Gesture-based image shape filtering

Provided are techniques for gesture-based image shape filtering. At least one keyword in a search request for an image is received. A first set of images that are candidate matches to the requested image are identified based on the at least one keyword. A gesture-based image for a filter request is received. The first set of images are filtered using the gesture-based image to identify a second set of images that are candidate matches to the requested image.

FIELD

Embodiments of the invention relate to gesture-based image shape filtering.

BACKGROUND

While searching for an image, a user may provide a search request via text input. Then, in response to the search request, search results for images that are possible matches to the desired image are displayed.

Some systems allow a user to submit a sketch of an image and attempt to locate the desired image based on the sketch. For example, a sketch of a circle (e.g., a round shaped object) may result in images for a ball, a circular building, a flower, furniture, food, etc. being returned.

SUMMARY

Provided is a method for gesture-based image shape filtering. At least one keyword in a search request for an image is received using a processor of a computer. A first set of images that are candidate matches to the requested image are identified, using the processor of the computer, based on the at least one keyword. A gesture-based image for a filter request is received using the processor of the computer. The first set of images are filtered, using the processor of the computer, using the gesture-based image to identify a second set of images that are candidate matches to the requested image.

Also, provided is a computer program product for gesture-based image shape filtering. The computer program product comprises a computer readable storage medium having program code embodied therewith, the program code executable by at least one processor to: receive at least one keyword in a search request for an image, identify a first set of images that are candidate matches to the requested image based on the at least one keyword, receive a gesture-based image for a filter request, and filter the first set of images using the gesture-based image to identify a second set of images that are candidate matches to the requested image.

Moreover, provided is a computer system for gesture-based image shape filtering. The computer system includes one or more processors, one or more computer-readable memories and one or more computer-readable, tangible storage devices, and program instructions, stored on at least one of the one or more computer-readable, tangible storage devices for execution by at least one of the one or more processors via at least one of the one or more memories, to: receive at least one keyword in a search request for an image, identify a first set of images that are candidate matches to the requested image based on the at least one keyword, receive a gesture-based image for a filter request, and filter the first set of images using the gesture-based image to identify a second set of images that are candidate matches to the requested image.

DETAILED DESCRIPTION

FIG. 1illustrates a computing environment in accordance with certain embodiments. InFIG. 1, a computing device100includes a search system110. The search system110includes a filter system120. The computing device100is coupled to a data store150. The data store150stores one or more images160, one or more image profiles162, and an index170.

With embodiments, an image profile162is stored for each image160(either together or separately). In certain embodiments, the image profile162may include one or more of: a radius, a circumference, edge information, and contour. The image profile162is used to store information for the image160in the index170for easier location of the image162.

With embodiments, a user may input a gesture-based image (e.g., a sketch or a pattern) using, for example, a touch-based interface (e.g., a computer having a touch screen). The user may use a finger, stylus, etc. to input the gesture-based image. The filter system120generates an image profile for the gesture-based image, compares the generated image profile to stored image profiles for other images to identify one or more candidate (e.g., possible) matches to the gesture-based image.

FIG. 2illustrates pairs200,210,220,230of gesture-based images and associated images in accordance with certain embodiments. For this example, a user is searching for images of a particular flower. Flowers are of different shapes, and there are some shapes that are not easy to describe with text input (e.g., at least one keyword) or voice input. As an example, even if there was a specific name for the shape of the particular flower, the user may not know the specific name of the shape. InFIG. 2, there are flowers204,214,224,234in different shapes, and, with embodiments, the user is able to enter a gesture-based image202,212,222,232to try to locate the associated flower204,214,224,234.

FIG. 3illustrates, in a flow diagram, operations for identifying one or more images in accordance with certain embodiments.FIG. 3is formed byFIGS. 3A and 3B. Control begins at block300with the search system110receiving a search request for an image. The search request may be by text input, voice input, and/or with a gesture-based image. In block302, in response to the search request, the search system110identifies a first set of images that are candidate matches to the requested image. In certain embodiments, the search system110and/or the filter system120queries one or more data stores (e.g., one or more different data sources) to obtain the first set of images. In block304, the search system110displays the first set of images. In block306, the filter system120receives a gesture-based image for a filter request. In block308, in response to the filter request, the filter system120filters the first set of images using the gesture-based image to identify a second set of images (which are a subset of the first set of images) that are candidate matches to the requested image. From block308(FIG. 3A), processing continues to block310(FIG. 3B).

In block310, the filter system120displays the second set of images. In block312, the filter system120determines whether another filter request has been received. If so, processing continues to block308(FIG. 3A), otherwise, processing ends. Thus, a user may create a new gesture-based image, and the filter system120re-filters the current set of images based on the new gesture-based image.

In certain embodiments, there is an initial text based image search followed by gesture-based image filtering. For example, if a user issues a search request with text input of “flower”, a first set of images are displayed. Then, if the user submits a gesture-based image (e.g., draws a circle), the first set of images are filtered to provide a second set of images. The filtering is based on comparing image profiles of the requested gesture-based image and the images in the first set of images.

FIG. 4illustrates, in a flow diagram, operations performed to obtain images based on a gesture-based image. Control begins at block400with the filter system120generating an image profile for the gesture-based image. In block402, the filter system120compares the generated image profile to image profiles of candidate images (e.g., the images in the first set of images found in block302ofFIG. 3A) to identity a second set of images that match the gesture-based image. In block404, the filter system120ranks the images in the second set based on various factors that indicate how closely the generated image profile for the gesture-based image and the stored image profiles for the candidate images match. For example, if edge matching is used for comparison of image profiles, then, an image in the second set matching more edges to the requested image is ranked higher than an image in the second set matching fewer edges to the requested image. In certain embodiments, the filter system120uses a least squares technique to identify which images are closer to the gesture-based image, with more similar patterns being given higher ranking.

In certain embodiments, to generate the image profile for the requested gesture-based image, the filter system120captures different touch points of the gesture-based image and creates an equation of a shape based on a distance between two adjacent, incremental points of the gesture-based image. In certain embodiments, at an initial touch point, there may be many adjacent touch points, so embodiments use some incremental distance apart and then perform the least square technique.

In certain embodiments, for stored images, the filter system120computes the image profiles and stores the image profiles. In certain embodiments, the image profiles describe edges of the image, where the edges form a shape.FIG. 5illustrates an edge500on an image in accordance with certain embodiments.

In certain embodiments, for the gesture-based image, the filter system120discovers shapes based on edges found from the gesture-based image. Next, based on image edge recognition techniques, the filter system120identifies the shapes of any stored image objects that may match the shape of the gesture-based image.

In certain embodiments, if more than one image is discovered that matches the gesture-based image, then the filter system120computes an image profile based on shape dimensions of a shape of in each image. A shape dimension may be described as a feature or measurement of an image. Shape dimensions include, but are not limited to, location of the shape, orientation of the shape, and distance between reference points of the shape, etc.

As an example, if the user wants to search for a flower, then the user may enter text input of “flower”, and this will return search results of flowers. Next, the user may enter a gesture-based image, and the filter system120returns filtered search results. The filter system120determines directional parameters from the gesture-based image. Continuing with the example, if the user enters a gesture-based patter of a “U”, then tulips may be returned, while, if the user enters a gesture-based image of an inverted “U”, snow drops may be returned.

In certain embodiments, since shape dimensions from images are relative to each other, the filter system120may use the ratio of shape dimensions between shapes to provide accurate image matching.FIG. 6illustrates matches of flowers to gesture-based images in accordance with certain embodiments. For example, inFIG. 6, pair600illustrates that a larger circle within a circle gesture-based image matches one type of flower, while pair610illustrates that a smaller circle within a circle matches another type of flower.

In certain embodiments, the filter system120identifies shape dimensions. In certain embodiments, a user may specify shape dimensions. As another example, if a curve and a circle are identified, then the shape dimensions of the curve may be a radius (r1) and a perimeter (p1), while the shape dimension of the circle is a radius (r2). The filter system120may apply a quotient dot product operation against these two shapes so that the equation ([r1, p1]/[r2]) results in 2 ratio values ([r1/r2, p1/r2]). Then, the filter system120stores the images along with the image profile and indexes the images accordingly. In certain embodiments, the images are indexed based on the image profiles.

In certain embodiments, when a user is searching for an image, the user provides keywords for a query (e.g., “flower”) and receives a first set of search results. Then, for filtering, with finger gestures, the user draws one or more shapes as part of the image query submission. Now, the filter system120computes an image profile query in a similar manner for processing an image and queries an index along with the keywords. In certain embodiments, the filter system may rank images based on the closest keyword and image profile matches from the index and return a list of ranked images.

In certain embodiments, if the list of images is large, the user may optionally refine the image ranking by providing additional gesture-based shapes. As shapes are being drawn, the filter system120adjusts the image ranking dynamically to provide interactive feedback to allow drill down to a desired image.

Thus, with various embodiments, the user of a gesture-based image may be used as part of the initial query (e.g., with text input) and/or as filtering of a result set of images from the initial query.

Additional Embodiment Details

The code implementing the described operations may further be implemented in hardware logic or circuitry (e.g., an integrated circuit chip, Programmable Gate Array (PGA), Application Specific Integrated Circuit (ASIC), etc. The hardware logic may be coupled to a processor to perform operations.

FIG. 7illustrates a computer architecture700that may be used in accordance with certain embodiments. In certain embodiments, computing device100may implement computer architecture700. The computer architecture700is suitable for storing and/or executing program code and includes at least one processor702coupled directly or indirectly to memory elements704through a system bus720. The memory elements704may include local memory employed during actual execution of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during execution. The memory elements704include an operating system705and one or more computer programs706.

Input/Output (I/O) devices712,714(including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I/O controllers710.

The computer architecture700may be coupled to storage716(e.g., any type of storage device; a non-volatile storage area, such as magnetic disk drives, optical disk drives, a tape drive, etc.). The storage716may comprise an internal storage device or an attached or network accessible storage. Computer programs706in storage716may be loaded into the memory elements704and executed by a processor702in a manner known in the art.

The computer architecture700may include fewer components than illustrated, additional components not illustrated herein, or some combination of the components illustrated and additional components. The computer architecture700may comprise any computing device known in the art, such as a mainframe, server, personal computer, workstation, laptop, handheld computer, telephony device, network appliance, virtualization device, storage controller, etc.