Object-sensitive image search

Methods and systems for object-sensitive image searches are described herein. These methods and systems are usable for receiving a query for an image of an object and providing a ranked list of query results to the user based on a ranking of the images. The object-sensitive image searches may generate a pre-trained multi-instance learning (MIL) model trained from free training data from users sharing images at websites to identify a common pattern of the object, and/or may generate a MIL model “on the fly” trained from pseudo-positive and pseudo-negative samples of query results to identify a common pattern of the object. As such, the user is presented with query results that include images that prominently display the object near the top of the results.

BACKGROUND

Text-based search engines traditionally retrieve relevant images or other elements (e.g., videos, documents, etc.) based on textual information. For example, a search engine may receive an image query “car,” and subsequently search billions of documents and ultimately provide relevant images based on textual information in the documents. These traditional search engines may determine “relevance” by considering such factors as link structure, link text, anchor text, or any other suitable textual information. For example, web images may be indexed with words from image titles, surrounding texts, or the like such that search engines determine a web image's relevance based on the image's title rather than the visual content of the image itself.

While text-based search engines may work well in returning the text-based documents, text-based search engines do not take into account the visual information of images, and therefore may provide inaccurate image-based query results.

SUMMARY

This summary is provided to introduce simplified concepts for object-sensitive image searches, which are further described below in the Detailed Description. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.

In some instances, the object-sensitive image searches described herein comprise re-ranking images based on the object information, which is obtained utilizing multi-instance learning (MIL). The technique is able to provide very high precision in returning images of objects (e.g., animals, plants, products, etc.) by prioritizing relevant query results in ranking lists for object-level queries by taking into account the visual information of search result images.

In one implementation, the object-sensitive image search techniques may be configured as an off-line system, which may utilize free public information to build a MIL model for object-sensitive re-ranking. In this implementation, a service provider's computing system (e.g., located on the “back end”) may train the MIL model.

In another implementation, the techniques may be configured as an on-line system, which may utilize pseudo-positive and pseudo-negative samples in query results to build a MIL model for object-sensitive re-ranking. In this implementation, a user's computing system (e.g., located on the “front end”) may train the MIL model.

DETAILED DESCRIPTION

Overview

This disclosure describes object-sensitive image searches. For instance, systems and methods are described for receiving a query for images of an object, generating query results comprising images that include the object, ranking the images, and providing a ranked list of query results to the user based on the ranking of the images. In some instances, the ranked list is based on a prominence (e.g., size, position, etc.) of the object within the images. For instance, in response to receiving a image query “car,” the systems and methods may return a ranked list of images that are ordered, at least in part, on the prominence of cars in the returned images—rather than simply the most relevant images as determined via the text or metadata associated with the images. As such, those images including large or centrally positioned cars may be provided at the top of the returned ranked list in this example.

Additionally or alternatively, the object-sensitive image searches may generate a pre-trained multi-instance learning model trained to identify a common pattern of the object (e.g., the car). In other examples, the object-sensitive image searches may generate a multi-instance learning model “on the fly” trained to identify a common pattern of the object.

As discussed above, while text-based search engines may provide very high precision in returning text-based documents, text-based search engines often do not take into account the visual information of images, and therefore may provide inaccurate image-based query results. However, this disclosure describes, in detail below, an object-sensitive image search system and method that may provide accurate image results to a user submitting an image-based query.

While the techniques described in this disclosure may be described with respect to images such as still images of objects, other forms of images are contemplated. For example, the techniques described in this disclosure may be described with respect to images such as videos, animations, or the like.

Illustrative Object-Sensitive Image Search Computing Environment

FIG. 1represents an illustrative computing environment in which an off-line approach and/or an on-line approach of object-sensitive image searches can be implemented. By way of example only,FIG. 1illustrates a user102performing a query104for images of an object (in this example, images of a “car”) via a computing device106. As illustrated, the computing device may include an object-sensitive ranking module108and a browser110stored in memory112and executable by processor(s)114. In some instances, however, the object-sensitive ranking module108may additionally or alternatively reside remotely from the computing device106, as discussed below.

Here,FIG. 1illustrates the computing device106communicatively coupled with a service provider116via a network(s)118. Service provider116may include one or more servers120(1)-120(N), which may comprise processor(s)122and memory124.FIG. 1further illustrates the memory124of the one or more servers120(1)-120(N) may include an object-sensitive ranking module126, a search engine128, and an image processing module130, a free image module132, and image data134, which may be executable by processor122.

Generally, when a user102submits a query104for an image of an object (e.g., “car”), the implementation illustrated inFIG. 1may utilize, in combination or individually, the object-sensitive ranking module126stored in memory124of the one or more servers120(1)-120(N) and/or the object-sensitive ranking module108stored in memory112of the computing device106. Both the object-sensitive ranking module108and the object-sensitive ranking module126may be configured to provide a ranked list of query results to the user102based on ranked images. For example, the computing environment illustrated inFIG. 1may utilize the object-sensitive ranking module126stored in memory124of the one or more servers120(1)-120(N) in an off-line approach to provide a ranked list of query results to the user102based on ranked images. Alternatively, the computing environment illustrated inFIG. 1may utilize the object-sensitive ranking module108stored in memory112of the computing device106in an on-line approach to provide a ranked list of query results to the user102based on ranked images.

In either instance, the module may provide a list that is ranked according to a prominence (e.g., size, position, etc.) of a searched-for object. For instance, in the example where the user102conducts a search for “car,” the module may analyze those images returned by traditional text-based techniques to determine a prominence of the cars therein. The module may then re-rank the images according to the detected prominence, such that an image that is largely occupied by a car may be ranked relatively higher than an image that includes a less-visible car or no car at all. By doing so, the object-sensitive ranking module108and/or126provides a better search experience to the user102than compared with traditional text-based-only techniques.

With respect to the off-line approach,FIG. 1illustrates the one or more servers120(1)-120(N) of the service provider116communicatively coupled, via the network(s)118, with a plurality of computing devices136(1)-136(N). Each of the computing devices136(1)-136(N) may comprise a browser138stored in memory140and executable by one or more processor(s)142. Further, each of the computing devices136(1)-136(N) may be utilized by a plurality of users144. These users144may employ the computing devices136(1)-(N) to share images at a plurality of websites, to submit a plurality of queries for images of objects, or the like.

In the off-line approach, the free image module132stored in memory124of the one or more servers120(1)-120(N) may monitor the queries submitted by the users at one or more search engines to determine which objects users tend to search for. With this information, the service provider116may select one or more objects for which to train multi-instance learning (MIL) models for use by the object-sensitive ranking module126. For instance, the service provider116may see that the users144most often search for “cars,” “flowers,” etc. and may correspondingly train MIL models for these respective objects.

After determining the objects for which to train MIL models, the free image module132may analyze images of these objects at a plurality of content provider(s)146(e.g., websites, etc.) to build pre-trained MIL models. For instance, the free image module132may analyze each photo at one or more photo sharing websites (e.g., Flickr®, etc.) that is associated (e.g., via a user-provided tag) with a particular object, such as “car.” The free image module130may then build a MIL model for the object “car” for use by the object-sensitive ranking module126. In this off-line implementation, each of the pre-trained MIL models may be trained to recognize a common pattern of the objects. With this information, the object-sensitive ranking module126may rank or re-rank search results for a subsequent query by comparing each image of the search results with the corresponding MIL model to determine whether the object is prominent or not within the image.

In contrast to the off-line approach, in the on-line approach the object sensitive ranking module108trains a MIL model “on the fly.” For example, when a user102submits a query104(e.g., “car”), the object-sensitive ranking module108may train a MIL model using the query results that are returned using traditional search-engine techniques. Here, in this implementation, the MIL model may be trained to recognize a common pattern of the object of the query104submitted by the user102. With this model, the object-sensitive ranking module108then re-ranks the images to place images that prominently display the searched-for object (e.g., “car”) near the top of the search results returned to the user.

Both the object-sensitive ranking module126stored in memory124of the one or more servers120(1)-120(N) and the object-sensitive ranking module108stored in memory112of the computing device106may utilize image data134stored in memory124, along with MIL models, to perform object-sensitive ranking and/or re-ranking. WhileFIG. 1illustrates image data134stored in memory124, image data134may be stored on memory112of computing device106. Both the object-sensitive ranking module126stored in memory124of the one or more servers120(1)-120(N) and the object-sensitive ranking module108stored in memory112of the computing device106may then utilize the image data134and the MIL models to perform object-sensitive ranking and/or re-ranking, whether the data is stored on memory124, memory112, or any other suitable memory store location.

Illustrative Off-Line Object-Sensitive Image Search System

FIG. 2is a block diagram of an example off-line approach of an object-sensitive image search system being implemented in the computing environment ofFIG. 1.FIG. 2illustrates that a back-end202of the system may comprise the memory124of the one or more servers120(1)-120(N) of the service provider116, while a front end204of the system may comprise memory112of the client computing device106. As discussed above, the memory124may include the object-sensitive ranking module126, the search engine128, the image processing module130, the free image module132, and the image data134for implementing, in part or in whole, object-sensitive image searches.

In this embodiment,FIG. 2illustrates that the free image module132may comprise a public query analyzer module206. The public query analyzer module206may further comprise a monitoring module208and an analyzing module210for monitoring and analyzing search-engine queries previously made by the users144. Generally, the public query analyzer module206may utilize the monitoring module208to monitor the queries submitted by the users144, and utilize the analyzing module210to determine which objects users144tend to search for. The public query analyzer module206may store the determined objects users144tend to search for in an object category212. The object category220may comprise a plurality of animals, plants, and/or products as objects users144tend to search for.

To identify the queries that users tend to search for, the public query analyzer module206may consult a query log214, which stores one or more queries216that the users144have previously conducted searches for. After analyzing the queries216submitted by the users, the free image module132may build a lexicon218containing each query216and its object220that belong to an object in the object category212. That is, the lexicon218of object queries may comprise a plurality of objects220from the plurality of queries216. For example, the plurality of objects220may comprise a plurality of animals, plants, and/or products that the public query analyzer module206may have determined users144tend to search for by monitoring the queries previously submitted by the plurality of users144. With knowledge of these objects, a MIL model may then be built for each of the objects, as described below.

To build a MIL model for each identified object220in the lexicon218, both positive training data222and negative training data224may then be gathered from one or more of the content providers146ofFIG. 1. In one example, the public query analyzer module206may analyze each photo at one or more photo-sharing websites (e.g., Flickr®, etc.) that is associated (e.g., via a user-provided tag) with a particular object, such as “car.” For example, the public query analyzer module206may label images as positive instances if users144have tagged the image with the object “car,” otherwise the images are label as negative instances. A MIL training module226may then train a pre-trained MIL model228to identify a common pattern of each object220. More specifically, the MIL training module226may train the pre-trained MIL models228utilizing both the positive training data222and negative training data224gathered for each object220in the lexicon218. Each of the pre-trained MIL models228may be stored in a set of MIL models230for use in providing a ranked list of query results to a user102.

FIG. 2further illustrates the off-line object-sensitive image search system includes the search engine128ofFIG. 1. The off-line object-sensitive image search system illustrated here inFIG. 2may receive a query104from browser110via the search engine128. Search engine128may include results module232, which may provide image(s)234associated with the query results. The image processing module130ofFIG. 1may include a segmentation module236and a feature extraction module238for processing the image(s)234provided by the search engine128. For example, the image processing module130may be configured to segment each of the images234into region(s)240and extract features from the region(s)240. The regions240and features may be used by a pre-trained MIL model228to select images234from the query results comprising a common pattern of an object220in a region240.

FIG. 2illustrates the object-sensitive ranking module126, illustrated inFIG. 1, may comprise a MIL selector module238configured to select a pre-trained MIL model228from the set of MIL models230for a query104that matches an object220in the received query. For instance, in response to receiving a query for “car,” the MIL selector module238may select a MIL model associated with the object “car.” Here,FIG. 2illustrates the pre-trained MIL model228may be configured for image selection244, which, as discussed above, may be for selecting an image234from the query results comprising the common pattern of the object220in the regions240.FIG. 2further illustrates the object-sensitive ranking module126may also comprise a ranking module242, which may comprise a region quality module248. The region quality module248may further comprise a region size module250, a region position module252, a region quantity module254, and a common pattern time module256. Generally, the region quality module248may provide for ranking image(s)234selected by the pre-trained MIL model228based on a quality of each of the region(s)240comprising the common pattern of the object220. For example, the region quality module248may utilize the region size module250, the region position module252, the region quantity module254, and/or the common pattern time module256to determine individually or in any combination a region size, a region position, a region quantity, and/or an amount of time a common pattern is in a region, respectively. The ranking module242may then provide a ranked or a re-ranked list of the query results based on the ranked images to a computing device106via a browser110for viewing by a user102. For instance, the ranking module242may re-order a set of image results to place images that are largely occupied by a “car” higher up on the list than those images that have small cars or cars that are not near the center of the image.

Illustrative Off-Line Object-Sensitive Image Search Process

The processes, described herein, are illustrated as a collection of blocks in a logical flow graph, which represent a sequence of operations that can be implemented in hardware, software, or a combination thereof. The collection of blocks is organized under respective entities that may perform the various operations described in the blocks. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the processes. The operations in the processes are aligned with possible devices/actors that may perform these operations. However, the illustrated arrangement is only illustrative and other device/actor arrangements may be used to implement to processes. This applies to each process described herein.

FIG. 3is a flowchart illustrating details of a process of an off-line approach of an object-sensitive image search implementation utilizing the example off-line object-sensitive image search system ofFIG. 2.

In this particular implementation, the process300may include block302, which may represent analyzing a plurality of queries from a query log, such as the query log214. For example, the public query analyzer module206may monitor the queries submitted by the users144at one or more search engines to determine which objects users tend to search for. With this information, the public query analyzer module206may select one or more objects for which to train MIL models for use by the object-sensitive ranking module126. The process300may further include block304, which represents generating a lexicon of object queries. As discussed above, the lexicon218ofFIG. 2may be generated from the query log214. Here, the free image module132may generate the lexicon218containing each query216and its object220that belong to an object in the object category212that the public query analyzer module206may have determined users144tend to search for.

The process300may also include block306, which represents generating positive training data222and negative training data224. Here, the public query analyzer module206may analyze photos at one or more photo sharing websites that are associated with the objects users tend to search for. The public query analyzer module206may label images as positive instances if users144have tagged the image with the object, otherwise the images are label as negative instances. For instance, the module206may identify those photos that have been tagged with “car” for use as positive training data222, as well as those photos that have not been tagged with “car” for use as negative training data224.

At block308, the MIL training module226may train MIL models to identify a common pattern of each object220. For example, block308may represent the MIL training module226training MIL models to identify a common pattern of each object220utilizing both the positive training data222and negative training data224ofFIG. 2. For instance, the MIL training module226may train a model for the object “car.” Block310, meanwhile, may represent the MIL training module226ofFIG. 2forming a set of pre-trained MIL models230comprising trained MIL models228.

The process300may also include block312, which may represent the search engine128receiving a query104from a user102, such as the query “car” or another query that includes the object “car.” Block312may be followed by block314, which may represent the search engine128generating query results, which may comprise image(s)234in instances where the user102is conducting an image-based search. Next, at block316the image processing module130may segment each of the returned images234into region(s)240and may extract features from these region(s)240. Block316may be followed by block318, which may represent an object-sensitive ranking module126selecting a pre-trained MIL model228. Here, the object-sensitive ranking module may comprise a MIL selector module238configured to select a pre-trained MIL model228from the set of MIL models230for a query104that matches an object220in the received query. For instance, the module may select a MIL model that has been trained for the object “car” in response to receiving the query “car” or a query that includes the term “car.”

The process300may also include block320, which may represent running the selected pre-trained MIL model228against the images234from the query results to identify those image(s)234that include a common pattern of an object220in one or more regions240. For example, block320may represent comparing the extracted features from the region(s)240to the common pattern of the object220to identify image(s)234from the query results that include the common pattern of the object220. The process300may then include block322, which may represent ranking image(s)234of the query results402. For example, block322may represent the ranking module242ranking the images234that are determined to include the common pattern of the object220with reference to the pre-trained MIL model228. Here, the ranking module242may rank the selected images234based on a quality (e.g., a region size, a region position, a region quantity, and/or an amount of time a common pattern is in a region) of each of the regions240comprising the common pattern of the object220. Finally, the process300may include block324, which may represent providing ranked images to a user (e.g., user102). For example, block324may represent providing the ranked list of the query results to the user based on the ranking of the selected image(s)234.

FIG. 4represents, in more detail, the off-line approach of the object-sensitive image search computing environment ofFIG. 1. Generally,FIG. 4may illustrate the off-line approach of the object-sensitive image search providing a ranked list of query results to a user based on the ranking of the images. Here,FIG. 4may illustrate a user102initiating a query104for an image of a “car” via the browser110of device106. Next,FIG. 4illustrates the device106passing the query104from the front-end204over to the search engine128on the back-end202. The search engine128may then generate query results402comprising image(s)234via the results module232. The results module232may utilize traditional techniques to generate the query results, such as performing text matching between the received query and metadata associated with the searched images (e.g., tags, etc.).

Subsequent to the results module generating query results402comprising image(s)234, the image processing module130may then receive the search result image(s)234for processing. For example, the image processing module130may segment each of the image(s)234into region(s)240using the segmentation module234. Here, the segmentation module234may use a JSEG image segmentation algorithm, as proposed inUnsupervised Segmentation of Color-Texture Regions in Images and Video, published August 2001, by Yining Deng, incorporated herein by reference, to segment each of the image(s)234into one or more region(s)240. WhileFIG. 4may illustrate the segmentation module234using a JSEG image segmentation algorithm, any other suitable image algorithm may be used to segment each of the image(s)234into region(s)240.

The image processing module130may also use the feature extraction module to extract features from each segmented region(s)240.FIG. 4illustrates the image processing module130may provide the image(s)234and respective region(s)240as image data134for use by the object-sensitive ranking module126. Further,FIG. 4illustrates the object-sensitive ranking module126may comprise the MIL selector module238, which may select pre-trained MIL models238from the set of MIL models230.

FIG. 4illustrates the set of MIL models230may have been generated by the free image module132. Here, the free image module132is illustrated as gathering training data from image-sharing websites. Specifically, the free image module132is illustrated inFIG. 4as being communicatively coupled with the network(s)118and monitoring users144performing image sharing acts (e.g., sharing images at websites, submitting queries for images of objects, or the like).FIG. 4illustrates that the free image module132may comprise the public query analyzer module206, the object query lexicon218, the MIL training module226, and the set of trained MIL models230.

Generally, the free image module132may be for generating MIL models using free training data from monitoring users sharing images at websites. Further, the MIL models are generally for identifying objects in images and handles the problems in which the labels are not available for each individual instance but only for bags of instances. For example, multi-instance learning involves labeling a bag as positive if it contains a positive instance, and labeling a bag as negative if all the instances in it are negative. Therefore MIL may be suitable for identifying objects in images, since generally information regarding whether images contain a specific object or not is available. More specifically, when applying MIL algorithms in image classification, each image may be a bag and the regions may be instances. In addition, MIL algorithms not only are able to classify bags but also identify positive instances. This means that, given a specific object such as “car” and several positive and negative training images (i.e., if an image contains the object, it is positive; otherwise it is negative), MIL algorithms may be able to judge whether a new image contains the object and identify which region contains the object. Here, in this implementation of the off-line approach of the object-sensitive image search, the free image module132may analyze public queries using the public query analyzer module206and generate a lexicon218of this data for use by the MIL training module226. The MIL training module226may then train the MIL models230for web images using the lexicon218of free training data from monitoring users sharing images at websites.

Turning back to object-sensitive ranking module126ofFIG. 4, the ranking module242may rank images selected by the trained MIL models230. For example, the ranking module242may rank the region(s)240based on a quality of each of the region(s)240. Specifically, the region quality module248may determine individually or in any combination a region size, a region position, a region quantity, and/or an amount of time a common pattern is in a region, as qualities of the region(s)240. The ranking module242may then generate a ranked list404of the query results image(s)142and provide the ranked list of query results to the user102. Here,FIG. 4illustrates the ranked list404may be provided to the browser110of computing device106for viewing by the user102.FIG. 4illustrates the ranked list404, provided by the off-line approach of the object-sensitive search, may comprise the largest, most clear image as a highest-ranking image406(i.e., a first image) and a smaller image408as a lower-ranking image (i.e., a last image in the example set). WhileFIG. 4illustrates the ranked list404comprising four images, the ranked list404may comprise any other number of one or more images.

Illustrative On-Line Object-Sensitive Image Search System

FIG. 5is a block diagram of an example on-line approach to an object-sensitive image search system being implemented in the computing environment inFIG. 1. Here, in this implementation, memory112of computing device106may comprise an object-sensitive ranking module108, which may comprise similar modules as the object-sensitive ranking module126of the off-line object-sensitive image search system described inFIG. 2. For example, the object-sensitive ranking module108may comprise the ranking module242. Further, and as discussed above, the ranking module242may comprise the region quality module248, which may comprise the region size module250, the region position module252, the region quantity module254, and/or the common pattern time module256. The MIL training module226and the browser110may also be stored in memory112of the on-line object-sensitive search system ofFIG. 5.

Here, in this implementation, the MIL training module226may utilize pseudo training samples502stored in memory112of computing device106to train a MIL model504.FIG. 5illustrates the pseudo training samples502may comprise pseudo-positive samples506and pseudo-negative samples508. WhileFIG. 5illustrates the memory112of computing device106comprising pseudo training samples502, pseudo training samples502may be stored in any suitable memory location, in whole or in part. For example, a portion of the pseudo training samples502may be stored in memory112, while another portion may be stored in memory124. Further, pseudo training samples502may be entirely stored in memory124, or alternatively, entirely in memory112.

FIG. 5further illustrates that the memory112may include a pseudo-relevance module510, which may select the pseudo training samples502. Here, the pseudo-relevance module510may utilize a re-ranking algorithm to select pseudo-positive samples506and pseudo-negative samples508from query results402generated by the results module232of search engine128. For example, the pseudo-relevance module510may utilize pseudo-relevance feedback (PRF) as described, by Rong Yan, Alexander Hauptmann and Rong Jin, inMultimedia Search with Pseudo-Relevance Feedback, published Jul. 24-25, 2003 and incorporated herein by reference, to select pseudo-positive samples506and pseudo-negative samples508. While the pseudo-relevance module510may utilize PRF as described inMultimedia Search with Pseudo-Relevance Feedback, any other relevance feedback, suitable for selecting positive and negative samples from query results402, may be utilized by the pseudo-relevance module510.FIG. 5further illustrates memory124may also comprise a query analyzer module512configured to determine the object220of a query submitted by user102.

Similarly, and as discussed above, in the on-line object-sensitive search system ofFIG. 5, the memory124may comprise the image processing module130. The image processing module130may include the segmentation module234and the feature extraction module236for processing the image(s)234generated by the search engine128. For example, the image processing module130may be configured to segment each of the images234into region(s)240and extract features from the region(s)240. Further, the segmented image(s)234and region(s)240may be stored as image data134in memory124. Here, in this implementation, the region(s)240and features may be used by a trained MIL model504to select image(s)234from the query results comprising a common pattern of an object220in the region(s)240.

FIG. 5illustrates the memory112may also comprise image data134. The image data134may be provided by the one or more sever(s)120(1)-120(N) to computing device106to identify, with use of the trained MIL model504, image(s)234from the query results comprising a common pattern of the object220. WhileFIG. 5illustrates the memory124storing the image processing module130, in other instances the memory112may store this module130. For example, the search engine128of the one or more sever(s)120(1)-120(N) may process a user's (e.g., user102) query (e.g., query104) and provide the query results to an image processing module130located in memory112of the computing device106.

Illustrative On-Line Object-Sensitive Image Search Process

FIG. 6is a flowchart illustrating details of a process of an on-line approach of an object-sensitive image search implementation utilizing the example on-line object-sensitive image search system ofFIG. 5. In this particular implementation, the process600may include block602, which may represent the search engine128receiving a query106for an image of an object220from a user102.FIG. 6illustrates block602may be followed by block604, which may represent the search engine128results module232generating query results which may comprise image(s)234of the object based on the query106. For instance, the results module232may utilize traditional text-matching techniques to produce preliminary query results, as discussed above.

The process600may also include block606, which may represent the image processing module130segmenting each of the image(s)234into region(s)240and extracting features from the region(s)240. The process600may continue with block608, which may represent the pseudo-relevance module510selecting pseudo training samples502. Here, the pseudo training samples502may be selected from the query results402generated by the results module232of the search engine128. Further, the pseudo-relevance module510may select pseudo-positive samples506and pseudo-negative samples508from the query results402generated by the results module232.

The process600may further include block610, which represents the MIL training module226training a MIL model504to identify a common pattern of the object220. For example, block610may represent the MIL training module226training the MIL model504“on the fly” to identify a common pattern of the object220utilizing both the pseudo-positive training samples506and pseudo-negative training samples508ofFIG. 5. WhileFIG. 5illustrates the memory112comprising the MIL training module226, in other instances the memory124may comprise the MIL training module226. For example, the MIL training module226may train the MIL model504on the back-end202and subsequently hand the trained MIL model504to the object-sensitive ranking module108on the front-end204. Further, the MIL training module226may also store the “on the fly” trained MIL model504in memory for later use. For example, the MIL training module226may store the trained MIL model504in a set of MIL models in memory for later use by the object-sensitive ranking module108. For example, a MIL selector module (e.g., MIL selector module238) may be included in memory112and may be configured to select a trained MIL model504from a set of MIL models for a query104that matches an object220of a user's102query104.

The block610may be followed by block612, which may represent selecting images234from the query results comprising a common pattern of the object220in the region240with use of the trained MIL model504. The process600may include block614, which may represent ranking images234of the query results402. For example, block614may represent the ranking module242ranking the images234identified with use of the trained MIL model504. Here, as discussed above, the ranking module242may rank the selected images234based on a quality (e.g., a region size, a region position, a region quantity, and/or an amount of time a common pattern is in a region) of each of the regions240comprising the common pattern of the object220. For example, in one implementation, the ranking module242may rank all the images234from a largest calculated region size comprising the common pattern as a highest-ranking image, to a smallest calculated region size comprising the common pattern as the lowest-ranking image. Block614may be followed by block616, which may represent re-ranking the query results based on the ranking of the selected images performed at block614. Process600may be completed at block618, which may represent providing the re-ranked query results to the user.

FIG. 7represents, in more detail, the on-line approach of the object-sensitive image search computing environment ofFIG. 1. Generally,FIG. 7may illustrate the on-line approach of the object-sensitive image search providing a re-ranked list of query results to a user based on the ranking of query images. Here,FIG. 7may illustrate a user102initiating a query104for an image of a “car” on the browser110of device106. Next,FIG. 7illustrates the device106passing the query104from the front-end204over to the search engine128on the back-end202. The search engine128, on the back-end202, may then generate query results402comprising image(s)234via the results module232. Subsequent to the results module232generating query results402comprising image(s)234, the image processing module130may then receive the query results402image(s)234for processing. For example, the image processing module130may segment each of the image(s)234into region(s)240using the segmentation module234. Similarly, as discussed above, the segmentation module234may use the JSEG image segmentation algorithm to segment each of the image(s)234into region(s)240. Again, whileFIG. 7may illustrate the segmentation module234using a JSEG image segmentation algorithm, any other suitable image algorithm may be used to segment each of the images into regions. The image processing module130may also use the feature extraction module to extract features from each segmented region(s)240.

FIG. 7illustrates the image processing module130may provide the image(s)234and respective region(s)240as image data134for use by the object-sensitive ranking module108on the front-end204. WhileFIG. 7illustrates the image processing module130processing and storing the image data134on the back-end202, the image processing module130may alternatively be located on the front-end204. For example, the image processing module130may be stored in memory112, where the image processing module130may then process and store the image data134.

Here, in this implementation of the on-line approach of the object-sensitive image search, the pseudo-relevance module510may select pseudo-positive samples506and pseudo-negative samples508as pseudo training samples502from the query results402generated by the results module232. The MIL training module226may then train a MIL model504to identify a common pattern of the object220in the region(s)240of the image(s)234. Further, the trained MIL model504may be configured for image selection228, in which the trained MIL model504may select, from the query results402, images comprising the common pattern of the object220in the region(s)240. The ranking module242comprising the region quality module248may then rank the image(s)234based on a quality of each of the region(s)240. Specifically, the region quality module248may determine individually or in any combination a region size, a region position, a region quantity, and/or an amount of time a common pattern is in a region, as qualities of the region(s)240. The ranking module242may then generate a ranked list of the query results image(s)142. The object-sensitive ranking module108may then re-rank the query results402based on the ranking of the selected images.

Here,FIG. 7illustrates the re-ranked list702may be provided to the browser110of computing device106for viewing by the user102.FIG. 7illustrates the re-ranked list702, provided by the on-line approach of the object-sensitive search, may comprise the largest, most clear image as a highest-ranking image704(i.e., a first image) and a smaller image706as a lower-ranking image (i.e., a last image in the example set). WhileFIG. 4illustrates the ranked list404comprising four images, the ranked list404may comprise any other number of images.

Conclusion

Although embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the disclosure is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as illustrative forms of implementing the embodiments.