Abstract:
A system performs user intention modeling for interactive image retrieval. In one implementation, the system uses a three stage iterative technique to retrieve images from a database without using any image tags or text descriptors. First, the user submits a query image and the system models the user&#39;s search intention and configures a customized search to retrieve relevant images. Then, the system extends a user interface for the user to designate visual features across the retrieved images. The designated visual features refine the intention model and reconfigure the search to retrieve images that match the remodeled intention. Third, the system extends another user interface through which the user can give natural feedback about the retrieved images. The three stages can be iterated to quickly assemble a set of images that accurately fulfills the user&#39;s search intention. They system can be used for image searching without text tags, can be used for initial text tag generation, or can be used to complement a conventional tagged-image platform.

Description:
RELATED APPLICATIONS 
       [0001]    This application claims priority to U.S. Provisional Patent Application No. 61/042,215 to Wen et al., entitled, “User Intention Modeling for Interactive Image Retrieval,” filed Apr. 3, 2008, and incorporated herein by reference. 
     
    
     BACKGROUND 
       [0002]    Content-based Image Retrieval (CBIR) has been extensively studied in recent years due to the explosive growth of online and offline image databases. Researchers in a number of different research areas have developed CBIR using different approaches. 
         [0003]    Researchers in the computer vision and machine learning areas tend to focus on fully automatic approaches that aim to train computers to automatically understand image content. Typical approaches include region-based image retrieval, image attention detection, and multi-instance learning. However, due to the extreme diversity of general image content, the computational cost, and the low-level nature of most vision-based image understanding algorithms, fully automatic CBIR is far from being a real application. 
         [0004]    Researchers in the multimedia processing community have taken a less ambitious approach by involving human interaction in the image searching process. One notable approach is the relevance feedback algorithm. It allows users to label positive and negative samples in order to iteratively improve the search results. This approach can indeed improve the search performance in some cases because of the human involvement. 
         [0005]    Unfortunately, the improvement is often limited and outweighed by the added trouble of manually labeling many samples. Like computer vision-based approaches, research on improving relevance feedback has focused on improving the feature extraction and automatic learning algorithms on the feedback samples. Inevitably, these approaches hit a similar bottleneck as the vision-based approaches, such as computational cost and the problem of using low-level features to describe high-level semantic content. 
         [0006]    The difficulties with CBIR and the intense demand for image search applications, especially for the Internet, have led commercial companies to take a different route to image searching/text-based image searching. Most current conventional image search engines take advantage of the cognitive ability of human beings by letting the human user label images with tags, and then conduct a text-based image search. This is a rather practical approach that can generate immediate results, but with great limitations. The acquisition of image tags, though it can be assisted by image metadata such as surrounding text and search annotations, can hardly obtain satisfactory results without brute force human labeling. Moreover, for large existing stock image collections and personal desktop photos, there is no surrounding text to assist the search. More importantly, images naturally contain much richer information than text, and thus can hardly be well represented by text alone. There is a great gap between text description and image content. The cliché “an image is worth a thousand words” is unfortunately true in most image search situations. Thus current text-based search results are far from satisfactory. 
       SUMMARY 
       [0007]    A system performs user intention modeling for interactive image retrieval. In one implementation, the system uses a three stage iterative technique to retrieve images from a database without using any image tags or text descriptors. First, the user submits a query image and the system models the user&#39;s search intention and configures a customized search to retrieve relevant images. Then, the system extends a user interface for the user to designate visual features across the retrieved images. The designated visual features refine the intention model and reconfigure the search to retrieve images that match the remodeled intention. Third, the system extends another user interface through which the user can give natural feedback about the retrieved images. The three stages can be iterated to quickly assemble a set of images that accurately fulfills the user&#39;s search intention. They system can be used for image searching without text tags, can be used for initial text tag generation, or can be used to complement a conventional tagged-image platform. 
         [0008]    This summary is provided to introduce the subject matter of user intention modeling for interactive image retrieval, which is further described below in the Detailed Description. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0009]      FIG. 1  is a diagram of an exemplary image retrieval framework. 
           [0010]      FIG. 2  is a block diagram of an exemplary intention-based image retrieval system. 
           [0011]      FIG. 3  is a diagram of an exemplary operational flow of the image retrieval system of  FIG. 2 . 
           [0012]      FIG. 4  is a diagram of an exemplary user interface displaying an intention modeling result on an intention list. 
           [0013]      FIG. 5  is a diagram of exemplary user interface of the image retrieval system. 
           [0014]      FIG. 6  is a diagram of an exemplary mini-toolbar interface for natural user feedback. 
           [0015]      FIG. 7  is a diagram of exemplary image retrieval results with and without intention deduction. 
           [0016]      FIG. 8  is a diagram of exemplary offline and online parts of the exemplary image retrieval system. 
           [0017]      FIG. 9  is a flow diagram of an exemplary method of iteratively refining a user&#39;s search intention for interactive image retrieval. 
       
    
    
     DESCRIPTION 
       [0018]    Overview 
         [0019]    This disclosure describes user intention modeling for interactive image retrieval. Instead of conventional techniques that try to retrieve images from the Internet or from a large database based on a descriptive text tag, the exemplary image retrieval system described herein turns the search focus first toward the user, and aims to provide accurate image retrieval by working through a refinement of the user&#39;s intention in seeking an image, i.e., modeling the visual details of what the user “has in mind.” This simple change in focus from object to user has dramatic implications for the efficacy of image search and retrieval. 
         [0020]    In one implementation, the exemplary system employs three interactive methods, components, or stages to allow the searching device to better capture the user&#39;s intention during content-based image retrieval. First, an exemplary “intention list” user interface (UI) induces a coarse determination of user intention, thereby breaking the search task down into smaller pieces, and narrowing the search space. Second, another exemplary interactive UI allows the user to draw multiple reference strokes on one or more images to specify user intention in detail, by being able to point out visual features or image aspects. The second exemplary method can combine clues from multiple examples to refine the intention results. Third, natural user feeding through associated UI&#39;s is utilized to collect both long-term and short term user relevance feedback data to boost the performance of the exemplary image retrieval system. 
         [0021]    In one implementation, the three exemplary stages just introduced are iterative and synergy between the three interactive mechanisms improves search efficiency, reduces user workload, and improves user experience. One advantage of the exemplary system is that the interacting mechanisms accommodate users with different expectations and intentions. Consequently the exemplary system greatly improves search efficiency and user experience across a broad spectrum of users. Exemplary UI designs also significantly improve the image retrieval performance. 
         [0022]    Significantly, the exemplary intention-based image retrieval system does not need to use any text tags or descriptive metadata. But on the other hand, the exemplary system can also be a powerful complement for conventional tag-based search systems, and provide functionality that boosts user experience to a level that seems impossible by conventional methods that are purely text-based. 
         [0023]    The exemplary system can also be used to accumulate or analyze image tags. For example, the exemplary system can help conventional tag-based systems to start from no tags at all, and gradually obtain tags through search-based annotation. In another scenario, the exemplary system can be used to post-process tag-based image search algorithms, to allow the user to handle ambiguous and noisy web image search results by simple interaction, which can greatly improve the user&#39;s experience of web-based image search engines. 
         [0024]    The exemplary user intention modeling described herein provides a quantifiable and principled approach to user intention modeling and applies the intention model to image retrieval. 
         [0025]    Exemplary System 
         [0026]      FIG. 1  shows an exemplary image retrieval framework  100  in which the image retrieval is based on modeling user intention. The image retrieval framework can be implemented on a single, standalone computing device, however a preferred distributed version is shown. A server  102  communicates with a user&#39;s (client) computing device  104  via the Internet  106  (or other network). The server  102  and the computing device  104  can be a desktop, notebook, or mobile computer, i.e., possessing a processor, memory, data storage, operating system, and other typical computer components. The server  102  hosts an exemplary intention-based image retrieval system  108 . The client computing device  104  also hosts an instance of the intention-based image retrieval system  108 ′ or components of a distributed version of the intention-based image retrieval system  108 ′. 
         [0027]    The intention-based image retrieval system  108  includes components to create an indexed image database  110 , obtaining images from available sources such as the Internet  106  or local filing systems and storage media. The image database  110  can exist on one or both of the server  102  and the client computing device  104 , or can exist in a distributed manner on a network. 
         [0028]    At the user&#39;s computing device  104 , the exemplary intention-based image retrieval system  108 ′ deploy exemplary user interfaces (UI&#39;s)  112  that receive user input to iteratively model the user&#39;s intention in seeking an image. In one implementation, each stage of the modeling uses one or more associated UI&#39;s  112 . The exemplary UI&#39;s  112  introduce respective interactive mechanisms at different levels of user-intention refinement to shorten the distance between the raw image database  110  and the user-specified image query target. 
         [0029]    Exemplary Engines 
         [0030]      FIG. 2  shows an example implementation of the intention-based image retrieval system  108  of  FIG. 1 , in greater detail. The intention-based image retrieval system  108  includes various engines that can be implemented in software and/or hardware. However, the illustrated components and configuration in  FIG. 2  is only one example for the sake of description. The intention-based image retrieval system  108  can be implemented in other configurations that may include variations in the number and type of components. 
         [0031]    Engine components are now listed. The illustrated intention-based image retrieval system  108  includes an image database generator  202 , an intention-modeling engine  204 , the images database  110 , a retrieval engine  208 , a buffer for image results  210 , and an optional interface for tag-based image platforms  212  i.e., for communicating with text or tag-based image platforms. 
         [0032]    The image database generator  202  may further include a crawler  214  to obtain images from local files or from the Internet  106 , an indexer  216 , and an image features extractor  218 . The features extractor  218  may include various engines and filters for extracting image features, such as a face detector  220 , a scene features extractor  222 , a color features extractor  224 , an attention modeling engine  226 , a facial local binary pattern (“LBP”) features extractor  228 , a texture features extractor  230 , . . . , and a color signature feature extractor  232 . The image database generator  202  creates a feature-indexed collection of images (or image pointers, such as URLs) that constitute the images database  110 . 
         [0033]    The intention-modeling engine  204  includes a UI engine  234  to generate and manage the UI&#39;s associated with each intention modeling stage, a coarse inference engine  236  to make an initial determination of user intention, an intention refinement engine  238  to determine more subtle aspects of the user&#39;s expectations, and a feedback iterator  240  to greatly boost the accuracy of modeling the user&#39;s intention in seeking an image. 
         [0034]    The coarse inference engine  236 , in turn, includes a query image input  242  to receive an image that can be used as a search criterion, and a query image parser  244  that includes an intention deduction engine  246 , that provides an initial intention assessment and places the intention assessment in the context of an intention list  248 . 
         [0035]    The intention refinement engine  238  includes a visual feature selection UI  250 , which in one implementation receives graphic input such as drawing, pencil, stylus, or paintbrush strokes that specify the visual aspects and features in example images that the user intends to find in an image being retrieved. In other words, the user can designate features across one or more images that should be present in the image(s) that the user is searching for. When the user designates features across multiple images, the feature combination engine  252  combines the features into coherent image search criteria. 
         [0036]    The feedback iterator  240  further includes a short-term module  254  that includes a relevant images designator  256  to collect user relevance feedback. For example, the relevant images designator  256  may provide a way for the user to designate a collection of images that are like the image(s) being sought. The feedback iterator  240  also has a long-term module  258  that may apply learned intention patterns  260  and historical log data  262  to the current short term user interaction with the relevant images designator  256 . That is, the long-term module  258  may detect that the user relevance feedback at the short-term module  254  is beginning to resemble a learned pattern  260 , and may invoke the learned pattern  260  to improve the image retrieval. 
         [0037]    The retrieval engine  208  receives modeled user intention data from any of the three stages of the intention-modeling engine  204  and configures a search from the query image, the designated visual features, and/or the user relevance feedback from the three stages. A search configuration engine  264 , therefore, includes a search method selector  266  to designate a search technique relevant to the current state of the modeled user intention, a feature selector  268  and a feature combiner  270  to form feature-based image search criteria (unless the feature combination engine  252  has already performed this task for second stage user-intention input), and a weight engine  272  to assign and prioritize an emphasis or importance to each search criterion thus created. 
         [0038]    In one implementation, the retrieval engine  208  uses the modeled intention data and/or the search configuration data to train a support vector machine (SVM)-based ranking engine  274 . The SVM-based ranking engine  274  drives an image scoring engine  276 , which prioritizes image search results retrieved from the image database  110 . 
         [0039]    Operation of the Intention-based Image Retrieval System 
         [0040]    The intention-modeling engine  204  solicits user input to iteratively model user intention as it relates to searching for and retrieving images that match the intention. In the initial query stage, the coarse inference engine  236  receives an image from the user via the query image input  242  and infers the intention of the user from the query image only. The query image parser  244  uses the intention deduction engine  246  to arrive at the most likely intentions, which are presented in the intention list  248 . Generally, at this stage, no additional user effort is required to inform the intention-modeling engine  204  of user intention. The entire image semantic domain is automatically divided and narrowed down into predetermined categories, so that better search results can be obtained. 
         [0041]    In a second stage, the visual feature selection UI  250  facilitates a multiple reference stroke interaction with the user. Multiple reference strokes provide input to handle cases in which adequate user intention cannot be expressed by the single query image. By asking the user to naturally draw strokes on multiple images at regions that the user thinks is “important”, the intention-modeling engine  204  learns which aspect in each of the images should be emphasized, and how such aspects should be combined. The stroke interactions of this second stage can also be used within just a single image to refine the derived user intention arrived at in the first stage. 
         [0042]    A third stage uses natural user feedback to help define user intention when user-submitted reference strokes in the second stage are not sufficient to describe the user intention. In one implementation, the relevant images designator  256  uses a collector, such as an “I like these” basket to enable the user to conduct relevance feedback naturally and effortlessly. As the user collects more subjectively “good” images to the basket via the feedback iterator  240 , the image retrieval system  108  incrementally improves the retrieved image results  210 . The long-term module  258  can leverage more data by matching the user behavior in the current session with learned patterns  260  and historical log data  262  from old sessions (which may also be from different users), using this borrowed information to help the current image search. 
         [0043]    As the intention-modeling engine  204  iterates from the first component (the coarse inference engine  236 ), to the third component (the feedback iterator  240 ), the amount of information leveraged by the retrieval engine  208  increases dramatically, while the user effort only increases moderately. 
         [0044]    Users have different performance needs, and therefore may choose to stop at any stage and use the current retrieved image results  210 . Novices may only use the default deduced intention provided on the intention list  248  by the intention deduction engine  246  to perform a one round query, while experts may want to use the reference stroke techniques of the intention refinement engine  238  and the natural feedback feature of the feedback iterator  240  to obtain very accurate results via providing more user information. 
         [0045]      FIG. 3  shows an example operational pipeline of the intention-based image retrieval system  108 . The user submits a query  302  via the query image input  242  that serves as initial search criterion. The intention deduction engine  246  determines the user&#39;s most likely intention within the context of predetermined categories (shown as a user intention list  248 ). The user can adjust  304  the results at this stage by using multiple reference strokes to designate features. Then formation of a search schema  264  follows. Retrieved search results  210  are displayed and the user submits feedback  306  that differentiates images that are aligned with the user intention from those images that are not. An image from the user feedback  240  may be looped back to the query image input  242  to start a subsequent iteration for refining the user intention. 
         [0046]    The operational pipeline of  FIG. 3  does not use any pre-assigned image tags, text, or metadata during formation of the search schema  264 . However, the exemplary image retrieval system  108  is not meant to replace tag-based image search methods. Yet, the exemplary system  108  can perform content-based image retrieval when tags are not available. Moreover, the exemplary image retrieval system  108  can easily be combined with tag-based techniques via the interface for tag-based platforms  212 , to provide multimodal search results. 
         [0047]    Determining User Intention From a Query Image 
         [0048]    In the coarse inference engine  236 , when the query intention of the user is provided via the image input  242 , then even a very rough (broad or general) intention greatly simplifies the image search since the search space is greatly narrowed down to a specific domain. For example, if the user intends to find human portraits, those images without faces are easily discarded. 
         [0049]    Conventional systems that consider user intention require the user to input categorical keywords or select from many predefined categories in order to determine a user intention. This causes the user additional effort and limits the potential image results because only a subset of the selected category is searched. 
         [0050]    The intention deduction engine  246  intelligently deduces the user intention in real time after the user submits the query image, and presents the most likely intentions in the “smart” intention list  248 , as shown in  FIG. 4 . The highlighted items in each of the shown intention lists  248  indicate the most probable intention deduced by the query image parser  244  for each query image. An experimental study has shown that the intention-modeling engine  204  deduces a correct user intention most of the time. 
         [0051]    In most cases, it takes no additional effort on the part of the user, other than submitting the query image, for the query image parser  244  to automatically determine a correct user intention category on the intention list  248  and thereby greatly narrow down the search. For example, as shown in  FIG. 4 , a first image  402  that contains a salient object results in a search for an “Object”; a second image  404  with scenery results in a search for a “Scene”; and a third image  406  with a distinct face results in a search for a “Portrait”. 
         [0052]    In applied experiments, it was observed that the user only rarely needs to modify the deduced search intention, but such modification is very easily accomplished by simply clicking manually on the desired search intention on the intention list  248 , as shown in  FIG. 4 . In one implementation, the user can click on a lock button to set the current intention, thereby fixing the search to only the selected search intention. 
         [0053]    Each user intention in the intention list  248  is associated with a specific search schema, in which the search configuration engine  264  selects optimal search features, feature combination, and search algorithms, and can be updated with accumulated user log data  262  to provide a better user experience when retrieving images with each of the search intentions. 
         [0054]    The user-experience of searching via the exemplary intention-based image retrieval system  108  is straightforward: the user submits a query image, and similar images are returned via the deduced intention. If the user changes the automatically proposed intention, then new retrieved image results  210  come up. The retrieved image results  210  are presented in a clean and straightforward user interface, for example, as shown in  FIG. 5 . 
         [0055]    In one implementation, for each of the displayed images in the retrieved image results  210 , a contextual mini-toolbar may appear when the user moves the mouse toward a specific boundary of each image, providing several functionalities related to the image, as shown in  FIG. 6 . 
         [0056]    It is worth noting that the mechanism of the query image parser  244  for forming an intention list  248  is quite different from conventional selection of a fixed search category, since in the query image parser  244  the user intention can be flexibly updated while the user searches with different query images. In this manner, user searches are conducted on the entire images database  110 , rather than in a subcategory of the images. The exemplary technique of searching by user intention results in each search using a different strategy to search the entire database  110  (rather than each search using a static search that searches just a categorical subset of the images). 
         [0057]    Without the intention deduction engine  246 , the retrieval engine  208  would have to perform image comparison and matching during a search using all features available, so that the aspect that the user wished to emphasize would be greatly diluted, and the search performance degraded significantly.  FIG. 7  shows a comparison of the retrieved image results  210  of two searches that begin with the same query image search criterion. Using exemplary intention deduction, each retrieved image is very similar in visual appearance and content meaning to the query image, providing accurate image search results. But when not using the exemplary intention deduction, some of the retrieved images may be vaguely similar but are irrelevant in their semantic meaning, e.g., a no-parking sign returned in a search for a national flag. 
         [0058]    Determining User Intention From Reference Stroke Interactions 
         [0059]    In some image searches, the intention-modeling engine  204  can use detailed information to determine the user&#39;s intention over just the intention deduction made from the single query image by the intention deduction engine  246 . This may be due to ambiguity in the query image, or difficulties in finding an example similar to the target image in order to begin a search. For instance, if the query image portrays two people, the intention-modeling engine  204  may not be able to tell which one of the two the user is focused on, or whether the user wants to search for both people at once. In an example, the user has only an image of a boy skiing to submit as a query image, but ultimately wants to find a different type of image with a portrait of the boy and a background of a girl skiing. 
         [0060]    The intention refinement engine  238 , therefore, allows the user to draw multiple strokes on images as an indication of a region of interest. A stroke is generally a drawn line, or a shape that selects by surrounding a visual object. For the user, operation of the intention refinement engine  238  is quite easy and straightforward: the user draws one or more strokes on regions of images that the user considers “good”. The feature combination engine  252  can combine the features designated by the multiple strokes to create a singular clue, cue, or search criterion for finding new images. For example, if the user wants to search for images with both a boy and his mother, but only has portraits of each separately, then the visual feature selection UI  250  allows the user to designate the boy on a first image and the mother on a second image. By specifying strokes on two images, images that have both the boy and mother will appear in the retrieved images display  210 . Thus, the intention refinement engine  238  can logically combine features across multiple images. 
         [0061]    In one implementation, the user-drawn strokes submitted at the visual feature selection UI  250  are analyzed on two levels. The intention refinement engine  238  seamlessly sends the user strokes to the search configuration engine  264  to let the user tune the search intuitively. First, the retrieval engine  208  considers the region of the image containing the stroke as an indication of the user&#39;s attention. In one implementation, the image retrieval system  108  uses this information to collaborate with an automatic attention detection algorithm, to adaptively find a user attention area in the images via the user strokes. The subsequent search is guided by these user attention results. For example, by designating a flower in a larger image, the user emphasizes the flower region, which leads to results that are more coherent with the user&#39;s intention of searching specifically for the flower. 
         [0062]    Second, if the stroke covers most of one dimension of a salient object (e.g., a depiction of a car, an animal, or a human face), then the stroke is a strong indication of the user&#39;s interest in this object. In this case, the intention refinement engine  238  may adjust the intention deduced in the first step at the coarse inference engine  236  so that the search configuration engine  264  may fashion a search schema more suited to the current, more nuanced user intention. For example, once a stroke covers a human face, face recognition will be added to the search schema. The SVM-based ranking engine  274  and the image scoring engine  276  then place images with people who have similar faces at the top of the retrieved images results. 
         [0063]    Determining User Intention From Natural User Feedback 
         [0064]    At times, even drawing strokes on single or multiple reference images via the intention refinement engine  238  cannot provide sufficient information to obtain accurate enough search results. In such cases, the feedback iterator  240  further leverages user input to allow the user to conduct relevance feedback through the relevant images designator  256 , a natural user interface. 
         [0065]    Conventional relevance feedback algorithms suffer from insufficient labeled data. Users are easily bored by “positive” and “negative” buttons attached to images. The feedback iterator  240  improves user experience by changing the mechanism of collecting labeled data. Instead of explicitly letting the user label an individual image as “positive” or “negative”, the relevant images designator  256  allows the user to add images to a collection of desirable images for the given search, dubbed in one implementation an “I like these” image basket. Dragging-and-dropping “good” images to the “I like these” image basket provides accumulatively better results in the iteratively retrieved image results  210 . That is, the resulting search is based on a collection of images and better results can be iteratively reposted each time an image is added. 
         [0066]    In one implementation, besides using drag-and-drop of an image into a basket of desirable images, the user can also use easy-to-access buttons in the mini-toolbar shown in  FIG. 6  to place images in the “I like these” image basket, e.g., without moving an input device, such as a mouse, very far. The mini-toolbar, such as that shown in  FIG. 6 , also greatly reduces the mouse movement distance needed to reach other frequently required functionalities related to the image, such as labeling an image as “not wanted” to remove it from the current search, putting the image into the “I like these” collection instead of dragging and dropping, viewing larger images, or tagging the image, if desired. 
         [0067]    For the image retrieval system  108 , implicit image labels are obtained at the feedback iterator  240  during the natural user feedback, and incrementally improve the search results  210 , which in turn stimulate the user to collect more “good” images. 
         [0068]    Long-Term Search Memory 
         [0069]    It has been observed that the way that different users understand the same image, although different in some ways, often has some correlation. Intuitively, the feedback iterator  240  can make use of this correlation and borrow information from previous queries to assist the current query. 
         [0070]    User behavior in the current search session is thus compared with sessions in the historical log data  262 , and several similar sessions may be found as matches or approximations. These matches can provide learned patterns  260  and other information for the current search, offering clues about both user intention and final target images. This is similar to accumulatively building an accurate ranking of search results by accumulating user input. Through such a process, the feedback iterator  240  makes use of previous records of other users as well as records of the current user. This greatly reduces the number of operations the user needs to perform to obtain desirable results. 
         [0071]    Generating An Image Database 
         [0072]    In one implementation, the exemplary image retrieval system  108  leverages several state-of-the-art computer vision and machine learning technologies.  FIG. 8  shows such an implementation of the image retrieval system  108  that uses a two-stage architecture: an offline stage  802  and an online stage  804 , i.e., offline indexing and online searching. Offline in this context means that the user is not searching for images online while the image database  110  is being built and indexed, but does not mean that the image database generator  202  itself is necessarily offline with respect to the Internet  106  or other network. 
         [0073]    In the offline stage  802 , images obtained by the crawler  214 , either from the Internet  106 , other network, or local file system  806  stored on a hard drive, are added into a queue of the indexer  216  in order for the features extractor  218  to obtain a series of features. The images are then stored to the structured images database  110 . Typical features extracted include face detection and alignment data  220 , facial Local Binary Pattern features  228 , scene descriptors  222 , texture features  230 , color features  224 , image attention model  226 , color signature  232 , etc. 
         [0074]    Online Operation 
         [0075]    In the online stage  804 , the user interactively operates the image retrieval system  108  through the exemplary UI&#39;s generated by the UI engine  234 , including the query image input (UI)  242 , the visual features selection UI  250 , and the relevant images designator (UI)  256 . These UI&#39;s allow the user to inform the image retrieval system  108  of the user&#39;s specific image query intention. 
         [0076]    The image retrieval engine  108  automatically configures itself with image features and feature combinations (e.g., via  252 ,  268 ,  270 ). The weight engine  272  assigns a relative importance factor to each feature and feature combination according to the search schema selected by the search method selector  266  for each intention. The intention-modeling engine  204  then takes the user&#39;s query image and user feedback as input, and trains the SVM-based ranking engine  274 . 
         [0077]    Finally, images in the database  110  are ranked according to scores given by the SVM-based ranking engine  274  and the image scoring engine  276 , then presented to the user as retrieved image results  210 . The user can then give feedback to the system through the feedback iterator  240 . With more feedback data, the image retrieval system  108  is automatically updated, and returns refined results  210  to the user. This looping occurs in an iterative manner until the user is satisfied with the current retrieved images results  210 . 
         [0078]    Interface For Tag-Based Platforms 
         [0079]    In one implementation, the image retrieval system  108  includes the interface for tag-based platforms  212 . The exemplary image retrieval system  108  can be combined with popular Internet image search engines to complement and boost the accuracy of text-based image searches. Many textual tags apply to multiple items. For example, an “apple” can be a fruit, a computer, or a music recording medium. “Palm” can apply to a type of tree or to the human hand. “Lincoln” can mean a person or a car. 
         [0080]    The exemplary image retrieval system  108  not only can filter images brought up via a text query, but can also provide visual uniformity across the retrieved image results  210 , when desired. For example, the image retrieval system  108  can ensure that “Lincoln” images returned during a search correspond to the user&#39;s intention of finding images of U.S. President Lincoln, and the image retrieval engine  108  can also ensure a return of images that only have a frontal view of Lincoln, for example, as opposed to side-views and other angles. This applies to the other features, for example, the image retrieval system  108  can return images that all include a specified dominant color. Thus, the image retrieval engine  108  can greatly enhance text-based image searching. 
         [0081]    Exemplary Methods 
         [0082]      FIG. 9  shows an exemplary iterative method  900  of user intention-based interactive image retrieval. In the flow diagram, the operations are summarized in individual blocks. The exemplary method  900  may be performed by combinations of hardware, software, firmware, etc., for example, by components of the exemplary image retrieval engine  108 . 
         [0083]    At block  902 , a query image is received. 
         [0084]    At block  904 , the user&#39;s search intention is inferred from visual contents of the image. It is worth noting that no text, tags, or metadata are used to characterize the query image. The user&#39;s intention is parsed from the query image itself. That is, the visual content of the query image is interpreted on the human perceptual level as high-level semantic information. 
         [0085]    At block  906 , the user&#39;s search intention is modeled. An initial coarse modeling places the user&#39;s rough search intention within a limited number of intention categories that can be displayed to the user as a list. In the rare event that the exemplary deduction techniques infer the incorrect intention, the user can designate the correct intention category with a single mouse click. 
         [0086]    At block  908 , a search is configured to match the intention. That is, different intention categories call for different search techniques. For example, if the user&#39;s intention is a portrait-type image, then the search technique includes a face detection algorithm. Relevant image features are also extracted from the query image to be combined into an image search criterion. 
         [0087]    At block  910 , images are retrieved from a database via the selected search technique and the search criteria fashioned from the extracted visual features of the query image. In one implementation, the method includes training a SVM-based ranking method, which scores the retrieved images with respect to how well they fulfill the modeled search intention. 
         [0088]    At branch number “1” and block  912 , the user decides whether the retrieved images meet expectations. If so, the method ends, but if not, then the method gathers more user input. 
         [0089]    At block  914 , another user interface is extended to receive the user&#39;s selection of visual features across one or more of the retrieved images. The user may select significant salient objects in one or more images, or may select part of an image that constitutes a visual feature to be used as a search criterion. The method then loops back to refining the intention model, at block  906 . 
         [0090]    After retrieving an updated set of retrieved images from the database, at branch number “2” and block  916 , the user decides whether the retrieved images meet expectations. If so, the method ends, but if not the method again collects further user input. 
         [0091]    At block  918 , another user interface is extended to receive the user&#39;s natural feedback about the latest set of retrieved images. In one implementation, the user drags and drops desirable images into a collection. With each drag, the set of retrieved images updates, via refining the intention model at block  906  and proceeding with reconfiguration of the search and retrieval of a new set of images. 
         [0092]    Then, at branch number “3” and block  920 , the user again decides whether the retrieved images meet expectations. If so, the method ends, but if not the method iteratively loops back to the beginning at block  902 , where one or more of the images of the latest set of retrieved images may be submitted as a query image to be parsed afresh for visual features that capture the user&#39;s search intention. 
         [0093]    Conclusion 
         [0094]    Although exemplary systems and methods have been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed methods, devices, systems, etc.