Patent ID: 7457825

Claim:
A method in a device for generating a search request for a multimodal query with a query image and query text, the query image being stored in electronic form, the method comprising: providing a collection of images and associated words; generating a word-to-image index that maps words to associated images and an image-to-related-information index that maps images to associated keywords; receiving a multimodal query that includes a query image and query text; identifying images of the collection based on textual relatedness between an associated word and the query text; wherein the images are identified by searching the word-to-image index to locate images with associated words that are related to the query text; selecting images of the identified images based on visual relatedness between an identified image and the query image, wherein the selecting comprising extracting a feature vector for the identified image, determining the distance between the extracted feature vector and the feature vector of each image of the collection, and selecting the images based on the determined distance; generating a search request based on keywords associated with the selected images, wherein the search request is generated from keywords of the image-to-related-information index; submitting the generated search request to a search engine for identifying documents related to the multimodal query; and providing an indication of the identified documents as a search result for the multimodal query.