Patent Document ID: 20120109858
Application ID: 12914653
Patent Status: 0

Claim One:
1. A computer-implemented method performed by a data processing apparatus, the method comprising: receiving a joint image-audio query from a client device, the joint image-audio query including query image data defining a query image and query audio data defining query audio; determining query image feature data from the query image data, the query image feature data describing image features of the query image; determining query audio feature data from the audio data, the query audio feature data describing audio features of the query audio; providing the query image feature data and the query audio feature data to a joint image-audio relevance model, the joint image-audio relevance model trained to generate relevance scores for a plurality of resources, wherein each resource includes resource image data defining a resource image for the resource and text data defining resource text for the resource, and wherein each relevance score is a measure of the relevance of corresponding resource to the joint image-audio query; ordering the resources according to the corresponding relevance score; and providing data defining search results indicating the order of the resources to the client device.