Patent ID: 11907337
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A method performed by one or more data processing apparatus, the method comprising:
for each image of a plurality of images:
processing pixels of the image by a textual generator model to obtain a set of phrases that are descriptive of the content of the image, wherein each phrase is one or more terms, wherein processing the pixels of the image by the textual generator model comprises:
receiving a plurality of outputs from a convolutional neural network of the textual generator model, each output of the plurality of outputs representing a possible query associated with the image;
projecting each output of the plurality of outputs to a low-dimensional vector, the low-dimensional vector representing an image embedding associated with a respective output;
producing a probability distribution for the plurality of outputs based on the low-dimensional vector associated with each output of the plurality of outputs, the probability distribution indicating an image-to-image similarity score for each output of the plurality of outputs; and
selecting a top-N most probable queries and associated similarity scores as the set of phrases to be obtained;

processing the set of phrases by a textual embedding model to obtain an embedding of predicted text for the image; and
processing the image using an image embedding model to obtain an embedding of image pixels of the image;

training a multimodal image classifier on the embeddings of predicted text for the images and the embeddings of image pixels for the images to produce, as output, labels of the output taxonomy to classify an image based on the image as input.