Patent Description:
Recently, large-scale, multimodal training of large machine-learned models (e.g., transformer-based models), etc. has led to improvements in many different domains, such as computer vision, language understanding, audio processing, etc. For example, in the domain of computer vision tasks, a single large pre-trained machine-learned model (e.g., a deep learning model) can often outperform multiple smaller, task-specific models. "<NPL>et al. , presents an investigation of a variety of Modality-Shared Contrastive Language-Image Pre-training (MS-CLIP) frameworks".

The invention is defined in the claims.

One example aspect of the present disclosure is directed to a computer-implemented method for pixel-based machine-learned models for multimodal vision-language tasks. The method includes obtaining a first image and textual content associated with the first image. The method includes rendering a second image that depicts the textual content associated with the first image. The method includes processing the first image and the second image with a machine-learned encoding model to respectively obtain a first image embedding and a second image embedding for an image embedding space comprising a plurality of image embeddings. The method includes training the machine-learned encoding model based on a difference between the first image embedding and the second image embedding.

Another aspect of the present disclosure is directed to a computing system for pixel-based machine-learned models for multimodal vision-language tasks. The computing system includes one or more processors. The computing system includes one or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations. The operations include obtaining a first image and textual content associated with the first image. The operations include rendering a second image that comprises the first image and a rendering of the textual content associated with the first image. The operations include processing the second image with a machine-learned image transformer model to obtain an image embedding of the second image for an image embedding space, wherein the image embedding space comprises a plurality of image embeddings generated using the machine-learned image transformer model. The operations include retrieving one or more image embeddings from the image embedding space based on a similarity between the one or more image embeddings and the image embedding of the second image. The operations include using the one or more image embeddings to perform a task associated with at least one of the first image or the textual content associated with the first image.

Another example aspect of the present disclosure is directed to one or more tangible, non-transitory computer readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations. The operations include obtaining textual content from a requesting entity. The operations include generating an image that depicts a rendering of the textual content. The operations include processing the image with a machine-learned image transformer model to obtain an image embedding of the image for an image embedding space, wherein the image embedding space comprises a plurality of image embeddings generated using the machine-learned image transformer model. The operations include retrieving one or more image embeddings of the plurality of image embeddings from the image embedding space based on a similarity between the one or image embeddings and the image embedding of the image. The operations include providing one or more images respectively associated with the one or more image embeddings to the requesting entity.

Generally, the present disclosure is directed to machine-learned models. More particularly, the present disclosure relates to exclusively pixel-based machine-learned encoding models for multimodal vision-language tasks. For example, a computing system can obtain a first image (e.g., an image depicting an animal, etc.) and textual content associated with the first image (e.g., a description of the species of animal, characteristics of the animal, etc.). The computing system can render an image that depicts the textual content associated with the first image. The computing system can process the first image and the second image with a machine-learned encoding model (e.g., a machine-learned image transformer model, etc.). The machine-learned encoding model can be a model that exclusively processes image data (e.g., pixels, etc.).

By processing the first and second images with the machine-learned encoding model, a first image embedding and a second image embedding can be obtained for an image embedding space. The image embedding space can include a plurality of image embeddings. The computing system can train the machine-learned encoding model based on a difference between the first image embedding and the second image embedding. For example, the computing system may utilize a contrastive learning process that minimizes a difference between the first image embedding and the second image embedding, and maximizes a difference between the pair of the first and second image embeddings and the rest of the image embeddings within the image embedding space. Once trained, the computing system can utilize the machine-learned encoding model to perform visual tasks, language tasks, or multimodal vision-language tasks. For example, the computing system can use the model to generate an image embedding from an image, renderings of textual content, or a combination of both, and then use the image embedding in conjunction with the image embedding space to perform various vision/language tasks (e.g., semantic image analysis, sentence classification, answer retrieval, image classification, etc.).

Aspects of the present disclosure provide a number of technical effects and benefits. As one example technical effect and benefit, conventional machine-learned models generally include a discrete model component for each modality of a multimodal task such as vision-language tasks. For example, many conventional models for vision-language tasks include an image encoder and a text encoder. As each discrete model component requires its own set of parameters, values, etc., the training of each model component incurs a substantial cost in computing resources (e.g., power, memory, bandwidth, compute cycles, storage, etc.). However, aspects of the present disclosure facilitate multimodal vision-language tasks with a single machine-learned model via rendering of textual content as an image, therefore eliminating the computing resource cost associated with training of multiple model components (e.g., discrete image and text encoders, etc.).

For another example, as described previously, conventional models generally utilize discrete image encoders and text encoders for multimodal vision-language tasks. However, text encoders often require extensive pre-processing of textual content before it can be properly processed by the text encoder. For example, many text encoders can only process token representations generated from the textual content, which requires the expenditure of substantial quantities of computing resources. Furthermore, many text encoders are language-specific, and require that textual content first be translated to the language in which the encoder was trained before processing. This translation also requires substantial quantities of computing resources, and introduces a considerable vector for decreasing model accuracy due to the errors, inaccuracies, and mistranslations inherent to machine translation of languages. For example, it can be challenging to tokenize certain language as the quantity of tokens available for tokenization is often limited. However, aspects of the present disclosure facilitate language-agnostic processing of textual content. In particular, by rendering textual content to an image, the machine-learned encoding models of the present disclosure can be trained to generate accurate embeddings without requiring any pre-processing of textual content (e.g., tokenization, machine translation, etc.), therefore generating more accurate results and eliminating the expenditure of computing resources for pre-processing that is required by conventional techniques.

<FIG> depicts a block diagram of an example computing system <NUM> that performs training of a pixel-based machine-learned encoding model for multimodal vision-language tasks according to example embodiments of the present disclosure. The system <NUM> includes a user computing device <NUM>, a server computing system <NUM>, and a training computing system <NUM> that are communicatively coupled over a network <NUM>.

The user computing device <NUM> can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

The user computing device <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the user computing device <NUM> to perform operations.

In some implementations, the user computing device <NUM> can store or include one or more pixel-based machine-learned encoding models <NUM>. For example, the pixel-based machine-learned encoding models <NUM> can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example pixel-based machine-learned encoding models <NUM> are discussed with reference to <FIG>.

In some implementations, the one or more pixel-based machine-learned encoding models <NUM> can be received from the server computing system <NUM> over network <NUM>, stored in the user computing device memory <NUM>, and then used or otherwise implemented by the one or more processors <NUM>. In some implementations, the user computing device <NUM> can implement multiple parallel instances of a single pixel-based machine-learned encoding model <NUM> (e.g., to perform parallel multimodal vision-language tasks across multiple instances of the pixel-based machine-learned encoding model).

More particularly, the pixel-based machine-learned encoding model <NUM> can be trained and utilized to perform computer vision tasks, language tasks, and multimodal vision-language tasks. In particular, the pixel-based machine-learned encoding model <NUM> can process image data (e.g., pixels, etc.) alongside textual content rendered as image data, to perform multimodal vision-language tasks. For example, textual content rendered as an image can be processed by the pixel-based machine-learned encoding model <NUM> to obtain an image embedding. The image embedding can be used to retrieve other image embeddings from image embedding space <NUM>. The image embedding space can include a plurality of other image embeddings. For example, the pixel-based machine-learned encoding model <NUM> may process large numbers of images, or pairs of images (e.g., two similar images, a first image and a rendering of textual content descriptive of the first image, etc.) to populate the image embedding space <NUM>. The retrieved image embeddings can be utilized to perform various multimodal vision-language tasks (e.g., textual classification, image classification, semantic text and/or image analysis, image retrieval, answer retrieval, etc.).

Additionally or alternatively, one or more pixel-based machine-learned encoding model <NUM> can be included in or otherwise stored and implemented by the server computing system <NUM> that communicates with the user computing device <NUM> according to a client-server relationship. For example, the pixel-based machine-learned encoding models <NUM> can be implemented by the server computing system <NUM> as a portion of a web service (e.g., a multimodal vision-language service). Thus, one or more models <NUM> can be stored and implemented at the user computing device <NUM> and/or one or more models <NUM> can be stored and implemented at the server computing system <NUM>.

The user computing device <NUM> can also include one or more user input components <NUM> that receives user input. For example, the user input component <NUM> can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

The server computing system <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the server computing system <NUM> to perform operations.

As described above, the server computing system <NUM> can store or otherwise include one or more pixel-based machine-learned encoding models <NUM>. For example, the models <NUM> can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example models <NUM> are discussed with reference to <FIG>.

The user computing device <NUM> and/or the server computing system <NUM> can train the models <NUM> and/or <NUM> via interaction with the training computing system <NUM> that is communicatively coupled over the network <NUM>. The training computing system <NUM> can be separate from the server computing system <NUM> or can be a portion of the server computing system <NUM>.

The training computing system <NUM> includes one or more processors <NUM> and a memory <NUM>. The one or more processors <NUM> can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory <NUM> can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory <NUM> can store data <NUM> and instructions <NUM> which are executed by the processor <NUM> to cause the training computing system <NUM> to perform operations. In some implementations, the training computing system <NUM> includes or is otherwise implemented by one or more server computing devices.

The training computing system <NUM> can include a model trainer <NUM> that trains the machine-learned models <NUM> and/or <NUM> stored at the user computing device <NUM> and/or the server computing system <NUM> using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and/or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer <NUM> can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

In particular, the model trainer <NUM> can train the pixel-based machine-learned encoding models <NUM> and/or <NUM> based on a set of training data <NUM>. The training data <NUM> can include, for example, a corpus of images and textual content that is associated with. For example, the corpus of images may include an image that depicts a giraffe. The textual content associated with the image may describe various characteristics of that particular giraffe or giraffes in general (e.g., height, weight, age, details regarding of the environment in which the image was captured, average lifespan, etc.). The model trainer <NUM> can generate a second image that includes a rendering of the textual content (e.g., rendering an image with the textual content, etc.). The model trainer <NUM> can process the image and the second image with the model(s) <NUM>/<NUM> to obtain a first image embedding and a second image embedding.

The model trainer <NUM> can train the model(s) <NUM>/<NUM> based on a difference between the first image embedding and the second image embedding. For example, the model trainer <NUM> may evaluate a contrastive loss function that minimizes the difference between the first image embedding and the second image embedding and maximizes the difference between (a) the pair of image embeddings including the first and second image embeddings and (b) the plurality of image embeddings in the image embedding space <NUM>. In other words, the contrastive loss function maximizes the similarity between the first image embedding and the second image embedding and minimizes the similarity between the first/second image embeddings and the other image embeddings in the embedding space. In such fashion, the model trainer <NUM> can train the model(s) <NUM>/<NUM> to perform multimodal vision-language tasks.

In some implementations, if the user has provided consent, the training examples can be provided by the user computing device <NUM>. Thus, in such implementations, the model <NUM> provided to the user computing device <NUM> can be trained by the training computing system <NUM> on user-specific data received from the user computing device <NUM>. In some instances, this process can be referred to as personalizing the model.

The model trainer <NUM> includes computer logic utilized to provide desired functionality. The model trainer <NUM> can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, the model trainer <NUM> includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer <NUM> includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

The machine-learned models described in this specification may be used in a variety of tasks, applications, and/or use cases.

In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine-learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.

In some implementations, the image data can depict text or natural language data. For example, the image data may include a rendering of the text or natural language data. The machine-learned model(s) can process the image data that depicts the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the image data that depicts the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the image data that depicts the text or natural language data to generate a latent text embedding output. As another example, the machine-learned model(s) can process the image data that depicts the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the image data that depicts the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the image data that depicts the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the image data that depicts the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the image data that depicts the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the image data that depicts the text or natural language data to generate a prediction output.

In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data that depicts speech data. The machine-learned model(s) can process the image data that depicts the speech data to generate an output. As an example, the machine-learned model(s) can process the image data that depicts the speech data to generate a speech recognition output. As another example, the machine-learned model(s) can process the image data that depicts the speech data to generate a speech translation output. As another example, the machine-learned model(s) can process the image data that depicts the speech data to generate a latent embedding output. As another example, the machine-learned model(s) can process the image data that depicts the speech data to generate an encoded speech output (e.g., an encoded and/or compressed representation of the speech data, etc.). As another example, the machine-learned model(s) can process the image data that depicts the speech data to generate an upscaled speech output (e.g., speech data that is higher quality than the input speech data, etc.). As another example, the machine-learned model(s) can process the image data that depicts the speech data to generate a textual representation output (e.g., a textual representation of the input speech data, etc.). As another example, the machine-learned model(s) can process the image data that depicts the speech data to generate a prediction output.

<FIG> illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device <NUM> can include the model trainer <NUM> and the training dataset <NUM>. In such implementations, the models <NUM> can be both trained and used locally at the user computing device <NUM>. In some of such implementations, the user computing device <NUM> can implement the model trainer <NUM> to personalize the models <NUM> based on user-specific data.

<FIG> depicts a block diagram of an example computing device <NUM> that performs training of a pixel-based machine-learned encoding model for multimodal vision-language tasks according to example embodiments of the present disclosure. The computing device <NUM> can be a user computing device or a server computing device.

<FIG> depicts a block diagram of an example computing device <NUM> that performs multimodal vision-language tasks using a pixel-based machine-learned encoding model according to example embodiments of the present disclosure. The computing device <NUM> can be a user computing device or a server computing device.

The central intelligence layer includes a number of machine-learned models. For example, as illustrated in <FIG>, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device <NUM>.

<FIG> depicts a data flow diagram for training a pixel-based machine-learned encoding model according to some embodiments of the present disclosure. More specifically, a computing system <NUM> (e.g., user computing device <NUM>, server computing system <NUM>, training computer system <NUM> of <FIG>, etc.), can obtain an image <NUM> and textual content <NUM> associated with the image <NUM>. For example, the image <NUM> and textual content <NUM> may be obtained from a corpus of training data for training of models for multimodal vision-language tasks (e.g., training dataset <NUM> of <FIG>, etc.). In some implementations, the image <NUM> and textual content <NUM> may be extracted via various web crawling techniques. For example, an image search can be performed (e.g., using an image search engine) and pairs of images and their associated metadata (e.g., alt-text descriptions, etc.) can be extracted to form the corpus of training data. For example, an image search may be performed for aircraft carriers. An image (e.g., an image of the U. Midway) and an associated alt-text description (e.g., "<NPL>") can be extracted from a website related to aircraft carriers using any type or manner of web crewing technique. The pair of the image and the alt-text description can be included in a corpus of training data.

The computing system <NUM> can generate an image <NUM> that includes a rendering of the textual content <NUM>. For example, the computing system <NUM> may utilize image renderer <NUM> to render the image <NUM>. It should be noted that the images <NUM> and/or <NUM> may be rendered using any type or manner of image format. For example, the image <NUM> may be rendered in a graphics interchange format (GIF) and the image <NUM> may be rendered in a joint photographic expert group (JPEG) format.

It should be noted that the appearance of the textual content <NUM> as rendered to image <NUM> is chosen only to more clearly depict that the textual content <NUM> has, in fact, been rendered, and is not native text. Rather, the textual content <NUM> can be rendered to the image <NUM> in any manner that facilitates various implementations of the present disclosure. In particular, the textual content depicted in image <NUM> is depicted as being rendered at a slanted angle. However, implementations of the present disclosure may also render the textual content of the image <NUM> at the center of the image <NUM> without any degree of slant. Alternatively, in some implementations, the computing system <NUM> may render the textual content <NUM> to the image <NUM> at a location other than the center of the image <NUM>. As such, it should be broadly understood that the textual content <NUM> can be rendered to the image <NUM> in any type or manner of font, positioning, slant, format, thickness, color, dimension (e.g., three-dimensional or pseudo three-dimensional text, etc.), etc.).

The machine-learned encoding model <NUM> can process the image <NUM> to obtain a first image embedding <NUM>, and can process the image <NUM> to obtain a second image embedding <NUM>. In particular, it should be noted that the machine-learned encoding model <NUM> can be a pixel-specific model that is trained exclusively to process images data (e.g., data that includes or otherwise describes the pixels that constitute an image, etc.). In some implementations, the machine-learned encoding model <NUM> can be a machine-learned image transformer model.

The computing system can utilize loss function evaluator <NUM> to train the machine-learned encoding model <NUM>. In particular, the loss function evaluator <NUM> can train the machine-learned encoding model <NUM> based on a difference between the first image embedding <NUM> and the second image embedding <NUM>. In some implementations, the difference between the first image embedding and the second image embedding may refer to a degree of similarity between the image embeddings <NUM>/<NUM>. For example, the computing system <NUM> may train the machine-learned encoding model <NUM> such as to increase the degree of similarity between the image embedding <NUM> and the image embedding <NUM>.

In some implementations, the loss function evaluator <NUM> may evaluate a loss function that evaluates (a) a difference between the first image embedding <NUM> and the second image embedding <NUM>, and (b) the pair of image embeddings <NUM>/<NUM> and a plurality of images included in the image embedding space <NUM>. In other words, the loss function can maximize the similarity between the first image embedding <NUM> and the second image embedding <NUM>, and can minimize the similarity between the first/second image embeddings <NUM>/<NUM> and the other image embeddings in the embedding space <NUM>.

For example, the loss function evaluated by the loss function evaluator <NUM> may be a contrastive loss function <NUM>. When evaluated, the contrastive loss function can maximize a difference between the first image embedding <NUM> and the second image embedding <NUM>. The contrastive loss function <NUM> can also maximize a difference between the pair of image embeddings <NUM>/<NUM> and the plurality of image embeddings within the image embedding space <NUM>. In such fashion, the contrastive loss function <NUM> can be utilized by the computing system <NUM> in conjunction with the loss function evaluator <NUM> to train the machine-learned encoding model <NUM> for multimodal vision-language tasks.

Although the image <NUM> is not depicted as including text, it should be noted that in some implementations the image <NUM> may also include a rendering of textual content. For example, as depicted, the image <NUM> depicts a black cat. The textual content <NUM> includes descriptors for the black cat depicted in image <NUM> (e.g., "black cat", "kitten", "maine coon cat", young cat", etc.). However, in some implementations, the image <NUM> may also include textual content. For example, the image <NUM> may include textual content that describes a source of the image <NUM> (e.g., "this image was retrieved from catpics.

As described, the computing system <NUM> can be trained to perform multimodal vision-language tasks. In particular, to do so, text inputs (e.g., textual content) is rendered on blank images, and subsequently dealt with entirely as images, including the initial patch embedding. By training the machine-learned encoding model <NUM> (e.g., a single vision transformer) contrastively, we obtain a single vision transformer model <NUM> that can understand both images and text through the single interface of vision and provides a single representation which can be used to solve image, image-language, and pure language understanding tasks. In particular, as described, the machine-learned model <NUM> can be trained by considering positive pairs of consecutive sentences sampled from a text corpus, pairs of translated sentences for different languages, pairs of back-translated sentences, as well as pairs of sentences with word dropout. Such text/text pairs can seamlessly be integrated into the contrastive training by supplementing batches of image/alt-texts with pairs of (rendered) text/text pairs.

Furthermore, alongside multimodal versatility, training and utilization of the machine-learned encoding model <NUM> according to implementations of the present disclosure alleviates common hurdles with text processing, namely the development of an appropriate tokenizer and vocabulary. This is particularly interesting in the context of a massively multilingual setup, where the text encoder has to handle dozens of languages.

<FIG> depicts a data flow diagram for performing multimodal vision-language tasks using a trained pixel-based machine-learned encoding model according to some embodiments of the present disclosure. More specifically, a computing system <NUM> can obtain an image <NUM>. The image <NUM> can include a rendering of textual content. For example, the computing system <NUM> may obtain an image (e.g., an image that depicts a cat) and textual content descriptive of the image. The computing system <NUM> may then generate an image that includes a rendering of the textual content, and then form an image <NUM> that includes both the image and the rendering of the textual content. Alternatively, the computing system <NUM> may render the textual content directly to the obtained image to form the image <NUM>. The computing system <NUM> can include a trained machine-learned encoding model <NUM> (e.g., training for multimodal vision-language tasks as described with regards to <FIG>, etc.). The computing system <NUM> can process the image <NUM> with the machine-learned encoding model <NUM> to obtain an image embedding <NUM>. The image embedding <NUM> can be any type or manner of encoding of the information of the image <NUM>.

The computing system <NUM> can include an image embedding space <NUM> (e.g., a collection of image embeddings that collectively from an image embedding space). The image embedding space <NUM> can include a plurality of image embeddings 308A-308N. For example, prior to processing the image <NUM> with the machine-learned encoding model <NUM>, the computing system <NUM> may process a large number of images with the machine-learned encoding model <NUM> to obtain the image embeddings 308A-308N, and then store the image embeddings 308A-308N within the image embedding space <NUM>. In some implementations, some of the image embeddings 308A-308N may be embeddings of images that are, or otherwise include, renderings of textual content. For example, image embedding 308A may be an image embedding of an image that depicts a dog. Image embedding 308B may be an embedding of an image that includes a rendering of corresponding textual content that describes the dog (e.g., "black dog; large dog; german shepherd", etc.).

In some implementations, the computing system <NUM> can retrieve an image embedding <NUM> of the plurality of image embeddings 308A-308N from the image embedding space <NUM>. For example, the computing system <NUM> may utilize image embedding retriever <NUM> to retrieve the image embedding <NUM>. The image embedding retriever <NUM> may retrieve the image embedding <NUM> based on a similarity between the image embedding <NUM> and the image embedding <NUM>.

For example, the image embedding retriever <NUM> may be instructed by the computing system <NUM> to select a single image embedding <NUM> from the plurality of image embeddings 308A-308N that is most similar to the image embedding <NUM>. The image embedding <NUM> may be an embedding of an image <NUM> that is a rendering of textual content associated with the image <NUM>. The image embedding retriever <NUM> may determine that the image embedding <NUM> is most similar to the image embedding <NUM> of the plurality of image embeddings 308A-308N.

It should be noted that, although the image <NUM> from which the image embedding <NUM> is generated is depicted in <FIG>, the image <NUM> is not necessarily stored within the image embedding space <NUM>, or the computing system <NUM> at all. Rather, in some implementations, the computing system <NUM> may obtain the image <NUM> after retrieving the image embedding <NUM>, and then provide the image <NUM>. For example, the computing system <NUM> may obtain textual content from a requesting entity (e.g., a user of a user computing device, etc.). The textual content may include a query from the user (e.g., "what cat breed is this?"). As depicted, the computing system <NUM> can render the query as an image to form the image <NUM>. The computing system can retrieve the image embedding <NUM> as previously described, can obtain the image <NUM> respectively associated with the image embedding <NUM>, and can provide the image <NUM> to the requesting entity. For example, the image embedding <NUM> may indicate a location from which the image <NUM> can be retrieved (e.g., a file repository, etc.). For another example, the computing system <NUM> may process the image embedding <NUM> with a generative machine-learned model to generate the image <NUM> (or a reconstruction of the image <NUM>) (e.g., using a machine-learned decoding model trained concurrently or subsequently with the machine-learned encoding model, etc.).

It should be noted that, as depicted, the image embedding <NUM> can encode specific characteristics of entities depicted within the image <NUM> as well as the textual content depicted within the image <NUM>. For example, as depicted, the textual content rendered in image <NUM> can be a query (e.g., "what cat breed is this?"). The entity depicted in the image <NUM> can be a specific breed of cat (e.g., a maine coon cat). The machine-learned encoding model <NUM> can generate the image embedding <NUM> such that the image embedding <NUM> retrieved based on its similarity to the image embedding <NUM> shares features of both the textual content and the entity of the image <NUM>. For example, as depicted, the image <NUM> associated with image embedding <NUM> is an answer to a query of the textual content of image <NUM> that is specific to the breed of the cat depicted in the image <NUM>. In such fashion, the machine-learned encoding model <NUM> can generate image embeddings (e.g., image embedding <NUM>) that are sufficiently detailed to enable complex multimodal vision-language tasks, such as answering multimodal queries.

Additionally, in some implementations, the textual content may also include possible answers that are all rendered as a single image. For example, a prediction submodel can be added to the machine-learned encoding model that is configured to predict a correct answer from a series of given possible answers. The textual content rendered to image <NUM> may be "what cat breed is this? A) maine coon; B) siamese; C) tabby cat; d) Ragdoll). The image embedding <NUM> can be generated from this image, and the image embedding retriever <NUM> can retrieve an image embedding <NUM> that selects one of the four multiple choice questions. In such fashion, the machine-learned encoding model <NUM> and the included prediction submodel can be trained to predict the correct answer of the four answers.

It should be noted that, although the textual content rendered in image <NUM> is written in the same language as the textual content rendered in image <NUM>, it is not necessary that the textual content of image <NUM> and the images respectively associated with image embeddings 308A-308N is all written in the same language. Rather, if the query depicted in image <NUM> was written in a language different than the language of the answer depicted in image <NUM>, the image embedding <NUM> may still be sufficiently similar to the image embedding <NUM> as to be retrieved by the image embedding retriever <NUM>.

It should be noted that, although the machine-learned encoding model <NUM> can facilitate multimodal query tasks (e.g., answer retrieval tasks), it is not limited to such tasks. Rather, the image embeddings generated using machine-learned encoding model <NUM> can be utilized in a variety of vision tasks, language tasks, and multimodal vision-language tasks (e.g., language translation tasks, a textual classification task that classifies textual content depicted by the third image, an image classification task that classifies the third image, a semantic analysis task that generates a semantic output for the third image, an image retrieval task, etc.). For example, the image <NUM> that is associated with image embedding <NUM> depicts textual content associated with image <NUM>. However, if computing system performs a task to retrieve semantically similar images to that of image <NUM>, the computing system may utilize the image embedding retriever to retrieve a large number of image embeddings 308A-308N from the image embedding space <NUM> that are respectively associated with images semantically similar to image <NUM> (e.g., images depicting elderly cats, etc.).

<FIG> depicts a flow chart diagram of an example method to perform according to example embodiments of the present disclosure. Although <FIG> depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method <NUM> can be omitted, rearranged, combined, and/or adapted in various ways without deviating from the scope of the present disclosure.

At <NUM>, a computing system obtains a first image and textual content associated with the first image. In some implementations, the first image includes a rendering of additional textual content different from the textual content associated with the first image. In some implementations, the additional textual content is written in a first language, and wherein the textual content associated with the first image is written in a second language different from the first language. In some implementations, the textual content is descriptive of the first image.

At <NUM>, the computing system renders a second image that includes the first image and a rendering of the textual content associated with the first image. In some implementations, rendering the second image that depicts the textual content associated with the first image includes modifying the textual content associated with the first image to obtain modified textual content, and rendering a second image that depicts the modified textual content. Additionally, or alternatively, in some implementations, the textual content is descriptive of the first image. For example, if the first image depicts a sunset on the beach, the textual content may describe characteristics of the sunset on the beach (e.g., "image depicts beach sunset; image captured in Caribbean", etc.).

At <NUM>, the computing system processes the first image and the second image with a machine-learned encoding model to respectively obtain a first image embedding and a second image embedding for an image embedding space that includes a plurality of image embeddings. In some implementations, the machine-learned encoding model comprises a machine-learned image transformer model.

At <NUM>, the computing system trains the machine-learned encoding model based on a difference between the first image embedding and the second image embedding. In some implementations, training the machine-learned encoding model includes evaluating a loss function that evaluates a difference between the first image embedding and the second image embedding, and a difference between (a) a pair of image embeddings comprising the first and second image embeddings and (b) the plurality of image embeddings in the image embedding space. For example, evaluating the loss function can include evaluating a contrastive loss function that minimizes the difference between the first image embedding and the second image embedding and maximizes the difference between (a) the pair of image embeddings comprising the first and second image embeddings and (b) the plurality of image embeddings in the image embedding space.

In some implementations, the computing system can further obtain a third image. The computing system can process the third image with the machine-learned encoding model to obtain a third image embedding. The computing system can retrieve a fourth image embedding from the image embedding space based on a similarity between the third image embedding and the fourth image embedding.

In some implementations, obtaining the third image includes obtaining a third image that depicts a rendering of second textual content. Retrieving the fourth image embedding can include retrieving a fourth image embedding from the image embedding space. The fourth image embedding can be based on an image that depicts one or more entities that correspond to the second textual content.

Alternatively, in some implementations, retrieving the fourth image embedding can include retrieving a fourth image embedding from the image embedding space. The fourth image embedding can be based on an image that depicts a rendering of second textual content associated with the third image.

In some implementations, the computing system can further use the fourth image embedding to perform a task. The task can include a textual classification task that classifies textual content depicted by the third image, an image classification task that classifies the third image, a semantic analysis task that generates a semantic output for the third image, an image retrieval task in which the third image includes a plurality of characteristics and the fourth image includes at least a portion of the plurality of characteristics, etc..

In some implementations, obtaining the third image includes obtaining a third image that depicts (a) one or more entities and (b) a rendering of second textual content descriptive of the one or more entities. Additionally, or alternatively, in some implementations, obtaining the third image includes obtaining a third image that depicts (a) one or more entities and (b) a rendering of second textual content descriptive of a query associated with the one or more entities.

In some implementations, the second textual content is further descriptive of a plurality of proposed answers to the query associated with the one or more entities.

Claim 1:
A computer-implemented method for pixel-based machine-learned models for multimodal vision-language tasks, comprising:
obtaining, by a computing system (<NUM>) comprising one or more computing devices, a first image (<NUM>) and textual content (<NUM>) associated with the first image;
rendering, by the computing system, a second image (<NUM>) that depicts the textual content associated with the first image;
processing, by the computing system, the first image and the second image with a machine-learned encoding model (<NUM>) to respectively obtain a first image embedding (<NUM>) and a second image embedding (<NUM>) for an image embedding space comprising a plurality of image embeddings; and
training, by the computing system, the machine-learned encoding model based on a difference between the first image embedding and the second image embedding, wherein training the machine-learned encoding model comprises evaluating a loss function that minimizes a difference between the first image embedding and the second image embedding in the image embedding space.