Multimodal Few-Shot Learning with Frozen Language Models

1 Introduction . Auto-regressive transformers have been shown to be very impressive models of natural language [ 42 ] . Large-scale language transformers exhibit several surprising abilities beyond that of standard text generation [ 4 , 31 ] . Perhaps most notably , they are few-shot learners ; they can learn to perform a new task from a few examples without any further gradient updates . Equipped with this ability , these models have been shown to rapidly adapt to new tasks and styles of generation via prompting ( e.g . switching from formal to informal language ) [ 4 ] , to retrieve relevant encyclopedic or general knowledge when primed with a relevant context ( e.g . answering questions such as ‘ When did the French Revolution begin ? ’ ) [ 34 , 1 , 28 ] and to use new words in appropriate ways straight after being taught what those words mean ( sometimes referred to as ‘ fast binding ’ ) [ 12 , 4 ] . Despite these impressive capabilities , such large scale language models are ‘ blind ’ to modalities other than text , preventing us from communicating visual tasks , questions or concepts to them . Indeed , philosophers and linguists have questioned whether an un-grounded language model can ever achieve true understanding of the language it processes [ 5 , 2 ] . Here , we present Frozen , a method for giving a pre-trained language model access to visual information in a way that extends its few-shot learning capabilities to a multimodal setting , without changing its weights . Frozen consists of a neural network trained to encode images into the word embedding space of a large pre-trained language model such that the language model generates captions for those images . The weights of the language model are kept frozen , but gradients are back-propagated through it to train the image encoder from 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ! '' # scratch ( Figure 2 ) . Although Frozen is trained on single image-text pairs , once trained it can respond effectively to interleaved sequences of multiple images and text . This allows users to ‘ prompt ’ it with several examples of new multimodal tasks before evaluating its performance , or to ‘ teach ’ it the name of a new visual category before immediately asking about that category . By exploiting its pre-trained language model , Frozen exhibits nontrivial zero-shot performance on multimdodal tasks that it was not trained on , such as visual question answering ( VQA ) . More surprisingly , it gets better at these tasks after seeing a handful of examples ‘ in-context ’ as in [ 4 ] , and also performs above chance on tests of fast category learning such as miniImageNet [ 43 ] . In each case , comparisons with ‘ blind ’ baselines show that the model is adapting not only to the language distribution of these new tasks , but also to the relationship between language and images . Frozen is therefore a multimodal few-shot learner , bringing the aforementioned language-only capabilities of rapid task adaptation , encyclopedic knowledge and fast category binding to a multimodal setting . Our goal in developing Frozen was not to maximise performance on any specific task , and in many cases it is far from state-of-the-art . Nonetheless , it performs well above trivial baselines across a wide range of tasks without ever seeing more than a handful of the training examples provided by these benchmarks . Moreover , as illustrated in Figure 1 , Frozen is a system for genuinely open-ended and unconstrained linguistic interpretation of images that often produces compelling output . To summarise , our contributions are as follows : 1 . We present Frozen , a modular , scalable and efficient approach to training vision front-ends for large language models . The resulting combined model retains all of the capabilities of large language models , but can also process text and image inputs in any arbitrary sequence . 2 . We show that such models transfer their capacity for rapid task adaptation , encyclopedic knowledge and fast category binding from a language-only to a multimodal setting , and verify that prompting them with both visual and language information can be strictly more effective than doing so with language information alone . 3 . We quantify these capabilities on a range of existing and new benchmarks , paving the way for future analysis of these capabilities . 2 Related Work . The Frozen method is inspired by lots of recent work . [ 26 ] show that the knowledge encoded in transformer language models can be a valuable prior for tasks involving reasoning and memory across discrete sequences , and even classification of images presented as sequences of spatial regions . In that approach , a small subset of the pre-trained language model weights are fine-tuned to the various final applications . In contrast , applying Frozen to different tasks does not involve any weight updates to the transformer whatsoever ; the system adapts to and improves at multimodal ( vision and language ) tasks as activations propagate through the model . The two studies thus reveal different ways in which knowledge acquired from text can transfer to non-linguistic settings . The effectiveness of prefix tuning [ 23 ] or prompt tuning [ 20 ] was another important motivation for Frozen . Prefix tuning is a method for prompting a language model to produce output of a particular style using gradient descent to learn a task-specific bias term which functions like the continuous embedding of a text prompt . Using prefix tuning , language models can be adapted to different natural language generation tasks like summarization . Frozen could also be considered a type of image-conditional prefix tuning , in which this continuous prefix is not a bias but an image-conditional activation produced by an external neural network . Learning to embed image representations into the ‘ word ’ space of a large pretrained language model was done previously by [ 16 ] . This work focused on image-text classification , and uses a BERT-style language model that is fine-tuned ( rather than frozen ) on multimodal data . [ 36 ] extend a similar image embedding+BERT system to create a generative model of text , using a pre-trained object extraction system to embed images into word space . Neither of these studies consider the problem of learning image-text correspondences in a few shots . A large body of work has applied either text-specific or multimodal representation-learning approaches like BERT [ 8 ] to visual question answering ( VQA ) and captioning ( see e.g . [ 25 , 40 ] and many more ) . In these approaches , models are first trained with aligned data on task-agnostic cross-modal objectives and then fine-tuned to specific tasks . This approach can yield state-of-the-art performance on a range of classification tasks . Unlike Frozen , the resulting systems are highly specialized to one task , and can not learn new categories or adapt to new tasks in a few shots . By contrast , [ 7 ] propose text generation as an objective for task-general multimodal models , yielding a system that , like Frozen , produces unconstrained language output . Unlike Frozen , they do not use a pre-trained model trained on text only , and do not consider zero or few-shot learning , instead updating all weights of the system with training data for each task they consider – thus , again , specializing the models to one task at a time . Similarly , [ 46 ] and [ 6 ] show that a large pre-trained language model as decoder can improve a captioning performance when training data is limited . Unlike Frozen , they use pre-trained frozen visual encoders or object extractors and fine-tune the pre-trained weights in the text decoder on the captioning data . Similarly , they do not consider zero or few-shot adaptation across different multimodal tasks . Past work has also explored alternative approaches for post-hoc combination of models for different modalities using latent variables [ 41 ] . Multimodal pre-training has recently been shown to enable strong zero-shot generalization in the discriminative setting using large-scale contrastive learning [ 29 , 14 ] . Also in a discriminative setting , [ 45 ] has observed signs of emergent few-shot-learning from large-scale training . In contrast , our work enables strong generalization to new multimodal tasks both zero-shot or few-shot with completely open-ended generative text output . 3 The Frozen Method Frozen is a method for grounding a large language model without changing its weights , closely related to prefix tuning [ 23 , 20 ] . Prefix tuning trains a task-specific continuous bias term to function like the embedding of a constant , static text prompt used for all test-time examples . Frozen extends this approach by making this prefix dynamic , in that it is not a constant bias but an input-conditional activation emitted by a neural network . 3.1 Architecture . Pre-trained Autoregressive Language Models Our method starts from a pre-trained deep autoregressive language model , based on the Transformer architecture [ 42 , 30 ] , which parametrizes a probability distribution over text y . Text is decomposed into a sequence of discrete tokens y = y1 , y2 , ... , yL by the SentencePiece tokenizer [ 18 ] . We use a vocabulary of size 32,000 . The language model makes use of an embedding function gθ which independently transforms each token into a continuous embedding tl : = gθ ( yl ) , as well as a transformer neural network fθ whose output is a vector of logits parameterizing a categorical distribution over the vocabulary . The distribution pθ ( y ) is represented as follows : log pθ ( y ) = ∑ l log pθ ( yl|y1 , y2 , ... , yl−1 ) = ∑ l fθ ( t1 , t2 , ... , tl−1 ) yl The model we start from is pre-trained , i.e . θ has been optimised via the standard maximum-likelihood objective on a large dataset of text from the internet . We use a 7 billion parameter transformer trained on the public dataset C4 [ 31 ] – previous work has shown that the multi-billion parameter scale is sufficient to exhibit the key capacities we are interested in studying [ 30 , 34 ] . Vision Encoder Our vision encoder is based on NF-ResNet-50 [ 3 ] . We define vφ as a function that takes a raw image and emits a continuous sequence to be consumed by the transformer . We use the final output vector of the NF-Resnet after the global pooling layer . Visual Prefix One important requirement is to represent images in a form that the transformer already understands : a sequence of continuous embeddings , each having the same dimensionality D as a token embedding tl . We therefore form the visual prefix by linearly mapping the vision encoder ’ s output to D ∗ k channels , and then reshaping the result as a sequence of k embeddings , each with dimensionality D. We call this sequence a visual prefix since it plays the same functional role in the transformer architecture as ( part of ) an embedding sequence of prefix tokens . We experimented using different number of tokens k , specifically 1 , 2 and 4 and found that 2 performs best , though certainly this would be sensitive to other architectural details . See Appendix for more details on the architecture . 3.2 Training . During training , we update only the parameters φ of the vision encoder using paired image-caption data from the Conceptual Captions dataset [ 37 ] . Our experiments show that fine-tuning θ hurts generalization , as much less paired image-caption data is available than the amount of text-only data used to pre-train θ . Training only the parameters φ makes our system modular – it can use an existing language model off the shelf – and also quite simple : we only train a visual encoder and rely on the capabilities of an existing language model . Following standard captioning systems [ 22 , 13 ] , we treat captioning as conditional generation of caption text y given an image x . We represent x as vφ ( x ) = i1 , i2 , ... , in and train φ to maximise the likelihood : log pθ , φ ( y|x ) = ∑ l log pθ , φ ( yl|x , y1 , y2 , ... , yl−1 ) = ∑ l fθ ( i1 , i2 , ... , in , t1 , t2 , ... , tl−1 ) yl Whilst the parameters θ are frozen , each element ik of the visual prefix receives gradients∑ l ∇ikfθ ( i1 , i2 , ... , in , t1 , t2 , ... , tl−1 ) yl , enabling the parameters of the visual encoder to be optimised with standard backpropagation and SGD ( Figure 2 ) . As the notation fθ ( i1 , i2 , ... , in , t1 , t2 , ... , tl−1 ) suggests , we present the visual prefix during training as if it were a sequence of embeddings occurring earlier in time than the caption ( token embeddings ) t1 , t2 , .... We use relative positional encoding [ 38 ] , which enables the transformer to generalize to prefix sequences where an image is not always in the first absolute positions , and where more than one image may be present . In particular , we use the version of relative attention described in transformerxlDai.We leave improvements of this simple scheme for future work .

Summary: The paper proposes a new GPT-2 style image-language model, Frozen, which is trained in two stages:  1. common GPT-2 pre-training on huge text-alone corpora 2. training of a Resnet image encoder while freezing the GPT-2 and text embedding weights, on the task of image captioning (i.e. language modeling conditional on two image encoding vectors with the same dimension as token embeddings).  Since GPT-2 exhibits few-shot abilities in language-alone tasks, the idea is Frozen can also benefit from the GPT-2 language pertaining and exhibit *multimodal* few-shot abilities by the minimal design of adding a visual encoder. Note that the freezing does not make training more efficient, since gradients have to pass through all Transformer layers to the visual encoder.  The Frozen model is applied to zero/few-shot learning of two downstream tasks: VQA and few-shot image classification.