Multimodal Few-Shot Learning with Frozen Language Models
Authors: Maria Tsimpoukelli
Jacob Menick
Serkan Cabi
S. M. Ali Eslami
Oriol Vinyals
Felix Hill
Venue: NeurIPS 2021 Poster (2021)
Keywords: Language Modeling
Multimodal
Few-shot Learning

Abstract: When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, we present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language). Using aligned image and caption data, we train a vision encoder to represent each image as a sequence of continuous embeddings, such that a pre-trained, frozen language model presented with this prefix generates the appropriate caption. The resulting system is a multimodal few-shot learner, with the surprising ability to learn a variety of new tasks when conditioned on examples, represented as a sequence of any number of interleaved image and text embeddings. We demonstrate that it can rapidly learn words for new objects and novel visual categories, do visual question-answering with only a handful of examples, and make use of outside knowledge, by measuring a single model on a variety of established and new benchmarks.

Multimodal Few-Shot Learning with Frozen Language Models
Maria Tsimpoukelli DeepMind mrts@deepmind.com com
Jacob Menick" DeepMind University College London jmeni cwwdeepmind..ccommmmm com
Serkan Cabi" DeepMind cabi cabi@deepmind com
S. M. Ali Eslami Deep.Mind aeslami wewlaam@@deeemiinn
Oriol Vinyals DeepMind vinyals@ddepmmidd.com
Felix Hill DeepMind felixhil1@deepmie com
Abstract
When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just few examples. Here, we present simple, yet effective. approach for transferring this few-shot learning ability to a multimodal setting (vision and language). Using aligned image and caption data, we train vision encoder to represent each image as sequence of continuous embeddings, such that a pre-trained, frozen language model prompted with this prefix generates the appropriate caption. The resulting system a multimodal few-shot learner, with surprising ability to learn a variety of new tasks when conditioned examples, represented as sequence of multiple interleaved image and text embeddings. We demonstrate that can rapidly learn words for new objects and novel visual categories, do visual question-answering answering with only a handful examples, and make use of outside knowledge, by measuring single model on variety of established and new benchmarks.
Introduction
Auto-regressive transformers have been shown to be very impressive models of natural language 42 Large-scale language transformers exhibit several surprising abilities beyond that of standard text generation 4 31. Perhaps most notably, they few-shot learners; they can learn to perform a new task from a few examples without any further gradient updates. Equipped with this ability, these models have been shown to rapidly adapt to new tasks and styles of generation via prompting (e.g. switching from formal to informal language) 4. to retrieve relevant encyclopedic or general knowledge when primed with relevant context (e.g. answering questions such 'When did the French Revolution begin?') 34 28 and to use new words in appropriate ways straight after being taught what those words mean (sometimes referred to 'fast binding') I12 4
Despite these impressive capabilities, such large scale language models are 'blind' to modalities other than text, preventing us from communicating visual tasks, questions concepts to them. Indeed, philosophers and linguists have questioned whether an un-grounded language model can ever achieve true understanding of the language it processes [5] 2). Here, present Frozen, method for giving pre-trained language model access to visual information in a way that extends its few-shot learning capabilities multimodal setting, without changing its weights. Frozen consists of a neural network trained to encode images into the word embedding space of large pre-trained language model such that the language model generates captions for those images. The weights of the language model are kept frozen, but gradients are pack-propagee through it to train the image encoder from
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Completion
This person is like @
This person is W
This pezson is Like
<DDS>
Model Completion
This was invented by Zacharias Janssen.
This was invented by Thomas Edison.
This was invented
Wzight brothers.
Model Completion
kith these can drive azound teack, overtaking and takiing speed
of take off frox city and fly accoss the sky to sonewhere would
With one of these I can
break ascuze bistding: unt ock the door
Curated samples with about five seeds required to get past well-known language model failure modes of either repeating text for the prompt or emitting text that does not pertain to the image. These samples demonstrate the ability to generate open-ended outputs that adapt both images and text, and to make use of facts that it has learned during language-only pre-training.
Figure I:
scratch (Figure 2). Although Frozen is trained on single image-text pairs, once trained it can respond
effectively to interleaved sequences of multiple images and text. This allows users to 'prompt' it with several examples of new multimodal tasks before evaluating its performance, to 'teach' it the name of new visual category before immediately asking about that category.
snall red beat on the water
By exploiting its pre-trained language model, Frozen ex- hibits nontrivial zero-shot performance on multimdodal tasks that it was not trained on, such as visual question answering (VQA). More surprisingly, gets better at these tasks after seeing a handful of examples 'in-context' as in 4) and also performs above chance on tests of fast cat- egory learning such minilmageNet [43]. In each case, comparisons with 'blind' baselines show that the model is adapting not only to the language distribution of these new tasks, but also to the relationship between language and images. Frozen is therefore multimodal few-shot learner, bringing the aforementioned sunguage-only capabilities of rapid task adaptation, encyclopedic knowledge and fast category binding to a multimodal setting.
14
Model Self Attention
* Frozen
T/ Vision
g9
Language Model
Text Embedder
small
red
boat
on
the HAter
Figure 2: Gradients through a frozen lan- guage model's self attention layers are used to train the vision encoder.
Our goal in developing Frozen was not to maximise performance on any specific task, and in many cases it far from state-of-the-an Nonetheless, performs well above trivial baselines across a wide range of tasks without ever seeing more than handful of the training examples provided by these benchmarks. Moreover, as illustrated Figure I Frozen is system for genuinely open-ended and unconstrained linguistic interpretation of images that often produces compelling output.
<E0S>
Stove
<80>>
This
<F083
Attantin Layers
Layors
Self Aftent Layere
Misio Encoder
Text Embedder
Vision Encoder
Text Embedddr
Vision Enooder
Text Emooddor
Vision Fnooder
Vision Eroonder
Vision Froondee
Fnoowader
Fimbedow
Fimbedder
Questiion: What coloout cos7
tnwented
Aho this? A:
biscket.
Questiion:
this? A:
thix?
The Wiright
Answez:
( 0-ehot VQA
(b) 1-shot outside-knowledde VQA
(6) Few-shot image classification
Figure 3: Inference- Time interface for Frozen. The figure demonstrates how we can support (a) visual question answering, (b) outside-knone -nowllee- question answering and (c) few-shot image classification via in-context learning.

---

To summarise, our contributions are as follows: 1. We present Frozen, modular, scalable and efficient approach to training vision front-ends for large language models. The resulting combined model retains all of the capabilities of large language models, but can also process text and image inputs in any arbitrary sequence. 2. We show that such models transfer their capacity for rapid task adaptation, encyclopedic knowledge and fast category binding from a language-only to multimodal setting, and verify that prompting them with both visual and language information can be strictly more effective than doing so with language information alone. 3. We quantify these capabilities range of existing and new benchmarks, paving the way for future analysis of these capabilities.
2
Related Work
The Frozen method is inspired by lots of recent work. [26 show that the knowledge encoded in transformer language models can be valuable prior for tasks involving reasoning and memory across discrete sequences, and even classification of images presented as sequences of spatial regions. In that approach, small subset of the pre-trained language model weights are fine-tuned to the various final applications. In contrast, applying Frozen to different tasks does not involve any weight updates to the transformer whatsoever; the system adapts to and improves at multimodal (vision and language) tasks as activations propagate through the model. The two studies thus reveal different ways in which knowledge acquired from text can transfer to non-linguistic settings.
The effectiveness of prefix tuning [23] or prompt tuning 20 was another important motivation for Frozen. Prefix tuning is method for prompting a language model to produce output of particular style using gradient descent to learn a task-specific bias term which functions like the continuous embedding of text prompt. Using prefix tuning, language models can be adapted to different natural language generation tasks like summarization. Frozen could also be considered type of mmage-conditional prefix tuning, in which this continuous prefix is not bias but an mange-condiionnl activation produced by an external neural network.
Learning to embed image representations into the 'word' space of large pretrained language model was done previously by [16]. This work focused on image-text classification, and uses a BERT-style language model that is fine-tuned (rather than frozen) on multimodal data. [36] extend similar image embedding+Bill BERT system to create a generative model of text, using a pre-trained object extraction system to embed images into word space. Neither of these studies consider the problem of learning image-text correspondences in a few shots.
A large body of work has applied either text-specific or multimodal prrresentation-learminn approaches like BERT g to visual question answering (VQA) and captioning (see e.g. 25 40 and many more). In these approaches, models are first trained with aligned data on task-agnostic cross- moosl objectives and then fine-tuned to specific tasks. This approach can yield state-of-the-ar performance on a range of classification tasks. Unlike Frozen, the resulting systems are highly specialized to one task, and cannot learn new categories or adapt to new tasks in a few shots.
By contrast, 7 propose text generation as objective for task- mask-general multimodal models, yielding a system that, like Frozen, produces unconstrained language output. Unlike Frozen, they do not use a pre-trained model trained on text only, and do not consider zero or few-shot learning, instead updating all weights of the system with training data for each task they consider thus, again, specializing the models to one task at a time. Similarly, [46] and 6 show that a large pre-trained language model as decoder can improve a captioning performance when training data is limited. Unlike Frozen, they use pre-trained frozen visual encoders object extractors and fine-tune the pre- pre-trained weights in the text decoder on the captioning data. Similarly, they do not consider zero or few-shot adaptation across different multimodal tasks. Past work has also explored alternative approaches for post-hoc combination of models for different modalities using latent variables [1
Multimodal pre-training has recently been shown to enable strong zero-shot generalization in the discriminative setting using large-scale contrastive learning 22 14] Also a discriminative setting, [45 has observed signs of emergent few-shot-learning from large-scale training. In contrast, our work enables strong generalization to new multimodal tasks both zero-shot or few-shot with completely open-ended generative text output.
3

---

3 The Frozen Method
Frozen is a method for grounding large language model without changing its weights, closely related prefix tuning [33 20]. Prefix tuning trains a task-specific continuous bias term function like the embedding of constant, static text prompt used for all test-time examples. Frozen extends this approach by making this prefix dynamic, in that it is not a constant bias but an input-conditional activation emitted by neural network.
3.1 Architecture
Pre-trained Autoregressive Language Models Our method starts from pre-trained deep auto- regressive language model, based on the Transformer architecture [42 30 which parametrizes probability distribution over text y. Text is decomposed into a sequence of discrete tokens = //., Mr by the SentencePiece tokenizer [8].. We use a vocabulary of size 32,000, The language model makes use of an embedding function go which independently transforms each token into a continuous embedding & == (o()),, as well as transformer neural network fe whose output a vector of logits parameterizing categorical distribution over the vocabulary. The distribution (y()) is represented as follows:
log (y)) Σ log (3n y2. 91-1) Σ folti, tn, 4-11
The model we start from is pre-trained, i.e. 0 has been optimised via the standard maximum-likelinn objective on a large dataset of text from the internet. We use a 7 billion parameter transformer trained on the public dataset C4 [31]- previous work has shown that the multi-billion parameter scale is sufficient to exhibit the key capacities we are interested studying 30
Vision Encoder Our vision encoder is based on NF-ResNet-50 [3].
We define as a function that takes raw image and emits continuous sequence to consumed by the transformer. We the final output vector of the NE-Resnet after the global pooling layer.
Visual Prefix One important requirement is to represent images in a form that the transformer already understands: sequence of continuous embeddings, each having the same dimensionality D as token embedding 4, We therefore form the visual prefix by linearly mapping the vision encoder's output to l channels, and then reshaping the result as a sequence of embeddings, each with dimensionality D. We call this sequence visual prefix since it plays the same functional role in the transformer architecture as (part of) an embedding sequence of prefix tokens. We experimented using different number of tokens k, specifically 1, and 4 and found that 2 performs best, though certainly this would be sensitive to other architectural details. See Appendix for more details on the architecture.
3.2 Training
During training, we update only the parameters / the vision encoder using paired image-caption data from the Conceptual Captions dataset [37]. Our experiments show that fine-tuning 0 hurts generalization, as much less paired image-caption data is available than the amount of text-only data used to pre-trate preitrain& f. Training only the parameters é makes our system modular it can use existing language model off the shelf - and also quite simple: we only train a visual encoder and rely on the capabilities of an existing language model.
Following standard captioning systems 22 13), we treat captioning as conditional generation of caption text y given an image x. We represent as us (x) i i2, ..., in and train to maximise the likelihood:
log (ylr)) = Σ log (u. V1, 92, V1 11
Σ Solin, m2, ini bi be be, 4-1

---

Whilst the parameters 0 are frozen, each element i& of the visual prefix receives gradients VV,,foli 12,... in &,, 12, .... 1- enabling the parameters of the visual encoder to be op-
timised with standard backpropagation and SGD (Figure 2).
As the notation fo( f2. ..., bzz 11. f2; ..., 42-1 suggests, we present the visual prefix during training as if were a sequence of embeddings occurring earlier in time than the caption (token embeddings) (i, l2,... We use relative positional encoding [38], which enables the transformer to generalize to prefix sequences where an image is not always in the first absolute positions, and where more than one image may be present. particular, we use the version of relative attention described in transformerxlDai.. We leave improvements of this simple scheme for future work.
3.3 Interface at Inference Time
At inference time, a vanilla language model, conditioned upon an arbitrary text prompt y1, 1/2 .... Up- generates text sequences y/+1 2 autoregressively. In Frozen it is straightforward to include images in such prompt by placing an image's embedding i,, 12 as a prefix to a text embedding subsequence $1, f2, .... tp. Because the transformer fe is modality-agnostic sub-sequence of text token embeddings with a sub-sequence of image embeddings in any arbitrary order. In Figure 3) we show how this support zero-shot visual guestion-answering (Figure 3 few-shot visual prrestion-answerigg
interleave
Figuuee 3p), and few-shot image classification (Figure 3b).
To evaluate these tasks, the model decodes output sequences greedily and these outputs are compared against the ground truth answers of the task following the normalization technique used in [19]. To probe the open-ended capabilities of Frozen, we decided not to use common practice of short-lists of pre-canned answers, even though in some tasks this may hurt its performance accuracy percentages.
3.4 Few-Shot Learning Definitions
The ability of Frozen to be conditioned on sequence of interleaved images and text allows it not only be able to perform different multimodal tasks, but also gives rise to different ways of 'inducing' the task to the model in order to improve its performance. We briefly define the terminology used in our settings, common amongst all the different tasks. See Figure 5) in the appendix for a visual illustration of these concepts.
Task induction Explanatory text that precedes the sequence of images and text. It is intended to describe the task to the model in natural language, for example "Please answer the question.'
Number of shots The number of distinct full examples of the task presented to the model prior the evaluated example. For example, in Visual Question-Answering, shot is an image along with the question and the answer.
For tasks involving fast category binding (e.g., few-shot image classification), we define further specific terminology. See also Figure 4h and Figure 6nin the appendix.
Number of ways The number of object classes in the task (e.g. dog vs cat).
Number of inner-shots The number of distinct exemplars from each category that are presented to the model (i.e. number of images of different dogs). In previous work with Minilmagenet, these were known as shots, but we modify the term here to distinguish from the more general usage of the term described above.
Number of repeats The number of times each inner-shot is repeated in the context presented to the model. We use this setting as an ablation to explore how the model integrates visual information about category.
4 Experiments: A Multi-Modal Few-Shot Learner
Our experiments are designed to quantify three capacities that should be characteristic of a Multi- Modal Few-Shot Learner: rapid adaptation to new tasks, fast access to general knowledge and fast binding of visual and linguistic elements. We train Frozen Conceptual Captions, a public dataset
5

---

n-shot Acc.
n=0
n=1
n=4
n-shot Acc.
n=0
n=1
n=4
Frozen
29.5
35.7
38.2
X
Frozen
5.9
9.7
12.6
X
Frozen seratch
0.0
0.0
0.0
X
Frozen 400mLM
4.0
5.9
6.6
X
Frozen Ginetuned
24.0
28.2
29.2
X
Frozen finetuned
4.2
4.1
4.6
X
Frozen train-blind
26.2
33.5
33.3
Frozen train-blind
3.3
7.2
0.0
X
Frozen VQA
48.4
Frozen VQA
19.6
X
Frozen VQA-blind
39.1
Frozen VQA-blind
12.5
X
Oscar 24
73.8
MAVEX 44
39.4
Table 1: Transfer from Conceptual Captions to VQAv2. The column indicates whether model uses training data from the VQAv2 training set. The row denoted Frozen urain-blind bind is the blind base- line described in |subsection 4. Frozen vQa is baseline which mixes VQAv2 training data.
Table 2: Transfer from Conceptual Captions to OKVQA. The column indicates if model uses training data from the OKVQA training set. Frozen does not train VQAv2 except in the baseline row, and it never trains on OKVQA.
that consists of around three million image-caption pairs 37.. We do early stopping on the validation set perplexity which usually reaches an optimum just after a single epoch with batch size 128. All experiments used the Adam optimizer with 81 0.9 and 82 0.95 and constant learning rate of 3e-4 unless otherwise noted. We operate on 224 x 224 images at both train and test-time. Images which are not square are first padded with zeroes square and then resized 224x224.
4.1 Rapid Task Adaptation
We first examine zero-shot and few-shot generalization from captioning to visual question-answering This is type of rapid adaptation from captioning behaviour to guestion-answerin behaviour analogous transfer from language modelling to open-domain -aastinnnnnnwringg in the text-only setting 34
We evaluate on the VQAv2 [10] validation set.
Zero-shot transfer from captioning to VQA We first observe that version of our model in which the ability to embed images into prefix is trained solely with a captioning objective can transfer moderately well visual question-answerttg
in the zero-shot setting, with no specific training towards that goal. We simply have to provide the system with image and textual prompt of the form Question: what colour is the dog sitting on the grass? Answer:, then observe how it completes the prompt. The ability to adapt to input of this form is presumably transferred from the training data of the pretrained language model component of the system.
The strength of the pre-trained language model in the system is double-edged sword. It powers the generalization abilities of Frozen but also enables the model perform surprisingly well without considering the visual input at all. To guard against this possibility we also train blind baselines, which image presented to the visual encoder blacked out, but the convnet weights are still trained (see Table 1). This amounts to prefix tuning [23] We outperform this blind baseline which also inherits the sho- learning abilities of the language model.
In these experiments we also include two additional and important baselines: Frozen finetuned in which the language model instead finetuned starting from the pretrained weights and Frozen scratch- wherein the whole system is trained from scratch andtto-end, both using the same dataset as Frozen. These baselines preferred a smaller learning rate of 1e-5. Results Table show that keeping the language model frozen generalizes substantially better to visual question guestion-answering answering than finetuning. The model trained from scratch is not able transfer all from captioning to VQA; we interpret this to suggest that the tremendous generalization abilities of large language models are reliant upon large-scale training datasets in which the task of predicting the next token mimics the test setting (here question-anse -nswering)) with non-negligible frequency.
Improving performance with few-shot learning More importantly, for the present work, we observe that the ability of the model to transfer knowledge from captioning and texi-modelling to visual question-answering improves the model presented with several examples of VQA data sequentially. We repeat the previous experiments with up to four examples of image-question-answee answer

---

Dosppett D ropostss i-ineen shots
Support from ImageNet
Quastion from Imageliet
(e) minilmageNet
Thok Induction
Model Completion
shoa??
blicket.
biScket.
30: Thise
from ImageNet
Question from VisualGenomee
biickett
(0) Fast vastvaA
[table)
Model Completion
2 way
Droepesda 2-amanathota
blicket.
biscket.
This
dix made oF?
the
wood
Figure 4: Examples of (a) the Open-Ended minilmageNet evaluation (b) the Fast VQA evaluation.
triples shown to the model as conditioning information in the continuous prefix sequence (using the interface in Figure 3).
These results are presented in Table I For contrast, we compare this performance condition in which we mix in some data from the VQAv2 training set with the captioning data. As we might expect, few-shot learning on four examples is outperformed by SGD on tens of thousands of examples, but few-shot performance clearly improves with more examples, and goes some way (38.2%) toward closing the gap from zero-shot performance (29.5%) to full SGD training performance (48.4%)
There are two important takeaways from the results presented in this section. First, they show that training a visual encoder through a pretrained and frozen language model results in system capable of strong out-of-distribution (zero-shot) generalization. Second, they confirm that the ability to rapidly adapt new tasks given appropriate entext is inherited from the pretrained language model and transfers directly to multimodal tasks.
4.2 Encyclopedic Knowledge
Here we study the extent which Frozen can leverage the encyclopedic knowledge in the language model towards visual tasks. The Conceptual Captions dataset is hypernymed (e.g. proper names are replaced with a general word like person). This enables us to rigorously study the transfer factual knowledge because all knowledge of named entities comes from language model pretraining.
Consequently, when we show the model an image of an airplane and ask "who invented this?" (Figure 1), the visual encoder has determined that the image contains an airplane, and the language model has used this to retrieve the factual knowledge that airplanes were invented by the Wright brothers, a fact which is referenced in the C4 training set through (text-only) articles about airplanes. This is fascinating chain of deduction. detailed analysis of this behaviour with more examples included in the Appendix (e.g. Figure Figure 10
Figure I1).
We bolster this finding quantitatively by evaluating performance OKYQA 27 a visual question- answering dataset designed to require outside knowledge in order to answer correctly. The pretrained language model's command of factual knowledge is of course dependent upon its scale, the performance of Frozen using pretrained language models of varying sizes: the base model with 7 billion parameters, and much smaller 400 million parameter language model pretrained on the same dataset. Table 2) shows the results: task performance scales with model size. Again finctuning performs worse than leaving the model frozen in of generalization performance. We stress that Frozen is never trained on OKVQA.
examine
4.3
Fast Word-to- Sisoooiiiuall-aeeeee Binding
In the multi-modal setting, fast-binding refers to model's ability associate a word with a visual category in few shots and immediately use that word in an appropriate way.

---

Open-Eu Open-Ended minilmageNNet and Real-Name minilmageNet To quantify the fast-binding capac- ity of of Frozen, we evaluate it on the minilmageNet meta-learning task [43]. Note that there are important differences with how we attempt minilmageNet and how it is approached in previous work. First, unlike standard meta-learning, we do not train Frozen on the (meta) task. Second, we evaluate Frozen in an open-ended fashion, where it must successfully generate correct category name (and then the EOS token) in order to be credited with a correct answer. Finally, although we use the same image classes the minilmageNNe test set, they are at higher resolution (224 x 2224 and with integer class labels [0, 1] replaced with nonsense words ('dax', 'blicket' etc). We make this adjustment because the nonsense words should have no (or less) intrinsic meaning to the language model than integers, whose relative order (for instance) should have been reflected in a massive text training corpus. We refer to this task as Open-Ended Ended minilmageNer. To assess how much difficulty is added by binding visual categories to nonsense words versus simply adapting to an image recognition task per se, we also consider version Real-Name minilmagenet in which visual categories in both the support set and the answer retain their original names. SeeuFigure 4a for an illustration.
On both versions of this evaluation, we experiment by exposing the model different numbers of inner-shots, repeats and task induction. On two-way Open-Enden minilmagenet, we observe that when Frozen is presented with a sequence of images and descriptions of new names for them, it is able to learn new names for the objects presented and then use these new names immediately with substantially above chance accuracy. Importantly, the ability of the model to these new words improves with more examples of the corresponding category. Notably, this upward trend is more pronounced when this supporting information involves different exemplars from the visual category (inner-shots) rather than repetitions a single exemplar (repeats). The fast-binding capacities of the model can thus be improved with richer and more varied visual support prompting.
On two-way Real-Name minilmagenet, we observe a similar trend but with higher absolute perfor- mance. This underlines the difficulty in Open-Ended minilmagenet introduced by having to assign novel words to categories that may otherwise be already known to the model, and because the real names may carry visual information leveraged from the captioning data the model was trained on.
In Table 4 we show that the observed effects on Ondn-Ended minilmagenet do not transfer to the 5-way setting, where Frozen is not significantly above chance. This shows that learning to bind five new names to five visual categories Frozen. As before, however, we do observe an upward trend in the model's capacity to return the actual name for a visual category among the five possibilities as the number of inner-shots or repeats increases. Further work is required and we look forward to progress in this more challenging setting.
single forward pass is beyond the current capabilities of
Task Induction Inner Shots Repeats
X
/
1
1
3
5
1
0
0
3
Frozen Frozen (Real-Name)
29.0
54.1
55.2
57.6
51.8
57.7
58.6
1.7
49.2
67.0
68.4
63.8
65.2
64.0
Frozen test-blind
48.5
46.7
45.3
Frozen test-blind (Real-Name)
1.0
12.6
33.0
ANIL Baseline 32)
73.9
81.7
84.2
Table 3: Performance of Frozen and baselines on Open Ended minilmageNet 2- 2-yay Tasks. Randomly picking between the two class labels (then emitting the EOS token) would yield 50% accuracy. As the model has to generate the answer, and is not counted correct if it paraphrases, this is not the best blind baseline, which is why we include open-ended blind baselines that also generate.
Fast-VQA and Guided- VQA As transformers are trained to model text, their attention weights learn to associate or "bind'- pairs of words across sentences. The experiments with minilmageNet show that this capacity can transfer directly to binding visual categories to their names, enabling the system generate the name demand. This raises the question of whether Frozen can integrate newly- visual category (and its names) more fully into the model's language system, so that it can, for instance, describe or answer questions about that category.
To test this capacity, we constructed a new task (Qust-VQQ- - out of two well-known datasets, ImageNet (35] and Visual Genome [17] For each question, the model is presented with nonsense
8

---

Task Induction
X
/
/
/
1
/
Inner Shots
1
3
1
1
Repeats
0
0
0
1
5
Frozen
18.0
21.3
22.4
22.1
21.5
21.1
20.9
Frozen (Real-Name)
0.9
12.4
34.0
31.0
32.0
33.2
33.8
Frozen test-blind
18.6
19.9
19.8
Frozen test-blind (Real-Name)
4.6
22.6
20.8
ANIL Baseline [32
45.5
57.7
62.6
Table 4: Performance Frozen and baselines on Open-Ended minilmageNet 5- 5-yay Tasks. Randomly picking between the five class labels (then emitting the EOS token) would yield 20% accuracy.
words ('dax' and 'blicket') and n images of the referents of those words (e.g. of a 'cat' or a 'dog') taken from ImageNet. is then asked question containing at least one those two words, about further image (taken from Visual Genome) in which both of the referents appear (see beegFigure 4p). As with minilmagenet, the words 'dax' and 'blicket' (and how they refer) should be new to Frozen, but the corresponding visual categories may be known from the Conceptual Captions training data, albeit by different names.
To quantify how much harder the introduction of new words for known categories makes this task, we also created variant (Guided-VQA) in which the original category names ('cat' or 'dog' are used instead of "dax" and 'blicket'. Guided- VQA is special case of Fast-VQA involving questions from Visual Genome, where the model is reminded what the important entities in the question look like prior to answering the question by labeling sample images with real category names. Guided- VQA does not require the same ability bind categories new words, but it does measure how well model can exploit task-relevant multimodal guidance when attempting new task in an otherwise zero-shot manner.
Past-VQQ and Guided-VQA are very challenging tasks because they are attempted without task- specific training, and because the underlying questions come from Visual Genome (VQAv2 images do not come with the necessary meta-data to construct the task). Visual Genome questions are particularly challenging because only single answer exists for each question. When scoring models, for simplicity we credit only an exact match with the output generated by the model, modulo the same post-proce prsseprocssiing applied for VQAv2. Because of the inherent difficulty of the task, we use strong baselines that can still utilize the large language model to verify strength of observed effects.
Fast-VQA
Guided-VOL
Inner Shots
0
1
3
Frozen
1.6
2.8
7.0
7.9
3.7
7.8
10.1
10.5
Frozen train-blind
0.7
0.3
1.3
0.4
1.9
2.3
3.7
3.7
Table 5: Performance of Frozen versus an equivalent blind model Fast and Guided-VOL VQA.
As shown in Table 5) the fact that the model improves with more shots in both Fast-VQA and Guided- VQA confirms that Frozen has some capacity to integrate novel words into its general capacity to process and generate natural language in multimodal context. It is notable that prefix-tuned model with no access to images improves moderately at Guided-VOA as more categories are presented, showing that additional linguistic cues (just being reminded of the words involved and the linguistic form of the task) goes some way to preparing for the upcoming question. As exemplified in FFigure 4| inspection of the model output confirms that in many cases it is indeed the multimodal (and not just linguistic) support that enables Frozen to improve performance as the number of shots increases. We observed that there are diminishing returns in performance gain as the number of shots increase. One possible explanation is that the shift from the training distribution of contexts with single images to multiple images causes inaccuracies in the model.
The Open-Ended minilmagenet, Real-Name minilmagenet, Fast- VQA and Guided-VOR -VQA evaluation sets are available to download at htttps:/fh229... gi github. i htt///rozee..eee

---

5 Discussion
5.1 Limitations
We believe this work is important proof-of-concept for a desired, much more powerful system capable of open-ended multimodal few-shot learning. Frozen achieves the necessary capacities to some degree, but key limitation is that it achieves far from otate-of-the-art performance specific tasks that it learns in a few shots, compared to systems that use the full training set for those tasks. As such, the main contribution of this work should be seen starting point or baseline for this exciting area of research of multimodal few-shot learning.
the
Further improvement can make the impressive zero-shot and few-shot generalization we observed more robust as reflected by higher accuracy and fewer seeds required to demonstrate our most compelling samples. Finally, there are many technical questions that were not explored in this proof- of-concept study, such as whether performance could be improved with more elaborate architectures for mixing vision and language. We leave the exploration of these possibilities to future investigations. The Open-Ende Ended minilmageNet, Real-Name minilmagenet, Fast-VQA and Guided-VQA benchmarks that we provide with this manuscript should facilitate the evaluation and analysis of future systems of this type.
5.2 Societal Impact
With the emergence of this new class of general purpose vision-language models, new capabilities of massive surveillance can be feasible. Both surveillance footage and publicly shared images can be analyzed for arbitrary questions without requiring any new labeled data or training of the system. As a mitigation for individuals, personal assistant software with similar capabilities can analyze publicly available documents about themselves to identify unintended exposures, even when novel concerns emerge either due to societal change or change of personal preferences.
Generative models of text that incorporate visual information can clevate the misuse of language- model generated content by making them even more convincing. Moreover, at this point we do not have sufficient tools to identify bias and toxicity issues of general purpose vision-guided language models. We invite the community to think about effective methods and benchmarks on this front.
More positively, systems like Frozen could be applied to assist visually impaired users of technology. Frozen's s ability to adapt to different styles of caption or question could enable a more personalised user experience in these cases.
There are environmental costs associated with training the large networks systems like Frozen. On the other hand, a system that can be trained once and then flexibly adapted to different settings could have a lower energy footprint overall than one that requires re-training for different applications.
5.3 Conclusion
We have presented method for transforming large language models into multimodal few-shot learning systems by extending the soft- poft-rrompting philosophy of prefix tuning (23] to ordered sets of images and text while preserving text prompting abilities of the language model. Our experiments
confirm that the resulting system, Frozen, is capable both of open-ended interpretation of images and genuinely multimodal few-shot learning even though the system is only trained to do captioning. One corollary of these results is that the knowledge required to quickly bind together or associate different words in language is also pertinent to rapidly binding language to visual elements across ordered set of inputs. This finding extends the conclusion of [26] that knowledge in transformer language models transfer to non-linguistic tasks the specific case of knowledge about few-shot learning.
Acknowledgements We wish to thank Sebastian Borgeaud and Jack Rae for preparing the pre- training text dataset and pretraining a selection of transformer language models, as well as Trevor Cai for help with experiments and infrastructure. We also wish to thank Pauline Luc, Jeff Donahue, Malcolm Reynolds, Andy Brock, Karen Simonyan, Jean- Baatistp Alayrac, Antoine Miech, Charlie Nash, Aaron van den Oord, Marc Deisenroth, Aida Nematzadeh, Roman Ring, Francis Song, Eliza Rutherford, Kirsty Anderson, Esme Sutherland, Alexander Novikov, Daan Wierstra, and Nando de Freitas for insightful discussions during the course of the project.
10

---

References
[1] Daniel Adiwardana, Minh- Minh-Thang Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppi- lan, Zi Yang. Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoe Le. Towards human-like open-domain chatbot. CoRR, 0ass22001.99777 2020.
[2] Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185-5198, 2020.
[3] Andrew Brock, Soham De, Samuel Smith, and Karen Simonyan. High perghpperfrmmnce large- scale image recognition without normalization. arXiv preprint arXiv:2 2021.
[4] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[5] David Chalmers. Gpt3 and general intelligence. Published in Daily Nous, 2021.
[6] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data-efficient adap- tation of pretrained language models for image captioning. arXiv preprint 1.Xiv:2102.10400 2021.
[7] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and aaaa--ann-langggge tasks via text generation. arXiv preprint arXiv:2102.0277 2021.
[8] Jacob Devlin, Ming-Wer Meei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /al..08800055 2018.
[9] Allen Institute for AI. C4 search. https://c4-seare apps allenai. org/ 2021-04-06.
Accessed:
[10] Yash Goyal, Tejas Khot, Douglas Summers-Ster Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 2017.
[11] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wer Wei Chang. detrieval-augmented language model pre-training, 2020.
Realm:
[12] Tracy H Heibeck and Ellen M Markman. Word learning in children: An examination of fast mapping. Child development, pages 1021-1034, 1987.
[13] MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. A comprehen- sive survey of deep learning for image captioning. ACM Computing Surveys (CsUR), 51(6):1336 2019.
[14] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yunhsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision, 2021.
[15] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, AI Borchers, Rick Boyle, Pierre luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau. Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami. Rajendra Gottipati. William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alck Jaworski, Alexander Kaplan, Harshit Khaitan, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary. Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of a tensor processing unit, 2017.
[16] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, Ethan Perez, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint (eXiiv:999.000 02950, 2019.
11

---

[17]
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017.
[18] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: /88. 2018.
[19] Georgia Tech Visual Intelligence Lab. Vqa python api and evaluation code. htttps///githut
com Com/GT-Visile Sor--isson--aae Lab/VQA
[20]
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2/04.08669, 2021.
[21] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021.
[22] Sheng Li, Zhiqiang Tao, Kang Li, and Yun Fu. Visual to text: Survey of image and video captioning. IEEE Transactions on Emerging Topics in Computational Intelligence, 3(4):297- 312, 2019.
[23] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprini arXiv:2 00100, 2021.
[24] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre- pre-training for vision-language tasks, 2020.
[25] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining tano-sgi visiolinguistic representations for siion-and-languue tasks. arXiv preprint arXiv:1908.0223 2019.
[26] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.0522 2021.
[27] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok- A visual question answering benchmark requiring external knowledge. In 2019 IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), pages 3190-3199 3199. IEEE Computer Society, 2019.
[28] Fabio Petroni, Tim Rocktäschel, Patricc S. H. Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases? CoRR, (00000969.. 2019.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[30] Alec Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
[31] Colin Raffel Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
[32] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv: 1909.09157, 2019.
[33] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
[34] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? arXiv preprint arXiv:2002.00 10 2020.
[35] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 1153):211-252,, 2015.
[36] Thomas Scialom, Patrick Bordes, Paul-Alexis Dray, Jacopo Staiano, and Patrick Gallinari. What bert sees: Cross-modal transfer for visual question generation. arXiv preprint arX12:2002.10832 2020.
12

---

[37] Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymeed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2556-2565, 2018.
[38] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations, 2018.
[39] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LoGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism, 2020.
[40] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguisti representations. arXiv preprint arXiv: 1908.085800 2019.
[41] Yingtao Tian and Jesse Engel, Latent translation: Crossing modalities by bridging generative models, 2019.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: 1706 03762, 2017.
[43] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning, arXiv preprint arXiv: (6t1::6600000000, 2016.
[44] Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. Multi-modal answer validation for knowledge-based vqa, 2021.
[45] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers, 2021.
[46] Zachary M Ziegler, Luke Melas- Serias,,rrizzi, Sebastian Gehrmann, and Alexander M Rush. Encoder- agnostic adaptation for conditional language generation. arXiv preprint 1908, 06938, 2019.
13


Reviews:
[{"review_id": "eBsfRceT_gM", "reviewer": "Reviewer_J1ff", "summary": "The paper proposes a new way to reuse large pre-trained language models and condition their generation on visual input. The authors build on the idea of pretext tuning and make pretext a (CNN) function on an image. By training the CNN for VQA, keeping the language model decoder fixed, the authors bring visual information into the input space of the language model. The authors propose several benchmarks for few-shot classification and captioning and show that their model demonstrates promised capabilities and achieves better than random results.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While it was standard for the computer vision community to reuse large pre-trained CNNs for image/video or even multi-modal tasks, doing the same with language models was not possible. This paper proposes to reuse the large pre-trained language models in a simple and elegant way. \nThe paper is very well written and clearly conveys the main point. I like how the authors demonstrate numerous capabilities that are enabled by such design, in the Experiments. I believe that this work may pave the way for new interesting research in multi-modal learning, especially given that the authors introduce new benchmarks.\n\nThe only drawback I see in this paper is the weak results in the introduced benchmarks. While this is not critical at this stage, it would be useful to have a better idea why the results are low, i.e. is it the difficulty of the task or simply not very capable individual components that are comprising the pipeline.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "_gvjlEojNAr", "reviewer": "Reviewer_DtyQ", "summary": "The paper proposes to utilize the zero-shot/few-shot generalization ability of frozen language models in a multi-modal setting, by projecting visual input features into the text input feature space, through moderate pre-training on Conceptual Captions (the LM is frozen during pre-training). \n\nThe paper designs several interesting experiment settings to show the model’s ability of task adaptation, utilizing pre-trained encyclopedia knowledge, and novel concept binding. \n\nMy main concern is that it is hard to judge whether the results in these experiments are significant or somewhat trivial. But given the novelty of the setting and the promise, I would lean towards acceptance but I look forward to the author’s response.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\n\nThe picture the paper tries to paint is appealing and makes sense. By projecting the visual features into the same features space as the input embedding space of a frozen LM, it is reasonable to expect that the Frozen LM can make use of this visual input information and we could tap into the great zero-shot/few-shot generalization ability of the Frozen LM.\n\nThe paper presents three interesting experiment settings that truly explore the potential usage of the model, i.e. to quickly adapt to new tasks and new concepts.\n\n\n\nCons:\n1. The paper somehow falls short as it does not deliver the same level of zero-shot ability as the original GPT3 on NLP tasks. But I won’t count this as a reason for rejection, as the authors have used reasonable methods and the unsatisfying results just show that this is an interesting and hard problem.\n\n2. Hard to judge whether the results are meaningful or trivial. The main metric to judge the significance of the results is the difference between Frozen and Frozen_blind. However, as all numbers are so far away from a reasonable baseline model, it is hard to judge the meaningfulness of the difference between model numbers.\nFor example, on VQA, a blind Frozen (without seeing the image) gets 33.3 while a Frozen gets 38.2. The numbers are so far away from reasonable baselines (for example, on VQA, a language-only model without any visual information can get over 40% acc), it makes it hard to say whether the 5 points improvement is significant or not.\n\nDetailed comments:\n3. Line 126-127: it is stated that fine-tuning the LM hurts performance, which is somehow in contrast to what is observed in V&L model pre-training (e.g., ViLBERT, VisualBERT, LXMERT). I wonder what would be the possible reason.\n\n4. Practically, if we just want a model with a good zero/few-shot ability on V&L tasks, we don't really need to fixate on keeping weights fixed during pre-training on multi-modal data? I see that the reason this paper keeps the weights frozen is simply because the performance is worse. I wonder if there are other reasons that we want to keep the weight frozen.\n\n5. Frozen_train-blind: Table 1 says Frozen_train-blind is mentioned in Section 4.1, but I cannot seem to find it mentioned in the main text.\n\n6. What is Frozen VQA? It says “mixing in data from VQA”. Does it mean mixing in VQA during pre-training?\n\n7. Table 1: The Frozen-finetune is a bit confusing, as it could mean fine-tuning LM during Conceptual Caption pre-training or fine-tuning on VQA with few examples. Judging from the context, I assume it is the former but it would be helpful to be clearer on this.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Nothing I could think of.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "BC0Oe0P3IZH", "reviewer": "Reviewer_GTTm", "summary": "This paper proposes a way to train a few-shot image-text neural network in which the text encoder weights are frozen, but used to train the image encoder from scratch. It provides an interesting training framework and evaluates few shot performance on image classification and vqa tasks, also studying several qualities of interesting, e.g. rapid task adaptation, encyclopedic knowledge, and fast concept binding.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The training paradigm is really cool, and definitely novel. The merit of this paper is strong as having few-shot learners that can adapt to tasks they've never seen before is very desirable. The paper is also generally well written and easy to follow. The primary drawback for the significance of this work is that the task performance is very low, but the authors do acknowledge that beating SOTA is not their intention. Also, the paper properly compares to \"blind\" baselines to prove that the network actually makes use of visual information, which is a necessary comparison and further verifies the merit of their work. \n\nOther positives include that comparison to prior work is thorough and there are clear distinctions made between this papers contributions and those of related work.\n\nOtherwise, there are some questions + comments:\n- L34: what is meant by \"ordered sets\"? do you just mean pairs of images and words as input? it's a bit confusing. \n- L73-80: the terminology of prefix vs. prompt vs. infix is not clear. Could you clarify the differences and revise the writing for clarity?\n- L122: it seems like the term \"n\" is used to reference different concepts/quantities throughout the paper, which gets confusing. In L22, is the number of visual prefix tokens = n? this should be explicitly stated. Then, in tables 1,2 n refers to the number of shots. Then, L274 refers to n as the number of images of the referents of words.\n- L130-138: is the difference between y and t that y are text tokens in their raw form, and t are their embeddings? from the notation it seems like they're the same thing, especially since x is defined on i_1, i_2\n- L30-138: why is there swapping between i_1, i_2 and then generalizing it to i_1, i_2, ...., i_n?\n- L136: can you define the relative position encoding? it's not defined/clear to me. Furthermore, did you ever actually check changing the image prefix positions? I would be curious to know how well the relative position encoding works, and how much the input ordering is needed as an inductive bias.\n- L154: how do we know that the \"different ways of inducing the task to the model\" aren't potentially harmful or biased to different groups?\n- Figure 4: The purpose of injecting nonsense words isn't explained early enough (it's only ever formally mentioned on L241-244), can you please address that sooner in the text. Also, why isn't there an answer in the upper row for the \"Question from ImageNet\"?\n- Number of shots vs. number of inner shots. Are there two differences here? The first being that number of inner shots is per category, and the second being that the \"full\" example isn't provided for inner shots? I am trying to connect it to the VQA example in the number of shots definition, in which the answer is provided in addition to the image and question.\n- Number of repeats. Repeating samples doesn't feel well motivated, and isn't really discussed in the text. Why is this done? I am not understanding how this helps us \"explore how the model integrates visual information\"\n- L174: is the word \"fast\" supposed to precede general knowledge? what about this approach makes the access to general knowledge fast? or is the point that we don't have to retrain the language model to be task specific? It just seems a bit redundant with \"rapid adaptation\" and \"fast binding\"\n- Table 1,2 Frozen_train-blind: why do these results do worse with n=4 shots?\n- Table 1,2: clarifying question- are all of these results with having 2 visual prefixes?\n- Table 1: How much VQAv2 data is mixed in?\n- L189-191. This is an important take-away, but it is not new - see Visual Dialogue without Vision or Dialogue, Massiceti et al. Maybe you already are familiar with related work showing this, but nothing is cited here, so it would be good to mention this confirms results already shown in prior work.\n- L207: Did you ever look at increasing the number of shots until full SGD training performance was reached? is it possible?\n- Fast binding: could the purpose of fast binding be better motivated? right now the use of nonsense words doesn't really provide intuition of how this is useful. Do you think we could extend fast binding to other languages? That seems like it would be an impactful use case.\n- Table 3: It seems like Frozen and Frozen (real name) plateau as both the number of inner shots or repeats increases, or performance even gets worse (repeats = 5, frozen real name). Why is this? This happens both for the 2-way and 5-way experiments.\n- L241-242: What does \"this allows the system to express its answers with word-like tokens\" mean?\n- L251-255 states that Frozen improves performance with more examples, which feels like an exaggeration of the results, given that the performance plateaus after 3 which is not addressed in the analysis.\n- Fast-VQA and Real-Fast-VQA: Not enough dataset details were provided on this in the main text, additional details on the number of samples etc would be helpful. Also, I'd reconsider using \"Fast\" in the name, it's a bit confusing. And \"Real-fast\" sounds like the data is now really fast. Is the word fast being used because the data is evaluated in a few shot setting?\n\nStyle suggestions, typos, etc.\n- L34: remove \"e.g.\"\n- L136: the ellipses here are an awkward way to end a sentence\n- L149: people that aren't native English speakers may not know what a \"stress test\" is. I'd suggest rewording.\n- L153: remove \"at\"\n- L215: the e.g. feels awkward here, consider removing\n- L258-259, 265: wording a bit weird/hard to follow\n- L283: \"a model is reminded what the important entities in the question look like\" is a bit confusing- is this because the question now has the real category names included? I would reiterate that explicitly.\n- L325-326: what does this mean? \n- L327: adding a hyphen after language may help the flow here. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. There isn't analysis of samples that were already correctly classified with the blind models. What did those reveal, did you look at them? This would provide better insight to this issue and also help researchers build upon this work to specifically target these biases.\n2. The classification baselines only explore at most a 5-way classification task, and with performance so low, it isn't particularly promising that results would extend well to a more realistic setting.\n3. The role of the relative position encoding is not studied, and so it is unclear how necessary the image-text embedding ordering is for the experimental results.\n4. The plateau or worsening of performance as the number of shots or repeats increases is not addressed, and is a bit concerning. Why would performance get worse with more examples?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "AuXgH6-kvpR", "reviewer": "Reviewer_qf5a", "summary": "The paper proposes a new GPT-2 style image-language model, Frozen, which is trained in two stages: \n1. common GPT-2 pre-training on huge text-alone corpora\n2. training of a Resnet image encoder while freezing the GPT-2 and text embedding weights, on the task of image captioning (i.e. language modeling conditional on two image encoding vectors with the same dimension as token embeddings).\n\nSince GPT-2 exhibits few-shot abilities in language-alone tasks, the idea is Frozen can also benefit from the GPT-2 language pertaining and exhibit *multimodal* few-shot abilities by the minimal design of adding a visual encoder. Note that the freezing does not make training more efficient, since gradients have to pass through all Transformer layers to the visual encoder.\n\nThe Frozen model is applied to zero/few-shot learning of two downstream tasks: VQA and few-shot image classification. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Significance**: The work is at the intersection of two important topics: multimodal (vision-language) learning and few-shot learning. Given the surprising zero/few-shot abilities of recent large language models (e.g. GPT-2/3, T5), it is natural to investigate how such abilities can be instantiated in a multimodal setup, which requires the learner not only for rapid task adaption and knowledge transfer, but also fast binding between visual features and language concepts. Such a study should be very impactful and pertaining both to cognitive science and AI, given how humans possess such strong abilities but not machines yet. \n\nThe work also relates to an important debate in NLP: can language models learn meanings from form (text corpora) alone? By leveraging a frozen language model to \"heavy-lift\" a visual encoder head, it probes to what extent can language learning enable learning and binding to visual concepts.\n\n**Originality**: The work is interesting and original.\n\n**Clarity**: The paper is well written and the main idea is simple and easy to follow. The figures are illustrative, though I was *slightly* confused by the left part of Figure 4 and what \"0-repeats, 0-shots, ..., 0-repeats, 2-inner-shots\" mean.\n\n**Quality**: It is interesting how few-shot enables some boosts in results compared to zero-shot. However, my main concern is the poor experimental results in general. Please see below. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Unfortunately, the results seem too low to be claimed surprising. With better experiments I'd definitely raise the score.\n\n- For VQA (Table 1), the gap between Frozen and Frozen (train-blind) seem very small, which suggests maybe the majority of correctly answered questions are thanks to language-alone learning? I'd like to see some non-trivial examples. \n\n- For few-shot classification (Table 3, 4), the accuracies are merely 50-something and 20-something percents for two and five way classification, while random baseline would achieve 50% and 20%. This makes me hard to believe Frozen achieves \"fast concept binding\". Besides, I am not sure if the Real-Name result (66%/34%) can prove the model achieves \"slow concept binding\" either, given the baseline's 84%/62% performance. \n\n- Given the poor lag behind visual-language baselines, is it possible to make a fairer comparison by controlling the training data to Frozen and baselines? For example, make sure Frozen and Oscar receives the same amount of training data (text corpora for language-alone pretraining, paired image-captions for visual encoder training), and then study the zero/few-shot test case? Showing the baselines performing a lot better with more training data isn't very helpful to determine how Frozen behaves. \n\n- Conceptually, why would language learning first enable visual learning later? Why not the other direction (visual learning first, language learning later), given that's more like how humans learn? I'm also curious about such a Frozen variant: mix the two training stages (language modeling, language modeling conditional on visual features) together and learn GPT-2 and visual encoder together. If such a variant is worse than Frozen, it better justifies the two-stage approach with freezing. \n\n- Is the visual encoder pre-trained anyway? It is not clearly stated in the main paper. If not, it might be interesting to evaluate the visual encoder alone, testing how much better at vision tasks it has been than a random encoder, thanks to the feedbacks from the language model. I'm also curious what if a pre-trained ResNet is used (frozen or fine-tuned) what will happen. If so, what type of pre-training is used, based on what data?", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a proof-of-concept for multimodal few-shot training of frozen language models — where the parameters of a pre-trained language model are kept fixed while an image encoder is trained to prompt it to generate text. Despite its simplicity, the approach demonstrates surprising generalization capabilities of the model when presented unseen visual concepts, although as pointed out by the authors themselves, the results are still far from what current models can achieve on these tasks and more work is needed to refine these ideas further. 
Decision: Accept (Poster)