Flamingo: a Visual Language Model for Few-Shot Learning
Authors: Jean-Baptiste Alayrac
Jeff Donahue
Pauline Luc
Antoine Miech
Iain Barr
Yana Hasson
Karel Lenc
Arthur Mensch
Katherine Millican
Malcolm Reynolds
Roman Ring
Eliza Rutherford
Serkan Cabi
Tengda Han
Zhitao Gong
Sina Samangooei
Marianne Monteiro
Jacob Menick
Sebastian Borgeaud
Andrew Brock
Aida Nematzadeh
Sahand Sharifzadeh
Mikolaj Binkowski
Ricardo Barreira
Oriol Vinyals
Andrew Zisserman
Karen Simonyan
Venue: NeurIPS 2022 Accept (2022)

Abstract: Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer, captioning tasks, which evaluate the ability to describe a scene or an event, and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.

Flamingo: a Visual Language Model for Few-Shot Learning
Jean-Baptiste Alayrac*#
Jeff' Donahue
Pauline Luc"
Antoine Miech
lain Barr
Yana Hasson
Karel Lene
Arthur Mensch
Katie Millican
Malcolm Reynolds
Roman Ring!
Eliza Rutherford
Serkan Cabi
Tengda Han
Zhitao Gong
Sina Samangooei
Marianne Monteiro
Jacob Menick
Sebastian Borgeaud
Andrew Brock
Aida Nematzadeh
Sahand Sharifzadeh
Mikolaj Binkowski
Ricardo Barreira
Oriol Vinyals
Andrew Zisserman
Karen Simonyan" #1
Equal contributions, ordered alphabetically, Equal contributions, ordered alphabetically,
Equal senior contributions
DeepMind
Abstract
Building models that be rapidly adapted novel tasks using only handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily inter- leaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question question-ansswring where the model is prompted with a question which it has to answer; captioning tasks, which evaluate the ability describe a scene or event; and close-ended tasks such multiple-choice visual guestion-answering For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.
deeepmind.com 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Corresponding authors:

---

Input Prompt
Completion
This as chinchilla, They found Chile.
This is shiha. They are Jopan.
faminge. They are found in the
This
Caribbean South America.
What is the tille this paintin? Answer. The Hallucinogenic Toreador.
pointing displayed? Answer: Museum,
What the mome of city where this
Arles.
Chiput: "Undergronda"
Outpat "ongresss
UNGERGRDUND
EENNGRESS
Ourput
"Soulomes"
2+1=3
S+6=11
3x6
3x6=18
Osspuc:
peopaganda poster depicting dressed as French emperur
Ourgut pink rom with flamingo Boot.
portrait Salvador Dall
Ourput
Napolzca hokiing
head.
pocce of cheese.
sanglois longs des violons de l'automme blessent
Pour qui sout ces serpents sifflent
Je sais un ceur but
langueur
munoicme.
pandas: 3
dogs:
giraffes:
I like reading
favourite play Hamleei 1
favorite book
Dreams from Father.
What happens to the man after himing ball?
falls down.
i iPod
o cityuape: Chicago
What mukes you think Chirago?
picture toddy
This is an apple with sicker on
think it' Chicago becouse of the Shedd Aquariun in the baacspraanll
What
They
laning
say? nicker soys "iPua" Where is the photo saken?
What
looks computes.
What the common thing
shout these thrse
looks lie it's takes in bockyand.
surpeiing?
Yes, is suppising.
Do think it printed handwrites?
What about this one? Which city is this and what Ganous landmark helped you recog nise city?
Whs is this pisture suppissnn you?
cattonn, imagee?
like it's hundwriten.
Ltbink it supeising because loddy lean are soi saulll found on the moon.
sticke??
This Tokyo. (hink it's Toloyo because of the Tokyo
white.
TIwwe.
Figure I: Selected examples of inputs and outputs obtained from Flamingo-80B. Flamingo can rapidly adapt to various image/video understanding tasks with few-shot prompting (top). Out of the box, Flamingo is also capable of multi-image visual dialogue (bottom). More examples in Appendix C.

---

150%
Flami (808) 32 shels
Previous zero/few shat SoLA
100.0%
Aggregated performance
125%
115
128
233
Perforance rolative
100%
L00
105-107
69
75
80
80
50%
Flamingo-808 Flamingo-9B
48
70.0%
25%
41
34
Flamingo 3B
YouCaok?
MSRVITQA
TeXtVOA
VisD
coco
VOAv2
VATEX
VizWiz
Hateful
Momes
OKVQA
MSVDQA
STAR.
FlicK30
IVOA
NexiQA
Number of shots
Figure 2: Flamingo results overview. Left: Our largest model, dubbed Flamingo, outperforms
state-of-the-art fine-tuned models on of the 16 tasks we consider with no fine-tuning. For the tasks with published few-shot results, Flamingo sets the new few-shot state of the Note: We omit RareAct, our 16th benchmark, as it is zero-shot benchmark with no available fine-tuned results to compare to. Right: Flamingo performance improves with model size and number of shots.
Introduction
One key aspect of intelligence is the ability to quickly learn to perform a new task given a short instruction [33, 70]. While initial progress has been made towards a similar capability in computer vision, the most widely used paradigm still consists of first pretraining on a large amount of supervised data, before fine-tuning the model on the task of interest [66, 118, 143]. However, successful fine- tuning often requires many thousands of annotated data points. In addition, it often requires careful per-task hyperparameter tuning and also resource intensive. Recently, multimodal vision-language models trained with contrastive objective [50, 85) have enabled zero-shot adaptation to novel tasks, without the need for fine-tuning. However, because these models simply provide a similarity score between text and an image, they can only address limited cases such classification, where finite set of outcomes is provided beforehand. They crucially lack the ability to generate language, which makes them less suitable to more open-ended tasks such as captioning visual question- answering. Others have explored visually-conditione language generation [17, 114, 119, 124, 132] but have not yet shown good performance in low-data regimes.
We introduce Flamingo, a Visual Language Model (VLM) that sets new state of the in few-shot learning on wide range of open-ended vision and language tasks, simply by being prompted with few input/output examples, as illustrated in Figure 1. Of the 16 tasks we consider, Flamingo also surpasses the fine-tuned state of the art on 6 tasks, despite using orders of magnitude less task-specific training data (see Figure 2). achieve this, Flamingo takes inspiration from recent work on large language models (LMs) which good few-shot learners [11, 18, 42, 86]. A single large LM can achieve strong performance on many tasks using only its text interface: a few examples of a task are provided to the model prompt, along with query input, and the model generates a continuation to produce a predicted output for that query. We show that the same can be done for image and video understanding tasks such as classification, captioning, or guestion-answering these can be cast as text prediction problems with visual input conditioning. The difference from a LM that the model must be able ingest a multimodal prompt containing images and/or videos interleaved with text. Flamingo models have this capability- -hey are visuall--conditionel conditioned autoregressive text generation models able to ingest sequence of text tokens interleaved with images and/or videos, and produce text as output. Flamingo models leverage two complementary pre-trained and frozen models: vision model which can "perceive" visual scenes and a large LM which performs basic form of reasoning. Novel architecture components are added in between these models to connect them in way that preserves the knowledge they have accumulated during computationally intensive pre- vre-training. Flamingo models are also able to ingest high- nigh-resolution images or videos thanks to a Perceiver-basee based [48] architecture that can produce small fixed number of visual tokens per image/video, given large and variable number of visual input features.
A crucial aspect for the performance of large LMs is that they are trained on large amount of text data. This training provides general-purpose generation capabilities that allows these LMs to perform well when prompted with task examples. Similarly, we demonstrate that the way we train the Flamingo models is crucial for their final performance. They are trained on a carefully chosen
3

---

Output: text
+
Pretrained and frozen
very serious cat.
Trained from scratch
LM block
n-th GATED XATTN-DENSE
Perceiver Resampler
Perceiver Resampler
1st LN block
GATED KATTN-DENSE
Vision Encoder
Vision Encoder
Processed text
<image> This is very cute dog. -0g.<30 <immgez>
This
Interleaved visual/text data
This is very cute dog.
This is
Flamingo architecture overview. Flamingo is family of visual language models (VLMs) that take as input visual data interleaved with text and produce free-form text as output.
Figure 3:
mixture of complementary large-scale multimodal data coming only from the web, without using any data annotated for machine learning purposes. After this training, Flamingo model can be directly adapted vision tasks via simple few-shot learning without any task-specific tuning.
Contributions. In summary, our contributions are the following: (i) We introduce the Flamingo family of VLMs which perform various multimodal tasks (such captioning, visual dialogue, from only a few input/output examples. Thanks to architectural
or visual snection-answeree innovations, the Flamingo models efficiently accept arbitrarily interleaved visual data and text as input and generate text in an open-ended manner. (ii) We quantitatively evaluate how Flamingo models be adapted various tasks via few-shot learning. We notably reserve large set of held- out benchmarks which have not been used for validation of any design decisions or hyperparameters of the approach. We these to estimate unbiased few-shot performance. (iii) Flamingo sets new state of the art in few-shot learning on a wide array of 16 multimodal language and image/video understanding tasks. On 6 of these 16 tasks, Flamingo also outperforms the fine-tuned state of the art despite using only 32 task-specific examples, around 1000 times less task-speciff training data than the current state of the art. With larger annotation budget. Flamingo can also be effectively fine-tuned to set a new state of the art on five additional challenging benchmarks: VQAv2, VATEX, VizWiz, MSRVTTQA, and HatefulMemes.
2 Approach
This section describes Flamingo: visual language model that accepts text interleaved with im- ages/videos as input and outputs free-form text. The key architectural components shown Figure chosen to leverage pretrained vision and language models and bridge them effectively. First, Perceiver Resampler (Section 2.1) receives spatio-temporal features from the Vision Encoder (obtained from either an image or video) and outputs a fixed number of visual tokens. Second, these visual tokens are used to condition the frozen LM using freshly initialised cross-attention layers (Section 2.2) that interleaved between the pretrained LM layers. These new layers offer an expressive way for the LM to incorporate visual information for the next-token prediction task. Flamingo models the likelihood of text y conditioned interleaved images and videos follows:
ply II (eeel
where ye is the ε-th language token of the input text, Vet is the set of preceding tokens, Zeg is the set of images/videos preceding token y/ model. The ability to handle interleaved text and visual sequences (Section 2.3) makes it natural to use Flamingo models for in-context few-shot learning, analogously to GPT-3 with few-shot text prompting. The model is trained diverse mixture of datascts as described in Section 2.4.
interleaved sequence and p is parametrized by a Flamingo

---

gated corse)
5. tmput visuat features
elpaaaettee, sottn gating paraneter 1911 'fe
attention
"ppliess GATED XATH OUNE Jayer.
LM Layer
KoVe|1]
Q:[V]
y y [un((T)ps_omm)) streestinn Reaa)
X
GATED XATIN-DENSE
FFW
Gatel resd forend (dans) Lever
010 M
Repiaar astr-attention FTF language
cross attention
ymy froeen./fely
return # cutpes visuiity cnformed language features
Vision
Language
input
input
Figure 4: GATED XATTN-DENSE layers. To condition the LM visual inputs, we insert new cross-attention layers between existing pretrained and frozen LM layers. The keys and values in these layers are obtained from the vision features while the queries derived from the language inputs. They are followed by dense feed-forward layers. These layers are gated so that the LM is kept intact initialization for improved stability and performance.
2.1 Visual processing and the Perceiver Resampler
from pixels to features. Our vision encoder is a pretrained and frozen Normalizer- Free ResNet (NFNet) [10] - we use the F6 model. We pretrain the vision encoder using a contrastive objective on our datasets of image and text pairs, using the two-term contrastive loss from Radford al. [85]. We use the output of the final stage, 2D spatial grid of features that is flattened to a 1D sequence. For video inputs, frames are sampled at FPS and encoded independently to obtain a 3D spatio-temporal grid of features to which learned temporal embeddings are added. Features are then flattened to ID before being fed to the Perceiver Resampler. More details on the contrastive model training and performance are given Appendix B.1.3 and Appendix B.3.2, respectively.
Vision Encoder:
from varying-size large feature maps to few visual tokens. This module connects the vision encoder to the frozen language model as shown in Figure 3. takes as input variable number of image or video features from the vision encoder and produces fixed number of visual outputs (64), reducing the computational complexity of the vision-text cross-attention. Similar to Perceiver [48] and DETR [13], we learn predefined number of latent input queries which are fed Transformer and cross-attend to the visual features. We show in our ablation studies (Section 3.3) that using such vision-languag resampler module outperforms plain Transformer and an MLP. provide an illustration, more architectural details, and pseudo-code Appendix A.I.I.
Perceiver Resampler:
2.2 Conditioning frozen language models on visual representations
Text generation is performed by a Transformer decoder, conditioned the visual representations produced by the Perceiver Resampler. We interleave pretrained and frozen text-only LM blocks with blocks trained from seratch that cross-attend the visual output from the Perceiver Resampler.
Interlcaving new GATED XATTN-DENSE layers within frozen pretrained LM. We freeze pretrained LM blocks, and insert gated cross-atiention dense blocks (Figure 4) between the original layers, trained from scratch. To ensure that at initialization, the conditioned model yields the same results the original language model, output a newly added layer by tanh(a) before adding it to the input representation from the residual connection, where is a layer-specific learnable scalar initialized to 0 [4]. Thus, at initialization, the model output matches that of the pretrained LM, improving training stability and final performance. In our ablation studies (Section 3.3), compare the proposed GATED XATTN-DENN
tanh-gating mechanism [41]. This multiplies the
layers against recent alternatives [22, 68] and explore the effect of how frequently these additional layers are inserted trade off between efficiency and expressivity. See Appendix A. 1.2 for more details.
Varying model sizes. We perform experiments across three models sizes, building on the 1.4B, 7B, and 70B parameter Chinchilla models [42]; calling them respectively Flamingo-3B. Flamingo-9B and
5

---

Flamingo- 80B. For brevity, we refer to the last as Flamingo throughout the paper. While increasing the parameter count of the frozen LM and the trainable vision-text GATED XATTN-DENSE modules, we maintain fixed-size frozen vision encoder and trainable Perceiver Resampler across the different models (small relative to the full model size). See Appendix B.1.1 for further details.
2.3 Multi-visual input support: per-image/video attention masking
The image-causal modelling introduced in Equation (1) obtained by masking the full text-to-image cross-attention matrix, limiting which visual tokens the model sees at each text token. At given text token, the model attends to the visual tokens of the image that appeared just before it in the interleaved sequence, rather than to all previous images (formalized and illustrated in Appendix A.1.3). Though the model only directly attends to a single image at a time, the dependency on all previous images remains via attentotnn in the LM. This single-image cross-attention scheme importantly allows the model to seamlessly generalise to any number of visual inputs, regardless of how many are used during training. In particular, we use only up to 5 images per sequence when training our interleaved datasets, yet our model is able to benefit from sequences of up to 32 pairs (or "shots") of images/videos and corresponding texts during evaluation. We show in Section 3.3 that this scheme is more effective than allowing the model to cross-attend to all previous images directly.
2.4 Training on mixture of vision and language datasets
We train the Flamingo models on a mixture of three kinds of datasets, all scraped from the web: an interleaved image and text dataset derived from webpages, image-text pairs, and video-text pairs.
M3W: Interleaved image and text dataset. The few-shot capabilities of Flamingo models rely on training on interleaved text and image data. For this purpose, we collect the MultiModal MassiveWad Web (M3W) dataset. We extract both text and images from the HTML of approximately 43 million webpages, determining the positions of images relative to the text based on the relative positions of the text and image elements in the Document Object Model (DOM). An example then constructed by inserting <image> tags in plain text at the locations of the images on the page, and inserting a special <EOC> (end of chunk) token (added the vocabulary and learnt) prior to any image and at the end of the document, From each document, we sample a random subsequence of I, = 256 tokens and take up to the first N images included in the sampled sequence. Further images are discarded in order to save compute. More details are provided in Appendix A.3.
Pairs of image/video and text. For our image and text pairs we first leverage the ALIGN [50] dataset, composed of 1.8 billion images paired with alt-text. To complement this dataset, we collect our own dataset of image and text pairs targeting better quality and longer descriptions: LTIP (Long Text & Image Pairs) which consists of 312 million image and text pairs. We also collect a similar dataset but with videos instead of still images: VTP (Video & Text Pairs) consists of 27 million short videos (approximately 22 seconds on average) paired with sentence descriptions. We align the syntax of paired datasets with the syntax of M3W by prepending <image> and appending <EOC> each training caption (see Appendix A.3.3 for details).
Multi-objective training and optimisation strategy. We train our models by minimizing weighted sum of per-dataset expected negative log-likelihoods of text, given the visual inputs:
M Am (.)
log 1og
where D. and Am are the mn-h dataset and its weighting, respectively. Tuning the per-dataset weights Am is key to performance. We accumulate gradients over all datasets, which we found outperforms "round-robin" approach [17]. We provide further training details and ablations in Appendix 1.2.
2.5 Task adaptation with few-shot in-context learning
Once Flamingo is trained, we use it to tackle a visual task by conditioning it on a multimodal interleaved prompt. We evaluate the ability of our models to rapidly adapt to new tasks using in- context learning, analogously to GPT-3 [11], by interleaving support example pairs in the form of (image, tezt) or (video, tezt), followed by the query visual input, to build prompt (details in

---

ORYQA
VQAV2 40
10000
MINVDSS (V)
VIIEX (V
FRARK IX
MSRVITOA (V)
1V0A
YouCook2 (V)
STAR (V)
WADII0 (0)
Teat C8
NeaIA
MadialMenns (D)
Rucket (V)
ZerolTew shot SOTA
43.3
[54
[114]
38,2
[12]
32.2
35.2
[S]
19.2
[33]
12.2
[.4]]
39.4
11.6
[:9]
66.1
[Ra]
40.7
[83
(0)
(16)
(4)
(0)
(0)
(0)
(0)
(0)
60x
(0)
(0)
41.2
49.2
7508
27.5
20.1
28.5
641.6
11.5
52.7
35.8
39.6
46.1
50.1
21.5
35.7
55.4
Floniggo:BB
43.3
53.2
85.0
33.0
50.0
34,0
72.0
14.9
35.7
64.6
41.3
47.3
52.7
22.4
53.6
32
45.9
$7.1
994
42.6
99.2
45.5
71.2
25.6
57.7
76.7
41.6
47.3
50.6
26.1
56.3
44.7
31.
79.4
3022
39.5
29.8
51.3
13.7
33.2
35.0
41.3
-5.0
31.8
23.0
37.00
37.9
Plamnggo 4B
49.3
56.3
93.1
36.2
51.7
34,9
72.6
18.2
37.7
70.8
42.8
50.4
53.6
24.7
62.7
32
$1.0
00.4
106.3
47.2
57.4
44.0
72.8
29.4
40.7
72.3
41.2
50.4
52.6
28.4
63.5
3056
30.3
$4.3
35.6
26.7
31.5
07.2
174
40.7
6011
391.7
3230
3500
25.7
46.4
Flamingo
32
57.8
57.4
63.1
67.6
1138
103.2
52.3
41.7
56.0
68.1
39.6
49.8
75.4
75.1
23.9
31.0
44.1
45.3
74.5
86.8
422
42.4
55.6
55.6
56.5
37.9
50.8
33.5
70.0
68.6
Pretminod FT SOTA
54.4
[34]
[190]
10.3
[124]
143.
47.9
128)
[153)
76.3
57.2
165)
00.4
11898
26.3
(91)
(135)
55.4
(132)
135.7
(128)
36.3
75.
179
(137)
54.7
[129)
25.2
751
[12]
(30
(108)
(444K)
(509K)
(27K)
(5008)
(303))
(308)
(1308)
(68)
(10K)
(468)
(1233)
(20K)
(383)
(98)
Table 1: Comparison to the state of the art. A single Flamingo model reaches the state of the art on a wide array of image (I) and video (V) understanding tasks with few-shot learning, significantly outperforming previous best zero- and few-shot methods with as few as four examples. More importantly, using only 32 examples and without adapting any model weights, Flamingo outperforms the current best methods fine-tuned on thousands of annotated examples on seven tasks. Best few-shot numbers are in bold, best numbers overall are underlined.
Appendix A.2) We perform open-ended evaluations using beam search for decoding, and close- ended evaluations using our model's log-likelihood to score each possible answer. We explore zero-shot generalization by prompting the model with two text-only examples from the task, with no corresponding images. Evaluation hyperparameters and additional details are given in Appendix B.1.5.
3 Experiments
Our goal is to develop models that can rapidly adapt to diverse and challenging tasks. For this, we consider a wide array of 16 popular multimodal image/video and language benchmarks. In order to validate model design decisions during the course of the project, of these benchmarks were used as part of our development (DEV) set: COCO, OKVQA. VQAv2, MSVDQA and VATEX. Performance estimates on the DEV benchmarks may be biased, as a result of model selection. We note that this is also the case for prior work which makes use of similar benchmarks validate and ablate design decisions. To account for this, we report performance on an additional set 11 benchmarks, spanning captioning, video question-answeriin visual dialogue and multi-choic question-answer answering tasks. The evaluation benchmarks are described in Appendix B.1.4. We keep all evaluation hyperparameters fixed across all benchmarks. Depending on the task, we use four few-shot prompt templates we describe in more detail in Appendix B. 1.5. We emphasize that we do not validate any design decisions on these 1l benchmarks and use them solely to estimate unbiased few-shot learning performance of our models.
well some less commonly explored capabilities such as
Concretely, estimating few-shot learning performance of model involves prompting it with set of support samples and evaluating it on a set of query samples. For the DEV benchmarks that are used both validate design decisions and byperparameters, as well as report final performance, therefore use four subsets: validation support, validation query, test support and test query. For other benchmarks, we need only the latter two. We report in Appendix 1.1 how form these subsets.
We report the results of the Flamingo models on few-shot learning in Section 3.1. Section 3.2 gives Flamingo fine-tune mune-t results. An ablation study is given in Section 3.3. Appendix B.2 provides more results including Flamingo's 's performance on the ImageNet and Kinetics 700 classification tasks, and our contrastive model's 's performance. Appendix C includes additional qualitative results.
3.1 Few-shot learning on vision-language tasks
Few-shot results. Results are given in Table 1. Flamingo outperforms by a large margin all previous zero-shot or few-shot methods on the 16 benchmarks considered. This is achieved with as few as four examples per task, demonstrating practical and efficient adaptation of vision models to new tasks. More importantly, Flamingo is often competitive with sfate-of-the methods additionally fine-tuned
7

---

Mothoo
VQAVS
Coco
VATEX
ViZWEz
MSRVITQA
Vibial
YouCook? valid
ManefoMeees
Jesk dev
sest std
lest
12st
lest dev
lest std
1254
valid
less std.
valid
less sod
lest seen
*.22 shots
67.6
115.8
65.1
19.8
31,0
56.8
86.8
56.0
70.0
138
65.4
47.4
1186
57.1
SotA
81.3
81.3
149.6
81.4
57.21
6006
46.8
752
78.4
138.7
54.7
73.7
84.61
[131]
(353)
(119
1153)
1001
1061
1511
1791
(125)
1152
135
[89)
[152]
Table 2:
Comparison to SotA when fine-tuning Flamingo. We fine-tune Flamingo on all nine tasks where Flamingo does not achieve SotA with few-shot learning. Flamingo sets a new SotA on five of them, outperfoming methods (marked with f) that use tricks such as model ensembling or domain-specific metric optimisation (e.g., CIDEr optimisation).
Ablated
Flamingo-33 3B original value
Changed
Param. count
Step
COCO CIDEat
OKVQA toplt
VQA+2 lpplt
MSVDQA loplt
VATEX CIDEx
Overall scvret
setting
valloc
Flomingo-38 model
3.2B
1.74s
86.5
42.1
55.8
36.3
53.4
wio Video-Tase Text pairs
3.28
1.42s
84.2
43.0
53.9
34.5
67.3
Training data
All data
wio Image- Text pains
Image-Teat Text puirs- LAION
3.213
3.28
0.95
1.74h
66.3
79.5
41.4
39.2
53.5
51.6
32.0
33.9
47.6
41.6
66.4
609
wio MFW
3,28
1.024
54.1
36.5
52.7
31.4
23.5
53.4
(11)
Optimisation
Accurslaaion
Round Robin
3,28
1.68%
76.1
39.8
52.1
33.2
40.8
62.9
(i0)
Tanh gaiigg
3.28
1.74s
78.4
40.5
52.9
35.9
47.5
(iv)
archisecture
Cross altention
GATED
VANILLA XATIN GRAFTING
2.4B
3.38
1.16s
1.746
79.2
80.6
41.5
36.1
50.8
53.4
32,9
32.2
50.7
47.8
66.9
63.1
(4)
Choss-atennion frequency
Every
Single in middle
Every 4th
2.88
2.38
1.02s
0.87%
71.5
82.3
42.7
38.1
50.2
55.1
29.1
34.6
50.8
42.3
68.8
59.8
Every 2nd
2,68
83.7
41.0
55.8
34.5
6i Resampler
Perveiver
MIP
Transformer
3.2B
3.28
1.85
1.81s
78.6
83.2
42.2
41.7
54.7
55.6
35.2
31.5
44.7
48.3
66.7
66.6
(vil)
Vision encoder
NFNet-F6
CLIP VIT-L/14
NENe--FO
2.98
3.18
1.45%
1.58s
73.8
76.5
41.6
40.5
53.4
52.8
31.1
33.2
12.9
44,5
64.9
62.7
(vil)
Froezing
(preerained)
(random init)
3,28
3.28
2.42s
2,42s
74.8
81.2
33.7
31.5
45.6
47.4
31.0
26.9
53.9
62.7
57.8
Table 3: Ablation studies. Each row should be compared to the baseline Flamingo run (top row).
Step time measures the time spent to perform gradient updates on all training datasets.
on up to hundreds of thousands of annotated examples. On six tasks, Flamingo even outperforms the fine-tuned SotA despite using a single set of model weights and only 32 task-specific examples. Finally, despite having only used the DEV benchmarks for design decisions, our results generalize well to the other benchmarks, confirming the generality of our approach.
Scaling with respect to parameters and shots. As shown in Figure 2, the larger the model, the better the few-shot performance, similar to GPT-3 [11]. The performance also improves with the number of shots. We further find that the largest model better exploits larger numbers of shots. Interestingly, even though our Flamingo models were trained with sequences limited to only 5 images on M3W, they are still able to benefit from up to 32 images or videos during inference. This demonstrates the flexibility of the Flamingo architecture for processing a variable number of videos or images.
3.2 Fine-tuning Flamingo as a pretrained vision-language model
While not the main focus of our work, we verify that when given more data, Flamingo models can be adapted task by fine-tuning their weights. In Table 2, we explore fine-tuning our largest model, Flamingo, for a given task with no limit on the annotation budget. In short, we do so by fine-tuning the model on a short schedule with a small learning rate by additionally unfreezing the vision backbone to accommodate a higher input resolution (details in Appendix B.2.2). We find that we can improve results over our previously presented in-context few-shot learning results, setting new state of the art on five additional tasks: VQAv2, VATEX, VizWiz, MSRVTTQA, and HatefulMemes.
3.3 Ablation studies
In Table 3, report our ablation results using Flamingo-3B on the validation subsets of the five DEV benchmarks with 4 shots. Note that we use smaller batch sizes and a shorter training schedule compared to the final models. The Overall score is obtained by dividing each benchmark score by its state-of-thh-aan (SotA) performance from Table and averaging the results. More details and results are given in Appendix B.3 and Table 10.
Importance of the training data mixture. As shown in row (i), getting the right training data plays crucial role. In fact, removing the interleaved image-text dataset M3W leads to a decrease of more
8

---

than 17% in performance while removing the conventional paired image-text pairs also decreases performance (by 9.8%). demonstrating the need for different types of datasets. Moreover, removing our paired video-text dataset negatively affects performance on all video tasks. We ablate replacing our image-text pairs (ITP) by the publicly available LAION-400M dataset [96], which leads to a
slight degradation in performance. We show in row (ii) the importance of our gradient accumulation strategy compared to using round-robin updates [17].
Visual conditioning of the frozen LM. We ablate the use of the 0-initialized tanh gating when merging the cross-attention output to the frozen LM output in row (iii). Without it, we see drop of 4.2% in our overall score. Moreover, we have noticed that disabling the 0-initialized tanh gating leads to training instabilities. Next, we ablate different conditioning architectures in row (iv). VANILLA XATTN, refers to the vanilla cross-attention from the original Transformer decoder [1 [115]. In the GRAFTING approach from [68], the frozen LM is used as with no additional layers inserted, and a stack of interleaved self-attention and cross-attention layers that take the frozen LM output are learnt from scratch. Overall, we show that our GATED LATTN-DENSE conditioning approach works best.
Compute/Memory vs. performance trade-off'. In row (V), we ablate the frequency at which we add new GATED XATTN-DENSE blocks. Although adding them at every layer is better, it significantly increases the number of trainable parameters and time complexity of the model. Notably, inserting them every fourth block accelerates training by 66% while only decreasing the overall score by 1.9%. In light of this trade-off, we maximize the number of added layers under hardware constraints and add a GATED KATTN-DENSE every fourth layer for Flamingo-99 and every seventh for Flamingo-80B We further compare in row (vi) the Perceiver Resampler to a ML. and vanilla Transformer given a parameter budget. Both underperform the Perceiver Resampler while also being slower.
Vision encoder. In row (vii), we compare our NFNet-F6 vision encoder pretrained with contrastive learning (details in Appendix B. 1.3) the publicly available CLIP ViT-L/14 [85] model trained at 224 resolution. Our NFNet-F6 has a +5.8% advantage over the CLIP ViT-L/14 and +8.0% over a smaller NFNet-FO encoder, which highlights the importance of using a strong vision backbone.
Freezing LM components prevents catastrophic forgetting. We verify the importance of freezing the LM layers at training in row (viii). If trained from scratch, we observe large performance de- crease of - 12.9%. Interestingly, fine-tuning our pretrained L.M also leads to a drop in performance of -8.0%. This indicates instance of "catastrophic forgetting" [71], in which the model progressively forgets its pretraining while training on a new objective. In our setting, freezing the language model is better alternative training with the pre-training dataset (MassiveTexxe Textt) in the mixture.
4 Related work
Language modelling and few-shot adaptation. Language modelling has recently made substantial
progress following the introduction of Transformers [115]. The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108]. In this work, we build on the 70B Chinchilla language model [42] as the base LM for Flamingo. Numerous works have explored techniques to adapt language models to novel tasks using a few examples. These include adding small adapter modules [43]. fine-tuning a small part of the LM [141], showing in-context examples in the prompt [11], or optimizing the
prompt [56, 60] through gradient descent. In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric
learning [24, 103, 1 12, 117] or meta-learning [6, 7, 27, 31, 91, 155).
When language meets vision. These LM breakthroughs have been influential for vision-language modelling. In particular, BERT [23] inspired a large body of lisionnlanguage work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 1 118, 121, 142, 143, 151]. We differ from these approaches as Flamingo models do not require fine-tuning on new tasks. Another family of vision-language models is based on contrastive learning [2, 5, 49, 50, 57, 74, 82, 85, 138, 140, 146]. Flamingo differs from contrastive models as it can generate text, although we build and rely upon them for our vision encoder. Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 1 116]. Concurrent works [17, 58, 119, 124, 154] also propose to formulate numerous vision tasks as text generation problems. Building on top of powerful pretrained language models has been explored in several recent works. One recent line of work [26, 68, 78, 1 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [7 1]. We follow this idea by freezing the

---

Chinchilla LM layers [42] and adding learnable layers within the frozen LM. We differ from prior work by introducing the first LM that can ingest arbitrarily interleaved images, videos, and text.
Web-scale vision and language training datasets. Manually annotated vision and language datasets costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139]. To alleviate this lack of data, numerous works [14, 50, 98, 110] automatically scrape readily available paired vision-text data. In addition such paired data, we show the importance of also training on entire multimodal webpages containing interleaved images and text as a single sequence. Concurrent work CM3 [1] proposes to generate HTML markup from pages, while we simplify the text prediction task by only generating plain text. We emphasize few-shot learning and vision tasks while CM3 [1] primarily evaluates on language-only benchmarks in zero-shot or fine-tuned setup.
5 Discussion
Limitations. First, our models build on pretrained LMs, and as side effect, directly inherit their weaknesses. For example. LM priors are generally helpful, but may play role in occasional hallucinations and ungrounded guesses. Furthermore, LMs generalise poorly to sequences longer than the training ones. They also suffer from poor sample efficiency during training. Addressing these issues can accelerate progress in the field and enhance the abilities of VLMs like Flamingo.
Second, the classification performance of Flamingo lags behind that of state-of-the-ar ff-therrtt contrastive models [82, 85). These models directly optimize for text-image retrieval, of which classification is a special case. In contrast, our models handle a wider range of tasks, such as open-ended ones. A unified approach to achieve the best of both worlds is important research direction.
Third, in-context learning has significant advantages over mundient-basees few-shot learning methods, but also suffers from drawbacks depending the characteristics of the application at hand. We demonstrate the effectiveness of in-context learning when access is limited to only a few dozen examples. In-context learning also enables simple deployment, requiring only inference, generally with no hyperparameter tuning needed. However, in-context learning is known to be highly sensitive to various aspects of the demonstrations [80, 148], and its inference compute cost and absolute performance scale poorly with the number of shots beyond this low-data regime. There may be opportunities to combine few-shot learning methods leverage their complementary benefits. We discuss the limitations of our work in more depth in Appendix D.I.
Societal impacts. In terms of socictal impacts, Flamingo offers number of benefits while carrying some risks. Its ability to rapidly adapt to broad range of tasks have the potential to enable non-exper users to obtain good performance in data-starved regimes, lowering the barriers to both beneficial and malicious applications. Flamingo is exposed to the same risks as large language models, such as outputting offensive language, propagating social biases and stereotypes, private information [42, 126). Its ability to additionally handle visual inputs poses specific risks such as gender and racial biases relating to the contents of the input images, similar to number of visual recognition systems [12, 21, 37, 97, 147]. We refer the reader to Appendix D.2 for a more extensive discussion of the societal impacts of our work, both positive and negative; well as mitigation strategies and early investigations of risks relating to racial or gender bias and toxic outputs. Finally we note that, following prior work focusing on language models [72, 81, 111], the few-shot capabilities of Flamingo could be useful for mitigating such risks.
well as leaking
Conclusion. We proposed Flamingo, general-purpose purpose family of models that can be applied to image and video tasks with minimal task-specific training data. We also qualitatively explored interactive abilities of Flamingo such as "chatting" with the model, demonstrating flexibility beyond traditional vision benchmarks. Our results suggest that connecting pre-trained large language models with powerful visual models is an important step towards general-purpose visual understanding.
Acknowledgments and Disclosure of Funding. This research was funded by DeepMind. We would like to thank many colleagues for useful discussions, suggestions, feedback, and advice, including: Samuel Albanie, Relja Arandjelović, Kareem Ayoub, Lorrayne Bennett, Adria Recasens Continente, Tom Eccles, Nando de Freitas, Sander Dieleman, Conor Durkan, Aleksa Gordié, Raia Hadsell, Will Hawkins, Lisa Anne Hendricks, Felix Hill, Jordan Hoffmann, Geoffrey Irving, Drew Jaegle, Koray Kavukeuoglu, Agustin Dal Lago, Mateusz Malinowski, Soña Mokrá, Gaby Pearl, Toby Pohlen, Jack Rae, Laurent Sifre, Francis Song, Maria Tsimpoukelli, Gregory Wayne, and Boxi Wu.
10

---

References
[1] Armen Aghajanyan. Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A causal masked multimodal model of the internet. arXiv:2201.07520, 2022.
[2] Jean-Baptiste Alayrac, Adria Recasens, Rosalia Schneider, Relja Arandjelović, Jason Rama- puram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self- supervised multimodal versatile networks. Conference Neural Information Processing Systems, 2020.
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In International Conference on Computer Vision, 2015.
[4] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley, ReZero is all you need: Fast convergence at large depth. In Uncertainty in Artificial Intelligence, 2021.
[5] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen time: joint video and image encoder for end-to-end retrieval. In International Conference on Computer Vision, 2021.
[6] Luca Bertinetto, João F. Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed-forward one-shot learners. Conference Neural Information Processing Systems, 2016.
[7] Luca Bertinetto, Joao F. Henriques, Philip S. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. arXiv: 1805.0813, 2018.
[8] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations Python+NumPy programs, 2018. URL http: / ht///gittt github. wwtchuu..oo//geee com/google, jaxx.
[9] John S. Bridle. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputiing, 1990.
[10] Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. peg-operformance large-scale image recognition without normalization, arXiv:2102.06171, 2021.
[11] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Conference on Neural Information Processing Systems, 2020.
[12] Joy Buolamwini and Timnit Gebru. Gender shades: commercial gender classification. In ACM Conference on Fairness, Accountabilityy and Transparency, 2018.
Intersectional accuracy disparities in
[13] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020.
[14] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In IEEE Computer Vision and Pattern Recognition, 2021.
[15] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv: 504.0035 2015.

---

[16] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng. and Jingjing Liu. UNITER: Universal image-text representation learning. In European Conference on Computer Vision 2020.
[17] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and -nd--and--ddn -anguage tasks via text generation. In International Conference on Machine Learning. 2021.
[18] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Hellst-el,,,,rn, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. wwXiv:2204.023111 2022.
[19] Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation on clip via vision-language knowledge distillation. In ACL Findings, 2022.
[20] Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In IEEE Computer Vision and Pattern Recognition, 2017.
[21] Terrance De Vries, Ishan Misra, Changhan Wang. and Laurens Van der Maaten. Does object recognition work for everyone? In IEEE Computer Vision and Pattern Recognition, 2019.
[22] Karan Desai and Justin Johnson. VirTex: Learning visual representations from textual annota- tions. IEEE Computer Vision and Pattern Recognition, 2021.
[23] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805, 2018.
[24] Carl Doersch, Ankush Gupta, and Andrew Zisserman. CrossTransforment spatially-aware few-shot transfer. Conference on Neural Information Processing Systems, 2020.
[25) Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In IEEE Computer Vision and Pattern Recognition, 2015.
[26] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. MAGMA- augmentation of generative models through adapter-based finetuning. arXiv:21 2021.
[27] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning. 2017.
[28] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. VIOLET: End-to-end video-language transformers with masked visual-token modeling.
arXiv:2111.12681, 2021.
[29] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and lnn-uuge representation learning. In Conference on Neural Information Processing Systems, 2020.
[30] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 1021.
12

---

[31] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E. Turner Meta-learning probabilistic inference for prediction. arXiv: 09921, 2018.
[32] Alex Graves. Generating sequences with recurrent neural networks. arXiv: 130.:0300000 2013,
[33] Thomas L. Griffiths, Frederick Callaway, Michael B. Chang, Erin Grant, Paul M. Krueger, and Falk Lieder. Doing more with less: meta-reasoning and meta-learning in humans and machines. Current Opinion in Behavioral Sciences, 2019.
[34] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao. KAT: A knowledge augmented transformer for saiio-aad-languuaee vxXiv2212.08811 2021.
[35] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. VizWiz grand challenge: Answering visual questions from blind people. In JEEE Computer Vision and Pattern Recognition, 2018.
[36] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. arXiv:2203. 16634, 2022.
[37] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In Eumppan Conference on Computer Vision, 2018.
[38] Lisa Anne Hendricks, John Mellor, Rosalia Schneider, Jean-Bapting Alayrac, and Aida Nematzadch. Decoupling the role of data, attention, and losses in multimodal transformers. Annual Meeting of the Association for Computational Linguistics, 2021.
[39] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv: 1606.08415 2016.
[40] Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020. URL http://:://iitht wwt/ coom/deepmind///e /mmddamuu......aa
[41] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.
[42] Jordan Hoffmaann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Eric Noland Tom Hennigan, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv:203.. /5556, 2022.
[43] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffiien transfer learning for NLP. In International Conference on Machine Learning, 2019.
[44] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classifica- 2018
tion.
[45] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up lision-language pre-training for image captioning, arXiv:2111.12233 2021.
[46] Lun Huang, Wenmin Wang, Jie Chen, and Xiao Xiao-Yong Wei. Attention on attention for image captioning. In International Conference on Computer Vision, 2019.
[47] Md Amirul Islam, Matthew Kowal, Sen Jia, Konstantinos G. Derpanis, and Neil D. B. Bruce. Global pooling, more than meets the eye: Position information is encoded channel-wise in CNNs. International Conference on Computer Vision, 2021.
[48] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. International Conference Machine Learning, 2021.
13

---

[49] Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge. MURAL: multimodal, multitask retrieval across languages. urXiv:2109.05 2021.
[50] Chao Jía, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. 2.X.x:2102.05500 2021.
[51] Alex Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. All in one: Exploring unified video-language pre-training. arXiv:2203.00 22ti:2200307303 000,, 2022.
[52] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv:1602.02410 2016.
[53] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.0833 2020.
[54] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The Hateful Memes Challenge: Detecting hate speech in multimodal memes. Conference on Neural Information Processing Systems, 2020.
[55] Hugo Larochelle. Few-shot classification by recycling deep learning. Invited Talk at the S2D-OLAD Workshop, ICLR 2021, 2021. URL https:/ fewshot
com/38955350/
[56] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv:2104.08691, 2021.
[57] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Conference on Neural Information Processing Systems, 2021.
[58] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv:2201.12086, 2022.
[59] Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. HERO: Hierarchical encoder for video+language omni-representation pre-training. arXiv:2005.00200, 2020.
[60] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. 10Xi::2001.000
2021.
[61] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vasion-language tasks. In European Conference on Computer Vision, 2020.
[62] Phillip Lippe, Nithin Holla, Shantanu Chandra, Santhosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, and Helen Yannakoudakis. A multimodal framework for the detection of hateful memes. arXiv:2012.. 128771,,, 2020.
[63] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? arXiv:2101.0688 2021.
[64] Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Optimization of image description metrics using policy gradient methods. In International Conference on Computer Vision, 2017.
[65] Yu Liu, Lianghua Huang, Liuyihang Song, Bin Wang, Yingya Zhang, and Pan Pan. Enhancing textual cues in
transformers for VQA. VizWiz Challenge 2021, 2021.
14

---

[66] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task task-agnostic vi- siolinguistic representations for vision-and ann-lang tasks. Conference on Neural Information Processing Systems, 2019.
[67] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang. Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. UniVL: A unified video and language pre- pre-training model for multi- modal understanding and generation. arXiv:2002.06335. 2020.
[68] Ziyang Luo, Yadong Xi, Rongsheng Zhang, and Jing Ma. VC-GPT: Visual conditioned GPT
for end-to-end generative usiionandd-anguage pre-training. arXiv:2201.12723 2022.
[69] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. IEEE Computer Vision and Pattern Recognition, 2019.
[70] Ellen M. Markman. Categorizattin and naming in children: Problems of induction. MIT Press, 1989.
[71] Michael McCloskey and Neil J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. The Psychology of Learning and Motivation, 1989.
[72] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Hyapbell-Gillinghmm, Geoffrey Irving. and Nat McAleese. Teaching language models to support answers with verified quotes. arXiv:2203. 11147, 2022.
[73] Antoine Miech, Sann-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. RareAct: A video dataset of unusual interactions. arxiv:2008.00018 2020.
[74] Antoine Miech, -ean-Bap Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In IEEE Computer Vision and Pattern Recognition, 2020.
[75] Tomas Mikolov, Martin Karafíát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. Interspeech, 2010.
[76] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv:2202.12837, 2022.
[77] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In ACM Conference on Fairness, Accountability, and Transparency, 2019.
[78] Ron Mokady, Amir Hertz, and Amit H. Bermano. ClipCap: CLIP prefix for image captioning. arXiv:2111.0977 0.99344, 2021.
[79] Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das. Large-scale pretraining for visual dialog: A simple state-of-the-art baseline. In European Conference on Computer Vision, 2020.
[80] Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Conference on Neural Information Processing Systems, 2021.
[81] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv:2202.00 03286, 2022.
[82] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Yu, Minh-Thang Lu- ong. Mingxing Tan, and Quoc V. Le. Combined scaling for zero-shot transfer learning. arXiv:2111.10050 2021.
[83] Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022.
15

---

[84] Yixuan Qiao, Hao Chen, Jun Wang, Yihao Chen, Xianbin Ye, Ziliang Li, Xianbiao Qi, Peng Gao, and Guotong Xie. Winner team Mia TextVQA Challenge 2021: Vision-and lang-auge representation learning with pre-trained sequence-to-sequence model. arXiv:2 15332, 2021.
[85] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. arXiv:2103. 000000000000.....000000000000000 2021.
[86] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driess- che, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark. Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals. Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving, Scal- ing language models: Methods, analysis & insights from training Gopher. arXiv:2112.11446, 2021.
[87] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. 1rXiv:1910.100
2019.
[88] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory optimizations toward training trillion parameter models. In International Conference for High Performance Computing, Networking, Storage and Analysis, 2020.
[89] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv:2204.06112 2022.
[90] Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In IEEE Computer Vision and Pattern Recognition, 2017.
[91] James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Conference on Neural Information Processing Systems, 2019.
[92] Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 2021.
[93] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. 1804 09300, 2018.
[94] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
[95] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan
16

---

Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. Multitask Prompted Training Enables Zero-Shot Task Generalization. In International Conference on Learning Representations, 2022.
[96] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv:2111.0214,, 2021.
[97] Carsten Schwemmer, Carly Knight, Emily D. Bello-Pardo, Stan Oklobdzija, Martijn Schoonvelde, and Jeffrey W. Lockhart. Diagnosing gender bias in image recognition systems. Socius, 2020.
[98] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual Captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics, 2018.
[99] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LMM Training multi-bille billi-billion parameter language models using model parallelism. wwXiv:2104.0869, 2019.
[100] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In JEEE Computer Vision and Pattern Recognition, 2019.
[101] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment 2021.
model. 1
[102] Lucas Smaira, João Carreira, Eric Noland, Ellen Clancy, Amy Wu, and Andrew Zisserman. A short note on the Kinetics-700-2001 human action dataset. arXiv:2010. 10864, 2020.
[103] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Conference on Neural Information Processing Systems, 2017.
[104] David R So, Wojeiech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V. Le. Primer: Searching for efficient transformers for language modeling. arXiv:2109.0866 2021.
[105] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. 1906. 02243, 2019.
[106] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pre-training of generic visual-linguistic representations. arXiv: 1908.085330 2019.
[107] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT: A joint model for video and language representation learning. In International Conference on Computer Vision, 2019.
[108] Ilya Sutskever, James Martens, and Geoffrey E. Hinton. Generating text with recurrent neural networks. In International Conference on Machine Learning, 2011.
[109] Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformer. In Conference on Empirical Methods in Natural Language Processing, 2019.
[110] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni. Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The new data in multimedia research. Communications of the ACM, 2016.
[111] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kuishreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts,
17

---

Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang. Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstem, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. LaMDA: Language models for dialog applications. arXiv:2201.08230 2022.
[112] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola, Rethinking few-shot image classification: good embedding is all you need? In European Conference on Computer Vision, 2020.
[113] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. Conference on Neural Information Processing Systems, 2019.
[114] Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Conference on Neural Information Processing Systems, 2021.
[115] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Conference on Neural Information Processing Systems, 2017.
[116] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In International Conference on Computer Vision, 2015.
[117] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. Conference on Neural Information Processing Systems, 2016.
[118] Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai. Zicheng Liu, Yumao Lu, and Lijuan Wang. UFO: A unified transformer for vision-language representation learning. arXiv:2111.10003 2021.
[119] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. arXiv:2202.03052, 2022.
[120] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective work best for zero-shot generalization? arXiv:2204.05883. 2022.
[121] Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. VLMo: Unified dision-language pre-training with cffture-of-modaliiyy experts. arXiv:2111.02358, 2021.
[122] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. VATEX: A large-scale, high-quality multilingual dataset for aedeo-and-languree research. In International Conference on Computer Vision, 2019.
[123] Yue Wang. Shafiq Joty, Michael Lyu, Irwin King, Caiming Xiong, and Steven Hoi. VD-BERT: A unified vision and dialog transformer with BERT. In Conference on Empirical Methods in Natural Language Processing, 2020.
[124] Zinui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. VLM Simple visual language model pretraining with weak supervision. arXiv:2 10904, 2021.
[125] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Gun, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le.. Finetuned language models are zero-shot learners. 100iv:2109001650 1021.
18

---

[126] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang. Myra Cheng. Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and lason Gabriel. Ethical and social risks of harm from language models. 12.ix2212.044359 2021.
[127] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari Morcos, Hongseok Namkoong. Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv:2203.0m 005422 2022.
[128] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua Tenenbaum, and Chuang Gan. STAR: Benchmark for Situated Reasoning in Real-World Videos. In Conference on Neural Information Processing Systems, 2021.
[129] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-QA: Next phase of question- answering to explaining temporal actions. In IEEE Computer Vision and Pattern Recognition, 2021.
[130] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017.
[131] Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. Zeroprompt: Scaling prompt-based pretraining to 1,000 tasks improves zero-shot generalization. arXiv:2201. 06910, 2022.
[132] Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feicht- enhofer, Florian Metze, and Luke Zeulemoyer. VLM: Task-agnostic video-language model pre-training for video understanding. arXiv:2105.0990 09996, 2021.
[133] Ming Yan, Haiyang Xu, Chenliang Li, Junfeng Tian, Bin Bi, Wei Wang, Weihua Chen, Xianzhe Xu, Fan Wang, Zheng Cao, Zhicheng Zhang, Qiyu Zhang, Ji Zhang, Songfang Huang, Fei Huang, Luo Si, and Rong Jin. Achieving human parity on visual question answering. arXiv:2111.0888 2021.
[134] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. arXi::2201.0428.. 2022.
[135] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In International Conference Computer Vision, 2021.
[136] Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of GPT-3 for few-shot knowledge-baadd based VQA. In National Conference on Artificial Intelligence (AAAI), 2021.
[137] Zhengyuan Yang, Yijuan Lu, Jianfeng Wang. Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. TAP: Text-aware pre-training for texI-VQA and text-caption. In IEEE Computer Vision and Pattern Recognition, 2021.
[138] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe NIu, Hang Xu, Xiaodan I Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre-training, arXiv:2111.0778 2021.
[139] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Annual Meeting of the Association for Computational Linguistics, 2014.
[140] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng. Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. arXiv:2111.1144 2021.
19

---

[141] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. BitFit: Simple carameter-efficien fine-tuning for transformer-basee masked language-modee models. arXiv:2106.10199 2021.
[142] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. MERLOT: Multimodal neural script knowledge models. Conference Neural Information Processing Systems, 2021.
[143] Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. MERLOT reserve: Neural script knowledge through vision and language and sound. In IEEE Computer Vision and Pattern Recognition, 2022.
[144] Andy Zeng. Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv:2204.000 005998 2022.
[145] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision trans- formers. arXiv:2106.04556 2021.
[146] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. LiT: Zero-shot transfer with locked-image text tuning. arXiv:2111.07991, 2021.
[147] Dora Zhao, Angelina Wang, and Olga Russakovsky. Understanding and evaluating racial biases in image captioning. IEEE Computer Vision and Pattern Recognition, 2021.
[148] Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, 2021.
[149] Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In National Conference on Artificial Intelligence (AAAI), 2018.
[150] Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and VQA. In National Conference on Artificial Intelligence (AAAI), 2020.
[151] Linchao Zhu and Yi Yang. ActBERT: Learning global-local video-text representations. In IEEE Computer Vision and Pattern Recognition, 2020.
[152] Ron Zhu. Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution. arXiv:20/2.08299, 2020.
[153] Xinxin Zhu, Longteng Guo, Peng Yao, Shichen Lu, Wei Liu, and Jing Liu. Vatex video cap- tioning challenge 2020: Multi-view features and hybrid reward strategies for video captioning. arXiv:1910.1 1102, 2019.
[154] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Xiaogang Wang, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-Perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. arXiv:2112.01522, 2021.
[155] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, 2019.
20

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5 for brief discussion and Appendix D.2 for the full discussion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] The code and the data are proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 3 and Appendix B.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No] We do not observe large enough variance in our training runs to justify the computation cost incurred by multiple training runs. For the largest models, it is not feasible within our compute budget.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Details can be found in Appendix B.1.2. In short, our largest run was trained on 1536 TPU chips for 15 days.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] We properly cited the prior methods on which our work based, as well as prior datasets when appropriate
(e.g., ALIGN).
(b) Did you mention the license of the assets? [N/A] The assets used are previous work for which we cited papers. We do mention the license of all visual assets we use for the figures of the paper in Appendix G.
URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] Our data was automatically scraped from million of webpages. See Datasheets [30] in Appendix F.
(c) Did you include any new assets either in the supplemental material
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [Yes] See Datasheets [30] in Appendix F.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
21


Reviews:
[{"review_id": "vQ0JJxwi7g", "reviewer": "Reviewer_T9XK", "summary": "The Flamingo model is a general few-shot solution for multimodal tasks. As input, the model can accept a multimodal prompt. The prompt contains a few examples of the desired task, with text, images, or videos as tokens. The model determines the task dynamics from the prompt and infers the desired task output. This model relies on three key ideas: (i) A frozen Pretrained Language Model (PLM) for training extensive knowledge, (ii) Reducing visual input to a fixed number of visual tokens with Perceiver, and (iii) Training on a web-based noisy image-text dataset (ALIGN) and a cleaner dataset (LTIP).", "questions": "Given the limited access to large-scale training data (e.g., ALIGN), how do the authors see future research on this topic?  Are the authors able to provide results from public data, such as LAION?\n", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is generally of high quality. Various tasks are studied in depth. Compared to previous few-shot methods, the performance is significantly better. Additionally, the work includes an extensive ablation study of different design choices, including pertaining, training data mixtures, mixing different tasks, and architecture choices. Moreover, the method is efficient and uses frozen PLM and vision encoders. The only trainable parts are intermediate modules (e.g., Perceiver and gated XATTN-Dense attention). \n\nNotably, I find few-shot supervision, and steering of large-scale PLM to solve different tasks to be human-like and an important research direction. \n\n\nWeaknesses:\n\n\n\n(W1) Data access is a major concern. As far as I know, the ALIGN dataset is not publicly available, so future research in this area will be limited. Data is a key factor for training, as indicated in the paper. \n\n(W2) A similar work, \"Multimodal Few-Shot Learning with Frozen Language Models,\" has many related ideas (e.g., frozen PLM, few-shot multimodal prompt solution). In addition, the study introduces a relevant benchmark, Open-Ended miniImageNet 2-Way Tasks.\nWhile the paper does cite this work, the paper didn't elaborate enough on the fundamental differences between the studies. Is it only using new inputs such as videos, as mentioned on L275?\n\n(W3) This is not a weakness, but a matter of taste. I found it a bit exhausting to go back and forth between the main paper and the appendix, especially with the appendix's size.  A concise but formal description of the method in the main paper (e.g., GATED XATTN-DENSE) would be greatly appreciated. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nJKbl-6iZOA", "reviewer": "Reviewer_KruY", "summary": "This paper introduces a new pre-trained vision-and-language model (VLM), Flamingo, built upon pre-trained vision and language models. The Flamingo model can take arbitrarily interleaved visual data and text as input and do open-ended text generation. \nThe Flamingo model shows promising results on multiple multimodal tasks in few-shot learning and can even outperform some SOTA models on some tasks. ", "questions": "- Do the authors plan to release the model and/or the dataset used to train the model? If so, when?\n- Do the authors plan to release an API like what OpenAI does to allow people to play with your model? \n- How can one trust the results in the paper (not by faith) if no one can reproduce the results?", "limitations": "- Missing details of the dataset introduced in the paper. No plan for the data and model release. \n- Inadequate reproducibility.\n\nSee the weaknesses mentioned above for details. ", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n-\n- This paper introduces a new pre-trained VLM built upon existing powerful pre-trained vision models and language models, and archives promising few-shot learning results on a wide array of vision-and-language tasks. \n- Useful tricks are proposed for bridging pre-trained vision models and language models, including vision feature resampling, inserting cross-modal attention into language models, etc.\n- A new dataset is also mentioned in the paper to pre-train the Flamingo model. \n- The paper is well written in general and easy to understand. \n\nWeaknesses: \n-\n- Taking arbitrarily interleaved visual data and text as input and generating text isn't new for VLM models; the CM3 [1] model does it already and even can do image generation in addition to text generation. Since CM3 is released months ago before the NeurIPS submission, the discussion about it in the paper isn't fair (\"CM3 follows the paper's similar approach\"). The existence of CM3 does not invalidate some of the main contributions of Flamingo, but it has to be fairly discussed, especially considering the similarities those two models have, e.g., the way to interleave and encode visual and text data in a sequence.\n- However, lots of implementation and dataset details are missing (especially the dataset), so reproducing the results in the paper by a third party is nearly impossible. \n- No data and model are attached, and no plan is mentioned in the paper. If the authors are not going to release the model and the data, I am not sure what the point is as no one else can reproduce the results (you have to trust the results reported in the paper blindly). Most researchers do not have the resources to re-train the model, and even with sufficient resources, a third party cannot reproduce the results without access to the data used in the paper.\nNeurIPS advocates open source research and cares so much for reproducibility, so non-reproducible research is not acceptable. \nIn that case, this paper would be mainly a PR and flag-planting paper for a big institute/company, and does minimal good (and arguably hurt) the research community. I think such kinds of research have to convince people first about the validity and reproducibility of their results before being accepted to top conferences.\n- Figure 1 is too good to be true. I know those results are most likely to be cherry-picked and supposed to be good. But without a detailed and fair analysis, they could be very misleading. \n\n[1] CM3: A Causal Masked Multimodal Model of the Internet, Aghajanyan et al., 2022. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inadequate Data and Algorithm Evaluation", "Responsible Research Practice (e.g., IRB, documentation, research ethics)"]}, "extra_text": {}}, {"review_id": "i5XJlpG9KMF", "reviewer": "Reviewer_mm3o", "summary": "This work introduced a new vision-language pretraining model, Flamingo, that aims at few-shot generalization on interleaved visual and textual data. The main architecture is based on Perceiver Resampler to connect a frozen language model and a visual encoder. In order to generate text conditioned on visual representations, the authors design a gated cross-attention module that takes in visual outputs within the pre-trained LM. The model is trained via multi-task joint training. A large-scale interleaved text and image dataset, M3W, is collected to enhance the few-shot capabilities. Experiments show that the proposed model outperforms prior state-of-the-art models on 16 tasks and surpasses the fine-tuned version on 6 tasks.", "questions": "- The model can only be used in uni-modal/multi-modal to text tasks, will and how the model can be generalized to a unified VL architecture, e.g. text to image, image to image?\n- Can prompt engineering techniques that are used for GPT-3 (or similar models) be applied to Flamingo?", "limitations": "The authors provide sufficient analysis of the potential limitations of the proposed method, including massive architecture seeking, and legacies of large language models.\nThe requirements of huge amounts of TPUv4 hardware make the model hard to be reproduced by acamadic researchers.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The demonstrated few-shot capabilities on vision-language tasks are impressive and can refresh current VL performance on various multimodal benchmarks.\n- The proposed architecture and gated cross-attention injected language models are technically sound and can be borrowed by future research on vision-language modeling.\n- Ablation studies on the proposed architecture demonstrated the effectiveness of each module design, which also provides further guidance on future VL architecture design.\n- The interleaved image-text dataset M3W is beneficial to VL community for future research on zero-shot learning.\n\nWeaknesses:\n- Similar to GPT-3 and other generative pretraining models, Flamingo requires a huge amount of model parameters (the smallest one is 1.4B and the performance drop is significant). How to distill the model to a smaller size may be worth discussing.\n- Most dataset and tasks are perception level understanding, such as VQA. How is the reasoning capability of the model, for example on Visual commonsense Reasoning?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XS40g9duY2r", "reviewer": "Reviewer_ZdRp", "summary": "This paper introduces an interesting way of combining large, pre-trained vision and language models for solving a wide range of multi-modal tasks. The proposed model, Flamingo, is able to ingest a sequence of text tokens interleaved with images and/or videos, and produces text as output. The authors further demonstrate the efficiency of this model on a wide range of visual-language benchmarks such as few-shot image captioning, reasoning and question-answering tasks where it outperforms many SOTA models. ", "questions": "- The few-shot VQA performance of Flamingo is impressive. However, I have a feeling that the model needs to be guided with text prompts to answer certain aspects related to an image. Is it possible for the model to answer a certain question without building the context? For e.g. In Fig.1 re: Flamingo image, what would be model response if you directly ask $\\textit{what is the difference between these three images?}$ without the initial leading questions. Similarly, in Fig. 11 re: street sign in NYC, what is the response if you directly ask $\\textit{Which park is nearby?}$\n\n- What is the model performance if you don't use pre-trained vision and language models i.e. training them from scratch with Perceiver Resampler etc.? Does the proposed framework not help in extracting useful low-level features from images and text?\n\n- Is it possible to visualise how textual embeddings map (or attend) to visual embeddings? I reckon, given the task (query prompt) this attention will change for the same image.\n\n- How does the pre-trained visual or language model affects (or limits) the performance of Flamingo? You have mentioned this in the limitations section but have you tried ablating those components? or do you have a failure example?\n\n- I am surprised that spatial grid positional encodings did not help while flattening 2D spatial features to a 1D sequence. In general, this should result in the loss of valuable spatial information. Can authors provide some insights on this?\n\n- What's the reason behind choosing NFNet for visual encoder? Did you try transformers (e.g. ViT) as a vision encoder where you can directly utilise tokens as inputs to Perceiver Resampler?\n\n- What does datasets $D_{m}$ refer to in Eq. 2? I thought the model is trained on only M3W dataset, constituting data from multiple documents. After some sifting through Appendix A, I found the relevant details in Sec. A.3.3. I think it would be a good idea to mention it in the Sec. 2.4 of main paper before introducing the equation.\n\n- For training, do you need the data to be arranged in certain task-specific order for it to generalize to the tasks in few-shot manner during evaluation?\n\n- Out of curiosity, what’s the idea behind naming this model Flamingo? ", "limitations": "Authors have adequately highlighted the limitations and societal impacts of this work. ", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The architecture for combining pre-trained visual and language models, while preserving their knowledge, is novel and would be beneficial for the community in adapting large pre-trained models effectively. \n- The paper empirically justifies almost all the aspects of design choices via adequate evaluations.\n- The proposed method demonstrates impressive performance on few-shot image captioning, reasoning and question-answering tasks.\n- The paper is very detailed, well-written and easy to follow..\n\n**Weaknesses:**\nI do not have major concerns with the paper but a suggestion to improve the readability of the paper. I have put my questions in the `questions` section below.\n- It’s hard to navigate the appendix of the paper. Could you please mention the exact sections of appendix for specific details while referencing it in the main paper? For e.g. see Appendix A.1.3 instead of just Appendix A.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VqP1qTLpaWt", "reviewer": "Reviewer_agGw", "summary": "This paper proposes to align a pretrained image encoder and a large pretrained generative text encoder to the same multimodal space, by training an additional perceiver resampler and gated cross-attention layers. The method is simple and effective. Experiments on various vision-language multimodal tasks show that the proposed model called Flamingo outperforms previous methods under the zero-shot and few-shot settings, and also achieves SOTA performances on several tasks under the finetuned setting.\n", "questions": "Will the trained models and pretraining datasets be released?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper is quite well written and easy to follow.\n2. The proposed method is simple and effective. \n3. The proposed vision-language pretraining dataset with interleaved images and texts is novel, and would be valuable to the community.\n4. Extensive experiments on various tasks and model sizes show the efficacy of the proposed model.\n\nWeakness:\n1. One of my main concerns is about the novelty. Aligning a pretrained image encoder and a pretrained text encoder to the same multimodal space has already been studied in previous works, though not at a scale as large as Flamingo. In particular, a very similar idea is proposed in [1], which aligns the CLIP image encoder and a pretrained BART model to the same model space via distillation methods with light additional projection heads. Similar to Flamingo, [1] also exhibits remarkable zero-shot performance on image caption and open-end VQA tasks. It would be better if this submission can clarify more on the connection and difference with [1] and other similar works.\n2. The good performance of the proposed models depends heavily on the large model size and large pretraining dataset size. This makes the pretraining and inference of these models very expensive.\n3. Some details about the datasets and experimental settings on downstream tasks are not very clear. E.g, \n(1) it is not clear what are the sources for collecting the interleaved image-text dataset.\n(2) Previous dual-stream vision-language models like CLIP are efficient for large-scale image-text retrieval because features can be precomputed. However, the image and texts are intertwined in the proposed Flamingo, and it is not clear how the proposed model can be used for retrieval tasks.\n\n[1] Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation, ACL Findings 2022.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept