SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
Authors: Zirui Wang
Jiahui Yu
Adams Wei Yu
Zihang Dai
Yulia Tsvetkov
Yuan Cao
Venue: ICLR 2022 Poster (2022)
Keywords: Vision-Language Pretraining
Multimodal Language Model
Weak Supervision

Abstract: With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks. However, the requirement for expensive annotations including clean image captions and regional labels limits the scalability of existing approaches, and complicates the pretraining procedure with the introduction of multiple dataset-specific objectives. In this work, we relax these constraints and present a minimalist pretraining framework, named Simple Visual Language Model (SimVLM). Unlike prior work, SimVLM reduces the training complexity by exploiting large-scale weak supervision, and is trained end-to-end with a single prefix language modeling objective. Without utilizing extra data or task-specific customization, the resulting model significantly outperforms previous pretraining methods and achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks, including VQA (+3.74% vqa-score), NLVR2 (+1.17% accuracy), SNLI-VE (+1.37% accuracy) and image captioning tasks (+10.1% average CIDEr score). Furthermore, we demonstrate that SimVLM acquires strong generalization and transfer ability, enabling zero-shot behavior including open-ended visual question answering and cross-modality transfer.

Published as a conference paper at ICLR 2022
SIMVLM: SIMPLE VISUAL LANGUAGE MODEL PRE- TRAINING WITH WEAK SUPERVISION
Zirui Wang" Jiahui Adams Wei Yu², Zihang Dai², Yulia Tsvetkov", Yuan Cao CCamegii Mellon University {ziruiw] 0ecss edu
"Google Research, Brain Team
{jiahuiyu, adamsyuwei, z sihangd, yuancac p @ggogle..coooo com University of Washington
{yuliat mulliat)ee fecs. washington. edu
ABSTRACT
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks. However, the requirement for expensive annotations including clean image captions and regional labels limits the seal- ability of existing approaches, and complicates the pretraining procedure with the introduction of multiple dataset-speeifie objectives. In this work, we relax these constraints and present minimalist pretraining framework, named Simple Visual Language Model VLMM, Unlike prior work, SiMVLM reduces the training complexity by exploiting large-scale weak supervision, and is trained end-to-end with a single prefix language modeling objective. Without utilizing extra data or task-specific customization, the resulting model significantly outper- forms previous pretraining methods and achieves new state-of-the-at results on a wide range of discriminative and generative vision-language benchmarks, includ- ing VQA (+3.74% vqa-score), NLVR2 (+1.17% accuracy), SNLI-VE (+1.37% ac- curacy) and image captioning tasks (+10.1% average CIDEr score). Furthermore. we demonstrate that SimVI.M VLM acquires strong generalization and transfer ability, enabling zero-shot behavior including open-ended visual question answering and cross-modality transfer.
INTRODUCTION
Self-supervised textual representation learning (Devlin et al.] 2018 Radford et al.) 2018 2019 Liu
let al.] 20199] Yang et al.) 2019: Raffel et al.) 2019] Brown al. 2020) based on Transtormers (Vaswani et al. 2017) has pushed the state the art on a wide range natural language pro- cessing (NLP) tasks (Rajpurkar et al. 2016 Wang et al. 2018 Sarlin al. 2020). One successful approach is to first pretrain the model (c.g. BERT on large-scale unlabled text corpora using masked language modeling (MLM) objective (Devlin al. 2018), followed by finetuning downstream tasks. While this cetraining-finetuniin paradigm has been widely adopted, recent work on autore- gressive language models (LM) (Radford et al. 2019 Brown et al.) (0200) such GPT-3 has shown strong performance without finetuning by utilizing few-shot prompts (Liu et al., 2021), suggesting the text guided zero-shot generalization is a promising alternative.
Motivated by the success of textual representation pretraining, various efforts have been made to build the multi-mode modal (visual and textual) counterpart. line of work (Tan & Bansal 2019 Lu et al. 2019; Li et al.) 2019 Chen et al. [020b] Li et al.) 2020; Su et al., 2020 Zhang et al. 2021 has explored Lision-language pretraining (VLP) that learns joint representation of both modali- ties to be finetuned on lanion-language (VL.) benchmarks, such as visual question answering (VQA) (Goyal et al., (017). In order capture the alignment between images and text, previous methods have extensively exploited two types of human-labeled datasets from multiple sources, which typi- cally consist of the following steps. Firstly, object detection datasets are used to train a supervised
"This work conducted Google.

---

Published as a conference paper at ICLR 2022
object detector (OD) which allows further extracting region-of -f-inererstt (ROI) features from images. Next, datasets with aligned image-text pairs are used for MLM pretraining of a fusion model that usually takes imput the concatenation of the extracted ROI features and the paired text. In addi- tion, due to the limited scale of human annotated data, various task-specific auxiliary losses have been introduced in order to improve performance. These design choices complicate the pretrain- ing protocol of VLP, creating a bottleneck for further quality improvement. What is more, such pretraining-fine fintuuningg based approaches usually lack the zero-shot capability, just like their lan- guage counterparts. In comparison, another line of work (Radford et al., 2021] Ramesh et al. 2021 Jia et al., 2021) utilizes weakly labeled/aligned data crawled from the web to perform pretraining, accieving good performance and certain zero-shot learning capability on image classification and image-text retrieval. Nonetheless, these methods mainly focus on specific tasks of consideration representation for VI. benchmarks.
and thus may not serve as a generic pretraining-finetune
In light of these disadvantages of the existing techniques, we are interested in building a VLP model that: (I) can be seamlessly plugged into the vvetraining-finetunin
paradigm and achieve competitive performance on standard VL benchmarks; (2) does not require a complicated pretraining protocol as in previous methods; and (3) has the potential towards text guided zero-shot generalization in cross-modal settings. To this end, we propose SimVLM, standing for Simple Visual Language Model, which significantly simplifies VLP by solely exploiting language modeling objectives on weakly aligned image-text pairs (Jia et al. 2021). In a nutshell, SimVI.X consists of the following components:
Objective. It is trained end-to-end from scratch with single objective of Prefix Language Modeling (PrefiXLM), which can not only naturally perform text generation as GPT-3, but also process contextual information in a bidirectional manner as BERT does.
The framework employs ViT/CoAtNet (Dosovisskiy et al. 2021 Dai et al., 2021) and directly takes raw images as inputs. These models can also fit the large-scale data and are readily compatible with the PrefixLMM objective.
Architecture.
Data. These setups relieve the requirement for object detection and allow the model to utilize the large-scale weakly labeled dataset, which has better potential towards zero-shot generalization.
Not only is VLM simpler, requiring neither object detection pretraining nor auxiliary losses, but also obtains better performance than previous work. Empirically, Sim VLM consistently out- performs existing VLP models and achieves new state-of-the art results on 6 VL benchmarks with- out additional data nor task-specific customization. Besides, it acquires stronger generalization in visual-language understanding that empowers zero-shot image captioning and open-ended VQA. In particular, Sim learns unified multimodal representation that enables zero-shot cross modd-lty transfer, where the model is finetuned on text-only data and directly evaluated on image-and test examples without further training. Our results suggest that generative VIP can not only match existing MLM-based methods on VL tasks but also demonstrate promising zero-shot potential.
RELATED WORK
Recent years have seen a rapid progress made vision-language pretraining (Uppal et 2020
Han et al.
1 Khan et al. 2021). While a variety of approaches have been proposed, a large portion of them require object detection for image region feature regression or tagging as part of
the pre- tre-training objectives TTan& & Bansal 2019 Su et al.
2020 Li et al., 2019; Chen et al. 2020b
Gan et al.
2020 Li al.) 2020 Yu et al. 2021 Li et al.
2021 Zhang et al. [221]] Hu et 2021
Cho et al.
202T These methods rely on strong object detection model like Fast(er) R-CN (Ren
et al.] al. (015)) 5) which is often trained on human annotated data sets like Visual Genome (Krishna et al.)
2016). Using such labeled training data as a prerequisite increases the cost of building the training pipeline, and makes the approach less scalable. Some recent efforts have also explored VLP without object detection module (Xu et al. 2021 Kim et al. 2021 Huang al. 2021). but they only use clean pretraining data with small scales and thus their zero- capability is limited.
On the other hand, multiple cross-modality loss functions have been proposed part of the training
objectives, for example image-text matching (Tan & Bansal] 2019 Lu et al. [2019] Xu et al. 202I),
masked region clasiification/feauue regression (Tan & Bansal 2019 Chen et al. 2020b)
object
2

---

Published as a conference paper at ICLR 2022
rurreng bapply
01
d11
raad
1
Transfomer Encoder
Transformer becader
XX
Taken Emibedding
postioonll emboocinng
PC patch-test embedding
Figure Illustration of the SimVLM model. This shows example of training with PrefixLM of an image-text pair. For text-only corpora, it is straightforward to remove the image patches and utilize textual tokens only.
attribute prediction (Xu et al. 2021), contrastive loss (Li et al. 2020 2021), word-region alignment (Chen et al., 2020b) word-patch alignment (Kim et al. 2021). They are often mixed with other objectives including image caption generation and masked language modeling to form compound pre-training losses. This creates the challenge of balancing among different losses and datasets, and thus complicates the optimization procedure.
Our work by contrast, follows a minimalist approach that takes raw image inputs and makes use of only the language modeling loss, without resorting to auxiliary models like faster R-CNN for image region detection. Motivated by recent works (Radford et al., [021]] Ramesh et al. 2021 Jia
al. 202T] Isimpoukelli et al. 202T) that illustrate zero-shot learning in certain image-text tasks, we train our model using large- scale weakly labeled data only. While concurrent work (Shen et al.)
202T] has explored building on top of models pretrained with such dataset, we focus on pretraining from scratch to explore the limit of generative VLP.
3 SIMVLM
3.1 BACKGROUND
The bidirectional Masked Language Modeling (MLM) has been of the most popular self- supervised training objectives for textual representation learning, As demonstrated by BERT (Devlin al. it based on the idea of denoising autoencoder such that the model is trained to recover the corrupted tokens in document. Specifically, given text sequence randomly sampled and corrupted sequence x.m constructed by replacing tokens in X,. with special [MASK] token. The training objective is to reconstruct x,, from the context X by minimizing the negative log-likelihoode
subset of tokens
are
(0) = -EEDD [log (x(x [x()))
(I)
where the trainable parameters the model and D is the pretraining data. This approach learns contextualized representations that can be further finetuned for downstream tasks. The MLM-style pretraining has been widely adopted in previous VLP models, whereby the input is an image-text pair and the model needs to predict masked tokens by leveraging image ROI features.
A Ateennatvvly, the unidirectional Language Modeling (L.M) trains the model to directly maximize the likelihood of the sequence under the forward autoregressive factorization:
T
(L = - [log Px(x) = E Σ
log Po (x (xx)
(2)
Compared with MLM, the LM pretraining has also been shown be highly effective for multiple NL.P tasks (Radford al. 2018). More importantly, it facilitates the model with strong generation
3

---

Published as a conference paper at ICLR 2022
capability that enables text induced zero-shot generalization without finetuning (Brown al. 2020 While MLM has become the de facto approach in VLP models reviewed above, the generative LM has been understudied.
3.2 PROPOSED OBJECTIVE: PREFIX LANGUAGE MODELING
Motivated by the zero-shot capability introduced by pre-training with LM loss, we propose pretain vision-languee representation using the Prefix Language Modeling (PrefixLM). PrefixLM differs from the standard LM such that it enables bi-directional attention on the prefix sequence (e.g. NeT, in Eq. (3) ), and only conducts autoregressive factorization on the remaining tokens (e.g. N2r, in Eq. 3D. During pretraining, prefix sequence of tokens of (a randomly selected) length T, truncated from input sequence and the training objective becomes:
CrreRaM EXWD [log Px(xxr,1xex,) - EXWD
log (x(xx|X(T,,)) XxT,)))))))
(3)
Intuitively, images can be considered as prefix for their textual descriptions they often appear before text in a web document. Therefore, for a given image-text pair, we prepend image feature sequence of length T; to the text sequence, and enforce the model to sample prefix of length T, 2 T; to calculate LM loss on text data only (an example is shown in Figure D Compared to prior MLM style VLP methods, our PrefixLM model under the not only enjoys the bidirectional contextualized representation as in MLM, but also can perform text generation similar to L.M.
to-sueeece framework
3.3 ARCHITECTURE
We adopt Transformer as the backbone of our model due to its success for both language and vision tasks (Devlin et al. 2018 Dosovitskiy al. 2021). Differently from standard LM, PrefiXLM enables bidirectional attention within the prefix sequence, and thus it is applicable for both decoder-only language models. In our preliminary experiments,
and encoder-decoder sequence-to-sequene found that the inductive bias introduced by encoder-decoder model which decouples encoding from generation is conducive to the improvement of downstream task.
An overview of our model architecture depicted Figure T For the visual modality, inspired by VIT Dosovitskiy al., 2021) and CoAtNet (Dai al., 2021). our model receives the raw image X E IH XC and maps it into flattened ID sequence of patches Xp E x as input for the
transformer, where D is the fixed hidden size of the transformer layers and T; = HW is the length of the image tokens for a given patch size P. Following Dai al. (02T), we use a convolution (Conv) stage consist of the first three blocks of ResNet (He et al. (016) to extract contextualized patches, which we find advantageous over the naive linear projection (cquivalent to x Conv layer) used in VIT, consistent with the observation from (Xiao et al. 2021). For the textual modality, we follow the standard practice to tokenize the input sentence into sub-word tokens (Kudo Richardson 2018), and the embeddings trainable ID positional embeddings for image and text inputs separately, and we additionally add 2D relative attention for the image patches within transformer layers (Dai et al. 2021). Notice that do not add extra modality type embeddings for which we found no improvement in our experiment. We study the effects of various components of the model in Section 4.4
learned for a fixed vocabulary. To retain positional information, add two
3.4 DATASETS
Since our approach does not rely on an object detection module and only operates with raw image patch inputs, we pretrain all model parameters from scratch using large-scale noisy image-text data, which has better potential for zero-shot generalization. Specifically, we use the image and alt-text pairs introduced in Jia et al. (2021), which are crawled from the web with minimal post-processing, On the other hand, our formulation of PrefixLM is modality-agnostic and thus we can additionally include text-only corpora to compensate for noisy text supervision in the alt-text data. As shown later our experiments, this unified PrefixLM formulation reduces the modality discrepancy and improves the model quality.

---

Published as a conference paper at ICLR 2022
VQA
NLVR2
SNLI-VE
CoCo Caption
NoCaps
Maltio De
test-dev
lest-std
dev
Test-P
dey
lest
B@4
M
C
Base sized Models
LXMERT
72:42
72.54
74.90
74.50
45.5
SOHO
73.25
73:47
76.37
77.32
85.00
84.95
VimV
77.87
78.14
81.72
81:77
84,20
84.15
39.0
32.9
134.8
24.0
94.8
13.1
46.6
Large-sized Models
UNITER
OSCAR
73.61
73.82
79.12
80.37
41.7
30.6
140.0
24.5
80,9
11.3
Villa
74.69
74.87
79.76
81.47
80.18
80.02
UNIMO
VinVT.
76.56
76.60
82.67
83.98
41.0
31.1
140.9
25.2
92.5
13.1
SimVLLMaag
79.32
79.56
84.13
84.84
85.68
85.62
40.3
33.4
142.6
24.7
108.5
14.2
47.5
Hage-sized Models
SimVI.M,
80.03
80.34
84.53
85.15
86.21
86.32
40.6
33.7
143.3
25.4
110.3
14.5
47.6
Table I: Single model results for vision-language pretraining methods on popular VL banchmarks. We report vqa-score for VQA, accuracy for NLVR2 and SNLI-VE, BLEU@4 for Multi30k and various metrics for image captioning @B@4 BLEU@4, M: METEOR, C: CIDEr, S: SPICE).
Compared to prior VLP methods consisting of two pretraining stages and multiple auxiliary ob- jectives, our model only requires one-pass pretraining using a single language modeling loss in an end-to-end manner, hence the name Simple Visual Language Model (SimVLM).
EXPERIMENTS
We conduct systematic experiments on a diversified set of visual-linguistic benchmarks, including visual question answering, image captioning, visual reasoning, visual entailment, and multimodal translation. We not only examine our model as general-purper VI. representation learning in the netraining-finetuniie paradigm, but also study its zero-shot generalization towards open-ended VI. understanding.
4.1 SETUP
Our models are implemented with the Lingvo framework (Shen et al. 2019). We follow the setup in ViT (Dosovitskiy et al. (021) to explore 3 variants of SiMVLM, namely "Base", "Large", and "Huge", such that each variant follows the same setting as its corresponding ViT variant. All models pretrained from scratch for about 1M steps the training set of ALIGN (Jia et al.] (02T] and the Colossal Clean Crawled Corpus (C4) dataset presented in Raffel et al. (2019). We mix the two pretraining datasets within each batch, which contains 4,096 image-text pairs (ALIGN) and 512 text-only documents (C4), sharded across 512 TPU v3 chips (Jouppi et al. 2017). ). More pretraining settings are detailed in Appendix B.1
After pretrained, our model is finetuned and evaluated on six vision-language benchmarks, including three discriminative tasks: VQA v2 (Goyal et al., 2017), SNLI-VE (Xie al., 2019), and NLVR2 (Suhr et al. 2018); as well as three generative tasks: CoCo captioning (Chen et al., 2013), NoCaps (Agrawal et al., (201)). and Multi30k (Elliott al.) 2016). We additionally examine its zero-shot generalization and performance on single-modality tasks. Details of tasks considered and the fine- tuning process are outlined in Appendix B.2
4.2 COMPARISON WITH EXISTING APPROACHES
To examine the quality of lasion-languuge pretraining, we first compare Sim VLM on the popular multi-modal tasks with statc-of-the-art (SOTA) VLP methods including LXMERT (Tan & Bansal 2019), VL-T5 (Cho et al. 2021), UNITER (Chen et al. (020b), OSCAR (i al. 2020). Villa (Gan al. 2020). SOHO (Huang al. 2021). UNIMO [i al. 2021). and VIn (Zhang al. 2021
As can be seen in Table T Sim VLM outperforms all existing models and achieves new SOTA results on all tasks considered, often by significant margin. This demonstrates generative pretraining approach is competitive with MLM-basee models and that simple framework with weak supervision sufficient learn high-quality multi-modal representations.
5

---

Published as a conference paper at ICLR 2022
CoCo Caption
NoCaps
Sctup
B@8
In
Near
Out
Overall
BUTT"
AoANet
supervised
39.5
29.3
129.3
23.2
M2 Transformet"
39.1
29.2
131.2
22.6
81.2
69.4
75,0
SinVLMoa:
9.5
11.5
24.0
7.5
83.2
84.1
82.5
83.5
Siue Siaa
zcro-shot
96.3
11.2
14.7
32.2
101.2
100.4
102.3
101.4
Sia
34.7
29.2
118.7
21.9
95.0
91.9
98.5
93.7
MinV
few-shot
35.4
102.5
106.0
Miae
36.8
31.5
131.3
24.0
111.8
110.6
111.0
110.4
OSCAR
41.7
30.6
140.0
24.5
85.4
84.0
80.3
83.4
VinVL!
pretrain fineruee
41.0
31.1
140.9
25.2
103.7
95.6
83.8
94.3
SinVLMMage
40.6
33.7
143.3
25,4
113.7
110.9
115.2
112.2
Table Image captioning results on CoCo Karpathy-test split and NoCaps validation split. For No- Caps, (In, Near, Out} refer to in-domain, near-domain and out-of-domain respectively. indicates Cider optimization. Model references: Annerson al. (2018) "Huang al. (2019) Cornia et al. (2020).
For the discriminative tasks, the Sim already outperforms all prior methods while using less capacity, and the SimVLM obtains almost points absolute score improvement compared to the previous SOTA (VinVL), pushing the single model performance above 80% on VQA for the first time. In addition, SimVLM also consistently outperforms prior methods on NLVR2 and SNLI-VE, illustrating its capability of processing more complex visual-linguistic reasoning. For the generation tasks including image captioning and image translation, Sim VLM also shows large improvements using naive finetuning techniques. Our model outperforms on the public "Karpathy" 5k test split of CoCo captioning as well as the NoCaps benchmark than prior methods trained with more complex reinforcement learning approach of CIDEr optimization (Rennie et al.] 2017). Finally, SimVLM is also effective for image translation of Multi30k from English German. These experiments demonstrate that our model can be seamlessly plugged into the pretraining- finetuning paradigm with superior performance, utilizing minimalist pretraining and finetuning procedures.
3 out of 4 metrics
4.3 ZERO-SHOP GENERALIZATION
A crucial benefit of generative modeling and scaling with weak supervision is the potential of zero- shot generalization. Models (Brown et al.] 2020) Radford et al.] [021] TTa et al.] (022) have been
shown capable of performing few-shot zero-shot transfer from pretrained models downstream datasets, even across language boundaries (Lample & Conneau 2019). In this section, we show- case three different settings of zero-shot applications less explored in prior VLP work, including transferring to unseen tasks, modalities and/or testing instances.
4.3. ZERO-SHIOT/FEW-SIIO IMAGE CAPTIONING
The pretraining procedure of VIM can be interpreted as a noisy image captioning objective on real-world web corpus. Thus, is natural to ask how well this caption ability generalizes to other datasets in a zero-shoUfewwshot manner. To this end, we take the pretrained SimVLM model, and directly decode on image captioning benchmarks for the zero-shot setting while finetune on 1% training data for 5 epochs for the few-shot setting. We also found that using prefix prompt "A picture of" improves the quality of decoded captions, similar to the finding in Radford et al. (2021).
As shown in Table 2 the zero-shot/few-sho performance (Appendix D of SimVLM is competi- tive with fully supervised baselines CoCo, and it also demonstrates strong generalization on the concept-rich NoCaps benchmark by achieving better scores than pretrained models. Figure 2 (a) illustrates sample captions generated by our model (Appendix A). SimVLM is able to not only cap- ture world concepts but also provide a detailed description of the visual input. For example, the decoded samples able to explain complex scenes with multiple objects (e.g. "people", "table with drinks", "dark restaurant"). Besides, the model also shows understanding of fine- -aa-grra abstraction such specific car brand and model (e.g. "Aston Martin". "Vantage"). Sim VLM even performs

---

Published as a conference paper at ICLR 2022
SNLI-VE SNLI Accap//ACCes
Multi30k Multi30k (T) B@4 M
SNLI-VE (T)
MNLI
Fully Supervised Baseline
EVE-Image
71.56 71.16
UNITER
78.59 / 78.28 85.00 84.95
SOHO
LIUM
23.8
35.1
GroundeeTTrans
15.8
31.2
Zero-Shot Cross-Modality Transfer
SinVLMooe
71.35/71.00
72.65 / 72.24
64.37 / 63.98
15.0
24.8
Sim SimVLM LMMgg
72.85 72.44
73.62/ / 73.23
66.97 / 66.31
17.7
30.1
Sim Viugg
73.56 73.08
74.24 / 73.86
67.45 / 66.97
18.2
32.6
Table 3: Zero-shot cross-modality transfer results on SNLI-VE and Multi30k. For SNLI-VE, the zero-shot model is finetuned on three source datasets: text-only SNLI-VE (Xie et al.) 2019), SNLI (Bowman et al. 2015), and MNLI (Williams al., 2017). For Multi30k, the model finetuned on text-only Multi30k data. Model reference: "(Specia et al. 2016).
robustly on challenging images that could be tricky for human, such as abstract or dark pictures. These illustrate that our model learns a wide range real-world concepts that generalize well in zero-shot manner.
4.3.2 ZERO-SHOT CROSS-MODALITY TRANSFER
Existing pretraining methods have been shown to be successful in transferring knowledge across heterogeneous data spaces. For example. multilingual language models (Devlin et al.) 2018 Lample & Conneau 2019) enable zero-shot cross-lingual transfer such that the model is only finetuned using training data from a source language (typically English) and evaluated on the target language without further training. Inspired by this setup, we explore a novel zero-shot cross-modality transfer paradigm of utilizing VLP models, and evaluate how well our model generalizes across modalities. Since text training data usually cheaper obtain compared to visual data, finetune Sim VLM on text-only downstream data and then directly evaluate the zero-shot transfer on joint VL tasks.
Specifically, We utilize SNLI-VE and Multi30k to examine the zero-shot transfer performance. For SNLI-VE, we finetune on three text-only NLJ datasets such that the premise sentence is used as the encoder's input while the hypothesis is fed to the decoder, and similar classifier head is trained on the embedding of the last token in the decoder. At inference, the finetuned model is evaluated by taking the premise image as the encoder input and the corresponding hypothesis sentence the decoder. As shown in Table 3 SimVLM performs competitively with fully supervised baselines including UNITER under the zero-shot setting. As sanity check, we also mask out the image feature to predict using the hypothesis only, and find our models can only obtain results close to random guess (average scores of 34.31 / 34.62). This results performance close random guess hence demonstrating the effectiveness of Sim VLM's cross- transfer ability.
In addition, SimVLM also capable of domain adaption by transferring from the MNLI dataset to SNLI-VE, whereby data comes not only from a different modality but also another domain. We also find possible to transfer across different languages and modalities using ViM Specif- ically, we utilize the German image captioning task from WMT 2016 of Multi30k for evaluation, where our model is finetuned on English-Gern German text-only translation data followed by decoding with image-only input in the encoder. Table shows that Sim VLM is capable of transferring knowl- edge across modalities and languages in generative tasks, achieving comparable performance to supervised baselines (decoded examples shown in Figure 2 (b)). These results suggest zero-shot cross-modality transfer emerges with the scaling of weakly labeled data.
4.3.3 OPEN-ENDED VQA
On the VQA benchmark, the best performing models to date formulate the problem as a discrimina- tive task of multi-label classification over predefined 3,129 answer candidates, often consisting of short factual terms. real-world applications, however, it is hard to define a closed set of candidate answers that covering all possible scenarios, making the true open-ended VQA challenging setup.
7

---

Published as a conference paper at ICLR 2022
Karpaily-test Out-domain
Partaal Train
Dev
In-domain
Overall
In-domain
Out-domain
Overall
Discriminative
UNITER
74.4
10.0
70.5
VL-T3
70.2
66.4
VL-BART
69.4
7.0
65.7
LLm
73.8
10.3
70.5
SimVLMupp
76.0
80.4
17.3
76.7
79.5
11.0
71.8
Sine
76.5
81.0
17.5
77.2
80.2
11.1
72.2
Gencrative
VL-TS
71.4
13.1
67.9
VL-BART
72.1
13.2
68.6
SimVI.Maa 73.2
78.3
25.8
75.2
77.1
27.1
71.3
75.2
72.5
SinVLMMpp 75.5
79.9
30.3
77.0
79.1
28.8
73.0
Table 4: Comparison of discriminative and generative VQA methods. "Dev" refers to standard vqa-score on the VQA validation split. "Karpathy-test" is the setup used in Cho et al. (2021) for evaluation the Karpathy split with answers. "Partial Train" refers to train the model only partial training data which contain subset of all candidate answers.
Generative models such Sim VL. provide an alternative solution towards this challenge by gen- erating free-form textual answers without being constrained to predefined answers. To this end, we finetune Sim VLM using the PrefiXLM loss described above where treat the concatenation of the image and the question as the prefix, and train the model to generate answers.
We then compare the generative approach with classification methods in Table 4 Firstly, we follow Cho et al. (202T) and evaluate model performance questions with rare an- swers the Karpathy-test split. Here, out- of-domain questions are defined as those with best-scoring answer not included the 3,129 candidates. Results show that VLM outper- forms both discriminative and generative base- lines all splits. More importantly, the gen- erative Sim VLM significantly improves on the out-of-domain split by over 17 points, demon- strating its strong generalization. However, this setup mainly focuses on rare answers and it re- mains unclear how well the model generalizes common unseen answers. We therefore pro-
Method
Acc@
SimCLRv2 Chen
2020a
79.8
DINO Caron
202
80.1
CLIP Radford et 2021
85.4
ALIGN (Jia et al.
202T
85.5
Sim
80.6
Sim
82.3
Sim VLMgge
83.6
Table 5: Linear evaluation on ImageNet classifi- cation, compared to state-of-the-ate representation learning methods.
ceed to investigate a more challenging setup where we randomly select 2,085 (about two-thirds of 3, 129) in-domain answers and partition both train and validation sets into two splits based on whether their best-scoring answers are included in the selected set not. We then only finetune SimVLM on the in-domain split of the train set and evaluate on the entire validation set. The "Par- tial Train" column in Table 4 shows that the generative Sim VLM is also competent in this setup by scoring reasonably well on over 1,000 unseen answers. Overall, we found the generative SimVi VLM performs competitively with its discriminative counterpart in the standard setup, and works generally better in the out-of-domain case.
Note that use the exact matching between generated answers and human labels for score caleu- lation in the above experiment, however it is possible that the model generates appropriate answers in different formats or synonyms. Therefore, in addition the quantitative study above, we show qualitative generation results in Figure 2 (c). It can be observed that Sim VLM is able to generate answers not included in the 3, 3,129 candidate set (e.g. "surgeon" and "wood carving"), demonstrating that Sim VLM can transfer knowledge from the pretraining corpus to VQA. It is thus natural to ask whether VLM can perform zero-shot VQA without finetuning at all. In our experiments, we found that ViM is able to "answer" by completing prompting sentences, (d). Nonetheless, we also observed that the model falls short in generating meaningful answers to the real questions. We hypothesize that this is due to the low quality of the pretraining data in which most textual descriptions are short and noisy. To verify our assumption, we continue the pretraining process the cleaner WIT dataset (Srinivasan et al. 2021 for 50k steps. Examples in Figure 2
shown in Figure 2
8

---

Published as a conference paper at ICLR 2022
show that open-ended VQA ability emerges in SimVLM such that it generate related responses after finetuning on the knowledge-richl wikipedia dataset.
4.4 ANALYSIS
Single-Modality Tasks. Since SimVLM per- forms well on joint vision-language bench- marks, natural to ask how well the learned representations perform tasks of sin- gle modality. We hope to gain deeper insights into the model behavior examining its per- formance on these benchmarks, but is not our intention achieve ttate-of-the-ar on single- modality tasks. In Table 7 (Appendix C). we compare VLM with existing VLP models on the GLUE benchmark (Wang et al. 2018),
Method
VQA score
No Pretraining
49.70
Decodcr-only
65.23
w/ LM
64.48
Sim VLL
67.43
w/ Image2Text
49.23
w/o Text2Text
65.25
w/o conv stage
63.11
where we mainly follow the text processing procedure in Raffel et al.] (2019) and train our model to classify the fully formatted input with- out token type embeddings. Sim performs better than existing VLP methods and compet- itively with BERT, indicating that it has good language understanding ability. Additionally, also compute the top-1 accuracy on Ima- geNet following the linear evaluation protocol Table Note that our model is not pre- trained with discriminative task such as the contrastive loss, hence we use an average pool- ing of encoder outputs image features. Re- sults verify that our model has also learned high-quality image representation.
w/ span corruption
w/ 2 conv biks
65.57
w/ 4 conv blks
66.55
w/ 10% ALIGN
66.71
w/ CC-3M
63.32
Table 6: Ablation study on VQA. "w/ LM" and "w/ span corruption" denote replacing the pro- posed PrefiXLM loss with different pretraining objective. "Image2Text" and "Text2Text" refer to the noisy image-text data and the text-only data used for pretraining. "conv blks" denotes number of ResNet blocks.
Ablation Study. study contributions from each model component, we conduct ablation study on SimaLM models with embedding dimension of 512 and 8 layers. We make compar- isons on VQA in Table 6 First, we compare secoder-decoder models with decoder-only models of comparable model size, and find that decoder-only model performs significantly worse VQA. This suggests the inductive bias of separating bidirectional encoding from unidirectional decoding beneficial for joint VI. representation learning. Next, we study the effectiveness of pretraining objectives and results show that the PrefiXLM objective outperforms both span corruption (Raffel al., 2019) and naive LM, illustrating the importance using unified objective formulation for both image-text and text-only data. Moreover, we ablate the contribution of datasets. While weakly aligned image-text data are required for bridging the gap between visual and textual representa- tions, text-only corpora also improves the model quality. This is probably because textual signals are extremely noisy in the former and thus the model relies on the later to acquire better language understanding. In addition, we experimented with 10% ALIGN and CC-3M (Sharma et al. 2018) datasets, and confirms the importance of data scaling. We then study the effect of the convolution stage and find it critical for VL performance. Following Dai et al. (202 1), we experiment with using either the first 2/3/4 ResNet Conv blocks, and empirically observe that the conv block setup works best. This indicates that image and text have different levels of representation granularity and thus utilizing contextualized patches is beneficial.
CONCLUSION
In this work, we present simple yet effective framework of vision-language pretraining. Unlike prior works using object proposal systems and auxiliary losses, our model processes whole image patches and is trained end-to-end with single prefix language modeling objective. Our work suggests a promising alternative to existing VLP paradigm and we hope our work may inspire future research generative VI.P.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We would like thank Hieu Pham, Chao Jia, Andrew Dai, Bowen Zhang, Zhifeng Chen, Ruoming Pang. Douglas Eck, Claire Cui and Yonghui Wu for helpful discussions, Krishna Srinivasan, Samira Daruki, Nan Du and Aashi Jain for help with data preparation, Chao Jia, Zhen Li, Jonathan Shen, Colin Raffel and Sharan Narang for assistance on experimental settings, and others in the Google Brain team for support throughout this project.
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 8948-8957, 2019.
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 6077-6086, 2018.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large anno- tated corpus for learning natural language inference. arXiv preprint arXiv:1508.0520 508.0536.. 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 4165, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in Neural Information Processing Systems, 33:22243-222555, 2020a.
Xinlei Chen, Hao Fang, Tsung-Y- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint /504 00325, 2015.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020b.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-anddllanuuage tasks via text generation. arXiv preprint 212.v22102.02779, 2021.
Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory trans- former for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10578-10587, 2020.
Zihang Dai, Hanxiao Liu, Quoe Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes, arXiv preprint arXiv:2106.04M8 04803, 2021.
Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1 180.0000 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recogni- tion at scale. In International Conference Learning Representations, 2021. URL https: //openzeview wtwt//ooum///
Desmond Elliott, Stella Frank, Khalil Sima' an, and Lucia Specia. Multi30k: Multilingual english- german image descriptions. arXiv preprint arXiv: 1605.00459 2016.
10

---

Published as a conference paper at ICLR 2022
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and- daiio-and-langeen and-language representation learning. arXiv preprint arXiv:2006. 000.0000066999 06195, 2020.
Yash Goyal, Tejas Khot, Douglas Commers-SSaay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6904-6913, 2017.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, and Dacheng Tao. A survey on visual transformer, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Xiaowei Hu, Xi Yin, Kevin Lin, Lijuan Wang, Lei Zhang, Jianfeng Gao, and Zicheng Liu. Vivo: Surpassing human performance in novel object captioning with visual vocabulary pre-training. In AAAI, February 2021.
Gao Huang, Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pp. 846-661. Springer, 2016.
Lun Huang. Wenmin Wang, Jie Chen, and Xiao-Yong Wei. Attention on attention for image cap- tioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1634-464,, 2019.
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre- traininiing for lanonnlanguage representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12976- 2021.
Taichi Iki and Akiko Aizawa. Effect of vision-and-languaage extensions on natural language under- standing vision-and -nd--and--agaee lan-uagg models. arXiv preprint arXiv:2104. 08066, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.055::
2021.
Norman P. Jouppi. Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre luc Cantin, Clifford Chao. Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexan- der Kaplan, Harshit Khaitan, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adri- ana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov. Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Er- ick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of a tensor processing unit, 2017.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey, 2021.
Wonjae Kim, Bokyung Son, and Iddo Kim. Vilt: Vision-and-language transformer without convo- lution region supervision, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 2016. URL wttpt:ttpps...... htapp::://xxiii wwwperiwvv..rrr/eeom abwgg//bs//in w0t..//110.... 07332
11

---

Published as a conference paper at ICLR 2022
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1888.06226 2018.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language, 2019.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang- UNIMO: Towards unified-mode modal understanding and generation via cross-modal contrastive learn- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- fics and the 11th International Joint Conference Natural Language Processing (Volume 1; Long Papers), pp. 2592-2607, Online, August 2021. Association for Computational Linguis- http53//12222.all///ee acl ael-longe 202
tics. doi:
URL https: http:////acaaattoooooooo org/2021,
Xiujun Li, Xi Yin, Chumyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang. Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semanties aligned pre- training for vision-language tasks. ECCV 2020, 2020.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing, 2021.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.5501, 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision- vision-and-languaal lnd-uang tasks. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Sys- tems, volume 32. Curran Associates, Inc., 2019. URL https:hhtts/::/oeeeel ngs neurips
Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859 2016
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 1xXi:2103.00022. 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910. 190.010000008 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 2016.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2 /2092, 2021.
12

---

Published as a conference paper at ICLR 2022
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran As- sociates, Inc., 2015. URL http:tppss//ppprreeeiiggg.g nww..s..errip.... ips wttppappe/211///11111111 pdf
Steven Rennie, Etienne Marcheret, Youssef Mrouch, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7008-7022 2017.
Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 4938-44964 4944,4947444 2020.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2556-22 2565, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P11 1238. URL https:://///m... httpt::t//aaaaaa https///accaaaacoooooooooo//////
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, et al. Lingvo: modular and scalable framework for
modeling, 2019.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and-lanm and-languagee tasks? arXiv preprint arXiv:2107.06383 2021.
Lucia Specia, Stella Frank, Khalil Sima' An, and Desmond Elliott. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 543-553, 2016.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia- image text dataset for multimodal multilingual machine learning, arXiv preprint arXiv:2103.01913, 2021.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre- training of generic linuul-iinguistic representations. International Conference on Learning Representations, 2020. URL http::ttts//// httpppss.//peeeeeeeeeeee http:///..ppppeeeeeeeeeeeeeeeees /oree//frrmii--eeyyPaaait
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00m 1.00491, 2018.
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5100-5111, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10. (0.18653/11/D19-1551 URL
D19-151 1114
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multi- modal few-shot learning with frozen language models. Advances in Neural Information Process- ing Systems, 34, 2021.
Shagun Uppal, Sarthak Bhagat, Devamanyu Hazarika, Navonil Majumdar, Soujanya Poria, Roger Zimmermann, and Amir Zadeh. Multimodal research in vision and language: A review of current and emerging trends, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008, 2017.
13

---

Published as a conference paper at ICLR 2022
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for 2017.
sentence understanding through inference. arXiv preprini
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better, 2021.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine- grained image understanding. arXiv preprint arXiv:/901.00 2019.
Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, and Fei Huang. E2E-VLP: End-to-end vision-language pre-training enhanced by visual learning. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 503-513, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 1021.acl-long.coo URL BoRL/httpps:/acaaatholoooo wtw.://ttthooooooooo/22222/22220 aclllacclllloo.. .2
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, pp. 5754-5764 2019.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vil: Knowl- edge enhanced vision-language representations through scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 3208-3216, 2021.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. VinvI: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5579- 5588, June 2021.
14

---

Published as a conference paper at ICLR 2022
IMAGE
PREFIX
OUTPUT
+ picture of"
+ picture of"
+ picture of"
+ picture of"
"a man driving yellow and black aston martin vantage on the road."
"a group of people sitting at table with drinks in dark restaurant."
"abstract drawing with grey and white triangles."
closeup of red seahorse in a dark aquarium."
(a)
+
+
"what is the profession of this person?"
"what is the doing?"
"vier mädchen schnee"
"ein hund im wasser"
"wood carving"
"surgeon"
(b)
(c)
"this building is located in"
"this food is kind of"
"what can a visitor do here?"
"where to abserve this animal?"
"american breakfast dish."
"the tower located in the city paris and has two restaurants."
ssyneey australia."
"the giant panda is native to central china."
(d)
(e)
Figure 2: Generated examples of SimVLM of various applications: (a) zero-shot image captioning (b) zero-shot cross cross-ty transfer on German image captioning (c) generative VQA (d) zero-shot visual text completion (e) zero-shot open-ended VQA.
A GENERATED EXAMPLES
Examples generated by SimVLM of various types are shown in Figure 2 We use either image-only image-text prefix inputs in the encoder, and use the decoder generate suffix text.
15

---

Published as a conference paper at ICLR 2022
CoLA
SST-2
RTE
MRPC
QQP
MNLI
QNLI
WNLI
BERT
54.6
92.5
62.5
81.9/87.6
90.6/87.4
84.2
91.0
48.8
VisualBERT
38.6
89.4
56,6
71.9/82.
89.4/86.0
81.6
87.0
53.1
UNITER
37.4
89.7
55.6
69.3/80.3
89.2/85.7
80.9
86.0
55.4
VL-BERT
38.7
89.8
55.7
70.6/81.8
89.0/85.4
81.2
86.3
53.1
VilBERT
36.1
90.4
53.7
69.0/79.4
88.6/85.0
79.9
83.8
55.4
LXMERT
39.0
90.2
57.2
69.8/80.4
75.3/75.3
80.4
84.2
46.0
Sim VLMMase
46.7
90.9
63.9
75.2/84.4
90.4/87.2
83.4
88.6
58.1
Table 7: Text-only task performance on the GLUE benchmark (Dev set). Results for BERT and
other VLP methods obtained from [ki & Aizawa (2021). The overall best result is bolded while underline signifies the best VLP model.
B EXPERIMENTAL DETAILS
B. PRETRAINING
Our models are pretrained according to the methodology described in Section former, each variant follows the same setting as corresponding ViT variant. For Conv stage, the first three blocks (excluding the Conv stem) of ResNet-101 and ResNet-152 (He et al., 2016) for our Base and Large models respectively, and a larger variant of ResNet-152 with more channels for the Huge model (matching its hidden dimension size). We always of 16x 16. During pretraining, we utilize the resolution of 224x224, resulting in a patch sequence of length 14x 14 visual tokens. For the textual input, we use vocabulary size of 32,000 and max sequence length of 256 in both the encoder and the decoder. We also share parameters between the embedding and the decoder softmax output layer (Press & Wolf, 2016). All parameters are shared across visual and textual inputs except the Conv stage and positional embeddings.
3 For the Trans-
fixed patch size
We pretrain on scrge-scale web datasets for both image-text and text-only inputs. For joint vision and language data, we exploit the training set of ALIGN (Jia et al. 202I), which contains about 1.8B noisy image-text pairs. Notice that we do not use any extra data preprocessing or filtering, except simple random resized cropping. For the text-only copora, we use the Colossal Clean Crawled Corpus (C4) dataset presented in Raffel et al. (2019) and followed their preprocessing steps. The dataset contains about 800GB of web crawled documents.
All models are pretrained for about 1M steps from scratch to optimize for the single PrefiXLM objective in Eq 3 We use the AdamW optimizer (Loshchilov & Hutter 2017) with 31 0.9, 82 0.999 and weight decay of 0.01. We warm up the learning rate for the first 2% of updates to a peak value of 5x100* and then linearly decay it afterwards. Dropout is used during the pretraining stage. We mix the two pretraining datasets within each batch, which contains 4,096 image-text pairs and 512 text-only documents, sharded across 512 TPU v3 chips (Jouppi et al. 2017)
B.2 FINETUNING
After pretraining, our model is finetuned various downstream tasks. Similar to the pretraining stage, we use the AdamW optimizer with the same Beta values, while we tune the learning rate in (1x10-5, 2x10-5, 5x10-5) We also enable regularization methods of Dropout (set 0.1) and stochastic depth (only applied to Conv stage and encoder with fixed dropout rate of 0.1) (Huang jet al. 2016) during the finetuning stage. Following standard practice, we use the corresponding dev split to find the best setting and report the result the test split. We consider 5 types of downstream tasks listed below:
Visual question answering: This task requires the model to answer questions about input images, and has been the most widely used VL benchmark. Following prior work, we use the VQA v2 (Goyal et al. 2017) and formulate the task as classification problem over 3,129 most frequent answers in the training set. The raw image and the corresponding question are used as inputs to the encoder and the decoder respectively, and task-specific linear classifier is trained predict answer
16

---

Published as a conference paper at ICLR 2022
based activation corresponding to the last question token from the decoder. We use resolution of x800x400 for the image and all positional parameters are adapted using linear interpolation.
The SNLI-VE (Xie et al. 2019) dataset is adapted from SNLI (Bowman al, 2013). which is originally designed to predict the relation between premise sentence and a hy- pothesis sentence either entailment, neutral contradiction, task known natural language inference (NLI). For the VL variant, the premise is based on the content of an image rather than textual descriptions. We finetune Sim VLM similarly VQA, such that the image and the sentence are fed to encoder and decoder separately, and the classifier is trained to predict the three relations.
Visual entailment:
Visual reasoning: The NLVR2 (Suhr al. 2018) dataset tests the model's ability of jointly rea- soning over the language and multiple images by asking whether a textual description is true based on pair of two images. Following Zhang al. (2021), we create two input pairs, each consisting of one image and the textual description, and generate output embeddings for both using the same setup above. The two embeddings are then concatenated for final prediction.
Image captioning: The captioning task requires a model to generate natural language descriptions of input images. We consider two datasets CoCo (Chen et al.] 2015) and NoCaps (Agrawal et al. 2019), both finetuned using the CoCo training data. For VLM, it is straighttorward to first encode the image in the encoder and then generate captions using the decoder. Note that in contrast to prior work that apply task-specific tricks such CIDEr optimization (Rennie et al. 2017). our model is trained with naive cross-entropy loss only.
Multimodal translation:
The goal of multimodal translation is to translate image descriptions in source language to target language, for which image inputs can be taken advantage as grounding signal. We train and evaluate on the Multi30k (Elliott et al. (016) dataset. We utilize the PrefixLM described in previous sections such that the source sentence, together with the image inputs, are fed the encoder, which will be translated to the target language by the decoder.
C MODEL PERFORMANCE ON LANGUAGE-ONLY TASK
We compare model with prior VLP methods natural language understanding (NLU) tasks the GLUE benchmark [Wang et al. 2018) in Table 7
D ERRATUM
We found an error in reporting the zero-shot COCO evaluations in the first version of this paper. This mistake does NOT affect all other results and the numbers have been updated. Meanwhile, also added few-shot results in addition to zero-shot results on both MsCOCO and NoCaps in Table to provide a more comprehensive view of capacities in Sim VLM models. Hence, our main claims and conclusions still hold.
17


Reviews:
[{"review_id": "YHyLABJUDzy", "reviewer": "Reviewer_PHG8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strength\n\n**Simplified pretraining procedure.** The most important point is to simplify a pretraining procedure. In recent works, MLM and image-text matching (ITM) losses are repeatedly used and additionally propose dataset-specific losses to exploit inductive biases. Unlike this trend, this work counters the general belief and gives a new insight to use the PrefixLM for vision-language pretraining (VLP) as a standalone objective. However, this paper's argument seems to be fragile for the reasons in the below weakness points.\n\n**Generalization and transfer knowledge.** The PrefixLM is a generative modeling loss compared to the MLM's classification loss. For this reason, it may be keen on zero-shot generalization. They show competitive or better performance on zero-shot image captioning, zero-shot cross-modality transfer, and open-ended visual question answering tasks.\n\n### Weakness\n\nThe PrefixLM is the paper's central argument, aside from the weak supervision dealt with in ALIGN (Jia et al., 2021) and others (e.g., CC 12M, Changpinyo et al., 2021). However, this paper lacks solid validation of their argument, as summarized below.\n\n**W1. Can the PrefixLM replace MLM?** The critical ablation study to see the effectiveness of PrefixLM compared to MLM is 1) limitedly shown on VQA and image captioning in Table 6, a limited number of downstream tasks compared to the other experiments, although it is a critical study to validate their argument. Moreover, 2) this ablation study is performed for text-only data, not vision-language data in Section 4.4 Ablation Study. For this reason, their argument is fragile to be nullified in the further validation in the current status.\n\n- Could you provide any evidence that the PrefixLM is better than MLM in the controlled experiment (ablation study) for multimodal data?\n\n**W2. Can the PrefixLM be used as a standalone objective?** For instance, the ITM loss is known to be effective combining with MLM (e.g., ViLT, Kim et al., 2021). They argue that a single objective is enough, but there is no validation of whether this is true. The natural questions on this matter include combining with ITM or other dataset-dependent losses would be helpful or result in mediocre performance.\n\n- Could you provide any evidence that PrefixLM + ITM method is not significantly different from PrefixLM only method? If not (PrefixLM + ITM is better), what would be the advantage of using the PrefixLM only? \n\n**W3. Fair comparison issue with the ALIGN dataset.** The crucial problems in the experiments are that 1) it is irreproducible for its inaccessible collected datasets outside of the associated groups, and 2) it is hard to assess the contribution in the state-of-the-art comparison due to different pretraining datasets, different model architectures, and pretraining objectives. The controlled experiment still has the problems as described in W1. \n\n- For the reproducibility, could you provide the scores pretraining on WIT (Srinivasan et al., 2021) only?\n\n\n### Writing\n\n- In the RHS of Eqn.3, if $t=T_p$, is $P(x_{T_p}|x_{[T_p,T_p]}, x_{< T_p})$ the intended one? Should $x_{[T_p,T_p]}$ be null?\n- It is ambiguous that \"which decouples encoding from generation is conducive to the improvement of downstream task\" in the 1st paragraph, Section 3.3. What do you mean by \"decouple\"?\n- In the 2nd paragraph of Section 3.3, \"we additionally add 2D relative attention for the image patches within transformer layers,\" which needs some elaboration for reproducibility and self-contained explanation.\n- Using C4 dataset was crucial? Why do you choose to use C4 additionally?\n- In the 4.1 Setup, you mentioned that \"512 text-only documents are in each batch,\" but, in this reading moment, cannot follow the context. \n- In Table 2, \"'Pre.' indicates the model is pretrained\" on what? Are you mentioning that the corresponding models are pretrained on CoCo or NoCaps? Here, all SimVLM is also pretrained on both ALIGN and C4 datasets, but you don't say all your model is pretrained.\n- In 4.3.1, please be aware that your Figure 2 is placed in Appendix and indicate explicitly. \n- In Table 2, could you specify the meaning of each row section? Why are two SimVLM_{huge} scores different in the 2nd and 3rd sections?\n- Explanation on transformer decoder is simply skipped. Could you provide implementational details for reproduction?", "summary_of_the_paper": "This paper proposes a Prefix Language Modeling (PrefixLM) objective for a pretraining procedure for multiple vision-language downstream tasks and zero-shot evaluations. They argue that it successfully replaces the masked language model (MLM). This work follows the notion of weak supervision suggested by ALIGN (Jia et al., 2021), having noisy image captions and holistic labels. Surprisingly, this simple pretraining approach achieves new state-of-the-art on \"a wide range of discriminative and generative vision-language benchmarks,\" and shows \"strong generalization and transfer ability\" in zero-shot settings.", "summary_of_the_review": "This paper shows that a simple pretraining method, the PrefixLM, using two datasets from the dataset of ALIGN (Jia et al., 2021) and the Colossal Clean Crawled Corpus (C4) dataset (Raffel et al., 2019) (optionally, along with WIT (Srinivasan et al., 2021)) achieves new state-of-the-art in various vision-language benchmarks. However, this work is narrowed to achieve the state-of-the-art performances while failing to show 1) better performance of the PrefixLM than MLM, 2) relationships with the other multimodal pretraining losses (e.g., ITM) 3) in the controlled experiment in terms of pretraining dataset and model architecture. The weaknesses W1-3 hinder this paper's recommendation since the main argument of the PrefixLM is in a vague position. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ClN7H4wXNhY", "reviewer": "Reviewer_wzgn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPros:\n\n1. The empirical results are strong. SimVLM takes an important step towards less human-label-dependent V&L models.\n\n2. Interesting results on some zero-shot tasks. In Sec 4.3.2, the authors introduce a new setting: train on a similar single-modality task and transfer directly to a multi-modality task. This is a new yet promising way of transferring to new V&L tasks.\n \n\nCons:\n\n\n1. Dataset deduplication. I do not see if the authors check whether downstream tasks data are present in the pre-training dataset. It is recommended to check the overlap.\n\n2. Image caption / SNLI-VE and VQA experiments are all grouped into the zero-shot setting. They are technically \"zero-shot\" but they actually differ quite a lot from others. SNLI-VE & VQA experiments are also not the \"full zero-shot\" setting where we simple take a model and directly get the desired output. I would recommend being more careful about the wording. \n\n3. For the experiment on SNLI-VE in Table 3, I think the same method could also be done for UNITER (i.e. train on SNLI and then transfer to SNLI-VE). It is an informative baseline to have.\n\n\nMinor points:\n\n4. Could the cross-modality experiments be extended to other tasks such as VQA? Could we train on  some similar text-only tasks and then transfer to VQA? The SNLI-VE is a good sign but SNLI-VE is a three-way classification task and is perhaps easier to transfer to than VQA.\n\n5. Efficiency concern. Training on CLIP/ALIGN-level data is costly. A seemingly more efficient way to use trained CLIP/Align model as bottom backbones and train a V&L transformer on top (e.g., Shen et al., 2021). With the same model scale (base), the VQA score is not so far apart (78.14 v.s. 76.48).\nShen et al., How Much Can CLIP Benefit Vision-and-Language Tasks?\n\n6. True Zero-Shot VQA? I wonder what prevents the authors from directly testing on VQA. One could give the image + question as the prefix and then calculate the generation probability for every answer candidate from the 3,129 answer pool. This would serve as a valuable reference point.\n", "summary_of_the_paper": "The paper proposes to pre-train a generative language model conditioned on a visual input on billion-scale web image-text data. Such a model can be then transferred to various vision-and-language tasks with ease. SimVLM establishes new SotA on several new tasks and shows promising zero-shot capacity in certain tasks.\n\n", "summary_of_the_review": "Overall, I think the cons are points that could be addressed and do not hurt the main merit of the paper. Thus I recommend acceptance but I would recommend addressing the cons if possible.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BPhhcW0mtfg", "reviewer": "Reviewer_Z7A6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### [Strengths]\n- Overall, I like this paper. The proposed method is very simple but effective and practical because the pretraining task is not complex and the used weakly aligend image-text data are relatively easy to acquire compared to those used in other methods such as CLIP. \n- Despite its simplicity, the performance are promising for various fine-tuning tasks.\n- The paper is easy to understand.\n- The experiments are thorough and support their hypothesis. \n\n#### [Weakness]\nDespite many strenghs of this paper, there are some room to be improved. My score is between 6 and 8 due to the issues below:\n- In introduction, the authors argued \"These design choices complicate the pretraining protocol of VLP, creating a bottleneck for further quality improvement.\" and \"these methods mainly focus on specific tasks of consideration and thus may not serve as a generic pretraining-finetuning representation for VL benchmarks.\". Are there any explicit evidences or refences for this arguement? \n- Figure 1 can be improved. When zooming up, some texts are not clear.\n- Reproduciblity is limited. The limited reproducibility by other research groups results from data issue. It is not trivial to construct 1.8B paired data even if they are weakly-aligned and train them. Also, I guess that the promising results of a simple pretraining task and architectures might benefit from large-scale size of its training data. Therefore, the ablation experiments on various training data sizes (e.g. 10%, 30%, 50%, and 100%) will be helpful for understanding the effects of the data size on the performances. If smaller size data can provide comparable results, the contributions of this work will be much enhanced. \n- Presenting parameter sizes of each model variant will help to understand the results. \n- Table 2 is not easy to read. In Table2, what is the difference betwwen the second and the third row groups? I guess the second is for zero-shot and the third one is for fine-tuning. This should be explicitly clarifed in Table for readability. Also, why is SimVLM_large checked only as pretraining? \n- The results in the main paper and the appendix are explained in the main manuscript. For example, even if Figure 2 and Table 7 exist in the appendix, there is no information on those. This harms the readability.\n- Table 3 can also be improved. Fine-tuned datasets and metrics are located at the same row. Image Masking results also might harm the readability. \n- Why the perfomances of SNLI-finetuned is better than those of SNLI-VE-finetuned? The authors need to clarify this reason.\n- Multimodal learning can provide more robust performance on different domain data like zero-shot image classification on line-draw images of CLIP. SimVLM also present more robust image classification performances on different domain data? In addition to classification accuracy, these results can enhance the contribution of this paper, if possible. \n- In Ablation study, why used SimVLM_small? How about the possibility that poor performance of decoder-only model resulted from its smaller model size? \n- Minor issues (typos)\n  - In 3.4 in p4, \"with raw image patch inputs.\" . --> , (comma)\n  - in 4.1 in p, \"Huge\".: . --> , (comma)\n  - Karpapth --> Karphathy \n", "summary_of_the_paper": "This paper proposes a simple but effective image-text multimodal representation learning method that leverages a transformer-based encoder-decoder using a simple prefix langemodel as pretraining task from large-scale noisy image-text aligned data. As fine tuning tasks, the authors evaluate their method (SimVLM) on VQA, NLVR2, SNLI-VE, CoCo caption, NoCaps, and Multi30k. With extensive experiments, this work presents promising few-shot and zero-shot performance results outperforming previous models.  ", "summary_of_the_review": "Overall, this is a good paper but there are some issues that should be improved, in particular, reproducibility and table for readability. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents SimVLM, a simpler generative VLP framework with billion-scale web image-text data, which has good zero-shot potential while remaining competitive on standard VL benchmarks. SimVLM achieves SotA on several tasks and shows promising zero-shot capacity in certain tasks. Most of the reviewers liked the work; they had concerns about data scaling, but the authors showed that SimVLM_small with the smaller cc3m dataset does not drop performance too much (although the large data scaling with their large-scale weakly aligned data is still important to achieve good zero-shot learning). All reviewers also mentioned the strong concern of reproducibility and data accessibility, so we encourage the authors to address this as clearly as possible via releasing models and safely-cleaned/anonymized data subsets, etc.
Decision: Accept (Poster)