Masked Vision and Language Modeling for Multi-modal Representation Learning
Authors: Gukyeong Kwon
Zhaowei Cai
Avinash Ravichandran
Erhan Bas
Rahul Bhotika
Stefano Soatto
Venue: ICLR 2023 poster (2023)
Keywords: Vision and language
multi-modal learning

Abstract: In this paper, we study how to use masked signal modeling in vision and language (V+L) representation learning. Instead of developing masked language modeling (MLM) and masked image modeling (MIM) independently, we propose to build joint masked vision and language modeling, where the masked signal of one modality is reconstructed with the help from another modality. This is motivated by the nature of image-text paired data that both of the image and the text convey almost the same information but in different formats. The masked signal reconstruction of one modality conditioned on another modality can also implicitly learn cross-modal alignment between language tokens and image patches. Our experiments on various V+L tasks show that the proposed method, along with common V+L alignment losses, not only achieves state-of-the-art performance by using a large amount of data but also outperforms the other competitors by a significant margin in the regimes of limited training data. 

Published as a conference paper at ICLR 2023
MASKED VISION AND LANGUAGE MODELING FOR MULTI-MODAL REPRESENTATION LEARNING
Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, Stefano Soatto AWS AI Labs
{gukyeong, zhaoweic, soattos 8ama com.aaaazccooommmmmomooooooooooa com
ABSTRACT
In this paper, study how to masked signal modeling in vision and language (V+L) representation learning. Instead of developing masked language model- ing (MLM) and masked image modeling (MIM) independently, we propose build joint masked vision and language modeling, where the masked signal of one modality reconstructed with the help from another modality. This is motivated by the nature of image-text paired data that both of the image and the text con- vey almost the same information but in different formats. The masked signal re- construction of one modality conditioned on another modality can also implicitly learn cross- modal alignment between language tokens and image patches. Our ex- periments on various V+L tasks show that the proposed method, along with com- mon V+L alignment losses, achieves state-of-the-art performance the regime of millions of pre-trainin pre-training data. Also, outperforms the other competitors by significant margin in limited data scenarios.
INTRODUCTION
Vision and language (V+L) representation learning has gained significant attention due to the trans- ferablility of the representations a diverse set of downstream tasks such as zero- or few-shot visual recognition (Jia et al., 2021; Radford et al., 2021; Tsimpoukelli et al., 2021), object detection (Cai al., 2022: Kamath et al.. 2021), information retrieval (Li al., 2022; 2021), and multi-modal generation (Ramesh et al., 2022; 2021) etc. This success is mainly driven by large-scale pre-training with paired image and text data. The current V+I. pre-training techniques particularly focus on the representation learning that characterizes the association between vision and language, and they are largely inspired by self-supervised learning techniques (Devlin et al., 2018; He et al., 2020) in uni-modal learning.
Masked signal modeling is a popular self-supervisory pre-training task (Devlin et al., 2018; Liu et al.. 2019; Yang et al., 2019; Bao et al., 2021; Xie et al., 2021; He et al., 2022), which aims at reconstructing the masked signals from the unmasked ones. has been independently explored in the domains of natural language processing (NLP) and computer vision (Devlin et al., 2018; Liu et al., 2019; Yang et al., 2019; Bao et al., 2021; Xie et al., 2021; : He et al., 2022). For example, in the domain of NLP, BERT (Devlin et al., 2018) and several follow-up works (Liu et al., 2019; Yang et al., 2019) utilize masked language modeling (MLM) where the model is expected to predict the masked text tokens using unmasked tokens. They have shown that MLM leads powerful generalization performance across diverse NLP tasks. In computer vision, shown in Figure (top-left), the masked image modeling (MIM) is to predict masked pixels or image patches using unmasked portions of the images. MIM has shown to be an effective pre-training task for learning visual representations (Bao al., 2021; Xie et al., 2021; He al., 2022).
While MLM and MIM have been actively explored in each domain, existing works do not fully leverage the masked multi-modal signal modeling in the domain of V+L. For example, as shown in Figure/ I (bottom-left), several approaches rely only on MLM with unmasked images and do not model the masked images (Duan et al., 2022; Li et al., 2022; 2021; 2019; Yang et al., 2022). In this case, the distribution of text given image, p[] can be leamee, but the distribution of image given text, P)) cannot. This will potentially lead to biased performance in cross- cross-modal retrieval tasks such as image-to-text text-to-image retrieval as shown in our experiments. Although there

---

Published as a conference paper at ICLR 2023
Masked Image Modeling
Masked Vision and Language Modeling
dog running towards green ball"
Masked Language Modeling in V+L Learning
"A dog is running lowards green ball"
dog is running towards green ball'
"A [MASK] is running towards green ball'
"A [MASK] is running towards green ball"
Figure 1: An overview of masked vision and language modeling. The left side shows existing
approaches and the right side highlights our proposed approach.
exist works that use both modality signals masked, they either use frozen object detector extract region-based visual features (Chen et al., 2020b; Li et al., 2020a; Lu et al., 2020; Su et al., 2019; Tan & Bansal, 2019) or mask the image tokens from a pre-trained image tokenizer instead of the raw RGB pixels (Dou et al., 2022; Fu al., 2021; Singh et al., 2022). These frozen object detector and image tokenizer not only require additional data for training but also prevent the V+L interactions from being learned end-to-end.
In this paper, we propose joint masked V+L modeling where the original signal is reconstructed by using its masked imput and the corresponding unmasked input from the other modality. As illus- trated in Figure (right part), although we exploit random masking, the dog face in the image can be used predict the masked text token "dog" and the text "green ball" can be used reconstruct the corresponding masked patches in the image. To ensure that the model uses information from both modalities, explicitly enforce the model utilize cross-attention to generate the joint represen- lations. Compared with the aforementioned existing works, our approach models both conditional distributions, TT] and PTT Also, the model trained end-to-end, without frozen bottleneck components that disturb learning interactions between V+L. By reconstructing one modality signal from the corresponding the other modality signal (e.g. reconstructing the text "dog" from the visual signals of dog face), the model implicitly learns the alignment between V+L. In addition, we ob- serve that the model trained for the joint masked V+L modeling becomes noticeably effective when the training data is limited. Overall, contributions are summarized as below:
1. We propose joint masked V+L modeling task. We show that models pre-trained with the proposed task, along with common V+L aligument tasks such as image-text matching, achieves state-of-the-ar performance
broad rage of V+L tasks.
2. We provide probabilistic interpretation of the proposed method and highlight the difference between ours and existing approaches terms of the V+L. joint distribution estimation.
3. We achieve significantly better performance than other V+L models in the regimes of limited training data, and only ~40% of data used by the state-of-the-att models is sufficient to match their performance.
2 RELATED WORK
Vision and language representation learning The methods in V+L representation learning can be categorized based on how the information is fused between the modalities to obtain the joint rep- resentations. We group the fusion techniques into three categories: 1) transformers with attention across modalities, 2) contrastive learning with large-scale pre-training data, 3) a hybrid form of learning with cross-attention and a contrastive loss. The attention across modalities has been widely used with image features extracted from thelh- object detectors and text features obtained from transformer encoders (Chen et al., 2020b; Li et al., 2020a: Lu al., 2020; Tan & Bansal, 2019; Zhang et al., 2021; Li et al., 2020b; Su et al., 2019; Li et al., 2019). While cross-attention effec- tively aligns V+L representations, it computationally expensive since all possible pairs of images
2

---

Published as a conference paper at ICLR 2023
Image Encoder (fx)
Image Gross modality Encoder (stm)
Image Cross-modallty Decoder (oth))
Pm
1m
Reconstruction
'A (MASK] rumning towards green ball
'Adog is running lowarde green bail
Text
Text Cross-modallty Encoder (irrs)
Token Classifier (a
running towards grean
Encoder
Unze)
Wy
ball
Figure 2: A framework of joint modeling of masked vision and language. The blue and green lines demonstrate the information flow for image and text reconstruction, respectively. The dotted lines indicate the cross-modal input of unmasked signals for generating attention.
Image (text) Encoder
Image
Crose-modally Encoder
Self- attention block
Feed-forward
Cross-
feed-forward
attention block
Self-attention
Self-attention
(a)
(b)
Figure Visualization of image (text) encoders and image (text) cross- modality encoders.
and texts need to be processed. On the contrary, the authors in (Jia al., 2021; Radford et al., 2021: Mokady et al., 2021; Shen et al., 2021; Yuan et al., 2021) show that contrastive learning with uni-modal encoders and millions of image-text pairs achieve powerful zero-shot performance in diverse
The contrastive learning-bass approaches do not rely on computationally expen- sive cross-attention but require excessively large amount of training data. Hence, combination of contrastive loss and cross-attention is utilized by complementing limitations of both approaches in (Li al., 2021: 2022; Yang al., 2022; Duan al., 2022). In particular, only image and text pairs that result in high similarity by uni-modal encoders are processed using the cross-attention layers to reduce the computational burden and improve the alignment.
Masked signal modeling is a commonly used pre-training objective in the aforementioned V+L models. It has been independently explored in each of vision and language domain. In the NLP domain, BERT and its variants (Devlin et al., 2018: Liu et al., 2019) achieve representations that can generalize to broad range of NLP tasks through MLM. Autoregressive language models (Radford al., 2018: 2019) which predict masked future tokens have shown to effective self-supervised learners. The success of the language models leads to several MIM techniques. BeiT (Bao et al., 2021) is trained to recover masked visual tokens which are obtained by a discrete variational autoen- coder (dVAE). In SimMIM (Xic et al., 2021) and MAE (He et al., 2022), transformers to recover masked patches an end-to-end fashion. The authors (Chen et al., 2020a) propose to autoregressively predict the unknown pixels to learn visual representations. In (Bachmann al., 2022), multiple vision modality data are masked and reconstructed to learn visual representations. In the domain of V+L learning, (Arici al., 2021) explores MIM and MLM for catalog data with short text attributes. V+L models with an object detector often aim at recovering only bounding box visual features (Chen al., 2020b; Li al., 2020a; Lu al., 2020; Tan & Bansal, 2019; Su et al., 2019). Several V+L models focus on predicting future text tokens without MIM (Wang et al., 2021: Yu et al., 2022; Alayrac al., 2022). While both MIM and MLM are explored (Geng al., 2022), the trained model is evaluated only on vision tasks. In (Dou et al., 2022; Fu et al., 2021; Singh et al., 2022: Wang et al., 2022), image tokens defined by image tokenizers trained with addi- tional 250 million images (Ramesh et al., 2021) or distillation from the CLIP model (Radford al., 2021) trained with 400 million image-text pairs (Peng et al., 2022) are reconstructed. eliminate these model and data dependencies, by directly recovering RGB pixels and text tokens from masked image patches and masked text tokens. Therefore, MIM and MLM are seamlessly integrated achieve generalizable V+L representations within simple training framework.
trained
our work,
METHOD
Our method has two types of pre-training objectives, which are 1) masked vision and language modeling and 2) multi-modal alignment. We explain each pre-training objective in this section.
3

---

Published as a conference paper at ICLR 2023
3.1 MASKED VISION AND LANGUAGE MODELING
The overall framework of masked vision and language modeling is shown in Figure 2. We use transformer- baasdddd encoders (Vaswani al., 2017) for both image and text streams. Given an image- text pair (I, T) an image encoder, Sim. is used to extract features, = (Uets ..... uN}, from the image input I. is the number of image patches and Uds is the encoding of the image class token, [CLS]. The text encoder, ftut, extracts features, input, T. .M is the number of text tokens and Wets is the encoding of the start token of a sentence, [START]. The image and the text encoder consist of 12 self-attention blocks as shown in Figure 3 (a). The image and the text features are further processed by image and text cross modds-ty encoders. The cross- modality encoders have 3 cross-attentin blocks as illustrated in Figure (b). The image (text) cross-modality encoder uses text (image) features to generate attentions. These cross-modality encoders can enhance the representation of one modality by interacting with another modality (Lu et al., 2020; Tan & Bansal, 2019).
.... WMM ), from the text {Wetss wi.....
For text masking. we follow BERT (Devlin et al., 2018) with minor modifications. In BERT, the original tokens are replaced with either the [MASK] token or random tokens. We use only the [MASK] token to replace tokens to be masked (Wettig et al., 2022). For image masking, we follow (He et al., 2022; Xie et al., 2021) and use random masking of raw image patches with a masking patch size of 32 x 32. Given that 224 224 images are divided into 16 16 patches for the image encoder, the large masking patch prevents the model from simply copying their neighborhood pixels for reconstruction (Xie et al., 2021).
Image and Text Masking:
We reconstruct the original signals of one modality from its masked input conditioned on the unmasked input of the other modality. Specifically, an original image. I, and a masked text, Tm. are used to reconstruct an original text, T, are used to reconstruct the original image, I. For image reconstruction, (Im. T) is first given to the image and the text encoders to obtain masked image features, Um. and unmasked text features, w. Following (Xie et al., 2021), we use both masked and unmasked patches to obtain Um- (Um, w) are further processed by the image cross-modality encoder, Jim where is used to compute cross-attentions. The output of gim is mapped back to the original RGB image space by an image cross- mods-ito decoder, gim which consist of cross-attention blocks followed by a fully connected layer (FC). Although existing work exploits a light-weight transformer decoder with only self-attention (He et al., 2022) or a simple linear mapping (Xie et al., 2021) for the image decoder, we use joint information between modalities to allow further interactions in decoding. For masked text reconstruction, a token classifier, gizt. which consists of a FC followed by softmax is applied to the output of the text cross-modality encoder, Jrnt, for the token prediction. The masked V+L modeling loss, LM LM, is defined as
Joint Reconstruction:
original text, T,, and similarly masked image, Lm. and
LMVLM = EUIDD Hly M Di (I, M(TarT 11
Dim (Jm, 1)
(I)
MLM
MIM
where dtat = -izz (gtwe (fim (T), fizt (Tm))) and dim- = de (gim (fom (Im), ftwe (T))). The loss is computed only for masked pixels and text tokens. Hence, the superscript M denotes data or features correspond to the masked signals. A pair of and T is sampled from the training dataset D, H M denotes cross-entropy, and yr is matrix that contains one-hot row vectors for the ground truth of masked text tokens. 22(-) is the number of pixels. When minimizing LMVLM, the model is enforced reconstruct the original signals by attending to the other modality signals. Cross-attending for reconstruction enables the model to learn interaction between V+L modalities.
3.2 MULTI-MODAL ALIGNMENT
In addition to the masked signal modeling tasks, we adopt two additional tasks to explicitly learn multi-modality alignment. The first one is an image-text contrastive (ITC) learning (Radford et al., 2021; Jia et al., 2021). For the k-th pair of image and text features out of the image and text encoders, two separate FC layers are used to project the image [CLS] token features and the text [START] token features to the same dimensional feature space with unit norm, 2 and 2 respectively. The loss. LITC is computed as
exp(2 =tas/T) EN ( 2b/e/ N exp(2%% zt =
exp(=ttm =b/a/T)
Σ k=1
Lire = N
log
+ log
(2)
4

---

Published as a conference paper at ICLR 2023
where N and are the batch size and the temperature scaling parameter, respectively. The second task is an image-text matching (ITM) (Chen et al., 2020b; Li et al., 2021; 2020b), predicting whether an image and text are aligned or not. The [CLS] ] and [START] token features from the image and text cross-modality encoders are iim and zoose respectively. To fuse these two features, we compute the element-wise product of z and zoros Etat (2 Ztst 2 and FC layer followed by softmax is applied to obtain the final prediction (Lu et al., 2019). For training, we use VITM = 1, when zrress and cross are pair. Otherwise, VITM 0. The loss, LITM. is defined as Fixt
zim
CITM = BU..T)-)(H((T) M, gft (2mm *2axt *xfr=*))]
(3)
Following (Li et al., 2021), sample in-batch hard negatives based on the distribution of the cosine
similarity between 2im and ztzt- The overall pre-training loss, L, is defined as L = LasV LM + LITO LrrM- We term our model trained with loss L as Mask (Masked Vision and Language Modeling).
3.3 PROBABILISTIC INTERPRETATION
We differentiate VL. from the existing V+I. models using masked signal modeling from a perspective of likelihood estimation. The training objective of masked signal modeling on uni- modal signals, X, focuses on learning the data distribution p(X) which is formulated by the law of total probability as p(X)= =xx., p(Xm) p(X] where X.m is an instance of masked signal from the set of all possible masked signals, .Mx. MIM or MLM learns the data distribution by maximizing ExmeMx p(X]Xm) (Bengio et al., 2013).
In V+L representation learning, the ultimate goal is to learn the joint distribution of multi-modal signals, T), However, the authors in (Sohn et al., 2014) pointed out that directly maximiz- ing the likelihood for the joint distribution is challenging because of the heterogeneous multi- modal data distributions. Instead, they show minimizing variation of information defined as -E (log p([[T) + ((T)() is sufficient to estimate the joint distribution. From a per- spective of variation of information, the limitations in existing works can be better understood. Several existing works attempted to approximate the joint distribution using MLM with unmasked image (Duan al., 2022; Li et al.. 2021; 2019; Yang al., 2022). In other words, p(T),, Tm) is maximized to learn the conditional distribution, p 77) but p(7(T) is not modeled, In other existing works (Chen et al., 2020b; Li et al., 2020a; Lu et al., 2020; Su et al., 2019; Tan & Bansal, 2019), where both modalities are masked, the visual masking is limited to mask the visual features extracted from a frozen object detector, v/(.), instead of the raw image pixels. In this case, the distributions (()(T) and ((])) are modeled instead of (T) and p(T))). This frozen feature extractor can bottleneck the direct estimation of the underlying data distribution. Mask is trained end- to-end to estimate both conditional distributions, p( (T) and p(T( which directly minimizes the variation of information. We hypothesize this modeling of conditional distributions for both modali- ties could lead to superior performance in both large-scale and limited data training scenarios, which we empirically demonstrated in Section 4.
4 EXPERIMENTS
4.1 PRE-TRAINING DATASETS AND DOWNSTREAM TASKS
We use the union of four datasets for pre- pre-training so that we can perform fair comparison with existing state-of-the-art methods (Chen et al., 2020b; Li et al., 2021). These datasets are Concep- tual Captions (CC) (Sharma et al., 2018), SBU Captions (Ordonez et al., 201 I), Visual Genome (VG) (Krishna et al., 2017), and COCO Captions (Lin et al., 2014). While VG and COCO contain captions annotated by humans, CC and SBU Captions are automatically collected from the web. The total number of unique images and image-text pairs in the four datasets are 4.1M and 5.2M, respectively. We term this pre-training dataset as the 4M dataset. We validate the pre-trained model on the following four downstream tasks:
Image-Text Retrieval: We perform text-to-image and image-to-text retrieval. We use the ITC and ITM losses of Section 3.2 for finetuning and the finetuned models are evaluated on COCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015). In addition, since COCO is used for pre- training, zero-shot retrieval performance is reported on Flickr30k. In (Li et al., 2021), the model finetuned on COCO is used for the zero-shot evaluation on Flickr30k. Although it may result in better perfor- mance, believe that using the finetuned model does not validate the zero-shot capability of the
5

---

Published as a conference paper at ICLR 2023
MSCOCO (5K)
Flickr30k (IK)
Method
# imuges
Image Retrieval
Text Retrieval
Image Retrieval
Text Retrieval
R@1 I
R@5
R@ 10
R@1 1
R
R 10
R@1 I
R@5
RØ 10
R@1
RØ5
R
ImageBERT (Qi al., 2020)
6M
50.5
78.7
87.1
66.4
89.8
94,4
73.1
92.6
96.0
87.0
97.6
99.2
UNITER (Chen et al., 2020b)
4M
52.9
79.9
88.0
65.7
88.6
93.8
75.6
94.1
96.8
87.3
98.0
99.8
VILLA (Gan et al., 2020)
4M
76.3
94.2
96.8
87.9
97.5
98.8
OSCAR (Li et 2020b)
4M
54.0
80.8
88.5
70.0
91.1
95.5
ALBEF
al., 2021)
56.8
81.5
73.1
91.4
82.8
96.7
94.3
99.4
Triple (Yang et al. 2022)
4M
59.0
83.2
89.9
75.6
92.8
96.7
84.0
96.7
98.5
94.9
99.5
99.8
Codebook (Duan et al., 2022)
4M
58.7
82.8
89.7
75.3
92.6
96.6
83.3
96.1
97.5 8
95.1
99,4
99,9
ALIGN (Jia al., 2021)
188
59.9
83.3
89.8
77.0
93.5
96.9
84.9
97.4
98.6
95.3
99.8
100.0
MaskVLM
4M
60.1
83.6
90.4
76.3
93.8
96.8
84.5
96.7
98.2
95.6
99.4
99.9
Table I: Comparison with finetuned state-of-the-an methods image-text retrieval. The gray row indicates that the model is trained with significantly larger number of data than MaskVL.M.
pre-trained model. Therefore, we use the pre-trained model directly for zero-shot evaluation. Fol- lowing (Li et al., 2021), we first retrieve top-k candidates using the similarity scores from the image and the text encoders. The top-k candidates are further processed by the cross- cross-modalin modalityy encoders to obtain the final retrieval results.
Visual Question Answering (VQA): Here, given an image and a question pair, the model should generate correct answer. The model is evaluated VQA v2 (Goyal al., 2017). We adopt the answer generation framework (Cho et al., 2021) and finetune the base model with a fusion encoder and an answer decoder. The model architectures are visualized in Figure 6 (a) of Appendix. The fusion encoder consists of one cross-attention block shown in Figure 3 (b). The output from the text cross-mo encoder is used as queries, and the image cross-modality encoder output is utilized to create attentions in the fusion encoder. The architecture of the answer decoder is the same as that of the text cross-modality encoder, but is trained with a language modeling loss to generate the answers. Specifically, the output of the fusion encoder is used for computing attentions and the answer tokens are autoregressively predicted. During inference, [START] token is used as an initial token to generate following answer tokens.
Natural Language for Visual Reasoning (NLVR): This tasks involves a binary classification with triplet, (text, imagel, image2). The goal here is to predict whether the text describes the pair of images. For finetuning, we feedforward (text, imagel) and (text, image2) separately to extract the features as shown in Figure 6 (b). The [CLS] token features of imagel and image2 from the image encoder are denoted as U1 and 02, respectively. The [START] token text features from the text encoder is w. These features are processed by the cross-modality encoders. The outputs of the image and text cross-modaliy, encoders are fused by element-wise multiplication. The fused features for both images are concatenated, and a classifier with two linear layers predicts whether the text is aligned with the image pair or not. NLVR2 (Suhr et al., 2018) is used for the evaluation.
Visual Entailment (VE): Given an image text pair, the task to classify the relationship between the image and the text into one of three categories: entailment, neutral, and contradictory. The element-wise product of the output from the image and the text mross-modality encoders forwarded to classifier of two linear layers for prediction. SNLI-VE (Xie et al., 2019) is used for evaluation.
4.2 IMPLEMENTATION DETAILS
We use a Visual Transformer (ViT) (Dosovitskiy et al., 2020) pre-trained on ImageNet (Deng et al., 2009) and a pre-trained RoBERTa from (Liu et al.. 2019) to initialize the image and the text encoder, respectively. We pre-train the model for 50 epochs when the 4M dataset is used and 30 epochs for all other experiments. A batch size of 512 used with 16 NVIDIA Tesla V100 GPUs All parameters are optimized using AdamW (Loshchilov & Hutter, 2017) with a weight decay of 0.05. Following (Xie et al., 2021), we use the image masking ratio of 60%. While 15% masking ratio is used for text in language models (Devlin et al., 2018; Liu et al., 2019), we use 30% since the paired image can provide additional information for text reconstruction. During pre-training, the learning rate is warmed up to 3 x 10 in the first 5 epochs and decayed to 3 10 using cosine scheduler. The learning rates for the image encoder and the text encoder are set to 10 which is less than that of the cross- modaaltyy encoders. An image size of 224 x 224 and RandAugment (Cubuk et al.,
6

---

Published as a conference paper at ICLR 2023
Flickr3kk (IK)
Method
images
Image Retrieval
Text Retrieval
R@1
R
10
R@s
10
ImageBERT (Qi al., 2020)
79.6
87.5
70.7
90.2
94.0
Unieoder-VL. (Li et al., 2020a)
3.8M
48.4
76.0
85.2
64.3
85.8
92.3
ViLT (Kim et ul., 2021)
4M
55.0
82.5
89,8
73.2
93.6
96.5
UNITER (Chen el al., 20220b
4M
66.2
88.4
92,9
80.7
95.7
98.0)
ALBEF (Li
2021)
68.2
88.6
93.0
84.9
97.2
99.0
FLAVA (Singh al., 2022)
68M
65.2
89.4
67.7
94.0
CLIP (Radford et al., 2021)
400M
68.7
90.6
93.2
88.0
98.7
99.4
ALIGN (Jia et al., 2021)
1.8B
75.7
93.8
96.8
88.6
98.7
99.7
MaskVL.M
4M
75.0
92.5
95.8
87.0
97.9
99.3
Table 2:
Zero-shot image-text retrieval performance on Flickr30k. The gray row indicates that the model is trained with significantly larger number of data than MaskVLM.
VQA
NLVR2
SNLI-VE
Method
images
test-dev
test
dev
test-P
val
test
VisuaBERTT (Li et al., 2019)
113K
70.80
71.00
67.40
67.00
VL-BERT (Su ul., 2019)
3.3M
71.16
LXMERT (Tan & Bansal, 2019)
180K
72.42
72.54
74.90
74.50
12-in-1 (Lu 2020)
3.3M
73.15
78.87
76.95
UNITER
al., 20206)
4M
72.70
77.18
77.85
78.59
78.28
VL-BARI/TS (Cho et al., 2021)
180K
71.30
73.60
ViLT (Kim al., 2021)
4M
70.94
75.24
76.21
OSCAR
2020b)
4M
73.16
73.44
78.07
78.36
VILLA (Gan et al., 2020)
4M
73.59
73.67
78.39
79.30
79.47
79.03
ALBRF al., 20221)
4M
74.54
74.70
80.50
80.14
80,30
Triple (Yang al., 2022)
4M
74.90
74.92
80.54
80.11
80.51
80,29
Codebook (Duan et al., 2022)
4M
74.86
74.97
80.50
80.84
80.47
80.40
FLAVA (Singh al., 2022)
68M
72.80
79.00
SinVLMaaze (Wang al., 2021)
1.BB
77.87
78.14
81.72
81.77
84,20
84.15
MaskVLM
4M
75.45
75.40
81.58
81.98
80.37
80.67
Table 3: Comparison with state-of-the-ar methods on VQA, NLVR2, and VE. The gray row indi- cates that the model is trained with significantly larger number of data than MaskVLM.
2020) are used. During finetuning, the image is resized 384 384 and the positional encoding is interpolated following (Dosovitskiy et al., 2020). More details can be found in Appendix.
4.3 EVALUATION ON IMAGE-TENT RETRIEVAL, VQA, NLVR, AND VE
We compare the finetuned image-text retrieval performance of the proposed MaskVLM with the state-of-the-arr methods in Table 1. The second column is the number of unique images used for pre- training and the retrieval performance is evaluated in terms of Recall @k (R@k) We do not directly compare with ALIGN (Jia et al., 2021) since it is trained with more than 300 times of data used for MaskVLM. However,
still highlight the small performance gap between Mask VLM and ALIGN. We achieve the best performance all Recall k metrics on both COCO and Flickr30k except for the image retrieval R @ 10 and text retrieval R@5 on Flickr30k. Compared to ALIGN, we even achieve higher R for image retrieval on COCO and text retrieval on Flickr30k. Table 2 shows the zero-shot retrieval performance of the state-of-the-ar methods on Flickr30k. MaskVLM achieves significant improvement over the second best method, ALBEF (Li et al., 2021), by 6.8 points at R @ for image retrieval. Given that ALBEF not trained for MIM, we hypothesize that ALBEF achieves the biased performance for text retrieval and Mask achieves the significant improvement in image retrieval by additional MIM which models p (T) While FLAVV exploits both MILM and MIM with the pre-trained image tokenizer, using 13 times more data than MaskVLM. LLM still outperforms FLAVA by 9.8 and 19.3 points at R I for image and text retrieval respectively. Compared with CLIP (Radford et al., 2021) which is trained with at least 76 times more data than MLs we still achieve higher R@ 1 for image retrieval by 6.3 points. In general, Mask achieves of of-te-of-the-att the-art performance in both finetuning and zero-shot experiments.
We report the accuracies on VQA, NLVR, and VE in Table 3. Except for Sim whose pre- training data more than 300 times larger than that of Mask VLM, we consistently achieve the best

---

Published as a conference paper at ICLR 2023
Dataset of samples)
Method
COCO IR
COCO TR
VQA
NLVR2
SNLI-VE
R
R@1
R
test-P
test
ALBEF
49.97
77.35
89.32
73.07
76.89
79.
79.19
CC 50% 00CO (2M)
Ours
56.36
81.98
73.22
92.00
74.24
79.81
79.47
79.69
79.50
25% (1.3)
ALBEF
48.09
75.63
64.24
87.20
72.65
75.20
76.78
78 78.84
78.96
Ours
55.11
81.28
72.18
91.48
74.17
79.13
78.94
79.35
79.76
CC 10% + COCO (0.9M)
ALBEF
73.57
84,98
72.14
74.81
74.64
78.51
78.36
Ours
54,04
80.74
70.04
91.24
73,93
78.62
77.19
79.11
79.60
Table 4: Downstream task performance with limited pre-training data.
performances in all these tasks except for the validation split of NLVR2. In particular, Mask better than the second best method by 0.43, 1.14, and 0.27 on the test splits of VQA, NLVR2, and SNLI-VE, respectively. Compared to the base model of Sim VLM, we narrow the accuracy gaps to 2.74% and 3.48% in test-std and test splits of VQA and VE, respectively. MaskVLM achieves higher accuracy than SimVLMpas in the test split of NLVR2 by 0.21%.
4.4 EVALUATION WITH LIMITED
DATA
We highlight the performance of MaskVLM limited data scenarios. In particular, we create three subsets of the 4M pre-traiming data by sam- pling 50%, 25%, and 10% of CC and combin- ing them with COCO. The number of image- text pairs in each subset is around 39%, 25%, and 16% of the 4M pre-training data which con- tain 5.2M pairs, respectively. We pre-train mod- els with these subsets of the data and analyze the downstream task performance in comparison with state-of-the-art methods. The results are re- ported in Table 4. Particularly, image and text re- trieval R@ performance on COCO is also visu- alized in Figure 4. We compare MaskVLM with
& 8 22 22
Becalgo
ALBEF
ALBE: Codkzook
Codebbook
B E.
MosMVLM
MosKVLM
2, of poited data (M)
2. of paired data (MI
Figure 4: R@1 plots for image retrieval (left) and text retrieval (right) on COCO using limited pre-training data.
the most recent state-of-the-art methods, which are ALBEF (Li et al., 2021) and Codebook (Duan et al., 2022). In Table 4, as the size of pre-training data becomes smaller from CC 50% + COCO
to CC 10% COCO, the performance gap between Mask VLM and ALBEF increases from 6.39 to 8.71 at R @ in COCO image retrieval (IR), 7.46 to 9.04 at R @ in COCO text retrieval (TR), 1.17 to 1.79 in VQA and 0.31 to 1.24 in the test set of SNLI-VE. In NLVR2 and VQA, MLsk trained with CC 10% + COCO achieves higher accuracy than ALBEF trained with CC50% + COCO, which contains more than twice of image-text pairs in CC 10% + COCO, In Figure 4, while Code- book shows competitive recall performance compared to the MaskVLM with the 4M dataset (5.2M pairs), the @@ differences in image and text retrieval, respectively, increase from 1.4 and 1.0 in the 4M dataset to 3.15 and 3.80 in CC 50% COCO. Our model trained with CC25% COCO outperforms Codebook trained with CC50% COCO by 1.90 and 2.76 points in terms of image and text retrieval R @ respectively. Since one of the main differences in MaskVLM compared to ALBEF and Codebook is the additional MIM, we believe that joint modeling of V+L contribute to better performance in limited data scenarios.
4.5 ABLATION STUDY
We perform an ablation study using different combinations of loss components to highlight the contribution of masked V+L modeling. We compare six models with the same architecture but with different loss functions for pre-training. We pre-train all models on the CC 50% + COCO dataset and compare finetuned and zero-shot retrieval performance on Flickr30k in Table 5. We note that zero-shot evaluation of the MLM + MIM model cannot be performed because the FC layers to compute ITM and ITC are not trained during pre-training. ITC and ITM are closely related to the retrieval task since they are used for finetuning and measuring the similarity between images and texts. However, MLM MIM still achieves significantly better finetuned and zero-shot performance than ITC. which shows that MLM + MIM alone learns meaningful V+L representations. Compared to ITC+ ITM in the finetuned performance, ITC ITM MLM achieves slightly improved R @
8

---

Published as a conference paper at ICLR 2023
Finetuned
Zero-shot
Loss
IR
TR
IR
TR
R@1
R@5
R@1
R@s
R@1 1
R@5
R@1
R@5
ITC ITC ITM MLM +MIM ITC + ITM MLM
65.10
89.88
80.10
96.90
55.08
80,90
68.40
90.00
79.96
95.56
92.30
98.90
69.50
89,54
82,40
96.60
76.08
94.40
90.30
98.80
80.34
95.82
92.00
99.30
70.74
90.92
84.40
97.10
ITC ITM + MIM
80.12
95.56
91.50
99.00
69.26
90.30
82.90
97.20
ITC + ITM MLM MIM
81.26
96.00
94.10
99.60
71.18
91.12
85.60
97.50
Table 5: Image-text retrieval evaluation on Flickr30k with different loss functions for pre-trainig pre-training.
Original
Masked
GT: boston terrier running lush green grass front while fence"
Masked: "[MASK] [MASK] [MASK] [MASK] [MASK] running on lush [MASK] [MASK] front
[MASK] [MASK] [MASK]]
Recon (mask): 'a brown of motion running on lush greens in front the green forest"
Recon (org): 'a brown and white dog is running on lush green grass in front of a white fence"
GT: three people in a cave Masked: three people [MASK] [MASK] [MASK] Recon (mask): 'three people sitting the row"
Recon (org): "three people standing in a cave"
Figure 5: Masked language modeling examples using masked and original images. "Recon (mask)" and "Recon (org)" denote reconstructed text from the masked image and the original image, respec- tively.
by 0.38 in image retrieval and degraded R@1 by 0.3 in text retrieval. When MIM alone is used with ITC ITM well, finetuned R@1 is improved by 0.16 and degraded by 0.8 for image and text retrieval, respectively, over ITC + ITM. On the other hand, when ITC + ITTM + MLM + MIM is used, the model achieves significant improvement of finetuned performance over ITC ITM MLM by 0.92 and 2.10 for R@] I image and text retrieval, respectively. ITC + ITM + MLM + MIM also obtains the best performance in zero-shot retrieval. This result further supports the advantage of joint modeling for masked V+L signals.
4.6 QUALITATIVE RESULTS
We perform a qualitative analysis to show the role of multi-modal information in the reconstruction of masked signals from our model. To be specific, we illustrate the prediction of masked text tokens with and without the corresponding images. This illustration highlights how MaskVLM effectively utilizes both modality information to complete the masked signal modeling task. Figure 5 shows the reconstruction of masked texts using original images ("Recon (org)") and masked images ("Recon (mask)"). In the first top example, when the model given masked text and a masked image which does not contain the "dog", the reconstruction is performed by using only available informa- tion such as image patches of "green grass". Thus, the prediction is limited to "a brown of motion" or "the green forest". However, when the original image is used for reconstruction, both "a brown and white dog" and "white fence" are reconstructed by accurately attending to the image. In the bottom example, the visible patches of the masked image contain few people, but lack background information. Consequently, the reconstruction with the masked image does not contain any back- ground information but the background "cave" is reflected in the reconstruction with the original image. Theses examples confirm that MaskVLM has learned to perform masked signal modeling using both V+L information.
CONCLUSION
We propose masked vision and language modeling as a pre-training task for learning V+L repre- sentations. We provide probabilistic interpretation to highlight the contribution of the proposed method and validate its advantages in both large-scale and limited data regimes. We consistently achieve the state-of-the-art performance in broad range of V+L tasks.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank Jiali Duan for providing results of the Codebook (Duan et al., 2022) with limited pre- training data in Figure 4.
REFERENCES
Jean-Bape Mon-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
Tarik Arici, Mehmet Saygin Seyfioglu, Tal Neiman, Xu, Son Train, Trishul Chilimbi, Belinda Zeng, and Ismail Tutar. Mlim: Vision-and and---an---aan lan--ugu model pre-training with masked language and image modeling. XrXiv preprint arXiv:2109 12178, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprini
arXiv:1607.064500 2016.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi- task masked autoencoders. arXiv preprint arXiv:2204.01678, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 20Xiv:2006.08254, 2021.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. Advances in neural information processing systems, 26, 2013.
Zhaowei Cai, Gukyeong Kwon, Avinash Ravichandran, Erhan Bas, Zhuowen Tu, Rahul Bhotika, and Stefano Soatto. X-detr: A versatile architecture for instance-wise vision-language tasks. arXiv preprint arXiv:2204.05622 2022.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691 1703. PMLR, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference computer vision, pp. 104-120. Springer, 2020b.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. International Conference on Machine Learning, pp. 1931-1942. PMLR, 2021.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icee, 2009.
Jacob Devlin, Ming- Heig-Wei&o Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048805 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Zi-Yi Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-ent vision- and-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 18166-18176, 2022.
10

---

Published as a conference paper at ICLR 2023
Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang. Yi Xu, Belinda Zeng, Chenyang Tao, and Trishul Chilimbi. Multi-modal alignment using representation codebook. arXiv preprint 2rXhv:2203.00048, 2022.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Vi- olet: End-to-end video-language transformers with masked visual-tokes modeling. arXiv preprint arXiv:2111. 1226811, 2021.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adver- sarial training for vision-and- -an-lang representation learning. Advances in Neural Information Processing Systems, 33:661 6-6628, 2020.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and Picter Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205. 2022.
Yash Goyal. Tejas Khot, Douglas Summers-Sture Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Chao Jia, Yinfei Yang, Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Aishwarya Kamath, Mannat Singh, Yann LoCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr-modulated detection for end-to-ent multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1790-1790, 2021.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantee alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1128-3137,20
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and- -nd--auuuagee transformer without convo- lution region supervision. In International Conference Machine Learning, pp. 5583-5594. PMLR, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsoureed dense image annotations. International journal of computer vision, 123(1):32-73, 2017.
Gen Li, Nan Duan, Yuejian Fang. Ming Gong, and Daxin Jiang. viicoder-vl: A universal encoder for vision and language by mrossl pre- mee-trannng.. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11336-1 1344, 2020a.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201. /2086, 2022.
11

---

Published as a conference paper at ICLR 2023
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: 2019.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang. Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semanties aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020b.
Tsung-YY Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / 1692, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05001, 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for aaiion-an-languaae tasks, Advances neural information processing systems, 32, 2019.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10437-10446, 2020.
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.097 2021.
Maria- Elena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv::2088 06366, 2022.
Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to to-phrase correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprint arXiv:2001.07900 2020.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition, pp. 413 IEEE, 2009.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp.
PMLR, 2021.
12

---

Published as a conference paper at ICLR 2023
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypermymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 2556-2565, 2018.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and aaiin-and-aanguaee tasks? arXiv preprint arXiv:2107.06388 2021.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Mar- cus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15638-15650, 2022.
Kihyuk Sohn, Wenling Shang, and Honglak Lee. Improved multimodal deep learning with variation of information. Advances in neural information processing systems, 27, 2014.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training 2019.
of generic visual-linguistic representations. arXiv preprint
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv: 1.00491, 2018.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. arXiv preprint arXiv: 1908.0099 2019.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, -122 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.1000 10904, 2021.
Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? arXiv preprint arXiv:2202.08005, 2022.
Ross Wightman. Pytorch image models.
pytorch- mmage-modedt 2019.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL https: /www. ac aclweb. org/ hagghhoohhooy/// 2020 emn 1p-dpmos 6.
13

---

Published as a conference paper at ICLR 2023
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-
grained image understanding. arXiv preprint arXiv:1901.06706. 2019.
Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:2111. 11098866 2021.
Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision- lasoualanguaae pre-training with triple contrastive learning. arXiv preprint arXiv:2202. 220.:20011, 2022.
Yi Yang and Shawn Newsam. Bag-of- wwaf-vvsual-wwote and spatial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, pp. 270 279,-279.120 2010.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding, Advances in neural information processing systems, 32, 2019.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint 2rXi::205.001117 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A foundation model for computer vision. arXiv preprint a111 2021.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 5579-595555 5588, 2021.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DETAILS ON FINETUNING FOR DOWNSTREAM TASKS
We explain implementation details for each of the downstream tasks. For all the downstream tasks, we use AdamW (Loshchilov & Hutter, 2017) with a weight decay of 0.05 and the cosine scheduler. An image size of 384 384 with RandAugment (Cubuk et al., 2020) is utilized and the positional encoding is interpolated following (Dosovitskiy et al., 2020). Except for the VQA task, we use the model achieves the best performance the validation set to report the performance on the test set. We use the last epoch model for the VQA evaluation.
Image-Text Text Retrieval: COCO (Lin et al., 2014) and Flickr30k (Plummer et al., 2015) are used to report the performance. To be specific, we follow data splits proposed in (Karpathy & Fei-Fei, 2015) and an average recall over image and text retrieval is used to find the best model in the validation set. The pre-trained model is finetuned for 15 epochs with batch size of 256 and learning rate of -er-pe--ttraee x 10
Visual Question Answering (VQA): For fair comparison with existing methods (Chen et al., 2020b; Li et al., 2021), we use training and validation sets from VQA v2.0 (Goyal et al., 2017) with subset of VQA samples from Visual Genome (Krishna et al., 2017) for training. Also, we report performance on both test-dev and test-std splits of VQA v2.0. Following (Li et al., 2021), we weight the loss for each answer based on its occurrence among all the answers. The model is finetuned for 15 epochs with batch size of 256 We use learning rate of 2 x 10-6 for the image and the text cross- cross-modast mosaliy encoders, the fusion encoder, and the answer decoder. For the image and the text encoders, a learning rate of 1 x 10-5 is used, The fusion encoder and the answer decoder are initialized by the last and all three blocks of the pre-trained text cross-modality encoder, respectively.
Natural Language for Visual Reasoning (NLVR): Data splits proposed in (Suhr et al., 2018) are used for finetuning and evaluation. The model finetuned for epochs with batch size of 128. Since the classifier is newly added after finetuming, classifier and 10-5 for the remaining parts of the model. Different from (Duan et al., 2022; Li et al., 2021; Yang et al., 2022), where the models require additional text-assignment pre-training step before finetuning, we directly finetune for simplicity.
use learning rate of 1 x 10-4 for the
Visual Entailment (VE): We follow data splits proposed in SNLI-VE (Xie et al., 2019). We finetune the model with a batch size of 256 for 5 epochs. Similar to the NLVR task, a learning rate of 10-4 is used for the classifier and x 10-5 is used for the remaining parts of the model.
Image
features
Image festures
Crose-nidallty Encoder (gtw)
Pi
Image Cross moddlly
cross
"True" "Faise'
F22
Encoder (stes)
Answer
Question fealures
Classifier
Text features
Text Crose-modallyy Encoder (Pize)
Fusion Encoder
Answer Decoder
Text
W
Cross-modallty Encoder (sxx)
CreAR
ZEEE
[START]
(a) VQA
(b) NLVR
Figure 6: An illustration of model architectures for VQA and NLVR.
A.2 REPRODUCIBILITY
We add more details of MaskVLM for reproducibility. We used the ImageNet pre- trained VIT (vit.base.patchle 422 from (Wightman, 2019) and the pre-trained RoBERTa (roberta-base
from Hugging Face (Wolf et al., 2020). The detailed model architectures are visualized in Figure 7. Following (Dosovitskiy et al., 2020), the image encoder uses layer normal- ization (Ba et al., 2016) before each multi-head attention block while the text encoder applies layer normalization after each multi-head attention block (post norm). For the image (text) cross- modality encoder, we adopt the post norm and use the outputs of the text (image) encoder as keys and values
15

---

Published as a conference paper at ICLR 2023
Add & Norm
Feed Forward
Add Norm
Feed
Mult-Head Atlention
X
Nomm
Add & Norm
Feed Forward
q
(t)
x 12
Multi-Hoad Attention
Norm
x12
Add & Norm
Multi-Haad Attention
Multi- Attention
kf
F'm (Wm)
(b)
(c)
Figure 7: Model architectures of (a) Image encoder, (b) Text encoder and (c) Image (text) cross-
modality encoder. The dotted lines in (c) denote key and value from the other modality for cross- attention.
MSCOCO (5K)
Flickr30k (IK)
Method
Image Reirieval
Text Retrieval
Image Retrieval
Text Retrieval
R@1
R@5
R@ 10
R@1
R@S R@10
R@I
R@S
R@10
R@1
R@5
R@ 10
ALBEF
56.8
81.5
89.2
73.1
91.4
96.0
82.8
96.7
98.4
94.3
99.4
99.8
MaskVLM (both)
59.5
83.4
90.2
76.0
93.4
96.9
83,9
96.6
98.3
95.1
99.9
100.0
MaskVLM (one)
60.1
83.6
90.4
76.3
93.8
96.8
84.5
96.7
98.2
95.6
99.4
99.9
Comparison of finetuned MaskVLMs with different masking strategies and ALBEF on image-text retrieval. (MaskVLM (One): masking one modality at a time for computing MLM and MIM losses. VLs (both): masking both modalities at the same time for reconstruction)
Table 6:
to compute cross-attention. To compute MIM and MLM, the self-attention outputs of the masked image features, Um. is used as queries and the unmasked text features, w, are used as keys and values in the image cross cross-modase modality encoder. For the text mross-modality encoder, the masked text features, Wm, are used as queries and the unmasked image features, v, are used as keys and values. To keep the framework simple, do not any loss weighting for each loss term and layer decay during finetuning.
A.3 ABLATION STUDY ON MASKING STRATEGIES
We study different masking strategies in computing MIM and MLM losses. In particular, we com- pare MaskVLM using one modality masked and the other modality unmasked for reconstruction (MaskVLM (one)) and Mask using both modalities masked at the same time for reconstruc- tion. We compare these two MaskVLM models with the state-of-the-art method, ALBEF in Table 6. We follow the experimental setup described in Section 4.1 and report the finetuming performance on image-text retrieval. The performance of masking one modality slightly better than masking both modalities at the same time (MaskVI.M (both)). However, we observed that both reconstruction strategies are still effective as they achieve higher R@1 for image and text retrieval in both COCO and Flickr30k compared to ALBEF.
time (MasVVLM (one)) was
A.4 ABLATION STUDY ON MASKING RATIO
We perform ablation study using different masking ratios for masked vision and language modeling. In particular, we pre-train MaskVLM with several combinations of image and text masking ratios on the CC 50% COCO dataset and report the finetuned image-text retrieval performance on Flickr30k in Table 7. We also report an average of R@k for image and text retrieval. When only image masking ratio is changed in the first three rows of the table, the difference between the maximum and the minimum of the average recall is 0.26 for image retrieval and 0.10 for text retrieval. This shows that MaskVLM achieves stable performance across the tested image masking ratios. From
16

---

Published as a conference paper at ICLR 2023
comparison between the second row and the last observe that increasing the text masking ratio from 0.15 to 0.3 leads to higher recall performance for both image and text retrieval.
Masking ratio (image / text)
Image
Retrieval
Text Retrieval
@
R@5
@@ 10
Average
R@
R R@5
R@ 10
Average
0.3 0.6 0.3
81.32
96.04
97.92
91.76
93.30
99.70
100.00
97.67
81.26
96.00
97.78
91.68
94.10
99.60
99.60
97.77
0.7 0.3
81.82
96.00
98.00
91.94
93.60
99.50
99.90
97.67
0.6 0.15
80.30
95.66
97.82
91.26
92.50
99.10
99.60
97.07
Table Finetuned image-text retrieval performance on Flickr30k with different masking ratios for masked vision and language modeling.
A.5 EVALUATION ON IMAGE RECOGNITION
We evaluate the image recognition performance of Mask Following CLIP (Radford et al., 2021), perform image classification directly using the pre-trained MaskVLM various image recognition datasets including UC Merced Land Use (Yang Newsam, 2010), MIT-67 (Quattoni & Torralba, 2009), CUB-200 (Wah et al., 201 1), Oxford Flowers (Nilsback & Zisserman, 2008), Caltech-256 (Griffin et al., 2007), and ImageNet-1K (Deng et al., 2009). We compare the Top- 1 accuracy of MaskVLM with ALBEF (Li et al., 2021) in Table 8. Both models pre-trained with the 4M dataset. Since during pre-training stage, both Mask VLM and ALBEF were initialized with the ImageNet pre-trained weights, the evaluation evaluation on other datasets is. We formulate image classification as image-to-text retrieval where the similarity scores between query image and all the class names are computed to retrieve top-1 class name. The similarity scores can be obtained using either ITC and ITT scores, and we report them separately. Also, the results of using prompt engineering as in CLIP are reported.
ImageNet is not strictly zero-shot but the
As shown in Table 8, MaskVLM consistently outperforms ALBEF across all the datasets. In par- ticular, prompt engineering improves the average accuracy across the datasets for MaskVLM but ALBEF achieves lower average accuracy with prompt engineering. This shows that MaskVLM can better align images with variants of text than ALBEF, which results in stronger V+I. V+I. representations of MaskVLM for the image recognition task.
Prompt Engineering
UC Mereed Land Use
Oxford Flowers
Method
MIT-67
CUB-200
Caltech256
ImageNet- IK
Average
ALBEF (TTC)
31.62
52.46
5.78
26.93
45.75
36.28
33.14
ALBEF (TTM)
26.29
56.19
5.68
21.68
41.79
35 35.41
31.17
ALBEF (ITC)
28.38
51.04
5.33
25.44
48.71
30.26
31,53
ALBEF (ITM)
15.81
29.70
3.90
18.86
26.57
12.53
17.90
MaskVLM (TTC)
41.52
55.60
4.97
26.61
60.68
33.16
37.09
MaskVLM (TTM)
29.52
58.21
10.36
36.14
58.95
34,59
37.96
MLs (ITC)
45.14
66.27
4.37
60,95
39.16
41.43
MaskVLM (ITM)
40.76
63.36
8.53
44.53
61.77
39.71
43.11
Table Top-1 accuracy of pre-trained MaskVLM and ALBEF on image recognition. ITC and ITM denote the alignment scores utilized to perform image classification.
A.6 ADDITIONAL EXAMPLES FOR THE QUALITATIVE ANALYSIS
We present additional examples for the qualitative analysis of MaskVLM in Figure 8. Similar to Figure 5, masked text tokens reconstructed with masked images ("Recon (mask)") and original images Reecoon (org)"). We highlight that MLM utilizes both V+L information to reconstruct the text which corresponds to the given image.
17

---

Published as a conference paper at ICLR 2023
Original
Masked
"people are fixing
house"
Masked:
"people are [MASK] [MASK] [MASK] of [MASK]*
Recon (mask): "people the wing of tree"
Recon (org): "people are working the roof of house"
ride her shoulders" Masked: young [MASK] is giving [MASK] [MASK] [MASK] ride on her [MASK]'
GT: "a young woman giving
Recon (mask): "a young girl is giving theons ride on her horse"
Recon (org): young mother giving baby
shoulders'
GT: 'a girl in a jean dreas is walking along a raise balance beam"
Masked:
jean [MASK] [MASK] along raise [MASK] [MASK]'
Recon (mask): 'a girl in a jean house mirrored along raise pink
Recon (org): girl in jean dress walking along raise wooden beam"
GT: 'a young girl wearing blue shirt marching band playing trumpe!" Masked: "a young girl wearing blue [MASK] marching [MASK] band [MASK] a [MASK]] Recon (mask): "a beautiful girl wearing blue hat marching in band of wing"
Recon (org): young girl wearing
marching
field"
Figure 8: Additional masked language modeling examples using masked and original images. "Re- con (mask)" and "Recon (org)" denote reconstructed text using the masked image and the original image, respectively.
A.7 STATISTICS OF THE PRE- TRAINING DATASET
In Table 9, we report the statistics of the 4M pre-training dataset that Mask is trained on. We note that some data urls provided in the web datasets become invalid, which may lead to slightly different number of image-text pairs depending on when the datasets are downloaded.
Dataset
# of image-text pairs of images
COCO
566,747
113,287
CC
2,912.317
2,912.317
SBU
1,000,000
1,000,000
VG
768,536
100,406
Total
5,247,600
4,126,010
Table 9: Statistics of the 4M pre-training dataset.
18


Reviews:
[{"review_id": "lNWQCGQ4UK", "reviewer": "Reviewer_kwqv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose masked language and image modeling to pre-train image and text representations for downstream tasks such as image-text retrieval and various image-language reasoning tasks (VQA, NLVR, visual entailment). The proposed model is trained on paired image-text data, where one modality is masked while the other is kept unmasked. Given these two inputs the goal is to reconstruct the masked modality. In addition, a contrastive text-image loss (like CLIP) and an image-text matching loss are used.\n\nThe authors compare against a multitude of baselines on the tasks, which for the wide majority they beat. In addition, they show that their method works better in the low-data regime, probably because of the additional masked image modeling objective. The losses in the model are then ablated on the image-text retrieval text and it is shown that adding MIM and MLM improves the model. Finally, qualitative results are presented showing how additional cross-modal information helps prediction. Ablations on masking strategy and ratios are given in the appendix.", "summary_of_the_review": "This paper presents a somewhat novel method for image-language pre-training and shows that it beats a large number of baselines. The problem is that it omits quite a few recent or concurrent baselines in the experimental and related work section. The claims of SOTA in the paper then seem somewhat dubious to me. See the weaknesses section for more details.", "strength_and_weaknesses": "Strengths:\n\nThe authors compare against a large number of baselines, and are able to show better performance in almost all instances.\n\nQualitative results as well as ablation of the objectives for pre-training provide compelling evidence that cross-modal masked modeling can help to produce better representations.\n\nWell designed experiment shows that in the low-data regime, adding a masked image modeling loss can improve performance by making more efficient use of the data.\n\nGood ablation studies of masking ratios and strategies in the appendix.\n\nWeaknesses:\n\nOne significant problem with the paper is that many recent and concurrent methods and references are missing. As a result, many of the tables that claim to compare MaskVLM to SOTA results do not actually:\n\n- Table 1 lists SOTA methods on MSCOCO and Flickr30k image-text retrieval with finetuning. It is however missing Florence, which beats all methods in the table.\n- Table 2 lists SOTA methods on zero-shot Flickr30k image-text retrieval. It is also missing Florence, and CoCA.\n- Table 3 lists SOTA methods on VQA but is missing the above methods and SimVLM and Flamingo.\n\nIf the claim is that MaskVLM achieves SOTA with specific qualifications (such as pre-training on a smaller dataset) this should be stated and shown explicitly. If the claim is that MaskVLM achieves SOTA outright, then I don't think this is true.\n\nSome papers that should be cited:\n- Florence\n- CoCA and SimVLM\n- Flamingo\n- MultiMAE\n- M3AE (could be optional I think)\n- BEiT-3 (this came out after ICLR submission deadline, but is relevant)\n\nIn addition, claims such as \"In the domain of V+L learning, there exist limited works on end-to-end joint modeling of V+L signals\" are untrue, in light of the above references.\n\nThe method and its presentation are also a tad complicated. It could help significantly if the explanation of the method was cleaned up a bit. Here are some things that I found confusing:\n\n- The first term in Eq. 1 appears to show the MLM loss is applied to unmasked tokens as well as masked tokens, as opposed to just masked tokens as in BERT. Is this a typo, or actually the case? And if so, why was this decision made?\n- In Eq. 1, there is an expectation over D on the second term but not the first. I'm assuming this is a typo? \n- One more nitpick for eq 1: I think $ g_{im}^{de} $ in the definition of $ \\phi_{im} $ should also take $ f_{txt}(T) $ as an argument, as the image decoder is also allowed to cross-attend to vision text information.\n- Why use an L1 loss for MIM rather than L2 like in MAE?\n- In order to fuse representations, element-wise multiplication is used (in the downstream task heads, and the ITM loss). This seems non-standard to me, so perhaps a reference to previous work that does this could be helpful.\n\nOne small nitpick: MAE is cited as a preprint, but it was published at CVPR 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kOu6X6hg_b", "reviewer": "Reviewer_RfUs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple multimodal vision and language mask model for multimodal training, called MaskedVLM. The main idea is to leverage both masked image patches with full text description and masked language tokens with full image patches for semantic align. \nAdditionally, the method employs two pretraining losses: image-text matching and CLIP-style image-text contrastive losses. \nAfter pretraining with 4M data, they evaluated the proposed MaskVLM on 4 standard multimodal downstream tasks: I2T, T2I, VQA, andMultimodal NLI. Experimental results look promising compared to other baseline methods and include ablation studies. ", "summary_of_the_review": "Overall, this paper proposes a simple but effective method and is easy to read. On the other hands, this paper has limited novelty and missed some important related work. If the author can address my concerns, I am willing to raise my score. I look forward to the author response. ", "strength_and_weaknesses": "### Strength\n* Both modality masking idea is simple and seems effective even if the idea is not very novel. \n* Experimental results seem promising with significant margins compared to the baseline methods. \n* Under limited pretraining data, this method show promising and competitive performances. This is meaninful for academic or small-scale industry research groups. \n* Overall, the paper is clear and easy to read. \n\n### Weakness\n* [Major] Even if the main idea is clear and simple, most of them are from other previous work such as MLM, MAE, and MIM. Of course, the novelty is not all. However, for technical contribution, the combination or integration of the previous ideas needs to be not trivial and challenging, considering the nature of ICLR. Also, the authors need to describe how to effectively integrate in details. Unfortunately, I could not find the details of their proposed method. For example, there is no detailed implementation on how to make joint integration for image decoder. Figure 2 did not show the details. Figure 3 has no architecture of image cross-modal decoder. If the page limitation is an issue, the author can use the appendix. \n* [Major] Some important related previous works are missed such as CoCa [Yu et al. 2022], SimVLM [Wang et al. 2022a], Florence [Yuan et al. 2021], and BEiT-3 [Wang et al. 2022b]. Among them, the author need to compare their method to SimVLM with discussion even if the pretraining dataset is not same. For the rest, the author need to introduce them in related work at least.\n* [Major] For limited dataset experiments, how is the pattern on more data? I wonder the performance of MaskVLM on larger data including CC12M. For example, ALBEF presented two versions such as 4M and 14M. I think the contribution of MaskVLM can be enhanced with the same setting of ALBEF-14M. That is, it will be meaninful under 14M setup for Figure 4. \n* [Minor] How is the performance on image recognition of this method, such as ImageNet-1k fine-tuning? Of course, image recongnition performance is out of scope of this paper but the comparable performance on image recongnition can improve the contribution of this paper. \n*  [Minor] If I correctly understand, the training data are doubled due to the proposed mask approach. How is the performance on the same computing cost?\n\n### References\n* [Yu et al. 2022] [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://arxiv.org/abs/2205.01917). arXiv:2205.01917\n* [Wang et al. 2022a] [SimVLM: Simple Visual Language Model Pretraining with Weak Supervision](https://arxiv.org/abs/2108.10904). ICLR 2022.\n* [Yuan et al. 2021] [Florence: A New Foundation Model for Computer Vision](https://arxiv.org/abs/2111.11432). arXiv:2111.11432.\n* [Wang et al. 2022b] [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://arxiv.org/abs/2208.10442). arXiv:2208.10442.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HAyHudBxRb", "reviewer": "Reviewer_UZ4d", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the use of joint masking for vision and language for learning representations from text and images. The proposed approach is tested on multiple tasks and achieves state-of-the-art results especially in the case of limited training data.", "summary_of_the_review": "A very interesting contribution with several experiments and a detailed ablation study.", "strength_and_weaknesses": "Strengths\nThe paper is well written and easy to follow. \n\nAn interesting approach based on jointly masking images and text for learning visual and text representations is presented.\n\nExperiments on multiple tasks and datasets are presented. A detailed ablation study is also conducted.\n\nWeaknesses.\n\nIn several occasions the authors claim that other methods have been trained on much larger datasets. As a consequence this makes comparison with these works problematic. Is it possible that these methods are trained on the same amount of data as the one used in this study? Alternatively, would it be possible that the proposed method is trained on much larger datasets?\n\nIn order to investigate the impact of the each loss term it would be desirable that only one of them is removed at a time. In other words, it would it would be very informative to also include results with the following losses (in Table 5): MLM + MIM + ITC and MLM + MIM + ITM.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DbXrhrLrMw", "reviewer": "Reviewer_cmqf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers training vision-language models by masked reconstruction. Specifically, the model is trained to reconstruct a masked image conditioned on the paired text, and vice versa. In addition to this task, the standard contrastive and ITM tasks are added during pretraining. The authors use public small-scale VL datasets for pretraining and beat baseline methods on retrieval and VQA/NLVR-type tasks. The authors also show that the proposed method works well in low-data regimes and that all tasks contribute to the final performance. \n\n", "summary_of_the_review": "The paper is well-written and presents a clear and intuitive idea that works well empirically. The authors use public datasets which improve reproducibility but there seem to be no error bars. The biggest issue is that multiple closely related papers which also focus on masked training for VL models are missing. When accounting for these papers, the novelty of the paper is decreased and the numbers in the tables might also need to be revised,", "strength_and_weaknesses": "Strengths:\n\n* The proposed method is simple and intuitive.\n* The paper relies on open-source datasets which significantly improves reproducibility.\n* The paper shows strong improvements over baseline methods.\n\nWeaknesses:\n\n* There are multiple missing references. The novelty of the proposed method should be discussed in light of these.\n* Given that there are missing references, the baselines in the tables might need to be revised. Thus, it is not clear how well the model performs compared to SOTA methods.\n* Error bars are missing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster