Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Authors: Junnan Li
Ramprasaath R. Selvaraju
Akhilesh Deepak Gotmare
Shafiq Joty
Caiming Xiong
Steven Hoi
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: vision and language pre-training
representation learning
image-text retrieval
visual question answering
vision-language reasoning

Abstract: Large-scale vision and language representation learning has shown promising improvements on various vision-language tasks. Most existing methods employ a transformer-based multimodal encoder to jointly model visual tokens (region-based image features) and word tokens. Because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions. In this paper, we introduce a contrastive loss to ALign the image and text representations BEfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. Unlike most existing methods, our method does not require bounding box annotations nor high-resolution images. In order to improve learning from noisy web data, we propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. We provide a theoretical analysis of ALBEF from a mutual information maximization perspective, showing that different training tasks can be interpreted as different ways to generate views for an image-text pair. ALBEF achieves state-of-the-art performance on multiple downstream vision-language tasks. On image-text retrieval, ALBEF outperforms methods that are pre-trained on orders of magnitude larger datasets. On VQA and NLVR$^2$, ALBEF achieves absolute improvements of 2.37% and 3.84% compared to the state-of-the-art, while enjoying faster inference speed. Code and models are available at https://github.com/salesforce/ALBEF.

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh D. Gotmare Shafiq Joty, Caiming Xiong, Steven C.H. Hoi
Salesforce Research
{junnan 1i, rselaaraa,, akhilesh. g shoi her.jttysshoiiaallssffreeee JSsalesforce.ccm
Abstract
Large-scale vision and language representation learning has shown promising improvements various vision-language tasks. Most existing methods employ transformer-basee multimodal encoder to jointly model visual tokens (region-based image features) and word tokens. Because the visual tokens and word tokens are unaligned, it is challenging for the multimodal encoder to learn image-text interactions. In this paper, introduce contrastive loss to ALign the image and text representations BÉfore Fusing (ALBEF) them through cross-modal attention, which enables more grounded vision and language representation learning. Unlike most existing methods, our method does not require bounding box annotations nor high-resolution images. To improve learning from noisy web data, propose momentum distillation, a self-training method which learns from pseudo-targets produced by a momentum model. We provide theoretical analysis of ALBEF from a mutual information maximization perspective, showing that different training tasks can be interpreted as different ways to generate views for an image-text pair. ALBEF achieves state-of-the performance on multiple downstream vision- language tasks. On image-text retrieval, ALBEF outperforms methods that are pre-trained orders of magnitude larger datasets. On VQA and NLVR2, ALBEF achieves absolute improvements of 2.37% and 8.8% compared to state-of- the-art, while enjoying faster inference speed. Code and models are available at stttps://gtten com st/ssallesfoooo Sew.essfrce/ALBBBB
Introduction
aatn-annn--Laauggee Pre-training (VLP) aims to learn multimodal representations from large-scale image-text pairs that can improve downstream Ession-and-Languaa (V+L) tasks. Most existing VLP methods (e.g. LXMERT [I, UNITER 22, OSCAR (i rely on pre-trained object detectors to extract region-bases based image features, and employ a multimodal encoder to fuse the image features with word tokens. The multimodal encoder is trained to solve tasks that require joint understanding of image and text, such as masked language modeling (MLM) and image-text matching (ITM).
While effective, this VLP framework suffers from several key limitations: (1) The image features and the word token embeddings reside in their own spaces, which makes it challenging for the multimodal encoder to learn to model their interactions; (2) The object detector is both annotation-expensin and compute-expensive, because it requires bounding box annotations during pre-training, and high- resolution (e.g. 600x 1000) images during inference; (3) The widely used image-text datasets [ S
are collected from the web and are inherently noisy, and cxisting pre-training objectives such as MLM may overfit to the noisy text and degrade the model's generalization performance.
We propose ALign BEfore Fuse (ALBEF), new VLP framework address these limitations. We first encode the image and text independently with secectto-frre image encoder and a text encoder. Then we use multimodal encoder fuse the image features with the text features through cross- modal attention. We introduce an intermediate image-text contrastive (ITC) loss on representations from the unimodal encoders, which serves three purposes: (1) aligns the image features and the text features, making it easier for the multimodal encoder to perform cross-modal learning; (2) it
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

improves the unimodal encoders to better understand the semantic meaning of images and texts; (3) it learns a common low-dimensional space to embed images and texts, which enables the image-text matching objective to find more informative samples through our contrastive hard negative mining.
To improve learning under noisy supervision, we propose Momentum Distillation (MoD), simple method which enables the model to leverage a larger uncurated web dataset. During training, we keep a momentum version of the model by taking the moving-average of its parameters, and use the momentum model to generate pseudo-targets as additional supervision. With MoD, the model is not penalized for producing other reasonable outputs that are different from the web annotation. We show that MoD not only improves pre-training, but also downstream tasks with clean annotations.
We provide theoretical justifications on ALBEF from the perspective of mutual information maximiza- tion. Specifically, we show that ITC and ML. maximize a lower bound on the mutual information between different views of an image-text pair, where the views are generated by taking partial information from each pair. From this perspective, our momentum distillation can be interpreted as generating new views with semantically similar samples. Therefore, ALBEF learns vision-language representations that are invariant to semantic-preserving transformations.
We demonstrate the effectiveness of ALBEF various downstream V+L tasks including image-text retrieval, visual question answering, visual reasoning, visual entailment, and supkly-supervised visual grounding. ALBEF achieves substantial improvements over existing state-of-the of-the-art methods. On image-text it outperforms methods that are pre-trained on orders magnitude larger datasets (CLIP 6 and ALIGN 77) On VQA and NLVR22 it achieves absolute improvements of 2.37% and 3.84% compared to the state-of-the-ar method VILLA N while enjoying much faster inference speed. We also provide quantitative and qualitative analysis ALBEF using Grad-CAM (.) which reveals its ability to perform accurate object, attribute and relationship grounding implicitly.
2 Related Work
2.1 Vision-Language Representation Learning
Most existing work on vision-language representation learning fall into two categories. The first category focuses on modelling the interactions between image and text features with transformer- based multimodal encoders [00
[8 Methods in this category achieve superior performance on downstream V+L tasks that require complex reasoning over image and text (e.g. NLVR² 19, VQA )20). but most of them require high-resolution input images and pre-trained object detectors. A recent method [2 improves inference speed by removing the object detector, but results in lower performance. The second category focuses on learning separate unimodal encoders for image and text [2 23 6 77). The recent CLIP (6) and ALIGN ( perform pre-training on massive noisy web data using contrastive loss, of the most effective loss for representation learning 24 25
[4 [5
They achieve remarkable performance on image-text retrieval tasks, but lack the ability model more complex interactions between image and text for other V+L tasks (2]].
26
2
ALBEF unifies the two categories, leading to strong unimodal and multimodal representations with superior performance on both retrieval and reasoning tasks. Furthermore, ALBEF does not require object detectors, a major computation bottleneck for many existing methods I
2.2 Knowledge Distillation
Knowledge distillation 28 aims to improve a student model's performance by distilling knowledge from teacher model, usually through matching the student's 's prediction with the teacher's. ss While most methods focus on distilling knowledge from a pre- cre-trained teacher model 28 29 30 31I [2))
online distillation [R 34] simultancously trains multiple models and use their ensemble as the
teacher. Our momentum distillation can be interpreted as a form of online self-distillation, where a temporal ensemble of the student model is used as the teacher. Similar ideas have been explored in semi-supervised learning (35), label noise learning 36). and very recently in contrastive learning (37] Different from existing studies, we theoretically and experimentally show that momentum distillation is a generic learning algorithm that can improve the model's 's performance on many V+L tasks.
3 ALBEF Pre-training
In this section, we first introduce the model architecture (Section Then we delineate the pre- training objectives (Section 3.2). followed by the proposed momentum distillation (Section 5.3
Lastly we describe the pre-training datasets (Section 3.4) and implementation details (Section 3.5

---

[CLS] embedding
ITM
MLM
hard negatives
multimodal encoder
momentum
Feed forward
update
Cross Attention
x6
momentum distillation
Self Attention
image encoder
text encoder
Momentum Model
CTO
Feed forward
negatives
forward
12x
X6
Self Attention
Image-Text Contrastive Loss
Self Attention
image input
text input
Figure 1: Illustration of ALBEF. It consists of an image encoder, text encoder, and multimodal encoder. We propose image-text contrastive loss to align the unimodal representations of an image-tex pair before fusion. An image-text text matching loss (using an-batch hard negatives mined through contrastive similarity) and masked-lanee waaaod-llangggmodddee modeling loss are applied to learn multimodal interactions between image and text. In order to imprave learning with noisy data, we generate pseudo-targets using the momentum model (a moving-average version of the hase model) as additional supervision during training.
3.1 Model Architecture
As illustrated Figure II ALBEF contains an image encoder, a text encoder, and multimodal encoder. We use 12-layer visual transformer ViT-B/16 [38 the image encoder, and initialize it with weights pre-trained on ImageNet-Ik from BT. An input image is encoded into a sequence of embeddings: (Wels, V1, Ux) ). where Vcls is the embedding of the [CLS] token. We use 6-layer transformer 39 for both the text encoder and the multimodal encoder. The text encoder is initialized using the first 6 layers of the BERT base 40 model, and the multimodal encoder is initialized using the last layers the BERT hase- The text encoder transforms an input text T into a sequence of embeddings { (Wels, W1, wN) ). which is fed to the multimodal encoder. The image features fused with the text features through cross attention at each layer of the multimodal encoder.
3.2 Pre-training Objectives
We pre-train ALBEF with three objectives: image-text contrastive learning (ITC) on the unimodal encoders, masked language modeling (MLM) and image-text matching (ITM) on the multimodal encoder. We improve ITM with online contrastive hard negative mining.
Image-Text Contrastive Learning aims learn better unimodal representations before fusion. learns similarity function s = (Vets) 9w (Wels). such that parallel image-text pairs have higher similarity scores. Iv and 9w are linear transformations that map the [CLS] embeddings to normalized lower-dimensional (256-d) representations. Inspired by MoCo [24] we maintain two queues to store the most recent M image-text representations from the momentum unimodal encoders. The normalized features from the momentum encoders are denoted as g', (v and g', (wwa)) We define
s(I,T) = Go (VcLs) (w'a)) and sTT, I) 9w (w..) Tg', (1(aa)
For each image and text, calculate the softmax-normalized image-to-text and text-to-image similarity as:
exp(s(I,Tmm)) ΣM T==l (opps((,TTmm//)
exp(s(T,
p (1)
$2i (()) (T) Em exp(s(TT,m))))
(I)
where learnable temperature parameter. Let y gi² and (T) denote the ground-truth one-hot similarity, where negative pairs have probability of O and the positive pair has a probability of The image-text contrastive loss defined as the cross-entropy between p and y.:
1 1 B H(y (I), (()) + (T),
Lite
2
3

---

"poler [MASK]'
"a man [MASS)] along a road front of nature summer"
waterfall woods
GT: standing
GT: remote
Top-5 pscudo targets:
Top-5 poeudo-targets:
pueude-taggtt
1. 200
1. walks
2. walking
beoutiful
3. water
3. runs
Itlle
pond
secret
5. wild
goes
secluded
GT: breakdown of the on the road
GT: the harbor small village
Top-5 preudo-targets
young woman get out
near
the harboou with boats and
2. a woman inspects her damaged car under a tree
2. replica of the sailing ship the harbour
a woman looking into car after locking her leys. Inside
3. ships in the harbor of the town
the harbor small village bosts lined alongside the geographical category village
young
broken
brewidown the on the road
Figure Examples of the pseudo-targets for MLM (Ist row) and ITC (2nd row). The pseudo-targets can capture visual concepts that are not described by the ground-trune truth text (e.g. "bcautiful waterfall", "young woman").
Masked Language Modeling utilizes both the image and the contextual text to predict the masked words. We randomly mask out the input tokens with a probability of 15% and replace them with the special token [MASK]] I
Let 7 denote a masked text, and p p (1, (I, denote the model's predicted
probability for masked token. MLM minimizes a cross-entropy loss:
mulm E H(y pmsk (I,
(3)
where mak is one-hot vocabulary distribution where the ground-trute token has probability of 1.
Image-Text Matching predicts whether pair of image and text is positive (matched) or negative (not matched). We use the multimodal encoder's output embedding of the [CLS] token as the joint representation of the image-text pair, and append a fully- connectdd (FC) layer followed by softmax to predict a two-class probability pit The ITM loss is:
Litm = E(.rr)=H((y" imm .p" im (I, T))
where nM is 2- 2-dimensional one-hot vector representing the ground-trute -ruth label.
We propose strategy to sample hard negatives for the ITM task with computational overhead.
negative image-text pair is hard if they share similar semantics but differ in fine-grained details.
We use the contrastive similarity from Equation to find in-batch hard negatives. For each image in mini-batch, we sample one negative text from the same batch following the contrastive similarity distribution, where texts that are more similar to the image have a higher chance to be sampled. Likewise, we also sample one hard negative image for each text.
The full pre- pre-training objective of ALBEF is:
L = Live + Emmm + Litm
(5)
3.3 Momentum Distillation
The image-text pairs used for pre- pre-training are mostly collected from the web and they tend to be noisy. Positive pairs are usually weakly-correlater the text may contain words that are unrelated to the image, or the image may contain entities that are not described in the text. For ITC learning, negative texts for an image may also match the image's content. For MLM, there may exist other words different from the annotation that describes the image equally well (or better). However, the one-hot labels for ITC and MLM penalize all negative predictions regardless of their correctness.
To address this, we propose to learn from pseudo-targets generated by the momentum model. The teacher which consists of ponooentialmovvin-avveeee
momentum model versions of the unimodal and multimodal encoders. During training, we train the base model such that its predictions match the ones from the momentum model. Specifically, for ITC, we first compute the image-text similarity using features from the momentum unimodal encoders as s' (I,T)) (((v'' I (u and s' eTT targets q and q by replacing with s' in Equation The ITCMoD loss is defined as:
continuously-evolvee
I'. (wees)) "' (v(,)) Then we compute soft pseudo-
emod = (1 - Lie + a Bi,TT 2 (I) || ( + (L (T) ll 2ta (T))]
(6)
2
following BERT, the replacements are 10% random tokens, 10% unchanged, and 80% [MASK]

---

Similarly, for MLM. let q ( T) denote the momentum model's prediction probability for the masked token, the MLMMoD loss is:
(I.
Cmod - mim + ( (I. || p pmik (I,
(7)
In Figure 2 wC show examples of the top-5 candidates from the aseudo-targetss which effectively capture relevant words/texts for image. More examples can be found in Appendix.
We also apply MoD to the downstream tasks. The final loss for each task is weighted combination of the original task's loss and the KL-divergence between the model's prediction and the geudo-targets, For simplicity, we set the weight a 0.4 for all pre-training and downstream tasks
3.4 Pre-training Datasets
Following UNITER [2].
construct our pre-training data using two web datasets (Conceptual Captions A. SBU Captions (5)) and two in-domain datasets (COCO [4]] and Visual Genome 442
The total number of unique images is 4.0M, and the number of image-text pairs is 5.1M. To show that our method is scalable with larger-scale web data, we also include the much noisier Conceptual 12M dataset
3 increasing the total number of images 1M. Details Appendix.
3.5
Implementation Details
Our model consists of BERT base with 123.7M parameters and a ViT-B/16 with 85.8M parameters. We pre-train the model for 30 epochs using a batch size of 512 on 8 NVIDIA A100 GPUs. We use the AdamW 44 optimizer with weight decay of 0.02. The learning rate is warmed-up to le in the first 1000 iterations, and decayed to le" - following a cosine schedule. During pre-training, we take random image crops of resolution 256 x 256 as input, and also apply Rand/uugmet 45 During fine-tuning, we increase the image resolution to 384 x 384 and interpolate the positional encoding of image patches following [33]. The momentum parameter for updating the momentum model is set as 0.995, and the size of the queue used for image-text contrastive learning is set as 65,536. We linearly ramp-up the distillation weight from 0 to 0.4 within the 1st epoch.
4 A Mutual Information Maximization Perspective
In this section, we provide an alternative perspective of ALBEF and show that maximizes a lower bound on the mutual information (MI) between different "views" of an image-text pair. ITC, MLM, and MoD can be interpreted as different ways to generate the views.
Formally, we define two random variables a and b as two different views of data point. In self- supervised learning 24 25 46, a and are two augmentations of the same image. In vision-language representation learning, its semantic meaning. We aim to learn representations invariant to the change of view. This can be achieved by maximizing the MI between and b. In practice, maximize lower bound on (I b) by minimizing the InfoNCE loss 47 defined as:
consider and b as different variations of an image-text pair that capture
exp(s(a, b)) Eie exp(s(a, b))
LZCE = -Bp(a.b)
log
(8)
where s(a, b) is a scoring function (e.g., a dot product between two representations), and B contains the positive sample b and | - 1 negative samples drawn from proposal distribution.
Our ITC loss with ooe-hot labels (Equation can be re-written as:
1 Bu(ITT
exp( (ep(s(1 (xp(s(I,TT/r)
log exp(s(T, ΣM eXp(s(T,I,)) Im)/T)
Ene -- log
(9)
ΣM exp(s(I,TTm)/)
Minimizing Lite can be seen as maximizing a symmetric version of InfoNCE. Hence, ITC considers the two individual modalities (i.e., / and 7' as the two views of an image-text pair, and trains the unimodal encoders to maximize the MI between the image and text views for the positive pairs.
Pur experiments show that 0.3, 0.4, 0.5 yield similar performance, with 0.4 slightly better
3 some urls provided by the web datasets have hecome invalid
we remove color changes from RandAugment because the text often contains color information
5

---

As shown in 48, we can also interpret MLM as maximizing the MI between masked word token and its masked context (i.e. image + masked text). Specifically, we can re-write the MLM loss with one-hot labels (Equation as
exp(( ) 1 T((, exp [(I,T))
Mmlm = -B [log
(10)
where (v)) + Rd is a lookup function in the multimodal encoder's output layer that maps word token into vector and V is the full vocabulary set, and [(T ?) is a function that returns the final hidden state of the multimodal encoder corresponding to the masked context. Hence, MLM considers the two views of an image-text pair be: (1) a randomly selected word token, and (2) the image + the contextual text with that word masked.
Both ITC and MLM generate views by taking partial information from image-text pair, through either modality separation or word masking. Our momentum distillation can be considered as generating alternative views from the entire proposal distribution. Take ITCMoD in Equation C as an example, minimizing (T. (T), (e((())))) is equivalent to minimizing the following objective:
exp(8' (1, Tw)/T)
exp(x(1, (m)
Σ j2t log P m
Pm (1) = (1)=
-
Y2U Σm= (I, log
(II)
ΣM cxp(s(I,TTm)/
It maximizes MI(I, Tm) for texts that share similar semantic meaning with the image / because those texts would have larger di (1). Similarly, ITCMoD also maximizes MI T)) for images that are similar to T. We can follow the same method to show that MLMMoD generates alternative views
y' E V for the masked word y y and maximizes the MI between y' and (I, (I, Therefore, our momentum distillation can be considered as performing data augmentation to the original views. The momentum model generates diverse set of views that are absent in the original image-text pairs, and encourages the base model to learn representations that capture view-ing enw-iivarannt semantic information.
5 Downstream V+L Tasks
We adapt the pre-trained model to five downstream V+L tasks. We introduce each task and our fine-tuning strategy below. Details of the datasets and fine-tuning hyperparameters are in Appendix.
Image-Text Retrieval contains two subtasks: image-to-text retrieval (TR) and text-to-image retrieval (IR). We evaluate ALBEF on the Flickr30K [49 and COCO benchmarks, and fine-tune the pre- trained model using the training samples from each dataset. For zero-shot retrieval on Flickr30K, evaluate with the model fine-tuned on COCO. During fine-tuning, we jointly optimize the ITC ITC learns an image-text scoring function based
loss (Equation 2 and the ITMM loss (Equation on similarity of unimodal features, whereas ITM models the fine-grained interaction between image and text to predict matching score, Since the downstream datasets contain multiple texts for each image, we change the ground-truth label of ITC to consider multiple positives in the queue, where each positive has a ground-truth probability of 1/ //#posiiives. During inference, we first compute the feature similarity score $itc for all image-text pairs. Then we take the top-k candidates and calculate their ITM score Sitm for ranking. Because i can be set to be very small, our inference speed is much faster than methods that require computing the ITM score for all image-text pairs [2] X
Visual Entailment (S (51) is a fine-grained visual reasoning task to predict whether the relationship between image and text is entailment, neutral, contradictory. We follow UNITER 2 and consider VE as a three-way classification problem, and predict the class probabilities using a multi-layer perceptron (MLP) on the multimodal encoder's representation of the [CLS] token.
Visual Question Answering (VQA 52 requires the model to predict an answer given an image and question. Different from existing methods that formulate VQA as multi answer classification problem [53 we consider VQA as an answer generation problem, similar to [54] Specifically, we
6-layer transformer decoder to generate the answer. As shown in Figure 3. the rugr-regressive answer decoder receives the multimodal embeddings through cross attention, and a start-of-sequencl token ([CLS]) is used the decoder's initial input token. Likewise, -od-of--equeece token ([SEP] is appended to the end of decoder outputs which indicates the completion of generation.
ressuts on SNLI-VE should be interpreted with caution because its test data has been reported to be noisy 50

---

cross-attention input
Image Encoder $
Multimodal Block
share parameters
image #2
share cross-attention layer
X6
Image Encoder
Multimodal Encoder
Answer Decoder
Image Encoder
Multimodal Block
Image #1
Multimodal Encoder
Image
[cLS]
Text Encoder
Text Encoder
question
text
(a) VQA
(b) NLVR
Figure 3:
The model architecture for VQA and NLVR². For VQA, we append an auto-regressive decoder to generate the answer given the quage-question embeddings. For NLVR², we replicate the transformer block within each layer of multimodal encoder enable reasoning two images.
The answer decoder is initialized using the pre- pre-trained weights from the multimodal encoder, and finetuned with conditional language-modeligg loss. For fair comparison with existing methods, we constrain the decoder to only generate from the 3,128 candidate answers 55 during inference.
Natural Language for Visual Reasoning (NLVR² 19 requires the model to predict whether text describes a pair of images. We extend our multimodal encoder to enable reasoning over two images. As shown in Figure Bp, each layer of the multimodal encoder is replicated to have two consecutive transformer blocks, where each block contains a self-attention layer, cross-attention layer, and feed-forward layer (see Figure D The two blocks within each layer are initialized using the same pre-trained weights, and the two cross-attention layers share the same linear projection weights for the keys and values. During training, the two blocks receive two sets of image embeddings for the image pair. We append a MLP classifier on the multimodal encoder's [CLS] representation for prediction.
For NLVR?, we perform an additional pre-training step to prepare the new multimodal encoder for encoding an image-pair. We design text-assignment (TA) task as follows: given a pair of images and text, the model needs to assign the text to either the first image, the second image, or none of them. consider as three-way classification problem, and use a FC layer on the [CLS] representation to predict the assignment. We pre-train with TA for only I epoch using the 4M images (Section 3.4).
specific textual description. We study the weakly-supervise setting, where no bounding box annotations are available. We perform experiments on the RefCOCO+ [56) dataset, and fine-tune the model using only image- text supervision following the same strategy as image-text retrieval. During inference, we extend to acquire heatmaps, and use them to rank the detected proposals provided by (53].
Visual Grounding aims localize the region in an image that corresponds
Grad-CAM
g
Experiments
6.1 Evaluation on the Proposed Methods
First, we evaluate the effectiveness of the proposed methods (i.e. image-text contrastive learning, contrastive hard negative mining, and momentum distillation). Table I shows the performance of the downstream tasks with different variants of method. Compared the baseline pre-training tasks (MLM+ITM), adding ITC substantially improves the pre-trained model's performance across
#Pre-train Images
TR
IR
SNLI-VE (test)
NLVR (lest-P)
VQA (lest-dev)
Training tasks
(flickr test)
MLM ITM
93.96
88,55
77.06
77.51
71.40
ITC + MLM + ITM
96.55
91.69
79.15
79.88
73.29
ITC + MLM ITMbard
97.01
92.16
79.77
80.35
73.81
4M
ITCMa MLM ITMMaad
97.33
92.43
79.99
80.34
74.06
Full (ITCMeD + MLMMMD + ITMhaed
97.47
92.58
80.12
80.44
74.42
ALBEF (Full MoDnowsstram
97.83
92.65
80.30
80.50
74.54
14M
ALBEF
98.70
94.07
80.91
83.14
75.84
Table 1: Evaluation of the proposed methods four downstream V+L tasks. For text-retrieval (TR) and image-retrieval (IR), report the average of R@ R and @@ 10. ITC: image-text contrastive learning. M: masked language modeling. ITMuad:: image-text matching with contrastive hard negative mining. MoD: momentum distillation, MoD MaDnownsecon: momentum distillation on downstrcam tasks.
7

---

Pre-train Images
Flickr30K (IK test set)
MSCOCO (5K test set)
Method
TR
IR
TR
TR
R@1
R@5
R@
R@1
R@5
R
R@
R@5
R 10
R@
R
R
UNITER
98.0
99.2
75.6
94.1
96.8
65.7
88.6
93.8
52.9
79.9
88.0
VILLA
87.9
97.5
98.8
76.3
94.2
96.8
OSCAR
70.0
91.1
95.5
54.0
80.8
88.5
ALIGN
1.2B
99.8
100.0
84.9
97.4
98.6
77.0
93.5
96.9
59.9
89.8
ALBEF
4M
99.8
82.8
96.7
98.4
73.1
91.4
96.0
56.8
81.5
89.2
ALBEF
14M
95.9
100.0
85.6
97.5
98.9
77.6
94.3
97.2
84.3
90.5
Table 2: Fine-tuned image-text retrieval results on Flickr30K and COCO datasets.
# Pre-train Images
Flickr30K (IK test set)
Method
TR
IR
R@
R@5
R@ 10
R@1 1
R@5
R@ 10
UNITER 2
4M
83.6
95.7
97.7
68.7
89.2
93.9
CLIP
400M
88.0
98.7
99.4
68.7
90.6
95.2
ALIGN
1.2B
88.6
98.7
99.7
75.7
93.8
96.8
ALBEF
4M
90.5
98.8
99.7
76.8
93.7
96.7
ALBEF
14M
94.1
99.5
99.7
82.8
96.3
98.1
Table 3: Zero-shot image-text retrieval results on Flickr30K.
VQA
NLVR
SNLI-VE
Method
Lest-dev
test-std
dev
lest-P
val
test
VisualBERT [1
70.80
71.00
67.40
67.00
VI-BBRT 10
71.16
LXMERT 11
72.42
72.54
74.90
74.50
12-in-1
12
73.15
78.87
76.95
UNITER
72.70
72.91
77.18
77.85
78.59
78.28
VL-BARTYTS 14
71.3
73.6
ViLT PI
70.94
75.24
76.21
OSCAR 6
73.16
73.44
78.07
78.36
VILLA 8
73.59
73.67
78.39
79.30
79.47
79.03
ALBEF (4M)
74.54
74.70
80.24
80.50
80.14
80.30
ALBEF (14M)
75.84
76.04
82.55
83.14
80.80
80.91
Table 4: Comparison with statc-of-the methods on downstream vision-language tasks.
all tasks. The proposed hard negative mining improves ITM by finding more informative training samples. Furthermore, adding momentum distillation improves learning for both ITC (row 4), MLM (row 5), and on all downstream tasks (row 6). In the last row, we show that ALBEF can effectively leverage more noisy web data to improve the pre- pre-training performance.
6.2 Evaluation on Image-Tess Text Retrieval
Table 2 and Table 3 report results on fine-tuned and zero-shot image-text retrieval, respectively. Our ALBEF achieves state-of-the-an performance, outperforming CLIP [ and ALIGN (7] which are trained on orders of magnitude larger datasets. Given the considerable amount of improvement of ALBEF when the number of training images increases from 4M to 14M, we hypothesize that it has potential to further grow by training on larger-scale web image-text pairs.
6.3 Evaluation on VQA, NLVR, and VE
Table 4 reports the comparison with existing methods on other V+L understanding tasks. With 4M pre- training images, ALBEF already achieves soate-of-the-ze performance. With 14M pre-training images, ALBEF substantially outperforms existing methods, including methods that additionally use object tags [ or adversarial data augmentation [ Compared to VILLA [. ALBEF achieves absolute improvements of 2.37% on VQA test-std, 3.84% on NLVR2 test-P, and 1.88% on SNLI-VE test. Because ALBEF detector-free and requires lower resolution images, it also enjoys much faster inference speed compared to most existing methods (>10 times faster than VILLA on NLVR²).
6.4 Weakly-supervised Visual Grounding
Table 5 shows the results on RefCOCO+, where ALBEF substantially outperforms existing meth- ods 57 58 (which use weaker text embeddings). The ALBEF variant computes Grad-CAM
8

---

"man with head down"
"giri with black tank
shirt"
Method
Val
TestA
TestB
ARN
[57
32.78
34.35
32.13
CCL [58]
34.29
36.91
33.56
ALBEF
51.58
60.09
40.19
ALBEF
58,46
65.89
46.25
Table
visual
Figure 4: Grad CAM visualization on the pross-attention maps the 3rd layer of the multimodal encoder.
grounding on RefCOCO+ (56] dataset.
Q: what is the right of Q: is this rice noodle soup?
Q: what is the man doing in the street? walking
Q: what does the truck the left sell? A: cream
A: yes
the soup? A: chopsticks
Figure 5: Brad-CAM visualizations on cross- aoss-atttntionn maps of multimodal encoder for the VQA model.
little girl holding a kitten next to fence"
"girl"
"holding"
"kitten"
"next"
"blue"
Figure 6: Grad-CAM visualizations on the cross-attention maps corresponding to individual words.
visualizations on the self-attention maps in the last layer of the image encoder, where the gradients are acquired by maximizing the image-text similarity Sitc. The ALBEF variant computes Grad-CAM on the cross-attention maps in the 3rd layer of the multimodal encoder (which is a layer specialized in grounding), where the gradients acquired by maximizing the image-text matching score 8itm- Figure ! provides few visualizations. More analysis is in Appendix.
We provide the Grad-CAM visualizations for VQA in Figure 5 As can be seen in Appendix, the Grad-CAM visualizations from ALBEF are highly correlated with where humans would look when making decisions. In Figure 6 show per-word visualizations for COCO. Notice how model not only grounds objects, but also their attributes and relationships.
6.5 Ablation Study
w/ hard negs
w/o hard negs k 128
Table studies the effect of vari- ous design choices on image-text
Flickr30K
Site
k: 16
k = 128
k 256
TR IR
97.30
98.60
98.57
98.57
98.22 (- 0.35)
retrieval. Since use Site filter top-k candidates during inference, we vary and report its effect. In general, the ranking result acquired by Sitm is not sensitive to changes in k. We also validate the effect of
90.95
93.64
93.99
93.95
93.68 0.31)
Table 6: Ablation study fine-tuned image-text retrieval. The av- erage recall the test is reported. We Site to filter top-k candidates and calculate their Simn score for ranking.
hard negative mining in the last column.
Table studies the effect of text- assignment (TA) pre-training and parameter sharing on NLVR We examine three strategies: (1) the two mutimodal blocks share all parameters, (2) only the cross-
w/ TA
w/o TA
NLVR?
share all share CA no share share all share CA no share
dev test-P
82.13
82.55
81.93 82.85
80.52
80.28 80.45
77.84 77.58
82.36
83.14
81.29
Table 7: Ablation study on NLVR
attention (CA) layers are shared, (3) no sharing. Without TA, sharing the entire block has better performance. With TA to pre-train the model for image-pair, sharing CA leads to the best performance.
7 Conclusion and Social Impacts
This paper proposes ALBEF, a new framework for vision-languas representation learning. ALBEF first aligns the unimodal image representation and text representation before fusing them with multimodal encoder. We theoretically and experimentally verify the effectiveness of the proposed

---

image-text contrastive learning and momentum distillation. Compared to existing methods, ALBEF offers better performance and faster inference speed on multiple downstream V+L tasks.
While our paper shows promising results on vision-language representation learning, additional analysis on the data and the model is necessary before deploying it in practice, because web data may contain unintended private information, unsuitable images, or harmful texts, and only optimizing accuracy may have unwanted social implications.
References
[1] Tan, H., M. Bansal. LXMERT: learning cross-modality encoder representations from transform- ers. K. Inui, J. Jiang, V. Ng, X. Wan, eds., EMNLP, pages 5099-5110, 2019.
[2] Chen, Y., L. Li, L.. Yu, et al. UNITER: universal represenntin learning. In ECCV,
vol. 12375, pages 104-120. 2020.
[3] Li, X., X. Yin, C. Li, al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, pages 121-137. 2020.
[4] Sharma, P., N. Ding, S. Goodman, et al. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In I. Gurevych, Y. Miyao, eds., ACL, pages 2556-2565, 2018.
[5] Ordonez, V., G. Kulkarni, T. L. Berg. Im2text: Describing images using I million captioned photographs. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. C. N. Pereira, K. Q. Weinberger, eds., NIPS, pages 1143-1151. 2011.
[6] Radford, A., J. W. Kim, C. Hallacy, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0002 2021.
[7] Jia, C., Y. Yang, Y. Xia, et al. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.059) 2021.
[8] Gan, Z. Y. Chen, L. Li, et al. Large-scale adversarial training for vision -nd--aad--laaaa language represen- tation learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin, eds., NeurIPS. 2020.
[9] Selvaraju, R. R., M. Cogswell, A. Das, et al. Grad-cam: Visual explanations from deep networks via gradient-bai based localization. In ICCV, pages 618-626. 2017.
[10] Su, W., X. Zhu, Y. Cao, et al. VI-bert: Pre-training of generic visual- visual-linguistic representations. In ICLR. 2020.
[11] Lu, J., D. Batra, Parikh, et al. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. H. M. Wallach, Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. B. Fox, R. Garnett, eds., NeurlPS, pages 13-23. 2019.
[12] Lu, J., V. Goswami, M. Rohrbach, et al. 12-in-1: Multi-task vision and language representation learning. In CVPR, pages 00434-10433, 2020.
[13] Li, L. H., M. Yatskar, D. Yin, et al. Visualbert: simple and performant baseline for vision and 0at//1908.0355,,, 2019.
language. arXiv preprint 1908.v:900.00355000
[14] Qi, D., L. Su, J. Song, et al. Imagebert: Cross-modal pre-training with large-scale weak- supervised image-text data. arXiv preprint urXiv:2001.07666, 2020.
[15] Li, G., N. Duan, Y. Fang, al. Unicoder-vi: A universal encoder for vision and language by cross-modal pre-training. In AAAI, pages 11336-11344 2020.
[16] Yu, J. Tang, W. Yin, et al. Ernie-vil: Knowledge enhanced vision vasion-language representations through scene graph. arXiv preprint arXiv:2006. 16934, 2020.
[17] Zhang, P., Li, X. Hu, et al. Vinvi: Making visual representations matter vision-language models. arXiv preprini arXiv:2101.0003
2021.
[18] Huang, Z., Z. Zeng, Y. Huang, et al. Seeing out of the box: End-to-end pre-training for lasion-language representation learning. arXiv preprint arXiv:2104.0311 00.111 2021.
[19] Suhr, A., S. Zhou, A. Zhang, et al. corpus for reasoning about natural language grounded in photographs. In A. Korhonen, D. R. Traum, L. Märquez, eds., ACL, pages 6418-6428. 2019.
10

---

[20] Antol, S., A. Agrawal, J. Lu, et al. VQA: visual question answering. In ICCV, pages 2425-2433 2015.
[21] Kim, W., B. Son, I. Kim. Vilt: Vision-and-language transformer without convolution or region supervision. arXiv preprint arXiv:2 2021.
[22] Faghri, F. D. J. Fleet, J. R. Kiros, et al. VSE++: improving visual -ssaatsemantic embeddings with hard negatives. In BMVC, page 12. 2018.
[23] Li, K., Y. Zhang, K. Li, al. Visual semantic reasoning for image-text matching. In ICCV, pages 4653 1563-4661. 2019.
[24] He, K., H. Fan, Y. Wu, et al. Momentum contrast for unsupervised visual representation learning. In CVPR. 2020.
[25] Chen, T., S. Kornblith, M. Norouzi, et al. simple framework for contrastive learning of visual representations. In ICML. 2020.
[26] J., P. Zhou, C. Xiong, et al. Prototypical contrastive learning of unsupervised representations. In ICLR. 2021.
[27] Li, J., C. Xiong, S. C. Hoi. Mopro: Webly supervised learning with momentum prototypes. In ICLR. 2021.
[28] Hinton, G., O. Vinyals, J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXivv/503.000 02031, 2015.
[29]
Zagoruyko, S., N. Komodakis. Paying more attention to attention: Improving the performance
of convolutional neural networks via attention transfer. In ICLR. 2017.
[30] Furlanello, T., Z. C. Lipton, M. Tschannen, et al. Born-again neural networks. In J. G. Dy, A. Krause, eds., ICML, pages 1602-1611. 2018
[3]] Touvron, H., M. Cord, M. Douze, al. Training effacnff image transformers & distillation through attention. arXiv preprint arXiv:2012. /2877, 2020.
[32] Sanh, V., L. Debut, J. Chaumond, et al. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXiv:1910.000 108, 2019.
[33] Zhang, Y., Xiang, T. M. Hospedales, Deep mutual learning. CVPR, pages 4320-4328. 2018.
[34] Anil, R., G. Pereyra, A. Passos, et al. Large scale distributed neural network training through online distillation. ICLR. 2018.
[35] Tarvainen, A., H. Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NIPS, pages 1195-1204. 2017.
[36] Li, J., R. Socher, S. C. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR. 2020.
[37] Cheng, R., B. Wu, P. Zhang, et Data-efficient language-supervised zero-shot learning with self-distillation. arXiv preprint wxXiv:2/04.0894 2021.
[38] Dosovitskiy, A., L. Beyer, A. Kolesnikov, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. 2021.
[39] Vaswani, A., N. Shazeer, Parmar, al. Attention all you need. In Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. N. Vishwanathan, R. Garnett, eds., NIPS, pages 5998-6008. 2017.
[40] Devlin, J., M. Chang, K. Lee, et al. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, T. Solorio, eds., NAACL, pages 4171-4186. 2019.
[41] Lin, T., Maire, J. Belongie, et al. Microsoft COCO: common objects in context, In D. J. Fleet, T. Pajdla, B. Schiele, T. Tuytelaars, eds., ECCV, vol. 8693, pages 740-755. 2014.
[42] Krishna, R., Y. Zhu, O. Groth, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJC,, 123(1):32-73, 2017.
[43] Changpinyo, S., P. Sharma, N. Ding, et al. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR. 2021.

---

[44] Loshchilov. I. F. Hutter.
Decoupled weight decay regularization.
arXiv preprint
arXiv:1711.05101, 2017.
[45] Cubuk, E. D., B. Zoph, J. Shlens, al. Randaugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, pages 702-703. 2020.
[46] Tian, Y., C. Sun. B. Poole, et al. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin, eds., NeurlPS. 2020.
[47] Oord, A. v. d.. Y. Li, O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv: 1807.03748, 2018.
[48] Kong, L., C. de Masson d' Autume, L. Yu, et al. A mutual information maximization perspective of language representation learning. In ICLR. OpenReview.net, 2020.
[49] Plummer, B. A., L. Wang, C. M. Cervantes, et al. Flickr30k entities: Collecting region-to-ppieee to-phrase correspondences for richer mmagetto-sentence models. In ICCV, pages 2641-2649, 2015.
[50] Do, V., O.-M. Camburu, Z. Akata, et al. e-snli-ve: Corrected visual-textual entailment with natural language explanations. arXiv preprint arXiv:2004. 033444,00..3774, 2020.
[51] Xie, N., F. Lai, D. Doran, al. Visual entailment: A novel task for fine-grained image understanding. arXiv preprini arXiv:1901.06706 2019.
[52] Goyal, Y., T. Khot, D. Sommers-Staay et al. Making the Vn VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6325-6334. 2017.
[53] Yu, L., Z. Lin, X. Shen, et al. Mattnet: Modular attention network for referring expression comprehension. CVPR, pages 1307-1315. 2018.
[54] Cho, J., Lei, H. Tan, al. Unifying vision-and-languaal tasks via text generation. arXiv preprint
2021.
[55] Kim, J., J. Jun, B. Zhang. Bilinear attention networks. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biande R. Garnett, eds., NIPS, pages 1571-1581. 2018.
[56] Yu, L., P. Poirson, S. Yang, et al. Modeling context in referring expressions. In B. Leibe,
Matas, N. Sebe, M. Welling, eds., ECCV, pages 69-85. 2016.
[57] Liu, X., L. Li. S. Wang, al. Adaptive reconstruction network for weakly supervised referring expression grounding. In ICCV, pages 2611-2620. 2019.
[58] Zhang, Z., Z. Zhao, Z. Lin, et al. Counterfactual contrastive learning fo weakly-supervised linguae grounding. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin, eds., NeurlPS. 2020.
[59] Karpathy, A., F. Li. Deep visual-semantic alignments for generating image descriptions. In CVPR, pages 3128-3137. 2015.
[60] Bowman, S. R., G. Angeli, C. Potts, et al. A large annotated corpus for learning natural language inference. L. Märquez, C. Callison- Burch,, J. Su, Pighin, Y. Marton, eds., EMNLP, pages 632-642, 2015.
[61] Yu, Z., J. Yu, Y. Cui, et al. Deep modular co-attention networks for visual question answering. In CVPR, pages 6281-6290 2019.
[62] Kazemzadeh, S., V. Ordonez, M. Matten, et al. Referitgame: Referring to objects in photographs of natural scenes. A. Moschitti, B. Pang, W. Daelemans, eds., EMNLP. 2014.
[63] Das, A., H. Agrawal, L. Zitmick, al. Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions? 2016.
12


Reviews:
[{"review_id": "iT2YPf1DH6r", "reviewer": "Reviewer_3hiK", "summary": "This paper proposes a new vision-language representation learning framework, ALBEF, which uses image-text contrastive (ITC), masked language modeling (MLM), and image-text matching (ITM) losses with momentum distillation to learn SOTA representations. The ITC and MLM losses are adapted to be convex combinations of the original loss and the KL divergence between the predicted probability distribution (of the image or text or masked tokens) and the soft pseudo targets obtained from the momentum model. \n\nThe pipeline is evaluated on image-text retrieval, visual entailment, NLVR, VQA, and grounding and shows improved performance on all tasks compared to prior work, and with fewer training samples in comparison to several prior methods. Qualitative visualizations also verify the quality of the learned representations, with convincing heat maps suggesting well grounded, semantically aligned embeddings. ", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper was well written and provided extensive quantitative and qualitative evaluation of the representation learning framework. While the individual losses being used were not \"original\" (contrastive learning has been applied on image-text data before, e.g., CLIP, ALIGN, masked language modeling is commonplace in both language and vision-language representation learning), the used of a momentum model to balance the original losses with a KL divergence term over pseudo target distributions is really interesting, and novel. Additionally, the experimental results are quite convincing, and ALBEF often outperforms prior work with fewer pretraining samples, which is very desirable.  \n\nMoreover, the paper clearly states it is trying to address the following problems with prior work:\n(1) image and text features residing in independent spaces\n(2) the use of object detectors\n(3) the overfitting of MLM to noisy text data\nAnd it successfully addresses these points. \n\nAn important result of this method is that training with momentum distillation does *not* hurt performance on downstream tasks with clean annotations, unlike prior work where performance can degrade in these settings. It is a significant contribution to use MoD and not degrade performance on clean data.\n\nAuthors also provide well-written analysis connecting the losses to maximizing a lower bound on MI. While this is not new information for the ITC loss, it is nice that the authors also connect the MLM loss to this setting. And for readers less familiar with infoNCE prior work, it provides a thorough description.\n\nQuestions:\n- The ablations in Table 1 verify that using both ITC and ITM (without and with hard negative mining) improves performance over MLM + ITM alone. I find this kind of surprising- why would ITM be complementary to ITC? They seem like very similar losses and contrastive losses already weigh hard negatives more than other samples, so I am wondering if there is any intuition to why the combination of both does better or how ITM inherently differs from ITC. \n- How was the alpha chosen for the MoD losses?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not spend significant time in the paper addressing limitations or societal impact, and the writing of the latter is a bit hand-wavy at the end in section 7. However, there are no critical limitations or negative societal impacts that needed to be obviously addressed by the contribution of this paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "UdR3mB_etLj", "reviewer": "Reviewer_XRzR", "summary": "This paper presents an vision-and-language pretraining framework that utilizes momentum distillation to address the issue of training on noisy web data. More specifically, it presents a self-training approach to learn from pseudo-targets that are generated by a momentum model. In contrast to existing approaches which simply use cross-modal attention to reason about concatenated sequences of image regions and words, this work also leverages contrastive learning to learn more effective representations before using cross-modal attention.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The submission is generally well-written and easy to follow. In particular, the figures are well-drawn and informative, which helps the reader to understand the approach. The related work section is also comprehensive and provides clear distinctions between this work and existing vision-and-language pretraining approaches. The proposed approach is trained using the same pretraining objectives including masked language modeling and image-text matching in existing approaches such as VL-BERT and VisualBERT.\n\nWhile the concept of knowledge distillation is not new, the application of it to improve the ability of the model to learn from uncurated datasets is interesting. In particular, with noisy text, it helps to improve pretraining by not penalizing the model for producing reasonable outputs that may correspond to the image or text. The provision of theoretical justifications on the benefits of the momentum distillation is also very helpful.\n\nIt is not clear how sensitive the proposed approach is to the distillation weight. It would be informative to include an ablation over the distillation weight on a small subset to determine its effect on the final performance. In addition, it is not specified why the text encoder is initialized with the first 6 layers of a pretrained BERT-base model while the multimodal encoder is initialized using the last 6 layers.\n\nThe effectiveness of this work is demonstrated via extensive quantitative experiments on multiple downstream tasks, including image-text retrieval and visual question answering, where it outperforms state-of-the-art approaches on all of the metrics. The significance of this work is that the learnt language and visual representations can be used to improve performance for training and evaluating on other downstream tasks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they have.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6.5"}, "extra_text": {}}, {"review_id": "KXtqfzHNVO2", "reviewer": "Reviewer_16yc", "summary": "In this paper, the authors study the vision-transformer-like vision-language-pretraining (VLP) methods. The raw image and text are first encoded by a vision transformer and a text transformer respectively, and are then sent into a multimodal transformer for fusion. There are two major technical contributions, i.e., 1) the proposed ITC loss and 2) the moving average “teacher” to generate the pseudo targets.  \n\n\nThe proposed method (ALBEF) outperforms previous E2E VLP methods, and achieves comparable performance to the VLP methods that take object region features.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. This is a solid work that improves the SOTA of the VLP with raw image input.\n\n2. The proposed image-text contrastive learning (ITC) and moving average teacher are intuitive and effectively improve the model performance. To my knowledge, the idea of aligning the visual and text modalities before sending them into the multimodal fusion as in ITC is new.\n\n3. The analyses in Section 4 and the Grad-CAM visualization provide good analyses of the proposed approach.\n\nWeaknesses: \n1. The first two rows in Table 1 show that the proposed ITC loss significantly improves the performance. I wish to hear discussions and comments on where the ITC could be useful. Is it specified to the proposed vision-transformer-related VLP models, or it could be extended to the Vilt-like structure [21] with linear projection only, or even to the conventional VLP methods [13,2,3] with detector features. \n\n2. The ablation of the model architecture might be necessary to understand the source of improvements, i.e., from the proposed losses and distillation, or the change in model architecture and patch size. Specifically, it might be necessary to ablate 1) different model sizes, e.g., small, base, large; 2) different patch sizes and numbers, e.g., ViT/32, /16, and input image size; 3) the influence of model initialization in Sec 3.1.\n\n3. A minor concern is the comparison to SOTA in Table 4. The compared methods are the BASE version of the SOTA [3,8]. The LARGE version of [3,8] yields similar performance to the reported numbers.\n\n4. A comment: The contribution of the momentum distillation and the analyses in Section 4 limit at applying the related techniques (moving average teacher in semi-supervised learning [33], the mutual information maximization perspective in NLP [A]) onto the VLP methods. Nonetheless, the proposed techniques and analyses do help the VLP study.\n\n[A] A mutual information maximization perspective of language representation learning\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impacts are discussed in the conclusion section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DiS1Oy7hwMs", "reviewer": "Reviewer_TH9r", "summary": "This paper introduces a method  to perform text-image matching and demonstrate that  it can be used to solve several language-vision downstream tasks  ranging from image-text retrieval, VQA and national language for visual reasoning among others. \n\n* Given an  image and a textual input visual and  textual embedding are obtained  that are then fed  to a multimodal encoder with  cross-attention layers. \n* The first key idea of this paper lies in the image-text contrastive loss employed between the image/text embeddings before feeding them to the multimodal  decoder with the intuition behind that being that forcing these embeddings to align before fusing them through the multi-modal encoder results into better representations for downstream tasks\n* The second key idea is in the usage of  a momentum model that generates pseudo-target labels by keeping a moving average that are then used for supervision at training time. \n\nAn in-depth discussion is provided:\n1) for each of the losses used,\n2) on the fact that the proposed approach maximizes a lower bound on the mutual information between different “views” of an image-text pair\n\nExtensive Results for several tasks and datasets are provided and strong results are reported. The qualitative results demonstrating where the learned attention is focusing for each downstream task are also quite insightful", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Well-written, well-motivated, easy to follow and understand what's happening and why the design choices (architecturally, and loss-wise) are done the way they're done. I find the method sufficiently novel in the context of \"aligning before fusing\" with the contrastive loss. The results reported  across all tasks are strong and SOTA. \n\n* As much as I liked reading the paper and found its results strong and mostly convincing I think it would be a fair question to ask the authors what they think the key novelty of this work is and maybe  discuss it explicitly. The reason I'm mentioning this is that one could argue that it's a combination of existing losses and modules (cross attention, momentum distillation) that are well-put together but at the end of the day there's not an important contribution there. \n\n* Why does the momentum distillation have such a small impact across all downstream tasks (Table 1). I could see why it intuitively makes sense (instead of single positive/negative label have a list of possible pseudo-gt ones) but in practice it seems not super useful?\n\n* While the limitations in terms of  societal impact is indeed provided in Sec. 7, there's no discussion in terms of weaknesses of the proposed approach. For example how fragile is the method in the pseudo-targets generated and what happens if they're consistently off. I'd encourage the authors to provide some additional details and insights here. \n\n\n## Post Rebuttal\n\nI'd like to thank the authors for taking the time to answer my (and the other reviewers') questions and for providing clarifications on their novelty and some experimental results. Overall I think this is a good paper with an interesting method along with a very solid experimental investigation. I'm happy to increase my score to \"7: Good paper, accept\". \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have devoted the last 4 lines of the paper to discuss potential limitations/challenges that one should address before deploying this method in real-life applications. There's a large discussion going on these days in the research community in the impact of training large language models and tasks like image-text matching where the visual domain comes in (especially in the context of raw web data) requires additional and careful consideration on what's used, how it's used, how owns the text/img data etc. I think the authors have done a sufficient job in this context but I think a little bit more of an in-depth discussion here would be beneficial. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "CN-D2fqzw0M", "reviewer": "Reviewer_z2Jk", "summary": "The paper proposes to learn joint vision and language representations by addressing several limitations in the existing methods. Firstly, they learn separate image and text representations using unimodal encoders without the need for bounding box annotations. Second, they fuse these two representations using a multimodal encoder based on cross modal attention and, third, they address the noise in large scale web based image-text datasets using a Momentum Distillation approach that generates pseudo-targets as additional supervision. They demonstrate the effectiveness of their approach on various downstream V+L tasks and gain substantial improvement over SOTA methods. Their main contributions are: removing the need for pre-trained object detector and high resolution images and combining it with the contrastive loss function for learning effective multi-modal representations, image-text contrastive learning loss (ITC). They also generate pseudo-targets for the ITC loss and the masked language modeling (MLM) loss using the momentum encoder model to address the weak correlations in the noisy image-text web data. This achieves high performance on both reasoning and retrieval tasks unlike other methods. The paper  is very well written and easy to follow and understand. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ALBEF model proposed in the paper learns joint vision and language representations using three pretext tasks/objective functions - image-text contrastive learning (ITC), masked language modeling (MLM) and image-text matching (ITM). These pre-trained embeddings are then fine-tuned on several V+L downstream tasks to show their effectiveness. The method is significantly novel and addresses the limitations of existing work. It shows consistent improvements over SOTA methods by using relatively fewer data samples for pre-training. \n\n1. In line 101, is w_cls the [CLS] token for the text inputs? For ITC loss, the similarity is computed based on these CLS embeddings, what would happen if instead it is computed based on the mean of the region/word-level embeddings?\n\n2. Can the loss function in Eq 1., be added at the multimodal encoder stage as well? \n\n3. It is not very clear how the momentum distillation learns to generate pseudo-targets. What is g’_v and g-_w in lines 148? Are these the average of the visual/textual features in the queue? How do they correspond to the image-text query to generate meaningful targets ?\n\n4. Are the outputs from the ITC loss in Eq 6, softer after adding the KL divergence term ? If so, is this an advantage over existing methods for MLM and ITC? Would doing label smoothing or adding softmax temperature yield similar advantages? \n\n5. In line 154-155, the authors say that they use  distillation loss for the downstream tasks, how does that work? Are the weights of this model fixed or fine-tuned? Or is it learned separately with the downstream task’s model?\n\n6. In VQA downstream task, lines 234, are both the answer decoder and multimodal encoder initialized with the weights of multimodal encoder? \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors show that their proposed method gains improvement with significantly fewer images (14M) as compared to the state-of-the-art ALIGN model that uses 1.2B images to train. They also discuss the negative impact of the work due to leakage of private information, social implications, etc if deployed in real world settings. These things have to be taken care of when using such models in practice. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes to tackle vision-language tasks by combining a contrastive loss used to first “align” the vision & language representations before “fusing” them via a cross-attention model trained via masked language modeling and image-text matching losses.

Reviewers were generally in agreement that this is a strong and well-written submission, proposing a well-motivated approach with convincing state-of-the-art quantitative and qualitative results.  While the overall objective and training procedure is rather complex, each component is ablated and demonstrated to be necessary for the method’s strong performance.

Some reviewers raised a few questions and concerns about missing ablations and design decisions. These questions were addressed satisfactorily by the authors’ responses. I strongly encourage the authors to incorporate the feedback and their provided answers into the camera-ready version of the submission where appropriate.

Given these strengths and especially given the recent interest in the area of vision-language representation and transfer learning, I recommend the submission for a spotlight presentation at NeurIPS.
Decision: Accept (Spotlight)