Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper proposes a new vision-language representation learning framework, ALBEF, which uses image-text contrastive (ITC), masked language modeling (MLM), and image-text matching (ITM) losses with momentum distillation to learn SOTA representations. The ITC and MLM losses are adapted to be convex combinations of the original loss and the KL divergence between the predicted probability distribution (of the image or text or masked tokens) and the soft pseudo targets obtained from the momentum model. 

The pipeline is evaluated on image-text retrieval, visual entailment, NLVR, VQA, and grounding and shows improved performance on all tasks compared to prior work, and with fewer training samples in comparison to several prior methods. Qualitative visualizations also verify the quality of the learned representations, with convincing heat maps suggesting well grounded, semantically aligned embeddings. 
main_review: This paper was well written and provided extensive quantitative and qualitative evaluation of the representation learning framework. While the individual losses being used were not "original" (contrastive learning has been applied on image-text data before, e.g., CLIP, ALIGN, masked language modeling is commonplace in both language and vision-language representation learning), the used of a momentum model to balance the original losses with a KL divergence term over pseudo target distributions is really interesting, and novel. Additionally, the experimental results are quite convincing, and ALBEF often outperforms prior work with fewer pretraining samples, which is very desirable.  

Moreover, the paper clearly states it is trying to address the following problems with prior work:
(1) image and text features residing in independent spaces
(2) the use of object detectors
(3) the overfitting of MLM to noisy text data
And it successfully addresses these points. 

An important result of this method is that training with momentum distillation does *not* hurt performance on downstream tasks with clean annotations, unlike prior work where performance can degrade in these settings. It is a significant contribution to use MoD and not degrade performance on clean data.

Authors also provide well-written analysis connecting the losses to maximizing a lower bound on MI. While this is not new information for the ITC loss, it is nice that the authors also connect the MLM loss to this setting. And for readers less familiar with infoNCE prior work, it provides a thorough description.

Questions:
- The ablations in Table 1 verify that using both ITC and ITM (without and with hard negative mining) improves performance over MLM + ITM alone. I find this kind of surprising- why would ITM be complementary to ITC? They seem like very similar losses and contrastive losses already weigh hard negatives more than other samples, so I am wondering if there is any intuition to why the combination of both does better or how ITM inherently differs from ITC. 
- How was the alpha chosen for the MoD losses?
limitations_and_societal_impact: The authors do not spend significant time in the paper addressing limitations or societal impact, and the writing of the latter is a bit hand-wavy at the end in section 7. However, there are no critical limitations or negative societal impacts that needed to be obviously addressed by the contribution of this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a method  to perform text-image matching and demonstrate that  it can be used to solve several language-vision downstream tasks  ranging from image-text retrieval, VQA and national language for visual reasoning among others. 

* Given an  image and a textual input visual and  textual embedding are obtained  that are then fed  to a multimodal encoder with  cross-attention layers. 
* The first key idea of this paper lies in the image-text contrastive loss employed between the image/text embeddings before feeding them to the multimodal  decoder with the intuition behind that being that forcing these embeddings to align before fusing them through the multi-modal encoder results into better representations for downstream tasks
* The second key idea is in the usage of  a momentum model that generates pseudo-target labels by keeping a moving average that are then used for supervision at training time. 

An in-depth discussion is provided:
1) for each of the losses used,
2) on the fact that the proposed approach maximizes a lower bound on the mutual information between different “views” of an image-text pair

Extensive Results for several tasks and datasets are provided and strong results are reported. The qualitative results demonstrating where the learned attention is focusing for each downstream task are also quite insightful
main_review: * Well-written, well-motivated, easy to follow and understand what's happening and why the design choices (architecturally, and loss-wise) are done the way they're done. I find the method sufficiently novel in the context of "aligning before fusing" with the contrastive loss. The results reported  across all tasks are strong and SOTA. 

* As much as I liked reading the paper and found its results strong and mostly convincing I think it would be a fair question to ask the authors what they think the key novelty of this work is and maybe  discuss it explicitly. The reason I'm mentioning this is that one could argue that it's a combination of existing losses and modules (cross attention, momentum distillation) that are well-put together but at the end of the day there's not an important contribution there. 

* Why does the momentum distillation have such a small impact across all downstream tasks (Table 1). I could see why it intuitively makes sense (instead of single positive/negative label have a list of possible pseudo-gt ones) but in practice it seems not super useful?

* While the limitations in terms of  societal impact is indeed provided in Sec. 7, there's no discussion in terms of weaknesses of the proposed approach. For example how fragile is the method in the pseudo-targets generated and what happens if they're consistently off. I'd encourage the authors to provide some additional details and insights here. 


## Post Rebuttal

I'd like to thank the authors for taking the time to answer my (and the other reviewers') questions and for providing clarifications on their novelty and some experimental results. Overall I think this is a good paper with an interesting method along with a very solid experimental investigation. I'm happy to increase my score to "7: Good paper, accept". 

limitations_and_societal_impact: The authors have devoted the last 4 lines of the paper to discuss potential limitations/challenges that one should address before deploying this method in real-life applications. There's a large discussion going on these days in the research community in the impact of training large language models and tasks like image-text matching where the visual domain comes in (especially in the context of raw web data) requires additional and careful consideration on what's used, how it's used, how owns the text/img data etc. I think the authors have done a sufficient job in this context but I think a little bit more of an in-depth discussion here would be beneficial.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an vision-and-language pretraining framework that utilizes momentum distillation to address the issue of training on noisy web data. More specifically, it presents a self-training approach to learn from pseudo-targets that are generated by a momentum model. In contrast to existing approaches which simply use cross-modal attention to reason about concatenated sequences of image regions and words, this work also leverages contrastive learning to learn more effective representations before using cross-modal attention.
main_review: The submission is generally well-written and easy to follow. In particular, the figures are well-drawn and informative, which helps the reader to understand the approach. The related work section is also comprehensive and provides clear distinctions between this work and existing vision-and-language pretraining approaches. The proposed approach is trained using the same pretraining objectives including masked language modeling and image-text matching in existing approaches such as VL-BERT and VisualBERT.

While the concept of knowledge distillation is not new, the application of it to improve the ability of the model to learn from uncurated datasets is interesting. In particular, with noisy text, it helps to improve pretraining by not penalizing the model for producing reasonable outputs that may correspond to the image or text. The provision of theoretical justifications on the benefits of the momentum distillation is also very helpful.

It is not clear how sensitive the proposed approach is to the distillation weight. It would be informative to include an ablation over the distillation weight on a small subset to determine its effect on the final performance. In addition, it is not specified why the text encoder is initialized with the first 6 layers of a pretrained BERT-base model while the multimodal encoder is initialized using the last 6 layers.

The effectiveness of this work is demonstrated via extensive quantitative experiments on multiple downstream tasks, including image-text retrieval and visual question answering, where it outperforms state-of-the-art approaches on all of the metrics. The significance of this work is that the learnt language and visual representations can be used to improve performance for training and evaluating on other downstream tasks.
limitations_and_societal_impact: Yes, they have.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes to learn joint vision and language representations by addressing several limitations in the existing methods. Firstly, they learn separate image and text representations using unimodal encoders without the need for bounding box annotations. Second, they fuse these two representations using a multimodal encoder based on cross modal attention and, third, they address the noise in large scale web based image-text datasets using a Momentum Distillation approach that generates pseudo-targets as additional supervision. They demonstrate the effectiveness of their approach on various downstream V+L tasks and gain substantial improvement over SOTA methods. Their main contributions are: removing the need for pre-trained object detector and high resolution images and combining it with the contrastive loss function for learning effective multi-modal representations, image-text contrastive learning loss (ITC). They also generate pseudo-targets for the ITC loss and the masked language modeling (MLM) loss using the momentum encoder model to address the weak correlations in the noisy image-text web data. This achieves high performance on both reasoning and retrieval tasks unlike other methods. The paper  is very well written and easy to follow and understand. 

main_review: The ALBEF model proposed in the paper learns joint vision and language representations using three pretext tasks/objective functions - image-text contrastive learning (ITC), masked language modeling (MLM) and image-text matching (ITM). These pre-trained embeddings are then fine-tuned on several V+L downstream tasks to show their effectiveness. The method is significantly novel and addresses the limitations of existing work. It shows consistent improvements over SOTA methods by using relatively fewer data samples for pre-training. 

1. In line 101, is w_cls the [CLS] token for the text inputs? For ITC loss, the similarity is computed based on these CLS embeddings, what would happen if instead it is computed based on the mean of the region/word-level embeddings?

2. Can the loss function in Eq 1., be added at the multimodal encoder stage as well? 

3. It is not very clear how the momentum distillation learns to generate pseudo-targets. What is g’_v and g-_w in lines 148? Are these the average of the visual/textual features in the queue? How do they correspond to the image-text query to generate meaningful targets ?

4. Are the outputs from the ITC loss in Eq 6, softer after adding the KL divergence term ? If so, is this an advantage over existing methods for MLM and ITC? Would doing label smoothing or adding softmax temperature yield similar advantages? 

5. In line 154-155, the authors say that they use  distillation loss for the downstream tasks, how does that work? Are the weights of this model fixed or fine-tuned? Or is it learned separately with the downstream task’s model?

6. In VQA downstream task, lines 234, are both the answer decoder and multimodal encoder initialized with the weights of multimodal encoder? 


limitations_and_societal_impact: The authors show that their proposed method gains improvement with significantly fewer images (14M) as compared to the state-of-the-art ALIGN model that uses 1.2B images to train. They also discuss the negative impact of the work due to leakage of private information, social implications, etc if deployed in real world settings. These things have to be taken care of when using such models in practice.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors study the vision-transformer-like vision-language-pretraining (VLP) methods. The raw image and text are first encoded by a vision transformer and a text transformer respectively, and are then sent into a multimodal transformer for fusion. There are two major technical contributions, i.e., 1) the proposed ITC loss and 2) the moving average “teacher” to generate the pseudo targets.  


The proposed method (ALBEF) outperforms previous E2E VLP methods, and achieves comparable performance to the VLP methods that take object region features.

main_review: Strengths: 
1. This is a solid work that improves the SOTA of the VLP with raw image input.

2. The proposed image-text contrastive learning (ITC) and moving average teacher are intuitive and effectively improve the model performance. To my knowledge, the idea of aligning the visual and text modalities before sending them into the multimodal fusion as in ITC is new.

3. The analyses in Section 4 and the Grad-CAM visualization provide good analyses of the proposed approach.

Weaknesses: 
1. The first two rows in Table 1 show that the proposed ITC loss significantly improves the performance. I wish to hear discussions and comments on where the ITC could be useful. Is it specified to the proposed vision-transformer-related VLP models, or it could be extended to the Vilt-like structure [21] with linear projection only, or even to the conventional VLP methods [13,2,3] with detector features. 

2. The ablation of the model architecture might be necessary to understand the source of improvements, i.e., from the proposed losses and distillation, or the change in model architecture and patch size. Specifically, it might be necessary to ablate 1) different model sizes, e.g., small, base, large; 2) different patch sizes and numbers, e.g., ViT/32, /16, and input image size; 3) the influence of model initialization in Sec 3.1.

3. A minor concern is the comparison to SOTA in Table 4. The compared methods are the BASE version of the SOTA [3,8]. The LARGE version of [3,8] yields similar performance to the reported numbers.

4. A comment: The contribution of the momentum distillation and the analyses in Section 4 limit at applying the related techniques (moving average teacher in semi-supervised learning [33], the mutual information maximization perspective in NLP [A]) onto the VLP methods. Nonetheless, the proposed techniques and analyses do help the VLP study.

[A] A mutual information maximization perspective of language representation learning

limitations_and_societal_impact: The limitations and societal impacts are discussed in the conclusion section.
review_title: null

Meta-review: This submission proposes to tackle vision-language tasks by combining a contrastive loss used to first “align” the vision & language representations before “fusing” them via a cross-attention model trained via masked language modeling and image-text matching losses.

Reviewers were generally in agreement that this is a strong and well-written submission, proposing a well-motivated approach with convincing state-of-the-art quantitative and qualitative results.  While the overall objective and training procedure is rather complex, each component is ablated and demonstrated to be necessary for the method’s strong performance.

Some reviewers raised a few questions and concerns about missing ablations and design decisions. These questions were addressed satisfactorily by the authors’ responses. I strongly encourage the authors to incorporate the feedback and their provided answers into the camera-ready version of the submission where appropriate.

Given these strengths and especially given the recent interest in the area of vision-language representation and transfer learning, I recommend the submission for a spotlight presentation at NeurIPS.
Decision: Accept (Spotlight)