Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces an architecture which supports multiple modalities (text, image and text-image), via  a novel transformer module (MoME), in which blocks consist of modality specific experts with shared self-attention.
The method achieves new SOTA on vision language classification tasks (VQA and NLVR2), as well as text retrieval and image retrieval, on MS-COCO 5k and Flickr 30K.
The model can be pre-trained stage-wise, using text-only and image-only large scale databases.

strengths_and_weaknesses: Strengths:
* support for text-only, image-only and text-image pretraining, which allows leveraging existing datasets, reducing the need  for large datasets with both modalities labeled.
* image-text contrastive learning used for pre-training;
* selection via multi-headed self attention of which expert to use for subsequent processing.
* detailed ablation study.

Weaknesses:
* 4M to 1B images for pre-training leads to ~6% improvement. It would be interesting to understand if similar accuracy could be achieved with less data, or, conversely, get a higher improvement from 250x data.
* (minor) No details about the architecture of the experts (it seems these are feed forward networks).
questions: How does this model compare to  the Flamingo model of Alayrac et al?
How would the model perform on Meta-Dataset of Triantafillou et al? Would the retrieval benefit from text pre-training? Would image pre-training improve results, due to the different data distributions and increased number of classes?
Are the Vision / Language and Vision-Language experts trained independently? Would it benefit the subsequent tasks if these experts were pre-trained upfront?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on Vision-Language Pre-training (VLP), in which they propose a unified vision-language pretrained model VLMO that jointly learns a dual encoder (i.e., the text and image encoders) and a fusion encoder (i.e., the multimodal encoder).

Specifically, the authors introduce the Mixture-of-Modality-Experts (MOME) Transformer to encode different modalities,  where each block contains a pool of modality-specific experts to  capture modality-specific information and a shared self-attention layer to learn the alignments between different modalities. Moreover, it can utilize large amounts of image-only and text-only data to perform stagewise pre-training. The pre-trained model can be used as a fusion encoder for classification tasks, or fine-tuned as a dual encoder for retrieval tasks.


strengths_and_weaknesses: Strengths:

1. The motivation of the proposed VLMO model makes sense to me, which can alleviate the limitations of existing single-stream and two-stream VLP models.

2. The paper is generally well organized, and mostly easy to follow. Although the pre-training tasks are generally not new, the proposed Mixture-of-Modality-Experts (MOME) Transformer is relatively novel.

3. Experimental results demonstrate the effectiveness of the proposed pre-training framework and stagewise pre-training strategies. 


Weaknesses:

I have several concerns as follows:

1.  In the pre-training stage, the proposed model is first trained with a MRM with the image-only data, followed by a MLM with the text-only data. The motivation is not clear. Is it possible to perform MLM first followed by MRM? Or is it better to perform alternative training?

2. The downstream tasks only include image-text classification/retrieval tasks, but fail to consider many other representative VL generation tasks such as image captioning.

questions: N/A
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new method called VLMo for multimodal pretraining. VLMo consists of a dual encoder and a fusion encoder with Mixture-of-Experts, where experts of different modalities process information of specified modality or modality combination. This method achieves state-of-the-art performance in several multimodal downstream tasks. 
strengths_and_weaknesses: Strengths:
1. The method achieves state-of-the-art performance in several multimodal downstream tasks. 
2. The method creatively incorporates mixture-of-experts with multimodal representation learning, which bring significant improvements in downstream tasks. 

Weaknesses:
1. Lack of analysis of the mixture-of-experts. As this is the key innovation of the paper, which is somewhat novel and to the my best of my knowledge the first time to see modality expert in multimodal pretraining, the paper should demonstrate clearly why it is an important design, how it contributes to the performance, how different experts function and cooperate to reach a better effect than the conventional fusion methods. 
2. Unclear introduction to the model details: a. The MoE is essentially not the conventional MoE [1] with routing, so there should be clarification to avoid misunderstanding. b. There should be a clear demonstration of the number of parameters. As there are 3 FFN experts in the model, the transformer layers of VLMo-large should consist of at least 700M parameters, let alone the embedding and other parameters. Directly stating that it is a large-size model without clarification misleads readers. c. Based on my understanding, though the model is much larger, it did not bring much increase in computation costs. This should be illustrated clearly. Also, the paper should clarify which layers to use vision and text experts for different modalities, and which layers to process them with the VL experts, instead of using words like "bottom layers" and "top layers". 
3. The design of stagewise pretraining requires further explanation as there are many details, e.g., why is such stage design? how about training them jointly? why is the self attention fixed in the second stage? The ablation study should be comprehensive, and make more fair comparisons with the baselines. 
4. VLMo achieves stunning performance in VQA, which surpasses even all the current methods as far as I know, but I find from the supplementary details that the VLMO-large++ for VQA uses a high resolution of 768 * 768. This requires explanation or ablation study. From the presented results, readers cannot figure out where the improvements over VLMo-Large come from, scaling data or scaling resolution. 
5. This work should implement the model on more downstream tasks, including visual entailment, referring expression comprehension, etc. They are essential benchmarks for multimodal pretrained models. Also, this work should compare VLMo with the state-of-the-art methods (3 months before the NeurIPS submission), including BLIP [2] and OFA [3]. Especially for the cross-modal retrieval, VLMo should be compared with BLIP. Furthermore, as VLMo is pretrained with image-text contrastive learning, it is encouraged to conduct zero-shot cross-modal retrieval experiments. 

[1]. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.
[2]. Li, J., Li, D., Xiong, C., & Hoi, S. (2022). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086.
[3]. Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., ... & Yang, H. (2022). Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. arXiv preprint arXiv:2202.03052.
questions: 1. Why does this work uses such a high resolution of 768 * 768 only for VQA, but uses 224*224 or 384 * 384 for the other tasks?
2. Different from the common practice, VLMo adds image-text contrastive learning to the pretraining tasks, which use two towers to process information from different modalities separately, incompatible with the fusion, and I believe that there should be 4 forward operations （2 forwards in the image and text encoder for contrastive learning, 1 for ITM, and 1 for MLM） for a single batch. Does this cause increase in training costs? Is it possible to provide an ablation study on the pretraining tasks? 

limitations: This paper did not point out limitations. It is suggested that the paper explain about lack of analysis in the expert design, lack of capabilities in generation tasks, and lack of further analysis about scaling, especially data scaling.
review_title: null

Meta-review: metareview: After the rebuttal and discussion, this paper unanimously receives positive rates. As the reviews show satisfaction on the authors’ feedback, the final draft needs to respect it accordingly. Additionally, reviewers suggest several post-rebuttal points, so the final draft may need to clarify them, including the comparison with Flamingo, the setup of stages and contributions of experts. 
confidence: Certain
Decision: Accept