Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed to use the vision transformer as the visual backbone for vision-language pre-training. To quantize the information flow transferred between vision and language modalities, the authors proposed a metric called Inter-modality flow that aggregates the attention weight across different layers. Experiments on VQA, visual entailment, and visual reasoning validates that the proposed method achieves state-of-the-art performance.
main_review: [Strength]

- Using visual transformers as the backbone to extract visual features is a neat and intuitive idea. The whole architecture is more consistent with the multi-modal transformers. The proposed IMF seems to have correlations with the downstream task performance, which might be a good indicator to measure the pre-trained vision language model without downstream tasks. 

[Weakness]

- My major concern of this paper is the lack of novelty. Visual parsing model is basically the vision transformer backbone, Masked feature regression is very similar to other pre-training objectives and has been used in previous work. The inter-modality flow can only measure the interaction between different modalities, 

- For masked feature regression, the proposed mask the token with top-k attention weight, is it possible to mask based on the proposed IMF metric? 

- The paper only tests with 3 downstream tasks (VQA, NLVR, Visual Entailment). Some other tasks which might require more vision-language interaction (VCR) are not tested in the paper. What is the IMF across different tasks? For example, is the IMF required in VQA different from NLVR? It will be good to add these ablation studies. 
limitations_and_societal_impact: The paper didn't provide texts about limitations and societal Impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces Swin Transformer in the visual domain for vision-language pretraining. Previous VL pretraining models feed image tokens and text tokens together into the cross-modal transformer, and the visual relationship learning and inter-modal alignment are encapsulated in the same transformer network. The authors propose that visual relationship between different visual objects or concepts is important for inter-modal alignment learning, and there should be independent processing of visual relationship in the framework. Motivated by this hypothesis, the authors add a visual-only transformer to learn visual relationship before feeding the image and text tokens together to the cross-modal transformer. In addition, the authors propose a metric named inter-modality flow (IMF) to measure the cross-modal interactions. They also propose masked feature regression as a new pretraining task. Pretraining is conducted on in-domain datasets COCO and Visual Genome. Evaluation is conducted on three downstream tasks VQA, visual reasoning, and visual entailment.
main_review: Strengths:

1. The motivation is clear and the method is clearly written.

2. The inter-modality flow is a novel metric to reflect the cross-modal interactions of different vision-language frameworks. The authors have deep discussion into this metric, and compare the IMF of the proposed approach and previous appraoches. The data distribution analysis in page 9 is also interesting and indicates that the proposed model learns better aligned vision-language features.

3. The vision-only structure for learning visual relationship of different objects has not been explored before. This paper makes the first attempt on introducing transformer-based visual relationship reasoning before the cross-modal transformer. I think it might inspire future work, discussions, and analysis.

Weakness:

1. The effectiveness of the swin transformer is not fully validated in the ablation study. To make fair comparisons, at least two experiments should be conducted. (1) Compare swin transformer with other vision transformers, such as ViT, to validate the superior of the swin transformer structure. (2) Compare with a baseline model that does not have the visual parsing module, i.e., the image patches are directly projected into visual tokens and then feed into the cross-modal transformer. This experiment can validate how well the visual parsing module works.

2. Why there is a vision-only transformer, but not a language-only transformer, before the cross-modal transformer. I would expect an ablation study experiment on adding a language parsing module which uses a transformer to process the input sentences. I am curious whether adding the language transformer will help or harm the performance.

3. The pretrained model is not evaluated on the downstream tasks of image captioning and text-image retrieval, which I think are two important downstream tasks for vision-langauge pretraining.

4. In Lines 298-299, "ViLT has much smaller IMF than other methods, as patch projections learn little about visual relationships." I do not totally understand it. I understand that smaller IMF indicates less inter-modality interactions, but how does it related to the visual relationships?

limitations_and_societal_impact: This paper does not discuss much on limitations and societal impact. I encourage the authors to add some discussions on the potential limitations, problems, or failure cases of the proposed method.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper provides a novel way of applying masked feature regression (MFR) to transformer-based visual encoders. In addition to this new way of MRM, the paper also provides a quantifiable measurement of how two modalities (image and text) interchange their information with the aids from the attention flow method. This quantity, the inter-modality flow (IMF), is then used to analyze the characteristics of different vision-and-language pretraining models.
main_review: This paper provides a novel way of applying masked feature regression (MFR) to transformer-based visual encoders. In addition to this new way of MRM, the paper also provides a quantifiable measurement of how two modalities (image and text) interchange their information with the aids from the attention flow method. This quantity, the inter-modality flow (IMF), is then used to analyze the characteristics of different vision-and-language pretraining models.

Question 1. I wonder if MFR can be applied to the features of non-transformer visual encoders (CNNs) by adopting correlations between spatial features other than attention weights. If so, I am eager to see how it can boost the performance of Pixel-BERT, which does not have a visual-related masked modeling objective.

Comment 1. The main model uses a Swin transformer as the visual encoder. Since Swin uses local window attention to build hierarchical structures out of vision transformer, it also has a local receptive field like CNN. Thus, the CNNs' limitations in visual relation learning mentioned in the abstract can also be applied to Swin. I believe this delineation should be revised.

Comment 2. The authors model each layer's inter-modality flow by *indexing* the *contextualized vector sequence* with the indices assigned to each modality before the first attention layer. However, the information of each index is mixed all together by processed through layers after layers. Thus, the vectors can not be considered a hundred percent text modality or a hundred percent image modality. The near-zero NMI at layers 8 to 11 of ViLT (figure 5) makes this speculation more concrete since the near-zero NMI means that two modalities are heavily mingled.

Question 2. To further analyze Comment 2, I am curious about how the indices are assigned to each cluster changes over layer in figure 5. Will the text and image indices be assigned to the same cluster for every layer?

### Remarks
[+] (Clarity) Overall, the paper is well written and easy to understand.  
[+] (Originality) I find originality out of both (1) using the attention weight to group the patches to mask and (2) employing the attention flow to analyze the inner interactions of the modalities.  
[+] (Significance) The proposed MFR is significant because it liberates vision-and-language pretraining methods' visual masked modeling from its pre-trained object detectors, which are used to obtain the target distributions of masked regions. However, MFR is only applicable to transformer-based visual encoders, limiting its usage for CNN visual encoders. (Question 1)  
[-] (Quality) The limitation of CNNs is too exaggerated. (Comment 1)    
[-] (Quality) The inter-modality flow should be further analyzed. (Comment 2 and Question 2)  
limitations_and_societal_impact: I think the authors fairly well addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a transformer based visual backbone for vision and language understanding. It uses a vision transformer backbone (Swin Transformer) to extract visual representation, which is then passed to a mask generation module. The visual tokens(including masked tokens) and text tokens embeddings are fed to a multimodal transformer model, where different pretraining objectives are applied. This architecture achieves state-of-the-art results in VQA, SNLI-VE, NLVR^2, Image-Text Retrieval. Authors propose an Inter Modality Flow metric that measure interaction between the visual and textual modalities and correlates well with the downstream results.
main_review: STRENGTHS

- It is impressive that the model achieves state-of-the-art results with a smaller input image dimension : 384x384 compared to previous works that use much larger image input dimensions like 600×1000 or 800×1333 which were difficult to train end to end due to huge computational cost. 
- Results on downstream tasks shows that with limited pretraining data but better visual representations, good accuracy can be achieved.
- The Inter Modality Flow metric seems to correlate well with the downstream results. This can act as a strong validation metric to monitor during pretraining on multimodal datasets. 

WEAKNESSES and REMARKS

- It is not clear to me from the ablation studies where the performance improvement is coming from. From Table 4. ablations, switching the backbone to SWIN-T resulted in the biggest improvement in accuracy. Is it because Swin Transformer has already learnt better visual features to start with and hence provides better results when multimodal pretraining is performed with visual features extracted from that? In general, in the vision and language community it has been observed that improved visual backbone helps to achieve superior downstream results on tasks like VQA etc. For example, the recent work [1], shows that a very powerful image backbone can generate representations of a richer collection of visual objects and concepts and hence get impressive downstream results in V&L tasks.

      [1] Zhang, Pengchuan, et al. "Vinvl: Revisiting visual representations in vision-language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

- For fair comparison with other end to end models like SOHO, authors should provide comparative results with SOHO model where it’s CNN encoder is replaced with a Swin Transformer of similar capacity as used in this work.
- It is not clear which Swin Transformer(Swin-T, Swin-S, Swin-B, Swin-L) is used for different experiments reported in Table 1, Table 2, Table 3. It will be better if parameter counts are also reported for different approaches in Table 1, Table 2, Table 3 for fair comparison and understanding whether the improvements correlate to param count. 
- MFR(Masked Feature Regression) metric in different forms has already been explored in previous works like VilBERT, UNITER etc. The only difference in this work as I understand is that the visual features are masked by masking all top-k attention weights for a particular randomly picked token. Given that, I am not sure this is a novel contribution. 
- The improvement from adding the MFR task is relatively very low (+0.5) improvement over no MFR. Is 0.5 a statistically significant improvement? It is usually better to take average of multiple runs when the gap is too small. 
- IMF metric is used for evaluating different models. It is not clear how IMF is used during the training process or how it helps to improve the model : whether it is used during pretraining as a validation metric to pick the best checkpoint or just used was a framework to evaluate different models after they are pretrained? If it is the former, then for fair comparison, IMF metric should also be used during pretraining of UNITER, SOHO etc to understand the value of the metric to give a better pertained model. However, IMF metric is not shown in ablation studies in Table 4, which makes me assume it was not used as a metric to monitor the model during pretraining.
- I am unsure about the added novelty of this work, it seems that majority of the performance improvements come from a better visual encoder backbone, and although the IMF metric shows the model to be better than SOHO/UNITER, it seems much of it is due to using a better vision encoder (Swin Transformer vs R/X-101). 


I would be interested to see the discussion and authors' replies to the concerns during the response stage. I am overall leaning towards improving the rating.
limitations_and_societal_impact: Please consider adding the Broader Impact Statement to the main paper rather than in the Appendix.
review_title: null

Meta-review: The paper initially received divergent recommendations, but after the detailed author response all reviewers lean to accept the paper.

I recommend accept under the expectation that authors will revise the paper according to the detailed author response, addressing the reviewers' concerns. This includes, but is not limited to the following points
1) clarifications & delineations, including to prior work.
2) Additionally results and ablations (tables) provided in author response
3) details on model size in FLOPs
[some can go to supplement if they don't fit in the page limit]
Decision: Accept (Poster)