Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to pre-train a generative language model conditioned on a visual input on billion-scale web image-text data. Such a model can be then transferred to various vision-and-language tasks with ease. SimVLM establishes new SotA on several new tasks and shows promising zero-shot capacity in certain tasks.


main_review: 
Pros:

1. The empirical results are strong. SimVLM takes an important step towards less human-label-dependent V&L models.

2. Interesting results on some zero-shot tasks. In Sec 4.3.2, the authors introduce a new setting: train on a similar single-modality task and transfer directly to a multi-modality task. This is a new yet promising way of transferring to new V&L tasks.
 

Cons:


1. Dataset deduplication. I do not see if the authors check whether downstream tasks data are present in the pre-training dataset. It is recommended to check the overlap.

2. Image caption / SNLI-VE and VQA experiments are all grouped into the zero-shot setting. They are technically "zero-shot" but they actually differ quite a lot from others. SNLI-VE & VQA experiments are also not the "full zero-shot" setting where we simple take a model and directly get the desired output. I would recommend being more careful about the wording. 

3. For the experiment on SNLI-VE in Table 3, I think the same method could also be done for UNITER (i.e. train on SNLI and then transfer to SNLI-VE). It is an informative baseline to have.


Minor points:

4. Could the cross-modality experiments be extended to other tasks such as VQA? Could we train on  some similar text-only tasks and then transfer to VQA? The SNLI-VE is a good sign but SNLI-VE is a three-way classification task and is perhaps easier to transfer to than VQA.

5. Efficiency concern. Training on CLIP/ALIGN-level data is costly. A seemingly more efficient way to use trained CLIP/Align model as bottom backbones and train a V&L transformer on top (e.g., Shen et al., 2021). With the same model scale (base), the VQA score is not so far apart (78.14 v.s. 76.48).
Shen et al., How Much Can CLIP Benefit Vision-and-Language Tasks?

6. True Zero-Shot VQA? I wonder what prevents the authors from directly testing on VQA. One could give the image + question as the prefix and then calculate the generation probability for every answer candidate from the 3,129 answer pool. This would serve as a valuable reference point.

summary_of_the_review: Overall, I think the cons are points that could be addressed and do not hurt the main merit of the paper. Thus I recommend acceptance but I would recommend addressing the cons if possible.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a simple but effective image-text multimodal representation learning method that leverages a transformer-based encoder-decoder using a simple prefix langemodel as pretraining task from large-scale noisy image-text aligned data. As fine tuning tasks, the authors evaluate their method (SimVLM) on VQA, NLVR2, SNLI-VE, CoCo caption, NoCaps, and Multi30k. With extensive experiments, this work presents promising few-shot and zero-shot performance results outperforming previous models.  
main_review: #### [Strengths]
- Overall, I like this paper. The proposed method is very simple but effective and practical because the pretraining task is not complex and the used weakly aligend image-text data are relatively easy to acquire compared to those used in other methods such as CLIP. 
- Despite its simplicity, the performance are promising for various fine-tuning tasks.
- The paper is easy to understand.
- The experiments are thorough and support their hypothesis. 

#### [Weakness]
Despite many strenghs of this paper, there are some room to be improved. My score is between 6 and 8 due to the issues below:
- In introduction, the authors argued "These design choices complicate the pretraining protocol of VLP, creating a bottleneck for further quality improvement." and "these methods mainly focus on specific tasks of consideration and thus may not serve as a generic pretraining-finetuning representation for VL benchmarks.". Are there any explicit evidences or refences for this arguement? 
- Figure 1 can be improved. When zooming up, some texts are not clear.
- Reproduciblity is limited. The limited reproducibility by other research groups results from data issue. It is not trivial to construct 1.8B paired data even if they are weakly-aligned and train them. Also, I guess that the promising results of a simple pretraining task and architectures might benefit from large-scale size of its training data. Therefore, the ablation experiments on various training data sizes (e.g. 10%, 30%, 50%, and 100%) will be helpful for understanding the effects of the data size on the performances. If smaller size data can provide comparable results, the contributions of this work will be much enhanced. 
- Presenting parameter sizes of each model variant will help to understand the results. 
- Table 2 is not easy to read. In Table2, what is the difference betwwen the second and the third row groups? I guess the second is for zero-shot and the third one is for fine-tuning. This should be explicitly clarifed in Table for readability. Also, why is SimVLM_large checked only as pretraining? 
- The results in the main paper and the appendix are explained in the main manuscript. For example, even if Figure 2 and Table 7 exist in the appendix, there is no information on those. This harms the readability.
- Table 3 can also be improved. Fine-tuned datasets and metrics are located at the same row. Image Masking results also might harm the readability. 
- Why the perfomances of SNLI-finetuned is better than those of SNLI-VE-finetuned? The authors need to clarify this reason.
- Multimodal learning can provide more robust performance on different domain data like zero-shot image classification on line-draw images of CLIP. SimVLM also present more robust image classification performances on different domain data? In addition to classification accuracy, these results can enhance the contribution of this paper, if possible. 
- In Ablation study, why used SimVLM_small? How about the possibility that poor performance of decoder-only model resulted from its smaller model size? 
- Minor issues (typos)
  - In 3.4 in p4, "with raw image patch inputs." . --> , (comma)
  - in 4.1 in p, "Huge".: . --> , (comma)
  - Karpapth --> Karphathy 

summary_of_the_review: Overall, this is a good paper but there are some issues that should be improved, in particular, reproducibility and table for readability.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a Prefix Language Modeling (PrefixLM) objective for a pretraining procedure for multiple vision-language downstream tasks and zero-shot evaluations. They argue that it successfully replaces the masked language model (MLM). This work follows the notion of weak supervision suggested by ALIGN (Jia et al., 2021), having noisy image captions and holistic labels. Surprisingly, this simple pretraining approach achieves new state-of-the-art on "a wide range of discriminative and generative vision-language benchmarks," and shows "strong generalization and transfer ability" in zero-shot settings.
main_review: 
### Strength

**Simplified pretraining procedure.** The most important point is to simplify a pretraining procedure. In recent works, MLM and image-text matching (ITM) losses are repeatedly used and additionally propose dataset-specific losses to exploit inductive biases. Unlike this trend, this work counters the general belief and gives a new insight to use the PrefixLM for vision-language pretraining (VLP) as a standalone objective. However, this paper's argument seems to be fragile for the reasons in the below weakness points.

**Generalization and transfer knowledge.** The PrefixLM is a generative modeling loss compared to the MLM's classification loss. For this reason, it may be keen on zero-shot generalization. They show competitive or better performance on zero-shot image captioning, zero-shot cross-modality transfer, and open-ended visual question answering tasks.

### Weakness

The PrefixLM is the paper's central argument, aside from the weak supervision dealt with in ALIGN (Jia et al., 2021) and others (e.g., CC 12M, Changpinyo et al., 2021). However, this paper lacks solid validation of their argument, as summarized below.

**W1. Can the PrefixLM replace MLM?** The critical ablation study to see the effectiveness of PrefixLM compared to MLM is 1) limitedly shown on VQA and image captioning in Table 6, a limited number of downstream tasks compared to the other experiments, although it is a critical study to validate their argument. Moreover, 2) this ablation study is performed for text-only data, not vision-language data in Section 4.4 Ablation Study. For this reason, their argument is fragile to be nullified in the further validation in the current status.

- Could you provide any evidence that the PrefixLM is better than MLM in the controlled experiment (ablation study) for multimodal data?

**W2. Can the PrefixLM be used as a standalone objective?** For instance, the ITM loss is known to be effective combining with MLM (e.g., ViLT, Kim et al., 2021). They argue that a single objective is enough, but there is no validation of whether this is true. The natural questions on this matter include combining with ITM or other dataset-dependent losses would be helpful or result in mediocre performance.

- Could you provide any evidence that PrefixLM + ITM method is not significantly different from PrefixLM only method? If not (PrefixLM + ITM is better), what would be the advantage of using the PrefixLM only? 

**W3. Fair comparison issue with the ALIGN dataset.** The crucial problems in the experiments are that 1) it is irreproducible for its inaccessible collected datasets outside of the associated groups, and 2) it is hard to assess the contribution in the state-of-the-art comparison due to different pretraining datasets, different model architectures, and pretraining objectives. The controlled experiment still has the problems as described in W1. 

- For the reproducibility, could you provide the scores pretraining on WIT (Srinivasan et al., 2021) only?


### Writing

- In the RHS of Eqn.3, if $t=T_p$, is $P(x_{T_p}|x_{[T_p,T_p]}, x_{< T_p})$ the intended one? Should $x_{[T_p,T_p]}$ be null?
- It is ambiguous that "which decouples encoding from generation is conducive to the improvement of downstream task" in the 1st paragraph, Section 3.3. What do you mean by "decouple"?
- In the 2nd paragraph of Section 3.3, "we additionally add 2D relative attention for the image patches within transformer layers," which needs some elaboration for reproducibility and self-contained explanation.
- Using C4 dataset was crucial? Why do you choose to use C4 additionally?
- In the 4.1 Setup, you mentioned that "512 text-only documents are in each batch," but, in this reading moment, cannot follow the context. 
- In Table 2, "'Pre.' indicates the model is pretrained" on what? Are you mentioning that the corresponding models are pretrained on CoCo or NoCaps? Here, all SimVLM is also pretrained on both ALIGN and C4 datasets, but you don't say all your model is pretrained.
- In 4.3.1, please be aware that your Figure 2 is placed in Appendix and indicate explicitly. 
- In Table 2, could you specify the meaning of each row section? Why are two SimVLM_{huge} scores different in the 2nd and 3rd sections?
- Explanation on transformer decoder is simply skipped. Could you provide implementational details for reproduction?
summary_of_the_review: This paper shows that a simple pretraining method, the PrefixLM, using two datasets from the dataset of ALIGN (Jia et al., 2021) and the Colossal Clean Crawled Corpus (C4) dataset (Raffel et al., 2019) (optionally, along with WIT (Srinivasan et al., 2021)) achieves new state-of-the-art in various vision-language benchmarks. However, this work is narrowed to achieve the state-of-the-art performances while failing to show 1) better performance of the PrefixLM than MLM, 2) relationships with the other multimodal pretraining losses (e.g., ITM) 3) in the controlled experiment in terms of pretraining dataset and model architecture. The weaknesses W1-3 hinder this paper's recommendation since the main argument of the PrefixLM is in a vague position.
review_title: null

Meta-review: This paper presents SimVLM, a simpler generative VLP framework with billion-scale web image-text data, which has good zero-shot potential while remaining competitive on standard VL benchmarks. SimVLM achieves SotA on several tasks and shows promising zero-shot capacity in certain tasks. Most of the reviewers liked the work; they had concerns about data scaling, but the authors showed that SimVLM_small with the smaller cc3m dataset does not drop performance too much (although the large data scaling with their large-scale weakly aligned data is still important to achieve good zero-shot learning). All reviewers also mentioned the strong concern of reproducibility and data accessibility, so we encourage the authors to address this as clearly as possible via releasing models and safely-cleaned/anonymized data subsets, etc.
Decision: Accept (Poster)