Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper scales masked autoencoders (MAE) to image-text multimodal data, it investigates the use of modality-agnostic unified encoders, which learns generalizable representations across a number of downstream tasks.
strength_and_weaknesses: Strengths:
(a) The paper is overall well-written, the problem well formulated and novel to me. Generalizing masked autoencoders to work on more modalities is definitely interesting to the community.
Weaknesses:
(a) There seems to be a lack of comparison with other baselines. The paper only compare with CLIP and MAE, while not other baselines, such as MultiMAE[1], using the same modality-agnostic setting, were not included.
(b) The author should also include a discussion on how is M3AE different from MultiMAE.
[1] MultiMAE: Multi-modal Multi-task Masked Autoencoders, ECCV 2022
clarity,_quality,_novelty_and_reproducibility: Well-motivated paper yet lacks discussion and comparison with other baselines.
summary_of_the_review: The paper is overall well-motivated and well-written, yet the experiment part seems to fall short with a lack of discussion and comparison with other baselines.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the multimodal representation learning in the masked auto-encoder way. The work is motivated by the fact that many existing multimodal learning approaches require a large amount of paired text-image data. This work builds upon MAE, extending it to text modality as well. The proposed method can handle both the unpaired image data and paired image-text data. The experiments show a clear improvement over MAE in several tasks, and show the effectiveness of the learned transferrable representation. 

--- post-rebuttal ---\
Thanks for the authors' response. I have carefully read the response and other reviewers' comments. I believe the current form of this paper  is not ready for ICLR publication. I keep my score. 
strength_and_weaknesses: Strength 

-The idea of applying masked autoencoder strategy to multimodality data is rational. The finding on higher mask ratio on text performs better than the low mask ratio on pure text is interesting. The explanation makes sense to me. 

-The experiments show the strength over MAE in different scenarios and applications.  

 

Weakness 

-There is other work also working on the similar motivation, i.e., leveraging unpaired data and paired text-image data for multimodal representation learning. BEIT-V3 [Wang et al.,], where they demonstrate an exceptional performance over many existing solutions including CLIP on a number of downstream tasks. It seems this paper misses the discussion on this work. And I’d also like to see the experimental comparison with BEIT-3 to show the advantages. 

[Wang et al.,] Wang et al.,Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks, https://arxiv.org/abs/2208.10442 

-I have a little concern on the effectiveness on the large-scale data, such as several hundreds of millions of data points or even larger. The current training is on CC-12M, and shows a good improvement over MAE. However, when the number of training samples gets larger, I am not sure if this improvement margin can be preserved.  

-Following the above point, thanks to the effort on the large scale data curation LAION, there are billions of image –text noisy pairs available. Please explain the application scenario of this method. 

-Figure 9, the reconstruction result from MAE is missing. I’d like to see a comparison to show the strength. 

-Other minors: In page 3, “For patches and tokens, we sample s random”, not sure what “s” means here. 

In page 7, there is a space in “M3AEoutperforms”  
clarity,_quality,_novelty_and_reproducibility: The presentation is clear to me. and I'm satisfied with the details of the technical points. I think this work can be easily reproduced. Regarding novelty, I'm ok with the proposed method to extend MAE to multimodal representation learning. While I like to see the clear comparison with other existing efforts like BEIT-3 with a similar motivation.
summary_of_the_review: Overall, the proposed solution is rational and mostly well presented. While my major concerns are lack of comparison with the existing method and no clear motivation given billions of paired text image data available.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an empirical study of M3AE trained on a large-scale image-text dataset, and find that multimodal masked autoencoder is able to learn generalizable representations that transfer well to downstream tasks. 
strength_and_weaknesses: Strength:

1. The paper is well written and this reviewer enjoyed reading it.

2. The proposed approach is simple and intuitive.

3. It shows promising results on image classification and out-of-distribution detection tasks.

Weakness:
1. The idea lacks of novelty. 

2. No experiments for vision-language tasks have been performed (e.g., text-to-image retrieval, VQA, visual reasoning, etc). The authors only fine-tune their pre-trained models for visual tasks, such as out-of-distribution detection and image classification. As the paper targets at multi-modal pre-training, it's essential to provide the experiments for vision-language down-stream tasks!

3. Figure 2 is confusing. For example, in the lower left corner, the text "Trees in a winter storm" even appears in the space of its right image. 

4. Ablation study experiments aren't sufficient, it's important to conduct the experiments, e.g., different mask ratios of MAE.
  
5. Why not use two separate encoders for visual and textual feature extraction? The proposed M3AE only use one encoder for these two kinds of modalities together, but this may make the models confused. it's better to perform ablation study to use two different encoders, so that we can figure out the necessity of such design.

6. The comparison is unfair, the proposed method uses more vision-language data (e.g. CC12M) for pre-training, and compare the models with MAE, which is only pre-trained on ImageNet. The performance gap between M3AE and CLIP is still very large. 
clarity,_quality,_novelty_and_reproducibility: 1. Clarity - good; the paper is well-written.

2. Quality - the experiments aren't sufficient. Please see the weakness 2,4 and 5.

3. Originality - ok. The proposed M3AE lacks of novelty.

4. Reproducibility - unclear. 
summary_of_the_review: The paper is about the new approach for pre-training vision-language models by the proposed multimodal masked autoencoder. Although the approach is interesting, it is incremental and not novel. Some of the experiments demonstrating the superiority of the proposed approach are also unclear.

The proposed M3AE haven't performed experiments on vision-language down-stream tasks, it's not clear that the approach can learn good multimodal representation. Although the model can achieve better performances compared with MAE, M3AE uses larger dataset (e.g. CC12M). The performance gap between M3AE and CLIP is large.

Some important ablation study experiments, such as different mask ratios of MAE and different encoders for visual/textual inputs, aren't conducted.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an extension of Masked Autoencoders (MAE) to the vision-language domain called Multi-Modal Masked Autoencoder (M3AE). Image-caption pairs are partially corrupted by masking some image patches and some language tokens. The non-masked image patches and language tokens are passed through an autoencoder based on transformers, which is trained to reconstruct the full image-caption, including masked parts. The embeddings of image patches that are learned by the transformer encoder are transferred to downstream image tasks, such as classification and out-of-distribution detection. 
strength_and_weaknesses: Strengths: The paper is easy to understand, with the methodology clearly presented. The results section includes some nice qualitative analysis and visualizations to help understand the representations that are learned by M3AE.

Weaknesses: 

1) The empirical support for the proposed approach is weak.

One of the main takeaways from the experiments (Figure 3-5) is that M3AE outperforms MAE in the downstream ImageNet classification task. This is a good sanity check, but it is not significant nor surprising. The M3AE framework involves predicting text token logits from image patches, which is similar to a supervised classification task where the text is providing labels for training the image encoder. On the other hand, MAE is completely unsupervised. 

It would be better to compare M3AE against other vision-language pretraining methods. However, no vision-language pretraining baselines are provided except for CLIP which outperforms M3AE by a large margin in Figure 3.

2) In the introduction, the authors make the strong claim that M3AE offers two benefits over contrastive pretraining methods, but neither benefit is demonstrated in the experiments.

First, the authors emphasize that contrastive learning "requires paired image-and-text data and therefore cannot leverage widely available unpaired data." This suggests that leveraging unpaired data will make M3AE perform better than CLIP. But as shown in Figure 3, CLIP does much better than M3AE. To support this claim, the authors should either (i) decrease the paired data for CLIP to e.g., 10-30% and show that M3AE does better than CLIP for the same amount of paired data in Figure 3, or (ii) add some unpaired image or text data for M3AE and show that it can do better than CLIP by leveraging this unpaired data.

Second, the authors claim that the "separation of image and text encoders" in contrastive learning "hinders the joint understanding of image and text", but they provide no proof of this. To support this claim, they should show that M3AE performs better than contrastive learning on a multimodal downstream task. Also, note that separation of image and text encoders is not necessary in contrastive learning (e.g., GLIPv2 enables fusion between image and text features before the contrastive objective), so this is not an inherent limitation of contrastive pretraining methods.

3) The authors write: "Surprisingly, we find that M3AE benefits from a higher text mask ratio (50-90%), in contrast to BERT whose standard masking ratio is 15%." This statement is misleading because it suggests that M3AE benefits from a higher text mask ratio compared to BERT on the same downstream task. But actually, BERT embeddings are evaluated on language tasks while M3AE embeddings are evaluated on image tasks. It does not make sense to compare BERT's text masking ratio with M3AE's, since M3AE's text representation is never evaluated in a downstream task.

clarity,_quality,_novelty_and_reproducibility: Clarity - good; the paper is clear and well-written.

Quality - the experiments needs work. The main experimental result (that M3AE outperforms MAE) is not significant enough (weakness 1), and the other main claims in the introduction are also flawed (weaknesses 2-3)

Originality - ok. The concept of masked auto-encoding is not novel, although extension to the multimodal setting merits some consideration.
summary_of_the_review: See weaknesses 1-3 above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 1. The paper presents a novel Multimodal Masked Autoencoder (M3AE) model that learns a unified encoder for both vision and language data via masked token prediction. 
2. The generalization of masked autoencoders to work on multiple modalities is novel and has proven effective. 
3. Similar ideas have been investigated previously, namely leveraging unpaired data and paired text-image data for multimodal representation learning. 
4. Additional experiments and comparisons on vision-language tasks should be conducted to verify the effectiveness of the proposed approach. 
5. The paper has some degree of scientific novelty, but may not meet ICLR's very high standards.
justification_for_why_not_higher_score: 1. Similar ideas have been investigated previously, namely leveraging unpaired data and paired text-image data for multimodal representation learning. 
2. Additional experiments and comparisons on vision-language tasks should be conducted to verify the effectiveness of the proposed approach. 
3. The paper has some degree of scientific novelty, but may not meet ICLR's very high standards.
justification_for_why_not_lower_score: 1. The paper presents a novel Multimodal Masked Autoencoder (M3AE) model that learns a unified encoder for both vision and language data via masked token prediction. 
2. The generalization of masked autoencoders to work on multiple modalities is novel and has proven effective.
Decision: Reject