Multimodal Masked Autoencoders Learn Transferable Representations
Authors: Xinyang Geng
Hao Liu
Lisa Lee
Dale Schuurmans
Sergey Levine
Pieter Abbeel
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multimodal
Self-supervised Learning
Pre-training
Masked Autoencoder
Representation Learning

Abstract: Building scalable models to learn from diverse, multimodal data remains an open challenge.
For vision-language data, the dominant approaches are based on contrastive learning objectives that train a separate encoder for each modality. While effective, contrastive learning approaches introduce sampling bias depending on the data augmentations used, which can degrade performance on downstream tasks. Moreover, these methods are limited to paired image-text data, and cannot leverage widely-available unpaired data. In this paper, we investigate whether a large multimodal model trained purely via masked token prediction, without using modality-specific encoders or contrastive learning, can learn transferable representations for downstream tasks. We propose a simple and scalable network architecture, the Multimodal Masked Autoencoder (M3AE), which learns a unified encoder for both vision and language data via masked token prediction. We provide an empirical study of M3AE trained on a large-scale image-text dataset, and find that M3AE is able to learn generalizable representations that transfer well to downstream tasks. Surprisingly, we find that M3AE benefits from a higher text mask ratio (50-90%), in contrast to BERT whose standard masking ratio is 15%, due to the joint training of two data modalities. We also provide qualitative analysis showing that the learned representation incorporates meaningful information from both image and language. Lastly, we demonstrate the scalability of M3AE with larger model size and training time, and its flexibility to train on both paired image-text data as well as unpaired data.


Under review as a conference paper at ICLR 2023
MULTIMODAL MASKED AUTOENCODERS LEARN TRANSFERABLE REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Building scalable models to learn from diverse. multimodal data remains an open challenge. For lasonala data, the dominant approaches are based on con- trastive learning objectives that train separate encoder for each modality. While effective, contrastive learning approaches introduce sampling bias depending on the data augmentations used, which can degrade performance on downstream tasks. Moreover, these methods are limited to paired image-text data, and cannot leverage widely-available unpaired data. In this paper, we investigate whether a large multimodal model trained purely via masked token prediction, without using modality-speciin encoders or contrastive learning, representations for downstream tasks. We propose a simple and scalable network architecture, the Multimodal Masked Autoencoder (M3AE), which learns uni- fied encoder for both vision and language data via masked token prediction. We provide an empirical study of M3AE trained on a large-scale image-text dataset, and find that M3AE is able to learn generalizable representations that transfer well to downstream tasks. Surprisingly, we find that M3AE benefits from higher text mask ratio (50-90%), ), in contrast to BERT whose standard masking ratio is 15%, due to the joint training of two data modalities. We also provide qualitative analy- sis showing that the learned representation incorporates meaningful information from both image and language. Lastly, we demonstrate the scalability of M3AE with larger model size and training time, and its flexibility to train both paired mmate-text data as well as unpaired data.
learn transferable
INTRODUCTION
With the rapid advances in neural architectures (Vaswani et al. 2017) and hardware performance, self-supervised pre-training has made tremendous progress in natural language processing (NLP) and vision (He et al, 2021 Devlin et al. 2018) Bao et al. 2021 Brown et al. 2020). The underlying idea, often referred as masked token prediction, is conceptually simple: the model learns predict removed portion of the data. Masked token prediction has enabled highly successful methods for pre-training in NLP and vision, including Transformer (Vaswani al., 2017). GPT (Brown al.) 2020), BERT (Devlin et al.) [2018). and MAE (He et al., 2021). These pre-trained representations have been shown to generalize well various downstream tasks. The cornerstone of these successes is that these methods excellently leverage large and diverse datasets. Indeed, with the scaling up of data diversity and model capacity, there is still no sign of plateau on generalization various downstream tasks (Devlin et al. 2018 He et al. 2021).
Driven by the successes in NLP and vision, there has been significant interest in improving visual representation learning by training on large and diverse multimodal datasets that contains both images and text. These datasets, such CC12M (Changpinyo et al., 202T) and YFCC100M (Thoeee ct al. 2015), often much more scalable than explicitly labeled datasets such ImageNet (Deng ctt al. 2009), and the diverse language data can provide rich supervision to train more generalizable representations.
The dominant paradigm for multimodal pre-training is cross- modal contrastive learning, such as CLIP (Radford et al.) 2021) and ALIGN (Jia al.) (021). These methods show that cross-modal contrastive learning models, trained on giant corpora of paired image-and-teed can generalize well to various downstream tasks. Despite these progresses, a major limitation for contrastive learning is that

---

Under review as a conference paper at ICLR 2023
Unmasked tokens
Mask
Encoder
Decoder
Patio
Patio
Palio
the
red
Red
entrance
Cetar
entranae
entrance
Figure I: Multimodal masked autoencoder (M3AE) consists of an encoder that maps language tokens and image patches shared representation space, and decoder that reconstructs the original image and language from representation.
it requires paired image-and-text data and therefore cannot leverage widely available unpaired data. In addition, contrastive learning based methods use separate encoders for image and text, making difficult for models to access information from different modalities at the same time. The separation of image and text encoders hinder the joint understanding of image and text.
To address the above limitations for visual representation learning, we propose a simple and scalable architecture called the multimodal masked autoencoders (M3AE) for learning a single unified model on large image and language data, without using modality-specific encoders contrastive learning. Based on MAE (He et al. 202T), M3AE trained purely via masked token prediction, Our key idea is to treat an image-ande and-aaddttee text pair long sequence of tokens consisting of embeddings of image patches and text. M3AE is trained simply by masking random patches of the input image and language tokens, and learning to reconstruct the masked pixels and text.
this paper, we provide an empirical study of M3AE trained on the multimodal CC12M (Changpinyo et al. 2021) dataset, and find that M3AE is able to learn generalizable representations that transfer well to downstream tasks such as image classification and out-of of distribution detection. We find that multimodal pre-training of M3AE CC12M achieves significantly higher performance on the ImageNet-1 k linear classification benchmark (Russakovsky et al. 2014) compared to pre-training on images only (MAE). Our strong results for M3AE demonstrate the generalization benefits of multimodal training for learning transferable representations across datasets.
Surprisingly, find that M3AE performs best when apply high mask ratio (75%) on language, while in contrast, language models like BERT (Devlin et al. 2018) conventionally use low mask ratio (15%) because language data highly semantic and information-denss We hypothesize that M3AE benefits from a higher mask ratio on text because it enforces a better joint understanding of vision and language during masked token prediction. We also provide qualitative analysis showing that the learned representation incorporates meaningful information from both image and language. Furthermore, we demonstrate the scalability of M3AE with larger model size and training time, as well as its flexibility train on both paired image-text data well unpaired data.
2 RELATED WORK
Self-supervised representation learning via reconstruction After the introduction Transform- ers (Vaswani al., 2017), self-supervised language modeling has made substantial progress in recent years. After pre-training on a large amount unlabeled data with reconstruction loss, Large-scale transformer language models like BERT (Devlin al. 2018) and GPT (Brown al. 2020) are highly successful in learning representations that generalize well to various downstream tasks. Taking inspiration from the success NLP, research have proposed wide variety of self-supervision method (Chen et al. [2020a] Dosovitskiy et al. 2020) [ao et al. 2021 He et al., (02T). iGPT (Chen al. 2020a) that operates sequences of pixels and reconstruct the unknown pixels. ViT (Doso-
2

---

Under review as a conference paper at ICLR 2023
vitskiy et al. 2020) studies masked patch prediction for self self-super learning. BEiT (Boo et al. 2021) proposes to predict discrete tokens (Van Den Oord et al. [2017] Ramesh et al.] (021). MAE (He) al. ) proposes randomly mask patches of the input image and reconstruct the missing pixels.
Heavily inspired by MAE and BERT, our M3AE brings together image and language data and learns
shared representation for both modalities by applying unified masked patch and token prediction objective.
Self-supervised representation learning via contrastive objectives Besides reconstruction, an- other major paradigm for self-supervised learning is contrastive learning, which models similarity and dissimilarity between two or more views of images or texts (Gao et al. al. 2021 Chen et al. 2020c He et al. 2020 den Oord et al.) 2018 Grill et al. 2020 Wu et al. 2018). SimCSE (Gao et al.
2021) proposes constructing positive sentence pair through applying Dropout. SimCLR (Chen et al.
2020b) studies applying random image augmentation for contrastive learning. Contrastive learning often rely heavily on data augmentation and can therefore introduce bias during training. Our M3AE does not rely on contrastive objectives so it can be applied without data augmentation.
Joint learning for language and image Learning representations for a single modality has high importance as extracts semantic formation from the raw data of modality. Learning a joint representation for several modalities is challenging since requires alignment between semantic information from different modalities, of which the information contained may vary drastically, Specifically, learning joint representation for vision and language has been a long standing problem in artificial intelligence. Recently, CLIP (Radford et al. (021)) successfully tackled this challenge by leveraging contrastive learning over large dataset of aligned text-image pairs. Several works followed this idea, further improving the joint representation. BLIP (Li et al. (022)) used noisy web data by bootstrapping the captions with synthetic ones. SLIP (Mu al. (021)) learned a joint
representation by combining CLIP (Radford et al. 2021) and SimCLR (Chen et al. (2020b) techniques and leveraging both a paired dataset, and a much larger image-only dataset. DeCLIP (Li et al. (02T) utilized more image-text pairs collected from CLIP (Radford et al. 2021) by adding multiple self-
supervised techniques. Inspired by BERT, other methods study cross-modie moddl matching loss (Chen
2019 Lu et al. 20199] Singh et al. 2021 Tan & Bansal 2019 Yu et al. 2022). FLAVVA (Singh et al. et al. datasets, Perceiver (Jacgle et al. 2021) proposes cross- cooss-ttention combining language and image modalities. CoCa (Yu et al. (022) combines cross-modal contrastive learning and autoregressive caption prediction. Our M3AE models provides a simple but effective alternative for learning joint representations by processing language tokens and image patches through shared secoder-decoded architecture. We train our model with masked token reconstruction loss, eliminating the need to handle each modality separately.
202T) employs both contrastive and multimodal training objectives on paired and image-only
MULTIMODAL MASKED AUTOENCODER (M3AE)
In this section introduce method, multimodal masked autoencoder (M3AE), which consists of an encoder that maps image and language to representation space, and a decoder that reconstructs the original image and language from the representation. We summarize the main architecture and training process of M3AE in Figure and Figure 2
Image-language masking. The first step of M3AE is to combine the language and image input into single sequence. Following standard natural language processing practice (Devlin et al.) 2018). tokenize the input text into a sequence of discrete tokens. For image input, we divide it into regular non-overlapping patches of pixels, following the practice of ViT (Dosovitskiy al.) 2020). Text tokens and image patches are then concatenated into single sequence.
For patches and tokens, we sample s random subset without replacement from a uniform distribution, and mask (i.e., remove) the remaining ones. A high masking ratio is applied to both text tokens and image patches, in order to eliminate information redundancy and make a sufficiently difficult task that cannot be easily solved by extrapolation from visible neighboring patches and tokens.
M3AE encoder. The M3AE architecture consists of two networks: an encoder and a decoder. The encoder is large transformer, following the architecuture of VIT (Dosovitskiy et al. 2020) and
3

---

Under review as a conference paper at ICLR 2023
Text prediction
Image prediction
A large bus sitting next very tall building
The stars in the blue sky
Multimodal Masked Autoencoders
lurge bus sitting next to a very building
Trees in winter storm
Façade of an old shop
The stars in the blue sky
Image text pairs
Images only
Figure 2: M3AE can learn representations from flexible mixture of image-text pairs and unpaired images using a unified model without relying on data augmentations.
BERT (Devlin et al. 2018). The encoder takes only unmasked (visible) language tokens and image patches as input. For language tokens, we first convert it into learnable embedding vectors and then apply ID positional encodings, following the standard practice (Devlin et al. 2018). For image patches, we a learnable lincar projection to convert them to image embeddings that have the same dimension as the language embeddings, and then apply 2D positional encodings, following the practice of MAE He et al. (202I). In order to distinguish the two different modalities, add two learnable vectors that represent language and images respectively to the corresponding modalities' embeddings. We call these "modality type encodings". Additionally, learnable CLS embedding (Devlin et al., 2018) is prepended to the beginning of the sequence. The combined language and image embeddings are then processed by series of transformer blocks obtain the final representation. Although the input consists of long sequences of image patches and text tokens, we can still train very large transformer encoders efficiently because the same only operates on small subset (e.g., 25%) of the full set.
M3AE decoder. Following MAE (He et al. 2021), we use a lightweight transformer-base decoder on the full set of tokens consisting of (i) encoded visible image patches, (ii) encoded visible text tokens, and (iii) mask tokens. Each mask token is shared, learned vector that indicates the presence of missing patch or token to be predicted. We add positional embeddings set in order to encode location information in mask tokens. We also add different set of modality type embeddings to visible tokens, similar the encoder. After the decoder transformer, we apply two linear projection output heads to compute the reconstruction. The image output head projects the decoder output corresponding to image patches to the same dimension as pixels in the original image patches. The language output head projects the decoder output of language to token logits. These output heads are then used for supervision during the self-supervised training of M3AE
all tokens in this full
Self-supervised training M3AE, Our M3AE reconstructs the input by predicting the pixel values for masked image patches and the token probabilities for masked language tokens. For image reconstruction, we compute the mean squared error (MSE) between the reconstructed and original images in the pixel space. For language reconstruction, reconstructed and original text. Our loss is a weighted sum of the image loss and the text loss. Similar to MAE (He et al. 2021 and BERT (Devlin et al.. 2018), compute the loss only the masked image patches and language tokens. Since M3AE processes image and language data uniformly by combining them into single sequence, a natural advantage for model is that it can be trained with the exact same loss on mixture of paired and unpaired data as shown in Figure 2 extending the applicability of our model beyond what possible with contrastive learning.
apply the cross entropy loss between the
significantly
EXPERIMENTS
4

---

Under review as a conference paper at ICLR 2023
In this section, we study the representation quality of M3AE. We aim to answer the following questions in our experiments: (1) Can M3AE learn generaliz- able visual representations that transfer well to down- stream tasks? (2) Does the learned representation incorporate meaningful information from both im- ages and language? (3) Does M3AE scale well with model size and training time?
90
Supervised
CLIP
80
Linear Accuracy
M3AE
70
60
To answer these questions, we first pre-train the M3AE model on a diverse shage-and-languaee dataset and evalute performance for downstream classification and out-of-distributin detection. We further evaluate the scalability of the model with re- spect to training epochs and model size. Finally, we provide detailed ablation study and qualitative analyses to inspect the quality of the learned repre- sentations.
50
40
0.1 0.3
1
Percentage paired image-text
Model
MAE
M3AF
CL.IP
Supv
Accuracy
44.6
61.3
69.0
81.8
M3AE text ratio
10%
20%
30%
100%
4.1 DATASETS
Accuracy
53.3
54.0
54.5
58.8
Pre-training datasets. M3AE is trained on Concep- tual 12M (CC12M) (Changpinyo et al. 202T). The original dataset images are provided in the form of internet URLs. Note that due to some expired URLs and non-English captions, we did not obtain the com- plete data in the dataset. For language data, we use the BERT tokenizer from Huggingface to tokenize the text. We provide more details about data preprocessing in Section A.
Figure 3: Comparison of M3AE, MAE, and CLIP ImageNet. M3AE significantly outperforms MAE. M3AE can flexibly leverage combination of paired image-text data and unpaired image only data. All models are ViT-B. MAE and M3AE are pretrained on CC12M for 50 epochs.
Downstream datasets. We assess model performance in a wider variety of distributions and tasks. We evaluate the image encoder transferability on ImageNet (Russakovsky et al.) 2014). We report top-1 validation accuracy of single 224x224 crop. We evaluate out-of-distributtin detection on CIFAR-100 and CIFAR-10 10 datasets (Krizhevsky et al. 2009).
4.2 EXPERIMENTS SETUP
Network architectures. Following MAE, we use ViT (Dosovitskiy et al., 2020) as the model architecture and consider three different sizes of ViT for the M3AE image and text encoder. We use the original ViT-B/16 and ViT-L/16 architectures (Dosovitskiy et al. 2020) for our encoder, as well as ViT-S/16 (Touvron et al. 2021) which is comparable to ResNet-50 in FLOPs and parameters. Following MAE (He et al. 2021), our decoder lightweight and has blocks and a width of 512. Full details about network architectures can be found in Section A.3
Pre-training setup. For a fair comparison with MAE, we train our model from scratch for the same number of epochs as MAE. The learnable temperature parameter is initialized to 0.01. The loss weights of image prediction and text prediction are set to and 0.5. The mask ratio for image and text are both set to 0.75. Refers to Section A.4 for more details.
Downstream evaluation setup. We evaluate our model transferability by performing linear classifi- cation frozen features, i.e., the pre-trained image encoder is fixed and serves as a feature extractor. After feature extraction, we train the linear classifier with the Adam W (Loshchilov & Hutter 2018) optimizer as the same He et al. (2021). More details be found in Section A.5
4.3 RESULTS
ImageNet Linear Classification. We evaluate performance on ImageNet under the linear classifica- tion setting. Linear classification, also called linear probing, is standard evaluation method used to

---

Under review as a conference paper at ICLR 2023
80
77:1
MAE M3AE
64.1
73.6
61.3
75-
Linear Accuracy
60
Finetuning Accuracy
67.8
73.1
50.7
50
66.0
44.6
40
55
35,6
49.0
31.5
50
M3AE MAE
30
46.8
ViT-S
ViT-B Model Size
ViT-L
2
Number of blocks finetuned
Figure 4: Left: Comparing the linear classification accuracy VIT model variants of different capacities (ViT- S/B/L). All models pre-trained for 50 epochs. M3AE scales well with model size, outperforming MAE in every setting. Right: Comparing finetuning different number of blocks VIT-L. All models are pre-trained for 50 epochs.
evaluate unsupervised or self-supervised representations. A randomly initialized final classification layer is trained while all other model weights are frozen.
Figure 3 shows the results linear classification. We report the results of VIT-B trained on ImageNet and CLIP (Radford al. (021) pre- pre-trained CC12M from prior work (Touvron et al. [221] Mu ct al. 2021). To study the flexibility of M3AE, we remove the text for portion of image-texi pairs, Le., 30% of paired image-text examples means 70% of CC12M image-text pairs become images only. lower percentage paired image-text data contains less information and therefore makes the task more difficult, since the model has infer the relation between visual and language concepts based on limited paired data.
The comparison between M3AE and the baselines are shown in Figure 3 M3AE significantly outperforms MAE by nearly 10 percent. CLIP strong baseline based on cross-modal contrastive learning. While
65
M3AE
63.9
61.0
Linear Accuracy
achieves higher accuracy than M3AF, it less flexible than our model since it can only use paired image-text data. In contrast, MBAE can leverage both paired image-text and unpaired image data without modifying the training procedure, as shown Fig- ureß] giving model strong potential to leverage diverse combination of unpaired single modality and multi-modal data. Notably, with M3AE pre-training, even adding 10% text annotations leads to signifi- cant boost accuracy over MAE (53.3% 45.2%).
55
50
45,9
45
0.15
0.5
0.75
0.9
Text mask ratio
Figure 5: Comparing M3AE with different text mask ratio. All models VYT-L trained for 50 epochs on CC12M. see that M3AE performs the best with a surprisingly high text mask ratio of 75%.
We make an important note that the linear classifica- tion performance of MAE pre-trained CC12M much lower than MAE pre-trained on ImageNet, and we hypothesize that such difference caused by the large domain gap between the two datasets. To confirm this hypothesis, we pre-trained a ViT-I. MAE on ImageNet for 800 epochs using the same hyperparameters top of our implementation, and obtained 73.5% accuracy linear classification, which exactly matches the original reported performance (He et al. 2). Thus, while our results cannot be directly compared the original MAE results (He al. 2021) pre-trained on ImageNet due to distribution mismatch, they demonstrate the strengths of multimodal training of M3AE for learning transferable representations across datasets.
ImageNet Fine-tuning, Following the fine-tuning experiment settings of MAE (He et al. 2021) perform partial fine-tuning pretrained models: fine-tune the last several layers while freezing

---

Under review as a conference paper at ICLR 2023
MAE on CIFAR-100 vs CIFAR-10 Standard Mahalanobis (UIROC=69001110
M3AE on CIFAR-100 vs CIFAR-10
Standard Mahalanobis AUROC=71 71. 415%
600
600
Count
400
test
Count
400
200
400 600 800 100012001400100118
250
500
750
1000 1250 1500 1750 2000
Score
MAE on CIFAR-100 vs CIFAR-10
M3AE on CIFAR-100 vs CIFAR-10 Max of Softmax Probs AUROC=78.495 78.457%
Max of Softmax Probs AUROC=77. 77.187%
in test
6000
test
Count
4000
out test
Count
4000
test
2000
2000
0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Score
Figure Out-of-distribution detection results CIFAR-100 (in-distribution) and CIFAR-10 (out-of- distribution). Upper shows results based on Mahalanobis outlier score, M3AE achieves 71.4% which higher than MAE's 69.0%. Lower shows results based on max over softmax score, M3AE achieves 78.5% which is also higher than MAE's 77.2%.
the others. Figure (right) shows that MBAEoutperforms MAE under different number of finetuned layers. This results suggest that M3AE representations more separable than MAE, which are also visualized in Section 4.4 We notice that with more layers being finetuned, the gap between M3AE and MAE becomes smaller. We believe the reason that there is large domain gap between internet image-text datasets and ImageNet images, therefore finetuning more layers may essentially destroy pretrained representations. Nonetheless, M3AE learns highly transferable representations that performs well even with the large domain gap between CC12M and ImageNet.
4.4 ANALYSIS
Model scaling, We also investigate the scaling behavior of M3AE with larger vision Transformer models. We pre-train MВAE and MAE with ViT-Small, ViT-Base and ViT-Large and perform ImageNet linear classification and finetuning using the learned representations. Figure 4sowws the effect of different model sizes. Our results indicate that M3AE scales well larger models, significantly outperforming MAE across different model sizes.
Dut-of-distribution detection. Some prior work demonstrated self-supervised learning approaches significantly improve OOD detection performance (Hendrycks al. 2019 2020 Fort al., (002I), where their self-supervised pre-training heavily relies on domain-specific data augmentations. We expect MAE perform well on OOD benchmarks and want study how M3AE performs compared with MAE.
the a author taking an elephant riding lesson. photo by < person
view of tiger head from the side
Figure 7: Visualization of attention between given text token and image patches on CC12M dataset. The text token for which we visualize the attention is bolded. We that the M3AE encoder is able attend to the correct objects.

---

Under review as a conference paper at ICLR 2023
We consider the difficult near-OOD as this is challenging and realistic problem; many methods can achieve high AUROC on the easier far-OOD benchmarks, but do not perform as well in near- near-OOD OOD tasks. The results shown in Figure 6 M3AE outperforms MAE in terms of both Mahalanobis outlier score (Lee et al. 2018) and max over sofumax score (Hendrycks et al. 2018).
Ablation text mask ratio. We also investigate the performance of M3AE under various text mask ratios. F shows holding the image patch mask ratio fixed (75%) and training for various text mask ratios. Surprisingly, the results indicate that M3AE benefits from high mask ratio (50%-90%) contrary to BERT (Devlin et al., 2018) whose typical masking ratio is 15%. We believe that this is the result of joint training of two modalities of data, where the masked language prediction can make use of information from both the visible language tokens and image patches.
Visualization of cross-modal attention weights. We are interested in what M3AE captures in multimodal attention weights. To do so, we visualize the M3AEencoder attention between given text token and all image patches, as well the attention between given image patch and all text tokens Yang & Zhang, (018)) in Figure and Figure M3AE learns to attend relevant concepts in both image and text, showing that our model able to infer relations between visual and language concepts.
close up view of one end the tar # #t filled with mas #fffaa car pone and lopped with staraw berries
100
Forming on
beillnng
Figure 8: Visualization of attention between a given image patch and all text tokens CC12M dataset The highlighted rectangle is image patch which we visualize the attention. Denser color the text denotes higher attention. visualization suggests that M3AE encoder is able attend to the correct words corresponding to the image patch.
Reconstruction visualization. We are interested in the reconstruction quality of pretrained М3АE. randomly sample examples from CC12M and the validation set ImageNet and show the results in Aegure9andd Figure In each reconstructed image, we include original unmasked tokens for better visual quality. We observe that our model infers holistic reconstructions across CC12M and ImageNet datasets, indicating it has learned numerous concepts.
Clustering analysis of representation. We perform t-SNE (Van der Maaten & Hinton 2008 visualizations of the learned representation of M3AE and MAE for 10 classes ImageNet validation
Figure 9: Masked image reconstruction ImageNet validation images. For each triplet, show the ground- truth (left), masked image (mid) and our M3AE reconstruction (right).
8

---

Under review as a conference paper at ICLR 2023
RICC
CARKIER
ECRREITE
Figure Masked image reconstruction CC12M images. each triplet, we show ground-truth (left), the masked image (mid) and our M3AE reconstruction (right).
tench
killer whale
Ice bear
bicycle-bullee boyyfte-bulttfor--two
hard disc mortar
garden spider
kuvasz
guenon
crash helmet
MAE
M3AE
t-SNE visualization for learned representations of 10 classes on ImageNet validation set. Left is MAE and right is M3AE. The representation of M3AE clusters much stronger together with the semantic labels compared to MAE representations.
Figure 11:
set in Figure TT Compared to MAE, M3AE successfully clusters together images that correspond to the same semantic label.
5 CONCLUSION
In this paper, propose M3AE, simple but effective model that learns multimodal representation from image and language data without the need for contrastive objectives. We show that by pre- training with diverse image and language data, our model can learn shared representations that generalize well to downstream tasks. Due to its flexibility and scalability, M3AE is especially suitable for learning from extremely large-scale datasets, and we envision that such pre-trained models can be broadly applicable in many downstream tasks, such as visual reasoning (Ding et al. 2021), dialog systems (Alayrac al. 2022) and language guided image generation (Ramesh al. 2021 2022).
One major limitation of this work is that the large-scale pre-training of M3AE consumes significant amount of energy. We hope that with the rapid progress of efficient hardware and renewable energy sources, this limitation can be overcome in the near future. Like any other general purpose representation learning models, our model could have both positive (e.g., enabling access to more information by improving machine translation) and negative (e.g., loss of jobs with more automation) impact on the society. These impacts are broadly applicable to pre-trainnt trained models in general and not specific this work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv: Arxiv-2204. 220..44944,1 2022.
Hangho Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 0rX.v:2/06.08254, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing
web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691- 1703. PMLR, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020c.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. arXiv preprint arXiv: Arxiv-1909, 11740, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 245-255. leee, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning. Advances in Neural Information Processing Systems, 34, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-diitrribttin detection. arXiv preprint arXiv: Arxiv-2/06..03004 2021.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence
embeddings. ArXiv, abs/2104.08821, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altch'e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach self-supervised learning. ArXiv, abs/2006. 07730. 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735, 2020.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv22111.0337,,222 2021.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure, arXiv preprint 1tf2::881246600 2018
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In Proceedings of the 36th International Conference on Machine Learning, ICMI, 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2712-2721. PMLR, 2019.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distributin robustness. arXiv preprint arXiv: Arxiv- 0000, 2020.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arXiv preprint arXiv: Arxin-2103.03206, 2021.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- vision-language understanding and generation. arXiv preprint arXiv:2201.12088 2022.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang. Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive aaguage--mmgg pre-training paradigm. arXiv preprint arXiv:2) 10.05208, 2021.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task agnostti visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv: Arxiv-1908.02265, 2019.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. arXiv preprint arXiv:2 12. 12. 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. 7748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 06125, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge (2014). arXiv preprint /
2(3), 2014.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. arXiv preprint arXiv:2 12.02/22444222. 2021.

---

Under review as a conference paper at ICLR 2023
Hao Tan and Mohit Bansal. Lxmert: Learning cross- mods-ttt encoder representations from trans- formers. arXiv preprint arXiv: Arxiv-1908. 1949,,0490, 2019.
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfc:100m: The data in multimedia research. arXiv preprint arXiv: Nrxiv-1503.0188 2015.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv,
2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3733 3742, 2018.
Jie Yang and Yue Zhang. Nerf++: An open-source neural sequence labeling toolkit. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018. http/ //aclweb.. wtt.///ntoooogggg.o.......
4013
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprint arXiv: Arxiv-2255.0199
2022.
12

---

Under review as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
A. PRE-TRAINING DATASETS
Conceptual 12M (CC12M Changpinyo et al. 2021) contains approximately 12M of image-text pairs, the original dataset images are provided in the form of internet URLs. Note that due to some expired URLs and oon-Enliish captions, we did not obtain the complete data the dataset.
A.2 DOWNSTREAM DATASETS
We evaluate the image encoder transferability on ImageNet (Russakovsky et al. 2014). We report top-
1 validation accuracy of a single 256x256 crop. We evaluate evaluate out-of out-of-distribttinn distribbution detection on CIFAR-I00 and CIFAR-I0 datasets (Krizhevsky et al., 2009). Table provides the detailed information of these datasets.
DATASET
Classes
Train size
Test size
Evaluation metric
CIFARIO
I0
50,000
10,000
Accuracy
CIFARIO0 100
100
50,000
10,000
Accuracy
ImageNet
1000
1, 1,, 167
50,000
Accuracy
Table 1: Details of downstream datasets
A.3 NETWORK ARCHITECTURES
Following MAE, we use ViT (Dosovitskiy et al. 2020) as the model architecture and consider three different sizes of ViT for the M3AE image and text encoder. The model consists of stack of standard Transformer blocks /Vaswani et al. (017)) and each Transformer block consists (Dosovitskiy self-attention and an MLP. We the original ViT-B/16 and ViT-L/16 architectures et al. 2020 for our encoder, as well as ViT-S/16 (Touvron et al. 2021) which is comparable to decoder is lightweight
multi-heau
ResNet-50 in FLOPs and parameters. Following MÄE (He et 20211, and has blocks and a width of 512. As in MAE, since our encoder and decoder have different width, adopt linear projection layer after the encoder match the dimension. For linear probing, use the auxiliary CLS token for training the classifier as done in MAE.
A.4 PRE-TRAINING HYPERPARAMETER
For the pre-training of M3AE and MAE, we follow the hyperparameters of the original MAE, We keep the optimizer, learning rate, weight decay the same the original MAE ImageNet. The only additional hyperparameters unique to M3AE are text token mask ratio and text token classification loss weight. We provide all the hyperparameters in Table where the same hyperparameters used to train network of all sizes and epochs. The base learning rate corresponds to the learning rate of 256 batch size, and it is linearly proportionally scaled according the actual batch size.
A.5 DOWNSTREAM EVALUATION HYPERPARAMETERS
For downstream tasks of linear classification on ImageNet and OOD detection on CIFAR, we use the same hyperparameters for M3AE and MAE, We list the hyperparameters for ImageNet 1K linear classification in Table 3 and OOD detection for CIFAR in Table
A.6 COMPUTATION RESOURCES
All the experiments are performed on the Google Cloud TPU platform. We implement our model using JAX and parallelize the large batch training across many TPUs with data parallelism. For all the pre-training, we use batch size 4096. We report the total amount of compute and the type of resources used Table
13

---

Under review as a conference paper at ICLR 2023
Hyperparameter
M3AE
MAE
Optimizer
AdamW
Base learning rate
1.5e-4
Weight decay
0,05
Optimizer momentum
P1 = 0.9, 02 = 0.95 4096
Batch size
Learning rate schedule
cosine decay 5
Warmup epochs
Image data augmentation Image patch mask ratio Text token mask ratio
RandomResizedCrop
0.75
0.75
N/A
Text token cross entropy loss weight
0.5
N/A
Table 2: Hyperparameters for pre-training M3AE and MAE CC12M
Hyperparameter
M3AE and MAE
Optimizer
LARS
Base learning rate
0.1
Weight decay
0
Optimizer momentum
0.9
Batch size
2048
Learning rate schedule
cosine decay
Epochs
90
Warmup epochs
Image data augmentation
RandomResizedCrop
Table 3: Hyperparameters for linear classification on ImageNet 1K
Hyperparameter
M3AE and MAE
Optimizer
AdamW
Base learning rate
0.001
Weight decay
0.05
Optimizer momentum
81 = 0.9, B2 = 0.999 1024
Batch size
Learning rate schedule Epochs
cosine decay
50
Warmup epochs
Image data augmentation
RandAugment
Label smoothing
0.1
Drop path
0.1
Table 4: Hyperparameters for fine tuning ImageNet.
Hyperparameter
M3AE and MAE
Optimizer
Adam W
Base learning rate
0.001
Weight decay
0.05
Optimizer momentum
31 = 0.9, 83 0.999 1024
Batch size
Learning rate schedule
cosine decay
Epochs
100
Warmup epochs
10
Image data augmentation
RandAAgment
Table 5: Hyperparameters fine tuning on CIFAR10.
14

---

Under review as a conference paper at ICLR 2023
Model
VIT-S
VIT-B
VIT-L
MAE 16.5h (v3-64)
8.5h (v3-128)
11.5h (v3-128)
M3AE
9.5h (v3-128)
5h (v3-256)
10h (v3-256)
Table TPU pod size and compute hours used for training 50 epochs of M3AE MAE on CC12M.
15


Reviews:
[{"review_id": "yUs8vu6LrR", "reviewer": "Reviewer_1MQ5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an empirical study of M3AE trained on a large-scale image-text dataset, and find that multimodal masked autoencoder is able to learn generalizable representations that transfer well to downstream tasks. ", "summary_of_the_review": "The paper is about the new approach for pre-training vision-language models by the proposed multimodal masked autoencoder. Although the approach is interesting, it is incremental and not novel. Some of the experiments demonstrating the superiority of the proposed approach are also unclear.\n\nThe proposed M3AE haven't performed experiments on vision-language down-stream tasks, it's not clear that the approach can learn good multimodal representation. Although the model can achieve better performances compared with MAE, M3AE uses larger dataset (e.g. CC12M). The performance gap between M3AE and CLIP is large.\n\nSome important ablation study experiments, such as different mask ratios of MAE and different encoders for visual/textual inputs, aren't conducted.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well written and this reviewer enjoyed reading it.\n\n2. The proposed approach is simple and intuitive.\n\n3. It shows promising results on image classification and out-of-distribution detection tasks.\n\nWeakness:\n1. The idea lacks of novelty. \n\n2. No experiments for vision-language tasks have been performed (e.g., text-to-image retrieval, VQA, visual reasoning, etc). The authors only fine-tune their pre-trained models for visual tasks, such as out-of-distribution detection and image classification. As the paper targets at multi-modal pre-training, it's essential to provide the experiments for vision-language down-stream tasks!\n\n3. Figure 2 is confusing. For example, in the lower left corner, the text \"Trees in a winter storm\" even appears in the space of its right image. \n\n4. Ablation study experiments aren't sufficient, it's important to conduct the experiments, e.g., different mask ratios of MAE.\n  \n5. Why not use two separate encoders for visual and textual feature extraction? The proposed M3AE only use one encoder for these two kinds of modalities together, but this may make the models confused. it's better to perform ablation study to use two different encoders, so that we can figure out the necessity of such design.\n\n6. The comparison is unfair, the proposed method uses more vision-language data (e.g. CC12M) for pre-training, and compare the models with MAE, which is only pre-trained on ImageNet. The performance gap between M3AE and CLIP is still very large. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "unb23F9yyM3", "reviewer": "Reviewer_ZSrM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper scales masked autoencoders (MAE) to image-text multimodal data, it investigates the use of modality-agnostic unified encoders, which learns generalizable representations across a number of downstream tasks.", "summary_of_the_review": "The paper is overall well-motivated and well-written, yet the experiment part seems to fall short with a lack of discussion and comparison with other baselines.", "strength_and_weaknesses": "Strengths:\n(a) The paper is overall well-written, the problem well formulated and novel to me. Generalizing masked autoencoders to work on more modalities is definitely interesting to the community.\nWeaknesses:\n(a) There seems to be a lack of comparison with other baselines. The paper only compare with CLIP and MAE, while not other baselines, such as MultiMAE[1], using the same modality-agnostic setting, were not included.\n(b) The author should also include a discussion on how is M3AE different from MultiMAE.\n[1] MultiMAE: Multi-modal Multi-task Masked Autoencoders, ECCV 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "kTJPdOHa6aK", "reviewer": "Reviewer_qu2E", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the multimodal representation learning in the masked auto-encoder way. The work is motivated by the fact that many existing multimodal learning approaches require a large amount of paired text-image data. This work builds upon MAE, extending it to text modality as well. The proposed method can handle both the unpaired image data and paired image-text data. The experiments show a clear improvement over MAE in several tasks, and show the effectiveness of the learned transferrable representation. \n\n--- post-rebuttal ---\\\nThanks for the authors' response. I have carefully read the response and other reviewers' comments. I believe the current form of this paper  is not ready for ICLR publication. I keep my score. ", "summary_of_the_review": "Overall, the proposed solution is rational and mostly well presented. While my major concerns are lack of comparison with the existing method and no clear motivation given billions of paired text image data available. ", "strength_and_weaknesses": "Strength \n\n-The idea of applying masked autoencoder strategy to multimodality data is rational. The finding on higher mask ratio on text performs better than the low mask ratio on pure text is interesting. The explanation makes sense to me. \n\n-The experiments show the strength over MAE in different scenarios and applications.  \n\n \n\nWeakness \n\n-There is other work also working on the similar motivation, i.e., leveraging unpaired data and paired text-image data for multimodal representation learning. BEIT-V3 [Wang et al.,], where they demonstrate an exceptional performance over many existing solutions including CLIP on a number of downstream tasks. It seems this paper misses the discussion on this work. And I’d also like to see the experimental comparison with BEIT-3 to show the advantages. \n\n[Wang et al.,] Wang et al.,Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks, https://arxiv.org/abs/2208.10442 \n\n-I have a little concern on the effectiveness on the large-scale data, such as several hundreds of millions of data points or even larger. The current training is on CC-12M, and shows a good improvement over MAE. However, when the number of training samples gets larger, I am not sure if this improvement margin can be preserved.  \n\n-Following the above point, thanks to the effort on the large scale data curation LAION, there are billions of image –text noisy pairs available. Please explain the application scenario of this method. \n\n-Figure 9, the reconstruction result from MAE is missing. I’d like to see a comparison to show the strength. \n\n-Other minors: In page 3, “For patches and tokens, we sample s random”, not sure what “s” means here. \n\nIn page 7, there is a space in “M3AEoutperforms”  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Rawpv6hLAd", "reviewer": "Reviewer_GbHq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an extension of Masked Autoencoders (MAE) to the vision-language domain called Multi-Modal Masked Autoencoder (M3AE). Image-caption pairs are partially corrupted by masking some image patches and some language tokens. The non-masked image patches and language tokens are passed through an autoencoder based on transformers, which is trained to reconstruct the full image-caption, including masked parts. The embeddings of image patches that are learned by the transformer encoder are transferred to downstream image tasks, such as classification and out-of-distribution detection. ", "summary_of_the_review": "See weaknesses 1-3 above. ", "strength_and_weaknesses": "Strengths: The paper is easy to understand, with the methodology clearly presented. The results section includes some nice qualitative analysis and visualizations to help understand the representations that are learned by M3AE.\n\nWeaknesses: \n\n1) The empirical support for the proposed approach is weak.\n\nOne of the main takeaways from the experiments (Figure 3-5) is that M3AE outperforms MAE in the downstream ImageNet classification task. This is a good sanity check, but it is not significant nor surprising. The M3AE framework involves predicting text token logits from image patches, which is similar to a supervised classification task where the text is providing labels for training the image encoder. On the other hand, MAE is completely unsupervised. \n\nIt would be better to compare M3AE against other vision-language pretraining methods. However, no vision-language pretraining baselines are provided except for CLIP which outperforms M3AE by a large margin in Figure 3.\n\n2) In the introduction, the authors make the strong claim that M3AE offers two benefits over contrastive pretraining methods, but neither benefit is demonstrated in the experiments.\n\nFirst, the authors emphasize that contrastive learning \"requires paired image-and-text data and therefore cannot leverage widely available unpaired data.\" This suggests that leveraging unpaired data will make M3AE perform better than CLIP. But as shown in Figure 3, CLIP does much better than M3AE. To support this claim, the authors should either (i) decrease the paired data for CLIP to e.g., 10-30% and show that M3AE does better than CLIP for the same amount of paired data in Figure 3, or (ii) add some unpaired image or text data for M3AE and show that it can do better than CLIP by leveraging this unpaired data.\n\nSecond, the authors claim that the \"separation of image and text encoders\" in contrastive learning \"hinders the joint understanding of image and text\", but they provide no proof of this. To support this claim, they should show that M3AE performs better than contrastive learning on a multimodal downstream task. Also, note that separation of image and text encoders is not necessary in contrastive learning (e.g., GLIPv2 enables fusion between image and text features before the contrastive objective), so this is not an inherent limitation of contrastive pretraining methods.\n\n3) The authors write: \"Surprisingly, we find that M3AE benefits from a higher text mask ratio (50-90%), in contrast to BERT whose standard masking ratio is 15%.\" This statement is misleading because it suggests that M3AE benefits from a higher text mask ratio compared to BERT on the same downstream task. But actually, BERT embeddings are evaluated on language tasks while M3AE embeddings are evaluated on image tasks. It does not make sense to compare BERT's text masking ratio with M3AE's, since M3AE's text representation is never evaluated in a downstream task.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject