SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $e$ -Mix is a generalized version of  $i$ -Mix [48], designed to consistently apply methods across both discrete and continuous domains by applying the mixup strategy in the embedding space.
- ShED is a generalized version of ELECTRA [15]. ShED constructs the pretext task, which involves predicting shuffled embeddings.
- Capri applies contrastive learning to the token level representation by randomly masking the token and treating different tokens as negative pairs.
- MAE aims to reconstruct the input. However, here, MAE employs a linear decoder for the continuous domain and no decoder for the discrete domain.
Additionally, we regard the randomly initialized encoder, referred to as the Baseline, as one of the baseline to check the effectiveness of self-supervised pretraining.
Architectures. Following [83, 85], we use 12 layers for the transformer encoder with the hidden size 256, and 8 attention heads. For the decoder, we fix the hidden size 128, and 4 attention heads. However, we choose an appropriate number of layers for the decoder to demonstrate the effect of the decoder for MAE. We also utilize different hyperparameters for each modality as other baselines, but we find that the hyperparameters can be shared across modalities (See Appendix B).
Pretraining and transfer learning. To evaluate our method, we pretrain each dataset 100K iterations and 100 epochs transfer learning, overall experiments by following [85]. We pretrain entire networks, i.e., encoder  $f_{\theta}$ , decoder  $g_{\theta}$ , and projection header  $h_{\theta}$ , but we utilize only the frozen encoder  $f_{\theta}$  on transfer learning. When pretraining, the masking ratio can differ from the datasets. For the masking ratio hyper-parameter, we choose the best value among candidates suggested by the prior work [85].
3.2 Main Experiments
In-domain linear evaluation. We evaluate the pretrained representation on each in-domain downstream classification task. We report the performance of a linear classifier trained on top of the frozen features. The results in Table 1 demonstrate that our proposed method, MetaMAE, achieves state-of-the-art performance across the entire dataset. For instance, we obtain  $16\%$  accuracy gain  $(53.6\% \rightarrow 69.4\%)$  on Genomics. Moreover, we note that MAE has achieved moderate performance compared to other self-supervised learning (SSL) methods on these benchmarks, but MetaMAE demonstrates the ability to enhance MAE and outperform other SSL approaches. For example, MetaMAE achieves the best performance on Pfam  $(44.7\% \rightarrow 62.3\%)$  and LibriSpeech  $(60.2\% \rightarrow 79.8\%)$  with significant improvement, here is where MAE reported in [85] (i.e., MAE with linear decoder) was not the best among baselines.
Table 2: Cross-domain linear evaluation performance across multiple modalities. We report the Spearman correlation for Stability and Fluorescence datasets, and the classification accuracy (\%) for the rest. * denote the results from the DABS 1.0 paper.
Pretrain data	Transfer data	Baseline	SSL Framework
e-Mix	ShED	Capri	MAE	MetaMAE
Genomics	Genomics-OOD	8.6	9.7	7.3	5.5	22.2	37.2
Pfam	SCOP	8.0	5.7	10.7	2.0	7.9	11.8
Secondary	52.4	53.7	67.6	49.5	62.5	65.9
Stability	0.31	0.39	0.53	0.26	0.40	0.53
Fluorescence	0.04	0.20	0.27	0.06	0.06	0.31
LibriSpeech	Audio MNIST	33.1*	80.4*	67.3*	53.6	45.1	89.5
Fluent Loc	62.1*	60.9*	60.2*	59.8	61.7	66.7
Fluent Act	26.2*	29.9*	30.5*	28.3	26.8	38.4
Fluent Obj	30.1*	39.9*	39.4*	33.1	32.0	49.3
Google Speech	4.9*	19.2*	20.7*	13.7	9.5	46.8
VoxCeleb1	0.6*	2.4*	2.8*	1.6	1.6	7.4
ImageNet32	CIFAR-10	24.2*	39.4*	39.6*	48.7	46.0	59.2
CUB	1.6*	3.9*	3.0*	3.7	3.1	6.3
VGG Flowers	9.0*	26.0*	13.0*	18.6	22.2	36.3
DTD	7.4*	8.8*	18.4*	14.7	14.2	20.9
Traffic Sign	14.3*	65.1*	27.5*	28.0	32.0	67.1
Aircraft	2.7*	10.2*	5.6*	6.4	5.9	16.4
Table 3: Linear classification accuracy (%) pretrained on a vision-language dataset, MSCOCO.
Pretrain data	Transfer data	Baseline	SSL Framework
e-Mix	ShED	Capri	MAE	MetaMAE
MSCOCO	VQA	53.4	57.6	53.1	52.9	54.2	69.7
Mismatched-caption	49.8	50.1	50.6	49.6	49.3	70.5
Cross-domain linear evaluation. We evaluate our method on a diverse set of cross-domain downstream tasks including both classification and regression. We employ a linear classifier, or regressor trained on the frozen features as the in-domain setup. Table 2 shows that MetaMAE outperforms all the baselines across all the benchmarks consistently, except for one specific dataset. For example, we obtain  $9\%$  accuracy gain  $(80.4\% \rightarrow 89.5\%)$  on the linear classification performance of transfer setup from LibriSpeech to Audio MNIST. It is important to note that cross-domain downstream tasks, due to their wider range of variations for each domain, are typically more challenging to consistently excel in compared to in-domain tasks. This significant performance improvement demonstrates the applicability of MetaMAE in various cross-domain transfer learning scenarios across the modalities.
Multi-modal dataset evaluation. One important future direction for the modality-agnostic SSL research community is to bind all modalities under a singular model [99, 108]. Here, we believe MetaMAE can be quite a promising method to tackle this problem, e.g., managing multiple modalities on a single model supplemented by domain-specific embedding modules. To this end, we verify the possibility of MetaMAE for tackling unified multi-modal self-supervised learning. As shown in Table 3, MetaMAE outperforms other modality-agnostic SSL methods on the vision-language tasks where we believe this multi-modal learning ability can help when unifying the modalities for SSL.
We perform an ablation study on six modalities: time-series (PAMAP2), tabular (HIGGS), speech (LibriSpeech), multi-spectral image (EuroSAT), and token (Pfam and Genomics). Throughout this section, we report the in-domain linear classification accuracy (\%), unless otherwise specified.
Component analysis. In Table 4, we demonstrate the necessity of each component in MetaMAE by adding each component one by one: Deeper decoder  $g_{\phi}$  with a Transformer architecture, latent optimization via gradient-based meta-learning, and the task contrastive loss  $\mathcal{L}_{\mathrm{task - con}}$ . We first found
Table 4: Ablation study on each component of MetaMAE, namely the use of the decoder, latent adaptation using gradient-based meta-learning (Gradient-based), and task contrastive learning (Task contrast). We report the classification accuracy (\%) across six different modalities.
Decoder	Gradient-based	Task contrast	PAMAP2	Genomics	EuroSAT	LibriSpeech	HIGGS	Pfam
×	×	×	85.3	53.6	86.3	33.3	70.0	44.7
✓	×	×	86.5	65.2	87.4	64.1	70.5	61.3
✓	✓	×	88.3	69.4	87.4	64.5	71.1	61.3
✓	✓	✓	89.3	69.4	88.5	79.8	71.5	62.3
Table 5: Effect of the decoder size of MAE on the classification accuracy  $(\%)$ . We use three different datasets across modalities.
decoder size	EuroSAT	Pfam	LibriSpeech
prev. best	87.4	54.7	60.2
0	86.3	44.7	33.3
2	86.7	61.4	68.1
4	87.4	61.3	64.1
6	86.7	61.4	74.1
Table 6: Effect of the nearby token (i.e., Nearby-  $\mathcal{S}$  ) selection ratio  $r$  on the classification accuracy  $\left( \% \right)$  . We use three different datasets across modalities.
r ratio	PAMAP2	HIGGS	Pfam
0	87.5	71.1	62.0
0.1	89.3	71.5	62.3
0.5	88.2	70.8	62.0
1.0	84.2	70.1	62.1
that incorporating a deep decoder is a critical component in our framework, enabling domain-agnostic capabilities similar to the success of MAE on the image domain [33]. Thus, we here suggest that improving MAE for the domain-agnostic is quite a promising direction to explore.
In addition, Table 4 verifies the contribution of meta-learning schemes to the performance of MetaMAE. We found that the gradient-based latent optimization rule, which includes the utilization of Nearby- $S$ , is more beneficial. We also confirm that task contrastive learning is a critical component in our framework like recent meta-learning frameworks [28, 60, 104]. Note that this task contrastive learning scheme is exclusively applicable in gradient-based approaches, emphasizing the significance of the gradient-based latent optimization method for MetaMAE.
Importance of decoder size for MAE. To verify the effect of decoder size for MAE, we evaluate the linear evaluation accuracy on datasets where the original MAE (i.e., no decoder) performed worse than other baselines. As Table 5 shows, we found that MAE can achieve the best performance compared to baselines by choosing the proper decoder size, yet there is room for enhancement as shown in Table 4. This result demonstrates the superiority of MAE for tackling modality-agnostic SSL problems, where we believe the development of MAE would be an important direction to investigate. In this respect, we believe MetaMAE will serve as an important baseline in this field.
Nearby supports. We further analyze the effect of  $r$ , i.e., the Nearby- $S$  ratio. We conduct the experiment with  $r \in \{0, 0.1, 0.5, 1.0\}$ . We note that  $r = 0$  indicates the gradient updates without any help of queries (i.e., direct reconstruction of  $S$ ), and  $r = 1$  denotes the gradient updates with the entire queries near the  $S$ . As shown in Table 6, this approach is found to be beneficial compared to the direct reconstruction, and the small ratio is suggested to be proper  $r$ , e.g.,  $r = 0.1$  is the best. This is because it effectively bridges the gap between the latent representation and the latents of masked tokens, thereby enhancing the encoding of knowledge required for the reconstruction task.
Computational efficiency. MetaMAE might be perceived as compute-inefficient when incorporating MAE due to the computational demands of second-order gradients; however, our findings suggest otherwise. Although MetaMAE increases the total training time of MAE by approximately 1.4 times (with the one-step adaptation), we have observed that it is much faster to achieve the best performance of MAE: in Figure 2, we compare the accuracy under the same training wall-clock time with MAE, e.g., 1.9 times faster on
(a) LibriSpeech
(b) PAMAP2
Figure 2: Computation efficiency comparison of MAE and MetaMAE. We report the pretraining wall clock time.
Self-supervised learning (SSL). SSL, i.e., learning without human supervision, recently has demonstrated substantial success across fields including, computer vision [32], natural language processing (NLP) [18], and speech recognition [2], frequently show better transferability and generalization ability compared to conventional pretraining methods, e.g., training on labeled datasets [11]. To learn such representation, SSL optimizes the loss on a pretext task that does not require any human labels. For instance, pioneer works for SSL proposed such tasks based on data reconstruction through auto-encoding [7] such as context prediction [19], and in-painting [67]. Later on, multiple SSL works found that utilizing the domain-specific inductive biases can effectively learn representations in a self-supervised manner, including, colorization [107], solving jigsaw puzzles [64], counting the number of objects [65], and rotation prediction [27], to name a few. More recently, contrastive representation learning has garnered significant attention in SSL [100, 32, 11]. This technique maximizes the similarities of similar (i.e., positive) pairs and minimizes the similarities of dissimilar (i.e., negative) pairs, rather than focusing on training an instance classifier. To generate such positive pairs, multiple works rely on domain-specific inductive biases such as data augmentations [47], i.e., different augmented views as a positive pair. In addition, recent advances have been made with the development of various architectural components: e.g., Siamese networks [44], self-distillation [32, 9], asymmetric architectures [29, 12], and utilization of Transformer architectures [9]. Despite the success of these strategies, most existing SSL frameworks rely heavily on domain-specific inductive biases, which limits their applicability to new modalities.
Modality-agnostic SSL. Recently, several streams of work have emerged focusing on the development of more generalized SSL methods, specifically modality-agnostic SSL. For example, DACL [90] and  $i$ -Mix [48] utilize the idea of mixup [105] to propose domain-agnostic contrastive learning, and  $e$ -Mix [83] generalizes the concept of  $i$ -Mix to be embedding-level instead of input-level. Capri [85], as a variant of CPC [88], contrasts the predicted representations from randomly masked tokens. [84] develops generative models to learn data-dependent distortions for contrast. Instead of contrastive learning, ShED [83] (a generalized version of ELECTRA [15]) constructs the pretext task of replacing token detection with a masking strategy. DABS 2.0 [85] proposes a method to generalize MAE [33] to be modality-agnostic. In their approach, however, decoders are not utilized for discrete domains like BERT [18], while only a linear decoder is employed for continuous domains. In this paper, we suggest an effective modality-agnostic latent optimization for learning representations by interpreting masked prediction for MAE [33] in a novel manner.
Masked Auto-Encoder (MAE). MAE [33], i.e., predicting the masked parts with a given unmasked parts, has been extended to multiple applications [109, 37] across various domains [18, 39]. Among them, the recent combination of MAE with Transformer architecture [18, 33, 5] has shown promise in tackling SSL scenarios. For instance, BERT [18] utilized MAE for natural language processing (NLP) tasks, incorporating a linear layer into its architecture. Furthermore, multiple variants of MAE show impressive performance in various domains, by suggesting modality-agnostic SSL [3, 4], architecture-agnostic SSL [97, 51], multi-modal pretraining [95], and generative pretraining frameworks [21, 52]. In this paper, we focus on improving the most basic form of mask-modeling (i.e., MAE) for constructing a modality-agnostic SSL framework which remains under-explored, despite its potential significance, through the lens of meta-learning. It is worth noting that our interpretation of viewing MAE as a meta-learning framework can be applied to any other masked-modeling-based SSL frameworks where we believe combining our meta-learning regularization to such SSL methods would be an interesting direction to explore.
Meta-learning. Meta-learning [86], i.e., learning to learn by extracting common knowledge over a task distribution, has emerged as a popular paradigm for enabling systems to adapt to new tasks in a sample-efficient way. Under various applications across domains (e.g., computer vision [78], natural language processing [30], and robotics [101]), there have been significant efforts to design a variety of meta-learning schemes, including gradient-based [22, 53] and amortization-based approaches [75, 61] such as metric-based [92, 79], and neural processes [25, 24, 41, 87]. Typically, recent works have combined gradient-based meta-learning (or iterative functional update) with amortization-based schemes to enhance adaptation performance [74, 103]. Furthermore, there have been varieties of amortization-based schemes (such as neural processes) that utilize the recent success of contrastive learning into meta-learning, i.e., task contrastive learning [28, 60]. In this paper, we interpret MAE as an amortization-based meta-learning, which is further enhanced via the benefit of model-agnosticism of gradient-based meta-learning and task contrastive learning.
In this paper, we tackle modality-agnostic self-supervised learning (SSL), an important problem of SSL that consists of multiple real-world applications. To this end, we explore the possibilities of the Masked Auto-Encoder (MAE) in tackling modality-agnostic SSL which is quite under-explored, despite its potential. We propose MetaMAE, a novel and effective SSL framework that enhances MAE with meta-learning. Our key idea is to interpret mask reconstruction task of MAE as a meta-learning task, which allows us to treat MAE as a meta-learning framework. Based on this novel interpretation, we suggest a unique integration with advanced modality-agnostic meta-learning methods to improve the generalization of MAE. Our experiments demonstrate that MetaMAE significantly improves the performance of modality-agnostic SSL approaches across a diverse range of modalities.
Limitations and future work. While MetaMAE becomes a state-of-the-art approach for modality-agnostic SSL problems, it still inherits a general limitation of the MAE, namely the modality-specific masking ratio, i.e., the masking ratio may differ across modalities. This is due to our shared design elements with MAE, which include masking, encoding, and decoding. Recent works propose design choices for the masking scheme [50, 94], including automation, where incorporating these ideas into MetaMAE would be an intriguing future research direction, potentially enhancing our approach to be an even more effective modality-agnostic SSL framework.
Potential negative impacts. SSL often requires a large computation and a large network capacity, therefore raising environmental concerns, e.g., carbon generation [77]. As MetaMAE is built upon the SSL method (i.e., MAE), practitioners may need to consider some computation for successful training. To address this issue, efficient training methods [80, 40], distilling knowledge to a smaller network [36], or network sparsity schemes [31, 46] would be required to ameliorate such problems.
We thank Kyungmin Lee for providing helpful feedbacks and suggestions in preparing an earlier version of the manuscript. We also thank Sang Keun Choe for technical suggestions on the PyTorch implementation of meta-learning. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST); No.2021-0-02068, Artificial Intelligence Innovation Hub; No.2022-0-00959, Few-shot Learning of Causal Inference in Vision and Language for Decision Making) and Samsung Electronics Co., Ltd (IO201211-08107-01).
[1] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In IEEE International Conference on Computer Vision, 2015.
[2] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems, 2020.
[3] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, 2022.
[4] A. Baevski, A. Babu, W.-N. Hsu, and M. Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In International Conference on Machine Learning, 2023.
[5] H. Bao, L. D. 0004, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2022.
[6] S. Becker, M. Ackermann, S. Lapuschkin, K.-R. Müller, and W. Samek. Interpreting and explaining deep neural networks for classification of audio signals. CoRR, abs/1807.03418, 2018.
[7] Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems, 2006.
[99] S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023.
[100] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[101] A. Xie, A. Singh, S. Levine, and C. Finn. Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning, pages 40-52. PMLR, 2018.
[102] Z. Xie, Z. Z. 0022, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. H. 0004. Simmim: a simple framework for masked image modeling. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[103] J. Xu, J.-F. Ton, H. Kim, A. R. Kosiorek, and Y. W. Teh. Meta-learning with iterative functional updates. In International Conference on Machine Learning, 2020.
[104] Z. Ye and L. Yao. Contrastive conditional neural processes. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[105] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[106] Q. Zhang, Y. Wang, and Y. Wang. How mask matters: Towards theoretical understandings of masked autoencoders. In NeurIPS, 2022.
[107] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In European Conference on Computer Vision, 2016.
[108] Y. Zhang, K. Gong, K. Zhang, H. Li, Y. Qiao, W. Ouyang, and X. Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802, 2023.
[109] L. Zhou, H. Liu, J. Bae, J. He, D. Samaras, and P. Prasanna. Self pre-training with masked autoencoders for medical image classification and segmentation, 2023.
In this section, we provide the implementation details of MetaMAE, including architectures and hyperparameters for MetaMAE when pretraining and evaluation.
Architectural details. We summarize our architectures in Table 7, with the hyperparameter notation referred from [89]. We use token embedding for encoder inputs, and apply positional embedding to both encoder and decoder inputs, as suggested by [89]. Specifically, token embedding separates the input data into fixed-size tokens, while positional embedding uses a fixed, absolute position represented by a combination of sine and cosine functions. We describe the token size for each specific dataset in Appendix C.
Table 7: A Pytorch-like architecture description of MetaMAE.  $n \in  \{ 2,4,6\} ,p \in  \{ 0,{0.1}\}$  are the hyperparameters.
Component	Layer descriptions
Encoder fθ	TransformerBlock(dmodel=256, dff=512, h=8, Pdrop=p, GELU, LayerNorm=True) × 12
Decoder gφ	TransformerBlock(dmodel=128, dff=256, h=4, Pdrop=0, GELU, LayerNorm=True) × n
Projector hψ	Linear(256, 1028), BatchNorm1d(1028), Linear(1028, 128)
**Pretraining details.** We summarize our selected hyperparameters for pretraining each dataset in Table 8. Following [85], we pretrain MetaMAE for 100k iterations utilizing the AdamW optimizer [56] with both a learning rate and weight decay set at 1e-4. The batch size for pretraining and the strategy for selecting the mask ratio are detailed in [83, 85]. For the MetaMAE-specific hyperparameters, we observe that a certain set of hyperparameters can generally work across modalities, e.g.,  $(\alpha, \lambda, \text{decoder depth}) = (0.5, 0.1, 4)$  (see Table 9 in Appendix B), or can be shared within each modality, e.g.,  $P_{\text{drop}} = 0$  for Token modality (see Table 10 in Appendix B). Nevertheless, we recommend modality-specific values for optimal performance (refer to Appendix B for hyperparameter sensitivity details). We set the temperature term for the contrastive loss  $\tau = 0.5$  and the Nearby- $S$  ratio  $r = 0.1$ . For latent adaptation, the latent representation undergoes a single-step update with the update magnitude denoted by  $\alpha$ .
Table 8: Hyperparameters of MetaMAE for pretrain datasets.
Modality
Dataset	Time-series	Tabular	MS Image	Token	Speech	RGB Image	Vision-Language
PAMAP2	HIGGS	EuroSAT	Genom	Pfam	Libri	WaferMap	ImageNet32	MSCOCO
MetaMAE-specific hyperparameters
α	0.5	1.0	0.1	0.1	0.1	0.1	0.1	0.1	0.5
λ	1.0	1.0	1.0	0.01	1.0	1.0	0.1	0.1	0.1
decoder depth	4	6	4	2	4	4	6	4	2
Pdrop	0.1	0.1	0	0	0	0	0	0	0
Hyperparameters from DABS benchmarks
mask ratio	0.85	0.50	0.85	0.50	0.15	0.85	0.15	0.85	0.5
batch size	256	256	64	32	128	64	128	64	64
We note that to scale up experiments, it is essential to facilitate distributed parallelism by using libraries such as BETTY [13] when utilizing PyTorch for meta-learning.