SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Huiwon Jang $^{A*}$  Jihoon Tack $^{A*}$  Daewon Choi $^{B}$  Jongheon Jeong $^{A}$  Jinwoo Shin $^{A}$ \
 $^{A}$ Korea Advanced Institute of Science and Technology (KAIST) $^{B}$ Korea University\
{huiwoen0516, jihoontack}@kaist.ac.kr
Despite its practical importance across a wide range of modalities, recent advances in self-supervised learning (SSL) have been primarily focused on a few well-curated domains, e.g., vision and language, often relying on their domain-specific knowledge. For example, Masked Auto-Encoder (MAE) has become one of the popular architectures in these domains, but less has explored its potential in other modalities. In this paper, we develop MAE as a unified, modality-agnostic SSL framework. In turn, we argue meta-learning as a key to interpreting MAE as a modality-agnostic learner, and propose enhancements to MAE from the motivation to jointly improve its SSL across diverse modalities, coined MetaMAE as a result. Our key idea is to view the mask reconstruction of MAE as a meta-learning task: masked tokens are predicted by adapting the Transformer meta-learner through the amortization of unmasked tokens. Based on this novel interpretation, we propose to integrate two advanced meta-learning techniques. First, we adapt the amortized latent of the Transformer encoder using gradient-based meta-learning to enhance the reconstruction. Then, we maximize the alignment between amortized and adapted latents through task contrastive learning which guides the Transformer encoder to better encode the task-specific knowledge. Our experiment demonstrates the superiority of MetaMAE in the modality-agnostic SSL benchmark (called DABS), significantly outperforming prior baselines. Code is available at https://github.com/alinlab/MetaMAE.
Self-supervised learning (SSL), i.e., learning without human supervision, recently has demonstrated substantial success across fields including, computer vision [32, 11, 29, 47, 33, 5, 102], natural language processing (NLP) [18, 49, 55, 70], and speech recognition [2, 38, 39]. The efficacy of SSL is derived by extracting transferable knowledge from unlabeled datasets, a feature that manifests significant utility for various downstream tasks such as classification and segmentation. As a result, SSL has become an indispensable technique in real-world applications (for instance, industrial contexts like medical imaging [26]), not only improving the performance on new datasets but also reducing a significant amount of computations and costs, e.g., expert annotation poses significant costs [68, 16]. However, despite the importance of SSL in such fields, recent advancements have been predominantly focused on specific domains (e.g., images and NLP) where the majority of existing SSL frameworks on such domains require modality-specific knowledge, thereby constraining the applicability and scalability of previous works across new modalities.
To tackle this issue, we draw attention to the recent success of the Masked Auto-Encoder (MAE) framework [33], which eliminates the need for modality-specific inductive biases. Initially presented as a generative model [91, 67], the MAE models the network to reconstruct the original input
Figure 1: An overview of the proposed Meta-learned Masked Auto-Encoder (MetaMAE): we adapt the amortized latent  $\mathcal{Z}_{\mathbf{x}}$  of the Transformer encoder  $f_{\theta}$  using gradient-based meta-learning on the Transformer decoder  $g_{\phi}$  to enhance the reconstruction, then maximize the alignment with optimized latent  $\mathcal{Z}_{\mathbf{x}}^{*}$  to guide the Transformer encoder towards improved predictions via task contrastive learning.
signal based on the randomly masked part of the signal. Recently, the integration of MAE with the Transformer architecture [33] has resulted in a powerful SSL framework for various domains, such as vision [33, 5, 10], NLP [18, 70], and tabular [59] datasets. For instance, it has been demonstrated that BERT [18], utilizing a Transformer encoder with a linear decoder, can effectively transfer to diverse tasks for the NLP domain. On the other hand, it has been suggested that utilizing a decoder of a deep Transformer architecture is essential for applying MAE within the vision domain [106], achieving remarkable performance [33]. Building on these insights, we found that MAE is quite a promising direction for modality-agnostic SSL: our experiment demonstrates that using a deep Transformer decoder for MAE significantly outperforms previous modality-agnostic SSL frameworks (see Table 5). In this paper, we suggest to further exploit the benefits of MAE to build a unified SSL framework.
Contribution. We propose Meta-learned Masked Auto-Encoder (MetaMAE), a novel modality-agnostic SSL framework that leverages the power of meta-learning; see the overview in Figure 1. Our key idea is to interpret MAE as a meta-learning framework, thereby improving the generalization through the use of advanced meta-learning schemes. To be specific, we interpret the data reconstruction itself as a task, where the Transformer meta-learner is adapted through amortization of the support set (i.e., unmasked tokens) to predict the query set (i.e., masked tokens). Based on this interpretation, we propose a novel integration of two advanced meta-learning techniques to enhance MAE; namely the use of gradient-based meta-learning [22] and task contrastive learning [28, 60].
- Latent Adaptation via Gradient-based Meta-learning: We suggest adapting the amortized latent to better reconstruct the given support set (and the nearby tokens) through gradient-based meta-learning on the decoder [74]. Then, the optimized latent is used to condition the decoder for the query prediction. This approach generally eases the task compared to the direct reconstruction, thereby streamlining and improving the task adaptation process.
- Task Contrastive Learning: To further leverage this optimized latent, we suggest utilizing task contrastive learning [28, 60]. Specifically, since both the optimized and predicted latents originate from the same task, we aim to maximize their similarity while minimizing their similarity with other tasks. This prompts the Transformer encoder to produce predictions closely aligned with the optimized latent, effectively guiding the Transformer to better encode the task knowledge.
We verify the efficacy of MetaMAE through extensive evaluations on multiple data modalities from modality-agnostic SSL benchmarks (i.e., DABS 1.0 [83] and 2.0 [85]), including time-series, tabular, discrete token, multi-spectral images, speech, and multi-modal datasets. Overall, our experimental results demonstrate strong results, consistently and significantly outperforming previous modality-agnostic SSL methods in linear evaluation. For instance, measured with classification accuracy  $(\%)$ , MetaMAE improves the prior state-of-the-art results by  $85.3 \rightarrow 89.3$  on PAMAP2 [71],  $53.6 \rightarrow 69.4$  on Genomics [72], and  $60.2 \rightarrow 79.8$  on LibriSpeech [66] datasets. Moreover, we also demonstrate that MetaMAE significantly improves the linear evaluation performance on cross-domain datasets, indicating the improved transfer ability of MAE through meta-learning.
2 Meta-Learning Modality-Agnostic Masked Auto-Encoder
In this section, we present Meta-learned Masked Auto-Encoder (MetaMAE), a novel and effective modality-agnostic self-supervised learning (SSL) framework. Our key contribution is to tackle the modality-agnostic SSL problem with in-depth utilization of MAE, which was quite under-explored in the field. Based on our novel interpretation that views MAE as a meta-learning framework (in Section 2.1), we improve the transfer ability of MAE by suggesting enhanced modality-agnostic meta-learning techniques including latent optimization-based meta-learning and task contrastive learning (in Section 2.2). Our framework is visually depicted in Figure 1, and the pseudo-code is provided in Algorithm 1.
Problem setup. We first describe the problem setup of our interest, modality-agnostic SSL. This problem aims to learn a transferable representation from the unlabeled dataset without utilizing the modality-specific inductive biases. For a given unlabeled pretrain dataset  $\mathcal{D}_{\mathrm{pretrain}} = \{\mathbf{x}_i\}_{i=1}^N$ , where  $\mathbf{x} \in \mathbb{R}^d$  represents an input sampled from a certain data-generating distribution in an i.i.d manner, our objective is to train an encoder  $f_{\theta}$  that can linearly separate the given labeled transfer dataset drawn from a similar or the same data-generating distribution.
2.1 Rethinking Masked Auto-Encoder as a Meta-Learning Framework
Meta-learning [86] aims to extract and utilize the knowledge from the distribution of tasks to better solve a relevant task. This problem is typically approached by training a meta-learner that can transfer its knowledge to a task-specific model through adaptation, where the performance of the meta-learner is evaluated on the basis of how well each adapted model performs on the corresponding task. To learn such a meta-learner, a standard way is to use a set of support set samples to adapt the task-specific model from the meta-learner and use another disjoint set of samples, called query set samples to evaluate the adaptation performance [92, 82].
Mask prediction as a modality-agnostic task. MAE is an SSL technique that trains an autoencoder to reconstruct the original input signal with a randomly masked part of the signal. To implement such a technique, recent works utilize the Transformer architecture for the autoencoder design which is necessary for successful training. To use Transformer for MAE, the input data is broken down into non-overlapping units coined tokens (e.g., patches for images, and words for languages) where such tokens are divided into two disjoint sets (unmasked and masked) for MAE modeling, i.e., the Transformer autoencoder predicts the masked token using the unmasked tokens.
Our key insight is to interpret the signal reconstruction of MAE as a meta-learning task, where two disjoint unmasked and masked token sets are viewed as support and query sets to adapt and evaluate the Transformer meta-learner. To be specific, the Transformer encoder extracts the task knowledge through amortization of the support set, where this amortized latent adapts the Transformer decoder to predict the query set of the task. Formally, for a given data sample  $\mathbf{x}$ , we first divide the signal into two disjoint sets, namely the support set  $S_{\mathbf{x}}$  and the query set  $\mathcal{Q}_{\mathbf{x}}$ , by utilizing the tokenizer operation  $\mathrm{tokenize}(\mathbf{x}) \coloneqq \{(m, \bar{\mathbf{x}}^{(m)})\}_{m=1}^{M} = S_{\mathbf{x}} \cup \mathcal{Q}_{\mathbf{x}}$ . Then, for a given Transformer encoder  $f_{\theta}$  and decoder  $g_{\phi}$ , MAE minimizes the discrepancy between the predicted token and the corresponding masked token (i.e., the query sample) as:
$$
\mathcal {L} _ {\mathrm {M A E}} (\theta , \phi ; \mathcal {Q} _ {\mathbf {x}}) := \sum_ {(q, \bar {\mathbf {x}} ^ {(q)}) \in \mathcal {Q} _ {\mathbf {x}}} d \left(\bar {\mathbf {x}} ^ {(q)}, g _ {\phi} ^ {(q)} \left(\mathcal {Z} _ {\mathbf {x}}\right)\right) \text {w h e r e} \mathcal {Z} _ {\mathbf {x}} = f _ {\theta} \left(\mathcal {S} _ {\mathbf {x}}\right), \tag {1}
$$
where  $d(\cdot, \cdot)$  is a discrepancy function:  $\ell_2$  norm for continuous (e.g., time-series, speech) and cross-entropy for discrete (e.g., token) datasets, respectively. Based on this interpretation, we improve the transfer ability of MAE (for modality-agnostic SSL) through a novel integration of two effective modality-agnostic meta-learning techniques to MAE.
2.2 MetaMAE: Improving Masked Auto-Encoder through Meta-Learning
We now describe our method, MetaMAE, which further improves the representation of MAE through a novel integration with advanced modality-agnostic meta-learning techniques. In a nutshell, MetaMAE operates by further optimizing the amortized latent of the Transformer encoder using gradient-based meta-learning. Then we maximize the alignment between the optimized and the amortized latents via contrastive learning, to guide the Transformer encoder to improve the generalization.
Algorithm 1 MetaMAE: Meta-Learning Modality-Agnostic Masked Auto-Encoder
Require: Unlabeled pretrain dataset  $D_{\mathrm{pretrain}}$  , weight hyperparameter  $\lambda$  , Nearby-  $S$  ratio  $r$  batch size  $B$  , learning rates  $\alpha ,\beta$    
1: Initialize  $\theta ,\phi ,\psi$  using the standard initialization scheme.   
2: while not done do   
3: Sample mini-batch  $\mathcal{B} = \{\mathbf{x}_i\}_{i = 1}^B$  from  $D_{\mathrm{pretrain}}$    
4: for  $i = 1$  to  $B$  do ▷ Note: we use the batch computation.   
5: Sample Support set  $S_{\mathbf{x}_i}$  and Query set  $\mathcal{Q}_{\mathbf{x}_i}$  from  $\mathbf{x}_i$    
6:  $\mathcal{Z}_{\mathbf{x}_i} = f_\theta (\mathcal{S}_{\mathbf{x}_i})$  ▷ Amortization through Transformer encoder.   
7: Sample  $\mathcal{N}(S_{\mathbf{x}_i};r)$  where  $|\mathcal{N}(S_{\mathbf{x}_i};r)| = r\times |\mathcal{Q}_{\mathbf{x}_i}|$  ▷ Sample the Nearby-S tokens.   
8:  $\mathcal{Z}_{\mathbf{x}_i}^* \gets \mathcal{Z}_{\mathbf{x}_i} - \alpha \nabla_{\mathcal{Z}_{\mathbf{x}_i}}\mathcal{L}_{\mathrm{MAE}}(\theta ,\phi ;\tilde{\mathcal{S}}_{\mathbf{x}_i})$  where  $\tilde{\mathcal{S}}_{\mathbf{x}_i} = \mathcal{S}_{\mathbf{x}_i}\cup \mathcal{N}(\mathcal{S}_{\mathbf{x}_i};r)$  ▷ Adapt the amortized latent.   
9: Compute MAE reconstruction loss  $\mathcal{L}_{\mathrm{grad}}^i$  with  $\mathcal{Z}_{\mathbf{x}^i}^*$  ▷ Eq. (2)   
10: Compute task contrastive loss  $\mathcal{L}_{\mathrm{task - con}}^i$  with  $\{\mathcal{Z}_{\mathbf{x}^i}\}_{i = 1}^B$  and  $\{\mathcal{Z}_{\mathbf{x}^i}\}_{i = 1}^B$  ▷ Eq. (4)   
11: Compute MetaMAE loss  $\mathcal{L}_{\mathrm{MetaMAE}}^i$  with  $\mathcal{L}_{\mathrm{grad}}^i$ $\mathcal{L}_{\mathrm{task - con}}^i$  and  $\lambda$  ▷ Eq. (5)   
12: end for   
13:  $\theta ,\phi ,\psi \gets \theta ,\phi ,\psi -\frac{\beta}{B}\sum_{i = 1}^{B}\mathcal{L}_{\mathrm{MetaMAE}}^i$  ▷ Update the entire networks.   
14: end while
Latent adaptation via gradient-based meta-learning. To further improve the generalization of MAE, we suggest utilizing the gradient-based meta-learning (i.e., model-agnostic meta-learning; MAML [22]) on the amortized latent space [74]. Specifically, we adapt the amortized latent of the support set to better reconstruct the support and the nearby tokens (of support tokens) by using the gradients of the decoder. Then, we utilize the optimized latent to condition the decoder to predict the query tokens. Here, our key idea is the use of nearby tokens when optimizing the latent, which turns out to be crucial for improved performance. Intuitively, optimizing such tokens induces an error correction on the latent, which eases the mask reconstruction (or prediction) task compared to the direct reconstruction, and thereby improves the task adaptation process [103].
Concretely, for a given support set  $S_{\mathbf{x}}$ , we select the nearby tokens of support tokens from the query set  $\mathcal{Q}_{\mathbf{x}}$ , namely  $\mathcal{N}(S_{\mathbf{x}};r) \subset \mathcal{Q}_{\mathbf{x}}$ , such that the cardinality is  $|\mathcal{N}(S_{\mathbf{x}};r)| = r \times |\mathcal{Q}_{\mathbf{x}}|$  with a ratio of  $r > 0$ . Then, we optimize the amortized latent  $\mathcal{Z}_{\mathbf{x}}$  to better reconstruct the support and the nearby tokens  $\tilde{\mathcal{S}}_{\mathbf{x}} := S_{\mathbf{x}} \cup \mathcal{N}(S_{\mathbf{x}};r)$  using the decoder gradient, then condition the meta-learner, i.e., the Transformer decoder  $g_{\phi}$ , to predict the query set  $\mathcal{Q}_{\mathbf{x}}$  as follows:
$$
\mathcal {L} _ {\text {g r a d}} (\mathbf {x}, \theta , \phi) := \sum_ {(q, \bar {\mathbf {x}} ^ {(q)}) \in \mathcal {Q} _ {\mathbf {x}}} d \left(\bar {\mathbf {x}} ^ {(q)}, g _ {\phi} ^ {(q)} \left(\mathcal {Z} _ {\mathbf {x}} ^ {*}\right)\right) \text {w h e r e} \mathcal {Z} _ {\mathbf {x}} ^ {*} = \mathcal {Z} _ {\mathbf {x}} - \alpha \nabla_ {\mathcal {Z} _ {\mathbf {x}}} \mathcal {L} _ {\mathrm {M A E}} (\theta , \phi ; \tilde {\mathcal {S}} _ {\mathbf {x}}) \tag {2}
$$
where  $\alpha > 0$  is the step size for the adaptation. One can easily extend the latent optimization to obtain  $\mathcal{Z}_{\mathbf{x}}^{*}$  with more than one gradient step where we found a single step adaptation is already quite effective yet showing computation efficiency compared to multiple iterations. Furthermore, we found that it is important to use the second-order gradients for the adaptation, i.e., backpropagation on the decoder adaptation gradient when optimizing the loss function, which enables the Transformer encoder to better amortize for the reconstruction task. Note that this gradient calculation on the decoder does not increase the computation too much, as using a smaller decoder size (compared to the encoder) is the key to the success of MAE [33].
Task contrastive learning. To further exploit the benefit of the gradient-based meta-learning, we suggest nudging the amortized latent to be as close as possible to the further optimized latent in Eq. (2). By doing so, the Transformer encoder is guided to better encode the reconstruction task knowledge as the optimized latent is further adapted with support and the nearby tokens. To effectively implement this concept, we utilize the idea of task contrastive learning [28, 60, 104]. Specifically, as both the optimized and amortized latents originate from the same task, we maximize the latent similarity within the same task while minimizing the similarity with other task latents.
Formally, let  $\mathcal{Z}_{\mathbf{x}}$  and  $\mathcal{Z}_{\mathbf{x}}^{*}$  be the amortized latent and optimized latent of the given input  $\mathbf{x}$  from a minibatch  $\mathbf{x} \in \mathcal{B}$ , respectively. We then use a non-linear projection network  $h_{\psi}$  and the average set pooling of latent tokens to obtain task-specific representation  $\mathbf{z}_{\mathbf{x}} = h_{\psi}(\mathrm{pool}(\mathcal{Z}_{\mathbf{x}}))$  and  $\mathbf{z}_{\mathbf{x}}^{*} = h_{\psi}(\mathrm{pool}(\mathcal{Z}_{\mathbf{x}}^{*}))$  for the task contrastive learning. For a given set of task-specific representations  $\mathcal{T} = \bigcup_{\mathbf{x} \in \mathcal{B}} \{\mathbf{z}_{\mathbf{x}}, \mathbf{z}_{\mathbf{x}}^{*}\}$ , the task contrastive objective is defined as follows:
$$
\mathcal {L} _ {\text {t a s k - c o n}} (\mathbf {x}, \theta , \psi) := \frac {1}{2} \left[ l _ {\text {c o n}} \left(\mathbf {z} _ {\mathbf {x}}; \mathbf {z} _ {\mathbf {x}} ^ {*}, \mathcal {T} \backslash \left\{\mathbf {z} _ {\mathbf {x}} ^ {*} \right\}\right) + l _ {\text {c o n}} \left(\mathbf {z} _ {\mathbf {x}} ^ {*}; \mathbf {z} _ {\mathbf {x}}, \mathcal {T} \backslash \left\{\mathbf {z} _ {\mathbf {x}} \right\}\right) \right] \tag {3}
$$
$$
\text {w h e r e} l _ {\mathrm {c o n}} \left(\mathbf {z}; \mathbf {z} ^ {+}, \left\{\mathbf {z} ^ {-} \right\}\right) := - \log \frac {\exp \left(\sin \left(\mathbf {z} , \mathbf {z} ^ {+}\right) / \tau\right)}{\exp \left(\sin \left(\mathbf {z} , \mathbf {z} ^ {+}\right) / \tau\right) + \sum_ {\mathbf {z} ^ {-}} \exp \left(\sin \left(\mathbf {z} , \mathbf {z} ^ {-}\right) / \tau\right)} \tag {4}
$$
where  $\mathrm{sim}(\mathbf{z},\mathbf{z}^{\prime})\coloneqq \mathbf{z}\cdot \mathbf{z}^{\prime} / \| \mathbf{z}\| \| \mathbf{z}^{\prime}\|$  be the cosine similarity and  $\tau >0$  is the temperature hyperparameter. From the perspective of contrastive representation learning, our task contrastive framework can be viewed as augmenting the positive pair. However, instead of using domain-specific inductive biases, we leverage gradient adaptation, thereby showing the possibilities of extending prior contrastive learning methods to modality-agnostic SSL frameworks.
Overall meta-learning objective. In the end, we derive a final training objective,  $\mathcal{L}_{\mathrm{MetaMAE}}$ : a meta-learning objective combining the latent adaptation Eq. (2) and the task contrastive learning Eq. (4). For a given hyper-parameter  $\lambda > 0$ , the meta-objective of MetaMAE becomes:
$$
\mathcal {L} _ {\text {M e t a M A E}} (\mathbf {x}, \theta , \phi , \psi) := \mathcal {L} _ {\text {g r a d}} (\mathbf {x}, \theta , \phi) + \lambda \mathcal {L} _ {\text {t a s k - c o n}} (\mathbf {x}, \theta , \psi) \tag {5}
$$
In this section, we demonstrate the effectiveness of the proposed framework by measuring the linear-evaluation performance under various datasets across modalities. We first describe our experimental setup (Section 3.1), and then we present the main experimental results (Section 3.2). We provide ablation studies regarding MetaMAE (Section 3.3).
We here briefly describe overall experimental setups. We provide further details of pretraining, evaluation, and hyperparameters in Appendix A.
Datasets. We select 8 sub-benchmarks from the DABS 2.0 benchmark [85], with categorizing the modalities for each sub-benchmark. We pretrain and transfer MetaMAE on the selected datasets:
- Time-series modality consists of datasets where the data is organized sequentially over time. In this paper, we use the PAMAP [71] dataset, which contains sensor signals from physical activity.
- Tabular modality refers to datasets where the data is structured in a table format, with rows (for instances) and columns (for attributes). We use the HIGGS [69] dataset from particle physics.
- Multi-spectral (MS) Image modality contains multi-channel 2D image datasets. We use the EuroSAT [34, 35] dataset, which consists of 13-channel satellite images.
- Token modality features datasets consisting of sequences of discrete units, similar to natural languages. We pretrain MetaMAE on both (a) the Genomics [72] dataset, subsequently transferring the learned model to the Genomics and Genomics-OOD datasets; and (b) the Pfam [20] dataset of proteins, followed by transfer learning to several tasks from the TAPE benchmarks [20], including Pfam, SCOP [23], Secondary Structure [43, 8], Stability [73], and Fluorescence [76].
- Speech modality includes 2D spectrograms of audio datasets. We pretrain MetaMAE on LibriSpeech [66], a large English audiobook corpus, and then transfer the model to datasets including LibriSpeech, Audio MNIST [6], Fluent Speech [57], Google Speech [96], and VoxCeleb1 [62].
- RGB Image modality comprises 3-channel 2D image datasets. We pretrain MetaMAE on (a) the ImageNet32 [17] dataset, which is scaled to  $32 \times 32$ , and transfer the pretrained model to datasets including CIFAR-10 [45], CUB [93], VGG Flowers [63], DTD [14], Traffic Sign [81], and Aircraft [58]; and (b) the WaferMap [98] dataset.
- Vision-Language modality comprises a combination of 3-channel 2D image and sequences of English text descriptions. We pretrain MetaMAE on MSCOCO [54], and then transfer the model to mismatched-caption detection [54] and the Visual Question Answering (i.e., VQA) tasks [1].
Table 1: In-domain linear evaluation performance across multiple modalities. We report F1-score (\%) for WaferMap and the classification accuracy (\%) for the rest. MS Image indicates the Multi-spectral image modality. *, and † denote the results from the DABS 1.0, and DABS 2.0 paper, respectively, where - of Capri results indicate that the pretraining loss divergence as described in [85].
Modality Dataset	Time-series PAMAP2	Tabular HIGGS	MS Image EuroSAT	Token	Speech Libri	RGB Image WaferMap
Genom	Pfam
Random initialization
Baseline	69.8†	54.8†	62.3†	37.2†	30.1	17.1*	77.7†
Self-supervised learning Framework
e-Mix	80.1	65.7	87.4	40.5	31.3	60.2	92.6
ShED	85.2	68.0†	61.5†	33.6	54.7	34.8*	92.4†
Capri	-	-	67.4†	23.5†	27.4	25.4	92.5†
MAE	85.3†	70.0†	86.3†	53.6	44.7	46.0	93.9†
MetaMAE	89.3	71.5	88.5	69.4	62.3	79.8	95.5
Note that the transferred datasets can be in-domain (i.e., same dataset) or cross-domain (i.e., different dataset). The details of the benchmarks are described in Appendix C.
Baselines. For the main experiments, we compare MetaMAE's performance with existing modality-agnostic self-supervised learning methods suggested by DABS 1.0 [83], and 2.0 [85]: