Title: Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder
Venue: NeurIPS 2023
Accepted: yes
Keywords: Self-Supervised Learning

Modality-Agnostic Self-Supervised Learning

Meta-Learning

Masked Auto-Encoder
Abstract: Despite its practical importance across a wide range of modalities, recent advances in self-supervised learning (SSL) have been primarily focused on a few well-curated domains, e.g., vision and language, often relying on their domain-specific knowledge. For example, Masked Auto-Encoder (MAE) has become one of the popular architectures in these domains, but less has explored its potential in other modalities. In this paper, we develop MAE as a unified, modality-agnostic SSL framework. In turn, we argue meta-learning as a key to interpreting MAE as a modality-agnostic learner, and propose enhancements to MAE from the motivation to jointly improve its SSL across diverse modalities, coined MetaMAE as a result. Our key idea is to view the mask reconstruction of MAE as a meta-learning task: masked tokens are predicted by adapting the Transformer meta-learner through the amortization of unmasked tokens. Based on this novel interpretation, we propose to integrate two advanced meta-learning techniques. First, we adapt the amortized latent of the Transformer encoder using gradient-based meta-learning to enhance the reconstruction. Then, we maximize the alignment between amortized and adapted latents through task contrastive learning which guides the Transformer encoder to better encode the task-specific knowledge. Our experiment demonstrates the superiority of MetaMAE in the modality-agnostic SSL benchmark (called DABS), significantly outperforming prior baselines.

Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder

Abstract

Despite its practical importance across a wide range of modalities, recent advances in self-supervised learning (SSL) have been primarily focused on a few wellcurated domains, e.g., vision and language, often relying on their domain-specific knowledge. For example, Masked Auto-Encoder (MAE) has become one of the popular architectures in these domains, but less has explored its potential in other modalities. In this paper, we develop MAE as a unified, modality-agnostic SSL framework. In turn, we argue meta-learning as a key to interpreting MAE as a modality-agnostic learner, and propose enhancements to MAE from the motivation to jointly improve its SSL across diverse modalities, coined MetaMAE as a result. Our key idea is to view the mask reconstruction of MAE as a meta-learning task: masked tokens are predicted by adapting the Transformer meta-learner through the amortization of unmasked tokens. Based on this novel interpretation, we propose to integrate two advanced meta-learning techniques. First, we adapt the amortized latent of the Transformer encoder using gradient-based meta-learning to enhance the reconstruction. Then, we maximize the alignment between amortized and adapted latents through task contrastive learning which guides the Transformer encoder to better encode the task-specific knowledge. Our experiment demonstrates the superiority of MetaMAE in the modality-agnostic SSL benchmark (called DABS), significantly outperforming prior baselines. Code is available at https://github.com/alinlab/MetaMAE.

1 Introduction

Self-supervised learning (SSL), i.e., learning without human supervision, recently has demonstrated substantial success across fields including, computer vision [32, 11, 29, 47, 33, 5, 102], natural language processing (NLP) [18, 49, 55, 70], and speech recognition [2, 38, 39]. The efficacy of SSL is derived by extracting transferable knowledge from unlabeled datasets, a feature that manifests significant utility for various downstream tasks such as classification and segmentation. As a result, SSL has become an indispensable technique in real-world applications (for instance, industrial contexts like medical imaging [26]), not only improving the performance on new datasets but also reducing a significant amount of computations and costs, e.g., expert annotation poses significant costs [68, 16]. However, despite the importance of SSL in such fields, recent advancements have been predominantly focused on specific domains (e.g., images and NLP) where the majority of existing SSL frameworks on such domains require modality-specific knowledge, thereby constraining the applicability and scalability of previous works across new modalities.

To tackle this issue, we draw attention to the recent success of the Masked Auto-Encoder (MAE) framework [33], which eliminates the need for modality-specific inductive biases. Initially presented as a generative model [91, 67], the MAE models the network to reconstruct the original input

Encoder

Decoder

⁞

Latent adaptation

Reconstruction

⁞

Projector

Forward

AvgPool

AvgPool

signal based on the randomly masked part of the signal. Recently, the integration of MAE with the Transformer architecture [33] has resulted in a powerful SSL framework for various domains, such as vision [33, 5, 10], NLP [18, 70], and tabular [59] datasets. For instance, it has been demonstrated that BERT [18], utilizing a Transformer encoder with a linear decoder, can effectively transfer to diverse tasks for the NLP domain. On the other hand, it has been suggested that utilizing a decoder of a deep Transformer architecture is essential for applying MAE within the vision domain [106], achieving remarkable performance [33]. Building on these insights, we found that MAE is quite a promising direction for modality-agnostic SSL: our experiment demonstrates that using a deep Transformer decoder for MAE significantly outperforms previous modality-agnostic SSL frameworks (see Table 5). In this paper, we suggest to further exploit the benefits of MAE to build a unified SSL framework.

Contribution. We propose Meta-learned Masked Auto-Encoder (MetaMAE), a novel modalityagnostic SSL framework that leverages the power of meta-learning; see the overview in Figure 1. Our key idea is to interpret MAE as a meta-learning framework, thereby improving the generalization through the use of advanced meta-learning schemes. To be specific, we interpret the data reconstruction itself as a task, where the Transformer meta-learner is adapted through amortization of the support set (i.e., unmasked tokens) to predict the query set (i.e., masked tokens). Based on this interpretation, we propose a novel integration of two advanced meta-learning techniques to enhance MAE; namely the use of gradient-based meta-learning [22] and task contrastive learning [28, 60].

Latent Adaptation via Gradient-based Meta-learning : We suggest adapting the amortized latent to better reconstruct the given support set (and the nearby tokens) through gradient-based metalearning on the decoder [74]. Then, the optimized latent is used to condition the decoder for the query prediction. This approach generally eases the task compared to the direct reconstruction, thereby streamlining and improving the task adaptation process.

Task Contrastive Learning : To further leverage this optimized latent, we suggest utilizing task contrastive learning [28, 60]. Specifically, since both the optimized and predicted latents originate from the same task, we aim to maximize their similarity while minimizing their similarity with other tasks. This prompts the Transformer encoder to produce predictions closely aligned with the optimized latent, effectively guiding the Transformer to better encode the task knowledge.

We verify the efficacy of MetaMAE through extensive evaluations on multiple data modalities from modality-agnostic SSL benchmarks (i.e., DABS 1.0 [83] and 2.0 [85]), including time-series, tabular, discrete token, multi-spectral images, speech, and multi-modal datasets. Overall, our experimental results demonstrate strong results, consistently and significantly outperforming previous modalityagnostic SSL methods in linear evaluation. For instance, measured with classification accuracy (%), MetaMAE improves the prior state-of-the-art results by 85.3 → 89.3 on PAMAP2 [71], 53.6 → 69.4 on Genomics [72], and 60.2 → 79.8 on LibriSpeech [66] datasets. Moreover, we also demonstrate that MetaMAE significantly improves the linear evaluation performance on cross-domain datasets, indicating the improved transfer ability of MAE through meta-learning.

2 Meta-Learning Modality-Agnostic Masked Auto-Encoder

In this section, we present Meta-learned Masked Auto-Encoder (MetaMAE) , a novel and effective modality-agnostic self-supervised learning (SSL) framework. Our key contribution is to tackle the modality-agnostic SSL problem with in-depth utilization of MAE, which was quite under-explored in the field. Based on our novel interpretation that views MAE as a meta-learning framework (in Section 2.1), we improve the transfer ability of MAE by suggesting enhanced modality-agnostic meta-learning techniques including latent optimization-based meta-learning and task contrastive learning (in Section 2.2). Our framework is visually depicted in Figure 1, and the pseudo-code is provided in Algorithm 1.

Problem setup. We first describe the problem setup of our interest, modality-agnostic SSL. This problem aims to learn a transferable representation from the unlabeled dataset without utilizing the modality-specific inductive biases. For a given unlabeled pretrain dataset D pretrain = { x i } N i =1 , where x ∈ R d represents an input sampled from a certain data-generating distribution in an i.i.d manner, our objective is to train an encoder f θ that can linearly separate the given labeled transfer dataset drawn from a similar or the same data-generating distribution.

2.1 Rethinking Masked Auto-Encoder as a Meta-Learning Framework

Meta-learning [86] aims to extract and utilize the knowledge from the distribution of tasks to better solve a relevant task. This problem is typically approached by training a meta-learner that can transfer its knowledge to a task-specific model through adaptation, where the performance of the meta-learner is evaluated on the basis of how well each adapted model performs on the corresponding task. To learn such a meta-learner, a standard way is to use a set of support set samples to adapt the task-specific model from the meta-learner and use another disjoint set of samples, called query set samples to evaluate the adaptation performance [92, 82].

Mask prediction as a modality-agnostic task. MAE is an SSL technique that trains an autoencoder to reconstruct the original input signal with a randomly masked part of the signal. To implement such a technique, recent works utilize the Transformer architecture for the autoencoder design which is necessary for successful training. To use Transformer for MAE, the input data is broken down into non-overlapping units coined tokens (e.g., patches for images, and words for languages) where such tokens are divided into two disjoint sets (unmasked and masked) for MAE modeling, i.e., the Transformer autoencoder predicts the masked token using the unmasked tokens.

Our key insight is to interpret the signal reconstruction of MAE as a meta-learning task, where two disjoint unmasked and masked token sets are viewed as support and query sets to adapt and evaluate the Transformer meta-learner. To be specific, the Transformer encoder extracts the task knowledge through amortization of the support set, where this amortized latent adapts the Transformer decoder to predict the query set of the task. Formally, for a given data sample x , we first divide the signal into two disjoint sets, namely the support set S x and the query set Q x , by utilizing the tokenize operation tokenize ( x ) := { ( m, ¯ x ( m ) ) } M m =1 = S x ∪Q x . Then, for a given Transformer encoder f θ and decoder g ϕ , MAE minimizes the discrepancy between the predicted token and the corresponding masked token (i.e., the query sample) as:

where d ( · , · ) is a discrepancy function: ℓ 2 norm for continuous (e.g., time-series, speech) and crossentropy for discrete (e.g., token) datasets, respectively. Based on this interpretation, we improve the transfer ability of MAE (for modality-agnostic SSL) through a novel integration of two effective modality-agnostic meta-learning techniques to MAE.

2.2 MetaMAE: Improving Masked Auto-Encoder through Meta-Learning

We now describe our method, MetaMAE, which further improves the representation of MAE through a novel integration with advanced modality-agnostic meta-learning techniques. In a nutshell, MetaMAE operates by further optimizing the amortized latent of the Transformer encoder using gradientbased meta-learning. Then we maximize the alignment between the optimized and the amortized latents via contrastive learning, to guide the Transformer encoder to improve the generalization.

Algorithm 1 MetaMAE: Meta-Learning Modality-Agnostic Masked Auto-Encoder

Require: Unlabeled pretrain dataset D pretrain , weight hyperparameter λ , NearbyS ratio r , batch size B , learning rates α , β .

Latent adaptation via gradient-based meta-learning. To further improve the generalization of MAE, we suggest utilizing the gradient-based meta-learning (i.e., model-agnostic meta-learning; MAML[22]) on the amortized latent space [74]. Specifically, we adapt the amortized latent of the support set to better reconstruct the support and the nearby tokens (of support tokens) by using the gradients of the decoder. Then, we utilize the optimized latent to condition the decoder to predict the query tokens. Here, our key idea is the use of nearby tokens when optimizing the latent, which turns out to be crucial for improved performance. Intuitively, optimizing such tokens induce an error correction on the latent, which eases the mask reconstruction (or prediction) task compared to the direct reconstruction, and thereby improves the task adaptation process [103].

Concretely, for a given support set S x , we select the nearby tokens of support tokens from the query set Q x , namely N ( S x ; r ) ⊂ Q x , such that the cardinality is |N ( S x ; r ) | = r ×|Q x | with a ratio of r > 0 . Then, we optimize the amortized latent Z x to better reconstruct the support and the nearby tokens ˜ S x := S x ∪ N ( S x ; r ) using the decoder gradient, then condition the meta-learner, i.e., the Transformer decoder g ϕ , to predict the query set Q x as follows:

where α > 0 is the step size for the adaptation. One can easily extend the latent optimization to obtain Z ∗ x with more than one gradient step where we found a single step adaptation is already quite effective yet showing computation efficiency compared to multiple iterations. Furthermore, we found that it is important to use the second-order gradients for the adaptation, i.e., backpropagation on the decoder adaptation gradient when optimizing the loss function, which enables the Transformer encoder to better amortize for the reconstruction task. Note that this gradient calculation on the decoder does not increase the computation too much, as using a smaller decoder size (compared to the encoder) is the key to the success of MAE [33].

Task contrastive learning. To further exploit the benefit of the gradient-based meta-learning, we suggest nudging the amortized latent to be as close as possible to the further optimized latent in Eq. (2). By doing so, the Transformer encoder is guided to better encode the reconstruction task knowledge as the optimized latent is further adapted with support and the nearby tokens. To effectively implement this concept, we utilize the idea of task contrastive learning [28, 60, 104]. Specifically, as both the optimized and amortized latents originate from the same task, we maximize the latent similarity within the same task while minimizing the similarity with other task latents.

Formally, let Z x and Z ∗ x be the amortized latent and optimized latent of the given input x from a minibatch x ∈ B , respectively. We then use a non-linear projection network h ψ and the average set pooling of latent tokens to obtain task-specific representation z x = h ψ ( pool ( Z x ) ) and z ∗ x = h ψ ( pool ( Z ∗ x ) ) for the task contrastive learning. For a given set of task-specific representations T = ⋃ x ∈B { z x , z ∗ x } , the task contrastive objective is defined as follows:

where sim( z , z ′ ) := z · z ′ / ∥ z ∥∥ z ′ ∥ be the cosine similarity and τ > 0 is the temperature hyperparameter. From the perspective of contrastive representation learning, our task contrastive framework can be viewed as augmenting the positive pair. However, instead of using domain-specific inductive biases, we leverage gradient adaptation, thereby showing the possibilities of extending prior contrastive learning methods to modality-agnostic SSL frameworks.

Overall meta-learning objective. In the end, we derive a final training objective, L MetaMAE : a metalearning objective combining the latent adaptation Eq. (2) and the task contrastive learning Eq. (4). For a given hyper-parameter λ > 0 , the meta-objective of MetaMAE becomes:

3 Experiments

In this section, we demonstrate the effectiveness of the proposed framework by measuring the linearevaluation performance under various datasets across modalities. We first describe our experimental setup (Section 3.1), and then we present the main experimental results (Section 3.2). We provide ablation studies regarding MetaMAE (Section 3.3).

3.1 Experimental Setup

We here briefly describe overall experimental setups. We provide further details of pretraining, evaluation, and hyperparameters in Appendix A.

Datasets. We select 8 sub-benchmarks from the DABS 2.0 benchmark [85], with categorizing the modalities for each sub-benchmark. We pretrain and transfer MetaMAE on the selected datasets:

Time-series modality consists of datasets where the data is organized sequentially over time. In this paper, we use the PAMAP [71] dataset, which contains sensor signals from physical activity.

Tabular modality refers to datasets where the data is structured in a table format, with rows (for instances) and columns (for attributes). We use the HIGGS [69] dataset from particle physics.

Multi-spectral (MS) Image modality contains multi-channel 2D image datasets. We use the EuroSAT [34, 35] dataset, which consists of 13-channel satellite images.

Token modality features datasets consisting of sequences of discrete units, similar to natural languages. We pretrain MetaMAE on both (a) the Genomics [72] dataset, subsequently transferring the learned model to the Genomics and Genomics-OOD datasets; and (b) the Pfam [20] dataset of proteins, followed by transfer learning to several tasks from the TAPE benchmarks [20], including Pfam, SCOP [23], Secondary Structure [43, 8], Stability [73], and Fluorescence [76].

Speech modality includes 2D spectrograms of audio datasets. We pretrain MetaMAE on LibriSpeech [66], a large English audiobook corpus, and then transfer the model to datasets including LibriSpeech, Audio MNIST [6], Fluent Speech [57], Google Speech [96], and VoxCeleb1 [62].

RGB Image modality comprises 3-channel 2D image datasets. We pretrain MetaMAE on (a) the ImageNet32 [17] dataset, which is scaled to 32 × 32 , and transfer the pretrained model to datasets including CIFAR-10 [45], CUB [93], VGG Flowers [63], DTD [14], Traffic Sign [81], and Aircraft [58]; and (b) the WaferMap [98] dataset.

Vision-Language modality comprises a combination of 3-channel 2D image and sequences of English text descriptions. We pretrain MetaMAE on MSCOCO [54], and then transfer the model to mismatched-caption detection [54] and the Visual Question Answering (i.e., VQA) tasks [1].

Note that the transferred datasets can be in-domain (i.e., same dataset) or cross-domain (i.e., different dataset). The details of the benchmarks are described in Appendix C.

Baselines. For the main experiments, we compare MetaMAE's performance with existing modalityagnostic self-supervised learning methods suggested by DABS 1.0 [83], and 2.0 [85]:

e -Mix is a generalized version of i -Mix [48], designed to consistently apply methods across both discrete and continuous domains by applying the mixup strategy in the embedding space.

ShED is a generalized version of ELECTRA [15]. ShED constructs the pretext task, which involves predicting shuffled embeddings.

Capri applys contrastive learning to the token level representation by randomly masking the token and treating different tokens as negative pairs.

MAE aims to reconstruct the input. However, here, MAE employs a linear decoder for the continuous domain and no decoder for the discrete domain.

Additionally, we regard the randomly initialized encoder, referred to as the Baseline, as one of the baseline to check the effectiveness of self-supervised pretraining.

Architectures. Following [83, 85], we use 12 layers for the transformer encoder with the hidden size 256, and 8 attention heads. For the decoder, we fix the hidden size 128, and 4 attention heads. However, we choose an appropriate number of layers for the decoder to demonstrate the effect of the decoder for MAE. We also utilize different hyperparameters for each modality as other baselines, but we find that the hyperparameters can be shared across modalities (See Appendix B).

Pretraining and transfer learning. To evaluate our method, we pretrain each dataset 100 Kiterations and 100 epochs transfer learning, overall experiments by following [85]. We pretrain entire networks, i.e., encoder f θ , decoder g θ , and projection header h θ , but we utilize only the frozen encoder f θ on transfer learning. When pretraining, the masking ratio can differ from the datasets. For the masking ratio hyper-parameter, we choose the best value among candidates suggested by the prior work [85].

3.2 Main Experiments

In-domain linear evaluation. We evaluate the pretrained representation on each in-domain downstream classification task. We report the performance of a linear classifier trained on top of the frozen features. The results in Table 1 demonstrate that our proposed method, MetaMAE, achieves state-of-the-art performance across the entire dataset. For instance, we obtain 16% accuracy gain (53.6% → 69.4%) on Genomics. Moreover, we note that MAE has achieved moderate performance compared to other self-supervised learning (SSL) methods on these benchmarks, but MetaMAE demonstrates the ability to enhance MAE and outperform other SSL approaches. For example, MetaMAE achieves the best performance on Pfam (44.7% → 62.3%) and LibriSpeech (60.2% → 79.8%) with significant improvement, here is where MAE reported in [85] (i.e., MAE with linear decoder) was not the best among baselines.

Cross-domain linear evaluation. We evaluate our method on a diverse set of cross-domain downstream tasks including both classification and regression. We employ a linear classifier, or regressor trained on the frozen features as the in-domain setup. Table 2 shows that MetaMAE outperforms all the baselines across all the benchmarks consistently, except for one specific dataset. For example, we obtain 9% accuracy gain (80.4% → 89.5%) on the linear classification performance of transfer setup from LibriSpeech to Audio MNIST. It is important to note that cross-domain downstream tasks, due to their wider range of variations for each domain, are typically more challenging to consistently excel in compared to in-domain tasks. This significant performance improvement demonstrates the applicability of MetaMAE in various cross-domain transfer learning scenarios across the modalities.

Multi-modal dataset evaluation. One important future direction for the modality-agnostic SSL research community is to bind all modalities under a singular model [99, 108]. Here, we believe MetaMAE can be quite a promising method to tackle this problem, e.g., managing multiple modalities on a single model supplemented by domain-specific embedding modules. To this end, we verify the possibility of MetaMAE for tackling unified multi-modal self-supervised learning. As shown in Table 3, MetaMAE outperforms other modality-agnostic SSL methods on the vision-language tasks where we believe this multi-modal learning ability can help when unifying the modalities for SSL.

3.3 Ablation study

We perform an ablation study on six modalities: time-series (PAMAP2), tabular (HIGGS), speech (LibriSpeech), multi-spectral image (EuroSAT), and token (Pfam and Genomics). Throughout this section, we report the in-domain linear classification accuracy (%), unless otherwise specified.

Component analysis. In Table 4, we demonstrate the necessity of each component in MetaMAE by adding each component one by one: Deeper decoder g ϕ with a Transformer architecture, latent optimization via gradient-based meta-learning, and the task contrastive loss L task -con . We first found

that incorporating a deep decoder is a critical component in our framework, enabling domain-agnostic capabilities similar to the success of MAE on the image domain [33]. Thus, we here suggest that improving MAE for the domain-agnostic is quite a promising direction to explore.

In addition, Table 4 verifies the contribution of meta-learning schemes to the performance of MetaMAE. We found that the gradient-based latent optimization rule, which includes the utilization of NearbyS , is more beneficial. We also confirm that task contrastive learning is a critical component in our framework like recent meta-learning frameworks [28, 60, 104]. Note that this task contrastive learning scheme is exclusively applicable in gradient-based approaches, emphasizing the significance of the gradient-based latent optimization method for MetaMAE.

Importance of decoder size for MAE. To verify the effect of decoder size for MAE, we evaluate the linear evaluation accuracy on datasets where the original MAE (i.e., no decoder) performed worse than other baselines. As Table 5 shows, we found that MAE can achieve the best performance compared to baselines by choosing the proper decoder size, yet there is room for enhancement as shown in Table 4. This result demonstrates the superiority of MAE for tackling modality-agnostic SSL problems, where we believe the development of MAE would be an important direction to investigate. In this respect, we believe MetaMAE will serve as an important baseline in this field.

Nearby supports. We further analyze the effect of r , i.e., the NearbyS ratio. We conduct the experiment with r ∈ { 0 , 0 . 1 , 0 . 5 , 1 . 0 } . We note that r = 0 indicates the gradient updates without any help of queries (i.e., direct reconstruction of S ), and r = 1 denotes the gradient updates with the entire queries near the S . As shown in Table 6, this approach is found to be beneficial compared to the direct reconstruction, and the small ratio is suggested to be proper r , e.g., r = 0 . 1 is the best. This is because it effectively bridges the gap between the latent representation and the latents of masked tokens, thereby enhancing the encoding of knowledge required for the reconstruction task.

Computational efficiency. MetaMAE might be perceived as compute-inefficient when incorporating MAE due to the computational demands of second-order gradients; however, our findings suggest otherwise. Although MetaMAE increases the total training time of MAE by approximately 1.4 times (with the one-step adaptation), we have observed that it is much faster to achieve the best performance of MAE: in Figure 2, we compare the accuracy under the same training wall-clock

(a) LibriSpeech

(b) PAMAP2

Libri (Speech)

PAMAP2 (Time-series)

Linear evaluation accuracy (%)

80

Linear evaluation accuracy (%)

90

MAE

MetaMAE

1.9xfaster

85

60

1.7xfaster

80

40

MAE

MetaMAE

75

0

50

100

150

200

0

10

20

30

40

50

Pretraining wall clock time (min)

Pretraining wall clock time (min)

time with MAE, e.g., 1.9 times faster on PAMAP2 dataset.

4 Related work

Self-supervised learning (SSL). SSL, i.e., learning without human supervision, recently has demonstrated substantial success across fields including, computer vision [32], natural language processing (NLP) [18], and speech recognition [2], frequently show better transferability and generalization ability compared to conventional pretraining methods, e.g., training on labeled datasets [11]. To learn such representation, SSL optimizes the loss on a pretext task that does not require any human labels. For instance, pioneer works for SSL proposed such tasks based on data reconstruction through auto-encoding [7] such as context prediction [19], and in-painting [67]. Later on, multiple SSL works found that utilizing the domain-specific inductive biases can effectively learn representations in a selfsupervised manner, including, colorization [107], solving jigsaw puzzles [64], counting the number of objects [65], and rotation prediction [27], to name a few. More recently, contrastive representation learning has garnered significant attention in SSL [100, 32, 11]. This technique maximizes the similarities of similar (i.e., positive) pairs and minimizes the similarities of dissimilar (i.e., negative) pairs, rather than focusing on training an instance classifier. To generate such positive pairs, multiple works rely on domain-specific inductive biases such as data augmentations [47], i.e., different augmented views as a positive pair. In addition, recent advances have been made with the development of various architectural components: e.g., Siamese networks [44], self-distillation [32, 9], asymmetric architectures [29, 12], and utilization of Transformer architectures [9]. Despite the success of these strategies, most existing SSL frameworks rely heavily on domain-specific inductive biases, which limits their applicability to new modalities.

Modality-agnostic SSL. Recently, several streams of work have emerged focusing on the development of more generalized SSL methods, specifically modality-agnostic SSL. For example, DACL [90] and i -Mix [48] utilize the idea of mixup [105] to propose domain-agnostic contrastive learning, and e -Mix [83] generalizes the concept of i -Mix to be embedding-level instead of input-level. Capri [85], as a variant of CPC [88], contrasts the predicted representations from randomly masked tokens. [84] develops generative models to learn data-dependent distortions for contrast. Instead of contrastive learning, ShED [83] (a generalized version of ELECTRA [15]) constructs the pretext task of replacing token detection with a masking strategy. DABS 2.0 [85] proposes a method to generalize MAE [33] to be modality-agnostic. In their approach, however, decoders are not utilized for discrete domains like BERT [18], while only a linear decoder is employed for continuous domains. In this paper, we suggest an effective modality-agnostic latent optimization for learning representations by interpreting masked prediction for MAE [33] in a novel manner.

Masked Auto-Encoder (MAE). MAE [33], i.e., predicting the masked parts with a given unmasked parts, has been extended to multiple applications [109, 37] across various domains [18, 39]. Among them, the recent combination of MAE with Transformer architecture [18, 33, 5] has shown promise in tackling SSL scenarios. For instance, BERT [18] utilized MAE for natural language processing (NLP) tasks, incorporating a linear layer into its architecture. Furthermore, multiple variants of MAE show impressive performance in various domains, by suggesting modality-agnostic SSL [3, 4], architecture-agnostic SSL [97, 51], multi-modal pretraining [95], and generative pretraining frameworks [21, 52]. In this paper, we focus on improving the most basic form of mask-modeling (i.e., MAE) for constructing a modality-agnostic SSL framework which remains under-explored, despite its potential significance, through the lens of meta-learning. It is worth noting that our interpretation of viewing MAE as a meta-learning framework can be applied to any other masked-modeling-based SSL frameworks where we believe combining our meta-learning regularization to such SSL methods would be an interesting direction to explore.

Meta-learning. Meta-learning [86], i.e., learning to learn by extracting common knowledge over a task distribution, has emerged as a popular paradigm for enabling systems to adapt to new tasks in a sample-efficient way. Under various applications across domains (e.g., computer vision [78], natural language processing [30], and robotics [101]), there have been significant efforts to design a variety of meta-learning schemes, including gradient-based [22, 53] and amortization-based approaches [75, 61] such as metric-based [92, 79], and neural processes [25, 24, 41, 87]. Typically, recent works have combined gradient-based meta-learning (or iterative functional update) with amortization-based schemes to enhance adaptation performance [74, 103]. Furthermore, there have been varieties of amortization-based schemes (such as neural processes) that utilize the recent success of contrastive learning into meta-learning, i.e., task contrastive learning [28, 60]. In this paper, we interpret MAE as an amortization-based meta-learning, which is further enhanced via the benefit of model-agnosticism of gradient-based meta-learning and task contrastive learning.

5 Discussion and conclusion

In this paper, we tackle modality-agnostic self-supervised learning (SSL), an important problem of SSL that consists of multiple real-world applications. To this end, we explore the possibilities of the Masked Auto-Encoder (MAE) in tackling modality-agnostic SSL which is quite under-explored, despite its potential. We propose MetaMAE, a novel and effective SSL framework that enhances MAE with meta-learning. Our key idea is to interpret mask reconstruction task of MAE as a meta-learning task, which allows us to treat MAE as a meta-learning framework. Based on this novel interpretation, we suggest a unique integration with advanced modality-agnostic meta-learning methods to improve the generalization of MAE. Our experiments demonstrate that MetaMAE significantly improves the performance of modality-agnostic SSL approaches across a diverse range of modalities.

Limitations and future work. While MetaMAE becomes a state-of-the-art approach for modalityagnostic SSL problems, it still inherits a general limitation of the MAE, namely the modality-specific masking ratio, i.e., the masking ratio may differ across modalities. This is due to our shared design elements with MAE, which include masking, encoding, and decoding. Recent works propose design choices for the masking scheme [50, 94], including automation, where incorporating these ideas into MetaMAE would be an intriguing future research direction, potentially enhancing our approach to be an even more effective modality-agnostic SSL framework.

Potential negative impacts. SSL often requires a large computation and a large network capacity, therefore raising environmental concerns, e.g., carbon generation [77]. As MetaMAE is built upon the SSL method (i.e., MAE), practitioners may need to consider some computation for successful training. To address this issue, efficient training methods [80, 40], distilling knowledge to a smaller network [36], or network sparsity schemes [31, 46] would be required to ameliorate such problems.

References

S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In IEEE International Conference on Computer Vision , 2015.

A. Baevski, Y. Zhou, A. Mohamed, and M. Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems , 2020.

A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning , 2022.

A. Baevski, A. Babu, W.-N. Hsu, and M. Auli. Efficient self-supervised learning with contextualized target representations for vision, speech and language. In International Conference on Machine Learning , 2023.

H. Bao, L. D. 0004, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations , 2022.

S. Becker, M. Ackermann, S. Lapuschkin, K.-R. Müller, and W. Samek. Interpreting and explaining deep neural networks for classification of audio signals. CoRR , abs/1807.03418, 2018.

Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems , 2006.

H. M. Berman, J. Westbrook, Z. Feng, G. Gilliland, T. N. Bhat, H. Weissig, I. N. Shindyalov, and P. E. Bourne. The protein data bank. Nucleic acids research , 28(1):235-242, 2000.

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In IEEE International Conference on Computer Vision , 2021.

M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning , pages 1691-1703. PMLR, 2020.

T. Chen, S. Kornblith, M. N. 0002, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning , 2020.

X. Chen and K. He. Exploring simple siamese representation learning. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

S. K. Choe, W. Neiswanger, P. Xie, and E. P. Xing. Betty: An automatic differentiation library for multilevel optimization. In International Conference on Learning Representations , 2023.

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR , pages 3606-3613, 2014.

K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations , 2020.

J. M. Clements, D. Xu, N. Yousefi, and D. Efimov. Sequential deep learning for credit risk monitoring with tabular financial data. arXiv preprint arXiv:2012.15330 , 2020.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , 2009.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT , 2019.

C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. In IEEE International Conference on Computer Vision , 2015.

S. El-Gebali, J. Mistry, A. Bateman, S. R. Eddy, A. Luciani, S. C. Potter, M. Qureshi, L. Richardson, G. A. Salazar, A. Smart, E. L. L. Sonnhammer, L. Hirsh, L. Paladin, D. Piovesan, S. C. E. Tosatto, and R. D. Finn. The pfam protein families database in 2019. Nucleic Acids Research , 2019.

P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning , 2017.

N. K. Fox, S. E. Brenner, and J.-M. Chandonia. Scope: Structural classification of proteins extended, integrating scop and astral data and classification of new structures. Nucleic Acids Research , 2014.

M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. Rezende, and S. A. Eslami. Conditional neural processes. In International Conference on Machine Learning , 2018.

M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. Eslami, and Y. W. Teh. Neural processes. arXiv preprint arXiv:1807.01622 , 2018.

F. C. Ghesu, B. Georgescu, A. Mansoor, Y. Yoo, D. Neumann, P. Patel, R. Vishwanath, J. M. Balter, Y. Cao, S. Grbic, et al. Self-supervised learning from 100 million medical images. arXiv preprint arXiv:2201.01283 , 2022.

S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations , 2018.

M. W. Gondal, S. Joshi, N. Rahaman, S. Bauer, M. Wuthrich, and B. Schölkopf. Function contrastive learning of transferable meta-representations. In International Conference on Machine Learning , 2021.

J.-B. grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Ávila Pires, Z. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent - a new approach to self-supervised learning. In Advances in Neural Information Processing Systems , 2020.

J. Gu, Y. Wang, Y. Chen, K. Cho, and V. O. Li. Meta-learning for low-resource neural machine translation. In EMNLP , 2018.

S. Han, J. Pool, J. Tran, and W. J. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems , 2015.

K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. B. Girshick. Masked autoencoders are scalable vision learners. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

P. Helber, B. Bischke, A. Dengel, and D. Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium , 2018.

P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2019.

G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. In Advances in Neural Information Processing Systems , 2014.

Z. Hou, X. Liu, Y. Cen, Y. Dong, H. Yang, C. Wang, and J. Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 594-604, 2022.

W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE Transactions on Audio, Speech &; Language Processing , 2021.

P.-Y. Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer. Masked autoencoders that listen. In Advances in Neural Information Processing Systems , 2022.

S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning , 2015.

H. Kim, A. Mnih, J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Teh. Attentive neural processes. In International Conference on Learning Representations , 2019.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

M. S. Klausen, M. C. Jespersen, H. Nielsen, K. K. Jensen, V. I. Jurtz, C. K. Soenderby, M. O. A. Sommer, O. Winther, M. Nielsen, B. Petersen, et al. Netsurfp-2.0: Improved prediction of protein structural features by integrated deep learning. Proteins: Structure, Function, and Bioinformatics , 87(6):520-527, 2019.

G. Koch, R. Zemel, and R. Salakhutdinov. Siamese neural networks for one-shot image recognition. In International Conference on Machine Learning , 2015.

A. Krizhevsky et al. Learning multiple layers of features from tiny images, 2009.

J. Lee, J. Tack, N. Lee, and J. Shin. Meta-learning sparse implicit neural representations. In Advances in Neural Information Processing Systems , 2021.

K. Lee and J. Shin. Rényicl: Contrastive representation learning with skew rényi divergence. In Advances in Neural Information Processing Systems , 2022.

K. Lee, Y. Zhu, K. Sohn, C.-L. Li, J. Shin, and H. Lee. i-mix: A domain-agnostic strategy for contrastive representation learning. In International Conference on Learning Representations , 2021.

M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Annual Conference of the Association for Computational Linguistics , 2019.

G. Li, H. Zheng, D. Liu, C. Wang, B. Su, and C. Zheng. SemMAE: Semantic-guided masking for learning masked autoencoders. In Advances in Neural Information Processing Systems , 2022.

S. Li, D. Wu, F. Wu, Z. Zang, and S. Z. Li. Architecture-agnostic masked image modeling from vit back to cnn. In International Conference on Machine Learning , 2023.

T. Li, H. Chang, S. K. Mishra, H. Zhang, D. Katabi, and D. Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Z. Li, F. Zhou, F. Chen, and H. Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835 , 2017.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision , 2014.

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio. Speech model pre-training for end-to-end spoken language understanding. In G. Kubin and Z. Kacic, editors, Proc. of Interspeech , pages 814-818, 2019.

S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report , 2013.

K. Majmundar, S. Goyal, P. Netrapalli, and P. Jain. Met: Masked encoding for tabular data. arXiv preprint arXiv:2206.08564 , 2022.

E. Mathieu, A. Foster, and Y. W. Teh. On contrastive representations of stochastic processes. In Advances in Neural Information Processing Systems , 2021.

N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel. A simple neural attentive meta-learner. In ICLR , 2018.

A. Nagrani, J. S. Chung, and A. Zisserman. Voxceleb: a large-scale speaker identification dataset. In INTERSPEECH , 2017.

M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing , pages 722-729. IEEE, 2008.

M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision , 2016.

M. Noroozi, H. Pirsiavash, and P. Favaro. Representation learning by learning to count. In IEEE International Conference on Computer Vision , 2017.

V. Panayotov, G. Chen, D. Povey, and S. Khudanpur. Librispeech: an asr corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing , 2015.

D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In IEEE Conference on Computer Vision and Pattern Recognition , pages 2536-2544, 2016.

M. Peplow. The 100,000 genomes project. British Medical Journal , 2016.

P. S. Pierre Baldi and D. Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communications 5 , 2014.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. In preprint , 2018.

A. Reiss and D. Stricker. Introducing a new benchmarked dataset for activity monitoring. In 16th International Symposium on Wearable Computers, ISWC , 2012.

J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. A. DePristo, J. V. Dillon, and B. Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems , 2019.

G. J. Rocklin, T. M. Chidyausiku, I. Goreshnik, A. Ford, S. Houliston, A. Lemak, L. Carter, R. Ravichandran, V. K. Mulligan, A. Chevalier, C. H. Arrowsmith, and D. Baker. Global analysis of protein folding using massively parallel design, synthesis, and testing. Science , 357 (6347):168-175, jul 2017.

A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations , 2019.

A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented neural networks. In International Conference on Machine Learning , 2016.

K. S. Sarkisyan, D. A. Bolotin, M. V. Meer, D. R. Usmanova, A. S. Mishin, G. V. Sharonov, D. N. Ivankov, N. G. Bozhanova, M. S. Baranov, O. Soylemez, et al. Local fitness landscape of the green fluorescent protein. Nature , 533(7603):397-401, 2016.

R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni. Green ai. arXiv preprint arXiv:1907.10597 , 2019.

V. Sitzmann, E. R. Chan, R. Tucker, N. Snavely, and G. Wetzstein. MetaSDF: Meta-learning signed distance functions. In Advances in Neural Information Processing Systems , 2020.

J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems , 2017.

N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 2014.

J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference on neural networks , pages 1453-1460. IEEE, 2011.

J. Tack, J. Park, H. Lee, J. Lee, and J. Shin. Meta-learning with self-improving momentum target. In Advances in Neural Information Processing Systems , 2022.

A. Tamkin, V. Liu, R. Lu, D. Fein, C. Schultz, and N. D. Goodman. Dabs: a domain-agnostic benchmark for self-supervised learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks , 2021.

A. Tamkin, M. Wu, and N. D. Goodman. Viewmaker networks: Learning views for unsupervised representation learning. In International Conference on Learning Representations , 2021.

A. Tamkin, G. Banerjee, M. Owda, V. Liu, S. Rammoorthy, and N. D. Goodman. Dabs 2.0: Improved datasets and algorithms for universal self-supervision. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS , 2022.

S. Thrun and L. Pratt. Learning to Learn . Springer, 1998.

M. K. Titsias, J. Schwarz, A. G. d. G. Matthews, R. Pascanu, and Y. W. Teh. Functional regularisation for continual learning with gaussian processes. In International Conference on Learning Representations , 2020.

A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017.

V. Verma, T. Luong, K. Kawaguchi, H. Pham, and Q. V. Le. Towards domain-agnostic contrastive learning. In International Conference on Machine Learning , 2021.

P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning , pages 1096-1103, 2008.

O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems , 2016.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2010-001 , 2010.

H. Wang, K. Song, J. Fan, Y. Wang, J. Xie, and Z. Zhang. Hard patches mining for masked image modeling. In IEEE Conference on Computer Vision and Pattern Recognition , 2023.

W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, and F. Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022.

P. Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209 , 2018.

C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. L. Yuille, and C. Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

M.-J. Wu, J.-S. R. Jang, and J.-L. Chen. Wafer map failure pattern recognition and similarity ranking for large-scale data sets. IEEE Transactions on Semiconductor Manufacturing , 28(1): 1-12, 2014.

S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519 , 2023.

Z. Wu, Y. Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE Conference on Computer Vision and Pattern Recognition , 2018.

A. Xie, A. Singh, S. Levine, and C. Finn. Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning , pages 40-52. PMLR, 2018.

Z. Xie, Z. Z. 0022, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. H. 0004. Simmim: a simple framework for masked image modeling. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

J. Xu, J.-F. Ton, H. Kim, A. R. Kosiorek, and Y. W. Teh. Metafun: Meta-learning with iterative functional updates. In International Conference on Machine Learning , 2020.

Z. Ye and L. Yao. Contrastive conditional neural processes. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations , 2018.

Q. Zhang, Y. Wang, and Y. Wang. How mask matters: Towards theoretical understandings of masked autoencoders. In NeurIPS , 2022.

R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In European Conference on Computer Vision , 2016.

Y. Zhang, K. Gong, K. Zhang, H. Li, Y. Qiao, W. Ouyang, and X. Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802 , 2023.

L. Zhou, H. Liu, J. Bae, J. He, D. Samaras, and P. Prasanna. Self pre-training with masked autoencoders for medical image classification and segmentation, 2023.

A Implementation details

In this section, we provide the implementation details of MetaMAE, including architectures and hyperparameters for MetaMAE when pretraining and evaluation.

Architectural details. We summarize our architectures in Table 7, with the hyperparameter notation referred from [89]. We use token embedding for encoder inputs, and apply positional embedding to both encoder and decoder inputs, as suggested by [89]. Specifically, token embedding separates the input data into fixed-size tokens, while positional embedding uses a fixed, absolute position represented by a combination of sine and cosine functions. We describe the token size for each specific dataset in Appendix C.

Pretraining details. Wesummarize our selected hyperparameters for pretraining each dataset in Table 8. Following [85], we pretrain MetaMAE for 100k iterations utilizing the AdamW optimizer [56] with both a learning rate and weight decay set at 1e-4. The batch size for pretraining and the strategy for selecting the mask ratio are detailed in [83, 85]. For the MetaMAE-specific hyperparameters, we observe that a certain set of hyperparameters can generally work across modalities, e.g., ( α , λ , decoder depth) = (0.5, 0.1, 4) (see Table 9 in Appendix B), or can be shared within each modality, e.g., P drop = 0 for Token modality (see Table 10 in Appendix B). Nevertheless, we recommend modalityspecific values for optimal performance (refer to Appendix B for hyperparameter sensitivity details). We set the temperature term for the contrastive loss τ = 0 . 5 and the NearbyS ratio r = 0 . 1 . For latent adaptation, the latent representation undergoes a single-step update with the update magnitude denoted by α .

We note that to scale up experiments, it is essential to facilitate distributed parallelism by using libraries such as BETTY [13] when utilizing PyTorch for meta-learning.

Evaluation details. In line with [85], we freeze the pretrained model and train either a linear classifier or a regressor for 100 epochs during the linear evaluation phase. We use the Adam optimizer [42] with both the learning rate and weight decay set as 1e-4. The batch size for this linear evaluation is set as described in [83, 85].

B Analysis on hyperparameter sensitivity

We here provide additional experiments on hyperparameters. This includes sharing various hyperparameters across modalities and conducting ablation studies with varying hyperparameters: α , λ , decoder depth, P drop, and the latent adaptation step size.

Sharing hyperparameters across modalities. As demonstrated in Table 9, MetaMAE shows robust performance regardless of the hyperparater selection. Notably, two or three major hyperparameters can be shared across all modalities, still outperforming prior methods. Furthermore, Table 10 indicates the resilience of MetaMAE's pretraining hyperparameters, especially at the intra-modality level.

Further ablation studies with varying hyperparameters. Table 11, 12, 13, and 14 show the sensitivity of hyperparameters on the PAMAP2 and WaferMap datasets. We observe that MetaMAE performs well even with non-optimal hyperparameters, except for the decoder depth and P drop, but we suggest finding better hyperparameters specific to each domain (e.g., λ = 0 . 1 for WaferMap). Regarding the decoder depth, we find that each modality requires an appropriate value, but generally, MetaMAE performs well with a decoder depth of 4. In Table 15, we observe that single-step adaptation effectively achieves good performance, and in some cases, even outperforms multiple-step adaptation due to the risk of overly decoder-specific support representation.

C Dataset details

We provide a summary of the considered datasets from the DABS benchmarks [83, 85] in Table 16. Note that we use the dataset split described in [83, 85].

Peer reviews:
comments: SUMMARY:
This paper presents Meta-learned Masked Auto-Encoder (MetaMAE), a novel modality-agnostic self-supervised learning (SSL) framework that leverages meta-learning to improve the transfer abilities of Masked Auto-Encoder (MAE). The authors reinterpret the mask reconstruction task of MAE as a meta-learning task and propose the integration of two advanced meta-learning techniques: gradient-based meta-learning and task contrastive learning. MetaMAE is evaluated on various data modalities from modality-agnostic SSL benchmarks, demonstrating significant improvements over previous modality-agnostic SSL methods in linear evaluation. The proposed approach also shows improved transferability on cross-domain datasets.

STRENGTHS:
1. The paper presents an innovative approach to modality-agnostic SSL by leveraging meta-learning and reinterpreting mask reconstruction in MAE as a meta-learning task.

2. This paper is well written.

3. MetaMAE consistently outperforms previous modality-agnostic SSL methods in linear evaluation, indicating its effectiveness.

4. Contrastive learning in MetaMAE can improve performance.

WEAKNESSES:
1. There are significant differences in pretraining and fine-tuning hyperparameters for various downstream tasks, such as masking ratio, batch size, and decoder depth, as shown in Appendix Table 2.

2. The paper lacks innovation.

3. In the ablation experiments, it can be observed that the proposed Gradient-based and Task contrast methods have minimal impact on most tasks.

QUESTIONS:
N/A

LIMITATIONS:
addressed
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this paper address modality-agnostic self-supervised learning (SSL), a significant yet under-researched problem with various real-world applications. They explore the potential of the Masked Auto-Encoder (MAE) in this context and propose SSL framework, MetaMAE, which enhances MAE with meta-learning. They reinterpret the mask reconstruction task of MAE as a meta-learning task, which in turn allows them to apply modality-agnostic meta-learning methods to improve MAE's generalization. Experimental results reveal that MetaMAE improve the performance of modality-agnostic SSL approaches across multiple modalities.\n\nSTRENGTHS:\nThis paper views Masked Auto-Encoder (MAE) from a meta-learning perspective, which is interesting.  \nThe paper is well-structured, allowing for easy comprehension of the information flow.   \nThe explanation on how support and query sets corresponds to tokens is clear.  \nThe experiment section is solid. The paper provides a comprehensive evaluation of the proposed method through thorough experiments and comparisons with existing models. \nThe approach shows great generalizability on cross-domain linear evaluation task.\n\nWEAKNESSES:\nThe experiment setup for meta-learning based gradient update could benefit from more explanation. Like how is the support and query set constructed during training for gradient updates.\n\nMeta-learning usually requires more compute due to the computation of second order gradients, and discussion on computational overhead could be necessary.\n\nDespite the variety of tasks for evaluation, most of the datasets used for evaluation is of small-scale.  \n\nOne important ablation study missing to validate the effectiveness of the proposed overall framework. (See question below)\n\nQUESTIONS:\nIn ablation study section, Tab.3, why not use only task contrast without gradient-based component to see how much of the gain comes from contrastive learning itself?\n\nAlso, how to balance the meta-learning gradient update and contrastive learning update? Would they influence each other? Is the hyper-parameter lambda grid searched?\n\nLIMITATIONS:\nMasking scheme limitation on MAE is discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper studies self-supervised learning (SSL) for multiple modalities (e.g. tabular data, images, text, speech). Central to this paper are Masked Auto-Encoders (MAEs), a popular SSL training paradigm where models are tasked to reconstruct inputs from randomly masked subsets. The authors interpret this reconstruction task as a meta-learning task, and based on this interpretation, propose a new method to improve the performance of MAE, called MetaMAE. The proposed method provides solid improvements over previous work on several datasets and modalities from the DABS benchmark.

STRENGTHS:
- The results presented in the paper are quite strong, surpassing previous results often by a large margin. As such, I believe this paper presents a solid advance over existing literature, and would be of interest to most researchers working on SSL.
- The paper is clear and well written.
- The experiments are solid and extensive, and there are many ablation studies which allow readers to gauge the importance of different design decisions and components of the method presented in this paper.

WEAKNESSES:
- It would be interesting to see if the proposed method can be used train a single model jointly on all the modalities and benchmarks, instead of training one separate model for each. One of the limitations discussed by the authors is that the masking ratio can differ across modalities, which might pose a problem. However, building a general model that could simultaneously perform all the tasks could further demonstrate the generality of the proposed method.

QUESTIONS:
No questions.

LIMITATIONS:
To the best of my knowledge the authors have properly discussed the limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

comments: SUMMARY:
This work interprets MAE as a meta-learning framework to formulate a modality-agnostic SSL framework. Specifically, they formulate the masked data and unmasked data (nearby tokens) as the support set and the query set, respectively. Moreover, the authors proposed to use gradient-based meta-learning on a decoder and task contrastive learning to optimize the encoder. Extensive experiments with ablation studies show the effectiveness of the proposed framework.

STRENGTHS:
+ The idea of leveraging meta-learning and MAE to address modality-agnostic SSL is novel and interesting.
+ The design of using gradient-based meta-learning on a decoder and task contrastive learning seems to be effective.
+ The experiments on various modalities and the detailed user study are convincing.

WEAKNESSES:
- It is necessary to further clarify the relation between the proposed method (i.e., MAE+meta-learning) and the task (i.e., modality-agnostic SSL), that is the motivation for formulating MAE as a meta-learning paradigm. For example, what's the advantage of  the meta-learning paradigm for modality-agnostic SSL? In the paper, the authors only explain that their formulation can "improve the generalization", which is not specific.
- It would be better to discuss the potential advantages of the proposed method to solve multimodal SSL.

QUESTIONS:
Please address my concerns in the paper Weaknesses.

LIMITATIONS:
The authors have addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26

comments: SUMMARY:
The paper titled "Modality-Agnostic Self-Supervised Learning by Meta-Learning" presents a novel framework called MetaMAE (Meta-learned Masked Auto-Encoder) for modality-agnostic self-supervised learning (SSL). The authors propose an innovative interpretation of Masked Auto-Encoder (MAE) as a meta-learning framework and enhance its generalization capabilities using advanced meta-learning techniques.

STRENGTHS:
The paper includes extensive evaluations on multiple data modalities, demonstrating the effectiveness of MetaMAE. The results show significant improvements over previous modality-agnostic SSL methods in linear evaluation.

WEAKNESSES:
I do not see any critical weakness for this paper. Authors have done pretty good job working out the details and limitations.

QUESTIONS:
I guess it is more of a vague questions, but in your experience, what might help handling diverse and complex data with this approach? What is keeping MetaMAE to become the one approach to rule them all?

LIMITATIONS:
Authors have done a good job addressing the concerns and limitations on the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MetaMAE, a novel framework to improve Masked Autoencoder (MAE) for modality-agnostic self-supervised learning. Unlike many SSL methods reliant on domain-specific inductive biases, MetaMAE enhances MAE with two meta-learning techniques: latent adaptation via gradient-based meta-learning to optimize amortized features and task contrastive learning to align amortized and optimized features. Conducting extensive experiments with modality-agnostic SSL benchmark DABS 1.0 and 2.0, MetaMAE consistently improves across diverse modalities like time series, tabular, images, and speech.\n\nSTRENGTHS:\n* **(S1)** This paper is well-motivated to enhance MAE with meta-learning strategies and design modality-agnostic pre-training methods. Extensive experiments on DABS and DABS 2.0 show the notable performance gains of the proposed MetaMAE, which highlights the potential of this research direction. Experimental details and the source code are provided for reproduction.\n\n* **(S2)** The overall presentation is smooth and easy to follow. The background and related works of the manuscript are well-studied.\n\nWEAKNESSES:\n* **(W1)** The designed method is straightforward and simple, while the intuition or analysis of why meta-learning strategies improve MAE for modality-agnostic pre-training is not well studied. Since the vanilla masked predictive learning frameworks like MAE and Data2Vec [1] are already modality-agnostic, the author should provide some empirical analysis to support the statements in Sec. 3.2. Or it will decrease the novelty of MetaMAE.\n\n* **(W2)** Some hyper-parameters need tuning for each modality, e.g., the decoder sizes and pre-training settings. As provided in the appendix, some sensitive hyper-parameters might have different behavior in different modalities.\n\n* **(W3)** Although the authors study a wide arrange of related papers, some recently proposed general pre-training frameworks should be included. For example, Masked Auto-Encoder methods like Data2Vec variants [1, 2] (modality-agnostic) and SimMIM variants [3, 4] (architecture-agnostic); Multi-modality pre-training like BEiT variants [5]; Generative Pre-training like VQ-based variants [6, 7] (can be applied to various modalities with some modifications). It is better to discuss the similarity between MetaMAE and these methods.\n\n* **(W4)** Typos should be double-checked before publishing, e.g., `pre-pre-training` in line 113.\n\n#### Reference\n[1] Alexei Baevski, et al. “data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language.” ICML, 2022.\n\n[2] Alexei Baevski, et al. “Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language.” ICML, 2023.\n\n[3] Chen Wei, et al. “Masked Feature Prediction for Self-Supervised Visual Pre-Training.” CVPR, 2022.\n\n[4] Siyuan Li, et al. “Architecture-Agnostic Masked Image Modeling - From ViT back to CNN.” ICML, 2023.\n\n[5] Wenhui Wang, et al. “Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks.” arXiv, 2023.\n\n[6] Patrick Esser, et al. “Taming Transformers for High-Resolution Image Synthesis.” CVPR, 2021.\n\n[7] Tianhong Li, et al. “MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis.” CVPR, 2023.\n\nQUESTIONS:\nSorry for the late review! No more questions besides the concerns mentioned in Weaknesses.\n\nLIMITATIONS:\nLimitations have been discussed by the authors and I found no more limitations.\n\n### Post-rebuttal\nThe authors' replies have tackled my concerns, thus, I raised the rating to 6. I hope the authors add more discussion with existing works and specify the limitations and future research directions for this topic.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-09"}