Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder
Authors: Huiwon Jang
Jihoon Tack
Daewon Choi
Jongheon Jeong
Jinwoo Shin
Venue: NeurIPS 2023 poster (2023)
Keywords: Self-Supervised Learning
Modality-Agnostic Self-Supervised Learning
Meta-Learning
Masked Auto-Encoder

Abstract: Despite its practical importance across a wide range of modalities, recent advances in self-supervised learning (SSL) have been primarily focused on a few well-curated domains, e.g., vision and language, often relying on their domain-specific knowledge. For example, Masked Auto-Encoder (MAE) has become one of the popular architectures in these domains, but less has explored its potential in other modalities. In this paper, we develop MAE as a unified, modality-agnostic SSL framework. In turn, we argue meta-learning as a key to interpreting MAE as a modality-agnostic learner, and propose enhancements to MAE from the motivation to jointly improve its SSL across diverse modalities, coined MetaMAE as a result. Our key idea is to view the mask reconstruction of MAE as a meta-learning task: masked tokens are predicted by adapting the Transformer meta-learner through the amortization of unmasked tokens. Based on this novel interpretation, we propose to integrate two advanced meta-learning techniques. First, we adapt the amortized latent of the Transformer encoder using gradient-based meta-learning to enhance the reconstruction. Then, we maximize the alignment between amortized and adapted latents through task contrastive learning which guides the Transformer encoder to better encode the task-specific knowledge. Our experiment demonstrates the superiority of MetaMAE in the modality-agnostic SSL benchmark (called DABS), significantly outperforming prior baselines.

Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder
Huiwon Jang** Jihoon Tack* Daewon Choi Jongheon Jeong' Jinwoo Shin
Korrea Advanced Institute of Science and Technology (KAIST) Kooee University (huiwoen0516, jihoontack] @kaist.ac.kr
Abstract
Despite its practical importance across wide range of modalities, recent advances in sufervupere learning (SSL) have been primarily focused on few well- curated domains, e.g., vision and language, often relying on their domain-spectin knowledge. For example, Masked Auto-Encoded (MAE) has become one of the popular architectures these domains, but less has explored its potential in other modalities. In this paper, we develop MAE as unified, modality-agnostic agnostic SSL framework. In turn, we argue meta-learning as key to interpreting MAE as a modality-annostic learner, and propose enhancements to MAE from the motivation to jointly improve its SSL across diverse modalities, coined MetaMAE as a result. Our key idea is to view the mask reconstruction of MAE as meta-learning task: masked tokens are predicted by adapting the Transformer meta-learner through the amortization of unmasked tokens. Based on this novel interpretation, we propose to integrate two advanced meta-learning techniques. First, we adapt the amortized latent of the Transformer encoder using gradient-basee the reconstruction. Then, we maximize the alignment between amortized and adapted latents through task contrastive learning which guides the Transformer encoder to better encode the task-specific knowledge. Our experiment demon- strates the superiority of MetaMAE in the modality-agnoste SSL benchmark (called DABS), significantly outperforming prior baselines. Code is available at https///ithhbbccm//iiilaaaaaaaa
meta-learning to enhance
Introduction
Self-supervised learning (SSL), i.e., learning without human supervision, recently has demonstrated
substantial success across fields including, computer vision [32, 11, 29, 47, 33, 5, 102], natural language processing (NLP) [18. 49, 55, 70]. and speech recognition [2, 38, 39]. The efficacy of SSL is derived by extracting transferable knowledge from unlabeled datasets, a feature that manifests significant utility for various downstream tasks such as classification and segmentation. As result, SSL has become indispensable technique in real-world applications (for instance, industrial contexts like medical imaging [26]). not only improving the performance on new datasets but also reducing significant amount of computations and costs, e.g., expert annotation poses significant costs [68, 16]. However, despite the importance SSL in such fields, recent advancements have been predominantly focused specific domains (e.g., images and NLP) where the majority of existing SSL frameworks on such domains require modality-speciff knowledge, thereby constraining the applicability and scalability of previous works across new modalities.
To tackle this issue, we draw attention to the recent success of the Masked Auto-Encoder (MAE) framework [33], which eliminates the need for modality-specine inductive biases. Initially presented as a generative model [91, 67], the MAE models the network to reconstruct the original input
Equul contributions
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Zx
Sx
Encoder
Projector
f0
Righ
AvgPool
7x
Reconstruction
Task Contrast
AvgPool
Zx
g Decoder
Forward
Latent adaptation
Figure I: An overview of the proposed Meta-learned Masked Auto-Encoder (MetaMAE); we adapt the amortized latent 2x of the Transformer encoder fo using gradient- meta-learning on the Transformer decoder Os to enhance the reconstruction, then maximize the alignment with optimized latent Z to guide the Transformer encoder towards improved predictions via task contrastive learning.
signal based on the randomly masked part of the signal. Recently, the integration of MAE with the Transformer architecture [33] has resulted a powerful SSL framework for various domains, such as vision [33, 5, 10J. NLP [18, 70], and tabular [59] datasets. For instance, has been demonstrated that BERT [18], utilizing Transformer encoder with a linear decoder, can effectively transfer to diverse tasks for the NLP domain. On the other hand, it has been suggested that utilizing decoder of a deep Transformer architecture is essential for applying MAE within the vision domain [106], achieving remarkable performance [33]. Building on these insights, we found that MAE is quite promising direction for modality-agnoste SSL: our experiment demonstrates that using deep Transformer decoder for MAE significantly outperforms previous mggnsity-agnostic SSL frameworks (see Table 5). In this paper, we suggest to further exploit the benefits of MAE to build a unified SSL framework.
Contribution. We propose Meta-learned Masked Auto-Encoder (MetaMAE), novel modality- agnostic SSL framework that leverages the power of meta-learning; see the overview in Figure 1. Our key idea is interpret MAE meta-learning framework, thereby improving the general- ization through the use of advanced meta-learning schemes. To be specific, we interpret the data reconstruction itself as task, where the Transformer meta-leamme is adapted through amortization of the support set (i.e., unmasked tokens) to predict the query set (i.c., masked tokens). Based on this interpretation, we propose novel integration of advanced meta-learning techniques to enhance MAE; namely the use of gradient-bact meta-learning [22] and task contrastive learning [28, 60].
Latent Adaptation via Gradient-based Meta-learning: We suggest adapting the amortized latent to better reconstruct the given support set (and the nearby tokens) through gradient- hased meta- learning on the decoder [74]. Then, the optimized latent is used to condition the decoder for the query prediction. This approach generally eases the task compared to the direct reconstruction, thereby streamlining and improving the task adaptation process.
Task Contrastive Learning: To further leverage this optimized latent, we suggest utilizing task contrastive learning [28, 60). Specifically, since both the optimized and predicted latents originate from the same task, we aim to maximize their similarity while minimizing their similarity with other tasks. This prompts the Transformer encoder produce predictions closely aligned with the optimized latent, effectively guiding the Transformer to better encode the task knowledge.
We verify the efficacy of MetaMAE through extensive evaluations on multiple data modalities from modality-agnostic SSL benchmarks (i.c., DABS 1.0 [83] and 2.0 [85]), including time-series, tabular, discrete token, multi-spectral images, speech, and multi-modal datasets. Overall, our experimental results demonstrate strong results, consistently and significantly outperforming previous modality- agnostic SSL methods in linear evaluation. For instance, measured with classification accuracy (%). MetaMAE improves the prior state-of-the-
results by 85.3 - 89.3 on PAMAP2 [71], 53.6 - 69.4 on Genomics [72], and 60.2 - 79.8 on LibriSpeech [66] datasets. Moreover, we also demonstrate that MetaMAE significantly improves the linear evaluation performance on cross-domain datasets, indicating the improved transfer ability of MAE through meta-learning.

---

2 Meta-Learning Modality-Agnostii Masked Auto-Encoder
In this section, we present Meta-learned Masked Auto-Encodoe (MetaMAE), a novel and effective mggasity- self-supervised learning (SSL) framework. Our key contribution is to tackle the mogality- SSL problem with in-depth utilization of MAE, which was quite under-explored in the field. Based on our novel interpretation that views MAE as a meta-learning framework (in Section 2.1). we improve the transfer ability of MAE by suggesting enhanced modality- meta-learning techniques including latent optimization-basee meta-learning and task contrastive learning (in Section 2.2). Our framework is visually depicted in Figure I, and the pseudo-code is provided in Algorithm 1.
Problem setup. We first describe the problem setup of our interest, modality-agnostic SSL.. This problem aims to learn a transferable representation from the unlabeled dataset without utilizing the modality-specifie inductive biases. For given unlabeled pretrain dataset Ppretrain {xx)== where x Id represents an input sampled from a certain data-generating distribution in an i.i.d manner, our objective is to train an encoder fe that can linearly separate the given labeled transfer dataset drawn from a similar or the same data-generating distribution.
2.1 Rethinking Masked Auto-Encoder Meta-Learning Framework
Meta-learning [86] aims to extract and utilize the knowledge from the distribution of tasks to better solve relevant task. This problem is typically approached by training meta-leamer that can transfer its knowledge to a task-specific model through adaptation, where the performance of the meta-learner is evaluated on the basis of how well each adapted model performs task. To learn such a meta-learner, a standard way is to use a set of support sel samples to adapt the task-specific model from the meta-learner and use another disjoint set of samples, called query set samples to evaluate the adaptation performance [92, 82].
the corresponding
Mask prediction as modallty-agnosste task. MAE is an SSL technique that trains autoencoder to reconstruct the original input signal with a randomly masked part of the signal. To implement such technique, recent works utilize the Transformer architecture for the autoencoder design which is necessary for successful training. To use Transformer for MAE, the input data is broken down into non-overlapping units coined tokens (e.g.. patches for images, and words for languages) where such tokens are divided into two disjoint sets (unmasked and masked) for MAE modeling, i.e., the Transformer autoencoder predicts the masked token using the unmasked tokens.
Our key insight is to interpret the signal reconstruction of MAE as a meta-learning task, where two disjoint unmasked and masked token sets are viewed as support and query sets to adapt and evaluate the Transformer meta-learner. To be specific, the Transformer encoder extracts the task knowledge through amortization of the support set, where this amortized latent adapts the Transformer decoder to predict the query set of the task. Formally, for a given data sample x, we first divide the signal into two disjoint sets, namely the support set Sx and the query set Qx by utilizing the tokenize m= Sx U Qx- Then, for a given Transformer encoder fe
operation tokenize(* (x) := {m, { and decoder Jo. MAE minimizes the discrepancy between the predicted token and the corresponding masked token (i.e., the query sample) as:
CHME(O, o; Qx) = Σ
d(x)), .90 (Zx)) where 2x = =f((S))
(I)
(g.2(0))E
where d(.,.) is a discrepancy function: l norm for continuous (e.g.. time-series, speech) and cross- entropy for discrete (c.g., token) datasets, respectively. Based on this interpretation, we improve the transfer ability of MAE (for modallty-agnostt SSL) through novel integration of two effective modality-agme agnostic meta-learning techniques to MAE
2.2 MetaMAE: Improving Masked Auto-Encoder through Meta-Learning
We now describe our method, MetaMAE, which further improves the representation of MAE through a novel integration with advanced modality-agnosti meta-learning techniques. In a nutshell, Meta- MAE operates by further optimizing the amortized latent of the Transformer encoder using gradient- based meta-lea Then we maximize the alignment between the optimized and the amortized latents via contrastive learning, to guide the Transformer encoder to improve the generalization.
3

---

Algorithm MetaMAE: Meta-ne Modality-Agnostic Masked Auto-Encoder
Require: Unlabeled pretrain dataset D Pretrain. weight hyperparameter A, Nearby-S ratio 7, batch size B, learning rates a, 3.
1: Initialize 0, Φ, & using the standard initialization scheme.
2: while not done do
3: Sample mini-batch B [x:)/1 from D pretrain
4: for i 1 B do
Note: we the batch computation.
5: Sample Support set Sx; and Query set 2x, from X; 2x, = SS((S)
6:
b. Amortization through Transformer encoder.
7:
Sample ((xxiir) where (V(Sx.))) [(xx,;))==r>> = 12xl
Sample the Nearby-S tokens.
8:
2x, 2x; aV VV C0s oz; Sx where Sx = Sx: UN((Sxir)
> Adapt the amortized latent.
9:
Compute MAE reconstruction loss Ligrad with Zxz
Eq. (2)
10:
Compute task contrastive loss Liesk-cou with (2x) and (2x) Compute MetaMAE loss Listanns with Cgrad Liaax-con and À
Eq. (4)
11:
D Eq. (5)
12:
end for
13:
8 B E. CHoTaNAR 0, o, db + 0, 0.
b Update the entire networks.
14:
end while
Latent adaptation via gradient-based meta-learning. To further improve the generalization of MAE, suggest utilizing the gradient-based meta-learning (i.e., model-agnostic meta-learning MAML [22]) on the amortized latent space [74]. Specifically, we adapt the amortized latent of the support set to better reconstruct the support and the nearby tokens (of support tokens) by using the gradients of the decoder. Then, we utilize the optimized latent to condition the decoder to predict the query tokens. Here, our key idea is the of nearby tokens when optimizing the latent, which turns out to be crucial for improved performance. Intuitively, optimizing such tokens induce an error correction the latent, which eases the mask reconstruction (or prediction) task compared to the direct reconstruction, and thereby improves the task adaptation process [103].
Concretely, for given support set Sx, select the nearby tokens of support tokens from the query set 2x, namely N rS) C 2x. such that the cardinality is ([Sx; = x |Ωxl with a ratio of r > O. Then, we optimize the amortized latent Zx to better reconstruct the support and the nearby tokens &x == Sx U UW((S;; using the decoder gradient, then condition the meta-learner, i.e., the Transformer decoder go, to predict the query set Qx as follows:
Ggrea(X,0,0) = Σ
d (x
96
(0) (2x) where 2 - - aV z. LHNE 60 Ox)
(2)
(E.
where > 0 is the step size for the adaptation. One can easily extend the latent optimization to obtain 2 with more than one gradient step where we found a single step adaptation is already quite effective yet showing computation efficiency compared to multiple iterations. Furthermore, we found that is important to use the second-order gradients for the adaptation, i.e., backpropagation on the decoder adaptation gradient when optimizing the loss function, which enables the Transformer encoder to better amortize for the reconstruction task. Note that this gradient calculation on the decoder does not increase the computation too much, using smaller decoder size (compared to the encoder) is the key to the success of MAE [33].
Task contrastive learning. To further exploit the benefit of the gradient-based meta-learning, we suggest nudging the amortized latent
as close as possible to the further optimized latent in Eq. (2). By doing so, the Transformer encoder is guided to better encode the reconstruction task knowledge as the optimized latent further adapted with support and the nearby tokens. To effectively implement this concept, we utilize the idea of task contrastive learning [28, 60, 104]. Specifically, as both the optimized and amortized latents originate from the same task, we maximize the latent similarity within the same task while minimizing the similarity with other task latents.

---

Formally, let 2x and 2 be the amortized latent and optimized latent of the given input from mini- batch x E B, respectively. We then use non ain-linear projection network hy and the average set pooling of latent tokens to obtain task-specific representation xx he (pool 2x)) and z he ( (pool 2x) for the task contrastive learning. For given set of task-specific representations 7 UxESS (oxs zzz. the task contrastive objective is defined as follows:
Loa (x, 8, 1/)
|con 2xi Z T \ (2x1) + Icon (z Zx, T \(zxt)
(3)
2
cxp (sim (2im(z22*) 2/T))
where lcon Gzn(2;;0) (2 {z } logg exp (sim(z, 2 )/7) + E2 exp (sim(z, 2 )/7)
(4)
where sim(z, == 2 z ///[|z]] |z||| z be the cosine similarity and T > O the temperature hyper- parameter. From the perspective of contrastive representation learning, our task contrastive framework can be viewed as augmenting the positive pair. However, instead of using homain-speciinn induc- tive biases, we leverage gradient adaptation, thereby showing the possibilities of extending prior contrastive learning methods to modality-agnose
SSL frameworks.
Overall meta-learning objective. In the end, derive final training objective, LHecaMAE' meta- learning objective combining the latent adaptation Eq. (2) and the task contrastive learning Eq. (4). For given Hlyper-paamette À > 0, the meta-objective of MetaMAE becomes:
LMataMaR(* (x, 0, ④2 1b) == Lerra x, 0, () + XLrasi con (x, 0, v))
3 Experiments
In this section, we demonstrate the effectiveness of the proposed framework by measuring the linear- evaluation performance under various datasets across modalities, We first describe our experimental setup (Section 3.1), and then we present the main experimental results (Section 3.2). We provide ablation studies regarding MetaMAE (Section 3.3).
3.1 Experimental Setup
We here briefly describe overall experimental setups. We provide further details of pretraining, evaluation, and hyperparameters in Appendix A.
Datasets. We select 8 sub-benchmarks from the DABS 2.0 benchmark [85], with categorizing the modalities for each sub-benchmark. We pretrain and transfer MetaMAE on the selected datasets:
Time-series modality consists of datasets where the data is organized sequentially over time. In this paper, we use the PAMAP [71] dataset, which contains sensor signals from physical activity.
Tabular modality refers datasets where the data is structured in table format, with rows (for instances) and columns (for attributes). We use the HIGGS [69] dataset from particle physics.
Multi-spectral (MS) Image modality contains multi-channel 2D image datasets. We use the EuroSAT [34, 35) dataset, which consists of 13-channel satellite images.
Token modality features datasets consisting sequences of discrete units, similar to natural lan- guages. We pretrain MetaMAE on both (a) the Genomics [72] dataset, subsequently transferring
the learned model the Genomics and Genomics-OOl OOD datasets; and (b) the Pfam [20] dataset of
proteins, followed by transfer learning to several tasks from the TAPE benchmarks [20], including
Pfam, SCOP [23], Secondary Structure [43, 8). Stability [73], and Fluorescence [76].
Speech modality includes 2D spectrograms of audio datasets. We pretrain MetaMAE Lib-
riSpeech [66]. large English audiobook corpus, and then transfer the model to datasets including
LibriSpeech, Audio MNIST [6], Fluent Speech [57], Google Speech [96], and VoxCelebl [62].
RGB Image modality comprises 3-channel 2D image datasets. We pretrain MetaMAE on (a) the ImageNet32 [17] dataset, which is scaled to 32 x 32, and transfer the pretrained model to datasets including CIFAR-10 [45]. CUB [93]. VGG Flowers [63], DTD [14], Traffic Sign [81]. and Aireraft [58]: and (b) the WaferMap [98] dataset.
Vision-Language modality comprises a combination of 3-channel 2D image and sequences of English text descriptions. We pretrain MetaMAE on MSCOCO [54], and then transfer the model to mismatched-captee detection [54] and the Visual Question Answering (i.c., VQA) tasks [1].
5

---

Table In-domain linear evaluation performance across multiple modalities. We report Fi-score (%) for MaferMap and the classification accuracy (%) for the rest. MS Image indicates the Multi- Multi-spectral image modality. * and denote the results from the DABS 1.0, and DABS 2.0 paper, respectively, where of Capri results indicates that the pretraining loss divergence as described in [85].
Modality
Time-series
Tabular
MS Image
Token
Speech
RGB Image
Dataset
PAMAP2
HIGGS
FuroSAT
Genom
Pfam
Libri
WaferMap
Random initialization
Baseline
69.81
54.81
62.3
37.2
30.1
17.1"
77.7
Self-super learning Framework
e-Mix
80.1
65.7
87.4
40.5
31.3
60.2
92.6
ShED
85.2
68.01
61.51
33.6
54.7
34.8"
92.4
Capri
67.4*
23.5
27.4
25.4
92.51
MAE
85.31
70.0
86.31
53.6
44.7
46.0
93,91
MetaMAE
89.3
71.5
88.5
69.4
62.3
79.8
95.5
Note that the transferred datasets can be in-domain (i.e., same dataset) or cross-domain (i.e., different dataset). The details of the benchmarks are described in Appendix C.
Baselines. For the main experiments, we compare MetaMAE's performance with existing modality- agnostic sulf-supe learning methods suggested by DABS 1.0 [83], and 2.0 [85]:
e-Mix is generalized version of i-Mix [48], designed consistently apply methods across both discrete and continuous domains by applying the mixup strategy in the embedding space.
ShED generalized version of ELECTRA [15]. ShED constructs the pretext task, which involves predicting shuffled embeddings.
Capri applys contrastive learning to the token level representation by randomly masking the token and treating different tokens negative pairs.
MAE aims to reconstruct the input. However, here, MAE employs a linear decoder for the continuous domain and decoder for the discrete domain.
Additionally, we regard the randomly initialized encoder, referred to as the Baseline, as one of the baseline to check the effectiveness of self-supervised pretraining.
Architectures. Following [83, 85], we use 12 layers for the transformer encoder with the hidden size 256, and 8 attention heads. For the decoder, fix the hidden size 128, and 4 attention heads. However, we choose an appropriate number of layers for the decoder to demonstrate the effect of the decoder for MAE. We also utilize different hyperparameters for each modality other baselines, but we find that the hyperparameters can be shared across modalities (See Appendix B).
Pretraining and transfer learning. To evaluate our method, we pretrain each dataset 100K iterations and 100 epochs transfer learning, overall experiments by following [85]. We pretrain entire networks, ie.. encoder fa. decoder ga. and projection header hg, but we utilize only the frozen encoder fe on transfer learning. When pretraining, the masking ratio can differ from the datasets. For the masking ratio paramerr, choose the best value among candidates suggested by the prior work [85].
3.2 Main Experiments
In-domain linear evaluation. We evaluate the pretrained representation on each in-domain down- stream classification task. We report the performance of a linear classifier trained on top of the frozen features. The results in Table demonstrate that our proposed method, MetaMAE, achieves state-of-the-an performance across the entire dataset. For instance, we obtain 16% accuracy gain (53.6% + 69.4%) on Genomics. Moreover, we note that MAE has achieved moderate performance compared to other self-supervised learning (SSL) methods on these benchmarks, but MetaMAE demonstrates the ability to enhance MAE and outperform other SSL approaches. For example, MetaMAF achieves the best performance on Pfam (44.7% - 62.3%) and LibriSpeech (60.2% - 79.8%) with significant improvement, here is where MAE reported in [85] (i.e., MAE with linear decoder) was not the best among baselines.

---

Table 2: Cross-domain linear evaluation performance across multiple modalities. We report the Spearman correlation for Stability and Fluorescence datasets, and the classification accuracy (%) for the rest. denote the results from the DABS 1.0 paper.
SSL Framework
Pretrain data
Transfer data
Baseline
e-Mix
ShED
Capri
MAE
MetaMAF
Genomics
Genomics-OOn
8.6
9.7
7.3
22.2
37.2
SCOP
8.0
5.7
10.7
2.0
7.9
11.8
Secondary
52.4
53.7
67.6
49.5
62.5
65.9
Pfam
Stability
0.31
0.39
0.53
0.26
0.40
0.53
Fluorescence
0.04
0.20
0.27
0.06
0.06
0.31
Audio MNIST
33.1"
80.4"
67.3"
53.6
45.1
89.5
Fluent Loc
62.1"
60.9"
60.2"
59.8
61.7
66.7
LibriSpeech
Fluent Act
26.2"
29.9"
30.5"
28.3
26.8
38.4
Fluent Obj
30.1"
39.9"
39.4"
33.1
32.0
49.3
Google Speech
4.9"
19.2"
20.7"
13.7
9.5
46.8
VoxCeleb1
0.6"
2.4"
2.8"
1.6
1.6
7.4
CIFAR-10
24.2
39.4"
39.6"
48.7
46.0
59.2
CUB
1.6"
3.9
3.0"
3.7
3.1
6.3
ImageNet32
VGG Flowers
9.0"
26.0"
13.0
18.6
22.2
36.3
DTD
7.4
8.8"
18.4
14.7
14.2
20.9
Traffic Sign
14.3"
65.1"
27.5"
28.0
32.0
67.1
Aircraft
2.7"
10.2"
5.6"
6.4
5.9
16.4
Table 3: Linear classification accuracy (%) pretrained on a vision-language dataset, MSCOCO.
SSL Framework
Pretrain data
Transfer data
Baseline
e-Mix
ShFD
Capri
MAE
MetaMAF
VQA Mismatched-caption
53.4
57.6
53.1
52.9
54.2
69.7
MSCOCO
49.8
50.1
50.6
49.6
49.3
70.5
Cross-domain linear evaluation. We evaluate our method on a diverse set of cross-domain down- stream tasks including both classification and regression. We employ linear classifier, regressor trained on the frozen features as the in-domain setup. Table 2 shows that MetaMAE outperforms all the baselines across all the benchmarks consistently, except for specific dataset. For example, we obtain 9% accuracy gain (80.4% + 89.5%) on the linear classification performance of transfer setup from LibriSpeech Audio MNIST. It is important to note that cross-domain downstream tasks, due to their wider range of variations for each domain, are typically more challenging to consistently excel in compared to in-domain tasks. This significant performance improvement demonstrates the applicability of MetaMAE in various cross-domain transfer learning scenarios across the modalities.
Multi-modal dataset evaluation. One important future direction for the modality-agnostic SSL research community is to bind all modalities under a singular model [99, 108]. Here, we believe MetaMAE can be quite a promising method to tackle this problem, e.g., managing multiple modalities on a single model supplemented by domain-specific embedding modules. To this end, we verify the possibility of MetaMAE for tackling unified multi-modal self-supervised learning. As shown in Table 3, MetaMAE outperforms other modality-agnostii SSL methods on the vision-language tasks where we believe this multi-modal learning ability can help when unifying the modalities for SSL.
3.3 Ablation study
We perform an ablation study on six modalities: time-series (PAMAP2). tabular (HIGGS), speech (LibriSpeech), multi-spectral image (EuroSAT), and token (Pfam and Genomies). Throughout this section, we report the in-domain linear classification accuracy (%), unless otherwise specified.
Component analysis. In Table 4, we demonstrate the necessity of each component in MetaMAE by adding each component one by one: Deeper decoder go with Transformer architecture, latent optimization via gradient-base based meta-learning, and the task contrastive loss Lisek-cou- We first found
7

---

Table 4: Ablation study on each component of MetaMAE, namely the use of the decoder, latent adaptation using gradient- based metanng craudient-based)) and task contrastive learning (Task contrast). We report the classification accuracy (%) across six different modalities.
Decoder
Gradient-
Task contrast
PAMAP2
Genomics
EuroSAT
LibriSpeech
HIGGS
Pfam
X
X
85.3
53.6
86.3
33.3
70.0
44.7
X
X
86.5
65.2
87.4
64.1
70.5
61.3
88.3
69.4
87.4
64.5
71.1
61.3
89.3
69.4
88.5
79.8
71.5
62.3
Table 5: Effect of the decoder size of MAE on the classification accuracy (%). We use three different datasets across modalities.
Table 6: Effect of the nearby token (i.e.. Nearby-S) selection ratio r on the classifi- cation accuracy (%). We use three different datasets across modalities.
decoder size
EuroSAT
Pfam
LibriSpeech
prev, best
87.4
54.7
60.2
ratio
PAMAP2
HIGGS
Pfam
86.3
44.7
33.3
0
87.5
71.1
62.0
86.7
61.4
68,1
89.3
71.5
62.3
4
87.4
61.3
64.1
0.5
88.2
70.8
62,0
6
86.7
61.4
74.1
1.0
84.2
70.1
62.1
that incorporating a deep decoder is a critical component in our framework, enabling Nomain-agnostic capabilities similar to the success of MAE the image domain [33]. Thus, here suggest that improving MAE for the domain-agnostic is quite a promising direction to explore.
In addition, Table 4 verifies the contribution of meta-learning schemes to the performance of Meta- MAE. We found that the gradient- pradeent-based latent optimization rule, which includes the utilization of Nearby-S, is more beneficial. We also confirm that task contrastive learning is critical component our framework like recent meta-learning frameworks [28, 60, 104]. Note that this task contrastive learning scheme is exclusively applicable in gradient-based approaches, emphasizing the significance of the gradient-based latent optimization method for MetaMAE.
Importance of decoder size for MAE. To verify the effect of decoder size for MAE, we evaluate the linear evaluation accuracy on datasets where the original MAE (i.e., no decoder) performed worse than other baselines. As Table 5 shows, we found that MAE can achieve the best performance compared baselines by choosing the proper decoder size, yet there is room for enhancement as shown in Table 4. This result demonstrates the superiority of MAE for tackling modality- SSL problems, where we believe the development MAE would be an important direction to investigate. In this respect, we believe MetaMAE will serve an important baseline in this field.
Nearby supports. We further analyze the effect of i.c.. the Nearby-S ratio. We conduct the experiment with E {0, 0.1, 0.5, 1.0]. We note that r = 0 indicates the gradient updates without any help of queries (i.e., direct reconstruction of S), and r entire queries near the S. As shown in Table 6, this approach is found to be beneficial compared to the direct reconstruction, and the small ratio suggested to be proper r. e.g., 0.1 the best. This is because it effectively bridges the gap between the latent representation and the latents of masked tokens, thereby enhancing the encoding of knowledge required for the reconstruction task.
denotes the gradient updates with the
Computational efficiency. MetaMAE might be perceived as compute-inefficient when incorporating MAE due to the com- putational demands of second-order gra- dients; however, our findings suggest oth- erwise. Although MetaMAE increases the total training time of MAE by approx- imately 1.4 times (with the one-ttep adap- tation), we have observed that it is much faster to achieve the best performance of MAE: in Figure 2, we compare the accu- racy under the same training wall-clock time with MAE, e.g., 1.9 times faster
Libri (Speech)
PAMAPP series
3.24 tuster
Meto/ME H4E
50
100
23
(a) LibriSpeech
(b) PAMAP2
Figure 2: Computation efficiency comparison of MAE and MetaMAE. We report the pretraining wall clock time.
PAMAP2 dataset.
8

---

4 Related work
Self-supervised learning (SSL). SSL, i.e., learning without human supervision, recently has demon- strated substantial success across fields including, computer vision [32], natural language processing (NLP) [18], and speech recognition [2], frequently show better transferability and generalization ability compared to conventional pretraining methods, e.g., training on labeled datasets [11]. To learn such representation, SSL optimizes the loss on pretext task that does not require any human labels. For instance, pioneer works for SSL proposed such tasks based on data reconstruction through auto-encoding [7] such context prediction [19], and in-painting [67]. Later multiple SSL works found that utilizing the domain-specific inductive biases can effectively learn representations in a self- supervised manner, including, colorization [107]. solving jigsaw puzzles [64], counting the number of objects [65], and rotation prediction [27], to name a few. More recently, contrastive representation learning has garnered significant attention in SSL [100, 32, 11]. This technique maximizes the simi- larities of similar (i.e., positive) pairs and minimizes the similarities of dissimilar (i.e., negative) pairs, rather than focusing on training instance classifier. To generate such positive pairs, multiple works rely on domain-specific inductive biases such as data augmentations [47], i.e., different augmented views positive pair. In addition, recent advances have been made with the development of various architectural components: e.g., Siamese networks [44], self-distillation [32, 9], asymmetric architectures [29. 12], and utilization of Transformer architectures [9]. Despite the success of these strategies, most existing SSL frameworks rely heavily on domain-specific inductive biases, which limits their applicability to new modalities.
Modality-agnoosii SSL. Recently, several streams of work have emerged focusing on the develop- ment of more generalized SSL methods, specifically modality-agnoste SSL. For example, DACL [90] and i-Mix [48] utilize the idea of mixup [105] to propose domain-agnostic contrastive learning, and e-Mix [83] generalizes the concept of i-Mix be embedding-leve' instead of input-level. Capri [85], as a variant of CPC [88], contrasts the predicted representations from randomly masked tokens. [84] develops generative models learn data-dependent distortions for contrast. Instead of contrastive learning. ShED [83] (a generalized version of EL ELECTRA (15]) constructs the pretext task of replacing token detection with a masking strategy. DABS 2.0 [85] proposes a method to generalize MAE [33] to be modality-agnostic. In their approach, however, decoders are not utilized for discrete domains like BERT [18], while only a linear decoder is employed for continuous domains. In this paper. we suggest an effective monality-agnostic latent optimization for learning representations by interpreting masked prediction for MAE [33] in a novel manner.
Masked Auto-Encoder (MAE). MAE [33], i.e., predicting the masked parts with a given unmasked parts, has been extended to multiple applications [109, 37] across various domains [18, 39]. Among them, the recent combination of MAE with Transformer architecture [18, 33, 5] has shown promise in tackling SSL scenarios. For instance, BERT [18] utilized MAE for natural language processing (NLP) tasks, incorporating a linear layer into its architecture. Furthermore, multiple variants of MAE show impressive performance in various domains. by suggesting modality-agnose SSL [3, 4]. architecture-agnosic SSL [97, 51], multi-modal pretraining [95], and generative pretraining frameworks [21, 52]. In this paper, we focus on improving the most basic form of madk-mod (i.e., MAE) for constructing modality-agoose
SSL framework which remains under-explored, despite its potential significance, through the lens of meta-learning It is worth noting that our interpretation of viewing MAE as a meta-learning framework can be applied to any other masked masked-modeling based SSL frameworks where we believe combining our meta-learning regularization to such SSL methods would be interesting direction to explore.
Meta-learning, Meta-learning [86], i.e., learning learn by extracting common knowledge over task distribution, has emerged as a popular paradigm for enabling systems to adapt to new tasks in a sample-efficient way. Under various applications across domains (e.g., computer vision [78], natural language processing [30], and robotics [101]), there have been significant efforts to design a variety of meta-learning schemes, including gradient-based [22, 53] and amortization-bactt
approaches [75, 61] such as metric-based [92, 79], and neural processes [25, 24, 41, 87]. Typically, recent works have combined gradient- based meta-learning (or iterative functional update) with amortization- based schemes to enhance adaptation performance [74, 103]. Furthermore, there have been varieties of amortization-bast schemes (such as neural processes) that utilize the recent success of contrastive learning into meta-learning, i.e., task contrastive learning [28, 60]. In this paper, we interpret MAE as an amortization-base meta-lea,ning which is further enhanced via the benefit of model-agnosticism of gradient- based meta- letanlea and task contrastive learning.
9

---

5 Discussion and conclusion
In this paper, we tackle modality-aggnssiie self-supervised learning (SSL), important problem of SSL that consists of multiple real-world applications. To this end, we explore the possibilities of the Masked auto-Encoder (MAE) in tackling modality- SSL which is quite under-explored, despite its potential. We propose MetaMAE, novel and effective SSL framework that enhances MAE with meta-learning. Our key idea task, which allows us to treat MAE as a meta-learning framework. Based on this novel interpretation, we suggest a unique integration with advanced modallty-agnostte meta-learning methods to improve the generalization of MAE. Our experiments demonstrate that MetaMAE significantly improves the performance of modality
interpret mask reconstruction task of MAE a meta-learning
SSL approaches across diverse range of modalities.
Limitations and future work. While MetaMAE becomes a state-of-the-ar art approach for modality- agnostic SSL problems, it still inherits general limitation of the MAE, namely the modality-speciffe masking ratio, i.e., the masking ratio may differ across modalities. This is due to our shared design elements with MAE, which include masking, encoding, and decoding. Recent works propose design choices for the masking scheme [50, 94], including automation, where incorporating these ideas into MetaMAE would be an intriguing future research direction, potentially enhancing our approach be an even more effective prodality-agnosti SSL framework.
Potential
negative impacts. SSL often requires large computation and a large network capacity, therefore raising environmental concerns, c.g., carbon generation [77]. As MetaMAE is built upon the SSL method (i.e., MAE), practitioners may need to consider some computation for successful training. To address this issue, efficient training methods [80, 40), distilling knowledge to a smaller network [36], or network sparsity schemes [31, 46] would be required to ameliorate such problems.
Acknowledgements
We thank Kyungmin Lee for providing helpful feedbacks and suggestions in preparing an earlier version of the manuscript. We also thank Sang Keun Choe for technical suggestions on the Py PyTorch implementation of meta-learning. This work was supported by Institute of Information & communi- cations Technology Planning & Evaluation (IIP)) grant funded by the Korea government (MSIT) (No.2019-0-00 2000-0000075, Artificial Intelligence Graduate School Program (KAIST); No.2021 100-0202, Artificial Intelligence Innovation Hub; No.2022-0 000220000959, Few-shot Learning of Causal Inference in Vision and Language for Decision Making) and Samsung Electronics Co., Ltd
References
[1] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In JEEE International Conference on Computer Vision, 2015.
[2] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, wav2vec 2.0: A framework for superuuee learning of speech representations. In Advances in Neural Information Processing Systems, 2020.
[3] A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, 2022.
[4] A. Baevski, A. Babu, W.-N. Hsu, and M. Auli Efficient self-supervised learning with contex- tualized target representations for vision, speech and language. In International Conference on Machine Learning, 2023.
[5] H. Bao, L. D. 0004, Piao, and F. Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2022.
[6] S. Becker, M. Ackermann, S. Lapuschkin. K.-R. Müller, and W. Samek. Interpreting and explaining deep neural networks for classification of audio signals. CoRR, abs/1807.03411 2018.
[7] Bengio, P. Lamblin, D. Popovici, and H. Larochelle, Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems, 2006.
10

---

[8] H. M. Berman, J. Westbrook, Z. Feng, G. Gilliland, T. N. Bhat, H. Weissig, N. Shindyalov, and E. Bourne. The protein data bank. Nucleic acids research, 28(1):235-242, 2000.
[9] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In JEEE International Conference on Computer Vision, 2021.
[10] M. Chen, A. Radford. R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[1]] T. Chen, Kornblith, M. 0002, and G. E. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
[12] X. Chen and K. He. Exploring simple siamese representation learning. In IEEE Conference On Computer Vision and Pattern Recognition, 2021.
[13] S. K. Choe, W. Neiswanger, P. Xie, and E. P. Xing. Betty: An automatic differentiation library for multilevel optimization. In International Conference on Learning Representations, 2023.
[14] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In CVPR, pages 3606-3613, 2014.
[15] K. Clark, M.-T. Luong, Q V. Le, and C. D. Manning. Electra: Pre-training text encoders as dis- criminators rather than generators. International Conference on Learning Representations, 2020.
[16] J. M. Clements, D. Xu, N. Yousefi, and D. Efimov. Sequential deep learning for credit risk monitoring with tabular financial data. arXiv preprint arXiv:2012.15330 2020.
[17] J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hicrarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
[18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computationall Linguistics: Human Language Technologies, NAACL-IILT, 2019.
[19] C. Doersch, A. Gupta, and A. Efros. Unsupervised visual representation learning by context prediction. In IEEE International Conference on Computer Vision, 2015.
[20] S. El-Gebali, J. Mistry, A. Bateman, S. R. Eddy, A. Luciani, S. C. Potter, M. Qureshi, I.. Richardson, G. A. Salazar, A. Smart, E, L. L. Sonnhammer, I. Hirsh, L. Paladin, D. Piovesan, S. C. E. Tosatto, and R. D. Finn. The pfam protein families database in 2019. Nucleic Acids Research, 2019.
[21] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[22] C. Finn, P. Abbeel, and S. Levine. networks. In International Conference on Machine Learning, 2017.
meta-learning for fast adaptation of deep
[23] N. K. Fox, S. F. Brenner, and J.-M. Chandonia. Scope: Structural classification of proteins extended, integrating scop and astral data and classification of new structures. Nucleic Acids Research, 2014.
[24] M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Tch, D. Rezende, and S. A. Eslami. Conditional neural processes. In International Conference on Machine Learning, 2018.
[25] M. Garnelo, J. Schwarz, Rosenbaum, Viola, D. J. Rezende, Eslami, and W. Teh. Neural processes. arXiv preprint arXiv:1807.01622 2018.

---

[26] F. C. Ghesu, B. Georgescu, A. Mansoor, Y. Yoo, D. Neumann, P. Patel, R. Vishwanath, J. M. Balter, Y. Cao, S. Grbic, et al. self-supervised learning from 100 million medical images. arXiv preprint arXiv:2201.0/22
2022.
[27] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference Learning Representations, 2018.
[28] M. W. Gondal, S. Joshi, N. Rahaman, S. Bauer, M. Wuthrich, and B. Schölkopf. Function contrastive learning of transferable metaarrepresennations. In International Conference on Machine Learning, 2021.
[29] J.-B. grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Ávila Pires, Z. Guo, M. G. Azar, Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, 2020.
[30] J. Gu, Y. Wang, Y. Chen, K. Cho, and V. O. Li. Meta-learning for low-resource neural machine translation. In EMNLP, 2018.
[31] S. Han, Pool, J. Tran, and J. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, 2015.
[32] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference Computer Vision and Pattern Recognition, 2020.
[33] He, Chen, Xie, Y. Li, Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In IEEE Conference on Computer Vision and Puttern Recognition, 2022.
[34] P. Helber, B. Bischke, A. Dengel, and D. Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, 2018.
[35] P. Helber, B. Bischke, A. Dengel, and Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
[36] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. In Advances in Neural Information Processing Systems, 2014.
[37] Z. Hou, X. Liu, Y. Cen, Y. Dong, Yang, C. Wang, and J. Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 594-604, 2022.
[38] W.-N. Hsu, B. Bolte, Y.HH. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE Transactions on Audio, Speech &; Language Processing, 2021.
[39] P.-Y. Huang, H. Xu, Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer. Masked autoencoders that listen. In Advances in Neural Information Processing Systems, 2022.
[40] S. loffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference Machine Learning, 2015.
[41] H. Kim, A. Mnih J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Teh. Attentive neural processes. In International Conference on Learning Representations, 2019.
[42] Kingma and J. Ba. Adam: method for stochastic optimization. International Conference on Learning Representations, 2015.
12

---

[43] M. S. Klausen, M. C. Jespersen, H. Nielsen, K. K. Jensen, V. Jurtz, C. K. Soenderby, M. O. Sommer, O. Winther, M. Nielsen, B. Petersen, al. Netsurfp-2.00 Improved prediction of protein structural features by integrated deep learning. Proteins: Structure, Function, and Bioinformatics, 87(6):520-527, 2019.
[44] G. Koch, R. Zemel, and R. Salakhutdinov. Siamese neural networks for one-shot image recognition. In International Conference on Machine Learning, 2015.
[45] A. Krizhevsky et al. Learning multiple layers of features from tiny images, 2009.
[46] J. Lee, J. Tack, N. Lee, and J. Shin. Meta-learning sparse implicit neural representations. In Advances in Neural Information Processing Systems, 2021.
[47] K. Lee and J. Shin. Rényicl: Contrastive representation learning with skew rényi divergence. In Advances in Neural Information Processing Systems, 2022.
[48] K. Lee, Y. Zhu, K. Sohn, C.-L. Li, J. Shin, and H. Lee. i-mix: A domain-agnosien strategy for contrastive representation learning. In International Conference on Learning Representations, 2021.
[49] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and I.. Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Annual Conference of the Association for Computational Linguistics, 2019.
[50] Li, H. Zheng, Liu, C. Wang, B. Su, and C. Zheng. SemMAE: Semantic-guided masking for learning masked autoencoders. In Advances in Neural Information Processing Systems, 2022.
[51] S. Li, D. Wu, F. Wu, Z. Zang, and S. Z. Li. Architecture-agnostic masked image modeling from vit back to cnn. In International Conference Muchine Learning, 2023.
[52] T. Li, H. Chang, S. K. Mishra, H. Zhang, D. Katabi, and D. Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In IEEE Conference Computer Vision and Pattern Recognition, 2023.
[53] Z. Li, F. Zhou, F. Chen, and H. Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09883 2017.
[54] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, 2014.
[55] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.1 1692, 2019.
[56] I. Loshchilov and Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
[57] L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio. Speech model pre-training for end-to-end spoken language understanding. In G. Kubin and Z. Kacic, editors, Proc. of Interspeech, pages 814-818, 818, 2019.
[58] S. Maji, F.. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification aircraft. Technical report, 2013.
[59] K. Majmundar, S. Goyal, P. Netrapalli, and Jain. Met: Masked encoding for tabular data. arXiv preprint wxXiv:2206.08564 2022.
[60] E. Mathieu, A. Foster, and Y. W. Teh. On contrastive representations of stochastic processes.
In Advances in Neural Information Processing Systems, 2021.
[61] Mishra, M. Rohaninejad, X. Chen, and Abbeel. simple neural attentive meta-learner. In ICLR, 2018
13

---

[62] A. Nagrani. J. S. Chung, and A. Zisserman. Voxceleb: large-scale speaker identification dataset. In INTERSPEECH, 2017.
[63] M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. pages 722-729. IEEE, 2008.
[64] M. Noroozi and Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, 2016.
[65] M. Noroozi, H. Pirsiavash, and P. Favaro. Representation learning by learning to count. In IEEE International Conference on Computer Vision, 2017.
[66] V. Panayotov, Chen, D. Povey, and Khudanpur. Librispeech: an asr corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2015.
[67] Pathak, P. Krahenbuhl, Donahue, T. Darrell, and A. Efros. Context encoders: Feature learning by inpainting. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2536-2544, 2016.
[68] M. Peplow. The 100,000 genomes project. British Medical Journal, 2016.
[69] P. S. Pierre Baldi and D. Whiteson. Searching for exotic particles in high-energy physies with deep learning. Nature Communications 5, 2014.
[70] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. preprint, 2018
[71] A. Reiss and D. Stricker. Introducing a new benchmarked dataset for activity monitoring, In 16th International Symposium on Wearable Computers, ISWC, 2012.
[72] J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. A. DePristo, J. V. Dillon, and B. Lak- shminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information P Processing Systems, 2019.
[73] G. J. Rocklin, T. M. Chidyausiku, I. Goreshnik, A. Ford, S. Houliston, A. Lemak, L. Carter, Ravichandran, K. Mulligan, Chevalier, C. H. Arrowsmith, and D. Baker. Global analysis of protein folding using massively parallel design, synthesís, and testing, Science, 357 (6347):168-) jul 2017.
[74] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019.
[75] A. Santoro, Bartunov, M. Botvinick, D. Wierstra, and Lillicrap. Meta-learning with memory augory-augmented neural networks. In International Conference on Machine Learning. 2016.
[76] K. S. Sarkisyan,
Bolotin, M. V. Meer, D. R. Usmanova, S. Mishin, G. Sharonov, D. N. Ivankov, N. G. Bozhanova, M. S. Baranov, O. Soylemez, et al. Local fitness landscape
the green fluorescent protein. Nature, 53377603):397-40, 2016.
[77] R. Schwartz, Dodge, N. Smith, and Etzioni. Green ai. arXiv preprint 1907v 2019.
[78] V. Sitzmann, E. R. Chan, R. Tucker, N. Snavely, and G. Wetzstein. MetaSDF: Meta-learning signed distance functions. In Advances in Neural Information Processing Systems, 2020.
[79] J. Snell, Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
[80] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014.
14

---

[81] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The german traffic sign recognition benchmark: multi-class classification competition. The 2011 international joint conference neural networks, pages 1453-1460. IEEE, 2011.
[82] J. Tack, J. Park, H. Lee, J. Lee, and J. Shin. Meta-learning with self-improving momentum target. In Advances in Neural Information Processing Systems, 2022.
[83] A. Tamkin, V. Liu, R. Lu, D. Fein, C. Schultz, and N. D. Goodman. Dabs: domain domain-agnostic benchmark for self-supervised learning. In Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks, 2021.
[84] A. Tamkin, M. Wu, and N. D. Goodman. Viewmaker networks: Learning views for unsu- pervised representation learning. In International Conference on Learning Representations, 2021.
[85] A. Tamkin, G. Banerjee, M. Owda, V. Liu, S. Rammoorthy, and N. D. Goodman. Dabs 2.0: Improved datasets and algorithms for universal supfrsuupn In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS, 2022.
[86] S. Thrun and L. Pratt. Learning to Learn. Springer, 1998.
[87] M. K. Titsias, J. Schwarz, A. G. d. G. Matthews, R. Pascanu, and Y. W. Teh. Functional regularisation for continual learning with gaussian processes. In International Conference on Learning Representations, 2020.
[88] A. van den Oord, Y. Li, and Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807.03888 2018
[89] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[90] V. Verma, T. Luong, K. Kawaguchi, H. Pham, and Q. V. Le. Towards domain dagain-agnostic contrastive learning. In International Conference Machine Learning, 2021.
[91] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference Machine Learning, pages 1096-1103 103, 2008
[92] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. In Advances Neural Information Processing Systems, 2016.
[93] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2010-001, 2010
[94] H. Wang, K. Song, Fan, Y. Wang. J. Xie, and Z. Zhang. Hard patches mining for masked image modeling. In IEEE Conference Computer Vision and Pattern Recognition, 2023.
[95] W. Wang. H. Bao, L. Dong, Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. Mohammed, S. Singhal, S. Som, and F. Wei. Image foreign language: Beit pretraining for all vision and vasion-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
[96] P. Warden. Speech commands: A dataset for limited-vocabullat speech recognition. arXiv preprint
2018.
[97] C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. L. Yuille, and C. Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In IEEE Conference Computer Vision and Pattern Recognition, 2022.
[98] M.-. Wu, J.-.. R. Jang, and J.-L. Chen. Wafer map failure pattern recognition and similarity ranking for large-scale data sets. IEEE Transactions on Semiconductor Manufacturing, 28(1): 1-12, 2014.
15

---

[99] S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua. Next-gpt: Any-to-any any multimodal Ilm. arXiv preprint drXiv:3309.0559, 2023.
[100] Z. Wu, Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE Conference Computer Vision and Pattern Recognition, 2018.
[101] A. Xie, A. Singh, S. Levine, and C. Finn. Few-shot goal inference for visuomotor learning and planning. In Conference Robot Learning, pages 40-52. PMLR, 2018.
[102] Z. Xie, Z. 0022, Y. Cao, Y. Lin, Bao, Yao, Q. Dai, and H. H. 0004. Simmim: simple framework for masked image modeling. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[103] J. Xu, J.-F. Ton, H. Kim, A. R. Kosiorek, and Y. Teh. Metafun: Meta-learning with iterative functional updates. In International Conference on Machine Learning, 2020.
[104] Z. Ye and L. Yao. Contrastive conditional neural processes. In IEEE Conference Computer Vision and Pattern Recognition, 2022.
[105] H. Zhang, M. Cissé, N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
[106] Zhang, Y. Wang, and Y. Wang. How mask matters: Towards theoretical understandings of masked autoencoders. In NeurlPS, 2022.
[107] R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In European Conference on Computer Vision, 2016.
[108] Zhang, K. Gong. Zhang, Li, Y. Qiao, W. Ouyang, and X. Yue. Meta-transformer: unified framework for multimodal learning. arXiv preprint arXiv:2307.10802. 2023.
[109] L. Zhou, H. Liu, J. Bae, J. He, D. Samaras, and P. Prasanna. Self pre-training with masked autoencoders for medical image classification and segmentation, 2023.
16

---

A Implementation details
In this section, we provide the implementation details of MetaMAE, including architectures and hyperparameters for MetaMAE when pretraining and evaluation.
architectures in Table 7, with the hyperparameter notation referred from [89]. We use token embedding for encoder inputs, and apply positional embedding both encoder and decoder inputs, as suggested by [89]. Specifically, token embedding separates the input data into fixed-size tokens, while positional embedding uses fixed, absolute position represented by combination of sine and cosine functions. We describe the token size for each specific dataset in Appendix C.
Architectural details. We summarize
Table 7: A Pytorch-like architecture description of MetaMAE. (2,4,6), p (0,0.1) the hyperparameters.
Component
Layer descriptions
Encoder fe
(dmma 256, dr 512, Phop GELU, LayerNorm=TTuu)) (Amoda 128, dr 256 Paoop 0, GELU, LayeNNomnn Linear 1028), Batchhorri1d(1035) Linear (1028, 128)
Decoder Ge
TransformerBl (dcmmao
Projector hy
Pretraining details. We summarize our selected hyperparameters for pretraining each dataset Table 8. Following [85], we pretrain MetaMAE for 100k iterations utilizing the Adam optimizer [56] with both a learning rate and weight decay set at le-4. The batch size for pretraining and the strategy for selecting the mask ratio we observe that certain set of hyperparameters can generally work across modalities, e.g., (o, X, decoder depth) (0.5, 0.1, 4) (see Table 9 in Appendix B), can be shared within each modality, e.g.. Proop = 0 for Token modality (see Table 10 in Appendix B). Nevertheless, we recommend modality- specific values for optimal performance (refer to Appendix for hyperparameter sensitivity details). We set the temperature term for the contrastive loss т = 0.5 and the Nearby-S ratio 0.1. For latent adaptation, the latent representation undergoes a single-step update with the update magnitude denoted by a.
detailed in [83, 85]. For the MetaMAEE-peccffe hyperparameters,
Table 8: Hyperparameters of MetaMAE for pretrain datasets.
Modality
Time-series
Tabular
MS Image
Token
Speech
RGB Image
Vision-Language
Dataset
PAMAP2
HIGGS
FuroSAT
Genom
Pfam
Libri
WaferMap
ImageNer32
MSCOCO
MeioMAE-specii hyperparameters
0.5
0.1
0.1
0.1
0.1
1.0
1.0
1.0
0.01
1.0
1.0
0,1
0.1
0,1
decoder depth
4
4
Parop
0.1
0.1
0
Hyperpanameterss from DABS benchmarks
mask ratio
0.85
0.50
0.85
0.50
0.15
0.85
0.15
0.85
batch size
256
256
128
64
128
We note that to scale up experiments, it is essential to facilitate distributed parallelism by using libraries such as BETTY [13] when utilizing PyTorch for meta-learnin.
Evaluation details. In line with [85], we freeze the pretrained model and train either a linear classifier or a regressor for 100 epochs during the linear evaluation phase. We use the Adam optimizer [42] with both the learning rate and weight decay set as le-4. The batch size for this linear evaluation is set as described in [83, 85].
17

---

B Analysis on hyperparameter sensitivity
We here provide additional experiments on hyperparametters. This includes sharing various hyper- parameters across modalities and conducting ablation studies with varying hyperparameters: a, A, decoder depth, Parop. and the latent adaptation step size.
Sharing hyperparameters across modalities. As demonstrated in Table 9, MetaMAE shows robust performance regardless of the hyperparater selection. Notably, or three major hyperparameters can be shared across all modalities, still outperforming prior methods. Furthermore, Table 10 indicates the resilience of MetaMAE's pretraining hyperparameters, especially the intra-modality level.
Table 9: Linear evaluation performance (%) across modalities. Sharing 2 and HPs denotes MetaMAE with additionally sharing more hyperparameters among the non-shared hyperparameters in Table 8 in Appendix A, which are (a, A) (0.5, 0.1) and (a, X, decoder depth) (0.5, 0.1, respectively. HP denotes hyperparameter.
Modality
Time-series
Tabular
MS Image
Token
Speech
RGB Image
Dataset
PAMAP2
HIGGS
EuroSAT
Genom
Pfam
Libri
WaferMap
e-Mix
80.1
65.7
87.4
40.5
31.3
60.2
92.6
ShED
85.2
68.0
61.5
33.6
54.7
34.8
92.4
Capri
67.4
23.5
27.4
25.4
92.5
MAE
85.3
70.0
86.3
53.6
44.7
46.0
93.9
MetaMAE (sharing HPs)
89.1
71.0
88.5
55.4
62.2
77.1
95.4
MetaMAE (sharing HPs)
89.1
71.1
88.5
66.7
62.2
77.1
95,4
MetaMAF (reported)
89.3
71.5
88.5
69.4
62.3
79.8
95.5
Table 10: Linear evaluation performance (%) with sharing all hyperparameters, except mask ra- tio, intra-modality level. For sharing HPs in intra-modal, use (a, A, decoder depth, Patop) (0.1, 0.01, 2,011,2,7 2,0) 0) and (0.1, 0.1,4, 0) for Token and RGB Image modalities, respectively. HP denotes hyperparameter.
Modality
Token
RGB Image
Dataset
Genom
Pfam
WaferMap
ImageNet32 - CIFARIO
MetaMAE (sharing HPs in intra-modality)
69.4
61.5
95.5
59.2 59.2
MetaMAE (reported)
69.4
62.3
95.5
Further ablation studies with varying hyperparameters. Table 11, 12, 13, and 14 show the the PAMAP2 and WaferMap datasets. We observe that MetaMAE performs well even with non-optimal hyperparameters, except for the decoder depth and Parop. but we suggest finding better hyperparameters specific each domain (e.g.. Regarding the decoder depth, we find that each modality requires an appropriate value, but generally, MetaMAE performs well with decoder depth of In Table 15, we observe that single-step adaptation effectively achieves good performance, and in some cases, even outperforms multiple-step adaptation due the risk of overly decoder-speciffc support representation.
sensitivity of hyperparameters
0.1 for WaferMap).
Table 11: Sensitivity of on Table 12: PAMAP2 and WaferMap.
Sensitivity of À Table 13: Sensitivity of decoder
PAMAP2 and WaferMap.
depth on PAMAP2 and WaferMap.
a
PAMAP2
WaferMap
PAMAP2
WaferMap
depth
PAMAP2
WaferMap
0.1
86.2
95.5
0.01
88.6
95.2
84,9
94.2
0.5
89.3
95.4
0.1
89.1
95.5
89.3
95.5
1.0
89.1
95.2
1.0
89.3
93.6
6
86.2
95.5
Table 14: Sensitivity of Parop on PAMAP2 and WaferMap.
Table 15: Sensitivity of latent adaptation
step size on PAMAP2 and WaferMap.
Paop
PAMAP2
WaferMap
step size
PAMAP2
WaferMap
0
79.4
95.5
89.3
95.5
0.1
89.3
94.7
5
89.6
94.9
18

---

C Dataset details
We provide summary the considered datasets from the DABS benchmarks [83, 85) in Table 16. Note that we use the dataset split described in [83, 85].
Table 16: Datasets considered for pretraining and linear evaluation in our experiments. "MS Image" denotes the Multi-spectral image modality. For Phase, "P" denotes pretraining and "F" denotes fine-tuning.
Modality
Dataset
classes
Input shape
Token shape
Batch size
Time-serie
PAMAP2 [71]
52 320
5
P
256 & 256
Tahular
HIGGS [69]
2
28
P&
256 & 256
MS Image
EuroSAT [35, 34]
13 x 64 x
8 x 8
P&
64 & 64
Genomics [72]
Genomics-OOD [72)
60
250
1
32
Pfam [20)
26 x 128
128 & 128
Token
SCOP [23]
1195
26 128
128
Secondary Structure [43, 8]
26 x 128
128
Stability [73]
26 125
128
Fluorescence [76)
26 x 128
128
LibriSpeech [66]
x 224 224
16
P&F
&
Audio MNIST [6]
224
64
Fluent Locations [57]
1 x 221
221
16 16
64
Speech
Fluent
Fluent Objects [57]
14
x 224 224
16 16
64
Google Speech [96]
VoxCeleb1 [62]
1251
x 221 221
16 16
64
waferMap [98]
3 x 32 32
4 4
P
128 & 128
ImageNet-32
1000
32
64
CIFAR-10 (45)
10
3 x 32 32
4
64
[93]
RGB Image
VGG Flowers [63]
102
x 32 32
4 1XX
64
DTD [14]
32
Traffic Sign (81)
43
32 32
4 1XL
64
AirCraft (58)
102
32
X.X.
64
MSCOCO [54]
224
224, 30552 32)
(16 x
Vision-Language
VQA [1]
(3 x
221
221, 30652 32)
(16 x 16,
64
Mismstched catpttcchedccptionnnnc (54)
x 224
224, 30552
32)
(16 x 16
64
19


Reviews:
[{"review_id": "JqE6Z2HiLp", "reviewer": "Reviewer_3Z8m", "summary": "This paper proposes MetaMAE, a novel framework to improve Masked Autoencoder (MAE) for modality-agnostic self-supervised learning. Unlike many SSL methods reliant on domain-specific inductive biases, MetaMAE enhances MAE with two meta-learning techniques: latent adaptation via gradient-based meta-learning to optimize amortized features and task contrastive learning to align amortized and optimized features. Conducting extensive experiments with modality-agnostic SSL benchmark DABS 1.0 and 2.0, MetaMAE consistently improves across diverse modalities like time series, tabular, images, and speech.", "questions": "Sorry for the late review! No more questions besides the concerns mentioned in Weaknesses.", "limitations": "Limitations have been discussed by the authors and I found no more limitations.\n\n### Post-rebuttal\nThe authors' replies have tackled my concerns, thus, I raised the rating to 6. I hope the authors add more discussion with existing works and specify the limitations and future research directions for this topic.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* **(S1)** This paper is well-motivated to enhance MAE with meta-learning strategies and design modality-agnostic pre-training methods. Extensive experiments on DABS and DABS 2.0 show the notable performance gains of the proposed MetaMAE, which highlights the potential of this research direction. Experimental details and the source code are provided for reproduction.\n\n* **(S2)** The overall presentation is smooth and easy to follow. The background and related works of the manuscript are well-studied.", "weaknesses": "* **(W1)** The designed method is straightforward and simple, while the intuition or analysis of why meta-learning strategies improve MAE for modality-agnostic pre-training is not well studied. Since the vanilla masked predictive learning frameworks like MAE and Data2Vec [1] are already modality-agnostic, the author should provide some empirical analysis to support the statements in Sec. 3.2. Or it will decrease the novelty of MetaMAE.\n\n* **(W2)** Some hyper-parameters need tuning for each modality, e.g., the decoder sizes and pre-training settings. As provided in the appendix, some sensitive hyper-parameters might have different behavior in different modalities.\n\n* **(W3)** Although the authors study a wide arrange of related papers, some recently proposed general pre-training frameworks should be included. For example, Masked Auto-Encoder methods like Data2Vec variants [1, 2] (modality-agnostic) and SimMIM variants [3, 4] (architecture-agnostic); Multi-modality pre-training like BEiT variants [5]; Generative Pre-training like VQ-based variants [6, 7] (can be applied to various modalities with some modifications). It is better to discuss the similarity between MetaMAE and these methods.\n\n* **(W4)** Typos should be double-checked before publishing, e.g., `pre-pre-training` in line 113.\n\n#### Reference\n[1] Alexei Baevski, et al. “data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language.” ICML, 2022.\n\n[2] Alexei Baevski, et al. “Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language.” ICML, 2023.\n\n[3] Chen Wei, et al. “Masked Feature Prediction for Self-Supervised Visual Pre-Training.” CVPR, 2022.\n\n[4] Siyuan Li, et al. “Architecture-Agnostic Masked Image Modeling - From ViT back to CNN.” ICML, 2023.\n\n[5] Wenhui Wang, et al. “Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks.” arXiv, 2023.\n\n[6] Patrick Esser, et al. “Taming Transformers for High-Resolution Image Synthesis.” CVPR, 2021.\n\n[7] Tianhong Li, et al. “MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis.” CVPR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3Z8m,\n\nWe sincerely appreciate your efforts and insightful comments to improve the manuscript. We address your comment below.\n\n---\n\n**[Q1] Why meta-learning improves MAE for modality-agnostic SSL?**\\\n**[A1]** We remind that the masked modeling of MAE can be interpreted as an *amortization-based meta-learning* framework, where we improve the performance by incorporating advanced meta-learning schemes designed for enhancing amortization-based meta-learning. Typically, it is well-known that using i) gradient-based meta-learning [1] and 2) task contrastive learning [2] *on the amortized latent* enables the encoder to learn more complex and diverse task distribution. This is equivalent to learning complex and diverse data distribution from the MAE perspective, as individual data is interpreted as a meta-learning task (see Line 165). In this regard, we believe further using more advanced meta-learning for amortization-based meta-learning, e.g., MetaFun [3], will be an interesting and promising direction to explore. \n\nWhile we genuinely wished to provide supporting experimental results to validate these claims for addressing the reviewer's request, please understand that we only had **10 hours** (before the rebuttal deadline) to prepare our response (hence, implementing and running these experiments within the time limit was hard). Nonetheless, in the final manuscript, we will incorporate the results of using advanced meta-learning schemes, such as MetaFun, for MAE to further validate our claim.\n\n[1] Rusu et al., Meta-Learning with Latent Embedding Optimization, ICLR 2019\\\n[2] Mathieu et al., On Contrastive Representations of Stochastic Processes, NeurIPS 2021\\\n[3] Xu et al., MetaFun: Meta-Learning with Iterative Functional Updates, ICML 2020\n\n---\n\n**[Q2] Some hyper-parameters differ across modalities.**\\\n**[A2]** First, we would like to note that it is quite a common consensus that hyper-parameters differ across modalities, e.g., BERT (for text) uses a mask ratio of 0.15, and MAE (for image) uses 0.75. Nevertheless, we believe the performance of MetaMAE is robust to the hyper-parameter selection; we found even after sharing most hyper-parameters across *all modalities*, MetaMAE consistently outperforms previous state-of-the-art results across all modalities (see Table 3 in Appendix A). Furthermore, we run *additional* hyper-parameter ablation studies (see Table 2 in the attached pdf), which further support that the choice of pre-training hyper-parameters of MetaMAE is particularly robust at the intra-modality level. For example, for a Token modality, shared hyper-parameters $(\\alpha, \\lambda, \\text{decoder depth}, P_\\text{drop}) = (0.1, 0.01, 2, 0)$ can achieve 69.4\\% on Genom and 61.5\\% on Pfam, which are very close (or identical) to the reported performance of MetaMAE (69.4\\%, and 62.3\\%, respectively) in our original submission. These results could be useful observations in practice when pretraining a certain dataset of the same modality we experimented with.\n\nFinally, we also agree with the reviewer that developing a unified way of selecting hyper-parameters across modalities is an important and interesting direction. In this perspective, we believe MetaMAE can contribute as a strong baseline for future research in this direction, showing a robust performance regardless of the hyper-parameter selection (see Table 3 in the attached pdf); two major hyper-parameters ($\\alpha, \\lambda$) can be additionally shared across all modalities while it shows competitive performance with the MetaMAE in our original submission. We will include all results in the final manuscript.\n\n---\n\n**[Q3] More discussion on the recent related works regarding general pre-training.**\\\n**[A3]** Thank you for your suggestion. In the revised manuscript, we will include a comprehensive discussion of other pre-training methods that use masked modeling. Note that our key novelty of MetaMAE is to interpret the masked modeling (of MAE) as an amortization-based meta-learning framework, where such an interpretation is also feasible for other masked modeling schemes as well. While we have primarily chosen MAE due to its effectiveness across modalities and simplicity, we believe extending our interpretation for other masked modeling methods would be an interesting direction to explore by combining (or developing) advanced meta-learning frameworks.\n\n---\n\n**[Q4] Typos?: “pre-pre-training”**\\\n**[A4]** As for the word you mentioned, “pre-pre-training”, we note that this is not a typo but refers to a specific method proposed in [1]. Nevertheless, we will go through another proofreading to ensure that there are no further typos in the final draft. \n\n[1] Singh et al., The effectiveness of MAE pre-pretraining for billion-scale pretraining, Arxiv 2023\n"}, {"review_id": "BlauVltq3w", "reviewer": "Reviewer_GPRA", "summary": "The paper titled \"Modality-Agnostic Self-Supervised Learning by Meta-Learning\" presents a novel framework called MetaMAE (Meta-learned Masked Auto-Encoder) for modality-agnostic self-supervised learning (SSL). The authors propose an innovative interpretation of Masked Auto-Encoder (MAE) as a meta-learning framework and enhance its generalization capabilities using advanced meta-learning techniques.", "questions": "I guess it is more of a vague questions, but in your experience, what might help handling diverse and complex data with this approach? What is keeping MetaMAE to become the one approach to rule them all?", "limitations": "Authors have done a good job addressing the concerns and limitations on the paper.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper includes extensive evaluations on multiple data modalities, demonstrating the effectiveness of MetaMAE. The results show significant improvements over previous modality-agnostic SSL methods in linear evaluation.", "weaknesses": "I do not see any critical weakness for this paper. Authors have done pretty good job working out the details and limitations. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GPRA,\n\nWe sincerely thank you for your positive feedback on our paper! We address your question below. If you have any other questions or suggestions, please let us know!\n\nBest, \\\nAuthors\n\n---\n\n**[Q1] What might help in handling diverse and complex data with this approach? What is keeping MetaMAE to become the one approach to rule them all?** \\\n**[A1]** We believe that our interpretation of MAE as a meta-learning framework will be helpful in learning more diverse and complex datasets. Since meta-learning is well-known for learning diverse and complex distributions of tasks to enhance the model’s generalization [1], we believe this interpretation would be helpful for learning complex data distributions (note that our key interpretation is to rethink data reconstruction as a task). Moreover, as recent studies highlight the potential of improving MAE for modality-agnostic SSL [2,3], we believe combining MAE with even more advanced meta-learning schemes (or developing a new meta-learning method) would be a promising and interesting future direction to explore.\n\n[1] Finn et al., Model-agnostic Meta-learning for Fast Adaptation of Deep Networks, ICML 2017 \\\n[2] Majmundar et al., MET: Masked Encoding for Tabular Data, NeurIPS 2022 Workshop TRL \\\n[3] Xu et al., Masked Autoencoders that Listen, NeurIPS 2022\n"}, {"review_id": "I6bJOFGzQx", "reviewer": "Reviewer_2wfj", "summary": "This work interprets MAE as a meta-learning framework to formulate a modality-agnostic SSL framework. Specifically, they formulate the masked data and unmasked data (nearby tokens) as the support set and the query set, respectively. Moreover, the authors proposed to use gradient-based meta-learning on a decoder and task contrastive learning to optimize the encoder. Extensive experiments with ablation studies show the effectiveness of the proposed framework.", "questions": "Please address my concerns in the paper Weaknesses.", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The idea of leveraging meta-learning and MAE to address modality-agnostic SSL is novel and interesting.\n+ The design of using gradient-based meta-learning on a decoder and task contrastive learning seems to be effective.\n+ The experiments on various modalities and the detailed user study are convincing.", "weaknesses": "- It is necessary to further clarify the relation between the proposed method (i.e., MAE+meta-learning) and the task (i.e., modality-agnostic SSL), that is the motivation for formulating MAE as a meta-learning paradigm. For example, what's the advantage of  the meta-learning paradigm for modality-agnostic SSL? In the paper, the authors only explain that their formulation can \"improve the generalization\", which is not specific.\n- It would be better to discuss the potential advantages of the proposed method to solve multimodal SSL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2wfj,\n\nWe sincerely thank you for your helpful feedback and insightful comments to improve the manuscript. We address your comments and questions below.\n\n---\n\n**[Q1] Relationship between the proposed method (i.e., MAE+meta-learning) and modality-agnostic SSL.** \\\n**[A1]** We would like to clarify that the essential challenge of modality-agnostic SSL is to develop a pretext task that requires limited inductive biases (i.e., modality-specific knowledge), due to the difference of each modality. To tackle this challenge, in this work, we introduce a novel combination of two modality-agnostic approaches, namely MAE and meta-learning.\n\nSpecifically, by rethinking MAE’s task as a meta-learning framework, we can enhance MAE’s performance toward a modality-agnostic SSL framework. Remark that recent studies [1,2] have demonstrated the potential of MAE to evolve into a modality-agnostic SSL, indicating that enhancing MAE in a modality-agnostic manner is a promising direction for advancing modality-agnostic SSL. Furthermore, meta-learning, which is typically modality-agnostic, offers a promising direction for further improvement in a modality-agnostic manner. \n\n[1] Majmundar et al., MET: Masked Encoding for Tabular Data, NeurIPS 2022 Workshop TRL \\\n[2] Xu et al., Masked Autoencoders that Listen, NeurIPS 2022\n\n---\n\n**[Q2] Potential advantages of the proposed method to solve multimodal SSL.** \\\n**[A2]** Thank you for the constructive suggestion. We indeed expect that our proposed MetaMAE can be further extended to handle multimodal SSL scenarios, e.g., by adapting recent works [1,2] on multimodal MAE. Table 1 in the attached pdf verifies the potential of MetaMAE toward multimodal as you suggested. We found that pretraining MetaMAE on MSCOCO (vision-language) dataset from the DABS benchmark outperforms other modality-agnostic SSL approaches for both the two vision-language downstream tasks (e.g., 57.6\\% -> 69.7\\% on VQA task). We thank the comment and will also discuss the potential of MetaMAE for multimodal datasets by including the result in the final manuscript.\n\n[1] Geng et al., Multimodal Masked Autoencoders Learn Transferable Representations, Arxiv 2022 \\\n[2] Bachmann et al., MultiMAE: Multi-modal Multi-task Masked Autoencoders, ECCV 2022\n"}, {"review_id": "edgMsmjijY", "reviewer": "Reviewer_b3vS", "summary": "This paper studies self-supervised learning (SSL) for multiple modalities (e.g. tabular data, images, text, speech). Central to this paper are Masked Auto-Encoders (MAEs), a popular SSL training paradigm where models are tasked to reconstruct inputs from randomly masked subsets. The authors interpret this reconstruction task as a meta-learning task, and based on this interpretation, propose a new method to improve the performance of MAE, called MetaMAE. The proposed method provides solid improvements over previous work on several datasets and modalities from the DABS benchmark.", "questions": "No questions.", "limitations": "To the best of my knowledge the authors have properly discussed the limitations of their work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The results presented in the paper are quite strong, surpassing previous results often by a large margin. As such, I believe this paper presents a solid advance over existing literature, and would be of interest to most researchers working on SSL.\n- The paper is clear and well written.\n- The experiments are solid and extensive, and there are many ablation studies which allow readers to gauge the importance of different design decisions and components of the method presented in this paper.", "weaknesses": "- It would be interesting to see if the proposed method can be used train a single model jointly on all the modalities and benchmarks, instead of training one separate model for each. One of the limitations discussed by the authors is that the masking ratio can differ across modalities, which might pose a problem. However, building a general model that could simultaneously perform all the tasks could further demonstrate the generality of the proposed method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer b3vS,\n\nWe sincerely thank you for your positive feedback and insightful comment to improve the manuscript. We address your comment below.\n\n---\n\n**[Q1] Training a single model jointly on all the modalities and benchmarks to build a general model that could simultaneously perform all the tasks.** \\\n**[A1]** Thank you for your insightful comment. We agree that it is a very interesting and important future direction for the modality-agnostic SSL research community to bind all modalities with a single model. Here, we believe MetaMAE can be quite a promising method to tackle this problem, e.g., by performing multiple modalities on a single model with domain-specific embedding modules as introduced in a recent work [1]. Furthermore, it is worth noting that MetaMAE outperforms other modality-agnostic SSL methods in paired multimodal datasets (see Table 1 in the attached pdf for MSCOCO results) where we believe this multimodal learning ability can help when unifying the modalities for SSL.\n\n[1] Zhang et al., Meta-Transformer: A Unified Framework for Multimodal Learning, Arxiv 2023\n"}, {"review_id": "xtvigThdzq", "reviewer": "Reviewer_9KfC", "summary": "The authors of this paper address modality-agnostic self-supervised learning (SSL), a significant yet under-researched problem with various real-world applications. They explore the potential of the Masked Auto-Encoder (MAE) in this context and propose SSL framework, MetaMAE, which enhances MAE with meta-learning. They reinterpret the mask reconstruction task of MAE as a meta-learning task, which in turn allows them to apply modality-agnostic meta-learning methods to improve MAE's generalization. Experimental results reveal that MetaMAE improve the performance of modality-agnostic SSL approaches across multiple modalities.", "questions": "In ablation study section, Tab.3, why not use only task contrast without gradient-based component to see how much of the gain comes from contrastive learning itself?\n\nAlso, how to balance the meta-learning gradient update and contrastive learning update? Would they influence each other? Is the hyper-parameter lambda grid searched?", "limitations": "Masking scheme limitation on MAE is discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper views Masked Auto-Encoder (MAE) from a meta-learning perspective, which is interesting.  \nThe paper is well-structured, allowing for easy comprehension of the information flow.   \nThe explanation on how support and query sets corresponds to tokens is clear.  \nThe experiment section is solid. The paper provides a comprehensive evaluation of the proposed method through thorough experiments and comparisons with existing models. \nThe approach shows great generalizability on cross-domain linear evaluation task.", "weaknesses": "The experiment setup for meta-learning based gradient update could benefit from more explanation. Like how is the support and query set constructed during training for gradient updates.\n\nMeta-learning usually requires more compute due to the computation of second order gradients, and discussion on computational overhead could be necessary.\n\nDespite the variety of tasks for evaluation, most of the datasets used for evaluation is of small-scale.  \n\nOne important ablation study missing to validate the effectiveness of the proposed overall framework. (See question below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9KfC,\n\nWe sincerely thank you for your helpful feedback and insightful comments. We address your comments and questions below.\n\n---\n\n**[Q1] Explanation of the experimental setup for meta-learning-based gradient update, e.g., how the support and query set is constructed during training.** \\\n**[A1]** Thank you for the suggestion to enhance our manuscript. During pretraining, we randomly split the tokenized data into two disjoint sets, denoted as the support and query sets. Note that support and query sets can be interpreted as unmasked and masked tokens, respectively, in the MAE context. For linear evaluation, we consider only the support set to obtain representations for both train and test samples, following the standard self-supervised learning setup. We will include the explanations for the experimental setup in the final manuscript.\n\n---\n\n**[Q2] Discussion of computational overhead due to the computation of second-order gradients.** \\\n**[A2]** Thank you for your insightful comment. It is true that second-order gradients require more computation, but we found that MetaMAE can be somewhat efficiently implemented by using one-step adaptation. Also, using a small decoder size compared to the encoder size is the key to MAE, thereby the gradient computation is quite efficient.\n\nFurthermore, we found that while MetaMAE increases the total training time of MAE by roughly 1.4 times, we have observed that it is much faster to achieve the best performance of MAE: in Figure 1 in the attached pdf, we compare the accuracy under the same training wall-clock time with MAE; for example 1.7 times faster on LibriSpeech. Thank you for your comment and will incorporate this result in the final manuscript.\n\n---\n\n**[Q3] Most of the datasets used for evaluation are small-scale.** \\\n**[A3]** We follow the evaluation protocol of the standard modality-agnostic SSL benchmark (i.e., DABS), which also consists of considerably large-scale datasets (e.g., LibriSpeech, a large-scale audio dataset, which is used for pretraining and evaluating well-known audio SSLs [1,2]), thereby showing the scalability of MetaMAE. Nonetheless, we agree that the size of some datasets in the DABS can be relatively small than other SSL benchmarks (as they are collected from practical applications), and evaluating more large-scale experiments can further demonstrate the scalability of MetaMAE. To this end, we refer our experimental results on MSCOCO (consisting of 328K image caption pairs), which was done to address the discussion on the potential advantages of MetaMAE on multimodal SSL. We found that MetaMAE outperforms other modality-agnostic SSL approaches for both the two vision-language downstream tasks (see Table 1 in the attached pdf; e.g., 57.6\\% -> 69.7\\% on the VQA task). In the final manuscript, we will include the above result and highlight the promise of MetaMAE for large-scale SSL training.\n\n[1] Baevski et al., wav2vec 2.0: A Framework for Self-supervised Learning of Speech Representations, NeurIPS 2020 \\\n[2] Hsu et al., HuBERT: Self-supervised Speech Representation learning by Masked Prediction of Hidden Units, ACM 2021\n\n---\n\n**[Q4] Missing ablation study: Why not use only task contrast without a gradient-based component?** \\\n**[A4]** We would like to clarify that the ablation study you suggested is not possible as the gradient-based component is essential for task contrast. To perform task contrastive learning, positive pairs must be defined, and in our framework, they were defined as pairs of amortized and adapted latents. Remark that the adapted latents are constructed from the gradient-based component, which means that we can exploit task contrastive learning thanks to the gradient-based component.\n\n---\n\n**[Q5] How to balance the meta-learning gradient update and contrastive learning update? Would they influence each other? Is the hyper-parameter lambda grid searched?** \\\n**[A5]** Thank you for the question. We employ a log-scaled grid search for the hyper-parameter $\\lambda$ to regulate only the contrastive loss. However, the two loss terms can be influenced by each other as larger $\\lambda$ can emphasize more similar representations for positive pairs. This makes the model learn toward smaller gradient differences between the amortized and adapted latents. We would like to note that $\\lambda = 0.1$ consistently outperforms the previous results. This indicates that $\\lambda$ is not sensitive to the specific modalities, yet finding specific values for each modality is helpful for achieving optimal performance."}, {"review_id": "GsR7EEz1v5", "reviewer": "Reviewer_Pe9M", "summary": "This paper presents Meta-learned Masked Auto-Encoder (MetaMAE), a novel modality-agnostic self-supervised learning (SSL) framework that leverages meta-learning to improve the transfer abilities of Masked Auto-Encoder (MAE). The authors reinterpret the mask reconstruction task of MAE as a meta-learning task and propose the integration of two advanced meta-learning techniques: gradient-based meta-learning and task contrastive learning. MetaMAE is evaluated on various data modalities from modality-agnostic SSL benchmarks, demonstrating significant improvements over previous modality-agnostic SSL methods in linear evaluation. The proposed approach also shows improved transferability on cross-domain datasets.", "questions": "N/A", "limitations": "addressed ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an innovative approach to modality-agnostic SSL by leveraging meta-learning and reinterpreting mask reconstruction in MAE as a meta-learning task.\n\n2. This paper is well written.\n\n3. MetaMAE consistently outperforms previous modality-agnostic SSL methods in linear evaluation, indicating its effectiveness.\n\n4. Contrastive learning in MetaMAE can improve performance.", "weaknesses": "1. There are significant differences in pretraining and fine-tuning hyperparameters for various downstream tasks, such as masking ratio, batch size, and decoder depth, as shown in Appendix Table 2.\n\n2. The paper lacks innovation.\n\n3. In the ablation experiments, it can be observed that the proposed Gradient-based and Task contrast methods have minimal impact on most tasks.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Pe9M,\n\nWe sincerely appreciate your efforts and insightful comments to improve the manuscript. We address your comment below.\n\n---\n\n**[Q1] Significant difference in hyper-parameters for various downstream tasks, e.g., masking ratio, batch size, and decoder depth.** \\\n**[A1]** For your concern, we first clarify that except for the additional hyper-parameters introduced by MetaMAE, we follow the same hyper-parameter selection strategy from the prior work, DABS [1] (i.e., the paper introducing the modality-agnostic SSL benchmark). Specifically, we use the same hyper-parameters for the linear evaluation (i.e., fine-tuning; see Lines 17-21 in Appendix) and also use the same hyper-parameter selection for pre-training (e.g., batch size, optimization hyper-parameters, and choose the best-performing mask ratio from candidates in [1] by following [1]).\n\nRegarding additional hyper-parameters of MetaMAE, it is worth noting that it is a common consensus that the hyper-parameter differs across modalities, e.g., BERT (for text) uses a mask ratio of 0.15, and MAE (for image) uses 0.75. Nevertheless, as reported in Table 3 of Appendix A, even sharing most hyper-parameters across *all modalities*, MetaMAE consistently outperforms previous state-of-the-art results across all modalities. Furthermore, we run *additional* hyper-parameter ablation studies (see Table 2 in the attached pdf), which further support that the choice of pre-training hyper-parameters of MetaMAE is particularly robust at the intra-modality level. For example, for a Token modality, shared hyper-parameters $(\\alpha, \\lambda, \\text{decoder depth}, P_\\text{drop}) = (0.1, 0.01, 2, 0)$ can achieve 69.4\\% on Genom and 61.5\\% on Pfam, which are very close (or identical) to the reported performance of MetaMAE (69.4\\%, and 62.3\\%, respectively) in our original submission. These results could be useful observations in practice when pretraining a certain dataset of the same modality we experimented with.\n\nFinally, we also agree with the reviewer that developing a unified way of selecting hyper-parameters across modalities is an important and interesting direction. In this perspective, we believe MetaMAE can contribute as a strong baseline for future research in this direction, showing a robust performance regardless of the hyper-parameter selection (see Table 3 in the attached pdf); two major hyper-parameters ($\\alpha, \\lambda$) can be additionally shared across all modalities while it shows competitive performance with the MetaMAE in our original submission. We will include all results in the final manuscript.\n\n---\n\n**[Q2] The paper lacks innovation.** \\\n**[A2]** We politely yet strongly disagree with this claim. We believe our approach is novel and innovative as highlighted by Reviewers GPRA, 2wfj, and Pe9M, where the key novelty is to interpret MAE as a meta-learning framework. This novel interpretation enables us to incorporate advanced meta-learning schemes which lead to an enhanced performance.\n\nFurthermore, we believe the experimental results also reflect our innovation. Typically, as SSL is becoming more important in multiple real-world industrial domains [1], the superiority of MetaMAE will be highly impactful for various applications across modalities. Tackling modality-agnostic SSL will be even more crucial as new types of modality datasets (or industrial applications) increase, where we believe MetaMAE will serve as an important future baseline in this field.\n\n[1] Tamkin et al., DABS: A Domain-agnostic Benchmark for Self-supervised Learning, NeurIPS 2021 Track on Datasets and Benchmarks\n\n---\n\n**[Q3] In the ablation experiments, Gradient-based and Task contrast methods have minimal impact on most tasks.** \\\n**[A3]** We politely disagree with your claim. Note the cases with maximal or significant improvements in Table 4 in the attached pdf. For instance, gradient adaptation increases by 1.7\\% on PAMAP2, and task contrastive learning improves by 15.3\\% on Librispeech. Moreover, to further address your concern, we conduct an additional ablation study on Genomics (See the same table above), where gradient adaptation improves by 4.2\\%. This result shows that our components contribute considerably large performance gains in some cases.\n\nFurthermore, we would like to note that the gradient/task contrastive consistently improves the performance across multiple modalities as we reported ablation experiments on a lot of modalities in Table 3 of the main paper. We remark that consistent improvement is an important aspect when targeting multi-modal domains (e.g., Table 1 in the attached pdf for your information), as it suggests that the approach generalizes well across different domains and is not tailored to optimize only specific ones."}]

Author rebuttal:
Dear reviewers and AC,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.
As reviewers highlighted, we propose a novel (GRPA, 2wfj, Pe9M, 3Z8m) and interesting (2wfj, b3vS, 9KfC) approach with strong empirical results (ALL Reviewers) and extensive experiments (GPRA, 2wfj, b3vS, 9KfC, 8Z8m). Our paper is also well-structured (b3vS, 9KfC, Pe9M, 3Z8m).

We appreciate your constructive comments on our manuscript. In the attached pdf, we have run the following additional experiments to clarify the reviewers’ comments:

- Computational efficiency of pretraining MetaMAE (Figure 1; To answer the Q2 of Reviewer 9KfC)
- Large-scale multimodal experiments (Table 1; To answer the Q2 of Reviewer 2wfj, Q1 of Reviewer b3vS, and Q3 of Reviewer 9KfC)
- Shared hyperparameters across intra or all modalities (Table 2 and 3; To answer the Q1 of Reviewer Pe9M, and Q2 of Reviewer 3Z8m)
- More datasets on ablation studies (Table 4; To answer the Q3 of Reviewer Pe9M)
 
We strongly believe that MetaMAE can be a useful addition to the NeurIPS community, in particular, due to the enhanced manuscript by reviewers’ comments helping us better deliver the effectiveness of our method.

Thank you very much! \
Authors.


Meta-review: This paper introduces a novel idea supported by strong empirical results. The reviewers appreciate the valuable contribution of this study.  In preparation for the camera-ready version, we kindly urge the authors to incorporate the feedback provided by the reviewers, which will undoubtedly further refine the quality and impact of the work.
Decision: Accept (poster)