SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation details. In line with [85], we freeze the pretrained model and train either a linear classifier or a regressor for 100 epochs during the linear evaluation phase. We use the Adam optimizer [42] with both the learning rate and weight decay set as 1e-4. The batch size for this linear evaluation is set as described in [83, 85].
B Analysis on hyperparameter sensitivity
We here provide additional experiments on hyperparameters. This includes sharing various hyperparameters across modalities and conducting ablation studies with varying hyperparameters:  $\alpha$ ,  $\lambda$ , decoder depth,  $P_{\mathrm{drop}}$ , and the latent adaptation step size.
Sharing hyperparameters across modalities. As demonstrated in Table 9, MetaMAE shows robust performance regardless of the hyperparameter selection. Notably, two or three major hyperparameters can be shared across all modalities, still outperforming prior methods. Furthermore, Table 10 indicates the resilience of MetaMAE's pretraining hyperparameters, especially at the intra-modality level.
Table 9: Linear evaluation performance (\%) across modalities. Sharing 2 and 3 HPs denotes MetaMAE with additionally sharing more hyperparameters among the non-shared hyperparameters in Table 8 in Appendix A, which are  $(\alpha, \lambda) = (0.5, 0.1)$  and  $(\alpha, \lambda, \text{decoder depth}) = (0.5, 0.1, 4)$ , respectively. HP denotes hyperparameter.
Modality	Time-series	Tabular	MS Image	Token	Speech	RGB Image
Dataset	PAMAP2	HIGGS	EuroSAT	Genom	Pfam	Libri	WaferMap
e-Mix	80.1	65.7	87.4	40.5	31.3	60.2	92.6
ShED	85.2	68.0	61.5	33.6	54.7	34.8	92.4
Capri	-	-	67.4	23.5	27.4	25.4	92.5
MAE	85.3	70.0	86.3	53.6	44.7	46.0	93.9
MetaMAE (sharing 3 HPs)	89.1	71.0	88.5	55.4	62.2	77.1	95.4
MetaMAE (sharing 2 HPs)	89.1	71.1	88.5	66.7	62.2	77.1	95.4
MetaMAE (reported)	89.3	71.5	88.5	69.4	62.3	79.8	95.5
Table 10: Linear evaluation performance (\%) with sharing all hyperparameters, except mask ratio, intra-modality level. For sharing HPs in intra-modal, we use  $(\alpha, \lambda, \text{decoder depth}, P_{\text{drop}}) = (0.1, 0.01, 2, 0)$  and  $(0.1, 0.1, 4, 0)$  for Token and RGB Image modalities, respectively. HP denotes hyperparameter.
Modality	Token	RGB Image
Dataset	Genom	Pfam	WaferMap	ImageNet32 → CIFAR10
MetaMAE (sharing HPs in intra-modality)	69.4	61.5	95.5	59.2
MetaMAE (reported)	69.4	62.3	95.5	59.2
Further ablation studies with varying hyperparameters. Table 11, 12, 13, and 14 show the sensitivity of hyperparameters on the PAMAP2 and WaferMap datasets. We observe that MetaMAE performs well even with non-optimal hyperparameters, except for the decoder depth and  $P_{\mathrm{drop}}$ , but we suggest finding better hyperparameters specific to each domain (e.g.,  $\lambda = 0.1$  for WaferMap). Regarding the decoder depth, we find that each modality requires an appropriate value, but generally, MetaMAE performs well with a decoder depth of 4. In Table 15, we observe that single-step adaptation effectively achieves good performance, and in some cases, even outperforms multiple-step adaptation due to the risk of overly decoder-specific support representation.
Table 11: Sensitivity of  $\alpha$  on PAMAP2 and WaferMap.
α	PAMAP2	WaferMap
0.1	86.2	95.5
0.5	89.3	95.4
1.0	89.1	95.2
Table 12: Sensitivity of  $\lambda$  on PAMAP2 and WaferMap.
λ	PAMAP2	WaferMap
0.01	88.6	95.2
0.1	89.1	95.5
1.0	89.3	93.6
Table 13: Sensitivity of decoder depth on PAMAP2 and WaferMap.
depth	PAMAP2	WaferMap
2	84.9	94.2
4	89.3	95.5
6	86.2	95.5
Table 14: Sensitivity of  $P_{\mathrm{drop}}$  on PAMAP2 and WaferMap.
Pdrop	PAMAP2	WaferMap
0	79.4	95.5
0.1	89.3	94.7
Table 15: Sensitivity of latent adaptation step size on PAMAP2 and WaferMap.
step size	PAMAP2	WaferMap
1	89.3	95.5
5	89.6	94.9
We provide a summary of the considered datasets from the DABS benchmarks [83, 85] in Table 16. Note that we use the dataset split described in [83, 85].
Table 16: Datasets considered for pretraining and linear evaluation in our experiments. "MS Image" denotes the Multi-spectral image modality. For Phase, "P" denotes pretraining and "F" denotes fine-tuning.
Modality	Dataset	# of classes	Input shape	Token shape	Phase	Batch size
Time-series	PAMAP2 [71]	12	52 × 320	5	P & F	256 & 256
Tabular	HIGGS [69]	2	28	1	P & F	256 & 256
MS Image	EuroSAT [35, 34]	10	13 × 64 × 64	8 × 8	P & F	64 & 64
Token	Genomics [72]	10	4 × 250	1	P & F	32 & 64
Genomics-OOD [72]	60	4 × 250	1	F	32
Pfam [20]	623	26 × 128	1	P & F	128 & 128
SCOP [23]	1195	26 × 128	1	F	128
Secondary Structure [43, 8]	4	26 × 128	1	F	128
Stability [73]	-	26 × 128	1	F	128
Fluorescence [76]	-	26 × 128	1	F	128
Speech	LibriSpeech [66]	40	1 × 224 × 224	16 × 16	P & F	64 & 64
Audio MNIST [6]	10	1 × 224 × 224	16 × 16	F	64
Fluent Locations [57]	4	1 × 224 × 224	16 × 16	F	64
Fluent Actions [57]	6	1 × 224 × 224	16 × 16	F	64
Fluent Objects [57]	14	1 × 224 × 224	16 × 16	F	64
Google Speech [96]	36	1 × 224 × 224	16 × 16	F	64
VoxCeleb1 [62]	1251	1 × 224 × 224	16 × 16	F	64
RGB Image	waerMap [98]	9	3 × 32 × 32	4 × 4	P & F	128 & 128
ImageNet-32 [17]	1000	3 × 32 × 32	4 × 4	P	64
CIFAR-10 [45]	10	3 × 32 × 32	4 × 4	F	64
CUB [93]	200	3 × 32 × 32	4 × 4	F	64
VGG Flowers [63]	102	3 × 32 × 32	4 × 4	F	64
DTD [14]	47	3 × 32 × 32	4 × 4	F	64
Traffic Sign [81]	43	3 × 32 × 32	4 × 4	F	64
AirCraft [58]	102	3 × 32 × 32	4 × 4	F	64
Vision-Language	MSCOCO [54]	80	(3 × 224 × 224, 30552 × 32)	(16 × 16, 1)	P	64
VQA [1]	2	(3 × 224 × 224, 30552 × 32)	(16 × 16, 1)	F	64
Mismatched-caption [54]	2	(3 × 224 × 224, 30552 × 32)	(16 × 16, 1)	F	64