Understanding and Enhancing Mask-Based Pretraining towards Universal Representations
Authors: Mingze Dong
Leda Wang
Yuval Kluger
Venue: NeurIPS 2025 poster ()
Keywords: Mask-based pretraining
self-supervised learning
high-dimensional statistics
computational biology
Primary area: machine_learning_for_sciences
TL;DR: We present a theoretical framework for mask-based pretraining using high-dimensional statistics and introduce R²MAE, a novel pretraining scheme that enhances self-supervised learning across diverse data domains.

Abstract: Mask-based pretraining has become a cornerstone of modern large-scale models across language, vision, and recently biology. Despite its empirical success, its role and limits in learning data representations have been unclear. In this work, we show that the behavior of mask-based pretraining can be directly characterized by test risk in high-dimensional minimum-norm ("ridge-less") linear regression, without relying on further model specifications. Further analysis of linear models uncovers several novel aspects of mask-based pretraining. The theoretical framework and its implications have been validated across diverse neural architectures (including MLPs, CNNs, and Transformers) applied to both vision and language tasks. Guided by our theory, we propose an embarrassingly simple yet overlooked pretraining scheme named *Randomly Random Mask AutoEncoding* (**R²MAE**), which enforces capturing multi-scale features from data and is able to outperform optimal fixed mask ratio settings in our linear model framework. We implement R²MAE in vision, language, DNA sequence, and single-cell models, where it consistently outperforms standard and more complicated masking schemes, leading to improvements for state-of-the-art models. Our code is available at [this URL](https://github.com/MingzeDong/r2mae).

Understanding and Enhancing Mask-Based Pretraining towards Universal Representations
Mingze Dong Yale University mingze. dowgoyale.gdgeeeooooo edu
Leda Wang
Yuval Kluger Yale University yuval. wwwggeryylle.ddee
Yale University leda. wang@yale... edu
Abstract
Mask-based pretraining has become cornerstone of modern large-scale models across language, vision, and recently biology. Despite its empirical success, role and limits in learning data representations have been unclear. In this work, we show that the behavior of mask-based pretraining can be directly characterized by test risk in high-dimensional minimum-norm ("ridge-less") linear regression, without relying on further model specifications. Further analysis linear models uncovers several novel aspects of mask-based pretraining. The theoretical framework and its implications have been validated across diverse neural architectures (including MLPs, CNNs, and Transformers) applied to both vision and language tasks. Guided by our theory, we propose an embarrassingly simple yet overlooked pretraining scheme named Randomly Random Mask AutoEncoding (R² MAE), which enforces capturing multi-scale features from data and is able to outperform optimal fixed mask ratio settings in our linear model framework. We implement M²MAE in vision, language, DNA sequence, and single-cell models, where it consistently outperforms standard and more complicated masking schemes, leading improvements for state-of-the-art models. Our code is available at this URL.
Introduction
Mask-based pretraining has emerged as a unifying paradigm for self-supervised learning across
natural language [1-4], vision [5-13], and biological domains [14-22]. This approach is prevalent particularly for data that cannot be presented sequentially, such as images and tabular data [23]. The representations learned through masked- based pretraining have consistently yielded state-of-the-an zero-shot and fine-tuning performances on diverse downstream tasks [1, 23, 24, 6, 16].
Despite the widespread success masked autoencoding pretraining schemes, fundamental questions remain about why and how it helps in learning meaningful data representations. Several theoretical works [25-28] investigated its underlying mechanism using different frameworks, yet two critical questions on the qualitative role of masking persist:
(Universality across different contexts) The scheme proves effective across various data domains, masking designs, and neural network architectures beyond transformers. This suggests its underlying mechanism is fundamental and architecture-agnosicc.
(Diversity across domains and tasks) The optimal behavior of mask pretraining differs significantly across contexts. In language modeling, BERT employs a moderate masking ratio of 15% [1]. while in vision, surprisingly high masking ratios (75%) produce superior representations despite removing most of the visual content [6]. Moreover, the optimal masking ratio varies even across different downstream tasks for single model [29].
The performance curves of models with different masking ratios have been explicitly characterized in several works [6, 29), which serve as foundation for several theoretical explanations [25, 28). An
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

intriguing phenomenon is the existence of a sweet-spot masking ratio that achieves optimal model performance. Additionally, several interesting quantitative behaviors of the performance curve, such as plateaus in the low-masking and caar-optimallmaskinn regimes, appear in number of cases [6].
To knowledge, no previous work has proposed theoretical framework general enough to address the aforementioned qualitative challenges, nor have they successfully explained these quantitative behaviors of mask pretraining schemes. Moreover, prior works do not explain the effect of model size in determining the optimal mask ratio [29]. In this work, our main contributions are the following:
1. We introduce a novel theoretical framework based on a high-dimensional linear regression setting tailored to mask prediction. We demonstrate that the test risk of this considered model recapitulates both qualitative and quantitative behaviors of diverse pretrained neural networks with respect to masking ratio in large-scale vision and language models.
2. We derive explicit expressions for the test risk under several cases using random matrix theory [30-32] with novel theoretical contributions. Our results suggest that previous observations mask pretraining behaviors can be explained by solely bias variinc decomposition.
3. We identify and validate several aspects of mask-based pretraining previously unexplained or overlooked-in various architectures across vision and language tasks: 1) The scheme is only beneficial in the overparametrized regime; 2) The optimal masking ratio is task and model-size-dependenn:; 3) It enforces feature magnitude disparity.
4. Building on insights from the linear model, we propose RºMAE, a simple but novel pretraining strategy that replaces fixed mask ratio with uniformly sampled mask ratios from a predefined range. RAMAE yields consistent improvements in vision, language, DNA, and single-cell pretraining, outperforming standard masking and various existing enhancement strategies on downstream zero-shot, linear probing. and fine-tuning tasks. R*MAE enforces models to capture different feature scales, and is able outperform optimal fixed masking ratio performance in real data and linear models under appropriate mask range settings.
2 Related works
Mask-based pretraining in language, vision, and biology. Mask pretraining has become a dominant self-supervised learning approach in recent years, with significant developments in language modeling, computer vision, and biology. In NLP, BERT introduced the Masked Language Model (MLM) objective where random 15% tokens are corrupted and predicted from context [1]. MLM has been adapted in numerous works with modifications [2-4]. Studies show optimal masking ratios may exceed the 15% default and vary by task [29], while dynamic mask scheduling may improve performance [33, 34]. Other approaches propose learnable masks during pretraining [35-37]. In computer vision, researchers drew inspiration from BERT to devise masked image modeling methods, explored in ViT and BEiT [5. 7]. He et al. [6] propose MAE, showing that images benefit from an extremely high mask ratio of 75% to achieve statc-of oft-of-thhhhaa- art results in downstream tasks. This finding sparked numerous empirical studies on evaluating and improving the MAE scheme [8-13, 38).
The mask-baser pretraining paradigm has also made inroads into biological data science, in particular for DNA sequences and single-cell gene expressions. Those DNA models are usually trained directly by the BERT pretraining objective [39, 16], whereas variants of mask rates were explored for single- cell self-supervised learning models ranging from 15% to 90% [18, 20, 21, 17, 40]. To the best of our knowledge, there are currently no successful improvements of mask-pretraining schemes in biological models beyond simply tuning masking rates. See Appendix A. for additional discussions.
Understanding neural networks through linear models. The connection between neural networks and linear models in the proportional regime has been extensively studied in recent years. Here the proportional regime refers to the asymptotic setting where the feature number d and the sample number both tend to infinity, with their limit ratio 7 = d/n E (0, c0). For instance, the double descent phenomenon, where test error decreases with overparameterizztton was characterized empirically in deep networks [41] and theoretically shown for high- dighesiinnsional ridge(-less) regression [42, 43]. Recent works also addressed generalized polynomial regimes where ? = d/n² E (0, (44-446. The equivalence between nonlinear models and linear Gaussian models with matching moment statistics, i.e., universality, have been demonstrated or conjectured in multiple settings [47-49]. Further background of high-diment dighediimnsional linear models can be seen in Appendix A.2.

---

Theoretical endeavors to understand mask pretraining, Recent theoretical investigations aimed to provide insights into mask-based pretraining objectives. Cao et [26] analyzed MAE's attention mechanism through integral kernels and Pan et al. [50] demonstrated autoencoders" capacity to preserve semantic information. Zhang et al. [25] suggests that masking creates implicit positive pairs relevant to contrastive learning. Yue et al. [27] reframed MAE as local contrastive learning where reconstruction loss contrasts different image regions. Kong et al. [28] developed a latent variable framework to explain the existence of optimal masking rate in MAE. To our knowledge, prior research has precisely characterized the quantitative phenomena observed in mask-based pretraining, nor can these approaches be readily generalized across data domains and masking designs.
3 A theoretical framework for mask-based pretraining using high-dimensional linear models
In this work, we formulate the feature-level mask autoencoding problem follows. Let "it....,Tdd]] E IRd+1 be an input sample, where indices {l...., d + 1} denote features (tabular data) or positions (image/language data). A binary mask = = (Zi, ; Id+1) ) E {0, 1هs+1 yields the corrupted input x' = x z. The model fo Rd+1 + Rd+1 is trained reconstruct the original values 2; for features where = O. Denoting the set of masked indices as Sm. {ill 0} and using the Mean Squared Error (MSE) loss L(a, b) = ||a bll2, the objective per sample is:
Σ L(J (J'()x)) 2:) SESm
The purpose of setting the feature dimensionality as + 1 will become clear in the next section. This approach, particularly when fo employs (transformer-basee aligns with prominent masked autoencoding methods like MAE (vision) [6]. BERT (language/DNA) [1, 14, 39, 16], and masked autoencoders for single-cell genomics [17, 18, 21].
encoder-decodes architecture,
3.1 Reduced linear model
To make exact analysis of this feature-level mask autoencoding problem feasible, we introduce two primary simplifications. First, we assume the model f is linear in its input x'' and has no bias term. Specifically, the reconstruction for the i-th original feature x, is given by:
(ax'), x'3. Bi, where Br E Rd+1 is coefficient vector specific to feature i.
Note that if z; is the feature being reconstructed (i.c., 2; = 0), then the i-th component of the input x' is (a'); 24 O. Consequently, the i-th component of Gr (B:); does not contribute to the prediction (x²);
(xy((")) (B:),, Second, we assume the coefficient vectors { dd+1 are independent sets of parameters across different target features i. This allows the problem to be treated as dl + parallel, though potentially coupled through data, regression-like tasks.
We next consider stylized version of one such reconstruction task. Let y (UI,..,, Un) E TR represent an arbitrary single feature from the original sample that we aim to reconstruct (e.g.. y = Fy for some k). We formulate its corresponding regression problem as follows, using n for the total number of training samples. The feature dimension is now as one feature is removed from æ E Id+ Let X E Ra be the matrix containing all original, unmasked training samples with feature removed. We henceforth denote the j-th row of X as Zy. We consider the following teacher model:
x + 6.
(3)
Here, 3 Rd is the ground-truth coefficient vector. The noise E (EI,..., 6x) E E with each entry
by i.i.d. and E(cy] 0,
[[ o². Each sample x, is assumed to have zero expectation E(x;]] = 0.
and covariance Σ- = _ 1. We denote d/n, r = || 3 //|| and k = =02/7²
In the random-mask autoencoding task, each feature chosen as target is selected with probability p. Thus, for the regression on y. the effective number of samples is ń ~ Binomial(n1,p)) p). We let ÿ e i be the vector of these target values, and X E & be the rows of X corresponding to these instances. By Hoeffding's 's concentration inequality, we have n/n = p + o(1) with high probability. Since we only deal with the proportional regime, suffices to assume the case of n/n = p for establishing asymptotic risk quantities in our framework.
3

---

The observed covariates are randomly masked version of Xubb Let Z E {0, 1|axd be random matrix where each entry 2;j is i..d. Bemoulli(1 p), with p the masking probability defined before. Then the covariate matrix is X = sub C Z E Raxd We consider the solution the following ridge-less regression in the proportional regime (d, ñ oo, with d/n + 9 (0, constant):
G lim arg min x-0-
[ x N|I IB]
lim (X X + Aaa X y. X-0
(4)
We are interested in the test risk of the model. For a new, unmasked sample xy E RR, it is of form:
Rx (3:3) ( zo,3)²
X E |1B BIL X
(5)
Relation with standard ridge-less regression framework. Our setup diverges from the standard ridge-less regression framework in two key aspects. First, the effective number of training samples, yip, is directly modulated by p. Second, the design matrix X exhibits a level of induced sparsity (or feature corruption) determined by p. As we will demonstrate, these two p-dependent factors lead to complex and distinct behaviors in the bias and variance of the estimator, compared to classical ridge-less regression. Further background and preliminaries are provided in Appendix A.2.
Test risk and model performance. The test risk for reconstructing feature y be viewed as feature- -wit generalization error, analogous to validation loss. Here, y can represent latent features, whose reconstruction error is connected to the validation loss in the original space through the model's decoding transformation. Therefore, this risk reflects the model's feature learning ability, which indicates its utility for downstream tasks like probing and fine-tuning. The correlation between MAE validation loss and fine-tuning performance, as noted in [8], supports this interpretation.
Relation with real network optimization. Beyond the key linear simplification, complexities such as mini-batch processing and multi-epoch training are not incorporated into our current setup. Our goal in this study is to develop minimal model that captures essential aspects of mask pretraining behaviors. A more detailed characterization of these additional factors remains future research.
Next token prediction. Our linear model addresses an independent sample-wist prediction setting. While aligns well with the mask-based pretraining task, it cannot adequately model the other prevalent pretraining procedure, i.e., autoregression, which is token-wise prediction task with strong contextual dependencies. We anticipate the latter task to exhibit distinct statistical behaviors, which may be revealed through the analysis of a more complex high-dimensional linear model.
3.2 Isotropic model
Here we present our main theoretical results regarding the test risk the considered high-dimensional linear model. We first consider the simplest case where the covariance matrix Σ = I.
Theorem (Isotropic model). When Σ = I, the test risk (5) can be asymptotically expressed as:
(D+a)r [1 [1- P)p)))
<< 1 (7 <);
lim n,d+oo
(x(83))
(6)
+
p(ptk) [-P-T-p)
マ> 1 ( > p).
The proof of the theorem, provided in Appendix B.2, extends well-known results from standard high-dimensional linear regression [43] by employing an isotropic local law for sample covariance matrices [30]. According to the formula, in the underparameterrize regime (5 < 1), the test risk a monotonically increasing function of p. In the overparameterized regime (7 > 1), the test risk exhibits non-monotonic behavior with respect to p. achieving its minimum at some p" E (0, 1). Depending on the value of 7, the test risk curve will either monotonically increase regarding p (y 1), exhibit transition at the threshold 7 = p (r 1). These predictions, including the phase transition phenomenon, are validated by simulations as shown in Fig. IA.
Nevertheless, this outcome is largely unconstructive, as the minimal risk achieved does not offer a substantial reduction compared to that of null prediction (i.e., 9 0, for which R2 (0; = r2). In the following sections, we will demonstrate that the benefit of masking is due to the conditional dependence between unmasked and masked features, key component missing in this setting.

---

3.3 Spiked covariance model
Identity covariance represents a special case without feature dependency, whose characterization effectively reduces to the standard ridge-less case. If the covariance involves interaction terms, a non-trivial standalone treatment would be required. Below, we consider spiked covariance model, E = I dvve where u E IRd is vector and á >> 0 is scalar. Below we denote the masked data covariance as Σ (1 E)²Σ + p(1 p) diag E). We characterize the limiting test risk of this rank-1 1 spiked covariance model in the overparametrized regime:
Corollary 1 (Limiting test risk of spiked covariance model). The test risk (5) has the following limit:
Rx Rx ():
lim n-+00
lim n 20000
Φg + c2 00) + 5(c(1 - Φ.) -32 +
+ cp3
+
x²
,2
pd . 2 B
where
c=
1+ 5(1 p)
63 A.3 1. (A,
+ ビ 1, o. = 2,0 (h,1 + Σ
(8)
TH I Σ) -))- 2)
sb >. (A,I - u TH (A,I I + Σ) 2)
and X is the unique non-negative solution of the fixed point equation ñ IT(E(E+AA) X.1)
The result a corollary of Theorem 2, which characterizes the asymptotics of the test risk in this setup. Theorem 2 and its proof are presented in Appendix B.3, along with moderate delocalization assumption required for the proof. The validity of our derived test risk expression is confirmed by numerical experiments (Fig. IB). Intuitively, when the spike level o is small, the setting effectively reduces to the identity covariance case, where the test risk does not significantly descend. When is large, the behavior of the bias term is mostly characterized by the quadratic term 8(c(1 o.) ) In this case, there can exist a "sweet-spot" masking ratio that minimizes the quadratic term achieving near-zero bias and near-optimal risk, especially when ô is large. This yields the desired descent behavior in real-world mask pretraining curves and is validated via simulations (Figs. IC, 3).
According to the quadratic term, the test risk and the optimal masking ratio both depend on the feature strength, defined as the alignment between 33 and Σ (reducing to 3 v in this case). stronger feature strength results in steeper test risk descent and higher optimal masking ratio (Figs. IB-C, 3). Finally, we empirically observed that higher masking leads to a greater disparity in prediction magnitude, E
[|| X 3 12 x between 3o aligned with Σ and those that are not (Figs. ID. 3).
3.4 General covariance models recapitulate real-world mask pretraining curves
For general covariance matrices, an analytic expression of test risk remains infeasible. Nevertheless, when is an eigenvector of Σ, the behavior of bias and variance terms can be revealed through a simplified form of the test risk, presented as Theorem 3 in Appendix B.4. Similar to the spiked covariance case, the test risk displays a descent with respect to the masking ratio p due to cancellation in the bias term. To verify our results, we simulated covariance models constructed by orthonormal projections of various spectrum distributions (Fig. IE, see Appendix for details). The non- monotonic pattern of the test risk emerges in all models, with stronger effects and higher optimal masking ratios for stronger signal BSs (those corresponding to higher eigenvalues in Σ, Fig. 1E). We also observed a comparable transition threshold where the minimum test risk gains an advantage over null prediction (Fig. IE), which may suggest a form of universality that warrants further theoretical investigation.
We further compared our results with real language models (BERT) pretrained by MLM with different masking ratios [29]. Even for the same set of models, the behavior of mask pretraining curves varies with respect to the evaluation dataset (Fig. 1F). The resulting family of curves aligns well with our observations in linear models (Fig. IE). In vision MAE models [6], the mask pretraining curve can exhibit unusual behavior with two plateaus: 1) Before the performance improves with respect to masking ratio, the model performance remains relatively stable; 2) A range of masking ratios where the model achieves similar near-optimal. Interestingly, with another latent space model (see Appendix for details), we faithfully reproduced the observed two plateaus real mask pretraining curves (Fig. 1G). Notably, another sample from the model results in different curve aligning with MAE linear probing performance (Fig. 4). Together, these comparisons suggest connection between real- world mask pretraining and our linear model framework, which we will further validate in the next section.
5

---

Identity covariance (2 =1)
Spiked covariance (Z =vv++l)
1.2 19
risk for y-5
Test y-O k=0.04
2.00
2 1.50
Theary (Cosits =
L.0
1.1 137
Hean- Str Dew
Mean Skd
1.25
Theory (Cosine - 11
0.5)
Thesry
0.75
Sinuation (Cosine (0.)
5
0.50
Simusaion (Coeme 1)
1x 6.0
04
0.50 6.75
0.25
2.0
Masking roti P
b=
2.0
b=10
Mast ng natoo
2.0
1-:00
Maaking
6-100
tample
sample
1.5
coei
somole
Nay ged
v/w2.75
0.
05
coot
6x100-
10
0.00 5:25 0.50 0.75
0.00 0.25 0.50 73
0.00 0.22 0.59 0.75
0.55 0.50 0.75
Nasting rate
Masking ratlo p
Hasking r
Masking ratlo
Uniform distributed
spectrum
distributed spectrum
space
6-957
3.399
0.047
5.229
8.003
1.0
8.343
8.601
3.722
5.300
10
4.500
6.11
A- 7.503
4-405
3- 2.403
A- 7.122
A.4812
2.19
0.
0.5
5.232
0.692
0.0
Meeking
0.5
4-4335
1.4075
0
60
0.5
1.859
2.209
Figenvalue threshold 2.335
Figenaaue treeshod 2,214
Figerwalue threshold 2,780
0.20
1.00
2-2.529
3.321
20.1
2-292
2.734
20.75
A=1000
1.00
0.05
2-2612
2.535
2.408
2214
1.1000
1.000
1.00
0.03
0.15
1.000
0.00
1.044
1.465
0.00
10
1.000
1.546
0.00
1.003
1.000
Eigenval Magnitude
Diperatuee Magnitude
Ralative Performance of RoBERTa across tasks
HAE buning socuracy IHe 2022)
y-50
Gatten actuney
1.4
122
G172
1.0
0.61
0.00
2.100
-0.33
0.150
0.143
0.2
0.5
0.5
0.5
Mulling Hal00 g
Maakang roloo P
Mosling
Figure 1: A-B. Plots of theoretical test risk and simulations (showing mean and standard deviation from 50 samples) against the masking ratio p for the identity covariance model Σ = I (A) and the 2000 (left), 4000 (right). For the latter model, each entry in v is i.i.d. sampled from ((00, 1) and then scaled to ensure |||/|| = 1.
spiked covariance model Σ = 8oo + I (B). For the former model,
200, 7 5, 5 10. C-D. Plots of mean simulation test risk (C) and mean magnitude ratio (D, defined as E | X
X | between cos(,31 v) 1 and cos( (a, v) = 0) over 50 samples in the spiked covariance model. n = 200. = 5. E. Normalized test risk of different covariance models plotted against masking ratio p, where & was selected different eigenvectors of the covariance matrix Σ (Upper). Histogram of covariance spectrum densities for each model above (Lower). The threshold where the minimal risk becomes smaller than the null risk is highlighted by a dashed blue line. F. Transformed error rates of fine-tuned BERT models evaluated on different benchmark tasks [29] (See Appendix C for details).
G. Plots of MAE fine-tuning accuracy on ImageNet- IK [6] and the normalized test risk of a latent space model against masking ratio.
3.5 Validating insights from linear models in real neural networks
Apart from reproducing existing observations, a successful theory should also provide hypotheses that can be empirically validated. Here summarize main predictions from our theoretical framework:
Mask-based pretraining only beneficial in the overparametrized regime. This is because it reduces risk through the bias term, which only appears in the overparametrized case. Moreover, for these overparametrized models, the optimal masking ratio should be dependent the model parameter size, which determines the limit ratio y thus also the test risk.

---

A
Clavsification Acturacy (Undepararatrivee))
Class fication Acturary (Overerenetrinee)
147
Hikiin cheored layer 1V,
parama
fiss two lypl params 25
Hidden cire deeand loreri e -64
fist two lypt pertim 26,304
8X38
Hidden dire (Seand liyer 216.
Fird two Syal partinc 20
Hidden cire (Seeand byyer 1,006
2xx00
01:02 0 0. DE 0J O5 0
0.3 02 2304 05 86 07 64
Mazking Rall
Mosking Noco
B
Poxt Moatlievel Reconstruction Eror
Average GLUF
SQUAD
Suns charrals 0
large
large basel
0.38
Duss charreds-
base
medirm
0.5
0.04
1.0
0.32
6.1.
02.
0.3
04
85
06
07
30
D
Embecding Din: 16
Embediing Cim:
Emnbedding Dim:
Embedding 1024
snttedrggsigg ww
Enteear re shrck vn Mark
Em1ien Shook ve Park
13.0-
12.5
12.0
22.3
0.50 Mascing Rat
0.50
Nasting Baris
Masting Ratio
Maskingg Ratio
Figure 2: A. Linear probing classification accuracy of MLPs in coaumtterinsufficiett (left) and sufficient (right) settings on MNIST. B. Pixel-level reconstruction error without masking for CNN models trained on CelebA. C. Impact of masking ratio on different RoBERTa model sizes (large > base > medium). Adapted from [29] licensed CC-BY 4.0. axes were flipped for consistency. D. Effective rank of MNIST embedding in overparametrized MLP models of different settings.
2. The performance curve regarding the mask ratio can differ by evaluation tasks even for the same set of pretrained models, due to different features required for the downstream task.
The most decisive support of our theory would be on the first point that cannot be explained via previous arguments centered on training data [25, 27, 28]. We validate these points on MultiLayer Perceptrons (MLPs) trained on MNIST, convolutional neural networks (CNNs) trained on CelebA, and large-scale RoBERTa transformer models [29]. For the former two setups, we pretrained encoder- decoder architectures by pixel-level mask reconstruction tasks. We refer to extensive comparisons performed in [29] for effects of mask ratio and RoBERTa model size pretraining performance. We implemented both parameter-insufficien and sufficient settings for MNIST, whereas the latter was used for evaluating CNNs and transformers. In MLPs, the linear probing error rate exhibits descent for all cerameter-sufffcice models, while the error rate first fluctuates then monotonically increases models (Fig. 2A). The transition observed in parameter-insufficcce models can be explained by the test risk of underparametrized linear models (? < 1, Fig. 1A).
for
For CNNs, the optimal reconstruction of original images was observed for different sets of intermedi- ate masking ratios across model sizes (Fig. 2B). As for linear probing, all models suddenly improve after the masking ratio increases to model-size-speccfic threshold (Figs. 5-7). Together, different CNN model sizes exhibit distinct optimal masking ratios (0.6, 0.7, 0.8 for 8, 16, 32 base channels respectively). For RoBERTa, larger models correspond to higher optimal masking ratio, which is further altered by the evaluation task (Fig. 2C). These evaluations provide strong support for our first prediction, which would not be addressed by existing explanations. Differences of optimal masking ratio across evaluation tasks for CNNs and RoBERTa models further support the second point.
We then explored whether the increased feature magnitude disparity in spiked covariance models (Fig. 1D) appears in real neural networks. Specifically, we evaluated the effective rank (ER) of MNIST image embeddings in MLP models. ER is defined as the entropy of sum-normalinn normalizzl matrix singular values and measures spectrum uniformity [51]. Except for the extremely small embedding case (6im=16), all parameter-sufficient models indeed exhibit decrease of ER with respect to the masking ratio (Fig. 2D), with curve patterns resembling those in Fig. 1D, confirming our hypothesis. This also aligns with the previously observed decrease of ER during training in vision MAEs [25].
7

---

4 RºMAE for universal representation learning
As final contribution of our work, we aim to employ our gained understanding to improve current mask pretraining schemes. Our theoretical framework highlights that different masking ratios selectively emphasize features of varying strength. Therefore, we conclude that essential to expose the model to a range of masking ratios during pretraining. We propose the simplest pretraining method that serves the purpose, which can be described and implemented in one line:
Expose the model to data corrupted with uniformly sampled masking ratio p ~ ((mmin: Pmax).
We term this scheme as Randomly Random Mask AutoEncoding MR²M.E). Despite its simplicity, it has not been implemented in prior works to our knowledge. Existing works focused on improving the mask-based pretraining objective mostly aim to learn adaptive masks during pretraining [38, 37, 52] or perform (deterministic) mask rate scheduling during training [33, 34]. Technically, the closest variant of R² MAE may be the training phase of mask diffusion language model (MDLM) [53, 54], which reconstructs tokens in unmasked to completely masked samples, constituting special case of ( ) = (0, 1). Nevertheless, MDLMs are used for generation instead of fine-tuning related tasks, and fine-tuning standard BERT models with MDLM does not affect/improve downstream task performance [54]. The issue of setting (Pmin: Pmax) (0, 1) for feature learning is apparent with our theoretical framework, as the test risk either degenerates or explodes when p = 0/1.
4.1 Evaluation of R*MAE on vision and language modeling
We first evaluated RAMAE on well-studied image and language pretraining tasks. Our implementa- tions closely follow established practices [6, 29]. For vision pretraining, we implemented different mask ratio settings on the same ViT-base MAE model as in [6]. The considered settings include:
1. Default MAE with constant masking ratio 0.75: 2. R MAE with masking rate p ~ (0.66, 0.9);
3. the training phase of MDLM [54] with masking rate p 2(0 1); 4. Dynamic MR [34] that linearly decreases masking ratio from 0.9 to 0.6; 4. High (0.9) and low (0.5) mask ratios. We trained all models for 150 epochs. While the training shorter than default 800-epoch experiments in [6], their evaluation shows predictable improvements from 100 to 1600 epochs in ViT-Large models. Therefore, we anticipate our results to be comparable across different settings despite suboptimal absolute accuracy. All models were then fine-tuned for classification for 100 epochs following [6].
As shown in Table 1, R marginally outperforms the best alternatives (default MAE and dynamic MR) and does not suffer from suboptimal MR as observed in high and low masking baselines. Across our experiments, MAM yields its smallest improvement for ViT-MAE, potentially for two reasons: 1) Its training involves significantly longer epochs with augmentation, which deviates from other experimental settings and our theoretical framework; 2) R²MAE's pre-training loss in ViT-MAE fluctuates, likely due to variable-length of unmasked token sequences, warranting future improvement.
Table 1: Fine-tuning accuracies of ViT-base MAE models [6] on ImageNet classification. In our benchmarks, masking scheme metrics outperforming optimal fixed MR settings are labeled red.
Metric MR 0.75 (default)
MR 0.9
MR 0.5
MDLM
Dynamic MR
2ºM (Ours)
Top1 Acc.
81.97
81.20
81.80
81.02
81.97
82.00
Top5 Acc.
96.02
95.68
95.93
95.60
96.04
96.05
For language modeling. we trained RoBERTa- base and RoBERTa- (named following [29]) models on the Web dataset for 10B tokens, and fine-tuned them on GLUE benchmarks. The reported accuracy for each task is the average of three runs with different random seeds, consistent with [29]. Similar to vision experiments, we evaluated: 1. Default MIM (MR 0.15); 2. R^MAF (p (0.10.5 15, 0.4)); 3. Dynamic MR [34] (0.4 to 0. 05)5): 4. MLM with fixed 0.4 MR. Our fine-tuning accuracies are comparable to those in [29]. In both models, R MAE achieves best performance in three tasks (MNLI, QQP, SST-2), achieving best overall rank, followed by dynamic MR (Table 2).
4.2 Evaluation of R MAM on DNA sequence and gene expression modeling
One focus of M is on biological data including DNA sequences and single-cell gene expression data, where the standard mask- pretraining scheme remains the prevalent choice [16-18, 21],
8

---

Table 2: GLUE fine-tuning accuracies of RoBERTa models with different pretraining settings.
RoBERTa-Mediund (52M)
RoBERTa- Base (125M)
Method
MNLI
QQP
SST-2
QNLI
Rank
MNL.
QQP
SST-2
QNLI
Rank
MLM default
80.8
89.8
89,9
86.3
3.25
81.5
90.7
91.7
87.8
3.00
Fixed MR 0.4
80.3
89.7
90.1
86.6
3.75
81.7
90.7
91.2
88.5
3.25
MDLM
79.4
89.8
89.3
85.4
4.50
80.3
90.3
91.4
87.7
4,50
Dynamic MR
80.8
90.1
90.5
87.1
1.50
81.8
90.7
91.4
89.1
2.00
MAMAE (Ours)
90.1
90.6
86.7
1.25
81.9
91.9
88.6
1.25
and improving the scheme pressing need. We evaluated 12-layer BERT style model for DNA sequence (GPN-MSA [16]), and 5-layer MLP encoder-decoder model for single-cell gene expression respectively. Apart from RAMAE, we implemented standard MLM/MAE with different masking ratios, MDLM [54]. dynamic MR [34], and learnable mask (named CL-MAE following [38], which effectively covers AutoMAE [52]). We also compared alternative DNA sequence and single-cell models [55-59, 39, 17. 21]. To evaluate if other masking strategies synergize with R^MAE, we further implemented the combination of R MAE with Dynamic MR or CL-MAE. After training, DNA models are evaluated through zero-shot missense/regulatory (CIivaar/OMIM) variant prediction tasks [60-62]. Gene expression models are evaluated using linear probing performances in predicting cell type, disease, and age across donors lung and brain atlas datasets [63, 64).
As shown in Tables 3-4 and 6, MAE achieves the best overall performance in both DNA and single-cell tasks. The only tasks without clear advantage are DNA missense variant prediction (where all best models achieved near-optimal performance) and cell type classification (where the target label artificially curated). Together, among all tested model domains (vision, language, DNA, single-cell), RAMAE is the only scheme that consistently outperforms standard MLM/MAE with best mask ratios, among the default value and min/max ratios used R² MAE. The consistent improvement in our well-controlled comparisons highlights robustness and generalizability of R RAMAE.
Interestingly, for the cases where Dynamic MR and CL-MAE outperform the baseline setting, combining them with M results in a disadvantage compared to RºMAR alone. For better understanding, we inspected specific DNA sequence classes with different prediction performances. The combination of RD MAE with CL improves classification of harder variants including 3'UTR and ncRNA, but the easier 5'UTR variants (Table 3). These results demonstrate that combining REMAE with additional designs may bring advantages in certain cases but not overall improvement.
Table 3: Comparison on DNA variant effect prediction, PAUROC, partial AUROC.
Clinvar (Missense)
OMIM (Regulatory)
OMIM subset class AUPRC
Methods
AUROC AUPRC
PAUROC
AUPRC
S'UTR
3'UTR
ncRNA
NT
0.601
0.652
0,500
0.001
0.010
0.001
0.000
phastCons- 100v
0.883
0.848
0.514
0.006
0.081
0.005
0.005
phyloP-241m
0.912
0.913
0.590
0.028
0.175
0.015
0.028
phyloP-100v
0.927
0.937
0.574
0.038
0.251
0.029
0.039
CADD
0.966
0.967
0.595
0.048
0.279
0.010
0.090
GPN-MS MPN-MMS (MLM)
0.970
0.974
0.644
0.127
0.331
0.044
0.102
MR 5%
0.967
0.970
0.647
0.134
0,330
0.048
0.171
MR 30%
0.970
0.974
0.645
0.131
0.335
0.047
0,081
MDLM
0.970
0.974
0.647
0.131
0.341
0.048
0.110
Dynamic MR
0.970
0.973
0.645
0.132
0.332
0.054
0.082
CL-MAE
0.968
0.972
0.644
0.117
0,328
0.056
0.128
M (Ours)
0.969
0,973
0.649
0.148
0.339
0.050
0.136
Dynamic MR
0.970
0.974
0.649
0.138
0.324
0.045
0.106
CL
0.967
0.971
0.643
0.139
0.330
0.058
0.192
+ CL (1s =
0.965
0.969
0.649
0.140
0,325
0.051
0.208

---

Table 4: Comparison for different single-cell models trained on brain SEA-AD dataset.
Cell state
Alzheimers AUROC
Age Spearman
Avg performance
Methods
BAcc.
ma
Cell
Donor
Cell
Donor
Score
Rank
Normalized exp.
0.798
0.738
0.571
0.611
0.129
0.511
0.560
ScGPT
0.784
0.693
0,549
0.556
0.065
0.272
0.486
12.67
scVI
0.826
0.740
0.631
0.731
0.201
0.502
0.605
7.00
MAE (MR 25%)
0.841
0.737
0.667
0.699
0.483
0.575
0.667
4.33
.R 10%
0.837
0.726
0.543
0.536
0,449
0.399
0.582
10.67
.R 50%
0.839
0.738
0.574
0.591
0.516
0.536
0.632
6.17
MDLM
0.831
0.719
0.667
0.686
0.543
0.622
0.678
6.83
Dynamic MR
0.838
0.735
0.662
0.694
0,444
0.446
0.636
7.50
CL-MAE
0.838
0.729
0.687
0.722
0,462
0.484
0.654
5,83
R MAM (Ours)
0.840
0.737
0.687
0.716
0.572
0.628
0.697
2.17
Dynamic MR
0.834
0.735
0.642
0.682
0.551
0.545
0.665
6.67
+ CL
0.836
0.730
0.684
0.719
0.570
0.559
0.683
4.83
CL (k:
0.837
0.734
0.676
0.707
0.511
0.506
0.662
6.17
4.3 MAM enforces learning multi-scale features and can outperform optimal MR
We further investigated the mechanism underlying the improvement of MaMAE. On real single-cell data, ReMAE achieves near-optimal reconstruction performance across entire masking range, whereas models trained with single, fixed MR are effective only within a narrower range (Tables 7- 8). This observation aligns with the intuition that RºMAE enforces learning multi-scale features, thereby enhancing downstream task performance. Intriguingly, low masking ratios (c.g.. 10%), R can even outperform a model trained specifically at that fixed MR on the reconstruction task. In our linear model framework, we found that with appropriate (Pmin Pmmx)) settings, R-MAE can surpass optimal fixed MR in terms of test risk across different covariance settings in most cases, even when the optimal MR is mildly misaligned with R² MAM masking range (Tables 5,9). These findings suggest additional beneficial properties of R² MAE that warrant future theoretical research.
Table 5: Normalized test risk of RºMAE (MR range 0.5-0.6) against optimal fixed MR and mean MR settings across different random seeds for Beta covariance and latent space models. The ground truth signal / is set to be the first eigenvector of covariance Σ in all cases. n 200, 5.
Beta Covariance Model
Latent Space Model
Seed
Best MR
Min Risk
MR 55%
MAM
Best MR
Min Risk
55%
RºMAE
0.55
0.520
0.520
0.504
0.55
0.611
0.611
0,599
0.53
0.606
0.612
0.599
0.65
0.641
0.673
0.662
0.58
0.626
0.631
0.615
0.36
0.662
0.667
0.653
32
0.51
0.543
0.563
0.549
0.59
0.640
0.659
0.634
42
0.53
0.649
0.658
0.645
0.38
0.640
0.643
0.629
Conclusions
In this work, we introduced and analyzed theoretical framework to elucidate mask-based pretraining in large-scale deep learning models. Motivated by this framework, we propose an extremely simple approach MAMAE, which is shown improve upon state-of-the-art self-supervised image, vision, DNA sequence, and single-cell models by solely modifying the pretraining objective.
Limitations. Explicit characterization of the test risk in more complex model settings (e.g.. R²MAE) requires new analysis tools and remains a direction for future research. Potential improvements of M with dedicated domain-specift designs also remains to be explored.
envision that our theoretical framework will serve as a basis for better understanding self-supervised pretraining, one of the most important components in modern deep learning and foundation models. Furthermore, our work addresses pressing need for building better models towards universal representations, with immediate impact for the (biological) AI community.
Broader impact.
10

---

Acknowledgements
The authors thank Theodor Misiakiewicz, Boris Landa and the anonymous reviewers for helpful discussions and feedback. Y.K. acknowledges support by NIH grants U54AG076043, U54AG079759, P50CA121974, R01GM131642, UMIDA051410, and U01DA053628.
References
[1] Jacob Devlin, Ming-Woo Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language lechnologies, volume I (long and shori papers), pages 4171-41 2019.
[2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1et.i.997..16.... 11692, 2019.
[3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67 2020.
[4] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V
Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 12Xiv:0010.11929 2020.
[6] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[7] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- 2021.
formers. arXiv preprint
[8] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Yixuan Wei, Qi Dai, and Han Hu. On data scaling in masked image modeling. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 10365-10374, 2023.
[9] Yuge Shi, N Siddharth, Philip Torr, and Adam Kosiorek. Adversarial masking for self- supervised learning. In International Conference Machine Learning, pages 20026-20040. PMLR, 2022.
[10] Yunjie Tian, Lingxi Xie, Jiemin Fang, Jianbin Jiao, and Qi Tian. Beyond masking: Demystifying token-based pre-training for vision transformers. Pattern Recognition, page 111386, 2025.
[11] Ronghang Hu, Shoubhik Debnath, Saining Xie, and Xinlei Chen. Exploring long-sequence masked autoencoders. arXiv preprint arXiv:2210.07224, 2022.
[12] Christoph Feichtenhofer, Yanghao Li, Kaiming He, al. Masked autoencoders as spatiotempo- ral learners. Advances in neural information processing systems, 35:35946-35958, 2022.
[13] Floris Weers, Vaishaal Shankar, Angelos Katharopoulos, Yinfei Yang, and Tom Gunter. Masked autoencoding does not help natural language supervision at scale. Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pages 23432-23444, 2023.
[14] Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dia-languaae in genome. Bioinformatics, 1711522212--120, 2021.

---

[15] Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert- 2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306. /5006, 2023.
[16] Gonzalo Benegas, Carlos Albors, Alan J Aw, Chengzhong Ye, and Yun S Song. A dna language model based on multispecies alignment predicts the effects genome- widd variants. Nature Biotechnology, pages 1-6, 2025.
[17] Haotian Cui, Chloe Wang. Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. scgpt: toward building a foundation model for single-cell multi-omics using generative ai. Nature Methods, 2 21(8):1470-1480) 2024.
[18) Christina V Theodoris, Ling Xiao, Anant Chopra, Mark D) Chaffin, Zeina R Al Sayed, Matthew C Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng. Shirley Liu, et al. Transfer learning enables predictions in network biology. Nature, 61877965)61666224 2023.
[19] Mingze Dong, Kriti Agrawal, Rong Fan, Esen Sefik, Richard A Flavell, and Yuval Kluger. Scaling deep identifiable models enables zero-shot charactcrization of single-cell biological states. bioRxiv, pages 2023-11, 2024.
[20] Anna C Schaar, Alejandro -apuuer, Giovanni Palla, Robert Gutgesell, Lennard Halle, Mariia Minaeva, Larsen Vornholz, Leander Dony, Francesca Drummer, Mojtaba Bahrami, et al. Nicheformer: a foundation model for single-cell and spatial omics. bioRxiv, pages 2024-04, 2024.
[21] Yanay Rosen, Yusuf Roohani, Ayush Agrawal, Leon Samotorcan, Tabula Sapiens Consortium, Stephen R Quake, and Jure Leskovec. Universal cell embeddings: A foundation model for cell biology. bioRxiv, pages 2023-11, 2023.
[22] Abhinav Adduri, Dhruv Gautam, Beatrice Bevilacqua, Alishba Imran, Rohan Shah, Mohsen Naghipourfar, Noam Teyssier, Rajesh Ilango, Sanjay Nagaraj, Chiara Ricci-Tam, et al. Pre- dicting cellular responses to perturbation across diverse contexts with state. bioRxiv, pages 2025-06, 2025.
[23] Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):119-366 2025.
[24] Till Richter, Mojtaba Bahrami, Yufan Xia, David S Fischer, and Fabian Theis. Delineating the effective use of self-supervised learning in single-cell genomics. Nature Machine Intelligence, pages 1-11, 2024.
[25] Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. Advances in Neural Information Processing Systems, 35:27127-27139) 2022.
[26] Shuhao Cao, Peng Xu, and David A Clifton. How understand masked autoencoders. arXiv preprint arXiv:2202.036670, 2022.
[27] Xiaoyu Yue, Lei Bai, Meng Wei, Jiangmiao Pang, Xihui Liu, Luping Zhou, and Wanli Ouyang. Understanding masked autoencoders from a local contrastive perspective. arXiv preprint arXiv:2310.01994, 2023.
[28] Lingjing Kong, Martin Q Ma, Guangyi Chen, Eric P Xing, Yuejie Chi, Louis Chii--Philippe Morency, and Kun Zhang. Understanding masked autoencoders via hierarchical latent variable models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7918-7928, 2023.
[29] Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling? arXiv preprint arXiv:2202.08005, 2022.
[30] Alex Bloemendal, László Erdos, Antti Knowles, Homn-TTzer Yau, and Jun Yin. Isotropic local laws for sample covariance and generalized wigner matrices. Electron. Probab, 19(33):1-53,
2014.
12

---

[31] Antti Knowles and Jun Yin. Anisotropic local laws for random matrices. Probability Theory and Related Fields, 169:257-352, 2017.
[32] Theodor Misiakiewicz and Basil Saeed, A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gev estimator. arXiv preprint arXiv:2403. 20X0:22403.08938, 2024.
[33] Dongjie Yang, Zhuosheng Zhang, and Hai Zhao. Learning better masking for better language model pre-training. arXiv preprint arXiv:2208. 10806, 2022.
[34] Zachary Ankner, Naomi Saphra, Davis Blalock, Jonathan Frankle, and Matthew L Leavitt. Dynamic masking rate schedules for mlm pretraining, arXiv preprint arXiv:2305.150%6 2023.
[35] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the association for computational linguistics, 8:64-77, 2020.
[36] Yoav Levine, Barak Lenz, Opher Lieber, Omri Abend, Kevin Leyton-Brown, Moshe Tennen- holtz, and Yoav Shoham. Pmi-masking: Principled masking of correlated spans. arXiv preprint arXiv:2010.01825, 2020.
[37] Nafis Sadeq, Canwen Xu, and Julian McAuley. Informask: Unsupervised informative masking for language model pretraining. arXiv preprint arXiv:2210. 2022.
[38] Neelu Madan, Picolae-Cätälle Ristea, Kamal Nasrollahi, Thomas B Moeslund, and Radu Tudor lonescu. Cl-mae: Curriculum-leann
masked autoencoders. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2492-2502, 2024.
[39] Hugo Dalla-Torre Liam Gonzalez. Javier Sendoza-Reville Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkhatim, et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, 22(2):287-297, 2025.
[40] Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang. Jianzhu Ma. Xuegong Zhang, and Le Song. Large-scale foundation model on single-cell transcriptomics. Nature methods, (8); 1491-1 2024.
[41] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116632):15849-1588 2019.
[42] Dominic Richards, Jaouad Mourtada, and Lorenzo Rosasco.
Asymptotics of ridge (less) regression under general source condition. In International Conference on Artificial Intelligence and Statistics, pages 3889-3897. PMLR, 2021.
[43] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. Annals of statistics, 50(2):949, 2022.
[44] Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics, 49(2), 2021.
[45] Chen Cheng and Andrea Montanari. Dimension free ridge regression. The Annals of Statistics, (100)2277-29122 2024.
[46] Hong Hu, Yue Lu, and Theodor Misiakiewicz. Asymptotics of random feature regression beyond the linear scaling regime. arXiv preprint arXiv: 2403.08100 2024.
[47] Andrea Montanari and Basil N Saced. Universality of empirical risk minimization. In Confer- ence on Learning Theory, pages 4310-4312. PMLR, 2022.
[48] Hong Hu and Yue M Lu. Universality laws for nigh-dimensional learning with random features. IEEE Transactions on Information Theory, 1932- 2022.
[49] Dominik Schröder, Hugo Cui, Daniil Dmitriev, and Bruno Loureiro. Deterministic equivalent and error universality of deep random features learning. In International Conference on Machine Learning, pages 30285-30320 PMLR, 2023.
13

---

[50] Jiachun Pan, Pan Zhou, and Shuicheng Yan. Towards understanding why reconssccttttttti pretraining helps in downstream tasks. arXiv preprint 0tti:22000038226, 2022.
[51] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606-610. IEEE, 2007.
[52] Haijian Chen, Wendong Zhang, Yunbo Wang, and Xiaokang Yang. Improving masked autoen- coders by learning where to mask. Chinese Conference on Puttern Recognition and Computer Vision (PRCV), pages 377-390. Springer, 2023.
[53] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zeutlemoye. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint arXiv: /904, 1904,09324, 2019.
[54] Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 17:130136-100 130184, 2024.
[55] Nadav Brandes, Grant Goldman, Charlotte Wang, Chun Jimmic Ye, and Vasilis Ntranos. Genome-wide prediction of disease variant effects with deep protein language model. Nature Genetics, 5509):512-1522, 2023.
[56] Philipp Rentzsch, Max Schubach, Jay Shendure, and Martin Kircher. Cadd-splice iamp-ppiic-iimproinngg genome-wide variant effect prediction using deep learning-derived splice scores. Genome medicine, 13:1-12, 2021.
[57] Katherine S Pollard, Mclissa Hubisz, Kate R Rosenbloom, and Adam Siepel. Detection of nonneutral substitution rates on mammalian phylogenies. Genome research, 20(1): 2010.
[58] Patrick F Sullivan, Jennifer RS Meadows, Steven Gazal, BaDoi N Phan, Xue Li, Diane Genereux, Michael X Dong. Matteo Bianchi, Gregory Andrews, Sharadha Sakthikumar, et al. Leveraging base-pair mammalian constraint to understand genetic variation and human disease. Science, (e06664)) 2023.
[59] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch- Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural 2023.
information processing systems,
[60] Melissa J Landrum, Shanmuga Chitipiralla, Garth R Brown, Chao Chen, Baoshan Gu, Jennifer Hart, Douglas Hoffman, Wonhee Jang, Kuljeet Kaur, Chunlei Liu, et al. Clinvar: improvements to accessing data. Nucleic acids research, 48(D1)::D35--D84) 2020.
[61] Damian Smedley, Max Schubach, Julius OB Jacobsen, Sebastian Köhler, Tomasz Zemojtel, Malte Spielmann, Marten Jäger, Harry Hochheiser, Nicole L Washington, Julie A McMurry et al. A whole-genome analysis framework for effective identification of pathogenic regulatory variants in mendelian disease. The American Journal of Human Genetics, 99(3):595-66) 606, 2016.
[62] Siwei Chen, Laurent C Francioli, Julia K Goodrich, Ryan Collins, Masahiro Kanai, Qingbo Wang, Jessica Alföldi, Nicholas A Watts, Christopher Vittal, Laura D Gauthier, et al. A genomic mutational constraint map using variation in 76, 156 human genomes. Nature, 625(7993): 92-100, 2024.
[63] Lisa Sikkema, Ciro Ramfrez-SSústeguu, Daniel C' Strobl, Tessa E Gillett, Luke Zappia, Elo Madissoon, Nikolay S Markov, integrated cell atlas of the lung in health and disease. Nature medicine, 2966))156311577 2023.
Zaragosi, Yuge Ji, Meshal Ansari, et al. An
[64] Mariano Gabitto, Kyle J Travaglini, Victoria M Rachleff, Eitan S Kaplan, Brian Long, Jeanelle Ariza, Ding, Joseph T Mahoney, Nick Dee, Jeff Goldy, et al. Integrated multimodal cell atlas of alzheimer's disease. Nature Neuroscience, 27(12):2366 27662)333662338, 2024.
[65] Gang Li, Heliang Zheng. Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: semantic-guided masking for learning masked autoencoders. Advances in Neural Information Processing Systems, 15:14290-14302 2022.
14

---

[66] Agrim Gupta, Jiajun Wu, Jia Deng, and Fei-Fei Li. Siamese masked autoencoders. Advances in Neural Information Processing Systems, 06:40676-40692 2023.
[67] Daniel M Bear, Kevin Feigelis, Honglin Chen, Wanhee Lee, Rahul Venkatesh, Klemen Kotar, Alex Durango, and Daniel LK Yamins. Unifying (machine) vision via counterfactual world modeling. arXiv preprint arXiv:2.30..01 01228, 2023.
[68] Jiawen Wang, Yinda Chen, Xiaoyu Liu, Che Liu, Dong Liu, Jianqing Gao, and Zhiwei Xiong. Dual form complementary masking for domain--aaatiin image segmentation. arXiv preprint arXiv:2507. / 12008, 2025.
[69] Edgar Dobriban and Yue Sheng. Distributed linear regression by averaging. The Annals of Statistics, 19-2)991-9943, 2021.
[70] Francisco Rubio and Xavier Mestre. Spectral convergence for general class of random matrices. Statistics & probability letters, 81 (15)5592-660, 2011.
[71] Edgar Dobriban and Stefan Wager. High- digh-dimensional asymptoties of prediction: Ridge regres- sion and classification. The Annals of Statistics, 46(1):247-279, 2018.
[72] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[73] Yaskov, Lower bounds on the smallest eigenvalue of sample covariance matrix. Electronic Communications in Probability, 19:083, 2014.
[74] Martin Herdegen, Gechun Liang, and Osian Shelley. Vague and weak convergence of signed measures. arXiv preprint arXiv:2205. 13207, 2022.
[75] Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811 11-30849, 2024.
[76] Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology, 19:1-5, 2018.
[77] Adam Gayoso, Romain Lopez, Galen Xing, Pierre Boyeau, Valeh Valiollah Pour Amiri, Justin Hong, Katherine Wu, Michael Jayasuriya, Edouard Mehlman, Maxime Langevin, et al. A python library for probabilistic analysis of single-cell omies data. Nature biotechnology. 40(2):
163-166, 2022.
[78] Romain Lopez, Jeffrey Regier, Michael B Cole, Michael Jordan, and Nir Yosef. Deep generative modeling for single-cell transcriptomics. Nature methods, 1555201003--1058 2018.
15

---

Appendix Contents
A Additional text
16
A.1
Further discussions on alternative mask pretraining schemes
16
A.2
Background on tighddimessionl linear regression
17
A.3
Implementations CL and MAM CL
17
Additional theoretical results and proofs
18
B.1
Statement and proof of technical lemmas
B.2
Proof of Theorem
21
B.3 Statement and proof of Theorem 2
B.4
Statement and proof of Theorem
24
Experimental details
25
C.1 Simulations
25
C.2 Evaluations on trained BERT and MAE models
26
C.3 MNIST
26
C,4 CelebA
................................
27
C.5 VIT MAE models
27
C.6 RoBERTa models
28
C.7 DNA sequence models
28
C,8 Single-cell gene expression models
29
Appendix Figures
31
Appendix Tables
36
A Additional text
A.1 Further discussions on alternative mask pretraining schemes
Approaches to improving mask-based pretraining can be broadly divided into two categories. The first category focuses on refining the masking scheme itself, i.c., optimizing the selection of pixels/tokens to be masked to maximize pretraining efficacy or downstream performance. Numerous efforts have explored this direction [35, 3, 36, 65]. A number of these schemes assume specific data structures, such as sequential information in text, and thus may not readily generalize across all data domains. A prominent recent direction in this category involves learning the masks themselves during training, for instance, by optimizing them to enhance the pretraining objective or, conversely, to adversarially challenge it [52, 38]. Apart from these general enhancement strategies, several works specifically design masking procedures emphasize specific downstream tasks [66-68].
Wettig al. [29] conducted an extensive evaluation of different masking strategies for BERT masked language models, including a number of those cited above. Their findings highlight that while the optimal masking ratio might vary across strategies, simple uniform random masking often suffices to achieve peak performance. In the single-cell genomics context, Richter et al. [24] evaluated various structured masking schemes (e.g., gene program and transcriptional factor-based masking) against uniform masking. all at fixed masking ratio. They observed no consistent overall advantage for the more complex, structured masking schemes over uniform random masking. These results from both language and genomics domains align, suggesting that highly sophisticated, domain-specific masking strategies may not always be necessary for effective pretraining. The comparable performance achieved by different masking schemes may serve support for the general applicability of our theoretical framework, which is established based on uniform masking.
The second category of approaches focuses on altering the masking rate. Wettig et al. [29] observed that higher masking ratios generally boost masked language modeling performance, particularly for larger models. Ankner et al. [34] further demonstrated that a dynamic masking schedule, gradually reducing the masking rate from 40% to 15%, improves performance, while the reverse schedule does not. A key insight from the influential Masked Autoencoders (MAE) work [6] is that extremely high masking rate for images can force the model to learn robust and generalizable representations through the reconstruction task, leading to improved downstream performance. Notably, our theoretical
16

---

framework highlights a potential limitation of existing mask pretraining schemes: employing single, static masking strategy- whetherr involves carefully designed masking pattern or masking ratio- may not be sufficient to optimally capture the diverse spectrum of features present in data.
A.2 Background on high-dimension lincar regression
The major theoretical focus in this work is the linear model
= X3+
E~N(00,'
where X E Ruxd is the design matrix and 3 R the true parameter. For estimator B, the out-of-sample prediction risk at a fresh covariate Dy admits the bias variance decomposition
Rx (x(3:3) B) = [E] - [ + Tr| (()()()
Bias
Variance
where = E. [2 the population covariance.
In the proportional regime d/n + 7 E (0, oo), the ridge regression estimator is of form Bx = (x X -
For fixed X, its bias and variance are
Bias X² pd
X X
AI) -1
E AI)
o² Tr X n
Variance
x
A streamlined way to capture asymptotics in proportional models is via the theory of deterministic equivalents [69, 32j. Two sequences of (possibly random) matrices An: B, E Rax* are declared asymptotically equivalent (denoted A, x B,,) if for every sequence O, bounded in trace norm,
Tr Tr,O,(4, - B,)] - 0, n + co.
Within this framework, Rubio and Mestre [70] showed that the resolvent of the sample covariance
(Σ -z= equivalent a deterministic matrix (a, - -1)=1 -1, where a, solves an explicit fixed-point equation. Such equivalences yield precise control over traces of analytic functions of random matrices and underpin modern high-dimensional risk calculations. The limiting risk of high-dimensional ridge regression has been extensively studied in the past. As a representative result, the exact asymptotic risk has been established in [71]. We refer to [30, 31] for more results on the local law, which asserts the convergence of the resolvent entrywise under high probability bounds, at scales finer than the global limit.
In [43], the authors study the behavior of ridgeless least squares interpolation in high- -igh-dimensional settings, where the model interpolates the training data perfectly analogous to overparametrized neural networks. Surprisingly, they show that such interpolating solutions can theoretically generalize well under certain conditions. Their work derives exact asymptotic expressions for the bias and variance of minimum-non morm interpolators in the overparameterized regime, using tools from random matrix theory and the theory of deterministic equivalents. The results serve as a basis for understanding neural network behavior from the lens of high-dimensional linear regression theory.
A.3
Implementations of CL and RºMAE- + CL
Inspired by recent curriculum learning (CL) approaches in MAE [38, 52], which often involve an adversarial mask generator and an easy-to-hard progression (c.g., by scheduling gradient coefficient for the mask generator [38]), we evaluated whether these approach improves self-supervised learning for our biological data settings. Given that our DNA sequence and single-cell gene expression models process entire input sequences rather than patches, we implemented CL by introducing learnable, positive coefficients that modulate the element-wise reconstruction loss for each feature. These coefficients are constrained to have a mean of one. We then applied a gradient scheduling mechanism to these loss coefficients: one setting used constant gradient multiplier of 1 (termed the h: = 0 setting, which learns an easy mask with the smallest loss value throughout pretraining). while another employed a dynamic multiplier decreasing from 1 to -1 to simulate an casy-to-hard progression.
17

---

Our initial evaluations showed that the A = 0 fixed curriculum led to severe learning degeneration and hampered performance (Tables 3, 5), which would be an anticipated outcome.
To address this and integrate these CL. principles with our R MAE framework, we propose to randomly sample masking ratios from predefined discrete set of I values, e.g., [min__ation ratio,..., mmx__atiio], For each of the discrete masking ratios, we learn an adaptive, positive weight vector w, E R4 (where d the feature dimension), which form the columns of a weight matrix W E R These weights dynamically adjust the importance of reconstructing each feature under that specific masking ratio j. To ensure balanced learning across features and masking ratios, we impose mass-conservation constraints on W such that:
d W =d,
Vi E (1,
W =4 (I 1) Ø=1
Σ
This approach aims to provide, on average, the same learning signal magnitude per feature, while still allowing each masking ratio to prioritize different feature subsets. In practice, these constraints are efficiently enforced using a few iterations of the differentiable Sinkhorn algorithm positive matrix Wo [72]. Wo itself is generated by a small MLP applied to the full uncorrupted data features (for single-cell models) or the transformer's s masked token representations (for DNA sequence models). Notably, this R"MAE-CL approach effectively resolved the learning degeneration observed in the simpler k- 0 setting without requiring additional regularizations [38] (Tables 1-4,6).
initial
B Additional theoretical results and proofs
In this section, with a slight abuse of notation, we denote sub as X for brevity.
B.1 Statement and proof of technical lemmas
Lemma (Bias-variann) variance decomposition for general covariance model). The test risk Rx (3. =
E 118-Bll - has the following decomposition (x (x(3 - (3, outside negligible set, which can be expressed as:
(3) = =|[0B=1++---00000 + X ullz.
(10)
(B, Σ i==1
3,8;
(X ) FET+EEE ΣX Jaaw +o2II
(11)
a=1
where we denote the projection matrix I1 E E --5=** x X, as well as
Z [j Zaj =1}, 2 {j Zd] 0), Ya [i] 2,z, Ez, B2g Vi [i]
(12)
u E RR.." ", === = X E2,
(13)
w = X (2g Ei Σ 2. 2. Ez.i), Va E [i], i,j E (4.
(14)
Proof. From the definition, our test error is given by
(x = E (w 3 - x 3) x = EE [[ - 3 x
where | x Ex. Note that we have the virianer decomposition
Rx (3: = | x) - 31 + Tr(Cov (3) ($) X B (3:9)
(15)
Vx
Since 3 = (X y
X e), We have
11 X] - 3 = [B[(([((( x) - x) = -13 + (X X X x11 (16)
Tr (3v x) = Tr(E(((@ B 1 -[ (X])
(17)
18

---

Here, X 2). We notice that the event (X,, 0} is the same as [Z = 0} except for a negligible set, in the following proof, we take them as two identical events. In this case, we have the following two relations: for any sample i,
(X x X 0; (X
X
,,,,, E2, Ez, "(Ezz,,D2z,. Xi,z, -,;
(18)
Therefore
X E X
E
Ev,, - X,, 22,2, 2,2,
Xi,
E2,2,
(19)
The remaining entries of U are equal to zero. For the bias term, we have that
xx (x 3) [[ SI | (X x)* EUiEI.
(00)
The latter term can further be simplified as
(x xx+ ΣV'B= X uu u.R.RRR, E2, E2,2,Bz
(21)
For the variance term, we have
d
B[(X- x)33T(X (X- - X) -
X| = Σ
8,8, w
f,j=1
W diaga (w : diaya (1 Xxc (Zij Eiz, Σ2, Ez,i))
(22)
With this relation, we have
Vx (3,3) = T=(Cov($1Xx2))
TT(TE((( EI ( TE
TE E|
Tr
(x+ (BI(X- x3 (x - -X -1) (XT)*
(23)
Tr o x ++ Σ
B.9, W (XT)-XX*)
1-==
Σ BeB Σ ((
wan + o²Tr ((T+})+)
Combining all results above, we get the final expressions for B (3, and Vx (3.
Lemma 2 (Deterministic Equivalence For Trace-class Statistics). For the model Σ = dvo +
with ||ll = 1, we have the following holds with high probability: for a, b E {3, (B,u), v}, and any >> 0,
there exists some constant C independent of n,
T T
dG (s)
S Cn 11/22e
(24)
s the
S (atp. d(la(s) + 8 oG (s) 8 d (str.) dHa(s)
T+(Et))
<Cn
(25)
Proof. Since X has i..d. rows and each row is ΣΣ -subausssian, by the deterministic equivalent of resolvent for random sub-gaussian sample covariance matrices, see e.g. [32, Theorem 4], for any two given D, K > O, we have that the following holds with probability at least 1 - C"n DD for 22 K):
Aa (xT*/++X1) -1 b x-aa + Σ) E) 1b
polylog(a) Vn(Aū)3/2 >a/a A X.) -,)-1b1, (26)
19

---

where A, is the unique solution of the fixed point equation
Añ A = TT(E(E +x1)
We also notice that for the eigenvalue decomposition X / -UUUU,,
Aa T / X ----**1 E11 - S Ala"U(AL
(X
d
(x) / (27)
min
where min represents the smallest non-zero singular value. standard using concentration on random subgaussian matrices, see e.g. [73], to get mmin (X)/ 2 C(a (5)mii((),7)) (E),7/)) with
overwhelming probability for d/ñ Tn/P > 7/p strictly different with 1. Therefore, combining the above results, we get
(nl A. 15/273/2
I.. (A.I+ Σ) a a E Σ)
(28)
the other hand, while Añ - 0+, the above fixed point equation still makes sense, and X, + p. such that
Tr(E(E+ +M)--) n.
We next claim that
D4a T (A.I ( 14a (n, I Σーー - CX,
(29)
while we notice that the the fixed point equation can be equivalently be written as
pd 7>.
/
A s dHa(s) f()+;A),
(30)
where Al represents the empirical spectral distribution E. a direct consequence, is not hard to see supp( S [(1 =p) 1+.). From [31, Lemma 2.2], X, nonnegative and monotone increasing
in A. Therefore, f(x; A) is non-decreasing on (0, oo) for any x 0 with lim f(z; X) 1, lim fz; 0) = O, lim (x; A) = -o for any À > O. We also have the natural upper and lower bound as fx; A) I f(z; A) S f(x; A), where f(æ; A) is the same as f(; X) while changing Ha to the dirac measure at (1 - p) and f(z; A) is the same as f(x; A) while changing H. to the dirac measure 1 + 6. So there exists further implies that fx; X) is uniformly bounded. Finally
constant C > 0, such that C-1 & p. X S.C. This
0zf(2; A) = pd
(x + s)2 dHo(s),
(31)
yx.2
for 8 101-))2,11 + 8 and E [C-1,]], the above bounded away from zero and above. Therefore, there exists another constant C such that Č-I < 0xf(r;; < Ć. To be concise, we replace C by max(C, C). Utilizing the implicit function theorem, we can get [0x X)] < for X [0, 1], and therefore (t, n+l & C.. Combining all the above arguments, would have
DLa T (A.I (o.++E)11 シ - u,a (U,I Σ) -)>l-11 s CA.
(32)
Taking À 1113/3 the right- side turns with the speed at least n 1/2+te, so we can get the final control
11/2tt
(33)
The remaining part of the proof for Tr( Σ) is analogous to the result above using [32, Theorem (46)), so we omit the full proof here.
20

---

B.2 Proof of Theorem 1
Proof. One important observation here that the bias term and the variance term are homogeneous for ||3||, so below we assume [| = 1 without loss of generality. Since we are under the isotropic
setting, Σzz- 0 for any indices set Z. Also, X has i.i.d. elements with variance p. By Lemma
1, the bias term is simply given by
x (B. 3) |ī = 3 (T((5 Σ 1)33,
(34)
while for the variance term, direct calculation shows w x Xoj=o dij. Therefore,
Vx (),
Σ
3,33
X
)aa XnIo 8iz + T (xTx)))
(35)
b,==1
d
X,, x0 +o (x(x))+)
1
(36)
an
a=1
To deal with the first term (I) in variance, we use the isotropic local law [30, Theorem 2.5]. De- fine (A) = X(XX / + - X1)
Qa(A) = Xx // / ( T / + 2/) then Qa =
82 Ra (n) and we consider À such that 0 < Im < 1, Re(A) ñ 22/3+< for some e' > 0. we have the following with high probability that
Im(m(A))
Ra)aaa m(a) S
n-1++
(37)
Im(
uniformly for a E [7].. Using the similar argument as in [43, A.3], for all real A 7/3346 we
get
Qa(A)aa axm(A)/ml A x - (3-1)/2
(38)
the following holds with high probability
(Qa(A)aa - axm(A)/n) 1 1x,,=0
S - - -
(39)
Hoeffding's 's inequality shows that for arbitrary small é >> 0, with probability at least 1 2d exp( -nt),
pm S ñ (1+)//2
X xx:=0 =0
a=1
and as a direct consequence,
Σ
B2
x
pd
Σ
x x..=0 pm S (1(+))/2
a=1
Combined with these, we have the following with some absolute constant C:
1 - pTT(Q=(A)) S (1-01-/)22222
(40)
similarly as (27), we have that there exists constant C only depend on 7n. such that with high probability, Tr (Qa(A)) - Tr (XXXT)+) S CA. Finally, up to some constant, we can bound the difference between (I) and p'r (xxX))+) by A + Cn (0fn(11-0) (1-c)/2x-3 -2 which converges to 0 for ñ (1-c)/6 To sum up. based on Lemma 2 with 8 0 and À n -2/3+t' we get that
B (8. 3
H.
dG20(s) (s) < Cnn 22/3+e
(41)
,2
21

---

where μ., is the solution of n = Tr((1 +
that is, μ., = (1 - p)ppp)ppppppp +)+-
Also, Cd (s) is the dirac measure p. This suggests Bx (8, (y p)/7 almost surely
when 7 > p. and (9, (1///)//2 + when 7 < p. is almost the same for us to use Lemma to get
that with high probability,
(3,
7(1 R1 - p) p p(1 p1- n.)² 7(1 - -D)3
+ 0
,2
which leads to our final result.
B.3 Statement and proof of Theorem 2
In this section, we first provide the delocalized signal assumption and several definitions, then present the statement and proof of Theorem 2.
Assumption (Delocalized signal). 3a 0, such that ||wl| = (dt-a) and || O(d-a).
Here we assume that and the direction of /3 should not be too sparse in order to establish concentra- tion properties of the masking process on our signals. This is purely technical, and we can select a sufficiently small to accommodate specific scenarios in the application.
Definition (Spiked Covariance Structure). For Σ = + dew" where 5 > 0 and || = 1, denote the masked covariance Σ = (1 - -) + p(1 - p) diag (). In other words, Σ = (1 p) + (1 pvv + p(1 p)8 diag(v @ v). Suppose Σ Σ x is the spectral decomposition of Σ with 1 + 2 31 2 8, 2 ... 2 ōa 2 (1 - p)?. We use Ha(s) := E'= 1.28, to represent the
empirical spectral distribution of Σ. We also denote the following (signed) empirical measures
G (s) ( .28,
,28,, ()
Σ (0, xiMoo, x .26, C
Σ (u, 3 2
Denote μe, to be the unique non-negative solution of
1- P
Px dHa(s), s s+p. p.
(42)
7
We then define the predicted bias and variance by
B(Ha, G G G. = /
He
&G (s)
+
( pô pp. T B
1 11 - )
?
/
s
dG (s)
st
s s p+
+ (-/
Me dd
1 8(1 - p) /
pô v B
dG (s)
2
8 fhx
8 1xx
(43)
p²5 (u (3)2 1 8(1 - pp)
(o+u. dHals)) + (otu.)) (= à
dG (s)
Villa Go, G.') == K K x+
(o+u,yz (H)())
(44)
Theorem 2 (Spiked covariance model). The test risk (5) can be decomposed as R (3:3) =
Bx Bx (3; + Vx (3:,) (3 with forms of the two terms available in Lemma 1. Suppose Assumption I holds, and for some arbitrary e that sufficiently small, assume d (a)p-)))/2) for a given in Assumption 1. then with overwhelming probability,
B x (8. B) 22
-B(Haa G99 G&v ) S Cd
(45)
Vx (3,
v(H G.v G
Cd
(46)
2²
Furthermore, if we assume Hag + H, G G88 Gee + G G + Gov then almost surely
R (x:3; B)
p2 + B(H, 33 G. GT) + V(H, G.O.)
(47)
22

---

Proof. One important observation here is that the bias term and the variance term are homogeneous ||312, so below we assume || 1 without loss of generality.
Here, we have Σ dovn T + I. In this case, we have
Σz.,, = ódz, + 1213
E2 = 1z.
1+ 10ZUUz 1
02 U2
(48)
4, = x,, Σ2, 2,2, 22, = -,,,, (Iz,
22 2, 1002, v2:B2;
1 + 02 1
8
8
X1,, 1 UZ 1|
22 12
(1-2,) 0)) B X,0,
(49)
1+ 81 1
So
(8 Z, 3
X+u = X diag,
Xv.
(50)
1 Uz 1
For 3 and v satisfies Assumption 1, utilizing Hoeffding's 's inequality, we know that with probability
1 -xxp( cxp(-dd),
8 ((1 2.) @ v) B
pô T B <C-d-(a-0)2
1 1+ 822 1
1 1+1(11 - p)1 ((])-)|[e1])
Therefore, denote := =ō p0pp--' T. (TT3/(1 + 8(1 - p)) and take union bound, we have the following holds with high probability that
pô v B
ttu-
u
S Ev
$ ca- Evll (a-
(51)
1+ 8(1 - p)
Therefore we have
xx (0, = İig + (xx))* U
EU'
Jev T +1
(52)
n + x
+ V (īI + x+u)
and by Lemma 2, we have that for a, b (B,0).
a Σt
(s)
s (1-c613
(53)
s ub
This leads to the fact that
113
14
dGB0(1)
(s)
5 On- (n-e111))
(54)
+ 14
X u 2
pô u.... T
dGg
C(nn C(n-((1))/) da(a-))/
(55)
1 8(1 p)
+ Hr
8
v T [II + xxxu)) - (
He dG (s) +
p N.TT 3 1 8(1 p) /
dG (s)
2
(56)
s P.
5 s
S C8(n (1-2)//2 + d (a-))
(57)
Combined with the result above, for 36)22. we finally get
Bx (,,
B G8 G G S cd
(58)
22
We next consider the variance term. For this Σ model, we have that
80zu,
1 Yai-a.Xaj-a EEi - E,z, Σz E2.1) = 1 Excroox
8ig +
(59)
1 + 8 811 2
23

---

Therefore, using Lemma 1, we have
d
(x
Σ
B,33
[X
+
Januy +xII (xTx))
f-=1
a=1
Σ
d à
3:33
Σx
Dan 0ii
6es + + Uzz 12
6riey
+ +nIr (x+x))*
11=1
a=l
d
7 ((x ((xT)+ 3 8f1x,tn Xaing + 582(1
( EX Jaa Σ B:8, x
Xaj Us2y
Σ
) ++E***
Ja Σ
1 8 Uz. 112
1m1
aml
+ Hill (xxx)
(((1-z)T(B00)) 1+ (v
[(xT)**X* am
+ (1 Z.) (.0 + ((XTx)*))
x)
(60)
Hoeffding's inequality gives
((1 - (B) v)) 2
63
<dd-(o-))/
1 1z
1 8+
as well as 1(1 Za)T (3 o 3) -l < da(a-))22 which holds uniformly for every S with Therefore, with high probability,
probability 1 -
Tr(Cov(8 - Tr ( (5+x) [ (++p+ 63"
1 5(1 - pT) )
- Tr Σ () d (a-1))
(61)
Note that by Lemma 2, we have the following with overwhelming probability that
Tr(E+E) S (s dHHa(s) + 8 ( d0
< <Cn <Ch-1/2t 11/24
(62)
(s+H.)² dHa(s)
To sum up, we have
Tr[Cov(B X
V(Ha, G G2 - (a-o1/
(63)
which is equivalent to
Vx ( V(Ha, Gāo .) S Cd y22
(64)
As a direct consequence, if we have Al + H, G + GB8 G&8.GBu + G G + G then by the definition of weak convergence, almost surely we have
G.e
R(,,3) + B(H, 88 6 ((") + ViH, R ,G").
(65)
y22
Here we are able tell the weak convergence of the (signed) measure Ge because the total variation
of this measure is given by |G = Σin 13, x (v. x.) S //,,(,,,.)) (v,
1,
so the sequence of G is also tight. See e.g. [74] for detailed argument.
B.4 Statement and proof of Theorem 3
Theorem 3 (General covariance model). For general Σ, assume is eigenvector of E with can be expressed as:
eigenvalue 17. The test risk R(B, (3) = E || B 12
Rx (B:3) X 1)BB
Σ 8,8 ) ΣX Ja waf + 2 x)))))
(2(3g)
(66)
24

---

Here we denote
Z, = Zoz = 1], 2 = [j] = 0}:
(67)
X' E Raxd
X X n) 2,
Z. xa =0; 0;
(68)
E Ez,))) wh E R", w diag. (1 Xay=0 (Eig Eiz, D Za
(69)
Proof. For the bias term, we have that
|[[() 31 | + X u u E R2 U, = x 222, 22,2 In our case, we consider to be an eigenvector of Σ with eigenvalue 1,. That is, we have
(70)
A2 Bz, + Dz
n33.-
(71)
Substituting 2z with n13z Σ2, Bz. in leads to 1
ui = 2, Σ 2, B2. - X1,2,3,.
(72)
Thus we can define v E RR, with uz = n1 x, 2. 2. Σ 22 .z.- Then we have
u = u' XB.
(73)
Plugging u' in the bias term, we have that
|E - 31 1| u -
(74)
That is, the term X + X canceled the bias term. Furthermore, we define X' e Raxd such that
x x ΣΣ 2, X 1,2 0. Then the bias term can be written
|E 31 = 1 1)
(75)
The variance term can be directly obtained due to Lemma 1.
Remark 1. Comparing the risk terms with standard ridge-less regression, we see that the masking introduces additional variance through the term:
Σ(XT)+XX waa
(76)
Here wil can be interpreted as the sum of covariances for pair-wise masked features in X conditioning on remaining unmasked features. As for the bias term, for eigenvectors ,3 with large eigenvalues 7)
replacing X with X' cancels the shrinkage effect in the projection matrix X X, which effectively reduces the bias term. In summary, if there are significant dependency between features in X resulting small w then a reduced risk of the masked regression compared to ordinary ridgeless regression is anticipated due to the bias term.
C Experimental details
C.1 Simulations
Across all simulations, input data matrices X Raax had rows sampled i.i.d. from (0 E). where the specific covariance Σ and dimensions (n, d) varied by experiment. Target values E R were generated as y = X,3 + e, with noise 6; ~ (0, 0.04). The ground-truth coefficient vector B E TRd was also experiment-speccffi. For each masking probability p E (0.05, 0.10, 00955), masked data X and targets ý were constructed as per our problem formulation. Regression estimates و were obtained either via the pseudo-inverse X ÿ (Figs. 1B-D, 3) or by solving (X X + ATa) X y with x = 10-6 (other figures), with negligible empirical difference between methods. Test risk was computed using 10n new test samples, and & was calculated with 50 repetitions per p through sampling different X from X. All error bars shown in this work indicate standard deviations.
25

---

Fig. 1A Here, E Id- The vector 33 was sampled from uniform distribution and normalized to || =. We evaluated two settings: 1) Overparamettrize: 2000, d 5n 10000 (? = 5); 2) Underparametrized:: n 4000, d = 0.5n 2000 (7- = 0.5). The theoretical risk was calculated using the formula in Theorem 1.
Figs. IB-D and 3 The covariance was Σ = La + óvv where E RR was a uniformly sampled vector (Fig. IB-D) controlled spike strength. Coefficients & with norm I were generated with B = cos Ov + sin Ou, where u x b (b a) v is the normalized component of uniformly sampled vector after removing its projection onto v. Parameters were n = 200, d = 5n = 1000. For theoretical risk calculation in these figures, A, was first obtained from ñ / A. = Tr(E(E A. "), where Σ (1 - p(1 p) diag(E) (E) and A 10- 8 Then, parameters (Ø3, Qu, 1, u, c) were calculated using Eq. (8), and the final risk via Corollary 1.
all-ones vector, both scaled to have norm I (Fig. 3), and e {1. 10, 100)
Fig. 1E The covariance Σ E IRdxd was constructed by one of three methods. In all cases, 500, = 5m- = 2500. For each Σ, f was an eigenvector of Σ corresponding to an eigenvalue at a specific quantile of its spectrum.
Uniform: Σ = Q
Xa)....,,aa@A Aa) with X ((1, 10) and Q random orthogonal matrix generated via QR decomposition of a randomly sampled Gaussian matrix.
Beta distributed: Similar 'uniform', except that the eigenvalues were sampled from Beta(2, 6) distribution then scaled to have min I and max 10.
Latent space model: Σ = Ia+ WWT, with W E R (g 0.5d) having i.i.d. Gaussian entries ((, (10 - /V/(vd+ + 2) This construction ensures that the eigenvalues of Σ approximately range from to 10.
Figs. 1G and 4 The covariance was Σ = = + WWT. W E Rdxq = QDR where Q Raxd and R E Kяxę were random orthogonal matrices (Haar distribution via and D E _dxq was a diagonal matrix with its non-zero
scipy scipy.stats. stats. grthho_grouprroooo entries set specified eigenvalue (e.g.. 100). Coefficients were B W (1, W W) for uniformly sampled vector 0 E RR. Parameters were n 100, d = 5000, q = 50.
Tables 5 and 9 The covariance matrices were the same those constructed for the Beta- distribute and latent space models in Fig. 1E. In the linear model, the implementation of R R2MAE is as follows. We first sample a row-wise masking ratio Pt ~ (Pmin: Pmax). )- This ratio p; is then used to sample the mask for each row of the data matrix. The resulting masked matrix Xsa is further used to construct X and calculate 3. Note that the removal probability for each row depends on 1 Pi, so the simplification of sample size as np from fixed MR settings is longer applicable. Therefore, to ensure a fair comparison, in the corresponding fixed MR settings, we sample each row with constant probability equal to the fixed MR. Finally, we test five random seeds for generating the model and masking matrices. Similar to previous experiments, the normalized test risks shown are the average values over 50 runs. For fixed MR settings, we tested MR values {0, 0.01, 0.02, ,..99). We confirmed that the MA results exactly match those of the fixed MR setting when Pmin = Pinax and the same random seed is used.
C.2 Evaluations on trained BERT and MAE models
For Fig. 1F, BERT fine-tuning accuracies at different masking ratios were obtained from [29] (data sourced from the GitHub repository). These reported accuracies were transformed follows. First, error rates were calculated as (1 accuracy). To normalize the y-intercepts, each curve was linearly extrapolated to masking ratio of zero using its values at masking ratios 0.15 and 0.3; each eurve was then vertically shifted so that its extrapolated value at 0% masking ratio became zero. Subsequently, each curve multiplied by unique scaling factor ensure all curves shared common slope for the line segment connecting their points at masking ratios 0.4 and 0.8. For Figs. 1G and 4, MAE fine-tuning and linear probing accuracies were directly obtained from [6].
C.3 MNIST
Dataset and model architecture. We used the standard MNIST dataset that consists of 60,000 training and 10,000 test grayscale images of handwritten digits at 28 x 28 pixels. We implemented
26

---

three-layer MLP with 784 dimensionn input (flattened images), first hidden layer with variable size (16 for underparameterrized setting, 512 for overparametrized setting), a second hidden layer of variable size (16, 32, 64, 256, or 1024 units), and 784-dimensional output layer. Each hidden layer uses ReLU activation with batch normalization, and the output layer uses sigmoid activation. The training objective is mean squared error (MSE) calculated on the masked pixels.
Training procedure. For each model setting, we tested masking ratios {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}. Models were trained for 15 epochs using Adam optimizer (learning rate 0.003, batch size 128). All experiments used PyTorch on NVIDIA A6000 GPU with fixed random seeds. Each experiment was finished in several minutes.
Evaluation. We performed digit classification for each pretrained model through linear probing. Specifically, we froze the first two layers of each pretrained model and trained only a new classification layer (mapping from the second hidden layer to 10 output classes) using cross-entropy loss. The linear probing classifier was trained for 15 epochs using Adam optimizer (learning rate 0.003).
C.4 CelebA
Dataset and model architecture. The CelebA dataset contains over 200,000 celebrity face images with 40 attribute annotations. Images were resized to 128 x 128 pixels using the official train- ing/validatioon/tst split. We used a U-Net with four downsampling blocks in the encoder, bottleneck, and four upsampling blocks in the decoder with skip connections. Each convolutional block contains two 3x3 convolutional layers with batch normalization and ReLU activation. The training objective is mean squared error (MSE) calculated on the masked pixels.
Training procedure. We tested base channel counts of {8, 16, 32) with masking ratios (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9). Each model was trained for 10 epochs using Adam optimizer (learning rate 0.001, batch size 256). All experiments used a NVIDIA 6000 GPU with fixed random seeds. Each experiment was finished in one hour.
Evaluation. We tested representation performance through inputting uncorrupted images and evaluating the reconstruction MSE of the output. For linear probing. we extracted the U-Net encoder and bottleneck, froze their weights, and trained classifier for the 40 CelebA attributes. The classifier included global average pooling. shared feature extraction layer (512 units with dropout), and 40 independent linear output heads. For each setting, the linear probing classifier was trained for 10 epochs using Adam optimizer (learning rate 0.001) and binary cross-entropy loss.
C.5 VIT MAE models
Dataset and model architecture. We used the biT- MAE model and the ImageNet- IK training split as pretraining data, following the MAE codebase [6].
Mask pretraining schemes. The patch tokens in the MAE input sequence are masked by one of the following strategies:
Fixed MR. A constant fraction of patch tokens in the sequence is masked. We tested MR values of (0.5, 0.75, 0.9). MR 0.75 the MAE default.
Dynamic MR. The masking ratio follows a linear decay: Pt
max {Prinns Praxx - Pmax Adecay ). Here, t represents the number of training epochs. We set Pmax 0.9, Pmii 0.6, and Adecay is chosen such that Pe linearly decays throughout the training. This mimics the scheme proposed in [34].
MDLM. For every mini-batch, a masking ratio p ~ (0, 1) is sampled. We use mean token mask loss for each MR (for all experiments), which is equivalent to w, = 1/k in the ELBO [54]. The implementation is very similar to the standard log-linear schedule of (()) [54].
MAMAE. For every mini-batch, a masking ratio p ~ 20(6.6 0.9) sampled.
Training procedure and evaluation. We trained all models for 150 epochs with 10 warmup epochs. All models were later fine-tuned on the ImageNet- IK training split and evaluated on the validation split. Other pretraining and fine-tuning configurations exactly follow the instructions in the MAE codebase. Each experiment was performed on one NVIDIA H100 GPU with the same fixed random seed, and each epoch took approximately 0.3 hours.
27

---

C.6 RoBERTa models
Dataset and model architecture. We used the HuggingFace RoBERTa-medium and RoBERTa-baa models, and the 10B token subset of Web sample-10BT, downloaded from HuggingFace) [75] as the training set. Although our implementation differs from [29] its training set and layer-norm design, we found the fine-tuning accuracies to be overall comparable.
Mask pretraining schemes. The tokens in the input sequence are masked by one of the following strategies:
Fixed MR. constant fraction p of tokens in the sequence masked. We tested MR values of {0.15, 0.4}. MR 0.15 is the MLM default, and an MR of 0.4 is recommended in [29].
Dynamic MR. The masking ratio follows a linear decay: Pt
max {Pmin Pmax - Pmax Adecay ). Here, represents the number of training steps. We set fmax 0.4, Pmin 0.15, and Xdecay is chosen such that Pr linearly decays throughout the training.
MDLM. For every mini-batch, masking ratio p 4(0,, 1) is sampled.
R'MAE. For every mini-batch, a masking ratio p ~ 410.15, is sampled.
Training procedure and evaluation. RoBERTa-base follows the HuggingFace default setting, while RoBERTa-medium overrides the following parameters: vocab_size-5000 50265, tidden_size=512 num hum_hidden__m 1ayddddn_ num 8 h intermediate_siiee saieeeeeesiee_e_______ We used Adam optimizer, a max sequence length of 128, an effective batch size of 2048, weight decay of 0.01, warmup ratio of 0.03, learning rate of models, and default linear learning rate decay. Fine-tuning
embeddings=514, max_possttion_eemodddnngss
7e-4/3e-4 for the was performed on the GLUE datasets (MNLI, QQP, SST-2, QNLI) for 5 epochs with learning rate of 2e-5 and a batch size of 32. The average accuracy of three funing runs was reported as the final accuracy, following [29]. Each experiment was performed on one NVIDIA H100 GPU with fixed random seeds, and finished in one day.
C.7 DNA sequence models
Dataset and model architecture. We adopted the GPN-MSA [16] framework, which a 12-layer transformer model with 12 attention heads per transformer layer. Benegas et al. [16] curated a training set comprising multiple-sequence alignment (MSA) from human DNA and 89 other species, with careful filtering and biological considerations; please refer to [16] for more details on the model and the training set. Each training sample consists 128-base pair (bp) window of human genome and its corresponding MSA, and the pretraining task is to predict the token (A/C/G/T) in the masked locations of human DNA, based on the input of other unmasked locations and the auxiliary MSA information.
Mask pretraining schemes. Before encoding, the raw input X corrupted as mask by one of the following strategies:
Fixed MR. constant fraction p of base pairs in the sequence is masked. p 15% corresponds to GPN-MSA default [16].
Dynamic MR. Same as the fixed MR case, except that the masking ratio follows linear decay Pt = max {Pmins Pmax Pmax Adecay }. Here represents the number of training
steps. We set Pmax 0.30, Pmin = 0.15 and choose À such that Pr linearly decays throughout training.
MDLM. For every binc- a masking ratio p ~ U(O, 1) is sampled.
RºMAE. For every mini-batch, a masking ratio p ~ L00005, 0.3) is sampled.
R'MAE + Dynamic MR. For every mini-batch, we sample the masking ratio p (pt. 0.3), where P: max {Pmin. Pmax Piax t Addecay J. To match masking ratio expec- tation with the Dynamic MR setting, we set Pmax = 0.30, Pmin = 0.00 and choose X such that pt linearly decays throughout training. Although we applied early stopping criteria based on validation loss to all models, only this model triggered an early stop, occurring at masking ratio of p: 0.10.
28

---

For CL approaches, we first describe their combinations with RAMAE, as their standalone implemen- tation results from straightforward simplification of R&MAE- + CL schemes.
R + CL (k=0). We implement token-wise MI.P layer with hidden space [128, 128] and ReLU activation that projects the transformer-leareds masked location representations to output P E RR"masxi xi Here 1 = 10 represents the length of the pre-defined mask ratio vector. We further implement row-wise projection layer with sigmoid activation plus 0.5 to obtain strictly positive entries (and to improve optimization stability), which we term P RRxi Finally, we apply K 10 iterations of the non-square Sinkhorn operator described in Appendix A.3: (y) = Sinkhorn (K (a))) For every
we now sample the mask ratio in [0.05, ..., 0.30] with length 10, and select the column of M (v) based on the selected mask ratio index. Then we multiply this column of M (y) to the element-wiss loss for the masked locations. The newly implemented layers are optimized together with the main reconstruction model, after being fixed for 5000 initial training steps.
R CL. Apart from the additional components described above, employed a dynamic multiplier to the gradient received by these additionally implemented layers, decreasing from to -1 throughout its training with a linear decay.
CL. The setting is effectively implemented by always selecting one fixed masking ratio (15%) in the MAM + CL setting.
Training procedure. Models were trained for 30000 steps using the defaults in [16] with Adam optimizer, learning rate le-4, and effective batch size 2048. All experiments used PyTorch on NVIDIA 6000 Ada GPUs with fixed random seeds. Each experiment takes 6.5 hours to complete.
Evaluation. We evaluated different models' performance in zero-shot predictions of pathological missense (Clinvar pathologic versus GnomAD common) and regulatory (OMIM pathologic versus GnomAD common) variants [60-62]. The inference was performed by vep inn-nnffeenee imple- mented in [16]. The evaluation sets as well as the scores of alternative models shown in Table I [55-59, 39] are provided by the original GPN-MSA work Huggingface. Partial AUROC (max FPR 0.001) was used in the OMIM evaluation to account for high imbalance of positive and negative classes.
C.8 Single-cell gene expression models
Dataset. We employed the Human Lung Cell Atlas dataset [63] and human brain MTG SEA-AD dataset [64]. Both datasets were downloaded from the CellXGene portal and were subsetted to 5000 highly variable genes (HVGs) using the default procedure in Scanpy [76]. For the HLCA dataset, we further filtered out cells that have fewer than 20 of these HVGs. After preprocessing, these datasets have 2161082 and 1378211 cells respectively, along with metadata of fine-grained cell types, Oocease/Alzheimas status (Alzheimer's 's Disease Neuropathologic Change, ADNC), and age labels.
Model architecture. For all settings that require pretraining from scratch, we implemented a 5 5-layer MLP encoder-decoder based architecture described as follows. The model receives corrupted count matrix X mask
R Peella Kinpus as input, where Ninput 5000:
Latent encoder E, (3-layer-MLP with Rhidden units and ReLU activations, the final layer being a linear projection from Whidden to Mlasent maps logarithmically transformed (cor- rupted) counts together with the dataset batch covariate to the latent space.
Decoder D (2-layer-MLP with Whidden units and ReLU activations, the final layer being projection from Thidden to Nimput with softmax activation) receives the embedding, observed library size, and batch covariates and produces regative-binomia parameters (My, (,) for every gene [77]. Batch normalizations are used in both encoders and decoders.
Objective. The objective is defined as the average negative reconstruction likelihood for the masked genes in the input data: = mask Zgemask log NB(x, (g 0,).
In all implemented models, we set Whidden 2000 and Mlatent 1000. These are much higher values than those of typical SI models [78, 77] and are comparable to latent space sizes in recent single-cell foundation models [17, 21].
29

---

Mask pretraining schemes. Before encoding. the raw count matrix X is converted to the masked matrix X, mask by one of following strategies:
Fixed MR. A constant fraction p of gene columns in the count matrix sampled once per mini-batch is replaced by zero.
Dynamic MR. Same as the fixed MR case, except that the masking ratio follows linear decay Pt max {Pmiu Pmax Pmax Adecay ). Here represents the number of training steps. Here Pmax = 0.5 and Pmin = 0.1, with À an dataset-specific parameter to enforce linear decay throughout training (1/300000 for HICCA, 1/150000 for SEA-AD).
MDLM. For every mini-batch, masking ratio p 4(0, 1) is sampled.
RAMAE. For every mini-batch, a masking ratio P (0011, 0.5) is sampled.
RºMAE + Dynamic MR. For every mini-batch, we sample the masking ratio p (Pt. Pumax), where Pt max (Pmin, Pinxx Pmax & Adecay) with the same parameter selections as the Dynamic MR setting.
For CL. approaches, we first describe their combinations with R R*MAF, as their standalone implemen- tation results from straightforward simplification of R + CL schemes.
R + CL (k=0). We implement an ML.P layer that projects the original count matrix (we pass transformed version, log (Xx /0) + 1) in practice) to two-layer MLP with hidden dims [128, 256] and ReLU activations, reshaped to output P R Tinpus x64 We further implement a row-wise projection layer with sigmoid activation plus 1e-9 to obtain strictly positive entries, which we term P E Rrispu x? Finally, we apply K = 4 iterations of the non-square Sinkhorn operator so that each row sums to and each column to Rinput: (v) = Sinkhorn (K) (P()). For every mini-batch, we now sample the mask ratio in the length-/ vector 10.10, 0.15, ..., 0.50], and select the column of (y) based on the sampled mask ratio index. Then the element-wise negative log-likelihood multiplied by this column of Myy as the training objective. The newly implemented layers are optimized together after being fixed for 5000 training steps.
MaMAF CL. Apart from the settings in R CL (k=0), employed dynamic multiplier to the gradient received by these additionally implemented layers, decreasing from 1 to -1 throughout its training with linear decay from 30000 to 120000 training steps.
CL (k=0), CL. These settings are effectively implemented by always selecting one fixed masking ratio in the above R MAE CL (k=0) and R + CL. settings respectively.
Vc In this setting, we no longer mask the data, and instead formulate variational posteriors and train the model using the evidence lower bound (ELBO) objective [78, 77].
Training procedure. Models were trained for 50 epochs using Adam optimizer (learning rate 1e-3, weight decay lc-4, batch size 400). 90% of data were selected as the training set and the remaining 10% was set as the validation set. All experiments used PyTorch on an NVIDIA 6000 GPU with fixed random seeds. Each experiment takes 1-2 hours to complete.
Evaluation. We evaluated different model embeddings' performance in identifying key metadata across donors through linear probing. Apart from previously described models, we used pretrained scGPT and UCE models output zero-shot embeddings of the preprocessed datasets [17, 21J. For scGPT, the dataset is further log-normalized according to the instructions [17]. We also evaluated the linear probing performance of the log normalized expression itself. We selected all cells from randomly sampled [0.6 x Total donors] donors in the dataset as the training set, and the remaining cells as the test set. Reference control donors from the SEA-AD datasets were removed. We further removed data without corresponding metadata for the respective regression/clasificttioo tasks. Specifically, we removed cells whose cell type was labeled as Unknown for the fine-grained cell state classification task (HLCA), and removed cell types that do not contain Alzheimer-specin subtypes in the SEA-AD dataset; we removed cells without age labels (HLCA) or containing broad categories instead of exact ages (SEA-AD) for the age regression task. After each removal, we subset the training and test sets so that each donor comprises a maximum of 200000 in the set 1000000 Paoone
"Atooor numbers for cell type classification task in SEA-AD), to further balance cell numbers across donors. The training and test sets for cell type classification in HLCA is obtained by removal after subsetting.
30

---

We performed ridge regression for regression tasks and logistic regression for classification tasks. GridSearchCV was utilized for selecting the best regularization parameters (mp 1ogspace 80,8, 20) to minimize regression MSE, np. logspace (-6, - 10) to maximize classification balanced accu- racy), and the training samples were separated by donors for five-fold cross-validation.
Finally, for classification tasks, we evaluated balanced accuracy and macro Fl score on the test set. For regression tasks, we evaluated both cell and donor level (obtained through averaging cell-level score per donor) Spearman т. The only exception is for the ADNC classification, where all methods perform poorly in terms of balanced accuracy and macro Fl scores. Therefore, we instead evaluated macro AUROC on both cell and donor levels.
For those models trained from scratch, additionally evaluated their performance in reconstructing randomly masked genes in the pretraining validation set. 'he masking ratios evaluated are [0. 1, 0.2, 0.3, 0.5, 0.7]. We calculated Pearson between the model output By and log-normalized gene expression per cell, and then averaged the Pearson r over all cells in the validation set (which stays the same across all methods tested).
D Appendix Figures
10000
1.5
15
coold 1/me
1.0
conti (onjA19-- 8-555
conti v/-5.75
coolS Vi+
Exx 10-1
0.55 0252 C.75
0/0 0
0.75 0
0.25 0.0
0.75
Voeddngg
Figure The spiked covariance model ( 800 +D, where 1/vd. Plots of mean simulation
test risk and prediction magnitude ratio (IE) || 2 and cos(33, v) = 0) over samples the spiked covariance model against the masking ratio p.
x [
[| | | between cos(B), v) =
200, 7 = 5.
Latent model, another random seed
MAE linear probing accuracy (He et al., 2022)
1.75
Imageket valicatin accrrsc
1.50
0.6
1.25
Normaltedd
0.5
1.00
0.75
0.50
0.3
0.25
0.2
0.00
0.2
1.0
Masking
Figure 4: Plots of the normalized test risk of a latent space model and MAE linear probing accuracy on ImageNet [6] against masking ratio. The covariance Σ for the latent space model another sample from the same generative process as in Fig. 1G. Note that there is slight horizonal shift between the two curves.
31

---

CelebA Attribute Classification Accuracy Median Scaled (n 8)
Medlan-Scoll Scall ed Accuracy (value/medii an)
Masking
-1.0
0.3 69
Wary Hair
Black Pair-
Blond Hair
Archod, Eyebrows
Big Nose-
Wearing SaeringgNNee Necitle
Oval OvaLFacc Face-
Bongs
Witaiin maaring Hat-
Pale Skin
Eyes
Gray Hair
Rony Cheekes
Streight Hair-
Chuiby
Eyeglasses
Hairline
Wearing Neckiace
Scleburns
Narrow Eyer.
Mustache
Geatee Big Lipe
No Blury
Wearing Slightly, Open
High, Cheekioonss
Smiling Male
Wearinng Uppsiik Attractive
Heavy Makeup
$,e, Coockcck She
Biishy Enshy_Fyybrod Eyetro
Double
Attribute (hierarchically clustered by scaien-scaled values)
Figure Median scaled accuracy of U-Net models (hase channel 8) on CeleBA classification tasks.
CelebA Attribute Classification Accuracy Median Scaled (n 16)
Medor-Scalee Accuracy (vllueieedian)
B3
Mouth Slightly Open
Smiling
High Chee Chee locnees
Attractive Hair.
Heavy, Makeuup
Lipstick
Black Hair-
Brovin Hair-
Pale Blin-
Double Chin-
Blurry
Big Lip--
Wearing Nedoace-- e-
Sideburee-
NairoN Leves-
Rosy Hairline
Gray Hair- Wearingg Hat
Wearing Necktle- e-
5. 3. Clock Shadow- Pointy Pointy_Nose Nose- We Wearing, Earnings
Big Nese-
Ovel face
Bangs- Youing
Arched Eyebrowt: Blond Hair-
Eveglesee
Wearing
Buahy
Attribute (hicrorchically clustered by median-scales values)
Figure 6: Median scaled accuracy of U-Net models (base channel = 16) on CeleBA classification tasks.
32

---

CelebA Attribute Classification Accuracy Median Scaled (n 32)
Med Med and---sslleed Accuirity (vailsimmedion
1.2
-10
Black Hair-
Gray Hair-
Receding Hairiiee
Wearing, slace-
Sidebums
Narrow Soorrow_Frt Eyes-
Mustache-
Goaten--
Big Lips
Resy Resy.chee Cheeks-
Bushy Syebrows
Wearing_/vee Nacktee-
Wearing Hat-
s.o Clock Shadow-
Strighht nightt Hair-
Pointy Nase
Beard.
Wearing Eariins
Arched syebrows.
Brown Hair-
Bangs
Oval FvaL Face-
Bags Under laves.
Big Mose-
Blend Hair
Mouth Slightly Open High Chookbones
Smilings
Waryy lair
Male
Wearing Lipsticke
Attractive
Heavy, Maklup
Gyegl
Attribute (hierarchically clustered by median-scaled
Figure 7: Median scaled accuracy of U-Net models (base channel = 32) on CeleBA classification tasks.
33

---

Comparing Different Model Configurations the Same Random Mask Test Set
with
Masked
7
7
7
7
7
7
Model with
Model with
Model with
Model with mask
Model with
Original
2
2
2
2
2
2
Model with mask ratib 0.1
Model with mask rabiow
Model with mask radio 0.5
Model with mask ratio 0.7
Model with mask ratlo 0.9
Original [Label: 1) Masked (Ratio: 0.20)
Model
with mask rabio 0.3
Model mask ratio= 0.5
Medel with maski ratio 0.7
Model with mask ratio 0.9
Original (Label: 0)
Kasked (Ratio: 0.75)
mask ratip- 0.1
with 4
Original [Label:
Masked (Ratio:
mask rabios
4
ql
4
4
9
(256. 256)
Figure 8: Comparison of overparametrized MLP reconstructions MNIST data across different training mask ratios. Original digits (first column) and their masked versions (second column) are followed by reconstructions from models with identical architecture but varying mask ratios during training. The second layer hidden dim 256 for all models.
34

---

Comparing Different Model Configurations the Same Random Mask Test Set
Original
Masked iRatio: 0.79)
Mask: 0.1
Mask: 0.3
Mask; 0.5
Mask: 0.7
Mosi: 0.9
Original
Masked |Ratio: 0.30)
Mask: 0.1
Mask: 0.3
Mask: 0.5
Mask: 0.7
Mesk:
Original
Masked IRatio: 59)
Mask: 0.3
Mask: 0.5
Mask:
Masi:
Original
(Ratlo: 0.18)
Mask:
Original
Modal Base Channels: (16, 16, 16, 16, 16)
Figure 9: Comparison of U-net reconstructions on CeleBA across different training mask ratios. Orig- inal images (first column) and their masked versions (second column) followed by reconstructions from models with identical architecture but varying mask ratios during training. All models have base channel 16.
35

---

E
Appendix Tables
Table 6: Comparison for different single-cell gene expression models trained Human Lung
Cell Atlas (HLCA). BAcc, Balanced Accuracy. For each specific task, pretraining scheme metrics outperforming optimal fixed masking ratio settings labeled red.
Cell state
Disease
Age Spearman
Avg performance
Methods
BAcc.
mo
BAcс.
FI marroo
Cell
Donor
Score
Rank
Normalized exp.
0.834
0.774
0.675
0.489
0.470
0.574
0.636
12.50
scGPT (Lung)
0.813
0.717
0.624
0.401
0.429
0.523
0.584
15.67
scGPT (All)
0.834
0.711
0.629
0,403
0.438
0.500
0,586
15,00
scGPT (CP)
0.816
0.696
0.613
0.389
0.431
0.521
0.578
16.67
UCE (4L)
0.808
0.702
0.631
0.417
0.436
0.518
0.585
15.50
UCB (33L)
0.800
0.699
0.619
0.419
0.447
0.540
0.587
15.50
scVI
0.897
0.804
0.767
0.626
0.556
0.618
0.711
9.67
MAE (MR 25%)
0.908
0.830
0.834
0.604
0.586
0.623
0.731
6,50
MR 10%
0.915
0.802
0.851
0.635
0.582
0.609
0.732
5.67
MR 50%
0.909
0.833
0.837
0.604
0.587
0.601
0.729
6.33
MDLM
0.903
0.806
0.829
0.560
0.577
0.622
0.716
8.83
Dynamic MR
0.919
0.829
0.850
0.651
0.571
0.597
0.736
5.00
CL-MAE
0.907
0.825
0.843
0.635
0.589
0.648
0.741
4,50
CL-MAE (A: 0)
0.801
0.667
0.773
0.493
0.530
0.563
0.638
13.83
R MAM (Ours)
0.915
0.812
0.853
0.651
0.595
0.641
0.744
Dynamic MR
0.914
0.842
0.835
0.597
0.616
0.658
0.744
4.17
CL
0.911
0.817
0.837
0.618
0.572
0,619
0.729
6.67
CL (k =
0.911
0.805
0.840
0.630
0.590
0.646
0.737
5.00
Table 7: Comparison of random masking reconstruction Pearson across different single-cell gene expression models trained on brain MTG SEA-AD dataset. MR, Masking Ratio.
Methods
MR 10%
MR 20%
MR 30%
MR 50%
MR 70%
SeV1
0.834
0.834
0.829
0.815
0.781
MAE (MR 25%)
0.843
0.847
0.846
0.840
0.819
MAE (MR 10%)
0,842
0.844
0,841
0.832
0.797
MAE (MR 50%)
0,841
0.845
0.843
0.842
0.830
MDLM
0.840
0.844
0.842
0.840
0.827
Dynamic MR
0.842
0.845
0.842
0.834
0.800
CL-MAE
0.841
0.844
0.845
0.838
0.817
MAMAE (Ours)
0.846
0.847
0,845
0.842
0.826
Dynamic MR
0.844
0.847
0.846
0.841
0.824
CL
0.836
0.839
0.839
0.836
0.825
CL (k: -0
0.844
0.846
0.845
0.840
0.823
36

---

Table 8: Comparison of random masking reconstruction Pearson across different single-cell gene
expression models trained on Human Lung Cell Atlas (HICA)). MR, Masking Ratio.
Methods
MR 10%
MR 20%
MR 30%
MR 50%
MR 70%
ScVI
0.712
0.744
0.746
0.733
0.698
MAE (MR 25%)
0.743
0.774
0.780
0.777
0.747
MAE (MR 10%)
0.733
0.770
0.777
0.769
0.733
MAE (MR 50%)
0.744
0.770
0.779
0.781
0.760
MDLM
0.729
0.760
0.766
0.764
0.739
Dynamic MR
0.742
0.772
0.777
0.772
0.737
CL-MAE
0.736
0.768
0.776
0.776
0.756
CL-MAE (ks
0.174
0.131
0.109
0.090
0,086
R MAE (Ours)
0.746
0.773
0.782
0.778
0.752
Dynamic MR
0.743
0.774
0.781
0.779
0.753
CL
0.731
0.763
0.772
0.770
0.746
CL 00)
0.733
0.767
0.777
0.770
0.742
Table 9: Normalized test risk of MAM (MR range 0.4-0.5) against optimal fixed MR and mean MR settings across different random seeds for Beta covariance and latent space models. The ground truth signal / is set to be the 10th quantile eigenvector of covariance Σ in all cases. 200, 7 5.
Beta Covariance Model
Latent Space Model
Sced
Best MR
Min Risk
MR 45%
MAMAE
Best MR
Min Risk
MR 45%
R R'MAE
2 0.43
0,859
0.865
0.855
0.45
0.826
0.826
0.823
12
0.43
0,863
0,865
0.859
0.42
0,832
0.837
0.830
22
0.54
0.862
0.898
0.890
0.37
0.848
0.851
0.842
32
0.43
0.817
0.822
0.814
0.33
0.852
0.865
0.859
42
0.36
0.817
0.830
0.819
0.43
0.814
0.818
0.806
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction do accurately reflect the paper's ' contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have specified limitations of the work with limitation section at the final of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
38

---

Answer:
[Yes]
Justification:
We have provided full set assumptions and correct proofs in our work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We do fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that affects the main claims and/or conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
39

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Our code and instructions are provided at attps://githun com/ MingzeDong/r2nn Detailed instructions are also provided in Appendix.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
Pllyc)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. http/uulic/ggeee weaiiccgguee//e/eee /ilee//CddSubbiie wwwaniisssoooooiiiiiiiioooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
We do specify all the training and test details necessary to understand the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report eтror bars suitably and correctly detined or other appropriate information about the statistical significance of the experiments. All error bars this work represent standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
40

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources needed to repro- duce the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurij: http:///eenuippcccpu.eiieee cc/public//etteeeeeeeeeeeeeeeeee Cerriic///iiiieeeeee htte/Ehhiiiisuiieeeeeee csGuidelineeees
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We described potential socictal impacts of our work in the "Broader impacts" section. We do not see particular negative societal impacts in work.
Guidelines:
41

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This work does not present data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credited assets. The license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
42

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasats has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
43

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our work does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https /topa://neuute http//rpps.ccccooeeeeeeeeeeeeeee cccc/oooeeeeecc//222///1/1. for what should or should not be described.
44


Reviews:
[{"review_id": "QgKAbMAu2w", "reviewer": "Reviewer_RbST", "summary": "Here, the authors propose a theoretical analysis that leads to a masking strategy for masked language models called \"R^2MAE\", and demonstrate its effectiveness in self-supervised representation learning for DNA and single-cell.", "questions": "(See questions in strengths/weaknesses)", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "This paper presents a generally strong theoretical analysis, which essentially argues that the success of the masked-based pretraining objective can be explained as a trade-off between minimizing bias and optimizing risk (i.e. ensuring that for each feature, the model learns which other features can be  used to reconstruct the feature, while minimizing overfitting). They do convincing empirical experiments to validate the key predictions of their theoretical analysis, showing that the masked-based pretraining is only beneficial in the overparameterized regime, and that optimal masking ratio differs by evaluation task.\n\nBased upon these insights, the authors propose a new masking strategy (R^2MAE), which contrary to previous adaptive masking proposals, simply samples masking ratio during pre-training at uniform, and demonstrate the effectiveness of this strategy on two different biological domains, DNA sequences and single-cell gene expression. These empirical results are strong, demonstrating that their proposed method can outperform even SOTA proposals on at least some kinds of tasks in their respective domains, on standardized benchmarks at least, and critically, frequently over fixed/dynamic masking ratio proposals.\n\nEven as the empirical result is strong, I'm struggling to connect their masking strategy proposal to the theory presented - I understand that different masking ratios have different bias-risk trade-offs, but how does sampling masking ratio at uniform necessarily lead to a better final model? By intuition, I'd expect this to lead to a model that's an \"expectation\" over all of these masking ratios (including ones with high bias/risk), so I'm not fully understanding why this is effective.", "was_revised": true, "final_justification": "The rebuttal by the reviewers do a good job of demonstrating the generality of their method, as well as show that their proposed method aligns well with the optimal risk. Although the connection between their theory and their proposed masking scheme still remains unclear to me, these are strong empirical results. My original score for this paper was high to begin with, so I will maintain my score.", "rebuttal": "Thank you very much for your comments. Our rebuttal is organized as follows. We will first present new results that address general feedback from all reviewers. They include new experiments on vision and language models and a strengthened connection between R$^2$MAE and our theoretical framework. We will next provide a point-by-point response to specific concerns.\n\n### **New Results 1: Experiments of R$^2$MAE on Vision and Language Models**\n\nIn general, reviewers appreciated our empirical study but expressed curiosity about whether R$^2$MAE would achieve SOTA performance in more established domains like vision and language modeling. We were fortunate to gain access to additional computational resources during the rebuttal period to perform these experiments.\n\n**Vision:** Our implementations closely follow established practices from the vision MAE work [1]. We implemented different mask ratio settings on the ViT-base MAE model using their official Pytorch codebase. The settings include: 1. Default MAE (constant 0.75 mask ratio, MR); 2. R$^2$MAE (p ~ U(0.6, 0.9)); 3. Dynamic MR [3] (linearly decreasing MR from 0.9 to 0.6); 4. High (0.9) and low (0.5) fixed MR baselines. Given the time limit, all models were trained for 150 epochs. While this is shorter than the 800 epochs in [1], their analysis shows predictable improvements with longer training schedules. Therefore, while our absolute accuracies may be suboptimal, we expect the relative performance across settings to be comparable and the conclusion to hold if trained longer. All other hyperparameters follow the defaults in [1]. Models were then fine-tuned for classification for 100 epochs, following [1].\n\nOur results below show R$^2$MAE achieves the best performance in both top-1 and top-5 accuracy. We found that R$^2$MAE introduces fluctuations in the pretraining loss, likely due to variable sequence lengths since MAE only encodes unmasked tokens. This presents a potential area for even further improvement.\n\n| ViT-base | Acc@1 | Acc@5 |\n|:---|:---:|:---:|\n| MAE default (MR 0.75) | 81.97 | 96.02 |\n| fixed MR 0.9 | 81.20 | 95.68 |\n| fixed MR 0.5 | 81.80 | 95.93 |\n| Dynamic MR | 81.97 | 96.04 |\n| R$^2$MAE | **82.00** | **96.05** |\n\n**Language:** We trained RoBERTa-base and RoBERTa-medium models on the FineWeb dataset for 10B tokens (max seq length 128, comparable to [2]) and fine-tuned them on GLUE benchmarks (MNLI, QQP, SST-2, QNLI). We evaluated: 1. Default MLM (MR 0.15); 2. R$^2$MAE (p ~ U(0.15, 0.4)); 3. Dynamic MR [3] (0.4 to 0.15); 4. MLM with a fixed 0.4 MR. Fine-tuning accuracies are comparable to those in [2]. The results show R$^2$MAE achieves the best overall rank, with a more pronounced advantage in the larger RoBERTa-Base model.\n\n| RoBERTa-Medium (52M) | MNLI | QQP | SST-2 | QNLI | Mean Rank |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| MLM default (MR 0.15) | 80.8 | 89.8 | 89.9 | 86.3 | 3.25 |\n| Fixed MR 0.4 | 80.3 | 89.7 | 90.1 | 86.6 | 3.50 |\n| Dynamic MR | 80.8 | **90.1** | 90.5 | **87.1** | 1.50 |\n| R$^2$MAE | **80.9** | **90.1** | **90.6** | 86.7 | **1.25** |\n\n| RoBERTa-Base (125M) | MNLI | QQP | SST-2 | QNLI | Mean Rank |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| MLM default (MR 0.15) | 81.5 | 90.7 | 91.7 | 87.8 | 3.00 |\n| Fixed MR 0.4 | 81.7 | 90.7 | 91.2 | 88.5 | 3.00 |\n| Dynamic MR | 81.8 | 90.7 | 91.4 | **89.1** | 2.00 |\n| R$^2$MAE | **81.9** | **90.8** | **91.9** | 88.6 | **1.25** |\n\n**Note:** In these fine-tuning evaluations, the advantage of R$^2$MAE appears smaller than in our linear probing or zero-shot evaluations. This is expected, as the identical fine-tuning pipeline can reduce differences from pre-training. Nevertheless, the consistent advantage of R$^2$MAE across tasks and domains strongly supports its effectiveness and generalizability.\n\n### **New Results 2: Explicit Connection between R$^2$MAE and the High-Dimensional Linear Regression Framework**\n\nInspired by reviewer comments, we evaluated R$^2$MAE within our linear model framework. Specifically, we simulated the latent space model from Fig. 1E. Intriguingly, we observed that for features of different strengths, R$^2$MAE's risk tends to align with the *optimal* risk, with its \"effective masking ratio\" decreasing as feature strength decays.\n\n| Normalized test risk | p=0.45 | p=0.5 | p=0.55 | p=0.6 | R$^2$MAE, p~U(0.45,0.6) | Effective p of R$^2$MAE |\n|---|---|---|---|---|---|---|\n| β = 1st Eig | 0.687 | 0.671 | 0.666 | 0.671 | 0.665 | ~0.55 |\n| β = 2nd Eig | 0.751 | 0.748 | 0.752 | 0.772 | 0.750 | 0.5-0.55 |\n| β = 3rd Eig | 0.839 | 0.842 | 0.849 | 0.872 | 0.842 | ~0.5 |\n| β = 4th Eig | 0.880 | 0.889 | 0.898 | 0.929 | 0.888 | 0.45-0.5 |\n\n\nOur new results support the intuition that R$^2$MAE achieves a “feature-adaptive” masking ratio that approximates dataset-specific optimal masking ratios, allowing it to outperform any fixed ratio in settings with multi-scale features. This is validated by real data (Tables 4 and 5), where R$^2$MAE achieves near-optimal reconstruction across its entire masking range (0.1-0.5), while fixed-rate settings only well cover a smaller range. This result aligns with the robust advantage of R$^2$MAE and strengthens the value of our work.\n\n### **Point-by-point response to other specific concerns**\n\nThe only concern from the reviewer is the connection between R²MAE and the linear model. We are grateful for the feedback which inspired the second part of our new results. Here, we highlight the difference between the R²MAE setting and the fixed mask ratio setting in the linear model framework. In the linear model, R²MAE translates to first sampling a row-wise masking ratio $p_i \\sim U(p_{\\text{min}}, p_{\\text{max}})$ and then using it to sample each row of the masking matrix $Z$. This introduces a subtle difference in the effective sample matrix $X_{sub}$, as the sampling probability for each row is no longer a constant ($p$) but depends on the specific value of $p_i$. Due to this difference, deriving the closed-form solution for the test risk becomes very challenging and remains a topic for future research. At the very least, this complexity suggests that the test risk of R²MAE is not identical to that of any fixed masking ratio. We further empirically validated this, as shown in the second part of our new results. We believe these results effectively address the concern. Please also refer to the first part of new results which contains additional validation in vision and language models that improve our work. Please let us know if there are any additional concerns.\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" CVPR. 2022.\n[2] Wettig, Alexander, et al. \"Should You Mask 15% in Masked Language Modeling?.\" EACL. 2023.\n[3] Ankner, Zachary, et al. \"Dynamic Masking Rate Schedules for MLM Pretraining.\" EACL. 2024."}, {"review_id": "ehInfqcICW", "reviewer": "Reviewer_yjuN", "summary": "This paper presents a novel theoretical framework to understand mask-based pretraining through the lens of high-dimensional ridgeless linear regression. The authors show that both qualitative and quantitative behaviors of masked autoencoding across domains (vision, language, biology) can be recapitulated through this simplified yet analytically tractable model. Based on insights from this theory, they propose a simple yet effective pretraining strategy called R2MAE (Randomly Random Mask AutoEncoding), which applies uniformly sampled masking ratios during pretraining. Empirical results on DNA sequence and single-cell gene expression data demonstrate that R2MAE outperforms existing state-of-the-art approaches across multiple metrics.", "questions": "1. Have you considered incorporating R2MAE into more general-purpose pretrained models (e.g., ViT or LLaMA) to evaluate generalization beyond biology?\n\n2. My understanding is that R²MAE randomly samples a different masking ratio for each input from a uniform distribution—is this correct? If so, based on your theoretical analysis, is it possible to determine an optimal masking ratio for each individual sample, rather than relying on random sampling?", "limitations": "The main limitation lies in the strong assumption of a linear model and the minimal interaction between features, which may oversimplify the complex dependencies present in real-world deep networks.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Theoretical Contributions: The use of high-dimensional linear models to explain masking behaviors is theoretically grounded. The analysis reveals key phenomena such as optimal masking ratios, phase transitions, and performance plateaus. Closed-form expressions for test risk under various covariance structures (isotropic, spiked, general) are novel and informative.\n\n2. Clear Empirical Validation: The authors validate theoretical predictions on diverse architectures (MLPs, CNNs, Transformers) and domains (vision, language, biology). Experiments on real biological data (DNA, single-cell) show that theoretical predictions align well with empirical trends.\n\n3. Practical Impact via R2MAE: Despite its simplicity, R2MAE consistently improves performance over standard and dynamic masking schemes in biological settings. The strategy is easy to implement and generalizable.\n\nWeaknesses:\n1. Theoretical-Algorithmic Gap: While this work provides novel insights into masked pretraining, some of the underlying technical assumptions are quite strong. In particular, the reliance on a linear model and simplified data distribution assumptions may limit the direct applicability of the theoretical findings to practical deep learning settings involving nonlinear architectures and complex data structures.\n\n2.  Simplified Feature Independence in $\\beta$ Estimation:  The assumption that each regression target ($\\beta_i$) is estimated independently or with minimal interaction may ignore feature co-adaptation, which is critical in practice.\n\n3. Minimal Comparison on General Datasets: Most empirical validations beyond theory focus on DNA and single-cell data. It would strengthen the paper to include results on standard vision/language benchmarks with R2MAE (e.g., ImageNet, GLUE).", "was_revised": true, "final_justification": "During the rebuttal, the authors address many of my concerns and I increased my score.", "rebuttal": "Thank you very much for your comments. Our rebuttal is organized as follows. We will first present new results that address general feedback from all reviewers. They include new experiments on vision and language models and a strengthened connection between R$^2$MAE and our theoretical framework. We will next provide a point-by-point response to specific concerns.\n\n### **New Results 1: Experiments of R$^2$MAE on Vision and Language Models**\n\nIn general, reviewers appreciated our empirical study but expressed curiosity about whether R$^2$MAE would achieve SOTA performance in more established domains like vision and language modeling. We were fortunate to gain access to additional computational resources during the rebuttal period to perform these experiments.\n\n**Vision:** Our implementations closely follow established practices from the vision MAE work [1]. We implemented different mask ratio settings on the ViT-base MAE model using their official Pytorch codebase. The settings include: 1. Default MAE (constant 0.75 mask ratio, MR); 2. R$^2$MAE (p ~ U(0.6, 0.9)); 3. Dynamic MR [3] (linearly decreasing MR from 0.9 to 0.6); 4. High (0.9) and low (0.5) fixed MR baselines. Given the time limit, all models were trained for 150 epochs. While this is shorter than the 800 epochs in [1], their analysis shows predictable improvements with longer training schedules. Therefore, while our absolute accuracies may be suboptimal, we expect the relative performance across settings to be comparable and the conclusion to hold if trained longer. All other hyperparameters follow the defaults in [1]. Models were then fine-tuned for classification for 100 epochs, following [1].\n\nOur results below show R$^2$MAE achieves the best performance in both top-1 and top-5 accuracy. We found that R$^2$MAE introduces fluctuations in the pretraining loss, likely due to variable sequence lengths since MAE only encodes unmasked tokens. This presents a potential area for even further improvement.\n\n| ViT-base | Acc@1 | Acc@5 |\n|:---|:---:|:---:|\n| MAE default (MR 0.75) | 81.97 | 96.02 |\n| fixed MR 0.9 | 81.20 | 95.68 |\n| fixed MR 0.5 | 81.80 | 95.93 |\n| Dynamic MR | 81.97 | 96.04 |\n| R$^2$MAE | **82.00** | **96.05** |\n\n**Language:** We trained RoBERTa-base and RoBERTa-medium models on the FineWeb dataset for 10B tokens (max seq length 128, comparable to [2]) and fine-tuned them on GLUE benchmarks (MNLI, QQP, SST-2, QNLI). We evaluated: 1. Default MLM (MR 0.15); 2. R$^2$MAE (p ~ U(0.15, 0.4)); 3. Dynamic MR [3] (0.4 to 0.15); 4. MLM with a fixed 0.4 MR. Fine-tuning accuracies are comparable to those in [2]. The results show R$^2$MAE achieves the best overall rank, with a more pronounced advantage in the larger RoBERTa-Base model.\n\n| RoBERTa-Medium (52M) | MNLI | QQP | SST-2 | QNLI | Mean Rank |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| MLM default (MR 0.15) | 80.8 | 89.8 | 89.9 | 86.3 | 3.25 |\n| Fixed MR 0.4 | 80.3 | 89.7 | 90.1 | 86.6 | 3.50 |\n| Dynamic MR | 80.8 | **90.1** | 90.5 | **87.1** | 1.50 |\n| R$^2$MAE | **80.9** | **90.1** | **90.6** | 86.7 | **1.25** |\n\n| RoBERTa-Base (125M) | MNLI | QQP | SST-2 | QNLI | Mean Rank |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| MLM default (MR 0.15) | 81.5 | 90.7 | 91.7 | 87.8 | 3.00 |\n| Fixed MR 0.4 | 81.7 | 90.7 | 91.2 | 88.5 | 3.00 |\n| Dynamic MR | 81.8 | 90.7 | 91.4 | **89.1** | 2.00 |\n| R$^2$MAE | **81.9** | **90.8** | **91.9** | 88.6 | **1.25** |\n\n**Note:** In these fine-tuning evaluations, the advantage of R$^2$MAE appears smaller than in our linear probing or zero-shot evaluations. This is expected, as the identical fine-tuning pipeline can reduce differences from pre-training. Nevertheless, the consistent advantage of R$^2$MAE across tasks and domains strongly supports its effectiveness and generalizability.\n\n### **New Results 2: Explicit Connection between R$^2$MAE and the High-Dimensional Linear Regression Framework**\n\nInspired by reviewer comments, we evaluated R$^2$MAE within our linear model framework. Specifically, we simulated the latent space model from Fig. 1E. Intriguingly, we observed that for features of different strengths, R$^2$MAE's risk tends to align with the *optimal* risk, with its \"effective masking ratio\" decreasing as feature strength decays.\n\n| Normalized test risk | p=0.45 | p=0.5 | p=0.55 | p=0.6 | R$^2$MAE, p~U(0.45,0.6) | Effective p of R$^2$MAE |\n|---|---|---|---|---|---|---|\n| β = 1st Eig | 0.687 | 0.671 | 0.666 | 0.671 | 0.665 | ~0.55 |\n| β = 2nd Eig | 0.751 | 0.748 | 0.752 | 0.772 | 0.750 | 0.5-0.55 |\n| β = 3rd Eig | 0.839 | 0.842 | 0.849 | 0.872 | 0.842 | ~0.5 |\n| β = 4th Eig | 0.880 | 0.889 | 0.898 | 0.929 | 0.888 | 0.45-0.5 |\n\n\nOur new results support the intuition that R$^2$MAE achieves a “feature-adaptive” masking ratio that approximates dataset-specific optimal masking ratios, allowing it to outperform any fixed ratio in settings with multi-scale features. This is validated by real data (Tables 4 and 5), where R$^2$MAE achieves near-optimal reconstruction across its entire masking range (0.1-0.5), while fixed-rate settings only well cover a smaller range. This result aligns with the robust advantage of R$^2$MAE and strengthens the value of our work.\n\n### **Point-by-point response to other specific concerns**\n\nIn response to the reviewer’s comments, we have performed extensive experiments on vision and language mask pretraining models. Our response to each question is provided as follows.\n\n**W1. Theoretical-Algorithmic Gap**: This is a good point. We acknowledge that our theoretical results are based on linear models that deviate from real neural networks, as mentioned in the subsection “Relation with Real Network Optimization” in Section 3.1. Nevertheless, a rich body of literature has applied ridgeless regression models and random matrix theory to successfully recapitulate neural network behaviors like double descent and the scaling law [4,5]. Despite the deviations, we believe that our extensive evaluations of neural network and linear model behaviors convincingly support the validity of our framework as a “working theory.”\n\n**W2. β Estimation**: Indeed, the β estimation procedure is a simplification that enables feature-wise analysis. Nevertheless, we argue that the “feature” here does not necessarily mean individual pixels or tokens. Intuitively, the MSE loss of a sample can be approximated by the reconstruction loss in a latent feature space, linked by a decoding transformation. This insight suggests that we can investigate mask reconstruction behavior by evaluating different eigenvectors of the design matrix covariance as β. Fully addressing the underlying non-linearity remains an open problem and a promising direction for future research.\n\n**W3, Q1. Comparison on General Datasets**: Thank you for raising this excellent point. We agree that incorporating results on image and language modeling would greatly improve our work. Please refer to the earlier section for these results. Overall, they demonstrate the superior performance of R²MAE, which aligns well with the results on biological data.\n\n**Q2. The possibility of determining an optimal masking ratio per sample**: This is a good question. First, as we demonstrated in Section 3.5 of the paper, the optimal masking ratio varies for different downstream tasks. Therefore, in the pretraining phase, finding a single optimal masking ratio may not be well-defined, as it defeats the purpose of training a universal model. Second, a real-world downstream task (e.g., image classification, GLUE tasks) can be highly complex and require learning features at different scales. In contrast, R²MAE learns a spectrum of features during pretraining through a feature-adaptive masking mechanism, a point we clarified earlier with our new results. We will incorporate relevant discussions on this topic in future versions of our work to make our presentation more self-contained. Please let us know if there are any additional concerns.\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" CVPR. 2022.\n[2] Wettig, Alexander, et al. \"Should You Mask 15% in Masked Language Modeling?.\" EACL. 2023.\n[3] Ankner, Zachary, et al. \"Dynamic Masking Rate Schedules for MLM Pretraining.\" EACL. 2024.\n[4] Hastie, Trevor, et al. \"Surprises in high-dimensional ridgeless least squares interpolation.\" Annals of statistics. 2022.\n[5] Bahri, Yasaman, et al. \"Explaining neural scaling laws.\" Proceedings of the National Academy of Sciences. 2024."}, {"review_id": "gJIwSy2aUC", "reviewer": "Reviewer_CpLL", "summary": "This paper presents a theoretical framework for understanding mask-based pre-training through the lens of high-dimensional linear regression. The authors demonstrate that masked autoencoding can be characterized by the test risk of ridgeless regression with masked covariates. Following these insights, they propose R2MAE (Randomly Random Mask AutoEncoding), which samples masking ratios uniformly during training. Their method achieves state-of-the-art results on DNA sequence and single-cell gene expression tasks.", "questions": "What is the relationship between this method and autoregressive modeling? Fully causal masks in fact also expose a uniformly sample masking ratio to every token in the sequence (first token has full masking, last token has almost no masking). Connections to this paradigm would be very valuable for further understanding.\n\nHow does R2MAE's performance scale with dataset size and diversity? A test on an even more studied domain with clearly highly non-homogenous data such as images would be even more convincing and would make me upgrade my score to a 6. If the method fails to improve performance on image encoding (beat MAE) why is that the case?\n\nHow does your framework relate to recent work on spatially-structured masking strategies in vision, such as SiamMAE (Gupta et al., 2023) or CWM (Unifying Machine Vision ... Bear et al., 2023) which utilize specifically structured masking strategies to obtain specific behaviors from the model (such as separation of appearance and motion information) which are not present in uniform, non biased masking approaches?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The reduction of masked pretraining to high-dimensional linear regression provides tractable closed-form analysis while reproducing key empirical phenomena like optimal masking ratios and plateau behaviors.\n- Theoretical predictions are validated across multiple architectures (MLPs, CNNs, Transformers) and datasets, confirming that benefits only emerge in overparameterized regimes.\n- Method achieves consistent improvements over existing methods on DNA and single-cell tasks\n- Rigorous mathematical analysis\n\nWeaknesses:\n- The proposed method (R^2MAE) is only evaluated on biological data despite claims of universality, with no results on standard vision or language benchmarks.\n- The linear model ignores optimization dynamics, architectural inductive biases, and nonlinear feature interactions that may be crucial to masked pretraining", "was_revised": true, "final_justification": "I maintain my score of accept", "rebuttal": "Thank you very much for your comments. Our rebuttal is organized as follows. We will first present new results that address general feedback from all reviewers. They include new experiments on vision and language models and a strengthened connection between R$^2$MAE and our theoretical framework. We will next provide a point-by-point response to specific concerns.\n\n### **New Results 1: Experiments of R$^2$MAE on Vision and Language Models**\n\nIn general, reviewers appreciated our empirical study but expressed curiosity about whether R$^2$MAE would achieve SOTA performance in more established domains like vision and language modeling. We were fortunate to gain access to additional computational resources during the rebuttal period to perform these experiments.\n\n**Vision:** Our implementations closely follow established practices from the vision MAE work [1]. We implemented different mask ratio settings on the ViT-base MAE model using their official Pytorch codebase. The settings include: 1. Default MAE (constant 0.75 mask ratio, MR); 2. R$^2$MAE (p ~ U(0.6, 0.9)); 3. Dynamic MR [3] (linearly decreasing MR from 0.9 to 0.6); 4. High (0.9) and low (0.5) fixed MR baselines. Given the time limit, all models were trained for 150 epochs. While this is shorter than the 800 epochs in [1], their analysis shows predictable improvements with longer training schedules. Therefore, while our absolute accuracies may be suboptimal, we expect the relative performance across settings to be comparable and the conclusion to hold if trained longer. All other hyperparameters follow the defaults in [1]. Models were then fine-tuned for classification for 100 epochs, following [1].\n\nOur results below show R$^2$MAE achieves the best performance in both top-1 and top-5 accuracy. We found that R$^2$MAE introduces fluctuations in the pretraining loss, likely due to variable sequence lengths since MAE only encodes unmasked tokens. This presents a potential area for even further improvement.\n\n| ViT-base | Acc@1 | Acc@5 |\n|:---|:---:|:---:|\n| MAE default (MR 0.75) | 81.97 | 96.02 |\n| fixed MR 0.9 | 81.20 | 95.68 |\n| fixed MR 0.5 | 81.80 | 95.93 |\n| Dynamic MR | 81.97 | 96.04 |\n| R$^2$MAE | **82.00** | **96.05** |\n\n**Language:** We trained RoBERTa-base and RoBERTa-medium models on the FineWeb dataset for 10B tokens (max seq length 128, comparable to [2]) and fine-tuned them on GLUE benchmarks (MNLI, QQP, SST-2, QNLI). We evaluated: 1. Default MLM (MR 0.15); 2. R$^2$MAE (p ~ U(0.15, 0.4)); 3. Dynamic MR [3] (0.4 to 0.15); 4. MLM with a fixed 0.4 MR. Fine-tuning accuracies are comparable to those in [2]. The results show R$^2$MAE achieves the best overall rank, with a more pronounced advantage in the larger RoBERTa-Base model.\n\n| RoBERTa-Medium (52M) | MNLI | QQP | SST-2 | QNLI | Mean Rank |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| MLM default (MR 0.15) | 80.8 | 89.8 | 89.9 | 86.3 | 3.25 |\n| Fixed MR 0.4 | 80.3 | 89.7 | 90.1 | 86.6 | 3.50 |\n| Dynamic MR | 80.8 | **90.1** | 90.5 | **87.1** | 1.50 |\n| R$^2$MAE | **80.9** | **90.1** | **90.6** | 86.7 | **1.25** |\n\n| RoBERTa-Base (125M) | MNLI | QQP | SST-2 | QNLI | Mean Rank |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| MLM default (MR 0.15) | 81.5 | 90.7 | 91.7 | 87.8 | 3.00 |\n| Fixed MR 0.4 | 81.7 | 90.7 | 91.2 | 88.5 | 3.00 |\n| Dynamic MR | 81.8 | 90.7 | 91.4 | **89.1** | 2.00 |\n| R$^2$MAE | **81.9** | **90.8** | **91.9** | 88.6 | **1.25** |\n\n**Note:** In these fine-tuning evaluations, the advantage of R$^2$MAE appears smaller than in our linear probing or zero-shot evaluations. This is expected, as the identical fine-tuning pipeline can reduce differences from pre-training. Nevertheless, the consistent advantage of R$^2$MAE across tasks and domains strongly supports its effectiveness and generalizability.\n\n### **New Results 2: Explicit Connection between R$^2$MAE and the High-Dimensional Linear Regression Framework**\n\nInspired by reviewer comments, we evaluated R$^2$MAE within our linear model framework. Specifically, we simulated the latent space model from Fig. 1E. Intriguingly, we observed that for features of different strengths, R$^2$MAE's risk tends to align with the *optimal* risk, with its \"effective masking ratio\" decreasing as feature strength decays.\n\n| Normalized test risk | p=0.45 | p=0.5 | p=0.55 | p=0.6 | R$^2$MAE, p~U(0.45,0.6) | Effective p of R$^2$MAE |\n|---|---|---|---|---|---|---|\n| β = 1st Eig | 0.687 | 0.671 | 0.666 | 0.671 | 0.665 | ~0.55 |\n| β = 2nd Eig | 0.751 | 0.748 | 0.752 | 0.772 | 0.750 | 0.5-0.55 |\n| β = 3rd Eig | 0.839 | 0.842 | 0.849 | 0.872 | 0.842 | ~0.5 |\n| β = 4th Eig | 0.880 | 0.889 | 0.898 | 0.929 | 0.888 | 0.45-0.5 |\n\n\nOur new results support the intuition that R$^2$MAE achieves a “feature-adaptive” masking ratio that approximates dataset-specific optimal masking ratios, allowing it to outperform any fixed ratio in settings with multi-scale features. This is validated by real data (Tables 4 and 5), where R$^2$MAE achieves near-optimal reconstruction across its entire masking range (0.1-0.5), while fixed-rate settings only well cover a smaller range. This result aligns with the robust advantage of R$^2$MAE and strengthens the value of our work.\n\n### **Point-by-point response to other specific concerns**\n\nIn response to the reviewer’s comments, we have performed extensive experiments on vision and language masked pretraining models. Our responses to each question are provided as follows.\n\n**Q1. Relation to next-token prediction.** Thank you for this very insightful question. Despite the apparent relevance, we believe next-token prediction cannot be adequately addressed by our current theoretical framework. This is because next-token prediction is a token-wise, rather than sample-wise, prediction task. These prediction tasks within an input sequence are highly dependent. This constitutes a clear gap with our current setup, and addressing it will require future research. This is an important point, and we will discuss it in the next version of our paper.\n\n**Q2. R²MAE's performance on larger and more diverse datasets.** Our results on vision and language modeling show that R²MAE provides consistent improvements (albeit sometimes marginal), even for larger models trained on diverse datasets across various data domains. Our evaluations on two configurations of RoBERTa models show that R²MAE provides consistent improvements across model configurations, whereas the performance of other settings varies. These findings align with our results on biological data and strengthen our conclusions.\n\n**Q3. Comparison with structured masking strategies.** This is another great point. The cited structured masking strategies are designed to enforce the model to learn specific patterns in the data of interest. Therefore, they may not be directly comparable to ordinary MAE (or R²MAE) pretraining. Nevertheless, the masking procedure in the cited works is an ordinary random mask with a fixed rate, a setting where incorporating R²MAE could potentially improve performance.\n\nMeanwhile, a variety of structured masking strategies have been proposed to directly improve general pretraining models, as discussed in the literature and in Appendix A.1 of our paper. The RoBERTa masking study [2] compared several masking strategies and concluded that these strategies do not outperform simply tuning the masking rate. Our evaluations of learnable masking strategies for biological data, as well as a parallel study on biological data [4], reached the same conclusion.\n\nThank you again for your thorough evaluation. Please let us know if our responses have fully addressed your concerns or if any additional information is needed.\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" CVPR. 2022.\n[2] Wettig, Alexander, et al. \"Should You Mask 15% in Masked Language Modeling?.\" EACL. 2023.\n[3] Ankner, Zachary, et al. \"Dynamic Masking Rate Schedules for MLM Pretraining.\" EACL. 2024.\n[4] Richter, Till, et al. \"Delineating the effective use of self-supervised learning in single-cell genomics.\" Nature Machine Intelligence. 2025."}, {"review_id": "5TsBGNaH4Y", "reviewer": "Reviewer_T5Vd", "summary": "This paper investigates the theoretical foundation and empirical behavior of mask-based pretraining, which is widely used in self-supervised learning for language, vision, and biological data. The authors propose a unified theoretical framework based on high-dimensional linear regression (including spiked covariance models) to explain why and when mask-based pretraining is effective, and how the optimal mask ratio depends on data and model size. Building on these insights, they introduce a simple pretraining method called R²MAE (Randomly Random Mask AutoEncoding), where the mask ratio is randomly sampled for each training batch. The method is evaluated on DNA sequence and single-cell gene expression tasks, showing modest improvements over traditional approaches. While the theoretical analysis is mathematically solid within linear settings, its extension to deep nonlinear models remains questionable, and the empirical gains are relatively small.", "questions": "See in Weaknesses", "limitations": "See in Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- The approach is easy to implement and performs well on some biological datasets, showing minor improvements over standard random masking.\n- The paper attempts to provide a theoretical explanation for the mask-based pretraining phenomenon, which is a meaningful direction.\n\n## Weaknesses\n\n1. The main idea is just randomizing the mask ratio, which is very similar to dynamic masking and scheduled masking from earlier works [1,2,3,4,5]. The paper doesn’t fairly compare or cite these, and the novelty is overclaimed.\n2. All the theoretical analysis is based on linear models, while real masked autoencoders are highly nonlinear deep networks. There’s a big gap between the theory and practice, so the “universality” claim isn’t convincing [1,6].\n3. Experiments are only on a few models and datasets, with no evidence for challenging or large-scale biological tasks. The generality claim is not well supported [1,7].\n4. The biological results show only marginal improvements, and there’s almost no discussion on whether these gains are meaningful for real science or medicine [7].\n5. The Introduction is poorly constructed—it promises universal advances for language and vision, but the paper only does very limited, niche biological experiments (DNA sequence and single-cell gene expression) or toy datasets like MNIST, with absolutely no evaluation on general language or vision tasks. There’s no bio-specific design either; the method is generic and not tailored at all.\n6. The writing is lengthy and dense, with key points buried in technical details and formulas.\n7. There are issues in the theoretical proofs. For example, in the spiked covariance analysis, the authors make strong assumptions (like high-dimensional limits and feature alignment) without really checking if they hold in practice. Many derivations just assume ideal Gaussian data and feature structures, but small deviations in real data can break the conclusions. The claim that an optimal mask ratio always exists under spiked covariance only holds under strict conditions, which are not verified in their experiments. Much of the analysis is quite similar to recent works like MaskTwins [8], which also use linear model or spiked covariance for mask theory, and while those papers have their own limitations, they actually discuss model assumptions and limitations more carefully. This paper overstates its theoretical contribution and doesn't clearly position itself with respect to these existing analyses.  \n\n### References\n\n1. Ankner et al., Dynamic masking rate schedules for MLM pretraining, arXiv:2305.15096 (2023). [link](https://arxiv.org/abs/2305.15096)\n2. Yang et al., Learning better masking for better language model pre-training, arXiv:2208.10806 (2022). [link](https://arxiv.org/abs/2208.10806)\n3. Wettig et al., Should you mask 15% in masked language modeling?, arXiv:2202.08005 (2022). [link](https://arxiv.org/abs/2202.08005)\n4. Bao et al., BEiT: BERT Pre-Training of Image Transformers, arXiv:2106.08254 (2021). [link](https://arxiv.org/abs/2106.08254)\n5. He et al., Masked Autoencoders Are Scalable Vision Learners, CVPR 2022. [link](https://arxiv.org/abs/2111.06377)\n6. Chen et al., TokenUnify: Scalable Autoregressive Visual Pre-training with Mixture Token Prediction, arXiv:2405.16847 (2024). [link](https://arxiv.org/abs/2405.16847)\n7. Dalla-Torre et al., Nucleotide transformer: building and evaluating robust foundation models for human genomics, Nature Methods, 2025. [link](https://www.nature.com/articles/s41592-024-02523-z)\n8. Wang et al., MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation, ICML 2025. [link](https://openreview.net/pdf?id=9CpeZ8BzPO)", "was_revised": true, "final_justification": "The consistent cross-domain improvements and notable 16.5% improvement in OMIM regulatory variant prediction demonstrate method robustness. Given these substantial additions and the potential for improved presentation, I am inclined to raise my assessment to borderline accept.", "rebuttal": "Thank you very much for your comments. Our rebuttal is organized as follows. We will first present new results that address general feedback from all reviewers. They include new experiments on vision and language models and a strengthened connection between R$^2$MAE and our theoretical framework. We will next provide a point-by-point response to specific concerns.\n\n### **New Results 1: Experiments of R$^2$MAE on Vision and Language Models**\n\nIn general, reviewers appreciated our empirical study but expressed curiosity about whether R$^2$MAE would achieve SOTA performance in more established domains like vision and language modeling. We were fortunate to gain access to additional computational resources during the rebuttal period to perform these experiments.\n\n**Vision:** Our implementations closely follow established practices from the vision MAE work [1]. We implemented different mask ratio settings on the ViT-base MAE model using their official Pytorch codebase. The settings include: 1. Default MAE (constant 0.75 mask ratio, MR); 2. R$^2$MAE (p ~ U(0.6, 0.9)); 3. Dynamic MR [3] (linearly decreasing MR from 0.9 to 0.6); 4. High (0.9) and low (0.5) fixed MR baselines. Given the time limit, all models were trained for 150 epochs. While this is shorter than the 800 epochs in [1], their analysis shows predictable improvements with longer training schedules. Therefore, while our absolute accuracies may be suboptimal, we expect the relative performance across settings to be comparable and the conclusion to hold if trained longer. All other hyperparameters follow the defaults in [1]. Models were then fine-tuned for classification for 100 epochs, following [1].\n\nOur results below show R$^2$MAE achieves the best performance in both top-1 and top-5 accuracy. We found that R$^2$MAE introduces fluctuations in the pretraining loss, likely due to variable sequence lengths since MAE only encodes unmasked tokens. This presents a potential area for even further improvement.\n\n| ViT-base | Acc@1 | Acc@5 |\n|:---|:---:|:---:|\n| MAE default (MR 0.75) | 81.97 | 96.02 |\n| fixed MR 0.9 | 81.20 | 95.68 |\n| fixed MR 0.5 | 81.80 | 95.93 |\n| Dynamic MR | 81.97 | 96.04 |\n| R$^2$MAE | **82.00** | **96.05** |\n\n**Language:** We trained RoBERTa-base and RoBERTa-medium models on the FineWeb dataset for 10B tokens (max seq length 128, comparable to [2]) and fine-tuned them on GLUE benchmarks (MNLI, QQP, SST-2, QNLI). We evaluated: 1. Default MLM (MR 0.15); 2. R$^2$MAE (p ~ U(0.15, 0.4)); 3. Dynamic MR [3] (0.4 to 0.15); 4. MLM with a fixed 0.4 MR. Fine-tuning accuracies are comparable to those in [2]. The results show R$^2$MAE achieves the best overall rank, with a more pronounced advantage in the larger RoBERTa-Base model.\n\n| RoBERTa-Medium (52M) | MNLI | QQP | SST-2 | QNLI | Mean Rank |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| MLM default (MR 0.15) | 80.8 | 89.8 | 89.9 | 86.3 | 3.25 |\n| Fixed MR 0.4 | 80.3 | 89.7 | 90.1 | 86.6 | 3.50 |\n| Dynamic MR | 80.8 | **90.1** | 90.5 | **87.1** | 1.50 |\n| R$^2$MAE | **80.9** | **90.1** | **90.6** | 86.7 | **1.25** |\n\n| RoBERTa-Base (125M) | MNLI | QQP | SST-2 | QNLI | Mean Rank |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| MLM default (MR 0.15) | 81.5 | 90.7 | 91.7 | 87.8 | 3.00 |\n| Fixed MR 0.4 | 81.7 | 90.7 | 91.2 | 88.5 | 3.00 |\n| Dynamic MR | 81.8 | 90.7 | 91.4 | **89.1** | 2.00 |\n| R$^2$MAE | **81.9** | **90.8** | **91.9** | 88.6 | **1.25** |\n\n**Note:** In these fine-tuning evaluations, the advantage of R$^2$MAE appears smaller than in our linear probing or zero-shot evaluations. This is expected, as the identical fine-tuning pipeline can reduce differences from pre-training. Nevertheless, the consistent advantage of R$^2$MAE across tasks and domains strongly supports its effectiveness and generalizability.\n\n### **New Results 2: Explicit Connection between R$^2$MAE and the High-Dimensional Linear Regression Framework**\n\nInspired by reviewer comments, we evaluated R$^2$MAE within our linear model framework. Specifically, we simulated the latent space model from Fig. 1E. Intriguingly, we observed that for features of different strengths, R$^2$MAE's risk tends to align with the *optimal* risk, with its \"effective masking ratio\" decreasing as feature strength decays.\n\n| Normalized test risk | p=0.45 | p=0.5 | p=0.55 | p=0.6 | R$^2$MAE, p~U(0.45,0.6) | Effective p of R$^2$MAE |\n|---|---|---|---|---|---|---|\n| β = 1st Eig | 0.687 | 0.671 | 0.666 | 0.671 | 0.665 | ~0.55 |\n| β = 2nd Eig | 0.751 | 0.748 | 0.752 | 0.772 | 0.750 | 0.5-0.55 |\n| β = 3rd Eig | 0.839 | 0.842 | 0.849 | 0.872 | 0.842 | ~0.5 |\n| β = 4th Eig | 0.880 | 0.889 | 0.898 | 0.929 | 0.888 | 0.45-0.5 |\n\n\nOur new results support the intuition that R$^2$MAE achieves a “feature-adaptive” masking ratio that approximates dataset-specific optimal masking ratios, allowing it to outperform any fixed ratio in settings with multi-scale features. This is validated by real data (Tables 4 and 5), where R$^2$MAE achieves near-optimal reconstruction across its entire masking range (0.1-0.5), while fixed-rate settings only well cover a smaller range. This result aligns with the robust advantage of R$^2$MAE and strengthens the value of our work.\n\n### **Point-by-point Response to Specific Concerns**\n\n**W1. Relevant works:** Thank you for the comment. We would like to note that we indeed cited all relevant literature mentioned in the review. Furthermore, we have a dedicated section (Appendix A.1) discussing previous works, and our benchmarks included several of these approaches. While R$^2$MAE is simple, it remains uncovered by previous work. Our theoretical framework is also novel in clarifying unexplained phenomena in mask-based pre-training. Given this, we respectfully disagree with the judgment that our work “does not fairly compare or cite these, and the novelty is overclaimed.” We are happy to address more specific comments if provided.\n\n**W2. Gap between the theory and practice:** We discussed this gap in the “Relation with real network optimization” part of Section 3.1. Despite the gap, using linear models to explain behaviors in neural networks is a well-established approach in the literature [4, 5]. To verify our findings generalize, we performed extensive evaluations on MLPs, CNNs, and Transformers (Figs. 1-2). Importantly, these evaluations highlighted several phenomena that are explained by our overparametrized linear model but not by alternative frameworks, suggesting our model serves as a valuable “working theory” of masked pre-training.\n\n**W3-W5. On the limitations of experimental results:**\n1.  **On \"marginal\" improvement:** We respectfully disagree. R$^2$MAE provides consistent and, in several cases, substantial performance improvements. For instance, we improved the PRAUC for the OMIM regulatory variant prediction task by 16.5% over the SOTA model reported in [6]. In brain gene expression data, we improved cell-level age prediction Spearman's r by 10.9% over an optimal MAE. While improvements are smaller in a number of cases, they are consistent across metrics, tasks, and data domains, including the new vision/language results. This improvement is a solid advance for a pre-training scheme that does not alter model architecture.\n\n2.  **On task relevance and scientific discovery:** Our tasks are well-established benchmarks. The DNA variant prediction task is widely used for evaluating DNA models (e.g., Nucleotide Transformer [NT] cited by the reviewer). The OMIM benchmark itself is a large-scale task evaluating over two million DNA sequences. Our experiments span a range of evaluation scenarios from zero-shot (DNA) and linear probing (gene expression) to fine-tuning (ViT, RoBERTa).\n\n3.  **On the lack of vision/language results:** We agree this is an important point and have now provided these results in the first section of our response.\n\n**W5-W6. On paper writing:** We appreciate the feedback. We are unclear how to address general criticisms like “lengthy and dense writing” but would be happy to modify our text if more specific suggestions are provided. Our introduction aims to motivate a general framework, and we did not find sentences that “promise universal advances for language and vision” as criticized. We believe these comments may stem from a misunderstanding.\n\n**W7. On theory assumptions and the MaskTwins paper:** Our theoretical work does not aim to prove properties of neural networks directly. Instead, we define a tractable linear model, analyze its properties, and then empirically evaluate if these properties hold in real networks. We empirically verified our findings across multiple data models (Fig. 1E) and network architectures (Fig. 2), in order to support our framework's validity.\n\nWe were not aware of MaskTwins as it was published after our submission deadline, but we will cite it in the next version. We have reviewed the paper and find the technical approach to be very different. MaskTwins uses compression sensing theory to derive bounds for signal discovery in a signal+noise model. Our work uses random matrix theory to characterize test risk in an overparameterized regression setting. The goal is also distinct. We would be happy to revise any part of our manuscript that the reviewer feels is an overstatement.\n\nThanks again for your thorough evaluation. Please let us know if there are any additional concerns.\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" CVPR. 2022.\n[2] Wettig, Alexander, et al. \"Should You Mask 15% in Masked Language Modeling?.\" EACL. 2023.\n[3] Ankner, Zachary, et al. \"Dynamic Masking Rate Schedules for MLM Pretraining.\" EACL. 2024.\n[4] Hastie, Trevor, et al. \"Surprises in high-dimensional ridgeless least squares interpolation.\" Annals of statistics. 2022.\n[5] Bahri, Yasaman, et al. \"Explaining neural scaling laws.\" PNAS. 2024.\n[6]  Benegas, Gonzalo, et al. \"A DNA language model based on multispecies alignment predicts the effects of genome-wide variants.\" Nature Biotechnology, 2025."}]

Author rebuttal:

Meta-review: This paper develops a theoretical framework for understanding mask-based pretraining through the lens of high-dimensional ridgeless linear regression, offering new insights into when and why masked autoencoding is effective and how optimal mask ratios depend on data and model size. Guided by this analysis, the authors introduce R2MAE (Randomly Random Mask AutoEncoding), a simple scheme that samples masking ratios uniformly during training. Applied to DNA sequence and single-cell gene expression tasks, the new method achieves SOTA or improved performance over standard masking strategies, demonstrating the value of their theoretically derived method.

The reviewers generally acknowledged the importance of the work in contributing a theoretical foundation about why and how masked autoencoding works. This by itself is an important contribution. Furthermore, the reviewers appreciated the presented, wide-range empirical validation of predictions from this theoretical work for a diverse set of network architectures. This full range approach in addressing an open scientific question in ML is impressive. Some concerns were raised over some of the linear and other approximations made in the analysis, as well as about the range of application presented in the original submission. The authors well addressed the latter by showing new empirical validation over a wide range of additional datasets that were all favorable for the new method. Inclusion of these results and some refinements on the clarity of the writing will turn this paper into a valuable contribution for the NeurIPS community.
Decision: Accept (poster)