How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders
Authors: Qi Zhang
Yifei Wang
Yisen Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Masked Autoencoders (MAE) based on a reconstruction task have risen to be a promising paradigm for self-supervised learning (SSL) and achieve state-of-the-art performance across different benchmark datasets. However, despite its impressive empirical success, there is still limited theoretical understanding of it. In this paper, we propose a theoretical understanding of how masking matters for MAE to learn meaningful features. We establish a close connection between MAE and contrastive learning, which shows that MAE implicit aligns the mask-induced positive pairs. Built upon this connection, we develop the first downstream guarantees for MAE methods, and analyze the effect of mask ratio. Besides, as a result of the implicit alignment, we also point out the dimensional collapse issue of MAE, and propose a Uniformity-enhanced MAE (U-MAE) loss that can effectively address this issue and bring significant improvements on real-world datasets, including CIFAR-10, ImageNet-100, and ImageNet-1K. Code is available at https://github.com/zhangq327/U-MAE.


How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders
Qi Zhang" Yifei Wang Yisen Wang 4.3f
Key Lab. of Machine Perception (MoE), School of Intelligence Science and Technology. Peking University 2 School of Mathematical Sciences, Peking University Institute for Artificial Intelligence, Peking University
Abstract
Masked Autoencoders (MAE) based on a reconstruction task have risen to be promising paradigm for self-supervised learning (SSL) and achieve state-of- the-art performance across different benchmark datasets. However, despite its impressive empirical success, there is still limited theoretical understanding of it. In this paper, we propose a theoretical understanding of how masking matters for MAE to learn meaningful features. We establish close connection between MAE and contrastive learning, which shows that MAE implicit aligns the mask- induced positive pairs. Built upon this connection, we develop the first downstream guarantees for MAE methods, and analyze the effect of mask ratio. Besides, as a result of the implicit alignment, we also point out the dimensional collapse issue of MAE, and propose a Uniformity-ehhanee effectively address this issue and bring significant improvements on real-world datasets, including CIFAR-IO, ImageNet-1000 and ImageNet- K. Code is available at https:/ pettps://gtthut http://ggttuuub.ommangggg33/ com//hangg337/U /U-MMM
MAE (U-MAE) loss that can
Introduction
Recently, self-supervised learning (SSL) has been proposed as promising paradigm for learning data representations without access to labels. Beside the popular contrastive learning methods [4, 14, 12, 27], recently there has been renaissance of reconstruction-bant
autoencoders for SSL, for example, MAE [15], BEiT [1], iBOT [32], and SimMIM [29], which demonstrate state-of-the of- art performance various downstream tasks. Taking MAE example, it learns to reconstruct the masked patches from the unmasked context with an encoder-decod architecture, from which we can that there
main components in MAF: masking and autoencoders. While for autoencoders, its canonical one, even equipped with expressive neural networks, still less competitive than modern SSL methods [4], which indicates that autoencoders may not be the key factor for the success of MAE. Then for masking, MAE needs a very large mask ratio, e.g., 75% patches of the input image masked out. Such large mask ratio will lead the loss of most image contents that hardly recoverable from the rest. Further considering that MAE deliberately chooses weak decoder with shadow architecture, believe that reconstruction might not be the ultimate goal of MAE in the learning process. Then, some natural questions are raised here:
What is the role of masking in MAE? How does it affect the downstream performance?
To answer these questions, firstly, we build close connection between MAE and contrastive learning. In particular, we find that masking could produce implicit positive pairs while MAE's reconstruction
"qqual Contribution.
Corresponding author: Yisen Wang (yisen.wangd)) @ pku.edu.cn).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

loss is lower bounded by an alignment loss on these positive pairs, Further for feature uniformity, we notice that although MAE will not fully collapse, it still suffers from dimensional collapse where features lie in a low-dimensional space and become highly alike. Inspired by the uniformity loss in contrastive learning, we propose a Uniformity-enhanee
MAE (U-MAE) to promote the feature diversity. Empirically, the proposed U-MAF attains significant improvements over MAE on the linear probing task across different benchmark datasets (CIFAR-IO I0, ImageNet- 100, and ImageNet-IK: and different ViT backbones, and it effectively eliminates the feature collapse issue of MAE. Secondly, the connection between MAE and contrastive learning enables us to establish the first theoretical guarantee for downstream classification among MAE methods. Our guarantees suggest that large mask ratio is indeed necessary for bridging semantically similar samples together.
We summarize our contributions as follows:
We propose a new theoretical understanding of MAE by establishing a formal connection between MAE and contrastive learning. In particular, we show that a small reconstruction loss implies better alignment of mask-induced positive pairs.
Built upon this connection, we establish the first theoretical guarantee on the downstream performance among MAE methods, which helps understand the role of high mask ratio.
We point out the dimensional collapse issue of MAE, and propose U-MAE that enhances feature diversity with a uniformity loss. Empirically, U-MAE improves the linear probing accuracy of MAE by large margin across different real-world datasets and backbones.
2 Related work
Self-Supervised Learning. Canonical deep learning relies labeled data (e.g.. ImageNet) train deep models. Instead, self-supervised learning (SSL) aims to learn meaningful representations from fully unlabeled data. A particular example is contrastive learning (CL) that learns align augmented samples in the feature space, which achieves remarkable success and largely closes the performance gap between supervised and self-supervised learning [4, 14, 12, 27]. Recently, Masked Image Modeling (MIM). stemming from the Masked Language Modeling (MLM) paradigm widely adopted in NLP (e.g.. BERT [9]), also shows promising results in visual representation learning, to name a few, BEiT [1]. iBOT [32], SimMIM [29], and MAE [15]. Among these MIM methods, the former three (similar to BERT) have close connections to contrastive learning (their objective can be directly formulated as contrastive loss [20]), while MAE has several key differences to BERT-like methods. As the name suggests, MAE is more like an autoencoder rather than a token predictor: adopts a pixel-level reconstruction loss, omits the masked tokens in the encoder input, and utilizes a fully non-linear encoder-decoder architecture. In this work, we mainly provide a theoretical understanding of the working mechanism of MAE, in particular, how it learns generalizable features. Meanwhile, our analysis can also be applied to other MIM methods by different specifications of the encoder-decoder and their inputs.
Understanding MAE. Despite the impressive success of MAE, the theoretical understanding of is largely underexplored. Among several concurrent attempts, Cao et al. [2] mostly focus on the attention mechanism of MAE through an integral kernel perspective. A similar theoretical analysis [23] also shows that autoencoders can preserve useful semantics in the pretraining data. However, we notice that these MAE analyses mostly focus on the autoencoder architecture, while they largely ignore the role of masking. As shown by He el al. [15], the patchwise masking strategy is a key component that distinguishes MAE from standard autoencoders, and different mask ratios p have a large impact on the downstream performance of pretrained features of MAF. Therefore, our work reconstruction task learns meaningful representations.
aims to explain how the masking-
Downstream Generalization of SSL. Motivated by the empirical success of SSL, many researchers try to theoretically understand how SSL works. Arora et al. [25] establish downstream guarantees of contrastive learning representations. Wang et al. [28] revise their bounds by resolving the class collision issues, and develop a new understanding of contrastive learning from the perspective of augmentation overlap. Haochen et al. [13] propose an augmentation graph framework and characterize the downstream performance of the cigen-decomposition solution. While these prior works focus on the contrastive learning method, there are little theoretical understanding on the downstream performance on MAE. In this work, we establish the first theoretical guarantee on the downstream performance among MAE methods and discuss the effect of mask ratios theoretically and empirically.

---

3 Masked Autoencoders Perform Implicit Contrastive Learning
In this section, we establish formal connection between MAE and contrastive learning by showing that MAE implicitly aligns positive input pairs induced from the masking mechanism. Specifically, in Section 3.2, we show how a small MAE loss implies small alignment loss. Motivated by this connection, in Section 3.3, we further study its feature collapse issue and point out that MAE suffers from dimensional feature collapse. To address this issue, in Section 3.4, we propose eniboreity-enhanced (U-MAE) loss to further promote feature diversity.
3.1 Mathematical Formulation for MAE
We begin by introducing mathematical formulation of MAE [15]. Given a natural image F from an unlabeled dataset Du. we first reshape it into n patches, denoted as E RR*x where s denotes the patch size (e.g., 16 x 16 in ViT [10]. Then, we draw random binary mask m E {0, 1 (drawing 0 with probability p. Le., the mask ratio), and obtain two complementary masked views of Æ:
11 = x T[m] E IR I2z = x[| m] E I_=2**,
(I)
where ni = (m(11))) p), n2 = np are integers satisfying π = ni + n2. We denote this random masking
process drawing 21, 22 from the joint distribution , xz) whose marginal distributions are Mi(zi]ãs [ā) and M2(xg2(), respectively. The MAE model h = g o f is an encoder-decoder architecture, where encoder f maps imputs I1 to a latent feature 21 f(11), and decoder g
maps the latent feature 2 back to the pixel space to reconstruct the complementary target view x22 Specifically, MAE adopts a simple mean square error (MSE) loss:
(h) = Eggeer aall (Ix()(x)) - =22l?
(2)
where the decoder output 22 h(x1) ((((())) is assumed to be Iz-normalized following the original paper of MAE saying that normalized target 22 yields better performance [15]. Although our analysis is based on MAE, it is quite general and can be naturally transferred to other Masked Image Modeling (MIM) frameworks, such as BEiT [1], iBOT [32], and SimMIM [29]. This is because their differences mainly lie in the implementation details, which is discussed in detail in Appendix C.
The Bipartite Mask Graph GM of MAE. We notice that MAE essentially learns to pair the two complementary views 21, 22 via the reconstruction task, which can be modeled by mask graph Gas-
Denote the set of all unmasked views as X; = {a1} } and the set of all masked views as X₂ {x2). where the two sets are assumed to be finite" (can be exponentially large), i.e., |&&|| N1, 122 = N2. The mask graph GM over the joint set X' Xi U X22 is defined here:
Node: each view x E X.
Edge: the edge weight Way,2g between any F1, E X' is defined as their joint probabil- ity M(11, x22) E,, Ex,((zi, 22 (T) In other words, there is an edge between two views (i.e., W21,22 > 0) if and only if they are complementary views generated by masking.
Considering the masking process in Eq. 1, there only exist edges between the two sets X1, and there is no edge within each itself. Thus, the mask graph Gm is bipartite graph. Its adjacency matrix can be simply defined as AM E R N where (AM)
W2i for Ij E X1, E X22. As long as p + 0.5, N1 # N2 and AM is not necessarily a square matrix. We can define the normalized adjacency matrix as Am = D2 1/ 1/2 where D1, D2 are the diagonal degree matrices with elements dz, En, Way,zz and da, Σ 2n Way respectively.
3.2 MAE Implicitly Aligns Positive Input Pairs
As the name suggests, MAE basically composed of two key designs: mask and autoencoders. For simplicity, we begin by assuming that the encoder-decoder architecture of MAE is capable of accomplishing the vanilla autoencoder task, i.e., reconstructing the original input.
Assumption 3.1. For any non-degenerate decoder g. assume the existence of a pseudo-inverse encoder Jg F such that the resulting pseudo autoencoder hg = go fg satisfies E,, (1,()))))))))))) =||2 S e, where x represents either unmasked data masked data .22
"This is used to avoid non essential nuances related to functional analysis. Our discussion can also be extended to the infinite data regime following Haochen et al. [13].
3

---

Mask Graph
Augmentation Graph
Randem in: MAE
w MMM M
Log otaagserr ae
L-MAC
-00
16
L-MAE
20
Av E R
E N
20
40
60
20
100
20
80
100
Cpecch
(a) mask graph and augmentation graph
(b) singular values
(c) effective rank
Figure I: (a) An illustration of the mask graph and the corresponding augmentation graph of MAE. Different colors denote different belonging classes. (b) Comparison of the singular values of learned features with 1) random initialization, 2) MAE loss, 3) our U-MAE loss. (c) The changing process of effective rank [24] of the encoded features trained with different objectives (MAE and U-MAB).
This assumption is likely to hold in practice, since deep networks already demonstrate excellent performance on the autoencoding task [19, 17]. Besides, some works show that the Transformer network adopted in MAE has universal approximation ability [30]. However, we note that vanilla autoencoder task cannot learn as meaningful representations as MAE. For example, without applying any masks, the linear probing accuracy of MAE drops dramatically from 61.2% to 17.4% on ImageNet-100 This indicates that the autoencoding ability (studied in [2, 23]) is not enough to explain the efficacy of MAE, which motivates us further explore the masking mechanism in MAE
MAE Performs Asymmetric Input-output Alignment on the Mask Graph. First, we show that the MAE loss can be lower bounded by an asymmetric alignment loss between the two complementary views I 22 using two-branch autoencoders h (real) and hg (pseudo), respectively.
Theorem 3.2. Under Assumption 1. the MAE loss can be lower bounded by
LMAE (h) 2 Lasym (h) + const, and Lasym (h) = (0x1,,,(x1)) (22) -tr(H
(3)
tr(H HMH
(4)
where H denotes the output matrix of on X1 whose 11 row Hz, V (a and Hg denotes the output matrix of hg on X2 whose T2-th row is ( (Hg)az = (aghhg(a))
As result, small MAE loss implies small alignment loss (as lower bound), which indicates that MAE will implicitly align the masked and unmasked views with an encoder-decoder architecture. However, is little bit confusing why aligning the two complementary views helps learn meaningful features. For a closer look, we find that via masking, MAE actually produces implicit connections among different input samples in the form of 2-hop connectivity. Consider pair of 2-hop input neighbors T1, _ E X that share a common complementary target view I2 E X2 (more likely to happen under a larger mask ratio p). as illustrated in Figure (a)). By enforcing 41 41, क to reconstruct the same output x2. MAE will implicitly map their features together. this way, the 2-hop input neighbors serve as positive pairs that will be implicitly aligned as in contrastive learning.
Motivated by this observation, we seek to establish formal connection between MAE and contrastive learning below. In particular, we will show that like the data augmentations in contrastive learning, the masking mechanism of MAE also implicitly introduces affinities between input samples and create (implicit) positive pairs. For a formal exposure, we define an augmentation graph GA for modeling the relationship between all input samples in Xi, which is different from the mask graph GM for modeling the input-output relationship between XG and X2.
The Augmentation Graph GA of MAE, We construct an augmentation graph G for modeling the mask-induced affinity between all unmasked views in X1 Specifically, for any two views I1, oτ E X1, we define their edge weight in the adjacency matrix A as the probability of having the same target view, i.e., A(z1 2²)) = M(x [[2) (x2)) 5. Note that the augmentation graph
"We can also construct augmentation graph G' the output space X2, where the (æz, xr))+tt) th element of the adjacency matrix A' is defined by A(x22 = Er, (M As the results are quite similar, we mainly take the input space as an example in this work.
M(zi M(x1, 22)/ M(x2)) can further be calculated by marginalizing x2l1).

---

has the same degree matrix D1 as the mask graph. Based on this formulation, we define a symmettric alignment loss w.E. the positive pairs (I', +J] A(1 2f) drawn according to the augmentation graph:
(5) MAE Performs Symmetric Input Alignment on the Augmentation Graph. Built upon the augmentation graph constructed above, we theoretically verify the intuition on 2-hop connectivity by establishing the relationship between the asymmetric input-output alignment loss on the mask graph and the symmetric input alignment loss on the augmentation graph in the following theorem.
Calign (h) = =.,,. h(x1)
Th(=t)
Theorem 3.3. The asymmetric alignment loss on the mask graph (Eq. (4)) can be lower bounded by the symmetric alignment loss on the augmentation graph (Eq. (5)):
Lasym (h) Z Calign (h) + const.
(6)
2
Proof Sketch. We provide a proof sketch of this inequality as it is the key to our analysis. We first symmetrize the asymmetric alignment loss Lasse (h) with an arithmetic inequality, and then establish its equivalence to the symmetric alignment loss Latign (h). The derivation highlights the intrinsic connection between the mask graph (adjacency matrix AM) and the augmentation graph (adjacency matrix A).
Lasym (h) = h((r))
-tr(II (r(HHAM
(reformulated to the mask graph (Theorem 3.2))
2 "IIET AM 12 12+ ++|HH113
(because tr(AB) S
2 (1|411 + 11)
1
2
Ur A H) 1
(because |H11? = Σ (eg (x2)) = 1)
= 2
422
Σ Am, ((((1))(((t))
(transformed to the augmentation graph)
2
1 2
2 Caligo (h)
(following the definition in Eq. (5))
Combining Theorem 3.2 and Theorem 3.3, we arrive at the main theorem showing that MAE's reconstruction loss can be bounded by the symmetric alignment loss of the positive input pairs (x1, =f) drawn according to the augmentation graph.
Theorem 3.4. Under Assumption 3.1, MAE's reconstruction loss (Eg. (2)) can be lower bounded by the alignment loss between positive pairs (x1, ~ A(x1, xt),
(M)E(h) 2 2 Lullgge (h) E + const -
E 2 _ h(11)
const.
In this way, we establish a close relationship between the two mainstream SSL paradigms (MAE and contrastive learning) by showing that a small MAE loss will imply small alignment loss of positive input pairs as in contrastive learning. Leveraging this connection, we could establish guarantees for the downstream generalization of MAE (discussed in Section 4).
Comparison to Contrastive Learning. Comparing the aligmment loss (Eq. (5)) to that of contrastive learning [22, 4. 13], we notice that the main difference lies in that contrastive learning aligns features in the latent space of the encoder f. while MAE aligns features in the output space with an encoder-decoder architecture h = f. Nevertheless, we also note that most variants of contrastive learning apply a nonlinear projection head g upon the encoder f before calculating the alignment loss [4, 5, 14, 12, 6], and MAE also adopts a shadow decoder g. Therefore, we may also regard the role of MAE's decoder as the projection head in contrastive learning. Theoretically, if we further assume the bi-Lipschitzness of the decoder, we can show that the MAE loss is further lower bounded by an alignment loss defined in the feature space.
Corollary 3.5. Under Assumption 3.I and the assumption that the decoder is T-bi--ipschitzz Le., V (81,22) 1/Tl// x2 < [1((;)) - g(x2))12 < -1|=1-1221 -I2 Then, the MAE reconstruction loss can be lower bounded by the alignment loss w.r.t. the encoder outputs:
(M) 2 -1/(2L) E Ex f(1)) f(x)) - + const.
(8)
5

---

3.3 The Feature Collapse Issue in MAE
We have revealed that the MAE loss is closely related to an alignment loss. However, it is well known that in contrastive learning, simply aligning the positive pairs will result in the full feature collapse, because the alignment loss can also be minimized when the encoder produces a constant feature for all inputs. There, various techniques are proposed to address the issue by incorporating an additional loss to encourage feature uniformity or feature decorrelation [4, 14, 31], or by asymmetric structural designs [12, 6, 3.. MAE does not implement any of these techniques, but its latent features still do not fully collapse. One would wonder how MAE attains this property. the following theorem, we show that minimizing the MAE loss can provably get rid of the full feature collapse.
= c, the MAE loss has a large Theorem 3.6. When the encoder fully collapses, i.e., Vx E X1, f(x) lower bound:
(h) 2 Var(xz),
where Var(z2) denotes the variance of masked targets computed on the training dataset.
MAE Can Avoid Full Feature Collapse. As the training data contain diverse images, the variance Var(z2) will be relatively large. Therefore, unlike the alignment loss in contrastive learning, the MAE loss cannot be minimized (to a small value) by a collapsed encoder. The main reason that alignment loss operates in a fully flexible latent space that allows a collapsed encoder to minimize the loss, while in MAE, the reconstruction loss adopts a parameter-invariant and sample-dependent target T2. In this case, fully collapsed encoder can no longer minimize the reconstruction loss w.r.f. Z2. which, in turn, means that MAE will not fully collapse.
MAE Still Suffers from Dimensional Collapse. Although MAE can avoid full feature collapse, it could still suffer from dimensional feature collapse where the learned features lie in a low dimensional subspace [16. 18], which also limits its representation power. We empirically examine this issue on ImageNet-1000 100. Figure (bb) shows that after learning, MAE's features indeed become more collapsed as there are fewer large singular values (indicating non-collapse dimensions). Quantitatively, Figure (c)) shows that the features of MAE gradually collapse during the training process, as the effective rank [24] becomes smaller and smaller. This shows that MAE suffers from an increasing degree of dimensional feature collapse. Next, we introduce an explicit regularization to address this issue.
3.4 U-MAE: Enhancing Feature Diversity with an Explicit Uniformity Regularization
To further enhance the feature diversity of MAE, inspired by the uniformity loss in contrastive learning [22, 4, 13], we propose the following Uniformity-enhaneed MAE (U-MAE) loss with an explicit regularization on feature uniformity through a coefficient X > 0,
LU (h) = LMXE (h) + X Luiit (f),
(10)
where (fmit()) = E,, E,- (f11 (x
(11)
and x- denotes an independently drawn unmasked view from Xi. Intuitively, the spectral uniformity loss Lfmi(()) [13] encourages small feature similarity between random unmasked views, which could effectively promote the feature diversity of all samples. As a preview of the results, we can observe from Figures 1(b) & I(c) that U-MAB effectively addresses the dimensional feature collapse issue and improves the effective feature dimensionality by a large margin.
Theoretically, combined with Corollary 3.5, we can show that the U-MAE loss is lower-bounded by the spectral contrastive loss with a specific choice of X.
Theorem 3.7. Denote the spectral contrastive loss (SCL) from Haochen et al. [13] as
(f(z) ( f(zT))² (12) (fcr(f) = (f) + Luni(f) = -2E,, f(z)) f(=T) + Ex
Under Assumption 3.1 and the assumption that the decoder is L-bi-Lipschitz, when X = 1/(4L), the U-MAE loss can be lower bounded by the SCL loss:
1
LU-MAE(h) 2 4L
-scu(f) - + const.
(13)
As result, minimizing the U-MAE loss will implicitly minimize the spectral contrastive loss among input views. Leveraging this inherent connection, in the next section, we further explain the important role of masking on the downstream generalization of MAE (as well as U-MAE).

---

4 Downstream Generalization of Masked Autoencoders
In Section 3, develop close connection between MAE and contrastive learning. Built upon this connection, in Section 4.1, we establish theoretical guarantees on the downstream performance of U-MAE leveraging the mask-induced augmentation graph introduced in Section 3.2. Built upon these theoretical insights, in Section 4.2, we further investigate the effect of mask ratios and explain the optimal mask ratio of MAE.
4.1 Theoretical Guarantees Downstream Classification
In this part, we characterize the downstream performance of U-MAE on the c-class linear classification task [25, 13], which is measured by the prediction accuracy of natural images w.r.t. their labels y(z) when applying a linear prediction head upon pretrained features. For simplicity, we adopt the mean classifier Py (2) = arg max W ff where W E Beek
x is the weight of the linear classification
head, and for y [c]. the y-th W Ez (21) contains the mean representation of the class y. Arora et al. [25] empirically show that the mean classifier can obtain comparable performance to learnable linear heads. By default, we set the uniformity coefficient À = 1/(4L) as in Theorem 3.7.
label error as a = Ex.x, [1(21) ) + y(ū)]. Then, for / h E H (the hypothesis class) with h go f. the downstream classification error of its encoder can be upper bounded by its U-MAE pretraining loss:
Theorem 4.1. Denote the
Pr(ÿ + (T) S L. Lu.ME(()) + cza + toLL c4s
(14)
where C1....,C are constants and C3 >
This theorem provides an upper bound on the downstream error of an encoder f with its U-MAE pretraining loss, which the first theoretical guarantee on the downstream performance of MAE methods. As an implication of this theorem, small U-MAE loss would provably imply small downstream classification error, which helps explain the good downstream generalization ability of MAE [15]. Besides, we establish common lower bound on the U-MAE loss that holds for all h H. As a large common lower bound of U-MAE loss indicates that the downstream error will always have a large upper bound, we should pursue small common lower bound in the following theorem.
Theorem 4.2. The U-MAE pretraining loss has the following common lower bound:
N1
1
the H. (UMAE(h) 2
X e + const,
(15)
4L
i=k+1
where \ ....2 AN, denote the eigenvalues of A.
Combining Theorem 4.1 and Theorem 4.2, we can see that the downstream error of MAE can be minimized with a small vanilla autoencoding error small label error a, and smaller magnitude of "residual eigenvalues". ie.., (AR+1, AN, Here, residual eigenvalues represent the high frequency components of the augmentation graph GA that cannot be fitted by k-dimensional features. The vanilla autoencoding error & depends on the capacity of the chosen model class H. while the label error and residual eigenvalues (AR+1,.... AN,} purely depend on the mask mask-indueed augmentation graph A.. Therefore, overall speaking, MAE with large capacity can have a smaller downstream error with smaller autoencoding error s. In the meantime, the masking ratio P should be properly chosen to ensure small label error a and small magnitude of residual eigenvalues (Ak+I...., AN, }. Indeed, He al. [15] show that the mask ratio has a decisive influence on the downstream performance of MAE. Therefore, in the next part, we discuss how the choice of mask ratio would affect the downstream generalization of MAE via the label error a and residual eigenvalues (AR+1,..., AN.).
4.2 The Effect of Mask Ratio on Downstream Generalization
A surprising fact in MAE is that the optimal mask ratio is pretty high, e.g., = 0.75, where most image contents are lost. Therefore,
wondering why such high mask ratio is necessary and how it helps (instead of damaging) the learning of data representations. In this part, we provide both theoretical and empirical insights this problem based on our proposed theory.
Theoretical Insights. As discussed above, Theorems 4.1 & 4.2 suggest that a small label error a and small magnitude of residual eigenvalues (Ak41..... AN, are important for good downstream
7

---

Mask Ratio
0.25
0.75
0.95
Car
Car 2
92
24
0.5
0.
014
(a) Masked views with different mask ratio
(b) The distance between (c) The relative distance be- intra-class and inter-class tween intra-class samples samples
and inter-class samples
Figure 2: (a) Appropriate mask ratio can generate similar views from different samples in the same class. (b) The influence of mask ratio on the average l2 distance on ImageNet- 100 between intra-class and inter-class samples. (c) The influence of mask ratio on the relative distance between intra-class and inter-class samples.
performance in MAE. In particular, we notice that the mask ratio p has a decisive influence on both factors. On the one hand, the label error will grow with larger mask ratio, but it is only very large under an extremely high mask ratio. Intuitively, a represents the possibility of recovering the original class y from the left patches after masking. As illustrated in Figure 2(a), small medium mask ratio, even a large one of 0.75, hardly alters the belonging class, and only when the mask ratio is extremely high, e.g., 0.95, we can hardly tell it is still car. On the other hand, according the spectral graph theory [7]. the residual eigenvalues f AN, } represent the high frequency components of the graph, which have large magnitude when the graph is less connected, e.g., having many disjoint components. Therefore, a small downstream error requires the augmentation graph G to have better connectivity. In this aspect, Figure 2(a) shows that when the mask ratio increases, many dissimilar patterns are masked out and the left patches can have an increasing level of similarity, particularly among intra-class samples, e.g., the tires of the two cars. Therefore, large mask ratio can effectively improve the connectivity of the augmentation graph by reducing sample variation and increasing inter-sample similarity. Considering both the effects on the label error and the residual components
f16+1...., AN, J. we notice that there is tradeoff in the choice of mask ratio: we should choose properly large one to increase graph connectivity, and avoid too large mask ratio it leads to the class mixture. In other words, a guiding principle is that mask ratio should be chosen such that the mask-induced graph connectivity should happen mostly among intra-class samples (no harm) instead of inter-class samples (inducing large label error). Below, we further examine this understanding on real-world data.
Empirical Investigation. To verify our perspective, we conduct experiments on CIFAR-10 with different mask ratios. We evaluate the distance between two images by calculating the average l2 distance between every pair of patches. As shown in Figure 2(b). we find that the average distance of intra-class images decreases with the increasing of mask ratio, which means that higher mask ratio leads to generating more edges between intra-class samples with better connectivity (i.e., decreasing magnitude of residual eigenvalues {Ak+1 ., AN. 1). Meanwhile, we also notice that high mask ratio will also lead to a smaller inter-class distance, which suggests an increasing label error Therefore, there exists tradeoff on the mask ratio to balance residual eigenvalues AA++I,...,AN and labeling error
The Sweet Spot for Mask Ratio. The discussion above reveals that mask ratio has an effect on both intra-class and inter-class connectivity, and a good mask ratio should be selected to balance the two sides such that intra-class distance is relatively high while inter-class distance is relatively low. Motivated by this, we compute the relative distance (intra-class over inter-class). As shown in Figure 2(c), the relative distance decreases first with the increasing mask ratio, showing that the intra-class distance decreases faster than the inter-class distance under small mask ratio. When >> 0.7, the relative distance becomes larger again, indicating that the difference between intra-class and inter-class edges disappears under too large mask ratio. The sweet spot lies in p 0.7, which is pretty close to the optimal mask ratio of MAE (p 0.75). This shows that our theoretical analysis of the effect of mask ratio aligns surprisingly well with the practice of MAE.
8

---

Table 1: Linear evaluation accuracy (%) and fine-tuning accuracy (%) of pretrained models by MAE loss and U-MAE loss with different ViT backbones on CIFAR- 10, ImageNet- 100, and ImageNet- IK. The uniformity regularizer term in the U-MAE loss significantly improves the linear evaluation performance of the MAE loss without hurting the performance of fine-tuning accuracy.
CIFAR-I0
ImageNet-100
ImageNet-IK
Downstream Task
Method
ViT-Tiny
ViT-Base
ViT-Base
ViT-Large
ViT-Base
ViT-Large
MAE
59.6
61.7
61.2
64.4
55,4
62.2
Linear Probing
U-MAE
68.9
70.2
67.5
72.8
58,5
65.8
MAE U-MAE
89.6
90.7
86.9
87.3
82.9
83.3
Fine-tuning
89.4
90.8
86.8
87.3
83.0
83.2
5
Experiments
In this section, we first present the main empirical results of our proposed U-MAE loss on different real-world datasets with different backbones. Then we conduct a series of experiments understand how well the U-MAE loss works.
5.1 Evaluation on Benchmark Datasets
To evaluate the effectiveness of the proposed U-MAE loss, extensive experiments are conducted on CIFAR-10 [21], ImageNet-100 [8], and ImageNet- [8].
Setup. We mainly follow the basic setup of MAE [15]: for the encoder, we adopt different variants of flexible one following
VIT [10], Le. ViT-Tiny, ViT-Base, and ViT-Large. For the decoder,
[15]. The mask ratio is set to 0.75. For U-MAE, the coefficient of the uniformity term is set to 0.01. On CIFAR-10, 10, pretrain the model for 2000 epochs with batch size 4096 and weight decay 0.05. On ImageNet-Iee 100 and ImageNet-Ie IK, we pretrain the model for 200 epochs with batch size 1024 and weight decay 0.05. We conduct both linear evaluation and non-linear fine-tuning on the pretrained encoder. For linear evaluation, we train linear classifier on the frozen pretrained encoder. As for non-linear fine-tuning, we train both the pretrained encoder and the linear classifier with the cross entropy loss.
Effectiveness of the Proposed U-MAE Loss. In Table 1, we compare the performance of original МAE loss and U-MAE loss on different benchmarks. We find that, on linear evaluation results, proposed U-MAE loss increases 8.9% CIFAR-10. 7.2 % on ImageNet- 100, and 3.4% on ImageNet- K with two different backbones. On fine-tuning results, our proposed U-MAE loss will not hurt the performance of fine-tuning results of MAE. The experimental results verify the effectiveness of the proposed U-MAE loss, which achieves better performance than the original MAE loss across different datasets and different backbones. Moreover, quickly evaluate the training process of the encoder, we set an online linear classifier to monitor the linear accuracy whose results can be found in Appendix B.5,
Extention to Other MIM Frameworks. We also verify our proposed method in another MIM framework SimMIM [29]. For SimMIM, use VIT-Base the encoder and use the linear decoder as [29]. We use the recommended mask ratio 0.6. Similar to U-MAE, Uniformity-enhann SimMIM loss (U-SimMIM) adds uniformity regularization term to the original reconstruction loss of SimMIM, where the coefficient of the uniformity term is set to 0.01. For ImageNet-100, we pretrain the model for 200 epochs with batch size 128 and weight decay 0.05. Linear evaluation is conducted in Table 2. We can see that the linear accuracy increases 6.8% with the uniformity regularizer, which further verifies the general property of our approach.
Table 2: Linear probing accuracy (%) of U-SimMIM (ViT-Base) on ImageNet- 100.
Table 3: Linear probing accuracy (%) of U- MAE with different coefficients (Eq. (10)).
SimMIM U-SimMIM
X
0
le-3
le-2
le-1
1e0
54.3
61.1
Acc
61.2
65.9
67.5
45.1
47.0

---

MAE Loss
U-MAE Loss
Tedt Accuracy (15
60
60
Original MAE Loss
U-MAE Loss
50
100
150
200
Linear Accuracy 61.2
Linear Accuracy : 67.5
Epoch
(a) t-SNE representations
(b) Comparison between original MAE loss and our U-MAE loss along training.
Figure 3: (a) Visualization of representations on random 10 classes of ImageNet- 100 trained with MAE loss and U-MAE loss. Our U-MAE loss significantly improves the class-clustering performance of the encoder. (b) The linear evaluation results during the training process. It shows that our U-MAE loss improves the downstream performance consistently along training.
5.2 Empirical Understandings
Visualization of Representations. To intuitively understand the improvement of our U-MAE loss on clustering intra-class samples, we use t-SNE [26] to visualize the representations trained with MAE loss and U-MAE loss ten random class of ImageNet-100 (detailed classes introduced in Appendix B.4). We find that with our uniformity regularizer term, the samples are much better- clustered corresponding to their ground-truth labels. To be specific, the red class ("hens") and the gray class ("indigo birds") are separated from others, this is because most of other classes are the animals living in the oceans while these two classes are more like the birds living the land the sky. Thus, these two classes are easier to be distinguished, especially with our uniformity regularizer.
Different Coefficients of the Regularizer Term. The most important hyper-parametee of our proposed U-MAE loss is the coefficient of the uniformity regularizer term. In Table 3, we present the results of linear evaluation of the regularizer term. We can see that the downstream performance increases when the coefficient increases from 0 to 0.01. However, the overlarge coefficient will also hurt the performance of U-MAE loss as the task of MAE will be overlooked.
ImageNet-100 100 trained with the U-MAE loss with different coefficients
Training process. To further compare the performance between the original MAE loss and U-MAE loss, we plot the linear evaluation accuracy on ImageNet- 100 during the training process in Figure 3(b). We can observe that our proposed U-MAE loss improves the performance of MAE with all different training epochs, which verifies the effectiveness of our proposed U-MAE loss.
Conclusion
In this paper, proposed new theoretical understanding of MAE by formally connecting MAE and contrastive learning. In particular, we show that a small MAE loss implies a good alignment between the mask-induced positive samples. Based this connection, further analyze the dimensional collapse issue of MAE and propose a new variant, named Uniformity-enhanced MAE (U-MAE), through adding an explicit feature uniformity regularization. Theoretically, this connection enables us to establish theoretical guarantees on the downstream classification error, which also provides theoretical insights on the choice of large mask ratio in MAE. Empirically, the proposed U-MAE successfully mitigates the dimensional collapse issue of MAE, and achieves consistent improvement on the linear probing task on CIFAR-10, ImgaeNet-1000 and ImageNet-
Acknowledgment
Yisen Wang is partially supported by the NSF China (No. 62006153), Project 2020BD006 supported by PKU-Baidu Fund, and Huawei Technologies Inc. We thank anonymous reviewers for constructive and helpful discussions.
10

---

References
[1] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers, In ICLR, 2022.
[2] Shuhao Cao, Peng Xu, and David A. Clifton. How to understand masked autoencoders. CoRR, 2022.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. In ICML, 2020.
[5] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models strong semi semi-supervised learners. In NeurlPS, 2020.
[6] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
[7] Fan RK Chung. Spectral graph theory, volume 92. American Mathematical Soc., 1997.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
[9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-IILT, 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[11] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-218, 1936.
[12] Jean-Bastien Grill, Florian Strub, Florent Altché, C. Tallec, Pierre H. Richemond, Elena Buchatskaya, C. Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: a new approach to self-supervised learning. In NeurIPS, 2020.
[13] Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
[14] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint wrXiv:2111.06377 2021.
[16] Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation in self-supervised learning. In CVPR, 2021.
[17] Li Jing. Jure Zbontar, et al. Implicit rank-minimizing autoencoder. In NeurIPS, 2020.
[18] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. ICLR, 2022.
[19] Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In ICLR, 2014.
[20] Lingpeng Kong, Cyprien de Masson d' Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. A mutual information maximization perspective of language representation learning. ICLR, 2020.
[21] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

---

[22] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 0081v:1807.03748, 2018.
[23] Jiachun Pan, Pan Zhou, and Shuicheng Yan. Towards understanding why mask-reconstrruttin pretraining helps in downstream tasks. arXiv preprint arXiv:2206. 2022.
[24] Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In EUSIPCO, 2007.
[25] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. A theoretical analysis of contrastive unsupervised representation learning. In ICML, 2019.
[26] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(66):557--26/) 260,,, 2008.
[27] Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. In NeurlPS, 2021.
[28] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.
[29] Zhenda Xie, Zheng Zhang. Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In CVPR, 2022.
[30] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-fo-sequence functions? In ICLR, 2020.
[31] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In ICML, 2021.
[32] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In ICLR, 2022.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] See Section 5.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "hdfkkUQH--B", "reviewer": "Reviewer_M1jb", "summary": "This paper attempts to understand how masked autoencoder (MAE) works and investigates why the masking ratio of MAE is important. On top of the analysis, the paper proposes a uniformity-promoting MAE loss which additionally regularizes the loss by enforcing uniformity. In experiments, the uniformity-promoting MAE shows better performance than the original MAE on CIFAR-10 and ImageNet-100 datasets. ", "questions": " - I wonder if the investigation and solution (U-MAE) can be transferred to other masked image modeling methods like BEiT [1], iBOT [2], SimMIM [3], and so on. Discussion about the transferability of those methods and relationships among such masked image modeling methods will be interesting. \n\n[1] Bao et al., Beit: Bert pre-training of image transformers, ICLR 2022\n[2] Zhou et al., iBOT: Image BERT Pre-Training with Online Tokenizer, ICLR 2022\n[3] Xie et al., SimMIM: A Simple Framework for Masked Image Modeling., CVPR 2022", "limitations": "the review will be updated in a couple of days.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n\t- Theoretical analysis of masked autoencoder gives somewhat new perspective and insight. \n\t- Proposed uniformity-promoting MAE seems a reasonable loss design and works well on CIFAR-10 and ImageNet-100. \n- Weaknesses\n\t- The experiments are weak, so it is hard to convince the paper's main claim. As in table 1, the experiments are conducted on CIFAR-10 and ImageNet-100 which are not standard for evaluating self-supervised learning methods. It is necessary to conduct at least one comparison with MAE *on ImageNet-1k* as in the original MAE paper.  \n\t- The main formulation and analysis are based upon the MAE framework, so I wonder if the observations and knowledge can be transferred to other masked image modeling methods like BEiT [1], iBOT [2], and SimMIM [3], and so on. Discussion about the transferability of those methods and relationships among such masked image modeling methods will be interesting. \n\t- It is hard to convince the claim in Figures 2 and 3(a) saying \"the higher mask ratio will generate more similar views of intra-class samples.\" at line 241. Figure 2 seems to cherrypick samples. It is not guaranteed that the surviving patches have similar semantic meanings in natural image datasets like ImageNet. In figure 3, a smaller L2 distance at a higher mask ratio may come from the loss of variety and diversity of image patches. \n\n\n[1] Bao et al., Beit: Bert pre-training of image transformers, ICLR 2022\n[2] Zhou et al., iBOT: Image BERT Pre-Training with Online Tokenizer, ICLR 2022\n[3] Xie et al., SimMIM: A Simple Framework for Masked Image Modeling., CVPR 2022", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a7FZ8Y_NqBx", "reviewer": "Reviewer_wuSY", "summary": "The authors propose a new theoretical understanding of Masked Autoencoders (MAE). From the theoretical perspective, this paper is interesting and novel. I agree with their contributions as stated in the paper.\n\n1. They prove that MAE loss is upper bounded by an implicit alignment loss and develop a new understanding of how MAE bridges the samples in the same class. Based on their analysis, they propose a new U-MAE loss.\n\n2. They establish a guarantee for the downstream performance of MAE with practical assumptions and compare the training process of contrastive learning and MAE.\n\n3. They verify that their U-MAE loss can significantly improve the performance of MAE on two datasets CIFAR-10 and ImaegNet-100. \n\n As a very early attempt, this paper provides theoretical contributions for the MAE study. Both the \"How to Understand Masked Autoencoders\" paper and this paper provide insights for the theoretical research for the MAE community.", "questions": "Suggestions can be found in the above.", "limitations": "Training MAE from scratch is very resource-consuming. But for future work, it is still better to do experiments on the large-scale datasets, like ImageNet-1K used in the MAE paper.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nMAE looks like a very straightforward model. But how to extract the mathematical essence of the simple model is very challenging. This paper contributes a novel understanding of MAE, and provides the experiments.   In particular, the authors propose a new theoretical understanding of why the choice of mask ratio is so important for MAE from a graph perspective. This is very insightful and inspiring.\n\nWriting is clear and easy to follow.\n\n\nWeaknesses -- some suggestions and minor issues.\n\n1. The mathematical formulations can be improved. \n\nFor example, Line 95 [k] and Line 105 [.] will make the readers confused about the brackets. \n\nIn Line 100, the tilde can be replaced as $\\in$, as D_u is defined as a set, not a distribution.\n\nIn Eq. (1), x_1 is defined as a vector, so it is better to use bold to denote the vector.\n\n2. More implementation details. \nIt is very interesting to see more training details, including how many GPUs, and how many hours for training. \n\n3. More explanation on experimental results.\n\nFor example, in Figure 4 (2), U-MAE loss obtained better visualizations. There are two clusters (grey and red) that are separated from other clusters. Why are they so distinctive from the others?  What are the 10 random classes?  What classes are the grey and red clusters presenting? What are the natures of these two classes in the original data domain?  Further analysis can give us more motivation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TypGK4EJiM6", "reviewer": "Reviewer_8imA", "summary": "The paper presents theoretical results of MAE, a recently popular approach for self-supervised visual representation learning. Two insights are provided: 1) MAE loss can be bounded by an implicit alignment loss, and switching to this loss appears to be helpful for MAE pre-training; 2) masking at a high ratio implicitly bridges different instances of the same class. Experiments are mainly done on small-scale datasets, CIFAR-10 and ImageNet-100.", "questions": "- Clarification: how is $f_g$ obtained in the U-MAE loss? According to the theory it seems $f_g$ is a hypothetical function that inverts the decoder, but it is not clear how such a decoder can be trained empirically.\n- Rebuttal request: I don't think Figure 3 is convincing enough. It just shows the L2 distances with intra-class examples and intra-instance views are decreasing as the mask ratio decreases. However, it is not clear what's the trend of the *overall* distances in the entire dataset, and what's the trend of the inter-class examples. I think it is more important to show the *relative* trend between intra-class and inter-class, rather than the *absolute* trend.", "limitations": "It's a theoretical paper, I do not find the paper addressing limitations and potential negative impact, which is fine to me.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) Analyzing and understanding state-of-the-art approaches, especially in self-supervised learning is of great significance.\n\n(+) While I have not checked the math carefully, I find the insights discovered in the paper interesting, and it also appears to be quite well-supported by the experiments.\n\n(-) The paper presents two results, one by treating MAE as implicit contrastive learning (Section 3), and one by building on a theory of generalization. While both are interesting, it is generally making the paper less focused (it could just be two separate works instead).\n\n(-) I don't like the writing style of the paper, where the derivations appear in the main text. I would hope to see clean results, key equations, and detailed insights in the main paper, and leave the derivation part in the appendix.\n\n(-) Figure 1 appears to be not so necessary, as MAE paper has already shown a similar trend on the standard, large-scale dataset ImageNet-1K. \n\n(-) Figure 2 is a bit artificial and less *convincing*. The visualization just shows a particular pair of examples. Many concerns can arise: what if the intra-class variation is so big that no shared, similar views can be found between two examples? what if the shared views are across two examples from *different* classes?\n\nOverall, I think the paper is presenting some interesting results and findings, but I don't think they are solid enough from the perspective of empirical justifications. I do hope my concerns can be addressed so that I am no longer on the fence for this paper. \n  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-lX_PfRO2Z", "reviewer": "Reviewer_qc3H", "summary": "The paper aims to provide a theoretical underpinning to explain the success of masked autoencoders (MAEs) in computer vision, with a particular focus on the importance of the mask ratio. To this end, the authors establish a connection between MAE training and contrastive self-supervised learning objectives, and leverage graph theory to establish a generalization bound. Furthermore, based on their analysis the authors propose a regularizer and test its effectiveness empirically, observing benefits on multiple data sets.", "questions": "- What is $\\tilde x_2^-$ in equation (9)? How does the last term relate to uniformity as suggested by the terminology uniformity-promoting MAE? I could see how it relates to alignment.\n- Why is (5) greater or equal the preceding expression? It seems it would be equal to the preceding expression.\n- How are the graph weights define exactly? $M_1$ and $M_2$n are probability distributions if I read the definitions correctly, so the definition in (10) wouldn’t resolve to a scalar weight.\n- What visualization technique is used to generate the plots in Figure 4? What data set are the results in Figure 5 obtained for?\n", "limitations": "It is unclear whether the approach really captures the mechanism underlying the success of MAE. At least I’m not convinced that the potentially incorrect derivations give additional insights on top of the intuitions e.g. about masking ratios given in the original MAE paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I believe the search for a connection between contrastive self-supervised learning and reconstruction based approaches like MAEs is interesting and could turn out fruitful. The attempt to link the Siamese structure in contrastive SSL with the encoder-decoder architecture used in MAEs looks promising at first. However, I have doubts about different elements of the derivation of the alignment, as well as the presentation and the experiments.\n\nThe major issues I found are the following:\n- It seems that then MAE encoder and decoder are incorretly defined in L93: Using the notation from the laper, the encoder $f$ maps $|m|$ image patches to $|m|$ tokens, which are then interleaved with $n - |m|$ mask tokens according to the locations of the masked patches and augmented with positional encodings. The resulting n tokens are then fed to the decoder $g$ which outputs $n$ patches. The authors then define $x_1$ corresponding to the stacked unmasked patches, and $x_2$ corresponding to the stacked masked masked patches. This means that $x_1$ and $x_2$ have different shapes unless the masking ratio is 0.5. As a consequence, the Lipschitz property cannot be applied as in L151 because $f(x_1, p_{x_1})$ and $f_g(x_2, p_{x_2})$ do not map into the same space. This would mean that the proof breaks down at this point. Interleaving $f$ and $f_g$ with the masked and unmasked tokens would bring them to comparable spaces, but considering the distance between the interleaved representations would not make much sense as this would amount to simply measuring the magnitude of the unmasked representation. I’m happy to change my assessment, but I don’t see how this can be fixed.\n- Concepts from prior work are often unsufficiently explained, e.g. the derivation of the spectral graph theory based generalization bound and the related quantities (e.g. the authors talk about why the graph they consider is directed before it is clear what the relevance of the graph is). Notations are often defined after they are used for the first time, others are not introduced at all such as some quantities in Theorems 4.1 and 4.2.\n- It is often unclear what the assumptions are and what is proven by theory. I believe a better way to structure the paper would be to clearly state the assumptions and then why they should be satisfied in practice (e.g. w.r.t. the Lipschitz assumption).\n\n\nMinor weaknesses:\n- L163 (and later) “regularizer” instead of “regular”\n- The last parenthesis in (5) should be deleted.\n- The choice of $f_g$ is important for the proposed regularization loss, but how it is chosen is only described in the appendix. It would be better to move the description to the main paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept