Self-Guided Masked Autoencoder
Authors: Jeongwoo Shin
Inseo Lee
Junho Lee
Joonseok Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: self-supervised learning
self-supervision
SSL
representation learning
Primary area: machine_vision

Abstract: Masked Autoencoder (MAE) is a self-supervised approach for representation learning, widely applicable to a variety of downstream tasks in computer vision. In spite of its success, it is still not fully uncovered what and how MAE exactly learns. In this paper, with an in-depth analysis, we discover that MAE intrinsically learns pattern-based patch-level clustering from surprisingly early stages of pre-training. Upon this understanding, we propose self-guided masked autoencoder, which internally generates informed mask by utilizing its progress in patch clustering, substituting the naive random masking of the vanilla MAE. Our approach significantly boosts its learning process without relying on any external models or supplementary information, keeping the benefit of self-supervised nature of MAE intact. Comprehensive experiments on various downstream tasks verify the effectiveness of the proposed method.

Self-Guided Masked Autoencoder
Jeongwoo Shin', Inseo Lee', Junho Lee', Joonseok Lee!:a.
SSooul National University, "Google Research {swswss, ian. lee, joon2003, joonseok)osmu. ac.
Abstract
Masked Autoencoder (MAE) is self-supervised approach for representation learn- ing. widely applicable to a variety of downstream tasks in computer vision. In spite of its success, it is still not fully uncovered what and how MAR exactly learns. In this paper, with an in-depth analysis, we discover that MAE intrinsically learns pattern-based patch-level clustering from surprisingly early stages of pre- training. Upon this understanding, we propose self- geff-guided masked autoencoder, which internally generates informed mask by utilizing its progress in patch clus- tering, substituting the naive random masking of the vanilla MAE. Our approach significantly boosts its learning process without relying on any external models or supplementary information, keeping the benefit of self- seff-supervised nature of MAE intact. Comprehensive experiments on various downstream tasks verify the effectiveness of the proposed method.
Introduction
Self-supervised learning has been an attractive direction to alleviate the substantial cost for data annotation. For example, Masked Language Modeling (MLM), predicting masked words of an input sentence, is demonstrated to capture contextual meaning of word by BERT 13] and GPT [8] Motivated from the success of MLM, Masked Image Modeling (MIM) has been introduced in computer vision, utilizing abundant unlabeled image data. Among them, Masked Autoencoder (MAE) 222, equipped with Vision Transformer (ViT) [15 -based asymmetric encoder-decoder structure, demonstrates that simple reconstruction of the RGB pixels for the masked patches enough to achieve competitive performance various downstream tasks.
In the wake of MAE's impressive performance, succession of studies have emerged aiming to augment its capabilities through the integration of informed masking techniques. These innovative endeavors leverage diverse sources of additional information, including attention maps generated by supervised VIT [28 knowledge learned by pre-trained self-supervised models [32 10 or
supplementary adversarial modules [45], all aiming at refining the quality of the masks. However, these prevailing approaches have merely applied informed masking without truly understanding the
mechanism of MAE, relying on external resources such as pre-trained models or labels.
To this end, we embark on an in-depth analysis through extensive experiments to understand the internal operation of MAE, is still fully uncovered what and how MAE exactly learns, despite the several prior endeavors [9] 41 61, 29]. Based on our analysis of MAE, we then explore the potential of MAE produce informed masks on its own. We first demonstrate that MAE intrinsically learns pattern-based patch-level clustering and this property emerges from extremely early stages of pre-training (Section 3.3). We then unveil the underlying mechanism of the mask tokens in the decoder (Section 3.4). Upon this understanding, we propose novel method to boost the training process of MAE via informed masks, generated an entirely unsupervised manner without incurring any external models supplementary information, unlike the previous informed masking methods.
Coorrsponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Input
(b) Random masking
(c) Bi-partitioned
Informed masking
(i) Informed with hint
image
feature
fealure
Figure 1: Illustration of our self-guided MAE.
Figure illustrates our model compared to the original MAE. Unlike the random masking (b), method generates informed masks covering the main object entirely (e-f) using the distinguishable patch representations (d) emerging from very early stage of the training. With the internally produced informed masks, MAE accelerates its training process of learning patch-level clustering, leading to clearer and finer embedding space (c, g). Our contributions are summarized follows:
We discover that MAE learns pattern- -bared patch-level clustering within each image, emerging from incredibly early stage of the pre-training process.
We propose new masking strategy, guided masked autoencoder, relying solely on internal quantification of the progress in patch-clustering, freee from external models or labels.
Our comprehensive experiments across various downstream tasks validate that our proposed method genuinely expedites the learning process of MAE.
2 Preliminary
Masked Autoencoder (MAE). MAE [22 aims to learn task-agnostic feature representations for various downstream vision tasks, e.g., classification, detection, or segmentation.
Given an image of size H x W, MAE first splits it into to same-sized P x P image patches. Each patch is linearly mapped to a d-dimensional embedding. As result, the input image represented as set of these features, denoted by X' "x."),..,xx"" (()), (x) Rd), where number of patches. We call (8) masks out a subset of patches in X. The set of masked and visible patches are denoted by X'm and Xy, respectively, where Xx, U X, X and Xm, X
HW/P p2 the 'patch' or 'token' embedding interchangeably. MAE randomly
MAE adopts an asymmetric incodrrdeccoer structure based on ViT 15] The encoder E takes X, as input and produces a same-sized set of embeddings, denoted by X = [x((1) x E(X,)) Through the encoding, the patch representations
updated to reflect the context of the entire image, only from the visible parts. Then, the decoder takes a set of n patch embeddings, denoted by
X = (t) (()) m if (() Xm, (i) = ( (i) E X. as input, where m Rd is learnable mask token. Each x X.m is substituted with a mask token m, and corresponding positional encoding is applied to distinguish them. The decoder targets to reconstruct the raw RGB pixels of Xm,. Once trained, only the encoder is deployed for downstream tasks.
Hierarchical Latent Variable Model. Kong et al. [29 recently discovers that the internal operation of MAE can be explained under the framework of a hierarchical latent variable model. There exists high-level shared information the input image, and equivalent to statistical dependency among the patches in N. The MAE encoder E E(X.) learns high-level latent variables by estimating

---

MAE Enc.
MAE Dec.
MoCo
VIT
MAE Ebc. MAE Dec.
MoCo
VIT
1.0
(a) Similarity score matrix (M. 196 x 196)
(a) Simllrityy score matrix (M, 196 x 196)
0.8
Image
0.4
0.4
(b) Similarity score patch (14 14)
0.2
(b) Similarity score to patch (14 x 14)
0.2
0.0
0.0
Imuge
(c) Attention CLS 14)
(c) Attention
(14 14)
Representations of image
Representations of image 2
Figure 2:
Relationships among the patch embeddings. (a) Pairwise similarity matrix for all 196 x 196 pairs of patches. (b) Similarity between the mean patch and all individual patches. (c)
Attention score of the class token.
the shared information è from the visible patches Xe, and the decoder D([E(X,); ml) performs the reconstruction task by inducing Xm from via the mask tokens.
Analysis of MAE
We study what and how MAE learns with concept of token relation (Section 3.1 and demon- strate that it learns pattern-based patch-level clustering (Section 3.2) from early stages of training (Section 3.3 We then illuminate the underlying mechanism of the MAE decoder (Section 3.4). In this section, we use the VIT-B MAE 22 pre-trained for 400 epochs on ImageNet-1K [2 and all experiments have been conducted 10% of ImageNet-Ii training set, unless noted otherwise.
3.1 Token Relation
In order to understand what MAE learns, we analyze its token embeddings with their quantified pair-wise relationships, ie., attention score matrix A and cosine similarity matrix M. For the input patches X E [n and the transformer weights WiQ.KVV E Rddd' for queries, keys, and values, respectively, E m and M R are given by
A Softmax X X (XW*)//XX /Vd)' (I)
M
| 2
X 12112
where X' = A(XWV) E Raxd' and x E Rd' indicates the i-th row of X'.
We present analysis with A and M under two settings. First, calculate them using complete of patches X' at the encoder as EXx). This ideal setting offers the most accurate è, suitable to analyze the features learned by MAE. As alternative, they can obtained from the practical setting of the decoder ((E((X)) ml) which contains mask tokens. Since only the visible tokens are exploited to estimate è, this setting would produce less accurate token relations compared to the former one.
3.2 What is Learned by MAE?
We investigate the distribution of patch relationships in the learned embedding space, using the last layer embeddings of EXX) and D(EE(X,;; ml)
(.) for 196 (14 14) patches of set-aside test images.
Qualitative Analysis. In Figure we compare the patch representations among different models (MAE, MoCo (23), and ViT [15]). Figure 2a depicts the normalized pairwise cosine similarity matrix (M) for all 196 196 patch pairs for a test image. The MAE encoder shows more polarized values, i.e., higher variance, indicating that patches are more clearly clustered. Figure 2b illustrates the cosine similarity between the mean of patches and all individual patches. In the examples in Figure 2 the background patches are majority, so the mean patch is closer the background. Patches

---

corresponding to the main object clearly show lower similarity to the mean (background), indicating that the MAE encoder has learned patch clustering based on visual patterss, Lee, texture and color. Similar results in the projected latent space are provided in Appendix B Figure 2c shows the attention scores of the class ([CLS]) token. As the class token not updated during self-supervised training, it does not carry particularly meaningful information and therefore could be regarded as a random vector. As a result, the class token does not lean towards any specific patch under self-supervision, and thus the score is distributed similarly to the relationship with the mean patch in Figure 2b In contrast, MoCo and ViT fail to clearly distinguish the patterns among the whole patches.
Despite the limited information X., the decoder also exhibits proficiency in grouping patches based on their patterns, albeit not as effective the encoder.
Quantitative Analysis. We additionally measure the feature variance (ar) and variance of the pairwise similarities (as), on the ImageNet-1K validation set:
(M.j M)² n(n -
x)
OF
OS
X
where x = + E, x x and M my Eij M. Higher OF indicates patch embeddings are
spread out more widely in the feature space, while higher as indicates stronger patch clustering.
In Table D the MAE encoder and decoder show signif- icantly higher OF and as compared to MoCo and ViT, suggesting that their patch embeddings are more diversely clustered in the embedding space rather than a sim- pler alternative, e.g., bi-partition. Given the significant utilization of fegeffreyenccy information (e.g.. pattern or quantitatively con-
Table 1: Feature variance (aF)) and sim-
ilarity variance (as).
Feature
E|or]
B[as]
MAE encoder
0.08
0.075
MAE decoder
0.11
0.059
texture) in MAE (Figure 7 firm that MAE effectively clusters patches based on their patterns. MoCo and ViT show significantly lower or and Øs. as they tend to learn a simpler form of feature maps, aligned with 42 To alleviate the concern that the large variance might be result of few extremely clustered features, instead of good separability, we additionally measure Normalized Mutual Information (NMI) [46] between queries and keys, which is an indicator of homogeneity in attention map [42). As shown in Figure 6 we confirm with non-zero NMI that MAE does not collapse to a few extremely separated feature groups.
MoCo 23
0.01
0.003
VIT 15
0.02
0.012
Summary. MAE learns patch-level clustering in an image based on their visual patterns. Operating only with visible tokens, the decoder learns a similar but less clearer trend than the encoder.
3.3 When Does MAE Learn Patch Clustering?
Given that the MAE learns patch clustering upon completion of pre-training, when does it start to learn them in pre-training? We answer this question by tracking the token relations of MAE
Evolving Bi-partitioning across Training. We start with the simplest form of token clusters, i.e., bi-partitioning. We cluster the patches into the two most prominent sub-groups by applying graph-cut to M from the final layer. Based on this clustering, we trace the mean of inter-cluster edge weights (Jimte) and mean of intra-cluster edge weights (Finma) with M and A, across the training.
Figure Ba shows Pinter and Pintra measured with M and A. We observe two notable patterns regarding the gap Minta - Minter: 1) the gap tends get larger along with the training steps, more prominently with the attention scores. 2) there is a clear margin between jintra and jinter from very early stages. The decoder also shows similar but less prominent trend.
Convergence of Token Relations. Going beyond investigating token clusters, we directly track the gap between the distribution of token relations during the training vs. upon completion. Specifically. we consider the mean KL divergence di()) in the i-th layer at the j-th epoch over a set of images D:
ō..5 [D]
Σ LED
(R.(RR) (T)R (1).
where N is total epochs and R (0) token relation matrix (e.g., M or A) computed with the i-th layer embeddings at the j-th epoch.
R #xxx3 RaX is a function mapping an input image I to

---

Enceder intra --- Encoder inter
-
Decuder intra - Decoder inter
Mean edge
0.010
0.008
0.4
0.006-
0.004
3
0.002
220
260
340
220 Epochs
280
340
400
Epochs
Similarity score
Attention score
(a) Bi-partitioning performance
Encoder 5
0.12
Decoder B
0.12
Encoder Ib 0:12
0.12
10
10
Decoder
0.08-
0.08
0.04
0.00
0.00
0.00
0.00
100
200 300-400 400
100 200 300 400
100 200 Sho 400 Epochs
200 son 4om
Epochs
Epochs
Epochs
Similarity score
Attention score
(b) KL divergence of token relations
Figure MAE learns patch clustering from very early stage of training process. (a) MAE widens the gap Jiinrr Pinter- (b) Token relations drastically converge at early epochs and then gradually
level off. Numbers in the legend denote the layer i. More details are provided in Appendix B
Figure 3b depicts 8;( for even-numbered layers up to 400 epochs, measured with M and A. clearly shows that 8 monotonically decreases, converging quickly early epochs, indicating that the patches begin to be clustered from early epochs. This result strongly implies that MAE learns the token relations at early epochs and progressively strengthens it along the rest of training. The decoder also shows a similar trend, but with less prominence.
Summary. MAE learns to cluster the patches from the early stage of training.
3.4 Operations of the Decoder
In previous experiments, we observe that the decoder ((((())); in the practical setting is still able to build complete token relation, which verifies that the decoder exploits the estimated shared information conveyed from the encoder F((X.) to complement the missing information in masked-out tokens (X,,) and reconstruct them. Connecting this to our discovery in Section 3.2 we claim that the pattern-based patch clustering learned by MAE conceptually corresponds this If the encoder is trained sufficiently, its output embeddings X" for the visible tokens would convey the general context (i.e., é) of the entire image. Then, through the decoding process, mask tokens are contextualized by selectively attending to X thereby possessing the essential information to represent the target patches X,,, originally derived from Therefore, by reversing this process, we assess if the encoder has been sufficiently trained to precisely associate the patches by quantifying deployed in (X²): ml). which is estimated by E(X)). Based on this idea, we propose a novel metric to measure it during training, which will be the key
proposed method in Section 4
Exploitation Rate. We pose that the overall attention weight on mask tokens the decoder is good indicator to quantify the amount of è utilized by the decoder. Specifically, we define the exploitation rate of the mask tokens over the decoder layers using the attention score matrix A (Equation []. which can be interpreted the special case of attention rollout [1. For the sets of token indices A and B, the exploitation rate T E E of the tokens in A construct the tokens in the 1-th layer is defined as the average attention weights relying on the tokens in A for the tokens in B:
(1)
ΣΣ
(5)
A-
iEB
5

---

where 1 = i.... L is the layer index, and A is the attention score matrix at the I-th layer. For A and B, we are interested in the set of mask tokens M = {i: x( Jm of visible tokens = {i: x()) X, and of all tokens O {1, (l...., n... For example, r!l r--v 0.7 indicates that the contextualized visible tokens consist of mask tokens (70%) and visible tokens (30%) on average.
Then, recursively accumulate these ratio across all layers to get the overall exploitation rate of the mask tokens. Formally, the accumulated exploitation rate Rell A- E IR of the tokens in a set A to construct the tokens in a set B up to the 1-th layer is defined by
RA-
(1) AA R TB B-B A-
(1) (1-1)
(6)
where 1 1. At the 1-th layer, tokens in B (denoted by B¹) consist of tokens in both A and B from the previous layer, A and B with their respective ratios of (1) left term (1) " R( means the overall exploitation rate of tokens in A to construct those in B',
"A-B and (1()) Thus, the
coming through A 1 and similarly, BB-8" - indicates that coming through B
Finally, the proportion of information from the visible tokens (R and that from the mask tokens (1e() (L) after 1-th layer with masking ratio of m are given by
R Ry_
R Ry-M + (1 m)
B --V
(1) M = m 1 M- + (1 m) R RM- (7)
Empirical Analysis. We measure the exploitation rate of visible tokens (RiLLo) and mask tokens (RM=)) in Figure 4 Surprisingly, R!!)) -M surpasses R2 after some moment, denoted by T' Heavy exploitation the mask tokens in every decoder layer strongly indicates that they truly hold substantial amount of shared information estimated by the encoder, which is more valuable than simple interpolation of visible patches (before T epochs) to represent masked out patches. We observe T == 50, but this may differ depending on the model dataset.
0.8
0.6
Layer
Rate
Layer 2
Layer
Layer
0.2-
40 T
100
400
800
Epochs
Figure 4: Exploitation rate.
Summary, When the encoder is sufficiently trained cluster patches, the encoder outputs reflect the shared information, and they are utilized to constitute mask tokens in the decoder. This means the mask tokens possess this patch clustering information and start to be intensely exploited for reconstructing masked-out patches. Thus, the mask tokens in the decoder that the mask tokens have patch clustering information conveyed from the encoder, sufficiently to cluster the patches. This process is verified via measuring the shared information learned by the encoder E(X,) during training by tracking the accumulated exploitation rate in Equation]] 7 Heavy exploitation of the mask tokens in decoder after T zs 50 implies that the encoder presently trained sufficiently to cluster the patches.
conversely infer from high exploitation rate of
Self-Guided Informed Masking
In Section 3.2 and Section 3.3 show that the MAE encoder learns patch clustering from early stage, allowing us to appropriately bi-partition the image into two major token clusters and to mask out one of them. In other words, can generate informed masks with MAE itself early in the pre-training phase and use these informed masks for the remainder of the training. To decide when exactly the MAE can properly cluster the patches, we use exploitation rate suggested Section (.44 which allows us to confidently generate informed masks at T epoch, ultimately leading to the design of our method.
From these observations in Section 3) we are motivated to leverage the patch relevance learned from the carly-stage expedite training, instead of relying random masking. Random masking delays inefficiently revisiting easily separable patches already
the learning of powerful patch-clusters clustered in the early stage which reflects the key dissimilarities among the image tokens.
Based on this idea, we propose our self-guided informed masking, which internally injects the information about the learned key dissimilarities by intensively masking one of the top two well-

---

Image
Rondom
Dipartiionn
Ours (lto))
Figure 5: Examples of Self-guided Informed masking, More examples and detailed explanations on method are displayed in Appendix A
separated clusters. We emphasize that MAE is still trained in a single stage; at epoch T, we begin generating informed masks and continue the training process without interruption.
Armed with our method, we can accelerate MAE focus on learning less distinguishable patches instead of wasting time repeating to discover the most prominent patterns. As our method purely relies on inherent metries during training, is completely free from any external models or extra information. A more detailed reasoning can be found in Appendix A
To achieve this, need to 1) bi-partition the image, 2) properly design the informed masks, 3) select the attention layer to construct the informed masks, and 4) decide when to start informed masking.
Bi-partitioning, To bi- sirpartiiion the image reflccing the learned key dissimilarities, we take Nor- malized Cut (Ncut) 44 to consider both dissimilarity between different clusters and similarity inside each cluster. We construct fully-connected undirected image graph with M (Equation 2 which consists of the patches and similarity between them as nodes and edges, respectively. To partition the set of all node indices O into two disjoint sets A and B, we minimize the Neut energy ((A, B)/S(A O) + ((A, B3/ (B, O), where ( 13) = EiEA,EB M.,-- As shown in [44], we can approximate the solution of this problem by calculating the second smallest eigenvector (y1) of the eigensystem (D M)y ADy. where D is diagonal matrix with D E, M.,, and D M is the Laplacian matrix. Finally, we bi- bi-partition the graph by thresholding y with its mean, yi. i.e. A
yily 2 yr} and = {ily < yr).
Object-centric Masking, Our approach stems from the observation that masking the entire image leads to learn patch clustering across the entire image; i.e., the reconstruction loss affects the whole image. To refine this process, we restrict the masking to object-centric regions. By narrowing the masking focus, our method guides the MAE to concentrate on learning patch clustering within the object regions; that is, the loss affects only the object-related parts, thereby accelerating the process of learning patch clustering object region. In this context, we aim mask out the cluster containing the main object, letting the model to learn feature representations for the foreground faster. Since we do not have access to the label, we take an indirect approach: the token with the largest absolute element in y tends to compose the main object. (55)
In reality, however, practical issues like imperfect bi-partitioning and varying cluster sizes complicat- ing batch processing, arise. To address, we rank the tokens by the relevance to C and mask out fixed ratio of the tokens based on the ranking. The relevance score S, of the i-th patch is defined
S.
where Xn CC
Σ 100
Xm 1l2 ||x00112'
As Xm represents the majority of C which mainly consists of the object, S, exhibits the relevance to the object by measuring the similarity of each patch to Xm. this manner, we can robustly mask out the whole object even with the noisy bi- partitiiion....
7

---

Though intensive masking on object leads to expedited feature learning on it, reconstruction fundamentally impossible if all tokens are masked, as there no clue to construct high-level information about it. To prevent such case, we add a few hint tokens, chosen either by uniformly randomly or proportional to S, with decaying ratio, following [32.. centric informed masking (4th row) based on bi-partitioning (3rd row). We provide minimum information for estimating shared information via the hint tokens (last row).
our object-
Appropriate Layer for Patch Clustering. We consider attention distance and Normalized Mutual Information (NMI) in Figure to decide with which layer embeddings to compute the similarity matrix among the patches. To obtain sufficiently meaningful token relation, we discard the early layers (-4th layer) from the candidates as they show extremely low NMI, which indicates relatively ho-
1.0-
Normalizedt lized value
0.8
0.6-
0.4
Altention distance
Normalized MI
mogeneous attention map. Then, we select the second last encoder layer with the highest attention distance, they have the highest potential awareness the global patterns.
0.0
3
6
121
Encoder
Decoder
Figure 6: MAE properties.
When to Start Informed Masking. Lastly, we need to determine when the model has learned patch clustering enough generate high-quality informed masks. Since the relative relationship among the tokens is enough for bi bi-partitioning, we generate them when the mask tokens start have comparable amount of information to the visible tokens. Based on Section 3.4 we quantify the shared information possessed by the mask tokens and start informed masking when it becomes comparable the information in visible tokens (1E(4) M- 2 R(L) R(22)) i.e, around T epochs in Figure
Experiments
5.1 Experimental Settings
Baselines. We compare our model to the original MAE [22] and Attention-driven Masking and Throwing (AMT) [88, recently enhanced MAE without requiring any external model label information. We exclude other models requiring external pre-trained models [10 32 or labeled data (8 for generating informed masks, since it is longer fair comparison.
We pre-train all competing models for 400 epochs on ImageNet- 12] and fine-tune on downstream tasks: image classification, object detection, and semantic segmentation. All experiments are conducted following the settings in original MAE 22, unless noted otherwise. We conduct experiments 8 NVidia A6000 GPUs (48GB).
Experimental Protocol.
Datasets. We use CIFAR-100 30
iNaturalist 2019 48 and CUB200-2011 49 for image clas-
sification. We fine-tune our model on COCO semantic segmentation.
36 for object detection, and on ADE20K 62 for
5.2 Performance on Downstream Tasks
Image Classification. The left side of Table compares the image classification performance on various datasets. Our method outperforms the baselines all datasets with both linear probing and fine-tuning, implying that the expedited training with our method leads to stronger feature representation after same epochs of training. We provide further analysis our boosted performance in Section 5.4 and extended training results in Appendix
Object Detection and Segmentation. We fine-tune a Mask R-CNN model (24 end-to-end on COCO with a ViT backbone for 90K iterations, and evaluate with the average precision for bounding box (Apbox) and for segmentation mask (AP Ammask) ) We also compare semantic segmentation performance on ADE20K using UperNet [57]. in Mean Intersection over Union (mloU). Our method outperforms baselines all fine-gran graiige tasks as shown the right side of Table 2 indicating that ours better
captures the fine details of the image with the same training session.

---

Table Performance on downstream tasks. LP and FT stand for Linear probing and Fine-tuning. respectively. Det. indicates the Object Detection task.
Task
Image Classification
Det. COCO AP
Segmentation
Dataset
ImageNet-IK
iNa2019
CIFAR
CUB
COCO Apmask
ADE20K
Metric
LP
FT
FT
FT
FT
mloU
MAE
22
61.4
82.5
78.7
89.3
81.8
43.0
38.9
45.0
AMT
38
61.7
82.8
76.0
87.8
80.8
42.8
36.6
43.1
Ours
62.9
83.2
78.9
90.0
82.8
43.3
39.3
45.2
5.3
Ablation Studies
Table 3 compares linear-probing performance of our method on image classification, with various settings. We fix the masking ratio to 0.75 for all the experiments in this section. Ablation studies on more factors including masking ratio be found in Appendix D The first group verifies that the later layers of the encoder yield the most accurate token relations, aligned well with our analysis in Section 4 The second group verifies that the hint tokens proposed in Section 4 are essential. Also, better performance with uniform sampling (Random) than with S;-based approach (Equation g indicates the importance of providing equal opportunity all clusters have visible tokens.
Table 3: Ablation studies, The default is highlighted in gray. Detailed analysis
be found in Appendix D
Layer
Target cluster
Hint strategy
Linear probing
Enc
Object
Random
62.3
Enc
Object
Random
62.4
Dec
Object
Random
62.7
Enc
Object
Si-based
62.5
Enc 11
Object
No hint
52.3
Enc 11
Object
Random
62.9
5.4 Analysis on the Learned Feature Space
We take a deeper look into our method for further insights on its improvements via various metrics with the ImageNet-IK IK validation set. We analyze with 0 unless noted otherwise.
We measure the weighted average distance of the attention operations between the query and key tokens within the image in Figure 7a Since it can be interpreted as the size of the receptive fields CNNs, higher attention distance our method suggests that has been better-trained at the same epoch, more globally capturing the image context.
Attention Distance [s].
Fourier Analysis 42 Figure shows the relative log amplitude of Fourier-trand tranffrrer representa- tions, which indicates the degree to which the model prioritizes either high- nigh-frequency (pattern) or low-frequency (shape) information. Our method utilizes high frequeeeyyncy components more intensively than MAE does, implying more powerful pattern- based clustering.
Mask Token Variance. We report the variance of mask token embeddings along the decoder layers with m = 0.75 Figure 7c As they carry high-level semantics of each potential patch cluster, higher variance among them indicates that the latent variables responsible for estimated shared information in each individual cluster has been diversified, implying that patches are grouped into finer clusters. Consistently higher variance along the layers of ours manifests its further progressed patch clustering.
clearly shows that ours captures finer patch embeddings and tighter boundaries than MAE. See Appendix C more examples and detailed explanations.
Qualitative Comparison. Figure I

---

Attentton distance tpx
110-
Oara MAE
-1.8
varlance
Log amplitude
MAE
90-
-2.2
0.18
70
0.18
50-
Ours MAE
0.10
0.00
30
3.0
121
6
121
5 Decoder
Encoder
Decoder
Encoder
Decoder
(a) Attention distance
(b) Fourier analysis
(c) Mask token variance
Figure Metrics explaining our performance gain. Layers left on the red dotted line belong the encoder, and the rest to the decoder.
Related Work
Masked Image Modeling (MIM). Inspired by Masked Language Modeling 13
8 MIM has been 37 51
widely applied in image [63
7
59
22
14
58
19
33
3
18
T6
35 and video 43 established
understanding 47 52 MIM with CNNs, focusing on the prediction of masked regions of an image. MLM also has been applied to VIT [5] e.g., BEiT 7 leverages visual tokens from dVAE reconstruction targets for masked image patches, and SimMIM [59] directly predicts raw pixel values as a regression problem. MAE 22
17
53
40
56
60
54
20
21
26
27
25
Context Encoder
also regresses the raw pixels, adopting asymmetric encoder-decoder architecture.
Informed Masking, Recent researches have considered to arm MAE with an advanced masking strategy [34 22 45
10
32
38) MST (34] and AttMask [28] have pioneered information ceaittionggiieed guided
masks, utilizing the attention maps of a supervised ViT. ADIOS 45 adopts adversarial training to get optimal masks. SemMAE (32 and AutoMAE [TO] leverage the powerful knowledge from pre-trained self-supervised ViTs, showcasing the synergistic fusion of informed masking with MAE, Despite effectiveness, these methods limited they require external model or rely labels, which make these models no longer fully self-supervised. To address this critical issue, AMT (38] extracts attention maps directly from the model during pre-training and generates informed masks from them.
Analysis on MAE. Nam al. 42 reports a comparative analysis between MIM and contrastive learning, highlighting how the mecoder-decoder architecture of MAE empowers the properties of MIM, although their discovery confined to token differentiation MIM models. Another avenues introduce theoretical architectures to elucidate MAE's behavior under specific assumptions (9 41 Hierarchical latent variable model 29 aligns well with our primary observations.
61 31.
Summary and Limitations
Unveiling the operation of Masked Autoencoder (MAE) which fundamentally learns pattern- based patch-level clustering, we expedite the MAE to learn patch clustering by incorporating informed mask derived from itself. Notably, our method does not require any other external models additional information. Superior results on extensive experiments demonstrate the effectiveness of our method.
Limitations. Our method may show less significant improvement when training with excessively fragmented images, e.g,, some dataset for segmentation tasks. In detail, since there would be numerous clusters within each image, masking specific clusters with informed masking may yield similar masks to random masking.
Acknowledgements
This work supported by Samsung Electronics Co., Ltd (10230414-05943-01. RAJ012377-80SD) by Youlchon Foundation (Nongshim Corp.), and by National Research Foundation (NRF) grants (No. 2021H1D3A2A03038607 / 50%, 65-3022--0036576 / 10%, RS- 2023-00222663 / 5%) and Institute for Information & communication Technology Planning & evaluation (ITP)) grants (No. RS-2024-00355131 25%, RS-2022-11220264/ 10%), funded by the government of Korea.
10

---

References
[1] Abnar and W. Zuidema. Quantifying attention flow in transformers. arXiv:2005.00928 2020.
[2] M. Assran, M. Caron, I. Misra, P. Bojanowski, Bordes, P. Vincent, A. Joulin, M. Rabbat, and
N. Ballas. Masked siamese networks for label-efficient learning. In ECCV, 2022.
[3] S. Atito, M. Awais, A. Farooq, Z. Feng, and Kittler. MC-SSLO.0: Towards multi-concept self-supervised learning. arXiv:2111. 2021.
[4] S. Atito, M. Awais, and J. Kittler. SIT: Self-supervised vision transformer. arXiv:2104.03602. 2021.
[5] R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir. MultiMAE: Multi-modal multi-task masked autoencoders. In ECCV, 2022.
[6] W. G. C. Bandara, N. Patel, A. Gholami, M. Nikkhah, M. Agrawal, and V. M. Patel. AdaMAE: Adaptive masking for efficient spatiotemporal learning with masked autoencoders. In CVPR, 2023.
[7] H. Bao, L. Dong, S. Piao, and F. Wei. BEiT: Bert pre-training of image transformers. arXiv:2106.08254, 2021.
[8] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In NeurlPS, 2020.
[9] S. Cao, P. Xu, and D. A. Clifton. How to understand masked autoencoders. 2rXIv:2202.03670.
2022.
[10] H. Chen, W. Zhang, Y. Wang, and X. Yang. Improving masked autoencoders by learning where to mask. arXiv:2303.06583 2023.
[11] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and Sutskever. Generative pretraining from pixels. In ICML, 2020.
[12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
[13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: tre-training of deep bidirectional transformers for language understanding. 1XX0:1810004800 2018.
[14] X. Dong, J. Bao, T. Zhang, Chen, W. Zhang, L. Yuan, D. Chen, F. Wen, N. Yu, and B. Guo. PeCo: Perceptual codebook for bert pre-training of vision transformers. In AAAI, 2023.
[15] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11 1929, 2020.
[16] Y. Fang, S. Yang, S. Wang. Y. Ge, Y. Shan, and X. Wang. Unleashing vanilla vision transformer with masked image modeling for object detection. In ICCV, 2023.
[17] C. Feichtenhofer, Y. Li, K. He, et al. Masked autoencoders spatiotemporal learners. In NeurlPS, 2022.
[18] Gao, Ma, H. Li, Z. Lin, J. Dai, and Y. Qiao. MCMAB: Masked convolution meets masked autoencoders. In NeurIPS, 2022.
[19] R. Girdhar, A. El-Nouby, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. OmniMAE: Single model masked pretraining on images and videos. In CVPR, 2023.
[20] A. Gupta, S. Tian, Y. Zhang, J. Wu, R. Martin-Martin, and L. Fei-Fei. MaskViT: Masked visual pre-training for video prediction. arXiv:2206.1188 2022.
[21] A. Gupta, J. Wu, J. Deng, and I., Fei-Fei. Siamese masked autoencoders. arXiv:2305. 14344, 2023.

---

[22] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
[23] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[24] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In ICCV, 2017.
[25] Z. Hou, Sun, Y.-K. Chen, Y. Xie, and S.-Y. Kung. MILAN: Masked image pretraining on language assisted representation. arXiv:2208.66049 2022.
[26] B. Huang, Z. Zhao, G. Zhang, Y. Qiao, and L. Wang. MGMAE: Motion guided masking for video masked autoencoding. ICCV, 2023.
[27] Z. Huang, X. Jin, C. Lu, Q. Hou, M.-M. Cheng, D. Fu, X. Shen, and J. Feng. Contrastive masked autoencoders are stronger vision learners. arXhv:2207.13532, 2022.
[28] I. Kakogeorgiou, S. Gidaris, B. Psomas, Y. Avrithis, A. Bursuc, K. Karantzalos, and N. Ko- modakis. What to hide from your students: Attention-guided masked image modeling. In ECCV, 2022.
[29] L.. Kong, M. Q. Ma, G. Chen, Xing, Y. Chi, L.-P. Morency, and K. Zhang. Understanding masked autoencoders via hierarchical latent variable models. In CVPR, 2023.
[30] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[31] J. D. Lee, Q. Lei, N. Saunshi, and J. Zhuo. Predicting what you already know helps: Provable self-supervised learning. In NewrlPS, 2021.
[32] G. Li, H. Zheng, D. Liu, C. Wang, B. Su, and C. Zheng. SemMAE: Semantic-guided masking for learning masked autoencoders. In NeurlPS, 2022.
[33] X. Li, Y. Ge, K. Yi, Hu, Y. Shan, and L.-Y. Duan. mc-BEiT: Multi-choice discretization for image bert pre-training. In ECCV, 2022.
[34] Z. Li. Z. Chen, F. Yang, W. Li. Y. Zhu, C. Zhao, R. Deng, L. Wu, R. Zhao, M. Tang, et al. MST: Masked self-supervised transformer for visual representation. In NewrIPS, 2021.
[35] F. Liang, Y. Li, and D. Marculescu. SupMAE: Supervised masked autoencoders are efficient vision learners. arXiv:2205. 2022.
[36] T.-Y. Lin, M. Maire, Belongie, J. Hays, P. Perona, D. Ramanan, Dollár, and C. Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[37] Y. Liu, S. Zhang, J. Chen, K. Chen, and D. Lin. PixMIM: Rethinking pixel reconstruction in masked image modeling. FxXiv:2303.02416, 2023.
[38] 7. Liu,J. Gui, and H. Luo. Good helper is around you: Attention-driven masked image modeling. In AAAI, 2023.
[39] N. Madan, N.-C. Ristea, K. Nasrollahi, T. B. Moeslund, and R. T. Ionescu. Cl-mae: Curriculum- learned masked autoencoders. In Proceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision, pages 2492-2502, 2024.
[40] Mun, M. Shin, G. Han, S. Lee, S. Ha, J. Lee, and E.-S. Kim. BaSSL: Boundary-aware self superviee learning for video scene segmentation. In ACCV, 2022.
[41] J. Pan, P. Zhou, and S. Yan. Towards understanding why mask-reconstruutton pretraining helps in downstream tasks. arXiv:2206.00 03826, 2022.
[42] N. Park, W. Kim, B. Heo, T. Kim, and S. Yun. What do self-supervised vision transformers learn? arXiv:2305.0000 3000000799, 2023.
[43] D. Pathak, P. Krahenbuhl, Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
12

---

[44] J. Shi and J. Malik. Normalized cuts and image segmentation. IEEE Transactions on pattern analysis and machine intelligence, 22(())8888990 900888905,200 2000.
[45] Y. Shi, N. Siddharth, P. Torr, and A. R. Kosiorek. Adversarial masking for self-supervised learning. In ICML, 2022.
[46] A. Strehl and J. Ghosh. Cluster ensembles a knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research, 3(12):583 2002.
[47] Z. Tong, Y. Song. J. Wang, and L.. Wang. VideoMAF: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, 2022.
[48] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and
S. Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
[49] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD birds-200-2011 dataset. Technical report, California Institute Technology. 2011.
[50] H. Wang, K. Song, J. Fan, Y. Wang. J. Xie, and Z. Zhang. Hard patches mining for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10375-10385, 2023.
[51] H. Wang. Y. Tang, Y. Wang, J. Guo, Z.-H. Deng, and K. Han. Masked image modeling with local multi-scale reconstruction. In CVPR, 2023.
[52] L. Wang. B. Huang. Z. Zhao, Z. Tong, Y. He, Y. Wang, Wang, and Y. Qiao. VideoMAE v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023.
[53] R. Wang, D. Chen, Z. Wu, Y. Chen, X. Dai, M. Liu, Y.G. Jiang, L. Zhou, and L. Yuan. BEVT: BERT pretraining of video transformers. In CVPR, 2022.
[54] Y. Wang, Z. Pan, X. Li, Z. Cao, K. Xian, and J. Zhang. Less is more: Consistent video depth estimation with masked frames modeling. In ACM MM, 2022.
[55] Y. Wang, X. Shen, S. X. Hu, Y. Yuan, J. L. Crowley, and D. Vaufreydaz. Self-supervised transformers for unsupervised object discovery using normalized cut. In CVPR, 2022.
[56] C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022.
[57] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018
[58] J. Xie, W. Li, X. Zhan, Z. Liu, Y. S. Ong, and C. C. Loy. Masked frequency modeling for self-supervised visual pre-training. arXiv:2206.07700 2022.
[59] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu. SimMIM: A simple framework for masked image modeling. CVPR, 2022.
[60] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas. VideoGPT: Video generation using vq-vae and transformers. arXiv:2 10157, 2021.
[61] Q Zhang, Y. Wang, and Y. Wang. How mask matters: Towards theoretical understandings of masked autoencoders. NeurIPS, 2022.
[62] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 227:302-321, 2019.
[63] J. Zhou, C. Wei, H. Wang, W. Shen, C. Xie, A. Yuille, and T. Kong. iBOT: Image bert pre-training with online tokenizer. arXiv:211..07832, 2021.
13

---

Appendix
A Method Elaboration
Detailed Reasoning for Our Method. As discussed in Section 2 the true shared information c exists for the entire token set X, which is equivalent to statistical dependency among the patches in X. With training, MAE learns to estimate this high-level latent variable č, which reflects the context of the entire image. Let us denote by Sm and S, for information specific masked out patches Xmm and visible patches Xr respectively, e.g., positional embeddings.
Since MAE cannot access Xm during training, the decoder is forced to reconstruct Xm via 1) simple interpolation using visible tokens, or 2) estimated statistical dependency among the entire tokens, i.e. è. As shown in Figure simple interpolation means reconstructing Xm, mainly with X, and s., which not directly related to Xm. leading to poor reconstruction result. However, due to the reconstruction loss, MAE is forced to improve the reconstruction quality, establishing high-level information é and performing the reconstruction based on it. As a result, at some moment, the encoder starts to map the visible tokens Xr to estimated shared information è for the whole token set X, and decoder exploits this hierarchical information to reconstruct the low-level information; i.e., the raw RGB pixels of Xm. This process is verified in Figure 4 in the main manuscript.
High-level information
(Latent space)
Sym
Sy
Low-level in information (RGB)
Xm
X
Figure I: Hierarchical latent variable model framework [29]. Assuming high-level shared informa- tion e exists among the whole tokens, MAE encoder learns to estimate from X, to reconstruct raw pixels of Xm. Here, shared information is equivalent to statistical dependency inside X. Sm and s, stand for information specific Xm, and X,, respectively. Dotted line indicates potential dependency.
Moreover, connecting this logic conceptually corresponds to pattern-basee patch clustering information. In other words, considering the pattern-basee patch clustering in MAE (as verified in Section B), it suggests that MAE clusters the patches and builds corresponding high-level variable containing è for each patch cluster.
discovery in Section 3.2
claim that this unknown
In summary, MAE learns to construct the latent variables for each potential patch cluster. However, considering the fact that MAE learns relevance among the patches from the extremely early stages in pre-training process (Section 3.3). it can be inferred that MAE with naive random masking is actually revisiting key dissimilarities in X, which exists between casily separable patches, every epoch wasting large portion of its training resources. Especially, when it comes to bi-partitioning (which the simplest form of key dissimilarities) MAE learns it from the very early epochs as verified in Fig. 3al
Based on this reasoning, we can enforce MAE to focus on learning hardly distinguishable tokens by guiding MAE to skip revisiting key dissimilarities by injecting the information about it as input. We can inject this information via informed masks, which possess key dissimilarities by intensively masking one of the bi-partitioned clusters, leading MAE to assign most of the training resource to learning relatively vague patch clusters in masked out patch sets.
As discussed in Section 4 our method generates informed masks by itself without using any external model requiring additional information. Recall that MAE generates informed masks after T ~ 50 epochs of training. Figure compares our informed masking with and without the hint tokens the random masking. also illustrates the bi-partitioned clusters extracted from MAE itself after 51 epochs, which are used for the internal generation of the informed masks. We observe in these examples that our mask out the target cluster even when the bi-partitioning is not perfect. For example, the target cluster in (e) consists of the portion of house and sky, but our method fully masks out the patches composing the house in the image. Similar results can be found in (j) and (k). Also, even when the foreground
Qualitative Analysis.
masking (Section 4 guarantees fully

---

Doop
Th Deportttoons Random
Ours
[hl
(4)
(h)
Figure II: Qualitative examples of informed masking on ImageNet training set. Based method, informed masks are generated after 51 epochs of pre-training with hint ratio of 0.05. Results clearly show that MAE in early training steps provides appropriate bi-partitioning information and successfully creates informed mask without using external models or additional information. We also note that, similarity-scon seartt--scr---bsseeee masking strategy yields robust informed mask even in the case when the bi-partitioning is imperfect.
is not clearly distinguished due to the barely discernibee patterns as in (i) and (I), we see that our approach still fully masks out the object. The success of relevance score strongly indicates that patch vectors are hierarchically clustered based on their visual patterns, as they are masked out in the order of pattern similarity with the mean patch vector.
We confirm from the examples that even in early epochs, MAE is able to appropriately bi-partition the image, which means it has already learned discriminate the image into two clusters. We also find that most of the examples are bi-partitioned into foreground and background, since the similarity edges between these two groups tend to have the weakest values. In summary, although MAE in the early epochs does not promise to provide perfectly discriminated object-centric cluster from the image, our proposed approach robustly builds object-centric masks through the introduction of the relevance score.
B
Token Relations
Patch Clustering in Projected Latent Space. Figure III illustrates the patch clusters few examples and their t-sne plots. We consider a graph = (V,E) for the given image, where V and correspond to patches and edges between them weighted by M in Eq. ( respectively. From this graph, we repeatedly apply Normalized Cut 44 to remove edges with the lowest relevance until the graph is split into predefined number (K) of clusters. We clearly
that tokens with similar visual patterns (color, texture) are 1) grouped together as the same patch cluster (2nd row) and 2) embedded closely in the latent space (last row). Apparent discrimination in the representation space supports the patch-level clustering.
Enhanced Feature Separability of MAE with Our Method. Based on our analysis on embedding space suggested in Equation 3. compare the vanilla MAE and our method the aspect feature separability with 800 epochs of training in Table M The results indicate that our method shows more diversified feature space via higher feature variance and similarity variance, aligned well with the analysis in Section
Table I: Feature variance (ar) and sim- ilarity variance (ass)
Feature
B(ar]
B[as]
MAE 22 Ours
0.082
0.075
0.096
0.079
KL Divergence of All Layers in MAE. We additionally provide KI divergence (KLD) of token relations for all layers in MAE as an extension of Section For the decoder, we use token relations with the intact input, i.e., (X)), for the criterion distribution in the KLD (Equation 4h. In other

---

Input image
Patch clusters
6o) Projection
Figure III: Illustrations of patch clusters learned by MAE. (a) Input images. (b) Similarity-basel patch clusters. (c) t-sne plots of the patch embeddings.
words, we compare the token relations from each epoch with masked inputs to the token relations from the last epoch with intact inputs. Due to this setting, KLD with decoder does not converge to zero at the final epoch in Figure IV
As shown Figure IV all layers but the first one in the decoder drastically converge at the early epochs with both of M than A. Encoder (F(X)) layers are much stabler and converge faster than decoder (D((E(X,), ml)) layers due the difference the amount of given information. Also, since the cosine similarity scores M directly compare the similarity among the tokens, strong convergence of M supports the observation that MAE intrinsically learns the patch-level clustering.
KLD of the attention scores in the first encoder layer is low at the first epoch, which implies that it learns homogeneous attention map rather than random values as discussed in Section [..4 The first layer of the decoder shows high KLD with the attention scores along with the training, because 1) the mask tokens are not contextualized yet (that is, mask token vectors does not represent the masked out patches at all), and 2) the index of each mask token is randomly selected for every epoch. On the other hand, KI.D with the similarity scores decreases along the epochs, because the similarity score matrix is calculated after the contextualization. This suggests that even single first layer in decoder has ability to properly exploit from the encoder to discriminate the patches although it is weaker than the later layers.
0.16
0.16
0.16-
0.16
0.12
Decoder 6
0.12
Encoder 6
0.12
Decoder 6
0.12
Encoder
0.08
0.08
0.08
0.08
10
0.04
0.04
0.04
0.04
0.00
0.00
0.00
0.00
100 200 300 400 Epochs
100 200 300 400
0
100 200
300 400 E Epochs
0
100 200 300 400
Epochs
Epochs
(a) Similarity score
(b) Attention score
Figure IV: KL divergence of the token relations between the final and intermediate epochs. Layer numbers are displayed the legend. All the layers but the first one in decoder show drastic decrement of (a) similarity score and (b) attention score at early epochs. The convergence speed and final converged values vary in layers.

---

VIT intra
inter
McCo intra
MoCo inter
intra
MAE inter
0.020
0.8
Mean edge
Mean edge
0.6
0.010
0.4
0.005
40
100
160
220 Epochs
280
340
400
40
100
160
220
280
340
400
Epochs
(a) Similarity score
(b) Attention score
Figure V: Bi-partitioning performance of various models. MAE, MoCo and ViT show different
trends of bi bi-partitioing performance in both of (a) similarity score and (b) attention score.
Further Experiments on VIT 15 and MoCo 23 We provide bi-partitioning performance and KL.
divergence token relations of ViT and MoCo for better understanding on our metrics in Figure We display the result of MAE encoder together for comparison. Before delving into the analysis, we note that the result of this experiment with ViT and MoCo is irrelevant to our main claims since ViT and MoCo do not learn patch clustering.
As MoCo yields homogeneous attention map resulting in simple form of embedding space, e.g. main object cluster and background cluster. the result of MoCo in Figure indicaate that the last epoch of MoCo has provided properly bi-partitioned patch groups. Consistent gap between mean inter-cluster (Himer) and mean intra-cluster (Jintra) edge weights of similarity score matrix M and attention score matrix A of MoCo supports this claim.
Unlike MAE or MoCo, embedding space of ViT does not guarantee to provide appropriate bi- partitioning results. As result, Figure V although the similarity score matrix M enlarges the gap between (iite and Mintra the attention score matrix A increases the Pinter rather than Piotm- This hardly interpretable pattern implies that the seeudo-ground truth for bi- partitioned patch groups generated at the last epoch is unstable or even incorrect.
In summary, only MAE explicitly shows its ability to clearly recognize key dissimilarities among the tokens, i.e., bi-partitioning information, from the extremely early stage of pre-training, and consistently escalates the gap between Minter and Jinara-
0.10
0.10
0.6
0.6
0.5
0.5
0.08
0.08
10
10
10
0.08
0.06
MoCo
MoCo
VIT
0.04
0.04
0.2
0.02
0.02
0.1
0.1
0.0
0.0
0.00
0.00
0
100
200
300
400
0
100
200
300
400
0
100
200
300
400
100
200 300 Epochs
100
Epochs
Epochs
Epochs
(a) Similarity score
(b) Attention score
Figure VI: KI divergence of token relations of various models. MoCo and ViT show weaker convergence of token relations in both of (a) similarity score and (b) attention score.
Figure VI shows the KL divergence of token relations from ViT and MoCo. Compared to the result of MAE in Figure IV both ViT and MoCo reveal gradual convergence of token relations and some layers exhibit their unstable convergence. Again, as ViT and MoCo do not learn patch-clustering, the experiment results of ViT and MoCo are off-topic to the main stream of our work.
iv

---

Mie
[+]K (1) K-
10
181 --
[n]
03 x-
1at K-
(hi 10---
021 K
(1) 1---
Figure VII: Qualitative comparison on ImageNet validation set. Patches are discriminated in more ine-grainee manner with our method. More diverse and finer patch clusters constructed in foreground verify our hypothesis that intensive masking on specific cluster leads to establish more diverse high-level latent variables.
C Qualitative Results
We provide more qualitative examples of patch clustering compared to vanilla MAE in Figure VII where we see that images are segmented into K clusters in unsupervised manner. Successful segmentation from recursive graph-cut suggests that features are hierarchically discriminated in the embedding space. Our method clearly shows more accurately clustered patches based on their pattern and also yields tighter boundary between the clusters for various types of images, i.e., object-centered images and those containing higher portion of background.
D Analysis on Ablation Studies
As displayed in Table our ablation study on layer selection for embedding extraction verifies the hypothesis on it (See Section 3 while showing the minor effect model performance relative to other factors. Especially, the last layer of the decoder shows higher performance than the early intermediate layers of the encoder. Since the decoder possesses the patch cluster information constructed through the entire encoder layers, it may have more appropriate bi-partitioning quality than using few early encoder layers, e.g., layer 3 or layer
To analyze the reason for the minor effect on layer selection, we display the examples of informed masks generated with bi- partitioned patch cluster from each layer in Figure VIII
Table II: Ablation studies on various factors. The default is highlighted in gray.
Layer
Target cluster
Hint strategy
Masking ratio
Linear probing
Enc 3
Object
Random
0.75
62.3
Enc 7
Object
Random
0.75
62.4
Dec 8
Object
Random
0.75
62.7
Enc 11
Object
S;-based
0.75
62.5
Enc 11
Object
No hint
0.75
52.3
Enc 11
Background
Random Random
0.75
61.1
Enc 11
Alternate
0.75
61.6
Enc 11
Object
Random
0.6
62.9
Enc 11
Object
Random
0.9
61.4
Enc 11
Object
Random
0.75
62.9
In Figure VIII we find that the later layer the encoder provides the most accurate bi bi-partitioning result compared to others. However, in spite of the improper patch clustering, each layer can build plausible informed mask (and often proper) based on our similarity-scor--aaseee based masking strategy. With simple image, e.g. (a), all layers are able to properly bi-partition the image leading to fully mask out the main object. With more complex images like (b), (c), (e) and (f), whether 1) the bi bi-partitioned cluster contains mixture of foreground and background or 2) only some patches of

---

Good Goog
Layer
Laper
Doon
Lapor
Laper
Figure VIII: Comparison of the Quality of the informed masks generated from different layers. Fach example is denoted by the index of the original image in Figure [I] Although early layers of the encoder and the last layer of the decoder yield inappropriate bi-partitioning result, our similarity- score-based masking strategy robustly alleviates this issue, leading to minor difference in performance the layer selection for generating informed mask.
the foreground are discriminated, method stably constructs the proper informed mask, aligning with the result in Figure [] In example (d), when the layer 7 is used, we observe that object-centric masks are successfully generated since the pattern of lizard similar to that of plants, despite failure in bi-partitioning where the discriminated foreground captures only the plants, missing the liazard. Also, although the decoder hardly captures the entire shape of the foreground. it precisely discriminates the salient patches belonging to the main objects, as expected to generate more accurate informed mask than the early or intermediate encoder layers. We also note that using the last layer of the encoder yields similar performance to our default setting (i.e., using the second last layer of the encoder) which could be preferred for its simplicity.
As shown the second group in the Table M it is essential to provide hint tokens for successful training. As displayed in Table [II] the training process results in too high loss without the hint tokens, while it is appropriately alleviated with them. This is because it is fundamentally impossible for the model reconstruct the whole foreground without any visible tokens belong it. In the aspect of patch clustering, MAE would lose an opportunity to construct high-level latent variables i.e., shared information, for the clusters specific to the foreground when trained without hint tokens.
Table III: Reconstruction loss (MSE) with 400 pre- pre-training epochs
according each training method.
Model
|
MAE 22
Ours (no hint)
Ours (with hint)
Loss
0.64
0.56
In addition to the ablation studies in Table 3 we also consider 1) the target cluster to be masked out and 2) masking ratio in the third and fourth group Table respectively. Object-centric informed masking leads to better performance compared to background-centric masking alternately masking foreground and background along the epochs, supporting our choice of object-centric masking strategy in Section [ For the masking ratio, although masking less regions (0.6) yields the same linear probing performance to the default one (0.75), it recommended to set masking ratio to 0.75 for more efficient training cost.
E
Comparison with Various MAES
compare the performance across various MAEs in Table IV grouping the models based on the incorporation of external training costs. For models that we were able to reproduce the re- sults (AMT [38, HPM (50), we report the reproduced results. For SemMAE (32], we refer the performance reported in the paper. For CL-MAE (39], report only the training time, as its
vi

---

reproduction is difficult due to the high training cost, and the results reported in the respective paper are not directly comparable due different experimental settings.
Table IV: Comparison with additional MAEs terms of linear probing performance. MAEs that utilize external resources additional parameterized modules highlighted in gray, indicating that they are not included as baselines for a fair comparison. The training time is reported as relative value the MAE, training for 400 epochs. The results show that method matches the performance other MAEs with the same or even less training cost. Our method requires only about one more step generate masks, which empirically increases the pre-training time about 0.25% for training 400 epochs, i.e., x.0025x training time compared to the vanilla MAE.
Method
of params.
Pre-train epochs
Linear probing Training time
Baselines using external pre-trained model
SemMAE
112M
800
68.7
6.3x
MAE 22
112M
800
63.8
2x
MAE 122
112M
1600
68.0
4x
Ours
112M
800
65.9
2x
Ours
112M
1600
68.7
4x
Baselines using additional module
HPM 50
138M
400
63.2
1.5x
CL-MAE
39
148M
400
6x
Baselines without external resource or additional module
AMT (38
112M
400
61.7
1x
MAB
112M
400
61.4
Ours
112M
400
62.9
1x
F Extended Training
We conduct extended pre-training sessions and report linear prohing performance on ImageNet-1K along the training epochs in Table[V] Our method consistently brings sustained performance gain after considerable length of training, i.e., for 1600 epochs.
Table V: Linear probing with ImageNet-1K
Pre-training epochs
200
400
800
1600
MAE p2
53.9
61.4
63.8
68.0
Ours
54.4
62.9
65.9
68.7
G
Computing Resources
conduct experiments on 8 NVidia A6000 GPUs (48GB) and it takes ~2.5 days on pre-training for 400 epochs. For 1600 epochs of pre-training, it takes about 10 days.
vii

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main contribution is discovery of the internal operations of MAE and a novel method to expedite its training based on the discovery. These points are clearly mentioned in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We mention limitations of our work Section 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
viii

---

Answer:
[NA]
Justification:
We do not claim any theoretical result or contribution.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide the detailed algorithm and hyperparameter settings in Section and Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
ix

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We plan to publicly provide our code used in this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pippscc/puullc//uuu /ttee///uuddde/oooSuuubbiiibii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings and details are elaborated in Section Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported due to the expensive computational cost.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
X

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computational cost for pre-training in Appendix G
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:p/s///aarsss.cooooooo//hhaa
Answer: [Yes]
Justification:
We strictly follow the ethics guideline with this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper mainly studies the internal operations conducted by the Masked Autoencoder model, and proposes how to expedite its training. As far as we see, there is no direct negative societal impacts from this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
xi

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our proposed method is mainly about expediting existing self- sulf-vuper model, Masked Autoencoder. Besides general risks of machine learning models, e.g. trained biased data, this work does not impose any high risk for misuse far concern.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We respectfully cited all assets use, following the license and terms of use properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
xii

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide detailed documentation regarding the code and trained models to be released.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does involve any human subject or crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve any human subject.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
xiii


Reviews:
[{"review_id": "uqPeE9oHTV", "reviewer": "Reviewer_LZWD", "summary": "This paper investigates the properties of MAE and introduces a better masking strategy for MAE. Analyses using similarity scores and attention show that MAE learns to cluster the patches at the early epochs. On the other hand, analyzing exploitation rates of various layers implies that the encoder is trained sufficiently to cluster the patches after the early training. Based on the observation, Self-guided Informed Masking is introduced to accelerate MAE, which searches for the foreground tokens and masks them except a few tokens as a hint. Quantitative comparison in the experiments section verifies its effectiveness on MAE.", "questions": "Please refer to the weaknesses part above", "limitations": "The authors provide a limitation at the end of the paper, but it does not seem like their limitation but the shortage of MAE.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper provides various analyses\n* The proposed Self-guided Informed Masking improves the baseline", "weaknesses": "* There is a missing link or jump between the analyses in Section 3 and the proposed method in Section 4. \n    * If the proposed Self-guided Informed Masking is based on the observations in Section 3, then the authors should verify whether the metrics reported in Section 3 are modified through the proposed masking strategy. However, the impacts of the proposed method are not investigated through the analyzing tools in Section 3.\n    * What is the connection between the exploitation rate analysis and the design choices made in Self-guided Informed Masking?\n\n\n* Similar work to Self-guided Informed masking\n    * SemMAE [1] also masks foreground tokens of the given image while leaving a few of them as cues for autoencoding. The authors should clarify how Self-guided Informed Masking differs from and improves upon SemMAE's approach\n\n* Lack of explanation\n    * Key terms are not defined early enough in the paper. This paper considers visual patterns as a key aspect of the MAE pre-training. However, I cannot find any definition or explanation of the term 'pattern' until page 4 (line 116), which hinders understanding the main arguments in the early sections.\n    * The term 'Hint strategy' should be explained in or near the caption of Table 3.\n\n\n* Inconsistent notation\n    * \\textit{M} is used to denote both the set of mask tokens in Section 3 and the cluster containing the main object in Section 4. If there's no specific link between these definitions, such duplicate notation may confuse readers.\n\n* Performance improvements are marginal in some benchmarks (e.g., ADE20K segmentation, COCO detection, iNat2019)\n\n\n\n[1] Li et al., \"SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders,\" NeurIPS, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed feedback and valuable insights provided by the reviewer, and we have tried to address all concerns and suggestions.\n\n__[W1-2] Connection between the explotation rate analysis in Sec. 3 and our method__\n\nWe thank the reviewer for this constructive feedback. This will be explicitly stated at the beginning of Sec. 4 in the main paper as detailed below.\n\nIn Sec. 3, we show that the MAE encoder learns patch clustering from an early stage through bi-partitioning and KL divergence analyses. As bi-partitioning is sufficient to distinguish tokens into two major token clusters, we can bi-partition the image and mask out one of them. This result indicates that we can generate informed masks with MAE itself early in the pre-training phase and use these informed masks for the remainder of the training.\nThen, the next question is _when exactly the MAE can properly cluster the patches_. When the encoder is sufficiently trained to cluster patches, the encoder outputs reflect this information. Then, they are utilized to constitute mask tokens in the decoder. This means the mask tokens possess this patch clustering information and start to be highly exploited for reconstructing masked-out patches. By reversing this order, it can be inferred that a high exploitation rate of mask tokens in the decoder indicates that mask tokens have proper patch clustering information conveyed from the encoder, implying that the encoder can cluster the patches. We verified this via exploitation rate analysis, showing that mask tokens start to be exploited as much as visible tokens from an early epoch ($T$ epoch). This finding allows us to confidently generate informed masks at epoch $T$, ultimately leading to the design of our method.\n\n__[W1-1] Analyzing tool in Sec. 3__\n\nFollowing your suggestion, we verify our method using the analysis tools in Sec. 3 in Table A in the rebuttal PDF. The results indicate that our method shows more diversified feature space via higher feature variance and similarity variance, aligned well with the analysis in Sec. 5. \n\n__[W2] Difference between SemMAE [1] and our method__\n\nBoth masking strategies aim to mask out important regions of the image. However, the fundamental difference between SemMAE and ours is that SemMAE is 'supervised' by an external pre-trained model which requires extra training cost for pre-training this model, while ours is completely 'self-supervised'. As SemMAE is supervised by external model, it is not guaranteed that SemMAE still learns what MAE actually learns, i.e., pattern-based patch clustering. The training process of SemMAE can be interpreted as indirect feature distillation via reconstruction task, since it heavily depends on the quality of the features extracted from the external model, e.g., iBoT [2] features containing semantic segmentation information. On the other hand, our work identifies the property of MAE (i.e., patch clustering) emerging during the training process and utilizes this observation to accelerate MAE's learning of this property. In this context, we refer to our method as providing 'acceleration' rather than 'performance improvement' because it speeds up MAE's ability to learn its inherent features, rather than enhancing its feature space with external resources. Since our method relies solely on MAE itself, it is completely free from relying on the attributes of feature representation from external models, and this is the key difference between our method and SemMAE.\n\n__[W3, W4] Lack of explanation and inconsistent notation__\n\nThanks for your thorough review. As suggested, we will define the term 'visual pattern' earlier in the paper and explain the hint strategy in the section pertaining to Table 3. Also, the notation $\\textit{M}$ in Sec. 4 will be replaced with an alternative notation.\n\n__[W5] Performance improvements__\n\nWe admit that the degree of improvement varies across various tasks, as the reviewer pointed out. We emphasize, however, that our method consistently improves performance across various tasks with different nature, e.g., requiring global understanding (classification) and pixel-level details (segmentation).\n\n__Additional comment on limitations__\n\nOur method may show less significant improvement when training with excessively fragmented images, especially for the segmentation task. In detail, since there would be numerous clusters within each image, masking specific clusters with informed masking may yield similar masks to random masking. We will note this as a limitation in the paper.\n\n[1] Li, Gang, et al. \"Semmae: Semantic-guided masking for learning masked autoencoders.\" Advances in Neural Information Processing Systems 35 (2022): 14290-14302.\n\n[2] Zhou, Jinghao, et al. \"ibot: Image bert pre-training with online tokenizer.\" arXiv preprint arXiv:2111.07832 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RkbiKe3d64", "reviewer": "Reviewer_ZFhW", "summary": "Masked autoencoding (MAE) based pre-training has been widely adopted recently. However, it is not fully uncovered what and how MAE exactly learns. In this paper, the authors provide an in-depth analysis of MAE and discover that it learns pattern-based patch-level clustering from early stages of pre-training. Upon this finding, the authors propose a self-guided masking strategy which generates informed masks in an unsupervised manner without relying on external models or labels. The experiments on various downstream tasks shows the effectivness of the masking strategy in MAE pre-training.", "questions": "Please look at the weakness section for the questions and suggestions.", "limitations": "Yes, the authors has adequately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides an in-depth analysis of what and how MAE learns during pre-training.\n- The authors use attention score and cosine similarity score to analyze the pairwise relationship of tokens using the last layer of the ViT-B encoder and the decoder. MAE based qualitative analysis shows that patches are clearly clustered as compared to MoCo and ViT. Quantitatively, the feature variance and similarity variance is high for MAE encoder and decoder which show it  clusters patches based on their patterns. The analysis provide another insight about MAE regarding when they start to learn patch clustering. \n- The exploitation rate of mask tokens and visible tokens show that every decoder hold substantial amount of shared information estimated by the encoder which means MAE is trained to efficiently cluster the patches.\n- Based on the findings, the authors propose a new called self-guided informed masking.\n- Experiments on various datasets (IN1k, COCO, iNat19 etc) shows the effectiveness of proposed masking strategy.", "weaknesses": "- Although the author shows the performance of masking strategy with 1600 epochs, is there any reason to use a model trained for 400 epochs for analysis? What is the motivation behind it? Did the authors only use 10% of training data during pre-training for analysis?\n- Given this patch clustering and estimated shared information, does it mean we need only few tokens for downstream tasks? Any analysis with linear layer fine-tuning and fewer visible tokens would be interesting?\n- It is still not clear if the authors pre-train the model in two stages? Did they pre-train the model with random masking for T epochs and then continue pre-training it with informed masking? Any remarks on that would be helpful.\n- Given the decoder has enough shared information to reconstruct the masked patches and ablation table shows the performance boost, can we directly use the features extracted from the decoder for downstream tasks ?\n- Can the authors shed light on the computation and the memory and compared the approach with random masking? Given the embeddings are extracted from either encoder or decoder for informed masked with bi-partition, would that increase the total pre-training time?\n- Any ablation on the masking ratio? how much hint is added in the informed mask?\n- Any learning comparison in terms of reconstruction loss, with hint or without hint. If the reconstruction task becomes harder and we know that the generalization of pre-training can be effected if the proxy task is too challenging.\n- The masking strategy looks a combination of normalized cut, [1] and [2]. Can the authors please give a remark on the differences of self-guided masking and these methods?\n\n[1] Self-supervised transformers for unsupervised object discovery using normalized cut.\n\n[2] SemMAE: Semantic-guided masking for learning masked autoencoders.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for positive comments and constructive feedbacks. We have tried to address each of the concerns raised.\n\n__[W1] Reason for the analyzing MAE with 400 pre-training epochs__\n\nTo use a 1600-epoch pre-trained MAE for analysis, we would need to train MAE, MoCo, and ViT for 1600 epochs for a fair comparison. Due to the excessive time and resources required for this setting, we set the training time to 400 epochs following the literature, MAE [1], MoCo [2], and ViT [3]. These seminal works have analyzed with 400 or less epochs, reflecting the fact that 400 epochs is sufficient to obtain a well-trained embedding space for comparison. Nevertheless, we have shown the results with 1600 epochs to demonstrate scalability and effectiveness of our method. \nAlso, all the anlayses have been conducted with models trained on the __whole IN1K training set__.\n\n__[W2] Using fewer tokens for downstream tasks__ \n\nWe think this is an interesting idea. Because MAE distinguishes features solely based on their visual patterns, if we can extract specific clusters of tokens tailored to a given downstream task, it would be feasible to use only these subsets of tokens for those tasks. Although this is beyond the scope of this paper, it would be a great future research topic.\n\n__[W3] Training process of our method__\n\nThe model is trained in a single stage; at epoch $T$, we begin generating informed masks and continue the training process without interruption. We will state this point more clearly in the revised paper.\n\n__[W4] Using decoder features for downstream tasks__\n\nIt is an interesting suggestion to use the decoder layers for downstream tasks, if the MAE is sufficiently trained. Using decoder features might have slight benefit for the tasks requiring pixel-level details, considering the fact that decoder features are closer to the pixel-level space. However, using decoder layers would be less efficient since it requires additional inference time. This would be an interesting future direction for this line of research.\n\n__[W5] Training cost__\n\nOur method only requires only one more step to generate masks, which empirically increases the pre-training time about 0.25% (for 400 epochs pre-training). We will make this clearer in camera-ready.\n\n__[W6] Ablation on masking ratio__\n\nWe provide an ablation study on masking ratio in Table C in the rebuttal PDF. A masking ratio of 0.6 shows slightly better performance (+0.1%), while a masking ratio of 0.9 shows degraded performance, which shows similar trend to the results in [1]. We had to conduct this experiment with 200 pre-training epochs due to the short rebuttal period, but will add a similar table with regular 400 epochs of pre-training in camera-ready.\n\nFor hint tokens, we start by providing 10% of the masked tokens and gradually decrease the ratio to 5% by the end. We will include this information as well.\n\n__[W7] Reconstruction loss with and without hints__\n\nAs shown in the ablation studies in Table 3 in main paper, the learning process can become too challenging with informed masking, making the reconstruction task difficult and ultimately resulting in degraded performance. The reconstruction losses (MSE) after 400 epochs of training for vanialla MAE, ours (with hint) and ours (without hint) are 0.41, 0.56, and 0.64 respectively as shown in Table D in rebuttal pdf.\n\nAlthough the model can fail to learn generalized feature with a too difficult task, we successfully addressed this issue by providing hint tokens and consistent performance improvements in various downstream tasks verify our claim.\n\n__[W8] Difference from SemMAE [4]__\n\nBoth masking strategies aim to mask out important regions of the image. However, the fundamental difference between SemMAE and ours is that SemMAE is 'supervised' by an external pre-trained model which requires extra training cost for pre-training this model, while ours is completely 'self-supervised'. As SemMAE is supervised by external model, it is not guaranteed that SemMAE still learns what MAE actually learns, i.e., pattern-based patch clustering. The training process of SemMAE can be interpreted as indirect feature distillation via reconstruction task, since it heavily depends on the quality of the features extracted from the external model, e.g., iBoT [5] features containing semantic segmentation information. On the other hand, our work identifies the property of MAE (i.e., patch clustering) emerging during the training process and utilizes this observation to accelerate MAE's learning of this property. In this context, we refer to our method as providing 'acceleration' rather than 'performance improvement' because it speeds up MAE's ability to learn its inherent features, rather than enhancing its feature space with external resources. Since our method relies solely on MAE itself, it is completely free from relying on the attributes of feature representation from external models, and this is the key difference between our method and SemMAE.\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] He, Kaiming, et al. \"Momentum contrast for unsupervised visual representation learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[3] Dosovitskiy, Alexey, et al. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" arXiv preprint arXiv:2010.11929 (2020).\n\n[4] Li, Gang, et al. \"Semmae: Semantic-guided masking for learning masked autoencoders.\" Advances in Neural Information Processing Systems 35 (2022): 14290-14302.\n\n[5] Zhou, Jinghao, et al. \"ibot: Image bert pre-training with online tokenizer.\" arXiv preprint arXiv:2111.07832 (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "82zoxrYHRY", "reviewer": "Reviewer_BsNN", "summary": "This paper demonstrates that Masked Autoencoders (MAEs) learn pattern-based, patch-level clustering and that they obtain this ability in the early stages of pre-training. Based on this finding, it proposes a self-guided Masked Autoencoder method, which utilizes self-attention map information to mask out meaningful foreground information. The experiments show that the proposed method outperforms the vanilla method as well as AMT, another self-attention guided MAE method.", "questions": "Please refer to the weaknesses section.", "limitations": "I don't find any ethical issues. Please refer to the weaknesses section.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- I enjoyed reading this paper, as it begins by analyzing the properties of MAEs to identify the drawbacks of the methods and then proposes a novel method based on these analyses. I believe such an attempt to motivate the method via empirical analyses should be encouraged. Additionally, this paper includes post-hoc analyses of the proposed method, which are helpful in understanding the method.\n- The direction of introducing a novel masking strategy in this study seems promising and could potentially lead to significant advancements in the field.\n- The proposed method outperforms both the vanilla MAE and AMT, not only in image classification tasks but also in dense prediction tasks.", "weaknesses": "I believe the main weaknesses of this paper are two-fold: soundness and novelty.\n\n- **Soundness:** I am not fully convinced by the claims in this paper. The following are examples, though not exhaustive.\n    - In L99, dissimilarities between the main objects and the mean values of tokens might not directly imply “the MAE encoder has learned patch clustering basded on visual patterns,” and it is insufficient to conclude that MAE can clearly distinguish between backgrounds and the object based on the results in Fig 2b. Alternatively, it might be possible to simplify or even skip Section 3.2, as [40] addresses a similar claim.\n    - Since MAE does not train a CLS token, I am not convinced that the results of Fig 2c lead to meaningful findings.\n    - In L126, even if it is true that “the decoder learns a similar but les clearer pattern than the encoder,” the main objective of MAE is training the encoder, not the decoder. Therefore, the implications of this statement are not clear.\n    - In L195, the drawbacks of random masking appear somewhat suddenly. I couldn’t find direct links between the analyses in Section 3 and the inefficiency of the random masking strategy.\n- **Novelty:** The core takeaways from the analyses and the proposed method are not groundbreaking.\n    - L33: Even though Figure III in the appendix elegantly shows the token-wise clustering properties of MAE, the claim regarding the \"MAE’s property that learns pattern-based patch-level clustering\" is already demonstrated in [40]. Furthermore, according to [40], this property is not unique to MAE but is general to MIM methods. Additionally, it might be an interesting finding that this property emerges in the *extremely early stages of training*, but I wouldn’t say the 40th or 100th epochs constitute an *extremely early stage*. It might not be an apples-to-apples comparison, but [22] demonstrates that 100 epochs are a sufficient setting to achieve reasonable performance.\n    - L37: As this paper also cited, the proposed method is not the only method that utilizes a self-attention mask to introduce an efficient masking strategy. Similarly, prior works, such as AttnMask [28], offer comparable insights.\n- **Writing:** Although the writing and organization are not the major weaknesses of this paper, as I understand it, there is room for improvement.\n    - This paper occasionally refers to figures from later sections at the beginning, causing me to go back and forth to understand the context fully.\n    - Should the paper introduce the (accumulated) exploitation rate instead of using NMI or self-attention distance?\n    - Although the proposed method is straightforward and easy to understand, including a diagram to overview the method would enhance its readability.\n    - I sometimes struggled to find the definitions of symbols and hyperparameters, which could be more clearly indexed or referenced.\n    - In Section 5.4, explicitly stating the implications of the experimental results would significantly improve the paper.\n- **Method:** Likewise, I believe there is room for improvement in the proposed method.\n    - In L218, this method utilizes bi-partitioning and assumes that the dominant tokens represent background parts. However, its effectiveness might depend on the ImageNet dataset. Images with many objects or those containing large objects could be considered; in such cases, this assumption might no longer hold.\n    - In L238, he method employs the second-to-last encoder layer to utilize the self-attention map. However, it would also make sense to use the last self-attention layer for the sake of simplicity, since I assume that there would only be minor performance degradation.\n    - For a fair comparison, the experiments neglect methods that utilize external information or models, but it would be great if this paper could compare the method and emphasize the pros and cons alongside those methods.\n\nOverall, I am on the fence but lean towards acceptance since the strengths outweigh the weaknesses. However, I believe there is room for improvement, and a major revision could significantly strengthen this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and constructive feedback; we have carefully considered your comments and made efforts to address each of the concerns raised. Due to length limitations, we have had to use references from the main paper. We apologize for any inconvenience this may cause.\n\n__[W1] Does Fig. 2 indicate that MAE clusters patches based on visual patterns?__\n\nWe understand that Fig. 2 may not be sufficient to claim visual pattern-based patch clustering. To support this, we refer to Figure III in Appendix, which qualitatively shows token clustering based on their visual pattern. Following your suggestion, we will refer to [40], which demonstrates that 1) MIM models can distinguish tokens and 2) MIM models focus on high-frequency information.\n\n__[W2] Attention map with CLS token__\n\nCorrect. Since the CLS token is not updated during training, it does not contain particularly meaningful information. The main point of Fig. 2c is that if patch vectors are well clustered, they can be distinguished by _any random vector_. We will clarify Fig. 2c in camera-ready.\n\n__[W3] Unclear statement (L126)__\n\nWe agree that MAE mainly aims to obtain a well-trained encoder. However, we discuss both encoder and decoder to show that the entire architecture of MAE aims to learn patch clustering, as our analysis targets the whole MAE. Also, we apologize for the misuse of the term 'pattern' in L126, which was confusing with 'visual pattern'. It would be better to be replaced with 'trend'.\n\n__[W4] Drawbacks of random masking (L195)__\n\nThanks for this clarifying question. In Sec. 3, we discover that _MAE is already well-trained to separate the image into major feature clusters in early epochs_. After the initial $T$ epochs, MAE becomes sufficiently trained to distinguish tokens into major clusters. Random masking, however, keeps producing easily separable training examples, without taking advantage of its learned knowledge so far. We propose a method to exploit this to expedite the learning process in Sec. 4. In this context, we describe it as 'delaying the learning of patch clustering' and identified this as a drawback of random masking. We will clarify this in the paper.\n\n__[W5-1] Difference from [40] (L33)__\n\nTo clarify, the demonstration in [40] about MIM models is confined to 'token distinguishment', distinct from our observation of 'pattern-based patch clustering'. We claim that merely distinguishing tokens is _not sufficient_ to establish our method, as it does not indicate that MAE can generate meaningful patch clusters for informed masking.\n\n__[W5-2] $T=40$ epochs is not early__\n\nIt may be subjective whether 40 epochs are extremely early or not, but our contribution lies in the discovery that MAE trained for less than $T$ epochs is insufficiently trained and would be inadequate for downstream tasks. As this is the empirical minimum, any pre-training method with $T$ or longer epochs will benefit from our method, and in this context we argue $T$ epochs to be an 'early' stage.\n\n__[W6] Difference from prior works (L37)__\n\nWhile our masking strategy shares some common aspects with them, the fundamental difference is that ours relies solely on MAE itself, established through our original analysis of MAE's internal operations. AttnMask [28] relies on supervised training with external labels, unlike our complete self-supervision. We highlight that our main contribution is achieving complete independence from any extra resources while successfully expediting the learning process of MAE.\n\n__[W7,9,10,11] Suggestions for writing quality__\n\nWe thank for the suggestions to improve our manuscript. We will reflect them (overview diagram, notation summary, and summary of experimental implications) in camera-ready.\n\n__[W8] Why do we need exploitation rate?__\n\nEmploying NMI or attention distance might look feasible to decide if the MAE is sufficiently trained to provide informed masks, but they pose the challenge of setting a proper threshold, since they are unbounded. Our exploitation rate quantifies the shared information within the mask tokens compared to the visible tokens, providing a bounded score and thereby easing the thresholding.\n\n__[W12] Effectiveness of bi-partitioning in diverse image contexts__\n\nWe acknowledge the concern regarding images with large objects or multiple objects.\n\nFor an image with a large object, only their sub-parts will be masked because the object occupies more than the masking ratio. This scenario still benefits from our method, as intensive masking on the main object still  leads to better representation (see Fig. II in Appendix).\n\nAs you pointed out, images with many objects might present a challenge for applying informed masking effectively. As there will be many small clusters, masking specific clusters with informed masking may yield similar masks to random masking. However, we emphasize that this limitation is not unique to our method but is a common issue across various informed masking strategies. We will explicitly note this limitation in the paper to provide a clearer understanding of the scope and applicability of our method.\n\n__[W13] Using the last encoder layer for informed masking (L238)__\n\nWe agree with your insight. We simply follow the analysis results and employ the second-last layer, but as pointed out, using the last layer has a minor impact on performance. We will note this in camera-ready.\n\n__[W14] Additional baselines__\n\nWe provide Table B in rebuttal PDF to compare with other methods that use external pre-trained models or additional parameterized modules. Thanks to the external pre-trained models or additional parameterized modules, they slightly outperform our method. However, we emphasize that our method is entirely independent of any extra resources, almost maintaining the original cost of MAE (+0.25% of additional cost), in contrast to other models which require extra cost for training external models and incorporating additional parameters into MAE.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PpBFQwXYfO", "reviewer": "Reviewer_Pvc4", "summary": "This paper proposes a novel masking strategy, Self-Guided Informed Masking for pre-training MAE. The authors found that MAE learns patch-level clustering of visual patterns at the early stages of training, which is demonstrated through an in-depth analysis. Based on this insight, the authors suggest bi-partitioning the image to identify and mask entire objects and then reconstruct them using a small number of hint tokens. Comprehensive experiments across various downstream tasks validate that the proposed strategy consistently improves the performance of MAE.", "questions": "[Q1] Can this approach be applied to other domains or modalities too? It would be more impactful if it could treat other domains since MAE is shown to be the modality-agnostic self-supervised learning [1-2].\n\n[1] Jang, Huiwon, et al. \"Modality-agnostic self-supervised learning with meta-learned masked auto-encoder.\" Advances in Neural Information Processing Systems 36 (2024). \\\n[2] Xie, Johnathan, et al. \"Self-Guided Masked Autoencoders for Domain-Agnostic Self-Supervised Learning.\" arXiv preprint arXiv:2402.14789 (2024).", "limitations": "They addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**[S1]** The paper provides a novel and interesting observation to understand the learning mechanism of MAE, along with a comprehensive analysis to justify the observation.\n\n**[S2]** The paper shows that the proposed approach consistently and significantly improves MAE’s performance on various downstream tasks.\n\n**[S3]** The overall writing is smooth and easy to follow.", "weaknesses": "**[W1]** Lack of baselines. \\\n**[W1-1]** Though they chose only MAE and AMT to show the effectiveness of the approach, it is too limited to fully demonstrate their claim. Specifically, guiding the mask with a pre-trained network should also be considered a baseline if it is not trained in a supervised manner, e.g., AttMask [1]. \\\n**[W1-2]** In addition, HPM [2] should be included as a baseline, since it also proposes an automatic masking strategy that does not use any external data or pre-trained model.\n\n**[W2]** Scalability. \\\nEven though other MIMs [3-4] already have become SOTA in the image domain, I believe the core advantage of MAE is scalability [5] compared to other SSLs, e.g., it performs significantly better with more pre-training epochs, larger models, or more data. The authors should demonstrate the scalability, e.g., by providing the accuracy curve and results on ViT-L/16.\n\n[1] Kakogeorgiou, Ioannis, et al. \"What to hide from your students: Attention-guided masked image modeling.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022. \\\n[2] Wang, Haochen, et al. \"Hard patches mining for masked image modeling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. \\\n[3] Assran, Mahmoud, et al. \"Self-supervised learning from images with a joint-embedding predictive architecture.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. \\\n[4] Wang, Haochen, et al. \"Droppos: Pre-training vision transformers by reconstructing dropped positions.\" Advances in Neural Information Processing Systems 36 (2024). \\\n[5] Singh, Mannat, et al. \"The effectiveness of MAE pre-pretraining for billion-scale pretraining.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We have tried to alleviate all the concerns raised.\n\n__[W1] Additional baselines__\n\nWe agree that it would be valuable to compare our method with others that rely on external pre-trained models or additional modules, as long as the comparison is fair. We included a table comparing our method to these models, including HPM [1] in rebuttal PDF. (We will update the table with unified 400 pre-training epochs in camera-ready.)\n\nUpon thorough understanding of AttnMask [2], the model suggested by the reviewer, we conclude that this approach DOES supervise the model with labels, making it no longer comparable with our self-supervised method. For this reason, we confirm to exclude this for fair comparison. We highlight that our main contribution is achieving complete independence from any extra resources while successfully expediting the learning process of MAE, as demonstrated in our analysis.\n\n__[W2] Scalability__\n\nWe thank the reviewer for the feedback on scalability. Scalability on pre-training epochs can be found in the supplementary materials (~1600 epochs). Although we would like to include what the reviewer suggested now, it is challenging to conduct experiments on larger model sizes and datasets during this short rebuttal period. We will add the experiments on ViT-L/16 with 400 pre-training epochs compared to MAE in camera-ready. \n\n__[Q1] Applicability across various domains and modalities__\n\nSince our method accelerates the training process of MAE without altering MAE's inherent learning (i.e., patch clustering), we believe it is applicable to any domain or modality where MAE itself is suitable.\n\n[1] Wang, Haochen, et al. \"Hard patches mining for masked image modeling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2] Kakogeorgiou, Ioannis, et al. \"What to hide from your students: Attention-guided masked image modeling.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GhqNMYlZCo", "reviewer": "Reviewer_3PrK", "summary": "This paper aims to enhance the Masked Autoencoder (MAE) approach for self-supervised learning in computer vision. The authors discovered that MAE intrinsically learns pattern-based patch-level clustering from the early stages of pre-training. This finding led them to propose a self-guided masked autoencoder, which internally generates informed masks based on the learned patch clustering, rather than relying on random masking. This method improves the learning efficiency without the need for external models or supplementary information, maintaining the self-supervised nature of MAE. Their approach involves generating informed masks that cover the main objects in an image more effectively than random masks. These masks are derived from the early-stage patch clustering learned by MAE, accelerating the training process and leading to clearer and more refined feature embeddings. The self-guided masked autoencoder leverages the inherent pattern-based clustering learned by MAE to produce more effective masks, enhancing the training efficiency and performance of the model on various computer vision tasks.", "questions": "1. Does this work perform the best under the default mask ratio (0.75) similar to other works?\n2. Have you considered using stochastic $S_i$ (weighted sampling) in the hinted method? In addition, I think sampling hint patches from different clusters might be useful.", "limitations": "The author has addressed the limitation in this paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The self-guided masked autoencoder is innovative, as it generates informed masks internally based on the model’s own learning progress. This method is also backed with sound motivations.\n2. The paper provides an in-depth analysis of the MAE’s learning process, revealing that pattern-based patch-level clustering occurs from the early stages of pre-training. This understanding contributes insights into the workings of MAE and can inform future research.\n3. This paper also presents comprehensive analysis on learned feature space, providing multiple views of understanding of this method.", "weaknesses": "1. The success of the self-guided masked autoencoder hinges on the accuracy of the initial patch clustering. If the initial clusters are not well-formed, the informed masks generated could be suboptimal, leading to less effective learning and reduced performance gains.\n2. The process of bi-partitioning and generating informed masks adds significant computational overhead. This extra complexity can increase the overall training time and resource consumption. It would be better if the author could include the training time or FLOPS for their method. \n3. Lack of baselines. The author has mentioned multiple informed masking strategies, and they should be included to this paper for comparisons (CL-MAE, ADIOS). Currently, only MAE and AMT served as baseline method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' positive comments and constructive feedback. We have made efforts to address each of the concerns raised.\n\n__[W1] Quality of informed masks__\n\nWe acknowledge the concern that initial clusters may not be well-formed for some images. Although we have shown that MAE properly clusters patches through the observations in Sec. 3, it is true that the clustering quality may sometimes be suboptimal. To address this issue, we proposed a 'relevance score' that measures the relationship of the entire token set to the bi-partitioned cluster and then generates masks based on this score (Sec. 4 L222). As a result, our approach yields robust informed mask even in the case when the clustering and bi-partitioning is imperfect, illustrated in Fig. II in Appendix A.\n\n__[W2] Training cost__\n\nOur method only requires only one more step to generate masks, which empirically increases the pre-training time about 0.25% (for 400 epochs pre-training). We will make this clearer in camera-ready.\n\n__[W3] Additional baselines__\n\nWe thank this consturctive feedback. As AMT is the only model soley relying on MAE archietcture itself, we chose it as baseline. We exclude CL-MAE [1] and ADIOS [2] from our baselines since both of them utilize additional module such as curriculum masking module or occlusion module which are parameterized and need to be trained in addition to MAE. The reported figures in their original papers are not comparable, since CL-MAE experimented only on a subset (20%, randomly selected 200 classes) of ImageNet-1K (IN1K), which does not align with our main purpose of self-supervision on large datasets. Also, as it takes about 6x training time for CL-MAE empirically, we will add CL-MAE in camera-ready due to the short rebuttal period. ADIOS was also experimented in its original paper with a downsized version (224->96) of a subset (10%, ImageNet-100) of IN1K and is not applied to the MAE architecture. Admitting their relevance, we will introduce CL-MAE and ADIOS in the related work section.\n\nReflecting the reviewer's point, we compare with two other models (SemMAE [3] and HPM [4]) using additional module or external model in Table B of the rebuttal PDF. Since these baselines incorporate additional parameters (external models or extra modules attached to MAE), they slightly outperform ours. However, these models incur significantly larger training cost for the additional parameters, while our method requires only a constant additional time (about 0.25% of the training cost of MAE) for mask generation relying solely on MAE itself without introduction of any additional resources.\n\n__[Q1] Ablation on masking ratio__\n\nWe provide an ablation study on masking ratio in Table C in the rebuttal PDF. A masking ratio of 0.6 shows slightly better performance (+0.1%), while a masking ratio of 0.9 shows degraded performance, which shows similar trend to the results in [5]. We had to conduct this experiment with 200 pre-training epochs due to the short rebuttal period, but will add a similar table with regular 400 epochs of pre-training in camera-ready.\n\n__[Q2] Hinting strategy__\n\nBoth ideas are very interesting, and we explored them in an ablation study. Consideration of using $S_i$-based sampling is shown in the ablation studies (Ln 273-277, Table 3) in main paper, where it exhibited slightly degraded performance. Sampling hint tokens from different clusters is presented in Table I in the supplementary material. We sampled hint tokens from two major clusters alternately throughout the training epochs, which also resulted in slightly lower performance compared to our default setting.\n\n[1] Madan, Neelu, et al. \"CL-MAE: Curriculum-Learned Masked Autoencoders.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.\n\n[2] Shi, Yuge, et al. \"Adversarial masking for self-supervised learning.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Li, Gang, et al. \"Semmae: Semantic-guided masking for learning masked autoencoders.\" Advances in Neural Information Processing Systems 35 (2022): 14290-14302.\n\n[4] Wang, Haochen, et al. \"Hard patches mining for masked image modeling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[5] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear all reviewers,

Thank you for your time and effort in reviewing our paper. We have carefully addressed all the concerns raised and incorporated the valuable suggestions. We kindly request that you re-evaluate our paper in light of these rebuttals.

Meta-review: This paper explores the properties of MAE and introduces an improved masking strategy for MAE. Specifically, it proposes a self-guided masked autoencoder that internally generates informed masks by leveraging its progress in patch clustering. All reviewers agree that the analysis is interesting and that the results are robust compared to baseline methods. Concerns regarding a more detailed analysis and differentiation from previous work have largely been addressed during the rebuttal. Final ratings ranged from Borderline Accept to Weak Accept. Given the intriguing findings, clear methodology, and significant results, the Area Chair recommends accepting the paper.
Decision: Accept (poster)