Exploring The Role of Mean Teachers in Self-supervised Masked Auto-Encoders
Authors: Youngwan Lee
Jeffrey Ryan Willette
Jonghee Kim
Juho Lee
Sung Ju Hwang
Venue: ICLR 2023 poster (2023)
Keywords: self-supervised learning
masked auto-encoder

Abstract: Masked image modeling (MIM) has become a popular strategy for self-supervised learning (SSL) of visual representations with Vision Transformers. A representative MIM model, the masked auto-encoder (MAE), randomly masks a subset of image patches and reconstructs the masked patches given the unmasked patches. Concurrently, many recent works in self-supervised learning utilize the student/teacher paradigm which provides the student with an additional target based on the output of a teacher composed of an exponential moving average (EMA) of previous students. Although common, relatively little is known about the dynamics of the interaction between the student and teacher. 
Through analysis on a simple linear model, we find that the teacher conditionally removes previous gradient directions based on feature similarities which effectively acts as a conditional momentum regularizer. From this analysis, we present a simple SSL method, the Reconstruction-Consistent Masked Auto-Encoder (RC-MAE) by adding an EMA teacher to MAE. We find that RC-MAE converges faster and requires less memory usage than state-of-the-art self-distillation methods during pre-training, which may provide a way to enhance the practicality of prohibitively expensive self-supervised learning of Vision Transformer models. Additionally, we show that RC-MAE achieves more robustness and better performance compared to MAE on downstream tasks such as ImageNet-1K classification, object detection, and instance segmentation.

Published as a conference paper at ICLR 2023
EXPLORING THE ROLE OF MEAN TEACHERS IN SELF- SUPERVISED MASKED AUTO-ENCODERS
1,2 Youngwan Lee' JJeffrey Willette 'Jonghee Kim
JJuoo Lee
SSun Ju Hwang
Electronics and Telecommunications Research Institute (ETRI), South Korea
2 Korea Advanced Institute of Science and Technology (KAIST), South Korea
ABSTRACT
Masked image modeling (MIM) has become a popular strategy for self-supervised learning (SSL) of visual representations with Vision Transformers. A representa- tive MIM model, the masked auto-encoder (MAE), randomly masks a subset of image patches and reconstructs the masked patches given the unmasked patches. Concurrently, many recent works in self-supervised learning utilize the studen- t/teacher paradigm which provides the student with an additional target based on the output of teacher composed of an exponential moving average (EMA) of pre- vious students. Although common, relatively little is known about the dynamics of the interaction between the student and teacher. Through analysis on a sim- ple linear model, find that the teacher conditionally removes previous gradient directions based on feature similarities which effectively acts as a conditional mo- mentum regularizer. From this analysis, we present a simple SSL method, the Reconstruction-Consisee
Masked Auto-Encoder (RC-MAE) by adding an EMA teacher MAE. We find that RC-MAE converges faster and requires less mem- ory usage than state-of-the-an self-distillation methods during pre-training, which may provide way to enhance the practicality of prohibitively expensive self- supervised learning of Vision Transformer models. Additionally, we show that RC-MAE achieves more robustness and better performance compared to MAE downstream tasks such as ImageNet- IK classification, object detection, and instance segmentation.
INTRODUCTION
The Transformer (Vaswani et al., 2017) is the de facto standard architecture in natural language processing (NLP), and has also surpassed state-of-the-ar Convolutional Neural Network (He et al., 2016; Tan & Le, 2019) (CNN) feature extractors in vision tasks through models such as the Vision Transformer (Dosovitskiy et al., 2021) (ViT). Prior to the advent of ViT's, self-supervised learn- ing (SSL) algorithms in the vision community (He et al., 2020; Chen al., 2020c; Grill al., 2020; Chen et al., 2021) utilized CNNs (e.g., ResNet (He et al., 2016)) as a backbone, performing instance discrimination pretext tasks through contrastive learning (He al., 2020; Chen et al., 2020c). Inter- estingly, self-distillation schemes (Grill et al., 2020; Caron et al., 2021) using teacher consisting of an exponential moving average (EMA) of the previous students, (i.e., a "mean" teacher) (Tarvainen & Valpola, 2017), have been shown to exhibit strong performance.
Inspired by the success of masked language modeling (MLM) pre-training in NLP, recent SSL ap- proaches (Bao et al., 2022; Zhou et al., 2022; Xie et al., 2022; He et al., 2022: Assran et al., 2022) in the vision community have proposed forms of masked image modeling (MIM) pretext tasks, using ViT-based backbones. MIM is simple pretext task which first randomly masks patches of im- age, and then predicts the contents of the masked patches (i.e., tokens) using various reconstruction targets, e.g., visual tokens (Bao et al., 2022; Dong et al., 2021), semantic features (Zhou al., 2022; Assran et al., 2022) and raw pixels (He et al., 2022; Xie et al., 2022). In particular, iBOT (Zhou et al., 2022) and MSN (Assran al., 2022) use self-distillation scheme for MIM by having the teacher network provide an encoded target (i.e., feature representation) to match the encoded features from
cqual contribution

---

Published as a conference paper at ICLR 2023
03.50
Recisstrente cesterrrtttt Gradiest Moniraton
01.3
Teacher Carronted Gradiant Montertam
Decader
EMA
Dosoder
$3.00
Presious Mamertoo
90
02.50
Inoider f4.
Encadar
62.25
Recumstrnctinn Groslient ggh
82.00
MAE
Patchity
RC-MAE
PoLe
200
400
630
800
1000
1206
3400
1600
Consistoncy Gradient (Cunditional)
original Input
Epech
Gradient Correction
RC-MAE
(c) Fine-Tuning on ImageNet-1K
Figure 1: Overview. (a): When the inputs which led to the previous gradients and current gra- dients are similar, the consistency gradient provides conditional correction, allowing the stu- dent to learn from newer knowledge. (b): In RC-MAE, the reconstructed patches from the student are compared with the original input (reconstruction loss L,), and with the predicted patches from the teacher (consistency loss L.). (c): ImageNet-Ie 1K Fine-tuning top-1 accuracy curve: RC-MAE achieves comparable accuracy (83 4%) at 800 epochs compared to MAE trained for 1600 epochs.
the original image at a semantic feature level (i.e, global abstraction). Methods using semantic- target representations exhibit strong performance image-level classification tasks. On the con- trary, SimMIM (Xie et al., 2022) and MAE (He et al., 2022) provide pixel-level reconstructions of masked patches, and lead to superior performance dense prediction tasks such as object detection and segmentation. However, self-distillation for pixel-level MIM has been under-explored as of yet.
A recent SSL approach, BOL (Grill et al., 2020), has shown that slight architectural asymmetry between a student and EMA teacher can create a stable model which outperforms previous con- trastive learning methods. The success of BYOL (Grill et al., 2020) inspired empirical (Chen & He, 2021) and theoretical (Tian et al., 2021) analyses into what enables BOLL to effectively learn and avoid collapse with the EMA Teacher during pre- pre-training. Still. despite the popularity of the EMA Teacher in SSL,, relatively little is known about how the teacher interacts with the student throughout the training process.
In this work, we explore the dynamics of self-distillation in pixel-level MIM, egg., MAE Through analyzing simple linear model, we investigate the dynamics between the gradients of an image re- construction loss and a teacher consistency loss, learning that the gradients provided by the teacher's s consistency loss conditionally adjust the current reconstruction gradient by a weighted mixture of previous gradients. The weights of the mixture previous features. Thus, the teacher acts like a conditional momentum regularizer. For example, Fig. 1(a) shows the case where the inputs which created the previous gradient momentum are sim- ilar to the ones which created the current gradients. In this case, the teacher makes a conditional correction remove the previous direction from the momentum, allowing the student to learn from the newer knowledge in the current batch. If however, the inputs which created both gradients are nearly orthogonal, the teacher would instead respond with minimal to correction. We derive this conditional gradient effect in Proposition 4. and show evidence in both a simple linear model as well as in deep ViT-based (Dosovitskiy et al., 2021) MAE model (Fig. 2).
derived from similarities between current and
To empirically validate our analysis of the contributions of EMA Teachers, we present a simple yet effective SSL. approach, the Reconstruction- Consistttnnnn Masked Auto-Encoder (RC-MAE), by equipping MAE with an EMA Teacher, and providing a consistency target. Additionally, we study the effects using different image masking strategies between the student and teacher models on the consistency objective, finding that using the same mask generally leads to better performance in both pre- pre-training and downstream tasks. The same mask tends to form an orthogonal objective (Fig. 3(b)) to the reconstruction loss, which has been shown (Suteu & Guo, 2019; Ajemian al., 2013) to be beneficial for multi-tasi models as there is limited interference between tasks. This observation may be of interest any future SSL works which leverage multiple pre-training objectives.
2

---

Published as a conference paper at ICLR 2023
Our experiments follow the same architecture, settings, and pre-training recipe as MAE (He al., 2022), and we find that the simple addition of a teacher (RC-MAE) consistently outperforms MAE in all model sizes (e.g.. ViT-S, ViT-B, and ViT-L) when fine-tuned for ImageNet classification. Ad- ditionally, we find that the teacher's ' conditional gradient correction we identified allows RC-MAE to converge faster compared to MAE (Fig. I())), and RC-MAE outperforms recent self-distillation methods, MSN and iBOT, on dense prediction tasks such as object detection and instance segmenta- tion. Furthermore, compared to recent self-distillation methods utilizing mean teacher, RC-MAE realizes more efficiency in computation and memory due to the fact that both networks receive only a subset of patches instead of the whole image. Our main contributions are as follows:
1. We analyze the contribution EMA Teachers in sulf--su learning, finding that the gradient provided by the teacher conditionally adjusts the current reconstruction gradient direction and magnitude conditioned on the similarity of current and previous features.
2. Using this knowledge, we propose simple, yet effective approach for self-supervised pre-training of Vision Transformers, the Reconstruction- Consistent Masked Auto- Encoder (RC-MAE), which improves over vanilla MAE terms of speed of convergence, adversarial robustness, and performance on classification, object detection, and instance segmentation tasks.
3. Thanks its simplicity, RC-MAE achieves greater savings in both memory and computa- tion compared to other state-of-the-ar self-distillation-boe MIM methods.
2 RELATED WORKS
In NLP, masked language modeling (MLM) is common for large-scale pre-training (Devlin et al., 2019; Radford et al., 2018) by predicting masked words. Similarly, VIT (Dosovitskiy et al., 2021; Liu et al., 2021; Lee et al., 2022) based masked image modeling (MIM) approaches (Zhou et al., 2022; Bao et al., 2022: He et al., 2022; Xie et al., 2022; Assran et al., 2022) for computer vision tasks have been proposed. These MIM approaches first apply a mask to patches of an image, and then the masked patches are predicted given the visible patches either at the token-level (Zhou al., 2022: Bao et al., 2022; Assran et al., 2022) or pixel-level (Chen et al., 2020b; Xie et al., 2022; He et al., 2022). Token-level masked patch prediction (Zhou et al., 2022; Assran et al., 2022; Bao al., 2022) predicts tokens or clusters of masked patches similar to MLM. Pixel-level prediction (Chen et al., 2020b; Xie et al., 2022; He et al., 2022) learns visual representations by reconstructing masked input patches the RGB pixel-level.
Additionally, self-distillation (Grill et al., 2020; Caron et al., 2021; Chen et al., 2021) has been deployed in MIM methods by utilizing teacher constructed from an exponential moving aver- age (EMA-Teacher) of student weights, providing an additional target for the student. iBOT (Zhou et al., 2022) gives full view of image (i.e., all patches) to the teacher network tokenizer, offering a token-level target of the masked patches. Also giving a masked view to the student and full view to the teacher, MSN (Assran et al., 2022) makes the output embeddings from an EMA-Teacher serve as semantic feature representation target to the student. Likewise, BootMAE (Dong et al., 2022) also adopts EMA-Teecher. providing a feature-level target the student on top of the pixel-level MIM approach. A key difference from these self- oelf-dstiilation MIM approaches is that RC-MAE provides only unmasked patches to the teacher and student. instead of the full image. As a result, RC-MAE shows better scalability compared with recent methods (see. Table 6).
online
PRELIMINARIES
The Masked Autoencoder. (MAE) (He et al., 2022) self-supervised approach with ViT en- coder f and decoder h, which randomly masks portion of input patches, and then reconstructs the masked patches given the visible patches. Given an image X RCHHW MAE patchifies X into N non-overlapping patches X E Nx(''))) with a patch size of P and randomly masks a subset of patches M (i.e.. mask tokens). The subset of visible patches is input to the encoder achieve latent representations: z = f(V). Then, the decoder & attempts to reconstruct A given the latent representations, γ M); where ÿ Nx(2-C) denotes the reconstructed patches. MAE
3

---

Published as a conference paper at ICLR 2023
utilizes the mean-squared error reconstruction loss C, which is only computed on masked patches:
Σ = = M 2c,
X - 2
(1)
The EMA Teacher. The mean teacher model (Tarvainen & Valpola, 2017) is a temporal ensemble of previous student weights which provides an additional target the student. Doing so has been shown to reduce the number of labels needed to achieve the same level of accuracy, and has become core part of recent state-of-the-oe art SSL approaches as reviewed in Section 2. Predictions from the student and teacher are compared via function such mean squared error (Tarvainen & Valpola, 2017) or cross-entropy (Caron al., 2021). Generally, the teacher updated after every gradient step on the student S, using exponential moving average of the student weights,
T(t) aT((=)) + (1-a)s() E " Ea'(1-a)s(>),
i-O
with a parameter a e (0, 1). The additional target forms a consistency loss Le between the teacher
and the student predictions. Considering the mean squared error loss, and Y' being the prediction from the teacher model,
Σ P 2112 M H.M
L. =
THE ROLE OF THE TEACHER
Although EMA teachers are common in recent SSI. approaches, relatively little is known about the interaction between the student and teacher. Through analysis of linear model that mirrors MAE+Teacher objective, we will show how the gradients linear model for the student S and teacher I consisting of single weight matrix, like an MAE, the objective is to reconstruct the original input x from a masked input X = x @ m, where @ is an elementwise multiplication and m is random binary mask with predefined masking ratio.
both models interact. Considering a
Proposition 4.1. With the reconstruction and consistency objective (Eqs. (1) and (3)), the gradient contribution of the teacher (V CC.) adjusts the direction and magnitude of the reconstruction gradi- ents (V sL,). The magnitude and direction of the adjustment from the teacher conditional based on the similarity between the current and previous features. With x representing an independent input from previous timestep,
VgLr + VsL. &
SS - x12
xx= V
|S8 - S1opGrad(13011
Sxx
xxx
SSx
TRx
(t-)
(4)
Six xx
Σ A
Six xx + SSx Txx
xx
Vgc,
Ve from
Proof. Please see Appendices B and B.1.1
The gradient of the consistency loss S&C is wholly represented by the last term the RHS. Inter- estingly, there is a dot product & & which gets distributed into every term of the sum. If we consider the possible cases for cos x) for x
the dot product as cosine similarity cos( x x | at a single arbitrary timestep are as follows cos( (x, x) (-1,0, (0.1),,
Case 1: cos x) 1). In this case, the resulting gradient from the last term on the RHS of Eq. (4) removes some amount residual memory of the direction of a previous gradient. A cosine similarity of [1| also means the inputs collinear, and the gradient invariant the sign of x &
Case 2: cos (x,( = O. In this case, There is zero contribution from the teacher for this term in the sum.
Case 3: /cos( (x. x) (0. 1). In this case, the component which contains the previous gradient will be weighted by the coefficient &
4

---

Published as a conference paper at ICLR 2023
In all cases, due to the sign of the last term on the RHS of Eq. (4), the gradient of consistency loss conditionally removes residual memory of previous gradients. The magnitude of the removal is likewise conditional, which can be seen by using the triangle inequality upper bound the final term of Proposition 4.1 to see that,
VV = a' aV'AV go- (( C s
Va'A\/Vss-oll
su-o [xx]]
(5)
i=1
1-1
leading to the conclusion that the magnitude and direction of the consistency gradient directly result from the similarities to features in the recent learning history, as a² e (0, 1) decays exponentially for distant timesteps in the past. The gradient of the student model can be bounded in similar fashion, but without the decaying a coefficients which makes the bound much looser in general (see Appendix B.1).
Empirical Test. To test for this effect in a linear model, we conducted experiment by training a linear model on data consisting of random samples variate normal distributions (see Appendix E.1 for further details). After each training iteration, we sampled an extra batch of data, and for each single point in the batch we constructed sequences consisting of two inputs described in Table 1. We then took a single gradient step and teacher update for the first input and calculated the gradient of the reconstruction and consistency loss on the second input.
Table
Input sequences used in Fig. 2
Name
Input
Description
Case (x.8)
same input twice (same)
IR from random multi-
Case 2
(x. 8)
different inputs (different)
Case
(x,x)
same input with different mask (similar)
Expected Outcome. Based on the similarity of the inputs, we would expect the consistency loss to produce a larger gradient for the same or similar inputs and a smaller gradient for different in- puts. Additionally, the direction of the reconstruction and consistency gradient should be closer to opposite for case 1, and closer to orthogonal for case 2, with case 3 falling somewhere in-between. In Fig. (top row), we in fact observe this trend, noting that the reconstruction loss produces a significantly larger gradient when the second gradient step is on different inputs due to the looser bound in Eq. (16).
Interpretation. This finding implies that the teacher plays role of something akin to gradient memory, where the teacher acts as a memory bank which retrieves the memory of recent gradients based matching query & to a key & in the memory bank. For novel inputs which do not match anything in recent memory, the teacher responds with a minimal correction, letting the student learn more from the reconstruction signal. If the query and key match, however, the teachers gradient will conditionally remove some directional information contained in the previous gradient. This allows the student to move in a direction which favors new knowledge gained from the current input, and cancels out some previous momentum. This process is illustrated in Fig. I(a). In Appendix D, we show that the same terms appear in the context of a deep model, with the dot product appearing at each semantic feature level. However, in a complex model with nonlinearities, the resulting gradient direction becomes harder to interpret. Even so, in Fig. 2 (bottom row), we empirically find the same underlying trend in the gradient norms and directions when analyzing RC-MAE (a VIT based model).
RECONSTRUCTION-CONSSSI
MASKED AUTO-ENCODER
In this section, utilize the analysis from Section 4, and present a simple self-supervised ap- proach, Picoostraction-Coniiin
Masked Auto-Encoder (RC-MAE). The overview of RC-MAE is illustrated in Fig. 1(b). RC-MAE is a ViT-based version of the simple linear model outlined in Sec- tion 4 where the total objective consists of reconstruction and consistency loss from the original image and a mean teacher, respectively. The teacher network shares the same architecture as the student, consisting of an encoder fe, (e.g., ViT (Dosovitskiy et al., 2021)) and decoder he.-
At timestep 6, the model parameters of the teacher 0, are updated to be the exponential moving average (EMA) of the student model parameters 0, (Eq. (2)). While BYOL (Grill et al., 2020), DINO (Caron et al., 2021), and MSN (Assran et al., 2022) make the student network mimic the teachers semantic-level representation, for RC-MAE, the consistency and reconstruction targets both pixel-level representations. We would expect that the two pixel-level objectives would lead to better performance on dense prediction tasks. Additionally, we would expect the conditional
5

---

Published as a conference paper at ICLR 2023
Same
Similar
Different
Teacher
Student
Codtoe Siilletty
600
200
-0.-
-1.
100
300
200
020
100
201
20
000
50
38C
30
303
4E0
30
Iterations
Iteratioos
Derratins
(a) V&
(b) VV
(c) CosSim( ( Vsc((+))
1,
0.36
0.31
Smilortt
0.5
0.25
0.00
0.31
0.8
0:15
0:0
0.01
0.5
250
1000
2331
11801
1291
1500
250
1001
2301
1100
1221
1330
1000
1100
1.501
1301
1131
1860
100
Herslion
Hettlioo
Hletaloons
(d) RC-MAE |V
(e) RC-MAE |
(1) CosSim(Vs (((
Figure 2: Top Row: Linear model. Bottom Row: RC-MAE (a ViT). Both models show the same trends. For three input sequences Table we performed one gradient step/teacher update and then calculated sf,, and V she at the next step. Figs. 2(a) and 2(d): V F. is larger when cos &, x) large (Eq. (5)). Figs. 2(b) and 2(e): The same and different lines do not follow the same order for the reconstruction loss. Figs. 2(c) and 2(f): The direction of sCC is conditioned on cos (x, x). All together: For v sLc, when cos( &, x) = 1 there tends to be larger move in a negative direction while cos( & x) 0 there tends to be smaller move in an approximately orthogonal direction.
momentum corrections from the consistency gradient to allow for quicker and more stable rate of convergence, due our interpretation in Section of the momentum corrections supplied by the teacher.
To investigate masking strategies for the teacher, define two types of RC-MAE, RC-MAE-S and RC-MAE-D, RC-MAE-S uses the same mask tokens for both student and teacher networks. Conversely, RC-MAE-D, uses different mask tokens for each network. For both variants, given an image, we randomly sample respective mask tokens M with a mask ratio (e.g., 75% like MAE). The visible patches are then given the student and teacher networks, which reconstruct all patches. With the consistency loss Eq. (3), the RC-MAE objective is:
[ FEM
reconstruction
consistency
where i is the token index and V, V" denote the reconstructed patches from the decoders of the student and teacher networks, respectively. For the reconstruction target, we use standard normalized patch pixels as done with MAE (He et al., 2022). For RC-MAE-D the student and teacher process different visible patches, and the total loss is only calculated on the student's mask token locations M,. Unless otherwise specified, RC-MAE means RC-MAE-S in the rest of this paper.
6 EXPERIMENTS
To validate the effectiveness of the EMA Teacher in RC-MAE, we compare RC-MAE to an oth- erwise identical MAE (He et al., 2022) baseline. For the EMA Teacher, We use ViT (Dosovitskiy et al., 2021) as an encoder and transformer decoder with the same depth and dimension as MAE. For fair comparison with MAE, follow the same implementation MAE's official code. For experiments, we pre- trait on ImageNet-Il IK and evaluate linear probing (LN) and end-to-end fine- tuning (FT) for classification and COCO object detection & instance segmentation for which

---

Published as a conference paper at ICLR 2023
Reconstruction Consistency gradients
MAE RC-MAF-D RC-MAE-S
1.4
Reconstructin 1.44
Probabiliy Dersity
800
1802
-0.25
-2.50
-3.25 0.00
0.5
0.5 9.75
Epoch
distribution
(a) Reconstruction loss
(b) Cosine similarity
Figure 3: (a): L, is high for RC-MAE-D compared with MAE, even though RC-MAE-D outper- forms MAE in downstream tasks (see Table 2). (b): RC-MAE-D likely causes conflict between L, and L. (described Appendix C), and the resulting cos Cr, V.,) shows a positive bias. Inter- estingly, RC-MAB-S shows a similar distribution as the special cases studied in Fig. 2(f) indicating that the same situations arise in an i.i.d. training setting.
Table 2: Different mask types for the teacher network in RC-MAE. The 'Same' means that the teacher network uses the same mask tokens as the student and 'Diff'' is the opposite. FT and LN denote end-to end fine-tuming and linear probing on ImageNet- I K, respectively.
Table 3: Different masking ratio. GPU time means pre-training time (hours) on 8 V100- 32GB GPUs environment.
Method
Epoch
Mask Ratio
FT
GPU Time
MAE
800
75%
83.2
125.5h
RC-MAE
800
75%
83.4
166.6h
Method
Mask
FT
LN
Aphox
Apr
RC-MAE
RC-MAE
800
800
80%
85%
83.4
83.2
136.4h
142.6h
MAE
83,4
67.3
50.3
44.9
MAE
1600
75%
83.4
256.5h
RC-MAE-D
Dift.
83,6
68.7
50.7
45.2
RC-MAE
1600
75%
83.6
331.1h
RC-MAE-S
Same
83.6
68.4
51.0
45.4
RC-MAE
1600
80%
83.5
283.3h
use the Mask R-CNN benchmark (Li et al., 2021) for dense prediction. implementation and training settings can be found in Appendix E.
Further details regarding
6.1 ABLATION STUDY
Mask token for Teacher: Same vs. Different. To investigate the effect of masking strategies between teacher and student, we evaluate RC-MAE variants against a plain MAE. Results are shown in Table 2. Both RC-MAE-S and RC-MAE-D outperform MAE, achieving higher accuracy gap in linear probing (1.4% 1.)%) compared to fine-tuning (0.2%). Notably, RC-MAE-S shows higher performance than RC-MAE-D on object detection and instance segmentation while the performance gap for classification slightly favors RC-MAE-D. D. To analyze the differences between RC-MAE- and RC-MAE-S, we recorded the cosine similarity between the reconstruction and consistency gradients throughout training Fig. 3(b). We found that RC-MAF-D resulted in cosine similarity with a positive bias, while RC-MAE-S appears to be centered near zero. We hypothesize that a conflict between the pixel-level reconstruction and consistency objectives causes the biased cosine similarity in RC-MAE-D, leading to lower pre-training and subsequent fine-tuning performance on these tasks. The base conflict likely stems from the fact that MAE models tend to give lower- quality reconstructions on visible patches. Therefore, when patch is masked in the student but not the teacher, the teacher provides a low-quality consistency target, interfering with the high-quality original input reconstruction target.
Masking ratio Training epoch. We compare to MAE using different masking ratios, measur- ing pre-training time on the same 8-GPU machine. Per-iteration, RC-MAE is slower due to the added teacher network. However, compared to MAE with 75% mask & 1600 epochs (default), RC-MAE with 80% mask & 800 epochs achieves the same level of performance while using 55% less time. Fig. (cc) illustrates fine-tuning accuracy curves of RC-MAE (75% mask & 800 epochs) and MAE (75% mask & 1600 epochs). It is easy observe that RC-MAE reaches higher accuracy quicker than MAE. As the teacher is the only difference between the models, the increased rate of convergence is direct result from the conditional teacher gradients.

---

Published as a conference paper at ICLR 2023
Table 4: End-to-end fine-tuning on ImageNet-IK. All results (top-1 acc.) are trained with 224 224 input. MAE (our impl.) were trained using the official code with the training configuration on the same V100-32GB GPUs environment by ourselves. Note that there are reproducing issues.
Method
Pre- Train Data
Pre-Train Epochs
ViT-S
ViT-B
ViT-L
Supervised. (Touvron al., 2021)
INIK w/ labels
300
79.9
81.8
82.6
DINO (Caron et al., 2021)
INIK
800
81.5
82.8
MoCo v3 (Chen et al., 2021)
INIK
300
81.4
83.2
84.1
BEiT (Bao et al., 2022)
I
800
81.7
83.2
85.2
MSN (Assran al., 2022)
INIK
600
83.4
iBOT (Zhou al., 2022)
INIK
800
82.3
84.0
84.8
BootMAF (Dong et al., 2022)
INIK
800
84,2
85.9
MAE (He et al., 2022)
INIK
1600
83.6
85.9
MAE (our impl.)
INIK
1600
81.8
83.4
85.5
RC-MAE
INIK
1600
82.0
83.6
86.2
Table 5: COCO object detection and segmentation using
Table 6: Resource Comparison with Self-distillation based MIM methods: GPU Memory and runtime mea- sured using V V100-32GB GPUs with batch size of 128 for ViT-L.
Mask R- R-CNN with ViT-Base backbone. For fair compar-
ison, we follow the benchmarking transfer learning proto-
col (Li et al., 2021).
Method
Pre Train Data
AP
Apmusk
Superivsed (He et al., 2022)
INIK w/ labels
47.9
42.9
Method
Memory
Throughpul
MoCo v3 (Chen et al., 2021)
INIK
47.9
42.7
MAE
84G
533 imgs/s
BEiT (Bao et al., 2022)
I
49.8
44.4
MSN
183G
78 imgs/s
MSN (Assran et al., 2022)
INIK
46.6
41.5
iBOT (Zhou et al., 2022)
INIK
47.3
42.2
iBOT
227G
123 imgs/s
MAE (He et al., 2022)
INIK
50.3
44.9
BootMAE
98G
376 imgs/s
RC-MAE
INIK
51.0
45.4
RC-MAE
95G
441 imgs/s
6.2 END-TO-END FINE-TUNING ON IMAGENET-IE 1K
Table 4 shows ImageNet fine-tuning results. Note that our reproduced MAE (e.g. ViT-B and ViT- L) the same 8 GPUs machine shows slightly lower accuracy than the original MAE due known reproduction issue. We assume that the slight difference accuracy stems from the different GPU environments. For all VIT backbone sizes, RC-MAE consistently outperforms the reproduced MAE. While ViT-S and ViT-B achieve lower accuracy than iBOT or BootMAE, RC-MAE with ViT- surpasses the state-of-the-art methods. These results suggest that the reconstruction-consiitee scheme could have stronger benefits for large-scale models.
6.3 OBJECT DETECTION AND SEGMENTATION ON COCO
To validate the pixel-level representation quality of RC-MAE, we analyze performance on object detection and instance segmentation on COCO (Lin al., 2014). Following the same training protocol (Li et al., 2021) and implementation details as MAE (He et al., 2022), we fine-tune Mask R-CNN (He al., 2017) with a ViT-Base backbone tre-trained by RC-MAE for 100 epochs. The training details are described in Appendix E.4. Table summarizes the detection and segmentation results. RC-MAE improves over the MAE baseline by 0.7% box AP and 0.5% mask AP. We note that both RC-MAE and MAE outperform iBOT by large margins, even though iBOT shows higher ImageNet classification accuracy. These results suggest that the pure pixel-level objectives allow models to learn richer representations at the region or pixel-level. Moreover, as illustrated in Fig. 4, while iBOT and MSN attend to uncorrelated regions given the query patches, RC-MAE and MAE focus more sharply the pertinent regions near the query patches. These results indicate that pixel-level MIM more advantageous than semantic-level MIM for dense prediction tasks.
6.4 RESOURCE COMPARISON: MEMORY AND COMPUTATION TIME
In terms of memory consumption and computation cost, we conduct a resource comparison with the self-distillation- based MIM methods, e.g., MSN, iBOT, and BootMAE as shown Table 6. More details about how to measure described in Appendix F22. Since adding the EMA Teacher to MAE, RC-MAE shows more memory usage and runtime. However, compared with state-of-the-an the-art self-distillation methods, RC-MAE requires less memory usage and computation cost. This result
8

---

Published as a conference paper at ICLR 2023
Query
RC-MAE
MAE
MSN
IBOT
MoCo v3
Figure 4:
Attention maps obtained by thresholding 10% mass of the query patch (*)). RC- MAE & MAE attend regions more precisely with respect to the query patch while MSN, iBOT, and MoCo v3 likely attend the unrelated region with the query patch.
Robustness Evaluation on ImageNet-variamt watteaaaaaaaammmmeeeee-- (Hendrycks & Dietterich, 2019), ImageNet-A (Hendrycks et al., 2021b), ImageNet-R (Hendrycks et al., 2021a), and ImageNet-SSktchh (Wang et al., 2019). Except for ImageNet-C which measured in terms of mean Corruption Error (mCE), top-1 accuracy is used as the metrie for IN-A,-R, and -Sketch.
Table 7:
Method IN-C (mCE J) IN-A (top-1 1)
IN-R (top-1 t) IN-Sketch (top-1 1)
MAE
51.7
35.9
48.3
34.5
RC-MAE
49.9
36.5
49.5
35.8
can be attributed to the fact that other methods feed the full image the teacher network, whereas RC-MAE only delivers visible patches (no mask patches) to the teacher same as the student.
6.5 ROBUSTNESS EVALUATION ON IMAGENET
We evaluate the robustness models on four ImageNet variants: ImageNet-C (Hendrycks Diet- terich, 2019),-A (Hendrycks al., 2021b).-R (Hendrycks al., 2021a), and -Sketch (Wang al., 2019), which are common benchmarks for evaluating robustness to perturbations. Table shows the robustness comparison with MAE using the ViT-B backhone. RC-MAE outperforms MAE consis- tently on all robustness datasets, indicating that the conditional gradient corrections from the teacher result more robust solution.
CONCLUSION
In this work, have provided an analysis and empirical verification of the nature of contributions from EMA teachers in self-supervised learning. We have found that teachers provide a conditional momentum correction, where the direction and magnitude is conditioned on current and previous feature similarities. Equipped with our analysis, we proposed a simple and effective self-supervised learning approach, RC-MAE. combining masked auto-encoder (MAE) with an EMA teacher. In our experiments, we observed that the teacher enables quicker convergence of RC-MAE which achieves a gain ImageNet classification with increased robustness. Furthermore, RC-MAE out- performs recent self-supervised models on dense prediction tasks. Since our aim is to analyze the dynamics of the EMA teacher, we design a simple approach to validate our findings, which results in better computation and memory efficiency compared to other self-distillation methods. For future work, we look forward to seeing further applications of our analysis. For example, could the condi- tional corrections from the teacher be adequately incorporated into an Adam-like optimizer? If so this would remove the need to have a full model copy as teacher and lead to cost savings and faster convergence in wide variety of settings outside of self-supervised learning.

---

Published as a conference paper at ICLR 2023
8 ACKNOWLEDGEMENT
This work was partly supported by Institute of Information & Communications Technology Plan- ning EVaIuation(IITP) grant funded by the Korea government(MSIT) (No. 2014-3-00123, De- velopment of High Performance Visual BigData Discovery Platform for Large-Scale Realtime Data Analysis, No. 2020-0-00000 Development of Previsional Intelligence based on Long-term Visual Memory Network), (No.2022-00 202220000222 Development of Artificial Intelligence Technology for Self- Improving Competency-Awee Aware Learning Capabilities), (No. RS-2022-00187223, Development of Large Korean Language Model Technology for Efficient Pre-training), and (No.2019-0-00075, Ar- tificial Intelligence Graduate School Program(KAIST).
REPRODUCIBILITY STATEMENT
In this section, we guide where the sections are for reproducibility.
Linear model experiments outlined in Section - Appendix E.1
ImageNet experiments such as pre-training, fine-tuning, linear probing, and robustness evaluation - Appendix E.3
COCO object detection and instance segmentation + Appendix E,4
REFERENCES
Robert Ajemian, Alessandro D' A'uuili,,, Helene Moorman, and Emilio Bizzi. A theory for how sen- sorimotor skills are learned and retained in noisy and nonstationary neural circuits. Proceedings of the National Academy of Sciences, 110(52):E5078 E5078-E88778 2013.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. arXiv preprint arXiv:2204.00 00141, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. ICLR, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pp. 9650- 9660, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In ICML, 2020a.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Nyya Sutskever. Generative pretraining from pixels. In ICML, 2020b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020c.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. empirical study of training self-supervised vision transformers. In ICCV, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 2019.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transform- ers. arXiv preprint arXiv:211 12. 2021.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining. In FCCV, 2022.
10

---

Published as a conference paper at ICLR 2023
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint (.Ki:v/700602677 2017.
Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-super learning. In NeurlPS, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari. Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In ICLR, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021b.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense prediction. CVPR, 2022.
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:21 1...: 1.11199 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitick.. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. Technical report, OpenAI, 2018.
Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal gradients. arXiv preprint arXiv: 1912:191, 2019.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
11

---

Published as a conference paper at ICLR 2023
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averagee consis- tency targets improve supi-super deep learning results. In NeurIPS, 2017.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICML, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In NeurIPS, 2019.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In CVPR, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arxiv: 1788:37088 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022.
12

---

Published as a conference paper at ICLR 2023
A
APPENDIX
We will briefly describe the contents of each section of this appendix below:
Appendix B: Detailed Derivations of Proposition 4.1 the Main Text.
Appendix C: An Intuitive Explanation of Differences Between RC-MAE-D and RC-MAE-
S.
Appendix D Does the Linear Model Extend to Deep Models?
Appendix E: Extra Experimental Details.
Appendix F: Additional Experiments Not Covered in the Main Text.
B LINEAR MODEL DERIVATION
With the weights S and T signifying the student and teacher weights (if no explicit timestep su- perseript (4) is given, then the current timestep is assumed), x and x representing the masked and unmasked inputs, and StopGrad(.) signifyin an operation which stops the propagation of the gra- dients, as the learning rate, and considering the following assumptions,
Assumption B.1. Both the masked input & and have zero mean.
Assumption B.2. For all geometric sums, we assume the sequence N is long enough so that ΣN a² a)
The gradient of loss w.r.t the full linear MAB+Teacher model is,
VsL(S, V,L, v (S,
(7a)
1
V SSX x 1 S
SS (Tx)
(7b)
Sxx
xxx (S x)xxx
(7c)
ssC.
VsC.
Remark B.1. In essence, the MAE objective learns the transformation from square diagonal blocks to off-diagonal blocks of a feature covariance matrix.
Setting the gradient of the reconstruction loss (which is the plain MAE objective) equal to zero, re-arranging and taking the expectation w.r.t the joint data and masking distribution p(x, x) can
see,
0 -5xx xx
(8a)
xx -5x
(8b)
(xxx) [xx Ep(x,)) [SX*]] [Pxx*
(8c)
E xxxxx xx ] S Ss pxx, x)xx dxdx
(8d)
E xx [xx (SE [XX]]
(8c)
Exxx =5Exxx
(8f)
(8g)
This means that the optimal S represents the transformation which takes the square positive semi- definite covariance matrix Σx and transforms it into another off diagonal block of the covariance matrix Ex.x, which is a part of some larger block covariance matrix,
Exx Dxx
Ex,
Dx
13

---

Published as a conference paper at ICLR 2023
B.1 WHAT THE ROLE OF THE TEACHER?
Although the student-teacher paradigm with EMA teacher is common Tarvainen & Valpola (2017); Grill et al. (2020); Assran et al. (2022); Zhou et al. (2022); Caron et al. (2021); Bao et al. (2022), it is still not well understood how the teacher objective directly interacts with and helps the student. In addition to the explanations in Section 4, here we will provide a complete derivation of Proposition 4.1.
B.1.1 PROOF OF PROPOSITION 4.1
The gradient contribution from the teacher and the consistency loss conditionally remove previous gradient directions based on similarities of current and previous features. Using the linear model derivation, the full gradient of an MAE+Teacher given by,
Proof.
sc VsL, VsL.
(9a)
Vs, 2
18x x12 x12+ V |5x StopGrad
(9b)
-5x xx + 55x - TXXX"
(9c)
288x - X² [1 - (1-a)8 8 a(1 a(1-a))(tt) +... +... a' (0(1-)))((--1)
(9d)
258x - Xx
(1 - a +
a (1- xx
(9e)
1=1
258x x
11- a)- Σ
a -o) St a
Σ Jobi
V
(()
xx
(91)
El
= 2888 - -X-
S+ Σ
(1 - a)
1 Σ
(())
x
(9g)
F=b-3
288x - -x - -Sxx
Σ a - a)
Σ
v sw
c(,)
xx*
(9h)
juH-s
Σ a' a) a)
Σ (()
-5xX
xx
xox
(9))
VeC.
VsC.
where Eq. (9f) is a result of the fact that s(t-1) can be stated in relation to by re-arranging 5(t-1) XV st- (t-1) = S, and further decomposition of 5(t-1) in the same way can yield any desired term in the sequence until reaching S(0), For a more concise expression, the indices the sums can be re-arranged like so,
14

---

Published as a conference paper at ICLR 2023
Vs = Vgh, VsLc
(10a)
= -xxX - xx
Σ (1 - a)
Σ
V son VsnL L()
xx
(10b)
jub-i
= -8x - xx A
Σ
Σ a'(1 - a) vGn
Xx
(10c)
j=b-i
1
S8X Xx A
1 (1 a) V L())
(10d)
i
= -8X xx
Σ
V stn (()
Σ a aa- ②8)
xx
(10e)
=0
= 58X Xx A
Σ V S sSn (1-1)
t+1
xx
(10f)
Σ (1-1) V s(o) 0 12x
SXX Xx A
(10g)
-XX Xx
Σ
L''V s() xx
(10h)
4sh,
Velc
where Eq. (10g) uses the fact that the full geometric sum converges to I and therefore a (t+1) s 0. We can then expand the gradient term to include the full gradient at the previous timesteps,
2 a'v stt ((1-))
ssC, + VsC. = - X²
Σ
Xx
(11a)
1=1
-Xx -XX
Σ
SXX
xx1 + (S-T)) (t-i)
Xx
(11b)
Pefi
Vsc.
With & signifying a different i.i.d. sample from the dataset which was present in the previous batch. Therefore, by the last term on the RHS, the gradient of the consistency loss composed of resid- ual effects from the previous gradient steps on different inputs which will have different overall effects based on the result of X& xx" and xx xxx Consider the cosine similarity cos( x x/ | |||||| the possible cases for cos( (x, X) are as follows cos x) (-1,0, 1, (0. 1), (-1,0)}
Case 1: cos(x, x) (-1,1). For this case, a cosine similarity of [1] means the points are collinear, existing the same line, which makes the gradient invariant to the sign of X x. The removal of the previous gradient direction will have the largest impact in this case.
Case 2: cos( &, x) O. For this case, & & = O, which zeros out the whole term. There is no removal of the previous gradient direction for this term in the recursion.
Case 3: |cos( & x)] E 1). In this case, the component which contains the previous gradient will be weighted by the coefficient Ç x=xTs
x x
ssC, + VsL. = SRRT XXT
(12a)
A generic linear model with a generic weight parameter O which has gradient in the form of BC (Ox x, can be interpreted gradient direction and magnitude for every component in
15

---

Published as a conference paper at ICLR 2023
the vector (Ox x) with a further conditional direction and magnitude projected to the components of @ by the final x' term. Both the reconstruction and consistency losses in Eq. (7) follow this form. Therefore, in each case above, the correction provided to S by the consistency loss is weighted conditionally based on the dot product x &, and the final projection of the current feature x 17
Remark B.2. The upper bound on the gradient of the consistency loss can be bounded with the triangle inequality.
We can examine the following term which uses the previous result from Eq. (10),
VgC. Σ a' Aa so-n ((t-))
xx
(13a)
distributing the the outer terms and using the triangle inequality, we can see that,
| Σ a s(
xx
V Xx
(14a)
The most significant (judged by a coefficients) term in the upper bound of the norm of the consis- tency loss when i 1. Therefore the norm of the consistency loss from the teacher likely to decrease if the current input is not similar to the previous input. Likewise, the most significant term in the upper bound increases when the current input has high similarity to the input x One should also note that the upper bound the reconstruction loss can be decomposed via a similar line of reasoning,
G(L,) Six Xx
(15a)
= (S((-1) - AA ste- C(t- ((t-1)) 1)xx))))(x)) Xx
(15b)
=s((0) - Σ v su S )xxx xxx
(15c)
s=1
Focusing only on the non constant terms, we can see that,
Σ (AV (t xx
| > V st-i ((1-1)
(16)
It is easy to see there are no exponentially decreasing coefficients on the terms for the previous time steps, making the upper bound for the reconstruction loss larger than that of the consistency loss. This should lead to the reconstruction loss having larger gradient contribution for novel inputs when compared to the consistency loss, empirically shown in Fig. 2.
C WHY DOES RC-MAE-S OUTPERFORM RC-MAE-D
RC-MAE-S uses the same mask between the teacher and the student, while RC-MAE-D uses different mask between the teacher and the student. Empirically, we observed difference in cosine similarity between the gradients of C, and L. for the different masking strategies the consistency loss depicted in (Fig. 3). In general, different masking strategy X and & forces SX towards the prediction of Tx but there no reason to expect that this objective aligns well with the original reconstruction target in practice.
For example, MAE models are known to give poor reconstructions on masked patches, because the masked patches do not receive any gradient signal during the pre-training phase. Therefore, on patches which are unmasked the student and masked in the teacher, RC-MAE-D provides a consistency objective which interferes with the reconstruction objective by giving noisy target of lower quality, which is the likely cause which leads to worse pre-training performance as shown in Fig. 3 and also lower finetuning performance shown in Table 2.
D Do THE FINDINGS FROM THE LINEAR MODEL EXTEND TO DEEP MODELS?
Although have analyzed simple linear model, more complex deep model with non-linearities can be broken down a similar way. To see this with simple network consisting of sin- gle hidden layer Ba(Ax) with elementwise non-linearity o, and quadratic consistency loss
16

---

Published as a conference paper at ICLR 2023
L. [Bo(Ax) (ox( where the teacher weights are {C, D}, and the student weights {A, B). The teacher at single layer can be expanded according to,
Do(C%)= Do (((1-)AA
a)) ([
(17a)
Do
(1 Σ
a (1 - a))((())) x
(17b)
Do
(1-a)A+ Σ
a1 a)
A+ Σ
ato
(G)
x
(17c)
121
s=t-s
Do
A+ Σ
a (1 a)
Σ
V ((()
x
(17d)
1-
jubi
Do
A+ AV a'A ((-))
x
(17c)
Do
A+ [[. xI...
x
(17f)
where the fourth step is result of rearranging the sums following the same steps Eq. (10). The term xT] results from fact that we know the previous gradient ends with the term &T, where & is another i.i.d. sample from a previous batch, due to the chain rule applied as follows,
OL.
OC
(Ax do (Ax) 0Ax
(18a)
04 (Ax) 0o OAX 0A
[BB( Do (Cx)) B diag(o' (iaagoo(AA))) (Ax))]* x²
(18b)
(BO(AR)) Do
([ [[++
2 a X[. (c>([...TT](b-.)))) x
B diag( a (Ax))
(Axg)
xT
(18c)
This shows that although intermediate representations are more complicated. they contain similar terms, and the dot product term is present at every intermediate representation. Having this term in each semantic feature space is useful, as the dot product in the high dimensional pixel space may carry less meaning. Empirically, in Fig. we observed the same overall effects in the deep ViT based RC-MAE (bottom row) as we did in the simple linear model (top row).
E EXPERIMENT DETAILS
E. LINEAR MODEL
To perform the linear model experiment outlined in Section 4, we first generated dataset consisting of 10 random Gaussian clusters. Each cluster has randomly sampled mean vector μ4; U-3,, 3), and each μ4; 3R With each mean vector, we also sample corresponding covariance matrix Ex Wishart(V.) with 54 degrees of freedom (1.5 x dimensionality). V; is diagonal matrix with diagonal being sampled from U..25,.335 .35). We then sample 200 datapoints from each of the 10 Gaussians, resulting in dataset with 1000 total instances. Further experimental settings can be found in Table 8. An example of the dataset two dimensions with four clusters can be seen in Fig. 5.
The student and teacher S, T' 32x32 consist of a single weight matrix with a bias parameter. We randomly mask the components of the input vector according to the masking ratio. The goal of the linear model is then to reconstruct the full input given the masked input with reconstruction C, and consistency L. loss according to the linear objectives in Eq. (7).
17

---

Published as a conference paper at ICLR 2023
7.5
label
8
255
255
-5.0
-7.
-5.0
5.0
Figure 5: Example of a dataset sampled with the procedure outlined in Appendix E.1. The only difference from the dataset in the experiment is the dimensionality and number of clusters which was minimized for visualization purposes.
Setting
Value
Runs
5
Iterations
500
Batch Size
32
Optimizer
SGiD+Momentum(0.9)
Learning Rate
0,001
Teacher Momentum (a)
0.9
Masking Ratio
0.5
Table 8: Linear Model Experiment Settings
E.2 IMPLEMENTATION DETAILS
We implement RC-MAE based on our baseline MAE (He al., 2022) by adding EMA Teacher which has the same architecture as student (i.e., MAE). To be specific, the EMA teacher consists of an encoder (e.g., ViT-B) and decoder (Transformer blocks), following and dimension size MAF, RC-MAE with VIT-S has 12 blocks of Transformer blocks with dimension size of 384 for an encoder and 4 blocks of Transformer blocks with a dimension size of 256 for decoder. RC-MAE with ViT-B has 12 blocks of Transformer blocks with dimension size of 768 for an encoder and 8 blocks of Transformer blocks with a dimension size of 512 for a decoder. RC-MAE with VIT-L has 24 blocks of Transformer blocks with dimension size of 1024 for an encoder and 8 blocks of Transformer blocks with dimension size of 512 for a decoder. For pre-training,
same number of blocks
optimize only the student network including encoder and decoder by the gradients while updating the teacher network (encoder & decoder) by EMA of the student weights as shown in Fig. 1(b). For downstream tasks, we only use the pre-trained encoder part from the teacher network as other self-distillation methods ((Caron et al., 2021; Assran et al., 2022; Zhou al., 2022; Dong et al., 2022)). We also test which one better between student and teacher for transfer learning in Appendix F.1.
E.3 IMAGENET EXPERIMENTS
We follow the implementation details of the official MAE (He et al., 2022) code for all pre-training, fine-tuning and linear probing. While He al. (2022) used 128 TPU-v3 cores, have tried to produce the baseline MAE and train our RC-MAE on the same local GPU environment, which has 8 NVIDIA V100 GPUs (32GB) for more accessibility in the community. Although the authors pro- vide the guideline using nodes with 8-GPUs (64-GPUs) in their official code written in Pytorch, we train RC-MAE and MAE on the same 8-GPUs environment. To do this, we use accum. iter
httpp:://gi thub. com wtt..tccooacceoceoeeeeeeeeeeeeee http://cccoooreeerrc/cccecoaeaee
18

---

Published as a conference paper at ICLR 2023
Table Pre-training setting.
Config
Value
optimizer
AdamW (Loshchilov & Hutter, 2019)
optimizer momentum
31, B2 = 0.9, 0.95 (Chen et al., 2020a)
weight decay
0.05
base learning rate
1.5e-4
batch size
4096
learning rate schedule
cosine decay (Loshchilov & Hutter, 2016)
warmup epochs
40
augmentation
RandomResizedCrop
mask ratio
75%
Table 10: End-to-end fine-tuning setting.
Config
Value
optimizer
AdamW
Table 11: Linear probing setting,
optimizer momentum
31.82 0.9, 0.999
Config
Value
weight decay
0.05
layer-wise decay
0.75 (S), 0.65 (B,L)
optimizer
LARS
base learning rate
5e-4
base learning rate
0.1
drop path
0.1 (S,B), 0.2 (L)
weight decay
0
batch size
1024
batch size
16,384
learning rate schedule
cosine decay
learning rate schedule
cosine decay
warmup epochs
5
warmup epochs
10
training epochs
300 (S), 100 (B.L)
training epochs
90
augmentation
RandAug(9, 0.5)
augmentation
RandomResizedCrop
label smoothing
0.1
mixup cutmix
0.8
1.0
which the authors provide for accumulating gradients. However, when using accum iter, a re- producing issue? occurs in that the re-produced result (MAE-ViT-B) achieves slightly lower per- formance e.g., for ViT-B, 84.4% vs. 84.6% in end-to-end fine-tuning as shown in Table 4.
Pre-training. Thanks to accum_i teer we can set the same effective batch size of 4096 as MAE, e.g., 256 (batch_size-pe gpu) & (GPUs) 2 (accumiite)) for both VTT-B and VIT-S. For ViT-S, we set 512 (batch size_pee pereeperrgee gpu) x 8 (GPUs) x I (accum_iter) we follow the linear learning rate scaling rule (Goyal al., 2017): Ir Ir x batch_size/26.. The detail settings shown in Table
End-to-End fine-tuning. Since MAE (He et al., 2022) does not have VIT-S result, train MAE and RC-MAE using ViT-S following the training protocol in BETT (Bao et al., 2022). For ViT- S. we use a layer-wise Ir decay of 0.75, stochastic drop path of 0.1, and fine-tuning epoch of 300. We can expect that a more suitable hyper-parameter search boosts performance. For ViT- L. can set the effective batch size 1024 by using accum_iter, egg., 64 (batch size per gpu) x 8 (GPUs) x (accum.iter) The detail settings are shown in Table 10.
Linear probing. We LARS (You et al., 2017) optimizer. The detail settings are shown in Table 11.
Robustness Evaluation on ImageNet-variante We use the same model weights (RC-MAE w/ViT- for 1600epoch) fine-tuned on the original ImageNet- IK as shown in Table 4 and only test without any specialized fine-tuning on the different validation sets, such as ImageNet-C (Hendrycks & Di-
19

---

Published as a conference paper at ICLR 2023
Table 12: Comparison between Student vs. Teacher networks on ImageNet- 1K evaluation. FT and LN denote fine-tuning and linear probing top-1 accuracies, respectively.
FT
LN
Checkpoint
top-1
top-5
top-1
top-5
Student
83.48
96.59
68.38
87.88
Teacher
83.58
96.64
68.41
87.94
Table Total GPU memory usage and throughput using ViT-L backbone. OOM represents that it not able to run on 8 V100-32GB GPUs due to "Out of memory".
GPU Memory
Usage
(GB)
Throughput (imgs/sec.)
Batch size
128
256
512
1024
128
256
512
1024
MAE (He et al., 2022)
84
103
140
213
533
853
1191
1442
MSN (Assran et al., 2022)
183
COM
OOM
00M
78
OOM
COM
OOM
iBOT (Zhou et al., 2022)
227
0OM
0OM
00M
123
00M
O0M
OOM
BootMAE (Dong et al., 2022)
98
116
154
225
376
557
701
813
RC-MAE
95
114
152
226
441
674
914
1101
etterich, 2019),-A (Hendrycks et al., 2021b),-R (Hendrycks et al., 2021a), and -Sketch (Wang et al., 2019).
E.4 OBJECT DETECTION AND SEGMENTATION ON COCO
Following the training protocol (Li et al., 2021) and implementation details MAE (He et al., 2022), we fine-tune Mask R-CNN (He et al., 2017) with a ViT-Base backbone COCO dataset. To adapt the vanilla ViT with the isotropic structure (i.e., single-scale) for FPN (Lin et al., 2017) in the backbone of Mask R-CNN, we equally divide the ViT into four blocks and apply convolutions to downsample upsample the intermediate features for building the multi-scale feature pyramid. For fair com- parison with MAE, we train both MAE and the proposed RC-MAE for 100 epochs with the same training protocol
the same 8 GPU environment: large-scale jitter (LSJ), AdamW (Loshchilov & Hutter, 2019) with half-periodie cosine learning rate decay, linear warmup, and drop path regulariza- tion (Huang et al., 2016) (e.g., 0. for ViT-Base). We use batch size of 16 per GPU), learning rate of 4e-5, and weight decay of 0.1. We implement models based on the reproduced orch code
F ADDITIONAL EXPERIMENTS
STUDENT VS. TEACHER NETWORKS.
To determine which is better between student and teacher networks for transfer learning, conduct fine-tuning and linear probing on ImageNet- by using each pre-trained weight. Table 12 shows the comparison between student and teacher networks. We find that the results of the teacher network are slightly higher than those of the student, which similar to DINO (Caron et al., 2021). Thus, unspecified, we use teacher network weight for downstream tasks.
F.2 RESOURCE COMPARISON: MEMORY COMPUTATION COST
In terms of memory consumption and computation cost, conduct resource comparison with the bascline method MAE 6 (He et al., 2022) and the self-distillation-bar MIM methods, e.g.,
'ttps:: / /ttgt//tttttt cwm/ http/hustl///// //MMeet
At the time of submission of this paper, the official code of benchmarking detection (Li al., 2021) was not released.
https: / / thub. com/ www.bookreeeerchhcoooookkkkkkkce
20

---

Published as a conference paper at ICLR 2023
(Assran et al., 2022), iBOT (Zhou al., 2022), and BootMAE" (Dong al., 2022). We compare GPU memory usage and throughput while pre-training with ViT-L/16 on V100-32GB GPUs shown in Table 13. Memory usage is directly measured by using nvidia-smi command. Throughput is an average speed (images/second) for an epoch. We follow the original configura- tion for each method except batch size for fair comparison. Batch sizes from 128 to 1,024 were employed to observe the scalability of the methods. We note that iBOT (Zhou et al., 2022) and MSN (Assran al., 2022) could not run with batch sizes from 256, while other methods could deal even with a batch size of 1,024. iBOT feeds both masked and unmasked patches into the student encoder by following the masked image modeling employed in BeiT (Bao al., 2022). Therefore, it requires 4x memory usage of MAE-based methods for the student encoder. In addition, for fur- ther performance gain, iBOT exploits multiple local crops. Although we removed the local crops, memory usage was still 164GB which is much higher than MSE-based methods. MSN takes lower masking ratio than other methods (0.5 vs. 0.75), resulting in 2x memory consumption for the student encoder.
MSN
21


Reviews:
[{"review_id": "uKpnbYpJOJ", "reviewer": "Reviewer_rLZG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission proposed a new simple yet effective method, named Reconstruction-Consistent Masked Auto-Encoder (RC-MAE), by equipping the latest ViT-based Masked image modeling (MIM) with mean teachers.\nThe authors derive some approximations (using a simple linear model) on MIM pretext setting to analyze the role of mean teachers. Some synthetic experiments are provided to support this mathematical analysis. Finally, experiments on classification (ImageNet-1k), and object detection (COCO) tasks show consistent improvement over MAE and with faster training time (RC-MAE, 800 epochs, 166.6 hours ~= MAE, 1600 epochs, 256.5 hours).", "summary_of_the_review": "Although the proposed method is effective, it is very straightforward and provides very much ``expected'' results. The analysis and conclusion from an extremely simplified linear model may not be the same as the actual ViT-based model.", "strength_and_weaknesses": "Strengths\n\n- simple and effective\n- proved mean teachers technique is compatible with ViT-based MAE empirically\n- slightly stronger representation and much faster self-supervised training but requires not very much extra memory/computation cost compared to MAE, making RC-MAE more scalable compared to MAE and more practical than other self-distillation MIM methods (MSN, iBOT)\n\nWeaknesses\n\n- The final solution is exactly the combination of two well-known techniques, which makes sense but is too straightforward. Also, introducing some kind of memory mechanism to improve MAE has been demonstrated effective in BootMAE, which proposed a momentum encoder using EMA update. I think in-depth analysis and a thorough comparison with BootMAE are required.\n- The accompanied mathematical analysis and the synthetic experiments are good. But it is an extremely simplification of the actual model (ViT), which makes me wonder if the conclusions drawn from the equations are applicable to RC-MAE.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h6H5vIYmahw", "reviewer": "Reviewer_jGgU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple SSL method, the Reconstruction-COnsistent Masked Auto-Encoder (RC-MAE) by adding an EMA teacher to MAE. Experiments show that RC-MAE converges faster and requires less memory usage than state-of-the-art self-distillation methods during pre-training. Compared with MAE, RC-MAE consistently outperforms on various tasks, including image classification on ImageNet-1K, object detection and instance segmentation on COCO.", "summary_of_the_review": "See Strength And Weaknesses", "strength_and_weaknesses": "Strength:\n+ This paper gives a analysis contributation of EMA teacher in self-supervised learning, and finds that the gradient provided by the teachers conditionally ajusts current gradient direction and magnitude conditioned on the similarity of features.\n+ EC-MAE achieves similarly performance for MAE with faster convergence speed, aversatial robustness.\n+ EC-MAE saves both memory and computation compared with self-distillation-based MIM methods.\n\n\nWeaknesses:\n+ The performance on ImagNet-1K is not impressive compared with recent concurrent works.\n+ Lack of downstream tasks like semantic segmentation tasks\n+ Why proposed method achieve better robustness, could you give some explanation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "bTRImqVXw8", "reviewer": "Reviewer_JA1p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper firstly analyzes the properties of mean teacher in a masked data modeling task with a simplified linear model. The mean teacher contributes to the training by gradient correction, and it is similar to the memory queue. Several empirical findings about the mean teacher are introduced based on the experimental observation.    \nNext, the paper proposes RC-MAE (Reconstruction-Consistent Masked Auto-Encoder), which add a EMA teacher for MAE. The proposed model achieves better performance compared to the baseline on classification, detection and segmentation benchmarks. Besides, RC-MAE costs less computation resource and obtains more stability.  ", "summary_of_the_review": "This paper explains the role of mean teacher in a MIM model and proposes RC-MAE that achieves better performance than MAE. With good quality and fair technical contributions, this paper is marginally above the acceptance threshold.", "strength_and_weaknesses": "**Strength**:    \n[1] The theoretical analysis of EMA teacher dynamics is solid.      \n[2] The reviewer appreciate that the RC-MAE objective does not introduce a new hyper parameter. The reconstruction term and consistency term simply add together without a balance.   \n\n\n**Weakness**:    \n[1] Although RC-MAE outperforms MAE on classification, detection and segmentation benchmarks, the improvement is not significant enough.     \n[2] With all the mathematical derivations, perhaps more insights and conclusions could be obtained.    ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "a2esSHneyu", "reviewer": "Reviewer_5RHA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper extends the Masked Auto-Encoders (MAE) with Vision Transformers by incorporating a Teacher-Student setup (asymmetric siamese model with exponential moving average update for one of the streams).  The proposal uses a reconstruction loss over the masked patches and adds a consistency loss over the reconstruction of both streams.  The experiments show an improvement over MAE in ImageNet-1K classification, and COCO detection and segmentation downstream tasks.\n\nThe paper also frames the Teacher-Student setup as a memory of past features that are used by the student to follow the gradient of the previous samples' features.  The analysis was done over a linear simplification of the Teacher-Student model, and an experiment shows the gradient responses on different cases of similarity and dissimilarity of the samples.", "summary_of_the_review": "The paper proposes a simple yet effective loss based on the reconstruction of patches using ViT as an encoder, and a two-stream asymmetric model with exponential moving average as update for one of the streams (teacher-student) that should agree in their reconstructions (consistency loss).  The claim of the authors of using the understanding of the teacher as a bank of gradients to propose the above mentioned setup is beyond me.\n\nThe setup is complicated, and tries to tie to topics and package them together.  Perhaps a simpler write-up with these two pieces presented individually could be easier to follow and allow them to shine.\n\nThe details about the implementation (encoders, decoders, and setup) is not presented.  The paper references the same implementation as He et al.'s setup, yet the differences for this proposal are not explained.", "strength_and_weaknesses": "Strengths:\n- The paper uses a simple extension to the mask autoencoders with ViT (MAE) by adding a consistency loss over an asymmetric siamese setup (Teacher-Student), following the success of recent asymmetric self-supervised models that rely on exponential moving average for the update of the other stream.\n- The results show improvement over the existing MAE, and two approaches for masking are explored and evaluated.\n- The role of the teacher as a memory bank of past gradients is explored.  A simple experiment is performed on a linear version of the Teacher-Student setup that shows the changes on the gradient w.r.t. the similarity of the features.  The same experiment is repeated with less explainability with more complex models.\n\nWeaknesses:\n- The claims of using the \"role of the teacher\" as basis to propose the extension of the model are not clear.  \n\n  In particular, the proposed loss of reconstruction and consistency is straightforward on the multi-stream setup.  In its current form, the paper reads as adding more complexity by trying to explain the bank of gradients as a basis for the extra consistency term in (6).  While the result is interesting, it doesn't seem to resonate with the proposal of updating the MAE.\n\n  Consider a simpler and cleaner approach of introducing the consistency and reconstruction loss, instead of adding unnecessary complexity to justify the claims.\n\n- It is not clear from the explanation in the Appendix how well the projection ($S\\hat{x}$) and the difference of the agreement between the teacher and student ($(S-T)\\hat{x}$) relate to the original data ($x$) also influence the final gradient.  \n\n  Wouldn't errors in the projection or difference between the streams also influence the gradients?  And that will also make a difference on the bank and what is stored in it?\n\n  Should the generic linear model also contain a term that shifts the vector $\\Theta \\tilde{x} -x$ depending on the difference of the streams as well? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster