Exploring Target Representations for Masked Autoencoders
Authors: xingbin liu
Jinghao Zhou
Tao Kong
Xianming Lin
Rongrong Ji
Venue: ICLR 2024 poster (2024)
Keywords: self-supervised learninf
masked image modeling
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Masked autoencoders have become popular training paradigms for self-supervised visual representation learning. These models randomly mask a portion of the input and reconstruct the masked portion according to assigned target representations. In this paper, we show that a careful choice of the target representation is unnecessary for learning good visual representation since different targets tend to derive similarly behaved models. Driven by this observation, we propose a multi-stage masked distillation pipeline and use a randomly initialized model as the teacher, enabling us to effectively train high-capacity models without any effort to carefully design the target representation. 
On various downstream tasks, the proposed method to perform masked knowledge distillation with bootstrapped teachers (dbot) outperforms previous self-supervised methods by nontrivial margins. We hope our findings, as well as the proposed method, could motivate people to rethink the roles of target representations in pre-training masked autoencoders.

Published as a conference paper at ICLR 2024
EXPLORING TARGET MASKED AUTOENCODERS
REPRESENTATIONS
FOR
Xingbin Liu
Jinghao Zhou
Tao Kong"*
Xianming Lin
Rongrong Ji
Xiamen University 2ByteDance
ABSTRACT
Masked autoencoders have become popular training paradigms for self-super visual representation learning. These models randomly mask a portion of the in- put and reconstruct the masked portion according to assigned target representa- tions. In this paper, we show that a careful choice of the target representation is unnecessary for learning good visual representation. Driven by this observation, we propose a multi-stage masked distillation pipeline and use a randomly ini- tialized model as the teacher, enabling to effectively train high-capacity mod- els without any effort to carefully design the target representation. On various downstream tasks of classification, transfer learning, object detection, and seman- tic segmentation, the proposed method to perform masked knowledge distillation with beotstrapped teachers (dBOT) outperforms previous self-supervised meth- ods by nontrivial margins. We hope our findings, as well as the proposed method, could motivate people rethink the roles of target representations pre-training masked autoencoders. The code and pre-trained models are publicly available at https:/ / thub. com http/luxiiingiii wwgiiiigiiiiio dbot.
INTRODUCTION
Masked Image Modeling (MIM) (He et al., 2022; Wei al., 2022a; Baevski al., 2022; Zhou al., 2021) has recently become an active research topic in the field of visual representation learning and establishes strong performance for vision recognition tasks, egg., image classification, object detection, and semantic segmentation, which also surpasses traditional supervised learning (Touvron al., 2021) mechanism. To be specific, MIM randomly masks a portion of the input and then reconstructs the masked portion according to the transformed target, formulated as
min E (M 2 (1 aw
(1) M)),
(1)
where "O" means element-wise product; M the patch mask; "x M" represents "unmasked patches" and vice versa; (-) is the learnable network to be pre-trained; T is the transformation function generating the reconstructed target. T can either image feature transformation method; M(..-) is the similarity measurement, e.g., 12-distance (He et al., 2022). masked image passed through the network fo(x M) to reconstruct the visual representation of the intact image with transformation T(x C (1 M)).
parameterized network or a traditional
crucial problem of MIM is how to choose the reconstructed target, i.e., T(-) in Eq. (1). Previous methods use disparate teacher networks to generate the reconstruction target. BEiT (Bao al., 2022) employs a pre-trained DALL-E (Ramesh et al., 2021) as the teacher network. In MaskFeat (Wei et al., 2022a), authors et al., 2021) features to perform MIM; MVP (Wei et al., 2022b) employs a multi-moddllty model, CLIP (Radford et al., 2021), which is pre-trained by rich image-text pairs. MAE (He et al., 2022) uses image pixels as the target, which functions likewise to randomly initialized teacher network, demonstrated in Appendix B. iBOT (Zhou et al., 2021) and data2vec (Baevski et al., 2022) the exponential moving average (EMA) strategy to update teacher's parameters ø. Though different methods differ in their architectural designs and optimization, the choice of the teacher network lies crucial for each method and calls for systematic study. In this work, we paraphrase a term Masked
HOG (Dalal & Triggs, 2005), MoCo (He et al., 2020) and DINO (Caron
"Equal contribution. 'Corresponding author(linxme @ @xmu.cdu.cn..

---

Published as a conference paper at ICLR 2024
Knowledge Distillation (MKD) to focus our discussion on special case of MIM where the target is generated by a parameterized network (teacher network), i.e., T-)) = (-). In this setting, T is the teacher network, and f is the student network.
The purpose of our work is to investigate whether a careful design of the teacher network for MKD matters. Such exploration is nontrivial given that different teacher networks contain different knowl- edge we endued into the teacher network, which may induce diverse behaviors for the student net- works. And the painstaking selection of the target representations in the field of MIM. To this end, we compare student networks distilled by four teacher networks with different computation pipelines, i.e., DINO (Caron et al., 2021) for contrastive learning. MAE (He al., 2022) for masked autoencoding, DeiT (Touvron et al., 2021) for supervised learning, and DALL-E (Ramesh et al., 2021) for autoregressive generation. Four teachers are all pre-trained on ImageNet-IK for a fair comparison. To our surprise, although the behaviors of the teacher networks are very different, the distilled student networks share similar characters after several stages of MKD: (i) the performance variance between student networks distilled from different teachers rapidly decreases. (ii) the model weights and output features across layers within the networks share similar properties.
Such observations indicate that the design of target representation is not essential when pre-trained with multi-stage, i.e., teacher networks do not matter with multi-stage masked knowledge distilla- tion. Exceptionally, knowledge distillation, and find that it performs as well as those initialized by pre-trained models with the exact same settings! Using random model as teachers not only avoids an extra pre-training stage, but also alleviates the painstaking selection of the target representations.
use randomly initialized model as teacher to perform multi-stage masked
Based on the above studies and observations, naturally propose to perform masked knowledge distillation with bootstrapped teachers, short as dBOT é. Specifically, masked knowledge distil- lation is performed repeatedly multiple stages. At the end of each stage, we assign the student's weight to the teacher and re-initialize the student's weight to continue masked knowledge distil- lation. With simple yet effective design that enables pre-training starting from randomly initial- ized teachers, dBOT achieves 84.5%, 86.6%, and 88.0% top-1 fine-tuning accuracy on ImtgeNet- 1K (Deng et al., 2009) with VIT-B/16, ViT-L/16, and ViT-H/14, respectively, significantly surpassing previous states of the art, MAE. Beyond that, dBOT achieves 52.7 and 56.0 Apbox for object de- tection on COCO (Lin et al., 2014), as well as 49.5 and 54.5 mloU for semantic segmentation on ADE20K (Zhou et al., 2017), with ViT-B/16 and ViT-L/16 respectively. We also explore MKD with teachers of larger sizes, further boosting model performances on various visual tasks.
RELATED WORK
2. SELF-SUPERVISED VISUAL LEARNING
Self-supervised learning is an active research topic recently. Early practices revolve around con- trastive learning (He al., 2020; Chen al., 2020; Grill et al., 2020; Caron et al., 2020; 2021) where the model output features of images transformed by different data augmentations are pulled together. With the development of Masked Language Modeling (MLM) in language pre-training (Devlin et al., 2019), researchers also introduce the training strategy of masked reconstruction to visual pre- training. BEiT (Bao al., 2022) uses the DALL-E (Ramesh al., 2021) to encode an image patch as the target for model reconstruction. iBOT (Zhou et al., 2021) uses an online teacher shifting the target from offline to online to make the target semantic meaningful. obtained from offline or online model as reconstruct target, MAE (He et al., 2022), SimMIM (Xie et al., 2022), and MaskFeat (Wei et al., 2022a) achieve good performance in masked-image recon- struction using lowelevel pixels or HOG (Dalal & Triggs, 2005) features. Among them, MAE uses an asymmetric encoder-decoder structure greatly increasing the training efficiency. data2vec (Baevski three modalities (vision, speech, and language)
addition using the token
et al., 2022) demonstrates good generalizations by reconstructing multiple neural network layer representations.
2.2 KNOWLEDGE DISTILLATION
Knowledge distillation (KD) is widely employed in model knowledge compression (Hinton al., 2015), which improves the performance of the smaller student model by distilling the knowledge learned from well-trained large teacher network. Further study e.g.relational KD (Park al.,
2

---

Published as a conference paper at ICLR 2024
computation
initialized teacher
classification 1st
object detetion
semantic segmentation
pipeline
0th
3rd 2nd
0th
1st 2nd
3rd
4th
0th 1st
2nd 3rd 4th
Supervised
DeiT
83.6 84.3 84.3
49.1 50.5
52.5
52,4
46.4 49.2 50.4 49.9
Contrastive
DINO
83.2 84.2 84.5 84.4
50.1
52.5 52.9
52.7
46.8 49.7 50.4 49.4
Autoregressive
DALL-E
81.1 83.5 84.4 84.3
31.9
51.0 52.7 52.5
31.9 49.6 49.3
Autoencoding
MAE
83.6 84.3 84.4 84.3 50.6
52.9 52.7 52.5
48.1 49.6 50.4 49.8
random
77.3 83.4 84.5
84.3
29.2
49.6
52.4
52.7
52.4
25.7
47.0 49.1 49.5 49.5
performance variance
2.24 0.37 0.07 0.04
9.54
1.23 0.17
0.12
9.19 1.15 0.54 0.23
Table 1: The top-1 classification accuracy ImageNet-IK, object detection AP-box COCO with Cascade Mask R-CNN, and semantic segmentation mloU on ADE20K with UperNet dBOT using different models the initialized teacher network. Note that all models are pre-trained on ImageNet-IK: including DALL-E,, for fair comparison. We perform distillation in each stage for 800 epochs. In the 1st stage, distill from initialized teacher to obtain student. In the subsequent (i.e., 2nd 3rd, etc.) stages, the obtained students are leveraged as bootstrapped teacher to distill a new student.
2019), contrastive KD (Tian et al., 2019), and latent feature KD (Romero et al., 2015) is conducted to improve the performance of vanilla KD. Beyond its prominence the field of supervised learning, KD recently cuts figure in supersupereiee learning. Concurrent work manages to adopt conven- tional feature distillation (Wei et al., 2022c) to match contrastive models with MIM-trained ones. Nevertheless, it shows negligible gains MIM-trained models such as MAE. BETT (Bao et al., 2022), MaskFeat (Wei et al., 2022a) and MVP (Wei et al., 2022b) could be seen as distilling knowl- edge from dVAE (Ramesh et al., 2021). HOG features (Dalal & Triggs, 2005) and ianguage-inducee model CLIP (Radford et al., 2021) within the discourse of MKD, respectively. Until now, there exists work conferring system-level study on the importance of how to choose adequate target representation or teacher networks to guide the learning of MKD.
3 DOES h-)-) MATTER IN MKD?
Given the general form masked knowledge distillation as shown in Eq. (1), in this section, we aim to investigate whether the careful design of the target, i.e., teacher network ha()), matters. Specifically, we want to answer three questions as follows:
Whether models distilled from different h-)-) differ in terms of their transfer perfor- mances?
Whether distilled models differ in terms of their weights and outputs?
If (+) does not matter, what matters more to close the gap between students distilled from different h.)?
To answer these questions, we employ the standard masked autoencoder framework (He et al., 2022) to give system-level study, introduced next.
Common setup. The architectural settings strictly follow (He et al., 2022). For the teacher net- work, the vanilla ViT (Dosovitskiy et al., 2021) with intact input. For the student network with masked input, we use the asymmetric encoder-decoder structure. The student's output is fur- ther projected to dimension the same as that of teacher's embedding. During pre-training, use Smooth L1 loss (Girshick, 2015) for the optimization of the student network, and the teacher network is kept fixed. Detailed settings are delayed to Appendix A.1. We pre-train models on ImageNet- K (Deng et al., 2009) and conduct evaluation under classification ImageNet, object detection on COCO (I.in al., 201 4), and semantic segmentation on ADE20K (Zhou al., 2017).
3.1 PRELIMINARY STUDY
We first investigate the effect of using networks initialized differently as teachers for masked knowl- edge distillation. Four canonical methods as pre-trained teachers are substantiated, each from a category distinguished based their computation pipelines, iee. DeiTT (Touvron et al., 2021) for
3

---

Published as a conference paper at ICLR 2024
supervised learning, DINO (Caron et al., 2021) for contrastive learning, DALL-E (Ramesh et al., 2021) for autoregressive generation, and MAE (He et al., 2022) for autoencoding. The results of initialized teacher at the 0th stage and of its distilled student at the 1st stage are shown in Table 1.
Different h.).) lead to similarly performed students. After the first stage of masked knowledge distillation, the student consistently outperforms teacher as shown in Table 1, yielding 1.8%, 1.0%, 2.4%, and 0.7% performance gains for four different (.)()) respectively. demonstrating the effective- ness of masked knowledge distillation for visual representation learning. Although the performance order of different (-) is reserved after the first stage of distillation, the students distilled from dif- ferent (.).) have closer downstream performances compared the original h-).). The performance variance drops from 2.24 to 0.37 after the first stage of distillation. The conclusion holds true for experiments on object detection and semantic segmentation.
3.2 DISTILLATION WITH MULTIPLE STAGES
Given the observations that better teacher generally induces better outperforming student, motivated to use the trained student as teacher to train new student repeatedly and study whether similar trend endures. If so, would like seek at what stage the performances saturate for differ- ent downstream tasks, as well as the discrepancy among the results incurred by different initialized teachers.
h.).) does not matter with multi-stage distillation. The performance gain is valid but decreases with multi-stage and eventually vanishes. Take MAE being the initialized teacher as an example, students outperform teachers by +0.7%, +0.1%, -0.1% for classification, +2.3, -0.2, -0.2 points for object detection, and +1.5, +0.8, -0.6 points, for semantic segmentation, from the 0th to the 3rd stage. Other teachers and downstream tasks share the same conclusion. Moreover, the performance gaps of students learned from different teachers decrease, especially after state as shown by the performance variance at different stages in the last row of Table 1. Take classification tasks for instance, the variance decreases along with the training stage, i.e., 2.24, 0.37, 0.07, 0.04, which reveals that the choice of h-(-) exerts little influence on the downstream performance. See Table for results of more downstream tasks. To demonstrate models' differences in terms of weights and outputs, we conduct property analysis in Sec. 6. Similar properties are found, which verify our conclusion.
A random (.).) works surprisingly well. Since the choice of ha(-) does matter, an intuitive experiment is to see what will happen when we employ random teacher, in which the parameters are randomly initialized at the 0th stage. To our surprise, using a random teacher achieves per- formances comparably with other pre-trained teachers. Compared to a randomly initialized model, distilled students with multiple stages achieve 6.1%. 20.4, and 21.3 performance gain on classifi- cation, object detection and semantic segmentation respectively. Empirically, object detection and semantic segmentation require one more stage to saturate compared to classification. The saturated results are on par with those induced by pre-trained teachers, which enables us to train state-of-the- art model more efficiently, without the need of an extra pre-training stage for the initialized teacher (e.g., contrastive learning DINO).
4 MKD WITH BOOTSTRAPPED TEACHERS
The study in Sec. 3 motivates us to propose a multi-stage distillation pipeline for pre-training. The entire pre-training undergoes multiple stages split by breakpoints. For each stage, we fix teacher network to obtain a stable visual representation, guiding the learning of student network. The pre- trained student model is then used as stronger teacher and distills knowledge to new subse- quent student, providing richer visual representations. We re-initialize the student network at each breakpoint. The above process repeats itself the teachers keep bootstrapped from the students, until performance saturation downstream tasks is observed. Hence, our strategy is to perform distilla- tion with bootstrapped teachers. We illustrate our framework in Fig. Ic and the conceptual relations with the other two paradigms in Fig. 1. By noting m as the momentum which indicates how fast the teacher's ' parameters 0, is updated from student's 's parameters 0., i.e., O, = m 0, + (1 - m) 0., present the following discussions.
4

---

Published as a conference paper at ICLR 2024
14:2
203 Mm
7
toxber
steket
XN
(a) BEİT, MaskFeat
(b) iBOT, data2vec
(c) dBOT
Conceptual comparison of three masked image modeling paradigms. The difference between the three paradigms is how the parameters of the teacher network are updated. (a): The parameters of the teacher network are frozen during the whole training process, constructing an offline teacher. (b): Exponential moving average is applied to correlate the parameters the student and teacher networks, constructing online teacher. (c): dBOT uses a multi-stage distillation pipeline, i.e., the parameters of the teacher network are frozen except at breakpoints, where we assign student parameters the teacher and re-initialize the student network.
Figure/ 1:
Relations with previous methods. One group of works leverages pre-trained teacher as in Fig. la, ie.. BEiT (Bao et al., 2022). The teacher requires an extra stage of pre-training and is kept fixed with be more semantic meaningful, prompting student's learning. Nonetheless, the pre-training of these teachers entails completely different computation pipeline (Wei et al., 2022a) and often addi- tional data (Wei et al., 2022b), complicating its practical use. Another group as in Fig. 1b works with random teacher in dispense with pre-trained ones. Starting from randomness, the teachers in iBOT (Zhou et al., 2021) and data2vec (Baevski et al., 2022), however, are bootstrapped from the student typically with m (0, 1). e.g., 0.9998 as (Baevski et al., 2022). Although bootstrap duces improving quality of the teacher's representation, the pipeline is plagued by its optimization instability and sensitivity towards hyper-parameten
1. Ideally, pre-trained teachers bear additional knowledge which is prone
We note that MAE uses identity mapping of pixels the target, which is observed to function similarly as shown in Appendix B.I. Despite its simplicity, such practice cludes synergy between the teacher and the student. Comparatively, dBOT is with = for every breakpoint and m = otherwise.
fixed random teacher with I,
EXPERIMENTS
5.1 PRE-TRAINING
Architecture. We use different capacity Vision Transformers (Dosovitskiy et al., 2021), i.e. ViT- B/16, VIT-L/16, and ViT-H/14 for dBOT. The input image of size 224x224 is first divided by linear projection head into non-overlap patch tokens total of 196 for ViT-B and ViT-L, and 256 for ViT-H. We exactly follow the common setup demonstrated in Sec. 3, egg., student with asymmetric encoder-decoder architecture, a teacher with intact input, etc.
Optimization. The learning rate is first linearly increased to the initial learning rate for the first 40 epochs and then cosine annealed O. The initial learning rate is as 1.5e-4 batch_size / 256, with batch size being 4096 for all models. We use the Adam optimizer (Loshchilov & Hutter, 2019) and Smooth L1 loss (Girshick, 2015) to optimize the parameters of student network. Stochastic drop rate are applied, 0.2 for ViT-B, 0.2 for ViT-L, and 0.3 for ViT-H. We use only center-crop and flipping for data augmentation. As shown in Table 1, the performance of different downstream tasks saturates at different stages. By default, we pre-train all models for classification with 2 stages, for object detection and semantic segmentation with stages.
5.2 IMAGENET RESULTS
We primarily focus on the end-to-end fine-tune tine-tuning performance and report the top-1 validation accu- racy ImageNet-1K (Deng al., 2009) dataset.
5

---

Published as a conference paper at ICLR 2024
Table 2: Comparison fine-tuning result of the previous methods on ImageNet-1K. -.k. We evaluate by the end-to-end fine-tuning protocol. All results are based on an image size of 224, except for VIT-H with an extra result with 448 image size. We perform distillation in each stage for 800 epochs and with stages (our default) in total.
Table 3: Semi-supervised learning 1K with different self- supervised models. 1% and 10% represent the label fraction. ViT-B is selected as the arch. All results are based on our implementation with the official pre-trained model.
ImageNet-
method
VTT-B
VIT-L
VIT-H
ViT-H448
supervised
82.3
82.6
83.1
MoCo v3
83.2
84.1
DINO
83.6
methods based on masked image modeling:
BEiT
83.2
85.2
method
1%
10%
iBOT
84.0
85.2
supervised
68.9
MAE 83.6
85.9
86.9
87.8
data2vec
48.7
71.2
data2vec 84.2
86.2
MAE
53.1
73.1
dBOT 84.5
86.6
87.4
88.0
dBOT
54.8
74.5
Table 4: Object detection and instance segmentation COCO and Semantic segmentation on ADE2OK. All results are based on our implementation with the official pre-trained model. We perform distillation in each stage for 800 epochs and with stages (default).
Apbox
A Apmask
mloU
mAce ViT-B
method
method
ViT-B
ViT-L.
ViT-B
VIT-L
ViT-B
VIT-L
ViT-L
supervised
49.8
51.2
43.2
44.5
supervised
47.4
49.9
DINO
50.1
43.4
iBOT
48.4
52.3
59.3
63.3
MAE
50.6
54.0
43.9
46.2
data2vec
48.2
59.5
iBOT
51.3
44.3
MAE
48.1
53.6
58.9
65.5
dBOT
52.7
56.0
45.7
48.2
dBOT
49.5
54.5
60.7
66.0
Evaluation setup. We sweep the base learning rate within range with batch size being 1024. We warm up the learning rate during the first 5 epochs to the initial learning rate and use cosine schedule for the rest of the epochs. We average all the patch tokens output from the last transformer block and pass them into a linear projection head for classification. We fine-tune ViT-B for 100 epochs and ViT-L and ViT-H for 50 epochs in total.
Comparison with previous results. We report the fine-tuning results on ImageNet-IK. mainly focusing on the comparison of the self-supervised and supervised methods. Supervised denotes the results reported in the MAE. As shown in Table 2, dBOT achieves remarkable results with different model capacities, demonstrating its scalability. We achieved top-1 evaluation accuracy of 84.5%, 86.6%, and 87.4% with ViT-B, ViT-L,, and ViT-H, yielding gains of 0.9%, 0.7%, and 0.5% compared to MAE. When fine-tuned with an image size of 448, dBOT further achieves accuracy of 88.0%, surpassing the results obtained by MAE.
Semi-supervised learning. To investigate the label efficiency of dBOT, we also show the semi- supervised results ImageNet-1K IK under different labeled data availability in Table 3. We focus on the comparison with self-supervised learning methods. The label-fraction sampling strategy follows (Chen et al., 2020). dBOT outperforms MAE by 1.7 and 1.4 points using 1% and 10% of the labels, respectively, showing higher label efficiency.
5.3 DOWNSTREAM TASKS
To further demonstrate the effectiveness, consider dense prediction tasks: object detection, mantic segmentation, and instance segmentation.
Objection detection and instance segmentation. We consider Cascade Mask R-CNN (Cai & Vasconcelos, 2019) the task head for object detection and instance segmentation with VIT-B and ViT-L on COCO (Lin et al., 2014). We report Aphox and Apmask for object detection and instance segmentation respectively. The results demonstrated in Table 4. dBOT outperforms the

---

Published as a conference paper at ICLR 2024
Table 5: Ablation study with ViT-B/16 on ImageNet-1K validation set. We report with the end-to- end fine-tuning top-1 accuracy (%). Ablation study is conducted with randomly initialized teachers. We note that models distilled from the pre-trained teachers generally share similar trends. Default settings are marked in gray vanilla denotes being 0 the breakpoint and 1 otherwise. co- bi denotes m is cosine annealed from value to
(a) Stage split number. 2-stage distillation works the best.
(b) Epoch for each stage. 2- stage distillation with 800 epochs for each stage works the best.
(c) Momentum update. The vanilla strategy explicitly splitting stages works the best.
pre-training epochs acc
pre-training epochs
acc
momentum
acc
T600
83.6
400-800
84.3
vanilla
84,5
800-800
84.5
800-400
84.3
0.9998
83.6
533-533-533
84,4
800-800
84.5
0.9999
83.9
800-1200
84.3
cosinc(0.996,1)
82.1
(d) Target normalization. Using patch representations w/o [LN] as targets works best.
(e) Student initialization. initializing the student's 's weight brcakpoints works best.
Re-
(f) Mask ratio. A mask ratio of
75% works best.
target norm
acc
student init
acc
mask ratio
acc
w/ [IN] w/o [LN] 84.5
84.3
w/o re-initialize
84.2
0.7
84.3
w/ re- re-initialize
84.5
0.75
84.5
0.8
84.2
previous self-supervised and supervised methods by large margin, setting result with both ViT-B and ViT-L. With ViT-B, dBOT achieves a Apbox of 52.7 and a Apmask of 45.7, outperforming the supervised baseline pre-training by 2.9 and 2.5 points, respectively. With ViT-L, such improvement is more prominent with 4.8 and 3.6 points respectively, showing the high scalability of dBOT for model capacity in downstream dense prediction tasks.
state-of-the-an
Semantic segmentation. We adapt UperNet (Xiao al., 2018) the task head for semantic segmentation with VTT-B and ViT-L on ADE20K (Zhou al., 2017). We report the mloU and mAcc for semantic segmentation, and the results are demonstrated in Table 4. We achieve the best performances on semantic segmentation compared to previous eeffssupee methods by a nontrivial margin. dBOT improves mloU from 47.4 to 49.5 with ViT-B, and 49.9 to 54.5 with ViT-L,, yielding gains 2.1 and 4.6 points respectively, compared to the supervised baseline. The improvement semantic segmentation is as significant in object detection.
5.4 ABLATION STUDY
Stage split number. We study the influence of stage number by splitting total training epochs of 1600 into varying distillation stages, from 0 to 2. Results are shown Table 5a. 2-stage distillation works the best (for classification task), achieving 84.5% accuracy. Splitting epochs to 3-stage brings 0.1% performance drop, while all splitting strategies obtain a top-1 accuracy higher than 83.6%, indicating its generalizability.
Epoch for each stage. Table 5b studies proper epochs needed for each stage in a 2-stage distilla- tion pipeline. With the 2nd stage distilling for 800 epochs, longer epochs for the 1st stage induces 0.2% improvement (84.3% vs. 84.5%). With the 1 stage distilling for 800 epochs, 800 epochs are enough for the 2nd stage since 1200 epochs incur no gain. Evenly splitting the epochs in 2-stage masked knowledge distillation achieves the best performance.
Momentum update. We use dBOT multi-stage distillation pipeline, which distill from momentum encoder with being () for every breakpoint and otherwise. We further investigate other momentum update strategies commonly used in self-supervised learning. Results are shown in Table 5c. The vanilla strategy works the best.
Target normalization. used target representation should be passed through the Layer Normalization (Ba al., 2016)
We study whether patch tokens obtained by the self- attention blocks to be

---

Published as a conference paper at ICLR 2024
140
140
140
140
140
120
120
100
100
100
100
100
80
80
60-
60
40
40-
40-
42
4.5.6
9001112
7.
101112
0
9100112
1123455 1.567.7
9101112
910012
140
140
Do
140
MAE
140
DALL
140
0.020m
200
100
100
100
100
40
42
2 3
7 9101112
0
6 101112
01234567 9101112
7 67
0 12
9101112
1
5
910012
Dell-2
DALL-4-2
random-2
Figure 2: Average attention distance of different heads w.r. layer number VIT-B with different teachers and their corresponding student distilled for 2 stages. The first row showcases the teachers while the second showcases the 2th stage distilled student. Models using different teachers achieve the same result. The distilled students obtain more local attention compared to the teachers.
0.57
0.57
6.57
0.5
037
037
0.47
6.17
0.4
0.4
0.47
0.57
0.57
0.37
037
0.2
0.2
0.7
0.2
0.1
6
101
0 6
9101112
4.0
9101112
112
9101112
910112
6.07
DIND-0
6.0
Def-01
0.0
MAE-
0.67
DALL---0
0.67
10000m00"
66.7
6.5
00
0.0
0.07
0.07
6.0
00
0.4
0.4
0.37
0.39
0.37
0.39
0.27
022
0.17
0:7
101112
9101
9301112
9301112
91011
DINO-2"
MAE-2
DALL-E
random-22 2
Figure 3: Singular value decomposition of different layers of ViT-B with different teachers and their corresponding student distilled for stages. The first row showcases the teachers while the second showcases the 2th stage distilled student. Models using different teachers achieve the same result.
layer [LN]. The accuracy of models after 2-stage distillation shown in Table 5d. Without passing through [LN], the patch tokens directly obtained from the transformer block make them less suitable as target representations guide students' learning.
Student initialization. We study whether student's weight should remain when entering the next stage of distillation. Specifically, we either keep the student's weight unchanged or re-initialize the student at each breakpoint. As shown in Table 5e, re-initializing the student's 's weight works the best.
Mask ratio. Table 5f shows the influence of the mask ratio on end-to-end fine-tuning. The optimal mask ratio for dBOT is 75%, the same as that in MAE.
PROPERTY ANALYSIS
We investigate the properties of models distilled from different teachers under certain criteria, ana- lyzing models' weights and outputs. Further, training efficiency is briefly discussed with previous methods.
Averaged attention distance. We compute averaged attention distance (Dosovitskiy et al., 2021), averaged over ImageNet-IP 1K val set, for each attention head of different blocks understand how local and global information flows into Transformers. Average attention distance for dBOT using DeiT, DINO, MAE, DALL-E, and random as teachers illustrated in Fig. 2. The higher the
8

---

Published as a conference paper at ICLR 2024
Table 6: Training time (s) per epoch for different methods with ViT-B/16,
Table 7: Results of classification (cls.) on INIK, ob- ject detection (det.) on COCO, and semantic segmen- tation (seg.) on ADE20K. For same-size teachers (col- ored gray), students are pre-trained with default settings. For bigger teachers, students are pre-trained for I-stage
ViT-L/16, and VIT-H/14. asym. de- notes whether use an asymmetric
encoder-decoder structure. All entries are tested on the same setting, i.e., with 32 NVIDIA A100-80G GPUs.
from 2-stage distilled teachers.
teacher
student
cls.
det.
seg.
method data2vec BEiT MAE dBOT
VIT-B
84.5
52.7
49.5
asym.
X
X
/
VIT-L
VIT-B
84.6 (+0.1)
53.1 (+0.4)
50.1 (+0.6)
ViT-B
169
166
79
109
VIT-H
84.6 (+0.1)
53.5 (+0.8)
50.8 (+1.3)
VIT-L
431
356
125
200
VIT-L
86.6
56.0
54.5
VIT-L
VIT-H
960
751
240
416
VIT-H
86.8 (+0.2) 56.1 (+0.1) 55.2 (+0.7)
attention distance, models' attention over an image is more global. Although the average attention distance of disparate initialized teachers varies greatly, their distilled students after multi-stage distil- lation exhibit similar behaviors, e.g., models' attention toward local or global contents. Additionally, dBOT achieves more local attention than previous works.
Singular value decomposition. We computed the percentage of top-k singular values (Wall et al., 2003) of the embedding w.r.t each layer. The results are averaged over the ImageNet- IK val set. We showcase the results with & varying from to 5. Singular value decomposition for dBOT using DeiT, DINO, MAE, DALL-E, and random as teachers are shown Fig. 3. The higher the percent- age, the models' output over an image is less correlated, indicating larger redundancy of its spatial representations thus less suitability for compression. Intuitively, random models at the 0th stage has the largest percentage given that pixel are merely randomly projected. The student networks distilled from different initialized teachers exhibit similar behaviors.
Training efficiency. We compute the training time per epoch for different methods in Table 6. With an asymmetric encoder-decoder architecture (asym.) as the default setup, dBOT performs slower than MAE, but much faster than data2vec and BEiT. Such advantage turns more significant with models of larger size.
7 DISTILL FROM BIGGER TEACHERS
Inspired by canonical practices in knowledge distillation (Hinton et al., 2015), we use larger teachers to distill smaller students, showeasing the potential of MKD general. Specifically, we attempt to use ViT-L/H as teacher networks distill ViT-B, and VIT-H as the teacher network to distill ViT- L. All larger teachers are first distilled for 2 stages with the default setup. We resize the image to 196 x 196 for ViT-H/14 to keep the length of its output the same as that of ViT-B/L. While do not find substantial gains on classification results, the results by distilling from ViT-H are significantly better for dense prediction tasks compared to the default setup, i.e., +0.8 points of Apbox and +1.3 points of mloU with ViT-B as the student. The performance gain in distilling ViT-L from ViT-H is diminished but still valid, i.e., +0.1 Apbox and +0.7 mloU. We also consider MKD with data-richer teachers, e.g. CLIP, as exploratory experiments and set new state-of of-the-art results for self-supervieed learning. Refer to Appendix C for details.
CONCLUSION
As special case of MIM, we formulate MKD upon which an empirical investigation is conducted about the influence of different target representations on self-supervised masked autoencoders. The study concludes that it is not necessary to carefully choose the target representation to learn good visual representations if distillation is performed in multiple stages (i.e., with bootstrapped teach- ers). Instead of initializing teachers with pre-trained models, we resort to random ones for simple practice. Without an extra stage of pre-training, dBOT achieves favorable performance image classification, object detection, and semantic segmentation. We hope our study and method will provide timely insights for self-supervised learning.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 100716607.5555 2016.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: general framework for self-supervised learning in speech, vision and language. In ICML., 2022.
Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. In ICLR, 2022.
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: high quality object detection and instance segmentation. TPAMI, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.
Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Ross Girshick. Fast R-CNN, In ICCV, 2015.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In ICAIS, 2010.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeruIPS, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in neural network. In NewrIPS Workshops, 2015.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019.
10

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scout Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitmets: Hints for thin deep nets. In ICLR, 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019.
Contrastive representation distillation. ICLR,
Hugo Touvron, Matthieu Cord, Mathhis Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Michael E Wall, Andreas Rechtsteiner, and Luis M Rocha. Singular value decomposition and prin- cipal component analysis. In A practical approach to microarray data analywis, 2003.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Yuan Wu, Alan Yuille, and Christoph Feichtenhofer.
Masked feature prediction for self-supervised visual pre-training. In CVPR, 2022a.
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. MVP: Mulimmodalit-ggudded visual pre-training. arXiv preprint arXiv:2203.05175. 2022b.
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation.
arXiv preprint 220.:222005.11111 2022c.
Tete Xiao, Yingcheng Liu, Bolci Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: A simple framework for masked image modeling, In CVPR, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image bert tra-iing with online tokenizer. In ICLR, 2021.

---

Published as a conference paper at ICLR 2024
Table Al: Pre-training setup. recipe is the pre-training recipe for dBOT cos. denotes cosine distance. C., d., and s. denotes downstream tasks of classification, object detection, and semantic segmentation respectively. drop path for the students.
config optimizer
default
recipe
AdamW 0.9
optim. momentum 81
optim. momentum B2
0.95
0.98
loss peak learning rate
Smooth L1
negative 3e-3
2.4e-3
learning rate schedule
cosine decay
batch size
4096
weight decay
0.05
stages epochs per stage warmup epochs augmentation aug. input scale asym. enc-dec drop path target w/ [IN] mask ratio
2 (c.). 3 (d./s.)
1
800
1600
40 RandomResizedCrop
10
(0.2.
(0.4, 1)
/
X
0.2 (B/L., 0.3 (H)
0.1 (B/L/HH)
x
0.75
0.4
Table A2: End-to-end fine-tuning setup. recipe is the pre-training recipe for dBOT
config optimizer
default
recipe m
Adam W
peak learning rate weight decay optim. momentum layer-wise decay batch size learning schedule warmup epochs
30.
{1,2,3,4)e44
0.05
B1, 32 = 0.9, 0.999
0.75
1024
cosine decay 5
epochs
100 (B), 50 (L/H)
augmentation
RandAug (9, 0.5)
label smoothing
0.1
mixup
0.8
cutmix drop path
1.0
0.2 (B/L), 0.3 (H) 0.1 (B), 0.2 (L), 0.3 (H)
A IMPLEMENTATION DETAILS
A.1 PRE-TRAINING
Default setup. We show our default pre-training setup in the second colum of Table A.. We use Xavier Uniform (Glorot & Bengio, 2010) to initialize the Vision Transformer (Dosovitskiy et al., 2021). Note that we use asymmetry stochastic drop path rate for students and teachers.
Setup for distillation from bigger teachers. We follow the default setup, except that different setup for stages. We first train larger-size teachers for 2 stages (in all downstream tasks) and use those to distill new students for stage (in all downstream tasks).
12

---

Published as a conference paper at ICLR 2024
Table A3: Object detection setup.
Table A4: Semantic segmentation setup.
config
value
config optimizer
value
optimizer
Adam W
Adam W
optim. momentum
B1. 02
0.9, 0.999
optim. momentum
B1. = 0.9, 0.999
peak learning rate
le-4
peak learning rate
(0.3.0.5,0.8,1,33)e4
batch size layer-wise decay weight decay
16
batch size layer-wise decay weight decay learning schedule steps
16 (0.65.0.75.0.85.0.93
0.75
0.05
0.05
learning schedule
step
cosine
epochs
12
16000
step epochs
8, 11
warmup steps
1500
drop path
0.2
drop path
OBI 0.2(L)
A.2 CLASSIFICATION
The default end-to-end fine-tuning recipe is shown in the second column of Table A2, following the common recipes (He al., 2022; Bao al., 2022) of VIT tuning for self-supervised models. The same recipe is applied when distilling from bigger teachers.
A.3 OBJECT DETECTION AND INSTANCE SEGMENTATION
We adopt the vanilla VIT with Cascade Mask R-CNN (Cai Vasconcelos, 2019) as the task head on COCO (Lin al., 2014) dataset for object detection and instance segmentation, following the common setup (Zhou et al., 2021). The default recipe shown in Table A3. cope with versatile image sizes, add relative position embedding instead of interpolating the absolute position em- bedding obtained during pre-training. For a fair comparison, we applied the same setup and sweep the learning rate and stochastic drop path rate for different methods.
A.4 SEMANTIC SEGMENTATION
We use vanilla VIT and UperNet (Xiao et al., 2018) as the task head on ADE20K (Zhou et al., 2017) dataset for semantic segmentation, following the common setup (Bao al., 2022). The default recipe is shown in Table A4. To cope with versatile image sizes, we add relative position embedding instead of interpolating the absolute position embedding obtained during pre-traint pre-training. For a fair comparison, we applied the same setup and sweep the learning rate and layer-wise decay for different methods.
B ADDITIONAL EXPERIMENTS
B. PIXELS vs. RANDOM MAPPING OF PIXELS
MAE performs masked image modeling using the image pixel as the reconstruction target. We directly alter the target to patch tokens obtained from the image fed into a randomly initialized network. We select two patch tokens as the reconstruction target, one is the token obtained using the last transformer block, and the other is the token obtained using linear projection, i.e., without any transformer block. After 400 epoch pre-training of ViT-B, the top-1 accuracy of the model on ImageNet- 1K obtained by the three different targets is shown below.
epoch
pixel
Oth block
12 block
400
83.3
83.2
83.2
1600
83.6
83.6
83.6
It can be derived that using the patch token obtained by randomly initialized network the target can achieve comparable results with a pixel as a target. A similar result proves that patch tokens obtained by randomly initialized can also serve good reconstruction target.
13

---

Published as a conference paper at ICLR 2024
B.2 LINEAR PROBING
We evaluate the linear probing performance of dBOT and MAE using ViT-B following the same setup MAE, the results of which shown below.
MAE
dBOT
67.8%
67.9%
dBOT achieves comparable linear probing performances with MAE.
C DISTILL FROM DATA-RICHER TEACHERS
We explore models pre-trained with richer data (i.e., CLIP (Radford et al., 2021) with 400M Image-Text pairs) as the initialized teacher to seek potential upper-bound of MKD.
C.I PRE-TRAINING
Compared to the default setup, there exist two major disparities of the pre-training recipes for models distilled from data-richer teachers, discussed next. The following practice is summarized as recipe detailed in Table A1.
Vanilla Architecture. We find that not using the asymmetric encoder-decoder architecture (He et al., 2022) is optimal, as shown in Table C5. While an asymmetric architecture generates momen- tum for bootstrapping models similar (Grill al., 2020), which lies crucial for distillation with random teachers, it hurts the performance when distilling with stronger pre-trained teachers.
Hypothetically, the significance of the decoder in asymmetrical encoder-decoder architecture lies in the need for separate layers to decode low-level details when the targets contain little semantics (e.g., pixels and random mappings of pixels). Such need is eased when the target contains high-level semantics (e.g.. DINO and CLIP). The existence of the decoder, in this case, may even restrain the encoder to grasp full knowledge from the teacher, inducing degraded performances.
1-Stage MKD. We use different models teachers to distill students for one stage with longer epochs, i.e., 1600. Results are shown in Table C6. Empirically, the performance gains for multi- stage MKD over 1-stage MKD decrease teachers' fine-tuning performance increases. Stronger teachers, such as DINO and MAE, induce similarly performed students with 1-stage MKD (1 x 1600) compared to 2-stage MKD (2x800).
Specifically, when using CLIP as the pre-trained teacher, the performance for 2-stage MKD is, to surprise, 0.9% lower than that of I-stage MKD. Understandably, although the fine-tuning result of the student after 1-stage distillation better than that CLIP, the student essentially trained on INIK and may not contain faithfully data information stored in the CLIP model. Therefore, strong teachers work well with I-stage MKD, especially for models pre-trained on extra richer data.
C.2 DOWNSTREAM TASKS
Implementation Details. For fine-tuning, we also use slightly different recipe from default one with smaller learning rates and drop path, dubbed as recipe detailed in Table A2. For object
Table C5: Image classification on INIK with DINO and CLIP as initialized teachers, well as random ones. Students with DINO and CLIP teachers are distilled for stage.
initialized teacher
pre-training data
asym. enc-dec
acc
random
INIK
84.5
random
INIK
X
83.8
DINO
INTK
/
84.4
DINO
INIK
X
84.8
CLIP
INIK + 400M ITP.
/
84.9
CLIP
IN1K + 400M ITp.
X
85.7
14

---

Published as a conference paper at ICLR 2024
Table C6: ImageNet-IK classification results of 1 stage masked knowledge distillation with dif- ferent teachers. Total epochs are shown in the format of (stagesxepochs. per stagsseee.............. A denotes performance gaps between entries x and x 1600.
pretraining epochs
random
DALL-E
DeiT
DINO
MAE
CLIP
0
77.3
81.1
81.8
83.2
83.6
84.8
1 1600
83.6
83.6
83.6
84.4
84.4
84.9
800
84.5
84.4
84.3
84.5
84.4
84.0
A
+0.9
+0.8
+0.7
+0.1
+0.0
-0.9
Table C7: Results of classification (cls.) INIK, object detection (det.) on COCO, and semantic segmentation (seg.) on ADE20K with CLIP (Radford et al., 2021) the teacher. Students are distilled for 1 stage. The det. results with CLIP teachers are with absolute positional embedding.
initialized teacher
student
cls.
det.
seg.
random
84.5
52.7
49.5
ViT-B
CLIP-B
85.7 (+1.2)
53.6 (+0.9)
52.9 (+3.4)
random
86.6
56.0
54.5
ViT-L
CLIP-L random
87.8 (+1.2)
56.8 (+0.8)
56.2 (+1.7)
87.4
VīT-H
CLIP-L
88.5 (+1.1)
detection, instance segmentation, and semantic segmentation, follow the default setup detailed in Appendices A.3 and A.4.
Results. Results for downstream tasks shown in Table C7. ViT-B distilled from CLIP-B achieves an 85.7% top-1 accuracy and a 52.9 mloU, surpassing all previous arts. With CLIP-L as the teacher, ViT-H with image resolution 448 achieves an 89.1 % top-1 accuracy, setting new state-of-the-art image recognition result.
C.3 CONFLICT WITH MAIN CONCLUSION
can be observed that MKD with CLIP (Radford et al., 2021) as the teacher performs much better than that with the random teacher and multi-stage distillation, which seems contradictory to main conclusion that teacher networks do not matter with multi-stage masked knowledge distillation. Notably, CLIP is trained with 400M image text pairs (300x larger than ImageNet-IK), which is drastically different setup from multi-stage distillation on ImageNet- only. Exploring CLIPP as a target representation gains popularity (Wei et al., 2022b) recently but is beyond the main scope of this paper. We present these results to corroborate the validity and to explore the upper bound of MKD in general. We note that the exact solution resolve the conflict is to perform multi-stage distillation using the CLIP's in-house 400M data to which we have no access. It is hypothesized that two results should be matched in light of experiments on ImageNet- -KK, which left future work.
15


Reviews:
[{"review_id": "RufsQ3YD5a", "reviewer": "Reviewer_aTJx", "summary": "This paper studies the role of target representation in Masked Image Modeling framework. Prior works each proposes a separate teacher network to generate the reconstruction target: BeiT uses DALL-E; MaskFeat uses HoG; MVP uses CLIP, without clearly justifying the necessity. This work finds that different choices of teacher network, including a random initialized one, leads to close performance in MIM training. Furthermore, it proposes a bootstrapped iterative MIM training pipeline, called dBOT, which shows improved performance.", "questions": "1. In table 1, I find a randomly initialized teacher network could achieve 77.3 accuracy on ImageNet (and similar for other datasets), which really seems impossible to me. Am I missing something here?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The observation that diffrent teacher models do not make a large difference in generating target representation in MIM learning is interesting. The high-level idea of this paper is easy to follow, and the model does show good performance on classification, detection, and segmentation tasks.", "weaknesses": "1. The proposed method dBOT and the observation about the choice of teacher model is somewhat unrelated. It is my opinion that the good performance of dBOT comes from multi-stage training, which I find not clearly motivated in this paper. For instance, if the teacher network is switched in to some other pretrained networks, I feel confident that this dBOT pipeline would still perform good.\n\n2. The observation of the insignificance of the choice of teacher network is somewhat aligned to the observation of MAE, which simply opts for raw image pixels and surpasses previous methods like BeiT or MaskFeat with more sophisticated target representation by a non-trivial margin. This is also validated from the results in appendix B.1, which shows *using the patch token obtained by a randomly initialized network as the target can achieve comparable results with a pixel as a target.*\n\n3. The authors claim that *Using a random model as teachers not only avoids an extra pre-training stage, but also alleviates the painstaking selection of the target representations*. But I think this has already been achieved by MAE, since reconstrucing raw image pixels does not involve pretraining, and there is no need to select target representations. Thus, I am not sure what is the practical value of the observation of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iKj5EvTSXO", "reviewer": "Reviewer_tvFW", "summary": "The paper analyses the role of target representations (reconstruct targets) in masked knowledge distillation for self-supervised learning (SSL). It finds that distilling from the output of a randomly initialised network results in performance and properties similar to distilling from pre-trained representations. Based on these observations, the work proposes a new SSL method, dBOT that employs multiple stages of distillation starting from a randomly initialized teacher network. The results indicate that the proposed SSL pretraining consistently outperforms prior SSL pretraining on the downstream tasks of image classification, semantic segmentation, and object detection.", "questions": "# Questions\n1. In context of the weaknesses above, do authors have evidence that suggests that performance saturates and not drops with more stages and training? \n2. **Sample efficiency.** SSL methods have been shown to improve with more training (Figure 7 of MAE [1]). Is the total number of pre-training iterations controlled across different methods in Table 2, 3, 4 and 5? \n4. The authors compute properties **within** model and show that these match across different choices of teachers (Figure 2 and 3). Do the learned layers **across distilled models** show any correspondences? eg. how do the cross-model CKA similarity maps [2] look?\n5. Table 6: Why is MAE faster than dBOT when both use an asymmetric encoder-decoder architecture?\n6. \"Additionally dBOT achieves more local attention than previous works\". Can authors clarify this comment in context of Figure 2? It seems like MAE's attention distance plot is similar to the distilled models'.\n7. Section 6. SVD line 5: Shouldn't lesser correlation in model's output result in lower redundancy?\n8. C.1. I think I understand why the need for a decoder could be \"eased when target contains high-level semantics\". But can the authors elaborate a bit on why the \"existence of decoder\" may hurt?\n\n\n# Minor typos/suggestions:\n1. Section 1: A masked image **is** passed through the \n2. Section 2.2: Page 3 last line: teacher and ~~bootstraps~~ bootstrap the teacher for stages\n3. Table 2 caption: ~~Comparison~~ Comparing fine-tuning result\n4. The method (section 4) suggests to continue repeating the distillation until a saturation in downstream task is observed. The authors may want to reword this as it is odd to have the SSL pretraining phase depend on a particular downstream task.\n\n# References\n\n[1] Masked Autoencoders Are Scalable Vision Learners. He et al. 2021\n\n[2] Do Vision Transformers See Like Convolutional Neural Networks? Raghu et al. NeurIPS 2021", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper makes a novel and interesting observation of the masked knowledge distillation being invariant to the initial teacher networks (Table 1).\n2. The proposed method is simple and achieves consistent improvements over prior SSL methods (models) across multiple downstream tasks (Table 2,3,4).\n3. The paper performs thorough ablation study (Table 5) and analysis of model weights and outputs (Section 6)\n4. The method seems to be stable with respect to minor changes in the pretraining setup. (Table 5)\n5. The work promises further improvements if data beyond ImageNet-1k is used (Appendix C.2 and C.3), including a result that shows model training from a CLIP-L teacher achieving 89.1% top-1 accuracy (new SoTA image recognition result)", "weaknesses": "Please look at the Questions section for suggestions on improving the draft. \n\nHere are a few minor concerns (that are a bit related to each other):\n1. **dBOT does not scale with more training?**. Results in Table 1 show a drop in performance on adding more stages, thus requiring a good stopping condition.\n2. **Different models for different tasks** as opposed to prior works that typically release a single general model\nThe purpose of SSL is to learn a single model that generalize across tasks. Given that the performance on different downstream tasks peaks at different stages (Table 1), it is not clear how to choose the number of distillation stages to arrive at a single general model that can work across tasks (even beyond the ones discussed - eg. for embodied navigation [1]). \n3. **Do the trends hold across random seeds?**. For a single random initialized teacher, Table 1 shows 2 stage being better for classification and stage 3 being better for detection/segmentation. It is not clear if trends like this (and the performance) hold across random seeds.\n\n[1] Offline Visual Representation Learning for Embodied Navigation. Yadav et al. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AqUDJa1rXi", "reviewer": "Reviewer_2u7d", "summary": "This work studies the effect of knowledge distillation in mask auto-encoders. The authors observe that the choice of teacher representation becomes inconsequential when employing multi-stage distillation. As a result, the authors propose a novel approach called MKD, which utilizes bootstrapped teachers initialized randomly. Notably, MKD yields significant performance improvements when compared to alternative methods.", "questions": "1. What is the recipe ⚗ in appendix A.1? I did not find the definition of this setting?\n\n2. Pls see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents a reasonable and novel story, and the conclusion regarding the teacher representation for Masked auto-encoders is convincingly demonstrated through solid preliminary experiments.\n\n2. The organization and writing style are clear, making the paper easily readable.\n\n3. The authors perform extensive and compelling experiments to validate the effectiveness of MKD, which results in significant performance improvements compared to the baselines.\n\n4. The analysis provided in Section 6 is highly appreciated.", "weaknesses": "1.  The conclusion of the paper is limited to Masked auto-encoders where the teacher and student models are pre-trained with the same data (IN-1k). However, this is not clearly stated in the main paper.\n\n2.  A While the authors mention the instability and sensitivity of other methods, it should be noted that MKD also requires careful pipeline design and hyper-parameter selection. For example,  MAE with fixed m (momentum updating teacher’s param) can achieve 84.3 at stage 2, however, in most cases in the ablation, MKD only outperforms MAE when the stage split number, epochs for each stage, and momentum parameter (m) and others are correctly set. This limits the practical application of MKD. This limits the practical application of MKD, as one could simply distill MAE for two stages with less time consumption and similar performance.\n\n3. The paper only explores the fine-tune setting of semi-supervised learning (SSL), which diminishes the differences between pre-training models. How about the linear-probe setting? Would teachers also matter? Additionally, it is unclear whether the models mentioned in Section 6 are supervisedly fine-tuned on IN or pre-trained models before fine-tuning.\n\n4. A minor weakness of the paper is that only l2 distillation is performed, neglecting other potential distillation methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P4HjW4fiMK", "reviewer": "Reviewer_wVuK", "summary": "This paper explores a quite interesting perspective of masking image modeling by first summarizing current methods with a more high-level architecture called masking knowledge distillation and then empirically demonstrate the learnt backbones of different target representation do not differ from each other with respect to both transfer results and weight distribution. Based on that, the authors propose a simple yet effective framework called dBOT to learn strong self-supervised visual representation.", "questions": "- Writing:\n  - Typo: \"self-supervised learning\" instead of \"self-supervised learninf\" in Keywords", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper has a clear formulation and writing architecture to present their motivation.\n- The experimental results are solid and completed to support their claim.\n- The observed phenomenon is quite interesting.", "weaknesses": "- This paper constructs tightly with the proposed masked knowledge distillation framework in Equ. 1, which consists of several basic components, including 1) the transferred backbone, 2) target representation, 3) asymmetric masking and 4) similarity measurement.\n- About target representation:\n  - The phenomenon observed in this paper can be more detailed phrased as, \"With long enough pre-training, different target representation demonstrate similar behavior\". Therefore, does that suggest that the observed conclusion only mask sense with a fixed dataset with long enough training? \n  - In other words, is the bottleneck of scalable visual representation learning not about the methodology but more about data?\n- About asymmetric masking:\n  - Following my question above, another perspective to understand this phenomenon is due to the limited scalability [1] of asymmetric masking architecture proposed in MAE, which is also utilized as the main architecture in this work. While in Swinv2 and EVA, the MIM pre-training has been quite important for ViT training of giant sizes. Does the implementation of masking also affect the scalability of pre-training?\n  - Moreover, does the invariance of target representation suggest masking operation is the key in MKD, which might also explain why the learnt representation has similar weight distribution?\n- About backbone architecture:\n  - Throughout the whole paper, the authors utilize the vanilla ViT of different sizes for both the student and teacher networks, while DeiT has shown that architecture discrepancy exists when distilling  between different architectures like CNN and ViT. Will that be the same for MKD, like does the phenomenon still hold with a ConvNext teacher?\n- Overall, I think this is a quite interesting paper, and my questions are more open-ended to further enhance the insight of the observed phenomenon for future development.\n\n[1] Zhai, Xiaohua, et al. \"Scaling vision transformers.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)