Deep Dynamic AutoEncoder for Vision BERT Pretraining
Authors: Honghao Chen
Xiangwen Kong
Xiangyu Zhang
Xin Zhao
Kaiqi Huang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Recently, masked image modeling (MIM) has demonstrated promising prospects in self-supervised representation learning. However, existing MIM frameworks recover all masked patches equivalently, ignoring that the reconstruction difficulty of different patches can vary sharply due to their diverse distance from visible patches. In this paper, we propose Deep Dynamic AutoEncoder (DDAE), a novel MIM framework that dynamically focuses on patch reconstructions with different degrees of difficulty at different pretraining phases and depths of the model. In addition to raw pixel regression, DDAE performs dynamic feature self-distillation for intermediate layers to learn semantic information. Our methodology provides more locality inductive bias for ViTs, especially in deep layers, which inherently makes up for the absence of local prior for self-attention mechanism. Moreover, our core design deep dynamic supervision can be migrated into existing MIM methods (e.g., MAE, BEiT-v2) seamlessly. The Experimental results demonstrate the effectiveness of our approach. As a tokenizer-free framework, the base-size DDAE can achieve 83.5% top-1 accuracy with only 100 epochs pretraining, surpassing MAE and BEiT pretrained for 800 epochs. For a longer pretraining schedule, DDAE achieves 84.3% top-1 accuracy on Imagenet-1K, and 49.3% mIoU on ADE20K for semantic segmentation.

Under review as a conference paper at ICLR 2023
DEEP DYNAMIC AUTOENCODER FOR VISION BERT PRETRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, masked image modeling (MIM) has demonstrated promising prospects in self-supervised representation learning. However, existing MIM frameworks recover all masked patches equivalently, ignoring that the reconstruction difficulty of different patches can vary sharply due to their diverse distance from visible patches. In this paper, we propose Deep Dynamic AutoEncoder (DDAE), a novel MIM framework that dynamically focuses on patch reconstructions with different degrees of difficulty at different pretraining phases and depths of the model. addition raw pixel regression, DDAE performs dynamic feature self-distillation for intermediate layers to learn semantic information. Our methodology provides more locality inductive bias for ViTs, especially in deep layers, which inherently makes up for the absence of local prior for self-attention mechanism. Moreover, our core design deep dynamic supervision can be migrated into existing MIM methods (e.g., MAE, BEÎT-v2) seamlessly. The experimental results demonstrate the effectiveness of our approach. As tokenizer-free framework. the base-size DDAE can achieve 83.5% top-1 accuracy with only 100 epochs pretraining, sur- passing MAE and BETT pretrained for 800 epochs. For longer pretraining sched- ule, DDAE achieves 84.3% top-1 accuracy on Imagenet- K, and 49.3% mloU on ADE20K for semantic segmentation.
INTRODUCTION
Aided by the rapid gains in hardware, deep learning has ushered in the era of big models and big data. Along with the ever-growing model capacity, the demand for data can easily reach hundreds of
millions (Dosovitskiy et al. 2020). which is not publicly accessible for labeled data. Self- Supervised Learning(SSL) frameworks, such as DINO (Caron al.) (021]. MOCO (Chen al., 2021). BeiT
[Boo al. 2021 etc., etc., have grown in concern in vision model pretraining without the need for labels.
In particular, the recently proposed Masked Image Modeling (MIM) methods (He et al. 2022 Xie
et al. 2022b Dong et al., [221] Wei et al.) 2022b Chen et al. 2022b; Dong et al. 2022 Chen al.
2022ch have shown remarkably impressive performance in a variety of vision tasks, demonstrating the promise unify computer vision and natural language processing(NIIP) pretraining (Peng al. 2022a).
Inspired by BERT (Devlin al. 2018) in NLP, MIM pretrains the encoder by reconstructing the masked image patches from visible patches. Existing MIM methods can be divided into two cate- gories according to the need for additional tokenizer. Two-stage methods: Represented by the pioneer work BEiT (Bao et al. 202T). two-stage methods firstly transform image patches into se- mantic visual tokens through a pre-trained tokenizer. then reconstruct the corresponding tokens of masked image patches to pretrain the encoder. Tokenizer needs to be offline pretrained with fixed model architectures and extra data (Zhang et al. 2019b] Ramesh et al. 2021 Radford et al. 2021), some methods further require an off-the--helf DNN as teacher to distill tokenizer pre-training (Peng ctt al., 2022a). One-stage methods: Taking the recent work MAE (He et al., 2022) as a represen- tative, MAE constructs an asymmetric encoder-decoder structure, and directly performs raw pixel regression of masked image patches. One-stage methods are tokenizer-free. the reconstruction tar- get includes raw pixels, features (e.g., HOG) and self-distillation targets (He et al.) 2022; Xie al. 2022b, Wei et al., 2022at Chen et al. 2002cc.. MIM methods enable Vision Transformers ViT'sto learn rich visual representations and show great potential in various downstream tasks.

---

Under review as a conference paper at ICLR 2023
Naturally, we ask why MIM works. Facilitated by self-attention mechanism, ViTs excel at model- ing long-range dependencies, but different from CNN, ViTs' lack of local prior(i.e., a pixel is more related to its neighbors than the distant pixels) largely leads to its need for large dataset pretraining (e.g.. JFT-300M) or special training techniques (e.g., a DeiT-style distillation method (Touvron et al., conjecture that the success of MIM comes from the fine-graines reconstruc-
(021b)). Therefore, tion of masked image patches. The generation task built by MIM enables ViTs to pay more attention to vicinity besides using global semantic reasoning. Thus, it inherently makes up for the absence of local prior in the structure of ViTs, which is essentially different from the discriminative task in supervised pretraining or contrastive learning.
However, success comes with remaining obstacles. At present, all MIM frameworks recover all patches equivalently, ignoring the fact that the reconstruction difficulty of different patches can vary sharply, and the semantic reasoning and local perception ability required for patches are not the same. Generally, recovering patches with more visible patches around will be simpler, as long as the model has sufficient local perception ability. In contrast, reconstruction with few visible patches around requires the model to have strong semantic reasoning ability, given that there is little access to neighboring information. Treating all pixels equally will neglect this demand for different properties of the model, inevitably limiting its representation ability. On the other hand, layers at different training phases naturally learn features at different levels. Therefore, we ask if there is way to focus on objectives with diverse characteristics as training progresses so that better representations can be learned overall.
Motivated by this observation and answering the question above, we propose simple yet effective SSL framework named Deep Dynamic AutoEncoder (DDAE). With deep dynamic supervision, the model dynamically focuses on different patches at different training phases. First, it attaches impor- tance patches close to visible ones, which only demands fair local perception to recover. Then compared with the beginning of training, the model further pays attention to the distant and difficult patches aided by the simpler patches, which have been attached more importance in the earlier pre- training phase. Specifically, we first define the reconstruction difficulty according to the distance between masked patches and visible ones, generating a distance map. Then, we exert different su- pervision signals (which are controlled by learnable parameters 3 for intermediate layers. As the training progresses, the model dynamically focuses on different regions with the update of B.
Since the pixel representations of perceptually similar images might be very different, only pixel- level reconstruction can be vulnerable to the perceptual difference between images (Dong et al.] 2021 Besides pixel regression, we further apply feature self-distillation to provide semantic in- formation. We directly align the encoder's intermediate features with the corresponding features of the momentum encoder in deep dynamic supervision, where the momentum encoder is updated by Exponential Moving Average (EMA) (Grill et al., 2020 He et al., 2020). It is worth noting that the feature distillation target here is the corresponding feature of the momentum encoder, while the regression targets for intermediate layers are all raw pixels. Dynamic loss is applied for both feature self-distillation and pixel regression.
Furthermore, we directly migrate the core design deep dynamic supervision to the representative methods in one-stage and two-stage, MAE (He et al., 2022) and BEiT-v2 (Peng et al. 2022a) re- spectively, surpassing original methods with nontrivial margins. Since deep dynamic supervision does not introduce any additional structure, it can be used as a general MIM pretraining plug-and- play module.
The contributions are summarized as follows:
We propose DDAE, a one-stage tokenizer-free framework for self-supervised representa- tion learning. Our core design deep dynamic supervision can be migrated to existing MIM approaches seamlessly, providing more local priors for ViTs to make up for their inherent structural defects.
Although deep supervision does not perform well in modern supervised learning, DDAE combines dynamic loss and deep supervision in a novel and effective way, making a step towards unleashing the potential of deep supervision in MIM.
2

---

Under review as a conference paper at ICLR 2023
The experimental results demonstrate the effectiveness of the proposed DDAE, we achieve state- oate--of-the-arr art performance among tokenizer- free methods and have competitive perfor- mance with many two-stage methods.
2 RELATED WORK
Local Prior for ViTs. Aided by the global receptive field of multi-head AMSS)))
mechanism, ViTs (Dosovitskiy et al. (020)) have significantly advanced computer vision tasks such as image classification (Wu et al.J 2020; Touvron et al.] 202 Yuan al.) 2021). object detec-
tion (Carion et al., 2020 Zhu et al., 2020; Beal et al. 2020), and semantic segmentation (Zheng) since self-attention does not have CNN's local prior, it often ignores local feature details, which largely leads to ViT's demand for large dataset pretraining. To solve, DeiT
jet al. (021).
(Touvron et al. 202 transfers CNN-based knowledge to vision transformer through a distillation token. VIT (Yuan al.] 2021) proposed using a tokenization module to recursively reorganize
the image to consider neighboring pixels. Local MSAs (Yang et al. 2019; Liu et al. [002 Chu
jet al. 2021) are proposed to calculate self-attention only within small windows, achieving better performance than global MSAs not only on small datasets but also on large datasets, e.g., ImageNet-
21K. Since local prior is the natural inductive bias of convolution, there are also works aimed
integrating convolution and vision transformer. Works like PVT (Wang et al. 2021 Iand CvT (Wu al. 202T) insert spatial reduction or convolution before global sell-attention, yielding the merits of self-attention and convolution. MixFormer (Chen et al. 2022a) and Conformer (Peng et al. (021) integrate the advantages by building parallel interactions. should be noted that all above methods fundamentally change ViTs' basic structure, bringing negative effects such as restricted receptive field or increased computational cost of downstream tasks. Different from them, our work focuses on providing local prior for vanilla ViTs without introducing any extra structural modification.
Masked Image Modeling. Inspired by BERT (Devlin et al. 2018) in NLP, MIM learn represen- tation by reconstructing masked image patches from visible patches. Existing MIM methods can be divided into two categories according to the need for the additional tokenizer. Two-stage meth-
ods: Represented by the pioneering work BEiT (Bao et al. 2021. two-stage methods (Bao al.,
2021 Dong et al. 2021: Wei et al. 20226 Chen et al.) 2022b] Peng et al., (0226) firstly trans- form image patches into semantic visual tokens through a pretrained discrete VAE (Ramesh al.,
2021 la visual tokenizer, then the corresponding tokens of masked image patches are reconstructed to pretrain the encoder. Nevertheless, the tokenizer needs to be offline pretrained with fixed model architectures and extra data (Zhang et al., 2019b] Ramesh et al. [201 Radford et al., (021), some methods even further require an aff-the-shelf DNN as teacher to distill tokenizer pre-training (Peng et al. 2022b), which complicates the process and increases cost. One-stage methods: MAE (He et al. 2022]constructs an asymmetric encoder-decoder structure, and directly performs raw pixel regression of masked image patches. SimMIM (Xie et al., 2022b) allows hierarchical transform- ers such as Swin (Liu et al. 202T) to be directly applied to MIM. MaskFeat (Wei et al. 2022a), Data2vec (Baevski et al. (022), BootMAE (Dong et al. (022). sDAE (Chen et al. (022c) explored the choice of reconstruction targets. However, existing MIM methods reconstruct all patches equiv- alently, which is obviously suboptimal. Several inpainting works explored the dynamic loss design (Pathak et al.] 2016 Yeh et al.] [016)] Yu et al. 2018: Wang et al.) (2018), ,, but there is essential dif- ference between MIM and inpainting. Inpainting focuses on the quality of the generated image, while MIM focuses on the representation of the encoder obtained from pretraining, Our DDAE is a tokenizer-free framework that focuses on dynamic reconstruction different pretraining phases.
Deep Supervision. Deep supervision methods are proposed to accelerate model's convergence and alleviate the problem gradient vanishment. Through applying aux layers to transmit the super- vision signal to the shallow layers. deep supervision has been used in carly classification models (Szegedy et al., 2015 Lee et al., 2015) and extended to other visual recognition tasks (Xie & Tu 2015| et al. (0066//Zang et al. (018b) 8b) Mosinska et al., 2018 /Zhang et al. 2018a). Despite these advances, modern CNN classification models rarely use auxiliary classifiers since directly ap- pending simple auxiliary classifiers top of the early layers of modern network hurts its perfor- mance (Huang et al. 2017). Several works utilize deep supervision to enhance the network through knowledge distillation et al., 2019 Zhang et al. 2019a). Zhang et al. (2022) proposed Con- trastive Deep Supervision to use contrastive learning signals for intermediate layers, but the signals of intermediate layers are still the same. Different from them, we introduce inconsistency in the
3

---

Under review as a conference paper at ICLR 2023
Enooder
Regresion
Regmosdon
Decodee
Encoder
Tiepe
Thapel
Tiept
Toagt
EMA Updale
Distillaton Decoder
Dynomie Dictilation
Momeotum Encoder
Vedile Tobans
Deroder Ourue
Mask
Distance Map
(a) Overall Framework of DDAE
(b) Deep Dynamic Loss
Figure 1: (a) Overall framework DDAE. Only visible patches fed into the encoder while full patches are fed into the momentum encoder. We perform raw pixel regression and feature self- distillation both with deep dynamic loss. (b)Deep dynamic loss. Lighter color indicates that the signal comes from the shallower layer of the encoder. Best viewed in color.
intermediate signals for the first time. To the best of our knowledge, this paper is also the first to apply deep supervision to the MIM task. State-of-the-art results show the great potential of deep supervision the application MIM.
METHODS
In this section, we firstly elaborate on the basic framework of MIM. Then introduce our pro- posed DDAE's two core designs termed Deep Dynamic Supervision and Deep Self-Distillation in Section 3.2 and Section respectively. Our framework is illustrated in Fig
3.1 PRELIMINARY
Formally, MIM firstly divides the input image X RRxxxCC into non-overlapping flattened patches x [x1, X2, X2, XN), where x; RPC according to the patch size P. and N (H x W)/P² is the number of patches. It then samples random binary mask m [m1. M2. ..., my), where m, E {0, 1} to mask portion of the flattened patches. The masked patches
- x m are discarded (He et al. 2022) or substituted by a learnable token [MASK] (Bao et al. 202I), and the rest patches Xı-Mr x C (1 - m) are used to reconstruct the dropped features or images to learn rich representations, The optimization target of MIM can be formulated as follow:
min E (Id (() [MASK] @ ml), XM) 0,0
(1)
X~D
where 3 denotes element-wise multiplication; fa(-) and d- are encoder and decoder respectively;
M(..-) .) is the similarity measurement, which varies in different works, e.g., la-distance in pixel space (He et al. 2022), perceptual loss in codebook space (Dong et al. (202I) self-distillation loss in feature space (Chen et al. 2022c). In our work. we use the Iı-distance as our measurement, M(a, b) = ||a bl 2 for both of the pixel reconstruction and self-distillation. To simplify the formulation, we ignore the mask token term and use d.(.) (-) to represent do([: [MASK] @ ml).
3.2 DEEP DYNAMIC SUPERVISION
Dynamic. Firstly, we define the difficulty reconstruction according to the distance between each masked patch and visible patches, generating distance map with distance transform D(-). For
4

---

Under review as a conference paper at ICLR 2023
60
62
64
66
64
10
03
64
66
64
14
80
02
0.4
10
(a) 50%
(b) 75%
(c) 90%
Figure 2: Distance map (normalized to [0, 1J) under different mask ratio. Darker color means larger distance from visible patches. High mask ratio leads to diverse distance for masked patches.
each masked token (m; 1), the distance transform assigns a number that is the Euclidean distance between that token and the nearest unmasked token in 2D space. Naturally, it is difficult recover patch that is far from visible ones and demands stronger semantic reasoning ability. On the contrary, reconstruction of a patch with visible ones nearby only requires fair local perception.
As shown in Fig 2 existing MIM methods often use a high mask ratio like 75% (which is proved to be critical to MIM's success (He et al. 2022)), ), values of the distance map vary diversely. Since the distance map is based on patches (14 x 14) rather than pixels (224 x 224), it is cost-free.
To guide the model to focus on different patches (corresponding to the requirement of distinctive properties) in different training phases, propose ha(+) to learn dynamic coefficient of the distance map to generate loss weight, directly applying dynamic focus to the loss weight corre- sponding patches. hg is derived follows:
exp(D(m;))) Em,=1 exp( (xp(D(m,)))
m, =
h(m,13)
(2)
m =0
We re-scale the loss weight map into [0, 1] according to the max value of the weights. When 3 0. larger distance leads to greater loss weight, so the model pays more attention to the reconstruction of difficult patches. Increasing the value of 3 can exacerbate this trend. Conversely, when 3 O, larger distance leads to smaller loss weight, and decreasing 3 results in more importance attached to simple patches. Along with the changing of B, the model dynamically focuses patches with diverse degrees recovery difficulty.
Deep supervision. As commented in the introduction, the reconstruction of patches with diverse difficulties requires distinct characteristics of the model, and layers at different depths naturally learn features different levels. Therefore, we apply dynamic loss the intermediate layers. Taking the standard VIT with B blocks as the encoder, we divide the blocks into K groups and extract the features the end of each group. For example, we K 4, then we extract output features of block 3, 6, 9 and 12 and feed them into the decoder respectively to recover masked patches. K is also called supervisory number, since is equal the number of supervisory signals. Then the loss function of pixel reconstruction Lpizet is derived as follows:
K
Lpiret (0, 3) = Xaha, (m) @ ||da, (((a)) (x)-M) x
(3)
where f(0(-) denotes to the output features of block-í encoder, and gi B denotes to the group index; de, is the regression decoder; are the scale factors of each group, and set to [0.75, 0.75, 0.75, 1] by default. Note that of different layers are independent of each other.
R
Deep dynamic supervision does not introduce any additional structure, it can be seamlessly mi- grated to existing MIM structures. We discussed its universality in Section 4.2 where we migrated to the representative MIM methods of one-stage and multi-stage.

---

Under review as a conference paper at ICLR 2023
3.3 DEEP SELF-DISE DILL-DIITILLATION
In addition to raw pixel regression, the layer-by-layer correspondence between intermediate fea- tures is more suitable for the design of deep dynamic supervision. Therefore, we designed Deep Self-Distillation based on BootMAF (Dong et al.) (022) to further strengthen the model through high-level semantic features. Specifically, the momentum encoder provides the features of masked patches of each layer as the target of self-distillation, so that the intermediate features of cach layer corresponding to the encoder have their own targets of self-distillatio.. The momentum encoder is updated by the encoder using the exponential moving average method (EMA). Formally, denoting the parameters of the encoder so as 0 and those of the momentum encoder før 0' we update 0 by:
- m0 + (1 - m)0
Here [0. 1) is momentum coefficient and set to 0.9999 by default. Note that 0' are not updated by proppprppaatioon but by equation 3 Jater each training step. The feature self-distillation also uses deep dynamic supervision as pixel regression. Note that the regression of raw pixels is one-vs-all, the features of all layers will be reconstructed by one regression decoder. The feature self-distillation is all-vs-all, which means the shallow features of the encoder are self-distille by the corresponding shallow features the momentum encoder, and the deep features are self-distilled by the corresponding deep features. Pixel regression and feature self-distillation use separate decoders, which both consist of two-layer transformer blocks. Since the decoders are light-weighted, the additional cost brought by deep supervision is acceptable. Now the deep self-distillation formula is:
K
Laiistai (0, = Σ
Xah (m) @ I|dea (90 (x1 (g.) (xM))1?
(5)
where fo(-) and for (-) are encoder and momentum encoder respectively; dæ is the distillation de- coder; À are the scale factors of each group, and set to [0.75, 0.75, 0.75, 1] by default. We add an L2 regularization term constrain the magnitude of Since L2 norm reaches minimum under uniform distributions, the L2 regularization loss here is to ensure that the loss mask does not deviate too far from uniform mask, this can effectively prevent beta from being too extreme(in which case The the model only focus on the hardest or simplest patches). The overall loss function is:
L Liizet + Liistill + A
Σ 1/g (m)ll2
(6)
where is the scale factor to tune the L2 regularization term and is to 0.5 by default.
4 EXPERIMENTS
4.1
IMPLEMENTATION
We conduct experiments on ImageNet- 1K without labels as the pretraining data for self-supervist learning. The input resolution is set as 224x224 during pretraining and partitioned into 16x 16 size patches. We pretrain the standard ViT small and base architectures, i.e., ViT-S/16 and ViT-B/16. The pixel regression decoder and feature distillation decoder both consist along with an extra linear projection head for predictions. The dimension for the pixel regression decoder is 512 while for feature distillation decoder is set the same as the encoder. We use block- wise masking with ratio of 75%. The data augmentation is only standard random cropping and horizontal flipping. All /3 are initialized as -0.5 by default. The pretraining settings are almost the same as BootMAE (Dong et al. 2022) (See Appendix [ for details).
transformer blocks,
4.2 IMAGE CLASSIFICATION
We evaluate both tie-ttning accuracy and linear probing accuracy on ImageNet- 1k. Table T the comparison with previous state-of-the-at MIM-based methods including one-stage and two- stage. As a tokenizer-free framework, our DDAE acheives consistent advantages both in a short schedule and longer schedule. In particular, with only 100 epochs pre-training. DDAE can achieve
presents

---

Under review as a conference paper at ICLR 2023
Table 1: Image classification accuracy (%) comparison ImageNet-1K of different methods with ViT-B as backbone. We report the fine-tuning and linear probing accuracy and our method DDAE outperforms previous self-supervised methods.
Methods
Extra Tokenizer
Pretraining Epochs
Finetune
Linear
DINO (Caron al.
2021
contrastive
300
82.8
78.2
MoCo-v3 (Chen ct al. 2021
contrastive
300
83.2
76.7
SplitMask NEl-Nouby
2021
w/
300
83.6
N/A
CAE (Chen ct al. b022b
300
83.6
64.1
MaskFcat (We1 cf al. 2022a
w/o
300
83.6
N/A
BootMAE (Dong cf al. 2022
w/o
100
83.0
N/A
SdAE CCen ct al.
2022c
w/o
100
83.5
60.3
(DAE(ours)
w/o
100
83.5
60.3
BEiT Bao et al.
2021
w/
800
83.2
56.7
MAE
He al.
2022
w/o
800
83.4
64.4
MAE
He al.
2022
w/o
1600
83.6
SimMIM XTe et al.
2022h
w/o
800
83.8
56.7
CAE (Chen ct al. (022bb
w/
1600
83.9
70.4
Mask Feat (Wei
2022a
w/o
1600
84.0
N/A
BootMAE (Dong et al.
2022
w/o
800
84.2
66.1
SdAE (Chen et al. 2022c
w/o w/o
800
84.0
N/A
(Durs(ours)
800
84.3
67.7
comparable performance with MAE using 1600 epochs pre-training and surpass 800 epochs pre- trained BETT. Furthermore, with a longer pretraining schedule, DDAE achieves 84.3% top-1 accu- state-of-the-ard
ImageNet- IK among one-stage self-supervised methods. Note that despite SdAE reaching comparable results with DDAE when pretrained for 100 epochs, method outperforms it with + 0.3% gains with 800 epoch schedule, demonstrating our consistent advantages both in a short schedule and a longer schedule.
racy, developing
In terms linear probing, our approach surpasses the above MIM methods with the same training epochs, but is not as good as the contrastive-bate methods. Contrastive-bass methods compare across images while MIM-based methods exploit the whole image structure, which may care about more than 1000 classes (Chen et al., 2022b). This phenomenon is also reported in MAE (He et al.) 2022] and BEiT (Bao al. (2021), so for MIM-based methods, fine-tuning measurement may better metric to validate their effectiveness.
4.3 ANALYSIS OF DDAE
Migrating Deep Dynamic Supervision Existing MIM methods. Although for simplicity we build our framework one-stage approach, our core design, Deep Dynamic Supervision(DDSS) compatible with other existing MIM methods. To demonstrate that, we conduct experiments to migrate DDS into representative ote-stage and two-stage MIM methods, MAF (He et al. (202)) and BEÎT-v2 (Peng et al. 2022a) respectively. We extract intermediate features of the encoder, then feed them into decoder together with the encoder's final output. The reconstruction targets are all set the same as the final output. Reconstruction losses for intermediate layers are multiplied by their corresponding dynamic loss weight as mentioned in Section [3.2] The plug-and and-play results reported in Table demonstrates our design's generality for both one-stage and two-stage approaches.
More local priors for ViTs. We conjecture that the fine-grained task built by MIM enables ViT to pay more attention to local information besides using global semantic inference, which inherently makes up for the absence of local prior in the structure ViTs. We depict the average attention dis-
Table 2: Plug-and-pla -ndy Experiments.
Methods
Extra tokenizer Pretraining Epochs
Finetune
MAE (He et al. 2022
w/o
300
82.9
DDS-MAE
w/o
100
83.2
BEÎT-v2 |eng et al. 2022a)
w/
100
83.9
DDS-BER -V
100
84.1
7

---

Under review as a conference paper at ICLR 2023
Figure 3: The attention map averaged over 12 attention heads between the class token and the patch tokens in the shallow layer (Block 3) of the ViT-B encoder pretrained ImageNet- IK. Top: Input image, Middle: MAE, and Bottom: our DDAE. We can see that DDAE has formed effective target percention attention in the verv earlv stage. while MAE still behaves to be rudimentarv.
Attention Distance
100
0000000
100
0 3000
80
80
60
60
40
20
20
0-
12
12
(a) MAE
(b) DDS-MAE
Figure 4: The averaged attention distance in different attention heads (dots) w.r.t the layer number on MAE (a) and DDS-MAE (b) with ViT-B as the backbone.
tance of vanilla MAE and DDS-MAE to demonstrate our design's effectiveness in providing more local priors in deep layers. Specifically, we compute averaged attention distance in each attention head of each layer (Dosovitskiy al.] (2000). We conduct experiments based on MAE since it is a simple enough MIM framework. As illustrated in Fig 4 DDS remarkably reduces the average attention distance of the model deep layers. This phenomenon indicates that through deep dy- namic supervision the model focuses more on local vicinity. thus obtaining more local priors in deep layers. Moreover, for DDS part, different heads in relatively deeper layers behave more di- versely, which may also contribute to DDS's s effectiveness (+ 1.22% gains over MAE on ViT-S and more than + 0.3% gains on ViT-B).
Analysis on dynamic curve of 3 and attention map, We depict the curve of dynamic coefficients changes during training for different layers in Fig For all intermediate layers, 3 drops sharply at the beginning, then rises slowly the training progresses. A smaller 3 value means that the model pays more attention to simple patches which are close to visible ones. Therefore, in the overall trend, each layer attaches more importance to the reconstruction of simpler patches and gradually pays more attention to difficult ones. We further show the shallow layer's attention map of MAE and DDAE in Fig Our DDAE forms target perception attention effectively in the very early stage, while MAE still behaves to be rudimentary.
A Alltioo on Deep Dynamic Supervision. We conduct ablation studies analyze the contributions of each component in Deep Dynamic Supervision. The models are finetuned on ImageNet- IK for 100 epochs. We ViT-S encoder and pretrain for 100 epochs. Table that either ablation degrades the performance, suggesting that both designs benefit the training dynamics.
Table 3: Ablation study Deep Dynamic Supervision. Here D-MAE means applying Dynamic Loss to MAE, DS-MAE means applying Deep Supervision. VIT-S is used as backbone.
Methods
Deep Supervision
Dynamic Loss
Finetune
MAE (He et al. 2022)
78.7
D-MAE
78.9
DS-MAE
79.7
DDS-MAE
79.9
8

---

Under review as a conference paper at ICLR 2023
for Block
for Block
0.475
-0.440
6 Eor Block
0,300
0.525
-0.450
-0.50
B Jor Block
-0.75
P
-0600
Block
-0.400
20
40
80
160
40
50
60
Epoch
Epoch
Overall Trend
(b) Zoomed-in Trend
Figure 5: 3 dynamic curve for intermediate layers. (a) is the overall trend from epoch epoch
100; (b) is the zoomed-in trend. Larger 33 means more importance is attached to difficult patches.
Table 4: Ablation study Deep Solf-Disiillatin. Here v means applying DDS on that part.
Methods
Pixel Regression
Feature Self-Distillation
Finetune
83.0
83.2
83.3
DDAE
83.5
Ablation on Deep Self-Distillation Then, study Deep Self Salf-Distillation in DDAE, conduct ablation experiments with ViT-B as encoder, pretrain for 100 epochs and finetune 100 epochs on ImageNet- IK. Results in suggess that applying DDS on Pixel and Feature both can improve the feature representation learned.
4.4 SEMANTIC SEGMENTATION
We evaluate the learned representation of DDAE on the ADE20K benchmark (Zhou et al. 2019
which consists of 25K images and 150 semantic categories. We use the UperNet (Xiao et al. 2018) task layer for semantic segmentation. We train Upernet 160K iterations with single-scale inference. The results are reported in Table with mean Intersection of Union (mloU) as the evaluation metric. Compared with previous state-of-the-an self-supervised models, our proposed DDAE outperforms all the one-stage methods, further validating the effectiveness of our framework.
Table 5: Semantic segmentation mloU (%) comparison on ADE20K. VIT-B is used as backbone.
Methods
Extra Tokenizer
Pretraining Epochs
mloU
BEiT Bao al.
202
w/ w/o w/o 2022b
800
45.6
SimMIM Xie ci al.
800
46.9
MAE
He et al.
2022
1600
48.1
PeCo Dong et al.
2021
w/
800
48.5
CAE
Chen al.
2022b
w/
800
48.8
SdAE (Chen et al.
(202cc)
w/o
800
49.0
BootMAE (Dong et 20222)
2022
w/o
800
49.1
DDAE(ours)
w/o
800
49.3
CONCLUSION AND LIMITIATIONS
In this paper, we propose a new framework DDAE with two core designs. (1) We present deep dynamic supervision enable dynamic focus on different patches through pretraining, providing progressively richer representation. (2) We propose to self-distill intermediate features of momen- tum encoder with dynamic supervision, which enhance the representation from a high semantic level. More importantly, we find our approach can bring more local priors for ViTs in deep layers, which we empirically find crucial for MIM's success. Although our DDAE is effective, the corre- lation between difficulty and distance is less relevant for background patches. How effectively distinguish foreground and background an unsupervised way and apply different weights a problem that needs to be further explored.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning speech, vision and language. arXiv preprint arXiv:2202.003555 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint (7Xiv:2106008254. 2021.
Josh Beal, Eric Kim, Eric Tzeng, Dong Huk Park, Andrew Zhai, and Dmitry Kislyuk. Toward transformer based object detection. arXiv preprint arXiv:2012.099 09955, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey 7agoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Qiang Chen, Qiman Wu, Jian Wang, Qinghao Hu, Tao Hu, Errui Ding, Jian Cheng, and Jingdong Wang. Mixformer: Mixing features across windows and dimensions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2022a.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self sulf-supe representa- tion learning. arXiv preprint arXiv:2202.0300 2022b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self- supervieeri vision transformers. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp- 9640-9649, 2021.
Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. arXiv preprint arXiv:2208. 00449, 2022c.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34:9355-9366, 2021.
MMSegmentation Contributors. Mmsegmentation, an open source semantic segmentation toolbox, 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transform- arXiv preprint arXiv:21 127 2021.
Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining.. arXiv preprint ar arXiv:2207.07111 07.1000701611.... 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Mathhis Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2/12.100 10740, 2021.
10

---

Under review as a conference paper at ICLR 2023
Mann-Batteen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Gao Huang. Danlu Chen, Tianhong Li, Felix Wu, Laurens Van Der Maaten, and Kilian Q Wein- berger. Multi-scale dense networks for resource efficient image classification. arXiv preprint arXiv: 1703.009444 2017.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pp. 1725-1732, 2014.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint wrXiv:1412.6900 2014.
Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- supervised nets. Artificial intelligence and statistics, pp. 562-570. PMLR, 2015.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022 2021.
lly Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
1rXiv:1711.05101 2017.
Agata Mosinska, Pablo Marquez-Neila, Mateusz Koziński, and Pascal Fua. Beyond the pixel-wise loss for topology-aware delineation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3136-3145, 2018.
Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estima- tion. In European conference on computer vision, pp. 483-499, Springer, 2016.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue. Trevor Darrell, and Alexci A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 367-376, 2021.
Zhiliang Peng, Li Dong, Hangho Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022a.
Zhiliang Peng, Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprini arXiv:2208. 06366, 2022b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8766 PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine PMLR, 2021.
Learning, pp.
11

---

Under review as a conference paper at ICLR 2023
Dawei Sun, Anbang Yao, Aojun Zhou, and Hao Zhao. Deeply- supervsueer knowledge synergy- In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 6997-7006, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021b.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 568-578, 2021.
Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiay Jia. Image inpainting via generative multi-column convolutional neural networks. Advances in neural information processing systems, 31, 2018.
Chen Wei, Haoqi Fan, Saining Xie, Yuan Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14668-14678, 2022a.
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: Multimodality-guine visual pre-training arXiv preprint w:Xiv:2203.05777 2022b.
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision. arXiv preprint arXiv:2006.03677 2020.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22-31, 2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings the the European conference on computer vision (ECCV), pp- 418 2018.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE international conference on computer vision, pp. 1395-1403, 2015.
Zhenda Xie, Zigang Geng, Jingeheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. arXiv preprint arXiv:2205.113. 13543, 2022a.
Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653 9563,9663, 2022b.
Baosong Yang, Longyue Wang, Derek Wong, Lidia Chao, and Zhaopeng Tu. Convolutional self- attention networks. arXiv preprint arXiv:1904.03100 2019.
Raymond Yeh, Chen Chen, Teck Yian Lim, Mark Hasegawa-Johnson and Minh N Do. Semantic image inpainting with perceptual and contextual losses. arXiv preprint vrXi::1007007539 2(3), 2016.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 2018.
12

---

Under review as a conference paper at ICLR 2023
Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang. Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-tokees vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 558-567, 2021.
Hanyuan Zhang, Hao Wu, Weiwei Sun, and Baihua Zheng. Deeptravel: neural network based travel time estimation model with auxiliary supervision. arXiv preprint arXiv:1802.02144 2018a.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019a.
Linfeng Zhang, Xin Chen, Junbo Zhang, Runpei Dong, and Kaisheng Ma. Contrastive deep super- vision. arXiv preprint urXiv:2207.05300 2022.
Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-vae: A variational autoencoder for directed acyclic graphs. Advances in Neural Information Processing Systems, 32, 2019b.
Zhenli Zhang. Xiangyu Zhang, Chao Peng, Xiangyang Xue, and Jian Sun. Exfuse: Enhancing feature fusion for semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 269-2844 2018b.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from -equeucc-t--scqueece perspective with transformers. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 6881-6888 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3)330--321, 2019.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection, arXiv preprint arXiv=2010.04159, 2020.
13

---

Under review as a conference paper at ICLR 2023
A TRAINING DETAILS
ImageNet Pretraining. We use Adam (Kingma & Ba 2014) as the optimizer and train the DDAE for 100/800 epochs to illustrate both the rapid convergence and higher upper limit. The learning rate set as 1.5e * batchsize, 256 with cosine learning rate decay and warmed up for 20/40 epochs respectively. The total batch size is as 4096. The weight decay is 0.05. We increase the momentum parameter from 0.999 to 0.99999 in the first 400 epochs. The mask ratio 75%.
ImageNet Fine-tuning, We follow the fine-tuning protocol in BEiT to use layer-wise learning rate decay as 0.75/0.65 for ViT-S/ViT-B respectively. The batch size is 1024, the warmup epoch is 5 and the weight decay is 0.05. For ViT-S, train 100 epochs with learning rate 1.6e-2, For ViT-B, We train 100 epochs with learning rate 8e-3.
Linear probing. We use the LARS (Karpathy et al., 2014) optimizer with momentum 0.9, We train the model for 90 epochs with a batch size of 16384. The warmup epoch is 10 and the base learning rate is 0.1. We set weight decay as zero.
ADE20K Semantic segmentation. We use UperNet (Xiao et al. 2018) based on mmsegmentation (Contributors] 2020). Following BEÎT, we use Adam (Loshchilov Hutter 2017) optimizer with initial learning rate 4e weight decay of 0.05 and batch size of 16 for 160k iterations. We warm up the learning rate for 1500 iterations and use a linear decay strategy. We set the layer decay as 0.65. Since ViTs can not output hierarchical features, we utilize four different scales FPNs to scale the feature map into different sizes. The input resolution 512 x 512. We report single-scale test results.
B MORE ABLATION STUDIES
Ablation on the supervisory number K. We further explored the impact of supervisory number K on performance. Specifically, we set K to be 1, 2, 4, and 6 respectively. The method to determine the block index is the same as that described in Section 3.2 As shown in Table 6 the additional benefits brought by more than four supervisory signals have been saturated. So we set K 4 as default in our framework.
Table 6: Ablation study on supervisory numbers K. Note all supervisions use Dynamic Loss.
K
Block Index
Finetune
1
122
78.9
2
66, 12
79.5
4
[3, 6, 9, 12
79.9
[2, 4,
6, 8, 10, 12]
79.9
Ablation on the choice 33 We also explored the choice of B. Firstly, we fix with different initialization, then set 3 as learnable parameters. Table demonttaess that learnable 3 benefits most. We also draw the 3 curves with different initialization in Fig 6 the final values of and the relative relationship among block 3,6,9 tend to be stable and robust initialization. We take B = [-5.5 -0.,, 0.5, 0.5] as initialization in our experiments by default.
Table 7: Ablation study on the choice B. Fixed means will not be updated in the pretraining.
B
Finetune Initialization
Fixed
[0, 0, 0,
79.7
Fixed
1.055 -0.5, - 0.5, 00.5
79.5
Learnable
00.0.0.0
79.6
Learnabl
-0.1, -0., -0.7
79.5
Learnable
-0.1
79.7
Learnable
-05, -05, -0.5
79.9
14

---

Under review as a conference paper at ICLR 2023
-0.3
for Block
B
-0.4
Block
Black
-0.40
-0.5
-0:445
-0.6
-0.45
-0.7
20
Epoch
Epoch
(a) Smaller shallow layer
(b) Smaller for shallow layer - zoomed
0.430
for Block
for Block
-0.2
Block
-0.45
6 for Block
-0.5
-1.450
20
40
60
80
100
40
50
Epoch
Epoch
(c) Smaller for deep layer
(d) Smaller for deep layer zoomed
Figure 6: B dynamic curve for different initialization. (a) Smaller values for shallow layer; (b) Smaller values for shallow layer zoomed in; (c) Smaller values for deep layer; (d) Smaller values for deep layer zoomed in. Different initialization ends with similar values of 3 and relative rela- tionship among different blocks.
C MORE VISUALIZATIONS
Our DDAE can form target-aware attention a very early stage (Block which sharp contrast to MAE. Here we show more shallow attention maps of DDAE and MAE in Fig. 7
15

---

Under review as a conference paper at ICLR 2023
Figure 7: More attention map in the shallow layer (Block 3) of the ViT-B encoder DDAE and MAE. The attention map is averaged over 12 attention heads between the class token and the patch tokens of the ViT-B encoder pretrained on ImageNet- Top: Input image, Middle: MAE, and Bottom: our DDAE.
16


Reviews:
[{"review_id": "tizwWVs80Sq", "reviewer": "Reviewer_x4N6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposes to introduce deep supervision into masked image modelling. Technically, there are two loss functions: the first one is to reconstruct pixel densities using tokens produced by intermediate transformer encoders; the second one seems to be enforcing the transformer encoder outputs to mimick those of corresponding outputs of an EMA version of the encoder (although I am not very sure about the details due to clarity issues). The primary selling point is to use a transformed (by equation 2) distance map that weights both losses so that patches with different distances to visible ones get different weights. This technique is considered 'dynamic' by the authors, in the tile, as the concentration parameter beta is learned in a layer-wise manner. Evaluations are done with downstreaming ImageNet fine-tuning and linear classification and an ADE20k fine-tuning experiments, reporting marginally improved performance. Codes are not provided or promised.\n\n[Post-rebuttal]\nThe method is conceptually novel and has some value. \nThe significance is left for experts on large-scale pre-training to judge.\nThe major issue is unclear presentation. Although I can guess how to implement it from these vague descriptions, people who are not familiar with topics involved may have some difficulty in understanding this. This is sub-bar for an ICLR paper, in terms of clarity.\nAfter several rounds of communication, the authors correct some factual errors (1-M v.s. M), and typos. They also tune-down some unjustified interpretations that may be misleading for the community. Remaining issues include:\n\n(1) The behavior of the regularization term in Equation.6 lacks clarity. The authors give an intuitive example in the response, which is not included into the paper. If included, this raises the clarity of this part above the bar. But I still recommend an analysis, as mentioned before: 'The impact of this regularization needs more (theoretical and experimental) analysis' in the first round and 'How gradients propagate to beta, through h() is not clear to me and this needs an analysis' in the second round. This is not a difficult thing, which only needs some gradient analysis and I don't understand why authors insist it is not necessary.\n(2) The training loop. After some clarification, I know that the implementation is aligned with my guessing. But the paper still lacks clarity. Including a training loop (like Algorithm 1 in the MoCo paper) is not difficult and I dont' understand why authors insist it is not necessary.\n\nAlthough it is disappointing that the authors still ignore my sincere suggestions on improving the clarity for a larger audience that are not familiar with these techniques, I have raised the recommendation to BA.", "summary_of_the_review": "The idea is fine, but the current version lacks in clarity. As for significance, I am not fascinated and others with hands-on experience in pre-training may have a better judgement.", "strength_and_weaknesses": "Strengths:\n+ I am familiar with and like deep supervision and distance transforms. They are techniques specific to computer vision due to their links to hierarchical representations and spatial affinity. This idea (if better developed) is something that I would like to cite and follow. Introducing deep supervision into masked image modeling is novel to my knowledge. Using distance transforms to weight training is a novel add-on too and makes sense in this masked image modelling setting.\n+ Although the method is unclear in several places, I can somewhat figure out how to implement it as the modules are standard ones.\n+ Although lacking in significance, the experimental results are conclusively better than baseline.\n+ The attempt to reveal the underlying mechanism is appreciated (fig.3 and 4), although I am not fully convinced.\n\nWeaknesses:\n- I think the major issue is clarity:\n- The notations make me confused. In equation 1, we use x_{1-M} to reconstruct x_M. But in equation 2 and 3, it seems that the paradigm changes? For equation 2, I believe we should reconstruct masked-out patches x_M right? Correct me if I am wrong. For equation 3, I think it is possible to distill the features of masked-out patches or remaining patches. But since the equation does not contain decoder I guess the input should be remaining patches x_{1-M} ? Having that said, if the distillation is done with only the encoder, how is the 'distillation decoder' in Fig.1 used?\n- Reading section 3.3 makes me lost. There is no formal description of the EMA procedure or the training loop. And what's worse, the figure, the loss and the notation seem to contradict each other. The sentence 'Deep dynamic supervision is shared with raw pixel regression' brings much more confusion. What is 'shared with'? According to Fig.1 I guess the deep dynamic supervision term refers to the whole training objective?\n- I think the beta regularization is the key to the success. According to my own experience, training this kind of parameters is not easy. The impact of this regularization needs more (theoretical and experimental) analysis. Since beta can be either negative or positive, regularizing its L1-norm is not valid. So I can see the authors use h for regularization, but the behavior is not readily clear to me.\n\n- Although I think the method is novel from several points of view, I have to point out several missing references. In the 'Deep Supervision' related works section, the authors claim that making deep supervision work for modern architecture is not easy. I agree and I would like to point out that papers like [A][B], which are the referred 'Contrastive Deep Supervision' paper's baselines, have studied the problem. \n\n[A] Deeply-supervised knowledge synergy, CVPR 2019\n[B] Be your own teacher: Improve the performance of convolutional neural networks via self distillation, ICCV 2019\n\n- My last major concern is about significance:\n- Personally, I only care about whether these pre-trained models can help my group develop better down-streaming robotic scene understanding algorithms. I feel excited about CLIP, Swin or MDETR, but the performance improvement of this one seems too marginal. I don't think this is a strong enough pre-training paper for ICLR. Maybe other colleagues with hands-on expertise on large-scale pre-training can better assess the significance.\n\nI have other comments:\n- I doubt he central claim of 'locality inductive bias in deeper layers'. I am not convinced why we we should have local bias in late stages. Meanwhile, I cannot understand why the deep dynamic supervision design allows this to happen.\n- 'gradually reconstructed vicinity' I am not convinced. There is no technical modules that allow gradual reconstruction.\n- 'this paper shows another potential use of deep supervision to make the network more diverse' This is too vague and I am not convinced by this 'diverse' claim.\n- The online calculation of distance transforms of random masks brings computation overhead, which should be analyzed.\n- The better convergence claim should be supported by curves instead of two datapoints.\n\nSeveral capical issues:\n- The Experimental results\n- Dynamic Loss\n- The Reconstruction targets", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jr9Fj4f7WLx", "reviewer": "Reviewer_aYDP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new MIM framework that dynamically focuses on patch reconstructions based on the degree of difficulty (i.e., the nearby visible patches) during pre-training. Besides, this paper proposes to self-distill intermediate features from the momentum encoder. Experiments show that it outperforms previous self-supervised methods on ImageNet for image classification and on ADE20K for semantic segmentation.", "summary_of_the_review": "Overall, I think this paper is a borderline paper and slightly below the acceptance threshold. I would like the authors to provide more justifications to their motivation to further judge this paper.", "strength_and_weaknesses": "Pros:\n\n- Interesting exploration to improving MIM by providing more local priors.\n- State-of-the-art performance among tokenizer-free methods. \n\nCons:\n\n- The motivation of this paper is not convincing. The reconstruction difficulty of patches are not necessarily correlated with distance from visible patches. For example, it may depend on how much information is included in the patch. The background patches will still be easy to reconstruct following regular patterns even if it is far away from visible patches, but the foreground patches (e.g., nose of a dog) are hard to guess and require stronger semantic reasoning.\n\n- The empirical improvement is too marginal. This is no clear advantage of the proposed method compared to SdAE(Chen et al., 2022c) according to the experiments. It has the same accuracy as SdAE for 100 epochs and only outperforms SdAE by 0.3% for 800 epochs.\n\n- This paper is not well-written. The introduction is too redundant and messy. Figure 1 is hard to follow. The caption is not self-contained.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PHr3FOLew7", "reviewer": "Reviewer_pPUj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduce a simple mechanism to allow the training of MIM model tacking the reconstruction difficulty into account. It also introduce deep supervision that can help the diversify the feature representation across different layers. Good performance are reported to justfiy its methods. ", "summary_of_the_review": "The presented approach is simple yet effective as shown in their benchmark results. The analysis is in general good. Thus I am leaning to accept. \n\nPost rebuttal:\nI decided to down-rate the score after the reviewer discussion. Although the core idea of weighting patch reconstruction is inspiring, the improvement from dynamic loss is comparably marginal compared to the deep supervision, which makes the current paper writing less proper as the majority discussion is focused on dynamic loss. Given the current reviewers opinions, I recommend the authors to rephrase the paper writing or provide stronger results to validate dynamic loss and submit to another venue. ", "strength_and_weaknesses": "Strength:\n- The intuition that reconstructing patches has different levels of difficulty makes a lot of sense. The proposed method is also simple, and the required modifications on top of the MAE framework are quite minimal. \n- The experimental results are solid. \n- The analysis of locality(eg. Fig 4) is interesting and inspiring. \n\nWeakness:\n- The author claims that deep supervision introduces diversification and mentions the independence of beta_i. However, the Fig 5 shows that the beta_i are highly correlated and mostly the same. The difference in beta_i as shown in zoom-in figure in Fig 5 right plot seems too brittle to support the hypothesis that lower layer focus on more simple patches and deeper layers focus on more difficult patches. Also, why is the beta curve for block 12 missing from Fig 5?\n- More discussion of how the initial beta is chosen is needed.  The paper initialize beta to -0.5 and the during the whole training beta varies from -0.6 to -0.425, very close to the initial value. What will happen if you initialize beta differently? \n\n \nOther comments\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LL0C4K-5id", "reviewer": "Reviewer_VeHD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designs a deep dynamic supervision mechanism that can be migrated into existing MIM methods.\nIt proposes to dynamically focus on patch reconstructions with different degrees of difficulty at different pretraining phases and depths of the model. \nFurther experiments demonstrate the effectiveness of proposed method.", "summary_of_the_review": "Overall, this paper is valid. However, the idea seems to be incremental. This prevents me from further increasing the score. \n\nAfter reading the rebuttal, I still feel that this paper lacks enough novelty and the main components proposed yeild minor improvements.  \n\nHence, I downgrade my rating and would not recommend to accept. ", "strength_and_weaknesses": "Strength\n+ The performance is good\n+ Detailed experiments demonstrate the effectiveness of each component \n\nWeakness\n+ The idea seems to be incremental. Add deep supervision is straight-forward and should be expected to work.\n+ It lacks motivation on using both of distillation decoder and regression decoder. It seems minor improvement when combing them. What is the experiment variance? \n+ Using extra tokenizer is not necessary a bad thing. Will proposed method be further improved when use extra tokenizer?\n+ What is the improvement on down-stream fine-tuning when compared to MAE?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4zBX7PWQdg", "reviewer": "Reviewer_A9rs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two practical regularization techniques for MAE training and the related quantitative results verify their efficacy. One is to exert different weights to the reconstruction loss of different patches based on how difficult it is to reconstruct these masked ones with visible ones. Another is to employ a momentum encoder to regularize the learned intermediate features. Experiments show these two designs are effective and complementary. They can accelerate training and further improve baseline performance, evaluated on two tasks, e.g., classification and segmentation.", "summary_of_the_review": "This paper gives two simple designs, a dynamic loss and deep supervision, to improve the current MAE training and downstream performance. These two designs are widely researched in inpainting and contrastive learning topics, and they perform quite well on MAE, especially on classification and semantic segmentation. The motivations are well-described and well-supported by the experiments. I believe it gives good practices for the development of MAE.", "strength_and_weaknesses": "Strengths:\n1) It gives a strong motivation for the proposed two designs. Applying spatially variant reconstruction losses to different masked patches based on their distances from the existing ones is a sound technique. The momentum encoder is also a typical method to improve representation learning.\n2) It conducts extensive and solid experiments. Classification and semantic segmentation are conducted along with several compared methods. Ablations are also sufficient.\n\nWeaknesses:\n1) Though dynamic loss is complementary to deep supervision (in Table 3), the sole improvement brought by dynamic loss is limited (only 0.2 compared to MAE) and the major benefits are provided by deep supervision. This experimental result weakens the key role of dynamic loss while the loss is stressed with the most pages. Undoubtedly, dynamic loss can accelerate training convergence, but it is expected to give better improvement if it is the most crucial part of this paper. Besides, such a dynamic loss design is a routine in inpainting literatures [1-4].\n> [1] Pathak, Deepak, et al. Context encoders: Feature learning by inpainting. CVPR. 2016.\n>\n> [2] Yeh, Raymond, et al. Semantic image inpainting with perceptual and contextual losses. arXiv preprint arXiv:1607.07539 2.3. 2016.\n>\n> [3] Yu, Jiahui, et al. Generative image inpainting with contextual attention. CVPR. 2018.\n>\n> [4] Wang, Yi, et al. Image inpainting via generative multi-column convolutional neural networks. NeurIPS. 2018.\n\n2) The analysis of the used momentum encoder is unsatisfying from my perspective. If we treat the masked input as a view of the input, then MAE can be considered as maximizing the similarity between the positive pair in contrastive learning. To this end, the usage of momentum encoder can be discussed in depth.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject