Masked Image Residual Learning for Scaling Deeper Vision Transformers
Authors: Guoxi Huang
Hongtao Fu
Adrian G. Bors
Venue: NeurIPS 2023 poster (2023)
Keywords: self-supervised learning
vision transformer
masked image modeling
TL;DR: a novel self-supervised learning method, called masked image residual learning, enables scaling deeper vision transformers.

Abstract: Deeper Vision Transformers (ViTs) are more challenging to train. We expose a degradation problem in deeper layers of ViT when using masked image modeling (MIM) for pre-training.
To ease the training of deeper ViTs, we introduce a self-supervised learning framework called  $\textbf{M}$asked $\textbf{I}$mage $\textbf{R}$esidual $\textbf{L}$earning ($\textbf{MIRL}$), which significantly alleviates the degradation problem, making scaling ViT along depth a promising direction for performance upgrade. We reformulate the pre-training objective for deeper layers of ViT as learning to recover the residual of the masked image.
We provide extensive empirical evidence showing that deeper ViTs can be effectively optimized using MIRL and easily gain accuracy from increased depth. 
With the same level of computational complexity as ViT-Base and ViT-Large, we instantiate $4.5{\times}$ and $2{\times}$ deeper ViTs, dubbed ViT-S-54 and ViT-B-48.
The deeper ViT-S-54, costing $3{\times}$ less than ViT-Large, achieves performance on par with ViT-Large.
ViT-B-48 achieves 86.2\% top-1 accuracy on ImageNet. 
On one hand, deeper ViTs pre-trained with MIRL exhibit excellent generalization capabilities on downstream tasks, such as object detection and semantic segmentation. On the other hand, MIRL demonstrates high pre-training efficiency. With less pre-training time, MIRL yields competitive performance compared to other approaches.

Masked Image Residual Learning for Scaling Deeper Vision Transformers
Guoxi Huang
Hongtao Fu
Adrian G. Bors!
Baidu Inc. huangguoxi @baidu.com
Huazhong University of Science and Technology m202173233 @ @hust.edu.cn
University of York
adrian.bors wor.ors@@orr....... york.ac.uk
Abstract
Deeper Vision Transformers (ViTs) are more challenging to train. We expose a degradation problem in deeper layers of VIT when using masked image modeling (MIM) for pre-training. ease training of deeper ViTs, we introduce a self- supervised learning framework called Masked Image Residual Learning (MIRL), which significantly alleviates the degradation problem, making scaling ViT along depth a promising direction for performance upgrade. We reformulate the pre- training objective for deeper layers of ViT as learning to recover the residual of the masked image. We provide extensive empirical evidence showing that deeper ViTs can be effectively optimized using MIRL and easily gain accuracy from increased depth. With the same level of computational complexity as ViT-Base and ViT-Large, we instantiate 4.5x and 2 deeper ViTs, dubbed ViT-S-54 and ViT-B-48. The deeper ViT-S-54, costing 3 less than ViT-Large, achieves performance par with ViT-Large. ViT-B-48 achieves 86.2% top-1 accuracy on ImageNet. On one hand, deeper ViTs pre-trained with MIRL exhibit excellent generalization capabilities on downstream tasks, such as object detection and semantic segmentation. On the other hand, MIRL demonstrates high pre-training efficiency. With less pre-training time, MIRL yields competitive performance compared to other approaches. Code and pretrained models available at: https://github.com// http://hhwb..cmmuuusssllluuuaaa.
Introduction
Transformer architecture [46] has become the de-facto standard in natural language processing (NLP). major driving force behind the success of Transformers in NLP is the self-supervised learning method called masked language modeling (MLM) [10]. MLM significantly expends the generalization capabilities of Transformers, with the underlying principle being very intuitive removing portions of sentence and learning to predict the removed content. Recent advancements in computer vision have been profoundly inspired by the scaling successes of Transformers in conjunction with MLM in NLP, successively introducing the Vision Transformer (ViT) [13] and masked image modeling (MIM) for training generalizable vision models. The concept of MIM is as straightforward as MLM; its pre-training objective is to predict masked image patches based the unmasked image patches, thereby capturing rich contextual information.
This paper first reveals that MIM can induce negative optimization in deeper layers of ViT. which not only constrains the generalization performance of ViT but also hinders its scaling along the depth dimension. Previous work [55, 37] suggests that deeper layers of VIT more properly pre-trained by using MIM, the conclusions of which contradict our observation. Another branch of work [6, 50, 4, 25, 37] tentatively suggests that, due the lack of semantie information in MIM, the shallower layers of ViTs are more effectively pre-trained than the deeper layers. In our preliminary experiments Sec. 2.2, we demonstrate that replacing the deeper Transformer blocks pre-trained by using MIM with randomly initialized blocks does not degrade performance, which supports our
Corresponding author. Work done when H. Fu intern Baidu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Loss
Decoder
Decoder
Decoder
Decoder
shallow
deep
shallow
shallow
deep
$801
Encoder
Encoder
Encoder
(a) MAR
(b) truncated MAE
(c) MIRL
Figure 1: Three MIM pre- pre-training schemes. For simplicity, the diagram omits the random masking
process. A complete ViT consists of shallow and deep parts. The dashed box indicates the part the model that is not involved in the pre- pre-training process.
81.75
vanilla MAE
trancated MAE
83.5
81,50
ViT-Base
82.9
$2.6
Accuracy
81.00
80.75
82.3
80.25
VIT-Small
81.7
vanilla MAE
trincated
80.00
$1.0
12
80.5
$1.0
82.0
82.5
83.0
833
Number of randomly initialized blocks
Accuracy (%)
Figure 2: Truncated MAE vs. MAE. The æ-axis represents the number of blocks replaced with randomly initialized blocks from the encoder's end after pre-training. ViT-S is used the en- coder to better observe the differences.
Figure Comparison MIRL, truncated MAE, and MAE. To maintain the same decoding putational cost, each decoder in the MIRL model contains blocks, while other models have 4 de- coder blocks.
statement. We hypothesize the negative pre-training effect enforced the deeper layers is due to depth restriction, which can be regarded as a degradation problem occurring in deeper ViTs.
We address the pre-training degradation problem in deeper layers of ViT by introducing Masked Image Residual Learning (MIRL) framework. We establish a multi-decoding process by segmenting the encoding blocks according their depth levels. Instead of letting the entire autoencoder learn to reconstruct the masked content, MIRL encourages the deep layers to Iearn latent features that are beneficial for recovering the image residual, distinct from the main image component. The diagram of the MIRL with 2 segments is illustrated in Figure 1c, where we divide the encoder into shallow and deep segments, and append separate decoder to each. The shallow segment learns reconstruct the main component of the masked content, while the deep segment is explicitly reformulated as learning the image residual. The MIRL framework is essentially equivalent shifting the pre-training objective of deeper layers of ViT from image reconstruction to image residual reconstruction. This simple yet effective concept image residual reconstruction significantly alleviates the degradation problem in deeper layers of ViT. making scaling ViTs along depth promising direction for improving performance. By extending the MIRL framework and dividing the encoder into more segments, as illustrated in Figure 4, we can train deeper ViTs and readily achieve accuracy gains from substantially increased depth. Consequently, we instantiate deeper encoders: ViT-B-24, ViT-B-48, and ViT-S-54, comprising 24, 48, and 54 Transformer blocks, respectively. Notably, with similar computational complexity, our deeper ViT variants deliver considerably better generalization performance than the wider ViT encoders (e.g. VIT-S-54 vs. VIT-B, ViT-B-48 ViT-L), thanks to the increased depth. Meanwhile, our experiments
Sec. 4.2 demonstrate that employing additional feature- level objectives [12, 6] or VGG loss [26, 11] can further improve performance, suggesting that the improvement directions of feature-level loss and MIRL are orthogonal and can complement each other.

---

Loss
random
masking
H
H
M
maii
res.
rosidual
visible patehes
M
visible token
M
masked token
encoding process
decoding
misin
Trsti it Tés
resicuaal
shorteus connection
M
Encoder Segments: 0 0 +
Decoders: Hi, Hy,
Figure 4: Example of the MIRL framework. The Transformer blocks in the VIT encoder split into four segments, D1, D2, 23, and Φ,. The output of each segment is then sent to its corresponding decoder. Shortcut connections are established between the shallower decoders H H2 and deeper decoders H4, H3. enabling deeper decoders to predict the masked image residual.
2
Preliminaries
2.1 Revisit masked image modeling
Following the paradigm of Masked AutoEncoder(MAA) [21], the input image split into a set of non-overlapping patches x = {x*) where x' E R PCC denotes the image patch in the i-th position with p2 resolution and channels. The image patches are further tokenized into visual tokens via linear projection, denoted by
[ == 2. Subsequently, random sampling strategy divides the indexes of patches into two subsets, V and M. where V n. M = 0. The image patches (x") and visual tokens (z)) with indexes V considered be visible to the encoder. Thereafter, the encoder with L blocks only takes as input visible tokens zy, and maps them to embedding features zY. The objective of MIM is to predict the unseen content from X by employing a decoder. learnable mask token M( is introduced after the encoder, which is placed in M masked positions. Then the full of encoded patches and mask tokens processed by small decoder to reconstruct the original image in pixels. The architecture of MAE can be described as:
z Fe(zy-1)
L..LL
2 (epa :iE M
(2)
Hu)).
where Fe(-) refers the 4-th Transformer block in the encoder, H-)) denotes a shallow decoder. The objective loss of MIM is given by
piixol
1
Σ
| x211
(4)
LM
P2C
iEM
where the reconstruction loss is only calculated in the masked positions.
2.2 A deep dive into autoencoders for MIM
We present three distinct autoencoder architectures in Figure 1 elarify motivation. Figure la shows the MAE framework [21]. Figure 1b depicts a truncated MAE with only the shallow VIT blocks in the encoder. Figure 1c presents simplified MIRL with two decoders connected the shallow and deep encoder segments. Models are pre-trained for 300 epochs with the same hyperparameters. Full setup details are Appendix A of the supplementary materials.
WW still perform the position embedding addition and class token concatenation processes as in VIT, but these steps omitted for notational simplicity.

---

Observation I: MIM pre-training can induce negative optimization in deeper layers of VIT. However, due the overwhelmingly positive pre-training effect that MIM bestows upon the earlier blocks. its adverse influence on the latter blocks remains undiscovered. Given a ViT encoder pre-trained with MIM (Figure la), we substitute the pre-trained weights in the latter blocks of the encoder with the random parameters and subsequently fine-tune the encoder on ImageNet-1 K. The curve plot of vanilla MAE, depicted in Figure 2, illustrates that applying random re-initialization to deeper blocks from the end of the encoder can improve performance. The initial point in Figure 2 indicates the result of an MAE model that has been pre-trained and subsequently fine-tuned without any random initialization. As more shallow blocks are randomly re-initialized, accuracy declines. Intriguingly, random initialization generalizes better than MIM pre-training in deeper layers of ViT, defying intuition.
Observation I:: Performing MIM pre-training on fewer layers can lead to better efficiency and effectiveness. A truncated MAE illustrated in Figure 1b requires less pre-training time than the vanilla MAE while still achieving better or comparable performance. For fine-tuning, we initialize a complete ViT model with pre-trained weights from the truncated MAE. Regarding the blocks that are not included in the truncated MAE, we apply random initialization. As shown in Figure 2, when truncating 3 blocks from the end of the encoder, the fine-tuned model has better accuracy than the rest ones. By using the truncated MAE, we only pre-train 4 blocks and achieve similar fine-tuning accuracy as the vanilla MAE, reducing pre-training cost by 66%.
Observation III: Learning to recover image residual is a more productive pre-training objective. A simplified version of MIRI., shown in Figure 1c, formulates the pre-training objective for deeper layers as learning image residual, promoting more vivid image detail recovery and imposing a positive pre-training effect on deeper layers. Figure 3 demonstrates that MIRL achieves the highest fine-tuning accuracy among the three MIM pre-training schemes.
In summary, Observations I and II expose a pre-training degradation problem in ViT's deeper layers, leading to sub-optimal solutions. The same issue is also observed in the BEiT [1] paradigm, potentially attributed to depth limitations in MIM. In Observation III, we introduce MIRL to alleviate the degradation problem in deeper layers. The rest of the paper demonstrates how we employ MIRL to tackle the challenges of training deeper Vision Transformers.
Method
3.1 Masked Image Residual Learning (MIRL)
Upon observing that deeper layers pre-trained by MIM underperform against those with random initialization, we infer that the weight parameters of these deeper layers have indeed been updated during MIM pre-training. but in an unfavorable direction. In contrast, the shallower layers demonstrate improved performance after MIM pre- pre-traini This leads us to speculate that the depth of the layers could be the root cause of the degradation problem.
To alleviate degradation in deeper ViTs during pre-training, we propose letting the deeper Transformer blocks learn to predict the residual of the masked image, rather than directly predicting the masked image itself. An overview of the MIRL framework illustrated in Figure 4. Specifically, the encoder is partitioned into multiple segments, with each segment being followed by a separate small decoder. Subsequently, we establish shortcut connections between the shallower and deeper decoders. We underscore that these shortcut connections constitute the core of our method. This configuration fosters a seamless collaboration between very shallow and deep Transformer blocks in corrupted image reconstruction: the shallower segment learns to reconstruct the main component of the masked image, while the deeper segment learns the image residual. During pre- pre-training, the established shorteut connections enable back-propagation to affect both the deeper and shallower layers simultaneously. This intertwined relationship between the deeper and shallower layers implies that the pre-training should either guide both towards a beneficial direction or lead both astray. With the introduction of the MIRL framework, our experimental results indicate that the shallower layers have, in essence, steered the deeper layers towards a more favorable direction.
Formally, we reformulate the encoder in Eq. (1) by evenly grouping the encoding Transformer blocks into G segments:
v2 = 2 (z 2g
g L..G,
(5)
4

---

where Py denotes stack of encoding blocks in the g-th segment. In the output of each encoding segment zy the masked positions are filled with shared masked token e[j]] denoted as u, = z U [eim] : E M) Subsequently, for the g-th segment and the (G-g 1i---+1)-tb segment selected from bottom-up and top-down directions, two separate decoders Hg and HG-s+1 are appended for feature decoding. Let consider that the g-th shallower segment learns mapping function 2g = (n)) producing the main component of the reconstructed image &g. Thereafter, we let E = Hc -+ g from the (G-g+1--th deeper segment asymptotically approximate the residual Eg = 8g. The objective loss Lgg for the g-th segment defined as:
1 M
Σ IEM
P
E312 -
M
1
Σ YEM
P
1'' -
(6)
Different from the residual learning in [23], image residual learning would not fit an identity mapping, considering that the inputs to the two segments are different. See Appendix B.2 for further discussions alternative form of Lg. The final loss function is formed by accumulating all 2
reconstruction loss terms:
Liotll = -Ezz1 XgG.
(7)
where Xg is the scaling coefficient, which is set to & by default. Additional pre-training objective losses, such the feature-level loss used in [6, 54] and the VGG loss [26]. can be employed to enhance performance. We provide the definitions of other loss terms Appendix B. However, indicated in the ablation study in Sec.4.2, incorporating additional loss terms introduces non- negligible overhead during pre- pre-training. By default, we solely utilize the per-pixel loss defined in (6).
Densely Interconnected Decoding (DID). We design densely interconnected decoding (DID) module, inserted into the decoders across different segments, enabling access to the features produced by previous segments. DID allows subsequent segments avoid relearning features already acquired in earlier segments, thereby enhancing representation diversity. Immediately following the self- attention module in the first Transformer block decoder Hg, we insert DID module, which ingeniously implemented using the Multi-Head Attention mechanism, MHA(Q, VVV:
DID 2/ 2 = BHMAAAA (1g, 1D-----., 20 (2) -...))))))
(8)
where the query Q is ug. while the key and value K, V are v 1y.,..... 2. The dense interconnection property of the DID module is rendered by the concatenation form in MHA's key K and value V where we concatenate the output from previous encoding segments, thereby enabling feature integration at different depth levels.
3.2 Scaling to deeper VIT
Without modifying the Transformer block design, we instantiate deeper VIT variants by simply stacking more blocks. We consider the embedding hidden sizes in the Transformer block of 384 and 768. The details of the deeper ViT's are provided Table 1. Choosing an appropriate hidden size is non-trivial for scaling up Vision Transformers, considering that larger hidden size (e.g., 1024 adopted ViT-Large and 1280 adopted ViT-Huge) can cause instability due to very large values in attention logits, leading (almost one-hot) attention weights with near-zero entropy. The instability of very wide ViTs also reported in [21, 9].
Table 1: Details of Vision Transformer scaling along the depth dimension.
Model
Depth
Hidden size
MLP size
Heads
ViT-S-54
54
384
1536
12
ViT-B-24
768
3072
12
ViT-B-48
48
768
3072
12
By utilizing MIRL,, we demonstrate that deeper ViTs exhibit stronger or comparative generalization capabilities when compared to their shallower and wider counterparts. With similar computational
5

---

VIT-8-48
85
VIT-8-24
Accuracy (%)
VIT-S-54
VIT-L
Accuracy (%)
84
VTTBB
VIT-B
84
VIT-S
82-
MIRL MAE
83
V-T-5-44
VIT-8-88
VIT-5
VIT-8
81
82
50M
100M 150M 200M
250M 300M 350M
800
1000
#parameters
pre-training epochs
Figure 5: Fine-tuning evaluation ImageNet versus model size. With a similar complexity, deeper ViTs outperform shallower ViTs. The
Figure 6: Training schedules. The performance of deeper ViTs has not yet plateaued even after 1000 pre-training epochs. We employ a step de- cay learning rate scheduler.
models are pre-trained for 300 epochs.
complexity, ViT-S-54 generalizes better than ViT-B. With only 31% of the computational cost of ViT-L, ViT-S-54 delivers performance on par with ViT-L. Notably, ViT-B-48 not only that it achieves higher performance but also provides a more stable training than VIT-L. This suggests that deepening VìT's could be a promising direction for enhancing vision model performance. Furthermore, MIRL helps to alleviate the training difficulties typically encountered in deeper ViTs, unlocking their potential and making them more effective for variety of downstream tasks.
4 Experiment
The proposed MIRL method is evaluated on image classification, object detection and semantic segmentation tasks. All models are pre-trained on ImageNet- I K and then fine-tuned in downstream tasks. The input size is 224 224, which is split into 196 patches with a size of 16 x 16.
Pre-training setup. We pre-train all models on the training set of ImageNet- with 32 GPUs. By default, VIT-B-24 is divided into 4 segments, while ViT-S-54 and ViT-B-48 are split into 6 segments, and others into 2. Each appended decoder has 2 Transformer blocks with an injected DID module. We follow the setup in [21], masking 75% of visual tokens and applying basic data augmentation, including random horizontal flipping and random resized cropping. Full implementation details are in Appendix A.
4.1 Instantiations of deeper ViT
We compare the performance of the deeper ViTs detailed in Sec.3.2 with the VIT instantiations presented in [13]. As illustrated in Figure 5, we can easily gain accuracy from increased depth by leveraging MIRL for pre-training,
particular, ViT-S-54, which has the same level of computational complexity as VIT-B but is 4x deeper than ViT-B, significantly outperforms ViT-B and even achieves performance on par with ViT-L pre-trained with MAE. Likewise, ViT-B-48 surpasses VIT-L while maintaining the same level of computational cost. Furthermore, the encoder pre-trained with MIRL consistently delivers higher performance than the one pre-trained with MAE.
4.2 Ablation studies
Various objective functions. Our approach is general masking modeling architecture, seamlessly complementing prior methods that propose various objective functions. We study the compatibility of the recent feature-level loss and perceptual loss (i.e. VGG loss) with our method. The results listed in Table 2a show that incorporating these objective functions can further improve the model, which demonstrates the generality of MIRL. Nonetheless, the additional loss terms will introduce heavy computational overhead. To accelerate our experiments, we have not used them by default.
Number of segments. The purpose of dividing encoder into more segments is to construct auxiliary reconstruction losses to facilitate the training of the intermediate layers. We observe that these auxiliary reconstruction branches amplify the gradient magnitude, potentially improving

---

Table 2: MIRL ablation experiments ImageNet-1K: 1K: We report the fine-tuning (ft) accuracy(%) for all models, which are pre-trained for 300 epochs. Unless specified otherwise, the encoder is ViT-B-24.
(a) Various objective functions. The en- coder is ViT-B.
(b) Numbers of segments (seg.) in the autoencoder of MIRL.
(c) Effect of Densely Intercon- nected Decoding.
objectives
MIRL
ft
time
#seg.
blocks per seg.
ft
model
DID
ft
pixel
83.5
1.0x
1
24
83.5
ViT-B-24
84.5
fixtt-ffeture
83.2
1.6x
2
12
84.3
ViT-B-24
/
84.7
pixcl+
83.6
1.8x
4
6
84.7
VIT-B-48
X 85,0
ViT-B-48
85.3
vig [26]
83.8
2.5x
6
4
84.6
(d) MIRL vs. simple multi-decoders Column A reports the performance gap between MIRL and multi-decoders.
(e) MIRL 13. coarse and fine separation.
model
depth
MIRL
A
multi multi-dccoders
method
ViT-B
ViT-B-24
ViT-B
83.5
0.3
83.2
MIRL
83.5
84.7
ViT-B-24
24
84.7
0.6
84.1
coarsc-to-fine
83.1
84.2
ViT-B-48
85.3
0.8
84.5
fine-to-coarse 82.9
84.2
Table 3: Image classification results on ImageNet-IK, IK. All models are pre-trained and fine-tuned with 224x224 input resolution. "IN" refer to ImageNet, while "FT" is the fine-tuning accuracy. "Epochs" refer to the number of pre-training epochs. The models pre-trained with extra data or very long schedules marked in gray. We report the best result in bold and the second best result(s) underlined.
Encoder
#params
FLOPs
Method
Training Data
Epochs
FT (%)
Supervised
INIK
82.3
MoCov3 [5]
INIK
300
83.2
BEiT [1]
DALLE25OMHINIK
800
83.2
SimMIM [56]
INIK
800
83.8
VIT-B
86M
16.8G
CIM [16]
DALLE250M-INIK
300
83.3
LocalMIM [49]
INIK
1600
84.0
MAE [21]
INIK
300/800
82.6/83.1
MAE [21]
INIK
1600
83.6
MIRL
INIK
300/800
83,5/84.1
ViT-S-54
96M
18.8G
MIRL
INIK
300/800
84.4/84.8
ViT-B-24
171M
33.5G
MIRL
INIK
300
84.7
Supervised
INIK
82.6
MaskFeat [52]
IN1K
1600
85.7
ConMIM [57]
IN1K
1600
85.5
VIT-L
307M
59.7G
HPM [48]
IN1K
800
85.8
MAE [21]
INIK
1600
85.9
MAE [21]
INIK
300/800
84,5/85.4 .4
ViT-B-48
341M
67.0G
MIRL
INIK
300/800
85.3/86.2
the optimization of deeper Transformer blocks. Table 2b reports the performance of MIRL with respect to various numbers segments. For ViT-B-24, the configuration with 2 segments shows lower accuracy than the with 4 segments. However, further splitting the encoder into more segments brings no more performance gain.
Effect of densely interconnected decoding. Considering that the auxiliary reconstruction branches from different segments adopt the same objective metric, the proposed DID establishes a feature reuse mechanism, preventing layers various depth levels from learning similar feature representations.

---

Table 2c demonstrates that the MIRL models embodying the DID module yield higher fine-tuning accuracy than those without the DID. Deeper models gain more advantages from DID.
MIRL. vs. simple multi-decoders. In order to demonstrate that image residual learning, powered by the shorteut connections, is the key effective training deeper ViTs, we construct a segmented autoencoder with multiple decoders, Unlike MIRL, each decoder in the multi-decoder model independently learns reconstruct the masked content. As shown in Table 2d, MIRL achieves substantially higher accuracy than the simple multi-decoder approach. Notably, the performance gap between MIRL and the multi-decoders widens as more Transformer blocks are stacked. When pre-training with multi-decoders the deeper ViT seems to gain accuracy from increased depth. However, this does not imply that the multi-decodes approach addresses the degradation problem. Since replacing the weights of its deeper layers with random weights does not lead to performance drop, the trivial improvement is attributed to the increased number of shallower layers.
MIRL vs. coarse and fine separation. As the reconstructed image residual shows some fine-grained details images, it intriguing know what pre-training results can be produced by replacing the reconstruction targets with the coarse and fine image components separated by using a Laplacian of Gaussian operator. We construct segmented autoencoder with multiple decoders, referred as "coarse-to-fine" in which the reconstruction targets of the shallower and deeper segments correspond the coarse and fine image components, respectively. "fine-to-caars"" denotes the reversed targets compared to the "coarse-tor to-feteoe""" configuration. Table 2e indicates that the segmented autoencoder with fine and coarse reconstruction targets achieves lower accuracy than MIRL, demonstrating that the main and residual components are not equivalent to the fine and coarse components.
Training schedules. So far, we have only trained our models using a relatively short pre-training schedule of 300 epochs. Note that deeper ViTs gain more advantages from longer pre-training schedules, compared shallower ViTs. We extend pre-training to 1000 epochs and record fine-tuning performance for various pre-training lengths. To resume pre-training from previous checkpoints, we use step decay learning rate scheduler, decaying the learning rate by factor of 10 at 90% and 95% of the specified pre- training length. Figure 6 shows that ViT-B tends to plateau after 800 pre- pre-training epochs, while ViT-S-54 keeps improving even after 1000 epochs. This implies that deeper ViTs' potential can be further unleashed by adopting a very long pre-training schedule, such as 1600 epochs.
4.3 Image classification on ImageNet-1K
We compare our models with previous results on ImageNet- IK Hyperparameters are provided in Appendix A. For ViT-B, MIRL pre-trained for 300 epochs achieves 83.5% top-1 fine-tuning accuracy, comparable to MAE (83.6%) pre-trained for 1600 epochs. Our pre-training is 5.3x shorter, demonstrating the high efficiency of MIRL. MIRL alleviates degradation in deeper ViTs, showing impressive generalization. In 800-epoch pre- pre-training scheme, the deeper encoder ViT- S-54 produces 84.8% accuracy, which is 1.7% higher than ViT-B (83.1%) pre-trained with MAE and only 0.6% lower than ViT-L (85,4%). ViT-B-48, with computational complexity similar to viT-L but 2x deeper, achieves 85.3% and 86.2% accuracy with 300 and 800-epoch pre-training schemes, outperforming the ViT-L models pre-trained by other listed methods. Furthermore, the deeper encoders can further benefit from very long pre-training schedules, as discussed Sec. 4.2.
4.4 Object detection and segmentation on COCO
To evaluate the generalization capabilities of our approach, we transfer our pre-trained models to the object detection task. The experiment is conducted MS COCO [30] account of its wide use. Following [21], we choose Mask R-CNN [22] as the detection framework and trained with the 1x schedule. For fair comparisons. we adopt the identical training configurations from midetection [3], and Average Precision (AP) is used as the evaluation metric. As summarized in Table 4, MIRL outperforms all the listed methods.
4.5 Semantic segmentation on ADE20K
We compare our method with previous results on the ADE20K [61] dataset, utilizing the UperNet framework for our experiments, based on the implementation provided by [1] (see Appendix A for training details). The evaluation metric is the mean Intersection over Union (mloU) averaged across all semantic categories. We employ pre- praitriine ViT-B-48 the backbone, which has a computational
8

---

Table 4: Object detection results with Mask R-CNN MS-COCO. The models pre-trained with extra data or very long schedules are marked in gray.
Method
Backhone
Pre-training Data
Epochs
Detection A AP
Segmentation A AP
DeiT [45]
VIT-B
46.9
41.5
BETT [1]
VIT-B
+NIK+D
800
46.3
41.1
MAE [21]
VIT-B
INIK
800
46.8
41.9
MAE [21]
VIT-B
INIK
1600
48.4
42.6
MIRL
VIT-B
INIK
800
49.3
43.7
MAE [21]
VIT-L
INIK
1600
53.3
47.2
MIRI
ViT-B-48
INIK
800
53.4
46.5
Table Semantic segmentation results on ADE20K. The models pre-trained with extra data or very long schedules are marked in gray.
Method
Pre-training Data
Backbone
Epochs
mloU
McCo v3 [5]
IN1K
VIT-I. VIT-L
300
49.1
BETT [1]
INIK+
800
53.3
MAE [21]
INIK
VIT-L
1600
53.6
MIRL
INIK
ViT-B-48
800
53.2
cost similar to VIT-L. The results summarized in Table 5. The segmentation model using ViT-B-48 achieves competitive results compared to ViT-L pre-trained with BEiT [1] and MAE [21]. This indicates that the instantiated deeper ViTs exhibit strong transferability to downstream vision tasks.
4.6 Limitation and discussion
While MIRL significantly alleviates training challenges for deeper ViTs, a comprehensive theoretical explanation for the effectiveness of image residual reconstruction in training deeper ViTs remains elusive. We provide some insights into why MIRL might work well for deeper ViTs: 1) By reformu- lating the pre-training objective recover the masked image's residual, MIRL implicitly encourages the model to focus on learning high-level contextual information and fine-grained details that are otherwise difficult to capture. 2) MIRL could stabilize gradient flow and enhance learning dynamics for deeper layers in ViTs, as evidenced by a larger gradient norm of the encoder in MIRL compared vanilla MIM (see gradient visualization in Appendix C). Despite these insights, further theoretical analysis and investigation are required to fully understand MIRL's effectiveness in training deeper ViTs. The deepest VIT presented in this research comprises only 54 blocks. We anticipate that depth of 54 far from the upper limit for scaling ViT along the depth dimension. These areas are left for future work.
5 Related Work
Self-supervised learning. After the triumph of BERT [10] and GPT [35] models in NLP, self- supervised learning (SSL. has undergone paradigm shift, replacing the conventional supervised learning approach [23, 43), and has made remarkable advancements in numerous domains [32, 60, 27J. Larger datasets, new training methods and scalable architectures [31, 13, 34, 59] have accelerated this growth. In computer vision, inspired by the success of BEiT [1], recent research [15, 29, 18, 24, 19, 20, 58, 2, 37, 33, 49] has explored adapting the transformer architecture to the task image self-supervised domain. After that, the emergence of MAE [21] has further led to resurgence of interest in reconstruction- based masked image methods, such as [21, 14, 51, 56, 11]. We are particularly intrigued by these masked methods, as they have shown state-of the art performance numerous transfer tasks and are computationally efficient. This has motivated us to introduce MIRL, novel approach that builds these methods.
Relation scaling models. Scaling deeper ConvNets [23, 43, 39, 42, 40, 41] is an effective way to attain improved performance, but the same cannot be easily achieved with ViTs [38]. While the

---

masked
res. main
residual
main
masked
res. main
residual
main
masked
res. main
residual
main
masked
res.
main
residual
main
Figure 7:
Visualization of MIRL. Example images are generated from the validation set on ImageNet.
Transformer architecture has succeeded in building large-scale language models [7, 53, 36, 44, 8, 17]. the implementation of scalable Transformers for visual models still significantly lags behind. Recent work [62, 28, 47, 9, 59] has endeavored explore deep Transformer-like models. These studies introduce necessary modifications to the original Transformer architecture, such parallel layers, altered layer norm positions, composite attention mechanisms, larger embedding dimensions, unique optimization strategies, and exhaustive hyperparameter searches. Although they have demonstrated commendable performance, they lack a guiding principle about how to deepen or enlarge the Transformer-lkke models. Contrary to previous methods, approach is rooted in in-depth analysis, dissecting the standard ViT architecture. This allows us to identify the challenges in fully realizing the potential deeper ViTs and develop effective solutions accordingly. Building upon the principles we proposed, we efficiently construct deep-scale ViT models.
Conclusion
In this paper, we first reveal performance degradation problem in Vison Transformers (ViTs) when pre-training with masked image modeling (MIM). Through an in-depth experimental analysis, we determine that the degradation is caused by the negative optimization effect of MIM enforced on deeper layers of VIT. We then introduce novel concept of masked image residual learning (MIRL) to establish self-supervised learning framework. aimed at alleviating the performance degradation problem. Leveraging MIRL, we unleash the potential of deeper ViTs and instantiate deeper encoders, including ViT-S-54, ViT-B-24 and ViT-B-48. These deeper ViTs variants exhibit superior generalization performance on downstream tasks.
Broader impacts. The proposed approach, which predicts content from training data statistics, may reflect biases with adverse socictal impacts and generate con-existent content, underlining the need for further research in this area.
10

---

References
[1] Hangbo Bao, Li Dong, and Furu Wei. BETT: BERT pre-training of image transformers. ICLR, 2021. 4, 7,
8.9
[2] Adrien Bardes, Jean Ponce, and Yann LeCun. Viereg: Varianee-ivariine--covraneee regularization for self- learning. arXiv preprint arXiv:2105.04900 2021.
[3] Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng,
Ziwei Liu, Jiarui Xu, et al. Mmdetcction: Open mmlab detection toolbox and benchmark. CoRR, 2019.
[4] Xiaokang Chen, Mingyu Ding, Xiaodi Wang. Ying Xin, Shentong Mo, Yunhao Wang. Shumin Han, Ping
Luo, Gang Zeng. and Jingdong Wang. Context autoencoder for self-supervised representation learning.
arXiv preprint (rxXiv2202.03026 2022.. 1
[5] Xinlei Chen, Saining Xie, and Kaiming Hc. An empirical study of training self-supervised vision transformers. In ICCV, 2021. 7.9
[6] Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang. Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. In ECCV, 2022. 2,
[7] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language
modeling with pathways. arXiv preprint arXiv:2204.023/// 2022. 10
[8] Hyung Won Chung, Le Hou, Shayne Longpre. Barret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finettunddd-e language models. arXiv
preprint ar Xiv:2210.. 1416, 2022. 10
[9] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22
billion parameters. preprint arXiv:2302.00 E04.22302.0544.. 2023. 5, 10
[10] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACI, 2019. 1.9
[1I] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprint arXiv:2111.12711
2021. 2.
[12] Xiaoyi Dong. Jianmin Bao, Ting Zhang. Dongdong Chen, Weiming Zhang. Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision hert pretraining. In ECCV, 2022.
[13] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth
16x16 16 words: Transformers for image recognition at scale. ICLR, 2021. 1, 6, 9
[14] Alaaeldin El-Nouby. Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave Are large-scale datasets necessary for self supervisdd pre- traiiriin?g? arXiv preprint arXiv:2112. 12.1400 2021. 9
[15] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised
representation learning. In ICMI. 2021. 9
[16] Yuxin Fang. Li Dong, Hangbo Bao. Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training. ICLR, 2023. 7
[17] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models
with simple and efficient sparsity. 2021. 10
[18] Spyros Cidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representa-
tions by predicting bags of visual words. In CVPR, 2020. 9
[19] Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthicu Cord, and Patrick Pérez. Online bag-of-visual-ve words generation for unsupervised representation learning. arXiv preprint a5x55:2012.11552.
2020. 9
[20] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self-supervised pretraining of
visual features in the wild. CVPR, 2021. 9
[21] Kaiming He. Xinlei Chen, Saining Xie. Yanghao Li. Piotr Dollár, and Ross Girshick. Masked autoencoders
are scalable vision learners. In CVPR, 2022. 3. S. 6, 7, 8,
[22]
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In ICCV, 2017. 8
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, 2016. 5.9
[24] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In ICMI., 2020. 9
[25] Ziyu Jiang, Yinpeng Chen, Mengchen Liu, Dongdong Chen, Xiyang Dai, Lu Yuan, Zicheng Liu, and Zhangyang Wang. Layer grafted pre-training: Bridging contrastive learning and masked image modeling for label-efficient representations, ICLR, 2023, 1
[26]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016. 5,7
[27] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver. Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General visual representation learning. In ECCV, 2020. 9
[28] Dmitry' Lepikhin, HyoukJoong Loc, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint 2006 16668, 2020. 10
[29] Junnan Li, Pan Zhou. Caiming Xiong. and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.0m 04966. 2020. 9

---

[30] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 8
[31] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV,
2018. 9
[32] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. JEEE Transactions on Knowledge and
Data Engineering, 2010. 9
[33] Xiangyu Peng, Kai Wang, Zheng Zhu, and Yang You. Crafting better contrastive views for siamesc representation learning. In CVPR, 2022. 9
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from
natural language supervision. In ICML, 2021. 9
[35] Alec Radford, Karthik Narasimhan, Tim Salimans, 1lya Sutskever, et al. Improving language understanding
by generative pre-training. OpenAl, 2018. 9
[36]
Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.
arXiv preprint arXivv/910 191110000......11111111111111181 2019. 10
[37]
Sucheng Ren, Fangyun Wei, Samuel Albanie, Zheng Zhang, and Han Hu. Deepmim: Deep supervision for
masked image modeling. arXiv preprint arXiv:2303. 20222 1, 9
[38] Carlos Riquelme, Joan Puigeerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In NeurlPS,
2021. 9
[39]
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-
tion. arXiv preprint arXiv: 1409, 1509,156, 2014. 9
[40] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint
arXiv:1505 AKKK:1505.00381 003870 2015. 9
[41] Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception-resnet
and the impact of residual connections on learning. In AAAI, 2017. 9
[42] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru
Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[43] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In
ICML,, 2019.
[44] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint
arXiv:2205.05133 2022. 10
[45] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers & distillation through attention. In ICMI., 2021. 9
[46] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz
Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017. 1
[47] Hongyu Wang, Shuming Ma, Li Dong. Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers 1.000 layers. arXiv preprint arXiv:2203.0055
2022. 10
[48] Haochen Wang. Kaiyou Song. Junsong Fan, Yuxi Wang, Jin Xie, and Zhaoxiang Zhang. Hard patches
mining for masked image modeling. In CVPR, 2023. 7
[49] Haoqing Wang, Ychui Tang, Yunhe Wang. Jianyuan Guo, Zhi-Hong Deng, and Kai Han. Masked image modeling with local multi-scale reconstruction. CVPR, 2023. 7,
[50] Shaoru Wang. Jin Gao, Zeming Li, Jian Sun, and Weiming Hu. A closer look at self-supervised lightweight vision transformers. arviv preprint Arov 14443, 2022. 1
[51] Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-super visual pre- pre-training. arXiv preprint arXiv:21 09133,,,9333,11 2021, 9 Wu, Alan Yuille, and Christoph Feichtenhofer. Masked
[52] Chen Wei, Haoqi Fan, Saining Xic, Chao-
feature prediction for self-supervised visual pre-training. In CVPR, 2022. 7
[53] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint X2v: 07682, 2022. 10
[54] Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. Tech Report,
2022. 5
[55] Zhennda Xie, Zigang Geng, Jingeheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. arXiv preprint arXiv:2205.1 13543, 2022. 1
[56] Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. CVPR, 2021. 7.9
[57] Kun Yi, Yixiao Ge. Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. ICLR, 2023. 7
[58] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised
learning via redundancy reduction. arXiv preprini urXiv:2103. (03.03000 2021. 9
[59] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In
CVPR, 2022. 9, 10
12

---

[60] Xiaohua Zhai, Joan Puigcerver. Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic. Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual adaptation benchmark. arXiv preprint arXiv:1910.0480. 2019.
[61] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[62] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang. Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. 1886, 2021. 10
13


Reviews:
[{"review_id": "cYx78wBrXf", "reviewer": "Reviewer_Cuwd", "summary": "* This paper propose a Masked Image Modeling pretraining framework,  which boost the pretraining performance in deeper deeper layers of ViT architecture.\n\n* Deeper vision transformers are hard to train, so authors introduce a new pretraining objective for mim, which can  alleviate the degradation problem in deeper vit.\n\n* MIRL segment the encoding blocks into several blocks(such as 4) and use different decoders to reconstruct the target.\n\n* various experiments are conducted, with less pre-training time, MIRL can achieve competitive performance compared to other mim approaches especially in deeper vit.", "questions": "please refer to the weakness", "limitations": "None", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* well motivated and well written, training and pretraining in deeper vit architecture is important, experiments show that pretraining in deeper vit-base can get similar result with vit-base.\n\n* The downstream task performance(in table 4) show superior performance than MAE framework, +2.5 bbox mAP, +1.8 mask AP.", "weaknesses": "* Since MIRL with deeper layers is different from the standard vit architecture(more deep than original vit), the fully supervised performance of the the same layers should be reported, the compare is a little in Table 3.\n\n* I am interested that would this framework be also work in convnet mask image modeling framework(such as simmim, spark) ? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question1 (Q1). Since MIRL with deeper layers is different from the standard vit architecture(more deep than original vit), the fully supervised performance of the the same layers should be reported, the compare is a little in Table 3.** \n\n**Answer 1 (A1).** We provide an additional comparison between ViT-B and ViT-S-54, shown in the below table, illustrating that a deeper ViT does not exceed a standard ViT when they do not experience the masked unsupervised training. We opt to not present these results in our original submission because we want readers to put more attention to the degradation problem that happened in the Masked pre-training process. \nWe use the same hyperparameters as provided in [1]. \n| Encoder | Method | ImageNet (%) |\n| -------- | -------- | :-------: |\n| ViT-B   | supervised    |              82.3|\n| ViT-S-54  | supervised   |         82.2|\n\n**Q2. I am interested that would this framework be also work in convnet mask image modeling framework(such as simmim, spark) ?**\n\n**A2.** Since our code was originally implemented for the ViT arch, we have not tested our MIRL framework by using pure ConvNets. However, we instead experiment with the hybrid architecture, ConvViT[2, 3], which is made up of both conv layers and transformer layers. When using ConvViT-B as an encoder, using a 300 epoch-pretraining scheme, MIRL generates 85.1\\% finetuning accuracy higher than the baseline (84.3\\%). \n\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Gao, Peng, et al. \"Convmae: Masked convolution meets masked autoencoders.\" arXiv preprint arXiv:2205.03892 (2022).\n\n[3] Dosovitskiy, Alexey, et al. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" arXiv preprint arXiv:2010.11929 (2020)."}, {"review_id": "6x7o8NfHxY", "reviewer": "Reviewer_Pj1d", "summary": "This paper clarifies that MIM pretraining can induce negative optimization in deeper layers of ViT through comparison between vanilla MAE and truncated MAE. Based on this observation, this paper proposes a MAE-based framework named Masked Image Residual Learning (MIRL) for alleviating the degradation problem and training deeper ViT. In MIRL, The objective of MIM is decoupled to recovering the main component of images from the features of shallower layers, and recovering the residual from the features of deeper layers. Ablation experiments shows that MIRL outperforms MAE and MIM baselines with multiple decoders, and help deeper ViTs (e.g., ViT-S-54 and ViT-B-48) achieve better performance compared to previous methods on ImageNet.", "questions": "1. The results in Table 2(d) show the comparison between MIRL and MIM baselines with multiple decoders. Did the baseline method here use DID? If not, can \"residual learning + DID\" significantly outperform \"multi-decoders + DID\"? \n\n2. A comparison of training time between MIRL and MAE should be included.\n\nOther questions have been mentioned in weaknesses.", "limitations": "The authors have addressed the limitations of this paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In this paper, the degradation problem in the MIM pretraining of ViT is illustrated by well-designed experiments.\n\n2. Training deeper ViT with MIM has not been widely studied in previous works.\n\n3. The idea of recovering the residual of images is interesting, and ablation experiments show the effectiveness of MIRL.", "weaknesses": "1. Although this paper has cited some previous works that provide supervision to different layers of ViT (like deepMIM[1]), there is no comparison between MIRL and these previous methods in this paper.\n\n2. Using multiple decoders in MIRL will significantly increase training time compared with MAE. A comparison of training time between MIRL and MAE should be included.\n\n3. On the dense prediction tasks, MIRL does not show significant improvement (even weaker performance) compared to MAE with a similar number of parameters. \n\n4. This paper says that MIRL alleviates the degradation problem in deeper layers of ViT. However, This paper does not provide evidence on whether the phenomenon in observation II still exists in the proposed method. \n\n5. The baseline results of pretraining deeper ViTs (like ViT-S-54 ViT-B-48) with MAE are not provided in this paper.\n\n[1] Sucheng Ren, Fangyun Wei, Samuel Albanie, Zheng Zhang, and Han Hu. Deepmim: Deep supervision for masked image modeling. arXiv preprint arXiv:2303.08817, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for considering our idea interesting. We address the reviewer's concerns as follows.\n\n**Q1. Although this paper has cited some previous works that provide supervision to different layers of ViT (like deepMIM[1]), there is no comparison between MIRL and these previous methods in this paper.**\n\n**A1.** Given that DeepMIM [1] utilizes pre-trained MAE models, drawing a direct comparison with our approach may not be meaningful. We were unable to utilize the code available on DeepMIM's GitHub page. However, the DeepMIM-MAE model bears resemblance to the simple \"multi-decoder model\" discussed in our ablation study, the results of which are presented in Table 2(d) on page 6 of the paper. In Lines 203-212 of the paper, we discuss the results and argue that multi-decoders are not as effective as MIRL.\n\n**Q2. Using multiple decoders in MIRL will significantly increase training time compared with MAE. A comparison of training time between MIRL and MAE should be included.**\n\n**A2.** In MIRL, each decoder is quite shallow, ensuring that there isn't a significant increase in the training time. We assessed the training time per epoch on a single V100 GPU, and the total pre-training (pt) time was determined by multiplying the epoch time by the number of training epochs. As illustrated in the table below, when achieving comparable fine-tuning (ft) accuracy, MIRL requires roughly 6 times less training time than MAE. This highlights the better efficiency of the proposed methodology.\n\n\n| method  | backbone | min/ep  | #pt epochs | total pt time (hours) | ft acc. (%) |\n| -------- |  :-------: |  :-------:  |  :-------: |   :-------: | :-------: | \nMAE  | ViT-B | 64 | 1600 | 1706  | 83.6|\nMIRL  | ViT-B | 57 | 300 | 285 |  83.5 |\n\n\n**Q3. On the dense prediction tasks, MIRL does not show significant improvement (even weaker performance) compared to MAE with a similar number of parameters.**\n\n**A3.** For the semantic segmentation task on ADE20K, MIRL yields an mIOU that is 0.4 lower than MAE's. However, our pre-training schedule is only half the duration of MAE's (i.e., 800 epochs vs. 1600 epochs). Given these closely matched results, the greater efficiency of MIRL deserves emphasis. \nMoreover, when pre-training for a longer schedule, we expect our model to deliver even better performance, as analyzed in Figure 6 from the paper. Additionally, even with fewer pre-training epochs, our method outperforms other approaches in both image classification and object detection tasks.\n\n\n\n**Q4. This paper says that MIRL alleviates the degradation problem in deeper layers of ViT. However, This paper does not provide evidence on whether the phenomenon in observation II still exists in the proposed method.**\n \n\n**A4.** To address this concern, we devise an additional model named \"truncated MIRL\". The concept is akin to the truncated MAE depicted in Figure 1(b) from the paper. It involves pre-training the early encoding blocks using MIRL, while the subsequent blocks are randomly initialized. As detailed in the table below, MIRL outperforms truncated MIRL (in which 3 blocks are not involved in pre-training, and the 5th block solely focuses on recovering the masked content) by 0.3%. This demonstrates that MIRL effectively pre-trains the deeper layers, outperforming random initialization. This also suggests that the phenomenon observed in Observation II does not exist in our method. Due to the space limitation for the paper, an additional diagram to depict truncated MIRL will be provided in the supplementary material of the revised version.\n\n\n\n| encoder | method  |  ft acc. (%) |\n| -------- |--------  |  :-------: |\n| ViT-S | MIRL |  82.3 |\n| ViT-S | truncated MIRL |  82.0 |\n| ViT-S | MAE |  81.0 |\n| ViT-S | truncated  MAE |  81.7 |\n\n\n**Q5. The baseline results of pretraining deeper ViTs (like ViT-S-54 ViT-B-48) with MAE are not provided in this paper.**\n \n**A5.** We have performed the relevant experiments and in the table below we provide comparisons between MAE and MIRL when using deeper ViTs. \n\n\n| encoder | method  | pre-training epochs|  ft acc. (%) |\n| -------- |--------  | :-------: | :-------: |\n| ViT-S-54 | MAE | 300 |  82.7 |\n| ViT-S-54 | MIRL | 300 |  84.4 |\n| ViT-B-48 | MAE | 300 |  84.5 |\n| ViT-B-48 | MIRL | 300 | 85.3 |\n\nBy tackling the degradation problem, MIRL consistently outperforms MAE and offers superior model scaling capabilities.\n\n\n**Q6. The results in Table 2(d) show the comparison between MIRL and MIM baselines with multiple decoders. Did the baseline method here use DID? If not, can \"residual learning + DID\" significantly outperform \"multi-decoders + DID\"?**\n\n**A6.** To address this concern, we quantize our method as MIRL =  multi-decoders + DID + \"residual learning\".\nAfter accounting for each of these components, as well as their various combinations, we provide the following results:\n\n| model  | multi-decoders  | DID | residual learning |Top-1 (%) |\n| -------- |  :-------:  |  :-------: | :-------: | :-------: | \nViT-B-48  |&cross;| &cross;| &cross;  |  84.3 |\nViT-B-48  |&#10004;| &cross;| &cross;  |  84.5 |\nViT-B-48  |&#10004;|&#10004;| &cross;  |  84.5 |\nViT-B-48  |&#10004;|&cross;| &#10004;  |  85.0 |\nViT-B-48  |&#10004;|&#10004;| &#10004;  |  85.3 |\n\nWe clarify that in Table 2(d) from the paper, the multi-decoder implementation utilizes DID. However, as indicated in rows #2 and #3 of the above table, DID does not enhance the model's performance when \"residual learning\" is excluded. As we mentioned in Lines 116-120 from the paper, \"residual learning\" represents the core of the proposed framework. Still, DID boosts the \"multi-decoders\" accuracy by ~0.1% when the encoder is ViT-B-24.\n\n\n\n[1] Ren, Sucheng, et al. \"DeepMIM: Deep Supervision for Masked Image Modeling.\" arXiv preprint arXiv:2303.08817 (2023). \n\n"}, {"review_id": "RiU5Uc8tqo", "reviewer": "Reviewer_igwC", "summary": "This paper delves into the degradation issue encountered in the deeper layers of Vision Transformer (ViT) and proposes a self-supervised learning framework called Masked Image Residual Learning (MIRL). MIRL reformulates the learning objective to recover the residual of the masked image and makes scaling ViT along depth a promising direction for enhancing performance. Experimental results demonstrate promising improvements on various downstream tasks with increased depth, highlighting the effectiveness of the MIRL method.", "questions": "Training the vision-Transformers with varying scales often requires distinct hyper-parameters. For instance, the optimal hyper-parameters for training ViT-Large and ViT-Small typically differ. It would be interesting to investigate if the proposed MIRL approach can alleviate this phenomenon when training vision Transformers with varying scales.\n\nOverall, this paper addresses an important problem and provides some insights through observations and the proposed solutions. While it leans more towards empirical findings, it would be beneficial to provide additional theoretical evidence to support the claims.", "limitations": "See the above comments.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Tackling the degradation issue in training deeper vision-Transformers (ViT) in the context of MIM pre-training is a practical problem and is also a under-explored area of research. \n\n2.This paper presents several interesting observations and shows the MIM introduces negative optimization in the deeper layers of ViT that would hinder further pre-training performances. Thre proposed method is simple and effective for solving this problem.\n\n3.The authors conduct extensive and comprehensive experiments that show the effectiveness and efficiency of the proposed method.", "weaknesses": "1.While the proposed method has shown promising results, it still lacks an in-depth explanation, e.g., choosing the training objective for encouraging deep layers to recover the image residual. The rationale behind dividing different segments and providing the proposed reconstruction terms could benefit from offering additional theoretical justifications, such as utilizing tools like Mutual Information.\n\n2.It would be beneficial to provide further clarity on the mechanism to ensure precise reconstruction of the main component and residual of the image in the shallower and deeper parts, respectively. Given that they share the same training objective, disentangling them poses a nontrivial challenge.\n\n3.An ablation study could be added to investigate whether assigning different regularization weights to different segment loss parts is beneficial, whether the MIRL approach incurs additional training time and other costs.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of our work, considering the area addressed in our paper as an important under-explored area of research. In the following we address the concerns of the reviewer.\n\n**Q1. While the proposed method has shown promising results, it still lacks an in-depth explanation, e.g., choosing the training objective for encouraging deep layers to recover the image residual. The rationale behind dividing different segments and providing the proposed reconstruction terms could benefit from offering additional theoretical justifications, such as utilizing tools like Mutual Information.**  ...\n\n**Overall, this paper addresses an important problem and provides some insights through observations and the proposed solutions. While it leans more towards empirical findings, it would be beneficial to provide additional theoretical evidence to support the claims.**\n\n**A1.** Indeed, as the reviewer mentioned and as we ourselves noted in Section 4.6, an in-depth theoretical explanation for our method remains elusive. \nAn intuitive explanation for our method can be found in our response to Reviewer DfMD's Question 2.\nAdditionally, we have offered some insights in Lines 259-264 from Section 4.6, entitled \"Limitation and discussion,\" from the paper, and have included a gradient analysis in Appendix C from the supplementary materials associated with the paper. This analysis illustrates that our method could stabilize gradient flows and enhance the learning dynamics for deeper layers in ViTs. \nWe plan to provide a thorough theoretical justification in a subsequent work.\n\nWe appreciate the suggestions of the reviewer. We will consider analyzing the mutual information to provide more insights of MIRL.\n\n\n\n**Q2. It would be beneficial to provide further clarity on the mechanism to ensure precise reconstruction of the main component and residual of the image in the shallower and deeper parts, respectively. Given that they share the same training objective, disentangling them poses a nontrivial challenge.**\n\n**A2.** In Line 213, the ablation study provided compares MIRL with \"coarse and fine\" separation, examining whether a precise reconstruction of both the main component and image residual can enhance performance. The \"fine and coarse\" separation resembles the main component and residual separation but with two unique training objectives. Such a precise reconstruction is not as effective as our default design.\n\nIn Appendix B.2 of the supplementary material submitted with the paper, we also investigated whether explicitly imposing a precise reconstruction can be beneficial by modifying the loss definition as follows:\n\n$ \\mathcal{L}_g^\\dag = \\frac{1}{|\\mathcal{M}| }\\sum _{i\\in \\mathcal{M}} \\frac{1}{2P^2C} \\big(\\| \\xi_g^i \\|^2_2 + \\| \\xi_g^i - \\hat{\\xi}_g^i \\|^2_2 \\big).  $\n\nThis design also falls short compared to our default design. While scaling term $\\| \\xi_g^i \\|^2_2$ with a small value such as 0.1, we achieve the same outcome as the default design.\n\n\n\n**Q3. An ablation study could be added to investigate whether assigning different regularization weights to different segment loss parts is beneficial, whether the MIRL approach incurs additional training time and other costs.**\n\n**A3.** Following the suggestion of the reviewer, we introduce weight coefficient $\\lambda_g$ into our loss definition for each segment, shown in the equation below, \n\n$ \\mathcal{L}_g = \\frac{1}{|\\mathcal{M}| } \\lambda_g\t\\sum _{i\\in \\mathcal{M}} \\frac{1}{P^2C} \\| \\xi_g^i - \\hat{\\xi}_g^i \\|^2_2 .$\n\nWe will add an ablation study for $\\lambda_g$ in the revised version.\n\nMIRL does not lead to an increase in training time. Instead, throughout the paper, we emphasize the high efficiency of our method. It is not that it only delivers superior results but it also requires shorter training times than the baselines. For detailed information on training time, please refer to our response to Reviewer Pj1d's Question 2, where we also provide a table with results.\n\n**Q4. Training the vision-Transformers with varying scales often requires distinct hyper-parameters. For instance, the optimal hyper-parameters for training ViT-Large and ViT-Small typically differ. It would be interesting to investigate if the proposed MIRL approach can alleviate this phenomenon when training vision Transformers with varying scales.**\n\n\n**A4.** We appreciate the insightful suggestions. Across all ViT variants, we've maintained consistent hyper-parameter settings, which encompass learning rate, batch size, weight decay, among others, during the pre-training phase. Tweaking these hyper-parameters has a minimal impact on the performance, according to our experiments. \nDuring the fine-tuning phase, we found our models to be insensitive to the change of droppath configuration, as mentioned in Appendix A.1 from the supplemental material submitted with the paper. \n\nHowever, as we have found empirically, using a larger layer-wise learning rate decay for deeper ViTs is currently unavoidable.\nWe will incorporate more details in the supplementary material of the revised version.\n"}, {"review_id": "Ehubm3GRME", "reviewer": "Reviewer_6uey", "summary": "In this paper, the authors propose a new mask image modeling method, which is helpful for deep ViT model pretraining.", "questions": "### strength\n1. The paper is well-written and easy to follow\n2. The proposed method starts from an interesting observation and is novel.\n3. The experiment results seem good.\n\n### Weakness\n1. The experiment lacks some necessary reuslts. For example, ViT-L is deeper than ViT-B and has 24 layers by default, it may benefit from the proposed method. This would be a fair comparison and further prove the effectiveness of the proposed method.\n2. Following 1, there is no baseline for the deeper models except Table.2(b). So when comparing with MAE, the gain of the proposed method is not clear. \n3. I'm confused about the design of ViT-S-54, why use 54 layers? if it uses 48 layers, its params and FLOPs will be more similar to ViT-B and the comparison would be more fair.\n4. I also have a question about the method design, why the segments are divided into two groups to predict content and residual? Take Fig4 as an example, denote input as x, if H1 predicts x0, H2 predicts x1 = x-x0, H3 further predicts x2 = x-x0-x1 and so on, such a sequential design seem more consistent with the model architecture, will it perform better?\n5. A question about detail, MAE uses the pixel-normed patch as the prediction target, which is critical for its performance. Is this also used in the proposed method and the reported MAE baselines(300/800)?\n6. Some baseline methods are missed. For example, MaskFeat[1] for ViT-B, Data2Vec[2], PeCO[3], BootMAE[4] and CAE[5] for ViT-B/L.\n7. A small typo in Table.2(d), should it be multi-decoders?\n\n[1] Masked Feature Prediction for Self-Supervised Visual Pre-Training\n\n[2]data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language\n\n[3]PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers\n\n[4]Bootstrapped Masked Autoencoders for Vision BERT Pretraining\n\n[5]Context Autoencoder for Self-Supervised Representation Learning\n \n", "limitations": "Please refer to Questions", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Please refer to Questions", "weaknesses": "Please refer to Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our observations interesting and method novel. In the following, we address the concerns of the reviewer.\n\n**Q1. The experiment lacks some necessary results. For example, ViT-L is deeper than ViT-B and has 24 layers by default, it may benefit from the proposed method. This would be a fair comparison and further prove the effectiveness of the proposed method.**\n\n**A1.** \nAs stated in Lines 153-157 from the paper, training ViT-L is very unstable due to the adoption of a large hidden dimension size.\nWhile performing MAE pre-training with ViT-L, we encountered the issue of \"NaN\" values. Each time when NaN occurs, we have to restart the training from the beginning epoch, as checkpoint resumptions could not solve the problem. This instability has also been reported in [1]. These training failures resulted in substantial waste of computational resources. Due to this instability, the actual computational cost for training was 2 to 3 times higher than the ideal cost, as we had to complete the pre-training of ViT-L after encountering failure in three separate attempts while using MAE.\nNonetheless, we assure that the experimental setup, theoretical derivations, and conclusions presented in this paper adhere to rigour and reproducibility. We hold the belief that the performance trends observed for ViT-B and ViT-S can be extended to ViT-L despite the encountered instability. \n\n\n\n\n**Q2. Following 1, there is no baseline for the deeper models except Table.2(b). So when comparing with MAE, the gain of the proposed method is not clear.**\n\n**A2.** The baselines for deeper models can be found in our response to Reviewer Pj1d's Question 5, including a table with results. According to these results, across various model scales, MIRL consistently outperforms MAE.\n\n\n\n**Q3. I'm confused about the design of ViT-S-54, why use 54 layers? if it uses 48 layers, its params and FLOPs will be more similar to ViT-B and the comparison would be more fair.**\n\n**A3.** The 54-layer ViT model features a slender structure, generally deemed challenging to train. Experimenting with such a slender model can further demonstrate our approach's ability in facilitating training deeper ViT models, aligning with the paper's theme of scaling deeper vision transformers.\nWe have presented a fair comparison between MAE and MIRL in Table 3 from the paper, where both employ the same encoder, specifically ViT-B.\n\n\n**Q4. I also have a question about the method design, why the segments are divided into two groups to predict content and residual? Take Fig4 as an example, denote input as x, if H1 predicts x0, H2 predicts x1 = x-x0, H3 further predicts x2 = x-x0-x1 and so on, such a sequential design seem more consistent with the model architecture, will it perform better?**\n\n**A4.** For insights into the intuition behind our method's design, kindly refer to our response to Reviewer DfMD's Question 2.\n\nThe reviewer's suggestion seems to parallel a design concept where one learns the residual of another residual. Although we haven't experimented with this specific design, we harbour reservations about such a \"residual-of-the-residual learning\" scenario. The residual of the residual may approach zero, potentially stalling weight updates.\n\n\n**Q5. A question about detail, MAE uses the pixel-normed patch as the prediction target, which is critical for its performance. Is this also used in the proposed method and the reported MAE baselines(300/800)?** \n\n**A5.** Yes. Our models as well as the reported MAE baselines used the normalized pixel values. \n\n**Q6. some baseline methods are missed. For example, MaskFeat for ViT-B, Data2Vec, PeCO, BootMAE and CAE for ViT-B/L.**\n\n**A6.** The result for MaskFeat[2] is included in Table 3, from page 7 of the paper. However, due to space constraints in the submission, the results of the other mentioned methods are not presented in Table 3. We will include comparisons with these methods in the revised version.\n\n\n\n**Q7. A small typo in Table.2(d), should it be multi-decoders?**\n\n**A7.** Thanks for pointing out the typo. Yes, it should be multi-decoders.\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proc. of the IEEE/CVF CVPR, 2022.\n\n[2] Wei, Chen, et al. \"Masked feature prediction for self-supervised visual pre-training.\" Proc. of the IEEE/CVF CVPR, 2022."}, {"review_id": "M3tuCPfZCu", "reviewer": "Reviewer_DfMD", "summary": "This paper finds that the deep layers of ViT fail to benefit from MIM pre-training. The authors replace deeper layers of MAE pre-trained ViTs with random initialization and demonstrate that this modified model achieves better performances than the original MAE pre-trained model. To this end, the authors conclude that MIM pre-training is only effective for shallow layers. To address this, Masked Image Residual Learning (MIRL) is proposed. Experiments under various benchmarks demonstrate the efficacy of the proposed MIRL method.", "questions": "I do not have any questions. Please refer to the weaknesses section. I am willing to raise my rating if the authors succeed in demonstrating that their improvements come from the proposed method and not from carefully tuned hyperparameters during fine-tuning.\n\n**Post Rebuttal Comments**\n\nThanks for the rebuttal and additional comments. My concerns are well addressed. Since using EMA in fine-tuning will not affect the performance significantly, the reviewer thinks it might be a fair comparison. Also, the explanation of why residual learning works is intuitive. Therefore, I will raise my rating to 6 and the authors are supposed to revise their paper according to our dialog, including (1) clarification about the EMA, (2) an intuitive explanation about why it works, and (3) comparison with CVPR'23 works.", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. The authors have systematically analyzed whether MIM pre-training is effective for deeper layers of ViTs.\n3. The proposed method and the motivation are highly connected.\n4. Experiments are sufficient and the code is provided in the supplementary material.", "weaknesses": "**Major Concerns:**\n\n1. **Unfair comparisons against other methods.** I have carefully checked the fine-tuning configuration of both MIRL (Table 2 in the supplementary material) and MAE [21], and find that *MIRL incorporates an extra exponential moving average (EMA)* while others do not, which appears to be an unfair comparison. It is better to fine-tune all models pre-trained by MIRL *without* EMA. Also, several hyper-parameters have been carefully tuned compared with MAE [21], such as base learning rate, batch size, and warmup epochs. Therefore, it is better to compare with MAE which is fine-tuned under the exact same configuration.\n2. **Lack of intuitive explanation of residual learning.** I recognize the efficacy of learning residual in MIM, but the underlying motivation remains unclear. In other words, given the observation that deep layers of ViT fail to benefit from MIM pre-training, why learning residual can alleviate this problem?\n3. **Observation I and II cannot be reflected in Figure 2 by using a single plot named \"truncated MAE\".** If my understanding is correct, models of these two observations are pre-trained in different ways. Specifically, the model of Observation I should be pre-trained by vanilla MAE while the model of Observation II should be pre-trained by truncated MAE. Therefore, these two observations cannot be reflected in Figure 2. It is better to plot three curves in Figure 2, including vanilla MAE, vanilla MAE with truncated fine-tuning (Observation I), and truncated MAE (Observation II).\n\n**Minor Questions/Suggestions:**\n\n4. Typo in Eq. (6). It should be $||\\mathbf{x}^i - \\hat{\\mathbf{x}}_g^i - \\hat{\\xi}_g^i||_2^2$.\n5. It is better to combine Table 1 in the main submission with Table 4 in the supplementary material to better compare the configuration (including #params and FLOPs) of different backbones.\n6. It is encouraged to plot vanilla MAE with ViT-B and ViT-L in Figure 6 to demonstrate the scaling behavior of the proposed method.\n7. It is better to compare deep ViTs pre-trained with vanilla MAE.\n8. The configuration of Table 2a is not clear. Is residual learning performed on both pixel and feature?\n9. The setting of Table 2d is confusing. Its caption says \"multi-encoder\" while in lines 203-212 is \"multi-decoder\".\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments. We answer the questions as follows.\n\n**Q1. Unfair comparisons against other methods.  MIRL incorporates an extra exponential moving average (EMA) while others do not, which appears to be an unfair comparison ... .**\n\n\n**A1.** We address this concern from three aspects:\n\n1. In our experiments, EMA does not significantly impact the fine-tuning accuracy on ImageNet. When fine-tuning ViTs without EMA, using the same hyperparameters as mentioned in [1], we observed similar accuracy as those obtained with EMA. For instance, when pre-trained for 300/800 epochs, ViT-S-54 achieved accuracies of 84.41%/84.79% with EMA and 84.34%/84.86% without EMA. Incorporating EMA requires adaptive adjustments to the learning rate and warm-up epochs to avoid overfitting.\n\n2. The reason we employ EMA is to enhance tuning performance on small datasets consisting of only a few hundred training samples, such as private industrial datasets. When tuning on these limited-scale datasets, we gravitate towards loading weights from the ImageNet fine-tuned model instead of the MIM pre-trained model because the MIM lacks semantic features. In this context, using the EMA-fine-tuned models yields better tuning accuracy on tiny datasets, especially when picking a non-final checkpoint, compared to counterparts without EMA. \n\n3. The reason EMA doesn't significantly impact performance could be attributed to the sufficiently long training period (e.g., spanning 800 pre-training epochs + 200/100 fine-tuning epochs). This allows models with or without EMA to likely converge to the same optimum. Nonetheless, EMA remains crucial for training models from scratch, as indicated in [1] (e.g., ViT-B achieves 82.3% accuracy with EMA and 82.1% without EMA). \n\n**Q2. Lack of intuitive explanation of residual learning.  Why learning residual can alleviate this problem?**\n\n**A2.** We explain the intuition behind our method as follows:\nUpon observing that deeper layers pre-trained by MIM underperformed compared to those with random initialization, we inferred that the weight parameters of these deeper layers had indeed been updated during MIM pre-training, but in an unfavorable direction.\nIn contrast, the shallower layers demonstrated improved performance after MIM pre-training.  This led us to speculate that the depth of the layers could be the root cause of the degradation problem. By introducing shortcut connections between the shallower and deeper layers, we allowed the back-propagation to affect both the deeper and shallower layers simultaneously, cooperating to restore the masked content.  This MIRL design tightly couples the deeper layers with the shallower ones, implying that MIRL should either guide both the deeper and shallower layers towards a beneficial direction or lead both astray. The results suggest that the shallower layers have, in essence, steered the deeper layers towards a more favorable direction. We will include this explanation in the revised version of the manuscript.\n \n**Q3. Observation I and II cannot be reflected in Figure 2 by using a single plot named \"truncated MAE\". If my understanding is correct, models of these two observations are pre-trained in different ways. Specifically, the model of Observation I should be pre-trained by vanilla MAE while the model of Observation II should be pre-trained by truncated MAE. Therefore, these two observations cannot be reflected in Figure 2. It is better to plot three curves in Figure 2 ... .**\n\n**A3.** We apologize for the confusion caused. In Figure 2, we included the results for vanilla MAE, vanilla MAE with truncated fine-tuning, and truncated MAE. Within the curve for vanilla MAE in Figure 2, the initial point indicates the result of an MAE model that has been pre-trained and subsequently fine-tuned without any random initialization. This specific point aligns with the fine-tuning result of vanilla MAE.  Meanwhile, the rest points in the vanilla MAE curve indicate the results of vanilla MAE with truncated fine-tuning. We will provide further clarity on this point in the revised version.\n\n**Q4. Typo in Eq. (6). It should be $\\| | \\mathbf{x}^i - \\hat{\\mathbf{x}}^i_g - \\hat{\\xi}_g^i | \\|^2_2.$ The setting of Table 2d is confusing. Its caption says \"multi-encoder\" while in lines 203-212 is \"multi-decoder\"**\n\n**A4.** We appreciate the reviewer for pointing out the typos. We correct them accordingly. Specifically, in the caption of Table 2d, \"multi-encoders\" will be corrected to \"multi-decoders\". \n\n**Q5. It is better to combine Table 1 in the main submission with Table 4 in the supplementary material to better compare the configuration ... .**\n\n**A5.** Thanks for the suggestion from the reviewer. We will combine Table 1 from the main submission with Table 4 in the supplementary material in the revised version.\n\n**Q6. It is encouraged to plot vanilla MAE with ViT-B and ViT-L in Figure 6 to demonstrate the scaling behavior of the proposed method.**\n\n**A6.** We agree with the reviewer's suggestion that including curves for MAE models can highlight the scaling capabilities of our method. In the revised version, we will add the curves of vanilla MAE with ViT-B and ViT-L in Figure 6.\n\n**Q7. It is better to compare deep ViTs pre-trained with vanilla MAE.**\n\n**A7.** Reviewer Pj1d raised a similar concern. Kindly refer to our response to Reviewer Pj1d's Question 5, and the table with results, for clarification. Across various model scales, MIRL consistently outperforms MAE.\n\n**Q8. The configuration of Table 2a is not clear. Is residual learning performed on both pixel and feature?**\n\n**A8.** The feature-level loss is only calculated at the end of the encoder, which means that the \"residual learning \" is only performed at the pixel level, as mentioned in Appendix B.1 from the supplementary material.\n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proc. of the IEEE/CVF CVPR, 2022."}]

Author rebuttal:

Meta-review: This paper focuses on the issue of performance degradation in the deeper layers of Vision Transformer (ViT) when applying masked image modeling. It introduces a new framework called MIRL. The paper underwent evaluation by five experts in the field, all of whom expressed strong interest in the problem being studied. Despite initial concerns raised before the rebuttal phase, the authors successfully addressed these concerns and received unanimous positive feedback from the reviewers.

The Area Chair agrees with the reviewers that the problem addressed in this paper is of great significance and has the potential to inspire further research in the field, thus recommending acceptance. The authors should seamlessly integrate all the newly added experiments and analyses into the final version of the manuscript.
Decision: Accept (poster)