Title: Masked Image Residual Learning for Scaling Deeper Vision Transformers
Venue: NeurIPS 2023
Accepted: yes
Keywords: self-supervised learning

vision transformer

masked image modeling
Abstract: Deeper Vision Transformers (ViTs) are more challenging to train. We expose a degradation problem in deeper layers of ViT when using masked image modeling (MIM) for pre-training.
To ease the training of deeper ViTs, we introduce a self-supervised learning framework called  $\textbf{M}$asked $\textbf{I}$mage $\textbf{R}$esidual $\textbf{L}$earning ($\textbf{MIRL}$), which significantly alleviates the degradation problem, making scaling ViT along depth a promising direction for performance upgrade. We reformulate the pre-training objective for deeper layers of ViT as learning to recover the residual of the masked image.
We provide extensive empirical evidence showing that deeper ViTs can be effectively optimized using MIRL and easily gain accuracy from increased depth. 
With the same level of computational complexity as ViT-Base and ViT-Large, we instantiate $4.5{\times}$ and $2{\times}$ deeper ViTs, dubbed ViT-S-54 and ViT-B-48.
The deeper ViT-S-54, costing $3{\times}$ less than ViT-Large, achieves performance on par with ViT-Large.
ViT-B-48 achieves 86.2\% top-1 accuracy on ImageNet. 
On one hand, deeper ViTs pre-trained with MIRL exhibit excellent generalization capabilities on downstream tasks, such as object detection and semantic segmentation. On the other hand, MIRL demonstrates high pre-training efficiency. With less pre-training time, MIRL yields competitive performance compared to other approaches.

Masked Image Residual Learning for Scaling Deeper Vision Transformers

Abstract

Deeper Vision Transformers (ViTs) are more challenging to train. We expose a degradation problem in deeper layers of ViT when using masked image modeling (MIM) for pre-training. To ease the training of deeper ViTs, we introduce a selfsupervised learning framework called M asked I mage R esidual L earning ( MIRL ), which significantly alleviates the degradation problem, making scaling ViT along depth a promising direction for performance upgrade. We reformulate the pretraining objective for deeper layers of ViT as learning to recover the residual of the masked image. We provide extensive empirical evidence showing that deeper ViTs can be effectively optimized using MIRL and easily gain accuracy from increased depth. With the same level of computational complexity as ViT-Base and ViT-Large, we instantiate 4.5 × and 2 × deeper ViTs, dubbed ViT-S-54 and ViT-B-48. The deeper ViT-S-54, costing 3 × less than ViT-Large, achieves performance on par with ViT-Large. ViT-B-48 achieves 86.2% top-1 accuracy on ImageNet. On one hand, deeper ViTs pre-trained with MIRL exhibit excellent generalization capabilities on downstream tasks, such as object detection and semantic segmentation. On the other hand, MIRL demonstrates high pre-training efficiency. With less pre-training time, MIRL yields competitive performance compared to other approaches. Code and pretrained models are available at: https://github.com/russellllaputa/MIRL .

1 Introduction

Transformer architecture [46] has become the de-facto standard in natural language processing (NLP). A major driving force behind the success of Transformers in NLP is the self-supervised learning method called masked language modeling (MLM) [10]. MLM significantly expends the generalization capabilities of Transformers, with the underlying principle being very intuitive - removing portions of a sentence and learning to predict the removed content. Recent advancements in computer vision have been profoundly inspired by the scaling successes of Transformers in conjunction with MLM in NLP, successively introducing the Vision Transformer (ViT) [13] and masked image modeling (MIM) for training generalizable vision models. The concept of MIM is as straightforward as MLM; its pre-training objective is to predict masked image patches based on the unmasked image patches, thereby capturing rich contextual information.

This paper first reveals that MIM can induce negative optimization in deeper layers of ViT, which not only constrains the generalization performance of ViT but also hinders its scaling along the depth dimension. Previous work [55, 37] suggests that deeper layers of ViT are more properly pre-trained by using MIM, the conclusions of which contradict our observation. Another branch of work [6, 50, 4, 25, 37] tentatively suggests that, due to the lack of semantic information in MIM, the shallower layers of ViTs are more effectively pre-trained than the deeper layers. In our preliminary experiments in Sec. 2.2, we demonstrate that replacing the deeper Transformer blocks pre-trained by using MIM with randomly initialized blocks does not degrade performance, which supports our

(a) MAE

(b) truncated MAE

(c) MIRL

shallow

deep

Encoder

Decoder

Decoder

Loss

deep

shallow

Encoder

Decoder

Loss

Decoder

shallow

deep

Encoder

Loss

0

1

2

3

4

5

6

7

8

9

10

11

12

Number of randomly initialized blocks

79.75

80.00

80.25

80.50

80.75

81.00

81.25

81.50

81.75

Accuracy (%)

vanilla MAE

truncated MAE

80.0

80.5

81.0

81.5

82.0

82.5

83.0

83.5

Accuracy (%)

ViT-Small

ViT-Base

81.0

82.6

81.7

82.9

82.3

83.5

vanilla MAE

truncated MAE

MIRL

statement. We hypothesize the negative pre-training effect enforced on the deeper layers is due to depth restriction, which can be regarded as a degradation problem occurring in deeper ViTs.

We address the pre-training degradation problem in deeper layers of ViT by introducing a M asked I mage R esidual L earning ( MIRL ) framework. We establish a multi-decoding process by segmenting the encoding blocks according to their depth levels. Instead of letting the entire autoencoder learn to reconstruct the masked content, MIRL encourages the deep layers to learn latent features that are beneficial for recovering the image residual, distinct from the main image component. The diagram of the MIRL with 2 segments is illustrated in Figure 1c, where we divide the encoder into shallow and deep segments, and append a separate decoder to each. The shallow segment learns to reconstruct the main component of the masked content, while the deep segment is explicitly reformulated as learning the image residual. The MIRL framework is essentially equivalent to shifting the pre-training objective of deeper layers of ViT from image reconstruction to image residual reconstruction. This simple yet effective concept of image residual reconstruction significantly alleviates the degradation problem in deeper layers of ViT, making scaling ViTs along depth a promising direction for improving performance. By extending the MIRL framework and dividing the encoder into more segments, as illustrated in Figure 4, we can train deeper ViTs and readily achieve accuracy gains from substantially increased depth. Consequently, we instantiate deeper encoders: ViT-B-24, ViT-B-48, and ViT-S-54, comprising 24, 48, and 54 Transformer blocks, respectively. Notably, with similar computational complexity, our deeper ViT variants deliver considerably better generalization performance than the wider ViT encoders ( e.g . ViT-S-54 vs . ViT-B, ViT-B-48 vs . ViT-L), thanks to the increased depth. Meanwhile, our experiments in Sec. 4.2 demonstrate that employing additional featurelevel objectives [12, 6] or VGG loss [26, 11] can further improve performance, suggesting that the improvement directions of feature-level loss and MIRL are orthogonal and can complement each other.

Decoders:

H 1 , H 2 , H 3 , H 4

…

Φ

2

H

1

M

…

visible patches

Φ

1

…

M

random

masking

H

4

…

M

M

H

2

Φ

3

H

3

…

M

M

…

…

…

M

M

Φ

4

Loss

: visible  token

: masked  token

M

: encoding process

: decoding process

Encoder Segments: Φ

1

- Φ

2

- Φ

3

- Φ

4

Decoders:

H 1 , H 2 , H 3 , H 4

: shortcut connection

main

main+res

residual

main

n+

residual

2 Preliminaries

2.1 Revisit masked image modeling

Following the paradigm of Masked AutoEncoder(MAE) [21], the input image is split into a set of non-overlapping patches x = { x i } N i =1 , where x i ∈ R P 2 C denotes the image patch in the i -th position with P 2 resolution and C channels. The image patches are further tokenized into visual tokens via a linear projection, denoted by z 0 = { z i 0 } N i =1 2 . Subsequently, a random sampling strategy divides the indexes of patches into two subsets, V and M , where V ∩ M = ∅ . The image patches ( x V ) and visual tokens ( z V 0 ) with indexes in V are considered to be visible to the encoder. Thereafter, the encoder with L blocks only takes as input visible tokens z V 0 , and maps them to embedding features z V L . The objective of MIM is to predict the unseen content from x M by employing a decoder. A learnable mask token e [M] is introduced after the encoder, which is placed in M masked positions. Then the full set of encoded patches and mask tokens is processed by a small decoder to reconstruct the original image in pixels. The architecture of MAE can be described as:

where F ℓ ( · ) refers to the ℓ -th Transformer block in the encoder, H ( · ) denotes a shallow decoder. The objective loss of MIM is given by

where the reconstruction loss is only calculated in the masked positions.

2.2 A deep dive into autoencoders for MIM

We present three distinct autoencoder architectures in Figure 1 to clarify our motivation. Figure 1a shows the MAE framework [21]. Figure 1b depicts a truncated MAE with only the shallow ViT blocks in the encoder. Figure 1c presents a simplified MIRL with two decoders connected to the shallow and deep encoder segments. Models are pre-trained for 300 epochs with the same hyperparameters. Full setup details are in Appendix A of the supplementary materials.

Observation I: MIM pre-training can induce negative optimization in deeper layers of ViT. However, due to the overwhelmingly positive pre-training effect that MIM bestows upon the earlier blocks, its adverse influence on the latter blocks remains undiscovered. Given a ViT encoder pre-trained with MIM (Figure 1a), we substitute the pre-trained weights in the latter blocks of the encoder with the random parameters and subsequently fine-tune the encoder on ImageNet-1K. The curve plot of vanilla MAE, depicted in Figure 2, illustrates that applying random re-initialization to deeper blocks from the end of the encoder can improve performance. The initial point in Figure 2 indicates the result of an MAE model that has been pre-trained and subsequently fine-tuned without any random initialization. As more shallow blocks are randomly re-initialized, accuracy declines. Intriguingly, random initialization generalizes better than MIM pre-training in deeper layers of ViT, defying intuition.

Observation II: Performing MIM pre-training on fewer layers can lead to better efficiency and effectiveness. A truncated MAE illustrated in Figure 1b requires less pre-training time than the vanilla MAE while still achieving better or comparable performance. For fine-tuning, we initialize a complete ViT model with pre-trained weights from the truncated MAE. Regarding the blocks that are not included in the truncated MAE, we apply random initialization. As shown in Figure 2, when truncating 3 blocks from the end of the encoder, the fine-tuned model has better accuracy than the rest ones. By using the truncated MAE, we only pre-train 4 blocks and achieve similar fine-tuning accuracy as the vanilla MAE, reducing pre-training cost by 66%.

Observation III: Learning to recover image residual is a more productive pre-training objective. A simplified version of MIRL, shown in Figure 1c, formulates the pre-training objective for deeper layers as learning image residual, promoting more vivid image detail recovery and imposing a positive pre-training effect on deeper layers. Figure 3 demonstrates that MIRL achieves the highest fine-tuning accuracy among the three MIM pre-training schemes.

In summary, Observations I and II expose a pre-training degradation problem in ViT's deeper layers, leading to sub-optimal solutions. The same issue is also observed in the BEiT [1] paradigm, potentially attributed to depth limitations in MIM. In Observation III, we introduce MIRL to alleviate the degradation problem in deeper layers. The rest of the paper demonstrates how we employ MIRL to tackle the challenges of training deeper Vision Transformers.

3 Method

3.1 Masked Image Residual Learning (MIRL)

Upon observing that deeper layers pre-trained by MIM underperform against those with random initialization, we infer that the weight parameters of these deeper layers have indeed been updated during MIM pre-training, but in an unfavorable direction. In contrast, the shallower layers demonstrate improved performance after MIM pre-training. This leads us to speculate that the depth of the layers could be the root cause of the degradation problem.

To alleviate degradation in deeper ViTs during pre-training, we propose letting the deeper Transformer blocks learn to predict the residual of the masked image, rather than directly predicting the masked image itself. An overview of the MIRL framework is illustrated in Figure 4. Specifically, the encoder is partitioned into multiple segments, with each segment being followed by a separate small decoder. Subsequently, we establish shortcut connections between the shallower and deeper decoders. We underscore that these shortcut connections constitute the core of our method. This configuration fosters a seamless collaboration between very shallow and deep Transformer blocks in corrupted image reconstruction: the shallower segment learns to reconstruct the main component of the masked image, while the deeper segment learns the image residual. During pre-training, the established shortcut connections enable back-propagation to affect both the deeper and shallower layers simultaneously. This intertwined relationship between the deeper and shallower layers implies that the pre-training should either guide both towards a beneficial direction or lead both astray. With the introduction of the MIRL framework, our experimental results indicate that the shallower layers have, in essence, steered the deeper layers towards a more favorable direction.

Formally, we reformulate the encoder in Eq. (1) by evenly grouping the encoding Transformer blocks into G segments:

where Φ g denotes a stack of encoding blocks in the g -th segment. In the output of each encoding segment z V g , the masked positions are filled with a shared masked token e [M] , denoted as u g = z V g ∪ { e [M] : i ∈ M} N i =1 . Subsequently, for the g -th segment and the ( G -g +1 )-th segment selected from bottom-up and top-down directions, two separate decoders H g and H G -g +1 are appended for feature decoding. Let us consider that the g -th shallower segment learns a mapping function ˆ x g = H g ( u g ) producing the main component of the reconstructed image ˆ x g . Thereafter, we let ˆ ξ g = H G -g +1 ( u G -g +1 ) from the ( G -g +1 )-th deeper segment asymptotically approximate the residual ξ g = x -ˆ x g . The objective loss L g for the g -th segment is defined as:

Different from the residual learning in [23], our image residual learning would not fit an identity mapping, considering that the inputs to the two segments are different. See Appendix B.2 for further discussions on an alternative form of L g . The final loss function is formed by accumulating all 2 G reconstruction loss terms:

where λ g is the scaling coefficient, which is set to 2 G by default. Additional pre-training objective losses, such as the feature-level loss used in [6, 54] and the VGG loss [26], can be employed to enhance performance. We provide the definitions of other loss terms in Appendix B.1. However, as indicated in the ablation study in Sec.4.2, incorporating additional loss terms introduces nonnegligible overhead during pre-training. By default, we solely utilize the per-pixel loss defined in Eq.(6).

Densely Interconnected Decoding (DID). We design a densely interconnected decoding (DID) module, inserted into the decoders across different segments, enabling access to the features produced by previous segments. DID allows subsequent segments to avoid relearning features already acquired in earlier segments, thereby enhancing representation diversity. Immediately following the selfattention module in the first Transformer block in decoder H g , we insert a DID module, which is ingeniously implemented using the Multi-Head Attention mechanism, MHA( Q,K,V ) :

where the query Q is u g , while the key and value K,V are [ z V g -1 , ..., z V 0 ] . The dense interconnection property of the DID module is rendered by the concatenation form in MHA's key K and value V where we concatenate the output from previous encoding segments, thereby enabling feature integration at different depth levels.

3.2 Scaling to deeper ViT

Without modifying the Transformer block design, we instantiate deeper ViT variants by simply stacking more blocks. We consider the embedding hidden sizes in the Transformer block of 384 and 768. The details of the deeper ViTs are provided in Table 1. Choosing an appropriate hidden size is non-trivial for scaling up Vision Transformers, considering that a larger hidden size ( e.g ., 1024 adopted in ViT-Large and 1280 adopted in ViT-Huge) can cause instability due to very large values in attention logits, leading to (almost one-hot) attention weights with near-zero entropy. The instability of very wide ViTs is also reported in [21, 9].

By utilizing MIRL, we demonstrate that deeper ViTs exhibit stronger or comparative generalization capabilities when compared to their shallower and wider counterparts. With similar computational

0M

50M

100M

150M

200M

250M

300M

350M

#parameters

81

82

83

84

85

Accuracy (%)

ViT-S-54

ViT-B-24

ViT-B-48

ViT-S

ViT-B

ViT-L

ViT-S

ViT-B

MIRL

MAE

200

400

600

800

1000

pre-training epochs

82

83

84

85

86

Accuracy (%)

ViT-B-48

ViT-S-54

ViT-B

complexity, ViT-S-54 generalizes better than ViT-B. With only 31% of the computational cost of ViT-L, ViT-S-54 delivers performance on par with ViT-L. Notably, ViT-B-48 not only that it achieves higher performance but also provides a more stable training than ViT-L. This suggests that deepening ViTs could be a promising direction for enhancing vision model performance. Furthermore, MIRL helps to alleviate the training difficulties typically encountered in deeper ViTs, unlocking their potential and making them more effective for a variety of downstream tasks.

4 Experiment

The proposed MIRL method is evaluated on image classification, object detection and semantic segmentation tasks. All models are pre-trained on ImageNet-1K and then fine-tuned in downstream tasks. The input size is 224 × 224 , which is split into 196 patches with a size of 16 × 16 .

Pre-training setup. We pre-train all models on the training set of ImageNet-1K with 32 GPUs. By default, ViT-B-24 is divided into 4 segments, while ViT-S-54 and ViT-B-48 are split into 6 segments, and others into 2. Each appended decoder has 2 Transformer blocks with an injected DID module. We follow the setup in [21], masking 75% of visual tokens and applying basic data augmentation, including random horizontal flipping and random resized cropping. Full implementation details are in Appendix A.

4.1 Instantiations of deeper ViT

We compare the performance of the deeper ViTs detailed in Sec.3.2 with the ViT instantiations presented in [13]. As illustrated in Figure 5, we can easily gain accuracy from increased depth by leveraging MIRL for pre-training. In particular, ViT-S-54, which has the same level of computational complexity as ViT-B but is 4 × deeper than ViT-B, significantly outperforms ViT-B and even achieves performance on par with ViT-L pre-trained with MAE. Likewise, ViT-B-48 surpasses ViT-L while maintaining the same level of computational cost. Furthermore, the encoder pre-trained with MIRL consistently delivers higher performance than the one pre-trained with MAE.

4.2 Ablation studies

Various objective functions. Our approach is a general masking modeling architecture, seamlessly complementing prior methods that propose various objective functions. We study the compatibility of the recent feature-level loss and perceptual loss ( i.e . VGG loss) with our method. The results listed in Table 2a show that incorporating these objective functions can further improve the model, which demonstrates the generality of MIRL. Nonetheless, the additional loss terms will introduce heavy computational overhead. To accelerate our experiments, we have not used them by default.

Number of segments. The purpose of dividing an encoder into more segments is to construct auxiliary reconstruction losses to facilitate the training of the intermediate layers. We observe that these auxiliary reconstruction branches can amplify the gradient magnitude, potentially improving

(b) Numbers of segments (seg.) in the autoencoder of MIRL.

(c) Effect of Densely Interconnected Decoding.

(e) MIRL vs . coarse and fine separation.

the optimization of deeper Transformer blocks. Table 2b reports the performance of MIRL with respect to various numbers segments. For ViT-B-24, the configuration with 2 segments shows lower accuracy than the one with 4 segments. However, further splitting the encoder into more segments brings no more performance gain.

Effect of densely interconnected decoding. Considering that the auxiliary reconstruction branches from different segments adopt the same objective metric, the proposed DID establishes a feature reuse mechanism, preventing layers at various depth levels from learning similar feature representations.

Table 2c demonstrates that the MIRL models embodying the DID module yield higher fine-tuning accuracy than those without the DID. Deeper models gain more advantages from DID.

MIRL vs . simple multi-decoders. In order to demonstrate that image residual learning, powered by the shortcut connections, is the key to effective training deeper ViTs, we construct a segmented autoencoder with multiple decoders. Unlike MIRL, each decoder in the multi-decoder model independently learns to reconstruct the masked content. As shown in Table 2d, MIRL achieves substantially higher accuracy than the simple multi-decoder approach. Notably, the performance gap between MIRL and the multi-decoders widens as more Transformer blocks are stacked. When pre-training with multi-decoders, the deeper ViT seems to gain accuracy from increased depth. However, this does not imply that the multi-decoder approach addresses the degradation problem. Since replacing the weights of its deeper layers with random weights does not lead to a performance drop, the trivial improvement is attributed to the increased number of shallower layers.

MIRL vs . coarse and fine separation. As the reconstructed image residual shows some fine-grained details images, it is intriguing to know what pre-training results can be produced by replacing the reconstruction targets with the coarse and fine image components separated by using a Laplacian of Gaussian operator. We construct a segmented autoencoder with multiple decoders, referred to as 'coarse-to-fine', in which the reconstruction targets of the shallower and deeper segments correspond to the coarse and fine image components, respectively. 'fine-to-coarse' denotes the reversed targets compared to the "coarse-to-fine" configuration. Table 2e indicates that the segmented autoencoder with fine and coarse reconstruction targets achieves lower accuracy than MIRL, demonstrating that the main and residual components are not equivalent to the fine and coarse components.

Training schedules. So far, we have only trained our models using a relatively short pre-training schedule of 300 epochs. Note that deeper ViTs gain more advantages from longer pre-training schedules, compared to shallower ViTs. We extend pre-training to 1000 epochs and record fine-tuning performance for various pre-training lengths. To resume pre-training from previous checkpoints, we use a step decay learning rate scheduler, decaying the learning rate by a factor of 10 at 90% and 95% of the specified pre-training length. Figure 6 shows that ViT-B tends to plateau after 800 pre-training epochs, while ViT-S-54 keeps improving even after 1000 epochs. This implies that deeper ViTs' potential can be further unleashed by adopting a very long pre-training schedule, such as 1600 epochs.

4.3 Image classification on ImageNet-1K

We compare our models with previous results on ImageNet-1K. Hyperparameters are provided in Appendix A. For ViT-B, MIRL pre-trained for 300 epochs achieves 83.5% top-1 fine-tuning accuracy, comparable to MAE (83.6%) pre-trained for 1600 epochs. Our pre-training is 5 . 3 × shorter, demonstrating the high efficiency of MIRL. MIRL alleviates degradation in deeper ViTs, showing impressive generalization. In an 800-epoch pre-training scheme, the deeper encoder ViTS-54 produces 84.8% accuracy, which is 1.7% higher than ViT-B (83.1%) pre-trained with MAE and only 0.6% lower than ViT-L (85.4%). ViT-B-48, with computational complexity similar to ViT-L but 2 × deeper, achieves 85.3% and 86.2% accuracy with 300 and 800-epoch pre-training schemes, outperforming the ViT-L models pre-trained by other listed methods. Furthermore, the deeper encoders can further benefit from very long pre-training schedules, as discussed in Sec. 4.2.

4.4 Object detection and segmentation on COCO

To evaluate the generalization capabilities of our approach, we transfer our pre-trained models to the object detection task. The experiment is conducted on MS COCO [30] on account of its wide use. Following [21], we choose Mask R-CNN [22] as the detection framework and trained with the 1 × schedule. For fair comparisons, we adopt the identical training configurations from mmdetection [3], and Average Precision (AP) is used as the evaluation metric. As summarized in Table 4, MIRL outperforms all the listed methods.

4.5 Semantic segmentation on ADE20K

We compare our method with previous results on the ADE20K [61] dataset, utilizing the UperNet framework for our experiments, based on the implementation provided by [1] (see Appendix A for training details). The evaluation metric is the mean Intersection over Union (mIoU) averaged across all semantic categories. We employ pre-trained ViT-B-48 as the backbone, which has a computational

cost similar to ViT-L. The results are summarized in Table 5. The segmentation model using ViT-B-48 achieves competitive results compared to ViT-L pre-trained with BEiT [1] and MAE [21]. This indicates that the instantiated deeper ViTs exhibit strong transferability to downstream vision tasks.

4.6 Limitation and discussion

While MIRL significantly alleviates training challenges for deeper ViTs, a comprehensive theoretical explanation for the effectiveness of image residual reconstruction in training deeper ViTs remains elusive. We provide some insights into why MIRL might work well for deeper ViTs: 1) By reformulating the pre-training objective to recover the masked image's residual, MIRL implicitly encourages the model to focus on learning high-level contextual information and fine-grained details that are otherwise difficult to capture. 2) MIRL could stabilize gradient flow and enhance learning dynamics for deeper layers in ViTs, as evidenced by a larger gradient norm of the encoder in MIRL compared to vanilla MIM (see gradient visualization in Appendix C). Despite these insights, further theoretical analysis and investigation are required to fully understand MIRL's effectiveness in training deeper ViTs. The deepest ViT presented in this research comprises only 54 blocks. We anticipate that a depth of 54 is far from the upper limit for scaling ViT along the depth dimension. These areas are left for future work.

5 Related Work

Self-supervised learning. After the triumph of BERT [10] and GPT [35] models in NLP, selfsupervised learning (SSL) has undergone a paradigm shift, replacing the conventional supervised learning approach [23, 43], and has made remarkable advancements in numerous domains [32, 60, 27]. Larger datasets, new training methods and scalable architectures [31, 13, 34, 59] have accelerated this growth. In computer vision, inspired by the success of BEiT [1], recent research [15, 29, 18, 24, 19, 20, 58, 2, 37, 33, 49] has explored adapting the transformer architecture to the task of image self-supervised domain. After that, the emergence of MAE [21] has further led to a resurgence of interest in reconstruction-based masked image methods, such as [21, 14, 51, 56, 11]. We are particularly intrigued by these masked methods, as they have shown state-of-the-art performance on numerous transfer tasks and are computationally efficient. This has motivated us to introduce MIRL, a novel approach that builds on these methods.

Relation to scaling models. Scaling deeper ConvNets [23, 43, 39, 42, 40, 41] is an effective way to attain improved performance, but the same cannot be easily achieved with ViTs [38]. While the

res. + main

residual

main

masked

res. + main

residual

main

res. + main

residual

mp

masked

res.+ main

residual

main

Transformer architecture has succeeded in building large-scale language models [7, 53, 36, 44, 8, 17], the implementation of scalable Transformers for visual models still significantly lags behind. Recent work [62, 28, 47, 9, 59] has endeavored to explore deep Transformer-like models. These studies introduce necessary modifications to the original Transformer architecture, such as parallel layers, altered layer norm positions, composite attention mechanisms, larger embedding dimensions, unique optimization strategies, and exhaustive hyperparameter searches. Although they have demonstrated commendable performance, they lack a guiding principle about how to deepen or enlarge the Transformer-like models. Contrary to previous methods, our approach is rooted in an in-depth analysis, dissecting the standard ViT architecture. This allows us to identify the challenges in fully realizing the potential deeper ViTs and develop effective solutions accordingly. Building upon the principles we proposed, we efficiently construct deep-scale ViT models.

6 Conclusion

In this paper, we first reveal a performance degradation problem in Vison Transformers (ViTs) when pre-training with masked image modeling (MIM). Through an in-depth experimental analysis, we determine that the degradation is caused by the negative optimization effect of MIM enforced on deeper layers of ViT. We then introduce a novel concept of masked image residual learning (MIRL) to establish a self-supervised learning framework, aimed at alleviating the performance degradation problem. Leveraging MIRL, we unleash the potential of deeper ViTs and instantiate deeper encoders, including ViT-S-54, ViT-B-24 and ViT-B-48. These deeper ViTs variants exhibit superior generalization performance on downstream tasks.

Broader impacts. The proposed approach, which predicts content from training data statistics, may reflect biases with adverse societal impacts and generate non-existent content, underlining the need for further research in this area.

References

Hangbo Bao, Li Dong, and Furu Wei. BEiT: BERT pre-training of image transformers. ICLR , 2021. 4, 7, 8, 9

Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906 , 2021. 9

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. CoRR , 2019. 8

Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.03026 , 2022. 1

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV , 2021. 7, 9

Yabo Chen, Yuchen Liu, Dongsheng Jiang, Xiaopeng Zhang, Wenrui Dai, Hongkai Xiong, and Qi Tian. Sdae: Self-distillated masked autoencoder. In ECCV , 2022. 1, 2, 5

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022. 10

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 , 2022. 10

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442 , 2023. 5, 10

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL , 2019. 1, 9

Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprint arXiv:2111.12710 , 2021. 2, 9

Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining. In ECCV , 2022. 2

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR , 2021. 1, 6, 9

Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112.10740 , 2021. 9

Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In ICML , 2021. 9

Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training. ICLR , 2023. 7

William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021. 10

Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In CVPR , 2020. 9

Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Online bag-of-visual-words generation for unsupervised representation learning. arXiv preprint arXiv:2012.11552 , 2020. 9

Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self-supervised pretraining of visual features in the wild. CVPR , 2021. 9

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022. 3, 5, 6, 7, 8, 9

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , 2017. 8

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016. 5, 9

Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In ICML , 2020. 9

Ziyu Jiang, Yinpeng Chen, Mengchen Liu, Dongdong Chen, Xiyang Dai, Lu Yuan, Zicheng Liu, and Zhangyang Wang. Layer grafted pre-training: Bridging contrastive learning and masked image modeling for label-efficient representations. ICLR , 2023. 1

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV , 2016. 2, 5, 7

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General visual representation learning. In ECCV , 2020. 9

Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668 , 2020. 10

Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966 , 2020. 9

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014. 8

Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV , 2018. 9

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering , 2010. 9

Xiangyu Peng, Kai Wang, Zheng Zhu, and Yang You. Crafting better contrastive views for siamese representation learning. In CVPR , 2022. 9

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021. 9

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAI , 2018. 9

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 , 2019. 10

Sucheng Ren, Fangyun Wei, Samuel Albanie, Zheng Zhang, and Han Hu. Deepmim: Deep supervision for masked image modeling. arXiv preprint arXiv:2303.08817 , 2023. 1, 9

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In NeurIPS , 2021. 9

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. 9

Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387 , 2015. 9

Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI , 2017. 9

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR , 2015. 9

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML , 2019. 9

Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131 , 2022. 10

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML , 2021. 9

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 2017. 1

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.00555 , 2022. 10

Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, and Zhaoxiang Zhang. Hard patches mining for masked image modeling. In CVPR , 2023. 7

Haoqing Wang, Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhi-Hong Deng, and Kai Han. Masked image modeling with local multi-scale reconstruction. CVPR , 2023. 7, 9

Shaoru Wang, Jin Gao, Zeming Li, Jian Sun, and Weiming Hu. A closer look at self-supervised lightweight vision transformers. arXiv preprint arXiv:2205.14443 , 2022. 1

Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:2112.09133 , 2021. 9

Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked

feature prediction for self-supervised visual pre-training. In CVPR , 2022. 7

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022. 10

Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. Tech Report , 2022. 5

Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. arXiv preprint arXiv:2205.13543 , 2022. 1

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. CVPR , 2021. 7, 9

Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. ICLR , 2023. 7

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230 , 2021. 9

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR , 2022. 9, 10

Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867 , 2019. 9

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR , 2017. 8

Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886 , 2021. 10

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper finds that the deep layers of ViT fail to benefit from MIM pre-training. The authors replace deeper layers of MAE pre-trained ViTs with random initialization and demonstrate that this modified model achieves better performances than the original MAE pre-trained model. To this end, the authors conclude that MIM pre-training is only effective for shallow layers. To address this, Masked Image Residual Learning (MIRL) is proposed. Experiments under various benchmarks demonstrate the efficacy of the proposed MIRL method.\n\nSTRENGTHS:\n1. This paper is well-written and easy to follow.\n2. The authors have systematically analyzed whether MIM pre-training is effective for deeper layers of ViTs.\n3. The proposed method and the motivation are highly connected.\n4. Experiments are sufficient and the code is provided in the supplementary material.\n\nWEAKNESSES:\n**Major Concerns:**\n\n1. **Unfair comparisons against other methods.** I have carefully checked the fine-tuning configuration of both MIRL (Table 2 in the supplementary material) and MAE [21], and find that *MIRL incorporates an extra exponential moving average (EMA)* while others do not, which appears to be an unfair comparison. It is better to fine-tune all models pre-trained by MIRL *without* EMA. Also, several hyper-parameters have been carefully tuned compared with MAE [21], such as base learning rate, batch size, and warmup epochs. Therefore, it is better to compare with MAE which is fine-tuned under the exact same configuration.\n2. **Lack of intuitive explanation of residual learning.** I recognize the efficacy of learning residual in MIM, but the underlying motivation remains unclear. In other words, given the observation that deep layers of ViT fail to benefit from MIM pre-training, why learning residual can alleviate this problem?\n3. **Observation I and II cannot be reflected in Figure 2 by using a single plot named \"truncated MAE\".** If my understanding is correct, models of these two observations are pre-trained in different ways. Specifically, the model of Observation I should be pre-trained by vanilla MAE while the model of Observation II should be pre-trained by truncated MAE. Therefore, these two observations cannot be reflected in Figure 2. It is better to plot three curves in Figure 2, including vanilla MAE, vanilla MAE with truncated fine-tuning (Observation I), and truncated MAE (Observation II).\n\n**Minor Questions/Suggestions:**\n\n4. Typo in Eq. (6). It should be $||\\mathbf{x}^i - \\hat{\\mathbf{x}}_g^i - \\hat{\\xi}_g^i||_2^2$.\n5. It is better to combine Table 1 in the main submission with Table 4 in the supplementary material to better compare the configuration (including #params and FLOPs) of different backbones.\n6. It is encouraged to plot vanilla MAE with ViT-B and ViT-L in Figure 6 to demonstrate the scaling behavior of the proposed method.\n7. It is better to compare deep ViTs pre-trained with vanilla MAE.\n8. The configuration of Table 2a is not clear. Is residual learning performed on both pixel and feature?\n9. The setting of Table 2d is confusing. Its caption says \"multi-encoder\" while in lines 203-212 is \"multi-decoder\".\n\nQUESTIONS:\nI do not have any questions. Please refer to the weaknesses section. I am willing to raise my rating if the authors succeed in demonstrating that their improvements come from the proposed method and not from carefully tuned hyperparameters during fine-tuning.\n\n**Post Rebuttal Comments**\n\nThanks for the rebuttal and additional comments. My concerns are well addressed. Since using EMA in fine-tuning will not affect the performance significantly, the reviewer thinks it might be a fair comparison. Also, the explanation of why residual learning works is intuitive. Therefore, I will raise my rating to 6 and the authors are supposed to revise their paper according to our dialog, including (1) clarification about the EMA, (2) an intuitive explanation about why it works, and (3) comparison with CVPR'23 works.\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a new mask image modeling method, which is helpful for deep ViT model pretraining.\n\nSTRENGTHS:\nPlease refer to Questions\n\nWEAKNESSES:\nPlease refer to Questions\n\nQUESTIONS:\n### strength\n1. The paper is well-written and easy to follow\n2. The proposed method starts from an interesting observation and is novel.\n3. The experiment results seem good.\n\n### Weakness\n1. The experiment lacks some necessary reuslts. For example, ViT-L is deeper than ViT-B and has 24 layers by default, it may benefit from the proposed method. This would be a fair comparison and further prove the effectiveness of the proposed method.\n2. Following 1, there is no baseline for the deeper models except Table.2(b). So when comparing with MAE, the gain of the proposed method is not clear. \n3. I'm confused about the design of ViT-S-54, why use 54 layers? if it uses 48 layers, its params and FLOPs will be more similar to ViT-B and the comparison would be more fair.\n4. I also have a question about the method design, why the segments are divided into two groups to predict content and residual? Take Fig4 as an example, denote input as x, if H1 predicts x0, H2 predicts x1 = x-x0, H3 further predicts x2 = x-x0-x1 and so on, such a sequential design seem more consistent with the model architecture, will it perform better?\n5. A question about detail, MAE uses the pixel-normed patch as the prediction target, which is critical for its performance. Is this also used in the proposed method and the reported MAE baselines(300/800)?\n6. Some baseline methods are missed. For example, MaskFeat[1] for ViT-B, Data2Vec[2], PeCO[3], BootMAE[4] and CAE[5] for ViT-B/L.\n7. A small typo in Table.2(d), should it be multi-decoders?\n\n[1] Masked Feature Prediction for Self-Supervised Visual Pre-Training\n\n[2]data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language\n\n[3]PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers\n\n[4]Bootstrapped Masked Autoencoders for Vision BERT Pretraining\n\n[5]Context Autoencoder for Self-Supervised Representation Learning\n\nLIMITATIONS:\nPlease refer to Questions", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into the degradation issue encountered in the deeper layers of Vision Transformer (ViT) and proposes a self-supervised learning framework called Masked Image Residual Learning (MIRL). MIRL reformulates the learning objective to recover the residual of the masked image and makes scaling ViT along depth a promising direction for enhancing performance. Experimental results demonstrate promising improvements on various downstream tasks with increased depth, highlighting the effectiveness of the MIRL method.\n\nSTRENGTHS:\n1.Tackling the degradation issue in training deeper vision-Transformers (ViT) in the context of MIM pre-training is a practical problem and is also a under-explored area of research. \n\n2.This paper presents several interesting observations and shows the MIM introduces negative optimization in the deeper layers of ViT that would hinder further pre-training performances. Thre proposed method is simple and effective for solving this problem.\n\n3.The authors conduct extensive and comprehensive experiments that show the effectiveness and efficiency of the proposed method.\n\nWEAKNESSES:\n1.While the proposed method has shown promising results, it still lacks an in-depth explanation, e.g., choosing the training objective for encouraging deep layers to recover the image residual. The rationale behind dividing different segments and providing the proposed reconstruction terms could benefit from offering additional theoretical justifications, such as utilizing tools like Mutual Information.\n\n2.It would be beneficial to provide further clarity on the mechanism to ensure precise reconstruction of the main component and residual of the image in the shallower and deeper parts, respectively. Given that they share the same training objective, disentangling them poses a nontrivial challenge.\n\n3.An ablation study could be added to investigate whether assigning different regularization weights to different segment loss parts is beneficial, whether the MIRL approach incurs additional training time and other costs.\n\nQUESTIONS:\nTraining the vision-Transformers with varying scales often requires distinct hyper-parameters. For instance, the optimal hyper-parameters for training ViT-Large and ViT-Small typically differ. It would be interesting to investigate if the proposed MIRL approach can alleviate this phenomenon when training vision Transformers with varying scales.\n\nOverall, this paper addresses an important problem and provides some insights through observations and the proposed solutions. While it leans more towards empirical findings, it would be beneficial to provide additional theoretical evidence to support the claims.\n\nLIMITATIONS:\nSee the above comments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper clarifies that MIM pretraining can induce negative optimization in deeper layers of ViT through comparison between vanilla MAE and truncated MAE. Based on this observation, this paper proposes a MAE-based framework named Masked Image Residual Learning (MIRL) for alleviating the degradation problem and training deeper ViT. In MIRL, The objective of MIM is decoupled to recovering the main component of images from the features of shallower layers, and recovering the residual from the features of deeper layers. Ablation experiments shows that MIRL outperforms MAE and MIM baselines with multiple decoders, and help deeper ViTs (e.g., ViT-S-54 and ViT-B-48) achieve better performance compared to previous methods on ImageNet.\n\nSTRENGTHS:\n1. In this paper, the degradation problem in the MIM pretraining of ViT is illustrated by well-designed experiments.\n\n2. Training deeper ViT with MIM has not been widely studied in previous works.\n\n3. The idea of recovering the residual of images is interesting, and ablation experiments show the effectiveness of MIRL.\n\nWEAKNESSES:\n1. Although this paper has cited some previous works that provide supervision to different layers of ViT (like deepMIM[1]), there is no comparison between MIRL and these previous methods in this paper.\n\n2. Using multiple decoders in MIRL will significantly increase training time compared with MAE. A comparison of training time between MIRL and MAE should be included.\n\n3. On the dense prediction tasks, MIRL does not show significant improvement (even weaker performance) compared to MAE with a similar number of parameters. \n\n4. This paper says that MIRL alleviates the degradation problem in deeper layers of ViT. However, This paper does not provide evidence on whether the phenomenon in observation II still exists in the proposed method. \n\n5. The baseline results of pretraining deeper ViTs (like ViT-S-54 ViT-B-48) with MAE are not provided in this paper.\n\n[1] Sucheng Ren, Fangyun Wei, Samuel Albanie, Zheng Zhang, and Han Hu. Deepmim: Deep supervision for masked image modeling. arXiv preprint arXiv:2303.08817, 2023.\n\nQUESTIONS:\n1. The results in Table 2(d) show the comparison between MIRL and MIM baselines with multiple decoders. Did the baseline method here use DID? If not, can \"residual learning + DID\" significantly outperform \"multi-decoders + DID\"? \n\n2. A comparison of training time between MIRL and MAE should be included.\n\nOther questions have been mentioned in weaknesses.\n\nLIMITATIONS:\nThe authors have addressed the limitations of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
* This paper propose a Masked Image Modeling pretraining framework,  which boost the pretraining performance in deeper deeper layers of ViT architecture.

* Deeper vision transformers are hard to train, so authors introduce a new pretraining objective for mim, which can  alleviate the degradation problem in deeper vit.

* MIRL segment the encoding blocks into several blocks(such as 4) and use different decoders to reconstruct the target.

* various experiments are conducted, with less pre-training time, MIRL can achieve competitive performance compared to other mim approaches especially in deeper vit.

STRENGTHS:
* well motivated and well written, training and pretraining in deeper vit architecture is important, experiments show that pretraining in deeper vit-base can get similar result with vit-base.

* The downstream task performance(in table 4) show superior performance than MAE framework, +2.5 bbox mAP, +1.8 mask AP.

WEAKNESSES:
* Since MIRL with deeper layers is different from the standard vit architecture(more deep than original vit), the fully supervised performance of the the same layers should be reported, the compare is a little in Table 3.

* I am interested that would this framework be also work in convnet mask image modeling framework(such as simmim, spark) ?

QUESTIONS:
please refer to the weakness

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-10