SlowGuess/NeurIPS_2023_OCR huggingface dataset:
the optimization of deeper Transformer blocks. Table 2b reports the performance of MIRL with respect to various numbers segments. For ViT-B-24, the configuration with 2 segments shows lower accuracy than the one with 4 segments. However, further splitting the encoder into more segments brings no more performance gain.
Effect of densely interconnected decoding. Considering that the auxiliary reconstruction branches from different segments adopt the same objective metric, the proposed DID establishes a feature reuse mechanism, preventing layers at various depth levels from learning similar feature representations.
Table 2c demonstrates that the MIRL models embodying the DID module yield higher fine-tuning accuracy than those without the DID. Deeper models gain more advantages from DID.
MIRL vs. simple multi-decoders. In order to demonstrate that image residual learning, powered by the shortcut connections, is the key to effective training deeper ViTs, we construct a segmented autoencoder with multiple decoders. Unlike MIRL, each decoder in the multi-decoder model independently learns to reconstruct the masked content. As shown in Table 2d, MIRL achieves substantially higher accuracy than the simple multi-decoder approach. Notably, the performance gap between MIRL and the multi-decoders widens as more Transformer blocks are stacked. When pre-training with multi-decoders, the deeper ViT seems to gain accuracy from increased depth. However, this does not imply that the multi-decoder approach addresses the degradation problem. Since replacing the weights of its deeper layers with random weights does not lead to a performance drop, the trivial improvement is attributed to the increased number of shallower layers.
MIRL vs. coarse and fine separation. As the reconstructed image residual shows some fine-grained details images, it is intriguing to know what pre-training results can be produced by replacing the reconstruction targets with the coarse and fine image components separated by using a Laplacian of Gaussian operator. We construct a segmented autoencoder with multiple decoders, referred to as "coarse-to-fine", in which the reconstruction targets of the shallower and deeper segments correspond to the coarse and fine image components, respectively. "fine-to-coarse" denotes the reversed targets compared to the "coarse-to-fine" configuration. Table 2e indicates that the segmented autoencoder with fine and coarse reconstruction targets achieves lower accuracy than MIRL, demonstrating that the main and residual components are not equivalent to the fine and coarse components.
Training schedules. So far, we have only trained our models using a relatively short pre-training schedule of 300 epochs. Note that deeper ViTs gain more advantages from longer pre-training schedules, compared to shallower ViTs. We extend pre-training to 1000 epochs and record fine-tuning performance for various pre-training lengths. To resume pre-training from previous checkpoints, we use a step decay learning rate scheduler, decaying the learning rate by a factor of 10 at  $90\%$  and  $95\%$  of the specified pre-training length. Figure 6 shows that ViT-B tends to plateau after 800 pre-training epochs, while ViT-S-54 keeps improving even after 1000 epochs. This implies that deeper ViTs' potential can be further unleashed by adopting a very long pre-training schedule, such as 1600 epochs.
4.3 Image classification on ImageNet-1K
We compare our models with previous results on ImageNet-1K. Hyperparameters are provided in Appendix A. For ViT-B, MIRL pre-trained for 300 epochs achieves  $83.5\%$  top-1 fine-tuning accuracy, comparable to MAE  $(83.6\%)$  pre-trained for 1600 epochs. Our pre-training is  $5.3\times$  shorter, demonstrating the high efficiency of MIRL. MIRL alleviates degradation in deeper ViTs, showing impressive generalization. In an 800-epoch pre-training scheme, the deeper encoder ViTS-54 produces  $84.8\%$  accuracy, which is  $1.7\%$  higher than ViT-B  $(83.1\%)$  pre-trained with MAE and only  $0.6\%$  lower than ViT-L  $(85.4\%)$ . ViT-B-48, with computational complexity similar to ViT-L but  $2\times$  deeper, achieves  $85.3\%$  and  $86.2\%$  accuracy with 300 and 800-epoch pre-training schemes, outperforming the ViT-L models pre-trained by other listed methods. Furthermore, the deeper encoders can further benefit from very long pre-training schedules, as discussed in Sec. 4.2.
4.4 Object detection and segmentation on COCO
To evaluate the generalization capabilities of our approach, we transfer our pre-trained models to the object detection task. The experiment is conducted on MS COCO [30] on account of its wide use. Following [21], we choose Mask R-CNN [22] as the detection framework and trained with the  $1 \times$  schedule. For fair comparisons, we adopt the identical training configurations from mmdetection [3], and Average Precision (AP) is used as the evaluation metric. As summarized in Table 4, MIRL outperforms all the listed methods.
4.5 Semantic segmentation on ADE20K
We compare our method with previous results on the ADE20K [61] dataset, utilizing the UperNet framework for our experiments, based on the implementation provided by [1] (see Appendix A for training details). The evaluation metric is the mean Intersection over Union (mIoU) averaged across all semantic categories. We employ pre-trained ViT-B-48 as the backbone, which has a computational
Table 4: Object detection results with Mask R-CNN on MS-COCO. The models pre-trained with extra data or very long schedules are marked in gray.
Method	Backbone	Pre-training Data	Epochs	Detection APb	Segmentation APm
DeiT [45]	ViT-B	-	-	46.9	41.5
BEiT [1]	ViT-B	IN1K+DALLE	800	46.3	41.1
MAE [21]	ViT-B	IN1K	800	46.8	41.9
MAE [21]	ViT-B	IN1K	1600	48.4	42.6
MIRL	ViT-B	IN1K	800	49.3	43.7
MAE [21]	ViT-L	IN1K	1600	53.3	47.2
MIRL	ViT-B-48	IN1K	800	53.4	46.5
Table 5: Semantic segmentation results on ADE20K. The models pre-trained with extra data or very long schedules are marked in gray.
Method	Pre-training Data	Backbone	Epochs	mIoU
MoCo v3 [5]	IN1K	ViT-L	300	49.1
BEiT [1]	IN1K+DALLE	ViT-L	800	53.3
MAE [21]	IN1K	ViT-L	1600	53.6
MIRL	IN1K	ViT-B-48	800	53.2
cost similar to ViT-L. The results are summarized in Table 5. The segmentation model using ViT-B-48 achieves competitive results compared to ViT-L pre-trained with BEiT [1] and MAE [21]. This indicates that the instantiated deeper ViTs exhibit strong transferability to downstream vision tasks.
4.6 Limitation and discussion
While MIRL significantly alleviates training challenges for deeper ViTs, a comprehensive theoretical explanation for the effectiveness of image residual reconstruction in training deeper ViTs remains elusive. We provide some insights into why MIRL might work well for deeper ViTs: 1) By reformulating the pre-training objective to recover the masked image's residual, MIRL implicitly encourages the model to focus on learning high-level contextual information and fine-grained details that are otherwise difficult to capture. 2) MIRL could stabilize gradient flow and enhance learning dynamics for deeper layers in ViTs, as evidenced by a larger gradient norm of the encoder in MIRL compared to vanilla MIM (see gradient visualization in Appendix C). Despite these insights, further theoretical analysis and investigation are required to fully understand MIRL's effectiveness in training deeper ViTs. The deepest ViT presented in this research comprises only 54 blocks. We anticipate that a depth of 54 is far from the upper limit for scaling ViT along the depth dimension. These areas are left for future work.
Self-supervised learning. After the triumph of BERT [10] and GPT [35] models in NLP, self-supervised learning (SSL) has undergone a paradigm shift, replacing the conventional supervised learning approach [23, 43], and has made remarkable advancements in numerous domains [32, 60, 27]. Larger datasets, new training methods and scalable architectures [31, 13, 34, 59] have accelerated this growth. In computer vision, inspired by the success of BEiT [1], recent research [15, 29, 18, 24, 19, 20, 58, 2, 37, 33, 49] has explored adapting the transformer architecture to the task of image self-supervised domain. After that, the emergence of MAE [21] has further led to a resurgence of interest in reconstruction-based masked image methods, such as [21, 14, 51, 56, 11]. We are particularly intrigued by these masked methods, as they have shown state-of-the-art performance on numerous transfer tasks and are computationally efficient. This has motivated us to introduce MIRL, a novel approach that builds on these methods.
Relation to scaling models. Scaling deeper ConvNets [23, 43, 39, 42, 40, 41] is an effective way to attain improved performance, but the same cannot be easily achieved with ViTs [38]. While the
Figure 7: Visualization of MIRL. Example images are generated from the validation set on ImageNet.
Transformer architecture has succeeded in building large-scale language models [7, 53, 36, 44, 8, 17], the implementation of scalable Transformers for visual models still significantly lags behind. Recent work [62, 28, 47, 9, 59] has endeavored to explore deep Transformer-like models. These studies introduce necessary modifications to the original Transformer architecture, such as parallel layers, altered layer norm positions, composite attention mechanisms, larger embedding dimensions, unique optimization strategies, and exhaustive hyperparameter searches. Although they have demonstrated commendable performance, they lack a guiding principle about how to deepen or enlarge the Transformer-like models. Contrary to previous methods, our approach is rooted in an in-depth analysis, dissecting the standard ViT architecture. This allows us to identify the challenges in fully realizing the potential deeper ViTs and develop effective solutions accordingly. Building upon the principles we proposed, we efficiently construct deep-scale ViT models.
In this paper, we first reveal a performance degradation problem in Vison Transformers (ViTs) when pre-training with masked image modeling (MIM). Through an in-depth experimental analysis, we determine that the degradation is caused by the negative optimization effect of MIM enforced on deeper layers of ViT. We then introduce a novel concept of masked image residual learning (MIRL) to establish a self-supervised learning framework, aimed at alleviating the performance degradation problem. Leveraging MIRL, we unleash the potential of deeper ViTs and instantiate deeper encoders, including ViT-S-54, ViT-B-24 and ViT-B-48. These deeper ViTs variants exhibit superior generalization performance on downstream tasks.
Broader impacts. The proposed approach, which predicts content from training data statistics, may reflect biases with adverse societal impacts and generate non-existent content, underlining the need for further research in this area.
[60] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019. 9
[61] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 8
[62] Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886, 2021. 10