SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To evaluate the performance on the pretext task, i.e., reconstructing dropped positions, we also report the averaged top-1 accuracy of position predictions on the ImageNet-1K validation set in Tabs. 1 to 4. We vary  $\gamma \in \{0,0.25,0.5,0.75\}$  and  $\gamma_{\mathrm{pos}} \in \{0.25,0.5,0.75,0.95\}$  when measuring the position prediction accuracy, and average the position accuracy among 16 different cases.
Main properties. Illustrated by Tabs. 1 to 4, we find sufficient evidence to support the three difficulties claimed in Sec. 1. (i) In Tab. 1, ViTs fail to learn highly semantic representations by simply solving the position reconstruction task, because the performances of  $\gamma = 0$  significantly lag behind. (ii) Discrepancies between pre-training and fine-tuning are revealed in Tab. 2 by setting
Table 3: Ablation study of  $\sigma$ . "  $\sigma = 0$  " means no position smoothing. "  $\rightarrow$  " denotes a linear decay schedule is performed on  $\sigma$ . We report the averaged top-1 accuracy of position predictions.
σ	ImageNet-1K	COCO detection	COCO segmentation	ADE20k
fine-tune	position	APb	AP50	AP75	APm	AP50	AP75	mIoU	aAcc
0	82.85	88.81	40.32	59.89	43.90	36.45	57.09	38.88	38.32	79.22
1	82.91	87.13	40.19	59.76	43.81	36.30	57.06	38.87	38.60	79.17
2	82.79	69.48	40.16	59.94	43.92	36.32	57.32	38.88	38.69	79.30
1 → 0	82.96	87.83	42.14	61.99	46.38	37.93	59.23	40.76	40.68	80.14
2 → 0	82.88	87.65	40.46	60.03	44.21	36.53	57.20	39.30	38.79	79.43
Table 4: Ablation study of attentive reconstruction. “ $\tau = \infty$ ” indicates no attentive reconstruction. We report the averaged top-1 accuracy of position predictions.
τ	ImageNet-1K	COCO detection	COCO segmentation	ADE20k
fine-tune	position	APb	APb50	APb75	APm	APm50	APm75	mIoU	aAcc
∞	82.84	88.66	40.56	60.48	44.15	36.78	57.64	39.46	38.38	79.36
0.1	82.96	87.83	42.14	61.99	46.38	37.93	59.23	40.76	40.68	80.14
0.5	82.86	87.78	40.73	60.33	44.77	36.71	57.72	39.38	38.80	79.62
$\gamma_{\mathrm{pos}} = 1$  , where we observe performance deterioration. (iii) Deciding which position to reconstruct precisely is crucial because by setting  $\sigma = 0$  in Tab. 3 and  $\tau = \infty$  in Tab. 4, ViTs have better position predictions task but perform worse on downstream tasks.
Performance on the position reconstruction task. In general, a positive correlation is observed between the accuracy of the downstream task and the averaged accuracy of predicted positions, indicating that the averaged accuracy of position predictions is a reliable indicator of how well the model is trained. However, there are some exceptions, i.e.,  $\sigma = 0$  in Tab. 3 and  $\tau = \infty$  in Tab. 4, where the model reconstructs more precise positions but performs worse on downstream benchmarks, indicating that deciding which position to reconstruct precisely is crucial.
Please refer to Supplementary Material for the detailed performance of the position reconstruction task, where we provide more evidence to support the three difficulties proposed in Sec. 1.
Patch mask ratio  $\gamma$ . We vary the patch mask ratio  $\gamma$  from 0 to 0.75 in Tab. 1. When  $\gamma = 0$ , the entire image is visible during pre-training, resulting in nearly perfect position predictions and making the task too simple for self-supervised learning (see Supplementary Material for details). However, the averaged accuracy of position predictions is quite low since the accuracy deteriorates quickly as we enlarge  $\gamma$  during evaluation. Adopting a larger  $\gamma$  for pre-training leads to better performance, especially on detection and segmentation benchmarks. Therefore, we can conclude that patch masking is essential in DropPos to prevent trivial solutions. However, deterioration is observed when  $\gamma = 0.9$ . This is because the model may be under-fitted using this extremely difficult pretext task.
Position mask ratio  $\gamma_{\mathrm{pos}}$ . We study the effectiveness of the position mask ratio  $\gamma_{\mathrm{pos}}$  in Tab. 2. DropPos appears to be more robust against different  $\gamma_{\mathrm{pos}}$  than  $\gamma$ . However, an appropriate  $\gamma_{\mathrm{pos}}$  is still necessary. A small  $\gamma_{\mathrm{pos}}$  leads to trivial solutions even if we have a large  $\gamma$ , while an extremely large  $\gamma_{\mathrm{pos}}$ , e.g.,  $\gamma_{\mathrm{pos}} = 1$ , results in discrepancies between pre-training and fine-tuning.
Position smoothing. By setting different values of  $\sigma_0$  and  $\sigma_T$ , we study the effectiveness of position smoothing. By setting  $\sigma = 0$ , we remove the position smoothing strategy, and the model tends to be overwhelmed by this position reconstruction task, leading to high accuracy in position prediction but poor performance on downstream tasks. However, when  $\sigma$  is too large, the smoothed positions become noisy, contributing to poor performance on both downstream tasks and position reconstruction. Additionally, equipped with a linear decay schedule of  $\sigma$ , DropPos is gradually guided, bringing improvements on both downstream tasks and position reconstruction.
Attentive reconstruction. Attentive reconstruction is studied in Tab. 4 by setting different values of temperature  $\tau$ , where  $\tau = \infty$  means no attentive reconstruction. Without attentive reconstruction, DropPos is able to obtain better position predictions but performs worse on downstream tasks. This is because we do not have to reconstruct the exact positions when different patches share similar visual appearances. This phenomenon can be mitigated by setting an appropriate  $\tau$ . A large  $\tau$  leads
Table 5: Comparison with previous methods on ImageNet-1K classification. All methods are evaluated by fine-tuning. The resolution of images is fixed to  $224 \times 224$ .  $\dagger$  means our implementation.  $\ddagger$  means the result is borrowed from [28].
method	venue	eff. ep.	ViT-B	ViT-L
supervised	-	80.9†	82.6‡
MAE [28]	[CVPR'22]	200	82.2†	83.3‡
DropPos	[Ours]	200	83.0	83.7
Contrastive Learning
DINO‡ [6]	[ICCV'21]	1600	82.8	-
MoCo v3‡ [13]	[ICCV'21]	600	83.2	84.1
Masked Image Modeling
BEiT‡ [3]	[ICLR'22]	800	83.2	85.2
MAE‡ [28]	[CVPR'22]	1600	83.6	85.9
SimMIM [64]	[CVPR'22]	800	83.8	-
SemMAE [34]	[NeurIPS'22]	800	83.4	-
LocalMIM [56]	[CVPR'23]	1600	84.0	-
HPM [54]	[CVPR'23]	800	84.2	85.8
Masked Image Modeling + Contrastive Learning
iBOT [72]	[ICLR'22]	1600	84.0	-
BootMAE [19]	[ECCV'22]	800	84.2	85.9
Position Reconstructing
DropPos	[Ours]	800	84.2	85.8
Table 6: Comparison with previous methods on downstream tasks. All methods take the ViT-B/16 [21] as the backbone and utilize Mask R-CNN [30] on COCO [38] object detection and instance segmentation, and UperNet [63] on ADE20k [71] semantic segmentation, respectively.  $\ddagger$  means the result is borrowed from [28].  $\dagger$  indicates our implementation, including pre-training and supervised fine-tuning, while  $\sharp$  represents we reproduce fine-tuning using the official pre-trained backbone. We perform  $1\times$  schedule of fine-tuning on COCO using ViTDet [36], and 80k iterations of fine-tuning on ADE20k using mmsegmentation [14].
method	venue	eff. ep.	COCO detection	COCO segmentation	ADE20k
APb	APb50	APb75	APm	APm50	APm75	mIoU	aAcc
MAE†[28]	[CVPR'22]	200	40.1	60.5	44.1	36.4	57.8	39.3	40.5	80.1
DropPos	[Ours]	200	42.1	62.0	46.4	37.9	59.2	40.8	40.7	80.1
MoCo v3# [13]	[ICCV'21]	600	43.7	65.7	47.7	39.1	62.0	41.8	44.7	81.5
MAE#[28]	[CVPR'22]	1600	47.3	68.2	52.5	42.4	65.3	45.6	47.0	82.7
BootMAE#[19]	[ECCV'22]	800	47.3	67.9	52.1	42.3	65.0	45.8	47.3	83.0
SemMAE#[34]	[NeurIPS'22]	800	45.6	66.2	55.2	40.9	63.3	44.4	44.9	82.0
LocalMIM#[56]	[CVPR'23]	1600	47.4	67.7	52.2	42.2	64.8	45.5	47.1	83.1
DropPos	[Ours]	800	47.7	68.3	52.8	42.6	65.3	46.2	47.8	82.8
to noisy position ground truths, leading to poor performance on both downstream tasks and position reconstruction, which is the same as a large  $\sigma$ .
4.2 Comparisons with previous results
We compare our proposed DropPos with the supervised pre-training baseline and a wide range of self-supervised methods, including (i) contrastive learning methods [6, 13], (ii) masked image modeling methods [3, 28, 34, 54, 56, 64], and (iii) their combinations [19, 72]. Effective pre-training epoch is used for fair comparison following [54, 72] since it accounts for the actual trained images/view. The detailed definition can be found in Supplementary Material. All methods are pre-trained with the same resolution, i.e.,  $224 \times 224$  on ImageNet-1K [48].
ImageNet-1K classification. We compare our DropPos with state-of-the-art alternatives on the ImageNet-1K [48] classification benchmark in Tab. 5. Notably, with only 200 epochs pre-training,
Figure 3: Qualitative results of position reconstruction. We evaluate position predictions with different  $\gamma$  but fix  $\gamma_{\mathrm{pos}} = 0.95$ . Black patches are masked during inference. The positions of those white patches are wrongly predicted, while the remaining patches are predicted correctly. For each tuple, we show results under (a)  $\gamma = 0$ , (b)  $\gamma = 0.25$ , (c)  $\gamma = 0.5$ , (d)  $\gamma = 0.75$ , and (e) the original image. DropPos manages to reconstruct precise positions.
Table 7: An in-depth analysis about whether the improved position sensitivity benefits downstream tasks or not. We freeze the backbone and train an extra linear patch classification head.  $75\%$  of position embeddings are randomly masked during linear probing. We can conclude that image classification indeed needs better location awareness and thus designing a position prediction pretext task is crucial and worth studying.
method	venue	Position prediction	ImageNet-1K Top-1
Pre-trained	Fine-tuned
MoCo v3 [13]	[ICCV'21]	43.2	78.0 ↑ 34.8	83.2
MAE [28]	[CVPR'22]	63.1	82.5 ↑ 19.4	83.6
DropPos	[ours]	77.3	88.0 ↑ 10.7	84.2
DropPos achieves  $83.0\%$  and  $83.7\%$  using ViT-B/16 and ViT-L/16 as the backbone, surpassing the supervised baseline by  $+1.1\%$ , and MAE [28] by  $+0.8\%$  and  $+0.4\%$  respectively. This empirical evidence demonstrates that enhancing the location awareness of ViTs indeed brings better visual representations. Furthermore, with 800 epochs of pre-training, DropPos manages to achieve competitive results compared with the state-of-the-art. Specifically, it achieves  $84.2\%$  and  $85.8\%$  using ViT-B/16 and ViT-L/16, respectively. Strikingly, DropPos reaches competitive results with BootMAE [19], which combines contrastive learning and masked image modeling.
COCO object detection and segmentation. We fine-tune Mask R-CNN [30] on COCO [38] with  $1 \times$  schedule, i.e., 12 epochs, using the configuration of ViTDet [36]. We take ViT-B/16 [21] as the backbone for all entries in Tab. 6. We regard  $\mathrm{AP}^{\mathrm{b}}$  and  $\mathrm{AP}^{\mathrm{m}}$  as the main metric for object detection and instance segmentation, respectively. For further comparison, we additionally report  $\mathrm{AP}_{50}^{\mathrm{b}}$  and  $\mathrm{AP}_{75}^{\mathrm{b}}$  for object detection, and  $\mathrm{AP}_{50}^{\mathrm{m}}$  and  $\mathrm{AP}_{75}^{\mathrm{m}}$  for instance segmentation. With only 200 epochs of pre-training, DropPos achieves  $42.1\% \mathrm{AP}^{\mathrm{b}}$  and  $37.9\% \mathrm{AP}^{\mathrm{m}}$ , outperforming MAE [28] by  $+2.0\%$  and  $+1.5\%$ , respectively. Note that these improvements appear to be more significant than those on the classification benchmark shown in Tab. 5, indicating that DropPos indeed contributes to better spatial reasoning abilities of ViTs. With 800 epochs of pre-training, DropPos achieves  $47.7\% \mathrm{AP}^{\mathrm{b}}$  and  $42.6\% \mathrm{AP}^{\mathrm{m}}$ , surpassing MAE [28] by  $+0.4\%$  and  $+0.2\%$ , respectively.
ADE20k semantic segmentation. We fine-tune UperNet [63] on ADE20k [71] with 80k iterations. We take ViT-B/16 [21] as the backbone for all entries in Tab. 6 and search for the optimal learning rate for each entry. We regard mIoU as the main metric for semantic segmentation. We additionally report aAcc for further comparison. With only 200 epochs of pre-training, DropPos achieves  $40.7\%$  mIoU, outperforming MAE [28] by  $+0.2\%$ . With 800 epochs of pre-training, DropPos achieves  $47.8\%$  mIoU, surpassing MAE [28] by  $+0.8\%$ .
Qualitative results. We load DropPos with 200 epochs of pre-training and provide qualitative results on this position reconstruction task in Fig. 3. DropPos manages to reconstruct the exact positions of most patches even in extreme circumstances, i.e.,  $\gamma = 0.75$  and  $\gamma_{\mathrm{pos}} = 0.95$ . This suggests that DropPos, as a self-supervised pretext task, indeed makes ViTs location-aware.
To explore whether the improved position sensitivity results in better feature representation and benefits to downstream tasks or not, we propose a metric to evaluate the position sensitivity.
Specifically, we freeze the backbone and train an extra linear position prediction head using the vanilla cross-entropy loss. Top-1 accuracies of position predictions before and after fine-tuning are reported in Tab. 7, and  $75\%$  of position embeddings are randomly masked during training. Higher values mean the model is better at modeling the position relationship. The top-1 accuracy on the ImageNet-1K [48] validation set after fine-tuning is also reported.
As shown in Tab. 7, the backbone performs better in position prediction after fine-tuning, indicating that image classification indeed needs strong abilities in modeling spatial relationships. It means that better position sensitivity corresponds to better performances on downstream tasks. This evidence suggests that our motivation, i.e., enhancing the location awareness of ViTs, is reasonable, and the topic is worth studying. By designing a position prediction pretext task, the backbone pre-trained by DropPos has better position modeling abilities, performing better on a variety of downstream tasks.
In this paper, we present DropPos, a novel, simple, and effective self-supervised pretext task that enhances the location awareness of ViTs. By masking patch tokens first and dropping positional embeddings next, ViTs are requesting to classify the position of each visible patch among all candidates based on partial visual appearance and a few anchor patches. In this way, we manage to avoid (i) discrepancies between pre-training and fine-tuning, (ii) trivial solutions, and (iii) reconstructing precise positions when unnecessary, resulting in improved spatial reasoning and understanding abilities of ViTs. Experiments across various benchmarks demonstrate the efficacy of DropPos, where DropPos consistently achieves competitive results compared with previous methods.
Discussion. Due to limited computational resources, we do not use DropPos to pre-train larger ViTs, e.g., ViT-H/14. Despite these limitations, a growing need to design a pretext task that effectively uncovers the representation learning capabilities of vision models is becoming evident. We hope our DropPos will inspire future work. Additionally, how to enhance location awareness of vision models for spatial reasoning tasks in a supervised manner is also valuable to study.
This work was supported in part by the Major Project for New Generation of AI (No. 2018AAA0100400), the National Natural Science Foundation of China (No. 61836014, No. U21B2042, No. 62072457, No. 62006231), and the InnoHK program.
[1] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. arXiv preprint arXiv:2301.08243, 2023. 3
[2] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning (ICML), pages 1298-1312. PMLR, 2022. 3
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations (ICLR), 2022. 1, 3, 4, 8, 16
[4] Mathilde Caron, Neil Houlsby, and Cordelia Schmid. Location-aware self-supervised transformers. arXiv preprint arXiv:2212.02400, 2022. 3
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS), 33:9912-9924, 2020. 2, 3, 15
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9650-9660, 2021. 3, 8, 15
[7] Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey on vision-language pre-training. Machine Intelligence Research, 20(1):38-56, 2023. 1
[8] Hajian Chen, Wendong Zhang, Yunbo Wang, and Xiaokang Yang. Improving masked autoencoders by learning where to mask. arXiv preprint arXiv:2303.06583, 2023. 2
[69] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Conference on Computer Vision (ECCV), pages 649-666, 2016. 1, 2
[70] Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1058-1067, 2017. 1
[71] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 633-641, 2017. 2, 3, 6, 8, 9, 16
[72] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Representations (ICLR), 2022. 3, 6, 8, 15
In this supplementary material, we first provide mode implementation details for reproducibility in Sec. A. Next, we provide more experiments in Sec. B. Finally, in Sec. C, we evaluate the performance of the position reconstruction task using pre-trained models under different settings, and we provide more evidence to support the proposed three difficulties in Sec. 1.
ViT architecture. We follow the standard vanilla ViT [21] architecture used in MAE [28] as the backbone, which is a stack of Transformer blocks [51]. Following MAE [28], we use the fixed 2D sine-cosine positional embeddings during pre-training. For the downstream classification task, we use features globally averaged from the encoder output for both end-to-end fine-tuning.
Effective training epochs. Following iBOT [72], we take the effective training epochs as the metric of the training schedule, due to extra computation costs brought by the multi-crop [5] augmentation, which is a widely used technique for contrastive methods. Specifically, the effective training epochs are defined as the actual pre-training epochs multiplied with a scaling factor  $r$ . For instance, DINO [6] is trained with 2 global  $224 \times 224$  crops and 10 local  $96 \times 96$  crops, and thus  $r = 2 + (96 / 224)^{2} \times 10 \approx 4$ . More details and examples can be found in [72].
A.1 ImageNet classification
For all experiments in this paper, we take ImageNet-1K [48], which contains 1.3M images for 1K categories, as the pre-trained dataset. By default, we take ViT-B/16 [21] as the backbone and it is pre-trained 200 epochs followed by 100 epochs of end-to-end fine-tuning. Implementation details can be found in the following table. Most of the configurations are borrowed from MAE [28]. The linear learning rate scaling rule is adopted:  $lr = lr_{\mathrm{base}} \times \mathrm{batch\_size} / 256$ . For supervised training from scratch, we simply follow the fine-tuning setting without another tuning. For ViT-B/16, pre-training and fine-tuning are conducted with 64 and 32 2080Ti GPUs, respectively. For ViT-L/16, pre-training and fine-tuning are conducted with 32 and 16 Tesla V100 GPUs, respectively.
config	pre-training	fine-tuning
optimizer	AdamW	AdamW
base learning rate	1.5e-4	1e-3
weight decay	0.05	0.05
momentum	β1, β2= 0.9, 0.95	β1, β2= 0.9, 0.999
layer-wise lr decay	1.0	0.8
batch size	4096	1024
learning rate schedule	cosine decay	cosine decay
warmup epochs	10 (ViT-B/16), 40 (ViT-L/16)	5
training epochs	200	100 (ViT-B/16), 50 (ViT-L/16)
augmentation	RandomResizedCrop	RandAug (9, 0.5) [16]
label smoothing	-	0.1
mixup [68]	-	0.8
cutmix [65]	-	1.0
drop path [31]	-	0.1
A.2 COCO object detection and segmentation
We take Mask R-CNN [30] with FPN [37] as the object detector. Following [28] and [54], to obtain pyramid feature maps for matching the requirements of FPN [37], whose feature maps are all with a stride of 16, we equally divide the backbone into 4 subsets, each consisting of a last global-window block and several local-window blocks otherwise, and then apply convolutions to get the intermediate feature maps at different scales (stride 4, 8, 16, or 32).
We perform end-to-end fine-tuning on COCO [38] for  $1 \times$  schedule with  $1024 \times 1024$  resolution, where 88,750 iterations of training with a batch size of 16 are performed. We simply follow the configuration of ViTDet [36], where the learning rate is 3e-4 and decays at the 78,889-th and 85,463-th iteration by a factor of 10. Experiments are conducted on 8 Tesla V100 GPUs.
Table S1: Ablation study of the initialization of the positional encoding. Fixed sin-cos position embeddings achieve the best performance.
initialization	learnable	ImageNet Top-1 Acc	ADE20k mIoU
sin-cos	×	82.96	40.68
sin-cos	✓	82.81	39.37
random	✓	82.48	38.72
Table S2: The performance when DropPos is equipped with Swin Transformers [40]. 200 epochs of pre-training and 100 epochs of fine-tuning are performed based on the implementation of UM-MAE [35]. We can conclude that DropPos works with Swin [40].
method	backbone	ImageNet-1K
UM-MAE [35]	Swin-Tiny [40]	82.04
DropPos	Swin-Tiny [40]	82.73
A.3 ADE20k semantic segmentation
We take UperNet [63] as the segmentation decoder following the code of [3, 14, 54]. Fine-tuning on ADE20k [71] for 80k iterations is performed. Specifically, each iteration consists of 16 images with  $512 \times 512$  resolution. The AdamW optimizer is adopted with an initial learning rate of 7e-4 and a weight decay of 0.05 with ViT-B. We apply a polynomial learning rate schedule with the first warmup of 1500 iterations following common practice [3, 14, 54]. When fine-tuning using backbones pre-trained with different methods, we search for the optimal learning rate or simply follow their official implementation for a fair comparison. Specifically, the learning rate is 1e-4 for [13, 28, 34], 4e-4 for [19, 56], respectively. All experiments are conducted on 8 Tesla V100 GPUs.
B More Experiments
The initialization of the positional encoding. DropPos uses fixed 2D sin-cos position embeddings by default. We ablate the initialization of position embeddings in Tab. S1 and it demonstrates that fixed sin-cos position embeddings achieve the best performance.
DropPos with Swin Transformers [40]. To verify the scaling property and the generalization of DropPos, we provide experiments when DropPos is equipped with the Swin Transformer. We follow the implementation of UM-MAE [35] and pre-train a Swin-Tiny [40] from scratch using DropPos. All models are pre-trained with 200 epochs and fine-tuned with 100 epochs, following the configuration of UM-MAE [35]. From Tab. S2, we can conclude that DropPos still works on Swin Transformers [40], and thus enhancing the location awareness of vision transformers is still worth studying.
C Performance of position reconstruction