DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions
Authors: Haochen Wang
Junsong Fan
Yuxi Wang
Kaiyou Song
Tong Wang
Zhaoxiang Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Self-Supervised Learning
Vision Transformer
Visual Representation Learning
TL;DR: We present DropPos, a novel self-supervised pretext task that enhances the location awareness of ViTs.

Abstract: As it is empirically observed that Vision Transformers (ViTs) are quite insensitive to the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident. To address this, we present DropPos, a novel pretext task designed to reconstruct Dropped Positions. The formulation of DropPos is simple: we first drop a large random subset of positional embeddings and then the model classifies the actual position for each non-overlapping patch among all possible positions solely based on their visual appearance. To avoid trivial solutions, we increase the difficulty of this task by keeping only a subset of patches visible. Additionally, considering there may be different patches with similar visual appearances, we propose position smoothing and attentive reconstruction strategies to relax this classification problem, since it is not necessary to reconstruct their exact positions in these cases. Empirical evaluations of DropPos show strong capabilities. DropPos outperforms supervised pre-training and achieves competitive results compared with state-of-the-art self-supervised alternatives on a wide range of downstream benchmarks. This suggests that explicitly encouraging spatial reasoning abilities, as DropPos does, indeed contributes to the improved location awareness of ViTs. The code is publicly available at https://github.com/Haochen-Wang409/DropPos.

DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions
Haochen Wang 1. Junsong Fan Yuxi Wang 1,4
Kaiyou Song?
Tong Wang" Zhaoxiang Zhang 1,,44
Center for Research on Intelligent Perception and Computing (CRIPAC),
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS),
Institute of Automation, Chinese Academy of Sciences (CASIA)
Megvii Technology Unniversity of Chinese Academy of Sciences (UCAS)
*Centre for Artificial Intelligence and Robotics, HKISI_CAS
fvanghaochen2022, junsong. fan, zhaoxiang. wha..gzh.nnniai.ccc... yuxivang@8@gmail.ccmmm com {songkaiyou, vangtong)mmegvii.com
Abstract
As it is empirically observed that Vision Transformers (ViTs) are quite insensitive the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident. To address this, we present DropPos, novel pretext task designed to reconstruct Dropped Positions. The formulation of DropPos is simple: we first drop large random subset positional embeddings and then model classifies the actual position for each non-overlapping patch among all possible positions solely based their visual appearance. To avoid trivial solutions, we increase the difficulty of this task by keeping only subser of patches visible. Additionally, considering there may be different patches with similar visual appearances, we propose position smoothing and attentive reconstruction strategies to relax this classification problem, since it is not necessary to reconstruct their exact positions in these cases. Empirical evaluations of DropPos show strong capabilities. DropPos outperforms supervised pre-training and achieves competitive results compared with state-of-the-arr self- supervised alternatives that explicitly encouraging spatial reasoning abilities, as DropPos does, indeed contributes to the improved location awareness of ViTs. The code is publicly available at https: // github..... com/ Maaoohhnn-Wang09////
wide range of downstream benchmarks. This suggests
Introduction
Learning extensible visual representations without any human annotations, known as self-supervised learning (SSL), has become research hotspot in the field of computer vision [7.9,26,27,p) 2,29,47,, 47,59, 19,69 62] since enables efficient transfer to various downstream benchmarks. To achieve this goal, researchers carefully design visual pretext tasks to produce appropriate supervision using only images [3, 18, 46, 62, 69, 70]. Two popular approaches among these are contrastive learning (CL) [62] and masked image modeling (MIM) [3]. both of which have demonstrated promising scaling behavior of vision models, particularly Vision Transformers (ViTs) [21]. Despite the success of these methods, ViTs are found to be relatively insensitive to the order of input tokens [13, 43, 67], leading to the hypothesis that they tend model the relationship between a set of unordered input tokens. Therefore, a natural question arises: beyond the current CL and MIM paradigms, whether a pretext task that explicitly enhances the positional awareness of ViT' can further improve their representation learning abilities?
"Correponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

agreement
image
positions
model
model
model
image
masked image
image
dropped position
(a) Contrastive learning.
(b) Masked image modeling-
(c) Dropped positions reconstructing.
Figure 1: Comparison between different pretext tasks. (a) Contrastive learning aims to maximize
the agreement between different views of one image. (b) Masked image modeling predicts specific contents of masked patches. (c) DropPos reconstructs positions solely based on visual appearance.
To answer this question, we begin by revisiting the forward procedure of ViTs. A sequence of positional embeddings (PEs) [51] is added to patch embeddings to preserve position information. Intuitively, simply discarding these PEs and requesting the model to reconstruct the position for each patch naturally becomes a qualified location-aware pretext task. By forcing the model only to utilize visual appearance for position reconstruction, the model is supposed to learn the shape relationships and layouts of image contents, thereby improving the ability to encode spatial visual content. However, despite the simplicity and intuitiveness of this idea, position-related methods such as [18, 44] have fallen far behind. We identify several difficulties in designing this type of paradigm:
(i) Discarding all PEs brings discrepancies between pre-training and the-tt because the model has never been exposed to any PEs during pre-training. (ii) Strengths of ViTs in modeling long-range dependencies may cause them to solve the task superficially, leading to trivial solutions thatfut to learn highly semantic representations by solving this simple task. (iii) Patches with similar visual appearances may result in confusing reconstruction targets, making it crucial to decide which position to reconstruct precisely.
Driven by this analysis, we present novel, straightforward, and highly effective pretext task for self sulf-suuer visual representation learning: reconstructing Dropped Positions (DropPos). DropPos tackles the above issues systematically: (i) To address discrepancies. simply dropping a large subset of PEs instead of discarding all of them works effectively. (ii) To avoid trivial solutions, we only keep subset of patches visible during pre-training, forcing ViTs to reconstruct the position of each visible patch with only partial inputs. (ii) To prevent ViTs from being overwhelmed by this particular task, we propose to relax this problem using position smoothing and attentive reconstruction strategies.
DropPos can be easily formulated into a simple patch-wise classification problem. Here is how works: First, input patches are randomly masked and a large subset of PEs corresponding to these visible patches is dropped. Then, ViTs are tasked with classifying the positions of visible patches among all possible candidates, relying on partial observations and few anchors (i.e., patches with unmasked PEs). Fig. illustrates two benefits of DropPos: requiring less prior knowledge of data augmentation techniques compared to CL, and (ii) eliminating the need for careful selection of target representation [39] and mask strategy [8, 54, 56] MIM does.
We conducted extensive experiments to evaluate the performance of DropPos in comparison to state-of-the-art alternatives across various downstream tasks. With only 800 epochs of pre-training, DropPos achieves 84.2% top-1 accuracy on ImageNet-I 1K [48] validation set using VIT-B/16 [21], outperforming MAE [28] pre-trained with 1600 epochs by +0.6%. Moreover, on COCO [38] object detection/segmeetationn and ADE20K [71] semantic segmentation benchmarks, which requires more abilities of spatial reasoning, DropPos surpasses MAE [28] consistently.
2 Related work
Self-supervised learning. The key challenge of self-supervised learning is the design of pretext tasks (18,28,38 32, 42, 44,445,50, 50, 54, 69], ie.,, how to produce appropriate supervision signals using only images. Among them, contrastive learning [5,9,26, 29, 58, 60, 62, has been popular recently, which seeks to maximize agreement between different views of the same images. Nevertheless, the

---

performance of contrastive- methods highly depends on carefully-designet data augmentation
techniques [5,6,9, 9, 11, 12, 72]. Another mainstream is masked image modeling [1,3, 28,28,5555 54, 56], which involves predicting specific contents masked patches [2, 3, 19, 20, 28, 54, 64, 72J. Unlike these methods, DropPos offers novel alternative for self-supervised ViTs: reconstructing dropped positions based on only visual clues. In this way, the model is urged to learn the spatial arrangement of local image patches, contributing to better localization abilities, which is important for spatial reasoning recognition tasks, e.g., segmentation [22, 53, 57, 57] and video understanding [17,25],
Position prediction in self-supervised learning. Doersch et al. [18] first split an image into 3x3 grids and train a network to predict the relative position of paired patches from the same image. This problem is formulated as an 8-way classification. Noroozi and Favaro [44] extended this approach to solve "jigsaw puzzles" by urging the model to predict the order of shuffled non- overlapping crops. These approaches were originally developed using ConvNets [33] and thus lacked the ability to learn long-range dependencies, which is crucial for position prediction. Very recently, Zhai et al. [67] revisited this type of pretext task in the scope of Transformers [51] by pre-training ViTs [21] to predict patch positions solely based on their visual appearance, while discarding positional embeddings. However, these pre-trained models have never been exposed to positional embeddings, leading discrepancies between pre-training and fine-tuning, and thus their performances are significantly inferior to sate-of-thhear alternatives [6, 23, 54]. Sameni et al. [49] come up with an auxiliary learning paradigm. Caron et al. [4] extended this idea by predicting the relative location of query (i.e., local crop) to the corresponding reference (i.e., global crop). However, [4] only examined the effectiveness on segmentation benchmarks [15, 24,44,,, 41, 71]. In contrast, DropPos incorporates a novel dropped position reconstruction objective and thus achieves competitive performances downstream tasks, including classfication [48], detection [38], and segmentation [38,71] 71].
resition-related objective and combine it with the popular contrastive
variety of
Positional embeddings in Vision Transformers. In the standard forward procedure of ViTs [21], learnable positional embeddings (PEs) are added with patch embeddings. Improving PEs and introducing more inductive bias into ViTs has become an active research area [23]. This is because some research shows that ViTs are surprisingly robust against permutations of the order of input patches. For instance, Chen et al. [13] demonstrate that the model performs well in classification even with no position embedding. suggesting that ViTs have not fully leveraged the positional information. Additionally, Naseer et al. [43] shows that ViT' are much less susceptible to patch shuffling perturbations than ConvNets. Therefore, we propose DropPos, a novel self-supervised pretext task that explicitly makes ViTs location-aware and promotes the emergence of spatial features.
3 Method
Preliminary: Vision Transformers. Given an image I E RlIxXXCC, it is first reshaped into a sequence of patches Ip E (N where ( (H, W) indicates the spatial resolution, C' is the number of channels, P is the patch size, and = HW/P2 the number of patches. A linear projection is then applied to Ip, mapping it to D dimensions to get patch embeddings E TR: ND.. Also, [CLS] token Xels E RD is used to aggregate the information. Position embeddings p E (NN+1)x are added to the patch embeddings to retain positional information. Then, z = [ x| θ p is the input mechanisms, where
of transformer blocks [51] which consists of stack of multi- head @ denotes element- seement-wisepe plus. In particular, Pu denotes the position embeddings of the [CLS] token, which will never be dropped.
3.1 DropPos
Fig. illustrates our proposed DropPos. We first randomly generate a binary patch mask M {0, 13A,, where 1 means visible and 0 means masked, respectively. Note that the [CLS] token, i.e., Zu, is always visible in our setting, and thus the length of M is N instead of N + 1 for simplicity. Given pre-defined mask ratio y E (0, 1). M supposed to subject to
N-1
Σ
-1-11-700
which means there are NN masked patches in total. It is worth noticing that we use patch masks to increase the difficulty of the pretext task, which is quite different from building pretext task to
3

---

13
18
Projector
Vision Transformer
Position Embedding
Position larget
Flatten
Position Mask Token
Figure 2: Illustration of DropPos. We first mask a large random subset of input images. Then, positional embeddings of visible patches are randomly dropped and [MASK] tokens are introduced. A lightweight projector is adopted to reconstruct those dropped positions. A simple patch-wise classification objective adopted and gray position targets do not contribute to training.
recover input images in MIM [3, 28]. Then, we denote Xvis gather(x, M) E R(I-))N x to be the visible tokens, and position embeddings (PEs) for these unmasked tokens are randomly dropped, where gather(-;) means only those unmasked patches (i.e., M; 1) are reserved. To address possible discrepancies, we aim drop subset of PEs of these visible patches instead of simply
discarding all of them. Thus, randomly generate binary positional mask Mpos E {0, 1] (1 Given a pre-defined position mask ratio 7pos E (0, 1), M is supposed to subject to
(l (1-))~-1
M pns = (1-7/pss)
[1 >)N,
which indicates that the model is supposed to reconstruct "pos 7) N dropped positions based on remaining (1 )1 7).N anchor patches and the [CLS] token. PEs for visible patches Pvis gath M) are gathered first. Then, aim to obtain the final PEs p' RR(17)NxD by replacing those dropped positions with a learnable [MASK] token:
Po-
if 0,
D
Pvis
Mi-1 =
pos
Pmasks
otherwise,
where = 1,.1....111)))) is the patch index, and Pmask E R is the [MASK] token. Finally, the input of the Transformer encoder is z' = [Xels Xvis| + p' +1]==))+1]<D
Implementation. Pre-training ViTs us- ing our DropPos can be implemented efficiently and requires few specialized operations. The process involves three stages prior forwarding features the encoder. First, we randomly mask a subset of embedded patch tokens. Fol- lowing MAE [28], the list of tokens shuffled randomly, and the last portion of the list is removed. Next, positional embeddings of those unmasked patches are randomly dropped using the same masking operation. Then, positional mask tokens are introduced to obtain the final positional embeddings, which are added to the patch tokens finally and encoded by ViT' followed by lightweight decoder. A simple classification objective is applied to those unmasked patches introduced as follows.
Algorithm Pseudo-Code of DropPos.
embedded patch taama/gaans__os: aatch/position mask ratio mask_token: learnable [MASK]
patch masking, [CLS] token is kept visible ida_via, x_via gather ida_via) position dropping
masking(x, ganna)
visible pstches
pos_embed - gpthee(pos__ebedd ids_vis)
ida_keep, ids_res masking(pos..cmeet gamna_pos)
poa_embed gather ids_keep)
concat vith mask tokens
pos_embed ([too___eeeeed naak_token])
reatore
pos_ eabed gather (pos_enbedd ids_res) add to patch tokens for forward pass return x_vis pos_embed

---

3.2 Pre-training DropPos
DropPos is pre-trained by reconstructing dropped positions. Concretely, as illustrated in Fig. 2 and presented in Sec. 3.1, an ViT [21] fe parameterized by A, takes z' as input. A lightweight decoder Po parameterized by v then projects fo(z') to patch-wise position predicitons IR(->))XN, We omit the [CLS] token here for simplicity. For each visible patch E {0, 1, ... (1 7)N- - 1]. softmax(o.) R& represents the probability density function of the predicted positions. Next, we have to produce appropriate ground truths, i.e., the actual position for each visible patch. To this end, we gather positions via y = gather (sther([0,11 I..... . 1]. M) E R()-)),, where vector is naturally the actual position for all patches before masking (w/o the [CLS] token).
(ff((zz))
[0, .,....... . - Overall, the objective is simply the vanilla cross-entropy loss given true positions Yiy-
(1-7)N-13 -
exp(o.)) 1ーー-ル-ー1 exp(oxx)
f=
Σ Σ
(1- (1-M Mpoe) one mo__hotyy.)) log
==
k==
Only patches with dropped positions, Le., orange patches in Fig. 2, contribute to this objective, which is reflected by the term (1 - M Mi in Eq. (4). Next, several techniques are proposed to prevent VITs from being overwhelmed by this particular task, making them pay more attention to model spatial dependencies instead of simply reconstructing positions.
Position smoothing. Considering that different categories (i.e, positions) are not completely independent under this setting, the classification problem in Eq. (4) is relaxed when the model has predictions close to the actual position. Specifically, a weight matrix w E (0, 1) is defined to measure similarities between different patches:
jist(i, σ²
wr.i exp
where dist (i,j) means the Euclidean distance between patch and j, and is hyper-parameter. [E N-1
The matrix then should be normalized by (i.j) w(i,j)/
k= w(i, k)) Here, w E R
indicates the smoothed ground truth when the actual position is Overall, the smoothed objective is:
(1-))N-1N) N-1
exp(o.)) (1-1)N-1
Lamooth
Σ
Σ (1 - (1-Mpos) w (y log
(6)
(xp()
j=
k=0
Furthermore, we relax the problem Eq. (6) by setting large a at the early stage, and we gradually decay the value of a to produce a challenging pretext task. A simple linearly decay is performed:
o₁ ( - σ0) oo.
where i and T' denote the iteration and the number of total steps, respectively. on and oy the initial value and final value of ar, respectively. We set dg = 1 and or = 0 by default.
Attentive reconstruction. Since there may be different patches that share similar visual appearance (e.g., R., two blue patches that represent the sky), it is not necessary to reconstruct their exact positions. Simply swapping these patches still maintains reasonable visual coherence. Therefore, we leverage the feature similarities between the [CLS] token and patch tokens to be an extra weight of Eq. (6):
(1-9)-1N-1
cxp(o.))
smooth
Σ
Σ (1 11- MI-M
Mpos) Ay. w (Viys log
(8)
Σ
Σ
(1-7)8-1 exp(ow)
where A N denotes the similarity matrix. Concretely, A, means the affinity between the [CLS] token and patch token i. Let f.1 e R and f E R ND be the features output by the encoder fo. and thus the affinity A; is computed by
exp(cos(fau) (fals, f.)/T)
A,
(9)
===0 exp(cos(ffas) : fz)/T)
where indicates the temperature parameter and is set 0.1 by default.
5

---

Table 1: Ablation study of patch mask ratio y. Note that "y = 0" means that the whole image is visible. We report the averaged top-1 accuracy of position predictions.
ImageNet-Ik IK
COCO detection
COCO segmentation
ADE20K
fine-tune
position
AP
AP
AP
APm
AP
AP
mloU
aAco
81.94
59.79
34.10
52.80
37.17
31.24
50.27
33.03
31.66
75.86
0.25
82.02
79.62
37.05
56.23
40.18
33.60
53.51
35.84
32.80
76.92
0.50
82.78
87.27
38.45
57.83
41.87
34.88
55.14
37.50
36.33
78.25
0.75
82.96
87.83
42.14
61.99
46.38
37.93
59.23
40.76
40.68
80.14
0.90
82.81
65.12
41.06
60.92
44.57
37.04
58.23
39.62
40.30
80.09
Table 2: Ablation study of position patch mask ratio "pas. Note that "Pos 1" means that we do not provide any reference patch, Lee,, all visible patches are randomly shuffled. We report the averaged top-1 accuracy of position predictions.
ImageNet-IK
COCO detection
COCO segmentation
ADE20K
Tpos
fine-tune
position
AP
AP
AP
APm
AP50
AP
mloU
aAcc
0.25
82.71
65.19
37.98
57.65
41.39
34.61
54.79
36,93
36.15
78.16
0.50
82.86
86.70
39.23
58.77
42.99
35.60
56.23
37.89
37.82
78.60
0.75
82.96
87.83
42.14
61.99
46.38
37.93
59.23
40.76
40.68
80,14
1.00
82.66
19.44
41.48
61.50
45.18
37.41
58.81
39.92
39.98
80.19
Experiments
Pre-training. We perform self-supervised pre-training on the ImageNet- [48] training set with a resolution of 224x224, By default, take ViT-B/16 [72] the backbone and perform 200 epochs of pre-training. The decoder is a stack of Transformer [51] and has depth of and a width of 512. Patch mask ratio y and position mask ratio Ypos are both 0.75 by default. Our implementation is based on HPM [54]. Details can be found in Supplementary Material.
Evaluation. We perform supervised training to evaluate our DropPos with end-to-end fine-tuning on ImageNet-1K [48] for classification. By default, 100 epochs of fine-tuning are performed following common practices [28, 54] for VIT-B/16 [21]. We report top-1 validation accuracy of single 224x224 crop. As for COCO [38], following previous methods [28, 54J, we take Mask R-CNN [30] with FPN [37] the detector. We perform end-to-end fine- fine-tuning
COCO [38] for 1 x schedule with 1024 x 1024 resolution. We report APb for object detection and APm for instance segmentation, respectively. Our implementation is based on detectron2 [61] and ViTDet [36]. For ADE20k [71], following previous methods [28, 54], we take UperNet [63] as the decoder and perform end-to-end fine-tuning on ADE20k [71] with 512x512 resolution for 80k iterations. We take mloU [24] the main evaluation metric. Our implementation is based on mmsegmentation [14].
4.1 Ablation studies
In Tabs. I to 4, we take the ViT-B/16 [21] pre-trained with 200 epochs on ImageNet-IK [48] as the backbone. We highlight the default settings of our DropPos. Concretely, if not specified, the default setting is ? 0.75, Tpos 0.75, ap 1, oT 0. and 0.1. By default, 100 epochs of fine-tuning on ImageNet-1K [48], xx schedule fine-tuning on COCO [38], and 80k iterations fine-tuning on ADE20k [71] are performed.
To evaluate the performance on the pretext task, i.e., reconstructing dropped positions, we also report the averaged top-1 accuracy of position predictions the ImageNet- IK validation set in Tabs. I to 4. We vary 7 E {0, 0.25, 0.5, 0.75] and "pos E {0.25, 0.5, 0.75, 0.95} when measuring the position prediction accuracy, and average the position accuracy among 16 different cases.
Main properties. Illustrated by Tabs. to 4, we find sufficient evidence to support the three difficulties claimed in Sec. 1. (i) In Tab. 1, VITs fail to learn highly semuntic representations by simply solving the position reconstruction task, because the performances of y = 0 significantly lag behind. (ii) Discrepancies between pre- pre-training and fine- fine-tuning are revealed in Tab. by setting

---

Table 3: Ablation study of a. "o 0" means no position smoothing. "4" denotes a linear decay schedule is performed on σ. We report the averaged top-1 accuracy of position predictions.
ImageNet-IK
COCO detection
COCO segmentation
ADE20K
fine-tune
position
AP
AP
AP,s
APm
AP50
AP
mloU
aAce
82.85
88.81
40.32
59.89
43.90
36.45
57.09
38.88
38.32
79.22
82.91
87.13
40.19
59.76
43.81
36.30
57.06
38.87
38.60
79.17
2
82.79
69.48
40.16
59.94
43.92
36.32
57.32
38.88
38.69
79.30
-> 0
82.96
87.83
42.14
61.99
46.38
37.93
59.23
40.76
40.68
80.14
82.88
87.65
40.46
60.03
44.21
36.53
57.20
39.30
38.79
79.43
Table 4: Ablation study of attentive reconstruction. "T= oo" indicates no attentive reconstruction. We report the averaged top-1 accuracy of position predictions.
7 ImageNet-IK
COCO detection
COCO segmentation
ADE20k
fine-tune
position
APM
AP
AP/s
AP"
AP5)
AP
mloU
aAce
80
82.84
88.66
40.56
60.48
44.15
36.78
57.64
39.46
38.38
79.36
0.1
82.96
87.83
42.14
61.99
46.38
37,93
59.23
40.76
40.68
80.14
0.5
82.86
87,78
40.73
60.33
44.77
36.71
57.72
39.38
38.80
79.62
Ypos = 1. where we observe performance deterioration. (iii) Deciding which position to reconstruct precisely is crucial because by setting = 0 in Tab. and r oo in Tab. 4, VITs have better position predictions task but perform worse on downstream tasks.
Performance on the position reconstruction task. In general, positive correlation is observed between the accuracy of the downstream task and the averaged accuracy of predicted positions, indicating that the averaged accuracy of position predictions is reliable indicator of how well the model trained. However, there are some exceptions, i.e., a = in Tab. and T = oo in Tab. 4, where the model reconstructs more precise positions but performs worse on downstream benchmarks, indicating that deciding which position to reconstruct precisely is crucial.
Please refer to Supplementary Material for the detailed performance of the position reconstruction task, where we provide more evidence to support the three difficulties proposed in Sec.
Patch mask ratio T, We vary the patch mask ratio y from 0 to 0.75 in Tab. 1. When y = 0, the entire image is visible during pre mre-trainin, resulting in nearly perfect position predictions and making the task too simple for self-supervised learning (see Supplementary Material for details). However, the averuged accuracy of position predictions is quite low since the accuracy deteriorates quickly as we enlarge during evaluation. Adopting a larger 7 for pre-training leads to better performance, especially detection and segmentation benchmarks. Therefore, we can conclude that patch masking is essential in DropPos to prevent trivial solutions. However, deterioration is observed when 7 = 0.9. This is because the model may be under-fitted using this extremely difficult pretext task.
Position mask ratio "ppas We study the effectiveness of the position mask ratio Ypos in Tab. 2. DropPos appears to be more robust against different "/pos than 7. However, appropriate Ypus is still necessary. small "ipos leads to trivial solutions even we have large y, while an extremely large / e.g., Ypos 1, results in discrepencies between pre-training and fine-tuning.
Position smoothing. By setting different values of ag and oT, we study the effectiveness of position smoothing. By setting O, we remove the position smoothing strategy, and the model tends to be overwhelmed by this position reconstruction task, lcading to high accuracy in position prediction but poor performance downstream tasks. However, when too large, the smoothed positions become noisy, contributing to poor performance on both downstream tasks and position reconstruction. Additionally, equipped with a linear decay schedule of o, DropPos is gradually guided, bringing improvements on both downstream tasks and position reconstruction.
Attentive reconstruction. Attentive reconstruction is studied in Tab. 4 by setting different values of temperature т, where 7 do means no attentive reconstruction. Without attentive reconstruction, DropPos is able to obtain better position predictions but performs worse downstream tasks. This because we do not have to reconstruct the exact positions when different patches share similar visual appearances. This phenomenon can be mitigated by setting appropriate A large 7 leads
7

---

Table 5: Comparison with previous methods on ImageNet-1K classification. All methods are evaluated by fine-tuning The resolution of images is fixed to 224x224. f means our implementation. j means the result borrowed from [28].
method
venue
eff. ep.
ViT-B
VIT-I
supervised
80.91
82.6
MAE [28]
[CVPR'22]
200
82.2
83.3*
DropPos
[Ours]
200
83.0
83.7
Contrastive Learning
DINO [6]
[ICCV'21]
1600
82.8
MoCo v3f [13]
[CCV'21]
600
83.2
84.1
Masked Image Modeling
BEiT [3]
[ICLR/2]]
800
83.2
85.2
MAE [28]
[CVPR/22]
1600
83.6
85,9
SimMIM [64]
[CVPR/22]
800
83.8
SemMAE [34]
[NcurPPS222]
800
83.4
LocalMIM [56]
[CVPR 23]
1600
84.0
HPM [54]
[CVPR'23
800
84.2
85.8
Masked Image Modeling + Contrastive Learning
iBOT [72]
[|CLR22]]
1600
84.0
BootMAE [19]
[BCCV'22]
800
84.2
85.9
Position Reconstructing
DropPos
[Ours]
800
84.2
85.8
Table 6: Comparison with previous methods downstream tasks. All methods take the ViT- B/16 [21] as the backbone and utilize Mask R-CNN [30] on COCO [38] object detection and instance segmentation, and UperNet [63] ADE20k [71] semantic segmentation, respectively. $ means the result is borrowed from [28]. indicates our implementation, including pre-training and supervised fine-tuning, while represents we reproduce fine-tuning using the official pre-trained backbone. We perform xx schedule of fine-tuning on COCO using ViTDet [36], and 80k iterations of fine-tuning
on ADE20k using mmsegmentation [14].
COCO detection
COCO segmentation
ADE20k
method
venue
eff. ep.
AP
AP
AP
APM
AP
AP
mloU
aAcc
MAE [28]
[CVPR'22]
200
40.1
60.5
44.1
36.4
57.8
39.3
40.5
80.1
DropPos
[Ours]
42.1
62.0
46.4
37.9
59.2
40.8
40.7
80.1
MoCo v [13]
[ICCV/21]
600
43.7
65.7
47.7
39.1
62.0
41.8
44.7
81.5
MAE [28]
[CVPR'22]
1600
47.3
68.2
52.5
42.4
65.3
45.6
47.0
82.7
BootMAE [19]
[ECCVV22]
800
47.3
67.9
52.1
42.3
65.0
45.8
47.3
83.0
SemMAEt [34]
[NeurIPS/22)
800
45.6
66.2
55.2
40.9
63.3
44.4
44.9
82.0
LocalMIM [56]
[CVPR'23]
1600
47.4
67.7
52.2
42.2
64.8
45.5
47.1
83.1
DropPos
[Ours]
800
47.7
68.3
52.8
42.6
65.3
46.2
47.8
82.8
to noisy position ground truths, leading to poor performance on both downstream tasks and position reconstruction, which is the same large a.
4.2 Comparisons with previous results
We compare our proposed DropPos with the supervised pre-training baseline and a wide range of self- supervised methods, including (i) contrastive learning methods [6. 13], (ii) masked image modeling methods [3,28, 34, 54, 56, 64], and (iii) their combinations [19, 72]. Effective pre-training epoch is used for fair comparison following [54, 72] since it accounts for the actual trained images/views. The detailed definition can be found in Supplementary Material. All methods are pre- are-trained with the same resolution, i.e., 224 ImageNet-IK [48].
ImageNet-1K classification. We compare our DropPos with state-of-the-art alternatives on the ImageNet-IK IK [48] classification benchmark in Tab. 5. Notably, with only 200 epochs pre-training,

---

Figure 3: Qualitative results of position reconstruction. We evaluate position predictions with different y but fix 7pos = 0.95. Black patches are masked during inference. The positions of those white patches are wrongly predicted, while the remaining patches are predicted correctly. For each tuple, we show results under (a) 0, (b) y 0.25, (c) y 0.5, (d) y 0.75, and (e) the original image. DropPos manages to reconstruct precise positions.
Table 7:
An in-depth analysis about whether the improved position sensitivity benefits downstream tasks or not. We freeze the backbone and train an extra linear patch classification head. 75% of position embeddings are randomly masked during linear probing. We can conclude that image classification indeed needs better location awareness and thus designing position prediction pretext task crucial and worth studying.
Position prediction Pre-trained
ImageNet-IK Top-1
method
venue
Fine-tuned
MoCo v3 [13]
[ICCV221]
43.2
78.0 1 34.8
83.2
MAE [28] [CVPR/22]
63.1
82.5
19.4
83.6
DropPos
[ours]
77.3
88.0 1 10.77
84.2
DropPos achieves 83.0% and 83.7% using ViT-B/16 and ViT-L/16 as the backbone, surpassing the supervised baseline by +1.1%, and MAE [28] by +0.8% and +0.4% respectively. This empirical evidence demonstrates that enhancing the location awareness of ViT's indeed brings better visual representations. Furthermore, with 800 epochs of pre-training, DropPos manages to achieve competitive results compared with the state-of-the-art. Specifically, it achieves 84.2% and 85.8% using ViT-B/16 and ViT-L/16, respectively. Strikingly, DropPos reaches competitive results with BootMAE [19], which combines contrastive learning and masked image modeling.
COCO object detection and segmentation. We fine-tune Mask R-CNN [30] on COCO [38] with x schedule, i.e., 12 epochs, using the configuration of ViTDet [36]. We take ViT-B/16 [21] as the backbone for all entries in Tab. 6. We regard APh and APm as the main metric for object detection and instance segmentation, respectively. For further comparison, we additionally report AP and AP&₂ for object detection, and AP and AP% for instance segmentation. With only 200 epochs of pre-training, DropPos achieves 42.1 1% AP and 37 97.9% APm, outperforming MAE [28] by +2.0% and +1.5%, respectively. Note that these improvements appear be more significant than those on the classification benchmark shown in Tab. 5, indicating that DropPos indeed contributes to better spatial reasoning abilities of ViTs. With 800 epochs of pre-training, DropPos achieves 47.7% APh and 42.6% APm, surpassing MAF [28] by +0.4% and +0.2%, respectively.
ADE20k semantic segmentation. We fine-tune UperNet [63] on ADE20k [71] with 80k iterations. We take ViT-B/16 [21] as the backbone for all entries in Tab. 6 and search for the optimal learning rate for each entry. We regard mloU as the main metric for semantic segmentation. We additionally report aAce for further comparison. With only 200 epochs of pre-training, DropPos achieves 40.79 7% mloU, outperforming MAE [28] by +0.2%. With 800 epochs of pre-training, DropPos achieves 47.8% mloU, surpassing MAE [28] by +0.8%.
Qualitative results. We load DropPos with 200 epochs of pre- pre-training and provide qualitative results on this position reconstruction task in Fig. 3. DropPos manages to reconstruct the exact positions of most patches even in extreme circumstances, i.e., 7 0.75 and Tpos = 0.95. This suggests that DropPos, as a self-supervised pretext task, indeed makes VIT's location-aware.
4.3 Analysis
To explore whether the improved position sensitivity results in better feature representation and benefits to downstream tasks not, we propose metric to evaluate the position sensitivity.

---

Specifically, we freeze the backbone and train an extra linear position prediction head using the vanilla cross-entropy loss. Top-1 accuracies of position predictions before and after fine-tuning are reported in Tab. 7, and 75% of position embeddings are randomly masked during training. Higher values mean the model is better at modeling the position relationship. The top-1 accuracy on the ImageNet- K [48] validation set after fine-tuning is also reported.
As shown in Tab. 7, the backbone performs better in position prediction after fine-tuning, indicating that image classification indeed needs strong abilities in modeling spatial relationships. It means that better position sensitivity corresponds to better performances on downstream tasks. This evidence suggests that our motivation, i.e., enhancing the location awareness of ViTs, is reasonable, and the topic is worth studying. By designing a position prediction pretext task, the backbone pre-traines trained by DropPos has better position modeling abilities, performing better on a variety of downstream tasks.
5 Conclusion
In this paper, we present DropPos, a novel, simple, and effective self-supervised pretext task that enhances the location awareness of ViTs. By masking patch tokens first and dropping positional embeddings next, ViTs are requesting to classify the position of each visible patch among all candidates based partial visual appearance and a few anchor patches. In this way, manage to avoid (i) discrepancies between pre-training and fine-tuning, (ii) trivial solutions, and (iii) reconstructing precise positions when unnecessary, resulting in improved spatial reasoning and understanding abilities of ViTs. Experiments across various benchmarks demonstrate the efficacy of DropPos, where DropPos consistently achieves competitive results compared with previous methods.
Discussion. Due to limited computational resources, we not use DropPos to pre-train larger ViTs, egg. ViT-H/14. Despite these limitations, a growing need to design a pretext task that effectively uncovers the representation learning capabilities of vision models is becoming evident. We hope our DropPos will inspire future work. Additionally, how to enhance location awareness of vision models for spatial reasoning tasks in a supervised manner is also valuable to study.
Acknowledgements
This work was supported in part by the Major Project for New Generation of AI (No. 2018AAA0100400), the National Natural Science Foundation of China (No. 61836014,
No. U21B2042, No. 62072457, No. 62006231), and the InnoHK program.
References
[1] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-emhedding predictive architecture. arXiv preprint 1rXiv:2301.08245 2023.
[2] Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning (ICML), pages
PMLR, 2022.
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations (ICLR), 2022. 1, 4, 8, 16
[4] Mathilde Caron, Neil Houlsby, and Cordelia Schmid, Location-aware self-supervised transformers. arXiv preprint Xiv: 22 12.02400 2022. 3
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NewrIPS), 33:9912-99924, 2020. 2, 3, 15
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pages 9650-9660, 2021. 3, 8, 15
[7] Fei-Long Chen, Du-Zhen Zhang. Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey on vision- haiiuae pre- pre-training. Machine Intelligence Research, 20(1):38 56, 2023. 1
[8] Haijian Chen, Wendong Zhang, Yunbo Wang. and Xiaokang Yang. Improving masked autoencoders by learning where to mask. arXiv preprint 2rXiv:3303,06383 2023. 2
10

---

[9] Ting Chen, Simon Korblith. Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML).
pages 1597-1607, 2020. 2,
[10] Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.0300
2022. 3
[11] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 2xXiv:000300429, 2020. 3
[12] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 15750-157558, 2021. 3
[13] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training sull--upe vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9640-9649, 2021. 1, 3, 8, 9, 16
[14] MMSegmentation Contributors. MMScgmentation: Openmmlab semantic segmentation toolbox and benchmark. https:// //ggthub..ttt///// cww/open mml mbm.a//mmmmggmtttaaaaa. 2020. 6, 8, 16
[15] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bert Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3213-333 2213-3223,22 2016. 3
[16] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), pages 702-703, 2020. 15
[17] Jiaxin Deng, Dong Shen, Haojie Pan, Xiangyu Wu, Ximan Liu, Gaofeng Meng. Fan Yang, Size Li, Ruiji Fu, and Zhongyuan Wang. A unified model for video understanding and knowledge embedding with heterogeneous knowledge graph dataset. In Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR), 2023. 3
[18] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1422-1430, 2015. 1, 2,
[19] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining. In European Conference on Computer Vision (ECCV), pages 247-264. Springer, 2022. 3, 8, 9, 16
[20] Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. In Proceedings of the AAAI Conference Artificial Intelligence (AAA)).2023.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. 1, 2, 3, S, 6, 8, 9, 15, 17, 18
[22] Du, Yujun Shen, Haochen Wang. Jingjing Fei, Wei Li, Liwei Wu, Rui Zhao, Zehua Fu, and Qingjie Liu. Learning from future: A novel self-training framework for semantic segmentation. Advances in Neural Information Processing Systems (NeurIPS), 2022. 3
[23] Philipp Dufter, Martin Schmitt, and Hinrich Schütze. Position information in transformers: An overview. Computational Linguistics, 48(3):733-73,, 2022. 3
[24] Mark Everingham, Luc Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision (IJC)), 88:303-308, 2009. 3, 6
[25] Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders spatiotemporal learners. Advances in Neural Information Processing Systems (NeurIPS), 15:35946-35955 2022. 3
[26] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self- supervieed learning. Advances in Neural Information Processing Systems (NeurIPS), 33:21271-2212 21284, 2020. 1,2
[27] Yuxian Gu, Jiaxin Wen, Hao Sun, Yi Song, Pei Ke, Chujie Zheng, Zheng Zhang, Jianzhu Yao, Lei Liu, Xiaoyan Zhu, et al. Eva2. 0: Investigating open domain chinese dialogue systems with large-scale pre-training. Machine Intelligence Research, 20/2):207 219, 2023.

---

[28] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li. Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 16000-16009, 2022. 2, 3, 4, 6, 8, 9, 15, 16
[29] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), pages 9729-9738, 2020. 1,2
[30] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2961-2969, 2017. 6, 8, 9,
[31] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV), 2016. 15
[32] Dahun Kim, Donghyeon Cho, Donggeun Yoo, and In So Kweon. Learning image representations by completing damaged jigsaw puzzles. In IEEE Winter Conference on Applicatioon of Computer Vision (WACV), pages 793 802, 2018. 2
[33] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 1 1):278--2324, 1998. 3
[34] Gang Li, Heliang Zheng. Daqing Liu, Chaoyue Wang. Bing Su, and Changwen Zheng. Semmae: Semantic- guided masking for learning masked autoencoders. In Advances in Neural Information Processing Systems (NeurlPS), 2022. 8. 16
[35] Xiang Li, Wenhai Wang, Lingfeng Yang, and Jian Yang. Uniform masking: Enabling mae pre- pre-training for pyramid- based vision transformers with locality. arXiv preprint arXiv:2205. 10063, 2022. 16
[36] Yanghao Li, Hanzi Mao. Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference Computer Vision (ECCV), pages 280-296 Springer, 2022.
6, 8, 9, 15
[37] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongic. Feature pyramid networks for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2112-21 2017. 6, 15
[38] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In European Conference on Computer Vision (ECCV), pages 740 -5500-755000 Springer, 2014, 2, 3, 6, 8, 9, 15
[39] Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target representations for masked autoencoders. arXiv preprint arXiv:2209.0:24 1, 2022. 2
[40] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transfomme: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF
International Conference on Computer Vision (ICCV), 2021. 16
[41] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam- Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 891-898, 2014.
[42] T Nathan Mundhenk, Daniel Ho, and Barry Y Chen. Improvements to context based self- sulf--upe learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9339-9333800 9348, 2018. 2
[43] Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-HHuur Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems (NeurIPS), 34:23296-23308, 2021. 1,3
[44] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision (ECCV), pages 69 84, 2016. 2,
[45] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1807.037473 2018. 2
[46] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2536-2544, 2016.
[47] Yixuan Qiu, Feng Lin, Weitong Chen, and Miao Xu. Pre-training in medical data: A survey. Machine Intelligence Research, pages 1-33, 2023. 1
[48] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 11:211-252 2015. 2. 3, 6, 8, 10, 15
12

---

[49] Sepehr Sameni, Simon Jenni, and Paolo Favaro. Representation learning by detecting incorrect location embeddings. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2023. 3
[50] Rodrigo Santa Cruz, Basura Fernando, Anoop Cherian, and Stephen Gould. Deeppermnet: Visual permutation learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1949-3957, 2017. 2
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurlPS), 30, 2017. 2, 3, 6, 15
[52] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning (ICML), pages 1096-1103, 2008.
[53] Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li. Liwei Wu, Yuxi Wang, and Zhaoxiang Zhang. Pulling targel to source: A new perspective on domain adaptive semantic segmentation. arXiv preprint
arXiv:2305. 13755, 2023. 3
[54] Haochen Wang. Kaiyou Song. Junsong Fan, Yuxi Wang. Jin Xie, and Zhaoxiang Zhang. Hard patches mining for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 10375- 1775-10385,,10 2023, 2, 3, 6, 8, 15, 16
[55] Haochen Wang, Yuchao Wang, Yujun Shen, Junsong Fan, Yuxi Wang, and Zhaoxiang Zhang. Using unreliable pseudo-lahels for label-effiiient semantic segmentation. arXiv preprint 2306:4330 2023.
[56] Haoqing Wang, Yehui Tang, Yunhe Wang. Jianyuan Guo, Zhi-Hong Deng, and Kai Han. Masked image modeling with local multi-scalc reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 2, 3, 8, 16
[57] Yuchao Wang. Jingjing Fei, Haochen Wang. Wei Li, Liwei Wu, Rui Zhao, and Yujun Shen. Balancing logit variation for long-tail semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19561-19577 2023. 3
[58] Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 4248-4254 2022. 2.3
[59] Ji-Rong Wen, Zi Huang, and Hanwang Zhang. Editorial for special issuc on large-scale pre-training: Data, models, and fine-tuning. Machine Intelligence Research, 20(2): 2023.
[60] Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Sepf-supervised visual representation learning with semantic grouping. Advances in Neural Information Processing Systems (NeurIPS), 2022. 2
[61] Yuxin Wa, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https:
// //github.cc
deeterrronn,eeeeee 2019. 6
[62] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3733-3742, 2018 1.2
[63] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In European Conference on Computer Vision (ECCV), pages 418-434, 2018. 8, 9, 16
[64] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9653-9663, 2022. 3, 8
[65] Sangdoo Yun, Dongyoon Han, Seong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pages 6023-6032, 2019. 15
[66] Sukmin Yun, Hankook Lee, Jachyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8354-8363, 2022. 2
[67] Shuangfei Zhai, Navdeep Jaitly, Jason Ramapuram, Dan Busbridge, Tatiana Likhomanenko, Joseph Cheng, Walter Talbott, Chen Huang, Hanlin Goh, and Joshua M Susskind. Position prediction as an effective pretraining strategy. In International Conference on Machine Learning (ICML), pages 26010- 26027, 2022. 3.3
[68] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Pau Paz. mixup: Beyond empirical risk minimization. 2018. 15
13

---

[69] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Conference
Computer Vision (ECCV). pages 649-666, 2016. 1,2
[70] Richard Zhang. Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross chasnel prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1058 1067. 2017.
[71] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through adc20k datasct. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition (CVPR), pages 633-644 641, 2017, 2, 3, 6, 8, 9, 16
[72] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xic, Alan Yuille, and Tao Kong, Image bert pre-training with online tokenizer. In International Conference on Learning Representations (ICLR), 2022.
3, 6, 8, 15
14

---

Supplementary Material
In this supplementary material, we first provide mode implementation details for reproducibility in Sec. A. Next, we provide more experiments Sec. B. Finally, Sec. C, we evaluate the performance of the position reconstruction task using pre-trained models under different settings, and we provide more evidence to support the proposed three difficulties in Sec. 1.
A Implementation details
VTT architecture. We follow the standard vanilla VIT [21] architecture used in MAE [28] as the backbone, which a stack of Transformer blocks [51]. Following MAE [28], we use the fixed 2D sine-cosine positional embeddings during pre-traine pre-training. For the downstream classification task, use features globally averaged from the encoder output for both end-to-end fine-tuning.
Effective training epochs. Following iBOT [72], we take the effective training epochs as the metric of the training schedule, due to extra computation costs brought by the multi-crop [5] augmentation, which is widely used technique for contrastive methods. Specifically, the effective training epochs defined as the actual pre-training epochs multiplied with a scaling factor r. For instance, DINO [6] trained with 2 global 224x crops and 10 local 96x96 crops, and thus 2+ (296/224)220 x 10 R More details and examples can be found in [72].
A.1 ImageNet elassification
For all experiments in this paper, we take ImageNet- 1K [48], which contains 1.3M images for 1K categories, as the pre-trained dataset. By default, we take ViT-B/16 [21] the backbone and it is pre-trained 200 epochs followed by 100 epochs of end-to-end fine-tuning, Implementation details can be found in the following table. Most of the configurations are borrowed from MAE [28]. The linear learning rate scaling rule adopted: Ir Iraase X batch_size / 256. For supervised training from scratch, we simply follow the fine-tuning setting without another tuning. For ViT-B/16, pre-training and fine-tuning are conducted with 64 and 32 2080Ti GPUs, respectively. For ViT-L/16, pre-training and fine-tuning are conducted with 32 and 16 Tesla V100 GPUs, respectively.
config
pre-training
fine-tuning
optimizer
AdamW
AdamW
base learning rate
1.5e-4
le-3
weight decay
0.05
0.05
momentum
Sr. _8 0.9, 0.95
B1, 8, 0.9, 0.999
layer-wise decay batch size Icarning rate schedule
1.0
4096
1024
cosine decay
cosine decay
warmup epochs
10 (ViT-B/16), 40 (ViT-L/16)
training epochs
200
100 (ViT-B/16) 50 (ViT-L/16)
augmentation
RandomResizedCrop
RandAug (9. 0.5) [16]
label smoothing
0.1
mixup [68]
0.8
cutmix [65]
1.0
drop path [31]
A.2 COCO object detection and segmentation
We take Mask R-CNN [30] with FPN [37] as the object detector. Following [28] and [54], to obtain pyramid feature maps for matching the requirements of FPN [37]. whose feature maps are all with a stride of 16, equally divide the backbone into 4 subsets, each consisting of last global-window block and several window blocks otherwise, and then apply convolutions to get the intermediate feature maps at different scales (stride 4, 8, 16, or 32).
perform end-to-end fine-tuning on COCO [38] for 1x schedule with 1024 x 1024 resolution, where 88,75 78770 iterations of training with a batch size of 16 are performed. We simply follow the configuration ViTDet [36], where the learning rate 3e-4 and decays at the 78,889-th and 85,463-th iteration by factor of 10. Experiments are conducted 8 Tesla V100 GPUs.
15

---

Table S1: Ablation study of the initialization of the positional encoding, Fixed sin-cos position
embeddings achieve the best performance.
ImageNet Top-1 Acc
ADE20K mloU
initialization
learnable
sin-cos
X
82.96
40.68
sin-cos
82.81
39.37
random
82.48
38.72
Table S2: The performance when DropPos is equipped with Swin Transformers [40]. 200 epochs of pre-training and 100 epochs of fine-tuning are performed based on the implementation of UM-MAE [35]. We can conclude that DropPos works with Swin [40].
method
backbone
ImageNet-IK
UM-MAE [35]
Swin-Tiny [40]
82.04
DropPos
Swin-Tiny [40]
82.73
A.3 ADE20k semantic segmentation
We take UperNet [63] as the segmentation decoder following the code of [3, 14, Fine-tuning on ADE20k [71] for 80k iterations is performed. Specifically, each iteration consists of 16 images with 512 resolution. The Adam optimizer is adopted with initial learning rate of 7e-4 and a weight decay 0.05 with ViT-B. We apply a polynomial learning rate schedule with the first warmup of 1500 iterations following common practice [3, 14. When fine-tuning using backbones pre-trained with different methods, we search for the optimal learning rate or simply follow their official implementation for fair comparison. Specifically, the learning rate is 1e-4 for [13, 28,3), 4e-4 for [19, (6), respectively. All experiments are conducted on 8 Tesla V100 GPUs.
B More Experiments
The initialization of the positional encoding, DropPos uses fixed 2D sin-cos position embeddings by default. We ablate the initialization of position embeddings in Tab. S1 and it demonstrates that fixed sin-cos position embeddings achieve the best performance.
DropPos with Swin Transformers [40]. To verify the scaling property and the generalization of DropPos, we provide experiments when DropPos is equipped with the Swin Transformer. We follow the implementation of UM-MAE [35] and pre-train Swin-Tiny [40] from scratch using DropPos. All models are pre-trained with 200 epochs and fine-tuned with 100 epochs, following the configuration of UM-MAE [35]. From Tab. S2, we can conclude that DropPos still works on Swin Transformers [40], and thus enhancing the location awareness of vision transformers is still worth studying.
C Performance of position reconstruction
In this section, we evaluate the performance of the position reconstruction task using pre-trained models under different settings. Specifically, we vary E {0, 0.25, 0.5, 0.75) and 7pos (0.25, 0.5, 0.75, 0.95] when measuring the position prediction accuracy. We report performance under different evaluation settings as well as the averaged accuracy among 16 different cases. From Tabs. S3 to S5, we find evidence to support the three difficulties for designing an appropriate position- related pretext task introduced in Sec. 1: (i) discrepancies between pre-training and fine-tuning, (ii) failing to learn highly semantic representations by solving this simple position reconstruction task, and (iii) difficult to decide which patch positions to reconstruct precisely.
study the effectiveness of different values of y during pre-training in Tab. S3. Interestingly, we find evidence for failing to learn highly semantic representations by solving this simple position reconstruction task. As illustrated by Tab. S3a, pre-trained model performs extremely well when set y 0 for evaluation but fails to keep this trend when enlarge This indicates that given
16

---

Table S3: Top-1 aceuracy of position reconstruction using ViT-B/16 [21] pre-trained with different mask ratio y. We underline the special parameter different from our default settings. Default settings highlighted in color. "Avg. acc" is the averaged top-1 accuracy over 16 different cases. We evaluate the performance using the same pre-trained model under different and Tpos- Larger 7 and Tpos indicates a more challenging task.
Praiiraining with 7 (Avg, ace: 59.79).
(b) Pre-training with 7 0.25 (Avg. acc: 79.62).
Ypos
0.25
0.50
0.75
0.95
avg.
Ypoe
0.25
0.50
0.75
0.95
avg.
0.00
99.37
99.26
99.15
98.34
99.03
0.00
99.24
99.26
99.18
98.92
99.15
0.25
87.97
87.45
86.20
70.79
83.10
0.25
98.67
98.81
98.63
97.21
98.33
0.50
55.37
55.11
49.84
22.96
45.82
0.50
93.96
93.62
90.01
62.11
84.93
0.75
12.99
14.85
12.85
4.21
11.23
0.75
50.02
47.79
35.03
11.46
36.08
Pre-training with 7 = (Avg. acc: 87.27).
(d) Pre-training with
Y 0.75 (Avg. 87.83).
Ypoa
0.25
0.50
0.75
0.95
avg.
Yppa
0.25
0.50
0.75
0.95
avg.
0.00
99.33
99.23
99.13
98.85
99.14
0.00
97.19
98.69
98.20
92.30
96.60
0.25
99.05
98.95
98.78
98.20
98.75
0.25
96.78
98.26
97.66
91.05
95.93
0.50
94.60
95.94
94.28
83.31
92.03
0.50
97.26
96.82
95.66
89.12
94.72
0.75
78.77
72.75
59.54
25.59
59.16
0.75
79.94
78.10
68.73
40.24
66.75
Table S4: Top-1 accuracy of position reconstruction using VIT-B/16 [21) pre-trained with different positional mask ratio Ypos- We underline the special parameter different from our default settings.
(a) Pre-training with Ypos 0.25 (Avg. acc: 65.19).
(b) Pre-training with Yoos 0.5 (Avg. acc: 86. 70).
ppos
0.25
0.50
0.75
0.95
Ypos
0.25
0.50
0.75
0.95
7
0.00
96.52
89.38
59.29
14.58
0.00
98.98
98.81
98.24
92.80
0.25
96.62
91.27
66.00
19.17
0.25
98.59
98.34
97.51
89.73
0.50
95.58
91.10
72.07
21.69
0.50
96.63
95.85
93.40
74.38
0.75
80.56
73.72
57.89
17.66
0.75
81.94
76.70
64.87
30.44
avg.
92.32
86.37
63.81
18.28
avg.
94.04
92.43
88.51
71.84
(c)
Pre-training with Tpos 0.75
(Avg. ace: 87.83).
(d) Pre-training with "pos = 1 (Avg. acc: 19.44).
Ypos
0.25
0.50
0.75
0.95
Ypos
0.25
0.50
0.75
0.95
0.00
97.19
98.69
98.20
92.30
0.00
15.57
23.24
20.41
23.59
0.25
96.78
98.26
97.66
91.05
0.25
12.51
21.10
24.30
29.41
0.50
97.26
96.82
95.66
89.12
0.50
7.76
14.18
25.56
45.01
0.75
79.94
78.10
68.73
40,24
0.75
3.34
6.00
12.53
26.45
avg.
92.79
92.97
90.06
78.18
avg.
9.80
16.13
20.70
31.12
the strength of VITs in modeling long-range dependencies, they have easily solved this task in superficial way, and thus pre-training with y = 0 becomes trivial for ViTs. this end, an appropriate y is necessary to increase the difficulty of the pretext task and avoid trivial solutions.
We study the effectiveness of different values of "/pos during pre-training in Tab. S4, and find evidence for discrepancies between pre-training and fine-tuning. As shown by Tab. S4d, the model fails to reconstruct accurate positions given some visible anchors. This because the model has never been exposed to any positional embeddings (PEs) during pre-training. Therefore, providing some anchors is necessary to address discrepancies. Also, may help the model focus on modeling relative relationships instead of simply reconstructing absolute positions.

---

Table S5: Top-1 aceuracy of position reconstruction using ViT-B/16 [21] pre-trained with different (i) a and (ii) т. We underline the special parameter different from our default settings.
(a) Pre-training with a = (Avg. acc: 88.81).
(b) Pre-training with (Avg. acc: 87.13).
7pos
0.25
0.50
0.75
0.95
Ypos
0.25
0.50
0.75
0.95
7
0.00
98.48
98.74
98.29
94.37
0.00
97.03
98.50
97.93
91.40
0.25
98.06
98.31
97.76
93.48
0.25
96.63
98.04
97.33
89.69
0.50
96.06
96,08
94.48
85,49
0,50
94.30
95.58
93.68
81.47
0.75
82.19
78.64
69.39
41.23
0.75
79.14
77.06
67.43
38.89
(c) Pre-training with a (Avg. acc: 69.48).
(d) Pre-training with
- 0 (Avg. acc: 87.83).
Ypos
0.25
0.50
0.75
0.95
Ypos
0.25
0.50
0.75
0.95
7
0.00
78.45
78.81
78.19
72.33
0.00
97.19
98.69
98.20
92.30
0.25
78.03
78.41
77.68
70.49
0,25
96.78
98.26
97.66
91.05
0.50
76.14
76.27
74.53
63.52
0.50
97.26
96.82
95.66
89.12
0.75
63.75
61.21
53.30
30.47
0.75
79.94
78.10
68.73
40.24
(c) Pre-training with a == - (Avg. acc: 87.65).
(f)
Pre-training with op (Avg. acc: 88.66).
Tpos
0.25
0.50
0.75
0.95
Ypos
0.25
0.50
0.75
0.95
7
0.00
97.63
98.61
97.93
91.30
0.00
98.66
98.31
97.72
91.22
0.25
97.22
98.19
97.46
89.96
0.25
98.54
98.17
97.46
91.00
0.50
94.99
95.85
94.06
82.50
0.50
96.85
96.20
94.52
84.64
0.75
80.33
77.92
68.42
40.11
0.75
83.57
79.30
70.04
42.29
Pre-training with
0.1
(Avg. 87.83).
(h)
Pre-training with
0.5 (Avg. 87.78).
Ppos
0.25
0.50
0.75
0.95
Ypos
0.25
0.50
0.75
0.95
0.00
97.19
98.69
98.20
92.30
0.00
97.68
97.94
97.82
91.78
0.25
96.78
98.26
97.66
91.05
0.25
97.83
97.91
97.53
91.01
0.50
97.26
96.82
95.66
89.12
0,50
96.44
96.06
94.52
84,49
0.75
79.94
78.10
68.73
40.24
0.75
80.16
77.16
66.01
40.18
We study the effectiveness of different values of "pos during pre-training in Tab. S4, and we find evidence for hard decide which patch positions reconstruct precisely. As shown by Tabs. S5a and S5f, the model achieves higher position prediction accuracy but performs worse on downstream tasks (please refer to Tabs. 3 and for downstream performances). Therefore, to prevent being overwhelmed by this particular position reconstruction task, techniques for relaxing the patch-wise classification problem become necessary, i.e., position smoothing, and attentive reconstruction.
18


Reviews:
[{"review_id": "w2u3Dz0EAY", "reviewer": "Reviewer_Pgbn", "summary": "This paper a method for self-supervised representation learning. Given a ViT architecture,  the authors propose to predict the  absolute position of masked positional embedings at random. Although the general direction is not new, the authors pose it in a simple an interesting way, that achieves good performance in the downstream tasks. ", "questions": "Please see above.", "limitations": "Yes, they have.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The proposed ides is simple yet effective, and is executed well.\n2) The presented is presented properly and is easy to follow.\n3) The authors evaluate their method properly on several downstream tasks, achieving acceptable performance.", "weaknesses": "The direction of obtaining a supervision signal from the patch positions in the ViT architecture has been explored before. The authors mention the majority of them in their related work section but do not discuss what the advantage of their proposed method is. The performance supports the effectiveness of their method, but they could discuss more explicitly their advantages and ablate on that. Moreover, a comparison with [1] would be informative.\n\n[1] Sameni et al., Representation Learning by Detecting Incorrect Location Embeddings, In AAAI 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Pgbn for the valuable time and constructive feedback. Point-to-point responses\nare provided below.\n\n**Q1: The advantage of the proposed DropPos should be discussed explicitly and lack of\ncomparison with [a].**\n\n**A1:** Appreciate! We would like to discuss the advantage of our DropPos explicitly. As mentioned in\nthe manuscript, there are three main difficulties in designing position-related self-supervised pretext\ntasks for ViTs. Specifically, (1) eliminating the Discrepancies between pre-training and fine-tuning,\n(2) avoiding trivial solutions to ensure highly semantic representations, and (3) reducing the impact of\nconfusing targets caused by similar visual appearances. All other methods cannot manage all of them,\nwhich is summarized in the following table.\n\n| Method | Eliminate Discrepancies | Avoid Trivial Solutions | Remove Confusing targets |\n|-------------------|-------------------------|-------------------------|--------------------------|\n| Zhai et al. [60]  | × | × | × |\n| Caron et al. [4]  | √ | × | √ |\n| Sameni et al. [a] | √ | × | √ |\n| DropPos           | √ | √ | √ |\n\n\nSpecifically, Zhai et al. [60] simply discard all positional embeddings during pre-training, and\nthus discrepancies arise. Therefore, the fine-tuning performances of [60] largely lag behind the\nstate-of-the-art.\n\nCaron et al. [4] propose to predict the relative location of a local crop to the corresponding global\ncrop, making it time-consuming and hard to learn highly semantic representations as this pretext task\nis somewhat too simple for powerful ViTs. ViTs may simply solve this task by comparing the texture\nof two given crops.\n\nSameni et al. [a] come up with an auxiliary position-related objective and combine it with the popular\ncontrastive learning paradigm. Therefore, the generalization abilities of learned representations\nare highly related to data augmentation techniques. Also, the position-related task itself proposed\nby [a] (without contrastive learning) may become a trivial solution, as identifying several mismatched\npositions is relatively easy for powerful ViTs.\n\nDropPos solves the mentioned three difficulties by dropping a subset of position embeddings, dropping\na large subset of patch tokens, and position smoothing and attentive reconstruction, respectively.\nDespite these advantages, the pre-training procedure of DropPos is ≈ 3× more efficient than [60, 4,\na] thanks to the patch masking stage.\n\nIn fact, all these three advantages have been ablated in the manuscript. Specifically, we ablated\nthe effectiveness of alleviating discrepancies in Table 2 (γ_pos = 0.75 *v.s.* γ_pos = 1), where an\nimprovement of +0.3% fine-tuning top-1 accuracy is observed. The second advantage is ablated\nin Table 1 (γ = 0.75 *v.s.* γ = 0), where an improvement of +1.02% fine-tuning top-1 accuracy is\nobserved. The effectiveness of eliminating confusing reconstruction targets is ablated in Table 3\n(σ = 1 → 0 *v.s.* σ = 0) and Table 4 (τ = 0.1 *v.s.* τ = ∞), where improvements of +0.11% and\n+0.12% fine-tuning top-1 accuracies are observed.\n\nWe will emphasize these results in the revision to further help address the concerns.\n\n**References**\n\n[a] Sepehr Sameni et al. Representation Learning by Detecting Incorrect Location Embeddings. In\nAAAI 2022."}, {"review_id": "prbxl07qUd", "reviewer": "Reviewer_e3Vr", "summary": "This paper introduces DropPos, a self-supervised pretext task designed to enhance the location awareness of Vision Transformers (ViTs). By dropping positional embeddings and reconstructing the positions of visible patches with some auxiliary strategies, DropPos improves spatial reasoning abilities in ViTs. Experimental results demonstrate the efficacy of DropPos, outperforming supervised pre-training and achieving competitive performance against state-of-the-art self-supervised methods. The paper provides good insights on enhancing location awareness in ViTs for future works.", "questions": "1.\tRegarding the DropPos method: Could you provide more specific details and clarity on the implementation of the DropPos method and the flowchart of the pseudo-code in Section 3.2? This would help readers better understand the proposed approach.\n2.\tStrengthening of the validation of motivation. Could you enhance the experimental analysis and discussion to provide more depth and intuition regarding the impact of DropPos on the position sensitivity of ViT? Can you consider conducting additional analyses or visualizations to better illustrate and explain the observed effects?\n3.\tPositional Encoding Initialization: It would be valuable to discuss the initialization strategy for positional encoding in the DropPos method. How was it initialized, and were different initialization strategies explored? Analyzing the impact of different positional encoding initialization strategies on the performance of DropPos would enhance the comprehensiveness of the study.\n4.\tScaling Properties and Generalization: Could you further investigate and discuss the scaling properties of the DropPos method on larger architectures, such as ViT-Huge? Is the method equally effective and robust on larger-scale models? Additionally, have you considered applying the DropPos method to other advanced vision transformers like Swin-Transformer and evaluating its performance?\n5.\tTransfer Learning and Robustness Evaluation: It is noted that there is a lack of studies on transfer learning using iNaturalists and Places datasets, as well as the robustness evaluation on ImageNet. I would like to see if the DropPos method remains effective on these benchmarks.\n6.\tThe Fig.~3 about the qualitative results of position reconstruction is somewhat confusing and requires a clearer explanation.\n", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe motivation is clear. The paper addresses the motivation of enhancing positional awareness and spatial reasoning abilities in vision transformers for pre-training.\n\n2.\tThe method is simple. The DropPos focuses on reconstructing dropped positions with some heuristics to avoid trivial solutions and ambiguities. Compare with the contrastive learning, it does not need complicated augmentations.\n\n3.\tThe method is effective. Compared to contrastive learning or masked image modeling, the proposed DropPos exhibits faster pretraining and slightly improved performance on several benchmarks.\n", "weaknesses": "1.\tThe initialization of the positional encoding for the DropPos is not discussed, and there is no analysis of the impact of the different PE initialization strategies for the proposed DropPos method.\n2.\tThe experiments are insufficient to validate the motivation. Although preliminary evidence is provided by experiments on downstream tasks such as detection and segmentation, which indicates that the DropPos enhances the location-awareness of ViT, the paper lacks more in-depth and intuitive experimental analysis and discussion to verify the strengthening of position sensitivity of ViT by the DropPos method.\n3.\tThe description of the DropPos method is not specific and clear enough, including the method implementation, the flowchat of pseudo-code in section 3.2. \n4.\tThe scaling properties of the DropPos on ViT are not explored in depth. For example, compared with the MAE method, It is questionable whether the method remains effective on Vit-Huge compared to the MAE. Experiments on more advanced vision transformers, such as Swin-Transformer, are also encouraged to be conducted.\n5.\tCompared with the MAE method, there is a lack of studies on transfer learning using iNaturalists and Places, and the robustness evaluation on ImageNet. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer e3Vr for the valuable time and constructive feedback. Point-to-point responses\nare provided below.\n\n**Q1: The initialization of the positional encoding.**\n\n**A1:** DropPos uses fixed 2D sin-cos position embeddings by default. We ablate the initialization of\nposition embeddings in the following table and it demonstrates that fixed sin-cos position embeddings\nachieves the best performance. We will add this in our revised version.\n\n| Initialization | Learnable | ImageNet | ADE20k |\n|---|---|---|---|\n| sin-cos | × | 82.96 | 40.68 |\n| sin-cos | √ | 82.81 | 39.37 |\n| random | √ | 82.48 | 38.72 |\n\n**Q2: The strengthening of improved position sensitivity.**\n\n**A2:** To systematically answer this question, we propose a metric to evaluate the model’s position\nsensitivity and explore the relationship between position sensitivity and performance on downstream\ntasks. Specifically, we freeze the backbone and train an extra linear position prediction head. Vanilla\ncross-entropy loss is used for training. Top-1 accuracies of position predictions before and after\nfine-tuning are reported, and 75% of position embeddings are randomly masked during training.\nHigher values mean the model is better at modeling the position relationship. The top-1 accuracy on\nthe ImageNet validation set after fine-tuning is also reported.\n\nPlease refer to Table 2 in the \"global\" response for detailed results.\n\nAs shown in the table, the backbone performs better in position prediction after fine-tuning, indicating\nthat *image classification indeed needs strong abilities in modeling spatial relationships*. It means that\n*better position sensitivity corresponds to better performances on downstream tasks*. By designing a\nposition prediction pretext task, the backbone pre-trained by DropPos has better position modeling\nabilities, performing better on a variety of downstream tasks.\n\n**Q3: The implementation of DropPos is not specific and clear enough.**\n\n**A3:** We apologize for the ambiguity. To clarify the flowchart of DropPos, we provide the pseudo-code\nfor computing the objective of DropPos. Please refer to the PDF in the \"global\" response for details.\nWe will add this and polish the method section to make it clearer and more readable.\n\n**Q4: The scaling properties and generalization of DropPos.**\n\n**A4:** The scaling property is worth studying when evaluating the effectiveness of a self-supervised\nalgorithm. Comparing performances between ViT-B and ViT-L, we can conclude that as the number\nof parameters in the model increases, the performance of DropPos is improving. However, the\nViT-Huge training cannot be finished in the short rebuttal period, we will try to provide the results in\nthe future revision.\n\nTo verify the scaling property and the generalization of DropPos, we provide experiments when\nDropPos is equipped with the Swin Transformer. We follow the implementation of UM-MAE [a] and\npre-train a Swin-Tiny from scratch using DropPos. All models are pre-trained with 200 epochs and\nfine-tuned with 100 epochs, following the configuration of UM-MAE [a].\n\nPlease refer to Table 1 in the \"global\" response for detailed results.\nFrom the table, we can conclude that DropPos still works on Swin Transformers, and thus enhancing\nthe location awareness of vision transformers is worth studying.\n\n**Q5: Lack of transfer learning results and the robustness evaluation.**\n**A5:** We recognize that these results are important to verify the generalization of a self-supervised\nalgorithm. However, MAE evaluated the performance on iNaturalists and Places365 by fine-tuning\non target datasets and the fine-tuning code has not been made publicly available. We will try to add\nthese experiments in future revisions.\n\nWe conduct robustness evaluation on ImageNet-Adversarial [b] and ImageNet-Rendition [c] using the\nsame models fine-tuned on the original ImageNet and only run inference on the different validation\nsets in the following table, which is exactly the same as MAE. As shown in the table, with only 800\nepochs of pre-training, DropPos achieves comparable or even better performances, demonstrating its\nrobustness.\n\n| Method | Backbone | Epoch | ImageNet-A [b] | ImageNet-R [c] | ImageNet |\n|---|---|---|---|---|---|\n| MAE | ViT-B | 1600 | **35.9** | 48.3 | 83.6 | \n| DropPos | ViT-B | 800 | 35.5 | **48.8** | **84.2** |\n| MAE | ViT-L | 1600 | **57.1** | **59.9** | **85.9** |\n| DropPos | ViT-L | 800 | 56.7 | 59.8 | 85.8 |\n\n**Q6: Figure 3 requires a clearer explanation.**\n\n**A6:** We apologize for the ambiguity. Figure 3 shows the qualitative results of position reconstruction\nunder different mask ratios γ. Black patches are masked during inference. The positions of those\nwhite patches are wrongly predicted, while the remaining patches are predicted correctly. From the\nfigure, we can conclude that DropPos manages to reconstruct precise positions even under extremely\ndifficult situations (e.g., γ = 0.75).\n\n**References**\n\n[a] Xiang Li et al. Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision\nTransformers with Locality. arXiv:2205.10063, 2022.\n\n[b] Dan Hendrycks et al. Natural adversarial examples. In CVPR, 2021.\n\n[c] Dan Hendrycks et al. The many faces of robustness: A critical analysis of out-of-distribution\ngeneralization. In ICCV, 2021."}, {"review_id": "QnZy2NuqUq", "reviewer": "Reviewer_HkQR", "summary": "This paper presents a simple yet effective approach for generative self-supervised representation learning on images, namely DropPos.  The proposed approach drops a large random subset of positional embeddings for visible tokens and classifies the actual position for these tokens via visual appearance. Experimentally, DropPos outperforms state-of-the-art self-supervised approaches on a wide range of downstream benchmarks including image classification, detection and segmentation.", "questions": "No. See Weaknesses.", "limitations": "No. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and organized.\n- The idea in this paper is simple yet effective, which brings something new in generative self-supervised representation learning.\n- The authors provide clear implementation details (e.g., Pseudo-Code), which makes it easier to be reproduced.\n- Detailed ablation study is conducted to verify the effectiveness of the proposed approach.\n- Achieving SOTA performance on various downstream tasks.", "weaknesses": "- As illustrated in (4), the cross-entropy loss is applied for dropped position supervision. Except for this loss, is additional loss used? e.g., the MSE reconstruction loss used in MAE.\n- In MAE, more layers are used in the decoder (i.e., 8). Here the decoder only consists of 2 layers. Is it because the dropped position classification task is easier than the patch reconstruction task?\n- What’s the overall training time for DropPos? Is it comparable or more efficient than existing self-supervised approaches?\n- For completeness, please add more dropout-guided work for discussion, e.g., DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks (CVPR23), which similarly employs dropout mechanism, but for spatial-attention dropout in videos. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer HkQR for the valuable time and constructive feedback. Point-to-point responses\nare provided below.\n\n**Q1: About the objective.**\n\n**A1:** DropPos uses only the cross-entropy loss mentioned in the manuscript, and the MSE loss used in\nMAE is not adopted. We will clarify this in our revised version.\n\n**Q2: About the decoder.**\n\n**A2:** In fact, when using decoders with different depths, the fine-tuning accuracy of MAE is almost\nthe same (please refer to Table 1a in MAE [27]). We adopt a shallower decoder simply because it is\nmore efficient. Moreover, we provide an ablation over the decoder depth in the following table. We\ntake ViT-B as the backbone and all models are pre-trained with 200 epochs. From the table, we can\ntell that DropPos appears to be robust against different decoder depths.\n\n| # blocks | ImageNet | ADE20k |\n|---|---|---|\n| 2 | 82.96 | 40.68 |\n| 8 | 82.88 | 40.05 |\n\n**Q3: About the overall training time.**\n\n**A3:** The training procedure of each iteration is as efficient as MAE, and the overall training time is\nhalf that of MAE since DropPos is pre-trained with only 800 epochs.\n\n**Q4: Lack of discussion with dropout-guided works.**\n\n**A4:** Appreciate! We provide a brief discussion with representative dropout-guided studies in the\nfollowing. [a] adaptively performs spatial-attention dropout in the frame reconstruction to facilitate\ntemporal correspondence learning in videos, leading to a stronger temporal matching learner in visual\nobject tracking and segmentation. [b] adopts feature-level dropout to the common weak-to-strong\npipeline in semi-supervised semantic segmentation, bringing a broader perturbation space and thus\nresulting in better performances.\n\nTo the best of our knowledge, DropPos is the first work that proposes a brand new self-supervised\npretext task using dropout on position embeddings.\n\n**References**\n\n[a] Qiangqiang Wu et al. DropMAE: Masked Autoencoders with Spatial-Attention Dropout for\nTracking Tasks. In CVPR 2023.\n\n[b] Lihe Yang et al. Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic\nSegmentation. In CVPR 2023."}, {"review_id": "F8mBky6qkC", "reviewer": "Reviewer_6DGy", "summary": "This paper introduces a novel approach to self-supervised representation learning for vision transformers, focusing on enhancing their positional awareness. The authors proposed a new pretext task called DropPos, which involves reconstructing the positions of dropped tokens in partial observations. By leveraging DropPos, which is a spiritual adaptation of MAE, the authors consistently achieve superior results compared to the state-of-the-art (SOTA) techniques in standard downstream tasks, including image classification, object detection, and semantic segmentation.", "questions": "1. Why do we need to compute Affinity as indicated in equation (9). Why can't we use the self-attention matrix (softmax(KQ'))?\n2. In Table 3, it will be also interesting to check the results of 2->1.\n3. What happens when ViT is pretrained with DropPos for longer epochs like 1600 epochs?\n4. In fig 3, how do we know that which predictions are correct? \nFor rebuttal, please also refer to the points mentioned in the weakness section.", "limitations": "The paper acknowledges a limitation in its conclusion, which the reviewer agrees with. Although the experiments conducted with ViT-B are deemed sufficient to demonstrate the potential of the proposed method, it is crucial to provide a comprehensive explanation and understanding of the model, regardless of its size. This ensures a robust analysis and comprehension of the proposed approach beyond the specific ViT-B architecture.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper effectively addresses the problem of incorporating inductive bias into vision transformers, and it takes an interesting approach by employing self-supervised learning (SSL) techniques. Enhancing the positional awareness of vision transformers is a significant aspect, as it can greatly improve their performance in downstream tasks that rely on location awareness and positional information. The proposed pretext task, DropPos, is simple yet effective, which aligns with the requirements of SSL. Additionally, such pretext tasks are efficient as they involve masking a substantial portion of the input data. The paper highlights that using tasks like DropPos eliminates the need for careful selection of target representation and mask strategy, as typically performed in mask image modeling. Furthermore, the paper is well-written and easy to follow. ", "weaknesses": "One potential weakness of the paper lies in its experimental evaluation. While the proposed pretext task is commended for its simplicity and effectiveness, it would have been valuable to compare the performance of a DropPos pretrained Vision Transformer (ViT) with a hierarchical transformer such as Swin, pretrained using mask image modeling [A, B]. This comparison would have provided insights into whether a carefully designed ViT architecture already addresses the need for positional awareness, rendering the pretext task redundant.\n\nFurthermore, the paper lacks an in-depth analysis of what the ViT is actually learning and how it achieves positional awareness. Quantifying the extent of positional-aware representation learning is crucial. Analysis such as intertoken distance within a layer, sparsity of attention weights, and linear probe results are missing, which would have shed light on the underlying reasons for the success of the proposed pretext task.\n\nA significant concern arises in the evaluation section where the authors reproduce most of the state-of-the-art (SOTA) results reported in other papers. However, the reproduced numbers are generally lower than the original reported results, without any mention of the differences. It would be important for the authors to investigate the causes of these discrepancies, considering factors such as differences in the experimental setup (e.g., number of GPUs, changes in batch size and learning schedules) or potential missing engineering tricks. Particularly concerning is the lack of significant improvement (wrt MAE) in object detection and segmentation results, which are crucial for spatial modeling.\n\nLastly, the authors missed referencing relevant literature on contrastive learning approaches designed for spatial modeling, such as [C]. In fact, [C] outperforms DropPos when trained for 200 epochs on the ADE20k dataset.\n\n[A] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai and Han Hu. SimMIM: A Simple Framework for Masked Image Modeling\n[B] Xiang Li, Wenhai Wang, Lingfeng Yang, Jian Yang. Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality.\n[C] Patch-level Representation Learning for Self-supervised Vision Transformers, CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 6DGy for the valuable time and constructive feedback. Point-to-point responses\nare provided below.\n\n**Q1: DropPos with Swin.**\n\n**A1:** We provide experiments when DropPos is equipped with Swin. We follow the implementation of\nUM-MAE [a] and pre-train a Swin-Tiny from scratch. Please refer to Table 1 in the \"global\" response\nfor detailed results. From the table, we can tell that *even a carefully designed ViT architecture has\nnot addressed the need for positional awareness yet.*\n\n**Q2: An in-depth analysis and the linear probing performance.**\n\n**A2:** We answer this question in the following three perspectives.\n\n(1) Following your suggestion, we illustrate the sparsity of attention weights and the inter-token\ndistance within a layer in Figure 1 and Figure 2 of the \"global\" response, respectively.\n- Figure 1 demonstrates that compared with MAE, *features of DropPos tend to have more sparse attention maps, especially at shallow layers.*\n- In Figure 2, we can conclude that compared with MAE, the shallow features of DropPos (depth 2 to 8) have lower distances, indicating smaller receptive fields. It means that *local patch relationships are more informative to help discriminate positions and enhance location awareness.*\n\n(2) To provide an in-depth analysis, we propose a metric to evaluate the model’s position sensitivity\nand explore the relationship between position sensitivity and performance on downstream tasks.\nSpecifically, we freeze the backbone and train an extra linear position prediction head using the\nvanilla cross-entropy loss. Top-1 accuracies of position predictions are reported, and 75% of position\nembeddings are randomly masked during training. Higher values mean the model is better at modeling\nthe position relationship. Please refer to Table 2 in the \"global\" response for detailed results. As\nshown in the table, the backbone performs better in position prediction after fine-tuning, indicating\nthat *image classification indeed needs strong abilities in modeling spatial relationships*. It means that\n*better position sensitivity corresponds to better performances on downstream tasks.* By designing a\nposition prediction pretext task, the backbone pre-trained by DropPos has better position modeling\nabilities, performing better on a variety of downstream tasks.\n\n(3) The linear probing accuracy of DropPos is 43.45% (ViT-B).\n\n**Q3: About the reproduced numbers, the insignificant improvements over MAE on detection\nand segmentation.**\n\n**A3:** The difference is due to the training iteration. For detection and segmentation tasks, we first\ndownload the pre-trained backbone and then perform end-to-end fine-tuning using the configuration\nof ViTDet [33] and mmsegmentation [14] for COCO and ADE20k experiments, respectively. *For\nefficient training, we perform 12 epochs of fine-tuning instead of 100 epochs on COCO, and 80k\niterations instead of 160k iterations on ADE20k*, following [48] and [18]. Therefore, although the\nreproduced numbers are lower than their original numbers, we have conducted a fair comparison. We\nwill clarify this and try to conduct experiments with longer schedules in our revised version.\n\nWe would like to point out that the improvement over MAE on the ADE20k semantic segmentation\nbenchmark is significant (+0.8 mIoU). As for COCO experiments, the improvements seem to be\na little bit incremental may be because ViTDet [33] was originally tuned based on MAE, and we\ndid not tune any parameters. However, consistent improvements over MAE are observed on COCO\nbenchmarks, verifying the effectiveness of DropPos.\n\n**Q4: Lack of comparison with [b].**\n\n**A4:** We will add a brief discussion on contrastive learning approaches designed for spatial modeling\nin our revised version. As for the performances, [b] used the multi-crop augmentation technique.\nTherefore, the effective pre-training epoch should be $200 \\cdot \\frac{2 \\cdot 224^2 + 8 \\cdot 96^2}{224^2} \\approx 700$ instead of 200.\nSimply comparing DropPos pre-trained with only 200 epochs seems to be unfair.\n\n**Q5: Affinity instead of self-attention.**\n\n**A5:** The self-attention map is a reasonable alternative. However, we empirically found that using\naffinity brings slightly better performances than using the self-attention map.\n\n**Q6: More experiments in Table 3.**\n\n**A6:** Appreciate! We conduct the suggested experiment. When $\\sigma = 1 \\to 0$, DropPos achieves 82.68 top-1 on ImageNet and 39.97 mIoU on ADE20k. This evidence indicates that reconstructing precise positions at the end of training is beneficial.\n\n**Q7: DropPos with 1600 epochs.**\n\n**A7:** We fail to pre-train a backbone with such a long schedule during the short rebuttal period due\nto limited computational resources. However, we hypothesize that DropPos is expected to perform\nbetter with a longer schedule, as the top-1 accuracy of position predictions is around 96% and still\nhas room to improve. We will try to add this experiment in the future.\n\n**Q8: About Figure 3.**\n\n**A8:** We apologize for the ambiguity. Figure 3 shows the qualitative results of position reconstruction\nunder different mask ratios γ. Black patches are masked during inference. The positions of those\nwhite patches are wrongly predicted, while the remaining patches are predicted correctly. From the\nfigure, we can conclude that DropPos manages to reconstruct precise positions even under extremely\ndifficult situations (e.g., γ = 0.75).\n\n**References**\n\n[a] Xiang Li et al. Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision\nTransformers with Locality. arXiv:2205.10063, 2022.\n\n[b] Sukmin Yun et al. Patch-level Representation Learning for Self-supervised Vision Transformers.\nIn CVPR 2022."}, {"review_id": "DzDv8Jxa9u", "reviewer": "Reviewer_S2Rp", "summary": "The paper introduces a novel self-supervised pretext task for Vision Transformers (ViTs), called DropPos. It aims to enhance the spatial reasoning or location awareness of ViTs, based on the observation that ViTs are often insensitive to the order of input tokens. DropPos works by dropping a large random subset of positional embeddings, then using the model to predict the actual position of each patch based solely on its visual appearance.\n\nThe paper identifies three major difficulties: (a) discrepancies between pre-training and fine-tuning (b) trivial solutions that fail to learn highly semantic representations by solving this simple task (c) patches with similar visual content. To prevent trivial solutions and increase task difficulty, this paper keeps only a subset of patches visible during the task. Given the potential similarity in visual appearances between different patches, the authors propose position smoothing and attentive reconstruction strategies. This relaxation allows for non-exact position reconstruction when exact positions are not critical. Quantitative evaluations demonstrate the effectiveness of DropPos, outperforming supervised pre-training and yielding competitive results against state-of-the-art self-supervised alternatives on various benchmarks. ", "questions": "I am not really sure so I put this in Questions instead of Weaknesses. After checking the main paper and supplementary, it seems the authors did not mention if the pre-training is conducted with DropPos only, or together with other pre-training tasks like in MAE. Could the authors answer whether DropPos is the only training objective?\n\nIf the answer is yes, the reviewer would hope the authors can discuss a bit about why such a very coarse-grained task can be more powerful than the counterparts. Does it mean the dense visual cues are not important in pre-training?\n\n\nOverall I am feeling this paper is a good trial, although it may need more in-depth analysis and appropriate discussion about relevant works.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The main claim is attractive. It would be quite interesting (and a bit counter-intuition) to see the vision transformers can learn a very good representation by such a simple patch predicting task, which is very coarse-grained. Particularly, the results of this paper look competitive to the mainstream pre-training tasks.\n\n2. This work provides extensive ablation studies to verify their design and to reveal some insights.\n\n3. It proposes Position smoothing and Attentive reconstruction to solve the problems like patches may share similar visual appearance.\n", "weaknesses": "\n1. The paper requires additional experiments and deeper analysis to substantiate some assertions.\n\nFor example, this paper starts from \"Vision Transformers (ViTs) are quite insensitive to the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident.\". \n\nAlthough this claim fits my intuition, it is weird that after citing some papers [13, 39, 60], the paper does not talk about this claim any more in the following sections. What would happen if messing up the order of input tokens? Or even further, should this really be viewed as a drawback? Insensitiveness to the order may also be a good property. The authors touch on this in the related work, but this claim needs more attention since it's the main drive of the paper.\n\nAlso, it's unclear if DropPos helps this issue. More specifically, if a model is pre-trained by DropPos and then finetuned in ImageNet, would it be more sensitive to the order of input tokens? Looking into this would help us understand if the improved performance comes from the model being more \"sensitive to the order of input tokens\".\n\n2. A recent work: Jigsaw-ViT: Learning jigsaw puzzles in vision transformer. \n\nAnother work Jigsaw-ViT proposes to include solving Jigsaw in the training of ViT, which is close to the task of this work. It is beneficial to include this for comparison.\n\n[1] Chen, Yingyi, Xi Shen, Yahui Liu, Qinghua Tao, and Johan AK Suykens. \"Jigsaw-ViT: Learning jigsaw puzzles in vision transformer.\" Pattern Recognition Letters 166 (2023): 53-60.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer S2Rp for the valuable time and constructive feedback. Point-to-point responses\nare provided below.\n\n**Q1: Additional experiments and deeper analysis are required to verify the motivation.**\n\n**A1:** We observed that the improved position sensitivity results in better feature representation and\nbenefits to downstream tasks. To verify this argument, we first propose a metric to evaluate the\nmodel’s position sensitivity. Specifically, we freeze the backbone and train an extra linear position\nprediction head using the vanilla cross-entropy loss. Top-1 accuracies of position predictions before\nand after fine-tuning are reported, and 75% of position embeddings are randomly masked during\ntraining. Higher values mean the model is better at modeling the position relationship. The top-1\naccuracy on the ImageNet validation set after fine-tuning is also reported.\n\nPlease refer to Table 2 in the \"global\" response for detailed results.\n\nAs shown in the table, the backbone performs better in position prediction *after* fine-tuning, indicating\nthat image classification indeed needs strong abilities in modeling spatial relationships. It means that\nbetter position sensitivity corresponds to better performances on downstream tasks. This evidence\nsuggests that our motivation, i.e., enhancing the location awareness of ViTs, is reasonable, and the\ntopic is worth studying. By designing a position prediction pretext task, the backbone pre-trained by\nDropPos has better position modeling abilities, performing better on a variety of downstream tasks.\n\n**Q2: Lack of comparison with [a].**\n\n**A2:** We apologize for the missed comparison. JigsawViT [a] explores solving jigsaw puzzles as an\nauxiliary objective in ViT for supervised image classification. The authors have demonstrated that the\nproposed extra objective brings consistent and significant improvements in supervised tasks. However,\nDropPos aims to design a brand new self-supervised pretext task that enhances the location awareness\nof ViTs, which is totally different. Therefore, it is relatively hard to compare the performances of\nthese two works under the same benchmark.\n\n**Q3: About the objective.**\n\n**A3:** Sorry for the ambiguity. DropPos uses only the cross-entropy loss mentioned in the manuscript,\nand the MSE loss used in MAE is not adopted. We will clarify this in our revised version.\n\n**Q4: The reason why such a very coarse-grained task can be more powerful than its counterparts.**\n\n**A4:** This is an interesting question. It may be because images are natural signals with heavy spatial\nredundancy [27]. Towards this issue, some interesting works explore the best target representations\nfor masked image modeling, e.g., [36] and [52]. They empirically found that raw RGB pixels may\nnot be the best choice. Using coarse-grained targets such as HoG [52] features even performs better.\nOn the other hand, instance discrimination or contrastive learning is also a very coarse-grained task.\nIt ignores the possibility of different samples belonging to the same category.\n\nTherefore, what matters in pre-training seems to be highly semantic clues, rather than dense cues. An\nappropriate pretext task is still worth exploring.\n\n**References**\n\n[a] Yingyi Chen et al. Jigsaw-ViT: Learning jigsaw puzzles in vision transformer. Pattern Recognition\nLetters 166 (2023): 53-60."}]

Author rebuttal:
To all reviewers:

Thank you so much for your careful review and suggestive comments. Following your suggestions, we present some extra figures and tables in the PDF. We also provide the pseudo-code for computing the objective of DropPos. Specifically,
- **@Reviewer e3Vr**, to clarify the flowchart of DropPos, we provide the pseudo-code for computing the objective in Algorithm 1.
- **@Reviewer 6DGy**, to evaluate the sparsity of attention maps of DropPos, we compare attention maps generated by MAE and DropPos at the first and the last Transformer blocks in Figure 1.
- **@Reviewer 6DGy**, to measure the receptive field of DropPos, we compare the mean attention distance of MAE and DropPos in Figure 2.
- **@Reviewer e3Vr** and **Reviewer 6DGy**, we conduct experiments in Table 1 to verify that DropPos also works with Swin Transformers.
- **@Reviewer e3Vr**, **Reviewer 6DGy**, and **S2Rp**, to explore the strengthening of improved location awareness, we provide an in-depth and intuitive experimental analysis in Table 2, where we freeze the pre-trained backbone and train an extra linear patch classification head.

Sincerely,

Authors.

Meta-review: All the five reviewers recommend acceptance. The authors should add the clarifications in their rebuttal to the final version.
Decision: Accept (poster)