Title: DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Self-Supervised Learning

Vision Transformer

Visual Representation Learning
Abstract: As it is empirically observed that Vision Transformers (ViTs) are quite insensitive to the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident. To address this, we present DropPos, a novel pretext task designed to reconstruct Dropped Positions. The formulation of DropPos is simple: we first drop a large random subset of positional embeddings and then the model classifies the actual position for each non-overlapping patch among all possible positions solely based on their visual appearance. To avoid trivial solutions, we increase the difficulty of this task by keeping only a subset of patches visible. Additionally, considering there may be different patches with similar visual appearances, we propose position smoothing and attentive reconstruction strategies to relax this classification problem, since it is not necessary to reconstruct their exact positions in these cases. Empirical evaluations of DropPos show strong capabilities. DropPos outperforms supervised pre-training and achieves competitive results compared with state-of-the-art self-supervised alternatives on a wide range of downstream benchmarks. This suggests that explicitly encouraging spatial reasoning abilities, as DropPos does, indeed contributes to the improved location awareness of ViTs. The code is publicly available at https://github.com/Haochen-Wang409/DropPos.

DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions

Abstract

As it is empirically observed that Vision Transformers (ViTs) are quite insensitive to the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident. To address this, we present DropPos, a novel pretext task designed to reconstruct Drop ped Pos itions . The formulation of DropPos is simple: we first drop a large random subset of positional embeddings and then the model classifies the actual position for each non-overlapping patch among all possible positions solely based on their visual appearance. To avoid trivial solutions, we increase the difficulty of this task by keeping only a subset of patches visible. Additionally, considering there may be different patches with similar visual appearances, we propose position smoothing and attentive reconstruction strategies to relax this classification problem, since it is not necessary to reconstruct their exact positions in these cases. Empirical evaluations of DropPos show strong capabilities. DropPos outperforms supervised pre-training and achieves competitive results compared with state-of-the-art selfsupervised alternatives on a wide range of downstream benchmarks. This suggests that explicitly encouraging spatial reasoning abilities, as DropPos does, indeed contributes to the improved location awareness of ViTs. The code is publicly available at https://github.com/Haochen-Wang409/DropPos .

1 Introduction

Learning extensible visual representations without any human annotations, known as self-supervised learning (SSL), has become a research hotspot in the field of computer vision [7,9,26,27,29,47,59,62] since it enables efficient transfer to various downstream benchmarks. To achieve this goal, researchers carefully design visual pretext tasks to produce appropriate supervision using only images [3,18,46, 52,62,69,70]. Two popular approaches among these are contrastive learning (CL) [62] and masked image modeling (MIM) [3], both of which have demonstrated promising scaling behavior of vision models, particularly Vision Transformers (ViTs) [21]. Despite the success of these methods, ViTs are found to be relatively insensitive to the order of input tokens [13, 43, 67], leading to the hypothesis that they tend to model the relationship between a set of unordered input tokens. Therefore, a natural question arises: beyond the current CL and MIM paradigms, whether a pretext task that explicitly enhances the positional awareness of ViTs can further improve their representation learning abilities?

<latexi sh

1_b

64="GNJr

Q

o

k

9

SfI

A

E/j

>

B7X

c

VD

g

K2

W

O

C

P

R

w

Z

+n

T0z3

y

v

d

u

L

Y

H

p

U

q

M

m

5

agreement

model

(a) Contrastive learning.

image

masked image

model

image

(b) Masked image modeling.

image

dropped position

⊕

(c) Dropped positions reconstructing.

positions

model

To answer this question, we begin by revisiting the forward procedure of ViTs. A sequence of positional embeddings (PEs) [51] is added to patch embeddings to preserve position information. Intuitively, simply discarding these PEs and requesting the model to reconstruct the position for each patch naturally becomes a qualified location-aware pretext task. By forcing the model only to utilize visual appearance for position reconstruction, the model is supposed to learn the shape relationships and layouts of image contents, thereby improving the ability to encode spatial visual content. However, despite the simplicity and intuitiveness of this idea, position-related methods such as [18, 44] have fallen far behind. We identify several difficulties in designing this type of paradigm:

(i) Discarding all PEs brings discrepancies between pre-training and fine-tuning because the model has never been exposed to any PEs during pre-training. (ii) Strengths of ViTs in modeling long-range dependencies may cause them to solve the task superficially, leading to trivial solutions that fail to learn highly semantic representations by solving this simple task. (iii) Patches with similar visual appearances may result in confusing reconstruction targets, making it crucial to decide which position to reconstruct precisely.

Driven by this analysis, we present a novel, straightforward, and highly effective pretext task for self-supervised visual representation learning: reconstructing Drop ped Pos itions (DropPos). DropPos tackles the above issues systematically: (i) To address discrepancies, simply dropping a large subset of PEs instead of discarding all of them works effectively. (ii) To avoid trivial solutions, we only keep a subset of patches visible during pre-training, forcing ViTs to reconstruct the position of each visible patch with only partial inputs. (iii) To prevent ViTs from being overwhelmed by this particular task, we propose to relax this problem using position smoothing and attentive reconstruction strategies.

DropPos can be easily formulated into a simple patch-wise classification problem. Here is how it works: First, input patches are randomly masked and a large subset of PEs corresponding to these visible patches is dropped. Then, ViTs are tasked with classifying the positions of visible patches among all possible candidates, relying on partial observations and a few anchors ( i.e. , patches with unmasked PEs). Fig. 1 illustrates two benefits of DropPos: (i) requiring less prior knowledge of data augmentation techniques compared to CL, and (ii) eliminating the need for a careful selection of target representation [39] and mask strategy [8, 54, 56] as MIM does.

We conducted extensive experiments to evaluate the performance of DropPos in comparison to state-of-the-art alternatives across various downstream tasks. With only 800 epochs of pre-training, DropPos achieves 84.2% top-1 accuracy on ImageNet-1K [48] validation set using ViT-B/16 [21], outperforming MAE [28] pre-trained with 1600 epochs by +0.6%. Moreover, on COCO [38] object detection/segmentation, and ADE20k [71] semantic segmentation benchmarks, which requires more abilities of spatial reasoning, DropPos surpasses MAE [28] consistently .

2 Related work

Self-supervised learning. The key challenge of self-supervised learning is the design of pretext tasks [18, 28, 32, 42, 44, 45, 50, 54, 69], i.e. , how to produce appropriate supervision signals using only images. Among them, contrastive learning [5,9,26,29,58,60,62,66] has been popular recently, which seeks to maximize agreement between different views of the same images. Nevertheless, the

performance of contrastive-related methods highly depends on carefully-designed data augmentation techniques [5,6,9,11,12,72]. Another mainstream is masked image modeling [1,3,10,28,54,56], which involves predicting specific contents masked patches [2, 3, 19, 20, 28, 54, 64, 72]. Unlike these methods, DropPos offers a novel alternative for self-supervised ViTs: reconstructing dropped positions based on only visual clues. In this way, the model is urged to learn the spatial arrangement of local image patches, contributing to better localization abilities, which is important for spatial reasoning recognition tasks, e.g. , segmentation [22,53,55,57,58] and video understanding [17,25].

Position prediction in self-supervised learning. Doersch et al. [18] first split an image into 3 × 3 grids and train a network to predict the relative position of paired patches from the same image. This problem is formulated as an 8-way classification. Noroozi and Favaro [44] extended this approach to solve 'jigsaw puzzles' by urging the model to predict the order of shuffled nonoverlapping crops. These approaches were originally developed using ConvNets [33] and thus lacked the ability to learn long-range dependencies, which is crucial for position prediction. Very recently, Zhai et al. [67] revisited this type of pretext task in the scope of Transformers [51] by pre-training ViTs [21] to predict patch positions solely based on their visual appearance, while discarding positional embeddings. However, these pre-trained models have never been exposed to positional embeddings, leading to discrepancies between pre-training and fine-tuning, and thus their performances are significantly inferior to state-of-the-art alternatives [6,13,28,54]. Sameni et al. [49] come up with an auxiliary position-related objective and combine it with the popular contrastive learning paradigm. Caron et al. [4] extended this idea by predicting the relative location of a query ( i.e. , local crop) to the corresponding reference ( i.e. , global crop). However, [4] only examined the effectiveness on segmentation benchmarks [15,24,41,71]. In contrast, DropPos incorporates a novel dropped position reconstruction objective and thus achieves competitive performances on a variety of downstream tasks, including classfication [48], detection [38], and segmentation [38,71].

Positional embeddings in Vision Transformers. In the standard forward procedure of ViTs [21], learnable positional embeddings (PEs) are added with patch embeddings. Improving PEs and introducing more inductive bias into ViTs has become an active research area [23]. This is because some research shows that ViTs are surprisingly robust against permutations of the order of input patches. For instance, Chen et al. [13] demonstrate that the model performs well in classification even with no position embedding, suggesting that ViTs have not fully leveraged the positional information. Additionally, Naseer et al. [43] shows that ViTs are much less susceptible to patch shuffling perturbations than ConvNets. Therefore, we propose DropPos, a novel self-supervised pretext task that explicitly makes ViTs location-aware and promotes the emergence of spatial features.

3 Method

Preliminary: Vision Transformers. Given an image I ∈ R H × W × C , it is first reshaped into a sequence of patches I p ∈ R N × ( P 2 C ) , where ( H,W ) indicates the spatial resolution, C is the number of channels, P is the patch size, and N = HW/P 2 is the number of patches. A linear projection is then applied to I p , mapping it to D dimensions to get patch embeddings x ∈ R N × D . Also, a [CLS] token x cls ∈ R D is used to aggregate the information. Position embeddings p ∈ R ( N +1) × D are added to the patch embeddings to retain positional information. Then, z = [ x cls ; x ] ⊕ p is the input of transformer blocks [51] which consists of a stack of multi-head self-attention mechanisms, where ⊕ denotes element-wise plus. In particular, p 0 denotes the position embeddings of the [CLS] token, which will never be dropped.

3.1 DropPos

Fig. 2 illustrates our proposed DropPos. We first randomly generate a binary patch mask M ∈ { 0 , 1 } N , where 1 means visible and 0 means masked, respectively. Note that the [CLS] token, i.e. , z 0 , is always visible in our setting, and thus the length of M is N instead of N +1 for simplicity. Given a pre-defined mask ratio γ ∈ (0 , 1) , M is supposed to subject to

which means there are γN masked patches in total. It is worth noticing that we use patch masks to increase the difficulty of the pretext task, which is quite different from building a pretext task to

CLS

M

14

M

M

6

4

M

1

0

+

+

+

+

+

+

+

+

+

Vision Transformer

CLS

Projector

16

14

13

7

6

4

2

1

0

0

Position Embedding

M

Position Mask Token

0

Position Target

recover input images in MIM [3,28]. Then, we denote x vis = gather ( x , M ) ∈ R (1 -γ ) N × D to be the visible tokens, and position embeddings (PEs) for these unmasked tokens are randomly dropped, where gather ( · , · ) means only those unmasked patches ( i.e. , M i = 1 ) are reserved. To address possible discrepancies, we aim to drop a subset of PEs of these visible patches instead of simply discarding all of them. Thus, we randomly generate a binary positional mask M pos ∈ { 0 , 1 } (1 -γ ) N . Given a pre-defined position mask ratio γ pos ∈ (0 , 1) , M pos is supposed to subject to

which indicates that the model is supposed to reconstruct γ pos (1 -γ ) N dropped positions based on remaining (1 -γ pos )(1 -γ ) N anchor patches and the [CLS] token. PEs for visible patches p vis = gather ( p 1: N , M ) are gathered first. Then, we aim to obtain the final PEs p ′ ∈ R (1 -γ ) N × D by replacing those dropped positions with a learnable [MASK] token:

where i = 0 , 1 , . . . , (1 -γ ) N is the patch index, and p mask ∈ R D is the [MASK] token. Finally, the input of the Transformer encoder is z ′ = [ x cls ; x vis ] ⊕ p ′ ∈ R [(1 -γ ) N +1] × D .

Algorithm 1 Pseudo-Code of DropPos.

Implementation. Pre-training ViTs using our DropPos can be implemented efficiently and requires few specialized operations. The process involves three stages prior to forwarding features to the encoder. First, we randomly mask a subset of embedded patch tokens. Following MAE [28], the list of tokens is shuffled randomly, and the last portion of the list is removed. Next, positional embeddings of those unmasked patches are randomly dropped using the same masking operation. Then, positional mask tokens are introduced to obtain the final positional embeddings, which are added to the patch tokens finally and encoded by ViTs followed by a lightweight decoder. A simple classification objective is applied to those unmasked patches introduced as follows.

3.2 Pre-training DropPos

DropPos is pre-trained by reconstructing dropped positions. Concretely, as illustrated in Fig. 2 and presented in Sec. 3.1, an ViT [21] f θ parameterized by θ , takes z ′ as input. A lightweight decoder g ψ parameterized by ψ then projects f θ ( z ′ ) to patch-wise position predicitons o = g ψ ( f θ ( z ′ )) ∈ R (1 -γ ) N × N . We omit the [CLS] token here for simplicity. For each visible patch i ∈ { 0 , 1 , . . . , (1 -γ ) N -1 } , softmax ( o i ) ∈ R N represents the probability density function of the predicted positions. Next, we have to produce appropriate ground truths, i.e. , the actual position for each visible patch. To this end, we gather positions via y = gather ([0 , 1 , . . . , N -1] , M ) ∈ R (1 -γ ) N , where vector [0 , 1 , . . . , N -1] is naturally the actual position for all patches before masking (w/o the [CLS] token). Overall, the objective is simply the vanilla cross-entropy loss given true positions y ij :

Only patches with dropped positions, i.e. , orange patches in Fig. 2, contribute to this objective, which is reflected by the term (1 -M i pos ) in Eq. (4). Next, several techniques are proposed to prevent ViTs from being overwhelmed by this particular task, making them pay more attention to model spatial dependencies instead of simply reconstructing positions.

Position smoothing. Considering that different categories ( i.e. , positions) are not completely independent under this setting, the classification problem in Eq. (4) is relaxed when the model has predictions close to the actual position. Specifically, a weight matrix w ∈ (0 , 1) N × N is defined to measure similarities between different patches:

where dist( i, j ) means the Euclidean distance between patch i and j , and σ is a hyper-parameter. The matrix then should be normalized by w ∗ ( i, j ) = w ( i, j ) / ( ∑ N -1 k =0 w ( i, k ) ) . Here, w ∗ i ∈ R N indicates the smoothed ground truth when the actual position is i . Overall, the smoothed objective is:

Furthermore, we relax the problem in Eq. (6) by setting a large σ at the early stage, and we gradually decay the value of σ to produce a challenging pretext task. A simple linearly decay is performed:

where t and T denote the iteration and the number of total steps, respectively. σ 0 and σ T are the initial value and final value of σ , respectively. We set σ 0 = 1 and σ T = 0 by default.

Attentive reconstruction. Since there may be different patches that share similar visual appearance ( e.g. , two blue patches that represent the sky), it is not necessary to reconstruct their exact positions. Simply swapping these patches still maintains reasonable visual coherence. Therefore, we leverage the feature similarities between the [CLS] token and patch tokens to be an extra weight of Eq. (6):

where A ∈ R N denotes the similarity matrix. Concretely, A i means the affinity between the [CLS] token and patch token i . Let f cls ∈ R D and f ∈ R N × D be the features output by the encoder f θ , and thus the affinity A i is computed by

where τ indicates the temperature parameter and is set to 0.1 by default.

4 Experiments

Pre-training. We perform self-supervised pre-training on the ImageNet-1K [48] training set with a resolution of 224 × 224. By default, we take ViT-B/16 [72] as the backbone and perform 200 epochs of pre-training. The decoder is a stack of Transformer [51] and has a depth of 2 and a width of 512. Patch mask ratio γ and position mask ratio γ pos are both 0.75 by default. Our implementation is based on HPM [54]. Details can be found in Supplementary Material .

Evaluation. We perform supervised training to evaluate our DropPos with end-to-end fine-tuning on ImageNet-1K [48] for classification. By default, 100 epochs of fine-tuning are performed following common practices [28, 54] for ViT-B/16 [21]. We report top-1 validation accuracy of a single 224 × 224 crop. As for COCO [38], following previous methods [28,54], we take Mask R-CNN [30] with FPN [37] as the detector. We perform end-to-end fine-tuning on COCO [38] for 1 × schedule with 1024 × 1024 resolution. We report AP b for object detection and AP m for instance segmentation, respectively. Our implementation is based on detectron2 [61] and ViTDet [36]. For ADE20k [71], following previous methods [28,54], we take UperNet [63] as the decoder and perform end-to-end fine-tuning on ADE20k [71] with 512 × 512 resolution for 80k iterations. We take mIoU [24] as the main evaluation metric. Our implementation is based on mmsegmentation [14].

4.1 Ablation studies

In Tabs. 1 to 4, we take the ViT-B/16 [21] pre-trained with 200 epochs on ImageNet-1K [48] as the backbone. We highlight the default settings of our DropPos. Concretely, if not specified, the default setting is γ = 0 . 75 , γ pos = 0 . 75 , σ 0 = 1 , σ T = 0 , and τ = 0 . 1 . By default, 100 epochs of fine-tuning on ImageNet-1K [48], 1 × schedule fine-tuning on COCO [38], and 80k iterations fine-tuning on ADE20k [71] are performed.

To evaluate the performance on the pretext task, i.e. , reconstructing dropped positions, we also report the averaged top-1 accuracy of position predictions on the ImageNet-1K validation set in Tabs. 1 to 4. We vary γ ∈ { 0 , 0 . 25 , 0 . 5 , 0 . 75 } and γ pos ∈ { 0 . 25 , 0 . 5 , 0 . 75 , 0 . 95 } when measuring the position prediction accuracy, and average the position accuracy among 16 different cases.

Main properties. Illustrated by Tabs. 1 to 4, we find sufficient evidence to support the three difficulties claimed in Sec. 1. (i) In Tab. 1, ViTs fail to learn highly semantic representations by simply solving the position reconstruction task , because the performances of γ = 0 significantly lag behind. (ii) Discrepancies between pre-training and fine-tuning are revealed in Tab. 2 by setting

γ pos = 1 , where we observe performance deterioration. (iii) Deciding which position to reconstruct precisely is crucial because by setting σ = 0 in Tab. 3 and τ = ∞ in Tab. 4, ViTs have better position predictions task but perform worse on downstream tasks.

Performance on the position reconstruction task. In general, a positive correlation is observed between the accuracy of the downstream task and the averaged accuracy of predicted positions, indicating that the averaged accuracy of position predictions is a reliable indicator of how well the model is trained. However, there are some exceptions, i.e. , σ = 0 in Tab. 3 and τ = ∞ in Tab. 4, where the model reconstructs more precise positions but performs worse on downstream benchmarks, indicating that deciding which position to reconstruct precisely is crucial .

Please refer to Supplementary Material for the detailed performance of the position reconstruction task, where we provide more evidence to support the three difficulties proposed in Sec. 1.

Patch mask ratio γ . We vary the patch mask ratio γ from 0 to 0.75 in Tab. 1. When γ = 0 , the entire image is visible during pre-training, resulting in nearly perfect position predictions and making the task too simple for self-supervised learning (see Supplementary Material for details). However, the averaged accuracy of position predictions is quite low since the accuracy deteriorates quickly as we enlarge γ during evaluation. Adopting a larger γ for pre-training leads to better performance, especially on detection and segmentation benchmarks. Therefore, we can conclude that patch masking is essential in DropPos to prevent trivial solutions . However, deterioration is observed when γ = 0 . 9 . This is because the model may be under-fitted using this extremely difficult pretext task.

Position mask ratio γ pos . We study the effectiveness of the position mask ratio γ pos in Tab. 2. DropPos appears to be more robust against different γ pos than γ . However, an appropriate γ pos is still necessary. A small γ pos leads to trivial solutions even if we have a large γ , while an extremely large γ pos , e.g. , γ pos = 1 , results in discrepencies between pre-training and fine-tuning .

Position smoothing. By setting different values of σ 0 and σ T , we study the effectiveness of position smoothing. By setting σ = 0 , we remove the position smoothing strategy, and the model tends to be overwhelmed by this position reconstruction task, leading to high accuracy in position prediction but poor performance on downstream tasks . However, when σ is too large, the smoothed positions become noisy, contributing to poor performance on both downstream tasks and position reconstruction. Additionally, equipped with a linear decay schedule of σ , DropPos is gradually guided, bringing improvements on both downstream tasks and position reconstruction.

Attentive reconstruction. Attentive reconstruction is studied in Tab. 4 by setting different values of temperature τ , where τ = ∞ means no attentive reconstruction. Without attentive reconstruction, DropPos is able to obtain better position predictions but performs worse on downstream tasks. This is because we do not have to reconstruct the exact positions when different patches share similar visual appearances . This phenomenon can be mitigated by setting an appropriate τ . A large τ leads

to noisy position ground truths, leading to poor performance on both downstream tasks and position reconstruction, which is the same as a large σ .

4.2 Comparisons with previous results

We compare our proposed DropPos with the supervised pre-training baseline and a wide range of selfsupervised methods, including (i) contrastive learning methods [6,13], (ii) masked image modeling methods [3,28,34,54,56,64], and (iii) their combinations [19,72]. Effective pre-training epoch is used for fair comparison following [54,72] since it accounts for the actual trained images/views. The detailed definition can be found in Supplementary Material . All methods are pre-trained with the same resolution, i.e. , 224 × 224 on ImageNet-1K [48].

ImageNet-1K classification. We compare our DropPos with state-of-the-art alternatives on the ImageNet-1K [48] classification benchmark in Tab. 5. Notably, with only 200 epochs pre-training,

DropPos achieves 83.0% and 83.7% using ViT-B/16 and ViT-L/16 as the backbone, surpassing the supervised baseline by +1.1%, and MAE [28] by +0.8% and +0.4% respectively. This empirical evidence demonstrates that enhancing the location awareness of ViTs indeed brings better visual representations . Furthermore, with 800 epochs of pre-training, DropPos manages to achieve competitive results compared with the state-of-the-art. Specifically, it achieves 84.2% and 85.8% using ViT-B/16 and ViT-L/16, respectively. Strikingly, DropPos reaches competitive results with BootMAE [19], which combines contrastive learning and masked image modeling.

COCO object detection and segmentation. We fine-tune Mask R-CNN [30] on COCO [38] with 1 × schedule, i.e. , 12 epochs, using the configuration of ViTDet [36]. We take ViT-B/16 [21] as the backbone for all entries in Tab. 6. We regard AP b and AP m as the main metric for object detection and instance segmentation, respectively. For further comparison, we additionally report AP b 50 and AP b 75 for object detection, and AP m 50 and AP m 75 for instance segmentation. With only 200 epochs of pre-training, DropPos achieves 42.1% AP b and 37.9% AP m , outperforming MAE [28] by +2.0% and +1.5%, respectively. Note that these improvements appear to be more significant than those on the classification benchmark shown in Tab. 5, indicating that DropPos indeed contributes to better spatial reasoning abilities of ViTs . With 800 epochs of pre-training, DropPos achieves 47.7% AP b and 42.6% AP m , surpassing MAE [28] by +0.4% and +0.2%, respectively.

ADE20k semantic segmentation. We fine-tune UperNet [63] on ADE20k [71] with 80k iterations. We take ViT-B/16 [21] as the backbone for all entries in Tab. 6 and search for the optimal learning rate for each entry. We regard mIoU as the main metric for semantic segmentation. We additionally report aAcc for further comparison. With only 200 epochs of pre-training, DropPos achieves 40.7% mIoU, outperforming MAE [28] by +0.2%. With 800 epochs of pre-training, DropPos achieves 47.8% mIoU, surpassing MAE [28] by +0.8%.

Qualitative results. We load DropPos with 200 epochs of pre-training and provide qualitative results on this position reconstruction task in Fig. 3. DropPos manages to reconstruct the exact positions of most patches even in extreme circumstances , i.e. , γ = 0 . 75 and γ pos = 0 . 95 . This suggests that DropPos, as a self-supervised pretext task, indeed makes ViTs location-aware .

4.3 Analysis

To explore whether the improved position sensitivity results in better feature representation and benefits to downstream tasks or not, we propose a metric to evaluate the position sensitivity.

Specifically, we freeze the backbone and train an extra linear position prediction head using the vanilla cross-entropy loss. Top-1 accuracies of position predictions before and after fine-tuning are reported in Tab. 7, and 75% of position embeddings are randomly masked during training. Higher values mean the model is better at modeling the position relationship. The top-1 accuracy on the ImageNet-1K [48] validation set after fine-tuning is also reported.

As shown in Tab. 7, the backbone performs better in position prediction after fine-tuning, indicating that image classification indeed needs strong abilities in modeling spatial relationships . It means that better position sensitivity corresponds to better performances on downstream tasks . This evidence suggests that our motivation, i.e. , enhancing the location awareness of ViTs, is reasonable, and the topic is worth studying. By designing a position prediction pretext task, the backbone pre-trained by DropPos has better position modeling abilities, performing better on a variety of downstream tasks.

5 Conclusion

In this paper, we present DropPos, a novel, simple, and effective self-supervised pretext task that enhances the location awareness of ViTs. By masking patch tokens first and dropping positional embeddings next, ViTs are requesting to classify the position of each visible patch among all candidates based on partial visual appearance and a few anchor patches. In this way, we manage to avoid (i) discrepancies between pre-training and fine-tuning, (ii) trivial solutions , and (iii) reconstructing precise positions when unnecessary , resulting in improved spatial reasoning and understanding abilities of ViTs. Experiments across various benchmarks demonstrate the efficacy of DropPos, where DropPos consistently achieves competitive results compared with previous methods.

Discussion. Due to limited computational resources, we do not use DropPos to pre-train larger ViTs, e.g. , ViT-H/14. Despite these limitations, a growing need to design a pretext task that effectively uncovers the representation learning capabilities of vision models is becoming evident. We hope our DropPos will inspire future work. Additionally, how to enhance location awareness of vision models for spatial reasoning tasks in a supervised manner is also valuable to study.

References

Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. arXiv preprint arXiv:2301.08243 , 2023. 3

Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning (ICML) , pages 1298-1312. PMLR, 2022. 3

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations (ICLR) , 2022. 1, 3, 4, 8, 16

Mathilde Caron, Neil Houlsby, and Cordelia Schmid. Location-aware self-supervised transformers. arXiv preprint arXiv:2212.02400 , 2022. 3

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS) , 33:9912-9924, 2020. 2, 3, 15

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 9650-9660, 2021. 3, 8, 15

Fei-Long Chen, Du-Zhen Zhang, Ming-Lun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey on vision-language pre-training. Machine Intelligence Research , 20(1):38-56, 2023. 1

Haijian Chen, Wendong Zhang, Yunbo Wang, and Xiaokang Yang. Improving masked autoencoders by learning where to mask. arXiv preprint arXiv:2303.06583 , 2023. 2

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML) , pages 1597-1607, 2020. 1, 2, 3

Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.03026 , 2022. 3

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020. 3

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15750-15758, 2021. 3

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 9640-9649, 2021. 1, 3, 8, 9, 16

MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation , 2020. 6, 8, 16

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, UweFranke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3213-3223, 2016. 3

Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW) , pages 702-703, 2020. 15

Jiaxin Deng, Dong Shen, Haojie Pan, Xiangyu Wu, Ximan Liu, Gaofeng Meng, Fan Yang, Size Li, Ruiji Fu, and Zhongyuan Wang. A unified model for video understanding and knowledge embedding with heterogeneous knowledge graph dataset. In Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR) , 2023. 3

Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 1422-1430, 2015. 1, 2, 3

Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Bootstrapped masked autoencoders for vision bert pretraining. In European Conference on Computer Vision (ECCV) , pages 247-264. Springer, 2022. 3, 8, 9, 16

Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2023. 3

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR) , 2021. 1, 2, 3, 5, 6, 8, 9, 15, 17, 18

Ye Du, Yujun Shen, Haochen Wang, Jingjing Fei, Wei Li, Liwei Wu, Rui Zhao, Zehua Fu, and Qingjie Liu. Learning from future: A novel self-training framework for semantic segmentation. Advances in Neural Information Processing Systems (NeurIPS) , 2022. 3

Philipp Dufter, Martin Schmitt, and Hinrich Schütze. Position information in transformers: An overview. Computational Linguistics , 48(3):733-763, 2022. 3

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision (IJCV) , 88:303-308, 2009. 3, 6

Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in Neural Information Processing Systems (NeurIPS) , 35:35946-35958, 2022. 3

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurIPS) , 33:21271-21284, 2020. 1, 2

Yuxian Gu, Jiaxin Wen, Hao Sun, Yi Song, Pei Ke, Chujie Zheng, Zheng Zhang, Jianzhu Yao, Lei Liu, Xiaoyan Zhu, et al. Eva2. 0: Investigating open-domain chinese dialogue systems with large-scale pre-training. Machine Intelligence Research , 20(2):207-219, 2023. 1

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16000-16009, 2022. 2, 3, 4, 6, 8, 9, 15, 16

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9729-9738, 2020. 1, 2

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2961-2969, 2017. 6, 8, 9, 15

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European Conference on Computer Vision (ECCV) , 2016. 15

Dahun Kim, Donghyeon Cho, Donggeun Yoo, and In So Kweon. Learning image representations by completing damaged jigsaw puzzles. In IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 793-802, 2018. 2

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. 3

Gang Li, Heliang Zheng, Daqing Liu, Chaoyue Wang, Bing Su, and Changwen Zheng. Semmae: Semanticguided masking for learning masked autoencoders. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. 8, 16

Xiang Li, Wenhai Wang, Lingfeng Yang, and Jian Yang. Uniform masking: Enabling mae pre-training for pyramid-based vision transformers with locality. arXiv preprint arXiv:2205.10063 , 2022. 16

Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision (ECCV) , pages 280-296. Springer, 2022. 6, 8, 9, 15

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2117-2125, 2017. 6, 15

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV) , pages 740-755. Springer, 2014. 2, 3, 6, 8, 9, 15

Xingbin Liu, Jinghao Zhou, Tao Kong, Xianming Lin, and Rongrong Ji. Exploring target representations for masked autoencoders. arXiv preprint arXiv:2209.03917 , 2022. 2

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021. 16

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 891-898, 2014. 3

T Nathan Mundhenk, Daniel Ho, and Barry Y Chen. Improvements to context based self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9339-9348, 2018. 2

Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems (NeurIPS) , 34:23296-23308, 2021. 1, 3

Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision (ECCV) , pages 69-84, 2016. 2, 3

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018. 2

Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2536-2544, 2016. 1

Yixuan Qiu, Feng Lin, Weitong Chen, and Miao Xu. Pre-training in medical data: A survey. Machine Intelligence Research , pages 1-33, 2023. 1

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV) , 115:211-252, 2015. 2, 3, 6, 8, 10, 15

Sepehr Sameni, Simon Jenni, and Paolo Favaro. Representation learning by detecting incorrect location embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2023. 3

Rodrigo Santa Cruz, Basura Fernando, Anoop Cherian, and Stephen Gould. Deeppermnet: Visual permutation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3949-3957, 2017. 2

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS) , 30, 2017. 2, 3, 6, 15

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning (ICML) , pages 1096-1103, 2008. 1

Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Liwei Wu, Yuxi Wang, and Zhaoxiang Zhang. Pulling target to source: A new perspective on domain adaptive semantic segmentation. arXiv preprint arXiv:2305.13752 , 2023. 3

Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, and Zhaoxiang Zhang. Hard patches mining for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10375-10385, 2023. 2, 3, 6, 8, 15, 16

Haochen Wang, Yuchao Wang, Yujun Shen, Junsong Fan, Yuxi Wang, and Zhaoxiang Zhang. Using unreliable pseudo-labels for label-efficient semantic segmentation. arXiv preprint arXiv:2306.02314 , 2023. 3

Haoqing Wang, Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhi-Hong Deng, and Kai Han. Masked image modeling with local multi-scale reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 2, 3, 8, 16

Yuchao Wang, Jingjing Fei, Haochen Wang, Wei Li, Liwei Wu, Rui Zhao, and Yujun Shen. Balancing logit variation for long-tail semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19561-19573, 2023. 3

Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4248-4257, 2022. 2, 3

Ji-Rong Wen, Zi Huang, and Hanwang Zhang. Editorial for special issue on large-scale pre-training: Data, models, and fine-tuning. Machine Intelligence Research , 20(2):145-146, 2023. 1

Xin Wen, Bingchen Zhao, Anlin Zheng, Xiangyu Zhang, and Xiaojuan Qi. Self-supervised visual representation learning with semantic grouping. Advances in Neural Information Processing Systems (NeurIPS) , 2022. 2

Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https: //github.com/facebookresearch/detectron2 , 2019. 6

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3733-3742, 2018. 1, 2

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In European Conference on Computer Vision (ECCV) , pages 418-434, 2018. 6, 8, 9, 16

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9653-9663, 2022. 3, 8

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6023-6032, 2019. 15

Sukmin Yun, Hankook Lee, Jaehyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8354-8363, 2022. 2

Shuangfei Zhai, Navdeep Jaitly, Jason Ramapuram, Dan Busbridge, Tatiana Likhomanenko, Joseph Y Cheng, Walter Talbott, Chen Huang, Hanlin Goh, and Joshua M Susskind. Position prediction as an effective pretraining strategy. In International Conference on Machine Learning (ICML) , pages 2601026027, 2022. 1, 3

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. 2018. 15

Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Conference on Computer Vision (ECCV) , pages 649-666, 2016. 1, 2

Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1058-1067, 2017. 1

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 633-641, 2017. 2, 3, 6, 8, 9, 16

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Representations (ICLR) , 2022. 3, 6, 8, 15

Supplementary Material

In this supplementary material, we first provide mode implementation details for reproducibility in Sec. A. Next, we provide more experiments in Sec. B. Finally, in Sec. C, we evaluate the performance of the position reconstruction task using pre-trained models under different settings, and we provide more evidence to support the proposed three difficulties in Sec. 1.

A Implementation details

ViT architecture. We follow the standard vanilla ViT [21] architecture used in MAE [28] as the backbone, which is a stack of Transformer blocks [51]. Following MAE [28], we use the fixed 2D sine-cosine positional embeddings during pre-training. For the downstream classification task, we use features globally averaged from the encoder output for both end-to-end fine-tuning.

Effective training epochs. Following iBOT [72], we take the effective training epochs as the metric of the training schedule, due to extra computation costs brought by the multi-crop [5] augmentation, which is a widely used technique for contrastive methods. Specifically, the effective training epochs are defined as the actual pre-training epochs multiplied with a scaling factor r . For instance, DINO [6] is trained with 2 global 224 × 224 crops and 10 local 96 × 96 crops, and thus r = 2+(96 / 224) 2 × 10 ≈ 4 . More details and examples can be found in [72].

A.1 ImageNet classification

For all experiments in this paper, we take ImageNet-1K [48], which contains 1.3M images for 1K categories, as the pre-trained dataset. By default, we take ViT-B/16 [21] as the backbone and it is pre-trained 200 epochs followed by 100 epochs of end-to-end fine-tuning. Implementation details can be found in the following table. Most of the configurations are borrowed from MAE [28]. The linear learning rate scaling rule is adopted: lr = lr base × batch _ size / 256 . For supervised training from scratch, we simply follow the fine-tuning setting without another tuning. For ViT-B/16, pre-training and fine-tuning are conducted with 64 and 32 2080Ti GPUs, respectively. For ViT-L/16, pre-training and fine-tuning are conducted with 32 and 16 Tesla V100 GPUs, respectively.

A.2 COCO object detection and segmentation

We take Mask R-CNN [30] with FPN [37] as the object detector. Following [28] and [54], to obtain pyramid feature maps for matching the requirements of FPN [37], whose feature maps are all with a stride of 16, we equally divide the backbone into 4 subsets, each consisting of a last global-window block and several local-window blocks otherwise, and then apply convolutions to get the intermediate feature maps at different scales (stride 4, 8, 16, or 32).

We perform end-to-end fine-tuning on COCO [38] for 1 × schedule with 1024 × 1024 resolution, where 88,750 iterations of training with a batch size of 16 are performed. We simply follow the configuration of ViTDet [36], where the learning rate is 3e-4 and decays at the 78,889-th and 85,463-th iteration by a factor of 10. Experiments are conducted on 8 Tesla V100 GPUs.

A.3 ADE20k semantic segmentation

We take UperNet [63] as the segmentation decoder following the code of [3, 14, 54]. Fine-tuning on ADE20k [71] for 80k iterations is performed. Specifically, each iteration consists of 16 images with 512 × 512 resolution. The AdamW optimizer is adopted with an initial learning rate of 7e-4 and a weight decay of 0.05 with ViT-B. We apply a polynomial learning rate schedule with the first warmup of 1500 iterations following common practice [3,14,54]. When fine-tuning using backbones pre-trained with different methods, we search for the optimal learning rate or simply follow their official implementation for a fair comparison. Specifically, the learning rate is 1e-4 for [13,28,34], 4e-4 for [19, 56], respectively. All experiments are conducted on 8 Tesla V100 GPUs.

B More Experiments

The initialization of the positional encoding. DropPos uses fixed 2D sin-cos position embeddings by default. We ablate the initialization of position embeddings in Tab. S1 and it demonstrates that fixed sin-cos position embeddings achieve the best performance.

DropPos with Swin Transformers [40]. To verify the scaling property and the generalization of DropPos, we provide experiments when DropPos is equipped with the Swin Transformer. We follow the implementation of UM-MAE [35] and pre-train a Swin-Tiny [40] from scratch using DropPos. All models are pre-trained with 200 epochs and fine-tuned with 100 epochs, following the configuration of UM-MAE [35]. From Tab. S2, we can conclude that DropPos still works on Swin Transformers [40], and thus enhancing the location awareness of vision transformers is still worth studying.

C Performance of position reconstruction

In this section, we evaluate the performance of the position reconstruction task using pre-trained models under different settings. Specifically, we vary γ ∈ { 0 , 0 . 25 , 0 . 5 , 0 . 75 } and γ pos ∈ { 0 . 25 , 0 . 5 , 0 . 75 , 0 . 95 } when measuring the position prediction accuracy. We report performance under different evaluation settings as well as the averaged accuracy among 16 different cases. From Tabs. S3 to S5, we find evidence to support the three difficulties for designing an appropriate positionrelated pretext task introduced in Sec. 1: (i) discrepancies between pre-training and fine-tuning, (ii) failing to learn highly semantic representations by solving this simple position reconstruction task, and (iii) difficult to decide which patch positions to reconstruct precisely.

We study the effectiveness of different values of γ during pre-training in Tab. S3. Interestingly, we find evidence for failing to learn highly semantic representations by solving this simple position reconstruction task. As illustrated by Tab. S3a, the pre-trained model performs extremely well when we set γ = 0 for evaluation but fails to keep this trend when we enlarge γ . This indicates that given

the strength of ViTs in modeling long-range dependencies, they have easily solved this task in a superficial way, and thus pre-training with γ = 0 becomes trivial for ViTs. To this end, an appropriate γ is necessary to increase the difficulty of the pretext task and avoid trivial solutions.

We study the effectiveness of different values of γ pos during pre-training in Tab. S4, and we find evidence for discrepancies between pre-training and fine-tuning. As shown by Tab. S4d, the model fails to reconstruct accurate positions given some visible anchors. This is because the model has never been exposed to any positional embeddings (PEs) during pre-training. Therefore, providing some anchors is necessary to address discrepancies. Also, it may help the model focus on modeling relative relationships instead of simply reconstructing absolute positions.

We study the effectiveness of different values of γ pos during pre-training in Tab. S4, and we find evidence for hard to decide which patch positions to reconstruct precisely. As shown by Tabs. S5a and S5f, the model achieves higher position prediction accuracy but performs worse on downstream tasks (please refer to Tabs. 3 and 4 for downstream performances). Therefore, to prevent being overwhelmed by this particular position reconstruction task, techniques for relaxing the patch-wise classification problem become necessary, i.e. , position smoothing, and attentive reconstruction.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel self-supervised pretext task for Vision Transformers (ViTs), called DropPos. It aims to enhance the spatial reasoning or location awareness of ViTs, based on the observation that ViTs are often insensitive to the order of input tokens. DropPos works by dropping a large random subset of positional embeddings, then using the model to predict the actual position of each patch based solely on its visual appearance.\n\nThe paper identifies three major difficulties: (a) discrepancies between pre-training and fine-tuning (b) trivial solutions that fail to learn highly semantic representations by solving this simple task (c) patches with similar visual content. To prevent trivial solutions and increase task difficulty, this paper keeps only a subset of patches visible during the task. Given the potential similarity in visual appearances between different patches, the authors propose position smoothing and attentive reconstruction strategies. This relaxation allows for non-exact position reconstruction when exact positions are not critical. Quantitative evaluations demonstrate the effectiveness of DropPos, outperforming supervised pre-training and yielding competitive results against state-of-the-art self-supervised alternatives on various benchmarks.\n\nSTRENGTHS:\n1. The main claim is attractive. It would be quite interesting (and a bit counter-intuition) to see the vision transformers can learn a very good representation by such a simple patch predicting task, which is very coarse-grained. Particularly, the results of this paper look competitive to the mainstream pre-training tasks.\n\n2. This work provides extensive ablation studies to verify their design and to reveal some insights.\n\n3. It proposes Position smoothing and Attentive reconstruction to solve the problems like patches may share similar visual appearance.\n\nWEAKNESSES:\n1. The paper requires additional experiments and deeper analysis to substantiate some assertions.\n\nFor example, this paper starts from \"Vision Transformers (ViTs) are quite insensitive to the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident.\". \n\nAlthough this claim fits my intuition, it is weird that after citing some papers [13, 39, 60], the paper does not talk about this claim any more in the following sections. What would happen if messing up the order of input tokens? Or even further, should this really be viewed as a drawback? Insensitiveness to the order may also be a good property. The authors touch on this in the related work, but this claim needs more attention since it's the main drive of the paper.\n\nAlso, it's unclear if DropPos helps this issue. More specifically, if a model is pre-trained by DropPos and then finetuned in ImageNet, would it be more sensitive to the order of input tokens? Looking into this would help us understand if the improved performance comes from the model being more \"sensitive to the order of input tokens\".\n\n2. A recent work: Jigsaw-ViT: Learning jigsaw puzzles in vision transformer. \n\nAnother work Jigsaw-ViT proposes to include solving Jigsaw in the training of ViT, which is close to the task of this work. It is beneficial to include this for comparison.\n\n[1] Chen, Yingyi, Xi Shen, Yahui Liu, Qinghua Tao, and Johan AK Suykens. \"Jigsaw-ViT: Learning jigsaw puzzles in vision transformer.\" Pattern Recognition Letters 166 (2023): 53-60.\n\nQUESTIONS:\nI am not really sure so I put this in Questions instead of Weaknesses. After checking the main paper and supplementary, it seems the authors did not mention if the pre-training is conducted with DropPos only, or together with other pre-training tasks like in MAE. Could the authors answer whether DropPos is the only training objective?\n\nIf the answer is yes, the reviewer would hope the authors can discuss a bit about why such a very coarse-grained task can be more powerful than the counterparts. Does it mean the dense visual cues are not important in pre-training?\n\n\nOverall I am feeling this paper is a good trial, although it may need more in-depth analysis and appropriate discussion about relevant works.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to self-supervised representation learning for vision transformers, focusing on enhancing their positional awareness. The authors proposed a new pretext task called DropPos, which involves reconstructing the positions of dropped tokens in partial observations. By leveraging DropPos, which is a spiritual adaptation of MAE, the authors consistently achieve superior results compared to the state-of-the-art (SOTA) techniques in standard downstream tasks, including image classification, object detection, and semantic segmentation.\n\nSTRENGTHS:\nThe paper effectively addresses the problem of incorporating inductive bias into vision transformers, and it takes an interesting approach by employing self-supervised learning (SSL) techniques. Enhancing the positional awareness of vision transformers is a significant aspect, as it can greatly improve their performance in downstream tasks that rely on location awareness and positional information. The proposed pretext task, DropPos, is simple yet effective, which aligns with the requirements of SSL. Additionally, such pretext tasks are efficient as they involve masking a substantial portion of the input data. The paper highlights that using tasks like DropPos eliminates the need for careful selection of target representation and mask strategy, as typically performed in mask image modeling. Furthermore, the paper is well-written and easy to follow.\n\nWEAKNESSES:\nOne potential weakness of the paper lies in its experimental evaluation. While the proposed pretext task is commended for its simplicity and effectiveness, it would have been valuable to compare the performance of a DropPos pretrained Vision Transformer (ViT) with a hierarchical transformer such as Swin, pretrained using mask image modeling [A, B]. This comparison would have provided insights into whether a carefully designed ViT architecture already addresses the need for positional awareness, rendering the pretext task redundant.\n\nFurthermore, the paper lacks an in-depth analysis of what the ViT is actually learning and how it achieves positional awareness. Quantifying the extent of positional-aware representation learning is crucial. Analysis such as intertoken distance within a layer, sparsity of attention weights, and linear probe results are missing, which would have shed light on the underlying reasons for the success of the proposed pretext task.\n\nA significant concern arises in the evaluation section where the authors reproduce most of the state-of-the-art (SOTA) results reported in other papers. However, the reproduced numbers are generally lower than the original reported results, without any mention of the differences. It would be important for the authors to investigate the causes of these discrepancies, considering factors such as differences in the experimental setup (e.g., number of GPUs, changes in batch size and learning schedules) or potential missing engineering tricks. Particularly concerning is the lack of significant improvement (wrt MAE) in object detection and segmentation results, which are crucial for spatial modeling.\n\nLastly, the authors missed referencing relevant literature on contrastive learning approaches designed for spatial modeling, such as [C]. In fact, [C] outperforms DropPos when trained for 200 epochs on the ADE20k dataset.\n\n[A] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai and Han Hu. SimMIM: A Simple Framework for Masked Image Modeling\n[B] Xiang Li, Wenhai Wang, Lingfeng Yang, Jian Yang. Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality.\n[C] Patch-level Representation Learning for Self-supervised Vision Transformers, CVPR 2022\n\nQUESTIONS:\n1. Why do we need to compute Affinity as indicated in equation (9). Why can't we use the self-attention matrix (softmax(KQ'))?\n2. In Table 3, it will be also interesting to check the results of 2->1.\n3. What happens when ViT is pretrained with DropPos for longer epochs like 1600 epochs?\n4. In fig 3, how do we know that which predictions are correct? \nFor rebuttal, please also refer to the points mentioned in the weakness section.\n\nLIMITATIONS:\nThe paper acknowledges a limitation in its conclusion, which the reviewer agrees with. Although the experiments conducted with ViT-B are deemed sufficient to demonstrate the potential of the proposed method, it is crucial to provide a comprehensive explanation and understanding of the model, regardless of its size. This ensures a robust analysis and comprehension of the proposed approach beyond the specific ViT-B architecture.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper presents a simple yet effective approach for generative self-supervised representation learning on images, namely DropPos.  The proposed approach drops a large random subset of positional embeddings for visible tokens and classifies the actual position for these tokens via visual appearance. Experimentally, DropPos outperforms state-of-the-art self-supervised approaches on a wide range of downstream benchmarks including image classification, detection and segmentation.

STRENGTHS:
- The paper is well written and organized.
- The idea in this paper is simple yet effective, which brings something new in generative self-supervised representation learning.
- The authors provide clear implementation details (e.g., Pseudo-Code), which makes it easier to be reproduced.
- Detailed ablation study is conducted to verify the effectiveness of the proposed approach.
- Achieving SOTA performance on various downstream tasks.

WEAKNESSES:
- As illustrated in (4), the cross-entropy loss is applied for dropped position supervision. Except for this loss, is additional loss used? e.g., the MSE reconstruction loss used in MAE.
- In MAE, more layers are used in the decoder (i.e., 8). Here the decoder only consists of 2 layers. Is it because the dropped position classification task is easier than the patch reconstruction task?
- What’s the overall training time for DropPos? Is it comparable or more efficient than existing self-supervised approaches?
- For completeness, please add more dropout-guided work for discussion, e.g., DropMAE: Masked Autoencoders with Spatial-Attention Dropout for Tracking Tasks (CVPR23), which similarly employs dropout mechanism, but for spatial-attention dropout in videos.

QUESTIONS:
No. See Weaknesses.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DropPos, a self-supervised pretext task designed to enhance the location awareness of Vision Transformers (ViTs). By dropping positional embeddings and reconstructing the positions of visible patches with some auxiliary strategies, DropPos improves spatial reasoning abilities in ViTs. Experimental results demonstrate the efficacy of DropPos, outperforming supervised pre-training and achieving competitive performance against state-of-the-art self-supervised methods. The paper provides good insights on enhancing location awareness in ViTs for future works.\n\nSTRENGTHS:\n1.\tThe motivation is clear. The paper addresses the motivation of enhancing positional awareness and spatial reasoning abilities in vision transformers for pre-training.\n\n2.\tThe method is simple. The DropPos focuses on reconstructing dropped positions with some heuristics to avoid trivial solutions and ambiguities. Compare with the contrastive learning, it does not need complicated augmentations.\n\n3.\tThe method is effective. Compared to contrastive learning or masked image modeling, the proposed DropPos exhibits faster pretraining and slightly improved performance on several benchmarks.\n\nWEAKNESSES:\n1.\tThe initialization of the positional encoding for the DropPos is not discussed, and there is no analysis of the impact of the different PE initialization strategies for the proposed DropPos method.\n2.\tThe experiments are insufficient to validate the motivation. Although preliminary evidence is provided by experiments on downstream tasks such as detection and segmentation, which indicates that the DropPos enhances the location-awareness of ViT, the paper lacks more in-depth and intuitive experimental analysis and discussion to verify the strengthening of position sensitivity of ViT by the DropPos method.\n3.\tThe description of the DropPos method is not specific and clear enough, including the method implementation, the flowchat of pseudo-code in section 3.2. \n4.\tThe scaling properties of the DropPos on ViT are not explored in depth. For example, compared with the MAE method, It is questionable whether the method remains effective on Vit-Huge compared to the MAE. Experiments on more advanced vision transformers, such as Swin-Transformer, are also encouraged to be conducted.\n5.\tCompared with the MAE method, there is a lack of studies on transfer learning using iNaturalists and Places, and the robustness evaluation on ImageNet.\n\nQUESTIONS:\n1.\tRegarding the DropPos method: Could you provide more specific details and clarity on the implementation of the DropPos method and the flowchart of the pseudo-code in Section 3.2? This would help readers better understand the proposed approach.\n2.\tStrengthening of the validation of motivation. Could you enhance the experimental analysis and discussion to provide more depth and intuition regarding the impact of DropPos on the position sensitivity of ViT? Can you consider conducting additional analyses or visualizations to better illustrate and explain the observed effects?\n3.\tPositional Encoding Initialization: It would be valuable to discuss the initialization strategy for positional encoding in the DropPos method. How was it initialized, and were different initialization strategies explored? Analyzing the impact of different positional encoding initialization strategies on the performance of DropPos would enhance the comprehensiveness of the study.\n4.\tScaling Properties and Generalization: Could you further investigate and discuss the scaling properties of the DropPos method on larger architectures, such as ViT-Huge? Is the method equally effective and robust on larger-scale models? Additionally, have you considered applying the DropPos method to other advanced vision transformers like Swin-Transformer and evaluating its performance?\n5.\tTransfer Learning and Robustness Evaluation: It is noted that there is a lack of studies on transfer learning using iNaturalists and Places datasets, as well as the robustness evaluation on ImageNet. I would like to see if the DropPos method remains effective on these benchmarks.\n6.\tThe Fig.~3 about the qualitative results of position reconstruction is somewhat confusing and requires a clearer explanation.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper a method for self-supervised representation learning. Given a ViT architecture,  the authors propose to predict the  absolute position of masked positional embedings at random. Although the general direction is not new, the authors pose it in a simple an interesting way, that achieves good performance in the downstream tasks.

STRENGTHS:
1) The proposed ides is simple yet effective, and is executed well.
2) The presented is presented properly and is easy to follow.
3) The authors evaluate their method properly on several downstream tasks, achieving acceptable performance.

WEAKNESSES:
The direction of obtaining a supervision signal from the patch positions in the ViT architecture has been explored before. The authors mention the majority of them in their related work section but do not discuss what the advantage of their proposed method is. The performance supports the effectiveness of their method, but they could discuss more explicitly their advantages and ablate on that. Moreover, a comparison with [1] would be informative.

[1] Sameni et al., Representation Learning by Detecting Incorrect Location Embeddings, In AAAI 2022.

QUESTIONS:
Please see above.

LIMITATIONS:
Yes, they have.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11