SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haochen Wang $^{1,3}$  Junsong Fan $^{1,4}$  Yuxi Wang $^{1,4}$  Kaiyou Song $^{2}$  Tong Wang $^{2}$  Zhaoxiang Zhang $^{1,3,4*}$
$^{1}$ Center for Research on Intelligent Perception and Computing (CRIPAC), State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA)
$^{2}$ Megvii Technology  $^{3}$ University of Chinese Academy of Sciences (UCAS)  $^{4}$ Centre for Artificial Intelligence and Robotics, HKISI_CAS
{wanghaochen2022, junsong.fan, zhaoxiang.zhang}@ia.ac.cn  
yuxiwang93@gmail.com {songkaiyou, wangtong}@megvii.com
As it is empirically observed that Vision Transformers (ViTs) are quite insensitive to the order of input tokens, the need for an appropriate self-supervised pretext task that enhances the location awareness of ViTs is becoming evident. To address this, we present DropPos, a novel pretext task designed to reconstruct Dropped Positions. The formulation of DropPos is simple: we first drop a large random subset of positional embeddings and then the model classifies the actual position for each non-overlapping patch among all possible positions solely based on their visual appearance. To avoid trivial solutions, we increase the difficulty of this task by keeping only a subset of patches visible. Additionally, considering there may be different patches with similar visual appearances, we propose position smoothing and attentive reconstruction strategies to relax this classification problem, since it is not necessary to reconstruct their exact positions in these cases. Empirical evaluations of DropPos show strong capabilities. DropPos outperforms supervised pre-training and achieves competitive results compared with state-of-the-art self-supervised alternatives on a wide range of downstream benchmarks. This suggests that explicitly encouraging spatial reasoning abilities, as DropPos does, indeed contributes to the improved location awareness of ViTs. The code is publicly available at https://github.com/Haochen-Wang409/DropPos.
Learning extensible visual representations without any human annotations, known as self-supervised learning (SSL), has become a research hotspot in the field of computer vision [7,9,26,27,29,47,59,62] since it enables efficient transfer to various downstream benchmarks. To achieve this goal, researchers carefully design visual pretext tasks to produce appropriate supervision using only images [3, 18, 46, 52, 62, 69, 70]. Two popular approaches among these are contrastive learning (CL) [62] and masked image modeling (MIM) [3], both of which have demonstrated promising scaling behavior of vision models, particularly Vision Transformers (ViTs) [21]. Despite the success of these methods, ViTs are found to be relatively insensitive to the order of input tokens [13, 43, 67], leading to the hypothesis that they tend to model the relationship between a set of unordered input tokens. Therefore, a natural question arises: beyond the current CL and MIM paradigms, whether a pretext task that explicitly enhances the positional awareness of ViTs can further improve their representation learning abilities?
(a) Contrastive learning.
(b) Masked image modeling.
Figure 1: Comparison between different pretext tasks. (a) Contrastive learning aims to maximize the agreement between different views of one image. (b) Masked image modeling predicts specific contents of masked patches. (c) DropPos reconstructs positions solely based on visual appearance.
(c) Dropped positions reconstructing.
To answer this question, we begin by revisiting the forward procedure of ViTs. A sequence of positional embeddings (PEs) [51] is added to patch embeddings to preserve position information. Intuitively, simply discarding these PEs and requesting the model to reconstruct the position for each patch naturally becomes a qualified location-aware pretext task. By forcing the model only to utilize visual appearance for position reconstruction, the model is supposed to learn the shape relationships and layouts of image contents, thereby improving the ability to encode spatial visual content. However, despite the simplicity and intuitiveness of this idea, position-related methods such as [18, 44] have fallen far behind. We identify several difficulties in designing this type of paradigm:
(i) Discarding all PEs brings discrepancies between pre-training and fine-tuning because the model has never been exposed to any PEs during pre-training. (ii) Strengths of ViTs in modeling long-range dependencies may cause them to solve the task superficially, leading to trivial solutions that fail to learn highly semantic representations by solving this simple task. (iii) Patches with similar visual appearances may result in confusing reconstruction targets, making it crucial to decide which position to reconstruct precisely.
Driven by this analysis, we present a novel, straightforward, and highly effective pretext task for self-supervised visual representation learning: reconstructing Dropped Positions (DropPos). DropPos tackles the above issues systematically: (i) To address discrepancies, simply dropping a large subset of PEs instead of discarding all of them works effectively. (ii) To avoid trivial solutions, we only keep a subset of patches visible during pre-training, forcing ViTs to reconstruct the position of each visible patch with only partial inputs. (iii) To prevent ViTs from being overwhelmed by this particular task, we propose to relax this problem using position smoothing and attentive reconstruction strategies.
DropPos can be easily formulated into a simple patch-wise classification problem. Here is how it works: First, input patches are randomly masked and a large subset of PEs corresponding to these visible patches is dropped. Then, ViTs are tasked with classifying the positions of visible patches among all possible candidates, relying on partial observations and a few anchors (i.e., patches with unmasked PEs). Fig. 1 illustrates two benefits of DropPos: (i) requiring less prior knowledge of data augmentation techniques compared to CL, and (ii) eliminating the need for a careful selection of target representation [39] and mask strategy [8,54,56] as MIM does.
We conducted extensive experiments to evaluate the performance of DropPos in comparison to state-of-the-art alternatives across various downstream tasks. With only 800 epochs of pre-training, DropPos achieves  $84.2\%$  top-1 accuracy on ImageNet-1K [48] validation set using ViT-B/16 [21], outperforming MAE [28] pre-trained with 1600 epochs by  $+0.6\%$ . Moreover, on COCO [38] object detection/segmentation, and ADE20k [71] semantic segmentation benchmarks, which requires more abilities of spatial reasoning, DropPos surpasses MAE [28] consistently.
Self-supervised learning. The key challenge of self-supervised learning is the design of pretext tasks [18, 28, 32, 42, 44, 45, 50, 54, 69], i.e., how to produce appropriate supervision signals using only images. Among them, contrastive learning [5, 9, 26, 29, 58, 60, 62, 66] has been popular recently, which seeks to maximize agreement between different views of the same images. Nevertheless, the
performance of contrastive-related methods highly depends on carefully-designed data augmentation techniques [5,6,9,11,12,72]. Another mainstream is masked image modeling [1,3,10,28,54,56], which involves predicting specific contents masked patches [2,3,19,20,28,54,64,72]. Unlike these methods, DropPos offers a novel alternative for self-supervised ViTs: reconstructing dropped positions based on only visual clues. In this way, the model is urged to learn the spatial arrangement of local image patches, contributing to better localization abilities, which is important for spatial reasoning recognition tasks, e.g., segmentation [22,53,55,57,58] and video understanding [17,25].
Position prediction in self-supervised learning. Doersch et al. [18] first split an image into  $3 \times 3$  grids and train a network to predict the relative position of paired patches from the same image. This problem is formulated as an 8-way classification. Noroozi and Favaro [44] extended this approach to solve "jigsaw puzzles" by urging the model to predict the order of shuffled non-overlapping crops. These approaches were originally developed using ConvNets [33] and thus lacked the ability to learn long-range dependencies, which is crucial for position prediction. Very recently, Zhai et al. [67] revisited this type of pretext task in the scope of Transformers [51] by pre-training ViTs [21] to predict patch positions solely based on their visual appearance, while discarding positional embeddings. However, these pre-trained models have never been exposed to positional embeddings, leading to discrepancies between pre-training and fine-tuning, and thus their performances are significantly inferior to state-of-the-art alternatives [6, 13, 28, 54]. Sameni et al. [49] come up with an auxiliary position-related objective and combine it with the popular contrastive learning paradigm. Caron et al. [4] extended this idea by predicting the relative location of a query (i.e., local crop) to the corresponding reference (i.e., global crop). However, [4] only examined the effectiveness on segmentation benchmarks [15, 24, 41, 71]. In contrast, DropPos incorporates a novel dropped position reconstruction objective and thus achieves competitive performances on a variety of downstream tasks, including classification [48], detection [38], and segmentation [38, 71].
Positional embeddings in Vision Transformers. In the standard forward procedure of ViTs [21], learnable positional embeddings (PEs) are added with patch embeddings. Improving PEs and introducing more inductive bias into ViTs has become an active research area [23]. This is because some research shows that ViTs are surprisingly robust against permutations of the order of input patches. For instance, Chen et al. [13] demonstrate that the model performs well in classification even with no position embedding, suggesting that ViTs have not fully leveraged the positional information. Additionally, Naseer et al. [43] shows that ViTs are much less susceptible to patch shuffling perturbations than ConvNets. Therefore, we propose DropPos, a novel self-supervised pretext task that explicitly makes ViTs location-aware and promotes the emergence of spatial features.
Preliminary: Vision Transformers. Given an image  $\mathbf{I} \in \mathbb{R}^{H \times W \times C}$ , it is first reshaped into a sequence of patches  $\mathbf{I}_p \in \mathbb{R}^{N \times (P^2C)}$ , where  $(H, W)$  indicates the spatial resolution,  $C$  is the number of channels,  $P$  is the patch size, and  $N = HW / P^2$  is the number of patches. A linear projection is then applied to  $\mathbf{I}_p$ , mapping it to  $D$  dimensions to get patch embeddings  $\mathbf{x} \in \mathbb{R}^{N \times D}$ . Also, a [CLS] token  $\mathbf{x}_{\mathrm{cls}} \in \mathbb{R}^D$  is used to aggregate the information. Position embeddings  $\mathbf{p} \in \mathbb{R}^{(N+1) \times D}$  are added to the patch embeddings to retain positional information. Then,  $\mathbf{z} = [\mathbf{x}_{\mathrm{cls}}; \mathbf{x}] \oplus \mathbf{p}$  is the input of transformer blocks [51] which consists of a stack of multi-head self-attention mechanisms, where  $\oplus$  denotes element-wise plus. In particular,  $\mathbf{p}_0$  denotes the position embeddings of the [CLS] token, which will never be dropped.
3.1 DropPos
Fig. 2 illustrates our proposed DropPos. We first randomly generate a binary patch mask  $\mathbf{M} \in \{0,1\}^N$ , where 1 means visible and 0 means masked, respectively. Note that the [CLS] token, i.e.,  $\mathbf{z}_0$ , is always visible in our setting, and thus the length of  $\mathbf{M}$  is  $N$  instead of  $N + 1$  for simplicity. Given a pre-defined mask ratio  $\gamma \in (0,1)$ ,  $\mathbf{M}$  is supposed to subject to
$$
\sum_ {i = 0} ^ {N - 1} \mathbf {M} _ {i} = (1 - \gamma) N, \tag {1}
$$
which means there are  $\gamma N$  masked patches in total. It is worth noticing that we use patch masks to increase the difficulty of the pretext task, which is quite different from building a pretext task to
Figure 2: Illustration of DropPos. We first mask a large random subset of input images. Then, positional embeddings of visible patches are randomly dropped and [MASK] tokens are introduced. A lightweight projector is adopted to reconstruct those dropped positions. A simple patch-wise classification objective is adopted and gray position targets do not contribute to training.
recover input images in MIM [3, 28]. Then, we denote  $\mathbf{x}_{\mathrm{vis}} = \mathrm{gather}(\mathbf{x},\mathbf{M})\in \mathbb{R}^{(1 - \gamma)N\times D}$  to be the visible tokens, and position embeddings (PEs) for these unmasked tokens are randomly dropped, where  $\mathrm{gather}(\cdot ,\cdot)$  means only those unmasked patches (i.e.,  $\mathbf{M}_i = 1$ ) are reserved. To address possible discrepancies, we aim to drop a subset of PEs of these visible patches instead of simply discarding all of them. Thus, we randomly generate a binary positional mask  $\mathbf{M}_{\mathrm{pos}}\in \{0,1\}^{(1 - \gamma)N}$ . Given a pre-defined position mask ratio  $\gamma_{\mathrm{pos}}\in (0,1)$ ,  $\mathbf{M}_{\mathrm{pos}}$  is supposed to subject to
$$
\sum_ {i = 0} ^ {(1 - \gamma) N - 1} \mathbf {M} _ {\text {p o s}} ^ {i} = (1 - \gamma_ {\text {p o s}}) (1 - \gamma) N, \tag {2}
$$
which indicates that the model is supposed to reconstruct  $\gamma_{\mathrm{pos}}(1 - \gamma)N$  dropped positions based on remaining  $(1 - \gamma_{\mathrm{pos}})(1 - \gamma)N$  anchor patches and the [CLS] token. PEs for visible patches  $\mathbf{p}_{\mathrm{vis}} = \mathrm{gather}(\mathbf{p}_{1:N},\mathbf{M})$  are gathered first. Then, we aim to obtain the final PEs  $\mathbf{p}' \in \mathbb{R}^{(1 - \gamma)N \times D}$  by replacing those dropped positions with a learnable [MASK] token:
$$
\mathbf {p} _ {i} ^ {\prime} = \left\{ \begin{array}{l l} \mathbf {p} _ {0}, & \text {i f} i = 0, \\ \mathbf {p} _ {\text {v i s}} ^ {i - 1}, & \text {i f} \mathbf {M} _ {\text {p o s}} ^ {i - 1} = 1, \\ \mathbf {p} _ {\text {m a s k}}, & \text {o t h e r w i s e ,} \end{array} \right. \tag {3}
$$
where  $i = 0,1,\ldots ,(1 - \gamma)N$  is the patch index, and  $\mathbf{p}_{\mathrm{mask}}\in \mathbb{R}^D$  is the [MASK] token. Finally, the input of the Transformer encoder is  $\mathbf{z}' = [\mathbf{x}_{\mathrm{cls}};\mathbf{x}_{\mathrm{vis}}]\oplus \mathbf{p}'\in \mathbb{R}^{[(1 - \gamma)N + 1]\times D}$ .
Implementation. Pre-training ViTs using our DropPos can be implemented efficiently and requires few specialized operations. The process involves three stages prior to forwarding features to the encoder. First, we randomly mask a subset of embedded patch tokens. Following MAE [28], the list of tokens is shuffled randomly, and the last portion of the list is removed. Next, positional embeddings of those unmasked patches are randomly dropped using the same masking operation. Then, positional mask tokens are introduced to obtain the final positional embeddings, which are added to the patch tokens finally and encoded
Algorithm 1 Pseudo-Code of DropPos.
# x: embedded patch tokens
# gamma/gamma_pos: patch/position mask ratio
# mask_token: learnable [MASK] token
# patch masking, [CLS] token is kept visible
ids.vis, _ = masking(x, gamma)
x.vis = gather(x, ids.vis)
# position dropping for visible patches
pos_embedding = gather(pos_embedding, ids.vis)
ids_keep, ids_res = masking(pos_embedding, gamma_pos)
pos_embedding = gather(pos_embedding, ids_keep)
# concat with mask tokens
pos_embedding = cat([pos_embedding, mask_token])
# restore
pos_embedding = gather(pos_embedding, ids_res)
# add to patch tokens for forward pass
return x.vis + pos_embedding
by ViTs followed by a lightweight decoder. A simple classification objective is applied to those unmasked patches introduced as follows.
3.2 Pre-training DropPos
DropPos is pre-trained by reconstructing dropped positions. Concretely, as illustrated in Fig. 2 and presented in Sec. 3.1, an ViT [21]  $f_{\theta}$  parameterized by  $\theta$ , takes  $\mathbf{z}'$  as input. A lightweight decoder  $g_{\psi}$  parameterized by  $\psi$  then projects  $f_{\theta}(\mathbf{z}')$  to patch-wise position predictors  $\mathbf{o} = g_{\psi}(f_{\theta}(\mathbf{z}')) \in \mathbb{R}^{(1 - \gamma)N \times N}$ . We omit the [CLS] token here for simplicity. For each visible patch  $i \in \{0,1,\dots,(1 - \gamma)N - 1\}$ ,  $\text{softmax}(\mathbf{o}_i) \in \mathbb{R}^N$  represents the probability density function of the predicted positions. Next, we have to produce appropriate ground truths, i.e., the actual position for each visible patch. To this end, we gather positions via  $\mathbf{y} = \text{gather}([0,1,\dots,N - 1],\mathbf{M}) \in \mathbb{R}^{(1 - \gamma)N}$ , where vector  $[0,1,\dots,N - 1]$  is naturally the actual position for all patches before masking (w/o the [CLS] token). Overall, the objective is simply the vanilla cross-entropy loss given true positions  $\mathbf{y}_{ij}$ :
$$
\mathcal {L} = - \sum_ {i = 0} ^ {(1 - \gamma) N - 1} \sum_ {j = 0} ^ {N - 1} (1 - \mathbf {M} _ {\mathrm {p o s}} ^ {i}) \cdot \operatorname {o n e \_ h o t} (\mathbf {y} _ {i j}) \cdot \log \left[ \frac {\exp (\mathbf {o} _ {i j})}{\sum_ {k = 0} ^ {(1 - \gamma) N - 1} \exp (\mathbf {o} _ {i k})} \right]. \tag {4}
$$
Only patches with dropped positions, i.e., orange patches in Fig. 2, contribute to this objective, which is reflected by the term  $(1 - \mathbf{M}_{\mathrm{pos}}^i)$  in Eq. (4). Next, several techniques are proposed to prevent ViTs from being overwhelmed by this particular task, making them pay more attention to model spatial dependencies instead of simply reconstructing positions.
Position smoothing. Considering that different categories (i.e., positions) are not completely independent under this setting, the classification problem in Eq. (4) is relaxed when the model has predictions close to the actual position. Specifically, a weight matrix  $\mathbf{w} \in (0,1)^{N \times N}$  is defined to measure similarities between different patches:
$$
w (i, j) = \exp \left(- \frac {\operatorname {d i s t} (i , j)}{\sigma^ {2}}\right), \tag {5}
$$
where  $\mathrm{dist}(i,j)$  means the Euclidean distance between patch  $i$  and  $j$ , and  $\sigma$  is a hyper-parameter. The matrix then should be normalized by  $w^{*}(i,j) = w(i,j) / \left(\sum_{k=0}^{N-1} w(i,k)\right)$ . Here,  $\mathbf{w}_i^* \in \mathbb{R}^N$  indicates the smoothed ground truth when the actual position is  $i$ . Overall, the smoothed objective is:
$$
\mathcal {L} _ {\text {s m o o t h}} = - \sum_ {i = 0} ^ {(1 - \gamma) N - 1} \sum_ {j = 0} ^ {N - 1} (1 - \mathbf {M} _ {\text {p o s}} ^ {i}) \cdot w ^ {*} (\mathbf {y} _ {i j}, j) \cdot \log \left[ \frac {\exp (\mathbf {o} _ {i j})}{\sum_ {k = 0} ^ {(1 - \gamma) N - 1} \exp (\mathbf {o} _ {i k})} \right]. \tag {6}
$$
Furthermore, we relax the problem in Eq. (6) by setting a large  $\sigma$  at the early stage, and we gradually decay the value of  $\sigma$  to produce a challenging pretext task. A simple linearly decay is performed:
$$
\sigma_ {t} = \frac {t}{T} \left(\sigma_ {T} - \sigma_ {0}\right) + \sigma_ {0}, \tag {7}
$$
where  $t$  and  $T$  denote the iteration and the number of total steps, respectively.  $\sigma_0$  and  $\sigma_T$  are the initial value and final value of  $\sigma$ , respectively. We set  $\sigma_0 = 1$  and  $\sigma_T = 0$  by default.
Attentive reconstruction. Since there may be different patches that share similar visual appearance (e.g., two blue patches that represent the sky), it is not necessary to reconstruct their exact positions. Simply swapping these patches still maintains reasonable visual coherence. Therefore, we leverage the feature similarities between the [CLS] token and patch tokens to be an extra weight of Eq. (6):
$$
\mathcal {L} _ {\text {s m o o t h}} ^ {\text {a t t n}} = - \sum_ {i = 0} ^ {(1 - \gamma) N - 1} \sum_ {j = 0} ^ {N - 1} (1 - \mathbf {M} _ {\text {p o s}} ^ {i}) \cdot A _ {\mathbf {y} _ {i j}} \cdot w ^ {*} (\mathbf {y} _ {i j}, j) \cdot \log \left[ \frac {\exp (\mathbf {o} _ {i j})}{\sum_ {k = 0} ^ {(1 - \gamma) N - 1} \exp (\mathbf {o} _ {i k})} \right], \tag {8}
$$
where  $\mathbf{A} \in \mathbb{R}^N$  denotes the similarity matrix. Concretely,  $A_i$  means the affinity between the [CLS] token and patch token  $i$ . Let  $\mathbf{f}_{\mathrm{cls}} \in \mathbb{R}^D$  and  $\mathbf{f} \in \mathbb{R}^{N \times D}$  be the features output by the encoder  $f_\theta$ , and thus the affinity  $A_i$  is computed by
$$
\mathbf {A} _ {i} = \frac {\exp \left(\cos \left(\mathbf {f} _ {\mathrm {c l s}} , \mathbf {f} _ {i}\right) / \tau\right)}{\sum_ {j = 0} ^ {N - 1} \exp \left(\cos \left(\mathbf {f} _ {\mathrm {c l s}}, \mathbf {f} _ {j}\right) / \tau\right)}, \tag {9}
$$
where  $\tau$  indicates the temperature parameter and is set to 0.1 by default.
Table 1: Ablation study of patch mask ratio  $\gamma$ . Note that “ $\gamma = 0$ ” means that the whole image is visible. We report the averaged top-1 accuracy of position predictions.
γ	ImageNet-1K	COCO detection	COCO segmentation	ADE20k
fine-tune	position	APb	APb50	APb75	APm	APm50	APm75	mIoU	aAcc
0.00	81.94	59.79	34.10	52.80	37.17	31.24	50.27	33.03	31.66	75.86
0.25	82.02	79.62	37.05	56.23	40.18	33.60	53.51	35.84	32.80	76.92
0.50	82.78	87.27	38.45	57.83	41.87	34.88	55.14	37.50	36.33	78.25
0.75	82.96	87.83	42.14	61.99	46.38	37.93	59.23	40.76	40.68	80.14
0.90	82.81	65.12	41.06	60.92	44.57	37.04	58.23	39.62	40.30	80.09
Table 2: Ablation study of position patch mask ratio  $\gamma_{\mathrm{pos}}$ . Note that "  $\gamma_{\mathrm{pos}} = 1$  " means that we do not provide any reference patch, i.e., all visible patches are randomly shuffled. We report the averaged top-1 accuracy of position predictions.
γpos	ImageNet-1K	COCO detection	COCO segmentation	ADE20k
fine-tune	position	APb	APb50	APb75	APm	APm50	APm75	mIoU	aAcc
0.25	82.71	65.19	37.98	57.65	41.39	34.61	54.79	36.93	36.15	78.16
0.50	82.86	86.70	39.23	58.77	42.99	35.60	56.23	37.89	37.82	78.60
0.75	82.96	87.83	42.14	61.99	46.38	37.93	59.23	40.76	40.68	80.14
1.00	82.66	19.44	41.48	61.50	45.18	37.41	58.81	39.92	39.98	80.19
Pre-training. We perform self-supervised pre-training on the ImageNet-1K [48] training set with a resolution of  $224 \times 224$ . By default, we take ViT-B/16 [72] as the backbone and perform 200 epochs of pre-training. The decoder is a stack of Transformer [51] and has a depth of 2 and a width of 512. Patch mask ratio  $\gamma$  and position mask ratio  $\gamma_{\mathrm{pos}}$  are both 0.75 by default. Our implementation is based on HPM [54]. Details can be found in Supplementary Material.
Evaluation. We perform supervised training to evaluate our DropPos with end-to-end fine-tuning on ImageNet-1K [48] for classification. By default, 100 epochs of fine-tuning are performed following common practices [28, 54] for ViT-B/16 [21]. We report top-1 validation accuracy of a single  $224 \times 224$  crop. As for COCO [38], following previous methods [28, 54], we take Mask R-CNN [30] with FPN [37] as the detector. We perform end-to-end fine-tuning on COCO [38] for  $1 \times$  schedule with  $1024 \times 1024$  resolution. We report  $\mathrm{AP^b}$  for object detection and  $\mathrm{AP^m}$  for instance segmentation, respectively. Our implementation is based on detectron2 [61] and ViTDet [36]. For ADE20k [71], following previous methods [28, 54], we take UperNet [63] as the decoder and perform end-to-end fine-tuning on ADE20k [71] with  $512 \times 512$  resolution for 80k iterations. We take mIoU [24] as the main evaluation metric. Our implementation is based on mmsegmentation [14].
4.1 Ablation studies
In Tabs. 1 to 4, we take the ViT-B/16 [21] pre-trained with 200 epochs on ImageNet-1K [48] as the backbone. We highlight the default settings of our DropPos. Concretely, if not specified, the default setting is  $\gamma = 0.75$ ,  $\gamma_{\mathrm{pos}} = 0.75$ ,  $\sigma_0 = 1$ ,  $\sigma_T = 0$ , and  $\tau = 0.1$ . By default, 100 epochs of fine-tuning on ImageNet-1K [48],  $1 \times$  schedule fine-tuning on COCO [38], and 80k iterations fine-tuning on ADE20k [71] are performed.