Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
Authors: Kang-Jun Liu
Masanori Suganuma
Takayuki Okatani
Venue: ICML 2025 poster (2025)
Keywords: Self-supervised learning; Instance segmentation pre-training; Object detection pre-training; Vision transformer
Primary area: deep_learning->selfsupervised_learning

Abstract: We present a novel self-supervised feature learning method using Vision Transformers (ViT) as the backbone, specifically designed for object detection and instance segmentation. Our approach addresses the challenge of extracting features that capture both class and positional information, which are crucial for these tasks. The method introduces two key components: (1) a positional encoding tied to the cropping process in contrastive learning, which utilizes a novel vector field representation for positional embeddings; and (2) masking and prediction, similar to conventional Masked Image Modeling (MIM), applied in parallel to both content and positional embeddings of image patches. These components enable the effective learning of intertwined content and positional features. We evaluate our method against state-of-the-art approaches, pre-training on ImageNet-1K and fine-tuning on downstream tasks. Our method outperforms the state-of-the-art SSL methods on the COCO object detection benchmark, achieving significant improvements with fewer pre-training epochs. These results suggest that better integration of positional information into self-supervised learning can improve performance on the dense prediction tasks.

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
Kang-Jun Liu'
Masanori Suganuma
Takayuki Okatani 12
Abstract
et al., 2021) for image feature extraction have advanced significantly. These approaches enable feature extraction from unlabeled images and, with larger training datasets, have improved performance in various downstream tasks (Deng et al., 2009; Geiger et al., 2012; Lin et al., 2014; Wu et al., 2015; Zhou et al., 2017).
We present a novel self-supervised feature learn- ing method using Vision Transformers (ViT) as the backbone, specifically designed for object detection and instance segmentation. Our ap- proach addresses the challenge of extracting fea- tures that capture both class and positional infor- mation, which are crucial for these tasks. The method introduces two key components: (1) po- sitional encoding tied to the cropping process in contrastive learning, which utilizes novel vec- tor field representation for positional embeddings; and (2) masking and prediction, similar to conven- tional Masked Image Modeling (MIM), applied in parallel to both content and positional embed- dings of image patches. These components en- able the effective learning of intertwined content and positional features. We evaluate our method against ofate-of-thheart approaches, pre-training ImageNet-1K K and fine-tuning downstream tasks. Our method outperforms the statc-of-the- art SSL methods on the COCO object detection benchmark, achieving significant improvements with fewer pre-training epochs. These results suggest that better integration of positional infor- mation into self-supervised learning can improve performance on the dense prediction tasks. Our code is available at https: //tl:///iitti tit.bb...bu...o.ggoooooo.... com/ KJ rc/ wwc/InteetwiinedSe.
While early methods did not restrict the scope of down- stream tasks, targeting a broad range from image classifica- tion to dense prediction tasks such as semantic segmenta- tion, recent years have seen a shift toward developing SSL methods tailored to specific downstream tasks. This shift stems from the recognition that the features required for image classification, which depends on global image-level representations, differ significantly from those necessary for dense prediction tasks, where pixel-level or patch-level features play a crucial role.
This study builds on recent research trends, focusing specif- ically on object detection (OD) and instance segmentation (IS) as downstream tasks. These tasks require the precise identification of individual object instances within an im- age, which necessitates extracting appropriate features from localized regions such as pixels, patches, or subregions. The core focus of this research is on effectively integrat- ing both the content information of these local regions and their positional information within the image into feature representations, as these are arguably crucial for OD and IS tasks.
Among SSL methods, contrastive learning is an early and foundational approach that continues to be widely adopted. In this method, two random crops are taken from a single image, augmented with random transformations, and the feature representations are trained to be similar in the feature space. This approach can be interpreted as focusing on learn- ing position-invarian features for the entire image. Subse- quently, pixel-level contrastive learning was introduced to better suit dense prediction tasks. Besides using crops, these methods pair individual pixels and train their features to be similar. A key challenge in this approach lies in determining which pixels to pair. To address this, various strategies have been proposed, including methods based on proximity in feature space (Li al.. 2022a; Su al., 2024), methods relying geometric positional correspondence (Yun et al.,
1. Introduction
In recent years, self-supervised learning (SSL) methods (Chen et al., 2020; He al., 2020; Caron et al., 2020; Grill al., 2020; Caron et al., 2021: Zhontar et al., 2021; Ermolov
'Graduate School of Information Sciences. Tohoku University, Miyagi, Japan FRIKEN Center for AIP, Tokyo, Japan. Correspon- dence to: Takayuki Okatani <okatani vision.is. tow...ss..ooou..........
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
Conventional
Ours
0,0
0,1
1.3
0.1, 0.1
0.1, 0.4
0.2, 0.5
0.2, 0.9
2.1
2,2
2,3
3.2
0.3, 0.1
0.3, 0.4 0.6, 0.5
0.6, 0.9
1,0
Figure 1. Proposed position encoding, Conventional methods always apply the same positional encoding to randomly cropped views. In contrast, the proposed method directly applies the positional encoding defined on the original image the cropped views.
2022; Lebailly & Tuytelaars, 2023), and hybrid approaches that balance these two criteria (Bardes et al., 2022; Lebailly al., 2024; Stegmüller al., 2023).
on a regular grid, yielding set of position embeddings of the patches. They are subsequently combined with the content embeddings of their corresponding image patches.
Another line of research focuses on Masked Image Model- ing (MIM) (Bao et al., 2022; He et al., 2022; Zhou et al., 2022a), a distinctly different approach. This method, de- signed to use with Vision Transformers (ViT) (Dosovitskiy al., 2021), partitions an image into patches and randomly masking subset of them. The model is trained to recon- struct the masked patches based on the context provided by the unmasked ones. Masked patches as reconstruction targets can be pixels as in MAE (He et al., 2022) or patch- level latent features as in iBOT (Zhou et al., 2022a). MIM is regarded as an effective way to learn feature representations that seamlessly combine both the content information and positional information of each patch.
The second component is that, unlike previous MIM (Bao et al., 2022; Peng et al., 2022; He et al., 2022), which apply masking only to the image content embeddings of patches, our method also applies it to position embeddings. In our approach, after the input image is patchified into set of patches before being fed into the ViT, masking and pre- diction are performed independently on both their content embeddings and the position embeddings. The underly- ing expectation is that by predicting the masked positional information from the remaining positional and content infor- mation, and vice versa, the model can extract features that intertwine both the image content and positional informa- tion. It is important to note that this specialized treatment of positional embeddings is applied only during training, allowing the standard positional embedding method of ViT to be seamlessly employed during inference.
In this paper, we propose novel SSL method specifically designed for OD and IS as downstream tasks, building on the contrastive learning framework. These tasks demand both content information and positional information to ac- curately classify object instances while distinguishing them from one another. Our method focuses on extracting feature representations that seamlessly integrate these two types of information. Although our motivation is similar to cent works such as DropPos (Wang et al., 2023) and LOCA (Caron et al., 2024), our approach introduces two novel components that fundamentally redefine how positional in- formation is utilized during training.
We experimentally compare the proposed method with ex- isting ofath-of-the-art approaches on the COCO detection dataset (Lin et al., 2014) in the standard setting, i.e., pre- training on ImageNet-1K (Deng et al., 2009) and fine-tuning COCO. The results show that the proposed method achieves significant performance improvements in the down- stream tasks of OD and IS, demonstrating the effectiveness of our approach.
The first the use of positional encoding tied the cropping process in contrastive learning; see Figure 1. In conven- tional SSL (Caron et al., 2021: Chen et al., 2021; Caron et al., 2024), the position encoding is not aligned with the crop- ping, meaning the same position embeddings are applied whether processing the full image or cropped sub-image. We propose representing positional encoding as a vector field with the same dimensions as the input image, which is then cropped in the same manner the image and sampled
2. Preliminaries
Contrastive self-supervised learning In SSL., a feature extraction model is trained on a pretext task, and a key is in the design of the task (Doersch et al., 2015; Zhang et al., 2016; Gidaris et al., 2018; Oord et al., 2018; Vincent et al., 2008). Contrastive methods (Oord et al., 2018: Chen et al.,
2

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
2020; He et al., 2020)' have proven particularly effective, maximizing the similarity of representations from different views of the same image-created through random crops and other diverse image augmentations. These methods ensure that the learned representations are invariant to these augmentations, and they mostly focus on learning image- level representations.
as the following loss:
N
CPatch
Po (ū: + Pt), Phatch (u P:) log phatch
m
(2)
where Ppacch (+) and ppatch (-) share the same network ar- P
chitecture and 0 is updated by gradient descent while O' is updated by the exponential moving average of 0. MIM has demonstrated strong performance in dense prediction tasks.
Self-distillation Self-distillation (Caron et al., 2021) is widely used approach in contrastive SSL. It transfers knowl- edge from a teacher model to a student model, with both models being updated simultaneously. The student is up- dated via gradient descent, while the teacher updated as a momentum copy of the student. Both models, parameterized by A and 6' respectively, share the same network architec- ture. Given an image x and its two views, and v, which are randomly cropped from and undergo random aug- mentations, knowledge is distilled through a cross-entropy loss:
Combining Eq. (1) and Eq. (2), the losses of iBOT (Zhou et al., 2022a) and DINOv2 (Oquab et al., 2024) can be summarized as follows:
_DINOw2= Cimage + Ceatch + oo KoLeo,
(3)
iBOT
where [KoL.on is defined as in (Sablayrolles et al., 2019) to increase feature diversity. These methods have demon- strated strong performance in both classification and dense prediction tasks.
Cimage = PPo |c (v)² log Pa ((CTs) (u),
(1)
3. Proposed Method
3.1. Outline
where P)CLS(() and PiCLSI (-) project [CLS] tokens in the VIT's input to a probability distribution over K dimen- sions. This can also be interpreted as an assignment to K learnable prototypes, with the student learning these assign- ments from the teacher.
The objective is to develop a SSL method tailored for ob- ject detection (OD) and instance segmentation (IS). Models pre-trained using this method are expected to efficiently extract the necessary features for these downstream tasks and deliver high accuracy with minimal fine-tuning, Fol- lowing recent studies on SSL methods, we adopt ViT as the backbone for image feature extraction.
iBOT (Zhou et al., 2022a) and DINOv2 (Oquab et al., 2024) Masked image modeling (MIM) (Bao et al., 2022; He et al., 2022) is an SSL method based on a different principle. Using ViT as the backbone, it divides the in- put image u into N patches and applies a linear mapping to each, resulting in N vectors [u.]/ MIM randomly masks subset of these vectors and trains a model to pre- diet the masked vectors from the unmasked ones. Specif- ically, let {m, E {0, Ji=1 be a random mask sampled according to ratio p E [0, 1]. A special token e (MASK) is introduced, replacing the patch embedding vector uy with a; = (1 (1-m;)- U, + m, e ([MSs] for = I,...,... NN. Posi- tion embeddings (p: ) are then added to â to obtain the integrated embeddings {û, + pt N. iBOT (Zhou et al., 2022a) and DINOv2 (Oquab et al., 2024) implement MIM within a self-distillation framework. The student model p ppatch (-) and the teacher model (.pach(() project the in- tegrated embeddings into probability distribution of K" dimensions. The patch-level self- liff-iitillation is formulated
As discussed in Section 1, OD and IS require isolating object instances in an image, making it essential have fea- ture representations that effectively integrate both content and positional information. To address this, the proposed method builds on existing approaches while introducing mechanisms to improve the treatment of positional informa- tion in feature learning. Specifically, two key components are introduced. The first is a positional encoding method that aligns with cropping in image-level contrastive (or joint- embedding) techniques. The second is patch-level mask- ing approach that masks not only image content but also positional information, incorporating both into the predic- tion target. Each of these components is detailed below.
3.2. Position Encoding Linked with Image Cropping
1.2.1. BASIC METHOD
In conventional contrastive methods, two cropped regions from the input image are treated as if they were complete, independent images, and positional embeddings are applied as such (Figure 1). Consequently, the positional and size information of the cropped regions within the original image
'The term "contrastive methods" in its narrow sense refers to methods that use both positive and negative samples. However, for simplicity in this paper, we also refer 'non-contrastive methods, which use only positive pairs, as contrastive.
3

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
Crop
(uask)
(tcca]
image
Image
Student
Teacher
Resize & shift
Add
Aoottem
Elac
Student
Teacher
Figure 2. Illustration of the proposed method. (a) The proposed method defines positional encoding directly on the input image. The position embedding for each view is determined in relation to its crop. Additionally, we introduce a virtual larger image, where the input image is assumed to occupy random scale and position. Positional embeddings are then calculated within this virtual image. The content embeddings of the image are obtained using standard methods from the original image and are added to the corresponding position embeddings. (b) Using these position embeddings, our method performs image-level feature alignment between views (C (Cimage) and independently executes masking and prediction for both position embeddings and content embeddings (CPos Coontent).
completely discarded, which can hinder the accurate iden- tification of object instances. To overcome this limitation, we propose a method that crops the positional information embedded in the original image alongside its RGB content.
Subsequently, Pa and pa sampled on regular grid corre- sponding the patches, and their values are combined with and u, The resulting inputs to the Vision Transformer (ViT) are {w + Pu,i and {uy Po, where p.,, and Po,i represent the sampled positional vectors for each patch.
The details are as follows. ViT divides the input image x into fixed number (N) of patches, which are then embedded via linear transformation into sequence of vectors {arg } The position of each patch within the image is encoded by positional vector pa which is added to the patch embedding I; to form I; + Py. The resulting sequence {z; + ps) is then fed into the input layer of ViT.
In practice, P represented as a 2D array of size w consisting of d-dimensional vectors, i.e., P Rexixd The size h are independent of the input image size and are treated hyperparameters. Due to the arbitrary nature of the crop operations tu and t., we interpolate the array P to obtain { and {p.,))
Conventional contrastive methods (Chen et al., 2021) extract two cropped views, u (x( and v to (2), from an input image x, and process each view independently in the same way. Specifically, is divided into patches and embedded as J which are combined with the fixed positional encodings {p:).. resulting in {u: + p. The same process is applied to v, yielding {1r + p Thus, while the cropped views u and v can vary in positions and sizes, the positional encodings p; remain unchanged independently of t, and f...
3.2.2. POSITION AND SCALE AUGMENTATION
By adopting the above approach, becomes possible to integrate the position and size of cropped views within the image into the feature representation. However, this method raises two concerns. First, while the approach encodes absolute positional information, the identification of object instances should typically rely only on the relative positional information between the two views; see Figure 1. Learning absolute positions directly and becoming overly dependent on them may lead to unintended consequences.
The proposed method introduces alternative approach to positional encoding. Let P denote smooth vector field, with its four corners aligned to those of the input image xr While x provides an RGB vector for each pixel, P encodes the spatial location of each pixel as a vector. Similar to conventional methods, two views, u (x) and e = tu (2), are cropped from x and divided into patches, yielding {u, and fo, )N 1- The same cropping transformations, t, and lup are then applied to P, generating cropped vector (P).
Second, there is notable difference in the spatial distri- bution and size of objects between ImageNet (Deng et al., 2009) images and those used in OD/IS tasks (e.g., COCO). In ImageNet, objects generally occupy a large, centralized region of the image, whereas in COCO, multiple objects of varying sizes appear across different parts of the image. Ne- glecting this distinction could result in performance issues.
To address these issues, we apply data augmentation over Pu, and p.. by randomly shifting and scaling them together
fields P.. = (P) and

---

Self-Supervieed Learning of Intertwined Content and Positional Features for Object Detection
within P; see Figure 2(a). Specifically, let A represent the smallest bounding rectangle that encloses Pu and Pe: as shown in Figure 2(a). A coordinate transformation ta consisting of scaling and translation, is applied to A. The scaling factor is /s/|AI, where |A] denotes the area of the region. The parameter s is randomly sampled from a Beta distribution, as will be discussed in Section 4.3, The trans- lation (displacement) is randomly sampled from a uniform distribution, ensuring that the new cropped regions remain within the hypothesized input image. With the application of l,, the positional encodings undergo combined transfor- mations, with the cropped fields becoming Pa. < (P)) and Pv f to (P)))
details, refer to Section 4.3.
We retain the original content masking and prediction from JIM--pecificallly: the selected vector u4 + Pu,i is modi- fied to C[MASKj + the position masking and prediction described above; see Figure 2(b) 3_ Thus, masking and prediction are applied symmetrically to both content and position codes. As il- lustrated in Figure 2(b), this process is applied only to the student side (i.e., only for the view u) in the teacher-student framework, similar to the hybrid models introduced after iBOT (Zhou et al., 2022a).
it is performed independently of
In summary, the position and content masking and predic- tion are implemented through the following loss functions:
This position and scale augmentation is expected to reduce excessive reliance on absolute positional information within the image and alleviate biases associated with the object scale distribution in ImageNet.
N
C
phos (u: + pu.;)) log Pppe(
u, + m2 e(posmAsK] + (1- mm Pu)),
(4)
3.3. Masked Position Prediction
The introduction of the above positional encoding method broadens the range of options for designing 'pretext tasks' (i.e.. loss functions) during training. In particular, we pro- pose utilizing positional information as a target for Masked Image Modeling (MIM) (Bao et al., 2022; Zhou et al., 2022a; He et al., 2022). Specifically, we extend the approach em- ployed in methods like iBOT and DINOv2 is tra- ditionally applied exclusively content embeddings to include positional embeddings. This extension aims hance feature representation learning, with a particular focus on fostering more effective integration of content and po- sitional information.
grontent
Σ
B peontent (u: + pu,;) TT log PongPaomm Prontent
1=1
m C[AAKK] + (1 - (1-mm)p mmmm") )u, + Pass,, (5)
where m. E {0, 1) and m E (0,, 13. are the sampled masks with a mask ratio p E [0,
It is worth noting that some existing methods also incorpo- rate position prediction; however, they predict precise po- sitions using location indicators (Wang al., 2023; Caron et al., 2024) or at the pixel level (He et al., 2022). In con- trast, our method predicts positional information within the feature (embedding) space, representing a fundamentally different approach.
The details are as follows. Recall that in our method, the in- put vector sequence is represented as {w + Pu where the positional encoding depends on the cropping of the view u. Our masked position prediction works as follows; as in MIM, vectors from the input sequence {u: + P.N. are randomly selected, and the position codes of the selected vectors are masked Specifically, if u; + Pu., is selected, it is modified as u; + [|PosmAs]] using a newly introduced special token [[PosMASK]- The resulting masked sequence is then fed into the VIT. Through preliminary experiments, we observed that when masking position embeddings, selecting patches in cross-shaped pattern produces better results compared to the box-wise selection used in iBOT and DINOv2 for Masked Image Modeling (MIM) with content embeddings. In those methods, patches are randomly selected in rectan- gular boxes, and all patches within each box are masked until the designated mask ratio p is achieved?. For further
3.4. Adaptation to DINOv2
While the proposed method can be adapted to other SSL methods, we focus on integrating it with DINOv2 due to its popularity and performance. The modification to the loss function is straightforward: we add the position masking and prediction loss from Eq. (4) to the original DINOv2 loss as follows:
DINOv2
Cours Cinaage Coontent + +XKot.oo C KoLeo + +CPss (6)
iBOT
where CkoLeo (Sablayrolles et al., 2019) enhances the diver- sity of image-level representations, and its weight AKoteo is set to 1.
We follow previous studies (Bao et al., 2022: Zhou et al., 2022a; Oquab et al., 2024) in determining p for both masked content and positional predictions, where is randomly selected from the range (0.1, 0.5).
'In our implementation. for each image in a batch, we randomly applied either content or position masking and prediction. each with 50% probability. These applications are mutually exclusive, meaning both types of masking are never applied the same image simultancously.
5

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
applied to both global and local crops, while masked con- tent/position prediction (Section 3.3) is applied only the global crops. Our training setup is largely based on DINOv2 (Oquab et al., 2024), with several modifications detailed in Appendix B.1. For baseline comparisons, we use a ViT-B backbone, while ablation studies are performed using a ViT- backbone. The weights for the comparison methods are sourced from their public repositories, except for DINOv2. Since DINOv2 was pre-trained on much larger dataset of 142M samples (Oquab et al., 2024), we conduct its training on ImageNet- K, referring to it DINOv2
Table COCO object detection instance segmentation and ADE20K semantic segmentation We report the results using both ViT-B/16 and VIT-S/16 backbones. The pre-trained weights for other methods are sourced from their official repositories, except for DINOv2), which is our reproduction on ImageNet-1K IK. The highest score is highlighted in bold, while the second-hige highest is underlined.
COCO
ADE20K mloU
Method
Elf. Ep.
Apoox
ApMask
VTT-Smal/116
DINO (2021)
3200
42.0
38.0
42.9
iBOT (2022a)
3200
43.8
39.1
44.8
Evaluation on COCO and ADE20K We evaluate the transferability of the features learned by our method on object detection and instance segmentation tasks using the COCO dataset (Lin et al., 2014). We also report perfor- mance on ADE20K (Zhou et al., 2017), in line with recent SSL studies (Locatello et al., 2020; Wang al., 2023). We fine-tune the pre-trained models on COCO and ADE20K as follows. For the COCO dataset, we follow the evalua- tion methodology from DropPos (Wang et al., 2023), using ViTDet (Li et al., 2022b) as our detection framework while removing window attention and relative position encodings from the backbone. For the ADE20K dataset, we adhere to the evaluation protocol from LOCA (Caron et al., 2024), using the linear decoder approach from Segmenter (Strudel et al., 2021), which utilizes minimal number of adapter layers, Additional implementation details are provided in Appendix B.
Mugs (2022b)
3200
41.3
37.2
45.3
CrIBo (2024)
1600
42.6
38.3
44.9
SelfPatch (2022)
1050
40.4
36.7
42.5
FLSL
700
45.5
40.5
41.4
CrOC (2023)
600
40.2
36.2
43.4
LOCA (2024)
40.1
36.0
44.8
DINOv2 (2024)
350
41.9
37.7
44.7
Ours
350
44.8
39.8
44.8
VTT-Base/16
DINO (2021)
1600
45.5
40.8
44.7
MAE (2022)
1600
48.1
43.2
46.2
iBOT (2022a)
1600
47.6
42.4
47.7
Mugs (2022b)
1600
47.0
42.0
47.7
DropPos (2023)
800
47.0
42.2
45.3
CriBo (2024)
800
45.4
40.5
45.6
LOCA (2024)
600
48.3
43.0
48.5
DINOv2 (2024)
47.7
42.4
47.5
Ours
350
49.2
43.8
48.4
Compared methods We compare our method with state- of-the-art SSL approaches. For general-purpor purpose SSL meth- ods, we include DINO (Caron et al., 2021), MAE (He et al., 2022), iBOT (Zhou et al., 2022a), and DINOv2 (Oquab et al., 2024). For SSL methods specifically designed for dense prediction tasks, we evaluate Mugs (Zhou al., 2022b), DropPos (Wang al., 2023), CrIBo (Lebailly et al., 2024), SelfPatch (Yun et al., 2022), FLSL (Su et al., 2024), and CrOC (Stegmüller et al.. 2023). Explanations for these methods are provided Section 5. For all methods, we use the implementations and pre- pre-trained weights available from their official repositories. Depending on the availability of configurations, we report results for only one of the two backbones (ViT-B/16 ViT-S/16) for some methods.
From an implementation perspective, only minor extensions the existing DINOv2 code are necessary. Specifically, we compute the additional loss mentioned above. In DINOv2, content masking and prediction were originally applied to 50% of the images in each batch. We now compute the addi- tional position loss for the remaining images. Additionally, the position augmentation described in Section 3.2.2 can be seamlessly integrated into DINOv2's 's data augmentation pipeline, without requiring any further modifications.
4. Experiments
4.1. Experimental Configuration
Pre-training on ImageNet-1K We pre-train our model on ImageNet- K (Deng et al., 2009) using the Adam W (Loshchilov, 2017) optimizer for 100 epochs. The multi- crop augmentation employed following the DINO series (Caron et al., 2021; Zhou et al., 2022a; Oquab al., 2024), specifically utilizing global crops and eight local crops. The proposed position encoding method (Section 3.2.2) is
4.2. Main Results
Object detection on COCO In Table I, we compare the performance of various SSL methods on the COCO dataset using ViT-S/16 and ViT-B/16 as the backbone.
With one exception, our method outperforms others, includ- ing ODD/S-specifit SSL methods. The exception is FLSL (Su et al., 2024) with the ViT-S backbone. Although the performance of FL.SI. consistent with the results reported
FFoloowing iBOT (Zhou al., 2022a), we caleulate the effec- tive number of epochs for each method.
6

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
Table 2. Effectiveness of position em- bedding sampling and masked posi- tion prediction (Pos
Table 3. Effects of hyper-parameters with the proposed position encoding method
Const
Beta.
(a) Distributions of scaling factor 8
(b) Resolutions of
Pos. Sampling
[ ApBux
A ApMask
Dist.
Ap
ApMask
Pos. Size
ApBox
ApMask
0.5
41.3
37.2
Const.
43.0
38.6
19 19
44.4
39.7
43.2
38.6
Uniform
43.7
39.0
50 50
44.8
39.8
Figure 3. Distributions of scaling factor s
43.7
39.1
Beta(2, 5)
44.8
39,8
/
/
44.8
39.8
settings of our method, we systematically ablate each com- ponent or hyperparameter. In all tables, the default configu- ration is highlighted with a blue background.
in the original paper, we have identified several issues in its official code
The comparison with other methods is as follows. Our method outperforms all the compared methods in this set- ting, including OD/IS- SSL methods. While Drop- Pos shows some improvement over its base model, DINO, it remains inferior to certain general-purpose methods. We hypothesize that this is due to its high position mask ratio of 94%, which likely hinders the learning of complex vi- sual patterns. With a ViT-B/16 backbone, LOCA performs only on par with the general-purpose MAE, whereas our method surpasses both by +0.9 A and +0.8 APMask In- terestingly, DINOv2t achieves performance comparable to iBOT while requiring only about one-fourth of the effective training epochs, likely benefiting from its broader design exploration.
Effectiveness of individual components Table 2 shows an ablation study of the two proposed components: the po- sition encoding method and the masked position prediction. Individually, they contribute improvements of +1.9 ApBox and +2.4 A Aploox, respectively. When combined, these modi- fications further boost performance by at least +1.1 Ap These results underscore the effectiveness of each com- ponent and indicate their complementary contributions to overall performance.
Scaling factor 8 As described in Section 3.2.2, the pro- posed positional encoding method simulates feature extrac- tion from small objects within an image by applying random scaling and translation transformations to the crop of the po- sition encoding field. The scaling factor 8 is sampled from a beta distribution. This approach aligns with the statistical distribution of object sizes in the input images during object detection. To evaluate the impact of different s distributions, we tested three scenarios: uniform distribution in the range [0.2, 1.0]. and Beta 5) as shown in Figure 3. Note that = 1.0 indicates no scaling, meaning the object scale from the original ImageNet image is preserved. The results, shown in Table 3a, indicate that accuracy improves progressively with s 1.0 ("Const') ) uniform, and beta distributions, in that order, verifying the proposed sampling method.
Semantic segmentation on ADE20K In Table we also report the performance of SSL methods on ADE20K. We observe that the perroffffning methods, including iBOT, Mugs, DINOv21, LOCA, and ours, show similar results, with our method not demonstrating significant improve- ment in semantic segmentation. This could be because semantic segmentation is primarily a pixel-level classifica- tion task. Notably, MAE and DropPos perform worse, likely due to the absence of augmentation invariance in contrastive learning, which may be critical for classification tasks. It is also worth mentioning that the performance of the compared methods is higher than previously reported in LOCA (Caron et al., 2024), likely due to the use of the AdamW optimizer and learning rate scheduler, as recommended by (Lebailly et al., 2024).
1.0 (referred to 'Const.'), )
Resolution of position encoding field As described in Section 3.2.2, the position-encoding vectors for each patch (c.g., Pu.i) are obtained by sampling at regular grid points with interpolation from field P, represented as a tensor of size Wp x hp X d. We evaluated the impact of varying the spatial resolution Wp x hp. hyperparameter in this representation. The results, shown in Table 3b, indicate that the method is not overly sensitive to resolution as a hyperparameter.
4.3. Ablation Study
We then examine the design choices in our method by eval- uating object detection performance on the COCO dataset. We use ViT-S backbone here. Starting with the default
'ee attempted to reproduce the SSL pre-training part based on their code. However, we discovered several bugs in the official repo and were unable to reproduce it, as of this writing. httpps: : //gi thub. commoom//////m / SuMM.
Position mask sampling strategy Through a preliminary study, we found that a cross-wiss wise mask sampling strategy,

---

Self-Supervieed Learning of Intertwined Content and Positional Features for Object Detection
Table 4. Ablating different position mask settings
(a) Position masking
(b) Image ratio in a batch w/ position or content masking
Pos. Mask Apoox ApMisk
Content vs. Pos.
Ap8ox
ApMask
Box -iis
Cross wise
Box-wise 44.4
39.7
100% w/ Masked Cont. Pred.
43.4
38.9
Cross-wise 44.8
39.8
100% w/ Masked Pos. Pred.
21.7
20.7
Figure 4. Position masking: Box-wise vs. Cross-wise
50% each
44.8
39.8
as shown in Figure 4, works effectively for masked position prediction. Table 4a compares this approach with the pop- ular box-wise sampling strategy used for MIM in previous studies, showing that cross-wiss wise sampling performs slightly better. We retain the box-wise sampling scheme for masking content vectors.
long been known pretext tasks in SSL (Noroozi & Favaro, 2016), into the SSL methods mentioned above. One such method in this line of work is DropPos (Wang et al., 2023), an extension of MAE. In addition to the core principle of MAE, which masks and predicts the content embedding of image patches, DropPos introduces a task where the posi- tional embedding of the patches is 'dropped' and predicted. Specifically, 75% of the tokens (u, + p;) are removed, and MAE is applied to the remaining 25%. Among these, 75% (i.e., 18.75% of the total tokens) have their positional em- beddings p; dropped, leaving only us as input. The task is to predict the dropped p; this configuration. LOCA an SSL method that processes two views of the input image (a reference view and smaller query view with overlap) and aims predict the query view position in the reference view coordinate by a single cross-attention layer. Within the cross-attention layer, it has to identify the overlap parts and predict their positions correctly. In these work, position prediction is formulated as an 1D classification problem:
Content masking vs. position masking Our method applies either content masking and prediction or position masking and prediction to each image in a batch in mu- tually exclusive manner. The selection is random, with default ratio of 50:50. For sanity check, we also eval- uate the configurations of 100:0 and 0:100. The results, shown in Table 4b, indicate that the 50:50 mix achieves the best performance, while applying position masking and prediction alone leads to a significant performance drop. These findings confirm the effectiveness of combining both approaches.
5. Related Work
Cloc = one hot (i)" log ploc ploc(x;;;
(7)
Dense contrastive learning Dense contrastive learn- ing (O Pinheiro et al., 2020; Wang et al., 2021; Xie et al., 2021; Bardes et al., 2022) focuses on learning pixel-level representations rather than image-level ones (Oord et al., 2018; Chen et al., 2020; He et al., 2020), with the goal of im- proving performance dense prediction downstream tasks such segmentation and detection. The challenge in pixel- level SSL (Yun et al., 2022; Li et al., 2022a; Lebailly et al., 2024; Bardes et al., 2022; Stegmüller et al., 2023; Lebailly & Tuytelaars, 2023) is positive samples matching problem. Their approaches can be summarized as either similarity- based or position-based, or both. Although (Lebailly al., 2024; Stegmüller al., 2023) also track the positions of cropped views in the original image, they still use the con- ventional position encoding. Therefore, they do not predict relative positions between views and focus only on visual content.
where one_hot(i) is a location indicator, and ploc(.) projects patch tokens (or masked tokens in DropPos) to probability distribution over N dimensions.
6. Summary
We presented a novel self-supervised learning method that learns pre-trained weights optimized for object detection and instance segmentation. The method introduces two key components. The first is position encoding aligned with cropped views in contrastive learning setting. This is achieved using a position embedding field, where embed- ding vectors are sampled on a regular grid corresponding to the geometry of the cropped view in the input image. Combined with the proposed position encoding augmenta- tion, which can be seamlessly integrated into existing SSL data augmentation pipelines, this approach leads to signifi- cant improvements on the COCO benchmark compared to DINOv2 (reproduced by us on ImageNet-IK3 The second component is the simultaneous masking and prediction of position and content embeddings, further enhancing perfor- mance on the COCO benchmark. Our method also performs
Self-supervised learning with position prediction Re- cently, several studies (Wang et al., 2023; Caron et al., 2024), have aimed to improve performance on dense prediction tasks by incorporating position prediction tasks, which have
8

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
comparably to the state-of-the-art LOCA the ADE20K dataset, where LOCA is specially tuned for this task. We hope this study sheds light on the potential of positional encoding in contrastive learning, an area that remains under- explored in the research community.
Contributors, M. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https:
github.com//n ww..ab/mmmegeeettteettt 2020.
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024.
Acknowledgments
This work was partly supported by JST [Moon-shot Re- search and Development), Grant Number [JPMJMS2032] and JSPS KAKENHI Grant Number 20H05952 and 23H00482.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 148-255. Icee, 2009.
Impact Statement
Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on com- puter vision, pp. 1422 1430, 2015.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
References
Bao, H., Dong, L., Piao, S., and Wei, F. BEit: BERT pre-training of image transformers. In International Con- ference on Learning Representations, 2022.
Bardes, A., Ponce, J., and LeCun, Y. VICRegl: Self- supervised learning of local visual features. In Advances in Neural Information Processing Systems, 2022.
Ermolov, A., Siarohin, A., Sangineto, E., and Sebe, N.
Whitening for self self-supe representation learning. In Proceedings of the 38th International Conference Ma- chine Learning, pp. 3015-3024, 2021.
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, pp. 9912-9924, 2020.
Geiger, A., Lenz, P., and Urtasun, Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference computer vision and pattern recognition, pp. 3354-3361. IEEE, 2012.
Caron, M., Touvron, H.. Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, 2021.
Gidaris, S., Singh, P., and Komodakis, N. Unsupervised representation learning by predicting image rotations. In International Conference Learning Representations, 2018.
Caron, M., Houlsby, N., and Schmid, C. Location-await self-supervised transformers for semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 117-127, 2024.
Grill, J.-B., Strub, F,, Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo. Z. Gheshlaghi Azar, M., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M. Bootstrap your latent A new approach to self-supervised learning. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F, and Lin, H. (eds.), Advances in Neural Information Processing Sys- tems, pp. 21271-21284, 2020.
Chen, T.. Kornblith, S., Norouzi. M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. Proceedings of the 37th International Conference on Machine Leurning, pp. 1697- 2020.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729- 2020.
Chen, X., Xie, S., and He, K. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1640-9649, 2021.

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint 2018.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F,, El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S. Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski,
Lebailly, T. and Tuytelaars, T. Global-local delf-distillation for visual representation learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 1441-1450, 2023.
Lebailly, T., Stegmüller, T., Bozorgtabar, B., Thiran, J.-P., and Tuytelaars, T. CriBo: Self-supervised learning via cross-image object-level bootstrapping. In The Twelfth International Conference on Learning Representations. 2024.
P. DINOv2: Learning robust visual features without su- pervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Park, N., Kim. W., Heo, B., Kim, T., and Yun, S. What do self-supervised vision transformers lear?? In The Eleventh International Conference on Learning Repre- sentations, 2023.
Lefaudeux, B., Massa, F., Liskovich, D., Xiong, W., Caggiano, V., Naren, S., Xu, M., Hu, J., Tintore, M., Zhang. S., Labatut, P., Haziza, D., Wehrstedt, L., Reizen- stein, J., and Sizov, G. xformers: A modular and hack- able transformer modelling library. https: //ggthub.. com/ htt////aeboooeeeeeeeeee
Peng, Z., Dong, L.., Bao, H., Ye, Q. and Wei, F. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022.
xfommers, 2022.
Li, C., Yang, J., Zhang, P., Gao, M., Xiao, B., Dai, X., Yuan, L.,, and Gao, J. Efficient self-supervised vision transformers for representation learning. In International Conference on Learning Representations, 2022a.
Sablayrolles, A., Douze, M., Schmid, C., and Jégou, H. Spreading vectors for similarity search. In International Conference on Learning Representations, 2019.
Li, Y., Mao, H., Girshick, R., and He, Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pp. 280-296, 2022b.
Stegmüller, T., Lebailly, T., Bozorgtabar, B., Tuytelaars, T., and Thiran, J.-P. Croc: Cross-view online clustering for dense visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7000-7009, 2023.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. Microsoft coco: Common objects in context. Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Strehl, A. and Ghosh, J. Cluster ensembles knowledge reuse framework for combining multiple partitions. Jour- nal of machine learning research, 3(Dec):583-617, 2002.
Strudel, R., Garcia, R., Laptev, L., and Schmid, C. Seg-
menter: Transformer for semantic segmentation. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 7262-7272 2021.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf,
T. Object-centric learning with slot attention. Advances in neural information processing systems, 33:11 1525-11588 2020.
Su, Q., Netchaev, A., Li, H., and Ji, S. Fisl: feature-level self-supervised learning. Advances in Neural Information Processing Systems, 36, 2024.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-196100 1103, 2008.
Noroozi, M. and Favaro, Unsupervised learning visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69-84 84, 2016.
O Pinheiro, P. O., Almahairi, A., Benmalek, R., Golemo, F., and Courville, A. C. Unsupervised learning of dense visual representations. Advances in Neural Information Processing Systems, 33:4489 4500, 2020.
Wang, H., Fan, J., Wang, Y., Song, K., Wang, T., and Zhang, Z. Droppos: Pre-training vision transformers by recon- structing dropped positions. In Thirty- Conference on Neural Information Processing Systems, 2023.
10

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
Wang, X., Zhang. R., Shen, C., Kong, T., and Li, L. Dense contrastive learning for self-supervised visual pre- training. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3024-3033, 2021.
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang. I.., Tang. X., and Xiao, J. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912- 1920, 2015.
Xie, Z., Lin, Y., Zhang, Z., Cao, Y., Lin, S., and Hu, H. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16684-16693, 2021.
Yun, S., Lee, H., Kim, J., and Shin, J. Patch-level represen- tation learning for self-supervised vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8354-8363, June 2022.
Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In Proceedings of the 38th International Con- ference on Machine Learning, pp. 12310-12320, 2021.
Zhang, R., Isola, P.. and Efros, A. A. Colorful image col- orization. In Computer Vision-ECCV 2016: 14th Eu- ropean Conference, Amsterdam, The Netherlands, Octo- ber 11-14, 2016, Proceedings, Part III 14, pp. 649-666. Springer, 2016.
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., and Kong, T. Image BERT pre-training with online tokenizer. International Conference on Learning Rep- resentations, 2022a.
Zhou, P., Zhou, Y., Si, C., Yu, W., Ng, T. K., and Yan, S. Mugs: A multi-granular self-supervised learning frame- work. arXiv preprint urXiv:2203.14411 2022b.

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
0.35
0.30
0.30
IROT MAB DeuPhe
(145
(Normalized
0.25
NMI
0.20
0.35
0.15
DOSD
0.30
DENDx2
1801
KAB
6.08
0.5-
0.00
8
111
11
12
Atlention Block ID
Allention Block ID
Figure Statistical differences attention maps across layers ViT models pre-trained with the compared SSL methods. Input images sourced from the COCO dataset. Normalized mutual information (NMI) and mean attention distance (MAD) are used metrics, following Park et al. (2023). See the text for more details.
A. Analyses of Attention Maps in Pre-trained Models
Existing SSL methods, including ours, incorporate positional information into feature learning in various ways. Beyond evaluating performance on downstream tasks, we can analyze how attention to patches (tokesn) is spatially distributed within the layers of the ViT, providing insights into the differences between methods. We refer to this spatial distribution as "patch attention" here. In the following, previous experiments, and evaluate them. All experiments are conducted using VTT-B/16 backbone, with input images standardized to 480 x 480.
use pre-trained models from each method, prepared using the same approach as in the
A.1. Patch attention diversity
Following Park et al. (2023), we first examine the diversity of attention maps and the effective receptive field size. It should be noted that while Park et al.'s analysis uses ImageNet-IK images, we use COCO images as described above. There are two metrics involved (for details, refer to Park et al. (2023)): normalized mutual information (NMI) (Strehl & Ghosh, 2002) and mean average distance (MAD) (Dosovitskiy et al., 2021). .. NMI is the mutual information between a query token g and a attention from q to kt, and p(q) assumed to be uniformly distributed across the image. Intuitively, this measures the diversity of attention maps. MAD measures the average distance between patch positions within an image, weighted by attention, representing the effective receptive field size in the VIT. The two metrics computed by averaging the heads in the attention computation in each layer. The results are shown in Figure 5.
key token k, based on the joint distribution p(g, k) = T 4)p(4). where <( () is the
First, it is clear that both metrics vary significantly across methods and layers within the same method. Dividing the methods into four categories- aategoris--image-leeee learning (DINO), patch-level learning (MAE and DropPos), hybrid methods (iBOT and DINOv2), and methods incorporating positional learning (LOOA and our method) behavior is similar to what was reported Park et al. (2023). Specifically, higher patch attention diversity desirable, but it is smaller in image-level methods (DINO) and relatively larger in methods that incorporate patch-level learning. The three methods that showed strong performance COCO LOCA, MAE, and our method-ewe exhibit similar behavior in both metries, particularly in the last three layers.
A.2. Visualization of patch attention in the final layer
Figure visualizes the attention all patches (tokens) in the image in the final layer, where single point (i.e., patch) in the input image is selected as the query. The attention is averaged across all heads. In the leftmost column of the figure, the position of the query patch is indicated by a red dot in the input image. Across all methods, it appears that patches "close" to the query patch-though in different senses senses-receds -reeeive greater attention. However, the interpretation of "closeness" varies across methods. To be specific, MAE can look widely but be biased to textures or colors easily. not enough to identify the instance. Moreover, the query point is in the background but close to the boundary of foreground object, attention
12

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
Input
MAB
DropPos
LOCA
iBOT
Ours
SESSUIEER
STOP
Figure Patch attention maps with sampled reference points queries. We visualize patch attention maps from the final layer. using the sampled reference points (indicated in red) as queries.
focused more on the foreground than the background, as shown in the fourth row. DropPos produces an attention map too locally without recognizing wider patterns. LOCA exhibits cross-shaped artifacts in its attention map. iBOT displays more focused attention than the above methods, but it seems to attend to the entirety of objects in the image without instance discrimination. This is especially evident in the bottom example, where the distinction between trees and the traffic sign is unclear. On the other hand, in the attention maps produced by our method, when the query is on the foreground, the object instance indicated by the query is clearly delineated. When the query is in the background, the background regions can be highlighted more accurately without mixing with the foreground. This behavior suggests that our method most suitable for object detection and instance segmentation, demonstrating that the proposed approach effectively achieves its goal.
B. Implementation details
B.I. Pre-training on ImageNet-1K
We follow the implementation of DINOv2 (Oquab et al., 2024) and adopt some settings from iBOT (Zhou et al., 2022a) due to the significantly smaller scale of training data, i.e., from 142M dataset to ImageNet-IK. We use the same hyperparameters for both the ViT-B and ViT-S backbones, except for the number of GPUs: 8 and 4, respectively. The details are provided in Table
13

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
We pre-train ViT-S on single node with A6000 GPUs and ViT-B on nodes with the same setup. For training time comparison, compare method with DINOv2f in our implementation on single node. Our method requires 1.3x longer training per epoch.
Table 5. Implementation details pre-training on ImageNet-IK
Config
#Epochs
100
Optimizer
AdamW
Base learning rate
2e-3
Warmup (fteoochs)
10
Layerwise Ir decay Patch emb. Ir decay Weight decay (cosine)
False
False
0.04 0.4
Drop path rate (lincar)
[0, 0.11
Teacher temp.
0.04 const.
Teacher momentum init.
0.992
Patch mask prob.
[0.1, 0.5]
Pos mask prob.
[0.1, 0.5]
Patch mask
box-wise
Pos. mask
cross-wise
Patch/pos mask ratio
50% vs. 50%
Output dim. (all heads)
65,536
Separate heads
True
Norm. last layer
True Beta(2,5)
Dist. of
Smin- Smax ((P) =
0.2, 1.0 50x50
P
Total batch size
512
B.2. Object detection on COCO with fine-tuning.
Following DropPos (Wang et al., 2023), we adopt ViTDet (Li al., 2022b) as our object detection framework, fine-tuning the entire model on the COCO object detection benchmark with 22.128 iterations and a total batch size of 64 (12 epochs). We 3e-4 learning rate for ViT-B backbone and le-4 learning rate for ViT-S backbone. Both learning rate decay the 19,667-th and 21,306-th iterations by factor of 10. To preserve the integrity of the pre-trained weights, we remove relative position encodings and window attentions from ViTDet, ensuring that the backbone remains as close possible to its original pre-trained configuration. We also employ checkpointing and efficient attention kernels (Lefaudeux et al., 2022) to optimize GPU memory usage.
B.3. Semantic segmentation on ADE20K with fine-tuning
For the ADE20K dataset, we follow LOCA (Caron et al., 2024), adopting the linear decoder protocol in Segmenter (Strudel et al., 2021) and training for 127 epochs with a batch size of 16 (resulting in a total of 160k iterations). We consider the optimizer and learning rate settings from Lebailly et al. (2024), and employ the AdamW optimizer, and sweep the weight decay (wd) across { (le-2, Ie-4) and the learning rate (Ir) across (8e-5, 3e-5, lc-5, 8c-6) with a min_Ir of 0.1x We report results in single scale, averagd over 2 runs. This largely improves the performances reported in LOCA. The codebase is based on Segmenter (Strudel al., 2021) and built using MMSegmentation (Contributors, 2020).
C. Qualitative studies of position masking: Box-wise Cross-wise
We visualize that vertical line artifacts occur with the box-wise strategy but are absent with the cross-wise strategy; see Figure 7. Furthermore, the superiority of the cross-war wise strategy is quantitatively validated in Table 4a.
14

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
Input
Box-wise
Cross-wise
STOP
Figure 7. Artifact comparison in attention maps: Box-wise vs. Cross-wise. Vertical line artifacts occur with the box-wise strategy but are absent with the cross-wise strategy
D. Comparison with Official DINOv2 and DINOv2-reg Backbones
We augment the results in the main paper with evaluations using pre-trained weights from DINOv2 (Oquab et al., 2024) and DINOv2-reg (Darcet et al., 2024), which were pre-trained on the large-scale LVD-142M, 14-M dataset. We adopt them in our fine-tuning experiments on COCO and ADE20K, shown in Table 6. is important note that their VIT-S/B models patch size
distilled from larger pre-trained model, ViT-g, and refined with larger resolution fine-tuning. Since they of 14x 14, we interpolate the patchifier kernel from 14x 14 to 16x 16 before fine-tuning to ensure a fair comparison.
DINOv2-reg and DINOv2 achieve 52.2 ApBox on the ViT-B/16 backbone and a 47.4 ApBox on the ViT-S/16 backbone respectively, which are at least +2 APBox higher than the other methods. Interestingly, we find that DINOv2-reg significantly outperforms DINOv2 with ViT-B but not with ViT-S.
15

---

Self-Supervised Learning of Intertwined Content and Positional Features for Object Detection
Table 6. Augmented results with official DINOv2/-reg (Oquab et al., 2024; Darcet et al., 2024) backbones.
(a) VIT-B/16 backbone.
VIT-S/16 backbone.
COCO
ADE20K
COCO
ADE20K
Method
Eff. Ep.
Ap
ApM
mloU
Method
Eff. Ep.
Ap
ApMask
mloU
DINO
45.5
40.8
44.7
DINO
3200
42.0
MAE
1600
48.1
43.2
46.2
iBOT
3200
43.8
39.1
44.8
iBOT
1600
47.6
42.4
47.7
Mugs
3200
41.3
37.2
45.3
Mugs
1600
47.0
42.0
47.7
CrlBo
1600
42.6
38.3
44.9
DropPos
800
47.0
42.2
45.3
SelfPatch
1050
40.4
36.7
42.5
CriBo
45.4
40.5
CrOC
600
40.2
LOCA
600
48.3
43.0
48.5
LOCA
600
40.1
36.0
44.8
DINOv2
350
47.7
42,4
47.5
DINOv2
350
41.9
37.7
44.7
Ours
350
49.2
43.8
48.4
Ours
350
44.8
39.8
44.8
On LVD-142M
On LVD-142M
DINOv2
51.1
45.3
$2.5
DINOv2
47.4
42.2
49.7
DINOv2-reg
52.2
46.3
54.3
DINOv2-reg
46.5
16


Reviews:
[{"review_id": "NjKUMos3KW", "reviewer": "Reviewer_v6Kv", "summary": "This work extends the teacher-student SSL approach of DINO v2 with an additional task with the goal to improve the dense prediction capability of the trained model. Specifically, during training the student network is either tasked with the alignment of masked positional encoding as well as the standard alignment of masked out content views. To this end, the authors do not apply the positional with respect to just the cropped image part, but with respect to the position of the crop in the original image.\n\n---\nupdate after rebuttal: I think the authors should have referenced the ground work done by DropPos in a more prominent manner, as it explains the motivation behind the global mask very well. Nevertheless, the idea to sample from a virtual larger image is an interesting contribution.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The method improves object detection on COCO, but the improvements in segmentation are limited. The fact that the position sampling on its own, even without the additional training task shows significant gains is interesting. The importance of the sampling strategy (constant, uniform, beta) is significant, but not explained or discussed extensively. Might the main benefit of the method be simply its adjustment to challenging smaller objects in the COCO dataset?**\n\nAs you correctly pointed out, the improvement on ADE20K is limited compared to object detection tasks. We believe this is because ADE20K is primarily a pixel-level classification task that does not require explicit spatial reasoning, and thus benefits less from relative positional encoding. (We included ADE20K to enable fair comparison with prior work.) In contrast, object detection tasks on COCO require more precise instance-level localization, which our method is explicitly designed to address.\n\nThank you also for your valuable comment regarding the sampling strategy. To simulate the long-tail distribution of object sizes in COCO, we introduced a Beta distribution. Since small objects are frequent and more difficult to detect, we considered this distribution more realistic for the training setting. In future work, we plan to analyze the impact of different sampling distributions in a more systematic manner, and to explore their generalization to other tasks and datasets.\n\nWe also appreciate your question on whether the improvement mainly stems from better handling of small objects. Indeed, we believe the position sampling is particularly effective for small objects. However, our method also improves attention diversity and instance discrimination capabilities (see Figures 5 and 6). These improvements are not limited to small objects but contribute to better object-level reasoning across a wide range of object sizes.\n\n**Q2. Why do you use a 50/50 split between content and positional encoding prediction? ...**\n\nThank you for pointing this out. As shown in Table 4b, using position prediction alone leads to a significant drop in performance, whereas content prediction alone yields reasonably strong results. Nonetheless, we found that a 50/50 split between content and position masking achieves the best overall performance (see the table below for more details).\n\nWe hypothesize that the effectiveness of this balanced setup stems from the fact that the position prediction task, while insufficient on its own, encourages the model to explicitly encode spatial relationships, which synergizes with content learning. The 50% allocation provides a sufficient signal for content feature extraction while still allowing position prediction to serve as a meaningful auxiliary task. We agree that this trade-off warrants further exploration, and we plan to investigate more dynamic or adaptive scheduling strategies in future work.\n\n| Content vs. Pos. | 100/0 | 75/25 | 50/50 | 25/75 | 0/100 |\n|------------------|-------|-------|-------|-------|--------|\n| **AP$^\\text{box}$**  | 43.4  | 44.2  | 44.8  | 44.2  | 21.7   |\n| **AP$^\\text{mask}$** | 38.9  | 39.5  | 39.8  | 39.5  | 20.7   |", "claims_and_evidence": "The main claim of this work is that the proposed sampling of the positional encoding and the additional task of masking and predicting positional encoding vectors between a student and a teacher view improves the dense downstream performance of the model. The experiments suggest that both these additions to the DINO V2 method are beneficial.", "methods_and_evaluation_criteria": "The method is pretrained on ImageNet1k and its dense prediction performance is evaluated on COCO and ADE20K. A standard setup to compare with other SSL methods. Ablations studies focus on the object detection and instance segmentation performance on COCO.", "theoretical_claims": "no theoretical claims.", "experimental_designs_or_analyses": "Yes. Experiments and ablations are valid. Besides pretraining in a comparable setup to other methods, the finetuning in both object detection as well as segmentation downstream tasks is performed with established pipelines.", "relation_to_broader_scientific_literature": "The focus on the role of the positional encoding itself and its utilization during pretraining are valid contributions to further improve contrastive self-supervised learning when it comes to downstream tasks that require dense prediction.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n- The method is simple and the authors provide a good motivation for their approach. To build upon the pipeline of DINO v2 makes a lot of sense and the authors describe the additions well. The paper is quite easy to follow.\n- The improvements on the downstream evaluations on COCO over the DINO v2 baseline are significant.\n- The authors provide a lot of insightful evaluations, ablation studies and qualitative examples.\n\nWeaknesses:\n- The method improves object detection on COCO, but the improvements in segmentation are limited. The fact that the position sampling on its own, even without the additional training task shows significant gains is interesting. The importance of the sampling strategy (constant, uniform, beta) is significant, but not explained or discussed extensively. Might the main benefit of the method be simply its adjustment to challenging smaller objects in the COCO dataset?", "other_comments_or_suggestions": "None.", "questions_for_authors": "Why do you use a 50/50 split between content and positional encoding prediction?  Given that training with just positional prediction does not work well and the model still needs to learn content based feature extraction with a sufficient number of training samples.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BaCqkE90e4", "reviewer": "Reviewer_mPP8", "summary": "The paper proposes and investigates a novel positional encoding method and an extension to DINOv2 loss that incorporates positional masking for better SSL pretraining of Object Detection and Instance Segmentation Vision Transformer backbones. The method achieved competitive performance on COCO and ADE20K. Ablation studies show the effectiveness of the proposed components, positional sampling, and positional masking. The study also investigates a statistical ground for its scale distribution for the positional sampling part and empirical support for its positional masking strategy and mixed content-position prediction.\n\n## update after rebuttal\n\nThis is an interesting work to me. I like that they incorporate relative positional information in the pretraining by using their proposed positional encoding. The experiments displays some questionable points, as noted by the reviewers, but are overall promising. So it come to whether we should take reasonably projected results or ask for apple-to-apple comparisons, and I'll leave it to you. \nThe comparison between masking strategies is not significant. However, I probably should not just focus on the numbers. This work displays a complete path from identifying the cause of the attention artifacts to solving it by designing a masking strategy. The question of how the attention artifacts are less relevant to the final outcome is not a burning question that hinders acceptance. \nI am not concerned of the motivation. The question of why contrasive SSL for segmentation is reasonable and worth asking, but not very constructive if taken very seriously. Should it be constructive, then why SSL at all, we can always finetune. \nTherefore I deem this work to be worth displaying to the community for its progress in processing position information with SSL for segmentation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. One weakness of this work is the unclear cause of the stronger vertical artifact, compared to horizontal, in the attention map when box-based masks are used.**\n\nTwo potential explanations may account for the artifact being more pronounced vertically than horizontally: the statistical bias inherent in the image content and the effect of the horizontal flip (hflip) applied during training. To investigate these factors, we conducted two experiments. Note that although typical contrastive learning applies hflip independently to each crop, our method applies it uniformly to the entire input image to maintain consistent spatial relationships across views.\n\nFirst, to assess the impact of image content, we rotated all training images 90° counterclockwise and then trained the model under otherwise identical conditions (with hflip applied as usual). As a result, the artifact also rotated 90°, appearing horizontally rather than vertically when we fed test images in their original orientation.\n\nNext, we augmented the data by randomly applying a vertical flip (vflip) at a 1:1 ratio alongside hflip. We expected this to mitigate or randomize any directional artifacts; however, the horizontal artifact persisted.\nThese findings suggest that the predominance of the vertical artifact is primarily driven by the intrinsic characteristics of the image content. It appears that ImageNet—and natural images in general—exhibit different statistical properties along the vertical and horizontal axes. Further investigation into this phenomenon is warranted for future work.\n\nWe provide a figure at the following URL illustrating the effect of each transformation (Hflip, Rot90, hflip/vflip) on the resulting attention maps for reference.\n\nhttps://drive.google.com/file/d/1uezT10nsOwK6KtWqvBSN-so6B6DdiRUe/view?usp=drive_link\n\n**Q1. Though this work is positioned for improving IS and OD performance, displaying ImageNet classification results could make this work more valuable to readers.**\n\nThank you for your suggestion. We will include the ImageNet classification results in the revised manuscript. In summary, our method's performance is slightly lower than that of DINOv2* (our reproduction on ImageNet-1k), yet it outperforms existing SSL methods designed for dense prediction. We report the linear probing performance below, following the protocol used in DINOv2.\n\n| IM1K; Linear Probing | ViT-B/16 | ViT-S/16 |\n|----------------------|----------|----------|\n| DINOv2*              | 78.2     | 74.4     |\n| Ours                 | 76.8     | 71.6     |\n\n\n**Q2. Is the choice of scale factor distribution based on intuition or on mathematical grounds?**\n\nWe chose it empirically for the intuition of a higher position precision requirement for small/medium regions. Please refer to our response to ***Reviewer v6Kv Q1.*** for further discussion.\n\n**Q3. Does an imbalanced content vs position image ratio improve the results?**\n\nThank you for your interest. We found that our method is robust to the content vs. position masking ratio. Please refer details in the table in our response to ***Reviewer v6Kv Q2.***.\n\n**To other comments**\n\nThank you for pointing out these typos. We will fix the duplicated “Table,” add the missing reference to Figure 3, and standardize the usage of “Figure” throughout the text.", "claims_and_evidence": "This work claims that feature representation integrating content and positional information is important for OD and IS tasks. By introducing new cropping-aware positional encoding and position-masking during training, the ViT models studied can learn such features better. The claim is well-supported by comparative studies to other recent works in image SSL and ablation studies of the proposed components.", "methods_and_evaluation_criteria": "Yes. The paper proposes novel positional embedding and modified DINOv2 loss, adding positional prediction to improve self-supervised learning for Object Detection and Instance Segmentation. The datasets of choice are COCO and ADE20K, which are commonly tested datasets for OD and IS, respectively. The evaluation metrics are box/mask average precision and mean intersection over union for COCO and ADE20K, respectively. It makes sense.", "theoretical_claims": "The paper is highly empirical with minimal theoretical claims. However, in Section 4.3, there is a claim that relative object scale in object detection follows beta distribution with no further explanation. The ablation study shows a connection between these two by displaying a better result. Should the writer want to make an empirical observation, it is better to state it clearly.", "experimental_designs_or_analyses": "Yes. The work follows the evaluation protocol of previous works. For OD on COCO, they use the protocol from DropPos and framework from ViTDet while removing windowed attention and RoPE. IS on ADE20K uses the protocol from LOCA and Segmenter’s linear decoder for minimal adaptation. Methods are compared under the same pipeline, and the results are strong under standard metrics: average precision for COCO and mean intersection over union for ADE20K.", "relation_to_broader_scientific_literature": "This work extends traditional discrete positional encoding to a normalized vector space and uses it to incorporate relative positional/scale information of image patches in the embedding process. The work demonstrated stronger performance than traditional sin/cos positional encoding on OD and IS tasks.\n\nThe authors discussed that the position inference target connects to DropPos and LOCA. This work combines it with pixel inference and proposed novel positional encoding to achieve better performance. Also, the contrasive learning target is based on iBOT and DINO, which are strong image self-supervised learning methods. This work expands SSL for OD and IS by adding augmentation in positional encoding and combining positional inference and pixel inference.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "One weakness of this work is the unclear cause of the stronger vertical artifact, compared to horizontal, in the attention map when box-based masks are used.", "other_comments_or_suggestions": "Though this work is positioned for improving IS and OD performance, displaying ImageNet classification results could make this work more valuable to readers. A line plot of different ratios between position masking and content masking v.s. performance is another thing that interests me, potentially others. \nL362 - Table Table 3a: delete one “Table”\nFigure 3 is dangling with no reference to it in the main text.\nL643 - Inconsistent usage of Figure and Fig.", "questions_for_authors": "1. Is the choice of scale factor distribution based on intuition or mathematical grounds?\n2. Why is horizontal attention artifact absent or weaker than vertical ones? Does it imply that the vertical position is harder to predict for images in the pretraining dataset?\n3. Does an imbalanced content vs position image ratio improve the results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "usUxBws0DP", "reviewer": "Reviewer_dcuT", "summary": "The paper proposes a novel self-supervised learning (SSL) framework tailored specifically for object detection (OD) and instance segmentation (IS). The key idea is to integrate positional information more effectively by introducing a learnable positional encoding field that is aligned with the image cropping process. In addition, the method employs a dual masking strategy where both the content and positional embeddings of image patches are masked and then predicted. The authors report improvements on COCO for both detection ($AP_{Box}$) and instance segmentation ($AP_{Mask}$), and they also evaluate on ADE20K for semantic segmentation.\n\n## Update after rebuttal\nI appreciate the authors’ efforts in preparing the rebuttal and addressing the major concerns I raised. While I had suggested experimenting with a larger backbone to facilitate a more direct comparison with a state-of-the-art method, I recognize that this concern is relatively minor. Given the authors’ thorough clarifications and convincing responses, I have decided to adjust my overall assessment to Weak Accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Have you experimented with stronger backbones such as ViT-Large/16?...**\n\nDue to insufficient computational resources, we have not been able to conduct such large-scale experiments comprehensively; however, preliminary results confirm the scalability of our approach. Nonetheless, we believe that the performance of our proposed method has been sufficiently demonstrated by the results reported in the paper.\n\n**Q2. Have you explored the performance of your method under limited-dataset conditions for OD and IS?...**\n\nWe thank the reviewer for the suggestion. To further evaluate our method's performance under limited training data conditions, we conducted fine-tuning experiments on OD and IS. The results are shown in the table below. Here, the labels 1/32, 1/8, etc. indicate that only 1/32, 1/8, etc. of the original dataset (randomly selected) was used for training, following the experimental settings of LOCA. These results confirm the superiority of our method even under limited-dataset conditions.\n\n| Method | 1/32 AP$^\\text{Box}$ | 1/32 AP$^\\text{Mask}$ | 1/8 AP$^\\text{Box}$ | 1/8 AP$^\\text{Mask}$ | 1/2 AP$^\\text{Box}$ | 1/2 AP$^\\text{Mask}$ | 1 AP$^\\text{Box}$ | 1 AP$^\\text{Mask}$ |\n|--------|----------------------|-----------------------|---------------------|----------------------|---------------------|----------------------|-------------------|--------------------|\n| MAE    | 31.0                 | 28.8                  | 33.1                | 31.1                 | 42.6                | 39.0                 | 48.1              | 43.2               |\n| LOCA   | 30.4                 | 28.2                  | 33.0                | 30.6                 | 42.3                | 38.5                 | 48.3              | 43.0               |\n| Ours   | 33.0                 | 30.8                  | 32.8                | 30.6                 | 43.3                | 38.9                 | 49.2              | 43.8               |\n\n**Q3. Could you elaborate on why you chose to focus specifically on OD/IS rather than addressing SS in a unified manner?**\n\nAs you correctly noted, both OD/IS and SS rely on pixel-level class and positional information, yet their requirements for positional precision differ. In OD/IS, it is crucial to accurately distinguish individual instances—even when they are adjacent or overlapping within the same class. Consequently, OD/IS tasks demand the ability to extract and integrate detailed positional information with class cues—precisely the capability our proposed method is designed to achieve. In contrast, SS primarily benefits from positional information that ensures the continuity and consistency of regions corresponding to the same class. Thus, the fine-grained, class-integrated positional cues essential for OD/IS are not only unnecessary for SS but, given the constraints of representational capacity, may even be counterproductive.\n\n**Q4. Have you considered evaluating your method on additional datasets dedicated specifically to OD or IS?**\n\nWe conducted experiments on LVIS (with ViT-B/16, and 12 fine-tuning epochs) and observed a similar tendency to that on COCO, as shown in the table below.\n\n| Method | AP$^\\text{Box}$ | AP$^\\text{Mask}$ |\n|--------|-----------------|------------------|\n| MAE    | 30.1            | 28.7             |\n| LOCA   | 29.6            | 29.6             |\n| Ours   | 30.6            | 29.6             |\n\n\n**To other weaknesses**\n\nWe appreciate the feedback. We will revise Figure 2 to include clearer annotations and breakdowns to better highlight the key components of our method. Additionally, we plan to release the full codebase to ensure reproducibility and facilitate further exploration by the community.", "claims_and_evidence": "Claims:\n- By aligning the positional encoding with the cropping process, the method preserves crucial spatial cues that are lost in conventional approaches.\n- Simultaneously masking and predicting both content and positional embeddings leads to improved feature representations, especially beneficial for OD and IS tasks.\n- The proposed approach achieves competitive or superior performance on COCO compared to existing methods.\n\nEvidence:\n- Experimental results on COCO indicate improvements in AP_Box and AP_Mask compared to several baselines.\n- Ablation studies demonstrate that both the cropping-aligned positional encoding and the dual masking strategy contribute to performance gains.\n\nHowever, the direct comparison with LOCA - which also focuses on incorporating positional information - is limited. Further experiments (e.g., using a ViT-Large/16 backbone or few-shot settings) would strengthen the evidence for the method’s claimed advantages.", "methods_and_evaluation_criteria": "The proposed method is well-motivated for OD and IS because these tasks require precise localization and instance-level discrimination, which rely heavily on accurate positional information. The evaluation uses standard benchmarks - COCO for OD/IS and ADE20K for semantic segmentation - and includes extensive ablation studies to assess different design choices.\n\nConcerns:\n- While semantic segmentation is evaluated, the paper emphasizes OD and IS as the primary target tasks; the authors should clarify why they separate these tasks given that semantic segmentation also requires fine positional accuracy.\n- To better emphasize OD and IS, additional datasets specific to each task could be included.", "theoretical_claims": "There are no formal theoretical proofs provided in the paper. The contribution is primarily empirical, with the rationale for the positional encoding strategy based on intuitive arguments and validated through experiments.", "experimental_designs_or_analyses": "The experimental design includes:\n- Pre-training on ImageNet-1K using a ViT-based architecture.\n- Evaluation on COCO for OD/IS and ADE20K for semantic segmentation.\n- Detailed ablation studies analyzing the effects of different hyperparameters and design choices (e.g., mask sampling strategies, resolution of the positional field).\n\nConcerns:\n- The current experiments, while solid, could benefit from closer alignment with the settings used in LOCA. For instance, experiments with a stronger backbone like ViT-Large/16 and evaluations under few-shot scenarios would provide more comprehensive evidence for the method’s effectiveness on OD/IS.", "relation_to_broader_scientific_literature": "The paper builds on a rich body of SSL work including methods like DINO, iBOT, and DINOv2, and it specifically addresses limitations in previous approaches that attempt to integrate positional information. Although the idea of leveraging positional cues is not entirely new, the paper’s approach of using a cropped positional field and dual masking is presented as an incremental yet meaningful improvement. However, since LOCA is the most direct competitor in this area, a more rigorous experimental comparison is needed to highlight the distinct advantages of the proposed method.", "essential_references_not_discussed": "The paper cites most of the relevant works in SSL and dense prediction tasks.", "other_strengths_and_weaknesses": "Strengths:\n- Provides comprehensive ablation studies that detail the contribution of each component.\n- Experimental results on COCO demonstrate improvements in both detection and instance segmentation metrics.\n\nWeaknesses:\n- Figure 2 could benefit from additional annotations or breakdowns to clarify the process and key components of the proposed method.\n- The lack of provided code hinders reproducibility and a deeper understanding of the implementation details.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Have you experimented with stronger backbones such as ViT-Large/16? It would be valuable to see experiments that use a higher-capacity model to evaluate how your approach scales. Positive results in this setting would not only solidify your claims about the method’s effectiveness for dense prediction tasks but also offer a more direct comparison to competitors like LOCA, which have explored similar configurations.\n2. Have you explored the performance of your method under limited-dataset conditions for OD and IS? By evaluating your approach in few-shot settings - where only a limited amount of training data is available - you could further substantiate the claim that your method learns robust representations that are effective even with minimal supervision. For example, if your method can achieve competitive performance with a few-shot setup similar to that reported for LOCA, it would significantly reinforce your argument regarding its suitability for real-world applications.\n3. Could you elaborate on why you chose to focus specifically on object detection (OD) and instance segmentation (IS) rather than addressing semantic segmentation (SS) in a unified manner? The current motivation appears to emphasize locating objects rather than handling instance-level prediction uniquely. This raises concerns about whether the method is truly specialized for OD and IS as opposed to SS. A more detailed explanation of the distinct challenges and requirements of OD and IS would help justify your targeted approach and clarify the specific advantages your method offers for these tasks.\n4. Have you considered evaluating your method on additional datasets dedicated specifically to OD or IS? Although COCO is a widely recognized benchmark and provides a strong basis for evaluation, incorporating experiments on another dataset would help demonstrate that the improvements observed are not unique to COCO.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mm62Icz36T", "reviewer": "Reviewer_GJjf", "summary": "Post-rebuttal\n\nAfter reading the comments by reviewers v6Kv and mPP8, I agree that contrastive learning for dense prediction can offer advantages over autoencoder-based methods, such as more efficient training without heavy fine-tuning. I also agree that the proposed method introduces some technical novelty beyond DropPos. For these reasons, I have updated my rating to weak accept.\n\n---\n\nThis paper enhances dense representation learning from contrastive learning by 1) using position encoding as a relative location within the entire image instead of renormalizing it in each cropped image, and 2) reconstructing position information to preserve spatial structure. As a result, the proposed method outperforms or is on par with previous dense contrastive learning methods such as MuGS, FLSL, and LOCA.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.  Why use contrastive learning for dense representation? ...**\n\nThe importance of this study can be explained from practical and exploratory perspectives. \nPractical Perspective: Although the CVPR2022 MAE results demonstrate high object detection performance, they incur extremely high computational costs during both pre-training and fine-tuning (specifically, 1600 epochs for pre-training and 100 epochs for object detection fine-tuning). In contrast, in our experiments (and those of related studies), pre-training is conducted in 350-700 epochs and fine-tuning requires only 12 epochs; under these conditions, MAE yields inferior results. In other words, our approach achieves satisfactory performance while significantly reducing computational cost. \nExploratory Perspective: Our results offer new insights into the challenge of determining what information should be extracted from images for dense prediction tasks. They also show that even within the framework of contrastive learning, these tasks are learnable, and there may be further room for improvement. We expect that our work will serve as a stepping stone for future discoveries.\n\n**Q2. How does this method perform on tasks beyond object detection and segmentation? ...**\n\nAs you correctly pointed out, our method, like existing approaches, also shows relatively low performance on classification tasks. The linear probing accuracy on ImageNet-1K is 76.8% (evaluated using ViT-B/16 with DINOv2’s linear evaluation protocol). However, we would like to emphasize that we do not consider this a disadvantage. Regardless of the relative \"level\" of the tasks (if anything, we consider object detection to be a higher-level task, as it involves both classification and geometric estimation), the two tasks are inherently different to some extent—for instance, in the way positional information is utilized. Therefore, we view this trade-off as a natural consequence.\n\n**Q3. & Q4. The technical novelty is limited.  ...;  Position reconstruction has already been widely explored. ...** \n\nThank you for your comment. We respectfully contend that our proposed method is genuinely novel and should not be regarded as merely a variant of existing approaches. While it shares the common objective of embedding positional information into feature representations, its approach differs significantly from methods such as DropPos, MP3, and LOCA.\n\nMoreover, because simple position prediction tasks tend to overfit, DropPos uses a high masking rate (94%) and MP3 removes all positional tokens. Our method avoids these measures, demonstrating both its effectiveness and novelty. Although there may be some adverse effects on image-level prediction (classification), this is not our primary target; as reported in the table below for few-shot classification following the settings in MSN[1], the performance drop in classification is smaller than that of other methods with similar objectives.\n\nFinally, regarding the concern about the justification of our method’s effectiveness, our results on dense prediction tasks—supported by comprehensive evaluations (in Table 1), ablation studies (in Table 2), and attention analyses (Figure 6)—clearly show that our approach outperforms conventional methods. We are prepared to conduct additional experiments if necessary and appreciate the comment regarding MP3, which we plan to address in the revised manuscript.\n\n| Method | 1 Img/class | 2 Img/class | 5 Img/class | 1%   |\n|---|--|---|----|------|\n| MAE    | 6.0±0.2  | 9.6±0.3 |  17.5±0.2 | 28.3  |\n| LOCA   | 31.9±0.2 | 40.3±0.4 | 49.7±0.1 | 56.9 |\n| Ours   | 39.2±0.3 | 48.3±0.4 | 56.9±0.1 | 63.0 |\n\nNote: Unlike MSN, we report linear probing results for MAE without partial fine-tuning.\n\n[1] Assran, Mahmoud, et al. \"Masked siamese networks for label-efficient learning.\" ECCV, 2022. (KJ-note. Adding the performance of iBOT for few-shot.)\n\n**Q5. Why not explore contrastive learning on multi-object images?...** \n\nIn our study, we primarily focused on learning feature representations that benefit dense prediction tasks. However, as you noted, SSL on multi-object/scene images has also garnered significant interest in the community. To explore this direction, we conducted similar experiments using the Object365 dataset—a dataset of a manageable size—instead of ImageNet-1K. Without any hyperparameter tuning, we achieved 48.7 mAP on the object detection task, which is nearly equivalent to the 49.1 mAP obtained with ImageNet-1K. This demonstrates that our approach performs comparably on more challenging, multi-object images. We plan to further evaluate our method across a wider range of tasks beyond dense prediction in future work.\n\n**To minor comments.**\n\nWe will revise the reference for DropPos to correctly cite NeurIPS 2023. We will also note the DINOv2 results by reporting the performance from its original model trained on larger datasets and highlighting the difference in training conditions and expected performance.", "claims_and_evidence": "I'll combine this section with \"Methods and Evaluation Criteria.\"", "methods_and_evaluation_criteria": "I have several concerns regarding the motivation, method, and evaluation.\n\nA. Why use contrastive learning for dense representation? Many works since MIM/MAE have shown that pixel-level objectives learn better dense representations than contrastive learning. For instance, Table 4 of the MAE (CVPR 2022) paper shows that ViT-B trained with MAE on IN-1k achieves 50.3 and 44.9 box and mask AP, respectively, outperforming this paper’s 49.2 and 43.8, as reported in Table 1. What advantage does this method have over MAE and its numerous follow-ups, including those combining MAE with contrastive learning?\n\nB. How does this method perform on tasks beyond object detection and segmentation? It is well known that dense contrastive learning involves a trade-off between high-level tasks (e.g., classification) and low-level tasks (e.g., detection). I suspect this method may degrade classification performance. Could you report linear probing accuracy on IN-1k to verify this?\n\nC. The technical novelty is limited. The idea of position-aware cropping has been explored since DenseCL, with various methods adopting their own variants, including LOCA, as mentioned in this paper. Using global positional embeddings may serve as an additional trick for ViT-based contrastive learning, potentially encoding the scale of cropped images. However, its precise effect is not well justified. For instance, it introduces a distribution shift in positional encodings between training and testing, which could harm image-level prediction, as full [0-1,0-1] encodings are always used at test time, potentially reducing scale invariance in classification.\n\nD. Position reconstruction has already been widely explored. This includes methods such as DropPos, as mentioned in the paper. Additionally, MP3 [1] is an example of a missing related work.\n\nE. Why not explore contrastive learning on multi-object images? IN-1k is mostly object-centric, but region-aware contrastive learning would be more impactful for complex, multi-object images. LAION would be a practical example, but early academic papers [2,3] have demonstrated this on smaller datasets like COCO. Notably, region-aware contrastive learning can also improve image classification by preventing representation collapse, benefiting not just detection and segmentation.\n\n[1] Position Prediction as an Effective Pretraining Strategy. ICML 2022.\\\n[2] Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations, and Dataset Biases. NeurIPS 2020.\\\n[3] Object-aware Contrastive Learning for Debiased Scene Representation. NeurIPS 2021.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Major concerns were discussed above. Here are some minor comments:\n\n- DropPos is from NeurIPS 2023, not 2024, meaning it was released more than a year ago.\n\n- The DINOv2 results are misleading since the model was designed for larger-scale datasets. However, this paper reimplemented it on IN-1k, leading to significantly lower performance compared to other methods. It would be better to report both the original DINOv2 and the reimplemented version while clearly stating that the original DINOv2 performs better due to its larger training dataset.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "There is a vast amount of work on (dense) contrastive learning, so I understand that the paper cannot cover all of them. However, it should at least cite [1], as it is highly relevant to the position prediction proposed in this paper.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "All my concerns have been discussed above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission discusses a method for self-supervised feature learning for object detection. Initially, the reviewers raised concerns about the similarity to DropPos, limited gains on segmentation, somewhat limited experimentation, limited novelty (DenseCL, LOCA, DropPos, etc.), and limited comparison to MAE-style methods. After the rebuttal and a lengthy discussion the reviewers arrived at an accept recommendation. The reviewers recognized the merits of the work while pointing out that the rebuttal wasn't able to resolve all of the aforementioned concerns (e.g., limited comparison to MAE-style methods). AC concurs with an unanimous accept recommendation and recommends a weak accept.
Decision: Accept (poster)