CR2PQ: Continuous Relative Rotary Positional Query for Dense Visual Representation Learning
Authors: Shaofeng Zhang
Qiang Zhou
Sitong Wu
Haoru Tan
Zhibin Wang
Jinfa Huang
Junchi Yan
Venue: ICLR 2025 Poster (2025)
Keywords: Self-supervised learning
Distillation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Dense visual contrastive learning (DRL) shows promise for learning localized information in dense prediction tasks, but struggles with establishing pixel/patch correspondence across different views (cross-contrasting). Existing methods primarily rely on self-contrasting the same view with variations, limiting input variance and hindering downstream performance. This paper delves into the mechanisms of self-contrasting and cross-contrasting, identifying the crux of the issue: transforming discrete positional embeddings to continuous representations. To address the correspondence problem, we propose a Continuous Relative Rotary Positional Query ({\mname}), enabling patch-level representation learning. Our extensive experiments on standard datasets demonstrate state-of-the-art (SOTA) results. Compared to the previous SOTA method (PQCL), our approach achieves significant improvements on COCO: with 300 epochs of pretraining, {\mname} obtains \textbf{3.4\%} mAP$^{bb}$ and \textbf{2.1\%} mAP$^{mk}$ improvements for detection and segmentation tasks, respectively. Furthermore, {\mname} exhibits faster convergence, achieving \textbf{10.4\%} mAP$^{bb}$ and \textbf{7.9\%} mAP$^{mk}$ improvements over SOTA with just 40 epochs of pretraining.

Published as a conference paper at ICLR 2025
CR2PQ: CONTINUOUS RELATIVE ROTARY POSITIONAL QUERY FOR DENSE VISUAL REPRESENTATION LEARN-
ING
Shaofeng Zhang', Qiang Zhou', Sitong Wu', Haoru Tan', Zhibin Wang", Jinfa Huang,, Junchi Yan Sch. of Computer Science & Sch. of Aitificial Intelligence, Shanghai Jiao Tong University INF Tech Co., Ltd., CCUHK, *HKU, PPeking University.
(sherrylone, yanjunchi ) @ jtu edu.en.
Code: https://githin com/ Sherrryone//rooor
ABSTRACT
Dense visual representation learning (DRL) shows promise for learning localized in- formation dense prediction tasks, but struggles with establishing pixel/patch cor- respondence across different views (cross-contrasting rely on self-contrasting the same view with variations, limiting input variance and hindering downstream performance. This paper delves into the mechanisms of self- contrasting and cross- cross-oontaasting, identifying the crux of the issue: transforming discrete positional embeddings to continuous representations. To address the corre- spondence problem, we propose a Continuous Relative Rotary Positional Query (CR2PQ), enabling patch-level representation learning. Our extensive experiments on standard datasets demonstrate state-of-the-ard (SOTA) results. Compared to the previous SOTA method (PQCL), our approach achieves significant improvements on COCO: with 300 epochs of pretraining, CR2PQ obtains 3.4% % mAp46 and 2.1 % mApmk improvements for detection and segmentation tasks, respectively. Furthermore, CR2PQ exhibits faster convergence, achieving 10.4% % mAp6 and 7.9% mApmk improvements over SOTA with just 40 epochs of pretraining.
Existing methods primarily
INTRODUCTION
Self-supervised representation learning (SSL) has been attracting increasing attention in deep learn- ing, whereby the prediction problem is often formulated unlabeled data. SSI. methods can mainly be divided into three categories: 1) generative-based methods (Goodfellow al., 2014; Donahue & Simonyan, 2019), learning generate samples in the input space; However, generation can be computationally expensive and may not be necessary for representation learning. 2) contextual-based methods (Gidaris al., 2018), designing pretext tasks (e.g.,, denoising auto-encoders (Vincent et al., 2008), context autoencoders (Zhang et al., 2016), etc) pretrain the backbone; 3) contrastive-based methods (Chen al., 2020a; He et al., 2020; Chen et al., 2020b; 2021), taking augmented views of the same image as positive pairs and others as negative pairs. Contrastive-bass methods have shown great promise in downstream tasks, e.g., image classffication/dettction (He et al., 2017; Cai & Vasconcelos, 2018) and video classification (Han et al., 2020).
pretext task with pre-training on
Contrastive learning is family of instance discrimination- based methods (Chen et al., 2020a; He et al., 2020; Chen et al., 2020b; 2021), which trains the network by distinguishing positive image-level samples from their negative counterparts given query anchors from
during the learning process. These general (instance-Level) contrastive learning (Tian et al., 2020a; Oord et al., 2018) 8) and its many variants (Caron al., 2021; Zhou et al., 2022; Song & Ermon, 2020) are of the most popular directions that achieved great success in the past few years and dominate other methods (gencrative-base) and cantextual-basede in the field of SSL, especially for linear and finetuning classification tasks.
Junchi Yan is the correspondence author, who is also affiliated with Shanghai Artificial Intelligence Laboratory. Work was partly supported by NSFC (62222607), Shanghai Municipal Science and Technology Major Project (0215H22)))002))

---

Published as a conference paper at ICLR 2025
However, these methods (Caron al., 2021; Chen al., 2020a; Grill al., 2020) are still less competitive on dense predictive tasks (Wang et al., 2021; Xie et al., 2021c), egg., detection (He et al., 2017) and segmentation (Xiao et al., 2018). The main reason is that instance-level contrastive methods aim to learn global-discriminatiee information, but lack spatial-sensitive information. To address this issue, some dense contrastive learning (DCL.) methods with pixel-wise (Xie al., 2021e; Wang et al., 2021) and patch-wise (Yun et al., 2022) contrastive objectives and frameworks are proposed. However, these methods still have other limitations: One main shorteoming of these dense contrastive learning methods is establishing the correspondence among pixels/patches usually requires billneaar interpolation, which is complex and heavily sensitive to random crop augmentation (in an extreme case, if two views have no intersection parts, there are no correspondence relation). One simple way to avoid mining the correspondence is inputting the same view (masked and unmasked versions) twice (Zhou et al., 2022). However, the variance of the inputs (masked and unmasked views) is much lower than inputting two different views, where the variance of two views has been proven to be the key to success in contrastive learning (Wang et al., 2022a; Tian et al., 2020b).
To address the correspondence problem, we propose a continuous rotary cesttionl-qqery-basee paradigm. Specifically, we first randomly crop two views from the image, and use the relative coordinate system and rotary positional embedding to represent the relative positions between the two views. Then, we take the rotary relative positional embedding as a query and reconstruct the latent representations (or RGB value) of one view from the other view. Through the relative coordinate system rotary positional query, our CR2PQ significantly simplifies the previous DCL paradigm (usually using GCN or other sub-network to learn the correspondence information), making the dense contrastive learning correspondence-fiees Our main contributions are:
i) Correspondence-fres DCL via relative coordinate system and continuous rotary positional embedding. Instead of simply employing RoPE to the vision domain (ID to 2D), we further transform the discrete positional embedding to continuous, taking the relative coordinates as input, which makes our CR2PQ correspondence-ffes
Besides, due to the simplicity of CR2PQ, our methods can be integrated into several popular representation learning paradigms, e.g., MIM-based, CL.-based, and Distillation-base based methods (See Table 4).
ii) Positional-aware query module. Instead of directly using the vanilla self attention (Vaswani et al., 2017) (which would hurt the downstream performance and bring more computational cost due to the extra query tokens, as the downstream task only inputs the raw tokens), we propose positional-awan cross attention module in the final block between query tokens and patches embeddings to learn semantic information of query tokens, and the cross attention only incurs a few extra parameters.
iii) New SOTA performance. We conduct exhaustive experiments on classification, detection, and segmentation tasks, where our CR2PQ achieves new SOTA results. In particular, CR2PQ outperforms previous DCL methods with a large range, especially on dense prediction tasks. Specifically, outperforms previous SOTA by 3.4%, and 2.1 % points for detection and segmentation on the MSCOCO dataset. CR2PQ surpasses previous SOTA PQCL 1.9% points on ADE20K semantic segmentation. We further conduct experiments to show its convergence speed, and we surprisingly find with few epochs (e.g.. 40) pretraining, CR2PQ outperforms previous SOTA DCL methods (Zhang et al., 2023a) 10.4% % and 7.9% scores on detection and segmentation on the MSCOCO dataset.
2 RELATED WORKS
Instance contrastive learning. Instance self-supervised learning aims to extract informative image representations by leveraging unlabeled data. It achieves this by applying contrastive objectives, which essentially bring similar representations closer while pushing dissimilar ones further apart. A key challenge in contrastive learning is handling negative examples, which are data points used to differentiate the representation of the target image. Memory-based methods like MoCo (He et al., memory bank for comparisons. SimCLR (Chen
2020; Chen et al., 2020b) store negative examples et al., 2020a), on the other hand, treats all other data points within the same batch as negative examples. To avoid the need for explicit negative examples, BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021) employ separate encoder and predictor networks with a stop-gradient mechanism. Barlow Twins (Zbontar et al., 2021) 1), ZeroCL (Zhang et al., 2021), 1). and VICReg (Bardes et al., 2022) achieve similar results by focusing on decorrelating features different levels. Building upon ZeroCL (Zhang et al., 2021), 1). ARB (Zhang et al., 2022) proposes aligning representations with an orthogonal base for computational efficiency. Inspired by the success of Vision Transformers (ViTs), recent work
2

---

Published as a conference paper at ICLR 2025
like MoCo v3 (He et al., 2020) and DINO (Caron et al., 2021) replacing convolutional neural networks (CNNs) with ViT backbones.
Dense contrastive self-supervised learning. In contrast to general contrastive learning, dense contrastive learning (Wang ct al., 2021; Yang et al., 2021; Ziegler & Asano, 2022; Ge et al., 2021) methods aim to learn spatial-sensitive information to provide a pretrained model to dense predictive tasks (e.g., object detection, instance segmentation and semantic segmentation). DenseCL. (Wang al., 2021) exploits the correspondence by sorting the similarities of pixels in the deep feature map, while PixPro (Xie et al., 2021c) utilizes the augmentation wrapper to get the spatial correspondence of the pixel intersection between two views. Furthermore, Detco (Xie et al., 2021a) tries to improve the performance of general contrastive learning approaches by augmenting multiple global and local views simultaneously. Inspired by PixPro, Resim (Xiao et al., 2021) uses Rol Pooling (Jiang et al., 2018) to extract feature vector from the associated feature map region for both views. On the basis of DenseCI., SetSim (Wang et al., 2022b) employs a threshold selection to filter out noisy backgrounds. With the development of ViT in SSL (Dosovitskiy et al., 2020), SelfPatch (Yun et al., 2022) treats the spatial neighbors of the patch as positive examples for learning more semantically meaningful relations among patches. On the basis of DINO (Caron et al., 2021), ADCLR (Zhang et al., 2023b) proposes patch-level contrasting via unmasked query tokens and cross-attention mechanism to avoid mining spatial correspondence. To further increase the variance of patch-level contrasting, PQCL (Zhang et al., 2023a) proposes to replace the unmasked query tokens with the relative positional embeddings, which further increase the difficulty of DCL, resulting in better performance in dense predictive tasks.
Masked Image Modeling (MIM). Masked Image Modeling (MIM) self-supervised learning technique where the model reconstructs masked portions of an image to learn meaningful repre- sentations. MIM methods can be categorized based on the information they reconstruct: (a) raw pixel values, (b) auxiliary features, and (c) masked patch embeddings. SimMIM (Xie et al., 2022) and MAE (He et al., 2021) ) (a) reconstruct the raw pixel values from masked or partially observed patches. MaskFeat (Wei et al., 2021) (b) incorporates HOG features as supervisory signals for richer semantics. CIM (Fang et al., 2022) (a) enhances robustness by adding perturbations to raw images. iBOT (Zhou et al., 2022) (c) is the first to predict token embeddings, but using the same view might influence downstream tasks. SIM (Tao et al., 2022) (c) addresses this by predicting masked patch embeddings from different view, but introduces higher computational cost. MIM research offers promising approach for learning powerful image representations for various computer vision tasks.
RoPE in vision domain. Relative Positional Encoding (RoPE) (Su et al., 2024) has emerged promising technique to improve vision transformers. Pioneering studies introduced RoPE to ViT- based architectures, with the Hybrid X-forme (Jeevan & Sethi, 2022) applying 1D RoPE. While this approach demonstrates the potential of RoPE, its limitations in capturing spatial relationships hinder performance on complex vision tasks like classification, detection, and segmentation. Additionally, evaluations using small datasets might not accurately reflect the effectiveness of RoPE on larger and more intricate datasets. Recent studies exploring 2D RoPE, such as EVA-02 (Fang al., 2023) and Unified-1O 2 (Lu et al., 2023), focused on language-related tasks or new model architectures. This work aims to bridge this gap by investigating the effectivvnesss of 2D RoPE in improving the performance of basic architectures on challenging vision tasks.
3 METHODOLOGY
3.1 PRELIMINARIES
APE and Vision Transformers. Denote an image by x CC where H W is the resolution of the image and C is the number of channels. Plain ViT (Dosovitskiy et al., 2020) treats the image
as sequence composed of non-overlapping patches (x()) g Ji=1, where each patch has fixed P x P resolution. Then, the patches are linearly transformed to D-dimensional patch embeddings (1) + P pas R²,, where E E DDCP2² is the linear projection and P E R is the positional embedding for the i-th patch. A [C'SS] token ([CLS)) RRD is subsequently prepended to the patch sequence to extract global information, so the resulting input sequence is represented as z = [z (CLL)) (1), 2 ..... z N).. Then, ViT uses a Transformer encoder ( (Vaswani et al., 2017) to generate both lmage-level (CTS.) token) and patch-level (other tokens). In line with SelfPatch (Yun
3

---

Published as a conference paper at ICLR 2025
Patchufy
Stadent
Continuous RoPE Cross Attention
Rolary Cross
patch
Rotary magee
Xx E
f
Aftcation
2-3
Ralotvee
Pooling
[CL5] A
Compute Continoous
[CLS]]
[CS)]
[CLS] T
Globo
Global
projecter d'r
Fretratndd
soaterr
View
f
spatch
Shuiie
projector
Local
Teseler
Figure I: framework the proposed CR2PQ. We first crop the image twice along with augmentations to generate view A and view В. Then, we compute the continuous relative position of the two views (illustrated the right figure). For view and view B, we feed them into the transformer encoder and the pre-traue pre-trained teacher network, respectively. Then, embedding to extract patch-level embeddings of view B through the rotary cross-attention block. Finally, the patch-level and global alignment objectives are added to learn dense representations.
the pre-caleulated rotary positional
et al., 2022), we use fo to denote the whole process of a ViT parameterized by 0:
fa(x) = fo ([2((CT..S)) ())
(N) ((C..)) (1))
(M)
(1)
where (CT.5) and (+) are the representations of the whole image and i-th patch, respectively.
Rotary positional embedding in 1-D. In Transformer models, self celf-attention plays crucial role in capturing relationships between words in sentence. However, self-attention mechanisms lack inherent knowledge of word order. Positional encoding techniques address this by injecting positional information into the model. Traditional discrete relative positional encoding (RPE) methods (take the discrete patch index as input, and return the positional embeddings), like relative positional bias (RPB), are often limited by the interaction with attention weights, which causes limited utilization of relative position. Thus, RoFormer (Su et al., 2024) proposes novel relative position embedding method: Rotary Position Embedding (RoPE). RoPE directly incorporates relative position information into the attention computation by applying trigonometric functions (sine and cosine) to key and query vectors. This allows RoPE to interact effectively with attention weights, leading to a comprehensive understanding of word order within the self-aftention process. Specifically, RoPE introduces the multiplication of Euler's s formula (c to key and query vectors as relative position embedding. ie., when n, m-th query and keys in qn Km E 1g1xXnead
RoPE is applied follows:
d
qne in kn eme imm
Attmn, Re Rela k = Re(q,, (i(n-m)) m)0
where Re[-] denotes real part of complex number and means complex conjugates.
3.2 THE PROPOSED CR2PQ
View Generation. Given the image x E Hxx'3 in training set, we first randomly crop the image twice to generate two views X and Xg. Then, we resize the two views to H x W x 3. Note that we randomly crop two views, therefore, the positional relationship of the two views might be either containing, overlapping. or non-overlapping. We record the absolute position of xB, respectively. Specifically, PA and Pa are composed of PA = (P. PAy PAr. P. (top location, left location, height and width) and Ps = {PB:: (PBi.PBj, PBj, PBL. PBw respectively.
of XA and pæ
Computing Relative Positional Coordinates. The core idea of our CR2PQ is to use the relative coordinate system and continuous rotary positional embeddings to represent the positional relationship between the two views. For simplicity, we use view A to reconstruct view B as an example. Specifically, Suppose the grid size of the feature map of the view A is KA. Then, the patch index
matrix of view can be written as:
(0,0)
(0, 1)
(0.
(0, KW 1)
(1, (1 v -1
(1,0)
(1,1)
(1,2)
rPA
(2,0)
(2
(2
(2, K -1)
(3)
(K
1,1)
(K
1,2)
(K - -1)
4

---

Published as a conference paper at ICLR 2025
Overlapping
Containing
Non-Overlap Nor-Overlapping
iew
B
View
Illustration three different scenarios (overlapping, containing, and non-overlappingg. The proposed relative coordinate system easily represent the relative coordinate matrix rp in both three scenarios. We always use the top-left corner of the view as the origin point and set the interval of the feature map of view A as 1.
Figure 2:
where K H/PSA. and KW W/PSA, and p is the patch size when inputting the view to the model. Observed through Eq. 3, we can recognize the patch index matrix rp.1 as a 2-D coordinates system, where the top-left element is the origin point and the scale of the coordinates system is 1. The detailed illustration can he found in Fig. 2. Since we set each grid size of the feature map of the anchor view 1, then the relative positional indices (relative coordinates) of the patches of the view B can be written as:
K (pm: PAi)
Pbb K (m om-1) 1)
rp (psA, (ps).ssB)
+
PAR
KB PAh
Row
(4)
KW
(paj PAj)
KW -nn
PBa
W
PAw
PAw
Column
where (rp") rp') is the m-th row n-th column element in the coordinate matrix rpg. psn is the patch size when inputting the view B to the model, and K H/psB. K W/psB. Note that Asa and psg are dependent on the patch size of the predefined backbone and teacher networks.
Continuous Relative Rotary Positional Embeddings. Given the coordinate indices matrix rpA of view A, suppose the hidden dimensions of each head of the backbone Dread, then we first convert query and key CIx(Pncaa)) (Dacaa/4) by considering the 2t-th dim as real part and 2t + 1-th dim as imaginary part. For each channel, following RoPE (Su et al., 2024), we utilize multiple frequencies O, in the key and query, i.e., 0g t/
where E {0, 1, 1,... Dread/45 /4). Then, for the m-th row and r-th column location the positional matrix in Eq. 3 of view A, the rotation matrix
Ryot E C(K K ) x Dnead/4 can be written as:
R nm.). rot
= el0m R.ma, (m,m,22+1)) R R (1,122++1)
elm 0 0st Dhead/2
(5)
R (m.n.2t) rot
i0en
i ilen Dread/2 S < Dhead
=
K" <(-(Pp--PA))
Psic PRR-KK (m-11 and
Then, for view B, can replace the m, in Eq. with
PAR
K PAh
K (pay PAs)
Pa KW (n-1) respectively. Then, the relative rotary matrix becomes KW PAu
PAw
50,
Kll (ra, PAs)
Pah K PAR
R
rela e
(m.n)
PAb
0 < < Dhead/2
(6)
x (Pa)-PA))
PM K (n-1) K
PAw
Dread /2 S
< Dread
where 2f-th dims are the real part and 2t + 1-th dims are the imaginary part.
Framework and Objectives. Given the cropped views XA, Xg, first calculate the relative positions rp and rp Then, we calculate the rotary matrix R, and relative rotary Ryeto via Eq. and Eq. 6, respectively. Then, feed the X the encoder network, which is composed of
5

---

Published as a conference paper at ICLR 2025
patchify layer and several transformer blocks. For each block, we replace the self-attention with:
KX-
exp
(R (m.") (R(at
(m..)))
(R.
Pot ( k(h,,))))) v v((h,))
(A,10)
Attn(Q, K, V)(
H h==
KX 2==0
(m.n) (Root m().))) T (R (h.a0) (Ro ((
h=0
w==
exp
(7)
where q (m. h (m.
(m.n) Wy, where h (m,,n) is the hidden representation of the patch located in m-th row and n-th column. Wo, W Wy are learnable parameters. Given the parameterized backbone by fo, we can calculate the representation of the view A by AΛ = (xA, Ryat). Then, the position-awan cross attention becomes:
WQ. Q. k(m,,)
(mm,n)
WK. K. and v(m,n)
K
K--1
exp
RReiia
(m,") ((rpp
(m.")))
TT (T (h.w) k(h,o))) v(h,1))
hPa
(8)
RR (m.n)
4(rp ((rp(mm))) (m)
))T (R (
k(h,))))))))))))))))))))()
1=0 h==0
==
exp
rela
(m.n) is given by Eq. 6, and rp (m,.)) the m-th row n-th column elements of the relative where Ryela
coordinate matrix rpn, derived by Eq. 4. (rrpg (m..)) ) (q(m.n) + hmaak) W where m is the learnable masked placeholder, which is also commonly used in other methods (Zhou et al., 2022: Xie et al., 2022), and Wo is the learnable parameters. After obtaining the hFPa, we feed the h* to the light-weight decoder 9,, to obtain Zg, and predict the embedding or RGB pixel values of the view B. Specifically, for distillation methods, adopt pretrained encoder f-)., and extract the hidden representations by z = f/ ()),, where E &K x xKğ Diee, where Drea is the output dimension of the teacher model. Finally, we compute the patch- objective by:
K
K B
(h,w) 2B
z (hhww)
Patch
K
Mcanwhile, we add another global objective term learn global-discriminatn information by:
W (haw)
k KW
LGlobal = g
h
z (h.w)
(10)
KH
W B
K
KW h=1 ==1
where g', is the global projector composed of linear layers and an activation function. Finally, we to balance the global and local objectives LCRePQ = Pattch + A LGtobal-
adopt a
4 EXPERIMENTS
Datasets. We conduct self-supervised pre-training on the ImageNet-IK IK (Deng et al., 2009) training set with 1,000 classes, as used in SSL for both MIM (He et al., 2021) and contrastive learning (Chen et al., 2020a). We also transfer the encoder pre-trained by CR2PQ on MS-COCO (Lin et al., 2014) and ADE20K (Zhou et al., 2017) datasets.
Pre-training hapee-parameters. Adamw (Loshchilov & Hutter, 2018) and batch size of 2048, distributed over 32 GPUs using ViT-S/16 (batch size per GPU 64). For ViT-B, the learning rate is linearly ramped up during the first 40 epochs to its base value determined with the following linear scaling rule (Chen et al., 2020a): bir = 1.5e-4, BatchSize=2048, and Ir = blr * BatchSize 256. For ViT-S, we set bir as 1.75e-4. After warmup, decay the learning rate with cosine schedule (Loshchilov & Hutter, 2016). We follow the data augmentations of BYOL (Grill et al., 2020) (color jittering, Gaussian blur, and solarization) with bicubic interpolation to adapt the position embeddings the scales.
line with CAE (Chen et al., 2022), we train with
Platform.
The experiments are performed on a workstation with 32 V000 GPUs by default (if not otherwise specified).
4.1 MAIN RESULTS
COCO object detection and segmentation. Setups. We evaluate pre-trained models on the COCO object detection and instance segmentation tasks (Liin et al., 2014). We evaluate model under two

---

Published as a conference paper at ICLR 2025
Table 1: Accuracy on MS-COCO. Mask R-CNN (He et al., 2017) and Cascade R-CNN (Cai & Vasconcelos, 2018) are adopted and trained with the Ix schedule. All the results are obtained by using our same finetune protocol for fair comparisons. Epoch refers to the number of pretraining.
Method
Rackhose
Framwork
AFpochs
Paaaam.
aViews.
Apli
Ohject Detection
AP
AP
Ap
Instance Segmestation
AFT
AP
MicssV2 (Chen et 20X0)
RecNet-50
200
23M
234'
38.9
59.2
42.4
35,5
56.2
37.8
SwAV (Caron et al., 2020)
ResNe::0
200
23M
x 22-4"
38.5
60.4
41.5
35.4
57.0
37.7
(Ning al... 2021)
ResNet-0
200
23M
2 x 224
40.3
59.9
44.3
36.4
57.0
39.2
ReSim (Xaa sl. 2021)
ResNei-50
200
23M
2 22.4
40.3
60.6
44.2
36.4
57.5
38.9
DelCo at. 202222
ResNe: 50
200
23M
22-4"
40.1
61.0
43,9
36.4
58,0
38.9
1021)
16T-
Mask RCNN
224
59,6
39.2
McBY (Xie al. 2021b)
VIT-S/16
22
224
41.1
63.7
44.8
37.3
60.3
39.8
DINO (Caon al., 2021)
VIT-S/16
300
22M
2x 224'
40.5
63.4
44.2
37.3
59.9
39.5
SelfPaech (Yim et al, 2022))
VIT-S116
200
22M
2 x 22/F
42.1
61.9
46.1
38.5
61.3
40.8
iBOT (Zhon et al. 2002)
VIT-S/16
200
22M
X 2244
42.6
65.7
47,0
39.0
61.7
41.3
POC1 (20nng
3023g)
VIT-5/16
200
234'
POCL (2hang 2023a)
VIT 1/
48.1
39.7
42.2
CR2PQ (Ouns)
VIT-8/16
200
22M
224
45.0
67.4
49.0
30.9
63.4
42.7
CR2PQ (Duns)
VIT-S/16
300
22M
2 N 224
47.4
69.3
52.0
41.8
65.9
44.7
DINO (Cron 2021)
VIT-376
300
22M
2 224"
45.2
47.8
38,0
61.2
41.7
SeifPasch
VIT-116
224'
62.0
42.6
DINO (Caron al.. 2012)
S/1
800
22M
96
46.8
06.7
50.3
40.6
43.2
IBOT (Zhou etal. 2023)
VIT-S/16
300
22M
224'
45.4
65.1
49.0
39.6
62.1
41.7
IBOT (Zhou et al., 2022)
VIT-S/16
Cascade RCNN
500
22M
x
226
10xx 962
49.4
68.7
53.3
42.6
65.6
45.8
PQCL (2hang ol., 20233)
VIE-S/16
200
22M
2
224
46.2
65.5
49.8
39,9
62.3
42.6
PQCT (Zhang ol., 2023a)
VIT-S/16
300
22M
2 224
47.7
67.0
$1.3
41.1
64.0
44.2
RaPE: 2024) (Scrates)
VIT-3/16
224"
33.4
44.9
29.2
CRZPO (Uurs)
VIT-3/16
22M
224
49.4
68.1
53.5
42.7
05.4
46.1
CR2PQ (Durs)
VIT-S/16
200
234
50.3
69.3
54.4
43.2
66.5
46.5
CR2PQ (Durs)
VIT-S/1
300
22M
2
224'
50.5
69.5
54.7
43.4
66.7
46.8
popular frameworks
(He et al., 2017) and Cascade R-CNN (Cai & Vasconcelos, 2018)
with the standard Ix schedule (12 epochs). In line with previous methods (Zhou et al., 2022: Zhang et al., 2023a), we adopt Adam (Loshchilov & Hutter, 2018) optimizer and set the learning rate as 3e- 4 with weight decay 0.05. Evaluation. MS COCO (Lin et al., 2014) is a large-scale object detection, segmentation, and captioning dataset: in particular, train 2017 and val 2017 splits contain 118K and 5K images, respectively. We follow the basic configuration of mmdetection (Chen et al., 2019) for fine-tuning Mask R-CNN (He et al., 2017) with FPN (Lin et al., 2017) under the standard 1x schedule. Results. Table shows the proposed CR2PQ can consistently outperform previous SOTA (Zhang al., 2023a) in both object detection and instance segmentation tasks. We evaluate CR2PQ with both 200 and 300 epochs pretraining. When applying the pre-trained model to Bascaded-RCNN, for 200 epochs pretraining without local views (Caron et al., 2020) and query views (Zhang et al., 2023a), CR2PQ surpasses PQCL (Zhang et al.. 2023a) with 100 epochs pretraining and one extra query view pretraining 1.7% point mAp6 and 1.6 % point mAP respectively. Besides, our CR2PQ with 300 epochs pretraining can outperform iBOT (800 epochs pretraining with 10 extra local views) by 1.1% % point mApb and 0.8% % point mAP When adopting the Mask-RCNN framework, under 200 epochs pretraining, our model CR2PQ outperforms PQCL by 1.9% point mAp and 0.6 % point mAPmA respectively. With 300 epochs pretraining, compared with previous SOTA PQCL (Zhang et al., 2023a), our CR2PQ achieves 3,4% mAP* and 2.1 % mAp improvements and segmentation tasks, respectively. We also evaluate the detection and segmentation performance without pretraining, i.e., directly using 2D RoPE (Su et al., 2024), where the 2D RoPE obtains much worse results than CR2PQ.
detection
ADE20K semantic segmentation. Setup. We evaluate semantic segmentation performances of pre-trained models on ADE20K (Zhou et al., 2017). which contains 150 fine-grained semantic categories and 25K training data. We finetune the pre-trained models on Semantic FPN (Lin et al., 2017) and UperNet (Xiao et al., 2018) with 40K and 160K iteration, respectively. Following the previous methods SelfPatch (Yun et al., 2022) and PQCL (Zhang et al., 2023a), we report three metrics: (a) mean intersection of union (mloU) averaged over all semantic categories, (b) all pixel accuracy (aAce), and (c) mean class accuracy (mAcc). Evaluation. ADE20K (Zhou et al., 2017) is semantic segmentation benchmark containing 150 fine-grained semantic categories and 25 images. We follow all the configurations of mmsegmentation (Contributors, 2020) for fine-tuning Semantic FPN (Lin et al., 2017) with 40K iterations and an input resolution of 5 512x512 12. We also perform large-scale fine-tuning experiments using UPerNet (Xiao et al., 2018) with 160K iterations and input resolution of 512x512. Results. As shown in Table 2, CR2PQ can outperform previous all methods under the same setting. Besides, we further evaluate DINO (Caron et al., 2021) 1) with 10 local views and 800 epochs pretraining (checkpoint is downloaded in their official repository). where CR2PQ gets 2.6% point improvements with only 200 epochs pretraining and only two global views. We guess the big improvements are because CR2PQ is patch-level distillation method, which makes our method more spatial-sensitin resulting the higher performance in dense prediction tasks.
7

---

Published as a conference paper at ICLR 2025
Table ADE20K semantic segmentation performances of the recent self-supervised approaches pre-trained on ImageNet. The metrics mloU, aAcc, and mAcc denote the mean intersection of union, all pixel accuracy, and mean class accuracy, respectively.
Method
Arch
Backhone
#fter
#Epochs
#Params
#Views
mloU
aAce
mAce
MoCo-v2 (Chen et al., 2020b)
FPN
ResNet50
40k
200
23M
2 x 224
35.8
77.6
45.1
SwAV (Caron et al., 2020)
ResNet50
40k
200
23M
2 224
44.9
DenseCL, (Wang et al., 2021)
FPN
ResNet50
40k
200
23M
2 224
37.2
78.5
47.1
MrcaV3 (Chen et al., 2021)
FPN
VIT-5/16
40k
300
23M
2 224"
35.3
78.9
45.9
MoBY (Xic 2021bb)
FPN
VIT-S/16
40k
300
23M
x 224
39.5
79.9
50.5
DINO (Caron et al. 2021)
FPN
S/16
40k
300
23M
2 224
38.3
79.0
49.4
DINO (Caron et 2021)
UperNet
VIT-S/16
160k
300
23M
2
42.3
52.7
SelfPatch (Yun et 2022)
FPN
S/16
40k
200
23M
x 2242
41.2
80.7
52.1
SelfPatch (Yun et 2022)
UperNet
VIT-S/16
160k
200
23M
2 2242
43.2
81.5
53.9
DINO (Caron et al., 2021)
UperNet
S/1
23M
224 962
44.4
iROT (Zhou et al., 2022)
UperNet
VIT-S/16
160k
200
23M
2 224
44.1
81.4
55.3
ADCLR (Zhang et al., 2023b)
UperNet
S/16
160k
200
23M
2 224
44.3
81.9
55.1
PQCL (Zhang al., 2023a)
UperNet
VII-S/16
160k
23M
224
56.1
CR2PQ (Ours)
UperNet
VIT
160k
200
23M
2 224
47.0
83.1
57.4
Table 3: Finetune top-1 1 and top-5 classification accuracies on ImageNet- IK with VIT-S and ViT-B. "PT Eps" and "FT Eps" mean the number of epochs for pretraining and fine-tuning, respectively.
Method
Backthone
#Params
Eps
Egs
Top-1
Top-5
Racktbone
#Params
Eps
Egg
Top-1
Top-5
Deil (Touxion al., 2021)
VIT-S
22M
300
200
79.9
V/T-B
SGM
300
100
$1.2
MAE (He et al., 2021)
VIE-
22M
1000
200
80.5
VIT-B
86M
300
100
83.2
Moeo V3 (He al. 3021))
VIT-S
22M
1600
200
81.4
VIT-B
86M
800
1000
83.2
DINO (He
2021)
VIT-S
22M
1600
200
81.5
VIT-B
86M
800
100
82.8
BBT (Zhou al., 2022)
VIT-S
200
81.8
WTTB
Tiny MIM (Oxhulak al., 2023)
VIT-S
22M
300
100
81.5
85.8
VIT-B
86M
300
83.4
96.3
(DarS)
VIT-S
22M
300
200
N2.2
86.1
VIT-B
86M
300
100
83.7
96.5
4.2 ABLATION STUDIES
Fine-tuning classification. Most of the previous dense contrastive learning methods (Yun et al., 2022; Xie et al., 2021c) show worse performance compared with global instance-level contrastive methods (Zhou et al., 2022; Caron et al., 2021), which is because the patch-level loss could inhibit the model from learning ciobal-disrimminatve information. However, different from previous DCL methods, we find CR2PQ could achieve comparable results on fine-tuning classification with the instance-level SSL methods. We guess that's because add the global objective in Eq. 10, which helps CR2PQ learn global-diseriminative information. Table 3 shows the fine-tuning accuracies with different backbones (ViT-S and ViT-B). With ViT-S, CR2PQ achieves 82.2 top-1 accuracy with only 300 epochs pretraining, which outperforms previous SOTA iBOT (Zhou et al., 2022) with 800 epochs pretraining by 0.4 point. With ViT-B, our methods achieve 83.7 top-1 accuracies with only 300 epochs pretraining, which outperforms under the same epochs pretraining.
Table 4: Comparisons with different teachers on the COCO datasets (detection and segmentation).
Teacher models and architectures. To ex- plore the effect of the teacher model, we test our CR2PQ with different teacher models. We also evaluate our CR2PQ without teacher mod- els (See EMA update and Pixel) in Table 4. Specifically, we adopt ViT-S (Dosovitskiy et al., 2020) as the student model and use VIT-B, ResNets (He et al., 2016), and VIT-L the teacher models. As the output dimensions of different teacher models could be variant (e.g., 14 x 14 768 for ViT-B/16, 7 7 2048 for
Stadest
Teacher
Epoch
EAPp
mAP
EMA update (Contrastiv))
46.1
30.3
Pixel (Masked Image Modeling)
45.2
40..0-10)
DINO (Caron et al., 2021) (Res 50)
47.4
41. (+2.0)
DINO (Carea al.. 2021) D)
47.0
(+1.3)
VIES
SinCER (Chen
3020laa (Res-5t)
300
47.1
(1.6(418)
MAE (He al., 2021
46.5
40.7--0.)
MAF (He et al., (VIT1)V
47.0
4141--16
IBOT (Zhou al., 2022) (VT
46.2
4100+12)
BOT (2hou all, (VIT)
46.5
41.5
ResNets), we change the grid size K B and KW of the relative positional index. More concretely, for ResNet-50 and ResNet- 101, we set the grid size of the positional embeddings to 224/7 32. For ViT-B/16, we set the grid size of the positional embeddings to 224/ 14 = 16. We mainly choose the teacher model pre-trained by some profound methods, including MAE (He et al., 2021), iBOT (Zhou et al., 2022), and DINO (Caron et al., 2021). We keep the batch size as 2048 and use VIT-S/16 as the student network to train CR2PQ with 300 epochs. Table 4 shows the results with different teacher models. We find using contrastive-base teacher model (Caron et al., 2021; Chen et al., 2020a), our CR2PQ could obtain more gains than using MIM- based (He et al., 2021; Zhou et al., 2022) teacher models. We guess that's because contrastive- based methods usually lear global- discriminative information, but lose local information. When using these contrastive- based methods, our CR2PQ could enhance the ability to capture the local information. In contrast, MIM-hased
8

---

Published as a conference paper at ICLR 2025
Table 5: Ablation on projector head. We pre-train all the models with 400 epochs and 2048 batch size on ImageNet-1k IK. We compare different architectures of heads. "Share weights" means the global and local projectors share the same weights. "Trans" means multi-head self-attention decoder.
Architecture
Share weights
Giooaal projector
Local projector
Top-1
Top-5
Framework
mAp
mAp
2-layer MLP
2-layer MLP
83.3
96.2
52.0
45.1
3-layer MI.P
3-layer MI.P
83.2
96.1
52.4
45.3
VIT-B/16
2-layer MLP
2-layer MLP
83.4
96.2
Cascade-RCNN
52.0
45.1
X
3-layer MLP
2-layer MLP
83.2
96.1
52.1
45.0
X
2-layer MLP
3-layer MLP
83.5
96.4
52.5
45.1
2- 2-layrr MI.P
2-layer Trans
83.7
96.5
53.0
45.6
430
41.5
46.5
C000
COCO
41.
TirgMIM
40.5
650
4000
22.5
Lau
Loss
Figure 3: Finetuning classification ImageNet-1K 1K (left), object detection (middle) and instance segmentation (right) performance on COCO with different global and local objective weights A. All the experimental settings keep the same for fair comparisons.
methods usually add the patch-level objective, and the pre- ore-trained models more spatial-sensitive but less -ibbil-dserimiinativ.. Therefore, when using MIM- based methods as the teacher model, our CR2PQ may lose some global-diseriminative information, leading fewer improvements than using contrastive-basee teacher models. Besides, It's noteworthy that our CR2PQ w/o pretrained teacher (using pixels as the teacher) can still outperform previous SOTA method (Zhang al., 2023a) with a large range, which further demonstrates the effectiveness of the continuous coordinate-basee task.
Loss weight of global semantic learning. As CR2PQ mainly focuses on dense prediction tasks, we set the weight of the global objective À = 0.5 as the default. To better balance the global- discriminative and spatial-sensitive information, we further conduct a set of experiments by switching A from 0 ~ 1. Specifically, we pre-train ViT-S/16 300 epochs with 2048 batch size. Then, for detection and segmentaion, we finetune the pretrained model on COCO dataset with Mask RCNN framework. Fig. 3 illustrate the funetuning classification accuracy on ImageNet, mAPb and map results on COCO with different global loss weight A. We find with larger X, CR2PQ obtains higher fine-tune accuracy on ImageNet-IK. However, correspondingly, large À makes CR2PQ learn less spatial information, resulting mAP score drops on the COCO dataset. We also find when setting À = O, the detection score is also lower than À 0.5, guess that's because too small À makes CR2PQ information, where detection tasks also require the discriminative
fail to learn global-discriminant information to help classify the object in classification head layer.
Projector head of global semantic learning. To further study the effect of the local and global projector head, we try different combinations of the two modules. Following SimSiam (Chen & He, 2021) and Barlow Twins (Zbontar et al., 2021), we mainly attempt to use 2-layer and 3-layer MLP and 2-layer transformer blocks as projector heads. Since the global projector inputs one global token, we only add the transformer block on the local projector module. Table 5 shows fine-tuned classification, detection and instance segmentation results on ImageNet- 1K and COCO datasets with different architectures. Different from previous contrastive methods (Chen & He, 2021; Zbontar et al., 2021), we find that lightweight global projectors can bring better classification accuracy than complex architectures, while employing two-layer transformer layer as local projector can bring more gains than simply using two-layer and three-layer MLP.
Pretraining Strategy. To show the effectiveness of our continuous rotary positional embedding, we conduct a set of experiments, including three types of positional embeddings, i.e., continuous sin-cos embedding, discrete rotary positional embedding (RoPE), learnable relative positional embeddings and discrete sin-cos embeddings. More concretely, for sin-cos embedding, we first calculate the rela- tive positional index proposed in Eq. 4. Then, for each index, we calculate the sin-cos embedding (He
9

---

Published as a conference paper at ICLR 2025
900
47.5
Counde BCNN axd
45.0
42.3
400
12
13
14
POCL peo Rcce)
F0OT (e7)04))
36
POCL (er (nal))
POCL/W lool)
PUCLL/ensten
280 (h00))
(R2PQ)0ua)
(RIPQ)
25
40
330
175
156
175
200
190
200
300
Epect
Tpecks
Fpecks
Figure 4: Finetuning Cascade-RCNN and Mask-RCNN (right) on detection (left) and segmentation (middle) on COCO dataset. We report the mAp6 and mAp performance for each 40 epochs.
al., 2021) through the given relative indices of each patch. For discrete positional embeddings, we perform pretraining in MAE, since the discrete position can not represent the relative positions of two views. Therefore, for each image, randomly mask the given image with 75% masking ratio. Then, we use the remain part to reconstruct the masked part. For learnable positional embedding. first calculate the relative positional index proposed in Eq. 4. Then, we use two-layer MIPP to map the tuple (h, w) to Doodel dimension as the positional embedding. For discrete sin-cos, we also adopt the MAE pretraining and the model degenerates MAE (He al., 2021) (replace the RGB supervision with teacher embeddings). Table 6 shows the classification and detection results on ImageNet and COCO, where our CR2PQ significantly outperforms peer methods.
Convergence and training efficiency. To ex- plore the convergence of the proposed CR2PQ,
Table 6: Ablation on different pretraining strat- egy. We pre-train ViT-S/16 with 300 epochs on
we conduct a group of experiments with differ- ent pretraining epochs. Fig. 4 shows the de- tection and segmentation results the COCO dataset with different pretraining epochs us- ing Cascade-RCNN and Mask-RCNN frame- works. We surprisingly find with Cascade- RCNN, CR2PQ can achieve 47.2 mAP*b
ImageNet-IK IK and fine-tune on other datasets.
Student
Positional embedding
Epoch
Acc
mApes
Continuous RoPE
82.2
50.5
Discrete RoPE
81.5
48.2
VIT-S/16
Continuous Sin-cos
300
81.9
47.8 (-2.7)
Learnable
81.3
46.9 (:3.6)
Discrete Sin-cos
81.1
45.9 (-4.6)
and 41.1 mApmk score with only 40 epochs pretraining, which obtains 10.4 improvements on mApbb
and 7.9 improvements mApmk scores detection and segmentation tasks, respectively. Besides, also find with local loss, PQCL converges slower than without using the local loss (when 40 epochs pretraining, PQCL (w/o local gets better results than w/ local)). However, with 200 epochs pretraining, PQCL (w/ local) outperforms PQCL (w/o local) by 0.7 mAp46 and 1.7 mAAmm scores. We guess that's because the patch-level loss could impede the model from learning global-discne disbbaldiscri infor- mation, which slows down the convergence rate. However, the patch-level loss also makes the model learn spatial- -sntil--enssiive information, resulting better performance on detection and segmentation under long epochs pretraining. In contrast, CR2PQ enjoys fast convergence, and it outperforms previous SOTA by a large range with only40 epochs pretraining.
CONCLUSION
In this paper, we propose CR2PQ to learn dense representations through continuous relative rotary positional embedding. Different from prior 2D RoPE works, we extend the discrete RoPE to continuous and demonstrate its effectiveness in learning dense representations from comprehensive experiments. Then, we conduct exhaustive ablation studies demonstrate the robustness of the proposed CR2PQ, where our method achieves new SOTA results on detection and segmentation tasks, outperforming previous dense contrastive learning SOTA with large range. Besides, our CR2PQ also achieves comparable results classification with previous methods. Finally, demonstrate the effectiveness of the proposed continuous RoPE through different positional embedding types.
While CR2PQ achieves state-of-thed -htt---rttt results, there are limitations to consider for future exploration: Sensitivity to Background-heand Views: The current method relies on randomly cropped views for contrastive learning (or distillation.. In some cases, these views might contain mostly background with little foreground content. This can lead the model to learn irrelevant information that hinders performance various downstream tasks focused the foreground objects.
Limitations.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianee-ivarranc---vvvaaeee oatp://aaaaaeatotaaaaaaaaerrrrar for superviee learning. In ICLR, 2022.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NewrIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. 2019.
arXiv preprint arXiv: 1906.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang. Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representationn learning. arXiv preprint arXiv:2202.03026 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint trXiv:2003.04297 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
MMSegmentation Contributors. MMSegmennatiin: Openmmlab semantic segmentation toolbox and
benchmark. https://githhei com/ caaaa/openmmiit ab/ http///mmegmeet... Lon, 2020.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. NeurIPS, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2020.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training. arXiv preprint arXiv:2202.03388) 2022.
Yuxin Fang, Quan Sun, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303. 1331, 2023.
Chongjian Ge, Youwei Liang, Yibing Song, Jianbo Jiao, Jue Wang, and Ping Luo. Revitalizing cnn attention via transformers in self-supervised visual representation learning. NewrIPS, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 080300772, 2018
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 27, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised leaming. NewrlPS, 2020.
11

---

Published as a conference paper at ICLR 2025
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representation learning. NeurIPS, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
Pranav Jeevan and Amit Sethi. Resource-efficient hybrid x-formers for vision. In WACV, 2022.
Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In ECCV, 2018.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022.
Tsung-Y Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1608 (00.00830..383, 2016
Ily Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision. language, audio, and action. arXiv preprint arXiv:2312.17117 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. 2018
Utku Ozbulak, Hyun Jung Lee, Beril Boga, Esla Timothy Anzaku, Ho-min Park, Amout Van Messem, Wesley De Neve, and Joris Vankerschaver. Know your self-supervised learning: A survey image-based generative and discriminative training. TMLR, 2023.
Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. NeurIPS, 2020.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024.
Chenxin Tao, Xizhou Zhu, Weijie Su, Gao Huang, Bin Li, Jie Zhou, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning, 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. ECCV, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? NeurIPS, 2020b.
Hugo Touvron, Matthieu Cord, Matthijs Douze. Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoina Manzagol. Extracting and composing robust features with denoising autoencoders. In ICMI, 2008
12

---

Published as a conference paper at ICLR 2025
Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen. On the importance of asymmetry for siamese representation learning. In CVPR, 2022a.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021.
Zhaoqing Wang, Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang, Mingming Gong, and Tongliang Liu. Exploring set similarity for dense self-superviser representation learning. In CVPR, 2022b.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yure Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:21 arXiv:2/ /2.09133, 2021.
Téte Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. ICCV, 2021.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In ICCV, 2021a.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self-supervised learning with swin transformers. arXiv preprint arXiv:2105.0453 04553,, 2021b.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In CVPR, 2021c
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. CVPR, 2022.
Ceyuan Yang, Zhirong Wu, Bolei Zhou, and Stephen Lin. Instance localization for self-supervised detection pretraining. In CVPR, 2021.
Sukmin Yun, Hankook Lee, Jachyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In CVPR, 2022.
Jure Zhontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. In ECCV, 2016.
Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-cl: Instance and feature decorrelation for fegetive-free symmetric contrastive learning. In /CLR, 2021.
Shaofeng Zhang, Lyn Qiu, Feng Zhu, Junchi Yan, Hengrui Zhang, Rui Zhao Hongyang Li, and Xiaokang Yang. Align representations with base: A new approach to self-supervised learning. In CVPR, 2022.
Shaofeng Zhang, Qiang Zhou, Zhibin Wang, Fan Wang, and Junchi Yan. Patch-level contrastive learning via positional query for visual pretraining. In ICML, 2023a.
Shaofeng Zhang, Feng Zhu, Rui Zhao, and Junchi Yan. Patch-level contrasting without patch correspondence for accurate and dense contrastive representation learning. In ICLR, 2023b.
Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre- pre-training with online tokenizer. In JCLR, 2022.
Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In CVPR, 2022.
13

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION AND EXPERIMENT SETUPS
Baselines. We consider recent advanced self-supervised methods based on the ResNets (He et al., 2016) and ViTs (Dosovitskiy et al., 2020) architectures: (a) self-supervised ResNets: SimCLR (Chen et al., 2020a), MoCo-v2 (Chen et al., 2020b), SwAV (Caron et al., 2020), Barlow Twins (Zbontar al., 2021), 1), ZeroCL (Zhang et al., 2021), ARB (Zhang al., 2022), DenseCL (Wang et al., 2021), ReSim (Xiao et al., 2021), and DetCo (Xie et al., 2021a); and (b) self- self-suue ViTs: DINO (Caron al., 2021), MoCo-v3 (Chen et al., 2021), MoBY (Xie et al., 2021b), iBOT (Zhou et al., 2022), SelfPatch (Yun et al., 2022), TinyMIM (Ozbulak et al., 2023).
Code of computing relative coordinates. Code shows the implementation of how to compute relative coordinates matrix of rp
def factorial (p A, P_B, A,,, :
P_A and P_B are given line 199 of this paper. # AA,, K_ are given in Eq.3 in this paper. width
"_per_grid_A P_A[3] / K_A
wigrid_bias (p_5[1] P_A[1]) / -_per_grid
w_per_grid_B P_B[3] / K_B
A.grid_scale = __per_grid__ / w_per_grid_A
height
10
h_per_grid_A P_A[2] / K_A
h_grid_bias (p_5[0] P_A[0]) / h_per_grid__
12
h_per_grid_B = P_5[2] K_B
13
h_grid_scale h_per_grid_B h_per_grid_
14
# compute coordinate matrix
15
staa,,, h_end h_grid K_B h h_gridd_calle w w_start, w_end = w_grid_bias + K_B w_grid_scale
17
grid_h torchharrageesstarttt___._
end=h_end, step=
h_grid_scale)
18
grid_w (srchh. end step= #_grid_scale
end=w_end,
19 grid torch. meshgrid hrid_ return grid
Listing 1: Computing Relative Coordinates
B EXPERIMENT ON LARGE-SCALE MODEL
To explore the scalability of the proposed CR2PQ, we conduct experiments with ViT-L, and evaluate the pretrained ViT-Base on ViTDet (Li al., 2022) detector. Specifically, we pre-train the ViT-Large with 800 epochs with batch size 2048, distributed on 16 A 100 GPUs with the base learning rate 1.5c-4. Table shows the results on ImageNet- IK using VIT-L backbone, where our CR2PQ can consistently outperform previous basclines on both 400 and 800 epochs pretraining, Table shows the results of the COCO dataset when using ViT-B as the backbone and ViTDet as the detector head.
Method
Architecture
Epoch
Acc@1
Method
mAp
mApmt
DINO (Caron et al., 2021)
ResNet-50
400
77.4
(Chen et al., 2021)
VIT-J/16
Scratch
48.1
42.6
MAE (He et al., 2021)
VIT-L/16
400
84.3
MAE (He et al., 2021)
51.1
45.6
CR2PQ (Teacher ResNer-50 DINO)
VIT-1/16
400
84.6
(Caron al., 2021)
49.0
MAE (He et al., 2021)
VIT-L/16
84.6
CR2PQ (Ours)
52.2
46.5
iBOT (Zhou
2022)
VIT-L/16
84.8
CR2PQ (Teacher ResNet-50 DINO)
VIT-L/16
800
85.3
Table 8:
Object detection and in- stance segmentation results on COCO
Table Finetuning classification results on ImageNet-I dataset using ViT-L.
datasets using ViT-Base and ViTDet.
14


Reviews:
[{"review_id": "beTsDW0Em4", "reviewer": "Reviewer_qBT6", "summary": "The paper introduces the Continuous Relative Rotary Positional Query to enhance dense visual contrastive learning by improving pixel/patch correspondence across different views. It addresses limitations in existing self-contrasting methods by transforming discrete positional embeddings into continuous representations. The proposed CR2PQ enables more effective patch-level representation learning, achieving state-of-the-art results and faster convergence in detection and segmentation tasks on the COCO dataset.", "questions": "Please refer to the weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Writing quality is good. The paper is well-structured, and clearly written.\n2. SOTA performance. The paper demonstrates the state-of-the-art performance on mainstream detection and segmentation datasets, such as COCO and ADE20K, which is impressive.\n3. Versatility of the method. The paper shows the simplicity of CR2PQ, which can be easily integrated into a variety of popular representation learning frameworks, such as mask-based learning, contrastive learning, and distillation methods.", "weaknesses": "1. Reliance on random cropping. Although random cropping can increase the variability of the input, its results may still be limited by the randomness of the cropping. In extreme cases, it may result in almost no overlap between the generated views, affecting the learning effect of the model.\n2. Computational complexity. Complex matrix operations are required when calculating relative position embedding and rotating embedding, which increases the burden in scenarios with limited computing power.\n\nP.S. There is an error in Figure 1. [CLS] should be global information, while patch is local information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "acjpBujscq", "reviewer": "Reviewer_p4vE", "summary": "The authors propose a distillation technique where a student is densely trained to match teacher features. The novelty comes from using 2D RoPE in the network as well as a cross-attention module with relative positional information. They show good empirical results on detection and segmentation.", "questions": "-Why use a pretraining network for the teacher? You are comparing with other baselines which some of which learn everything from scratch. This seems like a logical thing to try, have you tried that?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "-The empirical results are good and outperform previous SOTA.\n\n-I think this paper can be worthwhile to accept, I'm willing to improve my score based on the author's reply.", "weaknesses": "-L142: Relative positional encoding = RoPE?\n\n-L161: W_{pos} v.s. P_{pos} ?\n\n-The notation in equation 1 is confusing. It is as if the patches don’t interact with each other. I would use a new variable to define a patch representation. Also if f_\\theta denotes the ViT, why does it take z as input, which already contains the linear layer on the left side of the equation but not on the right side. I think the notation should be made more precise.\n\n-Equation 2 has some n and m mixed.\n\n-L219: “we set each patch size of the view A as 1”, but in L227 p_A (the patch size) is defined?\n\n-L228: There is a sentence “Since we set each grid size of the anchor view as 1.” What is that supposed to mean?\n\n-L297: If I’m not mistaken, the definition of q doesn’t make sense.\n\n-The first stated contribution is using 2D RoPE for SSL based methods. Then, in L358, shoud state “We also evaluate the detection and segmentation without pretraining i.e. directly using 2D RoPE”. First, that entry is only in Table 1 and not Table 2. Second, I think you should also independently show empirical evidence of your 2 first contributions (2D RoPE and cross-attention module) and report results for that.\n\n-In general, I think the paper could be more explicitaly precise with how sizes/positions are encoded e.g. is it relative to the original image input grid or relative to the crop?\n\n\n\nMinor:\n\n-L082: “as the downstream task only input”\n\n-If I’m not mistaken, there is a problem with sentence at L203 starting with i.e.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F6c07VovwY", "reviewer": "Reviewer_iV1q", "summary": "1. The paper introduces Continuous Relative Rotary Positional Query (CR2PQ), a novel method for dense visual representation learning.\nCR2PQ addresses the challenge of establishing pixel/patch correspondence across different views in dense contrastive learning (DRL) by transforming discrete positional embeddings to continuous representations.\n\n2. It utilizes a rotary positional embedding to represent the relative positions between two views and reconstructs the latent representations of one view from another through a rotary positional query.\n\n3. The method simplifies the dense contrastive learning paradigm by making it correspondence-free and integrates easily into various representation learning frameworks.\n\n4. Extensive experiments on standard datasets demonstrate state-of-the-art (SOTA) results, outperforming the previous SOTA method (PQCL) significantly in detection and segmentation tasks on COCO with improved mAP scores.", "questions": "1. What is the performance of the CR2PQ backbone performance on some strong detectors, such as DINO or Co-DETR?\n\n2. CR2PQ requires the teacher model to provide contrastive pairs, however, the performance does not improve as the model becomes larger (ViT-L vs ResNet50). The reviewer wonders about the performance of a larger model for the student. Does this approach work for a larger backbone as a student, such as ViT-L/ViT-G? The authors are suggested to validate the scalability of the method.\n\n3. Some small mistakes\n\n- The font of the paper is different from other papers. Should it be correct? \n\n- line 274, there is an overlap between the table and the caption.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. CR2PQ introduces a pioneering method for dense visual representation learning by utilizing continuous relative rotary positional embeddings, which is a significant departure from traditional discrete embeddings.\n\n2. The method achieves state-of-the-art results across various benchmarks, including object detection and segmentation tasks on COCO and semantic segmentation on ADE20K, outperforming previous leading methods by a considerable margin.\n\n3. The introduction of a positional-aware cross attention module enhances the learning of semantic information without incurring significant additional computational costs. CR2PQ's use of rotary positional embeddings makes it robust to various view augmentations, including random cropping, which is a common challenge in contrastive learning methods.\n\n4. The paper supports the method's strengths through extensive experiments and ablation studies, providing a thorough analysis of CR2PQ's performance under different conditions and configurations.", "weaknesses": "1. Experiments. The author should provide more scales of backbone to validate the scalability of the method. Most experiments are conducted on ViT-S. The reviewer understands the efficiency of the experiments, however, there should be some experiments on larger backbones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Wzom0ekHN", "reviewer": "Reviewer_MHTX", "summary": "The paper presents a novel self-supervised framework for dense visual representation learning, which avoids the need for explicit dense correspondences between local features across views. Instead, the framework reframes the task as predicting local representations from one view to another, guided by relative positional cues. It integrates rotary positional embeddings within the student model and distills knowledge from a pre-trained, frozen teacher model. This approach yields faster convergence and improved performance on standard benchmark evaluations.", "questions": "- In Table 1, what does the row \"RoPE\" exactly correspond to? A ViT-S/16 equipped with rotary positional embedding, randomly initialized and finetuned on the downstream task?\n\n- In Table 4, what does the row \"EMA update (Contrastive)\" exactly correspond to? Is the teacher randomly initialized?\n\n- At line 219. it is mentioned that the patch size of view A is set to 1, but then it is set to $p_{A}$. Can you clarify this?\n\n- At line 227: I suggest using another notation for $p_{A}$ as the patch size, as it is confusing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed self-supervised framework for dense visual representation learning is novel.\n- The method elegantly eliminates the need to establish explicit correspondence between local features across views by leveraging relative positional cues.\n- The performance on dense downstream tasks is thoroughly evaluated, showing faster convergence and achieving state-of-the-art results on standard benchmarks.", "weaknesses": "- The method differs from existing baselines in three key ways: (1) the use of rotary positional embeddings, (2) the use of a pre-trained, frozen teacher model, and (3) the proposed pretext task. This makes it challenging to assess the contribution of each component to the overall performance. Specifically, the fairness of the experimental setup is questionable, as other methods are trained from scratch while CR2PQ benefits from a pre-trained teacher. More ablation studies are needed to separate the impact of each element.\n\n- Overall, the writing is difficult to follow, with multiple notation inconsistencies, typos, and signs of negative vertical spacing used to fit within the page limit.\n\n- Equation 1 is misleading/incorrect as it suggests that the representation of a single patch is independent of its context.\n- Equation 2: The angle of the key seems incorrect.\n- Line 210: The image dimensions are inconsistent with line 157.\n- Line 214: Inconsistent use of $\\mathbf{p}{a}$ and $\\mathbf{p}{A}$.\n- Line 234: The notation is inconsistent with the left side of Equation 3.\n- Table 1: Framwork $\\rightarrow$ framework.\n- Figure 1: There seem to be inconsistencies in the notations used within the figure and also with respect to the method section.\n- \"pertaining\" $\\rightarrow$ \"pretraining\"/\"pre-training\" (11 occurrences).\n- Line 86: exhausted $\\rightarrow$ exhaustive.\n- Line 161: $\\mathbf{W}{pos}$ $\\rightarrow$ $\\mathbf{P}^{i}{pos}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)