Patch-Level Contrasting without Patch Correspondence for Accurate and Dense Contrastive Representation Learning
Authors: Shaofeng Zhang
Feng Zhu
Rui Zhao
Junchi Yan
Venue: ICLR 2023 poster (2023)
Keywords: self supervised learning
contrastive learning

Abstract: We propose ADCLR: \underline{A}ccurate and \underline{D}ense \underline{C}ontrastive \underline{R}epresentation \underline{L}earning, a novel self-supervised learning framework for learning accurate and dense vision representation. To extract spatial-sensitive information, ADCLR introduces query patches for contrasting  in addition with global contrasting. Compared with previous dense contrasting methods, ADCLR mainly enjoys three merits: i) achieving both global-discriminative and spatial-sensitive representation, ii) model-efficient (no extra parameters in addition to the global contrasting baseline), and iii) correspondence-free and thus simpler to implement. Our approach achieves new state-of-the-art performance for contrastive methods. On classification tasks, for ViT-S, ADCLR achieves 78.1\% top-1 accuracy on ImageNet with linear probing, outperforming our baseline (DINO) without our devised techniques as plug-in, by 1.1\%. For ViT-B, ADCLR achieves 79.8\%, 84.0\% accuracy on ImageNet by linear probing and finetune, outperforming DINO by 0.6\%, 0.4\% accuracy. For dense tasks, on MS-COCO, ADCLR achieves significant improvements of 44.3\% AP on object detection, 39.7\% AP on instance segmentation, outperforming previous SOTA method SelfPatch by 2.2\% and 1.2\%, respectively. On ADE20K, ADCLR outperforms SelfPatch by 1.0\% mIoU, 1.2\% mAcc on the segmentation task.

Published as a conference paper at ICLR 2023
PATCH-LEVEL CONTRASTING WITHOUT PATCH COR- RESPONDENCE FOR ACCURATE AND DENSE CON- TRASTIVE REPRESENTATION LEARNING
Shaofeng Zhang- 122, Feng Zhu", Rui Zhao 1.3, Junchi Yan
1 Moo Key Lab of Artificial Intelligence, Shanghai Jiao Tong University
Shanghai AI Laboratory
"SenseTime Group Limited
(sherrylone, yanjunchi ( jtu. enu.cn
(zhufeng, zhaoruá] @senset ime com
Code: attps://githe com/ shorryllooc/uuucyoooottt
ABSTRACT
We propose ADCLR: Accurate and Dense Contrastive Representation Learning, a novel self-supervised learning framework for learning accurate and dense vi- sion representation. To extract spatial-sensitvv information, ADCLR introduces query patches for contrasting in addition with global contrasting. Compared with previous dense contrasting methods, ADCLR mainly enjoys three merits: i) achieving both global-diseriminatiee and spatial-sensitin representation, ii) model- efficient (no extra parameters in addition to the global contrasting baseline), and iii) correspondence-fiee and thus simpler to implement. Our approach achieves new state-of-the-an performance for contrastive methods. On classification tasks, for ViT-S, ADCLR achieves 77.5% top-1 accuracy ImageNet with linear probing, outperforming our baseline (DINO) without our devised techniques as plug-in, by 0.5%. For VīT-B, ADCLR achieves 79.8%, 84.0% accuracy on ImageNet by linear probing and finetune, outperforming iBOT by 0.3%, 0.2% accuracy. For dense tasks, on MS-COCO, ADCLR achieves significant improvements of 44.3% AP on object detection, 39.7% AP on instance segmentation, outperforming previous SOTA method SelfPatch by 2.2% and 1.2%, respectively. On ADE20K, ADCLR outperforms SelfPatch by 1.0% mloU, 1.2% mAce on the segmentation task.
INTRODUCTION
Self-supervised representation learning (SSL) has been attracting increasing attention for deep learning, whereby a prediction problem is often formulated by a pretext task for pre-training with unlabeled data. SSL methods can mainly be divided into three categories: 1) Generative approaches (Goodfellow et al., 2014) learn generate samples in the input space. However, generation can be computationally expensive and may not be necessary for representation learning. 2) Contextual methods (Gidaris et al., 2018) design pretext tasks (denoising auto-encoders (Vincent al., 2008), context auto encoders (Zhang al., 2016), etc). 3) Contrastive methods (Jin et al., 2022; Zhang et al., 2022; Chen al., 2021; Caron al., 2021) take augmented views of the same image as positive pairs and others negative pairs. cnatrasttveebaseee methods have shown great promise e.g. in image classification/detectinn video classification (Caron et al., 2021). and others (Chen et al., 2021).
It has been recently shown (Chen al., 2020a; Wang et al., 2021) that existing contrastive learning in general aims to learn global-diseriminative features, which may lack spatial sensitivity (Yi et al., 2022), and it limits their ability downstream fine-tuning tasks, especially for dense vision tasks like detection and segmentation. Consequently, object-level (Wei et al., 2021; Hénaff et al., 2022) and pixel-level (Xie et al., 2021c; Wang al., 2021) contrastive objectives and frameworks proposed. Meanwhile, with the success of recent ViT-based visual backbones (Dosovitskiy et al., 2020; Liu al., 2021), patch-level contrastive approaches (Yun al., 2022) are devised, which
Junchi Yan is the correspondence author. Rui Zhao is also with Qing Yuan Research Institute, Shanghai Jiao Tong University. This work was in part supported by NSFC (62222607), Shanghai Municipal Science and Technology Major Project
and SenseTime Collaborative Research Grant.

---

Published as a conference paper at ICLR 2023
achieve ffate-of-the-at performance downstream dense tasks. However, there are mainly three disadvantages in these dense contrasting methods. i) It is hard to balance the global and patch- level losses in the dense contrasting methods (Xie et al., 2021c; Wang et al., 2021), causing their less competitive tunnee accuracy on the global classification task. ii) Establishing the correspondence among pixels/patches usually requires bilinear interpolation, which is complex and heavily sensitive to random crop augmentation (in an extreme case, if two views have no intersection parts, there's no correspondence relation). fifi each corresponding pixel/patch need be involved in the final contrastive loss, which is time-consuming and Dense Contrastive Representation Learning (ADCLR), which is more dobbal-iicrimmiatiie. spatial- correspondence-free and efficient. The main contributions of ADCLR are:
In this paper, we propose Accurate
1) Cross-view Query-based Patch-level Contrasting Paradigm:
For patch-level contrasting as recently used for dense tasks in negative-free Transformer-based SSL methods, we propose to augment two views, and perform contrasting crops from different views, for more effective learning with increased contrasting difficulty. The motivation for our cross-view design instead of the commonly- used single-view in existing patch-level contrasting is: it is non-triv and even impossible to establish patch correspondence within single view (especially adding random resized crop augmentation). While introducing two views and replacing the correspondence establishing with more feasible query could meanwhile increase the patch appearance variance for more difficult contrasting. The above module can be introduced to existing global contrasting only SSL baselines. As shown in Fig. 1, the [CLS] tokens are used extract global information, and the designed query patches are used to extract local information, making ADCLR both global- -oabal-dicrrmiiative and spatial-sensitive.
2) Robust Unidirectional Cross-Attention Scheme under the above Paradigm: The above patch- level contrasting paradigm can technically be prone to collapse to a trivial solution' (see more explanation in our theoretical analysis in Sec. 3.3) if we directly resort to the unidirectional self- attention scheme as used in the vanilla vision Transformers. Instead, we design unidirectional cross-attention, which takes both query patches and raw patches from raw images as input. For attention block, the data flow are: RP - RP. (RP,QP] QP,, QP, QP, (i + j) where RP and Q., means raw patches (including [CLS] token) and the i-th query patch, respectively.
3) Boosting baselines to new SOTA accuracy on classification and dense tasks: The proposed and negative-free SSL baselines e.g. DINO (Caron et al., 2021) and iBOT (Zhou et al., 2022). Our experimental results both linear probing, finetune classification as well as other downstream tasks show the effectiveness of ADCLR.
ADCLR can serve as a plugin based on existing Transformer-ban
2 RELATED WORKS
In this section, we review previous literature on general contrastive learning (for global tasks such as image classification) and dense ones (for downstream tasks like detection and segmentation).
General contrastive learning aims at learning the global information of images through enforcing contrastive objectives (e.g., InfoNCE loss, Hjelm et al. (2018)) on the final global image represen- tations (CCLS) token or from average pooling). The pivotal idea is to align the embeddings of two augmented views from the same image while preventing trivial solutions (a.k.a. degenerated represen- tations). To reach this target, MoCo (He et al., 2020; Chen et al., 2020b) employs a memory bank to store and update negative examples, from which the negative examples could be randomly sampled, whereas SimCLR (Chen et al., 2020a) treats all other data samples within the same training batch as negative examples. Due to the inconvenience and huge cost of using negative examples, later research turned to explore negative-example- free methods. BOL (Grill al., 2020) and SimSiam (Chen & He, 2021) design asymmetric architectures with additional predictors and sop-gradient to prevent using negative examples. Barlow Twins (Zbontar et al., 2021). I). ZeroCL (Zhang et al., 2021) and VICReg (Bardes et al., 2022) resort to feature-level decorrelation to mitigate trivial solutions. Inspired by the success of Vision Transformers (ViT), CNN backbones are gradually replaced with ViTs Chen et al. (2021); Caron et al. (2021). iBOT (Zhou et al., 2022) further incorporates contrastive learning with masked image modeling and has achieved of-tee performance on various tasks.
Dense contrastive learning, on the contrary, targets learning local information through regularizing different local regions of images. One common dense contrastive learning way to mine the correspondence of each pixel (CNNs) or patch (ViTs) in feature map (Wang et al., 2021; Yun et al.,
'The embedding vectors end up spanning a lower-dimensional subspace instead of the entire space.
2

---

Published as a conference paper at ICLR 2023
2022). DenseCL (Wang al., 2021) exploits the correspondence through sorting the similarities of pixels, while PixPro (Xie et al., 2021c) utilizes the augmentation wrapper to get the spatial correspondence of the pixel intersection between two views in the feature map. Furthermore, Detco (Xie et al., 2021a) tries to improve the performance of general contrastive learning approaches by augmenting multiple global and local views simultaneously. Inspired by PixPro, Resim (Xiao et al., 2021) uses Precise Rol Pooling (Jiang et al., 2018) to extract a feature vector from the associated feature map region for both views. On the basis of DenseCL, SetSim (Wang et al., 2022) employs a threshold selection to filter out noisy backgrounds. With the development of VIT in SSL (Chen al., 2021; Caron et al., 2021), SelfPatch (Yun et al., 2022) treats the spatial neighbors of the patch as positive examples for learning more semantically meaningful relations among patches.
3 METHODOLOGY
We first review recent SSL approaches with ViTs as backbones (Caron et al., 2021; Yun et al., 2022) in Sec. 3.1. Then we present ADCLR detailedly in Sec. 3.2 (we provide sketch in Fig. 1).
3.1 PRELIMINARIES
Vision Transformers. Denote an image by x E RCCHWW where x W is the resolution of the image and C' is the number of channels. Plain VIT (Dosovitskiy et al., 2020) treats the image x as sequence composed of non-overlapping patches fx(9 E RRP2)
fē=1> | where each patch has fixed P x P resolution. Then, the patches are linearly transformed to D-dimensional patch embeddings (+) Ex + W pos E RDD where E E T DxCP2 is the linear projection and W pos E IR is the positional embedding for the i-th patch. A [CLS] token (CCLS) E. is subsequently prepended to the patch sequence to extract global information, so the resulting input sequence is represented as
z = [z. CCLL, (1),, (2))
z (. Then, ViT uses a Transformer encoder (Vaswani et al., 2017) to generate both image-level ([CLS) token) and patch-level (other tokens) In line with SelfPatch (Yun et al., 2022), we use fo to denote the whole process of a ViT parameterized by 0:
fo(x) = fo ([2ICLS)
(1) (2))
z 1 = JOLS (x).f) (x), ((2)
((1)
(x), (x),
(N) (x))
(x)
where / p (x) and f/ (x) are the representations of the whole image and i-th patch, respectively.
Self-supervised learning with ViTs. Since our ADCLR is built on top of DINO and iBOT, shortly review DINO's framework and objective function. Given the image x. DINO constructs positive pair (xA, xB) through random augmentation. Then, DINO learns by maximizing the similarity between their representations. The generic form of image-level self-supervised loss is:
LINOO H (1r (1! (J(CT(99(A)) (xa)): .g (1 ICTS (xn)))) (xn)))
where ((a, aa log b is the cross entropy loss. sg(.) means stop-gradient operation. 9- is the MLP projector, which is commonly used in previous SSL methods (Chen et al., 2020a; Grill et al., 2020). 7 and 0' denote the exponential moving averages updated parameters in the teacher branch.
3.2 ADCLR: ACCURATE AND DENSE CONTRASTIVE REPRESENTATION LEARNING
Query crops. Although DINO is able to handle most cases (c.g., two augmented views contain the same objects), find that the [CLS] token may extract mismatched information (see the attention map of (CLS) in Fig. 1). To this end, we randomly crop each image to generate Q query crops, where is pre-defined hyper- parameer.. Then, we resize each query crop to the same resolution as raw patches (e.g.. 16 16 or 8 8): X R Denote the i-th query crop as x", we feed each query patch into a linear projector get its embedding z(9.), Different from raw patches, it's unnecessary to add positional embeddings to query patches as the region of the query crop may not in the augmented views. Then, the input embeddings can be formulated as:
z CCSS z (1)
(()) (2)
z (N)
z (11) (oz)
(92)
patches
query patches
3

---

Published as a conference paper at ICLR 2023
Embeekiings
Attention mup
Attention maap
mup
former Encoder
CLE Teken
Takee
Augmestation
Chop
CLS
JEMA
Quers Lg
Giobal Gu
Mismaiched
Matched
Matched
Asgmeriation
Beal
Root
U:S5
CLS
Embodkings
Figure I: Framework of the proposed ADCLR (left figure). Here set the number of patches set as 2 for better illustration. For each query patch, we add the contrastive loss Lq. The stop-gradien and momentum update operations added to prevent collapse. The right figure illustrates the average attention map of all the heads the last block. "Matched" means objects are from the same classes.
Note that (CLS) and raw patches (2), (N) will be equipped with positional embeddings.
Unidirectional cross attention mechanism. Instead directly feeding whole sequence (namely [CLS] token, raw patches, query patches) the Transformer encoder, which may hurt the perfor- mance of the downstream task (as only [CLS] token and raw patches are fed to the Transformer for downstream tasks), we slightly modify the attention mechanism. Given one [CCLS) token, raw patches sequence [z(1), (22)
[N]] and query patches sequence (2(mx), z((2) ... , 2z42)). Let Iz. CT.S) z(" z(2), z [M) and z" [z(m), 2(22), z (02)]. For z (CTS and raw patches, the propagation rule in the attention block is in line with plain Transformer, i.e.,
(Ox ddk
Attn (Qu, Ky,, Vx) )- Softmax
Vx'
(4)
where Q₂ = z z' K z'Wk. V z z'Wv, and Wo. WK. Wv are learnable weights. For query patches z (4))), the attention propagation rule is similar to [CLS] token, i.e.,
QgiqniKe ddz
Attn(Qz(v)) Kz' Vzr) = Softmax
Vv, 1sisQ.
(5)
where Q2(ez) z (%) (W..Wa. Wq Q- By Eq. 4 and Eq. 5, we can complete the attention block in modified Transformer. In nutshell, we can summarize the propagation rule as RP -> RR. {RP,QP) QP:} QP,Q + QP; (i + j). where RP and QP, means raw patches (including [CLS] token) and i-th query patch, respectively. We can find only raw patches and [CLS]] token linearly express the raw patches and [CLS] in the attention mechanism. This design can make the pretrained Transformer more suitable for downstream tasks without introducing extra parameters.
Objective function of ADCLR. By feeding the whole sequence of two views X and Xg to the modified Transformer fo and MLP projection head 9ty, obtain the final representation of the image h |CLS| raw patches aa and query patches h% Then, the overall objectives are: A:B
LADOLR = Hhh
(CLS)
[CLS]
h/
ICLS]
(9:)
(6)
1=1
Global
where À used balance global and local loss. The impact is studied in Sec. 4.3. ADCLR can also selectively add the MIM objective in iBOT Zhou et al. (2022) to further boost the accuracy.
3.3 NECESSITY OF OUR UNIDIRECTIONAL CROSS-ATTENTIO MECHANISM
Theorem Bidirectional self-attention in standard vision Transformer leads to collapse. Given of query patches 9 and a set of raw patches (x) contrasting with the standard vision Transformer like backbones that use bidirectional self-attention mechanisms can cause ADCLR easily to collapse is large enough (Q >> the total number of patches of the image).
4

---

Published as a conference paper at ICLR 2023
Table Linear and finetune Top-1 accuracy on ImageNet-IK, means using DALL-E (Reddy et al., 1) tokenizer in the pre-training stage. $ means these works are so far arXiv preprints to date.
means adding the mask image modeling loss proposed in (Zhou al., 2022)
Framework
Method
Model
#Params
PT Eps.
Lincar
FT Eps.
FT Acc. (%)
Scratch, DeiT
VTT-B
86M
0
300
81.8
Training from scrach
Scratch, MAE
VIT-B
86M
300
82.3
Scratch, Swin
Swin-B
88M
300
83,5
Supervised Pre-training
Supervised. SIMMIM
Swin-B
88M
300
100
83,3
Supervised, SimMIM
Swin-L
197M
300
100
83.5
MAE
VIT-B
86M
1600
67.8
100
83.6
BETT"
VIT-B
86M
300
37.6
100
83.0
Masked Image Modeling
CAL
VIT-B
86M
300
64.2
100
83.3
CAB
VIT-B
86M
800
68.3
100
83.6
CIM
VIT-B
86M
300
100
83.3
Moco V3
VIT-B
86M
800
76.5
100
83.2
DINO
VIT-B
86M
400
78.2
100
83.6
Contrastive Leanring
BOT
VIT-B
86M
400
79.5
83.8
ADCLR
VIT-B
86M
400
78.6
100
83.9
ADCLR*
VITBB
86M
400
79.8
100
84.0
Remarks. In fact, applying the standard bidirectional self-attention as used in vision Transformers not only incurs the solution collapse issue shown in Theorem 1, but also may hurt the performance, as will be shown in the ablation studies in Sec. 4.3. We guess this is because ADCLR introduces additional query patches during the pretraining stage, while the downstream task finetuning only involves raw patches and the bidirctional scheme would cause the mutual influence between query and raw patches, leading to the imput inconsistency between pretraining and downstream tasks, Therefore. we elaborately replace the bidirectional self-attention scheme by our devised unidirectional cross- attention one as such the above two issues are addressed directly.
EXPERIMENTS
4.1 EXPERIMENT SETUP
Dataset. We conduct self- self-supervised pre-training on the ImageNet-IK (Deng et al., 2009) training set, as commonly used in SSL methods for both MIM (He et al., 2021) and contrastive learning (Chen al., 2020a). ImageNet- I includes 1,000 classes which are even in distribution. We also transfer the encoder pretrained by ADCLR on MS-COCO (Lin et al., 2014), ADE20K (Zhou et al., 2017) and other fine- -raingrrindd classification datasets (Van Horn al., 2018) for downstream evaluation.
Baselines. We consider a variety of existing SSL methods based on the ResNet (He et al., 2016) and VIT (Dosovitskiy et al., 2020) architectures: (a) self-supervised ResNets: MoCo-v2 (Chen et al., 2020b), SwAV (Caron et al., 2020), Barlow Twins (Zbontar et al., 2021), ZeroCL (Zhang et al., 2021), ARB (Zhang et al., 2022), DenseCL (Wang et al., 2021), ReSim (Xiao al., 2021), and DetCo (Xie et al., 2021 a); and (b) self-supervised ViTs: DINO (Caron et al., 2021), MoCo-v3 (Chen et al., 2021), MoBY (Xie et al., 2021b), iBOT (Zhou et al., 2022) and SelfPatch (Yun et al., 2022).
Evaluation protocols. Standard SSL protocols is to either learn a linear classifier on frozen fea- tures (Chen et al., 2020a; He et al., 2020) or to finetune on downstream tasks (He et al., 2021; Chen et al., 2022). For linear evaluations, for training, and report accuracy on a central crop. For finetuning evaluations (detection and segmen- tation on MS-COCO (Lin al., 2014), segmentation networks with the pretrained weights to adapt with further training. In line with (Zbontar et al., 2021). also evaluate our method's transfer ability on small-scale and fine-grained classification dataset.
apply random resize crops and horizontal flips augmentation
ADE20K (Zhou et al., 2017)), initialize
4.2
MAIN RESULTS
We first validate the ADCLR framework used this study with the standard self-supervised bench- mark on ImageNet. Then, we study the performance of detection, segmentation, and transfer learning.
Finetune on ImageNet-lk. Table shows the results with four different learning schemes, i.e., finetune from scratch, supervised pre-training. MIM-based approaches, and contrastive learning. We report linear probing and finetune accuracy, respectively. Our finetune protocol is line with iBOT (Zhou et al., 2022), using strong regularization for training 100 epochs. The compared baselines
5

---

Published as a conference paper at ICLR 2023
Table 2: K-NN and linear top-1 accuracy on ImageNet-Ik. Epoch means effective pre-training epochs accounting for actual trained images/views, which is used iBOT (Zhou et al., 2022).
means using MIM loss and # #Views means the number of views used in the pretraining stage.
Method
Model
#Params.
imagex/s
#Views
Epoch
k-NN
Linear
Supervised
RN50
23M
1237
300
79.3
79.3
SimCLR (Chen al. 2020s)
RN50
23M
1237
1000
60.1
69.1
Moco'2 (Chen al., 20206)
RN50
23M
1237
2
800
61.9
71.1
ARB (Zhang et al. 2022)
RN50
23M
1237
2
1000
66.4
73.1
CNNs
Barlow Twins (Zhontar ct 2021)
RN50
23M
1237
2
1000
66.0
73.2
BYOL (Grill ct al., 2020)
RN50
1237
1000
64.8
74.4
SwAV (Caron
2020)
RNSO
23M
1237
800
65.7
75.3
DINO (Caron
2021)
RN50
23M
1237
3200
67.5
75.3
Moco V3
(Chen al., 2021)
VIT-S/16
22M
1007
1200
73.4
Moco V3 (Chen et al., 1
VIT-B/16
86M
312
1200
76.7
SwAV (Caron et al. 2020)
VIT-S/16
22M
1007
8
2400
66.3
73.5
DINO (Caron (Caron et 2021)
VIT-S/16
22M
1007
12
3200
74.5
77.0
DINO (Caron al,, 2021)
VIT-B/16
S6M
312
1600
76.1
ViT&
iROT (Zhou al., 2022)
VIT-S/16
1007
3200
75.2
iBOT (Zhou al., 2022)
VIT-B/16
86M
312
1600
77.1
79.5
ADCLR
VIT-S/16
22M
1007
12
3200
74.6
77.5
ADCLR
VIT-B/16
86M
312
12
1600
76.6
78.6
ADCLR*
VIT-S/16
22M
1007
12
3200
74.9
78.1
ADCLR
VIT-B/16
86M
312
12
1600
77.4
79.8
Table 3: Object detection and instance segmentation on MS-COCO, Mask R-CNN adopted and trained with the 1x schedule. All the results based the same implementation for object detection and instance segmentation. Epoch refers to the number pretraining epochs on ImageNet- 1K.
Detection AP
Segmentation
Method
Backhone
Epoch
#Param.
ApM
AP
Apmn
AP
AP
Moco-V2 Voco-V2C0 (Chen 2020b)
RN50
200
23M
38.9
59.2
42.4
35,5
56.2
37.8
SwAV (Caron al., 2020)
RN50
200
23M
38.5
60.4
41.4
35,4
57.0
37.7
DenseCL (Wang et al., 2021)
RN50
200
23M
40.3
59.9
44.3
36.4
57.0
39.2
ReSim (Xiao et 2021)
RN50
200
23M
40.3
60.6
44.2
36.4
57.5
38.9
DetCo (Xie et 10)
RN50
200
23M
40.1
61.0
43.9
36.4
58.0
38.9
Moco V3 (Chen et al., 2021)
VIT-S/16
300
23M
39.8
62.6
43.1
37.1
59.6
39.2
MoBY (Xie et al., 20212)
VIT-S/16
300
22M
41.1
63.7
44.8
37.3
60.3
39.8
DINO (Caron al., 2021)
VIT-S/16
300
22M
40.8
63.4
44.2
37.3
59.9
39.5
SeliPatch (Yun et al., 2022)
VIT-S/16
200
22M
42.1
64.9
46.1
38.5
61.3
40.8
ADCLR
VIT-S/16
200
22M
43.8
65.2
47.4
39.2
61.9
41.3
ADCLR
VIT-S/16
300
22M
44.3
65.4
47.6
39.7
62.1
41.5
MIM-based methods MAE (He et al., 2021), BEiT (Bao et al., 2021), CIM (Fang et al., 2022)
as well as contrastive methods (Caron et al., ;021; Zhou et al., 2022; Chen et al., 2021) and the combination of the two techniques: CAE (Chen et al., 2022) which is emerging.
Linear probing on ImageNet-lk. For linear probing evaluation, we compare ADCLR with both CNNs-based and ViTs-based contrastive learning methods. Following iBOT (Zhou et al., 2022), use effective training epochs to measure the images actually seen by ADCLR. Specifically, DINO and iBOT by default trained with 2 global crops with the crop size 224 and 10 local crops with 96
the size 96*96. The calculation formula is r = 2 + 224 )2 x 10 = 3.84 = 4. Moreover, ADCLR
introduces query crops, and the total seen image is 2 96 ) 10
16 )2 x 10 3.89 x 4. 924
222
The overall seen image brought by query crops is only 0.05, which can be negligible. We report A-NN and linear probing accuracy in Table 2. ADCLR achieves 77.5% % top-1 accuracy
ImageNet- with ViT-S, outperforming baseline DINO +0.5% in 3200 effective epochs. For ViT-B, ADCLR outperforms DINO and iBOT +0.4% and +0.3% % accuracy in 800 effective epochs, respectively.
4.2.1 TRANSFER LEARNING ACROSS DIFFERENT TASKS
MS-COCO Setup. We evaluate pre-trained models on the MS-COCO object detection and instance segmentation tasks. Here, all models are fine-tuned with Mask R-CNN (He et al., 2017) and FPN (Lin et al., 2017) under the standard Ix schedule. Following (Chen et al., 2022), we utilize multi-scale training and resize the image with the size of the short side between 480 and 800 and the long side larger than 1333. In pretraining stage, we set À = 0.5 and Q = 10 for extract better dense information. The pretrining is in line with SelfPatch (Yun et al., 2022). In finetune stage, distributed training

---

Published as a conference paper at ICLR 2023
on 8 Tesla V100 32G GPUs set batch size 32 and set the learning rate 3e-4 with the layer decay rate 0.75, which also follows SelfPatch.
Results. Table 3 shows the detection and segmentation performance MS-COCO. We give the results of both 200 and 300 epochs' pretraining to better demonstrate the advance of ADCLR. We find that Although both SelfPatch and ADCLR perform patch-level contrastive loss, ADCLR breaks the limitation of contrasting between the same view. The learning paradigm of query tokens seems to bring more variance than single view contrasting, increasing the difficulty of patch-level pretext tasks, and resulting in higher performance on detection and segmentation tasks. Specifically, under 200 epochs pretraining, ADCLR outperforms SelfPatch with +1.7% and +0.7% points on detection (APh)) and segmentation (APt*) tasks, respectively. Compared with baseline DINO (Caron et al., 2021), ADCLR outperforms DINO with large range, e.g., +2.2% % bounding box average precision, +1.2% % mask average precision higher than DINO, which benefits from contrasting on query patches.
Table 4: ADE20K semantic segmentation performances of the recent self- supervised approaches pre-trained on ImageNet. The metries mIoU, aAcc, and mAce denote the mean intersection of union, all pixel accuracy, and mean class accuracy, respectively.
Semantic mentation ADE20K. Se- tups. We also evaluate seman- tic segmentation performances of
seg- on
Method
Arch
Backbone
#Iter.
mloU
aAcc
m/Acc
MoCo-v2 (Chen al., 2020b)
FPN
ResNet50
40k
35.8
77.6
45.1
SwAV (Caron et ul., 2020)
FPN
ResNet50
40k
35,4
77.5
44,9
DenseCL (Wang et al., 2021)
FPN
ResNet50
40k
37.2
78.5
47.1
pre-trained models
MocoV3 (Chen ct al., 2021)
FPN
VIT-S/16
40k
35.3
78.9
45.9
on ADE20K (Zhou al., 2017), which includes
MoBY (Xic al., 2021b)
FPN
VIT-S/16
40k
39.5
79.9
50.5
DINO (Caron et al., 2021)
FPN
VIT-S/16
40k
38.3
79.0
49.4
DINO (Caron et ul., 2021)
UperNet
VIT-S/16
160k
42.3
80.4
52.7
150 fine-grained
SelfPatch (Yun et al., 2022)
FPN
VIT-S/16
40k
41.2
80.7
52.1
SelfPatch (Yun et al., 2022)
UperNet
VIT-S/16
160k
43.2
81.5
53.9
semantic categories
and 25k training data. In line with SelfPatch (Yun et al., 2022), report three metrics: the mean intersection of union (mloU) averaged over all semantic categories, all pixel accuracy (aAcc), and mean class accuracy (mAcc). We pretrain ADCLR in 200 epochs and finetune the pretrained backbone in different architectures (FPN (Lin et al., 2017) and UperNet (Xiao et al., 2018)) for fair comparisons. Results. The results are given in Table 4. When finetuning the FPN with 40k iterations, ADCLR outperforms SelfPatch and DINO + +1.2% % and +3.9 % mIoU points, respectively. In UperNet with 160k iterations, ADCLR outperforms SeliPatch and DINO +1.0% and +3.0% mloU points. We guess the gain mainly comes from two aspects, 1) patch-level contrasting enhances the ability to capture local information (see the gap between global method DINO and local method SelfPatch and ADCLR); 2) The learning paradigm of query tokens brings more accurate information (the gap between SelfPatch and ADCLR).
ADCLR
FPN
VIT-S/16
40k
42.4
81.1
54,2
ADCLR
UperNet
VIT-S/16
160k
44.2
81.8
55.1
Transfer learning on the small-scale dataset. We study transfer learning where we pre-train on ImageNet- IK and fine-tune several smaller datasets. We follow the training recipe and protocol used in DINO and iBOT. For pretraining, VIT-S and ViT-B are pretrained on 16 GPUs with 1024 batch size and 10 local views in 800 and 400 epochs, respectively. The results are reported in Table 5. While the results on several datasets (c.g., CIFARIO, CIFAR100, Flowers, and Cars) have almost platcaued, ADCLR consistently performs favorably against other SSL frameworks, achieving state-of-the-an transfer results. Specifically, on small-scale datasets, ADCLR outperforms baseline DINO (Caron et al., 2021) over 2.2% and 1.1% accuracy iNaturalist18 8 and iNaturalist19 datasets. We observe ADCLR achieve greater performance gain over DINO in fine-grained classification datasets like INaturalist 8. iNaturalist19, and Flowers than CIFARIO and CIFAR 100. indicating ADCLR (query token) is more adept at capturing local information. Compared with the ffte-of-the-all method iBOT (Zhou et al.. 2022). although ADCLR achieves similar results, iBOT spends more memory due to the MIM branch (under the same setting, 19.5G for iBOT and 16.7G for ADCLR).
4.3 ABLATION STUDY
To verify the robustness of ADCLR, we comprehensively study the sensitiveness of balance ratio X, number of query tokens Q, the effect of cross attention mechanism, and crop ratio.
7

---

Published as a conference paper at ICLR 2023
Table 5: Top-1 linear accuracy on transfer learning, all models are pretrained on ImageNet- IK
Arch.
Method
CIFARIO
CIFARI00
iNaturalist18
iNaturalist19
Flowers
Cars
Rand.
99.0
89.5
70.7
76.6
98.2
92,1
BEiT (Bao et al. 2021)
98.6
87.4
68.5
76.5
96.4
92.1
VIT-S
iBOT (Zhou et al., 2022)
99.1
90.7
73.7
78.5
98.6
94.0
DINO (Caron et al., 1)2
99.0
90.5
72.0
78.2
98.5
93.0
ADCLR
99.1
90.8
74.0
78.5
98.6
94.2
Rand.
99.0
90.8
70.7
76.6
98.2
92.1
BEiT (Bao et al,, 2021)
99.0
90.1
72.3
79.2
98.0
94.2
VIT-B
iBOT (Zhou et al., 2022)
99.2
92.2
74,6
79.6
98.9
94.3
DINO (Caron et al., 202
99.1
91.7
72.6
78.6
98.8
93.0
ADCLR
99.2
92.0
74.8
79.7
98.9
94.6
692
130T
H801
ADCLA (Cur))
ADCLB 10und)
68.3
66.
38.5
18.2
1822
362
680
ADCLR didnectional
ADCLA (undrrdtione
02
63
Number query
Number query views
(a) Effect of
(b) Effect of query crops Q
Uni-/Bi- directional attention
Figure 2: Top-1 linear classification accuracy ImageNet- IK with different hyper- parameters.
Effect of A. We conduct ablation experiments on different balance ratios À to explore the effect of local objective Looat. Specifically, we set batch size as 1024, and distribute training on 32 GPUs with 30 epochs as warm up out of the total 100 epochs. For efficiency, we only use two global views (224 * 224), and 10 query views (16*16) without local view (96*96). We report the results in Fig. 2(a). Compared with baseline DINO, ADCLR outperforms +1.3% top-1 accuracy when À = 0.5. With too large A, the accuracy will drop little, which we guess ADCLR RR would overemphasize the local information, but ignore the global information and leading to the decreasing accuracy of global task (classification). With smaller e.g. between 0.1 and 0.2, ADCLR is mainly determined by [CLS] token, which may bring unwanted contrasting loss (see Fig. 1) to hurt the final performance.
Effect of the number of query crops Q. We set batch size as 1024 and pretrain ADCLR with 100 epochs. In line with ablation on A, we only two global views without local views. Besides, choose the best value of A = 0.5 for this experiment. We illustrate the results in Fig. 2(b). The performance increases with the increase Q. which is similar the ensemble mechanism (with more base models, the ensemble model are more accurate). We find ADCLR saturates when Q = 9 and Q 10, which may be enough to capture local information. We also try large Q, but when we set Q 15, find there's only little improvement, and thus set Q 10 by default.
Effect unidirectional cross attention. As discussed in Sec. 3.2, we find unidirectional cross attention makes ADCLR more effective to downstream tasks (with only the raw patches and [CLS] tokens input). We conduct experiments by using unidirectional and bidirectional cross attention mechanisms and the results given in Fig. 2(c). When Q ADCLR (bidir) drops about 3%~~4% accuracy. When increasing the number query view Q, the accuracy the downstream classification task drops notably. When we set = 196 (same with query token), we find ADCLR (bidir) only gets about 50% accuracy, while ADCLR (unidir) gets 68.8% accuracy.
Table 6: Ablation study query crop ratio.
VIT-S, 100 epochs
0.1
0.15
0.2
ADCLR (Linear)
68.5
69.1
68.8
Effect of query crops ratio Since ADCLR introduces the query crops, further study the performance with different local and global crop ratios. Following iBOT (Zhou et al., 2022), we conduct the experiments by tweaking s, where is the scale dividing the local and global crops. The query crops and local crops are sampled from (0.05, s). The global crops are sampled from (0.4, 1). We fix the other hyper-pun parammer and pretrain ADCLR with 100 epochs. We report the linear classification accuracy in Table 6. We evaluate crop ratios smaller than 0.2 because of the low resolution query crop With a large ratio, low-resolution crops may lose lots of local information.
8

---

Published as a conference paper at ICLR 2023
Table 7: Ablation of the number of local views.
Effeet of local crops. We further conduct ablation on two 8-GPU machines on the number of local views reported in Ta-
100 epochs
300
epochs
CTup strullgyy
Top-1
time
Top-1
time
Imem.
2 x 2244
68.9
16.1h
73.9
47.6h
9.8G
ble 7. We fix the batch size 1024 and
x 224 + x 962
72.2
17.7h
75.3
51.0h
11.1G
evaluate it pretrained with 100 and 300
224 +.6 96a
74.6
20.8h
76.7
60.9h
13.4G
epochs. The multi-crop strategy follows SWAV (Caron et al., 2020), i.c., 2 global views (224*224) and 2, 6, 10 local views (96*96). Under the two-view setting, ADCLR achieves 68.9% top-1 accuracy with 100 epochs pretraining. With the same setting, we reproduce the result of DINO, resulting 67.7% top-1 accuracy with 100 epochs pretraining, where our ADCLR outperforms DINO +1.2% % accuracy. For multi-crops setting, we run DINO with 300 epochs, using 2 x 224² + 10 x 962 crop strategy, resulting 76.2% (0. % higher than reported in DINO) top-1 accuracy, ADCLR outperforms DINO by +0.8% top-1 accuracy. We find ADCLR gets higher accuracy under the two-crops strategy, which we guess using two global views for DINO is difficult to extract local information and the object in global views cannot be matched exactly. For ADCLR, the query tokens are cropped from the raw images with 16 pixels, which help capture accurate and dense information (see Fig. 1).
x 224
96"
75.1
24.7h
77.0
72.6h
16.7G
FURTHER DISCUSSION TO RELATED WORKS
We discuss the relation of ADCLR against similar approaches to better position our work.
Relation to DINO (Caron et al., 2021). Connection. ADCLR adopts the learning framework of DINO (EMA update, sharpening and centering). The global branch of ADCLR is the same as DINO. Differences. On the basis of DINO, ADCLR proposes a local branch to learn spatial-sensitive information. Besides, technically, ADCLR proposes the unidirectional cross attention mechanism.
Relation to iBOT (Zhou et al., 2022). Connection. Both iBOT and ADCLR learn global and local information. Differences. iBOT learns the local information by adding MIM objective, which is now widely developed in MAE (He al., 2021), SimMIM (Xie et al., 2021d). ADCLR learns the local information by maximizing the consistency of the same query crops in latent space (the variance caused by the attention mechanism of two views). Compared with iBOT (masks patches), query tokens in ADCLR may be out of the views, which increases the difficulty of local contrasting.
Relation to SelfPatch (Yun et al., 2022). Connection. Both ADCLR and SelfPatch adopt DINO's framework on the patch level to learn dense information. Differences. The idea of SelfPatch is very straightforward, i.c., regard top-k neighbors of query patch as positives, which may bring the noise (the selection of k). Besides, the local branch of SelfPatch requires input views has the same location, which decreases the learning difficulty. In contrast, ADCLR is simple yet also carefully designed. solve the
The local branch of ADCLR reserves the variance caused by augmentation (ADCLR positional shift caused by random resize and crop). Technically, SelfPatch designs the matching algorithm. For each query crop, it calculates the similarities and sorts them, which may bring lot more complexity. ADCLR only introduces Q query patches, where < Q < 10. The complexity brought by ADCLR can be ignored (ViT-B/16 and VYT-S/16 divide the image into 196 raw patches.)
6 CONCLUSION
This paper proposes ADCLR, new contrastive learning method aiming to capture accurate and local information for vision representation. By introducing "query crop". ADCLR could easily match spatial location without complex operation, e.g., bi-linear interpolation. To make ADCLR more adaptable to downstream tasks and prevent collapse, we design unidirectional cross attention. Experiments on classification, detection, and segmentation show the effectiveness.
Limitations. Currently, the paradigm of "query crops" only supports low resolution (mapping the cropped region into small patches), which may lose semantic information. Hence, we think the multi-scale paradigm could help ADCLR lot, which may be good extension in future works.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 2016.

---

Published as a conference paper at ICLR 2023
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-ivvariane--cvvaaaaaa heg.r/aaaaaaeeeoooooaoaeeeerrrrr for self-supervised learning. In ICLR, 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, NeurlPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and
Armand Joulin. Emerging properties in self-supervised vision transformers. ICCV, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In ICML, 2020a.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint arXiv:2202.03026, 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2020.
Yuxin Fang. Li Dong, Hangho Bao, Xinggang Wang, and Furu Wei. Comuppted image modeling for self-supervised visual pre-training. arXiv preprint #xXiv2222.003822 2022.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1r03.0:803.07728, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 27, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint 1.X6v221 2021.
Olivier J Hénaff, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman, João Carreira, and Relja Arandjelović. Object discovery and representation networks. ECCV, 2022.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilde Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: / 1808 06670, 2018.
10

---

Published as a conference paper at ICLR 2023
Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In ECCV, 2018.
Peng Jin, JinFa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David A Clifton, and Jie Chen. Exxpetation-maxiiizatiio contrastive learning for compact audeo-and-language representations. In NeurlPS, 2022.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In
Tsung Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 10012-10022, 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.033: 03983, 2016.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
Mr D Murahari Reddy, Mr Sk Masthan Basha, Mr M Chinnaiahgari Hari, and Mr N Penchalaiah. Dall-e: Creating images from text. UGC Care Group Journal, 2021.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021.
Zhaoqing Wang, Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang, Mingming Gong, and Tongliang Liu. Exploring set similarity for dense self-supervised representation learning. In CVPR, 2022.
Fangyun Wei, Yue Gao, Zhirong Wu, Han Hu, and Stephen Lin. Aligning pretraining for detection via object-level contrastive learning. NeurlPS, 2021.
Tete Xiao, Yingeheng Liu, Bolci Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018.
Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In ICCV, 2021.
Enze Xie, Jian Ding, Wenhai Wang. Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In ICCV, 2021a.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self-supervised learning with swin transformers, arXiv preprint 10X.::2005.0453,, 2021b.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring lexel- consistency for unsupervised visual representation learning. In CVPR, 2021c.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv: 211.211119986 2021d.
11

---

Published as a conference paper at ICLR 2023
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. arXiv preprint arXiv:2205.09611, 2022.
Sukmin Yun, Hankook Lee, Jachyung Kim, and Jinwoo Shin. Patch-level representation learning for self-supervised vision transformers. In CVPR, 2022.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. In ECCV, 2016.
Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-cl: Instance and feature decorrelation for negative-free symmetric contrastive learning. In /CLR, 2021.
Shaofeng Zhang, Lyn Qiu, Feng Zhu, Junchi Yan, Hengrui Zhang, Rui Zhao, Hongyang Li, and Xiaokang Yang. Align representations with base: A new approach to self-suuer learning. In CVPR, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In ICLR, 2022.
12

---

Published as a conference paper at ICLR 2023
A PROOF FOR THEOREM 1
Proof For simplicity, we consider in one layer single-head self-attention block, and remove the g. b re-scale parameters in layer normalization (Ba et al., 2016). Recall the self attention mechanism in Transformer; ie., Attn(x, Wq, WK, Wv)= Softma (x a) and Wv are learnable weights, and vd is the hidden dimension. Note that x R(++N+))xd includes (CT.S) token, raw patch sequence and query patch sequence. Let 8 (x.Wa)(x, Wx)
vwW, where WQ. WK
Vd
vd where (N+Q)ete S S S 'N+Qie since the normalized ||x, W 2al22 = Then, the input of two branches in contrastive learning becomes [x)xxx
e-1
, x for view A and [x9 xb, "B:'... xB..... X3 N+Q1 }forr view Note that we have X¹x = X3 for S isso Q. Then, the embedding of i-th patch z'A and zig through self-attention block can be written as:
N+Q
N+Q
1 1=0
X Wv,
2B =
B
B Wv. 0 0SiSN+Q
(7)
Recall the objective of negative-free contrastive learning, i.e. aligning the representations max sim(z'A.2gg for 0 si Q. where na is the embedding of (CLS) token. For simplicity, we use the mean square error (MSE) objective for analysis. Take the gradient into Eq. 7, we get:
Oc(= 0sA
06(=))28)) ZB))
OH(='A22b)
01( N+Q 23
Lj=o
NN+ 29 A
Wrl
(8)
0s
0s
where 3: means attention value of i-th patch to j-th patch in view branch. Since x is standardized
and Q >> N. Then, the numerator of Eq. 8 can be written as I| ||(E)= S x Σラ= s xxxx))
) Wy
Then, the network could easily get collapse solution, i.e., making $ so = $ for 0 S i.j S Q, leading
=
the C(2A,222) = O, since xAA = X for 0 < i I Q. Then, we complete the proof.
B IMPLEMENTATIONS
Backbone. We use the standard ViT architecture (Dosovitskiy et al., 2020). It has several blocks composed of multi-head self-attention (MHSA) (Vaswani et al., 2017), MLP, and LayerNorm (Ba et al., 2016). In line with DINO and iBOT, we mainly adopt ViT-S/16 and ViT-B/16 as backbones.
Pretraining hyper-parameters. In line with DINO, we train with Adamw (Loshchilov Hutter, 2018) and batch size of 1024, distributed over 32 GPUs using ViT-S/16. The learning rate is linearly ramped up during the first 10 epochs to its base value determined with the following linear scaling rule (Chen et al., 2020a): Ir = 0.0005, batchsize=256. After warmup, we decay the learning rate with a cosine schedule (Loshchilov & Hutter, 2016). The weight decay also follows a cosine scheduled from 0.04 to 0.4. The temperature is set to 0.1 while use linear warm-up for тe from 0.04 to 0.07 during the first 30 epochs. We follow the data augmentations of BYOL (Grill et al., 2020) (color jittering, Gaussian blur, and solarization) and multi-crop (Caron et al., 2020) with bicubic interpolation to adapt the position embeddings to the scales. We set À = 0.5 and query crop ratio from 0.05 to 0.15. The code with configuration details for reproducing will be publicly available.
class Attention Pytorch-liked code of unidirectional cross attention.
def (self, cls, x, query):
:param cls: B, 1, c
:param B, N, C
:param query: B, Q ***
ękv = self,
self.qkv gkv (torch ([cls, x, queryl, dim=1)) rreshape (B, 1+N+Q, 3, 3, 1, 4)
c // molf.num_headee peeruue (2, 0,
9, k, v qkv[0), gkv[l], [kv[2]
13

---

# query forward Published as a conference paper at ICLR 2023
cls and x forward
attn (ql;, ;, :1+N] 0 k(i, i, :1+N] .transpose (-2, -1))
self.scalo
attn attn. sow.sossttaaxxooo (dim==1))
attn self.attn_droo
x (attn B vl:, :, :1+N]) (Iansposse 2) rreshape (B, N+1, C)
attn iql;, ;, N+1:1 8 klt, ;, 1:N+1] .taansoose (-2, -1))
self.scale
attn attn. softmax (dim=-1)
attn self.attn aelf. (attn)
query = (attn 0 vi:, :, 1:N+1]] ttrnsposee (1, 2) (Beshape 1, C)
torch cat (xx, queryl, dim=l)
x self.projxx)
x sclf.proj_drop (x)
return xli, :1], xli, 1:N+1l, xI, N+1:], attn
14


Reviews:
[{"review_id": "ZhqPq7C6Cpd", "reviewer": "Reviewer_D4eS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on learning dense contrastive representation based on vision transformer backbone. To achieve this goal, the authors propose a clean, simple and effective framework. In detail, they perform the cross-attention between query patches and raw patches. Then the updated query patch features are positive pairs. Despite the local contrastive loss, they also add global contrastive loss. The proposed method outperforms the previous methods.", "summary_of_the_review": "I like the idea that perform attention between query patches and raw patches. Then contrast the query patch features.\nI also like the analyses related to the collapse.\nThere are some details are not doing well, such as selecting the query patches.\nIn summary, I prefer to accept this paper.", "strength_and_weaknesses": "Strengh\n1. The framework is clean, simple, and effective.\n2. This proposed method does not introduce too many extra parameters.\n3. The method outperforms the previous methods.\n\nWeakness\n1. This paper claims that [CLS] token may extract mismatched information. However, the query patch token may also has this problem, if the patch does not exist in one augmented view. \n2. How to select the query patches? Previous works like SoCo use selective search to select good patches. Selective search may also be used in this paper.\n\nOther questions\n1. In Table 7, are the local views used as raw patches or query patches?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xmu7HH8x2jz", "reviewer": "Reviewer_wYqQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a dense contrastive representation learning framework for visual feature learning in a self-supervised manner. The proposed framework considers a local patch query strategy for local contrasting together with a global contrasting. The method achieves superior performances on multiple challenging benchmarks for dense predictions. \n\n", "summary_of_the_review": "The idea of performing local and global contrasting seems to be interesting, but on the other hand, this would also bring extra computational overhead, especially when the number of image crops is high. The authors need to further clarify their advantage on this point. ", "strength_and_weaknesses": "Pros:\n+ the idea of combing local and global contrastive learning is interesting, and also important for dense prediction tasks, as local and global contexts provide different semantic information of the image. \n+ the results are strong. \n\nCons:\n- As shown in Table 7, the local patch query introduces a lot of extra memory. It seems that the method sacrifices the memory overhead to gain performance. How about a larger-batch size? Would the method still be able to train with a reasonable memory consumption?\n- The performance on ImageNet-1k seems to be minor compared to DINO and iBOT (only 0.3%). But from the comparison with state-of-the-arts on the dense prediction tasks, the gain is more clear. Could the authors explain the performance differences?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nc8sM1Im4Ka", "reviewer": "Reviewer_7W9J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about unsupervised learning of image classification. It is focused on patch-level contrastive by an attention mechanism. Experiments on ImageNet-1k, MSCOCO, ADE-20k verify the effectiveness. ", "summary_of_the_review": "The idea and experiments seem good, but the description of the method is not self-contain and clear.", "strength_and_weaknesses": "--Strength\n\nThe paper introduces ADCLR, a novel patch-level contrastive learning method, improves the quality of dense representation of unsupervised learning. The experiments show the competitive performance of the proposed method. The paper provides detailed information for reproduction, and opens a new direction for future research.\nThe improvement is marginal comparing to iBOT on ImageNet. \n\n--Weakness\n\nFigure 1 does not describe the method well, e.g., what do we get after the augmentation image goes through Transformer Encoder? Reconstructed image or deep features?\n\nIn page 4, \"...By feeding the whole sequence of two views xA and xB to...\"  I cannot figure out how to get xA and XB.  It seems z is input, and goes through of attention layer, but where does x comes from? what is two views?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IF1NNzIVtpx", "reviewer": "Reviewer_kB6d", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents ADCLR, a method for contrastive and self supervised representation learning. The main target task is dense prediction i.e. semantic segmentation and object detection, however the framework fits the classification task as well.\n\nWithin the general framework of DINO Caron et al. (2021)  authors state their contributions to be (i) cross-views are used for contrasting learning, (ii) the unidirectional cross attention module, and (iii) achieving SOTA.\n", "summary_of_the_review": "My recommendation is to accept this submission, the paper is sound and the results do support the claims.", "strength_and_weaknesses": "The first natural application is image classification, so ADCLR is benchmarked for that task. Common benchmarks in the field include (i) k-NN and (ii) top-K linear accuracy on imagenet and (iii) comparison against a strong supervised classifier on smaller datasets. The selection of methods (CNN and ViT) and datasets is comprehensive and shows  SOTA improvement.\n\nHowever, the central application is dense prediction i.e. semantic segmentation and object detection, so ADCLR is also benchmarked for those tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster