Soft Neighbors are Positive Supporters in Contrastive Visual Representation Learning
Authors: Chongjian GE
Jiangliu Wang
Zhan Tong
Shoufa Chen
Yibing Song
Ping Luo
Venue: ICLR 2023 poster (2023)
Keywords: contrastive learning
soft neighbors
visual correlation

Abstract: Contrastive learning methods train visual encoders by comparing views (e.g., often created via a group of data augmentations on the same instance) from one instance to others. Typically, the views created from one instance are set as positive, while views from other instances are negative. This binary instance discrimination is studied extensively to improve feature representations in self-supervised learning. In this paper, we rethink the instance discrimination framework and find the binary instance labeling insufficient to measure correlations between different samples. For an intuitive example, given a random image instance, there may exist other images in a mini-batch whose content meanings are the same (i.e., belonging to the same category) or partially related (i.e., belonging to a similar category). How to treat the images that correlate similarly to the current image instance leaves an unexplored problem. We thus propose to support the current image by exploring other correlated instances (i.e., soft neighbors). We first carefully cultivate a candidate neighbor set, which will be further utilized to explore the highly-correlated instances. A cross-attention module is then introduced to predict the correlation score (denoted as positiveness) of other correlated instances with respect to the current one. The positiveness score quantitatively measures the positive support from each correlated instance, and is encoded into the objective for pretext training. To this end, our proposed method benefits in discriminating uncorrelated instances while absorbing correlated instances for SSL. We evaluate our soft neighbor contrastive learning method (SNCLR) on standard visual recognition benchmarks, including image classification, object detection, and instance segmentation. The state-of-the-art recognition performance shows that SNCLR is effective in improving feature representations from both ViT and CNN encoders. 

Published as a conference paper at ICLR 2023
SOFT NEIGHBORS ARE POSITIVE SUPPORTERS IN CONTRASTIVE VISUAL REPRESENTATION LEARNING
Chongjian Ge
Jiangluu Wang"
Zhan Tong²
Shoufa Chen'
Yibing Song Ping Luo'
'The University of Hong Kong
The Chinese University of Hong Kong
Tencent AT Lab AAI Institute, Fudan University
rhettgee@cconeet. hku.hu hk
j jlwang&cuhk edu hk
zhantong... 20236gmai com
shoufacheccnnect. hhw..ee.ccccccc.......... hku.hk
y yibingsong . cv@gmail....mcmmmcmmm..........a com plucêcs .kk.. hk
ABSTRACT
Contrastive learning methods train visual encoders by comparing views (c.g., often created via group of data augmentations the same instance) from instance to others. Typically, the views created from one instance are set as positive, while views from other instances studied extensively to improve feature representations in self-supervised learning. In this paper, we rethink the instance discrimination framework and find the binary instance labeling insufficient to measure correlations between different samples. For an intuitive example, given a random image instance, there may exist other images in mini-batch whose content meanings are the same (i.e., belonging to the same category) or partially related (i.e., belonging to a similar category). How treat the images that correlate similarly to the current image instance leaves un- explored problem. We thus propose to support the current image by exploring other correlated instances (i.e., soft neighbors). We first carefully cultivate a candidate neighbor set, which will be further utilized to explore the highly-correlate stances. A cross- attsntattnnioon module is then introduced to predict the correlation score (denoted as positiveness) of other correlated instances with respect to the current one. The positiveness score quantitatively measures the positive support from each correlated instance, and is encoded into the objective for pretext training. To this end, our proposed method benefits in discriminating uncorrelated instances while absorbing correlated instances for SSL. We evaluate our soft neighbor contrastive learning method (SNCLR) on standard visual recognition benchmarks, including image classification, object detection, and instance segmentation. The state-of the art recognition performance shows that SNCLR is effective in improving feature representations from both ViT and CNN encoders.
negative. This binary instance discrimination is
in-
INTRODUCTION
Visual representations are fundamental to recognition performance. Compared to the supervised learning design, self-supervised learning (SSL) is capable of leveraging large-scale images for pretext learning without annotations. Mcanwhile, the feature representations via SSL are more generalizable to benefit downstream recognition scenarios (Grill et al., 2020; Chen et al., 2021; Xie et al., 2021; He et al., 2022; Wang et al., 2021). ). Among SSL methods, contrastive learning (CLR) receives extensive studies. In a CLR framework, training samples are utilized to create multiple views based on different data augmentations. These views are passed through a two-branch pipeline for similarity measurement (e.g., InfoNCE loss (Oord et al., 2018) or redundancy-reducee
loss (Zbontar et al., 2021)). Based on this learning framework, investigations on memory queue (He et al., 2020), large batch size (Chen et al., 2020b), encoder synchronization (Chen & He, 2021), and self-distillation (Caron al., 2021) show how to effectively learn self-supervised representations.
In a contrastive learning framework, the created views are automatically assigned with binary labels according to different image sources. The views created from the same image instance are labeled as positive, while the views from other image instances are labeled as negative. These positive
"Coreesooding author. We provide the homepage for this project

---

Published as a conference paper at ICLR 2023
Vanilla
(b) SWAV
(c) NNCLR
(d) SNCLR
Current Instances
Other Instances
Nearest Neighbors
Soft Neighbors
Training sample comparisons in the self-supervised contrastive learning framework. The vanilla contrastive learning method is shown in (a) where the current instance is regarded as positive while others are negative. In SwAV (Caron et al., 2020), the current instance is assigned to online maintained clusters for contrastive mcasurement. In NNCLR (Dwibedi et al., 2021), the nearest neighbor is selected as a positive instance to support the current one for the contrastive loss com- putation (Chen et al., 2020b). Different from these methods, our SNCLR shown in (d) measures correlations between the current instance and other instances identified from the candidate neighbor set. We define the instances that are highly-correlater to the current instance as soft neighbors (e.g., different pandas to the input image belonging the 'panda' category in Fig. 3). The similar- ity/correlation, which is based on the cross-attes -oos-ntttntionn computation from a shallow attention module, denotes the instance positiveness to compute the contrastive loss. (Best viewed in colors)
Figure I:
and negative pairs constitute the contrastive loss computation process. practice, we observe that this rudimentary labeling is not sufficient to represent instance correlations (i.e., similar to the semantic relations in supervised learning) between different images and may hurt the performance of learned representations. For example, when use ImageNet data (Russakovsky et al., 2015) for self-supervised pre-training, views from the current image instance 2o belonging to the 'bloodhound' category is labeled as positive, while views from other image instances belonging to the same category will be labeled as negative. Since we do not know the original training labels during SSL, images belonging the same semantic category will be labeled differently limit feature representations. On the other hand, for the current image instance belonging to the 'walker hound' category and views from another image instance 22 belonging the 'peacock' category are both labeled as negative, the views from are more correlated to the views from ny than those views from 222 This correlation not properly encoded during contrastive learning as views from both and T22 are labeled the same. Without original training labels, contrastive learning methods are not effective in capturing the correlations between image instances. The learned feature representations are limited to describing correlated visual contents across different images.
although views from one image instance I
Exploring instance correlations have arisen recently in SwAV (Caron et al., 2020) and NNCLR (Dwibedi al., 2021). Fig. 1 shows how they improve training sample comparisons upon the vanilla CLR. (a), the vanilla CLR method labels the view from the current image instance as positive while views from other image instances negative. This CLR framework is improved in (b) where AW assigns view embeddings of the current image instance to neighboring clusters. The contrastive loss computation based on these clusters rather than view embeddings. Another improvement upon (a) is shown in (c) where nearest neighbor (NN) view embeddings are utilized in NNCLR support the current views to compute the contrastive loss. The clustered CLR and NN selection inspire us to take ad advantage of both. We aim to accurately identify the neighbors that are highly correlated to the current image instance, which is inspired by the NN selection, Meanwhile, we expect to produce a soft measurement of the correlation extent, which is motivated by the clustered CLR. In the end, the identified neighbors with adaptive weights support the current sample for the contrastive loss computation.
In this paper, we propose to explore soft neighbors during contrastive learning (SNCLR), Our framework consists of two encoders, two projectors, and one predictor, which are commonly adopted in CLR framework design (Grill et al., 2020; Chen et al., 2021). Moreover, we introduce a candidate neighbor set to store nearest neighbors and an attention module compute cross-attention scores. For the current view instance, the candidate neighbor set contains instance features from other images, but their feature representations similar that of the current view instance. We then
2

---

Published as a conference paper at ICLR 2023
compute cross-attention score of each instance feature from this neighbor set with respect to the current view instance. This value is a soft measurement of the positiveness of each candidate neighbor contributing the current instance during contrastive loss computations. Basically, higher positiveness often stands for a higher correlation between two samples. To this end, our SNCLR incorporates soft positive neighbors to support the current view instance for contrastive loss computations. We evaluate our SNCLR on benchmark datasets by using both CNN and ViT encoders. The results of various downstream recognition scenarios indicate that SNCLR improves feature representations to state-of thteeoffthe-att th--hhe-rt art performance.
RELATED WORKS
Our SNCLR framework explores nearest neighbors self-supervised visual pre-training. In this section, we perform literature review self-supervised visual representation learning and nearest neighbor exploration in computer vision.
2.1 SELF-SUPERVISED VISUAL REPRESENTATION LEARNING
The objective of the SSL paradigm is to construct pre-text feature learning process without manually labeling efforts. Using unlabeled visual data (e.g.. images and videos), SSL methods design various pretexts to let these data supervise themselves. The learned features via supervvipn are more generalizable to benefit downstream scenarios where supervised learning adapts these pre-text features into specific visual tasks. Existing SSL methods be categorized into generative and discriminative frameworks. The generative methods typically introduce an autoencoder structure for image reconstruction (Vincent al., 2008; Rezende al., 2014; Chen et al., 2020a), or model the data and representation in joint embedding space (Donahue & Simonyan, 2019). Recently, breakthrough frameworks have been developed based on masked image modeling (MIM) (He et al., 2022; Bao et al., 2021; Tong et al., 2022). The visual data is partially masked and sent to an encoder-decoder structure for data recovery. By reconstructing pixels, MIM methods produce visual features that are more generalizable. However, these methods are specifically designed for vision transformers and are computationally intensive. Without downstream finetuning, off-the-shelf adoption of their pre-trained features does not perform favorably recognition scenarios.
Apart from the generative framework, the discriminative SSL methods aim to minimize the feature distances of views augmented from the same images while maximizing views from two different images. Among discriminative methods, one of the representative arts is contrastive learning. Based on the instance diserimination paradigm, studies improve the learned representations by exploring memory bank (He et al., 2020; Pan et al., 2021), features coding (Henaff, 2020; Tsai et al., 2021) 1), multi-modality coding (Tian al., 2020a; Tsai al., 2020), strong data-augmentation (Grill et al., 2020; Caron et al., 2020; Tian et al., 2020b), parameters-updatioe strategy (Chen & He, 2021; Grill al., 2020) and loss redevelopment (Zbontar et al., 2021: Ge et al., 2021; Bardes et al., 2021). Among these, the neighboring- based methods (Caron et al., 2020; Dwibedi et al., 2021; I; Azabou et al., 2021) leverage either the cluster sets or the nearest neighbors to model the correlation between views. This design mitigates the limitation in CLR that visual data belonging to the same category will be labeled differently. this work, we explore soft neighbors support the current instance sample. These neighboring instances are from other images and make partial and adaptive contributions that are measured via cross-attention values.
2.2 NEAREST NEIGHBOR EXPLORATION IN VISUAL RECOGNITION
The nearest neighbor (NN) has been widely applied in the computer vision tasks, such as image classification (Boiman al., 2008; McCann Lowe, 2012), object detection//sgmmenationn (Harini & Chandrasekar, 2012), domain adaption (Yang et al., 2021; Tang et al., 2021; Tommasi & Caputo, 2013), vision alignment (Thewlis et al., 2019; Dwibedi et al., 2019), and few-shot learning (Wang et al., 2019). Generally, the nearest neighbor method explores the comprehensive relations among samples to facilitate the computer vision tasks. Earlier attempts have also been made in the field of self-supervised visual pre-train For example, in the contrastive learning framework, different from comparing representations directly, SwAV (Caron et al., 2020) additionally sets up prototype feature cluster and simultaneously maintains the consistency between the sample features and the
3

---

Published as a conference paper at ICLR 2023
encoder
projector
predictor
2g
View
positiveness
attention module
W2
SNCER
Input x
View 32
NN(5))
Top
momentum
momentum projector
candidate
NNOs)
other instance
My
encoder
neighbar
neighhor
neighto
Memifiiction
An overview of SNCLR. Given an input instance we obtain two views (i.e., 21 and .2). After projection, we obtain yi and y2. We use y/2 to select K candidates from the candidate neighbor set C where samples are from other images. For each selected sample (N(22); (i E [l...., KD, we sent to the attention module together with 31 to compute a cross- cross-attention value w; This value is the soft measurement of (N(2) for contrastive computation.
Figure 2:
prototype features. More recently, NNCLR (Dwibedi al., 2021) utilizes an explicit support set (also known as a memory queue/bank) for the purpose of nearest neighbor mining. In these methods, the neighbors are regarded as either or to contribute to the current instance sample. This inaccurate we observe that neighbors are usually partially related to the current sample. Different from the above methods, we leverage an attention module to measure the correlations between neighboring samples and the current The instance correlations are formulated positiveness scores during contrastive loss computations.
PROPOSED METHOD
Fig. shows an overview of our SNCLR. We construct a candidate neighbor set to support the current view during contrastive learning. The support extent is measured softly as positiveness via cross-attention computations. We first revisit the CLR framework, then illustrate our proposed SNCLR and its training procedure. Moreover, we visualize selected neighbors for intuitive view of how they positively support the current instance sample.
3.1 REVISITING CONTRASTIVE LEARNING
We show how contrastive learning works based on our framework in Fig. 2. Given an input instance sample from batch X. we create two augmented views æ1 and m2. After feature extraction and projection, obtain their corresponding outputs y1 and y2. For other N 1 instances from this batch, we denote their projected view representations as v22 where i [1, 2.... N 1]. The v² is shown in the figure for simplicity. The contrastive loss can be generally written as:
exp (sim(y), 82)/T)
C llg exp (sim (y1, /2) + ΣnT 1 exp (sim (//))))
(I)
where x E X, 7 is temperature parameter, 1/1 and y/2 are positive pairs, y1 and y/ are negative pairs. The sim(-) is the operator for similarity measurement, which is usually set as cosine distance (Chen al., 2020b; He al., 2020). We can also use only positive samples and substitute y1 with 2 for sample similarity computation (Grill et al., 2020; Chen & He, 2021), where projected representation of y1 via a predictor as shown Fig. 2 (e.g.. c. the sequentially-conneetee MLP layers). After computing the loss, momentum update (He et al., 2020) or partial gradient stop (Chen Hc, 2021) utilized to train encoders. Note that in Eq. all the instances other than the current are labeled as negative. To explore their correlations, introduce SNCLR follows.
use to denote the
3.2 SOFT NEIGHBORS CONTRASTIVE LEARNING (SNCLR)
Our SNCLR introduces the adaptive pright/positiveness measurement based on a candidate neighbor set C. We follow the CLR framework illustrated in Sec. 3.1. For the current instance x, cultivate
4

---

Published as a conference paper at ICLR 2023
candidate neighbor set, where we select K candidate neighbors with projected representations NN(32); (i E [1,2,... K). The N.( denotes the nearest neighbor identification operation in the projected feature space. For each identified neighbor NN(22),, we sent it to the attention module together with y1 for cross-attention computations. This attention module predicts a positiveness value Wy. We use this value to adjust the contributions of (y(22); to 21 in contrastive learning. The loss function of our SNCLR can be written as:
1 N
EEE exp (2) NN((2)(/T) ii exp (z1 NN(32)/7T)+ + E Σj= exp (21 (N(12))//
CSNCLR
log
(2)
where 21 is the predictor output, we use NV(22 to denote y2, use NN )o to denote 3/2 wo is
set as 1, N,2 is the nearest neighbors search of each y/ to obtain K neighbors (i.e., NN(15 ) j E [1,2..., KD. The and NN(22); are partially positive pairs, while the ž and NN ) are negative pairs. If we exclude the searched neighbors of 1/2 and y/- Eq. 2 will degenerate to Eq. 1.
Candidate Neighbors. We follow Dwibedi et al. (2021) use a queue as candidate neighbor set C, where each element the projected feature representation from the momentum branch. When using sample batches for network training, we store their projected features in C and adopt first-in-first-nn -ir strategy to update the candidate set. The length of this set relatively large to sufficiently provide potential neighbors. When identifying K candidate neighbors based on this set, we use the cosine similarity as the selection metrie. The nearest neighbors search can be written as follows:
(1 = arg max (cos(y2, c), top, = =K,
dEC
where both 1/2 and E C are normalized before computation. The NN(-.) operation is utilized for both and 1/2 to identify the top-K corresponding neighbors.
Positiveness Predictions. We quantitatively measure the correlations of the selected K neighbors with respect to the current instance sample. Fig. 2 shows intuitive view of the candidate neighbors. We observe that the first two neighbors belong to the same category as the input sample (i.e., 'Cavalier King Charles Spaniel'), ), while the third and fourth neighhors are partially related (i.e., these two samples belong to the 'dog' category but not in 'Cavalier King Charles Spaniel' breed). Besides, other samples do not belong to dogs. To this end, we introduce an attention module to softly measure (not in a binary form) their correlations (also known as positiveness in this paper). This module contains two feature projection layers, cross-attention operator, and a nonlinear activation layer. Given two inputs yi and NN(22). the positiveness score can be computed as:
W; - Softmax (vi) (v1) x (a(NN(1)):)
where /1(-) and /2(-) are the projection layers of this attention module, is scaling factor to adjust positiveness uy. We have empirically tried using different structures of this attention module. including using parametric linear projections for fi (-) and f-)., and adding more projection layers after the nonlinear activation layer. The results of these trials are inferior than simply using Eq. by setting (.) and fz(-) as identity mappings without parameters. This may be due to the self- supervised learning paradigm, where the objective is to effectively train the encoder. A simplification of this attention module (i.e., with less attentiveness) will activate the encoder to produce more attentive features during pretext training.
Analysis: Computing contrastive loss with more instances. The soft weight cross-attention design empowers us to properly model the correlations between the current instance and others. This design enables us leverage more instances to compute the contrastive loss and thus brings two advantages. First, we enrich the diversity of positive instances to support the current one in a soft and adaptive way. Different from the original SSL frameworks that align augmented views with the current instance, SNCLR explores correlations between different instances. At the same time, our SNCLR mitigates the limitation that instances belonging to the same semantic category are labeled oppositely in CLR. Second, SNCLR introduces more negative samples for discrimination (i.e., the yellow instances in Fig. 2). The number of negative samples introduced in SNCLR is N times larger than existing methods (Chen et al., 2021; Dwibedi et al., 2021), where negative samples are from the current mini-batch Using more negative samples and partially related positive samples benefits SNCLR in discriminating uncorrelated instances while absorbing correlated instances during pretext training.
5

---

Published as a conference paper at ICLR 2023
giant panda
1.000
0.955
0.932
0.864
Wb 0.796
trolleybus
Vi = 1.000
U/ = 0.961
ug 0.827
1/4 0.789
W/ 0.269
Input Instance
Neighbor-1
Neighbor-2
Neighbor-3
Neighbor-4
Neighbor-5
Figure 3: Neighbors display. We show imput instances in the first column. Their nearest 5 neighbors are shown on the corresponding row for clarification. We normalize the scores to clarify its positive contribution to the current sample.
Network Training. We compute contrastive loss via Eq. 2, and update network parameters of the upper branch shown in Fig. 2 via pack-propagation Afterwards, we follow (Grill et al., 2020; Chen al., 2021) to perform moving average update the momentum networks (i.e., the momentum encoder and the momentum projector). We refer the readers to Appendix A for more details on the network training details.
Neighbors Display. We show selected neighbors with var input instances in Fig. 3. The encoder backbone of ViT-S is utilized for visualization, with 300 epochs of training. Given an input instance, we use Eq. 3 to select top-K K neighbors and show them in this figure. We use an input instance from the "giant panda' category in the first row. The category is unknown during SNCLR. The top neighbors all belong to the 'giant panda' with high positiveness values. This indicates our SNCLR effectively selects highly- highly-eorrelated neighbors and computes high positiveness scores for contrastive loss computations. On the second row, the input instance is from the 'trolleybus' category. The top 4 neighbors are from the same category with high positiveness scores, while the 5-th neighbor is from the 'ocean liner' category with smller positiveness score. That's to say, the high score of us corresponds to the same category while the lower score corresponds to less-correlated contents, which shows that our attention module is effective in modeling correlations between the input instance and each candidate neighbor. As shown in Eq. 4, we accordingly scale the positiveness with a factor. More visualization examples are presented in Appendix C.I.
4 EXPERIMENTS
In this section, we perform experimental validation on our SNCLR framework. The implementation details will be first introduced. Then, compare our proposed SNCLR to state-of-the-art SSL methods on prevalent vision tasks, including image classification, object detection, and instance segmentation. In the end, the critical component will be ablated for effectiveness validation.
4.1 IMPLEMENTATION DETAILS
Network architectures. We adopt prevalent vision backbones (i.e., Vision Transformer (ViT) (Doso- vitskiy al., 2021) and ResNet (He et al., 2016)) as the visual encoders. To fully explore the potential of our proposed SNCLR, we adopt three different encoder architectures for validation, including ResNet-50, ViT-S, and ViT-B. For the ResNet, the features by the final block are further processed by global context pooling operator (Gholamalinezhad & Khosravi, 2020) to extract the representations. For the ViTs, we follow (Dosovitskiy et al., 2021; Chen et al., 2021; 1; Caron et al., 2021) to leverage the [CLS] tokens as the representation vectors. The designs of the projectors and the predictors follow those from (Grill et al., 2020), where there are two fully-connected layers with batch normalization (loffe & Szegedy, 2015) and ReL.U activation (Agarap, 2018) in between.

---

Published as a conference paper at ICLR 2023
Table 1: Linear evaluations on ImageNet with top-1 accuracy (in %).. Both the ResNet and ViT en- coders are utilized. Prevalent methods, including SimLCR (Chen et al., 2020b), MoCo (He et al., 2020; Chen et al., 2020c; 2021), SimSiam (Chen & He, 20foM,nnfoMin (Tian al., 2020b). et al., 2020), BYOL (Grill et al., 2020), NNCLR (Dwibedi et al., 2021), DINO (Caron et al., 2021), MAE (He et al., 2022), and CAE (Chen et al., 2022). We highlight best results under the same model parameters constraints in bold. denotes the experimental results implemented by ourselves since there are no direct results provided. For ResNet, we show the accuracy results of the encoder trained with 100, 200, 400 and 800 epochs. For ViTs, the performances are reported with the encoders trained with 100, 200 and 300 epochs. All the results are reported without the multi-crop augmentation.
(Caron
(a) ResNet-50.
(b) ViT-S.
(c) ViT-B.
Method
100 200 400 800
Method BYOL
100
200 300
Method
100 200 300
SimCLR
66.5 68.3 69.8 70.4
71.0
BYOL
73.9
InfoMin Aug
70.1
MoCo v2
71.6
MoCo v3
76.2
SWAV
66.5 69.1 70.7 71.8
MoCo v3"
67.5
71.8 72.5
MoCo v2
67.4 69.9 71.0 72.2
NNCLR
69.1
71.9 75.9
SwAV
67.1
MoCo v3
68.1 72.2 73.0 74.1
SimCLR
73.9
DINO
72.5
SimSiam
68.1 70.0 70.8 71.3
MAE
61.5
NNCLR
69.2 70.7 74.0 74.5
NNCLR"
66.8
70.9 71.5
BYOL
66.5 70.6 73.2 74.3
SimCLR
69.0
MAEVTL
57.3 64.4 67.3
Barlow Twins
72.5 73.2
CAE
51.8
CAE
64.1
SNCLR(ours)
69.6 72.4 74.5 75.3
SNCLR(OurS)
68.5 72.5 73.4
SNCLR(ours)
71.4 74.3 76.8
Training configurations. The pretext training is conducted the ImageNet-IK dataset (Russakovsky et al., 2015) without labels. We follow the data augmentation strategies from (Grill et al., 2020) to create distorted views, which include the resolution adjustment, random horizontal flip, color distortions, Gaussian blur operations, etc. More details about data augmentation are described in Appendix A. For ResNet encoders, we adopt the LARS optimizer (You al., 2017) with cosine annealing schedule (Loshchilov & Hutter, 2016) to train the networks for 800 epochs with warm-up of 10 epochs. The learning rate is scaled linearly with respect to the batch size via = 0.3 x BatchSize/ 256 (Goyal et al., 2017), where 0.3 denotes the base learning rate. The update coefficient of the momentum network branches is set as 0.99. Unless otherwise stated, we utilize 30 soft neighbors train SNCLR. Different from ResNet, we leverage AdamW optimizer (Loshchilov & Hutter, 2017) to train ViTs for 300 epochs with a warm-up of 40 epochs. Following (Chen et al., 2021). the base learning rate is configured as 1.5 x 10-1. We refer the readers Appendix A for more detailed training configurations.
4.2 COMPARISON TO OHE APPROACHES
The representations learned by SNCLR are compared with other state-of-the-ar frameworks in various downstream tasks, including image classification, object detection, and instance segmentation.
Linear evaluation image classification. Following the standard linear classification proto- col (Grill et al., 2020), we freeze the parameters of the encoder backbones and additionally train a linear head. We use the SGD optimizer, with a momentum of 0.9, batch size of 4096, and weight decay of O, during linear training. The base learning rate is set as 0.1 for ResNets and 3 for ViTs.
As shown in Table we perform linear probe on encoders (i.e., fine-tuning the linear head with 90 epochs while keeping the encoders fixed) and report the linear evaluation with top-1 accuracy. Table la presents the classification results of ResNet-50 encoders with various self-supervised frameworks. It validates that our SNCLR consistently outperforms the prevalent SSL methods at different training epochs. For example, SNCLR obtains 74.5% top-1 accuracy with only 400 training epochs, which is even higher than almost all SSL frameworks trained with 800 epochs (e.g.. 74.3% with BYOL (Grill et al., 2020), and 71.8% with SWAV (Caron et al., 2020)). Even compared to the methods (Dwibedi et al., 2021) explicitly involving the neighbor information, SNCLR achieves the best performance of 75.3% top-1 accuracy at 800 epoch. We also evaluate the effectiveness of SNCLR on ViT encoders. Quantitative results in Table 1b reveal that absorbing the highly-correlan neighbors empowers the various ViT encoders in extracting more discriminating representations as well, thus facilitating image classifications (e.g., 73.4% with ViT-S and 76.8% with ViT-B).
7

---

Published as a conference paper at ICLR 2023
Table Image classification by using comi-supervised training on ImageNet. denotes the results implemented by ourselves since there are no direct results provided.
(a) Classification accuracy via ResNet-SO.
(b) Classification accuracy via ViT-S.
Method
Epoch
Top-1
Top-5
Method
Epoch
Top-1
Top-5
1% 10% 1% 10%
1%
10%
1%
10%
Supervised
25.4 56.4
48.4
80.4
MoCo v3*
300
46.9
58.3
72.1
79.4
BYOL"
300
44.2
54.5
71.8
77.9
SimCLR
800
48.3 65.6
75.5 87.8
NNCLR
300
45.8
55.6
71.7
78.0
NNCLR*
400
55.1 69.0 79.2 88.6
DINO
300
50.9
61.4
75.3
87.9
MoCo v3"
400
51.2
62.2
76.1 82.4
SNCLR(our))
300
51.2
61.9
76.0
88.4
BYOL
800
53.2
68.8
78.4 89.0
SNCLR(OUrS)
400
56.0
69.2 80.1 89.1
Linear evaluation on image classification with semi-supervised learning. The representations are also evaluated in the semi-supervised setting. After pretext training. we add an extra classifier head and fine-tune the overall networks with certain subset of ImageNet training samples. Following the previous works (Zbontar al., 2021; Grill et al., 2020), we leverage I % and 10% of the training set for fine-tuning. As shown in Table 2, by fine-tuning the networks on 10% training dataset from ImageNet- Ik, SNCLR outperforms other methods with both the RersNet and ViT backbones. The considerable top-1 accuracy in Table 2 validates that encoders trained with SNCLR are able to produce more generalizable representations.
Evaluation on object detection and segmentation. To compare the transfer learning abilities of the learned representations, we conduct evaluations on object detection and instance segmentation. We adopt the prevalent COCO datasets (Lin et al., 2014). The pre-trained encoder of SNCLR is extracted and integrated into Mask R-CNN (He al., 2017). For fair comparisons, the same encoder (i.e., ResNet-50) is leveraged the visual backbones. We follow the standard training configurations provided in (Chen et al., 2019) to fine-tune the detectors with a 'Ix' schedule (90k iterations).
Table 3 shows the evaluation results. Compared to the supervised methods in the first block, SNCLR advances both object detection (by 1.0% Aphb) and instance segmentation (by 1.0% APT*), Besides, SNCLR outperforms favorably prevalent SSL methods. Taking the visual encoder trained with 200 epochs for example, SNCLR achieves the comparable performance trained with more epochs (e.g.. 39.4% Ap for BYL with 800 epochs) or better performance (e.g., +2.5% AP* for Barlow Twins with 1000 epochs). Similar results could be observed in instance segmentation scenarios as well, where SNCLR consistently outperforms other SSL methods. The performance profits indicate that encoding the contents correlation facilitates exploring rich information in samples, thus benefiting the visual encoders in generating more transferable representations for detection and segmentation tasks.
4.3 ABLATION STUDIES
In the SNCLR framework, correlation extent among the current instance and neighbors in the candidate set encoded positiveness for pretext training. Thus, investigate the influence of the main components, including the positiveness, selected neighbors, and candidate neighhor set. Unless otherwise stated, the ViT-S encoder pre-trained with 300 epochs is utilized for ablation studies.
Importance of positiveness. Our main contribution is encoding the correlations as positiveness to train the visual encoders. We analyze the importance of positiveness to the recognition performance in the second block of Table 4. Taking 30 neighbors for example, when we remove the positiveness and label all the neighbors exactly positive samples, the top-1 accuracy decreases from 73.38% to 72.24%. indicates that the binary labeling strategy is insufficient to measure the correlation extent at the feature space, thus causing the encoders produce less discriminating representations compared with soft labeling methods.
Importance of neighbors. We analyze how the selected neighbors affect the recognition performance as well. As shown in the first two blocks of Table 4, the visual encoders achieve a performance gain of +1.2% (i.e., 72.24% v.s. 71.17%) with the assistance of neighbors during the pretext training. The performance profits may contribute to the abundant comparisons/suppors from positive samples, which facilitates the current instance to align more highly-correlated views. In the third block of
8

---

Published as a conference paper at ICLR 2023
Table 3: Transfer learning object detection and instance segmentation with the Mask R-CNN R50-FPN detector. We highlight the best two results in bold. We compare SNCLR with prevalant methods, including PIRL (Misra & Maaten, 2020), Barlow Twins (Zbontar et al., 2021), DetCo (Xie et al., 2021) and etc. Our method achieves favorable detection and segmentation performance.
COCO det.
COCO instance seg.
Method
Epoch
Ap
AP
APP5
Ap
AP
AP75
Rand Init Supervised
31.0
49.5
33.2
28.5
46.8
30.4
90
38.9
59.6
42.7
35.4
56.5
38.1
PIRL
200
37.5
57.6
41.0
34.0
54.6
36.2
SwAV
200
38.5
60.4
41.4
35.4
57.0
37.7
MoCo
200
38.5
58.9
42.0
35.1
55.9
37.7
MoCo v2
200
38.9
59,4
42.4
35,5
56.5
38.1
MoCo v2
800
39.4
59.9
43.0
35.8
56.9
38.4
Barlow Twins
1000
36,9
58.5
55.4
BYOL
200
39.1
59.5
42.7
35.6
56.5
38.2
BYOL
400
39.2
59.6
42.9
35.6
56.7
38.2
BYOL
800
39.4
59.9
43.0
35.8
56.8
38,5
DetCo
200
39.4
59.2
42.3
34.4
55.7
36.6
SNCLR (ours)
200
39.4
60.3
43.0
36.0
57.1
38.3
SNCLR (ours)
800
39.9
60.5
43.7
36.4
57.3
39.0
Table 4: Analysis on the positiveness and number of neighbors K.
Table 5: Analysis set length.
neighbor
positiveness
K
Top-1
length
Top-1
x
x
71.17
71.17
X
30
72.24
8000
72.25 (+1.08)
/
30
73.38 (+2.211
16000
72.67 (+1.50)
/
10
72.81 (+1.64)
32000
72.43 (+1.26)
/
30
73.38 (+2.21)
64000
73.02 (+1.85)
50
72.92 (+1.75)
128000
73.38 (42.21)
Table 4, we also conduct experiments to analyze the influence of the number of neighbors. It's empirically found that there trade-off for the neighbor number. On the one hand, sufficient neighbors are able to enrich the diversities of the positive comparisons. On the other hand, there is certain probability of including some correlation-agnostii instances, which shall be the negative samples, into positive neighbors. Involving the less-correlated positive samples may disturb the training process and hurts the recognition performance. Our results show that 30 neighbors achieve the best accuracy. Thus, we adopt 30 neighbors in experiments for simplicity.
Analysis the neighbor set length. We empirically found that large set increases the recognition performance. For example, results in Table show that the encoder trained with the largest set achieves the best accuracy of 73.38%. It indicates that with the capacity of the increasing, it will be possible for the current instance to find more correlated neighbors to softly support itself.
CONCLUSIONS
In this work, we propose contrastive learning framework, termed SNCLR, to encode the correlation extent between instances as positiveness for softly supporting the positive samples. Contrary to the previous works that utilize binary weights measure the positive and negative pairs, our SNCLR leverages the soft weights to adaptively support the current instances from their neighbors. Thanks to sufficiently exploring the correlation among samples, proposed SNCLR facilitate both the ResNet and ViT encoders in producing more generalizable and discriminating representations. Experiments on various visual tasks, including image classification, object detection, and instance segmentation, indicate the effectiveness of our proposed SNCLR.
Acknowledgement. This work is supported by the General Research Fund of HK No. No.27708720. No. No.172122 72121200 and No.17200622

---

Published as a conference paper at ICLR 2023
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv: 1.XXv:/80330837, 2018.
Mchdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C Johnson, Kiran Bhaskaran-Nair, Max Dabagia, Bernardo Avila-Pires Lindsey Kitchell, Keith Hengen, et al. Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprini arXiv:2 102. 10106, 2021.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2/06.08254, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: for self-supervised learning. International Conference on Learning Representations, 2021.
regularization
Oren Boiman, Eli Shechtman, and Michal Irani. In defense of nearest-neighbor based image classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2008.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9650-9660, 2021.
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng. Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang. Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 999600015,, 2019.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning. 2020b.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang. Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-superviser representation learning. arXiv preprint arXiv:2202.03026, 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in neural information processing systems, 32, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
10

---

Published as a conference paper at ICLR 2023
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: neighbr contrastive learning of visual representations. In IEEE/CVF International Conference Computer Viston, 2021.
Chongjian Ge, Youwei Liang, Yibing Song, Jianbo Jiao, Jue Wang, and Ping Luo. Revitalizing cnn attentions via transformers in self-supervised visual representation learning. In Advances in Neural Information Processing Systems, 2021.
Hossein Gholamalinezhad and Hossein Khosravi. Pooling methods in deep neural networks, review. arXiv preprint arXiv:2009.07483 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint / 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self- supervvid learning. Advances in Neural Information Processing Systems, 2020.
R Harini and C Chandrasekar. Image segmentation using nearest neighbor classifiers based on kernel formation for medical images. International Conference on Putlern Recognition, Informatics and Medical Engineering (PRIME-2012), 2012.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In IEEE/CVF International Conference on Computer Vision, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision leamners IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference on Machine Learning, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0 0ew.1660080098,,. 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17110:1711.05100 2017.
Sancho McCann and David Lowe. Local naive bayes nearest neighbor for image classification. In IEEF/CVF Conference Computer Vision and Pattern Recognition, 2012.
Ishan Misra and Laurens der Maaten. Self-supervised learning of pretext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /rXiv:/877.0885 1807.0874733 2018.
11

---

Published as a conference paper at ICLR 2023
Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and variational inference in deep latent gaussian models. In International Conference on Machine Learning, 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
Song Tang, Yan Yang, Zhiyuan Ma, Norman Hendrich, Fanyu Zeng, Shuzhi Sam Ge, Changshui Zhang, and Jianwei Zhang. Nearest neighborhood-basee based deep clustering for source data-absent unsupervised domain adaptation. arXiv preprint wxXi:2217.11285. 2021.
James Thewlis, Samuel Albanie, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of landmarks by descriptor vector exchange. In IEEE/CVF International Conference on Computer Vision, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 2020b.
Tatiana Tommasi and Barbara Caputo. Frustratingly easy nban domain adaptation. In IEEE/CVF International Conference on Computer Vision, 2013.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. arXiv preprini arXiv:2203. /2602, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self-super learning from a multi-view perspective. arXiv preprint arXiv:2006.05576 2020.
Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang. Han Zhao, Louis Louis-Philippe Morency, and Ruslan Salakhutdinov. Self-supervised representation learning with relative predictive coding. arXiv preprint 103.: 1275, 2021.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning, 2008
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Wang, WL Chao, KQ Weinberger, and L Simpleshot Van der Maaten. Revisiting nearest-neighbe classification for fhw-shot learning. arxiy 2019, arXiv preprint arxiv:991 (90.0666.. 2019.
12

---

Published as a conference paper at ICLR 2023
Ross Wightman. Pytorch image models.
attpss://ggtuub..conm/vvvviggga
pytorch- cerrch-image-mooee -wwgammmdeeeesa..seemmmmmm 2019.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping
Luo. Detco: Unsupervised contrastive learning for object detection. In IEEE/CVF International Conference on Computer Vision, 2021.
Shiqi Yang. Joost van de Weijer, Luis Herranz, Shangling Jui, et al. Exploiting the intrinsic neighbor- hood structure for source-free domain adaptation. Advances in Neural Information Processing Systems, 2021.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint /708 03888, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, 2021.
13

---

Published as a conference paper at ICLR 2023
In this appendix material, the experimental details will be further described, including the network architectures, data augmentations, and basic training configurations. In the end, more visualization results on the selected neighbors and experimental analysis will be presented.
A EXPERIMENTAL DETAILS
Network Architectures. We adopt both the ResNet-50, ViT-S, and ViT-B as visual encoders for experimental comparison. Specifically, as for ResNet-50, we follow the implementation of Torchvis sion Toolkit (Paszke et al., 2019). The 2-D feature map produced by the ResNet-50 encoder is globally pooled to produce the I-D representation. For ViTs, we leverage the implementa- tion from timm tcolkit (Wightman, 2019) and DeiT (Touvron al., 2021). The patch resolution of 16 x 16 is adopted for both ViT-S and ViT-B. The attached [CLS] ] token is extracted as the pro- duced representations. Each projector and predictor in our framework consists of fully-connected layer with batch normalization (loffe & Szegedy, 2015) and a ReL.U activation layer (Agarap, 201 8), followed by another fully-connected layer. Note that we utilize the identity mapping for the vector projection in the cross-attention module. Thus, there are no network parameters in this block.
Image Processing. We here illustrate how we create two distorted views via data augmentation. For the pretext training stage, we follow (Grill et al., 2020) to conduct two groups of data augmentation on the input image. Specifically, them to 224 x 224 resolution. Then the random color distortions, the solarization, and the Gaussian blur operations are applied these patches. In the end, we perform the random horizontal flip and normalization on the processed samples to feed the networks for training. For the data augmentation in the linear evaluation procedure, we follow (Chen et al., 2021; 2020c) to include random cropping. resizing, and horizontal flip for simple data transformation.
first randomly crop local patches from the inputs and resize
Training Configurations. The training configurations are different among ResNets and ViTs. For ResNet-50, the LARS optimizer (You et al., 2017) is adopted to train the networks for a total of 800 epochs. The learning rate follows cosine annealing schedule without restarting (Grill et al., 2020; Loshchilov & Hutter, 2016). For the first 10 epochs, we warm-up the learning rate linearly from the initial Ir = 10 to the base learning rate (Ir 0.3 x BatchSize (innhSiiee/25)) The network updating coefficient fixed as 0.99. We train SNCLR using 32 NVIDIA A100 GPUs with a batch size of 4096. utilized for training ViTs. We
Unlike the above training protocols, several main components leverage the AdamW optimizer (Loshchilov & Hutter, 2017) with a weight decay of 0. for training. The base learning rate is configured (Ir 1.5 x 10-1 BatchSize (256) with linear warm-up of 40 epochs. Besides, we follow the previous works (Caron et al., 2021; Chen et al., 2021) to configure the total training schedule 300 epochs. The other configurations (e.g.. batch size, cosine annealing learning rate, etc.) are identical to the ones utilized in training ResNets. During the initial training stage, we observe that neighbors are not correctly identified. This may be due the premature feature representations from the encoder. To mitigate incorrect support neighbor selections, we degrade Eq. 2 to Eq. 1 during the first m training epochs by not using neighbors. Then, we resume using Eq. 2 in SNCLR pretext training.
B VISUALIZATION OF NEIGHBORS
In this subsection, we present extended examples with more neighbors in Fig. 4. It's empirically found that the neighbors consistently share the related information/content with the input samples (e.g., the 'bananas' in the first group and the 'monkey' in the second group). We also observe that even though the input images consist of multiple objects (e.g., the fruits in the third group), our proposed methods are still able index the eighl--oorrelatd instances, including related objects. The support from such strong correlated neighbors facilitates our SNCLR in the pretext training stage to produce more discriminating features.
14

---

Published as a conference paper at ICLR 2023
Figure 4: Neighbors display. We show the input instance on the first column. Unlike Fig. 3, there are 10 neighbors in total presented with the ranking from highest positiveness score to the lowest score.
15

---

Published as a conference paper at ICLR 2023
Table Analysis on the memory overhead and computational complexity. We adopt the ViT-Small structure for comparison. All the experiments are tested on 8 NVIDIA A100-40G GPUs.
Set Length
8000
16000
32000
64000
128000
Additional GPU Memory (MB)
+8.19
+16.38
+32.77 2.755
+65.54
+131.07
Seconds per batch
2.687
2.717
2.824
2.883
C ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
C.I MEMORY USAGE AND COMPUTATIONAL COMPLEXITY
Since a large pool of neighborhood set should be maintained in the proposed SNCLR, we study the empirical computation cost and memory usage with respect to the set length. The additional GPU memory and training speed are chosen as the evaluation metrics. All the experiments are tested with ViT-Small on 8 NVIDIA A100-40G GPUs. For the training speed, all the time duration is measured in seconds averaged over 20 epochs. The results in Table 6 show that the computational overhead increases marginally with the set length. Specifically, when the set contains 128000 candidates of which the dimension is 256, it additionally costs 131.07 MB of GPU memory (i.e., 30420 MB 30551MB (+0.43%)). Considering the performance gains brought by the bigger set, such computational complexity is acceptable.
C.2 ABLATION STUDIES ON THE INDIVIDUAL BENEFIT OF NEIGHBORS
We study the individual performance of neighbors on either positive instances or negative instances. Specifically, three different strategies adopted to train ViT-Small for 300 epochs, including I) applying the soft neighbors on the positive instances; 2) applying the soft neighbors on the negative instances; 3) applying the neighbors both positive and negative samples. Experiments summarized in the Table show that implying the neighbors on both positive and negative samples achieves the best result in the linear probe evaluation. Besides, implying neighbors either positive negative samples leads an inferior performance compared with the aullyimplemented SNCLR, while being better than the baseline results. It indicates that the full implementation enables leverage more instances for discrimination and thus brings two advantages: 1) it enriches the diversity of positive instances for adaptive clustering; 2) introduces more negative samples for sufficient discrimination.
Table 7: Analysis on the individual benefits of soft neighbors.
Backbone
Positive
Negative
Top-1 (5)
VIT-Small
X
X
71.17
ViT-Small
V
X
71.88 (+0.71)
ViT-Small
X
/
72.43 (+1.26)
ViT-Small
/
/
73.38 (+2.21)
C.3 ANALYSIS ON THE SNCLR
We provide a conceptual comparison between previous methods (i.e., SwAV Caron et al. (2020), NNCLR Dwibedi et al. (2021)) and ours to illustrate why SNCLR performs better. The differences between ours and existing methods are summarized as follows: 1) Unlike SwAV Caron et al. (2020) that utilizes the fixed prototype to include the semantic-correlated samples, we explicitly compute the correlation/similartyy extent via cross- cross-attention and further encoder the correlation/similarryy into the training objective. To this end, the proposed SNCLR framework is able to adaptively distribute the representations according to the sample semantics during the training process. 2)Compared with NNCLR Dwibedi et al. (2021), SNCLR abandons the traditional binary instance discrimination. Instead, SNCLR first includes more positive neighbors (larger than one) and adopts soft discrimination fully support the current instance via cross-atted cross-attention similarities. To summarize. SNCLR explores an adaptive and soft way to treat the images that correlate similarly to the current instance, thus being more effective in the contrastive learning.
16


Reviews:
[{"review_id": "t-wKX6kqbj", "reviewer": "Reviewer_v9QH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a soft neighbor contrastive learning method (SNCLR) in the self-supervised learning(SSL) area. By pointing out the harshness of the binary positive and negative discrimination of the contrastive loss, the author introduces the soft positive by sampling the soft neighbor concept with the positiveness value. As the result, extensive experiments support the new method to achieve state-of-the-art results in the public benchmarks.\n ", "summary_of_the_review": "Overall, the paper is in good shape. Please take a look at the weakness. ", "strength_and_weaknesses": "Strength\n\nThe paper is clearly written and well organized. \nAlthough the major idea is extended from the existing works of NNCLR, it still need to be merited.\nThe experiments are well conducted and the details are clear.\n\nWeakness\n1. The difference between equation 2 and the common contrastive loss is the NN extension in both Positive and Negative terms. In the ablation study, it is better to show the individual benefit of the NN. For example, in three experiments: a NN extension is only applied on the Positive. b NN extension is only applied on the Negative c NN extension is applied on both.\n2. Since the performance gain in SSL is tend to be saturated, it's better to provide a standard deviation of the accuracy number for some gap which is less than 1% gain.\n3. In section 3.2, \"We have empirically tried using different structures of this attention module, including using parametric linear project..\" comparing to complex computation, how about did some positive mining, a common method used in Deep Metric Learning [i]. which may be more efficient?\n\n4. Some minor issues in the paper\nSection 3.1 \n\"For other N instance..\" should be N-1 instance.\n\"We can also use only positive samples and substitute y1 with z1..\" please define z1 in advance.\n\n\n[i]Xuan, H., Stylianou, A., & Pless, R. (2020). Improved embeddings with easy positive triplet mining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2474-2482).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "sVomUv3ev8", "reviewer": "Reviewer_EMnc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A contrastive learning based self-supervised visual representation learning method is proposed in this paper. Upon observing the positive and negative assignments of training samples, this paper proposes to construct a candidate neighbor set where top candidate neighbors are selected as positive samples. When contribution of these positive samples are measured via cross attentions and defined as positiveness. These samples are denoted as soft neighbors to constitute the contrastive loss computation process. Experiments on several benchmarks and evaluation protocols indicate the feature representations of the proposed SNCLR perform well.", "summary_of_the_review": "Overall, this is a good exploration on mining positive samples to support the contrastive learning computation. Sufficient experiments demonstrate the SNCLR is effective on both CNN and ViT encoders via various training settings and datasets.", "strength_and_weaknesses": "+ The observation that correlated samples are labeled oppositely largely exists in contrastive learning. This is a fundamental issue and less touched in the literature. Compared to the prior SwAV and NNCLR, the proposed SNCLR adaptively mines correlated samples in the candidate neighbor set. The mining process is based on the cross-attention module which effectively measures the sample similarity during the loss computation phase. Both the samples and their positiveness are delved and formulated into the contrastive learning process.\n\n+ Extensive experiments on standard benchmarks, with linear probe, semi-supervised training, transfer to detection and segmentations, with CNN and ViT backbones. Sota comparisons are performed as well. The overall writing is clear and the visualizations / figures are intuitive to represent the algorithm motivations and effectiveness.\n\n-Compared to clustering and nearest neighbor selection, which are conducted in SwAV and NNCLR, the proposed SNCLR achieves better performance in the experiments. While claiming the combination of both advantages is an indirect way to illustrate the SNCLR effectiveness, more analysis shall be taken on elucidating why SNCLR performs better, especially using cross-attentions. \n\n-Introducing more terms in Eq. 2 are shown to improve the performance. Any other alternatives to substitute current configurations (e.g., cross-attentions, weighted combination of terms)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TrNLNa1FgU6", "reviewer": "Reviewer_ZPKB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to improve self-supervised contrastive learning by leveraging soft neighbours. The main idea is to maintain a candidate neighborhood set and retrieve K nearest neighbors for each sample as its positive examples. A positiveness score is calculated using an attention module and used as a soft weight of the neighbor. Experimentally, they show the proposed method improves the feature representations for many downstream tasks.", "summary_of_the_review": "This paper proposes to leverage soft neighbors to improve contrastive learning, which is new and important.  The experiment also shows promising results. Therefore, I would recommend an accept.", "strength_and_weaknesses": "Strength \n\n+Using soft neighors is new in contrastive learning. The idea is well motivated and makes a lot sense. \n\n+The experimental evaluations are sufficient and shows better performance than SOTA\n\nWeakness\n\n-There is no analysis on the memory complexity of the algorithm because a large pool of neighborhood set should be maintained\n\n-How does the size of the neighborhood set affect the performance? \n\n-The NNCLR result reported in Table 1 (a) is 74.5, while the original NNCLR paper reports 75.6. What is the reason behind that? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster