Enhancing Contrastive Learning with Variable Similarity
Authors: Haowen Cui
Shuo Chen
Jun Li
Jian Yang
Venue: NeurIPS 2025 spotlight ()
Keywords: contrastive learning
variable similarity
Primary area: general_machine_learning
TL;DR: This paper enhances contrastive learning by introducing the variable similarity of augmented views.

Abstract: Contrastive learning has achieved remarkable success in self-supervised learning by pretraining a generalizable feature representation based on the augmentation invariance. Most existing approaches assume that different augmented views of the same instance (i.e., the *positive pairs*) remain semantically invariant. However, the augmentation results with *varying extent* may introduce semantic discrepancies or even content distortion, and thus the conventional (pseudo) supervision from augmentation invariance may lead to misguided learning objectives. In this paper, we propose a novel method called Contrastive Learning with Variable Similarity (CLVS) to accurately characterize the intrinsic similarity relationships between different augmented views. Our method dynamically adjusts the similarity based on the augmentation extent, and it ensures that strongly augmented views are always assigned lower similarity scores than weakly augmented ones. We provide a theoretical analysis to guarantee the effectiveness of the variable similarity in improving model generalizability. Extensive experiments demonstrate the superiority of our approach, achieving gains of 2.1\% on ImageNet-100 and 1.4\% on ImageNet-1k compared with the state-of-the-art methods.

Enhancing Contrastive Learning with Variable Similarity
Haowen Cui', Shuo 2.3, Jun Li', Jian Yang!* 'PA Lab, Nanjing University of Science and Technology, China
School of Intelligence Science and Technology, Nanjing University, China SCenter for Advanced Intelligence Project, RIKEN National Science Institute, Japan
Abstract
Contrastive learning has achieved remarkable success in self-supervised learning by pretraining generalizable feature representation based on the augmentation invariance. Most existing approaches assume that different augmented views of the same instance (i.e., the positive pairs) remain semantically invariant. However, the augmentation results with varying extent may introduce semantic discrepancies or even content distortion, and thus the conventional (pseudo) supervision from augmentation invariance may lead to misguided learning objectives. In this paper, we propose novel method called Contrastive Learning with Variable Similarity (CLVS) to accurately characterize the intrinsic similarity relationships between different augmented views. Our method dynamically adjusts the similarity based on the augmentation extent, and it ensures that strongly augmented views are always assigned lower similarity scores than weakly augmented ones. We provide a theo- retical analysis to guarantee the effectiveness of the variable similarity in improving model generalizability. Extensive experiments demonstrate the superiority of our approach, achieving gains of 2.1 % on ImageNet- 100 and 1.4% on ImageNet-I k compared with the state-of-the-ar methods.
Introduction
Learning effective feature representations 2 3 a fundamental task in machine learning, with profound implications for various applications, including image classification [45), object detection 7. and segmentation [ (9. recent years, self supe-sueeee learning has emerged as a leading
paradigm for unsupervised visual representation learning [10) [ 12 [3]. Among various pretext
tasks in self-supervised learning, contrastive learning constructs self-supervisory signals by treating different augmented views of the same image positive pairs, enabling the extraction of high-quality feature representations without extensive labeled data. This paradigm has achieved remarkable success, demonstrating its potential to bridge the gap between supervised and unsupervised learning.
"Corresponding Authors
*PCA Lab, Key Lab Intelligent Perception and Systems for Education, School of Computer Science and Engineering, Nanjing University of Science and Technology
Information Ministry of
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(8)
Fixed Similarity
(b)
Variable Similarity
Auginemto Views Negative Samples
Augmented Views
Negrive Samples
Input
Encoder
Stronger
Angmentttion
Cropping
Diysimin
Dissinilar
Similar
Simii
Optimization
Features
Direction
Feature Space
Feature Space
Feature Space
Figure 2: Example of learning framework with the cropping augmentation contrastive learn- ing. Despite the semantic inconsistency between different augmented views, current contrastive learning method equally optimize them towards the direction with the maximum similarity.
Figure I: Illustration of different similarity mea- sure in contrastive learning. (a) Fixed maximum similarity among augmented views of the same instance. (b) Variable similarity (ours) among augmented views of the same instance.
As illustrated in Fig. aa), the core mechanism of contrastive learning is to maximize the similarity of embeddings between different views generated by data augmentations. These augmented views share the same underlying semantic concepts, enabling the model to learn invariant representations and thereby improving its generalization ability [10] [4] [3]. However, the assumption that all augmented views equally valid representations of the original data is still flawed, as the semantic consistency over-cropping augmentation can produce views containing only background, leading significant semantic discrepancies between these views and others. This inconsistency may force the model to optimize the wrong direction, compromising its ability to learn meaningful representations.
between these views cannot always be guaranteed. As shown in Fig. 2
In fact, data augmentations inherently induce a spectrum of semantic consistency between views, rather than binary state of similar dissimilar. As shown in Fig. 2 the progressive loss of critical features becomes more pronounced as the extent of cropping augmentation increases. This observation suggests that the similarity usually varies dynamically, rather than remaining fixed. Inspired by the above observation, we argue that the similarity between augmented views in contrastive learning should be variable according to the extent of applied data augmentations, illustrated in Fig, [b). This variability allows the model to reflect the semantic variation introduced by data augmentations. In particular, the stronger data augmentation leads to significant information degradation, resulting in a lower similarity compared with weakly augmented views.
In this paper, we propose Contrastive Learning with Variable Similarity (CLVS), which models the similarity between views as a variable determined by their augmentation parameters. Specifically, we first build an augmentation-awaat views based on their augmentation parameters. Then, we introduce an alignment objective that constrains the similarity between augmented views to align with the predicted variable similarity, thereby guiding the representations to reflect varying semantic relations. Additionally, the loss function will penalize cases when the similarity of strongly augmented views exceeds that of weakly augmented views. To further validate our approach, we provide a theoretical analysis demonstrating that the generalization error bound of method can be effectively shrunk through the of variable similarity. Our method is general-purpos. technique that can be easily integrated into many existing contrastive learning frameworks enhance their performance. When applied MoCo 16 and SimSiam 12 our approach achieves significant improvements of 6.1% and 5.9% on the ImageNet- 100 dataset, demonstrating its strong generalizability and practical effectiveness.
module to predict the variable similarity between two augmented
In summary, our main contributions are as follows:
We enhance contrastive learning by introducing the variable similarity to capture variations between augmented views, supported by an augmentation-awan module that estimates accurate semantic similarity between these views based on their augmentation parameters.
We theoretically prove that the statistical variance of similarity decreases with the utilization variable similarity, thereby resulting a reduction the error bound of the generalization.
Experimental results conducted on standard benchmarks demonstrate the superiority of our method, surpassing the state-of-the-art methods by 2.1% on ImageNet- 100 and 1.4% on ImageNet- Ik, respectively.

---

2 Related Works
2.1 Contrastive Learning
Contrastive learning has recently emerged as a leading paradigm in self-supervised learning [O
[2] 13 17 18 19), gradually closing the performance gap with supervised learning. The core
principle of contrastive learning is to pull together the embedding features of positive pairs, which different augmented views of the same instance. This aligns with the broader objective of metric 23 24 25), where the goal is to learn representations that preserve semantic similarity in the embedding space. A crucial challenge in contrastive learning is to prevent the representation collapse, where all samples are mapped to the same representation, resulting in a loss of discriminative capability. Several negative-use used contrastive methods [1O
learning 220 21
26
T6 27 tackle
this issue using the InfoNCE criterion (28). This criterion pulls positive pairs together while pushing
negative pairs apart. Negative-free methods [13] 12 29 mitigate the collapse problem by leveraging
only positive pairs and introducing an asymmetric design, i.e., the stop-gradies mechanism [30] Additionally, some works 17 information content.
aim to alleviate the collapse problem by maximizing the
2.2 Augmentation Technique
Data augmentation plays crucial role in many representation learning tasks. It improves the quality and diversity of positive sample pairs, enhances the robustness of learned representations [14] 33), and can help mitigate collapse (34] Recent studies have explored various strategies to leverage data augmentation more effectively. For instance, AugSelf (35) preserves augmentation-aware information through an auxiliary model that predicts the difference between augmentation parameters of augmented views. HAIEV [36] argues that different augmentations should be treated unequally and proposes computing contrastive loss different network layers achieve broader distribution of augmentation invariance. LoGo 337 simultancously incorporates global and local views, encouraging local views to have distinct representations. EquiMod (38] introduces equivariance into contrastive learning, aiming to better capture the augmentation information and improve the robustness of learned representations. Other works focus on leveraging stronger augmentations: CLSA 33 aims to minimize the distribution divergence between weak and strong augmented views, while RényiCI. [40] generalizes representation learning by utilizing Rényi divergence as the learning objective. CoCor [1 focuses on exploiting the strength of the composite data augmentation quantified by frequency, but it does not explicitly model the augmentation parameters. As a result, it cannot ensure the variation within the same augmentation type or ensure that the similarity between augmented views aligns with the extent of augmentation.
While these methods typically treat all positive pairs equally similar, they usually ignore the fact that different augmented views may preserve varying levels of semantic content. To address this limitation, we propose the concept of variable similarity, which better captures the nuanced relationships between views with diverse augmentations.
3 Method
3.1 Preliminaries of Contrastive Learning
Contrastive learning aims to train generalizable feature encoder f R + RRd that maps input data from the original n-dimensiinal space to a d-dimensional feature space. Given an input instance IR" sampled from the dataset ID and distribution of data augmentations T, the training objective is to maximize the similarity between the embeddings 2 f (41 (x:) and ø2 = f12((=) of two augmented views (x) and (z()), where t1, t2 T. This can be formulated as minimizing the following general objective:
Leon (f) = REEDD, (((2,,
where ((.,)) is an empirical loss to evaluate the inconsistency between and Z2. For the traditional contrastive methods [10 11I the InfoNCE criterion (8 is typically used as the loss function:
c 22)(
[(21, x2) = -log
(2)
ΣN csim(=1
3

---

Weak Aug Cuy
Augmentation- Module
22
Parameter Prediction
Variable Similarity
Weak tur
Lye
Similarity-Driven Learming
$1.3
La
Strong
X;
Continoous
Aug firs
A Augmented
Variable
Aug
FCC
Feature Space
Fearure Space
Input
Views
Encoder
Embeddings
Similurity
Model Layer
Figure 3: Overview the CLVS. The encoder f extracts the embedding 2; of augmented views. The augmentation-awan module predicts the variable similarity between two augmented views. The similarity between the embeddings of two augmented views constrained by Lats to approximate the predicted variable similarity during training. Lre enforces the similarity to the weakly augmented view 81,2 larger than that to the strongly augmented view $1,3. By incorporating Ca, the augmentation- aware module &p gains the ability to perceive augmentations.
where .im(:;)) is the similarity measurement function, such as cosine similarity. r is temperature hyperparameter and N is the number of negative samples in mini-batch. This encourages the model to pull positive pairs closer while pushing negative pairs apart in the embedding space.
3.2 Variable Similarity in Contrastive Learning
To address the limitation of enforcing a fixed similarity for positive pairs in contrastive learning, we propose CLVS. Unlike traditional methods, CLVS dynamically adjusts the similarity between augmented views based on the parameters of applied augmentations. This dynamic adjustment enables the model to better capture semantic relationships and produce more robust representations.
Augmentation-Awant Learning. Our key idea to make the similarity between augmented views of an input instance consistent with the extent of their augmentations. Let w, (w!..., w(i denote the augmentation parameters for transformation tw., where the vector w represents the parameter of the j-th augmentation type (e.g., cropping, color jittering), and is the number of augmentation types. Inspired by (35,, we introduce an auxiliary task predict the differences in augmentation parameters. Specifically, for the j-th augmentation type, the difference between two and tw: is defined d;,2 = w w/. To make the learned representations sensitive to these differences, we build an augmentation-awwrr module & = (4p ..... sp"), where each &p IRd + R is implemented as a Multi-Layer Perceptron (MLP) for generating augmentation-awwal embeddings. Here, m is the embedding dimensionality. Each so projects the feature embeddings 21 and #2 (produced by the encoder f) into an augmentation-awaan latent space, yielding h = sp (z1) and hr o (z2). The concatenated embedding of h and h then fed into a predictor ęb² to estimate
the parameter difference di This prediction task encourages the features to retain augmentation- aware information, allowing them to better reflect the extent of augmentations, and thereby laying the foundation for variable similarity estimation. The learning process for this auxiliary task is summarized by:
Calf. Σ 5-
(concatcc) (x))), d.
(3)
The loss function & dynamically adjusts depending on the type of data augmentation.
Variable Similarity Alignment. Building the above auxiliary task, calculate the variable similarity through an adaptive selection mechanism governed by 4. In this paper, we use 8J,j to denote the predicted variable similarity, and s,, for the similarity between encoder outputs of two augmented views. For each augmentation type j. we calculate the similarity between h and h as:
3 Forr details, l, || d',2l l2 augmentations with parameters, e.g., cropping. color jittering. di log( (i d( log(1 2i for augmentations without parameters, e.g., flipping, grayscaling.
(og((,,))

---

(x))) (22)ll2), i.e., sim(".) represents cosine similarity, where || ll2 denotes the E2 normalization. This is motivated by the intuition that smaller augmentation parameter differences indicate more consistent transformations, and thus a higher semantic similarity. Since different types of augmentation have varying impacts on the similarity, we aggregate the similarities across all augmentation types, To ensure that the variable similarity captures the most significant discrepancies between views, we calculate the minimum similarity across all augmentation types. The variable similarity between two augmented views defined as:
$122 sim(h), (nn,h,,,a h2) - (z1)
$ =
min sim((p' (z1), (z2)).
(4)
(E()....)
To integrate the above predicted variable similarity into the contrastive learning framework, we align the cosine similarity between the embeddings of two augmented views with the predicted variable similarity via an lg loss:
Cati (f, Fe Partwwnny an (f 81.2) =Ea twj w2
min (E(....)))
sim(yp' (z1), (21)) (z2)) (aml)))smmm(m)) (11 z2)
By minimizing Cani, the model learns to adaptively adjust the similarity objective for each positive pair based on the predicted variable similarity, addressing semantic inconsistencies caused by data augmentations.
Relative Similarity Constraint. To ensure the validity of the predicted variable similarity, we introduce a constraint based on the observation that stronger augmentations cause greater information degradation, thereby reducing the semantic consistency between augmented views. Specifically, the similarity between view and its strongly augmented version should be lower than its similarity to weakly augmented view. Given a stronger augmentation twy. the relative similarity constraint can be formalized follows:
Cre ()) = Ea Loon ($1.2, $1,3)]
E2z,tr, w2ying (max(0, min (eim((((zz): (21), min (s)m))))(21), (z2)))) (6)
jEl.....e
jEl.....
Here, $1,3 represents the predicted variable similarity between the weakly augmented view lue (a) and the strongly augmented view ww (I)) The max operator in Eq. (6) ensures that the loss is only activated when the similarity to strongly augmented view exceeds that to a weakly augmented view, thus enforcing the desired relationship between augmentation strength and semantic consistency.
The overview of CLVS is shown in Fig. 3 The total training loss consists of four components: the base contrastive loss, the parameter prediction loss, the alignment loss for variable similarity, and the constraint loss for augmentation consistency, Formally, the total loss is defined as:
Lrotal Ccon (f) + wL,(f, sp, b) + XLats (f, sp) + (f
where w. A. and y are positive weight coefficients that balance the contributions of different loss terms. By combining these losses, our method preserves the core objective of contrastive learning while addressing the limitation caused by information degradation during data augmentation, resulting in more robust and semantically meaningful representations.
3.3 Theoretical Analysis of Variable Similarity
In this section, aim to demonstrate that our proposed learning algorithm enhances the General- ization Error Bound (GEB) 42 compared to traditional contrastive learning methods. The GEB typically characterizes how well model trained on empirical data performs on unseen samples. For self-supervised learning, although the learned encoders are used different recognition tasks, such an error bound of the learning objective can still provide quantitative result evaluate the reliability of the encoder on unseen test data. This is because that the lower generalization error is expected to bring about the smaller InfoNCE loss the test data, and thus the corresponding feature discriminability during test phase can be ensured. Here, we establish that the proposed new loss terms La. Catts and Ere contribute to tightening the GEB, thereby validating the efficacy of our method.
To provide a more detailed understanding, we consider the underlying data distribution 2, and intro- duce the expected risk, defined as Erotal (f, p;; ") - [Eeeele [Eroaa 99; (ta (.)))))) The ex- pected risk represents the true objective we aim to minimize, while the empirical risk Crotel (f, sp; 9)
5

---

Table 1: Top-1 accuracies of linear evaluation (%). All compared methods with ResNet-50 encoder are pretrained for 200 epochs on the ImageNet-100 100 dataset.
Table 2: Top-1 accuracies of linear evaluation (%). All compared methods with a ResNet-50 encoder are pretrained for 200 epochs on the ImageNet-1k dataset.
Method
Batch Size
Top-1
Method
Batch Size
Top-1
MoCo
[6
256
78.8
SimCLR 10
4096
68.3
SimSiam 12
256
79.1
MoCo [6
256
67.5
AugSelf 35
256
80.5
SimSiam [2
256
70.0
RényiCL 40
256
82.1
AugSelt [35
256
69.4
EquiMod 38
256
83.9
SwAV 26
4096
CoCor 41
256
83.7
EquiMod 38
256
67.6
GCA 44
256
72.4
GCA 44
256
56.1
INTL 32
256
83,9
INTL 32
256
69.9
MoCo+CLVS
256
(1.9
MoCo+CLVS
256
69.9
SimSiam+CLVS
256
(6.0
SimSiam+CLVS
256
71.4
Table 3: Linear evaluation accuracies (%) various datasets. All compared methods with ResNet- 50 encoder are pretrained for 200 epochs on the ImageNet-100 dataset.
Method
CIFARIO
CIFAR100
Caltech101
SUN397
Food
Flowers
Pets
MoCo [6
85.81
64.96
85.78
48.73
62.37
84.01
69.15
SimSiam
22
86.52
65.98
87.21
49.68
62.67
84.09
69.28
AugSelf
35
88. 10
68.48
88.95
50.65
65.56
88.60
71.93
RényiCL.
40
86.60
64,92
87.21
48.45
61.89
85.33
74.87
EquiMod
87.86
69.59
90.37
52.85
65.71
89.88
75.88
CoCor
86.89
66.36
87.85
49,92
62.48
87.95
76.04
GCA 45
76.69
51.87
72.24
37,20
44.96
68.21
49.55
INTL
87,55
67.79
88.77
51.62
63.05
87.75
75.22
MoCo+CIVS
87.29
67.22
88.93
65.07 (+2:20)
87.48
77.65
SinSiam+CIVV
89.92 (+3.am
71.37 (5542)
91.73 (+4.5))
54,80 (65.2))
68.97 (66,3%)
91.12
77.16 (47.55)
computed from finite dataset. The gap between these two quantities, known as the generalization error, reflects the model's ability to generalize beyond the training set.
Theorem 1. For any {f , learned from the objective Crotat (S, up) and any given constant ô (0, 1), we have that with probability at least
Prorat (f, sp; 9) Eronar(f (f.4. 2)1 S C1/(C2 + wo)(o(1 + p(), +)) [n(2/8)1/(2N),
where p(),7) = (ig CN > 0 is monotonically decreasing w.r.t and 7, Here C1/(C2 + ww) is monotonically decreasing w.rl. w, where C1, C2 > O are constants that independent o and 4p.
From the result presented in the existing work [43 which highlights the relationship between larger data volumes and improved model generalization. This decrease reflects the statistical principle that larger datasets provide more reliable approximations of the underlying data distribution, thereby reducing the generalization gap. Meanwhile, observe that the error bound becomes tighter as loss effectively constrains
the parameter w increases. This is because that the augmentation-sseciiie the consistency between similarities, thereby enhancing the model generalization ability. More importantly, we can find that such an error bound becomes lighter as À and 7 increase. Actually, this is due to the further decrease of the variance p().7) = (siy 3)/C², induced by the parameters À and 7. Intuitively, since the similarity values are constrained to be floats between 0 and I rather than those extreme values of either 0 or 1, they are more likely located in the central area of [o, 1]. and thus the variance is naturally reduced. Notably, our theoretical analysis is task-agnostic and focuses the generalization behavior of the training objective itself, rather than any specific downstream application.
In summary, the above theorem demonstrates that our method successfully improves the generalization performance of conventional contrastive learning algorithms by leveraging both increased data and the new loss terms.

---

Table 4: Transfer learning results VOC and COCO object detection tasks. All compared methods with ResNet-50 encoder are pretrained for 200 epochs on the ImageNet-100 dataset.
VOCU7+12
COCO
Method
APRo
APG
A133
APt
SM
MoCo
16
40.87
69.78
41.41
37.29
56.36
40.14
20.63
41.63
51.54
SimSlam
12
39.83
67.27
40.09
37.24
56.61
40.25
21.21
41.78
50.26
Augseli
35
46.96
73.88
50.27
37.91
57.10
41.03
21.58
42.44
51.36
RénysCL
40
41.59
72.76
41.63
37.82
57.16
40.46
21.90
42.37
51.31
EquiMod
88
32.24
63.17
28.05
35.30
54,95
37.82
18.11
39.65
49,03
CoCor
80.60
34,80
17.53
51.74
GCA 64
30,80
60.80
27.39
34.93
$4.04
37.17
20.09
39.05
48.34
50.47
78.64
38.14
57.05
41.07
20.46
McCo+CLVS
52.22
79.43
56.87
39.35
58,90
42.91
22.06.114h
44,02
54,75 63.30
SImSlam+CIVS
(3130) 19.13 58, 38.39(.11.3)
(11.30)
38 10:.00
57.51 (AU0
41.37 (-112)
21.71
(on2t)
42.58 (1190)
52.52 (12.0)
Table 5: Few-shot classification accuracy with 95% confidenceinterral averaged 2000 episodes on FC100, CUB200 and Plant Disease. (N, K) denotes N-way K-shot task. All compared methods with ResNet-50 encoder are pretrained for 200 epochs on the ImageNet-100 dataset.
FC100
CUB200
Plant
Method
(5, 1)
(5.5)
15)
(5, 5)
(5. 1)
(5,5)
16
38.84-0.39
54.53=0.38
41.63-+0.46
55.96:10.46
68.42:10.49
86.22:40.34
SimSiam
12
3.377
53.83=0.39
39.31+:043
55.14:00.47
71.00=0.50
88.39=0033
Augself
85
40.73:10.41
58.41=0.39
42.50:10.44
59.9710.45
73.66:0.49
90.23:10.31
RényiCL.
40
37.7910.36
55.75=0.39
42.30:10.47
58.79:10.46
73.51=0.466
89.67=10.11
EquiMod
38
44.83-0.441
62.7620.40
44.8110.46
62.92-10446
77.21
91.16=10.25
GCA 44
30.34:1..33
44.68=0.37
37.61=0041
52.68-10.45
64.63--0.51
86.03=10.33
INTL.
44.96+00.40
62.49=0.40
44.79++0.48
62.27-+0.48
77.05
91.92-+0.28
MoCo+CLVS
40.28:10.000..44
56.26=0.39
42.77=40.47
57.98:1-0
71.16:0.49
88. 88.554-0.332 (233
SinSiam+CCL.S
45,30 0..42 62,97 0.39
47.23 0.48 (-ters
63.57
0.47 weutt
78.70
-0,45
92.11 ++0.8 s0
4
Experiments
In this section, we first present the implementation details of the experiments. Then, we perform extensive experiments downstream tasks and compare method with existing state-of-the the-art methods. Finally, we provide ablation studies of the proposed method.
4.1 Implementation Details
We pretrain the standard ResNet-50 4 encoder on the ImageNet- and ImageNet- 1k 45 datasets. The encoder is pretrained for 200 epochs with the batch size of 256. Each model in & is implemented with 2 fully connected layers, and each predictor in consists
single fully connected layer. 1 in the training loss, respectively. To evaluate
We set w 0.5 following 335 À = 0.5, and effectiveness of CLVS in modeling variable similarity between positive samples, we adopt both the negative-used contrastive method MoCo [16] and the negative-free method SimSiam 12 as basclines.
including random cropping, color jittering, horizontal flipping, grayscale conversion, and gaussian blurring. For strong augmentations, we integrate RandAugment [46] into the weak augmentation, which automates the selection of augmentation types and their magnitudes. The parameters of augmentations are set according to [3].
For weak augmentations, we employ standard augmentation strategies 12
4.2
Main Results
In this subsection, we present experimental results on various downstream tasks. We compare CLVS with state-of-the-arn contrastive learning methods. For clarity, the best results highlighted in bold, while the second-best results are underlined.
Comparison Linear Evaluation. We initially pretrain CLVS and perform linear evaluation on the ImageNet- 100 and ImageNet- Ik datasets. The linear evaluation protocol follows the approach of [2 The experimental results are shown in Tab. I and Tab. 2 respectively. Our proposed method achieves
7

---

Table 6: Comparison with methods using stronger augmentations. Top-1 linear evaluation accuracies (%) and few-shot classification accuracy (%) are reported. All compared methods with a ResNet-50 encoder pretrained for 200 epochs the ImageNet- 1k dataset.
Linear evaluation
5-way
I-shot
5-way 5-shot
Method
ImageNet
Caltech101
PC100
Plant
FC100
Plant
CLSA 33
72.2
91.21
34.46
67.39
49.16
86.15
RényiCI 40
72.6
94.04
36.31
80.68
S3.39
94,29
CLVS
72.9
94.60
53.33
81.04
73.14
95.17
Query
Nearest Neighbors
Query
Nearest Neighbors
AugSelf
Ours
Figure 4: Visualization of retrieval results on the flowers and cars datasets. Red boxes indicate incorrect retrieval results, and green boxes indicate correct retrieval results.
Top-1 accuracy of 86.0% and 71.4% on the ImageNet-100 and ImageNet- 1 datasets, significantly surpassing state-of-the-arr methods. We also provide comparison of training time among CLVS and other methods in Appendix [D]
We further evaluate the performance of CLVS in transfer learning. We utilize the encoder pretrained on the ImageNet- 100 dataset and conduct linear evaluation on datasets: RIFARIO/100 [47), Caltech101 [88. SUN397 [49], Food [50], Flowers (5T), and Pets [52]. The experimental results are presented Tab. CLVS achieves the best performance across all datasets, demonstrating its strong generalization ability and robustness.
Comparison on Object Detection. We also evaluate our method on the object detection task. The encoder pretrained on the ImageNet-100 dataset is converted to a generalized R-CNN detector following [1] and fine-tuned on the PASCAL VOC (53) and COCO (44] datasets. As shown in Tab. CLVS demonstrates competitive performance on both datasets. Although CLVS performs slightly worse than CoCor on the smaller VOC dataset, where CoCor's 's strict similarity ordering is particularly effective in regularizing feature learning, it demonstrates a clear advantage on the larger and more complex COCO dataset. The performance improvement suggests that our proposed variable similarity can learn more generalizable feature representations.
Comparison on Few-shot Classification. In this experiment, we evaluate the performance of different contrastive learning methods on the few-shot classification task. We conduct experiments on FC100 (55), CUB200 [56), and Plant [57 datasets. Following the experimental setting of [35] we freeze the encoder pretrained on the ImageNet- 100 dataset and employ logistic regression for classification. Tab. presents the results of different methods under the 5-way 1-shot and 5-way 5-shot settings. In particular, the CUB and Plant datasets contain fine-graines labels, requiring models to have a strong generalizability to discern subtle differences. CLVS achieves excellent performance across all datasets, which can be attributed to the variable similarity mechanism that enables the model to capture fine-grained features, thereby significantly improving its generalization ability.
Comparison on Retrieval. We validate the effectiveness of our proposed method through visualiza- tion experiments. The image features are extracted by the encoder pretrained on the ImageNet-100 dataset, and the top-4 nearest neighbors are retrieved based on the cosine similarity metric. Fig. 4 presents the image retrieval results on the Flowers [5T] and Cars (58] datasets. The retrieved images of flowers exhibit a smooth transition in color, starting with light-colored flowers and gradually transitioning to darker shades. Similarly, the retrieved results of cars show a progressive variation in object structure, ranging from close-up views to distant perspectives. These reflect the ability of our variable similarity framework capture fine- mraigraieed visual attributes, ensuring that the retrieval results are not only semantically relevant but also visually coherent.
8

---

ImageNet- 100 with 200 epochs
Lrom Lid foots Leve IN-100
CIFARIO CIFARIO0
Top-1 Accuracy
85
79.1
86.5
66.0
80.5 (HU6)
88.1 (+1.0)
68.7
85.5 (+5.4)
(
71.1 (+5.)
80
85.0 (44.9)
70.8 (44.) 89.4
86.0 (15.))
89.9 (13.4)
(1.4
75
SinSiam
100
200
300
400
500
Table 7: Ablation studies for different loss terms of CLVS. All results are reported on linear eval- uation accuracy (%)
Batch Size
Figure 5: Top-1 accuracies of linear evaluation (%) with different batch sizes.
Table 8: Top-1 accuracies of linear evaluation (%) on various basclines. All methods with ResNet-50 encoder pretrained for 200 epochs the ImageNet- 100 dataset.
Method
ImageNet- 100 CIFARIO CIFARI00 Caltech101 SUN397
Food
Flowers
Pets
SimCLR 10
80,30
85,89
64.31
86.82
48.59
61.95
83.59
68.36
SimCLR + CLVS
83.48 (
88.15 (-228)
68.45 (11
89.18 (0220) (4160
88.03 (4:
73.15 (-4.0%)
BYOL 13 BYOL + CLVS
83.24
87
66.95
87.27
49.97
63,89
85,31
75.01
85.80 (-231))
89.86
(-254)
71.27 (4002))
89.92 (42
53.42 (63.5))
67.83 (41340)
89.88 (-0,37)
77.62 (-201)
SWAV 26
75.48
83.87
60.70
84.52
44.81
58.39
79.83
62.58
SwAV CLVS
81.08 (45,009
86.84
(-299)
66.76 180)
88.42 (630
51.14 (40.3)
64.13 (43.44)
87.27 (+7.44)
68.96 (-0.11)
Comparison with Methods Using Stronger Augmentations. To validate the capability of our proposed method in handling stronger augmentations, comparative experiments are conducted with several methods that employ stronger augmentations. Specifically, the proposed method is implemented using RényiCL as the baseline, with the encoder pretrained on the ImageNet- Ik dataset. We perform the linear evaluation on the ImageNet- Ik and Caltech101 datasets, and few-shot classification on FC100 and Plant datasets. The experimental results shown in Tab. [6 demonsstttte that the proposed variable similarity effectively mitigate the semantic inconsistency caused by stronger augmentations, resulting in a significant boost in linear evaluation performance.
4.3 Ablation Studies
In this subsection, we conduct ablation studies to evaluate the robustness of our method. All encoders pretrained for 200 epochs the ImageNet- 100 100 dataset.
Effect of Training Loss. To evaluate the improvement contribution of each loss term, we conduct ablation studies on the ImageNet-100 dataset. The experimental results are shown in Tab. 7 First, we introduce prediction loss based on the difference in augmentation parameters, which encourages the model to perceive augmentations, which yields slight improvement. Second, the alignment loss Latt drives the model to adjust embedding similarities in accordance with the semantic differences induced by data augmentations. This leads to substantial performance gain, as it directly aligns feature similarities with semantic similarity. Third, the consistency loss Lre constrains the similarity between strongly and weakly augmented views. This further enhances the model's 's representation consistency under diverse augmentations. Combining all these loss terms leads to the best performance, demonstrating their complementary effects and validating the effectiveness of our proposed method. We also conduct experiments to validate the effectiveness of the augmentation-awan module, which can be found in Appendix D
Compatibility with Other Baselines. We further integrate our proposed method into additional baseline models, including SimCLR [10), BYOL [33, and SwAV (26). These methods are pretrained
on ImageNet- 100 and perform linear evaluation on the datasets mentioned in subsection 4.2 The experimental results in Tab. demonsttate that our method consistently improves performance across all baseline models.
Effect of Batch Size. Given that contrastive learning is typically sensitive to batch size [10, we further investigate the impact of batch size on the performance of our proposed method. The experimental results are shown in Fig. 5 When the batch size increases, the accuracy of linear evaluation also improves. Even with a small batch size, our method maintains strong performance, demonstrating its good robustness.

---

5 Conclusion
In this paper, we introduced a variable similarity mechanism to extend the contrastive learning framework, which dynamically adjusts the similarity between views based on the extent of data augmentation. The variable similarity is estimated as the similarity between the refined features encoded by the augmentation-awal be fully considered. By introducing an alignment loss for similarity and relative similarity con- straint between strong and weak augmentations, our method successfully addresses the semantic inconsistency caused by data augmentations. We also provided a theoretical analysis demonstrating that the variable similarity enhances the generalization error bound, validating its effectiveness. Experimental results indicated that the variable similarity enables the model to learn more robust feature representations. However, our method is limited to positive samples and does not incorporate negative samples into the variable similarity mechanism. the future, we plan to extend the variable similarity to all samples to further optimize the contrastive learning framework.
module, so that the variations induced by augmentations can
Acknowledgements
This work was supported by National Natural Science Fund of China (Nos. U24A20330, 62361 and 62506155), and Provincial Natural Science Fund of Jiangsu (Nos. BK20251985)
References
[1] Jiexi Yan, Lei Luo, Cheng Deng, and Heng Huang. Unsupervised hyperbolic metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages I 12465-1244 2021.
[2] Jiexi Yan, Erkun Yang, Cheng Deng, and Heng Huang. Metricformer: A unified perspective of correlation exploring in similarity learning. Advances in Neural Information Processing Systems (NeurIPS), 35:33414-33423 2022.
[3] Jiexi Yan, Lei Luo, Cheng Deng, and Heng Huang. Adaptive hierarchical similarity metric learning with noisy labels. IEEE Transactions on Image Processing, 32:1245-1256, 2023.
[4] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[6] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions Pattern Analysis and Machine Intellignce, (976):117-1111 1149, 2016.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision (ECCV), pages 213-229, 2020.
[8] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 234-241, 2015.
[9] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems (NeurIPS), 117764--78777 2021.
[10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), pages 1697- 2020.
10

---

[11] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9729-99999 7779-,738, 2020.
[12] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 15750-15758, 2021.
[13] Jean-Base BanttBa Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems (NeurIPS), 33:21271 27112228444 2020.
[14] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances Neural Information Processing Systems (NeurIPS), 33:6827-6839, 2020.
[15] Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In International Conference on Learning Representations (ICLR), 2021.
[16] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.04297 2020.
[17] Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learning (ICML), pages 12310-12320 2021.
[18] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: iiiianceinvariacce-covaiii regular- ization for self-supervised learning. International Conference on Learning Representations (ICLR), 2022.
[19] Xing Wu, Chaochen Gao, Yipeng Su, Jizhong Han, Zhongyuan Wang. and Songlin Hu. Smoothed contrastive learning for unsupervised sentence embedding. In Proceedings of the International Conference on Computational Linguistics (COLING), 2022.
[20] Lei Feng, Senlin Shu, Nan Lu, Bo Han, Miao Xu, Gang Niu, Bo An, and Masashi Sugiyama. Pointwise binary classification with pairwise confidence comparisons. In International Confer- ence Machine Learning (ICML), pages 3252-3262, 2021.
[21] Songhua Wu, Tongliang Liu, Bo Han, Jun Yu, Gang Niu, and Masashi Sugiyama. Learning from noisy pairwise similarity and unlabeled data. Journal of Machine Learning Research, 23(307):1-34, 2022.
[22] Jiexi Yan, Lei Luo, Chenghao Xu, Cheng Deng, and Heng Huang. Noise is also useful: Negative correlation-steeret latent contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pages 31 31-40, 2022.
[23] Lei Feng, Senlin Shu, Yuzhou Cao, Lue Tao, Hongxin Wei, Tao Xiang, Bo An, and Gang Niu. Multiple-instance learning from unlabeled bags with pairwise similarity. IEEE Transactions Knowledge and Data Engineering, 35(1 I): 1199119991111 1609, 2023.
[24] Jiexi Yan, Zhihui Yin, Erkun Yang, Yanhua Yang, and Heng Huang. Learning with diversity: Self-expanded equalization for better generalized deep metric learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 19365-19374, 2023.
[25] Jiexi Yan, Cheng Deng, Heng Huang, and Wei Liu. Causality-invarian interactive mining for cross-modal similarity learning. IEEE Transactions on Putlern Analysis and Machine Intelligence, 46(9):6216-66 6230, 2024.
[26] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS), 33:9912-9924, 2020.

---

[27] Chun-Hsiao Yeh, Cheng- Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In European Conference on Computer Vision (ECCV), pages 668-684, 2022.
[28] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 00001100700004 2018.
[29] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in seff-superviee vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) pages 9650-9660, 2021.
[30] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning (ICML), pages 10268-10278, 2021.
[31] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In International Conference on Machine Learning (ICML), pages 3015-3024 2021.
[32] Xi Weng, Yunhao Ni, Tengwei Song, Jie Luo, Rao Muhammad Anwer, Salman Khan, Fa- had Shahbaz Khan, and Lei Huang. Modulate your spectrum in self-supervised learning. International Conference on Learning Representations (ICLR), 2024.
[33] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference Machine Learning (ICML), pages 12121-12132, 2021.
[34] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations (ICLR), 2022.
[35] Hankook Lee, Kibok Lee, Kimin Lee, Honglak Lee, and Jinwoo Shin. Improving transferability of representations via augmentation-awwrre self-supervision. Advances in Neural Information Processing Systems (NeurIPS), 11:17710-1777223 2021.
[36] Junbo Zhang and Kaisheng Ma. Rethinking the augmentation module in contrastive learning: Learning hierarchical augmentation invariance with expanded views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16650- 16659, 2022.
[37] Tong Zhang, Congpei Qiu, Wei Ke, Sabine Süsstrunk, and Mathieu Salzmann. Leverage your local and global representations: A new self-supervised learning strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16580-16589, 2022.
[38] Alexandre Devillers and Mathieu Lefort. Equimod: An equivariance module to improve visual instance discrimination. In International Conference on Learning Representations (ICLR), 2023.
[39] Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 45(5):5549-556) 2022.
[40] Kyungmin Lee and Jinwoo Shin. Rényiel: Contrastive representation learning with skew rényi divergence. Advances in Neural Information Processing Systems (NeurIPS), 35:6463-6477 2022.
[41] Zihu Wang, Yu Wang, Zhuotong Chen, Hanbin Hu, and Peng Li. consistent representations. Transactions on Machine Learning Research, 2024.
Contrastive learning with
[42] Shuo Chen, Gang Niu, Chen Gong, Jun Li, Jian Yang, and Masashi Sugiyama. marge- contrastive learning with distance polarization regularizer. In International Conference on Machine Learning (ICML), pages (673-1683,202)
12

---

[43] Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. A theoretical analysis of contrastive unsupervised representation learning. In Interna- lional Conference on Machine Learning (ICML), pages 5628 -688-,677-75 2019.
[44] Zihao Chen, Chi-Heng Lin, Ran Liu, Jingyun Xiao, and Eva L Dyer. Your contrastive learning problem is secretly a distribution alignment problem. Advances in Neural Information Processing Systems (NeurIPS), 2024.
[45] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 148-255, 2009.
[46] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pages 702-703, 2020.
[47] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[48] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshop, pages 178 178, 2004.
[49] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3485-3492, 2010.
[50] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. 1ood-101-mining discriminative components with random forests. In European Conference Computer Vision (ECCV), pages 446 461, 2014.
[51] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics & Image Processing (ICVGIP), pages 722-729, 2008
[52] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3498 3505, 2012.
[53] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voe) challenge. International Journal of Computer Vision, 88:303-338, 2010.
[54] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision (ECCV), pages 740-755, 2014.
[55] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in Neural Information Processing Systems (NeurlPS), 31, 2018.
[56] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
[57] Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in plant science, 7:1419, 2016.
[58] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, pages 554 561, 2013.
[59] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint 18X8::77080088888 2017.
13

---

[60] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 558-) 1989.
[61] Ross Girshick. Fast r-cnn. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1440-1448, 2015.
[62] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2961-2969, 2017.
[63] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[64] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9640-9649, 2021.
[65] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations (ICLR), 2019.
[66] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal Machine Learning Research, 9(1 1), 2008.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of this paper are accurately reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In section 5 we discuss the limitations of our work and identify them as valuable future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[Yes]
Justification:
The full set of assumptions and complete proofs can be found in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We claim the details of methods and the experiments settings in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: Code [Yes] DATA [No]
Justification: We will soon open-source the code, and the datasets used are publicly available for download by anyone.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//dddeeebmimiia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All detailed specifications are presented in the section and Appendix B.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We lack sufficient computational resources to calculate error bars for every experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix D.7
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We confirm that our research conducted the paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper don't have obvious societal impacts currently.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite and comply with the licenses of the public datasets we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper does not involve the usage of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy for what should or should not be described.
21

---

A Proof
In this section, we present the proof for theory in the paper. We first introduce the following lemma for proving Theorem
Lemma 1. For independent random variables $2,22..... tn T and a given function T R,
if E (i 1.2...., ......,, n.), the function satisfies
(tt.....,ty tn. w(t1..... "esti...,,...................... to...., txn] S Pi. O. it holds that P( (n) Elw(l,..... 6,)]
then for any given 2e 2p²/ /i. Pi.
m S
Proof. We prove Theorem by analyzing the perturbation (i.e., Ps in the above Eq. 9 of the loss function Cemp.
We denote that
1
N
(sim #2)/
(f: =
log
(10)
N
2j=1e eiim(21 23)/
and
e
sim 22)/"
enim(31, 22)/
J N
log
log
(11)
(sim(z),
@aim (21, 2;1/7
where (a, (0b, ) an arbitrary mini-batch from the sample space. Then we have that
w
gemm(31, 82)/7
sim((z, #2)/7
log
log T
N
eim( 22)/
(e (ffmmx #2)/
(3 2,1111)
1
<
log
N
@ #2) EE== e0tm((11 2,1/ 7 e 82)/7)
(Ci/ + logg 1 + Eic, (8is - 33)/C2)
(12)
2N
where C1, C2 > 0 are constants. Meanwhile, we have
-log csim(2), z2)/ Cemp (f, :0; 2) Eemp (f,4;;) By Lemma] [ we let that for all = 1, ,.
(sim(2),2)//> log 2===1 estm((1)))))
2 (sim(s), #,l/
(13)
P: (Ci/(C2 + w) llog( + -3)/C() -3)
(14)
2N
so that we have
In(2/8)
Comp (f. :0 2) - Comp <Cii/(C + +w))log(1 + Σ
(815 - -3)/C})
2N
(<)
1 2e 222/ /EN P.
2N [
2 1 2e /C2/ Magl ( m)/C))
1 2c
1 2e ln2/))
= 1 - 6,
(15)
(Ci/ (o/(0o+))loo((14.. (01 (
where 7)
and μ / /lm(2/8)/(2N) The proof completed.
2N
22

---

B Pretraining Setup and Evaluation Protocols
B.1 Prefraining Setup
In the experiments, we integrate CLVS into MoCo and SimSiam. Here, we provide the detailed pretraining setup used for each method:
MoCo. The learning rate is set to 0.03 and the temperature parameter for contrastive loss is
0.2. The projector consists of 2 MLP layers with an output dimension of 128. The memory queue size 65536 and the exponential moving average (EMA) parameter is 0.999.
0.05. The projector and predictor consist of 3 MLP layers and 2 MLP layers with an output dimension of 2048, respectively.
SimSiam. The learning rate
B.2 Linear Evaluation
The linear evaluation protocol on ImageNet- 100 and ImageNet- 1 datasets follows 12]. Specifically, we freeze the backbone and train classifier by minimizing the cross-entropy loss using the LARS
59
optimizer.
The linear evaluation protocol for transfer learning follows 355 The training datasets are split into train and validation set, with 90% for training and the remaining 10% for validation. The representation of x center-cropped images are extracted by the frozen backbone. The classifier trained by minimizing the Læ-regularized cross-entropy loss using a L-BFGS [60] optimizer.
B.3 Object Detection
The object detection protocol follows [1I. We train the Faster R-CNN 61 detector with a backbone of ResNet-55-CC [62]. The detector is fine-tuned end-to-end on all layers for 24000 iterations on the VOC dataset and 180000 iterations on the COCO dataset. During training. the image scale is resized to [480, 800) pixels for VOC and [640, 800) pixels for COCO, while fixed scale of 800 pixels is used during inference.
B.4 Few-Shot Classification
The few-shot classification protocol follows 355. We conduct logistic regression using representation
extracted by the frozen backbone from 224 x 224 images in an N-way K-shot episode.
C More Discussion on Related Work
Our method and Augself (35 both argue that the pngmentatiooninvariin assumption in traditional contrastive learning may harm semantic consistency. However, the two approaches address this issue from completely different perspectives. To be specific, AugSelf uses an auxiliary task to only preserve augmentation information in its forward phase. In contrast, our method further explores the augmentation information to explicitly modulate the variable cettraative-supervisis signal for both forward and backward phases, leading to three important technical differences:
Dynamic Similarity Adjustment, We introduce a variable similarity mechanism that dynamically adjusts the similarity supervision between two augmented views. Nevertheless, AugSelf does not incorporate such a dynamic mechanism, and thus cannot explicitly adjust the similarity supervision in the corresponding learning objective.
Relative Similarity Constraint. We further introduce a relative constraint that enforces the similarity between weakly augmented views be reasonably higher than that between weakly and strongly augmented views. This relational structure ensures the important semantic consistency, which is absent in AugSelf.
Augmentation-TT Type Aware Similarity Estimation. AugSelf treats augmentation types equally. In contrast, our method explicitly accounts for the transformation that induces the most significant semantic change and uses the minimum similarity across types as the target. This practice allows the model to focus on the most challenging transformation, leading to robust representation.
23

---

D Additional Experimental Results
D.1 Effect of Augmentation-Awarg Module
We conduct experiments with the augmentation-awalt module y. The experimental results in Tab. 9 demonstrate the effectiveness of yp, which benefits the representation robustness via reasonably predicting the similarity between views caused by different augmentations.
D.2 Effect of Minimum Similarity Strategy
We conduct experiments with different strategies of similarity selection in variable similarity. The perimental results in Tab. IT demonstrate the effectiveness the minimum similarity strategy. These results support hypothesis that focusing the most challenging cases enhances representation robustness.
Table 9: Effect of the augmentation-awar mod- ule in CLVS to the linear evaluation accuracy (%). All methods pretrained for 200 epochs on the ImageNet- 100 dataset.
Table 10: Top-1 1 accuracies of linear evalua- tion (%) with different strategies. All methods with ResNet-50 encoder pretrained for 200 epochs on the ImageNet-100 dataset.
Method
Baseline
w/o yp
4p
Method
Min
Max
Mean
MoCo+CIVS
78.8
83.1
84.5
McCo+CIVS
84.5
84.0
83.8
SimSiam+CLVS
79.1
84.5
86.0
SimSiam+CIVV
86.0
85.4
85.7
D.3
More Epochs
In the aforementioned experiments, we adopt the setup of 200 training epochs from prior work [1] To comprehensively evaluate model performance convergence, we conduct experiments with more training epochs. Specifically, we pretrain both CLVS and comparative methods on the ImageNet- 100 dataset for 800 epochs and perform linear evaluation. shown Tab. [1] our method consistently outperforms comparative methods. These extended results further corroborate and align with the findings presented in Table reinforcing the robustness and superiority of our proposed method.
Table 11: Top-1 accuracies of linear evaluation (%). All methods with ResNet-50 encoder are pretrained for 800 epochs on the ImageNet-100 100 dataset.
Method
ImageNet-
CIFARIO
CIFARI00
Caltech101
SUN3997
Fond
Flowers
Pets
MoCo [6
85.7
87.5
67.0
$1.3
62.8
84.9
SimSiam
81.6
88.1
68.7
89.7
52.1
64.6
88.0
75.3
AugSelf
35
83.3
89.7
71.7
91.4
53.2
68.3
90.5
77.0
RényiCL 40
86.6
90.0
69.8
90.5
$5.4
65.6
89.9
77.9
EquiMod 38
86.2
88.6
70.0
89.6
$4.0
66.7
89.3
79.7
GCA
74.0
75.2
49.7
78.0
37.3
43.6
70.6
52.2
INTL.
86.5
88.3
69.5
$3.6
89.0
79.0
MoCo+CLVS
88.7
88.4
68.3
90.6
54.9
66.7
88.9
78.7
SimSiam=CLVS
87.6
73.4
56.6
91.3
D.4
VIT Backbone
To further validate the effectiveness of our proposed method, we conduct experiments using the ViT 633 backbone. Recently, MoCo-v3 [64] explores training ViT backbone in self-supervised learning framework. Therefore, we implement our method into MoCo-v3. We pretrain the ViT-Small encoder on the Imagenet- 100 dataset for 200 epochs with the batch size of 1024. We follow the experimental settings in MoCo-v3, including Adam W optimizer 655) with linear learning rate warm-up for the first 40 epochs, a momentum of 0.9, and a weight decay of 0.1. A cosine learning rate schedule is applied to the encoder and predictor. The learning rate set to 1.5e-4 and temperature 0.2. The experimental results are shown in Tab. 12 The proposed method with the ViT backbone exhibits better performance across all datasets. This reveals the effectiveness of the variable similarity, which is applicable of the VIT backbone.

---

Table 12: Top-1 accuracies of linear evaluation (%). All methods with ViT-Small encoder pretrained for 200 epochs on the ImageNet- dataset.
Method
ImageNct-100 100 CIFARTO CIFAR100 Caltech101
SUN397
Food
Flowers
Pets
MoCo-v3 [1
78.4
86.2
66.4
82.2
47.1
60.5
83.7
66.9
MoCo-v3 + CLVS
80.7
86.9
68.5
85.3
49.5
63.9
86.8
69.7
D.5
Impact of Individual Data Augmentations
To better understand how different data augmentations contribute variable similarity estimation, we perform an ablation where each augmentation type is applied in isolation within the SimSiam framework. Specifically, we consider Random Cropping. Color Jittering. Gaussian Blurring, and Horizontal Flipping, and evaluate the similarity prediction using only the parameter difference from the chosen augmentation. As shown in Table 13 cropping and color jittering yield the strongest performance, indicating that view differences along spatial and color dimensions provide more informative cues for similarity estimation. In contrast, blurring and flipping lead slightly lower accuracy, likely because they have limited impact on semantic content: flipping only changes orientation and blurring mainly suppresses low-level details. This suggests that augmentations that alter semantically relevant aspects of the image are more effective for guiding similarity estimation.
Table 13:
Linear evaluation accuracy (%) with individual augmentations. The method uses ResNet- 50 encoder pretrained for 200 epochs on ImageNet- with only one augmentation type applied during similarity estimation.
Method
Random Cropping Color Jittering
Gaussian Blurring Horizontal Flipping
SimSiam + CLVS
85.1
84.9
84.4
84.2
D.6 Parametric Sensitivity
We systematically analyze the sensitivity of the loss hyperparameters w, À and y by varying each in 0.1, 0.5, 1. As shown in Table [4] exhibits the clearest trend, i.e., intermediate values consistently lead to the best results, while both small and large values reduce accuracy. The effect where performance peaks 0.5 and slightly declines 1. In contrast, y shows relatively stable behavior across different values, with a marginal gain at = 1, indicating lower sensitivity. Overall, these findings suggest that CLVS is robust hyperparameter variations within reasonable range, with optimal performance typically achieved at moderate settings.
similar,
Table 14: Parametric sensitivity of different hyperparameters in the training loss. Linear evaluation accuracy (%) of ResNet-50 backbone encoder pretrained for 200 epochs on the ImageNet- 100 dataset.
Parameter
Value
MoCo+CLVS
SimSiam+CLVS
0.1
84.1
85.4
0.5
84.9
86.0
83.9
85.1
0.1
84.4
85.8
X
0.5
84,9
86.0
1
84.5
85.2
0.1
84.1
85.8
Y
0.5
84.7
85.8
84.9
86.0
D.7 Training Time Comparison
We provide experiments record the training time of our method and the compared methods. Specifically, we use 4 NVIDIA A A100-SXM4440GB GPUs to train these methods, where the batch
25

---

size is set to 256. As shown in Tab. 15 although our method introduces an extra module sp it only increases the time by around 5% compared the baseline. This demonstrates competitive efficiency in terms of training time.
Table 15:
Time comparison of pro- posed method and compared methods on the ImageNet-1k Ik dataset (in minutes).
Table 16: NMI score our methods and com- pared method to quantify the class separation on the ImageNet-100 100 dataset.
Method
Time epoch
Method
NMI
SimSiam
12
24.22
SimSiam
0.69
RényiCL.
40
24.72
Augself
0.74
EquiMod
38
38.83
RényiCI.
0.68
INTL [32
29.17
CLVS
0.76
CLVS
25.78
D.8
Visualization Learned Representations
We show the t-SNE [66] visualizations of the representations learned by our proposed method and several methods on the ImageNet- 100 dataset. As shown Fig. 6 our proposed method leads to better class separation. This indicates that variable similarity facilitates the model to learn more discriminative feature representations. Furthermore, we also provide the NMI (Normalized Mutual Information) score to quantify the class separation in Tab. [6] Our method achieves the highest NMI score, aligning with the trends observed in t-SNE visualizations.
(a) SimSiam
(b) AugSelf
(c) RényiCI.
(d) CLVS
Figure 6: I-SNE visualizations of image representations from randomly selected 10 classes in the ImageNet- 100 validation set.
26


Reviews:
[{"review_id": "ZFZscvlHVV", "reviewer": "Reviewer_rS1f", "summary": "Different from most existing self-supervised learning (SSL) approaches, the authors argue that different augmentation methods can lead to semantic discrepancies. They use variable similarity to accurately characterize the intrinsic similarity relationships between different augmented views, thereby enhancing the performance of existing SSL approaches. In addition to empirical analyses, the authors also offer theoretical analysis to demonstrate the effectiveness of variable similarity.", "questions": "(1) I’m extremely curious about the similarities and differences between authors' paper and AugSelf [1]. As mentioned in **Weakness** (1), it appears that authors' paper and AugSelf share the same motivation. I strongly recommend that the authors could discuss the similarities and differences with AugSelf. This will enable readers to better understand your work. I’ll adjust my review score based on the discussion the authors provide.\n\n(2) Most self-supervised learning (SSL) papers conduct experiments on ImageNet-1k and typically provide results for 800-1000 epochs. However, the authors only provided the linear evaluation results for the first 200 epochs, e.g., in Tables 1 and 2. I’m curious whether the effectiveness of the method proposed by the authors will diminish as the number of epochs increases.\n\n[1] Improving Transferability of Representations via Augmentation-Aware Self-Supervision", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n(1) The paper is strongly motivated. As the assumption that different augmentation methods can lead to semantic discrepancies is reasonable and interesting.\n\n(2) The authors have provided numerous experiments to prove the effectiveness of their method.\n\n(3) The authors not only provide empirical analyses on variable similarity but also offer theoretical analysis.\n\n\n**Weakness:**\n\n(1) While the authors’ motivation is interesting, the paper AugSelf [1] seems to be the first one to propose a solution to the semantic discrepancies arising from augmentation methods. This is my biggest concern. The authors seem not to have provided a comprehensive discussion on this aspect.\n\n(2) Most self-supervised learning (SSL) papers conduct experiments on ImageNet-1k and typically provide results for 800-1000 epochs. However, the authors only provided the linear evaluation results for the first 200 epochs, e.g., in Tables 1 and 2.\n\n[1] Improving Transferability of Representations via Augmentation-Aware Self-Supervision", "was_revised": true, "final_justification": "I am writing to provide my justification for raising the score of this submission to 5.\n\nUpon careful review and considering the rebuttal, I have come to the conclusion that this paper demonstrates a high level of novelty. The innovative ideas presented in the paper show great potential to contribute to the field.\n\nOne of my major concerns initially was the comparison with AugSelf. However, during the rebuttal process, the authors effectively addressed this issue. They provided detailed explanations and additional analyses that successfully clarified the differences and advantages of their approach over AugSelf. This resolution of the concern has significantly enhanced my confidence in the paper’s quality.\n\nAfter thorough consideration of the overall content, the rebuttal, and the discussions with the authors, other reviewers, and you, I am convinced that this submission has reached the acceptance level. Therefore, I have decided to raise my score to 5.", "rebuttal": "Thank you for your positive and constructive comments! Our point-by-point responses are provided below.\n\n---\n\n**Comment_1:** Similarities and differences with AugSelf.\n\n**Response_1:** Thanks for your comment. As the reviewer suggested, our proposed method and AugSelf both argue that the augmentation-invariance assumption in traditional contrastive learning may harm semantic consistency. However, the two approaches address this issue from *completely different* perspectives.\n\nTo be specific, AugSelf uses an auxiliary task to only *preserve* augmentation information in its forward phase. In contrast, our method further explores the augmentation information to explicitly *modulate* the variable contrastive-supervision signal for both forward and backward phases, leading to three important technical differences:\n\n1. *Dynamic Similarity Adjustment*: We introduce a variable similarity mechanism that dynamically adjusts the similarity supervision between two augmented views. Nevertheless, AugSelf does not incorporate such a dynamic mechanism, and thus it cannot explicitly adjust the similarity supervision in the corresponding learning objective.\n\n2. *Relative Similarity Constraint*: We further introduce a relative constraint that enforces the similarity between weakly augmented views to be reasonably higher than that between weakly and strongly augmented views. This relational structure ensures the important semantic consistency, which is absent in AugSelf.\n\n3. *Augmentation-Type Aware Similarity Estimation*: AugSelf treats all augmentation types equally. In contrast, our method explicitly accounts for the transformation that induces the most significant semantic change and uses the minimum similarity across types as the target. This practice allows the model to focus on the most challenging transformation, leading to robust representation.\n\nWe will add the above discussion in the camera-ready version if this paper is finally accepted.\n\n---\n\n**Comment_2:** Effectiveness of more epochs.\n\n**Response_2:** Thanks for your comment. \n\nWe would like to clarify that the number of training epochs in our experiments follows the same setting as in previous works [R1, R2, R3], where 200 epochs are commonly used for evaluations on both the ImageNet-100 and ImageNet-1k datasets. Moreover, the results of Tab. 11 in the appendix show that our proposed method remains effective when trained for 800 epochs on the ImageNet-100 dataset, indicating that its effectiveness is successfully demonstrated with more training epochs.\n\nFor ImageNet-1k, here we follow the reviewer's suggestion to train our model with 800 epochs by using 8 NVIDIA Tesla V100 GPUs. The training setup follows the baseline method SimSiam, with a learning rate of 0.05, and batch size of 256. The corresponding results are shown in the following table.\n\n| Method | SimCLR | MoCo | SwAV | SimSiam | INTL | Ours |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Top-1 | 70.4 | 71.1 | 71.8 | 71.3 | 73.1 | **73.7** |\n\nAs shown in the above table, our method achieves the best top-1 accuracy on the ImageNet-1k dataset among all compared methods. The experimental results demonstrate the effectiveness of our proposed method with more training epochs.\n\nThanks a lot again for your very insightful comments on our paper! We sincerely hope that our responses can solve your main concerns.\n\n[R1] Contrastive Learning with Consistent Representations, TMLR, 2024.\n\n[R2] Contrastive Learning with Stronger Augmentations, TPAMI, 2023.\n\n[R3] On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learning, ICML, 2024."}, {"review_id": "3MKXl8atcP", "reviewer": "Reviewer_HTKK", "summary": "This paper introduces Contrastive Learning with Variable Similarity (CLVS), a novel extension to existing contrastive learning frameworks that dynamically adjusts the similarity objective between positive pairs based on the strength of their data augmentations. \nBy incorporating an augmentation-aware module to predict semantic consistency and enforcing both an alignment loss (to match predicted similarity) and a relative similarity constraint (to ensure stronger augmentations yield lower similarity), the authors demonstrate significant improvements over state-of-the-art baselines on multiple benchmarks. \nIn particular, CLVS achieves up to +6.1% gain on ImageNet-100 with MoCo and +5.9% with SimSiam, as well as consistent improvements across transfer learning, detection, few-shot classification, and retrieval tasks.\n\nCLVS addresses a meaningful limitation in contrastive learning with strong theoretical and empirical evidence, and its general-purpose design promises impact across a wide range of self-supervised tasks.", "questions": "See the weaknesses.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "I found the paper has several strong points:\n\n$Strengths$\n\ni). Clear Motivation & Novelty. \n\nThe observation that aggressive augmentations can introduce semantic distortion is well-motivated, and modeling similarity as a continuous variable rather than a binary constant is both intuitive and underexplored in prior work.\n\nii). Theoretical Support. \n\nA rigorous generalization error bound analysis (Theorem 1) shows how the additional loss terms tighten the contrastive learning objective’s generalization gap, providing solid theoretical backing.\n\niii). Comprehensive Experiments. \n\nExtensive evaluation on ImageNet-100/1k, seven transfer datasets, PASCAL VOC & COCO detection, three few-shot benchmarks, and retrieval tasks convincingly demonstrate CLVS’s broad applicability and robustness.\n\niv). Ablation & Compatibility. \n\nDetailed ablations dissect the contributions of each loss component and the minimum-similarity strategy, while integration with various baselines (SimCLR, BYOL, SwAV) shows the method’s versatility.\n\n\nI also want to list some questions and weaknesses here:\n\n$Weaknesses$\n\ni) Clarity of Notation (Sec. 3). Equations (4)–(6) introduce several similarity terms (e.g., s_1,2, s^_1,2) in quick succession. A consolidated notation table or a more gradual build-up may help readability.\n\nii) Parameter Sensitivity. While the paper sets weights ω, λ, γ empirically, a brief sensitivity analysis (e.g., grid search results) would help practitioners understand tuning requirements.\n\niii) Negative Sampling Extension. It seems that the current method focuses solely on positive pairs. A discussion or preliminary experiment on extending variable similarity to negative samples (e.g., weighting negatives by augmentation distance) would be a valuable direction.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of the novelty, theoretical analyses, and experimental results of our paper! Thanks also for your very insightful and constructive suggestions! Our point-by-point responses are as follows.\n\n---\n\n**Comment_1:** Clarity of Notation.\n\n**Response_1:** We appreciate the reviewer for pointing out this issue. To clarify, we summarize the meaning and role of each similarity term used:\n\n| Notation | Description |\n| :-: | :-: |\n| $\\hat{s}_{1,2}^j$ | Predicted cosine similarity under $j$-th augmentation |\n| $\\hat{s}_{1,2}$ | Minimum similarity across all augmentations |\n| $s_{1,2}$ | Cosine similarity of raw embeddings |\n\nWe will introduce the above notations with clearer textual explanation accompanying each equation if the paper is finally accepted.\n\n---\n\n**Comment_2:** Parameter Sensitivity.\n\n**Response_2:** Thanks for your comment. The sensitivity analyses of $ \\lambda $ and $ \\gamma $ have already been presented in Tab. 9 and 10 of our manuscript, respectively. The value of $ \\omega $ follows the same setting as in AugSelf [R1]. To further address the reviewer’s concern, we additionally conduct a sensitivity analysis for $ \\omega $, while fixing $ \\lambda = 0.5 $ and $ \\gamma = 1 $.\n\n| $ \\omega $ | 0.1 | 0.5 | 1 |\n| :-: | :-: | :-: | :-: |\n| MoCo+CLVS | 84.1 | **84.9** | 83.9 |\n| SimSiam+CLVS | 85.4 | **86.0** | 85.1 |\n\nThese results indicate that the performance is relatively stable across a reasonable range of $ \\omega $, suggesting that the method is not highly sensitive to this parameter.\n\n---\n\n**Comment_3:** Negative Sampling Extension.\n\n**Response_3:** Thanks for your comment. \n\nOur method focuses on adjusting the similarity objective for positive pairs, as they are directly generated from the same instance via data augmentations. In this case, the augmentation parameters provide a meaningful and grounded basis for modeling semantic similarity.\n\nExtending the concept of variable similarity to negative pairs is indeed an interesting direction. However, we note a fundamental difference: negative pairs typically consist of different samples, and their similarity arises not only from augmentations but also from inherent semantic differences. As such, using augmentation parameters alone to model the similarity of negative pairs may not be sufficient or reliable.\n\nOne potential solution could be inspired by information-theoretic principles: we may estimate the inherent similarity between samples (e.g., using approximations of mutual information), and combine it with augmentation-induced variation to form a more comprehensive similarity measure for negative pairs. This would allow the model to jointly consider intrinsic sample differences and augmentation-induced discrepancies when weighting negative similarities.\n\nWe believe this integration requires careful design and is beyond the scope of the current work, but we appreciate the reviewer’s suggestion and will consider it as a future research direction.\n\nThanks a lot again for your very detailed comments on our paper! We sincerely hope that our responses can further improve the clarity of the paper.\n\n[R1] Improving Transferability of Representations via Augmentation-Aware Self-Supervision, NIPS, 2021."}, {"review_id": "ff34zBs6GQ", "reviewer": "Reviewer_LeRi", "summary": "This paper proposes a novel contrastive learning method with variable similarity. The authors argue that different augmentations introduce varying degrees of semantic bias, and therefore, positive samples should not be treated as uniformly similar. They propose a variable similarity objective to dynamically adjust the similarity between positive pairs.\n\nThe key idea is intuitive, which builds an augmentation-aware module to predict the variable similarity based on augmentation parameters. The proposed similarity alignment loss and relative constraint loss enable the model to more accurately reflect the true semantic distance between augmented views, promoting better representation learning.\n\nThe paper supports this idea with both theoretical and empirical evidence. The authors provide a generalization error bound analysis to justify the improved learning behavior under variable similarity conditions. Nowadays, it is unusual to see some theoretical finds in deep learning and self-supervised learning. Empirically, they evaluate this method across multiple downstream tasks, demonstrating consistent robustness.", "questions": "Please refer to Weaknesses.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1.The motivation of this paper is clear and interesting, which points out the semantic similarity between positive samples varies depending on data augmentations. The proposed method comprises of various loss terms - they are intuitive, make sense and well formulated.\n \n2.The experimental results across multiple datasets demonstrate consistent performance gains, reflecting the robustness and generalizability of the proposed method. The validation of its general applicability highlights the method’s potential for broader use.\n\n\n3.This paper provides a clear and well-reasoned theoretical analysis. I appreciate the theoretical evidence of the method, as it enhances the rigor of the paper.\n\n4.This paper is well written, with a clear and logic structure. It is easy to follow the problem that the authors claimed and their solution.\n\n\n**Weaknesses**\n\nThere are some minor weaknesses in my opinion.\n\n1.It seems that CoCor also defines a similar notion of augmentation strength. What distinguishes this paper from CoCor? This has important implications for the novelty of this work.\n\n2.Although the method aggregates the minimum similarity across different augmentation types, it is not clear how individual augmentation operations affect the predicted similarity. The authors could conduct experiments on this.", "was_revised": true, "final_justification": "My concerns have been addressed, so I will maintain my positive rating.", "rebuttal": "We appreciate the reviewer's positive comments for our paper. We address the concerns raised point by point below.\n\n---\n\n**Comment_1:** Differences of proposed method with CoCor.\n\n**Response_1:** Thanks for your comment. We would like to clarify that CoCor and our proposed method differ significantly in both methodology and design philosophy.\n\nCoCor defines augmentation strength based on the frequency or number of transformations applied to a sample. Specifically, a composite augmentation with more operators is defined as a stronger augmentation. Based on this, CoCor proposed a data augmentation consistency, i.e., the similarity between the augmented view and the original input decreases with the strength of the augmentation. However, the learning of this similarity structure in CoCor is supervised by ground-truth class labels, which are required to guide the alignment of augmented views with their semantic categories. This makes the approach label-dependent and less applicable in fully unsupervised settings.\n\nIn contrast, our method is based on a different perspective of augmentation. Even though positive pairs are derived from the same data sample, different parameterizations within the same augmentation type will induce variations in the resulting views. Therefore, the similarity between such augmented views should not be assumed constant, but rather should vary in accordance with the applied augmentation parameters. We define similarity not based on the number of transformations, but rather based on the parameter variations within the same augmentation type. For instance, two crops with small scale differences are considered more similar than two crops with large scale difference. Our method predicts this similarity via an augmentation-aware module $ \\varphi $ and aligns it with the representation similarity in a self-supervised manner. This enables the model to capture semantic consistency from augmentation parameters, without relying on human-annotated labels. \n\nIn summary, while both approaches utilize augmentation-related cues, our method is fundamentally different in its definition of similarity, learning strategy, and reliance on supervision. We believe these differences not only constitute a clear methodological novelty, but also broaden the applicability of our approach to a wider range of unsupervised learning scenarios.\n\n---\n\n**Comment_2:** Effectiveness of individual augmentation.\n\n**Response_2:** Thanks for your comment. To better understand the effect of individual augmentation types on the predicted similarity, we conducted an ablation study to evaluate each augmentation’s contribution.\n\nSpecifically, we conduct this experiment based on the SimSiam method. We isolate the effect of each of the following augmentations: random cropping, color jittering, Gaussian blurring, and horizontal flipping. When predicting the similarity between two augmented views, we use only the parameter difference from a single augmentation type, effectively assessing its individual contribution to the variable similarity estimation. The following results are obtained:\n\n| Type | Cropping | Color Jittering | Gaussian Blurring | Flipping |\n| :-: | :-: | :-: | :-: | :-: |\n| Top-1 | 85.1 | 84.9 | 84.4 | 84.2 |\n\nThese results reveal several insights: Crop and color jitter yield stronger performance when used alone, suggesting that their parameter differences offer more informative cues about view similarity. In contrast, blurring and flipping result in slightly lower performance. This may be attributed to their limited influence on the image’s semantic content, as flipping alters only the orientation and blurring affects low-level details without substantially changing object-level features. Consequently, their limited semantic impact makes them less effective in providing informative variation for similarity estimation. \n\nThanks a lot again for your positive comments on our paper! We sincerely hope that our responses can further improve the clarity of the paper."}]

Author rebuttal:

Meta-review: This paper studies modifying the SSL by the strength of the augmentation between positive pairs, ensuring sensitivity to semantic biases found in the augmentation transformation. The approach appears to be useful across SSL algorithms / losses, and hence would have high use for practitioners. The reviewers had a few issues, notably with similarities to a few other works that build in sensitivity to augmentation parameters, but the authors well-distinguished their work for those, both in motivation and implementation.

I therefore recommend the paper is accepted as a spotlight.
Decision: Accept (spotlight)