Title: Enhancing Contrastive Learning with Variable Similarity
Venue: NeurIPS 2025
Accepted: yes
Keywords: contrastive learning

variable similarity
Abstract: Contrastive learning has achieved remarkable success in self-supervised learning by pretraining a generalizable feature representation based on the augmentation invariance. Most existing approaches assume that different augmented views of the same instance (i.e., the *positive pairs*) remain semantically invariant. However, the augmentation results with *varying extent* may introduce semantic discrepancies or even content distortion, and thus the conventional (pseudo) supervision from augmentation invariance may lead to misguided learning objectives. In this paper, we propose a novel method called Contrastive Learning with Variable Similarity (CLVS) to accurately characterize the intrinsic similarity relationships between different augmented views. Our method dynamically adjusts the similarity based on the augmentation extent, and it ensures that strongly augmented views are always assigned lower similarity scores than weakly augmented ones. We provide a theoretical analysis to guarantee the effectiveness of the variable similarity in improving model generalizability. Extensive experiments demonstrate the superiority of our approach, achieving gains of 2.1\% on ImageNet-100 and 1.4\% on ImageNet-1k compared with the state-of-the-art methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel contrastive learning method with variable similarity. The authors argue that different augmentations introduce varying degrees of semantic bias, and therefore, positive samples should not be treated as uniformly similar. They propose a variable similarity objective to dynamically adjust the similarity between positive pairs.\n\nThe key idea is intuitive, which builds an augmentation-aware module to predict the variable similarity based on augmentation parameters. The proposed similarity alignment loss and relative constraint loss enable the model to more accurately reflect the true semantic distance between augmented views, promoting better representation learning.\n\nThe paper supports this idea with both theoretical and empirical evidence. The authors provide a generalization error bound analysis to justify the improved learning behavior under variable similarity conditions. Nowadays, it is unusual to see some theoretical finds in deep learning and self-supervised learning. Empirically, they evaluate this method across multiple downstream tasks, demonstrating consistent robustness.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1.The motivation of this paper is clear and interesting, which points out the semantic similarity between positive samples varies depending on data augmentations. The proposed method comprises of various loss terms - they are intuitive, make sense and well formulated.\n \n2.The experimental results across multiple datasets demonstrate consistent performance gains, reflecting the robustness and generalizability of the proposed method. The validation of its general applicability highlights the method’s potential for broader use.\n\n\n3.This paper provides a clear and well-reasoned theoretical analysis. I appreciate the theoretical evidence of the method, as it enhances the rigor of the paper.\n\n4.This paper is well written, with a clear and logic structure. It is easy to follow the problem that the authors claimed and their solution.\n\n\n**Weaknesses**\n\nThere are some minor weaknesses in my opinion.\n\n1.It seems that CoCor also defines a similar notion of augmentation strength. What distinguishes this paper from CoCor? This has important implications for the novelty of this work.\n\n2.Although the method aggregates the minimum similarity across different augmentation types, it is not clear how individual augmentation operations affect the predicted similarity. The authors could conduct experiments on this.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Contrastive Learning with Variable Similarity (CLVS), a novel extension to existing contrastive learning frameworks that dynamically adjusts the similarity objective between positive pairs based on the strength of their data augmentations. \nBy incorporating an augmentation-aware module to predict semantic consistency and enforcing both an alignment loss (to match predicted similarity) and a relative similarity constraint (to ensure stronger augmentations yield lower similarity), the authors demonstrate significant improvements over state-of-the-art baselines on multiple benchmarks. \nIn particular, CLVS achieves up to +6.1% gain on ImageNet-100 with MoCo and +5.9% with SimSiam, as well as consistent improvements across transfer learning, detection, few-shot classification, and retrieval tasks.\n\nCLVS addresses a meaningful limitation in contrastive learning with strong theoretical and empirical evidence, and its general-purpose design promises impact across a wide range of self-supervised tasks.\n\nSTRENGTHS AND WEAKNESSES:\nI found the paper has several strong points:\n\n$Strengths$\n\ni). Clear Motivation & Novelty. \n\nThe observation that aggressive augmentations can introduce semantic distortion is well-motivated, and modeling similarity as a continuous variable rather than a binary constant is both intuitive and underexplored in prior work.\n\nii). Theoretical Support. \n\nA rigorous generalization error bound analysis (Theorem 1) shows how the additional loss terms tighten the contrastive learning objective’s generalization gap, providing solid theoretical backing.\n\niii). Comprehensive Experiments. \n\nExtensive evaluation on ImageNet-100/1k, seven transfer datasets, PASCAL VOC & COCO detection, three few-shot benchmarks, and retrieval tasks convincingly demonstrate CLVS’s broad applicability and robustness.\n\niv). Ablation & Compatibility. \n\nDetailed ablations dissect the contributions of each loss component and the minimum-similarity strategy, while integration with various baselines (SimCLR, BYOL, SwAV) shows the method’s versatility.\n\n\nI also want to list some questions and weaknesses here:\n\n$Weaknesses$\n\ni) Clarity of Notation (Sec. 3). Equations (4)–(6) introduce several similarity terms (e.g., s_1,2, s^_1,2) in quick succession. A consolidated notation table or a more gradual build-up may help readability.\n\nii) Parameter Sensitivity. While the paper sets weights ω, λ, γ empirically, a brief sensitivity analysis (e.g., grid search results) would help practitioners understand tuning requirements.\n\niii) Negative Sampling Extension. It seems that the current method focuses solely on positive pairs. A discussion or preliminary experiment on extending variable similarity to negative samples (e.g., weighting negatives by augmentation distance) would be a valuable direction.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDifferent from most existing self-supervised learning (SSL) approaches, the authors argue that different augmentation methods can lead to semantic discrepancies. They use variable similarity to accurately characterize the intrinsic similarity relationships between different augmented views, thereby enhancing the performance of existing SSL approaches. In addition to empirical analyses, the authors also offer theoretical analysis to demonstrate the effectiveness of variable similarity.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n(1) The paper is strongly motivated. As the assumption that different augmentation methods can lead to semantic discrepancies is reasonable and interesting.\n\n(2) The authors have provided numerous experiments to prove the effectiveness of their method.\n\n(3) The authors not only provide empirical analyses on variable similarity but also offer theoretical analysis.\n\n\n**Weakness:**\n\n(1) While the authors’ motivation is interesting, the paper AugSelf [1] seems to be the first one to propose a solution to the semantic discrepancies arising from augmentation methods. This is my biggest concern. The authors seem not to have provided a comprehensive discussion on this aspect.\n\n(2) Most self-supervised learning (SSL) papers conduct experiments on ImageNet-1k and typically provide results for 800-1000 epochs. However, the authors only provided the linear evaluation results for the first 200 epochs, e.g., in Tables 1 and 2.\n\n[1] Improving Transferability of Representations via Augmentation-Aware Self-Supervision\n\nQUESTIONS:\n(1) I’m extremely curious about the similarities and differences between authors' paper and AugSelf [1]. As mentioned in **Weakness** (1), it appears that authors' paper and AugSelf share the same motivation. I strongly recommend that the authors could discuss the similarities and differences with AugSelf. This will enable readers to better understand your work. I’ll adjust my review score based on the discussion the authors provide.\n\n(2) Most self-supervised learning (SSL) papers conduct experiments on ImageNet-1k and typically provide results for 800-1000 epochs. However, the authors only provided the linear evaluation results for the first 200 epochs, e.g., in Tables 1 and 2. I’m curious whether the effectiveness of the method proposed by the authors will diminish as the number of epochs increases.\n\n[1] Improving Transferability of Representations via Augmentation-Aware Self-Supervision\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}