SNAPHARD CONTRAST LEARNING
Authors: Changpu Meng
Jie Yang
Wanqing Li
Yi Guo
Venue: ICLR 2026 Poster (2026)
Keywords: Contrastive Learning
Hard Sample Screening
Contrastive Loss
Computational Geometry
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: In recent years, Contrastive Learning (CL) has garnered significant attention due to its efficacy across various domains, spanning from visual and textual modalities. A fundamental aspect of CL is aligning the representations of anchor instances with relevant positive samples while simultaneously separating them from negative ones. Prior studies have extensively explored diverse strategies for generating and sampling contrastive (i.e., positive/negative) pairs. Despite the empirical success, the theoretical understanding of the CL approach remains under-explored, leaving questions such as the rationale behind contrastive-pair sampling and its contributions to the model performance unclear.
This paper addresses this gap by providing a comprehensive theoretical analysis from the angle of optimality conditions and introducing the SnaPhArd Contrast Learning (SPACL). Specifically, SPACL prioritizes hard positive and hard negative samples during constructing contrastive pairs and computing the contrastive loss, rather than treating all samples equally. Experimental results across two downstream tasks demonstrate that SPACL consistently outperforms or competes favorably with state-of-the-art methods, showcasing its robustness and efficacy. A comprehensive ablation study further examines the effectiveness of SPACL's individual components to verify the theoretic findings.

Published as a conference paper at ICLR 2026
SNAPHARD CONTRAST LEARNING
Changpu Meng', Jie Yang': Wanqing Li', Yi Guo
'University of Wollongong
Westtern Sydney University
cm245@ucwmail.co edu.. au, jiey@uow. edu. au, wanging@uow. .d, guo@westeernsydney. au
ABSTRACT
In recent years, Contrastive Learning (CL) has garnered significant attention due to its efficacy across various domains, spanning from visual and textual modalities. A fundamental aspect of CL is aligning the representations of anchor instances with relevant positive samples while simultaneously separating them from nega- tive ones. Prior studies have extensively explored diverse strategies for generating and sampling contrastive (i.e., positive/negative) pairs. Despite the empirical suc- cess, the theoretical understanding of the CL approach remains under-explored, leaving questions such as the rationale behind contrastive pair sampling and its contributions the model performance unclear. This paper addresses this gap by providing comprehensive theoretical analysis from the angle of optimality conditions and introducing the SnaPhArd Contrast Learning (SPACI.). Specifi- cally, SPACI prioritizes hard positive and hard negative samples during construct- ing contrastive pairs and computing the contrastive loss, rather than treating all samples equally. Experimental results across three downstream tasks demonstrate that SPACL consistently outperforms or competes favorably with state-of- thete--rtr methods, showcasing its robustness and efficacy. A comprehensive ablation study further examines the effectiveness of SPACL's 's individual components to verify the theoretic findings.
INTRODUCTION
Contrastive Learning (CL) has proven be powerful self-supervised approach for reducing the reliance large volumes of labeled data in training deep neural networks. At its essence, CL harnesses self-augmented signals from input data and revolves via optimizing a contrastive loss function. Throughout training, the model generates discriminative features, bringing similar (or positive) samples closer in the representation space while pushing dissimilar (or negative) samples apart. Thanks to these discriminative features, CL has led to notable advancements in various do- mains, including image classification (Mildenberger et al., 2025), relation extraction (Wu et al., 2023), and visual question answering (Zou & Yin, 2025), to name few.
The effectiveness CL is predominantly tied to the strategic utilization of contrastive pairs (both positive and negative examples) to discover informative features. Consequently, extensive inves- tigations have been conducted, resulting in the identification of two primary categories within the literature. One category focuses explicitly on the generation of contrastive pairs. Techniques such as minibatch adaptation, data augmentation, and knowledge-basse methods have been explored to enhance the quality and diversity of these contrastive pairs. Among them, minibatch adaptation, exemplified by the MoCo series and SimCLR, leverages in-batch data and/or a global queue to re- trieve contrastive samples (Koromilas et al., 2024; Lin et al., 2024). Data augmentation techniques introduce variability and richness to the data, employing transformations at both instance and feature levels (Liu al., 2024c; Zheng et al., 2024; Liu al.. 2024c; Zhuo et al., 2024; Lei et al., 2024). Knowledge-batt based methods, the other hand, integrate domain knowledge and/or prior information to generate contrastive pairs, especially in applications such as social recommendation and machine reading comprehension (Animesh & Chandraker, 2025; Lin al., 2024; Li et al., 2024b; Lee al., 2024b; Kim et al., 2024). Alternatively, another category values the sampling of contrastive pairs,
*Correspodiigg authors.

---

Published as a conference paper at ICLR 2026
with a growing emphasis on utilizing hard (positive and negative) samples (Zhuang al., 2024a; Zou & Yin, 2025; Zhang et al., 2024). The sample hardness or difficulty refers to instances that present greater challenges or ambiguity for the model in differentiating them from counter class(es). The integration of such hard samples prompts the model discriminatory abilities via capturing subtle variations within the data. Accordingly, diverse strategies have been proposed to proficiently sam- ple contrastive pairs, including feature importance (Chin et al., 2024), model uncertainty (Liu et al., 2025a), and data bias (Li al., 2024c), among others.
Although generating contrastive pairs is crucial in CL, our emphasis lies in the second category, namely the strategic sampling of contrastive pairs. Despite the empirical success of pair sampling, gap persists in the theoretical understanding. Specifically, questions such as the motivation or ratio- nale behind sampling contrastive pairs. the contribution of hard samples to the model performance, and the conditions for solution optimality/collapee under the circumstances of hard samples remain open. Moreover, most existing research (Shou et al., 2025; Wang et al., 2024a; Wu et al., 2025a) focuses solely on sampling either positive or negative samples, with fewer considering both simulta- neously. Consequently, the selection of both positive and negative samples concurrently introduces another complex perspective. Our work in this paper provides a response to these issues, which is summarized as below:
A theoretical analysis on current Contrastive Learning methods. outlining conditions for optimality, and exploring potential issues such as solution collapse. Based on the analysis, screening- based Contrastive Learning algorithm is proposed to explicitly harness hard contrastive pairs.
Empirical evaluation performed on three representative downstream tasks, namely image classification, Knowledge- hensively assess the effectiveness of SPACI. Furthermore, an extensive ablation study carried out to examine the contribution of each individual component.
link prediction and out-of-domain detection, to compre-
2 DIVE DEEP INTO CONTRASTIVE LEARNING
2.1 CONTRASTIVE LEARNING FORMULATION
A large body of Contrastive Learning (CL) methods adopt dual-encoder architecture, where two encoders separately encode an anchor (query) and its associated positive (key) sample. This general structure underlies several well-known instantiations, such as Momentum Contrastive Learning (He et al., 2020; Chen et al., 2020b; Zhu et al., 2021; Liu et al., 2024a; Chen et al., 2024; Wu et al., 2025b)'. Let vi, /2,,, be the j-th negative/positive sample of an anchor sample z;, and N- be the total number of negative/posiiive samples for z, With a similarity measure by the dot product (i.e., (1) and hyperparameter as the temperature, the following InfoNCE loss is applied,
Σ N exp((f((z).
e" = arg min - Σ log 6
Elog
(1)
ΣN exp((f(z)) slzt))/)++)))))))))))))())) Σ (()((
to optimize the main encoder fz;) with model parameters of θ (omitting θ in f for ease of notation). In Eq. (1), g is the derivative encoder (often omitted during inference) and is often in- tentionally aligned with f. albeit updated differently. This alignment may involve incorporating a (very) small portion of the updated e from f in a momentum- based manner while maintaining other portions. This InfoNCE loss tends to be minimal when f(z)) closely matches its positive counterparts and diverges from negatives.
2.2 OPTIMIZATION
Write x; = f(x)), n, = g(2,,) and Pj = g(2,,,). We first note that the function of the temperature T to change the weighting scheme and does not affect our theoretic results, as verified in the
Unless otherwise specified. our analysis concerns dual-encoder based CL, where two branches produce representations for query-key pair (parameters may shared or partially decoupled via momentum). Meth- ods that use single backhone with asymmetric heads and stop-gradient mechanisms (e.g., BYOL (Grill et al., 2020), SimSiam (Chen & He, 2021)) 1) can often be mapped to the dual- sual-encoderr template by viewing the predic- tor/teacher head as effective second encoder. When such mapping is not appropriate, we explicitly note it in the text.

---

Published as a conference paper at ICLR 2026
derivations. Hence omit in our following analysis for simplicity. We further write sf"
Σj exp(x; Pj) and 8, = Σj N exp(x;, ny)- Then the objective from Eq. (1) is concisely to minimize the below w.r.t θ:
Σ, exp(x,, Pj)
L: Σ log
log $
1
En N
exp(x,, pj) Ej exp(x,, nj)
We point out that although n, and Pj are encoded by the derivative encoder g. constant during the optimization of f, and so are nj and pj w.r.f x, Accordingly, the gradient w.r. θ is
OC 00 = Σ
8:
0x; 1
8 dx;
where O is some tensor product derived from chain rule. In addition, f is typically a deep neural network with considerable complexity. The universal approximation property of neural networks implies the existence of + for arbitrary mappings between z and x such that x = f(z). Therefore, the optimization process can be streamlined by prioritizing the optimal values for x,. This involves first identifying the best x,, and then pursuing them with an appropriate @ as a regression problem. The motivation lies in the geometry of the unit hypersphere. That is, when ny and Py remain constant, the optimal x, corresponding to the minimum of L becomes predictable, or at least some of its properties are tangible. As a result, the stationary point for Eq. (3) (hereafter omitting for simplicity) is given by: ô (s- /s+) /0x = 0. This condition is further equivalent to the following constrained minimization problem:
exp((x, nj)
min |
(4)
Ej exp(x, Py)
or, alternatively, an explicit minimization problem (without constraints):
Σ, exp( x | n,)
min exp
(5)
Σ,
| P.)
We begin with the constrained version, offering a direct geometric interpretation, before showing that the unconstrained version closely resembles its constrained counterpart.
2.2.1 OPTIMALITY CONDITION OF THE CONSTRAINED VERSION
We first provide the following theorem establish the condition for solution collapsing with single positive sample. By "collapsing", we refer to the scenario where x p. indicating that the optimal solution coincides with the positive sample itself.
Theorem 2.1 (Symmetric negatives). there is only single positive sample p. i.e., N 1, the optimal solution x will collapse to the only positive one if and only if the following holds with some C
Σ3 w, C P. Vj, Wy
exp(nnj p)
=
Σ exp(n,.
The detailed proof and illustrations are in A.1.1. The symmetry condition in Theorem 2.1 indicates that when the convex reconstruction of negatives a scaled version of the single positive, the optimal solution collapses. It also implies that collapsing does not occur without symmetry. Consequently, we derive the following non col-collapsing condition as a corollary.
Corollary 2.2 (Non-collapsing condition for single positive). For the same setting as in Theo- rem 2.1, ve e RR (0), cp & conv(n, 3 then x + p. convt} } is given in Appendix Eq. (12).
Proof. This condition is relaxed version of Theorem 2.1, as Eq. (6) explicitly demands the convex reconstruction. When this condition, as stated this corollary, is not met, it becomes impossible to have x p.

---

Published as a conference paper at ICLR 2026
Remark 2.3. The condition outlined in Corollary 2.2 can also be conceptualized the absence of intersection between the projective space of p (the line connecting p and -p) and the convex body formed by all negatives. This condition holds potential when all the negatives collectively occupy less than half of the sphere, for instance, concentrating around a small sphere cap. However, if the negatives encompuss substantial portion of the sphere, the condition becomes less informative, and one must refer to Eq. (6) to assess feasibility.
related geometric visualization provided Fig. 7 (Appendix) offers a clear depiction of the inter- actions among instances. First, the coverage the negatives is primarily determined by the presence of hard samples. The easy ones are mostly concentrated around the opposite side of f(z) or x. The more of these easy ones, the more weights they attract, which is "fixing" force for the potential solution. In other words, nxx would be closer to them and hence more difficult for the solution to break free from them. This suggests filter out some easy negatives. The second is to apply the same logic to the positives. The easy positive instances that are close to [z) also serve as fixation points, expanding the convex hull for p(x). The consequence is not only discouraging variations in the encoder, but also potentially yielding multiple ambiguous solutions. Therefore, we should eliminate the easy samples, both negatives and positives, alleviate these potential issues.
Interestingly, when ||x|| is constrained to be on Smm11, we have the following asymptotic results on the number of negative instances. Proof and illustrations are in A.1.2.
Lemma 2.4. When 8mm11, i.e. ||x the norm of the gradient upper-bounded asymptoti- cally, i.e.
lim
|V 2
+00
Remark 2.5. The proof of Lemma 2.4 and its illustration in Fig. 8 (Apppendix clearly demonstrate that the length of the gradient VxC. is capped by the constant 2. Consequently, having an excessive number of negatives becomes unnecessary. That is, when N surpasses critical threshold, the contribution of negatives becomes negligible. In addition, from Eq. (15) we observe that
2
\V e 1.76.
This represents the maximum gradient length when N = 1 and N+ = 1. As N increases, x|Vl experiences slight increase and rapidly approaches the overall maximum of 2, as pre- dicted in Lemma 2.4.
We leave the optimality condition of the free (unconstrained) version to supplementary A.1.3.
SNAPHARD CONTRAST LEARNING
From the above analysis, it is evident that easy positives and negatives mainly act as fixation points that limit variability, whereas hard samples critically shape the optimization landscape. Motivated by this observation, we focus on the strategic generation and selection of hard positive and negative pairs, leading to our proposed SnaPhArd Contrast Learning (SPACL) method.
Generation and selection of hard positives (P!)). For the i-th original sample ă; we construct a candidate pool C, = (2.} (7) ;..., , 2(M) ), consisting 2, and its M augmented variants.
Prior methods typically adopt either (i) using (Z,, z((m)) (m E [1, M)) an anchor-positive pair, or (ii) discarding 2; and pairing (z(mi) z( ((m2) (m1, m2 E [1, In contrast, we define the hardness of candidate
C, by its average dissimilarity to the rest of the pool: h(u) = J.. jel-12 Evec ((u) d(u, v), where d(::.) denotes a distance (or negative similarity) measure and is implemented as the inner product to align with the previous analysis (Eq. (2)). We then perform it- erative selection construct P... Specifically, we first identify the anchor z; = arg maxuec, Iu(,,, and P.h {z.). At each subsequent step, we choose u arg max minvepy d(u, v),
and update hh <- Phh U {u'). This process repeats until IPA| = Xpa, where Xpa (< M) is hyperparameter controlling the number of selected hard positives.
Remark 3.1. This farthest-poin iterative selection maximizes the spread of Ph on the unit hyper- sphere, thereby enlarging conv(P!!). In light of Theorem 2.1 and Corollary 2.2, this geometric

---

Published as a conference paper at ICLR 2026
expansion mitigates collapse by preventing the anchor from coinciding with single positive and works hand hand with the choice of the negatives introduced next.
Generation and selection of hard negatives (2,). For the negatives of 2, standard strategies include (i) the in-batch scheme, where {ž, jæz are treated as negatives, or (ii) maintaining momentum-basee negative queue. Beyond these. we introduce an adversarial generator to further enrich the negative space. Specifically, the generator G is encouraged to produce adversarial neg- atives 2,- that closely resemble the hard positive set PAt while the discriminator D is trained to distinguish between genuine positives and generated negatives. Formally, the adversarial min-max objective is given by
min max Extepyl log D(z + Eg -G(x)) log (1 D) G
The resulting candidate set of negatives is Q, {ž, | 1 i) {2, }. From 2, we identify the
hard negative subset Q Q, characterized by high similarity to the anchor z, Formally, for every
2, -.= E 2. Vz, E Qi O we require
g())")() sim(((zi),g(z;
Σ sim(f(z)),g(() ".)) = X Σ
(im(f(z).9(27
(8)
2, EQ
where Ao E (0, 1] is a screening coefficient controlling the relative proportion of hard negatives. It is worth emphasizing that while an absolute quantity is utilized for selecting hard positives, relative threshold is adopted for hard negatives. This design choice is motivated by the fact that negatives are typically far more abundant and diverse than positives, and forcing fixed number may either introduce many trivial negatives or overlook informative ones. Employing a relative threshold for negatives therefore ensures adaptability across diverse datasets while maintaining balance. The efficacy of this design validated by ablation experiments, and the corresponding computational complexity is provided in Appendix A.4.
The relative screening strategy for hard negatives Qrr effectively controls the contribu- tion of negatives that lie closest to the positive convex hull conv (P.). From a geometric perspective, including many easy negatives (those far from Phy increase the convex hull of the negalives and hence increase the chance of collapsing as shown in Theorem 2.1. In contrast, including only hard negatives ensures that the conv Q lies close to souv(Ph)) thereby much likely to break the sym- metric condition in (6), and hence aligns with Corollary 2.2, the anchor cannot trivially project into the negative convex hull, which strengthens robustness against collapse and encourages more discriminative embeddings.
Remark 3.2.
Overall loss. Combining the above, the contrastive loss of SPACL defined as:
E! exp ( ((f(z)). h. ())/7)
LSPACI
Σ Elog&
Σ ph exp (If(z)) ( /)/ + +Σ exp (1f(z))
(9)
EXPERIMENTS
In this section, the proposed algorithm evaluated on three downstream tasks across multi- modal scenarios, including image classification (i.e., categorizing images into predefined classes), knowledge-gan grapee link prediction (i.e., inferring missing relations between entities) and out-of- domain intent detection(i.e,, determining whether input matches known intents or is unseen). We utilize the inner product function to estimate the similarity. The screening terms for hard pos- itive and negative are set Apr. 2 and Ao 0.95, respectively (the impact from these two hyperparameters are discussed in the ablation study). All experiments are trained with a machine equipped with an NVIDIA A100 GPU server.

---

Published as a conference paper at ICLR 2026
4.1 IMAGE CLASSIFICATION
Datasets. We evaluate our method on four widely used benchmarks: CIFAR-10, CIFAR-100, ImageNet-100, and ImageNet-IK. IK. All methods, including baselines and SPACL, are implemented with the ResNet(-50) backbone to minimize the impact from encoder variations and focus primarily on the optimization loss". In addition, we standardize the following settings: images are resized to 32x32 for CIFAR-10/100 and x224 for ImageNet-style datasets, and all models are trained for 200 epochs with batch size of 256. Other hyperparameters follow the default configurations from original implementations of each baseline to ensure a fair comparison. Detailed dataset descrip- tions, baselines and their settings are provided in Appendix 1. Their results are either directly resourced from original papers or reproduced using the officially released code.
Table I:
Comparison of baselines and SPACL across supervised, self supervisd, and weakly-
supervised learning paradigms on CIFAR-10/100 and ImageNet- 100/ datasets.
Category
Method
CIFARIO
CIFARIOD
ImageNet 100
ImageNet IK
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
Cross Emtropy
95.07
99.82
74.01
91.89
83.17
95.78
78.20
93.71
Supervised
SupCon
95.51
99.85
76.57
93,50
85.06
96.84
78.72
94.31
VarCon
95,94
99.87
78.29
93.59
86.34
96.96
79.36
94,37
SPACT (Ours)
97.04
99.92.sa1
80.84:
94.32
87.62:1u
ams
80.98:a11
95.45
SimCLR
91.52
99.78
70.67
92.01
71.54
91.56
20.31
90.37
MoCo V2
92.93
99.79
70.01
6.
78. 98
95.20
71.06
90.40
92.57
99.71
70.50
80.18
74.28
91.56
Self-supervised
SwAV
89.14
99.69
64.87
88.81
74.07
92.77
75.29
91.83
VicReg
92.09
99.73
68.51
90.91
95.06
73.25
91.06
Burlow Twins
92,70
99,80
71.02
91,95
80.83
95,24
73,26
91.10
SPACL (Ours)
94.19
99.87:143
72.28
94.39
81.72
95.55
76.59
93,41=ma:
Gralit
90.56
95.95
60.57
82.32
43.54
64.45
18.13
37.19
Colns
83.14
65.13
37.09
Wealty-supervised
MaskCon
91.63
98.72
65.52
83.
47.74
66.98
19.08
38.17
(PAr).
93.92.am
99.51
69,22
_6.00
50.18
70.51
22.08
17.
Results. Table I reports the classification accuracy on CIFAR-10, 10, CIFAR- 100, ImageNet-100, 100, and ImageNet- IK under different supervision paradigms. Overall, our method achieves the best perfor- mance across all datasets and supervision settings. On average, SPACL improves Top-1 accuracy by 1.5 to 4.5% compared to the strongest bascline in each category, while also delivering consistent gains Top-5 accuracy. For instance, under full supervision settings, SPACL reaches 97 97.04% Top- and 99.92% Top-5 on CIFAR-10, 10, outperforming VarCon by +1.10% Top-1. On 100AR-100, it attains 80.84% Top-1 and 94.32% Top-5, surpassing SupCon by +4.27% Top-1. On ImageNet-100) the method achieves 87.6 87.62% Top-1 and 97.84% Top-5, improving over VarCon by +1.28% Top- Finally, on full ImageNet-IK, SPACL delivers 80.98% Top-1 and 95.45% Top-5, which is +1.62% higher in Top-1 compared to VarCon, confirming its competitiveness even with large-scale bench- marks. These results confirm that SPACL provides an effective solution for visual representation learning under diverse learning paradigms.
4.2 LINK PREDICTION
The proposed SPACL is evaluated on a multimodal Knowledge Graph (KG) comple- tion task, ie., KG link prediction, where each entity is associated with both visual and textual modalities. We conduct experiments on three widely used benchmarks: WN9, FB15K-237 237, and FBISK. We follow standard evaluation metries and report Mean Reciprocal Rank (MRR) to- gether with @its@ 1/3/10, which measure how highly the correct tail (or head) entity is ranked among all candidates. Followed prior work (Li et al., 2024b; Liu et al., 2024b), hyperparame- ters for SPACL are tuned by grid search. Specifically, the embedding dimension is searched over {100, 500, 1000, 1500, 2000}, the numbers of nearest neighbors is varied in 2, 6). and the learning rate is drawn from {10-4,5 5 x 10-4, 10-3 10 10-1,1 2 x 10-1). Detailed dataset de- scriptions, baselines and their implementation settings are deferred to Appendix A.3.2. Baseline results are directly resourced from original papers.
Datasets.
As shown in Table 2, SPACI achieves the best prediction performance in terms of MRR and Hit@K. K. In particular, FB15K-23 -237 (where inverse relations are removed to eliminate trivial shortcuts, resulting in sparser and more challenging dataset), the ability to identify informative hard samples becomes especially important. By explicitly targeting such hard positives and negatives, SPACL delivers consistent improvements, attaining an MRR of 41 41.30% and surpassing the strongest baseline LMKGEE22 (40.00%) by +1.30% while also raising Hir@ 10 61 200 compared to 57. 7.80%
Results.
? An additional ablation study on different backbones presented in Appendix A.S.

---

Published as a conference paper at ICLR 2026
Table 2: Link prediction results on WN9, FB15K-237, and FB15K. We report MRR and Hit@K (K=1/3/10) A dash indicates the value is not available.
Models
PRI5K-237
FBISK
MRR
Hicde
Hir
Hire 10
MRR
Hite0
Hire0
Hiree 10
MRR
Hitt9
Hir
Hicee 1D
RESCAL
.706
.447
.356
.263
.393
.541
.644
.544
.708
.824
DistMult
.863
TransE
.665
.765
.816
.871
.313
.221
.347
.497
-676
.542
.787
.875
RotazE
.910
.901
9955
-926
.335
.240
.768
.522
.783
.727
.820
.877
QuarF
366
.401
.556
.333
.859
.900
Complex
IKRL
901
.044
9112
.288
.194
.284
.458
.594
.484
.768
OTKGR
923
911
930
947
MKGioommer
.266
.367
.504
RSME
.878
.912
9923
.344
.467
881
9924
IMF
.389
.287
.593
.377
.785
.944
TransAE
.905
898
908
1922
.199
.317
.463
.645
MoSE
LMKGE
-934
-917
.914
9669
.100
.306
.137
.585
.68
.810
.85
.918
LMKGEE
.931
9914
1940
9967
.944
.302
.431
.578
366
.442
885
.918
SPACE
.42
.927.am a55
.97
.413aamu
.M4saa
.457xmms
.612
_882
883
.81
936.am
from LMKGE The result demonstrates that SPACL provides consistent gains across datasets of varying difficulty levels, validating applicability to knowledge graph completion.
4.3
OUT-OF-DOME
INTENT DETECTION
Dataset. We also evaluate SPACL in a purely text-domain setting, where the goal is detect out- of-domain (OOD) samples. Two widely used benchmarks, BANKING and StackOverflow, are em- ployed. Following prior work (Zhou al., 2023: Li et al., 2025), we randomly preserve 25%, 50% and 75% of the total classes as in-domain (IND) classes, while the remaining classes serve OOD. For SPACL, the Adam optimizer with a learning rate of 1e-5 employed, and we conduct model training with batch size of 8 over 20 epochs. The classification performance is evaluated using two metries: Fl-scores for OOD (F1-OOD) and IND (FI-IND) classes. Detailed dataset descriptions, baselines and their implementation settings deferred Appendix A.3.3. Baseline results directly resourced from original papers or reproduced using the officially released code.
Table 3: OOD detection performances across numerous models on the BANKING and StackOver- flow datasets, according varying proportions (25 %, 50%, 75%) of training classes.
Ratio
Methods BANKING (F1-OOD)
BANKING (FI-IND) StackOverflow (F1-0OD) StackOverflow (FI-IND)
MОGB
88.29
74.50
94.42
87.80
25%
TBOS
92.70
79.70
95.52
85.35
SPACL
93.51:10.05
80.21:10.09
96.72=0006
86.24=0.14
MOGB
89.71
87.27
81.04
81.53
50%
TBOS
84.42
84.63
90.41
88.96
SPACL
85.0810.03
85.57-10.18
91.52::0.19
90.12:0.00 0.08
MOGB
75.52
71.27
75%
TBOS
74.94
89.23
75.58
28.40
SPACL
75.72:10.02
89.95:10.16
76.39=0.18
88.96=0.05
Results. As shown in Table 3, SPACL achieves competitive performance with recent baselines across the datasets, achieving the best in-domain accuracy and OOD detection performance on average. For instance, for StackOverflow, SPACL achieves F1-OOD scores of 96.72 and 76.39 for 25% and 75% training classes, respectively. This presents substantial improvement of 2.3 and 5.1 absolute points, respectively, over MOGB. These results clearly indicate that SPACL retains strong discriminative ability with purely textual data. Together with previous visual and multimodal exper- iments, it is worth nothing that the proposed SPACL algorithm is modality-annostic and generalizes effectively across visual, textual, and multimodal CL scenarios.
4.4 ABLATION STUDY
We further conduct ablation studies to examine how different components of SPACL contribute to its performance. We also investigate the impact of the positive and negative sample budgets towards the final performance. Due to space constraints, these detailed experimental settings and results are provided in Appendix A.6.
Model Breakdown. To begin with, we analyze the individual components SPACL to assess their relative contribution. Our method integrates four key aspects: (1) anchor selection, which maxi- mizes the angular spread of positives by initializing with the hardest instance in the candidate pool;

---

Published as a conference paper at ICLR 2026
(2) hard positive selection, which iteratively enlarges the convex hull of positives to enhance diver- sity and mitigate collapse; (3) adversarial negatives, where the adversarial training is adopted to produce hard negatives; and (4) negative screening. which applies relative threshold to filter out trivial negatives. Accordingly, we construct the following variants for comparison: "w/o Anc" dis- ables anchor selection via taking the original sample as the anchor; "w/o HP" removes farthest-point selection via taking random augmentation as positives; "w/o AN" discards adversarial negatives
while "w/o NS removes the screening mechanism.
72.3
694
43
81
80
79.
80.3
79.5
72
70.3
70.5
69
70
67.2
68.3
67.0
40.6
40.9
80
71
77.9
68.1
67
66
64.8
40
3922
78
39
68
77
67
64
76
63
Full
w/o HP
w/o AN
w/o NS
Full
w/o Anc
HP
w/o AN
wio NS
Full
w/o Anc
HP
w/o AN
w/o NS
Full
w/o Anc wio HP wio AN w/o NS
w/o
(a) CIFAR- 100
(b) CIFAR- 100
(c) CIFAR-100 100
(d) FB15K-237
fully-supervised
superviup
supervisd))))))d)
Figure 1: Examination on individual components from SPACL on CIFAR- under fully-, self-, and weakly-supervised settings, and on FB15K-237.
Fig. reports the results on CIFAR-100 and FB15K-237 across the four ablated variants, high- lighting the contribution of each component. Overall, all variants underperform compared to the full model, confirming that every component of SPACL plays a positive role. The most significant performance drop occurs in "w/o NS", demonstrating the critical role of negative screening. That is, without screening, a large number of trivial negatives are retained, which expands the nega- tive convex body and weakens the discriminative boundary. In addition, "w/o AN" and "w/o Anc" both lead to moderate decreases. The impact of "w/o Anc" reflects the role of anchor selection in maximizing angular spread, thereby reducing the risk of symmetry-indeee induucd collapse described in Theorem 2.1. Similarly, the drop in "w/o AN" shows that adversarially generated negatives help sharpen the boundary of the negative region, which is otherwise blurred when only in-batch or queued negatives are used. Finally, "w/o HP" results in the smallest decline, suggesting that farthest- point selection, while useful for enlarging the convex hull of positives and enhancing diversity, is less critical than maintaining a well-separated boundary between positives and negatives.
Positive Pair Construction. Next, evaluate the impact of anchor and positive sample config- urations. SPACL uses anchor selection and hard positive mining to improve diversity and avoid collapse. Accordingly, we construct two variants for comparison: "OAA" disables dynamic anchor selection by always choosing the original instance as the anchor and randomly sampling positives from its augmented views; and "AFP" draws both anchors and positives randomly from the aug- mented candidate pool.
OAA
OAA
OAA
OAA
Ful
Ful
Full
Full
73
64/5
63.8 8
36.2
75.0
65.
76.9
68
66 7
78,5 9
70,00
40,0
80.8
72.3
69.
41.3
AFP.
AFP.
AFP.
AFP.
(a) CIFAR-100
(b) CIFAR- 100
(c) CIFAR-100
(d) FB15K-237
(Fully-supervised)
supfr-super
Wieakly-supervisee
Figure 2: Impact the roles of dynamic anchor selection and hard positive mining, evaluated CIFAR-100 under fully-, self-, and weakly-supervised paradigms, and on FB15K-237.
As shown Fig. 2, both simplified variants perform worse than the full model, underscoring the importance of carefully designed positive pairs. The OAA variant shows particularly large drop under self- and weakly-supervised settings, reflecting the limitation of using fixed anchor. This observation is consistent with Theorem 2.1, which states that static anchor reduces angular disper-

---

Published as a conference paper at ICLR 2026
sion and compresses the positive convex hull. On the other hand, the AFP variant, while introducing anchor diversity, omits the structured expansion provided by farthest-point selection, resulting in less stable representation geometry. For instance, the experimental results show that CIFAR-100 seef--suervissd) measured by Top-1 accuracy, the full model achieves 72.28%, whereas OAA and AFP drop to 66.53% and 70,,,, respectively. On FB15K- using mean reciprocal rank (MRR) as the evaluation metric, the full model reaches 41 .3%, while OAA drops to 38.2% and AFP reaches 40.7%. These results further confirm the role of both anchor dynamics and positive selection in maintaining discriminative geometry. Overall, these findings highlight the value of jointly optimiz- ing anchors and positives, reinforcing the design motivation of SPACL 's positive construction.
Adversarial Negatives vs In-Bateh Negatives. To assess the effect of negative sample quality, we compare two variants. The first, "IB," applies the contrastive loss directly in-batch negatives, treating all co-occurring samples in the batch or queue as equally informative. The second, "IB+H," ranks the in-batch negatives and selects those with the highest similarity the anchor, but does adversarially generate new hard negatives. To isolate the effect of adversarial enhancement, both variants retain the same anchor selection and positive construction.
IB
IB
IB
72.3
41.3
79.1
70.8
39.8
69.2
66.0
75.6 6
67-6
64. 44
73.9
Ful
66.1
Ful
62.8 8
Ful
35.2
Ful
IB+H
H
IB+H
IB+H
(a) CIFAR-100 (uullysupervised)
(b) CIFAR- 100 sSelf-supervised)
(c) CIFAR-100 (Weakly-superrised)
(d) FB15K-237
Figure 3: Comparison of adversarial negative strategies on CIFAR-100 under fully-, self-, and weakly-supervised settings, and on FB15K-237.
As shown in Fig. 3, the IB+H variant consistently achieves higher performance than the IB base- line across benchmarks. This demonstrates that simply ranking and selecting the most informa- tive in-batch negatives already improves contrastive learning by focusing on harder instances rather than treating all negatives equally. These results are also consistent with the "w/o AN" variant in the Model Breakdown analysis, confirming the role of negative quality in shaping the optimization landscape. The further gains of SPACL over IB+H highlight the additional benefit adversarially generated boundary- negatives, showing that synthesized hard negatives complement original ones by tightening decision boundaries and strengthening discriminability.
Discriminator Generator
-0.
5.0
2.5
Loxk
200
Adversarial In Anchor
-24
In-aatch Anchor
Eoedh
-20
-10
10 Figure 5: Zoomed-in view for ad- versarial and in-bateh negatives the same anchor.
25 50 75 1:0 125 is 175 200
Figure Embedding visualiza- tion of the anchor and in-batch negatives.
Figure 6: Loss curve evolution of the generator and discriminator on CIFAR-100.
To further illustrate the effect of our adversarial negatives, we sample CIFAR-100 with batch size of 256, randomly select one original sample as the anchor, generate three negatives using our adversarial module, and visualize all embeddings via t-SNE, As shown Fig. 4 and 5, most in na-batcch negatives remain in distant clusters, whereas the adversarial negatives concentrate in the region surrounding the anchor, indicating that adversarial negatives effectively tighten the local decision boundary and potentially enhance the model capability.

---

Published as a conference paper at ICLR 2026
We also observe that our proposed adversarial module, to extent, mitigates the impact of false negatives, False negatives typically refer to samples that are semantically related to the anchor (with high-similarity) but are mistakenly treated negatives. Importantly, ated negatives help construct stronger and more informative hard negatives than those obtained via in-batch sampling, including potential false negatives, demonstrated in Fig. 4 and 5. As these ad- versarial negatives exhibit higher similarity, the proposed selection mechanism naturally prioritizes them, thereby reducing the likelihood that moderately similar samples (i.e., some potential false negatives) are selected as hard negatives. Yet, we note that explicitly identifying or eliminating false negatives is beyond the scope of this work, and we leave this future work.
adversarially gener-
In addition, to verify the stability of adversarial training, we also visualize both the generator and discriminator losses on the CIFAR-100 task, as shown in Fig. The discriminator loss exhibits a smooth, gradually increasing trend, while the generator loss decreases throughout training. We attribute this result to following reason: early in training, the generator might produce weaker adversarial examples, making them easy for the discriminator to classify, thereby resulting in high generator loss and low discriminator loss. As training progresses, the generator becomes stronger and produces harder samples, causing the discriminator loss to increase, while the generator loss continues to decrease as adapts to the discriminator. Importantly, both components exhibit stability and are optimized smoothly.
5 CONCLUSION
Contrastive Learning (CL) enforces proximity between anchors and positives while separating nega- tives, yet the theoretical basis of pair sampling and its impact on performance remain underexplored. This paper introduces the SnaPhArd Contrast Learning (SPACL) algorithm, which strategically screens hard contrastive pairs. Our method highlights the role of hard samples during construct- ing contrastive pairs and computing contrastive loss. Thorough theoretical analysis justifies hard sampling and discusses optimality conditions and potential collapsing. Empirical evaluation demonstrates the effectiveness SPACL. Future research could explore different strategies for mea- suring sample difficulty and exploring alternative perturbation methods. Additionally, extending our analysis to negative-free CL could further illustrate adaptability and broader applicability.

---

Published as a conference paper at ICLR 2026
REFERENCES
Chaitanya Animesh and Manmohan Chandraker. Tuned contrastive learning. In 2025 IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 7225-7234, 2025. doi: 10.119/WACYG004:2223...M
Peng Bao, Rong Yan, and Shirui Pan. Co-augmentation of structure and feature for boosting graph contrastive learning. Information Sciences,
2024. ISSN 0020-0255. doi: https://dii. URL https: /www. se encedi rect. com/ sc ence/
artiicle/e11/5002003544000 0tg/10..0016/..ns.220000
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg:
regulariza- tion for self-supervised learning. International Conference on Learning Representations, 2022.
URI https:: /ttt:pss///pppreviieee net forum? (0==xm6YD62D110
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https::/
ngs neurips ec ttappperfflls/pppprrreee SawpsedddepPPppe.pppp. Paper.pdf.
Zongsheng Cao, Qianqian Xu, Zhiyong Yang. Yuan He, Xiaochun Cao, and Qingming Huang. Otkge: Multi-modal knowledge graph embeddings via optimal transport. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems, volume 35, pp. 39090-39102. Curran Associates, Inc., 2022. URL httpss://proceeddingg .ew.eedrpdnn....eiieeeeee......e neurips ce hatpeerffiies/ppppppe file/ fallee/ffaa2beeccccocaoeeellldaa
Paperr-pp---- Senper-Confeeence.cg pdf.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin, Un- supervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran Associates, Inc., 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, October 2021.
Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. Routerde: Query-based router by dual contrastive learning for assembling large language models. In A. Globerson, L.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 66305-66328. Curran Associates, Inc., 2024. URL https: http///ppoeeeeenneeeeeee/o file FBle://2s-1B8cc00012211777775555 neurips cc cctccc///aaeeeeeeeeaee pteeeapper__e saapper_flle files/ plles//pppr/2224) Conferenceeeeeeeeee pdf.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for Proceed contrastive learning of visual representations. In Proceedings of the 37th International Conference
on Machine Learning, volume 119, pp. 1597-1607, 13-18 2020a.
Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, and Huajun Chen. Hybrid transformer with multi-level fusion for multimodal knowledge graph completion. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, pp. 904-915, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450387323, doi: 10.1 00.1155//47749533331922 URL https::///// /oii org/ htte:/...111//3666 11..111114474999 3531992.
Xinlei Chen and Kaiming He. Exploring simple Siamese representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pp. 15750-15758. June
2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020b.

---

Published as a conference paper at ICLR 2026
Xinlci Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
9640 9649, October 2021.
Zhi-Yi Yi Chin, Chich-Ming Jiang, Ching-Chun Huang, Pun-Yu Chen, and Wei-Chen Chiu. Masking improves contrastive self-supervised learning for convnets, and saliency tells you where. Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp- 2761-2770, January 2024.
Chandra Kiran Evuru, Sreyan Ghosh, Sonal Kumar, Ramaneswaran S, Utkarsh Tyagi, and Dinesh Manocha. CoDa: Constrained generation based data augmentation for low-resource NLP. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Com- putational Linguistics: NAACL 2024, pp. 3754-3769. Mexico City, Mexico, June 2024. As- sociation for Computational Linguistics. doi: 10.18653//1/2224.fmmm www.:224...iiing--aaaa naggs..2. URL attps::/accaaneee http:////aaaatttoy///// wtwp:lloyy...gg///0 org//024 fi findings-naacll 238/.
Chen Feng and Ioannis Patras. Maskcon: Masked contrastive learning for coarse-labelled dataset. In 2023 IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 19913- 19922, 2023.
doi: 10.1 O0.119/9CVP8227722203.000000
Chengjian Feng, Yujie Zhong, Zequn Jie, Weidi Xie, and Lin Ma. Instagen: Enhancing object detec- tion by training on synthetic dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14121-14130, June 2024.
Jun Gao, Wei Wang, Changlong Yu, Huan Zhao, Wilfred Ng, and Ruifeng Xu. Improving event representation via simultaneous weakly supervised contrastive learning and clustering. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Pupers), pp. 3036-3049, May 2022.
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap Your Own Latent: a new approach to self-supervised learning. Advances in 2020.
neural inf. processing systems,
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, June 2022.
Minda Hu, Muzhi Li, Yasheng Wang, and Irwin King. Momentum contrastive pre-training for question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4324-4330, December 2022.
Taeuk Jang and Xiaoqian Wang. Difficulty- bbaiid sampling for debiased contrastive representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion (CVPR), pp. 00099-24048, June 2023.
Jinchao Ji, Bingjie Zhang, Junchao Yu, Xudong Zhang, Dinghang Qiu, and Bangzuo Zhang. Relationship-awe contrastive learning for social recommendations, Inf. Sciences, 629:778-797, 2023. ISSN 0020-0255.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 18661-18673. Curran Associates, Inc., 2020. URL https: htttps:://procceddngg.... neursinnnnnnuuuuneee neurips ec cc/paper____ waceeerffffeseeeeeeeeeee pa1lee/ppper/22/////// pdf.

---

Published as a conference paper at ICLR 2026
Yejin Kim, Scott Rome, Kevin Foley, Mayur Nankani, Rimon Melamed, Javier Morales, Ab- hay K. Yadav, Maria Peifer, Sardar Hamidian, and H. Howie Huang. Improving content rec- ommendation: Knowledge graph-based semantic contrastive learning for diversity and cold-start users. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Compula- tional Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 8743-8755, Torino, Italia, May 2024. ELRA and ICCL. URL https: http///aaaaaataaaeaeee org/2024. Arec-main. 766/
Panagiotis Koromilas, Giorgos Bouritsas, Theodoros Giannakopoulos, Mihalis Nicolaou, and Yan- nis Panagakis. Bridging mini-batch and asymptotic analysis in contrastive learning: From In- foNCE to kernel-based losses. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st ternational Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 25276-25301, PMLR, 21-27 Jul 2024. URL https:/ httpss//pppppeeeeeeeeeeeee mmlr. html.
press/v235 htt///korommaaaaaa
Orest Kupyn and Christian Rupprecht. Dataset enhancement with instance-level augmentations. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision ECCV 2024, pp. 384 402, Cham, 2025. Springer Nature Switzerland. ISBN
978-3-031-73337-6
Nicholas Lee, Thanakul Wattanawong, Schoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. Llm2llm: Boosting Ilms with novel iterative data enhancement, 2024a. htLhttts:://a http://aaawww..oraa//////a////// 15042.
Unggi Lee, Sungjun Yoon, Joon Seo Yun, Kyoungsoo Park, YoungHoon Jung, Damji Stratton, and Hyeoncheol Kim. Difficulty-ocuseed contrastive learning for knowledge tracing with a large language model-based difficulty prediction. In Nicoletta Calzolari, Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evalu- ation (LREC-COLING 2024), pp. 4891 4900, Torino, Italia, May 2024b. ELRA and ICCL. URL main.. 438/.
https:
wwtoorg//////. 2024
Yiming Lei, Zilong Li, Yangyang Li, Junping Zhang, and Hongming Shan. Core: Learning consistent ordinal representations with convex optimization for image ordinal estimation. Par- wttp:/oorrrroooo1/....a..ll
tern Recognition, 156:110740 2024. ISSN 0031-3203. doi: www.//cceenceaartcceee/////e 2024.110748. URL https::///// http:://wwwwww.ieee.e http://wwww..eeiieccce rect. com 50C31320 000313203240049290
Meng Li, Bo Yang, Tao Xue, Yiguo Zhang, and Liangliang Zhou. Contrastive graph clustering via enhanced hard sample mining and cluster-guiding: Contrastive graph clustering... Multimedia
Syst., 30(6), November 2024a. ISSN 0942-4962.
doi: 10.10077/005330002--15511 URL 022-0156777.
https:/ //doi///iii// www/ooggg///oo http://.11100///10
100.100/0000 00:07//005300
Qingyang Li, Yanru Zhong, and Yuchu Qin. MoCoKGC: Momentum contrast entity encoding for knowledge graph completion. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 10940-14952, Miami, Florida, USA, November 2024b. Association for Computational Linguis- tics. doi: 10. (0.18653/v1/2001 www.:4..mmmlmmmi..... emm emmlp-maii
URL https: http:///aaaaaaaaaaeeeeeea org/2024.
ma ii.iii.... 832/.
Yanhua Li, Xiaocao Ouyang, Chaofan Pan, Jie Zhang, Sen Zhao, Shuyin Xia, Xin Yang, Guoyin Wang, and Tianrui Li. Multi-granularity open intent classification via adaptive bral decision boundary. Proceedings of the AAAI Conference Artificial Intelligence, 39(23);
24512-24520, Apr. 2025. doi: 10.1600%/aai.vv392.. v39:23.34630. URL https: //t::///jssss. aaai.org. index. index.php/ai AAAI www.A///attttteteeee/ea/aaaaaaae
Yingji Li, Mengnan Du, Rui Song. Xin Wang, Mingchen Sun, and Ying Wang. Mitigating social biases of pre-trained language models via contrastive self-debiasing with double data augmenta- tion. Artificial Intelligence, 332: 132:104143, 2024c. ISSN 0004-3702. doi: http:://doi.org/1010116

---

Published as a conference paper at ICLR 2026
j.artint.222.1004144 URL /ttp:////////....aaaaaaaa..aooaa 041//000433722400000
Jiashi Lin, Lifang Wang. Xinyu Lu, Zhongtian Hu, Wei Zhang, and Wenxuan Lu. Improving knowl- edge graph completion with structure-awary supervised contrastive learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11948-139553 Miami, Florida, USA, November 1/2024.eemll-mmaaii 772. URL
2024. Association for Computational Linguistics. doi: 10. https: https///aaaaaallogyo...o. org/ 2024 emnlp maimmmiin main.. 772/.
Han Liu, Jiaqing Zhan, and Qin Zhang. Uncertainty-awarr contrastive learning with hard negative sampling for code search tasks. Proceedings of the AAAI Conference on Artificial Intelligence, 39 (18):18807-18815, Apr. 2025a. doi: 10.1609/aaai htt00///aai..99i1... v39i18.34077 URL https: /tl:://ojj /ojs. aaai. org/index. php/ AAAI / article/viee wtteee//w000000 34070.
Junping Liu, Shijie Mei, Xinrong Hu, Xun Yao, Jack Yang, and Yi Guo. Seeing the wood for the trees: contrastive regularization method for the low-resource knowledge base question answer- ing. In Findings of the Association for Computational Linguistics, pp. 1085-1094, July 2022.
Min Liu, Hui Sheng, Ningyi Zhang, Panpan Zhao, Yugen Yi, Yirui Jiang, and Jiangyan Dai. Dsd- clnet: Dual-stream encoder and dual-level contrastive learning network for supervised multivari- ate time series classification. Knowledge- Based Systems, 292: 1 1638, 2024a. ISSN 0950-7051. doi: http/////n/e.ttteea///////////// com/
URL https: : /t/////ww// /www. sci scciencedireee rect.
Ruizhou Liu, Zongsheng Cao, Zhe Wu, Qianqian Xu, and Qingming Huang. Multimodal knowledge graph embeddings via lorentz-based contrastive learning. In 2024 IFEE International Conference on Multimedia and Expo (ICME), pp. 1-6, 2024b. doi: 10. N0.1100/CEE555554000000 000. 10687608.
Wenxiao Liu, Zihong Yang. Chaozhuo Li, Zijin Hong, Jianfeng Ma. Zhiquan Liu, Litian Zhang, and Feiran Huang. Contrastive learning with hard negatives for sentence embeddings. Ap- plied Soft Computing, 184:1 13685, 2025b. ISSN 1568-4946. doi: https://doi.org/10 101100.......ccc 2025. 1365.. URL https:/ stteps::/www.r /www www.s.seeneedeeet.... com/ cww.//cieecce/articleee/ppp $1 01468446625009900
Yue Liu, Xihong Yang. Sihang Zhou, Xinwang Liu, Siwei Wang, Ke Liang, Wenxuan Tu, and Liang Li. Simple contrastive graph clustering. JEEE Transactions on Neural Networks and Learning Systems, 2023.
Ziyang Liu, Chaokun Wang, and Cheng Wu. Graph contrastive learning with reinforcement aug- mentation. In Proceedings of the Thirly-Third International Joint Conference on Artificial In- telligence, IJCAI '24, 2024c. ISBN 978-1-956792-04-1 doi: 10.24963/jjea..02241 20024/24.... URL
/atpss//doos org/10. 24963/ 20243//2222222466666
David Mildenberger, Paul Hager, Daniel Rueckert, and Martin J. Menten.
A tale of two classes: Adapting supervised contrastive learning binary imbalanced datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10305-10314 June 2025.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML'11, pp. 809-816, Madison, WI, USA, 2011. Omnipress. ISBN 978 9781450306195.
Letian Peng, Yuwei Zhang, and Jingbo Shang. Controllable data augmentation for few-shot min- ing with chain-of-thour thought attribute manipulation, 2024. URL https://arxiv. / 2307.00 07099.
Tao Ren, Haodong Zhang, Yifan Wang, Wei Ju, Chengwu Liu, Fanchun Meng. Siyu Yi, and Xiao Luo. Mhge: Multi-scale hard sample mining for contrastive deep graph clustering. Information Processing & Management, 6244)4104084, 2025. ISSN 0306-4573. doi: https://doi.or/110.000 1.ipm.205..14444 104084. URL https: : www. aci encedi rect. com/ acience/article. p11/803064572200225

---

Published as a conference paper at ICLR 2026
Yuntao Shou, Xiangyong Cao, and Deyu Meng. Spegel: Self-supervised graph spectrum contrastive learning without positive samples. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-14, 2025. doi: 10. 202.
Liu Sun, Ming He, Shuai Qin, and Nianbin Wang. Robust augmentation- contrastive clustering with negative data mining. Engineering Applications of Artificial Intelligence, 148:110414, 2025. ISSN 0952-1976. doi: https://oo..ogg///looooli/ggggpp 110414. URL https: www.. sciencedirect
Zhiqing Sun, Zhi-Hon Zhi-Hong Deng, Jian- Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space, 2019. URL httht:/:: /ttas://aaxxtie arxiv. org abs/ /tabs//99... 10197. 97.
Hugo Touvron, Alexandre Sablayrolles, Matthijs Douze, Matthieu Cord, and Hervé Jégou. Grafit: Learning fine-grained image representations with coarse labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 874-884, October 2021.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Eric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2071-2080, New York, New York, USA, 20- 22 Jun 2016. PMLR. URL https: ppwpceeeiingg ngs html.
www.:iinnnmmmmml...mmmmm......ee mlr press/
Sheng Wan, Yibing Zhan, Shuo Chen, Shirui Pan, Jian Yang, Dacheng Tao, and Chen Gong. Boost- ing graph contrastive learning via adaptive sampling. IEEE Transactions on Neural Networks and Learning Systems, 35(1 35(11))15977) 11:1771-119988 2024. doi: 10. 2.223.
Lu Wang, Chao Du, Pu Zhao, Chuan Luo. Zhangchi Zhu, Bo Qiao, Wei Zhang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Contrastive learning with negative sampling correction, 2024a. URL https: // rxivv. orgoorrggoooo.. org/abs/////eeeeoooee/.eeeeeee /2001 08690.
Meng Wang. Sen Wang, Han Yang, Zheng Zhang, Xi Chen, and Guilin Qi. Is visual context really helpful for knowledge graph? representation learning perspective. In Proceedings of the 29th ACM International Conference on Multimedia, MM '21, pp. 2735-2743, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450386517. doi: 10.1145/3474085. 3475470. URL https: tot.doi.oooooooooooooooo wrrii.org//oe 11.....11111144.00 3475477.
Yanjie Wang, Xu Zou, Luxin Yan, Sheng Zhong, and Jiahuan Zhou. Snida: Unlocking few- shot object detection with non-linear semantic decoupling augmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 12544-12553, June 2024b.
Zikang Wang, Linjing Li, Qiudan Li, and Daniel Zeng. Multimodal data enhanced representation learning for knowledge graphs. In 2019 International Joint Conference on Neural Networks (JJCNN, pp. 1-8, 2019. doi: 10. 20000/CCNN2001.8852000
Ziwen Wang, Jiajun Fan, Thao Nguyen, Heng Ji, and Ge Liu. Variational supervised contrastive learning, 2025. URL https: // httpaaxivvvvro... abs /20.. 07413.
Jiantao WU, Sara Atito, Zhenhua Feng, Shentong Mo, Josef Kitler, and Muhammad Awais. Rethink- ing positive pairs in contrastive learning, 2025a. URL 18200.
wttp org absgg//b////44
Nannan WU, Hongdou Dong, Wenjun Wang, and Yiming Zhao. Dual encoder contrastive learning with augmented views for graph anomaly detection. In James Kwok (ed.), Proceedings of the Fourt International Joint Conference on Artificial Intelligence, IJCAI-25, pp. 3480-3488. International Joint Conferences Artificial Intelligence Organization, 8 2025b. doi: 10.24963/ 2025i.2025/387. URL https: : /ooi. org/10 00..0.00000 249634 hate:/jcaaa... 022000//87777 Main Track.
Ting Wu, Jingyi Liu, Rui Zheng, Qi Zhang. Tao Gui, and Xuanjing Huang. Enhancing contrastive learning with noise-guided attack: Towards continual relation extraction in the wild. CoRR, abs/2305.07085, 2023. URL https: do1. org/10. 48550/ arxiv. 2305. 07085.

---

Published as a conference paper at ICLR 2026
Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. Image-embodied knowledge repre- sentation learning, 2017. URL https: /ttpss//aaxiiv../eee 1ttpx//..rg//b///119..0.0 070020
Albert Xu, Jhih-Yi Hsieh, Bhaskar Vundurthy, Eliana Cohen, Howie Choset, and Lu Li. Mathe- matical justification of hard negative mining via isometric approximation theorem, 2022. URL
https: /arxivv./aiivv///aa/e/e org/ /bag//bb//// 2220.22...2 11173.
Yuanhong Xu, Qi Qian, Hao Li, Rong Jin, and Juhua Hu. Weakly supervised representation learning with coarse labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10593-10601, October 2021.
Bishan Yang. Wen tau Yih, Xiaodong He, Jianfeng Gao, and Deng. Embedding entities and relations for learning and inference in knowledge bases, 2015. URL pettps://arxiv.
abs/1412, 6575.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow twins: Self- supervised learning via redundancy reduction. Marina Meila and Tong Zhang (eds.). Pro- ceedings the 38th International Conference Machine Learning. volume 139 of Proceed- ings of Machine Learning Research, pp. 12310-12320. PMLR, 18-24 Jul 2021. URL https: / //proceedings. mlr wtt.ss..l............mmm
wwwps/eesssaeeeee /v139/ zbontar21a html.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X. Pham, Chang D. Yoo, and In So Kweon. How does SimSiam avoid collapse without negative samples? A unified understanding with self-supervised contrastive learning. In International Conference on Learming Representa- tions, 2022a.
Kevin Zhang and Zhiqiang Shen. i-mae: Are latent representations in masked autoencoders lin- early separable? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 7740-7749, June 2024.
Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13774-13784, June 2024.
Lilin Zhang, Chengpei Wu, and Ning Yang. Weakly supervised contrastive adversarial training for learning robust features from comi-supervised data, 2025. URL https://arxiv..
Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical un- derstandings of masked autoencoders. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Process- ing Systems, volume 35, pp. 27127-27139. Curran Associates, Inc., 2022b. URL htt.://eeeeeee...eeere.....peppp
Shaofeng Zhang, Meng Liu, Junchi Yan, Hengrui Zhang, Lingxiao Huang, Xiaokang Yang, and Pinyan Lu. M-Mix: Generating hard negatives via multi-sample mixing for contrastive learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2461-2470 2022c.
SHUAI ZHANG, Yi Tay, Lina Yao, and Qi Liu. Quaternion knowledge graph embeddings. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox, and R. Gameet (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: / http:sss///pooceeeessss.eesee neurips...e....p.. F1l1:/08222222377772221002222222 Pww.:699-apep.....
Yanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu, and Yongyi Mao. Unsupervised sen- tence representation via contrastive learning with mixing negatives. Proceedings of the AAAI Con- ference on Artificial Intelligence, 17310-111300117330 Jun. 2022d. doi: 10. 10..///a...viii 21428. URL https:: /ttes////sssoeee aaai. www.aii..or//iaaoa www.h/dww..ppppppppmmpeepppp AAPP/AAAA/// wwrrticeevvieeee hwtpw/wwwww/2// 221428.

---

Published as a conference paper at ICLR 2026
Yu Zhao, Xiangrui Cai, Yike Wu, Haiwei Zhang, Ying Zhang, Guoqing Zhao, and Ning Jiang. Mose: Modality split and ensemble for multimodal knowledge graph completion, 2022. URL petps://arxier org/ otg/abs/////ooooooos/oo/sss..a.s 2tt../221...00 08821.
Xu Zheng, Tianchun Wang,
Wei Cheng, Aitian Ma, Haifeng Chen, Mo Sha, and Dongsheng Luo. Parametric augmentation for time series contrastive learning. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: pttt::://ppnrevieett net / forum?id-
EIP L
Yunhua Zhou, Jianqiang Yang. Pengyu Wang, and Xipeng Qiu. Two birds one stone: Dynamic ensemble for OOD intent classification. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 19659-10673, Toronto, Canada, July 2023, Asso- ciation for Computational Linguisties. doi: 10.18653/v 118653311/22233 /2023.acllloong.oooo //aclanthology. wwtp:/tt../oooo.ooooo acl-loncclllnnggg 595/.
URL https:
Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 09306-10315, 2021.
Haojie Zhuang, Wei Emma Zhang, Chang Dong, Jian Yang, and Quan Sheng. Trainable hard neg- ative examples in contrastive learning for unsupervised abstractive summarization. In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 1589-1600, St. Julian's ss Malta, March 2024a. Association for Computational Linguistics. URL https: http////aaaaattoooa.oooo.o.oooaa org/ 2024 Findings-eacl. 110/.
Jin Zhuang, Xiao-Yuan Jing, and Xiaodong Jia. Mining negative samples on contrastive learning via curricular weighting strategy. Information Sciences, 668:120534, 2024b. ISSN 0020-0255. doi: https://doi.org wrt.://.11/11111110/1/1 1016/ ins. 120534. URL https: www. sciencedirect.
com/scien
Jiaming Zhuo, Yintong Lu, Hui Ning. Kun Fu, Bingxin Niu, Dongxiao He, Chuan Wang, Yuanfang Guo, Zhen Wang, Xiaochun Cao, and Liang Yang. Unified graph augmen- tations for generalized contrastive learning
graphs. In A. Globerson, L. Mackey, D. Belgrave. A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neu- ral Information Processing Systems, volume 37, pp. 37473-37503. Curran Associates, Inc., 2024. URI. https::/ http///ooeeeeeeeeeoooooooooeeeee ngs neurips cc/ hacc/paper_faae fi filess/pperr/2224 file/ file/41efc1222 /t//44/eff222292e2 (wt.29222cccffooeeeeecaaa888881.
Yuanhao Zou and Zhaozheng Yin. Alignment, mining and fusion: Representation alignment with hard negative mining and selective knowledge fusion for medical visual question answering. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29623-29633, June 2025.

---

Published as a conference paper at ICLR 2026
A
SUPPLEMENTARY MATERIALS
A.1 DETAILED ANALYSIS
A.1.1 PROOF FOR THEOREM 2.1
Proof. The Lagrangian Le from Eq. (4)
E, exp(x, nz) exp(xx. Pj)
L
X||x||2 -),
where À E R is the Lagrangian multiplier, The optimality conditions can be defined as Eq. (10).
E explx.p)11
oxp(xxn,)nn))- (,, (xp(x.n,))) exp(x.p:)))) 2Ax 0 (2, exp(x.p,))
(10)
|x|3=11
Note that here is meant to be the optimal solution, conventionally written x". The second equa- tion signifies the spherical distribution condition, i.e., x E Smm1, where m denotes the dimension of the representation space, i.e., R. and Sm is the sphere embedded in Rn with the dimension of m - 1. We can further derive
exp(x, nz)
exp(x,Ps)
st = 21st
nj
s x,
(11)
due to the non-negativity property of the sum of exponentials, ie., 8 and s are strictly positive. Note that Σ, exp(x, /8 1, and Σ, Pp)s* 1. It indicates the L.H.S of Eq. (11) reconstructions. To simplify,
represents two struction for a set of arbitrary vectors v;'s (N is the set size):
introduce the definition of convex recon-
conv(v; (v {a|Bw, E R, l,..., N, a = Σ W,V,, Σ w =1, and 2 0 Vi}. (12)
Then, have:
{
n(x) E conv (n -N= p(x) Σ, Djv p(x) E conv(p,)) UN+
n(x) = E,
exp(x.m))
where n(x) and pxx) are functions dependent on x. The optimality condition from Eq. (11) can then be simplified to:
n(x) p(x) = cx,
(13)
where e (2As a/s= ensures equality by choosing an appropriate value for A.
When there is only single positive, p(x) = p. From Eq. (13), is easy to see that if p, then n(x) = (1 + c)p. Hence, by setting C = I + 2A/s-, and s 1, we establish the necessary condition. The argument holds in both directions, confirming the "if and only if" scenario.
Fig. 7 further illustrates the optimality condition for the case of multiple positives. Note that both p(x) and n(x) must exist within the convex hulls of positives and negatives, respectively, despite the position of the optimal solution in sm-1

---

Published as a conference paper at ICLR 2026
n(x)
Figure 7: Illustration of optimality condition. The polygons tightly around positives (annotated as circles) and negatives (annotated diamonds) are comv (p: J and conv (ou nn: ) respectively.
A.1.2 PROOF FOR LEMMA 2.4
Proof. We follow Eq. (3) and expand the part before Q recognizing it is precisely the linear operator VxLe- Then we have the following computation by omitting the index i:
VxLe s 5/8t +1 dx
exp(x,
exp pj} st
nj
Dj
s+ +
conv N cometp,1
st +1
v = conu 1N conv con 1N+ )
(14)
Subsequently, Vx is bounded by:
2
(15)
N-c + where apply the facts that ||v|| < 2, st < N constrained to be on 1mn11 ensuring maxss = NTe. The claimed limit follows.
for fixed N and all vectors
e.
Fig. 8 illustrates the length bound function of N values. Notably, when N+ > 1, the initial value of VexLell experiences slight reduction, followed by more substantial increase as N increases, and the convergence for the gradient length is rapid. particular, when N 1000, >/&Cell z 2 applied for all cases (as casily derived from Eq. (15)). This offers another rationale for hard sample selection: ensure the most effective minimum s+ s in Eq. (14), to achieve the maximum gradient norm without incurring unnecessary computational overhead.
1.1.3 OPTIMALITY CONDITION FOR FREE (UNCONSTRAINED) VERSION
Now we turn to the alternative objective without constraints, í.e., the free version:
Σ, exp
[
x n
Σ, exp (x. n,)
C₁ E, exp
(16)
x ||x|
E, exp x.).)
Pj

---

Published as a conference paper at ICLR 2026
2.0
1.9
188
Vx
17
1.6
1.5
N
1.3 1.4
N
V
N
Figure 8: Maximum ||Vx[|| as a function of N for various N+ values.
x
Are of unit sphere
Ax V
x(
|||
Figure 9: Illustration of the orthogonal projection Ax-
where x
x/||i|| and ||||| 1. The difference to the constrained objective lies:
ac,
Eg exp
[
n2
X
xx [
ny
0x
(s+)2
8 E3
exp
| Dj / [
&8
[
)p;
(s+)2
Σ, exp (x, n2,) (I - xx )m3
| [x+(+ )2
8 Σ, exp (x, p.) (I (I-Xx]] )P,
with the application of the following identity
i
x
xx
Ox
X
| 3
where I is the identity matrix with compatible dimensions.
Write Ax = I xx /1
T and w (x) exp (x/ P,/nn) /st/-
exp (x, p,/nn) /st//- The optimality condition gives Σ, w (x) = Σ, w (x) Apparently Ax is a linear map that converts a vector to the one perpendicular to x. That is, Agv = v vx v (x/||*|)* v x/||||| as illustrated in Fig. 9. Therefore, Ax v resides in the hyperplane centered at the origin o, denoted as Px. This hyperplane has normal vector and an apparent dimensionality of m 1. Consolidating these observations, we have the following straightforward optimality condition:
n(x) p(x).
(17)

---

Published as a conference paper at ICLR 2026
where n(x) Σ (x)nz. n; Axnz, and likewise for p(x). We can see that n(X) e conv n. ) and p(x) E [P:) analogous to Eq. (13), The difference is the convex hulls are now Px orthogonal to x, instead of ambient space IR..
The optimality condition in Eq. (17) appears distinct from Eq. (13), However, they are closely related. We can express Eq. (17) to separate x or x, i.e.,
exp(x, nj
exp(x, n2) (x, ny)
n(x)= = Σ
exp(x, nj) (x, nz)
-n(x)
5
exp(x, ny)( nz)
= =n(x) - 5 x (write $
Similarly one can have p(x) = p - - The optimality condition derived from Eq. (17) can be reformulated as n(x) - p (s+ - cx, which is nearly identical Eq. (13), except for the term c. Nonetheless, the constrained version has straightforward geometric interpretation that can be easily visualized, as demonstrated in Fig. 7, while the free version involves an orthogonal projection onto Px.
Px
X2
Px
Figure 10: Illustration of optimality condition from the free version for and Xy. Hollow circles are positives and hollow diamonds are negatives. Corresponding solid ones are projections to Px, and Px,. The polygons are convex hulls of projected positives and negatives. Dotted lines show the projection direction. The optimal solution has to lie within the intersection the convex bodies.
Fig. illustrates the optimality conditions for X and X2 (two candidates of x). Apparently X2 cannot be the solution because the convex bodies of negatives and positives in Px, has no intersec- tion. Although X; is a potential candidate for the solution since convex hulls intersect in Px,, the condition in Eq. (17) must also be satisfied.
A.2 RELATED WORK
Contrastive Learning (CL) has garnered significant attention for leveraging input data addi- tional supervisory signal during model training (Chen et al., 2020a; He et al., 2020; Grill et al., 2020: Chen & He, 2021). Specifically, given anchor sample and encoder function f(), the

---

Published as a conference paper at ICLR 2026
general form of the contrastive loss is defined as follows:
sim (f(z), f(z²) >> sim ((()),((z))
(18)
where sim(.,-) represents a user-defined similarity function, and z* and z represent contrastive pairs of positive and negative instances w.r.f z, respectively. The effectiveness of CL hinges on the ability to ensure positive samples remain closely to anchors in the latent space, while negative ones are pushed apart. Recent years have witnessed remarkable progress in CL. involving wide range of techniques for the generation and sampling of positive/negative instances.
A.2.1 GENERATION OF CONTRASTIVE PAIRS
Contrastive pairs, also referred to as positive and negative samples, are of paramount importance in CL and serve as the foundation for the model training. Extensive research into their generation, including minibatch adaptation, data augmentation, and knowledge-basee methods, has significantly improved the data quality and diversity.
Minibatch adaptation primarily caches current or past in-batch data as positive/negative samples. The MoCo (Momentum Contrast) series, exemplified by MoCo-VI (He et al., 2020) and MoCo- V2 (Chen et al., 2020b), employs momentum-based queue to accumulate negatives from past minibatches. In contrast, V3 (Chen et al., 2021) I) and SimCLR (Chen et al., 2020a) abandon this separate queue, instead using augmented versions of different samples (not anchors) from the current minibatch as negatives.
Data augmentation involves instance- and feature-level transformations applied to original training samples. For instance-level augmentation in Computer Vision (Kupyn & Rupprecht, 2025; Wang et al., 2024b; Feng et al., 2024), common techniques include In Natural Language Processing (Peng et al., 2024; Lee et al., 2024a; Evuru et al., 2024), augmentation techniques might involve translation, rephrasing, synonym replacement, and etc. Some studies consider adversarial samples (Zhang et al., 2025; Liu et al., 2023), via dynamically generating negative samples as min-max problem. For feature-level augmentation, HNCSE (Liu et al., 2025b) performs feature-level augmentation by mix- ing hard negatives in the embedding space and applying positive mixing guided by the hardest neg- ative, further enriching contrastive pair diversity and difficulty for sentence embeddings. Moreover, STRFE (Bao al., 2024) co-augments structure and node features in the spectral domain-bbosting low-frequency components for structural consistency while enhancing high-frequent components for feature-Iever diversity- to yield harder negatives and stronger representations. Complementarily, RA-CC (Sun et al., 2025) combines linear and model-based nonlinear fcature-level augmentations and mines visual confusion categories to reweight negatives, thereby producing harder views and sharpening cluster discrimination. MoCo-FT (Zhu et al., 2021) proposes feature extrapolation and interpolation (a linear combination) of positives and negatives to create diverse contrastive pairs. Similarly, the MixCSE model (Zhang et al., 2022d) mixes positive and negative samples during the training process. M-Mix (Zhang et al., 2022c) employs Graph Neural Networks to adaptively se- lect multiple neighboring nodes, assigning them dynamic weights for generating negatives. Another line of research discards negatives and relies exclusively on positives. Two notable approaches, BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021), employ a stop-gradient operation to construct positives as a Now-pass-iittere version of anchor samples. A specific study by Zhang et al. (Zhang al.. 2022a) later provides a comprehensive analysis of SimSiam. Two key factors, known as de-centering and dimensional de-correlation, are identified to prevent solution collapsing within the context of negative-free CL.
In contrast to the aforementioned minibatch adaptation and data augmentation, knowledge-based methods leverage external knowledge and prior information. For example, et al. (Ji et al., 2023) propose utilizing the social networking relationships among users and the commonality relationships of items to generate positive samples for online recommendation. sion, cloze-like question-answer pairs are generated as positives (Liu et al., 2022; Hu et al., 2022). Furthermore, prior knowledge about co-occurring events is utilized in (Gao et al., 2022) to form contrastive pairs.
machine reading comprehen-
A.2.2 SAMPLING OF CONTRASTIVE PAIRS
In contrast to the generation of contrastive pairs, the sampling technique aims to select high-quality or hard positive and negative samples. In the context of CL. sample hardness can be defined by the

---

Published as a conference paper at ICLR 2026
feature proximity of negative sample to that of positive sample, and vice versa. In other words, the closeness in the representation space between contrastive pairs is a key criterion for determining sample hardness. The more similar contrastive pair is, the harder it is for model to differentiate them, which, in turn, enforces the model to learn/produce more discriminative representations.
To begin with, adaptive sampling is leveraged in graph CL to reweight negatives toward the most informative ones while suppressing false negatives via polarization regularization (Wan et al., 2024). In complementary vein, curricular weighting strategy gradually shifts attention from easy to hard negatives to improve convergence and robustness to false negatives (Zhuang et al., 2024b). From theoretical angle, hard-negative mining further justified through an isometric-approximatio view that clarifies when mining avoids collapse and why it benefits metric learning (Xu et al.. 2022). Likewise, difficulty-bast -ased sampling debiases contrastive learning by emphasizing bias-confticting hard examples using a triplet-driven biased encoder (Jang & Wang, 2023). Extending these ideas to software engineering. uncertuinty-aware embeddings are coupled with hard negative sampling to construct challenging query-code pairs and strengthen inter-/ntra-moddliig alignment (Liu et al., 2025a). Meanwhile, in contrastive graph clustering, enhanced hard sample mining with cluster- guiding explicitly repels hard negatives while attracting hard positives via pseudo-labels (Li et al., 2024a). Finally, a multi-scale hard sample mining mechanism upweights difficult pairs across gran- ularities to reinforce clustering performance (Ren et al., 2025).
A.3 EXPERIMENTAL DETAILS
We provide experimental setups three benchmarks to validate the generality of method. These tasks span vision, textual, and multimodal data, covering low- to high-resolution image recognition, textual understanding, and multimodal relational reasoning. Together, they demonstrate that the proposed SPACL algorithm consistently adapts across diverse modalities, dataset scales, and super- vision regimes.
A.3.1 IMAGE CLASSIFICATION DATASETS AND IMPLEMENTATION DETAILS
Datasets. We evaluate the proposed method on four widely used benchmark datasets: CIFAR- 10, CIFAR-100, ImageNet-100, and ImageNet-1K. CIFAR-10 and CIFAR-100 consist of 60,000 color images of size 32 x 32, with CIFAR-10 containing 10 classes (6,000 images per class) and CIFAR-100 containing 100 classes (600 images per class). ImageNet-
is a medium-scale subset of ImageNet, covering 100 categories with 1,300 training images and 50 validation images per class. ImageNet- IK includes more than 1.2 million training images and 50,000 validation images spanning 1,000 categories. Together, these datasets span wide range of scales from low-resolution small-scale tasks (CIFAR-10/100) to large-scale high-resolution recognition (ImageNet- 100/1K), IK), providing comprehensive basis for evaluating classification performance in contrastive learning.
Implementation. To demonstrate the generality of our framework, we conduct experiments under three supervision settings: fully-supervised, supersuue and weakly-supervised learning. This design enables thorough assessment of the robustness and adaptability of our method across diverse training scenarios.
For fully-supervised settings, we consider Cross-Entropy, the conventional training paradigm us- ing classification loss, SupCon (Khosla al., 2020), which enhances representation learning by enforcing intra- classs similarity and inter-class separation, and VarCon Wang et al., 2025), a recent extension that introduces variational modeling into contrastive learning.
In the self-supervised category, we evaluate several thth--ff methods. SimCLR (Chen et al., 2020a) and MoCo V2 (Chen et al., 2020b) employ instance discrimination with different design choices for negative sampling and momentum encoders. BYOL (Grill et al., 2020) removes ex- plicit negatives and learns by aligning predictions with target networks. SwAV (Caron et al., 2020) combines contrastive objectives with online clustering to assign prototypes. VicReg (Bardes et al., 2022) regularizes variance, invariance, and covariance to improve feature robustness. Barlow Twins (Zbontar et al., 2021) minimizes redundancy between representation dimensions enforce richer feature decorrelation.
For weakly-supervised basclines, we include methods that integrate limited label information with contrastive signals. Grafit (Touvron et al., 2021) couples supervised contrastive learning with instance-level self-supervision to mitigate under-clustering. CoIns (Xu et al., 2021) combines

---

Published as a conference paper at ICLR 2026
cross-entropy with contrastive objectives to balance label guidance and representation diversity. MaskCon (Feng & Patras, 2023) leverages coarse labels through a masked contrastive mechanism, refining inter-sample relations by filtering out noisy associations.
A.3.2 LINK PREDICTION: DATASETS AND IMPLEMENTATION DETAILS
Datasets. Evaluations are conducted on three widely used benchmarks, FB15K, FB15K-237, and WN9. FBISK originates from Freebase and provides medium-scale relational graph that is often augmented with multimodal side information (entity-linked images and short textual descriptions). In our setting, it contains 14,951 entities, 1,345 relation types, and 592,213 triples, split into 483,142 training. 50,000 validation, and 59,071 test triples. FBI5K-231 237 derived from FB 15K by removing inverse and redundant relations, yielding a harder and more reliable benchmark for generalization while largely preserving the entity universe and mitigating relation leakage, it consists contains 14,541 entities, 237 relation types, and 272,115/ 17,535 20,466 466 triples for training, validation, and test, respectively. WNN is small Knowledge Graph extracted from WordNet with 9 relation types and 6,555 entities, yielding 11,741 / 1,337 / 1,319 triples for training, validation, and test.
Implementation. We adopt the standard Knowledge Graph completion setting. That is, for each test triple (h, r, t), our inputs or queries can be (h, r, ?) or (?, r. f). where "?" denotes the unknown entity. We then use the trained model score all candidate entities and rank them accordingly. Additionally, the Mean Reciprocal Rank (MRR) and Hit Ratio at cutoffs 1, 3, and 10 (Hit@) 1/3/10) are employed as our evaluation metrics. MRR measures the average reciprocal rank of the correct entity in the prediction list, thus emphasizing how early the correct answer appears. Hit@k computes the proportion of test queries for which the correct entity is ranked within the top positions, with Hit@1, 1, @it@3, and @it@ 10 reflecting increasingly relaxed success criteria. In all cases, larger values indicate better link prediction performance.
We compare SPACL with a broad set of baselines covering both unimodal and multimodal Knowl- edge Graph embedding approaches. The unimodal group consists of tensor factorization models such as RESCAL (Nickel et al.. 2011), bilinear formulations like DistMult (Yang et al., 2015) and ComplEx (Trouillon et al., 2016), and translational distance models including TransE (Bor- des et al., 2013). RotatE (Sun et al., 2019), and QuatE (ZHANG et al., 2019). These methods capture relational structure solely from symbolic triplets. The multimodal group augments struc- tural embeddings with side information. IKRL. (Xie al., 2017) extends TransF by aligning visual features with entity embeddings, TransAE (Wang et al., 2019) integrates structural and visual sig- nals through auto-encoding, RSME Wang et al. (2021) introduces a gating mechanism filter noisy modalities, MKGformer (Chen et al., 2022) employs transformer-batt al., 2022) utilizes optimal transport align heterogeneous information sources. Recent works such as MoSE (Zhao et al., 2022) and MoCoKGC (Li et al., 2024b) explore multi-source integra- tion and momentum contrast, while LMKGE (Liu et al., 2024b) and LMKGE12 leverage Lorentzian geometry with contrastive learning to enhance semantic alignment.
fusion, and OTKGE (Cao
A.3.3 OUT-OF-DOMAN INTENT DETECTION: DATASETS AND IMPLEMENTATION DETAILS
Datasets. Two homlly-co OOD detection benchmarks are employed, including BANKING and StackOverflow. The BANKING dataset consists of 13,083 utterances. related BANKING services, and is annotated with 77 unique intents. The StackOverflow dataset, sourced from Kag- gle.com, encompasses 20,000 technical questions categorized into 20 distinct intents, with each class containing 1,000 samples.
Implementation. We compare SPACL with two recent baselines for OOD intent detection. MOGB (Li et al., 2025) constructs adaptive granular-ball decision boundaries to separate IND and OOD intents, while TBOS (Zhou et al., 2023) employs a tailored scoring strategy to jointly optimize IND and OOD detection. The widely used BERT (-base-uncasen former) is adopted as the base encoder. We fine-tune the model using standard hyperparameters commonly recommended in prior work (Li et al., 2025; Zhou et al., 2023). The AdamW optimizer employed, with the learning rate searched over {1e-5, 2e-5 50-5 5e-61 } the batch size over {16, 32), and the number of training epochs over (30, 503.
model (12-layer Trans-

---

Published as a conference paper at ICLR 2026
A.4 COMPUTATIONAL COMPLEXITY OF SPACL
The computational complexity of SPACL is summarized as follows. Let 6 denote the batch size, M the number of augmented views per sample, d the feature dimension, and Xps the number of hard positives per anchor. For each individual sample, we require:
(1) Positive selection: the time complexity is O( + Xpxx M). where the first term corre- sponds to constructing its similarity matrix and the second term comes from the greedy ranking. The overall complexity approximates ((d) d >
M > Xpr.
(2) Adversarial generation: the time complexity is ((Apps + 1)d (Apa + 1)d), where the first term corresponds to the MLP (generator) of ' + inputs and the second term comes from the second MLP (discriminator). The overall complexity approximates to (dd)).
(3) Negative screening: the time complexity is O(bd + b log b). where the first term corre- sponds to computing similarities between the anchor and all negatives, and the second term corresponds to the score ranking, thereby reducing the complexity approximately to (bd)).
Notably, all similarity scores (of anchor-positive and nngaov-ne have already been computed in steps (1) and (3). As result, the final complexity is Obdd + baf + d)d,, given the batch size b.
We then compare SPACL with traditional CL methods, such as MoCo v1 (He et al., 2020), MoCo v3 (Chen et al., 2021), SimCLR (Chen et al., 2020a), and SimSiam (Chen & He, 2021), as shown in Table 4, where K is the queue size for MoCo v1. Empirically, we also list all fine-tuning time on CIFAR-100 100 for 200 epochs using NVIDIA A100 GPU server.
Table 4: Comparison of computational complexity and training time.
Complexity source
Complexity
Training time
MoCo v1
ObbKd bK + bdl
4 hours 45 minutes
MoCo v3
O bd +bd + d²)
3 hours 51 minutes
SimCLR
O + bul + bul + b²)
hours 03 minutes hours 46 minutes
SimSiam
Obd + bd)
SPACL
O(dd + hat + b²d)
3 hours 26 minutes
We notice that the adversarial generation module is the main contributor to the additional compu- tation. Nevertheless, the overall computational load of our method is comparable that of MoCo v1/v3 and SimCLR, and our empirical results support this claim. In terms of memory usage, the additional components of SPACL compared to MoCo v1 are two MLPs: generator and a discrim- inator, both implemented as one- seee-hidden-laye MLPs. As a result, the additional memory usage is mainly determined by the number of hidden neurons (say V) as O(2Vd + Vd) O(Vi)).
A.5 GENERALIZATION TO DIFFERENT BACKBONES
To address the backbone generality of SPACL, we further evaluate SPACL by employing VIT-S/16 and ViT-B/16 on ImageNet-1K: IK. We strictly follow the setting from MoCo v3 (Chen et al., 2021), keeping the data augmentations and learning rate schedule unchanged, and simply substitute SPACL for the contrastive loss. For fairness, we also include SimCLR (Chen et al., 2020a), BYOL (Grill et al., 2020), and SwAV (Caron et al., 2020) under the same ViT training setup.
Table 5: Performance comparison between SPACI. with other CL. methods using ViT-based en- coders. Note that (+) denotes results are directly sourced from original papers, and the best resutls
are bold.
Model
MoCo v3
SimCLR
BYOL
SWAV
SPACL
ViT-S. 300-ep
72.5*
69.0*
71.0*
67.1*
74.3
VIT-B, 300-ep
76.5*
73.9*
73.9*
71.6*
78.6+0.10
As shown in Table 5, SPACL achieves the highest accuracy across both VIT-S/16 and ViT-B/16 encoders, outperforming other basclines. These results clearly confirm that the benefits of SPACL are not limited to convolutional backbones, but also exhibit strong performance on transformer-style encoders.

---

Published as a conference paper at ICLR 2026
To incorporate more baselines, also add DINO (Caron et al., 2021) as well as MAE (He et al., 2022), U-MAE (Zhang et al., 2022b), and i-MAE (Zhang & Shen, 2024) to our comparison, and report the results in Table 6. In this set of experiments, methods and evaluate them on CI, CIFAR-100, ImageNet- 100,, and ImageNet-IK.
ViT-B/16 encoder for all four
Table 6: Performance comparison between SPACL and DINO/MAE- methods. Note that (+) denotes results directly sourced from original papers, and the best results are in bold.
Method
CIFAR-10
CIFAR-100
ImageNet-100
ImageNet-IK
DINO
89.52
66.76
74.84
78.2*
MAE
90.78*
68.66"
72.23
67.48
U-MAE
68.90"
61.42
67.50*
58.50"
i-MAE
92.00*
69.50
74.63
70.21
SPACI (ours)
93.94+0.05
7.07
83.46
78.6+10.10
Across all four benchmarks, SPACL achieves the highest average performance, clearly surpassing both DINO and all MAE-based variants. For example, ImageNet-1000 the best-performing base- line DINO with 74.84%. In contrast, SPACL attains 83.46%, outerforming DINO by a sig- nificant margin of +8.62% points, demonstrating markedly stronger ability to learn discrimina- tive representations. In addition, SPACL also demonstrates greater robustness across datasets of varying resolution and scale, showing that its design generalizes effectively from small-scale im- ages SEJFAAR-1/100 to mid-scale (ImageNet- 100) and large-scale corpora (ImageNet- IK). Overall, SPACL consistently outperforms both DINO and the MAE-based methods, demonstrating superior performance.
A.6 IMPACT OF POSITIVE AND NEGATIVE SAMPLE BUDGET
We examine the effect of sampling budgets for hard examples within SPACL. Specifically, we focus on two tunable hyperparameters: the positive sample budget Aps, which controls how many hard positives are selected from the augmented pool P,, and the negative sample budget Aga, which determines what fraction of the combined in-batch and adversarial negatives are retained as hard negatives based on similarity-awant screening. To ensure a fair evaluation, we perform controlled ablations by varying one parameter at a time while fixing the other. That is, when analyzing the impact of Apr, we fix Ag 95%. Conversely, when varying Aor, we fix Xps. to 2 per anchor. This setup allows us to isolate the effect each factor independently and provides a clear understanding of their roles in shaping the overall performance.
+0.0
+0.2
+0.5
Performance (3)
0.0
Performncee
+1.0
+1.5
-0.6
CIFAR-100
+2.0
CIFAR-100
F815K-237
FB15K-237
.0.8
ImageNet-1K
+2.5
ImageNet-1K
2
3
8
0.75
0.85
0.90
Apn
Age
Figure 11: Performance change (A)
Figure 12:
Performance change (A)
across different Aps values.
across different Xoa values.
The aggregated result presented in Fig. 11 reveals a notable trend in screening positives within the context of SPACL with the supervised learning paradigm. Specifically, as the increase of Xps, there is corresponding decline in performance, with the highest performance achieved with Aps. = 2 (for CIFAR-100) or Xps 3 (for FB15K-237). One reason is that there are minimum constraints on the optimum with small Aps compared to other cases, that effectively increases the model learning capacity. Another reason could be attributed Eq. (3), where larger Xpn leads to larger s+ and decreases the norm of the gradient. Consequently, the gradient-based optimizer, commonly used deep learning, may struggle to efficiently update model parameters, thereby compromising performance. However, relying solely a single hard sample poses the risk of solution collapse, as indicated by Theorem 2.1. Consequently, we opt for a small Xpa with the value of 2 to mitigate

---

Published as a conference paper at ICLR 2026
this risk. In addition, the comparison results for negatives are presented in Fig, 12. Notably, models trained with smaller values Ag:. underperform compared to those using larger values, indicat- ing that insufficient negative samples lead to weaker performance. Conversely, with all negatives (equivalent to A* = 100%), performance drops substantially relative to the 90% setting (as shown in Table 1), highlighting the importance of balanced negative set. At last, also notice that, ImageNet-1k K is less sensitive to Aya. likely because of its higher intra-batch diversity, ie., a higher proportion of samples from different classes, reduces the probability that the additional samples selected by larger À values are trivial.


Reviews:
[{"review_id": "kCvYTQbP8w", "reviewer": "Reviewer_61L9", "summary": "The paper presents a theoretical and empirical investigation into contrastive learning (CL) with a focus on hard positive and negative sample selection. The authors first analyze the optimality and collapse conditions of CL through geometric and gradient-based formulations of the InfoNCE loss. Building upon this theoretical framework, they propose SPACL, a new algorithm that strategically emphasizes “hard” samples - instances that are challenging for the model to distinguish, while filtering out trivial ones.\n\nSPACL introduces two core innovations: 1) Hard Positive Selection: Identifies the most diverse and dissimilar augmented views via a farthest-point iterative process, thereby mitigating representation collapse. 2) Hard Negative Generation and Screening: Incorporates adversarially generated negatives and a relative screening mechanism to retain only the most informative (hard) negative samples.\n\nEmpirical results across multiple benchmarks, including CIFAR-10/100, ImageNet(-100), and three knowledge graph datasets (WN9, FB15K-237, FB15K), show consistent improvements over state-of-the-art methods such as SimCLR, MoCo, SupCon, and VarCon. Theoretical derivations are complemented by extensive ablation studies demonstrating the contribution of each module.", "questions": "1. Could the authors provide a more explicit analysis of SPACL’s computational and memory complexity relative to MoCo or SupCon? How feasible is adversarial negative generation in large-scale pretraining?\n\n\n2. Have the authors tested SPACL with transformer-based encoders (e.g., ViT, CLIP) to verify whether the observed benefits generalize beyond convolutional backbones?\n\n\n3. The hardness definition for positives depends on dissimilarity averaging. Have alternative formulations (e.g., mutual information or gradient-based difficulty) been explored?\n\n\n4. Since the adversarial component introduces an inner min–max loop, did the authors encounter stability issues during training? If so, how were they mitigated?\n\n\n5. The conclusion mentions potential applicability to negative-free CL frameworks (e.g., BYOL, SimSiam). Could the authors clarify how the proposed geometric analysis might translate to such setups?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper provides complete analyses of why and how sample hardness influences contrastive optimization, connecting gradient geometry and convex hull dynamics.\n\n2. The dual hard-sample selection (positive and negative) is elegantly motivated and empirically validated.\n\n\n3. Strong performance across multiple benchmarks and supervision levels, with consistent margins of improvement.\n\n\n4. Clearly isolate the role of each component (anchor selection, hard positives, adversarial negatives, and screening).\n\n\n5. Demonstrates applicability in both visual and relational (knowledge graph) contexts.", "weaknesses": "1. The derivations assume idealized settings (e.g., constant auxiliary encoder ggg, unit-sphere normalization) that may not hold in practical large-scale CL systems. The paper could better discuss these approximations.\n\n\n2. While effective, SPACL introduces additional overhead via adversarial negative generation and iterative positive selection. The paper lacks a quantitative runtime or resource comparison to baselines.\n\n\n3. Although two distinct domains are tested, the method’s performance in text or multimodal CL remains unexamined.\n\n\n4. Although λ parameters are ablated, the dependence of SPACL’s stability on these values may warrant further analysis, especially for large-scale datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Poot4mNzp6", "reviewer": "Reviewer_tz8p", "summary": "Contrastive learning has attracted considerable attention in recent years and has demonstrated strong empirical performance across various tasks. To address the issue of potential false negative samples in the construction of feature pairs, this paper conducts a comprehensive theoretical analysis from the perspective of optimality conditions and proposes the SPACL framework. Specifically, the method improves the process of constructing positive and negative pairs by selecting “hard” positive and “hard” negative samples, rather than treating all samples equally. The experimental results show that the proposed approach achieves excellent performance.", "questions": "Please refer to the weaknesses section. The main issues are concentrated on the performance analysis and experiments, as well as the need for further explanation or visualization in the ablation studies.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: This paper provides a relatively thorough theoretical analysis, outlining the optimality conditions and highlighting the beneficial role of hard samples in constructing feature pairs. The proposed geometric analysis offers a novel perspective and is compelling.\n\nS2: At the methodological level, SPACL integrates the selection of distant positive samples with adversarially generated hard negative samples, which effectively enhances the discriminative capability of the model and serves as a valuable complement to existing methods.\n\nS3: The extensive experimental results clearly validate the effectiveness of the proposed method, showing consistent and significant improvements over other methods across various benchmark datasets.\n\nS4: The paper provides a number of illustrative figures, which are helpful for understanding the proposed method.", "weaknesses": "W1: Intuitively, the proposed method could incur some additional computational cost. Thus, providing an analysis of memory usage and throughput would help to more comprehensively evaluate the efficiency of the method.\n\nW2: In the ablation study, the authors state that the performance drop in the “w/o AN” setting indicates that adversarially generated negative samples help sharpen the boundary of the negative region; otherwise, relying solely on batch-based or queue-based negatives would lead to a more blurred boundary. It would be helpful to provide a more fine-grained explanation or additional visualizations to further support this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GqMXtTHQIi", "reviewer": "Reviewer_PoEY", "summary": "This paper investigates the role of hard sample mining in Contrastive Learning (CL). The authors provide a theoretical analysis of the InfoNCE loss, deriving optimality conditions to motivate the importance of hard samples. Based on this analysis, they propose SnaPhArd Contrast Learning (SPACL), a method that systematically screens for and utilizes hard positive and hard negative samples. The paper reports strong empirical results on image classification and knowledge graph link prediction tasks, showing improvements over a selected set of baseline methods.", "questions": "- For clarity and reproducibility, we suggest moving key experimental details, such as the backbone architecture, from the appendix to the main paper.\n- Could you please clarify the computational overhead (e.g., in terms of training time or GPU memory) introduced by the SPACL components, particularly the adversarial generator and discriminator, compared to a simpler baseline like SimCLR?\n- It is recommended to provide more details on the usage of LLMs in research ideation/writing.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Clear Theoretical Grounding**: A major strength is the clean and elegant theoretical motivation derived from the InfoNCE loss function. This provides a principled \"why\" for the algorithm's design, which is commendable.\n- **Thorough Ablation Studies**: The paper includes an extensive set of ablation experiments that methodically dissect the SPACL algorithm. This rigorous analysis convincingly demonstrates the utility of each proposed component within its framework.", "weaknesses": "- **Limited Scope of Experimental Comparison**: The evaluation could be strengthened by including a broader set of baselines. Comparisons are notably missing against: (a) the current state-of-the-art paradigm of Masked Image Modeling (e.g., MAE), and (b) influential non-InfoNCE contrastive methods like DINO. This makes it difficult to assess the method's standing in the wider field of self-supervised learning.\n- **Lack of Discussion on Complexity**: The proposed method adds several components (e.g., an adversarial generator and a discriminator) that increase complexity. The paper would benefit from a discussion on the trade-off between this added complexity and the resulting performance gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JWAecFipQG", "reviewer": "Reviewer_Qn56", "summary": "This paper introduces SnaPhArd Contrastive Learning (SPACL), which focuses on hard positive and negative examples within the contrastive learning (CL) framework. The authors analyzed the optimality and crash risks of CL and propose a method designed to overcome these challenges by using carefully selected contrasting pairs. SPACL demonstrates empirical performance by outperforming existing SOTA methods on multiple downstream tasks, including image classification and knowledge graph link prediction.", "questions": "This paper verifies the effectiveness of SPACL on image classification and knowledge graph completion tasks. What challenges might SPACL encounter when dealing with other modalities or tasks? What are the advantages and disadvantages of SPACL when applied to different types of data or tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The theoretical analysis provides a unique perspective for contrastive learning.\n- Extensive experiments demonstrate the effectiveness of the method in this paper.\n- The writing is clear and well-structured.", "weaknesses": "- This paper presents a detailed theoretical analysis; however, its heavy reliance on numerous equations makes the ideas somewhat obscure. The core concepts may be difficult for non-specialist readers in the field to grasp. Adding a few intuitive illustrations might enhance the overall completeness of the paper.\n\n- While the paper demonstrates the efficacy of SPACL, the computational cost compared to other methods is not discussed in depth. Understanding the trade-offs in terms of computational requirements could be valuable, especially for real-world applications.\n\n- I noticed that the experiments in this paper were conducted on classic datasets such as CIFAR and ImageNet. I understand that these are the standard datasets used in previous contrastive learning works. However, I have a more exploratory question: given the rapid and relatively mature development of VLMs (including large models across various domains), can the proposed method in this paper truly provide a valuable tool for the academic or industrial community?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GVppGUWuO6", "reviewer": "Reviewer_MXc1", "summary": "This paper proposes a screening-based contrastive learning method to jointly sample hard positive and hard negative samples in contrastive learning for better representation diversity (via positives) and discriminability (via negatives). It also provides a theoretical analysis of sampling contrastive pairs and conditions for representation collapse.", "questions": "1. What does the error bar in Table1, Table 2 indicate? \n2. Please address questions in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Introduces a novel contrastive learning approach that jointly samples hard positives and negatives using a screening mechanism, with theoretical analysis supporting its design.\n2. Shows strong empirical performance in 4 image classification benchmarks across and compares its method to supervised, self-supervised, and weakly-supervised baseline methods. Also compares its method to link prediction baselines in 3 benchmarks.\n3. Provides a comprehensive ablation to show the effectiveness of the proposed method", "weaknesses": "1. In the supervised contrastive learning setting, the risk of false negatives is minimal since positives and negatives are explicitly defined by labels. However, in the self-supervised setting, false negatives can naturally occur when semantically similar samples are mistakenly treated as negatives. As SPACL’s focus on hard negatives can amplify the detrimental effect of false negatives, a discussion of false negatives would be beneficial.\n2. I understand that “relying solely on a single hard sample poses the risk of solution collapse, as indicated by Theorem 2.1”. However, Figure 4 shows that increasing the number of hard positives beyond 4 leads to lower performance. Could the authors explain why increasing the number of hard positives (which are intended to prevent collapse) can lead to worse performance compared to using only one hard sample?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)