Understanding Contrastive Learning via Distributionally Robust Optimization
Authors: Junkang Wu
Jiawei Chen
Jiancan Wu
Wentao Shi
Xiang Wang
Xiangnan He
Venue: NeurIPS 2023 poster (2023)
Keywords: contrastive learning
distributionally robust optimization
mutual information
TL;DR: The paper investigates the robustness of contrastive learning and its connection to distributionally robust optimization, introducing a novel loss function and providing insights into the role of temperature for improved performance.

Abstract: This study reveals the inherent tolerance of contrastive learning (CL) towards sampling bias, wherein negative samples may encompass similar semantics (\eg labels). However, existing theories fall short in providing explanations for this phenomenon. We bridge this research gap by analyzing CL through the lens of distributionally robust optimization (DRO), yielding several key insights: (1) CL essentially conducts DRO over the negative sampling distribution, thus enabling robust performance across a variety of potential distributions and demonstrating robustness to sampling bias; (2) The design of the temperature $\tau$ is not merely heuristic but acts as a Lagrange Coefficient, regulating the size of the potential distribution set; (3) A theoretical connection is established between DRO and mutual information, thus presenting fresh evidence for ``InfoNCE as an estimate of MI'' and a new estimation approach for $\phi$-divergence-based generalized mutual information. We also identify CL's potential shortcomings, including over-conservatism and sensitivity to outliers, and introduce a novel Adjusted InfoNCE loss (ADNCE) to mitigate these issues. It refines potential distribution, improving performance and accelerating convergence. Extensive experiments on various domains (image, sentence, and graph) validate the effectiveness of the proposal.

Understanding Contrastive Learning via Distributionally Robust Optimization
Junkang Wu Jiawei Chen 2* Jiancan Wu' Wentao Shi Xiang Wang" Xiangnan Hel
University of Science and Technology of China
"Zhejiang University
{jkwu0909, wujcan, xiangwang1223, ciangnanhe)@gmale com, sleepyhunt@zju. edu. en, shiventao1230mail www.ustcc.e... edu. cn
Abstract
This study reveals the inherent tolerance of contrastive learning (CL) towards sampling bias, wherein negative samples may encompass similar semantics (e.g. labels). However, existing theories fall short in providing explanations for this phenomenon. We bridge this research gap by analyzing CL through the lens of distributionally robust optimization (DRO), yielding several key insights: (1) CL essentially conducts DRO over the negative sampling distribution, thus enabling robust performance across a variety of potential distributions and demonstrating robustness to sampling bias; (2) The design of the temperature is not merely heuristic but acts as Lagrange Coefficient, regulating the size of the potential dis- tribution set; (3) A theoretical connection is established between DRO and mutual information, thus presenting fresh evidence for "InfoNCE as an estimate of MI" and new estimation approach for (-diverggnceebaseed generalized mutual information. We also identify CL's potential shortcomings. including over-conservatisn and sensitivity outliers, and introduce novel Adjusted InfoNCE loss (ADNCE) mitigate these issues. It refines potential distribution, improving performance and accelerating convergence. Extensive experiments on various domains (image, sen- tence, and graphs) validate the effectiveness of the proposal. The code is available https: /ttgt/gttttttt////////////t ctwm junkangw//// ADDCE..
Introduction
Recently, contrastive learning (CL) [1; 2; has emerged one of the most prominent sule-vupe methods, due to its empirical success in computer vision [4; 5; 6; 7; 8), natural language processing [9; 10] and graph [11; 12]. The core idea is to learn representations that draw positive samples (e.g., augmented data from the same image) nearby and push away negative samples (e.g., augmented data from different images). By leveraging this intuitive concept, unsupervised learning has even begun to challenge supervised learning.
Despite its effectiveness, recent study [13] has expressed concerns about sampling bias in CL. Negative samples are often drawn uniformly from the training data, potentially sharing similar semantics (e.g., labels). Blindly separating these similar instances could result in a performance decline. To address sampling bias, existing research has proposed innovative solutions that either approximate the optimal distribution of negative instances [13; 14] integrate auxiliary detection module to identify false negative samples [15]. Nevertheless, our empirical analyses challenge the prevailing understanding of CL. Our findings indicate that CL inherently exhibits robustness towards
Jiawei Chen and Xiangnan He are the corresponding authors. Xiang Wang also affiliated with Institute of Artificial Intelligence, Institute Dataspace, Hefei Compre- hensive National Science Center.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

sampling bias and, by tuning the temperature т, can even achieve comparable performance with representative methods for addressing sampling bias (e.g.. DCL [13] and HCL [14]). This revelation prompts two intriguing questions: !) Why does Cl. display tolerance to sampling bias? and 2) How can we thoroughly comprehend the role of the temperature t? Despite recent analytical work on CL. from various perspectives [13; 14; 16], including mutual information and hard- hard-mining, these studies fail to provide a satisfactory explanation for our observations.
To address this research gap, we investigate CL from the standpoint of Distributionally Robust Optimization (DRO) [17; 18; 19; 20; 21; 22]. DRO refers to a learning algorithm that seeks to minimize the worst-case loss over a set of possible distributions. Through rigorous theoretical analysis, we prove that CL essentially performs DRO optimization over a collection of negative sampling distributions that surround the uniform distribution, constrained by KL-divergence- measure (we term the DRO-type equivalent expression of CL. as CL.-DRO). By enabling CL to perform well across various potential distributions, DRO equips it with the capacity to alleviate sampling bias. Furthermore, our findings emphasize that the design of T is not merely heuristic but acts Lagrange Coefficient, regulating the size of the potential distribution set. By integrating the principles of DRO, we can also gain insights into the key properties of CL such as hard-mining, and investigate novel attributes such as variance control.
Furthermore, we delve into the properties of CL-DRO, transcending the boundary of KL divergence, but exploring a broader family of distribution divergence metrics - g-divergence. It exhibits greater flexibility and potentially superior characteristics compared to KL-divergence. Interestingly, we establish the equivalence between CL-DRR under any ó-divergence constraints and the corresponding tight variational representation of d-divergence [23]. This discovery provides direct theoretical support for the assertion that "InfoCCE is an estimate of mutual information (MI)", without requiring redundant approximation as before work [24; 3]. Additionally, it provides a novel perspective for estimating arbitrary <-divergence, consequently enabling the estimation of o-divergence-based mutual information.
In addition, we discover some limitations in CL through the lens of DRO. One is that the DRO paradigm is too conservative, focusing on worst-case distribution with giving unadvisedly over-large weights on hard negative samples. Secondly, the presence of outliers can significantly impact DRO [25], resulting in both drop in performance and training instability. In fact, both challenges stem from the unreasonably worst-case distribution weights assigned to negative samples. To address this weakness, we propose DDjusted InfoNCE (ADNCE) that reshapes the worst-case distribution without incurring additional computational overhead, allowing us to explore the potentially better negative distribution. Empirical studies show that our method can be applied to various domains (such images, sentences, and graphs), and yield superior performance with faster convergence.
2 Related Work
Contrastive Learning. With numerous CL methodologics (e.g., SimCLR [4], Moco [6]. BYOL [26]. SwAV [27], Barlow Twins [8]) having been proposed and demonstrated the ability to produce high-quality representations for downstream tasks, theoretical research in this field has begun to emerge. Wang al. [28] revealed the hard-mining property and Liu et al. [29] emphasized its robustness against dataset imbalance. Meanwhile, Xue et al. [30] showed that representations learned by CL provably boosts robustness against noisy labels, and Ko et al. [31] investigated the relationship between CL and neighborhood component analysis. More closely related to our work is the result of Tian et al. [16] that analyzed CL. from minmax formulation.
Distributionally Robust Optimization. In contrast to conventional robust optimization approaches [32; 33; 34], DRO aims to solve the distribution shift problem by optimizing the worst-case error in pre-defined uncertainty set, which is often constrained by o-divergence [20; 17]. Wasserstein distance [21; 19; 18] and shape constraints [35; 36; 37J. Meanwhile, Michel et al. [38; 39] parametrized the uncertainty set, allowing more flexibility in the choice of its architecture. Zhai et al. [25] focused on issues related to the existing DRO, such sensitivity to outliers, which is different from the aforementioned research.
Mutual Information. The MI between two random variables, X and Y. measures the amount of information shared between them. To estimate MI, the Donsker- Donsker-Varadhan Varadhan target [40] expresses the lower bound of the Kullback-Leibler (KL) divergence, with MINE [24] being its parameterized
2

---

version. However, according to studies [41; 42], the MINE estimator may suffer from high variance. To address this issue, InfoNCE [3] extends the unnormalized bounds to depend on multiple samples. thereby eliminating the constraints of both MINE's reliance on conditional and marginal densities. Another line of MI focuses on boundedness and low variance. Towards this end, RPC [43] introduced relative parameters to regularize the objective and FLO [44] leveraged Manchel-Legendrr optimization. Nevertheless, they are still hard to explain how MI optimization elucidates the properties of CL and facilitates effective CL in practice.
3 Contrastive learning as Distributionally Robust Optimization
In this section, we firstly uncover the tolerance of CL towards sampling bias. Then we analyze CL and r from the perspective of DRO. Finally, we empirically verify the above findings.
Contrastive Learning is conducted two random variables X, Y over feature space A. Suppose we have original samples 2; which obey the distribution Px. positive samples drawn from the distribution (ar, y ~ Py|x. and negative samples selected from the distribution (Ir, () Py. The goal of CL. is to learn an embedding go- : A -> Rdd that maps an observation (i.e., x. y, y into a hypersphere. Let (g) go (v)) be the similarity between two observations and on the embedding space. CL draws positive samples closer (i.e. f (z), 9o ())))) 1 for (a. yt ) Pxy) and pushes the negative samples away (i.e., (go(()) (v)) 4 for (x, v) PxPr). For brevity, in this work, we simply shorten the notations PYix as Po. and Py as Qo- Also, we primarily focus on InfoNCE loss LIsaNCE as a representative for analyses, while other CL's loss functions exhibit similar properties. InfoNCE can be written as follow:
Notation.
efo(zin)/f
LININNCE -Epx EE *//]] log Equ [c(o(/))/]]] -Exx Er log
Eq, (fole.0z)))
3.1 Motivation
In practical applications of contrastive learning (CL), negative samples (y) are typically drawn uniformly from the training data, which might have similar semantics (for instance, labels). This introduces a potential issue of sampling bias, as posited by Chuang et al. [13]. since similar sam- ples are being forcefully separated. Recent re- search [15; 13] observed that compared to the em- ployment of ideal negative sampling that selects instances with distinctly dissimilar semantics, the existence of sampling bias could result in signif- icant reduction in performance.
Table 1: InfoNCE has the ability to mitigate sampling bias. We compare the performance of various CL methods with/without fine-tuning 7 (marked with r*/ro) We also report relative improvements compared with SimCLR(7").
CIFARIO
STL.10
Model
Top-1
Top-1
SimCLR(7o)
91.10
0.5
81.05
0.5
SimCLR(r"))
92.19
0.3
87.91
0.2
DCL(m)) DCL(r")
92.00 92.00 (-0.2%)
0.5
84.26 (-4.2%)
0.5
92.09 (-0.1%)
0.3
88.20 (+1.0%)
0.2
HCL(T)) HCL(T")
92.12 (-0.0%)
0.5
87.44 (-0.5%)
0.5
In this study, we observe an intriguing phe- nomenon where CL inherently demonstrates re- silience to sampling bias. We empirically evaluate CL on two benchmark datasets, CIFARIO and STL10, with the findings detailed in Table 1. Notably, we make the following observations: 1) By fine-tuning the temperature т, basic SimCLR demonstrates sigmificant improvement, achieving perfor- mance levels comparable to methods specifically designed to address sampling bias (i.e., SimCLR(7") versus SimCLR(ro). DCL and HCL); 2) With an appropriately selected т, the relative improvements realized by DCL and HCL are marginal. These insights lead us to pose two compelling questions: 1) Why does CL exhibit tolerance to sampling bias? and 2) What role does T play, and why is it so important?
92.10 (-0.0%)
0.3
87.46 (-0,5%
0.2
3.2 Understanding CL from DRO
In this subsection, first introduce DRO and then analyze CL from the perspective of DRO.
NNoe that our formula adheres to the commonly used InfoNCE expression (c.g., the SimCLR setting [4]), in which the denominator does not include Er, f0 )/7]. This is also used in Yeh et al. [45].
3

---

A. Distributionally Robust Optimization (DRO).
The success of machine learning heavily relies the premise that the test and training distributions are identical (a.k.a. iid assumption). How to handle different data distributions (ak.a. distribution shift) poses great challenge machine learning systems. Fortunately, Distributionally Robust Optimization (DRO) provides potential solution to mitigate this problem. DRO aims to minimize the worst-case expected loss over a of potential distributions Q, which surround the training distribution Qo and are constrained by a distance metric Dó within a specified robust radius Formally, DRO can be written as:
CDRO max Ex[[(x; Q
s.t. D2(Q11Q0) S n
where Qo) measures the ş-divergence between two distributions, and L(x; 0) denotes the training loss on input æ. Intuitively. models incorporating DRO enjoy stronger robustness due to the presence of Q that acts as an "adversary", optimizing the model under distribution set with adversarial perturbations instead of single training distribution.
B. Analyzing CL from DRO. We first introduce the CL-DRO objective: Definition 3.1 (CL-DRRO. Let Po and Qo be the distributions of positive and negative samples respectively. Let be the robust radius. CL-DRO objective is defined as:
CL- = Ex EPo ( max Ee (fo(z,
S.I. (2(Q1)0)) S.
(3)
The objective CL-DRO be understood the DRO enhancement of the basic objective Liasiic - Ep Er, [o y=] [f, y)]], which aims to increase the embedding simi- larity between the positive instances and decreases that of the negative ones. CL-DRO imporves Trasic by incorporating DRO on the negative side, where fo(2, y) is optimized a range of potential distributions. Consequently, CL-DRO equips the model with resilience to distributional shifts of negative samples.
Theorem 3.2. By choosing KL divergence DEL (Q) = f Q da, optimizing CL-DRO (cf. Qo Equation (3)) is equivalent to optimizing CL (InfoNCE,cf. Equation (1));
(Bo fo(2, alDxz 7) + ++((Ba Q
CKL DRO
Exx (ool,o(
min max
a20,3
Qo
sola.stoo"00
=-Bp [a" (1) log Equ (mlal + Constant
=a Lngnnc Conslant,
(4)
where a, represent the Lagrange multipliers, and ar (<()) signifies the optimal value of a that minimizes the Equation (4), serving as the temperature T CI-
The proof is presented in Appendix A. Theorem 3.2 admits the merit of CL as it is equivalent to performing DRO on the negative samples. The DRO enables CL to perform well across various potential distributions and thus equips it with the capacity alleviate sampling bias. We further establish the connection between InfoNCE loss with the unbiased loss that employs ideal negative distribution Lunbiased Epx PE (fo(z, y+]] EEquaaa [fo v)]].
Theorem 3.3. [Generalization Bound] Let LiguNce be an estimation of InfoNCE with N negative samples. Given any finite hypothesis space of models, suppose So E [M1, M2) and the ideal negative sampling distribution Qideal satisfies DEL (Qidall (1Q)) I 17, we have that with probability at least 1 pt
Lumbiased S &LugUNCE + B(p,NT))
(5)
where N,T) ( M.)T))
N In( 211).
N-1-exp((Mc M.)T7)
As we mainly focus on model robustness on negative distribution, here we disregard the constant term present in Equation (4), and omit the error from the positive instances. The proof is presented in

---

Appendix A.2. Theorem 3.3 exhibits the unbiased loss is upper bound by InfoNCE if sufficiently large data size is employed (considering (p, N, + N - 0). This bound illustrates how impacts the model performance, which will be detailed in the next subsection.
3.3 Understanding r from DRO
Building on the idea of DRO, we uncover the role of r from the following perspectives:
A. Adjusting robust radius. The temperature and the robust radius 1/ conform to:
Corollary 3.4. [The optimal a Lemma 5 of Faury et al. [46]] The value of the optimal or (i.e.,
can be approximated as follow:
V Va.lfo(z.u))
(6)
where Vooloots v)] denotes the variance of fo(2, under the distribution Qo-
This corollary suggests that the temperature parameter T is function of the robust radius v and the variance of fo- practical applications, tuning T in CL. essentially equivalent to adjusting the robustness radius r7.
The aforementioned insight enhances our understanding of the impact of temperature on CL perfor- mance. An excessively large value of 7 implies a small robustness radius potentially violating the condition ( (Qideal Iu) I r) as stipulated Theorem 3.2. Intuitively, large delineates restricted distribution set in DRO that may fail to encompass the ideal negative distribution, leading diminished robustness. Conversely, as decreases, the condition D D [[Q) S 1 may be satisfied, but it expands the term (p,, N) and loosens the generalization bound. These two factors establish trade-off in the selection T, which we will empirically validate Subsection 3.4.
B. Controlling variance of negative samples.
Theorem 3.5. Given any é-divergence, the corresponding CL.DDR objective could be approximated mean-variance objective:
1 1 VoolJo( vil)];
LOLDRO ~ -Bpx L (PP, (Ba (ooo + 2r (12)
where (1) denotes the the second derivative value of Φ(-) at point 1, and V VQ. [fo] denotes the variance of f under the distribution Qo-
Specially, if we consider KL. divergence, the approximation transforms:
CKL CLO EE
PP fol (Equ v)] 1 2T
Vo.
(8)
The proof presented in Appendix A.4. Theorem 3.5 provides a mean-variance formulation for CL. Compared to the basic objective Luusie CL introduces an additional variance regularization on fluctuations in similarity among
negative samples, governed by the parameter т, which modulates extensively validated in recent negative samples. The efficacy of variance regularizers has been research [20; 47; 48]. We consider that variance control might be a reason of the success of CL.
C. Hard-mining Note that the worst-case distribution in CL-DRO can be written as Q" Qo (/7 (refer Appendix A.6). This implies that the model in CL optimized under exp(foee.)/1]
the negative distribution Q", where each negative instance is re-weighted by (x.yff(r.y))//1 This outcome vividly exhibits the hard-mining attribute of CL, with T governing the degree of skewness of the weight distribution. A smaller suggests
Egp (aplfa(x..)/)
skewed distribution and intensified hard-mining, while a larger 7 signifies a more uniform distribution and attenuated hard-mining. Our research yields similar conclusions those found in [16; 28), but arrive at this finding from novel perspective.
3.4 Empirical Analysis
In this section, we conduct the following experiments to verify the above conclusions.
5

---

Variance of Neg
Mean of Pos
Robustness experiments w.r.t
T=
0.9
T= 0.4
0.25
0.8
0.99
0.6
Relative ACC
t= 0.8
0.7
0.98
0.15
0.6
T=
Noise Ratio
0.5
0.96
0.5
0.8
0.95
1.0
0.00
200
Epoch
Epoch
Figure 2: controls the variance of negative samples score. We alter the value T and record corresponding variance of negative samples and mean value of positive samples.
Figure 1: varies with the level of sam- pling bias. We report the relative topl ac- curacy of SimCLR w.r.. different across different rate r of false negative samples.
varies with the level of sampling bias. To study the effect of I model performance, conduct experiments on the CIFAR 10 dataset and manipulate the ratio of false negative samples based on the ground-truth labels. ratio of 1 implies retaining all false negative instances in the data, while ratio of implies eliminating all such instances. The detailed experimental setup can refer to the Appendix B. From Figure we make
observations: There is an evident trade-off the selection of т. As increases from 0.2 to 1, the model's performance initially improves, then declines, aligning with our theoretical analysis. 2) As the ratio false negative instances increases (r ranges from Ó to 1), the optimal 7 decreases. This observation is intuitive a higher ratio of false negative instances implies larger distribution gap between the employed negative distribution and the ideal, thereby necessitating a larger robust radius 1) satisfy the condition DKz (Qideal ([Qu) S As Corollary 3.4, a larger robust radius corresponds to a smaller
controls the variances of negative samples. We experiment with an array of T values and track the corresponding variance of So y) for negative samples and the mean value of fo(xr, for positive samples in CIFAR10 10. As illustrated in Figure 2, we observe a decrease in the variance of negative samples and an increase the mean value of positive samples as diminishes. Although this observation might be challenging to interpret based on existing studies, it can be easily explained through Equation (8). Note that governs the impact of the variance regularizer. smaller brings larger variance penalty, resulting in reduced variance of negative instances. Simultaneously. as the regularizer's contribution on optimization intensifies, proportionately diminishes the impact of other loss component on enhancing the similarity of positive instances, thereby leading to lower mean value of positive instances.
MNeanVariiane objective achieves competitive per- formance. We replace the InfoNCE with the mean- variance objective (cf. Equation (8)) and evaluate the model performance. The results are shown in Table 2, Remarkably.sude a simple loss can can deliver compet- itive results InfoNCE, thereby affirming the correct- ness of the approximation in Theorem 3.5. Nevertheless, marginally underperforms compared to InfoNCE, as it merely represents second-order Taylor expansion of InfoNCE. However, it worth highlighting that the mean- mean-variance objective is more efficient.
Table 2: Mean-variance objective (MW) achieves comparable performance, We replace the loss function with variance penalty on negative samples.
Cifar10
St110
Model
Top-1
Top-1
SimCLR(Go) SimCLR(7")
91.10
0.5
81.05
92.19
0.3
87.91
MW
91.81
0.3
87.24
0.2
4 Relations among DRO, Constrastive Learning and Mutual Information
Through extensive theoretical analysis, we have far demonstrated that CL is fundamentally a CL-DRO objective that conduct DRO negative samples. And it is well known that CL is also equivalent to mutual information estimation. natural question arises: ly there a connection between CL-DRO and MI, as both are theoretical understanding of CL?

---

To make our conclusion more generalized, we would like to transcend the boundary of KL divergence, but exploring a broader family of g-divergence. First of all, we would like to extend the definition of MI to utilize d-divergence:
Definition 4.1 (Ø-MI), The d-divergence-baaee based mutual information is defined as:
L) = ((Xx))=DD/(XXXX)))XXXXXX))) Ex (Psoool(O)))
Comparing to KL-divergence, d--divergence showcases greater flexibility and may exhibits superior properties. o-MI, as a potential superior metrie for distribution dependence, has been studied by numerous work [49; 50].
Theorem 4.2. For distributions P. Q such that < Q let be a set of bounded measurable functions. Let CL-DRO draw positive and negative instances from and Q, marked as C ( Then the CL-DRO objective is the light variational estimation of d-divergence. In fact, have:
CL-DRO (P. Q).
(p(()))) = max /EF
DC (P,Q) = max
JEF Ep
- min Bols"(1 (J (S- A]]}-
(10)
XER
Here, the choice of & CL-DRO corresponds to the probability measures in Dp(PIlQ). And ø denotes the corvex conjugate.
The proof is presented in Appendix A.5. This theorem establishes the connection between CL-DRO and o-divergence. If we replace P with Po and Q with Qo. and simply hypothesis that the metric function fo has a sufficiently large capacity, we have the following corollary:
Corollary 4.3, Exx [Do (P)) is tight variational estimation of (X;
InfoNCE is tighter MI estimation. The prevailing and widely used variational approximation of d-divergences is the Eq 66" (f)]}. However, it has been observed by Ruderman et al. [51] that this expression is loose when applied to probability measures, as it fails to fully consider the fundamental nature of divergences being defined between probability distributions. To address this limitation, let us further assume, as we do for R., that [1[ = Under this common assumption, we arrive a more refined formulation: D&(P = fax ( [R [J] minXeR (A + IFq (b (ʃ {]])}. Notably, this result, embracing the infimum over the spirit of Ruderman et al. [51], appears to be consistent with our proof in Theorem 4.2. Hence, we argue that "InfoNCE is tighter MI estimation."
target (LDv), defined as D& (PllQ) max [Ep
Furthermore, this corollary provides rigorous and succinet methodology for deriving InfoNCE from Variational MI. Specifically, given 6(x) x log
--++ + for the case of KL-divergence, its conjugate (x(x) E" can be computed. Subsequently, by resolving the simple convex problem min (A+ rE (0 [(a.w)_A]] }]}, we derive the optimal choice of >, denoted as X = r log E.q [e (x.w)/) Insert- ing X into Equation (10). D Po (o)) simplifies to max, PE [[ (xx,+)) log Eo, (x..))))) which is the minus of the InfoNCE objective.
DRO bridges the gap between MI and InfoNCE. Although previous works such as MINE[24] and CPC[3] have demonstrated that InfoNCE limitations [41]. For instance, MINE uses a critic in Donsker- Varadhan target (IDV) to derive a bound that is neither an upper nor lower bound on MI, while CPC relies on unnecessary approximations in its proof, resulting in some redundant approximations. In contrast, Theorem 4.2 presents a rigorous and practical method for accurately estimating the tight lower bound of MI.
lower bound for MI, they still have significant
DRO provides general MI estimation. Although a strict variational bound for MI has been proposed in Poole al. [41], their discussion is limited the choice of KL-divergence. Theorem 4.2 is suitable
for estimating various O-.I. For example. if we consider the case of x3 divergence. given by Φ(z)= =
x2. The variational representation P || Qo) max, (Er. y+] [s v)) (/ v))) where Vo. v)] represents the variance of f on distribution Q. Our theoretical framework offers the opportunity to estimate flexible é-MI, which can be adapted to suit specific scenarios.
2/2 (x -)=. we could obtain convex conjugate 60(x)
becomes Y) = (Po
7

---

Method
5.1
Shortcomings of InfoNCE
Based on the understanding of CL from DRO perspective, some weaknesses of InfoNCE revealed:
Too conservative. Distributionally Robust Optimization (DRO) focuses on the worst-case dis- tribution, often leading to overemphasis the hardest negative samples by assigning them disproportionately large weights. More precisely, the weights are proportional to exp (x, (x(,)))TT] causing samples with the highest similarity to receive significantly larger weights. However, practice, the most informative negative samples are often found in the "head" region (e.g.. top-20% region with high similarity) rather than exclusively in the top- 1% highest region [52]. Overemphasizing these top instances by assigning excessive weights is typically suboptimal.
Sensitive outliers. Existing literature [25] reveals that DRO markedly sensitive to outliers, exhibiting significant fluctuations in training performance. This is because outliers, once they appear regions of high similarity, can assigned exceptionally high weights. Consequently, this phenomenon unavoidably influences the rate of model convergence during the training phase.
5.2 ADNCE
Our goal refine the worst-case distribution, aim- ing to assign more reasonable weights to negative instances. The aspiration to create flexible dis- tribution that can be adjusted to concentrate on the informative region. This is accomplished by incorpo- rating Gaussian-like weights, as defined below:
InfoNCE.rn0.07
InfoNCE,r=0.2
Adjusted DRO
InfoNCE,
Weight Scale
w(p=-07,.--100)
wfg=0.4.0=1.0)
wig=0011000.5))
1 fo M
w(fo(x V), & ov2
expl-
(11) where f and a are two hyper-parameter we could control. As is illustrated in Figure 3, μ controls the central region of weight allocation, whereby samples closer to μ have larger weights, while controls the height of the weight allocation in the central region. Intuitively, smaller results in more pronounced weight discrepancy among the samples.
-1.0
-0.5
0.0
1.0
Similarity Score
Figure 3: We visualize the training weight of negative samples w.rl. similarity score. InfoNCE (in BIUE) over-emphasize hard neg- ative samples, while ADNCE utilizes the weight (in ORANGE, GREEN, BROWN) to adjust the distribution.
Inserting the weights into InfoNCE formula, we ob- tain the novel ADjusted InfoNCE (ADNCE):
LADNCE -Eplfo (2, y ///] + log Eg, [w[ (uooo v), oleoo
(12)
where Zp.o denotes the partition function of w(f(x, (r,, u), n, o), iee. Zy = E2 (eglw((o(ue y), a),e pt, a)}-
It is noteworthy that proposed approach only involves two modified lines of code and introduce no additional computational overhead, as compared to InfoNCE [4]. Pytorch-style pscudocode for ADNCE is given in Appendix.
6 Experiments
We empirically evaluate our adjusted InfoNCE method - ADNCE, and apply it as modification to respective classical CL. models image, graph, and sentence data. For all experiments, y is treated as a hyper-parameter to adjust the weight allocation, where a is set to 1 by default. For detailed experimental settings, please refer Appendix.
6.1 Image Contrastive Learning
We begin by testing ADNCE on vision tasks using CIFARIO. CIFAR100, and STL10 datasets. We choose SimCLR [4] as the baseline model (termed InfoNCE(To)). We also include two variants

---

Table 3: Performance comparisons multiple loss formulations (ResNet50 backbone, batchsize
256). Top-1 accuracy with linear evaluation protocol. To means T = 0.5, T represents grid searching on and a-CL-direct is similar work [16] on CL theory. Bold is highest performance. Each setting repeated 5 times with different random seeds.
CIFAR10
STL10
CIFAR100
Model
100
200
300
400
100
200
300
400
100
200
300
InfoNCE (Ta)
85.70
89.21
90.33
91.01
75.95
78.47
80.39
81.67
59.10
63.96
66.03
66.53
InfoNCE (r")
86.54
89.82
91.18
91.64
81.20
84.77
86.27
87.69
62.32
66.85
68.31
69.03
a-CI-direct
87.65
90.11
90.88
91.24
80.91
84.71
87.01
87.96
62.75
66.27
67.35
68.54
ADNCE
87.67
90.65
91.42
91.88
81.77
85.10
87.01
88.00
62.79
66.89
68.65
69.35
Table Sentence embedding performance on STS tasks (Spearman's correlation, "all" setting). Bold indicates the best performance while underline indicates the second best on each dataset. results from SimCSE [9]; all other results are reproduced or reevaluated by ourselves.
Model
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
Gl embeddings (avg.)*
55.14
70.66
59.73
68.25
63.66
58.02
53.76
61.32
BERT -flow 4
58.40
67.10
60.85
75.16
71.22
64.47
SeiTeas--hhitning
57.83
66.90
60.90
75.08
71.31
68.24
63.73
66.28
CT-BERT
61.63
76.80
68.47
77.50
76.48
74.31
69.19
72.05
SimCSE-BERR (T))
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.25
SimCSE-BERT (T")
71.37
81.18
74.41
82.51
79.24
78.26
70.65
76.81
ADNCE-BERR BERT
71.38
81.58
74.43
82.37
79.31
78.45
71.69
77.03
(Ta)
70.16
81.77
73.24
81.36
80.65
80,22
68.56
76.57
SimCSE- RoBERTRaa
68.20
81.95
73.63
81.83
81.55
80.96
69.56
76.81
ADNCE-Ro RoBERTabae
69.22
81.86
73.75
82.88
81.88
81.13
69.57
77.10
of vanilla InfoNCE for comprehensive comparisons. One approach is applying a grid search for (termed InfoNCE(r")), which serves to confirm its importance discussed in Section 3.4. The other is from Tian el al. [16], which proposes a novel approach for setting for the value of r (termed o-CL-direct).
As shown in Table 3, a grid search for has crucial impact on the model performance. This impact is significant throughout the entire training process, from the early stage (100 epochs) to the later stage (400 epochs). Additionally, ADNCE exhibits sustained improvement and notably enhances performance in the early stages of training. In contrast, while a-CL-direct introduces novel approach for setting the value of т. its essence remains weighted towards the most difficult negative samples, hence yielding similar performance compared to fine-tuning using grid search. In Figure 4, we plot the training curve to further illustrate the stable superiority of ADNCE.
88
Table 5: Self-supervised representation learning on TUDataset: The baseline results excerpted from the published papers.
81-
Methods
RDT-B
NCI1
PROTEINS
DD
ee-
0-CL- rect
ARNCE
node2vec
54.9+1.6
57.5+3.6
sub2vec
71.540.4
52.811.5
53,015.66
88
graph2vee
75.8+110
73.2-11.8
73.312.1
84
InfoGraph
8..5+114
76.2+1111
74,4+0..
72.9+1.8
1991
JOAO
85.3
78.110.5
74.6.0..4
77.310.5
JOAOv2
86.411.
78.4.0.5
74.11111
77.411.2
76.
RINCE
90.9+0..6
78.6=0.4
74.7+...8
78.7+0044
Epach
GraphCL (ra)
89,5+0.8
77.9=0.4
74,4+0.55
78.6+00
GraphCL. (r*)
90.7+0.6
79.2-0.3
74.7+0.6
78.5++1.0
Learning curve for Top-1 accuracy by linear eval- uation on CIFARIO and STL10.
Figure
ADNCE
91.4-10.3
79.3-0.7
75.110.66
79.210.6

---

6.2 Sentence Contrastive Learning
We follow the experimental setup GraphCL [12] and conduct evaluations on semantic textual similarity (STS) tasks in an unsupervised setting. Specifically, we evaluate the model on STS 2012-2016, ben-hanch,,rk, and checkpoints of BERT and RoBERTa, and randomly sample sentences from the English Wikipedia.
To ensure fair comparisons, we use pre-trained
As observed in Table 4, ADNCE consistently outperforms InfoNCE, achieving an average Spearman's correlation of 77%. The ease of replacing InfoNCE with ADNCE and the resulting substantial performance improvements observed in BERT and RoBERTa serve testaments to the efficacy and broad applicability of ADNCE. Furthermore, the improvements of T over To emphasize the significance of selecting a proper robustness radius.
6.3 Graph Contrastive Learning
To study the modality-agnostic property of ADNCE beyond images and sentences, we conduct an evaluation of its performance TUDataset [53]. We classical graph CL model GraphCL [12] as the baseline. To ensure a fair comparison, we adhered to the same protocol used in GraphCL [12], employing techniques such as node dropout, edge perturbation, attribute masking, and subgraph sampling for data augmentation. We replace the vanilla InfoNCE with two different variants, namely GraphCL(T*) obtained through grid search, and our proposed ADNCE.
Table demonstrates that ADNCE outperforms all baselines with a significant margin on four datasets, especially when compared to three of-the-ar InfoNCE-baan based contrastive methods, GraphCL, JOAO, and JOAOv2, thereby setting new records on all four datasets. The improvements observed in GraphCL(T") relative GraphCL(TT) align with understanding of DRO.
7 Conclusion and Limitations
We provide a novel perspective on contrastive learning (CL) via the lens of Distributionally Robust Optimization (DRO), and reveal several key insights about the tolerance sampling bias, the role of r, and the theoretical connection between DRO and MI. Both theoretical analyses and empirical experiments confirm the above findings. Furthermore, we point out the potential shortcomings of CL from the perspective of DRO, such as weer-oonservatism and sensitivity to outliers. To address these issues, we propose a novel CL loss ADNCE, and validate its effectiveness in various domains.
The limitations this work mainly stem from two aspects: 1) Our DRO framework only provides theoretical explanation for InfoNCF--based methods, leaving crucial gap in the success of CL. without negative samples [4; 26). 2) ADNCE requires weight allocation to be adjusted through parameters and cannot adaptively learn the best reweighting scheme.
Acknowledgments and Disclosure of Funding
This work is supported by the National Key Research and Development Program of China (2021ZD0111S)) 1180)) the National Natural Science Foundation of China (62372399, 9227010114 14, 62302321), the Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJU-SIAS-00) and supported by the University Synergy Innovation Program of Anhui Province (GXXT-2022-040).
References
[1] Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application face verification. In CVPR (1), pages 539-546. IEEE Computer Society, 2005.
[2] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR (2), pages 1735-1742. IEEE Computer Society, 2006.
[3] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, CoRR, 01171807733333 2018.
10

---

[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In JCML, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 2020.
[5] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momen- tum contrastive learning. CoRR, abs/2003.04297, 2020.
[6] Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9726-9735. Computer Vision Foundation IEBE, 2020.
[7] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV volume 12356 of Lecture Notes in Computer Science, pages 776-794. Springer, 2020.
[8] Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In ICML. volume 139 of Proceedings of Machine Learning Research, pages 12310-12320. PMLR, 2021.
[9] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In EMNLP (1), pages 6894 6910. Association for Computational Linguisties, 2021.
[10] Lingpeng Kong, Cyprien de Masson d' Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama. A mutual information maximization perspective of language representation learning. In ICLR. OpenReview.net, 2020.
[11] Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan He, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 13052-13065. PMLR, 2022.
[12] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen. Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurlPS, 2020.
[13] Ching- Rao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In NeurIPS, 2020.
[14] Joshua David Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In ICLR. OpenReview.net, 2021.
[15] Tsai-Shien Chen, Wei-Chih Hung, Hung-Yu Yu Tseng, Shao-Yi Chien, and Ming-Hsuan Yang. Incremental false negative detection for contrastive learning. In ICLR. OpenReview.net, 2022.
[16] Yuandong Tian. Understanding deep contrastive learning via coordinate-wis optimization. In NeurlPS, 2022.
[17] John C. Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. CoRR, abs/ mos//881a 110008750, 2018.
[18] Ruomin Huang, Jiawei Huang, Wenjie Liu, and Hu Ding. Coresets for wasserstein distribution- ally robust optimization problems. In NeurlPS, 2022.
[19] Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: performance guarantees and tractable reformulations. Math. Program., 171(1-2):115-111 2018.
[20] Hongseok Namkoong and John C. Duchi. Variance-hased regularization with convex objectives. In NIPS, pages 2971-2980, 2017.
[21] Soroosh Shafieezadeh-Abaddhh Peyman Mohajerin Esfahani, and Daniel Kuhn. Distributionally robust logistic regression. In NIPS, pages 1576-1584, 2015.
[22] Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifying some distributional robust- ness with principled adversarial training. In ICLR. OpenReview.net, 2018.
11

---

[23] Rohit Agrawal and Thibaut Horel. Optimal bounds between f-divergences and integral prob- ability metrics. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 115-124. PMLR, 2020.
[24] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio,
R. Devon Hjelm, and Aaron C. Courville. Mutual information neural estimation. In JCML, volume 80 of Proceedings of Machine Learning Research, pages 530-539. PMLR, 2018.
[25] Runtian Zhai, Chen Dan, J. Zico Kolter, and Pradeep Ravikumar. DORO: distributional and outlier robust optimization. ICML, volume 139 of Proceedings of Machine Learning Research, pages 12345 11355, PMLR, 2021.
[26] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach self-supervised learning. In NeurIPS, 2020.
[27] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurlPS, 2020.
[28] Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In CVPR, pages Computer Vision Foundation / IEEE, 2021.
[29] Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In ICLR. OpenReview.net, 2022.
[30] Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive learning benefits robustness against label noise. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 24851-2489 PMLR, 2022.
[31] Ching-Yun Yun Ko, Jeet Mohapatra, Sijia Liu, Pin- Yu Chen, Luca Daniel, and Lily Weng. Revisiting contrastive learning through the lens of neighborhood component analysis: integrated framework. In ICML, volume 162 of Proceedings of Machine Leurning Research, pages 11387-11412. PMLR, 2022.
[32] Binbin Jin, Defu Lian, Zheng Liu, Qi Liu, Jianhui Ma, Xing Xie, and Enhong Chen. Sampling- decomposable generative adversarial recommender. Advances in Neural Information Processing Systems, 33:22629-22633 2020.
[33] Defu Lian, Qi Liu, and Enhong Chen. Personalized ranking with importance sampling. In Proceedings of The Web Conference 2020, pages 1093-1103, 2020.
[34] Chenwang Wu, Defu Lian, Yong Ge, Zhihao Zhu, and Enhong Chen. pafluence-drived data poisoning for robust recommender systems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[35] Henry Lam, Zhenyuan Liu, and Xinyu Zhang. Orthounimodal distributionally robust optimiza- tion: Representation, computation and multivariate extreme event applications. arXiv preprint arXiv:2111.07894, 2021.
[36] Henry Lam and Clementine Mottet. Tail analysis without parametric models: A worst-case perspective. Oper. Res., 6566))1696-1711, 2017.
[37] Xi Chen, Simai He, Bo Jiang, Christopher Thomas Ryan, and Teng Zhang. The discrete moment problem with nonconvex shape constraints. Oper. Res., 69(1):279-296, 2021.
[38] Paul Michel, Tatsunori Hashimoto, and Graham Neubig. Modeling the second player in distributionally robust optimization. In ICLR. OpenReview.net, 2021.
[39] Paul Michel, Tatsunori Hashimoto, and Graham Neubig. Distributionally robust models with parametric likelihood ratios. In ICLR. OpenReview.net, 2022.
[40] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, Communications on Pure and Applied Mathematics, 1975.
12

---

[41] Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A. Alemi, and George Tucker. On variational bounds of mutual information. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 5171-5180. PMLR, 2019.
[42] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual informa- tion estimators. In ICLR. OpenReview.net, 2020.
[43]
Hubert Tsai, Martin Q. Ma, Muqiao Yang, Han Zhao, Louis- Phiis-Philiipe Morency, and Ruslan Salakhutdinov. Self- sepervseee representation learning with relative predictive coding. In ICLR. OpenReview.net, 2021.
[44] Qing Guo, Junya Chen, Dong Wang. Yuewei Yang. Xinwei Deng, Jing Huang, Lawrence Carin, Fan Li, and Chenyang Tao. Tight mutual information estimation with contrastive fenchel-legendre optimization. NeurlPS, 2022.
[45] Chun-Hsiao Yeh, Cheng- Cho Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In ECCV (26), volume 13686 of Lecture Notes in Computer Science, pages 668- Springer, 2022.
[46] Louis Faury, Ugo Tanielian, Elvis Dohmatob, Elena Smirnova, and Flavian Vasile. Distribu- tionally robust counterfactual risk minimization. In AAAI, pages 3850-3857. AAAI Press, 2020.
[47] Jun-ya Gotoh, Michael Jong Kim, and Andrew B. Lim. Robust empirical optimization is almost the same mean-variance optimization. Oper. Res. Lett., 46(4):448-452 2018.
[48] Jaeho Lee, Sejun Park, and Jinwoo Shin. Learning bounds for risk-sensitive learning. In NeurlPS, 2020.
[49] Sever S Dragomir, Gluscevic, and Charles Edward Miller Pearce. Csiszár f-divergence, ostrowski's inequality and mutual information. Nonlinear Analysis-Theory Methods and Applications, 27(4):2375-2386 2001.
[50] Kyungmin Lee and Jinwoo Shin. Rényiel: Contrastive representation learning with skew rényi divergence. In NeurlPS, 2022.
[51] Avraham Ruderman, Mark D. Reid, Dario Gareía-Gareía, and James Petterson. Tighter varia- tional representations of I-divergences via restriction to probability measures. In ICML. icml.cc Omnipress, 2012.
[52] Tiffany Tianhui Cai, Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Are all negatives created equal in contrastive instance discrimination? CoRR, abs/2010.06682, 2020.
[53] Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. CoRR, abs/2007.08667 2020.
[54] XuanLong Nguyen, Martin J. Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory, 56(11):5847-5861, 2010.
[55] Jean-Baptiste Hiriart-Urruty and Claude Lemaréchal. Fundamentals of convex analysis. Springer Science & Business Media, 2004.
CemmarrchallFuuddanntttss
[56) Aharon Ben-Tal and Mare Teboulle. An old-new concept of convex risk measures: the optimized certainty equivalent. Mathematical Finance, 17(3):49-4776, 2007.
[57] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[58] Mchryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. 2018.
[59] Aharon Ben- Tal and Mare Teboulle. Expected utility, penalty functions, and duality in stochastic nonlinear programming. Management Science, (2(11):1445-1466 1986.
13

---

[60] Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. In ICCV, pages 10286-10295. IEEE, 2021.
[61] Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. CoRR, fos/2007.086633 2020.
14

---

A Appendix of Proofs
A.1 Proof of Theorem 3.2
Theorem 3.2. By choosing KI. divergence DRL. (2 2da dx, optimizing CL-DRO (cf.
To Equation (3)) is equivalent to optimizing CL (InfoNCE,ff Equation (1)):
Q
C LCLDDR Brx (foo,,
min max fo v)] -aliaaaDx((lll000 - n) 3(Boo
1)}]
420,9
Qo
Sala.wtaa"a (m)
-Exx (a' (1) log Eq, (cfolz.w)aa"())
Constant
= =a'(n) Cmfunc Constant,
(4)
where 8 represent the Lagrange multipliers, and o² (1) signifies the optimal value of that minimizes the Equation (4), serving as the temperature T in CL.
Proof. In all the following appendix, for the sake of conserving space and clarity of expression, we have opted to simplify fo(2, y) as so and omit Expx
To complete the proof, we start with giving some important notations and theorem.
Definition A.1 (d-divergence [54]). For any convex funtion é with @(1) = O, the d-divergence between Q and Qo is:
(13) if Q is not absolutely continuous with respect to Qu- Specially, when
(a(1)) == Eg. (o(Q//QQQ))
where (o( 6(x) = r log x - x + 1, d-divergence degenerates to the well-known KL divergence.
Definition A.2 (Convex conjugate [55]). We consider pair (A, B) of topological vector spaces and a bilincar form :) + E such that (A, B. (.--)) +)) form a dual pair. For a convex function f + R, d = (i E T (x) 0o) is the effective domain of f. The convex conjugate, also known as the Legendre-Fenee Fennchel transform, of f - IR is the function f B -> IR defined as
(b) sup(ab f(a)), beB
(14)
Theorem A.3 (Interchange of minimization and integration [56]). Let (S,,F) be a measurable space equipped with a-algebra F, [9, P) be the linear space of measurable real valued functions f: with ||fllp < co, and let X == [F, normal integrand, and define on X. Then,
11 to] Let g R be a
min / g(x(())w))P(C (P(w) = 1 min g[ w) dP(w) ZEX 5
(15)
sE
To ease the derivation, we denote the likelihood raito L(x, y) v)) (x. y). Note that the o-divergence between Q and Qo constrained, and thus L(.) well-defined. For brevity, we usually short L(x, y) as L. And in terms of Definition A.1 of d-divergence, the expression of CL-DRO
becomes:
DRO DRO
E2, [S] + max Bo. [So]]
s.1. EQ. [o(L)] <
(16)
Note that Eq, [foL] and (Q. [o[]] are both convex in L. We use the Lagrangian function solver:
CD CLDRRO = 1E [fo] + min max [fo]] a @E 220,8 L
[o(L)] - nl + 3(EQ. [L] -
So B L {(1))})
-EEp, + min {ar 8 max 20,
a
(17)
S0 + L -- L-
-El,[fol + min {an - + aEg. 020,8
mmxx b
1
10-8113 }]}
fo
=EE,, [f] + min {an 8 aE g 220,
The first equality holds due to the strong duality [57]. The second equality a re-arrangement for
optmizing L. The third equation follows by the Theorem A.3. The last equality is established based
15

---

on the definition of convex conjugate A.2. When we choose KL-divergence, we have ΦKL(Z)
log + 1. be deduced that k ex Then, we have:
10 So + 3 }}}
CKT. CL-DRO = LEp + min {az) 3 + aB.. a20,8
Ep,lfol + min {an aBQu le 10t1 - -11 020,8
-EEpo[fol + min {ar log 1oo "1] a20
(18)
=-Ep,[fol + min {
an + a log E Eq,
L. 1u
a20
efvl""
= -EEp [a log Eqo [elo/a]] + am
a LInfONCE + Conslant
Here at arg min {an + a log le }}.
A.2 Proof of Theorem 3.3
Theorem 3.3. |Generalization Bound] Let LInfuNCE be estimation of InfoNCE with N negative samples. Given any finite hypothesis space of models, suppose fo E [M1, M2] and the ideal negative sampling distribution Qideal satisfies DEL(Q (ideeid (1Qo) S 17, we have that with probability at least 1 - p
Lunbiased S EinfNNCE B(p,N,T)
where B(p,N,))
My expi M.)T) N-1-exp((M; (M
2 In(
21
Here we simply disregard the constant term present in Equation (4) as it does not impact optimization, and omit the error from the positive instances.
Proof. Before detailing the proof process, we first introduce pertinent theorem:
Theorem A.4 (Union Bound). For any two sets A, B and a distribution we have
D(AUB) < D(A) + D(B)
(19)
Theorem A.5 (McDiarmid's inequality [58]). Let X1, .n, be independent random variables, where X; has range X. Let X1 x ... Xn - R be any function with the (cI,..., Cn)- bounded difference property: for every i = 1,..., n. and every (Z1..... In) (2'1..., x,,) X1 Xn that differ only in the i-th coordinate (x, =z'z for all F i). we have If (x1,..., 2- (s' S cz. For any e > 0,
-2c
P( ,.), - EE X.)I 26) exp(
(20)
Σm
Now we delve into the proof. As Qideat satisfies DKI(Q[1QU) 17, we can bound Lunbiased with:
Lunbiased = Epp, [fo] + Equaeat [fe]
< -Ep [So] + max
Eg [Sa]
(21)
(2)
KK
where Qideal Q" denotes the ideal negative distribution and the worst-case distribution in CL-DDR. From the Theorem 3.2, have the equivalence between InfoNCE and CL-DRO. Thus here
16

---

choose CL-DRO for analyses. Suppose we have N negative samples, and for any pair of samples (or, (e), (Iz. Mj), we have the following bound:
exp (=)
IQ" (zx, We), vc) - Q (25,
sup (I v)) - exp (m.y)~Qo
(22)
where the first inequality holds as Q" (o( > O. The second inequality holds based on explfe/r]
the expression of Q = Qu Eagg exp (refer to Appendix A.6). Suppose ʃo E [M1, M2]. the
upper bound of sup(z.))~Qo IQ" (z. f) y) arrives if fo (2,, y) M2 for the sample (x, y) and
fo(2,y) y) M1 for others. We have sup (a.(z.))>Q (x) y)1 S N-1+ Ma)/-)
My M11)))))))1))/
By using McDiarmid's inequality in Theorem f.o, any E. we have:
PI - -[ 2 el S exp (-22
-22
N NN (xp((M))MM))7)) Mi)/T)
(23)
N
M2exp((M2 - Mi)/T)
Equation (23) holds for the unique model f. For any finite hypothesis space of models F f2. .. fir we combine this with Theorem A.4 and have following generalization error bound of the learned model f:
2e²
N - 1+ exp((M2 (Xi((()))))))))
Mi)/T)
IP [P[CCLDDRO -T TLinfoNCE) 2 o) S Σ
exp
N
M2exp((M2 - -M1//)
feF
(24)
-2e2
(N IN-1+ (x-M+ - 2
=
FF
exp
N
expexp((M)) 110xxp((M22MMMaaa001 MI)/T)
Let
26
N exp((M2 exp((M22-M)//)))) )3
M2 expl (MM (1171
p
[F]
exp
Mi)/T)
(25)
we get:
M1 exp ((M2 - Mi)/T)
N 2|1|
In(
(26)
N 1 + exp ((M2 M1)/T)
2
Thus, for Vp E (0, 1), we conclude that with probability at least 1
M2 exp ((M2 M1)/Ti
N
2|F
Lunhiased -.K.
CL. TLI
In(
(27)
N N exp ((M2 M2)/T)
2
A.3 Proof of Corollary 3.4
Corollary 3.4. [The optimal Lemma Faury al. [46]] The value of the optimal (i.e., 7) can be approximated as follow:
Voollfelz, 2)1/2m,
(6)
where V VQ (f v)] denotes the variance of (x( y) under the distribution Qo-
Proof. While Corollary 3.4 has already been proven in [46], we present a brief outline the proof here for the sake of completeness and ensure that our article is self-contained. To verify the relationship between and 77, we could utilize the approximate expression of InfoNCE (cf. Equation (31)) and focus the first order conditions for r. In detail, have:
1
+ inf (Eoo [So] a20
Velloo + on)
2a ((2)
To find the optimal value of a (or equivalently, T), we differentiate the above equation and set it to 0. This yields fixed-point equation
V No [So]
2m
The corollary gets proved.
17

---

A.4 Proof of Theorem 3.5
Theorem 3.5. Given any (-divergence, the corresponding CL-DRO objective could be approximated as mean-variance objective:
1 (aolfo(z.vil) (7)
DROD ~ -Exx [Ep, (Boooloo (1) 2r (12)
where (1) denotes the the second derivative value of Φ(-) at point 1, and V Ve Ifol denotes the variance of I under the distribution Qo-
Specially, if we consider KI. divergence, the approximation transforms:
1 CL-DRO -Epx EP, [o(x, (EEoo w)]
(8)
2r
Proof. We start with introducing useful lemma.
Lemma A.6 (Lemma A.2 of [47]). Suppose that o: RR R RU{+o) a closed, convex function
such that ((z) 2 Φ(1) 0 for all z, is two times continuously differentiable around 1, and ((1) 0, Then
(C) max 25 -((2}
1 1 1 21 o0(1) 152+ o({2)
(28)
=
Note that most of the ø-divergences [54] (e.g.. KL divergence, Cressie-Read divergence, Burg entropy, J-divergence. x2 -diistance, modified x -distancce and Hellinger distance) satisfy the smoothness
conditions. When n = 2, [<] R Ç + 2 15, Substituting this back to Equation (17) we have:
fo +
CR- = Ep [fol + min {an 3 + aBQ.
10
}}}
20,3
fo Ep, + min {an + aBQu 220,8
+ 8
1 1
+ )
a
2
2
a
(29)
1
-Ep + min {an) 3 E 20,3
[o
+ B
+
(So + 3)3}
2 20)2 (1)a
1
= Ep. + min {an Eq. lffe + 220,8
(fe + })
2
(1))
we differentiate it w.r.f. 3:
afan Bq. .o + 2 (1)a (So 3) 0
(30)
03
have 8r BQq [fo], and the objective transforms into:
1
[fo inf {an + Equ [ff 2 2012) (fo 3)31}
a20.8
1
Er,[fo] + inf
[E [fo + 2a (1) (So (oo( +an
(31)
a20
EP inf
[EQ, [fo] + 2a (1) Woolfol + an?
a20
Choosing KL-divergence, we have (1) 1. Substituting (7) into Equation (31) and ignoring the constant azy:
- Ep. + Ee. + 27 [fol
Then Theorem gets proved.
18

---

A.5 Proof of Theorem 4.2
Theorem 4.2. For distributions P, Q such that & Q let be a set of bounded measurable functions. Let CL-DRO draw positive and negative instances from and Q, marked as E CLD (P, Q). Then the CL-DRO objective is the tight variational estimation of d-vivergence. In fact, have:
(*(P)1Q)= = max - -o CCLLDR((..O) = max BeaxEpf]]e -in (f-) JEF
(10)
fEF
XER
Here, the choice of & CL-DRO corresponds to the probability measures (o(PI1Q). And o denotes the corvex conjugate.
Proof. Regarding this theorem, our proof primarily relies on the variational representation of ø- divergence and optimized certainty equivalent (OCE) risk. Towards this end, we start to introduce the basic concepts:
Definition A.7 (OCE [56]). Let X be a random variable and let be a convex, swwrr--emiconinuuus function satisfies u(0) 0, u² (1) 0, then optimized certainty equivalent (OCE) risk p(X) is defines as:
p(X)
inf {A B[u(J A)]}
(32)
XER
OCE a type of risk measure that is widely used by both practitioners and academics [59; 56]. With duality theory, its various properties have been inspiring in our study of DRO.
Definition A.8 (Variational formulation).
(P(1))))= = sup [Er[f] (o)o"(()))
(33)
fer
where the supremum is taken over all bounded real-valued measurable functions F defined on X.
Note that in order to keep consistent with the definition of CL-DRO, transform Equation (10) to
CCLLDR Q)
max
Ep Epls] min(-- - Eolø" B]]}
(34)
JEF
feF
BER
Our proof for this theorem primarily relies utilizing OCE risk bridge and can be divided into two distinct steps:
Step 1: maxfer - L. (P. Q) max Ep[f] mingeR(-2 + (f
Step 2: (&())) max [p[[] minger (-3 + " + ()]}
1. We show that maxjer - CL-DOO (P, Q) maxfex Ep [f] minger( -3 Eol B]]}.
Le DRO
[p[s]]
min max (F (F - n) 3(B [T]] 1)} a20,8
+ L o(L))}
[p - min 220
{an 3 + aB mmax
+ }} a + B |] 10
Er[s]
min
{on - 3 aB
(35)
a20,
Ep[f] min {a - 8 + a B
a f+b = =Ep - min { - Eolo"
Constant)
a
When a" 1, step gets proved.
19

---

= [fo, + min 2. We show that D*(P - ja macpeemmmmaaaa--- B2l0 (f + 3)1}.
Firstly, we transform Ep[s] to 18 [
dP as:
aqp
max R [s] - min { - R (
fEF
(36)
dP
max
min
{ - 3 + Eolø" (f +1]}
JeJ
dQ
3
Let f + 3 Y, we have:
dP
max
EQls
dQ - min
{ - 3 + Egl0 8]]}
EEF
8
dP
max
min Eq 10 - B) dQ
{ 8 + Eglo"
YEF
(37)
dP
dP
max
YEF
Eoly
dQ
6² (Y)] + min M2Q [1
dQ
dP
max
lY
++YY)++
YEF
dQ
The first equality follows from replacing f f 01 with Y. The second equality is re-arrangement for optmizing B. The third equation holds Eq [1 de O.
Applying Theorem A.3, the last supremum reduces to:
dP dQ
max EQ
Y
(V)
YEF
dP
[up sup (Y
dQ (Y))
YEF
'Q "de"
dP
(38)
=Eolo*
dP
EQ (ob( aQ )
(P(())
where the last equality follows from the fact that x===="""
=
This concludes the proof.
A.6
Proof of Q"
Proof. From theorm3.2, CL-DRO be rewrriten as:
fo + L ((L))}}
c LLD -Ep, [Sa] min {a" 1 + ax Egg [max
3
a
(39)
fo +
{a"n + a a 10
}1}
3
a*
For the inner optimzation, we can draw the optimal L for max
f0+8 ({L)} as:
(40)
For the outer optimization, we can draw the optimal 3 for ming {a"n - B + a a'LEol
11} as
a" log Eq,e f
(41)
Then we plug Equation (41) into Equation (40).
(42)
Eqg, le fe
20

---

Based on the definition of L, we can derive the expression for Q":
CaT
Qu
(43)
le 10
EQu
B Experiments
Figure shows PyTorch-style pseudocode for the standard objective, the adjusted InfoNCE objective. The proposed adjusted reweighting loss is very simple to implement, requiring only two extra lines of code compared to the standard objective.
pos neg
exp of inner products for positive examples
exp of inner products for negative examples
number of negative examples
temperature scaling center position
sigma
height scale
#InfoNCE
standard_loss -log(pos sum () / (pos sum () neg. ()))
10
11
#ADNCE
12
seight-1/(sigm. eqrt (2+pi)) exp -0.5 ((neg mu) (aigma)
13
/wight mean()
Adjusted_loss loss -log (pos. sum () (pos. sum (neg weight detach sum
Figure 5: Pseudocode for our proposed adjusted InfoNCE objective, as well as the original NCE contrastive objective. The implementation of our adjusted reweighting method only requires two additional lines of code compared to the standard objective.
B.1 Visual Representation
Model. For contrastive learning on images, we SimCLR [4] as our baseline and follow the same experimental setup [13]. Specifically, we use the ResNet-50 network as the backbone. To ensure fair comparison, we set the embedded dimension to 2048 (the representation used in linear readout) and project it into a 128 128-dimensional space (the actual embedding used for contrastive learning). Regarding the temperature parameter we use the default value To of 0.5 in most researches, and we also perform grid search on varying from 0.1 1.0 an interval of 0.1, denoted by r*. The best parameters for each dataset is reported in Table 6. Note that {-} indicates the range of hyperparameters that we tune and numbers bold are the final settings. For a-CL, we follow the setting [16], where and = 0.5. We use the Adam optimizer with learning rate of 0.001 and weight decay of 6. models are trained for 400 epochs.
Table 6: hyperparameters setting on each datasets.
DATASETS
CI
STL
CIFAR 100
B
0.0.1 0.2, 0.3, 0.4, 0.6.
00.1 0.2, 0.3, 0.4, 0.6. [ (0.55 0.6, 0.7. 0.8, 0.9
0.11, 0.2, 0.3, 0.4, 0.5,
O.5., 0.6, 0.7, 0.8, 0.9
0.5, 0.6, 0.7, 0.8, 0.9
(0.5, 1.0
[0.5. 1.0
(0.5, 1.0)
Noisy experiments in Section 3.4. To investigate the relationship between the temperature parameter (or 77) and the noise ratio, we follow the approach outlined in [13] and utilize the class information of each image to select negative samples as a combination of true negative samples and false negative samples. Specifically, Tratio = 0 indicates all negative samples are true negative samples, Tratio = 0.5 suggests 50% of true positive samples existing in negative samples, Tratio means uniform sampling.
21

---

Variance analysis in Section 3.4. To verify the mean-variame variance objective of InfoNCE, we adopt the approach outlined in [60] and record the negative prediction scores for 256 samples (assuming batch size of 256) in each minibatch. Specifically, we randomly select samples from batch to calculate the statistics and visualize them. (1) For positive samples, we calculate cosine similarity by taking the inner product after normalization, and retain the mean value of the 256 positive scores as "pss mean'. : (2) For negative samples, we average the means and variances of 256 negative samples to show the statistical characteristics of these N negative samples '(mean neg; var neg)'. We record this data at each training step to track score distribution throughout the training process.
B.2 Sentence Representation
For the sentence contrastive learning. we adopt the approach outlined in [9] and evaluate our method on popular STS datasets: STS tasks from 2012-2016, STS-B and SICK-R. We utilize the SentEval toolkit to obtain all 7 datasets. Each dataset includes sentence pairs which are rated on scale of 0 to 5, indicating the degree of semantic similarity. To validate the effective of our proposed method, flow, BERT-whitening, CT-BERT and SimCSE. The best parameters each dataset reported in Table 7. To ensure fairesss, employed the official code, which can be accessed at https: / //github.. com/ http//rrnnntoooo seriinnetoonnnlpp nlp/ SimCSE.
utilize several methods as baselines: average GloVe embeddings, BERT-
Table 7: hyperparameters setting on sentence CL. Note that {-} indicates the range of hyperparameters that we tune and the numbers in bold are the final settings.
DATASITS
SUNCSE-BERT..H 0.05,
SIMCSE-ROBERT.N
0.01,
0.15,
0.02,0.03.0.04 2.5.
0.9.
[0.5,
[0.3,
B.3
Graph Representation
For the graph contrastive learning experiments on TU-Dataset [61], we adopted the same experimental setup as outlined in [12]. The dataset statistics can be found Table To ensure fairness, we employed the official code, which can be accessed at https: //gith/ttttt com/ com/Shen- -Shen-La// /alab//rrahCL/// trre/mastes wwwpaasteeruuuuuuriieeee /ww.upueeiiieddd.
We made only modifications to the script by incorporating our ADNCE method and conducting experiments the dyper-paramette
{0.5, 0.6, 0.7, 0.8, 0.9, 1.0) and σ = 1 on most datasets. Each parameter was repeated from scratch five times, and the best parameter selected by evaluating the validation dataset. The best parameters for each dataset reported in Table 9.
We summarize the statistics of TU-datasets [61] for unsupervised learning in Table 8. Table 10 demonstrates the consistent superiority of our proposed ADNCE approach.
Table 8: Statistics for unsupervised learning TU TU-datasets.
DATASETS
CATEGORY
GRAPHS#
AVG. N#
AvG. DEGREE
NCTI
BIOCHEMICAL MOLECULES
4,110
29.87
1.08
PROTEINS
BIOCHEMICAL MOLECULES
1,113
39.06
1,86
DD
BIOCHEMICAL MOLECULES
1.178
284.32
715.66
MUTAG
BIOCHEMICAL MOLECULES
188
17.93
19.79
COLLAB
SOCIAL NETWORKS
5,000
74.49
32.99
RDT-B
SOCIAL NETWORKS
2,000
429.63
1.15
RDT-M
SOCIAL NETWORKS
2,000
429.63
497.75
IMDB-B
SOCIAL NETWORKS
1,000
19.77
96.53
22

---

Table 9: hyperparameters setting graph CL. Note that {-} indicates the range of hyperparameters that we tune and the numbers in bold are the final settings.
DATASETS
BEST T
NCT1
0.05, 0.10, 0.15, 0.20, 0.25)
0.2, 0.3 0.4,
0.5,
0.6,
0.7,
0.8, 0.9 }
0.5, 1.0
PROTEINS
0.05, 0.10, 0.150 0.20, 0.25)
0.5.
1.0.
2.0
0.5. 1.0
DD
0.005, 0.100..11.. 0.20, 0.25]
0.2. 0.3
0.5,
0.6. 0.7.
0.9
0.5. 1.0
MUTAG
0..0,. 0.10. 0.15, 0.20, 0.25|
0.2, 0,.00.444
0.6.
0.7. 0.8.0.8.00
0.5.
COLLAB
0.005, 0.10, 0.15, 0.20, 0.25)
0.2, 0.3
0.5,
0.6, 0.7, 0.8, 0.9
0.5, 1.0
RDT-B
0.005, 0. 0.15, 0.20, 0.25)
0.2,
0.5,
0.6, 0.7,
0.8 0.9
0.5, 1.0
RDT-M
0.055, 10,0, 0.15, 0.20, 0.25)
0.2, 0.3 ,.44 0.5,
0.6, 0.7,
0.8, 0.9
0.5, 1.0
IMDB-B
0.100, 0.20, 0.30, 0.40, 0.50)
0.2, 0.3 ,.44 0.5, 0.6, 0.7, 0.8, 0.9
0.5, 1.0
Table 10: Unsupervised representation learning classification accuracy (%) on TU datasets. compared numbers from except AD-GCL, whose statisties reproduced indicates the best performance while underline indicates the second best on each dataset.
The platform. Bold
PROTEINS
No PRE-TRAIN
65 17
72.7320.51
75.67:9.29
87.39=100
05.29:0.16
76.36 :0.25
48.4820.25
69.37::0.37
70.15
INPOGRAPH
20.20.111
0.
72.654
19.01-1111
70.05
62:50_114
53.46:1:05
73.032.0.7
74.02
GRAPBCL
27.87+0041
34.3940.45
78.6240.40
86.30=1.34
71.3661.15
89.53-0.84
55,99920.28
71.14+0.44
75.71
75.79:0.77
78.86-+8448
87.66-1.01
90.34-0.58
56.38--0.40
ADNCE
79.3018.07
75.10.19.15
79.2318.59
89.04
72.2611.10
01.39__..31
55.01100115
71.38
76.74
B.4 Additional ablation study
The primary motivation behind ADNCE to mitigate the issues of over-conservatism and sensitivity outliers. These limitations stem from the unreasonable worst-case distribution, which assigns excessive weights to the hardnest negative samples. Consequently, any weighting strategy capable of modulating the worst-case distribution to foeus
the informative region holds promising. In our ADNCE, we opted for Gaussian-like weights due to its flexibility and unimodal nature. However, alternative weighting strategies such Gamma, Rayleigh or Chi-squared could also be employed. The following experiment demonstrates that these alternative weighting strategies can yield comparable results Gaussian-like weights. The above table showcases the comparison of the
Table 11: Alternative weighting strategies for ADNCE.
WEIGHT STRATEGY
PROBABILITY DENSITY FUNCTION
CIFARIO
GAMMA-LIKE
w(x m, n) F(mjam mn
e
91.74
RAYLEIGH-LIKE CHI-SQUARED-LICS
m)) c
2m
91.73
w(z, 2m/2T((m/2) x n0 22/2211
91.99
GAUSSIAN- LIKE TARKEAANNE (ADNCE)
(==m² m, n//
91.88
TOP-1 accuracy performance CIFAR10 dataset under different weight strategies. The parameters m and n are utilized to denote the parameters within their corresponding probability density function, while the variable represents random variable. It is important to note that, due the domain definition of some PDFs being (0, +oo), we need to set x = prediction_score +
23

---

Table 12: Sensitivity analysis of μ.
0.1
0.3
0.5
0.7
0.9
CIFARIO
91.3
91.66
91.9
91.77
92.25
STLIO
87.84
88.22
87.56
88.48
88.45
CIFART00
69.34
69.31
68.70
69.24
68.95
Table 13: Sensitivity analysis a.
0.2
0.4
0.6
0.8
1.5
2
CIFARIO
90.07
91.85
92.02
91.77
91.72
91.69
91.94
STLI0
86.54
88.30
88.10
87.54
88.95
88.12
88.40
CIFARI00
67.38
69.36
69.52
69.01
68.70
69.24
69.42
B.5 Sensitivity analysis of and
The above tables showcase the comparisons of the TOP-1 accuracy performance on three dataset under different μ and a. As can be seen, changing the parameters μr and would impact the model performance, but not as dramatical as tuning the parameter т. (For example, on STL10 a from 1.0 0.2 just brings 2.7% performance drops, while changing T brings 7.6% performance gap.) This outcome indicates that tuning and a is not a significant burden. In most scenarios, it may suffice to set requiring only the tuning of μ within the range of 0.1 0.9 (can refer to Table 6, 7, 9 in Appendix B for more details).
24


Reviews:
[{"review_id": "BLSBg9f4bH", "reviewer": "Reviewer_CyLz", "summary": "In this work the authors demonstrate a connection between contrastive learning, in particular InfoNCE, and distributionally robust optimization. In contrastive learning algorithms it is typical during training that samples which are similar are treated as being different, ie a negative pair, since contrastive learning typically does not include class labels. Intuitively this would be a hinder the ability of contrastive learning algorithms to learn useful representations. Interestingly this doesn't seem to happen in practice, choosing a temperature parameter correctly can cause standard CL algorithms to perform as good as, or better than, robust variants, in the presence of false negative pairs.\n\nIn this work it is theoretically proven that InfoNCE is naturally distributionally robust, which explains the phenomenon from the last paragraph. This is done by proposing a distributionally robust version of InfoNCE and demonstrating that the loss is equivalent to InfoNCE with scaling and shifting. This work goes on to analyze the effect and selection of the temperature parameter, giving further insight into the behavior of InfoNCE. It is demonstrated that the distributionally robust InfoNCE is an estiamtor of mutual information. This work then proposes ADNCE as a way to overcome shortcoings of InfoNCE, in particular its sensitivity to outliers, i.e. the hardest negative pairs. This is done by weighting samples in the loss so that the most outlying negative pairs have lower weight in the ending risk. It is experimentally demonstrated that this method brings improvements.", "questions": "I don't have any questions really, the paper just needs more polish", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "- The theoretical result of this work is useful, reasonably nontrivial, and regards a topic that is of very large interest to the ML community. Very good!\n- ADNCE is an easily algorithm for a problem of signifiant interest, that brings consistent improvements. Also very good!", "weaknesses": "I spent a fair amount of time with this paper and I think its contributions are of high significance and importance, however I really feel that the exposition needs some improvement. I spent some time with the proofs and, while I was always able to decipher what was happening and found no errors, but they are written in a way that I found very hard to parse. While its far from the worst writing I have seen, there is quite a lot of the main text that was also difficult to precisely understand. I don't think the topicality here is so mathematically dense or advanced that this couldn't be written in a much clearer way. If this paper were very well written I would give this an 8, I think. Some example issues are listed below:\n\n- Line 63: What is \"the ultimate distribution\"?\n- Line 101 & Proof of Theorem 3.2 : it would be much clearer if $f_\\theta$ was instead $f_\\theta(x,y)$. This wouldn't take more space and it would be much easier to parse whats going in in the Proof of Theorem 3.2 if this were the case. \n- Theorem 3.2: It should be made clear that $\\alpha^*$ is a function of $\\eta$, this seems quite important. One could simply write $\\alpha^*(\\eta)$.\n- appx. line 10: I think $P$ in this line is supposed to be $Q$\n- appx. (17):  I think its worth writing the original constrained optimization before applying strong duality.\n- appx. (17): $\\max_L$ is again a bit vague. I'm assuming whats actually happening is that we are optimizing over $Q$ in the $L$ definition.\n- appx. line 21: \"fine definition\" is nonstandard, something like \"always exists\" or \"is well-defined\"\n- appx. line 29: this should be an $\\arg \\min$. Text like \"$\\alpha^*$ represents the optimal value of ...\" is needlessly confusing, just say \"$\\alpha^* = \\arg \\min \\cdots$.\"\n- Line 148, 165: Maybe I'm missing something, but I don't quite get what \"optimal\" means in these lines.\n- Theorem 3.3: What is $Q^{ideal}$? In what sense is it \"ideal\"? \n\nIf this work used more clear and precise notation I think it would be great, right now it looks like a rough draft. As an example improvement, I would rewrite in appx (17) the fist term in the last line \n$\\mathbb{E}_{P_0} [f_\\theta]$ \n\nas\n\n$\\mathbb{E}_{(X,Y)\\sim P_0}[ f_\\theta(X,Y)]$\n\nSome other errors:\n- \"and graph\" should be \"and graphs\"\n- Give the definitions of acronyms, eg DCL HCL RPL\n- There are many missing periods for sentences that end with equations, eg at the end of (1), (16),(17)\n- Line 70: \"downstrea\" [sic]\n- Page 3 footnote: sloppy ]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer CyLz:\nDear Reviewer,\n\nWe sincerely appreciate your recognition of our work and deeply regret any confusion caused by typographical errors or unclear notations within our paper. Your detailed comments are highly valued. In the revised version, we commit to meticulously refining the paper in accordance with your feedback, rectifying any typographical errors or unsuitable notations, and providing precise explanations for each definition. Moreover, we will incorporate a comprehensive notation table to augment the clarity of our work. In the following, we provide responses to the questions you have raised:\n\n**Q1: What is \"the ultimate distribution\" in line 63?**\n\nA1: Here the ultimate distribution refers to the worst-case distribution $Q^*$ that the model is optimized on, formally defined as $Q^*=\\operatorname*{argmax}\\_{Q}  \\mathbb{E}\\_{Q}[f\\_\\theta]  \\qquad s.t.  D\\_{\\phi}({Q}||{Q}\\_0) \\leq \\eta $. From the Appendix A.6, $Q^*$ in InfoNCE can be written as $Q^* = Q\\_0  \\frac{\\exp[f\\_\\theta / \\tau]}{E\\_{Q\\_0} \\exp[{f\\_\\theta / \\tau}]}$. Our ADNCE reshape the $Q^*$ by incorporating Gaussian-like weights. In the forthcoming revision, we will replace the term \"ultimate distribution\" with \"worst-case distribution\" to enhance clarity.\n\n\n**Q2: What \"optimal\" means in lines 148, 165?**\n\nA2: In this context, the optimal $\\alpha^*$ signifies the optimal value of $\\alpha$ that minimizes the initial formula in equation (4). Specifically, $\\alpha^*=\\operatorname*{argmin}\\_{\\alpha\\geq 0} \\operatorname*{min}\\_{\\eta_1} \\operatorname*{max}\\_{Q\\in \\mathbb{Q}} \\{ \\mathbb{E}\\_{Q}[f_\\theta] -  \\alpha [D\\_{KL}(Q||Q\\_0) -\\eta] + \\eta_1 (\\mathbb{E}\\_{Q_0} [\\frac{Q}{Q\\_0}]  - 1) \\}$. By comparing equation (4) with InfoNCE, we deduce that the optimal $\\alpha^*$ functions as the temperature $\\tau$. Additonal, we employ the optimal $\\alpha^*$ for analyses in Corollary 3.4 to gain insight into the nature of the temperature. We appreciate your insightful query and  will give more clear explanations of the optimal $\\alpha^*$ in the next version. \n\n\n**Q3: What is $Q^{ideal}$ in theorem 3.3?**\n\nA3: The term $Q^{ideal}$ denotes the ideal negative sampling distribution that selects instances with distinctly dissimilar semantics. It should be noted that in practical constrastive learning, the ideal $Q^{ideal}$ is not avaliable. As a substitute, the uniform distribution $Q_0$ is utilized, which may sample instances with similar semantics, thereby introducing a so-called sampling bias. Theorem 3.3 demonstrates that InfoNCE is resilient to this sampling bias, establishes theoretical bounded relations between the model directly trained on the ideal $Q^{ideal}$ and the model trained on $Q\\_0$ using InfoNCE."}, {"review_id": "V56hKIuyU9", "reviewer": "Reviewer_sD47", "summary": "The paper proposes a novel theoretical framework for understanding contrastive learning (CL) via distributionally robust optimization (DRO). \nUnder this framework, the paper derives that the InfoNCE loss can be interpreted as DRO with KL ball around the negative sampling distribution, and the paper leverages DRO insights to derive the optimal temperature parameter. Experiments are conducted which show that a novel modification of InfoNCE can lead to better sample complexities.", "questions": "1. In Corollary 3.4, how is optimality measured? In other words, what objective does the setting of Eqn 6 optimize, and which assumptions are needed for it to hold?\n2. Besides the math, is there any intuition on why DRO in the negative samples is what CL should strive for? In other words, why do we want to maximize the distance for all possible distributions of negative samples around the truth? (rather than the one we see data from). I think illustrating this intuition would be important to making this paper stronger.\n2b. Does this DRO insight applies to other types of CL losses, besides InfoNCE? \n3. What is Q^ideal? It is undefined before Line 154.\n4. Can you please shed more light on what is hard-mining and how DRO gives us insights that hasn't been uncovered before? The paper seems to assume that readers know what hard-mining is, but I am not aware of this phenomenon for CL.\n5. Can ADNCE also be explained with some modification of DRO? It would be nice to see if the new method is theoretically motivated as well, or simply a heuristic to avoid conservatism of DRO.\n", "limitations": "See weaknesses/questions.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The framework that the paper proposes is quite elegant mathematically and explains the InfoNCE loss from a rigorous mathematical perspective, which endows us with many insights. \n2. Using these insights, the paper proposes a new ADNCE loss that overcomes the issues of conservatism in DRO and shows some possibly promising experiments. (I am not an expert in CL so I cannot gauge how convincing these empirical improvements are).", "weaknesses": "1. The proposed approach introduces more hyperparameters, which define the weighting distribution. These hyperparameters can be a hassle to tune in practice, and it is unclear if the reported gains are simply from better hyperparameter tuning with these newly introduced hyperparameters.. It would be informative to show whether the new ADNCE is better for a large number of hyperparameters, or only a few.\n\nPlease also answer my questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer sD47:\n\nDear Reviewer,\n\nWe greatly appreciate your acknowledgement of our contributions and your insightful comments. In what follows, we provide responses to the questions you have raised:\n\n\n\n**Q1: Concerning on the burden of hyperparameter tuning in ADNCE**\n\nA1: Considering the two limitations of InfoNCE, namely its over-conservatism and sensitivity to outliers, we introduce ADNCE that employs Gaussian-like weights with only two additional hyperparameters ($\\mu$ and $\\sigma$). In most scenarios, it may suffice to set $\\sigma=1$, requiring only the tuning of $\\mu$ within the range of 0.1 to 0.9 (can refer to Appendix B.1 and Table 6 in Appendix for more details). To further explore the model's sensitivity to the hyperparameter $\\mu$, we have conducted additional experiments and got the following results:\n\n\n|    $\\mu$     | 0.1  | 0.3  | 0.5  | 0.7  | 0.9  |\n|----------|------|------|------|------|------|\n| CIFAR10  | 91.3 | 91.66| 91.9 |91.77 |92.25 |\n| STL10   |87.84 |88.22 |87.56 |88.48 |88.45 |\n| CIFAR100 |69.34 |69.31 |68.70  |69.24 |68.95 |\n\n\nThe table above presents a comparison of TOP-1 accuracy performance across three datasets at different values of $\\mu$.\nAs observed, the performance of our model does not exhibit high sensitivity to $\\mu$. Even when $\\mu$ is set to a less optimal value (for instance, $\\mu=0.1$ in CIFAR10), our ADNCE still outperforms the original InfoNCE. This outcome underscores the efficacy of our ADNCE, and indicates that tuning $\\mu$ and $\\sigma$ is not a significant burden.\n\n\n\n**Q2: In Corollary 3.4, how is optimality measured?**\n\nA2: In this context, $\\alpha^*$ denotes the optimal value of $\\alpha$ that minimizes the initial formula as presented in equation (4). Specifically, $\\alpha^*=\\operatorname*{argmin}\\_{\\alpha\\geq 0} \\operatorname*{min}\\_{\\eta\\_1} \\operatorname*{max}\\_{Q\\in \\mathbb{Q}} \\{ \\mathbb{E}\\_{Q}[f\\_\\theta] -  \\alpha [D\\_{KL}(Q||Q\\_0) -\\eta] + \\eta\\_1 (\\mathbb{E}\\_{Q\\_0} [\\frac{Q}{Q\\_0}]  - 1) \\}$. Corollary 3.4 aims to analyze the relationships between $\\alpha^*$ and $\\eta$. Here, we simply apply a second-order Taylor expansion approximation without imposing any additional assumptions, which renders Corollary 3.4 both general and applicable across various domains. Appendix A.3 provides more comprehensive details.\n\n\n\n**Q3: Is there any intuition on why DRO in the negative samples is what CL should strive for?**\n\nA3: In constrastive learning, a uniform distribution is often employed to sample negative instances. However, this distribution is not optimal and may select instances with similar semantics, thereby leading to a potential issue of sampling bias. DRO empowers CL with a resilience to such sampling bias. \n\nDRO can be intuitively understood as a specific adversarial method: it introduces adversarial perturbations to the negative distribution, and the model is subsequently optimized to resist these perturbations. Through this mechanism, DRO entables CL to perform well accross various potential distributions, thereby endowing it with robustness against sampling bias.\n\n\n**Q4: What is Q^ideal?**\n\nA4: Thanks for underlining this problem. We will provide a comprehensive explanation of Q^ideal in the next revision. The term $Q^{ideal}$ denotes the ideal negative sampling distribution that selects instances with distinctly dissimilar semantics. It should be noted that in practical constrastive learning, the ideal $Q^{ideal}$ is not avaliable. As a substitute, the uniform distribution $Q\\_0$ is utilized, which may sample instances with similar semantics, thereby introducing a so-called sampling bias. Theorem 3.3 is aimed at establishing theoretical bounded relationships between the model directly trained on the ideal $Q^{ideal}$ and the model trained on $Q\\_0$ using InfoNCE.\n\n**Q5: What is hard-mining and how DRO gives us insights that hasn't been uncovered before?**\n\nA5: Hard-mining refers to a phenomenon that CL adaptively puts more weights on hard negative samples, which have higher prediction score and are challenging to distinguish. While this phenomenon has also been illuminated in recent studies either by examining the gradient magnitude [43] or through coordinate-wise optimization [45], our work presents the hard-mining property from a novel perspective and explicitly provides an expression of the weight. \n\nIn addition to hard-mining, DRO unveils some attributes that have not been disclosed by recent work, including its robustness to sampling bias and the pivotal role of the temperature. Furthermore, the DRO perspective enhances our understanding of the limitations of InfoNCE, thereby inspiring us to develop a superior method ADNCE.\n\n\n**Q6: Can ADNCE be considered as a theoretical modification of DRO?**\n\nA6:  We are thankful for your insightful query. ADNCE is a heuristic approach designed to directly address the limitations of InfoNCE. While ADNCE may lack a formal theoretical foundation, this straightforward strategy indeed demonstrates superior performance across a variety of tasks. It would indeed be fascinating to explore the theoretical foundation of ADNCE or to devise a new method that incorporates a theoretical modification of DRO. We aspire to undertake such an endeavor in our future work. "}, {"review_id": "FwvjcETkeV", "reviewer": "Reviewer_rLbF", "summary": "This paper starts from the question why the naive form of CL is robust to sampling bias issue, resulting in empirical success in various areas. T this end, the authors first present the relationship between CL and DRO theoretically, where the DRO-constrained CL objective is conceptually equivalent to the objective of CL itself. They further show that the temperature in CL acts as a (inverse of ) robust radius in DRO constraint.  Based on these findings, the authors finally propose ADNCE, where the importance weight of negative samples follow Gaussian distribution. The proposed methods are validated via experiments on various domains.", "questions": "- Why controlling the variance of negative samples contributes to the success of CL?\n", "limitations": "Limitation has been addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper reveals the relationship between CL and DRO in a more comprehensive way to address the bias-tolerant behavior of CL.\n- Theoretical findings regarding \\tau are validated empirically.\n- The proposed framework and the ablative model based on the approximation of CL-DRO (Eq. 8) achieve meaningful performance improvement on the datasets.", "weaknesses": "- The design choice of ADNCE needs to be justified. Why such Gaussian-like weights are the most reasonable choice and can address the weakness mentioned in Sec. 5.1?\n- Missing sensitivity analysis of \\mu and \\sigma in (12), comparing to those of \\tau in InfoNCE. How much the results change as the hyperparameters vary?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer rLbF:\n\nDear Reviewer,\n\nWe appreciate your recognition of our contribution on the connection between CL and DRO. We also express our gratitude for your insightful inquiries regarding ADNCE. Below, we present responses to your comments:\n\n**Q1: Why such Gaussian-like weights are the most reasonable choice and can address the weakness mentioned ？**\n\nA1: Thanks for your insightful comment.  The primary motivation behind ADNCE is to mitigate the issues of over-conservatism and sensitivity to outliers. These limitations stem from the unreasonable worst-case distribution, which assigns excessive weights to the hardnest negative samples. Consequently, any weighting strategy capable of modulating the worst-case distribution to focus more on the informative region holds promising. In our ADNCE, we opted for Gaussian-like weights due to its flexibility and unimodal nature. However, alternative weighting strategies such as Gamma, Rayleigh or Chi-squared could also be employed. The following experiment demonstrates that these alternative weighting strategies can yield comparable results to Gaussian-like weights.\n\n|Weight Strategy| Probability density function|CIFAR10|\n|----------|----------|------|\n|Gamma-like| $w(x,m,n)=\\frac{1}{\\Gamma(m)n^m}x^{m-1}e^{-\\frac{x}{n}}$ |91.74 |\n|Rayleigh-like| $w(x,m)=\\frac{x}{m^2}e^{-\\frac{x^2}{2m^2}}$ |91.73 |\n|Chi-squared-like| $w(x,m)=\\frac{1}{2^{m/2}\\Gamma{(m/2)}}x^{m/2-1}e^{-{x}/{2}}$ |91.99|\n|Gaussian-like (ADNCE)|$w(x,m,n)=\\frac{1}{n\\sqrt{2\\pi}}e^{-\\frac{1}{2} (\\frac{x-m}{n})^2} $|91.88 |\n\nThe table above presents a comparison of TOP-1 accuracy performance on the CIFAR10 dataset across different weight strategies. The parameters $m$ and $n$ are utilized to denote the parameters within their corresponding probability density function, while the variable $x$ represents a random variable. It is important to note that, due to the domain definition of some PDFs being $(0,+\\infty)$, we need to set $x=prediction\\\\_score + 1$.\n\nAs such, any proposal that enables adjusting the weight distribution centered on the main regions of probability density is a reasonable choice.\n\n**Q2: Missing sensitivity analysis of $\\mu$ and $\\sigma$？**\n\nA2: Much thanks for underlining this point. Here we presented sensitivity analyses of $\\mu$ and $\\sigma$ on cifar10, cifar100, and stl10 datasets. The results ar presented as follows:\n\n|    $\\mu$     | 0.1  | 0.3  | 0.5  | 0.7  | 0.9  |\n|----------|------|------|------|------|------|\n| CIFAR10  | 91.3 | 91.66| 91.9 |91.77 |92.25 |\n| STL10   |87.84 |88.22 |87.56 |88.48 |88.45 |\n| CIFAR100 |69.34 |69.31 |68.70  |69.24 |68.95 |\n\n\n\n|    $\\sigma$   | 0.2  | 0.4  | 0.6  | 0.8  |  1   | 1.5  |  2   |\n|------------|------|------|------|------|------|------|------|\n|  CIFAR10   | 90.07| 91.85| 92.02| 91.77| 91.72| 91.69| 91.94|\n|  STL10   | 86.54| 88.30 | 88.10 | 87.54| 88.95| 88.12| 88.40 |\n|  CIFAR100   | 67.38| 69.36| 69.52| 69.01| 68.70 | 69.24| 69.42|\n\nThe above tables showcase the comparisons of TOP-1 accuracy performance on three datasets (CIFAR10, STL10, and CIFAR100) under different values of $\\mu$ and $\\sigma$. As can be seen, changing the parameters $\\mu$ and $\\sigma$ would impact the model performance, but not as dramatical as tuning the parameter $\\tau$. (For example, on STL10 $\\sigma$ from 1.0 to 0.2 just brings 2.7\\% performance drops, while changing $\\tau$ brings 12.6\\% performance gap.) This outcome indicates that tuning $\\mu$ and $\\sigma$ is not a significant burden. In most scenarios, it may suffice to set $\\sigma=1$, requiring only the tuning of $\\mu$ within the range of 0.1 to 0.9 (can refer to Table 6, 7, 9 in Appendix B for more details).\n\n\n**Q3: Why controlling the variance of negative samples contributes to the success of CL？**\n\nA3: Controlling the variance of negative samples is crucial for the success of contrastive learning, for two reasons. On the one hand, by regulating the variance of predicted scores for negative samples, we simultaneously reduce the variance in model loss. This, in turn, enhances the generalization capability and reliability of the model. Analogous, there are numerous  studies [16, 26, 35] on the trade-off between bias and variance in model loss, demonstrating significant advancements in performance and robustness enhancement. On the other hand, variance control also manifests as rigorous hard-mining, as evidenced by the gradient expression presented below.\n\n$$\n\\frac{d}{d\\theta}(\\mathbb{E}\\_{Q\\_0} [f\\_\\theta] + \\frac{1}{2\\tau} \\mathbb{V}\\_{Q\\_0} [f\\_\\theta]) = \\mathbb{E}\\_{Q\\_0} \\left[\\frac{d}{d\\theta}(f_\\theta)\\right] +  \\frac{1}{\\tau}  \\mathbb{E}\\_{Q\\_0} [(f\\_\\theta - \\mathbb{E}\\_{Q\\_0} [f\\_\\theta]) \\frac{d}{d\\theta}(f\\_\\theta)]\n$$\n\nHigher weights are allocated to harder negatives having higher predicted scores, capturing the core idea of hard mining. Without variance control, weights lack distinction. This necessitates using a substantially larger set of negatives, further complicating training."}, {"review_id": "6KumkGYqBl", "reviewer": "Reviewer_C48x", "summary": "This paper tackles the sampling bias problem in contrastive learning, where negative samples are usually sampled randomly from the marginal distribution and may contain false positive samples.\nThe authors provide a connection between the contrastive learning objective and distributionally robust optimization as a first part of the contributions:\nBy an optimal choice of the temperature parameter, the InfoNCE loss coincides with maximizing the loss for negative samples, leading to the worst-case negative sample distribution.\nConsequently, it is revealed that the temperature parameter admits a trade-off between the generalization error and the DRO radius.\nSecondly, the authors point out the two issues of standard contrastive learning:\nIt tends to overweigh negative samples with large distances (called over-conservatism) and be sensitive to outliers.\nTo mitigate these issues, the authors propose an alternative weight distribution over negative pairs to prevent overweighting of too-far negative samples.", "questions": "Major comments are listed first.\n\n- In Corollary 3.4 and Theorem 3.5, it is better to explain exactly what the \"approximation\" indicates.\n- With the proposed method ADNCE, can you discuss how we should choose the parameters $\\\\mu$ and $\\\\sigma$ in Section 5.2?\n\nMinor comments follow.\n\n- Typo in l.42: \"we proof\" -> \"we prove\"\n- Typo in l.82: \"Donsker-Varadah\" -> \"Donsker-Varadhan\"\n- Typo in footnote 1: The expectation looks strange.\n- Typo in l.142: $\\\\mathbb{E}\\_Q$ -> $\\\\mathbb{E}\\_{Q\\_0}$\n- In Theorem 3.2, the definition of $\\\\mathbb{Q}$ lacks. In Eq. (4), can $\\\\mathbb{E}\\_{Q\\_0}[Q/Q\\_0]$ be simply written as $\\\\int Qdx$?\n- Typo in l.156: \"satisfied\" -> \"satisfies\"\n- Typo in l.173: \"theorem 3.2\" -> \"Theorem 3.2\"\n- Typo in l.177: \"subsection 3.4\" -> \"Subsection 3.4\"\n- Typo in l.190 and l.191: $E\\_{Q\\_0}$ -> $\\\\mathbb{E}\\_{Q\\_0}$\n- In Theorem 4.2, it is better to mention $\\\\phi^\\*$ indicates the convex conjugate because this is the first place where it appears.\n- In Corollary 4.3, explaining what two random variables $X$ and $Y$ refer to make the statement complete is better.\n- In l.251, what do you compare CL-DRO with by the word \"tighter\"? By the way, the word \"tight\" indicates that equality can be attained for a given inequality, so the comparative \"tighter\" seems strange.\n- In l.257, I don't understand how \"DRO bridges the gap\" in the following paragraph.\n- In l.259, it is better to explain what $\\\\mathcal{I}\\_{DV}$ indicates.", "limitations": "The authors have adequately discussed the limitations in the conclusion.\n\nSocietal impacts do not apply to this work because this work is mostly theoretical.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- New insights on contrastive learning: One of the main contributions of this paper, the equivalent reformulation of contrastive learning with distributionally robust optimization, provides a new perspective to contrastive learning. In this picture, contrastive learning can be interpreted as optimizing the worst-case distribution over the negative samples. This interpretation can benefit us in developing a new method, as seen in Section 5.\n- An new interpretation of the temperature parameter: As a result of the reformulation, we observe that the temperature parameter in contrastive learning serves as the Lagrange multiplier for the DRO constraint, and lower temperature leads to stronger DRO constraint. This insight again contributes to the development of algorithms.", "weaknesses": "- Meaning of the connection between contrastive learning and DRO is not clear enough: Theorem 3.2 connects the DRO objective and the InfoNCE loss. Although it is very interesting and insightful, and the authors argue that contrastive learning can be regarded as DRO and thus mitigate the sampling bias, we could argue that DRO is merely contrastive learning with the sampling bias in the opposite way. This would undermine the robustness property of DRO can still suffer from the sampling bias residing in contrastive learning.\n- DRO interpretation may not explain hard mining: In Section 3.3.C and Section 5.1, the authors observe that the DRO equivalent formulation explains how contrastive learning puts larger weights to negative samples with larger $f\\_\\\\theta$. Although this seems correct, this phenomenon does not correspond to what is so-called \"hard mining.\" Indeed, Robinson et al. [38] (and many other works) define hard negative samples as \"points that are difficult to distinguish from an anchor point,\" which can be put in the current context as \"negative samples with small $f\\_\\\\theta$.\" This observation may explain why a \"strange\" trade-off is observed in Section 3.3.A such that stronger DRO constraint leads to looser generalization error, unlike what we expect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer C48x:\n\nDear Reviewer,\n\nMuch thanks for your detailed comments. In the revised version, we will meticulously polish the paper in accordance with your feedback, correcting typos and providing clear explanations of notations. Also, we find there may exist some misunderstandings.\n## We would like to make clarifications in the following:\n\n**M1: The meaning of the connection between CL and DRO is not clear enough: ... We could argue that DRO is merely contrastive learning with the sampling bias in the opposite way. This would undermine the robustness property of DRO can still suffer from the sampling bias residing in CL.**\n\nA1: It appears there exists a misunderstanding. It is not accurate to consider DRO as merely contrastive learning with the sampling bias. DRO, in essence, is a broad optimization framework aiming to minimize the worst-case expected loss across a range of potential distributions. Our work (Thm. 3.2) demonstrates that optimizing the InfoNCE loss is equivalent to optimizing a CL-DRO objective, which is a specific loss function utilizing DRO. However, it is not logically sound to extrapolate this specific instance to conclude that DRO in general is equivalent to CL with sampling bias.\n\nFurthermore, the issue of sampling bias arises from the inadequate specification of the negative sampling distribution $Q\\_0$. DRO itself does not involve how to specify $Q\\_0$. Hence, it is not valid to deduce that sampling bias is intrinsic to DRO. In fact, the adversarial nature of DRO equips CL with robustness to sampling bias.\n\n**M2: DRO interpretation may not explain hard mining: ...  Indeed, Robinson et al. [38] (and many other works) define hard negative samples as \"points that are difficult to distinguish from an anchor point,\" which can be put in the current context as \"negative samples with small $f\\_\\theta$.**\n\nA2: There seems to be a misunderstanding that we would like to clarify. Our definition of hard negative samples aligns with the definition provided in [38]. More specifically, $f\\_\\theta$ in our paper represents the similarity between two points (as stated on line 98) --- a larger $f\\_\\theta$ value indicates a higher degree of similarity. Therefore, negative samples with large $f\\_\\theta$ values are considered as hard negative samples, not those with small $f\\_\\theta$ values.\n\nRegarding the scenario with an overly stringent DRO constraint, the decline in generalization error can be intuitively understood. In such cases, the optimization process takes into account an excessively broad range of distributions, including those irrelevant or harmful ones that significantly deviate from the ideal distribution. This can negatively impact the performance of the model.\n\n## We further provide responses to the questions you have raised:\n\n**Q3: What the \"approximation\" indicates in Corollary 3.4 and Theorem 3.5?**\n\nA3: The term \"approximation\" refers to the estimation obtained by employing a second-order Taylor expansion on the objective function $\\mathcal{L}\\_{\\text{CL-DRO}}^{\\phi}$. By selecting the form of the Kullback-Leibler (KL) divergence as $\\phi^{(2)}(1)=1$, truncating it to the second order, and differentiating it with respect to $\\eta\\_1$, we can derive $\\mathcal{L}\\_{\\text{CL-DRO,2}}^{KL}$.\n\nRegarding the approximation of corollary 3.4, it is sufficient to differentiate $\\mathcal{L}\\_{\\text{CL-DRO,2}}^{KL}$ with respect to $\\alpha$ in order to obtain the final result.\n\nThe reviewer could refer to Appendix A.3 and Appendix A.4 for more details.\n\n**Q4: How we should choose the  $\\mu$ and $\\sigma$?**\n\nA4: In most datasets, it is sufficient to set $\\sigma=1$ and just needs to tune the hyperparameter $\\mu$ via grid search. The reviewer could refer to Tables 6, 7, and 9 in Appendix B for more details.\n\n**Q5: What do you compare CL-DRO with by the word \"tighter\"?**\n\nA5: Existing common variational approximation of $\\phi$-divergences is Donsker-Varadhan target ($I_{DV}$) : $D\\_{\\phi}(P||Q) \\coloneqq \\operatorname*{sup}\\_{{f}\\in \\mathcal{F}} \\{ \\mathbb{E}\\_P [f ] -  \\mathbb{E}\\_Q [\\phi^*(f)] \\}$\nwhich holds for an arbitrary finite measure in $\\mathcal{F}$—is loose when applied to probability measures as was first observed in Ruderman et al. [1]. This expression fails to take into account the fact that divergences are defined between probability distributions. If we additionally assume, as we do for $\\mathbb{R}$, that $\\mathbb{E}_P[1]=1$, then we have the tighter representation: $D\\_{\\phi}(P||Q)   \\coloneqq \\operatorname*{sup}\\_{{f}\\in \\mathcal{F}} \\mathbb{E}\\_{P} [f] - \\operatorname*{min}\\_{\\lambda \\in \\mathbb{R}} \\{ \\lambda + \\mathbb{E}\\_{Q} [\\phi^*(f-\\lambda)] \\}$\nThis result, using the infimum over $\\lambda$ in the spirit of Ruderman et al. [1], appears to have been independently proposed by Agrawal et al. [2]. As our proof in Theorem 4.2 revolves around the aforementioned equation, it is appropriate to refer to it as a tighter estimation of mutual information.\n\n[1] Ruderman et al. Tighter variational representations of f-divergences via restriction to probability measures. ICML2012\n\n[2] Agrawal et al. Optimal bounds between f-divergences and integral probability metrics. ICML2020\n\n**Q6: What's the meaning of \"DRO bridges the gap\"?**\n\nA6: While recent work such as MINE [2] and CPC [36] have demonstrated that InfoNCE is a lower bound for MI, their theoretical analyses still exhibit certain shortcomings [37]. MINE employs a critic in the dual variational form (I_DV) to establish a bound that is neither an upper nor a lower bound on mutual information. Simultaneously, CPC's proof relies on unnecessary approximations. Consequently, the challenge of bridging the gap between contrastive learning (CL) and mutual information estimation from a rigorous theoretical standpoint remains an open question. Fortunately, the power of DRO enables us to rigorously demonstrate the equivalence between CL and mutual information from a theoretical perspective."}]

Author rebuttal:
# Overall Rebuttal:
We thank all reviewers for taking the time to review our paper and for providing valuable and insightful feedback. We are delighted to see that our work has been recognized for its contributions and inspiration to the contrastive learning community, as mentioned by Reviewers $\color{red}{\text{CyLz}}$, $\color{green}{\text{sD47}}$, $\color{blue}{\text{C48x}}$, and $\color{orange}{\text{rLbF}}$.

We would like to express our gratitude to the reviewers for affirming the reliability and nontrivial nature of the theoretical framework we proposed from Distributionally Robust Optimization (DRO) perspective, as highlighted by Reviewers $\color{red}{\text{CyLz}}$, $\color{green}{\text{sD47}}$, and $\color{orange}{\text{rLbF}}$.

Regarding the introduction of the ADNCE loss, we are pleased to see that the reviewers found it both simple and significant, leading to effective results. We appreciate the positive feedback from Reviewers $\color{red}{\text{CyLz}}$, $\color{green}{\text{sD47}}$, and $\color{orange}{\text{rLbF}}$ in this regard.

We carefully considered the comments and suggestions provided by the reviewers, and we have addressed them point by point in our rebuttual. We believe and hope that our response adequately addresses the concerns raised by the reviewers.

Once again, we sincerely thank the reviewers for their valuable feedback, which has helped us improve the quality of our work.

Meta-review: This paper draws connections between false negative resilience in contrastive representation learning with distributionally robust optimization. The reviewers agree that this is novel and interesting, and that the proposed practical algorithmic intervention (the ADNCE loss) is an appealing consequence.

There remain some concerns about clarity, which the authors are encouraged to address in the revision.
Decision: Accept (poster)