Contrastive Predictive Coding Done Right for Mutual Information Estimation
Authors: Jongha Jon Ryu
Pavan Yeddanapudi
Xiangxiang Xu
Gregory W. Wornell
Venue: ICLR 2026 Poster (2026)
Keywords: information estimation
contrastive predictive coding
representation learning
noise contrastive estimation
density ratio estimation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a simple yet fundamental fix to InfoNCE for accurate and principled mutual information estimation.

Abstract: The InfoNCE objective, originally introduced for contrastive representation learning, has become a popular choice for mutual information (MI) estimation, despite its indirect connection to MI. In this paper, we demonstrate why InfoNCE should not be regarded as a valid MI estimator, and we introduce a simple modification, which we refer to as *InfoNCE-anchor*, for accurate MI estimation. 
Our modification introduces an auxiliary \emph{anchor} class, enabling consistent density ratio estimation and yielding a plug-in MI estimator with significantly reduced bias.
Beyond this, we generalize our framework using proper scoring rules, which recover InfoNCE-anchor as a special case when the log score is employed. 
This formulation unifies a broad spectrum of contrastive objectives, including NCE, InfoNCE, and $f$-divergence variants, under a single principled framework. 
Empirically, we find that InfoNCE-anchor with the log score achieves the most accurate MI estimates; however, in self-supervised representation learning experiments, we find that the anchor does not improve the downstream task performance.
These findings corroborate that contrastive representation learning benefits not from accurate MI estimation per se, but from the learning of structured density ratios.

Published as a conference paper at ICLR 2026
CONTRASTIVE PREDICTIVE CODING DONE RIGHT FOR MUTUAL INFORMATION ESTIMATION
J. Jon Ryu', Pavan Yeddanapudi', Xiangxiang Xu², Gregory W. Wornell' "Department of EECS, MIT. Cambridge, MA 02139, USA
DDepartment of Computer Science, University of Rochester, Rochester, NY 14627, USA {jonhha, pky, wwwp:/wwwwwm...o. xiangxianggudrochhster.eet
ABSTRACT
The InfoNCE objective, originally introduced for contrastive representation learning, has become a popular choice for mutual information (MI) estimation, despite its indirect connection to MI. In this paper, we demonstrate why InfoNCE should not be regarded as a valid MI estimator, and we introduce simple modification, which we refer to as InfoNCE- anchor, for accurate MI estimation. Our modification introduces an auxiliary anchor class, enabling consistent density ratio estimation and yielding plug-in MI estimator with significantly reduced bias. Beyond this, we generalize our framework using proper scoring rules, which recover InfoNCF-anchor as special case when the log score is employed. This formulation unifies a broad spectrum of contrastive objectives, including NCE, InfoNCE, and f-divergence variants, under single principled framework. Empirically, we find that InfoNCE-anchor with the log score achieves the most accurate MI estimates; however, in sulf-super representation learning experiments, we find that the anchor does not improve the downstream task performance. These findings corroborate that contrastive representation learning benefits not from accurate MI estimation per se, but from the learning of structured density ratios.
INTRODUCTION
Contrastive learning has become cornerstone of modern unsupervised representation learning, powering advances in computer vision (Chen et al. 2020), natural language processing (Mikolov ct al. 2013 Levy & Goldberg 2014), and beyond (Jaiswal et 2020 Hu et al. 2024). One of the
key ingredients in many contrastive methods is the InfoNCE objective (van den Oord et al. 201 1)).
While originally proposed as a representation learning framework, van den Oord et al. (2018) noted that the InfoNCE objective can be used to evaluate mutual information (MI), interpreting the objective as a variational bound on MI; see (van den Oord et al.) 2018 Appendix A). This interpretation has led to its widespread use for MI estimation, e.g., (Poole et al. 2019) Song & Ermon 2020a Gowri jet al. 2024 Lee & Rhee 2024). It is also widely known, however, that InfoNCE often yields a rather loose bound on MI (Poole al. 2019; Tschannen et al. 2020). As result, the InfoNCE estimator is generally considered a low-variance but bigh MI estimator (van den Oord et al. 2018). Although several proposals have been made to address this issue since its inception, the effectiveness (i.e., the low-variance property) and the limitation (i.e., the high bias) of InfoNCE remain poorly understood.
In this paper, we clarify the operational meaning of the InfoNCE objective, by showing that the objective should be understood as a variational lower bound of a statistical divergence different from establish sharp characterization of its the mutual information. Building on this clarification, relationship to the Kullback-Leibler (KL) divergence, revealing why the InfoNCE objective should not be regarded as a direct estimate of MI. We further argue that InfoNCE can be viewed as a density ratio estimation objective, while the critic (or its exponentiated form) estimates the density ratio p(u.v) only up an arbitrary function C(y), rendering it unsuitable for use in plug-in estimator. p())pt))
To address this limitation, introduce simple modification the variational objective, which we call InfoNCE-anchor, corresponding to an alternative divergence. In the new framework, the inclusion of an anchor enables the (exponentiated) critic to estimate the density ratio _la.)) directly, p(a)p(y) This adjustment facilitates consistent density ratio estimation and yields plug-in MI estimator that

---

Published as a conference paper at ICLR 2026
retains the low variance of InfoNCE while significantly reducing its bias. See Figure I for quick comparison with existing estimators, where the new plug-in in estimator based on InfoNCE-anchor demonstrates low-bias, low-variance performance.
We generalize our framework using tools from statistical decision theory, showing that InfoNCE- anchor corresponds to the log score, canonical example of proper scoring rule. This insight reveals that many contrastive objectives, including NCE, InfoNCE, and certain f f-divergence variants, can be unified under a single principled framework of density ratio estimation using proper scoring rules.
Empirically, we show that estimators induced by the log score yields state-of-the-art MI estimates across range of settings. In contrastive representation learning tasks, however, we find that multiple scoring rules yield similar performance, suggesting that MI estimation is not the primary driver of contrastive learning success. Instead, our results support that contrastive learning benefits from learning structured density ratios, regardless of whether the objectives are accurate MI estimators.
While InfoNCE has played a canonical role in contrastive representation learning, its rather loose association to MI estimation has historically fostered the misconception that representation learning essentially about maximizing MI; see, e.g., (Bachman et al., 2019. Wu et al. 2020). This paper clarifies why such an interpretation can be limiting and imprecise, and that contrastive representa- tion learning should instead be framed as representation learned to factorize pointwise MI (PMI)
p(a.y) log p(rr)pty]' or pointwise dependence (PD) p(z)p(y] All proofs can be found in Appendix G All
p(x.y)
logarithms are in base 2; hence KI. divergence and MI are measured in bits.
2 PRELIMINARIES
In this paper, we first review different types iiriationallboundss based MI estimators in the literature. and provide a taxonomy. We then delve into the InfoNCE estimator, and show why the InfoNCE estimator should not be considered as direct estimate for MI.
2.1 TYPES OF INFORMATION ESTIMATORS
Existing variational-be bound based MI estimators can be categorized into three principal categories as follows, based on the relationship of their optimization objectives to the final metrics to compute MI. Table summarizzss the representative estimators.
Type 1: Training and evaluation with single variational lower bound. These estimators optimize a tractable lower bound on the MI and use the same bound for evaluation. Examples include DV (Donsker & Varadhan 1975). NWJ (Nguyen et al. 2010), and InfoNCE vvan den Oord et al., 2018). While conceptually simple and natural, McAllester & Stratos (2020) showed that any distribution-free high-probability lower bound of MI is upper bounded by log N. where N is the sample size. This result implies that variational sower-bound-basee sample estimates of MI suffer from an inherent limitation.
Type 2: Training with variational lower bound, evaluation by plugging-in to another variational lower bound. These estimators optimize surrogate objective, often smoothed or stabilized for optimization, and then estimate MI via plug-in to a different bound such as DV or NWJ. Examples include MINE (Belghazi et al., 20188. JS (Hjelm et al. 2019), and SMILE (Song & Ermon 2020a). These methods often improve stability during training, but introduce additional sources of mismatch between optimization and evaluation. Note that the critique of McAllester & Stratos (2020) still applies this type of estimators.
Type 3: Training with variational lower bound, evaluation with a plug-in estimator. These estimators target to learn the density ratio p(a.y)
p(a)pty) directly and compute MI by plugging the estimated score function into the definition of MI. This includes recent methods like PCC/D-
RFC (T'ai et al. 2020). /-DIME (Letizia et al. 2024), and JSD-LB (Dorent et al. 2025). as well method to be proposed below. These approaches provide greater flexibility and potentially lower bias, side-stepping from the issue of the variational lower-bound approach, as they decouple density ratio learning from specific bounds.
2

---

Published as a conference paper at ICLR 2026
Table I: Overview of existing variational- -oroooonaa--oud-bbb-ee MI estimators. In this table, we use the standard critic parametrization, which aims to train c(z, y) ~ log p(z.))
p(z)p(2))
Estimator
Optimization objective Z(c) (loss)
Estimator (X:
DV Donsker & Varadhan
1975
(ev(e) & -Pat (c(,, u)] log Paceintoi lentmal Luwa(e) A -ulxas) jla, y1] + Bptolpts) (est=vot)
-Lv((e)
Type
NWJ (Nguyen
2010
-xwwa(e)
InfoNCE: (an den Cord el al.
2018
(1))NCC(e) è - (px( Eien log En elxcss)) LOILISSNCE (0) (see Eq.
CLAINCE(()
or NT-XEMI (C Chen al. 4120
MIIIffNNCE (ong & Eemon
202th
-xaLnNNNE (0)
Type 2
MINE Belghazi et al. 2018
CMINN(C) Epts. v). + EMAL 1o Cas(e) (Rotesslssp( a)M + Recclpte)) (sp(e(x, Cis(e)
-Lv((e)
J8 (Poole et a 2019)
Lxwa(e)
or NT-Logitiic CChnn et al. 2020
SMILE (Song Emmon 2020a
Letppeday (c)
Type 3
PCC/DDRFC (Tsai et al. 2020
Cis(c) / Cya(e) -2pptaa)leettao) + Excerutai Excenatanle (e (c)
Eecrap) [o(2; y)
J-DIME (Letizia al.
Bacaao) cl2, y)
JSD>-LB (Dorent 2025
Cas(e) (c) (see Eq.
Rp(xaa) [dlr a)
InfoNCE-anchor (this paper)
4 and Eq. 13
Ep(r.y [e(z, v)
2.2 DEMYSTIPYING THE INFONCE ESTIMATOR
Despite its inception as an objective for contrastive representation learning (van den Oord et al. 2018), InfoNCE has become widely considered as a MI estimator. In this section, we revisit the analytical foundation of the objective and disentangle what InfoNCE is claimed to measure from what InfoNCE indeed characterizes. Our goal is two-fold: (1) reveal the divergence that InfoNCE targets, and (ii) quantify the precise gap between that divergence and the mutual information. Before we proceed, we remark that the core of InfoNCE can be better described when we contrast two abstract distributions (ar) and (o)),, which can be replaced by p(æly) and p(r). respectively, if we wish to specialize it for mutual information.
Throughout, let 21 denote positive example drawn from the data distribution @l. and let 22. ........ 2K be negative examples drawn i.i.d. from noise distribution go- We let may - (I...... 25)) for i S as a shorthand. A score network (or critic) ræ: X + y is trained to assign large values to real samples and small values to negatives, and the InfoNCE loss compares re (x1) against the arithmetic mean ra (x²) over the whole batch.
ro(o)) Σ ro (xx)))
CIsONNCE (0) A -Dintoncu -E (o log
As we alluded to earlier, if we plug-in p(xly) and p(x) in place of 9ı (I) and (z))., respectively, then Epty) Eptt) [CI)tNcC[(O]] recovers the standard InfoNCE objective for two modalities.
The following statement from (van den Oord et al. 2018 Poole et al., 2019) is a widely known connection between the InfoNCE objective the KI. divergence, which provides a justification of the InfoNCE objective as an MI estimator for K sufficiently large. We present its proof in Appendix G. for completeness.
Proposition (1) S min(log K, ((g 9n)}.
Our first contribution to provide a right upper bound on DintaNCC(f)) which yields much sharper as a corollary.
bound on (1n than Proposition |
Theorem 2. For = E (K), define p(11.k 2) as p(ank((=) A @1 (Irz) Il (0(x))) Then, we have
DifoNcr(() < DK (0,, go) A 1 K
1 K
D (((()))())
Σ
p(T1K
K
K
z'=1
< <min<log K, D(1 || go) - log
(2
ll go) - 1) 1)++1) +1)}
K
The first inequality is tight if and only if (x) [[(=)
(a())

---

Published as a conference paper at ICLR 2026
This theorem establishes two key theoretical properties of the InfoNCE objective. First, the InfoNCE objective is a tight variational lower bound of D K-Js (91 qu), a generalization of -hasnonn divergence (JSD) which we call the K --way JSD. The InfoNCE objective becomes equal to the divergence Ds 90) if and only if rzzz X @ (z)/4((z)) Since it only learns the density ratio up multiplicative constant, one cannot use it for a plug-in estimator (i.e., Type 3 in Section 2.1) with the critic (i.e., the density ratio model) learned by InfoNCE. Second, the InfoNCE objective may be still far away from (qi qo) even for K such that log K 2 D || go). Concretely, suppose D(g |l (o) = 2. Then, the DImtONCE(B) (0) I 1.19 when K = even if log K 2 D || go), and
even for K 64, we have DImOONCE (8) S 1.93.... which is strictly smaller than the KL divergence D(1 || go) 2.
This clearly demonstrates that the InfoNCE objective (1) can never match the KI. divergence for any finite K, even when log K 2 D(g || go). Therefore, InfO is not a variational representa- tion of the KI. divergence. This contrasts with other Type estimators such as DV and NWJ, whose bounds become tight when the critic matches the true log-density ratio.
In the next section, we propose a modification the InfoNCE objective, such that the critic is learned to exactly estimate the density ratio go(x)*
(1(a)) and so that it can be used in a plug-in estimator for density ratio functionals such as mutual information.
3 TENSORIZED DENSITY RATIO ESTIMATION WITH ANCHOR
Consider two distributions (x(2) and @ (z)). To estimate the density ratio (x(x)
qn(xr using samples from
(z))) and g1 (z), we consider the following classification problem over X (hence tensorization),
where we define the class densities p(æ1:k| 2) for 0, I..... K as
class 0 (anchor) 40 (2))) ()(K))
class 1 qa (2) (x²) ... (x(xx)
class 2 (o(x1) (122) ... (o(x))
(I)
class K (0(2)))000)) (((1)))((42) ... (IX)
and the prior probabilities over the classes as p(z) = K+v for z = O, and p(z) K+v if E [K], for some v 2 0. As highlighted, class O plays a special role as an anchor, allowing us to estimate the density ratio without multiplicative ambiguity as long as v > By anchor, we mean that class 0 acts as fixed reference distribution, eliminating arbitrary scaling and ensuring identifiability, which will become precise in Theorem 3 below. We can take v 0 if K (recovering InfoNCE), but require O in the K = 1 case to avoid degeneracy. More succinetly, we can write, for 2 * 0,
|( (xx) go (0(x1))0(=2) ... (0(oo((())))))))))))))<<))) go(xx)
(xx))
(0(22) p(E1K 0).
In words, for A O, the class density is designed such that xa is drawn from @l, and the rest are from Qu We can write the marginal distribution over Ti:K as
p(x1k)) = (x)))))((22) ... go(xx) 1
K
ai (24)))
(x(z)) K k+
+
By Bayes' rule, the posterior probability p(z(=11)) is
if = 0
1 + > (xc)
p(z[T1.K)- p(z1.K 12)p(>)
1= (o())))
(xx))
(2)
p(I1K)
(
if E [K]
Σ K
1=1 (xe)
4

---

Published as a conference paper at ICLR 2026
This motivates to parameterize probabilistic classifier po(2| in the form of
if
+ ΣR (o( rx(
po (=(2 =
(3)
if E [K]
Σ i=l (o(
Applying the maximum likelihood estimation (MLE) principle, we can derive the population objective
K K + v
rØ(21)
(k)
E (o)
log
v Σ re(s)))
E (0)))))))))))
log
-
V
E K
ro(x;)
since maxp (p(e))(e..xle) Po x1: = ming (0). We call it the InfoNCE-anchor objec- tive. Suggested by the name, when K 2 and v O, it boils down to InfoNCE. In another extreme, when K = 1 and = 1, it becomes equivalent to the standard variational lower bound of Jensen-Shannon divergence (see Table TT. p. In the language of noise contrastive estimation, this provides unification of the standard NCE (Gutmann & Hyvärinen 2012) (K 1, 0), and the so-called ranking NCE objectives (Ma & (o11
(K 2, v 0).
Fisher Consistency. When 0, it readily follows from the MLE principle that InfoNCE-
characterizes the density ratio (1))) its global minimizer in the population and nonparametric limit.
(x())
Theorem 3 (Fisher consistency). Let l" arg ming (0) denote global optimizer of the
InfoNCE-anchor objective. Suppose that there exists On such that roo (x) 9a(2)*
If K 2 and vo(x) for almost every under qo- If K 2 2 with 0, there exists constant
(1(2)
ro- (I))
9i (w)
C >> 0 such that rg- (x) = (9((a) for go-almost every x. (o(a))
3.1 APPLICATION: DIVERGENCE ESTIMATION AND REPRESENTATION LEARNING
We can apply the InfoNCE-anchor objective to estimate MI or to learn representation when given a joint distribution p(æ,, v), in similar way to InfoNCE (van den Oord al. 2018). That is, for each we can apply the InfoNCE-anchor for (x) < p(æly) and (o(x) + p()) For the final objective, take expectation over y p(y):
(1) C K: (0) & Ep(s)
K
E Blees
log + Σ ro (xr,
ro( 8)
K
K K -
log
K Σ re ro y)
1/
When with K 2 2, boils down the original InfoNCE, and minimizing ( can only guarantee that for some function C(y), rg- (x, Cly) p(z)p(p) When applied representation learning, the vanilla InfoNCE (i.e., with 0) thus may lead an undesirable behavior due to uncontrollable C(y), whereas the anchor (i.e., v > 0) can remove such degeneracy. However, in our representation learning experiment, we observe that the anchor does not lead to the improvement of downstream task performance; see Section 4.3
p(a.))
With a minibatch of size B, we can implement the loss with anchor for K B as follows:
K K K+UB log V
Tbb
log
E
(b Tbj
K+UB
( Fbj
We note that the computational overhead introduced by the additional second term, compared to InfoNCE, negligible. We provide further discussion along with pseudocode in Appendix C
An altemnatiee approach is set (q. (x), (o(x)) + (p(z,, p(z)p(v))) Appendix

---

Published as a conference paper at ICLR 2026
The density ratio estimator is typically parameterized ra t- c where cz y) (the critic) is often a neural network. In representation learning, common choices are the exponen- Tau()
tial form ro ro( y) <- ex Teter (see, e.g., (van den Oord et al. 2018) or the direct form ro (z y) + 1 (ole)Taole + ( (see, e.g., (HaoChen al. 2021 ). such that fa( and go(y) are learned embeddings that approximate PMI or PD, respectively. Here, T > a temperature parameter.
3.2 INFONCE-ANCHOR INTERPOLATES DV AND NWJ BOUNDS WHEN K
One may ask about the behavior of InfoNCE-anchon when K - co. While we defer a rigorous statement (Theorem 10p to Appendix D we remark that InfoNCE-anchor, by setting to vary as
K + 0o such that / K - B for some / 2 O, we can show that InfoNCE-anchor behaves similar to a generalization of the DV bound, which can be rearranged yield
B (x) (oog - (3 + log BB+1 3++1 E R (rox))))(() S D(( go)-
When 3 O, this boils down to the standard DV bound. When 3 co, the left-hand side becomes (x) [log ro (ar)] - (x)(z) [rx(())(o(= + < D(g qo), which is the NWJ bound. Moreover, we can even show that this bound monotonically interpolates between the DV bound (tightest, / = 0) and the NWJ bound (loosest, oc). A similar asymptotic behavior of InfoNCE (i.e., for 0) was noted by Wang & Isola 2020), but specifically in the context of contrastive representation learning.
3.3 DISCUSSION ON EXISTING VARIANTS OF INFONCE ESTIMATOR
In this section, we discuss two existing variants of InfoNCE, which were proposed in the effort of fixing the aforementioned issues of InfoNCE as the MI estimator. We highlight why they are insufficient as a fundamental fix, and how different from our proposal.
a-InfoNCE. Poole et al. (2019) proposed an alternative estimator called a- a-InfoNCE, defined as
(x)
Da-InfoNCE (0) E (as (10( log aro( +
1-a K-1 Σxe re (x.)
for some a (0, 1- Note that setting a K recovers the original InfoNCE bound. For a < t. this quantity can neither be understood as a loss for classification nor be a variational lower bound for D(gs 9o). Lee & Shin (2022 Theorem 4.2) claimed that the a-InfoNCE objective a tight variational lower bound for a a-skew KL divergence D(g am + (1 a qu)) that is, De-InfoNCE( < D(g2 aq + (1 a ))) for [o, 211 and the equality be achieved. We find, however, the proof has a flaw and we can indeed show that, unless
we have
sup D Pa-InfoNCE(()) (0) D(g || aqi + (1 a)4))
(6)
in the nonparametric limit. We defer the flaw the proof, and the proof of Eq. (6) Appendix
MLInfoNCE, Song & Ermon 2020b) introduced the multi-label InfoNCE (MLInfoNCE) estimator defined as
m
ro( Σm (ra (xtr1) ) + Et Ee (o(
DMLIMINNEE (0) E mim (#w)) 11 go (
log
(7)
Song & Ermon 2200b Theorem 2) shows that DMLLIIONCE (0) < D(1 || 40). However, we note that this objective cannot be understood as loss derived from proper classification setup unlike InfoNCE-anchor.
3.4 EXTENSION WITH PROPER SCORING RULES
In the classification setup of Eq. ( density ratio estimation reduces to estimating the class probability
p(2(11:)) in Eq. (2) via the model po po(2(01:1) in Eq. (3) The cross-entropy loss in Eq. (4) is

---

Published as a conference paper at ICLR 2026
proper scoring rule, ensuring that the optimized model recovers the true posterior. More generally, once density ratio estimation is cast as class probability estimation, any proper scoring rule can be applied, yielding broad family of consistent objectives.
Here we start with a general description of the proper scoring rules (Gneiting & Raftery, 2007 Dawid al. 2012 Let 2 be discrete alphabet and let A be any alphabet. Suppose that have sample access to the underlying distribution p(a, 2) over AA x 2. The goal of class probability estimation (CPE) (Garcia-Garcia & Williamson| 2012) is to estimate the underlying class probability :7 A + (2), where (a) _ (p(z)a))zez, using samples from p(a,
To characterize a class probability estimator as the optimizer of an optimization problem, we consider tuple of loss functions À (Ag: ((2) se which we call scoring rule, whereby action ; A + (()) incurs loss A₂ ()))) for a data point (a, Then, we measure the performance of an action i by the expected loss (Apc..:))<((((a))) (M(a))]
Definition (Proper scoring rules), A scoring rule X: A(2) + R2 is a vector-valued loss function. scoring rule is said to be proper if v is optimal with respect to X, i.e., for any distribution p(a, 2),
().)) E arg min iAA+A(2)
Bp(a.s) [A= (a))))
If is the unique optimal solution with respect to X, then À is said to be strictly proper.
We note that most (strictly) proper scoring rules can be induced by a (strictly) differentiable convex function. For the sake of exposition, let 2 = {O...., M) concretely. Then, for a differentiable function : {1] x TRM -> TE, we define the W-indducd scoring rule as
^m(m) & (p. V (()) -V())
(8)
(
Proposition 5.
VV is (strictly) convex and twice disferentiable, X is (strictly) proper.
The canonical example is the log score, which results in InfoNCE-anchor in Eq. We present some examples of proper scoring rules and the generating convex functions Appendix E.5
Now, considering the classification setup in Eq. (T. let À = v be a strictly proper scoring rule over discrete alphabet Z {0.... ... K), induced by strictly convex function w: R + R. Applying the scoring rule to evaluate the score of the class probability po(2|z1:x] (in Eq. 3 with respect to the underlying distribution p(=)p((r1x< [2), we can write the population objective (to be minimized)
CK (mo) e E E)( (T (7 (x1.k)) ))],
where we use ()(])))) = (po l 2)z to denote the class probability vector. Let v7" (x de- note the underlying class probability (0(2(x1.x))=es The following statement subsumes Theorem 3 as a special case. In what follows, for a differentiable, convex function h: I2d + R. we define the Bregman divergence between x, y E Rd generated by h as BB (x, & h(x) - hly) (x(y))xxxx->>
Theorem 6. For v > O,
By (F((k) ro(
(1)) - (1²))))
V K +
Ey (20((2)))
where r" (x(1k) A (0(==))
sE(K] and ro(xu.)) A (ro(x=)) sE(KY V is convex, we have
v (F(oou))
(na) S -C*x K+v (e00
(10)
If Vo is (strictly) convex, the equality is achieved (and only if) re(x))
(x)
qo (x)
That is, in principle, we can minimize the DRE objective function in Eq. for any strongly convex, differentiable function v to estimate the density ratio. Beyond the consistency, stated in Eq. (IO), this corollary also shows that the DRE objective (with negation) can be understood as variational lower bound of some divergence between & (2)) and (z())) induced by V. defined as
7

---

Published as a conference paper at ICLR 2026
(up((1))o01)) (m)) (+((=))) ]. This is analogous that the InfoNCE-anchoo objective in Eq. (4) is variational lower bound of the K- K-way JSD Dx-Js(/1 (41-9//))) -00 We note that this extension can be viewed as a special application of the more general multi-distribution density ratio estimation studied by Yu et al. 202TT for the binary density ratio estimation.
Implementation. Similar to InfoNCE-anchor in Eq. (4) this objective function can be simplified further if the scoring rule satisfies mild symmetry condition; see Appendix F2
Alternative Characterization of Proper Scoring Rule. One minor limitation of the characterization in Theorem is that = O is not permitted In Appendix F.T, we provide an alternative characterization of proper scoring rules, which can be related to the above formulation via the perspective transformation, and admits v 0.
special case, and thus InfoNCE cannot be subsumed.
Special Cases. For the special case when K = 1 and V/ = 1, note that the right hand side becomes the
f-divergence Dy(1i || qo) when f V. That is, the objective boils down the standard variational lower bound on the /-divergence (guuyen et al. 2010), hence recovering the /-DIME objectives of al. (2023). We also note that the GAN-DIME
Letizia al. (2024) and f-MICL objectives of and HD-DIME estimators in (Letizia et al. (2024) are essentially identical to the estimators proposed
in sai et 2020). More examples can be found in Appendix
4 EXPERIMENTS
In this section, we show that InfoNCE-anchor outperforms existing estimators in MI estimation (Section 4.1) and downstream classification task (Section 4.2). We also report a negative result: anchor variants do not improve the representation quality of InfoNCE in self-supervised represen- tation learning tasks (Section 4.3). In all experiments, we set K B and use 1 as the default. Notably, this default setting consistently outperforms the baseline objectives without any hyperparameter tuning. We provide a detailed sensitivity analysis in Appendix H..4
4.1 MI ESTIMATION
We evaluate various neural MI estimators structured and unstructured data using the benchmark suite Lee & Rhee Experimtts cover three domains: multivariate Gaussian data, MNIST images, and BERT embeddings of IMDB reviews. To control ground truth MI, the benchmark employs same-class sampling for positive pairs and a binary symmetric channel to inject controlled noise. This allows systematic variation of MI from 2 to 10 bits in 2-bit increments. All experiments were done with batch size 64 and averaged over 20 random runs. Implementation details such as critic architectures and optimization setups are deferred to Appendix H.T
Figure summarizes the results. We emphasize that InfoNCE-anchan achieves both lower bias and comparable variance across domains. JSplugin (equivalent to InfoNCE- with K 1, = 1) performs comparably on Gaussians but deteriorates on and texts, highlighting the value of large K. We also evaluate Spherical, an InfoNCE-anchor variant induced by the spherical scoring rule (Gneiting & Rafiery] ); see Table Appendix inferior performance indicates that, despite the equivalence of strictly proper scoring rules, the log score remains the most effective in practice. Additional results for Gaussian with varying batch sizes can be found in Appendix H.1
tasks such as MNIST
F:3 Its
4.2 PROTEIN INTERACTION PREDICTION
As further demonstration of the effectiveness of InfoNCE-anchor, we perform an experiment from a recent study by Gowri et al. (2024). In the work, the authors examined protein embeddings derived from a pretrained protein language model (pLM), the ProtTrans5 model (Elnaggar et al.) (021I. and evaluated whether one can predict interactions between protein pairs (z, y), specifically, (K, (kinase, target) and (L, R) = (ligand, receptor) pairs in the considered setting. The interaction labels from the OmniPath database (Türei al. 2021). We ran the experiment following the same setup, with estimating the PMI using the JS, C--anho,,,,,, and a few other variational approaches, and using them to decide whether interaction exists by thresholding the PMI of given pair.
8

---

Published as a conference paper at ICLR 2026
INV
(a) Gaussian with cubic transformation.
WWV
Somerial
InfNCE
(h) MNIST.
1857
Inf MC2
KLLb 20
Sprcrial
InfeCC anchor
Texts.
Figure 1: Summary MI estimation results on the standard benchmark. Across all the cases, the proposed InfoNCE-anchor estimator (the rightmost column) consistently demonstrates low-bias, low-variance performance compared to the existing estimators. We note that JSplugin is equivalent to JSD-LB (Dorent al. 2025).
Kinase
Ligand
1.0
(b) Summary of AUROC performance on the Kinase and Ligand benchmarks (mean computed over 20 runs).
LMI
JS
(DRF)
0.4
Spherical
Objective
Kinase
Ligand
0.2
InfoNCE-anchor
0.74 +o.0s
0.87 +0.04
0.0
0.8 1.0
0.4
X (DRF)
0.76 40.07
0.92 t0.00
FPR
FPR
J8
0.77 08
0.95 10.02
InfoNCE anchor
1.0000 o0
10.01
(a) ROC curves for each problem instance.
Spherical
0.73
0.87
Figure 2: Summary of the protein interaction prediction experiment.
Figure and Figure 20 summarize the results. As shown in Figure 2a InfoNCE-anchor shows the best prediction results for both problem instances, while the JS estimator, which is a special case of InfoNCE -acchor when K and y 1, performs second best. This again demonstrates the practical benefit of large K for accurate density ratio estimation. We also recall that the standard InfoNCE objective is not suitable in this scenario, as it only estimates PMI up multiplication with an arbitrary function C(yy discussed in Section 3.1 We include the histograms of learned PMI values (Figure 5) as well as the ROC curves each estimator for different runs (Figure in Appendix H.2
4.3 SELF-SUPERVISED REPRESENTATION LEARNING
In earlier sections we showed that InfoNCE-anchor improves MI estimation and downstream tasks using the learned density ratio model. A natural question whether this benefit carries over to self-supervised learning (SSL), where InfoNCE is the standard objective. We therefore pretrain
ResNet-18 on CIFAR-100 using the solo-learn framework (da Costa et al. [2022). comparing several contrastive objectives under identical settings (batch size B 256, same optimizer), and evaluate representations via linear probing; see Appendix H.3 for results with different backbones.
Table 2 shows that InfoNCE continues to yield the strongest representations. Adding the anchor, despite improving density ratio estimation, does not translate into better SSL performance. This suggests that the uncontrollable multiplicative factor C(y) in InfoNCE is either nearly constant or irrelevant for representation learning. JS performs poorly, highlighting the importance large K, while spherical scores collapse entirely, likely due to unfavorable optimization dynamics.

---

Published as a conference paper at ICLR 2026
Table 2: Linear probing accuracy (%) after SSL pretraining. We used PD parameterization for Spherical and x². Detailed setups can be found in Appendix H.3
Objective
InfoNCE
InfoNCF-anchor
Spherical
JS
X²
Top-1 accuracy
65.98
65.74
4.33
61.69
65.59
Top-5 accuracy
89.69
89.24
17.91
87.33
88.4
On the Failure of Other Proper Scoring Rules. We highlighted the spherical scoring rule because it is a concrete example from the CrfoNCE-anchor family (see Table ( that consistently fails in practice. Among these objectives, the log scoring rule is the only instance that reliably works. As noted in Theorem 6 scoring rules are Fisher-consistent in the nonparametric and population limits; thus, their differences not due statistical inconsistency. Instead, the empirically observed discrepancies arise from the optimization landscape.
For the scoring rules that fail, observe that the training objectives plateau almost immediately and do not improve, indicating that the optimization geometry induced by these objectives unfavorable for representation learning realistic batch sizes, model parameterizations, and noise levels. While understanding the precise landscape properties that distinguish successful and unsuccessful instances an interesting direction, full characterization is beyond the scope of this work.
Structural Analysis of Learned Representations. To understand why the improved MI estimation of InfoNCB-anchor does not translate gains in self-supervised representation learning, we analyze the structural properties of the learned features. We compare the representations learned by standard InfoNCE and InfoNCE-anche using Centered Kernel Alignment (CKA) (Kornblith et al.) 2019) and the Uniformity and Alignment metrics (Wang& & Isola 2020). We observe high similarity between the two representations (CKA 0.88 for both the backbone and projector), suggesting that the anchor does not fundamentally alter the global feature geometry. The marginal improvements in Uniformity (0.350 vs. 0.357) are balanced by a slight degradation Alignment (-3.77 vs. -3.81). This suggests that while the anchor modification encourages more distributed feature space, it may slightly relax the local clustering of positive pairs, resulting in the observed parity in classification accuracy.
These findings suggest decoupling between MI estimation accuracy and representation quality, While the anchor class resolves the density-ratio identifiability issue necessary for accurate MI, standard InfoNCE already provides a sufficient training signal for learning separable features. This supports our conclusion that contrastive learning benefits primarily from the learning of structured density ratios rather than the absolute scale of mutual information.
Overall, these findings indicate that neither accurate MI estimation nor exact density ratio recovery is essential for high-quality representation. Our results suggest that what matters in SSL appears to be the factorization of PMI, benefit of large K, and favoralle optimization properties with the log score.
5 CONCLUDING REMARKS
We revisited InfoNCE and showed not a consistent MI estimator but a variational bound of some other divergence. We introduce InfoNCE-anchor, simple fix enabling consistent density ratio estimation within a unified scoring-rule framework. InfoNCE-anchan sets new state-of-the-an MI estimation benchmarks and aids predictive tasks, though it does not improve SSL performance, highlighting that accurately estimating MI is not essential for representation quality (Tsschannen et al. 2020). We hope our work clarifies the role of InfoNCE and MI estimation in contrastive learning.
ACKNOWLEDGMENTS
This work was supported, in part, by the MIT-IBM Watson AI Lab under Agreement No. W1771646, and by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number FA8750-19-2-110 conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed implied, of the Department of the Air Force or the U.S. Govermment. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
The views and
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Adv. Neural Inf. Proc. Syst., volume 32, 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proc. Int. Conf. Mach. Learn. pp. 531-540. PMLR, 2018. 2
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1):1-3, 1950. 21
James Chapman, Lennie Wells, and Ana Lawry Aguila. Unconstrained stochastic CCA: Unifying multiview and self-supervised learning. In Int. Conf. Learn. Repr. 2024. URL https::/openreevie
net/forum?id-PHit PHLVav8822
24
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proc. Int. Conf. Mach. Learn., pp. 1597-1607. PMLR, 2020.
25
T.M. Cover and J.A. Thomas. Elements of Information Theory. Number v. in A Wiley-Interscience
publication. Wiley, 2006. ISBN 9780471241959, URL httpss//bookks google e. com/books
jaD8DWAAQBAJ
15
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of self-supervised methods for visual representation learning. J. Mach. Learn. Res., 23(56); 1-6, 2022.
29
Philip Dawid, Steffen Lauritzen, and Matthew Parry. Proper local scoring rules on discrete sample spaces. Ann. Statist., pp. 593-608, 2012. 7
Alexander Philip Dawid and Monica Musio. Theory and applications of proper scoring rules. Metron, 12(2):169-183, 2014. 22
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain Markov process expectations for large time, Commun. Pure Appl. Math., 28(1): (B(1):1-44 1174,, 1975.
Reuben Dorent, Polina Golland, and William M Wells. Connecting Jensen Shannon and Kullback- Leibler divergences: A new bound for representation learning. In Adv. Neural Inf. Proc. Syst., 2025. URL
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Towards cracking the language of life 's code through self-supervised learning. IEFE Trans. Pattern Anal. Mach. Intell., 44:7112-7127, 2021.
Dario Garcia-Garcia and Robert C Williamson. Divergences and risks for multiclass experiments. In Conf. Learn. Theory, pp. 28 JMLR Workshop and Conference Proceedings, 2012. 7
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. J. Am. Statist. Assoc., 102(477):359-33 378, 2007. 7
I. J. Good. Rational decisions. J. R. Stat. Soc. B, 140))107-114, 1952. 22
Gokul Gowri, Xiaokang Lun, Allon Klein, and Peng Yin. Approximating mutual information of high- dimensional variables using learned representations. In Adv. Neural Inf. Proc. Syst., volume 37, pp. 132843-132875. 2024.
28
Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. J. Mach. Learn. Res., 13(2), 2012. 5 23
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for sele-super deep learning with spectral contrastive loss. In Adv. Neural Inf. Proc. Syst., volume 34, pp. 5000-5011, 2021. 6 24
11

---

Published as a conference paper at ICLR 2026
R Devon Hjelm, Alex Fedorov, Samuel Maroii--arrhhloon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In Int. Conf. Learn. Repr. 2019. URL https://opentevvee..n///ooouss
Bklr3jackK
2
Haigen Hu, Xiaoyuan Wang. Yan Zhang, Qi Chen, and Qiu Guan. A comprehensive survey contrastive learning. Neurocomputing, pp. 128645, 2024.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Techmologies, 9(1):2, 2020. |
Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach direct importance estimation. J. Mach. Learn, Res., 10: 1991-1444,, 2009. 24
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proc. Int. Conf. Mach. Learn., pp. 3519-3529. PMIR, 2019.
10
Kyungeun Lee and Wonjong Rhee. A benchmark suite for evaluating neural mutual information estimators on unstructured datasets. In Adv. Neural Inf. Process. Syst., Datasets Benchmarks Track, 2024. URL https:///openrevieww http://pwwwwoooeeeeeee/e///o/aaa
8 28
Kyungmin Lee and Jinwoo Shin. RényiCL: Contrastive representation learning with skew Rényi divergence. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Adv. Neural Inf. Proc. Syst., 2022. /ttp///////////////////////////a
16
Nunzio Alexandro Letizia, Nicola Novello, and Andrea M Tonello. Mutual information estimation via f-divergence and data derangements. In Adv. Neural Inf. Proc. Syst., 2024. URL https
Omer Levy and Yoav Goldberg. Neural word embedding as implicit matrix factorization. In Adv.
Neural Inf. Proc. Syst., volume 27, 2014. [
Yiwei Lu, Guojun Zhang, Sun Sun, Hongyu Guo, and Yaoliang Yu. SIS-MICL: Understanding and generalizing infoNCE-based contrastive learning. Trans. Mach. Learn. Res., 2023. ISSN 2835-8856. URL httt::://openreviieuneeee//aeeee 2De33uuuZ
g
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. Proc. Conf. Empir. Methods Nat. Lang. Process., pp. 3698-3707, 2018.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In Int. Conf. Artif. Int. Statist., pp. 875-884. PMLR, 2020. 2
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation word representa- tions in vector space. In Int. Conf. Learn. Repr., 2013.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory, 56 555447781, 2010.
3 24
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Proc. Int. Conf. Mach. Learn., pp. 5171-5180. PMLR, 2019.
24
Jongha Jon Ryu, Xiangxiang Xu, Hasan Sabri Melihcan Erol, Yuheng Bu, Lizhong Zheng, and Gregory W. Wornell. Operator SVD with neural networks via nested low-rank approximation. In Proc. Int. Conf. Mach. Learn., 2024. URL httpp::/ppenreviee.eee//ffraeeee wttpre/eeweeeeeeeeeeeeerorrrttaa 24
Jongha Jon Ryu, Abhin Shah, and Gregory W. Worell. A unified view learning unnormalized distributions via noise-contrastive estimation. In Proc. Int. Conf. Mach. Learn,, 2025. URL
23
12

---

Published as a conference paper at ICLR 2026
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In Int. Conf. Learn. Repr., 2020a. http://s///eeeeeeeeeee/////////r Bit.e/dNdD.XEENDDSSS
Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. In Adv. Neural Inf. Proc. Syst. volume 33, pp. 8161-8173, 2020b. S
16
Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Ann. Inst. Stat. Math., 60 (4):699-746, 2008.
22 24
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the Bregman divergence: unified framework of density-ratio estimation. Ann. Inst. Stat. Math., 64(5):1009 1044, 2012.
22 24
Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdi- nov. Neural methods for point-wise dependency estimation. In Adv. Neural Inf. Proc. Syst., 2020.
24
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In Int. Conf. Learn. Repr., 2020. URL
Dénes Türei, Alberto Valdeolivas, Lejla Gul, Nicolàs Palacio-Fscat, Michal Klein, Olga Ivanova, Márton Ölbei, Attila Gábor, Fabian Theis, Dezsö Módos, Tamás Koresmáros, and Julio Saez- Rodriguez. Integrated intra-and intercellular signaling knowledge for multicellular omics analysis. Mol. Syst. Biol., 1793), 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: /807.03807 2018.
25
Lichen Wang, Jiaxiang Wu, Shao-Lun Huang, Lizhong Zheng, Xiangxiang Xu, Lin Zhang, and Junzhou Huang. An efficient approach to informative feature extraction from multimodal data. In Proc. AAAI Conf. Artif. Intell., volume 33, pp. 5281-5288, 2019. 24
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In Proc. Int. Conf. Mach. Learn., pp. 9939-9939. PMLR,
2020. 6 10
Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual infor- mation in contrastive learning for visual representations. arXiv preprint 2305::2005.13149, 2020.
Lantao Yu, Yujia Jin, and Stefano Ermon. A unified framework for multi-distribution density ratio estimation. arXiv preprint arXiv:21 2021.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
Clarification on a- G-InfoNCE
14
A.1 Corrected Statement
14
A.2 Issue in the Original Proof
15
Alternative Approach MI Estimation
16
Pseudocode for InfoNCE-anchhn
17
D
Asymptotic Behavior of InfoNCE-Anchor
17
Theoretti Treatment of Proper Scoring Rules
18
Preliminaries and Definitions)
18
E,2
From Loss Function to Generating Function
19
E,3
One-to-One Correspondence"
20
E.4
Connection to Bregman Divergences
21
Examples Proper Scoring Rules
F
Details on Extensions with Proper Scoring Rules
22
F1
Alternative Characterization of Proper Scoring Rule
22
F2
On Implementation
F3
Examples of InfoNCE-anchor Type DRE Objectives
23
Deferred Proofs
24
G.1
Proof of Proposition
................................ ................................
24
2
Proof of Theorem
G.3 Proof of Proposition
26
G.4
Proof of Theorem 6
27
Experiment Details
28
H.
MI Estimation
28
H.2
Protein Interaction Prediction
28
H.3
Self-supervised Representation Learning
29
H.4
Impact of the Anchor Parameter
29
A CLARIFICATION ON a-INFONCE
In Section we noted that Lee Shin 2022 Theorem 4.2) is not valid as stated the nonpara- metric limit. Here, present corrected statement and explain why the argument in their proof does establish the claimed identity.
We first restate their statement.
Theorem (Lee & Shin 2022 Theorem 4.2). Let (0, f]- In the nonparametric limit,
sup Da InfoNCE (0) = D(g) aq + (1 or)go)
They also state an analogous result for the a-skewed version of MLInfoNCE,; for brevity, we focus on a-InfoNCE, since the same issue arises there.
Below, we give the correct characterization, which shows that the equality Theorem fails in general. We then identify the precise step where the original proof breaks down.
A.1 CORRECTED STATEMENT
We can prove the following:
Proposition 8. For any E [0, in the nonparametric limit,
sup D., Infuccef))
D(g1 aq + o))o),
14

---

Published as a conference paper at ICLR 2026
where the equality holds if and only 91 q0-
That is, if 91 * then we have strict inequality
sup Da-IntoNCE(O) > D(gs || agg + (1 - a)go).
which invalidates Theorem
We now prove the proposition.
Proof of Proposition g In the nonparametric limit, we consider 0, which satisfies ro 2()
go(2)*
Let (2(11:K)
(o(2x) (o (21 (x22)))) ... (o(xx) denote the class density for class E [K] as defined
in Eq. Tp. Then, for such On the a-InfoNCE objective can be written as
(1(1KK) apI(x1:K) + 1-a
D Da-InfoNCE(Po) (x)x) log
K-1 Σ (T(:K)))
K
D -D(p((11))) (F + - a
Σ Pa (x 2
K
Here, we invoke the monotonicity of the KI. divergence (Cover & Thomas 2006;: for any joint distributions p(æ, y) and q(r, v). if p(x) and q(x:) the marginal distributions, respectively, have
D(p(2, v) || ((x v)) = D(v(x) (((x))) + (p(()((((())))) Il [(vl)))]
2 D(p(z) |l g())),
where the equality holds if and only if p2tz) (Pp(y( || [((+)))] O. equivalently p(u|2)) <(v(x)) for p(z))almost everywhere (a.e.). Since the marginal distribution of the two distributions pı (x1:k) and api (fT.K) L-a Σ
2 over 2ı are (11))) and 9i (x1)
kー Σ==20 (x)) am (21) (1 a)) (21). respectively, have the following chain of in-
equalities:
sup D De-InIONCE( 2 D (anIntoNCe(Bo))
D (((((11)))
(pI(F1x))
1- a K
Σ (((kk)
5=2
2 D (1 (21)
aq (21) (z))
It remains to characterize the equality condition. From the monotonicity lemma, the last equality holds if and only if the following holds for d (x1)-a.e.
(1(11k) (x
+ 1-a E Pa (ark)
k-
Q1 (x1)
(0(
@ (x1) a
K
@1 (az)
(0()))
K
(o(
The only way the above equality can hold almost surely is that d (x)/go(x)) is constant almost everywhere, which implies @i qo.
A.2 ISSUE ORIGINAL PROOF
To prove this statement, Lee & Shin (2022) used the sandwich argument:
DDv a)go) D (f)) D(g ll aqg (1 a)go).
15

---

Published as a conference paper at ICLR 2026
If both inequalities held, since the DV bound is the tight lower bound on the KL divergence which becomes tight in the nonparametric limit, we can conclude equality in Theorem However, Proposi- tion 8 states stronger inequality than (a), and implies that (6) cannot hold.
Since (a) follows as corollary of Proposition 8 we now detail the specific flaw in their proof for (b). To argue (b), they invoke the NWJ bound. That is,
D q) DNwJ p, - (etee)logro(o)) Bo(o)Foo(a) + 11,
p(z) where the equality holds if and only if Fo(x)) q(z) q-almost everywhere. For fixed ; 2K they plug a test function
(0(
fo(
aro(x1) K-1 Ee= (0(x2)
(e: Q1, aq1 + (1 )))) which yields
T0(
D(q agi + - a)) B (z1) log
arr( + 1-a E ro (1xz)
ro(
Eng (2))+(
Lara(x1) K-1 Em=2ro0s ro(
1-a
Taking expectation over (x())) ... ...........0 (xx), we have D(g || ag (1 aa)
ra(z1)
InANNCCC (0) + (22) (o(x))
E (os) Lora(x)) + K-1 Ex=rro(ss T0(02)
I-a
To conclude D(g orgi + - a))o) 2 Do-InfoNCF (0), then claim that
ra(x)) I-a
(0)
po(()))...
Ran(())) Lara(z)) + K-1 Ex=
21
ro(2))( ro(xz)
for a E (0, 4. This argued based on the following propositions from (Song & Ermon 2020b which we rephrase for simplicity:
Proposition 9 (Song & Ermon then
2020b Propositions and 2). If Z1. Zz,... Zx are exchangaabl.,
Z1
if (0, 1 ifa E K 1
E
ak [1
7Z +
j= Z
That is, Lee & Shin 2022 apply this proposition to argue (c), by letting (Zi...., 4x) + (ro())), To (T)) Note, however, that cannot apply the proposition here, the random variables are not exchangeable under (z1...., ~ (agg( + (1 (x)((=))))((2))) (.
B ALTERNATIVE APPROACH TO MI ESTIMATION
As alluded to earlier Section 3.1 we can construct an alternative consistent objective function for p(a.v)
estimating the pointwise dependence p(m)pt))' That is, applying the InfoNCF-ancchr framework for qı (x) + p(z. v) and qo (z. 9) + p(x)p(z), we obtain
K
To( W1) log
K:w (0) A
E (elee.
K
Σ ro (0)
K -
E
log
4 Li=l re(x,,
16

---

Published as a conference paper at ICLR 2026
While this version results in different, yet consistent objective function, it is not preferable over the discussed approach in practice.
With this approach, when v = O, minimizing < (0) guarantees that for some C' > 0,
ro- (x, y) = C p(., v)
p(r)p(y)
This is guarantee analogous to the MLInfoNCE (Song & Ermon 2020b).
C PSEUDOCODE FOR INFONCE-ANCHOR
Here, provide pseudocode for the PyTorch implementation of the InfoNCE-anchor objective function.
def infonce with aath_anchor(scorees nu=1.0):
scores: [a, B]
scoresli, f(x_i, v_3)
nu: prior smoothing hyperparaneter
5
assert
device scores.device
scores.size(n)
10
joint tern
torch.zeros(B, devicesdevice) trrchharange(1, 1) 14 mask(i 1, i 2] itorch.inf
scores_aug torch.cat([
pog(o)(nu)> torch.ones(B, 1, device-device),
scoresl, dim=1) augnented
joint_ters (scores.diag() scores_aug. caare__ug..oogsueeppf.... (ogsumexxp(disss)mmmmn)(
independent tern neg. inf diag scores aug neg torch.cat(I
torch.zeros(B, device-device). diagonal (-torch.inf)
22
23
loglog(nu) torch, 1, device-device),
24
nes_inf_dias_me diaa__nsk scores
1, din-1) negative augmented
narginal_ tern (np.Log(nu) scores_aug_nege (ogssmesppdinnss)mmeddd
27
28
return (K / (K nu)) joint_ term (nu / (K nu)) marginal_ term
We note that, compared to InfoNCE, the additional term in lines 20-26 introduces an additional logsumexp
(B 1) tensor, so the loss computation has slightly larger constant factor compared to standard InfoNCE. However, the dominant cost in our experiments comes from the encoder forward and backward passes and from computing the pairwise score matrix. In this context, the extra logsumexp and concatenation are negligible. Our wall-clock measurements confirm that the end-to-end training time with InfoNCE-anchor is essentially the same as InfoNCE, across all settings.
D ASYMPTOTIC BEHAVIOR OF INFONCE-ANCEE
Theorem 10. Ifv/K + + as K for some 3 > 0, then
K Klog K
1 (o)g log(3 ( E (ro(x)))
lim K +0000
3+1 log 8
E
K +
B +
1
<
log B +
D(g qu) - log( 1).
3+1
8 +
The equality holds if and only ro()) when / 0.
(x) xx
when 3 > O, and ro(x) C.21 (x) 90(2)
for some C
17

---

Published as a conference paper at ICLR 2026
Rewriting lower bound the KL divergence, we have Eq. (5)
E DECISION-THEORE
TREATMENT OF PROPER SCORING RULES
This section serves as a decision-theoretie foundation proper scoring rules for conditional proba- bility estimation, which essential proving the main statements Section 3.4 i.e., Proposition and Theorem [ Appendix F.3 is marked with asterisk, which is included for completeness and can be safely skipped in the first reading.
E.1 PRELIMINARIES AND DEFINITIONS
We first note that
Ep(a,s) (+(aa) Ep(a) Σ ((a)))
which implies that we only need to study the conditional problem for each a, without the expectation p(a). We define the (conditional) risk of A(2) with respect n)" E A(2) as
(n
& )= = (n )(n))-
ZEZ
In particular, we denote
(n))) - dh (1) n7)
and call the pointwise Bayes risk with respect to 17", since we can write the Bayes-optimal risk as
Ep(a,e) (m((a) = (d (7))) = min
Px ):A-A(2)
given that X is proper.
Since the propriety of a scoring rule is independent of the distribution p(a) over A as alluded to earlier, restate the definition of propriety as follows. We define and denote the regres of with respect to 17" under À as
Reg (n || n² (n (1 (n )²)
Definition 11. A loss-function tuple À is said to be proper if Reg (i) 17")) 2 0 for any n. 17" and (1" || (ny)) = 0 for any v)" loss-function tuple À is said to be strictly proper if it is proper and Reg > (n) || n7" 0 if and only if f 17",, for any n²
We now state the characterization of differentiable (strictly) proper loss functions. If differentiable, we let
g^(n") e V 7'') = ((V,A(m) 17 n=) J''
(11)
which is the gradient of the pointwise risk function i - d (n m²) at n n Here, JA(n") Rmxm denotes the Jacobian of the matrix X: A(2) + R...e...
(J)(n")) A (1²)
On,
Theorem 12. scoring rule X (A+: A(2) +R)) is (strictly) proper if and only (1) the pointwise Bayes risk function + (n) is (strictly) concave over ((2) and (2) g (n) = 0 fo any 77' A22).
To prove the theorem, we first state a key technical lemma. Given a differentiable function f: V - I over subset V of an Fuclidean space, we define the Bregman distortion B V as
By(u, v) A f()) - f(v) (VS(v),u v).
If is convex, By 2) called the Bregman divergence generated by f. Finally, let ( 2 (m) denote the negative pointwise Bayes risk.
18

---

Published as a conference paper at ICLR 2026
Lemma 13. For any n,,", have
Reg (f) ll (1)n")))))
Byaln" (nr n) + (g) -.)"
Proof. By chain rule, we have
()) = X (m)
Therefore, by the definition of Bregman distortion, we have
B B (n' n) = (n= .n)
+ = + - (g) = Reg (1) + ( - (g)
-nd
which concludes the proof.
The first condition (1) imposes that the estimation problem becomes (strictly) not easier as we mix the class probabilities. The second condition (2) formalizes that if 17 is the underlying class probability, then 77 = 17" is a local minimizer of the conditional risk function + dn m7")
Now are ready to prove Theorem 12
Proof of Theorem 72 We first prove the only-if direction. If À is proper, then ( n" > d (n" '7") for any 17 and r7" by definition. That is, n + d² || n7") is stationary at n n and
thus the gradient (n) V d (f) |l mn 0 for any 77. Now, by the identity in Lemma 13
have B B f) = Reg (n) n"" 2 0 for any 17", n7 which implies that the function fa
7
is concave. Further, if À is strictly proper, then B B B,>(1) (n" which implies that fa is strictly concave.
n)) Reg" (n n²) > 0 for any n + n
For the converse, i.e., the if direction, can directly apply the identity in Lemma 13 and conclude
Reg (n n7") 2 B (n" .)) 2 0 by the convexity of 7 It is clear that À is strictly proper if 7 is strictly concave.
E.2 FROM LOSS FUNCTION TO GENERATING FUNCTION
Given loss function X, we define a corresponding generating function
V v ف (p, X(1))
for p E (1} x R$ so that we can write the pointwise Bayes risk at 17 as
(amn)) = d .7" - "
Then, is easy check that Proposition 14. If A is (strictly) proper, p + - is (strictly) convex.
Proof. If A is (strictly) proper, then the negative pointwise Bayes risk function 17" 16 u (p") is (strictly) convex by Theorem 12 Since the mapping is perspective function of
-n-"(n)))
+ L-( (p), a must be (strictly) convex.
Remark 15 (From generating function to loss function), Conversely, we can define loss function from a differentiable function v {1} x RM as follows:
v (p) p) {Pp)]
A (n) A
V
so that write the pointwise Bayes risk 77"
dx (n n7") -10 (p").
19

---

Published as a conference paper at ICLR 2026
E.3 ONE-TO- ONE-TO-ONE CORRESPONDENCE"
A natural question to ask is whether À + VoA and V + X are inverse mappings each other. Indeed, have the following propositions.
Proposition 16.
= (n)= (n) (g))
Hence, in particular, if À is proper, it readily follows that A U (n) = X
Proof of Proposition 16
First, we consider 2 E (1,,..,,M, M). Note that
(n.(n) θ (1, ... PM vA.(n))
Öp, ppp + P1 +... PM
= =no + ey, (n,(n)).
for any 0, ....., M. Thus, have
M
(1a(n)
Σ ==0
170 Σ pol 1el-n+ey V)a(1))
Pr Opz
=(-n+ey -ny, ey, Σ 112 VX=
(- + ey, er, gg, (n))).
This implies that
@n)(n)
1 (n)
Op=
0Ao(n)
M
(n)
{x Σ
Pz
Op:
Ap:
M @).(1)
->=(1)) Σ P== 0p:
2=0
=->((n) (n o., (n))
We now consider z 0. Observe that
(T)
(p) Σ Pz
Op:
M
(e)e(n) - (n - en. (n)).
=
Hence, we have
Ay (n) (VV (p). - -p)(p)
3
M
-No(n) Σ (2)x(1) Endd.(n) (n + e. e.. +e.gg(m)) (n))
= =oo(n) - (n + ey, 1o (n)).
This concludes the proof.
The following statement asserts that the generating function induced by the induced loss function of generating function corresponds to the original generating function. Proposition 17.
V (p) (p).
20

---

Published as a conference paper at ICLR 2026
Proof. By definition, it is easy to check that
T (p) = -p, (p))
M
(p(p)
= ((V(V(()) (p)) (p))++ +
Σ Pz
Op:
= -pp).
Therefore, there is one-to-one correspondence between (strictly) proper loss functions (A: A(2) 2 and (strictly) convex functions : {1} x R4 - RR.
E.4 CONNECTION TO BREGMAN DIVERGENCES
Note following proposition. Proposition 18.
Bya(1 (n=" .n) - Ber p).
The following corollary reveals that any CPE objective function induced by proper scoring rule can be understood Bregman divergence minimization. Corollary 19. If X is proper, then
Reg || ny*) = (p'
In other words, shows that a proper loss function À acts only through the form of the Bregman divergence (.)(-.)) In other words, À and X' are equivalent CPE loss functions if Baa(;,) Byx (t.)). This defines an equivalence class in the set of loss functions
A & {x: A(2) + 2 B (..)) (-(
We know that this set is always not empty, since Proposition implics that A E(V).
Consider subset
A.(w) (A: A(2) - E 0}.
The loss functions in this subset can be thought as canonical functions, as we require g (n) = 0 to check propriety in Theorem [2] Note that
(V))
since Lemma 24 establishes that g^* (n) = 0. A small open question is whether A is an unique element of Ao Remark 20 (Properization), We remark that for any A E A(V), map it another element X E Ao (,, by defining it as
X'(n) & A(n) + (n) (n)))(
is easy to check that X' E (v) indeed. One can think of this as a properization of a loss function X, since for convex function V,, any loss function À A(V) can be made into proper loss
E (D).
E.5 EXAMPLES OF PROPER SCORING RULES
We first start with proper binary scoring rules; see Table 3 Most of the examples can be found from (Geiiting & Raftery, 2007). We refer to rules generated from the w-iinducdd scoring rules (Eq. ( by asymmetric scoring rules, and the d--induced rules (Eq. 12 by symmetric rules.
II a = 2, famously known as the Brier score Brier 1950
Gneiting Raftery 2007)
Called the spherical score when 2 (Gneiting & Raftery 2007
When 1, boils down to the log
21

---

Published as a conference paper at ICLR 2026
Table Examples of strictly proper binary scoring rules.
Asymmetric scoring rule
(p)
A (n)) Af (m) (see Eq.
KL.IP (Sugiyama et al.
2008
plog
log
21 Do
Wo
Robust DRE (a: é {0,1
al-11 or afa=1)
ni +
(Sugiyama al. 2012
a(a-1)) Ima
Inverse log
log
log m 22 np
Symmetric scoring rule
( 171)
(n) (see Eq.
Log Good
1952
No log 7o + 77 log 77 10 +vf
log ni t
Power (a $ 1)
2
o(o-1) 1)
Sym. inverse log
log 1/ log 1
log ru + log 7 + 1
Pscudo-spherical (a. g {0. 13
2
(Gneiting & Raftery 2007
(ng top)
Now, by naturally extending the definition of the elementary generating functions for the binary scoring rules, we can derive their multi-ary counterparts as shown in Table We note that the multi- asymmetric scoring rules, when considered with our binary density ratio estimation framework below, boil down the ones induced by the binary scoring rules. Therefore, since the nontrivial examples are from extending the symmetric scoring rules, we omit the multiary version of asymmetric rules.
Table 4: Examples of symmetric strictly proper (M 1)-ary scores.
Symmetric scoring rule
(1)
A (1)) (see Eq.
Known
Log
(n, log 7)
log 17=
Power (a: & {0, 1})
|all
Tsallis scoring rule
a(a11 ala-1)
a
a
Dawid & Musio 2014)
Sym. inverse log
Σm= log
EM log 1/a +
Spherical (a & (0. 1})
(M+1)
||n||
(M+1) a-1
171a
F DETAILS ON EXTENSIONS WITH PROPER SCORING RULES
In this section, we provide technical materials deferred from Section 3.4 on the extensions with proper scoring rules.
F.1 ALTERNATIVE CHARACTERIZATION OF PROPER SCORING RULE
An alternative, yet equivalent representation of a proper scoring rule is based on a convex function (nn) over 7) E A( MJ). One can induce convex function (p(p) from convex function (()) by the perspective transformation:
[1:p]
( (1+p1 pM
1 P1 +... PM
Theorem 21. Given differentiable function Φ: A((0 : M)) + TR.
X * = (n. v,, (nn) 10 (1))
(12)
Proof. First, we can write
L (no) (n))
K" (n² Ep(xnx) (2'(x1)) 2 (mo()1))))) (n" x
22

---

Published as a conference paper at ICLR 2026
It is easy to check, from the definition of the V -induccdd scoring rule in Eq.
12
(n' A w (mo)) - (V,, no)
In particular,
(n' > (n'))
-(()))
Hence, we have
(n) x (ma)) - (17 Vo (n² = B
See the proof of Theorem in Appendix for a comparison.
We remark that, for a (strictly) convex function Φ V is (strictly) convex since the perspective transformation preserves the convexity, and thus
xx 2 (n x"=")">(m"))>>>>>>>>>>>>>>>>>>>) = -
We note that the right hand side is the Bayes optimal risk. other words, a convex function Φ(-) can characterize proper scoring rule as its (negative) Bayes-optimal risk.
Theorem 22. For 2 O.
( (k.((n)) Bp(s1.x) Bo (n" (x1.k), no (x(.))))
For and convex function 4 we have
- (no) S - -K (m" = (n (#1xk))]
Ifv> > 0, for strictly convex function 4 the equality is achieved if and only if (a(
[0(2)*
If v i.e., if there no anchor class O, we can only estimate the density ratio up to multiplicative constant, as the original InfoNCE guarantees.
F.2 ON IMPLEMENTATION
Here, we We say that scoring rule is (y1, 82 -invarian for y1 + 1/2 E yf Xy, (1) = Ayz (17') and Aya (n) Xy1 (n') for any n, n/ such that fly V/, for y & {y1, 12) and M/5 T222 and 1/ya 1/y
Proposition 23. If the scoring rule {21, 22) rea-innarian for any (21,22) (1,2,.... K}, we have
K
(n)) =
K Beleeraoee)---ooeae (o0))) (A] ( (no(T1K)))
K E Enn
(1 (xI.K))
(13)
F.3 EXAMPLES OF INFONCE-ANCIORR-T
DRE OBJECTIVES
Recall the examples of proper scoring rules in Appendix E.5 Table we first list the canonical consistent DRE objectives derived by asymmetric scoring rules (see Table B). As noted earlier, the tensorization of InfoNCE-anchor does not have any effect with asymmetric scoring rules, and the objectives boil down to the standard binary DRE objectives.
As noted in the last column of the table, these binary DRE objectives have been extensively used and studied in the various literature DRE, MI estimation, and representation learning. We mention in passing that a recent paper (Ryu et al. 2025), building on noise-contrastive estimation (Gutmann & Hyvärinen 2012). revealed connection between these rules and the maximum likelihood estimation principle.
In Table 6 list the InfoNCE-anccor-yye objectives based the symmetric scoring rules (see Table [4) Table 7 lists the corresponding InfoNCE-type objectives (i.e., without anchor). We remark that the Spherical objective in the main text corresponds to the last row in Table 6
23

---

Published as a conference paper at ICLR 2026
Table Examples consistent DRE objectives derived from asymmetric scoring rules (first half of Table Note that these objectives induced by asymmetric scoring rules do not depend on K and v.
Asym. scoring rule
Cxw(n)) (see Eq.
Known as
KLIEP (Sugiyama et al. 2008 in DRE.
Log
E log ra()) + go [+)
NWJ Nguyen et al. 2010) MI estimation.
Robust DRE
(Sugiyama et al. (012))
Power (ex & (0,
Bn (a(a))a 1-ar
Ego [ofor)
KLIEP (Sugiyama al. 2008) when + in LSIF (Kanmmor et al. 2009) when DRE.
In MI estimation/DRR, known X
DRF (Tsai score (Wag el a.. [2019) spectral contrastive
2020 In rep. lcarning, H-
(when 2)
En [r) + (
loss HaoChen el al. 2021), CCA (Chapman
et al. 2024
LoRA loss (Ryu al. 2024).
Inverse log
E (r((s)] Egu [z)]
Table 6: Examples of InfoNCE-anchor-tyype DRE objectives (v > 0), derived from symmetric scoring
rules (Table 4). Here, P,(x1:K) [1,
Po (xxx E R-++1 and po(x) & ra(x) The
objective in the first row corresponds our proposal InfoNCE-anchor, When K 1, v = it is
also known as JS (Poole et al. 2019) or NT-Logistics (Chen et al. 2020).
Sym. scoring rule
Ri (no) (see Eq.
Log
x
E (
log
Pa(z1) + K+N E log [
K+
Power (a & (0.1))
K+u
K (anin))
(a )" + Ima |
Po(i)))
Rto Ballan))
Pul k) I-a TPal
k>ll
K E. (mn))
Po (=(
Sym. inverse log
K+v
[|
po(x))
" K4v (x
(pplit)))) + (
Pseudo-spherical
K En
( po(an)
pelprrllllll )a kto Extten >ll (natrna)1))
(ax & 10)
K+o
(a 2)
K R+v Emtrmae)
po(a))
+ K+o Ben(n x)) x ())
G DEFERRED PROOFS
G.1 PROOF OF PROPOSITION
Proof of Proposition We have an alternative proof for loose upper bound
(0) + log K < D(1i |l go).
We first consider the NWJ variational lower bound of the KI. divergence:
D(gi |l gn) 2 E,,, [log r]
E [r] + 1.
Here the equality holds if and only if r(x) = (=(x) qi (21))0(x2)) -. go(=x) and go(x1)90(22) ... (xx) Applying the NWJ lower bound, we obtain
(m) For K 2, consider two distributions
D(a (x: || (x()))
D(a (21) (40 ... (o(x)) ll (00= ... (x(x)))))))))))))
2 E (x (x).r((1,................. IK)] Ep (12)) (( Fx))
Note that, again, the equality is attained if and only if
41 (x1)
go (x))
24

---

Published as a conference paper at ICLR 2026
Table 7: Examples of InfoNCE-type DRE objectives, derived from symmetric scoring rules (Table
Sym. scoring rule
k (No) (see Eq.
13
Known as
InfoNCE
van den
Log
E (x log re(e1)
[
Oord et al.
2018 MNT-
Xent CChnn al. 2020
Power (a & {0, 1))
1
ra(21) )0-1)
I-a
[
Sym. inverse log
By (m
lox a
ratm17
Pseudo-spiin spadoo-pheerial (a 1))
E (x
ra(in)
Tra
(a 2)
B (
rofan]
Now, we consider a specific (suboptimal) parameterization of r(21, RK) xK) + log ro(
in the following form:
Σk= re(#))
K
for some nonnegative-valued function ro: X R By symmetry, it is easy to show that
go )=t)))))) (r(1,, (x. =1.
Hence, the NWJ lower bound simplifies to
log (2)1 K Σ ro ro()))
(z) go(x)) 2 E (m)))
-K;o(0),
(14)
which concludes the proof.
G.2
PROOF OF THBOREM 2
Proof of Theorem
2
We start with the following upper bound
-K.: = lx
< lss
where the upper bound is achieved when pe (2 [1 = p(=(=1k)) or equivalently
This by the Gibbs inequality,
E D(p(z(=1x)) (=(=(<1))))) 0.
note that for = O, we have
(O): + log K < (og [1 + log K
@ (ar)
E (
log
4o (ar)
(15)
K (1(2.)
K Σ go(
Dis (p(r 1).... p(
-K))
The equality condition follows from the Gibbs inequality. This proves the first inequality. To prove the upper bound log K, continuing from Eq. (15) we have
-O) + log K < E (an (202)))))
log
To (23
K
(ma) (172)
==
(1
< Eyg (o)))
log
log K.
K 9o (m)
25

---

Published as a conference paper at ICLR 2026
For the second upper bound, apply Jensen's inequality with the concavity of the logarithmic function and obtain
1
li (3iz)
21 (1r2) g0(xz))
log K (0(22))
Σ
2 log
E (21 (x2)))))))"00000000
log
x (g
|| 10) -
K
2 log
P(u ll 90) - +
Here, x² l1 x) E denotes the chi-squared divergence between distributions p and The last incquality follows since x² go) 2 e(a ll qo) Rearranging the incquality proves the desired bound.
G.3 PROOF OF PROPOSITION S
To prove this proposition, we need the following lemma. The definition of the generating function g of differentiable loss function À is given in Eq. (1T) in Appendix E.T Recall that the definition of the induced loss function A for convex function V is in Eq. 8
Lemma 24. v is twice differentiable, g
(n) 0.
Proof of Proposition By Lemma 24 To Mr (p) is a perspective of the function p V (p). fA must be (strictly) concave if V is (strictly) convex. Hence, by Theorem 12 we conclude that A v is (strictly) proper.
we have g (n) Further, since ry + -fa² (n)
We now prove Lemma 24
Proof of Lemma 24
Consider
tr (n)
2 (n)
7=
dm:
==0
Note that Psr = To/ for x' = -1..., M, we have
Op.
Pz!
0
On:
1 1(==''
2 -1 l.....,.. M.
Case =. If = 0, A8 (1) = (V), p) (p)). Hence, we have
OAW (n)
Op. δ (VV (P)) Oyz Op
Orjo
Pz' (p)))
1
((p, (2 (V2)())))))
(16)
770
If 1 = - (V)) and thus
(x)(n)
Op 0 (V(
Omo
dn Opz
Op:
M
Pst 8 (P(P)
170
Ap: Op.
x'
(p)))
(17)
no
26

---

Published as a conference paper at ICLR 2026
From 16 and 17 have
M
(n)
0 (n)
= Σ
11= Ono
2 0
M
- V ( (ViV(D))))) (V)(())))
1/0
(p. Vi. + Σ (v
0.
Case 2: 2: M.
If 0, we have
0ND (n)
1 (p)):)
(18)
Om:
170
If f M,
0x (7)
0 (v(
On:
On: 0p:
0p: (P))
On: 0p Op:
(p)
(19)
0p Op:
Therefore, from 18 and 199 we have
(A)(1)
(1) Σ 1)=
Ōn=
==0
M
(
(V2((p)p); Σ
0p- 0p:
2=1
= (V²D(P)))= (V 1V (p)p))
Hence, we conclude that gx (n) = O.
G.4 PROOF OF THROREM 6
We note that, while the following proof self-contained, a more detailed technical discussion on the general relationship between proper scoring rule and Bregman divergence minimization in Appendix E.4
Proof of Theorem
Note that can write
C (mo) (n"" Ep(r)x)) (n (21:k). X (7o (n" (In:), X (f1x)))
(x)((()))))
Now, it easy to check that, we have
(n" A A = =no -nō(-W(p))) - - -V -V pul)
In particular,
(n',>" (p")
Hence, we have
( - (n" > (nn)) Bu (p" .P))
27

---

Published as a conference paper at ICLR 2026
From this expression, we have
(no) (n² Bp(eek) 110 By Bu (p=p(p;,) po)
=Bp(er pf2 (|TI (0" po)
p(= = O))p(exxx===))) By (p" po)
Since p(= = 0) = K+y and p(x1x)2 = 0) - go (20)))))((2)) -.. (0((xx))) by definition, this concludes the proof.
H EXPERIMENT DETAILS
This section provides the details on the experiments in the main text. All implementations are based on PyTorch and all experiments were performed on single NVIDIA GeForce RTX 3090. Our code to reproduce the results is made publicly available at https://gi thtps://///ttbb...... htwp:/tw.w//ooooggaaa///ooooooee
H.1 MI ESTIMATION
We conducted series of mutual information (MI) estimation experiments across three distinet data modalities; synthetic Gaussian variables, image-based representations from MNIST, and text embeddings derived from the IMDB dataset, using the standardized mibenchmark framework (Lee & Rhee 2024). Each experiment paired a 10-dimensional synthetic source variable X E R0 with a modality-speciffe target variable Y, varying in dimensionality depending on the data type. Across all experiments, we used consistent training configuration: models were optimized using Adam with a learning rate of le-4, trained in stepwise mode for 20,000 iterations.
Across all setups, we evaluated a fixed set of mutual information estimators, including NWJ, NWJ- Plugin. JS, JS- JS-Plugin, InfoNCE, InfoNCE-Anchor, Density Ratio Fitting, and Spherical, with both joint and separable critic types. The critic network in all cases was an MLP composed of two hidden layers with 512 units, ReL.U activations, and no normalization or dropout layers. Critic architectures projected inputs into shared 16-dimensional embedding space. For joint critics, X and Y pairs were concatenated and passed through a single encoder, whereas for separable critics, independent encoders g(x) and hzy) were used. We used batch size 64 for all experiments except the ones in Figure 3 We refer the readers to (Lee & Rhee 2024) and their codebase for the rest of the details including the data generation mechanism.
Figure 3 summarizes the result of MI estimation for the Gaussian experiment with cubic transforma- tion with varying batch sizes. It clearly shows that InfoNCE-anche exhibits consistent performance, but we note that JSplugin also performs remarkably well in this simple benchmark.
H.2 PROTFIN INTERACTION PREDICTION
We followed the same setup of Gowri et al. (0024), and here we briefly overview the essential part. We conducted experiments on two datasets derived from ProfTrans5-encode protein embeddings: one composed of 22,229 kinase-target pairs and another with 1,702 ligand-receptor pairs. Each protein is represented by 1,024-dimensional vector, and all embeddings were whitened and clipped to the range [-10, 10). Across 20 trials, 170 proteins were randomly selected and held out per trial, ensuring that no interaction in the training set included any of the held-out proteins. The remaining interactions were used for training a mutual information estimator.
Our approach trains a separable critic network to estimate the density ratio via the InfoNCE-anchor objective. The critic architecture is a MLP with 4 hidden layers, each containing 256 units, and outputs 32-dimensional embeddings for each input protein, separate encoders f(x) and g for each side of the pair. ReLU activation was used, and no normalization layers were applied by default. We used the Adam optimizer with a learning rate of le-4, batch size of 64, and 10,000 training steps. We implement early stopping with patience of 500 steps, based on validation loss, which is monitored every 500 iterations. The final model is selected based on the best validation performance and is then used estimate pointwise mutual information (PMT) for held-out protein pairs.
28

---

Published as a conference paper at ICLR 2026
We present ROC curves (Figure and histograms of learned PMI values (Figure for each estimator. These two figures clearly demonstrate that InfoNCE-an exhibit the best discriminative power.
H.3 SELF-SUPERVISED REPRESENTATION LEARNING
Here, provide details the objective functions considered in the experiment. We used the temperature parameter т = 0.2 throughout, unless stated otherwise.
InfoNCE: Log score, K B
0, PMI factorization.
InfoNCE-anchor: Log score, K- = B 1, == 1, PMI factorization.
JS: Log score, = v 1, PMI factorization.
Spherical: Spherical score, K B 1, 1, PD factorization.
x²: Asymmetric power score with a = 2, K = 1, V = 1. In this case, 7 0.1 was used.
We found that the PMI factorization was not effective for all scoring rules other than the log score. The rest of the experimental details can be found from the codebase of da Costa et al. (2022).
To confirm that the observation is not specific the ResNet backbone, we conducted additional representation learning experiments using two different ViT-hase and ViT-small backbones; see Table The reported numbers are top-1 accuracies on the validation set. These results support consistent conclusion that the anchor does not improve the quality of learned representation. In the revision, we will also add experiments on ImageNet to verify the trend.
Table 8: Linear probing top-1 accuracy (%) CIFAR-100 across different backbones.
InfoNCE-anchos
Backbone
InfoNCE
V=1 1
1024
R8sNet-18
65.98
65.74
ViT-small
63.35
62.33
62.32
ViT-base
67.57
65.22
65.15
H.4 IMPACT OF THE ANCHOR PARAMETER v
In the experiments in Section 4 we assumed v = by default. This setting is robust and consistently outperforms standard InfoNCE without any tuning. However, represents the relative weight of the anchor class, we analyze its impact to provide practical guidance.
Stability in MI Estimation. As shown in Figure 6 the estimator is highly stable across wide log-scale range of v. While the optimal v varies by dataset (e.g.. higher for Gaussian, lower for Text), the estimation error remains low throughout the sweep. This confirms that the method is not fragile to hyperparameter selection.
Gains in Downstream Tasks. Table shows that specific tasks benefit from larger values. For Kinase prediction, increasing v improves the AUROC from 0.81 to 0.85. This demonstrates that better MI estimation through the anchor modification directly translates to improved performance in specialized downstream applications.
Table Protein Interaction Prediction performance (AUROC) for varying anchor parameter v.
Dataset 2
1/ = 22
2
2
V/
2
Ligand
0.974.013
4.18
0.98 012
0.98 4.09
0.98 198...00 0.84
Kinase
0.81
0.834.052
0.824.050
0.85..052
Practical Guidance. We recommend y/ = 1 as a robust starting point, but tross-validation with coarse log-scale sweep (e.g., 29 to 2 20) can be performed for a fine- iin-ggrined tuning. In our SSL experiments, linear probe accuracy remained stable regardless of v, suggesting that the anchor's primary benefit is for tasks requiring precise density ratios.
29

---

Published as a conference paper at ICLR 2026
WVV
J5
Milaago
Boage
x Jupp (DHR)
InfORNCE
Sphenical
InfoRiE anchor
Inger
116
106
116
1oc 156
200
10 194
D0e
16c 206
126
14
251
(a) Gaussian (cubic) with batch size 16.
WWO
15
Ppage
(DWP)
InfoRCCE
Sphenical
InfoWCE anchor
lngen
316
306
Le 1h 226
50 35a
the the
(b) Gaussian (cubic) with batch size 64,
15
Pyage
(DW)
InteRnEE
Spherical
InfoNicbanncher
300
18c 1bh
206
58
91
10k 15t
(c) Gaussian (cubic) with hatch size 256.
Figure 3: Summary of MI estimation results on the standard benchmark on the Gaussian cubic data, with different batch sizes.
(0.74-0008)
(0.77700.08)
x (DRF) (0.7610.00))
Spherical (0.73+0.07)
1.0
PR
TPR
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.30 FPR
6.75
1.00
0.09
0.25
050
0.35
1.00
0.00
0.25
0.50
0.75
1.00
FAR
FPR
FPR.
FPR
(a) Kinase
0.004)
(0.95=0.02)
x²
Spherical (0.87=0.05)
InfoNcE-anchar (0.97=0.01)
1.0
0.8
177
TPR-FRR
TRR
TP8
7PA
0.00
0:25
0.50
0:75
1.00
0.00
0.25
050
0.7
100
0.20
0.25
0.50
0.75
1.00
0.00
0.25
0.50
3.75
3.00
0.00
0.2
0.0
0.75
1.00
19R
FM
199
(b) Ligand
Figure 4: ROC curves from different estimators.
30

---

Published as a conference paper at ICLR 2026
InfoNCF-anchor
0.8
label1
lobc1
-00
-5
10
-10
-5
10
-10
10
-10
-5
10
-10
-5
(a) Kinase-target pair prediction.
LMI
JS
(DRF)
Spherical
InfoNCE ConicE--nchor
0.4
-20
-20
-20
PM
PM1
(b) Ligand-receptor pair prediction.
Figure Histograms of pointwise MI (log p(a.)) from different estimators. p(a)p(y)
Mean Absalute Devistion (MAD)
Image
0.75
0.35
0.200
0.07
0.5
0.205
0.40
0.70
0.200
0.05
0.25
0.195
0.04
0.35
0.60
2
2
2/4
21
2/
2/r
Anchor parameter
v
Anchor parameter
Anchor parameter
Figure 6: Ablation study the anchor parameter
different datasets for MI estimation. The
estimator remains stable across a wide range of v, with optimal values being dataset- dependent.
31


Reviews:
[{"review_id": "UGvDXsem3j", "reviewer": "Reviewer_g8Nj", "summary": "**The paper critiques InfoNCE as a biased mutual information estimator due to its loose bound on KL divergence, introduces InfoNCE-anchor with an auxiliary class for consistent density ratio estimation, generalizes via proper scoring rules to unify contrastive objectives, and shows empirically that anchor enhances MI accuracy but not representation learning.", "questions": "1. How does varying the anchor parameter $\\nu$ affect finite-sample bias-variance tradeoffs across different $K$ values, especially near degeneracy points?\n2. Why do scoring rules beyond log score not improve representation learning despite the unified framework?\n3. How do the proposed estimators compare to recent unmentioned methods like those in Gowri et al. (2024) on MI tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written with precise formulations.\n2. The theoretical analysis provides a sharp upper bound on InfoNCE via $K$-way JS divergence, clarifying its high bias. \n3. The unified framework through proper scoring rules elegantly connects NCE, InfoNCE, and f-divergence variants.", "weaknesses": "1. Theorem 2 assumes known distributions $q_1$ and $q_0$ for equality conditions, but in practice with neural critics of finite capacity, the proportionality $r_θ$ ∝ $\\frac{q_1}{q_0}$ may not hold, leaving gaps in how approximation errors affect the bound's tightness.\n2. Critique of α-InfoNCE in Section 3.3 claims a proof flaw without supplying a counterexample or alternative derivation.\n3. The extension to proper scoring rules claims consistency for class probability estimation, but this paper does not derive explicit conditions ensuring all rules yield unbiased density ratios beyond the log score case, potentially limiting generalizability.\n4. The critique of existing InfoNCE variants (e.g., MLInfoNCE) is theoretically sound but lacks empirical validation, such as comparisons in MI estimation accuracy, which would strengthen the argument for InfoNCE-anchor's superiority.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7LnLuwNOby", "reviewer": "Reviewer_mNC7", "summary": "This paper revisits the InfoNCE objective, widely used in contrastive learning and mutual information (MI) estimation, and argues that it is not a reliable MI estimator due to its connection to a K-way Jensen-Shannon divergence rather than the KL divergence underlying MI. The authors propose a simple modification, InfoNCE-anchor, which introduces an auxiliary anchor class to enable consistent density ratio estimation without arbitrary scaling factors. They further generalize this to a framework based on proper scoring rules, unifying various contrastive objectives like NCE, InfoNCE, and f-divergence variants. Empirically, the approach achieves state-of-the-art MI estimation across Gaussian, image, and text benchmarks, and improves downstream tasks like protein interaction prediction. However, it does not enhance representation quality in self-supervised learning on CIFAR-100, suggesting that accurate MI estimation is not the key driver for contrastive representation learning success.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The analysis of InfoNCE's limitations is sharp and well-motivated, with a tight bound on its divergence (Theorem 2) that clarifies why it underestimates MI even for large K. The anchor modification is elegant and directly addresses the identifiability issue in density ratio estimation (Theorem 3). The generalization to proper scoring rules is a nice unification, recovering existing methods as special cases while providing a principled decision-theoretic foundation.\n2. Strong results in MI estimation benchmarks, where InfoNCE-anchor consistently shows low bias and variance compared to baselines like NWJ, JS, and DRF. The protein interaction prediction experiment is a practical application, demonstrating real-world utility with improved AUROC.\n3. By decoupling accurate MI estimation from representation learning benefits, the paper challenges common interpretations in the field and shifts focus toward structured density ratios or pointwise dependence.", "weaknesses": "1. While the SSL experiments are thorough, they are restricted to CIFAR-100 with a ResNet-18 backbone. It would be valuable to test on larger datasets or architectures (e.g., ViTs) to confirm if the lack of improvement holds more generally.\n2. The choice of ν=1 is defaulted without extensive tuning; sensitivity analysis (e.g., ν vs. performance) could reveal trade-offs, especially since asymptotic behavior links ν/K to bounds like DV/NWJ.\n3. The anchor introduces an extra term, potentially increasing compute for large batches. A brief discussion on efficiency relative to vanilla InfoNCE would be helpful.\n4. In SSL, did you observe any differences in learned representations (e.g., via CKA similarity or uniformity/alignment metrics) between InfoNCE and InfoNCE-anchor?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qNJOlMEsn2", "reviewer": "Reviewer_TDDy", "summary": "This work revisits the theoretical foundations of the InfoNCE objective, arguing that it is not a valid mutual information estimator. The authors propose InfoNCE-anchor, a simple modification introducing an auxiliary anchor class that yields consistent density ratio and MI estimation with reduced bias. This work further generalizes this approach via proper scoring rules, unifying several contrastive learning objectives under one framework. Empirically, InfoNCE-anchor provides the most accurate MI estimates but offers no improvement in downstream self-supervised tasks.", "questions": "- The paper reports that introducing the anchor does not improve downstream task performance, despite yielding more accurate MI estimates. Could the authors elaborate on the possible theoretical reasons for this discrepancy?\n- From an empirical standpoint, could the authors provide more analysis or visualization (e.g., representation similarity, clustering quality, or linear probe performance) to support the claim that contrastive learning benefits primarily from structured density ratio learning rather than accurate MI estimation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This study presents a theoretically grounded method to enhance existing mutual information (MI) estimation techniques and provides empirical evidence demonstrating its effectiveness.\n- The proposed “anchor” modification is straightforward yet addresses a subtle theoretical issue in density ratio identifiability.\n- The MI estimation experiments are comprehensive and show consistent advantages of the proposed method across different domains.", "weaknesses": "- Although theoretically neat, the proposed modification brings no tangible improvement to representation learning — arguably the main motivation for contrastive objectives.\n- Only a few relatively simple contrastive methods are considered; comparison with modern frameworks would strengthen the practical side.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)