Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning
Authors: Reuben Dorent
Polina Golland
William M Wells
Venue: NeurIPS 2025 poster ()
Keywords: mutual information
f-divergence
joint range
representation learning
Primary area: probabilistic_methods
TL;DR: A new lower-bound for representation learning connecting Jensen–Shannon and Kullback–Leibler divergences.

Abstract: Mutual Information (MI) is a fundamental measure of statistical dependence widely used in representation learning. While direct optimization of MI via its definition as a Kullback-Leibler divergence (KLD) is often
intractable, many recent methods have instead maximized alternative dependence measures, most notably, the Jensen-Shannon divergence (JSD) between joint and product of marginal distributions via discriminative losses. However, the connection between these surrogate objectives and MI remains poorly understood. In this work, we bridge this gap by deriving a new, tight, and tractable lower bound on KLD as a function of JSD in the general case. By specializing this bound to joint and marginal distributions, we demonstrate
that maximizing the JSD-based information increases a guaranteed lower bound on mutual information. Furthermore, we revisit the practical implementation of JSD-based objectives and observe that minimizing the cross-entropy loss of a binary classifier trained to distinguish joint from marginal pairs recovers a known variational lower bound on the JSD.
Extensive experiments demonstrate that our lower bound is tight when applied to MI estimation. We compared our lower bound to state-of-the-art neural estimators of variational lower bound across a range of established reference scenarios. Our lower bound estimator consistently provides a stable, low-variance estimate of a tight lower bound on MI. We also demonstrate its practical usefulness  in the context of the Information Bottleneck framework.
Taken together, our results provide new theoretical justifications and strong empirical evidence for using discriminative learning in MI-based representation learning.

Connecting Jensen-Shannon and Kullback-Leibler Divergences: A New Bound for Representation Learning
Reuben Dorent Inria reuben. dow.endirnoannni........
Polina Golland MIT polina@csail.mie mit.iidummit...u
William Wells III Harvard, MIT sw@bwh. harvard edu
Abstract
Mutual Information (MI) is fundamental measure of statistical dependence widely used in representation learning. While direct optimization of MI via its definition a Kullback-Leibee divergence (KLD) is often intractable, many recent methods have instead maximized alternative dependence measures, most notably, the Jensen- Shannon divergence (ISD) between joint and product of marginal distributions via discriminative losses. However, the connection between these surrogate objectives and MI remains poorly understood. In this work, we bridge this gap by deriving a new, tight, and tractable lower bound on KLD as a function of JSD the general case. By specializing this bound joint and marginal distributions, we demonstrate that maximizing the JSD-based information increases a guaranteed lower bound mutual information. Furthermore, JSD-based objectives and observe that minimizing the cross-entropy loss of a binary classifier trained to distinguish joint from marginal pairs recovers known variational lower bound on the JSD. Extensive experiments demonstrate that our lower bound tight when applied MI estimation. We compared lower bound to state-of-the-ar neural estimators of variational lower bound across range of established reference scenarios. Our lower bound estimator consistently provides a stable, low-variance estimate of a tight lower bound on MI. We also demonstrate its practical usefulness in the context of the Information Bottleneck framework. Taken together, our results provide new theoretical justifications and strong empirical evidence for using discriminative learning in MI-based representation learning.
revisit the practical implementation of
Introduction
Estimating and optimizing Mutual Information (MI) is central to many representation learning frameworks [8]8 R. The MI between two random variables U and V is defined as the Kullback- Leibler Divergence (KLD) between their joint distribution and the product of their marginals,
[U = DRulPUv || Pu @ pv]
(1)
Mutual information serves as natural measure of dependence between U and V. In representation learning, common goal is learn compact yet informative representations by maximizing the MI between encoded variables.
A popular class of methods for estimating MI involves learning discriminator that distinguishes between samples from the joint distribution and those from the product marginals. However, discriminator trained through separate optimization problem,
because the MI objective depends maximizing MI leads to a bilevel optimization problem, which intractable in practice. To bypass this challenge, tractable Variational Lower Bounds (VLBs) B 16 20 24 could be used, such the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Donsker- bound [3] to optimize a lower bound of the MI using data data-estimated expectations. Indeed, KLD belongs to the family of f-divergences, which have standard VLBs. However. VLBs are often unstable and underperforming in the context of representation learning g I8 13
In practice, many successful methods optimize alternative f-divergences, such the Jensen-Shannon divergence (JSD) [8] between the joint pvv and the product of marginals pV & pv, without knowing about the relationship between KLD and JSD. We refer this measure as (Ias),
Lus [U: = Dis(Pov || PU pv].
While optimizing JSD has been shown to reach competitive results, the reason why optimizing JSD-based dependence measure would increase standard MI, defined via KLD, remains unclear.
In this work, we bridge this gap by showing that maximizing JSD is equivalent to maximizing a lower bound on MI. Our main contribution new and optimal lower bound on KLD function of JSD,
E (Das (p || q)) DKL(P || q]
where E(-) is a strictly increasing function, analytically described below, that resembles Logit function. In particular, this result shows that maximizing [as also increases guaranteed lower bound on mutual information.
Furthermore, we revisit the practical optimization of LJs, We show that the f-divergence VLB for JSD is equivalent to optimizing the cross-entropy (CE) loss of specific discriminator. This discriminator trained distinguish between true pairs of data and false pairs generated by scrambling the true
pairs, a connection originally noted in the GAN paper [6]:
log 2 LCE S Dus(p || g)
While this bound is equivalent to the standard f-divergence VLB for JSD, we provide a new informition-ttheorriiic derivation that highlights the discriminator perspective. Beyond offering more intuitive interpretation, this formulation also makes explicit the gap due to suboptimal discriminators.
Together, these two results imply that minimizing the CE loss of a discriminator trained to separate joint from marginal samples leads to an increase lower bound on the mutual information:
(log 2 LCE) S E (Tus V)) I [U;
This work therefore provides a principled theoretical justification for existing representation learning approaches that maximize discriminative JSD-based losses to increase mutual information, and introduces a theoretically grounded alternative to commonly used variational lower bounds.
The rest of the paper is organized as follows. Section 2 reviews related work. Section 3 introduces 4 presents our new lower bound relating JSD and KLD,
the joint range of f-divergences. Section along with bound connecting cross-entropy loss and JSD. Section ( provides empirical evidence of the bound's tightness and its utility for representation learning tasks. Section cococlus this work.
2 Related work
Estimating Mutual Information: MI core concept in information theory, but its estimation remains notoriously difficult, particularly in high-dimensional or continuous settings with limited data. Traditional estimators can be broadly categorized as parametric or non-parametric. Parametric methods rely on strong distributional assumptions (e.g., Gaussianity), which often do not hold in practice. -orammraett approaches, such as histogram binning, k-nearest neighbors [10], and kernel density estimation [15] [II. assumption frre but suffer from poor scalability with respect to data dimensionaliiy and sample size, limiting their applicability in modern machine learning contexts.
To overcome these limitations, recent work has introduced variational lower bounds (VLBs) on MI that leverage the expressiveness of deep neural networks. key example MINE [3, which leverages the Donsker- Varadhan (DV) representation of the KLD to train neural network to maximize a lower bound on MI. A related estimator, NWJ [16), arises from the dual representation of /-divergences and shares similar objectives. However, theoretical work by McAllester & Stratos [14] shows that such estimators are fundamentally upper-bounded by log(n), where is the number of data samples. Moreover, both MINE and NWJ exhibit high variance, in part due to unstable estimation of the

---

partition function. To tackle this, Poole et al. (20] introduce hybrid VLB methods that generalize existing methods to trade off bias and variance. SMILE (4] further improves upon MINE by clipping the density ratio, which helps reducing the estimator variance. It also introduces a consistency check that exposes weaknesses in existing VL.B estimators, especially in high MI regimes, and highlights that optimizing VLBs does not necessarily imply an accurate estimation of a lower bound of MI.
Another lower bound approach is InfoNCE, introduced via contrastive predictive coding (CPC) [88 InfoNCE considers MI estimation as multi-way classification problem, producing stable and low- variance estimates. However, this estimator is biased and upper- upper-bounded bounded by log(b), where b the batch size, which limits its ability to capture high MI values.
Beyond VLB methods, another class of estimators decouples the estimation and optimization stages. NJEE 22 extends theoretical work by McAllester & Stratos [4] and estimates MI through entropy subtraction using neural density models. In contrast, DEMI K and PCM (26) reformulate MI esti- mation as a binary classification task: a model is trained to distinguish paired from unpaired samples, and discriminator outputs are then mapped to MI estimates. Interestingly, SMILE (24) can also be interpreted within this discriminator-based framework. As pointed out in [12.. its implementation! relies on maximizing an estimator of the JSD 17. with MI subsequently estimated from the outputs of a clipped discriminator. Overall, by disentangling optimization and estimation using a two-step procedure, these methods alleviate some instability issues of VLBs. More recently, Letizia al. [12] extended this setup by incorporating general f-divergences and empirically observed that the JSD (referred as GAN-DIME) provides the best tradeoff between bias and variance, outperforming alternatives based on KLDD or Hellinger distance with relatively low-variance MI estimation. While these two-step estimators are currently among the most accurate MI estimates in practice, they require retraining whenever the underlying joint distribution changes. This limitation is particularly restrictive in representation learning, where the latent variables evolve during optimization. As a result, these methods cannot be reused as plug-in MI estimators during training. In contrast, our bound can be directly optimized via JSD, using a single discriminator, enabling stable, end-to-end training.
Maximizing Mutual Information in representation learning: Several recent methods use MI maximization as principle for representation learning via Information Bottleneck [25]. VLB- based methods such as MINE and NWJ have also been applied to representation learning. although their practical limitations, such as high variance, make them less attractive compared to contrastive approaches g [8] Alternatively, contrastive predictive coding (CPC) [18] and its InfoNCE objective have shown strong empirical results across domains such as audio, vision, and NLP. However, since the estimation is upper-bounded by log(b), where b is the batch size, it requires a large batch size, which may be impractical computationally.
Another popular approach is Deep InfoMax from Hjelm el al. 8 which shifts the focus from strictly maximizing MI maximizing other measures of statistical dependence, particularly the JSD-based objective Las. Unlike KLD- bounds used in MINE or NWJ, the JSD is symmetric and bounded, often resulting in more stable optimization, and, unlike CPC and MINE, does not require a large batch size. Deep Info leverages a HSDD-based MI estimator introduced by Nowozin et al. [17]. To support the use of JSD-based optimization objective, Hjelm et al. g empirically show that for randomly sampled discrete sparse distributions, I correlates well with true MI, suggesting it can act as a meaningful proxy. In this work, we go further and provide a theoretical justification: we show that maximizing las provably increases lower bound on the true mutual information.
Jensen-Shannon divergence in GANs: The link between JSD and discriminator training has its roots in Generative Adversarial Networks (GANs), where Goodfellow et al. (6] showed that, under an optimal discriminator, the GAN objective minimizes an affine transformation of the JSD between the real and generated data distributions. This was later generalized in the f-GAN framework (I7). where GAN training corresponds to a variational lower bound on a general f-divergence, estimated via a learned discriminator. Chen et al. [4] proposed InfoGAN, which incorporates MI maximization within the GAN framework using a variational approximation originally introduced by Barber & Agakov 21. Rather than estimating the MI between joint and marginal distributions directly, InfoGAN targets the MI between a subset of latent variables and the generator's 's output, approximated using an auxiliary network and a variational lower bound from Barber & Agakov [2]. While InfoGAN brings iiformation-thhoreee principles into adversarial training, it does so a generative setting with a specific structural assumption on the latent space.
hnapss//gthhbbccmmmmmmogooognnaa wttp://///smmmmmmmmmiiiiiiee mi-estimator
3

---

.oint range of f-divergences
To establish novel lower bound mutual information, build on the concept of the joint range of f-divergences, which we summarize here. While this concept has been covered in classical information theory (27 7. our knowledge, it has not been utilized in the context of representation learning or mutual information estimation.
Let p and q be two probability distributions defined on the measurable space (X, F). For any convex function f (0, oo) - such that f(1) = 0, the f-divergence between p and q is defined as:
dp
(y(ll = 9
(6)
dg
Common examples include the Hellback-Leiblee divergence, total variation distance, and Jensen- Shannon divergence, each corresponding to a different choice of f.
Given two such convex functions f and g, the joint range of the pair (Dj; Dg) is the of all achievable values:
Rí. ((D) Py(plla) : P),
where P denotes the set of all probability distributions on some measurable space. In addition, the joint range over all k-dimensional categorical distributions [A] is defined as:
Rk:f ((D)((()))))))) (p(ll) p. probability measures over [k]]
The joint range Rig has several useful properties; it convex, and the lower envelope of the joint range the optimal lower bound relating the two divergences.
Remarkably, to characterize the entire joint range Ry it is sufficient to restrict to binary distributions,
iee Bernoulli distributions 7 127].
Theorem 3.1 (Harremoës- Vajda, 2011 77).
Rig co (R2.f.g)
where co denotes the convex hull.
Maximizing discriminator cross-entropy increases a lower bound on Mutual Information
In this section, we use f-divergence properties, including the previously introduced joint range, to demonstrate that training discriminator to differentiatee samples from the joint and product of the marginal distributions leads to maximizing a lower bound of the Mutual Information.
4.1 General lower bound on Kullback-Leibe divergence by Jensen-Shannon divergence
We introduce here a new optimal lower bound on the KLD in terms of the JSD, obtained by analyzing the joint range of the divergences.
The KLD (DKL) and JSD (Dis) between two probability distributions p and g are defined as
DRL || q) = E
log p q
Dis(e 4] DRI. Ip || m] + + DKI |l m]
(10)
where m = (p + 4) a mixture distribution of and
To derive the tightest lower bound of KLD in terms of JSD, analyze the joint range of the two divergences, i.c., the set of all possible pairs (x, y) E [0, log 2) x R+ such that:
T Dis(p || q] and
P
(11)
for all pairs of distributions p, that are compatible with the divergences.
Following Theorem 3.1 by Harremoës & Vajda [7], this joint range be fully characterized by restricting to be pairs of Bernouli distributions compatible with JSD and KLD,
Das B(p) B(r) and y = (KL[B(p) || B(r)),
(12)

---

6(Sa)
0.6
DRulp [1
+(12)
S3
Ω
0.4
((S))
((S))
8.000
0.2
0.4
S2
0.6
0.8
1.0
Dashp |I
Figure I: Left: Parameter space of two Bernoulli distributions. Right: Image (in gray) of the mapping 6, showing the joint range of the Sensen-Shannon and the Kullback-Leible divergences. The red curve is new lower bound.
for (n, v) E 10 x (0, 1) ((O, 0), 1)), where B(0) is Bernouli distribution with parameter 6. We therefore interested in the image of the mapping & defined
o: (n, v) + (Das || B(v)), DxL[B(p) || B(²)))
(13)
from the unit square [0, 1] (0, 1) U {(0, 0). (1, (1,1) 1)) into R??
Exploiting the symmetry of the divergences:
DRL. [(() B(v) DxL. [ - p) || B(1 v)],
(14)
Das B(4) B(v) DJs [ μ4) || B(1 -)],
(15)
we restrict our analysis to the image (St) the lower triangle Ω ((,,)) E [0, x] (0, s n).
Let us analyze the behavior of o on the three edges of the triangle as shown in Fig.
1. Diagonal (S1 μ v):
DBs(B(n) 11 B(n) = 0. Thus, points on the diagonal map to the origin: d(u p) (0.
DKL (B()) || B(]) = 0,
2. Lower Edge (S2 : - 0):
(KI(B(R) I| B(O] + +oo for any μ > The JSD remains finite, points on this edge map (0, log 2) {o0}.
3. Right edge (S3 1): For this segment, we can obtain:
[BI,[(() || B(v) = log v,
(16)
Das [B() || B(x) = log 2 [oog( + +v vlog
(17)
1
As v + 0, DKL. (B(1) B(v) - +00, while Das (B(1) || B(v)) increases its maximum, i.e. log
We now present the main result, which shows that maximizing the JSD increases lower bound on the KLD and that this lower bound corresponds to the image Sz.
Theorem 4.1. The optimal lower bound (i.e., equality is achieved for family of distributions) on Kullback-Leible divergence established by the Jensen-Shannal divergence is:
(Dss [p || q)) DRL(p q)
(18)
where E : l0g + IR is strictly increasing function defined implicitly by its inverse,
E- h (11( B (exp (-y))},
(19)
where (eep((y)) is a Bernoulli distribution with parameter exp (-y) [0, 1/2). =-1 has closed-form expression related to Eq. 17 and defined in Appendix (41
5

---

In particular, this result holds when and respectively the joint and the product of the marginals, leading to novel lower bound information theory between Lss and MI. Theorem 4.J]s proof relies on a conjectured functional property, supported by extensive numerical evidence (see Appendix [.T].
Approximation:
Although =\t does not have closed-form expression, it can be approximated point- wise using numerical solvers (example in C.2) Alternatively, we introduce a smooth differentiable
approximation using the Logit function, L x log --z (details in Appendix C.3
2og 1.0))
(20)
4.2 f-divergence lower bound on JSD from discriminator cross-entropy
Having established that the JS-based information (Ias) provides a lower bound on the MI, we now show that, in turn, the Jas can be lower bounded using the expected cross-entropy (CE) loss of discriminator trained to distinguish between dependent and independent pairs. Therefore, optimizing binary cross-entropy classifier effectively increases a lower bound on MI via the inequality chain.
We adopt standard binary classification setup estimate the JSD between the joint distribution PVV and the product of its marginals pu @ pv. Consider the mixture model:
(U.V) Z = 1 ~ puv, (U, V) Z This defines joint distribution over (u, U, z), denoted p(u, v. z), with marginal over (u, v):
pu @ pv, with Z ~ B(122).
(21)
p(u, v) m(u, 2) = 2 (u, v) 2p (u)
(22)
Since the JSD is a f- diveergeeee, it admits a variational representation
expressible as:
Lss [U: V DusiPov Pu @ pv] = 2 max
E
[t]
|- log(2 - exp(t))
(23)
PUV
@pv
where t: u u x V + (-o0, log 2) is variational function. Now, define the following reparameterization,
f(u, = (og(2qq(= = 1 u, v)),
(24)
where go = = 1 v) is meant to approximate p(= = 1 u, v), the optimal classifier.
Substituting Eq. 24 into Eq.
23 Las [U: V 2 max log +
(details in B we obtain the following lower bound:
[log (o 1/u, v)] +
E Per 88p
[og([ go(2 1/u, e))]
(25)
nov
Thus, we obtain
Ls UV: VI log min (C)
(26)
where (C) is the expected binary cross-entropy loss between qe and the true posterior.
This bound is tight in the non- non-parametric limit (infinite data and model capacity). An equivalent result was shown in the context of GAN discriminators (6) and also be derived from 23 via the
reparameterization d(u, v) = IL (go (==1 u, v)). In Appendix B.3 we present an alternative proof that quantifies the gap between the JSD lower bound and the one obtained from cross-entropy.
Combining
26 and (8) we obtain our new VLB (C) on MI as a function of a discriminator CE:
(CE(0) ()) = E (log2 Lox (0)) < (Tus VI) < VU:
(27)
4.3 Estimating Mutual Information via discriminators
While we introduced a new VLB on MI (Eq. 27 that can be estimated from data, we now show that it also enables the construction of an estimator of the true MI, detailed below. Beginning with the joint model of Eq. (21). we derive p( (u v) and then p(2=1ju,n) From this, it is easy to show that
p(==0)u,v)
1[U: [] =
[L(p(== u,))
(28)
PUV
where L(-) is the Logit transform. Details can be found in Appendix B.4
This result suggests two-step estimation procedure: train discriminator qo to distinguish between joint and marginal samples; 2) estimate the MI by plugging the approximate go into Eq. (28) and
approximate expectations by data averages. This two-step strategy is equivalent to GAN-DIME [2]

---

Experiments
In this section, we evaluate the tightness of our proposed MI lower bound, derived from the JSD, and compare it widely used variational MI estimators. Our evaluation is three-fold. First, provide an empirical analysis of the lower bound in the discrete setting, where both MI and JSD can be computed exactly. This allows to assess the tightness of the bound independently of neural approximation errors. Second, we assess the quality of our bound as a variational objective function in neural estimation tasks, using synthetic benchmarks introduced in prior work (0 24 12] These include Gaussian and non-Gaussian settings designed to challenge both the bias and variance of MI estimators across regimes of increasing complexity and mutual dependence. Finally, we demonstrate the practical usefulness of optimizing JSD as a proxy of MI in the context of the Information Bottleneck (IB) framework.
5.1 Tightness of the JSD lower bound in the discrete setting
We begin by analyzing the proposed MI lower bound in controlled discrete setup where both the true MI and the JSD between the joint distribution and the product of marginals are tractable. This allows assess whether our general optimal lower bound, defined for arbitrary distributions, remains tight when specialized to the MI setting, i.c. restricted to joint and their product of marginals.
We construct parameterized family of joint distributions P(A) E R that interpolate between the independent and perfectly dependent cases. The marginal distributions are uniform categorical Po = Pv 1 k where k is the number of categories. The joint PLy distribution is defined as:
plv (1- avPu Pv + a diag(Pu).
(29)
where a E [0, 1] controls the strength of dependence. For each value of a, we compute the ground- truth MI DKL (Pla) with our bound E(Iss V]), where the transformation defined (9)).
UV |I Pu @ Py] and the JSD Das P || Pu @ Py]. We then compare [UU; V]
Figure?]shove the resulting trajectories as a varies from 0 (independent variables) to 1 (maximally dependent variables) across sizes i E (2, 3, 5000.- For any given JSD value x, observe that there exists a discrete distribution with JS- JS-based information 1aS & a whose true mutual information nearly coincides with the lower bound E(z). This empirically demonstrates the tightness of our proposed bound. This result is particularly remarkable as it suggests that the optimal lower bound between JSD and KLD introduced in Theorem 4.1, originally derived for arbitrary pairs of distributions, remains tight when restricted to mutual information estimation, i.e., when comparing joint distributions to the product their marginals. In Appendix [ we further show that there exists an infinite family of discrete joint-marginal distribution pairs that lie exactly our bound, thereby demonstrating that the proposed lower bound remains tight even under this restriction. This implies that our JSDD-based lower bound offers tight proxy when maximizing mutual information.
500
JSD lower boand
Figure 2:
Mutual infor- mation and its JSD-based lower bound E(Lis) for a parameterized family of dis- crete joint distributions with known MI and JSD, varying in dependence strength (a) and number of categories (k)) The presence of MIs near the lower bound across settings empirically demonstrates the tightness of our JSD-based estimate.
200
Number categories (log soale)
100
50
DEu DeLL (Po
3
20
2
10
0.5
Daso Po PV)

---

MINE
NWJ
CPC
JSD-LB
Gaussian
10
10
10
10
MI [note]
5
10000
200000
10000 Stepe
200000
100000
20000
10000 Stops
20000
Steps
Steps
10
10
Cubic
MI [nsts]
5
5
T0G000 Steps
20000
10000
20000
T00000
20000
10003
20000
Stepe
Steps
Steps
True MI
Estimated
VLD
In(b)
Our MI estin.
Figure 3: Staircase MI estimation comparison for d = 5 and batch size 6 = 64. The Gaussian case is reported in the top row, while the cubic case is shown in the bottom row.
MINE
NWJ
CPC
JSD-LB
10
10
asinh
M1 [nats
5
%
10000
20000
10030 Stepe
20000
0
10000 Steps
201000
0
10000 Steps
20000
Steps
Half-cube
10
10
10
10
MI [nste]
%
10000
20000
100.00
200.00
10000
210000
100000
20000
Steps
Stepp
Steps
Steps
Student
MI [nats]
2
2
10000
0
00
10000
0
100000
10000
Steps
Stege
Steps
Stegs
4
Uniform
MI nate]
2
1
1000O0 Steps
200000
200000
100000
20000
Stepe
Steps
Steps
True MI
Estimatedd
VLB
In(b)
Our MI estim..
Figure 4: Staircase MI estimation comparison with batch size 64 for the asinh (d = 5), half-cube (d 5). Student (d 5), and uniform distributions.

---

5.2 Complex Gaussian and non-Gaussian distributions
We next evaluate our lower bound in continuous domains where MI estimated from samples using neural networks. Our goal is to assess how tight the bound remains when used variational objective and how well it compares to other neural MI VLB under complex data distributions.
Setup and metries We follow the protocol of Letizia et al. 12]. who benchmark MI estimators in scenarios where the true MI is known analytically. For each estimator, we report the value of the lower bound objective and the ground truth MI. For our method, additionally report the estimated MI obtained using the procedure described in Section 4.3 and equivalent to GAN-DIME 112.
All estimators implemented using fully connected discriminator with input dimension 2d, two hidden layers of 256 ReLU units, and one scalar output. Training is performed for 4000 steps using the Adam optimizer and batch size N 64, matching the architecture and hyperparameters from prior work for comparability. As the functiio E is strictly increasing, maximizing E (log 2 LCg) is equivalent to minimizing the cross-entropy loss Lc Therefore, the approximation of Ent is not used during optimization. Implementation detail are available in Appendix E.2
We compare our approach against other differentiable variational lower bounds that, unlike two-step estimators, can be directly optimized as objective functions for mutual information maximization:
1. MINE 3
based on the Donsker- Varadhan dual representation of the KLD;
2. NWJ 16]
based on another KLD dual representation, which is unbiased but less tight compared to MINE;
3. CPC 18 based on contrastive predictive coding with the InfoNCE lower bound;
4. JSD-LB (ours), based on ICE from 27 with the estimated expected cross-entropy loss.
Gaussian benchmarks In the Gaussian setting, we define two random variables ~ AO, Ia) and ~ (O, Ia). sampled independently, and construct = p + /I- p² N, where p is the correlation coefficient. the eubic case, we apply a nonlinear transformation v + v to V. This transformation introduces non- Gan-Gausiian marginals while keeping the same mutual information. In both cases, the MI is known: I [U;; V1
log(1 - p²))
Figure presents results. In the Gaussian setting, JSD-LB consistently provides tight and stable lower bound on the true MI, outperforming MINE, NWJ, and CPC, especially in high-MI regimes. While MINE and NWJ exhibit high variance, JSD-LB and CPC remain more stable. In particular, MINE's instability can lead to significant overestimation its lower bound, sometimes exceeding the true MI, whereas CPC remains limited by the contrastive bound log(b). In contrast, the estimated JSD-LB remains below the true MI, as expected for a valid lower bound, yet closely tracks it. Furthermore, our derived two-step MI estimator from (28) shown in green in Figures 3 4 and equivalent to GAN-DIME [12), surpasses other variational lower bound estimators.
In the cubic setting, JSD-LB provides a tight lower bound, while maintaining low-variance estimates compared to MINE and NWJ. Moreover, our two-step MI estimator achieves the most accurate MI estimation, outperforming all other VLBs.
Complex Gaussian and non-Gaussian benchmarks We also replicate the more challenging benchmark from Letizia et al. [1]. following recommendations from Czyž et al. [3] using complex Gaussian transformations (half-cube and asinh, Fig and Con-Gaussian distributions (Student and uniform distributions, Fig/4). Across these settings (see details in Appendix E.2). our lower bound and its derived MI estimations remain competitive. with lower variance VLBs, confirming that the proposed JSD-based objective generalizes beyond Gaussian settings.
5.3 Variational Information Bottleneck Experiments
To assess the practical utility of our JSD-LB in representation learning, we evaluate it within the Information Bottleneck (IB) framework [ [9], which aims to lear compact yet predictive representations T from inputs for targets V. by maximizing [T; I[T;U], We reproduce the

---

Table 1: Generalization performance (%) on MNIST dataset. Performance is evaluated by the mean classification accuracy on the MNIST test set after training on the MNIST training set.
Method
VIB 1
NIB 9
squared-ViI VIB 21
squared-NIB 21
DisenIB 19
JSD-LB
Testing
97.6
97.2
96.2
93.3
98.2
98.8
Table 2: Adversarial robustness performance (%) on MNIST dataset.
Method
VIB [
NIB g
squared-VII VIB 21
squared-NIB 21
Disen 19
JSD-LB
E 0.1
74.1
75.2
42.1
61.3
94.3
99.5
Training
e 0.2
19.1
21.8
8.7
24.1
81.5
96.0
C 0.3
3.2
5.9
9.3
68.4
91.4
6 == 0.1
73.4
75.2
42.7
62.0
90.2
94.6
Testing
6 0.2
20.8
23.6
9.2
24.5
80.0
89.6
8 = 0.3
4.2
3.4
5.9
9.9
67.8
86.1
Table Distinguishing in- and out-of-distribution test data for MNIST image classification (%).
t (resp.. .) indicates that larger (resp., lower) value is better.
Method
VIB 1
NIB 9
squared- VIB [21
squared-NIB 21
DisenIB [9
JSD-LB
TPR95
27.4
34.4
49.9
47.5
0.0
0.00
AUROC
94.6
94.2
86.6
85.6
99.4
100.0
AUPR
94.8
95.2
83.5
83.3
99.6
99.9
AUPR Out t
93.7
91.8
83.2
83.3
98.9
99.9
Detection Err +
11.5
11.9
20.0
15.0
1.7
0.1
MNIST experiments of Pan et al. [19, where U denotes an image and V its label, using the same architecture and setup. our setting (JSD-LB), I [T; V] is optimized by maximizing the JSD via cross-entropy loss. Following Pan et al. [19, we assess generalization, adversarial robustness, and out-of-distribution (OOD) detection. Robustness is evaluated by computing the average classification accuracy on adversarially perturbed inputs, where pixel-wise pertubation are obtained using one-step gradient-based attack with perturbation magnitudes E {0.1, 0.2, 0.33. OOD detection is tested on synthetic Gaussian noise samples, using standard OOD detection metries including True Positive Rate at 95% (TP955), Area Under the ROC Curve (AUROC), Area Under the Precision- Reccal Curve (AUPR) and Detection Error.
Our results presented in Tables and show that replacing the MI estimator with JSD-LB leads to state-of- the-art art performance in terms of generalization, adversarial robustness, and out-of-distributie robustness. Specifically, outperforms other Variational Information Bottleneck objectives [I 9 21 and the Disentangled Information Bottleneck approach [19]. This set of experiments shows that maximizing JSD via the cross-entropy loss has practical value for representation learning.
Conclusion
In this work, we addressed a fundamental gap in the theory of representation learning: the lack of formal guarantees linking Sensen-Shannon divergence (JSD), widely used in practice, to Kull- back-Leibler divergence (KLD), the basis of MI. We derived a new and optimal lower bound on KLD as a function of JSD, and showed how this leads provable lower bound on MI when comparing joint and marginal distributions. Furthermore, we demonstrated that minimizing the cross- entropy loss of binary classifier distinguishing between joint and marginal pairs increases lower bound on the MI, unifying existing discriminator-bann based objectives in representation learning with an information- -ieioottiteerticccc interpretation. Through extensive experiments in both discrete and continuous domains, we showed that our bound is tight and also performs competitively variational objective and as an estimator practice. Moreover, we demonstrate usefulness representation learning within the Information Bottleneck (IB) framework, reaching state-of-the-an performance. Altogether, this work provides a rigorous theoretical foundation and practical method for maximizing mutual information through discriminative representation learning.
10

---

Acknowledgment
We thank the anonymous reviewers for their insightful comments and helpful suggestions, which improved the quality of this paper. We would also like to thank Yury Polyanskiy for pointing us to the joint range. R.D. received a Marie Sklodowska-Curie grant No 101 101244 (project: SafeREG), This work was performed using HPC resources from GENCI-IDRIS (Grant 2024- SafeREG, 2025-SafeREG). The research leading to these results has received funding from the program 'Investissements d' a'vveiir" of Health (P41EB028741 and RO1EB032387).
W.W. is supported by the National Institutes
References
[1] Alexander A. Alemi and Ian Fischer and Joshua V. Dillon and Kevin Murphy. 2017. Deep Variational Information Bottleneck. In: International Conference on Learning Representations.
[2] Barber, David, & Agakov, Felix. 2004. The IM algorithm: a variational approach to information maximization. Advances in neural information processing systems, 16(320), 201.
(3] Belghazi, Mohamed Ishmael, Baratin, Aristide, Rajeshwar, Sai, Ozair, Sherjil, Bengio, Yoshua, Courville, Aaron, & Hjelm, Devon. 2018. Mutual information neural estimation. Pages 531-540 of: International conference on machine learning. PMLR.
[4] Chen, Xi, Duan, Yan, Houthooft, Rein, Schulman, John, Sutskever, llya, & Abbeel, Pieter. 2016. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29.
[5] Czyž, Pawel, Grabowski, Frederic, Vogt, Julia, Beerenwinkel, Niko, & Marx, Alexander. 2023. Beyond normal: On the evaluation of mutual information estimators. Advances in Neural Informa- tion Processing Systems, 36, 16957-16990.
[6] Goodfellow, lan, Pouget-Abadic Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, & Bengio, Yoshua. 2014. Generative adversarial nets. Advances in neural information processing systems, 27.
[7] Harremoës, Peter, & Vajda, Igor. 2011. On pairs of f-divergences and their joint range. IEEE Transactions on Information Theory, 57(6), 3230-3235.
[8] Hjelm, R Devon, Fedorov, Alex, Lavoie-Marchildon Samuel, Grewal, Karan, Bachman, Phil, Trischler, Adam, & Bengio, Yoshua. 2018. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 18X8v:880..00 06670.
[9] Kolchinsky, Artemy, Tracey, Brendan D. & Wolpert, David H. 2019. Nonlinear information bottleneck. Entropy, 21(12), 1181.
[10] Kraskov, Alexander, Stögbauer, Harald, & Grassberger, Peter. 2004. Estimating mutual infor- mation. Physical Review E Statistical, Nonlinear, and Soft Matter Physics, 69(6), 066138.
[11] Kwak, Nojun, & Choi, Chong-Ho. 2002. Input feature selection by mutual information based on Parzen window. IEEE transactions on pattern analysis and machine intelligence, 24(12), 1667-1671.
[12] Letizia, Nunzio Alexandro, Novello, Nicola, Tonello, Andrea M. 2024. Mutual Information Estimation via f-Divergence and Data Derangements. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems.
[13] Liao, Ruizhi, Moyer, Daniel, Golland, Polina, & Wells, William M. 2020. DEMI: Discriminative Estimator of Mutual Information. arXiv preprint arXiv:2010.01766.
[14] McAllester, David, & Stratos, Karl. 2020. Formal limitations on the measurement of mutual information. Pages 875-884 of: International Conference on Artificial Intelligence and Statistics. PMLR.
11

---

[15] Moon, Young-ll, Rajagopalan, Balaji, & Lall. Upmanu. 1995. Estimation of mutual information using kernel density estimators. Physical Review E, 52(3), 2318.
[16] Nguyen, XuanLong, Wainwright, Martin J, & Jordan, Michael 2010. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 1), 5847-5861.
[17]
Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29.
Nowozin, Sebastian, Cseke, Botond, & Tomioka, Ryota. 2016. f-gan:
[18] Oord, Aaron van den, Li, Yazhe, & Vinyals, Oriol. 2018. Contrastive Predictive Coding. arXiv preprint arXiv:1807.03744.
Representation Learning with
[19] Pan, Ziqi, Niu, Li, Zhang, Jianfu, Zhang, Liqing. 2021. Disentangled information bottleneck. Pages 9285-9293 of: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35.
[20] Poole, Ben, Ozair, Sherjil, Van Den Oord, Aaron, Alemi, Alex, & Tucker, George. 2019. On Variational Bounds of Mutual Information. Pages 5171-5180 of: International conference on machine learning. PMLR.
[21] Rodríguez Gálvez, Borja, Thobaben, Ragnar, & Skoglund, Mikael. 2020. The convex informa- tion bottleneck lagrangian. Entropy, 22(1), 98.
[22] Shalev, Yuval, Painsky, Amichai, & Ben-Gal, Irad. 2022. Neural joint entropy estimation. IEEE Transactions on Neural Networks and Learning Systems, 35(4), 5488-5500.
[23] Shannon, Matt. 2020. Properties of f-divergences and f-GAN training. arXiv preprint arXiv:2009.00557.
[24] Song, Jiaming, & Ermon, Stefano. 2019. Understanding the Limitations of Variational Mutual Information Estimators. arXiv preprint arXiv:1910.06222.
[25] Tishby, Naftali, Pereira, Fernando C., & Bialek, William. 1999. The information bottleneck method. Pages 368-377 of: Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing.
[26] Tsai, Yao-Hung Hubert, Zhao, Han, Yamada, Makoto, Morency, Louis-Philippe, & Salakhutdi- nov, Russ R. 2020. Neural methods for point-wise dependency estimation. Advances in Neural Information Processing Systems. 33, 62-72.
[27] Yury Polyanskiy, Yihong Wu. 2024. Information Theory From Coding to Learning. Cambridge University Press. https://people..de lids www.eiieeeciyyoooooppeaeeaaooooo pxpprp
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The two main contributions of the paper are both included: new, optimal and computationally tractable lower bound on Kullback-Lebler divergence as a function Jensen-Shannon
of Jensen Shannon divergence; b) proof that maximizing the cross-entropy loss of discriminator leads to an increase in a lower bound on the mutual information.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The manuscript acknowledges that the derived E function does not admit closed form and must be approximated numerically. Furthermore, we clearly state in the main manuscript that the proof of Theorem 4. relies on a conjectured functional property, supported by extensive numerical evidence
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
13

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All the assumptions are included in the statements of Theorems, and Corollaries (that are cross-referenced), All the complete proofs are reported in Appendix B and C.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: A link to the code is available. In addition, the complete neural architectures are specified Appendix E.2, together with the optimizer type, hyperparameters, and the hardware details. Finally, we report the other VLB objectives in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
14

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code along with README file detailing how to run the ex- periments. The first set of experiments does not require any external data, as all experiments are conducted using samples drawn from known probability distributions. For the second set, the widely used MNIST dataset is used.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurIPS code and data submission guidelines (https coapss.cc/pullic/guu guidee/ Sow.edeueeoooeuummiicce iearbbmmssiooooooiioii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are written in Appendix E.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
15

---

Justification: In line with prior work mutual information estimators, our "staircase" plots display MI estimates over training steps. The shaded regions around the curves visually convey the variance of the estimators, providing intuitive understanding of their stability and statistical behavior during optimization. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details are given in Appendix E.2.5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: There are no potential harms caused by the research process and no potential harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
16

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is foundational research. Mutual information estimation and Jensen- Shannon divergence are used for various machine learning algorithms and there are no direct paths to negative applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
17

---

We cite all datasets used in our experiments, as well as the original papers from which we adapt code implementations (mainly f-DIME) of ofte-offthe-ar mutual information estimators.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, www.ssppppssseeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We fully disclose the code and provide a readme file an anonymized repository, describing how to use it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmdel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: he paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
18

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No large language models were used part of the methodology technical contribution of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy /tap:::///eurriie cc. fc//oonferencee/2225//// for what should or should not be described.
19

---

A Appendix: Related MI estimators
In this section, present some existing methods and their formulas for estimating MI. As mentioned in the "Related work" Section bounds and two-step (optimization then estimation) methods.
2
these methods can be categorized as differentiable variational lower
A.1 Other differentiable Variational Lower Bounds (VLBs)
In this subsection, we review some of the main Variational Lower Bounds (VLBs) on MI. These differentiable bounds can be used directly as minimization criteria in the context of representation learning.
The mutual information neural estimator (MINE) is VLB on MI derived from the Donsker- Varadhan dual representation of the KLD 120
MINE 3
= { (MINEE (0) =B ((.. [To (u, v)] log (p( (e (elolu.e))
(30)
where f are the parameters of the functions To 4 V R which are defined using a deep neural network. key limitation of MINE is that it considers the logarithm of the expectation, which leads to biased estimation when approximated using data samples.
NWJ 16
Another VLB based on the KLD dual representation:
U V sup [N) = ((,.))))ov((uo)) (To v)] ((u.)))p [ol(a,))11) "}
(
(31)
This VLB is not as tight as MINE, i.e. (0) JMINTE (0) I TU; but its estimation is unbiased.
CPC
This VLB, based on contrastive predictive coding (CPC), is defined as
To(ure)))
(f)c(() = (((.u))-prvvs((.)))
log
(32)
N
E eTo(uros))
1=1
where b is the batch size and pvvv denotes the joint distribution of i.i.d. samples drawn from puv. While CPC estimates have typically low variances, they are upper bounded by log b, which introduces bias and restricts its ability to estimate high mutual information.
A.2 Two-step MI estimators
Alternatvell,, another class of estimators has been proposed. Unlike the previously presented VLB, their estimation two-step. First, a neural network is trained to approximate a function that is later used for estimating the MI. These methods are, therefore, learning, as that would involve solving bilevel (nested) optimization problems.
easily applied representation
Since MINE suffers from high-variance estimations, Song & Ermon 24 proposed to clip the outputs of the neural network Tg, which estimates the log-density ratio, when estimating the partition function:
SMILE 24
[S = (u..o))pvv(aa)) [To (u, log (E(u,)) (wlpv [elip (etalue)) eT,, ,e,,) (33)
where the parameter 2 0 is hyperparameter and the clip function is defined as follows:
clip(v, (ip(v,L, 1, u) = max (min (v, u).l)
(34)
The SMILE estimate ISMILE is not a VLB but converges to the IMIEE VLB when 7 - oc.
In practice, the implementation of SMILE is significantly different from MINE. Indeed, Song & Ermon proposed to train the neural network To, by maximizing a -hannnn MI estimator from 7 in the Deep InfoMax procedure [8]. SMILE is, therefore, a two-step estimator.
20

---

NJEE [22]
The Neural Joint Entropy Estimator (NJEE) 22 estimates the MI between discrete random variables by modeling entropy estimation as sequence of classification tasks. Let U denote the m-th component of discrete random vector U E IRd, and define U as the subvector containing the first components of U. Let V be another random variable. Given a batch of samples, NJEE estimates the entropy of Ü by training series of classifiers. Specifically, for each m, two neural networks Gg.,, (Um U"=-1) and Gg (U' V, U?=- are trained to predict Un respectively from its preceding components Um-1, and from both V and Um-1, using cross-entropy as the loss function. Once trained, the NJEE mutual information estimator is computed as:
(U:E(U;V)- (UU(UI) + Σ CE (Gem. (Um Un--)))
Σ m=1
CE (Gem. (Um V,Um-!)), (35)
m==
where Hu(U1) 1) is the estimated marginal entropy the first component. This two-stage procedure is designed for discrete random variables and does not directly extend to the continuous case.
is an f-divergence based two step method that decouples the optimization of a deep neural network To and the estimation of the MI. Let / denote convex lower semicontinuous function that satisfies f(1) The Fenchel conjugate of f is denoted f" Specifically, the network T is trained to maximize the following quantity:
f-DIME 12
JT(T) = B(())) (u.0)) (Tuu - - (T (
(36)
and the MI then estimated as:
[oo (IF') (f(u.
L = E (u
(37)
In practice, Letizia et al. [2 proposed to use three functions that are related to the Kullback- divergence, GAN, and Hellinger distance squared:
JKL. - u log(u)
SGAN u + ulog u (u + 1) log(u + 1) + log 4
SHD: (Vu-1)²
Experimentally, they found that fGAN achieves the best performance. Importantly, the fGAN is closely related to the f function associated with the Jensen-Shannon Divergence defined as:
1 fJSD u - 2
u +
2 )
u log - (u + 1) log(
(38)
B Appendix: Proofs of theorems and derivations
B.1 Proof of Theorem 4.1
We begin by deriving the expression for the JSD between two Bernoulli distributions.
Lemma B.1. Let p B(μ) and q = (v)) be two Bernoulli distributions with parameters μ and Then, the Lensen-Shannol Divergence between and is given by:
1
V
Das B()) || B(r) log 2 + 2
log
+ v log
1
11 +
1 1 p
1 - v
+ 2
(1 4) log
+ (1-
2)
log
(39)
2 11
2 11 v
Proof. The Kullback-Leible divergence between two Bernoulli distributions p and g with parameters and is given by:
DKL/p |l ql p log / (1 11) log
-
(40)
v
21

---

Given that the mixture (p + q) is a Bernoulli distribution with parameter m defined as:
24g 2
the JSD is
1
Das(B(n)
|| B(²]
DkL D [B({)
|| B(m)] + =DLLIB(U) B(m)]
2
1
2p
+ (1 (4) log 2(1 14)
poog
2
1 + v
- μ - V/
2v
+ (1 - 11 log 2(1 - v)
+
log
11 + v
μ -
1
log 2 +
μ log
+ =ollog
2
1 + v
μ + V/
1 H
1 -
+
(1 μ1) log
+ (1 v) log
2
11 v
2 1 V
Lemma B.2. The function E-1 defined by
1 + [0, log 2)
(41)
- log 2 [11 + e ") log(1 + ve- ) + ye ye-
C strictly increasing, and satisfies
(0) 0,
lim (v) = log 2. +00
(42)
Hence, E-1 is a bijection from R_ onto [0, log 2).
Proof. The function E-1 is obtained by finite compositions, additions and multiplication of elemen- tary C functions. Therefore, E-1 E (R()),
Differentiating, we obtain
(E- = - - log(1 + F) (1 + v) eny
1te-u - - ye-
(43)
= = [log(1 + e²)]
(44)
Since e > O and log(1 + e²) 0 for all 0, it follows that (E-1)(v) > 0. Hence, E-1 is strictly increasing on I.
Moreover, as + +00 e + 0 and thus lim y y = (y) = log 2. At = 0, we have
-- = log 2 [ + log(2) + 0 = 0. By continuity and strict monotonicity, the image of - is therefore [0, log 2), establishing bijectivity.
Lemma B.3. The function E, defined as the inverse of - in Eq. is C and strictly increasing.
Proof. Since - is C with strictly positive derivative on R,, the inverse function theorem implies that E is C and strictly increasing on [0, log 2).
We will now prove the main result of this work.
We denote Ω the lower triangle Ω= ((, E [0, (0, v S u). Lemma B.4. Let the mapping be defined as the lower triangle:
&: (H,v)
(45)
where j and k are defined as follows:
م(m, v) = Das ((4) B(v) and k(p,v) = DKL ((H) || B(r)
(46)
Then, the Jacobian of the mapping & is given by:
0i
01
1 [1[(()) - I[mm)]
1/2 [[(v) L(m)]
J(H,v)
By
(H.v)
L(n) (L)))
1=
(47)
where IL is the logit function.
22

---

Proof.
Using Lemma
B.1
the partial derivative
Li is given by: 8u
0j
(1,
v)
1
14
1
1
+ log
1
2
11
1
+ V
11 +
1 + V
1
1
1
1 - P +
1 -
(1
p4)
log
2
1 - V/
/ - V
1 - V
1
11
11
1
log
log
1
+
2
1 +
11
# v
11 +
2
1 +
1
log
log
2
1 +++
μ V
1
/
log
log 1 m
2
1
[L(4) [[m)]
2
By symmetry of the JSD:
0j
(1,1) = [[((v) L(m)]
(48)
Oy
2
Finally, using Eq
40
0p (M, v) log 1a(1 v)
Ōk:
(49)
1(1 - HI
-1( - L(V),
(50)
and
Ōk (p, Ov
11 p
(51)
1 -
Conjecture B.5. The determinant of the Jacobian defined in Eq. (77) is negative on the interior Ω ((,,)) E 10 x (0, 2} of R2 i.e.: (p, E S det(J(H,,))) <0.
(52)
While we have not found an analytical proof of the conjecture, it can be verified numerically, as shown in Figure The negative of the determinant of the Jacobian is always positive.
Theorem B.6. The optimal lower bound on Kullback-Leibler divergence established by the Jensen- Shannon divergence is:
E(Das (p |l 4) & DEL(P || q)
(53)
where E: 10g + R+ + strictly increasing function defined implicitly by its inverse.
E-1 y (B)( B (exp (-y))},
(54)
where B (exp(-y)) is a Bernoulli distribution with parameter exp (-y) E 1J. =-1 has closed-form expression related to Eq. 17
Proof. Following the Theorem 3.1 by Harremoës & Vajda 77]. the joint range between the JSD and the KLD can be fully characterized by restricting p, be pairs of Bernoulli distributions compatible with JSD and KLD,
Das B(p) II B(v)] and y Dxt. ((() || B(v)), for (H, v) [0, (0, 1). 0), (1, 1)}, where B(8) is Bemoulli distribution with parameter 6. We are therefore interested in the image of the mapping & defined as:
(55)
(p,v) (Bis[B(n) B()), DRL Dku (B()) || B(x)))
(56)
23

---

103
10
0.8
det (J (H, v)) (log scale)
10
0.6
10-3
10
0.4
10
0.2
10
10-9
0.2
0.4
0.6
0.8
μ
Figure 5: Visualization of the negative of the determinant of the Jacobian. We can observe that is
constantly positive, meaning that the determinant of the Jacobian is always negative.
from the unit square [0, 1] (0, 1) {(0, 0), (L, 1)} into R
Exploiting the symmetry the divergences:
(Rr,[(()) B(r) Dxr. [B(1 - 11) || B(1 v)]; Das[B(n) (v)) Das [B -4) 14) || B(1 v)],
(57)
(58)
we restrict our analysis to the image ((?) of the lower triangle Ω {(p, E [0, [] S p).
Using Conjecture B.5 the determinant of the Jacobian is negative for all inner points in P. According
to the open mapping theorem, the inner points are therefore mapped into interior points of the range. To determine the lower envelope of the image, 5Ω (S1, S2, 5z). Given that, S₁ and S2 respectively map to a single point (0, 0) and a horizontal line y 0o, Sy defines the lower envelope of the range and is, therefore, our bound.
examine the image of the boundary
B.2 Proof of Eq. (26)
The variational representation 17 27 of can be expressed as:
Lus [U: V] DJs(pov PU @ pv] = 1 max
1
log - exp(t))
(59)
2
PUv
8pv
where 4:44 x V (-o0, log 2) is variational function. Now, define the following reparameterization,
t(u, v) - log(220 = 1 k),)),
(60)
where go(z 1 u, v) is neural network.
24

---

Substituting 660 into (59].
we obtain 25
1
Jas IU; V
max
E Puv
[og(220 (2 1 u, [[)]
E @pv
|- log(2 - 2go(z 1 u))1
(61)
2
t
1
max
log
4 +
E PUV
Dog go = v] +
E [log(1 go(= = ||,
(62)
2
(63)
The expected eross-entropy (CE) under our classifier model in Eq. (1 is defined as:
LCE
E
|- log go w)]
log qo (2|u,, wi]]
(64)
(u,,,)))Pvv%
zmpp (u,v))puviz
or, expanding the expectation over 2,
LCE
E
I- 1og 1|u, u)] (u,v)~pu 8pv
E
[- log qo 0/u, u)]
(65)
(u.v)~puv
With Eqn. 62 obtain
J [U:V] 2 log 2 - min (Cn(
(66)
B.3 Alternative to Section 4.2 to obtain (26) and gap quantification
Here, we revisit the relationship between Jensen-Shannon divergence and the cross-entropy training loss. In our main manuscript, the relationship derived using properties of f-divergences. The following derivation is more discriminator-oriene and does not use f-divergence properties directly.
We use a standard binary classification setup to estimate the JSD between the joint distribution pov and the product of its marginals pa @ pv. Consider the mixture model:
Alternative proof to 26). Recall the classifier model of Equation 21
(U, V) Z ~ puv v), (U.V) Z ~ pu @ pv. with Z B(1/2).
(67)
This defines joint distribution over (u, U, 2). denoted p(u, u. =), with marginal over (u, v):
p(u, v) m(u, w) = Luv v) 1 (u).
(68)
Next we study the MI among (U, V) and Z under that model,
[(U V); 2| = Duu ll puv @ pzl (u, (u,
(69)
E (u.,a))-ovvx
log Puv (u, v v)pz(( (z)
(70)
log PUVIZ
(71)
zxpz (u.v)~peviz
m(u, w)
(expanding the expectation over 2)
(72)
PUv (u, v) log
Po @ pv(
E (a,0)~pUV
+
E (u.)) Spv
log
(73)
2
m(u,u) u)
m(u,v)
Therefore:
[CU V): = D |l Pu & pvl = L7s [U:V]
(74)
From this, we observe that, in the classifier setup above, the MI between (U, V) and Z equals the Jensen-Shannon divergence between U and V. A similar property holds for general JS divergences.
MI also characterizes the information loss by conditioning. In this setup,
[[U, V); Z| H [2] H [Z|U, V] log 2 H [2 U, V]
(75)
25

---

where the conditional entropy is defined as
H []
E (v,,,))-oovz
log Pogiz1e uu (2/1,
(76)
Combining Equations
74 and 75) yields
J [U: VJ log - H [2|U,
(77)
Suppose we define a variational distribution qa (=|u, v) then
H []
E
L + log (o (z|u,, v)]
(1,1:2)~Pevz
(78)
E
|- log (0(2,,
(u,,,)))puvz
E
E E~P2IUV
[- log logozzoo (u, u12) + log (o w)]
(u.))-ev
(79)
E (a.u,))~puvz
| log (2lu,,
Therefore,
[Z
E (u,o)-pov
(K (o e)]]
E (u.v,))-Puvx
log go u, u)]
CCR
(80)
The first term 5 on the right-hand side of (80 is the expected KL.D between the true posterior Pz UV and the model posterior go, which is always non-negative. The second term corresponds to the expected cross-entropy loss of the classifier qa.
Using the non-negativity of the KL divergence, we obtain the inequality:
H [Z LCE.
(81)
where Loe denotes the expected cross-entropy loss. Equality holds if and only if the model distribution ga(z u, v) is equal to the true posterior (2 (= u, v) almost everywhere.
Combining this with Equation
yields
Lıs [U:V] 2 log LCE
(82)
Thanks to this new derivation, we can explicitly quantify the gap between the true JSD lower bound E(Tss [U; VI) and the one obtained from the cross- gross-entropy ICE (0) as follows:
Lemma B.7. Let qo denote the discriminator, and let LCE(f) be its expected binary cross-entropy. The gap between our variational lower bound IcE (0) and the optimal lower bound E (Jas [U; VJ) given by
Elss VJ; ICr(0) xe 8 E' (ICE(0)),
(83)
where denotes the expected Kullback-Leie divergence between the true and approximate posteri- ors:
E (v.v)~puv
(Dkuvvzivvu, || go(z u, v)]
(84)
Proof. Combining Equations
77 and 80) obtain
Js UU: Vl log(2) LC 5.
(85)
Hence, the gap Ag between the optimal JSD lower bound and our bound (c) is
As (Las UU: v]) - - ((cc(())
(86)
= (log(2) Los (0) + 5) - = (log(2) - (ce(
(87)
Applying a first-order Taylor expansion around ICg leads to:
Ag s 8 E' (Jcn(0))
(88)
26

---

B.4 Proof of Eq. (28)
We derive here method for approximating the MI by way of the trained classifier, building on the setup of Section 4.2 and B.3
Under with the joint model of Eq. 688) (originally (1)).
ol) pzuv (2, u)
and (z)u()/ Pzuv (2, u, Duv (u, v)
(89)
(z(2)
Moreover, since 7 B(1/2), we have p(2 = =1) = p(2 = 0) Therefore,
log (z 1/,, v) PzIuv = u 2)
Pzuv (2 u,
PUVIz (u, 212 = log (u,(u,)) v)
log
log
Pzuv (2 = O, u,
(u, = (pu @ (v) (u, v) (90)
Using the definition of KLD,
log (uv v) (pu @ pv (2)
PzIv = 1 u, v)
DKL |l pu @ pvp
E (w,))~PUV
E (a.t)~PoV
log
pziov = 0 v)
(91)
and finally.
IU: DKL/Pov |l po 8 pv]==
[Lizio(===]] 1/u,
(92)
(u.t)~puv
where LL is the logit transform. Thus we can estimate the MI of puv by first training a classifier to approximate the true posterior, Pziuv, and after that use approximate the expectation by averaging over the 'true' data.
C Appendix: Approximation of the function E
In this section, we derive Equation 17 which is used to define the inverse E-1, This allows the use
of numerical solvers to approximate E. Finally, present the approximation of the function E using the Logit transform.
C.1 Derivation of Equation
17
Let's recall 17
Dus B(1) B(v)] = log 2
1
log(1 + v) vlog
V
(93)
2
(1
Proof. Using Lemma (B.I)
DJs (Js(B(1) | B(v)] log 2 +
1
log
1
+ v
V
+ - v) log
1-
(94)
2
+
1+v
1
V 1 1+
log 2
log(1 + v) v vlog
(95)
2
27

---

C.2 Python code for implementing 3
from scipy. optimize import root_s import numpy as np
Define the inverse function Xi (x) def inv_xi
up. exyx
termi up npg1og z)
term2 = 2 np. log (2 / (1 + 2))
return lp. (2) 0.5 (termi term2)
Define the function xi(x)
def xi)x)
result root_scalar (lambda y: (av_xi(y) bracket=[0 [0, 100],
method- 'brenqq
return result. root if result converged else None
Algorithm 1: Algorithmic implementation of the E\t function the inverse of its known inverse E-1
C.3 Proposed differentiable approximation of =
To enable the use of our bound within differentiable objective, we introduce smooth, differentiable approximation of the E function. This approximation is derived from the Taylor expansion the KI.
divergence in terms of the JSD. The lower bound is characterized by the following parametric curve:
Das (BL) II B(v] = - (-
(96)
y [BL[ B log
(97)
where E: log 2) R strictly increasing function implicitly defined by its inverse:
E- : + log 1 [111 ve log (1 + - + ye
(98)
2
Hence, we can express:
== E
(99)
Let I log x denote the Logit function. We then observe that:
x
+ +1)
= log
log
(100)
2
log 2
log 2
22 2-1
+
log
log 2
(101)
--1
log
[[1 + e log (1 (1+ + ye
2 Tog 2
log
(102)
[11 + (1+e-0) log (1 + ye
2log
1lo
log
(1+ log (1 + +e y 1)
(103)
The first-order Taylor expansion of the right-hand side of Eq. (103) around 0 is simply + This motivates approximating E using a scaled logit function. Through grid search, found that scaling the logit by factor of 1.15 yields the lowest median approximation error over I E (0, 1):
log ( + +1))
((x) z 1.15 L
(104)
As shown in Figure our proposed approximation closely matches the numerically estimated inverse obtained via a solver. Importantly, this formulation is fully differentiable and can directly integrated into a variational lower bound (VLB) objective in representation learning.
28

---

Numerical estimation via solver
17.5
Logit approximation
15.0
12.5
DKL P || q
10.0
7.5
5.0
2.5
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
D DJs p || q
Figure 6: Comparison between the numerical evaluation of using Algorithm T the proposed differentiable approximation from Eq. (104).
D Appendix: Tightness Analysis in the Mutual Information Setting
While our JSD-KLL KLD lower bound in Theorem 4.1 the tightest possible (i.e., equality is achieved for a family of distributions) between these two divergences for arbitrary distributions, one might expect tighter bound when restricted to the mutual information setting. i.e., when comparing joint distribution to the product of its marginals. Remarkably, as illustrated Figure of the main text, discrete pont-mmrginnl pairs lie close general bound. In this section, analytically show that there exists an infinite family of distributions in the mutual information setting that exactly lie on lower bound.
Lemma D.1. Let (U be pair of uniform categorical variables of dimension k E No that are fully dependent (i.e., V). Then:
IU; = [T:
(105)
Proof. Let k N+, and assume and V are fully dependent (i.c., V) uniform categorical variables of dimension k. Their joint and marginal distributions are given by:
Pu = Pv = t Pov = L
and consequently,
Pv = (1..))
Let = 2 (Pu @ Pv + Puv) denote the mixture distribution. Then M.j = 1/k+1/12 if = jj and Mi, 2ेे otherwise.
We now compute 1 10: V] and lıs V]:
&
1/k log k.
I0; VJ
log
(106)
E
1/k2
1=1
29

---

1/k
1/k²
k
1/k2
Lis [U: V =
log
+
log
log
x
2
(1/k: + 1/k2)
2
(1/k: 1/h2)
x2
1/(2k²)
1=1
:=1 1=1
(107)
1/2
2/k log 1-1/E log 1+1/E + k-1 log
(108)
2
log 2 1 {(oog(1 1/k) log
1/k 1+17k
(109)
(log k).
(110)
Combining Equations (166) and 110). we obtain
Jas [U; =
-E-
(111)
which completes the proof.
E Appendix: Experimental details
E.1
Diserete case experiments
In this section, we present a synthetic experiment designed to empirically validate the tightness of our proposed lower bound between mutual information (MI) and the Jensen-Shannon divergence (ISD), as introduced in Theorem 4.1 Unlike general divergences between arbitrary distributions, here we specialize the analysis to the mutual information setting, which corresponds to comparing a joint distribution to the product of its marginals.
We consider family of discrete joint distributions P(a) Rixk parameterized by dependence factor a E [0, 1], which interpolates between the independent and perfectly dependent cases. Specif- ically, fix the marginals to be uniform categorical distributions: Pt Pv 1 where denotes the number of categories. The joint distribution is constructed as a convex combination of the independent and deterministic cases:
Ply (1 - ov) @ Pv + a diag((P))
(112)
where diag(Pt) denotes the diagonal matrix with Po on its diagonal, ensuring perfect dependence along the diagonal. This construction ensures that:
When a 0, U and V are independent.
When 1, U V with probability one.
For each a, we compute the mutual information V(UVV) DKa(P || Py Pv] and the Jensen-Shannon divergence Dis(Ple) (rV || Pt @ Pv]. and compare them to the lower bound E(Dus[PPy || Pu & Pvl)), where E is defined in Equation 19
E.2
Complex Gaussian and non-Gaussian distributions experiments
In this section, we provide further details on the experimental setup described in Section 5.2 and present supplementary results for the linear and cubic Gaussian experiments.
E.2.1 Experimental setup
The neural network architectures employed in our experiments follow the joint architecture, as in 12 We do not the deranged architecture from [12), as it is incompatible with the CPC objective.
Joint architecture. Following the architecture described [12], we employ a fully connected feed-forward neural network where the input layer has dimensionality 2d, corresponding to the concatenation of two d-dimensional samples. The network comprises two hidden layers, each with 256 neurons and ReLU activations, and single scalar output neuron. During training, the network
30

---

receives as imput d² pairs (u, v) per iteration, where these paits are formed by taking all combinations of samples u and y independently drawn from the joint distribution P(a.o)-
Staircase In the Gaussian setting, we define two random variables U (0, La) and N ~ NN La), sampled independently, and construct V - p /1 p² N, where p the correlation coefficient.
To explore settings with non- Gaussian marginals while preserving the same mutual information, we apply nonlinear transformations V. In the cubic case, we use the mapping + va which preserves the dependency structure but introduces non-Gaussian marginals. Additionally, we benchmark two alternative transformations: the inverse hyperbolic sine (asinh), which shortens the tails of the distribution, and the half-cube transformation, which stretches them. These mappings allow us to evaluate the robustness of our method under controlled deviations from Gaussianity.
During the training procedure, every 4k iterations, the target value of the MI increased by nats, for 5 times, obtaining target staircase with 5 steps. The change in target MI is obtained by increasing which affects the true MI according to I [U; V]
log - p
Training procedure. Each neural estimator is trained using Adam optimizer, with learning rate 0.002, 31 = 0.9, B2 = 0.999. The batch size is initially set to b 64.
E.2.2 Analysis for different values of batch size
Our VLB is robust changes in batch size b. In Figures 7 and 9 observe that MINE and NWJ exhibit high variance, even for large batch sizes (e.g., 128), while JSD-LB and CPC remain more stable. Notably, MINE's instability can cause substantial overestimation of its lower bound, occasionally exceeding the true MI. CPC, on the other hand, provides low-variance estimates but is upper bounded by the contrastive limit log(b), which results in higher bias in high-MI regimes.
E.2.3 Analysis for different values of dimension d
We additionally perform an analysis for different values of dimension d. We report the achieved bias, variance, and mean squared error (MSE) corresponding to the four settings d (5, 10, 20, 100} in T We experimentally found that MINE produces estimate with high variance. NWJ is unstable in high dimension, and CPC is bounded by In(b) When repurposed for two-step MI estimation, our method surpasses other variational lower bound estimators across dimensions. In contrast, while the bias of JSD-LB increases with dimension d, its variance remains stable, making a reliable surrogate for MI maximization. Finally, when repurposed for two-step MI estimation, our method surpasses other variational lower bound estimators across dimensions.
E.2.4 Comparison of our two-step estimator with other two-step estimators
While our primary contribution is a novel variational lower bound (JSD-LB) that can be directly used for MI maximization, we additionally report results for our two-step estimator (Our MI estim.) alongside other two-step approaches to provide better context. Comparisons with SMILE, KL-DDME,, and GAN-DIME are presented in Tables and 6 The results show that our derived two-step MI estimator, which is equivalent to GAN-DIME up to a linear transformation, achieves competitive performance.
E.2.5 Computational time analysis
A critical characteristic is the computation time. The computational time analysis is developed server with CPU "Intel Xeon Platinum 8468 48-Core Processor" and an NVIDIA GPU H100 and reported in Table As in [12], we found that the influence of the MI lower bound objective functions on the algorithm's 's time requirements is minor.
31

---

MINE
NWJ
CPC
JSD-LB
Gaussian Mi justs]
10
10
10
10
300000 Steps
200000
10:00
20000
10000
20000
XXXM Steps
20000
10
Cubic MI juals]
%
20000
20000
Stegus
Stups
Scops
Steps
Estimated
In((b)
MI
Figure 7: Staircase MI estimation comparison for d = and batch size 32. The Gaussian case is reported in the top row, while the cubic case is shown in the bottom row.
MINE
NWJ
CPC
JSD-LB
Gaussian
10
18
MI judts
20000 Steges
200000
100000 Strps
20000
10000 Stepe
20000
20000
Steps
10
10
10
Cubic
julss
20000 Stogus
200000
%
100000
20000
10000
2000
%
100MI
Strps
Stepe
Steps
True
Estimatedd
VLB
In(b)
Our MI estimn
Figure 8: Staircase MI estimation comparison for d and batch size b 64. The Gaussian case reported in the top row, while the cubic case is shown in the bottom row.
MINE
NWJ
CPC
JSD-LB
Gaussian
10
MI (uals
%
20000
10000 Steps
20000
Stepe
Steps
Stepe
10
10
10
10
Cubic
MI Jaats
5
10000 Steps
20000
10000
20000
10000
2000
10000
20000
Stepe
Steps Our MI estim.
True
MI
Eatimaned
VLB
In(b)
Figure 9: Staircase MI estimation comparison for d 5 and batch size 6 128. The Gaussian case reported in the top row, while the cubic case is shown in the bottom row.
32

---

Table 4: Bias, variance, and mean squared error (MSE) of the MI estimators using the joint architec-
ture, when = 128, for the Gaussian setting with varying dimensionality d E (5, 10, 20, 100).
d
d
10
MI
2
4
6
8
10
2
4
6
8
10
a) Bias
MINE
0.07
0.11
0.12
0.05
-1.58
0.12
0.19
0.32
0.12
-1.26
NWJ
0.09
0.19
0.8
1.77
0.15
0.28
0.86
CPC
0.2
0.81
2.11
3.89
5.85
0.25
0.91
2.21
3.94
5.86
JSD-LB
0.64
1.22
1.75
2.22
2.62
0.77
1.55
2.33
3.07
3.75
Our MI estim.
0.1
0.09
0.12
0.19
0.1
0.17
0.21
0.28
0.35
0.29
b) Variance
MINE
0.05
0.14
0.41
2.05
12.23
0.06
0.15
0.49
1.63
8.87
NWJ
0.07
0.23
4.09
12.83
0.09
0.23
1.49
CPC
0.04
0.03
0.01
0.0
0.0
0.04
0.03
0.01
0.0
0.0
JSD-LB
0.03
0.06
0.12
0.23
0.72
0.03
0.06
0.11
0.19
0.35
Our MI estim.
0.07
0.13
0.22
0.36
0.93
0.08
0.15
0.25
0.44
0.73
c) Mean Square Error
MINE
0.06
0.15
0.43
2.05
14.72
0.08
0.19
0.59
1.64
10.46
NWJ 0.07
0.27
4.74
15.98
00
0.11
0.31
2.23
00
CPC 0.08
0.69
4.45
15.15
34.23
0.1
0.87
4.89
15.5
34.36
JSD-LB
0.45
1.55
3.17
5.16
7.57
0.63
2.47
5.56
9.62
14.43
Our MI estim.
0.08
0.13
0.23
0.39
0.94
0.11
0.2
0.33
0.56
0.82
d
20
d = 100
MI
6
8
10
4
a) Bias
MINE
0.19
0.38
0.8
0.82
0.7
1.0
1.36
2.61
3.24
4.0
NWJ
0.23
0.48
00
00
00
1.13
1.67
3.05
4.1
CPC
0.31
1.0
2.3
4.0
5.89
0.99
1.39
2.59
4.18
5,98
JSD-LB
0.88
1.8
2.77
3.73
4.67
1.42
2.36
3.54
4.79
6.06
Our MI estim.
0.26
0.39
0.52
0.64
0.65
1.22
1.67
2.22
2.83
b) Variance
MINE
0.08
0.19
0.65
1.62
5.74
0.19
0.21
0.99
1.15
1.75
NWJ
0.12
0.27
00
00
00
0.14
0.17
1.19
1.81
0C
CPC
0.06
0.04
0.02
0.01
0.0
0.18
0.04
0.03
0.01
0.01
JSD-LB
0.04
0.06
0.1
0.18
0.29
0.06
0.04
0.06
0.11
0.16
Our MI estim.
0.12
0.17
0.28
0.44
0.73
0.17
0.14
0.19
0.27
0.41
C) Mean Square Error
MINE
0.12
0.34
1.29
2.29
6.23
1.18
2.05
7.81
11.62
17.76
NWJ 0.17
0.5
00
00
30
1.41
2.95
10.46
18.61
06
CPC
0.15
1.04
5.32
15.97
34.65
1.15
1.96
6.72
17.47
35.83
JSD-LB
0.82
3.28
7.75
14.12
22.06
2.07
5.6
12.61
23.05
36.89
Our MI estim.
0.19
0.32
0.55
0.86
1.15
1.39
1.62
2.99
5.18
8.41
33

---

Table 5: Bias, Variance, and Mean Square Error SMILE, KL-DIME, GAN-DIME, and Our MI estimator across different b, and MI values for 5.
64
b
256
MI
4
8
10
2
6
10
a) Bias -0.23
SMILE
-0.59
-0.73
-0.68
-0.65
-0.28
-0.65
-0.81
-0.78
-0.68
KL-DIME
0.09
0.12
0.22
0.44
0.9
0.05
0.05
0.09
0.19
0.41
GAN-DIME
0.1
0.1
0.13
0.24
0.09
0.06
0.05
0.07
0.12
0.01
Our MI estim.
0.1
0.09
0.12
0.19
0.1
0.06
0.05
0.07
0.12
0.04
b) Variance
SMILE
0.08
0.13
0.22
0.33
0.72
0.03
0.07
0.1
0.18
0.3
KL-DIME
0.06
0.07
0.08
0.11
0.17
0.03
0.02
0.02
0.03
0.05
GAN-DIME
0.07
0.13
0.22
0.37
0.92
0.03
0.06
0.11
0.18
0.4
Our MI estim.
0.07
0.13
0.22
0.36
0.93
0.03
0.06
0.1
0.2
0.43
c) Mean Square Error
SMILE 0.13
0.48
0.74
0.8
1.14
0.11
0.49
0.75
0.8
0.77
KL-DIME 0.07
0.08
0.13
0.3
0.97
0.03
0.02
0.03
0.07
0.22
GAN-DIME
0.08
0.14
0.23
0.43
0.92
0.04
0.07
0.11
0.2
0.4
Our MI estim.
0.08
0.13
0.23
0.39
0.94
0.04
0.07
0.11
0.22
0.43
Table Bias, Variance, and Mean Square Error SMILE, KL-DDIM, GAN-DIME, and Our MI
estimator across different b, and MI values for 20.
b 64
256
MI
2
6
10
6
10
a) Bias
SMILE
-0.08
-0.24
-0.25
-0.24
-0.16
-0.17
-0.38
-0.43
-0.39
-0.28
KL-DIME
0.24
0.36
0.62
1.0
1.57
0.16
0.2
0.34
0.58
0.92
GAN-DIME
0.26
0.38
0.54
0.67
0.72
0.19
0.26
0.35
0.48
0.5
Our MI estim.
0.26
0.39
0.52
0.64
0.65
0.19
0.26
0.36
0.47
0.53
b) Variance
SMILE
0.12
0.16
0.27
0.46
0.77
0.06
0.07
0.12
0.2
0.32
KL-DIME
0.1
0.11
0.15
0.19
0.22
0.05
0.03
0.04
0.05
0.07
GAN-DIME
0.12
0.17
0.26
0.41
0.73
0.06
0.08
0.12
0.2
0.36
Our MI estim.
0.12
0.17
0.28
0.44
0.73
0.06
0.08
0.12
0.2
0.32
Mean Square Error
SMILE 0.12
0.22
0.33
0.52
0.79
0.08
0.21
0.3
0.35
0.41
KL-DIME
0.16
0.24
0.53
1.19
2.69
0.07
0.07
0.16
0.38
0.91
GAN-DIME
0.19
0.32
0.55
0.86
1.25
0.09
0.15
0.24
0.43
0.61
Our MI estim.
0.19
0.32
0.55
0.86
1.15
0.09
0.14
0.25
0.42
0.6
Table 7: Comparison of the time requirements (in minutes) to complete the 5-step staircase MI
(d 5) over the batch size.
b 32
b = 64
b = 128
b = 256
b 512
b 1024
MINE
0.54
0.46
0.52
1.0
2.67
9.07
NWJ
0.39
0.4
0.46
0.94
2.61
9,02
CPC
0.39
0.37
0.42
0.91
2.59
8.99
JSD-LB
0.45
0.48
0.51
0.96
2.62
9,05
34


Reviews:
[{"review_id": "MLsYuIIt7n", "reviewer": "Reviewer_KvyC", "summary": "The paper studies estimation of mutual information (MI), and introduces a new bound on MI based on the Jensen-Shannon divergence (JSD). The paper presents an overview of existing approaches to estimating MI, including variational lower bounds and discriminator-based approaches, alongside a discussion of limitations related to, e.g., estimator variance in high MI settings or difficulties around retraining of discriminators. An overview is provided of MI estimation and JSD in representation learning alongside a discussion of their connection to generative adversarial networks. The paper discusses existing work on joint ranges for f-divergences, showing how joint ranges can be characterized by studying the convex hull of joint ranges for Bernoulli distributions.\n\n\nThe main contribution of this paper is a variational bound on MI. The authors characterize the edges of the convex hull of the joint range of JSD and Kullback-Leibler divergence (KLD) on Bernoulli distributions. They show how one of these edges characterizes a lower bound on the KLD through the JSD with a form that is described by the JSD between arbitrary Bernoulli distributions and a degenerate Bernoulli with $p=1$. This result is then related to the discriminator cross-entropy (CE) also studied in the GAN literature showing how this CE bounds the JSD and thus the MI. Finally, a procedure for estimating the value of the MI is provided, similar to the GAN-DIME approach.\n\n\nExperimentally, the study shows that the MI lower bound is tight in a controlled discrete setting before proceeding to study a suite of established benchmarks for MI estimation. The experiments compare the proposed estimator against other variational lower bound MI estimators, and the proposed solution compares favorably with more accurate bounds and lower variance in both higher true MI settings and challenging non-Gaussian settings.", "questions": "These questions are largely extensions of the weaknesses discussed above. I would strongly consider increasing by score if any one of these aspects were addressed, e.g., through some further analysis of bias-variance of the estimator or consideration of slightly more complicated problem:\n\n* Do you have any characterizations, either theoretical or experimental, of the bias-variance trade-off for the presented bound/MI estimator compared to the baselines? - or the time-complexity?\n* How does the approach compare to baselines in higher dimensional versions of the problem you consider - or, ideally, on a more challenging problem?\n* Do you have a sense of how the presented bound (and the estimator) compares to, e.g., f-DIME or SMILE (even if the comparison should be done with due discussion of the limitations you point out)?", "limitations": "The authors discuss limitations, such as the necessity of numerical approximation in characterizing $\\Xi$ and the choice of baselines. A brief comment on the numerical verification of Conjecture B.3 as discussed in Sec. B1, could be warranted in the main paper.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper presents a novel theoretical bound which is both, to the best of my knowledge, novel and with sound proofs, drawing on and connecting existing research in information theory, representation learning, generative adversarial networks, and mutual information estimation.\nThe paper presents a clear overview of existing approaches and their strengths and limitations, and situates their contribution well based on this. \nThe paper provides well-documented code for recreating their experiments, including notebooks with further details and visualizations of, e.g., joint ranges. These same visualizations are used in the paper to provide a well-presented argument for the main contribution. \nThe paper presents convincing experimental results, validating both the tightness of the bound and showing good performance (compared to a subset of common MI estimators) of the MI bound and the direct MI estimator on a well-studied set of problems.\n\n\nThe main weakness, I find, is that the study, in contrast to preceding studies on MI estimation, does not present, e.g., bias-variance results (beyond some discussion based on the plots), consider only $d\\leq5$ cases, and does not consider more complicated problems. A lesser point is the, while well-argued, lack of consideration of broader set of baselines.\n\n*On bias-variance results and time complexity*\nI would find it to be strong additions to the paper, if the experiments could include analyses similar to previous, related work. Examples of such analysis include, e.g., time-complexity comparisons in Figure 8 or the bias-variance numerical results (such as Tables 2-5, Figures 9, 12-14) in Letizia et al. (2024). Similarly, bias-variance/gradient accuracies are considered in Figures 3-4, 6-7 of Poole et al. (2019) [17] and Figure 2 in Song & Ermon (2020) [20]. A minor note is that your reference gives only Poole and Ozair but the study is also authored by van den Oord, Alemi, and Tucker.\n\n*On problems problems beyond the staircase experiments*\nSimilarly, the experimental validation is limited to the staircase benchmarks (while challenging and relatively comprehensive, especially with the inclusion of, e.g., the Student-case) but does not include a consideration of a possibly more challenging problem, such as Poole et al.'s analysis of dSprites, Song & Ermon's analysis of MNIST datasets, or Letizia et al.'s consideration of MNIST. \n\n*On dimensionality*\nThe experimental analysis is restricted to lower-dimensionality ($d=5$) problems for the staircase MI estimation problems than is generally considered in these related works. For instance, Song & Ermon present results for the $d=20$ setting in their Figure 1, Letizia et al. (while they do restrict the analysis to $d=5$ and below for, e.g., Figures 2-4) compared the derangement and permutation strategies on a $d=20$ case, and Poole et al.'s results in Figure 2 are on $d=20$.\n\n*On baselines*\nThe authors highlight how, e.g., f-DIME variants are strong MI estimators with good bias variance trade-offs. The authors argue for the chosen subset of baselines (MINE, NWJ, CPC, i.e., without discriminator-based ones) based on them being variational lower bounds that do not require retraining when the joint changes and how, therefore, these can be used as estimators during training. I find this generally convincing, but I would find some comparison to a wider variety of baselines (including GAN-DIME and SMILE) valuable to contextualize the performance.", "was_revised": true, "final_justification": "My main concerns were related to bias-variance results, higher-dimensional experiments, more complicated problems, and comparisons to, e.g., SMILE. I find that the authors have adequately addressed each of concerns, showing, e.g., favourable bias-variance trade-off even in higher dimensions and utility of the proposed method in an information bottleneck experiment on a more complicated problem. I have improved my rating accordingly.", "rebuttal": "We thank the Reviewer for the positive and constructive feedback. We appreciate that the Reviewer clearly identified the key contributions of our work. As suggested, we have conducted additional experiments and revised the manuscript to include:\n\n* Bias-variance and time complexity analyses  \n* Extended experiments to higher dimensions\n* Comparisons to additional baselines, including two-step estimators such as SMILE and $f$-DIME\n\nBelow, we address each of the Reviewer’s points in detail.\n\n## Weaknesses\n\n**On bias-variance results and time complexity.** We appreciate the suggestion. In the revised manuscript, we include a detailed analysis of the bias, variance, and mean squared error (MSE) of our JSD lower bound, following protocols from Letizia et al. (2024) and Poole et al. (2019). These new results in the revised appendix D.2.3 (and reproduced in Tables 1 and 2) show that our JSD-LB consistently provides a stable lower bound on the true MI across dimensions:\n\n*  While MINE and NWJ exhibit high variance, JSD-LB and CPC remain more stable. In particular, MINE’s instability can lead to significant overestimation of its lower bound, sometimes even exceeding the true MI. While CPC provides low-variance estimates, they are upper bounded by the contrastive bound $\\log(b)$, resulting in higher MSE and bias in high MI regimes. We experimentally found that the bias of JSD-LB increases with dimension $d$, but its variance remains stable, making it a reliable surrogate for MI maximization.\n* When repurposed for two-step MI estimation, our method surpasses other variational lower-bound estimators across dimensions.\n* As in Letizia et al. (2024), we found that the influence of the MI lower-bound objective functions on the algorithm’s time requirements is minor. \n\n**High-dimensional experiments.** In response to the Reviewer’s concern about dimensionality, we extended our staircase experiments to higher dimensions (up to $d=100$). Results in Table 1 show that our method maintains a favorable bias-variance trade-off, while estimators such as MINE and NWJ have variance and stability that degrade with dimensionality.\n\n**More challenging benchmarks.**\nTo further demonstrate practical relevance, we added a new representation learning benchmark using the Information Bottleneck (IB) framework. Results presented in revised Appendix D.3.2 and reproduced in our response to Reviewer ifQp show that using JSD as a surrogate to MI leads to state-of-the-art downstream performance, outperforming other variational IB approaches. These experiments demonstrate that, in practice, maximizing JSD is an effective strategy for representation learning.\n\n**Comparison to Additional Baselines**. While our initial focus was on variational bounds that can be directly used for MI maximization, we agree that including comparisons with two-step estimators provides better context. In revised Appendix D.2.4  (referenced in our response to Reviewer zy6E), we now report comparisons to SMILE, KL-DIME, and GAN-DIME. The results show that our derived two-step MI estimator, which is equivalent to the GAN-DIME (up to a linear operation), allows reaching competitive performance.\n\n**Other Comments**.\n\n* On the Poole reference: Thank you for pointing this out. We have corrected the citation to include all co-authors.  \n* On limitations: We added a short comment on the numerical verification of Conjecture B.3 in the main text (Section 4.1).\n\n## Questions\n\n> “Do you have any characterizations, either theoretical or experimental, of the bias-variance trade-off for the presented bound/MI estimator compared to the baselines? \\- or the time-complexity?”\n\nAs mentioned above, we provide an experimental characterization of the bias-variance trade-off of our bound. We found that maximizing JSD leads to an increase in the lower bound of MI that is stable. A more competitive bias-variance tradeoff can be obtained by reusing the discriminator for two-step MI estimation.\n\n> “How does the approach compare to baselines in higher-dimensional versions of the problem you consider \\- or, ideally, on a more challenging problem?”\n\nSee our response above for higher-dimensional cases and a challenging Information Bottleneck problem.\n\n> “Do you have a sense of how the presented bound (and the estimator) compares to, e.g., f-DIME or SMILE (even if the comparison should be done with due discussion of the limitations you point out)?”\n\nSee our response above.\n\n## Rating\nWe appreciate the Reviewer recognizing the theoretical contributions. We hope that our answers and changes address the Reviewer’s concerns and can contribute to increasing their score.\n\n## Additional experiments\n\n**Table 1:** Bias, variance, and mean squared error (MSE) of the MI estimators using the joint architecture, when N = 64, for the Gaussian setting with varying dimensionality $d \\in $ {5,10,20,100}.\n\n|       |       |     |     | **d=5** |     |    | |     |     | **d=10** |     |     | |     |     | **d=20** |     |     | |     |     | **d=100** |     |     |\n|:-----|:-----:|:---:|:---:|:-------:|:---:|:---:|:---:|:---:|:---:|:--------:|:---:|:---:| :---:|:---:|:---:|:--------:|:---:| :---:|:---:|:---:|:--------:|:---:| :---:|:---:|\n|       | **MI** | **2** | **4** | **6** | **8** |  **10** | | **2** | **4** | **6** | **8** | **10** | | **2** | **4** | **6** | **8** | **10** | | **2** | **4** | **6** | **8** | **10** | \n**a) Bias**\n |       | MINE |  0.07 |  0.11 |  0.12 |  0.05 |  -1.58 |    |  0.12 |  0.19 |  0.32 |  0.12 |  -1.26 |    |  0.19 |  0.38 |  0.8 |  0.82 |  0.7 |    |  1.0 |  1.36 |  2.61 |  3.24 |  4.0 |\n |       | NWJ |  0.09 |  0.19 |  0.8 |  1.77 |  inf |    |  0.15 |  0.28 |  0.86 |  inf |  inf |    |  0.23 |  0.48 |  inf |  inf |  inf |    |  1.13 |  1.67 |  3.05 |  4.1 |  inf |\n |       | CPC |  0.2 |  0.81 |  2.11 |  3.89 |  5.85 |    |  0.25 |  0.91 |  2.21 |  3.94 |  5.86 |    |  0.31 |  1.0 |  2.3 |  4.0 |  5.89 |    |  0.99 |  1.39 |  2.59 |  4.18 |  5.98 |\n |       | JSD-LB |  0.64 |  1.22 |  1.75 |  2.22 |  2.62 |    |  0.77 |  1.55 |  2.33 |  3.07 |  3.75 |    |  0.88 |  1.8 |  2.77 |  3.73 |  4.67 |    |  1.42 |  2.36 |  3.54 |  4.79 |  6.06 |\n |       | Our MI estim.  |  0.1 |  0.09 |  0.12 |  0.19 |  0.1 |    |  0.17 |  0.21 |  0.28 |  0.35 |  0.29 |    |  0.26 |  0.39 |  0.52 |  0.64 |  0.65 |    |  1.1 |  1.22 |  1.67 |  2.22 |  2.83 |\n**b) Variance**\n |       | MINE |  0.05 |  0.14 |  0.41 |  2.05 |  12.23 |    |  0.06 |  0.15 |  0.49 |  1.63 |  8.87 |    |  0.08 |  0.19 |  0.65 |  1.62 |  5.74 |    |  0.19 |  0.21 |  0.99 |  1.15 |  1.75 |\n |       | NWJ |  0.07 |  0.23 |  4.09 |  12.83 |  inf |    |  0.09 |  0.23 |  1.49 |  inf |  inf |    |  0.12 |  0.27 |  inf |  inf |  inf |    |  0.14 |  0.17 |  1.19 |  1.81 |  inf |\n |       | CPC |  0.04 |  0.03 |  0.01 |  0.0 |  0.0 |    |  0.04 |  0.03 |  0.01 |  0.0 |  0.0 |    |  0.06 |  0.04 |  0.02 |  0.01 |  0.0 |    |  0.18 |  0.04 |  0.03 |  0.01 |  0.01 |\n |       | JSD-LB |  0.03 |  0.06 |  0.12 |  0.23 |  0.72 |    |  0.03 |  0.06 |  0.11 |  0.19 |  0.35 |    |  0.04 |  0.06 |  0.1 |  0.18 |  0.29 |    |  0.06 |  0.04 |  0.06 |  0.11 |  0.16 |\n |       | Our MI estim. |  0.07 |  0.13 |  0.22 |  0.36 |  0.93 |    |  0.08 |  0.15 |  0.25 |  0.44 |  0.73 |    |  0.12 |  0.17 |  0.28 |  0.44 |  0.73 |    |  0.17 |  0.14 |  0.19 |  0.27 |  0.41 |\n**c) Mean Squared Error**\n |       | MINE |  0.06 |  0.15 |  0.43 |  2.05 |  14.72 |    |  0.08 |  0.19 |  0.59 |  1.64 |  10.46 |    |  0.12 |  0.34 |  1.29 |  2.29 |  6.23 |    |  1.18 |  2.05 |  7.81 |  11.62 |  17.76 |\n |       | NWJ |  0.07 |  0.27 |  4.74 |  15.98 |  inf |    |  0.11 |  0.31 |  2.23 |  inf |  inf |    |  0.17 |  0.5 |  inf |  inf |  inf |    |  1.41 |  2.95 |  10.46 |  18.61 |  inf |\n |       | CPC |  0.08 |  0.69 |  4.45 |  15.15 |  34.23 |    |  0.1 |  0.87 |  4.89 |  15.5 |  34.36 |    |  0.15 |  1.04 |  5.32 |  15.97 |  34.65 |    |  1.15 |  1.96 |  6.72 |  17.47 |  35.83 |\n |       | JSD-LB |  0.45 |  1.55 |  3.17 |  5.16 |  7.57 |    |  0.63 |  2.47 |  5.56 |  9.62 |  14.43 |    |  0.82 |  3.28 |  7.75 |  14.12 |  22.06 |    |  2.07 |  5.6 |  12.61 |  23.05 |  36.89 |\n |       | Our MI estim.  |  0.08 |  0.13 |  0.23 |  0.39 |  0.94 |    |  0.11 |  0.2 |  0.33 |  0.56 |  0.82 |    |  0.19 |  0.32 |  0.55 |  0.86 |  1.15 |    |  1.39 |  1.62 |  2.99 |  5.18 |  8.41 |\n\n\nThe computational time analysis is developed on a server with CPU ”Intel Xeon Platinum 8468 48-Core Processor”\nand an NVIDIA GPU H100.\n\n**Table 2:** Comparison of the time requirements (in minutes) to complete the 5-step staircase MI ($d=5$) over the batch size $b$ using\n| | **b=32** | **b=64** | **b=128** | **b=256** | **b=512** |  **b=1024** | \n|:-----|:-----:|:-----:|:---:|:---:|:-------:|:-------:|\n| MINE | 0.54 | 0.46 | 0.52 | 1.0 | 2.67 | 9.07 |\n| NWJ | 0.39 | 0.4 | 0.46 | 0.94 | 2.61 | 9.02 |\n| CPC | 0.39 | 0.37 | 0.42 | 0.91 | 2.59 | 8.99 |\n| JSD-LB | 0.45 | 0.48 | 0.51 | 0.96 | 2.62 | 9.05 |"}, {"review_id": "4Hm6YY2gRR", "reviewer": "Reviewer_Snmy", "summary": "In order to optimize mutual information (MI), recent methods have trained discriminators to distinguish samples from the joint distribution from those from the product of marginals, using the Jensen-Shannon divergence (JSD).\nThis work provides theoretical and empirical evidence in favor of using discriminative losses on the Jensen-Shannon divergence, by showing that maximizing JSD is equivalent to maximizing a lower bound on the KLD-based (Kullback-Leibler divergence) MI.\nAuthors show that minimizing a cross entropy loss of a discriminator trained to separate joint from marginal samples increases the lower bound on MI.", "questions": "* What are the real-world implications of this work?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The theoretical proofs in the paper are outside of my area of expertise so I cannot comment on its strengths and weaknesses.\n\n**Strengths**\n* The staircase MI estimations from neural networks appear to be the most accurate compared to baselines.\n\n**Weaknesses**\n* Being a more application-oriented person, I wish there were more elements in the paper to help me understand the practical consequences of the tight lower bound on machine learning applications. Does it lead to better representations? Can you show a real-world application that benefits from it in terms of accuracy?", "was_revised": true, "final_justification": "No changes in my review, explanations seem reasonable but outside of my domain of expertise, I would still have liked to see direct results on real-world applications, but I understand it may be out of scope for theoretical work?", "rebuttal": "We thank the Reviewer for their time. We are pleased to clarify the practical implications of our theoretical contributions.\n## Weaknesses:\n\n**Practical consequences of the tight lower bound on machine learning applications.**\n\nOur primary objective is to provide a theoretical foundation for a practice that is already widespread in representation learning: maximizing JSD-based information using discriminators. For instance, this optimization strategy is a cornerstone of frameworks like Deep InfoMax [1], which has been successfully applied to a wide range of downstream tasks, such as image classification [1], graph node classification [2], molecular property prediction [3], recommender\nsystems [4] and remote sensing [5]. While JSD has often been used empirically as a proxy for mutual information, its theoretical justification remained unclear.\n\nOur work closes this gap: we demonstrate that maximizing JSD corresponds to maximizing a tight lower bound on the true mutual information. This result strengthens the theoretical underpinnings of many existing methods and provides assurance that optimizing JSD discriminative losses is a principled and effective strategy for maximizing mutual information.\n\n## Questions\n> Does it lead to better representations? Can you show a real-world application that benefits from it in terms of accuracy?\n\nWe thank the reviewer for this important question. To further demonstrate the practical relevance of our results, we conducted additional experiments using the Information Bottleneck (IB) framework. The IB framework aims to learn compact yet informative representations from an input variable that are predictive of a target variable. We included new experiments where the objective is to maximize JSD rather than alternative variational bounds. Following prior work [6], we assess the learned representations obtained via JSD maximization on generalization performance, adversarial robustness, and out-of-distribution detection using the MNIST dataset.\n\nOur findings show that:\n\n* Maximizing JSD information outperforms alternative variational IB baselines that rely on alternative mutual information bounds.\n\n* These empirical results support our theoretical claims and confirm that maximizing JSD not only has solid theoretical grounding but also leads to practically robust representations.\n\n> What are the real-world implications of this work?\n\nThe real-world implication of our work is that methods aiming to maximize mutual information, particularly in representation learning, can confidently rely on JSD-based objectives as a mathematically justified and effective surrogate.\n\n[1] Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., & Bengio, Y. (2018). Learning deep representations by mutual information estimation and maximization. In: ICLR.\n\n[2] Veličković, P., Fedus, W., Hamilton, W. L., Liò, P., Bengio, Y., & Hjelm, R. D.  (2019). Deep Graph Infomax. In: ICLR\n\n[3] Stärk, H., Beaini, D., Corso, G., Tossou, P., Dallago, C., Günnemann, S., & Liò, P. (2022). 3D Infomax improves GNNs for Molecular Property Prediction. In: ICML.\n\n[4] Yu, J., Yin, H., Xia, X., Chen, T., Li, J., & Huang, Z. (2023). Self-supervised learning for recommender systems: A survey. IEEE Transactions on Knowledge and Data Engineering, 36(1), 335-355\n\n[5] Wang, Y., Albrecht, C. M., Braham, N. A. A., Mou, L., & Zhu, X. X. (2022). Self-supervised learning in remote sensing: A review. IEEE Geoscience and Remote Sensing Magazine, 10(4), 213-247.\n\n[6] Pan, Z.; Niu, L.; Zhang, J.; and Zhang, L. 2021. Disentangled Information Bottleneck. In AAAI"}, {"review_id": "s9tn5QqTfU", "reviewer": "Reviewer_zy6E", "summary": "The paper derives a lower bound on the KLD as a function of the JSD and then of CE. The bound can be used to lower bound the mutual information (MI). It is then argued that the maximization of such a lower bound can be used in representation learning, where the maximization of the MI is often the problem. Some numerical results are then reported.", "questions": "In addition to the above comments, I would recommend to:\n\n- Possibly provide some analysis on how tight the bound is, besides providing numerical results. The fact that the JS is limited while the KL is not may already lead to the conclusion that the JS is a lower bound. \n- Is the function \"E\" defined in Theorem 41. unique? \n- Better describe the content of Section 4.3 and elaborate on how the bound is exploited to estimate MI. As it stands, Section 4.3 is somewhat disconnected from the rest. It appears as if MI was estimated exactly as in GAN-DIME. Is this the case? Please be more explicit. \n-  The comparison between the method in Section 4.3 and the lower bound on MI is discussed only in the numerical results section. Again, here, I am not sure what is what. In particular, in the text of Section 5.2, GAN-DIME is said to be the \"best\" estimator, and it is mentioned several times, but the plots do not show it, nor report curves labeled with it. Such curves should be explicitly reported. \n- The comparison of MI estimation does not consider all estimators but only some of them. For instance, SMILE is missing. \n- How does the estimation of the MI with the lower bound compare with the method in Sect. 4.3 and with GAN-DIME ?  \n- No analysis of the variance of the proposed MI estimator that exploits the lower bound is reported. \n- What is the advantage of using the MI estimation with the lower bound on the KL w.r.t. to the other estimators ? \n- The paper emphasizes the value of the proposed lower bound for representation learning, i.e,. to enable maximization fo the MI. But then no results are reported about it. Can you report such results and verify that they are superior to the others ? \n- Minor thing. Double check the right term of equation (6): dp/dq ?", "limitations": "Yes,", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper provides a lovely result, i.e., a connection between the KLD (and therefore MI) and the JSD. The bound can be exploited to bound and therefore maximize MI, this is the rationale of the paper. The major limit is that it is not clear whether the estimation of the MI by exploitation of such bounds can lead to superior performance with respect to state-of-the-art estimators.\nWhat is the result of maximizing MI via exploitation of the bound is not sufficiently discussed.", "was_revised": true, "final_justification": "I appreciated the discussion during the rebuttal phase. The paper provides some new insight on how to lower-bound the MI. The tightness of the bound is not formally proved, although the numerical results, in the considered settings, show it.", "rebuttal": "We sincerely thank the Reviewer for the time and effort spent on reviewing our paper. We appreciate that the Reviewer recognizes the novelty of the connection between KL divergence, JSD, and cross-entropy, and its application to mutual information (MI) estimation. Below, we respond to the specific concerns and clarify the theoretical and empirical contributions of our work.\n\n## Weaknesses\n> *“It is not clear whether the estimation of the MI by exploitation of such bounds can lead to superior performance with respect to state-of-the-art estimators.”*\n\nOur primary goal is to theoretically justify a practice that is already widespread in representation learning: maximizing JSD-based information. While JSD has been used empirically as a proxy for MI (e.g., in the popular Deep InfoMax framework), our contribution shows that maximizing JSD does **maximize a tight lower bound** on MI. This connection clarifies the mathematical rationale behind JSD-based information methods, which often work well in practice.\n\nAs a result, our initial experiments evaluated how this JSD-based bound performs compared to other plug-in MI estimators (MINE, NWJ, and CPC) compatible with representation learning.\n\nImportantly, while alternative two-step MI estimators, such as $f$-DIME or SMILE, have been shown to demonstrate state-of-the-art bias-variance tradeoffs, they require retraining whenever the underlying joint distribution changes. Therefore, once trained, they cannot be reused in representation learning, since the distribution of the latent variables is evolving during optimization. For this reason, we initially excluded them from our experiments. However, we agree with the Reviewer that they offer useful context and therefore conducted additional experiments as detailed below.\n\n## Questions\n> *\"Possibly provide some analysis on how tight the bound is...\"*\n\nYes, the JSD–KLD lower bound we derive is the *tightest possible* between these two divergences. Specifically, it corresponds to the lower envelope of the joint range of $(\\mathrm{KL}, \\mathrm{JSD})$ pairs over Bernoulli distributions, and we provide an explicit analytic expression via its inverse mapping (Eq. 4).\n\nWhile our result holds for arbitrary distributions, one might expect a tighter bound when restricted to the mutual information setting, i.e., when comparing a joint distribution to the product of its marginals. Remarkably, as shown in Figure 2 of our manuscript, there exist discrete joint-marginal pairs that lie very close to our general bound. To address the Reviewer’s comment, we performed an analytical derivation of the KL and JSD divergences between the joint and product of marginals for fully dependent uniform categorical variables of dimension $k$ (i.e., when $p_{UV}$ is diagonal and $\\alpha = 1$). We found that these pairs lie *on* our bound for any $k\\in\\mathbb{N}^{+}$, providing an infinite family of distributions within the mutual information setting on our bound. This result will be included in the Appendix and referenced in the main text. We thank the Reviewer for this suggestion, which has helped us strengthen the contribution of our work.\n\n> *\"Is the function E in Theorem 4.1 unique?\"*\n\nWe thank the Reviewer for this question. Yes, the function $\\Xi$ is unique and defined via its inverse $\\Xi^{-1}$, which maps JSD to a lower bound on KL. We added a proof that $\\Xi^{-1}:\\mathbb{R}^+\\mapsto[0,\\log2)$ is bijective in the revised manuscript. We clarify this point in the revised manuscript (Section 4.1).\n\n> *\"Better describe Section 4.3 and its relation to GAN-DIME.\" [...] \"GAN-DIME is said to be the best estimator, but not shown in plots.\"*\n\nThank you for pointing this out. Section 4.3 indeed proposes a two-step estimator, which turns out to be equivalent to GAN-DIME, up to a linear transformation. We now clarify this equivalence explicitly in the revised manuscript.\n\n> *\"The comparison of MI estimation does not consider all estimators (e.g., SMILE).\"*\n\nWe have now added comparisons to SMILE and KL-DIME. While these cannot be directly optimized during the training of representation methods (as they require a two-step optimization procedure), we agree that they offer useful context. These new results have been added in Appendix D.2.4 and are reproduced below in Table 1. Our results show that the two-step estimator derived from our bound performs on par with or better than KL-DIME and SMILE, especially in high-MI or high-dimensional settings. \n\n> *\"No analysis of variance is reported.\"*\n\nWe have added a detailed variance and MSE analysis in the revised version (Appendix D.2.3), which is reported in rebuttal to Reviewer KvyC. The key findings are:\n* Our lower bound (JSD-LB) shows significantly lower variance than MINE and NWJ, especially at higher MI values and larger dimensions.\n* The MSE of our lower bound is lower than that of CPC and NWJ.\n\n> *\"What is the advantage of this method over others?\"*\n\nThe main advantage is practical usability: many state-of-the-art MI estimators (e.g., GAN-DIME, SMILE) require a two-step training pipeline that is not compatible with end-to-end training in representation learning. Our bound can be directly optimized via JSD, using a single discriminator, enabling stable, end-to-end training. This provides a principled rationale for existing JSD-based methods and introduces a theoretically grounded alternative to common variational estimators like MINE.\n\n> *\"No results are reported about the bound’s value in representation learning.\"*\n\nTo strengthen the bound's value in representation learning, we have added new experiments in the Information Bottleneck (IB) framework. Results in revised Appendix D.3.2 and reproduced in our response to Reviewer ifQp show that maximizing JSD as a proxy for MI in IB leads to superior performance compared to standard MI estimators.\n\n> *\"Minor: Equation (6): please check if it should be dp/dq.\"*\n\nIt should indeed be $dp/dq$. For example, for the KL-divergence: $f:t\\mapsto t \\ln(t)$.\n\n## Conclusion\nWe hope the clarification, additional proofs, and new experiments address the Reviewer's concerns.\n\n## Comparison with two-step MI estimators\n**Table 1:** Bias, variance, and mean squared error (MSE) of the two-step MI estimators using the joint architecture, when $b \\in$ {64,258}, for the Gaussian setting with varying dimensionality $d \\in $ {5,20}.\n||||||||**d=5**|||||||||**d=20**|||||||||\n|:---|:---:|:-:|:-:|:---:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|||||**b=64**||||||**b=256**||||||**b=64**||||||**b=256**|||\n||**MI**|**2**|**4**|**6**|**8**|**10**||**2**|**4**|**6**|**8**|**10**||**2**|**4**|**6**|**8**|**10**||**2**|**4**|**6**|**8**|**10**|\n**a) Bias**\n||SMILE|-0.23|-0.59|-0.73|-0.68|-0.65||-0.28|-0.65|-0.81|-0.78|-0.68||-0.08|-0.24|-0.25|-0.24|-0.16||-0.17|-0.38|-0.43|-0.39|-0.28|\n||KL-DIME|0.09|0.12|0.22|0.44|0.9||0.05|0.05|0.09|0.19|0.41||0.24|0.36|0.62|1.0|1.57||0.16|0.2|0.34|0.58|0.92|\n||GAN-DIME|0.1|0.1|0.13|0.24|0.09||0.06|0.05|0.07|0.12|0.01||0.26|0.38|0.54|0.67|0.72||0.19|0.26|0.35|0.48|0.5|\n||Our MI estim.|0.1|0.09|0.12|0.19|0.1||0.06|0.05|0.07|0.12|0.04||0.26|0.39|0.52|0.64|0.65||0.19|0.26|0.36|0.47|0.53|\n**b) Variance**\n||SMILE|0.08|0.13|0.22|0.33|0.72||0.03|0.07|0.1|0.18|0.3||0.12|0.16|0.27|0.46|0.77||0.06|0.07|0.12|0.2|0.32|\n||KL-DIME|0.06|0.07|0.08|0.11|0.17||0.03|0.02|0.02|0.03|0.05||0.1|0.11|0.15|0.19|0.22||0.05|0.03|0.04|0.05|0.07|\n||GAN-DIME|0.07|0.13|0.22|0.37|0.92||0.03|0.06|0.11|0.18|0.4||0.12|0.17|0.26|0.41|0.73||0.06|0.08|0.12|0.2|0.36|\n||Our MI estim.|0.07|0.13|0.22|0.36|0.93||0.03|0.06|0.1|0.2|0.43||0.12|0.17|0.28|0.44|0.73||0.06|0.08|0.12|0.2|0.32|\n**c) Mean Square Error**\n||SMILE|0.13|0.48|0.74|0.8|1.14||0.11|0.49|0.75|0.8|0.77||0.12|0.22|0.33|0.52|0.79||0.08|0.21|0.3|0.35|0.41|\n||KL-DIME|0.07|0.08|0.13|0.3|0.97||0.03|0.02|0.03|0.07|0.22||0.16|0.24|0.53|1.19|2.69||0.07|0.07|0.16|0.38|0.91|\n||GAN-DIME|0.08|0.14|0.23|0.43|0.92||0.04|0.07|0.11|0.2|0.4||0.19|0.32|0.55|0.86|1.25||0.09|0.15|0.24|0.43|0.61|\n||Our MI estim.|0.08|0.13|0.23|0.39|0.94||0.04|0.07|0.11|0.22|0.43||0.19|0.32|0.55|0.86|1.15||0.09|0.14|0.25|0.42|0.6|\n\n## Tightness analysis\n\nLet $(U,V)$ be a pair of uniform categorical distributions of dimension $k$ that are fully dependent, i.e. $P_{U,V} = \\text{diag}(P_U)$, where $P_U=\\frac{1}{k}\\mathbf{1}$. Then $\\text{I}\\_{\\text{KL}}\\left[U,V\\right] = \\Xi\\left( \\text{I}\\_{\\text{JS}}\\left[U,V\\right] \\right)$.\n\n**Proof:**\n\nLet $k\\in\\mathbb{N}^{+}$. We assume that $U$ and $V$ are fully dependent (i.e. $U=V$) uniform categorical distributions of dimension $k$. Therefore, the matrix representation $P\\_{U}$, $P\\_{V}$ and $P\\_{UV}$ are respectively $\\frac{1}{k}\\mathbf{1}\\_{k}$,  $\\frac{1}{k}\\mathbf{1}\\_{k}$ and  $\\frac{1}{k} I\\_{k}$ . Consequently,\n $P\\_{U}\\otimes P\\_{V} = \\frac{1}{k^2}\\mathbf{1}\\_{(k,k)} $.\n\nLet $M = \\frac{1}{2} ( P\\_{U} \\otimes P\\_{V} + P\\_{U,V}) $ be the mixture. $M_{i,j}=\\frac{1/k+1/k^2}{2}$ if $i=j$ and $M_{i,j}=\\frac{1}{2k^2}$ otherwise.\n\nLet's calculate $\\text{I}\\_{\\text{KL}}\\left[U,V\\right]$ and $\\text{I}\\_{\\text{JS}}\\left[U,V\\right]$.\n\n$$ \\text{I}\\_{\\text{KL}}\\left[U,V\\right] = \\sum_{i=1}^{k} \\frac{1}{k} \\log \\frac{1/k}{1/k^2} = \\log(k) $$\n\n$$  \\mathrm{I}\\_{\\text{JS}}[U, V]= \\frac{1}{2} \\left( \\sum\\_{i=1}^{k} \\frac{1}{k} \\log \\frac{1/k}{\\frac{1}{2}(1/k + 1/k^2)} + \\sum\\_{i=1}^{k} \\frac{1}{k^2} \\log \\frac{1/k^2}{\\frac{1}{2}(1/k + 1/k^2)}  + \\sum\\_{i=1}^{k} \\sum\\_{j=1\\text{ s.t. }  j \\neq i}^{k} \\frac{1}{k^2} \\log \\frac{1/k^2}{\\frac{1}{2k^2}} \\right)$$ \n$$  \\quad \\quad \\quad \\quad = \\frac{1}{2} \\left(\\log \\frac{2}{1 + 1/k} + \\frac{1}{k} \\log \\frac{2/k}{1 + 1/k}  +  \\frac{k-1}{k} \\log(2) \\right)$$ \n$$  \\quad \\quad \\quad \\quad = \\log(2) - \\frac{1}{2} \\left( \\log(1 + 1/k) - \\frac{1}{k} \\log \\left( \\frac{1/k}{1 + 1/k} \\right) \\right)$$\n$$  \\quad \\quad \\quad \\quad = \\Xi^{-1}(\\log(k)) $$\n\nTherefore: \n$$ \\text{I}\\_{\\text{JS}}\\left[U,V\\right]  = \\Xi^{-1}(\\text{I}\\_{\\text{KL}}[U, V]). $$"}, {"review_id": "i2E1RNgQuW", "reviewer": "Reviewer_ifQp", "summary": "This paper presents a novel theoretical connection between the Jensen–Shannon divergence (JSD) and the Kullback–Leibler divergence (KLD), two widely used statistical divergence measures in representation learning. The authors derive a new, tight, and tractable lower bound on KLD as a function of JSD. By specializing this bound to joint and marginal distributions, they demonstrate that maximizing JSD-based information increases a guaranteed lower bound on mutual information (MI). The theoretical contributions are supported by rigorous derivations and extensive empirical evaluations.", "questions": "Can the authors provide a sensitivity analysis or theoretical discussion on how the bound evolves under suboptimal discriminators?\n\nWould the bound remain tight and competitive when used in real-world settings, such as self-supervised representation learning based model pretraining.\n\nMutual information estimation plays a central role in the Information Bottleneck (IB) principle, a widely adopted framework for removing redundant information from learned representations. To validate the practical benefits of JSD-LB, I suggest the authors conduct controlled experiments where it replaces conventional MI estimators in IB-based methods (e.g., [1,2]).\nSpecifically, evaluating its impact on downstream tasks such as generalization, adversarial robustness, out-of-distribution detection, and supervised disentanglement (e.g., swapped generation) would help quantify whether improved MI estimation of JSD-LB translates into better representation quality.\n\nPlease consider adding wall-clock comparisons, memory usage analyses, etc, especially in complex higher-dimensional representation space.\n\n[1]Deep Variational Information Bottleneck, ICLR 2017\n\n[2]Disentangled Information Bottleneck, AAAI 2021", "limitations": "The authors have discussed the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\nThe paper provides a mathematically rigorous lower bound of KLD in terms of JSD, addressing the theoretical gap in the understanding of discriminative MI estimators.\n\nThe proposed bound demonstrates impressive tightness in both theoretical derivation and empirical validation. Extensive experiments show that the estimator achieves consistently low variance and competitive performance compared to state-of-the-art MI estimators.\n\nWeakness\n\nWhile the paper presents a rigorous theoretical derivation connecting JSD and KLD, it assumes access to an optimal discriminator or models with infinite capacity. However, in practical learning scenarios, discriminators are implemented with finite-capacity neural networks trained on limited data, which may affect the tightness of the bound.\n\nThe experiments are largely restricted to synthetic setups where mutual information can be analytically computed. While these controlled settings are useful for theoretical validation, they limit the understanding of  JSD-LB’s performance in practical representation learning tasks.\n\nMoreover, the evaluation focuses on MI estimation itself but does not assess whether the proposed method improves actual representation learning outcomes, which is more critical in real-world applications.\n\nThe paper lacks a detailed comparison of computational cost or training efficiency relative to existing mutual information estimators.", "was_revised": true, "final_justification": "The authors' rebuttal effectively addresses my concerns about assumption, real-world applicability and efficiency.\n\nTherefore I tend to maintain my score.", "rebuttal": "We thank the Reviewer for their positive and detailed feedback. We are glad to see that the key strengths and contributions of the paper, especially the theoretical connection between JSD and KLD and the tightness of the proposed bound, were clearly appreciated. Below, we address the concerns and suggestions raised.\n \n## Weaknesses:\n\n**On the assumption of an optimal discriminator.** We agree that the tightest bound assumes access to an optimal discriminator and recognize that, in practice, discriminators are trained with limited capacity and finite data. In the revised manuscript, we have added a discussion (Section 4.3) on how the tightness of the bound may degrade under suboptimal discriminators. Specifically, we note that even when the discriminator is not optimal, its output remains a lower bound on JSD and thus still leads to a valid (though looser) lower bound on mutual information. Moreover, we show in the Appendix that we can estimate the gap between the true JSD lower bound and the one obtained from the cross-entropy $\\mathcal{L}_{CE}$ as:\n\n\\begin{equation} \\Xi(\\text{D}\\_{\\text{JSD}}\\left[p\\_{UV}  || p\\_{U} \\otimes p\\_{V}\\right]) - \\Xi(\\log2 - \\mathcal{L}\\_{CE}) \\approx \\delta \\cdot\\Xi'(\\text{D}\\_{\\text{JSD}})>0 \n\\end{equation}\n\nwhere $\\delta=\\mathbb{E}\\_{(u,v)\\sim \\tilde{p}\\_{UV}} \\text{D}\\_{\\text{KL}}[\\tilde{p}\\_{Z|UV}(u,v)\\;||q\\_{\\theta}(z| u,v)]$ is the KL divergence between the true posterior of the mixture model defined in Eq. (21) and the parametric approximate posterior $q\\_{\\theta}$.\n\n**Real-world settings and representation learning outcomes** To demonstrate the utility of our JSD-LB in practical representation learning tasks, we have added a new set of experiments in the revised Appendix D.3.2 using the Information Bottleneck (IB) framework. Specifically, we use the JSD information as a surrogate for MI. Following the experimental setting from Pan et al (2021) [1], we evaluate its impact on downstream tasks such as generalization, adversarial robustness, and out-of-distribution detection using the MNIST dataset.\n\nOur results, reproduced below in Table 1, show that replacing the MI estimator with JSD-LB leads to state-of-the-art performance in terms of generalization, adversarial robustness, and out-of-distribution robustness. Specifically, it outperforms other Variational Information Bottleneck objectives [2,3,4] and the Disentangled Information Bottleneck approach [1]. This set of experiments shows that maximizing JSD has practical value for representation learning.\n\n**Computational cost and training efficiency**\nWe thank the Reviewer for pointing this out. In the revised Appendix D.2.5, we now report the time requirements between JSD-LB and other estimators such as MINE and NWJ. Table 2 from the response to Reviewer KvyC shows the results. As in Letizia et al. (2024), we found that the influence of the MI lower-bound objective functions on the algorithm’s time requirements is minor.\n\n## Questions\n\n> Can the authors provide a sensitivity analysis or theoretical discussion on how the bound evolves under suboptimal discriminators?\n\nPlease find above a discussion of the sensitivity and a theoritical discussion on how the bound evolves under suboptimal.\n\n> Would the bound remain tight and competitive when used in real-world settings, such as self-supervised representation learning based model pretraining.\n\nEvaluating the tightness and competitiveness of our bound in real-world scenarios, such as self-supervised representation learning, would require access to the intractable true mutual information. While our main objective is to provide a theoretical justification for why maximizing JSD-based information effectively increases mutual information, such empirical evaluation is left for future work.\n\n> To validate the practical benefits of JSD-LB, I suggest the authors conduct controlled experiments where it replaces conventional MI estimators in IB-based methods (e.g., [1,2]). \n\nWe thank the Reviewer for this suggestion, which is indeed an excellent use case to demonstrate the practical value of our work. We conducted additional experiments where the JSD-LB was replaced with conventional MI estimators in IB-based methods, reported in Tables 1, 2, 3, and 4 below. The results show the effectiveness of our approach, outperforming other MI estimators and reaching comparable performance to disentanglement approaches. These results will be included in our revised manuscript.\n\n> Please consider adding wall-clock comparisons, memory usage analyses, etc, especially in complex higher-dimensional representation space.\n\nSee our response above and Table 2 from the response to Reviewer KvyC.\n \n\n## Variational Information Bottleneck Experiments\n\nInformation bottleneck results, where the objective is to compress the source random variable $U$ to a \"bottleneck\" random variable $T$, keeping the information relevant for predicting the target random variable $V$. This corresponds to the following maximization objective:\n$$I [ T; V ] - I [T;U] $$\n\nIn this set of experiments, we reproduce the experiments from [1] on MNIST. Specifically, the source random variable $U$ corresponds to an image and $V$ to its label. We employ the same model architecture to encode the variable $U$ and obtain the variable $T$, and directly report the corresponding results of [1].\n\nIn our setting (JSD-LB), $I [ T; V ]$ is maximized by maximizing the JSD via the cross-entropy loss.\n\n- **Information compression**\n\n   **Table 1:** Comparison of $I[T;V]$ obtained using non-disentanglement bottleneck approaches [2,3,4], with disentanglement [1] and via JSD optimization (ours).\n   \n  | **Dataset**       | **Training** | **Testing** | \n  |:--|:-:|:--:|\n  | H(V)            | 2.30         | 2.30        | \n  | VIB [2]              | 2.01         | 1.99        | \n  | NIB [3]              | 1.95         | 1.97        | \n  | squared-VIB [4]      | 1.94         | 1.95        | \n  | squared-NIB [4]      | 1.98         | 1.99        | \n  | DisenIB [1]      | 2.25     | 2.17    | \n  | JSD-LB       | **2.30**     | **2.29**    | \n\n\n- **Generalization**\n\n  Generalization performance in Table 2 is evaluated by the classification mean accuracy on MNIST test set after training the model on MNIST training set \n\n   **Table 2:** Generalization performance (%) on MNIST dataset.\n    | Method | *VIB [2]* | *NIB [3]* | *squared-VIB [4]* | *squared-NIB [4]* | *DisenIB [1]* | *JSD-LB* |\n    | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n    | Testing | 97.6 | 97.2 | 96.2 | 93.3 | 98.2 | **98.8** |\n\n\n\n- **Robustness to adversarial attack**\n\n  To evaluate adversarial robustness, we adopt a standard one-step gradient-based attack method following [5]. After training the model on the MNIST training set, we assess robustness by computing the average classification accuracy on adversarially perturbed inputs derived from both the training and test sets. These adversarial examples are created by applying a single gradient ascent step on the input with respect to the loss. The perturbation strength is controlled by the parameter $\\epsilon$, which we vary over the values {0.1, 0.2, 0.3}, corresponding to different pixel-wise perturbation magnitudes, as commonly done in prior work [5].\n\n    **Table 3:** Adversarial robustness performance (%) on MNIST dataset.\n    | | Method | *VIB [2]* | *NIB [3]* | *squared-VIB [4]* | *squared-NIB [4]* | *DisenIB [1]* | *JSD-LB* |\n    | :----: | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n    | **a) Training set** |\n    | | &epsilon; = 0.1 | 74.1 | 75.2 | 42.1 | 61.3 | 94.3 | **99.5** |\n    | | &epsilon; = 0.2 | 19.1 | 21.8 | 8.7 | 24.1 | 81.5 | **96.0** |\n    | | &epsilon; = 0.3 | 3.5 | 3.2 | 5.9 | 9.3 | 68.4 | **91.4** |\n     | **b) Testing set** |\n    | | &epsilon; = 0.1 | 73.4 | 75.2 | 42.7 | 62.0 | 90.2 | **94.6** | \n    | | &epsilon; = 0.2 | 20.8 | 23.6 | 9.2 | 24.5 | 80.0 | **89.6** | \n    | | &epsilon; = 0.3 | 4.2 | 3.4 | 5.9 | 9.9 | 67.8 | **86.1** | \n\n- **Out-of-distribution detection**\n\n    In line with prior work [1], we evaluate out-of-distribution (OOD) detection using a synthetic dataset composed of 10,000 samples drawn independently from a standard Gaussian distribution $\\mathcal{N}(0, 1)$. The goal is to determine whether the model can reliably distinguish in-distribution data from these OOD noise samples. For evaluation, we adopt common OOD detection metrics including False Positive Rate at 95% True Positive Rate (FPR95), Detection Error, Area Under the ROC Curve (AUROC), and Area Under the Precision-Recall Curve (AUPR).\n    \n   **Table 4:** Distinguishing in- and out-of-distribution test data for MNIST image classification (%).  ↑ (resp., ↓) indicates that a larger (resp., lower) value is better.\n\n   | Method | *VIB [2]* | *NIB [3]* | *squared-VIB [4]* | *squared-NIB [4]* | *DisenIB [1]* | *JSD-LB* |\n    | :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n    | TPR95 ↓ | 27.4 | 34.4 | 49.9   | 47.5  | **0.0** | **0.00** |\n    | AUROC ↑  | 94.6 | 94.2 | 86.6 | 85.6 | 99.4| **100.0** |\n    | AUPR In ↑  | 94.8 | 95.2 | 83.5| 83.3 | 99.6 | **99.9** |\n    | AUPR Out ↑ | 93.7 | 91.8 | 83.2 | 83.3 | 98.9 | **99.9** |\n    | Detection Err ↓ | 11.5 | 11.9 | 20.0 | 15.0 | 1.7 | **0.1** |\n\n[1] Pan, Z.; Niu, L.;  Zhang, J.; and Zhang, L. 2021. Disentangled Information Bottleneck. In AAAI\n\n[2]  Alemi, A. A.; Fischer, I.; Dillon, J. V.; and  Murphy, K. 2017. Deep variational information bottleneck. In ICLR.\n\n[3] Kolchinsky, A.; Tracey, B. D.; and Wolpert, D. H. 2019. Nonlinear information bottleneck. Entropy 21(12): 1181.\n\n[4] Rodrıguez Galvez, B.; Thobaben, R.; and Skoglund, M. 2020. The Convex Information Bottleneck Lagrangian. Entropy 22(1): 98.\n\n[5] Goodfellow, I. J.; Shlens, J.; and Szegedy, C. 2015. Explaining and Harnessing Adversarial Examples. In ICLR."}, {"review_id": "dujeuMpG2A", "reviewer": "Reviewer_VCK3", "summary": "This paper explores the connection between Kullback-Leibler divergence (KLD) and Jensen-Shannon divergence (JSD), with a focus on approximating mutual information (MI).  The authors derive a tight lower bound on KLD in terms of JSD.  The authors further propose to lower bound this surrogate with a variational approximation.   Finally, the paper empirically demonstrates that the proposed lower bound on MI is tight in a simple discrete distribution, and that the approximation is effective in a range of more challenging continuous settings.", "questions": "The authors refer to the proposed bound as \"optimal\" in several instances.  Yet, it is unclear what the authors mean by \"optimal bound\".  Is this just because it is tight?  If so then it is better to say that.  I also don't know that it is demonstrated as tight beyond a single example for a discrete model (and empirically at that).", "limitations": "Limitations are not sufficiently discussed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\nThis study has broad practical utility as it has become common practice to substitute JSD for KLD in MI estimation, particularly for representation learning tasks.  This practice is done despite a lack of evidence that there exists any connection between these two divergence measures.  This paper establishes that there is indeed a bound relationship, and furthermore that the JSD lower bound can be optimized in practical settings.\n\n**Weaknesses**\nThe proposed method optimizes a lower bound of a lower bound on MI (Eq. (5)).  The properties of this lower bound are not made entirely clear beyond (empirical) tightness in a simple discrete model (Sec. 5.1).  Moreover, this relies on an approximation of the bound proposed in Eq. (20).  However, it remains unclear why Eq. (20) is a valid approximation.  Despite the claim \"details in Appendix C.3\", the actual details provided are somewhat lacking as the approximation is presented without justification.  Nevertheless, Fig. 6 does qualitatively suggest that this is an accurate approximation, and this reviewer is willing to believe the claim.\n\nThe empirical results in Sec. 5.2 are more convincing as the methodology is demonstrated on a wider set of more practical examples.  Nevertheless, the behavior of the proposed estimator can be somewhat unreliable in certain settings.  This is particularly true in the Student's T distribution (Fig. 4) where the estimator is extremely high variance to the point that it seems unusable.  Indeed, the proposed estimator has the highest variance among all baselines in this regime.  \n\nThere are some mistakes in Sec. B.4 and also points that require clarification.  The definition of the conditionals in Eq. (79) seem to be reversed.  The second equality in Eq. (80) is unclear unless $p(z=0) = p(z=1)$ so that they cancel in the numerator and denominator, but it isn't obvious why this condition should hold.\n\nA higher-level, but somewhat minor, comment on the presentation: much of the motivation behind the provided method is that it is useful in representation learning.  Indeed, the authors mention representation learning several times in the abstract and introduction.  However, no representation learning is demonstrated in this paper, leaving the conclusion that this method is useful for representation learning entirely unsupported.\n\nSome detailed comments:\n* L43: \"KLD-based mutual information\" is a bit of a misnomer as MI is KLD-based by definition\n* L64: \"non-parametric\" should not be hyphenized (admittedly this is a stylistic preference)\n* L167: Shouldn't the domain be over a closed and half-open set $[0,1] \\times (0,1]$?\n* L212: The distribution $\\tilde{p}(z\\mid u,v)$ is introduced without definition\n* Fig. 2: I found this figure completely uninformative as it was unclear what I am looking at.  What is the color gradient region referring to?  How do I know if this is a good result?", "was_revised": true, "final_justification": "The authors did an adequate job of addressing my concerns in their rebuttal.  In particular, they clarified a few points of confusion and (most importantly) included an additional set of experiments on representation learning.  I think this latter set of experiments would really strengthen the paper and I would encourage the authors to report these results in the main text in a final version of the manuscript.", "rebuttal": "We thank the Reviewer for their thoughtful and detailed review. We are pleased that the theoretical rigor and practical relevance of our work were appreciated. In particular, we are glad that the Reviewer acknowledged that this work closes a gap between the practice and the theory. We address the raised concerns through additional experiments and clarifications, detailed below.\n\n## Weaknesses\n\n**Approximation of the Bound**\n\nWe are happy to clarify how we derived our approximation of the $\\Xi$ function. We began by observing that the equality $x = \\Xi^{-1}(e^{-y})$, where $\\Xi^{-1}$ is known analytically, leads to:\n\n$$\n\\mathbb{L}\\left(\\frac{1}{2} \\cdot (\\frac{x}{\\log 2} + 1)\\right)=\\log \\left( \\frac{\\frac{x}{\\log 2} + 1}{1 - \\frac{x}{\\log 2}} \\right)=  \\log \\left( \\frac{4 \\log 2}{(1 + e^{-y}) \\log(1 + e^{-y}) + y e^{-y}} - 1 \\right) .\n$$\n\nThen, we found that the first-order Taylor polynomial of the right term near $y=0$ is $y\\mapsto y$. This led us to approximating $\\Xi$ with a logit. We then conducted a grid search and found that scaling the logit function by a factor of 1.15 led to the best median approximation error over $x \\in (0,1)$, i.e.:\n\\begin{equation}\n    \\Xi(x) \\approx 1.15 * \\mathbb{L}\\left(\\frac{1}{2} \\cdot (\\frac{x}{\\log 2} + 1)\\right) .\n\\end{equation}\n\nImportantly, $\\Xi$ is a strictly increasing function. As a result, maximizing $\\Xi(\\text{I}\\_{\\text{JS}}[U, V])$ is equivalent to directly maximizing $\\text{I}_{\\text{JS}}[U, V]$. For this reason, the approximation of \\$\\Xi\\$ is not used during optimization; it is included solely for the reader’s interest and for interpretability.\n\nIn the revised manuscript:\n\n* We explain how this approximation arises from the Taylor expansion of the KL divergence in terms of JSD.\n* We include a plot comparing the approximation to the numerically evaluated $\\Xi$, highlighting the error.\n* We explicitly state that the optimization doesn't rely on the approximation of $\\Xi$.\n\n\n**High Variance in the Student’s T Experiments (Fig. 4)**\n\nWe agree that the Student’s T distribution poses challenges due to its heavy tails. While our method does exhibit higher variance in this specific setting, we note that this behavior is shared across estimators, and the challenge is characteristic of the distribution rather than the method alone.\n\n**Usefulness for Representation Learning**\n\nTo better support the applicability of our method in real-world scenarios, we included a new representation learning benchmark based on the Information Bottleneck (IB) framework (Appendix D.3.2). IB seeks to learn compressed representations that retain information relevant to predicting a target variable.\n\nFollowing prior work (e.g., Pan et al., 2018), we evaluate the resulting representations in terms of generalization, adversarial robustness, and out-of-distribution detection on MNIST.\n\nThe new results replicated in response to Reviewer ifQp show that:\n\n* Replacing conventional MI estimators with JSD-LB within the IB framework leads to superior performance on various downstream tasks, including generalization, adversarial robustness, and out-of-distribution detection.\n* This supports our claim that maximizing JSD is both a theoretically grounded and practically effective strategy for representation learning.\n\n**Clarifications and Corrections**\n\n* **Eqs. (79) and (80):** Thank you for pointing this out. We have corrected the reversed definitions in Eq. (79). Regarding Eq. (80), we clarified that $Z$ is a Bernoulli variable with parameter $\\frac{1}{2}$, ensuring equal marginal probabilities $p(z=0) = p(z=1)$, which justifies the cancellation in the expression.\n\n* **Fig. 2:** The purpose of this figure is to illustrate how tight our bound is for pairs of uniform categorical variables of varying dimensionality $k$, where both $(\\text{I}\\_{\\text{JS}}, \\text{I}\\_{\\text{KL}})$ can be computed analytically. Specifically, we show how this pair evolves as the joint distribution becomes more dependent.\nThe color gradient encodes the number of categories $k$: higher $k$ values correspond to trajectories that reach higher values of both $\\text{I}\\_{\\text{KL}}$ and $\\text{I}\\_{\\text{JS}}$. Each colored curve represents the trajectory of $(\\text{I}\\_{\\text{JS}}, \\text{I}\\_{\\text{KL}})$ as the joint distribution $p\\_{U,V}^{(\\alpha)}$ transitions from the independent case ($\\alpha = 0$) to the fully dependent case ($\\alpha = 1$).\nThe red curve corresponds to the theoretical lower bound we derive between $\\text{I}\\_{\\text{JS}}$ and $\\text{I}\\_{\\text{KL}}$, which is the tightest for arbitrary distributions. This figure demonstrates that the end of each trajectory lies exactly on the boundary. As suggested by Reviewer zy6E, we will include an analytical proof of this fact. Therefore, our bound is also tight in the mutual information setting, i.e., when comparing a joint distribution to the product of its marginals. We will revise the figure caption and the main text to make this interpretation more explicit.\n\n* **L43:** We revised the language to avoid the phrase “KLD-based MI,” instead referring to “standard MI, defined via KL divergence.”\n\n* **L64:** Corrected to “nonparametric,” without a hyphen.\n\n* **L167:** Thank you for pointing this out. The correct full domain is $[0,1] \\times (0,1) \\cup \\\\{ (0,0),(1,1) \\\\} $. Therefore, the the lower triangle is $\\Omega=\\\\{(\\mu,\\nu) \\in [0,1]\\times(0,1] \\mid \\nu \\leq \\mu\\\\}$. We corrected this in L165, L167, L169, L420, L444 and L446.\n\n* **L212:** The definition of the distribution $\\tilde{p}(z|u,v)=\\frac{\\tilde{p}(z)\\tilde{p}(u,v|z)}{p(u,v)}$, where each term is defined in Eq. (21) and (22) has been added when first introduced.\n\n## Questions\n> **It is unclear what the authors mean by \"optimal bound\".**\n\nIn this work, we derive the *tightest possible lower bound* between the Jensen–Shannon divergence (JSD) and the Kullback–Leibler divergence (KLD) by analyzing the joint range between KLD and JSD. This means that our bound is saturated. i.e., equality is achieved for a family of distributions (right edge $S_3$ of our triangle). In contrast to classical inequalities such as Pinsker’s inequality, which provides a loose bound, our bound is attainable and therefore the tightest.\n\nAlthough our bound is derived in full generality for arbitrary pairs of distributions $p$ and $q$, one might expect that restricting to structured pairs, such as joint and product-of-marginals (as in mutual information), could allow for a tighter bound. However, through empirical investigations on discrete distributions, where both JSD and KLD between joint and product-of-marginals can be computed exactly, we observed experimentally that such pairs already lie near the boundary of our derived lower bound (Figure 2). In our response to Reviewer zy6E, we proved that we can find an infinite family of joint and marginal discrete distributions **on** this bound. This suggests that our bound is also the tightest in the context of mutual information estimation."}]

Author rebuttal:

Meta-review: This paper derives a new lower bound on KL divergence in terms of Jensen-Shannon divergence (JSD), both of which are important tools in machine learning, and then estimates the mutual information (MI) based on the bound and the variational representation of the JSD. The MI estimator demonstrates favorable bias-variance trade-offs on synthetic datasets as compared to common baselines.

All reviewers recognize the theoretical contributions as novel and rigorous, providing important results for JSD-based objectives in representation learning, and are supportive of acceptance. In the rebuttal, the experiments were extended to include a wider array of baselines, including SMILE, and higher dimensional settings. The authors confirmed its advantages in the empirical bias-variance trade-off.

As a notable weakness, it remains unclear in theory how tight these bounds are, although verified empirically, noting that the envelope of the joint range of $f$-divergences focuses on worst-case distribution pairs. Related claims should be stated more cautiously. Besides the reviewers' comments, please support Theorem 3.1 with a more accurate reference.

Overall, the strengths outweigh the weaknesses. This work is well written and presents generalizable tools from information geometry that can be used in other scenarios as well. I recommend acceptance.
Decision: Accept (poster)