Neural Mutual Information Estimation with Vector Copulas
Authors: Yanzhi Chen
Zijing Ou
Adrian Weller
Michael U. Gutmann
Venue: NeurIPS 2025 poster ()
Keywords: mutual information
flow-based models
copulas
vector ranks
Primary area: general_machine_learning
TL;DR: A mutual information estimator based on recent vector copula theory, which explicitly disentangles the marginal distributions and dependence structure in estimation.

Abstract: Estimating mutual information (MI) is a fundamental task in data science and machine learning. Existing  estimators mainly rely on either highly flexible models (e.g., neural networks), which require large amounts of data, or overly simplified models (e.g., Gaussian copula), which fail to capture complex distributions. Drawing upon recent vector copula theory, we propose a principled interpolation between these two extremes to achieve a better trade-off between complexity and capacity. Experiments on state-of-the-art synthetic benchmarks and real-world data with diverse modalities demonstrate the advantages of the proposed method.

Neural Mutual Information Estimation with Vector Copulas
Yanzhi Chen', Zijing Ou?, Adrian Weller Michael U. Gutmann" University of Cambridge, "Imperial College London, *Alan Turing Institute, *University of Edinburgh
Abstract
Estimating mutual information (MI) is fundamental task in data science and machine learning. Existing estimators mainly rely on either highly flexible models (e.g., neural networks), which require large amounts of data, overly simpli- fied models (e.g., Gaussian copula), which fail to capture complex distributions. Drawing upon recent vector copula theory, we propose principled interpolation between these two extremes to achieve a better trade-off between complexity and capacity. Experiments on state-of-the-ar synthetic benchmarks and real-world data with diverse modalities demonstrate the advantages of the proposed estimator.
Introduction
Mutual information (MI) is a fundamental measure of the statistical dependence between random variables (RVs). Compared to other dependence measures, MI stands out due its equitability and generality [1, 2]: it can capture non-linear dependence of any form and can handle RVs with any dimensionalities. rendering a powerful measure for quantifying statistical dependence. science, MI is widely used to analyze the relationships between protein sequences [3] and gene profiles [4, 5], as well as assess feature importance and redundancy [6]. In machine learning, MI broadly serves as learning objective and regularizer [7, 8, 9, 10, 11, 121, with diverse applications to representation learning [7, 8, 13, 14], generative modeling [10], fairness and privacy [15, 16), etc.
data
wide range of powerful, neural MI estimators have been developed [17, 18, 19, 20, 21, 23]. Most of these estimators rely single, unconstruined network approximate certain quantities--uchh as the joint density p( xx y) or the density ratios p(x, neural networks universal functional approximators can, in theory, approximate arbitrary functions given sufficient data [24, 25], in practice we often only have small set of data. Indeed, theoretical studies have shown that such distribution-free treatment of MI estimation will inevitably suffer from requiring an exponential sample size [26. 27, 28, 29J. A straightforward remedy is to restrict the model simpler classes-for instance, assuming that the data is approximately Gaussian. However, these assumptions are often overly simplistic to capture complex distributions in reality.
MI estimation. While
Recent advances in vector copula theory [30] offer promising avenue for addressing this dilemma. Vector copula theory extends classical copula theory [31] by generalizing it from univariate to vector marginals. It reveals that the multivariate marginals and the dependence structure (i.e., the vector copula) of joint distribution are fully disentangled. This disentanglement motivates a more fine-gran -reinnd way for making assumption in MI estimation, where we impose lightweight yet reasonable assumptions solely on the vector copula rather than on the entire distribution. Crucially, the complexity of the vector copula can be adaptively adjusted through efficient vector copula selection, allowing for an optimal trade-off' between capacity and complexity. Experiments on state- of-the-art art synthetic benchmarks and real-world data demonstrate the competitiveness of our estimator against ofte-of-tth--a estimators. In summary, the main contributions of this work are three-fold:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

p(x)
p(ux)
p(uy)
p(y)
I(XY) ([ uy]
Figure 1: Overview of the proposed vector copula-based MI estimator (VCE), which explicitly disentangles the modeling of marginal distribution and dependence structure (i.e., the vector copula). VCE first respectively computes the vector ranks ux and uy corresponding to the two marginal variables X and with flow models (T). It then finds the vector copula c C from the vector copula pool C that best matches with the joint distribution p(ux uy) of the estimated vector ranks (2). Mutual information [[;; Y) is computed as the negative differential entropy of the vector copula c, which itself is irrelevant to the two marginal distributions p(x) and p(y).
We develop swvide-and-oonuee MI estimator based recent vector copula theory, which explicitly disentangles marginal distributions and dependence structure in MI estimation;
We reinterpret existing estimators through the lens of vector copula, revealing that they correspond to varying parameterization and learning strategies of vector copula with various trade-offs;
We provide consistency and error analysis of our estimator, along with extensive empirical evalua- tion diverse test cases covering multiple modalities, marginal patterns and dependence structures.
Code containing both our method and state-of-the-art neural estimators is available in [github repo].
2 Preliminaries
Throughout this work, we use upper case letters (e.g. X) to denote random variables and lower case letters (e.g. x) to denote their instances. We use 24 [0, 1] or μ to denote the uniform distribution on [0, 1]d and use A to denote Gaussian distribution on Rdd V denotes the gradient and Jxy denotes the Jacobian of y w.r.t x. The symbol # denotes the push-forward operation.
2.1 Mutual information and estimation
The mutual information (MI) between variables X and Y is defined as the Kullback-Leibler (KL) divergence between the joint distribution p(x, y) and the product of marginal distributions p(x)p(y):
p(x,y)
I(x:Y) = KLLp(x y) (ecr))pp((((p)))) (xx))(())]] = log p(x)p(y)
(1)
In this work, we consider estimating (X; from an empirical dataset D = (x(() y0 Several neural network-based methods have been developed for MI estimation:
Generative estimators. These methods leverage generative models to approximate the various distributions in (1) or their equivalents, and use the learned generative models to construct an MI estimate [17, 29, 18, 32, 19]. The accuracy of generative estimators crucially depends the quality of the learned generative models. Simpler models (e.g. Gaussian copula) are easy to learn but may fail to adequately capture the true data distribution [33, 34]. In contrast, complex models (e.g. flow-based models [35, 36, 37, 38) and diffusion models [19]) offer greater expressiveness but can challenging to optimize, in particular if the amount of data is insufficient or the data dimensionality is high.
Discriminative estimators. These methods train neural network f with samples x. y p(x, y) and samples x, ~ p(x)p(y) to estimate the density ratio p(x. y)/p(x)p(y) [20, 21, 39, 40, 41, 32, 23). Once trained, the learned density ratio can either be used in (1) or the Donsker- Varadhan (DV) representation [42] to obtain an MI estimate. Discriminative methods avoid directly modeling densities, however they are prone to the curse of high-discrepancy [40, 41, 29, 32), which occurs p(x,y y) and p(x (())(()) differ significantly- - for instances, cases with high MI or high-dimensional data. Several advanced methods were proposed to alleviate this issue, including clipping the network outputs [32], introducing reference distributions [41]. avoiding computing the partition function [23].

---

2.2 Vector copula
Vector copula theory. The recent vector copula theory [30] provides principled framework for modeling and analyzing the dependence between multivariate random variables. It extends classical copula theory [31] by considering 'vector' marginals. We begin by the concept of vector ranks:
Definition 1 (Vector rank). Let p be an absolutely continuous distribution on R2 with support in convex set. Let be the uniform distribution on [0, 1] There exists convex function ab such that V = p-ppand and #pp J.. V is called the vector rank associated with p [43].
When d = 1, vector rank reduces to standard scalar rank. Intuitively, vector rank transforms multi- variate distribution p to a (multivariate) uniform distribution μ.. entirely removing its characteristics.
In the text below, we slightly overload this definition and use the term "vector rank' to refer to both the vector rank function u(-) and also the corresponding random variable induced by this function.
Definition 2 (Vector copula). Let ux and uy be the vector ranks corresponding to p(x) and p(y) respectively. A vector copula C(ux, uy is cumulative distribution function on [0, 1]dx with uniform marginals on C(ux) = 0/0, 1]dx and C(uy) U[O, 1)dr, The probubilistic density function corresponding C is called vector copula density and is denoted as c(ux, uy) (30].
Given the above definition, we have the following result [30] generalizing the Sklar theorem [31].
Theorem 1 (Vector Sklar Theorem). Let X E R and Y E Rd be two random variables with joint distribution y)x,y) on Rdx+dr, For any absolutely continuous distributions p(x. y) with support in a convex set, there exist an unique function cl., ))) such that
p(x,y) = (((x))(()c(ux,ee)
(2)
where ux and uy are the vector ranks computed for and y respectively. The fanction e equals to the vector copula density associated with uxx and uy [30].
The vector Sklar theorem suggests that for a distribution pxx,y), . its marginal distributions and the joint dependence structure are entirely disentangled, with the latter fully characterized by the vector copula density c. Note that here we focus on the case of two RVs; we refer to [30] for general cases.
Instances of vector copula. We discuss several instances of vector copula related to our work. One important instance is the vector Gaussian copula [30]. This model assumes that the joint dependence structure admits a Gaussian structure, with its vector copula C-V being
CN (ux,uy)
(())) (uy))
(3)
where E [[Ix Exyl, [Ex,, Iy]] is p.s.d matrix whose blocks Ix E Rdxxxx and Iy Rd xdy are identity matrices. 4(-) is the cumulative distribution function of multivariate normal distribution and o(-) is the (element-wise) cumulative distribution function of univariate normal distribution. Equivalently, vector Gaussian copula can be defined by its data generation process: e ~ N(E E). ux = Φ(Esax). uy = Φ(E>dx), ). with égdx and E-dx being the first dx and the remaining dimensions of e respectively. An analytic expression for e can be derived accordingly.
Other useful instances of vector copula include t-vector copula, Archimedean vector copula and Kendall vector copula, which correspond different inductive biases about the dependence structure.
3 Methodology
In this section, we propose new mutual information (MI) estimator based vector copula theory. The core of our method is Theorem 2, which establishes a connection between MI and vector copula: Theorem 2 (MI is vector copula entropy). The mutual information [(X; Y) is the negative differential entropy of the vector copula density:
I(X;Y) = -H([-uuxcx uy
(4)
where ux and uy are the vector ranks corresponding to p(x) and ply) respectively.
Proof: Please refer to Appendix A.
3

---

Algorithm Vector copula MI estimate (VCH)
Algorithm Vector copula MI estimate" (VCE")
Input: data D = (x(0) y( Ji=1
Input: data D = fxx(") , y())
===1
Output: estimated Y
Output: estimated (YX;Y)
Parameters: flows fx copulas {er, (.M)
Parameters: flows fx. fy, ratio estimator r
Initialization: D Draain U Daat, K 1,
Initialization: reference copula c', D'
Marginal distributions learning learn fx with Dx = { by FM; learn fy with Dy = (1 ) by FM; for i in to n. do
Marginal distributions learning learn fx with Dx = [x
learn fy with Dy = {y (8) Ji=1 by FM; ) f=
by FM;
for in 1 n do
compute ú rank(J (fx (x(
compute ú
rank (fx (x ();;
compute u = rank(fy (y(()))) end for
compute end for
rank(Jr (y(+)))
Vector copula density estimation repeat
Vector copula density estimation repeat
set c(ux: uy)= k EK Prex(ux, uy);
sample ux uy D' - D-D'U until |D'|= = n
(1) (i)
d uy);
ē arg maxe uyy [lg (ogx, )y)];
L f- Fax byy [log cl (xx ]y)];
K + 2K;
train to classify samples from and D': set ć(ux, uy) r(ux, uy) c' dux. uy return ((x;Y) = Σ log c(u al))
until improvement va return f(X:V) 1.2
n Ei= log c(u .n
This theorem generalizes the results of [44, 45] from univariate to vector marginals' It establishes that MI depends solely on the vector copula, which itself invariant marginal distributions. Notably, the theorem also reveals that the pointwise mutual information (PMI) i.e. p(x. ((y) can equivalently be viewed as a density c(ux, uy) its own right, in contrast to the vast majority of existing works [46, 47, 48, 49] which continue to treat PMI as density ratio. This shift in perspectives opens us new possibility in the parameterization and learning of the PMI, including directly modeling it as a normalized density learned via MLE, as will be discussed later.
Theorem immediately suggests new divide-and-conquee approach for MI estimation: we can first estimate the vector ranks ux and uy, followed by subsequent learning of the vector copula c::
x 1 n
n Σ
log c(ú u
(5)
where åx. ûy and are the empirical estimates ux, uy and e respectively.
We discuss below several potential advantages of the above divide-and-cooquur estimation strategy:
By disentangling the modeling of marginals distribution and copula, we can use differently-sized models in their parameterization, avoiding using a single overly flexible or overly simplified model for the entire distribution. This leads to a better trade-off between model complexity and capacity;
By disentangling the learning of marginals and copula, we can reuse the pre-trained marginals across multiple copula choices with varying complexities, allowing model selection to be performed solely the copula space in a computational efficient way. also reduces overall learning difficulty.
In the following, we elaborate methods estimate the vector ranks and the vector copula respectively.
Building upon classic copula, the theory in [44, 45] only holds for bivariate cases, and generalizing their results to high-dimensional cases require non-trivial formulation "Alternatiely, one may also learn the marginals p(x), p(y) and the vector copula é jointly. However, joint learning be ill-posed [50]. Our ablation study in Appendix B2 suggests that separate learning more robust.
derivation ppreinsllyyyy our key contribution.

---

3.1 Marginal distribution learning
In this step, we learn the two marginal distributions p(x) and ply) with flexible flow-based models [35, 36, 37,38) 38] and use them to compute the vector ranks ux and uy.
Flow-based modeling of marginals. Let fx R2x flow-based models and let Pfx (x) and pfr (y) be the densities induced by fxx and fy respectively. We respectively learn fx and JY with data p(x) and data y p(y) by flow matching [38]:
Ráx and fr R + R be two
min E[CMM fx)),
min Bli fv
(6)
where LrM denotes the flow-matching loss [38]. Upon convergence, fx and fY respectively transform the two marginals to a standard normal distribution: (),,,) = #x() and I)( ~ #y
Vector ranks computation. With the learned flows fx and fy, we compute the vector ranks as:
rank (fx(x()
rank (fr (y()))
where rank,(e) = n+1 دド=1 1/c 2 (01 is the element-wise ranking function that computes the scalar ranks for each of the dimension in e. Given universal density approximators fx, fr, úx and üy serve consistent estimates of the true vector ranks ux and uy.
While the joint density p(x, y) is often challenging to estimate, the marginal distributions p(x) and p(y) are typically far easier to learn due their lower dimensionality. It is thus reasonable to expect that åx and ûy are close approximations to ux and uy in moderate dimensionality settings.
Remark. The above process of estimating ux and uy can be viewed as a generalization of classic copula transformation in MI estimation, where we compute vector ranks rather than scalar ranks.
3.2 Vector copula estimation
In this step, we learn the vector copula e with the previously estimated vector ranks úx and nt, leveraging model-based parameterization and careful model selection strategy.
Model-based parameterization of copula. As noted earlier, any parametric model can be used to represent the vector copula c, regardless of whether an analytical PMI is available. In this work, we parameterize e as a mixture of existing parametric vector copulas [30] from the copula pool, whose model complexity can be well controlled by tuning the number of mixture components:
c(ux, uy) = Σ PECk uy); uy);
where Σx 1= Pk = 1 and each C E C is selected from the predefined pool C of vector copulas. Any inductive bias about the dependence structure can used to guide copula selection. Here, we simply implement each c vector Gaussian copula and learn c by maximum likelihood estimate (MLE);
max Ellog c(ux, uy]]
In theoretical analysis, we analyze why this copula design is cheap yet reasonable modeling of c.
Efficient model selection. A key design in our method is the explicit exploration of the capac- ity-complexity trade-off in copula modeling, which is governed by the number of mixture compo- nents K. Here, we determine K by cross validation, using negative log-likelihood (NLL) as the criterion. This process is computationally cheap: each copula is already lightweight, involving no neural networks; furthermore, different copulas can be trained in parallel using one single loss.
Algorithm summarizes the main pipeline of the proposed vector copula-based estimator (VCE).
Remark. As an alternative to the above model-based parameterization, one may also adopt a reference- based parameterization for the vector copula, inspired by the design in [51]. Specifically, let c' be reference vector copula that easy to sample (e.g. a vector Gaussian copula). We can learn e by first estimating the density ratio = c/c' using samples from and e' [52, 41, 39], then recover the vector copula e as = e': see Algorithm 2. By parameterizing r as deep neural network, this method allows for more flexible modeling of c, at the cost of a less fine-grained control over its complexity.
5

---

Theoretical analysis
In this section, we analyze several important theoretical properties of the proposed VCE estimator. Proposition (Consistency of VCE), Assuming that (a) fx and fy are universal PDF approximators with continuous support and (b) the number of mixture components K in (8) is sufficiently large.
Define In (X;; Σ log c(ū i For every e O, there exists nlE) N, such that
- 1 <E, Vn 2 n()),
Proof. Please refer to Appendix A
Additionally, have the following result analyzing the estimation learned marginals Pfx (x), (y) and the estimated vector copula density é.
the quality of the
Proposition 2 (Error of vector copula-based MI estimate). Let ûx and ûy be the estimated vector ranks. Les c(ux, ûy) be the true joint distribution of úx and ûy, and o(ūx, ūy) its estimate. Assuming that sufficient Monte Carlo samples are used compute 1 (X; in (5), have
T(X:Y) - f(x;Y) ((x:))-((x;))) & [(xx) + H(ūx)) KT[c(xx, ûy )) ûy)]
where the first term the RHS vanishes as Pfx (x) p(x) and Pry (y) ply). In the limit of perfectly learned marginals, the ervor simpliftes to
I(x;Y)- ((X;))(())) K[Cclee,
with and being the true vector copula density and estimated vector copula density, respectively.
Proof. Please refer to Appendix A.
Proposition 2 decomposes the estimation error of the proposed VCE estimator into two components:
Marginal estimation (ūv)] > 0, which diminishes as both marginals are learned more accurately (recall that ideally, we have ux ~ [0, 1]dx and úy ~ 24[0, _]dr). ). For data with moderate dimensionality, we expect this bias to be small, the two marginals are with low-dimensionality. being casy to estimate.
Imperfect marginal estimations introduce bias given by {(ūx))
Dependence structure modeling error. This arises from the discrepancy between the estimated copula è and the true copula c. It depends on two factors: (a) capacity whether the parameterization of è is sufficiently expressive approximate c; and (b) complexity how easy è can be learned from the limited data. These factors highlight the importance of model selection for the copula c.
Proposition 3 (Vector Gaussian copula as second-order approximation). A vector Gaussian copula e corresponds to the second-order Taylor expansion of the true vector copula c* up to variable transformation.
Proof. Please refer to Appendix A.
This result explains choice of using mixture of Gaussian copulas cheap yet principled approximation to the true vector copula. A single vector Gaussian copula already offers a reason- able approximation of the true copula by capturing dependencies up second order; higher-order interactions, if necessary, can be modeled by adding mixture components in a fully controllable way.
Finally, we have the following result regarding cases with weakly dependent random variables (RVs).
Proposition 4 (Vector copula of independent RVs). The vector copula corresponding to the product of marginals (x. y) = p(x)p(y) is a vector Gaussian copula if p' (x. y)) is absolutely continuous.
Proof. Please refer Appendix A.
Proposition suggests that the two RVs X and Y are nearly independent, our estimator likely to provide accurate estimation of Y(X;Y) as the true vector copula is Gaussian-like, being close to the family of our copula design (8). For weakly dependent RVs, reasonable to expect that p(x, y) resembles vector Gaussian copula, with the difference captured by the additional components in (8).

---

5 Reinterpreting existing MI estimators
In this section, reinterpret existing MI estimators through the lens of vector copula theory, showing that they correspond to different parameterizations and learning strategies of the vector copula.
Reinterpreting discriminative estimators. Existing critic-based approach to MI estimation [20, 53, 39, 41, 32, 23] can be interpreted as parameterizing the vector copula c(ux, uy) using a feedfor- warding neural network f:
c(ux. uy) ) x ef
(10)
which is learned by discerning samples from the joint p(x. y) and the product of marginals p(y)
(via e.g contrastive learning). Specifically, recall that the optimal critic f in these methods corresponds to the log density ratio up to an additive constant C' [54]: f(x, y) = log p(x, y) //(pp/(()))))))))) (y(y)) + C, with the PMI itself equal to the vector copula density, as established by the vector Skalar theorem.
Compared our model-based parameterization of the vector copula density in (8), this neural network parameterization is more flexible and can potentially capture more complex dependence structures. However, discussed earlier, such distribution-fiet free parameterizations lack complexity control, which may lead to a poor varianee trade-off. Furthermore, discriminative methods learns the vector copula by comparing distributions, which can be challenging if they differ significantly (e.g., in high-MI cases, see [40, 41, 29, 32]. In contrast, our main method learns the vector copula by maximum likelihood estimate (MLE), which the most efficient consistent estimator for the copula.
Reinterpreting generative estimators. Many generative estimators for MI [17, 29, 18, 48, 32] require either learning the joint distribution p(x, y) p(x))((y<(uxx uy) or the conditional distri- bution p(y|x) x) = p(y )c(ux. uy) using single model. This process can be interpreted as learning the marginal distribution(s) and the vector copula simultaneously, with the two components parameterized jointly via a single generative model. Our method, on the contrary, explicitly separates the modeling and the learning of the marginal distributions p(x),p(y) from that of the vector copula c(ux, : uy control over model complexity, but also mitigates
This strategy not only enables a more the challenge of jointly learning the marginal distribution and the dependence structure- a strategy aligned with the spirit of classical copula transformations [47, 55, 56, 57] to simplify MI estimation.
We further discuss two recent works [48, 17] closely related to our work. These methods operate by respectively transforming the two RVs and Y by two flow- models, such that the joint distribution of the transformed data can be approximated by distribution with easy-to-compute MI (for instance, Gaussian distribution). Their practical methods, V-MIENF and DINE-Gaussian, can be reinterpreted assuming the dependence structure as a vector Gaussian copula (see Lemma in Appendix A4 for a detailed derivation):
(11) which corresponds to the case K in the VCE estimator and is accurate (only) if the true dependence is Gaussian-like like. The possibility of using non-Gaussian base distribution is also discussed in [48], albeit without practical implementation. Additionally, the marginals and the vector copula in their method are learned jointly rather than separately as in our method, and they continue to treat PMI as density ratio ((x,))/p(x))(()) unlike method which treats it as a density c(ux, uy).
c(ux, uy) ~ (uxx Ey:
6 Experiments
Baselines. We consider five representative neural estimators in the field: MINE [20], InfoNCE [21]. MRE [41], MINDE [19] and [48]. The first three methods are critic-based whereas the latter three are generative madel- MRE chosen as the representative of state-of-the-an discriminative methods, which is specifically designed to address the high nigh-discrepancy issue in these methods. MINDE is chosen to represent the state-of-the-art generative methods, which leverages powerful diffusion model in MI estimation. Further baselines are considered in Appendix B2.
Hyperparams. For the vector copula in VCE, we consider mixtures with 1, 4, 8, 16, components.
Neural architecture, optimizer and training details. Please refer to appendix B1 for more details.
In the following evaluation, we primarily focus on evaluating the VCE estimator (Algorithm 1), and present the results of the alternative VCE' estimator (Algorithm 2) in the appendix. All results are collected through 8 independent runs. Error bars reported are the standard deviations (std) of the runs
7

---

Talagyy
HREE
HEC DE
den
28
24
3335 30
23 oo
(a) X, Y N
(b) AX)
(c) AX By
(d) Student-t
(e) Swiss Roll
Figure 2: Comparing MI estimators under various dependence strengths p. Data in cases (b)(c) generated by first sampling X, Y ~ N as in case (a), then transforming them with the shown transformations. The dimensionalities the data in the five cases are 64, 32, 32, 32, respectively.
PRED
VCT
31
(a) Spiral
(b) A By3
(c) Student-t
(d) MoG 1
(c) MoG
Figure 3: Comparing different MI estimators under various data dimensionality d and fixed depen-
dence levels. MoG corresponds to mixture of Gaussians. Spiral corresponds to spiral transformation.
6.1 Synthetic distributions
Setups. In [58], a diverse set of models with known MI are developed to comprehensively evaluate MI estimators. We consider representative cases from this benchmark, further extending it by (a) considering varying dependence strengths for each chosen case; (b) employing mixing matrices A, B couple the dimensions in X and Y respectively. We also include the mixture models in [49] to enrich our tests. Together, our test cases cover non-Gaussianity, skewness, heterogeneous marginals, long tails, low-dimensional manifold structure, coupling dimensions, high-dimensionallty. varying dependence strengths and non-Gaussian dependence structure. Each test case contains n = 10 data.
Results. Figure 2 and Figure 3 compare the performance of different MI estimators Overall, VCE provides good estimates all scenarios, consistently ranking among the top performers.
Compared to discriminative methods e.g., MINE and InfoNCE, VCE demonstrates significant advantages, particularly high MI settings (e.g. strong dependence level p or high dimensionality d). This advantage may be because our method avoids directly comparing two highly distinet distributions as these methods, which challenging. The advantage may also attribute the better complexity-capan trade-off in our method, which avoids an overly powerful model for the copula.
Compared the generative method --MIENF, VCE demonstrates advantages in scenarios involving non-Gaussian dependence structures (see e.g. the MoG cases and 64D t-distribution). In such cases, A-MIENF's assumption of a Gaussian dependence structure falls short in capturing the true dependence structure. This underscores the pitfalls of using overly simplified model for the copula.
We specifically discuss two challenging cases highlighted in prior works [58, 19]: (a) Spiral transfor- mation, which highly transforms the original data; and (b) multivariate (-distribution with degree of freedom 1, which exhibit heavy-tailed dependence. For these two highly challenging scenarios, VCE and MINDE are the only two methods that can simultaneously provide reasonable estimates in both cases, with VCE outperforming MINDE in other settings (see e.g., Figure 2.c and Figure 3.b).
Comparison classic copula-base based MI estimators and further discriminative estimators is in Appendix B2.

---

(a) Gaussian plates
(b) Rectangles
Figure 4: The image dataset [59], which contains images of rectangles and Gaussian plates.
Method
Gaussian Plates
Rectangles
((X;Y) =1
((X;Y) 3
((x;)) =7
T(X;Y) =1
((xY) =
((X:Y) =
MINE
0.89 +: 0.07
2.86 0.24
0.227
0.81 0.13
2.57 + 0.26
5.39 + 0.23
InfoNCE
0.14
2.63
3.83
0.18
2.49
3.86 3.86
MRE
0.16
2.85
0.228
0.82
2.56 0.48
5.45 + 0.31
--MIENF
0.14
2.42
3.85 1:5:100 0.22
0.54
0.76 2.14
1.54 0.11
VCE
0.92 0.04
2.93 0.12
6.53 0.36
0.12
2.27 + 0.23
5.02 + 0.14
Table Comparing different MI estimators on the image benchmark proposed in [59].
6.2 Image dataset with known MI
Setups. We next consider the benchmark [59]. which contains correlated images X and Y; see Figure Here X E R 16x11 and Y E IR 16x and the ground truth I(x; Y) is known for this dataset. Following recent works [59, 3], we preprocess these high-dimensional image data by an autoencoder e: R6616 - Rd which proves effective in reducing data dimensionality while preserving key information. The quality of such compression w.r.t d' is analyzed theoretically and empirically in Appendix A5 and B2, based on which we set d' 16. A total number of 10, 000 data used. Note that while the dependence between X and Y are Gaussian for this dataset [59], the dependence structure for the compressed data can be non- Can-Gaus even if the compression is near-lossless.
Results. Table 1 compares the performance different MI estimators on this task. Our estima- tor consistently outperforms the recent .-MIENF estimator on this dataset, and it shows highly competitive performance against discriminative methods. However, our method performs slightly worse than discriminative methods in the Rectangles case. One reason why approach loses to discriminative approaches in the Rectangles case may be that the underlying dependence structure of the preprocessed data highly complex in this case, which is difficult model effectively with single vector Gaussian copula or even a reasonable mixture of such copulas. Discriminative methods, on the contrary, adopt neural network-based parameterization of the vector copula, being inherently more flexible. These results highlight the limitation of model-base parameterization of the vector copula density in certain cases. Nonetheless, our estimator still provides a highly reliable estimate.
6.3 Embeddings of language models
Setups. We further consider a real-world dataset in natural language processing. It consists of pairs of embeddings from language model (LM) [60, 61] computed the IMDB dataset [62], which contains negative or positive movie comments; see Table 2. The ground truth MI of this dataset unknown, but can be computed numerically accurately; see Appendix BI. A total number of n x 10s data are used. Similar to the previous task, we preprocess data by an autoencoder IR dim - 16 with diM being the dimensionality of the LM's embeddings. The quality of such compression is empirically studied in Appendix B2, which is near-lossless.
Results. Table 3 summarizes the results for this dataset. In this scenario, where the underlying mutual information (MI) is relatively low, our method does not show a significant advantage over discriminative methods. This is likely because for this dataset, the high-discrepancy issue [40, 41, 29, 32] is not significant, and discriminative methods offer a more flexible parameterization of the vector copula density than method (see Section 5). Nonetheless, close to discriminative methods, and it significantly outperforms the generative method --MIENE.
method still provides estimate
6.4 Further analysis and ablation studies
We conduct further analysis the effect of model selection and separate learning in Appendix B2.

---

1 (positive) I thought this was wonderful way to apend time on
(positive) If you like original gut wrenching laughter you will like
2 (negative) So im not big fan of Boll's work but then
(positive) This fantastic movie of three prisoners who become famous...
Table 2: The text benchmark, which contains reviews of positive or negative movie comments.
Method
T(X;Y) 2.1
I(X:Y) 0.9
Method
T(X;Y) 1.5
T(X:Y) 0.2
MINE
0.04
0.71 +: 0.05
MINE
1.42 :
0.18
InfoNCE
1.64 0.09
0.70 1: 0.06
InfoNCE
1.41 1: 0.03
0.19 + 0.04
MRE
1.72 + 0.07
1,23 + 0.02
MRE
1.23 + 0.09
0.31 + 0.09
M-MIENE
0.91 + 0.05
0.43 + 0.03
M-MMEN
0.73 1: 0.03
0.11 + 0.02
VCE
2.01 + 0.04
0.83 + 0.01
VCE
1.22 1 0.02
0.19 = 0.02
(a) Llama-3 13B
(b) BERT
Table 3: Comparing different MI estimators on the text dataset. Left: evaluation on the embeddings of Llama-3 13B model [61]. Right: evaluation the embeddings of BERT model [60].
7 Conclusion
In this work, we introduced a new mutual information (MI) estimator grounded in recent vector copula theory. A fundamental difference to existing approaches is the explicit disentanglement of marginal distributions and dependence structure in our method. This separation enables more flexible and fine-grained modeling, avoiding the pitfalls of both overly simplistic or excessively complex approaches, and reducing overall learning difficulty via strategic factorization of the original estimation problem. Extensive experiments demonstrate our method's effectiveness and robustness.
Beyond the development of practical estimator, our research also offers fresh perspectives on MI estimation. By viewing PMI as a density rather than density ratio, we open new avenues for modeling. Additionally, our approach to vector rank computation generalizes the classical copula transformation and holds promise as a versatile preprocessing step for a broad range of MI estimators. Finally, by reinterpreting existing estimators through the lens of vector copula theory, we obtain new insights into the parameterization and learning of different estimators and the underlying trade-off's
Copulas have been widely used for MI estimate [63, 55, 64, 56, 65, 33, 51, 45, 66). Existing methods primarily focus on classic copulas, where the copula transformation is applied independently to each univariate marginal to better account for the marginal-invariann invariant property of MI. This strategy has been shown to improve accuracy and reduce variance [56. 65]. We go one step further by using vector copulas, where the transformation jointly considers all dimensions of the multivariate marginals. This can be seen as generalization of classic copula transformation, where we not only consider MI's invariance element-wise bijections but also to any diffeomorphisms. Another key difference lies in that these works still treat PMI as a density ratio, whereas our work treats PMI as density.
We note that, while powerful, our estimator is not panacea. One limitation of our method is that relies on the two marginal distributions to be reasonably modeled. While marginal distributions are far easier to learn than the joint distribution, they can still be challenging to learn for high-dimensional data e.g., images. Fortunately, dimensionality reduction techniques [3. 13] help to mitigate this issue. Another limitation lies in the flexibility of our model-based parameterization of vector copula, which can be less flexible than neural network methods. However, as our method strikes a good trade-off between complexity and capacity across diverse cases, we consider it as a highly competitive method.
Acknowledgments
AW acknowledges supports from Turing AI Fellowship under grant EP/V025279/1, the Alan Turing Institute, and the Leverhulme Trust via the Leverhulme Centre for the Future of Intelligence. MUG supported in part by Gen AI - the AI Hub for Generative Models, funded by EPSRC (EP/Y028805/1). YC acknowledge supports from the Cambridge Trust and the Qualcomm Innovation Fellowship. ZO acknowledges supports from Lee Family Scholarship via Imperial College London.
10

---

References
[1] Justin B Kinney and Gurinder S Atwal. Equitability, mutual information, and the maximal information coefficient. Proceedings of the National Academy Sciences, (1)):3354-3359) 2014.
[2] Tm Cover, Ja Thomas, and J Wiley. Elements of information theory. Tsinghua University Pres, 2003.
[3] Gokul Gowri, Xiao-Kang Lun, Allon M Klein, and Peng Yin. Approximating mutual information of high-dimensional variables using learned representations. arXiv preprint arXiv:2409.02732 2024.
[4] Antonio Reverter and Eva KF Chan. Combining partial correlation and an information the- ory approach to the reversed engineering of gene co-expression networks. Bioinformaties, 24(21):2491-244 -497, 2008.
[5] Lin Song. Peter Langfelder, and Steve Horvath. Comparison of co-expression measures: mutual information, correlation, and model based indices. BMC bioinformatics, 13:1-21, 2012.
[6] Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on pattern analysis and machine intelligence, (((()))1226-2366 2005.
[7] R Devon Hjelm, Alex Fedorov, Samuel Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 86X7:71808.06670, 2018.
Karan Grewal, Phil Bachman,
[8] Michael Tschannen, Josip Djolonga, Paul Rubenstein, Sylvain Gelly, and Mario Lucie. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2019.
[9] Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual information contrastive learning for visual representations. arXiv preprint 1xXv:20055131499 2020.
[10] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ily Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016
[11] Yanzhi Chen, Michael U Gutmann, and Adrian Weller. Is learning summary statistics necessary for likelihood-free inference? In International Conference on Machine Learning, pages 4529- 4544, PMLR, 2023.
[12] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint 1rXiv:1612.00411 2016.
[13] Yanzhi Chen, Dinghuai Zhang, Michael Gutmann, Aaron Courville, and Zhanxing Zhu. Neural approximate sufficient statistics for implicit models. arXiv preprint arXiv:2010. /0079, 2020.
[14] Zexuan Qiu, Qinliang Su, Zijing Ou, Jianxing Yu, and Changyou Chen. Unsupervised hashing with contrastive information bottleneck. arXiv preprint Stl::2055.0638, 2021.
[15] Yanzhi Chen, Yingzhen Li, Adrian Weller, et al. Scalable infomin learning. Advances in Neural Information Processing Systems, 35:2226-2239, 2022.
[16] Sayedeh Leila Noorbakhsh, Binghui Zhang, Yuan Hong. and Binghui Wang. (Inf2Guard]: An [Information- Theooreticee framework for learning (Privacy-Presse Preserving) representations against inference attacks. In 33rd USENIX Security Symposium (USENIX Security 24), pages 2405- 2422, 2024.
[17] Bao Duong and Thin Nguyen. Diffeomorphic information neural estimation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7468-7475, 2023.

---

[18] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pages 1779-1788. PMLR, 2020.
[19] Giulio Franzese, Mustapha BOUNOUA, and Pietro Michiardi. Minde: Mutual information neu- ral diffusion estimation. In The Twelfth International Conference on Learning Representations, 2023.
[20] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference machine learning, pages 531-540. PMLR, 2018.
[21] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv-1807, 2018.
[22] Qing Guo, Junya Chen, Dong Wang, Yuewei Yang, Xinwei Deng, Jing Huang, Larry Carin, Fan Li, and Chenyang Tao. Tight mutual information estimation with contrastive fenchel-legendre optimization. Advances in Neural Information Processing Systems, 35:28319
2022.
[23] Nunzio Alexandro Letizia, Nicola Novello, and Andrea M Tonello. Mutual information estimation via f-divergence and data derangements. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[24] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359-366) 366, 1989.
[25] Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103-114, 2017.
[26] Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6): 1253, 2003.
[27] Yihong Wu and Pengkun Yang. Minimax rates of entropy estimation on large alphabets via best polynomial approximation. IEEE Transactions Information Theory, 6266):370--3772, 2016.
[28] Andrew R. Barron and Kai Guo. Convergence of smoothed empirical measures with applications to entropy estimation. IEEE Transactions on Information Theory, 686)):3366-3533,, 2022.
[29] David McAllester and Karl Stratos. Formal limitations on the measurement of mutual infor- mation. In International Conference on Artificial Intelligence and Statistics, pages 875-884. PMLR, 2020.
[30] Yanqin Fan and Mare Henry. Vector copulas. Journal of Econometrics, 234(():1228 150, 2023.
[31] M Sklar. Fonetions de répartition à n dimensions et leurs marges. In Annales de I'ISUP, volume 8, pages 229-231, 1959.
[32] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual informa- tion estimators. In International Conference on Learning Representations, 2019.
[33] Shashank Singh and Barnabás Póczos. Nomparanormal information estimation. In International Conference on Machine Learning, pages 3210-3219. PMLR, 2017.
[34] Yanzhi Chen and Michael U Gutmann. Adaptive gaussian copula abc. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1584-1592. PMLR, 2019.
[35] George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pages 2338-2347, 2017.
[36] Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. arXiv preprint 1906.v/906.044000 0006004032, 2019.
[37] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1-64, 2021.
12

---

[38] Yaron Lipman, Ricky TQ Chen, Heli Ben- Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2022.
[39] Ruizhi Liao, Daniel Moyer, Polina Golland, and William M Wells. Demi: Discriminative estimator of mutual information. arXiv preprint arXiv:2010.01766, 2020.
[40] Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratin estimation. Advances in neural information processing systems, 33:4905-4916, 2020.
[41] Akash Srivastava, Seungwook Han, Kai Xu, Benjamin Rhodes, and Michael U Gutmann. Estimating the density ratio between distributions with high discrepancy using multinomial logistic regression. Transactions on Machine Learning Research, 2023(3): 2023.
[42] Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, iv. Communications on pure and applied mathematics, 36(2):183-212 1983.
[43] Victor Chemozzukov, Alfred Galichon, Marc Hallin, and Mare Henry. Monge-kantorovich depth, quantiles, ranks and signs. Annals of Statisties, 45(1):223-2566 2017.
[44] Manuel Davy and Arnaud Doucet. Copulas:
new insight into positive time-frequency distributions. IEEE signal processing letters, 10(7):215-118, 2003.
[45] Jian Ma and Zengqi Sun. Mutual information is copula entropy. Tsinghua Science & Technology, 16(1):51-54, 2011.
[46] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, pages 531-540. PMLR, 2018
[47] Zhengyang Hu, Song Kang, Qunsong Zeng, Kaibin Huang, and Yanchao Yang. Infonet: Neural estimation of mutual information without test-time optimization. In Forty-first International Conference Machine Learning.
[48] Ivan Butakov, Aleksandr Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, and Alexey Frolov. Mutual information estimation via normalizing flows. Advances in Neural Information Process- ing Systems, 37:3027-3057, 2024.
[49] Pawel Czyž, Frederic Grabowski, Julia E Vogt, Niko Beerenwinkel, and Alexander Marx. The mixtures and the neural critics: On the pointwise mutual information profiles of fine distributions. arXiv preprint arXiv:2310.102400 2023.
[50] Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Nicolas Chapados, and Alexandre Drouin. Tactis-2: Better, faster, simpler attentional copulas for multivariate time series. arXiv preprint arXiv:2310.01327, 2023.
[51] David Huk, Mark Steel, and Ritabrata Dutta. Your copula a classifier disguise: classification- based copula density estimation. In International Conference on Artificial Intelligence and Statistics, pages 3790-3798. PMLR, 2025.
[52] Michael Gutmann and Aapo Hyvärinen.
estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international con- ference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[53] Sudipto Mukherjee, Himanshu Asnani, and Sreeram Kannan. Cemi: Classifier based conditional mutual information estimation. In Uncertainty in artificial intelligence, pages 1083-1093. PMLR, 2020.
[54] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
13

---

[55] Houman Safaai, Amn Onken, Christopher D Harvey, and Stefano Panzeri. Information estima- tion using nonparametric copulas. Physical Review 98(5):053302, 2018.
[56] Xianli Zeng, Yingcun Xia, and Howell Tong. Jackknife approach to the estimation of mutual information. Proceedings of the National Academy of Sciences, 115(40)9956-9996) 2018.
[57] Soumik Purkayastha and Peter X-K Song. fastmi: A fast and consistent copula- based non- parametric estimator of mutual information. Journal of Multivariate Analysis, 201:105270, 2024.
[58] Pawel Czyž, Frederic Grabowski, Julia Vogt, Niko Beerenwinkel, and Alexander Marx. Be- yond normal: On the evaluation of mutual information estimators. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[59] Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, Alexey Frolov, and Kirill Andreev. Information bottleneck analysis of deep neural networks via lossy compression. In The Twelfth International Conference on Learning Representations, 2024.
[60] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018
[61] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint urXiv:2407.21788, 2024.
[62] Andrew Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011 Association for Computational Linguistics.
[63] Yves-Laurent Kom Samo. Inductive mutual information estimation: A convex maximum- entropy copula approach. International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2021.
[64] Amor Keziou and Philippe Regnault. Semiparametric estimation of mutual information and related criteria: Optimal test of independence. IEEE Transactions on Information Theory, 63(1):57-71, 2016.
[65] Nunzio A Letizia, Nicola Novello, and Andrea M Tonello. Copula density neural estimation. IEEE Transactions on Neural Networks and Learning Systems, 2025.
[66] Gery Geenens. Towards a universal representation of statistical dependence. arXiv preprint urXiv:2302.0815// 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes],
The abstract and introduction directly reflects the exact contribution of the work.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes],
Justification:
Limitation is explicitly discussed in conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
15

---

Justification:
All proofs have a clear declaration of assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes],
Justification: Code is available in the provided repo, and the details of experiments and implementations are also documented in the appendix.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes],
Justification: source code is provided through the github link provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /niiss wwtppcccpppppeeeecccceeccccceeee /
seiisssssiiiyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes],
Justification: all experimental details are provided in Appendix B. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars (which correspond to the standard errors) reported, which collected from 8 independent runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes],
Justification: please see the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: https://neur:/i /ttpar///////aa/eeeeeeee ct. httprr/p....pppiiiehhhccccooedda
Answer: [Yes].
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: the work is general method and its societal impacts have been discussed in introduction and conclusion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: all code and data are taken from public available resource without violating any license and terms.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification: no new assets introduced this stage.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is only used for proofreading.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: htopp::/neeripss cc/Confferenccee/225/// for what should or should not be described.
21


Reviews:
[{"review_id": "rxKuti80yU", "reviewer": "Reviewer_DGA1", "summary": "The paper describes an approach to estimating mutual information that\nbridges the gap between neural models and simpler models, e.g.,\nGaussian copula.  For this, they propose to use Vector Copulas.\n\nExperiments show some advantages of the method.", "questions": "I would be helpful for the reader if the main text of the paper pointed to the appendix\nfor the proof of Thm 2  :)\n\nStronger experimental results would strengthen the work.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "MI estimation is an important area, e.g, for optimizing encoders\nin representation learning.\n\nThe theoretical development is mostly clear.\n\nIt would help the reader to define the push-forward notation (the symbol #)\n\nThe equivalence of PMI, usually stated as a density ratio,\nand the vector copula density seems very interesting, and as the authors show,\nit can be used to re-interpret important prior works.\n\nThe remainder of the paper demonstrates ways of applying the\nresulting ideas to some MI estimation problems, including\nmodeling the copula as a mixture of copula.\n\nThe results in table 1 show some mild improvements in MI estimation\nin comparison to other methods, though MRE is competitive.\n\nThe results in Figs 1 and 2 are very hard to read, so performance is\nhard to evaluate.\n\nI feel that the theoretical contributions outweigh the somewhat mild\nexperimental result improvements.", "was_revised": true, "final_justification": "I remain positive on this paper.", "rebuttal": "Thank you for your constructive feedback and the positive assessment! Your recognition that our work provides a novel perspective of PMI, enables important re-interpretion of prior works, and have good theoretical contributions are very encouraging. Below, we address your constructive criticisms regarding **empirical performance** and **presentation**.\n\n\n&nbsp;\n\n### **Stronger Empirical Performance**\n\nWe fully agree on the importance of strong empirical performance. In fact, **our estimator does demonstrate excellent performance across a wide range of settings**, though this is not easy to notice due to readability issues in Figures 1 and 2 (which we have now fixed). Below, we highlight scenarios where our method shines:\n\nTable 1. Scenarios where our method is a clear winner. Cases like A, C and D are particularly challenging and are pain points in existing estimators.\n\n|  | Challenges | Evidence |\n| --- | --- | --- |\n| **A** &nbsp; | High MI | See cases $\\rho$=0.8 in Figure 1 and cases $d$=64 in Figure 2 |\n| **B**  &nbsp;| High dimensionality | Figure 2, see cases $d$=64 |\n| **C** &nbsp; | Long-tailed distributions | Figure 1d & 2c (student-t), Figure 1b (exponential) |\n| **D** &nbsp; | Highly-transformed distributions | Figure 2a (Spiral) |\n| **E** &nbsp; | Hetereogenous marginals | Figure 1b |\n| **F** &nbsp; | Non-Gaussian dependencies | Figure 2c (student-t), Figure 2d & 2e (mixtures) |\n|||\n\n(Additionally, our method also gracefully scales to **image** and **text** data, either achieving the best performance or coming very close to the top performer in these cases.)\n\n\n\n\n&nbsp;\n\n### **Notations & Presentation**\n\nWe sincerely thank the reviewer for pointing out the issues with notation and presentation. We have made substantial improvements in the revised version, including:\n\n- (**Figures 1 and 2**) We have make them clean and more readible by (a) increasing the font and figure sizes; (b) avoiding clutter by moving some baseline to the appendix.\n- (**Theoretical proof pointer**) Following your suggestion, we have now pointed the reader to the relevant proof in the appendix for each theoretical result.\n- (**Notations clarity**) We have formally defined the push-forward operation (the symbol “#”) in Section 2 to improve clarity."}, {"review_id": "00hBWCcV1H", "reviewer": "Reviewer_A2bS", "summary": "The paper shows that MI can be recast as a vector copula estimation task, thereby generalising the previous results where the link between copulas and MI was established for univariate variables. A vector copula method is proposed to estimate MI in two steps, first by fitting separate flow models on each set of variables X and Y, and secondly, learning the dependence between X and Y with a mixture of Gaussian vector copulas. Theoretical arguments motivate the approach, and experiments show its competitiveness compared to existing methods for MI estimation.", "questions": "Overall, the paper is persuasive about the usefulness of vector copulas for MI. However, I am unsure about the practical implementation proposed in this paper. Before I can confidently advocate the acceptance of this paper, **I would like to see the first four weaknesses addressed**. Furthermore, critical questions I would kindly ask to be answered include:\n\n- Could you provide, at least in the appendix, some evidence of the obtained vector ranks being $\\mathcal{N}(\\mathbf{0},\\mathbf{I})$ for your experiments? Maybe a test for Gaussianity/independence, or showing the univariate ranks of the representations, or the covariance matrices, or metrics derived from these?\n\n- Are there other ways for obtaining vector ranks that you have considered? For instance, in the 2023 Vector Copulas paper, the authors consider an optimal transport problem solved through linear programming. How would these compare, computationally and results-wise or are they not appropriate to be used here?\n\n---\n\nBelow are minor points and typos that I believe should be corrected before this paper can be ready for acceptance, as well as a few questions I am curious about but that I do not need to be answered if time is too short:\n\n**Typos and notation**\n\n- L24 Do you mean 'suffer from requiring an exponentially large...'?\n- Definition 1, there is a missing dot to the statement?\n- Notation in L98-99, I find it more common to use $\\Phi$ for the CDF and reserve $\\phi$ for the PDF. A possible alternative to notation could be to use $\\Phi_d$ to distinguish the element-wise CDF from the joint?\n- L101 $\\epsilon_{d_X} $ should be $\\epsilon_{\\leq d_X} $ ?\n- L149 $c$ *as* a mixture?\n- L152 stricture -> structure\n- L158, of *a* copula?\n- Appendix L55, should say by Lemma 2 instead?\n- Appendix L89, these combined *results* lead to?\n- Appendix, define $\\Sigma$ in Lemma 3.\n- Throughout your proofs in the Appendix, at APDX L133-134, and more generally, to be more respectful to readers, I would replace 'it is clear' by the appropriate reason. Eg ' we identify this as a Gaussian density', 'By definition, it follows that...'.\n- Figure 1 and 2 could use bigger fonts for the axis and titles, same with the appendix.\n- L266 known *to be* challenging?\n- L297, please write what $d_L$ is equal to in your experiment.\n\n**Curiosity**\n\n- L194, Does a second-order Taylor expansion imply second-order dependencies? For instance, in classical copulas, Gaussians satisfy the simplifying assumption of vines - would this be the same for the vector Gaussian copula? Then, does this mean your V-Gaussian mixture is simplified?\n- L235, Do you test for or show in one of your examples? Time permitting, it would perhaps be nice to see.\n- L286-287, Could this also be because the neural estimators do not need to be normalised and might overestimate the true MI?", "limitations": "Limitations are generally addressed. However, I would appreciate more emphasis on the fact that vector ranks need to be standard Gaussians in the proposed approach, as it is critical for the theory and method to work. I believe things that would ameliorate this include putting more emphasis on this condition being met when relevant, and providing or suggesting some diagnostics or tests to check for this assumption being respected.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "I believe the paper offers a promising way forward in MI estimation, differentiating itself from the more common density ratio approaches that could lead to unnormalized estimates. The separation of learning vector marginals first and the vector copula second allows great control over the method’s complexity. Theoretical results are generally good (except Prop 4 which I have doubts about) and provide good arguments for the adoption of this method. Experiments are also convincing. \n\n---\n\n**Strengths**\n- The paper’s main novelty in Theorem 2 offers a theoretically grounded way of estimating MI, which possibly provides a sound direction for future progress in this field. \n- Their proposed estimator offers some good flexibility in terms of selecting an appropriate model capacity for the perceived complexity of a task, as well as targeting whether to add complexity to the vector marginals and/or the vector copula.\n- I found Proposition 3 to be very good motivation for the proposed modelling approach, exemplifying the capacity-complexity trade-off.\n\n---\n\nWhile the first part of the paper, up to L126 had me convinced by providing a sound motivation, I am very uncertain about Section 3.1, which holds me back from fully believing in the proposed approach. I also have some comments on some aspects of training feasibility, relations to previous work, the vector copula mixture, and the meaning of Proposition 4.\n\n**Weaknesses**\n\n- I am particularly sceptical about the obtained vector ranks as this is crucial for your theory to hold, and for your estimates to be meaningful:\n\nTo use a vector copula, you need vector ranks. To obtain those, you choose flow models. While it certainly is easier to estimate two separate flows in $d_X$ and $d_Y$ dimensions compared to one big flow in $d_X+D_Y$, this is still not at all obvious. L133-134 is crucial to your method, yet it is not emphasised enough and checked for in my opinion. \n- I am not sure joint learning of vector copulas is a well-posed problem:\n\nAt the bottom of page 3, and in the Appendix L251, you suggest learning vector ranks with the vector copula jointly. But is this well defined? I believe [TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series, ICLR 2024] showed that joint optimisation is ill-defined for regular copulas – would the same not be true here? Your experiments in the Appendix seem to suggest so…\n\n- It is not clear to me how the vector copula mixture obtains meaningfully different mixture components $c_k$:\n\nAs you optimise the MLE on observed vector ranks, how do you ensure that each mixture component will not obtain the same representation? There are no real details about the implementation you use that I could spot. \n\n- Two relevant previous works should be discussed in relevant sections of the text:\n\nIn Algorithm 2, your approach of using a DRE to estimate the ratio between a simple vector copula and the true vector copula has previously been used in the context of copulas in [Your copula is a classifier in disguise: classification-based copula density estimation, AISTATS 2025], in their Section 3, and I believe this should be acknowledged. Furthermore, in your Section 5, this same paper discusses many similar points to L207-220 in their Section 6. A second reference that I would like to see discussed is [Towards a universal representation of statistical dependence, by Gery  Geenens 2023], where the relationship between copulas and MI is discussed, notably in their Section 6 showing MI is a measure on non-independence while copulas fully capture dependence.\n\n- Proposition 4 I believe is a little misleading:\n\nIn your proof in the Appendix, before concluding in L150, you end up with a representation of an independent Gaussian in $d_Y+d_X$ dimensions. While this is a Gaussian, it is fully independent. How is your statement different from saying that an independent copula is a Gaussian copula because you can set the correlation matrix $\\Sigma$ to be diagonal while interpreting your data generating process to be with Gaussian variables? All this is saying is that a Gaussian vector copula can also capture the independence vector copula, at which point I fail to see the relevance of this statement, or am I missing something? \n\n---\n\n\n**Quality**: The motivation for the method is well articulated and supported with appropriate theoretical results, and experiments are rather exhausting covering multiple setups. However I am not sure about practical details of the proposed method, which I believe have to be addressed more.\n\n**Clarity**: The submission is overall clearly written and well organised with the only details missing being about the copula mixture and missing discussions of related work.\n\n**Significance**: I believe this paper presents good arguments for a possible shift in focus within the MI community, offering a new objective, providing links within the literature, and reinterpreting existing models for new insights. It also provides motivation to develop new vector copula models with MI constituting a useful application for them. \n\n**Originality**: New insights and properties of existing work are discussed. But some related work that I believe should be discussed is omitted. The introduction of vector copulas for MI, while not fully novel from the copula perspective, is motivated well and backed up by theoretical guarantees.", "was_revised": true, "final_justification": "The Authors clearly addressed all my concerns. They proposed additional diagnostics and a discussion of some related work, and promised to include those in the final version of the paper. I was already positive about this paper with a rating of 4, and I am increasing my rating to 5 after the discussion, as I confidently believe this paper is of sufficient quality and merit to be published at NeurIPS.", "rebuttal": "Thank you for your high-quality review and the detailed feedbacks! We are deeply grateful for your recognition of our work’s theoretical maturity, potential impact in creating a focus shift in MI community and our comprehensive evaluation. Below, we address your concerns, particular those regarding **the quality of the obtained vector ranks** and **related works**. \n\n&nbsp;\n\n&nbsp;\n\n### **W1. Sceptical about the quality of the obtained vector ranks** ###\n\nThis is a valid concern. We address it from two aspects: \n\n- (**Guaranteeing uniformity**) Each univariate rank  $\\mathbf{\\hat{u}}_d$ is guaranteed to be *perfectly* uniform in our method. Specifically, when computing the vector ranks, we applied element-wised empirical rank rather than Gaussian CDF; see the provided code. The description in the manuscript was indeed inaccurate, and we sincerely apologise for this inaccuracy. \n- (**Testing independence**) Following the reviewer’s insightful suggestion, we have performed a diagnostic to validate whether dimensions  $\\mathbf{\\hat{u}}_i$,  $\\mathbf{\\hat{u}}_j$ are independent. Tables 1 and 2 summarize the results. It is clear that $\\mathbf{\\hat{u}}_i$,  $\\mathbf{\\hat{u}}_j$ are indeed near-independent in moderate-to-high dimensional settings. We will include this diagnostic in the revised manuscript.\n\nTable 1. (Student-d distribution case). Inspecting the non-diagonal elements $\\Sigma_{ij}$ in the covariance matrix $\\Sigma$ of the estimated vector rank $ \\mathbf{\\hat{u}}$.\n\n|  | &nbsp; mean of $\\Sigma_{ij}$ &nbsp; | &nbsp; 5% quantile of $\\Sigma_{ij}$ &nbsp; | &nbsp; 95% quantile of $\\Sigma_{ij}$ &nbsp; |\n| --- | --- | --- | --- |\n| d=32 | &nbsp; 0.007 | &nbsp; -0.042 | &nbsp; 0.031 |\n| d=64 | &nbsp; 0.012 | &nbsp; -0.048 | &nbsp; 0.042 |\n|  |  |  |  |\n\nTable 2. (Spiral transformation case). Inspecting the non-diagonal elements $\\Sigma_{ij}$ in the covariance matrix $\\Sigma$ of the estimated vector rank $ \\mathbf{\\hat{u}}$.\n\n|  | &nbsp; mean of $\\Sigma_{ij}$ &nbsp; | &nbsp; 5% quantile of $\\Sigma_{ij}$ &nbsp; | &nbsp; 95% quantile of $\\Sigma_{ij}$ &nbsp;|\n| --- | --- | --- | --- |\n| d=32 | &nbsp; 0.007 | &nbsp; -0.053 | &nbsp; 0.055 |\n| d=64 | &nbsp; 0.009 | &nbsp; -0.079 | &nbsp; 0.066 |\n|  |  |  |  |\n\nWhile the above results indicate **good vector rank quality (perfect element-wise uniformity + low correlation)**, quality degration may still occur. The above test serves as a highly useful diagnostic for quality assessment. We sincerely thank the reviewer for prompting this checking, which has been instrumental in improving the rigor and practicality of our work.\n\n&nbsp;\n\n&nbsp;\n\n### **W2. Joint learning of vector copulas is not a well-posed problem** ###\n\nWe are on the same page! **Our prior mention of joint learning was merely for completeness, not to recommend it**. In fact, separate learning is a key design in our method, as motivated by l120–l126 and justified by the ablation study “Joint Learning vs. Separate Learning” in Appendix B. The reviewer's reference further supports this design. \n\n\n\n&nbsp;\n\n&nbsp;\n\n### **W3. How to obtain meaningfully different mixture components $c_k$ in the vector copula** ###\n\nThe answer is simple: **just randomly initialize the parameters of each mixture component**, following standard practice in Gaussian mixture modeling. This is analogous to how neural networks are initialized to avoid homogeneous weight configurations. If the true dependence is non-Gaussian, MLE should learn diverse components to attain a high likelihood.\n\n&nbsp;\n\n&nbsp;\n\n### **W4. Relevant previous works missing** ###\n\nThank you very much for pointing us to these highly relevant works, which we have now discussed properly in the paper:\n\n- (**Acknowledging [A]**). When introducing VCE’ (reference-based parameterization of vector copula), we cite [A] to explicitly acknowledge that similar method has been explored for classic copula. We have also related our Section 5 to the analysis in [B], thereby drawing a connection between the two works. \n- (**Discussing [B]**). We have now included a new paragraph discussing existing copula-based MI estimators, where both [A][B] and other relevant works are included.\n\n[A] Your copula is a classifier in disguise: classification-based copula density estimation, AISTATS 2025\n\n[B] Towards a universal representation of statistical dependence, 2302.08151.\n\n&nbsp;\n\n&nbsp;\n\n\n### **W5. Confusion on Proposition 4** ###\nWe feel that we are on the same page --- Proposition 4 indeed suggests that *a vector Gaussian copula model can perfectly represent any independent distribution* $p'(\\mathbf{x}, \\mathbf{y}) = p(\\mathbf{x})p(\\mathbf{y})$.\n\n\n&nbsp;\n\n&nbsp;\n \n\n### **Questions** ###\n- (**The quality of the obtained vector ranks?**) Thank you for this great suggestion. We have now included a diagnostic for assessing the quality of the learned vector rank $\\hat{\\mathbf{u}}$; please see our response to W1 above. More advanced tests can also be used, but we consider it as future works.\n\n- (**Other ways for obtaining vector ranks?**) The linear programming approach from the original vector copula paper can also be used, but we found their method computationally expensive for even moderate dimensional variables (e.g. 30D), so we use the flow-based approach instead.\n\n \n&nbsp;\n\n&nbsp;\n \n\n### **Curiosity**\n> **Is vector Gaussian copula mixture a simplification of vines?**\n\nThis is an inspiring ask! It could be, but a careful check is needed, as we work with high-dimensional vector Gaussian copula rather than the commonly used bivariate copula in vines.\n\n> **Do you test for or show in one of your examples that $\\mathcal{N}$-MIENF / DINE-Gaussian indeed learns a vector Gaussian copula?**\n\nWe have not shown this directly; however, the experiments in 6.1 may have implicitly verified this, as $\\mathcal{N}$-MIENF only works well in cases with Gaussian dependence (see Figs 2c, 2d, 2e).\n\n> **Could the better performance of neural estimators in Sec 6.2 also due to that they do not need to be normalised?**\n\nYes, particularly considering its deep connection to model flexibility and capacity. However, please don't forget that this flexibility comes at a price (no control over complexity).\n\n\n\n&nbsp;\n\n\n**Other issues**. Issues such as typos and notation issues have been fixed in the revised manuscript. We sincerely thank the reviewer for the very detailed checking and correction."}, {"review_id": "MXPNTo8Fyl", "reviewer": "Reviewer_Gion", "summary": "The paper  proposes an copula based approach (and algorithm) to compute  mutual information.\nExperimental results show that the proposed approach can outperform alternative algorithms.", "questions": "none", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**strengths:**\n\n- Extensive comparison with existing algorithms.\n- demonstrate good accuracy \n- code provided\n\n\n**weaknesses:**\n- Figures too small (Fig.1/2) \n- Appendices missing (?)\n- Table 2 not needed (?)", "was_revised": true, "final_justification": "NA", "rebuttal": "Thank you for your time and your work in reviewing our submission! Although being concise, your feedback is still valuable in improving our work. We truly appreciate your recognition of our work's merits, including our extensive evaluation, the reproducibility, and the overall strong performance. Below, we address your concerns.\n\n- **Figures too small**: We sincerely apologize for this. In the revised version, we have (a) resized all figures and (b) increased the font size to improve clarity.\n- **Appendix missing**: The appendix is actually provided as a separate file via the ‘Supplementary Materials’ link above. We have now merged it with the main text for better clarity.\n- **Table 2 may be redundant**: This could be true, but Table 2 may help readers to get a sense of the text dataset, as it is a relatively new dataset to the MI community. \n\nWe hope that we have addressed your concerns. Thank you again for reviewing our work. Please don’t hesitate to let us know if there are any additional comments or suggestions!"}, {"review_id": "mx6OGKKc28", "reviewer": "Reviewer_44kJ", "summary": "The paper utilizes the concept of vector copula introduced in [27] to develop a generative MI estimator (VCE). The authors discussed important properties of VCE such as consistency and convergence. They also commented on desirable features and interesting concepts such as separate learning/modeling and complexity-capacity trade-off. Moreover, given the interpretation of the PMI as a density rather than a density ratio, the authors studied neural-network free methodologies (e.g., MLE) to model the vector copula. The VCE estimator is compared with several established estimators (both discriminative and generative). Experiments show promising results for low-dimensional data and for low values of MI.", "questions": "Overall, I found this paper interesting and well-motivated, and I would like the authors to address the following questions to help clarify the contributions and potential impact:\n\n1. **Comparison with SOTA Methods.** Authors selected MRE [35] and MINDE [16] as SOTA estimators for discriminative and generative approaches, respectively. This is definitely arguable, at least for the discriminative approach as also SMILE [29] and $f$-DIME [36] are robust estimators. Moreover, [29] and [50] introduced common benchmarks and evaluation techniques which in this paper are not fully considered. Could the authors include richer comparisons with these methods? This would provide a more comprehensive evaluation and help establish where VCE stands relative to current best practices. Additionally, could the authors provide clear recommendations on when VCE should be preferred over other methods?\n\n2. **Bias-Variance Analysis.** The paper lacks discussion about bias-variance trade-offs, which is crucial for MI estimators. Could the authors provide a theoretical analysis or experimental evaluation of how the VCE estimator's bias and variance scale with sample size? This would help clarify the estimator's statistical properties and practical applicability. \n\n3. **Neural Network Parameterization.** The paper focuses on neural-network free methodologies for modeling the vector copula density. Given that the error depends on the KL divergence between $c$ and $c'$, could the authors provide at least one concrete example comparing neural network parameterization vs. classical methods for the vector copula density estimation? This would help understand the trade-offs and potential benefits of different modeling approaches, and could reveal whether the neural-network free approach is indeed still a valid path even for high dimensionality.\n\n4. **High MI and High Dimensionality Scenarios.** The experiments focus on low-dimensional data and low MI values, but the most challenging problems in MI estimation occur with high-dimensional data and high MI values. Could the authors extend their experiments to include high-dimensional scenarios (e.g., $d > 50$) and high MI values? This would demonstrate the method can be used for real-world applications and could significantly increase the significance of the paper if the VCE estimator shows advantages in these challenging regimes.\n\n5. **Comparison with Related Copula-based Methods.** The papers [R1, R2] briefly discuss the advantages of modeling the density ratio (not seen as a density) via classic copulas for the MI estimation task. I think it might be worthy add a comparison (at least written) with these papers. Similarly, can the authors elaborate more (perhaps add some sentences if not in the main also in the appendix about the difference between this paper and the paper in [40] as they both study the MI as negative differential entropy, but the former with the concept of vector copula?) I think it might help the reader better understand the novelty of this paper.\n\n6. **Double-Step Approach Validation.** The paper proposes a two-step approach: flow matching to obtain vector ranks followed by classical parametric methods (the authors mentioned about the possibility to train neural networks as well) to estimate the vector copula density. However, marginal distribution learning via flow matching generative models could not reach convergence and classical parametric methods are known to suffer from various limitations (e.g., model misspecification, curse of dimensionality). Could the authors provide a more thorough analysis of how this double-step approach actually improves upon existing single-step methods? Specifically, could they demonstrate quantitatively that the benefits of the conceptual separation outweigh the potential errors introduced by the two-step process? This would help validate whether the proposed approach genuinely advances the state-of-the-art or simply trades one set of problems for another.\n\n[R1] Samo, Yves-Laurent Kom. \"MIND: Inductive Mutual Information Estimation, A Convex Maximum-Entropy Copula Approach.\" arXiv preprint arXiv:2102.13182 (2021).\n\n[R2] Letizia, Nunzio A., and Andrea M. Tonello. \"Copula density neural estimation.\" arXiv preprint arXiv:2211.15353 (2022).", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper is technically sound and the theoretical analysis includes consistency and convergence/error proofs for the VCE estimator. The authors provide various experimental setups demonstrating the applicability of the method across different scenarios. The work is well-organized with clear sections and an accurate literature review, including a clear explanation of the vector copula concept and methodology. \n\nThe main contribution of the paper is the formalization of a novel MI estimator based on the concept of vector copula, with the validation of desirable properties. Another interesting contribution lies in interpreting the PMI as density itself rather than density ratio (which follows [27]), which could advance the field and shed light on how to treat the PMI. The authors are honest about limitations, which are clearly stated.", "was_revised": true, "final_justification": "The Authors have adequately addressed all my concerns, which are now resolved.", "rebuttal": "Thank you for your very insightful comments and the constructive criticisms. We believe many criticisms (for instance, no high-MI/high-dimensionality experiments, improper evaluation) were due to some misunderstandings, which we clarify below. Many of the requested comparisons/analyses can also be found in the provided appendix. Meanwhile, we have included new results and theories, as well as properly discussed related works to fully address your concerns.\n\n&nbsp;\n\n&nbsp;\n\n\n\n### **1a. Comparison with STOA Discriminative Methods**\n\nWe address your concerns on baselines from two aspects: (a) clarifying our baseline choice and (b) futher comparing with the suggested baselines.\n\n- (**MRE is a competitive baseline**) We fully agree that $f$-DIME and SMILE are STOA estimators, as we will note in our paper. That said, MRE is also highly competitive: it is verified on high-dimensional non-Gaussian data (80D) and high MI regimes (20+nats). Importantly, MRE shares key similarities with GAN-DIME---both methods estimate the ratio via classifier training and avoid computing the partition function. In practice, we found them almost equally strong, so we mainly compare with MRE for space efficiency.\n- (**Further comparison to SMILE**)  Following your suggestion, we have also comapred to SMILE; see the table below. The results highlight the advantanges of our method. A more complete comparison will be included in the manuscript.\n\nTable 1. Comparing SMILE vs VCE with 64D data. Results are collected from 8 independent runs. Code available in the provided code repo. \n\n|         &nbsp;&nbsp; |   &nbsp; Spiral &nbsp;   |   &nbsp; Cubic &nbsp;   |  &nbsp; Student-t &nbsp;  |   &nbsp; MoG  &nbsp;   |    &nbsp; MoT &nbsp;    |\n|-------|:----------:|:----------:|:-----------:|:----------:|:----------:|\n| SMILE-1 &nbsp;&nbsp; | &nbsp;1.54 ± 0.05&nbsp;| &nbsp;0.01 ± 0.02&nbsp;| &nbsp;1.82 ± 0.05&nbsp;| &nbsp;1.90 ± 0.06&nbsp;| &nbsp;1.14 ± 0.03&nbsp;|\n| SMILE-10 &nbsp;&nbsp; | &nbsp;3.22 ± 1.15&nbsp;| &nbsp;1.07 ± 0.08&nbsp;| &nbsp;5.12 ± 0.17&nbsp;| &nbsp;6.81 ± 0.20&nbsp;| &nbsp;3.22 ± 0.12&nbsp;|\n| SMILE-25 &nbsp;&nbsp; | &nbsp;2.72 ± 1.45&nbsp;| &nbsp;6.21 ± 0.34&nbsp;| &nbsp;4.60 ± 0.47&nbsp;| &nbsp;6.75 ± 0.20&nbsp;| &nbsp;3.08 ± 0.09&nbsp;|\n|   VCE   &nbsp;&nbsp; | &nbsp;8.27 ± 0.25&nbsp;| &nbsp;10.00 ± 0.30&nbsp;| &nbsp;9.76 ± 0.37&nbsp;| &nbsp;12.49 ± 0.37&nbsp;| &nbsp;7.20 ± 0.22&nbsp;|\n|    GT   &nbsp;&nbsp; | &nbsp;10.77&nbsp;      | &nbsp;10.77&nbsp;      | &nbsp;12.46&nbsp;      | &nbsp;13.19&nbsp;      | &nbsp;6.90&nbsp;      |\n|  |  |  |  |  |  |\n\nIf necessary, we are fully happy to replace MRE with $f$-DIME or SMILE in our main text. We wish it understandable that it was unrealistic to include every baseline due to space limit. \n\n&nbsp;\n\n&nbsp;\n\n### **1b. Issues with Benchmark & Evaluation Protocol** ###\n\nWe clarify that **our benchmark and protocols highly overlap with SOTA** benchmarks [44, 50, 51] and recent estimators (and **may even be richer**). See Table 2 for a comparsion. \n\nTable 2. Comparing the benchmarks and evaluation protocols in different works. \n\n|  | &nbsp;  **Synthetic distributions** | **Real-world dataset**  | **Evaluation protocol**|\n| --- | --- | --- | --- |\n| **$f$-DIME** | &nbsp;  Gaussian, cubic, half cube, asinh, uniform, student-t, swiss roll | MNIST & FMNIST | Direct comparison with GT + SC test$^†$\n| **$\\mathcal{N}$-MIENF** | &nbsp;  Gaussian, cubic, asinh+student-t, uniform, student-t  &nbsp; | Images with known MI | Direct comparison with GT |\n| **VCE** | &nbsp; Gaussian, cubic, tanh-and-exp, student-t, swiss roll, spiral, mixture distributions &nbsp; | Images with known MI, LLM embeddings &nbsp; | Direct comparison with GT |\n|  |  |  | |\n\n$^†$We do not employ the self-consistency (SC) test from [29], as all our test cases (including images and texts) have known and controllable ground truth (GT), requring not using this test.\n\n\n\n\n\n&nbsp;\n\n&nbsp;\n\n\n### **2. Bias-and-variance Analysis** ###\n\n- (**Empirical analysis**) Following your valuable suggestion, we have now included an empirical variance analysis of VCE. See Table 3 for an example. \n\nTable 3. Empirical variance of VCE with 64D data w.r.t sample size $n$. Reported values are the std across 16 randomly sampled batches.\n\n|                | n=50        | n=100       | n=500       | n=1000      | n=2000      | n=5000      | GT         | \n| -------------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ---------- |\n| Student-t (v=1) &nbsp; &nbsp;| 0.6242      | 0.4764      | 0.1814      | 0.1787      | 0.0507      | 0.0342      | 12.0022    |\n| Spiral       &nbsp; &nbsp;  | 1.4173      | 0.3176      | 0.3114      | 0.2741      | 0.1451      | 0.1218      | 10.7725    |\n|||||||||\n\n\n- (**Theoretical analysis**). We also derive a new proposition analyzing how the number of mixture components $k$ governs the bias-variance trade-off in vector copula modeling:\n\n(*Proposition 5*) For large sample size $n$, the expected KL divergence between the true vector copula $c$ and its $k$-component vector Gaussian copula mixture estimate $\\hat{c}_k$ satisfies\n\n$\n\\mathbb{E} \\left[ \\mathrm{KL}[c || \\hat{c}_k] \\right] \\leq C_1 \\cdot k^{-\\frac{2\\alpha}{d}} + C_2 \\cdot \\frac{k d \\log (C_3 n)}{n},\n$\n\nwhere $\\alpha, C_1, C_2, C_3 > 0$ are constants depending on the true vector copula solely, $d$ is the data dimension and $n$ is the sample size. Here, the 1st term on the RHS quantifies the **bias**, which decreases as $k$ grows and is irrelevant to the sample size $n$. The 2nd term on the RHS corresponds to the **variance**, which increases as $k$ grows and decreases as $n$ grows.\n\n\n\n&nbsp;\n\n&nbsp;\n\n### **3. Neural Parameterization vs Model-based Parameterization**\n\nThank you for highlighting this comparison --- we are on the same page! **Figure 1 in Appendix B2** precisesly perform this comparison. In the figure, we compare **VCE (model-based parameterization)** and **VCE’ (neural parameterization)** in diverse settings. The results clearly show the advantange of model-based parameterization in the considered settings. \n\nWe fully understand that the reviewer is skeptical regarding the applicability of classic methods in high-dimensional space. However, please note that we are not applying classic methods directly in the original space, but rather a transformed space learned by flow models. This transformation bridges the gap, making classic method viable in this space.\n\n&nbsp;\n\n&nbsp;\n\n\n### **4. High MI and High Dimensionality (d>50) Scenarios**\n\nWe clarify that our original submission actually include both these cases (though they might not be so easy to notice due to our presentation); see Table 4 below for a summary. Our method demonstrates either significant advantange or highly competitive performance in both cases.\n\nTable 4. MI and data dimesnionalities considered in this work. \n\n|  | Detail &nbsp; | Evidence |\n| --- | --- | --- |\n| High MI &nbsp; | 14nats non-Gaussian data, 7nats image and text data &nbsp; | Figure 1a-1e cases $\\rho=0.8$, Figure 2a-2e cases $d=64$  |\n| High dimensionality &nbsp; | 64D imcompressible data, 32x32 images*, 512D text embeddings*  &nbsp; | Figure 2a-2e cases $d=64$, Section 6.2, Section 6.3 |\n||||\n\n*For compressible data, we leverage injective flow (implemented as autoencoder + latent flow). The same autoencoder is used for all other estimators for fairness.\n\n\n&nbsp;\n\n&nbsp;\n\n### **5. Comparison to Related Copula Methods** ###\n\n- **Additional paragraph comparing our work with [R1, R2]**\n\n“*Copula has been widely used for MI estimate [R1, R2, other works]. Existing methods primarily focus on classic copulas, where the copula transformation is applied independently to each univariate marginal. This strategy has been shown to improve accuracy and reduce variance [R2]. We go one step further by using vector copulas, where the transformation jointly considers all dimensions of the multivariate marginals. This can be seen as a generalization of classic copula transformation, where we not only consider MI's invariance to element-wise bijections but also to any bijective function. Another key difference lies in that existing works [R1, R2, …] treats PMI as a density ratio, whereas our work treats PMI as a density.*“\n\n- **Texts clarifying the difference between our work and [40]**\n\n“*Similar to our work, prior work [40] also interprets MI as the differential entropy of copula. However, building upon classic copulas, their result only holds for scalar marginals, and generalizing their result to vector marginals require non-trivial formulation and derivation. Our work in contrast holds for vector marginals with any dimensionalities by leveraging vector copulas.*”\n\nWe promise that all these discussions will be included in the final version. Thank you once gain for highlighting this important comparison. \n\n&nbsp;\n\n&nbsp;\n\n\n\n\n\n### **6. Double Step Approach Validation** ###\n\nWe are on the same page again --- the ablation study **Joint Learning vs Separate Learning in Appendix B2** precisely performs this quantitative analysis. The results clearly demonstrate the benefits of the double step approach over single step.\n\nNote that even if the learned vector ranks are imperfect, double step learning can still be useful. This is because the first step will not alter the underlying MI. In this regard, computing vector ranks can be viewed as an informed data-preprocessing step to improve MI estimation --- akin to the scalar rank transformation used in the CODINE paper [R2].\n\n&nbsp;\n\n&nbsp;\n\n\n**Summary**: VCE is a robust and accurate MI estimator across diverse settings, including high-MI regimes (e.g. 14nats), high-dimensional data (e.g., 64D incompressible data, 512D text embeddings), long tails, non-Gaussian dependencies, heteogeneous marginals, etc. However, it may be is less suitable for incompressible data with very high dimensionality (>128D)."}]

Author rebuttal:

Meta-review: The paper shows that mutual information can be recast as a vector copula estimation task. This work generalizes previous results linking copulas and information for univariate variables. A vector copula method for estimation is clearly elaborated and tested, Theoretical arguments motivate the approach, and experiments show its competitiveness compared to existing methods for MI estimation. The paper is relevant because it addresses the very important area of MI estimation. The presented theoretical developments are sound and clear. In particular, Theorem 2 provides a sound direction for future progress. The proposed estimator offers flexibility for selecting an appropriate model capacity for the perceived complexity of a task. These strengths make this contribution relevant to NeurIPS.
Decision: Accept (poster)