Title: Neural Mutual Information Estimation with Vector Copulas
Venue: NeurIPS 2025
Accepted: yes
Keywords: mutual information

flow-based models

copulas

vector ranks
Abstract: Estimating mutual information (MI) is a fundamental task in data science and machine learning. Existing  estimators mainly rely on either highly flexible models (e.g., neural networks), which require large amounts of data, or overly simplified models (e.g., Gaussian copula), which fail to capture complex distributions. Drawing upon recent vector copula theory, we propose a principled interpolation between these two extremes to achieve a better trade-off between complexity and capacity. Experiments on state-of-the-art synthetic benchmarks and real-world data with diverse modalities demonstrate the advantages of the proposed method.

Neural Mutual Information Estimation with Vector Copulas

Abstract

Estimating mutual information (MI) is a fundamental task in data science and machine learning. Existing estimators mainly rely on either highly flexible models (e.g., neural networks), which require large amounts of data, or overly simplified models (e.g., Gaussian copula), which fail to capture complex distributions. Drawing upon recent vector copula theory, we propose a principled interpolation between these two extremes to achieve a better trade-off between complexity and capacity. Experiments on state-of-the-art synthetic benchmarks and real-world data with diverse modalities demonstrate the advantages of the proposed estimator.

1 Introduction

Mutual information (MI) is a fundamental measure of the statistical dependence between random variables (RVs). Compared to other dependence measures, MI stands out due to its equitability and generality [1, 2]: it can capture non-linear dependence of any form and can handle RVs with any dimensionalities, rendering it a powerful measure for quantifying statistical dependence. In data science, MI is widely used to analyze the relationships between protein sequences [3] and gene profiles [4, 5], as well as to assess feature importance and redundancy [6]. In machine learning, MI broadly serves as a learning objective and regularizer [7, 8, 9, 10, 11, 12], with diverse applications to representation learning [7, 8, 9, 13, 14], generative modeling [10], fairness and privacy [15, 16], etc.

Awide range of powerful, neural MI estimators have been developed [17, 18, 19, 20, 21, 22, 23]. Most of these estimators rely on a single, unconstrained network to approximate certain quantities-such as the joint density p ( x , y ) or the density ratios p ( x , y ) /p ( x ) p ( y ) -during MI estimation. While neural networks as universal functional approximators can, in theory, approximate arbitrary functions given sufficient data [24, 25], in practice we often only have a small set of data. Indeed, theoretical studies have shown that such distribution-free treatment of MI estimation will inevitably suffer from requiring an exponential sample size [26, 27, 28, 29]. A straightforward remedy is to restrict the model to simpler classes-for instance, assuming that the data is approximately Gaussian. However, these assumptions are often overly simplistic to capture complex distributions in reality.

Recent advances in vector copula theory [30] offer a promising avenue for addressing this dilemma. Vector copula theory extends classical copula theory [31] by generalizing it from univariate to vector marginals. It reveals that the multivariate marginals and the dependence structure (i.e., the vector copula) of a joint distribution are fully disentangled. This disentanglement motivates a more fine-grained way for making assumption in MI estimation, where we impose lightweight yet reasonable assumptions solely on the vector copula rather than on the entire distribution . Crucially, the complexity of the vector copula can be adaptively adjusted through efficient vector copula selection, allowing for an optimal trade-off between capacity and complexity. Experiments on stateof-the-art synthetic benchmarks and real-world data demonstrate the competitiveness of our estimator against state-of-the-art estimators. In summary, the main contributions of this work are three-fold:

p(x)

p(ux)

p(uy)

I(X;Y) = -H[c(ux,uy)]

①

2

①

We develop a divide-and-conquer MI estimator based on recent vector copula theory, which explicitly disentangles marginal distributions and dependence structure in MI estimation;

We reinterpret existing estimators through the lens of vector copula, revealing that they correspond to varying parameterization and learning strategies of vector copula with various trade-offs;

We provide consistency and error analysis of our estimator, along with extensive empirical evaluation on diverse test cases covering multiple modalities, marginal patterns and dependence structures.

Code containing both our method and state-of-the-art neural estimators is available in [github repo].

2 Preliminaries

Throughout this work, we use upper case letters (e.g. X ) to denote random variables and lower case letters (e.g. x ) to denote their instances. We use U [0 , 1] d or µ to denote the uniform distribution on [0 , 1] d and use N to denote Gaussian distribution on ❘ d . ∇ denotes the gradient and J x y denotes the Jacobian of y w.r.t x . The symbol # denotes the push-forward operation.

2.1 Mutual information and its estimation

The mutual information (MI) between variables X and Y is defined as the Kullback-Leibler (KL) divergence between the joint distribution p ( x , y ) and the product of marginal distributions p ( x ) p ( y ) :

In this work, we consider estimating I ( X ; Y ) from an empirical dataset D = { x ( i ) , y ( i ) } n i =1 . Several neural network-based methods have been developed for MI estimation:

Generative estimators . These methods leverage generative models to approximate the various distributions in (1) or their equivalents, and use the learned generative models to construct an MI estimate [17, 29, 18, 32, 19]. The accuracy of generative estimators crucially depends on the quality of the learned generative models. Simpler models (e.g. Gaussian copula) are easy to learn but may fail to adequately capture the true data distribution [33, 34]. In contrast, complex models (e.g. flow-based models [35, 36, 37, 38] and diffusion models [19]) offer greater expressiveness but can be challenging to optimize, in particular if the amount of data is insufficient or the data dimensionality is high.

Discriminative estimators . These methods train a neural network f with samples x , y ∼ p ( x , y ) and samples x , y ∼ p ( x ) p ( y ) to estimate the density ratio p ( x , y ) /p ( x ) p ( y ) [20, 21, 39, 40, 41, 32, 23]. Once trained, the learned density ratio can either be used in (1) or in the Donsker-Varadhan (DV) representation [42] to obtain an MI estimate. Discriminative methods avoid directly modeling densities, however they are prone to the curse of high-discrepancy [40, 41, 29, 32], which occurs if p ( x , y ) and p ( x ) p ( y ) differ significantly - for instances, cases with high MI or high-dimensional data. Several advanced methods were proposed to alleviate this issue, including clipping the network outputs [32], introducing reference distributions [41], avoiding computing the partition function [23].

2.2 Vector copula

Vector copula theory . The recent vector copula theory [30] provides a principled framework for modeling and analyzing the dependence between multivariate random variables. It extends classical copula theory [31] by considering 'vector' marginals. We begin by the concept of vector ranks:

Definition 1 (Vector rank) . Let p be an absolutely continuous distribution on ❘ d with support in a convex set. Let µ be the uniform distribution on [0 , 1] d . There exists a convex function ψ such that ∇ ψ # µ = p and ∇ ψ -1 # p = µ . u := ∇ ψ -1 is called the vector rank associated with p [43].

When d = 1 , vector rank reduces to standard scalar rank. Intuitively, vector rank transforms a multivariate distribution p to a (multivariate) uniform distribution µ , entirely removing its characteristics.

In the text below, we slightly overload this definition and use the term 'vector rank' to refer to both the vector rank function u ( · ) and also the corresponding random variable u induced by this function.

Definition 2 (Vector copula) . Let u X and u Y be the vector ranks corresponding to p ( x ) and p ( y ) respectively. A vector copula C ( u X , u Y ) is a cumulative distribution function on [0 , 1] d X + d Y with uniform marginals on C ( u X ) = U [0 , 1] d X and C ( u Y ) = U [0 , 1] d Y . The probabilistic density function corresponding to C is called vector copula density and is denoted as c ( u X , u Y ) [30].

Given the above definition, we have the following result [30] generalizing the Sklar theorem [31].

Theorem 1 (Vector Sklar Theorem) . Let X ∈ ❘ d X and Y ∈ ❘ d Y be two random variables with joint distribution p ( x , y ) on ❘ d X + d Y . For any absolutely continuous distributions p ( x , y ) with support in a convex set, there exist an unique function c ( · , · ) , such that

where u X and u Y are the vector ranks computed for x and y respectively. The function c equals to the vector copula density associated with u X and u Y [30].

The vector Sklar theorem suggests that for a distribution p ( x , y ) , its marginal distributions and the joint dependence structure are entirely disentangled, with the latter fully characterized by the vector copula density c . Note that here we focus on the case of two RVs; we refer to [30] for general cases.

Instances of vector copula . We discuss several instances of vector copula related to our work. One important instance is the vector Gaussian copula [30]. This model assumes that the joint dependence structure admits a Gaussian structure, with its vector copula C N being

where Σ = [[ I X , Σ XY ] , [Σ ⊤ XY , I Y ]] is a p.s.d matrix whose blocks I X ∈ ❘ d X × d X and I Y ∈ ❘ d Y × d Y are identity matrices. Φ( · ) is the cumulative distribution function of multivariate normal distribution and ϕ ( · ) is the (element-wise) cumulative distribution function of univariate normal distribution. Equivalently, a vector Gaussian copula can be defined by its data generation process: ϵ ∼ N ( ϵ ; 0 , Σ) , u X = ϕ ( ϵ ≤ d X ) , u Y = ϕ ( ϵ >d X ) , with ϵ ≤ d X and ϵ >d X being the first d X and the remaining dimensions of ϵ respectively. An analytic expression for c N can be derived accordingly.

Other useful instances of vector copula include t -vector copula, Archimedean vector copula and Kendall vector copula, which correspond to different inductive biases about the dependence structure.

3 Methodology

In this section, we propose a new mutual information (MI) estimator based on vector copula theory. The core of our method is Theorem 2, which establishes a connection between MI and vector copula:

Theorem 2 (MI is vector copula entropy) . The mutual information I ( X ; Y ) is the negative differential entropy of the vector copula density:

where u X and u Y are the vector ranks corresponding to p ( x ) and p ( y ) respectively.

Algorithm 1 Vector copula MI estimate (VCE)

Input:

data D = { x ( i ) , y ( i ) } n i =1

Output:

estimated ˆ I ( X ; Y )

Parameters:

flows f X , f Y , copulas { c 1 , ..c M }

Initialization:

D = D train ∪ D val , K = 1 ,

▷

Marginal distributions learning learn

X

with

=

D

{

x

learn

f

X

f

Y

with for

in

i

1

to

n

compute compute

end for

)

(

i

)

n

i

=1

}

}

n

i

f

=1

X

(

f

Y

x

(

(

y

(

by FM;

by FM;

i

;

))

)

i

)

u

ˆ

Y

D

(

i

)

do

X

ˆ

u

(

i

=

)

Y

▷ Vector copula density estimation repeat

Algorithm 2 Vector copula MI estimate' (VCE')

Input:

data D = { x ( i ) , y ( i ) } n i =1

Output:

estimated ˆ I ( X ; Y )

Parameters:

flows f X , f Y , ratio estimator r

Initialization:

reference copula c ′ , D ′ = ∅

▷ Marginal distributions learning learn f X with D X = { x ( i ) } n i =1 by FM; learn f Y with D Y = { y ( i ) } n i =1 by FM; for i in 1 to n do compute

rank

=

compute end for

x

(

(

y

(

(

i

i

)

)

(

i

)

X

(

i

)

Y

▷ Vector copula density estimation repeat

This theorem generalizes the results of [44, 45] from univariate to vector marginals 1 . It establishes that MI depends solely on the vector copula, which itself is invariant to marginal distributions. Notably, the theorem also reveals that the pointwise mutual information (PMI) i.e. p ( x , y ) /p ( x ) p ( y ) can equivalently be viewed as a density c ( u X , u Y ) in its own right, in contrast to the vast majority of existing works [46, 47, 48, 49] which continue to treat PMI as a density ratio . This shift in perspectives opens us new possibility in the parameterization and learning of the PMI, including directly modeling it as a normalized density learned via MLE, as will be discussed later.

Theorem 2 immediately suggests a new divide-and-conquer approach for MI estimation: we can first estimate the vector ranks u X and u Y , followed by subsequent learning of the vector copula c 2 :

where ˆ u X , ˆ u Y and ˆ c are the empirical estimates to u X , u Y and c respectively.

We discuss below several potential advantages of the above divide-and-conquer estimation strategy:

By disentangling the modeling of marginals distribution and copula, we can use differently-sized models in their parameterization, avoiding using a single overly flexible or overly simplified model for the entire distribution. This leads to a better trade-off between model complexity and capacity;

By disentangling the learning of marginals and copula, we can reuse the pre-trained marginals across multiple copula choices with varying complexities, allowing model selection to be performed solely in the copula space in a computational efficient way. It also reduces overall learning difficulty.

In the following, we elaborate methods to estimate the vector ranks and the vector copula respectively.

))

))

;

;

u

ˆ

ˆ

u

=

rank

(

(

f

f

X

Y

=

))

;

=

y

{

(

rank

i

(

rank

(

3.1 Marginal distribution learning

In this step, we learn the two marginal distributions p ( x ) and p ( y ) with flexible flow-based models [35, 36, 37, 38] and use them to compute the vector ranks u X and u Y .

Flow-based modeling of marginals . Let f X : ❘ d X → ❘ d X and f Y : ❘ d Y → ❘ d Y be two flow-based models and let p f X ( x ) and p f Y ( y ) be the densities induced by f X and f Y respectively. We respectively learn f X and f Y with data x ∼ p ( x ) and data y ∼ p ( y ) by flow matching [38]:

where L FM denotes the flow-matching loss [38]. Upon convergence, f X and f Y respectively transform the two marginals to a standard normal distribution: N ( 0 , I ) ≈ f X # p ( x ) and N ( 0 , I ) ≈ f Y # p ( y ) .

Vector ranks computation . With the learned flows f X and f Y , we compute the vector ranks as:

where rank d ( ϵ ) = 1 n +1 ∑ n j =1 1 [ ϵ d ≥ ϵ ( j ) d ] is the element-wise ranking function that computes the scalar ranks for each of the dimension in ϵ . Given universal density approximators f X , f Y , ˆ u X and ˆ u Y serve as consistent estimates of the true vector ranks u X and u Y .

While the joint density p ( x , y ) is often challenging to estimate, the marginal distributions p ( x ) and p ( y ) are typically far easier to learn due to their lower dimensionality. It is thus reasonable to expect that ˆ u X and ˆ u Y are close approximations to u X and u Y in moderate dimensionality settings.

Remark . The above process of estimating u X and u Y can be viewed as a generalization of classic copula transformation in MI estimation, where we compute vector ranks rather than scalar ranks.

3.2 Vector copula estimation

In this step, we learn the vector copula c with the previously estimated vector ranks ˆ u X and ˆ u Y , leveraging a model-based parameterization and a careful model selection strategy.

Model-based parameterization of copula . As noted earlier, any parametric model can be used to represent the vector copula c , regardless of whether an analytical PMI is available. In this work, we parameterize c as a mixture of existing parametric vector copulas [30] from the copula pool, whose model complexity can be well controlled by tuning the number of mixture components:

where ∑ K k =1 p k = 1 and each c k ∈ C is selected from the predefined pool C of vector copulas. Any inductive bias about the dependence structure can be used to guide copula selection. Here, we simply implement each c k as a vector Gaussian copula and learn c by maximum likelihood estimate (MLE):

In theoretical analysis, we analyze why this copula design is a cheap yet reasonable modeling of c .

Efficient model selection . A key design in our method is the explicit exploration of the capacity-complexity trade-off in copula modeling, which is governed by the number of mixture components K . Here, we determine K by cross validation, using negative log-likelihood (NLL) as the criterion. This process is computationally cheap: each copula is already lightweight, involving no neural networks; furthermore, different copulas can be trained in parallel using one single loss.

Algorithm 1 summarizes the main pipeline of the proposed vector copula-based estimator (VCE).

Remark . As an alternative to the above model-based parameterization, one may also adopt a referencebased parameterization for the vector copula, inspired by the design in [51]. Specifically, let c ′ be a reference vector copula that is easy to sample (e.g. a vector Gaussian copula). We can learn c by first estimating the density ratio r = c/c ′ using samples from c and c ′ [52, 41, 39], then recover the vector copula c as c = r · c ′ ; see Algorithm 2. By parameterizing r as a deep neural network, this method allows for a more flexible modeling of c , at the cost of a less fine-grained control over its complexity.

4 Theoretical analysis

In this section, we analyze several important theoretical properties of the proposed VCE estimator.

Proposition 1 (Consistency of VCE) . Assuming that (a) f X and f Y are universal PDF approximators with continuous support and (b) the number of mixture components K in (8) is sufficiently large. Define ˆ I n ( X ; Y ) := 1 n ∑ n i =1 log ˆ c (ˆ u ( i ) X , ˆ u ( i ) Y ) . For every ϵ > 0 , there exists n ( ε ) ∈ N , such that

Proof . Please refer to Appendix A

Additionally, we have the following result analyzing the estimation error w.r.t the quality of the learned marginals p f X ( x ) , p f Y ( y ) and the estimated vector copula density ˆ c .

Proposition 2 (Error of vector copula-based MI estimate) . Let ˆ u X and ˆ u Y be the estimated vector ranks. Let c (ˆ u X , ˆ u Y ) be the true joint distribution of ˆ u X and ˆ u Y , and ˆ c (ˆ u X , ˆ u Y ) its estimate. Assuming that sufficient Monte Carlo samples are used to compute ˆ I ( X ; Y ) in (5) , we have

where the first term on the RHS vanishes as p f X ( x ) → p ( x ) and p f Y ( y ) → p ( y ) . In the limit of perfectly learned marginals, the error simplifies to

with c and ˆ c being the true vector copula density and estimated vector copula density, respectively.

Proposition 2 decomposes the estimation error of the proposed VCE estimator into two components:

Marginal estimation error . Imperfect marginal estimations introduce a bias given by | H (ˆ u X ) + H (ˆ u Y ) | > 0 , which diminishes as both marginals are learned more accurately (recall that ideally, we have ˆ u X ∼ U [0 , 1] d X and ˆ u Y ∼ U [0 , 1] d Y ). For data with moderate dimensionality, we expect this bias to be small, as the two marginals are with low-dimensionality, being easy to estimate.

Dependence structure modeling error . This error arises from the discrepancy between the estimated copula ˆ c and the true copula c . It depends on two factors: (a) capacity - whether the parameterization of ˆ c is sufficiently expressive to approximate c ; and (b) complexity - how easy ˆ c can be learned from the limited data. These factors highlight the importance of model selection for the copula c .

Proposition 3 (Vector Gaussian copula as second-order approximation) . A vector Gaussian copula c N corresponds to the second-order Taylor expansion of the true vector copula c ∗ up to variable transformation.

This result explains our choice of using a mixture of Gaussian copulas as a cheap yet principled approximation to the true vector copula. A single vector Gaussian copula already offers a reasonable approximation of the true copula by capturing dependencies up to second order; higher-order interactions, if necessary, can be modeled by adding mixture components in a fully controllable way.

Finally, we have the following result regarding cases with weakly dependent random variables (RVs).

Proposition 4 (Vector copula of independent RVs) . The vector copula corresponding to the product of marginals p ′ ( x , y ) = p ( x ) p ( y ) is a vector Gaussian copula if p ′ ( x , y ) is absolutely continuous.

Proposition 4 suggests that if the two RVs X and Y are nearly independent, our estimator is likely to provide an accurate estimation of I ( X ; Y ) as the true vector copula is Gaussian-like, being close to the family of our copula design (8). For weakly dependent RVs, it is reasonable to expect that p ( x , y ) resembles a vector Gaussian copula, with the difference captured by the additional components in (8).

5 Reinterpreting existing MI estimators

In this section, we reinterpret existing MI estimators through the lens of vector copula theory, showing that they correspond to different parameterizations and learning strategies of the vector copula.

Reinterpreting discriminative estimators . Existing critic-based approach to MI estimation [20, 53, 39, 41, 32, 23] can be interpreted as parameterizing the vector copula c ( u X , u Y ) using a feedforwarding neural network f :

which is learned by discerning samples from the joint p ( x , y ) and the product of marginals p ( x ) p ( y ) (via e.g contrastive learning). Specifically, recall that the optimal critic f in these methods corresponds to the log density ratio up to an additive constant C [54]: f ( x , y ) = log p ( x , y ) /p ( x ) p ( y ) + C , with the PMI itself equal to the vector copula density, as established by the vector Skalar theorem.

Compared to our model-based parameterization of the vector copula density in (8), this neural network parameterization is more flexible and can potentially capture more complex dependence structures. However, as discussed earlier, such distribution-free parameterizations lack complexity control, which may lead to a poor bias-variance trade-off. Furthermore, discriminative methods learns the vector copula by comparing distributions, which can be challenging if they differ significantly (e.g., in high-MI cases, see [40, 41, 29, 32]. In contrast, our main method learns the vector copula by maximum likelihood estimate (MLE), which is the most efficient consistent estimator for the copula.

Reinterpreting generative estimators . Many generative estimators for MI [17, 29, 18, 48, 32] require either learning the joint distribution p ( x , y ) = p ( x ) p ( y ) c ( u X , u Y ) or the conditional distribution p ( y | x ) = p ( y ) c ( u X , u Y ) using a single model. This process can be interpreted as learning the marginal distribution(s) and the vector copula simultaneously, with the two components parameterized jointly via a single generative model. Our method, on the contrary, explicitly separates the modeling and the learning of the marginal distributions p ( x ) , p ( y ) from that of the vector copula c ( u X , u Y ) . This strategy not only enables a more fine-grained control over model complexity, but also mitigates the challenge of jointly learning the marginal distribution and the dependence structure-a strategy aligned with the spirit of classical copula transformations [47, 55, 56, 57] to simplify MI estimation.

We further discuss two recent works [48, 17] closely related to our work. These methods operate by respectively transforming the two RVs X and Y by two flow-based models, such that the joint distribution of the transformed data can be approximated by a distribution with an easy-to-compute MI (for instance, a Gaussian distribution). Their practical methods, N -MIENF and DINE-Gaussian, can be reinterpreted as assuming the dependence structure as a vector Gaussian copula (see Lemma 3 in Appendix A4 for a detailed derivation):

which corresponds to the case K = 1 in the VCE estimator and is accurate (only) if the true dependence is Gaussian-like. The possibility of using non-Gaussian base distribution is also discussed in [48], albeit without practical implementation. Additionally, the marginals and the vector copula in their method are learned jointly rather than separately as in our method, and they continue to treat PMI as a density ratio p ( x , y ) /p ( x ) p ( y ) , unlike our method which treats it as a density c ( u X , u Y ) .

6 Experiments

Baselines . We consider five representative neural estimators in the field: MINE [20], InfoNCE [21], MRE [41], MINDE [19] and N -MIENF [48]. The first three methods are critic-based whereas the latter three are generative model-based. MRE is chosen as the representative of state-of-the-art discriminative methods, which is specifically designed to address the high-discrepancy issue in these methods. MINDE is chosen to represent the state-of-the-art generative methods, which leverages powerful diffusion model in MI estimation. Further baselines are considered in Appendix B2.

Hyperparams . For the vector copula in VCE, we consider mixtures with 1 , 4 , 8 , 16 , 32 components.

Neural architecture, optimizer and training details . Please refer to appendix B1 for more details.

In the following evaluation, we primarily focus on evaluating the VCE estimator (Algorithm 1), and present the results of the alternative VCE' estimator (Algorithm 2) in the appendix. All results are collected through 8 independent runs. Error bars reported are the standard deviations (std) of the runs.

(a)

X,Y

∼ N

(b)

A

tanh

(

X

)

,

B

e

Y

(c)

A

X

3

,

B

Y

3

t

(e) Swiss Roll

f(x).g(y)=x,y

f(x).g(y)=Atanh(x),Bey

f(x),g(y)=Ax3,By3

Student-t (v=1)

SwissRoll

MINE

8.0

8.0

10

0.6

15

12

日

V

MRE

MINDE

N-MIENF

6.0

中

中

6.0

8

6

0.5

0.4

9

6

3

口

VCE

I(X;

4.0

2.0

市

4.0

2.0

四

X）

A

I(X;

0.3

0.2

0.1

0.0

0.0

0.0

-0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

-0.8-0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

-0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

-0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

-0.8 -0.6 -0.4 -0.2 0.0 0.2 0.4 0.6 0.8

p

p

(a) Spiral

(b)

A

X

3

,

B

Y

3

t

(d) MoG 1

(e) MoG 2

Spiral

f(x)，g(y)=Ax²，By3

Student-t (v=1)

MoG

MoG 2

(X;

12

10

8

6

4

口

VCE

MINE

MINDE

MRE

N-MIENF

Truth

1

X)/

10

8

4

2

中

I(X;

12

10

8

4

6

O

4

中

X

14

12

10

8

6

4

2

X)

5

3

2

1

8

7

6

4

Q

4

8

16

d

32

64

Q

4

8

16

d

32

64

Z-

4

00

16

d

32

64

4

8

16

d

32

64

O

4

8

16

d

32

64

6.1 Synthetic distributions

Setups . In [58], a diverse set of models with known MI are developed to comprehensively evaluate MI estimators. We consider representative cases from this benchmark, further extending it by (a) considering varying dependence strengths for each chosen case; (b) employing mixing matrices A , B to couple the dimensions in X and Y respectively. We also include the mixture models in [49] to enrich our tests. Together, our test cases cover non-Gaussianity, skewness, heterogeneous marginals, long tails, low-dimensional manifold structure, coupling dimensions, high-dimensionality, varying dependence strengths and non-Gaussian dependence structure. Each test case contains n = 10 4 data.

Results . Figure 2 and Figure 3 compare the performance of different MI estimators 3 . Overall, VCE provides good estimates in all scenarios, consistently ranking among the top performers.

Compared to discriminative methods e.g., MINE and InfoNCE, VCE demonstrates significant advantages, particularly in high MI settings (e.g. strong dependence level ρ or high dimensionality d ). This advantage may be because our method avoids directly comparing two highly distinct distributions as in these methods, which is challenging. The advantage may also attribute to the better complexity-capacity trade-off in our method, which avoids an overly powerful model for the copula.

Compared to the generative method N -MIENF, VCE demonstrates advantages in scenarios involving non-Gaussian dependence structures (see e.g. the MoG cases and 64D t -distribution). In such cases, N -MIENF's assumption of a Gaussian dependence structure falls short in capturing the true dependence structure. This underscores the pitfalls of using a overly simplified model for the copula.

We specifically discuss two challenging cases highlighted in prior works [58, 19]: (a) Spiral transformation, which highly transforms the original data; and (b) multivariate t -distribution with degree of freedom ν = 1 , which exhibit heavy-tailed dependence. For these two highly challenging scenarios, VCE and MINDE are the only two methods that can simultaneously provide reasonable estimates in both cases, with VCE outperforming MINDE in other settings (see e.g., Figure 2.c and Figure 3.b).

(a)

Gaussian plates

(b)

Rectangles

6.2 Image dataset with known MI

Setups . We next consider the benchmark [59], which contains correlated images X and Y ; see Figure 4. Here X ∈ ❘ 16 × 16 and Y ∈ ❘ 16 × 16 , and the ground truth I ( X ; Y ) is known for this dataset. Following recent works [59, 3], we preprocess these high-dimensional image data by an autoencoder e : ❘ 16 × 16 → ❘ d ′ , which proves effective in reducing data dimensionality while preserving key information. The quality of such compression w.r.t d ′ is analyzed theoretically and empirically in Appendix A5 and B2, based on which we set d ′ = 16 . A total number of 10 , 000 data is used. Note that while the dependence between X and Y are Gaussian for this dataset [59], the dependence structure for the compressed data can be non-Gaussian even if the compression is near-lossless.

Results . Table 1 compares the performance of different MI estimators on this task. Our estimator consistently outperforms the recent N -MIENF estimator on this dataset, and it shows highly competitive performance against discriminative methods. However, our method performs slightly worse than discriminative methods in the Rectangles case. One reason why our approach loses to discriminative approaches in the Rectangles case may be that the underlying dependence structure of the preprocessed data is highly complex in this case, which is difficult to model effectively with a single vector Gaussian copula or even a reasonable mixture of such copulas. Discriminative methods, on the contrary, adopt a neural network-based parameterization of the vector copula, being inherently more flexible. These results highlight the limitation of model-based parameterization of the vector copula density in certain cases. Nonetheless, our estimator still provides a highly reliable estimate.

6.3 Embeddings of language models

Setups . We further consider a real-world dataset in natural language processing. It consists of pairs of embeddings from a language model (LM) [60, 61] computed on the IMDB dataset [62], which contains negative or positive movie comments; see Table 2. The ground truth MI of this dataset is unknown, but it can be computed numerically accurately; see Appendix B1. A total number of n = 4 × 10 3 data are used. Similar to the previous task, we preprocess data by an autoencoder e : ❘ d LM → ❘ 16 , with d LM being the dimensionality of the LM's embeddings. The quality of such compression is empirically studied in Appendix B2, which is near-lossless.

Results . Table 3 summarizes the results for this dataset. In this scenario, where the underlying mutual information (MI) is relatively low, our method does not show a significant advantage over discriminative methods. This is likely because for this dataset, the high-discrepancy issue [40, 41, 29, 32] is not significant, and discriminative methods offer a more flexible parameterization of the vector copula density c than our method (see Section 5). Nonetheless, our method still provides an estimate close to discriminative methods, and it significantly outperforms the generative method N -MIENF.

6.4 Further analysis and ablation studies

We conduct further analysis on the effect of model selection and separate learning in Appendix B2.

(a) Llama-3 13B

X

Y

1

(positive)

I

thought this

was

a

wonderful

way to

spend

time

on ...

(positive)

If you

like

original

gut

wrenching

laughter

you

will

like

...

2

(negative)

So

im not

a

big

fan

of

Boll's

work but

then

...

(positive)

This a

fantastic

movie

of

three

prisoners who

become

famous...

Method

I

(

X

;

Y

)

≈

2

.

1

I

(

X

;

Y

)

≈

0

.

9

MINE

1.83

±

0.04

0.71

±

0.05

1.64

±

0.09

0.70

±

0.06

MRE

1.72

±

0.07

1.23

±

0.02

N

-MIENF

0.91

±

0.05

0.43

±

0.03

VCE

2.01

±

0.04

0.83

±

0.01

Method

I

(

X

;

Y

)

≈

1

.

5

I

(

X

;

Y

)

≈

0

.

2

MINE

1.42

±

0.04

0.18

±

0.02

1.41

±

0.03

0.19

±

0.04

MRE

1.23

±

0.09

0.31

±

0.09

N

-MIENF

0.73

±

0.03

0.11

±

0.02

VCE

1.22

±

0.02

0.19

±

0.02

(b) BERT

7 Conclusion

In this work, we introduced a new mutual information (MI) estimator grounded in recent vector copula theory. A fundamental difference to existing approaches is the explicit disentanglement of marginal distributions and dependence structure in our method. This separation enables more flexible and fine-grained modeling, avoiding the pitfalls of both overly simplistic or excessively complex approaches, and reducing overall learning difficulty via strategic factorization of the original estimation problem. Extensive experiments demonstrate our method's effectiveness and robustness.

Beyond the development of practical estimator, our research also offers fresh perspectives on MI estimation. By viewing PMI as a density rather than a density ratio, we open new avenues for modeling. Additionally, our approach to vector rank computation generalizes the classical copula transformation and holds promise as a versatile preprocessing step for a broad range of MI estimators. Finally, by reinterpreting existing estimators through the lens of vector copula theory, we obtain new insights into the parameterization and learning of different estimators and the underlying trade-offs.

Copulas have been widely used for MI estimate [63, 55, 64, 56, 65, 33, 51, 45, 66]. Existing methods primarily focus on classic copulas , where the copula transformation is applied independently to each univariate marginal to better account for the marginal-invariant property of MI. This strategy has been shown to improve accuracy and reduce variance [56, 65]. We go one step further by using vector copulas , where the transformation jointly considers all dimensions of the multivariate marginals. This can be seen as a generalization of classic copula transformation, where we not only consider MI's invariance to element-wise bijections but also to any diffeomorphisms. Another key difference lies in that these works still treat PMI as a density ratio, whereas our work treats PMI as a density.

We note that, while powerful, our estimator is not a panacea. One limitation of our method is that it relies on the two marginal distributions to be reasonably modeled. While marginal distributions are far easier to learn than the joint distribution, they can still be challenging to learn for high-dimensional data e.g., images. Fortunately, dimensionality reduction techniques [3, 13] help to mitigate this issue. Another limitation lies in the flexibility of our model-based parameterization of vector copula, which can be less flexible than neural network methods. However, as our method strikes a good trade-off between complexity and capacity across diverse cases, we consider it as a highly competitive method.

Acknowledgments

AW acknowledges supports from a Turing AI Fellowship under grant EP/V025279/1, the Alan Turing Institute, and the Leverhulme Trust via the Leverhulme Centre for the Future of Intelligence. MUG was supported in part by Gen AI - the AI Hub for Generative Models, funded by EPSRC (EP/Y028805/1). YC acknowledge supports from the Cambridge Trust and the Qualcomm Innovation Fellowship. ZO acknowledges supports from Lee Family Scholarship via Imperial College London.

References

Justin B Kinney and Gurinder S Atwal. Equitability, mutual information, and the maximal information coefficient. Proceedings of the National Academy of Sciences , 111(9):3354-3359, 2014.

Tm Cover, Ja Thomas, and J Wiley. Elements of information theory. Tsinghua University Pres, 2003.

Gokul Gowri, Xiao-Kang Lun, Allon M Klein, and Peng Yin. Approximating mutual information of high-dimensional variables using learned representations. arXiv preprint arXiv:2409.02732 , 2024.

Antonio Reverter and Eva KF Chan. Combining partial correlation and an information theory approach to the reversed engineering of gene co-expression networks. Bioinformatics , 24(21):2491-2497, 2008.

Lin Song, Peter Langfelder, and Steve Horvath. Comparison of co-expression measures: mutual information, correlation, and model based indices. BMC bioinformatics , 13:1-21, 2012.

Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on pattern analysis and machine intelligence , 27(8):1226-1238, 2005.

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2018.

Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations , 2019.

Mike Wu, Chengxu Zhuang, Milan Mosse, Daniel Yamins, and Noah Goodman. On mutual information in contrastive learning for visual representations. arXiv preprint arXiv:2005.13149 , 2020.

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems , 29, 2016.

Yanzhi Chen, Michael U Gutmann, and Adrian Weller. Is learning summary statistics necessary for likelihood-free inference? In International Conference on Machine Learning , pages 45294544. PMLR, 2023.

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 , 2016.

Yanzhi Chen, Dinghuai Zhang, Michael Gutmann, Aaron Courville, and Zhanxing Zhu. Neural approximate sufficient statistics for implicit models. arXiv preprint arXiv:2010.10079 , 2020.

Zexuan Qiu, Qinliang Su, Zijing Ou, Jianxing Yu, and Changyou Chen. Unsupervised hashing with contrastive information bottleneck. arXiv preprint arXiv:2105.06138 , 2021.

Yanzhi Chen, Yingzhen Li, Adrian Weller, et al. Scalable infomin learning. Advances in Neural Information Processing Systems , 35:2226-2239, 2022.

Sayedeh Leila Noorbakhsh, Binghui Zhang, Yuan Hong, and Binghui Wang. { Inf2Guard } : An { Information-Theoretic } framework for learning { Privacy-Preserving } representations against inference attacks. In 33rd USENIX Security Symposium (USENIX Security 24) , pages 24052422, 2024.

Bao Duong and Thin Nguyen. Diffeomorphic information neural estimation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 7468-7475, 2023.

Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning , pages 1779-1788. PMLR, 2020.

Giulio Franzese, Mustapha BOUNOUA, and Pietro Michiardi. Minde: Mutual information neural diffusion estimation. In The Twelfth International Conference on Learning Representations , 2023.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning , pages 531-540. PMLR, 2018.

Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints , pages arXiv-1807, 2018.

Qing Guo, Junya Chen, Dong Wang, Yuewei Yang, Xinwei Deng, Jing Huang, Larry Carin, Fan Li, and Chenyang Tao. Tight mutual information estimation with contrastive fenchel-legendre optimization. Advances in Neural Information Processing Systems , 35:28319-28334, 2022.

Nunzio Alexandro Letizia, Nicola Novello, and Andrea M Tonello. Mutual information estimation via f -divergence and data derangements. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks , 2(5):359-366, 1989.

Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks , 94:103-114, 2017.

Liam Paninski. Estimation of entropy and mutual information. Neural computation , 15(6):11911253, 2003.

Yihong Wu and Pengkun Yang. Minimax rates of entropy estimation on large alphabets via best polynomial approximation. IEEE Transactions on Information Theory , 62(6):3702-3720, 2016.

Andrew R. Barron and Kai Guo. Convergence of smoothed empirical measures with applications to entropy estimation. IEEE Transactions on Information Theory , 68(8):5326-5353, 2022.

David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics , pages 875-884. PMLR, 2020.

Yanqin Fan and Marc Henry. Vector copulas. Journal of Econometrics , 234(1):128-150, 2023.

M Sklar. Fonctions de répartition à n dimensions et leurs marges. In Annales de l'ISUP , volume 8, pages 229-231, 1959.

Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations , 2019.

Shashank Singh and Barnabás Póczos. Nonparanormal information estimation. In International Conference on Machine Learning , pages 3210-3219. PMLR, 2017.

Yanzhi Chen and Michael U Gutmann. Adaptive gaussian copula abc. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1584-1592. PMLR, 2019.

George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems , pages 2338-2347, 2017.

Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. arXiv preprint arXiv:1906.04032 , 2019.

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research , 22(57):1-64, 2021.

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations , 2022.

Ruizhi Liao, Daniel Moyer, Polina Golland, and William M Wells. Demi: Discriminative estimator of mutual information. arXiv preprint arXiv:2010.01766 , 2020.

Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems , 33:4905-4916, 2020.

Akash Srivastava, Seungwook Han, Kai Xu, Benjamin Rhodes, and Michael U Gutmann. Estimating the density ratio between distributions with high discrepancy using multinomial logistic regression. Transactions on Machine Learning Research , 2023(3):1-23, 2023.

Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on pure and applied mathematics , 36(2):183-212, 1983.

Victor Chernozhukov, Alfred Galichon, Marc Hallin, and Marc Henry. Monge-kantorovich depth, quantiles, ranks and signs. Annals of Statistics , 45(1):223-256, 2017.

Manuel Davy and Arnaud Doucet. Copulas: a new insight into positive time-frequency distributions. IEEE signal processing letters , 10(7):215-218, 2003.

Jian Ma and Zengqi Sun. Mutual information is copula entropy. Tsinghua Science & Technology , 16(1):51-54, 2011.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning , pages 531-540. PMLR, 2018.

Zhengyang Hu, Song Kang, Qunsong Zeng, Kaibin Huang, and Yanchao Yang. Infonet: Neural estimation of mutual information without test-time optimization. In Forty-first International Conference on Machine Learning .

Ivan Butakov, Aleksandr Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, and Alexey Frolov. Mutual information estimation via normalizing flows. Advances in Neural Information Processing Systems , 37:3027-3057, 2024.

Paweł Czy˙ z, Frederic Grabowski, Julia E Vogt, Niko Beerenwinkel, and Alexander Marx. The mixtures and the neural critics: On the pointwise mutual information profiles of fine distributions. arXiv preprint arXiv:2310.10240 , 2023.

Arjun Ashok, Étienne Marcotte, Valentina Zantedeschi, Nicolas Chapados, and Alexandre Drouin. Tactis-2: Better, faster, simpler attentional copulas for multivariate time series. arXiv preprint arXiv:2310.01327 , 2023.

David Huk, Mark Steel, and Ritabrata Dutta. Your copula is a classifier in disguise: classificationbased copula density estimation. In International Conference on Artificial Intelligence and Statistics , pages 3790-3798. PMLR, 2025.

Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 297-304. JMLR Workshop and Conference Proceedings, 2010.

Sudipto Mukherjee, Himanshu Asnani, and Sreeram Kannan. Ccmi: Classifier based conditional mutual information estimation. In Uncertainty in artificial intelligence , pages 1083-1093. PMLR, 2020.

Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning , pages 5171-5180. PMLR, 2019.

Houman Safaai, Arno Onken, Christopher D Harvey, and Stefano Panzeri. Information estimation using nonparametric copulas. Physical Review E , 98(5):053302, 2018.

Xianli Zeng, Yingcun Xia, and Howell Tong. Jackknife approach to the estimation of mutual information. Proceedings of the National Academy of Sciences , 115(40):9956-9961, 2018.

Soumik Purkayastha and Peter X-K Song. fastmi: A fast and consistent copula-based nonparametric estimator of mutual information. Journal of Multivariate Analysis , 201:105270, 2024.

Paweł Czy˙ z, Frederic Grabowski, Julia E Vogt, Niko Beerenwinkel, and Alexander Marx. Beyond normal: On the evaluation of mutual information estimators. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, Alexey Frolov, and Kirill Andreev. Information bottleneck analysis of deep neural networks via lossy compression. In The Twelfth International Conference on Learning Representations , 2024.

Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.

Yves-Laurent Kom Samo. Inductive mutual information estimation: A convex maximumentropy copula approach. In International Conference on Artificial Intelligence and Statistics , pages 2242-2250. PMLR, 2021.

Amor Keziou and Philippe Regnault. Semiparametric estimation of mutual information and related criteria: Optimal test of independence. IEEE Transactions on Information Theory , 63(1):57-71, 2016.

Nunzio A Letizia, Nicola Novello, and Andrea M Tonello. Copula density neural estimation. IEEE Transactions on Neural Networks and Learning Systems , 2025.

Gery Geenens. Towards a universal representation of statistical dependence. arXiv preprint arXiv:2302.08151 , 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] ,

Justification: The abstract and introduction directly reflects the exact contribution of the work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] ,

Justification: Limitation is explicitly discussed in conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] ,

Justification: All proofs have a clear declaration of assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] ,

Justification: Code is available in the provided repo, and the details of experiments and implementations are also documented in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] ,

Justification: source code is provided through the github link provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] ,

Justification: all experimental details are provided in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars (which correspond to the standard errors) are reported, which are collected from 8 independent runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] ,

Justification: please see the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] ,

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes] ,

Justification: the work is a general method and its societal impacts have been discussed in introduction and conclusion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: all code and data are taken from public available resource without violating any license and terms.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: no new assets introduced at this stage.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLM is only used for proofreading.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper utilizes the concept of vector copula introduced in [27] to develop a generative MI estimator (VCE). The authors discussed important properties of VCE such as consistency and convergence. They also commented on desirable features and interesting concepts such as separate learning/modeling and complexity-capacity trade-off. Moreover, given the interpretation of the PMI as a density rather than a density ratio, the authors studied neural-network free methodologies (e.g., MLE) to model the vector copula. The VCE estimator is compared with several established estimators (both discriminative and generative). Experiments show promising results for low-dimensional data and for low values of MI.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is technically sound and the theoretical analysis includes consistency and convergence/error proofs for the VCE estimator. The authors provide various experimental setups demonstrating the applicability of the method across different scenarios. The work is well-organized with clear sections and an accurate literature review, including a clear explanation of the vector copula concept and methodology. \n\nThe main contribution of the paper is the formalization of a novel MI estimator based on the concept of vector copula, with the validation of desirable properties. Another interesting contribution lies in interpreting the PMI as density itself rather than density ratio (which follows [27]), which could advance the field and shed light on how to treat the PMI. The authors are honest about limitations, which are clearly stated.\n\nQUESTIONS:\nOverall, I found this paper interesting and well-motivated, and I would like the authors to address the following questions to help clarify the contributions and potential impact:\n\n1. **Comparison with SOTA Methods.** Authors selected MRE [35] and MINDE [16] as SOTA estimators for discriminative and generative approaches, respectively. This is definitely arguable, at least for the discriminative approach as also SMILE [29] and $f$-DIME [36] are robust estimators. Moreover, [29] and [50] introduced common benchmarks and evaluation techniques which in this paper are not fully considered. Could the authors include richer comparisons with these methods? This would provide a more comprehensive evaluation and help establish where VCE stands relative to current best practices. Additionally, could the authors provide clear recommendations on when VCE should be preferred over other methods?\n\n2. **Bias-Variance Analysis.** The paper lacks discussion about bias-variance trade-offs, which is crucial for MI estimators. Could the authors provide a theoretical analysis or experimental evaluation of how the VCE estimator's bias and variance scale with sample size? This would help clarify the estimator's statistical properties and practical applicability. \n\n3. **Neural Network Parameterization.** The paper focuses on neural-network free methodologies for modeling the vector copula density. Given that the error depends on the KL divergence between $c$ and $c'$, could the authors provide at least one concrete example comparing neural network parameterization vs. classical methods for the vector copula density estimation? This would help understand the trade-offs and potential benefits of different modeling approaches, and could reveal whether the neural-network free approach is indeed still a valid path even for high dimensionality.\n\n4. **High MI and High Dimensionality Scenarios.** The experiments focus on low-dimensional data and low MI values, but the most challenging problems in MI estimation occur with high-dimensional data and high MI values. Could the authors extend their experiments to include high-dimensional scenarios (e.g., $d > 50$) and high MI values? This would demonstrate the method can be used for real-world applications and could significantly increase the significance of the paper if the VCE estimator shows advantages in these challenging regimes.\n\n5. **Comparison with Related Copula-based Methods.** The papers [R1, R2] briefly discuss the advantages of modeling the density ratio (not seen as a density) via classic copulas for the MI estimation task. I think it might be worthy add a comparison (at least written) with these papers. Similarly, can the authors elaborate more (perhaps add some sentences if not in the main also in the appendix about the difference between this paper and the paper in [40] as they both study the MI as negative differential entropy, but the former with the concept of vector copula?) I think it might help the reader better understand the novelty of this paper.\n\n6. **Double-Step Approach Validation.** The paper proposes a two-step approach: flow matching to obtain vector ranks followed by classical parametric methods (the authors mentioned about the possibility to train neural networks as well) to estimate the vector copula density. However, marginal distribution learning via flow matching generative models could not reach convergence and classical parametric methods are known to suffer from various limitations (e.g., model misspecification, curse of dimensionality). Could the authors provide a more thorough analysis of how this double-step approach actually improves upon existing single-step methods? Specifically, could they demonstrate quantitatively that the benefits of the conceptual separation outweigh the potential errors introduced by the two-step process? This would help validate whether the proposed approach genuinely advances the state-of-the-art or simply trades one set of problems for another.\n\n[R1] Samo, Yves-Laurent Kom. \"MIND: Inductive Mutual Information Estimation, A Convex Maximum-Entropy Copula Approach.\" arXiv preprint arXiv:2102.13182 (2021).\n\n[R2] Letizia, Nunzio A., and Andrea M. Tonello. \"Copula density neural estimation.\" arXiv preprint arXiv:2211.15353 (2022).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-21"}

comments: SUMMARY:
The paper  proposes an copula based approach (and algorithm) to compute  mutual information.
Experimental results show that the proposed approach can outperform alternative algorithms.

STRENGTHS AND WEAKNESSES:
**strengths:**

- Extensive comparison with existing algorithms.
- demonstrate good accuracy 
- code provided


**weaknesses:**
- Figures too small (Fig.1/2) 
- Appendices missing (?)
- Table 2 not needed (?)

QUESTIONS:
none

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that MI can be recast as a vector copula estimation task, thereby generalising the previous results where the link between copulas and MI was established for univariate variables. A vector copula method is proposed to estimate MI in two steps, first by fitting separate flow models on each set of variables X and Y, and secondly, learning the dependence between X and Y with a mixture of Gaussian vector copulas. Theoretical arguments motivate the approach, and experiments show its competitiveness compared to existing methods for MI estimation.\n\nSTRENGTHS AND WEAKNESSES:\nI believe the paper offers a promising way forward in MI estimation, differentiating itself from the more common density ratio approaches that could lead to unnormalized estimates. The separation of learning vector marginals first and the vector copula second allows great control over the method’s complexity. Theoretical results are generally good (except Prop 4 which I have doubts about) and provide good arguments for the adoption of this method. Experiments are also convincing. \n\n---\n\n**Strengths**\n- The paper’s main novelty in Theorem 2 offers a theoretically grounded way of estimating MI, which possibly provides a sound direction for future progress in this field. \n- Their proposed estimator offers some good flexibility in terms of selecting an appropriate model capacity for the perceived complexity of a task, as well as targeting whether to add complexity to the vector marginals and/or the vector copula.\n- I found Proposition 3 to be very good motivation for the proposed modelling approach, exemplifying the capacity-complexity trade-off.\n\n---\n\nWhile the first part of the paper, up to L126 had me convinced by providing a sound motivation, I am very uncertain about Section 3.1, which holds me back from fully believing in the proposed approach. I also have some comments on some aspects of training feasibility, relations to previous work, the vector copula mixture, and the meaning of Proposition 4.\n\n**Weaknesses**\n\n- I am particularly sceptical about the obtained vector ranks as this is crucial for your theory to hold, and for your estimates to be meaningful:\n\nTo use a vector copula, you need vector ranks. To obtain those, you choose flow models. While it certainly is easier to estimate two separate flows in $d_X$ and $d_Y$ dimensions compared to one big flow in $d_X+D_Y$, this is still not at all obvious. L133-134 is crucial to your method, yet it is not emphasised enough and checked for in my opinion. \n- I am not sure joint learning of vector copulas is a well-posed problem:\n\nAt the bottom of page 3, and in the Appendix L251, you suggest learning vector ranks with the vector copula jointly. But is this well defined? I believe [TACTiS-2: Better, Faster, Simpler Attentional Copulas for Multivariate Time Series, ICLR 2024] showed that joint optimisation is ill-defined for regular copulas – would the same not be true here? Your experiments in the Appendix seem to suggest so…\n\n- It is not clear to me how the vector copula mixture obtains meaningfully different mixture components $c_k$:\n\nAs you optimise the MLE on observed vector ranks, how do you ensure that each mixture component will not obtain the same representation? There are no real details about the implementation you use that I could spot. \n\n- Two relevant previous works should be discussed in relevant sections of the text:\n\nIn Algorithm 2, your approach of using a DRE to estimate the ratio between a simple vector copula and the true vector copula has previously been used in the context of copulas in [Your copula is a classifier in disguise: classification-based copula density estimation, AISTATS 2025], in their Section 3, and I believe this should be acknowledged. Furthermore, in your Section 5, this same paper discusses many similar points to L207-220 in their Section 6. A second reference that I would like to see discussed is [Towards a universal representation of statistical dependence, by Gery  Geenens 2023], where the relationship between copulas and MI is discussed, notably in their Section 6 showing MI is a measure on non-independence while copulas fully capture dependence.\n\n- Proposition 4 I believe is a little misleading:\n\nIn your proof in the Appendix, before concluding in L150, you end up with a representation of an independent Gaussian in $d_Y+d_X$ dimensions. While this is a Gaussian, it is fully independent. How is your statement different from saying that an independent copula is a Gaussian copula because you can set the correlation matrix $\\Sigma$ to be diagonal while interpreting your data generating process to be with Gaussian variables? All this is saying is that a Gaussian vector copula can also capture the independence vector copula, at which point I fail to see the relevance of this statement, or am I missing something? \n\n---\n\n\n**Quality**: The motivation for the method is well articulated and supported with appropriate theoretical results, and experiments are rather exhausting covering multiple setups. However I am not sure about practical details of the proposed method, which I believe have to be addressed more.\n\n**Clarity**: The submission is overall clearly written and well organised with the only details missing being about the copula mixture and missing discussions of related work.\n\n**Significance**: I believe this paper presents good arguments for a possible shift in focus within the MI community, offering a new objective, providing links within the literature, and reinterpreting existing models for new insights. It also provides motivation to develop new vector copula models with MI constituting a useful application for them. \n\n**Originality**: New insights and properties of existing work are discussed. But some related work that I believe should be discussed is omitted. The introduction of vector copulas for MI, while not fully novel from the copula perspective, is motivated well and backed up by theoretical guarantees.\n\nQUESTIONS:\nOverall, the paper is persuasive about the usefulness of vector copulas for MI. However, I am unsure about the practical implementation proposed in this paper. Before I can confidently advocate the acceptance of this paper, **I would like to see the first four weaknesses addressed**. Furthermore, critical questions I would kindly ask to be answered include:\n\n- Could you provide, at least in the appendix, some evidence of the obtained vector ranks being $\\mathcal{N}(\\mathbf{0},\\mathbf{I})$ for your experiments? Maybe a test for Gaussianity/independence, or showing the univariate ranks of the representations, or the covariance matrices, or metrics derived from these?\n\n- Are there other ways for obtaining vector ranks that you have considered? For instance, in the 2023 Vector Copulas paper, the authors consider an optimal transport problem solved through linear programming. How would these compare, computationally and results-wise or are they not appropriate to be used here?\n\n---\n\nBelow are minor points and typos that I believe should be corrected before this paper can be ready for acceptance, as well as a few questions I am curious about but that I do not need to be answered if time is too short:\n\n**Typos and notation**\n\n- L24 Do you mean 'suffer from requiring an exponentially large...'?\n- Definition 1, there is a missing dot to the statement?\n- Notation in L98-99, I find it more common to use $\\Phi$ for the CDF and reserve $\\phi$ for the PDF. A possible alternative to notation could be to use $\\Phi_d$ to distinguish the element-wise CDF from the joint?\n- L101 $\\epsilon_{d_X} $ should be $\\epsilon_{\\leq d_X} $ ?\n- L149 $c$ *as* a mixture?\n- L152 stricture -> structure\n- L158, of *a* copula?\n- Appendix L55, should say by Lemma 2 instead?\n- Appendix L89, these combined *results* lead to?\n- Appendix, define $\\Sigma$ in Lemma 3.\n- Throughout your proofs in the Appendix, at APDX L133-134, and more generally, to be more respectful to readers, I would replace 'it is clear' by the appropriate reason. Eg ' we identify this as a Gaussian density', 'By definition, it follows that...'.\n- Figure 1 and 2 could use bigger fonts for the axis and titles, same with the appendix.\n- L266 known *to be* challenging?\n- L297, please write what $d_L$ is equal to in your experiment.\n\n**Curiosity**\n\n- L194, Does a second-order Taylor expansion imply second-order dependencies? For instance, in classical copulas, Gaussians satisfy the simplifying assumption of vines - would this be the same for the vector Gaussian copula? Then, does this mean your V-Gaussian mixture is simplified?\n- L235, Do you test for or show in one of your examples? Time permitting, it would perhaps be nice to see.\n- L286-287, Could this also be because the neural estimators do not need to be normalised and might overestimate the true MI?\n\nLIMITATIONS:\nLimitations are generally addressed. However, I would appreciate more emphasis on the fact that vector ranks need to be standard Gaussians in the proposed approach, as it is critical for the theory and method to work. I believe things that would ameliorate this include putting more emphasis on this condition being met when relevant, and providing or suggesting some diagnostics or tests to check for this assumption being respected.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
The paper describes an approach to estimating mutual information that
bridges the gap between neural models and simpler models, e.g.,
Gaussian copula.  For this, they propose to use Vector Copulas.

Experiments show some advantages of the method.

STRENGTHS AND WEAKNESSES:
MI estimation is an important area, e.g, for optimizing encoders
in representation learning.

The theoretical development is mostly clear.

It would help the reader to define the push-forward notation (the symbol #)

The equivalence of PMI, usually stated as a density ratio,
and the vector copula density seems very interesting, and as the authors show,
it can be used to re-interpret important prior works.

The remainder of the paper demonstrates ways of applying the
resulting ideas to some MI estimation problems, including
modeling the copula as a mixture of copula.

The results in table 1 show some mild improvements in MI estimation
in comparison to other methods, though MRE is competitive.

The results in Figs 1 and 2 are very hard to read, so performance is
hard to evaluate.

I feel that the theoretical contributions outweigh the somewhat mild
experimental result improvements.

QUESTIONS:
I would be helpful for the reader if the main text of the paper pointed to the appendix
for the proof of Thm 2  :)

Stronger experimental results would strengthen the work.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03