InfoBridge: Mutual Information estimation via Bridge Matching

## Abstract

Abstract Diffusion bridge models have recently become a powerful tool in the field of generative modeling.
In this work, we leverage
them
to address another important problem in machine learning and information theory, the estimation of the mutual information (MI) between two random variables.
Neatly framing MI estimation as a domain transfer problem, we construct an unbiased estimator for data posing difficulties for conventional MI estimators.
We showcase the performance of our estimator on three standard MI estimation benchmarks, i.e., low-dimensional, image-based and high MI, and on real-world data, i.e., protein language model embeddings.

## 1 Introduction

Information theory offers an extensive set of tools for quantifying probabilistic relations between random variables.
It is widely used in machine learning for advanced statistical analysis (Berrett & Samworth, 2017; Sen et al., 2017; Duong & Nguyen, 2023b; Bounoua et al., 2024a),
assessment of deep neural networks’ performance and generalization capabilities (Tishby & Zaslavsky, 2015; Xu & Raginsky, 2017; Goldfeld et al., 2019; Steinke & Zakynthinou, 2020; Amjad et al., 2022; Butakov et al., 2024b),
self-supervised and semi-supervised learning (Linsker, 1988; Bell & Sejnowski, 1995; Hjelm et al., 2019; Stratos, 2019; Bachman et al., 2019; Veličković et al., 2019; van den Oord et al., 2019; Tschannen et al., 2020) and regularization or alignment in generative modeling (Chen et al., 2016; Belghazi et al., 2018; Ardizzone et al., 2020; Wang et al., 2024).

The majority of the aforementioned applications revolve around one of the central information-theoretic quantities – *mutual information* (MI).
Due to several outstanding properties, MI is widely used as an invariant measure of non-linear dependence between random variables.
Unfortunately, recent studies suggest that the curse of dimensionality is highly pronounced when estimating MI (Goldfeld et al., 2020; McAllester & Stratos, 2020).
Additionally, it is argued that long tails, high values of MI and some other particular features of complex probability distributions can make mutual information estimation even more challenging (Czyż et al., 2023).
On the other hand, recent developments in neural estimation methods demonstrate that sophisticated parametric estimators can achieve notable practical success in situations where traditional mutual information estimation techniques struggle (Belghazi et al., 2018; van den Oord et al., 2019; Song & Ermon, 2020; Rhodes et al., 2020; Ao & Li, 2022; Butakov et al., 2024a).
Among neural estimators, generative approaches are of particular interest, as they have proven to be effective in handling complex data (Duong & Nguyen, 2023a; Franzese et al., 2024; Butakov et al., 2024a).
Since MI estimation is closely tied to approximation of a joint probability distribution,
one can argue that leveraging state-of-the-art generative models, e.g., diffusion models, may result in additional performance gains.

#### Diffusion Bridge Matching.

Diffusion models are a powerful type of generative models that show an impressive quality of image generation (Ho et al., 2020; Rombach et al., 2022). However, they have some disadvantages, such as the inability to perform data-to-data translation via diffusion. To tackle this problem, a novel promising approach based on Reciprocal Processes (Léonard et al., 2014) and Schrödinger Bridges theory (Schrödinger, 1932; Léonard, 2013) have emerged. This approach is called the diffusion bridge matching and is used for learning generative models as diffusion processes for data-to-data translation. This type of models has shown itself as a powerful approach for numerous applications in biology (Tong et al., 2024; Bunne et al., 2023), chemistry (Somnath et al., 2023; Igashov et al., 2024), computer vision (Liu et al., 2023; Shi et al., 2023; Zhou et al., 2024), speech processing (Chen et al., 2023) and unpaired learning (Gushchin et al., 2024b; a; Shi et al., 2023).

#### Contributions.

In this work, we employ the Diffusion Bridge Matching for the MI estimation.

- 1.
Theory. We propose an unbiased mutual information estimator based on reciprocal processes, their diffusion representations and the Girsanov theorem (\wasyparagraph[4.1](https://arxiv.org/html/2502.01383v3#S4.SS1)).
- 2.
Practice. Building on the proposed theoretical framework and the powerful generative methodology of diffusion bridges, we develop a practical algorithm for MI estimation, named InfoBridge (\wasyparagraph[4.2](https://arxiv.org/html/2502.01383v3#S4.SS2)). We demonstrate that our method achieves performance comparable to existing approaches on low-dimensional benchmarks and superior performance on image data benchmarks, protein embeddings data benchmark and high MI benchmark (\wasyparagraph[5](https://arxiv.org/html/2502.01383v3#S5)).

#### Notations.

We work in $\mathbb{R}^{D}$, which is the $D$-dimensional Euclidean space equipped with the Euclidean norm $\|\cdot\|$. We use $\mathcal{P}(\mathbb{R}^{D})$ to denote the absolutely continuous Borel probability distributions whose variance and differential entropy are finite. To denote the density of $q\in\mathcal{P}(\mathbb{R}^{D})$ at a point $x\in\mathbb{R}^{D}$, we use $q(x)$.
We write $\mathrm{KL}\left(\cdot\middle\|\cdot\right)$ to denote the Kullback-Leibler divergence between two distributions.
We use $\Omega$ to denote the space of trajectories, i.e., continuous $\mathbb{R}^{D}$-valued functions of $t\in[0,1]$. We write ${\mathcal{P}(\Omega)}$ to denote the probability distributions on the trajectories $\Omega$ whose marginals at $t=0$ and $t=1$ belong to $\mathcal{P}(\mathbb{R}^{D})$; this is the set of stochastic processes. We use $dW_{t}$ to denote the differential of the standard Wiener process $W\in\mathcal{P}(\Omega)$. We use $Q_{|x_{0}}$ and $Q_{|x_{0},x_{1}}$ to denote the distribution of stochastic process $Q$ conditioned on $Q$’s values $x_{0}$ and $x_{0},x_{1}$ at times $t=0$ and $t=0,1$, respectively.
For a process $Q\in\mathcal{P}(\Omega)$, we denote its marginal distribution at time $t$ by $q(x_{t})\in\mathcal{P}(\mathbb{R}^{D})$, and if the process is conditioned on its value $x_{s}$ at time $s$,
we denote the marginal distribution of such a process at time $t$ by $q(x_{t}|x_{s})\in\mathcal{P}(\mathbb{R}^{D})$.

## 2 Background

#### Mutual information.

Information theory is a well-established framework for analyzing and quantifying interactions between random vectors.
In this framework, mutual information (MI) serves as a fundamental and invariant measure of the non-linear dependence between two $\mathbb{R}^{D}$-valued random vectors $X_{0},X_{1}$.
It is defined as follows:

$$ $I(X_{0};X_{1})\stackrel{{\scriptstyle\text{def}}}{{=}}\mathrm{KL}\left(\Pi_{X_{0},X_{1}}\middle\|\Pi_{X_{0}}\otimes\Pi_{X_{1}}\right),$ (1) $$

where $\Pi_{X_{0},X_{1}}$ and $\Pi_{X_{0}}$, $\Pi_{X_{1}}$ are the joint and marginal distributions of a pair of random vectors $(X_{0},X_{1})$.
If the corresponding PDF $\pi(x_{0},x_{1})$ exists, the following also holds:

$$ $I(X_{0};X_{1})=\operatorname{\mathbb{E}}_{\pi(x_{0},x_{1})}\log\frac{\pi(x_{0},x_{1})}{\pi(x_{0})\pi(x_{1})}.$ (2) $$

Mutual information is symmetric, non-negative and equals zero if and only if $X_{0}$ and $X_{1}$ are independent.
MI is also invariant to bijective mappings: $I(X_{0};X_{1})=I(g(X_{0});X_{1})$ if $g^{-1}$ exists and $g$, $g^{-1}$ are measurable (Cover & Thomas, 2006; Polyanskiy & Wu, 2024).

Brownian Bridge. Let $W^{\epsilon}$ be the Wiener process with a constant volatility $\epsilon>0$, i.e., it is described by the SDE $dW^{\epsilon}=\sqrt{\epsilon}dW_{t}$, where $W_{t}$ is the standard Wiener process. Let $W^{\epsilon}_{|x_{0},x_{1}}$ denote the process $W^{\epsilon}$ conditioned its on values $x_{0},x_{1}$ at times $t=0,1$, respectively. This process $W^{\epsilon}_{|x_{0},x_{1}}$ is called the Brownian Bridge (Ibe, 2013, Chapter 9).

#### Reciprocal processes.

Reciprocal processes are a class of stochastic processes that have recently gained attention of research community in the contexts of stochastic optimal control (Léonard et al., 2014), Schrödinger Bridges (Schrödinger, 1932; Léonard, 2013) and diffusion generative modeling (Liu et al., 2023; Gushchin et al., 2024a). In our paper, we consider a particular case of reciprocal processes which are induced by the Brownian Bridge $W_{|x_{0},x_{1}}^{\epsilon}$.

Consider a joint distribution $\pi(x_{0},x_{1})\in\mathcal{P}(\mathbb{R}^{D\times 2})$ and define the process $Q_{\pi}\in\mathcal{P}(\Omega)$ as a mixture of Brownian bridges $W_{|x_{0},x_{1}}^{\epsilon}$ with weights $\pi(x_{0},x_{1})$:

$$ $\displaystyle Q_{\pi}\stackrel{{\scriptstyle\text{def}}}{{=}}\int W^{\epsilon}_{|x_{0},x_{1}}d\pi(x_{0},x_{1}).$ $$

This implies that to get trajectories of $Q_{\pi}$ one has to first sample the start and end points, $x_{0}$ and $x_{1}$, at times $t=0$ and $t=1$ from $\pi(x_{0},x_{1})$ and then simulate the Brownian Bridge $W^{\epsilon}_{|x_{0},x_{1}}$. Due to the non-causal nature of trajectory formation, such a process is, in general, non markovian. The set of all mixtures of Brownian Bridges can be described as:

$$ $\displaystyle\big\{Q\in\mathcal{P}(\Omega)\text{ }s.t.\text{ }\exists\pi\in\mathcal{P}(\mathbb{R}^{D\times 2}):Q=Q_{\pi}\big\}$ $$

and is called the set of reciprocal processes (for $W^{\epsilon}$).

#### Reciprocal processes conditioned on the point.

Consider a reciprocal process $Q_{\pi}$ conditioned on some start point $x_{0}$. Let the resulting process be denoted as $Q_{\pi|x_{0}}$, which remains reciprocal. Then, if some regularity assumptions are met (Shi et al., 2023, Appx C.1) process $Q_{\pi|x_{0}}$ is known as the Schrödinger Föllmer process (Vargas et al., 2023). While $Q_{\pi}$, in general, not markovian, $Q_{\pi|x_{0}}$ is markovian. Furthermore, it is a diffusion process governed by the following SDE:

$$ $\displaystyle Q_{\pi|x_{0}}:dx_{t}=v_{x_{0}}(x_{t},t)dt+\sqrt{\epsilon}dW_{t},x_{0}\sim\delta(x_{0}),$ $\displaystyle v_{x_{0}}(x_{t},t)=\mathbb{E}_{q_{\pi}(x_{1}|x_{t},x_{0})}\left[\frac{x_{1}-x_{t}}{1-t}\right].$ (3) $$

Representations of reciprocal processes. The process $Q_{\pi}$ can be naturally represented as a mixture of processes $Q_{\pi|x_{0}}$ conditioned on their starting points $x_{0}$:

$$ $Q_{\pi}=\int Q_{\pi|x_{0}}d\pi(x_{0}).$ $$

Therefore, one may also express $Q_{\pi}$ via an SDE but with non-markovian drift (conditioned on $x_{0}$):

$$ $\displaystyle\hskip-5.69054ptQ_{\pi}:dx_{t}=v(x_{t},t,x_{0})dt+\sqrt{\epsilon}dW_{t},x_{0}\sim\pi(x_{0}),$ $\displaystyle\hskip-11.38109ptv(x_{t},t,x_{0})\!=\!v_{x_{0}}(x_{t},t)\!=\!\mathbb{E}_{q_{\pi}(x_{1}|x_{t},x_{0})}\left[\frac{x_{1}-x_{t}}{1-t}\right].$ (4) $$

#### Conditional Bridge Matching.

Although the drift $v_{x_{0}}(x_{t},t)$ of $Q_{\pi|x_{0}}$ in ([3](https://arxiv.org/html/2502.01383v3#S2.E3)) admits a closed form, it usually cannot be computed or estimated directly due to the unavailability of a way to easily sample from $\pi(x_{1}|x_{t},x_{0})$. However, it can be recovered by solving the following regression problem (Zhou et al., 2024):

$$ $\displaystyle\hskip-8.53581ptv_{x_{0}}\!=\!\operatorname*{arg\,min}_{u}\mathbb{E}_{q_{\pi}(x_{1},x_{t}|x_{0})}\left\|\frac{x_{1}-x_{t}}{1-t}-u(x_{t},t)\right\|^{2},$ (5) $$

which optimizes over drifts $u:\mathbb{R}^{D}\times[0,1]\rightarrow\mathbb{R}^{D}$. The same holds for the $Q_{\pi}$ and its drift $v(x_{t},t,x_{0})$ through the addition of expectation w.r.t. $\pi(x_{0})$:

$$ $\displaystyle\hskip-14.22636ptv=\operatorname*{arg\,min}_{u}\mathbb{E}_{q_{\pi}(x_{1},x_{t}|x_{0})\pi(x_{0})}\left\|\frac{x_{1}-x_{t}}{1-t}-u(x_{t},t,x_{0})\right\|^{2}=\!\operatorname*{arg\,min}_{u}\mathbb{E}_{q_{\pi}(x_{1},x_{t},x_{0})}\left\|\frac{x_{1}-x_{t}}{1-t}-u(x_{t},t,x_{0})\right\|^{2},$ $$

where $u:\mathbb{R}^{D}\times[0,1]\times\mathbb{R}^{D}\rightarrow\mathbb{R}^{D}$. Problem ([2](https://arxiv.org/html/2502.01383v3#S2.Ex6)) is usually solved with standard deep learning techniques. Namely, one parametrizes $u$ with a neural network $v_{\theta}$, and minimizes ([2](https://arxiv.org/html/2502.01383v3#S2.Ex6)) using stochastic gradient descent and samples drawn from $q_{\pi}(x_{0},x_{t},x_{1})$. The latter sampling is easy if one can sample from $\pi(x_{0},x_{1})$. Indeed, $q_{\pi}(x_{0},x_{t},x_{1})=q_{\pi}(x_{t}|x_{0},x_{1})\pi(x_{0},x_{1})$, and one can sample first from $\pi(x_{0},x_{1})$ and then from $q_{\pi}(x_{t}|x_{0},x_{1})$, which is the Brownian Bridge (Korotin et al.,, Eq 14).

Such a procedure of learning drift $v$ with a neural network is popular in generative modeling to solve a problem of sampling from conditional distribution $\pi(x_{1}|x_{0})$ and is frequently applied in the image-to-image transfer (Liu et al., 2023). The procedure of learning drift $v(x_{t},t,x_{0})$  ([2](https://arxiv.org/html/2502.01383v3#S2.Ex6)) is usually called the conditional (or augmented) bridge matching (De Bortoli et al., 2023; Zhou et al., 2024). In addition, such procedure can also be derived through the well-celebrated Doob $h$-transform (De Bortoli et al., 2023; Palmowski & Rolski, 2002) or reversing a diffusion (Zhou et al., 2024).

#### KL divergence between diffusion processes.

Consider two diffusion processes with the same volatility coefficient $\sqrt{\epsilon}$ that start at the same distribution $\pi_{0}$:

$$ $Q^{\alpha}:\;dx_{t}=f^{\alpha}(x_{t},t)\,dt+\sqrt{\epsilon}\,dW_{t},\quad x_{0}\sim\pi_{0}(x_{0}),\qquad\alpha\in\{A,B\}.$ (6) $$

By the application of the disintegration theorem (Léonard, 2014, \wasyparagraph1) and the Girsanov theorem (Øksendal, 2003, \wasyparagraph8.6) one can derive the KL divergence between these diffusions:

$$ $\displaystyle\mathrm{KL}\left(Q^{A}\middle\|Q^{B}\right)=\frac{1}{2\epsilon}\int_{0}^{1}\mathbb{E}_{q^{A}(x_{t})}\big[\|f^{A}(x_{t},t)-f^{B}(x_{t},t)\|^{2}\big]dt,$ (7) $$

where $q^{A}(x_{t})$ is the marginal distribution of $Q^{A}$ at time $t$.

This allows one to estimate the KL divergence between two diffusions with the same volatility coefficient and the same initial distributions, knowing only their drifts and marginal samples $x_{t}\sim q^{A}(x_{t})$.
This fact is widely used in Bridge Matching (Shi et al., 2023; Peluchetti, 2023), Diffusion (Franzese et al., 2024) and Schrödinger Bridge Models (Vargas et al., 2021; Gushchin et al., 2023).

## 3 Related Work

#### Mutual information estimators.

Mutual information estimators fall into two main categories: *non-parametric* and *parametric*.
Parametric estimators are also subdivided into *discriminative* and *generative* (Song & Ermon, 2020; Federici et al., 2023). Beyond this natural categorization, we also distinguish *diffusion-based* approaches to better position our method w.r.t. prior work.

Non-parametric estimators.
Classical MI estimation methods use non-parametric density estimators, such as kernel density estimation (Weglarczyk, 2018; Goldfeld et al., 2019) and $k$-nearest neighbors (Kozachenko & Leonenko, 1987; Kraskov et al., 2004; Berrett et al., 2019). The estimated densities are plugged into ([2](https://arxiv.org/html/2502.01383v3#S2.E2)), and MI is computed via Monte Carlo integration. While appealing in low-dimensional settings, these approaches have been shown to fail on high-dimensional or complex data (Goldfeld et al., 2019, \wasyparagraph5.3; Czyż et al., 2023, \wasyparagraph6.2; Butakov et al., 2024a, Table 1).

Non-diffusion-based generative estimators.
Parametric density models—such as normalizing flows and variational autoencoders—can be used to estimate MI via density estimation (Song & Ermon, 2020; McAllester & Stratos, 2020; Ao & Li, 2022; Duong & Nguyen, 2023a). However, results in (Song & Ermon, 2020, Figures 1,2) show that direct PDF estimation may introduce significant bias. To address this, recent works (Duong & Nguyen, 2023a; Butakov et al., 2024a; Dahlke & Pacheco,) avoid PDF estimation and instead estimate the density ratio in ([2](https://arxiv.org/html/2502.01383v3#S2.E2)), leveraging the invariance property of MI. While these methods perform better on synthetic benchmarks, they may still suffer from inductive biases introduced by simplified closed-form approximations.

Discriminative estimators.
Another class of MI estimators relies on training a classifier to distinguish between samples from $\pi(x_{0},x_{1})$ and the product of marginals $\pi(x_{0})\pi(x_{1})$, as in MINE (Belghazi et al., 2018), InfoNCE (van den Oord et al., 2019), and related methods (Song & Ermon, 2020). These approaches use variational bounds on KL divergence and offer parametric estimators suitable for high-dimensional and complex data. However, they suffer from theoretical limitations, such as high variance in MINE and large batch size requirements in InfoNCE (Song & Ermon, 2020). Recent high-MI and complex distributions based benchmarks further indicate that discriminative methods may underperform compared to generative approaches  (Franzese et al., 2024; Butakov et al., 2024a).

#### Neural Diffusion Estimator for MI (MINDE).

One of the most recent generative methods for MI Estimation is diffusion-based (Song et al., 2021) MINDE (Franzese et al., 2024). To estimate $\mathrm{KL}\left(\pi^{A}\middle\|\pi^{B}\right)$ the authors learn two standard backward diffusion models to generate data from distributions $\pi^{A}$ and $\pi^{B}$, e.g., for $\pi^{A}$:

$$ $\displaystyle Q^{A}:\underbrace{dx_{t}=[-f(x_{t},t)+g(t)^{2}s^{A}(x_{t},t)]dt+g(t)d\hat{W}_{t}}_{\text{backward diffusion}},\quad x_{T}\sim q^{A}_{T}(x_{T}),\vskip-28.45274pt$ (8) $$

where $f$ and $g$ are the drift and volatility coefficients, respectively, of the forward diffusion (Song et al., 2021), $d\hat{W}_{t}$ is the Wiener process when time flows backwards, and $q^{A}_{t}$ is the distribution of the noised data at time $t$ (Franzese et al., 2024, \wasyparagraph2, 3). The similar expressions hold for $\pi^{B}$ and $Q^{B}$. Then, the authors formulate a KL divergence estimator through the difference of diffusion score functions:

$$ $\displaystyle\mathrm{KL}\left(\pi^{A}\middle\|\pi^{B}\right)=\mathrm{KL}\left(Q^{A}\middle\|Q^{B}\right)=\int_{0}^{T}\mathbb{E}_{q^{A}_{t}(x_{t})}\left[\frac{g(t)^{2}}{2}\|s^{A}(x_{t},t)-s^{B}(x_{t},t)\|^{2}\right]dt+\mathrm{KL}\left(q^{A}_{T}\middle\|q^{B}_{T}\right).$ (9) $$

Here, $\mathrm{KL}\left(q^{A}_{T}\middle\|q^{B}_{T}\right)$ is the bias term, which vanishes only when diffusion has infinitely many steps, i.e., $T\rightarrow\infty$. When the diffusion score functions $s^{A}$ and $s^{B}$ ([8](https://arxiv.org/html/2502.01383v3#S3.E8)) are properly learned, one can draw samples from the forward diffusion $q^{A}_{t}(x_{t})=q^{A}_{t}(x_{t}|x_{0})q(x_{0})$ and compute the estimate of KL divergence ([9](https://arxiv.org/html/2502.01383v3#S3.E9)). In this way, the authors transform the problem of training the KL divergence estimator into the problem of learning the backward diffusions ([8](https://arxiv.org/html/2502.01383v3#S3.E8)) that generate data from noise.

To estimate mutual information, the authors propose a total of four equivalent methods, all based on the estimation of up to three KL divergences ([9](https://arxiv.org/html/2502.01383v3#S3.E9)) or their expectations.

## 4 Info Bridge Mutual Information estimator

In (\wasyparagraph[4.1](https://arxiv.org/html/2502.01383v3#S4.SS1)), we propose our novel MI estimator which is based on difference of diffusion drifts of conditional reciprocal processes. We explain the practical learning procedure in (\wasyparagraph[4.2](https://arxiv.org/html/2502.01383v3#S4.SS2)) and suggest some straightforward generalizations of our method in (\wasyparagraph[4.3](https://arxiv.org/html/2502.01383v3#S4.SS3)).

### 4.1 Computing MI through Reciprocal Processes

Consider the problem of MI estimation for random variables $X_{0}$ and $X_{1}$ with joint distribution $\pi(x_{0},x_{1})$. To tackle this problem, we employ reciprocal processes:

$$ $\displaystyle Q_{\pi}\stackrel{{\scriptstyle\text{def}}}{{=}}\int W^{\epsilon}_{|x_{0},x_{1}}d\pi(x_{0},x_{1}),Q^{\text{ind}}_{\pi}\stackrel{{\scriptstyle\text{def}}}{{=}}\int W^{\epsilon}_{|x_{0},x_{1}}d\pi(x_{0})d\pi(x_{1}).$ (10) $$

We show that the KL between the distributions $\pi(x_{0},x_{1})$ and $\pi(x_{0})\pi(x_{1})$ ([1](https://arxiv.org/html/2502.01383v3#S2.E1)) is equal to the KL between the reciprocal processes $Q_{\pi}$ and $Q^{\text{ind}}_{\pi}$, and decompose the latter into the difference of drifts.

###### Theorem 4.1 (Mutual Information decomposition) .

Consider random variables $X_{0},X_{1}$ and their joint distribution $\pi(x_{0},x_{1})$, that satisfies some regularity assumptions, (Shi et al., 2023, Appendix C). Consider reciprocal processes $Q_{\pi}$, $Q_{\pi}^{\text{ind}}$ induced by distributions $\pi(x_{0},x_{1})$ and $\pi(x_{0})\pi(x_{1})$, respectively, as in ([10](https://arxiv.org/html/2502.01383v3#S4.E10)). Then the MI between the random variables $X_{0}$ and $X_{1}$ can be expressed as:

$$ $\displaystyle I(X_{0};X_{1})=\frac{1}{2\epsilon}\int_{0}^{1}\mathbb{E}_{q_{\pi}(x_{t},x_{0})}\|v_{\text{joint}}(x_{t},t,x_{0})-v_{\text{ind}}(x_{t},t,x_{0})\|^{2}dt,$ (11) $$

where

$$ $\displaystyle v_{\text{joint}}(x_{t},t,x_{0})=\mathbb{E}_{q_{\pi}(x_{1}|x_{t},x_{0})}\left[\frac{x_{1}-x_{t}}{1-t}\right],$ (12) $\displaystyle v_{\text{ind}}(x_{t},t,x_{0})=\mathbb{E}_{q^{\text{ind}}_{\pi}(x_{1}|x_{t},x_{0})}\left[\frac{x_{1}-x_{t}}{1-t}\right].$ (13) $$

$v_{\text{joint}}$ and $v_{\text{ind}}$ are the drifts of the SDE representations ([4](https://arxiv.org/html/2502.01383v3#S2.E4)) of the reciprocal processes $Q_{\pi}$ and $Q^{\text{ind}}_{\pi}$.

Proof sketch: Starting from the quantity $\mathrm{KL}\left(Q_{\pi}\middle\|Q_{\pi}^{\text{ind}}\right)$, we first apply the disintegration theorem to express it as $\mathbb{E}_{\pi(x_{0})}[\mathrm{KL}\left(Q_{\pi|x_{0}}\middle\|Q_{\pi|x_{0}}^{\text{ind}}\right)]$ and apply it one more time to express it as $\mathrm{KL}\left(\pi(x_{0},x_{1})\middle\|\pi(x_{0})\pi(x_{1})\right)$. Finally, we decompose the KL between the conditional diffusion processes $\mathrm{KL}\left(Q_{\pi|x_{0}}\middle\|Q_{\pi|x_{0}}^{\text{ind}}\right)$ using Girsanov’s theorem. Full proof is provided in [Appendix B](https://arxiv.org/html/2502.01383v3#A2).

Once the drifts $v_{\text{joint}}$ and $v_{\text{ind}}$ are known, our Theorem [4.1](https://arxiv.org/html/2502.01383v3#S4.Thmtheorem1) provides a straightforward way to estimate the mutual information between the random variables $X_{0}$ and $X_{1}$ by evaluating the difference between the drifts $v_{\text{joint}}(x_{t},t,x_{0})$ ([12](https://arxiv.org/html/2502.01383v3#S4.E12)) and $v_{\text{ind}}(x_{t},t,x_{0})$ ([13](https://arxiv.org/html/2502.01383v3#S4.E13)) at points $x_{0},x_{t}$ sampled from the distribution of the reciprocal process $Q_{\pi}$ at times $0,t$. Regularity assumptions (Shi et al., 2023, Appendix C) are relatively mild and common for diffusion bridges generative modeling, i.e., they include restrictions such as finite first moment.

Remark. Our approach is different from MINDE, because we frame MI estimation as domain transfer task, while MINDE frames it as generative modeling task.
Our formulation allows unbiased
(^1^11In our case, by the word unbiased we mean that under full access to distributions and under the assumption that we can learn drift functions ideally, the estimation of the objective is unbiased.)
MI estimation  ([11](https://arxiv.org/html/2502.01383v3#S4.E11)) while MINDE has non zero bias term  ([9](https://arxiv.org/html/2502.01383v3#S3.E9)). Furthermore, the domain transfer perspective yields easier to learn trajectories and reduced variance in MI estimation [Tables 2](https://arxiv.org/html/2502.01383v3#S5.T2) and [3](https://arxiv.org/html/2502.01383v3#A4.T3). More details on the comparison with MINDE are presented in the Appendix [D](https://arxiv.org/html/2502.01383v3#A4).

Figure: Algorithm 1 InfoBridge. Training the model.

Figure: Algorithm 2 InfoBridge. MI estimator.

###### Theorem 4.1 (Mutual Information decomposition) .

### 4.2 Info Bridge. Practical optimization procedure

The drifts $v_{\text{joint}}$ and $v_{\text{ind}}$ of reciprocal processes $Q_{\pi}$ and $Q^{\text{ind}}_{\pi}$ can be recovered by the conditional Bridge Matching procedure, see (\wasyparagraph[2](https://arxiv.org/html/2502.01383v3#S2.SS0.SSS0.Px4)). We have to solve optimization problem ([2](https://arxiv.org/html/2502.01383v3#S2.Ex6)) by parametrizing $v_{\text{joint}}$ and $v_{\text{ind}}$ with neural networks $v_{\text{joint},\phi}$ and $v_{{\text{ind}},\psi}$, respectively, and applying Stochastic Gradient Descent on Monte Carlo approximation of ([2](https://arxiv.org/html/2502.01383v3#S2.Ex6)). The sampling from the distribution $q_{\pi}(x_{t},x_{0})$ of reciprocal process $Q_{\pi}$ at times $0,t$ is easy because:

$$ $\displaystyle q_{\pi}(x_{t},x_{0})=\mathbb{E}_{q_{\pi}(x_{1})}[q_{\pi}(x_{t},x_{0}|x_{1})]=\mathbb{E}_{q_{\pi}(x_{1})}[q_{\pi}(x_{t}|x_{1},x_{0})\pi(x_{0}|x_{1})].$ $$

Therefore, to sample from $q_{\pi}(x_{t},x_{0})$ it suffices to sample $x_{0},x_{1}\sim\pi(x_{0},x_{1})$ and sample from $q_{\pi}(x_{t}|x_{1},x_{0})$ which is again just a Brownian Bridge.

#### Vector field parametrization.

In practice, we replace two separate neural networks that approximate the drifts $v_{\text{joint}}(x_{t},t,x_{0})$ and $v_{\text{ind}}(x_{t},t,x_{0})$ with a single neural network that incorporates an additional binary input. Specifically, we introduce a binary input $s\in\{0,1\}$ to unify the drift approximations in the following way: $v_{\theta}(\cdot,1)\approx v_{\text{joint}}(\cdot)$ and $v_{\theta}(\cdot,0)\approx v_{\text{ind}}(\cdot)$. The introduction of an additional input is widely used for the conditioning of diffusion (Ho & Salimans, 2021) and bridge matching (Bortoli et al., 2024) models. We have empirically found that it provides a much more accurate estimation of mutual information due to correlation of bias in both drift functions.

We call our practical MI estimation algorithm InfoBridge. The drifts $v_{\theta}$ training procedure is described in Algorithm [1](https://arxiv.org/html/2502.01383v3#algorithm1) and the MI estimation procedure is described in Algorithm [2](https://arxiv.org/html/2502.01383v3#algorithm2).

### 4.3 Generalizations and implications

Generalizations. Our method admits several straightforward extensions.
For completeness, we present a method for the unbiased estimation of the general KL divergence in Appendix [C.1](https://arxiv.org/html/2502.01383v3#A3.SS1). According to the Theorem [C.1](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem1) the KL divergence between any two distributions can be decomposed into the difference of diffusion drifts in similar way to ([11](https://arxiv.org/html/2502.01383v3#S4.E11)).

Furthermore, our method can be extended to estimation of MI between random variables of different dimensionality (Appendix [C.5](https://arxiv.org/html/2502.01383v3#A3.SS5)) and naturally generalizes to the estimation of mutual information involving more than two random variables, known as interaction information (Appendix [C.4](https://arxiv.org/html/2502.01383v3#A3.SS4)). Practical procedures for these generalizations can also be derived in a similar way to (\wasyparagraph[4.2](https://arxiv.org/html/2502.01383v3#S4.SS2)).

Generative byproduct. Note that the learned drifts $v_{\theta}(\cdot,1)$ and $v_{\theta}(\cdot,0)$ define the distributions $\pi_{\theta,\text{joint}}(x_{1}|x_{0})\approx\pi(x_{1}|x_{0})$ and $\pi_{\theta,\text{ind}}(x_{1}|x_{0})\approx\pi(x_{1})$ as solutions to the corresponding SDEs ([4](https://arxiv.org/html/2502.01383v3#S2.E4)). One can simulate these SDEs by utilizing diffusion based generative modeling methods. Samples from $\pi_{\theta,\text{joint}}(x_{1}|x_{0})$ and $\pi_{\theta,\text{ind}}(x_{1}|x_{0})$ for the image based benchmark ([Section 5.2](https://arxiv.org/html/2502.01383v3#S5.SS2)) can be seen at Figure [5](https://arxiv.org/html/2502.01383v3#A5.F5). This is unnecessary for the MI estimation, but can be considered as an additional feature.

**Table 1: Mean MI estimates over $10$ seeds using $10$k test samples against ground truth (GT), adopted from Franzese et al. (2024). Color indicates relative negative (red) and positive bias (blue).**
| GT | Method Type | 0.2 | 0.4 | 0.3 | 0.4 | 0.4 | 0.4 | 0.4 | 1.0 | 1.0 | 1.0 | 1.0 | 0.3 | 1.0 | 1.3 | 1.0 | 0.4 | 1.0 | 0.6 | 1.6 | 0.4 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.2 | 0.4 | 0.2 | 0.3 | 0.2 | 0.4 | 0.3 | 0.4 | 1.7 | 0.3 | 0.4 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| InfoBridge(ours) | Bridge Matching | 0.3 | 0.5 | 0.3 | 0.4 | 0.4 | 0.4 | 0.4 | 0.9 | 1.0 | 1.0 | 1.0 | 0.3 | 1.0 | 1.3 | 1.0 | 0.4 | 1.0 | 0.6 | 1.7 | 0.4 | 1.0 | 1.0 | 1.0 | 1.0 | 0.9 | 0.9 | 1.0 | 1.0 | 1.0 | 0.0 | 0.0 | 0.2 | 0.3 | 0.2 | 0.5 | 0.3 | 0.5 | 1.3 | 0.4 | 0.4 |
| NVF | Flow | 0.2 | 0.4 | 0.3 | 0.6 | 0.4 | 0.4 | 0.4 | 1.0 | 1.0 | 1.0 | 1.0 | 0.3 | 1.0 | 1.3 | 1.0 | 0.4 | 1.0 | 0.6 | 1.5 | 0.4 | 1.0 | 1.0 | 1.0 | 0.8 | 0.5 | 0.6 | 0.9 | 1.0 | 1.0 | $\infty$ | $\infty$ | $\infty$ | 0.5 | 0.2 | -0.4 | 0.4 | 0.2 | 1.5 | 0.2 | 0.4 |
| JVF | 0.0 | 0.0 | 0.0 | 0.0 | 0.4 | 0.4 | 0.4 | 1.0 | 1.0 | 1.0 | 1.0 | 0.3 | 1.0 | 1.3 | 1.0 | 0.4 | 1.0 | 0.6 | 1.6 | 0.4 | 1.0 | 1.0 | 1.0 | 0.8 | 0.3 | 0.4 | 1.0 | 0.9 | 0.9 | 1.8 | 2.7 | 0.0 | 0.1 | 0.0 | 0.1 | 0.0 | 0.0 | 1.6 | 0.2 | 0.4 |  |
| MINDE–j | Diffusion | 0.2 | 0.4 | 0.3 | 0.4 | 0.4 | 0.4 | 0.4 | 1.2 | 1.0 | 1.0 | 1.0 | 0.3 | 1.0 | 1.3 | 1.0 | 0.4 | 1.0 | 0.6 | 1.7 | 0.4 | 1.1 | 1.0 | 1.0 | 1.0 | 0.9 | 0.9 | 1.1 | 1.0 | 1.0 | 0.1 | 0.2 | 0.2 | 0.3 | 0.2 | 0.5 | 0.3 | 0.4 | 1.7 | 0.3 | 0.4 |
| MINDE–c | 0.2 | 0.4 | 0.3 | 0.4 | 0.4 | 0.4 | 0.4 | 1.0 | 1.0 | 1.0 | 1.0 | 0.3 | 1.0 | 1.3 | 1.0 | 0.4 | 1.0 | 0.6 | 1.6 | 0.4 | 1.0 | 1.0 | 1.0 | 0.9 | 0.9 | 0.9 | 1.0 | 1.0 | 1.0 | 0.1 | 0.3 | 0.2 | 0.3 | 0.2 | 0.4 | 0.3 | 0.4 | 1.7 | 0.3 | 0.4 |  |
| MINE | Classical | 0.2 | 0.4 | 0.2 | 0.4 | 0.4 | 0.4 | 0.4 | 1.0 | 1.0 | 1.0 | 1.0 | 0.3 | 1.0 | 1.3 | 1.0 | 0.4 | 1.0 | 0.6 | 1.6 | 0.4 | 0.9 | 0.9 | 0.9 | 0.8 | 0.7 | 0.6 | 0.9 | 0.9 | 0.9 | 0.0 | 0.0 | 0.1 | 0.1 | 0.1 | 0.2 | 0.2 | 0.4 | 1.7 | 0.3 | 0.4 |
| InfoNCE | 0.2 | 0.4 | 0.3 | 0.4 | 0.4 | 0.4 | 0.4 | 1.0 | 1.0 | 1.0 | 1.0 | 0.3 | 1.0 | 1.3 | 1.0 | 0.4 | 1.0 | 0.6 | 1.6 | 0.4 | 0.9 | 1.0 | 1.0 | 0.8 | 0.8 | 0.8 | 0.9 | 1.0 | 1.0 | 0.2 | 0.3 | 0.2 | 0.3 | 0.2 | 0.4 | 0.3 | 0.4 | 1.7 | 0.3 | 0.4 |  |
| DoE (Gaussian) | 0.2 | 0.5 | 0.3 | 0.6 | 0.4 | 0.4 | 0.4 | 0.7 | 1.0 | 1.0 | 1.0 | 0.4 | 0.7 | 7.8 | 1.0 | 0.6 | 0.9 | 1.3 |  | 0.4 | 0.7 | 1.0 | 1.0 | 0.5 | 0.6 | 0.6 | 0.6 | 0.7 | 0.8 | 6.7 | 7.9 | 1.8 | 2.5 | 0.6 | 4.2 | 1.2 | 1.6 | 0.1 | 0.4 |  |  |
| DoE (Logistic) | 0.1 | 0.4 | 0.2 | 0.4 | 0.4 | 0.4 | 0.4 | 0.6 | 0.9 | 0.9 | 1.0 | 0.3 | 0.7 | 7.8 | 1.0 | 0.6 | 0.9 | 1.3 |  | 0.4 | 0.8 | 1.1 | 1.0 | 0.5 | 0.6 | 0.6 | 0.7 | 0.8 | 0.8 | 0.5 | 0.8 | 0.3 | 1.5 | 0.6 | 1.6 | 0.1 | 0.4 |  |  |  |  |
| KSG | 0.2 | 0.4 | 0.2 | 0.2 | 0.4 | 0.4 | 0.4 | 0.2 | 0.9 | 0.7 | 1.0 | 0.3 | 0.2 | 1.1 | 1.0 | 0.4 | 0.7 | 0.6 | 1.3 | 0.4 | 0.2 | 0.9 | 0.7 | 0.2 | 0.7 | 0.6 | 0.2 | 0.9 | 0.7 | 0.2 | 0.2 | 0.1 | 0.1 | 0.1 | 0.2 | 0.2 | 0.4 | 1.7 | 0.3 | 0.4 |  |
|  | Asinh @ St 1 × 1 (dof=1) | Asinh @ St 2 × 2 (dof=1) | Asinh @ St 3 × 3 (dof=2) | Asinh @ St 5 × 5 (dof=2) | Bimodal 1 × 1 | Bivariate Nm 1 × 1 | Hc @ Bivariate Nm 1 × 1 | Hc @ Mn 25 × 25 (2-pair) | Hc @ Mn 3 × 3 (2-pair) | Hc @ Mn 5 × 5 (2-pair) | Mn 2 × 2 (2-pair) | Mn 2 × 2 (dense) | Mn 25 × 25 (2-pair) | Mn 25 × 25 (dense) | Mn 3 × 3 (2-pair) | Mn 3 × 3 (dense) | Mn 5 × 5 (2-pair) | Mn 5 × 5 (dense) | Mn 50 × 50 (dense) | Nm CDF @ Bivariate Nm 1 × 1 | Nm CDF @ Mn 25 × 25 (2-pair) | Nm CDF @ Mn 3 × 3 (2-pair) | Nm CDF @ Mn 5 × 5 (2-pair) | Sp @ Mn 25 × 25 (2-pair) | Sp @ Mn 3 × 3 (2-pair) | Sp @ Mn 5 × 5 (2-pair) | Sp @ Nm CDF @ Mn 25 × 25 (2-pair) | Sp @ Nm CDF @ Mn 3 × 3 (2-pair) | Sp @ Nm CDF @ Mn 5 × 5 (2-pair) | St 1 × 1 (dof=1) | St 2 × 2 (dof=1) | St 2 × 2 (dof=2) | St 3 × 3 (dof=2) | St 3 × 3 (dof=3) | St 5 × 5 (dof=2) | St 5 × 5 (dof=3) | Swiss roll 2 × 1 | Uniform 1 × 1 (additive noise=0.1) | Uniform 1 × 1 (additive noise=0.75) | Wiggly @ Bivariate Nm 1 × 1 |  |

## 5 Experiments

We test our method on a diverse set of benchmarks with known ground truth values of MI.
To cover low-dimensional cases and some basic cases of data lying on a manifold,
we employ the tests by (Czyż et al., 2023).
Benchmarks from (Butakov et al., 2024b; a) are used to assess the method on manifolds represented as images. To evaluate our method on real-world data, we use protein language model embeddings and adopt the benchmark construction procedure proposed in (Lee & Rhee,). Finally, we evaluate our method on high mutual information tasks from (Czyż et al., 2023). In addition, we present an ablation study in Appendix [D.2](https://arxiv.org/html/2502.01383v3#A4.SS2), analyzing the impact of key hyperparameters, including the volatility coefficient $\epsilon$ and neural network architectures. We compare our method with a diverse collection of other MI estimation methods: diffusion based MINDE (Franzese et al., 2024), flow-based NVF, JVF (Dahlke & Pacheco,), classical MINE (Belghazi et al., 2018), InfoNCE (van den Oord et al., 2019), fDIME (Letizia et al., 2024), NWJ (Nguyen et al., 2010), KSG (Kraskov et al., 2004).

Across all four experimental setups, our method demonstrates consistently strong performance. Its main advantage emerges in challenging settings involving high-dimensional data and large mutual information (MI) values. In these regimes, our approach consistently outperforms both the diffusion-based MINDE estimator and classical MI estimation baselines.

### 5.1 Low-dimensional benchmark

The tests from (Czyż et al., 2023) focus on low-dimensional distributions with tractable mutual information.
Various mappings are also applied to make the distributions light- or heavy-tailed, or to non-linearly embed the data into an ambient space of higher dimensionality.

InfoBridge is tested with $\epsilon=1$ and a multi-layer dense neural network is used to approximate the drifts.
Our computational complexity is comparable to MINDE  (Franzese et al., 2024). For more details, please, refer to [Appendix E](https://arxiv.org/html/2502.01383v3#A5).
In each test, we perform $10$ independent runs with $100$k train set samples and $10$k test set samples.
We report the mean MI estimation in  [Tables 1](https://arxiv.org/html/2502.01383v3#S4.T1) and [14](https://arxiv.org/html/2502.01383v3#A5.T14).

Figure: (a) $16\times 16$ images (Gaussians)

Overall, the performance of our estimator is similar to that of MINDE (\wasyparagraph[3](https://arxiv.org/html/2502.01383v3#S3.SS0.SSS0.Px2)) and superior to the classical and flow based, methods, with the Cauchy distribution being the only notable exception. Unfortunately, the Cauchy distribution lacks the first moment, which poses theoretical limitations for Bridge Matching (Shi et al., 2023, Appendix C). However, we overcome this limitation using the tail-shortening $\operatorname{asinh}$ transform, which enables our method to achieve near-accurate MI estimates, see first two Asinh columns in [Tables 1](https://arxiv.org/html/2502.01383v3#S4.T1) and [14](https://arxiv.org/html/2502.01383v3#A5.T14).

### 5.2 Image data benchmark

In (Butakov et al., 2024b), it was proposed to map low-dimensional distributions with tractable MI into manifolds admitting image-like structure, thus producing synthetic images
(in particular, images of 2D Gaussians and Rectangles, see [Figures 5(a)](https://arxiv.org/html/2502.01383v3#A5.F5.sf1) and [5(b)](https://arxiv.org/html/2502.01383v3#A5.F5.sf2)).
By using smooth injective mappings, one ensures that MI is not alternated by the transform (Butakov et al., 2024a, Theorem 2.1). In the original works, it is argued that such benchmarks are closer to real data, and therefore closer to realistic setups.

Each neural algorithm is trained with $100$k train set samples and validated using $10$k samples. InfoBridge is tested with $\epsilon=1$ and we use a neural network with U-net architecture to approximate the drift.
Other experimental details are reported in [Section E.2](https://arxiv.org/html/2502.01383v3#A5.SS2) and the ablation study on neural network architecture and volatility coefficient $\epsilon$ is presented in [Section D.2](https://arxiv.org/html/2502.01383v3#A4.SS2).

**Table 2: Mean Average Error (MAE) averaged for all setups and random seeds, and standard deviations averaged over setups for Image based benchmark experiment. The best result is bolded.**
| Method | InfoBridge | InfoNCE | KSG | MINE | MIENF | NWJ | MINDE-C | MINDE-J |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MAE $\downarrow$ | 0.38 | 1.44 | 1.15 | 0.92 | 0.45 | 1.24 | 0.56 | 1.66 |
| Average std. $\downarrow$ | 0.07 | 0.04 | 0.02 | 0.13 | 0.08 | 0.08 | 0.43 | 0.45 |

We present our results for 16$\times$16 and 32$\times$32 resolution images with both Gaussian and Rectangle structure in [Figure 1](https://arxiv.org/html/2502.01383v3#S5.F1) and report aggregated MAE and std in Table [2](https://arxiv.org/html/2502.01383v3#S5.T2), while the samples from the learned conditional bridge matching models can be viewed in [Figures 5(c)](https://arxiv.org/html/2502.01383v3#A5.F5.sf3), [5(d)](https://arxiv.org/html/2502.01383v3#A5.F5.sf4), [5(e)](https://arxiv.org/html/2502.01383v3#A5.F5.sf5) and [5(f)](https://arxiv.org/html/2502.01383v3#A5.F5.sf6).

Our estimator is competitive, being consistently more precise and more stable than both MINDE-C and MINDE-J, and overall slightly better than the previous best-performing method:
MIENF (Butakov et al., 2024a).
In addition, our method delivers less variance in MI estimation w.r.t. random seeds, see [Table 2](https://arxiv.org/html/2502.01383v3#S5.T2).

### 5.3 Protein embeddings data

Figure: (a) Gaussian
Refer to caption: figures/high_mi/gaussian_plot.png

We evaluate InfoBridge on real-world data, i.e., conduct experiments on protein language model embeddings. Following the benchmark construction method from (Lee & Rhee,), we generate paired datasets with known mutual information (MI) by ensuring that only class labels are shared between variables $X$ and $Y$, making the MI analytically tractable (Lee & Rhee,, \wasyparagraph4.5). We use sequence embeddings from the ProtTrans5 model (Elnaggar et al., 2021), based on proteins from A. thaliana and H. sapiens, to create datasets with varying ground truth MI. The final number of training pairs is $20641$ and the dimensionality of embeddings is $1024$. In addition, the ablation study on the volatility coefficient $\epsilon$ is presented in [D.2](https://arxiv.org/html/2502.01383v3#A4.SS2).

Figure: Figure 3: Comparison of the selected estimators on the ProtTrans5 data. Along $x$ axes is ground truth $I(X_{0},X_{1})$, along $y$ axes is MI estimate $\hat{I}(X_{0},X_{1})$.
Refer to caption: figures/PT5_plot_human.png

InfoBridge with $\epsilon=1$ is trained for 100 epochs with similar to the low-dimensional benchmark (\wasyparagraph[5.1](https://arxiv.org/html/2502.01383v3#S5.SS1)) hyperparameters. We compare against MINE, InfoNCE, KSG, and MINDE. All the technical details can be found in Appendix [E.3](https://arxiv.org/html/2502.01383v3#A5.SS3). We present the results plot at Figure [3](https://arxiv.org/html/2502.01383v3#S5.F3) and MAE at Table [9](https://arxiv.org/html/2502.01383v3#A5.T9). One can see that InfoBridge is the only method to consistently estimate MI accurately; MINDE-C and MINDE-J are omitted as they drastically overestimate MI (e.g., MAE: MINDE-C 9.29 vs. InfoBridge 0.04).

### 5.4 High Mutual Information

To evaluate the performance of our method in high mutual information regimes, we conduct experiments on high-dimensional tasks introduced in (Czyż et al., 2023), including correlated uniform, smoothed uniform, correlated gaussian its half cube transformed variant. We systematically vary the dimensionality and mutual information of the random variables, setting $d\in\{20,40,80,160\}$ and $\text{MI}\in\{10,20,40,80\}$. Each experiment uses a training set of 100k samples and a test set of 10k samples. We evaluate our method with $\epsilon=0.01$ and show the results in Figure [2](https://arxiv.org/html/2502.01383v3#S5.F2). To keep narrative concise we provide additional baselines and experimental details, refer to Appendix [E.4](https://arxiv.org/html/2502.01383v3#A5.SS4).

As shown, our method yields the most accurate estimates compared to the ground truth, whereas MINE, InfoNCE, and fDIME fail to capture high mutual information values, and MINDE-C consistently underperforms relative to our approach.

## 6 Discussion

#### Potential Impact.

Our contributions include the development of novel unbiased estimator for the MI grounded in diffusion bridge matching theory. The proposed algorithm, InfoBridge, demonstrates superior performance compared to commonly used MI estimators on both synthetic and real-world challenging high-dimensional benchmarks. Also, our approach can be used to estimate the KL divergence and differential entropy [Sections C.1](https://arxiv.org/html/2502.01383v3#A3.SS1) and [C.2](https://arxiv.org/html/2502.01383v3#A3.SS2).

We believe that our work paves the way for new directions in the estimation of MI in high dimensions. This has potential real-world applications such as text-to-image alignment (Wang et al., 2024), self-supervised learning (Bachman et al., 2019), deep neural network analysis (Butakov et al., 2024b), and other use cases in high-dimensional settings.

Moreover, our approach offers opportunities for extension by exploring alternative types of bridges within reciprocal processes, for instance, variance-preserving SDE bridges (Zhou et al., 2024), or by incorporating advanced diffusion bridges techniques like time reweighting (Kim et al., 2025). Finally, for long-tailed data distributions, it may be possible to integrate long-tailed diffusion techniques (Yoon et al., 2023), extending the applicability of our approach.

#### Limitations.

We discuss limitations in Appendix [A](https://arxiv.org/html/2502.01383v3#A1).

#### LLM Usage.

Large Language Models (LLMs) were used only to assist with rephrasing sentences and improving the clarity of the text.

#### Reproducibility statement.

All the technical details that are required to reproduce the work are stated either in main experimental section (\wasyparagraph [5](https://arxiv.org/html/2502.01383v3#S5)) or [Appendix E](https://arxiv.org/html/2502.01383v3#A5). Additionally we provide code for the reproduction of experiments in supplementary. All the proofs for provided theoretical results are provided either in [Appendix B](https://arxiv.org/html/2502.01383v3#A2) or [Appendix C](https://arxiv.org/html/2502.01383v3#A3).

## References

- Amjad et al. (2022)
Rana Ali Amjad, Kairen Liu, and Bernhard C. Geiger.
Understanding neural networks and individual neuron importance via information-ordered cumulative ablation.
*IEEE Transactions on Neural Networks and Learning Systems*, 33(12):7842–7852, 2022.
doi: 10.1109/TNNLS.2021.3088685.
- Ao & Li (2022)
Ziqiao Ao and Jinglai Li.
Entropy estimation via normalizing flow.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 36(9):9990–9998, Jun. 2022.
doi: 10.1609/aaai.v36i9.21237.
URL [https://ojs.aaai.org/index.php/AAAI/article/view/21237](https://ojs.aaai.org/index.php/AAAI/article/view/21237).
- Ardizzone et al. (2020)
Lynton Ardizzone, Radek Mackowiak, Carsten Rother, and Ullrich Köthe.
Training normalizing flows with the information bottleneck for competitive generative classification.
In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), *Advances in Neural Information Processing Systems*, volume 33, pp. 7828–7840. Curran Associates, Inc., 2020.
URL [https://proceedings.neurips.cc/paper_files/paper/2020/file/593906af0d138e69f49d251d3e7cbed0-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2020/file/593906af0d138e69f49d251d3e7cbed0-Paper.pdf).
- Bachman et al. (2019)
Philip Bachman, R Devon Hjelm, and William Buchwalter.
Learning representations by maximizing mutual information across views.
In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 32. Curran Associates, Inc., 2019.
URL [https://proceedings.neurips.cc/paper_files/paper/2019/file/ddf354219aac374f1d40b7e760ee5bb7-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2019/file/ddf354219aac374f1d40b7e760ee5bb7-Paper.pdf).
- Belghazi et al. (2018)
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm.
Mutual information neural estimation.
In Jennifer Dy and Andreas Krause (eds.), *Proceedings of the 35th International Conference on Machine Learning*, volume 80 of *Proceedings of Machine Learning Research*, pp. 531–540. PMLR, 07 2018.
URL [https://proceedings.mlr.press/v80/belghazi18a.html](https://proceedings.mlr.press/v80/belghazi18a.html).
- Bell & Sejnowski (1995)
A J Bell and T J Sejnowski.
An information-maximization approach to blind separation and blind deconvolution.
*Neural Comput.*, 7(6):1129–1159, November 1995.
- Berrett & Samworth (2017)
Thomas Berrett and Richard Samworth.
Nonparametric independence testing via mutual information.
*Biometrika*, 106, 11 2017.
doi: 10.1093/biomet/asz024.
- Berrett et al. (2019)
Thomas B. Berrett, Richard J. Samworth, and Ming Yuan.
Efficient multivariate entropy estimation via $k$-nearest neighbour distances.
*Ann. Statist.*, 47(1):288–318, 02 2019.
doi: 10.1214/18-AOS1688.
URL [https://doi.org/10.1214/18-AOS1688](https://doi.org/10.1214/18-AOS1688).
- Bortoli et al. (2024)
Valentin De Bortoli, Iryna Korshunova, Andriy Mnih, and Arnaud Doucet.
Schrodinger bridge flow for unpaired data translation.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=1F32iCJFfa](https://openreview.net/forum?id=1F32iCJFfa).
- Bounoua et al. (2024a)
Mustapha Bounoua, Giulio Franzese, and Pietro Michiardi.
S$\omega$i: Score-based o-INFORMATION estimation.
In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), *Proceedings of the 41st International Conference on Machine Learning*, volume 235 of *Proceedings of Machine Learning Research*, pp. 4444–4471. PMLR, 21–27 Jul 2024a.
URL [https://proceedings.mlr.press/v235/bounoua24a.html](https://proceedings.mlr.press/v235/bounoua24a.html).
- Bounoua et al. (2024b)
Mustapha Bounoua, Giulio Franzese, and Pietro Michiardi.
S$\omega$i: Score-based o-information estimation.
In *ICML 2024, 41st International Conference on Machine Learning*, 2024b.
- Bunne et al. (2023)
Charlotte Bunne, Ya-Ping Hsieh, Marco Cuturi, and Andreas Krause.
The schrödinger bridge between gaussian measures has a closed form.
In *International Conference on Artificial Intelligence and Statistics*, pp. 5802–5833. PMLR, 2023.
- Butakov et al. (2024a)
Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, and Alexey Frolov.
Mutual information estimation via normalizing flows.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024a.
URL [https://openreview.net/forum?id=JiQXsLvDls](https://openreview.net/forum?id=JiQXsLvDls).
- Butakov et al. (2024b)
Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, Alexey Frolov, and Kirill Andreev.
Information bottleneck analysis of deep neural networks via lossy compression.
In *The Twelfth International Conference on Learning Representations*, 2024b.
URL [https://openreview.net/forum?id=huGECz8dPp](https://openreview.net/forum?id=huGECz8dPp).
- Chen et al. (2016)
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel.
Infogan: Interpretable representation learning by information maximizing generative adversarial nets.
In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), *Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain*, pp. 2172–2180, 2016.
URL [https://proceedings.neurips.cc/paper/2016/hash/7c9d0b1f96aebd7b5eca8c3edaa19ebb-Abstract.html](https://proceedings.neurips.cc/paper/2016/hash/7c9d0b1f96aebd7b5eca8c3edaa19ebb-Abstract.html).
- Chen et al. (2023)
Zehua Chen, Guande He, Kaiwen Zheng, Xu Tan, and Jun Zhu.
Schrodinger bridges beat diffusion models on text-to-speech synthesis.
*arXiv preprint arXiv:2312.03491*, 2023.
- Cover & Thomas (2006)
Thomas M. Cover and Joy A. Thomas.
*Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing)*.
Wiley-Interscience, USA, 2006.
- Czyż et al. (2023)
Paweł Czyż, Frederic Grabowski, Julia E Vogt, Niko Beerenwinkel, and Alexander Marx.
Beyond normal: On the evaluation of mutual information estimators.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=25vRtG56YH](https://openreview.net/forum?id=25vRtG56YH).
- (19)
Caleb Dahlke and Jason Pacheco.
Flow-based variational mutual information: Fast and flexible approximations.
In *The Thirteenth International Conference on Learning Representations*.
- De Bortoli et al. (2023)
Valentin De Bortoli, Guan-Horng Liu, Tianrong Chen, Evangelos A Theodorou, and Weilie Nie.
Augmented bridge matching.
*arXiv preprint arXiv:2311.06978*, 2023.
- Dosi & Roventini (2019)
Giovanni Dosi and Andrea Roventini.
More is different… and complex! the case for agent-based macroeconomics.
*Journal of Evolutionary Economics*, 29:1–37, 2019.
- Duong & Nguyen (2023a)
Bao Duong and Thin Nguyen.
Diffeomorphic information neural estimation.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 37(6):7468–7475, Jun. 2023a.
doi: 10.1609/aaai.v37i6.25908.
URL [https://ojs.aaai.org/index.php/AAAI/article/view/25908](https://ojs.aaai.org/index.php/AAAI/article/view/25908).
- Duong & Nguyen (2023b)
Bao Duong and Thin Nguyen.
Normalizing flows for conditional independence testing.
*Knowledge and Information Systems*, 66, 08 2023b.
doi: 10.1007/s10115-023-01964-w.
- Elnaggar et al. (2021)
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al.
Prottrans: towards cracking the language of life’s code through self-supervised learning.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 44:7112–7127, 2021.
- Fayad & Ibrahim (2023)
Ammar Fayad and Majd Ibrahim.
On slicing optimality for mutual information.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=JMuKfZx2xU](https://openreview.net/forum?id=JMuKfZx2xU).
- Federici et al. (2023)
Marco Federici, David Ruhe, and Patrick Forré.
On the effectiveness of hybrid mutual information estimation, 2023.
URL [https://arxiv.org/abs/2306.00608](https://arxiv.org/abs/2306.00608).
- Franzese et al. (2024)
Giulio Franzese, Mustapha BOUNOUA, and Pietro Michiardi.
MINDE: Mutual information neural diffusion estimation.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=0kWd8SJq8d](https://openreview.net/forum?id=0kWd8SJq8d).
- Goldfeld et al. (2020)
Z. Goldfeld, K. Greenewald, J. Niles-Weed, and Y. Polyanskiy.
Convergence of smoothed empirical measures with applications to entropy estimation.
*IEEE Transactions on Information Theory*, 66(7):4368–4391, 2020.
doi: 10.1109/TIT.2020.2975480.
- Goldfeld & Greenewald (2021)
Ziv Goldfeld and Kristjan Greenewald.
Sliced mutual information: A scalable measure of statistical dependence.
In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), *Advances in Neural Information Processing Systems*, 2021.
URL [https://openreview.net/forum?id=SvrYl-FDq2](https://openreview.net/forum?id=SvrYl-FDq2).
- Goldfeld et al. (2019)
Ziv Goldfeld, Ewout van den Berg, Kristjan H. Greenewald, Igor V. Melnyk, Nam H. Nguyen, Brian Kingsbury, and Yury Polyanskiy.
Estimating information flow in deep neural networks.
In *ICML*, 2019.
- Goldfeld et al. (2022)
Ziv Goldfeld, Kristjan Greenewald, Theshani Nuradha, and Galen Reeves.
$k$-sliced mutual information: A quantitative study of scalability with dimension.
In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), *Advances in Neural Information Processing Systems*, 2022.
URL [https://openreview.net/forum?id=L-ceBdl2DPb](https://openreview.net/forum?id=L-ceBdl2DPb).
- Greenewald et al. (2023)
Kristjan H. Greenewald, Brian Kingsbury, and Yuancheng Yu.
High-dimensional smoothed entropy estimation via dimensionality reduction.
In *IEEE International Symposium on Information Theory, ISIT 2023, Taipei, Taiwan, June 25-30, 2023*, pp. 2613–2618. IEEE, 2023.
doi: 10.1109/ISIT54713.2023.10206641.
URL [https://doi.org/10.1109/ISIT54713.2023.10206641](https://doi.org/10.1109/ISIT54713.2023.10206641).
- Gushchin et al. (2023)
Nikita Gushchin, Alexander Kolesov, Alexander Korotin, Dmitry Vetrov, and Evgeny Burnaev.
Entropic neural optimal transport via diffusion processes.
In *Advances in Neural Information Processing Systems*, 2023.
- Gushchin et al. (2024a)
Nikita Gushchin, Sergei Kholkin, Evgeny Burnaev, and Alexander Korotin.
Light and optimal schrödinger bridge matching.
In *Forty-first International Conference on Machine Learning*, 2024a.
- Gushchin et al. (2024b)
Nikita Gushchin, Daniil Selikhanovych, Sergei Kholkin, Evgeny Burnaev, and Alexander Korotin.
Adversarial schr$\backslash$” odinger bridge matching.
*arXiv preprint arXiv:2405.14449*, 2024b.
- Hjelm et al. (2019)
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.
Learning deep representations by mutual information estimation and maximization.
In *International Conference on Learning Representations*, 2019.
URL [https://openreview.net/forum?id=Bklr3j0cKX](https://openreview.net/forum?id=Bklr3j0cKX).
- Ho & Salimans (2021)
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
In *NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications*, 2021.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
*Advances in Neural Information Processing Systems*, 33:6840–6851, 2020.
- Ibe (2013)
Oliver Ibe.
*Markov processes for stochastic modeling*.
Newnes, 2013.
- Igashov et al. (2024)
Ilia Igashov, Arne Schneuing, Marwin Segler, Michael M Bronstein, and Bruno Correia.
Retrobridge: Modeling retrosynthesis with markov bridges.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Kappen (2024)
Bert Kappen.
Lecture notes in cds machine learning, September 2024.
- Kim et al. (2025)
Beomsu Kim, Yu-Guan Hsieh, Michal Klein, Marco Cuturi, Jong Chul Ye, Bahjat Kawar, and James Thornton.
Simple reflow: Improved techniques for fast flow models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=fpvgSDKXGY](https://openreview.net/forum?id=fpvgSDKXGY).
- Kingma & Ba (2017)
Diederik P. Kingma and Jimmy Ba.
Adam: A method for stochastic optimization, 2017.
- Kingma & Dhariwal (2018)
Durk P Kingma and Prafulla Dhariwal.
Glow: Generative flow with invertible 1x1 convolutions.
*Advances in neural information processing systems*, 31, 2018.
- (45)
Alexander Korotin, Nikita Gushchin, and Evgeny Burnaev.
Light schrödinger bridge.
In *The Twelfth International Conference on Learning Representations*.
- Kozachenko & Leonenko (1987)
L. F. Kozachenko and N. N. Leonenko.
Sample estimate of the entropy of a random vector.
*Problems Inform. Transmission*, 23:95–101, 1987.
- Kraskov et al. (2004)
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger.
Estimating mutual information.
*Phys. Rev. E*, 69:066138, Jun 2004.
doi: 10.1103/PhysRevE.69.066138.
URL [https://link.aps.org/doi/10.1103/PhysRevE.69.066138](https://link.aps.org/doi/10.1103/PhysRevE.69.066138).
- (48)
Kyungeun Lee and Wonjong Rhee.
A benchmark suite for evaluating neural mutual information estimators on unstructured datasets.
In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*.
- Léonard (2013)
Christian Léonard.
A survey of the schr$\backslash$” odinger problem and some of its connections with optimal transport.
*arXiv preprint arXiv:1308.0215*, 2013.
- Léonard (2014)
Christian Léonard.
Some properties of path measures.
*Séminaire de Probabilités XLVI*, pp. 207–230, 2014.
- Léonard et al. (2014)
Christian Léonard, Sylvie Rœlly, and Jean-Claude Zambrini.
Reciprocal processes. a measure-theoretical point of view.
*Probability Surveys*, 11:237–269, 2014.
- Letizia et al. (2024)
Nunzio Alexandro Letizia, Nicola Novello, and Andrea M Tonello.
Mutual information estimation via $f$-divergence and data derangements.
*Advances in Neural Information Processing Systems*, 37:105114–105150, 2024.
- Linsker (1988)
R. Linsker.
Self-organization in a perceptual network.
*Computer*, 21(3):105–117, 1988.
doi: 10.1109/2.36.
- Liu et al. (2023)
Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie, and Anima Anandkumar.
I^2sb: Image-to-image schr$\backslash$” odinger bridge.
*arXiv preprint arXiv:2302.05872*, 2023.
- McAllester & Stratos (2020)
David McAllester and Karl Stratos.
Formal limitations on the measurement of mutual information.
In Silvia Chiappa and Roberto Calandra (eds.), *Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics*, volume 108 of *Proceedings of Machine Learning Research*, pp. 875–884. PMLR, 08 2020.
URL [https://proceedings.mlr.press/v108/mcallester20a.html](https://proceedings.mlr.press/v108/mcallester20a.html).
- Nguyen et al. (2010)
XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan.
Estimating divergence functionals and the likelihood ratio by convex risk minimization.
*IEEE Transactions on Information Theory*, 56(11):5847–5861, 2010.
- Øksendal (2003)
Bernt Øksendal.
*Stochastic differential equations*.
Springer, 2003.
- Palmowski & Rolski (2002)
Zbigniew Palmowski and Tomasz Rolski.
A technique for exponential change of measure for markov processes.
*Bernoulli*, pp. 767–785, 2002.
- Peluchetti (2023)
Stefano Peluchetti.
Diffusion bridge mixture transports, schrödinger bridge problems and generative modeling.
*Journal of Machine Learning Research*, 24(374):1–51, 2023.
- Polyanskiy & Wu (2024)
Y. Polyanskiy and Y. Wu.
*Information Theory: From Coding to Learning*.
Cambridge University Press, 2024.
ISBN 9781108832908.
URL [https://books.google.ru/books?id=CySo0AEACAAJ](https://books.google.ru/books?id=CySo0AEACAAJ).
- Rhodes et al. (2020)
Benjamin Rhodes, Kai Xu, and Michael U. Gutmann.
Telescoping density-ratio estimation.
In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), *Advances in Neural Information Processing Systems*, volume 33, pp. 4905–4916. Curran Associates, Inc., 2020.
URL [https://proceedings.neurips.cc/paper_files/paper/2020/file/33d3b157ddc0896addfb22fa2a519097-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2020/file/33d3b157ddc0896addfb22fa2a519097-Paper.pdf).
- Rombach et al. (2022)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 10684–10695, 2022.
- Ronneberger et al. (2015)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
U-net: Convolutional networks for biomedical image segmentation.
In *Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18*, pp. 234–241. Springer, 2015.
- Rosas et al. (2019)
Fernando E Rosas, Pedro AM Mediano, Michael Gastpar, and Henrik J Jensen.
Quantifying high-order interdependencies via multivariate extensions of the mutual information.
*Physical Review E*, 100(3):032305, 2019.
- Runge et al. (2019)
Jakob Runge, Sebastian Bathiany, Erik Bollt, Gustau Camps-Valls, Dim Coumou, Ethan Deyle, Clark Glymour, Marlene Kretschmer, Miguel D Mahecha, Jordi Muñoz-Marí, et al.
Inferring causation from time series in earth system sciences.
*Nature communications*, 10(1):2553, 2019.
- Schrödinger (1932)
Erwin Schrödinger.
Sur la théorie relativiste de l’électron et l’interprétation de la mécanique quantique.
In *Annales de l’institut Henri Poincaré*, volume 2, pp. 269–310, 1932.
- Sen et al. (2017)
Rajat Sen, Ananda Theertha Suresh, Karthikeyan Shanmugam, Alexandros G Dimakis, and Sanjay Shakkottai.
Model-powered conditional independence test.
In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc., 2017.
URL [https://proceedings.neurips.cc/paper_files/paper/2017/file/02f039058bd48307e6f653a2005c9dd2-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2017/file/02f039058bd48307e6f653a2005c9dd2-Paper.pdf).
- Shi et al. (2023)
Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Arnaud Doucet.
Diffusion schrödinger bridge matching.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=qy07OHsJT5](https://openreview.net/forum?id=qy07OHsJT5).
- Somnath et al. (2023)
Vignesh Ram Somnath, Matteo Pariset, Ya-Ping Hsieh, Maria Rodriguez Martinez, Andreas Krause, and Charlotte Bunne.
Aligned diffusion schrödinger bridges.
In *Uncertainty in Artificial Intelligence*, pp. 1985–1995. PMLR, 2023.
- Song & Ermon (2020)
Jiaming Song and Stefano Ermon.
Understanding the limitations of variational mutual information estimators.
In *International Conference on Learning Representations*, 2020.
URL [https://openreview.net/forum?id=B1x62TNtDS](https://openreview.net/forum?id=B1x62TNtDS).
- Song et al. (2021)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
In *International Conference on Learning Representations*, 2021.
- Steinke & Zakynthinou (2020)
Thomas Steinke and Lydia Zakynthinou.
Reasoning About Generalization via Conditional Mutual Information.
In Jacob Abernethy and Shivani Agarwal (eds.), *Proceedings of Thirty Third Conference on Learning Theory*, volume 125 of *Proceedings of Machine Learning Research*, pp. 3437–3452. PMLR, 09–12 Jul 2020.
URL [https://proceedings.mlr.press/v125/steinke20a.html](https://proceedings.mlr.press/v125/steinke20a.html).
- Stimper et al. (2023)
Vincent Stimper, David Liu, Andrew Campbell, Vincent Berenz, Lukas Ryll, Bernhard Schölkopf, and José Miguel Hernández-Lobato.
normflows: A pytorch package for normalizing flows.
*Journal of Open Source Software*, 8(86):5361, 2023.
doi: 10.21105/joss.05361.
URL [https://doi.org/10.21105/joss.05361](https://doi.org/10.21105/joss.05361).
- Stratos (2019)
Karl Stratos.
Mutual information maximization for simple and accurate part-of-speech induction.
In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pp. 1095–1104, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
doi: 10.18653/v1/N19-1113.
URL [https://aclanthology.org/N19-1113/](https://aclanthology.org/N19-1113/).
- Tishby & Zaslavsky (2015)
Naftali Tishby and Noga Zaslavsky.
Deep learning and the information bottleneck principle.
*2015 IEEE Information Theory Workshop (ITW)*, pp. 1–5, 2015.
- Tong et al. (2024)
Alexander Y Tong, Nikolay Malkin, Kilian Fatras, Lazar Atanackovic, Yanlei Zhang, Guillaume Huguet, Guy Wolf, and Yoshua Bengio.
Simulation-free schrödinger bridges via score and flow matching.
In *International Conference on Artificial Intelligence and Statistics*, pp. 1279–1287. PMLR, 2024.
- Tschannen et al. (2020)
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic.
On mutual information maximization for representation learning.
In *International Conference on Learning Representations*, 2020.
URL [https://openreview.net/forum?id=rkxoh24FPH](https://openreview.net/forum?id=rkxoh24FPH).
- Tsur et al. (2023)
Dor Tsur, Ziv Goldfeld, and Kristjan Greenewald.
Max-sliced mutual information.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=ce9B2x3zQa](https://openreview.net/forum?id=ce9B2x3zQa).
- van den Oord et al. (2019)
Aaron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding, 2019.
URL [https://arxiv.org/abs/1807.03748](https://arxiv.org/abs/1807.03748).
- Vargas et al. (2021)
Francisco Vargas, Pierre Thodoroff, Austen Lamacraft, and Neil Lawrence.
Solving schrödinger bridges via maximum likelihood.
*Entropy*, 23(9):1134, 2021.
- Vargas et al. (2023)
Francisco Vargas, Andrius Ovsianas, David Fernandes, Mark Girolami, Neil D Lawrence, and Nikolas Nüsken.
Bayesian learning via neural schrödinger–föllmer flows.
*Statistics and Computing*, 33(1):3, 2023.
- Veličković et al. (2019)
Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm.
Deep graph infomax.
In *International Conference on Learning Representations*, 2019.
URL [https://openreview.net/forum?id=rklz9iAcKQ](https://openreview.net/forum?id=rklz9iAcKQ).
- Wang et al. (2024)
Chao Wang, Giulio Franzese, Alessandro Finamore, Massimo Gallo, and Pietro Michiardi.
Information theoretic text-to-image alignment, 2024.
URL [https://arxiv.org/abs/2405.20759](https://arxiv.org/abs/2405.20759).
- Weglarczyk (2018)
Stanislaw Weglarczyk.
Kernel density estimation and its application.
*ITM Web of Conferences*, 23:00037, 01 2018.
doi: 10.1051/itmconf/20182300037.
- Xu & Raginsky (2017)
Aolin Xu and Maxim Raginsky.
Information-theoretic analysis of generalization capability of learning algorithms.
*Advances in neural information processing systems*, 30, 2017.
- Yoon et al. (2023)
Eun Bi Yoon, Keehun Park, Sungwoong Kim, and Sungbin Lim.
Score-based generative models with lévy processes.
*Advances in Neural Information Processing Systems*, 36:40694–40707, 2023.
- Zhou et al. (2024)
Linqi Zhou, Aaron Lou, Samar Khanna, and Stefano Ermon.
Denoising diffusion bridge models.
In *The Twelfth International Conference on Learning Representations*, 2024.

## Appendix Contents

## Appendix A Limitations

Our approach is based on the diffusion bridge matching framework Peluchetti (2023); Shi et al. (2023).
While this class of models and its theoretical foundations have demonstrated strong potential in high-dimensional generative modeling Shi et al. (2023); Liu et al. (2023); Song et al. (2021), they also have certain limitations that, while often negligible in the context of generative modeling, can be more pronounced in other applications.

One such limitation, evident in our work, is the challenge of accurately approximating heavy-tailed distributions.
As can be seen in Table [1](https://arxiv.org/html/2502.01383v3#S4.T1), one-degree-of-freedom Student-t distribution, i.e., St (dof=1), also known as Cauchy distribution, has no first moment, and our method is not guaranteed to work in such a case due to theoretical requirements of bridge matching (Shi et al., 2023, Appendix C).
However, it is worth noting that such limitations are quite common for generative modeling and quite probably should be applicable to denoising score matching and diffusion models as well Franzese et al. (2024); Song et al. (2021). Another limitation of diffusion (and diffusion bridge matching) is that is requires of a lot of data samples at training stage. This could hinder the applicability of our method with low number of data samples.

## Appendix B Proofs

### B.1 Proof of Theorem 1

###### Proof.

Using the disintegration theorem and additive property of relative entropy (Léonard, 2014, Theorems 1.6, 2.4) at time $t=0$, we get:

$$ $\displaystyle\mathrm{KL}\left(Q_{\pi}\middle\|Q^{\text{ind}}_{\pi}\right)=\mathrm{KL}\left(\pi(x_{0})\middle\|\pi(x_{0})\right)+\mathbb{E}_{\pi(x_{0})}\big[\mathrm{KL}\left(Q_{\pi|x_{0}}\middle\|Q^{\text{ind}}_{\pi|x_{0}}\right)\big].$ $$

Note that since $Q_{\pi},Q^{\text{ind}}_{\pi}$ share the same marginals at time $t=0$, first KL term vanishes. Similarly, by using the disintegration theorem again for both times $t=0,1$, we get:

$$ $\displaystyle\mathrm{KL}\left(Q_{\pi}\middle\|Q^{\text{ind}}_{\pi}\right)=\mathrm{KL}\left(\pi(x_{0},x_{1})\middle\|\pi(x_{0})\pi(x_{1})\right)+\mathbb{E}_{\pi(x_{0},x_{1})}\big[\mathrm{KL}\left(Q_{\pi|x_{0},x_{1}}\middle\|Q^{\text{ind}}_{\pi|x_{0},x_{1}}\right)\big].$ $$

Recap that $Q_{\pi}$ and $Q^{\text{ind}}_{\pi}$ are both mixtures of Brownian Bridges. Therefore, $Q_{\pi|x_{0},x_{1}}=Q^{\text{ind}}_{\pi|x_{0},x_{1}}=W^{\epsilon}_{|x_{0},x_{1}}$ and $\mathrm{KL}\left(Q_{\pi|x_{0},x_{1}}\middle\|Q^{\text{ind}}_{\pi|x_{0},x_{1}}\right)=0$. Then the following holds:

$$ $\displaystyle\mathrm{KL}\left(Q_{\pi}\middle\|Q^{\text{ind}}_{\pi}\right)=\mathrm{KL}\left(\pi(x_{0},x_{1})\middle\|\pi(x_{0})\pi(x_{1})\right)=\mathbb{E}_{\pi(x_{0})}\big[\mathrm{KL}\left(Q_{\pi|x_{0}}\middle\|Q^{\text{ind}}_{\pi|x_{0}}\right)\big].$ $$

Moreover, processes $Q_{\pi|x_{0}}$ and $Q^{\text{ind}}_{\pi|x_{0}}$ are diffusion processes (\wasyparagraph[2](https://arxiv.org/html/2502.01383v3#S2.SS0.SSS0.Px3)). Then, by recalling ([7](https://arxiv.org/html/2502.01383v3#S2.E7)), we get:

$$ $\displaystyle\mathrm{KL}\left(\pi(x_{0},x_{1})\middle\|\pi(x_{0})\pi(x_{1})\right)=\mathbb{E}_{\pi(x_{0})}\big[\mathrm{KL}\left(Q_{\pi|x_{0}}\middle\|Q^{\text{ind}}_{\pi|x_{0}}\right)\big]=$ (14) $\displaystyle\frac{1}{2\epsilon}\int_{0}^{1}\mathbb{E}_{q_{\pi}(x_{t},x_{0})}\big[\|v_{\text{joint}}(x_{t},t,x_{0})-v_{\text{ind}}(x_{t},t,x_{0})\|^{2}\big]dt,$ $$

where drifts $v_{\text{joint}}$ and $v_{\text{ind}}$ are defined as in ([12](https://arxiv.org/html/2502.01383v3#S4.E12)) and ([13](https://arxiv.org/html/2502.01383v3#S4.E13)) respectively.
∎

###### Proof.

## Appendix C Extensions of our methodology

In this appendix section we present:

- •
Appendix [C.1](https://arxiv.org/html/2502.01383v3#A3.SS1). General KL divergence estimation method with proof and discussion.
- •
Appendix [C.2](https://arxiv.org/html/2502.01383v3#A3.SS2). Differential entropy estimation result with two possible algorithms and discussion.
- •
Appendix [C.3](https://arxiv.org/html/2502.01383v3#A3.SS3). Experimental illustrations to the entropy estimation algorithm.
- •
Appendix [C.4](https://arxiv.org/html/2502.01383v3#A3.SS4). Interaction information estimation method.

### C.1 KL divergence estimator

In this section, we present a general result for the unbiased estimation of KL divergence between any two distributions $\pi_{1}(x),\pi_{2}(x)\in\mathcal{P}(\mathbb{R}^{d})$ through the difference of drifts in the SDE formulation ([4](https://arxiv.org/html/2502.01383v3#S2.E4)) of reciprocal processes induced by these distributions, i.e.:

$$ $\displaystyle Q_{\pi_{1}}\stackrel{{\scriptstyle\text{def}}}{{=}}\int W^{\epsilon}_{|x_{0},x_{1}}d\pi_{1}(x_{1})dp(x_{0}),$ (15) $\displaystyle Q_{\pi_{2}}\stackrel{{\scriptstyle\text{def}}}{{=}}\int W^{\epsilon}_{|x_{0},x_{1}}d\pi_{2}(x_{1})dp(x_{0}).$ (16) $$

###### Theorem C.1 (KL divergence decomposition) .

Consider distributions $\pi_{1}(x_{1}),\pi_{2}(x_{1}),p(x_{0})\in\mathcal{P}(\mathbb{R}^{d})$, that satisfy some regularity assumptions (Shi et al., 2023, Appendix C), and reciprocal processes $Q_{\pi_{1}}$, $Q_{\pi_{2}}$ induced by distributions $\pi_{1}(x_{1}),\pi_{2}(x_{1})$ ([15](https://arxiv.org/html/2502.01383v3#A3.E15)), ([16](https://arxiv.org/html/2502.01383v3#A3.E16)). Then the KL divergence between distributions $\pi_{1}(x_{1})$ and $\pi_{2}(x_{1})$ can be represented in the following way:

$$ $\displaystyle\mathrm{KL}\left(\pi_{1}(x_{1})\middle\|\pi_{2}(x_{1})\right)=\frac{1}{2\epsilon}\int_{0}^{1}\mathbb{E}_{q_{\pi_{1}}(x_{t},x_{0})}\big[\|v^{\pi_{1}}(x_{t},t,x_{0})-v^{\pi_{2}}(x_{t},t,x_{0})\|^{2}\big]dt,$ (17) $$

where

$$ $\displaystyle v^{\pi_{1}}(x_{t},t,x_{0})=\mathbb{E}_{q_{\pi_{1}}(x_{1}|x_{t},x_{0})}\left[\frac{x_{1}-x_{t}}{1-t}\right],$ (18) $\displaystyle v^{\pi_{2}}(x_{t},t,x_{0})=\mathbb{E}_{q_{\pi_{2}}(x_{1}|x_{t},x_{0})}\left[\frac{x_{1}-x_{t}}{1-t}\right]$ (19) $$

are the drifts of reciprocal processes $Q_{\pi_{1}}$ and $Q_{\pi_{2}}$ respectively. Holds for any distribution $p(x_{0})$.

Our theorem allows us to estimate $\mathrm{KL}\left(\pi_{1}(x)\middle\|\pi_{2}(x)\right)$ knowing only the drifts $v^{\pi_{1}}(x_{t},t,x_{0})$ and $v^{\pi_{2}}(x_{t},t,x_{0})$, which can be recovered using conditional bridge matching (\wasyparagraph[2](https://arxiv.org/html/2502.01383v3#S2.SS0.SSS0.Px4)). Note that the expression ([17](https://arxiv.org/html/2502.01383v3#A3.E17)) is very similar to the expression ([11](https://arxiv.org/html/2502.01383v3#S4.E11)) in (\wasyparagraph[4.1](https://arxiv.org/html/2502.01383v3#S4.Thmtheorem1)).
Distribution $p(x_{0})$, can be considered as part of the design space and optimised for each particular problem.

The KL divergence is a fundamental quantity, and its estimator can have many applications, such as mutual information estimation or entropy estimation using results described in [C.2](https://arxiv.org/html/2502.01383v3#A3.SS2).

###### Proof.

Consider

$$ $\displaystyle\mathrm{KL}\left(\pi_{1}(x_{1})p(x_{0})\middle\|\pi_{2}(x_{1})p(x_{0})\right)=\underbrace{\mathrm{KL}\left(p(x_{0})\middle\|p(x_{0})\right)}_{=0}+\mathbb{E}_{p(x_{0})}\left[\mathrm{KL}\left(\pi_{1}(x_{1}|x_{0})\middle\|\pi_{2}(x_{1}|x_{0})\right)\right]=$ $\displaystyle=\mathbb{E}_{p(x_{0})}\left[\mathrm{KL}\left(\pi_{1}(x_{1})\middle\|\pi_{2}(x_{1})\right)\right]=\mathrm{KL}\left(\pi_{1}(x_{1})\middle\|\pi_{2}(x_{1})\right),$ $$

Next to get

$$ $\displaystyle\mathrm{KL}\left(\pi_{1}(x_{1})p(x_{0})\middle\|\pi_{2}(x_{1})p(x_{0})\right)=\frac{1}{2\epsilon}\int_{0}^{1}\mathbb{E}_{q_{\pi_{1}}(x_{t},x_{0})}\big[\|v^{\pi_{1}}(x_{t},t,x_{0})-v^{\pi_{2}}(x_{t},t,x_{0})\|^{2}dt\big],$ $$

one can repeat all the steps that were taken to show:

$$ $\displaystyle\mathrm{KL}\left(\pi(x_{0},x_{1})\middle\|\pi(x_{0})\pi(x_{1})\right)=\frac{1}{2\epsilon}\int_{0}^{1}\mathbb{E}_{q_{\pi}(x_{t},x_{0})}\big[\|v_{\text{joint}}(x_{t},t,x_{0})-v_{\text{ind}}(x_{t},t,x_{0})\|^{2}\big]dt,$ $$

in the proof of Theorem [4.1](https://arxiv.org/html/2502.01383v3#S4.Thmtheorem1).

∎

###### Theorem C.1 (KL divergence decomposition) .

###### Proof.

### C.2 Differential entropy estimator

A general result on the information projections and maximum-entropy distributions suggests a way of calculating differential entropy through the KL divergence estimation.

###### Theorem C.2 (Theorem 6.7 in Kappen ( 2024 ) ) .

Let $\phi\colon\mathbb{R}^{n}\to\mathbb{R}^{k}$ be any measurable function, an absolutely continous probability distribution $p(x)\in\mathcal{P}(\mathbb{R}^{d})$ and define $\alpha\stackrel{{\scriptstyle\text{def}}}{{=}}\operatorname{\mathbb{E}}_{p}\phi(x)$.
Now, for any $\theta\in\mathbb{R}^{k}$ consider an absolutely continuous probability distribution $q_{\theta}\in\mathcal{P}(\mathbb{R}^{d})$ with density:

$$ $q_{\theta}(x)=\exp(\left\langle\theta,\phi(x)\right\rangle-A(\theta)),\quad A(\theta)=\log\operatorname{\mathbb{E}}_{q_{\theta}(x)}e^{\left\langle\theta,\phi(x)\right\rangle}.$ $$

If there exists $\theta^{*}$ such that $p$ is absolutely continous w.r.t. $q_{\theta^{*}}$ and $\operatorname{\mathbb{E}}_{q_{\theta^{*}}(x)}\phi(x)=\alpha$, then

$$ $H(p)=H(q_{\theta^{*}})-\mathrm{KL}\left(p\middle\|q_{\theta^{*}}\right),$ $$

###### Corollary C.3 .

Let $X$ be a $d$-dimensional absolutely continuous random vector with probability density function $p$, mean $m$ and covariance matrix $\Sigma$.
Then

$$ $H(p)=H\left(\mathcal{N}(m,\Sigma)\right)-\mathrm{KL}\left(p\middle\|\mathcal{N}(m,\Sigma)\right),\qquad H\left(\mathcal{N}(m,\Sigma)\right)=\frac{1}{2}\log\left((2\pi e)^{d}\det\Sigma\right),$ $$

where $\mathcal{N}(m,\Sigma)$ is a Gaussian distribution of mean $m$ and covariance matrix $\Sigma$.

###### Corollary C.4 .

Let $X$ be an absolutely continuous random vector with probability density function $p$
and $\operatorname{\mathrm{supp}}X=S$, where $S$ has finite and non-zero Lebesgue measure $\mu(S)$.
Then

$$ $H(p)=H\left(\mathrm{U}(S)\right)-\mathrm{KL}\left(p\middle\|\mathrm{U}(S)\right),\qquad H\left(\mathrm{U}(S)\right)=\log\mu(S),$ $$

where $\mathrm{U}(S)$ is a uniform distribution on $S$.

Similar results can also be obtained for other members of the exponential family.
The first result ([C.3](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem3)) can be considered as a general recipe,
while the second one ([C.4](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem4)) can be useful when we have prior knowledge about the support of $X$ being restricted. Approach described in [C.2](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem2) is very flexible and can be considered as a generalization of the method used in Franzese et al. (2024).

In practice, to estimate the entropy of some probability distribution, it is sufficient to follow one of the described in LABEL:corollary:uniform_maximizes_entropy,corollary:Gaussian_maximizes_entropy. For example, if one uses Corollary [C.3](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem3):
1) estimate mean $m$ and covariance matrix $\Sigma$ using a set of data samples,
2) calculate entropy $H(\mathcal{N}(m,\Sigma))$ via the closed form expression,
3) calculate the KL divergence $\mathrm{KL}\left(p\middle\|\mathcal{N}(m,\Sigma)\right)$ via learning two conditional diffusion bridges models and utilizing our estimator ([17](https://arxiv.org/html/2502.01383v3#A3.E17)) from [C.1](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem1).

###### Theorem C.2 (Theorem 6.7 in Kappen ( 2024 ) ) .

###### Corollary C.3 .

###### Corollary C.4 .

### C.3 Entropy estimation experiments

We conduct entropy estimation experiments using our method proposed in [Section C.2](https://arxiv.org/html/2502.01383v3#A3.SS2). We estimate the entropy of high-dimensional ($D=10$) exponential and uniform distributions $\pi$ with varying distribution parameters. [Corollary C.3](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem3) (Gaussian) method is used for the both distributions and [Corollary C.4](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem4) (Uniform) method for the uniform distribution, since is has finite support. For each of the setups we prepare the dataset of 8000 training and 2000 test samples. Additionally we explore different reference distributions $p(x_{0})$, see [Theorem C.1](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem1), for the $\mathrm{KL}\left(\cdot\middle\|\mathcal{N}(m,\Sigma)\right)$ or $\mathrm{KL}\left(\cdot\middle\|U(S)\right)$ estimation. We explored four reference distributions $p(x_{0})$,: low ($\epsilon=0.1$), medium ($\epsilon=1$), and high ($\epsilon=10$) variance Gaussians (all with zero mean), and a ”data Gaussian” with mean and standard deviation derived from the training data.

Figure: Figure 4: Comparison of our InfoBridge combined with different methods [Corollary C.3](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem3) (Gaussian) and [Corollary C.4](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem4) (Uniform) and with different reference distributions $p(x_{0})$ on the entropy estimation task. Along $x$ axes is the ground truth entropy $H(X)$, along $y$ axes is the entropy estimate $\hat{H}(X)$. Each configuration is ran once.
Refer to caption: figures/entropy_exp.png

One can observe that InfoBridge is capable of entropy estimation on practice. Versions with low/medium variance gaussians generally show superior performance to high variance gaussian and data gaussian.
In the case of uniform distribution ([Corollary C.4](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem4)) method performs a lot better, suggesting that it is preferable in the cases when it can be applied (restricted support distributions).

### C.4 Interaction information

Here we propose the generalization of InfoBridge for the interaction information estimation, which is the generalization of mutual information for more than two random variables Rosas et al. (2019). Interaction information for random variables $X_{0},X_{1},X_{2}$ is defined by:

$$ $\displaystyle I(X_{0};X_{1};X_{2})\stackrel{{\scriptstyle\text{def}}}{{=}}I(X_{0};X_{1})-I(X_{0};X_{1}|X_{2})=$ $\displaystyle=\mathrm{KL}\left(\Pi_{X_{0},X_{1}}\middle\|\Pi_{X_{0}}\otimes\Pi_{X_{1}}\right)-\mathrm{KL}\left(\Pi_{X_{0},X_{1}|X_{2}}\middle\|\Pi_{X_{0}|X_{2}}\otimes\Pi_{X_{1}|X_{2}}\right).$ (20) $$

This definition can be generalized for more random variables in a similar way. Both MI information terms can be estimated using Theorem [4.1](https://arxiv.org/html/2502.01383v3#S4.Thmtheorem1) and our practical algorithm InfoBridge. Applications of interaction information include neuroscience Bounoua et al. (2024b), climate models Runge et al. (2019), econometrics Dosi & Roventini (2019).

### C.5 Mutual Information between random variables of different dimensionality

Here we outline the ways to estimate the mutual information between variables of different dimension. In turn this allows to calculate the mutual information between random variables from different modalities, e.g., text and images.

- •
Zero or noise padding. One can embed the lower dimensional variable into the higher dimensional space by padding the spare dimensions with zeros or noise. Such padding does not affect the mutual information, as (1) the padding can be decoupled from the original random variable, and (2) it remains uncorrelated with the other random variables.
- •
Employ the general KL estimation result. One can employ our general KL estimation result, i.e., Theorem [C.1](https://arxiv.org/html/2502.01383v3#A3.Thmtheorem1). There, we propose to add a third distribution, e.g., a point, normal distribution, or distribution of images, to start our two diffusion bridge matching models from. One can set $\pi_{1}(x)=q(x_{1},x_{2})$ and $\pi_{2}(x)=q_{1}(x_{1})q_{2}(x_{2})$, where $x_{1}$ and $x_{2}$ random variables of different dimension, i.e., $x_{1}\in\mathbb{R}^{d_{1}}$ and $x_{2}\in\mathbb{R}^{d_{2}}$.
- •
Autoencoder-based latent alignment. Another approach is to embed both random variables into a common latent space via autoencoders. Since a properly trained autoencoder preserves mutual information Butakov et al. (2024b), MI estimation can then be carried out in this shared latent space.

## Appendix D Additional discussion on method

### D.1 Conceptual comparison with MINDE

While MINDE also utilizes diffusion models for the MI estimation task, our method is built with totally different diffusion processes in mind. Utilization of regular diffusion generative modeling processes in MINDE has several disadvantages that our method particularly addresses.

First, by leveraging finite-time bridge processes instead of infinite-time diffusion (which in practice is approximated with a finite-time horizon), we provide an unbiased estimator. In particular, one can see that the MINDE MI estimator  ([9](https://arxiv.org/html/2502.01383v3#S3.E9)) has an intractable bias term in it, i.e., $\text{KL}(q^{A}_{T}\|q^{B}_{T})$, while our MI estimator  ([11](https://arxiv.org/html/2502.01383v3#S4.E11)) is exact.

Second, MINDE treats MI estimation as a generative modeling task, learning drifts to transform $\mathcal{N}(0,I)$ into $\Pi_{X_{1}}$ or $\Pi_{X_{1}\mid X_{0}}$. In contrast, our method frames it it as a domain transfer task, directly transforming input marginal $\Pi_{X_{0}}$ to distributions $\Pi_{X_{1}\mid X_{0}}$ or $\Pi_{X_{1}}$. This lead to two consequences:

- •
In domain transfer the diffusion bridge process is easier to learn, since it both starts and ends at data. This is a contrast to complex generation diffusion process that starts at noise and ends at data. The diffusion bridge approaches for domain transfer with diffusion process starting at data are known to be superior to conditional diffusion models Liu et al. (2023); Zhou et al. (2024).
- •
In our method each of the diffusion process starting at some particular point $x_{0}$ is a Schrödinger Föllmer process Vargas et al. (2023), which is a special instance of Schrodinger Bridge Bortoli et al. (2024); Vargas et al. (2021). Trajectories of Schrodinger Bridge processes with small volatility coefficient $\epsilon$ require less energy for translation and are relatively ”straight”. As a result, these trajectories are easy to follow, are expected to deliver less numerical error when dealt with and are easier to learn.

To empirically support the mentioned superiorities of our domain translation trajectories over MINDE’s data generation trajectories, we compute the variance of MI estimation, which is based on trajectory Monte Carlo integration (Eq [11](https://arxiv.org/html/2502.01383v3#S4.E11) for our method, Eq 19 Franzese et al. (2024) for MINDE). Specifically, we evaluate MINDE-C and our method on the image-based benchmark (\wasyparagraph [5.2](https://arxiv.org/html/2502.01383v3#S5.SS2)) using 16×16 rectangular images. We vary the number of sample tuples $\{x_{0}^{i},x_{t}^{i},x_{T}^{i}\}_{i=1}^{N}$ for MINDE-C and $\{x_{0}^{i},x_{t}^{i},x_{1}^{i}\}_{i=1}^{N}$ for our method and run 10 MI estimates per setting, and report the mean $\pm$ standard deviation in the Table [3](https://arxiv.org/html/2502.01383v3#A4.T3).

**Table 3: Comparison of MI estimation accuracy across different sample sizes.**
| Method \ N samples | 128 | 1024 | 4096 | 16384 | 65536 | 131072 |
| --- | --- | --- | --- | --- | --- | --- |
| InfoBridge (ours) | 9.79 $\pm$ 0.72 | 9.84 $\pm$ 0.34 | 9.9 $\pm$ 0.1 | 9.84 $\pm$ 0.06 | 9.86 $\pm$ 0.028 | 9.84 $\pm$ 0.028 |
| MINDE | 12.20 $\pm$ 6.65 | 11.46 $\pm$ 3.32 | 10.62 $\pm$ 3.94 | 12.59 $\pm$ 2.53 | 11.17 $\pm$ 1.38 | 10.67 $\pm$ 0.75 |

Our method achieves 1–2 orders of magnitude lower standard deviation in MI estimates compared to MINDE. Even with 1000$\times$ more samples, MINDE only approaches our accuracy. This is due to its reliance on complex noise-to-data trajectories, which make its estimates highly volatile and unreliable, unlike our more stable data-to-data paths. As shown in Figure [1](https://arxiv.org/html/2502.01383v3#S5.F1), MINDE consistently exhibits the largest and most unstable confidence intervals, undermining the reliability and precision of its MI estimates. Furthermore, every sample requires inference of a neural network and hence the reliable estimation of MI by MINDE requires several orders of magnitude more computational resources.

### D.2 Ablation study

Here we vary different important hyperparameters of our InfoBridge method.

Volatility coefficient $\epsilon$.
We evaluate different volatility coefficients on the image benchmark (\wasyparagraph[5.2](https://arxiv.org/html/2502.01383v3#S5.SS2)) and protein embeddings data (\wasyparagraph[5.3](https://arxiv.org/html/2502.01383v3#S5.SS3)). We run the image benchmark experiment on Gaussian 16x16 setting with ground truth MI=10 and the full protein embeddings data experiment with one random seed and report mean MAE error for the last 5 measurements during training, in both of setups. Results are presented in Table [4](https://arxiv.org/html/2502.01383v3#A4.T4) and Table [5](https://arxiv.org/html/2502.01383v3#A4.T5). One can see that MAE relatively stable for all the setups, while the optimal $\epsilon$ values is different for the both experiments.

One can notice that $\epsilon=1$ works relatively well on both of the setups. Furthermore, $\epsilon=1$ performs well on all the experiments we have done, we use it for the three out of four experiments in the main part of the paper and show its competitiveness on the high MI experiment in Appendix [E.4](https://arxiv.org/html/2502.01383v3#A5.SS4). In that light we call $\epsilon=1$ a robust choice suitable for all the data.

**Table 4: Mean Average Error (MAE) values on Image data benchmark (Gaussian 16x16) with ground truth MI=10 experiment for the different $\epsilon$ values. Each configuration (expect for $\epsilon=1$) is ran one time and 5 last MI measurements during training are averaged to produce the shown result. The best configuration is bolded and configuration used in the main part of the paper is underlined.**
| $\epsilon$ | 0.001 | 0.003 | 0.01 | 0.03 | 0.1 | 0.3 | 1 | 3 | 10 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MAE $\downarrow$ | 2.09 | 1.79 | 1.53 | 1.53 | 1.72 | 1.79 | 2.46 | 2.82 | 6.45 |

**Table 5: Mean Average Error (MAE) values on ProtTran5 experiment, averaged over all the ground truth MI values, for the different $\epsilon$ values. Each configuration (expect for $\epsilon=1$) is ran one time and 5 last MI measurements during training are averaged to produce the shown result. The best configuration is bolded and configuration used in the main part of the paper is underlined.**
| $\epsilon$ | 0.1 | 0.3 | 1 | 3 | 10 |
| --- | --- | --- | --- | --- | --- |
| MAE $\downarrow$ | 0.775 | 0.232 | 0.04 | 0.089 | 0.129 |

Number of parameters in a neural network. The width of the Unet Ronneberger et al. (2015) neural network architecture used in InfoBridge on Image benchmark experiments is varied to yield different number of parameters of a neural network. Numbers of base filters, $256,64,32,16,8$, yield number of parameters, $27M,1.7M,428K,108k,28k$. Similar to the ablation study on $\epsilon$ each configuration, except for $256$ number of filters and $27M$ parameters is run once, and the final five mutual information (MI) measurements during training are averaged to obtain the reported results. Outcomes are presented in [Table 6](https://arxiv.org/html/2502.01383v3#A4.T6). As one can see our InfoBridge is robust w.r.t. neural network architecture.

**Table 6: Comparison of the InfoBridge with different neural network architectures on the Image data benchmark (16 $\times$ 16 Gaussians setup) with ground truth MI=7.5. Each configuration is ran one time and 5 last MI measurements during training are averaged to produce the shown result. The configuration used in the main experimental part is underlined.**
| N Params | GT MI=7.5 (mean $\pm$ std) |
| --- | --- |
| 28k | $7.56\pm 0.3$ |
| 108k | $6.67\pm 0.15$ |
| 428k | $6.67\pm 0.13$ |
| 1.7M | $6.52\pm 0.07$ |
| 27M | $6.59\pm 0.17$ |

## Appendix E Experimental supplementary

In this section, additional experimental results and experimental details are described.

### E.1 Low-dimensional benchmark

#### Additional results.

In Table [14](https://arxiv.org/html/2502.01383v3#A5.T14) we present the results of low-dimensional benchmark Czyż et al. (2023) with precision of 0.01 nats.

#### Experimental details.

The benchmark implementation is taken from the official repository:

[https://github.com/cbg-ethz/bmi](https://github.com/cbg-ethz/bmi)

#### Info Bridge.

Neural networks are taken of almost the same architecture as in Franzese et al. (2024), which is MLP with residual connections and time embedding. Additional input $s$ described in (\wasyparagraph [4.2](https://arxiv.org/html/2502.01383v3#S4.SS2.SSS0.Px1)) and is processed the same way as time input. Number of parameters is taken depending on a dimensionality of the problem, see Table [7](https://arxiv.org/html/2502.01383v3#A5.T7). Exponential Moving Average as a widely recognized training stabilization method was used with decay parameter of $0.999$. For all the problems neural networks are trained during 100k iterations with $\epsilon=1$, batch size $512$, lr $0.0003$. Mutual Information is estimated by Algorithm [2](https://arxiv.org/html/2502.01383v3#algorithm2) with $N$ pairs of samples $\{x_{t}^{i},x_{0}^{i},t^{i}\}_{i=1}^{N}$, where $N$ is equal to the number of test samples times $10$, i.e., $100$k.

#### NVF and JVF.

The implementation of NVF and JVF is taken from the official repository:

[https://github.com/calebdahlke/FlowMI](https://github.com/calebdahlke/FlowMI)

and used with all the provided by authors default hyperparameters for the benchmark. The methods are ran on CPU and execution takes less then two hours for each benchmark problem run.

**Table 7: Neural networks hyperparametes for low-dimensional Czyż et al. (2023) benchmark. “Dim” - dimensionality of a MI estimation problem, “Filters” – number of filters in MLP, “Time Embed” – number filters in time embedding module, “Parameters” – number of overall neural networks parameters.**
| Dim | Filters | Time Embed | Parameters |
| --- | --- | --- | --- |
| $\leq$5 | 64 | 64 | 43K |
| 25 | 128 | 128 | 176K |
| 50 | 256 | 256 | 699K |

### E.2 Image data benchmark

#### Random seeds.

We run InfoBridge, MIENF, and MINDE with 3 random seeds, and all other methods with 5 seeds per experimental setup. In [Figure 1](https://arxiv.org/html/2502.01383v3#S5.F1) and [Table 2](https://arxiv.org/html/2502.01383v3#S5.T2), we report the mean performance across seeds, along with standard deviations and confidence intervals computed from the seed-wise variability.

#### Experimental details for Info Bridge.

The implementation of image data benchmark Butakov et al. (2024a) was taken from the official repository:

[https://github.com/VanessB/mutinfo](https://github.com/VanessB/mutinfo)

Following authors of the benchmark, Gaussian images were generated with all the default settings, Rectangle images were also generated with all the default settings, but minimum size of rectangle is $0.2$ to avoid singularities.
All the covariance matrices for the distributions defining the mutual information in the benchmark were generated without randomization of component-wise mutual information, but with randomization of the off-diagonal blocks of the covariance matrix.

To approximate the drift coefficient of diffusion U-Net Ronneberger et al. (2015) with time, conditional neural networks were used, special input $s$ was processed as time input. For all the tests, neural networks were the same and had $2$ residual layers per U-net block with $256$ base channels, positional timestep encoding, upscale and downscale blocks consisting of two resnet blocks, one with attention and one without attention. The number of parameters is $\sim$27M. During the training, $100$k gradient steps were made with batch size of 64 and learning rate $0.0001$. Exponential moving average was used with decay rate $0.999$. Mutual Information was estimated by Algorithm [2](https://arxiv.org/html/2502.01383v3#algorithm2) with $N$ pairs of samples $\{x_{t}^{n},x_{0}^{n},t^{n}\}_{i=n}^{N}$, where $N$ is equal to the number of test samples, i.e., $10$k. Nvidia A100 was used for the InfoBridge training. Each run (one seed) took around $6$ and $18$ GPU-hours for the $16\times 16$ and $32\times 32$ image resolution setups, respectively.

#### Visualization of learned distribution.

Figure [5](https://arxiv.org/html/2502.01383v3#A5.F5) presents samples generated from the learned distributions alongside ground truth samples from the dataset. It is evident that the samples from both $\pi_{\theta}(x_{1}|x_{0})$ and $\pi^{\text{ind}}_{\theta}(x_{1}|x_{0})$ closely resemble those drawn from $\pi(x_{0})$, indicating that the marginal distribution of $x_{0}$ is well approximated by the learned models.

Figure: (a) Gaussians data samples $x_{0}\sim\pi(x_{0})$
Refer to caption: figures/paired_gaussian_32_mi_5_eps_1_predict_vector_field_input_x_5.png

#### Experimental details for other methods.

In this part, we provide additional experimental details regarding other methods featured in [Figure 1](https://arxiv.org/html/2502.01383v3#S5.F1).
We report the NN architectures used for neural estimators in [Table 8](https://arxiv.org/html/2502.01383v3#A5.T8).

**Table 8: The NN architectures used to conduct the tests with images in [5](https://arxiv.org/html/2502.01383v3#S5).**
| NN | Architecture |
| --- | --- |
| GLOW,<br>$16\times 16$ ($32\times 32$)<br>images | $\times 1$:<br>$4$ ($5$) splits, $2$ GLOW blocks between splits,<br>$16$ hidden channels in each block, leaky constant $=0.01$<br>${}^{\times 2\;\text{in parallel}}$<br>$\times 1$:<br>Orthogonal projection linear layer${}^{\times 2\;\text{in parallel}}$ |
| Autoencoder,<br>$16\times 16$ ($32\times 32$)<br>images | $\times 1$:<br>Conv2d(1, 4, ks=3), BatchNorm2d, LeakyReLU(0.2), MaxPool2d(2)<br>$\times 1$:<br>Conv2d(4, 8, ks=3), BatchNorm2d, LeakyReLU(0.2), MaxPool2d(2)<br>$\times 2(3)$:<br>Conv2d(8, 8, ks=3), BatchNorm2d, LeakyReLU(0.2), MaxPool2d(2)<br>$\times 1$:<br>Dense(8, dim), Tanh, Dense(dim, 8), LeakyReLU(0.2)<br>$\times 3(4)$:<br>Upsample(2), Conv2d(8, 8, ks=3), BatchNorm2d, LeakyReLU(0.2)<br>$\times 1$:<br>Upsample(2), Conv2d(8, 4, ks=3), BatchNorm2d, LeakyReLU(0.2)<br>$\times 1$:<br>Conv2d(4, 1, ks=3), BatchNorm2d, LeakyReLU(0.2) |
| Critic NN,<br>$16\times 16$ ($32\times 32$)<br>images | $\times 1$:<br>[Conv2d(1, 16, ks=3), MaxPool2d(2), LeakyReLU(0.01)]${}^{\times 2\;\text{in parallel}}$<br>$\times 1(2)$:<br>[Conv2d(16, 16, ks=3), MaxPool2d(2), LeakyReLU(0.01)]${}^{\times 2\;\text{in parallel}}$<br>$\times 1$:<br>Dense(256, 128), LeakyReLU(0.01)<br>$\times 1$:<br>Dense(128, 128), LeakyReLU(0.01)<br>$\times 1$:<br>Dense(128, 1) |

MINDE. We use the official implementation of MINDE:

[https://github.com/MustaphaBounoua/minde](https://github.com/MustaphaBounoua/minde)

As a neural network, we adapt same U-net architecture as for our InfoBridge in image data experiments with approximately the same number of parameters. Training procedure hyperparameters do match the InfoBridge, i.e., 100k gradient steps with batch size 64 and learning rate 0.0001, exponential moving average with 0.999 decay rate. Mutual Information was estimated using 10 samples of $\{t^{i},x_{t}^{i}\}$ per each pair $\{x_{0}^{i},x_{1}^{i}\}$ in the test dataset. Nvidia A100 was used to train the models. Each run (one seed) took around 2 and 5 GPU-hours for the 16$\times$16 and 32$\times$32 image resolution setups.

MIENF. Under MIENF we consider gaussian-based $\mathcal{N}$-MIENF.
This method is based on bi-gaussianization of the input data via a Cartesian product of learnable diffeomorphisms Butakov et al. (2024a).
Such approach allows for a closed-form expression to be employed to estimate the MI.

With only minor stability-increasing changes introduced, we adopt the Glow Kingma & Dhariwal (2018) flow network architecture from Butakov et al. (2024a), which is also reported in [8](https://arxiv.org/html/2502.01383v3#A5.T8) (“GLOW”).
We used the normflows package (Stimper et al., 2023) to implement the model.
Adam (Kingma & Ba, 2017) optimizer was used to train the network
on $10^{5}$ images with a batch size $512$, and the learning rate decreasing from $5\cdot 10^{-4}$ to $10^{-5}$ geometrically.
For averaging, we used $3$ different seeds.
Nvidia A100 was used to train the flow models.
Each run (one seed) took no longer than four GPU-hours to be completed.

KSG. Kraskov-Stögbauer-Grassberger Kraskov et al. (2004) mutual information estimator is a well-known $k$-NN non-parametric method, which is very similar to unweighted Kozachenko-Leonenko estimator Kozachenko & Leonenko (1987).
This method employs distances to $k$-th nearest neighbors to approximate the pointwise mutual information, which is then averaged.

We used $k=1$ (one nearest neighbour) for all the tests.
The number of samples was $10^{5}$ for Gaussian images and $10^{4}$ for images of rectangles (we had to lower the sampling size due to degenerated performance of the metric tree-based $k$-NN search in this particular setup).
A single core of AMD EPYC 7543 CPU was used for nearest neighbors search and MI calculation.
Each run (one seed) took no longer than one CPU-hour to be completed.

MINE, NWJ, InfoNCE. These discriminative approaches are fundamentally alike: each method estimates mutual information by maximizing the associated KL-divergence bound:

$$ $I(X_{0};X_{1})=\mathrm{KL}\left(\pi({x_{0},x_{1}})\middle\|\pi(x_{0})\pi(x_{1})\right)\geqslant\sup_{T\colon\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R}}\mathrm{F}[T(x_{0},x_{1})],$ (21) $$

where $T$ is measurable, and $\mathrm{F}$ is some method-specific functional.
In practice, $T$ is approximated via a neural network, with the right-hand-side in ([21](https://arxiv.org/html/2502.01383v3#A5.E21)) being used as the loss function.

Motivated by this similarity, we use a nearly identical experimental framework to assess each approach within this category.
To approximate $T$ in experiments with synthetic images, we adopt the critic NN architecture from Butakov et al. (2024a), which we also report in [Table 8](https://arxiv.org/html/2502.01383v3#A5.T8) (“Critic NN”).

The networks were trained via Adam optimizer
on $10^{5}$ images with a learning rate $10^{-3}$, a batch size $512$ (with InfoNCE being the only exception, for which we used batch size $256$ for training and $512$ for evaluation due to memory constraints), and MAE loss for $10^{5}$ steps.
For averaging, we used $5$ different seeds.
Nvidia A100 was used to train the models.
In any setup, each run (one seed) took no longer than two GPU-hours to be completed.

AE-WKL Baseline

Here we include an additional baseline Auto Encoder + Weighted Kozachenko-Leonenko (AE-WKL) for the image based benchmark, which we exclude from the main part of the paper due to instability w.r.t. hyperparameters and huge inductive bias.

The idea of leveraging lossy compression to tackle the curse of dimensionality and provide better MI estimates is well-explored in the literature Goldfeld & Greenewald (2021); Goldfeld et al. (2022); Tsur et al. (2023); Fayad & Ibrahim (2023); Greenewald et al. (2023); Butakov et al. (2024b).
In our work, we adopt the non-linear compression setup from Butakov et al. (2024b), which employs autoencoders for data compression and weighted Kozachenko-Leonenko method Berrett et al. (2019) for MI estimation in the latent space.

However in this approach non-parametric MI estimator (WKL) approach heavily relies on the autoencoder latent space.

First, prior works suggest that non-parametric MI estimators are highly prone to the curse of dimensionality compared to NN-based approaches Goldfeld et al. (2020); Czyż et al. (2023); Butakov et al. (2024a) and hence WKL performance seriously deteriorates with growth of latent space dimensionality. In particular, Figures 2 and 3 in Goldfeld et al. (2020) and Table 1 in Butakov et al. (2024a) indicate that weighted Kozachenko-Leonenko estimator fails to yield reasonable estimates at all if the dimensionality reaches certain threshold.

Second, the performance of estimator depends on the amount of information preserved in the latent space by the autoencoder. If the dimension of latent space is less than dimension of the data manifold, the performance of the method should deteriorate as well.

We validate the AE-WKL approach with different dimensionality of autoencoder latent space in the Gaussian images setup. We increase the latent space dimensionality from $d=2$ (which is the intrinsic dimensionality of the dataset in question) to $d\in\{4,5,8\}$. We report our results in [Figure 6](https://arxiv.org/html/2502.01383v3#A5.F6). One can see while with $d=2$ latent space (which is the true latent dimensionality of gaussian images) AE-WKL approach achieves remarkable results, but with growth of dimension to $d\in\{4,5,8\}$ the performance starts to deteriorate with $d=8$ even leading to exploded estimates.

That leaves autoencoder approaches combined with non parametric MI estimators highly dependent on the successful latent space choice. This choice is trivial on the image based benchmark, where the dimensionality of data manifold is known by construction, however such information is usually unknown in real world scenario, which makes approaches like AE-WKL highly unreliable.

The autoencoders were trained using Adam optimizer
on $10^{5}$ images with a batch size $512$, a learning rate $10^{-3}$ and MAE loss for $10^{4}$ steps.
For averaging, we used $5$ different seeds.
Nvidia A100 was used to train the autoencoder model.
Each run (one seed) took no longer than one GPU-hour to be completed.

Figure: (a) $16\times 16$ images (Gaussians)

### E.3 Protein embeddings data

**Table 9: The best result is bolded and underlined and second best result is bolded.**
| Method | InfoBridge | MINE | InfoNCE | KSG | MINDE-C | MINDE-J |
| --- | --- | --- | --- | --- | --- | --- |
| MAE $\downarrow$ | 0.04 | 0.223 | 0.237 | 0.116 | 9.29 | 1342 |
| MAE Std | 0.01 | 0.017 | 0.007 | - | 1.1 | 57 |

#### Data.

We use the UniProt database proteins, their embeddings from prottrans_t5_xl_u50 Elnaggar et al. (2021) model and download them from [https://github.com/ggdna/latent-mutual-information](https://github.com/ggdna/latent-mutual-information). All proteins longer
than $12\cdot 10^{3}$ residues are excluded. These embeddings are then unit variance normalized. The final number of each protein embeddings (samples) used for training is $20641$ and the dimensionality is $1024$.

Methods implementation.

InfoBridge. Hyperparameters are the same as for low-dimensional benchmark Appendix [E.1](https://arxiv.org/html/2502.01383v3#A5.SS1), MLP has $\sim 900k$ parameters, weight decay is $0.001$, dropout is $0.2$ and the model is trained for $100$ epochs. Results for InfoBridge averaged over last 5 MI estimations during training.

MINDE. We leverage the official implementation of MINDE [https://github.com/MustaphaBounoua/minde](https://github.com/MustaphaBounoua/minde) in the configuration used for the low-dimensional benchmark, with $\sim 900k$ parameters MLP, lr $1e-4$, batch size of $256$, ema decay of $0.999$ and train model for $100$ epochs.

MINE, InfoNCE. These estimators are taken from the bmi library [https://github.com/cbg-ethz/bmi](https://github.com/cbg-ethz/bmi) with all the default parameters.

KSG. Implementation is taken from [https://github.com/ggdna/latent-mutual-information](https://github.com/ggdna/latent-mutual-information) and used with all the default parameters on $20000$ training samples.

All the neural methods were trained on Nvidia A100 and took less then one GPU-hour to complete. KSG estimation algorithm took approximately one hour to complete.

### E.4 High mutual information experiments

#### Half Cube transform.

We adopt the half-cube transform for Gaussian random variables as introduced in Czyż et al. (2023):

$$ $f(x)=x\sqrt{|x|}.$ $$

This transform lengthens the tails of Gaussian random variables, while preserving the MI due to $f(x)$ transform being reversible.

#### Additional experimental results.

In addition to the high MI experiment in the main text (\wasyparagraph [5.4](https://arxiv.org/html/2502.01383v3#S5.SS4)) here we present extended comparisons with other baselines: see Table [10](https://arxiv.org/html/2502.01383v3#A5.T10) for Gaussian random variables, Table [11](https://arxiv.org/html/2502.01383v3#A5.T11) for Gaussian random variables with half cube transform, Table [12](https://arxiv.org/html/2502.01383v3#A5.T12) for Correlated Uniform random variables, Table [13](https://arxiv.org/html/2502.01383v3#A5.T13) for Smoothed Uniform random variables. Overall our method still is the most reliable and accurate of the bunch.

As shown, our method performs well with both $\epsilon=1$ and $\epsilon=0.01$, with the latter providing slightly better results. For this reason, we adopt $\epsilon=0.01$ in the main text comparisons.

One can see that MINDE-J offers the closest to the ground truth MI in many setups, but we note that it drastically overestimates the MI in the d=160, MI=80 setup across all the distribution types. Such behaviour is typical for MINDE-J on complex datasets also, see (\wasyparagraph [5.2](https://arxiv.org/html/2502.01383v3#S5.SS2)) and (\wasyparagraph [5.3](https://arxiv.org/html/2502.01383v3#S5.SS3)) that is why we leave it out in the main text comparisons in (\wasyparagraph [5.4](https://arxiv.org/html/2502.01383v3#S5.SS4)).

**Table 10: Mutual Information estimates across increasing dimensions and mutual information levels for the Gaussian random variables. Best MI estimation is bolded and second best is underlined. ^† stands for method shown in the high MI experiment in the main text. Empty cell stands for method explosion.**
| Method | d=20, MI=10 | d=40, MI=20 | d=80, MI=40 | d=160, MI=80 |
| --- | --- | --- | --- | --- |
| InfoBridge (ours, $\epsilon=1$) | 9.85 | 19.99 | 39.31 | 78.86 |
| InfoBridge (ours, $\epsilon=0.01$)^† | 10.35 | 21.79 | 41.92 | 79.76 |
| MINDE-C^† | 9.59 | 16.96 | 35.51 | 56.27 |
| MINDE-J | 10.23 | 20.02 | 42.11 | 450 |
| MINE^† | 6.51 | 8.43 | 7.76 | 8.08 |
| InfoNCE^† | 5.14 | 5.40 | 4.76 | 3.80 |
| fDIME (KL, J) | 5.51 |  | 2.49 | 0.0 |
| fDIME (HD, J) | 5.24 | 3.66 | 1.14 | 0.0 |
| fDIME (GAN, J)^† | 6.88 | 6.71 | 3.54 | 0.0 |
| fDIME (KL, D) | 3.71 | 4.06 | 0.22 | 0.0 |
| fDIME (HD, D) | 4.01 | 4.24 | 0.23 | 0.0 |
| fDIME (GAN, D) | 4.5 | 4.86 | 0.32 | 0.0 |

**Table 11: Mutual Information estimates across increasing dimensions and mutual information levels for the Half Cube transformed Gaussian random variables. Best MI estimation is bolded and second best is underlined. ^† stands for method shown in the high MI experiment in the main text.**
| Method | d=20, MI=10 | d=40, MI=20 | d=80, MI=40 | d=160, MI=80 |
| --- | --- | --- | --- | --- |
| InfoBridge (ours, $\epsilon=1$) | 9.07 | 18.23 | 34.35 | 61.63 |
| InfoBridge (ours, $\epsilon=0.01$)^† | 9.92 | 20.69 | 39.6 | 84.24 |
| MINDE-C^† | 8.31 | 14.69 | 33.64 | 40.63 |
| MINDE-J | 9.28 | 16.49 | 43.3 | 308 |
| MINE^† | 3.18 | 3.0 | 3.28 | 3.74 |
| InfoNCE^† | 4.11 | 4.7 | 3.84 | 2.74 |
| fDIME (KL, J) | 7.52 | 6.84 | 11.3 | 8.18 |
| fDIME (HD, J) | 6.56 | 11.59 | 10.77 | 10.73 |
| fDIME (GAN, J)^† | 9.7 | 14.7 | 20.84 | 13.91 |
| fDIME (KL, D) | 5.41 | 6.82 | 6.67 | 6.76 |
| fDIME (HD, D) | 5.71 | 7.66 | 7.99 | 8.74 |
| fDIME (GAN, D) | 7.83 | 10.4 | 11.49 | 10.86 |

**Table 12: Mutual Information estimates across increasing dimensions and mutual information levels for the Correlated Uniform random variables. Best MI estimation is bolded and second best is underlined. ^† stands for method shown in the high MI experiment in the main text.**
| Method | d=20, MI=10 | d=40, MI=20 | d=80, MI=40 | d=160, MI=80 |
| --- | --- | --- | --- | --- |
| InfoBridge (ours, $\epsilon=1$) | 7.95 | 15.15 | 28.46 | 41.9 |
| InfoBridge^† (ours, $\epsilon=0.01$) | 8.65 | 16.93 | 33.69 | 57.73 |
| MINDE-C^† | 8.03 | 15.14 | 30.78 | 55.91 |
| MINDE-J | 8.8 | 17.56 | 34.59 | 539 |
| MINE^† | 6.12 | 7.9 | 7.92 | 7.11 |
| InfoNCE^† | 4.8 | 4.48 | 3.71 | 3.42 |
| fDIME (KL, J) | 7.6 |  | 6.1 | 0.99 |
| fDIME (HD, J) | 8.28 | 8.84 | 4.8 | 0.19 |
| fDIME (GAN, J)^† | 9.2 | 10.33 | 8.75 | 1.56 |
| fDIME (KL, D) | 5.62 | 5.1 | 2.71 | 0.1 |
| fDIME (HD, D) | 6.6 | 7.02 | 2.14 | 0.41 |
| fDIME (GAN, D) | 6.49 | 7.88 | 3.79 | 0.0 |

**Table 13: Mutual Information estimates across increasing dimensions and mutual information levels for the Smoothed Uniform random variables. Best MI estimation is bolded and second best is underlined. ^† stands for method shown in the high MI experiment in the main text.**
| Method | d=20, MI=10 | d=40, MI=20 | d=80, MI=40 | d=160, MI=80 |
| --- | --- | --- | --- | --- |
| InfoBridge (ours, $\epsilon=1$) | 6.4 | 12.83 | 24.21 | 29.83 |
| InfoBridge (ours, $\epsilon=0.01$)^† | 7.3 | 14.63 | 29.25 | 50.28 |
| MINDE-C^† | 6.53 | 13.23 | 24.98 | 44.2 |
| MINDE-J | 7.63 | 14.39 | 31.15 | 511 |
| MINE^† | 5.56 | 7.45 | 7.91 | 7.17 |
| InfoNCE^† | 3.84 | 3.67 | 3.34 | 3.14 |
| fDIME (KL, J) | 6.15 | -74 | 3.09 | 0.0 |
| fDIME (HD, J) | 5.9 | 8.12 | 3.95 | 0.0 |
| fDIME (GAN, J)^† | 7.75 | 8.49 | 4.75 | 0.0 |
| fDIME (KL, D) | 4.91 | 4.17 | 0.32 | 0.0 |
| fDIME (HD, D) | 4.59 | 4.82 | 0.37 | 0.0 |
| fDIME (GAN, D) | 5.65 | 5.65 | 0.33 | 0.0 |

#### Experimental Details

We use $100$k train samples dataset and $10$k test dataset for all the methods. Datasets are generated using mutinfo library github repository:

[https://github.com/VanessB/mutinfo](https://github.com/VanessB/mutinfo).

InfoBridge, MINDE, MINE, InfoNCE implementations were taken with exactly the same hyperparameters as for low-dimensional benchmark experiment, see Appendix [E.1](https://arxiv.org/html/2502.01383v3#A5.SS1).

fDIME implementation was taken from the official github repository:

[https://github.com/tonellolab/fDIME](https://github.com/tonellolab/fDIME),

with all the default hyperparameters. KL, HD, GAN stand for different divergences and J, D stand for Joint and Derranged architectures of fDIME.

Figure: Table 14: Mean estimate over $10$ seeds using $10\cdot 10^{4}$ test samples compared each against the ground-truth. Size of train dataset for every neural method is 100k. All the methods for comparison, except for NVF and JVF Dahlke & Pacheco, with InfoBridge were taken from Franzese et al. (2024). List of abbreviations ( Mn: Multinormal, St: Student-t, Nm: Normal, Hc: Half-cube, Sp: Spiral)

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 4
presentation: 2
questions: The typical condition for the application of Girsanov's theorem is that of Novikov. How do you suggest relaxing it to use InfoBridge with long tailed distributions as suggested in section 6?
soundness: 4
strengths: + The main contribution is much sought after: An unbiased, low variance estimate of mutual information has impact across fields. 

+ The paper's literature review is apt and does a great job motivating the problem.

+ The paper is full of illuminating insights tying information theory, diffusion models and stochastic calculus.
summary: This paper propose an unbiased estimator of mutual information constructed on bridge matching.
The evaluation is solely conducted by measuring the discrepancy from a ground truth mutual information.
weaknesses: + The evaluation is solely conducted by measuring the discrepancy from a ground truth mutual information. This is sound an necessary but also limited by itself. The paper would be more convincing if the proposed estimate of the mutual information were 
used as an objective, or its learned representations evaluated on downstream tasks.

+ The work could be made much more accessible. The background section runs two full pages, and the authors only introduce their method in page 5. We suggest the more direct route of introducing a simple example right after the introduction, and introduce the stochastic calculus concepts as needed.

+ The paper differentiate itself from MINDE[1] by 1) being unbiased while the latter is only asymptotically unbiased, 2) By presenting itself as a domain transfer task. Unfortunately, this shift in conceptual framing while thought provoking is only  indirectly evaluated through a reduction of the estimators in variance. This is a bit unfortunate as eliminating bias and reducing variance is enough to differentiate well enough from MINDE.

[1] Franzese, Giulio, Mustapha Bounoua, and Pietro Michiardi. "MINDE: Mutual information neural diffusion estimation." arXiv preprint arXiv:2310.09031 (2023).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Is MI provably independent of the choice of ϵ? in Theorem 4.1? If yes, please add a formal statement and a numeric ablation sweeping ϵ.
* What is the estimator variance as a function of t-grid resolution and batch size? Any control variates that reduce variance when t → 1 (where the (1-t)^{-1} factor amplifies noise)?  
* Does training two drifts (joint vs. independent) materially increase sample complexity vs. one-model approaches? Could a shared backbone with binary conditioning stabilize learning?  
* How would you share a NN between variables of different dimensionality?
* How does InfoBRIDGE behave on long-tailed or heavy-tailed marginals, beyond the cited remarks that classical estimators struggle there? Please add such stress tests. 
* Table 1: What are the reported metrics? Please add to caption description of the metric presented in the table.
* Figure 1: why MI estimation undershoots for Gaussians (a,b) but not for rectangles (c,d)?
* Discussion of basic implementation, training, and hyper-parameters details should be included in the main body. Include discussion of the computational cost.

Comment:

* Limitations should be discussed in the main body, not only in the appendix.
soundness: 3
strengths: * Conceptual novelty: Clear, principled decomposition of MI into a time-integral of drift differences between two bridge processes; different viewpoint than score-based MINDE.
* Methodological soundness: Builds on established conditional bridge matching; training and estimation procedures are straightforward (Alg. 1–2). 
* Practicality: Uses only joint samples and Brownian-bridge simulation (no explicit density estimation), which is attractive for complex data like images/embeddings.
summary: The authors propose InfoBRIDGE, a mutual information (MI) estimator that reframes MI estimation as a domain transfer problem between a joint distribution and the product of marginals using reciprocal processes built from mixtures of Brownian bridges. The required drifts are learned via conditional bridge matching with a neural net; training uses samples from the joint and a permuted batch to emulate independence, and evaluation computes the integral by Monte Carlo over Brownian-bridge samples. The authors argue this yields an unbiased MI estimator (given perfect drift learning) and report results on low-dimensional synthetic, image-based, high-MI benchmarks, and protein-embedding data.
weaknesses: * Unclear dependence on ϵ: The estimator formula scales with 1/(2ϵ). Please analyze or empirically demonstrate ϵ-invariance of the resulting MI and give guidance for choosing ϵ.  
* Bias/variance in practice: “Unbiased” is only in the idealized limit. Did you check the empirical calibration with confidence intervals vs. ground-truth MI? Did you check sensitivity to network size, training steps, and t-discretization?
* Independence construction: The independent process uses permuted pairs (x_0,\hat{x}_1). What is possible bias when datasets contain duplicated or highly correlated samples? Will a two-sample protocol (fresh marginal draws) materially changes results?
* Scope of experiments: The authors mention low-dimensional, image, high-MI, and protein settings; more detail is needed on metrics, baselines, and runtime/sample efficiency, especially versus strong discriminative bounds (e.g., InfoNCE variants).

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 1
questions: - What is $\delta()$ in equation (3)? It is not defined.
- In line 148 the authors refer to Problem (2). Does it refer to the second equation? That would be the MI definition. In the line below the authors write “minimizes (2)”, but they are not referring to the minimization of MI, I guess they are referring to the minimization of the long equation in line 145.
- In the pseudocode of algorithm 1 and in the code provided in the supplementary material, the authors perform a random permutation to obtain the samples from the product of marginal densities: 
y_samples_permuted = y_samples[torch.randperm(y_samples.shape[0])].
However, using such a function leads a certain number of elements of the shuffled vector to be positioned in the same initial position (before the permutation). This strategy was highlighted in f-DIME’s paper as problematic and in that specific case the authors showed that using a specific  discriminative architecture that mistake would lead to an upper bound in the MI estimate. For InfoBridge I do not see any upper bound. However, performing such a random permutation is logically incorrect and the authors should perform the experiments after correcting such a mistake. How does the performance of the proposed algorithm change after this correction? If it does not change, why? 
- Where are the experiments justifying the sentence in lines 314-315 about the additional binary input?
- Can the authors show a comparison of the computational complexity between the different algorithms used?
- Why the reference Lee and Rhee has no year?
- The discriminative models used during comparisons have the same size (number of parameters) of the generative estimators?
- Isn’t 100k train samples a very large number? How do the algorithms perform with a lower number of training samples?
- In Figure 2 the authors specify which version of MINDE they are using, but do not specify which f-DIME version. Can the authors specify it?
soundness: 2
strengths: - The proposed estimator relies on diffusion bridge models, thus leveraging a new technique for MI estimation
- The proposed generative MI estimator is unbiased, differently from MINDE which is biased
- The experimental results show the effectiveness of the proposed estimator
summary: The paper presents a novel unbiased MI estimator based on diffusion bridge models. The proposed estimator is tested on different low-dimensional and high-dimensional datasets.
weaknesses: - Although the proposed estimator leverages diffusion bridge models, which differ from diffusion models, the proposed framework appears as an extension of MINDE
- The paper is not well-written. The clarity could definitively be improved. For instance, SDE is not defined (stochastic differential equation). The equation in line 145 should be written in two lines, as it exceeds the borders of the paper. There are many typos: line 96 (conditioned its on values), line 264 (as a domain transfer task), multiple wrong references to equation (2) while I assume the authors wanted to refer to other equations. The same wrong reference to (2) happens also in line 300. Table 2 reports Mean Average Error, but I assume the authors used the Mean Absolute Error.
- In the experiments  the authors compare the proposed method with f-DIME. However, there is no reference of f-DIME in the related work section, so it should be inserted in the Discriminative estimators paragraph.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - For the experiment in Table 1, some methods do not require training. Do you only provide them with 10k test samples, or do they get access to 110k samples, i.e., the full train and test split. How would the method perform with only 10k samples for training and testing jointly?
- Do you have an intuition as of why higher MI values seem to lead to worst estimation in the image gaussian 16x16 case, for all estimators?
- “First, by leveraging finite-time bridge processes instead of infinite-time diffusion (which in practice is approximated with a finite-time horizon), we provide an unbiased estimator.” Why does discretization on the bridges process not induce a bias while it does for noise-to-data diffusion?
soundness: 3
strengths: - The authors theoretically validate the correctness of their new approach to use Bridge Matching for MI estimation.
- The difference with prior work on diffusion-based estimator MINDE is well-expressed and theoretically grounded.
- A rich set of standard benchmarks and ablation studies has been conducted.
summary: The paper introduces a new formulation of mutual information (MI) estimation as a domain transfer problem. In particular, it estimates MI by quantifying the difference between the drift functions of diffusion bridge processes associated with the joint distribution and the product of marginal (independent) distributions of the variables. The idea is interesting and improves previously studied diffusion-based estimators, to which the authors compare by running extensive benchmarks.
weaknesses: - The main weakness in the experimental evaluation is that the authors only consider very large sample sizes; 100k training datapoints, while classical approaches have also been tested for 5k or less. Although the main competitor, MINDE, also used 100k training points, it would be interesting to see if the method can also perform well with significantly fewer samples, e.g., 10k.
- While the MINDE estimator requires only the learned score functions to estimate mutual information, InfoBridge relies on full sampling of two diffusion bridge trajectories to approximate the drifts. It is not clear if this has implications for computational cost and scalability. It would be relevant for the authors to address and discuss the computational cost difference between the two methods.
- The paper refers several times to regularity conditions, such as the existence of finite first moments, as prerequisites for the main theorem (Theorem 4.1). However, these assumptions are not stated explicitly in the main text nor appendix. Since these are key hypotheses for the validity of the theoretical results, it would be important to summarize them clearly.
- The volatility coefficient $\epsilon$ is studied in the 16×16 Gaussian and protein embedding experiments, leading to a rule-of-thumb setting $\epsilon = 1$. However, the paper does not provide a detailed analysis of how $\epsilon$ influences the estimator’s bias and variance, nor a principled procedure for selecting it.
- Methods for MI estimation between random variables of different dimension are mentioned but not considered for experiments.
- As mentioned by the authors, heavy-tailed distributions cannot be modelled well.

Minor remarks:

- Some formulas and tables violate the space constraints (see e.g., page 3).
- Some references in the appendix are not rendered correctly.
- Reference: lee & Rhee missing year.


Decision: Accept (Poster)