SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ammar Fayad* MIT
Majd Ibrahim  
HIAST
Measuring dependence between two random variables is of great importance in various domains but is difficult to compute in today's complex environments with high-dimensional data. Recently, slicing methods have shown to be a scalable approach to measuring mutual information (MI) between high-dimensional variables by projecting these variables into one-dimensional spaces. Unfortunately, these methods use uniform distributions of slicing directions, which generally discard informative features between variables and thus lead to inaccurate quantification of dependence. In this paper, we propose a principled framework that searches for an optimal distribution of slices for MI. Importantly, we answer theoretical questions about finding the optimal slicing distribution in the context of MI and develop corresponding theoretical analyses. We also develop a practical algorithm, connecting our theoretical results with modern machine learning frameworks. Through comprehensive experiments in benchmark domains, we demonstrate significant gains in our information measure than state-of-the-art baselines.
Mutual information (MI) measures statistical dependence between two random variables by quantifying the amount of information gained about one variable from an observation of the other variable (Shannon, 1948). Despite its popularity in various fields (Marinoni and Gamba, 2017; Hamma et al., 2016), MI suffers from the curse of dimensionality: the estimation of mutual dependence is often stymied by the large dimension of the variables (Kriskov et al., 2004). Recently, slicing methods (Bonneel et al., 2015; Nadjahi et al., 2020) have demonstrated a scalable approach for estimating probability divergences and mutual dependence. Specifically, sliced mutual information  $(\mathcal{S}\mathcal{I};$  Goldfeld and Greenewald (2021)) measures the average of MI terms between one-dimensional projections of the high-dimensional variables, where the projecting slicing directions are sampled uniformly from a unit sphere. Because slicing methods do not compute the dependence directly in the high-dimensional space but in the low-dimensional space, these methods significantly reduce computation costs and are scalable. However, a critical problem of these slicing frameworks is that they generally require many slices for accurate estimation. As we detail later in the paper, the root cause of this problem is the uniform sampling of the slicing directions: discarding informative features about the variables and generating noisy slices by equally favoring important and irrelevant regions of the variables' spaces. Hence, the uniform sampling fails to capture complex relationships of variables which renders current slicing methods inefficient and can even lead to inaccurate dependence quantification.
Our contribution. With this insight, this paper aims to address the uniform sampling issue with the slicing approaches for improving the dependence measurement. Specifically, motivated by the success of Nguyen et al. (2021) in the generative modeling domain, we seek to find an optimal distribution of slicing directions in the context of MI that satisfies two criteria: 1) slices are distributed over maximally informative regions, and 2) slices are scattered over the unit sphere so that no
$^{*}$ afayad@mit.edu
relevant directions are discarded. We formalize these criteria in a regularized optimization problem and introduce a novel measure of dependence, the optimal sliced mutual information (denoted by  $\mathcal{ST}^*$ ).  $\mathcal{ST}^*$  provides a much more efficient performance compared to previous slicing methods and dependence measures (e.g., improvements of up to  $30\%$  in dependence detection power). We show an important benefit of  $\mathcal{ST}^*$ , which is that it can transform the fundamental work of Goldfeld and Greenewald (2021) (i.e.  $\mathcal{ST}$ ) to larger and more complex machine learning problems.
Several unique challenges arise when developing our regularized optimization problem. First, MI requires at least two slicing variables and the optimization should be over the space of the joint distribution of the slices, but the constraint is enforced over the marginals. We find that this challenge essentially corresponds to solving optimal transportation (OT) (Villani, 2009) problems: optimization over a set of couplings (of the distributions of slices) with given constraints. To that end, we leverage OT properties that enable us to derive many interesting theorems about  $\mathcal{S}\mathcal{I}^*$  connecting it to MI, SI, and differential entropy. The second challenge is the continuity problem: the cost function in OT is assumed to be upper semicontinuous in the topology of weak convergence for the maximizing solution to exist, but this continuity assumption no longer holds when dealing with information-theoretic quantities. We rigorously prove that, under very mild assumptions, a maximizing solution exists for any input variables regardless of their dimensions.
In summary, this paper bridges the gap between intractable information/dependence measures and modern computational frameworks by addressing the shortcomings of current slicing approaches. We investigate the optimality of slicing directions for MI with the following key contributions:
- Formalization of a new dependence measure (Sections 3.1 and 3.2). We introduce a novel and scalable dependence measure,  $\mathcal{S}\mathcal{I}^*$  with theoretical analyses of its properties and implications.
- Scalable estimator (Sections 3.3 and 4). We construct an optimal estimator with the tightest bounds and explicitly show that the effect of the variables' dimensions on the convergence rate is only up to a constant factor. We then employ deep neural networks to this optimal estimator to acquire an end-to-end  $S\mathcal{I}^*$  neural estimator and show that it is computationally efficient.
- Comprehensive evaluation of our approach (Section 6). We demonstrate significant gains of  $\mathcal{ST}^*$  in the accuracy of detecting complicated relations over state-of-the-art baselines on an extensive set of experiments.  $\mathcal{ST}^*$  also works excellently across challenging domains (e.g., representation and reinforcement learning), which is an unprecedented advantage and a feature that is largely missing from current literature on statistical dependence.
Notation. We follow standard notations from Villani (2009). Specifically, this paper considers Borel measures on Polish spaces; the latter are complete, separable metric spaces, equipped with their Borel  $\sigma$ -algebra. We denote the space of Borel probability measures on  $\mathcal{X}$  as  $\mathcal{P}(\mathcal{X})$ . If  $\mu$  is a Borel measure on  $\mathcal{X}$ , and  $T$  is a Borel map  $\mathcal{X} \to \mathcal{Y}$ , then  $T_{\#} \mu$  stands for the push-forward measure of  $\mu$  by  $T$ : it is a Borel measure on  $\mathcal{Y}$ , defined by  $T_{\#} \mu[A] = \mu[T^{-1}(A)]$  for any  $A \subset \mathcal{Y}$ , where  $\delta_x$  is the Dirac delta function at  $x$ . If  $\pi(d\theta d\varphi)$  is a probability measure in two variables  $\theta \in \mathcal{X}$  and  $\varphi \in \mathcal{Y}$ , its marginal (or projection) on  $\mathcal{X}$  (resp.  $\mathcal{Y}$ ) is the measure  $p_{\#}^{\Theta} \pi$  (resp.  $p_{\#}^{\Phi} \pi$ ), where  $p^{\Theta}(\theta, \varphi) = \theta$  and  $p^{\Phi}(\theta, \varphi) = \varphi$ . We focus on absolutely continuous random variables with bounded density functions. Let  $\theta^*: x \mapsto \theta^\top x$  and  $\gamma(\mathcal{X})$  is the uniform distribution on  $\mathcal{X}$ . For any two probability measures  $\mu, \nu$ ,  $d\mu/d\nu$  denotes the Radon-Nikodym derivative of  $\mu$  with respect to  $\nu$ . Furthermore, let  $(\psi \times \xi)(x,y) = (\psi(x),\xi(y))$  be the Cartesian product of two functions  $\psi, \xi$ . Denote by  $\mathcal{C}(\mathcal{X},\mathcal{Y})$  the set of continuous Borel maps from  $\mathcal{X}$  to  $\mathcal{Y}$ .  $\|\cdot\|$  denotes the Euclidean norm. Finally,  $\mathbb{S}^{d-1}$  is the unit sphere in  $\mathbb{R}^d$ , and  $\Omega_{d,k} \stackrel{\mathrm{def}}{=} \mathbb{S}^{d-1} \times \mathbb{S}^{k-1}$ .
Information theory. The mutual information (Shannon, 1948) characterizes the distance between the joint distribution  $P_{X,Y}$  and the product of its marginals  $P_{X} \otimes P_{Y}$ :
$$
\mathcal {I} (X; Y) = \mathrm {K L} (P _ {X, Y} | | P _ {X} \otimes P _ {Y})) = \int_ {\mathcal {X} \times \mathcal {Y}} \log \left(\frac {d P _ {X , Y}}{d P _ {X} \otimes P _ {Y}}\right) d P _ {X, Y},
$$
where  $\mathrm{KL}(\cdot ||\cdot)$  is the Kullback-Leibler divergence (Kullback and Leibler, 1951) which is lower semicontinuous in the topology of weak convergence (Posner, 1975). The Sliced mutual information (Goldfeld and Greenewald, 2021) is the mean of MI terms between one-dimensional projections of
the variables:
$$
\mathcal {I} (X; Y) = \oint_ {\Omega_ {d _ {x}, d _ {y}}} \mathcal {I} \left(\theta^ {\top} X; \varphi^ {\top} Y\right) d \boldsymbol {\gamma} (\theta) \otimes \boldsymbol {\gamma} (\varphi). \tag {1}
$$
In Equation (1),  $X \in \mathbb{R}^{d_x}$ ,  $Y \in \mathbb{R}^{d_y}$ ;  $\theta$  and  $\varphi$  are the slices corresponding to  $X$  and  $Y$ , respectively. We note that slices are independently and uniformly sampled from  $\mathbb{S}^{d_x - 1}$  and  $\mathbb{S}^{d_y - 1}$ . This uniformity causes redundancy in slices which deems optimality beneficial and necessary for the slicing process.
3 Definition and Theories of Slicing Optimality
In this section, we begin by motivating the definition of our novel measure by showing the advantage of our solution over uniform sampling. We then formalize our proposal as a regularized optimization problem. Lastly, we discuss our theoretical findings, such as the existence of an optimal slicing policy and the properties of  $S\mathcal{I}^*$ . These studies show that  $S\mathcal{I}^*$  possesses desired characteristics of a dependence measure.
3.1 Definition of  $\mathcal{S}\mathcal{I}^*$
We first motivate our solution in an example, in which we incorporate the following two optimality criteria into  $\mathcal{S}\mathcal{I}$ :
1. The projection directions are mainly concentrated into areas where the one-dimensional variables contain the maximum mutual information possible.
2. The slicing directions are also diversified over the whole sphere, ensuring that all regions with relevant information are visited.
Motivating example. Let  $Z \sim \mathcal{N}(0, I_3)$ ,  $\Lambda \sim \mathcal{N}(0, 0.1I_2)$ , and let  $X = [Z_1, Z_2]^\top$ ,  $Y = [Z_1, Z_3]^\top + \Lambda$ . In this example, we aim to estimate the average MI between one-dimensional projections of the two variables  $X$  and  $Y$  and show that our optimal distribution of slices yields superior performance over standard  $S\mathcal{I}$ . Specifically, by sampling slicing directions,  $\theta$  and  $\varphi$ , uniformly and independently from the 2-sphere (i.e., unit circle), we obtain  $S\mathcal{I}$ . To construct the distribution of slices that meets the former criteria, we first note that the slices which project only the first entries of  $X$ ,  $Y$
Figure 1: ROC AUC curve (left) and visualization of the custom distribution the slices (middle and right). Optimal distribution of slices can capture more information and yields a more accurate measure of dependence.
yield the maximum information because both variables share  $Z_{1}$  as their first entry (up to additive noise) while their second entries are independent. We leverage this observation and sample two-dimensional slices such that they are mainly concentrated around  $|\theta_0| = 1$ , and  $|\varphi_0| = 1$ , respectively. We also enforce the second criterion of slices diversity by choosing a few slices away from the cluster (See Figure 1 middle and right). We refer to Appendix D.1 for details.
Figure 1 (left) shows the area under the curve (AUC) of the receiver operating characteristic (ROC) as a function of the number of slices along with a visualization of the customized distribution of  $\theta, \varphi$ . Note the improvement in the number of slices needed until reaching an ROC AUC = 1, as our customized measure requires  $\sim 75\%$  fewer slices to reach perfect accuracy. This result shows that our proposed distribution can extract sufficient information more effectively.
Having established the importance of optimal slicing, we now formally define the optimal sliced mutual information  $(\mathcal{S}\mathcal{I}^{*})$  as a weighted average of information stored in one-dimensional projections of two random variables  $X,Y$ :
Definition 1 (Optimal sliced mutual information). Given two random variables  $X \in \mathbb{R}^{d_x}$ ,  $Y \in \mathbb{R}^{d_y}$  and  $\omega_X, \omega_Y \in [0, \pi/2]$ , define the following collection of probability measures  $\Sigma_{d,\omega} = \{\mu : \mu \in$
$\mathcal{P}(\mathbb{S}^{d - 1}), \mathbb{E}_{x,y \sim \mu}[\arccos |x^\top y|] \geq \omega\}$ . The Optimal Sliced Mutual Information can be expressed as:
$$
\mathcal {I} ^ {*} (X; Y) = \sup  \left\{\oint_ {\Omega_ {d _ {x}, d _ {y}}} \mathcal {I} \left(\theta^ {\top} X; \varphi^ {\top} Y\right) d \sigma (\theta , \varphi): p _ {\#} ^ {\Theta} \sigma \in \Sigma_ {d _ {x}, \omega_ {X}}, p _ {\#} ^ {\Phi} \sigma \in \Sigma_ {d _ {y}, \omega_ {Y}} \right\}. \tag {2}
$$
We call the distribution  $\sigma$  a slicing policy. Note that  $p_{\#}^{\Theta}\sigma$  refers to the first marginal distribution of  $\sigma$  (in other words, the law of the random slicing vector of  $X$ ,  $\Theta$ , i.e.  $\mathrm{law}(\Theta) = p_{\#}^{\Theta}\sigma$ );  $\mathrm{law}(\Theta) \in \Sigma_{d_x,\omega_X}$ . This observation also applies to  $\mathrm{law}(\Phi) \in \Sigma_{d_y,\omega_Y}$ . The supremum is taken to ensure that the slices convey sufficient information to quantify the relationship between  $X$  and  $Y$ , or equivalently to ensure that the slices are prevalent in maximally informative regions (i.e., criterion 1). The set over which the supremum is taken, thus, refers to the joint measures whose marginals produce slices that are scattered over the unit sphere of an appropriate dimension (i.e., criterion 2), where we use the  $\arccos|.||$  to measure slices diversity. Importantly, without the constraint, the measure might collapse to a Dirac delta probability measure at the max slicing vector (i.e. collapse to  $\sup_{\theta,\varphi}\mathcal{I}$ ). As such, the prominent feature of  $\mathcal{S}\mathcal{I}^*$  is that it reveals the maximum amount of information content stored in  $X,Y$  by searching for a distribution of slices that results in informative, noiseless, and diverse projections.
Remark 1.  $S\mathcal{I}^*$  is not a mutual information estimator and should not be used as a proxy of MI but rather as a new dependency measure. In fact, although it shares many properties with MI,  $S\mathcal{I}^*$  has multiple advantages such as scalability and efficiency, not to mention the cutting-edge performance on detecting complex relationships between high-dimensional random variables in nontrivial settings.
Remark 2. If  $d_x = d_y = 1, \mathcal{ST}^*(X;Y) = \mathcal{I}(X;Y)$ . Essentially, for one-dimensional r.v.s,  $\mathcal{ST}^*$  boils down to  $MI$ .
We also consider the optimality of slicing into  $k$ -dimensional subspaces by enforcing our same criteria on distributions over the Stiefel manifold (Chikuse, 2012). Appendix E details this definition.
3.2 Theoretical Properties of  $S\mathcal{I}^*$
We propose optimizing the slicing distribution towards one-dimensional projections with higher MI values. However, due to the lower semicontinuity of MI in the maximization problem, it is difficult to establish whether an optimal slicing policy exists. In the following, we prove the existence of a solution that our problem can converge to by assuming that for a given random variable  $X$  with pdf  $p_X, \int \| x\|^{\kappa}p_X(x)dx < \infty$  for some  $\kappa > 1$ :
Theorem 1 (Existence of an optimal slicing policy). For any random variables  $X$  and  $Y$  and any  $\omega_{X},\omega_{Y}\in [0,\pi /2]$ , there exists a slicing policy  $\sigma$  such that the information functional,  $\oint \mathcal{I}(\theta^{\top}X;\varphi^{\top}Y)d\sigma (\theta ,\varphi)$ , is maximized among all possible couplings of  $(law(\Theta),law(\Phi))\in \Sigma_{d_x,\omega_X}\times \Sigma_{d_y,\omega_Y}$
Proof. See Appendix A.1 for details.
We refer to such  $\sigma$  as an optimal slicing policy. Theorem 1 enables us to treat the problem as a reward-maximizing OT. Roughly speaking,  $S\mathcal{I}^{*}(X;Y)$  can be deemed as a reward-maximizing optimal transport problem, where the reward induced by slicing vectors  $\theta$  and  $\varphi$  is the amount of information they reveal about  $X$  and  $Y$ . As such, the formula in Equation (2) measures the maximum information gain possible among slicing policies.
Rényi (1959) and Bell (1962) postulate that a measure of dependence should satisfy properties found in Theorem 2:
Theorem 2 (Properties of  $\mathcal{S}\mathcal{I}^*$ ). For random variables  $X$  and  $Y$ , we have:
1.  $\mathcal{ST}^* (X;Y)$  is nonnegative and symmetric.
2.  $\mathcal{ST}^* (X;Y) = 0$  if and only if  $X$  and  $Y$  are independent.
3. If  $X_{n}, Y_{n}$  are sequences of random variables with joint distribution  $P_{X,Y}^{(n)}$  that converges pointwise to the joint distribution  $P_{X,Y}$ , then  $\lim_{n\to \infty}\mathcal{ST}^{\ast}(X_n;Y_n) = \mathcal{ST}^{\ast}(X;Y)$ .
4. Similar to MI,  $\mathcal{S}\mathcal{I}^*$  has a relative entropy form:  $\mathcal{S}\mathcal{I}^{*}(X;Y) = \sup_{\sigma} \underset{(\Theta, \Phi) \sim \sigma}{\mathbb{E}}[KL((\Theta^{*} \times \Phi^{*})_{\#} P_{X,Y}||\Theta_{\#}^{*}P_{X} \otimes \Phi_{\#}^{*}P_{Y})] = \sup_{\sigma} KL(\sigma \otimes (\Theta^{*} \times \Phi^{*})_{\#} P_{X,Y}||\sigma \otimes \Theta_{\#}^{*}P_{X} \otimes \Phi_{\#}^{*}P_{Y})$
Proof. See Appendix A.2 for details.
Now that we have established that  $\mathcal{S}\mathcal{I}^*$  possesses the desirable properties of a dependence measure, we emphasize the importance of using two slices:
Remark 3. Unlike existing slicing methods (Nguyen et al., 2021),  $\mathcal{S}\mathcal{I}^*$  requires two slicing variables  $\Theta$  and  $\Phi$  since using only one slice violates the important property (2) in Theorem 2. To illustrate, Let  $X\sim \mathcal{N}(0,I_2)$  and  $Y = \left( \begin{array}{cc}0 & \alpha \\ -\alpha & 0 \end{array} \right)X,\alpha \neq 0$ , clearly  $X$  and  $Y$  are dependent but  $\theta^{\top}X$  and  $\theta^{\top}Y$  are not for all  $\theta \in \mathbb{S}^{1}$  (the latter follows from the independence of the entries of  $X$ . See (Goldfeld and Greenewald, 2021)). The observation implies that for the such dependent  $X,Y$ ,  $\mathcal{S}\mathcal{I}^*(X;Y) = 0$ . We thus resort to using two slicing directions.
Next, we use the property (4) in Theorem 2 to represent  $\mathcal{S}\mathcal{I}^*$  using a discriminator function.
Corollary 1 (Discriminator-based form). Let  $\mu = (\Theta^{*} \times \Phi^{*})_{\#} P_{X,Y}, \nu = \Theta_{\#}^{*} P_{X} \otimes \Phi_{\#}^{*} P_{Y}$ . We write  $\mathcal{S}\mathcal{I}^{*}$  as:
$$
\mathcal {S I} ^ {*} (X; Y) = \sup  _ {\sigma} \mathbb {E} _ {(\Theta , \Phi)} \mathbb {E} _ {\mu} \Big [ \arg \max  _ {h} \big [ \mathbb {E} _ {\mu} [ \log (\varsigma \circ h (.)) ] - \mathbb {E} _ {\nu} [ 1 - \log (\varsigma \circ h (.)) ] \big ] \Big ],
$$
where  $\varsigma$  is the Sigmoid function and the discriminator  $h$  is defined on  $\mathbb{R}$ . This GAN-type form may be used to estimate  $S\mathcal{I}^*$  in Reproducing Kernel Hilbert Space similarly to Ghimire et al. (2021)'s approach for estimating KL.
Proof. See Appendix A.2.1 for details.
Moreover, we present a variational representation of  $\mathcal{S}\mathcal{I}^*$  which will be later used to construct a neural estimator:
Corollary 2 (Variational representation). Let  $(X,Y)\sim P_{X,Y}$  and  $(\Psi ,\Upsilon)\sim \gamma (\Omega_{d_x,d_y})$  . We have:
$$
\begin{array}{l} \mathcal {S I} ^ {*} (X; Y) = \sup  _ {T, f _ {1}, f _ {2}} \left\{\mathbb {E} \left[ T (\Psi , \Upsilon , f _ {1} (\Psi , \Upsilon) ^ {\top} X, f _ {2} (\Psi , \Upsilon) ^ {\top} Y) \right] - \right. \tag {3} \\ \left. \log \mathbb {E} \left[ \exp \left(T (\Psi , \Upsilon , f _ {1} (\Psi , \Upsilon) ^ {\top} \overline {{X}}, f _ {2} (\Psi , \Upsilon) ^ {\top} \overline {{Y}})\right) \right] \right\}, \\ \end{array}
$$
where  $(\overline{X},\overline{Y})\sim P_X\otimes P_Y$ . The supremum is taken over  $T\in \mathcal{C}(\Omega_{d_x,d_y}\times \mathbb{R}^2,\mathbb{R})$ ,  $f_{1}\in \mathcal{F}_{\omega_{X}}\stackrel {def}{=}\{f:f\in \mathcal{C}(\Omega_{d_x,d_y},\mathbb{S}^{d_x - 1}),f_{\#}\pmb {\gamma}(\Omega_{d_x,d_y})\in \Sigma_{d_x,\omega_X}\}$ $f_{2}\in \mathcal{F}_{\omega_{Y}}$ , where  $\mathcal{F}_{\omega_Y}$  is defined analogously to  $\mathcal{F}_{\omega_X}$ .
Proof. See Appendix A.2.2 for details.
Corollaries 1 and 2 provide useful representations of  $\mathcal{S}\mathcal{I}^*$  that make it compatible with modern machine learning algorithms and optimization models. Next, we highlight the concept of data processing that connects  $\mathcal{S}\mathcal{I}^*$  to  $\mathcal{I}$  and  $\mathcal{S}\mathcal{I}$ .
Theorem 3 (Data processing). For  $\omega_{X},\omega_{Y}\geq \pi /4$
$$
\mathcal {I} (X; Y) \geq \mathcal {S I} ^ {*} (X; Y) \geq \mathcal {S I} (X; Y). \tag {4}
$$
Notably,  $\mathcal{I}(X;Y) \geq \sup_{\theta, \varphi} \mathcal{I}(\theta^\top X; \varphi^\top Y) \geq \mathcal{S}\mathcal{I}^*(X;Y) \geq \mathcal{S}\mathcal{I}(X;Y) \geq \inf_{\theta, \varphi} \mathcal{I}(\theta^\top X; \varphi^\top Y)$ .
Proof. See Appendix A.3 for details.
Theorem 3 shows that  $\mathcal{S}\mathcal{I}^*$  is a generalization of  $\mathcal{S}\mathcal{I}$  because, for the given values of  $\omega_{X}$  and  $\omega_{Y}$ ,  $\Sigma_{d_x,\omega_X}$  and  $\Sigma_{d_y,\omega_Y}$  contain the uniform distributions over the respective sphere. On the other hand,  $\mathcal{S}\mathcal{I}^*$  does not exceed MI as a consequence of the Data Processing Inequality. However, due to the scalability and sample efficiency issues in MI,  $\mathcal{S}\mathcal{I}^*$  is a much better candidate for quantifying dependence than MI especially when the data have complex structures.
3.3 Estimation
A key property of  $S\mathcal{I}^*$  is its scalability which is intrinsic to slicing approaches, where the latter facilitate estimation from samples and do not suffer from the curse of dimensionality since computations are performed in low-dimensional subspaces. Here, we construct an optimal  $S\mathcal{I}^*$  estimator based on
one-dimensional information estimators. Let  $\widehat{\mathcal{I}}_n$  be a one-dimensional MI estimator over  $n$  samples whose absolute error is uniformly bounded by  $\delta (n)$ :
$$
\sup_{P_{S,Q}}\mathbb{E}[|\mathcal{I}(S;Q) - \widehat{\mathcal{I}}_{n}(S;Q)|]\leq \delta (n).
$$
Any optimal slicing policy  $\sigma^{*}$  that maximizes the information functional can be obtained by applying an appropriate transformation on the uniform measure on  $\Omega_{d_x,d_y}$ ; this transformation can be learned using neural networks. By the definition of the push-forward measure, we express  $\mathcal{S}\mathcal{I}^{*}(X;Y)$  as:
$$
\mathcal {S I} ^ {*} (X; Y) = \sup  _ {f _ {1} \in \mathcal {F} _ {\omega_ {X}}, f _ {2} \in \mathcal {F} _ {\omega_ {Y}}} \mathbb {E} _ {(\psi , v) \sim \boldsymbol {\gamma} (\Omega_ {d _ {x}, d _ {y}})} \left[ \mathcal {I} \left(f _ {1} (\psi , v) ^ {\top} X; f _ {2} (\psi , v) ^ {\top} Y\right) \right],
$$
where  $\mathcal{F}_{\omega_X},\mathcal{F}_{\omega_Y}$  are defined in Corollary 2. The approximation then becomes:
$$
\widehat {\mathcal {S I} ^ {*}} _ {n, m} (X; Y) \stackrel {\text {d e f}} {=} \sup  _ {f _ {1} \in \mathcal {F} _ {\omega_ {X}}, f _ {2} \in \mathcal {F} _ {\omega_ {Y}}} \frac {1}{m} \sum_ {j = 1} ^ {m} \left[ \widehat {\mathcal {I}} _ {n} \left(f _ {1} \left(\psi_ {j}, v _ {j}\right) ^ {\top} X; f _ {2} \left(\psi_ {j}, v _ {j}\right) ^ {\top} Y\right) \right]. \tag {5}
$$
Theorem 4 (Convergence Rate). The uniform error bound of  $\widehat{\mathcal{S}\mathcal{L}^*}_{n,m}(X;Y)$  is: