Title: On Slicing Optimality for Mutual Information
Venue: NeurIPS 2023
Accepted: yes
Keywords: Mutual information

Information Theory
Abstract: Measuring dependence between two random variables is of great importance in various domains but is difficult to compute in today's complex environments with high-dimensional data. Recently, slicing methods have shown to be a scalable approach to measuring mutual information (MI) between high-dimensional variables by projecting these variables into one-dimensional spaces. Unfortunately, these methods use uniform distributions of slicing directions, which generally discard informative features between variables and thus lead to inaccurate quantification of dependence. In this paper, we propose a principled framework that searches for an \textit{optimal} distribution of slices for MI. Importantly, we answer theoretical questions about finding the optimal slicing distribution in the context of MI and develop corresponding theoretical analyses. We also develop a practical algorithm, connecting our theoretical results with modern machine learning frameworks. Through comprehensive experiments in benchmark domains, we demonstrate significant gains in our information measure than state-of-the-art baselines.

On Slicing Optimality for Mutual Information

Abstract

Measuring dependence between two random variables is of great importance in various domains but is difficult to compute in today's complex environments with high-dimensional data. Recently, slicing methods have shown to be a scalable approach to measuring mutual information (MI) between high-dimensional variables by projecting these variables into one-dimensional spaces. Unfortunately, these methods use uniform distributions of slicing directions, which generally discard informative features between variables and thus lead to inaccurate quantification of dependence. In this paper, we propose a principled framework that searches for an optimal distribution of slices for MI. Importantly, we answer theoretical questions about finding the optimal slicing distribution in the context of MI and develop corresponding theoretical analyses. We also develop a practical algorithm, connecting our theoretical results with modern machine learning frameworks. Through comprehensive experiments in benchmark domains, we demonstrate significant gains in our information measure than state-of-the-art baselines.

1 Introduction

Mutual information (MI) measures statistical dependence between two random variables by quantifying the amount of information gained about one variable from an observation of the other variable (Shannon, 1948). Despite its popularity in various fields (Marinoni and Gamba, 2017; Hamma et al., 2016), MI suffers from the curse of dimensionality: the estimation of mutual dependence is often stymied by the large dimension of the variables (Kraskov et al., 2004). Recently, slicing methods (Bonneel et al., 2015; Nadjahi et al., 2020) have demonstrated a scalable approach for estimating probability divergences and mutual dependence. Specifically, sliced mutual information ( SI ; Goldfeld and Greenewald (2021)) measures the average of MI terms between one-dimensional projections of the high-dimensional variables, where the projecting slicing directions are sampled uniformly from a unit sphere. Because slicing methods do not compute the dependence directly in the high-dimensional space but in the low-dimensional space, these methods significantly reduce computation costs and are scalable. However, a critical problem of these slicing frameworks is that they generally require many slices for accurate estimation. As we detail later in the paper, the root cause of this problem is the uniform sampling of the slicing directions: discarding informative features about the variables and generating noisy slices by equally favoring important and irrelevant regions of the variables' spaces. Hence, the uniform sampling fails to capture complex relationships of variables which renders current slicing methods inefficient and can even lead to inaccurate dependence quantification.

Our contribution. With this insight, this paper aims to address the uniform sampling issue with the slicing approaches for improving the dependence measurement. Specifically, motivated by the success of Nguyen et al. (2021) in the generative modeling domain, we seek to find an optimal distribution of slicing directions in the context of MI that satisfies two criteria: 1) slices are distributed over maximally informative regions, and 2) slices are scattered over the unit sphere so that no

relevant directions are discarded. We formalize these criteria in a regularized optimization problem and introduce a novel measure of dependence, the optimal sliced mutual information (denoted by SI ∗ ). SI ∗ provides a much more efficient performance compared to previous slicing methods and dependence measures (e.g., improvements of up to 30% in dependence detection power). We show an important benefit of SI ∗ , which is that it can transform the fundamental work of Goldfeld and Greenewald (2021) (i.e. SI ) to larger and more complex machine learning problems.

Several unique challenges arise when developing our regularized optimization problem. First, MI requires at least two slicing variables and the optimization should be over the space of the joint distribution of the slices, but the constraint is enforced over the marginals. We find that this challenge essentially corresponds to solving optimal transportation (OT) (Villani, 2009) problems: optimization over a set of couplings (of the distributions of slices) with given constraints. To that end, we leverage OT properties that enable us to derive many interesting theorems about SI ∗ connecting it to MI, SI, and differential entropy. The second challenge is the continuity problem: the cost function in OT is assumed to be upper semicontinuous in the topology of weak convergence for the maximizing solution to exist, but this continuity assumption no longer holds when dealing with information-theoretic quantities. We rigorously prove that, under very mild assumptions, a maximizing solution exists for any input variables regardless of their dimensions.

In summary, this paper bridges the gap between intractable information/dependence measures and modern computational frameworks by addressing the shortcomings of current slicing approaches. We investigate the optimality of slicing directions for MI with the following key contributions:

Formalization of a new dependence measure (Sections 3.1 and 3.2). We introduce a novel and scalable dependence measure, SI ∗ with theoretical analyses of its properties and implications.

Comprehensive evaluation of our approach (Section 6). We demonstrate significant gains of SI ∗ in the accuracy of detecting complicated relations over state-of-the-art baselines on an extensive set of experiments. SI ∗ also works excellently across challenging domains (e.g., representation and reinforcement learning), which is an unprecedented advantage and a feature that is largely missing from current literature on statistical dependence.

Scalable estimator (Sections 3.3 and 4). We construct an optimal estimator with the tightest bounds and explicitly show that the effect of the variables' dimensions on the convergence rate is only up to a constant factor. We then employ deep neural networks to this optimal estimator to acquire an end-to-end SI ∗ neural estimator and show that it is computationally efficient.

2 Background

Notation. We follow standard notations from Villani (2009). Specifically, this paper considers Borel measures on Polish spaces; the latter are complete, separable metric spaces, equipped with their Borel σ -algebra. We denote the space of Borel probability measures on X as P ( X ) . If µ is a Borel measure on X , and T is a Borel map X → Y , then T # µ stands for the push-forward measure of µ by T : it is a Borel measure on Y , defined by T # µ [ A ] = µ [ T -1 ( A )] for any A ⊂ Y , where δ x is the Dirac delta function at x . If π ( dθdφ ) is a probability measure in two variables θ ∈ X and φ ∈ Y , its marginal (or projection) on X (resp. Y ) is the measure p Θ # π (resp. p Φ # π ), where p Θ ( θ, φ ) = θ and p Φ ( θ, φ ) = φ . We focus on absolutely continuous random variables with bounded density functions. Let θ ∗ : x ↦→ θ ⊤ x and γ ( X ) is the uniform distribution on X . For any two probability measures µ, ν , dµ/dν denotes the Radon-Nikodym derivative of µ with respect to ν . Furthermore, let ( ψ × ξ )( x, y ) = ( ψ ( x ) , ξ ( y )) be the Cartesian product of two functions ψ, ξ . Denote by C ( X , Y ) the set of continuous Borel maps from X to Y . ∥ . ∥ denotes the Euclidean norm. Finally, S d -1 is the unit sphere in R d , and Ω d,k def = S d -1 × S k -1 .

Information theory. The mutual information (Shannon, 1948) characterizes the distance between the joint distribution P X,Y and the product of its marginals P X ⊗ P Y :

where KL ( ·||· ) is the Kullback-Leibler divergence (Kullback and Leibler, 1951) which is lower semicontinuous in the topology of weak convergence (Posner, 1975). The Sliced mutual information (Goldfeld and Greenewald, 2021) is the mean of MI terms between one-dimensional projections of

the variables:

In Equation (1), X ∈ R d x , Y ∈ R d y ; θ and φ are the slices corresponding to X and Y , respectively. We note that slices are independently and uniformly sampled from S d x -1 and S d y -1 . This uniformity causes redundancy in slices which deems optimality beneficial and necessary for the slicing process.

3 Definition and Theories of Slicing Optimality

In this section, we begin by motivating the definition of our novel measure by showing the advantage of our solution over uniform sampling. We then formalize our proposal as a regularized optimization problem. Lastly, we discuss our theoretical findings, such as the existence of an optimal slicing policy and the properties of SI ∗ . These studies show that SI ∗ possesses desired characteristics of a dependence measure.

3.1 Definition of SI ∗

We first motivate our solution in an example, in which we incorporate the following two optimality criteria into SI :

The projection directions are mainly concentrated into areas where the one-dimensional variables contain the maximum mutual information possible.

The slicing directions are also diversified over the whole sphere, ensuring that all regions with relevant information are visited.

Motivating example. Let Z ∼ N (0 , I 3 ) , Λ ∼ N (0 , 0 . 1 I 2 ) , and let X =[ Z 1 , Z 2 ] ⊤ , Y =[ Z 1 , Z 3 ] ⊤ +Λ . In this example, we aim to estimate the average MI between one-dimensional projections of the two variables X and Y and show that our optimal distribution of slices yields superior performance over standard SI . Specifically, by sampling slicing directions, θ and φ , uniformly and independently from the 2-sphere (i.e., unit circle), we obtain SI . To construct the distribution of slices that meets the former criteria, we first note that the slices which project only the first entries of X,Y

1

4

16

128

Number of Slices

.5

.6

.7

.8

.9

1

AUC

SI (Optimal)

SI (Uniform)

MI

θ

0

θ

1

ϕ

0

ϕ

1

yield the maximum information because both variables share Z 1 as their first entry (up to additive noise) while their second entries are independent. We leverage this observation and sample twodimensional slices such that they are mainly concentrated around | θ 0 | = 1 , and | φ 0 | = 1 , respectively. We also enforce the second criterion of slices diversity by choosing a few slices away from the cluster (See Figure 1 middle and right). We refer to Appendix D.1 for details.

Figure 1 (left) shows the area under the curve (AUC) of the receiver operating characteristic (ROC) as a function of the number of slices along with a visualization of the customized distribution of θ, φ . Note the improvement in the number of slices needed until reaching an ROC AUC = 1, as our customized measure requires ∼ 75% fewer slices to reach perfect accuracy. This result shows that our proposed distribution can extract sufficient information more effectively.

Having established the importance of optimal slicing, we now formally define the optimal sliced mutual information ( SI ∗ ) as a weighted average of information stored in one-dimensional projections of two random variables X,Y :

Definition 1 (Optimal sliced mutual information) . Given two random variables X ∈ R d x , Y ∈ R d y and ω X , ω Y ∈ [0 , π/ 2] , define the following collection of probability measures Σ d,ω = { µ : µ ∈

P ( S d -1 ) , E x,y ∼ µ [arccos | x ⊤ y | ] ≥ ω } . The Optimal Sliced Mutual Information can be expressed as :

We call the distribution σ a slicing policy. Note that p Θ # σ refers to the first marginal distribution of σ (in other words, the law of the random slicing vector of X , Θ , i.e. law (Θ)= p Θ # σ ); law (Θ) ∈ Σ d x ,ω X . This observation also applies to law (Φ) ∈ Σ d y ,ω Y . The supremum is taken to ensure that the slices convey sufficient information to quantify the relationship between X and Y , or equivalently to ensure that the slices are prevalent in maximally informative regions (i.e., criterion 1). The set over which the supremum is taken, thus, refers to the joint measures whose marginals produce slices that are scattered over the unit sphere of an appropriate dimension (i.e., criterion 2), where we use the arccos | . | to measure slices diversity. Importantly, without the constraint, the measure might collapse to a Dirac delta probability measure at the max slicing vector (i.e. collapse to sup θ,φ I ). As such, the prominent feature of SI ∗ is that it reveals the maximum amount of information content stored in X,Y by searching for a distribution of slices that results in informative, noiseless, and diverse projections.

Remark 1. SI ∗ is not a mutual information estimator and should not be used as a proxy of MI but rather as a new dependency measure. In fact, although it shares many properties with MI, SI ∗ has multiple advantages such as scalability and efficiency, not to mention the cutting-edge performance on detecting complex relationships between high-dimensional random variables in nontrivial settings.

Remark 2. If d x = d y = 1 , SI ∗ ( X ; Y ) = I ( X ; Y ) . Essentially, for one-dimensional r.v.s, SI ∗ boils down to MI.

We also consider the optimality of slicing into k -dimensional subspaces by enforcing our same criteria on distributions over the Stiefel manifold (Chikuse, 2012). Appendix E details this definition.

3.2 Theoretical Properties of SI ∗

We propose optimizing the slicing distribution towards one-dimensional projections with higher MI values. However, due to the lower semicontinuity of MI in the maximization problem, it is difficult to establish whether an optimal slicing policy exists. In the following, we prove the existence of a solution that our problem can converge to by assuming that for a given random variable X with pdf p X , ∫ ∥ x ∥ κ p X ( x ) dx < ∞ for some κ > 1 :

Theorem 1 (Existence of an optimal slicing policy) . For any random variables X and Y and any ω X , ω Y ∈ [0 , π/ 2] , there exists a slicing policy σ such that the information functional, ∮ I ( θ ⊤ X ; φ ⊤ Y ) d σ ( θ, φ ) , is maximized among all possible couplings of ( law (Θ) , law (Φ)) ∈ Σ d x ,ω X × Σ d y ,ω Y .

We refer to such σ as an optimal slicing policy. Theorem 1 enables us to treat the problem as a reward-maximizing OT. Roughly speaking, SI ∗ ( X ; Y ) can be deemed as a reward-maximizing optimal transport problem, where the reward induced by slicing vectors θ and φ is the amount of information they reveal about X and Y . As such, the formula in Equation (2) measures the maximum information gain possible among slicing policies.

Rényi (1959) and Bell (1962) postulate that a measure of dependence should satisfy properties found in Theorem 2:

Theorem 2 (Properties of SI ∗ ) . For random variables X and Y , we have:

SI ∗ ( X ; Y ) = 0 if and only if X and Y are independent.

SI ∗ ( X ; Y ) is nonnegative and symmetric.

If X n , Y n are sequences of random variables with joint distribution P ( n ) X,Y that converges pointwise to the joint distribution P X,Y , then lim n →∞ SI ∗ ( X n ; Y n ) = SI ∗ ( X ; Y ) .

Similar to MI, SI ∗ has a relative entropy form: SI ∗ ( X ; Y ) = sup σ E (Θ , Φ) ∼ σ [ KL ( (Θ ∗ ×

□

Now that we have established that SI ∗ possesses the desirable properties of a dependence measure, we emphasize the importance of using two slices:

Remark 3. Unlike existing slicing methods (Nguyen et al., 2021), SI ∗ requires two slicing variables Θ and Φ since using only one slice violates the important property (2) in Theorem 2. To illustrate, Let

̸

are not for all θ ∈ S 1 (the latter follows from the independence of the entries of X . See (Goldfeld and Greenewald, 2021)). The observation implies that for the such dependent X,Y, SI ∗ ( X ; Y ) = 0 . We thus resort to using two slicing directions.

Next, we use the property (4) in Theorem 2 to represent SI ∗ using a discriminator function.

Corollary 1 (Discriminator-based form) . Let µ = (Θ ∗ × Φ ∗ ) # P X,Y , ν = Θ ∗ # P X ⊗ Φ ∗ # P Y . We write SI ∗ as:

where ς is the Sigmoid function and the discriminator h is defined on R . This GAN-type form may be used to estimate SI ∗ in Reproducing Kernel Hilbert Space similarly to Ghimire et al. (2021)'s approach for estimating KL.

Moreover, we present a variational representation of SI ∗ which will be later used to construct a neural estimator:

Corollary 2 (Variational representation) . Let ( X,Y ) ∼ P X,Y and (Ψ , Υ) ∼ γ (Ω d x ,d y ) . We have:

where ( X,Y ) ∼ P X ⊗ P Y . The supremum is taken over T ∈ C (Ω d x ,d y × R 2 , R ) , f 1 ∈ F ω X def = { f : f ∈ C (Ω d x ,d y , S d x -1 ) , f # γ (Ω d x ,d y ) ∈ Σ d x ,ω X } , f 2 ∈ F ω Y , where F ω Y is defined analogously to F ω X .

Corollaries 1 and 2 provide useful representations of SI ∗ that make it compatible with modern machine learning algorithms and optimization models. Next, we highlight the concept of data processing that connects SI ∗ to I and SI .

Theorem 3 (Data processing) . For ω X , ω Y ≥ π/ 4 ,

Proof.

Theorem 3 shows that SI ∗ is a generalization of SI because, for the given values of ω X and ω Y , Σ d x ,ω X and Σ d y ,ω Y contain the uniform distributions over the respective sphere. On the other hand, SI ∗ does not exceed MI as a consequence of the Data Processing Inequality. However, due to the scalability and sample efficiency issues in MI, SI ∗ is a much better candidate for quantifying dependence than MI especially when the data have complex structures.

3.3 Estimation

A key property of SI ∗ is its scalability which is intrinsic to slicing approaches, where the latter facilitate estimation from samples and do not suffer from the curse of dimensionality since computations are performed in low-dimensional subspaces. Here, we construct an optimal SI ∗ estimator based on

one-dimensional information estimators. Let ̂ I n be a one-dimensional MI estimator over n samples whose absolute error is uniformly bounded by δ ( n ) :

Any optimal slicing policy σ ∗ that maximizes the information functional can be obtained by applying an appropriate transformation on the uniform measure on Ω d x ,d y ; this transformation can be learned using neural networks. By the definition of the push-forward measure, we express SI ∗ ( X ; Y ) as 2 :

where F ω X , F ω Y are defined in Corollary 2. The approximation then becomes:

Theorem 4 (Convergence Rate) . The uniform error bound of ̂ SI ∗ n,m ( X ; Y ) is:

where U ∝ ( d -1 x + d -1 y ) 1 / 2 is a constant factor.

We note that the estimation rate depends on the dimensions of the problem d x , d y only up to the constant factor U . The explicit dependence of U on d x , d y has been a recent challenge which is now solved in Theorem 4. By imposing additional regularity on P X,Y , we can obtain δ ( n ) ≤ C ( ( n log n ) -s s +2 log n 1 -2 p ( s +2) + n -1 / 2 ) where s, p : 0 < s ≤ 2 ≤ p, and C > 0 (see Han et al. (2020); Goldfeld and Greenewald (2021)), which is significantly faster than the MI estimation rate, i.e. n -1 / ( d x + d y ) .

4 Neural Estimator for SI ∗

In this section, we introduce a practical implementation of SI ∗ (based on Corollary 2) by leveraging a neural network estimator and connect our theoretical results in Section 3 with modern machine learning frameworks. Let D def = { ( X k , Y k ) } m k =1 be a mini-batch with X ∈ R d x , Y ∈ R d y , and n = ⌊ m 2 ⌋ . We take ( X,Y ) ∼ { ( X k , Y k ) } n k =1 , ( X,Y ) ∼ { ( X k , Y k + n ) } n k =1 , 3 and (Ψ k , Υ k ) ∼ γ (Ω d x ,d y ) . Then, the neural network estimator SI ∗ W is computed as:

where f ( k ) i = f i (Ψ k , Υ k ) for i = 1 , 2; k ∈ [ n ] . λ 1 , λ 2 > 0 , and W def = [ w T , w 1 , w 2 ] are the parameters of neural networks T, f 1 , f 2 , respectively. We also follow Song and Ermon (2020) and account for the possible high-variance issues by using a smoothed version of Equation (6), where we clip the exp( · ) term in the second line between exp( -ρ ) and exp( ρ ) where ρ > 0 . We refer to Appendix C for the pseudocode of SI ∗ W . We later analyze the computational complexity of the estimator.

̸

.5

.6

.7

.8

.9

1

.5

.6

.7

.8

.9

1

AUC

Linear

.5

.6

.7

.8

.9

1

Parabolic

.5

.6

.7

.8

.9

1

Elliptical

.5

.6

.7

.8

.9

1

Sinusoidal

.5

.6

.7

.8

.9

1

Two Sinusoidals

.5

.6

.7

.8

.9

1

.5

.6

.7

.8

.9

1

AUC

.5

.6

.7

.8

.9

1

3

k

=4)

.5

.6

.7

.8

.9

1

3

Epicycloid (k=1)

.5

.6

.7

.8

.9

1

Spherical Harmonic 1

.5

.6

.7

.8

.9

1

Spherical Harmonic 2

5 Related Work

The main idea in quantifying dependence between random variables X,Y is to characterize the distance between their joint distribution P X,Y and the product of its marginals P X ⊗ P Y . Naturally, this problem has been studied in the area of optimal transportation theory (Mordant and Segers, 2022; Wiesel, 2021; Chuang et al., 2023). For example, Nies et al. (2021) defined the distance as the optimal transport cost on the spaces of the aforementioned distributions for a restricted set of cost functions. Other popular approaches study the distance covariance, which is the weighted Euclidean distance between the joint characteristic function and the product of the marginals' characteristic functions (Székely et al., 2007). Also, Reshef et al. (2011) introduced MIC, a measure of dependence for the 2 -variable relationships based on testing multiple grids with varying dimensions and searching for the grid with maximum information between the two variables. Notably, (Liu et al., 2021) benefit from the optimal transportation theory to calculate Squared-loss Mutual Information with a small number of paired samples (drawn from the joint distribution) and a large number of unpaired samples. Lastly, MID (Sugiyama and Borgwardt, 2013) uses the concept of fractal dimensions (Ott, 2002) to define the information dimension and shows benefits in various applications.

In the context of sliced statistical distances, Nguyen et al. (2021) proposed a similar solution to alleviate the problem of using a large number of slices and applied it to the Wasserstein (W-) distance by reformulating the distance as an optimization problem. While we share a similar motivation, the two slicing approaches have significant differences stemming from the fact that Sliced W-distances have only been studied in the context of generative modeling; they only require one slicing direction; and their cost function (1-Wasserstein distance) is continuous, which makes their problem a vanilla expectation maximization (EM) problem. In a different domain, Nguyen et al. (2020) proposed an improvement to relational regularized autoencoders by finding an important area of projections characterized by a von Mises-Fisher (vMF) distribution (Jupp and Mardia, 1979). Optimizing over the family of vMF distributions helps to identify important directions on the sphere ( ϵ parameter) and how much weight to put there compared to other directions ( κ parameter). We, however, do not restrict our optimization to a particular family of distributions since the optimal slicing policy might not take the form of vMF.

6 Experiments

We conduct extensive experiments to demonstrate SI ∗ 's efficacy. We compare SI ∗ 's dependence quantification quality to a diverse set of competitive baselines. Through careful analyses, we show SI ∗ 's scalability, computation speed, sample efficiency, and slicing efficiency. We further provide empirical evidence that SI ∗ works excellently across challenging representation learning and rein-

10

10 2

10 3

n

.5

.6

.7

.8

.9

1

AUC

10

10 2

10 3

m

10

10 2

10 3

10 4

n

Elliptical

10

10 2

10 3

m

Elliptical

forcement learning tasks. We refer to Appendix B for further analysis experiments and Appendix D for implementation details and hyperparameters provided at https://bit.ly/3foLke2 .

6.1 Effectiveness of SI ∗ as a Dependence Measure

We first illustrate the superiority of SI ∗ as a dependence measure by comparing against the following state-of-the-art measures: Mutual Information (MI) (Shannon, 1948; Kraskov et al., 2004), Sliced Mutual Information (SI) (Goldfeld and Greenewald, 2021), distributional sliced Wasserstein Dependence (WD) (Nguyen et al., 2021), Transport Dependency Coefficient (TD) (a.k.a Transport Correlation) (Nies et al., 2021), Mutual Information Dimension (MID) (Sugiyama and Borgwardt, 2013), Maximal Information Coefficient (Reshef et al., 2011), and Distance Correlation (dCor) (Székely et al., 2007). We perform ROC curve analysis to examine precision and recall in recognizing complex relationship types with the existence of uniformly distributed noise. We test on synthetic data that comprises positive samples that depict extremely complicated structures of P X,Y ∈ P ( R 3 × R 3 ) . In this setting, we conduct 10 statistical tests and cover a wide range of structures, including sinusoidal features, elliptical surfaces, spherical harmonic surfaces, and hyper-hypocycloids. We provide the full details (e.g. the mappings formulae) in Appendix D.2.

Results are reported in Figure 2 as a function of the noise ratio. Notably, SI ∗ surpassed all other dependency measures in discerning structure from noise and sometimes by a significant margin (up to 30% improvement in accuracy). Although SI ∗ has shown similar performance to some of the baselines in the linear and parabolic settings, a clear superiority of SI ∗ is manifested in the eight complicated geometries, such as the spherical harmonics and sinusoidals. As a result, Figure 2 clearly shows the advantages of our slicing optimality in highly non-trivial settings.

6.2 Sample and Slicing Efficiency of SI ∗

Supporting our theory in Section 3.3, we show that SI ∗ scales efficiently to higher dimensions. We test against SI and MI estimators (Kraskov et al., 2004) on two structures of common signal and elliptical. We calculate the AUC of the ROC while varying the number of samples or projections. Figure 3 shows that SI ∗ attains the perfect accuracy with less number of slices compared to the uniform-sampling-based baselines. As such, this empirical result supports our claim that SI ∗ succeeds in detecting the dependence between the variables with an improved sample and projection efficiency. Appendix B.2 provides additional experiments in higher dimensions.

6.3 Result on Representation Learning

Representation learning employs MI to discover useful representations by training a neural network encoder to maximize MI between its inputs and outputs. For example, Hjelm et al. (2019) introduces Deep InfoMax (DIM), a popular method for learning unsupervised representations, which improves the learning by including knowledge about the local structure of the input into the objective. We follow DIM settings and substitute MI with SI and SI ∗ as the information measure, respectively. We test these three methods along with BiGAN (Donahue et al., 2016) on the STL-10 (Coates et al., 2011) and CIFAR10 (Krizhevsky et al., 2009) datasets, which consist of high-dimensional images. Results in Tables 1 and 2 show that SI ∗ outperforms the baselines for all classifiers: high-level vector

representation ( Y ), the output of the previous fully-connected layer (fc) and the last convolutional layer (conv). As such, this experiment highlights the scalability aspect of SI ∗ with high-dimensional variables not to mention its excellent performance on a challenging ML task.

6.4 Result on Reinforcement Learning

Previous frameworks have studied MI objectives as regularizers to the reinforcement learning (RL) objective that involves high-dimensional variables of states, actions, and rewards (Nachum et al., 2019; Schwarzer et al., 2021). Since it is practically difficult to estimate MI in high dimensions, most works resort to learning representations of the high-dimensional data by projecting them into low-dimensional embeddings. Thanks to the slicing technique, these representations are no longer needed when using SI ∗ . We empirically evaluate the scalability of SI ∗ with an RL task, where we adapt the forward information objective in Rakelly et al. (2021) as:

where π denotes the agent's policy, A t ∼ π ( S t ) , S t +1 ∼ T ( S t , A t ) , and T is the transition dynamics. For SI, we use the same objective (by replacing SI ∗ with the dependence measure SI ( . ; . ) ). We run tests on three challenging Gym environments (Brockman et al., 2016), where the environment dimensions (observation dim, action dim) are as follows: Humanoid-v3: (378 , 17) ; Ant-v3: (113 , 8) ; Hopper-v3: (12 , 3) . Further details can be found in Appendix D.3. We report the results comparing SI ∗ , SI, and the original paper objective (MI). Figure 4 shows clear gains (up to 25% ) with a noticeable learning speed of SI ∗ compared to the baselines.

0.0

0.2

0.4

0.6

0.8

1

Timesteps (1e7)

0

2000

4000

6000

8000

10000

Return

(a) Humanoid-v3

0.0

0.2

0.4

0.6

0.8

1

Timesteps (1e7)

0

1000

2000

3000

4000

5000

6000

7000

8000

0.0

0.2

0.4

0.6

0.8

1

Timesteps (1e7)

1000

1500

2000

2500

3000

3500

4000

4500

6.5 Behavior Analysis

Convergence of SI ∗ . We validate the convergence rate of SI ∗ (Theorem 4). In Figure 5, we show the RMSE between the ground truth SI ∗ and the estimated in the setting where X and Y are normal random variables with 5 overlapping entries: Z ∼ N (0 , I 15 ) , Λ ∼ N (0 , 0 . 1 I 10 ) , X = Z [1 , 10] , Y = Z [6 , 15] +Λ . The heat map in Figure 5a shows convergence when n, m ( # samples and # slices, respectively) vary independently, while Figure 5b shows when one parameter is varying and the other is set to 10 3 . The MI estimator used is the Kozachenko-Leonenko estimator (Kraskov et al., 2004). We also test with X,Y ∈ R d with linear dependence to study the effect of the dimension on the convergence. Results are reported in Figure 5c with n and m varying together.

Computational complexity. Results in Figure 6 (left) show that the computational complexity of SI ∗ is of an order similar to that of SI . Note that, while SI ∗ is slower than SI , the difference is

10

32

100

320

1000

Number of Samples

10

32

100

320

1000

Number of Slices

0

.

05

0

.

10

0

.

15

0

.

20

RMSE

(a)

10

10 2

10 3

n (resp. m)

10

3

10

2

10

1

10 0

RMSE

SI

*

(m=10 3 , n varies)

SI

*

(n=10 3 , m varies)

MI

(b)

10

10 2

10 3

n=m

10

2

10

1

10 0

RMSE

d=10

d=100

(c)

insignificant not to mention the clear advantage of SI ∗ over other baselines given its superiority in detecting complex dependencies. We also analyze the effect of number of gradient update steps of f 1 , f 2 on the computation speed (Figure 6 (right)). Further results can be found in Appendix B.

13

2

9

2

10

2

11

2

12

2

13

n

2

-

3

2

-

2

2

-

1

2

0

2

1

Second

SI (

m

= 700)

SI

∗

(

m

= 700)

SI (

m

= 1000)

SI

∗

(

m

= 1000)

2

9

2

10

2

11

2

12

2

n

2

-

3

2

-

2

2

-

1

2

0

2

1

2

2

Second

10 Iteration

20 Iteration

50 Iteration

7 Concluding Remarks

We presented a novel dependence measure, called SI ∗ , that is scalable to high dimensions while being efficient regarding time, sample, and slicing complexity. We discussed its theoretical properties, proved that its estimation error depends on the problem dimensions only up to a constant factor, and empirically validated the competence of SI ∗ in detecting complicated dependencies against state-of-the-art dependence measures. We further placed SI ∗ into modern ML, where we proved the adequacy of our dependence measure on a more onerous set of tasks.

Limitations. Since the aim of this paper was to introduce optimality into slicing methods, SI ∗ is formalized to use two slicing variables. Although the current setting has proved effective and rather sufficient in complex machine learning scenarios, the extension into multivariate settings is encouraged for future works.

Broader impact. The findings and methodologies developed through this work can significantly improve the way we understand and analyze systems with complex relations. We believe this work will have noticeable impact on broader communities, since the study of quantifying information is active in research areas including physics, statistics, computational biology, economics, and neuroscience. At its current form, this work is mainly theoretical and does not impose any negative societal impact.

References

Bell, C. (1962). Mutual information and maximal correlation as measures of dependence. The Annals of Mathematical Statistics , pages 587-595.

Billingsley, P. (2013). Convergence of probability measures . John Wiley & Sons.

Bonneel, N., Rabin, J., Peyré, G., and Pfister, H. (2015). Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision , 51(1):22-45.

Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv preprint arXiv:1606.01540 .

Chikuse, Y. (2012). Statistics on Special Manifolds , volume 174. Springer Science & Business Media.

Chuang, C.-Y., Jegelka, S., and Alvarez-Melis, D. (2023). Infoot: Information maximizing optimal transport. In International Conference on Machine Learning , pages 6228-6242. PMLR.

Coates, A., Ng, A., and Lee, H. (2011). An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings.

Donahue, J., Krähenbühl, P., and Darrell, T. (2016). Adversarial feature learning. arXiv preprint arXiv:1605.09782 .

Donsker, M. D. and Varadhan, S. S. (1975). Asymptotic evaluation of certain markov process expectations for large time, i. Communications on Pure and Applied Mathematics , 28(1):1-47.

Fujimoto, S., Hoof, H., and Meger, D. (2018). Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587-1596. PMLR.

Ghimire, S., Masoomi, A., and Dy, J. (2021). Reliable estimation of kl divergence using a discriminator in reproducing kernel hilbert space. Advances in Neural Information Processing Systems , 34:10221-10233.

Ghourchian, H., Gohari, A., and Amini, A. (2017). Existence and continuity of differential entropy for a class of distributions. IEEE Communications Letters , 21(7):1469-1472.

Godavarti, M. and Hero, A. (2004). Convergence of differential entropies. IEEE Transactions on Information Theory , 50(1):171-176.

Goldfeld, Z. and Greenewald, K. (2021). Sliced mutual information: A scalable measure of statistical dependence. Advances in Neural Information Processing Systems , 34.

Goldfeld, Z., Greenewald, K., Nuradha, T., and Reeves, G. (2022). k-sliced mutual information: A quantitative study of scalability with dimension. arXiv preprint arXiv:2206.08526 .

Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR.

Hamma, A., Giampaolo, S. M., and Illuminati, F. (2016). Mutual information and spontaneous symmetry breaking. Phys. Rev. A , 93:012303.

Han, Y., Jiao, J., Weissman, T., and Wu, Y. (2020). Optimal rates of entropy estimation over lipschitz balls. The Annals of Statistics , 48(6):3228-3250.

Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. (2019). Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations .

Ioffe, S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. PMLR.

Jupp, P. E. and Mardia, K. V. (1979). Maximum likelihood estimators for the matrix von mises-fisher and bingham distributions. The Annals of Statistics , 7(3):599-606.

Kraskov, A., Stögbauer, H., and Grassberger, P. (2004). Estimating mutual information. Phys. Rev. E , 69:066138.

Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25.

Kullback, S. and Leibler, R. A. (1951). On information and sufficiency. The annals of mathematical statistics , 22(1):79-86.

Levy, P. S. (1955). Théorie de l'addition des variables aléatoires. The Mathematical Gazette , 39:344.

Liu, Y., Yamada, M., Tsai, Y.-H. H., Le, T., Salakhutdinov, R., and Yang, Y. (2021). Lsmi-sinkhorn: Semi-supervised mutual information estimation with optimal transport. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part I 21 , pages 655-670. Springer.

Loeve, M. (2017). Probability theory . Courier Dover Publications.

Marinoni, A. and Gamba, P. (2017). Unsupervised data driven feature extraction by means of mutual information maximization. IEEE Transactions on Computational Imaging , 3(2):243-253.

Mescheder, L., Nowozin, S., and Geiger, A. (2017). Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In International Conference on Machine Learning , pages 2391-2400. PMLR.

Mordant, G. and Segers, J. (2022). Measuring dependence between random vectors via optimal transport. Journal of Multivariate Analysis , 189:104912.

Nachum, O., Gu, S., Lee, H., and Levine, S. (2019). Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Representations .

Nadjahi, K., Durmus, A., Chizat, L., Kolouri, S., Shahrampour, S., and Simsekli, U. (2020). Statistical and topological properties of sliced probability divergences. Advances in Neural Information Processing Systems , 33:20802-20812.

Nguyen, K., Ho, N., Pham, T., and Bui, H. (2021). Distributional sliced-wasserstein and applications to generative modeling. In International Conference on Learning Representations .

Nguyen, K., Nguyen, S., Ho, N., Pham, T., and Bui, H. (2020). Improving relational regularized autoencoders with spherical sliced fused gromov wasserstein. arXiv preprint arXiv:2010.01787 .

Nies, T. G., Staudt, T., and Munk, A. (2021). Transport dependency: Optimal transport based dependency measures. arXiv preprint arXiv:2105.02073 .

Ott, E. (2002). Chaos in dynamical systems . Cambridge university press.

Piera, F. J. and Parada, P. (2009). On convergence properties of shannon entropy. Problems of Information Transmission , 45(2):75-94.

Popoviciu, T. (1935). Sur les équations algébriques ayant toutes leurs racines réelles. Mathematica , 9(129-145):20.

Posner, E. (1975). Random coding strategies for minimum entropy. IEEE Transactions on Information Theory , 21(4):388-391.

Rakelly, K., Gupta, A., Florensa, C., and Levine, S. (2021). Which mutual-information representation learning objectives are sufficient for control? Advances in Neural Information Processing Systems , 34.

Rényi, A. (1959). On measures of dependence. Acta Mathematica Academiae Scientiarum Hungarica , 10:441-451.

Reshef, D. N., Reshef, Y. A., Finucane, H. K., Grossman, S. R., McVean, G., Turnbaugh, P. J., Lander, E. S., Mitzenmacher, M., and Sabeti, P. C. (2011). Detecting novel associations in large data sets. science , 334(6062):1518-1524.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 .

Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., and Bachman, P. (2021). Dataefficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations .

Shannon, C. E. (1948). A mathematical theory of communication. The Bell system technical journal , 27(3):379-423.

Sønderby, C. K., Caballero, J., Theis, L., Shi, W., and Huszár, F. (2016). Amortised map inference for image super-resolution. arXiv preprint arXiv:1610.04490 .

Song, J. and Ermon, S. (2020). Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations .

Sugiyama, M. and Borgwardt, K. M. (2013). Measuring statistical dependence via the mutual information dimension. In Twenty-Third International Joint Conference on Artificial Intelligence . Citeseer.

Székely, G. J., Rizzo, M. L., and Bakirov, N. K. (2007). Measuring and testing dependence by correlation of distances. The annals of statistics , 35(6):2769-2794.

Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033.

Villani, C. (2009). Optimal transport: old and new , volume 338. Springer.

Wiesel, J. (2021). Measuring association with wasserstein distances. arXiv preprint arXiv:2102.00356 .

Yang, P. and Chen, B. (2018). Robust kullback-leibler divergence and universal hypothesis testing for continuous distributions. IEEE Transactions on Information Theory , 65(4):2360-2373.

Yoo, A. B., Jette, M. A., and Grondona, M. (2003). Slurm: Simple linux utility for resource management. In Workshop on job scheduling strategies for parallel processing , pages 44-60. Springer.

Peer reviews:
comments: SUMMARY:
The paper estalishes a new quantity for measuring the dependence between random variables. The definition is similar to mutual information and sliced mutual information. As (for rather obvious reasons) it takes values between these two existing measures, it has some desireable properties. A variational representation further makes it better compatible with neural estimators.

STRENGTHS:
The paper is well written, provides good references and approaches the challenging gap between theory of information measures and their estimation in practice.

WEAKNESSES:
It is difficult to see the benefit of the specific definition (2). There is no convincing argument why (2) is better than (1). Also an optimization over a set of prabability measures appears to be much more difficult to approximate than the comparatively "simple" integration over the uniform measure.
Many details are in appendices that I unfortunately have no time to read in the limited time given for the review process of many papers.

QUESTIONS:
Are there methods that try to approximate (1) by other means than uniform sampling? There are many methods for numerical integration that might also use the fact that not all directions are equally useful in approximating the integral.
Why is it necessary to change the definition of (1) and not simply try to use a better method to approximate (1) numerically?
Does there exist a one-dimensional MI estimator with finite \delta(n)? If not, can the supremum in the characterizing property of \delta(n) be restricted to a class where it is finite and all further results hold for this class?

LIMITATIONS:
Limitations were not addressed in the paper but discussed in an appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecently, sliced mutual information ($\\mathcal{SI}$) has been proposed as a statistical dependence measure that is scalable to high dimensions (Goldfeld and Greenewald, 2021). However, due to the use of uniform sampling for the slicing directions, $\\mathcal{SI}$ generally requires many slices for accurate estimation. This is because it doesn’t prioritize informative features about the variables and introduces noisy slices by treating both relevant and irrelevant regions of the variable space equally. This paper aims to improve $\\mathcal{SI}$ by finding an optimal distribution of slicing directions that satisfies two criteria: 1) slices are distributed over maximally informative regions, and 2) slices are scattered over the unit sphere. This new dependence measure is called optimal sliced mutual information, $\\mathcal{SI}^*$. The two criteria are then formalized as a regularized optimization problem in the definition of $\\mathcal{SI}^*$.\n\nThe contributions of this paper are as follows: (i) The authors introduce an improved version of $\\mathcal{SI}$ known as $\\mathcal{SI}^*$ and discuss its theoretical properties. In particular, they show that an optimal slicing policy exists, allowing them to treat the regularized optimization problem as a reward-maximizing optimal transport problem. (ii) The authors show that the convergence rate of the optimal estimator of  $\\mathcal{SI}^*$ depends on the dimensions only up to a constant factor. In addition, they develop a neural estimator for $\\mathcal{SI}^*$ which can be used in various machine learning problems. (iii) The authors perform several experiments to show the effectiveness and applications of $\\mathcal{SI}^*$. In particular, they show that: (a) $\\mathcal{SI}^*$ outperforms state-of-the-art measures in discerning structure from noise, (b) $\\mathcal{SI}^*$ is more sample- and slice-efficient compared to $\\mathcal{SI}$, and (c) $\\mathcal{SI}^*$ performs well on representation and reinforcement learning problems.\n\nSTRENGTHS:\nThe paper improves $\\mathcal{SI}$ by challenging the uniform sampling assumption in the original definition. The paper is original as it proposes a novel, improved measure of $\\mathcal{SI}$ that can capture statistical dependency better by retaining as much relevant information as possible. The theoretical result on the existence of the “optimal” slicing policy is interesting and non-trivial. The experiments clearly showcase the effectiveness of $\\mathcal{SI}^*$ as a dependence measure and how it can be applied to various machine learning tasks. The paper shows how $\\mathcal{SI}^*$ is an effective statistical dependence measure that works well in high dimensions. As many machine learning tasks involve high-dimensional variables, this measure will prove useful in practice. Overall, the paper is well-structured and was an interesting read.\n\nWEAKNESSES:\nWhile the paper has some interesting points, there are several weaknesses that should be addressed to make the paper more readable.\n\n1. While the definition is $\\mathcal{SI}^*$ is intriguing, the justification for the two criteria for “optimality” that motivated the definition is inadequate. (i) Why is it relevant or important to consider the maximally informative regions? (ii) If we accept that the maximally informative regions are important, why do we need criterion 2? The authors should explain more clearly why diversity of slices is important when measuring dependence between random variables.\n\n1. From the definition of $\\mathcal{SI}^*$, wouldn’t it be better to call it something like “maximum $\\mathcal{SI}$”?\n\n1. In Figure 2, the experiments are done for variables in $\\mathbb{R}^3$. However, $\\mathcal{SI}^*$, like $\\mathcal{SI}$, should continue to be scalable to high dimensions. Thus, the experiments should be done in higher dimensions (> 100), so we can test if the three neural networks involved in the estimation of $\\mathcal{SI}^*$ scale well with larger dimensions of inputs. \n\n1. Parts of the paper were hard to understand and could have been explained more clearly. (i) For example, the terms in the neural network estimator (Section 4) need more details and better explanation. (ii) In the motivating example (Section 3.1), it is not clear what the AUC of the ROC refers to or how it is computed in this case. The Appendix provides some details for another experiment but it is not clear if that applies here. Also, it seems that Fig. 1 is not an ROC AUC curve despite the caption stating as such. (iii) The setup and assumptions for the experiments require more details beyond that given in the paper and appendix. \n\n1. Since one claim in the paper is better sample and slice efficiency of $\\mathcal{SI}^*$, is it possible to provide theoretical results along this direction? The results in the paper seem to indicate that $\\mathcal{SI}^*$ has a similar convergence rate as $\\mathcal{SI}$, thus reducing its significance. \n\n1. Another claim in the paper is the potential of $\\mathcal{SI}^*$ to bring the approach to more complex machine learning scenarios. In this context, more experimental evaluation is necessary for both the representation learning and reinforcement learning scenarios. The current evaluation (e.g., Table 1) is not to the state of the art, in fact, a shallow vanilla CNN without data augmentation has 86% accuracy on CIFAR-10. This makes it difficult to interpret the significance of the results.\n\nQUESTIONS:\nIn addition to the list of weaknesses above, below are some questions and general suggestions for the authors to address:\n\n1. In Goldfeld and Greenewald’s paper, they show that $\\mathcal{SI}$ can increase with more processing of the random variables. Can $\\mathcal{SI}^*$ increase with more processing as well?\nCan we apply chain rule to $\\mathcal{SI}^*$?\n\n1. In Appendix B.1, the authors computed the RMSE between the ground truth and empirical values of MI and $\\mathcal{SI}^*$ as a function of the number of samples and slices (where applicable). Please include the relevant results for $\\mathcal{SI}$ as well.\n1. In Appendix B.1, the authors computed the RMSE between the ground truth $\\mathcal{SI}^*$ and the estimated $\\mathcal{SI}^*$. It is not clear how the ground truth $\\mathcal{SI}^*$ is computed in this case. In Goldfeld and Greenewald’s paper, they provided an expression for Gaussian $\\mathcal{SI}$ (Example 1). Is it possible to come up with similar expression for $\\mathcal{SI}^*$?\n\n1. In Appendix C, it seems that the algorithm is lacking in detail and clarity. For example, the authors should make it clear that $m$ number of random slices are being sampled. It is also not clear in the algorithm what the loss functions used for training the various neural networks are. Overall, the description of the algorithm seems a bit scattered and could be significantly made clearer.\n\n1. Goldfeld and Greenewald have considered extending $\\mathcal{SI}$ to $k-\\mathcal{SI}$ where instead of projecting to 1 dimension, the random variables are projected to $k$ dimensions. Can $\\mathcal{SI}^*$ be easily extended to the $k$ dimensions case?\n\n1. Minor comments: (i) Fix caption of Fig. 1 to not refer to the ROC curve. (ii) In Appendix D.1, $\\phi$ should be $\\varphi$.\n\nLIMITATIONS:\nThe limitations are discussed in Appendix E.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This work based on existing ideas of providing a tractable independence metric for high dimensional random variables, the proposed metric named "Optimal sliced mutual information" (SI*) improves upon existing method of uniform slicing to an adaptive optimal slicing with scalability design in mind. It has been shown that the proposed method has desirable properties to be a dependency metric. The estimator is shown to be faster converging than previous MI both theoretically and empirically. In addition, this method is applied on several machine learning tasks to show its effectiveness.

STRENGTHS:
This work discovers weakness in previous work in terms of slicing efficiency. The idea is well motivated and explained well in introductory sections. The properties of the proposed metric is well explained with easy to understand examples and visualization. Examples in the empirical study showcases various in field applications which implies wide impact upon adoption.

WEAKNESSES:
The scalability of the NN estimator of SI* is only studied at slicing size level. A remark about the scalability with $d_x, d_y$ similar to convergence analysis can be beneficial.

QUESTIONS:
There are multiple places where SI and $\mathcal{SI}$ are used interchangeably. e.g. l.50 l.282 Table 1, l.331 and more.
The provided implementation is short scaffolding code and missing many important pieces which prevents further cross-check of the implementation against the claim in the paper.

LIMITATIONS:
Limitations are well addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new dependence measure based on the sliced mutual information with both the theoretical analysis and the modern machine learning experiments. This measure is not strictly the mutual information but still could capture the correlation between two distributions. \nThe experiments are done in several different domains in eluding image classification, reinforcement learning. Their main idea is to replace the original uniformly sampled slices to the one whose directions could be diversified over the whole sphere as well as mainly concentrates on the interested area.\n\nSTRENGTHS:\nThe article is well written, starting from an easy understanding toy example, followed by the motivation and explanation. The authors also clarify several important concepts that the reviewer will raise during the reading. The theoretical analysis is detailed and clearly written. \n- The observation of the the uniform distribution in sliced mutual information is insightful\n- Extensive experiments are done in different domains\n\nWEAKNESSES:\nHave two minor concern and hope to hear from the authors,\n- In the theoretical part, the authors prove the existence of the optimal solution using the optimal transport's view. However, during the neural estimation, the problem is simplified to gradient descend with Lagrange constraints. The hyper parameters listed in equation (6) are not clearly described. \n - The efficiency is not as good as the previous methods (ten times comparing with SI. Which factors cause the computation speed to be slow?\n\nQUESTIONS:\n- Could you please explain what is the definition of AUC that shown in this paper. Does the evaluation treat as a binary classification of dependence and independence? Since this metrics are used in at least two experiments, could you please put more words on this part? \n- Many different mutual information estimator are mentioned in the paper, including the discriminator-based form and variational representation learning. There are also many other mutual information estimator, why did the author choice MINE[1]? \n- What neural estimator do you use in Table 1 DIM(MI)? \n- How do you choose $\\lambda_1$, $\\lambda_2$, is there any ablation study on this part? \n\n\n\n[1] Mutual Information Neural Estimation, 2021\n\nLIMITATIONS:\nNo negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}