Sliced Mutual Information: A Scalable Measure of Statistical Dependence

Abstract: Mutual information (MI) is a fundamental measure of statistical dependence, with a myriad of applications to information theory, statistics, and machine learning. While it possesses many desirable structural properties, the estimation of high-dimensional MI from samples suffers from the curse of dimensionality. Motivated by statistical scalability to high dimensions, this paper proposes sliced MI (SMI) as a surrogate measure of dependence. SMI is defined as an average of MI terms between one-dimensional random projections. We show that it preserves many of the structural properties of classic MI, while gaining scalable computation and efficient estimation from samples. Furthermore, and in contrast to classic MI, SMI can grow as a result of deterministic transformations. This enables leveraging SMI for feature extraction by optimizing it over processing functions of raw data to identify useful representations thereof. Our theory is supported by numerical studies of independence testing and feature extraction, which demonstrate the potential gains SMI offers over classic MI for high-dimensional inference.


Reviews and author responses:

 I am fully satisfied with the answers provided by the authors. I think that the authors did a tremendous job in taking into account all the reviews and answering the question and telling what can be added for  the final version.

About my question (2) relative to dispersions in 4.1 and 4.2: I was thinking exactly about Fig 1 and 2, and  was wondering about the std  (or other dispersion measurement) when repeating the MC trials, i.e. not reporting only the mean over the 200 trials, but also the dispersion. Now, if it's small, it's not relevant.

I concur with the other reviewers that adding some elements about why the violation of DPI is a relevant feature will improve the work.

As a conclusion, I maintain my score recommanding acceptance.
 

 Thank you for your detailed response. I am satisfied with the answers to my concerns, and I think the discussion on the curse of dimensionality would be a nice addition to the paper, as it helps provide a more nuanced picture of Theorem 1 and Corollary 3.

After reading the other reviews, I would also like to encourage the authors to further clarify why the violation of DPI by SMI is not a negative feature in the context of their work. 

I am keeping my initial score and recommend acceptence.

The paper presents an approximation to the high dimensional mutual information by repeatedly randomly projecting the high dimensional variables down to a single dimension and computing the mutual information in the low dimensional space.  The notion that breaking the data processing inequality is a useful property is odd. If the measure doesn't conform to the DPI then that means it's approximation to the mutual information becomes increasingly poor. The fact that neural networks have layers of features which progressively improve prediction performance is an issue with the large space the initial data ends up in, and the fact that the mutual information requires exponential quantities of compute and data to estimate high dimensional informations. The neural net isn't finding a more informative representation, it's finding one with a more easily understood manifold. I agree that the sliced projection is also likely to force things to have more understandable manifolds, and that this might be a desirable quality, but it's not because it doesn't obey the DPI. The information isn't created by computation, it's made more accesible (or inaccessible) by computation.

I don't think that pairwise independence is sufficient for property 5 in Proposition 1 to hold. Pairwise independence doesn't preclude larger interactions between sets of variables, only that each pair has no measurable interaction. As with the regular mutual information, the presence of additional variables can cause previously independent variables to become dependent (e.g.  spouse nodes in a Bayesian Network become dependent when conditioning on the shared child node). Presumably it requires mutual independence?

In section 3.1 the SMI estimator is claimed to be free from the curse of dimensionality. The error bound for it given in corollary 2 has a constant term which depends on the dimensions of both of the inputs and is unspecified.  This would seem to reintroduce the curse of dimensionality into the estimation.

In section 3.2 the term linear seems overloaded. I think it's used to mean the features are linear projections, but the interaction between the two projections could be non-linear and it might be useful to clarify this point.

Also in section 3.2, the variable Z appears in line 218, but has not previously been defined. I would assume it means X_1 as it's the partition of X into X_2 (i.e. Y) and some other variable, but this should be made clear. The text just before proposition 4 states that the most informative feature in X is X_1, even though Y is a copy of X_2 and so X_2 is maximally informative about Y.

Update:

Thanks for the response. I think that the clarifications around the DPI are necessary so as not to confuse the reader, and the responses to the reviews wrt the curse of dimensionality also are necessary updates, but overall I think this paper is a useful contribution. The experimental study only partially covers the parameter space of the measure.  The experiments in 4.1 would be more useful if the number of samples and slices were allowed to vary independently of each other, to see if there is anything complicated in the interaction.

 We thank the reviewer for the considerable number of hours spent on our manuscript and for the resulting feedback. Below we address the specific comments/concerns raised.

1. __Originality:__ We believe that SMI is novel and that its exploration as a measure of dependence goes beyond the resolution of mere (sliced) statistical divergences. Similarly to how Shannon's MI has properties that go beyond the fact that is can be defined via KL divergence. In both cases, the specific choice of distributions between which the divergence is measured, namely $P_{X,Y}$ versus $P_X\otimes P_Y$ is key and unlocks additional properties that these measure of dependence posses (e.g., chain rule, decompositions, etc.).

    Concerning the estimation base on on DV representation, we first note that this is not the only estimator considered in the paper. Section 3.1 proposes the $\hat{\mathsf{SI}}_{n,m}$ estimate based on a scalar MI estimator and Monte Carlo integration. This is the estimator that we mostly focus on and for which we provide theoretical guarantees. The DV-based one is only mentioned and used in Section 4.3 (for feature extraction). That said, we think that this neural estimator is interesting and its convergence rates warrant a thorough theoretical exploration, which we plan to do in future work. We do not view the fact that similar techniques were previously used for MI estimator as problematic or colliding with our work in any way.

2. __Large-scale experiments and computational complexity.__ We agree with the reviewer that downstream experiments utilizing SMI is an interesting and promising future direction. However, this goes beyond the scope of this 9 page paper, which is primarily a theoretical work aiming to inspire practical followup works of exactly that type. Indeed, we are intensely interested in exploring such applications ourselves. Proof-of-concept experiments are included here for illustration of the theoretical bounds; not (yet) as proof of practical engineering usefulness. We thank the reviewer for the long list of references, all of which we will explore as possible research avenues. We will also add mentions to those in the summary section of the revised manuscript. 

    Regarding computational complexity, in the revision we plan to add a pseudo-code of the SMI estimator along with a comment on its computational complexity ($O(m (d_x+d_y) n + m A(n))$ where $A(n)$ is the computational complexity of the MI estimator used to compute the the MI between 1D slices). 

3. __Kozachenko-Leonenko estimator reference:__ The Kraskov et al. paper mentioned by the reviewer is indeed cited in out work (see reference [15]). As suggested, we will use that reference in Section 4.2, rather than [24] that is currently cited there. We preferred to use the Kozachenko-Leonenko-based SMI estimator (as opposed to the DV-based one) when possible since it falls under the framework of Section 3.1 and is therefore coupled with the theoretical performance guarantees derived therein. 

4. __Definition of SMI is repeated twice:__ The first mention of SMI in the introduction is informal and is there merely to give a since of the proposed quantity (before the formal definition). The formal definition is given only once, in Definition 1 on page 3. 

5. __Giving the content away:__ We do not understand what the reviewer means here. The goal of an introduction section is to give an overview of the paper and its contributions. This is exactly what our introduction does.

6. __Breaking DPI:__ Our intention in discussing the lack of a DPI for SMI is not to suggest that this in itself is intrinsically good. In a sense, the opposite is true: SMI is capable of finding a good representation of the data and as a result DPI is broken. MI is agnostic to those transformations and there cannot be used as an objective for, e.g., feature extraction. SMI, on the other hand, is a sensible objective for such tasks. We acknowledge that some of the current wording might be confusing. We will edit these discussions in the revision so as to make this point clear. 

7. __Figure formatting:__ We will expand the size of the figures so they are readable. We are not sure what the reviewer means by $\epsilon$.

8. __Wording:__ We agree that the statement that 'SMI can be created from computation' can be worded better. We only meant to say that processing can increase SMI (as opposed to MI that obeys DPI). We will clarify this in the revision. Regarding other terms mentioned by the reviewer ('figure of merit' and 'SMI nullifies')---these are standard English terms and we do not see a problem here. 

9. __Limitations and societal impact:__ The reviewer is concerned that the practical utility of SMI in large-scale tasks in yet to be demonstrated. The sliced Wasserstein distance is then given as an example of a slicing-based measure for which superior performance was shown on real-world datasets. We kindly remind to the reviewer that sliced Wasserstein distances were originally introduced more than a decade ago in 

    *J. Rabin, G. Peyr`e, J. Delon, and M. Bernot. "Wasserstein barycenter and its application to texture mixing."  In Scale Space and Variational Methods in Computer Vision (SSVM), 2011.*

    The above paper explored preliminary theoretical properties and provided empirical results on simple synthetic examples. It took several years until followup works demonstrated the utility of sliced Wasserstein distances for real-world tasks such as generative modeling. We view our paper similarly, as introducing a new measure of information, justifying its sensibility via a thorough theoretical (structural and statistical) exploration, and providing proof-of-concept experiments for illustration of the theory. We are confident that followup works will adopt SMI and demonstrate it usefulness on real-world tasks, and plan to pursue such research directions ourselves. A discussion to that effect will be added to the revision.

10. __Minor comments:__ Thank you for spotting these. We will introduce all the corrections in the revision.


 We thank the reviewer for the constructive and positive feedback. Below we address the specific comments/concerns raised.

1. __Two slicing directions:__ The reviewer asked why both $\Theta$ and $\Phi$ are used for slicing, as opposed to just using one of them for both variables. This is a very interesting question; thank you for bringing this up. The original formulation was inspired by the canonical correlation coefficient $\rho_{\mathsf{CCA}}(X,Y):=\sup_{\theta,\phi}\frac{\mathsf{cov}(\theta^\intercal X,\phi^\intercal Y)}{\sqrt{\mathsf{var}(\theta^\intercal X)\mathsf{var}(\phi^\intercal Y)}}$. While this corresponds more directly to the max version of SMI, i.e., $\mathsf{mSI}(X;Y):=\sup_{\theta,\phi}\mathsf{I}(\theta^\intercal X,\phi^\intercal Y)$, we kept the slicing across two directions in the average formulation, which is the main focus of the paper. There are several benefits to this formulation:
    1. It is invariant to rotations of the spaces in which $X$ and $Y$ take values. If only one slicing variable was used, then a rotation of either $X$ or $Y$ would change $\mathsf{SI}(X;Y)$, seemingly not a desirable property for a measure of information. 
    2. It naturally gives rise to one of the most important properties of SMI, which is that $\mathsf{SI}(X;Y)=0\ \iff X$ and $Y$ are independent. Our proof clearly relies on having two different slicing directions (see line 10 of the supplementary material) and we don't expect this property to hold with a single slicing direction.
    3. It fares more naturally with variables of different dimensions, although the reviewer is right that one can always pad with zeros. 

    We acknowledge that our two-directional slicing technique is notably different from traditional slicing methods and will add a discussion of the above after the definition of SMI. We will also look for an example that contradicts Property 1 from Proposition 1 when only a single slicing direction is used. 


2. __Dispersion in empirical results:__ Figures 1 and 2 are computed using 200 random repeated experiments (Monte Carlo trials), so error bars would be too small to see. Regarding dispersion, RMSE and AUC ROC are already dispersion measures themselves. It is therefore not clear to us what it would mean to create a trial-to-trial variation of these quantities. If the reviewer can clarify this point, we are happy to add empirical results to that effect. 

3. __Computational cost:__ We will add a pseudo-code of the SMI estimator to the supplement along with a comment on its computational complexity ($O(m (d_x+d_y) n + m A(n))$ where $A(n)$ is the computational complexity of the MI estimator used to compute the the MI between 1D slices). 

    For max SMI, since it involves optimization, we do not have an expression for its computational complexity at this time. We would also like to kindly stress that max SMI is *not* being used as an objective for feature extraction in the paper (only the average SMI is used). In fact, while $\mathsf{SI}(X;Y)$ can be used for feature extraction, Proposition 4 effectively shows that mSMI is not useful for that purpose since $\sup_{\mathrm{A}_x,\mathrm{A}_y,b_x,b_y} \mathsf{mSI}(\mathrm{A}_x X + b_x; \mathrm{A}_y Y + b_y)=\mathsf{mSI}(X;Y)$ and invariance, e.g., to linear extractors, is recovered. 


 We thank the reviewer for the constructive and helpful feedback. Below we address the specific comments/concerns raised.

1. __Two slicing directions:__ The reviewer asked why both $\Theta$ and $\Phi$ are used for slicing, as opposed to just using one of them for both variables. This is a very interesting question; thank you for bringing this up. The original formulation was inspired by the canonical correlation coefficient $\rho_{\mathsf{CCA}}(X,Y):=\sup_{\theta,\phi}\frac{\mathsf{cov}(\theta^\intercal X,\phi^\intercal Y)}{\sqrt{\mathsf{var}(\theta^\intercal X)\mathsf{var}(\phi^\intercal Y)}}$. While this corresponds more directly to the max version of SMI, i.e., $\mathsf{mSI}(X;Y):=\sup_{\theta,\phi}\mathsf{I}(\theta^\intercal X,\phi^\intercal Y)$, we kept the slicing across two directions in the average formulation, which is the main focus of the paper. There are several benefits to this formulation:
    * It is invariant to rotations of the spaces in which $X$ and $Y$ take values. If only one slicing variable was used, then a rotation of either $X$ or $Y$ would change $\mathsf{SI}(X;Y)$, seemingly not a desirable property for a measure of information. 
    * It naturally gives rise to one of the most important properties of SMI, which is that $\mathsf{SI}(X;Y)=0\ \iff X$ and $Y$ are independent. Our proof clearly relies on having two different slicing directions (see line 10 of the supplementary material) and we don't expect this property to hold with a single slicing direction.
    * It fares more naturally with variables of different dimensions, although the reviewer is right that one can always pad with zeros. 

    We acknowledge that our two-directional slicing technique is notably different from traditional slicing methods and will add a discussion of the above after the definition of SMI. We will also look for an example that contradicts Property 1 from Proposition 1 when only a single slicing direction is used. 


2. __Relation to sliced divergences:__ This is a great point and we agree that the paper needs to discuss it. SMI can indeed be obtained from sliced KL divergence with $P=P_{X,Y}$ and $Q=P_X\otimes P_Y$; this is the claim in Property 3 of Proposition 1. The Nadjahi et al. 2020 paper mentioned by the reviewer is an excellent reference on the topic, which we will cite in the revision. However, the sample complexity result of Theorem 1 therein doesn't imply our Theorem 1 since Nadjahi et al. plug in empirical measures to estimate the sliced divergence, while the framework of Theorem 1 is more general and allows for arbitrary scalar mutual information estimators. Nevertheless, there is a clear and interesting connection between SMI and sliced divergences, which we will explicitly discuss in the revision. In particular, as the reviewer suggested, we will address the similarities and differences between the definitions, results, utility, etc.

3. __Curse of dimensionality:__ We agree with the reviewer that clarification of the statement that 'SMI does not suffer from the curse of dimensionality' is due. What we mean is that $n$ and $d$ are decoupled in the convergence rate (unlike the $n^{-1/d}$ rate for classic MI). The ambient dimension can, of course, still enter bounds via the constant, but the effect of $d$ is significantly milder than in the classic case. Specifically, $d$ can enter the bound from Theorem 1 in two places, which we discuss below:
    * $d$ can enter is via the $\delta(n)$ term, which is the scalar MI (i.e., between projections) estimation error. We do not expect this term to have a strong dependence on $d$, which we validate in our experiments.
    * As the reviewer suggests, $d$ can also enter in the Monte Carlo estimation error. As can be seen from line 129 of the supplementary material, the variance of the SMI terms enters the estimation error analysis. We then further bound it by MI (Property 2 of Proposition 1), and then use the assumption that $\sup_{\theta, \phi} \mathsf{I}(\theta^\intercal X;\phi^\intercal Y)\leq M$ (in the current document we stated the assumption as being on $\mathsf{I}(X;Y)$, in the revision we will replace this with the above tighter and more accurate statement that we actually used in the proof). This bound effectively absorbs the dimension-dependent factors of the MC estimation error, and $M$ may very well depend on $d$. However we do not anticipate this dependence to be strong since the slice is still fundamentally a 1-dimensional MI. 
    * Lastly, we note that there may be dependence on dimension of the constant $C$ from Corollary 2 (which is associated with the specific instantiation of the scalar MI estimator). Specifically, we use Theorem 2 from [25] particularized to the case where distributions are sub-Gaussian. The resulting constant therein may depend on $p,s,d,L,\beta$, where the latter is the sub-Gaussian constant. We apply the bound for the projected distributions $p_{X_\theta}$, $p_{Y_\phi}$, and $p_{X_\theta,Y_\phi}$, which are compactly supported and are therefore sub-Gaussian. Clearly, $p$ and $s$ are fixed, $d=1,2$, $L$ is independent of $d$, but $\beta$ is related to the support of the said distributions, and is thus dependent on $d$. 
    Nevertheless, we reemphasize that this constant is bounding the error of estimating scalar MI (of each slice), which is unlikely to grow fast with increasing ambient dimension. 

    These are all interesting and important points to discuss, which we will do in the revision, after stating Theorem 1 and Corollary 2.


2. __Scaling of existing MI and deep learning-based algorithms:__ We first point out that, as mentioned in the paper, the $n^{-1/d}$ convergence rate for MI estimation is minimax, i.e. all algorithms, including neural network based ones, cannot beat this rate at least in the worst case. Additionally, neural methods such as MINE have been shown to suffer from significant limitations, e.g., in

    *D. McAllester and K. Stratos, "Formal limitations on the measurement of mutual information." International Conference on Artificial Intelligence and Statistics, 2020.*

    That said, neural network based MI algorithms often do prove useful in practical high-dimensional settings. This does not mean that they violate the theoretical bounds 
but rather that when insufficient samples are available they still compute some kind of useful measure of dependence. This computed measure is treated as a proxy for MI, but practically oriented papers often do not delve into how good of a proxy this actually is or quantify what exactly is being measured.

    SMI can also be thought of in this sense (as a useful proxy for MI), but our approach is markedly different. We propose as a new dependence measure in its own right, call and describe it as such, and then explore its structure to justify it and study its statistical properties to show it can be estimated with relative ease. We believe that this principled approach is crucial for clarity and transparency in the machine learning community, and provides a solid stepping stone for applications to come. 


3. __Minor comments:__ Thank you for spotting these. We will introduce all the corrections in the revision.

4. __SMI versus MI performance in Figure 1:__ This is a great point and we will gladly include results to that effect in the revision. Specifically, we will add estimation error convergence rates for high-dimension MI estimation via the kNN method from [15], the EDGE estimator from [16], as well as MINE [17]. In addition, convergence rates for SMI estimation via S-MINE (see Section 4.3) will be added to the current kNN-based estimate plots for completeness. Rates for these methods will be shown for several dimensions with the goal of empirically validating the scalability to high-dimensions versus lack thereof of SMI versus MI, respectively. 


5. __Proof of Corollary 3:__ Thank you for the comment. We will add the proof to the supplement in the revision.

6. __Discuss limitations and societal impact:__ We will add a discussion about SMI being a new measure of dependence between random variables that (although related) is different from classic MI. We will also highlight potential negative impacts of treating SMI as an estimate of MI and provide an example to support that. E.g., the example from Section 3.2 shows that for any $0<a<\infty$ we have $\mathsf{I}\big(g_a(X);Y\big)=\infty$ while $\mathsf{SI}\big(g_a(X);Y\big)$ is finite (note that there was a typo in the submitted version; the correct setup has $Y=X_1$ and no $Z$).


 We thank the reviewer for the constructive and helpful feedback. Below we address the specific comments/concerns raised.

1. __Breaking DPI:__ Our intention in discussing the lack of a DPI for SMI is not to suggest that this in itself is intrinsically good. In a sense, the opposite is true: like the reviewer suggests, SMI is capable of finding good manifolds and as a result DPI is broken. MI is agnostic to those manifolds and thus cannot be used as an objective for, e.g., feature extraction. SMI, on the other hand, is a sensible figure of merit for such tasks. We acknowledge that some of the current wording might be confusing. We will edit these discussions in the revision so as to make this point clear. 

    The reviewer also mentioned that lack of DPI means that the 'approximation of mutual information becomes increasingly poor'. We would like to kindly stress that SMI is *not* proposed as a proxy of MI but rather as a new measure of dependence between random variables. The theoretical exploration of SMI properties (Section 3) aims to justify it as a meaningful such measure, while the statistical study (Section 3.1) shows that SMI can be estimated efficiently even in high-dimensional settings. We will highlight this point in the revision to avoid any potential confusion. Generally, Shannon's MI is not the only possible measure of information, although it is probably the most popular. To name a few alternatives, there are Arimoto's [A], Csisz{\'a}r's [B], and Sibson's [C] mutual information formulations, all targeting a notion of $\alpha$-mutual information (via Renyi divergence).

    [A]  S. Arimoto, "Information measures and capacity of order $\alpha$ for discrete memoryless channels," in Topics in Information Theory, Proc. Coll. Math. Soc. J`anos Bolyai. Keszthely, Hungary: Bolyai, 1975, pp. 41–52.

    [B] I. Csiszar, "Generalized cutoff rates and R`enyi’s information measures," IEEE Trans. Information Theory, vol. 41, no. 1, pp. 26–34, Jan. 1995.

    [C] R. Sibson, "Information radius,'' Z. Wahrscheinlichkeitstheorie und Verw.
Geb., vol. 14, pp. 149–161, 1969.

2. __Pairwise versus mutual independence in Property 5:__ The reviewer is completely right: 'pairwise' is a typo that will be corrected to 'mutual' in the revision. The property effectively shows that SMI follows the same tensorization property as classic MI. Thank you for catching this error. 

3. __No curse of dimensionality.__ We agree with the reviewer that clarification of the statement that 'SMI does not suffer from the curse of dimensionality' is due. What we mean is that $n$ and $d$ are decoupled in the convergence rate (unlike the $n^{-1/d}$ rate for classic MI). The ambient dimension can, of course, still enter bounds via the constant, but the effect of $d$ is significantly milder than in the classic case. More specifically, as Theorem 1 shows, $d$ can only enter via the $\delta(n)$ term, which is the scalar MI (i.e., between projections) estimation error. We do not expect this term to have a strong dependence on $d$, which we validate in our experiments.

4. __Overloading the term 'linear':__ We agree with the reviewer that our wording can be better and will revise this part for clarity. Specifically, we will refer to this as 'linear processing' throughout. 

5. __Typo in the example from Section 3.2:__ The reviewer is correct, there is indeed a typo in that example: $Y$ should be $X_1$ and $Z$ should be $X_2$. This will be fixed in the revision. Thank you for noticing this.

6. __Limitations and societal impact.__ To better cover the parameter space, in the revision we will add a 3D plot of the RMSE versus values of the pair $(m,n)$. It would be interesting to see if any unexpected trends emerge. Thank you for the suggestion.

Mutual information (MI) is a measure of the statistical dependence between random variables, which has found application in several fields, including information theory, statistics, machine learning and more recently deep learning. However, in practice, MI is a computational bottleneck when the random variables of interest take values in high-dimensional spaces: its sample complexity (i.e., the error induced by the estimation of MI from $n$ samples with dimension $d$) is in $n^{-1/d}$.

This work proposes an alternative measure to MI that scales well to a large number of samples or dimension, called "sliced mutual information" (SMI). This novel measure of statistical dependence builds on the literature of "sliced" metrics: several recent work have proposed to "slice" a metric (or divergence) defined on the space of probability measures, to alleviate its computational and statistical issues on high-dimensional settings (the most popular example is the sliced Wasserstein distance). SMI then corresponds to a sliced version of MI (Definition 1), and is shown to provide theoretical benefits over MI, which are then illustrated on practical settings. A sliced version of conditional MI and entropy are also introduced (Definitions 2 and 3, respectively).

Specifically, the authors prove that SMI shares similar properties with MI (Proposition 1), boils down to a simple form for gaussian random variables (Example 1), and admits a variational form that is analogous to the Donsker-Varadhan representation for MI (Proposition 3). Then, the authors establish the sample complexity of SMI under different regularity assumptions on the random variables (Theorem 1, Corollary 1, Corollary 2): in all these cases, SMI yields a dimension-free sample complexity, as opposed to MI which suffers from the curse of dimensionality. The authors also explain how SMI can be used to extract the most informative features, defined as linear transformations of the random variables (Proposition 4) or shallow neural networks (Corollary 3); this task cannot be done with MI because of the data processing inequality, which SMI does not verify.  Finally, several experiments are conducted, mainly on synthetic data, to illustrate the theoretical findings and encourage the use of SMI over MI on high-dimensional settings (Section 4).
  To the best of my knowledge, the slicing technique has been used to design new metrics for probability measures, whose practical advantages have mostly been illustrated on generative modeling applications. In that sense, the contribution of this work is original: the authors extend the scope of slicing by applying it to design a novel measure for statistical dependence, that can be used to extract informative features. 

I believe this work is very interesting for the NeurIPS community, since SMI can be applied to various statistical and machine learning problems, while providing significant computational and statistical benefits over the traditionally used mutual information on high-dimensional settings. In particular, results on feature extraction (sections 3.2 and 4.3) seem very useful for the deep learning community, who aims at designing theoretically-grounded methods.

This paper is very clear and well written; some minor inaccuracies are listed at the end of my review. While the claims seem technically sound overall, I have some concerns which prevent me from a giving a higher score for now:

1) Sliced MI and its variants rely on a slicing technique that is different from the traditional one, since the random variables are not projected along the same direction on the sphere (for example, in Definition 1, there are two independent random variables $\Theta$ and $\Phi$, while there is usually just one). The authors do not justify this choice although it is an important difference. Maybe this choice is motivated by the fact that the random variables do not necessarily have the same dimension; but in that case, the "padding" strategy proposed in "Sliced Gromov-Wasserstein" (Vayer at al.) could have been used. 

2) As recalled by the authors in Section 2 (paragraph "Mutual information and entropy"), MI is equal to a specific evaluation of the KL divergence. This means that sliced MI can be seen as sliced KL divergence, and maybe general results from "Statistical and Topological Properties of Sliced Probability Divergences" (Nadjahi et al.) could have been applied to derive the sample complexity. Therefore, I encourage the authors to address the difference between that paper and their contribution (e.g., the proof of Theorem 1 in Section A.4).

3) The authors emphasize that SMI offers a dimension-free sample complexity, thus beats the curse of dimensionality. I am not fully convinced that the derived bounds do not contain any implicit dependency on the dimension; for example, the literature on the sliced Wasserstein distance ("Max-Sliced Wasserstein Distance and its use for GANs", Deshpande et al.) argue that the Monte Carlo estimation (which is also used in this work) induce a significant approximation error on high-dimensional settings.

4) In the introduction, MI is said to have proven useful in machine learning and deep learning applications, but also to be a computational bottleneck on high-dimensional settings. Therefore, I would like the authors to clarify how existing algorithms that combine MI and deep learning scale with the data dimension.

Minor inaccuracies:

l.68: the definition of pushforward measure should be recalled.

l.119: "Conditional SMI" -> "Conditional sliced entropy"

Proposition 1: the notation in l.128 should be consistent with l.81, in order to clarify the analogy between the chain rules of SMI and MI.

l.141: what is "sliced differential entropy"?

Proposition 3: the set over which $g$ is defined seems wrong; $\mathbb{S}^{d_x -1} \times \mathbb{S}^{d_y -1} \times \mathbb{R} \times \mathbb{R}$ would make more sense.

l.218: $Y$ and $Z$ should be $X_1$ and $X_2$ respectively.

l.245: "adaption" -> "adaptation"

Figure 1: it would be relevant to compare the performances of SMI and MI on this experiment, and try a very large value of dimension $d$, in order to further illustrate the statistical and computational benefits of SMI.

The proof of Corollary 3 should be given in the supplementary document for completeness, even it follows directly from Section A.7 (at least the main steps should be explained).
 The authors did not address the limitations and societal impact of their work.

The authors define a sliced version of Mutual Information (MI), which allows them to have a computationnally efficient manner to estimate a surrogate of MI. This SMI is built in a manner reminiscent of the sliced approach to Wasserstein distance: it computes the average of MI between 1D projection of the random variables of interest.
Many theoretical properties of this SMI are proved in the article. Some of them follows classical properties of MI and recovering them is interesting for the relevance of SMI.
Some properties are different, especially regarding the Data Processing Inequality: it is not satisfied by SMI. The authors give  an interpretation for this fact and explains that it can be leveraged in computational inference by learninh representations that are moe useful after processing . SMI can then be used as objective to be maximised for that.
Some numerical experiments on feature extraction and independence testing support the theoretical results.   The article is an original article that is well written. It contains a very good combination of technical results (which are not easy) and discussions with insight about the usefulness of the proposed SMI.

Strong aspects:
- the definition of the SMI makes sense and appears to be a novel idea, in the  continuity of the projected statistical distance as it was done for the Wasserstein distance;
- numerous properties are derived, showing that SMI share many properies of MI
- the advantage for the estimation are clearly described and justified  
- the authors appear to be very knowledgeable of estimation of MI and they make numerous comments comparing their proposed SMI and existing methods in SOTA
- the numerical empirical results are really interesting, and reinforce what appear to be first a theoretical work, but not only.

Questions:
- why choose to have 2 independent projections to define the SMI, while in slices Wasserstein, the same orientation is used for projection of the two measures? Was having the same projection considered? If so, does it lead to inferior properties ?
- for the empirical results, I think that 4.1 and 4.2 would benefit from having some dispersion values  (std for instance) reported over several realizations, in order to see if the gain is statistically stable or not. 
- the computational cost could be stated more directly; also, what is the cost of max SMI as an objective of feature extraction ? 

Weak Points:
- the mathematical derivations are difficult to follow after the initial definitions (but it might be a shortcoming on the reader side, me, as I think that, using the Supplementaries, that every elements is well covered)
-  the discussion is done in term of convergence rate, but not in term of computational cost.

 The work is mostky theoretical and I don't see relevant poinst to comment.

The authors proposed a sliced version of mutual information, following the idea of the sliced Wasserstein distance, to address the difficulties of MI estimation in high dimensions. Some appealing properties of SMI are investigated. Some numerical experiments are provided to support the claim.  Originality. 
The research is somewhat novel. While there has not been any work on sliced mutual information, sliced Wasserstein distance has been heavily investigated in recent years. 
But the DV (Donsker-Varadhan representation) estimator used for the actual computation of sliced mutual information statistics is also very standard. Some theories presented are trivial extensions of mutual information.

Quality.
This method features simple implementations and interesting theoretical properties. While it attempts to improve the scalability of MI-based applications in high dimensions, the current draft failed to convincingly demonstrate its advantage over its existing counterparts. Moreover, the experiments are not adequate. It does not compare to existing methods (NWJ [1], InfoNCE [2], TUBA [3], CLUB [4], KNN [5], etc) and also needs to report statistics like computation time, stability, and results such as varying the dimensions. The applications in real data are not enough. The authors only used MNIST and the setups are proofs-of-concept, which does not have any practical utility. Instead, the authors should follow the feature extraction experiment setups from either the MINE paper, DIM paper [6], or the SimCLR paper [7].

[1] XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847–5861, 2010.

[2] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.

[3] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In ICML. PMLR, 2019.

[4] Cheng, Pengyu, et al. "Club: A contrastive log-ratio upper bound of mutual information." International Conference on Machine Learning. PMLR, 2020.

[5] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 69(6):066138, 2004.

[6] Hjelm, R. Devon, et al. "Learning deep representations by mutual information estimation and maximization." arXiv preprint arXiv:1808.06670 (2018).

[7] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

In Line 146, the author proposed SMI admits a variational form in the spirit of the Donsker-Varadhan representation. However, in section 4.2, the authors did not compare SMI with DV, but with Kozachenko-Leonenko. For Kozachenko-Leonenko estimator, it is better to cite [8]. Also, experiment details are missing.

[8] Kraskov, Alexander, Harald Stögbauer, and Peter Grassberger. "Estimating mutual information." Physical review E 69.6 (2004): 066138.

Clarity. 
The organization of this paper is okay. Most of the basic concepts are well defined. In terms of presentation, the writing of this paper needs a revision. There are many language issues, making it difficult to follow. For examples,
1.	The definition of SMI is repeated twice in the paper. 

2.	As for the content, the authors pretty much give away the most important part of entire paper in Section 1.1. And it is a bit concerning that some of the statements are misleading. In particular, the violation of DPI is not appealing property, and it definitely cannot explain why extracted features often are more useful for inference than the raw data. 

3.	The figures are not formatted according to publication standards, for example, the font size of the texts is too small and barely readable (Figures 1, 3, 4). Captions are not self-contained (Figures 3, 4), thus they are confusing (eg. what do Ax and Ay mean (is it in Prop. 4)? Also, what is epsilon).

4.	Some statements and expressions are head-scratchingly confusing. For example:  line 9: SMI can be created from computation. line 22: it was adopted as a figure of merit. line 41: SMI nullifies.

Minor :
line 24: considered random variable (random variable of interests)

line 26: received (has received)

line 27: setups (settings)

line 82: to be Markovian, the arrow from z cannot point back to y

line 89: be generated from computation

line 171: should differentiate equation 7 and citation 7

line 148: sup index is incorrect

line 217: ($x_1$ $ax_2$) miss a comma here

line 237: what does case 1 mean
 This paper will have a limited impact on the community as it fails to show strong evidence that the proposed method outperforms existing solutions in practical settings. In comparison, the sliced Wasserstein distance papers showed superior performance in a comprehensive set of benchmarks including challenging real-world datasets.

Meta-review: The paper has received overwhelmingly positive reviews, where reviewerss largely agreed on the originality/novelty and the significance of the proposed quantity SMI. I also join the reviewers in that seeing the curse of dimensionality can be circumvented by random projections in fields other than probability metrics is interesting and exciting. Provided the authors will resolve the confusion about the DPI and rephrase their corresponding sentences, I am recommending an acceptance.

Decision: accept