Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Sliced mutual information (SMI) serves as a surrogate measure of dependence to classic MI, but a quantitative characterization of how its estimation rates depend on the dimension remains obscure.
This paper naturally extends the original SMI definition to k-SMI and provides a comprehensive analysis on how k-SMI's estimation rates depend on dimension.
The authors utilize the neural estimation framework to estimate k-SMI, with optimal convergence rates.
They discuss numerical considerations and perform a thorough simulation study and real data analysis, showing the favorable performance of their method.
strengths_and_weaknesses: * The authors provide a comprehensive analysis of the quantitative account of the scalability question of k-SMI.
They provide ample theoretical analysis on it.

* The authors apply a neural estimation framework to estimate k-SMI.
What's more, they provide this estimation with theoretical guarantees, which claims optimal convergence rates.

* The simulation study and real data analysis are very sufficient.
The authors illustrate the effectiveness from many aspects.
questions: * For the proposed measure k-SMI, there seems to be a little lack of innovation.
It is a natural extension of SMI.
And I don't find the advantages of k-SMI($k>1$) over SMI.
Furthermore, the computational cost of k-SMI is larger than SMI.

* There is not any discussion about the complexity of the proposed method.
Though the proposed method can be estimated by a neural network with optimal convergence rates, the computational cost is also important in practice.
The proposed method need to solve an optimization problem by neural network, which may be a disadvantage compared with other dependence measurements.
Some discussion and empirical results are needed.

* There seem to be many existing methods that measure the dependence between random variables without using a neural network.
The authors may compare their algorithm with these methods and clarify the advantages and disadvantages. 
limitations:
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes k-SMI (k-sliced mutual information) as a measure of statistical dependence defined by averaging MI (mutual information) terms between k-dimensional projections of high dimensional random variables, which is an extension of SMI (sliced mutual information). Although SMI avoids the curse of dimensionality in mutual information estimation by averaging
over the one-dimensional slices, the bounds rely on high-level assumptions that are hard to verify in practice and do not explicitly capture dependence on the ambient dimension, which is crucial for understanding scalability. The authors extend this theory to k-dimensional
projections and provide justifications with explicit dependence on k and the dimensions of data via numerical experiments. Additionally, they provide an application to sliced infoGAN, showing that k-SMI can successfully replace classic MI even in applications with more intricate
underlying structure.
strengths_and_weaknesses: Strengths:
* The originality is good. While the approach is a straightforward extension of SMI, the authors provide significantly improved theoretical results that also apply to SMI.
* The theoretical results are good as the authors provide a comprehensive quantitative study of dependence on dimension of k-SMI, encompassing the MC error, performance bounds for neural estimators, and asymptotics of the population k-SMI as dimension increases.
* The paper is generally clear and well-written.

Weaknesses
* I'm not entirely convinced of the significance of this work. See questions below.
* The experiments are decent but could use some improvement. See questions below.

questions: Significance
I recognize that estimating MI in high dimensions is a hard problem. This approach of k-SMI is interesting in that regard. However, it seems to me that one could just as easily perform some kind of dimensionality reduction first (e.g. using diffusion maps, or even PCA) and then estimating the MI on the reduced dimensions. This would likely be computationally cheaper as the MC step in k-SMI is unlikely to be computationally fast. Also, this would allow for nonlinear projections if the dimensionality reduction method is nonlinear. In contrast, k-SMI only performs linear projections. What advantage, if any, does k-SMI have over this approach?

Experiments
* In line 312 — the result of the independence test, It seems the sinusoidal model (a) and the common signal model (b) perform differently. In (a), it only works when k = 1 when d is large; In particular, when d = 20, the AUC is pretty small for 2-SMI. On the other hand, both 1-SMI and 2-SMI work pretty well in (b) even when d is large enough. Although the difference of the two models is clear, how they could make such a difference in result is not discussed.
* Again in the independence test, it is shown that there is potential gain of higher k, but I am worrying about whether the simulated data is too artificial, since Z and V are both simulated from multivariate Gaussian with identity covariance matrices (line 282); clearly model (a) does not provide similar results. So I think experimenting with the theory on more models would be preferred.
* In line 318 — the experiment of sliced InfoGAN, it is only shown that the k-SMI successfully disentangles the latent factors with k = 1. Are there any other results from other k values? In addition, it would be nice if some quantitative measure of performance could be given for this experiment.
limitations: Limitations were generally not discussed. Are there circumstances where the authors expect k-SMI to fail in some way?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes $k$-sliced Mutual Information (k-SMI) which is an extension of sliced Mutual Information (SMI). The difference between SMI and k-SMI is that k-SMI projects original measures to k-dimensional subspaces instead of unidimensional subspaces. Similar to SMI, the authors derive structural properties for k-SMI including Identification of independence, Bounds, Relative entropy and variational form, Entropy decomposition, Chain rule, and Tensorization. The authors derive bounds on the error of Monte Carlo (MC)-based estimates of k-SMI using the continuity of differential entropy in the 2-Wasserstein metric. Moreover, the author explores asymptotics of the population k-SMI, Gaussian approximation results, and neural estimation of SMI. On the application side, k-SMI is applied to sliced InfoGAN for disentangled representation learning.
strengths_and_weaknesses: # Strengths

## Originality 
* This is the first work that proposes a variant of mutual information from $k$-dimensional subspace projections. The corresponding theoretical analysis is also original. 

## Quality and Significance
* The investigation of the paper is important for scalable and no curse of dimensionality mutual information estimation

## Clarity
* The paper is well-written and easy to follow.

# Weaknesses
* Sampling uniformly from Stiefel manifold requires using QR decomposition that makes Monte Carlo estimation of k-SMI slower than SMI. Therefore, some discussion and empirical investigation are needed.
* It is hard to see the benefits of k-SMI over 1-SMI from the empirical experiments. Also, the comparison of Info-GAN and sliced Info-GAN does not show the superior of k-SMI over k-SMI.
questions: * Why we should consider $k$-SMI instead of 1-SMI in practice?
* What is the computational complexity of the projection step in k-SMI?
limitations: This is fundamental work, hence, there is no foreseen negative social impact.
review_title: null

Meta-review: metareview: While the reviewers have raised concerns, mainly about the significance of k-SMI, they also acknowledged that the theoretical contributions of the paper are solid. On the other hand, I do agree with the authors on the fact that the main contribution of the paper is the theoretical analysis that can be trivially applied to SMI as well, rather than the introduction of k-SMI itself. Hence, I will go ahead and recommend a borderline acceptance for the paper. However, I strongly suggest the authors to take into account the reviewers' concerns, and reword their contributions to emphasize their theoretical contributions, which also clarify the behavior of SMI.  
confidence: Less certain
Decision: Accept