Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors leverage sliced mutual information and propose a tractable infomin learning algorithm. Compared to the existing methods, the proposed algorithm requires neither adversarial training nor a neural-based mutual information estimator. Instead, the authors replace the mutual information term with a sliced version in the infomin objective, and derive an analytic approximation to sliced mutual information. Moreover, solving hi and gj in the sliced mutual information together yields an upper bound of the analytic solution, and thus more efficient in practice. The authors compare the proposed method with 4 different baselines, including the non-parametric method and adversarial training method. They evaluate them on three tasks: in the independence test, they show that sliced mutual information is less effective than the adversarial training methods, but they require longer training time; in the fairness experiment, under a similar level of utility, the proposed method has significantly better fairness than the baselines; in the disentangled representation learning experiment, they show a similar observation to the fairness experiment that the proposed method achieves much better disentanglement without sacrificing the utility a lot.
strengths_and_weaknesses: **Strengths**:
- The paper is overall well written. The theoretical analysis is sound and well motivated.
- The idea of using proxy metrics instead of directly working on the intractable MI is interesting
- The experimental setup is sound with different baselines

**Weaknesses**:
- I think the paper misses one important and related paper [1], and thus lacks further discussion of one important implementation of infomin optimization that leverages neural-based methods to give a trackable estimation of the upper bound of mutual information.   This should serve as a strong baseline and compare with the proposed method, non-parametric methods, and adversarial training methods as well. I am expecting to see more quantitative analysis on the comparisons. 
- I am also looking forward to seeing more larger-scale experiments, as the paper is claimed to be “scalable”. For example, the CLUB MI estimator [1] is known to be applied to large language models like BERT to improve the adversarial robustness [2]. 
- The paper misses some important implementation details: (1) a brief introduction and citation to the datasets Dsprite and CMU-pie are missing. (2) how the networks h,g are trained needs more details, e,g, the training objective. (3) I am also looking forward to seeing a detailed pseudo algorithm in terms of the details mini-batch learning to improve the reproducibility.
- I am also expecting to see how is the sliced MI related to the real MI. It might be better to give some toy examples where MI can be calculated and see how close the methods are.

[1] Cheng, Pengyu, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan and Lawrence Carin. “CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information.” ICML (2020).

[2] Wang, Boxin, Shuohang Wang, Yu Cheng, Zhe Gan, R. Jia, Bo Li and Jingjing Liu. “InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective.” ICLR (2021)

questions: Please answer the questions in the weaknesses section. 

Both your method and MI upper bound [1] utilize neural networks for MI estimation, could the authors elaborate more on the advantages of your method?

I am willing to increase my scores if my questions and concerns are resolved.

limitations: See the weakness section.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose an adversary-free method for informin-based representation-learning based on the recently-proposed sliced mutual information estimator, noting the optimisation difficulties and costs associated with the adversarial approaches that have long been the go-to methods for tasks such as disentanglement and fair-representation learning. This is achieved by substituting the empirical approximation of sliced mutual  information (SI) with a closed-form solution that can be well-approximated with K-degree polynomials, leading to a problem that can be efficiently solved using canonical correlation analysis. The authors establish error bounds on said analytic approximation and show that the entailed functions can be solved in parallel for each slice without violating the objective.
Experiments are performed on a number of datasets spanning the interrelated tasks of independence  testing, disentanglement, and fair-representation learning, with Renyi correlation between the learned representation and Y (aiming to maximise) and T (aiming to minimise) as the evaluation metric. The proposed method, Slice, is shown to  outperform other infomin-learning approaches by a significant
margin while remaining computationally efficient.

strengths_and_weaknesses: - The paper builds on recent research on mutual-information estimation to develop a method that is both adversarial free
and computationally-efficient, and how it might be practically applied to problems of disentanglement and fair-representation
learning -- to my knowledge, the analytic approximation, derived from canonical correlation analysis, is novel within the
given context, and seems to be both theoretically and practically sound.
- The motivation for the seeking out non-adversarial alternatives to infomin learning is clearly established and the description of the method is generally well-structured and easy-to-follow, though I feel there is some room for improvement in the choice of notation (such as in the decision to use D and d for the dimensionality of Z and T, respectively).
- The analysis of the results is satisfactory; the conclusion is serviceable but feels rushed and I'm not convinced there's sufficient evidence given in the paper to 'verify' the failure modes of adversarial methods.
- The method is benchmarked against a good range of relevant baselines and using an appropriate suite of datasets
(though it would be nice to see variation in the domain of the fairness datasets) and results are aggregated over multiple
replicates in all cases. The procedures associated with each set of experiments are outlined in sufficient detail.
- While the existing baselines are solid, for the fair-representation learning experiments it would be nice to see a simpler 
adversarial cross-entropy-based baseline a la DANN/LAFTR, given its prominence in the literature, despite its inability
to capture higher order dependencies like its Renyi counterpart.
- Given the statements made about the convergence of the adversarial methods, I feel it would also be of interest to include ablations in which the number of adversarial steps is increased enough to determine whether this is a relevant factor, or whether the problems with these methods truly are of a more fundamental nature.
- The method, Slice, seems to perform impressively in terms of both expectation and variance. While Slice does incur decent computational overhead compared the simpler baselines, such as 'Pearson', it is nonetheless shown to be significantly more efficient than its adversarial competitors which suffer additional problems due to their parametric/minimax nature. The results included in the main paper, together with those included in the Appendices, support these conclusions convincingly enough, with the experimental setups used to produce them largely consistent with that of past work.
- While the authors do discuss the relationship between Renyi correlation and DP, and good rationale for adopting solely the former, it would nonetheless be good to additionally include the the latter for the evaluation of the method in the context of representation learning, given the conventions of the pre-existing literature on the problem.
- The images in Figures 2 and 3 comparing how Slice and an adversarial method (which one?) cope with label-swapping are rather small and there's no indication as to what the label corresponds to within the figures themselves or their captions. Those things aside, I think it would be helpful if the differences between the methods were highlighted as it's not immediately obvious where exactly one should be looking.

questions: -
limitations: I am satisfied with the extent to which the the limitations and ethics of the work have been addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a method for infomin learning based on information theory tools from previous work, that estimates the Mutual Information (MI) in the “sliced space”, which intuitively is a facet of MI.
strengths_and_weaknesses: Strengths
- The method seems explained well in terms of theoretical justifications
- The experiments seem to be provided with sufficient details

Weaknesses
- In the disentanglement experiment other metrics could be used to make sure that models are learning disentangled representations, e.g. MIG, factorVAE score, etc

questions: - On the disentanglement experiment more recent work could be considered as baseline, for example LieGroupVAE (https://arxiv.org/pdf/2106.03375.pdf) ?
- In the disentanglement experiment qualitative Figure2-3. Should latent traversals also be explored, to understand if the representation is disentangled?
- In the fairness experiment ρ∗(Z, T ) seems to get non-comparable results for the baselines. Should other better performing methods be considered as baselines?

limitations: One of the claims is the scalability, however a dedicated experiment demonstrating how the model scales compared to previous baselines is missing.
review_title: null

Meta-review: metareview: **Summary**: This paper develops an infomin-based representation method that based on the recently-proposed sliced mutual informaiton estimator. Unlike other methods, the proposed approach does not rely on an adversarial objective and provides tractable proxy-metric that eliminates the need for neural estimators of the mutual information. Experiments on independence tests, disentangled representation learning and algorithmic fairness aim to illustrate both improved utility and higher scalability. 

**Strengths**: Reviewers we overall positively predisposed towards this paper. They noted that this is a well-written paper, with sound and well-motivated theoretical analysis [d477, hqSG]. The proposed method, which derives from canonical correlation analysis is novel and computationally efficient [d477]. Experiments are sound and satisfactory, with benchmarks that include a good range of datasets and baselines. Reviewer *dNaL* notes good results in terms of both expectation and variance, in addition to improved computational efficiency relative to adversarial methods.

**Weaknesses**: Reviewers also noted limitations. Reviewer *dn477* noted a missing reference to CLUB (Chen et al., ICML, 2020) which would be a strong neural baseline. Several reviewers found that scalability claims are not strongly supported and that larger-scale experiments might strengthen the paper in this context [d477, hqSG]. More generally, reviewers were concerned that the submission lacks certain important implementation details [d477, dNAL]. In terms of the experiments, reviewers had a number of suggestions, including a comparison between slice MI to the analytically calculated MI for some toy example [d477], a comparison simpler a simpler adversarial cross-entropy based methods for the fairness experiments (e.g. DANN/LAFTR) [dNaL], a comparison to LieGroupVAE [hqSG], and reporting of disentanglement metrics such as MIG and the FactorVAE score [hqSG]. 
    
**Reviewer Author Discussion**: While the authors were not able to carry out larger-scale experiments, they provided an ablation study to further support claims of scalability.  They also added discussion of CLUB, and clarified that DANN/LAFTR are similar to the Neural TC baselines, clarified that disentanglement scores cannot be computed for the vector-valued quantities that are under consideration for this paper. Reviewer *D477* raised their score 5->6, reviewer *hqSG* raised their score 6->7.

**Reviewer AC Discussion**: Reviewers unfortunately did not respond to the AC during the discussion phase. The AC takes this as a signal that reviewers do not object to acceptance, but also do not champion it.  

**Overall Recommendation**: This submission is just about above the bar for an accept, though lack of a clear champion among reviewers somewhat limits confidence. 
confidence: Less certain
Decision: Accept