Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces SDRL, a representation learning algorithm for supervised learning.
SDRL aims to enforce three constraints to the learned representation g(x) :
1) That all the information from X about Y is preserved: Y independent to x given g(x)
2) That it's distributionally rotation invariant: Law(O g(x)) = Law (g(x))
3) That all it's coordinates are independent: Law(g(x)_i) = Law(g(x)_j) for all i, j

Since 2 and 3 are equivalent to asking that g(x) is Gaussian, they look for a pushforward map g such that g(x) is Gaussian.
In particular, they pick the optimal transport one given by Brenier's theorem.

I think the paper needs more work in the following on 4 very important aspects:

A) The paper is not very well motivated in my opinion. Why do we want to learn a representation with statistically independent and rotation invariant coordinates? I know many works already try to achieve this goal, but this paper doesn't do a good job at motivating it. It merely cites a bunch of other papers that do so, but reading the paper I am left with the impression of "we want to come up with a representation with properties 1, 2, 3. Why? Ask other people". It doesn't feel like I'm given a sufficient explanation of why I should be reading this paper and the reality is that if I wasn't asked to review it I would have left it in the pile of papers in my desk half way through the introduction. This is very concretely seen in the sentence "We also require that the representation is rotation invariant in distribution. Such an invariance property is an important characteristic in many visual and morphological tasks \cite{}". This is a third of the problem you're trying to solve, it is a bit crazy that all the motivation it has is a citation to two papers that are not mentioned anywhere else in the paper.

B) The method is very complicated. Why use the distance covariance rather than the empirical risk to assure that the information from X about Y is preserved in G(X)? I know the authors have several choices for V, but the choice they pick is quite particular and a complicated one at it. Without much motivation or experiments ablating this choice it's hard to justify it. It's hard to convince the reader to bother like this.

C) The learning bounds are pretty irrelevant. the n^-{2/(2+d)} bound, while minimax optimal, is useless in high dimensions (high meaning d > 10, which is usually what we are interested in representation learning). While minimax optimal ***in the distributionally worst case***, it is irrelevant in practice (and it is not an advance in any theoretical field). It's simply taking extra space in the paper, it really brings nothing to the table in my opinion.

D) The experiments are extremely underwhelming. For such a complicated method does not bring any significant benefit in the real classification datasets, and a very minor one in the regression ones. In regression, furthermore, an important baseline is missing: the last layer of an NN trained with a regression and / or ordinal regression loss. It seems unfair to compete with the NN inductive bias in the author's algorithm vs linear algorithms.

Overall, the authors take us through an interesting but overly complicated mathematical journey with a poor motivation, irrelevant learning bounds, and no empirical evidence that the journey was useful whatsoever.
review_title: Review for "Sufficient and Disentangled Representation Learning"

final_score: rating: 7: Good paper, accept
review_content: This work proposes a method, SDRL, for learning sufficient and disentangled representations, with the additional property that the representations should also be rotation-invariant. Together with the disengtangled property, the ration-invariant property specifies the distribution of representations to be isotropic Gaussians. On the other hand, the repressentations are required to be sufficient for predicting the target labels. These two goals motivate the Lagrangian formulation of the objective function, based on which the authors apply two different estimators for these two goals. Experiments are extensive, with a good mixture of synthetic and real-world data. I particularly like the visualization of the learned representations on the synthetic data, which well corroborates the theoretical claims in the first part of the paper. 

Overall, I think the paper has a sound logic and clear presentation. The only downside is that the novelty is a bit limited: both the sufficiency criterion (see Tishby et al. 2015) and the disentangled criterion (mutual independency) have well been studied in the literature, despite using other divergence measures, e.g., mutual information and KL divergence. That being said, I haven't seen similar work using exactly the same divergences as the ones in this paper, e.g., distance correlation and f-divergence in this context. 

A few more minor comments and questions:
-   I could understand representations that are rotation-invariant are desirable in vision tasks, but I am wondering is this requirement too strong for other applications, e.g., in language and speech? Other than the technical convenience brought by this property, is there any other reason that could motivate this criterion? 

-   In Eq. (2), shouldn't the domain of g be R^p instead of R^d? Otherwise the intersection of M and F would be empty. 

-   If my understanding about Lemma 2.1 is correct, it is the pushforward of \mu under T to be an isotropic Gaussian distribution, right? The original Z = g(X) is only sufficient, but not disintangled nor rotation-invariant. If this is the case, then I am not sure I totally agree with the second comment after Theorem 4.2, where the authors remark that the discriminator network is the pushforward function T. My understanding is that the representer network R is used to approximate the composite of T and g, so that R_\sharp \mu_X = R^* in Eq. (3), while the discriminator network D is only served as the witness function in the definition of f-divergence in Eq. (6). Please clarify this point, thanks!

-   The line above Eq. (9), typo: no \lambda is needed before the V term.

-   IMO the second assumption of Theorem 4.3 is quite strong: it requires the density to be lower bounded by a positive constant c_1, which does not even hold for Gaussians. On the other hand, the compactness of the domain helps to reconcile this assumption, and I understand that this is a standard assumption in the analysis of nonparametric density estimation/regression.

review_title: A solid work that aims to learn sufficient and disentangled representations, with strong motivations, sound theoretical justification and extensive empirical validations. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new representation learning framework for supervised learning 
In order to achieve good prediction accuracies while maintaining some desired properties. 
The sufficiency and disentangled properties of the representation are formalized in
this paper to achieve this goal. Most results in the literature have focused on
the unsupervised setting, therefore this framework is novel to me.
Below I have a few comments to revise the paper. 

To achieve sufficiency, this paper proposes to use the distance covariance defined in Section 4. 
This seems to be related with statistical correlation,
can this be written out in the paper? As when I read the proof of 
theorem 4.2, I do not know what is the \rho(R,R*). It seems not 
to be defined in the paper. 
Also is there any relation between V[z,y] with the sufficient statistics (Fisher 1922), as mentioned in the introduction? 
From (5) I then understand that this covariance V replies on the metric on X and Y. 
For multi-class classification problems, it is not always clear how to choose
a metric on Y. I recommend the authors to clarify this choice
at least in the numerical experiments. 

In the statement of Theorem 4.3, should \lambda = O(1) be strictly positive? The objective (7) is formalized as a bi-level optimization problem, and the computation problem is addressed using recently proposed methods based on gradient flows. There is a typo 2 lines before Section 5, should the rate be (log n) n^{-2 / (2+d)} ? 

Section 6 presents numerical results. I recommend to clarify what is Y in the classification problem 
and how the metric is chosen. Figure 1 should be significantly improved. Figure 1 illustrates well that the learnt features are sufficient, but it is unclear whether the features are disentangled. To make the results more convincing, I suggest to plot the learned representation directly (maybe 
in Appendix) to check whether it looks like an isotropic 2d Gaussian. Any statistical test of Gaussianity would be more convincing. 

What is the distance correlation computed in Figure 2? 
V[z,y] is called the distance covariance, so I guess it is something 
else. Is the Table 3 showing the training or test accuracy? 

Typo: 
- in the definition of V[z,y] on page 4, the constant c_m should be c_q? 
- 3 lines before (2), N_d (0, I_d) -> N (0, I_d)?
- The (a) and (e) in Figure 1 are not the learned features, but the original data.
- page 19: matric entropy -> metric entropy


review_title: This framework is novel to me, but the paper needs to be revised

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors present a new representation learning algorithm that trades off between a sufficiency condition (that is, the label should be independent of the input conditioned on the representation) and what they call a "disentangling" condition - that the representation vectors should be independent of one another and rotationally invariant. While the first condition has been used to define disentangled representations, the second is not standard. From the condition of rotational invariance, they require that the distribution over representations is isomorphic to a uniform Gaussian. They arrive at a loss with two terms, the first is a distance correlation between labels and representation, and the second is a divergence between the representation and a uniform Gaussian. In this sense, the regularization term looks quite similar to a VAE while the loss term looks quite similar to standard classification losses. The regularization is represented as a maximum over another loss, leading to a GAN-like coupled optimization problem.

The paper is written with quite a lot of complicated math. While I did not find anything wrong with the math, it did seem like at times it was meant more to obfuscate and impress and was often not really necessary. The experiments were almost entirely either comparisons against classic statistical methods (section 6.1 and 6.2, Table 1) or experiments on MNIST, FashionMNIST and CIFAR-10 (6.2, Table 3). Given the explosion of different deep learning methods for classification, representation learning, and disentangling in the last decade, it seems like the comparisons against classic statistical methods is missing the point - it seems highly likely that other deep learning methods could work just as well. In the experiments on MNIST, FashionMNIST and CIFAR-10, the authors set quite a high bar for themselves. These are 3 of the most over-studied datasets in all of machine learning - literally thousands, maybe tens of thousands of papers have been written on various deep learning algorithms applied to these datasets in the last several years. The experiments presented here do not seem to be a proper apples-to-apples comparison, however. A standard MLP is used for the SDRL (the method developed in this paper) while a DenseNet architecture trained with a classification loss is used for the "convolutional network" method. I was not able to find details on the dCorAE architecture. To properly reduce the number of possible confounding factors, the same architecture should be used with both the SDRL and the baseline methods, with different objective functions. Also, the classification numbers presented on CIFAR-10 do not seem to be state-of-the-art. According to Papers With Code, the DenseNet is able to achieve 96% accuracy on CIFAR-10, and even newer methods are able to reach 99% accuracy without additional data.

My biggest objection to the paper, however, is that it seems completely unrelated to disentangling. The only experiments presented are on visualization and classification - no results on standard disentangling tasks are presented. The rotational invariance condition seems completely against the grain of disentangling research. Despite disagreements over the exact definition of disentangling, there is at least broad agreement that a disentangled representation is one in which certain directions in latent space are privileged over others, and align with "true" latent factors in the world. While I disagree with the probabilistic definition of disentangling given in Locatello et al (2018), and prefer the geometric definition of Higgins et al (2018), I do agree with the point made in Locatello that, if different unique directions in latent space are not identifiable, then disentangling is not possible. Yet in this paper, the non-identifiability of different directions is given as a *necessary condition* for disentangling. This is the completely contrary to how the term is usually used. Even in the beta-VAE, which does have a rotationally-invariant loss function, the invariance is broken by requiring that the approximate posterior from the encoder has a diagonal covariance. Given this, I would recommend that the authors remove any reference to disentangling, and rewrite this purely as an alternative approach to supervised learning.
review_title: A GAN-like approach to representation learning that has little to do with disentangling

Meta-review: This paper aims to present a new representation learning framework for supervised learning based on finding a representation such that the input is conditionally independent given the representation, the components of the representations are independent and the representation is rotation-invariant. While there were both positive and negative assessments of this paper by the reviewers, there are 3 major concerns that lead me to recommend rejecting this paper:
1. Most importantly, experiments do not seem to be conclusive as they do not properly ablate the specific aspects of this method. More specifically, the authors compare their deep learning based approach with non-deep learning approaches but do not compare against deep learning baselines. This makes it impossible to assess the merit of the proposed approach (which also appears to be complicated) over much simpler baselines.
2. The required properties of the representations do not seem to be properly motivated. 
3. The paper refers to their produced representations as disentangled representations. As pointed out by AnonReviewer4, this appears not to be consistent with prior uses of that word in the community.

Decision: Reject