Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a new self-supervised learning technique that learns to map image inputs to "disentangled" vectors iteratively by alternating two steps. First, all of the data is assumed to be in the same subset and they learn a representation by minimizing a SimCLR like contrastive loss (here the positive pairs are augmented versions of the same image, while negative samples are other images in the batch). In the second step, they find the partition of the dataset into 2 subsets that maximize the contrastive loss. This new partition is added to the set of all partitions considered, and we repeat the two steps (again learning the representation that minimizes contrastive loss over all partitions and then finding the next partition and so on). They use a regularized form of the contrastive loss, which adds a term taken from earlier work ("Invariation Risk Minimization"). This terms measures the magnitude of the gradient of the contrastive loss wrt to the temperature parameter (at temperature=1). They evaluate their technique against alternatives on a wide range of disentanglement metrics and downstream classification accuracy, and show their technique performs well.
main_review: Overall, I think the paper is quite interesting and should be of interest to the community. I especially like the idea of finding the worst partition in terms of the representation learned so far and re-learning the representation based on this partition. I also found the evaluations in the paper quite comprehensive.

I think the main issue with the paper is that it makes rather strong claims but fails to support some of these. For example, the authors argue that their technique can disentangle the full semantic space (i.e., learn a fully disentangled representation) in a fully unsupervised manner. (in contrast to other SSL techniques, which can only learn to disentangle the semantic information wrt hand-crafted augmentations.) However, to me it is not clear how this squares with the Locatello et al. result that shows that fully unsupervised learning of disentangled representations is not possible. They mention in passing that Higgins et al.'s group theoretic definition of disentanglement solves the problem with Locatello et al.'s result but this is not clear to me. Locatello et al. in essence shows that there are multiple equivalent representations and it is impossible to pick one over another as more disentangled without some supervision. And Higgins et al. also accept this fact as they point out in their paper that their disentanglement definition assumes a "natural" decomposition is already given (i.e., what it means to be disentangled). I should say that I found it difficult to fully grasp what the authors' claim is, so please let me know if I misunderstood anything.

This brings me to my other point. I think the paper right now is unfortunately hard to read. The paper gets rather technical at times, and it is hard to follow the argument (for example, discussion after Lemma 1 in Section 4 is hard to follow). While I appreciate the authors' presenting theoretical results, I think the paper would benefit greatly if the technique could be better motivated on an intuitive level, and the technical arguments similarly could be presented on an intuitive level.

Please also see my further comments below:
- Figure 2 is hard to understand. What is being classified on the left panel? Also, it is not clear what I should see on the heatmaps on the right.
- line 105-106, the authors say "Higgins et al., ..., resolves the previous controversial points" wrt to Locatello et al. for example. As I mentioned above, this is not clear to me. It'd be worth expanding this a little bit further.
- What is the effect of regularizer term from "Invariant Risk Maximization"? The authors say it "regularizes phi to be invariant across subset in a partition". What does this mean exactly? Can you give an intuitive example of what this term does? What would happen if you didn't have this term?
- line 155, the authors say "f is a collapsed representation that maps all inputs to a fixed vector, which is impossible in today’s deep model training". What does this mean? It is impossible because of the regularization or tricks like EMA that people use. I don't see how the referenced work [88] is relevant to this point either.
- To me it seemed like finding the partition that maximizes loss is akin to finding hard negatives. Is this intuition correct? If so, would it be useful to mention sth along these lines in the paper?
- For the results in Table 2, how do you combine IP-IRM with SimCLR etc.?
- In the checklist, the authors mention limitations are discussed in section 6, but I couldn't see this.
- I understand that the authors may not have had enough time to get standard deviations on all of the results, but it'd be nice to add these. (I have seen the results in appendix).
- While reading this paper, I was reminded of the paper "What Should Not Be Contrastive in Contrastive Learning" ()https://arxiv.org/abs/2008.05659). This paper learns essentially a separate subspace for each augmentation (which encourages the representations not to throw away information like SimCLR etc. might do). It'd be interesting to see how well their method compares to this. (To be clear, this is totally optional. I understand the authors might not have time to work on this.)
- typos:
  - line 74, "attributed first part" -> "attributed to first part"
  - line 218, "regrading"
  - line 228, "What does IP-IRM feature look like?" -> "What do IP-IRM features look like?"
  - line 239, "regardless the other semantic" -> "regardless of the other semantic"


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper proposes a scalable approach to disentangled representation learning using the self supervised learning framework, rather than the typically used generative model framework. Their approach is principled, being grounded in the Higgins et al definition of disentangling from the symmetry perspective. It also appears to be scalable, which is a very exciting step in the subfield of unsupervised disentangled representation learning.
main_review: The paper is very well written and makes a very exciting step in unsupervised disentangled representation learning. I am happy to argue for its acceptance. Saying this, I do have a few concerns.

1) Is K a hyperparameter? How should one choose it? What were the choices for CMNIST and 3D shapes? Why does increasing it result in reduced performance? How do the authors propose to scale their algorithm to disentangling many different subspaces without the ability to increase K?

2) What are the augmentations used, and how are the negative examples chosen? If the authors just augment the existing SSL algorithms with their native augmentations/negative examples with the proposed disentangling framework, then it should be stated in text, as otherwise the reader is left wondering. Is there a way to choose augmentations that would not interfere with disentangling, as the authors suggested e.g. that colour information was not disentangled because it was subsumed by the augmentations.

Minor: Line 147 "In particular, SSL and fully-supervised learning are two special cases of Gi-disentanglement" - I think it would be better to move it out of the lemma, as this is not really a part of it, and instead add this to the following more in depth discussion of this point later in the section.




limitations_and_societal_impact: No
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors focus on the problem of learning unsupervised representations that are also disentangled. To this end, they build on top of the idea of invariant risk minimization and propose Iterative Partition-based Invariant Risk Minimization (IP-IRM) which iteratively assigns examples to partitions and constrains invariance in the loss among these partitions. They justify their approach mathematically according to a formal definition of disentanglement. The then perform multiple experiments qualitatively and quantitatively demonstrating the learned representations are disentangled. Furthermore, they show promising results on Out-of-distribution classification tasks. 
main_review: Strengths:

1. The paper is well written, flows well, and is easy to understand. 

2. The problem is very relevant. There has been a great deal of work on learning self-supervised representations in computer vision. However, much work has focused on improving image classification numbers and less on how well the representation is disentangled. Disentanglement is an essential foundation for abstract reasoning.

3. The paper is very well motivated mathematically and conceptually. They show, using a formal definition of disentanglement, why IP-IRM is a sensible approach.

4. Quantitative and qualitative experimentation is very thorough and convincing. 

Weaknesses:

1. For the OOD experiments on Imagenet, NICO, and related, the improvements in performance over the state of the art is smaller.

Overall:

The paper addresses an important but underexplored topic in self-supervised learning: disentangled representations. The authors firmly justify IP-IRM mathematically with Higgins's definition of disentanglement. Evaluation of the approach is thorough, and the quantitative and qualitative results convincing. I believe this would be a good contribution to the conference. 
limitations_and_societal_impact: Yes, they have addressed limitations and social impacts.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a self-supervised learning model which is a variant of the Invariant Risk Minimization with an iterative partition mechanism. The method can encourage the self-supervised learning of a disentangled representation. Empirical results validate the effectiveness of the proposed model on self-supervised learning and disentangled representation learning.
main_review: Originality:
The proposed method depends on the Invariant Risk Minimization model. The disentanglement validation depends on the symmetry-based (SB) disentangled representation definition. The overall learning setting is the same as the general self-supervised learning. The core technical originality I personally believe is the iterative partition mechanism, but I would also consider the combination of IRM, SSL, and SB-disentanglement is novel and original as well as they are not directly related.
Quality:
1. The experimental results clearly validate the disentanglement and self-supervised learning quality as they improve the existing SSL baselines by large margins. 
2. However, my main concern is about the justification part (Section 4) of the paper, where I am not fully convinced by the proofs of Lemma 1 (and therefore Theorem 1). In Line 155, the paper states there are two possible cases if Lemma 1 doesn't hold. But why are there only these two cases? It looks like the line 155-156 and the appendix line 61-62 try to describe the same thing: (1) main paper line 155-156: 'f is a collapsed representation that maps all inputs to a fixed vector, which is impossible in today's deep model training [88]'; (2) appendix line 61-62: 'The representation is not equivariant w.r.t. the action of \Pi G_i. As discussed in Lemma 1, this is impossible in today's deep model training [26].' However, nothing in the Lemma 1 discussion is related to the appendix statement about the equivariance with the product of subgroups, neither does the cited paper ([26] of the appendix). This makes me very confused about the correctness of the proof B.1 in the appendix. 
In main paper lines 156-159, the paper cites [35] to say there is always a linear projection. But why is it related to the group G_{t+1}? And why is it related to the proof of Lemma 1?
If the SB-disentanglement is actually realized as claimed in section 4, can the authors show the specific learned groups (or subgroups) empirically? As long as I can see in the disentanglement experiments, what the paper tries to emphasize is the separation of latent subspaces, which corresponds to traditional disentangled representation learning without considering the groups that can actually act on them.
3. The unsupervised disentanglement experiment only reports the means of the metrics and no standard deviations. Typically models should be reported with runs of multiple random seeds to get an evaluation on disentanglement performance.
Clarity:
There could be some improvements.
1. There should be some preliminaries about IRM. At least a general introduction about what problem IRM is trying to solve, why it works, and why the SSL task falls into its setting.
2. The justification part (Section 4) should be improved on its proof of Lemma 1 and Theorem 1 (see my concerns in the Quality part).
Significance:
The direction of improving the disentanglement property of self-supervised learning representations is promising and important. The results are significant and I believe it will benefit the representation learning community.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper proposes a novel approach to an important and popular problem, and in doing so provides a natural but novel application of *another* important and popular line of work (IRM). The work is more theoretically justified than most in the area, and has reasonably satisfying experimental results. Overall, it will make a fine contribution to the conference. To maximize that, please make sure to incorporate the outcome of the reviewer discussions – some of which seemed fairly illuminating – into the final version of the paper.
Decision: Accept (Spotlight)