Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Image representations learned using BYOL, a contrastive method, show better disentanglement than those learned with disentanglement-specific methods. Compares across models in toy datasets and one real-world dataset, CelebA.
main_review: The paper makes the previously unreported finding that contrastive learning produces "group disentangled" representations and outperform other disentanglement models (as measured by standard disentanglement measures). It offers a nice literature review, exposition is clear and provides many implementation details that should help reproducibility.

Though presented results are solid, it would be interesting to see the complete set of results (probably in Appendix), namely Fig. 4 and Tab. 2 (or Tab. 1) for other datasets. It might greatly benefit from some insight (perhaps preliminary) on why contrastive representations are disentangled when supervised/unsupervised representations would not. It is also unclear to me the effect of the contrastive learning pre-selection (sec 7, last paragraph) in the disentanglement performance, how are the thresholds in Tab. 7 selected, how many contrastive models are discarded in this process and would disentanglement performance change if the same threshold was used for all factors or if this selection was not performed at all---as would be in practice where this factors are unknown.

Questions:
* It is not clear how the MI is computed for Fig. 3, I believe somehwere in the Appendix it said the distribution is approximated with a histogram, but some details (or a pointer to the details that are in Appendix) would be appreciated.
* I noticed the best validation FactorVAE metric across normalizations (0.916 in Tab. 4) is the same as the one reported in Tab.1, can you confirm this is by chance and not because the same dataset split was used for both.

Minor:
* abs: explored->explore
* Sec 1, p. 4: and Swav (Caron et al., 2020) etc -> , SWav (Caron et al., 2020), etc | and Swav (Caron et al., 2020).
* Sec 2, p. 2: "which maximize the information and the generated image"  seems to be missing words.
* Sec 2, last p.: drop the "or not"
* Sec 3.2: without notion otherwise -> unless otherwised stated
* Fig 2: add labels to the bars, e.g., "Contrastive representation" and "Ground truth factors"
* Tab 1, caption: contratstive -> contrastive
summary_of_the_review: Overall a nice paper that makes some hefty claims and as such, I would like to see more results, have some questions answered and discuss with other reviewers before I can recommend acceptance. Showing results hold for other contrastive methods would also go a great deal in showing the validity of them and increase the impact of this work but it might be outside its scope.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper applies the contrastive BYOL method on a set of datasets, which are mostly well-established in the disentanglement community.  Here, they show empirical evidence that BYOL successfully learns disentangled representations. 
main_review: - Contributions: As your results are purely empirical I think statements like "We show that a contrastive method learns group-disentangled representations" might be troublesome since readers could expect theoretical guarantees.
- Contributions: You say that your contrastive method (which is confusing since you just the usual BYOL method without modifying it) achieves state-of-the-art performance; however, I do not see a comparison with reasonable models, e.g., [1], [2], [3].
- Abstract: The statement "Prior methods achieved initial successes on simplistic synthetic datasets but failed to scale to complex real-world datasets." is unclear - to which prior methods are you referring?
- Related work/Discussion: (1) Regarding the Zimmermann et al. 2021 paper, you say they didn't perform any quantitative analysis. However, this paper even proposes a new dataset of 3D objects and demonstrates the success of a contrastive method on it. (2) Furthermore, the comparison seems a bit skewed (please correct me if I'm missing something here), since your primary point of criticism of that work are the (strong) theoretical assumptions - yet, your paper does not contain any theoretical but just empirical results. (3) Finally, you say that you "build a series of quantitative benchmarks", yet, later in the paper, it seems like you only re-use existing datasets. If you mean that you tried evaluating disentanglement on real-world datasets, you should cite previous papers that did so, e.g. [2]. Can you please specify what you mean by "build" and make this more precise? 
- Major Results: How does your group disentanglement property differ from existing metrics in this field? I think a discussion should be added here. Your current description of it via Disentlement, Compactness, and Completeness sounds a lot like the metric suggested by [4] (i.e., the DCI metric) - what makes it special?
- Major Results: I don't really see how the lottery ticket hypothesis is connected to disentanglement. At the heart of your argument, you say that you only say you suspect this to be the reason but do not give any reasoning as to why you think so - you only describe the lottery ticket hypothesis before.
- Table 1, 2: Over how many random seeds did you average? What is the error of the score? That information is necessary to decide whether the results of BYOL are significantly better than those of the other methods.
- Table 3, 4: Is it correct that these results correspond to just a single random seed?
- Table 4, 5.5: You only discuss the result of BN; however, it remains unclear whether the BN results are really significantly worse than the other results. Furthermore, there is no discussion of the IN results, even though, that number is drastically lower than that of BN.
- Introduction/Related work/Results: You did not include the best-working VAE-based disentanglement method, namely SlowVAE [1].
- 5.4: (1) How is the reduction to 40 feature dimensions performed? Do you mean that you changed the architecture such that the output is only 40 dimensional or did you post-process the 1000 and 128-dimensional features, respectively? (2) Given that CelebA does not contain ground-truth latent variables, how did you compute the disentanglement metrics? This is a crucial aspect that is missing at the moment.

[1]: Klindt, David, et al. "Towards nonlinear disentanglement in natural data with temporal sparse coding." \
[2]: Khemakhem, Ilyes, et al. "ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA." \
[3]: Zimmermann, Roland S., et al. "Contrastive Learning Inverts the Data Generating Process." \
[4]: Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations.
summary_of_the_review: Even though the idea of the paper to look further into the disentanglement properties of contrastive methods is interesting, I cannot recommend accepting it in its current form due to a lack of comparison with existing methods, impreciseness in the language/results, and too few explanations about the observed results.

**Update**: After the rebuttal, I increased my score by one level from 3 to 5 - yet, there are still open issues that prevent me from recommending acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work explores the disentangling properties of contrastive methods. The authors discover that contrastive methods, particularly BYOL, learns disentangled representations with just a change of normalization method in the encoder. The work also proposes a new concept called "group disentanglement", which is a relaxed version of the original disentanglement. BYOL learns representations with group disentanglement and achieves SOTA on disentanglement benchmarks on not only synthetic image datasets but also a real world dataset.
main_review: Strengths:
- The simplicity of the method: BYOL with group norm instead of batch norm in the encoder.
- The proposed concept of group-disentanglement is a more relaxed but also more realistic property in real world scenarios. My feeling is that redundancies are needed in both model parameters (as shown in lottery ticket and dropout) and representations (group disentanglement).
- There are comprehensive experiments and ablation studies to support the claims in the paper: on synthetic datasets, a real world dataset CelebA (which I think is a significant step to study disentanglement), and important hyperparameters.

summary_of_the_review: I am not very familiar with the field of disentanglement study, so I am not able to judge very well the significance of this work. However I like the simple and clear idea (to study disentanglement properties of contrastive methods) and the comprehensive experimental results. I especially like the concept of group disentanglement and the improvement of benchmarks on a real world dataset. Therefore I am recommending an accept.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper makes an interesting empirical observation - BYOL representations have better disentanglement properties, according to some metrics, than current specialized methods. Furthermore, the authors found that the selection of the normalization function affects the results. The authors also claim the dimensions are "group disentangled" although this is only shown on one dataset.
main_review: Before knowing that we were assigned this paper for review, we were excited about the claims of this paper and replicated the results. We are therefore now familiar with the paper. Although the results in the paper can be replicated, the claims are a little misleading. We will explain below:

Strengths:
* The observation that BYOL features have better disentanglement properties than specialized methods is interesting.
* The observation that group disentanglement appears for the dSprites and Shape3D datasets is interesting, but its importance might be overstated
* A useful study of the effect of the normalization method on the disentanglement quality.
* The paper for the most part is nicely written and has attractive figures

Weaknesses:
* The presented metrics are incomplete. DCI is not provided on most of the synthetic experiments. Beyond dSprites (which is an easy dataset) only factorVAE is reported which gives a limited comparison between the methods.
* Misleading claim - representations are group disentangled - first of all, we did not find this to be correct for SmallNORB which is the hardest synthetic dataset. More importantly, we found that even for the easiest dataset, the only reason why it looks to experience group disentanglement is that mutual information is only computed at the single dimension level. However, if we take the 20 dimensions that are least imformative about an attribute, we could still classify the attribute with near perfect accuracy. So the claim of group disentanglement is misleading, there are just a group of dimensions that each individually seems to have information about a distinct factor, but the group is not disentangled at all.
* I'm not sure if this is actually claimed but it might be confusing to the reader who might think that group normalization is related to group disentanglement . In fact, the experiments showed this is not the case. Choosing a single group is fine. So it's really about replacing batchnorm by layernorm type normalization. 
* No analysis is provided to explain why BYOL representations have these attributes. As the method here is not applicative by itself, its main utility is the theoretical understanding. We not think that such an understanding is provided here.
summary_of_the_review: To summarize, the empirical observation that BYOL features are better disentangled than other methods is interesting. However, there are two important limitations: i) there is very little analysis trying to explain why this happens, and as no applications were shown, better understanding and analysis should be the main product of this research but they are lacking. ii) the experiments are incomplete - the group disentanglement is only shown for one dataset, our replication did not observe it for SmallNORB, such selection of results might make readers reach the wrong conclusions about the properties of the method. I suggest more complete experiments and more analysis.

###################

Our discussion with the authors and the poor experimental design reduced our confidence in this paper - we change our recommendation to rejection.
review_title: null

Meta-review: The paper provides additional empirical evidence that self-supervised learning methods can help disentangling factors of variation in a dataset. That said, the paper can benefit from better framing and perhaps comparison with existing work (e.g., https://arxiv.org/abs/2102.08850 and https://arxiv.org/abs/2007.00810). Furthermore, the authors acknowledge that there was a bug in their code, which I believe should at least lead to softening the claims about group disentanglement. Accordingly, please consider revising the paper and re-submitting to other venues.
Decision: Reject