Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper takes the negative-free contrastive learning methods as an example to explore the disentanglement property of the self-supervised methods experimentally. To address the limitations of existing disentangled metrics in high-dimensional representation models, the author proposes a new decoupling metric-MED based on mutual information. Experiments on real-world datasets and high-dimensional representation space demonstrate this metric's superiority and applicability.
strengths_and_weaknesses: Strengths:
1. Existing work on Disentangled Representation Learning is limited to the generative model. This paper empirically studies the disentanglement property of negative-free contrastive learning, which is an exploratory work.
2. This paper proposes a new metric, MED/top-K MED, which extends the decoupling metric to high-dimensional space.
3. The paper is well organized, so the reader can get the gist of the article. The authors clearly describe the proposed method. In addition, the experimental results show the effectiveness of the proposed method.

Weaknesses:
1. The author designed a version of MED/top-k MED, to evaluate the disentanglement. Section 5.4 analyzes the effect of dimension on Top-2 MED. However, the effect of dimension on MED is unclear. In the upper part of Table 1, the authors provide results in 1000-dimensional representation space. However, the paper does not directly show results in lower-dimensional spaces (no additional PCA required, just set the projection dimension), such as 100 or 200.
2. The author found that contrastive learning without negatives learned a well-disentangled subspace of latent representation from experiments. It is not yet known how this subspace and the subspace learned by disentangled representation learning perform on downstream tasks, such as classification tasks.
questions: Refer to Weaknesses
limitations: The authors have not addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides experiments on disentanglement for negative-free contrastive learning. The results indicate that current metrics have limitations on this setup. As a result, the authors present a novel metric for evaluating disentanglement in the proposed scenario.	

strengths_and_weaknesses: Strengths. 
- Investigating and expanding current metrics for quantifying disentanglement can address an important issue in deep learning.  
- The assessment of current issues on disentangled metrics is done properly 
- They assess disentanglement on real-datasets rather than synthetic
- Validation of the new metric with qualitative results on generative factors

Weaknesses. 
Missing comparison with other recently proposed metrics. 
- https://openreview.net/pdf?id=HJgK0h4Ywr
- https://openreview.net/pdf?id=EbIDjBynYJ8
- https://arxiv.org/abs/2106.03375

questions: In section 4 the authors show how the MED metric finds sub-sample of disentangled factors, however a correspondent analysis for traditional metrics could also be performed for comparison?
limitations: The work validates the proposed metric empirically, however would be also interesting to investigate the theoretical justification of this method.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper tries to empirically study the disentanglement of negative sample free contrastive learning methods, e.g. BYOL, SimSiam. The authors find that the existing disentanglement metric does not fit with the disentanglement of high-dimension feature representation space. Thus, the authors propose a new running time-efficient metric named “Mutual information based Entroy Disentanglement” and MED for short. The authors evaluate the new metric on some popular synthetic datasets and a real-world dataset CelebA and argue that negative sample free contrastive learning methods can learn a well-disentangled subset of representation.
strengths_and_weaknesses: Strength: 
1. The paper points out previous methods’ drawback and propose a time-efficient metric that is designed for high-dimension space.  
2. The author report various ablation studies to show the properties and effectiveness of the new metric, i.e. uniqueness of factor-representation correspondence, and influence from manipulating factors.

Weakness:
1. The main weakness is that the authors do not theoretically prove the new metric is sound. The major message of this paper is that the previous disentanglement metrics are not good enough and the proposed new metric can fix these problems. However, the paper only showed the drawback of previous metrics and for the new metric, the authors only empirically show it may work e.g. Figure 4. Why should we believe the new metric is better than other methods? Why the value of MED can be regarded as a measure of disentanglement property? From my perspective, the best way to prove a proposed metric work is theoretical analysis, i.e. [1], and the paper misses that part. I recommend the authors consider a high-dimension linear case and prove that MED can beat some other metrics or prove some property of MED. 
2. Also, in Section 3.2, the high-level intuition of MDE is not clear. The metric has three parts, Equation (1) (2) (3) and it is quite complicated. I did not get a clear insight, i.e., $R_{ij}$ used in $\rho_i$ and $S_i$. Why should we combine them in the way in Equation (3)? The last paragraph of Section 3.2 should be longer and provide more explanation. 
3. The authors argue that contrastive learning can learn a well-disentangled subset of representation without negative samples. The conclusion here is weak. There are many further important questions that the paper does not give answers to. Why the model can learn a well-disentangled subset of representation without negative samples? What is the disentanglement difference between methods with and without negative samples? Is there any high-level intuition or further suggestion about this conclusion? More discussion is needed here. 

[1] Kornblith, Simon, et al. "Similarity of neural network representations revisited." International Conference on Machine Learning. PMLR, 2019.
questions: 1. The paper only reports performance on CelebA. Why does not evaluate MED on ImageNet?
2. In line 155, it is easy to get ground truth factors for synthetic datasets. How to get ground truth factors for real datasets?
3. In Equation (1), how to calculate Mutual Infomation for two scalars, or do they represent a distribution over the whole dataset?
4. For Partial Disentanglement Evaluation Metric, how to know which k should be used? Binary search?
5. In line 324, how to get k=3 and how to know the latent space dimension of a real dataset? 

limitations: Limitations are the theoretical analysis mentioned above.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper empirically studied the disentanglement property of self-supervised methods, such as MoCo, BarlowTwins, and BYOL. Besides, the authors validated the disagreement of current disentanglement metrics for the models with high-dimensional latent space. The authors proposed a new metric based on mutual information to measure high-dimensional representations. Massive experiments conducted on synthetic datasets and a real-world dataset showed that negative-free contrastive methods can learn a disentangled subset of representation. 

Contribution:
1. This work studied the disentanglement properties of negative-free contrastive models for the first time.
2. This work proposed a new metric for high-dimensional models and a selection strategy to pick a disentangled subset of representation.
strengths_and_weaknesses: Strength:
1. The experiments were comprehensive and massive to cover popular conventional disentanglement methods and negative-free contrastive models.
2. This paper aims to address the disentanglement measurement of high-dimensional representations and bring negative-free contrastive learning into disentanglement learning.

Weakness:
1. line 34: “latent representation, This” → “latent representation. This”
2. Comparing metrics for a model in a subfigure may better show the disagreement or agreement of metrics in Figure 4.
3. We know Orientation is hard to be disentangled, but the authors still need to show ALL experimental results without selection.
4. What is manipulating a factor in section 4.3? Getting a set of images by traversing one factor? Could you color the latent index with the factor you picked? I cannot see the existence of a well-disentangled subset for orientation.
questions: The authors claim that disentanglement property remains unexplored in negative-free contrastive learning, but I find some relevant papers, e.g., Contrastive Disentanglement in Generative Adversarial Networks. Could the authors discuss the differences? Why negative-free contrastive learning is important?

How to prove the superiority of your proposed metric? Beta VAE, Factor VAE score, and DCI disentanglement show positive results for BYOL. 



limitations: The authors did not show the superiority of the proposed metric. They need to find some cases where all metrics fail to measure the disentanglement except the proposed metric.
review_title: null

Meta-review: metareview: There was a consensus among reviewers that this paper should be accepted. The key convincing arguments that this paper studies a novel setting: how to measure the disentanglement in high-dimensional spaces. For this, the authors perform extensive experiments and come up with a novel metric. The reviewers further felt that concerns raised in the initial reviews were subsequently addressed in the author rebuttal.
confidence: Certain
Decision: Accept