Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a framework to model disentangled directions for pretrained models. Such an approach mitigates the problems with poor generation quality arising while training models with additional regularization terms to force disentanglement. The underlying idea is contrastive-based: similar image variations are caused by changing the same factors in contrast to the remaining image variations. The proposed framework is model-agnostic: it can be applied to GANs, VAEs and flow models.
main_review: Strengths:
- The approach does not require any specific training.
- There is no fixed generative model type: it can be applied to GANs, VAEs and Flow models.
- The method significantly outperforms previous models in terms of disentanglement metrics.
- The method is quite stable to random seeds.
- The authors provide a thorough ablation study, report the model accuracy with std due to random seeds, check the model sensitivity to the values of hyperparameter T.

Weaknesses:
- The approach requires many 'tricks' and parts to work: Navigator, Contrastor consisting of two weight sharing encoders, contrastive approach, hard negatives flipping. Each component requires its own set of hyperparameters. The overall performance gain is significant, and the necessity is partially covered in the Ablation study section. But I wonder if it is needed to have two encoders with shared weights or it is possible to have only one? Is it required to tune hyperparameters for every component?
summary_of_the_review: Overall, the proposed method is outperforming previous approaches in terms of disentanglement scores. My main concern is the general complexity of the model. 

UPD: I keep my score the same.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes DisCo, a framework that learns disentangled representations from pretrained entangled generative models. Extensive experimental results show that DisCo outperforms many baselines in both quantitative and qualitative evaluations.
main_review: Pros:
1) The proposed method is novel and achieves SOTA results in disentanglement while ensuring good generation quality.
2) Extensive experiments and ablation studies.
3) In general, the paper is well written and easy to read. 

Cons:
1) There are still some flaws in the proposed method.
2) Some details about how to compute MIG and DCI for discovering-based methods are missing.
3) MIG and DCI metrics are out-of-date and may not well characterize disentanglement.

summary_of_the_review: 1) Novelty:
- The authors propose a novel and interesting idea of learning disentangled representations by reusing the decoder/generator of pretrained generative models and only learning the directions that lead to disentanglement. 

2) Correctness of the method:
- Is there an absolute operator in Eq. 1?
- $\mathcal{V} \in \mathbb{R}^{J}$ is not mathematically correct, it should be $\mathcal{V} \subset \mathbb{R}^J_+$ since $v$ is always >= 0.
- Why are the sizes of the query key set and the positive key set different? They should be the same in contrastive learning. If the authors use multiple positive keys per query key, the size of the positive key set should be $B \times N$ instead of $N$.
- I think the formula of NCE in Eq. 2 is incorrect. The sum in the nominator should be outside the $\log$ and should be treated as the mean over $N$. Which equation in (van de Oord et al., 2018) did the authors refer to? 
- I would like to see mathematical proof of why the BCELoss $\mathcal{L}\_{logit}$ (Eq. 3) is a lower bound of the NCELoss $\mathcal{L}\_{NCE}$ (Eq. 2). I have never seen such result in (van de Oord et al., 2018) or anywhere.
- The loss Eq. 3 is indeed an upper bound of the negative mutual information (if the two sums in Eq. 4 is replaced by mean) although it has a different form from InfoNCE (van de Oord et al., 2018). I think the authors simply use the loss without really understanding what it is.
- I do not really understand how contrastive learning help disentangle the right factors (or directions) if the latent code z and the shift $\epsilon$ are sampled randomly for both query and positive samples. Could the authors elaborate more on this?

3) Clarity in presentation:
- It seems that the flipping of negative samples into positive samples in Eq. 7 plays an important role in the model’s performance but is not carefully analyzed in the paper. I am curious about the main cause of flipping. Is it due to the poor selection of positive and negative samples at the first step? How does the flipping rate change during learning? Could the authors show a curve of this? In Table 4, I see that negative flipping does not improve the performance if one-hot regularization is not enforced. Could the authors explain this?
- How MIG and DCI are computed for discovering-based methods that only use GAN? It seems that these discovering-based methods use no encoder which is required to compute MIG and DCI.

4) Choosing metrics:
- I think the authors should use more up-to-date disentanglement metrics (e.g. JEMMIG [1]) when evaluating their method. The problem with MIG and DCI is that MIG was shown to capture only modularity, DCI consists of 3 separate sub-metrics and each of them only captures one aspect of disentanglement [2]. Could the authors tell which sub-metric of DCI you are using for evaluation? 

[1] Theory and Evaluation Metrics for Learning Disentangled Representations, Do & Tran, ICLR 2020

[2] Measuring Disentanglement - A Review of Metrics, Zaidi et al., 2020
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel representation learning technique to disentangle the latent space of pre-trained generative models, by discovering semantically meaningful directions in them.

The method trains a navigator and a delta-contrastor network, which consists of 2 encoders sharing weights. First, random samples are perturbed along the directions obtained from the navigator. The perturbed vectors are then decoded with the pre-trained generator, then encoded and the difference between 2 samples are taken. The output is in the variation space, where a contrastive learning technique clusters together the samples that were perturbed with the same direction.

main_review: Good:

The idea is very simple and easy to implement.

The paper is very well written and easy to understand.

There are extensive ablations that show the effect of design choices and hyper-parameters.

The qualitative results look very good for disentangling, the proposed method preserves e.g. the identity much better when changing other attributes, like smile or baldness.

Quantitatively the proposed method shows better performance than the baseline for many datasets and 3 different kinds of generative model: GAN, VAE and Flow. This is impressive and shows the methods generality.

Bad:

Although the paper explained the method well from the perspective of reproducibility, it does not explain why the method should chose semantically meaningful directions. One can imagine a shortcut scenario, where the method learn 0.5a+0.5b and 0.5a-0.5b directions, where a and b are perfect semantically meaningful directions. In principle the training loss could be minimised with this solution as well (?). The reason why this does not happens is because of the implicit biases in the networks (?). But then why is this method performing better than prior works?

"(ii) the factors are embedded in the pretrained model, severing as an inductive bias for unsupervised disentangled representation learning."
This still allows for the mixed solution 0.5a+0.5b and 0.5a-0.5b.

I think the following statement is incorrect, it should be removed:
"A composed of 3 fully-connected layers performs poorly, indicating the disentangled directions of the latent space W of StyleGAN is nearly linear."
- W is nearly linear because there are good directions in it, and a linear method can perform well in it.
- The 3 layer network fails for some other reason, in principle it should work at least as good as the linear model, as it has the preresentation capacity.

The method is very sensitive to the ratio between positive and negative samples. A very good tuning is needed, which is shown in the paper for most hyper-parameters. One might think that the gains come from the extensive tuning rather than the proposed idea itself.

minor:
- Although the images are resized to 64x64, it would be nice to see full resolution results with e.g. the StyleGAN2 generator. Or was the generator also retrained with reduced size images (for faster training I guess)?
- some typos and grammar could be fixed, e.g. "... generative model are two-ford: ..."

summary_of_the_review: I think the paper contains good practical ideas and a very extensive experimental evaluation.

The main weakness of the paper is the lack of deep insights. The reader learns a simple idea that is easy to implement and works well in practice, but does not get an answer to the "why" question.

Overall the good outweighs the bad.

--------- UPDATE ---------

I have read the other reviews and the rebuttals. The authors have clarified some details in the experiments. I think the experiments are strong and give valuable data for future research. I raise my score to accept.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to learn disentangled representations via contrastive learning on well-pretrained generative models. Extensive experiments are conducted on various datasets and the results validate the effectiveness of the method.
main_review: Strengths:
1. Learning disentangled representations via contrastive learning on pretrained models is an interesting direction for this community.
2. The introduced method is intuitive, simple, and effective.
3. Extensive experiments on disentanglement learning and high-resolution datasets are conducted with satisfying results obtained.
4. The introduced entropy-based domain loss and the hard negative flipping technique are effective and practical.
5. The proposed method is general for multiple generative models.

Problem:
For qualitative results on StyleGAN2 (Fig. 4 and Fig. 17 - 20), I wonder is there any manual selection on the layers to modify in the w space (like done in the methods GS and CF) or the modification happens globally on all layers in the w space?
summary_of_the_review: The introduced method is simple, effective, and general for disentanglement learning. I recommend accept for this paper.
review_title: null

Meta-review: The paper proposes a framework, named Disentaglement via Contrast (DisCo), to learn disentangled representations via contrastive learning on well-pretrained generative models. The method aims at simultaneously discovering semantically meaningful directions in pretrained generative models and training and encoder to extract them. The method uses contrastive learning where random samples perturbed along the discovered directions are regularised to be similar. The method is versatile and can be applied to various pretrained non-disentangled generative models including GAN, VAE, and Flow. Extensive experimental evaluation shows the benefits of the approach.

The authors provided a strong rebuttal addressing many of the concerns raised by the reviewers, including running new experiments (such as adding the JEMMIG metric to measure disentanglement as requested by Reviewer sBQs). This led to all reviewers recommending to accept the work.

The paper provides an exhaustive empirical evaluation testing several models and results are convincing. This was highlighted by all reviewers.

While the high level description of the method is clear, in practice the method is quite sophisticated requiring many heuristics (e.g. entropy-based domination loss or flipping hard negatives). This requires tuning several hyperparameters and complicates the message. This is mitigated by an ablation study presented by the authors highlighting the importance of each component. This was highlighted by Reviewer j95X and the AC agrees. The paper does provide implementation details, and reproducibility is not a concern.

Related to this point, Reviewer Go6R points out that the paper falls short in providing clear explanations on why the method is able to find meaningful semantic directions, and on where do the gains of the proposed model come from. While the paper could improve in this direction, the proposed empirical validation is convincing.

Overall the paper presents an interesting method that performs well in practice. All reviewers recommend accepting the work. The AC agrees with this recommendation.
Decision: Accept (Poster)