Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes to relax the assumption of statistical independence used in many disentangled representation learning works. The paper proposes to only assume that the support of the latent factors’ distribution factorizes, which is a weaker constraint than statistical independence. Specifically, the paper proposes a training criterion that minimizes a Hausdorff set-distance between the latents’ actual support and its factorization. The paper contains extensive experiments on three datasets with varying degrees of correlation settings between ground truth generative factors. The proposed approach achieves better DCI-D scores as compared to previous works. The learnt representations from the proposed approach also achieve better classification performance in “distributional shift” settings.
strength_and_weaknesses: Strengths
- The proposed relaxed assumption of factorization of support instead of independence provides more flexibility, allowing correlations in the training data to be learnt.
- The paper contains extensive experiments to support the proposed approach. Consistent improvements on Shapes3D, DSprites, MPI3D show the robustness of the proposed approach.
- Improvements in classification performance in “distributional shift” settings show that the proposed approach can facilitate generalization, especially in the case of strong correlation in the training distribution. 
- The paper is well-written and easy to comprehend.


Weaknesses:

- The paper describes a problem in previous works, proposes a solution and shows that it works. This usually covers the most important components to evaluate an approach but the paper could be much more useful for the research community if the paper can provide a higher-level picture of where the proposed approach fits in this research area, for example:
    - A discussion on what different ways could provide the factorization of latents’ support and why the proposed HFS criterion is a better choice among them.
    - What are the drawbacks of the proposed approach? Should everyone replace the methods from previous works with the proposed method in all cases? If not, in what settings does the proposed approach fare better than others and in what settings it might be better to not use it? Does the proposed approach take more computational resources? Is it a more difficult objective to train? 
- The paper draws its motivation from the fact that real-world datasets contain correlations between generative factors and therefore, the independence assumption is not very suitable for these real-world settings. But, the paper doesn’t include any experiments on uncontrolled real-world datasets, for example, a good first step could be the CelebA dataset.
- The paper mentions that previous “successful” works enforce the independence assumption even more strongly when it doesn’t hold for realistic data distributions. That should result in these approaches failing poorly on realistic datasets. Can the authors provide any findings (even from other papers) to support this argument? This again emphasizes the importance of at least one experiment with realistic data for this paper (where ideally, the previous works fail poorly, perhaps even more than vanilla VAE and the proposed approach works much better).
- The paper only considers one disentanglement evaluation metric — DCI-D. Even according to the cited justification Dittadi et al. (2021), both DCI and MIG are considered to be the best. Why do the authors choose to focus only on DCI? What are the results in terms of MIG? Also, it would be better if the evaluation metric(s) being used in the paper is explained briefly.
- Can the authors explain why HFS is working the best for no correlation setting? Shouldn't independence assumption work perfectly in this case and therefore, existing works have an advantage?
- Similarly, why do HFS versions of beta-VAE and beta-TCVAE work better than HFS? If the assumptions in beta-VAE and beta-TCVAE are not suitable for correlated settings, their criterion should hurt and not facilitate disentanglement, right?
- In Figure 2, I would have expected a consistent trend from top to bottom in the first few columns but there seem to be some training correlations in between (3rd, 7th, and 8th rows) that are better than the ones surrounding them. Why is that so? How exactly is the correlation made more severe in this case? 
- Minor:
    - Definition 1.1 seems insufficient. This definition would be satisfied by the case when there is only one dimension that contains all the generative factors, the rest all being just noise. I think the authors meant one-to-one mapping between the latent dimensions and the generative factors (as mentioned in the next sentence). If so, this definition should be fixed accordingly.
    - It would be easier for the reader to understand if the authors can explain in more detail why the independence assumption doesn’t hold for the case in Figure 1, perhaps contrasting it with how the distribution would be in the case of independence.
    - Table 1, column 2, HFS is second best (bold) instead of beta-TCVAE+HFS.
clarity,_quality,_novelty_and_reproducibility: Clarity - The paper is clearly written and easy to read. Most of the decisions made are explained with justifications. Some of my concerns are mentioned in weaknesses above.

Quality - The research in the paper is of high quality. The paper describes a problem in previous works, proposes a reasonable solution and shows with extensive experimentation that it works.

Novelty - The proposed approach is the first to my knowledge to relax the independence assumption in previous disentangled representation learning works in a general manner (without explicit auxiliary variables or specific prior models).

Reproducibility - The authors provide a lot of their experimental details in the main paper and appendices. They also promise to release their code after publication.
summary_of_the_review: I think the paper tackles an important problem in disentangled representation learning research, provides a reasonable solution and supports their approach with solid experiments. I have some concerns about the reasoning behind some experimental results, the lack of any experiments with realistic data and providing a more balanced picture of the proposed approach that I have described in more detail in the weakness section. But overall, I think the paper would be useful for the research community and hence I am inclining towards acceptance.

-------------------------------------------------------------------------------------------------------------------------------------------------------
The authors' response satisfactorily answers all my concerns. Therefore, I am keeping my acceptance recommendation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To avoid the unrealistic assumption of factor independence in traditional disentangled representation learning, this paper proposes to relax this assumption to factorized support, and proposes a Hausdorff-distance-based regularization. The authors conduct experiments on 3 classical datasets to show the improved disentanglement brought by the proposed method. Additionally, ablation studies show that the proposed method has  promising potential on out-of-distribution settings. 
strength_and_weaknesses: Strength:
1.	This paper is well motivated, aiming to tackle the problem of the unrealistic assumption of traditional disentangled representation learning.
2.	This paper proposes a new and effective training criterion through a Hausdorff distance term, which can be easily combined with existing disentanglement methods.
3.	The experiments show that the proposed method can improve the disentanglement and the out-of-distribution generalization ability.
Weaknesses:
1.	Some intuitions can be further explained, e.g., in section 2.2 the situation that breaks the factorized distribution can have a factorized support. It will be more convincing to give an example which does not have a factorized support will fail to disentangle, more intuitively show the relationship between factorized support and disentanglement.
2.	Since this paper claims to aim at the realistic scenario of disentangled representation learning, it is better to conduct experiments on real world datasets instead of the synthetic datasets(at least for the out-of-distribution setting.).
3.	The compared disentangled baselines seem to be out-of-date, it is better to incorporate the more recent disentangling methods. 

clarity,_quality,_novelty_and_reproducibility: Clarity is excellent, quality is good, novelty is good, reproducibility is good because the authors promise to release the code after review. 
summary_of_the_review: I have carefully checked the main body of this paper(including the introduction, method and experiments part.) I do not check the supplementary files.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper tackles a key problem in disentanglement - the independence assumption made by most methods does not hold in practice. Instead, the paper suggests requiring independent supports between factors which can be satisfied even when correlation exists. The paper proposes to measure support independence by the empirical Hausdorff distance, but find that doing so is infeasible. Instead, it measures the hausdorff distance on slices of the representations (specifically pairs of factors). While not powerful enough on its own, the authors show the Hausdorff loss term is very helpful when combined with existing approaches. Benefits are shown, particularly when strong correlations exist in the data. Good results are also shown for OOD generalization.    
strength_and_weaknesses: + Removing the assumption of independence between factors of variation in disentanglement models is of high importance.
+ The approach of independent supports may be a promising direction for tackling this task and appears novel.
+ The formulation of the loss term seems reasonable (but does not seem to be the entire story)
+ Experimental results do seem to confirm the claims of the paper (but note weaknesses) 

- The loss term is only able to detect second order correlations, which is not quite the same as ensuring factorized support. While the authors explained why this did it, it would have been interesting to understand the performance loss due to this relaxation.
- The experiments are somewhat self-serving in the sense the correlation is linear between pairs of terms, which is also what the loss term is optimising. How would it deal with higher order correlations? 
- The loss term, combined with reconstruction was insufficient and required the KL divergence with the normal distribution, which encourages no correlation. While the authors tried to address it in the paper, I wish that they justify this finding in a clearer way.
- The datasets useed are somewhat easy. How does this method performance on the more challenging SmallNorb?
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written and clear.
Quality: The main claims seem to be supported, the experimental results and ablation are well done.
Novelty: I am not aware of previous approaches to unsupervised disentanglement based on support factoriization.
Reproducibility: Code was provided.
summary_of_the_review: Overall, I find the idea interesting and the experimental evidence positive. Please address the weaknesses above. I am leaning towards acceptance.

#######################

Post-rebuttal: The authors mostly addressed my concerns, I keep my rating and recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper moves away from the independence assumption in disentangled representation learning and relaxes it to factorized support assumption. It regularizes the Hausdorff distance between the latents actual support and its factorization to uncover correlated factors. Experiments are presented on synthetic datasets to demonstrate the effectiveness of the method in ground truth factor learning and in classification under controlled distribution shifts. While some reviewers expressed concerns over the use of synthetic datasets in the experiments, all reviewers are overall positive about the paper and acknowledge the novelty of the approach presented in the paper along with its effectiveness on the standard benchmarks used in the disentanglement literature. 


justification_for_why_not_higher_score: No experiments on real datasets.
justification_for_why_not_lower_score: One of the first papers to present a method for learning disentangled factors that does away with independence assumption.
Decision: Accept: poster