Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an unsupervised approach to achieve group-based disentangled representation learning, as opposed to existing environment-based approaches. A theoretical framework for the group-based VAE was proposed, and implementations of the group and isomorphism reported. Experiments were performed on four common benchmark datasets, with comparisons to several representative VAE-based disentanglement framework (beta-vae, annealVAE, factorVAE, beta-TCVAE), demonstrating the groupified VAE achieves better disentanglement in many metrics. Qualitative evaluations demonstrate the cyclic representation space learned by the groupified VAE.
main_review: Strength:

This paper appears to be the first in bringing and realizing the concept of group-based disentanglement in unsupervised VAE. The method was rigorously motivated and backed by theoretical analysis. The experiments were relatively thorough in terms of the datasets, metrics, and comparison disentanglement VAE methods considered. The improvement of metrics, while moderate to marginal in some cases, were consistent across most metrics and datasets.

The visuals of the cyclic representation shown in section 5.2 and Fig 5 are especially interesting, as well the latent space shown in Fig 3 demonstrating the benefit of the isomorphism loss.

Weakness:

Some of the metrics seem to be weaker than usually reported in the comparison methods. E.g., The MIG metrics reported for dSprites seemed to be rather low for each of the "original" VAE methods compared to those seen in literature (e.g., Fig 3 in [1]). Please clarify, especially since the margin of improvements in many metrics were less than 1 standard deviation of the statistics.

While it is appreciated that the presented work considers a "unsupervised" learning of representations in comparison to existing works that uses the interaction with the environment as supervision to adopt the group-based definition, it would be still desirable to see a comparison of performance to understand what is the price to pay to go from a supervised to unsupervised setting. This is somewhat related to the comment above regarding avoiding using weak baselines.

In the final manuscript, please adjust the figures and tables such that they 1) appear in the order they're referred to in the text and 2) are spaced as close to the text description as possible. Right now Fig 4 and Fig 5 are referred to first, before Fig 3 and 2 are discussed in the text.

summary_of_the_review: This is an overall well written manuscript describing a novel method backed by in-depth theoretical analysis. The experiments were well designed and evaluations relatively thorough. There are some questions that can be addressed and improvements that can be made to the current manuscript, but overall I consider this to be an interesting work that will have good discussion potential for the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a theoretical framework on the learning of group-based disentanglement representations. It proposes a method to learn a cyclic group representation with the Abel loss and Order loss based on VAEs. Experiments validate the effectiveness of the proposed method on improving existing disentanglement VAE models.
main_review: Strengths:
1. The authors provide a theoretical framework on the learning of group-based disentangled representation, which is a step forward to the more general guaranteed disentanglement learning.
2. The experimental results validate the disentanglement effect of the 'Groupify' operation on multiple base models and synthetic datasets.
3. The proposed technique is compatible with all the information-theory-based baseline models. 

Weaknesses:
1. The introduction of $\Phi$ group is unnecessarily complex. Do the authors mean the $\Phi$ group is a direct product of $m$ cyclic groups of order n (or even simpler, $m$-dim Torus)? Is each cyclic subgroup represented by $(sin((2\pi z))/n), cos((2\pi z)/n))$, with z being the output of the VAE encoder? The generator of each subgroup is the action of rotating each circle by step $2\pi/n$? I see no necessity of incorporating more concepts from Group Theory like 'congruence class modulo n', 'n-th root unity group', 'additive group of integers modulon'. To me they are all the same in the context of this paper.
2. The mention of 'n-th dihedral group' is not informative, or even misleading. Dihedral groups are generated by only two subgroups, rotation and flip, which is not very related to the group used in this method (a product of m cyclic groups of the same order n).
3. The Abel loss and Order loss can only encourage the representation to maintain a group structure of m-dim torus. However, this does not solve the unidentifiability problem that motivates this method (the impossibility of disentanglement mentioned in the abstract, introduction section, and the paragraph before Sec. 4.1) because the ground-truth correspondence between the group action on the world space and its action effect on the image space is still unknown. The real problem here is not how to learn a group that is isomorphic to an m-dim cyclic group, but how to make sure the learned cyclic subgroups correspond to the interpretable variations shown in the image space. This paper proposes a method to learn a cyclic-group representation, which is valuable by itself in some way, but it still falls into the unidentifiability problem proposed in Locatello et al. 2019b.
4. I don't think $\phi\phi^{-1}=e$ can replace $\phi\phi^{n-1}=e$ in the Order loss. $\phi\phi^{n-1}=e$ ensures the cyclic structure of order n, but $\phi\phi^{-1}=e$ holds in any group with arbitrary order. The equation $\phi^{-1}=\phi^{n-1}$ holds when the cyclic structure has **already been learned**, but $\phi\phi^{-1}=e$ alone cannot help the learning of a cyclic structure.
5. It looks like the proposed VAE model is a special case of the Commutative Lie Group VAE proposed in https://arxiv.org/abs/2106.03375 with the learned group representation being replaced by a pre-defined direct product of cyclic groups. They should be more clearly discussed and compared in the main paper.
6. Currently only the synthetic datasets provided in the open-sourced disentanglement library (https://github.com/google-research/disentanglement_lib) are used in the experiments. More results on some real-world datasets like CelebA should be provided to validate the model's generalization ability.
7. The paper only shows the effectiveness of the method as an incremental technique to improve the existing disentanglement models, but does not show its stand-alone performance. If the proposed constraints are effective enough, they should outperform the existing SOTA models by being directly added to a vanilla VAE (as all the baseline models do). 
summary_of_the_review: I believe the theoretical contribution of this paper is valuable to the group-based disentanglement learning (or general disentanglement learning) community, but I still have concerns listed in the main review section. I currently rate this paper as weak reject, but I will be glad to improve my score if my concerns are properly solved.

======

After rebuttal:
Thanks for the authors' response to my comments. I think a large part of my concerns has been addressed. I will increase the score to 6 weak accept.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a theoretical framework for unsupervised representation disentanglement (in the sense of Higgins et al.) based on three constraints (group structure, data and model). It further describes a learning method for satisfying these constraints, and experimentally shows some performance gain on traditional disentanglement metrics. 

My intuitive understanding of the constraints is a follows:
- the model constraint is satisfied by construction of the agent group, whose structure is transferred from the latent space by the model.
- the group structure constraint ensures that this transfer is an isomorphism, and thus that the agent group is actually a group.
- the data constraint ensures that the agent group actually matches the ground truth generative factors. 

Please clarify if one of my statements is incorrect.
main_review: Strengths:
- The mathematical part of the paper and experiments appear sound.
- The isomorphism loss is to my knowledge novel, and seems to bring performance gain across datasets and models.

Weaknesses:
-  The paper is at best ambiguous and at worst misleading concerning whether or not it claims to solve the problem of unsupervised representation disentanglement
    - *"we offer an option [...] for the inductive bias that existing VAE-based models lack"*
    - *"Finally, we provide a learning model based on the existing VAE-based methods in an effort to fulfill the three conditions."*
    - *"we are the first to provide a theoretical framework to make the formal group-based mathematical definition of disentanglement practically applicable to unsupervised representation disentanglement."*
   - *"making the learned representation conform to the group- based definition without relying on the environment"* 

   But the paper never theoretically shows that the data constraint is actually satisfied, and instead relies on a necessary condition of the data constraint. This greatly undermines the theoretical contribution of the paper.
- I find the mathematical writing to be overall quite obfuscated, which complicates assessment of the theoretical contribution.

As actionnable feedback:
- I would recommend to be more open about which theoretical guarantees the proposed framework actually provides.
- Could you please give intuitive explanations of the role of the isomorphism loss, its novelty and comparison to related work, and why it brings this performance gain? 
summary_of_the_review: I find this paper to be overselling the importance of its theoretical contribution, and will therefore recommend rejection. I am looking forward to discussion with the authors, and will reconsider my rating if the authors respond to the two points listed as actionnable feedback in my main review. In particular I'd like to hear more about the isomorphism loss.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new theoretical framework for achieving unsupervised representation disentanglement based on $n$-th dihedral group. Further, it proved three sufficient conditions on the model, group structure, and data respectively. Evaluations on multiple benchmark datasets demonstrate the proposed framework achieves better mean performance with smaller variances compared to some VAE models.
main_review: Pros:
1. Proposed a new theoretical framework for achieving unsupervised representation disentanglement based on $n$-th dihedral group.
2. Provided theoretical analysis for the proposed framework via model and group structure
3. Conducted extensive experiments to verify the proposed framework

Cons:
1. Missing some latest works on disentanglement. The related work section is a bit weak since it only discussed the VAE models before the year 2019. In fact, there are a lot of related works about disentangled representation learning after 2019, such as TamingVAE, ControlVAE, and other works [1,2,3,4].
2. Please also compare the proposed framework with the baselines in recent work as mentioned above.

3. In Table 1, for dSprites dataset, the MIG score is much lower than the result illustrated in the literature $\beta$-TCVAE and ControlVAE. Please clarify and explain it. 

[1] Srivastava, Akash, Yamini Bansal, Yukun Ding, Cole Hurwitz, Kai Xu, Bernhard Egger, Prasanna Sattigeri, Josh Tenenbaum, David D. Cox, and Dan Gutfreund. "Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modelling." arXiv preprint arXiv:2010.13187 (2020). 

[2] Shao, H., Yao, S., Sun, D., Zhang, A., Liu, S., Liu, D., ... & Abdelzaher, T. (2020, November). ControlVAE: Controllable variational autoencoder. In International Conference on Machine Learning (pp. 8655-8664). PMLR.

[3] Kim, M., Wang, Y., Sahu, P., & Pavlovic, V. (2019). Bayes-factor-vae: Hierarchical bayesian deep auto-encoder models for factor disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2979-2987).

[4] Lezama, J. (2018, September). Overcoming the disentanglement vs reconstruction trade-off via jacobian supervision. In International Conference on Learning Representations.
summary_of_the_review: This paper proposed a new theoretical framework for achieving unsupervised representation disentanglement. My concern is that this work only compares it with the baselines before 2018, so it is hard to say whether it outperforms the recent baselines.
review_title: null

Meta-review: The submission provides a theoretical framework on the learning of group-based disentanglement representations and proposes a novel method to learn such representations.

The reviewers appreciated the novel perspective of the paper in introducing the concept of group-based disentanglement in unsupervised VAE. Furthermore, the approach was considered to be soundly theoretically motivated and experiments to be extensive. There was a lively discussion between reviewers and authors about certain ambiguities in the manuscript; however, they seem to have been largely resolved to the reviewers' satisfaction.

While there was a reviewer with a very low confidence recommending rejection, this paper brings an indisputably interesting novel perspective to the learning of unsupervised representations and I thus recommend acceptance.
Decision: Accept (Poster)