Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a novel method for disentangling latent representations, specifically for use with variational auto-encoders. This is accomplished by treating the latent codes as a tensor product of 1-spheres which results in their forming an n-dimensional torus where n is the number of latent factors. Additionally, the authors propose a new metric for analysing the quality of the decompositions. These methods are well founded and make intuitive sense, and the authors supplement this theoretical motivation with an array of tests comparing to several existing methods on several different datasets. The results demonstrate that the methods work well both qualitatively and quantitatively.
main_review: Strengths: I enjoyed this paper. It is well written and tackles an important problem of disentanglement of latent factors in VAEs. The method of decomposing onto a set of circles that form a D-dimensional torus makes sense and is clearly described. The empirical tests are thorough, presented honestly, and support the theoretical intuition both qualitatively and quantitatively.  The figures are well done and aid in the flow. Figures 2 and 3 certainly demonstrate the effectiveness of the method!

Weaknesses: I struggled to find any major problems with this paper. My main unresolved questions about the method are: (1) how does it scale with the dimensions of the tori and (2) how it preforms when a factor is not well represented as a circle. These are addressed by the authors in the conclusion. I do not think (1) is a major weakness as I think the simplicity of the idea offsets a problem which may not arise in practice, this method will work well when the data are well explained by a few latent factors. Question (2) is central to the method, but I also think it raises a qualm that is beyond the scope of this paper. Since these were my main questions, I would like to see the heatmaps as in Figure 2 for the other experiments described in the “Dependence on beta and on D” section, but these could be added to an appendix. The reason is I would like to see the behaviour as D is varied as this would reveal more clearly how the method handles over and underspecification of the latent space. 

I would also like to see a deeper investigation of the proposed DC-score metric, but, as is, it makes sense and is effective. Any further development of this element of the paper is likely beyond the scope.

Minor quibbles: 
1) Equation (9) could be rewritten so it is less cramped and easier to parse.
2) The terms alpha_k in the same section only take on values 0 and 1, otherwise the result would not be a torus. Do the authors intend for this to be a future modelling choice point? If not, I think the notation could be cleaned up to improve clarity. 


summary_of_the_review: I think the paper is well composed, interesting, and convincing. Its methodology is well founded and is clearly communicated, and the empirical results strongly support method. I think it also opens many important directions for subsequent research. For these reasons, barring something major that I have overlooked, I recommend acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an autoencoder where the latent space is defined on a torus. A D-dimensional torus is represented by the tensor product of D unit circles. They argue the torus induces disentanglement as the analogous of entropy entanglement in quantum physics. Using 5 datasets, they show both the reconstruction performance and disentanglement score (Eastwood & Williams, 2018) are high.
main_review: This study provides interesting results when using the torus as the representation space. I think the idea is novel. The utility is evaluated 5 datasets, and the results are convincing. I enjoyed reading this paper. 

The main drawback of this paper is the lack of theoretical analysis. While the motivation of introducing the torus is based on theory, the property of the proposed latent representation (5) is not analyzed. For example, could you provide a lower bound of DC score when we employ (5)?

I'm puzzled about the connection between this study and quantum physics. I'm not an expert of quantum physics, and I have no idea what are entanglement property, Von Neumann entropy, and so on. Please elaborate on these concepts and how they are related to the torus representation. I believe this revision significantly enhances the manuscript.

Minor comments:
- Typo: n in (5) seems to be D.
- The sentence below (8): the index (a, \alpha) should be an integer, but why they are distributed from the Gaussian?
summary_of_the_review: The concept of the torus representation is novel and its practical value is empirically convincing. However, there is no assessment in terms of theory. The connection to quantum physics is not well explained.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new autoencoder architecture that achieves better recovery of the latent factor structure in artificial datasets according to previously established metrics: disentanglement, completeness, and informativeness.  The key idea is to nonlinearly project a small number of latent factors into a higher-dimensional space, based on the topology of the torus, in such a way that prevents an arbitrary linear rotation of the latent factors from yielding an equivalent representation.  The approach is compared against existing approaches for variational autoencoders focused on learning disentangled representation, and achieves higher disentanglement and completeness, as well as competitive or best informativeness (reconstruction) on a majority of datasets considered.
main_review: Strengths
 - Method is simple yet innovative
 - The intuition behind the method is explained
 - Integrates very well with previous frameworks in the literature
 - Evaluation is convincing, including a direct demonstration of a strong linear match between ground truth and learned representations
 - Discussion of results satisfactorily accounts for why the method performs relatively worse in one dataset
 - The discussion frankly acknowledges a limitation of the method in terms of dealing with large numbers of latent causes

Weaknesses
 - Evaluation is based on artificial datasets, similar to the rest of the literature.  An attempt to use the method on real data, even if inconclusive, would be informative, especially since one of the limitations of the method is in dealing with large number of latent causes (which would presumably be much higher in real data than for simulations).
 - The explanations of the success of the method in terms of quantum physics concepts fall short of giving actual insight into why the method works.  Why are the physics analogies appropriate in this problem, if they are?
 - It is unclear whether the torus topology is as crucial as claimed.  If we ignore the circular structure, the method greatly resembles basis expansion methods used in linear models with higher-order interactions (https://en.wikipedia.org/wiki/Interaction_(statistics)).  Is nonlinear basis expansion the real secret ingredient to making this approach work, or does the torus topology (transforming angles into points on a 2d circle) play an additional role?
 - Is Lasso appropriate for evaluating toroidal representations, given the possibility of rotation?
 - How were the tuning parameters (beta, etc.) tuned?
 - I am baffled by how the learned factors can represent z2, z3, and z4 linearly in Figure 2, given the rotational invariance inherent in the toroidal representation.  Is it because of R, G, B are mostly represented in the linear part of the basis expansion v_orient?

Minor comments
 - The convention (as well as the rules of English) seems to have "autoencoder" as a single word, not two, but I could be mistaken.
 - In eq. 9, "N(0,1)" should be "N(0, sigma)"?
summary_of_the_review: This is an innovative method that looks very promising given the limitations of the evaluation on artificial data.  The paper is not perfect, and could improve in being more critical of the approach proposed, as well as reproducibility in terms of describing the evaluation procedure.  But the results clearly show an advantage over previous approaches on a variety of artificial datasets, which is technically impressive even if the practical impact of the method on real data is still unknown.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to use the tensor product of n unit circles to learn the disentangled representation. Experiments on synthetic datasets show the effectiveness of this technique.
main_review: Strengths:
The method proposed is simple and looks easy to reimplement.

Weaknesses:
1. The motivation 'entropy of entanglement' is not clearly explained. Considering this is not a commonly used concept in the disentangled representation learning community, I would suggest the authors provide more explanations on it. This may include (1) an intuitive descriptive introduction about what 'entropy of entanglement' is; (2) a detailed explanation on its relation to the disentangled representation learning; (3) a few experiments conducted on the disentanglement learning datasets to validate and demonstrate the claimed connections to 'entropy of entanglement'; (4) why it can solve the unidentifiability problem (Locatello et al., 2019); (5) how this 'entropy of entanglement' motivates the use of the tensor product of n unit circles as the representation.
2. The 4-th paragraph in the Sec. introduction says the n-torus $T^n$ has a low entanglement property. I wonder do the commonly used n-dim vector representations have low or high entanglement property? Which type is higher? How to compare these two types of representations quantitatively by 'entanglement property'?
3. I can see there are two steps of new techniques incorporated in the proposed method. One is the usage of multiple cyclic representations with each circle indexed by two 1-dim latent variables (defined by Eq. (6)(7)(8)). The other one is the flattening operation defined by the tensor product $\otimes$ in Eq. (5). Can the authors explain why they are motivated individually? More specifically, (1) why n unit circles are better than traditional vectors and (2) why we must use the tensor product to forward them to the decoder?
4. As the authors have noticed by themselves, the proposed tensor product method cannot generalize to a large number of latent dimensions because the dimensionality of the representation increases exponentially with the torus dimension. I think the authors should provide a solution to this obvious problem in this paper with experimental support.
5. No ablation study on the number of torus dimensions is provided in the experiment section.
6. Experiments on some complex datasets like CelebA should be provided to show the generalization ability of this method.
summary_of_the_review: Based on my concerns, I think this paper requires some significant improvements and I rate it as rejected for now.
review_title: null

Meta-review: Three reviewers had a positive impression of this paper, two of them were willing to champion it. The main positive aspects mentioned by these reviewers were clarity, methodological strength, novelty and convincing experimental evaluation. On the other hand, the was one clearly negative vote, raising issues about the proposed concept of 'entropy of entanglement' and about the use of tensor products. It seems that after the rebuttal, this reviewer was still not fully convinced. In my opinion, however, the rebuttal addressed most of these points of criticism in a clear and transparent way, so I recommend acceptance.
Decision: Accept (Poster)