Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents the idea that the current formulation of disentangled latent representations of data that have been presented are implausible in the sense that the factors are often not actually independent and cannot be learned or generated as independent. Instead the authors put forth the idea of transformations of data that are equivariant to the latent space representation as a formulation of disentangled factors. The authors use group theoretical constructs such as shift and rotation operators to show that a latent space representation should be equivariant such transformations. In other words, if a latent space representation is rotated, it should still reconstruct correctly, because the reconstruction loss should be trained on a rotated version of the image. 

The key strengths of this paper are the examples that showcase the lack of ability to learn independent latent factors. Figure 1 displays the failure to learn rotation as a factor in the MNIST digit dataset. Figure 2 is even more convincing in that it shows that the orbits of the different factors cannot be mapped to one another and thus cannot be truly independent. 

Second, I believe that the idea that is better stated in the introduction  on how disentanglement can be framed is valuable: “ In this framework, the factors of variation are different subgroups  acting on the dataset, and the goal is to learn representations where separated (of the data) subspaces are equivariant  to distinct subgroups.” Theoretically the authors are proposing an operational view of the latent factors as separate transformations on the data, and the representation as having subspaces equivariant to the transformations. Definition 1 is trying to state the same idea but is much less clear to the average ML reader

More generally, the authors should work harder to communicate this to the ML audience. The group theoretical background from the appendix should be in the background section, particularly the idea of equivariance and group operations. 

The key weakness is that their new formulation of disentanglement is that it is definitional does not give a plan of how this should be done. Based on their description it seems as if the dataset has to come with a set of known operations on the data (like rotations) that are equivariant. How would such operations be learned de novo from the data? It seems as if the framework requires learning two things separately 1. A latent representation of the data 2. A set of equivariant operations on the data (that are perhaps cyclic generators of an orbit). It is not clear how this would be learned. 


review_title: Interesting conceptual formulation, not practically developed 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The authors proposed a new way to disentangle affine transformations without topological defects. This paper made several theoretical contributions including a new definition of disentanglement and demonstration of the topological defects in existing disentanglement methods. Experimentally, this paper showed how their proposed shift operator model is powerful when dealing with topological defects.

Disentanglement is a relatively challenging task due to the lack of clear definition and the lack of a robust evaluation method. The authors did a good job providing new theoretical definitions and providing empirical and qualitative results to support their claims. The main weakness of the paper is the lack of quantitative metrics to evaluate their approach and compare with others. In addition, the model doesn’t appear to be very flexible as it requires that the transformation is known in advance.

Strengths:
+ Overall, the paper is well written and contains a good review of advances in the theory of disentanglement.
+ The idea of addressing topological defects for disentanglement appears novel. 
+ Using operators on the entire latent space is a new direction for the study of disentanglement. The authors’ viewpoint that “isolating factors of variation” is different from “mapping these factors into distinct subspaces”, and how they propose a new definition based on this viewpoint is interesting.


Weaknesses:
- Lack of quantitative evaluation metrics. The MSE in the appendix is not enough for quantifying disentanglement. 
- Since this paper focuses on disentanglement, at least Factor-VAE, one of the other representative disentanglement VAE models should be considered when doing the model evaluation.
-  Baseline models should be optimized in a more comprehensive manner (e.g., currently the selection of beta is {4, 10, 100, 1000} and latent dimension is {10, 30}). It’s unclear whether these models have been well optimized, or what measures are used to optimize the models for this task. 
- Because the method requires that the transformation is known in advance, this limits the flexibility of the approach. 
- How different transformations impact each other is not shown experimentally - there is only an example on Fig 3E showing some visual results, but this should be elaborated on further given the goal of the paper.

Minor points:
- The complex version of the shift operator is used. It would be interesting to show another version and their differences. 
- Latent traversals results appear to be rather sparse. It would be interesting to show how the variation exists inside the model via dense traversals and the computing of generated images variation with different latent traversals.
- Rotations may be more challenging to learn. 2000 examples may be insufficient for the model to learn this transformation correctly.

review_title: Review of "Addressing the Topological Defects of Disentanglement"

final_score: rating: 3: Clear rejection
review_content: **Summarize what the paper claims to contribute.**
The authors claims to show that disentanglement into subspaces by a continuous encoder is impossible for any finite group acting on Euclidean space
The authors claim to introduce an alternative definition of disentanglement that is more flexible and leads to a 

**Strengths:**
The authors consider the problem of disentangled representation learning which is of considerable interest to the community
The authors approach the problem by imposing structure through their disentangled operators

**Weaknesses:**
The reliance of the “impossibility of disentanglement” proof seems to rely heavily on the example of the perturbed triangle. The example and its assumptions seem fairly rigid and unnatural and I am unconvinced this captures the reality of disentangled representation learning with auto-encoding networks.
The approach of adding structure by means of a transformation operator was also used in [1,2] which are cited but not compared against. Instead the authors compare against various VAEs which do not impose any external structure which does not seem particularly appropriate.
If I understand correctly, the paper seems to be based on a mischaracterization of the arguments in [3]

**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.**
Reject. See weaknesses

**Supporting arguments for your recommendation.**
While the authors tackle an interesting problem and propose an interesting solution, the arguments on which the paper is based seem flawed.

**Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment.**
As I understand, the argument is against the utility of the *linear* disentangled representation in [3]. The more flexible definition the authors propose seems quite close to *disentangled representation* in [3], please clarify the difference.
Moreover, it seems the authors suggest the definition of disentangled representations proposed in [3] requires that subspaces corresponding to factors of variation are single dimensional (section 2) which is not the case, please clarify.
How does the approach compare against other methods, namely [1,2,4] that use structure to encourage disentangling of the representation?
Section 2 asserts that the VAE and its variants do not learn disentangled representations and uses PCA to show this is true. I expect that if this same analysis were used in the structured case, a similar result would be found, in particular, since the rotation matrix interacts with multiple dimensions of the latent code. Perhaps my intuition is incorrect, please clarify.

**Provide additional feedback with the aim to improve the paper.**
Perhaps a rewording could clarify: (Supervised Disentanglement) is composed of a 2x2 diagonal block... →  is a block diagonal matrix with a 2x2 rotation matrix in the upper left block and 1s on the remaining diagonals
(just after 11) The authors state that most deep networks are differentiable, my understanding is that the common ReLU networks are not differentiable but subdifferentiable

**Possible typos:**
(VAE, beta-VAE and CCI-VAE) the ”4s” → the ``4s”
(Dfn of a group; identity element) g_k e_G = e_G g_k = e_G → g_k e_G = e_G g_k = g_k

**Post rebuttal**
I thank the authors and other reviewers for their comments and discussion. While the direction the authors pursue is of unquestionable merit, I remain unconvinced that the work as it stands is sufficiently impactful for this venue. 

[1] Falorsi, Luca, et al. "Explorations in homeomorphic variational auto-encoding." arXiv preprint arXiv:1807.04689 (2018).
[2] Connor, Marissa, and Christopher Rozell. "Representing Closed Transformation Paths in Encoded Network Latent Space." AAAI. 2020.
[3] Higgins, Irina, et al. "Towards a definition of disentangled representations." arXiv preprint arXiv:1812.02230 (2018).
[4] Cohen, Taco, and Max Welling. "Learning the irreducible representations of commutative lie groups." International Conference on Machine Learning. 2014.
review_title: Interesting idea, improper contex

final_score: rating: 7: Good paper, accept
review_content: The paper first shows that existing approaches to latent space disentanglement perform poorly when the latent space topology (usually Euclidean) does not match the actual data topology, using rotation equivariance as an example. This analysis culminates in a general impossibility theorem for this type of disentanglement. The authors then propose a relaxed definition of disentanglement and show that it can be realized by means of a shift operator in latent space. Theoretical and empirical results demonstrate the superiority of the new approach. This is a very interesting idea that represents significant progress in an important problem. 

Unfortunately, the current organization of the paper does not work well: the authors devote too much space (half of the paper!) to the explanation of the problem, and too little (barely one page) to its solution. This leaves the reader with many unanswered questions about how the new method works and what its crucial details are. Some of these questions are later dealt with in the appendix, but this is too late.

My main suggestion for improvement is therefore to move most of section C 3.1 to the main text and allocate the required space by shortening the motivation (up to section 3.2) and possibly the discussion of multiple transformations in section 4.3. Content that would get lost by this change should be moved to the appendix.

More minor points are:
* The authors repeatedly refer to "recent success of ... distributed operators", but do not cite and discuss any prior work. Please add appropriate references to the introduction or related work.
* Rotations and translations are continuous transformations, whereas the proposed shift operator is discrete. Does this discretization introduce rounding errors or other artifacts? How many discretization levels are needed, and how can this number be determined? Does discretization have undesirable limitations? Such potential limitations should at least be acknowledged. Ideally, these questions should be investigated experimentally (but this can be left for future work if infeasible in the present paper). 
* In appendix E.1, results for rotations are an order of magnitude better than those for translations. Why is this the case?
* Figure 3E: It is hard to judge if the results align with the ground truth. Preferably, the ground truth should be displayed for reference.
* Is it necessary to design the network according to a-priory knowledge of the relevant group transformations, or can this be inferred automatically? For example, what happens if the latent space implements a group that does not correspond to any symmetry in the data?

I'm willing to raise my rating if these points (in particular, the relocation of section C 3.1) are suitably addressed in an updated version of the submission.
review_title: A very interesting paper that needs to be restructured before publication.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the notion of disentanglement in a group representation theoretic setting. Disentangling is sometimes conceptualized as mapping distinct factors (e.g. position / orientation) to distinct subspaces. It is shown theoretically that such a naive notion of disentangling is impossible for topological reasons, and this is confirmed empirically. An alternative definition of disentanglement is given, where instead of confining the effect of each transformation to a subspace, an operator is used that acts on the whole latent space (this operator is chosen as a shift operator, which works for cyclic groups). It is shown empirically that an autoencoder with a shift operator in latent space is better able to learn rotations and translations.

The paper does a good job explaining why the naive notion of disentangling leads to topological problems, and convincingly backs this up with experiments as well. The insight is not new to me personally, but I can't find a reference that explains it and I think it is not widely understood, so I consider this an important contribution to the (very muddled) discourse on disentangling.

Definition 1 provides a new definition of disentangling. However, the statement is not very precise, and I am not convinced that it can reasonably be considered as a definition of disentanglement. The definition is: 

"A representation is said to be disentangled with respect to a particular decomposition of a symmetry group into subgroups, if there is a family of known operators acting on this representation, potentially distributed across the full latent, where each operator is equivariant to the action of a single subgroup."

Based on the rest of the paper, I think this means that we have for each subgroup G_i an operator phi_i(g) acting on the latent space. The definition does not make it clear that we wish the encoder to be equivariant wrt this operator and some operator acting on the input space, but I will assume that is what is meant (otherwise, having an operator acting on the latent space is a rather vacuous requirement on the encoder/representation). The definition does speak of the operator being equivariant, which I will take to mean that it is a group representation, i.e. phi(gg') = phi(g)phi(g'). The operator being distributed I will take to mean that phi(g) can be any linear map, not necessarily acting trivially on a subspace or being (block-) diagonal / reduced. 

The definition mentions that each subgroup should have its own operator, but since all of them act on the whole subspace this seems to a trivial constraint. Indeed if we have a representation of the whole group acting on the latent space, simply restricting it to each subgroup gives us a representation of the subgroups. I would further note that what is done in practice in the paper is different from this definition, because we have one latent space per operator, not multiple operators acting on the same space.

Under this interpretation, I don't see how the definition is saying anything else than that the network should be equivariant wrt some representation of the group acting on the input and output space. Although equivariance is a good property for various reasons, it does not seem to me to be reasonable definition of disentangling by itself. Indeed, the identity map satisfies this constraint trivially.

It may be that I have misunderstood definition 1, but this strengthens the case for making it mathematically precise.

Even if one can question whether Def 1 is a good formalization of disentangling, the paper does show empirically that it is easier to learn an equivariant encoder/decoder when the latent operator is a shift operator or a diagonalized complex version of it, rather than a disentangled operator (with one 2x2 rotation matrix block and an identity block; fig 3b). Although I don't know if these two approaches have been compared before, several older papers consider similar models to the shift operator model. 

For instance, in a sequence of papers Memisevic & Hinton considered factorized RBMs that do something similar. Cohen & Welling described a representation-theoretic version of this model which is very similar what is presented in this paper (at least the linear AE), and also gave a definition of disentangling (under this definition, the complex diagonal shift operator is disentangled while the original shift operator is not). Models with a stack of multiple operators were considered by Sohl-Dickstein et al.

If one wishes to define a notion of disentangling based on subgroups and representations, it may be worth investigating subgroup adapted / Gelfand-Tsetlin bases.

In summary, I think this paper contains several interesting observations and results, and I think the general direction is very interesting and deserves further study. However, I'm not convinced that this paper provides a good definition of disentangling, the experiments although convincing and well executed are restricted to simplified domains, and some of the insights / methods presented in the paper are already present in earlier work. Nevertheless I hope the authors will not be discouraged, and continue to work on this important and fundamental problem using the tools of representation theory.

References
Memisevic & Hinton, Learning to Represent Spatial Transformations
with Factored Higher-Order Boltzmann Machines, 2010
Sohl-Dickstein, Wang, Olshausen, An unsupervised algorithm for learning Lie group
transformations, 2010
Cohen & Welling, Learning the Irreducible Representations of Commutative Lie Groups, 2014
Wakin, Donoho, Choi, Baraniuk, The multiscale structure of non-differentiable image manifolds, 2005

----
Post-discussion update:
Having read the other reviews, author response and updated paper, I still think this paper is borderline. The insight that disentangling transformations as naively defined is impossible for topological reasons is valid and interesting, but seems to have been already observed by others, e.g. Falorsi et al. Nevertheless the paper does a good job explaining this so it could be useful, as some authors seem to not know about this issue. The definition of disentangling still seems a bit vague to me, and I'm not convinced of practical applicability of the proposed method. 
review_title: Review

Meta-review: This is a borderline case (quite comparable to the other borderline case in my batch). The paper has received careful reviews and based on my weighting of the different arguments I arrive at an average score between 5.75 and 6.. The authors present some worthwhile ideas related to disentanglement that deserves more attention and that could spark more research in this direction. At the same time, the level of novelty and significance of this work remains a bit limited. Taken together the paper is likely not compelling enough to be among the top papers to be selected for publication at ICLR. 
Decision: Reject