Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper makes two contributions:
1 - It presents a framework for measuring linear symmetry based disentanglement (LSBD). In particular, it addresses the challenge that LSBD is only defined with respect to an optimal decomposition of the latent space $Z = Z_1 \oplus \ldots \oplus Z_k$ that needs to be identified by the metric.
2 - The metric serves as additional supervision in the training of a VAE on toy datasets with at most two symmetry transformations. The results show quantitative improvements in terms of disentanglement.
main_review: ## Recommendation
I am recommending to accept this submission which offers solid mathematical treatment 
- the background, problem and goal of the metric are clearly and intuitively explained
- the dispersion approach promises to reduce the level of transformation supervision needed by the metric
- section 4.2 has some very nice aspects, in particular (i) extending to non-uniform sample distributions via a reparametrization of the measure using a transitivity assumption is very rigorous and elegant (ii) the final definition of the metric in equation (7) is clean and general.
- the practical implementation in section 4.3 is sound

The recommendation is weak because I see the following limitations to the significance the approach:
- the LSBD-VAE seems to make the assumption that the latent state decomposition $Z = Z_1 \oplus \ldots \oplus Z_k$ is now a given of the problem, which simplifies differentiability of the metric and seems to contradict the prior definitions. I would argue that this diminishes the originality of the method. Also, this would explain the results described in 7.3 that "LSBD Representations Also Satisfy Previous Disentanglement Notions"
- despite the claims of limited supervision, I believe the amount of supervision used (transformation-labeled batches, l193) is important, which explains why LSBD-VAE outperforms previous approaches so clearly. 
- the experimental evaluation is restricted to toy datasets.


## Remarks
- The PCA step (l139-143) is done independently for each subgroup. What happens if different subgroups are projected on the same dimensions. Is this sound?
- Is the upper bound on l155 theoretically proven?
limitations_and_societal_impact: The authors have satisfyingly addressed this question.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Summary: 
This paper aims to operationalize the symmetry-based disentanglement idea proposed in Higgins et al., 2018, and proposes a novel loss function that can be used to both evaluate and learn disentangled representations (including learning from datasets where only a subset of samples are fully labeled). The paper includes some experimental validation of the proposed ideas on synthetic datasets. It also includes cross-comparisons between both the LSBD-based metric and method to state of the art disentangling metrics and methods, which demonstrate the complementary aspects of the LSBD method (e.g., its ability to capture multi-dimensional disentangled factors, for example rotations in a euclidean plane).

Overall, the paper is clearly written and makes a positive contribution in pushing for disentangling metrics that are motivated by a novel formalism. The impact of this paper is slightly limited due to the strength of supervision required (full supervision of all factors of variation) for implementing both the D_LSBD metric and the loss.
main_review: Originality
* Novel method for evaluating and learning disentangled representations.
* The objective function can be trained in a semi-supervised method with limited fully-supervised data (although the level of supervision required is extremely high)
* You have done thorough comparison to unsupervised disentanglement approaches, but it would also be useful to know how your method fares against standard semi-supervised approaches, e.g., [1], which could be adapted for a continuous supervised variable.

Quality/Clarity
* There are multiple ways of representing different transformations. For example, rotations can be represented either as 2D coordinates (as you have assumed here) or as 1D angles. In the case of unsupervised disentangling, it’s often unclear what the most parsimonious representation should be - how could the uncertainty over representations be handled in your framework?
* For your LSBD-VAE: It would be interesting to know are the relative contributions of L_{LSBD} and the adjusted likelihood (reconstructing from $\rho(g_m) \cdot \bar{z}$ rather than z_m) to the disentangling performance in the empirical results section. 

Significance
* The supervised set must have full labels for all factors/groups. If even one factor is missing, then this method cannot be applied, since we do not know the full set of transformations that relate different data points to each other. It is not clear to me how this assumption can be relaxed with the proposed metric/objective: if we lack labels for only a single factor, then it is no longer possible to explain the dataset in terms of actions on a single input sample.
* The above also means that experiments can only be done on synthetic datasets, where all factors are known.

References
[1] Kingma, Diederik P., et al. "Semi-supervised learning with deep generative models." Advances in neural information processing systems. 2014. Kingma, D. P., Mohamed, S., Rezende, D. J., & Welling, M. (2014). Semi-supervised learning with deep generative models. In Advances in neural information processing systems (pp. 3581-3589)
limitations_and_societal_impact: The authors have adequately addressed limitations and potential negative impact
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work focuses on developing a metric evaluation metric and a semi-supervised VAE-based model for linear symmetry-based disentanglement (LSBD)[1]. When assuming knowledge of linearly disentangled group representation $\rho$, the authors deduce that LSBD is fulfilled if the last requirement is met (equivariance of a map h). The authors propose to measure the equivariance by simply quantify the equivariance of the encoding map w.r.t. to the group and observations. Although a straightforward thought, computing the metric is challenging. Therefore, the authors propose an approximation by calculating the dispersion of the inverse group representation obtained through a PCA. Similar to the evaluation metric, the authors propose LSBD-VAE, a semi-supervised version of $\Delta$-VAE to regularise the latent space based on its dispersion. They show through empirical evaluation that their method outperforms other unsupervised disentanglement approaches w.r.t. to the proposed evaluation metric.

main_review: _Originality, significance, and general assessment_

The symmetry-based approach to disentanglement is theoretically sound and appealing. However, the work did not provide a proposal on learning the appropriate representation that fulfills the group-based disentanglement requirements. This work tries to fill this gap by proposing the LSBD evaluation metric and semi-supervised VAE. Although I found this research direction very exciting, I have some concerns about this work:
* Strong assumptions which may not hold beyond synthetic data: This work assumes explicit knowledge of the data and its structure for the LSBD evaluation metric and method. For each data pair, we need to know the corresponding group and linear transformation. This is quite a strong assumption, even if the semi-supervised version only assumes knowledge of some thousands of data pairs. I worry that we cannot judge from this work whether this is a realistic setting and whether some thousands of data pairs will be enough to generalize in a non-synthetic setting. 
* Fair benchmark comparison: LSBD-VAE has explicit knowledge of the groups of pairs of data instances. Even though only optimized in a semi-supervised fashion, this gives the optimization more signals than in an unsupervised setting. Therefore, comparing LSBD-VAE to only unsupervised methods is not fair. Locatello et al.[2] proposed a weakly-supervised approach that might be a more suitable comparison.
* Applicability: I do think with disentanglement, it is not about just perfectly learning to disentangle the factors of variation, but also about the promise that disentangled representations generalize better, are fairer, and more robust. This work only shows that the model can disentangle well w.r.t. the evaluation metric they proposed. However, the part of generalization, e.g., reconstruction ability and performance on downstream tasks, has not been investigated.
* Usefulness: The big question that remains unanswered after reading this paper is: "Why should I use this metric?" It is not clear why this evaluation metric is superior to the other 5-6 existing ones. What do other metrics fail to capture, which the LSBD can? Can you show some examples, theoretically or empirically, where the other metrics but LSBD fail?

_Quality and clarity_

In general, I could follow the paper and its structure. W.r.t. related work and the base model that LSBD is using, I think the authors should elaborate more details to make the paper self-contained:
* Related work (l.47-51): This discussion is very vague and does not describe what these works do. Can you elaborate on these works and how they are different from the semi-supervised approach?
* $\Delta$-VAE was used as the base for the semi-supervised approach. However, it was barely described. For myself, it helped reading through the original paper to have a better understanding of what precisely the method does. However, for the work to be self-contained, it might be helpful to the reader to have a brief description of $\Delta$-VAE.

_Comments and questions_

* Figure 6 & ll.316-319: I do not necessarily agree with "good scores on traditional disentanglement metrics do not necessarily imply good $D_{LSBD}$ scores". In particular, Figure 6b) shows that Beta, Factor, and MOD scores are proportional to $D_{LSBD}$. High Beta, Factor, and MOD scores correspond to lower $D_{LSBD}$, and lower Beta, Factor, and MOD scores correspond to higher. Can you elaborate?
* LSBD evaluation metric: Except for the difference in the "exact" values, it is not clear at all why your evaluation metric should be preferred to the other ones?
* What is the percentage of labels used (e.g., for the results plotted in Figure 5)? You only mentioned the absolute number of labels, not the relative fraction.
* All baseline disentanglement models are trained in an unsupervised manner. In contrast, the proposed model knows the groups and uses that in a semi-supervised fashion. I think a fairer comparison would be [3] as it uses weak supervision between pairs of data. This is quite similar to the way LSBD-VAE uses data pairs with the knowledge of its group and transformation. In my opinion, this work can be seen as a weakly-supervised approach. The paper might improve from discussing the similarities with [2].
limitations_and_societal_impact: The work does not discuss any potential societal impact. Adding such a discussion would improve the paper's overall quality. For instance, in that discussion, the authors could discuss both positive (interpretability, implications for understanding and evaluating fairness/bias in models) and negative impact (using representations for discriminating against specific subgroups and thus, compromise fairness) of disentangled representation learning.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new metric $D_{LSBD}$ to measure Linear Symmetry Based Disentanglement. The metric is motivated from an idealized setting before outlining a practical upper bound that can be realized. The authors also apply their new metric as an additional loss term to a $\Delta$-VAE model and show that it greatly enhances Linear Disentanglement while also agreeing with past notions of Disentanglement. The experiments, while simple and using only $SO(2)$ as a subgroup, are demonstrative of the proposed approach.
main_review: I really enjoyed the clarity of the presentation in this paper. In particular, using dispersion as a means to define a disentanglement measure is very intuitive and also novel. Moreover, most of the expository figures are well placed and serve as a nice visual guide to much of the theory presented. Overall, I find this paper interesting and a small step towards quantifying linear symmetry based disentanglement. 

There are a few comments and mild concerns that I have however which I hope the authors can clarify in their responses. I believe the theory is sound but the assumptions of the theory must be scrutinized. The assumption that the action of $G$ on $X$ is regular is quite a strong assumption and many datasets will have no hope of satisfying this. It would be good to get a quantification of the degradation of performance with respect to the new metric if modify one of the datasets in such a way that the action is no longer free. The second, and perhaps, the biggest gripe I have with this paper is the over reliance on $SO(2)$ as a subgroup. I understand why this subgroup was chosen as it is easy to perform experiments on but much of the practical theory currently does not generalize beyond this group.  However, the paper is currently framed as a more general framework and at the moment it is non-trivial to see how this would be generalized to even simple groups like $E(2)$. As a result, I would ask the author's to either tone down the main claims or repurpose them to be for $SO(2)$.  This to me is the biggest limitation of this work, even though the new disentanglement metric itself is interesting. Moreover, in section 4 the optimization was done over the set of all possible disentangled subspaces ---i.e. finding $\rho$ but in section 5  line 163 for LSBD-VAE one key assumption is that you have a linearly disentangled group representation. This seems at first glance a major decoupling of the theory from practice. It would be great to get some insight into this.

Questions/Minor Remarks:
- Why do we need a change of basis to expose the invariant subspace? Specifically, why can't we consider the total map $h' = (PCA \circ h) (x)$ to be the equivariant encoding function?
- What is the $\rho$ in $X$ space? I think you only mention $\rho$ in $Z$ space
- I think projecting the mean to the latent submanifold is a bit ad hoc. What you really want is to use the Reynolds operator to average over the subgroup.
- Line 58: Not all groups can be decomposed (outside of trivial decompositions). Please amend this statement.
- Line 181: is the scalar curvature is 0 is this not just a flat space rather than S^1? I believe I have a minor misunderstanding here.
- Figure 5, it seems like adding more supervision for the Airplace class (yellowish orange bar) doesn't seem to help as much as for Arrow and Pixel class. Can you maybe shed light into why this is the case?
- Fig 6a is illegible. A table here would be better.
- I would highly recommend an Algorithm box to explain the training procedure of LSBD-VAE. Often it was difficult to decouple the theory from the actual objectives (with the exact equations) being optimized.





limitations_and_societal_impact: N/A
review_title: null

Meta-review: The reviewers agreed that the proposed metric for linear symmetry based disentanglement is solid.  However, the three main criticisms were:
1) The lack of comparison with semi-supervised or weakly-supervised alternative methods, considering that this method requires labeled data,
2) The lack of empirical checking that LSBD correlates with downstream properties that are more directly useful in concrete tasks, and
3) The fact that SO(2) was the only group investigated in depth in the paper, despite the presentation as a general method.

Overall, the paper is not in bad shape, but the limited cope of applicability and empirical comparisons mean it is slightly below the bar.  Other examples of symmetry groups, correlation with metrics such as robustness, and comparison with semi-supervised methods would all substantially improve the paper.
Decision: Reject