Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper follows on the work of Higgins et al. 2018 that used linear symmetry based disentangled (LSBD) representations where real-world transformations provide some structure in the data that can be leveraged. The main contribution of this paper is to provide a metric to measure the quality of disentanglement in the learned representation. The paper makes some assumptions on the samples in the dataset and assumes that there are some group actions that relate one data point to another. Under this assumption, the paper derives a simple and easy-to-compute disentanglement measure. Second, this paper shows a method that can work with partial supervision to learn disentangled representation. This is done by using some data with supervision on the underlying transformation between the data points, and another set of data points where no labeling information is given. The proposed method uses a diffusion variational autoencoder. 

Pros:

1) This paper proposes a metric that can be used to quantify disentangled representations. Disentangled representation is one of the overloaded terms in the community, and there have been multiple interpretations without any clear evidence as to whether the learned representation is disentangled or not.
2) The problem formulation and setup has been nicely illustrated using observation and inference function modeling the different factors of variation in the data, and group actions that allow the transformation of one data point to another. 


Cons:

1) My first concern is regarding the strong assumptions used in this paper. It is hard for me to think of real world data, where we can have simple group actions that map one data point to another. This restricts the use of the disentanglement measure proposed in this paper or real data. 

2) While the paper clearly argues about the importance of having a measure for disentanglement, it does not tie this to the effectiveness of the downstream application. In practice, we are always interested in disentangled representations that also capture all the relevant information from the original data. While it might be possible for the inference function h to satisfy the equation h(g_n . x_1) = \rho (g_n). h(x_1)  as given in equation (2),  but this may not mean that the learned representation contains the information associated with the original data points. 

3) The batch size used for the supervised data points is 2. This implies that there is a group action between only two samples. I am a bit concerned that this does not capture the generality of the proposed formulation proposed in section 5. 

4) The main algorithm proposed in this paper is based on the diffusion variational autoencoder (Peez et al. 2020), and the method to handle unsupervised data is just by alternating the training between supervised and unsupervised data. There is not much analysis or ablation study on how much unsupervised data can be used.  

5) No comparison is made with other disentanglement methods that use partial supervision in the form of set membership. 

review_title: Quantifying and Learning Disentangled Representations with Limited Supervision

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper contributes an interesting new measure for disentanglement which consists in: (1) applying to each latent representation a (supervised) linear transformation mapping it to the origin representation (2) measuring the average distance of these canonicalized representation to the origin to obtain a mean discrepancy measure of disentanglement. By adding this measure as an additional loss term in a topological VAE, the authors show that the model can learn disentangled representations with a limited amount of supervision on some simple image datasets.

Strong points:
- the measure of disentanglement proposed is promising and novel.
- the mathematical derivations and the simulation results are convincing.

Weak points:
- some claims in the paper are misleading because they imply that the measure proposed is unsupervised, although it is totally supervised (see below for details).
- some technical aspects of the paper lack clarity (see below for details).
- the measure of disentanglement proposed requires a lot of prior knowledge on the data and on the structure of the representation: for all pairs of data points on which the measure is calculated, we need to know what group element this transformation corresponds to, and we need to know to which linear transformation this group element corresponds to in the latent representation. This is impractical in many datasets where the transformations are unknown, and in many models where the latent equivariant operators are learned and not pre-specified.

I recommend to reject this paper due to the clarity concerns and concerns about the validity of the claims, unless they can be addressed satisfactorily during the rebuttal period.

Main concerns to be addressed:
1) It is impossible to understand from the abstract, intro and related work section whether the proposed measure of disentanglement requires supervision about the transformation between pairs of data points or not. In fact, the measure is 100% supervised. I agree that the model proposed using this measure as an additional loss term is only supervised on part of the data, but the measure itself is supervised. Here are examples of misleading claims:
-Abstract: "Although several works focus on learning LSBD representations, none of them provide a metric to quantify disentanglement. Moreover, such methods require supervision on the underlying transformations for the entire dataset, and cannot deal with unlabeled data."
-Related Work: "Moreover, their methods require supervision on the transformation relationships among datapoints for the entire training dataset."
2) Some technical aspects of the paper are unclear:  
- LSBD is clearly defined, SBD is mentioned multiple times but never defined.
- the ∆VAE model is never described, nor its architecture. The reader is required to read another paper to understand how this model works.
- "An easy-to-compute metric to quantify LSBD given certain assumptions (see Section 4), which acts as an upper bound to a more general metric (derived in Appendix C)." This more general metric is not described in the main text, and it is unclear what the underlying motivation is for this alternative metric (and reading the technical description in the appendix did not help me understand the motivation).
- Enigmatic discussion points: "Our LSBD metric and method require a number of assumptions, as explained in Section 4. This limits the applicability of the metric and method, but also provides a clear direction to what needs to be done to obtain and quantify LSBD representations if these assumptions are relaxed.", "Moreover, our metric is in fact an upper bound to a more general metric (see Appendix C), which is however less straightforward to compute." What are the clear directions on what needs to be done? How to think of this more general metric?

Additional feedback:
- The LSBD assumptions at restated three times in the main text (p2,3,4), which is unnecessarily redundant.

review_title: An interesting metric of disentanglement, but claims and technical aspects need to be clarified

final_score: rating: 4: Ok but not good enough - rejection
review_content: =============================================================================================================

Summary:
The paper starts from the Linear Symmetry-Based Disentanglement (LSBD) in [1]. The existing approaches to evaluate disentanglement require the supervision of the dataset and it is impossible for the unlabeled data. In this regard, the authors propose the new quantifying metric for disentanglement under the 'limited supervision' setting. Also, under this metric, authors provide a VAE-based framework that exploits limited supervision for the proposed metric.

==============================================================================================================

Reason for score:

Overall, I vote for clear rejection. I was hooked by the title and abstract. However, I could not find any novelty and insights to compare several measures for the disentanglement score. Furthermore, experiments are not enough to support the proposed method (see cons). I suggest the resubmission with more experiment results and justifications of their proposed methods. 

=============================================================================================================

Strong points (pros) :

(1) Mathematical definitions to interpret disentanglement are clear (but complex). The quality of writing is not bad.

(2) I like the trial to quantify disentanglement under a limited-supervised setting since it could be applied to real-world settings.

==============================================================================================================

cons : 

(1) The comparison with the other disentanglement metrics under supervised data is crucial. The proposed metric which can be applied under a weakly supervised setting should have a certain amount of consensus with other metrics. The authors need to show through experiments and provide justification by pointing out similarities and differences with the other metrics. This part is very important to persuade reviewers and readers. 

(2) I was caught off guard at the experiments since there doesn't exist any baselines in Figure 4. The authors need to add baselines to compare with $\Delta VAE$.

(3) The idea underlying paper is quite overlapped with [2]. As a baseline, [2] can be used to analyze the performance of $\Delta VAE$. Also, the authors need to emphasize the novelty of their method by comparing it with [2].

=====================================================================================================

Minor :

The word "limited supervision" is quite confusing. Instead, I suggest "weakly-supervised". 

======================================================================================================

After rebuttal :

Thank you for the responses. I'm not sure why the authors didn't perform the experiments on the correlation between the previous factor-based disentanglement scores and the proposed disentanglement score in the limited supervised setting. For example, if there are 5 factors, I propose to evaluate the proposed disentanglement score for every possible pair of the factors (10 pairs) and average the scores. I believe this paper handles the valuable topic but it is not enough to be accepted since the experiments, which are crucial I believe, are omitted (comparison with the other disentanglement score, baselines to $Delta$VAE, comparison with [2]). Also, I concerned that other readers might be confused with the ("factor-based" disentanglement and "symmetric-based" disentanglement ) and (limited-supervision and weakly-supervision) (a new section should be added to handle these topics if this paper should be accepted). Furthermore, discussion with the related works is not enough.


I lower my confidence rate to 3 (5->3) and vote for weak reject (3->4). But, I hope this paper would be accepted after revisions in the future.

============================================================================================================

References 

[1] beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, ICLR 2017.

[2] Weakly-Supervised Disentanglement Without Compromises, ICML2020.
review_title: Official Review 4

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper aims to operationalize the symmetry-based disentanglement idea proposed in Higgins et al., 2018, and proposes a novel loss function that can be used to both evaluate and learn disentangled representations (including learning from datasets where only a subset of samples are fully labeled). The paper includes some experimental validation of the proposed ideas on synthetic datasets. 

Overall, the paper is clearly written and makes a positive contribution in pushing for disentangling metrics that are motivated by better formalism than what is commonly used. The paper would be stronger if it included rigorous comparisons to state of the art disentangling metrics and methods, and a discussion of how the requirement of full supervision of all factors of variation for at least a portion of the dataset can be relaxed.

Weaknesses:
* The supervised subset mentioned in section 6 must have full labels for all factors/groups. If even one factor is missing, then this method cannot be applied, since we do not know the full set of transformations that relate different data points to each other. It is not clear to me how this assumption can be relaxed with the proposed metric/objective: if we lack labels for only a single factor, then it is no longer possible to explain the dataset in terms of actions on a single input sample.
* The above also means that experiments can only be done on synthetic datasets, where all factors are known.
* This paper would be stronger if it (a) showed a comparison of the LSBD metric to commonly used metrics for disentangling, such as those compared in Locatello et al., 2018. and (b) compared the proposed model to other disentangling models, both supervised and unsupervised. An implicit comparison to diffusion VAE can be found in the results in Figure 4, when the number of labeled pairs is zero, but a more thorough comparison is required to understand the value of the model and metric.

Strengths:
* Novel method for evaluating and learning disentangled representations.
* Can leverage unlabeled data.

Clarity
* In the experiments, the g’s are given by the data. But how are the \rho functions implemented? Are they specified in advance?
Section 4, bullet point 3: “each observation is a transformed version of another observation, and that transformation is unique”. This seems like a fairly strong statement that is often violated: for example, rotating and flipping a square can easily map to the same X. It would be useful to discuss:
- how does the metric/model behave when these assumptions are violated in a domain?
- how does the loss react to subspaces that are scaled differently, or subspaces of differing dimensionality? In the experiments I believe all the subspaces are the  same size, but if one was in R^2 and another in R^100, the squared-distances would be dominated by the latter and the loss may not work as well.
review_title: Needs comparisons with other disentangling metrics/methods, and explanations for how the method can scale to realistic datasets

Meta-review: This paper proposes a new metric to measure symmetry-based disentanglement and uses this metric to optimize diffusion VAEs on a set of small, synthetic datasets. In general, reviewers found the theoretical framework introduced to be interesting and relevant, but there were a number of concerns regarding the empirical evaluation in the paper and the clarity of many of the claims, particularly wrt the need for strong supervision (pairs of data points with a known transformation between them) for both evaluating the metric and for training by regularizing the proposed metric. I'd encourage the authors to focus on the improvement points suggested by reviewers, most notably by improving the empirical evaluation by adding detailed ablations and comparisons (e.g., exploring the relative amount of supervision needed, comparisons to previous approaches) and clarity regarding the supervision required. As such, I recommend that it be rejected in its current form.  
Decision: Reject