Quantifying and Learning Disentangled Representations with Limited Supervision
Authors: Loek Tonnaer
Luis Armando Pérez Rey
Vlado Menkovski
Mike Holenderski
Jacobus W. Portegies
Venue:  (2021)
Keywords: Representation Learning
Disentanglement
Group Theory

Abstract: Learning low-dimensional representations that disentangle the underlying factors of variation in data has been posited as an important step towards interpretable machine learning with good generalization. To address the fact that there is no consensus on what disentanglement entails, Higgins et al. (2018) propose a formal definition for Linear Symmetry-Based Disentanglement, or LSBD, arguing that underlying real-world transformations give exploitable structure to data.

Although several works focus on learning LSBD representations, such methods require supervision on the underlying transformations for the entire dataset, and cannot deal with unlabeled data. Moreover, none of these works provide a metric to quantify LSBD.

We propose a metric to quantify LSBD representations that is easy to compute under certain well-defined assumptions. Furthermore, we present a method that can leverage unlabeled data, such that LSBD representations can be learned with limited supervision on transformations. Using our LSBD metric, our results show that limited supervision is indeed sufficient to learn LSBD representations.

Disentangled (SBD) Under review as a conference paper at ICLR 2021
QUANTIFYING AND LEARNING DISENTANGLED
REPRESENTATIONS WITH LIMITED SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
Learning low-dimensional representations that disentangle the underlying factors of variation in data has been posited as an important step towards interpretable machine learning with good generalization. To address the fact that there no consensus what disentanglement entails, Higginn et al. (2018) propose formal definition for Linear Symmetry-Basee Disentanglement, or LSBD, arguing that underlying real-world transformations give exploitable structure data.
Although several works focus learning LSBD representations, such methods require supervision on the underlying transformations for the entire dataset, and cannot deal with unlabeled data. Moreover, none of these works provide a metric to quantify LSBD.
We propose metric to quantify LSBD representations that is easy to compute under certain well-defined assumptions. Furthermore, we present method that can leverage unlabeled data, such that LSBD representations can be learned with limited supervision on transformations. Using our LSBD metrie, our results show that limited supervision is indeed sufficient learn LSBD representations.
INTRODUCTION
Disentangled representation learning aims to create low-dimensional representations of data that separate the underlying explanatory factors of variation in data. These representations provide an interpretable (Sarhan et al. (209)) and useful tool for various purposes, such as noise removal (Lopez et al. 2018), continuous learning (Achille et al., 2018), and visual reasoning (van Steenkiste et al.
2019
However, there is no consensus about the exact properties that characterize a disentangled represen- tation. Higgins al. (2018) provide formal definition for Symmetry-Baan and Linearly SBD (LSBD) data representations, building upon the idea that representations should reflect the underlying structure of the data. In particular, they argue that variability in the data comes from transformations in the real world from which the data is observed. Having a formal definition of disentanglement can serve as paradigm for the evaluation of disentangled representations.
Although several methods have been proposed to learn SBD or LSBD representations, none of them provide clear metric for quantifying the level of disentanglement in these representations, Quessard et al. 2020) introduce a loss term that measures the complexity of the transformations acting on their learned representations based on the number of parameters needed, but this term does not directly characterize disentanglement. [Caselle- Dupré et al.) (2019) only evaluate the performance of their learned representations when used particular downstream task.
Moreover, existing methods require information about the transformation relationships among data points for the entire training dataset. This information is used to produce models that enforce the properties of SBD or LSBD representations and can be considered example, this supervision can consist of the parameters the transformation that connects pair of data points, such as a rotation angle. Obtaining this supervision the transformations for dataset can be an expensive task that requires expert knowledge.
form of supervision. For
In this work focus characterizing and quantifying LSBD and developing method capable of obtaining LSBD representations by using a limited amount of supervision on the transformation properties of dataset. The main contributions of this paper are:

---

Under review as a conference paper at ICLR 2021
1. An co-mppute metric to quantify LSBD given certain assumptions (see Section which acts as an upper bound to a more general metric (derived in Appendix D).
2. A partially supervised method obtain LSBD representations that, during training, can also use data without supervision on the transformation relationships.
2 RELATED WORK
The concept of disentanglement comes from the intuition that data can be described in terms of set of independent explanatory factors that constitute the variability of the data. In probabilistic model- ing, these factors are interpreted random independent unobserved latent variables that condition the data generation process (Kulkarni et al.) 2015; Higgins et al.) 2016, Chen al.] [016 2018). Nevertheless, there is no consensus about the exact properties that disentangled representations should have. Evidence of this is the wide range of metrics used to characterize such representa- tions (Locatello et al., (0118). However, a reasonable expectation is that such representations should capture and separate the variations and properties of the data.
Recent work has turned the attention to capturing not only the independent explanatory factors of dataset, but also the transformations that those factors undergo in data generation. Examples of methods that attempt to produce representations with similar transformation properties to those of the explanatory factors of data (Cohen & Welling 2015 Worrall al. 2017 Sosnovik al. 2019).
Transformations of the real world determine the variability of data and its structure. These called symmetry transformations have long been studied in Physics (Gross) 1996) and have been formalized with group theory. Such transformations often affect only a subset of the properties that describe the real world and leave the rest invariant.
The connection between disentanglement and symmetries has recently been formalized by Higgins et al. (2018) into the definitions of Symmetry- Disentangled (SBD) and Linearly SBD (LSBD) representations. Quessard al. (2000); Caselles- Dupré et al. (2019) propose methods to obtain such SBD and/or LSBB) representations, but their methods require supervision on the transformation relationships among datapoints for the entire training dataset. Moreover, there no clear metric for the level of disentanglement in their methods.
3 SYMMETRY-BASER DISENTANGLEMENT
Higgins et al. (2018) provide a formal description of disentanglement that connects the symmetry transformations affecting the real world (from which data is generated) to the internal representa- tions of a model. The definitions are grounded in concepts from group theory, for a more detailed description of these concepts please refer Appendix
These definitions assume the following setting. Wi is the set of possible world states, with underlying symmetry transformations that are described by group G and its action G W W W. K groups G = G x x Gk- Data obtained via an observation function W + X that maps world states to observations in a data space X. A model's internal representation of data is modeled with the inference function + Z that maps data to the representation space Z. Together, the observation and the inference constitute the model's 's internal representation of the real world f W + Z with fww = h b(w).
In particular, G can be decomposed as the direct product
The definitions for Symmetry-Baan -Based Disentangled (SBD) and Linearly SBD (LSBD) representations formalize the requirement that model's internal representation f W - Z should reflect and disentangle the transformation properties of the real world. The definition of SBD can be found in Appendix B In particular, our work focuses on LSBD representations in which the transformation properties of the model's internal representations should linear. The exact definition is as follows:
Linearly Symmetry-Based Disentangled (LSBD) Representations A model's internal represen- tation f + Z where Z is a vector space, is LSBD with respect to the group decomposition
G G1 ... x Gk Kgft
2

---

Under review as a conference paper at ICLR 2021
there is decomposition of the representation space Z 21 ... Zx into K vector subspaces,
there are group representations for each subgroup in the corresponding vector subspace
G2 + (Z()z), k E (I.....K] K}
the group representation p G - GZ( for G in Z acts on Z as p(g)
(g)) 21: ................................ PK (gk) =x),
for g (..,...,, ..)) E and : = (z1, ...... =k) with gk E Gg and 24 E Zg.
the map f is equivariant with respect to the actions of G on W and Z, i.e. for all E W and E G it holds that f(g w) = p(g) f(w).
Symmetry-based disentanglement with respect to the data space X We do not directly observe W. only data observations in X, thus it is most practical to evaluate (L)SBD with respect to the inference map h : X + Z. However, the definitions for SBD and LSBD representations are given with respect to the world states representation map ʃ W Z. If the action of G on W is regular and b injective though, a representation f is (L)SBD the inference restricted to all possible observations is also (L)SBD. In this case, we can evaluate the same conditions for (L)SBD only for the function h. For more detailed description on these conditions consult Appendix
ASSUMPTIONS
In this paper we assume setting where an underlying group structure and its action on the data space and representation space are provided expert knowledge, which allows to provide metric that quantifies Linear Symmetry-Based Disentanglement (LSBD) and a method to achieve it. Although these assumptions limit the settings in which our disentanglement metric and method can be used, we believe
valuable because (1) provides a good basis for future research into more general metries and methods by making clear which assumptions should be relaxed,
(2) it gives good insights into the effect of the level of supervision on transformations (as shown by our results in Section 7). and (3) there are many practical scenarios where having this type of expert knowledge a reasonable assumption.
More concretely, we make the following assumptions about the available information about the data and its transformations:
1. Let W be the set of real world states. Let X be the data space, where observations are obtained. Let Z be the representation space where we wish to model this data; Z is vector space with given basis.
2. We have Lie group G and regular action of G W that describes the underlying sym- metries of W. The group decomposes as the direct product of groups G = G₁ x x... x Gx-
3. The observation function b W + X injective. Therefore, we can define group action for G on the image of b, i.e. G x b(W) - b(W) as g = b(g-bb-(a)) b-r(()) (see Appendix C for more details). This action also regular. Practically, this implies that each observation is transformed version of another observation, and this transformation is unique.
4. We have an action : G x Z + Z that is linear and has known representation p G GZ) i.e. for any g E G we can determine p(g).
5. The representation p G GL(Z) is linearly disentangled, i.e. there is known decom- position 2 Z1 + @ Zy such that p(g) 2 = (p1(91) 21...., (gK) zx), with Ok Gk, = Z, for k- 1....,K K.
6. There exists embedded submanifold ZG C 2 such that the action of G on Z restricted to ZG is regular, and ZG is invariant under the action. Only ZG will used as the codomain of the inference map, h X - 7c-
the special orthogonal group S0(2) of 2D rotations Let's look at an example that satisfies the assumptions above and reasonably occur in practice. Consider a windmill that can be rotated around a fixed vertical axis, with blades that can rotate independently of the windmill's orientation. The dataset consists of observations (i.e. 2D images) of this windmill, where the only
Example:
3

---

Under review as a conference paper at ICLR 2021
variability is in the rotations of the windmill and of its blades, all other aspects (e.g. camera angle, lighting conditions, object position, etc.) are fixed. Thus, the variability in this dataset can be described as two independent 2D rotations, see Figure
(
2(1)
(2)
(g. c
[g. e)
(1)
(2)
((g)
201)
(e.g)
(e.g) g)
Figure 1: Illustration of LSBD) representation with respect to the decomposition G= (0)) x SO(2) for the windmill example, 3D model by (Poly). The action of G on the real world is decomposed as independent transformations: the rotation of the windmill with respect to vertical axis and the rotation of the blades. The inference function X - Z maps images into R + R On Z. the action of G the independent rotation of each vector subspace. The action of G on Zg S' x S C Z is regular.
We satisfy the assumptions above as follows:
1. The world states W can be described by the angle the entire windmill and the angle of the blades, these angles are independent of each other. The data space X is the space of images, i.e. arrays of pixel values. A suitable representation space Z will become apparent soon.
2. It is well-knows that 2D rotations can be described by the special orthogonal group SO(2), the transformations underlying this dataset can be described with the group decomposi- tion G = G₁ x G2 = SO( x SO()). The corresponding action is indeed regular; any world states connected with unique transformation (i.e. pair of rotations)
3. If the windmill looks different from each angle, and the blades are always visible, then the observation function b: W X is indeed injective.
4. Each subgroup G SO(2) has representation Pk SO(2) GL(R²) as rotation matrices for vectors in I=2 so we can use Z = Z1 @ Ze = IR2 + R² as the representation space, and the direct sum p = P1 P2 the group representation.
5. Defined in this way, the representation is linearly disentangled.
6. The actions of the subgroups G; on the subspaces Z, are not regular, but they become regular when restricted the unit circle or 1-sphere 5 = {= E R² ||||| = =1]. which is submanifold of Z, 2 and invariant under the action. Thus, as the codomain of the inference map h, to obtain regular group action of on Zc.
ZG S S
Note that for windmill with e.g. identical blades, rotation of radians corresponds to the identity transformation, and angles he defined mod g.
4

---

Under review as a conference paper at ICLR 2021
5 QUANTIFYING LINEAR SYMMETRY-BASED DISENTANGLEMENT
In this section we provide metric that is easy to compute and characterizes LSBD under the as- sumptions of Section 4 by quantifying the equivariance of the inference map with respect the group action on X and Z. Moreover, in Appendix D we show that this metric is upper bound to a more general LSBD metric that can theory be evaluated for any encoder.
Characterizing LSBD representations The assumptions from Section f ensure that all criteria of the LSBD definition (see Section 3 are satisfied, except for equivariance of the inference map h: X + Zg with respect to the action of on X and Z. Thus, to evaluate whether h is disentan- gled, only need to check if h(g x) g h(x) holds for all X and g G.
Since the action of G on X is regular (see Section 4) the transformations in dataset X = {xn N (with 2.n X) can be fully described using only N - transformations. In particular, is convenient to describe all other data points in terms the first one: X' = {gn. I1 J.V L where 9I = e is the group identity. Note that {gn J can be used to describe the transformation between any two points Ty E X (assuming access to the inverse transformations as well), since 2i 9i (9) x.y)
n=1
Equivariance can therefore be evaluated for X' using only N comparisons, by testing whether hgn, x11) p(gn) h(x1) for 2, ...., N.. To simplify the notation, write the data represen- tations as 2n = h(x,) for n = i,...,,,................. N. Some rewriting shows that we can evaluate equivariance by checking whether
21 -) -... p( AN-
This formulation not only gives an efficient way checking perfect equivariance (and thus LSBD), but also allows for an efficient way to quantify the divergence from this ideal situation, as we will show next.
Quantifying LSBD Equation (2) shows how to check if learned data representations (zn L.W are LSBD, given the transformations ] but it also provides perspective to quantify how well the representations are disentangled if perfect disentanglement is not achieved. The transformed representations (p(g.')) zn J should ideally all be the same, so their dispersion is reasonable measure to quantify equivariance (where no dispersion means perfect equivariance).
An efficient way measure dispersion is by computing the sample variance. For sample {a, } in Euclidean space, this is computed as x Ew= ||a,, - all. where a = 1 E an the sample mean. In our work however, representations are modeled in a submanifold ZG that itself is typically not Euclidean. But by defining a metric d on this manifold, we can generalize the sample mean with the Fréchet mean a = mean = arg min Ex d2(an, a), and then compute the Fréchet variance + > dc (a(,,, a).
Therefore, we propose the following metric (lower is better) to quantify equivariance, and thus LSBD (under the assumptions of Section 4p. for set of representations {2n]n=1 S Zg with in- between transformations {gn]
LLSBD Σ d (p((gn)) -n. -2.,)) Z),
(3)
with 2 : mean ==1
mean(p(9n 2n);
N
(4)
for metric d defined on the manifold Zc. In particular, d should be preserved under the group representation for this metric to be sensible.
In the example from Section 4 we saw ZG x In particular, each circle S (a R² | 1} is a submanifold of Euclidean space. We choose Euclidean distance as the metric d, since is well-defined on the manifold and preserved under the group representation p. Moreover, for any two 2, E 5!, the squared distance between them can be efficiently computed as
d((2 ||2 21122 = 2T 2T
(5)
5

---

Under review as a conference paper at ICLR 2021
To compute the squared distance between points in S' x S' we simply sum up the squared distances per submanifold S With this metric, the Fréchet mean has simple solution
クm=1 Zn
mean(zn) (2n) n=1
(6)
11
>====
METHODOL.OGY
We present a method to learn LSBD representations (as defined in Section given some transforma- tion labels, under the assumptions in Section 4 We emphasize that there are two main components that lead to LSBD representations:
1. a suitable topology for the representation space Zc. to be able to model the (linear) action of the underlying symmetry group correctly, and
2. some supervision transformations to ensure equivariance w.r.t. the predefined group representation.
Our method consists of an unsupervised and a supervised part, such that we can learn from unlabeled data as well as from data with additional supervision on transformations. First we formalize what we mean by transformation labels, then we outline the unsupervised and supervised parts of our method.
Method dataset: observations with limited transformation labels For our method we assume access to disjoint datasets X' and y of observations in the data space X, where we have addi- tional information on the transformations between elements in y. More precisely, we assume that Y consists of L batches of M data points each, where all data points in batch can be expressed with known group element acting on the first member of the batch. Formally, this dataset can be described as
y {[ Pim
9im Fn 1==1)= 1/
where the first group transformation for each batch corresponds to the identity 911 beled dataset consists N M L data points with no information about the transformations among the data points.
The unla-
Unsupervised learning: VAE with suitable topology To model data representations for the un- labelled dataset X' in a submanifold ZG of Euclidean space, we use a Diffusion Variational Au- toencoder (A VAE) (Perez Rey et al. 2020). This is Variational Autoencoder (Kingma & Welling 2014 Rezende et al. 2014) that is capable of encoding data into a closed Riemannian latent space.
AVAEs use any closed Riemannian manifold embedded in Euclidean space latent space (or latent manifold), provided that a certain projection function from the Euclidean embedding space into the latent manifold is known and the scalar curvature of the manifold is available. The VAE uses parametric family of posterior approximates obtained from a diffusion process over the latent manifold. To estimate the intractable terms of the negative ELBO, the reparameterization trick is implemented via random walk.
In the case of S1 as latent (sub)manifold space, and the projection function II R22 -> S normalizes points in the embedding space: (T(2) = The scalar curvature of S1 is 0. It straightforward extend these properties latent space S1
consider the Euclidean space R22 as the embedding
Supervision: enforcing equivariance with transformation labels Caselles G et al. (2019)
proved that (L)SBD representations cannot be inferred from training set of unlabeled observations, but that access to the transformations between data points needed. They therefore use a training set of observation pairs with given transformation between them.
NNte that this function is not defined if ΣN 2n. = 0, but this isn't an issue in practice. "This projection function not defined for hut this value does in practice.

---

Under review as a conference paper at ICLR 2021
However, posit that only limited amount of supervision is sufficient, if we have access larger dataset unlabeled observations as well. Since obtaining supervision on transformations typically more expensive than obtaining unsupervised observations, it is desirable to limit the amount of supervision needed.
Therefore, we augment the unsupervised A VAE with supervised method that makes of the transformation-labelee batches from V. This method makes use of the same principles from Sec- tion [] but on batch level instead of full dataset. By alternating the unsupervised and supervised training phases, we have a method that makes use of both unlabeled and transformation-labee observations.
eicoderr
ø1
(g:) ) 2
decodar
2/2
oncodar
Z2
p(92)
deonder
Aoz
mean ment
(p(gm)) am)
2
entoder
2M
pgu) 2
doooder
Prx
LLSBD
CKL
meean
RE
Figure 2: Overview of the computation flow of the supervised training method for a batch IM
{Im Jm=1 {gm 21 JMM. (where 91 = e).
Figure 2 illustrates the supervised part of our method for a given transformation-lablle batch {x1, ................................ Far} {-1, 21:....9M:
from y The supervised part of the method uses the same encoder and decoder networks from the unsupervised part, but an extra processing step added between encoding and decoding the full batch, which encourages equivariance given transformation-label.. The procedure a forward pass is as follows:
the
Compute latent variables (zı....: )M by sampling from the posteriors g(=(=m) (i.e. regular forward pass through the encoder and sampling layer of AVAE).
2. Compute M
2m).
3. Compute p(gm)= z and the parameters of p(z) z) for (i.e. a regular forward pass of p(gm) through the AVAE decoder).
We train the full network to optimize the following loss function:
Li
RE + LKL + LLSBD
(8)
logp(smll(9..) 2)
Σ ([((()))))))( +y Σ d (zm: p(gm) z).
Here LRE + LKL is essentially the regular ELBO used to train AVEE, but instead of single sample from q(2 fm) to compute CRE, we used the "corrected" sample p(gm) Z. This encourages the decoder follow the required group structure.
EXPERIMENTS & RESULTS
We designed experiments datasets with known group decomposition test the following:
1. Study LSBD representations obtained with our methodology.
To avoid notational cluttering omit the index describing the hatch to which each data point belongs.

---

Under review as a conference paper at ICLR 2021
>
7
>
>
7
>
7
(a) Square Translation
(b) Arrow Rotation
(c) ModelNet40 Airplane
Figure 3: Example images from each of the datasets used. Each image corresponds example data point for combination of two factors, e.g. color and orientation, The factors change horizon- tally and vertically and the boundaries of each dataset example are periodic.
2. Evaluate the stability and the behavior of the method using our LSBD metric from Section for a range of numbers labeled pairs L across several training repetitions.
All datasets contain 64 pixel images, with a known group decomposition G SO x S0(2) describing the underlying transformations. For each subgroup fixed number of 64 transformations selected. Each image is generated from single initial data point upon which all possible group actions are applied, resulting in datasets with N = 4096 images. The datasets exemplify differ- ent group actions of 50(2): periodic translations, in-plane rotations, out-of-plane rotations, and periodic shifts, see Figure 3 For more details, see Appendix
For the supervised datasets we use pairs (i.e, batches of size M = 2) of transformation-labeee data points. From dataset of size N, randomly sample L < N/2 batches of unique data points without replacement. Each pair is labeled with description the transformation between the two data points in the pair.
We choose M = for our experiments since it is the most limited setting for our supervised method. Higher values of M would provide stronger supervision, so successful results with M 2 imply that good results can also be achieved for higher values of M (but not necessarily vice versa).
In our experiments, we use a fixed weight 7 100 and vary the number of batches L. We train the AAE for 300 epochs per experiment using the Adam optimizer, each epoch consists of a full unsupervised and supervised phase. For each experiment 10 repetitions were trained and evaluated by computing the LSBD metric across the training dataset. Details of the A AVA architecture can be found in Appendix
Square
Rotation
ModelNet40 Airplane
2.0
2.0
2.0
1.5
1.5
1.5
Cisao
L1000
Li000
1.0
1.0
0.0
0.0
0.0
256
512
768
1024
12280
1536
2048
256
512
768
1024
1280
1536
1791
7048
256
512
768
110024
1280
1536
1792
2048
Number of Labeled
Number of Labeled Pairs
Number of Labeled Pairs
Figure 4: Square Translation, Arrow Rotation and Modelnet40 Airplane datasets' box plots for the
LSBD metric evaluated over 10 training repetitions for different numbers of labeled pairs L and 7 100.
The results obtained for the three datasets presented in Figure 4 using the LSBD metric from Section 5 (lower is better). The results show that for all three datasets, only a limited amount of supervision is needed to obtain LSBD representations consistently multiple runs. In particular,
8

---

Under review as a conference paper at ICLR 2021
observe that by providing transformation labels for disjoint pairs of data points only, model can learn a more general description of the underlying transformations that connect the entire dataset.
8 CONCLUSION & FUTURE WORK
In this work, specified some well-defined assumptions that allow to define easy-to-compute metric to quantify Linearly Symmetry-Basee Disentanglement (LSBD), as defined by Higgins et al.) 20188. Moreover, presented method obtain such LSBD representations, making of unlabeled observations as well as a limited amount of supervision on transformations.
Using our LSBD metric, we showed experimentally that method indeed learn LSBD repre- sentations, using only limited amount of supervision on transformations, by making use of unla- beled observations well.
Our LSBD metric and method require number of assumptions, as explained in Section 4 This limits the applicability of the metric and method, but also provides a clear direction towards more general approach; by studying how we can relax these well-formal rell-formalize assumptions.
Moreover, our metric is in fact an upper bound to a more general metric (see Appendix D which is however less straightforward to compute. Characterizing how to compute this metric in various situations will allow for better comparison with other methods aiming to obtain LSBD representa- tions.
REFERENCES
Alessandro Achille, Tom Eceles, Loic Matthey, Christopher Burgess, Nick Watters, Alexander Lerchner, and Irina Higgins. Life-long disentangled representation learning with cross-domain latent homologies. In Advances in Neural Information Processing Systems, pp. 9873-997388 2018.
Hugo Caselles-Dupre Michael Gareia Ortiz, and David Filliat. Symmetry- disentangled rep- resentation learning requires interaction with environments. In Advances in Neural Information Processing Systems, pp. 4606-4615, 2019.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2615-2625, 2018
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp- 2172-2180 2016.
Taco S. Cohen and Max Welling. Transformation properties of learned visual representations. In 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Pro- ceedings, number 3, pp. 1-11, 2015.
Blender Online Community. Blender - a 3D modelling and rendering package Blender Foundation, Stichting Blender Foundation, Amsterdam, 2020.
www. b ender. org
D. J. Gross. The role of symmetry in fundamental physics. Proceedings of the National Academy of Sciences of the United States of America, 93(25):14256-1422) 1996. ISSN 00278424. doi: 10.1073/pnas 93.25.14255.555
Brian C Hall Lie Groups, Lie Algebras, and Representations, volume 222 of Graduate Texts in Mathematics. Springer International Publishing, Cham, 2015. ISBN 978-3-319-13466- URL hhtt:://lin springeriieeeee com/ wwtp:paigg.........10.00.0
6. doi:
978-3-319-13477
Irina Higgins, Loie Matthey, Xavier Glorot, Arka Pal, Benigno Uria, Charles Blundell, Shakir Mo- hamed, and Alexander Lerchner. Early Visual Concept Learning with Unsupervised Deep Learn ing. 2016. hRI httptttpp///iiittoo/o/t/tt
05579

---

Under review as a conference paper at ICLR 2021
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerehner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.022300 2018.
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Confer- ence on Leurning Representations (ICLR), 2014. ISBN 1312.61 14v10. doi: 10.1051000000 1051/00004-666// 201527329.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in neural information processing systems, pp. 2539- 2547, 2015.
Francesco Locatello, Stefan Bauer, Mario Lucie, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled repre- sentations. arXiv preprint arXiv:1811.12359, 2018.
Romain Lopez, Jeffrey Regier, Michael Jordan, and Nir Yosef. Information constraints on auto- encoding variational Bayes. Advances in Neural Information Processing Systems, 2018-Decee (1):6114-6125 2018. ISSN 10495258.
Luis A. Perez Rey. Vlado Menkovski, and Jim Portegies. Diffusion variational autoencoders. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI- 20, pp. 2704 2710, 2020. doi: 00224963/ijccii22220373. URL https://doi..o. 24963/ 1 caa .2000/375
Poly. Windmill. http:://poo...oogll..cc//////// ber 2020.
IM3xXu Accessed 25th Octo-
Robin Quessard, Thomas D. Barrett, and William R. Clements. Learning Group Structure and Disentangled Representations of Dynamical Environments. pp. 1-10, 2020. URL http://
wrxiv. wwt.w//rggbbbb/....oooo/o 06991
arxiv
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31 International Con- ference on International Conference on Machine Learning Volume 32, pp. II-1278, 2014.
Mhd Hasan Sarhan, Abouzar Eslami, Nassir Navab, and Shadi Albarqouni. Learning Interpretable Disentangled Representations Using Adversarial VAEs. volume 4, pp. 37-44. Springer Interna- )5.
tional Publishing, 2019. ISBN 978-3-030-33399-1, doi: 1111007/978-3000-33333-
N. Sedaghat and T. Brox. Unsupervised generation of a viewpoint annotated car dataset from videos.
In IEEE International Conference on Computer Vision (ICCV), 2015.
Stefano Soatto. Steps Towards Theory of Visual Information: Active Perception, Signal-toSyybbl Conversion and the Interplay Between Sensing and Control. 2011. att/ttp: htt/atttp//axxx.ortt /rrx
abs bbss/1110 2053
Ivan Sosnovik, Michal Szmaja, and Arnold Smeulders. Scale- Buuiv-riui Steerable Networks. Inter- national Conference on Learning Represeetatiinss, ICLR 2020 Conference Track Proceedings, pp. 1-14, 2019. URL http:://aaaxiiiii wttp://wwwppp//////////////// wts/19// 11093
Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? In Advances in Neural Information Processing Systems, pp. 14245-14258, 2019.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhanbetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3D ShapeNets: A Deep Representation for Volumetric Shapes. 2014. ISSN 10636919, doi: 10. 10.1/09/ 2015 2.2988808 URL. Utt.:ttppp//aaii..oo//b///////
10

---

Under review as a conference paper at ICLR 2021
APPENDIX
A PRELIMINARIES: GROUP THEORY
In this appendix, we summarize some concepts from group theory that are important to understand the main text of the paper. Group theory provides useful language to formalize the notion of symmetry transformations and their effects. For a more elaborate discussion we refer the reader to the book from Hall (015) on group theory.
Group A group is non-empty set G together with a binary operation o GG G - G that satisfies three properties:
1. Associativity: For all f,9, h G. it holds that f (g h) = (f g) o h.
2. Identity: There exists unique element E G such that for all g E G it holds that e o
g e = g.
3. Inverse: For all g E there exists an element g e G such that g = =gog--
Direct product Let G and G' be two groups. The direct product, denoted by G G', is the group
with elements (g.g') E G x G' with E G and g' E G", and the binary operation o : G G' G G" such that (g.g') ) (h, h²) (g o h ht).
Lie group A Lie group is a group where G is a smooth manifold, this means it can be described in a local scale with set of continuous parameters and that one can interpolate continuously between elements of G.
Group action Let be set and G a group. The group action of G on is function GA: G A + A that has the properties
1. GA(e,z) = x for all E A
2. GAlg, (Galg Galg o , a) for all g. E G and E A
Regular action The action of G on A is regular if for every pair of elements o, a' E A there exists unique g E G such that g a'.
Group representation A group representation of G in the vector space V is function p: G + GL(V) (where GL(V) is the general linear group on V) such that for all g, E p(g o ogg' p(g) p(g") and p(e) = ll, where lly is the identity matrix.
The direct sum of two representations G G GL(() in V and P2 G + GL(V") in V" is group representation #1 p2 GL(V e V') over the direct sum V + V". defined for V and r'
Direct sum of representations
E V
as:
(er 2 (2) (v, (p)(g) v. (2(g)
(10)
B SYMMETRY-BASED DISENTANGLEMENT DEFINITION
Given the assumptions stated in Section 3 the definition for Himmetry-Based Disentangled (SBD) representations introduced by (Higgins al. 2018 as follows:
Symmetry- Baased Disentangled (SBD) Representations Suppose that the group G is decom- posed as the direct product of groups, = G, x x... x Gx which models the transformations of the world states via an action G W W on W. A representation f is SBD representation with respect to = G x ... Gk
To avoid notational clutter, we write Ga(g. a) g where the set A on which G acts can be inferred from the context.
11

---

Under review as a conference paper at ICLR 2021
there is action :G -
the map / W - Z is equivariant with respect the actions of G on W and Z, i.e. ww E W and E G it holds that [[g w) = f(w), and
there is a decomposition Z = 21 x x Zx such that each Z, is invariant to the action of all but one subgroup. This means that for 2, E Z4 g 2 = 2, for g E Gj j # and is only affected by Gi.
C SYMMETRY- HAMETRY-BASED DISENTANGLEMENT IN THE DATA SPACE
is injective, then we can define an action of on the possible observations b(W) X. The injectivity of assures unique observations for different world states (in cases where b is not injec- tive, it is possible to address this through active sensing (Soatto 2011), so assuming injectivity is reasonable in practice). If b is injective then restricted its image it is bijective and the action : x 6( (W) - b(W) of G on b(W) as I = b(g (x)).
define
If the action of G on W is regular then is equivariant with respect to the actions of G on W and B(W). In such case, if LSBD with respect to the decomposition of G for the available data then is also LSBD. This means that in such setting only need to focus the disentanglement of the inference map h, as this will assure the disentanglement of the model's representation / of the world.
D GENERAL METRIC
In this appendix, we show how the disentanglement metric LLSBD in the main text (with a specific choice for the metric d) is an upper bound for a more general LSBD metric MLSBD. The more general metric can be defined for any encoding map from data space X to latent space Z and data probability measure μ on X satisfies the following condition. We assume the data (probability) measure y equals the pushforward Gx(,, x1) of probability measure on G by the function (1,,1)), i.e.
=
= (Hg E G Gx(g. A))
for Borel subsets A C X. Here Gx denotes the action of G on X and æ1 base point. The situation of discrete dataset {.,, Jo {g: .1 ) corresponds to the case in which and μ following empirical measures on the group G and data space X respectively
gg
We define the more general LSBD metric MLSBD follows
MLSBD =
p -(gg (1) p(g))-1.1111 hhl' x 21)dv(g')
(11)
inf PEP(G.2)
dv(g)
p.h.
where the norm || Ill. is Hilbert-space norm depending the representation the encoding map h : X + Z, and the data measure μ. Moreover, P(G, Z) denotes the set of linear representa- tions of G in Z. From now on, will denote the norm as || ||-.
To describe the norm || Assume that P splits in irreducible representations Ps G - Z, where Z Zi @ ..... Z₁ for some E N. We will define new inner product +). on as follows. First all we declare Z, and Z to be orthogonal with respect to () if o We denote by n; the orthogonal projection on Z,-
|| we start with an arbitrary inner product (.,-) on the linear latent space Z.
For v. w E Zg, we set
(v, w) = x-1 /
9 (p(g) v, v.plg) w) w)dm(g)
(12)
12

---

Under review as a conference paper at ICLR 2021
where is the (bi-invariant) Haar measure normalized such that m(G) and set
A = / Ja [=x((())))
(13)
the integral on the right-hand side is strictly positive and otherwise we set A == This construc- tion completely specifies the new inner product, and it has the following properties:
the subspaces Z, are mutually orthogonal
pi(g) is orthogonal on Z, for every g G, in other words ps maps the orthogonal group on Zz. Moreover, maps to the orthogonal group on Z. This follows directly from the bi-invariance of the Haar measure and the definition of (+:-)..
If is the orthogonal projection to Zi then
J Im (m(((()))))d(()))
(14)
if the integral on the left is strictly positive.
We now explain why LLSBD is upper bound for MLSBD, in case Z in the main text is endowed with the norm || I].. Note also that in the example with SO(2) rotations in the main text, the norm || Il is just the ordinary Euclidean norm.
First all, because the infimum in the definition of MLSBD. for any representation considered in the main text we find
MLSBD S f( pl0) hig 21) p(g') h(g x1 21) s))dv())) dv(g)
(15)
Finally, it is a property of the mean that for every E Z the right-hand side is smaller than
J. [1060) hlg 21) cll. dv(g).
(16)
Written out for discrete dataset, letting =n M(r)), and taking the mean from Equation 6
c: mean(zn) =:
(17)
n=1
we find that
N
MLSRD S -(la(a))-1
. - =| LLSAD
(18)
n=1
We will now give an alternative expression for the disentanglement metrie MLSBD, since will more visibly relate to the definition of equivariance. Let E P(G, Z) be a linear representation of G in Z. By expanding the inner product (or by using usual computation rules for expectations and variances), we first find that
1. p()) p()))- 1000-hhhg---0
h(g-21) - p(g')= .hlg =i)de(l') (o(
dv(s)
( (1o(0)- 1. -hgg x1 m1)l1 du(g) - -111 p(g)) hig (lg-a))du(g)) modrrel
(19)
1 |
2 (1((g)- hig x1) - p(g')=1 h(g x ( (odd)))du))))
We now use that p maps to the orthogonal group for (:: so that we can write the same expression as
h((gg MM(oog g') .1) hig' =))ledul)))u())
(20)
This brings us to the alternative characterization of MLSBD as
MLSBD= = inf pEP(G.2) S
h(g
(21)
13

---

Under review as a conference paper at ICLR 2021
In particular, if for every data point there is unique group element Iu such that 9x 21, the disentanglement metric MLSBD can also be written as
inf PEP(G.Z) 2 J//// llll g - g2 -
(22)
in which the equivariance condition appears prominently. The condition becomes even more appar- ent v fact the Haar measure itself, in which case the metric equals
1 . [lp(g)) o h(g x) h(z))l 2
inf PEP(G,2)
(23)
Although this expression would also make for a natural choice of metric in general (when v is not necessarily the Haar measure), this choice is not directly practical as one does not have access to data points g x for arbitrary
E DATASETS: ADDITIONAL. INFORMATION
Square Translation This dataset consists of a set of images of a black background with square of 8 white pixels. The dataset is generated applying vertical and horizontal translations of the white square considering periodic boundaries.
Arrow Rotation This dataset consists of a set of images depicting a colored arrow given orientation. The dataset is generated by applying cyclic shifts of its color and in-plane rotations. The cyclic color shifts were obtained by preselecting a fixed set of 64 colors from a circular hue axis. The in-plane rotations were obtained by rotating the arrow along an axis perpendicular to the picture plane over 64 predefined positions.
ModelNet40 Airplane The ModelNet40 Airplane dataset consists of dataset of renders obtained using Blender v2.7 Community (2020) from 3D model of an airplane within the ModelNet40 dataset Wu et al. the airplane's 's color and its orientation with respect to the camera. The orientation was changed via rotation with respect vertical axis (out-of-plane rotation). The colors of the model were selected from a predefined cyclic set of colors similar to the arrow rotation dataset.
& Brox 2015). We created each image by varying two properties:
F AVA ARCHITECTURE
Table 1: Encoder architecture
Input shape
Layer
Output shape
(64, 64, d)
Conv2D
(64, 64, 64)
(64, 64, 64)
MaxPool2D
(32, 64)
(32, 32, 64)
Conv2D
(32, 32, 64)
(32, 32, 64)
MaxPooI2D
(I6, 16, 64)
(16, 16, 64)
Conv2D
(16, 16, 64)
(16, 16, 64)
MaxPool2D
(8, 8, 64)
(8, 8, 64)
Flatten
8*8*64
8*8*64
Dense
64
Dense
4+1
The architecture of the VAE encoder and decoder are shown in Tables and respectively. For grayscale images (Square Translation). d 1, for RBG images, d = 3. All convolutional layers use a kernel size of 3 x with 64 filters, stride and are followed by a ReLU activation (with the exception of the final decoder layer, which uses a sigmoid activation to produce pixel values between and 1). MaxPooling layers use have pool size of x 2. The number units in the dense (or fully connected) layers is given by their output shape. Dense layers followed by ReLU activation, except for the last layer of the encoder. UpSampling layers enlarge the first two dimensions by factor of 2, using nearest neighbor interpolation.
14

---

Under review as a conference paper at ICLR 2021
Table 2: Decoder architecture
Input shape
Layer
Output shape
4
Dense
64
Dense
8*8*64
8*8*64
Reshape
(8, 8, 64)
(8, 8, 64)
UpSampling21>
(16, 16, 64)
(16, 16, 64)
Conv2D
(16, 16, 64)
(16, 16, 64)
UpSampling2D
(32, 32, 64)
(32, 32, 64)
Conv2D
(32, 32, 64)
(32, 32, 64)
UpSampling2D
(64, 64, 64)
(64, 64, 64)
Conv2l
(64, 64, d)
The first 4 dimensions of the encoder output projected onto the latent manifold St S to represent the mean parameter of the posterior, the final dimension represents the parameter log t, for which we limit the values between -10.0 and -5.0 for numerical stability. See Perez Rey et al. 2020) for more details about these parameters and the sampling process.
15


Reviews:
[{"review_id": "ZHhZTAnVym0", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper contributes an interesting new measure for disentanglement which consists in: (1) applying to each latent representation a (supervised) linear transformation mapping it to the origin representation (2) measuring the average distance of these canonicalized representation to the origin to obtain a mean discrepancy measure of disentanglement. By adding this measure as an additional loss term in a topological VAE, the authors show that the model can learn disentangled representations with a limited amount of supervision on some simple image datasets.\n\nStrong points:\n- the measure of disentanglement proposed is promising and novel.\n- the mathematical derivations and the simulation results are convincing.\n\nWeak points:\n- some claims in the paper are misleading because they imply that the measure proposed is unsupervised, although it is totally supervised (see below for details).\n- some technical aspects of the paper lack clarity (see below for details).\n- the measure of disentanglement proposed requires a lot of prior knowledge on the data and on the structure of the representation: for all pairs of data points on which the measure is calculated, we need to know what group element this transformation corresponds to, and we need to know to which linear transformation this group element corresponds to in the latent representation. This is impractical in many datasets where the transformations are unknown, and in many models where the latent equivariant operators are learned and not pre-specified.\n\nI recommend to reject this paper due to the clarity concerns and concerns about the validity of the claims, unless they can be addressed satisfactorily during the rebuttal period.\n\nMain concerns to be addressed:\n1) It is impossible to understand from the abstract, intro and related work section whether the proposed measure of disentanglement requires supervision about the transformation between pairs of data points or not. In fact, the measure is 100% supervised. I agree that the model proposed using this measure as an additional loss term is only supervised on part of the data, but the measure itself is supervised. Here are examples of misleading claims:\n-Abstract: \"Although several works focus on learning LSBD representations, none of them provide a metric to quantify disentanglement. Moreover, such methods require supervision on the underlying transformations for the entire dataset, and cannot deal with unlabeled data.\"\n-Related Work: \"Moreover, their methods require supervision on the transformation relationships among datapoints for the entire training dataset.\"\n2) Some technical aspects of the paper are unclear:  \n- LSBD is clearly defined, SBD is mentioned multiple times but never defined.\n- the ∆VAE model is never described, nor its architecture. The reader is required to read another paper to understand how this model works.\n- \"An easy-to-compute metric to quantify LSBD given certain assumptions (see Section 4), which acts as an upper bound to a more general metric (derived in Appendix C).\" This more general metric is not described in the main text, and it is unclear what the underlying motivation is for this alternative metric (and reading the technical description in the appendix did not help me understand the motivation).\n- Enigmatic discussion points: \"Our LSBD metric and method require a number of assumptions, as explained in Section 4. This limits the applicability of the metric and method, but also provides a clear direction to what needs to be done to obtain and quantify LSBD representations if these assumptions are relaxed.\", \"Moreover, our metric is in fact an upper bound to a more general metric (see Appendix C), which is however less straightforward to compute.\" What are the clear directions on what needs to be done? How to think of this more general metric?\n\nAdditional feedback:\n- The LSBD assumptions at restated three times in the main text (p2,3,4), which is unnecessarily redundant.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VCoNR_r6lme", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=============================================================================================================\n\nSummary:\nThe paper starts from the Linear Symmetry-Based Disentanglement (LSBD) in [1]. The existing approaches to evaluate disentanglement require the supervision of the dataset and it is impossible for the unlabeled data. In this regard, the authors propose the new quantifying metric for disentanglement under the 'limited supervision' setting. Also, under this metric, authors provide a VAE-based framework that exploits limited supervision for the proposed metric.\n\n==============================================================================================================\n\nReason for score:\n\nOverall, I vote for clear rejection. I was hooked by the title and abstract. However, I could not find any novelty and insights to compare several measures for the disentanglement score. Furthermore, experiments are not enough to support the proposed method (see cons). I suggest the resubmission with more experiment results and justifications of their proposed methods. \n\n=============================================================================================================\n\nStrong points (pros) :\n\n(1) Mathematical definitions to interpret disentanglement are clear (but complex). The quality of writing is not bad.\n\n(2) I like the trial to quantify disentanglement under a limited-supervised setting since it could be applied to real-world settings.\n\n==============================================================================================================\n\ncons : \n\n(1) The comparison with the other disentanglement metrics under supervised data is crucial. The proposed metric which can be applied under a weakly supervised setting should have a certain amount of consensus with other metrics. The authors need to show through experiments and provide justification by pointing out similarities and differences with the other metrics. This part is very important to persuade reviewers and readers. \n\n(2) I was caught off guard at the experiments since there doesn't exist any baselines in Figure 4. The authors need to add baselines to compare with $\\Delta VAE$.\n\n(3) The idea underlying paper is quite overlapped with [2]. As a baseline, [2] can be used to analyze the performance of $\\Delta VAE$. Also, the authors need to emphasize the novelty of their method by comparing it with [2].\n\n=====================================================================================================\n\nMinor :\n\nThe word \"limited supervision\" is quite confusing. Instead, I suggest \"weakly-supervised\". \n\n======================================================================================================\n\nAfter rebuttal :\n\nThank you for the responses. I'm not sure why the authors didn't perform the experiments on the correlation between the previous factor-based disentanglement scores and the proposed disentanglement score in the limited supervised setting. For example, if there are 5 factors, I propose to evaluate the proposed disentanglement score for every possible pair of the factors (10 pairs) and average the scores. I believe this paper handles the valuable topic but it is not enough to be accepted since the experiments, which are crucial I believe, are omitted (comparison with the other disentanglement score, baselines to $Delta$VAE, comparison with [2]). Also, I concerned that other readers might be confused with the (\"factor-based\" disentanglement and \"symmetric-based\" disentanglement ) and (limited-supervision and weakly-supervision) (a new section should be added to handle these topics if this paper should be accepted). Furthermore, discussion with the related works is not enough.\n\n\nI lower my confidence rate to 3 (5->3) and vote for weak reject (3->4). But, I hope this paper would be accepted after revisions in the future.\n\n============================================================================================================\n\nReferences \n\n[1] beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework, ICLR 2017.\n\n[2] Weakly-Supervised Disentanglement Without Compromises, ICML2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PZsP-9R09cm", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper follows on the work of Higgins et al. 2018 that used linear symmetry based disentangled (LSBD) representations where real-world transformations provide some structure in the data that can be leveraged. The main contribution of this paper is to provide a metric to measure the quality of disentanglement in the learned representation. The paper makes some assumptions on the samples in the dataset and assumes that there are some group actions that relate one data point to another. Under this assumption, the paper derives a simple and easy-to-compute disentanglement measure. Second, this paper shows a method that can work with partial supervision to learn disentangled representation. This is done by using some data with supervision on the underlying transformation between the data points, and another set of data points where no labeling information is given. The proposed method uses a diffusion variational autoencoder. \n\nPros:\n\n1) This paper proposes a metric that can be used to quantify disentangled representations. Disentangled representation is one of the overloaded terms in the community, and there have been multiple interpretations without any clear evidence as to whether the learned representation is disentangled or not.\n2) The problem formulation and setup has been nicely illustrated using observation and inference function modeling the different factors of variation in the data, and group actions that allow the transformation of one data point to another. \n\n\nCons:\n\n1) My first concern is regarding the strong assumptions used in this paper. It is hard for me to think of real world data, where we can have simple group actions that map one data point to another. This restricts the use of the disentanglement measure proposed in this paper or real data. \n\n2) While the paper clearly argues about the importance of having a measure for disentanglement, it does not tie this to the effectiveness of the downstream application. In practice, we are always interested in disentangled representations that also capture all the relevant information from the original data. While it might be possible for the inference function h to satisfy the equation h(g_n . x_1) = \\rho (g_n). h(x_1)  as given in equation (2),  but this may not mean that the learned representation contains the information associated with the original data points. \n\n3) The batch size used for the supervised data points is 2. This implies that there is a group action between only two samples. I am a bit concerned that this does not capture the generality of the proposed formulation proposed in section 5. \n\n4) The main algorithm proposed in this paper is based on the diffusion variational autoencoder (Peez et al. 2020), and the method to handle unsupervised data is just by alternating the training between supervised and unsupervised data. There is not much analysis or ablation study on how much unsupervised data can be used.  \n\n5) No comparison is made with other disentanglement methods that use partial supervision in the form of set membership. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GnerCxDGvDn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper aims to operationalize the symmetry-based disentanglement idea proposed in Higgins et al., 2018, and proposes a novel loss function that can be used to both evaluate and learn disentangled representations (including learning from datasets where only a subset of samples are fully labeled). The paper includes some experimental validation of the proposed ideas on synthetic datasets. \n\nOverall, the paper is clearly written and makes a positive contribution in pushing for disentangling metrics that are motivated by better formalism than what is commonly used. The paper would be stronger if it included rigorous comparisons to state of the art disentangling metrics and methods, and a discussion of how the requirement of full supervision of all factors of variation for at least a portion of the dataset can be relaxed.\n\nWeaknesses:\n* The supervised subset mentioned in section 6 must have full labels for all factors/groups. If even one factor is missing, then this method cannot be applied, since we do not know the full set of transformations that relate different data points to each other. It is not clear to me how this assumption can be relaxed with the proposed metric/objective: if we lack labels for only a single factor, then it is no longer possible to explain the dataset in terms of actions on a single input sample.\n* The above also means that experiments can only be done on synthetic datasets, where all factors are known.\n* This paper would be stronger if it (a) showed a comparison of the LSBD metric to commonly used metrics for disentangling, such as those compared in Locatello et al., 2018. and (b) compared the proposed model to other disentangling models, both supervised and unsupervised. An implicit comparison to diffusion VAE can be found in the results in Figure 4, when the number of labeled pairs is zero, but a more thorough comparison is required to understand the value of the model and metric.\n\nStrengths:\n* Novel method for evaluating and learning disentangled representations.\n* Can leverage unlabeled data.\n\nClarity\n* In the experiments, the g’s are given by the data. But how are the \\rho functions implemented? Are they specified in advance?\nSection 4, bullet point 3: “each observation is a transformed version of another observation, and that transformation is unique”. This seems like a fairly strong statement that is often violated: for example, rotating and flipping a square can easily map to the same X. It would be useful to discuss:\n- how does the metric/model behave when these assumptions are violated in a domain?\n- how does the loss react to subspaces that are scaled differently, or subspaces of differing dimensionality? In the experiments I believe all the subspaces are the  same size, but if one was in R^2 and another in R^100, the squared-distances would be dominated by the latter and the loss may not work as well.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new metric to measure symmetry-based disentanglement and uses this metric to optimize diffusion VAEs on a set of small, synthetic datasets. In general, reviewers found the theoretical framework introduced to be interesting and relevant, but there were a number of concerns regarding the empirical evaluation in the paper and the clarity of many of the claims, particularly wrt the need for strong supervision (pairs of data points with a known transformation between them) for both evaluating the metric and for training by regularizing the proposed metric. I'd encourage the authors to focus on the improvement points suggested by reviewers, most notably by improving the empirical evaluation by adding detailed ablations and comparisons (e.g., exploring the relative amount of supervision needed, comparisons to previous approaches) and clarity regarding the supervision required. As such, I recommend that it be rejected in its current form.  
Decision: Reject