Quantifying and Learning Linear Symmetry-Based Disentanglement
Authors: Loek Tonnaer
Luis Armando Pérez Rey
Vlado Menkovski
Mike Holenderski
Jacobus Willem Portegies
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Representation Learning
Disentanglement

Abstract: The definition of Linear Symmetry-Based Disentanglement (LSBD) formalizes the notion of linearly disentangled representations, but there is currently no metric to quantify LSBD. Such a metric is crucial to evaluate LSBD methods and to compare to previous understandings of disentanglement. We propose $\mathcal{D}_{\mathrm{LSBD}}$, a mathematically sound metric to quantify LSBD, and provide a practical implementation. Furthermore, from this metric we derive LSBD-VAE, a semi-supervised method to learn LSBD representations. We demonstrate the utility of our metric by showing that (1) common VAE-based disentanglement methods don't learn LSBD representations, (2) LSBD-VAE as well as other recent methods \emph{can} learn LSBD representations, needing only limited supervision on transformations, and (3) various desirable properties expressed by existing disentanglement metrics are also achieved by LSBD representations.

Quantifying and Learning Linear Symmetry-Based Disentanglement
Anonymous Author(s)
Affiliation
Address
email
Abstract
The definition of Linear Symmetry-Baie -Based Disentanglement (LSBD) formalizes the notion of linearly disentangled representations, but there is currently no metric quantify LSBD. Such metric is crucial to evaluate LSBD methods and compare to previous understandings of disentanglement, We propose DLSBD, a mathematically sound metric to quantify LSBD, and provide practical implemen- tation. Furthermore, from this metric we derive LSBD-VAE, a semi-supervised method to learn LSBB representations. We demonstrate the utility of metric by showing that (1) common VAE-based disentanglement methods don't learn LSBD representations, (2) LSBD-VAE as well other recent methods can learn LSBD representations, needing only limited supervision on transformations, and (3) various desirable properties expressed by existing disentanglement metrics are also achieved by LSBD representations.
2
10
11
Introduction
14
Learning low-dimensional representations that disentangle the underlying factors of variation in data considered an important step towards interpretable machine learning with good generalization. To address the fact that there is no consensus on what disentanglement entails and how to formalize it, Higgins al. (2018) propose a formal definition for Linear Symmetry-Ban Based Disentanglement, or LSBD, arguing that underlying real-world symolddymmmee symmetries give exploitable structure to data.
15
18
17
18
However, there is currently metric quantify LSBD. Such metric is crucial properly evaluate methods aiming to learn LSBD representations and to relate LSBD to previous definitions disentan- glement. Although previous works have evaluated LSBD by measuring performance downstream
20
21
tasks (Caselles-Duns Dupré et al., 2019) or by measuring specific traits related to LSBD (Painter et al.) 2020 Quessard al. 2020), none of these evaluation methods directly quantify LSBD according its well-formalized definition. 24
22
23
We propose DLSBD. well-formalized and generally applicable metric that quantifies the level of LSBD in learned data representations. We show an intuitive justification this metric, as well its theoretical derivation. We also provide practical implementation compute DLSBD for common symmetry groups. Furthermore, we show that our metric formulation can be used to derive semi-supervised method to learn LSBD representations, which call LSBD-VAE. To make LSBD-VAE more widely applicable, we also demonstrate how to disentangle symmetric properties from other non-symmetrie properties, and how quantify this disentanglement with DISBD.-
26
28
30
31
32 show the utility of DLSBDD by quantifying L.SBD in a number of settings, a variety of datasets with underlying SO(2) symmetries and other non-symmetric properties. First, we evaluate common Vaseb disentanglement methods and show that most don't learn LSBD representations. Second, evaluate LSBD-VAL and other recent methods that specifically target LSBD, showing that they
34
Submitted to Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

38 can obtain much better LSBD scores while needing only limited supervision on transformations.
37 Third, we compare DLSBD with existing disentanglement metrics, showing that various desirable properties expressed with these metrics are also achieved by LSBD representations.
2 Related Work
40
Plenty of works have focused on learning and quantifying disentangled representations recently, but research has shown that there is little consensus about the exact definition of disentanglement and methods often do not achieve it as well as they proclaim (Locatello et al., 2019). To introduce some much-needed formalization, Higgins et al. (2018) proposed define disentanglement with respect to symmetry transformations acting on the data. They used group theory to provide two formal definitions, which we refer as (Linear) Symmetry- -aased Disentanglement, (L)SBD. In this paper we focus only on LSBD, not SBD.
42
44
46
Several methods have been proposed to learn LSBD representations Gaselles-Dupte et al. 2019 Painter et al. 2020 Quessard et al. 2020). These methods also learn to represent the transformations acting on the input data, assuming various levels of supervision methods have previously focused on capturing transformations of the data outside the context of disentanglement well (Cohen and Welling 2015 Sosnovik et al. 2019 Worrall et al. 2017).
48
these transformations. Other
50
3
Linear Symmetry-Based Disentanglement
52
Higgins et al. (2018) provide formal definition of linear disentanglement that connects symmetry 54
59
transformations affecting the real world (from which data observed) to the internal representations of model. The definition is grounded in concepts from group theory, we provide a more detailed description of these concepts in the Supplementary Material.
55
56
57 The definition considers group G of symmetry transformations acting on the data space X through the group action G x X + X. In particular, G' can be decomposed the direct product of K groups G = Gi X x GK- A model's internal representation of data is modeled with the encoding function h X Z that maps data the embedding space Z. The definition for Linearly Symmetry- Based Disentangled (LSSB) representations then formalizes the requirement that model's 's encoding should reflect and disentangle the transformation properties of the data, and that the transformation properties of the model's encoding should be linear. The exact definition is as follows:
59
61
63
64
Definition: Linear Symmetry-Based Disentanglement (LSBD) A model's encoding map
65
X + Z, where Z a vector space, is LSBD with respect to the group decomposition G
86
G1 ... Gk if
87
1. there is a decomposition of the embedding space Z = Zi ..... + ZK into K vector subspaces,
68
2. there are group representations for each subgroup in the corresponding vector subspace G2 - (Z,( k E (1....,A K}
70
71
3. the group representation p G > GL(Z) acts on as p(g)
(p)(g)) 21, ................................ .PK (gk) =k),
for g -01....,gk" 9K) E G and (æ1 2x) Z with gk E Gy and 2 Zμ. 4. the map h is equivariant with respect to the actions of G on X and Z, i... for all X and y E holds that ligg x)) = p(g) h(x).
73
74
Furthermore, we say that a group representation p is linearly disentangled with respect to the group decomposition = G2 x x... x Gg it satisfies criteria to from the LSBD definition above.
76
'The original definition actually considers an additional set of world states W. but our definition is more practical and can be shown be the same under mild conditions, Supplementary Material.

---

77
4
Quantifying LSBD: DLSBDD
4.1
Intuition: Measuring Equivariance with Dispersion
79
To motivate our metric, let's first assume a set- ting in which suitable linearly disentangled group representation is known. Let's further assume that the dataset of observations can be expressed with respect to G acting on some base {gn- Jo m==
80
h(ao)
En' Al ) h((w))
81
83
point zo E X, i.e. {Ir, J.V Formally, this assumes that the action of G on 86 X is regular. In this case, we can use the inverse group elements In to transform each data point toward the base point To. i.c.
plom
84
85
87
86
99n 20)
Z
Z
To = 91 x1 = =... -N IN-
(2)
Figure 1: A dataset of images from a rotating ob- ject expressed in terms of the group G' = SO(2) acting on a base image xg- possible to quan- tify the level of LSBD of an encoding map h by measuring its equivariance with respect to group representation p. Since all data has been gener- ated from To. equivariance can be measured as the dispersion of the points (p(g=t) h((r))
89
Since p linearly disentangled, we only need to measure the equivariance of the encoding map quantify LSBD. Equivariance is achieved when hgg x) p(g) h(z). for all G, x E
90
91
92
93
X. Given the dataset described above, we can
check this property for E {an J and y E
94
95 (9 m= particular, from Equation ( we
can see that we have equivariance if
p(gN') h(za)) = p(o ())))(()==))))))))))())))))))( = ... p(ox') MM(N))
97 This not only characterizes perfect equivariance, but also allows for an efficient way to quantify how close we are true equivariance, by measuring the dispersion of the points {p(g. -(xx))) Given suitable norm
98
l|z in Z, we can thus quantify LSBD in this setting
N
1 N
Σ
p(on h((a))
Σ plg h(xw()
(4)
N
iec we compute the mean of (p(gm)) 1(xx)) ) and use the average squared distance to this mean for points in {p(g, h(x,,))== as our LSBD metrie, see Figure
100
101
102
However, this formulation requires knowing the right linearly disentangled group representation and suitable norm in Z. Moreover, implicitly assumes uniform probability measure over the group elements {gn I mn In the next section we formulate our metric for a more general setting.
103
104
105
4.2
DLSBD: A Metric for LSBD
106
Generalizing the ideas from the previous section with concepts from measure theory, we propose metric to measure the level I LSBD of any encoding h : X -> given a data probability measure μ on X, provided that μ can be written as the pushforward Gx (:, 2o) a of some probability measure
107
108
109
y on by the function G (:, xо) for some base point zg. More formally,
(AA)= Gx (t, u = v (lg E G Gx .r) E A)),
(5)
110 for Borel subsets A X. Note that this is only possible the action Gx transitive.
111 For example, the situation of dataset with N datapoints { n=1 = {gn xo |o corresponds to the case in which and μ are empirical measures the group G and data space X, respectively:
Σ E
(6)
NNote that {on 1 can he used to describe all known group transformations hetween elements in the dataset by means of composition and inverses, since x; = gs (9 ..). Thus suffices to check equivariance for these group transformations.
Note that do not actually need know zo nor h(xo).
3

---

113 We define the metric DLSBD for an encoding h and a measure μ as
(a [ 1. hig--1a- -x)) - Man pl dr(g),
DLSED
inf PEP(G.2)
with Mpph, [ hhl') zo))d(l'),
114 where the norm Il-1lo... is Hilbert-space norm depending on the representation p. the encoding map
115 h: X Z, and the data measure j.. More details of this norm can be found in the Supplementary
118 Material. Moreover, P(G, 2) denotes the set of linearly disentangled representations of G in Z.
117 Lower values of DLSBD indicate better disentanglement, zero being optimal.
4.3
Practical Computation of DLSBD
119
There are two main challenges for computing the metric of Equation 7 First, to calculate the integrals in the formula, all possible datapoints that can be expressed g Zn with g E G
121
G ... X Gk must be available. Second, the infimum of the integrals over all possible linearly
122
disentangled representations must be estimated. This requires finding the possible invariant subspaces
123
Z= = Z 1 ... @ Zx induced by the encoding h over which the group representations are disentangled.
124
We present a practical implementation of an upper bound to DLSBD for an encoding function h given dataset X' generated by some known group transformations. In particular, this approximation of LSED is designed for group decomposition G G1 x ... x Gk where each G = SO(D,) with fl..... K} the group of rotations in Dj dimensions. This implementation approximates the integrals of Equation (7) by using the empirical distribution of X. The invariant subspaces of Z to the subgroup actions are found by applying a suitable change basis. In the new basis, the disentangled group representations are expressed in parametric form whose parameters are optimized to find the tightest bound to DLSBD. See Figure 2 for an intuitive description of the process.
125
126
127
128
129
131
132
Assume there is dataset X'' that can be modeled in terms of the group decomposition G = G1 X-. Gk. For each G subgroup there set of known group elements Gy C Gy uniformly sampled such that the dataset is described in terms of all elements in G = G1 x ... x Gx and a base point To as .g....... g9)
133
134
135 X
Za I9k E Gk. k E {1...., K}}.
136
For each subgroup Gk we construct a set of encoded data 2g S Z whose variability should only de- pend on the action of Gg. The set Z& is given by Ze (= gK 19, 99 (1,.... K}}.
137
138
in which
zye (g) 9k) = h h((gh....................... .Kk) 2O)
h((g)....... B...ggg.... g6+1 1........... 9k) 20). (8)
g'EG2
139
Similar to (Cohen and Welling 2014), we find suitable change of basis that exposes the invariant
140
subspace Z& corresponding to the k-tth subgroup Gg. The new basis is obtained from the eigenvectors
141
resulting from applying Principal Component Analysis (PCA) Zu. Each element in Zμ is projected
into the first D2 eigenvectors. The new set is denoted as Z C Dz with elements z (gI...... gK) S
142
RD Dx that are the projected versions of zn (g)....,,
143
144
Quessard et al. 2020) describes how one could parameterize the subgroup representations of SO(Da)
145
for arbitrary Di but here we will focus on G₁ = S2))). In this case, we can parameterize each subgroup representation in terms of single integer parameter w E Z as (gk)(gk) corresponding to 2 x 2 rotation matrix whose angle of rotation is w multiplied by the known angle associated approximate the M Mp.h,zo from
147
to the group element Ok G₁= SO(2). For this subgroup Equation as Mr, given by
149
MR
Σ (0L....)))
( z'(g....., .9)).
150 Similar to Equation 7 we would like to find the optimal Pw. that minimizes the integral over the group representations. We can define parameter search space f C Z, e.g. Ω = [-10, 10) for finding
152 the optimal w E Ω that minimizes the dispersion, this is expressed in the following equation
D = min WER
Σ
IPp )- - "(91.....he - Mbwll2
G
(10)

---

h(g7k x0)
h(x0)
3k
PCA
Pkive
Z
h(gh -00
Z
R
R2
Figure 2: Consider dataset modeled by group decomposition G X Gk acting on zo and is embedded in latent space 7 via h. In this example the subgroup G5 SO(2) models the rotations of an airplane. Other subgroups Gzs could also be acting e.g. changes in airplane color. The first step to calculate the disentanglement of Gje is to construct set of data embeddings 2, S Z whose variability is due to GR. These embeddings are then projected into a 2-dimensional space through PCA. For these projected embeddings we can describe the group representations in simple parametric form Pk. For given Pk,w the equivariance of Ge is measured as the dispersion after applying the action of the inverse group representation Pk.w
153 Each DISED measures the degree of equivariance of the projected embeddings for each k-th subgroup corresponding to the best fitting group representation. The upper bound to the metric is finally obtained
155
by averaging across all subgroups DLSBD S 1 Σ
Ek=1 LSBDD K
K D
5
Learning LSBD Representations: LSBD-VAE
156
In this section present LSBD-VAE semi-supervised VAE-based method to learn LSBD repre- sentations. The main idea is to train an unsupervised Variational Autoencoder (VAE) (Kingma and Welling 2014; Rezende al. 2014) with suitable latent space topology, and use our metric as an additional loss term for batches of transformation-labelle data.
158
159
180
Asumptions LSBD-VAE requires some knowledge about the group structure G that is to be disentangled. Concretely, the group and its decomposition = G x ... x Gk should be known, well as suitable linearly disentangled group representation p G + GL(Z) and latent space 21 + ... @ Zg. Moreover, we assume there exists an embedded submanifold Zg S such that the action of G on Z restricted to Zg is regular, and Zg is invariant under the action. Only Za will then be used as the codomain for the encoding map, h X - Zg-
182
164
Z
166
167 We demonstrate the assumptions above for the common group structure G = SO(2) x SO(2). For the group representation p P1 @ P2. with Z 22 3 2 we can use rotation matrices in RR2 for # and p2. We can then use 1-spheres S {= E 2 ||=||| = 1) for the embedded submanifold:
168
Za S St. this case, the action of G on Z restricted to Zg is indeed regular, and Zg is invariant under the action.
170
172
Unsupervised Learning Latent Manifold To learn encodings only on the latent manifold Za. we use a Diffusion Variational Autoencoder (A VAF) (Perez Rey et al. 2020). A VAFS can any closed Riemannian manifold embedded in Fuclidean space a latent space (or latent manifold), provided that certain projection function from the Euclidean embedding space into the latent manifold known and the scalar curvature of the manifold available. The A VAE uses a parametric family of posterior approximates obtained from a diffusion process over the latent manifold. To estimate the intractable terms of the negative ELBO. the reparameterization trick is implemented via random walk.
173
175
178
177
178
179
180
In the of S' latent (sub)mani consider 2 as the Euclidean embedding space, and the projection function R IT: IR2 + 5t normalizes points in the embedding space: IT(=) 2/|=]. The scalar curvature of 5 O.
181
182
This projection function is not defined for but this value does occur in practice.
5

---

183
Semi-Supprvised Learning with Transformation Labels
Duupe et al. (2019) proved that LSBD representations cannot be inferred from training unlabeled observations, but that access to the transformations between data points is needed. They therefore use training set of observation pairs with given transformation between them.
184
186
187
However, we posit that only a limited amount of supervision is sufficient. Since obtaining supervision on transformations is typically more expensive than obtaining unsupervised observations, it is desirable to limit the amount of supervision needed.
188
189
augment the supervised AAE with a super- vised method that makes use of transformation-labelee batches, i.e. batches (Imm M m=1 such that Im Im 41 for m = 2,...,M, M, where the transformations 9m (and thus their group representations p(gm)) are known and referred to transfor- mation labels. The simplified version
Therefore,
191
21
enc.
(())) )
dec.
193
122
enc.
+ p(
dec.
Maz
195
1
198
M [m
187
198
enc.
p(gM) .Z
dec.
Mox
189
200
of the metric from Equation (4) can then be used for each batch as an ad- ditional loss term (with 2g 21). as
LxL
CLSBD
202
CRE
it is differentiable under the assump-
204
tions described above (using the Eu- clidean norm).
Figure 3: Overview of the supervised part of LSBD-VAR
205
206
We make small adjustment Equation (4p for the purpose of method, since the mean computed there does not typically lie on the latent manifold Zg. Thus, we use the projection II from the A VAE
207
208
project the mean onto Zg. Writing the encodings as pm hxmm), the additional loss term for
209
transfomatiion-labele batch {Im {Im]m=1
then becomes
M Σ
LLSBD
M p(am')
1
Σ
2m II
p(gml)
2m
(11)
M
210
where 91 = the group identity.
Moreover, instead of feeding the encodings 2m to the decoder, use p(gm) where
II M ΣM p(gm²)) -mm)))))mmmmmmmmm.m This encourages the decoder to follow the required group structure. This only affects the reconstruction loss component of the AAE.
=m).
213
214 Figure 3 illlsttrates the supervised part of our method for censformation-labell batch {xm JM The loss function the regular ELBO (but with adjusted decoder input as described above) as used in A VAE plus an additional term LLSSD. where is weight hyperparameter to control the influence of the supervised loss component. By alternating unsupervised and supervised training (using the same encoder and decoder), we have method that makes use of both unlabeled and rransformatioo-llbelledd observations.
215
216
217
219
Experimental Setup
220
We evaluate the disentanglement of several models on three different image datasets (Square, Arrow, and Airplane) with a known group decomposition G = SO( x SO(2) describing the underlying transformations. For each subgroup fixed number of | = 64 with & {1, 2} transformations is selected. The datasets exemplify different group actions of SO(2): periodic translations, in-plane rotations, out-of-plane rotations, and periodic huc-shifts.
222
224
226
In real settings, not all variability in the data can be modelled by the actions of group. Therefore. we also evaluate the same models on two datasets ModelNet40 (Wu et al.) 2014) and COIL-100 Nene et al. 1996) that consist of images from various objects (i.e. non-symmetric variation) under known out-of-plane rotations (SO(2) symmetries). In many settings it is easy to obtain labels for such rotations, e.g. when the camera or object angle controlled by an agent. See Figure for examples of the datascts. For more details, see the Supplementary Material.
227
228
230

---

Square
(b) Arrow
(c) Airplane
(d) ModelNet40
(e) COIL-100
Figure 4: Example images from each the datasets used. Each row shows different examples from a single factor changing.
232
For the Square, Arrow, and Airplane datasets we test LSBD-VAL VAE with transformation-labee batches of size M 2. More specifically. for each experiment we randomly select L disjoint pairs of data points, and label the transformation between the data points in each pair. We vary the number of labeled pairs from 0 (corresponding to Á VAE) to N/2 (in which case each data point involved in exactly one labeled pair). We set the weight of the supervised loss component to = 100 for all experiments. We choose M = 2 for our experiments since it is the most limited setting for LSBD-VAE Higher values of M would provide stronger supervision, so successful results with M 2 2 imply that good results can also be achieved for higher values of M (but not necessarily vice versa).
233
234
235
237
239
241
For the COIL-100 and ModelNet40 datasets, we train LSBD- LSBD-VAE on batches containing images of one particular object from all different angles (72 and 64 for COIL- 100 and ModelNet40, respectively). Each batch labelled with transformations (g) e...... (gM: e). where Im represent rotations, and the unit transformation indicates that the object is unchanged. To represent the rotations we use S latent space as in A VAE, whereas for the object identity we use a 5-dimensional Euclidean space with standard Gaussian prior as in regular VAEs. LSBD is measured as the disentanglement of rotations in the latent space. For these experiments we used 7
242
243
245
247
248
We furthermore test number of known disentanglement methods for comparison, including tra- ditional disentanglement methods as well methods focusing LSBD, In particular, we use emennttggeee.n..mmmmmmmmmmmnmnne
249
250
disentangl ement lib (Locatello et al. 2019) to train a regular VAE (Kingma and Welling 2014
251
Rezende et al. 2014 4), 3- 3-VAE (Higgins et al. 2017). CC-VAE (Burgess et al.J (018). Factor VAE (Kumar et al. 2018). Furthermore we evaluate the method from Quessard et al. (2020) that focuses on LSBD. We also tested forwardVAF (Caselles-Dupre)
252
Kim and Mnih, 2018 1). and DIP-VAE-II
253
et al.) 2019), but show only limited results since we were not able to reproduce any reasonable results our datasets.
255
256
We use encodings from all these methods to evaluate DLSBD, as well as common traditional disentan- glement metrics from disentang] ement 1ib: Beta (Higgins et al. 2017), Factor (Kim and Mnih
257
258
2018). SAP (Kumar et al. 2018). DCI Disentanglement (Eastwood and Williams 2018), Mutual Information Gap (MIG) (Chen et al. 2018). and Modularity (MOD) (Ridgeway and Mozer 2018).
259
280
More information about the architecture, epochs and hyperparameters can found in the Supplemen- tary Material. For the traditional disentanglement methods trained on Square, Arrow and Airplane datasets the latent spaces have 4 dimensions, since these are the minimum number of dimensions necessary learn LSBD representations for underlying SO(2) SO(2) symmetry group,
262
264
(Higgins et al. 2018 Caselles- Dupré et al. 2019). For COIL-100 and ModelNet40 we use latent spaces with 7 dimensions for fair comparison with the LSBD-VAE method.
285
Results: Evaluating LSBD with DT.RD
286
287
We now highlight four key observations from our experimental results. In particular, we differentiate between methods (VAE, 3- 3-VAE, CC-VAE, FACTOR, DIP-I, DIP- and metries (BETA, FACTOR, SAP, DCI, MIG, MOD) that approach disentanglement in the traditional sense, and
268
270
methods (A VAF,,, QUESSARD, L.SDD VAE) and metric (DIsBD) that focus specifically on LSBD..
The full quantitative results can be found in the Supplementary Material.
7

---

272
7.1
Standard Disentanglement Methods Don't Learn LSBD Representations
273
Figure 5 summarizes the DISED scores (lower is better) for all methods on all datasets. Bars show the mean scores over 10 runs for each method, the vertical lines represent standard deviations. SAB/TV indicates our method trained on I. labelled pairs (LSBBD-
274
275
corresponds to the unsupervised A VAE), LSBD- VAF/ful indicates our method trained on batches containing single object all known transformations (for datasets with non-symmetric variation). Note that LSBD- VAE obtained very good scores (nearly 0) on the Arrow and Pixel datasets, hence the missing bars.
277
279
0.8
0.7
1.75
1.5
6.5
0.4
1.0
dataget
Arrow
Airbane
COIL 100
Mode/Not40
0.5
Pxxl
02-
0.1
0.0
VAE
CC-VAE
Factovas
DIRVARS
Querer Qunraard
LSRO- VADS
Lo8D- VAEG5G
LOND- VAE.12
LS00 VAc/708
LeaD VAE/L200
Lsmn vAt/ VAE/1536
LsaD LSRD-VAEr VAE/-722227
(sRD VAE22008
VAS pysse
PacroniA
Guescund
LSED-VAED
LSRD VASAH
(sap VAU
(a) Datasets with SO(2) x (O))) symmetries
(b) Datasets with SO(2) and non-symmetric variation
Figure 5: LLSBD scores for all methods on all datasets
280
None of the traditional disentanglement methods achieve good DLSBD scores, even if they score well on other traditional disentanglement metrics. This implies that LSBD isn' 't achieved by traditional methods. Moreover, from the full results in the Supplementary Material we see that the traditional methods on these datasets do not achieve good scores on all traditional metries. In particular, SAP,
281
282
283
DCI, and MIG scores are low. We believe this is result of the cyclic nature of the symmetries underlying our datasets, further emphasizing the need for disentanglement methods that can capture such symmetries.
285
287
The SAP and MIG scores measure to what extent generative factors are disentangled into a single latent dimension. However, since the factors in our dataset are inherently cyclic due to their symmetry structure, they cannot be properly represented in a single latent dimension, as shown by Peree Rey
298
289
290
et al. 2020). Instead, at least two dimensions are needed to continuously represent each cyclic factor our data. A similar conclusion was made by Musselee-Dupree et al. (2019) Painter et al.) (2020).
291
292
DCI disentanglement measures whether latent dimension captures at most one generative factor. This is accomplished by measuring the importance of each latent dimension in predicting the true generative factor using boosted trees. However, since the generative factors are eyclic, the performance of the boosted tree classifiers is far from optimal, thus providing more importance to several dimensions in predicting the generative factors and giving overall lower DCI scores,
294
296
287
7.2
LSBD- VAE and other LSBD Methods Can Learn LSBD Representations with Limited Supervision on Transformations
298
299
From Figure 5 we observe that methods focusing specifically on LSBD can score higher on DLSSD,, showing that they are indeed more suitable to learn LSBD representations. In particular, LSBD-VAE got very good DLSBD scores for all datasets. Moreover, our experiments on the Arrow, Airplane, and Pixel datasets also show that only limited supervision suffices obtain good DLSBD scores with low variability.
300
301
302
303
We only partially managed to reproduce the results from Quessard et al. (2000) our datasets. Their method scored fairly well on the Airplane, ModelNet40, and COIL- 100 datasets, but did not do well on the Square and Arrow dataset in our experiments.
305
306
307
Furthermore, we tested VorWardVAE by [Caselles-Dupr et al. (0019, but we did not manage to produce any reasonable results datasets, trying both their original architecture and the
8

---

309
architecture we used for our other experiments. Therefore, we do not include scores for this method.
310
We did however manage reproduce VAE'ss results the Flatland dataset, which was used in their paper. For those experiments, we computed mean PLSBD score of 0.012 with standard deviation 0.001 over 10 runs, indicating that Forward indeed learns LSBD representations for Flatland.
312
314
7.3
LSBD Representations Also Satisfy Previous Disentanglement Notions
315
Our results also indicate that LSBD captures various desirable properties that are expressed by traditional disentanglement metrics. Figure we compare DLSBD scores with scores for previous disentanglement metrics, for all our experiments. Note that for DLSBD lower is better, whereas for all other metrics higher is better. As we noted before, good scores on traditional disentanglement metrics don't necessarily imply good DLSBD scores. Conversely however, methods that score well on DLSBD also score well on many traditional disentanglement metrics, often even outperforming the traditional methods. In particular, from the full results (see Supplementary Material) we see that LSBD- VAE matches or outperforms the traditional methods on the BETA, FACTOR and MOD metrics, and achieves much better scores for the DCI metric where traditional methods scored poorly.
316
317
318
320
322
324
The MIG and SAP scores are still low for methods focusing on LSBD. This expected however, as explained earlier in Section 7.1 This was also observed by Painter et al. 2020) for different datasets.
325
1.0
1.0
meiit vell
0.6-
Facter
0.6
Beta
metrc
5AP
DO
2.4
MIG
MOD
0.0
2.0
0.75
1.25
1.75
2.00
0.50
0.25
3.02
1.50
1.75
D,050
(a) Scatter plot
(b) Binned by DLSBD value
Figure 6: Comparing DISSDD previous disentanglement metrics
326
8
Conclusion
327
We presented DLSBD. a metric to quantify Linear Symmetry-Based Disentanglement (LSBD) as defined by Higgins al. (2018). We further used this metric formulation to motivate LSBD-VAE semi-supervised method to learn LSBD) representations given some expert knowledge on the underlying group symmetries that are to be disentangled.
328
330
331
We used DLsBd to evaluate various disentanglement methods, both traditional methods and recent methods that specifically focus LSBD, and showed that LSBD- VAE can learn LSBD representa- tions where traditional methods fail to do so. We also compared Dr.sed to traditional disentanglement metrics, showing that LSBD captures many of the same desirable properties that are expressed by existing disentanglement methods. Conversely, we also showed that traditional disentanglement methods and metrics do not usually achieve or measure LSBD.
332
333
334
335
336
337
Challenges that remain are expanding and testing LSBD- VAE and DlsBd on different group struc- tures, towards more practical applications, well as focusing the utility of LSBD representations for downstream tasks.
339
The work is fairly theoretical, and practical methods derived from this work have no obvious negative societal impact. However, the ideas presented are relevant to representation learning and could be, in particular, used in computer vision and agent control applications.
Broader Impact
341

---

References
343
344
Burgess, C. P.. Higgins, I., Pal, A., Matthey, L.., Watters, N., Desjardins, G., and Lerchner, A. (2018). Understanding disentangling in B-VAE. arXiv preprint arXiv: wwxxv:/804.03599
348
Caselles- Duupré, H., Ortiz, M. G., and Filliat, D. (2019). Symmetry-based disentangled representation learning requires interaction with environments. In Advances in Neural Information Processing Systems, pages 4606 4615.
348
349
Chen, T. Q., Li, X., Grosse, R. B., and Duvenaud, D. K. (2018). Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pages 2615- 2625.
351
Cohen, T. and Welling, M. (2014). Learning the irreducible representations of commutative Lie groups. 31st International Conference on Machine Learning, pages 3757-3770.
353
354
Cohen, T. S. and Welling, M. (2015). Transformation properties of learned visual representations. In 3rd International Conference Learning Representations.
355
356
Eastwood, C. and Williams, C. K. (2018), A framework for the quantitative evaluation disentangled representations. In International Conference Learning Representations.
358
Higgins, I., Amos, D., Pfau, D., Racaniere, S., Matthey, L., Rezende, D., and Lerchner, A. (2018). Towards a definition of disentangled representations. arXiv preprint arXiv:
359
Higgins, I. Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerehner, A. (2017). 3-VAE: Learning basic visual concepts with constrained variational framework. In International Conference Learning Representations.
361
383
Kim, H. and Mnih, A. (2018). Disentangling by factorising. In International Conference on Machine Learning, pages 2649-2658.
384
Kingma, D. and Welling, M. (2014). Auto-Encoding Variational Bayes. In International Conference on Learning Representations.
386
387
Kumar, A., Sattigeri. P., and Balakrishnan. A. (2018). Variational inference of disentangled latent
388
concepts from unlabeled observations. International Conference on Learning Representations.
369
Locatello, F., Bauer, S., Lucic, M., Gelly, s., Schölkopf, B., and Bachem, O. (2019). Challenging
common assumptions in the unsupervised learning of disentangled representations, In International Conference on Machine Learning.
371
372
Nene, S. A., Nayar, S. K., Murase, H., et al. (1996). Columbia object image library (coil-20).
Painter, M., Prugel-Bennece A., and Hare, J. (2020). Linear disentangled representations and
374
unsupervised action estimation. Advances in Neural Information Processing Systems, 33.
375
Perez Rey, L. A., Menkovski, V., and Portegies, J. (2020). Diffusion variational autoencoders. In Proceedings the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20,
376
377
pages
Quessard, R., Barrett, T. D., and Clements, W. R. (2020). Learning Group Structure and Disentangled Representations of Dynamical Environments. Advances in Neural Information Processing Systems, 33.
379
381
Rezende, J., Mohamed, S., and Wierstra, (2014). Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 3 International Conference on International Conference on Machine Learning-Volue Volume 32.
383
384
Ridgeway, K. and Mozer, M. C. (2018). Learning deep disentangled embeddings with the f-statistic loss. In Advances Neural Information Processing Systems, pages 185-194.
385
386
Sosnovik, I., Szmaja, M., and Smeulders, (2019). Scale-Fquivariant Steerable Networks. Interna- tional Conference on Learning Representations, pages 1-14.
10

---

388
Worrall, D. E., Garbin, S. J., Turmukhaabetov. D., and Brostow, G. J. (2017). Harmonic networks:
309
Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., and Xiao, J. (2014). 3D ShapeNets: A Deep Representation for Volumetric Shapes.
392
Checklist
393
394
1. For all authors...
395
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
396
(b) Did you describe the limitations of your work? [Yes]
398
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
389
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
401
402
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
404
(b) Did you include complete proofs of all theoretical results? [N/A] The theoretical contributions are new definitions and methodology, these have been motivated but do not require proofs.
405
406
407
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] In Supplemen- tary Material
409
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In paper or Supplementary Material
412
413
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] Each experiment was run 10 times, we report means and standard deviation, and show error bars where needed.
414
416
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Supplementary Material we described the hardware. Tracked time of training for experiments with LSBD-VAE and Quessard approach, also reported in the Supplementary Material. The times for training the traditional methods in disentanglement Jibb were not measured.
417
419
420
4. If you are using existing assets (c.g., code, data, models) or curating/releasnng new assets...
422
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] In Supplementary Material
(c) Did you include any new assets either in the supplemental material as URL?? [Yes]
425
Included code in the Supplementary Material
428
(d) Did you discuss whether and how consent was obtained from people whose data you're 're usin//curating? [N/A] No consent was needed.
427
428
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] It clearly doesn't.
430
5. If you used crowdsourcing or conducted research with human subjects...
431
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
432
435
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
436
11


Reviews:
[{"review_id": "Z1EbOwtWkd", "reviewer": "Reviewer_KFfN", "summary": "This work focuses on developing a metric evaluation metric and a semi-supervised VAE-based model for linear symmetry-based disentanglement (LSBD)[1]. When assuming knowledge of linearly disentangled group representation $\\rho$, the authors deduce that LSBD is fulfilled if the last requirement is met (equivariance of a map h). The authors propose to measure the equivariance by simply quantify the equivariance of the encoding map w.r.t. to the group and observations. Although a straightforward thought, computing the metric is challenging. Therefore, the authors propose an approximation by calculating the dispersion of the inverse group representation obtained through a PCA. Similar to the evaluation metric, the authors propose LSBD-VAE, a semi-supervised version of $\\Delta$-VAE to regularise the latent space based on its dispersion. They show through empirical evaluation that their method outperforms other unsupervised disentanglement approaches w.r.t. to the proposed evaluation metric.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "_Originality, significance, and general assessment_\n\nThe symmetry-based approach to disentanglement is theoretically sound and appealing. However, the work did not provide a proposal on learning the appropriate representation that fulfills the group-based disentanglement requirements. This work tries to fill this gap by proposing the LSBD evaluation metric and semi-supervised VAE. Although I found this research direction very exciting, I have some concerns about this work:\n* Strong assumptions which may not hold beyond synthetic data: This work assumes explicit knowledge of the data and its structure for the LSBD evaluation metric and method. For each data pair, we need to know the corresponding group and linear transformation. This is quite a strong assumption, even if the semi-supervised version only assumes knowledge of some thousands of data pairs. I worry that we cannot judge from this work whether this is a realistic setting and whether some thousands of data pairs will be enough to generalize in a non-synthetic setting. \n* Fair benchmark comparison: LSBD-VAE has explicit knowledge of the groups of pairs of data instances. Even though only optimized in a semi-supervised fashion, this gives the optimization more signals than in an unsupervised setting. Therefore, comparing LSBD-VAE to only unsupervised methods is not fair. Locatello et al.[2] proposed a weakly-supervised approach that might be a more suitable comparison.\n* Applicability: I do think with disentanglement, it is not about just perfectly learning to disentangle the factors of variation, but also about the promise that disentangled representations generalize better, are fairer, and more robust. This work only shows that the model can disentangle well w.r.t. the evaluation metric they proposed. However, the part of generalization, e.g., reconstruction ability and performance on downstream tasks, has not been investigated.\n* Usefulness: The big question that remains unanswered after reading this paper is: \"Why should I use this metric?\" It is not clear why this evaluation metric is superior to the other 5-6 existing ones. What do other metrics fail to capture, which the LSBD can? Can you show some examples, theoretically or empirically, where the other metrics but LSBD fail?\n\n_Quality and clarity_\n\nIn general, I could follow the paper and its structure. W.r.t. related work and the base model that LSBD is using, I think the authors should elaborate more details to make the paper self-contained:\n* Related work (l.47-51): This discussion is very vague and does not describe what these works do. Can you elaborate on these works and how they are different from the semi-supervised approach?\n* $\\Delta$-VAE was used as the base for the semi-supervised approach. However, it was barely described. For myself, it helped reading through the original paper to have a better understanding of what precisely the method does. However, for the work to be self-contained, it might be helpful to the reader to have a brief description of $\\Delta$-VAE.\n\n_Comments and questions_\n\n* Figure 6 & ll.316-319: I do not necessarily agree with \"good scores on traditional disentanglement metrics do not necessarily imply good $D_{LSBD}$ scores\". In particular, Figure 6b) shows that Beta, Factor, and MOD scores are proportional to $D_{LSBD}$. High Beta, Factor, and MOD scores correspond to lower $D_{LSBD}$, and lower Beta, Factor, and MOD scores correspond to higher. Can you elaborate?\n* LSBD evaluation metric: Except for the difference in the \"exact\" values, it is not clear at all why your evaluation metric should be preferred to the other ones?\n* What is the percentage of labels used (e.g., for the results plotted in Figure 5)? You only mentioned the absolute number of labels, not the relative fraction.\n* All baseline disentanglement models are trained in an unsupervised manner. In contrast, the proposed model knows the groups and uses that in a semi-supervised fashion. I think a fairer comparison would be [3] as it uses weak supervision between pairs of data. This is quite similar to the way LSBD-VAE uses data pairs with the knowledge of its group and transformation. In my opinion, this work can be seen as a weakly-supervised approach. The paper might improve from discussing the similarities with [2].", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work does not discuss any potential societal impact. Adding such a discussion would improve the paper's overall quality. For instance, in that discussion, the authors could discuss both positive (interpretability, implications for understanding and evaluating fairness/bias in models) and negative impact (using representations for discriminating against specific subgroups and thus, compromise fairness) of disentangled representation learning.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "XufJXl6dwpQ", "reviewer": "Reviewer_CUVD", "summary": "This paper proposes a new metric $D_{LSBD}$ to measure Linear Symmetry Based Disentanglement. The metric is motivated from an idealized setting before outlining a practical upper bound that can be realized. The authors also apply their new metric as an additional loss term to a $\\Delta$-VAE model and show that it greatly enhances Linear Disentanglement while also agreeing with past notions of Disentanglement. The experiments, while simple and using only $SO(2)$ as a subgroup, are demonstrative of the proposed approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really enjoyed the clarity of the presentation in this paper. In particular, using dispersion as a means to define a disentanglement measure is very intuitive and also novel. Moreover, most of the expository figures are well placed and serve as a nice visual guide to much of the theory presented. Overall, I find this paper interesting and a small step towards quantifying linear symmetry based disentanglement. \n\nThere are a few comments and mild concerns that I have however which I hope the authors can clarify in their responses. I believe the theory is sound but the assumptions of the theory must be scrutinized. The assumption that the action of $G$ on $X$ is regular is quite a strong assumption and many datasets will have no hope of satisfying this. It would be good to get a quantification of the degradation of performance with respect to the new metric if modify one of the datasets in such a way that the action is no longer free. The second, and perhaps, the biggest gripe I have with this paper is the over reliance on $SO(2)$ as a subgroup. I understand why this subgroup was chosen as it is easy to perform experiments on but much of the practical theory currently does not generalize beyond this group.  However, the paper is currently framed as a more general framework and at the moment it is non-trivial to see how this would be generalized to even simple groups like $E(2)$. As a result, I would ask the author's to either tone down the main claims or repurpose them to be for $SO(2)$.  This to me is the biggest limitation of this work, even though the new disentanglement metric itself is interesting. Moreover, in section 4 the optimization was done over the set of all possible disentangled subspaces ---i.e. finding $\\rho$ but in section 5  line 163 for LSBD-VAE one key assumption is that you have a linearly disentangled group representation. This seems at first glance a major decoupling of the theory from practice. It would be great to get some insight into this.\n\nQuestions/Minor Remarks:\n- Why do we need a change of basis to expose the invariant subspace? Specifically, why can't we consider the total map $h' = (PCA \\circ h) (x)$ to be the equivariant encoding function?\n- What is the $\\rho$ in $X$ space? I think you only mention $\\rho$ in $Z$ space\n- I think projecting the mean to the latent submanifold is a bit ad hoc. What you really want is to use the Reynolds operator to average over the subgroup.\n- Line 58: Not all groups can be decomposed (outside of trivial decompositions). Please amend this statement.\n- Line 181: is the scalar curvature is 0 is this not just a flat space rather than S^1? I believe I have a minor misunderstanding here.\n- Figure 5, it seems like adding more supervision for the Airplace class (yellowish orange bar) doesn't seem to help as much as for Arrow and Pixel class. Can you maybe shed light into why this is the case?\n- Fig 6a is illegible. A table here would be better.\n- I would highly recommend an Algorithm box to explain the training procedure of LSBD-VAE. Often it was difficult to decouple the theory from the actual objectives (with the exact equations) being optimized.\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "BdSfJtnTGKQ", "reviewer": "Reviewer_j9A8", "summary": "Summary: \nThis paper aims to operationalize the symmetry-based disentanglement idea proposed in Higgins et al., 2018, and proposes a novel loss function that can be used to both evaluate and learn disentangled representations (including learning from datasets where only a subset of samples are fully labeled). The paper includes some experimental validation of the proposed ideas on synthetic datasets. It also includes cross-comparisons between both the LSBD-based metric and method to state of the art disentangling metrics and methods, which demonstrate the complementary aspects of the LSBD method (e.g., its ability to capture multi-dimensional disentangled factors, for example rotations in a euclidean plane).\n\nOverall, the paper is clearly written and makes a positive contribution in pushing for disentangling metrics that are motivated by a novel formalism. The impact of this paper is slightly limited due to the strength of supervision required (full supervision of all factors of variation) for implementing both the D_LSBD metric and the loss.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n* Novel method for evaluating and learning disentangled representations.\n* The objective function can be trained in a semi-supervised method with limited fully-supervised data (although the level of supervision required is extremely high)\n* You have done thorough comparison to unsupervised disentanglement approaches, but it would also be useful to know how your method fares against standard semi-supervised approaches, e.g., [1], which could be adapted for a continuous supervised variable.\n\nQuality/Clarity\n* There are multiple ways of representing different transformations. For example, rotations can be represented either as 2D coordinates (as you have assumed here) or as 1D angles. In the case of unsupervised disentangling, it’s often unclear what the most parsimonious representation should be - how could the uncertainty over representations be handled in your framework?\n* For your LSBD-VAE: It would be interesting to know are the relative contributions of L_{LSBD} and the adjusted likelihood (reconstructing from $\\rho(g_m) \\cdot \\bar{z}$ rather than z_m) to the disentangling performance in the empirical results section. \n\nSignificance\n* The supervised set must have full labels for all factors/groups. If even one factor is missing, then this method cannot be applied, since we do not know the full set of transformations that relate different data points to each other. It is not clear to me how this assumption can be relaxed with the proposed metric/objective: if we lack labels for only a single factor, then it is no longer possible to explain the dataset in terms of actions on a single input sample.\n* The above also means that experiments can only be done on synthetic datasets, where all factors are known.\n\nReferences\n[1] Kingma, Diederik P., et al. \"Semi-supervised learning with deep generative models.\" Advances in neural information processing systems. 2014. Kingma, D. P., Mohamed, S., Rezende, D. J., & Welling, M. (2014). Semi-supervised learning with deep generative models. In Advances in neural information processing systems (pp. 3581-3589)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed limitations and potential negative impact", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "5aPZT1ZXUYi", "reviewer": "Reviewer_UCXu", "summary": "This paper makes two contributions:\n1 - It presents a framework for measuring linear symmetry based disentanglement (LSBD). In particular, it addresses the challenge that LSBD is only defined with respect to an optimal decomposition of the latent space $Z = Z_1 \\oplus \\ldots \\oplus Z_k$ that needs to be identified by the metric.\n2 - The metric serves as additional supervision in the training of a VAE on toy datasets with at most two symmetry transformations. The results show quantitative improvements in terms of disentanglement.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Recommendation\nI am recommending to accept this submission which offers solid mathematical treatment \n- the background, problem and goal of the metric are clearly and intuitively explained\n- the dispersion approach promises to reduce the level of transformation supervision needed by the metric\n- section 4.2 has some very nice aspects, in particular (i) extending to non-uniform sample distributions via a reparametrization of the measure using a transitivity assumption is very rigorous and elegant (ii) the final definition of the metric in equation (7) is clean and general.\n- the practical implementation in section 4.3 is sound\n\nThe recommendation is weak because I see the following limitations to the significance the approach:\n- the LSBD-VAE seems to make the assumption that the latent state decomposition $Z = Z_1 \\oplus \\ldots \\oplus Z_k$ is now a given of the problem, which simplifies differentiability of the metric and seems to contradict the prior definitions. I would argue that this diminishes the originality of the method. Also, this would explain the results described in 7.3 that \"LSBD Representations Also Satisfy Previous Disentanglement Notions\"\n- despite the claims of limited supervision, I believe the amount of supervision used (transformation-labeled batches, l193) is important, which explains why LSBD-VAE outperforms previous approaches so clearly. \n- the experimental evaluation is restricted to toy datasets.\n\n\n## Remarks\n- The PCA step (l139-143) is done independently for each subgroup. What happens if different subgroups are projected on the same dimensions. Is this sound?\n- Is the upper bound on l155 theoretically proven?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have satisfyingly addressed this question.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agreed that the proposed metric for linear symmetry based disentanglement is solid.  However, the three main criticisms were:
1) The lack of comparison with semi-supervised or weakly-supervised alternative methods, considering that this method requires labeled data,
2) The lack of empirical checking that LSBD correlates with downstream properties that are more directly useful in concrete tasks, and
3) The fact that SO(2) was the only group investigated in depth in the paper, despite the presentation as a general method.

Overall, the paper is not in bad shape, but the limited cope of applicability and empirical comparisons mean it is slightly below the bar.  Other examples of symmetry groups, correlation with metrics such as robustness, and comparison with semi-supervised methods would all substantially improve the paper.
Decision: Reject