Unsupervised Disentanglement with Tensor Product Representations on the Torus
Authors: Michael Rotman
Amit Dekel
Shir Gur
Yaron Oz
Lior Wolf
Venue: ICLR 2022 Poster (2022)
Keywords: Variational Auto-Encoder
Disentanglement Learning

Abstract: The current methods for learning representations with auto-encoders almost exclusively employ vectors as the latent representations.  In this work, we propose to employ a tensor product structure for this purpose. This way, the obtained representations are naturally disentangled. In contrast to the conventional variations methods, which are targeted toward normally distributed features, the latent space in our representation is distributed uniformly over a set of unit circles. We argue that the torus structure of the latent space captures the generative factors effectively. We employ recent tools for measuring unsupervised disentanglement, and in an extensive set of experiments demonstrate the advantage of our method in terms of disentanglement, completeness, and informativeness. The code for our proposed method is available at https://github.com/rotmanmi/Unsupervised-Disentanglement-Torus.

Published as a conference paper at ICLR 2022
UNSUPERVISED DISENTANGLEMENT WITH TENSOR PRODUCT REPRESENTATIONS ON THE TORUS
Michael Rotman', Amit Shir
Shir Gur!, Yaron Oz²*, Lior Wolf! School of Computer Science, Tel-Aviv University, Israel
2 Univrses, Sweden
School of Physies and Astronomy, Tel-Aviv University, Israel
4 Simons Center for Geometry and Physics, Stony Brook, USA
ro manaami@post.tauue tau.ta.aaa..i...... amit www.eeeeiivsseee...... com, shir gaaz.ggu@es.tau wtt.aaaaaa........ ya ronozetauees atw.:/ttttax.........t... wol facs tau.ac.u............... i1
ABSTRACT
The current methods for learning representations with auto-encoders almost ex- clusively employ vectors as the latent representations. In this work, we propose to employ lensor product structure for this purpose. This way, the obtained representations are naturally disentangled. In contrast to the conventional varia- tions methods, which are targeted toward normally distributed features, the latent space in representation is distributed uniformly over set of unit circles. We argue that the torus structure of the latent space captures the generative fac- tors effectively. We employ recent tools for measuring unsupervised disentangle- ment, and an extensive set of experiments demonstrate the advantage of our method terms of disentanglement, completeness, and informativeness. The code for our proposed method is available at https:///itthb: com rotmanii/ Dnsupervisedd-Dieeee wewnngleeeettttto
INTRODUCTION
Unsupervised learning methods of disentangled representations attempt to recover the explanatory factors of variation in the data. The recovered representation, is expected to be: (i) disentangled, ie., each element should vary one of the generative factors (ii) complete in the sense that all generative factors ability to relate via a low capacity model (e.g., a linear model) the generative factors and the latent representation
be controlled by the latent representation and (iii) informative, that is the
has been shown that solving this task without further assumptions is impossible, since there is an infinite number of bijective functions f that map between and (c) (Locatello et al. 2019) with all corresponding generative models having the same marginal distributions of the observations. In this work. we propose use representations that are tensor product of elements, which take values on unit circles S claiming that this structure is suitable for the effective recovery of the underlying modes of variation.
This claim follows from entropy considerations. One wishes to have a representation that has a low entropy of entanglement. The entropy of entanglement is measured by evaluating the number of non-zero eigenvalues of the Schmidt decomposition of the representation. If representation can be described by only one term, which is an outer product of the orthogonal basis functions, has the lowest possible entanglement entropy, zero. If representation requires the of more than one term in the decomposition, it is entangled and its entanglement entropy can be quantified, e.g. by the Von Neumann entropy constructed from non-zero eigenvalues.
The tensor product unit circles, S! takes the shape of an n-torus T = (S')), and has a low entanglement property. This representation has the advantage that it capture the periodicity structure of the generative factors, where each circle controls a different aspect the original distribution. Unlike other generative models, which rely the Gaussian distribution prior, 7 is

---

Published as a conference paper at ICLR 2022
compact manifold, and therefore any function that acts on it, such as decoder, has interpolate only, but not extrapolate when generating new instances.
In this work, present the -'E, a Variational Auto Encoder whose latent space resides on the T torus manifold. In an extensive set of experiments with four different datasets, we compare the torus latent representations with others proposed in the literature. We show clear advantage of the torus representation in terms of various measures of disentanglement, completeness, informativeness and propose a new metrie, the DC-score, which assesses the combined disentanglement and com- pleteness performance. We present a detailed quantitative and qualitative analysis that supports our method.
2 RELATED WORK
Generative models aim create new instances that are indistinguishable from given distribution. Auto encoders (Kramer 1991) achieve this by constructing the identity function using a composition of two components, an encoder and a decoder. The encoder tasked with compressing the input into a latent representation with a dimension smaller than the input's, s, whereas the decoder is responsible for reconstructing the original input from the latent space.
Such auto encoders usually overfit to their training data and are very limited when tasked with generating new samples. Instead of mapping each input instance to a constant vector, Variational Auto Encoders (Kingma & Welling] 2013) (VAE) map each instance to pre-defined distribution, by minimizing both the encoder-decoder reconstruction loss and a KL-divergence term that depends on the prior distribution. VAE's are capable of generating new instances by sampling a latent vector from the predefined distribution; however, the interpretation of latent space components is typically obscure and the mapping between them and the dataset properties may be complex.
In order to increase the latent space interoperability, various modifications of the original VAE were introduced. These variants try to achieve factorization of the latent space components with each component corresponding to specific simple characteristic of the dataset. In B8- VAE (Higgins et al. 2016), the weight of the KL-divergence term is increased w.r.t. the reconstruction term, which results in better factorization of the latent space. However, this introduces a new hyper-parameter and departs from the theoretical derivation of the ELBO term. For example, large 3 parameters may prevent the conditional distribution from modeling the data faithfully. DIP-VAE (Kumar et al.) (017)) introduces a regularizer that constrains the covariance matrix of the posterior distribution. This term is added the original VAE loss function, along with two hyper-parameters, and helps achieve better factorization. Factor- VAF (Kim Mnih) 2018) adds a Total Correlation (TC) approximation penalty over the produced latent codes and introduces additional dyyer-parametes.
2.1 DISENTANGLEMENT METRICS
In order to evaluate the expressiveness of different generative models, plethora of disentanglement definitions and metrics were suggested in the literature (Do & Tran 2019: [Eatwood & Williams) 2018), see Locatello al. (2019) for comprehensive review. In this work, we adopt the defini- tions and metrics introduced in Eastwood & Wiiliams (2018) which provides a successful set of disentanglement metrics.
The authors of Eastwood & Williams 2018 introduce three metrics, Disentanglement, Completeness and Informativeness (DCI), to quantify the disentanglement properties of the latent space and its ability to characterize the generative factors of a dataset. Given a dataset that was generated using a set of K generative factors, I' E R and that it is required to learn latent codes, RD of D D-dimensional representation. Ideally, in an interpretable representation, each generative factor zi would correspond to only one latent code c.. It is also beneficial if the mapping is linear, as the correlation between generative factors and codes can then be easily assessed. Generating new instances that are indistigguishable from the original dataset distribution requires that the latent codes cover the whole range of the generative factors. Furthermore, such a representation provides the ability to modify a specific property of a generated instance by directly tuning the corresponding latent code.
2

---

Published as a conference paper at ICLR 2022
The DCI metrics aim to quantify the relationship between codes and generating factors by having single number that characterizes the relative importance of each code Ca. in predicting a factor, 2, which in turn defines an importance matrix Rai. To construct the importance matrix, K regressors are trained to find a mapping between 2; and C, 2 = fe(C). In this work, we follow Fastwood & Williams (2018) and infer the importance matrix using lasso linear regressor's weights, Wia. by Rio
Once the importance matrix Rai is obtained, the DCI metrics can be defined explicitly. The disen- tanglement is given by
rank (R)
D Σ
D
Po DoD
K
where
Raj
E, Raj
D. -1 = 1 Σ Pax logg Pak:
Paj =
Σx=1
Ruk
Pa.
E. Roo
(2)
High disentanglement means that each entry of the latent vector č correspondss to only one element in 2 (in linear sense), that is, each code element, Cu. affects one generating factor. The disentanglement metric defined above differs from Eastwood & Williams/ (018) by a correction factor rank(At) When the rank of R is equal to the generative factor s dimension, this correction equals 1, and does not affect the metric. However, it does make a difference when the number of codes is smaller than what is needed account for all generating factors. Typically, one assumes that there are at least as many expressive codes as the number of factors. When this is not the case, it means that even if we have disentanglemmnt among the expressive codes, their number is not sufficient, and correction accounts for this. Note that the correction has no influence if there are irrelevant code elements in addition sufficient number of expressive ones. The p factors handle cases where some of the code dependence on the factors is very weak (namely irrelevant codes), while other codes depend mainly on one factor and hence should be of equal importance.
K
The completeness is defined by 1
D C, -11-H((P)) 1 Σ logn Pas:
Raj Σb= Roj
C=
Pas
In contrast to the disentanglement metric, which considers a weighted mean over the generative factors, motivated by the fact that irrelevant units in č should be ignored, here all codes are treated equally.
A situation in which each factor is explained by only one element of the code results in high completeness.
The informativeness is the MSE between the ground -rund factors, z, and the predicted values, fy (7).
K
[10 - fy (0)
I
=
K Edataset
(4)
Disentanglement and completeness values range between zero and one, where one is the best value, while the best value for informativeness zero.
Disentanglement and completeness alone do not provide sufficient information on whether a rep- resentation factorizes properly. For example, consider the case of representation where only one generative factor is described by all the codes. While this representation is completely disentangled, is not complete, and is therefore meaningless. In order to have meaningful representation, both disentanglement and completeness need to be high. Thus, we introduce new score, called the DC-score, which accounts for both disentanglement and completeness. We define the DC-score as the geometric mean of the two metrics, DC-score VDC. This way, only cases where both scores are high will result in a high DC-score. Furthermore, the score will favor cases where both disentanglement and completeness are comparable rather than having different values while having the same arithmetic mean.
'throughout the paper use a, b, c.... and i, k,... letters for indices of the codes é' and the factors / components respectively.
3

---

Published as a conference paper at ICLR 2022
m mg in my mi mg mimss mi m²
m
"
m
m m3
V(m)
mi
Figure An example of how a tensor product two circles, T = x S is converted to flattened latent representation that can be fed into the decoder. Each point the torus can be deseribed by two angles. Each angle represents an Ma tuple. The tensor product of the ma tuples is concatenated with their respective 0 component to produce the input to the decoder.
METHOD
Assume a pre-determined set of vectors sampled from unknown distribution X. each vector containing K independent factors, 5 E RK The training dataset contains samples ø, F(T)), == L,....,, .V that are generated using generative function F. In unsupervised disentanglement, the learner has access nor F, and receives only the set of samples, (x=))=1
The learner has to recover a set representation vectors č that are linked to žσ in a way that is bijective and disentangled, necessary to be able to sample new representation vectors Grew, and to obtain a generative function G such that the new generated samples are from the same distribution of F()), where E~ X.
Section 2.1 Furthermore, in order to generate
samples, it is also
In this work, we view the latent representation vector Ca the angles associated with a list of D two- dimensional unit vectors my IR², 1 S S D,i.e, Va, ||mal 1. Using theses two-dimensional vectors we define
Uprod
vec
(
prood
P1..00 prod
mi
Vorient (m) ........... mb)
(5)
where & is the outer product operator, oa {0, 1) and vee is the vectorization operation (equivalent to flattening the tensor). Define the operator V, that given m.,.... mDD concatenates Uprod together with Vorients V (m (mi,..., mD) [pprod Vorient]. Fig. depicts the for D) 2.
The vector v = V (m1...., mD) resides in a vector subspace T defined by only a set of D parameters. The additional D elements of Vorient required to ensure that the mapping V is bijective. For example, this can be seen by examining the case of D 2: let m. = (cos 1, Oi), m2 (cos 02, sin 02), m² m and m/ --2 Then my X m2 m² Ø mly
A natural way of acquiring a random point on the circle, S1, is by sampling two independent Gaussians, m th ~ N (pgg oft)). Each tuple of these vectors then normalized to have a unit norm,
m
m
(6)
(m2) (m²)
Assume that the elements, ma follow the normal distribution with zero mean and standard deviation 1, mg* ~ (0, 1), then the vectors m. follow the uniform distribution on S
For the purpose of obtaining the distribution parameters of m the encoder is applied on an instance,
o8 op
(z) =
o
u
The reparametization trick (Kingma & Welling 2013) is then used to obtain a set of normally distributed vectors. Denote by S the sampling operator, sample the coding vector
M, = {{(
m m
m
}} S(e(xr)), mp
(8)
4

---

Published as a conference paper at ICLR 2022
22
Figure 2: Heatmap of the generative factors vs. the codes for the teapots dataset with our architecture with six circles. The factors ordered by azimuth and elevation angles, followed by RGB values. The disentanglement, completeness and DC-scores are 0.67, 0.59 and 0.64, respectively. One can clearly see the high disentanglement property where each code mostly controls only one factor and high completeness where most factors are controlled by one code (e.g. 0g controlling 23).
where each component, (a, aia) of m, = D, = 0, is sampled from N(p. (pP.o.) .o)).
mD) The decoder G then acts on V (m1,..., m.)) to generate new instance, .. The requirement that the generated instance is indistinguishable from the original distribution (x) translates minimizing
the ELBO loss function. First, it maximizes the log-likelihood probability of E to be similar to the training sample reconstructs by having an 12 loss term. Second, to encourage uniform distribution on the torus, it contains KL-divergence term with respect to N (0, 1p) The overall expression for the loss function used during the optimization the networks e and G is
G (v (M1)) =1|" - BDrr (p (m) (m)
-
where 8 hyperparameter introduced in B- V-V (Higgins et al. 2016 and rr N (0, Lp).
As the latent representation resides on the torus, sub-vectors 2D components of m, may be described by D angles, &g2 E (0, 2w] (the angles 0₂ are identified with the codes c.). In order to generate new instances, identify, m cos 0,, m = sin 0. and apply the decoder G (V (m)) to these elements.
The torus topology, T", is both compact and periodic. Since it is compact, for every sampled 6... there is a nearby 0' from the training set, thus, the network G can be viewed as an interpolator. This in contrast vectors sampled over RDD in which G has to extrapolate. Furthermore, being periodic, this topology is able to exploit a periodic structure in the generative factors. Consider a common generative factor associated with rotations. rotation cannot be represented by a single non-compact variable; therefore encoding rotation in latent space requires the entanglement of two components. However, on the torus, only one compact dimension can be used identify this generative factor.
4 EXPERIMENTS
We compare our method to the leading VAE architectures from the current literature. The baseline methods are (i) 3-VAE (Higgins et al. 2016), (ii) DIP-VAE-II (Kumar et al. 2017). and (iii) Factor-VAE (Kim & Mnih 20188.. The code for all methods was taken from the Pytorch-VAl VAE repository (Subramanian, 2020) (Apache License 2.0). The network architectures for both the
encoder, decoder and the Factor- VAE's discriminator (which we do not use) follow those in Eastwood
Williams (2018), see Table I Each horizontal line Table T a skip-connection
All methods were trained using ADAM (Kingma & Ba 2014) with learning rate of 0.0001 and a batch size of 144. The hyperparameters used for DIP- VAE were Ading = 10 and Aodf = 1.0. All methods were trained for maximum number of 50 epochs, and the metrics were evaluated
5

---

Published as a conference paper at ICLR 2022
using the model that performs best on the validation set with respect to MSE. Furthermore, each baseline method was evaluated using two latent representations sizes, D = 10, 128 for all datasets.
For our method, 7 we use 8 circles, (5*)² 8 This results in V E R136, leaving the number of parameters in the decoder comparable to the other baseline methods. Each experiment was evaluated with five different seeds and the mean value is reported.
In order to compute the DCI metrics, we used a linear lasso regressor based on the implementation of Eastwood & Williams (2018) (MIT License) and Zaidi et al. (2020) (Apache License 2.0). For each generative factor, a lasso regressor fits between the normalized codes obtained by the encoder, e, to a normalized factor, namely fa (e) for each 2,. For each factor, the parameter for the regressor using 10-fold cross-valiaation from the set { {10-6, 10-5, 10-4, 10-s 10-2, 0.1, 0.2, 0.4, 0.8, 1). picking the one with the minimal the MSE error. The weights of the lasso regressor are then used to construct the W R R matrix from which the metrics computed as explained in section 2.1
The Fréchet Inception Distance (Heusel et al. 2017) (FID) is used to assess the quality of image generation. Specifically, we use the implementation of /Seiteer (2020) (Apache License 2.0) and report the FID at the final average pooling features of the Inception3 NN (Szegedy et al.) 2016).
The teapots dataset (Moreno et al.,, 2016 contains 200, 000 64 64 images of camera-centeres teapots rendered using a set of 5 generative factors, azimuth that is sampled from ~ U [0, 2n), elevation that is sampled from ~ U [0, 3) and three RGB channels, each sampled from U [0, 1]. All VAEs are trained over subset of 160, 000 (the generative factors unknown to all architectures), whereas the disentanglement metrics are evaluated on the remaining 40, 000 images.
The Cars3D dataset (Reed al. 2015) contains 183 CAD models of different cars projected into 64 x 64 image with four elevations and 24 azimuths. These three generative factors are split between training set of 14054 images and validation set of 3514 images.
The 2dshapes Dataset, introduced here and released to the public (MIT License), contains 200, 000 64 x 64 images of four possible shapes, triangles, squares, pentagons and hexagons. All shapes in the dataset are centered in the image and have a white background. The factors controlling the images are the following: specific shape drawn from a discrete uniform distribution, a scale value between ~ U[20, 40). rotation angle 0 ~ 2m), and color RGB values r ~ U(0, 1], g [0, 1]. b U|0,, 1] all sampled uniformly. Since the shapes are symmetric, different angles could correspond to the same image. The training set contains a subset of 160, 000 instances, whereas the disentanglement metrics are evaluated on the remaining 40, 000 validation instances.
The 3dshapes dataset (Burgess & Kim 2018) contains 480, 000 64 x 64 images of four possible objects, a cylinder, a tube, a sphere and box rendered with linearly spaced scale values, 10 hues linearly spaced, floor wall and object hue values and fifteen different orientations, which amounts to six generative factors.
The dSprites Dataset (Matthey et al. 2017) contains 737, 280 64 x 64 binary images of three shapes, a square, an ellipse and a heart with six different, equally spaced scale values, 40 different orientations, and 32 x 32 possible positions in the plane, amounting to five generative factors. 589, 824 instances were used for training and evaluation was performed on the remaining 147, 456 validation instances.
Results Across all the datasets (Tables 2). our - architecture obtains the highest DC-score, which combines the disentanglement and completeness scores, compared with the baselines. In all of these cases the generated images are meaningful, as can be seen from the relatively low FID scores. Furthermore, for teapots, 2dshapes and dSpriites we also achieve the best reconstruction MSE and FID scores. In the 3dshapes dataset 3- 3-VAE, which achieves the highest score. Qualitative results for the influence of each circle for the teapots and 2dshapes datasets can be seen in Fig. 3
obtain the best MSE score and comparable FID scores to
For the Cars3D dataset the results are quite different, even though our DC-score is highest. The best performing method in terms of the MSE and FID scores is VAE-II 128, which is not among the best performing methods for the other datasets. One plausible reason is that this dataset, although having only three generative factors, hides a much richer high-dimensional distribution. The factors controlling the car model (183 in total) implicitly contain many characteristics of a car, which requires a high dimensional latent space. Nonetheless, our method achieves the second best FID score for this dataset.

---

Published as a conference paper at ICLR 2022
(a)
Figure 3: The latent representation of the VAE. Each S circle corresponds to different factor of the dataset that is varied as the angle changes. (a) The 5 most expressive circles on the Teapot dataset. (b) The 6 most expressive circles on the 2dshapes dataset.
Table 1: Architectures for the encoder e, the decoder G and the discriminator used for Factor- VAE. These architectures follow the ones in Eastwood & Williams (2018).
Encoder
Decoder
Discriminator
conv
FC d 8192
+ 1000
BN, ReLU, x 64 conv
BN, ReLU, 3 312 conv
BN, LeakyReLU 0.2 FC 1000 - 1000
BN, ReLU,
conv
BN, ReLU, 3 x 3.12 conv
BN, ReI.U, 3 128 conv
BN, ReLUU, 3 3 256
BN, Lea.kyakl..UU.. 0.2
BN, ReLU,
conv
BN, ReLU, 3 x 35656 conv
FC 1000 -> 1000
BN, ReLU, 3 256 conv
BN, ReLU, 3 3 128
BN, LeakyReLU 0.2
BN, ReLU, x 3 512 conv
BN, ReLU, 3 x 3 128 conv
FC 1000 +
BN, RcLU, 512 conv
BN. RcLU, 3 64 conv
BN, ReLU, x 3 512 conv
BN. ReLU, 3 3 64 conv
8192 +
BN, ReLU, x conv, tanh
Dependence on 3 and on D In order to examine the sensitivity of our method to different tori dimensions, well as the influence of the 3 parameter KL-divergence, examined the DC-score and reconstruction error on the Teapot dataset, using four topologies, T T5, T and T and using five different 3 coefficients, 0, 1, 3.3 6, 9 (0 corresponds to Kl.-divergence). As can be seen in Fig. 4 removing the KI-divergence completely. leads to a decrease in the reconstruction error; however, the DC-score also decreases. The VAE exhibits the worst reconstruction error not surprisingly, can only encode four generative factors, whereas there are five generative factors in the dataset. Both the T5EVAE, V-EA and VAE have the capacity of encoding all generative factors, however, the encoder of the VAE should ignore three of the circles whereas the encoders of the T T.-E and T should ignore at most one circle. Thus, when increasing 3, the induced angles on 7 tend to more uniform distribution, which entangles between themselves. This does not happen in 5--VAE and 6 VAF, since there is at most one extra component that can entangled.
Heatmap analysis Disentanglement results can be qualitatively evaluated through a visual inspection of the 2d heatmaps the generative factors z, vs. the codes ca (the angles 0, our case). In Figure 2 we show an example of the heatmap results for using our method with D = 6, namely using a 6 VAE. The disentanglement and completeness properties are clearly seen in the 21, 22, 23, 24 dependence 05. 82, f.. 01 respectively, while images help us better understand cases where one of the scores is high and the other is low, or whether specific regressor is suited for capturing the functional dependence.
worse completeness for the 2u factor. Such
7

---

Published as a conference paper at ICLR 2022
0.6
100
MISE:
00
0.2
-10-3
0
5
B
(a)
(h)
(c)
Figure 4: The performance dependence on the latent torus and parameter of the KL-divergence term with five generating factors.(a) The accumulated DC-score. (b) The reconstruction error. (c) The FID score. We see a clear advantage for a latent space of at least the same dimension as the number of generating factors.
Table 2: Results all the datasets.
Dataset
Method
DC-Score
D
C
I
MSE
FID
3-VAE 10
0.1796
0.1902
0.1702
0.0263
0.0178
154,9922
3-VAE 128
0.1286
0.1715
0.0964
0.0123
0.0187
159.7412
DIP-VAE-II 10
0.1226
0,2480
0.0676
0.0534
0.0186
151.7267
teapots
DIP-VAE-II 128
0.0966
0.1592
0.0593
0.0180
0.0063
62.8249
Factor-VAL 10
0.2526
0.2183
0.3119
0.0601
0.0167
110.7673
Factor-VAl VAE 128
0.3891
0.3564
0.4761
0,0586
0.0061
70.9153
VAE
0.6230
0.6558
0.5919
0.0332
0.0012
15.7170
3-VAE 10
0.2898
0.2913
0.2883
0.0335
0.0145
143.7522
3-VAE 128
0.1733
0.2748
0.1093
0.0216
0.0165
178.0128
HI--VAEI1 10
0.1701
0.2525
0.1342
0,0650
0,0224
221,5138
2d shapes
DI--AAE-1 11 128
0.1125
0.2354
0.0543
0,0378
0.0131
78.1923
Factor- VAE 10
0.1875
0.2104
0.1676
0.0660
0.0214
122,2750
Factor-VAL VAE 128
0.5165
0.3935
0.6847
0.0844
0.0062
86.3864
7*-VAR
0.7021
0.7274
0.6777
0.0321
0.0009
17.3992
3-VAE
0.5121
0.5319
0.4934
0.0420
0.0020
18.3127
3-VAE 128
0.1219
0.1739
0.0854
0.0070
0.0023
20.0576
DIP-VAE-II 10
0.0838
0.2211
0.0329
0.0867
0.0046
186.8153
3dShapes
DIP-VAE-II 128
0.1052
0.1799
0.0617
0.0185
0.0115
267.8511
Factor- VAE 10
0.1759
0,2195
0.1418
0.0738
0.3748
139.5286
Factor-VAr 128
0.1884
0.2166
0.1641
0.0236
0.0024
62,3407
NAE
0.7112
0.8301
0.6140
0.0731
0.0009
22.2525
3-VAE 10
0.1962
0.2917
0.1321
0,0897
0.1152
393.0866
3-VAE 128
0.1447
0.2469
0.0848
0.0709
0.1153
284.6031
DIP-VAE-I1 10
0.2090
0.3979
0.1106
0.0789
0.0629
259.0138
Cars3D
DIP-VAE VAE-II 128
0.1226
0.2261
0.0665
0.0492
0.0206
191.8011
Factor- 10
0.2551
0.3800
0.1729
0.0961
0.0817
274.608 674.6059
Factor-VAr VAE 128
0.3630
0.6047
0.2351
0.0892
0.0738
315.6645
V*E
0.4623
0.5984
0.3571
0.0783
0.0465
242,2127
3-VAE 10
0.2047
0.2600
0.1613
0.0843
0.01
77.3847
3-VAR 128
0.1160
0.1837
0.0733
0.0548
0.0125
77.4040
DIP-VAF-II 10
0.1886
0.2673
0.1335
0.0897
0.0209
134.6060
dSprites
DIP-VAE-II 128
0.1065
0.2237
0,0509
0,0645
0.0218
168.3634
Factor-VAr 10
0.3020
0.2619
0.3908
0.0932
0.1529
153.1589
Factor- VAE 128
0.1654
0.2373
0.1154
0.0571
0.0419
101.6020
VAE
0.3396
0.3681
0.3161
0.0767
0.0066
65.7015
8

---

Published as a conference paper at ICLR 2022
5 DISCUSSION
We proposed and analyzed a low entangled latent representation of the generating factors of a distribution. The latent space has the topology of a torus which is direct product of circles. Each circle controls one generative factor, and moving along the circle changes the corresponding aspect of the distribution, Our latent space construction resembles the phase space structure of classical, integrable physical system, whose dynamics can be described using action-angle canonical variables as motion on a phase space torus. The dimension of the torus is the number of the system's 's degrees of freedom, and each point on it specifies the location in space and the velocity of the system at given instant in time. The coordinates of the point on each circle correspond a particular periodic cycle to which the whole motion has been decomposed. Analogously, the coordinates on each circle of a point in the latent space torus specify one of the basic properties of an element of the dataset to which the description of that element has been decomposed. The possible intriguing relationship between the integrable structure dynamical systems and its decomposition into its basic harmonic motions and the decomposition of the dataset into its basic constituents can be valuable in relating dynamical systems to the process of learning.
The task of unsupervised disentanglement learning is complicated by the fact that given any dis- entanglement representation there is an infinite family of entangled representations that have the same marginal distribution (Locatello et al.) 2019). Therefore, an inductive bias needed for the task. Our solution in this work is based on the concept of entanglement entropy in quantum physics. There are non- con-generic states in the Hilbert space that have low entanglement, and special class of them are the product states, whose entanglement entropies are identically zero. We employed these states for our disentanglement representation.
indeed easy to see that these provide a special representation, since any random unitary (or orthogonal) matrix acting them will make our representation entangled and hence destroy the disentanglement properties of our solution. Thus, in a sense, inductive bias is the use of the special non-generic states (of measure zero) in the Hilbert space for our representation.
We introduced DC-score to jointly quantify the disentanglement and completeness of the torus latent representation and calculated it, as well as various other measures in a large set of numerical experiments. For the comparison we used five different types of datasets. We varied the dimension of the latent space torus and showed a clear advantage of the torus latent description compared to others when the torus dimension is equal or greater than the number of generating factors. In order to properly assess the cases where the number of expressive codes is lower than the number of generative factors, we introduced correction to the disentanglement score, and hence our DC-score.
One limitation of our proposed low entangled representation is in scaling to datasets containing a large number of generative factors. The dimensionality of increases exponentially with the dimension of the torus latent representation and is, therefore. limited in describing these situations. This can perhaps be remedied by a different choice of embedding, egg. working with a direct sum of tensor products, which increases polynomially. Another limitation of the model may be in its ability to effectively describe property of the dataset that is not naturally encoded cirele, but is, rather, non-compact in nature.
While our torus latent representation performs better compared to other approaches, one may still wonder why it did not achieve the perfect DC-score 1. This can be explained by the fact that the generation of the datasets assumes a certain number of generating factors, while in fact there could be more, some indirectly combined. For example, in a case where multiple underlying factors are combined into one explanatory factor, the disentanglement may overshoot, thereby reducing the obtained score.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
All the code for reproducing all experiments presented in this work is available in the supplement.
ACKNOWLEDGMENTS
This project has received funding from the European Research Council (ERC) under the European Unions Horizon 2020 research and innovation program (grant ERC CoG 725974). The work of Y.O. supported in part by the Israeli Science Foundation Center of Excellence. The contribution of the first author is part of a Ph.D. thesis research conducted at TAU.
REFERENCES
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://gssiibbooooeeemmmml//ssss 2018.
Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representations. arXiv preprint (408.099000990 2019.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. International Conference on Learning Representations, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. arXiv preprint 1Ef::1700608500 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2016.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on PMLR, 2018.
Machine Learning, pp.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6441, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rXiv:1312.6114, 2013.
Mark Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233-22 243, 1991.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled
latent concepts from unlabeled observations. arXiv preprint arXiv: 2017.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing disentangled representations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings the 36th International Conference Machine Learning, volume 97 of Pro- ceedings of Machine Learning Research, pp. 4114-4124. PMLR, 09-15 Jun 2019. URL http:://proceedingg .wl...r....... www.::sss...p.p..ppm...mmme www./1//oaooll1laaaa..... html
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement http:///w..o/deeem////pptttteeee
testing sprites dataset.
2017.
Pol Moreno, Christopher KI Williams, Charlie Nash, and Pushmeet Kohli. Overcoming occlusion with inverse graphics. In European Conference Computer Vision, pp. 170-185. Springer, 2016.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee, Deep visual maalogy-making Advances in neural information processing systems, 28: 18:1222-126 2015.
10

---

Published as a conference paper at ICLR 2022
Maximilian Seitzer. pytorch-fide -aoreh-fidd:aa FID Score for PyTorch. http::///gtthub..cm//msiiiiie
pytorch-fidd fid, August 2020. Version 0.1.1.
A.K Subramanian. Pytorch-vaee httppp/s/ggtttub....m///tttttttp
2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Julian Zaidi, Jonathan Boilard, Ghyslain Gagnon, and Marc-André Carbonneau. Measuring disentan- glement: A review of metrics. arXiv preprint arXiv: 2020..
A GENERATIVE FACTORS HEATMAPS
In this section we provide additional heatmap examples for different tori which provide qualitative insight. Figures 5 and 6 present the heatmaps for our method on T and 7 torus topologies, respectively, with their D, C and DC scores.
When the number of generative factors larger than the number of torus dimensions, D, one can expect some mixing between the generative factors and the latent codes. This can be observed for instance for 7 in Fig. 5
As the dimension of the latent codes increases, see Fig. 6 the D, C and DC-score metrics improve. This effect on the disentanglement results from the prefactor rank(R) in equation which depends on the ratio of the numbers of latent codes and generative factors when the latter is larger.
Figure 5: Heatmap of the generative factors the codes for the teapot dataset with architecture with four circles (T*). The disentanglement, completeness and DC-scores are 0.36, 0,43 and 0.39, respectively. One can observe, for example that the second code 0, controls all three RGB values.
B ON THE CONNECTION TO QUANTUM PHYSICS
The assumption there is set of elementary generative factors 21, i 1, n that are disentangled from each other, and from which the data distribution can be constructed. The task is to learn these elementary factors and represent them by latent codes Ca, a = 1,..., m. Ideally we would have liked to have and bijective map ff that associates any C. one and vice versa.
We are working with a linear representation and since quantum mechanics a linear theory it inspires choice of the latent space representation. In quantum mechanics states represented by vectors Hilbert space and the operators that act on them are linear. In our case, the generative factors constitute state and do the latent codes. The linear regressor relating them is linear operator.
We represent each elementary factor 2; as state (qubit) in a Hilbert space (2,) F and the complete set of elementary factors by state (quantum register of qubits) in the tensor product of these

---

Published as a conference paper at ICLR 2022
-25
-25
60
00
-2.
2.5
-25
-25
96
-25
2.5
Figure 6: Heatmap of the generative factors vs. the codes for the teapot dataset with our architecture with eight circles (I*). The disentanglement, completeness and DC-scores are 0.69, 0.61 and 0.65, respectively.
Hilbert spaces (21,..., =n) 121) ...1=,) n-qubit states can represented in such a way as a product state and the reason that we can do that the assumption that are disentangled.
F .... F. Note, that only measure zero of E E
Entanglement can viewed as measure of how far the state is from being product state. One way to measure it is by taking the supremum the absolute value of the inner product of the state with all the possible product states. Another measure is the Von-Neumann entropy that is calculated by constructing density matrix p as the outer product of [=ı, Zn), decomposing z; to e.g. two subsets A and B such that E H & HB and tracing out part B, p = Trap. The eigenvalues of & encode the entanglement between and B. Specifically, the Von-Neumann entropy is Syx -Trp log ' and it vanishes iff the state [21, 2n.) is a product state.
Returning our case, construct the latent space representation using tensor product since should capture the above structure of the generative factors. The measure of disentanglement D. quantifies by the Plog P term which is the analog of the Von-Neumann entropy.
Note that there is a difference between our case and quantum mechanics: in quantum mechanics an elementary qubit state represented by two angles (0, σ) while in our case and elementary latent code is one angle parametrizing a circle of the torus.
One may inquire what would have gone wrong our analysis if chose that space of latent codes to be another curved compact manifold such as sphere. In such a case we should have covered the manifold with local patches such that there would a linear map between the generative factors and the latent codes on each patch and transition function between them. Such representation would be entangled and will fail, e.g. on the sphere near the pole since rotating by the azimuthal angle would not change any property.
12


Reviews:
[{"review_id": "qNBSntJIpj", "reviewer": "Reviewer_4Dgs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This study provides interesting results when using the torus as the representation space. I think the idea is novel. The utility is evaluated 5 datasets, and the results are convincing. I enjoyed reading this paper. \n\nThe main drawback of this paper is the lack of theoretical analysis. While the motivation of introducing the torus is based on theory, the property of the proposed latent representation (5) is not analyzed. For example, could you provide a lower bound of DC score when we employ (5)?\n\nI'm puzzled about the connection between this study and quantum physics. I'm not an expert of quantum physics, and I have no idea what are entanglement property, Von Neumann entropy, and so on. Please elaborate on these concepts and how they are related to the torus representation. I believe this revision significantly enhances the manuscript.\n\nMinor comments:\n- Typo: n in (5) seems to be D.\n- The sentence below (8): the index (a, \\alpha) should be an integer, but why they are distributed from the Gaussian?", "summary_of_the_paper": "The authors propose an autoencoder where the latent space is defined on a torus. A D-dimensional torus is represented by the tensor product of D unit circles. They argue the torus induces disentanglement as the analogous of entropy entanglement in quantum physics. Using 5 datasets, they show both the reconstruction performance and disentanglement score (Eastwood & Williams, 2018) are high.", "summary_of_the_review": "The concept of the torus representation is novel and its practical value is empirically convincing. However, there is no assessment in terms of theory. The connection to quantum physics is not well explained. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TF5GC7r3WU8", "reviewer": "Reviewer_DucY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: I enjoyed this paper. It is well written and tackles an important problem of disentanglement of latent factors in VAEs. The method of decomposing onto a set of circles that form a D-dimensional torus makes sense and is clearly described. The empirical tests are thorough, presented honestly, and support the theoretical intuition both qualitatively and quantitatively.  The figures are well done and aid in the flow. Figures 2 and 3 certainly demonstrate the effectiveness of the method!\n\nWeaknesses: I struggled to find any major problems with this paper. My main unresolved questions about the method are: (1) how does it scale with the dimensions of the tori and (2) how it preforms when a factor is not well represented as a circle. These are addressed by the authors in the conclusion. I do not think (1) is a major weakness as I think the simplicity of the idea offsets a problem which may not arise in practice, this method will work well when the data are well explained by a few latent factors. Question (2) is central to the method, but I also think it raises a qualm that is beyond the scope of this paper. Since these were my main questions, I would like to see the heatmaps as in Figure 2 for the other experiments described in the “Dependence on beta and on D” section, but these could be added to an appendix. The reason is I would like to see the behaviour as D is varied as this would reveal more clearly how the method handles over and underspecification of the latent space. \n\nI would also like to see a deeper investigation of the proposed DC-score metric, but, as is, it makes sense and is effective. Any further development of this element of the paper is likely beyond the scope.\n\nMinor quibbles: \n1) Equation (9) could be rewritten so it is less cramped and easier to parse.\n2) The terms alpha_k in the same section only take on values 0 and 1, otherwise the result would not be a torus. Do the authors intend for this to be a future modelling choice point? If not, I think the notation could be cleaned up to improve clarity. \n\n", "summary_of_the_paper": "The paper presents a novel method for disentangling latent representations, specifically for use with variational auto-encoders. This is accomplished by treating the latent codes as a tensor product of 1-spheres which results in their forming an n-dimensional torus where n is the number of latent factors. Additionally, the authors propose a new metric for analysing the quality of the decompositions. These methods are well founded and make intuitive sense, and the authors supplement this theoretical motivation with an array of tests comparing to several existing methods on several different datasets. The results demonstrate that the methods work well both qualitatively and quantitatively.", "summary_of_the_review": "I think the paper is well composed, interesting, and convincing. Its methodology is well founded and is clearly communicated, and the empirical results strongly support method. I think it also opens many important directions for subsequent research. For these reasons, barring something major that I have overlooked, I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S7e1al6EWZb", "reviewer": "Reviewer_YGh8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n - Method is simple yet innovative\n - The intuition behind the method is explained\n - Integrates very well with previous frameworks in the literature\n - Evaluation is convincing, including a direct demonstration of a strong linear match between ground truth and learned representations\n - Discussion of results satisfactorily accounts for why the method performs relatively worse in one dataset\n - The discussion frankly acknowledges a limitation of the method in terms of dealing with large numbers of latent causes\n\nWeaknesses\n - Evaluation is based on artificial datasets, similar to the rest of the literature.  An attempt to use the method on real data, even if inconclusive, would be informative, especially since one of the limitations of the method is in dealing with large number of latent causes (which would presumably be much higher in real data than for simulations).\n - The explanations of the success of the method in terms of quantum physics concepts fall short of giving actual insight into why the method works.  Why are the physics analogies appropriate in this problem, if they are?\n - It is unclear whether the torus topology is as crucial as claimed.  If we ignore the circular structure, the method greatly resembles basis expansion methods used in linear models with higher-order interactions (https://en.wikipedia.org/wiki/Interaction_(statistics)).  Is nonlinear basis expansion the real secret ingredient to making this approach work, or does the torus topology (transforming angles into points on a 2d circle) play an additional role?\n - Is Lasso appropriate for evaluating toroidal representations, given the possibility of rotation?\n - How were the tuning parameters (beta, etc.) tuned?\n - I am baffled by how the learned factors can represent z2, z3, and z4 linearly in Figure 2, given the rotational invariance inherent in the toroidal representation.  Is it because of R, G, B are mostly represented in the linear part of the basis expansion v_orient?\n\nMinor comments\n - The convention (as well as the rules of English) seems to have \"autoencoder\" as a single word, not two, but I could be mistaken.\n - In eq. 9, \"N(0,1)\" should be \"N(0, sigma)\"?", "summary_of_the_paper": "This paper proposes a new autoencoder architecture that achieves better recovery of the latent factor structure in artificial datasets according to previously established metrics: disentanglement, completeness, and informativeness.  The key idea is to nonlinearly project a small number of latent factors into a higher-dimensional space, based on the topology of the torus, in such a way that prevents an arbitrary linear rotation of the latent factors from yielding an equivalent representation.  The approach is compared against existing approaches for variational autoencoders focused on learning disentangled representation, and achieves higher disentanglement and completeness, as well as competitive or best informativeness (reconstruction) on a majority of datasets considered.", "summary_of_the_review": "This is an innovative method that looks very promising given the limitations of the evaluation on artificial data.  The paper is not perfect, and could improve in being more critical of the approach proposed, as well as reproducibility in terms of describing the evaluation procedure.  But the results clearly show an advantage over previous approaches on a variety of artificial datasets, which is technically impressive even if the practical impact of the method on real data is still unknown.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1zsnWT48MZ", "reviewer": "Reviewer_UzjD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe method proposed is simple and looks easy to reimplement.\n\nWeaknesses:\n1. The motivation 'entropy of entanglement' is not clearly explained. Considering this is not a commonly used concept in the disentangled representation learning community, I would suggest the authors provide more explanations on it. This may include (1) an intuitive descriptive introduction about what 'entropy of entanglement' is; (2) a detailed explanation on its relation to the disentangled representation learning; (3) a few experiments conducted on the disentanglement learning datasets to validate and demonstrate the claimed connections to 'entropy of entanglement'; (4) why it can solve the unidentifiability problem (Locatello et al., 2019); (5) how this 'entropy of entanglement' motivates the use of the tensor product of n unit circles as the representation.\n2. The 4-th paragraph in the Sec. introduction says the n-torus $T^n$ has a low entanglement property. I wonder do the commonly used n-dim vector representations have low or high entanglement property? Which type is higher? How to compare these two types of representations quantitatively by 'entanglement property'?\n3. I can see there are two steps of new techniques incorporated in the proposed method. One is the usage of multiple cyclic representations with each circle indexed by two 1-dim latent variables (defined by Eq. (6)(7)(8)). The other one is the flattening operation defined by the tensor product $\\otimes$ in Eq. (5). Can the authors explain why they are motivated individually? More specifically, (1) why n unit circles are better than traditional vectors and (2) why we must use the tensor product to forward them to the decoder?\n4. As the authors have noticed by themselves, the proposed tensor product method cannot generalize to a large number of latent dimensions because the dimensionality of the representation increases exponentially with the torus dimension. I think the authors should provide a solution to this obvious problem in this paper with experimental support.\n5. No ablation study on the number of torus dimensions is provided in the experiment section.\n6. Experiments on some complex datasets like CelebA should be provided to show the generalization ability of this method.", "summary_of_the_paper": "This paper proposes to use the tensor product of n unit circles to learn the disentangled representation. Experiments on synthetic datasets show the effectiveness of this technique.", "summary_of_the_review": "Based on my concerns, I think this paper requires some significant improvements and I rate it as rejected for now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers had a positive impression of this paper, two of them were willing to champion it. The main positive aspects mentioned by these reviewers were clarity, methodological strength, novelty and convincing experimental evaluation. On the other hand, the was one clearly negative vote, raising issues about the proposed concept of 'entropy of entanglement' and about the use of tensor products. It seems that after the rebuttal, this reviewer was still not fully convinced. In my opinion, however, the rebuttal addressed most of these points of criticism in a clear and transparent way, so I recommend acceptance.
Decision: Accept (Poster)