Year: 2021

Abstract:
The current methods for learning representations with auto-encoders almost exclusively employ vectors as the latent representations. In this work, we propose to employ a tensor product structure for this purpose. This way, the obtained representations are naturally disentangled. In contrast to the conventional variations methods, which are targeted toward normally distributed features, the latent space in our representation is distributed uniformly over a set of unit circles. We argue that the torus structure of the latent space captures the generative factors effectively. We employ recent tools for measuring unsupervised disentanglement, and in an extensive set of experiments demonstrate the advantage of our method in terms of disentanglement, completeness, and informativeness. The code for our proposed method is available as a supplementary material.

heading: 1 INTRODUCTION
text: Unsupervised learning methods of disentangled representations attempt to recover the explanatory factors z of variation in the data. The recovered representation, c, is expected to be: (i) disentangled, i.e., each element in c should vary one of the generative factors in z (ii) complete in the sense that all generative factors z can be controlled by the latent representation c, and (iii) informative, that is the ability to relate via a low capacity model (e.g., a linear model) the generative factors z and the latent representation c.
It has been shown that solving this task without further assumptions is impossible, since there is an infinite number of bijective functions f that map between c and z = f (c) (Locatello et al., 2019) with all corresponding generative models having the same marginal distributions of the observations. In this work, we propose to use representations that are a tensor product of elements, which take values on unit circles S1, claiming that this structure is suitable for the effective recovery of the underlying modes of variation.
This claim follows from entropy considerations. One wishes to have a representation that has a low entropy of entanglement. The entropy of entanglement is measured by evaluating the number of non-zero eigenvalues of the Schmidt decomposition of the representation. If a representation can be described by only one term, which is an outer product of the orthogonal basis functions, it has the lowest possible entanglement entropy, zero. If a representation requires the use of more than one term in the decomposition, it is entangled and its entanglement entropy can be quantified, e.g. by the Von Neumann entropy constructed from non-zero eigenvalues.
The tensor product of n unit circles, S1, takes the shape of an n-torus Tn = (S1)n, and has a low entanglement property. This representation has the advantage that it can capture the periodicity structure of the generative factors, where each circle controls a different aspect of the original distribution. Unlike other generative models, which rely on the Gaussian distribution as a prior, Tn is a compact manifold, and therefore any function that acts on it, such as a decoder, has to interpolate only, but not extrapolate when generating new instances.
In this work, we present the TD-VAE, a Variational Auto Encoder whose latent space resides on the TD torus manifold. In an extensive set of experiments with four different datasets, we compare the torus latent representations with others proposed in the literature. We show a clear advantage of the torus representation in terms of various measures of disentanglement, completeness, informativeness and propose a new metric, the DC-score, which assesses the combined disentanglement and com-
pleteness performance. We present a detailed quantitative and qualitative analysis that supports our method.

heading: 2 RELATED WORK
text: Generative models aim to create new instances that are indistinguishable from a given distribution. Auto encoders (Kramer, 1991) achieve this by constructing the identity function using a composition of two components, an encoder and a decoder. The encoder is tasked with compressing the input into a latent representation with a dimension smaller than the input’s, whereas the decoder is responsible for reconstructing the original input from the latent space.
Such auto encoders usually overfit to their training data and are very limited when tasked with generating new samples. Instead of mapping each input instance to a constant vector, Variational Auto Encoders (Kingma & Welling, 2013) (VAE) map each instance to a pre-defined distribution, by minimizing both the encoder-decoder reconstruction loss and a KL-divergence term that depends on the prior distribution. VAE’s are capable of generating new instances by sampling a latent vector from the predefined distribution; however, the interpretation of latent space components is typically obscure and the mapping between them and the dataset properties may be complex.
In order to increase the latent space interoperability, various modifications of the original VAE were introduced. These variants try to achieve a factorization of the latent space components with each component corresponding to a specific simple characteristic of the dataset. In β-VAE (Higgins et al., 2016), the weight of the KL-divergence term is increased w.r.t. the reconstruction term, which results in better factorization of the latent space. However, this introduces a new hyper-parameter and departs from the theoretical derivation of the ELBO term. For example, large β parameters may prevent the conditional distribution from modeling the data faithfully. DIP-VAE (Kumar et al., 2017) introduces a regularizer that constrains the covariance matrix of the posterior distribution. This term is added to the original VAE loss function, along with two hyper-parameters, and helps achieve better factorization. Factor-VAE (Kim & Mnih, 2018) adds a Total Correlation (TC) approximation penalty over the produced latent codes and introduces an additional hyper-parameter.

heading: 2.1 DISENTANGLEMENT METRICS
text: In order to evaluate the expressiveness of different generative models, a plethora of disentanglement definitions and metrics were suggested in the literature (Do & Tran, 2019; Eastwood & Williams, 2018), see Locatello et al. (2019) for a comprehensive review. In this work, we adopt the definitions and metrics introduced in Eastwood & Williams (2018) which provides a successful set of disentanglement metrics.
The authors of Eastwood & Williams (2018) introduce three metrics, Disentanglement, Completeness and Informativeness (DCI), to quantify the disentanglement properties of the latent space and its ability to characterize the generative factors of a dataset. Given a dataset that was generated using a set of K generative factors, ~z ∈ RK , and that it is required to learn latent codes, ~c ∈ RD of D-dimensional representation. Ideally, in an interpretable representation, each generative factor zi would correspond to only one latent code ca. It is also beneficial if the mapping is linear, as the correlation between generative factors and codes can then be easily assessed. Generating new instances that are indistinguishable from the original dataset distribution requires that the latent codes cover the whole range of the generative factors. Furthermore, such a representation provides the ability to modify a specific property of a generated instance by directly tuning the corresponding latent code.
The DCI metrics aim to quantify the relationship between codes and generating factors by having a single number that characterizes the relative importance of each code,1 ca, in predicting a factor, zi, which in turn defines an importance matrix Rai. To construct the importance matrix, K regressors are trained to find a mapping between zi and ~c, ẑi = fi(~c). In this work, we follow Eastwood & Williams (2018) and infer the importance matrix using a lasso linear regressor’s weights, Wia, by Ria = |Wia|.
1throughout the paper we use a, b, c, . . . and i, j, k, . . . letters for indices of the codes ~c and the factors ~z components respectively.
Once the importance matrix Rai is obtained, the DCI metrics can be defined explicitly. The disentanglement is given by
D = rank (R) K
D∑
a=1
ρaDa , (1)
where
Da = 1−HK(Pa) = 1 + K∑
k=1 Pak logK Pak, Paj = Raj∑K k=1Rak , ρa = ∑ j Raj∑ bk Rbk . (2)
High disentanglement means that each entry of the latent vector ~c corresponds to only one element in ~z (in a linear sense), that is, each code element, ca, affects one generating factor. The disentanglement metric defined above differs from Eastwood & Williams (2018) by a correction factor rank(R)K . When the rank of R is equal to the generative factor’s dimension, this correction equals 1, and does not affect the metric. However, it does make a difference when the number of codes is smaller than what is needed to account for all generating factors. Typically, one assumes that there are at least as many expressive codes as the number of factors. When this is not the case, it means that even if we have disentanglement among the expressive codes, their number is not sufficient, and our correction accounts for this. Note that the correction has no influence if there are irrelevant code elements in addition to a sufficient number of expressive ones. The ρ factors handle cases where some of the code dependence on the factors is very weak (namely irrelevant codes), while other codes depend mainly on one factor and hence should not be of equal importance.
The completeness is defined by
C = 1 K
K∑
j=1
Cj , Cj = 1−HD(P̃j) = 1 + D∑
a=1 P̃aj logD P̃aj , P̃aj = Raj∑D b=1Rbj . (3)
In contrast to the disentanglement metric, which considers a weighted mean over the generative factors, motivated by the fact that irrelevant units in ~c should be ignored, here all codes are treated equally. A situation in which each factor zi is explained by only one element of the code ca results in high completeness.
The informativeness is the MSE between the ground-truth factors, z, and the predicted values, fj (~c),
I = 1 K
K∑
j=1
Edataset [ |zj − fj (~c)|2 ] . (4)
Disentanglement and completeness values range between zero and one, where one is the best value, while the best value for informativeness is zero.
Disentanglement and completeness alone do not provide sufficient information on whether a representation factorizes properly. For example, consider the case of a representation where only one generative factor is described by all the codes. While this representation is completely disentangled, it is not complete, and is therefore meaningless. In order to have a meaningful representation, both disentanglement and completeness need to be high. Thus, we introduce a new score, called the DC-score, which accounts for both disentanglement and completeness. We define the DC-score as the geometric mean of the two metrics, DC-score = √ DC. This way, only cases where both scores are high will result in a high DC-score. Furthermore, the score will favor cases where both disentanglement and completeness are comparable rather than having different values while having the same arithmetic mean.

heading: 3 METHOD
text: Assume a pre-determined set of vectors sampled from an unknown distribution χ, each vector containing K independent factors, ~z ∈ RK . The training dataset contains samples xI = F (~zI), I = 1, . . . , N that are generated using a generative function F . In unsupervised disentanglement, the learner has no access to ~z nor to F , and receives only the set of samples, {xI}NI=1.
The learner has to recover a set of representation vectors ~cI that are linked to ~zI in a way that is bijective and disentangled, see Section 2.1. Furthermore, in order to generate new samples, it is also necessary to be able to sample new representation vectors ~cnew, and to obtain a generative function G such that the new generated samples are from the same distribution of F (~z), where ~z ∼ χ. In this work, we view the latent representation vector ~c as the angles associated with a list of D two-dimensional unit vectors ma ∈ R2, 1 ≤ a ≤ D, i.e., ∀a, ‖ma‖ = 1. Using the two-dimensional vectors we define
vprod ≡ vec ( vα1...αDprod ) , vα1...αDprod = m α1 1 ⊗ · · · ⊗mαDD , vorient ≡ ( m01, . . . ,m 0 D ) , (5)
where ⊗ is the outer product operator, αa ∈ {0, 1} and vec is the vectorization operation (equivalent to flattening the tensor). Next, define the operator V , that given m1, . . . ,mD, concatenates vprod together with vorient, V (m1, . . . ,mD) = [vprod; vorient]. The vector v = V (m1, . . . ,mD) resides in a vector subspace of R2D+D, defined by only a set of D parameters. The additional D elements of vorient are required to ensure that the mapping V is bijective. For example, this can be seen by examining the case of D = 2: let m1 ≡ (cos θ1, sin θ1), m2 ≡ (cos θ2, sin θ2), m′1 ≡ −m1 and m′2 ≡ −m2. Then m1 ⊗m2 = m′1 ⊗m′2. A natural way of acquiring a random point on the circle, S1, is by sampling two independent Gaussians, m̂αka ∼ N (µαka , σαka ). Each tuple of these vectors is then normalized to have a unit norm,
mαka = m̂αka√
(m̂0a) 2 + (m̂1a)
2 . (6)
Assume that the elements, m̂αka , follow the normal distribution with a zero mean and a standard deviation of 1, m̂αka ∼ N (0, 1), then the vectors ma follow the uniform distribution on S1. For the purpose of obtaining the distribution parameters of m̂αka , the encoder e is applied on an instance, x,
e (x) = [ µ01 µ11 , σ01 σ11 . . . µ0D µ1D , σ0D σ1D ] . (7)
The reparametization trick (Kingma & Welling, 2013) is then used to obtain a set of normally distributed vectors. Denote by S the sampling operator, we sample the coding vector as
~MI ≡ {(
m01 m11
) , . . . , ( m0D m1D )} = S(e(xI)) , (8)
where each component, (a, αa) of m, a = 1 . . . D, α = 0, 1 is sampled from N (µαa ,σαa ). The decoder G then acts on V (m1, . . . ,mD) to generate a new instance, x̃. The requirement that the generated instance is indistinguishable from the original distribution F (χ) translates to minimizing the ELBO loss function. First, it maximizes the log-likelihood probability of x̃ to be similar to the training sample it reconstructs by having an l2 loss term. Second, to encourage a uniform distribution on the torus, it contains a KL-divergence term with respect to N (0,1D). The overall expression for the loss function used during the optimization of the networks e and G is
L = ∑
I
[∣∣∣G ( V ( ~MI )) − xI ∣∣∣ 2 − βDKL (p (m̂) ||rI) ] , (9)
where β is a hyperparameter introduced in β-VAE (Higgins et al., 2016) and rI ∼ N (0,1D). As the latent representation resides on the torus, sub-vectors 2D components of ma may be described by D angles, θa ∈ [0, 2π] (the angles θa are identified with the codes ca). In order to generate new instances, identify, m0a = cos θa , m 1 a = sin θa . and apply the decoder G (V (m)) to these elements.
The torus topology, TD, is both compact and periodic. Since it is compact, for every sampled θa, there is a nearby θ′a from the training set, thus, the network G can be viewed as an interpolator. This is in contrast to vectors sampled over RD, in which G has to extrapolate. Furthermore, being periodic, this topology is able to exploit a periodic structure in the generative factors. Consider a common generative factor associated with rotations. A rotation cannot be represented by a single non-compact variable; therefore encoding a rotation in a latent space requires the entanglement of two components. However, on the torus, only one compact dimension can be used to identify this generative factor.

heading: 4 EXPERIMENTS
text: We compare our method to the leading VAE architectures from the current literature. The baseline methods are (i) β-VAE (Higgins et al., 2016), (ii) DIP-VAE-II (Kumar et al., 2017), and (iii) Factor-VAE (Kim & Mnih, 2018). The code for all methods was taken from the Pytorch-VAE repository (Subramanian, 2020) (Apache License 2.0). The network architectures for both the encoder, decoder and the Factor-VAE’s discriminator (which we do not use) follow those in Eastwood & Williams (2018), see Table 1. Each horizontal line in Table 1 denotes a skip-connection.
All methods were trained using ADAM (Kingma & Ba, 2014) with a learning rate of 0.0001 and a batch size of 144. The hyperparameters used for DIP-VAE were λdiag = 10 and λoff-diag = 1.0. All methods were trained for a maximum number of 50 epochs, and the metrics were evaluated using the model that performs best on the validation set with respect to MSE. Furthermore, each baseline method was evaluated using two latent representations sizes, D = 10, 128 for all datasets. For our method, T 8-VAE, we use 8 circles, ( S1 )8
. This results in V ∈ R136, leaving the number of parameters in the decoder comparable to the other baseline methods. Each experiment was evaluated with five different seeds and the mean value is reported.
In order to compute the DCI metrics, we used a linear lasso regressor based on the implementation of Eastwood & Williams (2018) (MIT License) and Zaidi et al. (2020) (Apache License 2.0). For each generative factor, a lasso regressor fits between the normalized codes obtained by the encoder, e, to a normalized factor, namely fi(~c) for each zi. For each factor, the α parameter for the regressor using 10-fold cross-validation from the set { 10−6, 10−5, 10−4, 10−3, 10−2, 0.1, 0.2, 0.4, 0.8, 1 } , picking the one with the minimal the MSE error. The weights of the lasso regressor are then used to construct the W ∈ RK×D matrix from which the metrics are computed as explained in section 2.1.
The Fréchet Inception Distance (Heusel et al., 2017) (FID) is used to assess the quality of image generation. Specifically, we use the implementation of Seitzer (2020) (Apache License 2.0) and report the FID at the final average pooling features of the Inception3 NN (Szegedy et al., 2016).
The teapots dataset (Moreno et al., 2016) contains 200, 000 64 × 64 images of camera-centered teapots rendered using a set of 5 generative factors, azimuth that is sampled from ∼ U [0, 2π), elevation that is sampled from ∼ U [ 0, π2 ) and three RGB channels, each sampled from U [0, 1]. All VAEs are trained over a subset of 160, 000 (the generative factors are unknown to all architectures), whereas the disentanglement metrics are evaluated on the remaining 40, 000 images.
The Cars3D dataset (Reed et al., 2015) contains 183 CAD models of different cars projected into 64× 64 image with four elevations and 24 azimuths. These three generative factors are split between a training set of 14054 images and a validation set of 3514 images.
The 2dshapes Dataset, introduced here and released to the public (MIT License), contains 200, 000 64 × 64 images of four possible shapes, triangles, squares, pentagons and hexagons. All shapes in the dataset are centered in the image and have a white background. The factors controlling the images are the following: specific shape drawn from a discrete uniform distribution, a scale value between s ∼ U [20, 40], rotation angle θ ∼ U [0, 2π], and color RGB values r ∼ U [0, 1], g ∼ U [0, 1], b ∼ U [0, 1] all sampled uniformly. Since the shapes are symmetric, different angles could correspond to the same image. The training set contains a subset of 160, 000 instances, whereas the disentanglement metrics are evaluated on the remaining 40, 000 validation instances.
The 3dshapes dataset (Burgess & Kim, 2018) contains 480, 000 64 × 64 images of four possible objects, a cylinder, a tube, a sphere and a box rendered with 8 linearly spaced scale values, 10 hues linearly spaced, floor wall and object hue values and fifteen different orientations, which amounts to six generative factors.
The dSprites Dataset (Matthey et al., 2017) contains 737, 280 64×64 binary images of three shapes, a square, an ellipse and a heart with six different, equally spaced scale values, 40 different orientations, and 32× 32 possible positions in the plane, amounting to five generative factors. 589, 824 instances were used for training and evaluation was performed on the remaining 147, 456 validation instances.
Results Across all the datasets (Tables 2–6), our T 8-VAE architecture obtains the highest DC-score, which combines the disentanglement and completeness scores, compared with the baselines. In all of these cases the generated images are meaningful, as can be seen from the relatively low FID scores. Furthermore, for teapots (Table 2), 2dshapes (Table 3) and dSprites (Table 6), we also achieve the best reconstruction MSE and FID scores. In the 3dshapes dataset (Table 4) we get the best MSE score and comparable FID scores to β-VAE, which gets the highest score.
For the Cars3D dataset (Table 5) the results are quite different, even though our DC-score is highest. The best performing method in terms of the MSE and FID scores is DIP-VAE-II 128, which is not among the best performing methods for the other datasets. One plausible reason is that this dataset,
although having only three generative factors, hides a much richer high-dimensional distribution. The factors controlling the car model (183 in total) implicitly contain many characteristics of a car, which requires a high dimensional latent space. Nonetheless, our method achieves the second best FID score for this dataset.
Dependence on β and on D In order to examine the sensitivity of our method to different tori dimensions, as well as the influence of the β parameter on KL-divergence, we examined the DC-score and reconstruction error on the Teapot dataset, using four topologies, T 4, T 5, T 6 and T 8, and using five different β coefficients, 0, 1, 3, 6, 9 (0 corresponds to no KL-divergence). As can be seen in Fig. 4, removing the KL-divergence completely, leads to a decrease in the reconstruction error; however, the DC-score also decreases. The T 4-VAE exhibits the worst reconstruction error - not surprisingly, as it can only encode four generative factors, whereas there are five generative factors in the dataset. Both the T 5-VAE, T 6-VAE and T 8-VAE have the capacity of encoding all generative factors, however, the encoder of the T 8-VAE should ignore three of the circles whereas the encoders of the T 5-VAE and T 6-VAE should ignore at most one circle. Thus, when increasing β, the induced angles on T 8 tend to a more uniform distribution, which entangles between themselves. This does not happen in T 5-VAE and T 6-VAE, since there is at most one extra component that can be entangled.
Heatmap analysis Disentanglement results can be qualitatively evaluated through a visual inspection of the 2d heatmaps of the generative factors zi vs. the codes ca (the angles θa in our case). In Figure 2 we show an example of the heatmap results for using our method with D = 6, namely using a T 6-VAE. The disentanglement and completeness properties are clearly seen in the z1, z2, z3, z4 dependence on θ5, θ2, θ0, θ1 respectively, while we see worse completeness for the z0 factor. Such images help us better understand cases where one of the scores is high and the other is low, or whether a specific regressor is suited for capturing the functional dependence.

heading: 5 DISCUSSION
text: We proposed and analyzed a low entangled latent representation of the generating factors of a distribution. The latent space has the topology of a torus which is a direct product of circles. Each
circle controls one generative factor, and moving along the circle changes the corresponding aspect of the distribution. Our latent space construction resembles the phase space structure of a classical, integrable physical system, whose dynamics can be described using action-angle canonical variables as motion on a phase space torus. The dimension of the torus is the number of the system’s degrees of freedom, and each point on it specifies the location in space and the velocity of the system at a given instant in time. The coordinates of the point on each circle correspond to a particular periodic cycle to which the whole motion has been decomposed. Analogously, the coordinates on each circle of a point in the latent space torus specify one of the basic properties of an element of the dataset to which the description of that element has been decomposed. The possible intriguing relationship between the integrable structure of dynamical systems and its decomposition into its basic harmonic motions and the decomposition of the dataset into its basic constituents can be valuable in relating dynamical systems to the process of learning.
The task of unsupervised disentanglement learning is complicated by the fact that given any disentanglement representation there is an infinite family of entangled representations that have the same marginal distribution (Locatello et al., 2019). Therefore, an inductive bias is needed for the task. Our solution in this work is based on the concept of entanglement entropy in quantum physics. There are non-generic states in the Hilbert space that have low entanglement, and a special class of them are the product states, whose entanglement entropies are identically zero. We employed these states for our disentanglement representation. It is indeed easy to see that these provide a special representation, since any random unitary (or orthogonal) matrix acting on them will make our representation entangled and hence destroy the disentanglement properties of our solution. Thus, in a sense, our inductive bias is the use of the special non-generic states (of measure zero) in the Hilbert space for our representation.
We introduced a DC-score to jointly quantify the disentanglement and completeness of the torus latent representation and calculated it, as well as various other measures in a large set of numerical experiments. For the comparison we used five different types of datasets. We varied the dimension of the latent space torus and showed a clear advantage of the torus latent description compared to others when the torus dimension is equal or greater than the number of generating factors. In order to properly assess the cases where the number of expressive codes is lower than the number of generative factors, we introduced a correction to the disentanglement score, and hence to our DC-score.
One limitation of our proposed low entangled representation is in scaling to datasets containing a large number of generative factors. The dimensionality of v increases exponentially with the dimension of the torus latent representation and is, therefore, limited in describing these situations. This can perhaps be remedied by a different choice of embedding, e.g. working with a direct sum of tensor products, which increases polynomially. Another limitation of the model may be in its ability to effectively describe a property of the dataset that is not naturally encoded in a circle, but is, rather, non-compact in nature.
While our torus latent representation performs better compared to other approaches, one may still wonder why it did not achieve the perfect DC-score 1. This can be explained by the fact that the generation of the datasets assumes a certain number of generating factors, while in fact there could be more, some indirectly combined. For example, in a case where multiple underlying factors are combined into one explanatory factor, the disentanglement may overshoot, thereby reducing the obtained score.
REPRODUCIBILITY STATEMENT
All the code for reproducing all experiments presented in this work is available in the supplement.

heading: A GENERATIVE FACTORS HEATMAPS
text: In this section we provide additional heatmap examples for different tori which provide a qualitative insight. Figures 5 and 6 present the heatmaps for our method on T 4 and T 8 torus topologies, respectively, with their D, C and DC scores. When the number of generative factors is larger than the number of torus dimensions, D, one can expect some mixing between the generative factors and the latent codes. This can be observed for instance for T 4 in Fig. 5.
As the dimension of the latent codes increases, see Fig. 6, the D, C and DC-score metrics improve. This effect on the disentanglement results from the prefactor rank(R)K in equation 1, which depends on the ratio of the numbers of latent codes and generative factors when the latter is larger.

heading: B ON THE CONNECTION TO QUANTUM PHYSICS
text: The assumption is there is a set of elementary generative factors zi, i = 1, .., n that are disentangled from each other, and from which the data distribution can be constructed. The task is to learn these elementary factors and represent them by latent codes ca, a = 1, ...,m. Ideally we would have liked to have n = m and a bijective map f that associates to any ca one zi and vice versa.
We are working with a linear representation and since quantum mechanics is a linear theory it inspires our choice of the latent space representation. In quantum mechanics states are represented by vectors in Hilbert space and the operators that act on them are linear. In our case, the generative factors constitute a state and so do the latent codes. The linear regressor relating them is a linear operator.
We represent each elementary factor zi as a state (qubit) in a Hilbert space |zi〉 ∈ F and the complete set of elementary factors by a state (quantum register of n qubits) in the tensor product of these Hilbert spaces |z1, ..., zn〉 = |z1〉 · · · |zn〉 ∈ H = F ⊗ F · · · ⊗F . Note, that only a measure zero of n-qubit states can be represented in such a way as a product state and the reason that we can do that is the assumption that zi are disentangled.
Entanglement can be viewed as a measure of how far the state is from being a product state. One way to measure it is by taking the supremum on the absolute value of the inner product of the state with all the possible product states. Another measure is the Von-Neumann entropy that is calculated by constructing a density matrix ρ as the outer product of |z1, ..., zn〉, decomposing zi to e.g. two subsets A and B such that ρ ∈ HA ⊗ HB and tracing out part B, ρA = TrBρ. The eigenvalues of ρA encode the entanglement between A and B. Specifically, the Von-Neumann entropy is SV N = −TrρA log ρA and it vanishes iff the state |z1, ..., zn〉 is a product state. Returning to our case, we construct the latent space representation using a tensor product since it should capture the above structure of the generative factors. The measure of disentanglement Da quantifies it by the −P logP term which is the analog of the Von-Neumann entropy. Note that there is a difference between our case and quantum mechanics: in quantum mechanics an elementary qubit state is represented by two angles (θ, φ) while in our case and elementary latent code is one angle parametrizing a circle of the torus.
One may inquire what would have gone wrong in our analysis if we chose that space of latent codes to be another curved compact manifold such as a sphere. In such a case we should have covered the manifold with local patches such that there would be a linear map between the generative factors and the latent codes on each patch and a transition function between them. Such a representation would be entangled and will fail, e.g. on the sphere near the pole since rotating by the azimuthal angle would not change any property.

Review summaries:
The paper presents a novel method for disentangling latent representations, specifically for use with variational auto-encoders. This is accomplished by treating the latent codes as a tensor product of 1-spheres which results in their forming an n-dimensional torus where n is the number of latent factors. Additionally, the authors propose a new metric for analysing the quality of the decompositions. These methods are well founded and make intuitive sense, and the authors supplement this theoretical motivation with an array of tests comparing to several existing methods on several different datasets. The results demonstrate that the methods work well both qualitatively and quantitatively.

The authors propose an autoencoder where the latent space is defined on a torus. A D-dimensional torus is represented by the tensor product of D unit circles. They argue the torus induces disentanglement as the analogous of entropy entanglement in quantum physics. Using 5 datasets, they show both the reconstruction performance and disentanglement score (Eastwood & Williams, 2018) are high.

This paper proposes a new autoencoder architecture that achieves better recovery of the latent factor structure in artificial datasets according to previously established metrics: disentanglement, completeness, and informativeness.  The key idea is to nonlinearly project a small number of latent factors into a higher-dimensional space, based on the topology of the torus, in such a way that prevents an arbitrary linear rotation of the latent factors from yielding an equivalent representation.  The approach is compared against existing approaches for variational autoencoders focused on learning disentangled representation, and achieves higher disentanglement and completeness, as well as competitive or best informativeness (reconstruction) on a majority of datasets considered.