Learning Disentanglement in Autoencoders through Euler Encoding
Authors: Jaehoon Cha
Jeyan Thiyagalingam
Venue: Submitted to ICLR 2023 (2023)
Keywords: disentanglement
disentangling
linear disentangled representations
autoencoder
latent space
factorizing
latent-space factorization
latent-space regularization

Abstract: Noting the importance of factorizing (or disentangling) the latent space, we propose a novel, non-probabilistic disentangling framework for autoencoders, based on the principles of symmetry transformations that are independent of one another. To the best of our knowledge, this is the first deterministic model that is aiming to achieve disentanglement based on autoencoders without pairs of images or labels, by explicitly introducing inductive biases into a model architecture through Euler encoding. The proposed model is then compared with a number of state-of-the-art models, relevant to disentanglement, including symmetry-based and generative models based on autoencoders. Our evaluation using six different disentanglement metrics, including the unsupervised disentanglement metric we propose here in this paper, shows that the proposed model can offer better disentanglement, especially when variances of the features are different, where other methods may struggle. We believe that this model opens several opportunities for linear disentangled representation learning based on deterministic autoencoders.

Under review as a conference paper at ICLR 2023
LEARNING DISENTANGLEMENT IN AUTOENCODERS THROUGH EULER ENCODING
Anonymous authors Paper under double-blind review
ABSTRACT
Noting the importance of factorizing (or disentangling) the latent space, we pro- pose novel, probbabbaiilitti disentangling framework for autoencoders, based on the principles of symmetry transformations that are independent of one an- other. To the best of our knowledge, this is the first deterministic model that is aiming to achieve disentanglement based on autoencoders without pairs of im- ages or labels, by explicitly introducing inductive biases into model architecture through Euler encoding. The proposed model is then compared with a number state-of-the-an art models, relevant to disentanglement, including symmetry- based and generative models based on autoencoders. Our evaluation using six different disentanglement metrics, including the unsupervised disentanglement metric we propose here in this paper, shows that the proposed model can offer better disen- tanglement, especially when variances of the features different, where other methods may struggle. We believe that this model opens several opportunities for linear disentangled representation learning based on deterministic autoencoders.
INTRODUCTION
Learning generalizable representations of data is one of the fundamental aspects of modern machine learning (Rudin et al.] 2022). In fact, better representations are more than a luxury now, and is a key to achieving generalization, interpretability, and robustness of machine learning models (Bengio et al. 2013: Brakel & Bengio 2017 Spurek et al. (020). One of the primary and desired character- istics of the learned representation is factorizability or disentanglement, so that latent representation can be composed of multiple, independent generative factors of variations. The disentanglement process renders the latent space features to become independent of one another, providing a basis for a set of novel applications, including scene rendering, interpretability, and unsupervised deep learning (Eslami et al. 2018 [iten al. 2020, Higgins al. (02T).
Deep generative models, particularly that build on variational autoencoders (VABs) (Kingma & Welling 2013: Kumar et al. [2177: Higgins et al. [2077] Tolstikhin al., 2018 Burgess et al. 2018]] Chen et al. 2018 |Buggess et al. 2018]] Kim & Mnih 2018] et al., (019), have shown to be effective in lcarning factored representations. Although these approaches have advanced the disen- tangled representation learning by regularizing the latent spaces, there are number of issues that limit their full potential: (a) VAE-based models consist of two loss components, and balancing these loss components is a well known issue (Asperti & Trentin 2020) (b) it is almost impossible to honor the idealized notion of having a known prior distribution for VAEs in practical settings (Takahashi al. 2019 Asperti & Trentin 2020 Zhang al., 2020 Aneja et al., 2021) and, (c) factorizing the aggregated posterior in the latent space does not guarantee corresponding uncorrelated representa- tions (Locatello et al. (2019). An alternative approach for achieving disentangled representations through seeking irreducible representations of the symmetry groups (Cohen & Welling) 2014| Hig- gins et al. 2018, Painter et al. 20200, Tonnaer al., 2022). where the aim is to find latent space transformations that are independent of one another, underpinned by well-defined mathematical framework(s) based on group theory. As this group of methods exploits the notion of transitions between samples, they require pairs of images representing the transitions (Cohen & Welling [0144] Painter et al. 2020) or equivalent labels (Tonnaer et al. 2022). Regardless of the approach, as shown in Locatello et al. (2019), it is fundamentally impossible to learn disentangled representations with- out having inductive biases on either the model or the dataset, and both VAE- and symmetry-bast -bssed approaches exemplify implicitly embedding inductive bias.

---

Under review as a conference paper at ICLR 2023
(d)
(c)
(k) LSBDVAE
DAI
AE
VAE
(-VAE
6-TCVAE CAI-VEE
FVAE
InfoVAE
DIPVAE
WAE
Figure Latent spaces learned by different models. Ideal learned latent space should cover two-dimensional grid (Higgins et al. 2018). The first, second and third rows show the latent spaces learned from three datasets, namely, XY, 2D Arrow, and 3D Airplane datasets, respectively. Columns correspond to different models stated at the bottom of every column. It can be seen that the proposed model, DAE, achieves the best disentanglement.
Despite the advances, we note number of issues the existing approaches about how they address disentangiement. Firstly, the majority of the VAE-based approaches are probabilistie, and such, the quality of the disentanglement depends on ideal or near-ideal priors, and on the process of learn- ing the correct posteriors for given data. Secondly, the majority of symmetry-basee disentangling approaches need pairs of images or labels, even in the unsupervised setting, owing to the require- ments around inductive bias. Thirdly, none of these models conform to the formal definition of linear disentangled representation proposed by |Higgins et al. (2018). Finally, and most importantly. none of the existing approaches have the unsupervised approach for introducing inductive biases (required for disentanglement) both on the models and on the datasets, essentially demanding labels or image pairs. Motivated by these shortcomings, in this paper, we propose a novel approach for deriving disentangled representation learning, with the following key contributions: We,
propose a totally unsupervised approach for introducing inductive bias into the model and data, without requiring pairs of images or labels,
propose non- prnbbabbabilittcc approach that does not involve any priors or learning posteriors,
ensure that our approach conforms to the formal definition of linearly disentangled rep- resentation as defined in Higgins et al. (2018).
propose unsupervised metric, namely, Grid Fitting Score (GF-Score). to quantify the disentanglement, echoing the aspiration of an ideal disentanglement measure outlined in Higgins al. (2018), and
demonstrate the implementation of formally defined disentanglement using autoencoders.
Disentangling Auto-Encoder (DAE), offers theo- retically sound framework for learning independent multi-dimensional vector subspaces, and hence towards learning disentangled representations. To the best of our knowledge, this the first tempt to actually implement disentanglement approach using deterministic autoencoders, espe- cially without pairs of images or labels, and hence in a truly unsupervised manner. We provide glimpse into the capability of the proposed model for disentanglement using three datasets compared against ten other models, which are either autoencoder-baas based disentangled models, that do not require any labels or pairs of inputs in Figure
As such, the proposed approach, which
probabilistic models or symmetry-
The rest of this paper is organized as follows. In Section we review the related work, focusing on VAE-based and symmetry-baser approaches. This is then followed by derivation of AE-based non-probabilistic approach for deriving disentangled representations in Section 3 Section 4 we perform detailed evaluation to decide the overall performance of the proposed model, using ten baseline models, six datasets, and six disentanglement metrics, and discuss our findings. We then conclude the paper in Section 5 with directions for further research. Given the space constraints, highlight the prominent results in the main part of the paper, while providing the remaining set of results and relevant material as part of the Appendix.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
2.1 DISENTANGLEMENT
Disentangled representation learning focuses on learning set of independent factors containing useful but minimal information for given task, such that their variations orthogonal to each other while accounting for the entire dataset (Bengio et al. 2013 Higgins et al. 2018). This essen- tially entails method or set of methods for decoupling correlations between latent variables. A large body of work around disentanglement, and the ideal properties of disentangled representation
can be found in RRdggeay (0T6); Eastwood & Williams 20088; Ridgeway & Mozer (2018); Zaidi
iet al. (2020). Among number of desirable properties of disentangled representation, modular- ity, compactness and explicitesss are three critically important properties. number of metrics have been proposed in the literature to quantify these properties (Higgins et al. 20117, Kim & Mnih 2018
Eastwood & Williams 2018: Chen al.) [208]] Do & Tran 2019) Sepliarskaia et al. (019)) In our work, we use the notions outlined in Zaidi et al. (220)), where the metrics are divided into three classes, namely, Intervention- -based,, Predictor-based, and Information-bass metrics. These metrics are all used in a supervised manner and can be of indicators for the robustness of the representation to noise, and for the non-line non-linearity of the relationships between learnt representations and ground truth factors, in addition to the three properties outlined above.
2.2 AUTOENCODER-BANE PROBABILISTIC MODEL.
AE-based probabilistic generative models are realized by replacing the conventional encoder Eg and decoder Do with probabilistic counterparts. The probabilistic encoder, denoted by q (0(2(z)) [z)) is used approximate the intractable true posterior, and the probabilistic decoder, denoted by po (== is used to reconstruct the æ from z (Kingma & Welling, (013). The majority of the previous work on disentangled representation learning are based on probabilistic models, particularly building on VAE. They enforce regularization in the latent space that either regularizes the approximate posterior ls 2 a)) the aggregate posterior g(z) &/ Σ 2 2 (z)())) ), as summarized in T'schanenn et al. 2018). The overall objective of the majority of the VAE-based methods can be expressed as:
Lrecon (Φb 0) + Lreg
where ( is regularizer of the concerned generative model. A carefully designed regularizer should enable the model to achieve better disentanglement, either by controlling the capacity of the latent space, or by measuring the total correlation between latent variables. However, it is worth noting that factorizing aggregated posterior using regularizers does not guarantee linear disentangled representations (Locatello et al. 2019). We summarize the regularization terms of seven state-of- the-art generative models in Appendix A (See Columns 2 and 3 of Table 3
2.3 SYMMETRY-BASHH DISENTANGLINNG MODEL.
While Higgins et al. 2018) proposed formal definition of linear disentangled representations, it was generic, so that no specific architecture, model or technique were defined. As such, does not provide an actual mechanism for learning such disentangled representations, albeit providing formal definition, which is essential for this purpose. From the definitions in Higgins et al. (2018),
symmetry group can be decomposed as product of multiple subgroups, if suitable subgroups can be identified. This can provide an intuitive method for disentangling the latent space, if sub- groups that independently act on subspaces of a latent space can be found. If actions applied on each of the subgroups affect only the corresponding subspace, these actions are called disentangled group actions. In other words, disentangled group actions only change a specific property of the state of object, and leaves the other properties invariant. If there is a transformation in vector space of representations, corresponding to a disentangled group action, the representation is called disentangled representation.
The concept and implementation of symmetry- based disentangled representations were proposed using pairs of images in Cohen & Welling (2014). However, owing to the limitation around commu- tative Lie groups, upon which this model is built upon, the real world applicability of the technique from Cohen & Welling (2014), especially across range of diverse datasets, limited. Following
3

---

Under review as a conference paper at ICLR 2023
formal definition for linear disentangled representations in Higgins al. (2018), there has been considerable amount effort to learn the transitions between images (Caselles Dupré et al. 2019 Quessard al. 2020 Painter et al. 2020). The transitions between images are learned by treating each transition as sequence of transitions until the base transition relies on pairs of images and by using additional networks. An alternative approach is to rely labels, for example, in TTonneer iet al. (2022), where they propose two Diffusion VAE-based methods (Rey et al.) 2019). namely semi-supervised and unsupervised, along with new metric called LSBD (Linear Symmetry-Based Disentanglement metric). The former model relies on labels, while the latter does not. As such, the latter model is directly relevant to our work, and, we use this as one of the baselines for our evaluation (See Section
FRAMEWORK FOR DAE
The deterministic, and hence, non-prots con-proaabilistic, approach we propose here, builds on the autoen- coder architecture (rather than variational autoencoders). We provide the relevant background on the disentangled representations from Higgins et al. (2018) in the Appendix A.2 In this section, define necessary mathematical framework and a corresponding neural network architecture imple- menting the proposed disentangling autoencoder.
3.1 ASSOCIATION BETWEEN THE DISENTANGLED REPRESENTATION AND AUTOENCODER
The definition of disentangled group actions from Higgins et al. 2018 assumes that group G can be decomposes direct product G = G1 X...X G To relax the condition, we consider a group G, which is generated by S {8i, $2, ... 8n} subject Let W be set of world-states and suppose we have a group action G x W - W. Then, we say that the action is disentangled by the relation R if there decomposition W W1 and actions :< 8, > xW, + W,, i { {I...., n such that:
of R of relations among elements in S.
W,,
1. (.11...,55)) (Wi, Wr.) = (5f W1. th wn) and,
2. if any elements g E can be written uniquely in the form = of s.. n for some 6, E Z by the relation R.
disentangling latent space relies on finding a corresponding group action : G - Z so that the symmetry structure of W reflected in an agent's 's representations, Z. This can be achieved if the following condition is satisfied:
With the definition of an equivariant map in place A.2
((w) = f[ w) Vg E G, w W.
where f W + Z is a mapping from world-states to an agent's representations. However, in general, one cannot control the nature of the generative process W - O leading from world- states to observations, O. In addition, without loss of generality, we can easily assume that the generative process an equivariant map.
Theorem 3.1. Suppose a generative process is an equivariant map satisfying 9 b(w) = b(g w) Vg E G.w W. Then, there exists function f that satisfies (2) if an inference process h: O > Z is an equivariant map satisfying.
g --h( hig- o) GgEEO
Proof. Proof in the Appendix
A.3
Following the Theorem 3.1 this assumption leads the fact that the goal of disentangling is the same as finding an inference process h O + Z satisfying,
g h()) h(g o) Vg E G, Although there is no guarantee that one can find a compatible action G Z + 2 satisfying l is bijective then (4) can be expressed follows,
(4)
4
g hlg h (z))
(5)
4

---

Under review as a conference paper at ICLR 2023
However, if h is a bijective function, simple neural network-based models cannot learn the overall cquivariant map. Yet, the equivariant map, such as one outlined in equation S can be learned by autoencoders with inductive biases both on the model and the datasets, which is the central contri- bution of this paper. To show this mapping, let h and h h-1 be an encoder, Egg, and a decoder, Da. of autoencoder. Then, the group action
G Z Z be defined as follows:
G % idc x Do
Es
GxO
7
This shows that the equivariant map can, indeed, be learned by an autoencoder. However, this is not without number of challenges, which we discuss in Section below.
3.2 INTRODUCING DISENTANGLED REPRESENTATIONS INTO AUTOENCODERS
In deriving a disentangled representation. it is worth noting that, a vector addition, basic and natural transformation, in the latent space enables natural transition between latent variables and nth root of unity is cyclic group. We achieve a disentangled group action by the relations R in the latent space by map (sf ...., 8,,55) (e ...., eia..c) and W Z, for some as E (0, 1). However complex numbers are undesirable in machine learning and so this is achieved by introducing an Euler encoding, E I,", into the AE architecture. We defined E as follows:
E(z) (202(2 sinei,ssnnnneaas s2m(2 c2 sin(2222): where n is the the number of dimensions of the latent space.
cos((2=n) (oss(2))
sin(2
(6)
Theorem 3.2. Let E be Euler encoding and A 2n + R be an injective linear transformation where 2n. For o (0, 1) and E {1, ...y T.). let T R R. by (xx) (x), Et +
an Ln). Then A. (x))(z)) = A E(T) (z)) if and only if = j and a = B.
Proof. For E R", let E(T, A E(T)) (z)) and define
I2ii1)
cos(2
sin (2
S7
sin(2Ta)
cos(22r
I2(n-x)
Since E(T, (z)) S. E(z), A (8) - SS) -))- E(()
(a) If + then A is zero transformation. which is contradiction.
(b) If then B + kt where & E Z, henee, = 8.
Since (x)) S; E(()) and Sp is orthogonal transformation. the Euler encoding after translation on Z can be considered as an orthogonal transformation of E(z), which enables the changes of output from changes of different latent dimensions to be orthogonal. Nevertheless, there are still a number of practical challenges to overcome. These are: (a) Number of Elements in least the relative ratio of the number of elements between the subgroups are not known a priori. This has crucial role introducing inductive biases on datasets, (b) Robustness Small Perturbations: Because the proposed approach for disentanglement is deterministic, the model is not resilient to small perturbations (e.g., noise) (Camuto et al., 2021). ), which is essential for the model to behave in robust manner when presented with unseen examples, and (c) Spatial Distribution of Features: An ideal factorized latent space must have the features spatially distributed an equally likely manner. However, the equivariant map we discussed above alone may not be sufficient address this issue. Although is possible to address some these concerns from a theoretical stand point, nearly all of these addressable by carefully designing architecture that exploits both the AE and the equivariant map principle discussed above, achieving the best possible disentanglement. We discuss these details in Section 3.3 below.
Subgroup: The number of possible elements in the subgroups N; (j 1,...,,. nt),
5

---

Under review as a conference paper at ICLR 2023
A a matrix that coustrains cach feature in the latent space
...
Encoder
Normalisation Incerpolation
Fuler
Decoder
Reconstruction loss
Illustration of the DAE architecture. The model includes the Euler encoding, and the outputs from the interpolation layer are mapped to cosine and sine values.
Figure 2:
3.3 ARCHITECTURE OF THE DAE
In mapping our theory to an architecture, we build on the AE model, which constitutes an encoder, that maps the observation space O factorized latent space Z', followed by the disentangling process that dettorizee/disentangeee
the latent space Z to Z', and finally the decoding layer, that maps the factorized latent Z' the regenerated observation space O Each of the concerns that were discussed in Section 3.2 (a) through (c) are handled by network of layers in our architecture. We show this model Figure and describe how this model addresses the relevant concerns next.
(a) Number of Elements in Subgroup: Although the number of elements in a subgroup is not known priori, the number or the relative ratio of the possible number elements across sub- groups can be estimated using techniques that can extract the variance information from compressed information, such as principal component analysis (PCA) (Jollifec 2002), independent component through neural networks (Kingma & Welling 2013 Burgess et al., 2018 Mondal et al. 2021). In this paper, for the reasons of simplification, will be using the PCA technique. Since the singular values from PCA are proportional to the variances of the principal components compressed data, these values are used to obtain a relative ratio of the number of possible element in the subgroups (Wall al. (003). Then, all singular val- ues are divided by the maximum values, and rounded to one decimal places, and then values smaller than are replaced with hyperparameter a. The relevant algorithm shown in Algorithm [ in the Appendix. Let A be the relative ratio of the possible number of elements across subgroups obtained from Algorithm I
analysis (ICA) (Hyvärinen & Oja, 2000).
(b) Uniform Spatial Distribution of Features: To ensure that each feature is equally/likely dis- tributed across the latent space and falls within the (0, 1) (which lets A (z)) A E(T, (z)) if and only if i j and ,9 in Theorem 3.2). we introduce normalization layer, where we ap- ply batch min-max normalization the outputs of the encoder. As minimum and maximum values vary from batch (mini-batch to batch (mini-batch), values during the training process, and use them during the test phase, akin to a batch normalization layer (loffe & Szegedy 2015). The minimum and maximum values are also initialized close to the middle point of [0. 1) to facilitate learning. This is then followed by scaling by A to account for different number of possible elements for different features.
update the moving minimum and maximum
Robustness is achieved by introducing an Interpolation layer that performs Gaussian interpolation the output of the normalized latent space, follow- ing Vincent et a (2010); [Berthelot et al.) (2018). Gaussian interpolation is used to map unseen examples to known examples, and make the latent space locally smooth. However, since the proposed model is deterministic, it is important to map number of unseen examples to the learned representations. This is achieved by adding weight-sensitive Gaussian noise the outputs of the previous layer during training, which is obtained based on the closest proximal distance of each dimension of the representations. The relevant algorithm shown in Algorithm 2 in the Appendix. It is worth noting that this layer will not be used during the inference phase.
(c) Robustness to Small Perturbations:
3.4 A NOVEL METRIC FOR QUANTIFYING DISENTANGLEMENT GF-SCORE
Nearly all of the existing set of metrics outlined in the literature for quantifying linear symmetry- based disentanglement require ground truth labels. Here, propose metrie, namely, Grid

---

Under review as a conference paper at ICLR 2023
Fitting Score (GF-Score), to achieve the same purpose without the need for labels. Our hypothesis is that performing independent disentangled actions on a symmetry group causes the corresponding subspace to form grid-shape latent space. This can be exploited by generating square grid to include latent variables, and by measuring the mean of the minimum distances from the square grid to latent variables signify the quality of disentanglement. If the latent variables fit perfectly into the square grid, it would imply that the model achieves perfect linear disentanglement, and we can mark this zero score. Therefore, the lower the GF-Score is, the better the disentanglement is. The relevant algorithm is shown in Appendix A
4 EVALUATION AND RESULTS
4.1 EVALUATION METHOD
Our evaluation involves comparing the performance of the proposed approach against ten baseline models across six datasets using six disentanglement metries. The code will be publicly available when the paper is published. We outline these details below.
Datasets: One of the critical challenges around evaluating disentanglement identifying suitable datasets. It is difficult to identify common datasets to study this problem. In the literature, differ- ent datasets have been used for different purposes. For example, dSprite (Matthey et al.] 2017). 3D Chair (Burgess & Kim 2018) and CclebA (Liu et al. 2015) datasets have been used in 3-VAE, 3-TCVAE, and FVAE. Although these datasets are useful to understand the traversal order of the latent space, they lack clear underlying group structure. As such, here, utilize the datasets that have been first utilized in [Higgins et al. (2018), with relevant enhancements, which we describe in the Appendix (See (00). In addition to this dataset, we use five more datasets containing clear un- derlying group structure, namely, 2D Arrow, 3D Airplane (Tonnaer et al., 2022), 3D Teapots (East- Kim 2018) and 3D Face Model (Paysan et al., 2009) datasets. Finally, to demonstrate the performance on complex datasets, we use the Blood Cell Acevedo et al.. (2020) and the Sprites |Red et al. (015)) datasets (see results in the Appendix).
wood & Williams 2018 3D Shape (Burgess
Baseline Models: We considered ten different baselines models for our evaluation, namely, plain AE, vanilla VAE, 3-VAE, 3-TCVAE, CCI- VAE, FVAE, InfoVAE, DIPVAE, WAE and LSBDVAE. For DIPVAE, we only test DIPVAE-I, owing to the reasons of that DIPVAE-II model works better only for cases where the dimension of the latent space is larger than the ground truth factors, which is not the case for us. Furthermore, the proposed technique is purely an AE-based method, we have not included any GAN-specific basclines. To render a fair evaluation mechanism. we used the same encoder and decoder architectures, and same latent space dimensions (for each baseline model), which are used in Higgins et al.] (211)); ) Kim & Mnih 2018); Quessard et al.] (2020): Tonnaer et al. (202)) throughout the evaluation. Please the Appendix (SeelA.8) for additional details.
As discussed in Section 2.1 a large number of metrics can be used to study the performance of disentanglement, depending on the nature of the dataset, access ground truth, availability of latent factors, and the number of dimensions in the latent space. We use two types of metries: (a) Visualization of the latent space, and (b) Numerical disentanglement score. The former permits one to visualize the orthogonality between features, and can be used to demonstrate how the latent traversal is achieved by the model and grid structures in the latent space. The sec- ond approach provides a quantifiable method for assessing the disentanglement. Collectively, we have used six metries, including five supervised metrics accounting for each of the disentanglement metric classes (see Section 2.1 namely, z-diff and z-min from the intervention-batt the predictor-based and jemmig and deimig from the information- based metric classes in order to measure disentanglement, completeness and informativeness, along with GF-Score (See 3.4). In Lo- catello et al. (2019), it was shown that variances of all metrics are large with random seeds and it disturbs the comparison between different models. Hence, we run all the models each data set for 20 different random seeds and select the random seed with the highest total score over these metrics.
Performance Metrics:
dci-rf from
4.2 RESULTS AND DISCUSSIONS
Our evaluation has produced a considerable volume of results, and for the reasons of brevity, present two sets of results here, namely, (i) we show the reconstructions of latent traversals for the 2D Arrow, 3D Airplane, XYCS, 3D shape, 3D Teapots and 3D Face Model datasets in Figure 3
7

---

Under review as a conference paper at ICLR 2023
K 2
9
7
(
(b)
(C)
(d)
(c)
Figure 3: Reconstructions of latent traversals across each latent dimension obtained by the DAE for the (a) 2D Arrow (color and shape), (b) 3D Airplane (color and shape), (c) XY CS (æ position, position, color and shape), (d) 3D Shape (floor hue, wall hue, object hue, scale, shape and orienta- tion), (e) 3D Teapots (azimuth, elevation, red, green, blue and extra) and (f) 3D Face Model datasets (face id, azimuth, elevation and lighting).
Table I: Differences in scores between XY/XYCS datasets. Absolute and percentage of change from XY to XYCS are shown. Percentage changes closer to the zero are desirable.
Disentanglement scores
Models
z-diff' 1
2-var
dci-rf
jemmig
dcimig f
DAE
1.00/1.00 (0.0%)
1.00/1.00 (0.0%)
0.99 0.99/0.94 /0.9//94444 (-5.0%)
0.81 0.81/0.67 /0.67 (-17.2%)
0.80 /o.0/o.75 (-6.2%)
8-VAE
1.00/0.79 (-21.0%))
/..0/..48. (-54.0%)
0.91 /a.us (-91.2%)
0.65 /0.18 (-72.3%)
0.63/o /a. (-79.3%)
3-TCVAE
1.00 (-28.0%))
1.00 /000/..411 (-59.0%)
0.93 /a.15 (-83.8%)
0.70/o.24 (.65.7%)
0.69 /o. (-79.7%%
CCI-VAE
1.00 1.00/1.0000.000
1.00/1.00 (0.0%)
0.98 //..0.... (-35.0%)
0.78/o. (-39.7%)
0.76/o 46 (-39.4%)
EVAE
1.00/1.00 (0.0%)
1.00/o.91 91 (-9.0%)
/9.6/..18 (-80.8%)
0.73/o.21 21 (-69.5%)
0.70/o. 13 (-80.5%)
Info
1.00/0.9((-8.. (-8.0%)
1.00 /o.00//..44 (-46.0%)
0.90 0.90/o.21 (-76.6%)
0.64/o.27 (-57.8%)
0.67 /o. 6-7)
DIPVAE
1.00 1.00/1000000 (0.0%)
1.00 /o. (-56.0%)
os /0.32 (-67.3%)
0.78/o. (-64.1%)
0.78 1.75/.111 (-85.8%)
LSBDVAE
1.00 1.00/1.00 (0.0%)
1.00 1.00/o.78 (6.////66622.2 (-24.0%)
0.96 0.96/0.38 (-60.43))
0. 0.72/o.30 (-68.3%)
0.70 (-60.0%)
along with the percentage of changes when color and shape features are added to XY dataset in Table [ and, (ii) we present the disentanglement scores of top two performing models across all datasets for all metrics in Tables and 2 We provide the remaining set of results (reconstructions of latent traversals and disentanglement scores of all models), and other relevant details (such as hyper-parameters, and network architectures A.7 and A.8) part of the Appendix.
4.2.1 LATENT SPACE VISUALIZATION
We show the disentangled (2D) latent space for the XY, 2D Arrow and 3D Airplane datasets, which have two underlying factors, in Figure [ (also see Table 14 in Appendix A for details of relevant hyperparameters) As can be seen in the Figure D the proposed model, in general, provides the ideal grid-shape outlined in |Higgins et al. (2018). The plain AE, vanilla VAE, Info- and WAE models offer the worst performance. Other models, such as 8-VAE, 3-TCVAE, CCI-VAE, and DIPVAE models also come closer to the ideal pattern in the three datasets, and thus most models are able to disentangle x and y positions in XY datasets, and rotation and color factors in 2D Arrow and 3D Airplane datasets. However, when color or shape feature is added to the XY dataset (i.e., for XYC, X and XYCS datasets), the disentanglement can become a significant challenge, other than for the proposed model (See Figure in Appendix). In addition to these, pairs of latent spaces, and reconstructions of latent traversals (across each latent dimension) of six datasets for the DAE are shown in Figure 3) (Also the Appendix A.11 A.11
for more results.)
4.2.2 DISENTANGLEMENT SCORES
We present the supervised disentanglement scores and their percentage changes when color and shape features are added to the XY dataset in Table 1
with the changes measured relative to the
8

---

Under review as a conference paper at ICLR 2023
Table 2: Disentanglement scores for the 2D Arrow, 3D Airplane, 3D Teapots, 3D Shape and 3D
Face Model datasets
Datasets
2D Arrow
3D Airplane
3D Teapots
3D Shape
3D Face Model
Metrics/Models
DAE
DIPVAE
DAE
DIPVAE
DAE
DIPVAE
FVAE
DAE
DAE
3-TCVAE
2-diff
1.00
1.00
1.00
0.85
0.96
1.00
0.96
1.00
1.00
1.00
0.93
1.00
0.92
dci-ef
0.88
0.85
0.80
0.54
0.89
0.84
0.99
0.99
0.65
0.59
jemmig
0.80
0.75
0.79
0.51
0.54
0.52
0.86
0.87
0.48
0.53
doimig f
0.79
0.72
0.75
0.43
0.53
0.50
0.88
0.90
0.17
0.47
(e (c=100)
0.30
2.55
0.19
7.66
0.002
0.26
0.20
0.0009
0.02
0.37
XY dataset, and it is worth noting that the added features have smaller variances than x and positions. As can be seen, in general, nearly, all models suffer from the performance drop except few. The CCI-VAE is the only model that performs as good as the proposed model for the z-diff and z-min metrics. The proposed model shows the smallest drop across three remaining metries, namely, dci-rf, jemmig and dcimig. While the largest drop in the proposed model is 17.2%, the scores fall by between 35.0% and 91.2% for the other models.
The disentanglement scores for the top two performing models for all datasets (except the XYS dataset) are shown in Tables[aand and 2 with the best performing model highlighted boldface. From these results (including those in the appendix), the following observations can be drawn: Firstly, the proposed model outperforms all models across all metrics for the 2D Toy (covering XY, XYC, SY XYCS datasets), 3D Airplane and 3D Teapots datasets (See Table for the remaining five datasets, DAE offered the best score across four of those datasets (2D Arrow, 3D Airplane, 3D Teapots and 3D Face Model) while offering the second best performance for the 3D Shape dataset. Where the DAE offered the second best performance, DAE still achieve higher scores jemmig and deimig and the same scores in z-diff and z-min. Thirdly, for the 2D Toy dataset, the proposed model maintains the reconstruction loss as small as possible whilst offering improved disentanglement scores (See Figures| 25 28). On the other hand, the reconstruction losses for the 3-VAE, 3-TCVAE and CCI-VAE models increase along with their disentanglement scores. Finally, GF-Score shows that the proposed model perfectly fits the latent space into grid structure across all datasets and baseline models. Based on the GF-Score from Table 4 to 10 model without
Table 7. Secondly,
regularizer, such as AE, fails form a grid structure in the latent space.
5 CONCLUSIONS
In the context of representation learning, being able to factorize or disentangle the latent space di- mensions is crucial for obtaining latent representations that are composed of multiple, independent factors of variations. Literature around disentanglement methods are either predominantly super- vised semi-supervised, and such, either labels pairs of images required or achieved via factorizing the aggregated posterior in the latent space.
In this paper, we presented non-probabilistic, deterministic model, namely, disentangling autoen- coder or DAE, addressing number of issues found in the literature. We also demonstrated how realize the disentanglement conceptualized in |Hggins al. (2018) for the first time, especially without requiring labels or pairs of images. Our approach exploits the Buler encoding that makes the subspaces of the latent space independent of one another. Along with the architectural details, we also presented a novel metric for quantifying disentanglement, namely, GF-Score. Our detailed eval- uations, performed against large number of AE-bbased models, using considerably large number of metrics show that our model can easily offer superior disentanglement performance when com- pared against a number of existing methods across a number datasets.
Although the results are encouraging, number of aspects remain to investigated, including, eval- uation of the proposed model on datasets that lack underlying group structure, understanding the effect of the choice of the latent dimension on the outcomes, and to evaluate different latent space smoothing algorithms, to mention a few.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Andrea Acevedo, Anna Merino, Santiago Alférez, Ángll Molina, Laura Boldá, and José Rodellar. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data in brief, 30, 2020.
Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances Neural Information Processing Systems, 34, 2021.
Andrea Asperti and Matteo Trentin. Balancing reconstruction error and kullback-leibler divergence in variational autoencoders. IEEE Access, 1199440-199444) 2020.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 178 1828, 2013.
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. arXiv preprint arXiv:/807.000 2018.
Philemon Brakel and Yoshua Bengio. Learning independent features with adversarial nets for non- linear ica. arXiv preprint arXiv: 2017.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://gihhb..oom/ccemmm/////sss 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in f-vae. arXiv preprint arXiv:1804.0 03599, 2018
Alexander Camuto, Matthew Willetts, Stephen Roberts, Chris Holmes, and Tom Rainforth. To-
wards a theoretical understanding of the robustness of variational autoencoders. In International Conference on Artificial Intelligence and Statistics, pp. 3565-3573. PMLR, 2021.
Hugo Caselles-Dupré, Michael Garcia Ortiz, and David Filliat. Symmetry- -hased disentangled
representation learning requires interaction with environments. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buce E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. arXiv preprint arXiv:1802.0499// 2018.
Taco Cohen and Max Welling. Learning the irreducible representations of commutative lie groups
In International Conference on Machine Learning, pp. 1755-1763, PMLR, 2014.
Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representations. arXiv preprint arXiv:1908.0990 2019.
David Steven Dummit and Richard M Foote. Abstract algebra, volume 3. Wiley Hoboken, 2004.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation disen- tangled representations. In International Conference on Learning Representations, 2018.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 360(6394): 1210, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. Iclr, 2(5):6, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 18/2 02230, 2018.
10

---

Under review as a conference paper at ICLR 2023
Irina Higgins, Le Chang, Victoria Langston, Demis Hassabis, Christopher Summerfield, Doris Tsao, and Matthew Botvinick. Unsupervised deep learning identifies semantic disentanglement in single inferotemporal face patch neurons. Nature communications, 12(1):1-14, 2021.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications.
Neural networks, 13(4-5):411 1-430, 2000.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015.
Raban Iten, Tony Metger, Henrik Wilming, Lídia Del Rio, and Renato Renner. Discovering physical concepts with neural networks. Physical review letters, 124(1):010508 2020.
Ian T Jolliffe. Principal component analysis for special rypes of data. Springer. 2002.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning. pp. 1649-26588 PMLR, 2018.
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
arXiv: 14, 2013.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference disentan- gled latent concepts from unlabeled observations. arXiv preprint (rxxiv:/711.0044 2017.
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement 2017.
testing sprites dataset.
Arnab Kumar Mondal, Himanshu Asnani, Parag Singla, and AP Prathosh. Flexae: flexibly learning latent priors for wasserstein auto-encoders. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of o Proceedings of Machine Learning Research, pp. 525-535. PMLR, 27-30 Jul 2021.
Matthew Painter, Adam Prugel-Bennede and Jonathon Hare. Linear disentangled representations and unsupervised action estimation. Advances in Neural Information Processing Systems, 33: 13297-13307, 2020.
Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. A 3d face model for pose and illumination invariant face recognition. In 2009 sixth IEEE international conference on advanced video and signal based surveillance, pp. 296-301. leee, 2009.
Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representations and group structure of dynamical environments. Advances in Neural Information Processing Systems, 33:19727-19737, 2020.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making Advances in neural information processing systems, 28, 2015.
Luis A Pérez Rey, Vlado Menkovski, and Jacobus W Portegies. Diffusion variational autoencoders. arXiv preprint (90::::9900099111 2019.
Karl Ridgeway. A survey of inductive biases for factorial representation-teaming. arXiv preprint 1rXiv:1612.05229, 2016.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. arXiv preprint arXiv: 002205532, 2018.

---

Under review as a conference paper at ICLR 2023
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. In- terpretable machine learning: Fundamental principles and 10 grand challenges. Staristics Surveys, 16:1-85, 2022.
Anna Sepliarskaia, Julia Kiseleva, Maarten de Rijke, et al. Evaluating disentangled representations. arXiv preprini arXiv:1910.05550 2019.
Przemyslaw Spurek, Aleksandra Nowak, Jacek Tabor, Lukasz Maziarka, and Stanislaw Jastrzebski. Non-linear ica based on namer-wold metric. In International Conference on Neural Information Processing, pp. 294-305. Springer, 2020.
Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Varia- tional autoencoder with implicit optimal priors. Proceedings of the AAAI Conference on Artificial Intelligence, (3000))5066-5073 2019.
I Tolstikhin, Bousquet, Gelly, and B Schölkopf. Wasserstein auto-encoders. In International Conference on Learning Representations (ICLR 2018). OpenReview. net, 2018.
Loek Tonnaer, Luis Armando Perez Rey, Vlado Menkovski, Mike Holenderski, and Jim Portegies. Quantifying and learning linear symmetry-based disentanglement. In Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 21584-21608. PMLR, 17-23 Jul 2022.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation learning. arXiv preprint arXiv: 18/2.05069, 2018.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with local denoising criterion. Journal of Muchine Learning Reseurch, 1 (((De):33311-3408 2010.
Michael E Wall, Andreas Rechtsteiner, and Luis M Rocha. Singular value decomposition and prin- cipal component analysis. In A practical approach to microarray data analysís, pp. 91-109. Springer, 2003.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. arXiv preprint arXiv:21100 10. 14795. 2021.
Julian Zaidi, Jonathan Boilard, Ghyslain Gagnon, and Marc-André Carbonneau. Measuring disen- tanglement: A review of metrics. arXiv preprint arXiv:2012.09276, 2020.
Zijun Zhang, Ruixiang Zhang, Zongpeng Li, Yoshua Bengio, and Liam Paull. Perceptual genera- tive autoencoders. International Conference on Machine Learning, pp. 1-298-11306. PMLR,
2020.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pp. 5885-5892, 2019.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 A REVIEW OF GROUP THEORY
More details of definitions and theorems can be found in Dummit & Foote (004)).
Definition A.1. A group an ordered pair (G,*) where is set and * is a binary operation on G satisfying the following axioms:
1. Associatiiiy; (a b) xc= a (b* for all a, b.
2. Identity: there exists an element in G, called an identity of G, such that for all E G, a * E =
3. Inverses: for each a E G there is an element a= of G, called an inverse of a, such that
a a= aa *
We shall write the operation b as ab.
Definition A.2. A group action of a group G on set A map G A A (g. a) g. satisfying the following properties:
1. 91 (g2 a) (9192) a, for all 91 92 G. A. and
2. e = a. for all a e A.
Definition A.3. Let G be a group. The subset of is a subgroup of G if is nonempty and J,y H implies x= E H and xy E H.
Definition A... A group H is cylic if H can be generated by a single element, ie., there is some element x E H such that H {x"In 7), and will be denoted by < x >.
A.2 DISENTANGLED REPRESENTATION
The notion of a disentangled representation is mathematically defined using the concept of symmetry in Higgins et al. (018). For example, horizontal and vertical translations are symmetry transforma- tions in two-dimensional grid, and, hence, such transformations change the location of an object in this two-dimensional grid. From the definitions of a symmetry group in Higgins al. (2018), symmetry group can be decomposed as a product of multiple subgroups, if suitable subgroups can be identified. This render intuitive method to disentangle the latent space, if subgroups that independently act on subspaces of latent space, can be found. If actions by transformations each subgroup only affect the corresponding subspace, the actions called disentangled group actions. In other words, disentangled group actions only change specific property of the state of an object, and leaves the other properties invariant. If there is transformation in vector space of represen- tations, corresponding to a disentangled group action, the representation is called a disentangled representation. We reproduce the formal definitions of disentangled group action and disentangled representation from Higgins et al. (2018), as Definitions A.5 and A.6 respectively.
Definition A.S. Suppose that we have a group action : G + X, and the group G decomposes as a direct product G G1 subgroups be referred to as and "6 respectively. Then, the action is disentangled if there is decomposition X X2 x Xn, and actions ; G, x X, + X,, [1,..., ,n) such that:
Gn. Let the action of the full group, and the actions of each
(g "...... .gn) (X1, ,,)) = (9) X1, ... Inn Xn)
for all gi E G; and x; E X,,
Now, to derive the definition of a disentangled representation from the definition of disentangled group action, consider set of world-states, denoted by W. Furthermore, assume that: (a) there is generative process b : W - O leading from world-states to observations, O, (b) and an inference process h O Z leading from observations an agent's representations, Z. With these, consider the composition f : W + Z. f = h b. In terms of transformation, assume that these transforma- tions represented by group G of symmetries acting W via an action G x W - W.
13

---

Under review as a conference paper at ICLR 2023
The overarching goal of disentangling the latent space relies finding corresponding action
x Z + so that the symmetry structure of W reflected in Z. other words, an action on Z corresponding to the action W is desirable. This can be achieved if the following condition is satisfied:
g ((w) = f(g w) EgEG,
In other words, the action, should commute with S. which adheres the definition of the equiv- ariant map, and thus, f is equívariant map, as shown below.
G W
W
ida x f
2
x Z
From Higgins et al. (2018), a disentangled representation can be defined as follows:
Definition A.6. The representation Z is disentangled with respect to G1 x x... Gn if
1. There is an action G Z Z.
2. The map f + equivariant between the actions on W and Z, and
3. There is a decomposition Z -2 ... x Zn or Z Z1 $ ... O Zn such that each Z; is fixed by the action of all Gj. J # and affected only G
A.3 PROOF
Proof of Theorem 3.1
Proof. Suppose that there b is equivariant map defined in Theorem 3.1 and fi is equivariant map. Then
g s(w) = -g- -gh((((w)) -igg b(w))
(9)
(10)
libblg w))
(11)
f( w))
(12)
(13)
Ng E W.
A.4 COMPARISON OF DIFFERENT VAE-BASED MODELS
In our evaluation, we compare the proposed model against seven other VAE-based derivatives, namely, vanilla VAE, 3-VAE, 3-TCVAE, CCI-VAE, FVAE, InfoVAE and WAE. All these mod- els vary based on the underlying regularizer Lyeg( (6). For example, the 3-VAE model constraints on the latent space using 3 to limit the capacity of the latent space, which encourages the model to learn the most efficient representation of the data. The regularization term of these different models (Column 2) are summarized in Table along with relevant notes (Column 3).
14

---

Under review as a conference paper at ICLR 2023
Table Comparison of different VAE-based models w.r. the regularizers they employ.
Model
(o)
Notes
VAE
(eulga(=)e)ilp(t))
8-VAE
3Pxa(1p(2jj))1p p(z))
Usually, 8 is greater tham
В-ТCVAE
T(2.z)
(PPg) ((z,)) Σ, Pec(o(z)))l((ss)
T(, is mutual information
CCI-VAB
8 PPL [p(=)) -]
C is capacity
FVAE
DKL( (xcl_ + II, ((,)))
( Maximum Discrepancy Cov covariance and (+( the ourput eneader.
Ecey (oupst)
DIPVAE
+ LL
We set 10Aod suggested in Kumar 2017
WAE
AM Dige (2(m)
A is regularization coeflicient
LSBDVAE
Die (ne (2)2) [[)) + ADrsan
A Diet. is divergence used Diffusion VAE
A.5 ALGORITIMMS
Algorithm 1: Obtaining A using PCA
Input: X: the entire dataset and o: hyperparameter less than
Output: [w1, 122
[,]
If n > 2:
S 91 S2,
s singular values from PCA(X)
$
51, 82,
5n S S/maz(S)
W1, W2.
wn): round to one decimal place of S (S of each dataset shown in
Table 12)
If there exists such that wy < 1, then w; = Ok
Otherwise:
A a1
Algorithm 2: Interpolation layer
Input: A mini-batch:
{x (xI,.... J.n).
Output: [y]...., Vm
Let x; (I/ (.)))=1,.....)
for k in {1, (1.2...,,1]
for {1, m) {k)
Denote weight w jit..,t mndd(ffztt)
5 x w where NO0,
Algorithm 3: Grid fitting score method
Input: Z [Z.. .... Z:n]: matrix consisting of all latent variables obtained by model.
Each row corresponds to one latent variable.
Output: S
for in {1, 2, 2.., n 1}
for j in {ē,
1, .... n}
Denote Z'd = [2 Z a two dimensional subspace of Z
Create GiJ; a set of variables from a square grid that fits Z
Let d'-1 =
for in range(len(C GD
d = distance G.. id kee?
Si. d'd S: the average of Si.i
15

---

Under review as a conference paper at ICLR 2023
A.6
DISENTANGLEMENT SCORES
Table 4: Disentanglement scores for the XY dataset
Models/Metrics
7-diff
z-var f
dci-rf
jemmig
deimig
GF (e 100) 4 MSEL
DAE
1.00
0.81
0.80
0.21
0.006
AE
1.00
1.00
0.79
0.55
0.50
8.67
0.003
NAE
1.00
1.00
0.84
0.58
0.54
8.32
0.002
8-VAE
1.00
1.00
0.91
0.65
0.63
3.82
0,003
8-TCVAE
1.00
1.00
0.93
0.70
0.69
0.004
CCI-VAE
1.00
1.00
0.98
0.78
0.76
3.25
0.009
FVAE
1.00
1.00
0.96
0.73
0.70
5.58
0.003
Info
1.00
1.00
0.90
0.61
0.60
8.06
0.002
DIPVAE
1.00
1.00
0.98
0.78
0.78
2.64
0,003
WAE
1.00
1.00
0.44
0.35
0.27
5.82
0.003
LSBDVAE
1.00
1.00
0.96
0.72
0.70
5.47
0.003
Table 5: Disentanglement scores for the XYCS dataset
Models/Metrics
z-diff f
Z-var f
dci-rf
jemmig
dcimig
GF (e 100 4
MSE
DAE
1.00
1.00
0.94
0.67
0.75
0.01
0.002
AE
0.95
0.55
0.24
0.22
0.13
18.37
0,001
NAE
0.82
0.30
0.09
0.21
0.08
0.58
0,001
8-VAE
0.79
0.46
0.08
0.18
0.13
0.44
0.001
8-TCVAE
0.72
0.41
0.15
0.24
0.14
0.61
0.002
CCI-VAE
1.00
1.00
0.63
0.47
0.46
0.25
0.002
FVAE
1.00
0.91
0.18
0.21
0.13
0.10
0.004
INfO
0.92
0.54
0.21
0.27
0.14
0.44
0.001
DIPVAE
0.32
0.26
0.001
WAE
0.93
0.38
0.06
0.15
0.05
0.42
0.003
LSBDVAE
1.00
0.76
0.38
0.30
0.28
0.54
0.001
Table 6: Disentanglement scores for the 2D Arrow dataset
Models/Metries
z-diff
z-var
dci-rf t
jemmig t
deimig t
GF (e (0-100)
MSE 4
DAE
1.00
0.85
0.88
0.80
0.79
0.30
0.014
AE
1.00
0.30
0.00
0.29
0.17
4480.97
0.012
VAE
0.99
0.85
0.01
0.19
0.05
16.36
0.059
BAV
1.00
0.88
0.34
0.62
0.55
7.71
0.034
3-TCVAF
1.00
0.87
0.83
0.77
0.73
5.48
0.011
CCI-VAE
1.00
0.84
0.79
0.75
0.70
3.07
0.017
FVAE
1.00
0.90
0.46
0.58
0.52
17.09
0.022
InfOVAE
1.00
0.33
0.22
16.75
0.029
DIPVAE
1.00
0.96
0.85
0.75
0.72
2.55
0.010
WAE
1.00
0.62
0.07
0.24
0.10
1510.34
0.006
LSBDVAE
1.00
0.83
0.83
0.80
0.76
4.07
0.008
16

---

Under review as a conference paper at ICLR 2023
Table 7: Disentanglement scores for the 3D Airplane dataset
Models/Metrics
7-diff f
7-var f
dci-rf
jemmig
deimig
GF (e 100
MSF +
DAE
1.00
1.00
0.80
0.79
0.75
0.28
0.011
AE
1.00
.038
0.00
0.22
0.14
2505.92
0.003
VAE
1.00
1.00
0.01
0.36
0.28
14.96
0.013
8-VAE
1.00
0.09
0.44
7.92
0.015
8-TCVAE
1.00
0.96
0.28
0.52
0.44
15.11
0.009
CCI-VAE
1.00
0.95
0.13
0.49
0.41
5.60
0.013
FVAE
1.00
1.00
0.01
0.39
0.29
14.06
0,011
IAE
1.00
0.98
0.17
0.37
11.86
0.014
DIPVAE
1.00
0.96
0.54
0.51
0.43
7.66
0,004
WAE
1.00
0.82
0.06
0.32
0.17
1392.81
0.002
LSBDVAE
1.00
0.94
0.18
0.49
0.41
11.90
0.002
Table 8: Disentanglement scores for the 3D Teapots dataset
Models/Metries
7-diff
dei-rf
jemmig
dcimig
GF (e 100)
MSE
DAE
1.00
1.00
0.89
0.54
0.53
0.0031
0.002
AE
0.89
0.45
0.16
0.16
0.05
23.22
0.001
VAE
1.00
0.77
0.43
0.29
0.20
0.24
0.001
8-VAE
0.93
0.80
0.47
0.29
0.20
0.14
0.001
8-CVAE
1.00
0.83
0.67
0.35
0.30
0.16
0.001
CCI-VAE
0.91
0.65
0.42
0.35
0.16
0.02
0.003
FVAE
1.00
0.79
0.50
0.17
0.001
InfoVAE
1.00
0.73
0.47
0.31
0,23
0.15
0.001
DIPVAE
1.00
1.00
0.84
0.52
0.50
0.13
0.001
WAE
0.78
0.52
0.15
0.15
0.04
0.20
0.001
LSBDVAE
1.00
0.68
0.51
0.33
0.25
0.15
0.001
Table 9: Disentanglement scores for
Shape dataset
Models/Metrics
z-diff f
z-var f
dci-rf ?
jemmig
deimig
GF (e (e-100) 1
MSE
DAE
1.00
0.93
0.99
0.87
0.90
0.0013
0.0008
AE
0.95
0.70
0.26
0.19
0.13
0.33
0.0006
VAE
0.96
0.81
0.31
0.27
0.22
0.14
0.0008
8-VAE
1.00
0.94
0.93
0.80
0.82
0.01
0.0022
3-TCVAE
0.89
0.75
0.79
0.77
0.74
0.02
0.0012
CCI-VAE
0.96
0.72
0.56
0.59
0.08
0.0008
FVAE
1.00
1.00
0.99
0.86
0.88
0.04
0.0007
0.90
0.80
0.35
0.22
0.16
0.20
0.0007
DIPVAE
0.93
0.79
0.68
0.49
0.52
0.20
0.0007
WAE
0.74
0.45
0.09
0.17
0.08
0.14
0.0017
LSBDVAE
1.00
0.92
0.51
0.42
0.40
0.15
0.0006
Table 10: Disentanglement scores for the 3D Face Model dataset
Models/Mctries
z-diff
z-var
dci-rf 1
jemmig 1
dcimig t
GF (e 100
MSE +
DAE
1.00
1.00
0.65
0.48
0.47
0.02
0.004
AE
0.69
0.47
0.10
0.19
0.05
100.60
0.003
VAE
1.00
0.69
0.47
0.34
0.20
0.47
0.002
3-VAE
1.00
0.92
0.60
0.52
0.44
0.34
0.005
3-TCVAE
1.00
0.92
0.59
0.53
0.47
0.37
0.005
CCI-VAE
1.00
0.90
0.61
0.53
0.44
0.31
0.002
EVAE
1.00
0.64
0.49
0.35
0.22
0.39
0.003
Info
1.00
0.90
0.57
0.50
0.38
0.44
0.002
DIPVAE
1.00
0.36
0.29
0.37
0.003
WAF
1.00
0.78
0.20
0.20
0.13
0.39
0.006
LSBDVAE
1.00
0.88
0.61
0.54
0.43
0.46
0.002
17

---

Under review as a conference paper at ICLR 2023
Table 11: Performance effects when removing the Euler layer (E) or the normalization layer (N)
Datasets
XY
XYC
XYS
Metrics/Moodels
DAE
w/o E
w/o N
DAE
w/o E
w/o
DAE
w/o E
w/o N
7-diff f
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
7-var 1
1.00
1.00
1.00
1.00
0.68
0.58
1.00
0.80
0.36
dci-rf f
0.99
0.59
0.95
0.99
0.44
0.26
0.99
0.54
0.11
jemmig
0.81
0.38
0.71
0.82
0.50
0.27
0.77
0.48
0.20
deimig f
0.70
0.83
0.63
0.27
GF (e 100)
0.21
1.00
0.86
0.03
0.07
0.48
0.02
0.21
2.07
Datascts
XYCS
2D Arrow
3D Airplnac
Metrics/Models
DAE
w/o
w/o N
DAE
w/o E
w/o
DAE
w/o
w/o N
z-diff f
1.00
0.99
0.82
1.00
1.00
1.00
1.00
1.00
1.00
z-var f
1.00
0.82
0.17
0.85
0.65
0.57
1.00
1.00
0.40
dci-rf
0.94
0.57
0.07
0.88
0.04
0.00
0.80
0.00
0.00
jemmig
0.67
0.47
0.15
0.80
0.19
0.25
0.79
0.35
0.13
deimig
0.75
0.59
0.06
0.79
0.53
0.19
0.75
0.27
0.12
GF (e 100) +
0.01
0.21
0.59
0.30
6.73
1213204.08
0.28
1.97
143594.36
Datasets
3D Teapots
3D Shape
3D Face Model
Metrics/Models
DAE
w/o
w/o N
DAE
w/o
w/o N
DAE
w/o E
w/o N
Z-diff f
1.00
0.99
1.00
1.00
0.87
0.90
1.00
0.99
0.73
7-var
1.00
0.81
0.92
0.93
0.70
0.74
1.00
0.57
0.42
dci-rf f
0.89
0.43
0.62
0.99
0.65
0.62
0.65
0.28
0.26
jemmig
0.54
0.22
0.40
0.87
0.53
0.50
0.48
0.31
0.22
deimig f
0.53
0.16
0.31
0.90
0.61
0.47
0.47
0.12
GF (e 100)
0.0031
0.16
0.32
0.0013
0.47
0.44
0.02
0.32
0.71
18

---

Under review as a conference paper at ICLR 2023
A.7 HYPERPARAMETERS
Table 12: S values for different datasets
Dataset
S
XY
[1.0, 1.0]
XYC
[1.0, 1.0, 0.8]
XYS
[1.0, 1.0. 0.8]
XYCS
[1.0, 1.0. 0.8, 0.8]
2D Arrow
[1.0, 1.01
3D Airplane
[1.0, 1.0]
3D Teapots
[1.0, 0.8, 0.8, 0.4, 0.3, 0.3) [1.0, 1.0, 1.0, 1.0, 0.5, 0.5]
3D Shape
3D Face Model
[1.0, 0.4, 0.4, 0.3]
Table 13: All hyperparameters for models.
Model
Values for XYCS dataset
Extra values for the other datasets
DAE (a)
0.5, 0.5, 0.1, 0.05, 0.01]
3-VAE (3)
2.0, 8.0, 16.0, 32.0, 64.0, 128.0
8-TCVAR (3)
2.0,4.0. 8.0, 16.0. 32.0, 64.0, 128.0
CCI-VAE (C)
1100.0, 500., 1000.0]
[50.0]
FVAE (r)
1.00, 10.0, 20.00, 40.0,4
Info (A)
[100.0,500. 1000.0]
[50.0,2000.0]
DIPVAE (A)
1.0, 5.0,, 10.0, 20.0, 50.0, 100.0
WAE (A)
11.0, 5.0, 10.0, 20.0, 30.0, 40.0, 50.0
LSBD (A)
[1.0, 10.0, 100.0)
Table 14: Best hyperparameters for models for the 2D datasets.
Model Dataset
XY
XYC
XYS
XYCS
2D Arrow
DAE (a)
1.0
0.001
0.001
0.01
0.5
8-VAE (5)
16
32
64
16
16
8-TCVAE (8)
32
128
128
32
8
CCI-VAE (C)
500
100
100
FVAE (Y)
40
500
Info (X)
100
20
500
50
DIPVAE (A)
50
100
2
100
20
WAE (A)
20
50
LSBD (A)
100
Table 15: Best hyperparameters for models for the 3D datasets.
/odel Dataset
3D Airplane
3D Teapots
3D Shape
3D Model
DAE (a:)
0.05
0.01
0.01
0.05
3-VAE (5)
6
64
16
3-TCVAE (3)
6
32
32
CCI-VAE (C)
500
50
100
100
FVAE (7)
20
1
5
1
InfoVAE (A)
2000
DIPVAE (A)
100
20
1
2
WAE (A)
5
10
50
50
LSBD (A)
10
100
10
10
19

---

Under review as a conference paper at ICLR 2023
A.8 ENCODER AND DECODER ARCHITECTURES
Table 16: Architecture for the 2D Toy Dataset
Encoder
Decoder
Input 84 84 image 10x Conv J, BN, LReLU
3x3 Conv 4. Sigmoid
10 Conv 1. BN, LReLU
10x 10 16 Conv L, BN, LReLU FC 64
10x 10 Conv 1. BN, LReLU
FC 256, LReLU
FC The number of features
FC 64, LReLU
Table 17: Architecture for the 2D Arrow and 3D Airplane Datasets
Encoder
Decoder
Input x4x64x x image
3 1 Conv &
4x4 4 32 Conv J. BN, LReLU
4x4 1 Conv f. BN, LReLU
4x 4 32 Conv J. BN, LReLU
4x Conv f. BN, LReLU
4x 4 64 Conv J. BN, LReLU
4x4 32 Conv f. BN, LReLU
4 4 64 Conv 4, BN, LRcLU
4x4 64 Conv f, BN, L.ReLU
FC 256
FC 1024, LRRLU
2
FC 256, LReLU
Table 18: Architecture for the 3D Teapots Dataset
Encoder
Decoder
Input 64 64 x 3 image
3 1 Conv 4
4 32 Conv J. BN, ReLU
4 3 Conv 1, BN, ReLU
4 4 32 Conv J, BN, ReLU
4 32 Conv f. BN, ReLU
4x 4 64 Conv 4. BN, ReLU
4 32 Conv f, BN, ReLU
4x 1 64 Conv 4, BN, ReLU
4x4 64 Conv BN, ReLU
FC 128
FC 1024, LReLU
FC
128, LReLU
Table 19: Architecture for the 3D Shape Dataset
Encoder
Decoder
Input 64 64 x image
3x33 Conv 4. Sigmoid
32 Conv J. BN, LReLU
4x 3 Conv 1, BN, I.ReLU
4x 4 32 Conv J. BN, LReLU
4x4 32 Conv f, BN, LReLU
4x 4 64 Conv J. BN, LReLU
4x4 32 Conv 1, BN, LReLU
4x 4 64 Conv J. BN, LReLU
4x4 64 Conv BN, LReLU
FC 256
FC 1024, LReLU
FC
FC 256, LReLU
Table 20: Architecture for the 3D Face Model Dataset
Encoder
Decoder
Input x4x64x image 4x4 32 Conv 4, BN, LReLU
3x3 Conv 4. Sigmoid
4x4 1 Conv 1. BN, LRRLU
4x4 32 Conv J, BN, LReLU
4x4 4 32 Conv 1, BN, LReLU
4x4 64 Cony J. BN, LReLU
4x4 32 Conv 1. BN, LReLU
4 64 Conv 4, BN, LReLU
4x4 64 Conv 1, BN, LReLU
FC 128
FC 1024, LReLU
PC 4
FC 128, LReLU
20

---

Under review as a conference paper at ICLR 2023
A.9 SYSTEM AND MODEL CONFIGURATIONS
All of our experiments were run on a single hardware consisting two DGX2 nodes, collectively consisting of 32-V100 GPUs, 1.5GB GPU RAM, and 3TB System RAM. Encoder and decoder architecture are the same in all experiments. Encoder has two convolutional layers followed by Batch Normalization layer and LeakyReLU activation. After convolutional layers. there is one fully- connected layer with 64 nodes and another layer which maps to the latent space. The decode part is symmetric to the encoder part. for CCI-VAF set as 25 for 2D Toy dataset and as 50 for all other datasets.
A. DATASET
powiior
postion
culor
shape
Figure 4: Four factors in datasets. x and y positions have 53 elements, color has elements and shape has 3 elements.
1. 2D Toy Dataset: This dataset has objects with three shapes (S) (a circles, rectangles and diamonds), and variations to their x and y positions and color information (more specifically, the brightness). This is rather small, but very effective, dataset. There are 53 unique v positions (X), 53 unique positions (Y) and 5 colors (C). We create XY, KYC,XYS and XYCS sub-datasets to show the differences of the latent space when the combination of categorical and continuous factors are presented.
2. 2D Arrow Dataset (Tonnaer et al. 2022): This dataset has 4096, three-channel RGB, x 64 x 3 images of 2D object (arrow) with ground truth factors of 64 colors and 64 fixed in in-plane rotations.
3. 3D Airplane Dataset [Tonnaer al. 2022): This dataset also has 4096, three-channel RGB, 64x644 x4xx images of a 3D object (airplane within the ModelNet40 dataset (Wu et al. 2015)) with ground truth factors of 64 colors and 64 rotations with respect to a vertical axis.
4. 3D Teapots Dataset (Fastwood & Williams 2018): This dataset has 200, 000, three-channel RGB, 64 x 64 x 3 images of a 3D object (teapot) with ground truth factors of independently sampled from its respective distribution: azimuth U0, 2TT). elevation
U[0, ' /2), and three colors, namely, red (R), green (G) and blue (B), sampled with R ~ [ 1], G ~ 1], and B ~ (0, 1]. This dataset very effective evaluate model when all factors are independently from the uniform distributions.
5. 3D Shape Dataset (Burgess & Kim 2018): This dataset has 480, 000, three-channel RGB, 64 x 64 x images of 3D objects with ground truth factors of four shapes, eight scales, 15 orientations, 10 floor color, 10 wall colors, and 10 object colors.
6. 3D Face Model Dataset (Paysan et al. 2009): This dataset has 127, 050, gray-scale, 64 x 64 images of 3D faces with ground truth factors of 50 different face ids, 21 azimuth, 11 elevation and 11 lighting conditions.
7. Blood Cell Dataset (Acevedo et al. 2020): This dataset has 17, 092, 360 x 364 x 3 images of eight types of normal peripheral blood cells. We used the 28 x 28 x resized dataset followed by Yang et al. (202T).
8. Sprites Dataset (Reed et al. 2015): This dataset has 93, 312, 64 x 64 x 3 images from 1296 unique characters with ground truth factors of the color of hair, tops, body, bottom and actions.
21

---

Under review as a conference paper at ICLR 2023
A.11 LATENT SPACES AND RECONSTRUCTIONS OF LATENT TRAVERSALS ACROSS EACH LATENT DIMENSION
(a) Ideal X- relationship
(b) Ideal X-C relationship
(c) Ideal X-S relationship
(4) Ideal C-S relationship
Figure 5: Ideal relationships between X-Y, X-C, X-S and C-S features.
shops
poetier
powiier
(a) X-Y relationship in DAE
(b) X-C relationship in DAE
(c) X-S relationship in DAE
(d) C-S relationship in DAE
Figure 6: Relationships between X-Y, X-C, X-S and C-S features in DAE.
dboce
poe
Boc
X-Y relationship in AE
(b) X-C relationship AE
(c) X-S relationship
(d) C-S relationship AB
Figure 7: Relationships between X-Y, X-C, X-S and C-S features in AE,
shaps
poettor
X-Y relationship VAE
relationship VAE
relationship
(d) selationship VAE
Figure 8: Relationships between X-Y, X-C, X-S and C-S features in VAE.
22

---

Under review as a conference paper at ICLR 2023
por
(a) X-Y relationship in 3-VAE
(b) X-C relationship in 3-VAE
(c) X-S relationship in 3-VAE
(d) C-S relationship in 2-VAE
Figure 9; Relationships between X-Y, X-C, X-S and C-S features in 3-VAE.
ahoce
ahoce
pos
poctior
X-Y relationship in 8-TCVAE
(b) X-C relationship 8-TCVAE
(c) X-S relationship
C-S relationship
8-TCVAE
8-TCVAE
Figure 10: Relationships between X-Y, X-C, X-S and C-S features in 3-TCVAE.
peettr
(a) X-Y relationship in
(b) X-C relationship in
(c) X-S relationship in CCI-VAE
(d) C-S relationship
CCIVAE
CCI-VAE
CCI-VAE
Figure 11: Relationships between X-Y, X-C, X-S and C-S features in CCI-VAE. VAE.
share
Shape
poliier
X-Y relationship in FVAE
(b) selatiooiss in FVAE
(c) X-S relationship FVAB
C-S relationship F'VAE
Figure 12: Relationships between X-Y, X-C, X-S and C-S features in FVAE.
23

---

Under review as a conference paper at ICLR 2023
shappe
relationship in InfoVAE
(b) relationship in InfoVAE
(c) relationship
relationship InfoVAE
InfoVAE
Figure 13: Relationships between X-Y, X-C, X-S and C-S features in Info VAE.
dhice
(a) X-Y relationship in
(b) X-C relationship in
(0) X-S relationship in
(d) C-S relationship in
DIPVAE
DIPVAE
DIPVAE
DIPVAE
Figure 14: Relationships between X-Y, X-C, X-S and C-S features in DIPVAE.
xpootier
relationship WAE
(b) X relationship WAE
X-S relationship WAE
(d) C-S relationship in WAE
Figure 15: Relationships between X-Y, X-C, X-S and C-S features in WAE.
200 Bor
X ppoc
ON
relationship
relationship L.BDDVA..
relationship L.SRDVAR.
relationship
LSBDVVA
ISBDVAE
Figure 16: Relationships between X-Y, X-C, X-S and C-S features LSBDVAE.
24

---

Under review as a conference paper at ICLR 2023
slace
(+) DAE
(b) ATE
ppsdos
CAP
(c) VAE
(d) 3-VAE
CAL
e 3-TCVA
O CCL-VA
Xpatoo
( PYAE
(h) InfoVAE
spatonn
① DIPVAE
i WAF
rpatee
(6 LSRDVAE
Figure 17: Reconstructions of latent traversals across each latent dimension in the XY CS dataset.
25

---

Under review as a conference paper at ICLR 2023
7 >
<
V
(3) DAE
(b) AE
-
7
mm
K K
1
V
R
(c) VAE
(d) 8 VAE
00
ab
7
K K
E
K
y
y
(e) 8 T-VE
( CCI-VAE
7 7 K
K
V
V
V
(g)
(h)
トソン >
-
>
V
レ >
7
K K
DIPVAE
- RA
ト
retater
K
K
7
LSBDVAE
Figure 18: Reconstructions of latent traversals across each latent dimension in the 2D Arrow dataset.
26

---

Under review as a conference paper at ICLR 2023
ebrte
VAE
3 VAE
009
noter
c 3-TCVAE
( CCI-VEE
009
Lg EVAB
(b) InfoVAE
DIPYAE
(i) WAE
retatee
LSBDVAE
Figure 19: Reconstructions of latent traversals across each latent dimension in the 3D Airplane dataset.
27

---

Under review as a conference paper at ICLR 2023
(a) DAE
(b) VAE
(e) (3-VAE
(d) 9-TCVAE
(e) PVAE
InfoVAE
DEr
(b) LSBDVAE
(g) DIPVAE
Figure 20: Reconstructions of latent traversals across each latent dimension in the 3D Teapots dataset. Due to the space limit, we omit the result from AE, CCI-VAE and WAE, which have low scores in 3D Teapots dataset.
28

---

Under review as a conference paper at ICLR 2023
MM
100
(a) DAE
(b) VAE
Alpa
(A-VAE
(6) 3-TCVVE
3M94
XM04
(e)
(
( DIPYAE
(b) LSBDVAE
Figure 21: Reconstructions of latent traversals across each latent dimension in the 3D Shape dataset.Due to the space limit, omit the result from AE, InfoVAE and WAE, which have low scores 3D Shape dataset.
29

---

Under review as a conference paper at ICLR 2023
Ro
Plok
19h0n0
Ignnng
(a) DAE
(b) AE
19h0ng
Igninn
(c) VAE
(d) 3-VAE
Tyd
(e) SCVVA
( CCI-VAE
(e) FVAE
(h) InfoVAE
Tlck
elewsten
Igning
DIPVAE
6 WAE
Bock
(k) LSHDVAE
Figure 22: Reconstructions of latent traversals across each latent dimension in the 3D Face Model dataset.
30

---

Under review as a conference paper at ICLR 2023
( DAE
(b) VAE
Graay
whet
(-VAE
(6) 3-TCVAE
(e)
(1
aetont
(g) DIPYAE
(b) LSBDVAE
Figure 23:
Reconstructions of latent traversals across each latent dimension the Sprites dataset. Due to the space limit, we omit the result from AE, Info and WAE. The ground truth factors are the color of hair, tops, body, bottom and actions.
31

---

Under review as a conference paper at ICLR 2023
(a) DAE
(b) AE
(c) VAE
3-VAE
(c) 3-TCVVE
( CCI-VAE
(g) FVAE
(h) InfoVAE
@ DIPYAE
( WAE
(k) LSBDVAE
Figure 24:
Reconstructions of latent traversals across each latent dimension in the Blood Cell dataset. Some of the models independently learn size and shape of blood cells. For example, for DAE, the first dimension in the latent space corresponds to the size of cells and the second and the third dimensions correspond to changes in shape in different directions.
32

---

Under review as a conference paper at ICLR 2023
Devttannemmet scores for ralaset
Dientanglanctt sares for dalaset of VAE
0.20
1.0
6.20
2:d9
6035
do
6.25
6.30
jemmig
0.300
jammi
0.055
02
6.000
1.0
0.1
2.02
5.025
Obsentsrglement
TCVAE
Disentargllmen: for ofCO VME
7:d7
0.6
0.20
domp
0.200
domp
6005
6.055
2.0
15.0
14.2
2286
0.000
200.0
500.
1300.0
0.000
Gicentanglement scares for XY dotaset of FVAE
Dienttngfonerk soores for dataset of infoie
0.200
0.220
6015
dirf
6015
do-e
Scote
6,300
Iemnia
6.000
(rnemia
0.055
1.0
5.2
15.0
25.9
30.0
40.2
50
200
300.0
1200.0
Disen/ong fo detaset DUPMAE
Civenta WAA
0.320
2:d7
0.6
40/1
0.35
0.355
8:000
0.300
seettannleement scores XY ditaset LSBOUME
0.200
0015
0.330
[emnig
domp
1.5
15,0
200.0
Figure 25: Disentanglement scores with the XY dataset with respect to hyperparameters. For DAE, Wi and we obtained by Algorithm| T are the same in this dataset. Hence, the desirable value for both Wi and W2 is 1. The result shows that when a, lower than 0.05, is assigned to the second dimension, the disentanglement scores also become lower.
33

---

Under review as a conference paper at ICLR 2023
Osentanglomart scores frr autaset D4E
Dieetanniommr immen zares for XYC sanase of HAE
6.35
500e
domp
6.30
(enonia
domng
0.005
0.4
6.005
0.000
2.5
5.0
16.0
32.0
56.2
220.
0.000
Disertanglemeen scones
Disentangle man: scang:
0.300
6.300
c41
0.5
0.15
0.7
8.000
0.000
Ossertnglement Scores XYC datacet FVAE
eenntnnglmmert scores for XVC do sarastt of Infort
did
0.0
0.20
jemmio
dump
0.20
(eniiia
doms
6.05
0.4
6.05
0.000
0.000
1.0
12.0
22.0
36.0
40.2
50.0
10.0
10.3
Ossontnnllemert scores datases
Disentanglemen: scanes dataset
0.220
6.200
2:0T
2:01
0.
0.355
0.6
0.4
12.0
20.2
35.0
se mounttnngmmmetkkkcm us for XYC dataset of LSSOVAE
50me
0.20
(smmg dome
6.05
0.000
1.0
12.0
200.0
Figure 26: Disentanglement scores with the XYC dataset with respect to hyperparameters.
34

---

Under review as a conference paper at ICLR 2023
Denttanglomnnteoo sconss for XYS datasat D4E
Dientargiemar iomer zores for XYS satazet of VAE
0.8
6.35
500e
domp
6.30
(enomia
domp
0.005
6.005
0.000
2.5
9.0
12.0
32.0
54.3
2205
0.000
Oisertanglemmen scores 5 0.300
Cisertanglemen:
6.300
c41
0.15
0.
0.000
0.5
6,000
Dsentanglement scores for XYS dataset of FVAE
Dbentaing lement sooees for XYS sataset of INDMAE
did
0.20
jemmio
dump
0.20
(rniiia
dums
66055
6.05
0.000
0,000
1.0
12.0
22.0
36.0
40.3
50.0
10.0
26.0
16.3
Ossnntnnlommett scores Krs dateser DIPUSE
Disentanleemen danset
0.020
6.200
2:07
2:01
0.5
0.355
62
12.0
20.2
35.0
Dlue Duentaanglemmmsscor for XYS dataset of LSSOVAE
0.20
(smmg dome
6.05
0.000
1.0
12.0
200.
Figure 27: Disentanglement scores with the XYS dataset with respect to hyperparameters.
35

---

Under review as a conference paper at ICLR 2023
Disertanieemnn 2VC5 dataset of DAE
Deertarglemmnntscoe XYCS of VAE
6.35
do
domp
6.30
(enonia
domp
0.205
6.005
0.000
2.5
5.0
18.0
32.0
56.2
:205
0.000
Dsaamnaglemmettsoooo scores of
Diseniang acont
0.300
0.300
c41
0.15
02
0.4
0.000
0,000
Diontanglement soores es for XYCS dataset FYME
Ols tanitannoeeeeoooeeeooo for XVCS dataset of NINAME
did
jemmia
(eniiia
0.20
dump
0.20
domp
6005
6.05
1.0
5.0
20.0
50.0
100.
200
201.0 426.2 506.0
0,000
100.0
506.0
1200.0
0,000
seeettanglamenttsooo sconce
Disorrnneemmentno scores
0.220
2:0T
6.200
2:01
2mmivr
6.355
0.5
02
0.2
20.2
30,0
Disentang memnnt scones XVCS dataset LSBOVAE
50me
0.20
(cmmg domg
6.055
0.000
1.0
12.0
200
Figure 28: Disentanglement scores with the XYCS dataset with respect to hyperparameters.
36


Reviews:
[{"review_id": "yr86g6mRzjL", "reviewer": "Reviewer_JNBF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author proposed a new perspective for learning disentangled representation. The Euler encoding is introduced to force the latent space to achieve a linear disentangled representation. What's more, the author adopts the PCA, normalization layer, and Gaussian interpolation to address the issues caused by the Euler encoding. The author also designed a new metric for comparing the disentanglement of the representation. The proposed method is validated on several toy datasets.\n ", "summary_of_the_review": "Overall, the ideal is novel. The author provided a new insight for representation learning with Euler encoding.\nHowever, the experiments on the nature image dataset are not given. Some classic metrics are not adopted in the comparison experiments.", "strength_and_weaknesses": "\nStrengths:\n1. The author proposed a new perspective for learning disentangled representation, which introduces the Euler encoding to force the latent space to achieve a linear disentangled representation.\n2. The author provides a detailed theoretical analysis.\n3. The proposed method achieves good results on toy datasets.\n\n\nWeaknesses:\n1. Some classic metrics are not adopted to measure the differences between the proposed method and SOTA methods, such as Disentanglement, Completeness, Informativeness， et al.  [1]\n2. Some related multi-dimensional disentangling methods (which are not probabilistic)  with AE as backbone are not included [2,3].\n3. The proposed method is only validated on the toy / synthetic dataset. The author is suggested to add experiments on natural image datasets.\n4. Some typos, such as ' of each of', 'commutative Lie groups'\n5. I am concerned that the proposed method will fail on natural images. The reasons are concluded as follows: the PCA achieves poor results on complicated nature images; The nature images contain more noise.\n\nReferences:\n[1]A Framework for the Quantitative Evaluation of Disentangled Representations[J]. International Conference on Learning Representations, 2018.\n[2]Dual Swap Disentangling, NIPS 2018.\n[3]Image-to-image translation for cross-domain disentanglement, NIPS 2018.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gxSgiRBD5y", "reviewer": "Reviewer_9cKh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a non-probabilistic approach to use autoencoder for learning disentanglement in unsupervised manner. It also proposes a new metric to quantify the disentanglement. ", "summary_of_the_review": "Overall the paper proposes a novel framework to address the problem of disentanglement with unsupervised data. Though the paper is overall sound, the evaluation could be improved by highlighting the differences achieved on some downstream tasks as it's unclear whether the gains are significant. ", "strength_and_weaknesses": "The paper explains the new framework in detail and compares against other autoencoder models on a range of disentanglement metrics.\nAt the same time, all the comparisons are against other autoencoder models and ignores other types of unsupervised deterministic models e.g. \nLearning the Multilinear Structure of Visual Data\nM Wang, Y Panagakis, P Snape, S Zafeiriou\nCVPR 2017\nThe current formulation does not cover the case of disentanglement with missing data.\nEvaluations are based purely on disentanglement metrics and it is not clear how much difference in downstream tasks some gains achieve. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EGim7EiYtfo", "reviewer": "Reviewer_LTGT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an **autoencoder** architecture based on **trigonometric functions** for **unsupervised disentangled representation learning**, introduced a new disentanglement metric (based on a strong assumption), and evaluated the proposed architecture on six datasets.", "summary_of_the_review": "The author tried to solve a challenging problem (unsupervised disentanglement) and conducted abundant experiments. However, the grid assumption is unreliable, their theory does not support the proposed method, and the detailed group theoretical treatment in experiments is unclear. Therefore, I cannot recommend this paper for acceptance to ICLR.", "strength_and_weaknesses": "## Strengths\n\n- Unsupervised disentanglement is an important problem.\n- The author used the algebraic definition of disentanglement.\n- The author compared the proposed architecture with many autoencoder-based methods.\n\n## Weaknesses\n\n- The author tried to support the proposed method with theoretical guarantees, but the theorems seem not to guarantee the learned representation is disentangled.\n- The proposed metric, Grid Fitting Score, relies on the assumption that \"performing independent disentangled actions on a symmetry group causes the corresponding subspace to form a grid-shape latent space\", which may not hold (details below).\n- The groups and group actions in experiments are not given, so the experimental settings are a bit questionable.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4Xu4f6KrZs", "reviewer": "Reviewer_T3ur", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a deterministic VAE for unsupervised disentanglement learning with Euler encodings. By introducing an architectural inductive bias with the Eular layer, this approach can learn disentanglement without using independence regularization terms or additional supervisions. In addition, the authors present several ways to tackle practical issues with PCA, min-max normalization, and Gaussian interpolation. An interesting disentanglement metric that does not need label information is also proposed. Experiments were conducted on several benchmarks with different metrics.", "summary_of_the_review": "The main idea is quite good and this paper includes several interesting points. However, the in-depth analyses of the proposed method are missing and thus the sources of good performance are not identifiable. Furthermore, the empirical support is weak in terms of the applicability for real-world data. I thus find it difficult to argue for acceptance of the work.", "strength_and_weaknesses": "Strength\n- I think designing good explicit inductive biases is particularly important for unsupervised disentanglement learning. The architectural design that enforces the neworks to learn disentangled features can resolve many difficulties in previous methods: For example, the necessity of independence-inducing regularization loss (which can cause the tradeoff between disentanglement and reconstruction) can be removed. In this aspect, I believe this work addressed an important problem and did a good job.\n- I feel the Euler encoding is technically sound and the mathematical notations and derivations are quite solid.\n- The experiments are comprehensive in terms of the number of datasets and evaluation metrics.\n- The paper is well backed-up by supplementary material that contains sufficient details of the method and experiments and presents additional results.\n\nWeakness\n- I was not able to find ablation studies and/or in-depth discussions to argue the effectiveness of the three elements described in Section 3.3. For example, how did the authors set the hyperparameter alpha in Algorithm 1 and what happens with varying the value of alpha? What happens if different normalization schemes are used or any normalization is not applied? Is the Gauassian interpolation layer indeed necessary? It would be very helpful to add experimental/theoretical support regarding them.\n- The architectures used in the experiments consist of vanilla convolutional and transposed convolutional layers with relatively shallow depths. I feel they are a bit outdated; I would recommend the authors to include experimental results using the networks with residual blocks and deeper depths.\n- My major concern is the applicability for real-world datasets; the proposed method was validated melely on simulated datasets. Although many unsupervised disentanglement algorithms also face this issue regarding their practical usage, it would be very helpful to present visual disentanglement results on CelebA or other real-world images to show the potential of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject