Year: 2018

Abstract:
Recent work has shown increased interest in using the Variational Autoencoder (VAE) framework to discover interpretable representations of data in an unsupervised way. These methods have focussed largely on modifying the variational cost function to achieve this goal. However, we show that methods like β-VAE simplify the tendency of variational inference to underfit causing pathological overpruning and over-orthogonalization of learned components. In this paper we take a complementary approach: to modify the probabilistic model to encourage structured latent variable representations to be discovered. Specifically, the standard VAE probabilistic model is unidentifiable: the likelihood of the parameters is invariant under rotations of the latent space. This means there is no pressure to identify each true factor of variation with a latent variable. We therefore employ a rich prior distribution, akin to the ICA model, that breaks the rotational symmetry. Extensive quantitative and qualitative experiments demonstrate that the proposed prior mitigates the trade-off introduced by modified cost functions like β-VAE and TCVAE between reconstruction loss and disentanglement. The proposed prior allows to improve these approaches with respect to both disentanglement and reconstruction quality significantly over the state of the art.

heading: 1 INTRODUCTION
text: Recently there has been an increased interest in unsupervised learning of disentangled representations. The term disentangled usually describes two main objectives: First, to identify each true factor of variation with a latent variable, and second, interpretability of these latent factors (Schmidhuber, 1992; Ridgeway, 2016; Achille & Soatto, 2017). Most of this recent work is inspired by the β-VAE concept introduced in Higgins et al. (2016), which proposes to re-weight the terms in the evidence lower bound (ELBO) objective. In Higgins et al. (2016) a higher weight for the Kullback-Leibler divergence (KL) between approximate posterior and prior is proposed, and putative mechanistic explanations for the effects of this modification are studied in Burgess et al. (2017); Chen et al. (2018). An alternative decomposition of the ELBO leads to the recent variant of β-VAE called βTCVAE (Chen et al., 2018), which shows the highest scores on recent disentanglement benchmarks.
These modifications of the evidence lower bound however lead to a trade-off between disentanglement and reconstruction loss and therefore the quality of the learned model. This trade-off is directly encoded in the modified objective: by increasing the β-weight of the KL-term, the relative weight of the reconstruction loss term is more and more decreased. Therefore, optimization of the modified ELBO will lead to latent encodings which have a lower KL-divergence from the prior, but at the same time lead to a higher reconstruction loss. Furthermore, we discuss in section 2.4 that using a higher weight for the KL-term amplifies existing biases of variational inference, potentially to a catastrophic extent.
There is a foundational contradiction in many approaches to disentangling deep generative models (DGMs): the standard model employed is not identifiable as it employs a standard normal prior which then undergoes a linear transformation. Any rotation of the latent space can be absorbed into the linear transform and is therefore statistically indistinguishable. If interpretability is desired, the modelling choices are setting us up to fail.
We make the following contributions:
• We show that the current state of the art approaches employ a trade-off between reconstruction loss and disentanglement of the latent representation.
• In section 2.3 we show that variational inference techniques are biased: the estimated components are biased towards having orthogonal effects on the data and the number of components is underestimated.
• We provide a novel description of the origin of disentanglement in β-VAE and demonstrate in section 2.4 that increasing the weight of the KL term increases the over-pruning bias of variational inference.
• To mitigate these drawbacks of existing approaches, we propose a family of rotationally asymmetric distributions for the latent prior, which removes the rotational ambiguity from the model. This approach resembles independent component analysis (ICA) for variational autoencoders.
• We propose to use a prior which allows a decomposition of the latent space using independent subspace analysis (ISA) and demonstrate that this prior leads to disentangled representations even for the unmodified ELBO objective. This removes the trade-off between disentanglement and reconstruction loss of existing approaches.
• An even higher disentanglement of the latent space can be achieved by incorporating the proposed prior distribution into the existing approaches β-VAE and β-TCVAE. Since the prior distribution already favours a disentangled representation, the new method dominates previous in terms of the trade-off between disentanglement and model quality.

heading: 2 BACKGROUND
text: We briefly discuss previous work on variational inference in deep generative models and two modifications of the learning objective that have been proposed to learn a disentangled representation. We discuss characteristic biases of variational inference and how the modifications of the learning objective actually accentuate these biases.

heading: 2.1 DISENTANGLED REPRESENTATION LEARNING
text: Variational Autoencoder Kingma & Welling (2014) introduce a latent variable model that combines a generative model, the decoder, with an inference network, the encoder. Training is performed by optimizing the evidence lower bound (ELBO) averaged over the empirical distribution:
LELBO = Eqφ(z|x) [log pθ(x|z)]−DKL(qφ(z|x)‖p(z)) , (1)
where the decoder pθ(x|z) is a deep learning model with parameters θ and each zl is sampled from the encoder zl ∼ qφ(z|x) with variational parameters φ. When choosing appropriate families of distributions, gradients through the samples zl can be estimated using the reparameterization trick. The approximate posterior qφ(z|x) is usually modelled as a multivariate Gaussian with diagonal covariance matrix and the prior p(z) is typically the standard normal distribution.
β-VAE Higgins et al. (2016) propose to modify the evidence lower bound objective and penalize the KL-divergence of the ELBO:
Lβ-ELBO = Eqφ(z|x) [log pθ(x|z)]− βDKL(qφ(z|x)‖p(z)) , (2)
where β > 1 is a free parameter that should encourage a disentangled representation. In Burgess et al. (2017) the authors provide further thoughts on the mechanism that leads to these disentangled representations. However we will show in the following that this parameter introduces a trade-off between reconstruction loss and disentanglement. Furthermore, we show in section 2.4 that this parameter amplifies biases of variational inference towards orthogonalization and pruning.
β-TCVAE In Chen et al. (2018) the authors propose an alternative decomposition of the ELBO, that leads to the recent variant of β-VAE called β-TCVAE. They demonstrate that β-TCVAE allows to learn representations with higher MIG score than β-VAE (Higgins et al., 2016), InfoGAN (Chen et al., 2016) and FactorVAE (Kim & Mnih, 2018). The authors propose to decompose the KL-term in the ELBO objective into three parts and to weight them independently:
Epθ(x) [DKL(qφ(z|x)‖p(z))] = = DKL(qφ(z|x)‖qφ(z)pθ(x)) +DKL(qφ(z)‖ ∏ j qφ(zj)) + ∑ j DKL(qφ(zj)‖p(zj)) . (3)
The first term is the index-code mutual information, the second term is the total correlation and the third term the dimension-wise KL-divergence. Because the index-code mutual information can be viewed as an estimator for the mutual information between pθ(x) and qφ(z), the authors propose to exclude this term when reweighting the KL-term with the β weight. In addition to the improved objective, the authors propose a quantitative evaluation score for disentanglement, the mutual information gap (MIG). They propose to first estimate the mutual information between a latent factor and an underlying generative factor of the dataset. The mutual information gap is then defined as the difference of the mutual information between the highest and second highest correlated underlying factor.

heading: 2.2 RELATED WORK
text: Recent work has shown an increased interest into learning of interpretable representations. In addition to the work mentioned already, we briefly review some of the influential papers: Chen et al. (2016) present a variant of a GAN that encourages an interpretable latent representation by maximizing the mutual information between the observation and a small subset of latent variables. The approach relies on optimizing a lower bound of the intractable mutual information. Kim & Mnih (2018) propose a learning objective equivalent to β-TCVAE, and train it with the density ratio trick (Sugiyama et al., 2012). Kumar et al. (2017) introduce a regularizer of the KL-divergence between the approximate posterior and the prior distribution. A parallel line of research proposes not to train a perfect generative model but instead to find a simpler representation of the data (Vedantam et al., 2017; Hinton et al., 2011b). A similar strategy is followed in semi-supervised approaches that require implicit or explicit knowledge about the true underlying factors of the data (Kulkarni et al., 2015; Kingma et al., 2014; Reed et al., 2014; Baydin et al., 2017; Hinton et al., 2011a; Zhu et al., 2017; Goroshin et al., 2015; Hsu et al., 2017; Denton et al., 2017).

heading: 2.3 ORTHOGONALIZATION AND PRUNING IN VARIATIONAL INFERENCE
text: There have been several interpretations of the behaviour of the β-VAE (Chen et al., 2018; Burgess et al., 2017). Here we provide a complementary perspective: that it enhances well known statistical biases in VI (Turner & Sahani, 2011) to produce disentangled, but not necessarily useful, representations. The form of these biases can be understood by considering the variational objective when written as an explicit lower-bound: the log-likelihood of the parameters minus the KL divergence between the approximate posterior and the true posterior
LELBO = log pθ(x)−DKL(qφ(z|x)‖pθ(z|x)) (4) From this form it is clear that VI’s estimates of the parameters θ will be biased away from the maximum likelihood solution (the maximizer of the first term) in a direction that reduces the KL between the approximate and true posteriors. When factorized approximating distributions are used, VI will therefore be biased towards settings of the parameters that reduce the statistical dependence between the latent variables in the posterior. For example, this will bias learned components towards orthogonal directions in the output space as this reduces explaining away (e.g. in the factor analysis model, VI breaks the degeneracy of the maximum-likelihood solution finding the orthogonal PCA directions, see appendix B.8). Moreover, these biases often cause components to be pruned out (in the sense that they have no effect on the observed variables) since then their posterior sits at the prior, which is typically factorized (e.g. in an over-complete factor analysis model VI prunes out components to return a complete model, see appendix B.8). For simple linear models these effects are not pathological: indeed VI is arguably selecting from amongst the degenerate maximum likelihood solutions in a sensible way. However, for more complex models the biases are more
severe: often the true posterior of the underlying model has significant dependencies (e.g. due to explaining away) and the biases can prevent the discovery of some components. For example, VAEs are known to over-prune (Burda et al., 2015; Cremer et al., 2018).
2.4 β-VAE EMPHASIZES ORTHOGONALIZATION AND PRUNING
What happens to these biases in the β-VAE generalization when β > 1? The short answer is that they grow. This can be understood by considering coordinate ascent of the modified objective. With θ fixed, optimising q finds a solution that is closer to the prior distribution than VI due to the upweighting of the KL term in 2. With q fixed, optimization over θ returns the same solution as VI (since the prior does not depend on the parameters θ and so the value of β is irrelevant). However, since q is now closer to the prior than before, the KL bias in equation 2 will be greater. These effects are shown in the ICA example in appendix B.8. VI (β = 1) learns components that are more orthogonal than the underlying ones, but β = 5 prunes out one component entirely and sets the other two to be orthogonal. This is disentangled, but arguably leads to incorrect interpretation of the data. This happens even though both methods are initialised at the true model. Arguably, the β-VAE is enhancing a statistical bug in VI and leveraging this as a feature. We believe that this can be dangerous, preventing the discovery of the underlying model.

heading: 3 LATENT PRIOR DISTRIBUTIONS FOR UNSUPERVISED FACTORIZATION
text: In this section we describe an approach for unsupervised learning of disentangled representations. Instead of modifying the ELBO-objective, we propose to use certain families of prior distributions p(z), that lead to identifiable and interpretable models. In contrast to the standard normal distribution, the proposed priors are not rotationally invariant, and therefore allow interpretability of the latent space.

heading: 3.1 INDEPENDENT COMPONENT ANALYSIS
text: Independent Component Analysis (ICA) seeks to factorize a distribution into non-Gaussian factors. In order avoid the ambiguities of latent space rotations, a non-Gaussian distribution (e.g. Laplace or Student-t distribution) is used as prior for the latent variables.
Generalized Gaussian Distribution A generalized version of ICA (Lee & Lewicki, 2000; Zhang et al., 2004; Lewicki, 2002; Sinz & Bethge, 2010) uses a prior from the family of exponential power distributions of the form
pICA(z) ∝ exp ( −τ ||z||pp ) (5)
also called generalized Gaussian, generalized Laplacian or p-generalized normal distribution. Using p = 2/(1 + κ) the parameter κ is a measure of kurtosis (Box & Tiao, 1973). This family of distributions generalizes the normal (κ = 0) and the Laplacian (κ = 1) distribution. In general we get for κ > 0 leptokurtic and for κ < 0 platykurtic distributions. The choice of a leptokurtic or platykurtic distribution has a strong influence on how a generative factor of the data is represented by a latent dimension. Fig. 2 depicts two possible prior distributions over latents that represent the
(x,y) spatial location of a sprite in the dSprites dataset (Matthey et al., 2017). The leptokurtic distribution expects most of the probability mass around 0 and therefore favours a projection of the x and y coordinates, which are distributed in a square, onto the diagonal. The platykurtic prior is closer to a uniform distribution and therefore encourages an axis-aligned representation. This example shows how the choice of the prior will effect the latent representation.
Obviously the normal distribution is a special instance of the class of Lp-spherically symmetric distributions, and the normal distribution is the only L2-spherically symmetric distribution with independent marginals. Equivalently (Sinz et al., 2009a) showed that this also generalizes to arbitrary values of p. The marginals of the p-generalized normal distribution are independent, and it is the only factorial model in the class of Lp-spherically symmetric distributions.
We investigate the behaviour of Lp-spherically symmetric distributions as prior distributions for p(z) in the experiments in section 4.

heading: 3.2 INDEPENDENT SUBSPACE ANALYSIS
text: ICA can be further generalized to include independence between subspaces, but dependencies within them, by using a more general prior, the family of Lp-nested symmetric distributions (Hyvärinen & Hoyer, 2000; Hyvärinen & Köster, 2007; Sinz et al., 2009b; Sinz & Bethge, 2010).
Lp-nested Function To start, let’s consider functions of the form( |z1|p0 + (|z2|p1 + |z3|p1) p0 p1 ) 1 p0 , (6)
with p0, p1 ∈ R. This function is a cascade of two Lp-norms. To aid intuition we provide a visualization of this distribution in figure 1a, which depicts (6) as a tree that visualizes the nested structure of the norms. We call the class of functions which employ this structure Lp-nested.
Lp-nested Distribution Given an Lp-nested function f and a radial density ψ0 : R 7→ R+ we define the Lp-nested symmetric distribution following Fernandez et al. (1995) as
pISA(z) = ψ0(f(z))
f(z)n−1Sf (1) , (7)
where Sf (1) is the surface area of the Lp-nested sphere. This surface area can be obtained by using the gamma function:
Sf (R) = Rn−12n ∏ i∈I
∏l1 k=1 Γ [ ni,k pi ] pli−1i Γ [ ni pi
] , (8) where li is the number of children of a node i, ni is the number of leaves in a subtree under the node i, and ni,k is the number of leaves in the subtree of the k-th children of node i. For further details we refer the reader to the excellent work of Sinz & Bethge (2010).
Independent Subspace Analysis The family of Lp-nested distributions allows a generalization of ICA called independent subspace analysis (ISA). ISA uses a subclass of Lp-nested distributions, which are defined by functions of the form
f(z) =   n1∑ j=1 |zj |p1  p0 p1 + · · ·+  n∑ j=n1+···+nl−1+1 |zj |pl  p0 pl  1 p0 , (9)
and correspond to trees of depth two. The tree structure of this subclass of functions is visualized in figure 1b where each vi, i = 1, . . . , l0 denotes the function value of the Lp-norm evaluated over a node’s children. The components zj of z that contribute to each vi form a subspace
Vi = { zj ∣∣∣ j = a . . . b with a = i−1∑ k=1 nk + 1, b = a+ ni } . (10)
Sinz & Bethge (2010) showed that the subspaces V1, . . . ,Vl0 become independent when using the radial distribution
ψ0(v0) = p0v
n−1 0 Γ [ n p0 ] s n p0 exp
( −v p0 0
s
) (11)
which we can interpret as a generalization of the Chi-distribution.
ISA-VAE We propose to choose the latent prior pISA(z) (Eq. 7) with f(z) from the family of ISA models of the form of Eq. 9, which allows us to define independent subspaces in the latent space. The Kulback-Leibler divergence of the ELBO-objective can be estimated by Monte-Carlo sampling. This leads to an ELBO-objective of the form
LISA-VAE = Ez∼qφ(z|x) [log pθ(x|z) + log pISA(z)− log qφ(z|x)] , (12) which only requires to compute the log-density of the prior that is readily accessible from the density defined in Eq. 7. As discussed in Roeder et al. (2017) this form of the ELBO even has potential advantages (variance reduction) in comparison to a closed form KL-divergence.
ISA-TCVAE The proposed latent prior can also be combined with the β-TCVAE approach and we get the objective
LISA-TCVAE = Ez∼qφ(z|x) [log pθ(x|z)]− Iq(z;x)− βDKL(qφ(z)‖ ∏ j qφ(zj))− ∑ j DKL(qφ(zj)‖pISA(zj)) ,
(13)
where Iq denotes the index code mutual information. To compute the terms in Eq. 13, Chen et al. (2018) propose a Monte-Carlo sampling approach called minibatch-weighted sampling, which also only requires to compute the log density of the prior.
Sampling and the Reparameterization Trick If we want to sample from the generative model we have to be able to sample from the prior distribution. Sinz & Bethge (2010) describe an exact sampling approach to sample from an Lp-nested distribution, which we reproduce as Algorithm 1 in the appendix. Note that during training we only have to sample from the approximate posterior qφ, which we do not have to modify and which can remain a multivariate Gaussian distribution following the original VAE approach. As a consequence, the reparameterization trick can be applied (Kingma & Welling, 2014).
Experiments in the following section demonstrate that the proposed prior supports unsupervised learning of disentangled representation even for the unmodified ELBO objective (β = 1).

heading: 4 EXPERIMENTS
text: In our experiments, we evaluate the influence of the proposed prior distribution on disentanglement and on the quality of the reconstruction on the dSprites dataset (Matthey et al., 2017), which contains images of three different shapes undergoing transformations of their position, scale and rotation.
Disentanglement Metrics To provide a quantitative evaluation of the disentanglement we compute the disentanglement metric Mutual Information Gap (MIG) that was proposed in Chen et al. (2018). The MIG score measures how much mutual information a latent dimension shares with the underlying factor, and how well this latent dimension is separated from the other latent factors. Therefore the MIG measures the two desired properties usually referred to with the term disentanglement: a factorized latent representation, and interpretability of the latent factors. Chen et al. (2018) compare the MIG metric to existing disentanglement metrics (Higgins et al., 2016; Kim & Mnih, 2018) and demonstrate that the MIG is more effective and that the other metrics do not allow to capture both properties in a desirable way.
Reconstruction Quality To quantify the reconstruction quality, we report the expected (log-)likelihood of the reconstructed data Eqφ(z|x) [log pθ(x|z)]. In our opinion this measure is more informative than the ELBO, frequently reported in existing work, e.g. (Chen et al., 2018), especially when varying the β parameter, the weighting of the KL term, which is part of the ELBO.
Comparison Baselines Chen et al. (2018) demonstrate that β-TCVAE, a modification of the βVAE, enables learning of representations with higher MIG score than β-VAE (Higgins et al., 2016), InfoGAN (Chen et al., 2016), and FactorVAE (Kim & Mnih, 2018). Therefore we choose to compare against β-TCVAE and β-VAE in our experiments.
Architecture of the Encoder and Decoder To allow an accurate comparison we use the same architecture for the decoder and encoder as presented in Chen et al. (2018). We reproduce the description of the encoder and decoder in appendix A.5
Choosing the ISA-layout We follow the practice of Sinz et al. (2009b) and perform a search over different values for the parameters of the ISA model and choose the model with the highest disentanglement (MIG) score. A comparison of different parameter settings and more details about this procedure are provided in appendix A.1

heading: 4.1 SUPPORT OF THE PRIOR TO LEARN DISENTANGLED REPRESENTATIONS
text: First, we investigate the ability of the prior to support unsupervised learning of disentangled representations for the unmodified ELBO-objective. Figure 3a depicts the structure of the latent representation after training for ISA-VAE, a combination of the Lp-nested ISA prior with the standard VAE approach. Because our prior allows independent subspaces the latent space becomes interpretable even when using the unmodified ELBO objective with β = 1. The plots were produced with the reference implementation of Chen et al. (2018) for visualizing latent representations for the dSprites dataset. When combining the ISA-model with β-TCVAE and varying the β parameter, a model with a high disentanglement score of MIG = 0.54 can be reached. This is the so far highest score reported for dSprites in the literature.

heading: 4.2 QUANTITATIVE COMPARISON OF THE DIFFERENT APPROACHES
text: This benefit of the proposed prior, that it encourages disentangled representations becomes even more obvious in our quantitative comparison. We compare the approaches ISA-VAE and ISATCVAE, that use the proposed Lp-nested prior pISA with their respective counterpart β-VAE (Higgins et al., 2016) and β-TCVAE (Chen et al., 2018) that use the standard normal prior pN . Because the amount of disentanglement depends on the choice of the parameter β, we vary β in the interval between 1 and 6 with a stepsize of 0.5. We compare the performance of the four different approaches in figure 4 with 16 experiments for each value of β. Clearly in both cases a higher disentanglement score can be achieved already for smaller values of β with ISA-β-VAE and ISA-β-TCVAE in comparison to the original approaches. Even when choosing the individually best value of β, that reaches the highest MIG score for each method, the poposed approaches reach a higher MIG score than their respective counterparts. Fig. 5 depicts the distribution of MIG scores for the individually best value of β: Both average and mean of the MIG scores are higher for the variants that use the ISA-model.

heading: 4.3 TRADE-OFF BETWEEN DISENTANGLEMENT AND RECONSTRUCTION LOSS
text: Since the proposed prior facilitates learning of disentangled representations, not only a higher disentanglement score can be reached, but also higher scores are reached for smaller values of β, when compared to the original approaches. This leads to a clear improvement of the trade-off between disentanglement and reconstruction loss. The improvement of this trade-off is demonstrated in figure 4, where we plot both the disentanglement score and the reconstruction loss for varying values
of β. ISA-β-VAE and ISA-β-TCVAE reach high values of the disentanglement score for smaller values of β which and at the same time preserves a higher quality of the reconstruction than the respective original approaches. At the same time we observe that with the proposed prior the quality of the reconstruction decreases at a smaller rate than for the original approaches.
This improvement of the trade-off between disentanglement and the reconstruction loss becomes also obvious in figure 5b where we plot the MIG score with respect to the reconstruction loss for the individually best value of β. The proposed approaches ISA-VAE and ISA-TCVAE allow higher MIG scores than their respective base lines while at the same time providing a better reconstruction quality, almost reaching the reconstruction quality of the standard, non-disentangling VAE. Interestingly the plot also shows that the increase of the MIG score for the baseline method β-TCVAE comes at the cost of a much lower reconstruction quality. This difference in the reconstruction quality becomes obvious in the quality of the reconstructed images. Please refer to the appendix where we present latent traversals in appendix A.3 and image reconstruction experiments in appendix A.4. With the proposed approach ISA-TCVAE the reconstruction quality can be increased significantly while at the same time providing a higher disentanglement.

heading: 5 CONCLUSION
text: We presented a structured prior for unsupervised learning of disentangled representations in deep generative models. We choose the prior from the family of Lp-nested symmetric distributions which enables the definition of a hierarchy of independent subspaces in the latent space. In contrast to the standard normal prior that is often used in training of deep generative models the proposed prior is not rotationally invariant and therefore enhances the interpretability of the latent space. We demonstrate in our experiments, that a combination of the proposed prior with existing approaches for unsupervised learning of disentangled representations allows a significant improvement of the trade-off between disentanglement and reconstruction loss.

heading: A APPENDIX

heading: A.1 CHOOSING THE ISA-LAYOUT
text: We vary l0 between 4 and 10 and choose the same value for l1 = l2,...,l0 between 2 and 10. We set the parameter range of the exponents pi to pi ∈ [0.9, 2.4] with a discretization step size of 0.1, which includes lepto- and platokyrtic distributions. Fig. 2 depicts how lepto- and platykurtic distributions at the child subspaces lead to different representations of the x and y coordinate. Because the MIG metric evaluates axis-alignment of the latent dimensions to the underlying factors, here the x and y coordinate, platykurtic priors in general achieve a higher MIG score. The child subspaces share the same parameter p1 = p2,...,l0 and we choose the exponent of the root node as p0 6= p1 to ensure independence of the subspaces. To study the influence of the layout on the reconstruction quality and MIG score we compare the results for different values of p0, p1,...,5 and l1, and vary the value of β in the interval β ∈ [1, 4] with a step size of 0.5 and repeat each experiment four times. We compare four layouts with the highest MIG score for each subspace layout in Fig. 6 where we plot the mean and standard error of MIG score and reconstruction loss.
A.2 SAMPLING FROM Lp-NESTED SYMMETRIC DISTRIBUTIONS
Algorithm 1: Exact sampling algorithm for Lp-nested symmetric distributions from Sinz & Bethge (2010) Input : The radial distribution ψ0(v0) of an Lp-nested symmetric distribution pLp for the Lp-nested function f Output: Sample x from pLp
1. Sample v0 from a beta distribution β[n, 1] 2. For each inner node i of the tree associated with f , sample the auxiliary variable si from a
Dirichlet distribution Dir [ ni,1 pi , . . . , ni,l1 pi ] where ni,k are the number of leaves in the subtree under node i, k. Obtain coordinates on the Lp-nested sphere within the positive orthant by si 7→ s 1 pi i = ũi (the exponentiation is taken component-wise)
3. Transform these samples to Cartesian coordinates by vi · ũi = vi,1:li for each inner node, starting from the root node and descending to lower layers. The components of vi,1:li constitute the radii for the layer direct below them. If i = 0, the radius had been sampled in step 1
4. Once the two previous steps have been repeated until no inner node is left, we have a sample x from the uniform distribution in the positive quadrant. Normalize x to get a uniform sample from the sphere u = xf(x)
5. Sample a new radius ṽ0 from the radial distribution of the target radial distribution ψ0 and obtain the sample via x̃ = ṽ0 · u
6. Multiply each entry xi of x̃ by and independent sample zi from the uniform distribution over {−1, 1}.

heading: A.3 DISENTANGLEMENT REPRESENTATIONS AND LATENT TRAVERSALS
text: We follow standard practice established in Chen et al. (2018) for visualizing latent representations and additionally show images generated by traversals of the latent along the respective axis. The red and blue color coding in the first column denotes the value of the latent variable for the respective x,y-coordinate of the sprite in the image. Colored lines indicate the object shape with red for ellipse, green for square, and blue for heart.
ISA-layouts: ISA-VAE: l0 = 5, l1,...,5 = 5, p0 = 2.1, p1,...,5 = 2.2 ISA-TCVAE l0 = 5, l1,...,5 = 4, p0 = 2.1, p1,...,5 = 2.2
A.4 IMAGE RECONSTRUCTION RESULTS

heading: A.5 MODEL ARCHITECTURE (PYTORCH)
text: The models were trained with the optimization algorithm Adam (Kingma & Ba, 2015) using a learning rate parameter of 0.01
All unmentioned hyperparameters are PyTorch v0.41 defaults.
c l a s s MLPEncoder ( nn . Module ) : d e f i n i t ( s e l f , o u t p u t d i m ) :
s u p e r ( MLPEncoder , s e l f ) . i n i t ( ) s e l f . o u t p u t d i m = o u t p u t d i m
s e l f . f c 1 = nn . L i n e a r ( 4 0 9 6 , 1200) s e l f . f c 2 = nn . L i n e a r ( 1 2 0 0 , 1200) s e l f . f c 3 = nn . L i n e a r ( 1 2 0 0 , o u t p u t d i m )
s e l f . conv z = nn . Conv2d ( 6 4 , o u t p u t d i m , 4 , 1 , 0 )
# s e t u p t h e non− l i n e a r i t y s e l f . a c t = nn . ReLU( i n p l a c e =True )
d e f f o r w a r d ( s e l f , x ) : h = x . view (−1 , 64 ∗ 64) h = s e l f . a c t ( s e l f . f c 1 ( h ) ) h = s e l f . a c t ( s e l f . f c 2 ( h ) ) h = s e l f . f c 3 ( h ) z = h . view ( x . s i z e ( 0 ) , s e l f . o u t p u t d i m ) r e t u r n z
c l a s s MLPDecoder ( nn . Module ) : d e f i n i t ( s e l f , i n p u t d i m ) :
s u p e r ( MLPDecoder , s e l f ) . i n i t ( ) s e l f . n e t = nn . S e q u e n t i a l (
nn . L i n e a r ( i n p u t d i m , 1 2 0 0 ) , nn . Tanh ( ) , nn . L i n e a r ( 1 2 0 0 , 1 2 0 0 ) ,
nn . Tanh ( ) , nn . L i n e a r ( 1 2 0 0 , 1 2 0 0 ) , nn . Tanh ( ) , nn . L i n e a r ( 1 2 0 0 , 4096)
)
d e f f o r w a r d ( s e l f , z ) : h = z . view ( z . s i z e ( 0 ) , −1) h = s e l f . n e t ( h ) mu img = h . view ( z . s i z e ( 0 ) , 1 , 64 , 64) r e t u r n mu img
Architecture of the encoder and decoder which is identical to the architecture in Chen et al. (2018).
B TOY EXAMPLES SHOWING BIASES IN VI AND β-VI
This section provides the details of the toy examples that reveal the biases in variational methods.
First we will consider the factor analysis model showing that VI breaks the degeneracy of the maximum-likelihood solution to 1) discover orthogonal weights that lie in the PCA directions, 2) prune out extra components in over-complete factor analysis models, even though there are solutions with the same likelihood that preserve all components. We also show that in these examples the β-VI returns identical model fits to VI regardless of the setting of β.
Second, we consider an over-complete ICA model and initialize using the true model. We show that 1) VI is biased away from the true component directions towards more orthogonal directions, and 2) β-VI with a modest setting of β = 5 prunes away one of the components and finds orthogonal directions for the other two. That is, it finds a disentangled representation, but one which does not reflect the underlying components.

heading: B.1 BACKGROUND
text: The β-VAE optimizes the modified free-energy, Fβ(q(z1:N ), θ), with respect to the parameters θ and the variational approximation q(z1:N ), Fβ(q(z1:N ), θ) = Eq(z1:N )(log p(x1:N |z1:N , θ))− βKL(q(z1:N )||p(z1:N )). Consider the case where M = 1β is a positive integer, M ∈ N, we then have
Fβ(q(z1:N ), θ) = N∑ n=1 [ Eq(zn)(M(β) log p(xn|zn, θ))−KL(q(zn)||p(zn)) ] In this case, the β-VAE can be thought of as attaching M replicated observations to each latent variable zn and then running standard variational inference on the new replicated dataset. This can equivalently be thought of as raising each likelihood p(xn|zn, θ) to the power M . Now in real applications β will be set to a value that is greater than one. In this case, the effect of β is the opposite: it is to reduce the number of effective data points per latent variable to be less than one M < 1. Or equivalently we raise each likelihood term to a power M that is less than one. Standard VI is then run on these modified data (e.g. via joint optimization of q and θ).
Although this view is mathematically straightforward, the perspective of the β-VAE i) modifying the dataset, and ii) applying standard VI, is useful as it will allow us to derive optimal solutions for the variational distribution q(z) in simple cases like the factor analysis model considered next.

heading: B.2 FACTOR ANALYSIS
text: Consider the factor analysis generative model. Let x ∈ RL and z ∈ RK . for n = 1...N
zn ∼ N (0, I), xn ∼ N (Wzn, D) where D = diag([σ21 , ..., σ2D]) (14)
The true posterior is a Gaussian p(zn|xn, θ) = N (z;µz|x,Σz|x) where µz|x = Σz|xW >D−1x and Σz|x = (W>D−1W + I)−1. (15)
The true log-likelihood of the parameters is
log p(x1:N |θ) = N∑ n=1 logN (xn,0,WW> +D)
= −N 2 log det(2π(WW> +D))− 1 2 N∑ n=1 x>n (WW > +D)−1xn
= −1 2 N [ log det(2π(WW> +D)) + trace((WW> +D)−1(µxµ > x + Σx)) ]
Here we have defined the empirical mean and covariance of the observations µx = 1N ∑N n=1 xn and Σx = 1 N ∑N n=1(xn − µx)(xn − µx)> i.e. the sufficient statistics. The true likelihood is invariant under orthogonal transformations of the latent variables: z′ = Rz where RR> = I.
Interpreting β-VI as running VI in a modified generative model (see previous section) we have the new generative process
for n = 1...N zn ∼ N (zn;0, I), for m = 1...M(β)
xn,m ∼ N (Wzn, D) where D = diag([σ21 , ..., σ2D]) We now observe data and set xn,m = xn.
The posterior is again Gaussian p(zn|xn, θ,M(β)) = N (zn; µ̃z|x(β, n), Σ̃z|x(β)) where
µ̃z|x(β, n) = Σ̃ −1 z|x(β)M(β)W >D−1xn and Σ̃z|x(β) = (M(β)W>D−1W + I)−1
Here we have taken care to explicitly reveal all of the direct dependencies on β. Mean-field variational inference, q(zn) = ∏ k qn,k(zk,d), will return a diagonal Gaussian approximation to the true posterior with the same mean and matching diagonal precision,
q(zn|xn, θ,M(β)) = N ( zn; µ̃z|x(β, n),Σq(β) ) where Σ−1q (β) = diag ( Σ̃−1z|x(β) ) We notice that the posterior mean is a linear combination of the observations µ̃z|x(β, n) = R(β)xn where R(β) = Σ̃z|x(β)M(β)W>D−1 are recognition weights. Notice that the recognition weights and the posterior variances are the same for all data points: they do not depend on n. The free-energy is then
F(q, θ, β) = Eq(z)(log p(x|z))−KL(q(z)|p(z)) with the reconstruction term being
Eq(z)(log p(x|z)) = − 1
2β N∑ n=1 x>n (D −1 − 2R>W>D−1 +R>W>D−1WR)xn
− N 2β log det(2πD)− N 2β trace(W>D−1Σq)
= −N 2β
( trace ( (D−1 − 2R>W>D−1 +R>W>D−1WR)(Σx + µxµ>x ) ) + log det(2πD) + trace(W>D−1WΣq) ) (16)
and the KL or regularization term being
KL(q(z)|p(z)) = −NK 2 − N 2 log det(Σq) + N 2 trace(Σq) + 1 2 N∑ n=1 x>nR >Rxn
= −N 2
( K + log det(Σq)− trace(Σq)− trace(R>R(Σx + µxµ>x )) ) .
We will now consider the objective functions and the posterior distributions in several cases to reason about the parameter estimates arising from the methods above.

heading: B.3 EXPERIMENT 1: MEAN FIELD VI APPLIED TO FACTOR ANALYSIS YIELDS THE PCA DIRECTIONS
text: Consider the situation where we know a maximum likelihood solution of the weights WML. For simplicity we select the solution WML which has orthogonal weights in the observation space. We
then rotate this solution by an amount θ so that W ′ML = R(θ)WML. The resulting weights are no longer orthogonal (assuming the rotation is not an integer multiple of π/2). We compute the loglikelihood (which will not change) and the free-energy (which will change) and plot the true and approximate posterior covariance (which does not depend on the datapoint value xn).
First here are the weights are aligned with the true ones. The log-likelihood and the free-energy take the same value of -17.82 nats.
1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 z1
1.00
0.75
0.50
0.25
0.00
0.25
0.50
0.75
1.00
z 2
posterior true variational approx.
0.8 0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 x1
0.8
0.6
0.4
0.2
0.0
0.2
0.4
0.6
0.8
x 2
weight rotation data covariance original weight rotated weight
Second, here are the weights rotated π/4 and the log-likelihood is -17.82 nats and the free-energy -57.16 nats.
0.6 0.4 0.2 0.0 0.2 0.4 0.6 z1
0.6
0.4
0.2
0.0
0.2
0.4
0.6
z 2
posterior true variational approx.
0.8 0.6 0.4 0.2 0.0 0.2 0.4 0.6 0.8 x1
0.8
0.6
0.4
0.2
0.0
0.2
0.4
0.6
0.8
x 2
weight rotation data covariance original weight rotated weight
When varying the rotation away fom the orthogonal setting, θ, the plots above indicate that orthogonal settings of the weights (θ = mπ/2 where m = 0, 1, 2, ...) lead to factorized posteriors. In these cases the KL between the approximate posterior and the true posterior is zero and the free-energy is equal to the log-likelihood. This will be the optimal free-energy for any weight setting (due to the fact that it is equal to the true log-likelihood which is maximal, and the free-energy is a lower bound of this quantity.) For intermediate values of θ the posterior is correlated and the free-energy is not tight to the log likelihood.
Now let’s plot the free-energy and the log-likelihood as θ is varied. This shows that the free-energy prefers orthogonal settings of the weights as this leads to factorized posteriors, even though the loglikelihood is insensitive to θ. So, variational inference recovers the same weight directions as the PCA solution.
0 /2 3 /2 2
43.43
43.42
43.41
43.40
43.39
43.38
43.37 43.36 ev id en ce
dependence of free-energy on weight rotation ELBO log-likelihood
The above shows that the bias inherent in variational methods will cause them to break the symmetry in the log-likelihood and find orthorgonal latent components. This occurs because orthoginal components result in posterior distributions that are factorized. These are then well-modelled by the variational approximation and result in a small KL between the approximate and true posteriors.

heading: B.4 EXPERIMENT 2: MEAN FIELD VI APPLIED TO OVER-COMPLETE FACTOR ANALYSIS PRUNES OUT THE ADDITIONAL LATENT DIMENSIONS
text: A similar effect occurs if we model 2D data with a 3D latent space. Many settings of the weights attain the maximum of the likelihood, including solutions which use all three latent variables. However, the optimal solution for VI is to retain two orthogonal components and to set the magnitude of the third component to zero. This solution a) returns weights that maximise the likelihood, and b) has a factorised posterior distribution (the pruned component having a posterior equal to its prior) that therefore incurs no cost KL(q(z)||p(z|x, θ)) = 0. In this way the bound becomes tight. Here’s an example of this effect. We consider a model of the form:
x = α√ 2 [ 1 1 ] z1 + β√ 2 [ 1 1 ] z2 + ρ√ 2 [ 1 −1 ] z3 + (17)
We set α2 + β2 = 1 so that all models imply the same covariance and set this to be the maximum likelihood covariance by construction. We then consider varying α from 0 to 1/2. The setting equal to 0 attains the maximum of the free-energy, even though it has the same likelihood as any other setting.
0.0 0.1 0.2 0.3 0.4 0.5 magnitude of additional component
35
30
25
20
15
ev id
en ce
dependence of free-energy on weight magnitude ELBO log-likelihood
B.5 EXPERIMENT 3: THE β-VAE ALSO YIELDS THE PCA COMPONENTS, CHANGING β HAS NO EFFECT ON THE DIRECTION OF THE ESTIMATED COMPONENTS IN THE FA MODEL
How does the setting of β change things? Here we rerun experiment 1 for different values of β.
0 /2 3 /2 2
44.0
43.9
43.8
43.7
43.6
43.5
43.4
ev id
en ce
dependence of free-energy on beta log-likelihood ELBO beta = 1 ELBO, beta = 1.5 ELBO, beta = 2
In this example, changing β in this example just reduces the amplitude of the fluctuations in the free-energy, but it does not change the directions found. A similar observation applies to the pruning experiment.
Increasing β will increase the uncertainty in the posterior as it is like reducing the number of observations (or increasing the observation noise, from the perspective of q).

heading: B.6 SUMMARY OF FACTOR ANALYSIS EXPERIMENTS
text: The behaviours introduced by the β-VAE appear relatively benign, and perhaps even helpful, in the linear case: VI is breaking the degeneracy of the maximum likelihood solution in a sensible way: selecting amongst the maximum likelihood solutions to find those that have orthogonal components and removing spurious latent dimensions. This should be tempered by the fact that the β generalization recovered precisely the same solutions and so it was necessary to obtain the desired behaviour in the PCA case.
Similar effects will occur in deep generative models, not least since these typically also have a Gaussian prior over latent variables, and these latents are initially linearly transformed, thereby resulting in a similar degeneracy to factor analysis.
However, the behaviours above benefited from the fact that maximum-likelihood solutions could be found in which the posterior distribution over latent variables factorized. In real world examples, for example in deep generative models, this will not be case. In such cases, these same effects will cause the variational free-energy and its β-generalization to bias the estimated parameters far away from maximum-likelihood settings, toward those settings that imply factorized Gaussian posteriors over the latent variables.
B.7 INDEPENDENT COMPONENT ANALYSIS
We now apply VI and the β free-energy method to ICA. We’re interested the properties of the variational objective and the β-VI objective and so we 1. fit the data using the true generative model to investigate the biases in VI and β-VI 2. do not use amortized inference, just optimizing the approximating distributions for each data point (this is possible for these small examples).
The linear independent component analysis generative model we use is defined as follows. Let x ∈ RL and z ∈ RK .
for n = 1...N for k = 1...K
zn,k ∼ Student-t(0, σ, v), xn ∼ N (Wzn, D) where D = diag([σ21 , ..., σ2D])
We apply mean-field variational inference, q(zn) = ∏ k qn,k(zk,d), and use Gaussian distributions for each factor qn,k(zn,k) = N (zn,k;µn,k, σ2n,k). The free-energy is computed as follows: The reconstruction term is identical to PCA: an avergage of a quadratic form wrt to a Gaussian, which is analytic. The KL is broken down into the differential entropy of q which is also analytic and the cross-entropy with the prior which we evaluate by numerical integration (finite differences). There is a cross-entropy term for each latent variable which is one reason why the code is slow (requiring N 1D numerical integrations). The gradient of the free-energy wrt the parameters W and the means and variances of the Gaussian q distributions are computed using autograd.
In order to be as certain as possible that we are finding a global maximum of the free-energies, all experiments initialise at the true value of the parameters and then ensure that each gradient step improves the free-energy. Stochastic optimization or a procedure that accepted all steps regardless of the change in the objective would be faster, but they might also move us into the basis of attraction of a worse (local) optima.

heading: B.8 EXPERIMENT 1: LEARNING IN OVER-COMPLETE ICA
text: Now we define the dataset. We use a very sparse Student’s t-distribtion with v = 3.5. For v < 4 the the kurtosis is undefined so the model is fairly simple to estimate model (it’s a long way away from the degenerate factor analysis case which is recovered in the limit v →∞). We use three latent components and a two dimensional observed space. The directions of the three weights are shown in blue below with data as blue circles.
10 5 0 5 10 x1
10
5
0
5
10
x 2
First we run variational inference finding components (shown in red below) which are more orthogonal than the true directions. This bias is in this directions as this reduces the dependencies (explaining away) in the underlying posterior.
10 5 0 5 10 x1
10
5
0
5
10
x 2
Second we run β-VI with β = 5. Two components are now found that are orthogonal with one component pruned from the solution.
20 15 10 5 0 5 10 15 20 x1
20
10
0
10
20
x 2
In this case the bias is so great that the true component directions are not discovered. Instead the components are forced into the orthogonal setting regardless of the structure in the data.

heading: B.9 SUMMARY OF INDEPENDENT COMPONENT ANALYSIS EXPERIMENT
text: The ICA example illustrates that this approach – of relying on a bias inherent in VI to discover meaningful components – will sometimes return meaningful structure (e.g. in the PCA experiments above). However it does not seem to be a sensible way of doing so in general. For example, explaining away often means that the true components will be entangled in the posterior, as is the case in the ICA example, and the variational bias will then move us away from this solution. The β-VI generalisation only enhances this undesirable bias.

Review summaries:
This paper presents a methodology to bring together independent subspace analysis and variational auto-encoders. Naturally, in order to do that, the authors propose a specific family of prior distributions that lead to subspace independence the Lp-nested distribution family. This prior distribution is then used to learn disentangled and interpretable representations. The mutual information gap is taken as the measure of disentanglement, while the reconstruction loss measures the quality of the representation. Experiments on the sPrites dataset are reported, and comparison with the state of the art shows some interesting results.