Year: 2020

Abstract:
We make two theoretical contributions to disentanglement learning by (a) defining precise semantics of disentangled representations, and (b) establishing robust metrics for evaluation. First, we characterize the concept “disentangled representations” used in supervised and unsupervised methods along three dimensions– informativeness, separability and interpretability–which can be expressed and quantified explicitly using information-theoretic constructs. This helps explain the behaviors of several well-known disentanglement learning models. We then propose robust metrics for measuring informativeness, separability, and interpretability. Through a comprehensive suite of experiments, we show that our metrics correctly characterize the representations learned by different methods and are consistent with qualitative (visual) results. Thus, the metrics allow disentanglement learning methods to be compared on a fair ground. We also empirically uncovered new interesting properties of VAE-based methods and interpreted them with our formulation. These findings are promising and hopefully will encourage the design of more theoretically driven models for learning disentangled representations.

heading: 1 INTRODUCTION
text: Disentanglement learning holds the key for understanding the world from observations, transferring knowledge across different tasks and domains, generating novel designs, and learning compositional concepts (Bengio et al., 2013; Higgins et al., 2017b; Lake et al., 2017; Peters et al., 2017; Schmidhuber, 1992). Assuming the observation x is generated from latent factors z via p(x|z), the goal of disentanglement learning is to correctly uncover a set of independent factors {zi} that give rise to the observation. While there has been a considerable progress in recent years, common assumptions about disentangled representations appear to be inadequate (Locatello et al., 2019).
Unsupervised disentangling methods are highly desirable as they assume no prior knowledge about the ground truth factors. These methods typically impose constraints to encourage independence among latent variables. Examples of constraints include forcing the variational posterior q(z|x) to be similar to a factorial p(z) (Burgess et al., 2018; Higgins et al., 2017a), forcing the variational aggregated prior q(z) to be similar to the prior p(z) (Makhzani et al., 2015), adding total correlation loss (Kim & Mnih, 2018), forcing the covariance matrix of q(z) to be close to the identity matrix (Kumar et al., 2017), and using a kernel-based measure of independence (Lopez et al., 2018). However, it remains unclear how the independence constraint affects other properties of representation. Indeed, more independence may lead to higher reconstruction error in some models (Higgins et al., 2017a; Kim & Mnih, 2018). Worse still, the independent representations may mismatch human’s predefined concepts (Locatello et al., 2019). This suggests that supervised methods – which associate a representation (or a group of representations) zi with a particular ground truth factor yk – may be more adequate. However, most supervised methods have only been shown to perform well on toy datasets (Harsh Jha et al., 2018; Kulkarni et al., 2015; Mathieu et al., 2016) in which data are generated from multiplicative combination of the ground truth factors. It is still unclear about their performance on real datasets.
We believe that there are at least two major reasons for the current unsatisfying state of disentanglement learning: i) the lack of a formal notion of disentangled representations to support the design of proper objective functions (Tschannen et al., 2018; Locatello et al., 2019), and ii) the lack of robust evaluation metrics to enable a fair comparison between models, regardless of their architectures or
design purposes. To that end, we contribute by formally characterizing disentangled representations along three dimensions, namely informativeness, separability and interpretability, drawing from concepts in information theory (Section 2). We then design robust quantitative metrics for these properties and argue that an ideal method for disentanglement learning should achieve high performance on these metrics (Section 3).
We run a series of experiments to demonstrate how to compare different models using our proposed metrics, showing that the quantitative results provided by these metrics are consistent with visual results (Section 4). In the process, we gain important insights about some well-known disentanglement learning methods namely FactorVAE (Kim & Mnih, 2018), β-VAE (Higgins et al., 2017a), and AAE (Makhzani et al., 2015).

heading: 2 RETHINKING DISENTANGLEMENT
text: Inspired by (Bengio et al., 2013; Ridgeway, 2016), we adopt the notion of disentangled representation learning as “a process of decorrelating information in the data into separate informative representations, each of which corresponds to a concept defined by humans”. This suggests three important properties of a disentangled representation: informativeness, separability and interpretability, which we quantify as follows:
Informativeness We formulate the informativeness of a particular representation (or a group of representations) zi w.r.t. the data x as the mutual information between zi and x:
I(x, zi) = ∫ x ∫ z pD(x)q(zi|x) log q(zi|x) q(zi) dz dx (1)
where q(zi) = ∫ x pD(x)q(zi|x) dx. In order to represent the data faithfully, a representation zi should be informative of x, meaning I(x, zi) should be large. Because I(x, zi) = H(zi)−H(zi|x), a large value of I(x, zi) means that H(zi|x) ≈ 0 given that H(zi) can be chosen to be relatively fixed. In other words, if zi is informative w.r.t. x, q(zi|x) usually has small variance. It is important to note that I(x, zi) in Eq. 1 is defined on the variational encoder q(zi|x), and does not require a decoder. It implies that we do not need to minimize the reconstruction error over x (e.g., in VAEs) to increase the informativeness of a particular zi.
Separability and Independence Two representations zi, zj are separable w.r.t. the data x if they do not share common information about x, which can be formulated as follows:
I(x, zi, zj) = 0 (2)
where I(x, zi, zj) denotes the multivariate mutual information (McGill, 1954) between x, zi and zj . I(x, zi, zj) can be decomposed into standard bivariate mutual information terms as follows:
I(x, zi, zj) = I(x, zi) + I(x, zj)− I(x, (zi, zj)) = I(zi, zj)− I(zi, zj |x)
I(x, zi, zj) can be either positive or negative. It is positive if zi and zj contain redundant information about x. The meaning of a negative I(x, zi, zj) remains elusive (Bell, 2003).
Achieving separability with respect to x does not guarantee that zi and zj are separable in general. zi and zj are fully separable or statistically independent if and only if:
I(zi, zj) = 0 (3)
If we have access to all representations z, we can generally say that a representation zi is fully separable (from other representations z6=i) if and only if I(zi, z6=i) = 0.
Note that there is a trade-off between informativeness, independence and the number of latent variables which we discuss in Appdx. A.7.
Interpretability Obtaining informative and independent representations does not guarantee interpretability by human (Locatello et al., 2019). We argue that in order to achieve interpretability, we should provide models with a set of predefined concepts y. In this case, a representation zi is
interpretable with respect to yk if it only contains information about yk (given that zi is separable from all other z6=i and all yk are distinct). Full interpretability can be formulated as follows:
I(zi, yk) = H(zi) = H(yk) (4)
Eq. 4 is equivalent to the condition that zi is an invertible function of yk. If we want zi to generalize beyond the observed yk (i.e., H(zi) > H(yk)), we can change the condition in Eq. 4 into:
I(zi, yk) = H(yk) or H(yk|zi) = 0 (5)
which suggests that the model should accurately predict yk given zi. If zi satisfies the condition in Eq. 5, it is said to be partially interpretable w.r.t yk.
In real data, underlying factors of variation are usually correlated. For example, men usually have beard and short hair. Therefore, it is very difficult to match independent latent variables to different ground truth factors at the same time. We believe that in order to achieve good interpretability, we should isolate the factors and learn one at a time.

heading: 2.1 AN INFORMATION-THEORETIC DEFINITION OF DISENTANGLED REPRESENTATIONS
text: Given a dataset D = {xi}Ni=1, where each data point x is associated with a set of K labeled factors of variation y = {y1, ..., yK}. Assume that there exists a mapping from x to m groups of latent representations z = {z1, z2, ..., zm} which follows the distribution q(z|x). Denoting q(zi|x) = ∑ z6=i
q(z|x) and q(zi) = EpD(x) [q(zi|x)]. We define disentangled representations for unsupervised cases as follows: Definition 1 (Unsupervised). A representation or a group of representations zi is said to be “fully disentangled” w.r.t a ground truth factor yk if zi is fully separable (from z6=i) and zi is fully interpretable w.r.t yk. Mathematically, this can be written as:
I(zi, z6=i) = 0 and I(zi, yk) = H(zi, yk) (6)
The definition of disentangled representations for supervised cases is similar as above except that now we model q(z|x, y) instead of q(z|x) and q(z) = ∑ x,y pD(x, y)q(z|x, y).
Recently, there have been several works (Eastwood & Williams, 2018; Higgins et al., 2018; Ridgeway & Mozer, 2018) that attempted to define disentangled representations. Higgin et. al. (Higgins et al., 2018) proposed a definition based on group theory (Cohen & Welling, 2014) which is (informally) stated as follows: “A representation z is disentangled w.r.t a particular subgroup yk (from a symmetry group y = {yk}Kk=1) if z can be decomposed into different subspaces {zi} H i=1 in which the subspace zi should be independent of all other representation subspaces z6=i, and zi should only be affected by the action of a single subgroup yk and not by other subgroups y 6=k.”. Their definition shares similar observation as ours. However, it is less convenient for designing models and metrics than our information-theoretic definition.
Eastwood et. al. (Eastwood & Williams, 2018) did not provide any explicit definition of disentangled representations but characterizing them along three dimensions namely “disentanglement”, “compactness”, and “informativeness” (between z any yk). A high “disentanglement” score (≈ 1) for zi indicates that it captures at most one factor, let’s say yk. A high “completeness” score (≈ 1) for yk indicates that it is captured by at most one latent zj and j is likely to be i. A high “informativeness” score1 for yk indicates that all information of yk is captured by the representations z. Intuitively, when all the three notions achieve optimal values, there should be only a single representation zi that captures all information of the factor yk but no information from other factors y6=k. However, even in that case, zi is still not fully interpretable w.r.t yk since zi may contain some information in x that does not appear in yk. This makes their notions only applicable to toy datasets on which we know that the data x are only generated from predefined ground truth factors y = {yk}Kk=1. Our definition can handle the situation where we only know some but not all factors of variation in the data. The notions in (Ridgeway & Mozer, 2018) follow those in (Eastwood & Williams, 2018), hence, suffer from the same disadvantage.
1In (Eastwood & Williams, 2018), the authors consider the prediction error of yk given z instead. High “informativeness” score means this error should be close to 0.

heading: 3 ROBUST EVALUATION METRICS
text: We argue that a robust metric for disentanglement should meet the following criteria: i) it supports both supervised/unsupervised models; ii) it can be applied for real datasets; iii) it is computationally straightforward, i.e. not requiring any training procedure; iv) it provides consistent results across different methods and different latent representations; and v) it agrees with qualitative (visual) results. Here we propose information-theoretic metrics to measure informativeness, independence and interpretability which meet all of these robustness criteria.

heading: 3.1 METRICS FOR INFORMATIVENESS
text: We measure the informativeness of a particular representation zi w.r.t. x by computing I(x, zi). If zi is discrete, we can compute I(x, zi) exactly by using Eq. 1 but with the integral replaced by the sum. If zi is continuous, we estimate I(x, zi) via sampling or quantization. Details about these estimations are provided in Appdx. A.10.
If H(zi) is estimated via quantization, we will have 0 ≤ I(x, zi) ≤ H(zi). In this case, we can divide I(x, zi) by H(zi) to normalize it to the range [0, 1]. However, this normalization may change the interpretation of the metric and lead to a situation where a representation zi is less informative than zj (i.e., I(x, zi) < I(x, zj)) but still has a higher rank than zj because H(zi) < H(zj). A better way is to divide I(x, zi) by log(#bins).

heading: 3.2 METRICS FOR SEPARABILITY AND INDEPENDENCE
text: MISJED We can characterize the independence between two latent variables zi, zj based on I(zi, zj). However, a serious problem of I(zi, zj) is that it generates the following order among pairs of representations: I(zf,i, zf,j) > I(zf,i, zn,j) > I(zn,i, zn,j) ≥ 0 where zf,i, zf,j are informative representations and zn,i, zn,j are uninformative (or noisy) representations. This means if we simply want zi, zj to be independent, the best scenario is that both are noisy and independent (e.g. q(zi|x) ≈ q(zj |x) ≈ N (0, I)). Therefore, we propose a new metric for independence named MISJED (which stands for Mutual Information Sums Joint Entropy Difference), defined as follows:
MISJED(zi, zj) = Ĩ(zi, zj) = H(zi) +H(zj)−H(z̄i, z̄j) = H(zi) +H(zj)−H(zi, zj) +H(zi, zj)−H(z̄i, z̄j) = I(zi, zj) +H(zi, zj)−H(z̄i, z̄j) (7)
where z̄i = Eq(zi|x)[zi] and q(z̄i) = EpD(x) [q(z̄i|x)]. Since q(z̄i) and q(z̄j) have less variance than q(zi) and q(zj), respectively, H(zi, zj)−H(z̄i, z̄j) ≥ 0, making Ĩ(zi, zj) ≥ 0.
To achieve a small value of Ĩ(zi, zj), two representations zi, zj should be both independent and informative (or, in an extreme case, are deterministic given x). Using the MISJED metric, we can ensure the following order: 0 ≤ Ĩ(zf,i, zf,j) < Ĩ(zf,i, zn,j) < Ĩ(zn,i, zn,j). If H(zi), H(zj), and H(z̄i, z̄j) in Eq. 7 are estimated via quantization, we will have Ĩ(zi, zj) ≤ H(zi) + H(zj) ≤ 2 log(#bins). In this case, we can divide Ĩ(zi, zj) by 2 log(#bins) to normalize it to [0, 1].
WSEPIN and WINDIN A theoretically correct way to verify that a particular representation zi is both separable from other z 6=i and informative w.r.t x is considering the amount of information in x but not in z6=i that zi contains. This quantity is the conditional mutual information between x and zi given z6=i, which can be decomposed as follows:
I(x, zi|z 6=i) = I(x, zi)− I(x, zi, z6=i) = I(x, zi)− (I(zi, z6=i)− I(zi, z6=i|x)) = I(x, zi)− I(zi, z6=i) + I(zi, z6=i|x) (8)
I(x, zi|z6=i) is useful for measuring how disentangled a representation zi is in the absence of ground truth factors. I(x, zi|z6=i) is close to 0 if zi is completely noisy and is high if zi is disentangled2.
2Note that only informativeness and separability are considered in this case.
For models that use factorized encoders, zi and z6=i are usually assumed to be independent given x, hence, I(zi, z6=i|x) ≈ 0 and I(x, zi|z6=i) ≈ I(x, zi)− I(zi, z6=i) which is merely the difference between the informativeness and full separability of zi. For models that use auto-regressive encoders, I(zi, z6=i|x) > 0 which means zi and z6=i can share information not in x. We can also compute I(x, zi|z6=i) in a different way as follows:
I(x, zi|z 6=i) = I(x, (zi, z6=i))− I(x, z6=i) = I(x, z)− I(x, z6=i)
If we want zi to be both independence of z6=i and informative w.r.t x, we can only use the first two terms in Eq. 8 to derive another quantitive measure:
Î(x, zi|z6=i) = I(x, zi)− I(zi, z6=i) = I(x, zi|z6=i)− I(zi, z6=i|x) (9)
However, unlike I(x, zi|z6=i), Î(x, zi|z6=i) can be negative. To normalize I(x, zi|z6=i), we divide it by H(zi) (H(zi) must be estimated via quantization). Note that taking the average of I(zi, x|z6=i) over all representations to derive a single metric for the whole model is not appropriate because models with more noisy latent variables will be less favored. For example, if model A has 10 latent variables (5 of them are disentangled and 5 of them are noisy), and model B has 20 latent variables (5 of them are disentangled and 15 of them are noisy), B will always be considered worse than A despite the fact that both are equivalent in term of disentanglement (since 5 disentangled representations are enough to capture all information in x so additional latent variables should be noisy). We propose two solutions for this issue. In the first approach, we sort I(x, zi|z6=i) over all representations in descending order and only take the average over the top k latents (or groups of latents). This leads to a metric called SEPIN@k3 which is similar to Precision@k:
SEPIN@k = 1
k k−1∑ i=0 I(x, zri |z6=ri)
where r1, ..., rL is the rank indices of L latent variables by sorting I(x, zi|z6=i) (i = 1, ..., L). In the second approach, we compute the average over all L representations z0, ..., zL−1 but weighted by their informativeness to derive a metric called WSEPIN:
WSEPIN = L−1∑ i=0 ρiI(x, zi|z6=i)
where ρi = I(x,zi)∑L−1 j=0 I(x,zj)
. If zi is a noisy representation, I(x, zi) ≈ 0, thus, zi contributes almost nothing to the final WSEPIN.
Similarly, using the measure Î(x, zi|z6=i) in Eq. 9, we can derive other two metrics INDIN@k4 and WINDIN as follows:
INDIN@k = 1
k k−1∑ i=0 Î(x, zri |z6=ri) and WINDIN = L−1∑ i=0 ρiÎ(x, zi|z6=i)

heading: 3.3 METRICS FOR INTERPRETABILITY
text: Recently, several metrics have been proposed to quantitatively evaluate the interpretability of representations by examining the relationship between the representations and manually labeled factors of variation. The most popular ones are Z-diff score (Higgins et al., 2017a; Kim & Mnih, 2018), SAP (Kumar et al., 2017), MIG (Chen et al., 2018). Among them, only MIG is theoretically sound and provides correct computation of I(x, zi). MIG also matches with our formulation of “interpretability” in Section 2 to some extent. However, MIG has only been used for toy datasets
3SEPIN stands for SEParability and INformativeness 4INDIN stands for INDependence and INformativeness
GT factor latent code
MIG RMIG
like dSprites (Matthey et al., 2017). The main drawback comes from its probabilistic assumption p(zi, yk, x
(n)) = q(zi|x(n))p(x(n)|yk)p(yk) (see Fig. 1). Note that p(x(n)|yk) is a distribution over the high dimensional data space, and is very hard to robustly estimate but the authors simplified it to be p(n|yk) if x(n) ∈ Dyk (Dyk is the support set for a particular value yk) and 0 otherwise. This equation only holds for toy datasets where we know exactly how x is generated from y. In addition, since p(n|yk) depends on the value of yk, it will be problematic if yk is continuous.
RMIG Addressing the drawbacks of MIG, we propose RMIG (which stands for Robust MIG), formulated as follows:
RMIG(yk) = I(zi∗ , yk)− I(zj◦ , yk) (10)
where I(zi∗ , yk) and I(zj◦ , yk) are the highest and the second highest mutual information values computed between every zi and yk; zi∗ and zj◦ are the corresponding latent variables. Like MIG, we can normalize RMIG(yk) to [0, 1] by dividing it by H(yk) but it will favor imbalanced factors (small H(yk)).
RMIG inherits the idea of MIG but differs in the probabilistic assumption (and other technicalities). RMIG assumes that p(zi, yk, x(n)) = q(zi|x(n))p(yk|x(n))p(x(n)) for unsupervised learning and p(zi, yk, x (n)) = q(zi|y(n)k , x(n))p(y (n) k , x
(n)) for supervised learning (see Fig. 1). Not only this eliminates all the problems of MIG but also provides additional advantages. First, we can estimate q(zi, yk) using Monte Carlo sampling on p(x(n)). Second, p(yk|x(n)) is well defined for both discrete/continuous yk and deterministic/stochastic p(yk|x(n)). If yk is continuous, we can quantize p(yk|x(n)). If p(yk|x(n)) is deterministic (i.e., a Dirac delta function), we simply set it to 1 for the value of yk corresponding to x(n) and 0 for other values of yk. Our metric can also use p(yk|x(n)) from an external expert model. Third, for any particular value yk, we compute q(zi|x(n)) for all x(n) ∈ D rather than just for x(n) ∈ Dyk , which gives more accurate results.
JEMMIG A high RMIG value of yk means that there is a representation zi∗ that captures the factor yk. However, zi∗ may also capture other factors y6=k of the data. To make sure that zi∗ fits exactly to yk, we provide another metric for interpretability named JEMMIG (standing for Joint Entropy Minuses Mutual Information Gap), computed as follows:
JEMMIG(yk) = H(zi∗ , yk)− I(zi∗ , yk) + I(zj◦ , yk)
where I(zi∗ , yk) and I(zj◦ , yk) are defined in Eq. 10.
If we estimate H(zi∗ , yk) via quantization, we can bound JEMMIG(yk) between 0 and H(yk) + log(#bins) (please check Appdx. A.12 for details). A small JEMMIG(yk) score means that zi∗ should match exactly to yk and zj◦ should not be related to yk. Thus, we can use JEMMIG(yk) to validate whether a model can learn disentangled representations w.r.t a ground truth factor yk or not which satisfies the definition in Section 2.1. Note that if we replace H(zi∗ , yk) by H(yk) to account for the generalization of zi∗ over yk, we obtain a metric equivalent to RMIG (but in reverse order).
To compute RMIG and JEMMIG for the whole model, we simply take the average of RMIG(yk) and JEMMIG(yk) over all yk (k = 1, ...,K) as follows:
RMIG = 1
K K−1∑ k=0 RMIG(yk) and JEMMIG = 1 K K−1∑ k=0 JEMMIG(yk)

heading: 3.4 COMPARISON WITH EXISTING METRICS
text: In Table 1, we compare our proposed metrics with existing metrics for learning disentangled representations. For deeper analysis of these metrics, we refer readers to Appdx. A.8. One can easily see that only our metrics satisfy the aforementioned robustness criteria. Most other metrics (except for MIG and Modularity) use classifiers, which can cause inconsistent results once the settings of the classifiers change. Moreover, most other metrics (except for MIG) use Eq(zi|x)[zi] instead of q(zi|x) for computing mutual information. This can lead to inaccurate evaluation results since Eq(zi|x)[zi] is theoretically different from zi ∼ q(zi|x). Among all metrics, JEMMIG is the only one that can quantify “disentangled representations” defined in Section. 2.1 on its own.

heading: 4 EXPERIMENTS
text: We use our proposed metrics to evaluate three representation learning methods namely FactorVAE (Kim & Mnih, 2018), β-VAE (Higgins et al., 2017a) and AAE (Makhzani et al., 2015) on both real and toy datasets which are CelebA (Liu et al., 2015) and dSprites (Matthey et al., 2017), respectively. A brief discussion of these models are given in Appdx. A.1. We would like to show the following points: i) how to compare models based on our metrics; ii) the advantages of our metrics compared to other metrics; iii) the consistence between qualitative results produced by our metrics and visual results; and iv) the ablation study of our metrics.
Due to space limit, we only present experiments for the first two points. The experiments for points (iii) and (iv) are put in Appdx. A.4 and Appdx. A.5, respectively. Details about the datasets and model settings are provided in Appdx. A.2 and Appdx. A.3, respectively. In all figures below, “TC” refers to the γ coefficient of the TC loss in FactorVAEs (Kim & Mnih, 2018), “Beta” refers to the β coefficient in β-VAEs (Higgins et al., 2017a).
Informativeness In Figs. 2a and 2b, we show the average amount of information (of x) that a representation zi contains (the mean of I(zi, x)) and the total amount of information that all representations z contain (I(z, x)). It is clear that adding the TC term to the standard VAE loss does not affect I(z, x) much (Fig. 2b). However, because zi and zj in FactorVAEs are more separable than those in standard VAEs, FactorVAEs should produce smaller I(zi, x) than standard VAEs on average (Fig. 2a). We also see that the mean of I(zi, x) and I(z, x) consistently decrease for β-VAEs with higher β.
Separability and Independence If we only evaluate models based on the separability of representations, β-VAE models with large β are among the best. These models force latent representations
to be highly separable (as in Fig. 3a, we can see that the max/mean/min values of I(zi, z6=i) are equally small for β-VAEs with large β). In FactorVAEs, informative representations usually have poor separability (large value) and noisy representations usually have perfect separability (≈ 0) (Fig. 4a). Increasing the weight of the TC loss improves the max and mean of I(zi, z6=i) but not significance (Fig. 3a).
Using WSEPIN and SEPIN@3 gives us a more reasonable evaluation of the disentanglement capability of these models. In Fig. 3b, we see that β-VAE models with β = 10 achieve the highest WSEPIN and SEPIN@3 scores, which suggests that their informative representations usually contain large amount of information of x that are not shared by other representations. However, this type of information may not associate well with the ground truth factors of variation (e.g., z3, z6 in Fig. 4c). The representations of FactorVAEs, despite containing less information of x on their own, usually reflect the ground truth factors more accurately (e.g., z5, z8, z7 in Fig. 4a) than those of β-VAEs. These results suggest that ground truth factors should be used for proper evaluations of disentanglement.
Interpretability Using JEMMIG and RMIG, we see that FactorVAE models can learn representations that are more interpretable than those learned by β-VAE models. Surprisingly, the worst FactorVAE models (with TC=10) clearly outperform the best β-VAE models (with β = 10). This result is sensible because it is accordant with the visualization in Figs. 4a and 4c.
Comparison with Z-diff In (Chen et al., 2018), the authors have already shown that MIG is more robust than Z-diff (Higgins et al., 2017a) so we compare our metrics with MIG directly.
Comparison with MIG On toy datasets like dSprites, RMIG produces similar results as MIG (Chen et al., 2018). Please check Appdx. A.13 for more details.
Comparison with “disentanglement”, “completeness” and “informativeness” In Fig. 6, we show the differences in evaluation results between JEMMIG/RMIG and the metrics in (Eastwood & Williams, 2018). We can easily see that JEMMIG and RMIG are much better than “disentanglement”, “completeness” and “informativeness” (or reversed classification error) in separating FactorVAE and β-VAE models. Among the three competing metrics, only “informativeness” (or I(z, yk)) seems to be correlated with JEMMIG and RMIG. This is understandable because when most representations are independent in case of FactorVAEs and β-VAEs, we have I(z, yk) ≈ I(zi∗ , yk) ≈ I(zi∗ , yk)− I(zj◦ , yk). “Disentanglement” and “completeness”, by contrast, are strongly uncorrelated with JEMMIG and RMIG. While JEMMIG consistently grades standard VAEs (β = 1) worse than other models (Fig. 5a), “disentanglement” and “completeness” usually grade standard VAEs better than some FactorVAE models, which seems inappropriate. Moreover, since “disentanglement” and “completeness” are not well aligned, using both of them at the same time may cause confusion. For example, the model “28_beta20” has lower “disentanglement” score yet higher “completeness” score
than the model “32_beta50” (Figs. 6a and 6b) so it is hard to know which model is better than the other at learning disentangled representations.
From Figs. 7a and 7b, we see that “disentanglement” and “completeness” blindly favor β-VAE models with high β without concerning about the fact that representations in these models are less informative than representations in FactorVAEs (Fig. 7c). Thus, they are not good for characterizing disentanglement in general.
Disentanglement and “completeness” are computed based on a weight matrix with an assumption that the weight magnitudes for noisy representations are close to 0. However, this assumption is often broken in practice, thus, may lead to inaccurate results (please check Appdx. A.9 for details).
Comparison with “modularity” “modularity” and “explicitness” (Ridgeway & Mozer, 2018) are similar to “disentanglement” and “informativeness” (Eastwood & Williams, 2018) in terms of concept, respectively. However, they are different in terms of formulation. We exclude “explicitness” in our experiment because computing it on dSprites is time consuming. In Fig. 8a, we show the correlation between JEMMIG and “modularity”. We consider two versions of “modularity”. In the first version (Fig. 8b), I(zi, yk) is computed from the mean of zi ∼ q(zi|x). This is the original implementation provided by (Ridgeway & Mozer, 2018). In the second version (Fig. 8c), I(zi, yk) is computed from q(zi|x). We can see that in either case, “modularity” often gives higher scores for standard VAEs than for FactorVAEs. It means that like “disentanglement”, “modularity” itself does not fully specify “disentangled representations” defined in Section 2.1.

heading: 5 CONCLUSION
text: We have proposed an information-theoretic definition of disentangled representations and designed robust metrics for evaluation, along three dimensions: informativeness, separability and interpretability. We carefully analyze the properties of our metrics using well known representation learning models namely FactorVAE, β-VAE and AAE on both real and toy datasets. Compared with existing metrics, our metrics are more robust and produce more sensible evaluations that are compatible with visual results. Based on our definition of disentangled representation in Section 2.1, WSEPIN/JEMMIG are the two key metrics in case ground truth labels are unavailable/available, respectively.

heading: A APPENDIX
text: A.1 REVIEW OF FACTORVAES, β-VAES AND AAES
Standard VAEs are trained by minimizing the variational upper bound LVAE of− log pθ(x) as follows: LVAE = EpD(x) [ Eqφ(z|x) [− log pθ(x|z)] +DKL (qφ(z|x)‖p(z)) ] (11)
where qφ(z|x) is an amortized variational posterior distribution. However, this objective does not lead to disentangled representations (Higgins et al., 2017a).
β-VAEs (Higgins et al., 2017a) penalize the KL term in the original VAE loss more heavily with a coefficient β 1:
Lβ-VAE = EpD(x) [ Eqφ(z|x) [− log pθ(x|z)] + βDKL (qφ(z|x)‖p(z)) ] Since EpD(x) [DKL (qφ(z|x)‖p(z))] = Iφ(x, z) +DKL (qφ(z)‖p(z)), more penalty on the KL term encourages qφ(z) to be factorized but also forces z to discard more information in x.
FactorVAEs (Kim & Mnih, 2018) add a constraint to the standard VAE loss to explicitly impose factorization of qφ(z):
LFactorVAE = LVAE + γDKL ( qφ(z)‖
∏ i qi(zi)
) (12)
where DKL (qφ(z)‖ ∏ i qi(zi)) ≥ 0 is known as the total correlation (TC) of z. Intuitively, γ can be large without affecting the mutual information between z and x, making FactorVAE more robust than β-VAE in learning disentangled representations. Other related models that share similar ideas with FactorVAEs are are β-TCVAEs (Chen et al., 2018) and DIP-VAEs (Kumar et al., 2017).
The loss of AAEs (Makhzani et al., 2015) is derived from the standard VAE loss by removing the term Iφ(x, z):
LAAE = EpD(x) [ Eqφ(z|x) [− log pθ(x|z)] ] +DKL (qφ(z)‖p(z))
Different from the losses of β-VAEs and FactorVAEs, AAE loss is not a valid upper bound on − log pθ(x).

heading: A.2 DATASETS
text: The CelebA dataset (Liu et al., 2015) consists of more than 200 thousands face images with 40 binary attributes. We resize these images to 64 × 64. The dSprites dataset (Matthey et al., 2017) is a toy dataset generated from 5 different factors of variation which are “shape” (3 values), “scale” (6 values), “rotation” (40 values), “x-position” (32 values), “y-position” (32 values). Statistics of these datasets are provided in Table 2.

heading: A.3 MODEL SETTINGS
text: For FactorVAE, β-VAE and AAE, we used the same architectures for the encoder and decoder (see Table 3 and Table 45), following (Kim & Mnih, 2018). We trained the models for 300 epochs with mini-batches of size 64. The learning rate is 10−3 for the encoder/decoder and is 10−4 for the discriminator over z. We used Adam (Kingma & Ba, 2014) optimizer with β1 = 0.5 and β2 = 0.99. Unless explicitly mentioned, we use the following default settings: i) for CelebA: the number of
5Only FactorVAE and AAE use a discriminator over z
latent variables is 65, the TC coefficient in FactorVAE is 50, the value for β in β-VAE is 50, and the coefficient for the generator loss over z in AAE is 50; ii) for dSprites: the number of latent variables is 10.

heading: A.4 CONSISTENCE BETWEEN QUANTITATIVE AND QUALITATIVE RESULTS

heading: A.4.1 CELEBA
text: Informativeness We sorted the representations of different models according to their informativeness scores in the descending order and plot the results in Fig. 9. There are distinct patterns for different methods. AAE captures equally large amounts of information from the data while FactorVAE and β-VAE capture smaller and varying amounts. This is because FactorVAE and β-VAE penalize the informativeness of representations while AAE does not. Recall that I(zi, x) = H(zi)−H(zi|x). For AAE, H(zi|x) = 0 and H(zi) is equal to the entropy of N (0, I). For FactorVAE and β-VAE, H(zi|x) > 0 and H(zi) is usually smaller than the entropy of N (0, I) due to a narrow q(zi)6.
6Note that H(zi) does not depend on whether q(zi) is zero-centered or not
In Fig. 9, we see a sudden drop of the scores to 0 for some FactorVAE’s and β-VAE’s representations. These representations zi are totally random and contain no information about the data (i.e., q(zi|x) ≈ N (0, I)). We call them “noisy” representations and provide discussions in Appdx. A.7. We visualize the top 10 most informative representations for these models in Fig. 10. AAE’s representations are more detailed than FactorVAE’s and β-VAE’s, suggesting the effect of high informativeness. However, AAE’s representations mainly capture information within the support of pD(x). This explains why we still see a face when interpolating AAE’s representations. By contrast, FactorVAE’s and β-VAE’s representations usually contain information outside the support of pD(x). Thus, when we interpolate these representations, we may see something not resembling a face.
Separability and Independence Table 5 reports MISJED scores (Section 3.2) for the top most informative representations. FactorVAE achieves the lowest MISJED scores, AAE comes next and βVAE is the worst. We argue that this is because FactorVAE learns independent and nearly deterministic representations, β-VAE learns strongly independent yet highly stochastic representations, and AAE, on the other extreme side, learns strongly deterministic yet not very independent representations. From Table 5 and Fig. 11, it is clear that MISJED produces correct orders among pairs of representations according to their informativeness.
Interpretability We report the RMIG scores and JEMMIG scores for several ground truth factors in the CelebA dataset in Tables 6 and 7, respectively. In general, FactorVAE learns representations that agree better with the ground truth factors than β-VAE and AAE do. This is consistent with the qualitative results in Fig. 12. However, all models still perform poorly for interpretability since their RMIG and JEMMIG scores are very far from 1 and 0, respectively. We provide the normalized JEMMIG and RMIG scores for all attributes in Fig. 13.

heading: A.4.2 DSPRITES
text: Informativeness From Fig. 14, we see that 5 representations of AAE have equally high informativeness scores while the remaining 5 representations have nearly zeros informativeness scores. This is because AAE needs only 5 representations to capture all information in the data. FactorVAE also
needs only 5 representations but some are less informative than those of AAE. Note that the number of ground truth factors of variation in dSprites dataset is also 5.
Separability and Independence Fig. 15 shows heat maps of MISJED scores for the three models.
Interpretability From Tables. 8 and 9, we see that FactorVAE is very good at disentangling “scale”, “x-position” and “y-position” but fails to disentangling “shape” and “rotation”. However, FactorVAE still performs much better than β-VAE and AAE. These results are consistent with the visual results in Fig. 16.
Also note that in FactorVAE, the RMIG scores for “scale” and “x-position” are quite similar but the JEMMIG score for “scale” is higher than that for “x-position”. This is because the quantized distribution (with 100 bins) of a particular representation zi fits better to the distribution of “x-position” (having 32 possible values) than to the distribution of “scale” (having only 6 possible values).

heading: A.5 ABLATION STUDY OF OUR METRICS
text: Sensitivity of the number of bins When estimating entropy and mutual information terms using quantization, we need to specify the value range and the number of bins (#bins) in advance. In this paper, we fix the value range to be [-4, 4] since most latent values fall within this range. We only investigate the effect of #bins on the RMIG and JEMMIG scores for different models and show the results in Fig. 17 (left, middle).
We can see that when #bins is small, RMIG scores are low. This is because the quantized distributions Q(zi∗) and Q(zj◦) look similar, causing I∗(zi∗ , yk) and I◦(zj◦ , yk) to be similar as well. When #bins is large, the quantized distribution Q(zi∗) and Q(zj◦) look more different, leading to higher RMIG scores. RMIG scores are stable when #bins > 200, which suggests that finer quantizations do not affect the estimation of I(zi, yk) much.
Unlike RMIG scores, JEMMIG scores keep increasing when we increase #bins. Note that JEMMIG only differs from RMIG in the appearance of H(zi∗ , yk). Finer quantizations of zi∗ introduce more information about zi∗ , hence, always lead to higher H(zi∗ , yk) (see Fig. 17 (right)). Larger JEMMIG scores also reflect the fact that finer quantizations of zi∗ make zi∗ look more continuous, thus, less interpretable w.r.t the discrete factor yk.
We provide a detailed explanation about the behaviors of RMIG and JEMMIG w.r.t #bins in Appdx. A.11. Despite the fact that #bins affects the RMIG and JEMMIG scores of a single model, the relative order among different models remains the same. It suggests that once we fixed the #bins, we can use RMIG and JEMMIG scores to rank different models.
Sensitivity of the number of samples From Fig. 18 (left, right), it is clear that the sampling estimation is unbiased and is not affected much by the number of samples.
Sensitivity of sampling in high dimensional space One thing that we should concern about is the performance of our metrics when the number of latent representations (#latents) is large (or z is high-dimensional). In Fig. 19a, we see that the informativeness of an individual representations zi is not affected by #latents. When we increase #latents, additional representations are usually noisy (I(zi, x) ≈ 0). The total amount of information captured by the model (I(x, z)), by contrast, highly depends on #latents (Fig. 19b). Unusually, increasing #latents reduces I(x, z) instead of increasing it. We have not found the final answer for this phenomenon but possible hypotheses are: i) On a high dimensional space where most latent representations are noisy (e.g. #latents=20), q(z|x) may look more similar to q(z), causing the wrong calculation of log q(z|x)q(z) , or ii) when #latents is large,
q(z|x) = ∏L−1 i=0 q(zi|x) is very tiny, thus, may lead to floating point imprecision7. In Fig. 19c, we
7We tried q(z|x) = exp (∑L−1 i=0 log q(zi|x) ) and it gives similar results as q(z|x) = ∏L−1 i=0 q(zi|x).
see that increasing #latents increases I(zi, z6=i). This makes sense because larger #latents means that z 6=i will contain more information. However, the change of I(zi, z6=i) is sudden when #latents change from 10 to 15, which is different from the change of #latents from 5 to 10 or 15 to 20. Recall that I(zi, z6=i) = H(zi) + H(z 6=i) − H(z). Since H(zi) can be computed stably, we only plot H(z6=i)−H(z) and show it in Fig. 19d. We can see that when #latents = 20, H(z6=i) ≈ H(z) which means we cannot differentiate between q(z6=i) and q(z). The instability of computation for high dimensional latents becomes clearer in Fig. 19e as I(x, zi|z6=i) = I(x, z) − I(x, z6=i) can be < 0 when #latents = 15 or 20. This causes the instability of WSEPIN in Fig. 19f despite the results look reasonable. JEMMIG and RMIG are calculated on individual latents so they are not affected by #latents and can provide consistent evaluations for models with different #latents.

heading: A.6 EVALUATING INDEPENDENCE WITH CORRELATION MATRIX
text: For every x(n) sampled from the training data, we generated m = 1 latent samples z(n,m)i ∼ q(zi|x(n)) and built a correlation matrix from these samples for each of the models FactorVAE, β-VAE and AAE. We also built another version of the correlation matrix which is based on the Eq(zi|x(n))[zi] (called the conditional means) instead of samples from q(zi|x
(n)). Both are shown in Fig. 20. We can see that the correlation matrices computed based on the conditional means incorrectly describe the independence between representations of FactorVAE and β-VAE. AAE is not affected because it learns deterministic zi given x. Using the correlation matrix is not a principled way to evaluate independence in disentanglement learning.

heading: A.7 TRADE-OFF BETWEEN INFORMATIVENESS, INDEPENDENCE AND THE NUMBER OF
text: LATENT VARIABLES
Before starting our discussion, we provide the following fact:
Fact 2. Assume we try to fill a fixed-size pool with fixed-size balls given that all the balls must be inside the pool. The only way to increase the number of the balls without making them overlapped is reducing their size.
In the context of representation learning, a pool is x with size H(x) which depends on the training data. Balls are zi with size H(zi). Fact. 2 reflects the situation of AAE (see Fig. 21 left). In AAE, all zi are deterministic given x so the condition “all balls are inside the pool” is met. H(zi) ≈ the entropy of N (0, I) which is fixed so the condition “fixed-size balls” is also met. Therefore, when the number of latent variables in AAE increases, all zi must be less informative (i.e., H(zi) must decrease) given that the independent constraint on the latent variables is still satisfied. This is empirically verified in Fig. 22 as we see the distribution of Eq(zi|x(n))[zi] over all x
(n) ∼ pD(x) becomes narrower when we increase the number of representations from 65 to 200. Also note that increasing the number of latent variable from 65 to 100 does not change the distribution. This suggests that 65 or 100 latent variables are still not enough to capture all information in the data.
FactorVAE, however, handles the increasing number of latent variables in a different way. Thanks to the KL term in the loss function that forces q(zi|x) to be stochastic, FactorVAE can break the
constraint in Fact 2 and allows the balls to stay outside the pool (see Fig. 21 right). If we increase the number of latent variables but still enforce the independence constraint on them, FactorVAE will keep a fixed number of informative representations and make all other representations “noisy” with zero informativeness scores. We refer to that capability of FactorVAE as code compression.

heading: A.8 ANALYSIS OF EXISTING METRICS FOR DISENTANGLEMENT
text: In this section, we analyze recent metrics, including Z-diff score (Higgins et al., 2017a; Kim & Mnih, 2018), Separated Attribute Predictability (SAP) (Kumar et al., 2017), Mutual Information Gap (MIG) (Chen et al., 2018), Disentanglement/Compactness/Informativeness (Eastwood & Williams, 2018), Modularity/Explicitness (Ridgeway & Mozer, 2018).
The main idea behind the Z-diff score (Higgins et al., 2017a; Kim & Mnih, 2018) is that if a ground truth generative factor yk (k ∈ {0, 1, ...,K}) is well aligned with a particular disentangled representation zi (although we do not know which i), we can use a simple classifier to predict k using information from z. Higgins et al. (Higgins et al., 2017a) use a linear classifier while Kim et. al. (Kim & Mnih, 2018) use a majority-vote classifier. The main drawback of this metric is that it assumes knowledge about all ground truth factors that generate the data. Hence, it is only applicable for a toy dataset like dSprites. Another drawback lies in the complex procedure to compute the metric, which requires training a classifier. Since the classifier is sensitive to the chosen optimizer, hyper-parameters and weight initialization, it is hard to ensure a fair comparison.
The SAP score (Kumar et al., 2017) is computed based on the correlation matrix C between the latent variables z and the ground truth factors y. If a latent zi and a factor yk are both continuous, the (square) correlation Ci,k between them is equal to
Cov2(zi,yk) Var(zk)Var(yk)
and is in [0, 1]. However, if the factor yk is discrete, computing the correlation between continuous and discrete variables is not straightforward. The authors handled this problem by learning a classifier that predicts yk given zi and used the balanced8 prediction accuracy as a replacement. Then, for each factor yk, they sorted C:,k in the descending order and computed the difference between the top two scores. The mean of the difference scores for all factors was used as the final SAP score. The intuition for this metric is that if a latent zi is the most representative for a factor yk (due to the highest correlation score), then other latent variables z6=i should not be related to yk, and thus, the difference score for yk should be high. We believe the SAP score is more sensible than Z-diff but it is only suitable when both the ground truth factors and the latent variables are continuous as no classifier is required. Moreover, if we have K discrete ground truth factors and L latent variables, the number of classifiers we need to learn is L×K, which is unmanageable when L is large. The MIG score (Chen et al., 2018) shares the same intuition as the SAP score but is computed based on the mutual information between every pair of zi and yk instead of the correlation coefficient. Thus, the MIG score is theoretically more appealing than the SAP score since it can capture nonlinear relationships between latent variables and factors while the SAP score cannot. The MIG score, to some extent, reflects the concept “interpretability” that we discussed in Section 2 in the main text.
8To achieve balance, the classifier uses the same number of samples for all categories of yk during training and testing
Eastwood et. al. (Eastwood & Williams, 2018) proposed three different metrics namely “disentanglement”, “completeness”, and “informativeness” to quantify disentangled representations. These metrics are computed based on a so-called “important matrix”R whose elementRik is the relative importance of zi (w.r.t other z6=i) in predicting yk. More concretely, for each factor yk (k = 0, ...,K−1), they train a regressor (LASSO or Random Forest) to predict yk from z and use the weight vector provided by this regressor to define R·k. The “disentanglement” score Di quantifies the degree to which a latent zi captures at most one generative factor yk. Di is computed as Di = (1−HK(Pi·)) where HK(Pi·) = ∑K−1 k=0 −Pik logPik and Pik =
Rik∑K−1 k′=0 Rik′ which can be seen as the “proba-
bility” of predicting yk instead of y6=k from zi. Similarly, the “completeness” score Ck quantifies the degree to which a ground truth factor yk is captured by a single latent zi (i = 0, ..., L − 1), computed as Ck = 1−HL(P̃·k) where HL(P̃·k) = ∑L−1 i=0 −P̃ik log P̃ik and P̃ik =
Rik∑L−1 i′=0 Ri′k . The
“informativeness” score describes the total amount of information of a particular factor yk captured by all representations z. However, the authors use the prediction error Ek of the k-th regressor to quantify “informativeness” instead of I(yk, z). Despite being well-motivated, these metrics still have several drawbacks. First, using three different metrics to quantify disentangled representations is not as convenient as using a single metric like MIG (Chen et al., 2018). For example, how can we compare two models A and B if A has a better “disentanglement” score but a worse “completeness” score than B? Second, these metrics do not apply for categorical factors with C classes since in this case the model weight is not a vector but an L× C matrix. Third, defining the pseudo-distribution Pik =
Rik∑K−1 k′=0 Rik′ seems ad hoc because i) the weight magnitudes Rik are unbounded and can vary
significantly (see Appdx. A.9), and ii) Pik strongly depends on the available ground truth factors (e.g. the value of Pik will change if we only consider 2 instead of 5 factors).
Ridgeway et. al. (Ridgeway & Mozer, 2018) proposed two metrics called “modularity” and “explicitness” that have similar interpretations as “disentanglement” and “informativeness” discussed above but differ in implementation. Specifically, they compute the “modularity” score
Mi for a representation zi as Mi = 1 − ∑K−1 k=0 (I(zi,yk)−Tik) 2
I2(zi,yk∗ )×(K−1) where k∗ = argmaxkI(zi, yk) and
Tik =
{ I(zi, yk∗) if k = k∗
0 otherwise . Like the “disentanglement” score Di, Mi is also ad hoc and is
undefined when the number of ground truth factors is 1. The “explicitness” score Ek for each ground truth factor yk is computed as the ROC curve of a logistic classifier that predicts yk from z. It turns out that Ek is just a way to bypass computing I(yk, z).
A.9 THE MUTUAL INFORMATION MATRIX I(zi, yk) AND THE IMPORTANCE MATRIX Rik
In Fig. 23, we compare our mutual information matrix I(zi, yk) with the counterpart in (Ridgeway & Mozer, 2018) and the importance matrix Rik in (Eastwood & Williams, 2018). It is clear that all matrices can capture disentangled representations (those highlighted in red) well since their corresponding values are high compared to other values in the same column. However, the matrix I(zi, yk) in (Ridgeway & Mozer, 2018) usually overestimates noisy representations since it uses Eq(zi|x)[zi] instead of q(zi|x). The matrix Rik in (Eastwood & Williams, 2018) sometimes assign very high absolute values for noisy representations since the regressor’s weights are unbounded. These flaws make the metrics in (Ridgeway & Mozer, 2018) and in (Eastwood & Williams, 2018) inaccurate and unstable, especially “modularity” and “disentanglement” since they require normalization over rows.

heading: A.10 COMPUTING METRICS FOR INFORMATIVENESS, SEPARABILITY AND INTERPRETABILITY
text: The metrics for informativeness, separability and interpretability in Section. 3 requires computing H(zi), H(zi|x), H(z6=i), H(z), and H(zi, yk). We can compute these entropies via quantization or sampling. Quantization is only applicable when zi is a scalar. If zi is a high-dimensional vector, we need to use sampling. Below, we describe how to compute H(z) via sampling and H(zi) via quantization. Other cases can be derived similarly.
Computing H(z) via sampling H(z) = −Eq(z) [log q(z)]
= −Eq(z,x) [ logEpD(x) [q(z|x)] ] = − 1
M M∑ m=1
[ log 1
N N∑ n=1 q ( z(m)|x(n) )] (13)
= − 1 M M∑ m=1
[ log 1
N N∑ n=1 ( L∏ i=1 q ( z (m) i |x (n) ))]
(14)
In Eq. 13, we use Monte Carlo sampling to estimate the expectations outside and inside the log function. The corresponding sample sizes are M and N . In Eq. 14, we use the assumption q ( z(m)|x(n) ) = ∏L i=1 q ( z (m) i |x(n) ) . Please note that the entropy H(z) computed via sampling can be negative if z is continuous since we use the density function q(z|x).
Computing H(zi) via quantization We can compute H(zi) via quantization as follows: H(zi) = − ∑ si∈S Q(si) logQ(s)
where S is a set of all quantized bins si corresponding to zi; Q(si) is the probability mass function of si. To ensure consistency among different zi as well as different models, we apply the same value range for all latent variables. In practice, we choose the range [−4, 4] since most of the latent values fall within this range. We divide this range into equal-size bins to form S. We can compute Q(si) as follows:
Q(si) = 1
N N∑ n=1 Q ( si|x(n) ) We compute Q ( si|x(n) ) based on its definition, which is:
Q(si|x(n)) = ∫ b a q(zi|x(n)) dzi (15)
where a, b are two ends of the bin si.
There are two ways to compute Q(si|x(n)). In the first way, we simply consider the unnormalized Q ′ (si|x(n)) as the area of a rectangle whose width is b − a and height is q(z̄i|x(n)) with z̄i at the center value of the bin si. Then, we normalize Q ′ (si|x(n)) over all bins to get Q(si|x(n)). In the second way, if q(zi|x(n)) is a Gaussian distribution, we can estimate the above integral with a closed-form function (see Appdx. A.14 for detail).

heading: A.11 RELATIONSHIP BETWEEN SAMPLING AND QUANTIZATION
text: DenoteHs(zi|x) andHq(zi|x) to be the sampling and quantization estimations of an entropyH(zi|x), respectively. Because Hs(zi|x) is the expectation of log q(zi|x), Hq(zi|x) is the expectation of logQ(zi|x), and Q(zi|x) ≈ q(zi|x)× bin width if the bin width is small enough, there exists a gap between Hs(zi|x) and Hq(zi|x), specified as follows:
Hq(zi|x) = Hs(zi|x)− log(bin width) = Hs(zi|x)− log (
value range #bins ) = Hs(zi|x)− log (value range) + log (#bins)
Since Q(zi) = EpD(x) [Q(zi|x)] and q(zi) = EpD(x) [q(zi|x)], we have Q(zi) ≈ q(zi)× bin width. Thus, Hq(zi) and Hs(zi) also exhibit a similar gap as Hs(zi|x) and Hq(zi|x):
Hq(zi) = Hs(zi)− log(bin width) However, this gap disappears when computing the mutual information I(zi, x) since:
Iq(zi, x) = Hq(zi)−Hq(zi|x) = (Hs(zi)− log(bin width))− (Hs(zi|x)− log(bin width)) = Hs(zi)−Hs(zi|x) = Is(zi, x)
In fact, one can easily prove that:
lim #bins→+∞ Iq(zi, x) = Is(zi, x)
.
Similar relationships between sampling and quantization also apply for H(zi, yk) and I(zi, yk). They are clearly shown in Fig. 24.
In summary,
• Sampling entropies such as Hs(zi|x) or Hs(zi) are usually fixed but can be negative since q(zi|x) or q(zi) can be > 1. However, these entropies can still be used for ranking though it is not easy to interpret them. • Quantized entropies such as Hq(zi|x) or Hq(zi) can be positive if the bin width is small
enough (or #bins is large enough). The growth rate is − log(bin width) (or log(#bin)). Because limx→+∞ log x = +∞, Hq(zi|x) and Hq(zi) cannot be upper-bounded.
• The mutual information I(zi, x) is consistent via either quantization or sampling. Unlike the entropies, I(zi, x) is well-bounded even when zi is continuous, thus, is suitable to be used in a metric. However, when #bins is small, the approximation Q(zi) ≈ q(zi)× bin width does not hold and quantization estimation can be inaccurate.

heading: A.12 NORMALIZING JEMMIG
text: Recall that the formula of the unnormalized JEMMIG(yk) is H(zi∗ , yk)− I(zi∗ , yk) + I(zj◦ , yk). If we estimate H(zi∗ , yk) via quantization, the value of the unnormalized JEMMIG(yk) will vary according to the bin width (or value range and #bins) (as shown in Fig. 17 (left)). However, we can still rank models by forcing them using the same bin width (or the same value range and #bins). To avoid setting these hyper-parameters, we can estimate H(zi∗ , y) via sampling. In this case, the value of the unnormalized JEMMIG(yk) only depends on q(zi|y) which is fixed after learning. Ranking disentanglement models using the unnormalized JEMMIG(yk) is somewhat similar to ranking generative models using the log-likelihood.
Using the unnormalized JEMMIG(yk) causes interpretation difficulty. We could normalize JEMMIG(yk) as follows:
Hq(zi) +H(yk)− 2I(zi∗ , yk) + I(zj◦ , yk) Hq(u) +H(yk)
(16)
where Hq(zi) is a quantization estimation of H(zi), hence, greater than 0; Hq(u) is an entropy that bounds Hq(zi) but does not depend on q(zi|x). Intuitively, u should be uniform. The main problem is how to find an effective value range [a, b] of zi that satisfies 2 conditions: i) most of the mass of zi falls within that range, and ii) H(u) is the bound of H(zi) if u ∈ [a, b]. However, before solving this question, we try to answer a similar yet easier question: “Given a Gaussian random variable z ∼ N (µ, σ), what is the value range of a uniform random variable u such that H(u) ≥ H(z)?”. Assume u ∈ [a, b], the entropy of u is H(u) = log(b− a) while the entropy of z is H(z) = 0.5 log(2πeσ2). We have:
H(z) ≤ H(u) ⇔0.5 log(2πeσ2) ≤ log(b− a) ⇔σ √ 2πe ≤ b− a
Thus, to ensure H(u) to be an upper bound of H(z), we should choose the value range of u to be at least σ √ 2πe. If σ = 1, this range is about 4.1327. If we also want [a, b] to capture most of the mass of z, a should be µ− σ2 √ 2πe and b should be µ+ σ2 √ 2πe.
Come back to the main problem, since q(zi) = EpD(x) [q(zi|x)] and q(zi|x) is usually a Gaussian distribution N (µi, σi), we can choose a, b as follows:
a = min ( µ (1) i − σ (1) i
2
√ 2πe, ..., µ
(N) i −
σ (N) i
2
√ 2πe ) , and
b = max ( µ (1) i + σ (1) i
2
√ 2πe, ..., µ
(N) i +
σ (N) i
2
√ 2πe
)
One may wonder that different methods can choose different value ranges [a, b] to normalize JEMMIG so how to ensure a fair comparison among them using the normalized JEMMIG. A simple solution is using the same value range [a, b] for different models. In this case, b− a should be large enough to cover various distributions. We can write Eq. 16 as follows:
Hq(zi) +H(yk)− 2I(zi∗ , yk) + I(zj◦ , yk) Hq(u) +H(yk)
= Hs(zi)− log(value range) + log(#bins) +H(yk)− 2I(zi∗ , yk) + I(zj◦ , yk)
Hs(u)− log(value range) + log(#bins) +H(yk)
= Hs(zi)− log(value range) + log(#bins) +H(yk)− 2I(zi∗ , yk) + I(zj◦ , yk)
log(#bins) +H(yk) (17)
Since the fraction in Eq. 17 is smaller than 1, increasing #bins will increase this fraction but still ensure that it is smaller than 1. This means the normalized JEMMIG is always in [0, 1] despite #bins.

heading: A.13 COMPARING RMIG WITH OTHER MIG IMPLEMENTATIONS
text: RMIG has several advantages compared to the original MIG (Chen et al., 2018) which we refer as MIG1: i) RMIG works on real datasets, MIG1 does not; ii) RMIG supports continuous factors, MIG1 does not. On toy datasets such as dSprites, RMIG produces almost the same results as MIG1 (Fig. 25 (left)). We argue that the small differences between RMIG and MIG1 scores in some models are caused by either the quantization error of RMIG (when #bins=100) or the sampling error of MIG1 (when #samples=10000).
Locatello et. al. (Locatello et al., 2019) provided an implementation9 of MIG which we refer as MIG2. MIG2 is theoretically incorrect in two points: i) it only uses the mean of the distribution q(zi|x(n)) instead of the whole distribution q(zi|x(n)), and ii) the bin range and width varies for different zi. The performance of MIG2 is, thus, unstable. We can easily see this problem by comparing the right plot with the left plot in Fig. 25. MIG2 usually overestimates the true MIG1 when evaluating β-VAE models with a large β (e.g. β ∈ {20, 30, 50}). We guess the reason is that in these models, q(zi|x(n)) usually has high variance, hence, using the mean of q(zi|x(n)) like MIG2 leads to the wrong estimation of I(zi, yk).

heading: A.14 DEFINITE INTEGRAL OF A GAUSSIAN DENSITY FUNCTION
text: Assume that we have a Gaussian distribution N (µ, σ). The definite integral of its density function within the range [a, b] denoted as G(a, b) can be computed as follows:
G(a, b) = ∫ b a 1 σ √ 2π exp ( −(x− µ)2 2σ2 ) dx
= 1
2 ( erf ( b− µ σ √ 2 ) − erf ( a− µ σ √ 2 )) 9https://github.com/google-research/disentanglement_lib
Although erf(·) does not have analytical form, we can compute its values with high precision using polynomial approximation. For example, the following approximation provides a maximum error of 5× 10−4 (Def, 2019):
erf(x) ≈ 1− 1 (1 + a1x+ a2x2 + a3x3 + a4x4) 4 , x > 0
where a1 = 0.278393, a2 = 0.230389, a3 = 0.000972, a4 = 0.078108.

heading: A.15 REPRESENTATIONS LEARNED BY FACTORVAE
text: We empirically observed that FactorVAE learns the same set of disentangled representations across different runs with varying numbers of latent variables (see Appdx. A.18). This behavior is akin to that of deterministic PCA which uncovers a fixed set of linearly independent factors10 (or principal components). Standard VAE is theoretically similar to probabilistic PCA (pPCA) (Tipping & Bishop, 1999) as both assume the same generative process p(x, z) = pθ(x|z)p(z). Unlike deterministic PCA, pPCA learns a rotation-invariant family of factors instead of an identifiable set of factors. However, in a particular pPCA model, the relative orthogonality among factors is still preserved. This means that the factors learned by different pPCA models are statistically equivalent. We hypothesize that by enforcing independence among latent variables, FactorVAE can also learn statistically equivalent factors (or q(zi|x)) which correspond to visually similar results. We provide a proof sketch for the hypothesis in Appdx. A.16. We note that Rolinek et. al. (Rolinek et al., 2018) also discovered the same phenomenon in β-VAE.

heading: A.16 WHY FACTORVAE CAN LEARN CONSISTENT REPRESENTATIONS?
text: Inspired by the variational information bottleneck theory (Alemi et al., 2016), we rewrite the standard VAE objective in an equivalent form as follows:
min q(z|x)
I(x, z) s.t. Rec(x) ≤ β (18)
where Rec(x) denotes the reconstruction loss over x and β is a scalar.
In the case of FactorVAE, since all latent representations are independent, we can decompose I(x, z) into ∑ i I(x, zi). Thus, we argue that FactorVAE optimizes the following information bottleneck objective: min q(z|x) ∑ i I(x, zi) s.t. Rec(x) ≤ β (19)
We assume that Rec(x) represents a fixed condition on all qi(z|x). Because I(x, zi) is a convex function of q(zi|x) (see Appdx. A.17), minimizing Eq. 19 leads to unique solutions for all q(zi|x) (Note that we do not count permutation invariance among zi here).
To make Rec(x) a fixed condition on all qi(z|x), we can further optimize p(x|z) with z sampled from a fixed distribution like N (0, I). This suggests that we can add a GAN objective to the original FactorVAE objective to achieve more consistent representations.
A.17 I(x, z) IS A CONVEX FUNCTION OF p(z|x)
Let us first start with the definition of a convex function and some of its known properties. Definition 3. Let X be a set in the real vector space RD and f : X → R be a function that output a scalar. f is convex if ∀x1, x2 ∈ X and ∀λ ∈ [0, 1], we have:
f(λx1 + (1− λ)x2) ≤ λf(x1) + (1− λ)f(x2)
Proposition 4. A twice differentiable function f is convex on an interval if and only its second derivative is non-negative there.
10When we mention factors in this context, they are not really factors of variation. They refer to the columns of the projection matrix W in case of PCA and the component encoding functions q(zi|x) in case of deep generative models.
Proposition 5 (Jensen’s inequality). Let x1, ..., xn be real numbers and let a1, ..., an be positive weights on x1, ..., xn such that ∑n i ai = 1. If f is a convex function on the domain of x1, ..., xn, then
f ( n∑ i=1 aixi ) ≤ n∑ i=1 aif(xi)
Equality holds if and only if all xi are equal or f is a linear function.
Proposition 6 (Log-sum inequality). Let a1, ..., an and b1, ..., bn be non-negative numbers. Denote a = ∑n i=1 ai and b = ∑n i=1 bi. We have:
n∑ i=1 ai log ai bi ≥ a log a b
with equality if and only if aibi are equal for all i.
Armed with the definition and propositions, we can now prove that I(x, z) is a convex function of p(z|x). Let p1(z|x) and p2(z|x) be two distributions and let p?(z|x) = λp1(z|x) + (1− λ)p2(z|x) with λ ∈ [0, 1]. p?(z|x) is a valid distribution since p?(z|x) > 0 ∀z and ∫ x ∫ z p?(z|x)p(x) dz dx = 1. In addition, we have:
p?(z) = ∫ x p?(z|x)p(x) dx
= ∫ x (λp1(z|x) + (1− λ)p2(z|x)) p(x) dx
= λ ∫ x p1(z|x)p(x) dx+ (1− λ) ∫ x p2(z|x)p(x) dx = λp1(z) + (1− λ)p2(z)
We write I(x, z) = λI1(x, z) + (1− λ)I2(x, z) as follows:
I(x, z) =λ ∫ x p(x) ∫ z p1(z|x) log p1(z|x) p1(z) dz dx+
+ (1− λ) ∫ x p(x) ∫ z p2(z|x) log p2(z|x) p2(z) dz dx
= ∫ x p(x) ∫ z ( λp1(z|x) log λp1(z|x) λp1(z) + (1− λ)p2(z|x) log (1− λ)p2(z|x) (1− λ)p2(z|x) ) dz dx
≥ ∫ x p(x) ∫ z p?(z|x) log p?(z|x) p?(z) dz dx (20)
=I?(x, z)
where the inequality in Eq. 20 is the log-sum inequality. This completes the proof.

heading: A.18 EXPERIMENTS TO SHOW THAT FACTORVAE LEARNS CONSISTENT REPRESENTATIONS
text: We first trained several FactorVAE models with 3 latent variables on the CelebA dataset. After training, for each model, we performed 2D interpolation on every pair of latent variables zi, zj (i ≤ j) and decoded the interpolated latent representations back to images for visualization. We found that the learned representations from these models share visually similar patterns, which is illustrated in Fig. 26. It is apparent that all images in Fig. 26 are derived from a single one (e.g. we can choose the first image as a reference) by switching the rows and columns and/or flipping the whole image vertically/horizontally. The reason why switching happens is that all latent variables of FactorVAE are permutation invariant. Flipping happens due to the symmetry of q(zi) which is forced to be similar to p(zi) = N (0, I).
We then repeated the above experiment on FactorVAE models containing 65, 100, 200 latent variables, but replacing 2D interpolation on pairs of latent variables with conditional 1D interpolation on individual latent variables to account for large numbers of combinations. We sorted the latent variables zi of each model according to the variance of the distribution of Eq(zi|x(n))[zi] over all data samples x(n) ∼ pD(x) in descending order. Fig. 27 shows results for the top 10 latent variables (of each model). We can see that some factors of variation are consistently learned by these models, for example, those that represent changes in color of the image background. Because these factors usually appear on top, we hypothesize that the learned factors should follow some fixed order. However, many pronounced factors do not appear at the top, suggesting that the sorting criterion is inadequate. We then used the informativeness metric defined in Sec. 4.1 to sort the latent variables. Now the “visual consistency” and “ordering consistency” patterns emerge, (see Fig. 28). We also observed that the number of learned factors is relatively fixed (around 38-43) for all models despite that the number of latent variables varies significantly from 65 to 200.

Review summaries:
This paper defines precise semantics of disentanglement representations and presents evaluation metrics to evaluate such representations. The authors provides information-theoretic characterization disentangled representations along three dimensions: informativeness, separability, and interpretability; and propose metrics to measure them. The authors argue that:- Informativeness can be defined as the mutual information between a particular representation (or a group of representations) w.r.t the data---I(x,z_i) - Separability between two representations can be achieved if they do not share any common information about the data---I(x,z_i,z_j) = 0 - Interpretability with respect to a concept is achieved if a representation contains information about the concept---I(z_i, y_k) = H(z_i) = H(y_k).The authors define a disentangled representation as a representation that is fully separable and fully interpretable and propose a suite of metrics to evaluate disentangled representations.Finally, the authors evaluate several representation learning methods (FactorVAE, betaVAE, AAE) using these metrics on toy and real datasets.

The paper presents a new set of metrics for evaluating disentangled representations in both supervised and unsupervised settings. Disentangled representations are evaluated along three dimensions: informativeness, separability, and interpretability. While previous work offers metrics for similar dimensions (e.g., (Eastwood & Williams, 2018)), the paper suggests that the metrics of the submission are superior to (Eastwood & Williams, 2018), based on a comparison between FactorVAE and Beta-VAE.