Theory and Evaluation Metrics for Learning Disentangled Representations
Authors: Kien Do
Truyen Tran
Venue:  (2020)
Keywords: disentanglement
metrics

Abstract: We make two theoretical contributions to disentanglement learning by (a) defining precise semantics of disentangled representations, and (b) establishing robust metrics for evaluation. First, we characterize the concept “disentangled representations” used in supervised and unsupervised methods along three dimensions–informativeness, separability and interpretability–which can be expressed and quantified explicitly using information-theoretic constructs. This helps explain the behaviors of several well-known disentanglement learning models. We then propose robust metrics for measuring informativeness, separability and interpretability. Through a comprehensive suite of experiments, we show that our metrics correctly characterize the representations learned by different methods and are consistent with qualitative (visual) results. Thus, the metrics allow disentanglement learning methods to be compared on a fair ground. We also empirically uncovered new interesting properties of VAE-based methods and interpreted them with our formulation. These findings are promising and hopefully will encourage the design of more theoretically driven models for learning disentangled representations. 

Published as a conference paper at ICLR 2020
THEORY AND EVALUATION METRICS FOR LEARNING DISENTANGLED REPRESENTATIONS
Kien Do and Truyen Tran
Applied Al Institute, Deakin University, Geelong, Australia (dkdo, truyen. tran) @deak eddu.a
ABSTRACT
We make theoretical contributions to disentanglement learning by (a) defin- ing precise semantics of disentangled representations, and (b) establishing robust metrics for evaluation. First, we characterize the concept "disentangled represen- tations" used in supervised and unsupervised methods along three dimensions- informativeness, separability and whih can be expressed and quantified explicitly using information-thhorred constructs. This helps explain the behaviors of several well-known disentanglement learning models. We then pro- pose robust metrics for measuring informativeness, separability, and interpretability. Through comprehensive suite of experiments, we show that our metrics correctly characterize the representations learned by different methods and are consistent with qualitative (visual) results. Thus, the metrics allow disentanglement learning methods be compared on fair ground. We also empirically uncovered new interesting properties of VAE-based methods and interpreted them with our formu- lation. These findings are promising and hopefully will encourage the design of more theoretically driven models for learning disentangled representations.
INTRODUCTION
Disentanglement learning holds the key for understanding the world from observations, transferring knowledge across different tasks and domains, generating novel designs, and learning compositional concepts (Bengio et al. 2013) Higgins et al. 2017b Lake et al. 2017 Peters et al. 2017 Schmidhuber 1992). Assuming the observation disentanglement learning is to correctly uncover set of independent factors {z, that give rise to the observation. While there has been a considerable progress recent years, common assumptions
generated from latent factors via p(æ(z). the goal of
about disentangled representations appear to be inadequate (Locatello et al. 2019).
Unsupervised disentangling methods are highly desirable as they assume no prior knowledge about the ground truth factors. These methods typically impose constraints to encourage independence among latent variables. Examples of constraints include forcing the variational posterior q to be similar factorial p(=) (Burgess al.) 2018 Higgins al. 2017a), forcing the variational aggregated prior ((2) to be similar to the prior p(=) Makhzzni et al. 2015), adding total correlation loss (Kim & Mnih 2018). forcing the covariance matrix of ((z) be close the identity matrix (Kumar al. (017). and using a kernel-based measure of independence (Lopez et al. 20188). However, it remains unclear how the independence constraint affects other properties of representation. Indeed, more independence may lead to higher reconstruction eтror in some models (Higgins et al. 2017aa Kim & Mnih 2018). Worse still, the independent representations may mismatch human's 's predefined concepts (Locatello et al. 2019). This suggests that supervised methods which associate a representation (or group of representations) z; with a particular ground truth factor yk may more adequate. However, most supervised methods have only been shown to perform well on toy datasets (Harsh Jha et al.) 2018 Kulkarni et al.] 20155] Mathicu et al. (2016) in which data are generated from multiplicative combination of the ground truth factors. It is still unclear about their performance real datasets.
We believe that there are at least two major reasons for the current unsatisfying state of disentangle- ment learning: i) the lack of formal notion of disentangled representations support the design of proper objective functions (Tschannen et al., 20188) Locatello et al.) 2019). and ii) the lack of robust evaluation metries enable fair comparison between models, regardless of their architectures or

---

Published as a conference paper at ICLR 2020
design purposes. To that end, contribute by formally characterizing disentangled representations along three dimensions, namely informativenesss, separability and interpretability, drawing from concepts in information theory (Section 2 We then design robust quantitative metrics for these prop- erties and argue that an ideal method for disentanglement learning should achieve high performance on these metrics (Section 3
We run a series of experiments to demonstrate how to compare different models using our proposed metrics, showing that the quantitative results provided by these metrics are consistent with visual results (Section(). In the process, we gain important insights about some well-known disentanglement learning methods namely FactorVAE (Kim & Mnih 2018), 3-VAE (Higgins al. 2017ap, and AAE (Makhzani et al. 2015).
RETHINKING DISENTANGLEMENT
Inspired by (Bengio al. 2013 Ridgeway 2016), we adopt the notion of disentangled representation learning as "a process of decorrelating information in the data into separate informative representa- tions, each of which corresponds to a concept defined by humans". This suggests three important properties of a disentangled representation: informativeness, separability and interpretability. which we quantify as follows:
Informativeness We formulate the informativeness of particular representation (or group of representations) 2; w.r.t. the data x as the mutual information between 2; and 2:
J.]. pD(L) log
((((z)) dz dx ((2;)
(1)
where q(z;) = J2 (z(2)((Z())) dr. In order to represent the data faithfully, representation 2; should be informative of x, meaning [(x, 2.) should be large. Because [x,, z) H(2)) H (=)
a large value of [2,, 21) means that (zz(ar) x given that ((x) can be chosen to be relatively fixed. In other words, if 2; is informative w.r.t. I, q(2(<z) usually has small variance. It is important to note that T(x, 2;) in Eq. 1 is defined on the variational encoder q(2, and does not require decoder. It implies that we not need to minimize the reconstruction error over (e.g.. in VAEs) to increase the informativeness of a particular 2,.
Separability and Independence Two representations 2i. 2j are separable w.r.t. the data if they do not share common information about x, which can be formulated follows:
T(x, Zi, 23) =
where I (x, 2,, 2;) denotes the multivariate mutual information (MeGill 1954) between x, 2; and 2j- [x, Ri, 2j) can be decomposed into standard bivariate mutual information terms as follows:
1x,,2,
(xx, + 1 (xx 23) 1 (zz 2z)) 1 [z,, 2,) 1 [(, zy (x)
[(x 2,,2,) 21) can be either positive negative. It is positive if 2; and 2, contain redundant information about z. The meaning of a negative [z, 2,, 2z) remains clusive (Bell 2003).
Achieving separability with respect to does not guarantee that 2; and 2j separable in general. 22 and 2j are fully separable or statistically independent if and only if:
I(2s, 2z) =
(3)
we have access to all representations 2, we can generally say that a representation 2; is fully separable (from other representations zy:e) if and only if ([,, 0.
Note that there is a trade-off between informativeness, independence and the number of latent variables which we discuss in Appdx. A.7
Interpretability Obtaining informative and independent representations does not guarantee in- terpretability by human (Locatello et al. (219).. We argue that in order to achieve interpretability, should provide models with set of predefined concepts y. In this case, representation 2; is
2

---

Published as a conference paper at ICLR 2020
interpretable with respect to yı if it only contains information about yv (given that 2; is separable from all other 241 and all yk are distinct). Full interpretability can be formulated as follows:
I T2, yk) H(z+) H()))
(4)
Eq. 4 is equivalent to the condition that is an invertible function of yk- If we want 2 to generalize beyond the observed ys (i.e., (z:) > Hyy))), we can change the condition in Eq. into:
T(z vk) = (ys) or (Ux)22)
(5)
which suggests that the model should accurately predict ys given z;. If 2; satisfies the condition in Eq. 5 it is said to be partially interpretable w.r.t yk-
In real data, underlying factors of variation usually correlated. For example, men usually have beard and short hair. Therefore, it is very difficult to match independent latent variables to different ground truth factors the same time. We believe that in order achieve good interpretability. we should isolate the factors and learn one at time.
2.1 AN INFORMATION-
DEFINITION OF DISENTANGLED REPRESENTATIONS
Given a dataset D = { J where each data point æ is associated with a set of K labeled factors of variation y {vı, ..., yk). Assume that there exists mapping from x to m groups of latent representations 2 = {21, 22,..., 2m which follows the distribution g Denoting (<2(1)) = E,,, g(2 x)) and (=x4 Epp(s) [0(2,/=)] We define disentangled representations for unsupervised cases follows:
Definition (Unsupervised). A representation or group of representations is said be "fully dis- entangled" w.r.t a ground truth factor yı if 2; is fully separable (from 2ạ1) and 2; is fully interpretable w.r.t yk- Mathematically, this can be written
[zz, zys1) 0 and [(z;, un) = H H Ma)
The definition of disentangled representations for supervised cases is similar as above except that now we model g=1|2,)) 3) instead of q(2 (r) and q(=) = -..,y PD (2. (2)2,,,))
Recently, there have been several works (Eastwood & Williams 2018 Higgins al. 2018 Ridgeway & Mozer 2018) that attempted to define disentangled representations. Higgin et, al. (Higgins et al.
2018) proposed definition based on group theory (Cohen & Welling 2014) which is (informalyy)
stated as follows: "A representation 2 is disentangled w.r.t a particular subgroup y/c (from a symmetry group {yu ( if 2 can be decomposed into different subspaces { in which the subspace ze should be independent of all other representation subspaces Zys, and =; should only affected by the action of single subgroup y/k and not by other subgroups yya."" Their definition shares similar observation as ours. However, it is less convenient for designing models and metrics than our coormation-theorete definition.
Eastwood ct. al. [Eastwood & Williams 2018) did not provide any explicit definition of disentangled representations but characterizing them along three dimensions namely "disentanglemen"" "com- pactness", ", and "informativeness" (between any yk). A high "disentanglement" score (z2 1) for 2; indicates that captures at most one factor, let's say yk- A high "completeness" score (= for yk indicates that it is captured by at most one latent zj and j is likely to be i. A high "informativeness" acordfoor for yı: indicates that all information of yk is captured by the representations 2. Intuitively. when all the three notions achieve optimal values, there should be only a single representation #; that captures all information of the factor y/: but no information from other factors Vytk- However, even in that case, z; is still not fully interpretable w.r.t z/k since z; may contain some information in æ that does not appear in yk- This makes their notions only applicable to toy datasets which we know that the data x only generated from predefined ground truth factors y [ux 1 Our definition can handle the situation where we only know some but not all factors of variation in the data. The notions in (Ridgeway& & Mozer 2018) follow those in (Eastwood & Williams 2018), hence, suffer from the same disadvantage.
In Fastwood & Williams 2018) 8). the authors consider the prediction error of yh given instead. High "informativeness" score means this error should be close
3

---

Published as a conference paper at ICLR 2020
3 ROBUST EVALUATION METRICS
We argue that a robust metric for disentanglement should meet the following criteria: i) it supports both supervisedd/ussuprviiss straightforward, i.c. not requiring any training procedure; iv) it provides consistent results across different methods and different latent representations; and it agrees with qualitative (visual) results. Here we propose information-theorent metrics to measure informativeness, independence and interpretability which meet all of these robustness criteria.
models; ii) it can be applied for real datasets; iii) it is computationally
3.1 METRICS FOR INFORMATIVENESS
We measure the informativeness a particular representation 2; w.r.t. x by computing [(x, 2;). If 2; discrete, we can compute / (xr, z4) exactly by using Eq. but with the integral replaced by the sum. If 2; is continuous, we estimate (z, =i) via sampling or quantization. Details about these estimations are provided Appdx.
If (2,) is estimated via quantization, we will have 0 < [(x, 2.) < Hz+). In this case, we can divide (x, 2;) by (2;) to normalize to the range [0, 1J. However, this normalization may change the interpretation of the metric and lead to a situation where representation is less informative than 2, (i.e., I(x, 2;) < I (x, 22)) but still has higher rank than 2j because (22)) < H(z)). A better way is to divide (x, 24) by log (#bins).
3.2 METRICS FOR SEPARABILITY AND INDEPENDENCE
MISJED We can characterize the independence between two latent variables 25. 2j based on [2, 2j). However, serious problem of (zz, 2ÿ) is that it generates the following order among pairs of representations:
[ z1.)) > (Z Zn.)) (Znniis =n.j) 0
where Σr,s žr, are informative representations and An.i. Zn.j are uninformative (or noisy) represen- tations. This means we simply want 2, 2j to be independent, the best scenario that both are noisy and independent (e.g. (<)())) q(2 #N, 1)). Therefore, we propose a new metric for independence named MISJED (which stands for Mutual Information Sums Joint Entropy Difference), defined as follows:
MISJED( 23)) = T(2,, = H(2) + ((2,) H(2,, Ej)
= H(z;) + H(2z) H( 23) + H(z,, 23) - H(2,, žz) I(zi, 23) + H 2,) H(;,, 2z)
where 2; = [x(e and g(z,) = Epp(a) (d(2,(z)) [.) Since ((z,) and ((3)) have less variance than ((2)) and g(2)), respectively, H (21, 2,) - H 22) 2 0. making I(24, 24) 2 0.
To achieve small value of [(=,; zz), two representations 2,, 2j should be both independent and informative (or, in an extreme case, are deterministic given x). Using the MISJED metric, we can ensure the following order: < f(21,, < T(21,5 2n.z) < T(Zn,i 2n.)) If H(2). H(2)), and H 2z) in Eq. 7 are estimated via quantization, we will have I(2, 22) S H(;;) + H(2z) 2 log(#bins). In this case, we can divide f(2;, 25) by 2 log (#bins) normalize it to [0, 1].
WSEPIN and WINDIN A theoretically correct way to verify that particular representation z, is both separable from other 2µ1 and informative w.r.t is considering the amount of information in but not in T that 2; contains. This quantity is the conditional mutual information between r and 2; given 244, which can be decomposed follows:
[(x, 2 (244 = [[x, x4) - [[x, 25, 2441)
[(x, z4) (1(21 - [ [41(z))
= 1xx, x4) - (211 + 1(2,,, (41 [x)
(8)
[x, 2/(24s) ) is useful for measuring how disentangled representation 2; is in the absence of ground truth factors. [xx 2(241) close if z; completely noisy and is high if 2, is disentangled
Note that only informativeness and separability are considered in this case.
4

---

Published as a conference paper at ICLR 2020
For models that use factorized encoders, 2, and 244 usually assumed to be independent given I, hence, I [2,, 2=112) ~ 0 and [x, 2ı(2z1) = [ (z, 2,) - [2,, (44s) which is merely the difference between the informativeness and full separability of 25. For models that use auto-regressive encoders, T(z,, 241/=) > O which means and 2 can share information not in x.
We also compute [2x, zi 21(244) in different way follows:
[(x, z4 21(24)) = [1x, (2,, 2xz1) 2441)
[xx 2) [z, 2411)
If we want to be both independence of zyz and informative w.r.t x, we can only use the first two terms in Eq. 8 to derive another quantitive measure:
[(x zi zz(zzt) = [(x, 2i) - [(z,, Zys))
-Ix zi (2(2)))) 1 (zis Zzt1(2))
However, unlike [xx, 2: 2012) [xx 2; (44) can be negative.
To normalize I (ax, 24 244)) we divide it by (2,) (24) must be estimated via quantization). Note that taking the average of [2, x|241 over all representations to derive single metric for the whole model not appropriate because models with more noisy latent variables will be less favored. For example, if model A has 10 latent variables (5 of them are disentangled and 5 of them are noisy), and model B has 20 latent variables of them are disentangled and 15 of them are noisy), B will always be considered worse than A despite the fact that both are equivalent in term of disentanglement (since 5 disentangled representations are enough to capture all information in æ so additional latent variables should be noisy). We propose two solutions for this issue. In the first approach, we sort [[x, 24 = over all representations descending order and only take the average over the top & latents (or groups of latents). This leads metric called SEPIN @ A which is similar Precision @k:
SEPIN @
[(x, 2r, 124r,)
where 71, ..., PL is the rank indices of L latent variables by sorting
(i 1, ..... L).
In the second approach, we compute the average over all L representations 20, 2L-1 but weighted by their informativeness derive metric called WSEPIN:
WSEPIN Σ pal 2((41)
where P: I(x,21) Σ/ 1(x,,,)
If is noisy representation, [(x, 24) Ro O, thus, z; contributes almost nothing to the final WSEPIN.
Similarly, using the measure (1x, in Eq. 9 we derive other metries INDIN @ and
WINDIN as follows:
INDIN @h
Σ ia,, 124r)) and WINDIN Σ p=((2,21244))
3.3 METRICS FOR INTERPRETABILITY
Recently, several metrics have been proposed to quantitatively evaluate the interpretability of repre- sentations by examining the relationship between the representations and manually labeled factors of variation. The most popular ones are Z-diff score (Higgins et al.) 2017a; Kim & Mnih 2018). SAP (Kumar et al.] 2017), MIG (Chen et al.] 2018). Among them, only MIG is theoretically sound and provides correct computation of [ 2)). MIG also matches with our formulation of "interpretability" Section to some extent. However, MIG has only been used for toy datasets
"SEPIN stands for SEParability and INformativnness *INDIN stands for INDependence and Nnformativeness
5

---

Published as a conference paper at ICLR 2020
GT Factor
Input
latonn code
Input
MIG
RMIG
MIG
RMIG
(a) Unsupervised
(b) Supervised
Figure Differences in probabilistic assumption of MIG and Robust MIG.
like dSprites (Matthey et 2017). The main drawback comes from its probabilistic assumption p(z;, Mss _x))) = ((2g (=) p 2z ( yy (p()) (see Fig. 1]. Note that p(z(*) yy) is distribution over the high dimensional data space, and is very hard to robustly estimate but the authors simplified to be p(n)y)) if ( E Dy (D the support set for a particular value yis) and otherwise. This equation only holds for toy datasets where we know exactly how is generated from y. In addition, since p(n (yx)) depends on the value of yıe. it will be problematic if yv is continuous.
RMIG Addressing the drawbacks of MIG, we propose RMIG (which stands for Robust MIG), formulated as follows:
RMIG = [22-, vv) - (zze ((zje.380 Vk.)
(10)
where T(2,,, ykk) and [(zy=, Mke are the highest and the second highest mutual information values computed between every 2; and yki 2;- and 2jo are the corresponding latent variables. Like MIG, we can normalize RMIG(y)) to [0, 1] by dividing it by (yg) but it will favor imbalanced factors (small
(y))).
RMIG inherits the idea of MIG but differs in the probabilistic assumption (and other technicalities). RMIG assumes that p(zi, yk, ((1)) (<((<<()) )p(v (x())) p(((()) for unsupervised learning and
p(z; Vk. _x)))) q<= (n) (x(()) )p(kk" (x) for supervised learning (see Fig. Ip. Not only this eliminates all the problems of MIG but also provides additional ad advantages. First, we can estimate (2;, M&) using Monte Carlo sampling on p(z(")). Second, p(y (x()) is well defined for both discrete/continuous and determiniisticcstochasiic p(y (z.(m)). If ys is continuous, we can quantize p()cc==(() If p(uk (x())) is deterministic (i.e., a Dirac delta function), we simply set it to 1 for the value of yv corresponding to xx()) and for other values of 1/s- Our metric can also use p(y (x()) from an external expert model. Third, for any particular value yk. we compute ((2///000) for all (n))) E D rather than just for (n()) E Dy,, which gives more accurate results.
JEMMIG high RMIG value ys means that there a representation 2;- that captures the factor yk- However, 2;- may also capture other factors V/t of the data. To make sure that z fits exactly to yu. we provide another metric for interpretability named JEMMIG (standing for Joint Entropy Minuses Mutual Information Gap), computed follows:
JEMMIG(yL)
H yk) yk) 1(2,. Mee + (2))
where (2x,, We) and (2) are defined in Eq. 10
we estimate H(zz-yys) via quantization, we can bound JEMMIG(yJ) between and H(n)) log(#bins) (please check Appdx. A. for details). A small JEMMIG(Js:) score means that zQ- should match exactly to yk and 2jo should not be related to ykk Thus, we can use JEMMIG(ye) to validate whether model can learn disentangled representations w.E.t a ground truth factor yk not which satisfies the definition in Section 2.1 Note that if we replace H (zp, Ee,38 ,ykk) by (ykk) to account for the generalization of 2;- yk. obtain metric equivalent to RMIG (but in reverse order).
To compute RMIG and JEMMIG for the whole model, we simply take the average of RMIG(y:) and JEMMiG(y)) over all yı (k: 1,... K) as follows:
K-1
K-1
RMIG K
Σ
(M and JEMMIG =
JEMMIG
K
O
R==

---

more separable Published as a conference paper at ICLR 2020
3.4 COMPARISON WITH EXISTING METRICS
In T we compare our proposed metrics with existing metrics for learning disentangled repre- sentations. For deeper analysis of these metrics, we refer readers Appdx. One can easily see that only our metrics satisfy the aforementioned robustness criteria. Most other metrics (except for MIG and Modularity) use classifiers, which can cause inconsistent results once the settings of the classifiers change. Moreover, most other metrics (except for MIG) use 12)( instead of ((=((a)) for computing mutual information. This can lead to inaccurate evaluation results since Eg(z, a²) [+i] is theoretically different from 21 ~ ((z; (r). Among all metrics, JEMMIG is the only one that can quantify "disentangled representations" defined in Section. 2.
its own.
Metrics
#classifiers
classifier
nonlineur
continuous factors
real data
relationship
a( (ar)
Z.difT'
lineur/majority
SAP
L X K
threshold value
x
X
v
MIG
none
x
x
Disentanglemeet
LASSO//
Completeness
K
x/V
x
x
x
random forest
Informariveness
Modularity
0
none
V
X
X
Explicitness
K
0B3-V5-fest
x
X
X
x
logistic regressur
WSEPIN'
none
4 /
1
WINDINT
none
/
RMIG
JEMMIG
0
nOne
Table 1: Analysis of different metrics for disentanglement learning. L and K are the numbers of latent variables and ground truth factors, respectively. Metrics marked with * are self-contained. Metrics marked with do not require ground truth factors of variation.
4 EXPERIMENTS
We use our proposed metries to evaluate three representation learning methods namely FactorVAE (Kim & Mnih 2018), 3-VAE (Higgins et al. 2017a) and AAE (Makhzani et al. 2015) on both real and toy datasets which are CelebA (Liu et al. 2015) and dSprites (Matthey et al. 2017). respectively. brief discussion of these models are given Appdx. A.I We would like to show the following points: i) how to compare models based on our metrics; ii) the advantages of our metrics compared to other metrics; iii) the consistence between qualitative results produced by our metrics and visual results; and iv) the ablation study of our metrics.
Due to space limit, we only present experiments for the first two points. The experiments for points (ii) and (iv) are put in Appdx. A.4 and Appdx. A.5 respectively. Details about the datasets and model settings are provided in Appdx. A.2 and Appdx. A.3 respectively. In all figures below, "TC" refers to the coefficient of the TC loss in Factor (Kim & Mnih 2018), "Beta" refers to the coefficient in 3-VAES (Higgins et al. 2017a).
Informativeness In Figs. 2a and 2b we show the average amount of information (of x) that representation 2; contains (the mean of I(2;, and the total amount of information that all representations 2 contain (I(2, r). It is clear that adding the TC term to the standard VAE loss does not affect x22,x)) much (Fig. 2b However, because 2; and zj in FactorVAEs
than those in standard VAEs, Factor VAFS should produce smaller [(2, x) than standard VAES on average (Fig. 2a). We also see that the mean of (z) and T(2,2)) consistently decrease for B-VAFs with higher 3.
Separability and Independence If we only evaluate models based on the separability of represen- tations, B-VAE models with large 3 are among the best. These models force latent representations
7

---

Published as a conference paper at ICLR 2020
24.
Ir
(a) mean of I (2i, 2)
(b) [(2, x)
Figure 2: The informativeness and the total information some FactorVAE and 3-VAE models. For each hyperparameter, we report the mean and the standard error of 4 different runs.
30
model
(a) maa//mea//min of 244)
(b) WSEPIN
(c) SEPIN@3 3
Figure 3: [2z; zy4), WSEPIN and SEPIN @3 of some FactorVAE and 3-VAE models.
to highly separable (as in Fig- 3a we can see that the max/mean/mmn values of I(zs, Zyy) are equally small for 3-VAEs with large 3). In FactorVAEs, informative representations usually have poor separability (large value) and noisy representations usually have perfect separability (* 0) (Fig. 4ap. Increasing the weight of the TC loss improves the max and mean of (2), 241) but not significance (Fig. 3a).
Using WSEPIN and SEPIN @ gives more reasonable evaluation of the disentanglement capabil- ity of these models. Fig. 36 we see that 3- 3-VAE models with 3 = 10 achieve the highest WSEPIN and SEPIN@3 scores, which suggests that their informative representations usually contain large amount of information mation may associate well with the ground truth factors of variation (e.g., 23, 26 in Fig. [4c). The representations of Factor VAEs, despite containing less information of x on their own, usually reflect the ground truth factors more accurately (e.g., 25, 28, 27 in Fig. 44) than those 3-VAEs. These results suggest that ground truth factors should be used for proper evaluations of disentanglement.
that are not shared by other representations. However, this type of infor-
1tw. R9
RDJ
3.7577 53200
(STS 5156, IN5O L
ISEP
8E0 2636
40) 25553
XPG-1100977
421 (S2F-3 TE5O, 3.7013
RR0 23511
000 4175 MRD-1.5455
LS2F=2
(870-3.1705)
(SEP=32425
9P0-0 0250Y
24 20020 ARO-0 001777
21 1S200 0002, (00000000001
2I5 (S8P 0423.
R#0 00000
0252 me-we (300)
21 (amw0 0932. IMPO-0 00011
216 (S8m 2411
meDwo (2200)
2001
0907, 0003)
7131 ISEP 0.2000
pino-
(a) VAE (TC=10)
(b) VAE
(c) 3-VA-VEE (3 10)
Figure 4: Visualization of the representations learned by representative FactorVAE, VAE, and 3-VAE models with separability (1,,, 24:)) and informativeness (1 (I x)) scores. Representations are sorted by their separability scores.
Interpretability Using JEMMIG and RMIG, we see that FactorVAB models can learn represen-
tations that are more interpretable than those learned by 3- 3-VAE models. Surprisingly. the worst Factor models (with TC=10) clearly outperforn the best 3- VAE models (with B = 10). This result sensible because is accordant with the visualization in Figs. and 4c
Comparison with Z-diff In (Chen et al. 2018). the authors have already shown that MIG is more robust than Z-diff (Higgins et al. 70) so compare our metrics with MIG directly.
8

---

Published as a conference paper at ICLR 2020
Dxta
3.0
05
20
358
JANE
(a) JEMMIG
(b) RMIG
(c) JEMMIG VS. RMIG
Figure 5: (a) and (b): Unnormalized JEMMIG and RMIG scores of several FactorVAE and 3-VAE
models. (c): Correlation between JEMMIG and RMIG.
Comparison with MIG On toy datasets like dSprites, RMIG produces similar results as MIG (Chen et 2018). Please check Appdx. A..33 for more details.
0.70
0
X2
0.65
D'ontt spremeet
0.55
Comp
0.50
0.6
0.45
Magie
0.5
0.40
0.4
1000
035
JEMNNG
FHVIG
EVeric
(a) JEMMIG vs. Disentanglement
(b) JEMMIG Completeness
(c) JEMMIG vs. Error
1-
0.70
0.55
0B-
0.50
0:55
0
0.50
0.6
0.45
0.3
12 12
0.40
035
0.50
075
1.50
1.25
1.50
175
0.25
0.75
3.00
1.25
1.50
1.75
025 0.50
3.25 3.50
1.75
RMG
RMNG
RMG
(d) RMIG vs. Disentanglement
(e) RMIG vs. Completeness
(f) RMIG vs. Error
Figure 6: Comparison between JEMMIG/RMIG and the metrics in (Eastwood & Williams, 2018) Because the competing metrics do not apply for categorical factors (see Appdx. A.8 for detailed analysis), we exclude the "shape" factor during computation. Following (Eastwood & Williams 2018), use LASSO classifiers with the L1 coefficient is 0.002. Blue dots denote FactorVAE models and orange dots denote 3-VAE models.
Comparison with "disentanglement,, "completeness" and "informativeness" In Fig. we show the differences in evaluation results between JEMMIG/RMIG and the metrics in (Eastwood & Williams 2018). We can easily see that JEMMIG and RMIG are much better than "disentanglement" "completeness" and "informativeness" (or reversed classification error) in separating FactorVAE and 3-VAE models. Among the three competing metrics, only "informativeness" (or [[2, yts) seems to be correlated with JEMMIG and RMIG. This is understandable because when most representations are independent in case of PactorVAB. and 3-VA,s, we have T(2, Uv.) s T(2 M.)) ~ T(2-- ,(2+,)))) [2 "Disentanglemennt" and "completeness", by contrast, are strongly uncorrelated with JEMMIG and RMIG. While JEMMIG consistently grades standard VAEs (3 = 1) worse than other models (Fig. 5a "disentanglement" and "completeness" usually grade standard VAEs better than some Factor VAE models, which seems inappropriate. Moreover, since "disentanglement" and "completeness" are not well aligned, using both of them at the same time may cause confusion. For example, the model "28 beta20" has lower "disentanglement" score yet higher "completeness" score

---

Published as a conference paper at ICLR 2020
TC
TC
TC
20
Flodd
1984
(a) Disentanglement
(b) Completeness
(c) Error
Figure 7: "disentanglement". "completeness" and "informativeness" (error) scores of several Factor- VAE and 8-VAE models.
FLyyne
1.02
3.60
Ragel
0.91
0.85
0.000
J30t C
7054
FISCE
(a) JEMMIG vs. Modularity
(b) Modularity (original)
(c) Modularity (correct)
Figure 8: (a): Comparison between JEMMIG and "modularity" (#bins=100). (b) and (c): "modularity" scores of several FactorVAE and 3-VAE models. The original version computes [[z;, y&s using Eg(s, a) [2]] while the correct version compute [zi, using (==:(ar).
than the model "32 "etaee0 (Figs. 6a and 6b) so it is hard to know which model is better than the other at learning disentangled representations.
From Figs. 7a and 7b we see that "disentanglement" and "completeness" blindly favor 3 3-VAE models with high 9 without concerning about the fact that representations in these models are less informative than representations in FactorVAEs (Fig. 7c). Thus, they are not good for characterizing disentanglement general.
Disentanglement and "completeness" are computed based on weight matrix with an assumption that the weight magnitudes for noisy representations are close to However, this assumption is often broken in practice, thus, may lead to inaccurate results (please check Appdx. A.9 for details).
Comparison with "modularity" "modularity" and "explicitness" (Ridgeway & Mozer 2018) similar "disentanglement" and "informativened (Eastwood & Williams 2018) in terms of concept, respectively. However, they are different terms of formulation. We exclude "explicitness" in our experiment because computing on dSprites is time consuming. In Fig. Sa we show the correlation between JEMMIG and "modularity". We consider two versions of "modularity". In the first version (Fig. (b). [(2,, yk is computed from the mean of 2; q(2][2). This is the original implementation provided by (Ridgeway & Mozer 2018). 3. In the second version (Fig. [c). [(2,, Vkk is computed from (2, .z). We can see that in either case, "modularity" often gives higher scores for standard VAFs than for FactorVAEs. It means that like "disentanglement" "modularity" itself does not fully specify "disentangled representations" defined in Section 2.1
CONCLUSION
We have proposed an rnfformation-theoreee definition of disentangled representations and designed ro- bust metrics for evaluation, along three dimensions: informativeness, separability and interpretability. We carefully analyze the properties of our metries using well known representation learning models namely FactorVAE, 3-VAE and AAE on both real and toy datasets. Compared with existing metrics, metrics more robust and produce more sensible evaluations that are compatible with visual results, Based on our definition of disentangled representation in Section 2.I, WSEPINJEMMIG are the two key metrics in ground truth labels are unavailable/availabee respectively.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Error function. http://www...ppeeii.ooo//////rra
May 2019.
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410. 2016.
Anthony Bell. The co-information lattice. In Proceedings the Independent Component Analysis and Blind Signal Separation: ICA, volume 2003, 2003.
the Fifth International Workshop on
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-1825 2013.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerehner. Understanding disentangling in beta-vae. arXiv preprint DrXiv:1804.03599 2018.
Tian Qi Chen, Xuechen Li. Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. arXiv preprint arXiv: 2018.
Taco Cohen and Max Welling. Learning the irreducible representations commutative lie groups.
In International Conference on Machine Learning, pp. 1755-1763, 2014.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. 2018.
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and VSR Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 805-820, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017a.
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murra Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. arXiv preprint arXiv: 1707.033,, 2017b.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations, arXiv preprint arXiv: 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. ICML, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1eXiv:1412.6980. 2014.
Tejas Kulkamni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in Neural Information Processing Systems, pp. 2549-2547, 2015.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint urXiv:1711.0088 2017.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. ICMI, 2019.
11

---

Published as a conference paper at ICLR 2020
Romain Lopez, Jeffrey Regier, Michael I Jordan, and Nir Yosef. Information constraints auto- encoding variational bayes. In Advances in Neural Information Processing Systems, pp. 61 1125, 2018.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprini wrXiv:151..0564: 2015.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In Advances in Neural Information Processing Systems, pp. 5040-5048, 2016.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http://giiibbcoomeeeeppppdddddds 2017.
William McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory, 4(4):93-111, 1954.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. MIT press, 2017.
Karl Ridgeway. survey of inductive biases for factorial representation learning. arXiv preprint arXiv:1612.05299, 2016.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. In Advances Neural Information Processing Systems, pp. 185-194, 2018.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea directions (by accident). arXiv preprint arXiv: /8/2.06775 2018.
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation, 4(6):863-879, -799 1992.
Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 61(3):61 1999.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation learning, arXiv preprint arXiv: 1812.88069 2018.
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
A. REVIEW OF FACTORVAES, 3-VAES AND AAES
Standard VAEs are trained by minimizing the variational upper bound L of log (z) follows:
(x) (=)+) [- log pe [z|=)] + Dxz (90(2)z) [p())]
(11)
where 4a (=|x)) is an amortized variational posterior distribution. However, this objective does not lead to disentangled representations (Higgins al. 2017a).
3-VAEs (Higgins et al. 2017a) penalize the KI. term in the original VAE loss more heavily with
coefficient 3 >> 1:
E pp)(s) (=la)) |- [ogp)(r|>>]]] 3Dxz (1<(=|=)lp(=)))
Since (x) [D (gs (=<=))))]) (a 2) + DE (4s (=) (=)]]p(=)), more penalty on the KL term encourages (2) to be factorized but also forces to discard more information in I.
FactorVAEs (Kim Mnih 2018) add a constraint to the standard VAE loss to explicitly impose factorization of 4
C FactorVAE LVA + DDKE go [
(12)
where DK (ga(2) II. (xx)) 2 0 is known as the total correlation (TC) of z. Intuitively, y can be large without affecting the mutual information between and æ, making FactorVAE more robust than 3-VAE in learning disentangled representations. Other related models that share similar ideas with PactorVAEs are are 3-TCVAEs (Chen et al. 2018 and DIP-VAEs (Kumar et al. 2017).
The loss of AAES (Makhzani et al. 2015) is derived from the standard VAE loss by removing the term (x, 2):
AAE
Epp(x) (=++))) |- log po (x|=)]] + DEL (go())llp(=))
Different from the losses of B-VAEs and FactorVAEs, AAE loss is not valid upper bound on log (I)).
A.2 DATASETS
The CelebA dataset (Liu al. attributes. We resize these images to 64 x 64. The dSprites dataset (Matthey et al.) 2017) is toy dataset generated from differrnt factors of variation which are "shape" (3 values), "scale" (6 values), "rotation" (40 values), "x-position" (32 values), "y-position" (32 values). Statistics of these datasets are provided in Table
2015) consists of more than 200 thousands face images with 40 binary
Dataset
#Train
#Test
Image size
CelebA
162,770
19,962
64x64x3
dSprites
737,280
0
64x64x] 1
Table 2: Summary of datasets used in experiments.
A.3 MODEL SETTINGS
For Factor VAE, 3- VAE and AAE, we used the same architectures for the encoder and decoder (see Table 3 and Table 4 following (Kim Mnih 2018). We trained the models for 300 epochs with mini-batches of size 64. The learning rate is 10 for the encoder/decoder and is 10-1 for the discriminator over z. We used Adam (Kingma & Ba 2014) optimizer with 3 0.5 and 83 = 0.99. Unless explicitly mentioned, we use the following default settings: i) for CelebA: the number of
*Only Vac and AAE discriminator over
13

---

Published as a conference paper at ICLR 2020
latent variables is 65, the TC coefficient in FactorVAE is 50, the value for 3 in 3- is 50, and the coefficient for the generator loss over & in AAE is 50; ii) for dSprites: the number of latent variables is 10.
Encoder
Decoder
Discriminator Z
dims: x43
dim: 65
2 dim: 65
conv (4, 4, 32), stride 2, ReL.U
FC x 1 x 256, ReLU
FF 1000, LReLU]
conv (4, 4, 32), stride 2, ReLU
deconv (4, 64), stride 1, valid, ReLU
FC
conv (4, 4, 64), stride 2, ReLU
deconv (4, 4, 64), stride 2, ReLU
D(z):
conv (4, 4, 64), stride 2, ReLU
deconv (4, 4, 32), stride 2, ReLU
conv (4, 256), stride valid, ReLU
deconv (4, 4, 32), stride 2, ReLU
FC 65 dim:
deconv (4, 4. 3), stride 2, ReLU
x dim: 64x64x3
Table 3: Model architectures for CelebA.
Encoder
Decoder
Discriminator Z
dims: 64x64x1
2 dim:
dim:
conv (4, 4, 32), stride 2. ReLU
FC 128, ReLU
SX
1000, LReLUJ
conv (4, 4, 32), stride 2, ReLU
FC 4x4x64, ReLU
FC
conv (4, 4, 64), stride 2, ReI.U
deconv (4, 4, 64), stride ReI.U
D(=):
conv (4, 4, 64), stride 2, ReLU
deconv (4, 4, 32), stride 2, ReLU
FC 128, ReLU
deconv (4, 4, 32), stride 2, ReLU
FC
deconv 1), stride ReLU
dim: 10
dim: 64x64x1
Table 4: Model architecture for dSprites.
A.4 CONSISTENCE BETWEEN QUANTITATIVE AND QUALITATIVE RESULTS
A.4.1 CELEBA
Informativeness We sorted the representations of different models according to their informative- ness scores the descending order and plot the results in Fig. There are distinct patterns for different methods. AAE captures equally large amounts of information from the data while FactorVAD and B- VAE capture smaller and varying amounts. This because FactorVAE and 3-VAE penalize the informativeness of representations while AAE does not. Recall that I (2,, z) H (z;) H (2 (z).
For AAE, H (2, xx) and (2;) equal to the entropy of A/ I). For FactorVAE and 3- VAE, H (= > 0 and H is usually smaller than the entropy of (0, T) due to narrow g(=; 6
(a) VAE (TC=50)
(b) 1.8--AB (8=50)
(c) AAE (Gz=50)
Figure 9: Normalized informativeness scores (bins= of all latent variables sorted in descending order.
'Nte that (2,) does not depend whether g(2,) is zero-centered
14

---

Published as a conference paper at ICLR 2020
In Fig. sudden drop of the scores 0 for some FactorVAE's and 3. 3-VAE's representations.
These representations 24 are totally random and contain no information about the data (i.c., ( R3 ()),))). We call them "noisy" representations and provide discussions in Appdx. A.7
We visualize the top 10 most informative representations for these models in Fig. 10 AAE's representations more detailed than FactorVAE's and B-VAE's, suggesting the effect of high informativeness. However, AAE's representations mainly capture information within the support of pD(x). This explains why still see face when interpolating AAE's representations. By contrast, FactorVAE's and 3-VAE's representations usually contain information outside the support of pD()). Thus, when we interpolate these representations, we may see something not resembling face.
408 2012
2401020
0
0
0:
(a) VactorVAE (TC=50)
(b) 3-VE (3=50)
(c) AAE (Gz=50)
Figure 10: Visualization the top informative representations. Scores are unnormalized.
Separability and Independence Table 5 reports MISJED scores (Section 3.2) for the top most informative representations. FactorVAE achieves the lowest MISJED scores, AAE comes next and 8- VAE is the worst. We argue that this is because FactorVAE learns independent and nearly deterministic representations, 3-VAE learns strongly independent yet highly stochastic representations, and AAE, on the other extreme side, learns strongly deterministic yet not very independent representations. From Table 5 and Fig. []] it clear that MISJED produces correct orders among pairs of representations according to their informativeness.
MISJED
(unnormalized)
21, 22
21, 23
21,2-1
21,2-2
2-1,2-2
2-1, 8-3
TAE
0.008
0.009
2.476
2.443
4.858
4.892
3-VAE
0.113
0.131
3.413
3,401
6.661
6.739
AAE
0.022
0.023
0.022
0.021
0.021
0.020
Table Unnormalized MISJED scores (#bins = 50, 10% data). 21, 22, and 2- 2_22 2_3 denote the top 3 and the bottom latent variables sorted by the informativeness scores in descending order. Boldness indicates best results.
(a) FactorVAE (TC=50)
(b) 3-VAE (3=50)
(c) AAE (Gz=50)
Figure 11: Normalized MISJED scores of all latent pairs sorted by their informativeness.
15

---

Published as a conference paper at ICLR 2020
Interprefability We report the RMIG scores and JEMMIG scores for several ground truth factors in the CelebA dataset Tables and respectively. In general, FactorVAE learns representations that agree better with the ground truth factors than 3-VAE and AAE do. This is consistent with the qualitative results in Fig. 12 However, all models still perform poorly for interpretability since their RMIG and JEMMIG scores very far from and 0, respectively. We provide the normalized JEMMIG and RMIG scores for all attributes in Fig.
RMIG (normalized)
Bangs
Black Hair
byeglasses
Goatee
Male
Smiling
H=0.4256
H=0.5500
H=0.2395
H=0.2365
H=0.6801
H=0.6923
FactorVAB
0.1742
0.0430
0.0409
0.0343
0.0060
0.0962
B-VAB
0.0176
0.0223
0.0045
0.0325
0.0094
0.0184
AAE
0.0035
0.0276
0.0018
0.0069
0.0060
0.0099
Table 6: Normalized RMIG scores (#bins=100) for some factors. Higher is better.
JEMMIG (normalized)
Bangs
Black Hair
Eyeglasses
lasses
Goatee
Male
Smiling
H=0.4256
H=0.5500
H=0.2395
H=
H=0.6801
H=0.6923
FactorVAB
0.6118
0.6334
0.6041
0.6616
0.6875
0.6150
B- B-V
0.8632
0.8620
0,8602
0.8600
0.8690
0.8699
AAE
0,8463
0.8613
0.8423
0.8496
0.8644
0.8575
Table 7: Normalized JEMMIG scores (#bins=100) for some factors. Lower is better.
Kones MMlgaaoo 1742, 42ne1
Block Nai IM soowo DANO, Madt KNSeE
Sendlay LMI capwt orec Hud aoons
(a) FactorVAE (TC=50)
(b) 3-VAE (3=50)
(c) AAE (Gz=50)
Figure 12: Top representations that most correlated with ground truth factors. For each representation, we show its mutual information with the ground truth factor.
A.4.2 DSPRITES
that representations of AAE have equally high informa- tiveness scores while the remaining 5 representations have nearly zeros informativeness scores. This because AAE needs only 5 representations capture all information in the data. FactorVAF also
Informativeness From Fig. 4
16

---

Published as a conference paper at ICLR 2020
Facteres PaleVA
6.8-
Grap Hoat
for
yrtar the
Houtt Stp
Rer Hith Coeate Block Fiss growe Big Lise Eiie
por Rustarie
gloee Guorts
the soeer the
howe Phinte Jor ovel Mase
BaY. be
JEMMIG (normalized)
6.55
Factensn Setave
A42
6.45
0.33
600
Bangg aored Sorting peesing
Amertto abuff
pur Bacc porarig Geoperee
Humeca
Gooeve
No Brard
Greg.te Ohiice
wesinng to
Artedt dye
eooyy
upyy the Fre sar sraring
Mase
Ord Pare
chuslty
So
strigis
thoter
Cbock
inumng Ave
(b) RMIG (normalized)
Figure 13: Normalized JEMMIG and RMIG scores for all attributes in the CelebA dataset. We sorted the JEMMIG and RMIG scores of the FactorVAB in ascending and descending orders, respectively.
needs only 5 representations but some are less informative than those of AAE. Note that the number of ground truth factors of variation in dSprites dataset also
(a) Vac
(b) 3-VAE
(c) AAE
Figure 14: Normalized informativeness scores (bins=100) of all latent variables sorted in descending order.
(a) FactorVAE
(b) 3-VAE
(c) AAE
Figure 15: Normalized MISJED scores (bins=100) of all latent pairs sorted by their informativeness.
Separability and Independence Fig. 15 shows heat maps of MISIED scores the three models.
Interpretability From Tables. Tables.Baadd3 we see that Factor is very good at disentangling "scale", "x-position" and "y-position" but fails to disentangling "shape" and "rotation". However, FactorVAB still performs much better than 3- B-V and AAE. These results are consistent with the visual results in Fig. 16
17

---

Published as a conference paper at ICLR 2020
Also note that in FactorVAE, the RMIG scores for "scale" and "x-position" are quite similar but the JEMMIG score for "scale" is higher than that for "x-position". This is because the quantized distribution (with 100 bins) of a particular representation 2; fits better to the distribution of "x "x-position" (having 32 possible values) than to the distribution of "scale" (having only 6 possible values).
Shape
Scale
Rotation
Pos X
Pos Y
FactorVAB VAE
0.2412
0.7139
0.0523
0.7198
0.7256
3-VAE
0.0481
0.1533
0.0000
0.4127
0.4193
AAE
0.0053
0.0786
0.0098
0.3932
0.4509
Table 8: Normalized RMIG scores (bins=100).
Shape
Scale
Rotation
Pos X
Pos Y
FactorVAE
0.6841
0.3422
0.7204
0.2908
0.2727
3-VAE
0.8642
0.8087
0,9199
0,5629
0,5576
AAE
0.8426
0.8143
0.8665
0.5738
0.5258
Table 9: Normalized JEMMIG scores (bins=100).
(a) Vac (Shape)
(b) 3-VAE (Shape)
(c) AAE (Shape)
Scale steedwrd 33.91 mmreeo 1998)
Scilo J016 Dicc 1511, EM1 ESST, Hel 2018
Scare (0500) pVM 23:4 Hot
(d) Factor VAE (Scale)
(c) B- B-E (Scale)
(f) AAE (Scale)
Ruttation G-60003, EWM-- Hed 50001
gaMinG 2033
Extationn (PPRG E033 12986-0.2655
Factor (Rotation)
(h) 3-VAE (Rotation)
AAE (Rotation)
Tou SINEGHC.735E JEMN -O 2200 1H-3460711
ITM C-0-4277 JVM 11-2-4557)
11930-0 2932 IWWI- 4E57+
Factor (Pos X)
(k) 3-VAE (Pos X)
(I) AAE (Pos X)
jentiiud (5M0 2297 Hckono
(MM firk 5144, Burtud BRA7)
ri (12L (HE/- 4632)
(m) VactorVAE (Pos Y)
(n) 3-VAE (Pos Y)
(0) AAE (Pos
Figure 16: Top 3 representations sorted by their mutual information with different ground truth factors.
A.S ABLATION STUDY OF OUR METRICS
Sensitivity of the number of bins When estimating entropy and mutual information terms using quantization, we need to specify the value range and the number of bins (#bins) in advance. this paper, we fix the value range be [-4, 4] since most latent values fall within this range. We only investigate the effect of #bins on the RMIG and JEMMIG scores for different models and show the results in Fig. 17 (left, middle).
18

---

Published as a conference paper at ICLR 2020
We can see that when #bins is small, RMIG scores are low. This is because the quantized distributions Q(z+)) and ((z))) look similar, causing I (23-, yk) and (zjo, ylu) to be similar as well. When #bins is large, the quantized distribution ((2+-) and (2zye) look more different, leading to higher RMIG scores. RMIG scores are stable when #hins > 200, which suggests that finer quantizations do not affect the estimation of [= yk) much.
Unlike RMIG scores, JEMMIG scores keep increasing when we increase #bins. Note that JEMMIG only differs from RMIG in the appearance of (2 yy).. Finer quantizations of z;- introduce more information about 2i. hence, always lead to higher [2+,, yh) (see Fig. 17 (right)). Larger JEMMIG scores also reflect the fact that finer quantizations of 2;- make 2;- look more continuous, thus, less interpretable w.r.t the discrete factor yk-
We provide a detailed explanation about the behaviors of RMIG and JEMMIG w.r. #bins Ap- pdx. I. Despite the fact that thins affects the RMIG and JEMMIG scores of single model, the relative order among different models remains the same. It suggests that once we fixed the #bins, we can use RMIG and JEMMIG scores to rank different models.
0.5
0.5
0.7
0e
bele20
t:10
1:20
1:20
k:30
1:50
1c50
0.3
0.5
bea20
betal0
bea20
bc510
bets30
bea100
0.
300
300
400
Figure 17: Dependences of RMIG (normalized), JEMMIG (normalized) and 1 Ek= H Mk) on the number of bins. The dataset is dSprite.
Sensitivity of the number of samples From Fig. 18 (left, right), it is clear that the sampling estimation is unbiased and is affected much by the number of samples.
Sensitivity of sampling in high dimensional space One thing that we should concern about is the performance of our metries when the number of latent representations (#latents) is large (or is high-dimensionnl). In Fig. 19a we see that the informativeness of an individual representations 2; not affected by #latents. When we increase #latents, additional representations are usually noisy (1(z,, x) R 0). The total amount of information captured by the model (1x z)), by contrast, highly depends on #latents (Fig. 19b). Unusually, increasing #latents reduces [(x, instead of increasing it. We have not found the final answer for this phenomenon but possible hypotheses are: i) On high dimensional space where most latent representations are noisy (e.g. #latents=20), q(2(x) may
look more similar to q(z), causing the wrong calculation of log q[=)
g(z(=) ii) when #latents is large,
(<222) M a(2 (z) is very tiny, thus, may lead floating point imprecision In Fig. 19c
We tried (<]/=) exp
D11 log ((z()))))) and gives similar results as a(=lx>) I1 (2/(x))
2.4
0:30
2.5
1c20
2.2
1050
2.0
1:20
t:50
1.0
betald
1.0
beta50
1.5
beta30
betad0
belan0
1.4
10000 20000 30000
4050n
50000
10000
20500
30000
40000
50000
#somp es
@ @samples
Figure 18: Dependences of JEMMIG and WSEPIN on the number of samples. All models have 10 latent variables. The dataset is dSprites.
19

---

Published as a conference paper at ICLR 2020
that increasing #latents increases [2, 2 This makes sense because larger #latents means that 24z will contain more information. However, the change of [22 241) is sudden when #latents change from 10 to 15, which is different from the change of #latents from 5 to 10 or 15 to 20. Recall that (2, Zgs) = (2;) + H (2+s) H(2). Since H(;;) can be computed stably, we only plot H (2 (z) and show it in Fig. 19d
19d We can that when #latents 20, (2 (=) which
means we cannot differentiate between ((<+zs) and q(z). The instability of computation for high dimensional latents becomes clearer in Fig. 19e [x, 2; (244)
[(,, 2442 can be when #latents = 15 or 20. This causes the instability of WSEPIN in Fig. 19r despite the results look reasonable. JEMMIG and RMIG are calculated on individual latents they are not affected by #latents and can provide consistent evaluations for models with different #latents.
[xx
1.50
0.00
1.55
-0.25
2.0
1.00
x1
1.5
0.75
-0.75
1.0
-1.00
0.25
0.0
0.00
-1.0
10
15
20
15
20
15
#latents
Alatents
#latents
#latents
(a) I (2x, 24)
(b) 1 (zz, 2)
(c) 1 (2,, 2nhh))
(d) H (24c) (2)
2.5
10x, 21 11x-1 0000
JEMAACAYR W
JEMWIN
3.5
0.5
0
10
15
0.0
0.0
20
#latents
elatents
elatents
*latents
(e) (xx, 2) [x,, Z+te)
(f) WSEPIN
(g) JEMMIG(J+)
(h) JEMMIG
Figure 19: Dependences of various quantitative measures the number of latents. All measures computed via sampling. The model used in this experiment is 3-VAE with 3 = 10.
A.6 EVALUATING INDEPENDENCE WITH CORRELATION MATRIX
For every (n)) sampled from the training data, we generated m = latent samples 2 q(2g) (x()) and built a correlation matrix from these samples for each of the models FactorVAE, 3-VAE and AAE. We also built another version of the correlation matrix which is based on the E m(0) (n)) [i]] (called the conditional means) instead of samples from {(2](=("))). Both are shown in Fig. 20 We can see that the correlation matrices computed based on the conditional means incorrectly describe the independence between representations of FactorVAE and 3- VAV. AAE is not affected because it learns deterministic 2; given æ. Using the correlation matrix is not a principled way to evaluate independence in disentanglement learning.
(n.m)
20

---

Published as a conference paper at ICLR 2020
(a) FactorVAE (stochastic)
(b) 3 --V (stochastic)
(c) AAE (stochastic)
(d) VactorVAE (deterministic)
(e) 3-VAE (deterministic)
(1) AAE (deterministic)
Figure 20: Correlation matrix of representations learned by Factor VAE, 38-AEE and AAE.
A.7 TRADE-OFF BETWEEN INFORMATTIVNESSS, INDEPENDENCE AND THE NUMBER OF LATENT VARIABLES
Before starting our discussion, we provide the following fact:
Fact 2. Assume we try to fill fixed-size pool with fixed-size balls given that all the balls must be inside the pool. The only way to increase the number of the balls without making them overlapped reducing their size.
Zn
x
AAE
FactorVAE
Illustration of representations learned by AAE and FactorVAF. A big red eircle represents the total amount of information that x contains or H(x) which limited by the amount training data. Blue circles are informative representations 2 and the size of these circle indicates the informativeness of 24. Green circles are noisy representations AAE does not contain only Factor does.
Figure 21:
In the context of representation learning, pool is with size (xx) which depends on the training data. Balls are z, with size H (z:). Fact. 2 reflects the situation AAE (see Fig. left). In AAE, the condition "all balls are inside the pool" is met. H(z) as
all z; deterministic given the entropy of (0, which fixed so the condition "fixed-size balls" also met. Therefore, when the number of latent variables in AAE increases, all 2, must be less informative (i.e., (22, must decrease) given that the independent constraint on the latent variables is still satisfied. This empirically verified in Fig. 22 as we see the distribution of E (m(=)) [2]] over all 2((1) ~ pD (x) becomes narrower when increase the number of representations from 65 to 200. Also note that increasing the number of latent variable from 65 to 100 does not change the distribution. This suggests that 65 or 100 latent variables are still not enough to capture all information in the data.
Factor VAE, however, handles the increasing number of latent variables in a different way. Thanks to the KI. term in the loss function that forces q(2 (2)) to be stochastic, FactorVAF can break the
21

---

Published as a conference paper at ICLR 2020
(a) Z dim=65
(b) z dim=100
(c) z dim=200
Figure 22: Distribution of E Bg(ss/am))(2) over all x(1) ~ pz( of a particular representation 2; for different AAE models.
constraint in Fact and allows the balls stay outside the pool (see Fig. 21 right). If we increase the number of latent variables but still enforce the independence constraint on them, Factor VAE will keep a fixed number of informative representations and make all other representations "noisy" with zero informativeness scores. We refer to that capability of Factor VAE as code compression.
A.8 ANALYSIS OF EXISTING METRICS FOR DISENTANGLEMENT
In this section, analyze recent metrics, including Z-diff score (Higgins al. 2017a Ta
Kim & Mnih
2018), Separated Attribute Predictability (SAP) (Kumar et al. 2017)) Mutual Information Gap (MIG)
(Chen al. 2018), (Ridgeway & Mozer 2018).
(Eastwood & Williams 2018)
Modularity/Expliiineee
The main idea behind the Z-diff score (Higgins al. 2017a Kim & Mnih 2018) is that if ground truth generative factor Yi (k E {0, I,.., K () is well aligned with a particular disentangled representation z; (although we not know which i), we can use a simple classifier to predict using information from Higgins et al. (Higgins et al. (017a) use a linear classifier while Kim et. al. (Kim & Mnih 2018) use a majority-vote classifier. The main drawback of this metric is that assumes knowledge about all ground truth factors that generate the data. Hence, it is only applicable for a toy dataset like dSprites. Another drawback lies in the complex procedure to compute the metric, which requires training classifier. Since the classifier is sensitive to the chosen optimizer, hyper-paramette and weight initialization, is hard to ensure a fair comparison.
The SAP score (Kumar et al. 2017) is computed based the correlation matrix C' between the latent variables z and the ground truth factors If a latent and a factor are both continuous,
the (square) correlation Ci,A between them equal to Cov' (as.M)) Nar(zx)Nar(2n)) and in [0, 1]. However, if the factor y/ discrete, computing the correlation between continuous and discrete variables not straightforward. The authors handled this problem by learning classifier that predicts yk given 2; and used the balanced prediction accuracy as a replacement. Then, for each factor yes they sorted ParainceeIreediction
C in the descending order and computed the difference between the top two scores. The mean of the difference scores for all factors was used as the final SAP score. The intuition for this metric is that if latent 2, is the most representative for factor yk (due to the highest correlation score), then other latent variables Zyes should not be related to 3/k. and thus, the difference score for yı should be high. We believe the SAP score is more sensible than dififf but it is only suitable when both the ground truth factors and the latent variables are continuous as no classifier is required. Moreover, if we have K discrete ground truth factors and L latent variables. the number of classifiers we need to learn is L x K, which is unmanageable when L is large.
The MIG score (Chen al. 2018) shares the same intuition as the SAP score but is computed based on the mutual information between every pair of and yı instead of the correlation coefficient. Thus, the MIG score is theoretically more appealing than the SAP score since it can capture nonlinear relationships between latent variables and factors while the SAP score cannot. The MIG score, to some extent, reflects the concept "interpretability" that we discussed Section 2 in the main text.
To achieve balance, the classifier uses the same number of samples for all categories of yk during training and testing
22

---

Published as a conference paper at ICLR 2020
Eastwood et. al. (Eastwood & Williams 2018) proposed three different metrics namely "disentan- glement", "completeness" . and "informativvness" to quantity disentangled representations. These metrics are computed based so-called "important matrix" R whose element Rag is the relative im- portance of 2; other z44)) ) in predicting yk- More concretely, for each factor yı (k = O..., K- 1), they train regressor (LASSO or Random Forest) predict yk from 2 and the weight vector provided by this regressor to define R. The "disentanglement" score D. quantifies the degree to which a latent 2; captures at most one generative factor yk- Dj is computed as D; = (1 Hx (P.)) Rik which can be seen as the "proba- where Hx(P,) = Σx= -Px log Pa and Pik ΣE= Rag
bility" of predicting yk instead of Uz4 from 2,. Similarly, the "completeness" score Cs quantifies the degree which ground truth factor ys is captured by single latent 2; (i 0, .... L 1), computed as C 1 (P.))) where Hy (P.x) = E -P:s log Pik and Pax Ee== Rura
R4g The
"informativeness" score describes the total amount of information a particular factor y/ captured by all representations However, the authors use the prediction error Eg of the k-th regressor to quantify "informativeness" instead of [(),, 2). Despite being well-motivated, these metrics still have several drawbacks. First, using three different metrics to quantify disentangled representations is not as convenient as using a single metric like MIG (Chen et al. 2018). For example, how can we compare two models A and B if A has better "disentanglement" score but worse "completeness" score than B? Second, these metrics do not apply for categorical factors with C' classes since in this case the model weight is not a vector but an C matrix. Third, defining the pseudo-distribution seems ad hoc because i) the weight magnitudes Ri are unbounded and can vary
Rut Pas Σx Ri significantly (see Appdx. A.9., and ii) Pax strongly depends the available ground truth factors (e.g. the value of P.x will change if we only consider 2 instead of 5 factors).
Ridgeway al. (Ridgeway & Mozer, 2018) proposed two metries called "modularity" and "explicitness" that have similar interpretations as "disentanglement" and "informativeness" dis- cussed above but differ in implementation. Specifically, they compute the "modularity" score ΣK (J(an >-Ta) where k* argmax (zis yk) and P(.. (x(K
Mi for representation 2; M; 1
T&k { [1 (()) wx-) k
Like the "disentanglement" score D,, M, is also ad hoc and is
otherwise
undetined when the number of ground truth factors is 1. The "explicitness" score Esg for each ground truth factor yı: is computed as the ROC curve of a logistic classifier that predicts yı: from z. It turns out that Eg is just a way to bypass computing I [ykk,
A.9 THE MUTUAL INFORMATION MATRIX 1(21, Uk) AND THE IMPORTANCE MATRIX Rae
In Fig. 23 we compare our mutual information matrix I(2 yk) with the counterpart in (Ridgeway & Mozer 2018) and the importance matrix R& in (Eastwood & Williams) 2018). It is clear that all matrices can capture disentangled representations (those highlighted in red) well since their corresponding values are high compared to other values in the same column. However, the matrix [(2;, yk)) in (Ridgeway & Mozer, 2018) usually overestimates noisy representations since it uses Eg(:; w) [=i] instead of g(>[)²). The matrix R2 in (Eastwood & Williams 2018) sometimes assign very high absolute values for noisy representations since the regressor's s werghts are unbounded. These flaws make the metries in (Ridgeway & Mozer (018)) and in (Eastwood & Williams, 2018) inaccurate and unstable, especially "modularity" and "disentanglement" since they require normalization over rows.
A. COMPUTING METRICS FOR INFORMATIVENESS, SEPARABILITY AND INTERPRETABILITY
The metrics for informativeness separability and interpretability in Section. 3 requires computing (z,)) H (2) H24,), (z), and H(,, (k). ). We can compute these entropies via quantization or sampling. Quantization is only applicable when 2; is a scalar. If 2; is high high-ddmensional vector, we need to use sampling. Below, we describe how to compute (2) via sampling and H (2,) via quantization. Other cases can be derived similarly.
23

---

Published as a conference paper at ICLR 2020
610.6368
8.0856
[1.8789
8.8055
[[0.639
0056
9.37
0.8857
0.8053]
[I 6.0942
DX:407
0.6351
0.0189]
[0. 9801
0.9892 6.
0.0023 0.
[8.2885
8.0377
3,8
6.1497
6,8233
6-
42,6492
8.
В.
[9.
0.0991 8.
0.
0.0003]
0.1224
0.0946
8.838
0.8252
2895
6.
3,9004
8.
B.
0.0002
0.0091
0.0002
0.0001]
10.108
0.0734
9.0753
0.0591
0.154
1.353
60.784
0.0001 0.0003
0,0712 0.1421
0.0419
53.6852
1.3094
[0. 0918
0.0412 B, 0377 [22222]]
0, 02533
[0.072
8. 0357 0.0315
2. 7242
020111
0.0058
0,5509
8.9995
0.054
10.
0881 8.
6094 8.8601
[0.1011
0.0537 9.1408 6.3 0.0164
52.20
0.
[0. 2654 0.0612
B. 8484
0.8148 0.8124]
28.2835
1. 0858
9.0649
8.8197 0,019
L.58229
0.4147
1026
0.0272
[8.8776 0.0162
0.0375 0.0578 [2.657]
8.6328 0.044 2.655
1.1383
[0.2371
0.52 0.0041 0.002311
[0.2991
8. 0.0614 B.1647 0.6875 0.8044]
6.5873
1.1907
1657
0.0747]]
(a) (2,, ye) w. (2(()))
(b) (2i, yk) W.O. ((2a(22)
(c) Rie
Figure 23: (a): Our mutual information matrix [(2 yk), (b): The mutual information matrix
[2,, Uk) ) in (Ridgeway & Mozer 2018). (c): The importance matrix Ra in (Eastwood & WWillias 2018). In (a) and (b), the columns corresponding to the following ground truth factors: "shape", "scale", "rotation", "x-position", "y-position". In (c), the column for "shape" is excluded because the metrics in (Eastwood & Williams 2018) do not support categorical factors. Values corresponding to disentangled representations are highlighted in red. Defective values are highlighted in green. The model is FactorVAE with TC=20.
Computing (2) via sampling
Hz)) -B,(=) [log [(>)]
- (x,,,.) [log Bop(ta) [[(=/=)]]
M
N
Σ
log
1
Σ
(13)
M
N
m=l
n=1
M
1
log
(14)
M
N
In Eq. 13 we use Monte Carlo sampling to estimate the expectations outside and inside the log function. The corresponding sample sizes are M and N. In Eq. [4 we use the assumption
q (z(m) (10)))(()) (n)))) I1 ạ (mm) (,x()) Please note that the entropy (z) computed via sampling can be negative if is continuous since we use the density function g(=lx))
Computing (2) via quantization We can compute (2)) via quantization as follows:
H(z)=- -
(s:)) log (s)) S.ES
where S is set of all quantized bins 8; corresponding to Q(s:) is the probability mass function of 8,. To ensure consistency among different 2, as well as different models, we apply the same value range for all latent variables. In practice, we choose the range [-4, 4] since most of the latent values fall within this range. We divide this range into equal-size bins to form S.
We can compute ((s:) as follows:
( Σ (c)(e()))
n=1
We compute Q (sg(z(()) based on its definition, which is:
Q(s.[=(")) = 1 ((2s)z(()) dzy
(15)
where a, b are two ends of the bin s;.
There are two ways to compute ((s)(a(())) the first way, we simply consider the unnormalized Q' (Ng (x)) as the area of a rectangle whose width is b a and height is ((={(=:()) with ž, at the center value of the bin 8t. Then, we normalize Q' (m)(a(()) over all bins to get ((). In the second way, if g(2,[=(")) is a Gaussian distribution, we can estimate the above integral with a closed-form function (see Appdx. A. detail).
24

---

Published as a conference paper at ICLR 2020
JEMMIG (quantized)
RMIG (quantized)
1.0
2 JEMMIG (sampling)
0.0
0.5
1.0
1.5
RMIG (sampling)
(a)
(b)
Figure 24: Correlation between the sampling (#samples=100000) and quantized (value range=[-4, 4],
#bins=100) estimations of JEMMIG/RMIG. In the subplot (a). the red line y x log(bin (bin width) while in the subplot (b), the red line is = x. Blues denotes Factor VAE models and oranges denotes 8-VAE models. The dataset is dSprites.
A.11 RELATIONSHIP BETWEEN SAMPLING AND QUANTIZATION
Denote (2(2())) and Hg (z (2g (x) be the sampling and quantization estimations of an entropy 1/ (2) respectively. Because H3 (z (r) is the expectation of log q(2; (z). Hg(z(rr is the expectation of
log (z)[2r) and [z) zs ((2/))) bin width if the bin width is small enough, there exists a gap between (z,la) and Hy [z), specified as follows:
(z)2)) (2(2))))) log (bin width)
vvalue range
H2(2))))) log
#bins
H,(=.l>) - log (value range) + log (#bins)
Since Q(2i) = Eyp (n) (Q(2)aa) and g(2)) (a) [q( [g(2g(z)], we have Q(z;) = g(2;) x bin width. Thus, (z,) and (2,) also exhibit a similar gap as H. Hz(zz(z) [x) and Hg (=al<r)
Hq (2.) H2(2)) - log(bin width)
However, this gap disappears when computing the mutual information I(2,, since:
lg(z,,)) = (z;) (z)z)(r)
(H, log(bin width)) (H/(r))r) log(bin width))
HH(z;)) (z:lz)
15(2;
In fact, one can easily prove that:
lim
(4(2, = T,(21,)))
Whins #
Similar relationships between sampling and quantization also apply for H(2, yn) and 7(2,, vk). They clearly shown in Fig. 24
In summary,
Sampling entropies such as Hz(z,(æ)) or H2(2,) usually fixed but can be negative since (2,(22) or q(2;) can be > 1. However, these entropies can still be used for ranking though it not easy to interpret them.
Quantized entropies such Hg(2(()) or Hy(2) be positive if the bin width is small enough (or #bins is large enough). The growth rate is - (bi width) (or log(#tin)) Because limz-+to log x +oo, (=(2)))) and H₃(=x) cannot be upper-bounded.
25

---

Published as a conference paper at ICLR 2020
The mutual information 22) is consistent via either quantization sampling. Unlike the entropies, [2,, z) is well-bounded even when 2; is continuous, thus, suitable to be used in metric. However, when #bins is small, the approximation Q(2i) g(2;) x bin width does not hold and quantization estimation can be inaccurate.
A.12 NORMALIZING JEMMIG
Recall that the formula of the unnormalized (EMMIG(()) is ( If estimate H (z y1k) via quantization, the value of the unnormalized JEMMIG will vary according to the bin width (or value range and #bins) (as shown in Fig. [ [7 (left)). However, we can still rank models by foreing them using the same bin width (or the same value range and #bins). To avoid setting these hyper-papne parameters, we can estimate H (2+ .V) via sampling, In this case, the value of the unnormalized (FMMIG(J1)) only depends on q(2; (y) which fixed after learning. Ranking disentanglement models using the unnormalized JEMMIG(J:) is somewhat similar to ranking generative models using the log-likelihood.
( I(2 . yr) + I(2jo, [
Using the unnormalized JEMMIG(J))) causes interpretation difficulty. We could normalize JEMMIG( as follows:
Hz(z)) + H(yk) 21 yk)) + (Z)).8k) Hg(u) H(y)
(16)
where (2(24) quantization estimation of H H24,, hence, greater than 0; Hq(u) is an entropy that bounds Hg(z)) but does not depend on {(2a (z). Intuitively, u should be uniform. The main problem how find an effective value range [a, b) of 2; that satisfies conditions: i) most of the mass of 2; falls within that range, and ii) (u) is the bound of (2;)) if before solving this question, we try to answer a similar yet easier question: "Given Gaussian random variable ~ N(u,o,, what is the value range of uniform random variable such that H (u)) 2 H(z)?". Assume u E [a, b). the entropy of u is Hu) = log(b - a) while the entropy is H 0.5 log(21m (2gT2))))) We have:
bj. However,
H(z)= S H(u)
+0.5 log (2Tee²)) S log(b - a)
2 S b
Thus, to ensure (u) to be an upper bound of (z),, we should choose the value range of u to be at least 2TTe. If a 1, this range is about 4.17 If we also want [a, b) to capture most of the mass of z, a should be μ & /2ire and b should be + g /
Come back the main problem, since q(2,) (w) [((z][r)] and q(2, (a)) is usually Gaussian distribution (us, ox), choose b as follows:
(1)
(N)
= min
P4
2ne ....... 14
(N)
oi 2Te
and
2
2
(N)
b = max
f4
(1)
+
2TC. 2TT.....
(N)
+
2
2
2
One may wonder that different methods can choose different value ranges (a, bj to normalize JEMMIG so how to ensure fair comparison among them using the normalized JEMMIG. A simple solution is using the same value range [a, b) for different models. In this case, 5 a should be large enough to cover various distributions. We write Eq. 16 as follows:
26

---

Published as a conference paper at ICLR 2020
0.6
0.6
0.5
0.5
MIG (Chen ak.]
0.4
MIG (Locatallo et.
0.4
1200
hgrad2
0.3
16:00
0.2
0.1
12.000
0.2
Figure 25: Left: Correlation between our RMIG (#bins=100) and the original MIG (Chen et al. 2018 (#samples=10000), Right: Correlation between our RMIG (#bins=100) and the implementation of MIG in (Locatello et al. 2019) (#bins=100). Experiments are conducted on the dSprites dataset.
Hg(z)) + Hyy)) (1 2.. yk.)
1z BR)
(u(u) + (yn)
(z,),) log((aa (value range) + (obgis + (1s)) (2(- yk.) [2500,
(1) - log( (value range) + log(#bins) + HH
H (x,)) log(value range) + log (binn) + H (yk) 21(2 yk) + (2)0, 9k)
(17)
log(#bins) + H (3x.)
Since the fraction in Eq. 17 is smaller than 1, increasing #bins will increase this fraction but still ensure that it is smaller than 1. This means the normalized JEMMIG is always in [0, 1] despite #bins.
A.13 COMPARING RMIG WITH OTHER MIG IMPLEMENTATIONS
RMIG has several advantages compared to the original MIG (Chen et 2018) which we refer as MIG1: i) RMIG works on real datasets, MIG1 does not; ii) RMIG supports continuous factors, MIG1 does not. On toy datasets such as dSprites, RMIG produces almost the same results as MIG1 (Fig. 25 (left)). We argue that the small differences between RMIG and MIGI scores in some models are caused by either quantization error of RMIG (when #bins=100) or the sampling error MIG1 (when 15000les=10000).
Locatello et. (Locatello et al. 2019) provided an implementation of MIG which we refer as MIG2. MIG2 is theoretically incorrect in two points: only uses the mean of the distribution q(2,) (=)) instead of the whole distribution (²)(a=()), and ii) the bin range and width varies for different 2;- The performance of MIG2 is, thus, unstable. We can easily see this problem by comparing the right plot with the left plot in Fig. 25 MIG2 usually overestimates the true MIGI when evaluating 8- models with large 3 (e.g. 3 E (20,30, 50)). We guess the reason is that in these models, ((z(=(())) usually has high variance, hence, using the mean of (m)) like MIG2 leads to the wrong estimation of [(2, y1).
A.14 DEFINITE INTEGRAL OF A GAUSSIAN DENSITY FUNCTION
Assume that we have a Gaussian distribution (n, a). The definite integral of its density function within the range (a, b) denoted G(a, b) can be computed as follows:
(x - 11)2)
G(a,b) b)
exp
dx
0
202
1
a
erf
erf
2
2
ov2
27

---

Published as a conference paper at ICLR 2020
Although erf(-) does not have analytical form, compute its values with high precision using polynomial approximation. For example, the following approximation provides maximum error of 5 x 10 Def 2019):
erf(a)
(1 a1 a22² ن.و + a1z444 x
where 0.278393, 0.230389, 43 = 0.000972, a4 0.078108.
A.15 REPRESENTATIONS LEARNED BY FACTORVAE
We empirically observed that FactorVAE learns the same set of disentangled representations across different runs with varying numbers of latent variables (see Appdx. Appdx.(A.18). This behavior is akin to that of deterministic PCA which uncovers a fixed set of linearly independent factor 100 (or principal components). Standard VAE is theoretically similar to probabilistic PCA (pPCA) (Tipping & Bishop 1999) as both assume the same generative process p(x, =) po(z)z)p(z). Unlike deterministic PCA, PPCA learns invarinnnnvvriant family of factors instead of identifiable set of factors. However, in particular pPCA model, the relative orthogonality among factors is still preserved. This means that the factors learned by different pPCA models are statistically equivalent. We hypothesize that by enforcing independence among latent variables, FactorVAF. can also learn statistically equivalent factors (or ((2; (x)) which correspond to visually similar results. We provide proof sketch for the hypothesis in Appdx. A.16 We note that Rolinek et. al. (Rolinek et al. 2018) also discovered the same phenomenon 3-VAE.
A.16 WHY FACTORVAE CAN LEARN CONSISTENT REPRESENTATIONSS"
Inspired by the variational information bottleneck theory (Alemi al. 2016), we rewrite the standard VAE objective in an equivalent form as follows:
min [xx,2) 2) s.t. Rec(z) s B <=|(²)
(18)
where Rec(x) denotes the reconstruction loss over x and 3 is scalar.
In the case of VactorVAF, since all latent representations are independent, into Σ, T(xx,z) Thus, we argue that Factor optimizes the following information bottleneck objective:
decompose T(r 2)
min Σ [I,, 2:) s.t. Rec(x) < ((2()))
(19)
We assume that Rec(a) represents a fixed condition on all g: (=(r))). Because /(r,, 2x) convex function of (2))2z) (see Appdx. A.17). minimizing Eq. leads to unique solutions for all g(2(2))
(Note that we do not count permutation invariance among 2 here).
To make Rec(x) fixed condition on all (2(22), we can further optimize p(x[2) with sampled from a fixed distribution like (000 I). This suggests that we can add a GAN objective to the original FactorVAE objective achieve more consistent representations.
A.17 (x,=) IS A CONVEX FUNCTION OF p(=|x)
Let us first start with the definition of a convex function and some of its known properties.
Definition 3. Let X be set in the real vector space RD and f X - R be function that output scalar. is convex Va 22 E X and vλ E [o, 1), we have:
f(Az1 + A)æ²) S Af(z1) + +1 (x)
Proposition 4. twice differentiable function f is convex derivative is non-negative there.
interval if and only its second
Whenn we mention factors in this context, they are not really factors of variation. They refer to the columns of the projection matrix W in case of PCA and the component encoding functions g(2; (r) in case of deep generative models.
28

---

Published as a conference paper at ICLR 2020
Proposition 5 (Jensen's is inequality). Let 21,..... En be real numbers and let a1...., On be positive weights on Ti,.. Zn such that Ex a, 1. If convex function on the domain of Iı w. Fyg then
Σ 0,
S Σ axf(x,)
Equality holds if and only if all na are equal or a linear function.
Proposition 6 (Log-sum inequality). Let a1, and bi,... ba be non-negative numbers. Denote
-i ag and 2im1 by. We have:
Σ
O. log
ai 2 a log
b;
with equality if and only if 9; equal for all i.
Armed with the definition and propositions, we can now prove that ((x z) is a convex function of p(=|z)) Let pi (z(xr) and (=(2(x) be two distributions and let p. (2(xx) = App (=|æ:) + (1 (=lar=
with À [0, 1]. p. (2|xz ) a valid distribution since p. (r) > Ve and Jg S, p. (2)x))(()) dz dz
1. In addition, we have:
J (.(ell))p(o) dr
p.(z)
/ (Ap((e))) + +11 ()p2(=/z)) p(z) de S (=(x)))((z) dx + (1 A) / pa(s|x)p(x) dr
Apı(=) - ()p2(z)
We write T(xx = A7, 2) + A)T2(1, =) follows:
(1(2(z)
1(x,2) =\ p(x)
P1 (2(2x))) log
dz dx+
p1(2)
+ +11 f p(x) p2 (2(2)2)) log p2(2(x) p2(z)
dz dr
p(x) /. ()p((=/=)) log Xpi + (1 -))22(z(x) log (1- 1)p2(2lx)
(21()
da dx
2 p(z) (=(x))))og px(2(z) dz dx (2)
(20)
=I, 2)
where the inequality in Eq. 20 is the log-sum inequality. This completes the proof.
A.18 EXPERIMENTS TO SHOW THAT FACTORVAR LEARNS CONSISTENT REPRESENTATIONS
We first trained several FactorVAE models with 3 latent variables on the CclebA dataset. After training, for each model, we performed 2D interpolation on every pair of latent variables 2;, 2j (i < j) and decoded the interpolated latent representations back to images for visualization. We found that the learned representations from these models share visually similar patterns, which is illustrated in Fig. 26 It is apparent that all images in Fig. 26 are derived from single one (e.g. can choose the first image as a reference) by switching the rows and columns and/or flipping the whole image verticall arttially/horizontallyy The reason why switching happens is that all latent variables of Factor VAE are permutation invariant. Flipping happens due to the symmetry q(2,) which forced to be similar p(=;) (0, T).
29

---

Published as a conference paper at ICLR 2020
B
(a) TC=50
(b) TC=10
Figure 26: Random traversal on the latent space of FactorVAF. We can easily see the visual resem- blance among image regions corresponding the same number.
(a) TC=10, 7_dim=65
TC=50, 7_dim=65
(c) TC=50, 7_dim=100
(d) TC=50, 7_dim=200
Figure 27: Top 10 representations sorted by the variance of the distribution of (2., [2:] over all (()).
We then repeated the above experiment on FactorVAE models containing 65, 100. 200 latent variables, but replacing 2D interpolation pairs of latent variables with conditional ID interpolation individual latent variables to account for large numbers of combinations. We sorted the latent variables 2; of each model according to the variance of the distribution of E. (=t [ over all data
samples (n))) ~ (D( in descending order. Fig. 27 shows results for the top 10 latent variables (of each model). We can that some factors of variation are consistently learned by these models, for example, those that represent changes in color of the image background. Because these factors usually appear on top, we hypothesize that the learned factors should follow some fixed order. However, many pronounced factors do not appear at the top. suggesting that the sorting criterion is inadequate. We then used the informativeness metric defined in Sec. 4.1 to sort the latent variables. Now the "visual consistency" and "ordering consistency" patterns emerge, (see Fig. 28 We also observed that the number of learned factors relatively fixed (around 38-43) for all models despite that the number of latent variables varies significantly from 65 to 200.
200R
100000
1AAAAA
UDDDD 10000
(a) TC=10, x_dim=65
(b) TC=50, -__imm=65
(c)
TC=50, z_dim=100
(d) TC=50, 2_dim=200
Figure 28: Top 10 representations sorted by informativeness scores. We can clearly see the consistency of representations across different runs.
30


Reviews:
[{"review_id": "ryevsg7M5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper presents a new set of metrics for evaluating disentangled representations in both supervised and unsupervised settings. Disentangled representations are evaluated along three dimensions: informativeness, separability, and interpretability. While previous work offers metrics for similar dimensions (e.g., (Eastwood & Williams, 2018)), the paper suggests that the metrics of the submission are superior to (Eastwood & Williams, 2018), based on a comparison between FactorVAE and Beta-VAE.\n\nStrengths:\n\n(1) The metrics of the paper are well motivated from an information-theoretic standpoint. While, in some cases, the metrics themselves are straight applications of information theory (e.g., informativeness metric == mutual information), the authors came up with new metrics when existing information-theoretic definitions could be fooled by, e.g., introducing noisy and independent latent representations z.\n\n(2) Experimental results show that these metrics are better than previous metrics at discriminating between FactorVAE and Beta-VAE (in an experimental setting where the former is clearly superior to the latter).\n\n(3) As opposed to much of prior work, these metrics do not require any training, which can be considered a plus as they do not require adaptation for each (sub)domain.\n\n(4) The paper is overall well written and clear. I very much enjoyed reading it.\n\nWeaknesses:\n\n(1) My main concern with the paper, which makes me vote for “weak accept” instead of “accept”: The metrics of the paper are compared against previous metrics (Eastwood & Williams, 2018) on *only two* types of disentangled representations, namely FactorVAE and Beta-VAE, and in one experimental condition. (There is plenty of other experimental material in the appendix which also introduces AAE, but none of it seems to compare against previous *metrics*). I think comparing metrics on only two systems is somewhat problematic, and we don’t know how general the results are. I would have preferred to see FactorVAE and Beta-VAE evaluated with fewer hyperparameter choices to allow introducing more variational approaches. Could you (the authors) at least evaluate AAE against Eastwood & Williams too? This concern is not just mere quibbling, as the authors have shown (e.g., in Section 3.2) that specific edge cases can fool naïve metrics (e.g., by introducing noisy and independent latent variables), and there may be other edge cases that the authors have not considered. I think evaluating new metrics against previous work (i.e., metrics) on only two underlying systems is not very convincing. I acknowledge that the paper offers *lots* of experiments – the full pdf is 30 pages (!) – but I think some of the existing ones could go to leave space for evaluations using more underlying VAE/baselines/edge-case systems.\n\n(2) The paper presents six metrics (MI, MISJED, WSEPIN, WINDIN, RMIG, JEMMIG) along three dimensions. While each individual metric makes sense, I feel the paper lacks a discussion section that ties these pieces together and suggests a way of using these metrics conjointly across the three dimensions towards building better-disentangled representations (the paper has a “discussion” section, but it is very short and actually more of a conclusion). The more metrics we have, the more chances each underlying VAE model can “win” on one of the metrics, which is not particularly enlightening. \n\n(3) The paper is not self-contained, and some parts are almost impossible to understand without familiarity with (Kim & Mnih, 2018) and (Higgins et al., 2017a). It uses technical terms of these papers without explanations (e.g., TC is not even spelled out it seems). Hyperparameters of these papers (e.g., beta, gamma) are used without explanation.\n\n(4) There is no related work section. Such a section could, e.g., make what is borrowed from (Kim & Mnih, 2018; Higgins et al., 2017a) more understandable.\n\nOverall, I think it is a nice paper that makes significant contributions to the problem of building better disentangles representations thanks to better evaluation metrics. The empirical support for some of the claims (e.g., superiority to (Eastwood & Williams, 2018)) is a bit weak, but other strengths mentioned above largely make up for that.\n\nMinor comments:\n\nThe definitions of SEPIN@k and INDIN@k don’t seem quite right. The summation iterates over z_0, …, z_k-1, leaving off z_k, …, z_L-1, but the latter variables might contain some z’s with lowest mutual information with x. The way the ‘sorted’ function is written only has the effect of reordering z_0, …, z_{k-1} among themselves, never considering any of the z_k and above whatever their MI’s are, which is probably not what the authors meant. Perhaps ‘sorted’ is intended to both sort and rename z’s, but if z’s are indeed renamed I think this should be indicated mathematically otherwise the equation is wrong (e.g., (z’_1, …, z’_{L-1}) = sorted_by_MI(z_1, …, z_{L_1})).\n\nTypos in WSEPIN and WINDIN definitions: “0 = 1” -> “i = 1”?\n\nFigure 1: “Image”? The paper is written in general terms and doesn’t seem to assume otherwise x is an image representation (or does it?).\n\nSection 3.4: “Table. 1” -> “Table 1”\n\nTable 1 vs. title of Section 3.1: The author’s informativeness metric doesn’t have a name, but the section title contains “informativeness,” which could be confusing vs. “informativeness” in Table 1 which is a completely different metric.\n\nPage 8, interpretability: “TC=10”. Do you mean “TC loss” here? I presume 10 is the value of a hyperparameter of the FactorVAE paper (gamma?), and not the actual value of the TC term. Same question for Figure 9 and later figures of the appendix.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Syg_EIAutH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper defines precise semantics of disentanglement representations and presents evaluation metrics to evaluate such representations. The authors provides information-theoretic characterization disentangled representations along three dimensions: informativeness, separability, and interpretability; and propose metrics to measure them. The authors argue that:- Informativeness can be defined as the mutual information between a particular representation (or a group of representations) w.r.t the data---I(x,z_i) - Separability between two representations can be achieved if they do not share any common information about the data---I(x,z_i,z_j) = 0 - Interpretability with respect to a concept is achieved if a representation contains information about the concept---I(z_i, y_k) = H(z_i) = H(y_k).The authors define a disentangled representation as a representation that is fully separable and fully interpretable and propose a suite of metrics to evaluate disentangled representations.Finally, the authors evaluate several representation learning methods (FactorVAE, betaVAE, AAE) using these metrics on toy and real datasets.\n\nI think this paper addresses an important problem of quantifying and measuring disentangled representations. The proposed metrics are reasonably sound and the authors provide an extensive set of experiments to show how to use them in practice. \n\nI have one comment regarding the sheer number of metrics that are presented in the paper and their practical usage. How do the authors see them being used in the future to compare models? Is one of the main arguments of the paper to encourage other people to use all of these metrics? Are all of them needed, given some of them seem to correlate with others? I think it would be helpful to highlight a few metrics or aggregates of them to inform future research in this area.\n\nAlso, for the paper to be more self-contained, I think the authors should include a short discussion about models that they compare in the experiments.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Bkg0XRg6KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a mathematically-grounded set of axes to describe the effectiveness of latent representations: informativeness (as the mutual information between input and z), separability (I(x, zi, zj) = 0, i!=j), and interpretability. They accompany these with metrics to evaluate representations across those criteria.\n\nThey use this to evaluate B-VAE, FactorVAE and AAE. Tl;dr B-VAE are the most separable, FactorVAE are the most interpretable.\n\nI think the paper serves as a great primer for people who are not familiar with disentangled representations, and also proposes a necessary vocabulary for understanding the trade-offs of different representation disentangling methods.\n\nThe paper is too long, you could cut the final paragraph of S2.1 without losing anything (that's half a page already). You can (and should) edit this down. I don't think this paper should be longer than 8 pages.\n\nP.s. did you really cite \"Error Function\" to the wikipedia page for Error function?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript proposes and evaluates new metrics for measuring the quality of disentangled representations for both supervised and unsupervised settings. The contributions include conceptual definitions and empirical evaluation.

In reviews and discussion, the reviewers and AC note missing or inadequate empirical evaluation with many available methods for learning disentangled representations. On the writing, reviewers mentioned that the conciseness of the manuscript could be improved. The reviewers also mentioned incomplete references and discussion of prior work, which should be improved.
Decision: Accept (Poster)