PROGRESSIVE LEARNING AND DISENTANGLEMENT OF HIERARCHICAL REPRESENTATIONS
Authors: Zhiyuan Li
Jaideep Vitthal Murkute
Prashnna Kumar Gyawali
Linwei Wang
Venue:  (2020)
Keywords: generative model
disentanglement
progressive learning
VAE

Abstract: Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE). However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation is compromised. Motivated by the concept of “starting small”, we present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions. The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new  representations at different levels of abstraction. We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works on two benchmark datasets using three disentanglement metrics, including a new metric we proposed to complement the previously-presented metric of mutual information gap. We further present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations. By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the first attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations.

Published as a conference paper at ICLR 2020
PROGRESSIVE LEARNING AND DISENTANGLEMENT OF HIERARCHICAL REPRESENTATIONS
Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali & Linwei Wang
Golisano College of Computing and Information Sciences Rochester Institute of Technology
Rochester, NY 14623, USA
{z17904, Jvm652 pkg21 1kkg218,,12 82, www.ei.aanggeriei.....
ABSTRACT
Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE). However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation compromised. Motivated by the concept of "starting small", we present strategy to progressively learn independent hi- erarchical representations from high- to low-levels of abstractions. The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new representations at different levels abstraction. We quantitatively demonstrate the ability of the presented model improve disentanglement in comparison to existing works on two benchmark data using three disentanglement metrics, including a new metric we proposed complement the wwvioussl--preenntes metric of mutual information gap. We fur- ther present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations. By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the first attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representation [
INTRODUCTION
Variational auto-encoder (VAE), popular deep generative model (DGM), has shown great promise in learning interpretable and semantically meaningful representations of data (Higgins et al. (2017) Chen et al. 2018); Kim & Mnih (2018); Gyaaalli et al. (2019). However, VAE has not been able to fully utilize the depth of neural networks like its supervised counterparts, for which fundamental cause lies in the inherent conflict between the bottom-up inference and top-down generation process (Zhao et al. (2017); Li et al. (2016)): while the bottom-up abstraction is able to extract high-level representations helpful for discriminative tasks, the goal of generation requires the preservation of all generative factors that are likely at different abstraction levels. This issue was addressed in recent works by allowing VAEs to generate from details added at different depths of the network, using either memory modules between top-down generation layers (Li et al. (2016)), or hierarchical latent representations extracted at different depths via a variational ladder autocncoder (VLAE, Zhao et al. (2017))
However, it is difficult to learn to extract and disentangle all generative factors at once, especially at different abstraction levels. Inspired by human cognition system, Elman (1993) suggested the im- portance of "starting small" two aspects of the learning process of neural networks: incremental input in which network is trained with data and tasks of increasing complexity, and incremental memory in which the network capacity undergoes developmental changes given fixed external data and tasks - both pointing to incremental learning strategy for simplifying complex final task. Indeed, the former concept of incremental input has underpinned the success of curriculum learning [engii et al. (2015).. In the context of DGMs, various stacked versions of generative adversarial networks (GANs) have been proposed to decompose the final task of high-resolution image gener- ation into progressive sub-tasks of generating small large images (Denton et al. (2015); Zhang
Source code available at //g www..cco/...h.c.ooooccc.oo
991 /proviAN

---

Published as a conference paper at ICLR 2020
et al. (2018) The latter aspect of "starting small" with incremental growth of network capacity is less explored, although recent works have demonstrated the advantage of progressively growing the depth of GANs for generating high-resolution images (Karras et al. (2018); :); Wang et al. (0118)
These works, so far, have focused on progressive learning as a strategy to improve image generation.
We are motivated investigate the possibillity use progressive learning strategies to improve learn- ing and disentangling of hierarchical representations. At a high level, the idea of progressively or sequentially learning latent representations has been previously considered in VAE. In Gregor et al. (2013). the network learned to sequentially refine generated images through recurrent networks, In Lezama (2019), a teacher-student training strategy was used to progressively increase the number of latent dimensions in VAE to improve the generation of images while preserving the disentangling ability of the teacher model. However, these works primarily focus on progressively growing the capacity of VAE to generate, rather than to extract and disentangle hierarchical representations.
In comparison, in this work, we focus on 1) progressively growing the capacity of the network to extract hierarchical representations, and 2) these hierarchical representations are extracted and used in generation from different abstraction levels. We present a simple progressive training strategy that grows the hierarchical latent representations from different depths of the inference and generation model, learning from high- to low-levels of abstractions as the capacity of the model architecture grows. Because it can be viewed as progressive strategy to train the VLAE presented in Zhao et al. (2017), we term the presented model pro- VLAE. We quantitatively demonstrate the ability of VLAE to improve disentanglement on two benchmark data sets using three disentanglement metrics, including new metric we proposed to complement the metric of mutual information gap (MIG) previously presented in Chen et al. (2018). These quantitative studies include comprehensive comparisons to 3-VAE (Higgins et al. (2017). VLAE (Zhao et al. (2017)), and the teacher-student strategy as presented in (Lezama (2019)) different values of the hyperparameter B. We further present both qualitative and quantitative evidence that Pro-VLAE is able to first learn the most abstract representations and then progressively disentangle existing factors or learn new factors at lower levels of abstraction, improving disentangling of hierarhical representations in the process.
RELATED WORKS
A hierarchy of feature maps can be naturally formed in stacked discriminative models (Zeiler & Fergus| 2014)). Similarly, in DGM, many works have proposed stacked- VAFS as a common way to learn hicrarchy of latent variables and thereby improve image generation (Sønderby et al. (2016);
Bachman (016)) Kingma et al. (2016)). However, this stacked hierarchy is not only difficult to train as the depths increases (Sønderby al. (0166); Bachmaan (2016)) but also has an unclear benefit for learning either hierarchical or disentangled representations: as shown in Zhao et al. (2017), when fully optimized, it is equivalent to a model with a single layer of latent variables. Alternatively, in- stead of a hierarchy of latent variables, independent hierarchical representations at different abstrac- tion levels can be extracted and used in generation from different depths of the network (Rezende) et al. (2014); Zhao et al. (2017)). A similar idea was presented in Li et al. (2016) to generate lost details from memory and attention modules at different depths of the top-down generation process. The presented work aligns with existing works (Rezende al. (2014): /Zho et al. (2017)) in learning independent hierarchical representation from different levels of abstraction, and we look to facilitate this learning by progressively learning the representations from high- low-levels.
Progressive learning has been successful for high-quality image generation, mostly in the setting of GANs. Following the seminar work of Elman (1993), these progressive strategies can be loosely grouped into two categories. Mostly, in line with incremental imput, several works have proposed to divide the final task of image generation into progressive tasks of generating low-resolution to high-resolution images with multi-scale supervision (Denton et al.] (2015): Zhang et al.] (2018). Alternatively, in line with incremental memory, a small number of works have demonstrated the ability to simply grow the architecture of GANs from a shallow network with limited capacity for generating low-resolution images, to a deep network capable of generating super-resolution images KKrrra et al. (2018); Wang et al. (2018)) This approach was also shown to be time-efficient since the early-stage small networks require less time to converge comparing to training a full network from the beginning. This latter group of works provided compelling evidence for the benefit of pro- gressively growing the capacity of a network to generate images, although its extension for growing the capacity of a network to learn hierarchical representations has not been explored.
2

---

Published as a conference paper at ICLR 2020
Limited work has considered incremental learning of representations in VAE. In Gregor al. 2015)) recurrent networks with attention mechanisms were used to sequentially refines the details in gen- erated images. It however focused on the generation performance of VAE without considering the learned representations. In Lezama (2019), a teacher-student strategy was used to progressively grow the dimension of the latent representations in VAF. Its fundamental motivation was that, given a teacher model that has learned to effectively disentangle major factors of variations, progressively learning additional nuisance variables will improve generation without compromising the disentan- gling ability of the teacher the latter accomplished via a wwwlyppropoed Jacobian supervision. The capacity of this model grow, thus, is by design limited the extraction of nuisance variables. In comparison, we are interested in more significant growth of the VAE capacity to progressively learn and improve disentangling of important factors of variations which, as we will later demon- strate, is not what the model in Lezama (2019) is intended for. In addition, neither of these works considered learning different levels of abstractions at different depths of the network, and the pre- sented pro- VLAE provides simpler training strategy to achieve progressive representation learning.
Learning disentangled representation is primary motivation of our work, and an important topic in VAE. Existing works mainly tackle this by promoting the independence among the learned latent factors in VAE (Higgins et al. (2017); Kim & Mnih] (2018): Chen et al.] (2018)). The presented progressive learning strategy provides novel approach to improve disentangling that is different to these existing methods and a possibility to augment them in the future.
3 METHODS
3.1 MODEL: VAE WITH HIERARCHICAL REPRESENTATIONS
We assume generative model p(æ; =) p(ar| (()) for observed and its latent variable z. To learn hierarchical representations of æc we decompose z into {2), 22, with z1(1 = 1, 2, 3, ..., L) from different abstraction levels that are loosely guided by the depth of neural net- work as in Zhao et al. (217)) We define the hierarchical generative model pa as:
p(a, 2) p(zl 21, 22, ZL) II P(Z)
Note that there no hierarchical dependence among the latent variables in common hicrarchical latent variable models. Rather, similar to that in Rezende et al.) (2014) and Zhao et al. (2017), z 's are independent and each represents generative factors at an abstraction level not captured in other levels. We then define an inference model qø to approximate the posterior as:
q(z1; 22, 21| = II ((=i(h/(()) 1=1
where (x()) represents a particular level of bottom-up abstraction of x. We parameterize pe and 4s structure and, as in /Zha et al. (2017), we approximate the abstraction level with the network depth. The full model is illustrated in Fig. [[], with a final goal to maximize a modified evidence lower bound (ELBO) of the marginal likelihood of data a::
with an
log)(a)) 2 L = Eyyt|j) (()gp((())) -3K [g((<()
(3)
where L denotes the Kullback-Leible divergence, prior p(x) is set to isotropic Gaussian (0, I) according standard practice, and / is hyperparameter introduced in Higgins al. (2017) to promote disentangling, defaulting to the standrd ELBO objective when ,
3.2 PROGRESSIVE LEARNING OF HIERARCHICAL REPRESENTATION
We present a progressive learning strategy, as illustrated in Fig. I to achieve the final goal in equa- tion 3 by learning the latent variables ž progressively from the highest (1 = L) to the lowest 1 level of abstractions. We start by learning the most abstraction representations at layer L as show in Fig. [a). In this case, model degenerates to vanilla VAE with latent variables æ at the deepest layer. We keep the dimension of 21. small to start small in terms of the capacity to learn latent representations, where we define the inference model at progressive step = 0 as:
21. (h)) oz h4 = (hi-1); for I = 1,2, - I and ho ar,
(4)
3

---

Published as a conference paper at ICLR 2020
hz
BL
26
hz
SL
ML-1
81
B1
A1
Praguos Toring
Figure 1: Progressive learning hierarchical representations. White blocks and solid lines are VAE models at the current progression. a is fade-in coefficient for blending in the new network com- ponent. Gray circles and dash line represents (optional) constraining of the future latent variables.
and the generative model as:
9L fZ()) 91 (( x= fo
(5)
where ff μLL and σr are parts of the encoder architecture, fi are parts of the decoder architecture, and D the distribution of æ parametrized by fú (go). which can be either Bernoulli or Gaus- sian depending on the data. Next, as shown in Fig. [ we progressively grow the model to learn 2L-1.... 22, 21 from high low abstraction levels. At each progressive step s move down one abstraction level, and grow the inference model by introducing new latent code:
...,
z ((uu())) oi 1= -
Simultaneously, we grow the decoder such that it can generate with the new latent code as:
gi (mm(()): g(+1)), I - L-s.
where my includes transposed convolution layers outputting feature map in the same shape g1+1, and [ denotes a concatenation operation. The training objective at progressive step is then:
Cpr Emltz 1. (m) [logp(a z1,, 21, 21, 21, 21.-s) 3 Σ KL(g(z)=))lp(x))
By replacing the full objective in equation (3) with sequence of the objectives in equation (8) as the training progresses, we incrementally learn to extract and generate with hicrarchical latent representations from high to low levels of abstractions. Once trained, the full model as shown in Fig. c) will used for inference and generation, and progressive processes are no loner needed.
3.3 IMPLEMENTATION STRATEGIES
Two important strategies are utilized to implement the proposed progressive representation learning- First, directly adding new components to a trained network often introduce sudden shock to the gradient: VAEs, this often leads the explosion of the variance in the latent distributions. avoid this shock, we adopt the popular method of "fade-in" (Karras et al. (2018) to smoothly blend the new and existing network components. In specific, we introduce a "fade-in" cocfficient a to equations (6) and (7) when growing new components in the encoder and the decoder:
21 (ahh), (ahi)) gi = f ([ami zz);; gt+1));
where a increases from 0 to I within certain number of iterations (5000 in our experiments) since addition of the new network components μ1,or, and mi.
Second, we further stabilize the training by weakly constraining the distribution of z/'s before they are added to the network. This can be achieved by a applying KI. penalty, modulated by a small coefficient y to all latent variables that have not been used in the generation at progressive step s:
L-s-1
Epre-trained
Σ
{I(((>(2))lb<<)))
(10)
1-1
4

---

Published as a conference paper at ICLR 2020
where y is set to 0.5 in our experiments. The final training objective at step then becomes:
C Lpro + Lepre-trained
(11)
Note that the latent variables at the hierarchy lower than L - are neither meaningfully inferred nor used in generation at progressive step 8, and Crrentrained merely intends to regularize the distribution of these latent variables before they are added to the network. In the experiments below, we use both "fade-in" and Lere- trained when implementing the progressive training strategy.
3.4 DISENTANGLEMENT METRIC
Various quantitative metrics for measuring disentanglement have been proposed (Higgins al. 2017]: Kim & Mnih (2018); Chen et al. (2018)) For instance, the recently proposed MIG metrics CChen al. (2018)) measures the gap of mutual information between the top two latent dimensions that have the highest mutual information with a given generative factor. A low MIG score, therefore, suggests an undesired outcome that the same factor is split into multiple dimensions. However, if different generative factors are entangled into the same latent dimension, the MIG score will not be affected.
Therefore, we propose a new disentanglement metric to supplement MIG by recognizing the entan- glement of multiple generative factors into the same latent dimension. We define MIG-sup as:
J Σ (T [nnorm (2y;
max Taorm (2,; vk)), kg(
(12)
where z is the latent variables and v the ground truth factors, k-5) argmax, Anorm (zj; 0g), is the number of meaningful latent dimensions, and noom (2j; v&)) is normalized mutual information [2,; //// (rk)) Considering MIG and MIG-sup together will provide a more complete measure of disentanglement, accounting for both the splitting of one factor into multiple dimensions and the encoding of multiple factors into the same dimension. In an ideal disentanglement, both MIG and MIG-sup should be 1, recognizing one-to-one relationship between generative factor and latent dimension. This would have a similar effect to the metric that was proposed in Eastwood & Williams (2018), although MIG-based metrics do not rely training extra classifiers regressors and are unbiased for hyperparameter settings. The factor metric (Kim & Mnih (2018) also has similar properties with MIG-sup, although MIG-sup is stricter penalizing any amount of other minor factors in the same dimension.
EXPERIMENT
We tested the presented pro- VLAE on four benchmark data sets: dSprites (Matthey et al. 2017 3DShapes (Burgess & Kim (2018), MNIST (LeCun et al. (1998). and CelebA (Liu et al. 2015 where the first two include ground-trute generative factors that allow us to carry out comprehen- sive quantitative comparisons of disentangling metrics with existing models. In the following, first quantitatively compare the disentangling ability of Pro-VLAE in comparison to three existing models using three disentanglement metrics. We then analyze pro-VLAR from the aspects of how learns progressively, its ability to disentangle, and its ability to learn abstractions at different levels.
Comparisons in quantitative disentanglement metries: For quantitative comparisons, we consid- ered the factor metric in Kim & Mnih (2018). the MIG in Chen et al. (0018). and the MIG-sup pre- sented in this work. We compared pro- pro-VLA VLAE (changing 3) with beta- beta-VAE (Higgins et al.) 2017)) VLAE (Zhao et al. (0017) as hierarchical baseline without progressive training, and the teacher- student model (Lezana (2019)) as the most related progressive VAE without hierarchical represen- tations. All models were considered at different values of /3 except the teacher-student model: the comparison of 3- VAF, VILAE, and the presented pro-VI.A VLAE thus also provides an ablation study on the effect of learning hierarchical representations and doing in progressive manner.
For fair comparisons, we strictly required all models to have the same number of latent variables and the same number of training iterations. For instance, if hierarchical model has three layers that each has three latent dimensions, a non-hierane oin-hiiaaal model will have nine latent dimensions; progressive method has three progressive steps with 15 epochs of training each, non proggreeeee method will be trained for 45 epochs. Three to five experiments were conducted for each model at each B value, and the average of the top three is used for reporting the quantitative results in Fig.
5

---

Published as a conference paper at ICLR 2020
30shapes
4
MNG UUC--MU
600
0-
FMM
PMM
P-M
YLAE
0.3
1900
20:0
2009
2523
2973
reconstruction
reconstruction
dipritte
PAM stafer-
42
Teotor reattrc
120
300
50e 120
100
Quantitative comparison of disentanglement metrics. Each point is annotated by the 3 value and averaged over top three best random seeds for the given / on the give model. Left to right: reconstruction errors vs. disentanglement metrics of factor, MIG, and MIG-sup, higher value indicating better disentanglement in each metric.
Figure 2:
50Shapes
dbprites
04-
0.4
20
29
HIG
HIO
P-VAR VLAE
J-MAE
0.5
0.0
0.1
0.2
9.3
0.5
Figure 3: MIG vs. MIG-sup following a similar presentation in Fig. 2 A better disentanglement
should have higher MIG and higher MIG-sup, locating at the top-right quadrant of the plot.
As shown, for MIG and MIG-sup, VLAE generally outperformed 3-VAE at most B values, while PLo-VLAE showed a clear margin of improvement over both methods. With the factor metric, pro- VLAE was still among the top performers, although with smaller margin and larger overlap with VLAE on 3DShapes, and with 3-VAB (3 = 10) on dSprites. The teacher-student strategy with Jacobian supervision in general had low to moderate disentangling score, especially on 3DShapes. This is consistent with the original motivation of the method for progressively learning nuisance variables after the teacher learns to disentangle effectively, rather than progressively disentangling hierarchical factors of variations as intended by pro-VLAE. Note that pro- Pro-VLAE in general per- formed better with a smaller value of (3 < 20), suggesting that progressive learning already had an effect of promoting disentangling and high value of 3 may over- verrpromote disentangling at the expense of reconstruction quality.
Fig. 3 shows MIG vs. MIG-sup scores among the tested models. As shown, results from Pro-VIAE were well separated from the other three models at the right top quadrant of the plots, obtaining simultaneously high MIG and MIG-sup scores a clear evidence for improved disentangling ability.
Fig. provides images generated by traversing each latent dimension using the best VLAE (3 8), the best VLAE (3 10), and the teacher-student model on 3DShapes data. As shown, pro- VLAE learned to disentangle the object, wall, and floor color in the deepest layer; the following hierarchy of representations then disentangled objective scale, orientation, and shape, while the lowest-level of abstractions ran out of meaningful generative factors to learn. In comparison, the VLAE distributed six generative factors over the nine latent dimensions, where color was split across

---

Published as a conference paper at ICLR 2020
GuTS
YLAE
Teacher-student model
Factor Eesto-noblic
Factor Peclor-mttcc0.007
Figure 4: Traversing each latent dimension in pro- PLAB (3 = 8), VLAE ( = 10), and teacher-
student model. The hierarchy of the latent variables noted by brackets on the side.
Generatated images randoms sample
[0xz]] Total:10.61
Generatated images from random sample
latent
[jca]
Total:12.43
Progg Progressve step
33
Z2
0.02
2
Tolnh:12
Total:12.00 09
Progrentie step
Z3
94
Z3
11:43
Z2
10.38
Z
0.02
Tetal:18.02
Total:18.19
Progrossive stop
Z
Z3
9:16
Z
Z2 8.36
Z1
0.000
Z1 0.001
Ours. NIG:0.9524, MIC 44000.9..8,..... Factor:0.9055
Teacher-student model. MIC:0.3091, MIC-sup:0 Flctor: 0.4027
Figure 5: Progressive learning of hierarchical representations. At each progression and for each 21, the row of images are generated by randomly sampling from its prior distributions while fixing the other latent variables (this is NOT traversing). The green bar at each row tracks the mutual information T(x; 21), while the total mutual information T(x; 2) is labeled on top.
the hierarchy and sometimes entangled with the object scale (in 22). The teacher-student model was much less disentangled, which will delve into further in the following section.
Information flow during progressive learning: further understand what happened during pro- gressive learning, use mutual information [ =t) as surrogate track the amount of informa- tion learned in each hierarchy of latent variables # during the progressive learning. We adopted the approach in Chen et al. 2018 to empirically estimate the mutual information stratified sampling.
Fig. 5 shows an example from 3DShapes. At progressive step O, pro-VAE was only learning the deepest latent variables in Ø3. discovering most of the generative factors including color, objective shape, and orientation entangled within ø3. At progressive step 1, interestingly, the model was able to "drag" out shape and rotation factors from 23 and disentangle them into Z2 along with new scale factor. Thus [(x; 23) decreased from 10.59 to 6.94 while [(x; =2) increased from 0.02 to 5.98 in this progression, while the total mutual information [(x; 2) increased from 10.61 to 12.84, suggesting the overall learning of more detailed information. Since 3DShapes only has 6 factors, the lowest-level representation 21 had nothing to learn in progressive step 2, and the allocation of mutual information remained nearly unchanged. Note that the sum of T(xr, 21' does not equal to [(x, 2) and Jower Σ I(x, 21) [ x) suggests the amount of information that is entangled.
In comparison, the teacher-stude student model was less effective in progressively dragging entangled representations newly added latent dimensions, suggested by the slowing changing of (ar, 2;)'s

---

Published as a conference paper at ICLR 2020
/
6
2
4
5
4
4
4
6
4
1
1
6
8
7
4
6
6
3
4
1
6 4 4 4 3
4 4 7 9
4
4
2
6
4
1
1
0
2
8
0
0
4
7
4
3
4 4
4
6
2
7
2
4
4
4
4
0
4
Deep
Shallow
Figure 6: Visualization of hierarchical features learnt for MNIST data. Each sub-figure is generated by randomly sampling from the prior distribution of ž at one abstraction level while fixing the others. The original latent code inferred from a image with digit "0". From left to right: 23 encodes the highest abstraction: digit identity; 22 encodes stroke width; and ž encodes other digit styles.
cariall
Figure 7: Visualization of hierarchical features learnt for CelebA data. Each subfigure is generated by traversing along a selected latent dimension in each row within each hierarchy of z 's. From left to right: latent variables 24 to zi progressively learn major (e.g., gender in 24 and smile in 2ạ) to minor representations (e.g. wavy-hair in z2 and eye-shadow in z1) in a disentangled manner.
during progression and the larger value of Tover- This suggests that, since the teacher-student model was motivated for progressively learning nuisance variables, the extent to which its capacity can grow for learning new representations is limited by two fundamental causes: 1) because it increases the dimension of the same latent vectors at the same depth, the growth of the network capacity is limited in comparison to pro-VLAE, and 2) the Jacobian supervision further restricts the student model to maintain the same disentangling ability of the teacher model.
Disentangling hierarchical representations: We also qualitatively examined VLo- on data with both relatively simple (MNIST) and complex (CelebA) factors of variations, all done in un- supervised training. On MNIST (Figure C). while the deepest latent representations encoded the highest-level features in terms of digit identity, the representations learned at shallower levels en- coded changes in writing styles. In Figure 7 we show the latent representation progressively learned in CelebA from the highest to lowest levels of abstractions, along with disentangling within each level demonstrated by traversing one selected dimension at a time. These dimensions are selected as examples associated with clear semantic meanings. As shown, while the deepest latent rep- resentation 24 learned to disentangle high-level features such as gender and race, the shallowest representation 21 learned to disentangle low-level features such as eye-shadow. Moreover, the num- ber of distinct representations learned decreased from deep to shallow layers. While demonstrating disentangling by traversing each individual latent dimension or by hierarchically-tearne represen- tations has been separately reported in previous works (Higgins et al. (2017): ZZhoo et al. (2017)), to our knowledge this is the first time the ability of a model to disentangle individual latent factors in a hierarchical manner has been demonstrated. This provides evidence that the presented progressive strategy of learning can improve the disentangling of first the most abstract representations followed by progressively lower levels of abstractions.
8

---

Published as a conference paper at ICLR 2020
5 CONCLUSION
In this work, we present a progressive strategy for learning and disentangling hierarchical represen- tations. Starting from simple VAE, the model first learn the most abstract representation. Next, the model learn independent representations from high- to low-levels of abstraction by progres- sively growing the capacity of the VAE deep to shallow. Experiments on several benchmark data sets demonstrated the advantages of the presented method. An immediate future work is to include stronger guidance for allocating information across the hierarchy of abstraction levels, either through external multi-scale image supervision internal cetemmttiio-thooettie regularization strategies.
REFERENCES
Philip Bachman. architecture for deep, hierarchical generative models. In Advances in Neural Information Processing Systems, pp. 4826-4834, 2016.
Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conferences on Machine Learning. 2015.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. hutp://ghhhn.coomdeeppmm////hsss 2018.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp- 2610-2620, 2018
Emily I. Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. Advances in Neural Information Processing Systems, pp. 1486-1494 2015.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. In International Conference Learning Representations, 2018.
Jeffrey I. Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71-99, 1993.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. In International Conference on Machine Learning, 2015.
Prashnna Kumar Gyawali, Zhiyuan Li, Cameron Knight, Sandesh Ghimire, B Milan Horacek, John Sapp, and Linwei Wang. Improving disentangled representation learning with the beta bernoulli process. In IEEE International Conference on Data Mining, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, 2018
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743-4751. 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEBE, 86(1 1):2278-2324, 1998.

---

Published as a conference paper at ICLR 2020
José Lezama. Overcoming the disentanglement reconstruction trade-off via jacobian supervision. In International Conference on Learning Representations, 2019.
Chongxuan Li, Jun Zhu, and Bo Zhang. Learning to generate with memory. In International Conference Machine Learning, pp. 1177-1186, 2016.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730-3738, 2015.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://githbb.c https:/ggihbb.coo//eeppnnn// http:///..ccccomm////pssssasaaae 2017.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International Conference Machine Learning. 2014.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther, Ladder variational autoencoders. Advances in neural information processing systems, pp. 3738-3746, 2016.
Yifan Wang, Federico Perazzi, Brian McWilliams, Alexander Sorkine- Hormung, Olga Sorkine- Hornung, and Christopher Schroers. A fully progressive approach to single-image super- resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 864-873, 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pp. 818 833..33.35 Springer, 2014.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan++; Realistic image synthesis with stacked generative adversarial net- works. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1947-1999,, 2018.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep gener- ative models. In International Conference Machine Learning, pp. 4091 1099, 2017.
10

---

Published as a conference paper at ICLR 2020
APPENDIX
A EXAMPLES OF PERFORMANCE OF DIFFERENT METRICS
MIG: 0.1295, MIG-sup:0.4224. factor: 0.6653
Figure 8: An example of factor being encoded in multiple dimensions. Each is traverse for one dimension (dimension order adjusted for better visualization). Notice that both diml and dim2 are encoding floor-color, both dim3 and dim4 are encoding wall-color, and both dim5 and dim6 are encoding object color. Therefore, the MIG is very low since it penalizes splitting one factor to multiple dimensions. On the other hand, the MIG-sup and factor-metric is not too bad since one dimension mainly encodes one factor, even though there are some entanglement of color-vs-shape and color-vs-scale.
MIG: 0.9524, MIG-sup:0.948B, factor: 0.9975
MIG: 0.9131, MIG-sup:0.7544 factor: 0.8313
Figure 9: An example of one dimension containing multiple factors. Each row is traverse for one dimension (dimension order adjusted for better visualization). Notice that both models achieve high and similar MIG because all 6 factors are encoded and no splitting to multiple dimensions. However, the right-hand side model has much lower MIG-sup and factor-metric than the left-hand side model. Because both scale and shape are encoded in dim5, while dim6 has no factor. Both MIG-sup and factor-metric penalize encoding multiple factors in dimension. Besides, MIG-sup is lower and drops more than factor-metric because MIG-sup is stricter in this case.
11

---

Published as a conference paper at ICLR 2020
B A CLOSER COMPARISON WITH VLAE
B.1 TWO DIMENSIONAL TRAVERSING ON MNIST DATASET
O
F
5 4 15661;
1
S
SSS
37307 6
S S
393J
0
S
F FIST
20
L
S 1
1 1135
F
2
5 S 6
5 V
3 2
3
7 777771
P V
G
3
25
9
177777 7
J 2
3
P
3
P
1
E 7
3 7
1 P
2 f
3 V
7 V P
7 P
7 4 4
1171
2
Q
S
2
Figure 10: MNIST traversing results following the same generation strategy and network hierarchy as those presented in Figure (Zhao et al. (2017). The network has layers and 2 dimensional latent code each layer. Each image is generated by traversing each of the two-dimensional latent code one layer, while randomly sampling from the other layers. From left to right: The top layer 23 encodes the digit identity and tilt; #2 encodes digit width (digits around top-left are thicker than digits around bottom--ight); and the bottom layer 21 encodes stroke width. Compared to VLAE, the representation learnt in the presented method suggests smoother traversing digits and similar results for digit width and stroke width.
B.2 INFORMATION ALLOCATION EACH LAYER
Table 1: Mutual information [(x; 2)) between data and latent codes at each 1-th depth of the network, corresponding to the qualitative results presented in Fig. 4 and Fig. 6 on 3Dshapes and MNIST data sets. Both VI.AE and the presented VLAE models have the same hierarchical architecture with layers and 3 latent dimensions for each layer. Compared to VLAE, the presented method allocates information in a more clear descending order owing to the progressive learning.
3DShapes
I(x; 23)
I(x; 22)
I(x; 21)
total [(x; 2)
VLAE
4.41
4.69
5.01
12.75
VLAE
6.94
6.07
0.00
13.02
MNIST
1(x; 23)
[(x; Ø2)
[(x; 21)
total [(x; x²)
VLAE
8.28
8.89
7.86
11.04
PLO-VLAE
9.83
8.24
6.28
10.93
12

---

Published as a conference paper at ICLR 2020
C ABLATION STUDY ON IMPLEMENTATION STRATEGIES
Table 2: The effect of progressive implementation strategies, i.e. "fade-in" and pre-trained KL penalty, successful training rates. We conducted 15 ablations experiments that each has 4 sub- experiments. As shown, the pro- VLAE cannot be trained successfully without the presented imple- mentation strategies, while each of the strategies helps stabilize the progressive training.
no strategies 0.0
pre-trained KL only fade-in only
Both
0.667
0.733
0.867
D CLOSER INVESTIGATION OF INFORMATION FLOW OVER LATENT VARIABLES
In this section, we present additional quantitative results how information flow among the latent variables during progressive training. We conducted experiments on both 3DShapes and MNIST data sets, considering different hierarchical architectures including combination of different num- ber of latent layers L and different number of latent dimensions Zdim for each layer. Each experi- ment repeated three times with random initializations, from which the mean and the standard deviation mutual information [ 21) were computed.
As shown in Tables for all hierarchical architectures, the information amount in each layer is captured in clear descending order, which aligns with the motivation of the presented progressive learning strategy. Generally, the information also tends to flow from previous layers suggesting a disentanglement of latent factors new latent layers are added. This is especially obvious for 3DShapes data where the generative factors are better defined.
layers,
In addition, models with small latent codes (Zdim 1) are not able to learn the same amount of information (total [xx, 2)) as those with larger latent codes (2dim = 3). The variance of information in each layer in the former also appears to be high. We reason that it may be because that the model trying to squeeze too much information into small code, resulting in large vibrations during progressive learning. On the other hand, while a model has large latent codes (L 4, 7dim 3), the information flow becomes less clear after the addition certain layers. Overall, assuming there K generative factors and there are D dimensions in total available in model, idcally would like to design the model such that D K. However, since is unknown most data, L and 2dim become hyperparameters that need to be tuned for different data sets.
Table 3: 3DShapes, L = 2, Adim
progressive step
T(x; 22)
T(x; 21)
total [(r; 2) 10.68 + 0.19
0
10.68 + 0.19 7.22 0.30
1
5.94
12.88 0.20
Table 4: 3DShapes, L 3, 2dim
progressive step
T(x; 23) T(x; 22)
T(x;; 21)
total [(x; 2)
0
10.16 1 0.113
10.16 + 0.13
9.76 + 0.05
7.36 + 0.10
13.00 + 0.02
2
6.83 + 1.37
6.66 + 0.17
5.80 + 0.41
13.07 + 0.02
13

---

Published as a conference paper at ICLR 2020
Table 5: 3DShapes, L = 4, Zdin =1
progressive step
24);
1x1;23)
1(r; x2)
1(1;
total [ 2)
4.89 0.03
4.89 + 0.03
4.77 + 0.04
3.55 0.04
4.66 0.04
3.75
2.70 + 0.10
10.67 + 0.09
4.55 #
3.53
+ 0.19
2.17 + 0.14
11.72 0.03
Table MNIST, 3, 2dim
progressive step
[(;; 2a)
[(x; 22)
T(a; 21)
total I(x; 2)
5.86 + 1.19
5.86 = 1.19
3.62 + 1.04
4.64 + 2.83
7.62 2.63
3.88 0.75
4.99 + 0.98
2.37 + 0.77
8.25 1.65
Table 7: MNIST, L = 3, 2dim =
progressive step
T(x; 23) 10.08 + 0.10
T(x; 22)
7(x; 21)
total [(x; 2)
10.08 - 0.10
9,97 + 0.05
8.03 + 0.17
11.01 + 0.04
9.91 + 0.04
8.09 + 0.07
6.27 0.02
11.02 + 0.02
Table 8: MNIST, L = 4, Zdim = 3
progressive step
I(x;
I(x; 23)
I(x; Z2)
I(x: 21)
total [(x;
10.06 0.22
10.06 1 0.22
10.11 0.06
7.93 + 0.08
10.98 - 0.02
10.06 0.08
8.1 + 0.04
6.39 - 0.12
10.98 + 0.06
9.99 + 0.09
8.11 + 0.03
6.45 + 0.15
3.52 + 0.07
11.03 + 0.03
14


Reviews:
[{"review_id": "ryx1b4oaYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduce pro-VLAE, an extension to VAE that promotes disentangled representation learning in a hierarchical fashion.\nEncoder and decoder are made of multiple layers and latent variables are not only present in the bottleneck but also between intermediate layers; in such a way, it is possible to encode information at different scales, hence the hierarchical representation. Latent variables can be learned in an incremental way, by making them visible to the whole model progressively, so that as more latent variables become available, they encode lesser and lesser abstract factors.\n\nExperiments are carried out on two benchmarks for disentanglement with annotations and pro-VLAE is compared to other methods in the state of the art.\nHere, the authors introduce an extension of the Mutual Information Gap (MIG) metric, namely MIG-sup: it penalizes when multiple generative factors are encoded in the same latent variable. Qualitative results are also shown for 2 non-annotated datasets.\n\nPROS\n- The idea is fresh, well explained and experiments are sufficiently thorough. The novelty introduced is enough, provided that not much literature has explored progressive representation learning in the context of disentanglement.\n- Results suggest that this is a promising direction for disentangling representations as pointed out by the authors in the conclusions.\n- We appreciated the smart solutions for what concerns the implementation and training stabilization.\n\nCOMMENTS/IMPROVEMENTS\nTo improve the quality of the paper, consider the following comments:\n\n- For the sake of completeness, experiments on Information flow should be also quantitative: it would be interesting to see how the information is captured by the latent variables on average on multiple runs, possibly trying different numbers of latent variables z_i.\n- In sec 3.1 \"z from different abstraction\" is too vague and should be better formalized.\n- In sec 2: \"the presented progressive learning strategy provides an entirely different approach to improve disentangling that is ORTHOGONAL to these existing methods and a possibility to augment them in the future.\": you should change to 'different'.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkxu1H8y9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a method for training Variational Ladder Autoencoder (VLAE) using a progressive learning strategy. In comparison to the generative model using a progressive learning strategy, the proposed method focuses not only on the image generation but also on extracting and disentangling hierarchical representation.\n\nOverall, I think the purpose of this paper should be written clearly. It is not clear whether the purpose is learning the disentangled representation or the hierarchical representation. In my opinion, I think the focus of the proposed method lies in the hierarchical representation through progressive learning, but the experiments are involved more with disentanglement. Furthermore, I believe the authors need to explain the relationship between hierarchical representation and disentangled representation. In particular, it is not clear why learning hierarchical representation is helpful for disentangled representations.\n\nThe qualitative experiments are not convincing since the proposed model looks worse in both the reconstruction and hierarchical disentanglement for MNIST dataset than the base model VLAE, as shown in Figure 5 in [1]. Regarding the metric used in the experiments, the authors mention that the proposed disentanglement metric MIG-sup is what they first developed for one-to-one property, but it seems that it was already proposed in [2]. In addition, the proposed metric requires ground truth for the generative factors, so its usage is limited and not practical.\n\nI think this work is similar to [3] in that both learn disentangled representations by progressively increasing the capacity of the model. I think the authors need to discuss about this work.\n\nAblation studies should be presented to verify the individual effects of the progressive learning method and implementation strategies on performance, respectively.\n\nIn Figures 2 and 3, the performance gap in the reconstruction error of the proposed method is greater than the base model when beta changes from 20 to 30. Therefore, it is necessary to show if it is robust against the hyperparameter beta. \n\nThere is no definition of v_k in Equation (12), so it is difficult to understand the proposed metric clearly.\n\nIn summary, I do not think the paper is ready for publication. \n\n[1] Learning Hierarchical Features from Generative Models, Zhao et al., ICML 2017\n[2] A Framework for the Quantitative Evaluation of Disentangled Representations, Eastwood et al., ICLR 2018\n[3] Understanding disentangling in beta-VAE, Burgess et al., NIPS 2017 Workshop on Learning Disentangled Representations\n\n\n-------------------------------------\nAfter rebuttal:\n\nThanks for the revision of the paper and the additional experiments.\n\nThe authors' comments and further experiments address most of my concerns. In particular, new experiments show that pro-VLAE performs quantitatively and qualitatively better than VLAE. Also, Figure 10 and the result of the information flow experiment using MNIST show that the first layer learns the intended representations properly.\n\nI appreciate the authors’ efforts put into the rebuttal, and the results of additional experiments are reasonably good. Therefore, I increase my final score to 6: Weak Accept.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJe7sjL9tB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an approach to incrementally learn hierarchical representations using a variational autoencoder (VAE). This is shown to be useful qualitatively and quantitatively in terms of disentanglement in the representations.\n\nTo learn the hierarchy, the authors use a ladder architecture based on variational ladder autoencoder (VLAE) but incrementally activate the lateral connections across the layers at varying depth of the encoder and the decoder. A vanilla VAE is first trained. Followed by adding stochastic later connections and then retraining the updated architecture. This combined with beta-VAE inspired upweighting of the KL term leads to learning a hierarchy of representations. Each level of the hierarchy, the representations are disentangled. \n\nInspired by progressive GANs, the authors employ ````\"fade-out\" when traversing the hierarchy. \n\nThe authors also introduce a new metric to capture the one-to-one mapping of the ground truth factors to the latent dimensions.\n\nAblation studies by varying/removing fadeout compared to incremental learning will be useful. Can fade-out (different weighting of each level) be added directly to VLAE without incremental learning? \n\nOverall the paper is well motivated and easy to read. The results look impressive and the learned hierarchy and latent traversals are convincing. A more thorough comparison with VLAE will make the paper stronger.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel way to learn hierarchical disentangled latent representations by building on the previously published Variational Ladder AutoEncoder (VLAE) work. The proposed extension involves learning disentangled representations in a progressive manner, from the most abstract to the more detailed. While at first the reviewers expressed some concerns about the paper, in terms of its main focus (whether it was the disentanglement or the hierarchical aspect of the learnt representation), connections to past work, and experimental results, these concerns were fully alleviated during the discussion period. All of the reviewers now agree that this is a valuable contribution to the field and should be accepted to ICLR. Hence, I am happy to recommend this paper for acceptance as an oral.
Decision: Accept (Spotlight)