Title: PRELIMINARY THEORETICAL TROUBLESHOOTING IN VARIATIONAL AUTOENCODER

Abstract: Variational Autoencoder plays an important role in disentangled representation learning. However, it is found facing posterior collapse problem and learning multiple variants in one factor. What would be learned by variational autoencoder (VAE) and what influence the disentanglement of VAE? This paper tries to preliminarily address VAE's intrinsic dimension, real factor, disentanglement and indicator issues theoretically in the idealistic situation and implementation issue practically through noise modeling perspective in the realistic case. On intrinsic dimension issue, due to information conservation, the idealistic VAE learns and only learns intrinsic factor dimension. Besides, suggested by mutual information separation property, the constraint induced by Gaussian prior to the VAE objective encourages the information sparsity in dimension. On disentanglement issue, subsequently, inspired by information conservation theorem the clarification on disentanglement in this paper is made. On real factor issue, due to factor equivalence, the idealistic VAE possibly learns any factor set in the equivalence class. On indicator issue, the behavior of current disentanglement metric is discussed, and several performance indicators regarding the disentanglement and generating influence are subsequently raised to evaluate the performance of VAE model and to supervise the used factors. On implementation issue, the experiments under noise modeling and constraints empirically testify the theoretical analysis and also show their own characteristic in pursuing disentanglement.

1. INTRODUCTION:

Variational AutoEncoder(VAE)s (Kingma & Welling (2013) , Rezende et al. (2014) ) have shown their powerful human-like abilities: modelling causal relationship, unsupervisedly extracting disentangled factors/representation (Bengio et al. (2013) ) and generating signals with abundant diversities in a "latent-factor-controllable" way. Those capabilities enable the knowledge transferring through shared causes/factors among different tasks/experiences, emphasized as the important human advantages against the current machine by Lake et al. (2016) and compling with the ideal mental imagery mechanism in memory and thinking. Benefitted from those capabilities, VAEs have been widely applied to various applications, including disentangled representations learning of images and time series (Higgins et al. (2016) , Kulkarni et al. (2015) , Mathieu et al. (2016) , (Fabius & van Amersfoort (2014) ), few-shot and transfer learning (Rezende et al. (2016) , Higgins et al. (2017b) , Higgins et al. (2017a) ), causal relationships modeling (Louizos et al. (2017) ), pixel trajectory predicting (Walker et al. (2016) ), joint multi-modal inference learning (Suzuki et al. (2016) ), increasing diversity in imitation learning (Wang et al. (2017) ), generation with memory (Li et al. (2016) ) and etc.

However, the lack of public theoretical study regarding the generating and inference procedure induced by VAEs is tripping the research process:

• Intrinsic Dimension Issue: Could the VAE learn the intrinsic number of factors underlying the data? • Real Factor Issue: Could the VAE learn the real generating factors underlying the data or just some fantasies? • Disentanglement Issue: What are the need and range induced by the word "disentanglement"?

• Indicator Issue: Could the effectiveness of current disentanglement metric be guaranteed?

All those lacks distillation would make researchers hard to effectively compute through their knowledge from the experiments and incline to make some avoidable arguments and considerations.

This paper will first discuss the properties of the idealistic VAE1 to target the aforemention issues and then moving to

• Implementation Issue: Could the aforemention analysis be instructive in real implementation?

For Intrinsic Dimension Issue, the information conservation theorem shows that idealistic VAE learns and only learns the intrinsic factor dimension illustrated in Fig.

(1). To Disentanglement Issue, the clarification on disentanglement is subsequently made. For Real Factor Issue, the factor equivalence properties shows that idealistic VAE possibly learns all factors in equivalent class rather than exactly pre-specified real factors. For Indicator Issue, limitations of the current disentanglement metric are analyzed and several new indicators are introduced.

After that, for Implementation Issue, we relax the discussion to the case that decoding procedure are not deterministic through noise modeling perspective. The experiments empirically testify that the knowledge derived form the idealistic case could be applied to the realistic sampling case and demonstrate the behaviors of different noise assumptions as well as our indictors. The illustration of the information conservation theorem 1. Suppose that the oracle data, denoted by random variable x, is generated by y (with P independent unit Gaussian random variables) with a homeomorphism mapping x = φ(y). Idealistic VAE will be forced to learn the factor z (with H independent unit Gaussian random variables) that generates the x with a homeomorphism mapping x = ψ(z). It yields z = ψ -1 • φ(y) and y = φ -1 • ψ(z). Then according to the information conservation theorem, it must hold that H = P .

2. GAUSSIAN-VAE MODEL:

Gaussian-VAE (Kingma & Welling (2013) , Rezende et al. (2014) ) is an scalable unsupervised representation learning model (Higgins et al. (2016) ), and since Gaussian distribution can be continuously and reversibly mapping to many other distributions, the theoretical analysis on it is also instructive for other continuous latent factors VAE.

Gaussian-VAE assume that input x is generated by several independent Gaussian factors z, that is p θ (z) = N (z|0, I H ). The generating/decoding process is modeled as p θ (x|z) and the inference/encoding process q φ (z|x) is treated as the approximate posterior distribution. Both of them are parameterized by the neural network with parameter θ and φ.

In VAE setting, the approximate inference method is applied to maximizing the variational lower bound of p θ (x) = p θ (x|z)p θ (z)dz,

L(q) = E z∼q φ (z|x) log p θ (x|z) -D KL (q φ (z|x)||p θ (z)) ≤ log p θ (x).

(1)

2.1. IDEALISTIC VAE:

In order to get assess to aforemention issues, we will start the analysis from the idealistic situation: An idealistic VAE model means that it can perfectly encode the signal into "used" factors and perfectly decode the "used" factors to original input signal and the factors follows i.i.d unit Gaussian distribution.

The idealistic VAE discussed in this literature should also under the following the Deterministic Assumption on q φ (x|z) and p θ (z|x). If the factors of x are well understood, the generation process should be deterministic, that is p θ (x|z) = δ(x = ψ(z)). We limit the consideration that q φ (z|x) = δ(z = ψ -1 (x)) is also deterministic as well for simplicity of analysis in this paper. For more complex situation, this consideration could be also basic and instructive.

We try to address aforemention issues by disregarding the training procedure and direct considering the idealistic VAE's behavior.

3. ON INTRINSIC DIMENSION ISSUE:

In order to get asses to the intrinsic dimension issue, we will present the information conservation theorem. It states some basic truths, e.g. two independent Gaussian and three independent Gaussian cannot be the generating factor of each other under continuous mapping. The theorem thus further illustrates that idealistic VAE learns and only learns the intrinsic factor dimension.

From the perspective of VAE objective, we will also show that the constraint induced by Gaussian prior, plays the lasso on the mutual information which encourage to clip down the small information dimension and promotes information sparsity in factors. In order to derive this perspective, the mutual information separation theorem and objective decomposition theorem are subsequently raised.

3.1. INFORMATION CONSERVATION:

Theorem 1 (Information Conservation). Suppose that z = (z 1 , • • • , z H ) and y = (y 1 , • • • , y P ) are sets of H and P (H = P ) independent unit Gaussian random variables, respectively, then these two sets of random variables can not be the generating factor of each other. That is, there are no continuous functions f : R H → R P and g : R P → R H such that z = g(y) and y = f (z).

Proof in Appendix B. The principle of the theorem is visually illustrated in Fig. 1 .

3.2. SEPARATION OF THE MUTUAL INFORMATION:

The mutual information regarding the factors learned by the inference/encoder network and the signal x can be a good quantity for evaluating the generating influence. That is,

EQUATION

In order to understand and estimate which factor of the VAE was learnt and influenced the generating process, I encoder (x; z h ) can be taken as a rational indicator2 . If we assume that z 1 , z 2 , • • • , z H is conditional independent given x3 , it can yield a useful result as the following.

Theorem 2 (Mutual Information Separation). Let z 1 , • • • , z H be independent unit Gaussian distribution, and z 1 , z 2 , • • • , z H be conditional independent given x. Then

EQUATION

Proof in Appendix B. This theorem suggests that if the learnt q φ (z) can factorize and the q φ (z|x) can factorize, then the consideration of each I encoder (z h ; x) won't be excess or lose information. Besides, when those term comes in the optimization objective, then it can start the negotiation between the information preservation and dimension reduction and play the role of lasso that clip down the factor in dimension with small mutual information.

Theorem 3 (Objective Decomposition). The terminology follows the aforemention definitions and if the involved KL-divergence and mutual information is well defined, then

EQUATION

)

Proof in Appendix B. The theorem demonstrates that the second term in variation lower bound in Eq. ( 11) is capable of controlling both the mutual information of x and z induced by the encoder network as well as the similarity of the learnt q φ (z) and the prior p θ (z). Further, the theorem suggests that it possesses the lasso capacity of clipping down the non-intrinsic factor dimension to some extent, visually demonstrated in Fig.

(2). 

4. ON DISENTANGLEMENT ISSUE:

Inspiration on Disentanglement: According to the information conservation theorem 1, the independent unit Gaussian factor assumption forms a strong inductive bias and facilitates the model incline to achieve most efficient coding. Under this assumption, the number of the learnt "used" factors of idealistic VAE should be the same as the true factors number under some assumptions such as the learnt q φ (z) should equal p θ (z) and decode/encode process is continuous and reversible. Empirically, sometimes, though the number latent of factors sometimes is pre-specified larger, only a small amount of unit Gaussian variables regarding the factors of VAE have been used while q φ (z|x) is close to deterministic. The theorem helps provide an interpretation to explain this phenomenon.

Here, in order to avoid the ambiguity of the terminology of disentanglement in this paper, we make the following clarification.4 

• The disentanglement of the learnt representation/factors in this literature refers to two parts depicted in Theorem 1:

-the factors are closer to be independent with each other, -the factors incline to be able to generate the oracle signal and to be inferred perfectly from the oracle signal through a continuous procedure/mapping. • The "disentanglement" refers to the closeness of the learnt factors to the pre-specified independent factors/concpets that can generate the oracle signal and be perfect inferred through a continuous procedure/mapping such as the independent semantic/visual factors.

Therefore, the estimation for D KL (q φ (z)||p θ (z)) that reflects the divergence of the learnt factor distribution and the i.i.d. unit Gaussian prior can be good a indicator to supervise the independence of the factors and served to quantitatively assess the disentanglement of each extracted factor, while the similarity regarding the original signal and reconstruction place another part of the disentanglement.

The "disentanglement" will be shown hard to be obtained in an unsupervised manner. Concretely, even in the idealistic cases, the extracted factors tend to possess the intrinsic number of latent factors of the model, while there are still possibly large variations of these factors due to it can be obtained only in the equivalent class induced by the pre-specified factors as proved in the next section.

5. ON REAL FACTOR ISSUE:

As for real factor issue, Gaussian Factor Equivalence theorem, (i.e. linear orthogonal transformation of Gaussian factor set are still gaussian factor set.), and Linear Factor Equivalence Class will be presented. They states that idealistic VAE are possibly learns any factors set in the factors equivalence class, and we should not expect "one-to-one" correspondence by disentanglement.

5.1. FACTORS EQUIVALENCE:

Theorem 4 (Gaussian Factor Equivalence). Suppose that z

= (z 1 , • • • , z H ) is a set of H indepen- dent unit Gaussian random variables. Let Q ∈ R

H×H be an orthogonal matrix and then y = Qz is also a set of H independent unit Gaussian random variables. Besides, z and y can generate each other through a linear homeomorphism mapping.

Proof in Appendix B. This theorem implies that there are a class of unit Gaussian random variables which can generate each other and have equivalent conservation information, as indicated by the following theorem.

Theorem 5 (Linear Gaussian Factor Equivalence Class).

[z] = {y|y = Qz, Q ∈ R H×H be the orthogonal mapping.} Then ∀y ∈ [z], y is a set of H independent unit Gaussian random variables and can generate z through an linear homeomorphism mapping.

The theorem clarifies that if Gaussian-VAEs have an linear matrix multiplication freedom degree of learning the factors, then the factors in the equivalence class can all be possibly learnt.

The empirically results tally with the above analysis(see Fig. 3 ). Suppose the visual semantic concepts can be viewed as a set of independent Gaussian variables (z = It can be seen that changing one factor results in multiple semantic factor changes in a comprehensible manner rather than the "one-to-one" correspondence which reflexed analysis regarding generating factor equivalence.

(z rotation , z gender , z with-glass , • • • ) T ) which are desired to be captured and learnt by VAEs, while the model is also possible to learn the independent factor set y

= (y 1 , • • • ) T = Qz in the equiv- alence class [z].

This explains why changing one factor like y 1 sometimes empirically results in change in multiple visual concepts.

This perspective suggests that it's actually hard to obtain the "disentangled representation" that exactly "one-to-one" corresponds to the "independent semantic representation" even though they are in the same equivalence class. As a result, the idealistic VAE model just tend to learn the "entangle representation" if we do preset "oracle generating factors" belonging to the equivalent class.

However, though those conclusions might be upsetting, it seems not be biology impossible. A neuron in hippocampus of animals was suggested to combinatorially possess several representation capabilities. E.g., Aronov et al. (2017) found that some neurons in rat's hippocampus involved in spatial representation also were involved in representing sound frequencies after training rats by a tasks that required them to use a joystick to manipulate sound in frequency continuously.

6. ON INDICATOR ISSUE:

6.1 LIMITATION OF THE EXISTING DISENTANGLEMENT METIC Higgins et al. (2016) proposed a "simulated factor" based "disentanglement" metric on the simulation datasets. However, according to Gaussian factor equivalence theorem 4 that even idealistic VAE will still learn the factors in the equivalence class, their metric could be effective sometimes for disentanglement but might suffer instability when evaluating the VAE in different trials (detailed in Appendix C).

Further, this metric could be hardly calculated in the real datasets to provide direct feedback of the "disentanglement". The reason is that it must pre-know the generating factors expected to be learnt.

6.2. PROPOSED INDICATORS:

In order to quantify the disentanglement performance5 as well as the I encoder (x; z), we assume that q * (z) is a factorized zero mean Gaussian estimation for q φ (z).

We can then list the indicators for assessing latent factor disentanglement: Definition 1 (Estimation for Ex∼p data (x) D KL (q φ (z|x)||p θ (z))).

DKL (q:

φ (z|x)||p θ (z)) = 1 M M m=1 D KL (q φ (z|x m )||p θ (z)).

(5)

Definition 2 (Estimation for I encoder (x; z)).

EQUATION

Definition 3 (Estimation for I encoder (x; z h ) which quantifies the influence of each factor).

EQUATION

Note that the above indicators 2-4 need the value of q * (z), we now introduce how to calculate this term based on Theorem 3. Through the minimization equivalence, we know that

EQUATION

the q * (z) can then be obtained by gradient method from solving the following optimization problem.

EQUATION

7 IMPLEMENTATION ISSUE

7.1. VAE WITH NOISE MODELING:

In order to testify the idealistic consideration in real situation, we are not going to learn all the factors or equivalently we assume that datasets have noise 6 (this situation correspond to that p θ (x|z) is not deterministic since the major factors z only forms a subset of the whole factors. 7 ), we integrate the noise modeling into our model:

p θ (x|z) = N (x|G(z), σ 2 I d ),

where σ 2 is either manually enumerated or adaptive learned. Noise modeling can be found crucial in influencing disentanglement in experiment since it would actually define the factors aimed to be learnt and subsequently influence the learnt intrinsic dimension suggested by information conservation property of VAE.

7.2. NOISE MODELLING WITH AUXILIARY CONSTRAINT:

The entangled representation can be caused by the over-large of searching space of q φ (z|x).

If the learned q φ (z) = q φ (z|x)p data (x)dx has a big divergence to p θ (z), then the VAE model tends to learn the entangle representation as it violates the one part of the disentanglement (clarified in section 4). Actually, in the VAE model, what we want is to search in the space that (Higgins et al. (2016) , Higgins et al. (2017b ), Larsen et al. (2015) , Mathieu et al. ( 2016)) suggest it an effective way. 6 Noise can be viewed as the generating factor that we are not interested in. 7 Notice the indeterministic p θ (x|z) could lead to the indeterministic p θ (z|x), but since the minor factors are supposed to have less influence on x, it could not bother the deduction using the knowledge we derived too much.

q φ (z) is possibly similar to p θ (z)8 . By implementing this ideal, we add auxiliary upper bound Ex∼p data (x) D KL (q φ (z|x)||p θ (z)) (detailed in Theorem 3) of D KL (q φ (z)||p θ (z)) to the original objective. This equivalently leads to the approach of β-VAE raised by Higgins et al. (2016) .

EQUATION

where β > 1.

7.2.1 RELATION OF VAE AND β-VAE UNDER GAUSSIAN NOISE ASSUMPTION WHEN σ 2 IS PRE-SPECIFIED

Equivalent objective of σ 2 pre-specified Gaussian noise VAE:

E z∼q φ (z|x) x -G(z) 2 2 -2σ 2 D KL (q φ (z|x)||p θ (z)).

Equivalent objective of σ 2 pre-specified as σ 2 pre Gaussian noise β-VAE:

E z∼q φ (z|x) x -G(z) 2 2 -2βσ 2 pre D KL (q φ (z|x)||p θ (z)),

where we call βσ 2 pre the normalized variance. It's shown that when the σ 2 is pre-specified, manually tuning it is the same as manually tuning β with a fixed σ 2 pre (for example σ 2 pre = 1) under Gaussian noise assumption. This equivalence saves our time for extra experiment studying the behaviors of this two cases and we call those two case noise specified β-VAE.

8. EXPERIMENT 8.1 DATASET:

MNIST is a database of handwritten digits (Lcun et al. (1998) ). CelebA (Liu et al. (2015) ) is a largescale celebfaces attributes datasets and only its images are used in our experiments. More details are in Appendix D. The extensive comparison of Gaussian-noise modeling β-VAE and Gaussiannoise with specified variance (β)-VAE is made on MNIST based on our indicators to exploring the disentanglement as well as to testify the theorem and analysis from the idealistic case to the realistic sampling case. The experiments on CelebA will be auxiliary to further support the generating factor equivalence class theorem9 .

By setting β as different values, we compare the performance of β-VAE with and without prespecified noise on MNIST. We specifically listed the result of β(=1)-VAE in all cases. More details can be found in Appendix D.2.

8.1.1. ON IMPLEMENTATION ISSUE:

• Noise modeling/specification influence the disentanglement.

The noise specifications and modeling significantly influence the model evidence quantitatively and reconstruction qualitatively, as clearly shown in Fig. (4 ).

The noise specifications significantly influence the divergence regarding q φ (z) and p θ (z) and noise specified and noise learning β-VAE achieve similar disentanglement quantitatively based on the similar indicator behaviours of DKL (q φ (z)||p θ (z)) in • Auxiliary constraints can influence the disentanglement in a different way.

The prominent difference induced by auxiliary constraints would be its stronger suppression on Ĩencoder (x; z) and the number of influential factors although changing the noise level also possess this capability indirectly. It's somewhat obvious to see and compare value of different indictors under the different β setting by sliding on the green/blue line. The bigger β, the lower DKL (q φ (z)||p θ (z)) and roughly the better reconstruction and hypothesis learnt. However, it's more interesting that in regard to the normalized variance, noise learning β-VAE enhances the suppression on Ĩencoder (x; z), as depicted in Fig. (6a ) and that is comprehensible since β-VAE is minimizing the auxiliary constraints both I encoder (x; z) + D KL (q(z)||p(z)) based on Theorem 3.

• Ĩencoder (x; z h ) effectively determines the "used" factors and VAEs incline to learn the intrinsic factor dimension in realistic sampling case when the disentanglement achieves in some extent. 2) and ( 16), the indicator Ĩencoder (x; z h ) determines the "used" factors. According to those figures, under the suitable noise assumption, VAEs automatically suppress the auxiliary factors and learn the intrinsic factor dimension as it still capable to have good reconstruct abilities and its D KL (q φ (z)||p θ (z)) closer to zero. This phenomenon has already suggested by the information conservation theorem 1.

• Factor equivalence is generally hold and VAEs possibly learn any factor set in the equivalence class. 16) and Table 1 . 

9. FUTURE WORK:

From the perspective of representation learning:

• It is interesting that the topology properties of oracle signal are used to obtain the proof for the information conservation theorem. Other situations including that data owns several connected components can be further considered and would uncover the efficient coding properties of discrete factors. • When q φ (z|x) is far from deterministic, the discussion would be crucial for many other general purposes induced by word disentanglement. Those study may further extend to the case that data containing different dimension manifolds and to the core VAEs' pattern separation/completion/generalization capabilities.

A APPENDIX: NOISE MODELING DEDUCTION A.1 GAUSSIAN NOISE DEDUCTION:

The VAE objective (i.e., the variational lower bound) can be treated as function of the θ and φ where noise variance parameters σ 2 are contained in θ,

EQUATION

)

Here the SGVB estimator in (Kingma & Welling ( 2013)),

LB (θ, φ, x m ) = [ 1 L L l=1 log p θ (x m |z l )] -D KL (q φ (z|x m )||p θ (z)

) is used. Note that the noise variance σ 2 is also taken as an optimization variable in the model, making the model capable of better adapting noise variation of data in practical cases in a totally automatic way, instead of a manually set manner.

Given multiple data points from a dataset X, we can construct an estimator of the mean marginal likelihood lower bound of the full dataset, based on minibatches

EQUATION

where the minibatch X M = {x m } M m=1 is a randomly drawn sample set of M datapoints from the full dataset X. Such a lower bound also constitutes an important indicator for model evidence in latter experiment. We call it the empirical variational lower bound (EVLB) in the following.

Note that LB (θ, φ, x m ) L(θ, φ, x m ) and we can deduce that

EQUATION

)

The last inequality holds due to D KL (p data (x)||p θ (x)) ≥ 0.

The alternative optimization strategy can be readily utilized to design the algorithm for solving the model by iteratively updating the noise parameter and the network ones. During the optimization process, the objective can be monotonically increasing, and thus the algorithm can be guaranteed to be convergent.

The algorithm is summarized as follows:

Optimization for parameters except σ 2 :

Optimization for σ 2 :

σ 2 = L l=1 M m=1 x m -G(z m,l ) 2 2 dM L

. (Close form solution in regard to LM (θ, φ, X M ).)

Direct gradient method to the transformed variable log simga = log σ ∈ R can be implemented to lift the lower bound LM as a result to increase the likelihood as well.

A.2 MOG NOISE DEDUCTION:

The noise ε in real situation might be more complex than a simple Gaussian, like that existed in real photographs (Plotz & Roth (2017) ). We thus try to further ameliorate the noise setting as a mixture of Gaussian(MoG) noise. Such noise modeling strategy has been widely verified to be effective in applications, like matrix factorization (Meng & Torre (2014) ) and robust principal component analysis (Zhao et al. (2014) ). That is, we assume that

EQUATION

)

Let c d ∈ {0, 1} K be the latent indicator random one-hot variable,

K k=1 c dk = 1, for the MoG- noise component of pixel indexed by d. Let Π = [π 1 , • • • , π K ] and Σ = [σ 2 1 , • • • , σ 2 K ]

be the ratio and variance of each component, respectively. Let Π, Σ be contained in θ. The conditional joint distribution turns to be

EQUATION

)

The posterior distribution q φ (z, c|x) can be factorized as q φ (z|x)q(c|x, z), where q φ (z|x) will be direct learnt and the alternative of q(c|x, z), q(c|x, e) will be set to the last step p θ (c|x, e) in regard to EM procedure. The lower bound of log p θ (x) is then reformulated as follows:

EQUATION

Similar to the Gaussian case, the reparamerization trick is implemented,

EQUATION

where z = En(x) + Σ z|x 1/2 (x)e.

By utilizing the SGVB estimator, we get,

EQUATION

Given an input dataset X, we can then construct an estimator to the mean marginal likelihood lower bound of the full dataset, based on minibatches, as follows:

EQUATION

where z m,l = En(x m ) + Σ z|x 1/2 (x m )e (l) and the minibatch X M = {x m } M i=1 is a randomly drawn sample of M datapoints from the full dataset X.

Then let

EQUATION

where z old = En old (x) + Σ old z|x 1/2 (x)e, and we can get

EQUATION

)

The EM algorithm can be naturally employed to solve the model. The implementation steps are listed as follows:

Step 1. Expectation Step.

Set q(c|x m , e (l)

EQUATION

where z m,l : z m,l old = En old (x m ) + Σ old z|x (x m )e (l) . The Objective in Maximization Step is obtained as the following,

EQUATION

Step 2. Maximization Step:

Fix: q(c|x, e) determined in the Expectation Step.

EQUATION

Update [Π, Σ] and {θ, φ}/[Π, Σ] by alternative optimization strategy.

Update Π, Σ: note here z m,l : z m,l old = En old (x m ) + Σ old z|x (x m )e (l) , and we can easily get the closed-form updating formula for these parameters:

EQUATION

)

Update {θ, φ}/[Π, Σ]: gradient methods with respect to {θ, φ}/[Π, Σ]. Note here z m,l = En(x m )+ Σ z|x (x m )e (l) .

The algorithm can then be summarized as follows:

1. Initialize the coefficient of {θ, φ}/[Π, Σ] and the coefficient of noise ε: Π,Σ.

2. Sample e from N (0, I H ) to obtain e 1 , • • • , e M [One for each element sample in the mini batch in the next step (L here is set to 1)].

3. Sample a mini batch X M from p data (x).

4. Implement EM algorithms as aforementioned (approximate inference for q(c, z|x)): Expectation: calculate γ dmk . Maximization: update [Π, Σ], Update {θ, φ}/[Π, Σ] with gradient methods.

5. Goto 3: Until Trigger End-Criterion.

B APPENDIX: PROOF:

Proof. For theorem 1. Proof by Contradiction. Suppose those two function exist, and we will show that they will be inverse mapping of each other and the homeomorphism mapping of R H and R P . Since R H and R P have different topology structures (P = H), the homeomorphism mapping will not exist.

z = g(y) = g(f (z)) ∀z ∈ R H ⇒ g • f = I H y = f (z) = f (g(x)) ∀y ∈ R P ⇒ f • g = I P

Since both f and g are continuous, there is a homeomorphism mapping between R H and R P and it leads to the contradiction.

Proof. For theorem 4. We only need to test the mean and variance of y.

E(y) = E(Qz) = Q E(z) = 0 Cov(y, y) = QCov(z, z)Q T = QIQ T = I

Therefore, y is another set of H independent unit Gaussian random variables. Since x = Q T y, z and y can generate each other with an linear homeomorphism mapping.

Proof. For theorem 2,

I(z 1 , • • • , z H ; x) = p(z 1 , • • • , z H , x) log p(z 1 , • • • , z H , x) p(z 1 , • • • , z H )p(x) dz 1 • • • dz H dx = p(z 1 , • • • , z H , x) log H h=1 p(z h |x) H h=1 p(z h ) dz 1 • • • dz H dx = H h=1 p(z h , x) log p(z h |x) p(z h ) dz h dx = H h=1 I(z h ; x).

Proof. For theorem 3.

E x∼p data (x) D KL (q φ (z|x)||p θ (z)) = q φ (z|x)p data (x) q φ (z|x)p data (x) p θ (z)p data (x) dx = q φ (z|x)p data (x) q φ (z|x)p data (x) q φ (z)p data (x) q φ (z) p θ (z) dx = I encoder (x; z) + D KL (q φ (z)||p θ (z)). (27) B.1 AUXILIARY EXPLANATIONS FOR INDICATORS

Corollary 1. The terminology follows the aforemention definitions and if the involved KLdivergence and mutual information be well defined then

EQUATION

The proof of corollary 1 is the same as that of theorem 3. This corollary suggests that the estimation in definition 2 provides another upper bound for the capacity of the encoder network. Empirically, this estimation is a much tighter estimation than using the estimation in definition 1.

Corollary 2. The terminology follows the aforemention definitions and if the involved KLdivergence and mutual information be well defined then

EQUATION

)

The corollary is an direct result of theorem 3 and corollary 1. It suggests that the estimation in definition 4 is a lower bound for D KL (q φ (z)||p θ (z)).

Definition 5 (Another Estimation for D KL (q φ (z)||p θ (z))).

EQUATION

Empirically, DKL (q φ (z)||p θ (z)) and DKL (q φ (z)||p θ (z)) shown the same estimation results on M-NIST.

Definition 6 (Another estimation for I encoder (x; z)).

EQUATION

Definition 7 (Another estimation for I encoder (x; z h ) which quantifies the influence of each factor).

Īencoder (x; z:

EQUATION

C APPENDIX: ANALYSIS ON THE "DISENTANGLEMENT" METRIC RAISED IN β-VAE (HIGGINS ET AL. ( 2016))

The terminology inherits those in the β-VAE paper. The main idea of that "disentanglement" metric is to create a statistic point z dif f relevant to the model for each simulated factor respectively and then to use a linear classifier to project the statistic point to the corresponding index of the simulated factor. If the statistic points induced by the model are easy to be separated then the model is thought to learn "disentangled" representation.

Here, we will argue that even for the idealistic VAE model that follows the disentanglement conditons 4 could still receive bad score under that performance metric in some situations.

Suppose that the true simulated factors v follows N (0, I H ). Then the learnt "used"10 factor z can be in the equivalence class [v] according to theorem 4. Concretely, there exists an orthogonal transformation Q such that z = Qv.

Suppose that the simulated factors with index y of v is fixed. Suppose v 1 y-f ixed and v 2 y-f ixed are two random variable representing the samples from the y-fixed v. Then the factors inferred by the idealistic VAE turns to be z 1 = Qv 1 y-f ixed and z 2 = Qv 2 y-f ixed . In order to calculate the statistic point z dif f (y) = E |z 1 -z 2 |, we first calculate the mean and variance of (z 1 -z 2 ).

EQUATION

Therefore, z dif f (y) can be obtained through the diagonal value of 2I -2q y q T y . That is,

EQUATION

From the above equation, the location of statistic point is unique determined by (q 2 y1 , • • • , q 2 yH ). When (q 2 y1 , • • • , q 2 yH ) is close to the vertex of the unit cubic for each y then all the statistic points turn to be easily separated.

However, from the perspective of the objective, all the orthogonal Qs are with the same potential to be learnt. It seems not to be with a small probability that statistic points of different indexes take similar location. For instance, if H = 2 and Q =

1 √ 2 1 √ 2 -1 √ 2 1 √ 2 then z dif f (1) = z dif f (2)

cannot be separated while the representation still follows the disentanglement conditions.

Among different trials, the Q might contribute to that "disentanglement" metric but also might not. That explains why that metric is unstable.

D APPENDIX: EXPERIMENT DETAILS:

We set L to 1, and minibatch size M to be 100 in all practices. All the pixels value have been linear normalized in to [0,1].

D.1 CLARIFICATION ON THE CORRECTION ON RESULTS:

We find in the last version that the code on calculating q * (z) is wrong. Concretely, the objective of the KL-divergence of two Gaussian is incorrect calculated. That influences the estimation of 2017)). Some other works advanced the mechanism under the VAE generation/inference processes. Typical works include the iterative attention generation/inference mechanism raised by Gregor et al. (2015) , normalizing flow proposed by Rezende & Mohamed (2015) that enhanced the expressive ability of the approximate posterior and its variants (Kingma et al. (2016) ).

Despite the improvement to the VAE itself, some other efforts were made by the ensemble between GAN with VAE. E.g., Larsen et al. (2015) unified GAN and VAE to obtain a better reconstruction and a high-level abstracts visual features embedding. Mathieu et al. ( 2016) also unified GAN and VAE but put emphasis on disentangling factors of variation. GANs without auxiliary design would learn the data distribution disregarding its noise level though suffer from unstable training and mode collapsing (Salimans et al. (2016) ) while VAEs would assume a decomposition of the noise and oracle clean datapoint regarding the noise data with an auxiliary prior on the distribution regarding the factors.

Besides, many efforts were made by regularization on the factor distribution or factor generating effect. E.g., Makhzani et al. (2015) introduced an adversarial loss into the latent space of the autoencoder which in idealistic case could learn any kind factor/lantent distribution including those contributing to the disentangled factors/representation. InfoGAN, raised by Chen et al. (2016) , introduced the infomax principle to GAN by adding an auxiliary mutual information regularization which enabled the inference of GANs and led to a better disentangled representation as well.

Recently, there is a new VAE variation is proposed by Higgins et al. (2016) who introduced the β-VAE framework which enhanced the constraints regarding the KL-divergence of the posterior and prior distribution of VAE and showd a novel disentanglement performance. This method has obtained a better performance as compared with conventional VAE methods, especially on its flexible tuning a compromising a parameter beta between the KL-divergence term and the likelihood term (the variational lower bound). 

:

That is, roughly, the VAE whose decoding procedure and encoding procedure are both deterministic.

If I encoder (x; z h ) = 0, it yields x and z h are independent with each other. The bigger I encoder (x; z h ), the more information z h conveys regarding x.

It follows the real implementation assumption that Σ z|x (x) = diag(σz1(x), • • • , σzH (x)).

Notice this clarification is based on the assumption that q φ (z|x) is deterministic. If not, then continuous and reversible mapping of encoder constrain should be loosen and also the reversibility of the decoder should be loosen. If we further demand the enhancement of the pattern separation and completion ability, that is, to make the hyperspheres induced by the observation points in the factor domain fully fill up the whole compact factor ball, then some auxiliary constrains including the restriction on mutual information I encoder (x; z) (defined in Section 3.2) induced by the encoder network need to be introduced.

Visual recognition could also provide a way to get assess to the factors equivalent class and supervise the disentanglement. If we assume that the most of the visual concept/factors follow the assumption regarding the "disentanglement", it is rational to qualitatively measure the interpretability of extracted latent factors by human perception to infer the disentanglement. Besides, previous empirical evidences of VAE applications

Jensen Shannon Divergence and other integral probability metric which can be good choice and directly be optimized through a adversarial format(Makhzani et al. (2015)) as well. However, in practice, we were defeated by the unstability of training GAN-like model.

The noise assumption is slightly changed into mixture of gaussian in Appendix A.2 for CelebA.

The auxiliary unused factor is innocuous for the subsequent analysis.

Figures and Tables:

Figure1: Idealistic VAE learns and only learns the intrinsic factor dimension. The illustration of the information conservation theorem 1. Suppose that the oracle data, denoted by random variable x, is generated by y (with P independent unit Gaussian random variables) with a homeomorphism mapping x = φ(y). Idealistic VAE will be forced to learn the factor z (with H independent unit Gaussian random variables) that generates the x with a homeomorphism mapping x = ψ(z). It yields z = ψ -1 • φ(y) and y = φ -1 • ψ(z). Then according to the information conservation theorem, it must hold that H = P .

Figure 2: The sparsity of mutual information occurs; Ĩencoder (x; z h ) determines the "used" factors; disentangled VAE pursues the intrinsic factors dimensions; generating factor exists equivalence class. Noise learning β-VAE (β = 10, equivalent σ 2 = 0.112): Ĩencoder (x; z h ), σ 2 z h and qualitatively influential factor traversals. The top pulse subgraph: Ĩencoder (x; z h ) of each factor. The bottom reverse pulse subgraph: the estimated variance σ 2 z h of each factor. The montages: influential factor traversals. We select those factor traversals with visually most interpretable/comprehensive effects to present and the whole influential factor traversals are listed in appendix 13. The phenomenon of the multiple semantic change induced by the same learnt factor and the encoding of same semantic among different learnt factor tallies with factor equivalence class theorem 4. The similar plot of its counterpart with specified normalized noise can be found in Fig.(16) in Appendix.

Figure 3: We should not expect "one-to-one" correspondence by disentanglement. One-shot traversal & generating factor equivalence class demonstration. The images are generated by MoG-2 β(=40) VAE trained on CelebA. The seed image is obtained out from the datasets. Each block represents the traversal of the generating factor from [-3 + z seedh , +3 + z seedh ]. (a) corresponds to face color white-yellow & female-male change. (b) corresponds to face color white to yellow change. (c) corresponds to background yellow to blue change. (d) corresponds to hair color white to black & face width change.It can be seen that changing one factor results in multiple semantic factor changes in a comprehensible manner rather than the "one-to-one" correspondence which reflexed analysis regarding generating factor equivalence.

Fig. (5a) and the number of normal variance factors Fig. (5b) in regard to the normalized variance.

Figure 4: Noise specification/modeling influence the learned hypothesis and the reconstruction. Blue Line: the EVBL (defined in Appendix A.1) of different specified σ 2 VAE [correspond to prespecified σ 2 pre β-VAE illustrated in Section 7.2.1 with equivalent σ 2 = βσ 2 pre ] . Green Line: the EVLB of noise learning β-VAE with different specified β [normalized to σ 2 = βσ 2 learnt for convenient comparing] . Green Pentagram: the EVLB of Gaussian noise learning VAE. Other Figures: their reconstructions on the testing set. The bigger EVLB, the better hypothesis that model learnt.

Figure 6: β-VAE is more suppressive on the mutual information. (a) Ĩencoder (x; z) of different VAE models & (b) Number of influential generators of different VAE models

The reflections of the generating equivalence properties 5.1, that is, single factor could result in multiple semantic concepts change and same semantic concept could be encoded in different factors, are again well demonstrated by Fig. (7), Fig. (2), Fig. (

Figure 7: CelebA: Generating Factors Traversal of MoG-2 β(=40)-VAE.

Figure 8: Learned σ 2 of different β setting

Variants and factors correspondence on CelebA ( MoG-2 β(=40)-VAE)

Table content: <table><tr><td colspan="7">Variant glass height blue to yellow* black to white* half bright half gloomy*</td></tr><tr><td>Factor</td><td>73</td><td colspan="2">37,45 13, 96,40,45,118</td><td>7</td><td/><td>110</td></tr><tr><td colspan="7">Variant face(big to small) lighting face lighting skin color(white to yellow)</td></tr><tr><td>Factor</td><td/><td>63,77,82</td><td>73,90</td><td>120,110</td><td/><td>102,96,28,63,82</td></tr><tr><td colspan="7">Variant head direction neck length hair color gender mouth open to close</td></tr><tr><td>Factor</td><td/><td>26,31</td><td>102</td><td>120</td><td>28</td><td>8</td></tr><tr><td colspan="3">* represents background change.</td><td/><td/><td/></tr></table>

was proposed byKingma & Welling (2013) andRezende et al. (2014) to implement the efficient learning and inference in directed probabilistic models regarding continuous latent variables with intractable posterior distributions and in scalable datasets. They introduced a network inference/recoginition model to represent the approximate posterior distribution and utilized reparameterization trick for stochastic joint optimization of a variational lower bound containing the parameters of both the generative/decoder and inference/recoginition/encoder models.After being raised, many VAE variations have been proposed to boost VAE's capabilities in generation quality and/or disentanglement of the learned representation. In these methods, multiple efforts were made by improving the generative and inference network structures. Typical works along this line include the convolution/de-convolution structure raised byKulkarni et al. (2015) and ladder structure raised byZhao et al. (

Table content: <table><tr><td>D.5 NETWORK STRUCTURE</td><td/><td/><td/></tr><tr><td>Dataset</td><td>Optimiser</td><td/><td>Architecture</td></tr><tr><td/><td>Adam</td><td>Input</td><td>28x28x1</td></tr><tr><td/><td>1e -3</td><td>Encoder</td><td>Conv 32x4x4,32x4x4 (stride 2).</td></tr><tr><td>Mnist</td><td>Epoch 200</td><td>Latents</td><td>FC 256. ReLU activation. 128</td></tr><tr><td/><td/><td>Decoder</td><td>FC 256. Linear. Deconv reverse of encoder.</td></tr><tr><td/><td/><td/><td>ReLU activation.</td></tr><tr><td/><td>Adam</td><td>Input</td><td>64x64x3</td></tr><tr><td/><td>1e -4</td><td colspan="2">Encoder Conv 32x4x4,32x4x4,64x4x4,64x4x4 (stride 2).</td></tr><tr><td>CelebA</td><td>Epoch 20</td><td>Latents</td><td>FC 256. ReLU activation. 128/32</td></tr><tr><td/><td/><td>Decoder</td><td>FC 256. Linear. Deconv reverse of encoder.</td></tr><tr><td/><td/><td/><td>ReLU activation.</td></tr><tr><td/><td>Adam</td><td>Input</td><td>192x168x1</td></tr><tr><td/><td>1e -4</td><td colspan="2">Encoder Conv 32x4x4,32x4x4,64x4x4,64x4x4 (stride 2).</td></tr><tr><td>Extended Yale Face B</td><td>Epoch 2002</td><td>Latents</td><td>FC 256. ReLU activation. 128</td></tr><tr><td/><td/><td>Decoder</td><td>FC 256. Linear. Deconv reverse of encoder.</td></tr><tr><td/><td/><td/><td>ReLU activation.</td></tr><tr><td/><td>Adam</td><td>Input</td><td>192x168x1</td></tr><tr><td/><td>1e -4</td><td colspan="2">Encoder Conv 32x4x4,32x4x4,64x4x4,64x4x4 (stride 2).</td></tr><tr><td>Extended Yale Face B</td><td>Epoch 1460</td><td>Latents</td><td>FC 256. ReLU activation. 128</td></tr><tr><td/><td/><td>Decoder</td><td>FC 256. Linear. Deconv reverse of encoder.</td></tr><tr><td>(Network Parameterized Noise)</td><td/><td/><td>ReLU activation.</td></tr></table>

Peer review (iclr2018):
 This paper studies the importance of the noise modelling in Gaussian VAE. The original Gaussian VAE proposes to use the inference network for the noise that takes latent variables as inputs and outputs the variances, but most of the existing works on Gaussian VAE just use fixed noise probably because the inference network is hard to train. In this paper, instead of using the fixed noise or inference network for the noise, the authors proposed to train the noise using Empirical-Bayes like fashion. The algorithm to train noise level for the single Gaussian decoder and mixture of Gaussian decoder is presented, and the experiments show that fitting the noise actually improves the ELBO and enhances the ability to disentangle latent factors. I appreciate the importance of noise modeling, but not sure if the presented algorithm is a right way to do it. The proposed algorithm assumes the Gaussian likelihood with homoscedastic noise, but this is not the case for many real-world data (MNIST and Color images are usually modelled with Bernoulli likelihood). The update equations for noises rely on the simple model structure, and this may not hold for the arbitrary complex likelihood (or implicit likelihood case). In my personal opinion, making the inference network for the noise to be trainable would be more principled way of solving the problem. The paper is too long (30 pages) and dense, so it is very hard to read and understand the whole stuff. Remember that the ‘recommended’ page limit is 8 pages. The proposed algorithm was not compared to the generative models other than the basic VAE or beta-VAE.