DynamicVAE: Decoupling Reconstruction Error and Disentangled Representation Learning
Authors: Huajie Shao
Haohong Lin
Qinmin Yang
Shuochao Yao
Han Zhao
Tarek Abdelzaher
Venue:  (2021)
Keywords: disentangled representation learning
dynamic learning
Variational Autoencoder
PID contoller

Abstract: This paper challenges the common assumption that the weight $\beta$, in $\beta$-VAE, should be larger than $1$ in order to effectively disentangle latent factors. We demonstrate that $\beta$-VAE, with $\beta < 1$, can not only attain good disentanglement but also significantly improve reconstruction accuracy via dynamic control. The paper \textit{removes the inherent trade-off} between reconstruction accuracy and disentanglement for $\beta$-VAE. Existing methods, such as $\beta$-VAE and FactorVAE, assign a large weight to the KL-divergence term in the objective function, leading to high reconstruction errors for the sake of better disentanglement. To mitigate this problem, a ControlVAE has recently been developed that dynamically tunes the KL-divergence weight in an attempt to \textit{control the trade-off} to more a favorable point. However, ControlVAE fails to eliminate the conflict between the need for a large $\beta$ (for disentanglement) and the need for a small $\beta$ (for smaller reconstruction error). Instead, we propose DynamicVAE that maintains a different $\beta$ at different stages of training, thereby \textit{decoupling disentanglement and reconstruction accuracy}. In order to evolve the weight, $\beta$, along a trajectory that enables such decoupling, DynamicVAE leverages a modified incremental PI (proportional-integral) controller, a variant of proportional-integral-derivative controller (PID) algorithm, and employs a moving average as well as a hybrid annealing method to evolve the value of KL-divergence smoothly in a tightly controlled fashion. We theoretically prove the stability of the proposed approach. Evaluation results on three benchmark datasets demonstrate that DynamicVAE significantly improves the reconstruction accuracy while achieving disentanglement comparable to the best of existing methods. The results verify that our method can separate disentangled representation learning and reconstruction, removing the inherent tension between the two. 

Under review as a conference paper at ICLR 2021
DYNAMICVAE: DECOUPLING RECONSTRUCTION ER- ROR AND DISENTANGLED REPRESENTATION LEARN- ING
Anonymous authors Paper under double-biee blind review
ABSTRACT
This paper challenges the common assumption that the weight B, in 8- V-V, should be larger than in order to effectively disentangle latent factors. We demonstrate that B- VAE, with 3 < 1, can not only attain good disentanglement but also signifi- cantly improve reconstruction accuracy via dynamic control. The paper removes the inherent trade-off between reconstruction accuracy and disentanglement for 3 3-VAE. Existing methods, such as 3-VAE and FactorVAR, assign large weight the KL-divergence term in the objective function, leading to high reconstruction errors for the sake of better disentanglement, To mitigate this problem, a Con- trolVAE. has recently been developed that dynamically tunes the KL-divergence weight in an attempt to control the trade-off off more favorable point. However, Con fails to eliminate the conflict between the need for large 3 (for disentanglement) and the need for small (for smaller reconstruction error). Instead, we propose Dynamic VAF that maintains different (3 at different stages training, thereby decoupling disentanglement and reconstruction accuracy. In order to evolve the weight, 3, along trajectory that enables such decoupling, Dy- namic leverages a modified incremental controller (PID) algorithm, and employs
pnrpportinalliinteraal controller, a
variant of proportional-integr integral-derivativn a moving average as well as a hybrid annealing method to evolve the value of KL-divergence smoothly in tightly controlled fashion. We theoretically prove the stability of the proposed approach. Evaluation results on three benchmark datasets demonstrate that DynamicVAn VAE significantly improves the reconstruction accuracy while achieving disentanglement comparable to the best of cxisting methods. The results verify that our method can separate disentangled representation learning and reconstruction, removing the inherent tension between the two.
INTRODUCTION
The goal of disentangled representation learning is to encode input data into a low-dimensional space that preserves information about the salient factors of variation, that each dimension of the
representation corresponds to a distinct factor in the data (Bengio et al. 2013 Locatello et al. 2020
van Steenkiste et al.) 2019). Learning disentangled representations benefits variety of downstream
tasks (Higgins et al. 2018 Lake et al.) 2017 Locatello et al. 2019c a Denton et al. 2017 Mathieu
al. 2019), including abstract visual reasoning (van Steenkiste al. 2019), zero-shot transfer
learning (Burgess et al. 2018 Lake et al. 2017 Higgins et al. 201 7a) and image generation (Nie) al. 2020). just name few. Due its central importance in various downstream applications. there 1s abundant literature on learning disentangled representations. Roughly speaking, there are two lines of methods towards this goal. The first category includes supervised methods (Chen Batmanghelich, 2019 Locatello et al.] 2019c (Shu et al. 2019 [Bouchacourt et al.) 2018 Nie et al. 2020 Yang et al. 2015), where external supervision (e.g.. data generative factors) is available during training to guide the learning of disentangled representations, The second line of works focus on unsupervised methods (Chen et al. [016 2018 Burgess et al. 2018: Kim & Mnih 2018 Denton ct al. 2017 Kumar et al. [0188] Fraccaro al. (017), which substantially relieve the needs have external supervisions. For this reason, in this paper, we mainly focus on unsupervised disentangled representation learning.

---

Under review as a conference paper at ICLR 2021
One major challenge of unsupervised disentanglement learning is that there exists trade-off between reconstruction quality of the input signal and the degree of disentanglement in the latent represen- tations. Let us take 3-VAE and its variants (Burgess et al.) 2018 Chen et al., 2018 Higgins et al.) 2017a) as an example. These methods assign a large and fixed weight /3 in the objective function to improve the disentanglement at the cost of reconstruction quality, which is highly correlated with accuracy in downstream tasks (van Steenkiste et al., 22119] Locatello et al. 2020). In order to improve the reconstruction quality, researchers have proposed dynamic learning approach, Con- troI (Shao et al., 2020), to dynamically adjust the weight on the KL term in the VAE objective to better balance the quality of disentangled representation learning and reconstruction error. However, while Control VAE allows better control of the trade-off between disentangled representation learning and reconstruction error, it does not eliminate it. One still achieved at the expense of the other. The contribution of this paper, compared to the above state of the art, lies in demonstrating that with the proper design, the trade-off between disentangled representation learning and reconstruction error is completely eliminated. Both objectives can be attained at the same time in decoupled fashion, without affecting each other. More specifically, we observe that if B was kept high in the beginning of training then lowered later in the process, the objectives are decoupled allowing each to be independently optimized. To the authors' knowledge, this work is the first to attain such decoupled optimization of both quality of disentanglement and reconstruction error.
Our Contributions: In this paper, we propose a novel unsupervised disentangled representation learning method, dubbed as DynamicVAF, that turns the weight of 3-VAR (3 1) (Burgess et al.) 2018, Higgins et al. 2017a) into a small value (B < 1) to achieve not only good disentanglement but also a high reconstruction accuracy via dynamic control. We summarize the main contributions of this paper as follows.
We propose a new model, DynamicVAE, that leverages an incremental PI controller and moving average to evolve the desired KL-divergence along trajectory that enables decoupling of two objectives: high-quality disentanglement and low reconstruction error.
We provide the theoretical conditions parameters of the PI controller guarantee stability of DynamicVAF.
We experimentally demonstrate that approach turns the weight of B-VAE (3 > 1) to B S 1, achieving higher reconstruction quality yet comparable disentanglement compared to prior approaches (e.g., FactorVAB). Thus, our results verify that the proposed method indeed decouples disentanglement and reconstruction accuracy without hurting each other's 'ss performance.
2 PRELIMINARIES
3-VAE and its Variants: 3- V-V (Higgins al. 2017b Chen et al. method for learning disentangled representations of the data generative factors (Bengio et al., (2033).
2018) a popular unsupervised
Compared to the original VAE, B-E incorporates an extra hyperparameter 3(3 > 1) as the weight of the KI. term in the VAE objective:
C = E (x (e 3DKz p(z)).
(1)
In order to discover more disentangled factors, in other variants, practitioners further add a constraint on the total information capacity, C, to control the capacity of the latent channels (Burgess al. 2018) to transmit information. The constraint can be formulated as an optimization method:
L3 = Eg (xlx) [log (x|z)] - -
-C,.
(2)
where / is a large and fixed hyperparameter. As result, when the weight , is large (e.g. 100), the algorithm tends to optimize the second term in (2) leading to much higher reconstruction error.
PID Control Algorithm: The PID is a simple yet effective control algorithm that can stabilize system output to a desired value via feedback control (Stooke et al. 2020 Aström et al. 2006).. The PID algorithm calculates an error, e(t), between set point (in this case, the desired KL-divergencee) and the current value of the controlled variable (in this case, the actual KL-divergence), then applies a correction in a direction that reduces that error. The correction is the weighted sum of three terms, one proportional to the error (called P), one that is the integral of error (called I), and one that is the derivative of error (called D); thus, the term PID. The derivative term is not recommended for noisy systems, such ours, reducing the algorithm Pl control. The canonical form of PI controller
2

---

Under review as a conference paper at ICLR 2021
(applied to control B(t)) is the following:
B(t) - k,e()) + K; Σ e(i)
(3)
,=0
where ((t) is the output of controller, which (in our case) is the used 3 during training at time i; e(() is the error between the output value and the desired value time i; Kp, K; denote the coefficients for the P term and I term, respectively. Eq. (3) may be rewritten in incremental form, as follows:
(t)) = (t) B(t - 1),
(4)
where ((0) can be set as needed (as we show later), and:
AB (t( e(l 1)] K₁e(!).
(5)
This paper adopts nonlinear incremental form of the PI controller, described later in Section 3
3 THE DYNAMICVAE ALGORITHM
The goal of disentangled representation learning (Burgess et al. 2018) is to maximize the log likelihood and simultaneously stabilize the Ki--divegeence to target value C. can be formulated as the following constrained optimization problem:
max 6.0
E (x/x)) [log pe (x/z)],
s.t. DKL(gs(zix) p[)) = C
(6)
In order achieve good trade-off between disentanglement and reconstruction accuracy, we attempt to design controller to dynamically adjust B(t) in the following VAE objective to stabilize the KL-divergence to the desired value C:
Ld (x)x) h)) [log po x 2)] -
p(z)).
The contribution of Dynamic VAE is to evolve (t)) along a good trajectory to achieve decoupling between disentanglement and reconstruction error.
To reach this goal, we need to address the following two challenges:
1. (()) should dynamically change from large value to small one. Specifically, at the beginning of training, (() should be large enough to disentangle latent factors. After that, 3(()) is required gradually drop small value optimize the reconstruction.
2. (() should not change too fast or oscillates too frequently. When ((t) drops too fast or oscillates, it may cause KL-divergence to grow with large value. Consequently, some latent factors may emerge earlier so that they can potentially be entangled with each other.
In this paper, we propose methods deal with these two challenges, summarized below.
A non-linear incremental PI controller: Fig. [ (a) shows the designed non-linear PI controller that dynamically adjusts the weight 3(t) in the KL term of the 3-VAE, based on the actual KL-divergence, (t). Specifically, it first samples the output KL-divergence, yK (t), at training step f. Then we use the difference e(t) between the sampled KL--divegeence time with the desired value, C, as the feedback to PI controller to tune (().. The corresponding PI algorithm is denoted by
p(t) Kgo(-e(t)) K; e())
(8)
where (-) is a sigmoid function; Kp. K; are positive parame-rrseeters for and term respectively. As mentioned earlier, we need a large (8() in the beginning to control the KL-divergence from small value to a large target value so that the information can be transmitted through the latent channels per data sample. Accordingly, we adopt an incremental form of the PI controller in Eq. (8), and initialize it to a large value:
3(1) = (1( + B(t 1);
where
A (,lo(-e(1)) - o(-e(l 1))] K,e(t).
(10)
and (8(0) is a large initial value. When the PI controller is initialized large value 3(0), it can quickly produce (small) KL-divergennce during initial model training, preventing emergence of entangled factors.
3

---

Under review as a conference paper at ICLR 2021
Ky (-e(t)
Desired KL
C
e(t)
P
B(t)
VAE objective
SGD
(x)(t)
(dDiveggecce
0.8
Σ
g(B())
0.4
+
K
e(i)
0.0
10000
20000 30000
40000
traininng staps
(a) Pl controller
(b) KL. varies with training steps
Figure 1: (a) PI control algorithm. (b) Time response of KL-divergence when 3 = 130.
Moving average: Since our model is trained with mini-batel data, it often contains noise that causes (()) to oscillate. In particular, when ((t) plunges during training, it would cause KI--divereecce to rise too quickly. This may lead multiple latent factors coming together to be entangled. To mitigate this issue, we adopt moving average to smooth the output KL-divergence as the feedback of PI controller below.
v(() = aa-
OU-T (1
Σ (a)K i=t-T
(11)
where os denotes weight and T denotes the window size of past training steps.
Hybrid annealing: Control systems with step (input) function (i.e., those where the set point can change abruptly) often suffer from an overshoot problem. An overshoot is temporary overcompensa- tion, where the controlled variable oscillates around the set point. In our case, it means that the actual KL-divergence may significantly (albeit temporarily) exceed the desired value, when set point is abruptly changed. This effect would cause some latent factors to come out earlier than expected, and be entangled, thereby producing poor-quality disentanglement. To address this problem, hybrid annealing method that changes the set point more gradually, as illustrated in Fig. [ijn Appendix. It combines step function with ramp function to smoothly increase the target KL-divergence in order to prevent overshoot and thus better disentangle latent factors one by one.
develop
The combination of the above three methods allows SyynamicVAF evolve B()) along favorable trajectory to separate disentanglement learning and reconstruction optimization. We summarize the proposed incremental algorithm in Algorithm in Appendix B
3.1 STABILITY ANALYSIS DYNAMICVAE
We further theoretically analyze the stability of the proposed DynamicVAF. This work is the first to offer the necessary and sufficient conditions that control hyperparameters should satisfy order to guarantee the stability of KL-divergence, when 3 is manipulated dynamically during the training process of a (variant of) 3-VAE.
To this end, our first step is to build the state space model for our control system. Throughout the paper, the state variable training step /. defined as xx()) 3()). Accordingly, the model of incremental PI controller can be written as:
xxt + 1) rrt) =
1))1 -1,())))))
(12)
where error e((), as shown in Fig. [a). )). is given by e(l) = - 1). Here y(() is dynamic model about the time response of the output KL divergence, (K) According to Liu & Theodorou (2019), stochastic gradient descent (SGD) for optimizing an objective function can be described by first-order dynamic model. Our experiment, illustrated in Fig. [ig. also shows that the response (t) in the open loop system approximately meets negative exponential function, which further verifies that our system is first-order dynamic system. We hence use the first-order dynamic model to describe it below.
dy + = ag(x).
(13)
di
where is positive hyperparameter to describe the dynamic property, and g(x) is a mapping function between the actual KI Ki-ddieegeecce and B()). Since Dynamic VAE is discrete control system with
4

---

Under review as a conference paper at ICLR 2021
sampling period Т, 1, the above first-order dynamic model can be reformulated
y(t) y(t 1) + ay(t) = ag(x(tt) = v(t) = 1 + y(t - 1) 1 g(x(t)).
(14)
Now let (t) =(t),
=t) =2())
y(t 1), rz) y(t -2), then Eqs. 12 and
14 can be rewritten
as the following state space equations.
(1(++ 1) = (t) K.IC =2 K, о) (az((1) 0) ((0=(()=200)as(())
1x(t+1)==
a -+ 1+a =2(1) - 20))
1(++1)==r(() fa(m (1)) aa))
(1S)
In order to analyze the stability of the above non-linear state space model, one commonly used method is to linearize it at equilibrium point (Hughes 2015). In this paper, use the following equilibrium point:
(x x5) (C),
(16)
where g (-) denotes the inverse function and x2 zz. Next, we apply the first-order Taylor expansion to the above Eq. 15 yielding
x
(17)
where
x(t) -0 (x1 xÿ, x2)
12) 25, at( 231
(18)
and A is the Jacobian matrix equilibrium point x", defined in Eq. 20) in Appendix C After this linearization, we can prove the stability of the proposed method as the modulus of eigenvalue À of A smaller than 1, described in the following theorem.
Theorem 3.1. Let
0 and assume g'(x) O, Vo > 0. Then DynamicVAE is stable at the equilibrium point C' if and only if the parameters the PI controller, K, and Kp. satisfy the following conditions
4(1 a)
K₂ + K2
(19)
0.5K (zt)² 2- 185-2(15 K 8K1(1 a))g + +81 +
K; 0, K,
We provide the detailed proof in Appendix
Remark 3.1. The assumption of g (æ) 0, V/x basically asks that the KL in the objective be monotonously decreasing function of the coefficient 8(t), and we also further empirically corroborate its validity on two benchmark datasets as shown in Appendix C.I In addition, we choose K, and K; that meet the above conditions 19 to verify the stability of Dynamic VAE in Appendix C.I
4 EXPERIMENTS
In this section we evaluate the performance of Dynamic and compare it against existing baselines. including ControlVAF (Shao et al. 2020 V-E (Higgins et al.) 2017b). 3-VAEB (Burgess al., 22018), FactorVAE (Kim & Mnih 2018), and VAE (Kingma & Welling 2013). We conduct experiments on three benchmark datasets: dSprites (Burgess et al. 2018), MNIST (Chen et al.) 2016))
and 3D Chairs (Aubry et al. 2014). The detailed model configurations and hyperparameter settings are presented in Appendix A Source code will be publicly available upon publication.
4.1 RESULTS AND ANALYSIS
Dsprites Dataset: We first evaluate the performance of DynamicVAE on learning disentangled representations using dSprites. Fig. (a) and (b) illustrate the comparison of reconstruction error and the hyperparameter (tt) (using random seeds) for different approaches. We can observe from (KL=20) has much lower reconstruction error (about 11.8) than 3-VAE and FactorVAF and is comparable to the basic VAE and ControlVAE. This is because DynamicVAE dynamically adjusts the weight, (()), balance the disentanglement and reconstruction. Specifically,
Fig. 2 (a) that Dynamic VAE VAE
5

---

Under review as a conference paper at ICLR 2021
Recordtruztion Logs
120
140
PVAA
DynamicVE BVASS
Hyperparomeen g00)
10a
DynamevnE
BIVAE)
13
18
81
Ladoratiion
100
VAE
XL Divergenc
12
80
10a
60
20
10a
0K
250K
500K 750K 1000K
1250K
OK
250K
500K 750K 1000K 1250K
0K
200K
400K
600K
80OK 1000K
Training stops
Training steps
Training steps
(a) Reconstruction loss
(b) 3(1)
(c) Disentangled factors
Figure 2: (a) shows the comparison of reconstruction error on dSprites using 5 random seeds. DynamicVAB (KI,=20) has comparable reconstruction errors as the basic VAE. (b) shows that DynamicVAB turns the weight of 3-VAF into small value less than 1. (c) shows an example of Dynamic VAE on disentangling factors as the total KL-divergence increases.
Table 1: RMIG for different methods averaged over random seeds. The higher the better.
Models/Metric
pos.
pos. y
Shape
Scale
Orientation
RMIG
DynamicVAF (KI,=20)
0.7166
0.7179
0.2004
0.6530
0.1024
0.4781 + 0.0172
ControlVAB (KI=200)
0.6802
0.6597
0.0956
0.6040
0.1081
0.4295 + 0.0865
FactorVAE (7 = =10)
0.7482
0.7276
0.1383
0.6262
0.1412
0.4763 1 0.0513
3-VAE, (7 = 100)
0.5666
0.5763
0.4353
0.3814
0.0631
0,4045 1 0.0345
3-VAE, (B 4)
0.1635
0.1047
0.1391
0.3958
0.0127
0.1632 1 0.0626
VAE
0.0359
0.0243
0.0116
0.1507
0.0039
0.0452 + 0.0326
DynamicVAI automatically assigns a large B(4) the beginning of training in order to obtain good disentanglement, and then its weight gradually drops to less than 1 at the end of optimization, as shown Fig. 2 (b). contrast, 3-VAE and FactorVAE have large and fixed weight in the objective so that their optimization algorithms tend to optimize the KI--divergence term (total correlation term for Factor VAE), leading to higher reconstruction error. For ControlVAE, it can also dynamically tune 3(t) to control the value of KL-divergence, but its disentanglement performance degrades with the increase of KL-divergence (i.e., decrease of weight) as illustrated in Table [ In addition, Fig. ([c) illustrates an example of per-factor KL-divergence in the latent code as the total information capacity (KL-divergence) increases from 0.5 to 20. We can see that Dynamic VAE disentangles all the five data generative factors, starting from position (z and y) to scale, followed by orientation and shape.
Next, we use a robust disentanglement metric, robust mutual information gap (RMIG) (Do & Tran 2020), to evaluate the disentanglement of different methods. We can from Table that JynamicVAF VAE has a comparable RMIG score to the VAE, but it has much lower reconstruction error illustrated in Fig. Moreover, Dynamic VAB has higher RMIG score but lower reconstruction error than 3-VAE models. We also find that our method achieves much better disentanglement than ControlVAE for comparable reconstruction accuracy. Hence, Dynamic VAE is able improve the reconstruction quality yet obtain good disentanglement.
Qualitatively, we also visualize the disentanglement results of diffeernnt models in Fig. 3 We can observe that DynamicVAE disentangles all the five generative factors on dSprites. However, ControlVAE is not very effective to disentangle all the factors when its KL-divergence is set to large value, such as 20. Furthermore, 88 BAE (7 100) disentangles four generative factors and mistakenly combines the scale and shape factors together (in the third row). The other methods do not perform well for disentanglement.
MNIST and 3D Chair Datasets We also evaluate the proposed method on the other two datasets: MNIST and 3D Chairs. Fig. iillustratss some samples of the disentangled factors for Dynamic VAE on MNIST. We also verify that our method achieves better disentanglement compared with the other methods, shown in Appendix D In addition, method with 3 < 1 can significantly improve the reconstruction accuracy than VAE as illustrated in Fig. 10 Appendix [
We also demonstrate that DynamicVAl VAE can learn many different data generative factors another challenging dataset, 3D Chairs. We can observe from Fig. 1htat our method disentangles six different latent factors, such wheels, and leg height and azimuth, same as Factor VAE (Kim & Mnih 2018).

---

Under review as a conference paper at ICLR 2021
DynamicVAE (KL-20)
Con (KL=20)
Basic VAE
dhoce oten
VactorV/AE ty 10)
B-VAE, IY 100)
B-VAE BB 4)
Chape CRETE COO
Figure 3: Rows: latent traversals ordered by the value KL-divergence with the prior in a descending
order. We initialize the latent representation from seed image, and then traverse single latent code in range of 3, 3), while keeping the remaining latent code fixed.
O
0
O
O
0
0
0
O
4
4
4 4
4
4 4
4
4
4
4
4
4
4
4
4
4
4
4
4
4 4
4
4
7 7
7
7
7
7
7
7
7
7
7
f 8 9 9
9
9
9
9 9 9 9
9
9
9
9
9
9
9
9
(a) Rotation
(b) Thickness
(c) Style from left to right
Figure 4: Latent traversals MNIST for DynamicVAE can be seen that our method can disentangle four different factors: rotation, thickness, size(width) and writing style.
Decoupled Reconstruction and Disentanglement Additionally, we show that the proposed Dy- namic VAE is able to decouple the reconstruction and disentanglement learning into two phases, overcoming the problem balancing the trade-off between reconstruction and disentanglement Fig. 6 illlsstrates the RMIG score and reconstruction loss with the increase of training steps after all the factors are disentangled (before 800, 000). It can be seen that RMIG score of our method remains stable as the reconstruction loss drops. Therefore, the proposed method does not introduce any conflict between reconstruction optimization and disentangled representation learning.
DynamicVAE
15.00
0.80
Recon loss
H
d
d
wheels
A
d
d
Reconstrudion
leg height
H
H
12.50
0.60
nzimuth
back langth
d
A
Piskness
body
1000K
Training steps
1200K
1400K
1600K
Figure 5: Sample traversals for the six latent fac- Figure 6: Averaged RMIG score and reconstruc- tors in our model on 3D Chairs.
tion loss vary with training steps.
4.2 ABLATION STUDIES
To compare the performance of DynamicVAE and its variants, we perform following ablation studies:
Dynamic VAE-P: it uses positional PI controller with initialization to large 3(0), instead of the incremental PI initialized to large 3(0), to tune the weight on KL term in the VAE objective. DynamicVAE--eee solely adopts step function without ramp function for our annealing method. VinamicVAE-4 this model directly uses the output KL-divergence at time as a feedback PI controller without using moving average smooth it.

---

Under review as a conference paper at ICLR 2021
Table 2: RMIG for different methods averaged over 5 random seeds. The higher is better.
Models/Metric
pos.
pos. y
Shape
Scale
Orientation
RMIG
Dynamic VAE
0.7166
0.7179
0.2004
0.6530
0.1024
0.4781 0.0172
DynamicVAEI
0.7376
0.7317
0.0992
0.6400
0.1120
0.4641 0.0240
Dynamic VAE-step
0.7209
0.7143
0,0664
0.6218
0.1543
0,4555 0.0355
Dynamic VAE-t
0.7152
0.7110
0.0997
0.6267
0.1322
0,4570 + 0,0182
Table shows the comparison of RMIG score for Dynamic VAE and its variants. It can be observed that Dynamic outperforms the other methods in terms overall RMIG score. We also find that VAE-step does not perform well because the ramp function is removed from our annealing method, leading to overshoot of PI controller. As a result, it makes the other factors come out earlier and entangled each other. Thus, annealing method. addition, we can see that the proposed moving average and incremental PI control algorithm also play critical role to improve the disentanglement.
conclude the importance of adding ramp function for our
RELATED WORK
Disentangled representation learning can be divided into main categories: unsupervised learning and supervised learning.
Supervised disentanglement learning (Mathieu et al. 2014, Reed et al.) 2014) requires the prior knowledge of some data generative factors from human annotation train the model. Some studies (Locatello al. 2019a b c) figure out that it is hard to achieve reliable and good disentanglement without supervision. For supervised learning, the limited labeling information can help ensure a latent space of the VAE with desirable structure w.r.t to the ground-truth latent factors. In order to reduce human annotations, researchers tried to develop weakly supervised learning (Bouchacourt al,, 2018 Hosoya 2019 Locatello et al., 2020) to learn disentangled representations. However, these methods still require explicit human labeling or assume the change of the two observations is small. In practice, is unrealistic for initial learners to discover the data generative factors in most real world scenarios.
2016 Siddharth al. 2017 Kingma al.
For unsupervised learning methods, the recent approaches mainly build on Variational Autoencoders (VAEs) (Kingma & Welling 2013) and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014). InfoGAN (Lin et al. 2020) is the first scalable unsupervised learning method for disentan- gling. It, however, suffers from training instabilities and does not perform well in disentanglement learning (Higgins et af. 2018). so most recent works are largely based on VAEs models. The VAE models, such as 3-VAE (3 1), FactorVAE and 3-TCVAE (Chen et al. 2018), often suffer from high reconstruction errors in order to obtain better disentanglement, since they add a large weight to terms in the objective. To address this issue, [Shao al. (2020) develops controllable variational autoencoder (ControlVAE) to dynamically tune 3 to achieve the trade-off between reconstruction quality and disentanglement. However, ControlVAr VAE fails to decouple disentanglement and recon- struction accuracy because the computed & may oscillate during training, since it uses step function as annealing method and frequently adjusts 3 at each step. In this paper, we propose a new method, Dynamic VAE, that can separate disentangled representation learning and reconstruction.
6 CONCLUSION
This paper proposed a novel dynamic learning method, DynamicVAE, to address the trade-off problem between reconstruction and disentanglement. Our method is able to turn the weight of 3-VAE to small value (3 < 1) to achieve good disentanglement against the previous default consumption, 3 > 1. Specifically, we leveraged an incremental PI controller, moving average and a hybrid annealing to stabilize the KL-divergence decouple the disentanglement and reconstruction. We further theoretically prove the stability of the proposed Dynamic VAF. The evaluation results demonstrate Dynamic VAE can significantly improve the reconstruction accuracy meanwhile attaining good disentanglement. It decouples disentanglement and reconstruction accuracy without introducing conflict between them.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Karl Johan Aström, Tore Hägglund, and Karl Astrom. Advanced PID control, volume 461. ISA-The Instrumentation, Systems, and Automation Society Research Triangle .... 2006.
Mathieu Aubry, Daniel Maturana, Alexei A Efros, Bryan C Russell, and Josef Sivic. Seeing 3d chairs: exemplar part-based 2d-3d alignment using large dataset of cad models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3762-3769, 2014.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learningg A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-188 2013.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Thirty-Second AAAI Confer- ence on Artificial Intelligence, 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint 18K4::8804.0399, 2018.
Junxiang Chen and Kayhan Batmanghelich. Weakly supervised disentanglement by pairwise similar- ities. arXiv preprint arXiv: 1906.0/044, 2019.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Πla Sutskever, and Picter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp- 2172-2180, 2016.
Emily L Denton al. Unsupervised learning of disentangled representations from video. In Advances in neural information processing systems, pp. 4414-4423, 2017.
Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representations. In Proceedings of ICLR, 2020.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. In Advances in Neural Information Processing Systems, pp. 3601-3610, 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- 2014.
fion processing systems, pp.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. Proceedings of ICLR, 2017a.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR, 2(5))6, 2017b.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 1812 02.02200 2018.
Haruo Hosoya. Group-based learning of disentangled representations with generalizability for novel contents. In IJCAI, pp. 2506-2513, 2019.
Jonathan L Hughes. Applications of stability analysis to nonlinear discrete dynamical systems modeling interactions. 2015.

---

Under review as a conference paper at ICLR 2021
IS Isa, BCC Meng, Z Saad, and NA Fauzi. Comparative study of pid controlled modes automatic water level measurement system. In 2011 IEEE 7th International Colloquium on Signal Processing and its Applications, pp. 237-242. IEEE, 2011.
Eliahu Ibraham Jury. Theory and application of the z-transform method. 1964.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2654-2663, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114 14, 2013.
arXiv preprint
Durk P Kingma, Shakir Mohamed, Danilo Jimenez. Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference Learning Representations, 2018.
Brenden M Lake, Tomer D Ullman, Joshua Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Zinan Lin, Kiran K Thekumparampil, Giulia Fanti, and Sewoong Oh. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In international conference machine learning, 2020.
Guan-Horg Liu and Evangelos A Theodorou. Deep learning theory review: An optimal control and dynamical systems perspective. arXiv preprint 1rXiv:1908.100 10920, 2019.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairnsss of disentangled representations. In Advances in Neural Information Processing Systems, pp. 1461 146114, 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of dis- entangled representations. In international conference on machine learning, pp. 4114-4124, 2019b.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations, 2019c.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. arXiv preprint arXiv:2002.02886,, 2020.
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. International Conference on Machine Learning, pp. 4402-441 1402-4412, 2019.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In Advances in neural information processing systems, pp. 5040-5044 2016.
Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debhath, Anjul Patney, Ankit B Patel, and Anima Anandkumar. Semi-supervised stylegan for disentanglement learning. Proceedings of ICML, pp. arXiv-2003, 2020.
Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee. Learning disentangle factors of variation with manifold interaction. International Conference on Machine Learning, pp. 1431- 1439, 2014.
Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. Controlvae: Controllable variational autoencoder. Proceedings of the 37th International Conference Machine Learning (ICMI.), 2020.
10

---

Under review as a conference paper at ICLR 2021
Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. arXiv preprint arXiv:1910.09972,.000 2019.
Narayanaswamy Siddharth, Brooks Paige, Jan-Willem Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In Advances in Neural Information Processing Systems, pp- 5925-5935, 2017.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. arXiv preprini urXiv:2007.33664 2020.
Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? In Advances in Neural Information Processing Systems, pp. 14245-14258, 2019.
Jimei Yang. Scott E Reed, Ming-Hsuaa Yang, and Honglak Lee. Weakly-supervised disentangling with recurrent transformations for 3d view synthesis. In Advances in Neural Information Processing Systems, pp. 1099-1107, 2015.
Jerzy Zabezyk. Mathematical control theory. An Introduction, 1992.

---

Under review as a conference paper at ICLR 2021
A MODEL CONHIGURATIONS AND HYPERPARAMETER SETTINGS
We summarize the detailed model configurations and hyperparameter settings for Dynamic VAE below.
Following the same model architecture of 3-VAE, we adopt convolutional layer and deconvolutional layer for our experiments. We use Adam optimizer with 31 = 0.90, 82 = 0.99 and a learning rate tuned from 10 We set Kp and K; for Pl algorithm to 0.01 and 0.005, respectively. The weight (8() for incremental Pl controller is initialized with 150, 100 and 50 for dSprites. MNIST and 3D Chairs, respectively. The batch size is set to 128. Using the similar methodology in (Burgess al. 2018).
we train a single model by gradually increasing KL-divergence from 0.5 to a desired value C' with a step function s and ramp function for every M training steps, as shown in Fig. In the experiment. we set the step, to 0.15 per M = 6, 000 training steps (including 5, 000 in step function and 1, 000 in ramp function) as the information capacity (desired KL- divergence) increases from 0.5 until to 20, 26 and 18 for dSprites, MNIST and 3D Chairs datasets respectively. In addition, the window size = 5 with equal weight a. Our model adopts the same encoder and decoder architecture as 3- and Control VAE except for plugging in PI control algorithm, as illustrated in Table 3 and Table 4
of moving average
Table 3: Encoder and decoder architecture for disentangled representation learning on dSprites and MNIST.
Encoder
Decoder
Input 6-1 64 binary image
Inpot g
4 conv, 32 ReL.U. stride
FC. 256 ReLU.
4 conv. 32 ReL.U. stride
upconv. 256 ReLUU. stride
x 4 conv, 64 ReLU, stride
4 x upconv, 64 ReLU,, stride
4 x 4 conv, G1 ReLU, stride
4 x 4 upсonv, 6-1 strde 2
x 4 conv. 256 ReLUU stride I
4 x 4 upconi: 32 ReLU, stride 2
FC 256. FC. 2 x 10
4 x upeanv: 32 ReLU. stride 2
Table 4: Encoder and decoder architecture for disentangled representation learning on 3D Chairs.
Encoder
Decoder
Input 64 x 64 x
Input E 16
4 4 cony. 32 ReLU, stride
FC, 256 ReLU.
x 4 conv, 32 ReLU, stride
4 x upcons: 256 ReLU, stride 2
4 conv, 64 ReI.U. stride
4 upcone: 64 ReI.U. stride
4 conv. 64 ReLUU stride
upcune: ReLU' stride
x 4 conv, 256 ReLU, stride
4 x upconv, 32 ReLU, stride 2
FC 256, FC, 2 x 10
4 x upсoιv. 32 ReLU, stride 2
A.1 PI PARAMETER TUNING AND SET POINT GUIDELINES
We can tune PI parameters by following the conditions that guarantee the stability of the proposed method in Eq. (9) in Appendix C In addition, (0)) is initialized to sufficiently large value in order to guarantee the KL-divergence is closed to zero at the beginning of model training. On the other hand, the choice of desired value of KL-divergence (set point) is very simple. Since our method achieves good disentanglement when B = 1, we can set its desired value to equal or lower than the KI KL-diieegeece of the original VAE as it converges.
12

---

Under review as a conference paper at ICLR 2021
Actual KL.
Desired KL
KL Annealing
200K
220K
240K
260K
280K
300K
Dimensional KL
210
200K
220K
240K
260K
280K
300K
Figure 7: Top: A hybrid annealing method that combines step function with ramp function. Bottom: dimwise KL. divergence.
Algorithm 1 Incremental PI Control
1: Input: desired KL C, cocfficients Kp, Ki, Bmin. iterations N,, window
10:
A(t - 1) < 8min then
11:
dI(() <- 0 // wind up
2: Output: weight (() training step t
12:
end if
3: Initialization: (()) 150 (100), g)z(()) (0)
13:
d3(t) +- dP(() dT(()
4: for = 1 N
14:
B(t) dī(t) + 5(t 11)
5: Sample KL-divergence, (1)
15:
if (((()) < Bmnn then
6: v(t) Σi-r (1)Kz())
16:
((t) + Bmin
7: e(t) + C - =t(()
17:
end if
8:
dP(t) Kplo(-e(t)) o0- 1))]
18:
Return ((t)
9:
dI(t) Kie(t)
19:
end
A.2 HYBRID ANNEALING METHOD
Fig. 7 shows the hybrid annealing method that combines step function with ramp function gradually increase the desired value of KL--dieergence for DynamicVAB. VAE.
B ALGORITHM
We summarize the incremental PI algorithm in Algorithm
C PROOF OF STABILITY IN THEOREM 3.1
In this section we provide the omitted proof in the main paper about Theorem 3.1 For convenience purpose, we first restate Theorem 3.1 below.
Theorem 3.1. Let a > 0 and assume gz(x) < 0, Vr > 0. Then Dynamic VAE is stable at the equilibrium point C. if and only if the parameters of the PI controller, K; and Kp, satisfy the
13

---

Under review as a conference paper at ICLR 2021
following conditions
4(1 +
Kp + Ki <
ag' (2]
(19)
0.5K (x²)²
2[Kp - -8K, + ())g 8(1 + a)
K, 0, Kp
Proof. At high level, the proof goes by showing the spectral norm of the Jacobian matrix A to be strictly less than I under the given condition, which is both sufficient and necessary for stability. To start with, recall that the Jacobian matrix A at equilibrium point x" is defined by
8f
0f1
0f
Owa
deg
K1 K4
K2
K3
A
0/2
0f2
0/2
K5
0
(20)
dzs
0fa
öfs
0
801
8w2
02g
x==
where
0f1
K1 0r1
(21a)
=1=2]
0f
1
K₂ Ou2
K; + (,0(12 C) o C)] = Kp + Ki
(21b)
m2
4
0f
K3 0E3
K2o
-) -
[)] Kp
(21c)
23=1;
K4 Oe
f
a
g (x1)
(21d)
21
1
0f
1
K₂ 0x2
(21e)
1==2
1+0
fʃ
(21f)
0r3
0
0.3
0f3
Dz
===2;
0.
Ozz 12=0; =
=,
dzz
===2] 0.
(21g)
In order to guarantee the stability of our state space model, the modulus of eigenvalue of A should be smaller than 1, i.e., | < 1. By definition, the eigenvalues of A can be obtained by computing the roots of the following characteristic polynomial:
^ K₁
K22
K3 0
del(XT A)
- K4
A- K5
-1
(22)
- (K1 + Ks) - (KiKs - K225-KK2,,A-- - K3K₁=0
Instead of computing the (complex) roots of the above cubic polynomial analytically, we use the
following bilinear transformation (Jury) 1964) to map the unit circle |X| < 1 to the left half plane
such that its real root less than 0 (Hughes) 2015):
A
A E+1 --1
(23)
Substituting À in Eq 22 with 23 we have
b 2z + big + by = 0,
(24)
where
by = K1 + K3 + K,K3 K5 K
b2 K1 + Kg K - +
(25)
bi - K1 Ks K1 K, K244 433 43
bn Ki Ks K,K5 K2Kg KgK4+1
Clearly, using the above transformation, we know that |\| < iff the real part of is less than 0, i... Re(g) O. In order to ensure Re(g) 0, based Routh- stability criterion (Zabezyk)
14

---

Under review as a conference paper at ICLR 2021
30
Exp fun fitting
Exp fun fitting
25
Data points
25
points
di-divergence
20
(-divergence
20
15
10
5
0
20
40
80
100
25
50
75
100
125
B(t)
B(t)
(a) MNIST dataset
(h) dSprites dataset
Figure 8: (a) g(x(t)) on MNIST dataset. (b) g(x(t)) dSprites dataset.
1992), bo, b1, 62, 6y should satisfy the following sufficient and necessary condition.
bo b 0
(26)
b2 >
b, b, bgbg
Substitute Eqs.
(21). (24g and 26 into the above system of inequalities, yielding
by 4a + (K, + 2ag)ag'
2(1 + a)
bg 4(1 + a) + (Kp + K₁ (a))))))
>
(1+ a)
(27)
b,b2 bodo -0.5K (x1)² -2 2a(K, (1)gg 8a(1 + +a)
0
(1 a)
Kju 1+a" g (xi) > 0
bo
To complete the proof, recall that a > 0 and we assume g'(x) < 0, Va. Hence the coefficients of PI controller, K, and K; Eq. (7). need to satisfy the following conditions.
Kg + 2K, > 4(2 + a)
ag'(x;)
K, + K₁ < 4(1 + a)
(28)
ug' (xi+)
0.5K ag (27)² 2[pp 8K, + +))lg + 8(1 a) >
K; >0
Since Kp 0, K, > 0 in our designed PI control algorithm and g (z(z)) O, further simplify
as
K₁ + K; 4(1 + a)
ag' (x+)
(29)
0.5K3 (x1)²
- [1 +
0
K; 0, 0,K, 00
Therefore, as K₂p and K; meet the above conditions 29 our Dynamic VAE would be stable at the set point, which is verified by the following experiments on different datasets.
15

---

Under review as a conference paper at ICLR 2021
1.0
1.75
1.50
KL Divergence
KL Divergence
1.25
1.00
0.75
0.4
0.2
0.25
0.0
20000
40000 training steps
60000
0
5000
10000 training steps
15000
20000
(a) MNIST when 3 80
(b) dSprites when 3 = 130
Figure 9: Time response of KL-divergence under different B on MNIST and dSprites datasets respectively
C.I VERIFICATION ON BENCHMARK DATASETS
We first verify the validity of assumption that g'(x) 0 in Theorem 3.1 Fig. 8 illustrates the relationship between ((t) and the actual KL when model training converges on dSprites and MNIST datasets. We can observe that the actual output KL-divergence and B(t) have highly negative correlation, which means g' (x) < O.
Next, going to verify the stability of the proposed DynamieVAR MNIST and dSprites datasets. On MNIST dataset, its mapping function g(x) in Fig. 8 (a) can be approximately obtained by curve fitting with the following negative exponential function:
g(x(t)) 26.38 exp(-0.04762(tt)
(30)
And the corresponding derivative is
(x((()) - 1.26 exp( (..066))))))) < -1226.
(31)
In addition, we introduce how to obtain the hyperparameter
our dynamic model in Eq. (14)
Assume that KL-divergence converges certain value C' in the open loop control system during model training, then the dynamic model in Eq. (14) can be rewritten as
y(t) y(t 1) + at()) = aC"
(32)
When y() 0, and the sampling period of system is T, the corresponding solution is given by
y(t) C² exp( at))
(33)
In order to obtain the value of a, one commonly used method in control theory is to set : as we have y(t") = C²(1 exp(-1)) = 0.632C". In this way, we can derive a based on the training steps as KI divdevegee reaches 63.2% of final value C' (Isa et al. 2011) in the experiments, as shown in Fig. 9 For MNIST dataset, we can get the hyperparameter
around based on the
time response of Kl.-divergence in the open loop system, as shown Fig. (a).
Similarly, the derivative of mapping function on dSprites can be approximately expressed by
g' (z(()) = 3.2 exp( - -0.121x(t)) < -3.2.
(34)
In addition, we can get the hyperparameter a 2507 around based on the time response of KL- divergence in the open loop system, shown in Fig. (b).
We summarize the parameters a and g (=(()) for different datasets in the following Table
In this paper, choose Kp 0.01 and K; 0.005 with the parameters in Table to validate model meets the conditions in Eq. (29). In addition, our experimental results in Section further demonstrate that method stabilize the KL-divergence to the points.
16

---

Under review as a conference paper at ICLR 2021
Table 5: Parameters summary for different datasets
Dataset
a
g'(x(1)) min
MNIST
-1.26
5000
dSprites
-3,2
2506
D EXTRA EXPERIMENTS ON MNIST
D.1 EVALUATION ON RECONSTRUCTION QUALITY
Fig. 10 shows the comparison of reconstruction loss and weight 3(1) for different methods. can be observed that DynamicVAE and Control VAE have comparable reconstruction accuracy to the basic VAE, but they have better disentanglement than it, as shown Fig. 4 and 15 addition, can see that Dynamic VAE has better reconstruction quality than the two 8- VAE models.
DynamicVAE
10a
750
P-VAE
Reconstruction Loss
700
ControlVAF
Hyperparameter ((O)
650
FactorVAE
10a
DynamicVAL
VAEE
500
10"
CorEreNAF
450
400
250K
500K 750K 1000K 1250K 1500K
250K 500K 750K 1000K 1250K 1500K
Training steps
Training steps
(a) Reconstruction loss
(b) Weight
Figure 10: Performance comparison of different methods.
D.2 MNIST LATENT TRAVERSALS FOR BASELINES
We present some samples of latent traversals for the baseline methods. We find that DynamicVAF outperforms ControlVAE in term rotation factor as illustrated Fig. and though they have comparable disentanglement score. Moreover, Dynamic VAE has better disentanglement than 3-VAE and FactorVAS VAB in the following figures.
O
O
0
0
O
O
O O
O
0
0
O
O
O
O
O
O 4
4
4
4
4
4
4
4
4
7
4
4
4
4
4
4
& 7
4
4
4
4
4
4
4
4
4
7 7
7
7
7
7
7
7 9
7
7
7
7
7
7
7
7
7
9999
9 9
4
9
9
9
9
9
9
9
9
9
9
(a) Rotation
(b) Thickness
(c) Style from left to right
Figure
Latent traversals on MNIST for ControlVAE.
17

---

Under review as a conference paper at ICLR 2021
O
D
D
O
O
O
4 4
4
4
4
4
L
4 4
4
4
4 4
4
4
4
4
4
4
4
4
4
4
7
7
7
7
9
9
9
9
4
9
9
9
9
(a) Rotation
(b)
Thickness
(c) Style from left to right
Figure 12: Latent traversals on MNIST for FactorVAE.
6
6
0
0
O
O 4
4 4 4 4
4
4
4
4
4
4
4 4
4
4
4
4
4
4
4
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7 7 9
7
9
9
9
9
9 4 9
O
9
9
9
9 9
q 4 4
9
9
9
9
9
9
9
9 9
(a) Rotation
(b)
Thickness
Style from left to right
Figure 13: Latent traversals on MNIST for B-VAE (B 10).
0
D
D
D
0
1 A 4 L L Y 7 7
V
4 4 4
4 4
4
4
4
4
4
4
4
4 4 4
7 7
]
7
7
7
7
7
7
7
7 7
A<
4
4
9
9
9 9 9
9
9
9
9 9
9
9
9
9
9
9
9
9
9 4
(a)
Rotation
(b) Thickness
(c) Style from left to right
Figure 14: Latent traversals MNIST for 3-VAE, (7 100).
O
O
O
4
L
L
4
4
4
4
4
4
4
4 o 4
4
4
4
4
4
7
7
7
7
7
7
7
7
7
9
9
T
7
9
9
9
9
9 9
q a
9
9
9
9
9
9
9
G
(a)
Rotation
(b) Thickness
(c) Style from left to right
Figure 15: Latent traversals on MNIST for the basic VAE.
18


Reviews:
[{"review_id": "xEMTvZTbbyN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Positive points:\n\n- Dynamically controlling the beta parameter of beta VAE is an interesting idea\n\nNegative points:\n\n- 1. The authors do not put their work into the context of the closely related information bottleneck principle\n- 2. The authors do not provide further details on several mechansims of their proposed method:\n  - 2. a) How is the constraint threshold C on the KL divergence term chosen? Does it have to be tuned for every dataset? Can it be tuned at all in the unsupervised setting, when no information about the ground truth factors is available?\n  - 2. a) Why does a constraint on the KL divergence result in a decoupling of reconstruction quality and disentanglement?\n  - 2. b) Is the beta schedule of starting with a large beta value which is then reduced a result of the control algorithm or is it pre-defined by the Hybrid annealing method the authors describe in section 3?\n- 3. Experiments:\n  - 3. a) FactorVAE has not converged yet, and it is the closest competitor.\n  - 3. b) The RMIG scores of FactorVAE, ControlVAE and DynamicVAE reported in Table 1 are not significantly different and therefore the score of DynamicVAE should not be in bold font.\n- 4. Missing references\n\n\n1. Missing references and context to related work:\n\nThe method seems closely related to the information bottleneck method (Tishby et al. 2000, Kirsch et al. 2020): the constraint on the KL divergence term to remain below a fixed value, defines an upper bound on the bandwith of the latent channel. The authors should explore this connection further, or at least should discuss the relation between the proposed method and the information bottleneck method.\n\n(Stuehmer et al. 2020) et al. propose a structured prior that also reduces the trade-off between reconstruction accuracy and disentanglement. It would be interesting to compare the performance of the proposed method to their methood ISA-VAE.\n\n\n2. Method\n\n2. a) How is the constraint threshold C on the KL divergence term chosen? Does it have to be tuned for every dataset? Can it be tuned at all in the unsupervised setting, when now information about the ground truth factors is available?\n\n2. b) Is the beta schedule of starting with a large beta value which is then reduced a result of the control algorithm or is it pre-defined by the Hybrid annealing method the authors describe in section 3?\n\n\n3. Experiments\n\nFig. 2 a) depicts the reconstruction error of the proposed method and different baselines for up to 1250K iterations. Figure 6 depicts the reconstruction loss of the the proposed method for up to 1600K iterations. \n\n3. a) FactorVAE has not converged yet (Fig. 2 a), and it is the closest competitor. Further iterations are recquired to assess if there is a difference in the reconstruction error between the proposed method and FactorVAE.\n\n3. b) The RMIG scores of FactorVAE, ControlVAE and DynamicVAE reported in Table 1 are not significantly different and therefore the score of DynamicVAE should not be in bold font. Are the reported error values standard deviation? This needs to be explained in the caption of the table.\n\n3. c) How does the proposed PI-controller based approach compare to a simpler approach of beta-VAE with a fixed beta schedule: After a pre-defined number of iterations beta goes down to a lower value. Does the dynamic control have any benefits?\n\n3. d) Related to 3 c): Is the schedule for the beta parameter that is generated by the PI-controller significantly different for different datasets?\n\n\nConclusion: Dynamically controlling the beta parameter is an interesting idea but this paper has several shortcomings and is not ready for publication.\n\n\nReferences:\n\nTishby et al., The information bottleneck method, ACCCC 1999\nKirsch et al., Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning, arXiv 2020\nStuehmer et al., Independent Subspace Analysis for Unsupervised Learning of Disentangled Representations, ICML 2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "ly54xXa1OJ1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In $\\beta$-VAE, one challenge is to choose the hyper-parameter $\\beta$ that controls the trade-off between the reconstruction quality and the disentanglement. This paper proposes a method called DynamicVAE. Rather than using a fixed hyperparameter $\\beta$, the method leverages a modified incremental Proportional-integral (PI) controller, which dynamically tunes $\\beta$ at different stages of training. The method is tested on benchmark datasets.\n\nThe paper is not difficult to follow. The idea of dynamically tuning the hyper-parameter looks interesting. However, there exists a previous method called control-VAE, which also dynamically tunes $\\beta$ with a PI controller. Although it is reported in the paper that the proposed method outperforms controlVAE, it is not clear to me how it differs from it. Therefore, I am not sure whether the proposed method has sufficient novelty.\n\nAs shown in Table 1, DynamicVAE outperforms FactorVAE, but the difference is tiny. In particular, Dynamic VAE is better in disentangling shape and scale, while FactorVAE is better in disentangling pos. x, pos. y and orientation. The results make me doubt whether DynamicVAE consistently outperforms FactorVAE in various datasets. It would make the paper more convincing if the authors can report quantitative results on more datasets.\n\nThe authors claim that the proposed method decouples reconstruction and disentanglement. However, no quantitative measurements for reconstruction are reported. Therefore, it is not clear whether the proposed method outperforms the baselines in terms of reconstruction.\n\nIn summary, I do not suggest accepting this paper. I am not convinced that the proposed method has sufficient novelty. More experimental results might be necessary.\n\n==============================================================================================\n\nThanks for the authors' response. I am still inclined to reject this paper. Compared to the existing ControlVAE, the contributions of this paper looks incremental. More experimental results might be necessary to make the paper more convincing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "For3EgHTeU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary \nThe paper proposes DynamicVAE, a variant of ControlVAE that is claimed to decouple the optimization reconstruction error and disentanglement, hence mitigating the inherent trade-off between the two.\n#### Pros:\n- The paper sheds light on the tradeoff problem in disentangled representation learning from an optimization perspective.  \n- They empirically show that their proposed method can in fact decouple the two optimization objectives. \n\n#### Cons:\n- It will be very helpful if the authors can clarify the exact difference between DynamicVAE and ControlVAE. My understanding is that the main difference lies in their usage of Hybrid Annealing, which is certainly important but somewhat incremental given the overall results.\n- My main concern with the paper is that it lacks extensive empirical evaluation.  \n  1. Currently, the baselines used are very week. Please consider using baseline methods such as [1] or others that also tackle the trade-off problem. \n  2. Please consider using more challenging datasets like SmallNorb or Cars (see [2] for options). This is important since dSprites can be fully specified using only the disentangled factors. For these difficult datasets please consider doing a quantitative evaluation. Currently, I only see qualitative evaluation for Chairs (which indeed is a reasonable dataset).\n  3. What values of beta and gamma did you try for the baseline methods? The RMIG numbers reported for beta vae are very low and different from what RMIG paper reports.\n\n#### Comments:\n- It is known that the pixel-wise reconstruction score does not give a clear picture of the sample quality [3]. Please consider using FID (see [4]).\n- I am interested in knowing how the high KL of 20 affects the NLL of the model? Also, how is the sample quality of generated samples?\n\nMy current rating is primarily based on the fact that the paper requires a lot more evaluation with competitive baselines on more relevant datasets. Also, clarifying the difference from ControlVAE will really help.\n\n[1] Overcoming the Disentanglement vs Reconstruction Trade-off via Jacobian Supervision\n[2] Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations\n[3] Generating Diverse High-Fidelity Images with VQ-VAE-2\n[4] Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modelling", "extra_scores": {}, "extra_text": {}}, {"review_id": "621aQGPY-pm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a strategy for controlling the beta value of a beta-VAE during training using approaches from control theory that allow it to target a designated level of KL divergence between the encoder distribution and the prior.  This is done in a way that aims to achieve good reconstructions while maintaining disentangling performance.  It can be viewed as a refinement of the ControlVAE approach of Shao et al 2020, varying only in the specifics of the strategy used.\n\nThough the approach is sensible and empirical results of the work are reasonable, I believe the contribution of the work is too small and too specific for publication at ICLR.  None of the high-level ideas are new, having already been introduced in Shao et al 2020, with the contribution effectively equating to making relatively low-level adjustments to the precise approach of that work.  This makes the content very niche and I believe it will be of limited interest to the ICLR community.   Beyond this, I also have serious concerns about the fact that this is an improvement on a somewhat outdated and discredited approach to disentanglement (the beta-VAE), along with the fact that the paper suffers from very serious overclaiming (such as the repeated incorrect claim of \"removing the inherent trade-off between reconstruction accuracy and disentanglement).  I also have some misgivings with the experimental evaluation as quantitative assessment of disentanglement is only provided for one dataset and reconstruction comparisons for only two of the three datasets; not enough to reach any solid conclusions about the approaches' performance, particularly once we consider how inconsistent disentangling methods and metrics are known to be across random seeds (e.g. the gains over ControlVAE do not look statistically significant).\n\n*Strengths*\n- The paper is reasonably well written and easy to follow.\n- The general approach is sensible and the control-theory approaches used to control beta are well-principled.\n- The experimental results are reasonable, albeit not particularly impressive.\n\n\n*Weaknesses*\n- The contributions are very small and specific, constituting low-level changes relative to Shao et al 2020.\n- The approach builds on the beta-VAE which is not only far from the state-of-the-art, but actively discredited in various ways (e.g. Mathieu et al 2019 and Locatello et al 2019).  Though overwise quite well-referenced, the paper critically omits any discussion of these limitations and I feel like the community has somewhat moved on from the beta-VAE by now; I do not think this is a line of work which will last the test of time or which has any particularly useful transferable ideas.  Given that unsupervised disentanglement methods have rather limited applications at the moment, I thus do not feel the work adds much to the community.\n- The paper severely overclaims: it results suggest that it is potentially able to improve on the Pareto front between reconstruction and disentanglement, but it is blatantly untrue that it completely removes the trade-off in the way that is repeatedly claimed.  Firstly, one still has to have a target KL and how this is fixed naturally induces a trade-off (with higher targets being more focused on disentanglement).  Secondly, there are various reasons why there will always be a degree of trade-off between these objectives regardless of the approach used (e.g. the best reconstructions are achieved if the autoencoder is completely noiseless, but removing the noise also removes any pressure for the latent space to structure in any way as one can just construct a lookup table), such that the claims clearly cannot hold.\n- As explained earlier, the numerical experimental results are not really extensive enough to confirm the approach offers meaningful improvements across a range of tasks.  For example, disentanglement is only quantitively evaluated on one dataset (Dsprites) and even more reruns are needed because the differences to the ControlVAE do not look like they are statistically significant.\n- I think there are much stronger baselines that could have been considered such as a) disentangling approaches published since the FactorVAE, b) using hierarchical VAEs which has shown to already give a much better trade-off (see eg arXiv:1906.00230), c) more heuristical ways of varying beta, and d) a FactorVAE where the scaling on the total correlation term is varied.  Relatedly, the FactorVAE is still rapidly improving its reconstruction loss with more iterations in Fig 2a, making the comparisons a little unfair as it seems like it might quickly catch up if the training was run for longer.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is in general well written and easy to follow, and the considered approach of controlling beta is sensible. However, all reviewers identify shortcomings in the empirical analysis of the proposed method (missing comparison with stronger baselines, convergence issues of the considered baselines, considered datasets, etc.). Furthermore, compared to the ControlVAE the contribution of the paper seems limited; and the empirical evaluation is insufficient to claim superior results in general. The authors did not address most of the concerns raised by the reviewers in their rebuttal. The authors can improve their paper substantially by performing the experimental results proposed by the reviewers and clarifying differences to the ControlVAE—but in its current form the paper does not meet the standard of ICLR.

Decision: Reject