Title: FAVAE: SEQUENCE DISENTANGLEMENT USING IN-FORMATION BOTTLENECK PRINCIPLE

Abstract: We propose the factorized action variational autoencoder (FAVAE), a state-of-theart generative model for learning disentangled and interpretable representations from sequential data via the information bottleneck without supervision. The purpose of disentangled representation learning is to obtain interpretable and transferable representations from data. We focused on the disentangled representation of sequential data since there is a wide range of potential applications if disentanglement representation is extended to sequential data such as video, speech, and stock price data. Sequential data is characterized by dynamic factors and static factors: dynamic factors are time dependent, and static factors are independent of time. Previous works succeed in disentangling static factors and dynamic factors by explicit modeling the priors of latent variables to distinguish between static and dynamic factors. However, this model cannot disentangle representations between dynamic factors, such as disentangling "picking" and "throwing" in robotic tasks. In this paper, we propose new model that can disentangle multiple dynamic factors. Since our method does not require modeling priors, it is capable of disentangling "between" dynamic factors. In experiments, we show that FAVAE can extract the disentangled dynamic factors.

1. INTRODUCTION:

Representation learning is one of the most fundamental problems in machine learning. A real world data distribution can be regarded as a low-dimensional manifold in a high-dimensional space (Bengio et al., 2013) . Generative models in deep learning, such as the variational autoencoder (VAE) (Kingma & Welling, 2013) and the generative adversarial network (GAN) (Goodfellow et al., 2014) , are able to learn low-dimensional manifold representation (factor) as a latent variable. The factors are fundamental components such as position, color, and degree of smiling in an image of a human face (Liu et al., 2015) . Disentangled representation is defined as a single factor being represented by a single latent variable (Bengio et al., 2013) . Thus, if in a model of learned disentangled representation, shifting one latent variable while leaving the others fixed generates data showing that only the corresponding factor was changed. This is called latent traversals (a good demonstration of which was given by Higgins et al. (2016a)1 ). There are two advantages of disentangled representation. First, latent variables are interpretable. Second, the disentangled representation is generalizable and robust against adversarial attacks (Alemi et al., 2016) .

We focus on the disentangled representation learning of sequential data. Sequential data is characterized by dynamic factors and static factors: dynamic factors are time dependent, and static factors are independent of time. With disentangled representation learning from sequential data, we should be able to extract dynamic factors that cannot be extracted by disentangled representation learning models for non-sequential data such as β-VAE (Higgins et al., 2016a; b) and InfoGAN (Chen et al., 2016) . The concept of disentangled representation learning for sequential data is illustrated in Fig. 1 . Consider that the pseudo-dataset of the movement of a submarine has a dynamic factor: the trajectory shape. The disentangled representation learning model for sequential data can extract this shape. On the other hand, since the disentangled representation learning model for non-sequential data does not consider the sequence of data, it merely extracts the x-position and y-position. There is a wide range of potential applications if we extend disentanglement representation to sequential data such as speech, video, and stock market data. For example, disentangled representation learning for stock price data can extract the fundamental trend of a given stock price. Another application is the reduction of action space in reinforcement learning. Extracting dynamic factors would enable the generation of macro-actions (Durugkar et al., 2016) , which are sets of sequential actions that represent the fundamental factors of the actions. Thus, disentangled representation learning for sequential data opens the door to new areas of research.

Very recent related work (Hsu et al., 2017; Li & Mandt, 2018 ) separated factors of sequential data into dynamic and static factors. The factorized hierarchical variational autoencoder (FHVAE) (Hsu et al., 2017) is based on a graphical model using latent variables with different time dependencies. By maximizing the variational lower bound of the graphical model, the FHVAE separates the different time dependent factors such as the dynamic and static factors. The VAE architecture developed by Li & Mandt (2018) is the same as the FHVAE in terms of the time dependencies of the latent variables. Since these models require different time dependencies for the latent variables, these approaches cannot be used disentangle variables with the same time dependency factor.

We address this problem by taking a different approach. First, we analyze the root cause of disentanglement from the perspective of information theory. As a result, the term causing disentanglement is derived from a more fundamental rule: reduce the mutual dependence between the input and output of an encoder while keeping the reconstruction of the data. This is called the information bottleneck (IB) principle. We naturally extend this principle to sequential data from the relationship between x and z to x t:T and z. This enables the separation of multiple dynamic factors as a consequence of information compression. It is difficult to learn a disentangled representation of sequential data since not only the feature space but also the time space should be compressed. We created the factorized action variational autoencoder (FAVAE) in which we implemented the concept of information capacity to stabilize learning and a ladder network to learn a disentangled representation in accordance with the level of data abstraction. 

EQUATION

where z is latent variable, D KL is the Kullback-Leibler divergence, and q (z|x) is an approximated distribution of p (z|x). D KL (q (z|x) ||p (z|x)) reduces to zero as the ELBO L VAE increases; thus, q (z|x) learns a good approximation of p (z|x). The ELBO is defined as

EQUATION

where the first term, E q(z|x) [log p (x|z)], is a reconstruction term used to reconstruct x, and the second term D KL (q (z|x) ||p (z)) is a regularization term used to regularize posterior q (z|x). Encoder q (z|x) and decoder p (x|z) are learned in the VAE.

Next we will explain how β-VAE extracts disentangled representations from unlabeled data. β-VAE is an extension of the coefficient β > 1 of the regularization term D KL (q (z|x) ||p (z)) in the VAE.

The objective function of β-VAE is

EQUATION

where β > 1 and p (z) = N (0, 1). β-VAE promotes disentangled representation learning via the Kullback-Leibler divergence term. As β increases, the latent variable q (z|x) approaches the prior p (z) ; therefore, each z i is pressured to learn the probability distribution of N (0, 1). However, if all latent variables z i become N (0, 1), the model cannot reconstruct x. As a result, as long as z reconstructs x, β-VAE reduces the information of z.

3. PRELIMINARY: ORIGIN OF DISENTANGLEMENT:

To clarify the origin of disentanglement, we will explain the regularization term. The regularization term has been decomposed into three terms (Chen et al., 2018; Kim & Mnih, 2018; Hoffman & Johnson, 2016) :

EQUATION

) where z j denotes the j-th dimension of the latent variable. The second term, which is called "total correlation" in information theory, quantifies the redundancy or dependency among a set of n random variables (Watanabe, 1960) . β-TCVAE has been experimentally shown to reduce the total correlation causing disentanglement (Chen et al., 2018) . The third term indirectly causes disentanglement by bringing q (z|x) close to the independent standard normal distribution p (z). The first term is mutual information between the data variable and latent variable based on the empirical data distribution. Minimizing the regularization term causes disentanglement but disturbs reconstruction via the first term in Eq. ( 4). The shift C scheme was proposed (Burgess et al., 2018) as a means to solve this conflict:

EQUATION

where constant shift C, which is called "information capacity," linearly increases during training. This shift C can be understood from the point of view of an information bottleneck (Tishby et al., 2000) . The VAE can be derived by maximizing the ELBO, but β-VAE can no longer be interpreted as an ELBO once this scheme has been applied. The objective function of β-VAE is derived from the information bottleneck (Alemi et al., 2016; Achille & Soatto, 2018; Tishby et al., 2000; Chechik et al., 2005) .

EQUATION

)

where C is the information capacity and x is the empirical distribution. Solving this equation by using Lagrange multipliers drives the objective function of β-VAE Eq. ( 5) with β as the Lagrange multiplier (details in Appendix B of (Alemi et al., 2016) ). In Eq. ( 5), information capacity C prevents I (x, z) from becoming zero. In the information bottleneck literature, y typically stands for a classification task; however, the formulation can be related to the autoencoding objective (Alemi et al., 2016) . Therefore, the objective function of β-VAE can be understood using the information bottleneck principle.

4. PROPOSED MODEL: DISENTANGLEMENT FOR SEQUENTIAL DATA:

Our proposed FAVAE model learns disentangled and interpretable representations from sequential data without supervision. We consider sequential data

EQUATION

For sequential data, we replace x with (x 1:T ) in Eq. 5. The objective function of the FAVAE model is

EQUATION

where p (z) = N (0, 1). The variational recurrent neural network (Chung et al., 2015) and stochastic recurrent neural network (Fraccaro et al., 2016) extend the VAE model to a recurrent framework. The priors of both networks are dependent on time. The time dependent prior experimentally improves the ELBO. In contrast, the prior of our model is independent of time like those of the stochastic recurrent network (Bayer & Osendorfer, 2014) and the Deep Recurrent Attentive Writer (DRAW) neural network architecture (Gregor et al., 2015) ; this is because FAVAE is disentangled representation learning rather than density estimation. For better understanding, consider FAVAE from the perspective of IB. As with β-VAE, FAVAE can be understood from the information bottleneck principle. max I (z;

EQUATION

where x1:T follows an empirical distribution. These principles make the representation of z compact while reconstruction of the sequential data is represented by x 1:T (see Appendix A).

4.1. LADDER NETWORK:

Figure 2 : FAVAE architecture.

An important extension to FAVAE is a hierarchical representation scheme inspired by the VLAE (Zhao et al., 2017) . Encoder q (z|x 1:T ) within a ladder network is defined as

EQUATION

)

where l is a layer index, h 0 ≡ x 1:T , and f is a time convolution network, which is explained in the next section. Decoder p (x 1:T |z) within the ladder network is defined as

zL = g l (z L ) (12) zl = g l (z l+1 + gate (z l )) , (13) x 1:T ∼ r (x 1:T , z0 ) (14)

where g l is the time deconvolution network with l = 1, • • • , L -1, and r is a distribution family parameterized by g 0 (z 0 ). The gate computes the Hadamard product of its learnable parameter and 

4.2. HOW TO ENCODE AND DECODE:

There are several mainstream neural network models designed for sequential data, such as the long short-term memory model (Hochreiter & Schmidhuber, 1997) , the gated recurrent unit model (Chung et al., 2014) , and the quasi-recurrent neural network QRNN (Bradbury et al., 2016) . However, the VLAE has a hierarchical structure created by abstracting a convolutional neural network, so it is simple to add the time convolution of the QRNN to our model. The input data are x t,i , where t is the time index and i is the dimension of the feature vector index. The time convolution considers the dimensions of feature vector j as a convolution channel and performs convolution in the time direction:

EQUATION

)

where j is the channel index. The proposed FAVAE model has a network similar to the VAE one regarding time convolution and a loss function similar to the β-VAE one (Eq. ( 8)). We used the batch normalization (Hinton et al., 2012) and ReLU as activation functions though other variations are possible. For example, 1d convolutional neural networks use a filter size of 3 and a stride of 2 and do not use a pooling layer.

5. MEASURING DISENTANGLEMENT:

While latent traversals are useful for checking the success or failure of disentanglement, quantification of the disentanglement is required for reliably evaluating the model. Various disentanglement quantification methods have been reported (Eastwood & Williams, 2018; Chen et al., 2018; Kim & Mnih, 2018; Higgins et al., 2016b; a) , but there is no standard method. We use the mutual information gap (MIG) (Chen et al., 2018) as the metric for disentanglement. The basic idea of MIG is measuring the mutual information between latent variables z j and a ground truth factor v k . Higher mutual information means that z j contains more information regarding v k .

MIG ≡ 1 K K k=1 1 H(v k ) I z j (k) ; v k -max j =j (k) I (z j ; v k )

, where j (k) ≡ arg max j I (z j ; v k ), and H (v k ) is entropy for normalization. In our evaluation we experimentally measure disentanglement with MIG. 

7. EVALUATION:

We evaluated our model experimentally using three sequential datasets: 2D Reaching, 2D Wavy Reaching, and Gripper. We used a batch size of 128 and the Adam (Kingma & Ba, 2014) optimizer with a learning rate of 10 -3 .

7.1. 2D REACHING:

To determine the differences between FAVAE and β-VAE, we used a bi-dimensional space reaching dataset. Starting from point (0, 0), the point travels to goal position (-0.1, +1) or (+0.1, +1). There are ten possible trajectories to each goal; five are curved inward, and the other five are curved outward.

The degree of curvature for all five trajectories is different. The number of factor combinations was thus 20 (2x2x5). The trajectory length was 1000, so the size of one trajectory was [1000x2].

We compared the performances of β-VAE and FAVAE trained on the 2D Reaching dataset. The results of latent traversal are transforming one dimension of latent variable z into another value and reconstructing something from the traversed latent variables. β-VAE, which is only able to learn from every point of a trajectory separately, encodes data points into latent variables that are parallel to the x and y axes (3b, 3c). In contrast, FAVAE learns through one entire trajectory and can encode disentangled representation effectively so that feasible trajectories are generated from traversed latent variables (3d, 3e).

7.2. 2D WAVY REACHING:

To confirm the effect of disentanglement through information bottleneck, we evaluated the validity of our model under more complex factors by adding more factors to the 2D Reaching dataset. Five factors in total generated data compared to the three factors that generate data in 2D Reaching. This modified dataset differed in that four out of the five factors affect only part of the trajectory: two of them affect the first half, and the other two affect the second half. This means that the model should be able to focus on a certain part of the whole trajectory and be able to extract factors related to that part. A detailed explanation of these factors is given in Appendix B.

We compared various models on the basis of MIG to demonstrate the validity of our proposed model in comparison of a time convolution AE in which a loss function is used only for the autoencoder (β = 0), FAVAE without the ladder network and information capacity C, and FAVAE with the ladder network and information capacity C. As shown in Table 1 , FAVAE with the ladder network and C had the highest MIG scores for 2D Reaching and 2D Wavy Reaching. This indicates that this model learned a disentangled representation best. Note that for 2D Reaching, the best value for C was small, meaning that there was little effect from adding C (since the dataset was simple, this task can be solved even if the amount of information of z is small). The latent traversal results for 2D Wavy Reaching are plotted in Fig. 5 . Even though not all learned representations are perfectly disentangled, the visualization shows that all five generation factors were learned from five latent variables; the other latent variables did not learn any meaningful factors, indicating that the factors could be expressed as a combination of five "active" latent variables. We tested our model for β = 300. The use of a ladder network in our model improved disentangled representation learning and minimized the reconstruction loss. The graph in Fig. 6 shows the MIG scores for networks with different numbers of ladders. The error bars represent the standard deviation for ten repetitive trials. Using all three ladders resulted in the minimum reconstruction loss with the highest MIG score (green curve) except for "Higher Ladder One". "Higher Ladder One" has a large reconstruction error.

7.3. SPRITES DATASET.:

To evaluate the effectiveness of the video dataset, we trained our model with the Sprites dataset which is used in (Li & Mandt, 2018) . This dataset has sequential length = 8 RGB video data with 3 × 64 × 64. This data set consists of static factor and dynamic factor. We note that the motion is not created with the combination of dynamic factors, and each motion exists individually (detail is E.2). Table 4 show the factors used in our experiment. We executed disentangled representation learning by using the FAVAE model with β = 20, C = [0.3, 0.17, 0.06] and network architecture used for this training is explained in Section F.1. Fig. 7 shows the results of latent traversal, and we chose two z values to change from z = 3 to 3. Since this dataset is composed of discrete factors, we show two z values at a time.The latent variables in the 1st ladder extract expressions of motion (4th z in 1st ladder), pants color (5th z in 1st ladder), direction of character (6th z in 1st ladder) and the shirts color (7th z in 1st ladder). The latent variables in the 2nd ladder extract expressions of the hair color (1st z in 2nd ladder) and the body color (2nd z in 2nd ladder). FHVAE can extract the disentangled representations between static factors and dynamic factor in high dimension dataset.

8. SUMMARY AND FUTURE WORK:

Our factorized action variational autoencoder (FAVAVE) generative model learns disentangled and interpretable representations via the information bottleneck from sequential data. Evaluation using three sequential datasets demonstrated that it can learn disentangled representations. Future work includes extending the time convolution part to a sequence-to-sequence model (Sutskever et al., 2014) and applying the model to actions of reinforcement learning to reduce the pattern of actions.

REFERENCES:

Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.

A INFORMATION BOTTLENECK PRINCIPLE FOR SEQUENTIAL DATA.

Here the aim is to show the relationship between FAVAE and the information bottleneck for sequential data. Consider the information bottleneck object:

EQUATION

is expanded from Alemi et al. ( 2016) to sequential data. We need to distinguish between x1:T and x 1:T , where x 1:T is the true distribution and x1:T is the empirical distribution created by sampling from the true distribution. We maximize the mutual information of true data x 1:T and z while constraining the information contained in the empirical data distribution. We do this by using Lagrange multiplier:

EQUATION

where β is a constant. For the first term,

EQUATION

where H (x 1:T ) is entropy, which can be neglected in optimization. The last line is Monte Carlo approximation. For the second term,

EQUATION

As a result,

EQUATION

For convenience of calculation, we use x i sampled from mini-batch data for both the reconstruction term and the regularization term. This is only an approximation. If the information bottleneck principle is followed completely, it is better to use different batch data for the reconstruction and regularization terms.

B EXTRACTED FACTORS OF EACH LADDER IN FAVAE:

We expect the ladder network can disentangle representations at different levels of abstraction. In this section, We check the factor extracted in each ladder by using 2D Reaching and 2D Wavy. Table 2 shows the counting index of latent variable with the highest mutual information in each ladder network. In Table 2 , the rows represent factor and the columns represent the index of the ladder networks. The factor 1 (goal left / goal right) in 2D Reaching and the factor 1 (goal position) in 2D wavy Reaching were extracted to the most frequently in the latent variable in 3rd ladder.

Since the latent variables have 8 dimensions for the 1st ladder, 4 dimensions for the 2nd ladder and 2 dimensions for the 3rd ladder, the 3rd ladder should be the least frequent when factors are randomly entered for each z. Especially long-term and short-term factors are clear in the 2D wavy Reaching dataset. In 2D Wavy Reaching dataset, there is distinct difference between factors of long and short time dependency. The "goal position" is the factor which affect the entire trajectory, and other factors affect half length of the trajectory (Fig. 9 ).In our experiment the goal of the trajectory which affect the entire trajectory tended to be expressed in the 3rd ladder. In both datasets, only factor 1 represents goal positions while others represent shape of the trajectories. Since factor 1 has different abstraction level from others, factor 1 and others result in different ladders such as ladder 3 and others. Reaching. The best C was decided by the value of KL divergence loss when we experimented to allow reconstruction with C = 0, 0, 0.

D GRIPPER:

To evaluate the potential application of our model to robotic tasks, we constructed a robot endeffector simulation environment based on Bullet Real-Time Physics Simulation2 . The environment consisted of an end-effector, two balls (red and blue), and two baskets (red and blue) in a bi-dimensional space. The end-effector grabs one of the balls and places it into the basket with the same color as the ball. The data factors include movement "habits." For example, the end-effector could reach the targeted ball by either directly moving toward the ball obliquely or first moving above of the ball and then lowering itself until it reached the ball (perpendicular movement). The end-effector could drop the ball from far above the basket or place it gently in the basket. Each factor could affect different lengths among the data; e.g., "the plan to place the ball in the basket" factor affects different lengths per datum since the initial distance from the target ball to the corresponding basket may differ. This means that the time required to reach the ball should differ. Note that input is a value such as gripper's position, not an image. See Appendix B for a more detailed explanation. 

F.2 FHVAE:

For the FHVAE experiments, we used the code from the implementation at Github4 . We used recurrent setting with LSTM encoders and decoder with unit size 256 and batch size 80. Dimensions of latent variable z1 and z2 are 7 for each. We used Adam optimizer with learning rate 0.001. We applied goal position factors (2 classes for 2D Reaching and 5 classes for 2D wavy Reaching) as label inputs. We varied α from 1.0 to 30.0, and the best α was 20.0 for 2D Reaching and 1.0 for 2D wavy reaching.

:

This demonstration is available at http://tinyurl.com/jgbyzke

This environment is available at https://pybullet.org

This code is available at https://github.com/yatindandi/Disentangled-Sequential-Autoencoder

This code is available at https://github.com/wnhsu/FactorizedHierarchicalVAE

Figures and Tables:

(a) β-VAE accepts sequential data. (b) Latent traversal of β-VAE. (c) FAVAE accepts sequential data. (d) Latent traversal of FAVAE.

Figure 1: Illustration of how FAVAE differs from β-VAE. β-VAE does not accept data sequentially; it cannot differentiate data points from different trajectories or sequences of data points. FAVAE considers a sequence of data points, taking all data points in a trajectory as one datum. For example, for a pseudo-dataset representing the trajectory of a submarine (1a,1c), β-VAE accepts 11 different positions of the submarine as non-sequential data while FAVAE accepts three different trajectories of the submarine as sequential data. Therefore, the latent variable in β-VAE learns only the coordinates of the submarine, and the latent traversal shows the change in the submarines position. On the other hand, FAVAE learns the factor that controls the trajectory of the submarine, so the latent traversal shows the change in the submarines trajectory.

β-VAE, 1st z. (c) β-VAE, 2nd z. (d) FAVAE, 1st z in 3rd ladder.(e) FAVAE, 1st z in 1st ladder.

Figure 3: Visualization of latent traversal of β-VAE and FAVAE. On one sampled trajectory (red), each latent variable is traversed and purple and/or blue points are generated. The color corresponds to the value of the traversed latent variable. 3a represents all data trajectories of 2D reaching.

Figure 4: MIG score and reconstruction loss for different values of β. Blue line represents results with information capacity C greater than zero; red line represents results with C set to e zero. Note that x axis is plotted in log scale.

Figure 5: Visualization of latent traversal for 2D Wavy Reaching. Each plot was decoded by traversing one latent variable; different colors represent trajectories generated from different values of same latent variable, z.

Fig. 4. We evaluated reconstruction loss and the MIG score for various values of β using three ladder networks (Fig. 6) with a different value of C for each ladder: C = [LowerLadder, MiddleLadder, HigherLadder]. One setting used was C = [0, 0, 0], meaning that C was not used; another setting was C = [20, 1, 5], meaning that C was adjusted on the basis of values of KL-Divergence for β = 1 and C = [0, 0, 0]. When C was not used, the model could not reconstruct data when β was high; thus, disentangled representation was not learned well when β was high. When C was used, the MIG score increased with β while reconstruction loss was suppressed.

Figure 6: Evaluation of ladders (left); structure used for each experiment (right).

Figure 7: Visualization of latent traversal of FAVAE. The horizontal axis represents sequence and the vertical axis represents the difference of z values.

Figure 8: Visualization of learned disentangled representations for Gripper dataset. Traversed latent variable shows changing position of blue objects:(b) (9th z in 1st ladder), changing target ball from red one to blue one and moving accordingly:(c) (1st z in 2nd ladder), reaching ball using different plan:(d) (2nd z in 2nd ladder), changing plan to place ball from drop ball to placing it gently:(e) (4th z in 2nd ladder).

learned the disentangled factors of the Gripper dataset. Example visualized latent traversals are shown in Fig.8. The traversed latent variable determined which factors were disentangled, such as the initial position of blue ball and basket (b), the targeted ball (red or blue) (c), plan to reach the ball (move obliquely or move perpendicularly) (d), and plan to place the ball in the basket (drop the ball or placing it in the basket gently) (e). These results indicate that our model can learn generative factors such as disentangled latent variables for robotic tasks, even though the length of the data affected by each factor may differ for each datum. We used FAVAE in a two-ladder network with 12 and 8 latent variables with β=1000.

Figure 9: Visualization of factors of 2D Wavy Reaching training data.

Figure 10: Motions in Sprites dataset. Row 1/2/3 represent walk left/forward/right, row 4/5/6 represent spellcast left/forward/right, and row 7/8/9 represent slash left/forward/right' respectfully.

Disentanglement scores (MIG, and Reconstruction loss) with standard deviations by 10 seeds for the different model. Best results are shown in bold.Hsu et al., 2017;Li & Mandt, 2018) graphically disentangle static and dynamic factors in sequential data such as speech data and video data(Garofolo et al., 1993;Pearce & Picone, 2002). In contrast, our model performs disentanglement by using a loss function (see Eq. 8). The advantage of the graphical models is that they can control the interpretable factors by controlling the priors time dependency. Since dynamic factors have the same time dependency, these models cannot disentangle dynamic factors. A loss function model can disentangle sets of dynamic factors as well as disentangle static and dynamic factors.

Table content: <table><tr><td>Model</td><td colspan="2">2D Reaching</td><td colspan="2">2D wavy Reaching</td></tr><tr><td/><td>MIG</td><td>Reconstruction</td><td>MIG</td><td>Reconstruction</td></tr><tr><td>Time Convolution AE</td><td>0.21(27)</td><td>0.18(13)</td><td>0.04(3)</td><td>0.06(1)</td></tr><tr><td>FAVAE (without ladder and C)</td><td>0.34(21)</td><td>0.62(115)</td><td>0.22(14)</td><td>23.6(279)</td></tr><tr><td colspan="2">FAVAE (with ladder and without C) 0.41(13)</td><td>0.03(5)</td><td>0.26(7)</td><td>0.08(2)</td></tr><tr><td colspan="2">FAVAE (without ladder and with C) 0.34(21)</td><td>0.62(115)</td><td>0.11(10)</td><td>1.47(73)</td></tr><tr><td>FAVAE (with ladder and C)</td><td>0.41(13)</td><td>0.03(5)</td><td>0.34(28)</td><td>1.81(170)</td></tr><tr><td>6 RELATED WORK</td><td/><td/><td/><td/></tr><tr><td>Several recently reported models (</td><td/><td/><td/><td/></tr></table>

Counting the index of latent variable in each factor by 10 seeds. The detail of factor is in Table4.

Table content: <table><tr><td/><td/><td colspan="3">1st ladder 2nd ladder 3rd ladder</td></tr><tr><td/><td>factor 1</td><td>1</td><td>1</td><td>8</td></tr><tr><td>2D Reaching</td><td>factor 2</td><td>10</td><td>0</td><td>0</td></tr><tr><td/><td>factor 3</td><td>10</td><td>0</td><td>0</td></tr><tr><td/><td>factor 1</td><td>3</td><td>0</td><td>7</td></tr><tr><td/><td>factor 2</td><td>8</td><td>0</td><td>2</td></tr><tr><td>2D wavy Reaching</td><td>factor 3</td><td>8</td><td>0</td><td>2</td></tr><tr><td/><td>factor 4</td><td>9</td><td>1</td><td>0</td></tr><tr><td/><td>factor 5</td><td>9</td><td>0</td><td>1</td></tr></table>

Disentanglement scores (MIG, and Reconstruction loss) with standard deviations by 10 seeds for the different model in sequence length = 100. Best results are shown in bold.

Table content: <table><tr><td>Model</td><td colspan="2">2D Reaching</td><td colspan="2">2D wavy Reaching</td></tr><tr><td/><td>MIG</td><td>Reconstruction</td><td>MIG</td><td>Reconstruction</td></tr><tr><td>FHVAE</td><td>0.62(17)</td><td>0.0003(5)</td><td>0.19(6)</td><td>0.051(36)</td></tr><tr><td>Time Convolution AE</td><td>0.06(4)</td><td>0.014(7)</td><td>0.26(6)</td><td>0.019(2)</td></tr><tr><td>FAVAE (without ladder and C)</td><td>0.52(6)</td><td>0.007(1)</td><td>0.28(2)</td><td>0.058(3)</td></tr><tr><td>FAVAE (without ladder with C)</td><td>0.47(4)</td><td>0.005(0)</td><td>0.26(1)</td><td>0.069(4)</td></tr><tr><td colspan="2">FAVEA (with ladder without C) 0.34(11)</td><td>0.005(1)</td><td>0.31(8)</td><td>0.022(2)</td></tr><tr><td>FAVAE (with ladder and C)</td><td>0.35(7)</td><td>0.007(1)</td><td>0.40(7)</td><td>0.055(4)</td></tr></table>

Peer review (iclr2019):
 This paper proposes an extension to VAE to model sequential datasets and extract disentangled representations of their evolution. It consists of a straight extension of CCI-VAE (Burgess et al 2018) to accept sequential data, combining it with a Ladder VAE architecture (VLAE, Zhao et al 2017). They show early results on fitting toy domains involving 2D points moving in space (reaching, reaching in sub-sequences with complex dynamics, gripper domain). Overall, I think this is an interesting piece of work, which proposes a good model extension and assessment of its characteristics. The model is well presented, the different components are sufficiently motivated and they perform just enough experiments to showcase the effectiveness of their method, although with some reservations. Critics: 1. The comparison to Beta-VAE is a straw man, and I’m not sure it’s a valid way to introduce your model. You are basically saying that treating sequential data as if it was non-sequential is bad, which is clearly not surprising? Hence any comparison with Beta-VAE that you show, Figure 1 and Figure 3, are not appropriate (the caption of Fig 1 is particularly bad in that aspect). A more correct comparison would be to directly feed x_1:T to a Beta-VAE and see what happens, maybe with a causal time-convolution as well if you want to avoid 3D filters. 2. You are also not comparing to the FHVAE model you present in your Introduction, which would have been nice to see, given that your model is simpler and requires less supervision. Does FAVAE perform better than these? 3. Section 3 could use a citation to Esmaeili et al 2018, which breaks out the ELBO even further and compares multiple models in a really nice way (e.g. Table A.2). Overall Section 2 and 3 feel a bit long and pedantic, you could just point people to the original papers and move some of the justification to Appendix (e.g. the IB arguments are not that required for your model. ).  The main point you want to put across is that you want to have your “z” compress a full trajectory x_1:T, under a single KL pressure (i.e. last sentence of Section 4). 4. Figure 3 was hard to interpret at first, specifically for panels b and c. Maybe if you showed the “sampled trajectory” only once in another plot it would make it clearer. 5. Time-convolution seems to wrongly be using the opposite indexing? With z_tk = .sum_{p=0}^{H} x_{t+p}, you have an anti-causal filter which looks at the future x_t’ for a z_t? That does not sound right? Also, you should call these “causal convolutions”, which is the more standard term. 6. The exact format of the observations was never clearly explained. From 7.1 I understood that you input 2D positions into the models, but what about the Gripper?  As you’re aware, Beta-VAE and others usually get RGB images as inputs, hence you should make that difference rather clear. This is a much simpler setting you’re working in. 7. Did you anneal the C as was originally proposed in Burgess et al 2018? With which schedule? This was not clear to me. The exact choices of C for the different Ladder levels lacked support as well. An overall section in the Appendix about the parameter ranges you tried, the architectures, the observation specifications, the optimisation schedule etc etc would be useful. 8. I appreciate the introduction of the MIE metric, which seems to slightly improve over MIG in a meaningful way. However, it would be good to show situations where the two metrics disagree and why MIE is better, because in the current experiments this is unclear. 9. Overall the Gripper experiments seem to merit a more complete assessment. Figure 7 was hard to understand, and I am not sure it shows clearly any disentangled factors. Its caption was strange too (what are the “(1, 8)”, “(2, 1)” things referring to?). 10. I would have liked more interpretation and comments on why the Ladder is needed, and why FAVAE (without Ladder and C) does so badly in Table 1. 11. It would be good to know if you really find that the different levels of the Ladder end up extracting different time scales, as you originally claim it can. There are no results supporting this assumption in the current version. 12. Figure 4B uses a bad scale, which makes it hard to assess what happens between the two C conditions for Beta .approx 100, where they seem to differ the most in Fig 4A. 13. Figure 5 could use titles/labels indicating which “generative factors” you think are being represented. Just compare them to your Figure 8 in Appendix. 14. Figure 6 MIE scores look all within noise between models considered. How sensitive is the metric to actual differences in the disentanglement? Overall, I think this is an interesting improvement to disentangled representations learning, but suffers a bit from early experimental results. I would still like it to be shown at ICLR though as it really fits the venue. I*m happy to improve my score given some improvements on the points mentioned above. References: - Burgess et al., 2018: https://arxiv.org/abs/1804.03599 - Zhao et al., 2017: https://arxiv.org/abs/1702.08396 - Esmaeili et al., 2018: https://arxiv.org/abs/1804.02086