Year: 2021

Abstract:
Learning disentangled representations leads to interpretable models and facilitates data generation with style transfer, which has been extensively studied on static data such as images in an unsupervised learning framework. However, only a few works have explored unsupervised disentangled sequential representation learning due to challenges of generating sequential data. In this paper, we propose recurrent Wasserstein Autoencoder (R-WAE), a new framework for generative modeling of sequential data. R-WAE disentangles the representation of an input sequence into static and dynamic factors (i.e., time-invariant and time-varying parts). Our theoretical analysis shows that, R-WAE minimizes an upper bound of a penalized form of the Wasserstein distance between model distribution and sequential data distribution, and simultaneously maximizes the mutual information between input data and different disentangled latent factors, respectively. This is superior to (recurrent) VAE which does not explicitly enforce mutual information maximization between input data and disentangled latent representations. When the number of actions in sequential data is available as weak supervision information, R-WAE is extended to learn a categorical latent representation of actions to improve its disentanglement. Experiments on a variety of datasets show that our models outperform other baselines with the same settings in terms of disentanglement and unconditional video generation both quantitatively and qualitatively.

heading: 1 INTRODUCTION
text: Unsupervised representation learning is an important research topic in machine learning. It embeds high-dimensional sensory data such as images and videos into a low-dimensional latent space in an unsupervised learning framework, aiming at extracting essential data variation factors to help downstream tasks such as classification and prediction (Bengio et al., 2013). In the last several years, disentangled representation learning, which further separates the latent embedding space into exclusive explainable factors such that each factor only interprets one of semantic attributes of sensory data, has received a lot of interest and achieved many empirical successes on static data such as images (Chen et al., 2016; Higgins et al., 2017; Dupont, 2018; Chen et al., 2018; Rubenstein et al., 2018b;a; Kim & Mnih, 2018). For example, the latent representation of handwritten digits can be disentangled into a content factor encoding digit identity and a style factor encoding handwriting style.
In spite of successes on static data, only a few works have explored unsupervised representation disentanglement of sequential data due to the challenges of developing generative models of sequential ∗Equal contribution. †Part of his work was done before joining Tencent. ‡His work was done before joining Amazon. §His work was done before joining Texas A&M University.
data. Learning disentangled representations of sequential data is important and has many applications. For example, the latent representation of a smiling-face video can be disentangled into a static part encoding the identity of the person (content factor) and a dynamic part encoding the smiling motion of the face (motion factor). The disentangled representation of the video can be potentially used for many downstream tasks such as classification, retrieval, and synthetic video generation with style transfer. Most of previous unsupervised representation disentanglement models for static data heavily rely on the KL-divergence regularization in a VAE framework (Higgins et al., 2017; Dupont, 2018; Chen et al., 2018; Kim & Mnih, 2018), which has been shown to be problematic due to matching individual instead of aggregated posterior distribution of the latent code to the same prior (Tolstikhin et al., 2018; Rubenstein et al., 2018b;a). Therefore, extending VAE or recurrent VAE (Chung et al., 2015) to disentangle sequential data in a generative model framework (Hsu et al., 2017; Yingzhen & Mandt, 2018) is not ideal. In addition, recent research (Locatello et al., 2019) has theoretically shown that it is impossible to perform unsupervised disentangled representation learning without inductive biases on both models and data, especially on static data. Fortunately, sequential data such as videos often have clear inductive biases for the disentanglement of content factor and motion factor as mentioned in (Locatello et al., 2019). Unlike static data, the learned static and dynamic factors of sequential data are not exchangeable.
In this paper, we propose a recurrent Wasserstein Autoencoder (R-WAE) to learn disentangled representations of sequential data. We employ a Wasserstein metric (Arjovsky et al., 2018; Gulrajani et al., 2017; Bellemare et al., 2017) induced from the optimal transport between model distribution and the underlying data distribution, which has some nicer properties (for e.g., sum invariance, scale sensitivity, applicable to distributions with non-overlapping supports, and better out-of-sample performance in the worst-case expectation (Esfahani & Kuhn, 2018)) than the KL divergence in VAE (Kingma & Welling, 2014) and β-VAE (Higgins et al., 2017). Leveraging explicit inductive biases in both sequential data and model, we encode an input sequence into two parts: a shared static latent code and a dynamic latent code, and sequentially decode each element of the sequence by combining both codes. We enforce a fixed prior distribution for the static code and learn a prior for the dynamic code to ensure the consistency of the sequence. The disentangled representations are learned by separately regularizing the posteriors of the latent codes with their corresponding priors.
Our main contributions are summarized as follows: (1) We draw the first connection between minimizing a Wasserstein distance and maximizing mutual information for unsupervised representation disentanglement of sequential data from an information theory perspective; (2) We propose two sets of effective regularizers to learn the disentangled representation in a completely unsupervised manner with explicit inductive biases in both sequential data and models. (3) We incorporate a relaxed discrete latent variable to improve the disentangled learning of actions on real data. Experiments show that our models achieve state-of-the-art performance in both disentanglement of static and dynamic latent representations and unconditional video generation under the same settings as baselines (Yingzhen & Mandt, 2018; Tulyakov et al., 2018).

heading: 2 BACKGROUND AND RELATED WORK
text: Notation Let calligraphic letters (i.e. X ) be sets, capital letters (i.e. X) be random variables and lowercase letters be their values. Let D(PX , PG) be the divergence between the true (but unknown) data distribution PX (density p(x)) and the latent-variable generative model distribution PG specified by a prior distribution PZ (density p(z)) of latent variable Z. Let DKL be KL divergence, DJS be Jensen-Shannon divergence and MMD be Maximum Mean Discrepancy (MMD) (Gretton et al., 2007).
Optimal Transport Between Distributions The optimal transport cost inducing a rich class of divergence between the distribution PX and the distribution PG is defined as follows,
W (PX , PG):= inf Γ∼P(X∼PX ,Y∼PG) E(X,Y )∼Γ[c(X,Y )], (1)
where c(X,Y ) is any measurable cost function and P(X ∼ PX , Y ∼ PG) is the set of joint distributions of (X, Y) with respective marginals PX and PG.
Comparison between WAE (Tolstikhin et al., 2018) and VAE (Kingma & Welling, 2014) Instead of optimizing over all couplings Γ between two random variables in X , Bousquet et al.
(2017); Tolstikhin et al. (2018) show that it is sufficient to find Q(Z|X) such that the marginal Q(Z) := EX∼PX [Q(Z|X)] is identical to the prior P (Z), as given in the following definition,
Definition 1. For any deterministic PG(X|Z) and any function G : Z → X ,
W (PX , PG) = inf Q:QZ=PZ EPXEQ(Z|X)[c(X,G(Z))]. (2)
Definition 1 leads to the following loss DWAE of WAE based on a Wasserstein distance,
inf Q(Z|X) EPXEQ(Z|X)[c(X,G(Z))] + β D(QZ , PZ), (3)
where the first term is data reconstruction loss, and the second one is a regularizer that forces the posterior QZ = ∫ Q(Z|X)dPX to match the prior PZ (Adversarial autoencoder (AAE) (Makhzani et al., 2015) shares a similar idea to WAE). In contrast, VAE has a different regularizer EX [DKL(Q(Z|X), PZ))] enforcing the latent posterior distribution of each input to match PZ . In (Rubenstein et al., 2018a;b), it is shown that WAE has better disentanglement than β-VAE (Higgins et al., 2017) on images, which inspires us to design a new representation disentanglement framework for sequential data with several innovations.
Unsupervised disentangled representation learning Several generative models have been proposed to learn disentangled representations of sequential data (Denton et al., 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018; Hsieh et al., 2018; Sun et al., 2018; Tulyakov et al., 2018). FHVAE in (Hsu et al., 2017) is a VAE-based hierarchical graphical model with factorized Gaussian priors and only focuses on speech or audio data. Our R-WAE employing a more powerful recurrent prior can be applied to both speech and video data. The models in (Sun et al., 2018; Denton et al., 2017; Hsieh et al., 2018) are based on the first several elements of a sequence to design disentanglement architectures for future sequence predictions.
In terms of representation learning by mutual information maximization, our work empirically demonstrates that explicit inductive biases in data and model architecture are necessary to the success of learning meaningful disentangled representations of sequential data, while the works in (Locatello et al., 2019; Poole et al., 2019; Tschannen et al., 2020; Ozair et al., 2019) are about general representation learning, especially on static data.
The most related works to ours are MoCoGAN (Tulyakov et al., 2018) and DS-VAE (Yingzhen & Mandt, 2018), which have the ability to disentangle variant and invariant parts of sequential data and perform unconditional sequence generation. Tulyakov et al. (2018) is a GAN-based model that can be only applied to the setting in which the number of motions is finite, and cannot encode the latent representation of sequences. Yingzhen & Mandt (2018) provides a disentangled sequential autoencoder based on VAE (Kingma & Welling, 2014). Training VAE is equivalent to minimizing a lower bound of the KL divergence between empirical data distribution and generated data distribution, which has been shown to produce inferior disentangled representations of static data than generative models employing the Wasserstein metric (Rubenstein et al., 2018a;b).

heading: 3 PROPOSED APPROACH: DISENTANGLED RECURRENT WASSERSTEIN AUTOENCODER (R-WAE)
text: Given a high-dimensional sequence x1:T , our goal is to learn a disentangled representation of timeinvariant latent code zc and time-variant latent code zmt , along the sequence. Let zt = (z
c, zmt ) be the latent code of xt. Let Xt, Zt, Zc and Zmt be random variables with realizations xt, zt, z
c and zmt respectively, and denote D = X1:T . To achieve this goal, we define the following probabilistic generative model by assuming Zmt and Z c are independent,
P (X1:T , Z1:T ) = P (Z c) T∏ t=1 Pψ(Z m t |Zm<t)Pθ(Xt|Zt), (4)
where P (Z1:T ) = P (Zc) ∏T t=1 Pψ(Z m t |Zm<t) is the prior in which Zt = (Zc, Zmt ), and the decoder model Pθ(Xt | Zt) is a Dirac delta distribution. In practice, P (Zc) is chosen as N (0, I)
and Pψ(Zmt |Zm<t) = N (µψ(Zm<t),σ2ψ(Zm<t)), µψ and σψ are parameterized by Recurrent Neural Networks (RNNs). The inference model Q is defined as
Qφ(Z c, Zm1:T |X1:T ) = Qφ(Zc|X1:T ) T∏ t=1 Qφ(Z m t | Zm<t, Xt), (5)
whereQφ(Zc|X1:T ) andQφ(Zmt | Zm<t, Xt) are also Gaussian distributions parameterized by RNNs. The structures of the generative model (4) and the inference model (5) are provided in Fig. 1.

heading: 3.1 R-WAE MINIMIZES A PENALIZED FORM OF A WASSERSTEIN DISTANCE
text: The optimal transport cost between two distributions PD and PG with respective sequential variables X1:T (X1:T∼PD) and Y1:T (Y1:T∼PG) is given as follows,
W (PD, PG) := inf Γ∼P(X1:T∼PD,Y1:T∼PG) E(X1:T ,Y1:T )∼Γ[c(X1:T , Y1:T )], (6)
P(X1:T∼PD,Y1:T∼PG) is a set of all joint distributions with marginals PD and PG respectively. When we choose c(x,y) = ‖x−y‖2 (2-Wasserstein distance) and c(X1:T , Y1:T ) = ∑ t ‖Xt−Yt‖2 by linearity, it is easy to derive the optimal transport cost for disentangled sequential variables.
Theorem 1. With deterministic P (Xt|Zt) and any function Yt = G(Zt), we derive
W (PD, PG) = inf Q:QZc=PZc ,QZm
1:T =PZm 1:T ∑ t EPDEQ(Zt|Z<t,Xt)[c(Xt, G(Zt))], (7)
where QZ1:T = QZcQZm1:T is the marginal distribution of Z1:T when X1:T ∼ PD and Z1:T ∼ Q(Z1:T |X1:T ) and PZ1:T is the prior. Based on the assumptions, we have an upper bound,
W (PD, PG) ≤ inf Q∈S ∑ t EPDEQ(Zt|Z<t,Xt)[c(Xt, G(Zt))], (8)
where the subset S is S = {Q : QZc = PZc , QZm1 = PZm1 , QZmt |Zm<t = PZmt |Zm<t} .
In practice, we have the following objective function of our proposed R-WAE based on Theorem 1,
T∑ t=1 EQ(Zt|Z<t,Xt)[c(Xt, G(Zt))] + β1 D(QZc , PZc) + β2 T∑ t=1 D(QZmt |Zm<t , PZmt |Zm<t), (9)
where D is an divergence between two distributions, and the second and third terms are, respectively, regularization terms for Zc and Zmt . In the following, we will present two different approaches to calculating the regularization terms in section 3.2 and 3.3. Because we cannot straightforwardly estimate the marginals Qφ(Zc) and Qφ(Zmt |Zm<t), we cannot directly use KL divergence in the two regularization terms, but we can optimize the RHS of (9) by likelihood-free optimizations (Gretton et al., 2007; Goodfellow et al., 2014; Nowozin et al., 2016; Arjovsky et al., 2018) when samples from all distributions are available.
3.2 DJS PENALTY FOR Zc AND MMD PENALTY FOR Zm
The prior distribution of Zc is chosen as a multivariate unit-variance Gaussian, N (0, I). We can choose penalty DJS(QZc ,PZc) for Zc and apply min-max optimization by introducing a discriminator Dγ (Goodfellow et al., 2014). Instead of performing optimizations in high-dimensional input data space, we move the adversarial optimization to the latent representation space of the content with a much lower dimension. Because the prior distribution of {Zmt } is dynamically learned during training, it is challenging to use DJS to regularize {Zmt } with a discriminator, which will induce a third minimization within a min-max optimization. Therefore, we use MMD to regularize {Zmt } as samples from both distributions are easy to obtain (dimension of zmt is less than 20 in our experiments on videos). With a kernel k, MMDk(Q,P ) is approximated by samples from Q and P (Gretton et al., 2007). The regularization terms can be summarized as follows and we call the resulting model R-WAE(GAN) (see Algorithm 1 in Appendix for details):
D(QZc ,PZc) = DJS(QZc ,PZc); D(QZmt |Zm<t ,PZmt |Zm<t) = MMDk(QZmt |Zm<t ,PZmt |Zm<t). (10)
3.3 SCALED MMD PENALTY FOR Zc AND MMD PENALTY FOR Zm
MMD with neural kernels for generative modeling of real-world data (Li et al., 2017; Bińkowski et al., 2018; Arbel et al., 2018) motivates us to use only MMD as regularization in Eq. (9),
D(QZc ,PZc) = MMDkγ (QZc ,PZc); D(QZmt |Zm<t ,PZmt |Zm<t) = MMDk(QZmt |Zm<t ,PZmt |Zm<t), (11)
where kγ is a parametrized family of kernels (Li et al., 2017; Bińkowski et al., 2018; Arbel et al., 2018) defined as kγ(x,y) = k(fγ(x), fγ(y)) and fγ(x) is a feature map, which is more expressive and used for Zc with equal or higher dimension than Zmt . The details of optimizing the first term MMDkγ (QZc ,PZc) in Eq. (11) is provided in Appendix D based on scaled MMD (Arbel et al., 2018), a principled and stable technique for training MMD-based critic. We call the resulting model R-WAE(MMD) (see Algorithm 2 in Appendix for details).

heading: 3.4 WEAKLY SUPERVISED DISENTANGLEMENT WITH A KNOWN NUMBER OF ACTIONS
text: When the number of actions (motions) in sequential data, denoted by A, is available, we incorporate a categorical latent variable a (a one-hot vector whose dimension is A) to enhance the disentanglement of the dynamic latent codes of the motions. The inference model for a is designed as qφ(a|x1:T , zm1:T ). Intuitively, the action is inferred from the motion sequence to recognize its label. Learning such a categorical distribution requires a continuous relaxation of the discrete random variable in order to backpropagate its gradient. Let α1, · · · , αA be the class probabilities, we can obtain a sample ã = (y1, · · · , yA) from its continuous relaxation by first sampling g = (g1, · · · , gA) with gj ∼ Gumbel(0, 1) and then applying transformation ãj = exp((logαj +gj)/τ) ∑ i exp((logαi+gi)/τ), where τ is a temperature parameter controlling the approximation. To learn the categorical distribution using the reparameterization trick, we use a regularizer DKL(qφ(ã|x1:T , zm1:T ), p(ã)), where p(ã) is a uniform Gumbel-Softmax prior distribution (Jang et al., 2016; Maddison et al., 2016). The motion variable is augmented as zRt = (z m t ,a), and learning joint continuous and discrete latent representation of image data has been extensively discussed in (Dupont, 2018) (see Fig. 1(c,d) for illustrations).

heading: 4 ANALYZING R-WAE FROM AN INFORMATION THEORY PERSPECTIVE
text: Theorem 2. If the mutual information (MI) between Z1:T and X1:T is defined in terms of the inference model Q, I(Z1:T ;X1:T ) = EQ(X1:T ,Z1:T )[logQ(Z1:T |X1:T ) − logQ(Z1:T )], where Q(X1:T , Z1:T ) = Q(Z1:T |X1:T )P (X1:T ) and Q(Z1:T ) = ∑ X1:T
Q(X1:T , Z1:T ), we have a lower bound:
I(Z1:T ;X1:T )≥ T∑ t=1 EPD [ EQφ [logPθ(Xt |Zt)−logP (D)]−EQφ(Zc|X1:T )[logQφ(Z c)−logP (Zc)] ]
− T∑ t=1 EPD [ EQφ(Zmt |Zm<t,Xt)[logQφ(Z m t |Zm<t)−logP (Zmt |Zm<t) ] . (12)
Theorem 2 shows that R-WAE maximizes a lower bound of the mutual information between X1:T and Z1:T , which theoretically guarantees that R-WAE learns semantically meaningful latent representations of input sequences. With constant removed, the RHS of (9) and (12) are the same if D is KL divergence. In spite of being theoretically important, Theorem 2 with KL divergence cannot be directly used for the regularization terms of R-WAE in practice, because we cannot straightforwardly estimate the marginals Qφ(Zc) and Qφ(Zmt |Zm<t) as discussed previously. From Eq. (9) and (12), we can obtain the following theorem.
Theorem 3. When its distribution divergence is chosen as KL divergence, the regularization terms in Eq. (9) jointly minimize the KL divergence between the inference model Q(Z1:T |X1:T ) and the prior model P (Z1:T ) and maximize the mutual information between X1:T and Z1:T ,
KL(Q(Zc)||P (Zc)) = EpD [KL(Q(Zc|X1:T )||P (Zc))]− I(X1:T ;Zc), (13) KL(Q(Zmt |Zm<t)||P (Zmt |Zm<t))=EpD [KL(Q(Zmt |Zm<t, X1:T )||P (Zmt |Zm<t)]− I(X1:T ;Zmt |Zm<t),
where the mutual information is defined in terms of the inference model as in Theorem 2.
Theorem 3 shows that, even if adopting KL divergence, the regularization in the loss of R-WAE still improves over the one in vanilla VAE, which only has the first term in the RHS of Eq. (13). The two mutual information terms explicitly enforce mutual information maximization between input data and unexchangeable disentangled latent representations, Zc and Zmt . Therefore, R-WAE is superior to recurrent VAE (DS-VAE).

heading: 5 EXPERIMENTS
text: We conduct extensive experiments on four datasets to quantitatively and qualitatively validate our methods. The baseline methods for comparisons are DS-VAE (Yingzhen & Mandt, 2018) and MoCoGAN (Tulyakov et al., 2018). We train our models on Stochastic moving MNIST (SMMNIST), Sprites, and TIMIT datasets under a completely unsupervised setting. The number of actions (motions) is used as prior information for all methods on MUG facial dataset. The detailed descriptions of datasets, architectures, and hyperparameters are provided in Appendix C, D, and G, respectively.

heading: 5.1 QUALITATIVE RESULTS ON DISENTANGLEMENT
text: We encode two original videos on the first and fourth row in Fig. 2 and generate videos on the second and third row by swapping corresponding {zc} and {zm1:T } between videos for style transfer. Fig. 2(left) shows that even testing on the long sequence (trained with T = 100), our R-WAE can disentangle content and motions exactly. In Fig. 2(right), we do the same swapping on Sprites. We can see that the generated swapped videos have exact same appearances and actions as the corresponding original ones. On the MUG
dataset, it is interesting to see that we can swap different motions between different persons. PPPPPPPMethods Datasets EER zc = 16 ↓ zm = 16 ↑
FHVAE 5.06% 22.77% DS-VAE 5.64% 19.20% R-WAE 4.73% 23.41%
Table 1: EER on TIMIT speech dataset under the same dimension setting of segment-level zc and sequence-leve zm for FHVAE (Hsu et al., 2017), DS-VAE (full q) (Yingzhen & Mandt, 2018) and R-WAE(MMD), respectively. Small EER is better for zc and larger EER is better for zm.
PPPPPPPMethods Datasets Sprites SM-MNIST
actions content digits DS-VAE(S) 8.11% 3.98% 3.31% R-WAE(S) 5.83% 2.45% 1.78% DS-VAE(C) 10.37% 4.86% 4.26% R-WAE(C) 7.72% 3.31% 2.84%
Table 2: Comparison of averaged classification errors. On Sprites dataset, fix one encoded attribute and randomly sample others. On SM-MNIST dataset, we fix the encoded zc and randomly sample the motion sequence from the learned prior pψ(zmt |zm<t). We cannot quantitatively verify the motion disentanglement on SM-MNIST.

heading: 5.2 QUANTITATIVE RESULTS
text: SM-MNIST and Sprites Datasets We quantitatively evaluate the disentanglement of our RWAE(MMD). In Table 2, "S" denotes a simple encoder/decoder architecture, where the encoders in both our model and DS-VAE (Yingzhen & Mandt, 2018) only use 5 layers of convolutional and deconvolutional networks adopted from DS-VAE (Yingzhen & Mandt, 2018). "C" denotes a complex encoder/decoder architecture where we use Ladder network (Sønderby et al., 2016; Zhao et al., 2017) and ResBlock (He et al., 2016), provided in Appendix E. On SM-MNIST, we get the labeled latent codes {zc} of test videos {x1:T } with T = 10 and randomly sample motion variables {zm1:T } to get labeled new samples. We pretrain a classifier and predict the accuracy on these labeled new samples. The accuracy on SM-MNIST dataset is evaluated on 10000 test samples. On Sprites, the labels of each attribute(skin colors, pants, hair styles, tops and pants) are available. We get the latent codes by fixing one attribute and randomly sample other attributes. We train a classifier for each attribute and evaluate the disentanglement of each attribute. The accuracy is based on 296 × 9 test data. Both DS-VAE and R-WAE(MMD) have extremely high accuracy (99.94%) when fixing hair style attribute, which is not provided in Table 2 due to space limit. As R-WAE(GAN) and R-WAE(MMD) have similar performance on these datasets, we only provide the results and parameters of R-WAE(MMD) to save space. There are two interesting observations in Table 2. First, the simple architecture has better disentanglement than the complex architecture overall. The reason is that the simple architecture has sufficient ability to extract features and generate clear samples to be recognized by the pretrained classifiers. But the simple architecture cannot generate high-quality samples when applied to real data. Second, our proposed R-WAE(MMD) achieve better disentanglement than DS-VAE (Yingzhen & Mandt, 2018) on both corresponding architectures. The attributes within content latent variables are independent, our model can further disentangle the factors. Compared to DS-VAE, these results demonstrate the advantages of R-WAE with implicit mutual information maximization terms. Due to space limit, we also include similar comparisons on a new Moving-Shape dataset in Appendix I. As the number of possible motions in SM-MNIST is infinite and random, we cannot evaluate the disentanglement of motions by training a classifier. We fix the encoded motions {zm1:T } and randomly sample content variables {zc}. We also randomly sample a motion sequence {zm1:T } and randomly sample contents {zc}. We manually check the motions of these samples and almost all have the same corresponding motion even though the sequence is long (T = 100). TIMIT Speech Dataset We quantitatively compare our R-WAE with FHVAE and DS-VAE on the speaker verification task under the same setting as (Hsu et al., 2017; Yingzhen & Mandt, 2018). The evaluation metric is based on equal error rate (EER), which is explained in detail in Appendix C. The lower EER on zc encoding the timbre of speakers is better and the higher EER on zm encoding linguistic content is better. From Table 1, our model can disentangle zc and zm well. We can see that our R-WAE(MMD) has the best EER performance on both content attribute and motion attribute. In Appendix H we show by style transfer experiments that the learned dynamic factor encodes semantic content which is comparable to DS-VAE. MUG Facial Dataset We quantitatively evaluate the disentanglement and quality of generated samples. We train a 3D classifier on MUG facial dataset with accuracy 95.11% and Inception Score 5.20 on test data (Salimans et al., 2016). We calculate Inception score, intra-entropy H(y|v), where y is the predicted label and v is the generated video, and inter-entropy H(y) (He et al., 2018). For a comprehensive quantitative evaluation, Frame-level FID score, introduced by (Heusel et al., 2017), is also provided. From Table 2, our R-WAE(MMD) and R-WAE(GAN) have higher accuracy, which
means the categorical variable best captures the actions, which indicates our models achieve the best disentanglement. In table 2, the Inception score of R-WAE(GAN) is very close to Inception Score of the exact test data, which means our models have the best sample quality. Our proposed R-WAE(GAN) and R-WAE(MMD) have the best frame-level FID scores, compared with DS-VAE and MoCoGAN. The orignal DS-VAE (DS-VAE(NA)) (Yingzhen & Mandt, 2018) without leveraging the number of actions performs worst, which shows that incorporating the number of actions as prior information does enhance the disentanglement of actions.

heading: 5.3 UNCONDITIONAL VIDEO GENERATION
text: SM-MNIST dataset Fig. 4 in Appendix E provides generated samples on the SM-MNIST dataset by randomly sampling content {zc} from the prior p(zc) and motions {zm1:T } from the learned prior pψ(z m t |zm<t). The length of our generated videos is T = 100 and we only show randomly chosen videos of T = 20 to save file size. Our R-WAE(MMD) achieves the most consistent and visually best sequence even when T = 100. Samples from MoCoGAN (Tulyakov et al., 2018) usually change digit identity along the sequence. The reason is that MoCoGAN (Tulyakov et al., 2018) requires the number of actions be finite. Our generated Sprites videos also have the best results but are not provided due to page limit.
MUG Facial Dataset Fig. 3 and Fig. 5 in Appendix E show generated samples on MUG dataset by randomly sampling content {zc} from the prior p(zc) and motions zRt = (a, zmt ) from the categorical prior p(a) (latent action variable a is a one-hot vector with dimension 6) and the learned prior pψ(zmt |zm<t). We show generated videos of length T = 10. DS-VAE (Yingzhen & Mandt, 2018) used the same structure as ours. Fig. 5 shows that DS-VAE (Yingzhen & Mandt, 2018) and
MoCoGAN (Tulyakov et al., 2018) have blurry beginning frames {xt} and even more blurry frames as time t evolves. While our R-WAE(GAN) has much better frame quality and more consistent video sequences. To have a clear comparison among all three methods, we show the samples at time step T = 10 in Fig. 3, and we can see that DS-VAE has very blurry samples with large time steps.

heading: 6 CONCLUSION
text: In this paper, we propose recurrent Wasserstein Autoencoder (R-WAE) to learn disentangled representations of sequential data based on the optimal transport between distributions with sequential variables. Our theoretical analysis shows that R-WAE simultaneously maximizes the mutual information between input sequential data and different disentangled latent factors. Experiments on a variety of datasets demonstrate that our models achieve state-of-the-art results on the disentanglement of static and dynamic latent representations and unconditional video generation. Future research includes exploring our framework in self-supervised learning and conditional settings for text-to-video and video-to-video synthesis.
Acknowledgement Jun Han thanks Dr. Chen Fang at Tencent for insightful discussions and Prof. Qiang Liu at UT Austin for invaluable support.

heading: APPENDIX FOR RECURRENT WASSERSTEIN AUTOENCODER

heading: APPENDIX A: PROOF OF THEOREM 1
text: In the following, we provide the proof of Theorem 1.
Theorem 1 For PG defined with deterministic PG(X|Z) and any function Y = G(Z),
W (PD, PG) = inf Q:QZc=PZc ,QZm
1:T =PZm 1:T T∑ t=1 EPDEQ(Zt|Xt)[c(Xt, G(Zt))], (14)
where QZ1:T is the marginal distribution of Z1:T when X1:T ∼ PD and Z1:T ∼ Q(Z1:T |X1:T ) and PZ1:T is the prior. Based on the assumptions, the constraint set is equivalent to
W (PD, PG) ≤ inf Q∈S ∑ t EPDEQ(Zt|Xt)[c(Xt, G(Zt))], (15)
where the set S = {Q : QZc = PZc , QZm1 = PZm1 , QZmt |Zm<t = PZmt |Zm<t}.
Proof: Consider the sequential random variables D = X1:T and Y1:T , the optimal transport between the distribution for D = X1:T and the distribution for Y1:T induces a rich class of divergence,
W (PD, PG) := inf Γ∼P(X1:T∼PD,Y1:T∼PG) E(X1:T ,Y1:T )∼Γ[c(X1:T , Y1:T )] (16)
where P(X1:T ∼ PD, Y1:T ∼ PG) is a set of all joint distributions of (X1:T , Y1:T ) with marginals PD and PG, respectively.
When we choose c(x,y) = ‖x − y‖2, c(X1:T , Y1:T ) = ∑ t ‖Xt − Yt‖2 by linearity. It is easy to derive the optimal transport for distributions with sequential random variables,
W (PD, PG) = inf Q:QZ1:T =PZ1:T ∑ t EPDEQ(Zt|Xt)[c(Xt, G(Zt))]. (17)
Based on our assumption, the marginal distribution of the inference model satisfies Q(Z1, · · · , ZT ) = Q(Zc)Q(Zm1 , · · · , ZmT ) = Q(Zc) ∏ t Q(Zmt |Zm<t). (18) The prior distribution P (Z1, · · · , ZT ) satisfies P (Z1, · · · , ZT ) = P (Zc)P (Zm1 , · · · , ZmT ) = P (Zc)
∏ t P (Zmt |Zm<t). (19)
Since the set S is a subset of {Q : QZ1:T = PZ1:T }, we derive the inequality (15).

heading: APPENDIX B: PROOF OF THEOREM 2
text: In the following, we provide the proof of Theorem 2. To make the notations easy to read, we use the density functions of corresponding distributions.
The joint generative distribution is p(x1:T , z1:T ) = pψ(z1:T )pθ(x1:T |z1:T ), where pψ(z1:T ) is the prior distribution and pθ(x1:T |z1:T ) is the decoder model. And the corresponding joint inference distribution is qφ(x1:T , z1:T ) = pD(x1:T )qφ(z1:T | x1:T ). If the MI between z1:T and x1:T is defined in terms of the inference model q, we have the following lower bound with step-by-step derivations:
I(z1:T ;x1:T )= Eq(x1:T ,z1:T )[log qφ(z1:T |x1:T ) qφ(z1:T ) ] (20)
= Eq(x1:T ,z1:T )[DKL(qφ(z1:T |x1:T ), p(z1:T |x1:T ))+log p(z1:T |x1:T )−log qφ(z1:T )] ≥ EpD [Eq(z1:T |x1:T )[log p(x1:T |z1:T ) + log p(z1:T )− log qφ(z1:T )−log p(D)]]
≥ T∑ t=1 Ep(D) [ Eqφ(zt|xt)[log pθ(xt|zt)] ] −Ep(D)[Eqφ(zt|xt)[log qφ(z c)−log p(zc)] ]
− T∑ t=1 Ep(D) [ Eqφ(zmt |xt)[log qφ(z m t |zm<t)− log p(zmt |zm<t) + log p(D) ] ,
where we use Bayesian’s rule p(z1:T |x1:T ) = pθ(x1:T |z1:T )p(z1:T )p(D) . Maximizing the MI between z1:T and x1:T achieves state-of-the-art results in disentangled latent representation by using different regularizers for the static and dynamical latent variable with different priors (Hjelm et al., 2018). In practice, incorporating the mutual information I(zmt ,xt) between element xt and motion z m t might facilitate the disentanglement of the dynamical latent variable zmt .
Theorem 3 When its distribution divergence is chosen as KL divergence, the regularization terms in Eq. (9) jointly minimize the KL divergence between the inference model Q(Z1:T |X1:T ) and the prior model P (Z1:T ) and maximize the mutual information between X1:T and Z1:T ,
KL(Q(Zc)||P (Zc)) = EpD [KL(Q(Zc|X1:T )||P (Zc))]− I(X1:T ;Zc). KL(Q(Zmt |Zm<t)||P (Zmt |Zm<t)) = EpD [KL(Q(Zmt |Zm<t, X1:T )||P (Zmt |Zm<t)]− I(X1:T ;Zmt |Zm<t).
Proof: Denote XD = X1:T . As in the proof of Theorem 2, the mutual information between Z1:T and X1:T is defined in terms of the inference model Q, and we use the density functions of corresponding distributions to make the notations easy to read. Thus,
Q(Z1:T ) = EpDq(z1:T |x1:T ).
According to the definition of mutual information, we have
I(X1:T ;Z c) = EpD ∑ zc pD(x1:T )q(z c|x1:T ) log pD(x1:T )q(z c|x1:T ) pD(x1:T )q(zc)
= EpD ∑ zc q(zc|x1:T ) log q(zc|x1:T ) q(zc)
= EpD ∑ zc q(zc|x1:T ) log q(zc|x1:T ) p(zc) − EpD ∑ zc q(zc|x1:T ) log q(zc) p(zc)
= EpD ∑ zc q(zc|x1:T ) log q(zc|x1:T ) p(zc) − ∑ zc q(zc) log q(zc) p(zc) = EpD [KL(Q(Zc|X1:T )||P (Zc))]−KL(Q(Zc)||P (Zc))
Therefore,
KL(Q(Zc)||P (Zc)) = EpD [KL(Q(Zc|X1:T )||P (Zc))]− I(X1:T ;Zc).
Similarly, we can prove the second equality in the theorem.

heading: APPENDIX C: DATASETS
text: Stochastic Moving MNIST(SM-MNIST) Dataset Stochastic moving MNIST (SM-MNIST) consists of sequences of frames of size 64× 64× 1, containing one MNIST digit moving and bouncing off edges of the frame (walls). We use one digit instead of two digits because two moving digits may collide, which changes the content of the dynamics and is inconsistent with our assumption. The digits in SM-MNIST move with a constant velocity along a trajectory until they hit at wall at which point they bounce off with a random speed and direction.
Sprites Dataset We follow the same steps as in Yingzhen & Mandt (2018) to process Sprites dataset, which consists of animated cartoon characters whose clothing, hairstyle, skin color and action can be fully controlled. We use 6 variants in each of 4 attribute categories (skin colors, tops, pants and hair style) and there are 64 = 1296 unique characters in total, where 1000 of them are used for training and the rest of them are used for testing. We use 9 action categories (walking, casting spells and slashing, each with three different viewing angles.) The resulting dataset consists of video sequences with T = 8 frames of size 64× 64× 3. MUG Facial Dataset We use the MUG Facial Expression Database (Aifanti et al., 2010) for this experiment. The dataset consists of 86 subjects. Each video consists of 50 to 160 frames. To use the same network architecture for the whole video datasets in this paper, we cropped the face regions and scaled to the same size 64× 64× 3. We use six facial expressions (anger, fear, disgust, happiness,
sadness, and surprise). To ensure there is sufficient change in the facial expression along a video sequence, we choose every other frame in the original video sequences to form training and test video sequences of length T = 10. 80% of the videos are used for training and 20% of the videos are used for testing.
TIMIT Speech Dataset The TIMIT dataset (Garofolo, 1993) contains broadband 16k Hz of phonetically-balanced read speech. A total of 6300 utterances (5.4 hours) are presented with 10 sentences from each of 630 speakers. The data is preprocessed in the same way as in (Yingzhen & Mandt, 2018) and (Hsu et al., 2017). The raw speech waveforms are first split into sub-sequences of 200ms, and then preprocessed with sparse fast Fourier transform to obtain a 200 dimensional logmagnitude spectrum, computed every 10ms, i.e., we use T = 20 for sequence x1:T . The dimension of xt is 200.
Now we explain the detail of the evaluation metric, equal error rate (EER), used on TIMIT dataset. Letwtest be the feature of test utterance xtest1:T andw
target be the feature of test utterance xtarget1:T . The predicted identity is confirmed if the cosine similarity betweenwtest andwtarget, cos(wtest,wtarget) is greater than some threshold used in Dehak et al. (2010). The equal error rate (EER) means the false rejection rate equals the false acceptance rate (Dehak et al., 2010). In the following, we will discuss the two choices of feature wtest for evaluations of all methods,
µc = 1
N N∑ i=1 Eq(zc|xi1:T )[z c],
which is used to evaluate the disentanglement of zc;
µm = 1
NT N∑ i=1 T∑ j=1 Eq(zmt |xi1:T )[z m t ],
which is used to evaluate the disentanglement of zm. For more details, please refer to (Dehak et al., 2010; Yingzhen & Mandt, 2018; Hsu et al., 2017). We use the same network architecture as in Yingzhen & Mandt (2018) for a fair comparison on speech dataset. As the input dimension of speech is low, the encoder/decoder network is a 2-hidden-layer MLP with the hidden dimension 256.

heading: APPENDIX D: CHOICES OF REGULARIZERS
text: In the following, we will discuss the choice of regularizers in R-WAE. To make notations easy to read, we use density functions for corresponding distributions. In both R-WAE(GAN) and R-WAE(MMD), we use the same regularizer for D(q(zmt |zm<t), p(zmt |zm<t)). We also add a KLdivergence regularization term on zm to stabilize training. In the experiments, we assume inference model q(zc|x1:T ) is a Gaussian distribution with parameters mean µc and diagonal variance matrix σc. Inference model q(zmt |xt, zm<t) is a Gaussian distribution with parameters meanµm and diagonal variance matrix σm. For the prior distribution, we assume p(zmt |zm<t) is a Gaussian distribution with parameters mean µψm and diagonal covariance matrix σ ψ m. For regularizing the motion variables, we just use MMD without introducing any additional parameter, MMDk(q(zmt |zm<t), p(zmt |zm<t)), and we choose mixture of RBF kernel (Li et al., 2017), where RBF kernel is defined as k(x,y) = exp(−‖x−y‖ 2
2σ2 ). With samples {z̃i} n i=1 from the posterior q(z̃ c) and samples {zi}ni=1 from the prior p(zc), MMDk(q(z̃ c), p(zc)) is defined as
MMDk(q(z̃ c), p(zc))=
1 n(n− 1) ∑ i6=j k(zi, zj)+ 1 n(n− 1) ∑ i 6=j k(z̃i, z̃j)− 1 n2 ∑ i,j k(z̃i, zj). (21)
The difference between R-WAE(MMD) and R-WAE(GAN) is how to choose metrics for the regularizer D(QZc ,PZc), where PZc is the prior distribution and QZc is the posterior distribution of the inference model.
R-WAE(MMD) The regularizer D(QZc ,PZc) is chosen as,
D(QZc ,PZc) = MMDkγ (Q(Zc),P (Zc)),
where the scaled MMD MMDkγ (Q(Z c),P (Zc)) is chosen as
MMDkγ (QZc ,PZc) = M̂MDkγ (QZc , PZc)
1 + 10EP̂ [‖∇fγ(zc)‖2F ] ,
where the function fγ(zc) is the kernel feature map and M̂MDkγ (QZc , PZc) is defined in the following. When we have samples {z̃ci}ni=1 from Q(Zc) and samples {zci}ni=1 from P (Zc),
M̂MDkγ (Q(Z c), P (Zc)) =
1 n(n− 1) ∑ i 6=j k(fγ(z c i ), fγ(z c j)) +
1 n(n− 1) ∑ i 6=j k(fγ(z̃ c i ), fγ(z̃ c j))
(22)
− 1 n2 ∑ i,j k(fγ(z̃ c i ), fγ(z c j)),
where the RBF kernel k is defined on scalar variables, k(x, y) = exp(−‖x−y‖ 2
2 ). To avoid the situation where the generator gets stuck on a local optimum, we apply spectral parametrization for the weight matrix (Miyato et al., 2018). The feature map fγ is updated L steps at each iteration. To overcome posterior collapse and inference lagging, we will update the inference model per iteration of updating the decoder model for L steps during training (He et al., 2019). See Algorithm 1 for details.
R-WAE(GAN) For the regularizer DJS(QZc ,PZc), we introduce a discriminator Dγ . The loss is as follows, L = Ezc∼p(zc)[logDγ(zc)] + Ez̃c∼q(z̃c)[log(1−Dγ(z̃c)))], (23) where p(zc) is the prior distribution and q(z̃c) is the posterior distribution of the inference model. To stabilize the training of the min-max problem in GAN-based optimization (23), a lot of stabilization techniques have been proposed (Thanh-Tung et al., 2019; Mescheder et al., 2018; Gulrajani et al., 2017; Petzka et al., 2017; Roth et al., 2017; Qi, 2017). Let samples {zc} are from the prior p(zc) and {z̃c} are from the inference posterior q(z̃c). In our R-WAE(GAN), we will adopt the regularization from Mescheder et al. (2018) and Thanh-Tung et al. (2019),
L − λE[‖(∇Dγ)ẑc‖2], (24) where ẑc = αzc + (1− α)z̃c, α ∈ U(0, 1) and (∇Dγ)ẑc is evaluated its gradient at the point ẑc.
Algorithm 1 R-WAE(GAN) Input: regularization coefficient β and content prior p(zc) Goal: learn encoders qφ(zc|x1:T ) and qφ(z m t |xt, zm<t), prior pψ(zmt |zm<t), discrimi-
natorDγ , and decoder pθ(xt|zt), where zt = (zc, zmt ) while not converged do
for step 1 to L do Sample batch X = {xt} Sample {zc} from prior p(zc) and {zmt } from prior pψ Sample {z̃c, z̃mt } from encoders qφ Update discriminator Dγ and encoders qφ with loss given by (9), (10) end for Update pθ and prior pψ with loss given by (9) and (10).
end while
Algorithm 2 R-WAE(MMD) Input: regularization coefficient β and content prior p(zc) Goal: learn encoders qφ(zc|x1:T ) and qφ(z m t |xt, zm<t), prior pψ(zmt |zm<t), feature
map fγ and decoder pθ(xt|zt), where zt = (zc, zmt ) while not converged do
for step 1 to L do Sample batch X = {xt} Sample {zc} from prior p(zc) and {zmt } from prior pψ Sample {z̃c, z̃mt } from encoders qφ Update feature map fγ and encoders qφ with loss given by (9), (11) end for Update pθ and prior pψ with loss given by (9) and (11).
end while

heading: 6.1 APPENDIX E: UNCONDITIONAL VIDEO GENERATION
text: Fig. 4 provides generated samples on the SM-MNIST dataset by randomly sampling content {zc} from the prior p(zc) and motions {zm1:T } from the learned prior pψ(zmt |zm<t). The length of our
generated videos is T = 100 and we only show randomly chosen videos of T = 20 to save file size. Our R-WAE(MMD) achieves the most consistent and visually best sequence even when T = 100. Samples from MoCoGAN (Tulyakov et al., 2018) usually change digit identity along the sequence. The reason is that MoCoGAN (Tulyakov et al., 2018) requires the number of actions be finite.
Fig. 5 shows unconditional video generation with T = 10 on MUG facial dataset. DS-VAE in (b) is improved by incorporating categorical latent variables. The figures should be viewed with Adobe Reader to see video.
(a) R-WAE(GAN) (b) DS-VAE (c) MoCoGAN
Figure 5: Unconditional video generation with T = 10 on MUG facial dataset. DS-VAE in (b) is improved by incorporating categorical latent variables. The figures should be viewed with Adobe Reader to see video.

heading: APPENDIX F: LATENT MANIFOLD VISUALIZATION
text: We encode the test data {x1:T } of SM-MNIST with T = 10 to get the content codes {zc} using our R-WAE(MMD). We visualize two-dimensional (2D) manifold of {zc} using t-SNE (Maaten & Hinton, 2008). In Fig. 6, different colors correspond to the digit identities of the latent codes {zc} of test videos on SM-MNIST. This indicates that {zc} encoded by our R-WAE(MMD) exactly captures the invariant information (digits) of the test data. The latent motion codes are sequential and cannot be visualized.

heading: APPENDIX G: MODEL ARCHITECTURE AND HYPER-PARAMETERS
text: In the inference model, we use an encoder network, defined in Fig. 7 (a) to extract latent feature ht defined in Fig.1. We use a decoder network to reconstruct x̂t from the hidden state ht, defined in Fig.1. For the discriminator Dγ in R-WAE(GAN), we use a 4-layer fully-connected neural
network (FC NN) with respective dimension (256, 256, 128, 1). For the feature map fγ with a scalar output for the RBF kernel of R-WAE(MMD), we use a 4-layer fully-connected neural network with respective dimension (256, 256, 128, 1). After encoding xt, we get extracted latent feature ht. We use Fig. 8(a) and Fig. 8(b) to infer the content variable zc and motion variables zmt . When the Gumbel latent variable is incorporated into our weakly-supervised inference model, we use Fig. 8(c) to infer the Gumbel latent variable a. The latent content variable zc and latent motion variable zmt are concatenated as input to an LSTM after an FC NN to output hidden state ht for reconstructing x̂t using the decoder. For our weakly-supervised model, the latent content variable zc, latent motion variable zmt and latent action variable a are concatenated as input to an LSTM after an FC NN to output hidden state ht for reconstructing x̂t using the decoder. We use Adam optimizer (Kingma & Ba, 2015) with β1 = 0.5 and β2 = 0.9.
Architecture on SM-MNIST, Sprites and TIMIT Datasets We use the same architecture on SM-MNIST and Sprites dataset, as shown in Fig. 9. The details of the parameters of the networks are provided in Fig. 9. As R-WAE(GAN) and R-WAE(MMD) have similar performance on SMMNIST and Sprites (see Sprites results in Table 4), we only provide the results and parameters of R-WAE(MMD) to save space. At each iteration of training the decoder pθ(xt|zt) and the prior pψ(z m t |zm<t), we train the encoder parameters qφ and the feature map fγ for R-WAE(MMD) with L steps. The results on SM-MNIST and Sprites datasets are evaluated after 500 epochs. On SM-MNIST dataset, we use a Bernoulli cross-entropy loss and choose L = 5. The penalty coefficients β1 and β2, are, respectively, 5 and 20. The learning rate for the decoder model is 5× 10−4 and the learning rate for the encoder is 1× 10−4. The learning rate for fγ is 1× 10−4. On Sprites dataset, we use an L2 reconstruction loss and choose L = 5 steps. The penalty coefficients β1 and β2 are, respectively, 10 and 60. The learning rate for the decoder model is 3× 10−4 and the learning rate for the encoder is 1× 10−4. The learning rate for Dγ in R-WAE(GAN) or fγ in R-WAE(MMD) is 1× 10−4. We use a decayed learning rate schedule on both datasets. After 50 epochs, we decrease all learning rates by a factor of 2 and after 80 epochs decrease further by a factor of 5. On TIMIT speech dataset, we use the same encoder and decoder architecture as that of DS-VAE. The dimension of hidden states is 256 and the dimensions of zc and zmt are both 16.
ResBlock1 down 64*3*3 self-attention
ResBlock2 down 128*3*3 ResBlock3 down 256*3*3 ResBlock4 down 512*3*3
ResBlock5 down 1024*3*3 Reshape output to (N, 1024× 2× 2)
FC NN
Table 5: Encoder Network Architecture.
FC NN and Reshape input to (N, 2048, 2, 2) ResBlock1 up 1024*3*3 ResBlock2 up 512*3*3 ResBlock3 up 256*3*3 ResBlock4 up 128*3*3
self-attention ResBlock5 up 64*3*3
Conv 3*3*3, activation=sigmoid
Table 6: Decoder Network Architecture.
Table 7: Encoder Network Architecture.
Figure 10: Network parameters on encoder network and decoder network on MUG facial dataset. We adopt ResBlock down and up from Brock et al. (2019). The dimensions of zc, zmt , ht, a are 150, 16, 180 and 6 respectively. The batch size on MUG facial dataset are 30 and the length of video sequence for training is T = 8.
Architecture on MUG Facial Dataset The details of the architecture parameters of the networks for MUG facial dataset are provided in Fig. 9. The results on MUG facial dataset are evaluated after 800 epochs. For the regularizer DKL(qφ(a|x1:T , zm1:T ), p(a)), we choose the coefficient of this categorical regularizer to be 50. We use an L2 reconstruction loss and choose L = 5 steps. For R-WAE(MMD), the penalty coefficients β1 and β2 are, respectively, 10 and 50. For R-WAE(GAN), the coefficients β1 and β2 of the penalties are, respectively, 5 and 60. The learning rate for the decoder model is 5 × 10−4 and the learning rate for the encoder is 2 × 10−4. The learning rate for Dγ in R-WAE(GAN) or fγ in R-WAE(MMD) is 2× 10−4. We use the same decayed learning rate schedule as described on SM-MNIST and Sprites datasets. This architecture can be applied to improve the compression rate (?).

heading: APPENDIX H: ADDITIONAL RESULTS ON AUDIO DATA
text: Swapping Static and Dynamic Factors on Audio Data Here we present results of swapping static and dynamic factors of given audio sequences. Results are given in Figure 11. Each heatmap subplot is of dimension 80 × 20 and visualizes the spectrum of 200ms of an audio clip, in which the mel-scale filter bank features are plotted in the frequency domain (x-axis represents temporal domain with 20 timesteps and y-axis is the value of frequencies). We collect these heatmaps in a matrix where the static factors in a row are kept the same and each column shares the same dynamic factor. It can be observed that in each column, the linguistic phonetic contents as reflected by the formants along x-axis are kept almost the same after swapping. Likewise, the timbres are reflected as the harmonics in the spectrum plot. This can be concluded by observing that the horizontal light stripes which represents the harmonics are kept consistent in a row. Moreover, we perform
identity verification experiment as conducted in DS-VAE (Yingzhen & Mandt, 2018). Similar to cross reconstruction, zcfemale and z c male (or f
female and fmale in DS-VAE) are swapped for two sequences {xfemale} and {xmale}. By an informal listening test of the original-swapped speech sequence pairs, we confirm that the speech content is preserved and identity is transferred (i.e. female voice usually has higher frequency).
APPENDIX I: ADDITIONAL RESULTS ON A MOVING-SHAPE VIDEO DATA
Generation Results on Moving Shapes We report results on a Moving-Shape dataset in Table 9 and Fig. 12. The Moving-Shape synthetic dataset was introduced in Balaji et al. (2018) which has 5 control parameters: shape type (e.g. triangle and square), size (small and large), color (e.g. white and red), motion type (e.g. zig-zag, straight line and diagonal) and motion direction. In Table 9, TFGAN (Balaji et al., 2018) encoder and decoder architectures are considered less expressive compared with BigGAN (Brock et al., 2019) architectures. Similar to results in Table 2, with more complex and expressive architecture, learning disentangled representation is harder. The results in Table 9 and Fig. 12 demonstrate that R-WAE produces better disentanglement and generation performance than DS-VAE both quantitatively and qualitatively. Qualitative difference of fixing zm and sampling zc for DS-VAE and R-WAE is not that obvious and thus not shown.

Review summaries:
This paper extends the Wasserstein autoencoder for learning disentangled representations from sequential data. The latent variable model considered contains separate latent variables capturing global and local information respectively, each of which is regularized by a divergence measuring the marginal posterior $Q_z$ and the prior $P_z$. An optional auxiliary discrete latent is introduced to incorporate inductive bias for discrete local features (e.g., type of actions). To estimate the divergence terms, the authors propose to use MMD for the recurrent local latents since the prior distribution evolves over time; for the global latent, the authors presented two options: discriminator-based Jenson-Shannon Divergence estimate (the same as adversarial autoencoder proposed in Makhzani et al., 2016) and scaled MMD (Arbel et al., 2018). The connection between the proposed objective and mutual information maximization is outlined in Section 4. Experimental results show that the proposed R-WAE model outperforms baseline DS-VAE/FHVAE/MocoGAN