Disentangled Recurrent Wasserstein Autoencoder 
Authors: Jun Han
Martin Renqiang Min
Ligong Han
Li Erran Li
Xuan Zhang
Venue: ICLR 2021 Spotlight (2021)
Keywords: Sequential  Representation Learning
Disentanglement
Recurrent Generative Model

Abstract: Learning disentangled representations leads to interpretable models and facilitates data generation with style transfer, which has been extensively studied on static data such as images in an unsupervised learning framework. However, only a few works have explored unsupervised disentangled sequential representation learning due to challenges of generating sequential data. In this paper, we propose recurrent Wasserstein Autoencoder (R-WAE), a new framework for generative modeling of sequential data. R-WAE disentangles the representation of an input sequence into static and dynamic factors (i.e., time-invariant and time-varying parts). Our theoretical analysis shows that, R-WAE minimizes an upper bound of a penalized form of the Wasserstein distance between model distribution and sequential data distribution, and simultaneously maximizes the mutual information between input data and different disentangled latent factors, respectively. This is superior to (recurrent) VAE which does not explicitly enforce mutual information maximization between input data and disentangled latent representations. When the number of actions in sequential data is available as weak supervision information, R-WAE is extended to learn a categorical latent representation of actions to improve its disentanglement. Experiments on a variety of datasets show that our models outperform other baselines with the same settings in terms of disentanglement and unconditional video generation both quantitatively and qualitatively.

Published as a conference paper at ICLR 2021
DISENTANGLED RECURRENT WASSERSTEIN AUTOEN- CODER
Jun Han PCG, Tencent
Martin Renqiang Min NEC Laboratories America rengiangêness labs. com
Ligong Han Rutgers University hanl gong@gma com
jurhanjhetteneett.com tencent.eceeccoe com
Li Erran Li :
Xuan Zhang 8
Alexa AI, Amazon
Texas A&M University float Matertlazzeeee www.aaaeeeeaaa....mm.a
erranlli i gmail. com
ABSTRACT
Learning disentangled representations leads to interpretable models and facilitates data generation with style transfer, which has been extensively studied static data such as images in an unsupervised learning framework. However, only a few works have explored unsupervised disentangled sequential representation learning due to challenges of generating sequential data. In this paper, we propose recurrent Wasserstein Autoencoder (R- WAE), new framework for generative modeling of sequential data. R-WAE disentangles the representation of an input sequence into static and dynamic factors (i.e., time-invariant and time-varying parts). Our theoretical analysis shows that, R-WAE minimizes an upper bound of penalized form of the Wasserstein distance between model distribution and sequential data distribution, and simultaneously maximizes the mutual information between input data and different disentangled latent factors, respectively. This is superior to (recurrent) VAE which does not explicitly enforce mutual information maximization between input data and disentangled latent representations. When the number of actions in sequential data is available as weak supervision information, R- WAE is extended to learn a categorical latent representation of actions to improve its disentanglement. Experiments outperform other baselines with the same settings in terms disentanglement and unconditional video generation both quantitatively and qualitatively.
variety of datasets show that our models
INTRODUCTION
Unsupervised representation learning is an important research topic in machine learning. It embeds high-dimensional sensory data such as images and videos into a low-dimensional latent space in an unsupervised learning framework, aiming at extracting essential data variation factors to help downstream tasks such as classification and prediction (Bengio et al. (013). In the last several years, disentangled representation learning, which further separates the latent embedding space into exclusive explainable factors such that each factor only interprets one of semantic attributes of sensory data, has received lot of interest and achieved many empirical successes
static data such as
images (Chen et al,, 2016 Higgins et al. 2017 Dupont 2018; Chen et al.] 2018 Rubenstein et al. 2018b Kim & Mnih 2018). For example, the latent representation of handwritten digits can be disentangled into a content factor encoding digit identity and a style factor encoding handwriting style. In spite of successes on static data, only few works have explored unsupervised representation disentanglement of sequential data due to the challenges of developing generative models of sequential
'Equal contribution.
1 Part of his work was done before joining Tencent.
I His work was done before joining Amazon.
6 His work done before joining Texas A&M University.

---

Published as a conference paper at ICLR 2021
data. Learning disentangled representations of sequential data is important and has many applications. For example, the latent representation of a smiling- faaiing--cce video can be disentangled into a static part encoding the identity of the person (content factor) and dynamic part encoding the smiling motion of the face (motion factor). The disentangled representation of the video can be potentially used for many downstream tasks such as classification, retrieval, and synthetic video generation with style transfer. Most of previous unsupervised representation disentanglement models for static data heavily rely on the KL-divergence regularization in a VAE framework (Higgins et al. 2017 Dupont 2018 Chen et al. 2018 Kim & Mnih 2018), which has been shown to be problematic due to matching individual instead of aggregated posterior distribution of the latent code the same prior (Tolstikhin) 2018 8 Rubenstein et al. 2018b a). Therefore, extending VAE or recurrent VAE (Chung et al.
et al.
2015) to disentangle sequential data in generative model framework (Hsu et al. |017: Yingzhen & Mandi 2018) is not ideal. In addition, recent research (Loocatlllo et al.] 2019) has theoretically shown that is impossible to perform unsupervised disentangled representation learning without inductive biases on both models and data, especially
static data. Fortunately, sequential data such as videos often have clear inductive biases for the disentanglement of content factor and motion factor mentioned in (Locatello et al. (2019). Unlike static data, the learned statie and dynamic factors of sequential data are not exchangeable.
In this paper, propose a recurrent Wasserstein Autoencoder (R- (R-WAE) learn disentangled representations of sequential data. We employ a Wasserstein metric (Arjovsky et al. 2018 Gulrajani al. 20177 Bellemare et al./ (2177) induced from the optimal transport between model distribution and the underlying data distribution, which has some nicer properties (for c.g., sum invariance, scale sensitivity, applicable distributions with non-overlapping supports, and better out-of-sample performance in the worst-case expectation (Esfahani & Kuhn 2018)) than the KL divergence in
VAE (Kingma Welling, 2014) and B B-VAE (Higgins et al. 2017). Leveraging explicit inductive biases in both sequential data and model, we encode an input sequence into two parts: a shared static latent code and a dynamic latent code, and sequentially decode each element of the sequence by combining both codes. We enforce fixed prior distribution for the static code and learn a prior for the dynamic code to ensure the consistency of the sequence. The disentangled representations are learned by separately regularizing the posteriors of the latent codes with their corresponding priors.
Our main contributions are summarized as follows: (1) We draw the first connection between mini- mizing a Wasserstein distance and maximizing mutual information for unsupervised representation disentanglement of sequential data from an information theory perspective: (2) We propose two sets of effective regularizers learn the disentangled representation in completely unsupervised manner with explicit inductive biases in both sequential data and models. (3) We incorporate a relaxed discrete latent variable to improve the disentangled learning of actions on real data. Experiments show that our models achieve state-of-the-an performance in both disentanglement of static and dynamic latent representations and unconditional video generation under the same settings as baselines (Yingzhen & Mandi 2018 Tulyakov et al. 2018).
2 BACKGROUND AND RELATED WORK
Notation Let calligraphic letters (i.e. X¹) be sets, capital letters (i.e. X) be random variables and lowercase letters be their values. Let (Dxx, Pc) be the divergence between the true (but unknown) data distribution Px (density p(x)) and the latent-variable generative model distribution Pa specified by prior distribution Pz (density p(z)) of latent variable Z. Let DEL be KL divergence. Dus be Consen-Shannov divergence and MMD be Maximum Mean Discrepancy (MMD) (Gretton et al. 2007].
Optimal Transport Between Distributions The optimal transport cost inducing a rich class of divergence between the distribution Px and the distribution Pe is defined as follows,
inf W (Pxx, Pa)== TMP(X RxP(X~Px.-~Pa)
L Exxxx-rlxXX 1 Y)).
(I)
where c(X, Y) is any measurable cost function and P(X ~ Px,Y Y ~ Po) is the set of joint distributions of (X, Y) with respective marginals Px and Pa.
Comparison between WAE (Tolstikhin et al., 2018) and VAE (Kingma & Welling 2014) In- stead of optimizing over all couplings I" between two random variables in X, Bousquet al.)
2

---

Published as a conference paper at ICLR 2021
(0017) : Tolstikhin al. (2018) show that it is sufficient to find Q(Z(x) such that the marginal Q(2)= == Ex-Px [012 X is identical to the prior P(Z), as given in the following definition,
Definition 1. For any deterministic (z( and any function G: 2- XX
inf Erx (Q(zx) G(Z)) W(Px, Q:Qz=P2
Definition Dleads to the following loss DWAE of WAE based on Wasserstein distance,
inf Px Lo1z x))c(X, G(Z)) + D(Q2, ((2x))
(3)
where the first term is data reconstruction loss, and the second one is a regularizer that forces the posterior Qz Q 2 X )dPx to match the prior Pz (Adversarial autoencoder (AAE) (Makhzani ct al. 2015) shares similar idea to WAE). In contrast, VAE has a different regularizer Ex (Z( Pz)) enforcing the latent posterior distribution of each input to match Pz.
In (Rubenstein et al. 2018a b). it is shown that WAE has better disentanglement than 3-VAE (Higgins) ct al.] on images, which inspires to design a new representation disentanglement framework for sequential data with several innovations.
Unsupervised disentangled representation learning Several generative models have been pro- posed to learn disentangled representations of sequential data (Denton et al. [2017] Hsu al. [017] Yingzhen & Mandi 2018 Hsieh et al., 2018:) [Sun et al.) 2018) Tulyakov et al.) (0188). FHVAE in (Hsu 2017) is VAE- hierarchical graphical model with factorized Gaussian priors and only focuses on speech or audio data. Our R- R-WAE employing a more powerful recurrent prior can be applied both speech and video data. The models in (Sun al. 2018 Denton et al., 2017 Hsich et al. 2018) are based on the first several elements of a sequence to design disentanglement architectures for future sequence predictions.
In terms of representation learning by mutual information maximization, our work empirically demonstrates that explicit inductive biases in data and model architecture are necessary to the success of learning meaningful disentangled representations of sequential data, while the works in (Locatello et al. 2019 Poole et al., 2019) TTschannnn et al. 2020 Ozair et al. 2019) are about general representation learning, especially on static data.
The most related works to ours are MoCoGAN (Tulyakov et al. 2018) and DS- VAE (Yingzhen & Mandt 2018), which have the ability to disentangle variant and invariant parts of sequential data and perform unconditional sequence generation. Tulyakov et al. (2018) is a GAN-based model that can be only applied to the setting in which the number of motions is finite, and cannot encode the latent representation of sequences. Yingzhen & Mandi (2018) provides a disentangled sequential autoencoder based VAE (Kingma Welling (0014). Training VAE is equivalent minimizing lower bound of the KL divergence between empirical data distribution and generated data distribution, which has been shown to produce inferior disentangled representations of static data than generative models employing the Wasserstein metric (Rubenstein et al.) 2).
3 PROPOSED APPROACH: DISENTANGLED RECURRENT WASSERSTEIN AUTOENCODER (R-WAE)
Given a high- digh-dimennional sequence Fi.7. our goal is to learn a disentangled representation of time- invariant latent code z" and time-varian latent code 2!"h along the sequence. Let =; = (2",²(") be the latent code of x;- Let Xt, Zr, 7 and 7,n be random variables with realizations æ,, 2t, 2" and z!" respectively, and denote D X1.T. To achieve this goal, we define the following probabilistic generative model by assuming Z. and Z are independent.
T
P(XI. ZiT) = P(Z)) II P 22 [=1
(4)
where P(Z = P(Z")
L J Pz Z is the prior in which Z, = (Z",Z,"), and the de- coder model Px( 7,) is Dirac delta distribution. In practice, P(Ze) is chosen N I)
3

---

Published as a conference paper at ICLR 2021
h2
(a) Generative Model (b) Inference Model (c) Weakly Supervised Generative (d) Weakly Supervised Inference
Figure I: Structures of proposed sequential probabilistic models. Sequence i17 is disentangled into static part x" and dynamic parts {x]" j. (a) Sequence is generated by randomly sampling (=", from priors and concatenating them as input into LSTM get hidden state h2 for the decoder; (b) x" is inferred from I1.T with an LSTM, and "!"" is inferred from h, and 2 with another LSTM; (c) is the same as (a) except concatenating additional categorical a; (d) categorical latent variable a is inferred from the dynamic latent codes. The detailed structures of the encoder and decoder are in the supplementary material.
and (Z( 2 = (M. of (Zm)). Ms and σg are parameterized by Recurrent Neural Networks (RNNs). The inference model Q is defined as
(7 Z X1T) Q (7 X NIT)
Q6 T Xt),
(5)
where (2 and (ZT" t Xe) are also Gaussian distributions parameterized by RNNs.
The structures of the generative model (4) and the inference model (5) are provided in Fig. [
3.1 R- R-WAE MINIMIZES A PENALIZED FORM OF A WASSERSTHIN DISTANCE
The optimal transport cost between two distributions Pp and Pa with respective sequential variables XI. (XuT~PP) and YI.T (fTT~P)) is given as follows,
W(Po, Pc) =
inf
B(x.m.M.))-r(((X1.r Yi.T)),
(6)
P(xi:-~Po,hha is set of all joint distributions with marginals Pp and Pg respectively.
When we choose c(æ; y) = || - y 2 (2- (2-Wasserstein distance) and c(X1:T, Y1.)) = E, || Y/ll2 by linearity, it is easy to derive the optimal transport cost for disentangled sequential variables.
Theorem 1. With deterministic P(X.) and any function Y, G((z), we derive
inf W(Po, 602z==Pe..aan P2T
ExppEorravessx ((2. G(Z.))..
where QZi = Qz z is the marginal distribution of Zve when X ~ Pp and Z ~ Q(Zr:lXX::) and Pz, is the prior. Based on the assumptions, we have upper bound,
W(Po, Pσ) S inf Σ EEryEEazzizz.xxx(((.... G(Z)))
QES
where the subset S is S = 1Q Qz- Pz-, Qz Pz Q211 2 12
In practice, we have the following objective function of our proposed R- WAE based on Theorem I
Σ
IDQ2= P2.) B2 Σ P2m
where Do is an divergence between two distributions, and the second and third terms are, respectively, regularization terms for Ze and Z In the following, we will present two different approaches to calculating the regularization terms in section 3.2 and 3.3. Because we cannot straightforwardly
estimate the marginals Q (Z() and ( Z ) we cannot directly use KL divergence in the two regularization terms, but ctt al. 2007] Goodfellow et al. 2014 Nowozin et al. 2016 Arjovsky et al. 2018) when samples from all distributions available.
optimize the RHS of (9) by likelihood-fre optimizations (Gretton
4

---

Published as a conference paper at ICLR 2021
3.2 Dus PENALTY FOR Z AND MMD PENALTY FOR Znh
The prior distribution of Z is chosen as multivariate unit-variance Gaussian, (0, f). We can choose penalty Jas (Q Qze zz. Pz-) for Ze and apply min-max optimization by introducing discriminator Dr (Goodfellow et al. 2014). Instead of performing optimizations in high-dimensional input data space, we move the ad adversarial optimization to the latent representation space of the content with a much lower dimension. Because the prior distribution of {Z,"} is dynamically learned during training, it is challenging to use Js to regularize { with a discriminator, which will induce third minimization within a min-max max optimization. Therefore, we use MMD to regularize {Z,") } as samples from both distributions are easy to obtain (dimension of 2 is less than 20 in our experiments on videos). With kernel k, MMD (Q, P) is approximated by samples from Q and P (Gretton et al. 2007). The regularization terms can be summarized as follows and we call the resulting model R R-WAE(GAN) (see Algorithm[]ja Appendix for details):
D(Qz=,P2-) = Das(Qz PP)); D(Q2T1 Pzt = MMD (QMD)(Q221ZZZ)) (Q2T1 Prizz)
(10)
3.3 SCALED MMD PENALTY FOR 7o AND MMD PENALTY FOR nn
MMD with neural kernels for generative modeling of real-world data (Li ct al. 2017 Binkowski et al. 2018 Arbel et al. 2018)) motivates to use only MMD as regularization in Eq. 9
D(Q2-, P2-) = MMDs,, (2-r, D(Qz7) z Pzrzz/
) = (MZx 2T. Pzrzz) ) (11)
where ky a parametrized family of kernels [Li et al. 2017 Binkowski et al. 2018 Arbel et al. 2018) defined as kt (z. 3) (xff(x)), (z)) and f-y (z) is a feature map, which is more expressive and used for Z with equal or higher dimension than 2,r. The details of optimizing the first term MMDR, (Qzr.P-.) in Eq. 111 is provided in Appendix D based scaled MMD (Arbel al. 2018), a principled and stable technique for training MMD-basee critic. We call the resulting model R. R-WAE(MMD) (see Algorithm 2 in Appendix for details).
3.4 WEAKLY SUPERVISED DISENTANGLEMEST WITH KNOWN NUMBER OF ACTIONS
When the number of actions (motions) in sequential data, denoted by A, is available, we in- corporate a categorical latent variable a one-hot vector whose dimension is A) to enhance the disentanglement of the dynamic latent codes of the motions. The inference model for a designed as go [=(a)
Intuitively, the action inferred from the motion sequence to recognize its label. Learning such categorical distribution requires continuous relaxation of the discrete random variable in order to backpropagate its gradient. Let aj. a be the class probabilities, we
obtain sample ā = (111... .AA) from its continuous relaxation by first sampling g = (g). .... Ia) with ~ Gumbel 1) and then applying transformation á = exp((log aj + g3) (1)/TT //T) E, exp( Oi + gr)/r), where is temperature parameter controlling the approximation. To learn the categorical distribution using the reparameterization trick, we use a regularizer Wir, (go(@) (ā .. x (.p. (((()))) where p(ā) is uniform Gumbel-Softtax prior distribu- tion (Jang et al., 2016; Maddison et al. 20166). The motion variable is augmented as z (=?",, a)), and learning joint continuous and discrete latent representation of image data has been extensively discussed in (Dupont 2018) (see Fig. (c.d) for illustrations).
ANALYZING R- WAE FROM AN INFORMATION THEORY PERSPECTIVE
Theorem 2. If the mutual information (MI) between Zur and Xur is defined in terms of the Zur))
inference model Q T(Zr:; X1T)) = Z Tog Q (( log (ZZrr),, where
Q(XI: ZIT) = Q( X and Q(Zi.)) = Ex.r Q(x1:1 ZIrT), we have a lower bound:
T
I(ZI.: X1.T) 2)
E Ep, [ 1 oog Px -og Dog Q (Z9) log P[
T
Epp EQg (Z. 10g (z) Pot
(12)
5

---

Published as a conference paper at ICLR 2021
7
7
7
7
8
8
8
8
7
7
7
7
8
8
8
(a) t=;; t=10; t550;
t=100
(b) |=1;
1=3;
1=:;
(c) t=
1=3;
t=
Figure 2: Illustration of disentangling the motions and contents of two videos on the test data of SM-MNIST (T 100), Sprites (T = 8) and MUG dataset 8). The first row and fourth row are original videos. The second row and third row are generated sequences by swapping the respective motion variables while keeping content variable the same (sampled at time steps for illustrations).
Theorem shows that R-WAE maximizes a lower bound of the mutual information between X and LT., which theoretically guarantees that R-WAE learns semantically meaningful latent repre- sentations of input sequences. With constant removed, the RHS of (9) and (12) are the same if [ is KL divergence. In spite of being theoretically important, Theorem 2 with KL divergence cannot be directly used for the regularization terms of R-WAE in practice, because cannot straightforwardly estimate the marginals Q (2) and Q Z 2 as discussed previously.
From Eq. 99 and 12
can obtain the following theorem.
Theorem 3. When its distribution divergence is chosen KL divergence, the regularization terms in Eq. (9) jointly minimize the KL divergence between the inference model Q Q(Z(T)Xur) and the prior model P P(Zra) and maximize the mutual information between X1.1 and ZI:r,
Epp (XX(((Z"X))))IP(ZT 1(X Z"), KL(Q(Z,") (22)) --1) (X," ((T)) 1X |
(13)
where the mutual information defined in terms of the inference model as in Theorem
Theorem shows that, even if adopting KL divergence, the regularization the loss of R- WAE still improves over the one in vanilla VAE, which only has the first term in the RHS of Eq. 13) The two mutual information terms explicitly enforce mutual information maximization between input data and unexchangeable disentangled latent representations, 2e and Zi Therefore, R-WAE is superior to recurrent VAE (DS-VAE).
EXPERIMENTS
We conduct extensive experiments on four datasets quantitatively and qualitatively validate our methods. The baseline methods for comparisons are DS-VAE (Yingzhen & Mandi 2018) and MoCoGAN (Tulykovv et al., 2018). We train our models on Stochastic moving MNIST (SM- MNIST), Sprites, and TIMIT datasets under completely unsupervised setting. The number of actions (motions) is used as prior information for all methods MUG facial dataset. The detailed descriptions of datasets, architectures, and hyperparameters are provided in Appendix C, D, and G, respectively.
5.1 QUALITATIVE RESULTS ON DISENTANGLEMENT
We encode two original videos on the first and fourth row Fig. 2 and generate videos on the second and third row by swapping corresponding {20} and {<(")]] between videos for style transfer. Fig. leftt shows that even testing on the long sequence (trained with T 100), our R- R-WAE can disentangle content and motions exactly. In Fig. right), do the same swapping on Sprites. We can see that the generated swapped videos have exact same appearances and actions the corresponding original ones. On the MUG

---

Published as a conference paper at ICLR 2021
dataset, it is interesting to see that can swap different motions between different persons.
Datasets
EER
Datasets
Sprites content
SM-MNIST
Methods
=== 164
x"
161
Methods
actions
digits
FHVAE
5.06%
22.77%
VAE(S)
8.11%
3.98%
3.31%
DAE
5.64%
19.20%
R-WAE(S)
5.83%
2.45 %
1.78% %
R- R-WAE
4.73%
23.41%
DS VAE(C)
10.37%
4.86%
4.26%
R- R-WAB(CC
7.72%
3.31 %
2.84% %
Table 1: EER on TIMIT speech dataset
under the same dimension setting of segment-level = and sequence-leve ="
Table 2: Comparison of averaged classification errors. On Sprites dataset, fix one encoded attribute and ran- domly sample others. On SM-MNIST dataset, we fix the encoded z" and randomly sample the motion sequence from the learned prior Pel (z7") [z =))). We cannot quantita- tively verify the motion disentanglement on SM-MNIST.
for FHVAE (Hsu et al., 2017). DS-VAE (full q) (Yingzhen & Mandt (018) and R WWAA(MMM)), respectively. Small EER is better for 2" and larger EER is better for mm
5.2 QUANTITATIVE RESULTS
SM-MNIST and Sprites Datasets We quantitatively evaluate the disentanglement of our R- WAE(MMD).. In Table 2 "S" denotes simple encoder/decoder architecture, where the encoders in both our model and DSE (Yingzhen & Mandi (018) only use 5 layers of convolutional and deconvolutional networks adopted from DS-VAE (Yingzhen & Mandi 2018). "C" denotes a complex
encoder/decoder architecture where we use Ladder network (Sønderby et al. 2016 Zhao et al. 2017) and ResBlock (He et al. 2016). provided in Appendix E. On SM-MNIST, we get the labeled latent codes } of test videos { } with T = 10 and randomly sample motion variables {>TT]] to get labeled new samples. We pretrain a classifier and predict the accuracy on these labeled new samples. The accuracy on SM-MNIST dataset is evaluated on 10000 test samples. On Sprites, the labels of each attributc(skin colors, pants, hair styles, tops and pants) are available. We get the latent codes by fixing one attribute and randomly sample other attributes. We train classifier for each attribute and evaluate the disentanglement of each attribute. The accuracy is based on 296 x 9 test data. Both VAE and R- WAE(MMD) have extremely high accuracy (99. (9.94%) when fixing hair style attribute, which is not provided in Table 2 due to space limit. As R- WAE(GAN) and R- R-WAE(MMD) have similar performance on these datasets, we only provide the results and parameters of R-WAE(MMD) to save space. There are two interesting observations in Table 2 First, the simple architecture has better disentanglement than the complex architecture overall. The reason is that the simple architecture has sufficient ability to extract features and generate clear samples to be recognized by the pretrained classifiers. But the simple architecture cannot generate high-quality samples when applied to real data. Second, our proposed R- WAE(MDD)) achieve better disentanglement than DS- DS-VAE (Yingzhen) & Mandi 2018) on both corresponding architectures. The attributes within content latent variables are independent, our model can further disentangle the factors. Compared to DS- VAE, these results demonstrate the advantages of R-WAY RAE with implicit mutual information maximization terms. Due to space limit, also include similar comparisons As the number of possible motions in SM-MNIST is infinite and random, we cannot evaluate the disentanglement of motions by training classifier. We fix the encoded motions (>Pir) and randomly
new Moving- Shape dataset in Appendix I.
sample content variables {=e). We also randomly sample a motion sequence {> and randomly
sample contents )= We manually check the motions of these samples and almost all have the same corresponding motion even though the sequence is long (I' = 100).
TIMIT Speech Dataset We quantitatively compare our R-WAE with FHVAE and DS- VAE on the speaker verification task under the same setting as (Hsu et al. evaluation metric is based on equal eтror rate (EER), which is explained in detail in Appendix C. The lower BER on z" encoding the timbre of speakers is better and the higher EER on z" encoding linguistic content is better. From Table our model can disentangle x" and xn well. We can see that our R- R-WAA(MMD) has the best EER performance on both content attribute and motion attribute. In Appendix H we show by style transfer experiments that the learned dynamic factor encodes semantic content which comparable to DS-.
2017 Yingzhen & Mandi 2018). The
MUG Facial Dataset We quantitatively evaluate the disentanglement and quality of generated samples. We train a 3D classifier on MUG facial dataset with accuracy 95. 15.11% and Inception Score 5.20 on test data (Salimans et al. 2016). We calculate Inception score, intra-entropp y is the predicted label and is the generated video, and inter-entropy Hy) (He et al. 2018). For comprehensive quantitative evaluation, Frame-level FID score, introduced by (Heusel et al. (017). is also provided. From Table 2, our R-WAF(MMD) and R-WAAGGAN) have higher accuracy, which
(vylv) v). where
7

---

Published as a conference paper at ICLR 2021
Methods
MocoGAN
DS-VAE(NA)
DS-VAE(W)
R-WAE(MMD)
R-WAE(GAN)
Metries
Accuracy
75.50%
66.73%
82.84
88.62%
90.15%
Intra-entropy.
0.26
0.28
0.23
0.17
0.15
Inter-entropy 1.78
1.77
1.78
1.79
1.79
Inception Score 4.60
4.44
4.71
5.05
5.16
FID
16.95
18.72
14.79
12.21
10.86
Table 3: Quantitative results on generated samples from the MUG facial dataset. "DS- VAE(NA)"
means that number of actions is not incorporated (Yingzhen & Mandt 2018). In "DS- VAE(NA)", samples are generated by fixing the encoded motions and randomly sampling content variable from the prior. Samples DS-VAE(W), R-WAE(MMD) and R-WAE(GAN) are generated by incorporating the prior informationnnumber of actions) into the model.
means the categorical variable best captures the actions, which indicates our models achieve the best disentanglement. In table 2, the Inception score of R-WAE(GAN) is very close to Inception Score of the exact test data, which means our models have the best sample quality. Our proposed R (-WAE(GAN) and R- WAE(MMD) have the best frame-level FID scores, compared with DS VAE and MoCoGAN. The orignal DS- VAE (DS- VAE(NA)) (Yingzhen & Mandt 2018)) without leveraging the number of actions performs worst, which shows that incorporating the number of actions as prior information does enhance the disentanglement actions.
5.3 UNCONDITIONAL VIDEO GENERATION
SM-MNIST dataset Fig. in Appendix E provides generated samples on the SM-MNIST dataset by randomly sampling content { from the prior p(z*) and motions {>ī-r} } from the learned prior (= - The length of our generated videos T 100 and we only show randomly chosen videos of T 20 to save file size. Our R-WAE(MMD) achieves the most consistent and visually best 100. Samples from MoCoGAN (Tulyakov al. 2018 usually change digit identity along the sequence. The reason is that MoCoGAN (Tulyakov et al.) 2018) requires the number of actions be finite. Our generated Sprites videos also have the best results but are not provided due to page limit.
sequence even when
(a) R- WAE(GAN)
(b) DS- VAE
(c) MoCoGAN
Figure 3: Unconditional video generation on MUG dataset, where the sample at time step T = 10 is
chosen for clear comparison. DS- VAE in (b) is improved by incorporating categorical latent variables. Samples of the video sequence are given in Appendix E.
MUG Facial Dataset Fig. 3 and Fig. in Appendix E show generated samples MUG dataset by randomly sampling content { from the prior p(z*) and motions z = (a. xx from the categorical prior (a) (latent action variable is one-hot vector with dimension 6) and the learned prior p (z?") >x))) We show generated videos of length = 10. DS- VAE (Yingzhen & Mandt 2018) used the same structure ours. Fig. 5 shows that DS VAE (Yingzhen & Mandt 2018) and
8

---

Published as a conference paper at ICLR 2021
MoCoGAN Tulyakov et al. 2018) have blurry beginning frames {z,} and even more blurry frames as time t evolves. While our R-WAE(GAN) has much better frame quality and more consistent video sequences. To have a clear comparison among all three methods, show the samples time step = 10 in Fig. 3 and we can see that DS-VAL ES-VAE has very blurry samples with large time steps.
CONCLUSION
In this paper, we propose recurrent Wasserstein Autoencoder (R-WAE) to learn disentangled repre- sentations of sequential data based the optimal transport between distributions with sequential variables, Our theoretical analysis shows that R-WAE simultaneously maximizes the mutual informa- tion between input sequential data and different disentangled latent factors. Experiments on variety of datasets demonstrate that our models achieve state-of-the-art results on the disentanglement of static and dynamic latent representations and unconditional video generation. Future research in- cludes exploring our framework in self supervised learning and conditional settings for text-to-video and video-to-video synthesis.
Acknowledgement Jun Han thanks Dr. Chen Fang at Tencent for insightful discussions and Prof. Qiang Liu at UT Austin for invaluable support.
REFERENCES
Niki Aifanti, Christos Papachristou. and Anastasios Delopoulos. The mug facial expression database. In 11th International Workshop on Image Analyxis for Multimedia Interactive Services WIAMIS 10. IEEE, 2010.
Michael Arbel, Dougal Sutherland, Mikolaj Binkowski, and Arthur Gretton. On gradient regularizers for mmd gans. In NIPS, 2018
Martin Arjovsky, Soumith Chintala, and Léon Boltou. Wasserstein gan. 2018.
2018.
Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf. Tfgan: Improving conditioning for text-to-video synthesis. 2018.
Mare G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The cramer distance solution biased wasserstein gradients. arXiv preprint 1rXiv:1705.10743 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learningg review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 2013.
Mikolaj Binkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.011 0140011 2018.
Olivier Bousquet, Sylvain Gelly, Ilya Tolstikhin, Carl-Johann Gabbiel,,, and Bernhard Schoelkopf. From optimal transport generative modeling: the vegan cookbook. arXiv preprint arXiv:1705.07641 2017.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ICLR, 2019.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In NewrIPS, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In NIPS, 2016.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances Neural Information Processing Systems 28, pp. 2980-2988. Curran Associates, Inc., 2015.
Najim Dehak, Patrick Kenny, Réda Dehak, Pierre Dumouchel, and Pierre Quellet. Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing, 2010.
Emily L Denton et al. Unsupervised learning of disentangled representations from video. In NIPS, 2017.
Emilien Dupont. Learning disentangled joint continuous and discrete representations. In NIPS, 2018.

---

Published as a conference paper at ICLR 2021
Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming, 2018.
John Garofolo. Timit acoustic phonetic continuous speech corpus. Linguistic Data Consortium, 1993, 1993.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, NIPS, 2014.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kemnel method for two-sample-prodee
In NIPS, 2007.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville, Improved training of wasserstein gans. In NIPS, 2017.
Jiawei He, Andreas Lehrmann, Joseph Marino, Greg Mori, and Leonid Sigal. Probabilistic video generation using holistic attribute control. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 452-467, 2018.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatin Lagging inference networks and posterior collapse in variational autoencoders. ICLR, 2019.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In NeurIPS, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Leamingg basic visual concepts with constrained variational framework. In ICLR, 2017.
Devon Hjelm, Alex Fedorov. Samuel Mavoie-Marchildon Karan Grewal, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: (ooo::8088.67770 2018.
Jun-Ting Hsieh, Binghin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In NewrIPS, 2018.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representa- tions from sequential data. In NIPS, 2017.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax 2016.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In ICML, 2018
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. NIPS, 2014.
Chun-Liang Li, Wei-Cheng Chang. Cheng, Yiming Yang, and Barnabás Póczos, Mmd gan: Towards deeper understanding of moment matching network. In NIPS, 2017.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.0077 2016.
Alireza Makhzani, Jonathon Shlens. Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoen- coders. orXiv preprint arXiv: /5/ 1.06644, 151.5445,,,
2015.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? JCML, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ICI.R 2018.
10

---

Published as a conference paper at ICLR 2021
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. I-gan: Training generative neural samplers using variational divergence minimization. NIPS, pp. 271-279, 2016.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasser- stein dependency measure representation learning. Advances Neural Information Processing Systems, 2019.
Henning Petzka, Asja Fischer, and Denis Lukovnicov. On the regularization of wasserstein gans. NIPS, 2017.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In ICML, 2019.
Guo-Jun Qi. Loss-sensitive generative adversarial networks lipschitz densities. arXiv:1701.06266 2017.
arXiv preprint Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In NIPS, 2017.
Paul K Rubenstein, Bernhard Schoelkopf. and Hyy Tolstikhin. On the latent space of wasserstein auto-encoders. arXiv preprint arXiv:1802.00 0020636.. 2018a.
Paul K Rubenstein, Bernhard Schoelkopf, and Ilya Tolstikhin. Learning disentangled representations with wasserstein auto-encoders. 2018b.
Tim Salimans. Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques training gans. In NIP. pp. 2234-2242, 2016.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Spren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In NIPS, 2016.
Ximeng Sun, Huijuan Xu, and Kate Saenko. A two-stream variational adversarial network for video generation. arXiv preprint arXiv:/812.0103: 2018.
Hoang Thanh-Tung, Truyen Tran, and Svetha Venkatesh. Improving generalization and stability of generative adversarial networks. ICI.R, 2019.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bemhard Schoelkopf. Wasserstein auto-encoders. ICLR, 2018.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gclly, and Mario Lucic. On mutual information maximization for representation learning. ICLR, 2020.
Sergey Tulyakow, Ming-Yu Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content video generation. In CVPR, 2018.
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In ICML, 2018.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Leamning hierarchical features from deep generative models. In ICML. 2017.

---

Published as a conference paper at ICLR 2021
APPENDIX FOR RECURRENT WASSERSTEIN AUTOENCODER
APPENDIX PROOF OF THEOREM
In the following, provide the proof of Theorem 1.
Theorem For Pg defined with deterministic Pc(X|Z) and any function Y = =G(Z),
inf W(PD, Pc) Q:Qze=P2- QuTT= -P t=1
Σ Brg E (x1z,x,))(((x G(Z)))]
(14)
where Qz is the marginal distribution of Zı.T when XI.T ~ PD and Z1. ((ZZiT|X111 and Peur is the prior. Based on the assumptions, the constraint set is equivalent
W(PD, Pa) S inf QES
ΣEP Ep, (dx G(Z.))]
(15)
where the set S = Q2- P QZT PZT OZT m Pzrizz
Consider the sequential random variables D - XI.T and YI.T; the optimal transport between the distribution for X1.T and the distribution for YıT induces a rich class of divergence,
Proof:
inf (P', Pc) == PP
EE [x ((x.n...m>-rll(X,n M1.r)]
(16)
where P( PD. MıT ~ Pa) is a set of all joint distributions of (X1:T, MI.T with marginals Po and Pg. respectively.
When we choose c(at, y) || yl1 c(X1:2 YIT) E, Y11 2 by linearity, It is easy to derive the optimal transport for distributions with sequential random variables,
W(Pp, Pa) Q:Q PzuT
inf
ΣE E,,,,x(2,,) G(Z))))
(17)
Based on our assumption, the marginal distribution of the inference model satisfies
Q (ZT) Q(Z))((TT))
ZT) Q(7)) Q z
(18)
The prior distribution P(Z1, ,Zr) satisfies
P(Z1. Zr) = P(Z9) -=)= = P(z))
P(Z.TI
(19)
Since the set S is subset of {Q QZ = Pzur }, we derive the inequality (5))
APPENDIX B: PROOF OF THEOREM 2
In the following, we provide the proof of Theorem 2. To make the notations easy to read, we use the density functions of corresponding distributions.
The joint generative distribution
p( .1.T) = (2
where Pe (Z1:T) is the prior distribution and po ( 21:)) is the decoder model. And the corre- sponding joint inference distribution is go ( #1.T) =
x1.T).
If the MI between 1 and 1. is defined in terms of the inference model g, we have the following lower bound with step-by-step derivations:
9 (0 (x1r)
[ ( B (y( [log
(20)
(0(2).T)
Eg [Dr.. (g (z ( ) 1
go (z
2 Epp pl (1r) + log1(21)) log (217)) llgg -
E [ Pa
(og log BBnn
Σ Ep(P) log
log [0;""""">>> log (D)).
12

---

Published as a conference paper at ICLR 2021
where Bayesian's rule p(Z1:r| (T:r)
po pe ( Maximizing the MI between p(D)
F1T and #1.7 achieves state-of-the-art results in disentangled latent representation by using different
regularizers for the static and dynamical latent variable with different priors (Hjelm et al. 2018). In practice, incorporating the mutual information [[= 2te) between element 2; and motion 2 might facilitate the disentanglement of the dynamical latent variable 2²""
Theorem When its distribution divergence chosen as KL divergence, the regularization terms in Eq. (9) jointly minimize the KT. divergence between the inference model Q Z1r X1.T) and the prior model P(ZI:)) and maximize the mutual information between XI.T and ZI:T,
pp IXL(QQQZZ - ((X 1:7; 2 (I(QQZZ." ~ P(, = pp (KL (1:1)
KL(Q(Z>))
P( P( 2 I(X1:T Z x
Proof: Denote X - XI.T As in the proof Theorem 2, the mutual information between Zi.T and X1:T is defined in terms of the inference model Q. and we use the density functions of corresponding distributions to make the notations easy to read. Thus,
Q(Zi:) Epp
According to the definition of mutual information, we have
log (
1(1i:: 29) = Epp Σ
(D(()1T))((2
log g (11T)
Epp Σ (20211111)
g(2²)
2
(2((TT)))
log g(2%)
Epp Σ ((2 (1TT) log
Epp Σ
g((2 (11r))
p(z*)
p(2²)
x
9x xx1T)
log ((27)
Epp Σ q(< ( log
Σ 7(2)))
p(zc)
p(zc)
Epp
( -
Therefore,
(L(Q(Z>))P((2))
Epp ( ((cci)))--((-iie 1(1:;; 29).
Similarly, we can prove the second equality in the theorem.
APPENDIX C: DATASETS
Stochastic Moving (NIST(SM-MNIST) Dataset Stochastic moving MNIST (SM-MNIST) con- sists of sequences of frames of size 64 x 64 containing one MNIST digit moving and bouncing off edges of the frame (walls). We use one digit instead of two digits because two moving digits may collide, which changes the content of the dynamics and is inconsistent with our assumption. The digits in SM-MNIST move with a constant velocity along a trajectory until they hit at wall at which point they bounce off with a random speed and direction.
Sprites Dataset We follow the same steps as in Yingzhen & Mandi (0118) to process Sprites dataset, which consists of animated cartoon characters whose clothing, hairstyle, skin color and action can be fully controlled. We use 6 variants in each of 4 attribute categories (skin colors, tops, pants and hair style) and there are 6 1296 unique characters total, where 1000 of them are used for training and the rest of them are used for testing. We use 9 action categories (walking, casting spells and slashing, each with three different viewing angles.) The resulting dataset consists of video sequences with T 8 frames of size 64 x 64 x 3.
MUG Facial Dataset We use the MUG Facial Expression Database (Aiantii et al., 2010) for this experiment. The dataset consists of 86 subjects. Each video consists of 50 to 160 frames. To use the same network architecture for the whole video datasets in this paper, we cropped the face regions and scaled to the same size 64 x 64 x 3. We use six facial expressions (anger, fear, disgust, happiness,
13

---

Published as a conference paper at ICLR 2021
sadness, and surprise). To ensure there is sufficient change in the facial expression along video sequence, we choose every other frame in the original video sequences to form training and test video sequences of length T = 10. 80% of the videos are used for training and 20% of the videos are used for testing.
TIMIT Speech Dataset The TIMIT dataset (Garofolo 1993) contains broadband 16k Hz of phonetically-baaancee read speech. total of 6300 utterances (5.4 hours) are presented with 10 sentences from each of 630 speakers. The data is preprocessed in the same way as in (Yingzhen & Mandi 2018) and (Hsu et al., 2017). The raw speech waveforms are first split into sub-sequences of 200ms, and then preprocessed with sparse fast Fourier transform obtain 200 dimensional log- magnitude spectrum, computed every 10ms, i.c., we use T = 20 for sequence T1:T. The dimension of at is 200.
Now we explain the detail of the evaluation metric, equal error rate (EER), used on TIMIT dataset. Let *h be the feature of test utterance test and waarget be the feature of test utterance target predicted identity is confirmed the cosine similarity between w test and w target, cos w test w target) is greater than some threshold used in Dehak et (2010).. The equal error rate (EER) means the false rejection rate equals the false acceptance rate (Dehak et al.) (2010). In the following, we will discuss the two choices of feature w te for evaluations of all methods,
FTTT The
E o in [
which is used to evaluate the disentanglement of 20;
N
E (= mt [<]"),
NT
which is used to evaluate the disentanglement " For more details, please refer to (Dehak et al., 2010: Yingzhen & Mandt 2018 Hsu et al., 2017). We the same network architecture in Yingzhen & Mandt (2018) for a fair comparison on speech dataset. As the input dimension of speech low, the encoder/decoder network is 2-hidden-layer MIPP with the hidden dimension 256.
APPENDIX D: CHOICES OF REGULARIZERS
In the following, we will discuss the choice of regularizers in R-WAE. To make notations easy to read, we use density functions for corresponding distributions. In both R- R-WAE(GAN) and R- WAEEMMDD), we use the same regularizer for (2]") nm =>%),, p(2 2m = We also add KL- assume inference
divergence regularization term on 2 to stabilize training. In the experiments, model gf2 (..)) a Gaussian distribution with parameters mean μ. and diagonal variance matrix ee- Inference model q(2," I = is a Gaussian distribution with parameters mean pm and diagonal variance matrix d For the prior distribution, we assume p(= > is a Gaussian distribution with parameters mean 1% and diagonal covariance matrix at For regularizing the motion variables, just use MMD without introducing any additional parameter, MMD 2212%") p(2?") 2=)))) and we choose mixture of RBF kernel (Li et al., 2017), where RBF kernel is defined as k(æ, y)
With samples {Er) from the posterior ((2") and samples {21) from the prior p(2"), MMD, (((EE))p(())))) is defined as
exp(- || u 20t
MMDD< (())))())))))))))))))))))))))))) nln - E((,, n(n 1)
1
Σ 2(E,,2)) isti
1 k(2,, (21) 112
The difference between R- R-WAE(MMD) and R- R-WAE(GAN) is how to choose metrics for the regu- larizer D(Q z- P2=),, where Pz- is the prior distribution and Qz is the posterior distribution of the inference model.
R- WAE(MMD)
The regularizer D(Qz,,Pz: is chosen as,
D(Q2., MMD ((()),(((
14

---

Published as a conference paper at ICLR 2021
where the scaled MMD MMDk, (Q(Z)).((Z)) is chosen
MMDR. (Qze, Pz-)
MM (Qz., 1 1+ 10E, p I (z
where the function f=)) is the kernel feature map and MMD (Qze, Pz-) is defined in the following. When we have samples (2])i=) from (29) and samples { from P(Z)),
MMD (0(2),, n(nn 11 i79
Σ
(())(() + n(n 1) it1
1
Σ
(2),
(22)
Σ ((CE))))))))
where the RBF kernel l is defined on scalar variables, k(r,, y) = exp( 1ap(-==-wll To avoid the situation where the generator gets stuck on local optimum, we apply spectral parametrization for the weight matrix (Miyato et al. 2018). The feature map f, is updated L steps at each iteration. To overcome posterior collapse and inference lagging, we will update the inference model per iteration of updating the decoder model for L steps during training (He et al. (019).. See Algorithm | for details.
-
R- (-WAE(GAAN For the regularizer ((z)(2zz2)))) we introduce discriminator Dy. The loss is as follows,
-= [log (x + E E3 [l (E,)))
(23)
where p(=") is the prior distribution and (2") stabilize the training of the min-max problem in GAN-based optimization (23), lot of stabilization
) is the posterior distribution of the inference model. To
techniques have been proposed (Thanh- Tung et al. 2019] Mescheder et al. 2018) Gulrajani et al.
2017
Petzka al. 2017 Roth al. 2017: 2017)
Let samples (=²) are from the prior p(=*) and
{ are from the inference posterior q("") In our R- R-WAD(GAN), we will adopt the regularization
from Mescheder al. (2018) and Thanh- Tung et al. (2019).
C - -->AE[]]( D,) l1²],
(24)
where 8 = az + - a) E ((0, 1) and (VD,)a> is evaluated its gradient at the point 2².
Algorithm (--AAAGANN)
Algorithm 2 R- R-WAB(MMD)
Input: regularization coefficient & and con- tent prior p(=))
Input: regularization coefficient 3 and con- tent prior p(=*) Goal: learn encoders. (0(2"(z1r)) and ( feature map fy, and decoder pe (xF where 2,
Goal: learn encoders (0(2) and (27" [2, > prior Ppi (2(") (=))) discrimi-
{0<2"" =%) prior (= [
nator D.,, and decoder pa (=(2)(zt). where 2;
(z" x²
(z (z))
while not converged do
while not converged do for step 1 to L do
for step to L do
Sample batch X = (x+)
Sample batch = (ax)
Sample { from prior ((=)) and {-(")
Sample {zr} from prior p(29) and {<m")
from prior Pv
from prior Pv
Sample { 27T) from encoders ds
Sample (,") from encoders d
Update discriminator D., and encoders
Update feature map f, and encoders 4a
ls with loss given by (9) (T))
with loss given by
end for
end for
Update pa and prior Pv with loss given by p and (TO).
Update pa and prior Pa with loss given by 99) and (IT)
end while
end while
6.1 APPENDIX E: UNCONDITIONAL VIDEO GENERATION
Fig. provides generated samples on the SM-MNIST dataset by randomly sampling content {**)
from the prior p(2") and motions {=m from the learned prior (2))) The length of our
15

---

Published as a conference paper at ICLR 2021
6
4
2
7
>
6
5
5
8
7
5
B
9
7
3
8
5
9
E
6
4
8
3
3
/
9
6
5
6
6
3
7
O
4
8
3
0
3
5
7
0
5
8
3
5
0
5
2
/
4
6
4
8
7
4
0
5
8
(a) R WAE(MMD)
(b) DS- VAE
(c) MoGoGAN
Figure 4: Unconditional video generation on SM-MNIST: (a) Sequences (length=20) in R- WAE(MMD) are randomly taken from generated samples with T 100 to save pdf size; (b) Generated videos by DS- VAE (Yingzhen & Mandi 2018) with T' 20; (c) Generated videos by MoCoGAN (Tulyakov et al. 2018) with = 20. The figures should be viewed with Adobe Reader
to see video.
generated videos is T 100 and only show randomly chosen videos of T 20 save file size. Our R- R-WAE(MMD) achieves the most consistent and visually best sequence even when T = 100. Samples from MoCoGAN (Tulyakov al.) 2018 usually change digit identity along the sequence. The reason is that MoCoGAN (Tulyakov et al. 2018) requires the number of actions be finite.
Fig. 5 shows unconditional video generation with T 10 on MUG facial dataset. DS-VAE in (b) is improved by incorporating categorical latent variables. The figures should be viewed with Adobe Reader to see video.
(a) R-WAEEGANN
(b) DS-VAE
(c) McCoGAN
Figure 5: Unconditional video generation with T 10 MUG facial dataset. DS- OS-VAE in (b) is improved by incorporating categorical latent variables. The figures should be viewed with Adobe Reader video.
APPENDIX F: LATENT MANIFOLD VISUALIZZTION
We encode the test data {wiT}} of SM-MNIST with = 10 to get the content codes {2(} using our R- R-WA. We visualize two-dimensional (2D) manifold of {2*} using (-SNE (Maaten & Hinton 2008 In Fig. 5 different colors correspond to the digit identities of the latent codes - of
test videos SM-MNIST. This indicates that {2*) encoded by our R- R-WAE(MMD) exactly captures the invariant information (digits) of the test data. The latent motion codes are sequential and cannot be visualized.
16

---

Published as a conference paper at ICLR 2021
Figure 6: Visualizing 2D manifold content code {=*} encoded from R-WAB(MMD) on SM- MNIST by t-SNE (Maaten & Hinton 2008).
APPENDIX G: MODEL ARCHITECTURE AND RYPER-PARAMETERS
FC NN
FC NN
ResBlock5
ResBlock1
h
h2
h
ResBlock4
ResBlock2
h2
ResBlock3
ResBlock3
ResBlock2
ResBlock4
ResBlock1
ResBlock5
C
Ct
(a) Encoder Network
(b) Decoder Network
Figure 7: Structures of the encoder network and decoder network. (a) The ResBlock in the encoder network consists of convolutional network adopted from [Bock et al. (2019) named "ResBlock down". After each Resblock, we use FC network get latent feature h' for 0, ... (Ladder Network (Sønderby et al.) 2016 Zhao et al.) (0177), whose dimensions are the same. [h3 ha h h h² h. are concatenated into latent feature hes where h is defined in Fig We use deconvolutional network adopted from [Brock al. (2019), named "ResBlock up". (b), the hidden state fe of an LSTM, defined in Fig evenly split into hh' h² h3 h22 h. h1, And the ResBlock in decoder network consists of deconvolutional network adopted from Brock et al. 2019). We use leaky relu activation for all ResBlocks.
In the inference model. h, defined in [ We use a decoder network to reconstruct é, from the hidden state he, defined in [ For the diseriminator D. in R- R-WAF(GAN),
encoder network, defined in Fig. 7 (a) to extract latent feature
4-layer fully-connected neural
17

---

Published as a conference paper at ICLR 2021
network (FC NN) with respective dimension (256, 256, 128, 1). For the feature map fy with scalar output for the RBF kernel of R- WAE(MMD)) we use a 4-layer fully-connected neural network with respective dimension (256, 256, 128, 1). After encoding 2;, we get extracted latent feature h4 We use Fig. a(a) and Fig. b)) to infer the content variable z" and motion variables =;" When the Gumbel latent variable incorporated into our weakly-supervised inference model, we Fig.Bol to infer the Gumbel latent variable a. The latent content variable x" and latent motion variable "!" are concatenated input to an LSTM after an FC NN to output hidden state h2 for reconstructing S; using the decoder. For our seakly-supervised model, the latent content variable 2", latent motion variable " and latent action variable a are concatenated as input to an LSTM after an FC NN to output hidden state h, for reconstructing åt using the decoder. We use Adam optimizer (Kingma & Ba 2015) with 3² = 0.5 and 3 0.9.
Sprites
Methods R- R-WAL(GAN) (S)
actions
content
3.73%
2.00%
R- R-WAE(MMD) (S)
5.83%
2.45%
R- R-WAE(GAN) (C)
3.13%
3.31%
R- R-WAE(MMD) (C)
7.72%
3.31%
Table 4: Results of (-WAA)GAAN) and R- R-WAF(MMD) on Sprites dataset.
Architecture on SM-MNIST, Sprites and TIMIT Datasets We use the same architecture on SM-MNIST and Sprites dataset, as shown in Fig. 9 The details of the parameters of the networks are provided in Fig. 9 As R-WAE(GAN) and R-WAE(MMD) have similar performance
SM- MNIST and Sprites (see Sprites results in Table 4), we only provide the results and parameters of save space. At each iteration of training the decoder py (g (z,)2z) 21) and the prior Py (2." (==)), we train the encoder parameters 9, and the feature map f, for R WAE(MMD) with L steps. The results SM-MNIST and Sprites datasets are evaluated after 500 epochs. On SM-MNIST dataset, we use a Bernoulli cross-entropy loss and choose I, = 5. The penalty coefficients 31 and 82, are, respectively, 5 and 20. The learning rate for the decoder model
R- R-WAE(MMD)
x 10- and the learning rate for the encoder is x 10 The learning rate for f, is 1 x 10m4, On Sprites dataset, we use an L2 reconstruction loss and choose L = 5 steps. The penalty coefficients 31 and 32 are, respectively, 10 and 60. The learning rate for the decoder model is 3 x 10 and the learning rate for the encoder is 1 x 10-4, The learning rate for D., in R- R-WAE(GAN) or fa R- R-WAE(MMD) decayed learning rate schedule on both datasets. After 50 epochs, we decrease all learning rates by factor of and after 80 epochs decrease further by factor of 5. On TIMIT speech dataset, we use the same encoder and decoder architecture that of DS- VAE. The dimension of hidden states is 256 and the dimensions x and x7" are both 16.
1 x 10-4. We use a
Z
2 x
2c R a
FC NN
NN
FC NN
FC NN +
FC NN
LSTM
LSTM
FC NN
LSTM
LSTM
FC NN
FC NN
Por
h
hr
h+
(a) infer x
(b) infer x
(c) infer
(d) output le for decoder (e) output h; for weakly decoder
Figure 8: Network architectures in addition to encoder/decoder network with ht defined in Fig. 7 (a) Network structure to infer the content variable z" from sequence 3 #1.7; (b) Network structure to infer content variable z!"; (c) In inference model, we introduce an additional Gumbel random variable a inferred by motion sequences {>m" ; (d) Content variable z" and motion variable z!"" are concatenated into an LSTM for the decoder model; (e) In weakly-superrised inference model, content variable 25, motion variable z and Gumbel random variable are concatenated into an L.TT for the decoder model.
18

---

Published as a conference paper at ICLR 2021
ResBlock1 down 64*3*3
FC NN and Reshape input to (N, 2048, 2, 2)
self-attention
ResBlock1 up 1024*3*3
ResBlock2 down 128*3*3
ResBlock2 up 512*3*3
ResBlock3 down 256*3#3
ResBlock3 up 256*3#3
ResBlock4 down 512#3#3
ResBlock4 up 128*3#3
ResBlock3 down 1024*3#3
self-attention
Reshape output to (N, 1024 x 2 x 2)
ResBlock5 up 64*3*3
FC NN
Conv 34343, activation-sigmoid
Table 5: Encoder Network Architecture.
Table 6: Decoder Network Architecture
Figure 9: Network parameters on encoder network and decoder network on SM-MNIST and Sprites datasets. We adopt ResBlock down and up from Brock et al. (2019). The dimensions of =", =²" hr are 120, 12 and 150 respectively. The batch size on both SM-MNIST and Sprites dataset are 60 and the length of video sequence for training T 8.
ResBlock1 down 64*3#3
FC NN and Reshape to (N, 3072, 2, 2)
self-attention
ResBlock1 up 1536*3#3
ResBlock2 down 128*3
ResBlock2 up 768 3*3
ResBlock3 down 256*3#3
ResBlock3 up 384#3#3
ResBlock4 down 512#3#3
ResBlock4 up 192#3*3
ResBlock5 down 102443#3
self-attention
Reshape output to (N, 1024 x 2
ResBlock5 up 96*3*3
FC NN
Conv 3*3#3, activation=siimmod
Table 7: Encoder Network Architecture.
Table 8: Decoder Network Architecture
Figure 10: Network parameters on encoder network and decoder network on MUG facial dataset. We adopt ResBlock down and up from Brock et al. (219). The dimensions of 2 2!" h2, a are 150, 16, 180 and 6 respectively. The batch size on MUG facial dataset are 30 and the length of video sequence for training is T 8.
Architecture on MUG Facial Dataset The details of the architecture parameters of the networks for MUG facial dataset are provided in Fig. g The results on MUG facial dataset are evaluated after 800 epochs. For the regularizer DKL (9 (a 21:11 x. >. xPPr),((a) we choose the coefficient of this categorical regularizer to be 50. We use L2 reconstruction loss and choose L = 5 steps. For R- R-WAE(MMD), the penalty coefficients 33 and 32 are, respectively. 10 and 50. For R- G-WAB(GAN), the coefficients 81 and B32 of the penalties are, respectively, 5 and 60. The learning rate for the decoder model is 5 x 10 and the learning rate for the encoder is 2 x 10 The learning rate for D, in R- R-WAE(GAN) or f- in R- W-AEMMMD 2 x 10-4, We use the same decayed learning rate schedule as described on SM-MNIST and Sprites datasets. This architecture be applied to improve the compression rate (?).
APPENDIX H: ADDITIONAL RESULTS ON AUDIO DATA
Swapping Static and Dynamic Factors on Audio Data Here we present results of swapping static and dynamic factors of given audio sequences. Results are given in Figure IT Each heatmap subplot is of dimension 80 x 20 and visualizes the spectrum of 200ms of an audio clip, in which the mel-scale filter bank features are plotted in the frequency domain (x-axis represents temporal domain with 20 timesteps and y-axis is the value of frequencies). We collect these heatmaps in a matrix where the static factors in a row are kept the same and each column shares the same dynamic factor. It can be observed that in each column, the linguistic phonetic contents as reflected by the formants along x-axis are kept almost the same after swapping. Likewise, the timbres reflected as the harmonics in the spectrum plot. This can be concluded by observing that the horizontal light stripes which represents the harmonics are kept consistent in row. Moreover, we perform
19

---

Published as a conference paper at ICLR 2021
Figure 1I: Cross generation of 16 audio clips forms 17 x 17 matrix. The first column and the first row are spectrum visualization of the original sequences. Subplot at the (i+ 1) row and (j + 1) column represents the reconstruction of i-th static factor and j-th dynamic factor.
20

---

Published as a conference paper at ICLR 2021
identity verification experiment as conducted in DS VAE (Yngghen & Mandt 2018 Similar to cross reconstruction, "fomat and maale (or ffomale and frmale in DS- VAE) are swapped for two sequences {reemale } and {T J. By informal listening test of the original-swapped speech sequence pairs, we confirm that the speech content is preserved and identity is transferred (i.e. female voice usually has higher frequency).
APPENDIX I: ADDITIONAL RESULTS ON MOVING-SHAPE VIDEO DATA
Static Factor Pred. Acc.
Dynamic Factor Pred. Acc.
DS- VAE (TFGAN) DS-VAE (BigGAN) R-WAE (TEGAN)
77.47%
72.45%
75.37%
70.85%
80.50%
83.60%
R-WAE (BigGAN)
75.27%
80.00%
Table 9: Prediction accuracy generated video data, the experiment setting here is similar to Table the main text. For predicting the static factor, we fix the static latent representation 2" and randomly sample zm and examine whether the static information preserved in the generated video (if so, the static attributes should be correctly predicted by a pretrained video classifier). For predicting the dynamic factor, we perform corresponding experiments analogously.
DS-VAE
R-WAE
Fix Content Sample Motion
Figure 12: Results of fix =f and sample =" using TFGAN (Balaji et al. 2018) architectures. The first in each subfigure real video sequences. The generated motion of moving objects by DS-VAE contains abrupt jumps and is not smooth, while R- R-WAE is able to generate motion of various types including zig-zag -ag, diagonal and straight line.
Generation Results on Moving Shapes We report results on a Moving Shange-SSheeeaa dataset in Table 9 and Fig. 12 The Moving-Shape synthetic dataset was introduced in [Balaii et al. (2018) which has control parameters: shape type (e.g. triangle and square), size (small and large), color (c.g. white and red), motion type (e.g. zig-zag, straight line and diagonal) and motion direction. In Table TFGAN (Balaji et al. (018)) encoder and decoder architectures are considered less expressive compared with BigGAN (Brock et al.) 2019) architectures. Similar to results in Table 2, with more complex and expressive architecture, learning disentangled representation is harder. The results and Fig. 12 demonstrate that R- WAE produces better disentanglement and generation
in Table 9 performance than DS VAE both quantitatively and qualitatively. Qualitative difference of fixing T and sampling zd for DS- VAE and R- WAE not that obvious and thus not shown.
21


Reviews:
[{"review_id": "NkTLPf0DCy0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes R-WAE to learn disentangled representations. Benefit from the characteristics of WAE, this paper shows that R-WAE can better disentangle the sequential data into content space and motion space. R-WAE achieves state-of-the-art performance in both disentanglement and unconditional generation.\n\n\nReasons for score: \nOverall, I vote for acceptance. The proposed method has theoretical foundations and shows excellent results.\n \n\nPros: \n \n-- The paper provided strict theoretical formulations, like the comparison between WAE and VAE, how WAE can generalize to the sequential format, and the connection between mutual information(MI) and the objective function of R-WAE. The experiments support the theorems.\n \n-- The authors provide sufficient experiments on multiple datasets. \nThe experiments cover tasks of various domains, including video and audio, which indicate the proposed method could be easily generalized to different tasks.\n \n-- Many architectures are investigated, including WAE-MMD, WAE-GAN, and simple/complex encoder.\n\n\nCons: \n\n-- When generation (Fig. 1 (a)), the dependency between h in different time steps is considered. However, during the inference phase (Fig. 1 (b)), the dependency is ignored. Any good reason? \n\n-- The illusions of Figure 5 do not keep the same across frames, while z^m shouldn’t change illusion. More analysis and discussion are probably needed for this result.\n \n-- The comparison between R-WAE(GAN) and R-WAE(MMD) can be further discussed. The comparison is shown in Table 3. The results show that R-WAE(GAN) performs better, but the reason is unclear. \n \n-- For the audio experiments, ASR results or phoneme classification might be needed to support that z^m keeps the local information. It would be better to provide audio demos of your cross reconstruction result shown in Figure 11.\n\n-- Figure 6 shows that WAE usually gives a tighter gap between classes of z^c, since WAE computes divergence between Q(Z^c) and P(Z^c), which causes z^c spread over the entire latent space; while VAE gives a noticeable gap between different classes of contents (such as the one shown in Figure 3 of scalable FHVAE), which leaves space for the unseen contents. Would the tighter gap cause issue when training and testing data are mismatched? If the testing data has some contents that have never been seen during the training, what would happen? (for example, training on MUG than testing on VoxCeleb video data) Will the z^c of the unseen content data be forced mapped to the content in the training set instead of keeping its own information?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "M2v8oJlfmHG", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on learning disentangled representations for sequential data. This paper proposed recurrent Wasserstein Autoencoder (R-WAE), which disentangles the factors of variations into time-variant and time-invariant ones. The theoretical analysis is provided by the method. Experiments show that R-WAE outperforms baselines on several datasets.\n\nIn general, the paper is easy to follow. The method looks relatively straight-forward, because it extends a recurrent VAE framework by replacing the reconstruction loss term with a Wasserstein distance. The superiority of the method is supported by both theoretical analysis and experimental performance.\n\nOn page 6, the authors claim that the regularization term in R-WAE is superior to a VAE model. But it is not clear to me why a recurrent VAE model cannot adopt the same regularization term. If this is possible, then I believe an ablation study is useful. Because it helps us understand the performance improvement is due to the introduction of Wasserstein distance or the regularization or both.\n\nIn section 3.4, it is mentioned that the number of actions is provided as weak supervision. Since it looks like the supervision does not contain much information, I am wondering how the performance will be impacted if such supervision is not provided. In addition, I want to know whether the learned categorial variable $a$ is consistent with the ground-truth actions for the data.\n\nIn table 3, intra-entropy $H(y)$ is reported. I am not sure why we should care about this value and why a lower value implies better performance. It looks to me that a lower $H(y)$ means that the model is more likely to give the same prediction for different samples. It is not clear to me why we want the model to have such property.\n\nIn general, I suggest accepting this paper, because the proposed method is supported by theoretical analysis and gives a good experimental performance.\n\n\n\nMinor:\n\nEq. (13) involves the mutual information term $I(X_{1:T};Z^c)$. Which distribution does $Z^c$ follows? It is the posterior distirubiton $P(Z^c | X_{1:T})$ or its variational approximation $Q(Z^c | X_{1:T})$?\n\nOn page 6, Eq. (21) should be changed to Eq. (13).\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "K04E8qXlLjk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper extends the Wasserstein autoencoder for learning disentangled representations from sequential data. The latent variable model considered contains separate latent variables capturing global and local information respectively, each of which is regularized by a divergence measuring the marginal posterior $Q_z$ and the prior $P_z$. An optional auxiliary discrete latent is introduced to incorporate inductive bias for discrete local features (e.g., type of actions). To estimate the divergence terms, the authors propose to use MMD for the recurrent local latents since the prior distribution evolves over time; for the global latent, the authors presented two options: discriminator-based Jenson-Shannon Divergence estimate (the same as adversarial autoencoder proposed in Makhzani et al., 2016) and scaled MMD (Arbel et al., 2018). The connection between the proposed objective and mutual information maximization is outlined in Section 4. Experimental results show that the proposed R-WAE model outperforms baseline DS-VAE/FHVAE/MocoGAN\n\nPros:\n- Extending Wasserstein autoencoder to model sequential data and learn disentangled representations is novel and well-motivated\n- Experimental results demonstrate the advantage compared with the baseline models in terms of disentanglement performance and generated data quality\n\nCons:\n- The appendix requires proofreading - the authors left derivations in the appendix, but it contains incomplete sentences and inconsistent notations. For example, in Appendix D, “We just use MMD without…” is incomplete. $P_{Z^c}$ and $Q_{Z^c}$ are used as example for the vanilla MMD computation while according the main text $P_{Z^m}$ and $Q_{Z^m}$ should be the one whose penalty is computed with that.\n- Ablation studies are lacking. How sensitive the performance is with respect to the latent variable dimensions and whether the tricks (updating encoder for $L$ times) are used or not. It is hard to conclude whether the proposed formulation is the key to the superior performance.\n- How is the performance like when using the vanilla MMD divergence for $Z^c$ without a neural kernel? The authors justify the decision by saying the scaled MMD is more expressive and proper for higher dimensional $Z^c$. It will be nice to show the benefit empirically; moreover, the $Z^c$ used for TIMIT and Sprites are not of higher dimensions than those for $Z^m$.\n- Why is R-WAE (GAN) only shown for the MUG dataset? Can authors also present the results for TIMIT/Sprites/SM-MNIST?\n- This paper should cite adversarial autoencoder (Makhzani et al., 2016) for the discriminator based JS divergence estimate, which proposed the same penalty first.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach for learning disentangled static and dynamic latent variables for sequence data. In terms of learning objective, the paper extends Wasserstein autoencoder to sequential data, and this approach is novel and well-motivated; the aggregated posterior for static variables comes out naturally and plays an important role for regularization (this appears to be new for sequence data). The authors also studies how to model additional categorical variables for weakly supervised learning in real scenarios. The main steps (generation and inference) were illustrated by graphical models with clarity, and rigorous statements are provided to back them up. Experimental results demonstrate the advantages of proposed method, in terms of disentanglement performance and generation quality.

The reviewers think this paper makes nice contributions to the sequential generative model community.
Decision: Accept (Spotlight)