Contrastively Disentangled Sequential  Variational Autoencoder
Authors: Junwen Bai
Weiran Wang
Carla P Gomes
Venue: NeurIPS 2021 Poster (2021)
Keywords: contrastive learning
sequential VAE
disentanglement
mutual information
self-supervised learning
generative models

Abstract: Self-supervised disentangled representation learning is a critical task in sequence modeling. The learnt representations contribute to better model interpretability as well as the data generation, and improve the sample efficiency for downstream tasks. We propose a novel sequence representation learning method, named Contrastively Disentangled Sequential Variational Autoencoder (C-DSVAE), to extract and separate the static (time-invariant) and dynamic (time-variant) factors in the latent space. Different from previous sequential variational autoencoder methods, we use a novel evidence lower bound which maximizes the mutual information between the input and the latent factors, while penalizes the mutual information between the static and dynamic factors. We leverage contrastive estimations of the mutual information terms in training, together with simple yet effective augmentation techniques, to introduce additional inductive biases. Our experiments show that C-DSVAE significantly outperforms the previous state-of-the-art methods on multiple metrics. 

Contrastively Disentangled Sequential Variational Autoencoder
Junwen Bai Cornell University jb2467@corne11.c edu
Weiran Wang
Carla Gomes Cornell University gomesêcs cornell.
Google weiranvangggoogle.ccom com
Abstract
Self-supervised disentangled representation learning is a critical task in sequence modeling. The learnt representations contribute better model interpretability as well as the data generation, and improve the sample efficiency for downstream tasks. We propose novel sequence representation learning method, named Con- trastively Disentangled Sequential Variational Autoencoder (C-DSVAE), to extract and separate the static (time-invariant) and dynamic (time-variant) factors in the latent space. Different from previous sequential variational autoencoder methods, we use novel evidence lower bound which maximizes the mutual information between the input and the latent factors, while penalizes the mutual information between the static and dynamic factors. We leverage contrastive estimations of the mutual information terms in training, together with simple yet effective augmenta- tion techniques, to introduce additional inductive biases. Our experiments show that C-DSVAE significantly outperforms the previous state-of-the the--rtt methods on multiple metrics.
Introduction
The goal of self-supervised learning methods is to extract useful and general representations without any supervision, and to further facilitate downstream tasks such as generation and prediction [[] Despite the difficulty of this task, many existing works have shed light on this field across different domains such as computer vision 2 3 5), natural language processing [ 7 g and speech processing 9 10
(also see huge number of references in these papers). While the quality of the learnt representations improves gradually, recent research starts to put more emphasis on learning disentangled representations. This is because disentangled latent variables may capture separate variations of the data generation process, which could contain semantic meanings, provide opportunity to remove unwanted variations for a lower sample complexity of downstream learning 14 15), and allow more controllable generations [16] 17 18]. These advantages lead to rapidly growing research area, studying various principles and algorithmic techniques for disentangled representation learning [19 20 21 22
II
12 33
23 24 25 26). One concern raised in (24] is that without any inductive bias, it would be extremely hard to learn meaningful disentangled representations. On the other hand, this concern could be much alleviated in the scenarios where the known structure of the data can be exploited.
In this work, we are concerned with the representation learning for sequence data, which has unique structure to utilize for disentanglement learning. More specifically, for many sequence data, the variations can be explained by dichotomy of static (time-invariant) factor and dynamic (time- variant) factors, each varies independently from the other. For example, representations of video recording the movements of cartoon character could be disentangled into the character identity (static) and the actions (dynamic). For audio data, the representations shall be able to separate the speaker information (static) from the linguistic information (dynamic).
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

1
X2
decoder
motion augmentation
content augmentation
Z1
7/2
Zf
S
Z
Z
S
LT
encoder
LSTM
LSTM
LSTM
X1
22
X1-T
Figure 1: The illustration of our C-DSVAE model. Left panel the general structure of the sequence- lo-sequence auto-encoding process: each frame is passed to the LSTM cell; dynamic factors 21.T are extracted for each time step; the static factor is extracted by summarizing the full sequence; the generation/reconsruutiio of frame depends on and 2,. Right panel depicts the contrastive learning module of C-DSVAE: #1.T the motion augmentation of 21.7 and æi.r the content augmentation of 21:7; dynamic factors 21:T of 11-T can be seen as the positive sample for the anchor 21.7 in the contrastive estimation w.r.t. the motion, and similarly the static factor of ai.r can be viewed as the positive sample for w.r.t. the content.
We propose Contrastively Disentangled Sequential Variational Autoencoder (C-DSVAE), a method seeking for clean separation of the static and dynamic factors for the sequence data. Our method extends the previously proposed sequential variational autoencoder (VAE) framework, and performs learning with a different different evidence lower bound (ELBO) which naturally contains mutual information (MI) terms to encourage disentanglement. Due to the difficulty in estimating high dimensional complex distributions (c.g., for the dynamic factors), we further incorporate the contrastive estimation for the MI terms with systematic data augmentation techniques which modify either the static or dynamic factors of the input sequence. The new estimation method turns out be more effective than the minibatch sampling based estimate, and introduces additional inductive biases towards the invariance. To our knowledge, we are the first to synergistically combine the contrastive estimation and sequential generative models in principled manner for learning disentangled representations. We validate C-DSVAE on four datasets from the video and audio domains. The experimental results show that our method consistently outperforms the previous state-of-the-art (SOTA) methods, both quantitatively and qualitatively.
2 Method
We denote the observed input sequence as Z1.7 = {I1, 22, .T} where x, represents the input feature time step (c.g., 24 could be video frame or the spectrogram feature of a short audio segment), and I' is the sequence length. The latent representations are divided into the static factor and the dynamic factors ž1r where z; the learnt dynamic representation time step i.
2.1 Probabilistic Model
We assume that in the ground-truth generation process, 2, depends こςι = {20, 21, .... 21-1) where 2o = O, and the observation æ; independent of other frames conditioned on z; and s. Furthermore, assume the static variable and the dynamic variables 21.7' independent from each other, i.e.,
p(s, 21.1) = (())p(2):r) Formally, let 2 = (s, 21:7) and we have the following complete likelihood
p 2) =)(2)(((=)1)))
p(s)) II ((alkee)) [(
(I)
where p(z) is the prior for sampling z. Our formulation captures the general intuition that we can separate the variations of the sequence into the time-dependent dynamic component (described by 2 2('s), and static component (described by 8) which remains the same for different time steps in the same sequence but differs between sequences. For example, in the cartoon character video

---

(see FigureB), the hair, shirt and pants should be kept the same when the character walks around, but different videos can have different characters. Similarly, for a speech utterance, the phonetic transcription controls the vocal tract motion and the sound being produced over time, but the speaker identity remains the same for the whole utterance. In this work, we refer to the dynamic component as "motion" and the static component as "content".
For the prior distributions, we choose p(s) to be the standard Gaussian NO, I). and p(z. (<()) to be ((u(=<ss, (z<+)) where (u(-) and (.) are modeled by LSTMs [27]. In operations, the drawn sample s shared throughout the sequence. To draw a sample of 24, we first take sample of 2;-1 as
the input of the current LSTM cell. Then forwarding the LSTM for one step gives us the distribution of 2,, from which we draw a sample with the reparameterization trick [28].
To extract the latent representations given only the observed data F1. where the motion and content mixed together, we hope to learn a posterior distribution (=1(x1:)) where the two components disentangled. That is, similar to the prior, our posterior should have a factorized form:
T
(=[[21.7) (( s/xrrr) =
= g(s(<1r)) II (( zg.).
(2)
[=1
The posterior distributions are also modeled by LSTMs. A nested sampling procedure, resembling that of the prior, applied to the posterior of dynamic variables. Similar parameterizations of dynamic variables by recurrent networks have been proposed in prior works
(29 30
31. The standard
loss function for learning the latent representations is an ELBO 32 (also see a derivation in
Appendix AA.I: D::
max P.4
Exrropo Eg(( ()) (log((xirla)) - K [[g(())r)))
(3)
where pD is the empirical data distribution. Under the parameterization of the posterior where s and Z1:T are mutually independent, the KL-divergence term reduces to
kLig(=|=1 (tt///=rr)))(p(()) = K [L [Ip(s)] + KL Lggg(z (1rr ((P)(((
(4)
where the second term is approximated with the sampled trajectories of the dynamic variables 21:7.
2.2 Our approach: Mutual information-base disentanglement
Several existing sequence representation learning methods 33 33 18] are built on top of the loss function 33. However, this formulation also brings several issues. The KL-divergence regularizes the posterior of the static or dynamic factors to be close to the corresponding priors. When modeling them with powerful neural architectures like LSTMs, it possible for the KL-divergence to be close to zero, yet at the same time, the posteriors become non-informative of the inputs; this is common issue for deep generative models like VAE (34) Techniques have been proposed alleviate this issue, including adjusting the relative weights of the loss terms to regularize the capacity of posteriors
20. replacing the individual posteriors the KI. terms with aggregated posteriors 35
36
37 and
enforcing latent structures (such as disentangled representations) in the posteriors 23 22
38
Our principled approach for learning useful representations from sequences is inspired by these prior works, and at the same time incorporates the unique sequential structure of the data. Without inductive biases, the goal of disentanglement can hardly be achieved since it is possible to find entangled s and 21:7 that explain the data equally well (ín fact, by Theorem of [24]. there could exist infinitely many such entangled factors). The problem may seem less severe in our setup, since is shared across time and it is hard for such s to capture all dynamics. Nevertheless, since both and 2 are used in generating z;, it is still possible for 2, to carry some static information. In the extreme case where each 2; encompasses 8, would no longer be indispensable for the generation. This issue motivated prior works to optimize the (estimate of) mutual information among latent variables and inputs 33 18 39.
Our method seeks to achieve clean disentanglement of and Z1.T by optimizing the following objective function, which introduces additional MI terms to the vanilla ELBO in (3)
max Em.ropo E logp(21r(()) - 100--KKA KL[g(=)llp(2))
P-9
Emprppp By(= in [ooo/o(((prll)l (KL
K (Llg(aur(a (p(()))))
+ I (s; (.)) + [,( - I, ())
(5)
3

---

where the aggregated posteriors are defined as g(2) q(s, æ1:r) =
q(s) pp (1)())1))))))....)))))))))())))) ( pp (g(2): [1.T]], and the MI terms are defined as [s: 21.7)
B log <( (21:1; (1:T) (( [log s(uurlr))) (and (2(21: () defined
((a)
9(#1/7)
similarly). The intuition behind ( is simple: besides explaining the data and matching the posteriors
with priors, 21.T and shall contain useful information from Ii:T while excluding the redundant information from each other. We further justify (5) by showing that it still forms a valid ELBO.
Theorem 1. With our parameterization of (s, 21:T), (5) is valid lower bound of the data log- likelihood Eurr=pp log(Fi:7).
The full proof can be found in Appendix A.2 With this guarantee, can follow the spirit of [20 23 to add and adjust the additional weight coeffcients a to the KL terms, 3 to the MI terms Ig F1.T) (21:1; (1:r), , and y (=):
Berroro By(s|enr ( + (1,
(6)
(21::) -1,
It remains to estimate the objective 6 for optimization. The KL term for 21.r is estimated with
the standard Monte-Carlo sampling, using trajectories of 21:T [2). The KL term for s can be estimated analytically. For the MI terms, attempt two estimations. The first estimation uses standard mini-batch weighted sampling (MWS) following 23 of 1 and the complex dependency among time steps, it may be hard for MWS estimate the distributions accurately, so we also explore non-parametric contrastive estimations for Ig (8; (1:2) and Ig (21:2: with additional data augmentations, which we will detail below.
33 Due to the high dimensionality
2.3 C-DSVAE: Contrastive estimation with augmentation
A contrastive estimation of ( can be defined as follows
C(aLT) = Epp log ( vt.) Σÿ= (121, 21.2
+ log(n + 1)
(7)
where x is "positive" sequence, while x² l,..., n is collection of n "negative" sequences. When ((21:T, (1:T) ((arrlzur) one can show that (7) approximates T (1:1:, see Ap- pendix fo4 a proof. To implement (7), 21:T is the trajectory obtained by using the mean each time step from ((=))( for a input sequence F1.7, while rH.r a randomly sampled sequence from the minibatch. A similar contrastive estimation is defined for as well. To provide meaningful positive sequences that encourage the invariance of the learnt representations, we obtain ii by systematically perturbing 21.7. In Sec A.4 we discuss how the augmented data give good estimate of 7 under additional assumptions.
q(>i/r)
Content augmentation The static factor (e.g., the character identity in videos the speaker in audios) is shared across all the time steps, and should not he affected by the exact order of the frames. We therefore randomly shuffle or simply reverse the order of time steps to generate the content augmentation of I1.7 and denote it by i..T The static and dynamic latent factors of zf.7. modeled by q, are denoted by sf and zf.7.. This is an inexpensive yet useful strategy, applicable both audios and videos. Similar ideas were introduced in 33 albeit not used for contrastive estimations.
Motion augmentation In motion augmentation, we would like to maintain the dynamic factors (c.g., actions movements) while replacing the content with meaningful alternative. Thanks to the recent efforts in contrastive learning, multiple effective strategies have been proposed. For video datasets, we adopt the combination of cropping, color distortion, Gaussian blur and reshaping 4 40). For audio datasets, we use classical unsupervised voice conversion algorithms 4 42). The motion augmentation of æ1.r is denoted by 2TT. with the static factor s and dynamic factors 21.7 estimated by q.
Note that .. is the motion augmentation of æır,, and øir in turn is also the motion augmentation of T-r Similarly, 8 and are mutually the "positive" sample to each other w.n.t. the static factor. During the training, it is efficient to generate the augmentations for each sequence in the minibatch,

---

45
45
4
5
5
4
5
45
25
5
5
5
45
5
45
45
45
45
5
45
45454
5
5
45
Data augmentations on SM-MNIST and Sprites. Left panel: SM-MNIST. The first row the raw input sequence of moving digits. The second reverses the sequence order of the raw sequence, serving as the content augmentation. The third row stretches the frames and enhances the color, which changes the digit styles but does not change the digit movements and thus forms a motion augmentation. Right panel: Sprites. The first row is the raw input character video. The second row is the content augmentation with random order. The third row motion augmentation produced by distorting colors and adding Gaussian noise.
Figure 2:
and apply contrastive estimation on both the original data and the augmented data. This leads the following final estimates:
+
( ~ (((())+(()))
(8)
where we use ((21: (Z1:T), 2irr) exp(sim(21:r //T) with indicating the latent variables for the positive/augmented or negative sequences (extracted from q). sim(-..) .) is the cosine sim- ilarity function and r 0.5 temperature parameter. This form of & using cosine similarity is widely adopted in contrastive learning 4 43
as it removes one degree of freedom (length) for high dimensional feature space. Plugging 8 into ( gives our final learning objective with con- trastive estimation. We name method Contrastively Disentangled Sequential Variational Encoder (C-DSVAE), and a full model illustration is shown Figure !
In practice, we find the contrastive estimation is more effective than the MWS based on Gaussian probabilities (see an ablation study in Appendix F. Interestingly, we observe that with only the contrastive estimation and augmentations, Ig (21r3; (last term in (5)) decreases even if we do include its estimation in our optimization process (see Appendix E} indicating that a good disentanglement between 21.7" and could be attained through contrastive estimations solely.
As will become evident in the experiments, and static factors than previous methods. We believe this is partly due to the inductive bias introduced by our approach: in order to consistently map two sequences with the same motion but different contents (which vary independently from the motion according to the generation process ( to the same latent representation 21,T, must discard the information of the inputs regarding s; similar arguments hold for the static variables. At the same time, we enforce that the extracted 21:2 and together should reconstruct F1.T that no information lost the auto-encoding process. To our knowledge, we are the first to use the contrastive estimation with augmentations in the sequential VAE scenario, for pushing the disentanglement of variations.
method achieves cleaner separation of the dynamic
3 Related Work
In terms of the general frameworks, disentangled representation learning methods can be categorized into GAN-based [44] and VAE-based (28) approaches. GAN- based methods like MoCoGAN [45] aim at generating videos from content noise and motion noise. These models typically do not explicitly learn the encoder from inputs to the latent variables, rendering them less applicable for representation learning.
The VAE framework parameterizes both the encoder (variational posterior) and the decoder (recon- struction), and optimize them jointly with a well-defined ELBO objective. The encoder allows us to directly control and reason about the properties of the extracted features, which can be straight- forwardly used for downstream tasks. Many variants of VAE were proposed to encourage the disentagleneen//nterrezbliill of individual latent dimensions. 3-VAE (00) adds a coefficient the per-sample KL-divergence term to better constrain the information bottleneck. FactorVAE [22] and
5

---

explicitly separate out the total correlation term, and adjust its weight coefficient in the objective to encourage the dinensnnsional disentanglement of the aggregated posterior. From a theoretical standpoint, [24] points out that without further inductive biases, it is impossible to learn disentangled representations with the standard VAE.
3-TCVAE [33
To handle the sequence data, vanilla VAE is extended to the recurrent version, and prior works have explored different approaches for separating the content and the motion. FHVAE [46) designs hierarchical VAE model which uses "sequence-deeeendett variables" (corresponding to the speaker factor) and "sequence- independent variables" (corresponding to the linguistic factors) for modeling speech sequences, and performs learning with the per-sequence ELBO while respecting the hierarchy in designing the posterior. FHVAE does not use additional loss terms for encouraging the disentan- glement. Different from FHVAE, VS [32 explicitly models the static and dynamic factors in its graphical model and its posterior has factorized form. few recent works (including ours) inherit the clean formulation of DS in terms of the graphical model and prior/posterio parameterization. V3E 33 introduces additional loss terms to the objective of DS VAE in an ad-hoc fashion, such as triplet loss that encourages the invariance of the learnt static factors by permuting the frame order (spiritually similar to our contrastive estimation for (s, (:T)). an additional prediction loss on 21.7 leveraging external supervision for the motion labels, and an MI term of [(s, =1:T) to be minimized. Our C-DSVAE differs from S3VAE in that we naturally introduce the MI terms from the fundamental principle of VAE, and augmentations instead of external supervision for learning the static and dynamic factors. Also based on the DSVAE formulation, R- WAE [8] proposes to replace the distance measure between the aggregated posterior and the prior with the Wasserstein distance, in the belief that the KI. divergence is too restrictive. In R-WAF, Maximum Mean Discrepancy (MMD) based estimation or GAN-based estimation of Wasserstein distance is used, either of which requires heavy tuning of hyperparameters. Another similar method, IDEL [47). optimizes the additional set of MI terms similar ours. However, in our work, show that these MI terms can be naturally derived from the fundamental principle of VAE, and we additionally propose the contrastive estimation and data augmentations strengthen them, which was not done before.
beequence-independen
Another relevant research area is nonlinear ICA, which tries to understand the conditions under which the disentanglement can be achieved, possibly with contrastive learning or VAE [88 49 Our general approach is well-alis well-aligned with the current understanding in this direction. That is, while general disentanglement (per-dimensional disentanglement) is hard achieve without stringent assumptions on the data generation process, we can seek some certain level of disentanglement (in our case, the group-wise wise disentanglement between the static and dynamic factors) given auxiliary information (which we provide through augmentations). Our intuitive arguments for the reason of separation (see the last paragraph of Sec 2.3) show that VAE and contrastive learning may be complementary to each other, and combining them could potentially gain better disentanglement.
50 25 26).
Experiments
We compare C-DSVAE with the ofate-of-the-art sequence disentanglement learning methods: FH- VAE (46). MoCoGAN (45), DSVAE 32]. S3VAE 33 and R-WAE (18), with the same experimental setups used by them.
4.1 Datasets
Sprites [1 is a cartoon character video dataset. Each character's (dynamic) motion be catego- rized into three actions (walking, spellcasting, slashing) and three directions (left, front, right). The (static) content comprises of each character's skin color, tops color, pants color and hair color. Each color has six variants. Every sequence is composed of 8 frames of RGB images with size 64 x 64.
MUG [52 is facial expression video dataset. The static factor corresponds to single person's identity. Each individual performs six expressions (motion): anger, fear, disgust, happiness, sadness and surprise. Following (45). each sequence contains 15 frames of RGB images with size 64 x 64
(after resizing).
SM-MNIST (53) Stochastic Moving MNIST) is dataset that records the random movements of two digits. Each sequence contains 15 gray scale images of size 64 x 64. Individual digits are collected from MNIST.

---

Table Disentanglement metrics on Sprites.
Table 2: Disentanglement metrics on MUG.
Methods
Acct
ISt
H(ylx) H(y)
Methods
Acct
1St
H(ylx), H(y)
MoCoGAN
92.89%
8.461
0.090
2.192
MoCoGAN
63.12%
4.332
0.183
1.721
DSVAE
90.73%
8.384
0.072
2.192
DSVAE
54.29%
3.608
0.374
1.657
R-WAE
98.98%
8.516
0.055
2.197
R-WAE
71.25%
5.149
0.131
1.771
S3VAE
99.49%
8.637
0.041
2.197
S3VAE
70.51%
5.136
0.135
1.760
C-DSVAE 99.99% 8,871 0.014
2.197
C-DSVAE 81.16% 5.341
0.092
1.775
Table 3: Disentanglement metrics on SM-MNIST.
Table 4: Disentanglement metrics on TIMIT.
Methods
Accf
Ist
H(ylx)
H(y)1
Methods content EER, motion EER
MoCoGAN
74.55%
4.078
0.194
0.191
FHVAE
5.06%
22.77%
DSVAE
88.19%
6.210
0.185
2.011
DSVAE
5.64%
19.20%
R-WAE
94.65%
6,940
0.163
2.147
R-WAE
4.73%
23.41%
S3VAE
95.09%
7.072
0.150
2.106
S3VAE
5.02%
25.51
C-DSVAE
97.84% 7.163
0.145
2.176
C-DSVAE
4.03%
31.81% %
is a corpus of read speech for acoustic-phonetic studies and speech recognition. The utterances are produced by American speakers of eight major dialects reading phonetically rich sentences. Following [46 [8], we extract per- framframe spectrogram features (with shift size of 10ms) from audio, and segments of 200ms duration (20 frames) are chunked from the original utterances and then treated independent sequences for learning. All datasets are separated into training, validation
TIMIT 54
and testing splits following 46 33 18
4.2 Disentanglement Metries
The quantitative performance measures all the models from two aspects: first, by fixing either the static dynamic factors and randomly sample the other, how well the fixed factor can be recognized; and second, with the randomly sampled factor, how different the generated sequence is from the original one. To this end, we pretrain separate classifier C to identify the static or dynamic factors (if available). The classifier C' is carefully trained with the full supervision and thus qualifies as judge.
We use five metrics to evaluate the disentanglement performance: accuracy, Inception Score, inter- entropy, intra-entropy and equal error rate. Accuracy (Acc) measures how well the fixed factor can be identified by the classifier C. Inception Score (IS) [55] computes the KL-divergence between the conditional predicted label distribution p(y(ær1:)) and the marginal predicted label distribution p(y) from C. Inter-Entropy similar to IS, but measures the diversity solely on the marginal predicted label distribution p(y); higher inter-entropy indicates that the generated sequences more diverse. Intra-Entropy measures the entropy over p(y( ()); a lower intra-entropy means more confident predictions. Equal error rate (EER) (56) is used in the TIMIT experiment evaluate the speaker identification accuracy.
4.3 Hyperparameters and Architectures
As is commonly done in VAE learning (20). in (6) we add coefficients a to the KL terms, 3 to the contrastive terms, while the coefficient ? of [(s 21:7) fixed be 1. In our experiments, a is tuned over (0.6. 0,.6 0.9. 1.0, 2.0} and / is tuned over (0.1, 0.2, 0.5, 0.7, 1.0, 2.0, 5.0). We use the Adam optimizer [57) with the learning rate chosen from {0.0005. 0.001, 0.0015, 0.002) through grid search. Models are trained until convergence and the one with the best validation accuracy would be chosen for testing. Our network architecture follows 33 motion priors and posteriors are both parameterized by uni-directional LSTMs, which output μ and a at each time step for the dynamic factors, while the content posterior produces j and a for the static factor of the whole sequence with bi-directional LSTM. See Appendix C.2 for further details.
7

---

2
1:
Figure 3: Manipulate the static and dynamic factors on Sprites. Row 1, 2, 4 are the raw test input sequences, while row 3, are manipulated generations. Left panel: Row 3 uses from row 1 and Z1. from row 2. Row 5 uses from row and &1.T from row 4. Right panel: Row 3 uses 21.7 from row I and s from row 2. Row 5 uses #1. from row and s from row
4.4 Quantitative Results
Tables 2 3
demonstrate the disentanglement evaluations on the datasets; 1 means the higher the value the better the performance, and + means the reverse. In Table [ we fix #1.T and randomly sample from p(s) for the evaluation. We observe that R- S3,,E, C-DSVAE can all deliver high accuracies in predicting a total of 9 classes (3 actions x 3 directions), but C-DSVAE outperforms all the others. On the other hand, when fix s and randomly sample 21:7, all the methods achieve near near-perfeer perfect accuracies for predicting character identities (and thus we do not show this result here). Similarly, in Table 2 we evaluate the ability fixing 21:7- C-DS VAE outperforms R- RAF and S3VAE by over 10% relatively in the prediction accuracy, and its intra-entropy is lower by 30% relatively. Table Mablemeeasure. how well the static factor s, which corresponds to one of the total 100 digit combinations, can be recognized when the dynamic factors are randomly sampled (the dynamic factors are continuous and thus hard to be categorized). While R-WAE, S3VAE, and C-DSVAE can all generate high-quality moving digits (see Sec 4.5). C-DSVAE non-trivially outperforms the others on all metrics.
maintain the facial expression by sampling s and
For TIMIT, we extract both the content factor (s, corresponds to the speaker identity) and the dynamic factors (21:7, correspond to the linguistic content) for the sequences. And for each of them, compute the cosine similarity of representations between pairs of sequences, based on which we perform thresholding classify if the two sequences are produced by the same speaker. Through
varying the threshold, we compute the EER for the speaker verification task [46] 32). For a well disentangled latent space, expect the EER (motion EER) to be high. The EER results are given in Table Compared with the previous SOTA by R-WAE, C-DSVAE reduces the content EER by over 10% relatively.
(content EER) to be low, while the EER 21:T
4.5 Qualitative Results
We now compare different methods qualitatively with the following tasks: given the extracted static factor s and dynamic factors 21,7, we fix one of them and manipulate the other to see if we can observe the desired variations in the generated sequences. Figure [ gives example generations for Sprites. We observe that the character identities are well preserved when we fix 8 (left) and the motion is well maintained when we fix 21-T (right), demonstrating the clean disentanglement by C-DSVAE. Figure 4 compares the generated sequences when the motion (facial expression) is fixed for C-DSVAE and S3VAE on MUG. We replace 21:7 for each input sequence with the one from the first row, and therefore the generated sequences are expected to have the smiling expression. C-DSVAE and S3VAE both generate recognizable smiling faces and preserve the individual's 's identity, but the sequences generated by S3VAE have blurs and are less sharp. In Figure [3 we fix the digit movements and take the content (digit identities) from various other sequences. Entangled factors might cause the extracted 1 to carry information of s. Row 3 of Figure shows such example produced by R- R-WAE where the motion factor of one digit carries the content information of "9" from row 1.
8

---

(a) C-DSVAE
(b) S3VAE
Figure 4: Fix the facial expression on MUG. Row 1, 2, 4 are the raw test sequences with different facial expressions. We replace 21:7 of row 2, 4 with 21:T of row and compare the performance of C-DSVAE and S3VAE. Hence row 5 are expected to display happiness well. Our C-DSVAE's generations are clean and sharp, while S3VAE produces blurred sequences (zoom in to see the details).
92
92
9
9
2
9
2
7
27
92
92
9
2
9
2
9
2
7
3
4
Φ
4
*
4
04
4
4
8
*
4
0+
0
40
40
4
d
4
6
a
4
4
4
00
60
9
0
9
4
0
4
8
8
89
8 9
8
%
8
98
98
3
89
89
8
9
98
98
98
9
8
8
9
8
8
8 9
8
7
38
8
3
8
8
9
8 9
3
3
(a) C-DSVAE
(b) R- R-A
Figure 5: Fix the movement and replace the digits. Row 1, 2, 4 are input test sequences with different digits and movements. Row 3, 5 use 21.r from row 1 while retaining their own The sequences
generated by C-DSVAE are clean and consistent, while R- WAE sometimes produces blurred or incorrect digits.
1( (1))) by MWS
[(5; ())) by MWS
2
train w. MWS
train w. MWS
train w. contrastive
train w. contrastive
training time
training time
Figure 6: Learning curves of the MI terms (estimated by MWS) between latent factors and the input sequences, when the training uses either MWS or the contrastive estimation for the MI terms. Though learning with different estimation methods, the MI curves for the two estimation methods are close to each other. Left: Sprites. Right: TIMIT.
Note that S3VAE and R-WAB are both capable of generating recognizable videos, which is validated by classifier C with good accuracies. Nevertheless, the results here show that C-DSVAE can learn

---

Table Ablation study of the augmentations on SMMNIST.
Methods
Acct
1St
H(ylx)
H(y)t
DSVAE
88.19%
6.210
0.185
2.011
C-DSVAE w/o content aug C-DSVAE w/o motion aug C-DSVAE
92.02%
6.360
0.184
2,083
95.25%
6.471
0.175
2.093
97.84%
7.163
0.145
2.176
Table 6: Ablation study of the augmentations on TIMIT.
Methods
content EER,
motion EER
DSVAE
5.64%
19.20%
C-DSVAE w/o content aug C-DSVAE w/o motion aug C-DSVAE
5.09%
24.30%
4.31%
31.09%
4.03%
31.81% %
better disentangled factors for the high-quality generation. Results of additional generation tasks be found in Appendix where we swap the latent factors from different sequences.
4.6 Other analysis
As mentioned in See 2 we attempt to estimate the MI terms using either MWS or the contrastive estimation with augmentations. In Figure 6, we show the MWS estimation of [(s; (1.7)) improves when the C-DSVAE training objective optimizes the contrastive estimation of ((s;)):)) This means the contrastive estimation could be surrogate for MI in generative models, and its additional inductive bias can be main contributor to the superior performance (see Appendix for quantitative comparisons).
Regarding the augmentations, it is intuitive adopt both the content augmentation and the motion augmentation. Having both of them can not only boost the learning of the dynamic static factors, but also improves the disentanglement. Adding only one of them could lead to unbalanced or entangled representation learning. As shown in Table content augmentation alone can improve accuracy by 3.8%, and motion augmentation alone can improve accuracy by 7.1 1. Two augmentations together can bring gain of 9. 9.1%. Similarly, in Table though adding only content augmentation or motion augmentation can outperform DSVAE, combining them leads to the largest gain.
Furthermore, in Appendix E.1 we show that even don't include (, 21-)) in the optimiza- tion, training on just the contrastive terms with augmentations can already lead to the decrease of [s,s, 21:T). ). Appendix G the transition of digits illustrates our learnt latent space is smooth and meaningful.
5 Conclusion
In this paper, propose Contrastively Disentangled Sequential Variational Autoencoder (C-DSVAE) to learn disentangled static and dynamic latent factors for sequence data without external supervision. Our learning objective is novel ELBO derived differently from prior works, and naturally encourages disentanglement. C-DSVAE uses contrastive estimations of the MI terms to further inject the inductive biases. Our method achieves the state-of-the-arr performance on multiple datasets, terms of the disentanglment metrics and the generation quality. In the future, we plan to extend C-DSVAE to other domains, such text, biology, agriculture and weather prediction. We will also improve our model's 's ability to capture long-range dependencies.
6 Acknowledgment
This work is supported by Hatch Federal Capacity Funds "3110006036 Hatch Bobea NYC-121437
10

---

References
[I] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1798-188 1828, 2013.
[2] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03744 2018.
[3] R Devon Hjelm, Alex Fedorov, Samuel Marohi-n Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018.
[4] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pages 1597-1607. PMLR, 2020.
[5] Adam Golinski, Reza Pourreza, Yang Yang, Guillaume Sautiere, and Taco S Cohen. Feedback recurrent autoencoder for video compression. In Proceedings of the Asian Conference on Computer Vision 2020.
[6] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227-2237, 2018.
[7] Jacob Devlin, Ming- ding-Weiie Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pages 4171-4186, 2019.
bidirectional transformers for language understanding.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33. pages 1877-1901, 2020.
[9] Alexei Baevski and AbdeIrahman Mohamed. Effectiveness of self self-supervised pre-training for asr. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7694-7698. IEEE, 2020.
[10] Yu-An Chung and James Glass. Generative pre-training for speech with autoregressive predic- tive coding. In IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 3497-3501. IEHF, 2020.
[11] Weiran Wang, Qingming Tang, and Karen Liveseu. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6889-6893. IEEE, 2020.
[12] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0:
A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33, 2020.
[13] Junwen Bai, Weiran Wang, Yingbo Zhou, and Caiming Xiong. Representation learning for sequence data with deep autoencoding predictive components. In International Conference on Learning Representations, 2021.
[14] Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In International Conference on Learning Representations, 2017.
11

---

[15] Emily L Denton and vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In I. Guyon. U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan. and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, 2017.
[16] Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal. A generative adversarial approach for zero-shot learning from noisy texts. Proceedings of the IEEE conference computer vision and patlern recognition, pages 1004-1013, 2018
[17] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator what you need for high-resolution video synthesis. In International Conference on Learning Representations, 2020.
[18) Jun Han, Martin Renqiang Min, Ligong Han, Li Erran Li, and Xuan Zhang. Disentangled recurrent wasserstein autoencoder. In International Conference on Learning Representations. 2021.
[19] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, volume 29, pages 2172-2180, 2016.
[20] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-vae: Learning basic visual concepts with a constrained variational framework. International Conference Learning Representations, 2016.
[21] Marco Fraccaro, Simon Kamronn, UIrich Paquet, and Ole Winther. A disentangled recogni- tion and nonlinear dynamics model for unsupervised learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareett, editors, Advances in Neural Information Processing Systems, volume 30, 2017.
[22] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learning, pages 2649-2658. PMLR, 2018.
[23] Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, volume 31, 2018.
[24] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pages 4114-4124. PMLR, 2019.
[25] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
[26] Francesco Locatello, Ben Poole, Gunnar Rätsch. Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348-6359. PMLR, 2020.
[27] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(): 1735-175 1780, 1997.
[28] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[29] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, 2015.
[30] Anirudh Goyal, Alessandro Sordoni, Marc-Alexande Côté, Nan Rosemary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. In Advances Neural Information Processing Systems, volume 30, 2017.
12

---

[31] Rahul G Krishnan, Uri Shalit, and David Sontag. Deep kalman filters. arXiv preprint 1rX.v.151..05121, 2015.
[32] Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. In International Conference on Machine Learning, pages 5670-5677 PMLR, 2018.
[33] Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3vae: Self-supervised sequential vae for representation disentanglement and data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6538-6547, 2020.
[34] Samuel Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, 2016.
[35] Jakub Tomczak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statisties, 2018.
[36] Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy, Fixing a broken elbo. In International Conference on Learning Representations, 2018.
[37] Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Variational autoencoder with implicit optimal priors. In Proceedings of the AAAI Conference Artificial Intelligence, pages 5066-5073 2019.
[38] Michael Tschannen, Olivier Bachem, and Mario Lucie. Recent advances autoencoder-based representation learning. arXiv preprint arXiv: S.fiv:/8//.00005 2018.
[39] Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Information-theoretii regularization for learning global features by sequential vae. Machine Learning, 110(8):2239222) 2266, 2021.
[40] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, volume 32, 2019.
[41] Werner Verhelst and Marc Roelands. An overlap-add technique based on waveform similarity (wsola) for high quality time-scale modification of speech. In IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 2. pages 554-557. IEEE, 1993.
[42] Jonathan Driedger and Meinard Müller. A review of time-scale modification of music signals. Applied Sciences, 6(2):57, 2016.
[43] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, 2020.
[44] Ian Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, 2014.
[45] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1526-1533 IEEE Computer Society, 2018.
[46] Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and inter- pretable representations from sequential data. In Advances in Neural Information Processing Systems, volume 30, 2017.
[47] Pengyu Cheng, Martin Renqiang Min, Dinghan Shen, Christopher Malon, Yizhe Zhang, Yitong Li. and Lawrence Carin. Improving disentangled text representation learning with information- theoretic guidance. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 7530-7541, 2020.
[48] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, 2016.
13

---

[49] Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources.
In Artificial Intelligence and Statistics, pages 460-469. PMLR, 2017.
[50] Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. Uncertainty in Artificial Intelligence, pages 217-227. PMLR, 2020.
[51] Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, 2015.
[52] Niki Aifanti, Christos Papachristou, and Anastasios Delopoulos. The mug facial expression database. In 1 Ith International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pages 1-4. IEEE, 2010.
[53] Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In International Conference on Machine Learning, pages 1174-1183. PMLR, 2018.
[54] J. Garofolo, Lori Lamel, W. Fisher, Jonathan Fiscus, D. Pallett, N. Dahlgren, and V. Zue. Timit acoustic-phonetic continuous speech corpus. Linguistic Data Consortium, 1992.
[55] Zhiming Zhou, Weinan Zhang, and Jun Wang. Inception score, label smoothing, gradient vanishing and log(d(x)) alternative. arXiv preprinl arXiv: (rXiv:/708.01729, 2017.
[56] Najim Dehak, Patrick Kenny, Réda Dehak, Pierre Dumouchel, and Pierre Ouellet. Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing. 19(4):788-777 7988,99887- 2010.
[57] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
14


Reviews:
[{"review_id": "PxJsj5xfFEW", "reviewer": "Reviewer_wLW8", "summary": "The focus of this work is disentangled representation in sequential data. It merges existing approaches/views of contrastive learning, disentanglement, and dynamical data modeling, into a unified framework with the goal to extract better representations of sequential data (video, audio experiments). \nThe paper is well written and self-contained which makes it easy to follow. The authors made an effort to present related literature (with useful remarks on differences) and in the experiments, they compare to the most relevant state-of-art approaches. \nMany insightful results are redelegated to the supplementary even though the paper could have content on 9 pages (submitted version has 8).  Although this paper builds on previous works, the detailed analysis of both theoretical and empirical findings makes their contribution in terms of contrastive disentanglement valuable to the community. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "From the ablation study in F.1 seems like motion augmentation is more important than content augmentation. Do you have any intuition on that? What would be an example dataset where the static augmentation will matter more?\n\nAre there any results that show why stochastic prior for the latent space is better than deterministic one?\n\nGiven the generative nature of the model, would it be possible to use C-DSVAE for generating counterfactuals for example? I assume this will depend on how much control you have on the generative process, if besides disentangling static from dynamic factors, could you control for some factorized variables.\n\nSince the approach relies on inductive biases, what would be the implication of changing the current ones with for example physics-inspired inductive biases? The proposed contrastive estimation with augmentation is tailored to video and audio data and I wonder how this could be generalized for datasets of different nature. Is there any alternative that can be considered?\n\nSome relevant references for deep learning approaches to state-space models could be mentioned, such as Kalman VAEs [1]  which also split the learning mechanics into static and dynamic.\n\n\n[1] Fraccaro, M., Kamronn, S., Paquet, U., & Winther, O. (2017). A disentangled recognition and nonlinear dynamics model for unsupervised learning. arXiv preprint arXiv:1710.05741. \n\n------------------------------------------------------------\nUpdate after Author Feedback and Discussion\n------------------------------------------------------------\n\nI thank the authors for the clarification. My questions and concerns have been mostly addressed, and I still argue for acceptance of the paper.  A minor note to avoid ambiguity would be to maybe if not in the title, at least in the abstract the authors should mention that the method (implementation) in the current form, only applies to video and audio data.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no potential negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "MIvJnbryUle", "reviewer": "Reviewer_RjYQ", "summary": "This paper proposes to use contrastive learning methods to improve disentangled sequential representation learning. The proposed C-DSVAE maximizes a novel evidence lower bound that introduces and maximizes the mutual information between latents and input data, both terms are naturally estimated from contrastive loss. The experiments are comprehensive and convincing.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenghs:\n1. The paper is well-written and well-organized.\n2. The formulation in Equation (5) is very interesting. I like the idea of using contrastive learning to naturally estimate the mututal information between inputs and latents. The same idea can be found in WAEs, but contrastive learning smartly leaverages data augmentation and makes the training easier and more robust.\n3. Experiments are extensive and convincing.\n\n\nQuestions:\n1. Line 97-98, why this makes the problem \"less severe\"? Is there any evidence to support that \"it is hard for $s$ to capture the dynamics\"? How to exclude the possibility of posterior collapse where all information is encoded in $s$ (this is likely if $p(z_{1:T})$ is hard to train, especially like Sprites has only 8 frames)?\n2. Evaluation:\n    1. Line 261, are '4 5' and '5 4' considered as two combinations? The numbers in Table 3 are quite high, but it seems impossible for a classifier to tell the order of two digits. Can the authors provide more explanation?\n    2. Line 294, why reporting MWS if \"it may be hard to estimate the distributions accurately\" (as stated in line 119)? Can the author provide more typical evaluation metrics such as MIG?\n3. Samples:\n    1. Can authors provide generation samples from prior on MUG and TIMIT?\n    2. Figure 13 is hard to judge, can the authors provide audio samples like gender swapping in [29] (DSVAE)?\n4. The results on MUG looks quite sharp and nice. I am curious how the model perform in higher resolution or more complicated video sequence (a slightly harder case would be uncropped MUG faces at higher resolution).\n5. Regarding to relating works, I think that the proposed bound is very related to the existing work [\\*]. It would be nice to discuss it in the revision of the paper. Specifically, [\\*] introduces a similar bound with corresponding MI terms.\n\n[\\*] Cheng, Pengyu, et al. \"Improving disentangled text representation learning with information-theoretic guidance.\" arXiv preprint arXiv:2006.00693 (2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed limitations and societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "AvJe4fjKVWs", "reviewer": "Reviewer_r1U7", "summary": "This paper tackles learning disentangled representations for sequential data through contrastive estimation methods to estimate mutual information terms. Content and motion data augmentation approaches are proposed for disentangling static and dynamic factors in a sequential VAE.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper is built on existing works but has good experimental results. In terms of originality, the ELBO formulations/modifications seem to be direct counterparts of the standard VAE to sequential data (however, theorem 1 seems to be new), and the connection between contrastive estimation and mutual information has its own history, with the CPC paper among the first to discuss the connection between contrastive estimation and mutual information. Regardless, this combination of data-augmented contrastive estimation with disentanglement seems novel and interesting.\n\nOne concern is perhaps a lack of fairness during comparison. Since the proposed method uses data augmentation during training, the comparisons to baselines should also contain data augmentation. The main difference is how the proposed method exploits the grouping of augmented sequences, while the baseline methods assume all sequences are independent. The current evaluation seems unfair because the proposed method may simply be doing better by seeing more samples, so it'd be good to see ablation experiments where baseline methods are trained on the data-augmented samples.\n\nSince this contrastive estimation objective depends on having a large amount of negative samples, what is the change in performance across different values of n for Eq 6? How does this compare to MWS when changing the size of the minibatch? Both estimators are biased when using finite samples, so it'd be interesting to see if the bias from contrastive estimation is smaller in practice.\n\nIs the contrastive objective with cosine similarity still an estimator for mutual information? If you plot it, does it show similar values to the MWS estimator?\n\nSince this introduces an additional hyperparameter compared to MWS, how sensitive is the temperature tau? \n\nClarity: Papers on contrastive estimation using data augmentation should probably be referenced in section 2.3. The definition of positive and negative should be more clearly explained in this context. Prior works that make use of cosine similarity should also be cited. The authors should also comment on why cosine similarity is used, instead of other similarity metrics.\n\n----\n\nUpdate after author response: Thanks for the thorough response. The additional experiments sound good to me, and I have updated my score. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Seeing that improvements are from the contrastive estimation objective instead of just seeing more data-augmented samples would benefit this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "2bvHUc7p3b6", "reviewer": "Reviewer_DVU8", "summary": "The paper makes contributions to the research on disentangled sequential representation learning based on Variational Autonecoders (VAES). Towards this end, it\n\n1)  proposes an objective function, which theoretically constitutes a lower bound of the marginal likelihood, while containing interpretable and intuitive mutual information loss terms for encouraging disentanglement between a static and a sequence of dynamic latent factors.\n\n2) proposes a non-parametric estimation of the aforementioned mutual information terms that leverages data-augmentation to further improve performance.\n\n3) demonstrates both quantitative and qualitative performance gains compared to other works on a variety of disentanglement metrics and video datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the paper is easy to follow, and the proposed method is both theoretically and intuitively motivated while tackling an important problem. The results (especially the qualitative of section 4.5) are compelling. However, some points both in the presentation and in the experiments can be improved in order to better demonstrate the proposed method. In particular,\n\n(1) In section 2.3, the implementation of the proposed estimate is not very clear. For example, how many n 'negative sequences are considered and what computational overhead sampling these sequences is it induced compared to the MWS estimate? It would be interesting to demonstrate convergence wrt to n and compare with the MWS estimate. Similarly, for the \\phi term how many samples are considered when estimating the aggregated posterior q(x1:T) in the denominator? How the positive sequence x+ is obtained?\n\n(2) More importantly, it is not clear what improvement p  presented in Tables 1-4, is attributed to the augmentation presented in 2.3 and what improvement to the modified ELBO of equation (5). I think the proposed approach would be better demonstrated via an ablation study on at least one dataset that includes :\ni) results with the objective in Equation (5) and the MWS estimate,\nii) results with the objective in Equation (5) and the estimate in Equation (6) without data augmentation\niii) performance of one competitive model with data augmentation when applicable\n\nIf such an ablation study is provided I would be happy to increase my score.\n\n(3) Regarding the writing, \ni. there are several errors that need to be addressed (page 8: we fix the digit movements and *take*, in the conclusion C-DSVAE uses ... to further *inject*). \nii. in equation 5, i think it would ease the reading if the KL between the aggregate posteriors is clearly defined and compared to the KL term of Eq 3 (I had to resort to the appendix for the precise definition).\n\n(4) As future research, it would be interesting to enforce disentanglement on feedback VAEs like the one proposed in \"Feedback Recurrent Autoencoder for Video Compression\",  and investigate the effect on more complex temporal patterns/ compression.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper applies contrastive learning to improve disentanglement in sequential latent variables. The idea is based on mutual information as used in many disentangled latent variable models, and then apply contrastive learning techniques as to estimate the mutual information.

Reviewers find the proposed approach novel and interesting. Initially they raise concerns on justifications of the proposed objective and potential limitation of the experimental results. In author feedback, authors conducted extra experiments to explain the intuition of the proposed objective. 

I think the idea of applying contrastive learning to disentangled sequential models is quite natural, although indeed I am not aware of other existing work that applies this idea to sequential VAEs. Experimental results indeed show advantage of the proposed approach, although it would be useful to discuss more on intuitions and analyse the key advantage of contrastive learning as compared with other mutual information based disentanglement techniques. 
Decision: Accept (Poster)