Contrastively Disentangled Sequential Variational Autoencoder
Year: 2021

Abstract:
Self-supervised disentangled representation learning is a critical task in sequence modeling. The learnt representations contribute to better model interpretability as well as the data generation, and improve the sample efficiency for downstream tasks. We propose a novel sequence representation learning method, named Contrastively Disentangled Sequential Variational Autoencoder (C-DSVAE), to extract and separate the static (time-invariant) and dynamic (time-variant) factors in the latent space. Different from previous sequential variational autoencoder methods, we use a novel evidence lower bound which maximizes the mutual information between the input and the latent factors, while penalizes the mutual information between the static and dynamic factors. We leverage contrastive estimations of the mutual information terms in training, together with simple yet effective augmentation techniques, to introduce additional inductive biases. Our experiments show that C-DSVAE significantly outperforms the previous state-of-the-art methods on multiple metrics.

heading: 1 Introduction
text: The goal of self-supervised learning methods is to extract useful and general representations without any supervision, and to further facilitate downstream tasks such as generation and prediction [1]. Despite the difficulty of this task, many existing works have shed light on this field across different domains such as computer vision [2, 3, 4, 5], natural language processing [6, 7, 8] and speech processing [9, 10, 11, 12, 13] (also see a huge number of references in these papers). While the quality of the learnt representations improves gradually, recent research starts to put more emphasis on learning disentangled representations. This is because disentangled latent variables may capture separate variations of the data generation process, which could contain semantic meanings, provide the opportunity to remove unwanted variations for a lower sample complexity of downstream learning [14, 15], and allow more controllable generations [16, 17, 18]. These advantages lead to a rapidly growing research area, studying various principles and algorithmic techniques for disentangled representation learning [19, 20, 21, 22, 23, 24, 25, 26]. One concern raised in [24] is that without any inductive bias, it would be extremely hard to learn meaningful disentangled representations. On the other hand, this concern could be much alleviated in the scenarios where the known structure of the data can be exploited.
In this work, we are concerned with the representation learning for sequence data, which has a unique structure to utilize for disentanglement learning. More specifically, for many sequence data, the variations can be explained by a dichotomy of a static (time-invariant) factor and dynamic (timevariant) factors, each varies independently from the other. For example, representations of a video recording the movements of a cartoon character could be disentangled into the character identity (static) and the actions (dynamic). For audio data, the representations shall be able to separate the speaker information (static) from the linguistic information (dynamic).
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
We propose Contrastively Disentangled Sequential Variational Autoencoder (C-DSVAE), a method seeking for a clean separation of the static and dynamic factors for the sequence data. Our method extends the previously proposed sequential variational autoencoder (VAE) framework, and performs learning with a different evidence lower bound (ELBO) which naturally contains mutual information (MI) terms to encourage disentanglement. Due to the difficulty in estimating high dimensional complex distributions (e.g., for the dynamic factors), we further incorporate the contrastive estimation for the MI terms with systematic data augmentation techniques which modify either the static or dynamic factors of the input sequence. The new estimation method turns out to be more effective than the minibatch sampling based estimate, and introduces additional inductive biases towards the invariance. To our knowledge, we are the first to synergistically combine the contrastive estimation and sequential generative models in a principled manner for learning disentangled representations. We validate C-DSVAE on four datasets from the video and audio domains. The experimental results show that our method consistently outperforms the previous state-of-the-art (SOTA) methods, both quantitatively and qualitatively.

heading: 2 Method
text: We denote the observed input sequence as x1:T = {x1, x2, ..., xT } where xi represents the input feature at time step i (e.g., xi could be a video frame or the spectrogram feature of a short audio segment), and T is the sequence length. The latent representations are divided into the static factor s, and the dynamic factors z1:T where zi is the learnt dynamic representation at time step i.

heading: 2.1 Probabilistic Model
text: We assume that in the ground-truth generation process, zi depends on z<i = {z0, z1, ..., zi−1} where z0 = 0, and the observation xi is independent of other frames conditioned on zi and s. Furthermore, we assume the static variable s and the dynamic variables z1:T are independent from each other, i.e., p(s, z1:T ) = p(s)p(z1:T ). Formally, let z = (s, z1:T ) and we have the following complete likelihood
p(x1:T , z) = p(z)p(x1:T |z) = [ p(s) T∏ i=1 p(zi|z<i) ] · T∏ i=1 p(xi|zi, s) (1)
where p(z) is the prior for sampling z. Our formulation captures the general intuition that we can separate the variations of the sequence into the time-dependent dynamic component (described by zi’s), and a static component (described by s) which remains the same for different time steps in the same sequence but differs between sequences. For example, in the cartoon character video
(see Figure 3), the hair, shirt and pants should be kept the same when the character walks around, but different videos can have different characters. Similarly, for a speech utterance, the phonetic transcription controls the vocal tract motion and the sound being produced over time, but the speaker identity remains the same for the whole utterance. In this work, we refer to the dynamic component as "motion" and the static component as "content".
For the prior distributions, we choose p(s) to be the standard Gaussian N (0, I), and p(zi|z<i) to be N (µ(z<i), σ2(z<i)) where µ(·) and σ(·) are modeled by LSTMs [27]. In operations, the drawn sample s is shared throughout the sequence. To draw a sample of zt, we first take a sample of zt−1 as the input of the current LSTM cell. Then forwarding the LSTM for one step gives us the distribution of zt, from which we draw a sample with the reparameterization trick [28].
To extract the latent representations given only the observed data x1:T where the motion and content are mixed together, we hope to learn a posterior distribution q(z|x1:T ) where the two components are disentangled. That is, similar to the prior, our posterior should have a factorized form:
q(z|x1:T ) = q(z1:T , s|x1:T ) = q(z1:T |x1:T )q(s|x1:T ) = q(s|x1:T ) T∏
i=1
q(zi|z<i, x≤i). (2)
The posterior distributions are also modeled by LSTMs. A nested sampling procedure, resembling that of the prior, is applied to the posterior of dynamic variables. Similar parameterizations of dynamic variables by recurrent networks have been proposed in prior works [29, 30, 31]. The standard loss function for learning the latent representations is an ELBO [32, 33] (also see a derivation in Appendix A.1):
max p,q Ex1:T∼pDEq(z|x1:T ) [log p(x1:T |z)−KL[q(z|x1:T )||p(z)]] (3)
where pD is the empirical data distribution. Under the parameterization of the posterior where s and z1:T are mutually independent, the KL-divergence term reduces to
KL[q(z|x1:T )||p(z)] = KL[q(s|x1:T )||p(s)] +KL[q(z1:T |x1:T )||p(z1:T )] (4) where the second term is approximated with the sampled trajectories of the dynamic variables z1:T .

heading: 2.2 Our approach: Mutual information-based disentanglement
text: Several existing sequence representation learning methods [32, 33, 18] are built on top of the loss function (3). However, this formulation also brings several issues. The KL-divergence regularizes the posterior of the static or dynamic factors to be close to the corresponding priors. When modeling them with powerful neural architectures like LSTMs, it is possible for the KL-divergence to be close to zero, yet at the same time, the posteriors become non-informative of the inputs; this is a common issue for deep generative models like VAE [34]. Techniques have been proposed to alleviate this issue, including adjusting the relative weights of the loss terms to regularize the capacity of posteriors [20], replacing the individual posteriors in the KL terms with aggregated posteriors [35, 36, 37] and enforcing latent structures (such as disentangled representations) in the posteriors [23, 22, 38].
Our principled approach for learning useful representations from sequences is inspired by these prior works, and at the same time incorporates the unique sequential structure of the data. Without inductive biases, the goal of disentanglement can hardly be achieved since it is possible to find entangled s and z1:T that explain the data equally well (in fact, by Theorem 1 of [24], there could exist infinitely many such entangled factors). The problem may seem less severe in our setup, since s is shared across time and it is hard for such s to capture all dynamics. Nevertheless, since both s and zi are used in generating xi, it is still possible for zi to carry some static information. In the extreme case where each zi encompasses s, s would no longer be indispensable for the generation. This issue motivated prior works to optimize the (estimate of) mutual information among latent variables and inputs [33, 18, 39].
Our method seeks to achieve clean disentanglement of s and z1:T , by optimizing the following objective function, which introduces additional MI terms to the vanilla ELBO in (3):
max p,q Ex1:T∼pDEq(z|x1:T )[log p(x1:T |z)]−KL[q(z)||p(z)]
= Ex1:T∼pDEq(z|x1:T )[log p(x1:T |z)]− (KL[q(s|x1:T )||p(s)] +KL[q(z1:T |x1:T )||p(z1:T )]) + Iq(s;x1:T ) + Iq(z1:T ;x1:T )− Iq(z1:T ; s) (5)
where the aggregated posteriors are defined as q(z) = q(s, z1:T ) = EpD [q(s|x1:T )q(z1:T |x1:T )], q(s) = EpD [q(s|x1:T )], q(z1:T ) = EpD [q(z1:T |x1:T )], and the MI terms are defined as Iq(s;x1:T ) = Eq(s,x1:T ) [ log q(s|x1:T )q(s) ] , Iq(z1:T ;x1:T ) = Eq(z1:T ,x1:T ) [ log q(z1:T |x1:T )q(z1:T ) ] (and Iq(z1:T ; s) is defined similarly). The intuition behind (5) is simple: besides explaining the data and matching the posteriors with priors, z1:T and s shall contain useful information from x1:T while excluding the redundant information from each other. We further justify (5) by showing that it still forms a valid ELBO.
Theorem 1. With our parameterization of (s, z1:T ), (5) is a valid lower bound of the data loglikelihood Ex1:T∼pD log(x1:T ).
The full proof can be found in Appendix A.2. With this guarantee, we can follow the spirit of [20, 23] to add and adjust the additional weight coefficients α to the KL terms, β to the MI terms Iq(s;x1:T ), Iq(z1:T ;x1:T ), and γ to Iq(z1:T ; s),
Ex1:T∼pDEq(z|x1:T )[log p(x1:T |z)]− α(KL[q(s|x1:T )||p(s)] +KL[q(z1:T |x1:T )||p(z1:T )]) + β(Iq(s;x1:T ) + Iq(z1:T ;x1:T ))− γIq(z1:T ; s). (6)
It remains to estimate the objective (6) for optimization. The KL term for z1:T is estimated with the standard Monte-Carlo sampling, using trajectories of z1:T [32]. The KL term for s can be estimated analytically. For the MI terms, we attempt two estimations. The first estimation uses a standard mini-batch weighted sampling (MWS) following [23, 33]. Due to the high dimensionality of z1:T and the complex dependency among time steps, it may be hard for MWS to estimate the distributions accurately, so we also explore non-parametric contrastive estimations for Iq(s;x1:T ) and Iq(z1:T ;x1:T ) with additional data augmentations, which we will detail below.

heading: 2.3 C-DSVAE: Contrastive estimation with augmentation
text: A contrastive estimation of I(z1:T ;x1:T ) can be defined as follows
C(z1:T ) = EpD log φ(z1:T , x
+ 1:T )
φ(z1:T , x + 1:T ) + ∑n j=1 φ(z1:T , x j 1:T ) + log(n+ 1) (7)
where x+ is a "positive" sequence, while xj , j = 1, . . . , n is a collection of n "negative" sequences. When φ(z1:T , x1:T ) =
q(x1:T |z1:T ) q(x1:T )
, one can show that (7) approximates Iq(z1:T , x1:T ); see Appendix A.4 for a proof. To implement (7), z1:T is the trajectory obtained by using the mean at each time step from q(z1:T |x1:T ) for a input sequence x1:T , while xj1:T is a randomly sampled sequence from the minibatch. A similar contrastive estimation is defined for s as well. To provide meaningful positive sequences that encourage the invariance of the learnt representations, we obtain x+1:T by systematically perturbing x1:T . In Sec A.4, we discuss how the augmented data give good estimate of (7) under additional assumptions.
Content augmentation The static factor (e.g., the character identity in videos or the speaker in audios) is shared across all the time steps, and should not be affected by the exact order of the frames. We therefore randomly shuffle or simply reverse the order of time steps to generate the content augmentation of x1:T and denote it by xc1:T . The static and dynamic latent factors of x c 1:T , modeled by q, are denoted by sc and zc1:T . This is an inexpensive yet useful strategy, applicable to both audios and videos. Similar ideas were introduced in [33] albeit not used for contrastive estimations.
Motion augmentation In motion augmentation, we would like to maintain the dynamic factors (e.g., actions or movements) while replacing the content with a meaningful alternative. Thanks to the recent efforts in contrastive learning, multiple effective strategies have been proposed. For video datasets, we adopt the combination of cropping, color distortion, Gaussian blur and reshaping [4, 40]. For audio datasets, we use classical unsupervised voice conversion algorithms [41, 42]. The motion augmentation of x1:T is denoted by xm1:T , with the static factor s
m and dynamic factors zm1:T estimated by q.
Note that xm1:T is the motion augmentation of x1:T , and x1:T in turn is also the motion augmentation of xm1:T . Similarly, s and s
c are mutually the "positive" sample to each other w.r.t. the static factor. During the training, it is efficient to generate the augmentations for each sequence in the minibatch,
and apply contrastive estimation on both the original data and the augmented data. This leads to the following final estimates:
Iq(z1:T ;x1:T ) ≈ 1
2 (C(z1:T ) + C(zm1:T )), Iq(s;x1:T ) ≈
1 2 (C(s) + C(sc)) (8)
where we use φ(z1:T (x1:T ), x∗1:T ) = exp(sim(z1:T , z ∗ 1:T )/τ) with ∗ indicating the latent variables for the positive/augmented or negative sequences (extracted from q). sim(·, ·) is the cosine similarity function and τ = 0.5 is a temperature parameter. This form of φ using cosine similarity is widely adopted in contrastive learning [4, 43] as it removes one degree of freedom (length) for high dimensional feature space. Plugging (8) into (6) gives our final learning objective with contrastive estimation. We name our method Contrastively Disentangled Sequential Variational Encoder (C-DSVAE), and a full model illustration is shown in Figure 1.
In practice, we find the contrastive estimation is more effective than the MWS based on Gaussian probabilities (see an ablation study in Appendix F). Interestingly, we observe that with only the contrastive estimation and augmentations, Iq(z1:T ; s) (last term in (5)) decreases even if we do not include its estimation in our optimization process (see Appendix E), indicating that a good disentanglement between z1:T and s could be attained through contrastive estimations solely.
As it will become evident in the experiments, our method achieves a cleaner separation of the dynamic and static factors than previous methods. We believe this is partly due to the inductive bias introduced by our approach: in order to consistently map two sequences with the same motion but different contents (which vary independently from the motion according to the generation process (1)) to the same latent representation z1:T , we must discard the information of the inputs regarding s; similar arguments hold for the static variables. At the same time, we enforce that the extracted z1:T and s together should reconstruct x1:T so that no information is lost in the auto-encoding process. To our knowledge, we are the first to use the contrastive estimation with augmentations in the sequential VAE scenario, for pushing the disentanglement of variations.

heading: 3 Related Work
text: In terms of the general frameworks, disentangled representation learning methods can be categorized into GAN-based [44] and VAE-based [28] approaches. GAN-based methods like MoCoGAN [45] aim at generating videos from content noise and motion noise. These models typically do not explicitly learn the encoder from inputs to the latent variables, rendering them less applicable for representation learning.
The VAE framework parameterizes both the encoder (variational posterior) and the decoder (reconstruction), and optimize them jointly with a well-defined ELBO objective. The encoder allows us to directly control and reason about the properties of the extracted features, which can be straightforwardly used for downstream tasks. Many variants of VAE were proposed to encourage the disentanglement/interpretability of individual latent dimensions. β-VAE [20] adds a coefficient to the per-sample KL-divergence term to better constrain the information bottleneck. FactorVAE [22] and
β-TCVAE [23] explicitly separate out the total correlation term, and adjust its weight coefficient in the objective to encourage the per-dimensional disentanglement of the aggregated posterior. From a theoretical standpoint, [24] points out that without further inductive biases, it is impossible to learn disentangled representations with the standard VAE.
To handle the sequence data, vanilla VAE is extended to the recurrent version, and prior works have explored different approaches for separating the content and the motion. FHVAE [46] designs a hierarchical VAE model which uses "sequence-dependent variables" (corresponding to the speaker factor) and "sequence-independent variables" (corresponding to the linguistic factors) for modeling speech sequences, and performs learning with the per-sequence ELBO while respecting the hierarchy in designing the posterior. FHVAE does not use additional loss terms for encouraging the disentanglement. Different from FHVAE, DSVAE [32] explicitly models the static and dynamic factors in its graphical model and its posterior has a factorized form. A few recent works (including ours) inherit the clean formulation of DSVAE in terms of the graphical model and prior/posterior parameterization. S3VAE [33] introduces additional loss terms to the objective of DSVAE in an ad-hoc fashion, such as a triplet loss that encourages the invariance of the learnt static factors by permuting the frame order (spiritually similar to our contrastive estimation for I(s, x1:T )), an additional prediction loss on z1:T leveraging external supervision for the motion labels, and an MI term of I(s, z1:T ) to be minimized. Our C-DSVAE differs from S3VAE in that we naturally introduce the MI terms from the fundamental principle of VAE, and use augmentations instead of external supervision for learning the static and dynamic factors. Also based on the DSVAE formulation, R-WAE [18] proposes to replace the distance measure between the aggregated posterior and the prior with the Wasserstein distance, in the belief that the KL divergence is too restrictive. In R-WAE, Maximum Mean Discrepancy (MMD) based estimation or GAN-based estimation of Wasserstein distance is used, either of which requires heavy tuning of hyperparameters. Another similar method, IDEL [47], optimizes the additional set of MI terms similar to ours. However, in our work, we show that these MI terms can be naturally derived from the fundamental principle of VAE, and we additionally propose the contrastive estimation and data augmentations to strengthen them, which was not done before.
Another relevant research area is nonlinear ICA, which tries to understand the conditions under which the disentanglement can be achieved, possibly with contrastive learning or VAE [48, 49, 50, 25, 26]. Our general approach is well-aligned with the current understanding in this direction. That is, while general disentanglement (per-dimensional disentanglement) is hard to achieve without stringent assumptions on the data generation process, we can seek some certain level of disentanglement (in our case, the group-wise disentanglement between the static and dynamic factors) given auxiliary information (which we provide through augmentations). Our intuitive arguments for the reason of separation (see the last paragraph of Sec 2.3) show that VAE and contrastive learning may be complementary to each other, and combining them could potentially gain better disentanglement.

heading: 4 Experiments
text: We compare C-DSVAE with the state-of-the-art sequence disentanglement learning methods: FHVAE [46], MoCoGAN [45], DSVAE [32], S3VAE [33] and R-WAE [18], with the same experimental setups used by them.

heading: 4.1 Datasets
text: Sprites [51] is a cartoon character video dataset. Each character’s (dynamic) motion can be categorized into three actions (walking, spellcasting, slashing) and three directions (left, front, right). The (static) content comprises of each character’s skin color, tops color, pants color and hair color. Each color has six variants. Every sequence is composed of 8 frames of RGB images with size 64× 64. MUG [52] is a facial expression video dataset. The static factor corresponds to a single person’s identity. Each individual performs six expressions (motion): anger, fear, disgust, happiness, sadness and surprise. Following [45], each sequence contains 15 frames of RGB images with size 64× 64 (after resizing).
SM-MNIST ([53], Stochastic Moving MNIST) is a dataset that records the random movements of two digits. Each sequence contains 15 gray scale images of size 64 × 64. Individual digits are collected from MNIST.
Table 1: Disentanglement metrics on Sprites.
Methods Acc↑ IS↑ H(y|x)↓ H(y)↑
MoCoGAN 92.89% 8.461 0.090 2.192 DSVAE 90.73% 8.384 0.072 2.192 R-WAE 98.98% 8.516 0.055 2.197 S3VAE 99.49% 8.637 0.041 2.197
C-DSVAE 99.99% 8.871 0.014 2.197
Table 2: Disentanglement metrics on MUG.
Methods Acc↑ IS↑ H(y|x)↓ H(y)↑
MoCoGAN 63.12% 4.332 0.183 1.721 DSVAE 54.29% 3.608 0.374 1.657 R-WAE 71.25% 5.149 0.131 1.771 S3VAE 70.51% 5.136 0.135 1.760
C-DSVAE 81.16% 5.341 0.092 1.775
TIMIT [54] is a corpus of read speech for acoustic-phonetic studies and speech recognition. The utterances are produced by American speakers of eight major dialects reading phonetically rich sentences. Following [46, 18], we extract per-frame spectrogram features (with a shift size of 10ms) from audio, and segments of 200ms duration (20 frames) are chunked from the original utterances and then treated as independent sequences for learning. All datasets are separated into training, validation and testing splits following [46, 33, 18].

heading: 4.2 Disentanglement Metrics
text: The quantitative performance measures all the models from two aspects: first, by fixing either the static or dynamic factors and randomly sample the other, how well the fixed factor can be recognized; and second, with the randomly sampled factor, how different the generated sequence is from the original one. To this end, we pretrain a separate classifier C to identify the static or dynamic factors (if available). The classifier C is carefully trained with the full supervision and thus qualifies as a judge.
We use five metrics to evaluate the disentanglement performance: accuracy, Inception Score, interentropy, intra-entropy and equal error rate. Accuracy (Acc) measures how well the fixed factor can be identified by the classifier C. Inception Score (IS) [55] computes the KL-divergence between the conditional predicted label distribution p(y|x1:T ) and the marginal predicted label distribution p(y) from C. Inter-Entropy is similar to IS, but measures the diversity solely on the marginal predicted label distribution p(y); a higher inter-entropy indicates that the generated sequences are more diverse. Intra-Entropy measures the entropy over p(y|x); a lower intra-entropy means more confident predictions. Equal error rate (EER) [56] is used in the TIMIT experiment to evaluate the speaker identification accuracy.

heading: 4.3 Hyperparameters and Architectures
text: As is commonly done in VAE learning [20], in (6) we add coefficients α to the KL terms, β to the contrastive terms, while the coefficient γ of I(s; z1:T ) is fixed to be 1. In our experiments, α is tuned over {0.6, 0.9, 1.0, 2.0} and β is tuned over {0.1, 0.2, 0.5, 0.7, 1.0, 2.0, 5.0}. We use the Adam optimizer [57] with the learning rate chosen from {0.0005, 0.001, 0.0015, 0.002} through grid search. Models are trained until convergence and the one with the best validation accuracy would be chosen for testing. Our network architecture follows [33]: motion priors and posteriors are both parameterized by uni-directional LSTMs, which output µ and σ at each time step for the dynamic factors, while the content posterior produces µ and σ for the static factor of the whole sequence with a bi-directional LSTM. See Appendix C.2 for further details.

heading: 4.4 Quantitative Results
text: Tables 1, 2, 3, 4 demonstrate the disentanglement evaluations on the datasets; ↑ means the higher the value the better the performance, and ↓ means the reverse. In Table 1, we fix z1:T and randomly sample s from p(s) for the evaluation. We observe that R-WAE, S3VAE, C-DSVAE can all deliver high accuracies in predicting a total of 9 classes (3 actions× 3 directions), but C-DSVAE outperforms all the others. On the other hand, when we fix s and randomly sample z1:T , all the methods achieve near-perfect accuracies for predicting character identities (and thus we do not show this result here). Similarly, in Table 2, we evaluate the ability to maintain the facial expression by sampling s and fixing z1:T . C-DSVAE outperforms R-WAE and S3VAE by over 10% relatively in the prediction accuracy, and its intra-entropy is lower by 30% relatively. Table 3 measures how well the static factor s, which corresponds to one of the total 100 digit combinations, can be recognized when the dynamic factors are randomly sampled (the dynamic factors are continuous and thus hard to be categorized). While R-WAE, S3VAE, and C-DSVAE can all generate high-quality moving digits (see Sec 4.5), C-DSVAE non-trivially outperforms the others on all metrics.
For TIMIT, we extract both the content factor (s, corresponds to the speaker identity) and the dynamic factors (z1:T , correspond to the linguistic content) for the sequences. And for each of them, we compute the cosine similarity of representations between pairs of sequences, based on which we perform thresholding to classify if the two sequences are produced by the same speaker. Through varying the threshold, we compute the EER for the speaker verification task [46, 32]. For a well disentangled latent space, we expect the EER on s (content EER) to be low, while the EER on z1:T (motion EER) to be high. The EER results are given in Table 4. Compared with the previous SOTA by R-WAE, C-DSVAE reduces the content EER by over 10% relatively.

heading: 4.5 Qualitative Results
text: We now compare different methods qualitatively with the following tasks: given the extracted static factor s and dynamic factors z1:T , we fix one of them and manipulate the other to see if we can observe the desired variations in the generated sequences. Figure 3 gives example generations for Sprites. We observe that the character identities are well preserved when we fix s (left) and the motion is well maintained when we fix z1:T (right), demonstrating the clean disentanglement by C-DSVAE. Figure 4 compares the generated sequences when the motion (facial expression) is fixed for C-DSVAE and S3VAE on MUG. We replace z1:T for each input sequence with the one from the first row, and therefore the generated sequences are expected to have the smiling expression. C-DSVAE and S3VAE both generate recognizable smiling faces and preserve the individual’s identity, but the sequences generated by S3VAE have blurs and are less sharp. In Figure 5, we fix the digit movements and take the content (digit identities) from various other sequences. Entangled factors might cause the extracted z1:T to carry information of s. Row 3 of Figure 5b shows such an example produced by R-WAE where the motion factor of one digit carries the content information of "9" from row 1.
Note that S3VAE and R-WAE are both capable of generating recognizable videos, which is validated by our classifier C with good accuracies. Nevertheless, the results here show that C-DSVAE can learn
better disentangled factors for the high-quality generation. Results of additional generation tasks can be found in Appendix D, where we swap the latent factors from different sequences.

heading: 4.6 Other analysis
text: As mentioned in Sec 2, we attempt to estimate the MI terms using either MWS or the contrastive estimation with augmentations. In Figure 6, we show the MWS estimation of I(s;x1:T ) improves when the C-DSVAE training objective optimizes the contrastive estimation of I(s;x1:T ). This means the contrastive estimation could be a surrogate for MI in generative models, and its additional inductive bias can be a main contributor to the superior performance (see Appendix F for quantitative comparisons).
Regarding the augmentations, it is intuitive to adopt both the content augmentation and the motion augmentation. Having both of them can not only boost the learning of the dynamic or static factors, but also improves the disentanglement. Adding only one of them could lead to unbalanced or entangled representation learning. As shown in Table 5, content augmentation alone can improve the accuracy by 3.8%, and motion augmentation alone can improve the accuracy by 7.1%. Two augmentations together can bring a gain of 9.1%. Similarly, in Table 6, though adding only content augmentation or motion augmentation can outperform DSVAE, combining them leads to the largest gain.
Furthermore, in Appendix E.1, we show that even if we don’t include Iq(s, z1:T ) in the optimization, training on just the contrastive terms with augmentations can already lead to the decrease of Iq(s, z1:T ). In Appendix G, the transition of digits illustrates our learnt latent space is smooth and meaningful.

heading: 5 Conclusion
text: In this paper, we propose Contrastively Disentangled Sequential Variational Autoencoder (C-DSVAE) to learn disentangled static and dynamic latent factors for sequence data without external supervision. Our learning objective is a novel ELBO derived differently from prior works, and naturally encourages disentanglement. C-DSVAE uses contrastive estimations of the MI terms to further inject the inductive biases. Our method achieves the state-of-the-art performance on multiple datasets, in terms of the disentanglment metrics and the generation quality. In the future, we plan to extend C-DSVAE to other domains, such as text, biology, agriculture and weather prediction. We will also improve our model’s ability to capture long-range dependencies.

heading: 6 Acknowledgment
text: This work is supported by Hatch Federal Capacity Funds "3110006036 Hatch Bobea NYC-121437".

Review summaries:
The focus of this work is disentangled representation in sequential data. It merges existing approaches/views of contrastive learning, disentanglement, and dynamical data modeling, into a unified framework with the goal to extract better representations of sequential data (video, audio experiments).  The paper is well written and self-contained which makes it easy to follow. The authors made an effort to present related literature (with useful remarks on differences) and in the experiments, they compare to the most relevant state-of-art approaches.  Many insightful results are redelegated to the supplementary even though the paper could have content on 9 pages (submitted version has 8).  Although this paper builds on previous works, the detailed analysis of both theoretical and empirical findings makes their contribution in terms of contrastive disentanglement valuable to the community. 

The paper makes contributions to the research on disentangled sequential representation learning based on Variational Autonecoders (VAES). Towards this end, it  1)  proposes an objective function, which theoretically constitutes a lower bound of the marginal likelihood, while containing interpretable and intuitive mutual information loss terms for encouraging disentanglement between a static and a sequence of dynamic latent factors.  2) proposes a non-parametric estimation of the aforementioned mutual information terms that leverages data-augmentation to further improve performance.  3) demonstrates both quantitative and qualitative performance gains compared to other works on a variety of disentanglement metrics and video datasets.

This paper proposes to use contrastive learning methods to improve disentangled sequential representation learning. The proposed C-DSVAE maximizes a novel evidence lower bound that introduces and maximizes the mutual information between latents and input data, both terms are naturally estimated from contrastive loss. The experiments are comprehensive and convincing.