Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper tackles learning disentangled representations for sequential data through contrastive estimation methods to estimate mutual information terms. Content and motion data augmentation approaches are proposed for disentangling static and dynamic factors in a sequential VAE.
main_review: Overall, this paper is built on existing works but has good experimental results. In terms of originality, the ELBO formulations/modifications seem to be direct counterparts of the standard VAE to sequential data (however, theorem 1 seems to be new), and the connection between contrastive estimation and mutual information has its own history, with the CPC paper among the first to discuss the connection between contrastive estimation and mutual information. Regardless, this combination of data-augmented contrastive estimation with disentanglement seems novel and interesting.

One concern is perhaps a lack of fairness during comparison. Since the proposed method uses data augmentation during training, the comparisons to baselines should also contain data augmentation. The main difference is how the proposed method exploits the grouping of augmented sequences, while the baseline methods assume all sequences are independent. The current evaluation seems unfair because the proposed method may simply be doing better by seeing more samples, so it'd be good to see ablation experiments where baseline methods are trained on the data-augmented samples.

Since this contrastive estimation objective depends on having a large amount of negative samples, what is the change in performance across different values of n for Eq 6? How does this compare to MWS when changing the size of the minibatch? Both estimators are biased when using finite samples, so it'd be interesting to see if the bias from contrastive estimation is smaller in practice.

Is the contrastive objective with cosine similarity still an estimator for mutual information? If you plot it, does it show similar values to the MWS estimator?

Since this introduces an additional hyperparameter compared to MWS, how sensitive is the temperature tau? 

Clarity: Papers on contrastive estimation using data augmentation should probably be referenced in section 2.3. The definition of positive and negative should be more clearly explained in this context. Prior works that make use of cosine similarity should also be cited. The authors should also comment on why cosine similarity is used, instead of other similarity metrics.

----

Update after author response: Thanks for the thorough response. The additional experiments sound good to me, and I have updated my score. 
limitations_and_societal_impact: Seeing that improvements are from the contrastive estimation objective instead of just seeing more data-augmented samples would benefit this paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper makes contributions to the research on disentangled sequential representation learning based on Variational Autonecoders (VAES). Towards this end, it

1)  proposes an objective function, which theoretically constitutes a lower bound of the marginal likelihood, while containing interpretable and intuitive mutual information loss terms for encouraging disentanglement between a static and a sequence of dynamic latent factors.

2) proposes a non-parametric estimation of the aforementioned mutual information terms that leverages data-augmentation to further improve performance.

3) demonstrates both quantitative and qualitative performance gains compared to other works on a variety of disentanglement metrics and video datasets.
main_review: In general, the paper is easy to follow, and the proposed method is both theoretically and intuitively motivated while tackling an important problem. The results (especially the qualitative of section 4.5) are compelling. However, some points both in the presentation and in the experiments can be improved in order to better demonstrate the proposed method. In particular,

(1) In section 2.3, the implementation of the proposed estimate is not very clear. For example, how many n 'negative sequences are considered and what computational overhead sampling these sequences is it induced compared to the MWS estimate? It would be interesting to demonstrate convergence wrt to n and compare with the MWS estimate. Similarly, for the \phi term how many samples are considered when estimating the aggregated posterior q(x1:T) in the denominator? How the positive sequence x+ is obtained?

(2) More importantly, it is not clear what improvement p  presented in Tables 1-4, is attributed to the augmentation presented in 2.3 and what improvement to the modified ELBO of equation (5). I think the proposed approach would be better demonstrated via an ablation study on at least one dataset that includes :
i) results with the objective in Equation (5) and the MWS estimate,
ii) results with the objective in Equation (5) and the estimate in Equation (6) without data augmentation
iii) performance of one competitive model with data augmentation when applicable

If such an ablation study is provided I would be happy to increase my score.

(3) Regarding the writing, 
i. there are several errors that need to be addressed (page 8: we fix the digit movements and *take*, in the conclusion C-DSVAE uses ... to further *inject*). 
ii. in equation 5, i think it would ease the reading if the KL between the aggregate posteriors is clearly defined and compared to the KL term of Eq 3 (I had to resort to the appendix for the precise definition).

(4) As future research, it would be interesting to enforce disentanglement on feedback VAEs like the one proposed in "Feedback Recurrent Autoencoder for Video Compression",  and investigate the effect on more complex temporal patterns/ compression.


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to use contrastive learning methods to improve disentangled sequential representation learning. The proposed C-DSVAE maximizes a novel evidence lower bound that introduces and maximizes the mutual information between latents and input data, both terms are naturally estimated from contrastive loss. The experiments are comprehensive and convincing.
main_review: Strenghs:
1. The paper is well-written and well-organized.
2. The formulation in Equation (5) is very interesting. I like the idea of using contrastive learning to naturally estimate the mututal information between inputs and latents. The same idea can be found in WAEs, but contrastive learning smartly leaverages data augmentation and makes the training easier and more robust.
3. Experiments are extensive and convincing.


Questions:
1. Line 97-98, why this makes the problem "less severe"? Is there any evidence to support that "it is hard for $s$ to capture the dynamics"? How to exclude the possibility of posterior collapse where all information is encoded in $s$ (this is likely if $p(z_{1:T})$ is hard to train, especially like Sprites has only 8 frames)?
2. Evaluation:
    1. Line 261, are '4 5' and '5 4' considered as two combinations? The numbers in Table 3 are quite high, but it seems impossible for a classifier to tell the order of two digits. Can the authors provide more explanation?
    2. Line 294, why reporting MWS if "it may be hard to estimate the distributions accurately" (as stated in line 119)? Can the author provide more typical evaluation metrics such as MIG?
3. Samples:
    1. Can authors provide generation samples from prior on MUG and TIMIT?
    2. Figure 13 is hard to judge, can the authors provide audio samples like gender swapping in [29] (DSVAE)?
4. The results on MUG looks quite sharp and nice. I am curious how the model perform in higher resolution or more complicated video sequence (a slightly harder case would be uncropped MUG faces at higher resolution).
5. Regarding to relating works, I think that the proposed bound is very related to the existing work [\*]. It would be nice to discuss it in the revision of the paper. Specifically, [\*] introduces a similar bound with corresponding MI terms.

[\*] Cheng, Pengyu, et al. "Improving disentangled text representation learning with information-theoretic guidance." arXiv preprint arXiv:2006.00693 (2020).
limitations_and_societal_impact: The authors discussed limitations and societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The focus of this work is disentangled representation in sequential data. It merges existing approaches/views of contrastive learning, disentanglement, and dynamical data modeling, into a unified framework with the goal to extract better representations of sequential data (video, audio experiments). 
The paper is well written and self-contained which makes it easy to follow. The authors made an effort to present related literature (with useful remarks on differences) and in the experiments, they compare to the most relevant state-of-art approaches. 
Many insightful results are redelegated to the supplementary even though the paper could have content on 9 pages (submitted version has 8).  Although this paper builds on previous works, the detailed analysis of both theoretical and empirical findings makes their contribution in terms of contrastive disentanglement valuable to the community. 
main_review: From the ablation study in F.1 seems like motion augmentation is more important than content augmentation. Do you have any intuition on that? What would be an example dataset where the static augmentation will matter more?

Are there any results that show why stochastic prior for the latent space is better than deterministic one?

Given the generative nature of the model, would it be possible to use C-DSVAE for generating counterfactuals for example? I assume this will depend on how much control you have on the generative process, if besides disentangling static from dynamic factors, could you control for some factorized variables.

Since the approach relies on inductive biases, what would be the implication of changing the current ones with for example physics-inspired inductive biases? The proposed contrastive estimation with augmentation is tailored to video and audio data and I wonder how this could be generalized for datasets of different nature. Is there any alternative that can be considered?

Some relevant references for deep learning approaches to state-space models could be mentioned, such as Kalman VAEs [1]  which also split the learning mechanics into static and dynamic.


[1] Fraccaro, M., Kamronn, S., Paquet, U., & Winther, O. (2017). A disentangled recognition and nonlinear dynamics model for unsupervised learning. arXiv preprint arXiv:1710.05741. 

------------------------------------------------------------
Update after Author Feedback and Discussion
------------------------------------------------------------

I thank the authors for the clarification. My questions and concerns have been mostly addressed, and I still argue for acceptance of the paper.  A minor note to avoid ambiguity would be to maybe if not in the title, at least in the abstract the authors should mention that the method (implementation) in the current form, only applies to video and audio data.
limitations_and_societal_impact: There is no potential negative societal impact of this work.
review_title: null

Meta-review: This paper applies contrastive learning to improve disentanglement in sequential latent variables. The idea is based on mutual information as used in many disentangled latent variable models, and then apply contrastive learning techniques as to estimate the mutual information.

Reviewers find the proposed approach novel and interesting. Initially they raise concerns on justifications of the proposed objective and potential limitation of the experimental results. In author feedback, authors conducted extra experiments to explain the intuition of the proposed objective. 

I think the idea of applying contrastive learning to disentangled sequential models is quite natural, although indeed I am not aware of other existing work that applies this idea to sequential VAEs. Experimental results indeed show advantage of the proposed approach, although it would be useful to discuss more on intuitions and analyse the key advantage of contrastive learning as compared with other mutual information based disentanglement techniques. 
Decision: Accept (Poster)