Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper proposes R-WAE to learn disentangled representations. Benefit from the characteristics of WAE, this paper shows that R-WAE can better disentangle the sequential data into content space and motion space. R-WAE achieves state-of-the-art performance in both disentanglement and unconditional generation.


Reasons for score: 
Overall, I vote for acceptance. The proposed method has theoretical foundations and shows excellent results.
 

Pros: 
 
-- The paper provided strict theoretical formulations, like the comparison between WAE and VAE, how WAE can generalize to the sequential format, and the connection between mutual information(MI) and the objective function of R-WAE. The experiments support the theorems.
 
-- The authors provide sufficient experiments on multiple datasets. 
The experiments cover tasks of various domains, including video and audio, which indicate the proposed method could be easily generalized to different tasks.
 
-- Many architectures are investigated, including WAE-MMD, WAE-GAN, and simple/complex encoder.


Cons: 

-- When generation (Fig. 1 (a)), the dependency between h in different time steps is considered. However, during the inference phase (Fig. 1 (b)), the dependency is ignored. Any good reason? 

-- The illusions of Figure 5 do not keep the same across frames, while z^m shouldn’t change illusion. More analysis and discussion are probably needed for this result.
 
-- The comparison between R-WAE(GAN) and R-WAE(MMD) can be further discussed. The comparison is shown in Table 3. The results show that R-WAE(GAN) performs better, but the reason is unclear. 
 
-- For the audio experiments, ASR results or phoneme classification might be needed to support that z^m keeps the local information. It would be better to provide audio demos of your cross reconstruction result shown in Figure 11.

-- Figure 6 shows that WAE usually gives a tighter gap between classes of z^c, since WAE computes divergence between Q(Z^c) and P(Z^c), which causes z^c spread over the entire latent space; while VAE gives a noticeable gap between different classes of contents (such as the one shown in Figure 3 of scalable FHVAE), which leaves space for the unseen contents. Would the tighter gap cause issue when training and testing data are mismatched? If the testing data has some contents that have never been seen during the training, what would happen? (for example, training on MUG than testing on VoxCeleb video data) Will the z^c of the unseen content data be forced mapped to the content in the training set instead of keeping its own information?


review_title: The proposed method has theoretical foundations and shows promising results.

final_score: rating: 7: Good paper, accept
review_content: This paper focuses on learning disentangled representations for sequential data. This paper proposed recurrent Wasserstein Autoencoder (R-WAE), which disentangles the factors of variations into time-variant and time-invariant ones. The theoretical analysis is provided by the method. Experiments show that R-WAE outperforms baselines on several datasets.

In general, the paper is easy to follow. The method looks relatively straight-forward, because it extends a recurrent VAE framework by replacing the reconstruction loss term with a Wasserstein distance. The superiority of the method is supported by both theoretical analysis and experimental performance.

On page 6, the authors claim that the regularization term in R-WAE is superior to a VAE model. But it is not clear to me why a recurrent VAE model cannot adopt the same regularization term. If this is possible, then I believe an ablation study is useful. Because it helps us understand the performance improvement is due to the introduction of Wasserstein distance or the regularization or both.

In section 3.4, it is mentioned that the number of actions is provided as weak supervision. Since it looks like the supervision does not contain much information, I am wondering how the performance will be impacted if such supervision is not provided. In addition, I want to know whether the learned categorial variable $a$ is consistent with the ground-truth actions for the data.

In table 3, intra-entropy $H(y)$ is reported. I am not sure why we should care about this value and why a lower value implies better performance. It looks to me that a lower $H(y)$ means that the model is more likely to give the same prediction for different samples. It is not clear to me why we want the model to have such property.

In general, I suggest accepting this paper, because the proposed method is supported by theoretical analysis and gives a good experimental performance.



Minor:

Eq. (13) involves the mutual information term $I(X_{1:T};Z^c)$. Which distribution does $Z^c$ follows? It is the posterior distirubiton $P(Z^c | X_{1:T})$ or its variational approximation $Q(Z^c | X_{1:T})$?

On page 6, Eq. (21) should be changed to Eq. (13).


review_title: The proposed method is supported by theoretical analysis and gives a good experimental performance.

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper extends the Wasserstein autoencoder for learning disentangled representations from sequential data. The latent variable model considered contains separate latent variables capturing global and local information respectively, each of which is regularized by a divergence measuring the marginal posterior $Q_z$ and the prior $P_z$. An optional auxiliary discrete latent is introduced to incorporate inductive bias for discrete local features (e.g., type of actions). To estimate the divergence terms, the authors propose to use MMD for the recurrent local latents since the prior distribution evolves over time; for the global latent, the authors presented two options: discriminator-based Jenson-Shannon Divergence estimate (the same as adversarial autoencoder proposed in Makhzani et al., 2016) and scaled MMD (Arbel et al., 2018). The connection between the proposed objective and mutual information maximization is outlined in Section 4. Experimental results show that the proposed R-WAE model outperforms baseline DS-VAE/FHVAE/MocoGAN

Pros:
- Extending Wasserstein autoencoder to model sequential data and learn disentangled representations is novel and well-motivated
- Experimental results demonstrate the advantage compared with the baseline models in terms of disentanglement performance and generated data quality

Cons:
- The appendix requires proofreading - the authors left derivations in the appendix, but it contains incomplete sentences and inconsistent notations. For example, in Appendix D, “We just use MMD without…” is incomplete. $P_{Z^c}$ and $Q_{Z^c}$ are used as example for the vanilla MMD computation while according the main text $P_{Z^m}$ and $Q_{Z^m}$ should be the one whose penalty is computed with that.
- Ablation studies are lacking. How sensitive the performance is with respect to the latent variable dimensions and whether the tricks (updating encoder for $L$ times) are used or not. It is hard to conclude whether the proposed formulation is the key to the superior performance.
- How is the performance like when using the vanilla MMD divergence for $Z^c$ without a neural kernel? The authors justify the decision by saying the scaled MMD is more expressive and proper for higher dimensional $Z^c$. It will be nice to show the benefit empirically; moreover, the $Z^c$ used for TIMIT and Sprites are not of higher dimensions than those for $Z^m$.
- Why is R-WAE (GAN) only shown for the MUG dataset? Can authors also present the results for TIMIT/Sprites/SM-MNIST?
- This paper should cite adversarial autoencoder (Makhzani et al., 2016) for the discriminator based JS divergence estimate, which proposed the same penalty first.

review_title: review

Meta-review: This paper presents an approach for learning disentangled static and dynamic latent variables for sequence data. In terms of learning objective, the paper extends Wasserstein autoencoder to sequential data, and this approach is novel and well-motivated; the aggregated posterior for static variables comes out naturally and plays an important role for regularization (this appears to be new for sequence data). The authors also studies how to model additional categorical variables for weakly supervised learning in real scenarios. The main steps (generation and inference) were illustrated by graphical models with clarity, and rigorous statements are provided to back them up. Experimental results demonstrate the advantages of proposed method, in terms of disentanglement performance and generation quality.

The reviewers think this paper makes nice contributions to the sequential generative model community.
Decision: Accept (Spotlight)