Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper introduces the SlowVAE to model transitions (of position, size, etc) in single-object videos. A Laplace prior conditioned on the latents at the previous step is used to learn the transitions, which the authors argue are naturally sparse.

My biggest concern regarding this work is the construction of datasets: the authors effectively “consider pairs of images [x_{t-1}, x_t], which differ effectively in only a few factors of variation due to the sparse prior.” While the marginal distributions of factor transitions may no doubt be sparse in natural settings, I would guess the sparsity is not independent across factors. For example: a video where a camera moves sporadically will show the position factors and size of an object changing sparsely but simultaneously. Could you augment Fig 1 and appendix H to show the joint distributions in addition to the marginals?

Moreover, the LAP procedure has the following rejection rule: “if all factors remain constant (no transition), the sample is rejected as the pair would not result in any temporal learning signal.” Surely there needs to be some static pairs if you want to be close to natural videos? 

Given KITTI is the only dataset you show results on whose transitions are natural (albeit not 3D-natural; see caveat in question 2 below), it would be nice to see more evidence how the SlowVAE would generalize and cope in scenarios different from the ones you’ve constructed. If the LAP procedure indeed samples from marginal transition statistics rather than the joint distributions, I don't believe the samples would resemble "natural transitions". Learning disentangled representations from data which is engineered to show changes in one generative feature at a time seems like cheating because we usually don't have easy access to such data (unless an agent actively interacts with an object).

Strengths:
- Good literature review
- Comparison to relevant SOTA models, linking to prior work on disentanglement as well as nonlinear ICA, on a variety of metrics.
- Solid experiments and well-presented results

Questions:
1) Have you tried working with sequences rather than pairs of images? I assume your ELBO can be readily extended to sequential data. Are there additional challenges in working with sequences?
2) Do you think there's room for bias in the natural statistics you computed on Youtube-VOS and KITTI-MOTS? For instance, from the fact that object masks are projections of 3D objects onto 2D frames? Consider the bicycle example (row 1) at https://youtube-vos.org/. In 3D, the position of the bicycle changes smoothly as the cyclist carries it. But in 2D there is a noticeable jump in the center of mass of the bicycle's mask (relative say to the body of the rider). Could this possibly explain why you found a heavier-tailed Laplace distribution to be a good fit for the transitions in Youtube-VOS and KITTI-MOTS?
3) How do natural transition statistics look like in multi-object datasets? How do you envision extending your work to tackle those?
4) In 3.3, you suggest alpha = 1 helps break the rotational symmetry of the ELBO by making axis-aligned representations optimal. Could you substantiate this claim? Theorem 1 does not immediately suggest an identifiability advantage for alpha = 1.
5) On the Natural Sprites dataset, why is it that discrete transitions give SlowVAE an advantage over PCL (MCC score 52.6 versus 50.2 and all other metrics) whereas continuous transitions are disadvantageous (MCC score 49.1 versus 51.7)? Why work with discrete transitions at all?
review_title: Does the motivation hold up more generally?

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Summary: 

The paper addresses the problem of disentangling the underlying generative factors from data with a particular focus on dynamic natural data. It provides evidence that transitions of objects in natural movies can be characterised by temporally sparse distributions. A novel proof based on a sparse prior on temporally adjacent observations is provided, allowing to recover true latent variables up to permutations and sign flips, and improving the disentangling performance over existing methods. Two new datasets with measured natural dynamics are also proposed to enable evaluation on more realistic scenarios.

##################################################################

Strengths:
- The paper addresses the important problem of unsupervised learning of disentangled representations. Overall, it is well written and easy to read.
- A new framework of non-linear ICA is introduced, showing evidence to the hypothesis that natural scenes have sparse temporal transitions.
- Two new benchmarks of natural and unconstrained data are presented which would enable future works on more realistic scenarios.
- The paper provides extensive experiments and comparisons to relevant baselines, including both qualitative analysis and quantitative results using different metrics.

##################################################################

Weaknesses:
- The existing datasets present in DisLib are considered unnatural since they assume data is i.i.d and define a prior on the number of factors to be changed. The datasets presented in this work represent incrementally more realistic scenarios where sparse transitions are imposed in addition to data with natural continuous generative factors and data with transitions from unstructured natural videos. In the Laplace transitions dataset, the rate of the Laplace prior λ is sampled from a uniform distribution λ ∼ U (1, 10) which may result in changes in many factors (when λ is small). Although this allows to have fair comparison with Locatello et al., (2020), this setting may sometimes result in non-sparse transitions between image pairs which is not aligned with the statistics of natural transitions as discussed throughout the paper. This would not be expected if one does not read Appendix 4. Hence, it would be good to clarify this particular point in the main paper. 
- In the Natural Sprites dataset, pairs are produced by only varying the position and scale (with real transitions extracted from the YouTube measurements) while color, shape, and orientations are fixed. While fixing the color and shape can be understood to follow natural transitions of objects, it is unclear why the orientation is not varied in this case?
- For the Kitti masks dataset, continuous natural transitions are considered in all underlying factors. Table 2 shows a clear improvement when the temporal distance between sampled frames ∆t = 5 compared to ∆t = 1. What would be the effect of further varying this parameter?
- From Figure 4, authors raised up an interesting observation for the rotation factor, where SlowVAE shows three sinusoidal oscillations with different frequencies matching the three distinct rotational symmetries of the shapes present in the dataset. However, from the MCC measurements, we can notice that all methods including SlowVAE struggle in disentangling the rotation factor. Here, it would be interesting to further discuss this limitation. 

##################################################################

Final update: Authors addressed all my comments in the rebuttal making significant improvements to the revision. I've increased my score. 

review_title: Generally good paper on disentanglement in more natural data, with extensive evaluation.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
# Summary

This paper introduces a novel VAE-based model with the aim to improve unsupervised disentangling of latent factors in visual data.  This model differs from previous disentangling models in that it takes short (2-frame) videos as input instead of static images.  The model is equipped with a Laplace prior over the dynamics of the video to help it align its representation to axes of sparse temporal dynamics.  The intuition is that the temporal dynamics of natural visual stimuli vary sparsely according to some choice of factors, and that choice of factors is exactly what “disentangled” should refer to.  The authors show that their model achieves better disentangling than previous static-image methods according to a number of metrics.

# Pros

* The interpretation of disentangling as a basis in which the distribution of temporal dynamics of video is sparse is valuable.  Previous approaches to disentangling have been plagued by non-identifiability, and this new interpretation of disentangling is a natural and operational solution.  I fully agree with the authors that temporal information is essential for representation-learning, and hope this paper can help accelerate the field in that direction.
* The paper is clear and well-written.
* The experiments are very thorough in terms of comparisons to previous models and evaluation with previous metrics.
* The application of the Mean Correlation Coefficient (MCC) as a metric for disentangling is good --- I think it is simpler and clearer than many existing disentangling metrics.
* The latent embedding plots are a nice way to visualize latent representations.  While they don’t show what effects non-matched generative factors have on the latent coordinates, they offer valuable information about the latent embedding that is complementary to the commonly used latent traversals.

# Cons

My biggest suggestion is to include an ablation study.  Aside from PCL (which isn’t variational so lives in a different world), there are no existing disentangling models on videos to fairly compare the authors’ model to.  Consequently, it is very important to perform ablation experiments.  More specifically, after reading the paper I have a burning question:  How important is the Laplace prior over transitions?  In other words, can the model work with just a KL regularization between the posteriors for consecutive timesteps?  I think it’s quite possible the answer is “yes” --- simply having a KL regularization instead of the Laplace prior should disentangle better than static-image VAEs because the diagonal posterior will be pressured to align with the transition dynamics.  So I wouldn’t be surprised if the model does quite well with just a KL instead of the Laplace, and that would be a simpler model with two fewer hyperparameters.  So please do this experiment --- regardless of the outcome, the results will be very valuable for readers considering using your model.

Aside from that, I have only a couple minor suggestions:
* Figure 2 is a notationally confusing.  You use z subscripts to indicate time in the lower part of the figure, but in the top part of the figure z subscripts indicate component index.  Maybe make the component index a superscript, or at least make the z in the bottom part of the figure bold so the boldness distinguishes vector from scalar.
* Perhaps make a note that the MCC metric doesn’t work well for discrete latents like shape in dSprites.  For example, in Figure 4 the SlowVAE permutes the shape ordering (e.g. as compared to PCL) and gets a low MCC score for that, but that low score is a drawback of the metric, not the model.  This isn’t unique to MCC --- most other metrics (except MIG) would fall for this too.  But perhaps for discrete latents (ones where we don’t care about the ordering of a discrete set of values), MCC can take the highest over all permutations.  I don’t think it’s necessary to do this, but perhaps you can add a sentence that mentions it so readers understand why the shape score is low.

# Conclusion:

Overall, I recommend this paper to be accepted.  It is very topical, since disentangling has recently been receiving increasing attention, and by incorporating temporal cues into disentangling in VAEs this paper could help steer the field in a productive new direction.  I only hope the authors do the suggested ablation experiment (I think practitioners would appreciate those results).

review_title: Recommendation to Accept

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper starts with an observation that temporal transitions in sequences of natural images are sparse, which is supported by data collected from two big datasets (youtube-vots and kitti-mots). This suggests using a sparse prior for temporal transitions of latent variables when modelling naturalistic scenes. The authors then introduce SlowVAE, a model for temporal independent component analysis (ICA), that depends on such a sparsity prior, and prove that latent variables are identifiable under this model up to permutation and sign flips, which, they say, is stronger than any previous result. Additionally, this work introduces a number of datasets of increasing complexity (and similarity to natural datasets) for testing disentanglement as well as it performs a large scale and detailed evaluation of the introduced model.

The paper is very well written and full of convincing arguments. The related work section (#2) is very well thought-through and extensively describes links to the relevant literature. The model (sections 3.1 and 3.3) is well described--I especially liked section 3.4. which describes assumption made by the theory, and how these assumptions can be violated in practice, which is then followed by extensive experimentation showing that the theory can work even when assumptions are validated. It would be of a great benefit to the community if a similar section was present in other papers.

I concede that I did not understand the proof sketch in sec 3.2 nor the corresponding figure 2, and I did not the full proof in the appendix, therefore I cannot speak to the correctness of the identifiability claim.

I have two remarks about eq. 4.
1) if \gamma != 0, this is not an ELBO, strictly speaking, which is contrary to what the text suggests.
2) We can read that the mean \mu(x_{t-1}) is used as a "single sample" to approximate the last term of that equation. Strictly speaking, this can lead to biased estimates of that term, specifically when the mapping from z_{t-1} to the statistics of p(z_t | . ) is non-linear. Additionally, in high-dimensional distribution, the mode itself is a very unlikely sample. Why is this ok to take the mean in the context of this equation?

The evaluation is extensive (two baselines, one of SOTA from ICA and disentangled representations literature, and 14 datasets). Figure 4, which shows plots of true generative factor vs matched latent variable, is superb. It is a really good method of visualising disentanglement--I agree with the authors that this is a much better way then showing latent traversals. 

The paper seems to be very good, though I am no expert on ICA. I strongly recommend acceptance. The reason I am not giving a higher score is that the significance to the community seems not that high, but please correct me if I am wrong.

Some further questions:
1) in sec 3.1. you write "we assume that the noise [in g] is modeled indirectly as a latent variable". Does that mean that g is itself a latent-variable model?
2) The appendix includes a "broader impact" statement, which strongly suggests that this paper was previously rejected from NeurIPS. May I ask what the main points of criticism were?

Some further remarks:
* In the first paragraph of intro you write "although untrue in the literal sense". Why is that? It seems true to me.
* "this problem" a few lines below the above is unclear.
* Table 2 appears BEFORE Table 1 in the text. This is very confusing!
* It is unclear what "Natural" in Table 1 refers to. I assumed that is refers to the introduces "Natural Sprites" dataset, but maybe I am wrong?
* A hyphen in latex should be typed as "---" and there should be no spaces between the hyphen and the surrounding text; you have one e.g. in  the conclusion.

UPDATE: T`he authors' response addressed all my remarks. I've increased the score.
review_title: Very well written and executed papers, good job!

Meta-review: This paper proposes a model for learning disentangled representations by assuming the slowness prior over transitions between two frames. The model is well justified theoretically, and evaluated extensively experimentally. The results are good, and all reviewers agree that this paper is among the top papers they have reviewed. For this reason, I am pleased to recommend this paper for an Oral.
Decision: Accept (Oral)