Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Review of “Unsupervised-Learning of time-varying features”

This work looks at using a conditional VAE-based approach to model transformations of sequential data (transformations can be spatial, such as rotation of MNIST, or temporal, i.e. screenshots of a car racing game). Like how our own visual system encodes differences in time and space [1], they show that in generative modelling with VAEs, “there is an advantage of only learning features describing change of state between images, over learning the states of the images at each frame.”

Such an encoding allows the model to ignore features that are constant over time, and also makes it easier to represent data in a rotating curved manifold. They demonstrate this using data collected from CarRacing-v0 task (a task where agents have to learn to drive from pixel observation of a top-down track), and also on MNIST where the digits are rotated around the center of an image. They provide interesting analysis of the latent space learned and show that indeed this approach can handle both stationary and non-stationary features well (in CarRacing-v0). For MNIST, they compare the latent space learned from transformations (z_dot) and show that this approach can encode image geometric transformations quite well.

While this paper is interesting and highlights advantages of modeling transformations of sequential data, I don't think the contributions are currently sufficient for ICLR conference (right now it is a good workshop paper IMHO). For it to be at the conference level, I can make a few suggestions of things that will bring it there, hopefully the authors can take these points as feedback to help improve the work:

1) Would be great to see how this approach can compare to existing proposed algorithms (i.e. TD-VAE as cited)? Are there problems where this approach will perform really well that current methods are inadequate?

2) As the method is based on an RL-task, would the latent representation learned be useful for an RL agent that relies on the latent code across several representative RL tasks (in both sample efficiency, and/or terminal performance)?

I don't mean to discourage the authors (esp as an Anon Reviewer #2...), as I like the direction of the work, and also appreciate that a lot of effort has gone into this work. I hope to see the authors take the criticism to make their work better. Good luck!

[1] Concetta1988, Feature detection in human vision: A phase-dependent energy model


review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper presents a VAE model. The authors consider time series data and claim that in this situation it is better to model the transformations in the latents instead of each datum independently. The setup is reasonable and seems novel, but since it stops at image registration, which is a well-known existing model I cannot qualify the paper as novel. The paper is mostly clear, some claims are not backed up by experiments and the experiments are lacking. As I motivate below I find the current content more at a workshop level than a conference paper. 

Major issues:
* This paper can become a conference paper in two ways in my opinion. 1) It either needs to show that richer modeling has benefits (if anything it would seem from fig 5 that this is not the case). A way towards that would be to take data where there are no simple transformations that we can introduce and show that it discovers reasonable ones. And 2) show on some highly varying temporal domain that this is better than differences of z. 
* on page 2 " the initial assumption that the time-series must be stationary can be fulfilled" -- The data doesn't have to comply with our standards of stationarity. A more sensible formulation is we add these additional constraints to our model which are correct if the data is stationary. 
* On page 3 "to make sure that the latent space can be interpreted". This is a very strong claim, it implies that if we do this the latent space will always be interpretable, which I think is false and definitely not backed up by experiments.
* The conditions on \dot{z} are interesting and potentially useful and they should be explored in experiments. Putting them in or not does it really make the sense that we think it should make ? Ideally in a setup where the data is not trivial.
* I am not sure what insight a reader can possibly get from figure 3. 
* Given the final image-registration setup I find that the following citations are necessary: 
jaderberg et al. Spatial transformer networks, Shu et al. Deforming auto-encoders: unsupervised disentangling of shape and appearance. 
Minor issues:
* the authors should number all equations. 
* In their first equation (not numbered) the indices go beyond N+1. 
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper presents a VAE architecture that separates a fixed content representation and time varying features of an image, that can be used to learn a representation of image transformations in a temporal setting.

The paper is well written and the ideas presented are interesting, but in my opinion not novel enough or thoroughly demonstrated to justify acceptance:

- there is a very relevant work that is not mentioned by the authors and that can be seen as a generalization of the model presented in this paper: "Disentangled Sequential Autoencoder" by Li and Mandt (ICML 2018), which introduces a model that is also disentangling a content and a temporal representation of sequential data. This is basically the more general model introduced by the authors of this submission in the beginning of section 2, without all the assumptions made in the rest of section 2. A comparison with this related work would help assess the differences in terms of modelling power and in performances.

- The assumptions made in this work are fairly strong for most interesting applications, in particular the fact that the content cannot change across time steps.

- To me, the issue with the novelty of this model would not be a big problem if the authors focused more on showing its usefulness in different applications (e.g. medical domain or RL as mentioned in the conclusions). However, the authors only demonstrate the TEVAE on relatively simple experiments that are only tailored to simple image transformations.

review_title: Official Blind Review #1

Meta-review: This work proposes a VAE-based model for learning transformations of sequential data (the main here intuition is to have the model learn changes between frames without learning features that are constant within a time-sequence). All reviewers agreed that this is a very interesting submission, but have all challenged the novelty and rigor of this paper, asking for more experimental evidence supporting the strengths of the model. After having read the paper, I agree with the reviewers and I currently see this one as a weak submission without potentially comparing against other models or showing whether the representations learned from the proposed model lead in downstream improvements in a task that uses this representations.
Decision: Reject