Venue: ICLR 2018 Workshop

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content:  
==========================================
Summary
==========================================
 
The paper proposes Moving Symbols, a means to generate synthetic video datasets with the capability of controlling changes on apperance and motion of objects occurring in the video. The main goal of the proposed dataset is to allow systematic experimentation of video prediction methods.

----------------
Novelty
----------------

Even though the proposed dataset is an extension of the Moving MNIST dataset (Srivastava et al., ICML'15), its contribution increases the benchmarking capabilities of the dataset.

----------------
Clarity
----------------

The paper is clear and its content is easy to follow.

----------------
Significance
----------------

The proposed dataset is a timely addition to recent interest on video prediction.
In addition, it allows for the systematic evaluation of these methods.

----------------
Quality
----------------

The manuscript displays good quality. Contributions are properly motivated, the evaluation protocol is clear and observations made on the evaluation are adequately discussed.


==========================================
PROs:
- Clear and easy to follow.
- Need of the proposed dataset is properly motivated.
- Observations of the conducted experiments are adequately discussed.
- Evaluation code is made available.


CONS:
- Even though the evaluation is appropriate, it is limited to a single existing method, i.e. MCNet (Villegas et al. ICLR'17).
==========================================

review_title: A  means to allow for systematic evaluation of video-based methods

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new dataset, which can be seen as a variation (in spirit at least) of the Moving MNIST dataset (Srivastava et al., 2015). Specifically, instead of having the same data classes in both training and test set, the proposed dataset has different ones (e.g., MNIST like in training, Omniglot like in test). The reason is that by having the same "semantic" distributions in both training and test sets, it's harder to judge whether the semantics themselves influence the learning of dynamics of objects over time. What is more, different from Moving MNIST, where the  image source files and speed of dynamics are hard coded, in this  paper they are sampled on the fly.

The dataset is an interesting contribution, and there is not much to say to make it better. I would note that sampling on the fly is better to evaluate the generalization capabilities of the algorithms. However, at the same time it makes comparisons between different methods from different papers somewhat arbitrary. It would be good if the authors offered also "comparison-friendly" dataset hyperparameters, so that new papers can compare with previous methods on equal grounds. Another variation that could also be considered is the scale of symbols, e.g., their size can vary over time. To make this more general, other geometric transformations can be considered, such as skewing, or even symbol morphings? Of course, in the end each variation should serve a purpose, namely what type of dynamics are tested here?
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: A dataset is introduced where appearance and motion statistics are varied in a controllable manner between train and test time, and various splits are proposed to test capabilities of current video prediction methods to generalize to novel motion or appearance profiles between train and test.
The dataset is tested on the model of Villegas et al 2017.
The conclusion is it does not generalize to novel content or motion profiles.
Though controlling such train/test dissimilarity statistics seems like a great idea, it is not clear that a learning algorithm or even the human brain can or should generalize to much novel motion profiles or appearances. This though is a different discussion. i find the dataset very valuable for evaluation purposes.

review_title: a dataset with controllable amount of train/test motion-content differences for video prediction 

Meta-review: Congratulations, your paper was accepted to the ICLR workshop.
Decision: Accept