Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposed a new synthetic dataset (CATER) for video understanding. The authors argue that since current video datasets are heavily biased over static scenes and object structures, it is unclear whether modern spatial-temporal video models can learn to reason over temporal dimension. In order to address this problem, they design this fully observable synthetic dataset which is built upon CLEVER, along with three tasks that are customized for temporal understanding. They further conduct a variety of experiments to benchmark state-of-the-art video understanding models and show how those models more or less struggle on temporal reasoning. 

Overall this paper is well-written and easy to follow. The problem is well-motivated, and the claims are mostly supported. The diagnosis in this paper provides useful insights that could be contributive to both vision and learning communities. 

My primary concern is to what extent can the new dataset (CATER) add to existing video datasets that are also explicitly designed for long term spatial-temporal reasoning, such as video VQA datasets TGIF-QA[1]/SVQA[2]. In addition to the comparison between CATER and three action recognition datasets (Kinetics/UCF101/HMDB51) as presented in Table 3., it would be more interesting to see how video understanding models that are specifically designed for those video VQA datasets will perform on CATER. 

[1] Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2758–2766, 2017.
[2] Xiaomeng Song, Yucheng Shi, Xin Chen, and Yahong Han. Explore multi-step reasoning in video question answering. In 2018 ACM Multimedia Conference on Multimedia Conference, pages 239–247. ACM, 2018.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The paper introduces CATER: a synthetically generated dataset for video understanding tasks. The dataset is an extension of CLEVR using simple motions of primitive 3D objects to produce videos of primitive actions (e.g. pick and place a cube), compositional actions (e.g. "cone is rotated during the sliding of the sphere"), and finally a 3D object localization tasks (i.e. where is the "snitch" object at the end of the video).  The construction of the dataset focuses on demonstrating that compositional action classification and long-term temporal reasoning for action understanding and localization in videos are largely unsolved problems, and that frame aggregation-based methods on real video data in prior work datasets, have found relative success not because the tasks are easy but because of dataset bias issues.

A variety of models from recent work are evaluated on the three proposed tasks, demonstrating the validity of the above motivation for the construction of the dataset.  The primitive action classification task is "solved" by nearly all methods and only serves for debugging purposes.  The compositional action classification task is harder and shows that incorporating LSTMs for temporal reasoning leads to non-trivial performance improvements over frame averaging.  Finally, the localization task is challenging, especially when camera motion is introduced, with much space for improvement left for future work.

I am positive with respect to acceptance of this paper.  It is a well-argued, thoughtful dataset contribution that sets up a reasonable video understanding dataset.  The authors recognize that since the dataset is synthetically generated it is not necessarily predictive of how methods would perform with real-world data, but still it can serve a useful and complementary role similar to the one CLEVR has served in image understanding.

I have a few minor comments / questions / editing notes that would be good to address:
- The random baseline isn't described in the main text, it would be good to briefly mention it (this will also help to clarify why the value is particularly high for tasks 1 and 2)
- The grid resolution ablation results presented in the supplement are actually quite important -- they demonstrate that with a small increase in granularity of the grid the traditional tracking methods begin to be the best performers. As this direction (of increased resolution to make the problem less artificial) is likely to be important, a brief discussion of this finding from the main paper text would be appropriate
- p3 resiliance -> resilience
- p4 objects is moved -> object is moved
- p6 actions itself -> actions themselves; builds upon -> build upon
- p7 looses all -> loses all; suited our -> suited to our; render's camera parameters -> render camera parameters; to solve it -> to solve the problem
- p8 (Xiong, b;a) and (Xiong, b) -> these references are missing the year; models needs to -> models need to
- p9 phenomenon -> phenomena; the the videos -> the videos; these observation -> these observations; of next -> of the next; in real world -> in the real world

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper introduces a new synthetic video understanding dataset, borrowing many ideas from the visual question answering dataset CLEVR. The new dataset is the first to account for all of the following fundamental aspect of videos: temporal ordering, short- and long term reasoning, and control for scene biases. Due to the inherent biases in available action recognition datasets, models that simply averages video frames do nearly as well as models that take temporal dependencies into account. In contrast, the authors show that with the proposed dataset, models without spatiotemporal reasoning largely fail.

The paper should be accepted as it addresses a major shortcoming of all existing video understanding datasets. It does a good job at summarizing the deficiencies in existing datasets, clearly motivating the need for a new dataset. The claims are backed up with solid experiments, ablating models and data parameters adequately. It is mostly well-written (except for section 4 which would benefit from extensive proofreading) and does a good job at covering relevant work. One drawback is of course the synthetic nature and limited domain of objects and actions. On the other hand, this makes the setup highly controllable and reliable. I like the fact that each task comes both with both static and moving camera.

Improvements and Questions:
Some relevant datasets are missing. For example, the Moving MNIST and Robot Pushing datasets could be added to Table 1.

I suggest having a train / validation / test split (like CLEVR), rather than just a train and validation split.

In particular for Task 3 more frames seem to give dramatic improvement. Why did you not run with more than 64 frames? 

Did you consider downsampling the videos to allow running on all the frames?

I’m missing details on the resolution of the generated videos?

review_title: Official Blind Review #3

Meta-review: The paper proposed a new synthetically generated video dataset (CATER) for benchmarking temporal reasoning. The dataset is based on the CLEVR dataset and provides videos make up of primitive actions ("rotate", "pick-place", "slide", "contain") that can be combined to form for complex actions.
The paper also benchmarks a variety of methods on three proposed tasks (atomic action classification, composite action classification, and 'snitch' localization) and demonstrates that while it is possible to get high performance on atomic action classification, the other two task are still challenging and requires temporal modeling.  

Overall, all reviewers found the paper to be well written and easy to follow, with care given to the dataset construction, as well as the task definitions and experiment setup and analysis.  The paper received strong scores from all reviewers (3 accepts).  Based on the reviewer comments, the authors further improved the paper by adding additional relevant datasets for comparison and providing missing details pointed out by the reviewers.  After the rebuttal, the reviewers remained positive.
Decision: Accept (Talk)