Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Why are the pathways completely separated? Isn't motion dependent on appearance? Could anything be gained by funneling appearance features earlier into the motion network?
review_title: Independence

final_score: rating: null
review_content: Very clear and interesting paper! I have a few questions before writing my review:

1) The qualitative results seem to only include (almost) static cameras. Could you please include new examples with moving cameras (fairly common in UCF-101, some KTH videos too) and discuss? I am especially interested in seeing how the good quantitative results translate in this setting, in particular for large camera movements. I wonder if your motion encoder can handle this complexity, or if it entangles this and fails (following the same reasoning you applied to justify decoupling appearance and motion). The latter case would then suggest potential improvements by separating further in two motion decoders, as often done in more traditional computer vision (where decoupling camera / background motion from the foreground helps, even when doing simple camera motion compensation).

2) Related to the first question, and in order to grasp better what the actual quantitative improvements represent (although the small qualitative results partially address this already), could you please compare to a stupid baseline where all the pixels from the last frame are simply copy-pasted? From the qualitative examples, it seems that the focus on small motions might reflect a bias of the datasets (or at least their subset where frame prediction methods truly differ), in which case it would be helpful to know if the models do not simply learn to be very conservative (as there is no explicit regularization discouraging that). I personally find the good results of all frame prediction methods very static :-)

3) Could you discuss a bit why you only experiment on human action recognition benchmarks and the implied potential dependency on action labels versus applying your approach on random videos, as done for instance by Vondrick et al. with their VGAN model (NIPS'16)?

4) Have you done any experiment to see if your MCNet+RES model has learned any useful representation for action recognition and compared to the aforementioned VGAN results on UCF-101? I am hopeful, as your architecture and working assumptions are similar to the two-stream network of Simonyan and Zisserman (as you mention), and this model is among the state of the art in action recognition (including on UCF-101, especially the latest variants like the Temporal Segment Networks of Wang et al at ECCV'16).
review_title: Pre-review questions

final_score: rating: 7: Good paper, accept
review_content: This paper introduces an approach for future frame prediction in videos by decoupling motion and content to be encoded separately, and additionally using multi-scale residual connections. Qualitative and quantitative results are shown on KTH, Weizmann, and UCF-101 datasets.

The idea of decoupling motion and content is interesting, and seems to work well for this task. However, the novelty is relatively incremental given previous cited work on multi-stream networks, and it is not clear that this particular decoupling works well or is of broader interest beyond the specific task of future frame prediction.

While results on KTH and Weizmann are convincing and significantly outperform baselines, the results are less impressive on less constrained UCF-101 dataset.  The qualitative examples for UCF-101 are not convincing, as discussed in the pre-review question.

Overall this is a well-executed work with an interesting though not extremely novel idea. Given the limited novelty of decoupling motion and content and impact beyond the specific application, the paper would be strengthened if this could be shown to be of broader interest e.g. for other video tasks.
review_title: well-executed but limited novelty and impact

final_score: rating: null
review_content: You are training VGG size networks on quite small datasets, and you seem to use the same architecture for all datasets. I didn't find any information on pre-training. Didn't you have any issues with overfitting?
review_title: VGG

final_score: rating: null
review_content: The proposed method shows strong results with realistic motion on KTH.  However on the less constrained UCF-101, the qualitative examples show realistic frames but very little motion, especially related to the foreground action of interest as would be desired.  Can you comment on this? 
review_title: static quality of MCNet on UFC-101

final_score: rating: 7: Good paper, accept
review_content: 1) Summary

This paper investigates the usefulness of decoupling appearance and motion information for the problem of future frame prediction in natural videos. The method introduces a novel two-stream encoder-decoder architecture, MCNet, consisting of two separate encoders -- a convnet on single frames and a convnet+LSTM on sequences of temporal differences -- followed by combination layers (stacking + convolutions) and a deconvolutional network decoder leveraging also residual connections from the two encoders. The architecture is trained end-to-end using the objective and adversarial training strategy of Mathieu et al.

2) Contributions

+ The architecture seems novel and is well motivated. It is also somewhat related to the two-stream networks of Simonyan & Zisserman, which are very effective for real-world action recognition.
+ The qualitative results are numerous, insightful, and very convincing (including quantitatively) on KTH & Weizmann, showing the benefits of decoupling content and motion for simple scenes with periodic motions, as well as the need for residual connections.

3) Suggestions for improvement

Static dataset bias:
In response to the pre-review concerns about the observed static nature of the qualitative results, the authors added a simple baseline consisting in copying the pixels of the last observed frame. On the one hand, the updated experiments on KTH confirm the good results of the method in these conditions. On the other hand, the fact that this baseline is better than all other methods (not just the authors's) on UCF101 casts some doubts on whether reporting average statistics on UCF101 is insightful enough. Although the authors provide some qualitative analysis pertaining to the quantity of motion, further quantitative analysis seems necessary to validate the performance of this and other methods on future frame prediction. At least, the results on UCF101 should be disambiguated with respect to the type of scene, for instance by measuring the overall quantity of motion (e.g., l2 norm of time differences) and reporting PSNR and SSIM per quartile / decile. Ideally, other realistic datasets than UCF101 should be considered in complement. For instance, the Hollywood 2 dataset of Marszalek et al would be a good candidate, as it focuses on movies and often contains complex actor, camera, and background motions that would make the "pixel-copying" baseline very poor. Experiments on video datasets beyond actions, like the KITTI tracking benchmark, would also greatly improve the paper.

Additional recognition experiments:
As mentioned in pre-review questions, further UCF-101 experiments on action recognition tasks by fine-tuning would also greatly improve the paper. Classifying videos indeed requires learning both appearance and motion features, and the two-stream encoder + combination layers of the MCNet+Res architecture seem particularly adapted, if they indeed allowed for unsupervised pre-trainining of content and motion representations, as postulated by the authors. These experiments would also contribute to dispelling the aforementioned concerns about the static nature of the learned representations.

4) Conclusion

Overall, this paper proposes an interesting architecture for an important problem, but requires additional experiments to substantiate the claims made by the authors. If the authors make the aforementioned additional experiments and the results are convincing, then this paper would be clearly relevant for ICLR.

5) Post-rebuttal final decision

The authors did a significant amount of additional work, following the suggestions made by the reviewers, and providing additional compelling experimental evidence. This makes this one of the most experimentally thorough ones for this problem. I, therefore, increase my rating, and suggest to accept this paper. Good job!
review_title: Interesting architecture for an important problem, but requires additional experiments.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a method for predicting video sequences in the lines of Mathieu et al. The contribution is the separation of the predictor into two different networks, picking up motion and content, respectively.

The paper is very interesting, but the novelty is low compared to the referenced work. As also pointed out by AnonReviewer1, there is a similarity with two-stream networks (and also a whole body of work building on this seminal paper). Separating motion and content has also been proposed for other applications, e.g. pose estimation.

Details :

The paper can be clearly understood if the basic frameworks (like GANs) are known, but the presentation is not general and good enough for a broad public.

Example : Losses (7) to (9) are well known from the Matthieu et al. paper. However, to make the paper self-contained, they should be properly explained, and it should be mentioned that they are "additional" losses. The main target is the GAN loss. The adversarial part of the paper is not properly enough introduced. I do agree, that adversarial training is now well enough known in the community, but it should still be properly introduced. This also involves the explanation that L_Disc is the loss for a second network, the discriminator and explaining the role of both etc.

Equation (1) : c is not explained (are these motion vectors)? c is also overloaded with the feature dimension c'.

The residual nature of the layer should be made more apparent in equation (3).

There are several typos, absence of articles and prepositions ("of" etc.). The paper should be reread carefully.

review_title: null

Meta-review: Here is a summary of the reviews and discussion:
 
 Strengths
 Idea of decoupling motion and content is interesting in the context of future frame prediction (R3, R2, R1)
 Quantitative and Qualitative results are good (R1)
 
 Weaknesses
 Novelty in light of previous multi-stream networks (R3, R2)
 Not clear if this kind of decoupling works well or is of broader interest beyond future frame prediction (R3) AC comment: I donÕt know if this is too serious a concern; to me the problem seems important enough -- making an advancement that improves a single relevant problem is a contribution
 Separation of motion and content already prevalent in other applications, e.g., pose estimation (R2)
 UCF-101 results are not so convincing (R3)
 Clarity could be improved, not written with general audience in mind (R2)
 Concern about static bias in the UCF dataset of the learned representations (R1, R2, R3) AC comment: The authors ran significant additional experiments to respond to this point
 
 The authors provided a comprehensive rebuttal which convinced R1 to upgrade their score. After engaging R2 and R3 in discussion, both said that the paper had improved in its subsequent revisions and would be happy to see the paper pass. The AC agrees with the opinion of the reviewers that the paper should be accepted as a poster.
Decision: Accept (Poster)