Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The results that progressively over time adapt and improve is very interesting to see.
Unfortunately, from the Figures and description I cannot say I exactly understand how information flows, e.g., boundary conditions are missing from Eq. 1-4 , which makes then hard to follow.
If you could help us follow better the information flow in the model, (maybe with a gif? ) that would be great, since the model architecture is very different than what most people have seen so far.

One question is: In your Figure 6, the finetuned model, has it seen this particular video sequence during finetuning, or it is a test sequence you are showing?

review_title: Model architecture - finetuning

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Learning about the physical structure and semantics of the world from video (without supervision) is a very hot area in computer vision and machine learning.
In this paper, the authors investigate how the prediction of future image frames (inherently unsupervised) can help to deduce object/s structure and it's properties (in this case single object pose, category, and steering angle, (after a supervised linear readout step))

I enjoyed reading this paper, it is clear, interesting and proposes an original network architecture (PredNet) for video frame prediction that has produced promising results on both synthetic and natural images.
Moreover, the extensive experimental evaluation and analysis the authors provide puts it on solid ground to which others can compare.

The weaknesses:
- the link to predictive coding should be better explained in the paper if it is to be used as a motivation for the prednet model.
- any idea that the proposed method is learning an implicit `model' of the `objects' that make up the `scene' is vague and far fetched, but it sounds great.

Minor comment:
Next to the number of labeled training examples (Fig.5), it would be interesting to see how much unsupervised training data was used to train your representations.
review_title: Review

final_score: rating: null
review_content: # Motivation
The leap/link from `predictive coding' to the `PredNet' architecture is not very clear from the text.
Can you clarify this?

# The PredNet Model
How did you decide that the `PredNet' architecture was good for deducing the structure of the world by predicting future video frames? This is unclear from the text. Was it arrived at through trial and error? Is there prior work that has already shown the architecture to be promising for this application?

# Experiment 3.1
How many different faces are there? What kinds of variations are present across the different faces?

Deducing the latent orientations/categories of the faces from the representation of the network via ridge-regression/linear-svm respectively is a supervised learning process, is that right?
Doesn't that go against the philosophy of deducing the object/s structure and properties from video in an unsupervised manner?

Who proposed the CNN-LSTM decoder? 
Why not compare your results to other methods that predict future image frames (in your paper you cite 9 papers predicting future frames)?

# Experiment 3.2
What is a linear readout?
What is the input to the fully supervised methods?
Do you notice any differences in prediction accuracy between sequences in which there are moving objects vs sequences in which only ego-motion is present?

# Discussion
What other structures (apart from single object pose, and steering angle) do you think could be deduced using this method?
How did you address the original goal of deducing an object's structure in natural scenes?

"predicting future frames requires at least an implicit model of the objects that make up the scene", do you have any experimental evidence that you have captured the properties of objects in a scene?
Rather, the main message of the paper seems to be that learning representations in an unsupervised manner is useful for supervised training methods to get better prediction accuracy. Is that a fair comment?

review_title: Several questions!

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Paper Summary
This paper proposes an unsupervised learning model in which the network
predicts what its state would look like at the next time step (at input layer
and potentially other layers).  When these states are observed, an error signal
is computed by comparing the predictions and the observations. This error
signal is fed back into the model. The authors show that this model is able to
make good predictions on a toy dataset of rotating 3D faces as well as on
natural videos. They also show that these features help perform supervised
tasks.

Strengths
- The model is an interesting embodiment of the idea of predictive coding
  implemented using a end-to-end backpropable recurrent neural network architecture.
- The idea of feeding forward an error signal is perhaps not used as widely as it could
  be, and this work shows a compelling example of using it. 
- Strong empirical results and relevant comparisons show that the model works well.
- The authors present a detailed ablative analysis of the proposed model.

Weaknesses
- The model (esp. in Fig 1) is presented as a generalized predictive model
  where next step predictions are made at each layer. However, as discovered by
running the experiments, only the predictions at the input layer are the ones
that actually matter and the optimal choice seems to be to turn off the error
signal from the higher layers. While the authors intend to address this in future
work, I think this point merits some more discussion in the current work, given
the way this model is presented.
- The network currently lacks stochasticity and does not model the future as a
  multimodal distribution (However, this is mentioned as potential future work).

Quality
The experiments are well-designed and a detailed analysis is provided
in the appendix.

Clarity
The paper is well-written and easy to follow.

Originality
Some deep models have previously been proposed that use predictive coding.
However, the proposed model is most probably novel in the way it feds back the
error signal and implements the entire model as a single differentiable
network.

Significance
This paper will be of wide interest to the growing set of researchers working
in unsupervised learning of time series. This helps draw attention to
predictive coding as an important learning paradigm.

Overall
Good paper with detailed and well-designed experiments. The idea of feeding
forward the error signal is not being used as much as it could be in our
community. This work helps to draw the community's attention to this idea.
review_title: Good paper, nice example of using the idea of feeding forward error signals.

final_score: rating: null
review_content: Most of the questions I had on reading the paper were answered by the ablative analysis in the appendix. Thank you for having that!

(1) For the experiments on natural image sequences (KITTI dataset), it is mentioned that the models were trained with loss only on the activity of the lowest error layer. Did having the loss at higher layers as well not help ? Since the loss is "grounded" in the lowest layer only and all other layers are measuring losses in learned representations, I would suspect it would make this a hard optimization problem. Is that the only reason or is something else at play ?

(2) For the classification experiments, the representations were read out after one step of running the model. Does it make sense/help to run the model for longer, feeding in the same image at each step ? In other words, since one would expect the prediction after 1 step to not be very sharp, are there any benefits to letting the model recurse on the same image till it can make sharper predictions ?

(3) For the steering angle experiments, steering angle is a function of an image sequence, not a single image. It is not immediately clear how the Autoencoder Model (Fig 5, right) uses sequence information in predicting the steering angle. Also for the comma.ai model and ResNet50 could you please clarify how these were trained ? A brief description of what the objective function was, and where sequence information was used would help put these models in context.
review_title: Why loss at input layer only and other questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: An interesting architecture that accumulates and continuously corrects mistakes as you see more and more of a video sequence.

Clarity: The video you generated seems very helpful towards understanding the information flow in your network, it would be nice to link to it from the paper.

 "Our model with hyperparameters optimized for KITTI underperforms the model of Finn et al. (2016), but outperforms the previous state-of-the-art model by Mathieu et al. (2016)."

 It is not clear how different are the train and test sequences at the moment, since standard benchmarks do not really exist for video prediction and each author picks his/her favorite. Underperforming Finn et al 206 at the H3.6m Walking videos is a bit disappointing.


review_title: an interesting architecture for future prediction inspired by deep predictive coding

Meta-review: This paper proposes an interesting architecture for predicting future frames of videos using end-to-end trained deep predictive coding.
  The architecture is well presented and the paper is clearly written. The experiments are extensive and convincing, include ablation analyses, and show that this architecture performs well compared to other current methods.
 Overall, this is an interesting, solid contribution.
Decision: Accept (Poster)