Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: In Figure 2, the model feeds back the generated output at each time step. But during training does the model use the ground truth as input at each time step ? This seems to be the case from Figure 3 : X_{t_n -1} is the input not \tilde{X_{t_n-1}}. Just want to confirm if this is indeed the case.
review_title: Feed-in ground truth or generated output during training ?

final_score: rating: null
review_content: Results indicate the proposed models have a harder time generalizing when the heights of the train and test towers are not the same. In all experiments only one tower height at a time is used for training. Does training with multiple heights lead to better generalization on unseen heights?

Section 5 seems to be cut off at the end, could the authors upload a complete version?
review_title: Update and Experiment Request

final_score: rating: 3: Clear rejection
review_content: *** Paper Summary ***

The paper proposes to learn a predictive model (aka predict the next video frames given an input image) and uses the prediction from this model to improve a supervised classifier. The effectiveness of the approach is illustrated on a tower stability dataset.

*** Review Summary ***

This work seems rather preliminary in terms of experimentation and using forward modeling as pretraining has already been proposed and applied to video and text classification tasks. Discussion on related work is insufficient. The end task choice (will there be motion?) might not be the best to advocate for unsupervised training.

*** Detailed Review ***

This work seems rather preliminary. There is no comparison with alternative semi-supervised strategies. Any approach that consider the next frames as latent variables (or privileged information) can be considered. Also I am not sure if the supervised stability prediction model is actually needed once the next frame is predicted. Basically the task can be reduced to predict whether there will be motion in the video following the current frame or not (for instance comparing the first frame and last prediction or the density of gray in the top part of the video might work just as well). Also training a model to predict the presence of motion from the unsupervised data only would probably do very well. I would suggest to stir away from task where the label can be inferred trivially from the unsupervised data, meaning that unlabeled videos can be considered labeled frames in that case.

The related work section misses a discussion on previous work on learning unsupervised features from video (through predictive models, dimensionality reduction...) for helping classification of still images or videos [Fathi et al 2008; Mabahi et al 2009; Srivastava et al 2015]. More recently, Wang and Gupta (2015) have obtained excellent ImageNet results from features pre trained on unlabeled videos. Vondrick et al (2016) have shown that generative models of video can help initialize models for video classification tasks. Also in the field of text classification, pre training of classifier with a language model is a form predictive modeling, e.g. Dai & Le 2015.

I would also suggest to report test results on the dataset from Lerrer et al 2016 (I understand that you need your own videos to pre train the predictive model) but stability prediction only require still images.

Overall, I feel the experimental section is too preliminary. It would be better to focus on a task where solving the unsupervised task does not necessarily imply that the supervised task is trivially solved (or conversely that a simple rule can turn the unlabeled data into label data).

*** Reference ***

Fathi, Alireza, and Greg Mori. "Action recognition by learning mid-level motion features." Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 2008.
Mobahi, Hossein, Ronan Collobert, and Jason Weston. "Deep learning from temporal coherence in video." Proceedings of the 26th Annual International Conference on Machine Learning. ACM, 2009.
Srivastava, Nitish, Elman Mansimov, and Ruslan Salakhutdinov. "Unsupervised learning of video representations using lstms." CoRR, abs/1502.04681 2 (2015).
A. Dai, Q.V. Le, Semi-supervised Sequence Learning, NIPS, 2015
Unsupervised learning of visual representations using videos, X Wang, A Gupta, ICCV 2015 
Generating videos with scene dynamics, C Vondrick, H Pirsiavash, A Torralba, NIPS 16

review_title: This work seems rather preliminary in terms of experimentation and using forward modeling as pretraining has already been proposed and applied to video and text classification tasks. Discussion on related work is insufficient. The end task choice (will there be motion?) might not be the best to advocate for unsupervised training.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary
===
This paper trains models to predict whether block towers will fall down
or not. It shows that an additional model of how blocks fall down
(predicting a sequence of frames via unsupervised learning) helps the original
supervised task to generalize better.

This work constructs a synthetic dataset of block towers containing
3 to 5 blocks places in more or less precarious positions. It includes both
labels (the tower falls or not) and video frame sequences of the tower's
evolution according to a physics engine.

Three kinds of models are trained. The first (S) simply takes an image of a
tower's starting state and predicts whether it will fall or not. The
other two types (CD and CLD) take both the start state and the final state of the
tower (after it has or has not fallen) and predict whether it has fallen or not,
they only differ in how the final state is provided. One model (ConvDeconv, CD)
predicts the final frame using only the start frame and the other
(ConvLSTMDeconv) predicts a series of intermediate frames before coming
to the final frame. Both CD and CLD are unsupervised.

Each model is trained on towers of a particular heigh and tested on
towers with an unseen height. When the height of the train towers
is the same as the test tower height, all models perform roughly the same
(with in a few percentage points). However, when the test height is
greater than the train height it is extremely helpful to explicitly
model the final state of the block tower before deciding whether it has
fallen or not (via CD and CLD models).


Pros
===

* There are very clear (large) gains in accuracy from adding an unsupervised
final frame predictor. Because the generalization problem is also particularly
clear (train and test with different numbers of blocks), this makes for
a very nice toy example where unsupervised learning provides a clear benefit.

* The writing is clear.


Cons
===

My one major concern is a lack of more detailed analysis. The paper
establishes a base result, but does not explore the idea to the extent
to which I think an ICLR paper should. Two general directions for potential
analysis follow:

* Is this a limitation of the particular way the block towers are rendered?

The LSTM model could be limited by the sub-sampling strategy. It looks
like the sampling may be too coarse from the provided examples. For the
two towers in figure 2 that fall, they have fallen after only 1 or 2
time steps. How quickly do most towers fall? What happens if the LSTM
is trained at a higher frame rate? What is the frame-by-frame video
prediction accuracy of the LSTM? (Is that quantity meaningful?)
How much does performance improve if the LSTM is provided ground truth
for only the first k frames?

* Why is generalization to different block heights limited?

Is it limited by model capacity or architecture design?
What would happen if the S-type models were made wider/deeper with the CD/CLD
fall predictor capacity fixed?

Is it limited by the precise task specification?
What would happen if networks were trained with towers of multiple heights
(apparently this experiment is in the works)?
I appreciate that one experiment in this direction was provided.

Is it limited by training procedure? What if the CD/CLD models were trained
in an end-to-end manner? What if the double frame fall predictor were trained
with ground truth final frames instead of generated final frames?


Minor concerns:

* It may be asking too much to re-implement Zhang et. al. 2016 and PhysNet
for the newly proposed dataset, but it would help the paper to have baselines
which are directly comparable to the proposed results. I do not think this
is a major concern because the point of the paper is about the role of
unsupervised learning rather than creating the best fall prediction network.

* The auxiliary experiment provided is motivated as follows: 
"One solution could be to train these models to predict how many blocks have
fallen instead of a binary stability label."
Is there a clear intuition for why this might make the task easier?

* Will the dataset, or code to generate it, be released?



Overall Evaluation
===
The writing, presentation, and experiments are clear and of high enough
quality for ICLR. However the experiments provide limited analysis past
the main result (see comments above). The idea is a clear extension of ideas behind unsupervised
learning (video prediction) and recent results in intuitive physics from
Lerer et. al. 2016 and Zhang et. al. 2016, so there is only moderate novelty.
However, these results would provide a valuable addition to the literation,
especially if more analysis was provided.

review_title: Good work, though more detailed analysis would be helpful

final_score: rating: 5: Marginally below acceptance threshold
review_content: Paper Summary
This paper evaluates the ability of two unsupervised learning models to learn a
generalizable physical intuition governing the stability of a tower of blocks.
The two models are (1) A model that predicts the final state of the tower given
the initial state, and (2) A model that predicts the sequence of states of this
tower over time given the initial state. Generalizability is evaluated by
training a model on towers made of a certain number of blocks but testing on
towers made of a different number of blocks.

Strengths
- This paper explores an interesting way to evaluate representations in terms of
  their generalizability to out-of-domain data, as opposed to more standard
methods which use train and test data drawn from the same distribution.
- Experiments show that the predictions of deep unsupervised learning models on
  such out-of-domain data do seem to help, even though the models were not
trained explicitly to help in this way.

Weaknesses

- Based on Fig 4, it seems that the models trained on 3 blocks (3CD, 3CLD)
  ``generalize" to 4 and 5 blocks.  However, it is plausible that these models
only pay attention to the bottom 3 blocks of the 4 or 5 block towers in order to
determine their stability. This would work correctly a significant fraction of
the time. Therefore, the models might actually be overfitting to 3 block towers
and not really generalizing the physics of these blocks. Is this a possibility ?
I think more careful controls are needed to make the claim that the features
actually generalize. For example, test the 3 block model on a 5 block test set
but only make the 4th or 5th block unstable. If the model still works well, then
we could argue that it is actually generalizing.

- The experimental analysis seems somewhat preliminary and can be improved. In
  particular, it would help to see visualizations of what the final state looks
like for models trained on 3 blocks but test on 5 (and vice-versa). That would
help understand if the generalization is really working. The discriminative
objective gives some indication of this, but might obfuscate some aspects of
physical realism that we would really want to test.  In Figure 1 and 2, it is
not mentioned whether these models are being tested on the same number of blocks
they were trained for.

- It seems that the task of the predicting the final state is really a binary
  task - whether or not to remove the blocks and replace them with gray
background. The places where the blocks land in case of a fall is probably quite
hard to predict, even for a human, because small perturbations can have a big
impact on the final state. It seems that in order to get a generalizable
physics model, it could help to have a high frame rate sequence prediction task.
Currently, the video is subsampled to only 5 time steps.

Quality
A more detailed analysis and careful choices of testing conditions can increase
the quality of this paper and strengthen the conclusions that can be drawn from
this work.

Clarity
The paper is well written and easy to follow.

Originality
The particular setting explored in this paper is novel.

Significance
This paper provides a valuable addition to the growing work on
transferability/generalizability as an evaluation method for unsupervised
learning. However, more detailed experiments and analysis are needed to make
this paper significant enough for an ICLR paper.

Minor comments and suggestions
- The acronym IPE is used without mentioning its expansion anywhere in the text.

- There seems to be a strong dependence on data augmentation. But given that
  this is a synthetic dataset, it is not clear why more data was not generated
in the first place.

- Table 3 : It might be better to draw this as a 9 x 3 grid : 9 rows corresponding to the
models and 3 columns corresponding to the test sets. Mentioning the train set is
redundant since it is already captured in the model name. That might make it
easier to read.

Overall
This is an excellent direction to work and preliminary results look great.
However, more controls and detailed analysis are needed to make strong
conclusions from these experiments.
review_title: Good preliminary work, more controls and detailed analysis are needed.

Meta-review: A summary of strengths and weaknesses brought up in the reviews:
 
 Strengths
 -Paper presents a novel way to evaluate representations on generalizability to out-of-domain data (R2)
 -Experimental results are encouraging (R2)
 -Writing is clear (R1, R2)
 
 Weaknesses
 -More careful controls are needed to ascertain generalization (R2)
 -Experimental analysis is preliminary and lack of detailed analysis (R1, R2, R3)
 -Novelty and discussion of past related work (R3)
 
 The reviewers are in consensus that the idea is exciting and at least of moderate novelty, however the paper is just too preliminary for acceptance as-is. The authors did not provide a response. This is surprising because specific feedback was given to improve the paper and it seems that the paper was just under the bar. Therefore I have decided to align with the 3 reviewers in consensus and encourage the authors to revise the paper to respond to the fairly consistent suggestions for improvement and re-submit. Mentime, I'd like to invite the authors to present this work at the workshop track.
Decision: Invite to Workshop Track