Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors address the problem of discovering and predicting with hierarchical structure in data sequences of relevance to planning. Starting with the kinds of data that have been used recently in video prediction, the authors aim at learning a sequence of keyframes (i.e., subsets of frames forming the overall sequence) that in a suitable sense "summarize" the overall trace. As they rightly note, many alternate models struggle with making good long term predictions in part because they focus on all levels of prediction equally.

The technical approach is to pose the problem as one of inferring the temporal location of each of these key frames and then to interpolate with a model to generate intermediate frames. One could try to make either step sophisticated - the authors choose to make the keyframe selection more sophisticated and interpolation simpler. The paper first described the KeyIn model in terms of a probabilistic model of jointly finding the Ks and then the inpainted Is. This can become delicate, so the authors propose a relaxation that is more forgiving when the keyframe locations are being searched for. Learning is driven by a reconstruction loss of finding the approximate location, locally interpolating and then seeing if this accords with the training data. This is all implemented with an LSTM based NN architecture which seems sound to me.

I feel the paper is taking on the right kinds of questions, looking for ways to inject the right kind of structure. I do have some concerns about the overall formulation:

1. Much of the paper is focussed on rather clean images where nothing extraneous is happening. In reality, the backgrounds of real images is not so benign and other extraneous dynamics might interfere. While I understand this is a step towards the long term goal, I wonder if the end result is a bit too incremental in the absence of some attempt to explore this source of (lack of) robustness.

2. In §6.3, the authors try to demonstrate that the number of keyframes parameter can be wrong by a little bit but these are still small ranges. In realistic images it is likely that the total number of keyframes selected by such an algorithm is much larger due to extraneous events. This is why a proper robustness study is crucial on more realistic input. As it, in anything other than the trivial dot on black background, the precision-recall numbers are fairly modest. This will likely degenerate into noise in most camera-based images of the kind seen by a real robot. So, how much confidence should we expect to have in the approach's generality?

3. For the baselines, the true good baseline might have been a human annotation that tells us how people really conceptualise the structure. With data such as pushing, this might not be so different from the simple visual inspection, but again with real data this will vary. The paper would really be much stronger if these were addressed.


 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper introduces a variational objective to train a model which can jointly select keyframes of a video and generate the intervening frames to produce a resultant video. The model is provided an initial set of frames as context. At training time the model always learns to produce N*J frames, where N is the number of keyframes and J is a fixed number of frames to generate for each keyframe. The authors compare their method for selecting informative keyframes on a number of baselines and show an improvement over these baselines. The problem is interesting and well-motivated, but I have some concerns with the proposed approach and experiments. As such, I am a weak reject.

comments / questions:
- Equation 3 lacks context. Initially, when looking at the authors' objective it seems that the inner expectation should be taken with respect to the joint time indices for the current and next keyframe. Only later after equation 4 do they mention that they always predict a fixed number of frames J. 
- The need for normalizing over the first T timesteps in equation 4 seems quite messy. Is it guaranteed that all of the needed keyframes will actually be within the first T timesteps? How does this work in practice?
- Many important details of the inference procedure are relegated to the appendix. For example, there are no details for extracting which of the 60 keyframes that were trained for a sequence (due to the fixed length sequences) should be selected at test time. Looking at the appendix, it is clear that the approach requires an extensive planning algorithm at inference time, which seems like an important component.
- The authors prominently highlight that their method is fully differentiable, yet they train in two stages while freezing weights. Why isn't the model trained end-to-end? The stated reason for doing so is that this "simple" two-stage procedure improves optimization. What exactly happens if you don't do this two stage training process? Does it fail to learn? Some experimental numbers would be nice to see. 
- The authors do not compare their method to any strong keyframe prediction baselines. Considering there is existing work in keyframe prediction, it seems important to highlight the difference between other competing models, rather than relying on simple baselines. Why don't they use self-information/surprisal as a baseline i.e., by training an autoregressive model on the frames and then picking the N frames with the largest -log(p)? This is a metric that has been investigated frequently and has better interpretability than defining a new measure of surprise. Note that Kipf et al. (2019) uses this notion of surprisal as well.
- Sauer et al. (BMVC 2019) should likely be cited as it does very similar keyframe analysis. Also, as the ICML 2019 conference had already concluded by the ICLR submission deadline, is it really fair to state the work with Kipf et al. (2019) was conducted in parallel?
- Why does the model trained to learn a fixed number of timesteps for the intermediate frames? Did they investigate jointly predicting the indices for the current and next timesteps? It seems like it would greatly simplify their inference scheme if they did this. If they tried that approach and it failed, maybe that should be mentioned in the paper (with an explanation as to why it fails).
- In the literature review, when discussing hierarchical temporal structure, the authors state: "However, these models rely on autoregressive techniques for text generation and are not applicable to structured data, such as videos." Autoregressive techniques have been investigated in relation to videos; in fact, the authors later describe papers that have used autoregressive techniques for modeling videos.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper introduces a model trained for video prediction hierarchically: a series of significant frames called “keyframes” in the paper are first predicted and then intermediate frames between keyframes couples are generated. The training criterion is maximum likelihood with a variational approximation. Experiments are performed on 3 different video datasets and the evaluation is performed for 3 tasks: keyframe detection, frame prediction and planning in robot videos.
The idea of generating an abstraction or a summary of a video via a sequence of important frames is attractive and could probably be used in different contexts. The proposed model is new and the authors introduce some clever ideas in order to train it. The evaluation work is important and the authors propose different settings for this evaluation. 
The paper also present weaknesses. First the motivation for keyframes generation should be better developed: the model does not perform better than baselines for video frames prediction so that keyframes generation should be motivated by other applications. Planning as proposed by the authors could be one, but in this case it should be more developed. The main weakness is however the technical presentation which is painful to follow. When it is possible to get a general picture of what is done, it is quite difficult to figure out exactly how the model works. A global rewriting and maybe a better focus are required for publication. The probabilistic model (section 3.1) is relatively clear, even if it could be improved. It seems that the generation of a keyframe and the prediction of the corresponding time (tau^n)  are independent (eq. 3). This could be commented. Also it seems that in eq. 3 the log(K|z..) term should be inside an expectation. Section 4 was difficult to decipher for me. My understanding is that instead of sampling from a multinomial during training, you bypass this non differentiable operation by using what you call “soft targets” thus obtaining a differentiable objective (eq. 4). Is that true? In any case, the procedure should be made a lot clearer. The “intermediate frame” passage also remained confuse for me.
Considering the experiments, the authors make an important effort in order to evaluate different aspects of their model. In a fisrt step, they evaluate the ability of the model to generate significant keyframes using a detection setting.  It is not clear how they define ground truth frames for this evaluation. Those ground truth frames are defined as the frames where the movement in the image changes, which is easy on the Brownian movement dataset but what about the others? Also the baselines used in this comparison are weak. In the paper of Denton, they suggest some way to detect surprise and apparently this is not what you used. This should be justified/ commented. For keyframe modeling the proposed model behaves similarly to the baselines and even performs worse than the simpler “jumpy” model. Concerni g the paragraph about the selection of the number of predicted keyframes, it is not clear what is the reference (ground truth) number of target keyframes.
 The planning experiments are interesting, but difficult to follow at least from the main text.
Overall, I think that there are several interesting ideas and realizations. They should be better put in perspective and explained.

----- post rebuttal -----------

Thanks for the detailed answer. The paper is largely improved both for the style and the comparisons. But still requires further improvements. I will keep my score.

review_title: Official Blind Review #1

Meta-review: The paper is interesting in video prediction, introducing a hierarchical approach: keyframes are first predicted, then intermediate frames are generated. While it is acknowledge the authors do a step in the right direction, several issues remain: (i) the presentation of the paper could be improved (ii) experiments are not convincing enough (baselines, images not realistic enough, marginal improvements) to validate the viability of the proposed approach over existing ones.

Decision: Reject