Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a generative model of videos composed of a background and a set of 2D objects (sprites). Optimization is performed under a VAE framework.

The authors' proposal of an outer product of softmaxed vectors (resulting in a 2D map that is delta-like), composed with a convolution, is a very interesting way to achieve translation of an image with differentiable parameters. It seems to be an attractive alternative to more complicated differentiable resamplers (such as those used by STNs) when only translation is needed.

Below I have made some comments regarding parts of the text, especially the experiments, that are not clear. The experimental section in particular seems rushed, with some results only alluded to but not given, not even in the appendix.

For an extremely novel and exotic proposal, showing only synthetic experiments could be excused. However, though there is some novelty in the method, it is disappointing that there isn't even an attempt at trying to tackle a problem with real data.

I suggest as an example aerial videos (such as those taken from drone platforms), since the planar assumption that the authors make would most probably hold in that case.

I also suggest that the authors do another pass at proof-reading the paper. There are missing references ("Fig. ??"), unfinished sentences (caption of Fig. 5), and the aforementioned issues with the experimental exposition.
review_title: .

final_score: rating: null
review_content: 1. Why does Algorithm 1 include both the delta-convolution and an STN? It seems that one would choose one or the other (Section 2) to implement a translation operator, not use both.

2. In the experiments, it seems that the Convolutional-PUN uses an LSTM, while the STN-PUN uses an RNN. If so, this gives the Convolutional-PUN an unfair advantage. Why wasn't the same base RNN architecture used for both? If both use LSTMs, which is reasonable, the text should be clear that this is the case.

3. Drawing conclusions from results that are not in the main text or even appendix (!) is very sloppy. In one instance (paragraph right before section 4.2) it is claimed that they can be obtained from the source code. In another (third paragraph of section 4.2) it is claimed that the proposed method generates high likelihood frames for longer than the baseline, but this is not shown or quantified anywhere. Can the authors comment on why this was not included even as appendix material?

review_title: Questions

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a generative model of video sequence data where the frames are assumed to be generated by a static background with a 2d sprite composited onto it at each timestep.  The sprite itself is allowed to dynamically change its appearance and location within the image from frame to frame.  This paper follows the VAE (Variational Autoencoder) approach, where a recognition/inference network allows them to recover the latent state at each timestep.

Some results are presented on simple synthetic data (such as a moving rectangle on a black background or the “Moving MNIST” data.  However, the results are preliminary and I suspect that the assumptions used in the paper are far too strong too be useful in real videos.  On the Moving MNIST data, the numerical results are not competitive to state of the art numbers.

The model itself is also not particularly novel and the work currently misses some relevant citations.  The form of the forward model, for example, could be viewed as a variation on the DRAW paper by Gregor et al (ICML 2014).  Efficient Inference in Occlusion-Aware Generative Models of Images by Huang & Murphy (ICLR) is another relevant work, which used a variational auto-encoder with a spatial transformer and an RNN-like sequence model to model the appearance of multiple sprites on a background.

Finally, the exposition in this paper is short on many details and I don’t believe that the paper is reproducible from the text alone.  For example, it is not clear what the form of the recognition model is…  Low-level details (which are very important) are also not presented, such as initialization strategy.
review_title: Mostly incremental generative model of video data with preliminary experimental results

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents an approach to modeling videos based on a decomposition into a background + 2d sprites with a latent hidden state. The exposition is OK, and I think the approach is sensible, but the main issue with this paper is that it is lacking experiments on non-synthetic datasets. As such, while I find the graphics inspired questions the paper is investigating interesting, I don't think it is clear that this work introduces useful machinery for modeling more general videos.

I think this paper is more appropriate as a workshop contribution in its current form.
review_title: Experimental results are too preliminary

final_score: rating: null
review_content: Is there a reason why your stn based model seems to diverge? (fig. 4) What do the training set curves look like? Also, have you considered a LSTM baseline that uses spatial structure like your convolutional pun model? It would help disentangle whether your pun model is helping more or just using spatial information via a convolution is good enough. 
review_title: divergence in figure 4

Meta-review: The strengths and weaknesses pointed out by the reviews were:
 
 Strengths
 Interesting methodology to achieve differentiable translation of an image (R1, R2)
 Graphics-motivation is unique, inspiring (R1)
 
 Weaknesses
 Experiments not clear (R2)
 Novelty is not enough to justify synthetic-only experiments (R1,R2)
 Missing important citations (R3)
 Lacking details, concern with reproducibility (R3)
 
 The authors acknowledged the reviews with a short sentence but did not provide any feedback or revisions.
 
 The AC and PC agree with the reviewers that the paper presents interesting preliminary work which is more suitable for a workshop in its current form.
Decision: Invite to Workshop Track