Title: Perception Updating Networks: On architectural constraints for interpretable video generative models
Authors: Eder Santana
Jose C Principe
Venue: ICLR 2017
Keywords: Structured prediction
Unsupervised Learning

Abstract: We investigate a neural network architecture and statistical framework that models frames in videos using principles inspired by computer graphics pipelines. The proposed model explicitly represents "sprites" or its percepts inferred from maximum likelihood of the scene and infers its movement independently of its content. We impose architectural constraints that forces resulting architecture to behave as a recurrent what-where prediction network.

TL;DR: Decoupled "what" and "where" variational statistical framework and equivalent multi-stream network 

Review discussion:
Is there a reason why your stn based model seems to diverge? (fig. 4) What do the training set curves look like?
Also, have you considered a LSTM baseline that uses spatial structure like your convolutional pun model? It would help disentangle whether your pun model is helping more or just using spatial information via a convolution is good enough.
Dear reviewer, thanks for your comments. Please, see the answers below:
1) Its not diverging, this is the test set. STN based model overfit much faster and didn't generalize as well as the other models we tested. The only model we could get to generate videos for several time steps without getting blurred was the conv PUN.
2) Note that the conv PUN models uses a single convolution per step, a convolutional LSTM would require many more, making the computational and memory requirements very hard to compare. That being said, we already investigating a scalable version of PUN that uses conv LSTMs, conv LSTM + PUN compositions, etc. The experiments in the present paper focused on modular tests and understanding of the pieces. Extensive hyperparameter search should follow in future work.
1. Why does Algorithm 1 include both the delta-convolution and an STN? It seems that one would choose one or the other (Section 2) to implement a translation operator, not use both.
2. In the experiments, it seems that the Convolutional-PUN uses an LSTM, while the STN-PUN uses an RNN. If so, this gives the Convolutional-PUN an unfair advantage. Why wasn't the same base RNN architecture used for both? If both use LSTMs, which is reasonable, the text should be clear that this is the case.
3. Drawing conclusions from results that are not in the main text or even appendix (!) is very sloppy. In one instance (paragraph right before section 4.2) it is claimed that they can be obtained from the source code. In another (third paragraph of section 4.2) it is claimed that the proposed method generates high likelihood frames for longer than the baseline, but this is not shown or quantified anywhere. Can the authors comment on why this was not included even as appendix material?
Thanks for your review. We will use your suggestions to improve our work.
This paper presents a generative model of video sequence data where the frames are assumed to be generated by a static background with a 2d sprite composited onto it at each timestep.  The sprite itself is allowed to dynamically change its appearance and location within the image from frame to frame.  This paper follows the VAE (Variational Autoencoder) approach, where a recognition/inference network allows them to recover the latent state at each timestep.
Some results are presented on simple synthetic data (such as a moving rectangle on a black background or the “Moving MNIST” data.
However, the results are preliminary and I suspect that the assumptions used in the paper are far too strong too be useful in real videos.  On the Moving MNIST data, the numerical results are not competitive to state of the art numbers.
The model itself is also not particularly novel and the work currently misses some relevant citations.  The form of the forward model, for example, could be viewed as a variation on the DRAW paper by Gregor et al (ICML 2014).  Efficient Inference in Occlusion-Aware Generative Models of Images by Huang & Murphy (ICLR) is another relevant work, which used a variational auto-encoder with a spatial transformer and an RNN-like sequence model to model the appearance of multiple sprites on a background.
Finally, the exposition in this paper is short on many details and I don’t believe that the paper is reproducible from the text alone.
For example, it is not clear what the form of the recognition model is…  Low-level details (which are very important) are also not presented, such as initialization strategy.
Thanks for your review. We will use your suggestions to improve our work.
This paper presents an approach to modeling videos based on a decomposition into a background + 2d sprites with a latent hidden state. The exposition is OK, and I think the approach is sensible, but the main issue with this paper is that it is lacking experiments on non-synthetic datasets.
As such, while I find the graphics inspired questions the paper is investigating interesting, I don't think it is clear that this work introduces useful machinery for modeling more general videos.
I think this paper is more appropriate as a workshop contribution in its current form.
Thanks for your review. We will use your suggestions to improve our work.
This paper proposes a generative model of videos composed of a background and a set of 2D objects (sprites). Optimization is performed under a VAE framework.
The authors' proposal of an outer product of softmaxed vectors (resulting in a 2D map that is delta-like), composed with a convolution, is a very interesting way to achieve translation of an image with differentiable parameters. It seems to be an attractive alternative to more complicated differentiable resamplers (such as those used by STNs) when only translation is needed.
Below I have made some comments regarding parts of the text, especially the experiments, that are not clear. The experimental section in particular seems rushed, with some results only alluded to but not given, not even in the appendix.
For an extremely novel and exotic proposal, showing only synthetic experiments could be excused.
However, though there is some novelty in the method, it is disappointing that there isn't even an attempt at trying to tackle a problem with real data.
I suggest as an example aerial videos (such as those taken from drone platforms), since the planar assumption that the authors make would most probably hold in that case.
I also suggest that the authors do another pass at proof-reading the paper. There are missing references ("Fig. ??"), unfinished sentences (caption of Fig. 5), and the aforementioned issues with the experimental exposition.
Thanks for your review. We will use your suggestions to improve our work.

Meta-review:
role: PC
data: comment: The strengths and weaknesses pointed out by the reviews were:
 
 Strengths
 Interesting methodology to achieve differentiable translation of an image (R1, R2)
 Graphics-motivation is unique, inspiring (R1)
 
 Weaknesses
 Experiments not clear (R2)
 Novelty is not enough to justify synthetic-only experiments (R1,R2)
 Missing important citations (R3)
 Lacking details, concern with reproducibility (R3)
 
 The authors acknowledged the reviews with a short sentence but did not provide any feedback or revisions.
 
 The AC and PC agree with the reviewers that the paper presents interesting preliminary work which is more suitable for a workshop in its current form.
role: Author
data: value: comment: We appreciate the reviews and suggestions. We will submit the paper to the Workshop Track.

Decision: Invite to Workshop Track