Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a technique for unsupervised / self-supervised decomposition of a scene into an arrangement of layered grids of offset sprites drawn from a simultaneously learned sprite sheet. The main contributions are the network architecture and training objective.
main_review: (An earlier version of this review interpreted the paper as only having effective applications to images that were originally built from a grid of tiles where the ground truth grid size was matched to the grid sized used for analysis or a precise multiple of it. A clarification of the data preprocessing pipeline clarified the situation. Several critical comments that were founded on that observation have been removed here. Main score changed from 3 to 7.)

The abstract suggests “Our framework offers a promising approach for discovering recurring visual patterns in image collections without supervision,” and very good results are shown for Super Mario Bros. It might seem that the technique relies on a careful coherence between the grid structure it uses for analysis and a tile grid being used to generate the original data used for evaluation. However, the fact that a reasonably effective reconstruction is found for tennis while an ineffective reconstruction is found for the Moby Dick text image shows that the situation is not as simple as being specific to tile-oriented images.

The way the this approach "wear its representation 'on its sleeve'" seems to be both a strength and weakness. Reifying the recurring visual patterns as a library of sprite images makes the library easy to audit, but bakes certain features into the pattern representation that we'd normally like representations to abstract over, e.g. robustness to slight variations scale, pose, lighting, etc. Scaling the brightness of an input image to 50% wouldn't effect most other approaches, but it would seem to have a major impact the scene reconstruction ability of this approach. Changing the scale of the images 20% between train and test might also break this approach. These problems do not seem to be unrecoverable within the proposed approach -- the "transform predictor" could be changed to account for more transformations -- but the evaluations performed in the paper don't probe the limits of the translation-only design choice.

Overall, the results *do* show a clear improvement over approaches like SPACE in settings where translation is the only transformation needed. (SPACE was also evaluated on a 3d-rooms dataset where objects varied in scale and lighting.) The state of the art is being advanced in this paper.

limitations_and_societal_impact: Some limitations of this work are pointed out and visually represented in the Limitations section. The Moby Dick text example is particularly useful because it represents a case where a grid of sprites transformed by only scaling *should* be sufficient for excellent reconstruction (but the demonstrated reconstruction quality is low, presumably because the grid used for analysis does not allow for enough density in the right places, forcing the system to learn character-clusters for which it doesn't have enough capacity).

The significance of transforming sprites with only translation is not well problematized (particularly given the scale/lighting variation seen for objects in the 3d-rooms dataset used to evaluate related work).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method to disentangle sprite-based video animations into recurring visual elements in a self-supervised manner. Specifically, it decompose a frame into grids and try to reconstruct the frame by drawing elements from a dictionary of discovered sprites for each grid. By training with this reconstruction objective, the model could discover a set of disentangled sprite elements. The authors show the effectiveness of their method both through qualitative discovery results as well as quantitative segmentation metrics. With these results, the authors demonstrate that their method work much better compared to previous methods in terms of discovering these recurring visual elements.    

main_review: + The topic of self-supervised discovery of recurring visual elements is very interesting. Though has been explored before, the proposed method tackles this problem with a very explicit and intuitive grid-based modeling approach. 
+ The ides is well executed, the qualitative results are convincing to show that the method works pretty well in the video game animation domain. Also, as the quantitative results demonstrated, the proposed method has a large advantage compared to previous methods in visual discovery and disentanglement. 
+ The paper is well-written and easy to follow. 
+ The authors have promised to release the code and models. 
+ I appreciate the discussion on the limitation of the proposed approach. 

Regarding the approach
- The biggest concern I have about this work is that, though demonstrated to work very well on video game frames, it's largely unknown how well it can generalize to the domain of real videos. Even with the example shown in Fig. 14, the video is relatively simple and with clean background. It would be nice to see how it works in more challenging scenarios, e.g. evaluate segmentation quality on datasets like DAVIS. 
- I understand this paper is trying to make a step towards self-supervised visual discovery and appreciate the choice of video game frames as a controllable and easy-to-source testbed to develop prototypes. With that, I'm not sure what is the utility of the editing GUI demonstrated in Figure 9? Since for video games, we already have the full control of all disentangled elements, why do we need to develop such an application? 
- In Eq 3, I'm curious how important are those regularization terms? Does the method learn at all without them? 

Typos
- In Figure 3, the bottom-right texts should be "switches p^l_j"?

Overall, I appreciate the direction taken and the approach developed in this paper and believe it would be inspiring for the community. 

####### Post rebuttal #######
I thank the authors for their response, they are useful in further clearing some questions I had about the work. I'd like to maintain my score to accept this paper. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a generative layered model of images based on sprites — the idea is that there is a dictionary of (possibly transparent) sprites which are arranged on layers which are, together with an optionally learned background layer, composited back-to-front to form a given image.  The task is to learn the sprite dictionary and their poses and depth ordering on a given frame.  The paper shows successful results on frames from Atari/SNES era video games (which actually did use sprites) and some intriguing (but very much preliminary) one-off results on real images, training on patches sampled at random from those images.

main_review: I like several things about this paper. First, there is pretty good execution in that technically the model make makes sense and these models are never easy to get right (it’s often easy for these models to “cheat”, e.g. by putting all of their information on one layer). It also seems like a clearly novel model and also outperforms some natural competitors at least on old video game data.

On the other hand the main weakness is that the proposed model certainly does not seem terribly general — though it is interesting that the authors are able to show that it can be a generic dictionary learning approach (even if not a very good one).

Overall I’m a bit biased in favor of this paper because the model is elegant and thought provoking so I would like to see it published, with the admission that the motivation for working on such a model is not particularly compelling and the application space seems narrow.

Some more detailed comments/questions:
* Why partial convolutions? I was not familiar with this concept and looked at the reference, but it was not clear what was used as the mask for the partial convolutions in this paper and why it was necessary (is there an ablation for this?)
* In general, how does one decide on background canvas size?
* I notice that motion cues are referenced a few times in paper, but don’t seem to actually be used by model? It *does* makes a lot of sense to assume that sprites move smoothly across the background --- and perhaps have a learnable transition model through the sprites to reflect appearance change of an object.
* Other related work ( recent layered decomposition models based on similar alpha compositing and reconstruction assumptions) that I recommend adding:
 ** Omnimatte (https://omnimatte.github.io/) 
 ** Multiplane images (e.g., https://augmentedperception.github.io/deepview/)
 ** Efficient inference in occlusion-aware generative models of images (https://arxiv.org/abs/1511.06362)


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a framework for unsupervised disentanglement of images from 2d games into sprites. The framework consists of learning a sprite dictionary, predicting corresponding sprites based on the information from the frame and rendering these sprites. The framework is supervised with reconstruction loss plus some regularization.

Overall the solution makes sense and seems well-thought, however the task 2d game decomposition is not that broad.

main_review: Positive:
- The solution to the problem makes sense and seems well thought.
- The paper contains an extensive evaluation and ablation study.
- The paper tries to apply the proposed method for real world data, e.g. scanned text, president photos and tennis matches.

Negative:
- The paper considers a 2d game decomposition task in computer vision fashion, however it is questionable if this task has any practical usefulness. Indeed one could use game sprites from game source code and similarly took their arrangement from there.

- The proposed method is very game specific, each game it requires to tune many parameters: depending on the size of sprites in the game the size of the grid need to be adapted; background model need to be adapted as well, for example in the game like Mario separate background for each lever need to learned (here only one lever is considered); number of layers need to selected if there are more than 2 layers in the game.

Questions:
- In Fig 8. (Mario) The "layer 2" contains some scores and letters renderings which are practically unreadable, however when they are used in reconstruction they look almost perfect, how did this happen? Is the layer just overplayed with the background for rendering or some other post processing is applied?

- For "Transform Predictor" why only shifts are considered, is it possible to use arbitrary affine transformations?

Some minor issues:
- Figure 10 and Figure 6 should be Tables 10 and Table 6.



limitations_and_societal_impact: Limitations is discussed in the paper and in the main review. There is no negative impact.
review_title: null

Meta-review: The submission proposes a self-supervised approach to represent images as a composition of elements from a dictionary of patches onto a canvas.

Reviewers felt that the paper tackles an interesting problem, is well-executed and provides convincing results for the video game animation domain. The main concern among reviewers is generalizability to more complex domains. The authors respond by pointing out that sprite-based animation and game content is already challenging enough for current approaches. They also point to Section 3.4 of the submission, which shows preliminary results on natural images and videos, noting that a richer image formation model would be required to obtain photorealistic results.

Reviewer vWXs was concerned that the proposed approach might be relying too much on task specificities to recover good sprites, which was addressed by the authors by clearing up confusion on how the data is scaled during preprocessing. Reviewer vWXs remains concerned that the translation-only design choice makes the approach brittle to variations in scale, pose, lighting, etc. between the training and evaluation settings, but notes that the approach can recover from that weakness through a better "transform predictor".

Given that overall reviewers agree that the submission advances the state-of-the-art with a new and interesting approach, I recommend acceptance.
Decision: Accept (Poster)