MarioNette: Self-Supervised Sprite Learning
Authors: Dmitriy Smirnov
MICHAEL GHARBI
Matthew Fisher
Vitor Campagnolo Guizilini
Alexei A Efros
Justin Solomon
Venue: NeurIPS 2021 Poster (2021)
Keywords: self-supervised learning
dictionary learning
instance segmentation
2d graphics

Abstract: Artists and video game designers often construct 2D animations using libraries of sprites---textured patches of objects and characters. We propose a deep learning approach that decomposes sprite-based video animations into a disentangled representation of recurring graphic elements in a self-supervised manner. By jointly learning a dictionary of possibly transparent patches and training a network that places them onto a canvas, we deconstruct sprite-based content into a sparse, consistent, and explicit representation that can be easily used in downstream tasks, like editing or analysis. Our framework offers a promising approach for discovering recurring visual patterns in image collections without supervision. 

MarioNette: Self-Supervised Sprite Learning
Dmitriy Smirnov MIT
Michaël Gharbi Adobe Research
Matthew Fisher Adobe Research
Vitor Guizilini Toyota Research Institute
Alexei A. Efros UC Berkeley
Justin Solomon
MIT
Abstract
Artists and video game designers often construct 2D animations using libraries of sprites-texturee patches objects and characters. We propose deep learning approach that decomposes sprite-based video animations into disentangled rep- resentation of recurring graphic elements self-supervised manner. By jointly learning dictionary of possibly transparent patches and training network that places them onto a canvas, we deconstruct sprite-based based content into sparse, con- sistent, and explicit representation that be easily used in downstream tasks, like editing or analysis. Our framework offers a promising approach for discovering recurring visual patterns in image collections without supervision.
Since the early days of machine learning, the accepted unit of image synthesis has been the pixel. But while the pixel grid is a natural representation for display hardware and convolutional generators, it does not easily permit high-level reasoning and editing.
In this paper, we take inspiration from animation to consider an atomic unit that is richer and easier edit than the pixel: the sprite. In sprite-based animation, popular early technique for drawing cartoons and rendering video games, an artist draws collection of patches-a sprite sheet- consisting of texture swatches, characters various poses, static objects, and so on. Then, each frame is assembled by compositing subset of the patches onto canvas. By reusing the sprite sheet, authoring new content requires minimal effort and can even be automated procedurally.
Our goal is to invert this process, simultaneously tackling unsupervised instance segmentation and dictionary learning. Given image dataset, e.g., frames from sprite-based video game, train a model that jointly learns a 2D sprite dictionary, capturing recurring visual elements in an image collection, and explains each input frame combination of these potentially transparent sprites. Whereas standard CNN-based generators hide their feature representation in their intermediate layers, our model wears its representation "on its sleeve": by explicitly compositing sprites from its learnt dictionary onto background canvas, rather than synthesizing pixels from hidden neural features, it provides a rexdily-interpreeabble visual representation.
Our contributions include the following:
We describe grid-based anchor system along with learned dictionary of textured patches (with transparency) to extract a sprite-based image representation.
We propose method learn the patch dictionary and the grid-based representation jointly, in differentiable, end-to-end fashion.
We compare past work learned disentangled graphics representations for video games. We show how our method offers promising avenues for further work towards identifying visual patterns in more complex data such as natural images and video.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Related Work
Decomposing visual content into semantically meaningful parts for analysis, synthesis, and editing is a long-standing problem. We review the most closely related work.
Layered decompositions. Wang and Adelson [54] decompose videos into layers undergoing
temporally-varye warps for compression. Similarly, Flexible Sprites (24) and Kannan et al. [27] represent videos with full-canvas semi-transparent layers to facilitate editing. Like Flexible Sprites, we adopt translation-only motion but restrict transformations to small neighborhoods around anchors, making inference tractable with many (2 100) sprites. Other methods decompose videos with moving subjects, such as humans, into independent layers, enabling matting [4]] and retiming of individual actions [40]; unlike sprite-based techniques, motion and appearance are not disentangled. Sbai et al. 48 use layered representation as inductive bias in a GAN with solid colored layers. Automatic decompositions into "soft layers" according to texture, color, semantic features have been used in image editing 2]. Gandelsman et al. [12] use deep image priors [52] to separate images into layer pairs. Huang and Murphy [19] introduce recurrent architecture to output multiple layers sequentially. Reddy et al. [46] discover patterns in images via differentiable compositing.
Interpretable generators for neural synthesis. Neural networks improve the fidelity and realism of generative models [1] 28 but limit control and interpretability [5] 6 8 16] Several works explore
interpretability using differentiable domain-specific functions. Hu et al. [8] Li al. [3I] constrain the generator to sets of parametric image operators. Mildenhall et al. [44] use ray-marching prior and rendering model encode a radiance field for novel view synthesis. Neural textures (51) replace RGB textures on 3D meshes with nigh-dimensional features. Rendering under new views enables ciessconsistent editing. Lin et al. 33 use spatial transformers in their generator to obtain geometric transformations, We synthesize frames by compositing 2D sprites undergoing rigid motions, enabling direct interpretation and control over appearance and motion.
Object-centric representations. Our learned sprites reveal, segment, and track object instances. Similarly, Slot Attention (37 extracts object-centric compositional video representations. However, our sprites are interpretable- motion and appearance are direct outputs-aan and our model scales to more objects per scene. SCALOR (23] handles up to 100 instances but does not produce common dictionary or handle diverse sprites. While SPACE (34) decomposes images into object layers, it tends embed sprites in the background, providing no control. Our method achieves higher IoU of recurring sprite patterns (see $3.I). Stampnet [53] discovers and localizes objects but focuses on simpler, synthetic datasets. MONet (7] decomposes images into multiple object regions using attention. Earlier attention mechanisms leverage pattern recurrence (9] 30 and motion cues [ to identify individual objects. Recent works use parametric primitives as image building blocks 32
Applying our sprite decompositions to video games, we can learn about dynamics and gameplay, benefiting downstream agents 17 26 and aiding content-authoring for research and game devel- opment, as in Procedural Content Generation [50). GameGAN [29 synthesizes new frames from controller input. They split rendering into static and dynamic components but render full frames, without factorization into parts. Their generator is difficult to interpret: appearance and dynamics are entangled within its parameters.
Compression. Appearance consistency and motion compensation are central to video compression [4] We model videos compositions of moving sprites, factoring redundancy in the input. This draws inspiration from works like DjVu 15] and Digipaper (20), which compress scanned documents by separating them into background layer and foreground text. Image epitomes (25) summarize and compress image shape and appearance into a miniature texture. Our sprite dictionary fills a similar
38 42
role, providing superior editing control.
Method
We start with an input sequence of RGB frames {II,..., In) with resolution w h. Our goal is to decompose each frame 1; E 3x into set of possibly overlapping sprites, organized into é depth layers, selected from a finite-size dictionary. The dictionary is collection of trainable latent codes {21,..., } that are decoded into RGBA sprites using neural network generator 2$2.

---

TRANSFORM PREDICTOR
extract local
T
TT
E ENCODER
sprite
tmunsform sprites
composite
imput
aneboos switches B
pes-anchoo sprines
reconstruction
I-c
sprite codes
SPRITE GENERATOR
background
sprites dictionary
Figure 1: Overview. We jointly learn a sprite dictionary, represented as a set of latent codes decoded by generator, well encoder network that embeds frame into grid of latent codes, or anchors. Comparing anchor embeddings to dictionary codes lets us assign a sprite to each grid cell. Our encoder also outputs binary switch per anchor turn sprites and off. After compositing, obtain a reconstruction of the input. Our self--uuer training optimizes a reconstruction loss.
Our training pipeline is illustrated in Figure 1 We first process each input frame with a convolutional encoder to produce grids of feature vectors, one grid per depth layer ($2.2). The grids are lower resolution than the input frame, with downsampling factor proportional to the sprite size. We call the center of each grid cell an anchor. We compare each anchor's feature vector against the dictionary's 's latent codes, using a softmax scoring function, to select the best matching sprite per anchor ($..). Using our sprite generator, we decode each anchor's matching sprite. This gives us grid of sprites for each of the & layers. To factorize image patterns that may not align with our anchor grid, allow sprites to move in small neighborhood around anchors ($24)) We composite the layers from back to front onto the output canvas to obtain our final reconstruction ($2.5). Optionally, the background modeled as special learnable sprite that covers the entire canvas.
We train the dictionary latent codes, frame encoder, and sprite generator jointly on all frames, compar- ing our reconstruction the input ($2.)). This self-supervised procedure yields representation that is sparse, compact, interpretable, and well-suited for downstream editing and learning applications.
2.1 Dictionary and sprite generator
The central component of our representation is global dictionary of m textured patches sprites D = [PI..... Pm.) where each P. E I2 4xxx is an RGBA patch. Our sprites have an alpha channel, which allows them to be partially transparent, with possibly irregular (i.e., non-square) boundaries. This is useful for representing animations with multiple depth layers and also allows to learn sprites smaller than their maximal resolution, if necessary, by setting alpha dictionary is shared among all frames; we reconstruct frames using only sprites from the dictionary.
around the boundary. The
set of trainable latent codes (zi,...: ²m)) with 2; E Rd. We decode these codes into RGBA sprites using a fully- connected sprite generator P: G (zz). This latent representation allows us to define similarity metric over the latent space, which we use to pair anchors with dictionary sprites to best reconstruct the input frame (..3). At test time, can forego the sprite generator and edit the RGBA sprites directly. Unless otherwise specified, we set latent dimension to d = 128 and patch size to & = 32.
Instead of optimizing for RGBA pixel values directly, we represent the dictionary
We randomly initialize the latent codes from the standard normal distribution. Our sprite generator first applies zero-mean unit-variance normalization- Layer Normalization [3] without an affine transformation to each latent code 2; individually. followed by one fully-connected hidden layer with 8d features, Group Normalization [55). and ReLU activation. We obtain the final sprite using a fully- connected layer with sigmoid activation to keep RGBA values in [0, 1]. Latent code normalization crucial stabilize training and keep the latent space in a compact subspace as the optimization progresses. See 3.3 for ablation study of this and other components.

---

2.2 Layered frame decomposition using sprite anchors
We seek a decomposition that best explains each input frame using dictionary sprites. We exploit translation invariance and locality in our representation: our sprites are "attached" to regular grid of reference points, or anchors, inspired by [3] 47 Each anchor has at most one sprite; we call inactive if it has none.
We give the sprites freedom of motion around their anchors to factorize structures that may not be aligned with the anchor grid. This local- or, Eulerian-viewport
makes inference tractable and avoids the pitfalls of tracking the global motion of all the sprites across the canvas (a La- grangian viewpoint). To enable multiple layers with sprite occlusions, we output & >> 1 anchor grids for each frame (f in our experiments). Figure 2 illustrates our layered anchor grids and local sprite transformations.
place spriliss al detive anchion
apply local transoous
apply sandam
cecedening
composil:
distiorary
Figure Layered sprite decomposition with local anchors. We assign at most one sprite per anchor and predict local transformation of each placed sprite around its anchor. To allow for occlusions be- tween sprites, we use multiple sprite layers, which we compose back to front to obtain the final image.
We use a convolutional encoder & to map the wx h RGB frame to grids of anchors, with resolution 2w: x 2h: Each anchor in layer / is represented by feature vector af E IRd char- acterizing local image appearance around the anchor and an active/inactive switch probabil- ity p. E [0, 1]. Our frame encoder contains (k) downsampling blocks, which use partial convolutions (36] with kernel size 3 and stride 2 (for downsampling), Group Normaliza- tion, and Leaky ReLU. It produces a tensor of intermediate features for each layer, which are normalized with LayerNorm. From these, we obtain the anchor switches with an MLP with one hidden layer of size d followed by Group Normalization and Leaky ReLU. We get anchor features using a linear projection followed by LayerNorm. The encoder architecture illustrated Figure
projection
layer anchors a
switches
layer anchors
input frame
GroupNomm
switches p
LeakyReLU
LayerNorm
Figure 3: Encoder architecture.
2.3 Per-anchor sprite selection
Once we have the layered anchor grids for the input frame, we need to assign sprites to the active anchors. We do this by scoring every dictionary element against each anchor j at layer using softmax over dot products between dictionary codes and anchor features:
exp(cl (v/aa)) 2 exp (c) =x/vaa)"
Recall that both the anchor features and dictionary latent codes are individually normalized using Layer Normalization operator. Restricting both latent spaces to a compact subspace helps stabilize the optimization and avoid getting stuck in local optima. During training, each anchor's 's sprite is weighted combination of the dictionary elements, masked by the anchor's active probability:
772
=
Σ S P:.
This soft patch selection allows gradients propagate to both dictionary and anchor features during training. Except for natural image and video datasets, at test time, we use hard selections, i.e., for each anchor, we pick the sprite (S) == P.) with highest score sig and binarize the switches p\. E 10,1).
2.4 Local sprite transformations
In real animations, sprites rarely perfectly align with our regular anchor grid, so, to avoid learning several copies of the same sprites (e.g., all sub-grid translations of a given image pattern), we allow sprites to move around their anchors. In our implementation, we only allow 2D translations of up to 2/2 the sprite size each side of the anchor, i.c., T (2/) 8/) E 1/22 8/2]2.

---

Slot Attentionn
SPACE
Reconstration
Foreground
Rack gorkggodd
Reconstaation
Layor
Spotto Diccionary
While all three methods obtain good reconstructions, SPACE only recognizes a few sprites, and Slot Attention does not yield meaningful decomposition. We decompose the entire foreground and learn dictionary.
Figure 4: Comparison to SPACE 44
and Slot Attention 37
We use a convolutional network to predict the translation offsets from the anchor's sprite and a crop of the input frame centered around the anchor, with identical spatial dimensions. This network follows the architecture of & followed by an MLP with single hidden layer of size d, Group Normalization, and Leaky ReLU. Specifically, we concatenate the image crop and the anchor's sprite S along the channel dimension and pass this tensor through this network to obtain the æ' and v/ offsets. An output layer projects two dimensions (horziontal and vertical shift) and applies tanh to restrict the range. We apply the shifts to the patches using a spatial transformer [21].
2.5 Compositing and reconstruction
Each anchor in layered representation is equipped with sprite S) and transformation T.. For each layer I, we transform the sprites in their anchor's local coordinate system and render them onto the layer's canvas, initialized as fully transparent. Because of the local transformation. neighboring sprites within a layer may overlap. When this happens, we randomly choose an ordering. in Figure This random permutation encourages our model to either avoid overlapping sprites within the same layer or make the sprite colors agree in the overlap region, since these are the only rendering regardless of the random z-ordering. Note that sprites
options that yield the distinct layers are not shuffled. The shuffling prevents the network from abusing the compositing to cover patches with others from the same layer.
We optionally learn a background texture to capture elements that cannot explained using sprites, This can be thought of as special patch of resolution greater than that of single frame. For each frame, we learn a (discrete) position offset in the background from which to crop. We represent these offsets as discrete pixel shifts using softmax classification (independently for each spatial dimension). We found this encoding better behaved than using a continuous offset with a spatial transformer--te the discrete encoding allows the gradient signal to propagate all shifts rather than the weak local gradient from bilinear interpolation (see for an ablation). We combine the background shows learned background.
and sprite layers via standard alpha compositing (5]. Figure X
In some experiments, we use a simpler background model: fixed solid color, determined by analyzing the data before training. In this variant, we sample 100 random frames, cluster the pixel values into 5 clusters using k-means, and choose the largest cluster center as background color.
2.6 Training procedure
Our pipeline is fully differentiable. We train the latent codes dictionary, sprite generator, frame encoder, transformation predictor, and background layer jointly, minimizing Loz distance between our reconstructions and ground truth frames. We also employ two regularizers: Beta distribution prior
5

---

Reconstruction
Foreground
Segmentation
Input
Ground Truth
Segmentation
Ours
Reconstruction
Layer
Layer 2
Segmentation
Sprite Dictionary
Figure 5:
Qualitative comparison to SPACE [34] on the synthetic game dataset. We show ground
truth sprite segmentations as well those obtained from the learned SPACE foreground and from our learned sprites. While SPACE only learns several of the sprites, we reconstruct the entire foreground using our dictionary.
on switches and dictionary element scores favors values close 0 and an L1 loss on switches favors a sparser solution without superfluous patches. Our final loss function for single input is:
2 2h
C(-) wh
10-1 10-11111 1twh
Beta
2)ta(2,2)(s{g)) +eet))))BBBe222t Beta ())))) + Aspuree
(3)
where O is the result of compositing the background and sprite layers; we optimize {s }. {p/) J. and O. We set Ayparse 0.005 and train for 200,000 steps (~20 hours) with ABeta = 0.002 and finetune for 10,000 steps with ABeta 0.1. For natural images and video, we set λnea = O. We use the Adam W (39] optimizer on GeForce GTX 1080 GPU, with batch size and learning rate 0.0001, except for the background module (learning rate 0.001 when used).
3 Experimental Results
We evaluate our self-supervised decomposition several real (non-synthetic) datasets, compare to related work, and conduct an ablation study. In figures, we use a checkerboard show transparency. Dictionary order is determined by sorting along 1-dimensional t-SNE embedding of the sprite latent codes. We find this ordering tends to group semantically similar sprites, making the dictionary easier to interpret and manipulate. While models are trained with dictionary of 150 patches, not all patches end up being used; we only show the used patches.
3.1 Comparisons
While to our knowledge no prior works target differentiable unsupervised sprite-based reconstruction, we compare to two state-of-the-arr methods that obtain similarly disentangled representations.
In Figure we compare to SPACE 34 and Slot Attention 37 The former decomposes scene into a foreground layer consisting of several objects as well as background, segmented into three layers. The latter deconstructs a scene into discrete "slots." We train both methods to convergence using their default parameters. While both reconstruct the input frames faithfully, SPACE only recognizes few sprites in its foreground layer, and Slot Attention does not provide semantically meaningful decomposition. In contrast, not only does our method model the entire scene using learned sprites, but also it factors out the sprites to form consistent, sparse dictionary shared for the entire sequence.
Additionally, we evaluate on monthetically-generate sprite-based game from [T], which is made of sprites on a solid background. We compare quantitatively to SPACE in Figure 6 and show qualitative results in Fi Since we have ground truth segmentation of each scene into sprites, we compute a matching between learned dictionary patches and sprites by associating each dictionary patch with the sprite that most frequently overlaps across the dataset. We visualize dictionary patches next to their respective sprites. We also use this labeling to compute segmentation metrics. In particular, we report mean IoU in the multiclass case (where each sprite distinct class) as well is the binary case (foreground/baccgroumn)) Because SPACE does not learn common dictionary, are unable to

---

obtain a labeling for its foreground elements and, consequently, cannot evaluate its multiclass metric. For the binary metric, we obtain a significantly higher value, since SPACE defers many sprites the background, whereas our method learns the sprites as dictionary elements.
Inpur
Flow Based
Ours
To show that model learns more than simple motion features,
we also compare to two conventional (non-learning) baselines. In Figure aa), we compare segmentation of frame obtained by clustering optical flow directions using k-means (inspired by Liu al. B35)) to one generated using our learned decomposition. The flow-based approach is unable to capture many of the details in the frame. In (b), we show the normalized dictionary obtained using an online dictionary learning method 43]. Because this method does not have the inductive biases of our model, the resulting dictionary is not easily interpretable or editable.
Go Segmentation
(b) Online Dictionary Learning
Figure 7: Comparison to con-
3.2 Sprite-based game deconstruction
ventional baselines.
We train on Fighting Hero (one level, 5,330 frames). Nintendo Super Mario Bros. (one level. 2,220 frames), and ATARI Space Invaders (5,000 frames). We use patch size k 32 for Mario and Fighting Hero and k 16 for Space Invaders. For Fighting Hero, we learn a background, as described 2.5
The sprites, background, and example frame reconstructions are shown in Figure Our model successfully disentangles foreground from background and recovers a reasonable sprite sheet for each game. Having reverse-engineered the games, we can use the decom- position for applications like editing. In Figure 9 we demonstrate a GUI that allows the user to move sprites around the screen.
Soating franne
adack Icarnad apoit
monid lcaniol spilc
Figure 9: Editing GUI.
3.3 Ablation Study
Model
PSNR
We show an ablation study on the Mario data. We train our full model, one with smaller 16 16 patches, another with larger 64 x 64 patches, a model with a smaller dictionary (25
Smaller patches Full
28.85 + 0.95
28.04 + 0.72
No LayerNorm
26.05 1: 0.45
elements), a model without LayerNorm, and one where use a straight-through estimator [22) to learn discrete switches p in lieu of Beta regularization. We train each model with five random seeds and report the reconstruction PSNR means and standard deviations Figure [0] This experiment verifies the importance of LayerNorm in our architecture and shows that the trick is ineffective in our setting. Though the smaller patches model achieves slightly higher mean PSNR than our full model, more of the sprites are split across dictionary patches (Figure I]. illustrating how the patch size choice sets an inductive bias for our decomposition.
Smaller dictionary
23.80 +: 1.38
Larger putches
23.63
1.05
Straight shrough switches
22.15 1: 0.25
Figure 10: Mario data across five random seeds.
Ablation study on
Figure 11: Mario dictionary with 16x 16 patches.
We also justify our choice for learning background shifts via classification ($2.5) rather than regression, i.c., using spatial transformers. Figure [ shows the background learned using a spatial transformer. In contrast to our full model (Figure 8 the original background is not discovered, and most of the canvas unused. We suspect that this due to lack of gradient signal from background pixels that do not get rendered at each training step.
Figure 12: Background learned with spatial transformer.
Method
Reconstruction PSNR
Mean IoU (multiiolss)
Mean IoU (binary)
SPACE Ours
31.9
0.0361
38.54
0.6497
0.7352
Figure 6: Quantitative comparison SPACE 34)
We report PSNR to evaluate overall reconstruction
quality as well as mean IoU for multiclass and binary foreground/backgromm segmentation problems. Our method recognizes significantly more sprites than SPACE, resulting higher mean loU.
7

---

Input
Reconstruction
Layer
Layer
Leamed Background
Sprite Dictionary
TBB
mnsis
Input
Reconstrnition
Layer
Layer
Sprite Dictionary
Input
Reconstruction
Tayer
Tayer
Sprite Dictionary
Sprite-based game decompositions. Our self-supervised technique recovers compact dictionary of semantically meaningful sprites representing characters (or their body parts) and props. The first example shows our learned background texture; the others use solid color as background.
Figure 8:
3.4 Future Directions and Limitations
While our method designed with sprite-based animation in mind, it can generalize to natu- ral images and videos. An exciting direction for future work is to incorporate more expres- sive transformations so as to discover recurring content in generic videos. Here, obtain pre- liminary results using our approach and achieve interesting decompositions cations to our sprite-based model.
Min
MiD
Input
Reconstrnction
Sprite Dictionary
Figure 13: We factorize the recurring elements of single photograph into a compact dictionary.
without modifi-
show results a tennis video (4,000 frames). The model learns parts of the player's ' body (head, limbs, shirt, etc.) as sprites and captures most of the tennis court in the learned background. By simply selecting the player sprites in the dictionary, we segment the entire video clip.
In Figure 14|
Our model can also discover recurring patterns in a single natural image. We train random crops of 768 x 512 photograph from the 2013 US Presidential Inauguration which contains many repeating elements such stairs, columns, and people. With a dictionary 39 32 32 sprites (39.936 pixels). we recover much of the detail of the original 393,21 pixels.
AP Photo/Cliff Owen

---

Input
Reconstraction
Layer
Layer 2
Segmentation
Leamndd Background
Spritc Dictionary
Labeled Sprite Distionary
Figure 14: Segmentation of natural videos. Despite its simplistic motion and appearance model, our approach can be applied to real-world videos. By selecting the few sprites corresponding to the tennis player, we can quickly obtain a segmentation of the full vidco sequence.
sut down on an old wooden settle, carv bench on the Battery. At one end a rumii still further adorning with his jaik- and diligently working away at the space be He was trying his hand
traat down on an old woomEn acttle eury heneh the Ihtterr it end rmmi at forther adorning it with his qneiu hnile and dihgently gorlingg
at the ree he tlasaa trying land di a hin undor t: di t mahe minet headwav. thonghh
ship under didn't make much headway. thonght.
Input
Reconstruction
Sprite Dictionary
Figure 15:
Reconstruction of scanned text excerpt, illustrating some limitations of our method. Because letters are densely packed within the text and lack motion cues, our model learns sprites comprising more than single glyph and suffers from high reconstruction error.
We demonstrate further limitations of our approach by applying it to automatic font discovery. We train on random 128 x 128 crops of six scanned pages of Moby Dick, each of approximately 500x800 resolution. Figure 15 shows input text excerpt, our reconstruction, and the learned dictionary.
This dataset differs significantly from our other testing datasets. Each input frame consists of many densely packed sprites (~~ 100 glyphs in each 128 128 crop), and many individual glyphs consist of smaller repeating elements. We hypothesize that because of these issues, combined with lack of motion cues between frames, with multiple glyphs and others with just partial glyphs. Incorporating priors tailored to regularly structured and dense data like text is direction for future research.
do achieve perfect reconstruction, learning certain sprites
4 Conclusion
We present self-supervised method to jointly learn patch dictionary and frame encoder from a video, where the encoder explains frames as compositions dictionary elements, anchored on regular grid. By generating layers of alpha-masked sprites and predicting per-sprite local trans- formation, we recover fine-scale motion and achieve high-quality reconstructions with semantically meaningful, well-separated sprites. Applied to content with significant recurrence, approach recovers structurally significant patterns.
Understanding recurring patterns and their relationships is central to machine learning. Learning to intelligently in video games or the physical world requires breaking experiences down into elements between which knowledge can be transferred effectively. Our sprite- decomposition provides an intuitive basis for this purpose. In this work, we focus on a simplified video domain. In the future, we would like to expand the range of deformations applied to the learned dictionary elements, such as appearance or shape changes. Our work opens significant avenues for future research to explore recurrences and object relationships in more complex domains.

---

Acknowledgements
The MIT Geometric Data Processing group acknowledges the generous support of Army Research Office grants W911NF210016 and 1991N11110299 of Air Force Office of Scientific Research award FA9550-19- 19-00-199---11,, of National Science Foundation grants I18-1838071 and CHS-1955697, from the CSAIL Systems that Learn program, from the MIT-IBM Watson AI Laboratory, from the Moyota-CSAIL Joint Research Center, from a gift from Adobe Systems, from an MIT.naao Immersion Lab/NCSOFT Gaming Program seed grant, and from the Skoltech-MIT Next Generation Program. This work was also supported by the National Science Foundation Graduate Research Fellowship under Grant No. 1122774.
References
[1] Y. Aksoy, T. O. Aydin, A. Smolié, and M. Pollefeys. Unmixing-based soft color segmentation for image manipulation. ACM Transactions on Graphics (TOG), 36(2):1-19, 2017.
[2] Y. Aksoy, T.-H. Oh, S. Paris, M. Pollefeys, and W. Matusik. Semantic soft segmentation. ACM
Transactions Graphics (TOG), 373, 2018.
[3] J. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv: 167.v:660, 2016.
[4] Bachu and K. M. Chari. A review motion estimation in video compression. In International Conference on Signal Processing and Communication Engineering Systems, pages 250-256. IEEE, 2015.
[5] D. Bau, J.-Y. Zhu, H. Strobelt, B. Zhou, J. B. Tenenbaum, W. T. Freeman, and A. Torralba. GAN dissection: Visualizing and understanding generative adversarial networks. arXiv:1811.1059) 2018.
[6] D. Bau, J.-Y. Zhu, J. Wulff, W. Peebles, H. Strobelt, B. Zhou, and A. Torralba. Seeing what GAN cannot generate. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 4502-4511. 2019.
[7] C. P. Burgess, L. Matthey, N. Watters, R. Kabra, I. Higgins, M. Botvinick, and A. Lerchner. MONet:
Unsupervised scene decomposition and representation. arXiv:1901.1 1XXv:19011.11900 2019.
[8] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. InfoGAN: In- terpretable representation learning by information maximizing generative adversarial nets. arXiv:1606.044 00000....67..000 2016.
[9] E. Crawford and J. Pincau. Exploiting spatial invariance for scalable unsupervised object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 3684-3692, 2020.
[10] R. Dubey, P. Agrawal, D. Pathak, T. L. Griffiths, and A. A. Efros. Investigating human priors for playing video games. In ICML, 2018.
[11] S. Eslami, N. Heess, T. Weber, Y. Tassa, D. Szepesvari, K. Kavukcuoglu, and G. E. Hin- ton. Attend, infer, repeat: Fast scene understanding with generative models. arXiv preprint 1XXv:1603.08575 2016.
[12] Y. Gandelsman, A. Shocher, and M. Irani. "Double-DIP": Unsupervised image decomposition Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 11026-1 10026-11035, 2019.
via coupled seep-image-pprorss
[13] R. Girshick. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, pages 1440-1448, 2015.
[14] I. J. Goodfellow, Pouget-Abadit M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks.
2014.
[15] P. Haffner, L. Bottou, P. G. Howard, and Y. LeCun. DjVu: Analyzing and compressing scanned documents for internet distribution. Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR 99 (Cat. No. PR00318) 18), pages 625-628. IEEE, 1999.
[16] E. Härkönen, A. Hertzmann, J. Lehtinen, and S. Paris. Ganspace: Discovering interpretable GAN controls. arXiv:2004.02546, 2020.
10

---

[17] Z. He, J. D. Liu, H. He, and D. Barber. Tracking by animation: Unsupervised learning of multi-object attentive trackers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 2818-1277, 2019.
[18] Y. Hu, H. Hc, C. Xu, B. Wang. and S. Lin. Exposure: A white-box photo post-processing framework. ACM Transactions on Graphics (TOG), 3712) 2018.
[19] J. Huang and K. Murphy. Efficient inference in occlusion-awar generative models of images. ICLR Workshops, 2016.
[20] D. Huttenlocher, P. Felzenszwalb, and W. Rucklidge. Digipaper: A versatile color document image representation. In Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348), volume 1, pages 213-223. IEEE, 1999.
[21] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial transformer networks. arXiv:1506.0202. 02025. 2015.
[22] E. Jang. S. Gu, and B. Poole. Categorical reparameterization with Gumbel-softmax. arXiv preprint arXiv: 2016.
[23] Jiang, S. Janghorbani, G. De Melo, and S. Ahn. Scalor: Generative world models with scalable object representations, arXiv preprint arXiv:1910.02380 2019.
[24] N. Jojic and B. J. Frey. Learning flexible sprites in video layers. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pages I-I. IEEE, 2001.
[25] N. Jojic, B. J. Frey, and A. Kannan. Epitomic analysis of appearance and shape. In Proceedings Ninth IEEE International Conference on Computer Vision, pages 3441 vol.1, 2003.
[26] N. Justesen, P. Bontrager, J. Togelius, and S. Risi. Deep learning for video game playing. IEEE Transactions on Games, 12(1):1 2019.
[27] A. Kannan, N. Jojic, and B. J. Frey. Generative model for layers of appearance and deformation. In AISTATS, volume 2005, page 1. Citeseer, 2005.
[28] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 81 11100-811, 2020.
[29] S. W. Kim, Y. Zhou, J. Philion, A. Torralba, and S. Fidler. Learning to simulate dynamic envi- ronments with GameGAN. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2020.
[30] A. R. Kosiorek, H. Kim, I. Posner, and Y. W. Teh. Sequential attend, infer, repeat: Generative modelling of moving objects. arXiv preprint arXiv: 01744, 2018
[31] T.-M. Li, M. Gharbi, A. Adams, F. Durand, and J. Ragan-Kelley. Differentiable programming for image processing and deep learning in halide. ACM Transactions Graphics (TOG), (4):1-13, 2018
[32] T.-M. Li, M. Lukáë, M. Gharbi, and J. Ragan-Kelley. Differentiable vector graphics rasterization for editing and learning. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.
[33] C.-H. Lin, E. Yumer, O. Wang, E. Shechtman, and S. Lucey. ST-GAN: Spatial transformer generative adversarial networks for image compositing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9455-9464, 2018
[34] Z. Lin, Y.-F. Wu, S. V. Peri, W. Sun, G. Singh, F. Deng, J. Jiang, and S. Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020.
[35] C. Liu, A. Torralba, W. T. Freeman, F. Durand, and E. H. Adelson. Motion magnification. ACM transactions on Graphics (TOG), 24(3):519-55 526, 2005.
[36] G. Liu, K. J. Shih, T.-C. Wang, F. A. Reda, K. Sapra, Z. Yu, A. Tao, and B. Catanzaro. Partial convolution based padding. In arXiv preprint arXiv:1811.111 181.11188, 2018.
[37] Locatello, Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, Uszkoreit, A. Doso- vitskiy, and T. Kipf. Object-centric learning with slot attention. arXiv:2006.1 15055, 2020.
11

---

[38] S. Lombardo, J. HAN, C. Schroers, and S. Mandt. Deep generative video compres- sion. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc. E. Fox, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL
pdf
[39] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[40] E. Lu, F. Cole, T. Dekel, A. Zisserman, D. Salesin, W. T. Freeman, and M. Rubinstein. Layered neural rendering for retiming people in video. ACM Trans. Graph., 2020.
[41] E. Lu, Cole, T. Dekel, A. Zisserman, W. T. Freeman, and M. Rubinstein. Omnimatte: Associating objects and their effects in video. In CVPR, 2021.
[42] G. Lu. W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao. Dve: An end-to-end deep video compression framework. arXiv: 1812v.881.0000 112000101, 2018.
[43] J. Mairal, F. Bach, J. Ponce, and G. Sapiro. Online learning for matrix factorization and sparse coding. Journal of Machine Learning Research, 1 11(1), 2010.
[44] B. Mildenhall, P. Srinivasan, M. Tancik, 'T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, pages 405-421. Springer, 2020.
[45] Porter and T. Duff. Compositing digital images. In Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques, pages 253-259, 1984.
[46] P. Reddy, P. Guerrero, M. Fisher, W. Li, and N. J. Mitra. Discovering pattern structure using differentiable compositing. ACM Trans. Graph. (Proceedings of SIGGRAPH Asia 2020), (9()): 262:1-262:15 15, 2020. doi: https://doi.org: .rgg/00.g000/ 100
[47] Redmon, S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 779-788, 2016.
[48] O. Shai, Couprie, and M. Aubry. Unsupervised image decomposition in vector layers. In IEEE International Conference on Image Processing, Abu Dhabi, UAE, 2020. IEEE.
[49] D. Smirnov, M. Fisher, V. G. Kim, R. Zhang, and J. Solomon. Deep parametric shape predictions using distance fields. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 561-570, 2020.
[50] A. Summerville, Snodgrass, M. Guzdial, C. Holmgård, A. K. Hoover, A. Isaksen, A. Nealen, and J. Togelius. Procedural content generation via machine learning (PCGML). IEEE Transac- tions on Cames, 10(3):257-277, 2018
[51] Thies, M. Zollhöfer, and M. Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions Graphics (TOG), 38(4): 1-12, 2019.
[52] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9446-9454, 2018.
[53] J. Visser, A. Corbetta, V. Menkovski, and F. Toschi. StampNet: unsupervised multi-class object discovery. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2951-2955. IEEE, 2019.
[54] J. Y. A. Wang and E. H. Adelson. Representing moving images with layers. IEEE Transactions on Image Processing, 3(5):625-638, 1994.
[55] Y. Wu and K. He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pages 3-19, 2018
12


Reviews:
[{"review_id": "U3_YcyqpeKH", "reviewer": "Reviewer_JHYp", "summary": "The authors propose a generative layered model of images based on sprites — the idea is that there is a dictionary of (possibly transparent) sprites which are arranged on layers which are, together with an optionally learned background layer, composited back-to-front to form a given image.  The task is to learn the sprite dictionary and their poses and depth ordering on a given frame.  The paper shows successful results on frames from Atari/SNES era video games (which actually did use sprites) and some intriguing (but very much preliminary) one-off results on real images, training on patches sampled at random from those images.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like several things about this paper. First, there is pretty good execution in that technically the model make makes sense and these models are never easy to get right (it’s often easy for these models to “cheat”, e.g. by putting all of their information on one layer). It also seems like a clearly novel model and also outperforms some natural competitors at least on old video game data.\n\nOn the other hand the main weakness is that the proposed model certainly does not seem terribly general — though it is interesting that the authors are able to show that it can be a generic dictionary learning approach (even if not a very good one).\n\nOverall I’m a bit biased in favor of this paper because the model is elegant and thought provoking so I would like to see it published, with the admission that the motivation for working on such a model is not particularly compelling and the application space seems narrow.\n\nSome more detailed comments/questions:\n* Why partial convolutions? I was not familiar with this concept and looked at the reference, but it was not clear what was used as the mask for the partial convolutions in this paper and why it was necessary (is there an ablation for this?)\n* In general, how does one decide on background canvas size?\n* I notice that motion cues are referenced a few times in paper, but don’t seem to actually be used by model? It *does* makes a lot of sense to assume that sprites move smoothly across the background --- and perhaps have a learnable transition model through the sprites to reflect appearance change of an object.\n* Other related work ( recent layered decomposition models based on similar alpha compositing and reconstruction assumptions) that I recommend adding:\n ** Omnimatte (https://omnimatte.github.io/) \n ** Multiplane images (e.g., https://augmentedperception.github.io/deepview/)\n ** Efficient inference in occlusion-aware generative models of images (https://arxiv.org/abs/1511.06362)\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "TYgAv-Wqgcb", "reviewer": "Reviewer_vWXs", "summary": "The paper introduces a technique for unsupervised / self-supervised decomposition of a scene into an arrangement of layered grids of offset sprites drawn from a simultaneously learned sprite sheet. The main contributions are the network architecture and training objective.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(An earlier version of this review interpreted the paper as only having effective applications to images that were originally built from a grid of tiles where the ground truth grid size was matched to the grid sized used for analysis or a precise multiple of it. A clarification of the data preprocessing pipeline clarified the situation. Several critical comments that were founded on that observation have been removed here. Main score changed from 3 to 7.)\n\nThe abstract suggests “Our framework offers a promising approach for discovering recurring visual patterns in image collections without supervision,” and very good results are shown for Super Mario Bros. It might seem that the technique relies on a careful coherence between the grid structure it uses for analysis and a tile grid being used to generate the original data used for evaluation. However, the fact that a reasonably effective reconstruction is found for tennis while an ineffective reconstruction is found for the Moby Dick text image shows that the situation is not as simple as being specific to tile-oriented images.\n\nThe way the this approach \"wear its representation 'on its sleeve'\" seems to be both a strength and weakness. Reifying the recurring visual patterns as a library of sprite images makes the library easy to audit, but bakes certain features into the pattern representation that we'd normally like representations to abstract over, e.g. robustness to slight variations scale, pose, lighting, etc. Scaling the brightness of an input image to 50% wouldn't effect most other approaches, but it would seem to have a major impact the scene reconstruction ability of this approach. Changing the scale of the images 20% between train and test might also break this approach. These problems do not seem to be unrecoverable within the proposed approach -- the \"transform predictor\" could be changed to account for more transformations -- but the evaluations performed in the paper don't probe the limits of the translation-only design choice.\n\nOverall, the results *do* show a clear improvement over approaches like SPACE in settings where translation is the only transformation needed. (SPACE was also evaluated on a 3d-rooms dataset where objects varied in scale and lighting.) The state of the art is being advanced in this paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations of this work are pointed out and visually represented in the Limitations section. The Moby Dick text example is particularly useful because it represents a case where a grid of sprites transformed by only scaling *should* be sufficient for excellent reconstruction (but the demonstrated reconstruction quality is low, presumably because the grid used for analysis does not allow for enough density in the right places, forcing the system to learn character-clusters for which it doesn't have enough capacity).\n\nThe significance of transforming sprites with only translation is not well problematized (particularly given the scale/lighting variation seen for objects in the 3d-rooms dataset used to evaluate related work).", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "8fsTk_Z4rv3", "reviewer": "Reviewer_dp3z", "summary": "The paper presents a framework for unsupervised disentanglement of images from 2d games into sprites. The framework consists of learning a sprite dictionary, predicting corresponding sprites based on the information from the frame and rendering these sprites. The framework is supervised with reconstruction loss plus some regularization.\n\nOverall the solution makes sense and seems well-thought, however the task 2d game decomposition is not that broad.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive:\n- The solution to the problem makes sense and seems well thought.\n- The paper contains an extensive evaluation and ablation study.\n- The paper tries to apply the proposed method for real world data, e.g. scanned text, president photos and tennis matches.\n\nNegative:\n- The paper considers a 2d game decomposition task in computer vision fashion, however it is questionable if this task has any practical usefulness. Indeed one could use game sprites from game source code and similarly took their arrangement from there.\n\n- The proposed method is very game specific, each game it requires to tune many parameters: depending on the size of sprites in the game the size of the grid need to be adapted; background model need to be adapted as well, for example in the game like Mario separate background for each lever need to learned (here only one lever is considered); number of layers need to selected if there are more than 2 layers in the game.\n\nQuestions:\n- In Fig 8. (Mario) The \"layer 2\" contains some scores and letters renderings which are practically unreadable, however when they are used in reconstruction they look almost perfect, how did this happen? Is the layer just overplayed with the background for rendering or some other post processing is applied?\n\n- For \"Transform Predictor\" why only shifts are considered, is it possible to use arbitrary affine transformations?\n\nSome minor issues:\n- Figure 10 and Figure 6 should be Tables 10 and Table 6.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations is discussed in the paper and in the main review. There is no negative impact.", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "2d09yIa8L8", "reviewer": "Reviewer_7ZfW", "summary": "This paper proposes a method to disentangle sprite-based video animations into recurring visual elements in a self-supervised manner. Specifically, it decompose a frame into grids and try to reconstruct the frame by drawing elements from a dictionary of discovered sprites for each grid. By training with this reconstruction objective, the model could discover a set of disentangled sprite elements. The authors show the effectiveness of their method both through qualitative discovery results as well as quantitative segmentation metrics. With these results, the authors demonstrate that their method work much better compared to previous methods in terms of discovering these recurring visual elements.    \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The topic of self-supervised discovery of recurring visual elements is very interesting. Though has been explored before, the proposed method tackles this problem with a very explicit and intuitive grid-based modeling approach. \n+ The ides is well executed, the qualitative results are convincing to show that the method works pretty well in the video game animation domain. Also, as the quantitative results demonstrated, the proposed method has a large advantage compared to previous methods in visual discovery and disentanglement. \n+ The paper is well-written and easy to follow. \n+ The authors have promised to release the code and models. \n+ I appreciate the discussion on the limitation of the proposed approach. \n\nRegarding the approach\n- The biggest concern I have about this work is that, though demonstrated to work very well on video game frames, it's largely unknown how well it can generalize to the domain of real videos. Even with the example shown in Fig. 14, the video is relatively simple and with clean background. It would be nice to see how it works in more challenging scenarios, e.g. evaluate segmentation quality on datasets like DAVIS. \n- I understand this paper is trying to make a step towards self-supervised visual discovery and appreciate the choice of video game frames as a controllable and easy-to-source testbed to develop prototypes. With that, I'm not sure what is the utility of the editing GUI demonstrated in Figure 9? Since for video games, we already have the full control of all disentangled elements, why do we need to develop such an application? \n- In Eq 3, I'm curious how important are those regularization terms? Does the method learn at all without them? \n\nTypos\n- In Figure 3, the bottom-right texts should be \"switches p^l_j\"?\n\nOverall, I appreciate the direction taken and the approach developed in this paper and believe it would be inspiring for the community. \n\n####### Post rebuttal #######\nI thank the authors for their response, they are useful in further clearing some questions I had about the work. I'd like to maintain my score to accept this paper. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes\n", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns spotted. \n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes a self-supervised approach to represent images as a composition of elements from a dictionary of patches onto a canvas.

Reviewers felt that the paper tackles an interesting problem, is well-executed and provides convincing results for the video game animation domain. The main concern among reviewers is generalizability to more complex domains. The authors respond by pointing out that sprite-based animation and game content is already challenging enough for current approaches. They also point to Section 3.4 of the submission, which shows preliminary results on natural images and videos, noting that a richer image formation model would be required to obtain photorealistic results.

Reviewer vWXs was concerned that the proposed approach might be relying too much on task specificities to recover good sprites, which was addressed by the authors by clearing up confusion on how the data is scaled during preprocessing. Reviewer vWXs remains concerned that the translation-only design choice makes the approach brittle to variations in scale, pose, lighting, etc. between the training and evaluation settings, but notes that the approach can recover from that weakness through a better "transform predictor".

Given that overall reviewers agree that the submission advances the state-of-the-art with a new and interesting approach, I recommend acceptance.
Decision: Accept (Poster)