MarioNette: Self-Supervised Sprite Learning

Artists and video game designers often construct 2D animations using libraries of sprites—textured patches of objects and characters . We propose a deep learning approach that decomposes sprite-based video animations into a disentangled representation of recurring graphic elements in a self-supervised manner . By jointly learning a dictionary of possibly transparent patches and training a network that places them onto a canvas , we deconstruct sprite-based content into a sparse , consistent , and explicit representation that can be easily used in downstream tasks , like editing or analysis . Our framework offers a promising approach for discovering recurring visual patterns in image collections without supervision . Since the early days of machine learning , the accepted unit of image synthesis has been the pixel . But while the pixel grid is a natural representation for display hardware and convolutional generators , it does not easily permit high-level reasoning and editing . In this paper , we take inspiration from animation to consider an atomic unit that is richer and easier to edit than the pixel : the sprite . In sprite-based animation , a popular early technique for drawing cartoons and rendering video games , an artist draws a collection of patches—a sprite sheet— consisting of texture swatches , characters in various poses , static objects , and so on . Then , each frame is assembled by compositing a subset of the patches onto a canvas . By reusing the sprite sheet , authoring new content requires minimal effort and can even be automated procedurally . Our goal is to invert this process , simultaneously tackling unsupervised instance segmentation and dictionary learning . Given an image dataset , e.g. , frames from a sprite-based video game , we train a model that jointly learns a 2D sprite dictionary , capturing recurring visual elements in an image collection , and explains each input frame as a combination of these potentially transparent sprites . Whereas standard CNN-based generators hide their feature representation in their intermediate layers , our model wears its representation “ on its sleeve ” : by explicitly compositing sprites from its learnt dictionary onto a background canvas , rather than synthesizing pixels from hidden neural features , it provides a readily-interpretable visual representation . Our contributions include the following : • We describe a grid-based anchor system along with a learned dictionary of textured patches ( with transparency ) to extract a sprite-based image representation . • We propose a method to learn the patch dictionary and the grid-based representation jointly , in a differentiable , end-to-end fashion . • We compare to past work on learned disentangled graphics representations for video games . • We show how our method offers promising avenues for further work towards identifying visual patterns in more complex data such as natural images and video . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Related Work . Decomposing visual content into semantically meaningful parts for analysis , synthesis , and editing is a long-standing problem . We review the most closely related work . Layered decompositions . Wang and Adelson [ 54 ] decompose videos into layers undergoing temporally-varying warps for compression . Similarly , Flexible Sprites [ 24 ] and Kannan et al . [ 27 ] represent videos with full-canvas semi-transparent layers to facilitate editing . Like Flexible Sprites , we adopt translation-only motion but restrict transformations to small neighborhoods around anchors , making inference tractable with many ( ≥100 ) sprites . Other methods decompose videos with moving subjects , such as humans , into independent layers , enabling matting [ 41 ] and retiming of individual actions [ 40 ] ; unlike sprite-based techniques , motion and appearance are not disentangled . Sbai et al . [ 48 ] use a layered representation as inductive bias in a GAN with solid colored layers . Automatic decompositions into “ soft layers ” according to texture , color , or semantic features have been used in image editing [ 1 , 2 ] . Gandelsman et al . [ 12 ] use deep image priors [ 52 ] to separate images into layer pairs . Huang and Murphy [ 19 ] introduce a recurrent architecture to output multiple layers sequentially . Reddy et al . [ 46 ] discover patterns in images via differentiable compositing . Interpretable generators for neural synthesis . Neural networks improve the fidelity and realism of generative models [ 14 , 28 ] but limit control and interpretability [ 5 , 6 , 8 , 16 ] . Several works explore interpretability using differentiable domain-specific functions . Hu et al . [ 18 ] , Li et al . [ 31 ] constrain the generator to sets of parametric image operators . Mildenhall et al . [ 44 ] use a ray-marching prior and rendering model to encode a radiance field for novel view synthesis . Neural textures [ 51 ] replace RGB textures on 3D meshes with high-dimensional features . Rendering under new views enables view-consistent editing . Lin et al . [ 33 ] use spatial transformers in their generator to obtain geometric transformations . We synthesize frames by compositing 2D sprites undergoing rigid motions , enabling direct interpretation and control over appearance and motion . Object-centric representations . Our learned sprites reveal , segment , and track object instances . Similarly , Slot Attention [ 37 ] extracts object-centric compositional video representations . However , our sprites are interpretable—motion and appearance are direct outputs—and our model scales to more objects per scene . SCALOR [ 23 ] handles up to 100 instances but does not produce a common dictionary or handle diverse sprites . While SPACE [ 34 ] decomposes images into object layers , it tends to embed sprites in the background , providing no control . Our method achieves a higher IoU of recurring sprite patterns ( see §3.1 ) . Stampnet [ 53 ] discovers and localizes objects but focuses on simpler , synthetic datasets . MONet [ 7 ] decomposes images into multiple object regions using attention . Earlier attention mechanisms leverage pattern recurrence [ 9 , 30 ] and motion cues [ 11 ] to identify individual objects . Recent works use parametric primitives as image building blocks [ 32 , 49 ] . Applying our sprite decompositions to video games , we can learn about dynamics and gameplay , benefiting downstream agents [ 17 , 26 ] and aiding content-authoring for research and game development , as in Procedural Content Generation [ 50 ] . GameGAN [ 29 ] synthesizes new frames from controller input . They split rendering into static and dynamic components but render full frames , without factorization into parts . Their generator is difficult to interpret : appearance and dynamics are entangled within its parameters . Compression . Appearance consistency and motion compensation are central to video compression [ 4 , 38 , 42 ] . We model videos as compositions of moving sprites , factoring redundancy in the input . This draws inspiration from works like DjVu [ 15 ] and Digipaper [ 20 ] , which compress scanned documents by separating them into a background layer and foreground text . Image epitomes [ 25 ] summarize and compress image shape and appearance into a miniature texture . Our sprite dictionary fills a similar role , providing superior editing control . 2 Method . We start with an input sequence of n RGB frames { I1 , . . . , In } with resolution w×h . Our goal is to decompose each frame Ii ∈ R3×w×h into a set of possibly overlapping sprites , organized into ` depth layers , selected from a finite-size dictionary . The dictionary is a collection of trainable latent codes { z1 , . . . , zm } that are decoded into RGBA sprites using a neural network generator ( §2.1 ) . Our training pipeline is illustrated in Figure 1 . We first process each input frame with a convolutional encoder to produce ` grids of feature vectors , one grid per depth layer ( §2.2 ) . The grids are lower resolution than the input frame , with a downsampling factor proportional to the sprite size . We call the center of each grid cell an anchor . We compare each anchor ’ s feature vector against the dictionary ’ s latent codes , using a softmax scoring function , to select the best matching sprite per anchor ( §2.3 ) . Using our sprite generator , we decode each anchor ’ s matching sprite . This gives us a grid of sprites for each of the ` layers . To factorize image patterns that may not align with our anchor grid , we allow sprites to move in a small neighborhood around anchors ( §2.4 ) . We composite the layers from back to front onto the output canvas to obtain our final reconstruction ( §2.5 ) . Optionally , the background is modeled as a special learnable sprite that covers the entire canvas . We train the dictionary latent codes , frame encoder , and sprite generator jointly on all frames , comparing our reconstruction to the input ( §2.6 ) . This self-supervised procedure yields a representation that is sparse , compact , interpretable , and well-suited for downstream editing and learning applications . 2.1 Dictionary and sprite generator . The central component of our representation is a global dictionary of m textured patches or sprites D = { P1 , . . . , Pm } , where each Pi ∈ R4×k×k is an RGBA patch . Our sprites have an alpha channel , which allows them to be partially transparent , with possibly irregular ( i.e. , non-square ) boundaries . This is useful for representing animations with multiple depth layers and also allows to learn sprites smaller than their maximal resolution , if necessary , by setting alpha to zero around the boundary . The dictionary is shared among all frames ; we reconstruct frames using only sprites from the dictionary . Instead of optimizing for RGBA pixel values directly , we represent the dictionary as a set of trainable latent codes { z1 , . . . , zm } , with zi ∈ Rd . We decode these codes into RGBA sprites using a fullyconnected sprite generator Pi = G ( zi ) . This latent representation allows us to define a similarity metric over the latent space , which we use to pair anchors with dictionary sprites to best reconstruct the input frame ( §2.3 ) . At test time , we can forego the sprite generator and edit the RGBA sprites directly . Unless otherwise specified , we set latent dimension to d = 128 and patch size to k = 32 . We randomly initialize the latent codes from the standard normal distribution . Our sprite generator first applies zero-mean unit-variance normalization—Layer Normalization [ 3 ] , without an affine transformation—to each latent code zi individually , followed by one fully-connected hidden layer with 8d features , Group Normalization [ 55 ] , and ReLU activation . We obtain the final sprite using a fullyconnected layer with sigmoid activation to keep RGBA values in [ 0 , 1 ] . Latent code normalization is crucial to stabilize training and keep the latent space in a compact subspace as the optimization progresses . See §3.3 for an ablation study of this and other components . 2.2 Layered frame decomposition using sprite anchors . We seek a decomposition that best explains each input frame using dictionary sprites . We exploit translation invariance and locality in our representation ; our sprites are “ attached ” to a regular grid of reference points , or anchors , inspired by [ 13 , 47 ] . Each anchor has at most one sprite ; we call it inactive if it has none . We give the sprites freedom of motion around their anchors to factorize structures that may not be aligned with the anchor grid . This local—or , Eulerian—viewpoint makes inference tractable and avoids the pitfalls of tracking the global motion of all the sprites across the canvas ( a Lagrangian viewpoint ) . To enable multiple layers with sprite occlusions , we output ` > 1 anchor grids for each frame ( ` = 2 in our experiments ) . Figure 2 illustrates our layered anchor grids and local sprite transformations . We use a convolutional encoder E to map the w×h RGB frame Ii to ` grids of anchors , with resolution 2wk × 2h k . Each anchor j in layer l is represented by a feature vector alj ∈ Rd characterizing local image appearance around the anchor and an active/inactive switch probability plj ∈ [ 0 , 1 ] . Our frame encoder contains log2 ( k ) − 1 downsampling blocks , which use partial convolutions [ 36 ] with kernel size 3 and stride 2 ( for downsampling ) , Group Normalization , and Leaky ReLU . It produces a tensor of intermediate features for each layer , which are normalized with LayerNorm . From these , we obtain the anchor switches with an MLP with one hidden layer of size d followed by Group Normalization and Leaky ReLU . We get anchor features using a linear projection followed by LayerNorm . The encoder architecture is illustrated in Figure 3 .

Summary: This paper proposes a method to disentangle sprite-based video animations into recurring visual elements in a self-supervised manner. Specifically, it decompose a frame into grids and try to reconstruct the frame by drawing elements from a dictionary of discovered sprites for each grid. By training with this reconstruction objective, the model could discover a set of disentangled sprite elements. The authors show the effectiveness of their method both through qualitative discovery results as well as quantitative segmentation metrics. With these results, the authors demonstrate that their method work much better compared to previous methods in terms of discovering these recurring visual elements.