MarioNette: Self-Supervised Sprite Learning

Artists and video game designers often construct 2D animations using libraries of sprites—textured patches of objects and characters . We propose a deep learning approach that decomposes sprite-based video animations into a disentangled representation of recurring graphic elements in a self-supervised manner . By jointly learning a dictionary of possibly transparent patches and training a network that places them onto a canvas , we deconstruct sprite-based content into a sparse , consistent , and explicit representation that can be easily used in downstream tasks , like editing or analysis . Our framework offers a promising approach for discovering recurring visual patterns in image collections without supervision . Since the early days of machine learning , the accepted unit of image synthesis has been the pixel . But while the pixel grid is a natural representation for display hardware and convolutional generators , it does not easily permit high-level reasoning and editing . In this paper , we take inspiration from animation to consider an atomic unit that is richer and easier to edit than the pixel : the sprite . In sprite-based animation , a popular early technique for drawing cartoons and rendering video games , an artist draws a collection of patches—a sprite sheet— consisting of texture swatches , characters in various poses , static objects , and so on . Then , each frame is assembled by compositing a subset of the patches onto a canvas . By reusing the sprite sheet , authoring new content requires minimal effort and can even be automated procedurally . Our goal is to invert this process , simultaneously tackling unsupervised instance segmentation and dictionary learning . Given an image dataset , e.g. , frames from a sprite-based video game , we train a model that jointly learns a 2D sprite dictionary , capturing recurring visual elements in an image collection , and explains each input frame as a combination of these potentially transparent sprites . Whereas standard CNN-based generators hide their feature representation in their intermediate layers , our model wears its representation “ on its sleeve ” : by explicitly compositing sprites from its learnt dictionary onto a background canvas , rather than synthesizing pixels from hidden neural features , it provides a readily-interpretable visual representation . Our contributions include the following : • We describe a grid-based anchor system along with a learned dictionary of textured patches ( with transparency ) to extract a sprite-based image representation . • We propose a method to learn the patch dictionary and the grid-based representation jointly , in a differentiable , end-to-end fashion . • We compare to past work on learned disentangled graphics representations for video games . • We show how our method offers promising avenues for further work towards identifying visual patterns in more complex data such as natural images and video . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Related Work . Decomposing visual content into semantically meaningful parts for analysis , synthesis , and editing is a long-standing problem . We review the most closely related work . Layered decompositions . Wang and Adelson [ 54 ] decompose videos into layers undergoing temporally-varying warps for compression . Similarly , Flexible Sprites [ 24 ] and Kannan et al . [ 27 ] represent videos with full-canvas semi-transparent layers to facilitate editing . Like Flexible Sprites , we adopt translation-only motion but restrict transformations to small neighborhoods around anchors , making inference tractable with many ( ≥100 ) sprites . Other methods decompose videos with moving subjects , such as humans , into independent layers , enabling matting [ 41 ] and retiming of individual actions [ 40 ] ; unlike sprite-based techniques , motion and appearance are not disentangled . Sbai et al . [ 48 ] use a layered representation as inductive bias in a GAN with solid colored layers . Automatic decompositions into “ soft layers ” according to texture , color , or semantic features have been used in image editing [ 1 , 2 ] . Gandelsman et al . [ 12 ] use deep image priors [ 52 ] to separate images into layer pairs . Huang and Murphy [ 19 ] introduce a recurrent architecture to output multiple layers sequentially . Reddy et al . [ 46 ] discover patterns in images via differentiable compositing . Interpretable generators for neural synthesis . Neural networks improve the fidelity and realism of generative models [ 14 , 28 ] but limit control and interpretability [ 5 , 6 , 8 , 16 ] . Several works explore interpretability using differentiable domain-specific functions . Hu et al . [ 18 ] , Li et al . [ 31 ] constrain the generator to sets of parametric image operators . Mildenhall et al . [ 44 ] use a ray-marching prior and rendering model to encode a radiance field for novel view synthesis . Neural textures [ 51 ] replace RGB textures on 3D meshes with high-dimensional features . Rendering under new views enables view-consistent editing . Lin et al . [ 33 ] use spatial transformers in their generator to obtain geometric transformations . We synthesize frames by compositing 2D sprites undergoing rigid motions , enabling direct interpretation and control over appearance and motion . Object-centric representations . Our learned sprites reveal , segment , and track object instances . Similarly , Slot Attention [ 37 ] extracts object-centric compositional video representations . However , our sprites are interpretable—motion and appearance are direct outputs—and our model scales to more objects per scene . SCALOR [ 23 ] handles up to 100 instances but does not produce a common dictionary or handle diverse sprites . While SPACE [ 34 ] decomposes images into object layers , it tends to embed sprites in the background , providing no control . Our method achieves a higher IoU of recurring sprite patterns ( see §3.1 ) . Stampnet [ 53 ] discovers and localizes objects but focuses on simpler , synthetic datasets . MONet [ 7 ] decomposes images into multiple object regions using attention . Earlier attention mechanisms leverage pattern recurrence [ 9 , 30 ] and motion cues [ 11 ] to identify individual objects . Recent works use parametric primitives as image building blocks [ 32 , 49 ] . Applying our sprite decompositions to video games , we can learn about dynamics and gameplay , benefiting downstream agents [ 17 , 26 ] and aiding content-authoring for research and game development , as in Procedural Content Generation [ 50 ] . GameGAN [ 29 ] synthesizes new frames from controller input . They split rendering into static and dynamic components but render full frames , without factorization into parts . Their generator is difficult to interpret : appearance and dynamics are entangled within its parameters . Compression . Appearance consistency and motion compensation are central to video compression [ 4 , 38 , 42 ] . We model videos as compositions of moving sprites , factoring redundancy in the input . This draws inspiration from works like DjVu [ 15 ] and Digipaper [ 20 ] , which compress scanned documents by separating them into a background layer and foreground text . Image epitomes [ 25 ] summarize and compress image shape and appearance into a miniature texture . Our sprite dictionary fills a similar role , providing superior editing control . 2 Method . We start with an input sequence of n RGB frames { I1 , . . . , In } with resolution w×h . Our goal is to decompose each frame Ii ∈ R3×w×h into a set of possibly overlapping sprites , organized into ` depth layers , selected from a finite-size dictionary . The dictionary is a collection of trainable latent codes { z1 , . . . , zm } that are decoded into RGBA sprites using a neural network generator ( §2.1 ) . Our training pipeline is illustrated in Figure 1 . We first process each input frame with a convolutional encoder to produce ` grids of feature vectors , one grid per depth layer ( §2.2 ) . The grids are lower resolution than the input frame , with a downsampling factor proportional to the sprite size . We call the center of each grid cell an anchor . We compare each anchor ’ s feature vector against the dictionary ’ s latent codes , using a softmax scoring function , to select the best matching sprite per anchor ( §2.3 ) . Using our sprite generator , we decode each anchor ’ s matching sprite . This gives us a grid of sprites for each of the ` layers . To factorize image patterns that may not align with our anchor grid , we allow sprites to move in a small neighborhood around anchors ( §2.4 ) . We composite the layers from back to front onto the output canvas to obtain our final reconstruction ( §2.5 ) . Optionally , the background is modeled as a special learnable sprite that covers the entire canvas . We train the dictionary latent codes , frame encoder , and sprite generator jointly on all frames , comparing our reconstruction to the input ( §2.6 ) . This self-supervised procedure yields a representation that is sparse , compact , interpretable , and well-suited for downstream editing and learning applications . 2.1 Dictionary and sprite generator . The central component of our representation is a global dictionary of m textured patches or sprites D = { P1 , . . . , Pm } , where each Pi ∈ R4×k×k is an RGBA patch . Our sprites have an alpha channel , which allows them to be partially transparent , with possibly irregular ( i.e. , non-square ) boundaries . This is useful for representing animations with multiple depth layers and also allows to learn sprites smaller than their maximal resolution , if necessary , by setting alpha to zero around the boundary . The dictionary is shared among all frames ; we reconstruct frames using only sprites from the dictionary . Instead of optimizing for RGBA pixel values directly , we represent the dictionary as a set of trainable latent codes { z1 , . . . , zm } , with zi ∈ Rd . We decode these codes into RGBA sprites using a fullyconnected sprite generator Pi = G ( zi ) . This latent representation allows us to define a similarity metric over the latent space , which we use to pair anchors with dictionary sprites to best reconstruct the input frame ( §2.3 ) . At test time , we can forego the sprite generator and edit the RGBA sprites directly . Unless otherwise specified , we set latent dimension to d = 128 and patch size to k = 32 . We randomly initialize the latent codes from the standard normal distribution . Our sprite generator first applies zero-mean unit-variance normalization—Layer Normalization [ 3 ] , without an affine transformation—to each latent code zi individually , followed by one fully-connected hidden layer with 8d features , Group Normalization [ 55 ] , and ReLU activation . We obtain the final sprite using a fullyconnected layer with sigmoid activation to keep RGBA values in [ 0 , 1 ] . Latent code normalization is crucial to stabilize training and keep the latent space in a compact subspace as the optimization progresses . See §3.3 for an ablation study of this and other components . 2.2 Layered frame decomposition using sprite anchors . We seek a decomposition that best explains each input frame using dictionary sprites . We exploit translation invariance and locality in our representation ; our sprites are “ attached ” to a regular grid of reference points , or anchors , inspired by [ 13 , 47 ] . Each anchor has at most one sprite ; we call it inactive if it has none . We give the sprites freedom of motion around their anchors to factorize structures that may not be aligned with the anchor grid . This local—or , Eulerian—viewpoint makes inference tractable and avoids the pitfalls of tracking the global motion of all the sprites across the canvas ( a Lagrangian viewpoint ) . To enable multiple layers with sprite occlusions , we output ` > 1 anchor grids for each frame ( ` = 2 in our experiments ) . Figure 2 illustrates our layered anchor grids and local sprite transformations . We use a convolutional encoder E to map the w×h RGB frame Ii to ` grids of anchors , with resolution 2wk × 2h k . Each anchor j in layer l is represented by a feature vector alj ∈ Rd characterizing local image appearance around the anchor and an active/inactive switch probability plj ∈ [ 0 , 1 ] . Our frame encoder contains log2 ( k ) − 1 downsampling blocks , which use partial convolutions [ 36 ] with kernel size 3 and stride 2 ( for downsampling ) , Group Normalization , and Leaky ReLU . It produces a tensor of intermediate features for each layer , which are normalized with LayerNorm . From these , we obtain the anchor switches with an MLP with one hidden layer of size d followed by Group Normalization and Leaky ReLU . We get anchor features using a linear projection followed by LayerNorm . The encoder architecture is illustrated in Figure 3 .

Summary: The authors propose a generative layered model of images based on sprites — the idea is that there is a dictionary of (possibly transparent) sprites which are arranged on layers which are, together with an optionally learned background layer, composited back-to-front to form a given image.  The task is to learn the sprite dictionary and their poses and depth ordering on a given frame.  The paper shows successful results on frames from Atari/SNES era video games (which actually did use sprites) and some intriguing (but very much preliminary) one-off results on real images, training on patches sampled at random from those images.