Unsupervised Image to Sequence Translation with Canvas-Drawer Networks

Encoding images as a series of high-level constructs , such as brush strokes or discrete shapes , can often be key to both human and machine understanding . In many cases , however , data is only available in pixel form . We present a method for generating images directly in a high-level domain ( e.g . brush strokes ) , without the need for real pairwise data . Specifically , we train a ” canvas ” network to imitate the mapping of high-level constructs to pixels , followed by a high-level ” drawing ” network which is optimized through this mapping towards solving a desired image recreation or translation task . We successfully discover sequential vector representations of symbols , large sketches , and 3D objects , utilizing only pixel data . We display applications of our method in image segmentation , and present several ablation studies comparing various configurations . 1 INTRODUCTION . When an architect reviews a design file , his eyes witness raw image data . But in his head , he is quickly converting these color values into the lines and shapes they represent . Recent works in image classification ( Krizhevsky et al. , 2012b ; Simonyan & Zisserman , 2014 ; Szegedy et al. , 2016 ) have shown that with deep neural networks , computer agents can learn similar recognition behaviors . The difference appears , however , in how humans and computers produce images . Neural networks generally output color matrices that fully detail each pixel , leading to artifacts and blurriness which many recent works have attempted to combat ( Goodfellow et al. , 2014 ; Karras et al. , 2017 ) . Humans , on the other hand , draw and design on higher-level domains such as brush strokes or primitive shapes . In this work , our aim is to develop computer agents that can operate on a similar level , generating sequences of high-level constructs rather than raw pixels . While a large amount of past work has been done on reproducing sequences given an observation ( Vinyals et al. , 2015 ; Xu et al. , 2015 ) , these methods rely on a pairwise dataset of real images and their corresponding sequences , which can be expensive to produce . We focus on the case where such a dataset is not available , and real images are represented only as pixels . Instead , we define a correct sequence as one that recreates a desired image when passed through a given rendering program , such a digital painting software or 3D engine . We present an end-to-end system for training high-level , sequence-based drawing networks without paired data , by optimizing through a learned approximation of such rendering programs . Our contributions are as follows : • We establish a simple domain-agnostic method for imitating the behavior of a nondifferentiable renderer as a differentiable canvas network . • We propose a framework for training sequence-based drawing networks for translation between a low-level and high-level domain ( e.g . pixels and brush strokes ) , without the need for a paired dataset . The main novelty is in utilizing the canvas network to calculate a tractable loss function between generated sequences and real pixel images . • We develop a method of efficiently extending the canvas-drawer framework to cases where the desired output is a much higher resolution ( e.g . 512x512 sketches ) and consists of hundreds of sequences , through the introduction of sliding and hierarchical architectures . We validate our method on a wide range of tasks , including translation of Omniglot symbols and large sketches into sequences of bezier curves , segmentation of architectural floorplans into parametrized bounding boxes , and recreating 3D scenes from a series of 2D viewpoints . We qualitatively show that trained drawer networks easily extend to out-of-distribution examples unseen during training , and we conduct a series of ablation studies on variations such as number of drawing steps , overlap in sliding drawer networks , and the inclusion of a two-layer hierarchy . 2 RELATED WORK . Previous works have attempted to learn a mapping between images and their corresponding stroke sequences . ( Simhon & Dudek , 2004 ) learns to refine coarse human sketches by training Hidden Markov Models . ( Graves , 2013 ) presents a general scheme for producing sequences with recurrent neural networks , and ( Ha & Eck , 2017 ) builds on this specifically in the case of producing sketches from a latent space . These methods all depend on a dataset of sketches represented in vector form , whereas our work deals with the case where such a dataset is not available . Other works have approached the sketch generation problem from a reinforcement learning perspective . ( Xie et al. , 2013 ) models a drawing agent to manipulate a digital brush and recreate oriential paintings . ( Ganin et al. , 2018 ) considers an end-to-end reinforcement learning agent , and proposes the use of a discriminator as a more accurate reward function . In general , reinforcement learning methods can be unstable and often depend on large amounts of training samples . Our work details a supervised approach where we directly calculate a gradient estimate by modeling a nondifferentiable drawing program as a differentiable canvas network . 3 FORMULATION . 3.1 PROBLEM STATEMENT . In our problem , we have a set of pixel imagesX , and we wish to produce the corresponding sequence of high-level constructs Y . We will refer to this sequence Y as a sequence of actions . While we do not have a dataset of X and Y pairs , we can infer that the correct Y for a given X is one that recreates X when passed through a renderer R ( Y ) . We assume we have access to such a program R ( Y ) , but it is non-differentiable . Instead , we train a canvas network C ( Y ) such that C ( y ) ≈ R ( y ) for y ∼ Y . We can than train a drawer network D ( X ) to minimize the pixel distance between C ( D ( X ) ) and X . Note that the above description is for image recreation . In translation tasks , our data instead consists of X , X ′ pairs where X is a given hint image and X ′ is a target image we would like to produce as a sequence of actions . Our objective is then to minimize the pixel distance between C ( D ( X ) and X ′ . 3.2 CANVAS . In the proposed framework , we view the renderer as a non-differentiable program that maps from a state xn and action yn to the next state xn+1 . We assume there is some starting state x0 , and that states are Markovian and contain all past information . We also assume we know the distribution of possible actions . As gradients can not be passed through the rendering program , we train a differentiable canvas network to act as an approximator by imitating the rendering program for any possible state-action pair . We achieve this by sampling rollouts of ( xn , yn , xn+1 ) from the renderer , selecting actions yn randomly and periodically resetting xn to the initial state x0 . We then train our canvas network to recreate xn+1 , given xn and yn . 3.3 DRAWER . We define the drawer as a conditional recurrent network that runs forN timesteps . At every timestep n , the drawer D ( X , xn ) witnesses original image X and the state xn , and selects an action yn+1 . This state-action pair is given to the renderer R ( x , y ) , which then outputs the next state xn+1 which is given back to the drawer . The true objective of the drawer is to minimize the pixelwise distance between the final state of the renderer Rfinal ( xN−1 , yN ) and the target image X ′ . However , as R ( x , y ) is not differentiable , we instead approximate it with canvas network C ( x , y ) . As each action yn+1 and state xn+1 depends only on the hint image and previous state , the entire graph Cfinal ( xN−1 , yN ) can be viewed as a function of X and x0 . When training the drawer , we sample X , X ′ pairs from a dataset of pixel images and optimize towards minimizing pixel distance between Cfinal ( xN−1 , yN ) and X ′ . x0 is fixed and is generally a blank image . It is important that we train only the parameters of the drawer network , as the canvas network must be frozen to retain its imitation of the rendering program . In practice , we augment the image inputs with a coordinate layer ( Liu et al. , 2018 ) , and utilize an LSTM layer ( Hochreiter & Schmidhuber , 1997 ) to capture time-dependencies in the drawer . To assist gradient flow and increase imitation accuracy , we found it useful to define xn+1 as a pixelwise maximum between C ( xn , yn+1 ) and xn . 3.4 SLIDING For many real-world cases , image data is large and contains many complex structures . Naive recurrent networks become intractable to train past a few dozen timesteps , and higher resolution images require networks with an increasingly larger amount of parameters . To mitigate this issue , we extend our method with the idea of a sliding drawer network . Specifically in the case of images , convolutional kernels ( Krizhevsky et al. , 2012a ) have proven to be useful as features in pixel space are generally translation invariant . We take this notion one step further , and assume that drawing behaviors are also translation invariant . We follow this logic and structure a drawing network that slides across the X and Y axes of an image . For every small section of the larger image , our drawer network predicts a short sequence of actions to perform . To prevent artifacts along the borders of sections , we found it useful to slide our drawing network such that overlap occured within neighboring sections . More details on the sliding architecture , as well as the hierarchical variant , are provided in the experiments section below . 4 EXPERIMENTS In the following section , we validate our claims through a range of qualitative results and quantitative ablation studies . It is important to note while pixelwise distance is a helpful directional benchmark , our true goal is to produce accurate high-level actions ( e.g . brush strokes , rectangles ) , which are evaluated qualitatively . 4.1 CAN CANVAS-DRAWER PAIRS BE USED . TO VECTORIZE SYMBOLS IN AN UNSUPERVISED MANNER ? In a motivating example , we attempt to recreate MNIST digits ( LeCun , 1998 ) and Omniglot symbols ( Lake et al. , 2015 ) in a sequence of four bezier curves , akin to physical brush strokes . Each high-level action represents a single curve , defined by two endpoints and a control point . Our state consists of a 64x64 black-and-white image matrix . Since our task is to recreate the symbol provided , both the hint image and the target image are the same . A drawer networks lasting four timesteps can accurately produce sequential bezier curves representing distinct segments of the MNIST digits . While non-LSTM drawers achieve roughly equal pixelwise loss , LSTM drawers are more consistent in segmenting digits into concrete curves . In a drawer lasting twenty timesteps , there is a noticeable increase in pixelwise performance , however the curves produced are bended , akin to sketching with tiny strokes rather than smooth motions . We 1Gregor et al . ( 2015 ) ; Ganin et al . ( 2018 ) ; He ( 2016 ) predict this is due to a mismatch in curve thickness , and in later experiments we consider bezier curves with variable thickness to address this issue . When trained on recreating Omniglot symbols , the drawer network learns a robust mapping that is capable of also recreating symbols that were out of distribution , such as MNIST digits or hand-drawn examples ( Figure 6g , right column ) .

Summary: To generate a sequence of high-level visual elements for recreation or translation of images, the authors propose differentiable "canvas" networks and "drawer" networks based on convolutional neural networks. One of the main ideas is the replacement of the "canvas" networks instead of non-differentiable "renderer" to end-to-end train the whole model with mean-squared error loss. It seems to be a novel approach to optimize drawing actions. It is reasonable to use separate networks to approximate the behavior of renderer and to fix the parameters of the "canvas" networks to maintain the pretrained rendering capability.