Character Generation through Self-Supervised Vectorization

1 INTRODUCTION . While , innately , humans sketch or write through strokes , this type of visual depiction is a more difficult task for machines . Image generation problems are typically addressed by raster-based algorithms . The introduction of generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) , variational autoencoders ( VAE ) ( Kingma & Welling , 2013 ) and autoregressive models ( Van Oord et al. , 2016 ) has led to a variety of applications . Style transfer ( Gatys et al. , 2015 ; Isola et al. , 2017 ) , photo realistic image generation ( Brock et al. , 2018 ; Karras et al. , 2019 ) , and super resolution ( Ledig et al. , 2017 ; Bin et al. , 2017 ) are some of the significant instances of the advancing field . Additionally , Hierarchical Bayesian models formulated by deep neural networks are able to use the same generative model for multiple tasks such as classification , conditional and unconditional generation ( Hewitt et al. , 2018 ; Edwards & Storkey , 2016 ) . These raster-based algorithms can produce high quality images , yet they can not benefit from the leverage that higher level abstractions bring about . Vector-level image representation intrinsically prevents models from generating blurry samples and allows for compositional image generation which eventually may contribute to our understanding of how humans create or replicate images ( Lake et al. , 2017 ) . This idea , with the introduction of sketch-based datasets such as Omniglot ( Lake et al. , 2012 ) , Sketchy ( Sangkloy et al. , 2016 ) , and QuickDraw ( Ha & Eck , 2017 ) has triggered a significant body of work in recent years . Stroke based image generation and parsing has been addressed with both vector supervised models and self-supervised generation . Of these , one prominent algorithm is Bayesian Program Learning ( Lake et al. , 2015 ) , where a single model can be utilized for 5 tasks in the Omniglot challenge : ( i ) parsing , ( ii ) unconditional generation or , ( iii ) generating exemplars of a given concept , ( iv ) generating novel concepts of a type , and ( v ) one-shot classification . This approach is also shown to be scalable when supported by the representative capabilities of neural networks ( Feinman & Lake , 2020b ; a ) , however , it requires stroke-level or vector supervision , which is costly to obtain or simply nonexistent . VAE/RNN ( Ha & Eck , 2017 ; Cao et al. , 2019 ; Chen et al. , 2017 ; Aksan et al. , 2020 ) and Transformer based models ( Ribeiro et al. , 2020 ; Lin et al. , 2020 ) are other common methods applied to vector based image generation . Although impressive results have been presented , stroke-level supervision is required to train these models . Figure 1 : Our drawing agent can accomplish four different tasks . From left to right : it can generate novel characters , parse a given character into its strokes , generate new exemplars for a given character , and generate novel concepts ( i.e . characters ) given a type ( i.e . alphabet ) . Ours is the first stroke-based method to tackle all of the generation and parsing tasks in the Omniglot Challenge , without requiring any stroke-level supervision . Recently , self-supervised ( i.e . the absence of strokelevel supervision ) strokebased image generation has been addressed with Reinforcement Learning ( RL ) ( Ganin et al. , 2018 ; Mellor et al. , 2019 ; Huang et al. , 2019 ; Schaldenbrand & Oh , 2020 ) . We call this approach self-supervised vectorization , since the vectorization of images is learned using only rasterimages as supervision . These methods mostly focus on image reconstruction and their exploration in generation is limited . For example , none of them address the conditional generation problem , or , they need the number of actions ( i.e . strokes ) as input . In this paper , we propose a self-supervised reinforcement learning approach where we train a drawing agent for character generation and parsing . Our drawing agent operates on the stroke-level ( i.e . vector ) representation of images . At each time step , our agent takes the current canvas as input and dynamically decides whether to continue drawing or stop . When a ‘ continue ’ decision is made , the agent outputs a program specifying the stroke to be drawn . A non-differentiable renderer takes this program and draws it on the current canvas . Consequently , a raster image is produced stroke-by-stroke . We first train this agent for two tasks by formulating appropriate loss functions : ( i ) unconditional character generation and ( ii ) parsing . Unconditional character generation is the task of generating a novel concept1 ( i.e . character ) given a dataset of concepts . For this task , our loss function includes the following components : an adversarial loss produced by a discriminator to make generated characters as “ real ” as possible , and two data fidelity losses assessing the conformity of the current canvas with the statistical properties of the overall dataset . We also use an additional entropy loss to prevent mode collapse . In the parsing task , the goal for our agent is to reconstruct a given character ( in raster-image ) by drawing it through strokes using as few of them as possible . We utilize the same action space and environment as in the unconditional generation model , only difference being the input fed to the policy is a complete canvas to be reconstructed . Our reward function in this task has two components : a fidelity reward that indicates how much of a stroke is consistent with the target image and a penalty that increases with every ‘ continue ’ action being taken . This model explicitly learns the vectorization of the input raster-image in a self-supervised manner . Next , we show that our parsing model can be exploited for exemplar generation ( i.e . a novel drawing of a given character ) and novel concept generation from type ( i.e . novel character generation given an alphabet of 10 characters ) without any further training . Given a character , the policy network of our parsing model outputs a distribution over the action space where likelihood of actions at each time step eventually allows us to generate variations of the input image . For novel concept generation conditioned on a type ( i.e . alphabet ) , we compose a stroke library by parsing the provided inputs . As we sample strokes from this library , we observe novel samples forming , in coherence with the overall structure of the alphabet . To the best of our knowledge , we are the first to tackle these tasks with a self-supervised approach that operates on stroke space . Through experiments we show that our agent can successfully generate novel characters in all three ways ( unconditionally , conditioned on a given alphabet , conditioned on a given character ) , and parse and reconstruct input characters . For both exemplar generation and type conditioned novel concept generation , we provide LPIPS ( Zhang et al. , 2018 ) , L2 and SSIM measures between input samples and generated images . Our contributions in this paper are two-fold : ( i ) we present a drawing agent that can successfully handle all of the generation and parsing tasks in the Omniglot challenge in a self-supervised , stroke- 1Omniglot challenge terminology . based manner – such a model did not exist ( ii ) we provide for the first time perceptual similarity based quantitative benchmarks for the ‘ exemplar generation ’ and ‘ type conditioned novel concept generation ’ tasks . 2 RELATED WORK . The main purpose of this work is to present a self-supervised approach in order to solve the generation and parsing tasks in the Omniglot Challenge ( Lake et al. , 2015 ) , by capturing the stroke-level representation of images . Here we initially examine the supervised and self-supervised approaches to Omniglot challenge . Then , we review the work on image vectorization . And lastly , we touch upon the research on program synthesis in the context of this study . Omniglot Challenge Omniglot dataset of world alphabets was released with a set of challenges : parsing a given letter , one shot classification , generating a new letter given an alphabet , generating a novel sample of a character , and unconditional generation . Omniglot letters have samples that are conditionally independent based on the alphabet-character hierarchy , hence , a distinctive approach to achieve all these tasks is Hierarchical Bayesian modeling ( Lake et al. , 2015 ) , ( Lake et al. , 2013 ) . As the Omniglot letters included human strokes as labels , the compositional and causal nature of letters are leveraged to model the generation process . Later , neurosymbolic models are also shown to be successful for unconditional generation ( Feinman & Lake , 2020a ) and conceptual compression for multiple tasks presented within the Omniglot Challenge ( Feinman & Lake , 2020b ) . However , without the stroke set that generated a concept , these tasks become more difficult . The idea of sequential image generation is examined by recurrent VAE models ( Rezende et al. , 2016 ) , ( Gregor et al. , 2015 ) , ( Gregor et al. , 2016 ) . DRAW ( Gregor et al. , 2015 ) and Convolutional DRAW ( Gregor et al. , 2016 ) were able to generate quality unconditional samples from MNIST and Omniglot datasets respectively . DRAW is proposed as an algorithm to generate images recurrently . The network is able to iteratively generate a given image by attending to certain parts of the input at each time step . Convolutional DRAW improved the idea with an RNN/VAE based algorithm that can capture the global structure and low-level details of an image separately in order to increase the quality of generations . Later , it is shown that Hierarchical Bayesian Modeling can be improved by the representational power of deep learning and attentional mechanisms in order to achieve three of the five Omniglot challenges ( Rezende et al. , 2016 ) . Another novel idea to leverage Bayesian modeling to tackle Omniglot Challenge was performing modifications on the VAE architecture to represent hierarchical datasets ( Edwards & Storkey , 2016 ) ( Hewitt et al. , 2018 ) . The significance of these studies is that they were able obtain latent variables to describe class-level features effectively . Despite the ability to utilize the same model for different problems ( one-shot classification , unconditional and conditional generation ) , raster-based one-step generative models have two disadvantages we want to address . First , they can not leverage the higher level abstraction and quality comes with working on a vector space . Secondly , one-step generation does not provide an interpretable compositional and causal process describing how a character is generated . In this work , we combine the advantages of two groups of aforementioned models with an agent operating on stroke representation of images that uses only raster images during training . Thus , we aim to solve all three generative and the parsing ( reconstruction ) tasks of the Omniglot challenge . We show that the model trained for reconstruction can also be adopted as a tool that captures the compositional structure of a given character . Without any further training , our agent can solve exemplar generation and type conditioned novel concept generation problems . Image Generation by Vectorization — With Stroke Supervision Sketch-RNN ( Ha & Eck , 2017 ) is the first LSTM/VAE based sketch generation algorithm . It is later improved to generate multiclass samples ( Cao et al. , 2019 ) and increase the quality of generations by representing strokes as Bezier curves ( Song , 2020 ) . The idea of obtaining a generalizable latent space by imagestroke mapping is studied by many ( Aksan et al. , 2020 ; Das et al. , 2021 ; Bhunia et al. , 2021 ; Wang et al. , 2020 ) . In CoSE ( Aksan et al. , 2020 ) , the problem is articulated as ‘ completion of partially drawn sketch ’ . They achieved state of the art reconstruction performance by utilizing variable-length strokes and a novel relational model that is able to capture the global structure of the sketch . The progress in stroke representation is continued with incorporation of variable-degree Bezier curves ( Das et al. , 2021 ) , and capturing Gestalt structure of partially occluded sketches ( Lin et al. , 2020 ) . Self Supervised Vectorization Self-supervised vector-based image generation problem has been approached by RL based frameworks ( Zhou et al. , 2018 ) , ( Ganin et al. , 2018 ) , ( Mellor et al. , 2019 ) , ( Huang et al. , 2019 ) , ( Schaldenbrand & Oh , 2020 ) , and ( Zou et al. , 2020 ) . In SPIRAL ( Ganin et al. , 2018 ) , unconditional generation and reconstruction tasks are tackled with adversarially trained RL agents . Succeeding research enhanced the reconstruction process by a differentiable renderer , making it possible for agents to operate on a continuous space ( Huang et al. , 2019 ; Schaldenbrand & Oh , 2020 ) . In order to avert the computational expense of RL based algorithms , end-to-end differentiable models are developed through altering the rendering process ( Nakano , 2019 ) or formulating the generation process as a parameter search ( Zou et al. , 2020 ) . More recently , a differentiable renderer and compositor is utilized for generating closed Bezier paths and the final image respectively ( Reddy et al. , 2021 ) . This method led to successful interpolation , reconstruction , and sampling processes . Most related to our work is SPIRAL where both reconstruction and unconditional generation is studied through self-supervised deep reinforcement learning . However , our approach has some significant differences . First , in SPIRAL each stroke is also represented as a Bezier curve , yet , the starting point of each curve is set as the final point of the previous curve . In our model , all control points of the Bezier curve are predicted by the agent at each time step . Hence , the agent has to learn the continuity and the compositionality of the given character in order to produce quality samples . Secondly , SPIRAL provides a generative model that works through a graphics renderer without addressing the conditional generation problem . They show impressive results on both natural images and handwritten characters . While we provide a solution for multiple generative tasks , we have not explored our model in the context of natural images . Another approach that presents a similar scheme to the reconstruction problem is “ Learning to Paint ” ( Huang et al. , 2019 ) . In Learning to Paint , the proposed model is utilized specifically for reconstruction . When reconstruction is considered , the main difference of our model is that since we try to model a human-like generation process , our agent outputs a single stroke at each time step with the environment being altered throughout this process while in Learning to Paint , 5 strokes are predicted by the agent at each time step . As a major difference from previous studies , our agent decides whether to stop or keep drawing before generating a stroke . This enables the agent to synthesize an image with as few actions as possible when motivated with our reward formulations . Self Supervised Program Synthesis Our method essentially outputs a visual program that depends only on the rastered data . In that sense , studies on Constructive Solid Geometry ( CSG ) are also related . Different RL frameworks for reconstruction of a given CSG image , that is essentially a composition of geometric shapes , are proposed ( Ellis et al. , 2019 ; Zhou et al. , 2020 ) . The former considered parsing as a search problem that is solved by using a read-eval-print-loop within a Markov Decision Process . The latter adopted a Tree-LSTM model to eliminate invalid programs and the reward is considered to be the Chamfer distance between the target image and current canvas . 3 METHOD Our model consists of a policy network and a ( nondifferentiable ) renderer . At time step t , the policy network takes the current canvas , Ct – a raster-image , as input and outputs two distributions , πB and πS . The first distribution , πB , is for stroke ( i.e . Bezier curve ) parameters and the second one , πS , is for the continue/stop decision . From the first distribution , we randomly sample a stroke defined by its 7 parameters ( x-y coordinates of start , end , control points of the quadratic Bezier curve , and a brush-width ) . From the second distribution , we randomly sample a decision . If the decision happens to be ‘ continue ’ , we add the newly sampled stroke to the current canvas , Ct , increment time ( i.e . t ← t + 1 ) and restart . If the decision was to ‘ stop ’ , then Ct is returned as the final output . Our model is able to handle parsing and different generation tasks , and the processing pipeline we just described is common in all these tasks . What changes among tasks is the reward functions and/or training procedures , which we explain below . Unconditional Generation The task of ‘ generating new concepts ’ as dubbed in Omniglot challenge , is essentially unconditional sampling from a distribution obtained from the whole Omniglot training set . Here , the model is asked to generate completely novel samples ( i.e . characters ) without any constraints . For this task , at each time step t , we calculate an instantaneous reward , rt , that has three components : rt = D ( Ct ) + λ1align ( Ct , I ) + λ2N ( |Ct| ; µ , σ ) . ( 1 ) The first term is a reward based on a discriminator to make generated characters as ‘ real ’ as possible . D ( · ) is a discriminator that outputs the “ realness ” score of its input canvas . We train it in an adversarial manner by using the generated examples as negatives and the elements of the input dataset as positives . The second term is a clustering-based data fidelity reward . The function align ( Ct , I ) measures the alignment between the current canvas Ct and another canvas I , which is a randomly selected cluster center at the beginning of each episode . The cluster centers are obtained by applying k-means on all characters in the input dataset . align basically counts the number of intersecting on-pixels ( between the two canvases ) minus the number of non-intersecting on-pixels in Ct , and divides this quantity by the number of on-pixels in I . The final term assesses the conformity of the current canvas with the dataset in terms of the number of on-pixels . N ( |Ct| ; µ , σ ) evaluates a normal distribution with ( µ , σ ) at |Ct| which is the number of on-pixels in the current canvas . We obtain ( µ , σ ) by fitting a normal distribution to the on-pixel counts of characters in the training set . We observed that the second and third terms accelerate learning as they guide the exploration within the vicinity of real characters . During training , instead of using the instantaneous reward , rt , we use the difference of successive rewards , i.e . rt − rt−1 . In order to encourage exploration and avoid mode collapse , we use an entropy penalty term as αmax ( 0 , KL ( [ πB , πS ] , U ) − τ ) . ( 2 ) Here , KL indicates KL-divergence and U is the uniform distribution . This term first measures the divergence between the uniform distribution and πB , πS , the distributions output by the policy network . Then , through the hinge function , if the divergence exceeds a threshold ( τ ) , this term activates and increases the penalty . The policy network and the discriminator D are updated alternatingly after 256 images are generated at each iteration . We employ the REINFORCE algorithm ( Williams , 1992 ) to update the weights of the policy network . Discriminator is trained using hinge loss . In order to stabilize the discriminator and keep the Lipschitz constant for the whole network equal to 1 , Spectral Normalization is applied at each layer ( Miyato et al. , 2018 ) . Throughout the training , we kept the balance ratio between generated and real samples at 3 . Image Reconstruction by Parsing In the “ parsing ” task , the goal is to reconstruct the given input image by re-drawing it through strokes as accurately as possible . To this end , we formulate a new reward function with two terms : a fidelity reward that indicates how much of a stroke is consistent with the input image ( using the “ align ” function introduced above ) and a penalty that is added with every time increment represented by t as ‘ continue ’ decisions being made : rt = align ( St , Ct ) − λ1t , ( 3 ) where St is the newly sampled stroke and Ct is the current canvas ( input ) . Second term simply acts as a penalty for every ‘ continue ’ action . The first term ensures the sampled stroke to be well-aligned with the input and the second term forces the model to use as few strokes as possible . There is no need for a discriminator . This model explicitly learns the vectorization of the input raster-image in a self-supervised manner . Apart from the different reward function , another crucial difference between the training of the unconditional generation model and the parsing model is how the input and output are handled . In unconditional generation , the newly-sampled stroke is added to the current canvas , whereas in parsing , we do the opposite : the sampled stroke is removed ( masked out ) from the current canvas , and the returned final canvas is the combination of all sampled strokes until the ‘ stop ’ decision . λ , α and τ in Equations 1 , 2 , and 3 are hyperparameters adjusted experimentally . ( see ‘ Training Details ’ in Appendix B ) . Generating New Exemplars In this task , a model is required to generate a new exemplar ( i.e . a variation ) of an unseen concept ( i.e . character ) . To the best of our knowledge , we are the first to tackle this task in a self-supervised stroke-based setting . Most importantly , we do not require any training to achieve this task . We utilize our parsing network described in the previous section to capture the overall structure of a given letter . In order to produce new exemplars , we randomly sample different parsings ( a set of strokes ) from the distribution generated by the agent . In order to eliminate ‘ unlikely ’ samples , we compute the likelihood of the parsing given the resulting policy , and apply a threshold . Generating Novel Concepts from Type In this task , the goal is to to generate a novel concept ( i.e . character ) given a previously unseen type ( i.e . alphabet ) consisting of 10 concepts . The novel concepts should conform to the overall structure , that is , the stroke formulation and composition of the given type ( alphabet ) . We , again , tackle this challenge using our parsing network without any further training . To do so , we first parse all input images into its strokes . For each input image , we sample five stroke sets from the stroke-parameters distribution output by the policy network . During the sampling process , we again use the likelihood-based quality function described in the previous section . We add all the strokes sampled during this process to form a stroke library . Here the strokes are stored with the time steps they are generated . Noting that the number of strokes sampled for a given character is not constant , we approximate a distribution for stopping actions . This process provides a stroke set representing the structure of letters and the way they are composed , that is , we can exploit the compositionality and causality of an alphabet . Throughout the character generation process , a stroke is sampled at each time step belonging to that particular group of the library . The sampled strokes are summed together to obtain the final canvas .

Summary: The paper proposes a self-supervised reinforcement learning approach to train a drawing agent for character generation and parsing. The drawing agent operates on the stroke-level (i.e. vector) representation of images. Different from the one-step generative model, the proposed method can capture the compositional structure of a given character. It is interesting to tackle these tasks with a self-supervised approach and reinforcement learning that operates on stroke space.