Character Generation through Self-Supervised Vectorization

1 INTRODUCTION . While , innately , humans sketch or write through strokes , this type of visual depiction is a more difficult task for machines . Image generation problems are typically addressed by raster-based algorithms . The introduction of generative adversarial networks ( GAN ) ( Goodfellow et al. , 2014 ) , variational autoencoders ( VAE ) ( Kingma & Welling , 2013 ) and autoregressive models ( Van Oord et al. , 2016 ) has led to a variety of applications . Style transfer ( Gatys et al. , 2015 ; Isola et al. , 2017 ) , photo realistic image generation ( Brock et al. , 2018 ; Karras et al. , 2019 ) , and super resolution ( Ledig et al. , 2017 ; Bin et al. , 2017 ) are some of the significant instances of the advancing field . Additionally , Hierarchical Bayesian models formulated by deep neural networks are able to use the same generative model for multiple tasks such as classification , conditional and unconditional generation ( Hewitt et al. , 2018 ; Edwards & Storkey , 2016 ) . These raster-based algorithms can produce high quality images , yet they can not benefit from the leverage that higher level abstractions bring about . Vector-level image representation intrinsically prevents models from generating blurry samples and allows for compositional image generation which eventually may contribute to our understanding of how humans create or replicate images ( Lake et al. , 2017 ) . This idea , with the introduction of sketch-based datasets such as Omniglot ( Lake et al. , 2012 ) , Sketchy ( Sangkloy et al. , 2016 ) , and QuickDraw ( Ha & Eck , 2017 ) has triggered a significant body of work in recent years . Stroke based image generation and parsing has been addressed with both vector supervised models and self-supervised generation . Of these , one prominent algorithm is Bayesian Program Learning ( Lake et al. , 2015 ) , where a single model can be utilized for 5 tasks in the Omniglot challenge : ( i ) parsing , ( ii ) unconditional generation or , ( iii ) generating exemplars of a given concept , ( iv ) generating novel concepts of a type , and ( v ) one-shot classification . This approach is also shown to be scalable when supported by the representative capabilities of neural networks ( Feinman & Lake , 2020b ; a ) , however , it requires stroke-level or vector supervision , which is costly to obtain or simply nonexistent . VAE/RNN ( Ha & Eck , 2017 ; Cao et al. , 2019 ; Chen et al. , 2017 ; Aksan et al. , 2020 ) and Transformer based models ( Ribeiro et al. , 2020 ; Lin et al. , 2020 ) are other common methods applied to vector based image generation . Although impressive results have been presented , stroke-level supervision is required to train these models . Figure 1 : Our drawing agent can accomplish four different tasks . From left to right : it can generate novel characters , parse a given character into its strokes , generate new exemplars for a given character , and generate novel concepts ( i.e . characters ) given a type ( i.e . alphabet ) . Ours is the first stroke-based method to tackle all of the generation and parsing tasks in the Omniglot Challenge , without requiring any stroke-level supervision . Recently , self-supervised ( i.e . the absence of strokelevel supervision ) strokebased image generation has been addressed with Reinforcement Learning ( RL ) ( Ganin et al. , 2018 ; Mellor et al. , 2019 ; Huang et al. , 2019 ; Schaldenbrand & Oh , 2020 ) . We call this approach self-supervised vectorization , since the vectorization of images is learned using only rasterimages as supervision . These methods mostly focus on image reconstruction and their exploration in generation is limited . For example , none of them address the conditional generation problem , or , they need the number of actions ( i.e . strokes ) as input . In this paper , we propose a self-supervised reinforcement learning approach where we train a drawing agent for character generation and parsing . Our drawing agent operates on the stroke-level ( i.e . vector ) representation of images . At each time step , our agent takes the current canvas as input and dynamically decides whether to continue drawing or stop . When a ‘ continue ’ decision is made , the agent outputs a program specifying the stroke to be drawn . A non-differentiable renderer takes this program and draws it on the current canvas . Consequently , a raster image is produced stroke-by-stroke . We first train this agent for two tasks by formulating appropriate loss functions : ( i ) unconditional character generation and ( ii ) parsing . Unconditional character generation is the task of generating a novel concept1 ( i.e . character ) given a dataset of concepts . For this task , our loss function includes the following components : an adversarial loss produced by a discriminator to make generated characters as “ real ” as possible , and two data fidelity losses assessing the conformity of the current canvas with the statistical properties of the overall dataset . We also use an additional entropy loss to prevent mode collapse . In the parsing task , the goal for our agent is to reconstruct a given character ( in raster-image ) by drawing it through strokes using as few of them as possible . We utilize the same action space and environment as in the unconditional generation model , only difference being the input fed to the policy is a complete canvas to be reconstructed . Our reward function in this task has two components : a fidelity reward that indicates how much of a stroke is consistent with the target image and a penalty that increases with every ‘ continue ’ action being taken . This model explicitly learns the vectorization of the input raster-image in a self-supervised manner . Next , we show that our parsing model can be exploited for exemplar generation ( i.e . a novel drawing of a given character ) and novel concept generation from type ( i.e . novel character generation given an alphabet of 10 characters ) without any further training . Given a character , the policy network of our parsing model outputs a distribution over the action space where likelihood of actions at each time step eventually allows us to generate variations of the input image . For novel concept generation conditioned on a type ( i.e . alphabet ) , we compose a stroke library by parsing the provided inputs . As we sample strokes from this library , we observe novel samples forming , in coherence with the overall structure of the alphabet . To the best of our knowledge , we are the first to tackle these tasks with a self-supervised approach that operates on stroke space . Through experiments we show that our agent can successfully generate novel characters in all three ways ( unconditionally , conditioned on a given alphabet , conditioned on a given character ) , and parse and reconstruct input characters . For both exemplar generation and type conditioned novel concept generation , we provide LPIPS ( Zhang et al. , 2018 ) , L2 and SSIM measures between input samples and generated images . Our contributions in this paper are two-fold : ( i ) we present a drawing agent that can successfully handle all of the generation and parsing tasks in the Omniglot challenge in a self-supervised , stroke- 1Omniglot challenge terminology . based manner – such a model did not exist ( ii ) we provide for the first time perceptual similarity based quantitative benchmarks for the ‘ exemplar generation ’ and ‘ type conditioned novel concept generation ’ tasks . 2 RELATED WORK . The main purpose of this work is to present a self-supervised approach in order to solve the generation and parsing tasks in the Omniglot Challenge ( Lake et al. , 2015 ) , by capturing the stroke-level representation of images . Here we initially examine the supervised and self-supervised approaches to Omniglot challenge . Then , we review the work on image vectorization . And lastly , we touch upon the research on program synthesis in the context of this study . Omniglot Challenge Omniglot dataset of world alphabets was released with a set of challenges : parsing a given letter , one shot classification , generating a new letter given an alphabet , generating a novel sample of a character , and unconditional generation . Omniglot letters have samples that are conditionally independent based on the alphabet-character hierarchy , hence , a distinctive approach to achieve all these tasks is Hierarchical Bayesian modeling ( Lake et al. , 2015 ) , ( Lake et al. , 2013 ) . As the Omniglot letters included human strokes as labels , the compositional and causal nature of letters are leveraged to model the generation process . Later , neurosymbolic models are also shown to be successful for unconditional generation ( Feinman & Lake , 2020a ) and conceptual compression for multiple tasks presented within the Omniglot Challenge ( Feinman & Lake , 2020b ) . However , without the stroke set that generated a concept , these tasks become more difficult . The idea of sequential image generation is examined by recurrent VAE models ( Rezende et al. , 2016 ) , ( Gregor et al. , 2015 ) , ( Gregor et al. , 2016 ) . DRAW ( Gregor et al. , 2015 ) and Convolutional DRAW ( Gregor et al. , 2016 ) were able to generate quality unconditional samples from MNIST and Omniglot datasets respectively . DRAW is proposed as an algorithm to generate images recurrently . The network is able to iteratively generate a given image by attending to certain parts of the input at each time step . Convolutional DRAW improved the idea with an RNN/VAE based algorithm that can capture the global structure and low-level details of an image separately in order to increase the quality of generations . Later , it is shown that Hierarchical Bayesian Modeling can be improved by the representational power of deep learning and attentional mechanisms in order to achieve three of the five Omniglot challenges ( Rezende et al. , 2016 ) . Another novel idea to leverage Bayesian modeling to tackle Omniglot Challenge was performing modifications on the VAE architecture to represent hierarchical datasets ( Edwards & Storkey , 2016 ) ( Hewitt et al. , 2018 ) . The significance of these studies is that they were able obtain latent variables to describe class-level features effectively . Despite the ability to utilize the same model for different problems ( one-shot classification , unconditional and conditional generation ) , raster-based one-step generative models have two disadvantages we want to address . First , they can not leverage the higher level abstraction and quality comes with working on a vector space . Secondly , one-step generation does not provide an interpretable compositional and causal process describing how a character is generated . In this work , we combine the advantages of two groups of aforementioned models with an agent operating on stroke representation of images that uses only raster images during training . Thus , we aim to solve all three generative and the parsing ( reconstruction ) tasks of the Omniglot challenge . We show that the model trained for reconstruction can also be adopted as a tool that captures the compositional structure of a given character . Without any further training , our agent can solve exemplar generation and type conditioned novel concept generation problems . Image Generation by Vectorization — With Stroke Supervision Sketch-RNN ( Ha & Eck , 2017 ) is the first LSTM/VAE based sketch generation algorithm . It is later improved to generate multiclass samples ( Cao et al. , 2019 ) and increase the quality of generations by representing strokes as Bezier curves ( Song , 2020 ) . The idea of obtaining a generalizable latent space by imagestroke mapping is studied by many ( Aksan et al. , 2020 ; Das et al. , 2021 ; Bhunia et al. , 2021 ; Wang et al. , 2020 ) . In CoSE ( Aksan et al. , 2020 ) , the problem is articulated as ‘ completion of partially drawn sketch ’ . They achieved state of the art reconstruction performance by utilizing variable-length strokes and a novel relational model that is able to capture the global structure of the sketch . The progress in stroke representation is continued with incorporation of variable-degree Bezier curves ( Das et al. , 2021 ) , and capturing Gestalt structure of partially occluded sketches ( Lin et al. , 2020 ) . Self Supervised Vectorization Self-supervised vector-based image generation problem has been approached by RL based frameworks ( Zhou et al. , 2018 ) , ( Ganin et al. , 2018 ) , ( Mellor et al. , 2019 ) , ( Huang et al. , 2019 ) , ( Schaldenbrand & Oh , 2020 ) , and ( Zou et al. , 2020 ) . In SPIRAL ( Ganin et al. , 2018 ) , unconditional generation and reconstruction tasks are tackled with adversarially trained RL agents . Succeeding research enhanced the reconstruction process by a differentiable renderer , making it possible for agents to operate on a continuous space ( Huang et al. , 2019 ; Schaldenbrand & Oh , 2020 ) . In order to avert the computational expense of RL based algorithms , end-to-end differentiable models are developed through altering the rendering process ( Nakano , 2019 ) or formulating the generation process as a parameter search ( Zou et al. , 2020 ) . More recently , a differentiable renderer and compositor is utilized for generating closed Bezier paths and the final image respectively ( Reddy et al. , 2021 ) . This method led to successful interpolation , reconstruction , and sampling processes . Most related to our work is SPIRAL where both reconstruction and unconditional generation is studied through self-supervised deep reinforcement learning . However , our approach has some significant differences . First , in SPIRAL each stroke is also represented as a Bezier curve , yet , the starting point of each curve is set as the final point of the previous curve . In our model , all control points of the Bezier curve are predicted by the agent at each time step . Hence , the agent has to learn the continuity and the compositionality of the given character in order to produce quality samples . Secondly , SPIRAL provides a generative model that works through a graphics renderer without addressing the conditional generation problem . They show impressive results on both natural images and handwritten characters . While we provide a solution for multiple generative tasks , we have not explored our model in the context of natural images . Another approach that presents a similar scheme to the reconstruction problem is “ Learning to Paint ” ( Huang et al. , 2019 ) . In Learning to Paint , the proposed model is utilized specifically for reconstruction . When reconstruction is considered , the main difference of our model is that since we try to model a human-like generation process , our agent outputs a single stroke at each time step with the environment being altered throughout this process while in Learning to Paint , 5 strokes are predicted by the agent at each time step . As a major difference from previous studies , our agent decides whether to stop or keep drawing before generating a stroke . This enables the agent to synthesize an image with as few actions as possible when motivated with our reward formulations . Self Supervised Program Synthesis Our method essentially outputs a visual program that depends only on the rastered data . In that sense , studies on Constructive Solid Geometry ( CSG ) are also related . Different RL frameworks for reconstruction of a given CSG image , that is essentially a composition of geometric shapes , are proposed ( Ellis et al. , 2019 ; Zhou et al. , 2020 ) . The former considered parsing as a search problem that is solved by using a read-eval-print-loop within a Markov Decision Process . The latter adopted a Tree-LSTM model to eliminate invalid programs and the reward is considered to be the Chamfer distance between the target image and current canvas . 3 METHOD Our model consists of a policy network and a ( nondifferentiable ) renderer . At time step t , the policy network takes the current canvas , Ct – a raster-image , as input and outputs two distributions , πB and πS . The first distribution , πB , is for stroke ( i.e . Bezier curve ) parameters and the second one , πS , is for the continue/stop decision . From the first distribution , we randomly sample a stroke defined by its 7 parameters ( x-y coordinates of start , end , control points of the quadratic Bezier curve , and a brush-width ) . From the second distribution , we randomly sample a decision . If the decision happens to be ‘ continue ’ , we add the newly sampled stroke to the current canvas , Ct , increment time ( i.e . t ← t + 1 ) and restart . If the decision was to ‘ stop ’ , then Ct is returned as the final output . Our model is able to handle parsing and different generation tasks , and the processing pipeline we just described is common in all these tasks . What changes among tasks is the reward functions and/or training procedures , which we explain below . Unconditional Generation The task of ‘ generating new concepts ’ as dubbed in Omniglot challenge , is essentially unconditional sampling from a distribution obtained from the whole Omniglot training set . Here , the model is asked to generate completely novel samples ( i.e . characters ) without any constraints . For this task , at each time step t , we calculate an instantaneous reward , rt , that has three components : rt = D ( Ct ) + λ1align ( Ct , I ) + λ2N ( |Ct| ; µ , σ ) . ( 1 ) The first term is a reward based on a discriminator to make generated characters as ‘ real ’ as possible . D ( · ) is a discriminator that outputs the “ realness ” score of its input canvas . We train it in an adversarial manner by using the generated examples as negatives and the elements of the input dataset as positives . The second term is a clustering-based data fidelity reward . The function align ( Ct , I ) measures the alignment between the current canvas Ct and another canvas I , which is a randomly selected cluster center at the beginning of each episode . The cluster centers are obtained by applying k-means on all characters in the input dataset . align basically counts the number of intersecting on-pixels ( between the two canvases ) minus the number of non-intersecting on-pixels in Ct , and divides this quantity by the number of on-pixels in I . The final term assesses the conformity of the current canvas with the dataset in terms of the number of on-pixels . N ( |Ct| ; µ , σ ) evaluates a normal distribution with ( µ , σ ) at |Ct| which is the number of on-pixels in the current canvas . We obtain ( µ , σ ) by fitting a normal distribution to the on-pixel counts of characters in the training set . We observed that the second and third terms accelerate learning as they guide the exploration within the vicinity of real characters . During training , instead of using the instantaneous reward , rt , we use the difference of successive rewards , i.e . rt − rt−1 . In order to encourage exploration and avoid mode collapse , we use an entropy penalty term as αmax ( 0 , KL ( [ πB , πS ] , U ) − τ ) . ( 2 ) Here , KL indicates KL-divergence and U is the uniform distribution . This term first measures the divergence between the uniform distribution and πB , πS , the distributions output by the policy network . Then , through the hinge function , if the divergence exceeds a threshold ( τ ) , this term activates and increases the penalty . The policy network and the discriminator D are updated alternatingly after 256 images are generated at each iteration . We employ the REINFORCE algorithm ( Williams , 1992 ) to update the weights of the policy network . Discriminator is trained using hinge loss . In order to stabilize the discriminator and keep the Lipschitz constant for the whole network equal to 1 , Spectral Normalization is applied at each layer ( Miyato et al. , 2018 ) . Throughout the training , we kept the balance ratio between generated and real samples at 3 . Image Reconstruction by Parsing In the “ parsing ” task , the goal is to reconstruct the given input image by re-drawing it through strokes as accurately as possible . To this end , we formulate a new reward function with two terms : a fidelity reward that indicates how much of a stroke is consistent with the input image ( using the “ align ” function introduced above ) and a penalty that is added with every time increment represented by t as ‘ continue ’ decisions being made : rt = align ( St , Ct ) − λ1t , ( 3 ) where St is the newly sampled stroke and Ct is the current canvas ( input ) . Second term simply acts as a penalty for every ‘ continue ’ action . The first term ensures the sampled stroke to be well-aligned with the input and the second term forces the model to use as few strokes as possible . There is no need for a discriminator . This model explicitly learns the vectorization of the input raster-image in a self-supervised manner . Apart from the different reward function , another crucial difference between the training of the unconditional generation model and the parsing model is how the input and output are handled . In unconditional generation , the newly-sampled stroke is added to the current canvas , whereas in parsing , we do the opposite : the sampled stroke is removed ( masked out ) from the current canvas , and the returned final canvas is the combination of all sampled strokes until the ‘ stop ’ decision . λ , α and τ in Equations 1 , 2 , and 3 are hyperparameters adjusted experimentally . ( see ‘ Training Details ’ in Appendix B ) . Generating New Exemplars In this task , a model is required to generate a new exemplar ( i.e . a variation ) of an unseen concept ( i.e . character ) . To the best of our knowledge , we are the first to tackle this task in a self-supervised stroke-based setting . Most importantly , we do not require any training to achieve this task . We utilize our parsing network described in the previous section to capture the overall structure of a given letter . In order to produce new exemplars , we randomly sample different parsings ( a set of strokes ) from the distribution generated by the agent . In order to eliminate ‘ unlikely ’ samples , we compute the likelihood of the parsing given the resulting policy , and apply a threshold . Generating Novel Concepts from Type In this task , the goal is to to generate a novel concept ( i.e . character ) given a previously unseen type ( i.e . alphabet ) consisting of 10 concepts . The novel concepts should conform to the overall structure , that is , the stroke formulation and composition of the given type ( alphabet ) . We , again , tackle this challenge using our parsing network without any further training . To do so , we first parse all input images into its strokes . For each input image , we sample five stroke sets from the stroke-parameters distribution output by the policy network . During the sampling process , we again use the likelihood-based quality function described in the previous section . We add all the strokes sampled during this process to form a stroke library . Here the strokes are stored with the time steps they are generated . Noting that the number of strokes sampled for a given character is not constant , we approximate a distribution for stopping actions . This process provides a stroke set representing the structure of letters and the way they are composed , that is , we can exploit the compositionality and causality of an alphabet . Throughout the character generation process , a stroke is sampled at each time step belonging to that particular group of the library . The sampled strokes are summed together to obtain the final canvas .

Summary: This paper presents an approach using reinforcement learning to parse and generate characters. Experiments are performed both with the omniglot challenge and MNIST datasets. In the context of omniglot, in addition to unconditional generation and parsing, results are also shown on the exemplar generation and type conditioned generation tasks. The proposed approach only uses pixel/raster level information during training and does not use stroke or vector data.