Creative Sketch Generation

1 INTRODUCTION . The true sign of intelligence is not knowledge but imagination . – Albert Einstein From serving as a communication tool since prehistoric times to its growing prevalence with ubiquitous touch-screen devices – sketches are an indispensable visual modality . Sketching is often used during brainstorming to help the creative process , and is a popular creative activity in itself . Sketch-related AI so far has primarily focused on mimicking the human ability to perceive rich visual information from simple line drawings ( Yu et al. , 2015 ; Li et al. , 2018 ) and to generate minimal depictions that capture the salient aspects of our visual world ( Ha & Eck , 2018 ; Isola et al. , 2017 ) . Most existing datasets contain sketches drawn by humans to realistically mimic common objects ( Eitz et al. , 2012 ; Sangkloy et al. , 2016 ; Jongejan et al. , 2016 ; Wang et al. , 2019 ) . In this work we focus on creative sketches . AI systems that can generate and interpret creative sketches can inspire , enhance or augment the human creative process or final artifact . Concrete scenarios include automatically generating an initial sketch that a user can build on , proposing the next set of strokes or completions based on partial sketches drawn by a user , presenting the user with possible interpretations of the sketch that may inspire further ideas , etc . ∗The work was done when the first author interned at Facebook AI Research . AI for creative sketches is challenging . They are diverse and complex . They are unusual depictions of visual concepts while simultaneously being recognizable . They have subjective interpretations like aesthetics and style , and are semantically rich – often conveying a story or emotions . To facilitate progress in AI-assisted creative sketching , we collect two datasets – Creative Birds and Creative Creatures ( Figure 1 ) – containing 10k creative sketches of birds and generic creatures respectively , along with part annotations ( Figure 2 right columns ) . To engage subjects in a creative exercise during data collection , we take inspiration from a process doodling artists often follow . We setup a sketching interface where subjects are asked to draw an eye arbitrarily around a random initial stroke generated by the interface . Subjects are then asked to imagine a bird or generic creature that incorporates the eye and initial stroke , and draw it one part at a time . Figure 2 shows example sketches from our datasets . Notice the larger diversity and creativity of birds in our dataset than those from existing datasets with more canonical and mundane birds . We focus on creative sketch generation . Generating novel artifacts is key to creativity . To this end we propose DoodlerGAN – a part-based Generative Adversarial Network ( GAN ) that generates novel part appearances and composes them in previously unseen configurations . During inference , the model automatically determines the appropriate order of parts to generate . This makes the model well suited for human-in-the-loop interactive interfaces where it can make suggestions based on user drawn partial sketches . Quantitative evaluation and human studies show that our approach generates more creative and higher quality sketches than existing approaches . In fact , subjects prefer sketches generated by DoodlerGAN over human sketches from the Creative Birds dataset ! Our datasets , code , and a web demo are publicly available 1 . 2 RELATED WORK . Sketches have been studied extensively as a visual modality that is expressive yet minimal . The sparsity of sketches compared to natural images has inspired novel modelling techniques . We discuss existing sketch datasets and sketch generation approaches . Other related work includes sketch recognition ( Yu et al. , 2015 ; Li et al. , 2018 ) , sketch-based image retrieval ( Yu et al. , 2016 ; Liu et al. , 2017 ; Ribeiro et al. , 2020 ) and generation ( Gao et al. , 2020 ; Lu et al. , 2018 ; Park et al. , 2019 ) . An overview of deep learning approaches for sketches can be found in this survey ( Xu et al. , 2020 ) Sketch datasets . Existing sketch datasets such as TU-Berlin ( Eitz et al. , 2012 ) , Sketchy ( Sangkloy et al. , 2016 ) , ImageNet-Sketch ( Wang et al. , 2019 ) and QuickDraw ( Jongejan et al. , 2016 ) are typically focused on realistic and canonical depictions of everyday objects . For instance , sketches in the Sketchy dataset ( Sangkloy et al. , 2016 ) were drawn by humans mimicking a natural image . Sketches in the QuickDraw dataset ( Jongejan et al. , 2016 ) were collected in a Pictionary-like game setting – they were drawn under 20 seconds to be easily recognized as a target object category . This is in stark contrast with how people engage in doodling as a creative activity , where they take their time , engage their imagination , and draw previously unseen depictions of visual concepts . These depictions may be quite unrealistic – including exaggerations or combinations of multiple categories . Our datasets contain such creative sketches of birds and generic creatures . See Figures 1 and 2 . Our data collection protocol was explicitly designed to engage users in a creative process . Also note that while not the focus of this paper , our datasets are a valuable resource for sketch segmentation approaches . See Section A in the Appendix for further discussion . 1songweige.github.io/projects/creative_sketech_generation/home.html Sketch generation . Earlier studies generated a sketch from an image via an image-to-image translation approach ( Isola et al. , 2017 ; Song et al. , 2018 ; Li et al. , 2019b ) . Subsequently , fueled by the release of large-scale free-hand sketch datasets , methods that generate human-like sketches from scratch have interested many researchers . One of the early works in this direction was SketchRNN ( Ha & Eck , 2018 ) – a sequence-to-sequence Variational Autoencoder ( VAE ) that models the temporal dependence among strokes in a sketch . Later approaches ( Chen et al. , 2017 ; Cao et al. , 2019 ) incorporated a convolutional encoder to capture the spatial layout of sketches . Reinforcement learning has also been studied to model the sequential decision making in sketch generation ( Zhou et al. , 2018 ; Huang et al. , 2019 ; Mellor et al. , 2019 ) . In contrast with generating entire sketches , sketch completion approaches generate missing parts given a partial sketch . SketchGAN ( Liu et al. , 2019 ) adopted a conditional GAN model as the backbone to generate the missing part . Inspired by the large-scale language pretraining approaches ( Devlin et al. , 2019 ) , SketchBERT ( Lin et al. , 2020 ) learns representations that capture the “ sketch gestalt ” . We propose a part-based approach , DoodlerGAN , for sketch generation . Compared to previous approaches ( Ha & Eck , 2018 ; Cao et al. , 2019 ) that learn to mimic how humans draw , our goal is to create sketches that were not seen in human drawings . The idea of generating different components as parts of the overall sketch can be traced back to compositional models ( Chen et al. , 2006 ; Xu et al. , 2008 ) . We explore these ideas in the context of deep generative models for sketches . Also relevant are approaches that exploit spatial structure when generating natural images – foreground vs. background ( Yang et al. , 2017 ) or objects and their spatial relationships ( Johnson et al. , 2018 ) . A discussion on creative tools based on sketch generation can be found in Section B the Appendix . 3 CREATIVE SKETCHES DATASETS . To facilitate progress at the intersection of machine learning and artificial creativity in the context of sketches , we collected two datasets : Creative Birds and Creative Creatures . The first is focused on birds , and the second is more diverse and challenging containing a variety of creatures . 3.1 DATASET COLLECTION . Both datasets were collected on Amazon Mechanical Turk using a sketching web interface2 . In order to engage subjects in a creative exercise , and encourage diversity across sketches , the interface generates a random initial stroke formed by connecting K keypoints on the canvas via Bezier curves . These points were picked via heuristics such that the strokes are smooth with similar lengths and have limited self occlusion ( see Figure 9 in the Appendix ) . Next , inspired by a process doodling artists ( e.g. , Michal Levy ) use , subjects were told to add an eye to the canvas wherever they like . They were then asked to step back , and visualize how the initial stroke and the eye can be incorporated in a creative sketch of a bird or arbitrary creature ( depending on the dataset ) . Subjects were then asked to draw one part of the bird or creature at a time , indicating via a drop down menu which part they were drawing . Each part can be drawn using multiple strokes . For Creative Birds , the options include 7 common parts of birds ( head , body , beak , tail , mouth , legs , wings ) . For Creative Creatures , the 16 parts ( e.g. , paws , horn , fins , wings ) cover terrestrial , aquatic , and aerial creatures . In our pilot studies we found that providing subjects this structure of drawing one part at a time increased the quality of sketches , and giving subjects the option to pick parts themselves made the exercise more natural and increased the diversity of sketches . As a by product , each stroke has a corresponding part annotation ( see Figure 2 ) , which can be a rich resource for sketch segmentation and part-based sketch recognition or generation models . Subjects are required to add at least five parts to the sketch ( in addition to the eye and initial stroke ) before they can submit the sketch . Once done with adding parts , subjects were given the option to add additional details to the sketch . Finally , they were asked to add a free-form natural language phrase to title their sketch to make it more expressive . In this work we do not use the details and phrases , but they may be useful in the future for more detailed sketch generation and automatic creative sketch interpretation . Except for Figure 1 , all example sketches shown in the paper are without the details unless noted otherwise . In addition to contributing to the creativity and diversity of sketches , the initial random strokes also add constraints , making sketch creation more challenging . Subjects have more constraints 2Our interface can be seen at https : //streamable.com/jt4sw1 when asked to draw only birds in Creative Birds , so we give them less starting constraints ( K = 3 keypoints in the initial strokes ) , but for Creative Creatures subjects have more freedom in the animals they draw so we provide more constraints ( K = 6 ) . Example initial strokes for both datasets and some insights from pilot data collection can be found in Figure 9 and Section C in the Appendix . 3.2 DATA ANALYSIS . We collected 10k sketches for both datasets . Filtering out sketches from workers who did not follow the instructions well , we have 8067 and 9097 sketches in Creative Birds and Creative Creatures respectively . See Figure 2 for random examples of sketches from both datasets . For comparison , we also show birds from other existing datasets . Notice that the sketches in our datasets are more creative , and sketches from Creative Creatures are more diverse and complex than those in Creative Birds . We conducted a human study comparing 100 sketches from Creative Birds to 100 from QuickDraw across 5 subjects each . Our sketches were rated as more creative 67 % of the time . More example sketches from our datasets can be found in Figures 12 and 13 in the Appendix . In Figure 14 in the Appendix we also show examples where different sketches incorporate similar initial strokes , suggesting room for creativity and variation that a generative model could learn . Sketches of individual parts are shown in Figure 10 and 11 in the Appendix . Analysis of the part annotations , including the order in which parts tend to be drawn , is provided in Section D in the Appendix .

Summary: The contribution of this paper is twofold. First, the authors introduce a pair of manually collected (via AMT) datasets of creative sketches (birds and general creatures) each along with part annotations. Second, they propose a part-based Generative Adversarial Network for the generation of unseen compositions/configurations of novel parts (legs, body, etc.) for creative sketches. The latter is quantitatively and qualitatively (via human studies) evaluated.