Creative Sketch Generation
Authors: Songwei Ge
Vedanuj Goswami
Larry Zitnick
Devi Parikh
Venue: ICLR 2021 Poster (2021)
Keywords: creativity
sketches
part-based
GAN
dataset
generative art

Abstract: Sketching or doodling is a popular creative activity that people engage in. However, most existing work in automatic sketch understanding or generation has focused on sketches that are quite mundane. In this work, we introduce two datasets of creative sketches -- Creative Birds and Creative Creatures -- containing 10k sketches each along with part annotations. We propose DoodlerGAN -- a part-based Generative Adversarial Network (GAN) -- to generate unseen compositions of novel part appearances. Quantitative evaluations as well as human studies demonstrate that sketches generated by our approach are more creative and of higher quality than existing approaches. In fact, in Creative Birds, subjects prefer sketches generated by DoodlerGAN over those drawn by humans!

Published as a conference paper at ICLR 2021
CREATIVE SKETCH GENERATION
Songwei Ge
Vedanuj Goswami & C. Lawrence Zitnick Facebook Al Research {vedanuj, satnickkeff eff.coofooooooooooooo com
University of Maryland, College Park songweige umd. edu
Devi Parikh
Facebook AI Research
Georgia Institute of Technology parikh@ga tech. edu
ABSTRACT
Sketching or doodling is popular creative activity that people engage in. How- ever, most existing work in automatic sketch understanding or generation has focused on sketches that are quite mundane. In this work, we introduce two datasets of creative sketches - Creative Birds and Creative Creatures contain- ing 10k sketches each along with part annotations. We propose DoodlerGAN a part-based Generative Adversarial Network (GAN) to generate unseen com- positions of novel part appearances. Quantitative evaluations as well human studies demonstrate that sketches generated by our approach are more creative and of higher quality than existing approaches. In fact, in Creative Birds, subjects prefer sketches generated by DoodlerGAN over those drawn by humans!
INTRODUCTION
The true sign of intelligence is not knowledge but imagination. Albert Einstein
From serving as a communication tool since prehistoric times to its growing prevalence with ubiq- uitous touch-screen devices sketches an indispensable visual modality. Sketching often used during brainstorming to help the creative process, and a popular creative activity in itself.
Sketch-related Al far has primarily focused mimicking the human ability to perceive rich visual information from simple line drawings (Yu et al. 20155] Li et al.) 2018) and to generate min- imal depictions that capture the salient aspects of our visual world (Ha & Eck 2018 Isola et al.) 2017). Most existing datasets contain sketches drawn by humans to realistically mimic common
objects (Fitz al. 2012, (Sangkloy al. 2016 Jongejan et al. 2016 Wang et 2019).
Jannn
Figure 1: Cherry-picked example sketches from our proposed datasets: Creative Birds (left) and Creative Creatures (right). See random examples in Figure and Figures 12 and 13 in the Appendix.
In this work we focus on creative sketches. Al systems that can generate and interpret creative sketches can inspire, enhance or augment the human creative process final artifact. Concrete scenarios include automatically generating an initial sketch that a user can build on, proposing the next set of strokes completions based partial sketches drawn by user, presenting the user with possible interpretations the sketch that may inspire further ideas, etc.
'The work done when the first author interned at Facebook AT Research.

---

Published as a conference paper at ICLR 2021
QuickDraw Bird Bongee-jn at al, 2016)
TU-Berlin Bird (Eit et 20.2)
Sketchy Bird (Sanghloy et al, 2016)
Creative Birds
Creative Birds pantlabees)
Creative Creatures (wi partiabels)
Figure 2: Random sketches from existing datasets (left) and our creative sketches datasets (right).
AI for creative sketches is challenging. They are diverse and complex. They are unusual depictions of visual concepts while simultaneously being recognizable. They have subjective interpretations like aesthetics and style, and are semantically rich - often conveying a story or emotions.
To facilitate progress in Al-assisted creative sketching. we collect two datasets Creative Birds and Creative Creatures (Figure ! containing 10k creative sketches of birds and generic creatures respectively, along with part annotations (Figure columns). To engage subjects in a creative exercise during data collection, we take inspiration from process doodling artists often follow. We setup a sketching interface where subjects are asked to draw an eye arbitrarily around a random initial stroke generated by the interface. Subjects then asked imagine bird or generic creature that incorporates the eye and initial stroke, and draw it one part at time. Figure shows example sketches from our datasets. Notice the larger diversity and creativity of birds in dataset than those from existing datasets with more canonical and mundane birds.
We focus creative sketch generation. Generating novel artifacts is key creativity. To this end propose DoodlerGAN - a part-based Generative Adversarial Network (GAN) that generates novel part appearances and composes them in previously unseen configurations. During inference, the model automatically determines the appropriate order of parts to generate. This makes the model well suited for human-in-the-logg, interactive interfaces where it can make suggestions based on user drawn partial sketches. Quantitative evaluation and human studies show that our approach generates more creative and higher quality sketches than existing approaches. In fact, subjects prefer sketches generated by DoodlerGAN over human sketches from the Creative Birds dataset!
Our datasets, code, and a web demo are publicly available
2 RELATED WORK
Sketches have been studied extensively as a visual modality that is expressive yet minimal. The sparsity of sketches compared to natural images has inspired novel modelling techniques. We dis- cuss existing sketch datasets and sketch generation approaches. Other related work includes sketch recognition (Yu al. 2015 Li et al. (2018). sketch-based image retrieval (Yu al.) 2016 Liu al.]
2017 Ribeiro et al.) 2020) and generation (Gao et al.] 2020 Lu et al., 2018 Park et al.
2019). An
overview of deep learning approaches for sketches can be found in this survey (Xu et al.
2020)
Sketch datasets. Existing sketch datasets such as TU-Berlin (Eitz et al., 2012), Sketchy (Sangk- loy et al. 2016), ImageNet-Sketch (Wang et al. (211)) and QuickDraw (Jongejan et al. (2016) are typically focused on realistic and canonical depictions of everyday objects. For instance, sketches in the Sketchy dataset (Sangkloy et al. 2016) were drawn by humans mimicking a natural image. Sketches in the QuickDraw dataset (Jongejan et al. 2016)
collected in Pictionary- like game setting- they were drawn under 20 seconds to be casily recognized as a target object category. This in stark contrast with how people engage in doodling time, engage their imagination, and draw previously unseen depictions visual concepts. These de- pictions may be quite unrealistic- including exaggerations or combinations of multiple categories. Our datasets contain such creative sketches of birds and generic creatures. See Figures [ Our data collection protocol was explicitly designed to engage users in creative process. Also note that while not the focus of this paper, our datasets are valuable resource for sketch segmentation approaches. See Section A in the Appendix for further discussion.
creative activity, where they take their
songwei ge. git.b.. htt.://wwwpppppeeee..ereeaaeeeer
2

---

Published as a conference paper at ICLR 2021
Sketch generation. Earlier studies generated sketch from an image via image-to-ings to-inage trans- lation approach (Isola et al.] 2017; Song et al.) 2018 Li et al. 2019b). Subsequently, fueled by the release of large-scale hand sketch datasets, methods that generate like sketches from scratch have interested many researchers. One of the early works in this direction was SketchRNN (Ha & Eck 2018)- -a a sequence-to-sequenee Variational Autoencoder (VAE) that mod- els the temporal dependence among strokes in sketch. Later approaches (Chen et al., 2017 Cao al. 2019) incorporated convolutional encoder to capture the spatial layout of sketches. Rein- forcement learning has also been studied to model the sequential decision making in sketch gen- eration (Zhou et al.) 2018: Huang et al. 2019 Mellor et al.] (019). In contrast with generating entire sketches, sketch completion approaches generate missing parts given partial sketch. Sketch- GAN (Liu et al. 2019) adopted a conditional GAN model as the backbone to generate the missing part. Inspired by the large-scale language pretraining approaches (Devlin et al. 2019), Sketch- BERT [Lnn et al.] 2020) learns representations that capture the "sketch gestalt". We propose a part-based approach, DoodlerGAN, for sketch generation. Compared to previous approaches (Ha & Cao et al., (019) that learn to mimic how humans draw, our goal is to create sketches that were not seen in human drawings. The idea of generating different components parts of the overall sketch can be traced back to compositional models (Chen et al. 2006] Xu et al. 2008). We explore these ideas in the context of deep generative models for sketches. Also relevant are approaches that exploit spatial structure when generating natural images foreground vs. background (Yang et al., 2017) or objects and their spatial relationships (Johnson et al. 2018). A discussion on creative tools based on sketch generation can be found in Section B the Appendix.
Eck 2018
3 CREATIVE SKETCHES DATASETS
To facilitate progress at the intersection of machine learning and artificial creativity in the context of sketches, we collected two datasets: Creative Birds and Creative Creatures. The first is focused on birds, and the second is more diverse and challenging containing a variety of creatures.
3.1 DATASET COLLECTION
Both datasets were collected on Amazon Mechanical Turk using a sketching web interface In order to engage subjects in creative exercise, and encourage diversity across sketches, the inter- face generates a random initial stroke formed by connecting K keypoints on the canvas via Bezier curves. These points were picked via heuristics such that the strokes are smooth with similar lengths and have limited self occlusion (see Figure 9 in the Appendix). Next, inspired by a process doo- told to add eye to the canvas wherever they
dling artists (e.g.. Michal Levy) use, subjects like. They were then asked to step back, and visualize how the initial stroke and the eye can be incorporated in creative sketch of bird or arbitrary creature (depending on the dataset).
Subjects were then asked to draw one part of the bird or creature at a time, indicating via a drop down menu which part they were drawing. Each part can be drawn using multiple strokes. For Creative Birds, the options include 7 common parts of birds (head, body, beak, tail, mouth, legs, wings). For Creative Creatures, the 16 parts (e.g., paws, horn, fins, wings) cover terrestrial, aquatic, and acrial creatures. In our pilot studies we found that providing subjects this structure of drawing one part at time increased the quality of sketches, and giving subjects the option to pick parts themselves made the exercise more natural and increased the diversity of sketches. As by product, each stroke has corresponding part annotation (see Figure 2. which can be rich resource for sketch segmentation and part-based sketch recognition or generation models. Subjects required to add at least five parts to the sketch (in addition to the eye and initial stroke) before they can submit the sketch.
Once done with adding parts, subjects were given the option to add additional details to the sketch. Finally, they were asked to add a free-form natural language phrase to title their sketch to make more expressive. In this work we do not use the details and phrases, but they may be useful in the future for more detailed sketch generation and automatic creative sketch interpretation. Except for Figure I all example sketches shown in the paper without the details unless noted otherwise.
In addition to contributing to the creativity and diversity of sketches, the initial random strokes also add constraints, making sketch creation more challenging. Subjects have more constraints
"Our interface can he at wthttps://streeaeee e com/ jt4w/]
3

---

Published as a conference paper at ICLR 2021
Part selector:
part
Crass Entropy
Sye selected
generotor (oys)
selector
x-N(0.11
Predicted part
stroke
Input partial sieatch
Classifier
Intermediate pertal
Partt soloctor
Fieed selected
k=
Part gensrator
Ihcod)
Part generator:
Intermociate partial skatch
(-1)(0.))
(oeak)
GAN
Finel skeech
Input partial sketch
Encoder
StyleGAN2 generator
Discriminator
Interediatie portial sketch N
Completion aignt
selector
(a) Training Phase
(b) Inference Phase
Figure 3: DoodlerGAN. (a) During training, given an input partial sketch represented as stacked part channels, a part selector is trained to predict the next part to be generated, and a part generator trained for each part to generate that part. (b) During inference, starting from a random initial stroke, the part selector and part generators work iteratively to complete the sketch.
when asked to draw only birds in Creative Birds, so we give them less starting constraints (K keypoints in the initial strokes), but for Creative Creatures subjects have more freedom in the animals they draw so we provide more constraints (K 6). Example initial strokes for both datasets and some insights from pilot data collection be found in Figure and Section C in the Appendix.
3.2 DATA ANALYSIS
We collected 10k sketches for both datasets. Filtering out sketches from workers who did not follow the instructions well, we have 8067 and 9097 sketches in Creative Birds and Creative Creatures respectively. See Figure 2 for random examples of sketches from both datasets. For comparison, we also show birds from other existing datasets. Notice that the sketches in our datasets are more creative, and sketches from Creative Creatures are more diverse and complex than those in Creative Birds. We conducted a human study comparing 100 sketches from Creative Birds to 100 from QuickDraw across 5 subjects each. Our sketches were rated as more creative 67% of the time. More example sketches from our datasets can be found in Figures 12 and 13 in the Appendix. In Figure[14[in the Appendix also show examples where different sketches incorporate similar initial strokes, suggesting room for creativity and variation that a generative model could learn. Sketches of individual parts shown in Figure 10 11 in the Appendix. Analysis of the part annotations, including the order in which parts tend to be drawn, is provided in Section Din the Appendix.
4 DOODLERGAN; A PART-BASED SKETCH GENERATION MODEL
Objects are typically configuration of parts (e.g., animals have two four legs, mouth below two eyes). Moreover, humans often sketch by drawing one part at a time. We approach creative sketch generation by generating novel appearances of parts and composing them in previously unseen con- figurations. Another benefit of parts is that creative sketches exhibit large diversity in appearance, but this complexity is significantly reduced if the sketches are decomposed into individual parts.
Our approach, DoodlerGAN, is bast- GAN that sequentially generates one part at a time, while ensuring each step that the appearance of the parts and partial sketches comes from corre- sponding distributions observed in human sketches. Humans do not draw parts in the same order across sketches, but patterns exist. To mimic this, and to adapt well to a (future) human-in-the-loo, setting, DoodlerGAN automatically determines the order of parts. Concretely, DoodlerGAN con- tains two modules: the part generator and the part selector, as shown in Figure Given part-based representation of partial sketch, the part selector predicts which part category to draw next. Given a part-based representation of partial sketch and part category, the part generator generates a raster image of the part (which represents both the appearance and location of the part).
4

---

Published as a conference paper at ICLR 2021
4.1 ARCHITECTURE
We represent sketches as raster images (i.e., grids of pixels) as opposed to vector representations of strokes. This makes detailed spatial information readily available, which helps the model better assess where a part connects with the rest of the sketch. A vector representation captures the low- level sequence, which may not be relevant to the overall quality of the generated sketch (e.g.. a circular head will look the same whether it is drawn clockwise or counter-clockwise). Our part- based approach models the sequence at an appropriate level of abstraction (parts).
The part generator is a conditional GAN based on the StyleGAN2 architecture (Karras et al. 2019 2020). We generate sketches at a 64 x 64 resolution. We use a 5-layer StyleGAN2 generator with [512, 256, 128, 64, 32) output channels and starting from x 4 x 64 constant feature map. To encode the input partial sketch, we use a 5-layer CNN with [16, 32, 64, 128, 256] output channels. Each layer contains two convolutional layers with 3 x 3 kernels followed by a LeakyRelu activation with a negative slope 0.2. Inspired by the design of U-Net (Ronneberger et al. 20155,, we downsample the intermediate feature map after each encoder layer and concatenate it channel-wise with the corresponding layers in the generator. See dashed lines in Figure 3a This gives the generator access to the hierarchical spatial structure in the input sketch.
The input partial sketch is represented by stacking each part as a channel, along with an additional channel for the entire partial sketch. If a part has not been drawn, the corresponding channel is zero image. We find a part-based representation to be crucial for predicting appropriate part locations. For instance, without the part channels, the generated legs were often not connected the body.
Following StyleGAN2, we borrow the discriminator architecture from (Karras et al. 2018). We give the discriminator access to the input partial sketch and the corresponding part channels, so that the discriminator is able to distinguish fake sketches from real ones not just based on the appearance of the part, but also based on whether the generated part is placed at an appropriate location relative to other parts (e.g., heads are often around the eyes). Specifically, similar to the generator, the input to the discriminator is an image with (number of parts 1) channels. In Figure Bal the difference between the real and fake data fed into the discriminator occurs in the red channel where the fake data contains the generated part (as opposed to the real onc), and the last white channel where the generated (vs. real) part is combined with the input partial sketch using max operation.
The generator and discriminator parameters are not shared across parts. In our experiments, a unified model across parts failed to capture the details in parts such as wings. We also experimented with fine tuning the model end-to-end but generally observed inferior generation quality.
For the part selector, use the same architecture the encoder but with linear layer added at the end to produce logits for different parts. The part selector is also expected to decide when the sketch is complete and no further parts need to be generated. Therefore, the output dimension of the linear layer is set to (number of parts + 1). We convert the human sketches in our dataset to (partial sketch, next-part) pairs and use that train the part selector in supervised way.
4.2 TRAINING AND INFERENCE
With strong conditional information, the discriminator in the part generator gets easily optimized and consequently provides zero gradient for the generator. As a result, the generator gets stuck in the early stages of training and only generates the zero image afterwards. We introduce a sparsity loss as an additional regularizer, which is the L2 norm between the sum of pixels in the generated and real parts, encouraging the model to generate parts of similar sizes as the real parts. This helps stabilize the training and helps the generator learn even when little signal is provided by the discriminator. During training, we also augment the training sketches by applying small affine transformations to the vector sketch images before converting them to raster images. During inference, we follow the same procedure as data collection: we first automatically sample an initial stroke from our interface (see Figure 9 and then use the part selector and part generators iteratively to complete the sketch.
Conditional GANs have been shown to not be sensitive to noise, generating the same sample for different noise vectors (Zhu et al., 2017, Donahue et al.) 2017). We encountered similar issues, especially for parts drawn later stage when more conditional information is available. This is expected; the more complete the partial sketch, the fewer ways there are to reasonably complete it. We found that our model is sensitive to minor perturbations to the input partial sketch. To increase
5

---

Published as a conference paper at ICLR 2021
Initial Stroke
SketchRNN
Uncond.
SketchRNN
Cond.
StyleGAN2
Uncond.
StyleGAN2
Cond.
Percentage
-based
DoodlerGAN
(a) Creative Birds
(b) Creative Creatures
Figure 4: Random generations from each approach. DoodlerGAN generations noticeably higher quality compared to baselines. More DoodlerGAN generations are in Figure in the Appendix.
the diversity in generations, we propose simple trick call conditioning perturbation. We apply random translations sampled from Ao0, 2) to the input partial sketch to generate multiple parts and then translate the generated parts back to align with the original partial sketch.
For further details of training, data augmentation and inference see Section F in the Appendix.
5 RESULTS
We quantitatively and qualitatively (via human studies) evaluate our approach compared to strong baselines and existing state-of-the-ar approaches trained on creative datasets:
StyleGAN2 Unconditional. We train StyleGAN2 using the same hyperparameters and data aug- mentation settings used for DoodlerGAN, to generate the entire sketches in step. To avoid mode collapse, we add the minibatch discriminator layer (Salimans et al. 2016) in its discriminator, This represents a state-of-the-art approach in image generation.
StyleGAN2 Conditional. This approach is the same as the one described above, but we condition the one-step generation on the initial stroke (encoded using the encoder as in DoodlerGAN).
RKN Unconditional. We use the SketchRNN model (Ha & Eckk 2018) trained on our datasets. This represents the state-of-the-art in sketch generation. We optimized the architecture (encoder, decoder and latent space sizes, temperature 7) and used heuristic post processing to elim- inate obvious failure cases adapt this approach the best we could to our datasets.
SKetchRNN Conditional. This approach is SketchRNN described above, except during inference fix the first few points and the pen states based the random initial stroke. These fed as input to continue the sequential generation of the sketch.
Persentage-based. This approach is DoodlerGAN, except instead of using parts, we divide the sketch into 20% chunks (based on the order in which strokes were drawn) and use them as "parts". comparison this approach allows us to demonstrate the effectiveness of semantic parts.
Randomly selected generations from each approach are shown in Figure 4 In StyleGAN2 gener- ations we can see general bird contour but clean details are lacking. In SketchRNN we can see the generated eye but later strokes do not form a coherent sketch. The failure to generate complex sketches with SketchRNN has also been reported in (Ha & Fek] 2018). DoodlerGAN generates no- ticeably higher quality sketches, with different parts of the sketch clearly discernible. More sketches generated by DoodlerGAN can be found in Figure [5
In human evaluation, we also compare bird generations from DoodlerGAN trained on Creative Birds to the numan--daaw bird sketches from QuickDraw to represent the "best" (oracle) generator one can hope to get by training QuickDraw. (A natural equivalent for Creative Creatures is unclear.) In addition to the five generation- based methods, we also compare our method with a strong retrieval- based method in Section in the Appendix.

---

Published as a conference paper at ICLR 2021
To evaluate, generate 10, 000 sketches using all approaches based on randomly sampled set of previously unseen initial strokes (for the conditional upproaches).
5.1 QUANTITATIVE EVALUATION
For quantitative evaluation, consider two metrics used in previous studies: Fréchet inception distances (FID) (Heusel et al. 2017) and generation diversity (GD) (Cao et al. 2019). For this, we trained an Inception model on the QuickDraw3.8N) dataset (Xu et al. 2020). See Section G in the Appendix for more training details. In Table I we see that DoodlerGAN has the best FID score relative to other approaches, while maintaining similar diversity.
Table 1: Quantitative evaluation of DoodlerGAN against baselines on Fréchet inception distances (FID), generation diversity (GD), characteristic score (CS) and semantic diversity score (SDS). The methods with the best scores are in bold.
Creative Birds
Creative Creatures
Methods
FID(L)
GD(t)
CS(t)
FID(L)
GD(()
CS(t)
SDS(t)
Training Data
19.40
0.45
18.06
0.60
1.91
SketchRNN Uncond.
79.57
17.19
0.20
60.56
15.85
0.43
1.22
SketchRNN Cond.
82.17
17.29
0.18
54.12
16.11
0.48
1.34
StyleGAN2 Uncond.
71.05
17.49
0.23
103.24
14.41
0.18
0.72
StyleGAN2 Cond.
130.93
14.45
0.12
56.81
13.96
0.37
1.17
Percentage-based 103.79
15.11
0.20
57.13
13.86
0.41
1.17
DoodlerGAN 39.95
16.33
0.69
43.94
14.57
0.55
1.45
We also introduce two additional metrics. The first is the characteristic score (CS) that checks how often a generated sketch is classified to be a bird (for Creative Birds) creature (for Creative Creatures) by the trained Inception model. The higher the score - the more recognizable a sketch as a bird creature the better the sketch quality. The second is the semantic diversity score (SDS) that captures how diverse the sketches are in terms of the different creature categories they represent (this is more meaningful for the Creative Creatures dataset). Higher better. Note that GD captures a more fine-grained notion of diversity. For instance, if all generated sketches are different dog sketches, GD would still be high but SDS would be low. See Section H in the Appendix for further details about these two metrics. In Table [ we see that DoodlerGAN outperforms existing approaches on both metries. In fact, DoodlerGAN even outperforms the human sketches on the Creative Birds dataset! This trend repeats in human evaluation.
5.2 HUMAN EVALUATION
Automatic evaluation of image generation and creative artifacts are open research questions. There- fore, ran human studies on Amazon Mechanical Turk (AMT). Specifically, pairs of sketches one generated by DoodlerGAN and the other by a competing approach 1 and ask which one (1) is more creative? (2) looks more like bird / creature? (3) they like better? (4) is more likely to be drawn by a human? For the conditional baselines, we also ask (5) in which sketch the initial stroke (displayed in a different color) better integrated with the rest of the sketch? We evaluated 200 random sketches from each approach. Each pair was annotated by 5 unique subjects.
showed subjects
Figure 5 shows the percentage of times DoodlerGAN is preferred over the competing approach. We also plot the Bernoulli confidence intervals (p 0.05). Values outside the band are statistically significantly different than 50% rejecting the null hypothesis that both approaches are equally good. For Creative Birds, DoodlerGAN significantly outperforms the five baselines on all five questions. even beats the real human-drawn sketches, not just from QuickDraw, but also from our Creative Birds dataset most dimensions! Creative Creatures is a more challenging dataset. DoodlerGAN outperforms other approaches but not human sketches. DoodlerGAN not statistically significantly better (or worse) than SketchRNN Conditional in terms of creativity. Note that creativity in itself may not be sufficient an arbitrary pattern of strokes may seem creative but may not be recognized bird creature. The combination of creativity and looking like bird creature is more

---

Published as a conference paper at ICLR 2021
SketchRNN Unconditional
StyleGAN2 Unconditional
Percentage-based QuickDraw Birds
Creative
SketchRNN Conditional
StyleGAN2 Conditional
times DoodlerGAN wins
80
60
40
20
Creative
Like
Human drawn
Initial
Creative
Human drawn stroke integrated
Initial
like bird
stroke integrated
like creature
Figure 5:
Human evaluation results on Creative Birds (left) and Creative Creatures (right). Higher values > DoodlerGAN is preferred over the approach more often.
holistic view, which we believe is better captured by the overall metrie of which sketch subjects like better. DoodlerGAN statistically significantly outperforms SketchRNN that metric. Similarly, subjects can not differentiate between DoodlerGAN and SketchRNN Conditional terms of being more likely be drawn by humans. Note that unlike some other Al tasks (e.g., recognizing image content), an average human need not be the gold standard for creative tasks such as sketching.
5.3 NOVELTY
Figure T6 in the Appendix shows generated sketches and their nearest neighbor training sketches using Chamfer distance. see that the closest training sketches are different, demonstrating that DoodlerGAN is generating previously unseen creative sketches. In fact, by evaluating sketch classification model trained on QuickDraw on our generated Creative Creatures sketches, we identify several sketches that have high response (>0.25) more than one category. We find that model has generated hybrids of a penguin and mouse, a panda and a snake, a cow and a parrot (see Figure(o) Yu & Nickerson, 2011
strong indication of creativity (Boden 1998
Squirrel Sea turtle
Kangaroo Parrot
Frog Penguin
Parrot Penguin
Horse Cat
Shark Bird
Cow
Panda Snake
Raccoon Owl
Penguin Mouse
Bear Sea turtle
Monkey
Parrot
Parrot
Figure 6: DoodlerGAN generates sketches hybrid creatures
In the Appendix, we include several other evaluation studies to demonstrate (1) DoodlerGAN's
ability to generate a diverse set of parts in an interpretable fashion where smoothly varying the input noise one layer changes the part shape and to another layer changes the position (Section (2)
DoodlerGAN's ability to complete sketch better than retrieval-basee the improvements in sketch quality due to the different design choices made in the part generator and selector (Section K Finally, also briefly discuss heuristic method to convert the generated raster images to a vector representation (Section D that enables rendering sketches at arbitrary resolutions. This conversion process adds unintended but delightful aesthetic to the sketches! In human studies we find that subjects prefer this aesthetic to the raw generated images 96% of the times. We also ran all our human studies with this aesthetic and find that similar trends hold.
approach (Section (3)
8

---

Published as a conference paper at ICLR 2021
CONCLUSION
In this work we draw attention to creative sketches. We collect two creative sketch datasets that we hope will encourage future work in creative sketch interpretation, segmentation, and generation. In this paper we focus on the latter, and propose DoodlerGAN part-based GAN model for creative sketch generation. We compare our approach existing approaches via quantitative evaluation and human studies. We find that subjects prefer sketches generated by DoodlerGAN. In fact, for birds, subjects prefer DoodlerGAN's sketches over those drawn by humans! There is significant room for improvement in generating more complex creative sketches (e.g.. Creative Creatures). Future work also involves exploring human-machine collaborative settings for sketching.
ACKNOWLEDGMENTS
The Georgia Tech effort was supported part by AFRL, DARPA, ONR YIP and Amazon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies endorsements, either expressed or implied, of the U.S. Government, or any sponsor.
REFERENCES
Margaret A Boden. Creativity and Artificial Intelligence. Artificial intelligence, 1998.
Nan Cao, Xin Yan, Yang Shi, and Chaoran Chen. Al-Sketcher: A Deep Generative Model for Producing High-quality Sketches. AAAI, 2019.
Hong Chen, Zi Jian Xu, Zi Qiang Liu, and Song Chun Zhu. Composite Templates for Cloth Model- ing and Sketching. In CVPR, 2006.
Yajing Chen, Shikui Tu, Yuqi Yi, and Lei Xu. Sketch-pix2seq: A Model Generate Sketches of Multiple Categories. (70Kv0/779.0440 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL, 2019.
Ernesto Diaz-Aviles.
Dreaming of Electric Sheep. 2018. URL https:/ medium. com/
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial Feature Learning. In ICLR, 2017.
Mathias Eitz, James Hays, and Marc Alexa. How Do Humans Sketch Objects? SIGGRAPH, 2012.
Chengying Gao, Qi Liu, Qi Xu, Limin Wang, Jianzhuang Liu, and Changqing Zou. SketchyCOCO: Image Generation From Freehand Scene Sketches. In CVPR, 2020.
David Ha and Douglas Eck. A Neural Representation of Sketch Drawings. In ICLR, 2018.
Christopher Hesse. edges2cats, 2017. URL htw.hhppps//affeeelleee.ooo/p/pp
html
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NeurlPS, 2017.
Zhewei Huang, Wen Heng, and Shuchang Zhou. Learning to Paint With Model-based Deep Rein- forcement Leamning In CVPR, 2019.
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image Translation with Conditional Adversarial Networks. In CVPR, 2017.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image Generation from Scene Graphs. In CVPR, 2018.

---

Published as a conference paper at ICLR 2021
Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox- Fox-Gieg. Quick, Draw! The Data, 2016. URL htttpt:///uickkrreeeeeeee http///w..c.aa..tttttoooo..o...a com/cata Licensed under (CC-BY 4.0).
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In ICLR, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. In CVPR, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila, Analyz- ing and Improving the Image Quality of StyleGAN. In CVPR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Google Creative Lab. AutoDraw 2017. URL httpp::/eexoerinntttt....tooooo autodraw
Ke Li, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, and Honggang Zhang. To- ward Deep Universal Sketch Perceptual Grouper. IEEE Transactions on Image Processing, 2019a.
Lei Li. Changging Zou, Youyi Zheng, Qingkun Su, Hongbo Fu, and Chiew-Lan Tai. Sketch- R2CNN: An Attentive Network for Vector Sketch Recognition. 1rXiv:1811.08170, 2018.
Yijun Li, Chen Fang, Aaron Hertzmann, Eli Shechtman, and Ming- Hsuan Yang. Im2Pencil: Con- Ming-Hsuan trollable Pencil Illustration from Photographs. In CVPR, 2019b.
Hangyu Lin, Yanwei Fu, Xiangyang Xue, and Yu-Gang Jiang. Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning Sketch Gestalt. In CVPR, 2020.
Fang Liu, Xiaoming Deng, Yu-Kun Lai, Yong-Jin Liu, Cuixia Ma, and Hongan Wang. SketchGAN: Joint Sketch Completion and Recognition With Generative Adversarial Network. CVPR, 2019.
Li Liu, Fumin Shen, Yuming Shen, Xianglong Liu, and Ling Shao. Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval. In CVPR, 2017.
Yongyi Lu, Shangzhe Wu, Yu-Wing Wing Tai, and Chi-Keung Tang. Image Generation from Sketch Constraint Using Contextual GAN. In ECCV, 2018.
John FJ Mellor, Eunbyung Park, Yaroslav Ganin, Igor Babuschkin, Tejas Kulkarni, Dan Rosenbaum, Andy Ballard, Theophane Weber, Oriol Vinyals, and SM Eslami. Unsupervised Doodling and Painting with Improved SPIRAL. arXiv: 1xXiv:990000107. 2019.
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. GauGAN: Semantic Image Syn- thesis with Spatially Adaptive Normalization. In SIGGRAPH Real-Time Live!, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. In ICLR, 2015.
Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti. Sketchformer: Transformer-base Representation for Sketched Structure. In CVPR, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomed-
ical Image Segmentation. In MICCAI, 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs. In NeurIPS, 2016.
Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies. ACM Transactions on Graphics (TOG), 2016.
Rosalia Schneider and Tinne Tuytelaars. Example- -Bsed Sketch Segmentation and Labeling Using CRFs. ACM Transactions on Graphies (TOG), 2016.
10

---

Published as a conference paper at ICLR 2021
Peter Selinger. Potrace: polygon- based tracing algorithm. Potrace (online), http://potrace. source- forge. net/potrace. paf (00009077-01,, -01, 2003.
Jifei Song, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Learning Sketch with Shortcut Cycle Consistency. In CVPR, 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In CVPR, 2016.
Tijmen Tieleman and Geoffrey Hinton. rmsprop: Divide the gradient by a running average of its recent magnitude. Coursera lecture, 2017.
Haohan Wang, Songwei Ge, Eric P. Xing, and Zachary C. Lipton. Learning Robust Global Repre- sentations by Penalizing Local Predictive Power. In NeurlPS, 2019.
Xingyuan Wu, Yonggang Qi, Jun Liu, and Jie Yang. SketchSegNet: A RNN Model for Labeling Sketch Strokes. In MLSP, 2018.
Peng Xu, Timothy M. Hospedales, Qiyue Yin, Yi-Zhe Song, Tao Xiang, and Liang Wang. Deep Learning for Free-Hand Sketch: A Survey and A Toolbox. arXiv:2001.02600, 2020.
Zijian Xu, Hong Chen, Song-Chun Zhu, and Jiebo Luo. A Hierarchical Compositional Model for FaceRepresentation and Sketching. PAMI, 2008
Jianwei Yang, Anitha Kannan, Dhruv Batra, and Devi Parikh. LR-GAN: Layered Recursive Gener- ative Adversarial Networks for Image Generation. ICLR, 2017.
Lumin Yang, Jiajie Zhuang, Hongho Fu, Kun Zhou, and Youyi Zheng. SketchGCN: Semantic Sketch Segmentation with Graph Convolutional Networks. arXiv:2003.00675 2020.
Lixiu Yu and Jeffrey V Nickerson. Cooks Cobblers? Crowd Creativity through Combination. In CHI, 2011.
Qian Yu, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy Hospedales. Sketch-a-Net that Beats Humans. In BMVC, 2015.
Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, and Chen-Change Loy. Sketch Me That Shoe. In CVPR, 2016.
Tao Zhou, Chen Fang, Zhaowen Wang, Jimei Yang, Byungmoon Kim, Zhili Chen, Jonathan Brandt, and Demetri Terzopoulos. Learning to Sketch with Deep Q Networks and Demonstrated Strokes. 1rXiv:1810.0553 2018.
Jun Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell. Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward Multimodal maage-to-Image Translation. In NeurIPS. 2017.
A SKETCH SEGMENTATION DATASETS
Previous sketch segmentation datasets (Schneider & Tuytelaars 2016 Yang et al. 2020 Li et al.) 2019a Wu et al. 2018 are too sparsely annotated to effectively train deep models. For example, the SPG dataset (Li et al. 2019a) contains 800 sketches per category with part annotations. Note that most approaches train one model per category. The SketchSeg (Wu et al. 2018) dataset con- tains 100 labeled sketches per category, then further augmented by automatically generating more sketches with a trained SketchRNN (Ha & Eck 2018) model. Our proposed datasets are exhaus- tively annotated all 10k sketches in both datasets have part annotations.

---

Published as a conference paper at ICLR 2021
B EXISTING CREATIVE TOOLS BASED ON SKETCH GENERATION
The growing availability of sketch generation approaches has spurred a variety of creative ap- plications and art projects. The book Dreaming of Electric Sheep (Diaz-Aviles 2018 inspired by Philip K. Dick's famous book contains 10,000 sheep sketches generated by DCGAN (Rad- ford al. 2015) trained on the QuickDraw dataset (Jongejan et al. 2016). Quick, Draw! (quickdraw aww wit withgoogle com) recognizes what user is drawing as they draw it. Auto- Draw (Lab) (017) returns high quality illustrations or sketches created by experts that match a novice's sketch. Edges2cats (Hessel (017) is an online tool based on pix2pix (Isola et al., (017) that converts free-hand drawings realistic images. Our dataset and approach can be used to boost more applications related to draw a creative sketch.
C INSIGHTS FROM PILOT DATA COLLECTION PILOT
We found that when using shorter initial stroke, the generic creatures were often recognizable as specific animals. But with longer strokes, we more frequently found fictional creatures that are high quality (look like creatures) but not recognizable Piloting longer initial strokes with birds indicated that subjects had trouble incorporating the initial stroke well while maintaining coherence. Shorter strokes led simpler sketches (still significantly more complex and creative than Quic Figure making Creative Birds a more approachable first dataset for creative sketch generation, and Creative Creatures notch above in difficulty.
real animal, indicating higher creativity.
D STROKE- AND PART-LEVEL ANALYSIS
We compare our Creative Birds and Creative Creatures datasets to previous datasets in Table 2 As a proxy for complexity, we report the average stroke length (normalized by the image size) and number of strokes across the bird sketches in each dataset. We see that our datasets have the longest and most number of strokes. The difference is especially stark w.r.t. QuickDraw (twice as long and three times as many strokes), one of the most commonly used sketch datasets.
Table 2: Stroke statistics
normalized stroke length
number of strokes
Sketchy
5.54 + 2.02
15.57 10.54
Tu-Berlin
5.46 + 1.75
14.94 = 10.18
Draw
4.00 + 2.97
6.85 + 4.19
Creative Birds (Ours)
7.01 + 2.89
20.74 + 13.37
Creative Creatures (Ours)
8.40 + 3.47
22.65 14.73
Next we analyze the part annotations in our datasets. Table shows the number of sketches contain- ing each of the parts. Notice that each part in Creative Creatures has significantly fewer examples (in the extreme case, only 302 for paws.) than most parts in Creative Birds, making Creative Creatures additionally challenging. Most parts in Creative Birds are present in more than half the sketches except for the mouth, which we find often subsumed in beak. For Creative Creatures, the numbers of sketches containing different parts has higher variance. Common parts like body and head have similar occurrence in both datasets. Example parts are shown in Figures IO and II] Note that the initial stroke can be used as any (entire or partial) part, including body but will not be annotated as such. This means that sketches without an annotated body likely still contain a body.
In Figures 7 and 8 we analyze the order in which parts tend to be drawn in both datasets.
E EXAMPLE SKETCHES, PARTS, AND INITIAL STROKES
See Figures 12 and 13 for example sketches from the Creative Birds and Creative Creatures datasets respectively. See Figures 10 and [ example parts from both datasets. See Figure for examples of the initial strokes used in the data collection and sketch generation process.
12

---

Published as a conference paper at ICLR 2021
Table 3: Part statistics
Creative Birds
total 8067
eye 8067
wings 7764
legs 5564
head
body
tail
beak
mouth 1447
7990
8477
5390
7743
total 9097
eye 9097
arms 2862
beak 2163
mouth 6143
body
ears
feet 2431
fin
7490
3009
1274
Creative Creatures
hands 1771
head 6362
hair 3029
horns 1738
legs 3873
nose 3428
paws 302
tail
wings
3295
1984
arm6
0.05
0.02 6.05 0.00
0.14
0.06 0.05
0:01
0.06
0:23
0644 0.02
0.17
0.04 0.02
0.0% 0.01
baat
0.09
DCaK-0.03
0.01 0.00
pay 0.03 0.02
0.00 0.02
0.02 0.05
0.52 0.00
mouth
0.06
05
0.00 0.18 0.10
0.09
0.02
0.05
0:39
0.00 0.06 0.07
body
0 12 0.04 0.00 0.02 0.06 0.07
0.04 0.04 0.04
0:09 0.03
0:17 0.03 0.00
0.11 0.00
0.03
book
1o0d
bwy
0.22
beak
head
6.01
6:31
0.02
0:32
0.02
0:25
0.08
0.04
0.110.07
0.05
0.08 0:15
n10
n. nu
Seet
0.01
0.01
Seat
1n-0
0.10 6.03
0.00
0.03
0.12 0.07 0.07 0.02 0:04
1M 6.36 0:10 0.0 6.01 AMM 0.00 OCT 8:10 m. 0.0% 0.04
0.10 0.02 9.06 0.07 0.01 0.00 0.03 0.07 0.02
0.10 0.07 0:05 0.05 0.05
0.05 0.00
0.11 0.06
0.01
1egs
logs-
6.DM
c.03
0.1
hair 0.02
twnds- 0 03 0.02 0.13 0.13 0:05 0.07 0.02 0.00 0.01 0:05 0.07 0.17 0.06 0.02
0.08 0.01
mouth
mouth
6.08
0.10
Twrrch
tall
4..4
0.06
0.05
0.200.07000
0.000.0000
0.00
horms
04 0.02
0.20 0.13 0:00 0.04 0.04 0.05 0.03 0.05 0.01 0.13 0:31 0.03 6.07 0.04
otrga
0.:01
wings-
0.10
0.09
0.02
bonak
Nead
Ieas-
0.02
0.06 0.17 0.01 0.07 0.04 0.04 0.01 0.04 0.0.
0.1L
koge
0.01
6.03
0.16 0.10 0.02 0:02 0.09 0.02 (.CB 0.04 0.04 0.00 0.02
0.0T 0.02
(a) Creative Birds
708e
0.05
paws
0.07 0.01 0.12 0.07 0.00 0.64 0.02 0.00 0.03 0.63 0.02 0.07 0:33 0.02
0:20 0.09
pawe
0.00
0.03 0.05
0.00 0.04
168
wings
(0.03 5.07 0.07 032 0:00 0.08 0:01 0.07 0.03 9.6 0.01 9:20 0.04 0.00
5:10 0.1
bask modto
1000
(b) Creative Creatures
Figure 7: For each dataset, on the left is the probability that the part is the first part drawn (after the eye, which subjects are asked to draw first). On the right is the probability that conditioned on the row part being drawn, the column part is drawn next. In Creative Birds, we see that sketches typically start with either the beak (in case the initial stroke already provides part of the head), or the head. In both cases, the next part is often the body. From there either the wings or legs tend to be drawn. The tail tends to follow. The mouth often drawn last. Similar patterns can be seen in Creative Creatures, except with larger variety due to the larger number of part options and larger diversity of creatures that subjects draw.
F IMPLEMENTATION DETAILS
Training. We found that the default learning rates used in the StyleGAN papers lead to over- optimization of the discriminator during the early training phase and prevent effective optimization of the generator on our datasets. Therefore, we did grid search on the discriminator and generator learning rates in {10-3 (10-3.5-3 10-4004--00 5 5-1 and batch sizes in (8, 40, 200]. looking for training stabil- ity and the high generation quality. We picked a learning rate of 10-4 and a batch size of 40 for
both the discriminator and generator. We use the Adam optimizer Kingma & Ba (2014) following StyleGAN papers (Karras et al. 2019 (020)) with 31 = 0, 32 0.99, = 10 We multiply tra factor equals to 0.01 to the sparsity loss. As for the part selectors for both datasets, we train each on 80% of the data with learning rate 2- and batch size 128 for 100 epochs when the training losses are observed to reach the plateau and a reasonable testing accuracy is achieved on the other 20% data. Specifically, we get 63.58% and 48.86% part selection test accuracy the Creative Birds and Creative Creatures datasets respectively. Our training time of each creature and bird part generator is approximately and 2 days on single NVIDIA Quadro GV100 Volta GPU.
Data augmentation. As discussed in Section 4 we apply a random affine transformation to the vector sketches as data augmentation step in addition to the random horizontal flips during training.
13

---

Published as a conference paper at ICLR 2021
0.50
0.42
0.38
0.26
0.54
0.
65
0.74
0.62
0:30
0.15
0.64
0.60
0.43
0:71
0.71
0.68
0.58
0.63
0.80
0.82
mouth
0.43 0.60
0.66
0.66
body
0.74 0.35
0.57
0.50
0.63
88
0.79
0.73
0:62
0.21
0.78
0.89
0.55
0:89
0.91
0.84
00.46
0,20
0.40
0.37
0.50
03
0.46
0.59
0.60
0.17
0.39
0.54
0.46
0.79
0.69
0.61
0.58
beak-
0.67
0.33
body
0.33
0.50
0.20 0.200.00..7
05.00.000
0.33
0:39
0.55
0.55
0.43
0.66 0.41
head-
0.67 0.80 0.50
0.50
0:04
0.90 0 5888
00.26
0.52
0.19
hair
00.38
0.27 0.41 0.50 0.53
0.13
0.40
0.67
0.54 0.42
legs
0.10 0.04
0.06 0.50
0.56
46 0.35
0.11 28 0.10 0.44 0.50
6.60 0.54
hands
0.111
0.17
0.34
0.18 0.35 0. 45 0.48 0.47 0.50
0.09 0.54
0.46
0.37 0.63
0.67
0.44
head
-0.85
0.63
0.84
0.79 0:83 0. 87 0.81 0:87 0.91
0.50 0.92
0.92
0.80 0.96 0.97 0.68
0.06 0.54
vings
0. 19 0.12
0.65 o. 46 0.50
homs
0.36
0:41
0.46
0.51 0.73 0.59
baok
bedy
heaad
legs
mouth
tall
wings
legs
0..40
0.46
0.59 0.54
0.42 0.76
nose
0.557 0.18
0.40
0.45
0.54 0. 57 0.36 0.60 0.63 0.20 0.49
0.58
0.50 0.75 0.61 0.54
0.229
0.14
0.20
0.11 0:21 0. 24 0.16 0.33 0.37
0.04 0.27
0.24
0.25 0.50 0.44 0.57
(a) Creative Birds
tall
0.29
0.16 0.09 0.39 40 0.46 0.33 0.08 0.41 0.39 0.56 0.50
wings
0.32
0.43 braal
0.16 0.44
0.58 0.56
0.46
0.43 pawe
0.61 0.50 Fail
arms
moouth
hair
hands
torms
lags
nove
wings
(b) Creative Creatures
Figure 8: Across all sketches that contain both parts, show the probability that the part along the column was drawn (at some point) after the part along the row. In the Creative Birds dataset, the smaller parts such the legs and mouth are often drawn after the larger parts including the head and body. In the Creative Creatures dataset, the head is often drawn before other parts. Hands, paws, feet, legs tend be drawn later.
7
4
4r
P
P
W
4
4
2
C
3
V
n
A
8
A
Figure 9: Example short (left) and long (right) random initial strokes used as prompt in our data collection and as the first step in our sketch generation approach. The short strokes were used for Creative Birds and long for Creative Creatures. Thick strokes are shown for visibility.
Specifically, we used two sets of affine transformation parameters: 1) random rotation with angles 0 E 24[-150, 15%], random scaling with ratio E (0.9,, 1.1], random translation with ratio & E 41-0.01, 0.01]. and fixed stroke width = 2 pixels; 2) E (1-4459455] s E 24(0.55 1.25], t E 2|-0.05, 0.05], and / 2/ pixels. We found that larger transformation and longer training time especially useful when training part generators on the Creative Creatures dataset which is more challenging and parts often have fewer examples than Creative Birds. We also found
14

---

Published as a conference paper at ICLR 2021
3
a1ll
1
Figure 10: Random examples of parts from Creative Birds. Thick strokes displayed for visibility. Top to bottom. left to right: eye, head, body, beak, mouth, wings, legs, tail, details. The number of examples each block roughly proportional to the percentage of sketches that have that part.
15

---

Published as a conference paper at ICLR 2021
a
3
r
C
C
c
Random examples of parts from Creative Creatures. subset of parts are shown, with preference for those not in common with Creature Birds. Thick strokes displayed for visibility. Top to bottom, left to right: hair, hands, head, horns, arms, body, cars, feet, fin. The number of examples in each block is roughly proportional to the percentage of sketches that have that part.
Figure 11:
16

---

Published as a conference paper at ICLR 2021
Figure 12: Random example sketches from our Creative Birds dataset. Entire sketches (including the "details") are shown.
17

---

Published as a conference paper at ICLR 2021
8000
Figure 13: Random example sketches from our Creative Creatures dataset. Entire sketches (includ- ing the "details") are shown.
18

---

Published as a conference paper at ICLR 2021
w1
(a) Creative Birds example
(b) Creative Birds example
(c) Creative Creatures example
3
(d) Creative Creatures example 2
Figure 14: Sketches with similar initial strokes from Creative Birds and Creative Creature datasets. Similar initial strokes have diverse sketches that incorporate them. Note that it is much easier to find sketches with similar initial strokes in the Creative Birds dataset because the initial strokes shorter. This also indicates that the Creative Creatures dataset is more challenging.
19

---

Published as a conference paper at ICLR 2021
2
4
*
3
3
2
x
z
4
27
4
8
R
$
x
8
a
$
&
x
1
6
T4
+0
A
(a) Creative Birds.
2
2
3
(b) Creative Creatures.
Figure 15: Uncurated creative sketches generated by DoodlerGAN.
20

---

Published as a conference paper at ICLR 2021
Q
4
m
X
1
x
+
#
4
1
R
x
x
4
8
2
2
8
x
*
4
8
A
2
t
x
4
2
Q
2
N
8
g
(a) Bird sketches generated by DoodlerGAN.
(b) Nearest Neighbors in the training data.
XT
d
22
x
8
P
40
(c) Creature sketches generated by DoodlerGAN.
(d) Nearest Neighbors in the training data.
Figure 16: Nearest neighbors of the sketches generated by DoodlerGAN showing that the genera- tions do not mimic the training data.
that the larger transformation also helps to increase the variation in the generation based on the noise but the general quality of generated parts suffers on the Creative Birds dataset. Consequently, we apply the larger augmentation when training the creature part generators and bird eye generator and smaller augmentation when training the other models. And we train the creature part generators for 60, 000 steps and bird part generators for 30, 000 steps.
Inference. For Creative Birds, we noticed that the quality of the generated sketch was higher when the generated eye was not too small and not too close to the initial stroke. Motivated by this observa- tion, we used the following trick: We sample 10 eyes for each initial stroke and rank them based on the pixel sum and distance to the initial stroke. We combine the ranks using Borda count and pick the highest ranked eye for the following steps. Using this we see improvement in the generation quality for Creative Birds, but not Creative Creatures.
Since humans in our data collection process were asked add least five parts the sketch, employ the same rule for our part selector - only after five parts have been added, the selector is given the option predict stop. Once part has been drawn, that part can not be selected again.
21

---

Published as a conference paper at ICLR 2021
G DETAILS OF MODEL TRAINED TO COMPUTE FID AND DIVERSITY SCORES
FID computes the similarity between two image distributions. It first embeds the images into feature space defined by trained Inception model (Szegedy al. 2016). It then fits multivariate Gaussian to the feature vectors from both sets of images. Finally, it computes the Fréchet distance between the two Gaussians.
To this end, we trained an Inception model on the QuickDraw3 8M dataset (Xu et al. 2020). Specif- ically, we preprocess the dataset based on the instructions the paper (Xu al., (020) and render each vector sketch into 64 x 64 raster image. The dataset contains 345 classes and each class contains 9, 000 training samples, 1, 000 validation samples, and 1, 000 test samples. We train the Inception model with the R.MSprop optimizer (Tieleman & Hinton 2017) for 50 epochs. We use initial learning rate of 0.045 with an exponential decay at a rate 0.98 after every epoch. The trained model achieved reasonable test accuracy 76.42% ( of-te-of-the-ar accuracy is 80.51 % reported in SketchMate (Xu et al. 2020 using 224 x 224 images).
Generation diversity calculates the average pairwise Euclidean distance between the features caleu- lated for sketches with a trained model to reflect the diversity of the generation. To be consistent, use the same Inception model trained QuickDraw3.8M to calculate these features.
H PROPOSED METRICS
Here we provide more details about the the characteristic score (CS) and semantic diversity score (SDS) metrics introduced in Section 5.1 We first construct the label sets B and C which include the labels from the 345 QuickDraw classes that represent a bird or creature. The details the two define
sets be found below. Then for the Creative Birds and Creative Creatures datasets, the characteristic score as the percentage the time a generated sketch is classified as a label in B and C respectively. The characteristic score gives basic understanding of the generation quality, indicating whether they are recognizable as the relevant concepts (birds and creatures) we used to create the dataset. We define the semantic diversity score as follows:
SDS Pc
PI
log
Pt --=--p
Pi log
PC
Pc
Pc
LEC
where pr represents the average probability that sketch in the generation collection is classified as label and pc represents the average probability that generated sketch is classified creature ie., PC = >iec Pt. The intuition similar to Inception Score (IS) (Salimans et al. 2016). Except that SDS does not penalize sketch for receiving a high probability for more than one label. In fact, a sketch that looks like a combination two creatures is probably quite creative! Such examples shown in Figure i in the paper. Thus, in SDS only look at the sum of the label distribution of the generation collection to capture two aspects of creativity: Across generations 1) the distribution should have high variety across creature categories (measured by the entropy of the marginal creature distribution) and 2) the distribution should have the most mass on creature categories.
The bird label set B and creature label set C from the QuickDraw categories are as follows:
B = [bird, duck, flamingo, parroi]
C [ant, bear, bee, bird, butterfly, camel, cat, cow, crab, crocodile, dog, dolphin, duck, elephant, fish, flamingo, frog, giraffe, hedgehog, horse, kangaroo, lion, lobster, monkey, mosquito, mouse, octopus, owl, panda, parrot, penguin, pig. rabbit, raccoon, rhinoceros, scorpion, sea turtle, shark, sheep, snail, snake, spider, squirrel, swan, tiger, whale, zebra]
I MULTIMODAL PART GENERATION
Conditional GANs tend to ignore the input noise and base their generations completely on the condi- tional information (Isola et al.) 2017). We encounter similar challenges in our approach, especially for the parts later in the order when the conditional information is strong. For the eye generator, when the conditional signal weak (just the initial stroke), the model responds to the noise well. We demonstrate this with the style mixing experiment conducted in the StyleGAN paper (Karras)
22

---

Published as a conference paper at ICLR 2021
Random translation
Part generation Reverse translation
Figure 17: Conditioning perturbation trick illustration.
(a) Bird eye generations with mixed (b) Bird tail generation without (c) Bird tail generation with condi- styles.
conditioning perturbation.
tioning perturbation.
Figure 18: We show various parts generated by our model when conditioned on different noise vectors. The eye generator naturally responds to the noise we see intuitive style mixing results in (a). For the other parts, we show that diverse parts can be generated (different appearances and positions) using our conditioning perturbation trick. See text for details.
et al. 2019). Specifically, given the same initial strokes, we sample two random noise vectors and feed them into different layers of the generator. In Figure each row has the same input noise for the lower generator layers and each column has the input same noise for the higher generator layers. We see that the lower-layer noise controls the position of the generated eyes while the higher-layer controls the size and shape of the eyes.
However, for generators of later parts, the generation tends have little variation given different noise vectors illustrated by the tail generation results in Figure T8b Although the generator does not respond to the noise, we find that is very sensitive to the conditional information. Specifically, if we slightly translate the input partial sketch, the generator generates part with different appear- ance and positions. As shown in the Figure 17 we randomly translate the input partial sketch, feed them to the generator, and then translate the generated parts back to the correct positions. Figure [8c shows different generations for the tail given the same input partial sketch. We call this trick condi- tioning perturbation. This trick is motivated by the formulation in SketchRNN (Ha & Eck 2018) where GMM parameters instead of deterministic points are used as the RNN output as mechanism introduce variation.
J GENERATIVE V.S. RETRIEVAL
In addition to the generation- methods (StyleGAN2, SketchRNN), that we compared to the main paper, we now consider strong retrieval-based baseline. Specifically, we hold 5% of the sketches our datasets to use as query sketches. We extract the first N parts from these sketches and use them to match against partial sketches containing N parts from the remaining 95% of the dataset using the average Chamfer distance across parts. The rest of the matched sketch is returned complete the query partial sketch.
23

---

Published as a conference paper at ICLR 2021
A limitation of this approach is that the quality of the match will deteriorate if the application quires multiple completions the same query partial sketch (e.g., to present as options to a human- in-the-loop). A retrieval approach will have to retrieve farther neighbors to find more completions. A generative approach can simply generate multiple samples as completions, with no systematic degradation in quality across samples as shown in Figure T8c
We evaluate this via human studies, we hypothesize a scenario where 32 candidate completions are desired. We compare the 32nd retrieved completion with "32nd" sample from DoodlerGAN (in our approach there is no natural ordering to the generations), and ask human subjects in which sketch the query is better integrated. We experiment with 200 queries containing and 3 parts each. DoodlerGAN is statistically significantly better than the retrieval approach, preferred 54% and 55% of the times with 2- and 3-part queries respectively.
Recall that the retrieval approach, by definition, reproduces human sketches completions. On the other hand, we saw Figure [66 that DoodlerGAN generates sketches that are different from training data, making it conceptually more creative approach.
K ABLATION STUDY
These ablation studies demonstrate the effectiveness of different design choices in DoodlerGAN.
Part generator: Figure 19a shows sketches generated if the entire partial sketch used as the conditioning information without the part-wise channel representation. We see that it is difficult for the part generator to learn the spatial relationships between different parts. Notice that the head (red part) often does not include the eye inside it. Out of sample of 512 head generations, we find that 57.4% of the time the generated head does not surround the eye when not using part channels, compared to 88.7% of the time with part channels. Figure 19b [9b] shows generations without the U-Net architecture, directly concatenating the encoder output with the noise vector instead. We see that the quality of the part composition is much worse than when using the U-Net (Figure 19ch which encodes the spatial information of the conditioning sketch in the feature map.
$2
B
8
2
2
$
$
Yor
28
m
Y
$
2
3
20
xx
d
18
4
3
$
p
\
x
d
1
y
(a) Without part channels.
(b) Without U-Net connections.
(c) Full model.
Figure 19: Ablation studies on the part generator.
Part selector: Figure 20 shows sketches generated if instead of using a part selector, we generate parts in fixed order (the most common order in the dataset). We find that the use of a part selector allows the model to use the initial stroke much more effectively. Consider the two sketches with red boundaries shown in Figure 20a They have the same initial strokes. When using a part selector, head is not generated and the initial stroke is used as the head. Whereas the generation with fixed order still generates the head and this error propagates to subsequent part generations. Similar observation can be made for beak generation in the two sketches with blue boundaries shown in Figure 20a Furthermore, Part selector plays an important role in choosing appropriate part lists and stopping the generation in appropriate step. For instance, it is not reasonable to generate all 16 part in every creature sketch, which greatly undermines the quality as shown in Figure 20b] A part selector is also important if one wants to implement method in human-in-the-lonn scenario
24

---

Published as a conference paper at ICLR 2021
where the model would need to identify what the human has already drawn, and then decide what to draw next.
B
A
$
1
Sy
Predicted order
Fixed order
Predicted order
Foxed order
(a) Creative Birds.
(b) Creative Creatures.
Figure 20: Ablation study the part selector.
L CONVERTING GENERATED SKETCHES TO VECTOR IMAGES
DoodlerGAN generates sketches in raster image format. One disadvantage of raster images is that their resolution can not easily increased without introducing artifacts. The benefit of vector images is that they can be rendered at arbitrary resolutions. We convert raster images of sketches to vector representatiio (SVGs) using the open source tool mkbi aakbitmap, which relies on the pot race program (Selinger 2003)) based polygon tracing algorithm. The parameters used are shown in Table Examples conversions are shown Figure 21 Notice that in addition to the change in format, the conversion also introduces different aesthetic to the sketch.
We conduct a human study to evaluate this aesthetic. We found that subjects prefer the new aesthetic 96% of the times! Note that this is while keeping the resolution of the two aesthetics the same. We hypothesize that this may be because the converted images do not have any blurry artifacts and the strokes are smoother. For completeness, Figure 21 we also show couple of these converted sketches at a higher resolution.
Table 4: mkbi i tmap & potrace parameters
Parameters
threshold grey value
bandpass filter radius
scale
interpolation
backend
path type
values 0.2
2
cubic
svg
group
25

---

Published as a conference paper at ICLR 2021
(a) Raster images (top) and corresponding converted vector images (bottom) on Creative Birds datasct.
(b) Raster images (top) and corresponding converted vector images (bottom) on Creative Creatures dataset.
(c) serrk-picked generations from DoodlerGAN converted to vectorized representation and displayed at higher resolution.
Figure 21: Converting DoodlerGAN's sketches from raster images to vector representations
26


Reviews:
[{"review_id": "nPbTeeOR7z6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper put forward two Creative datasets - Creative Birds and Creative Creatures, which have corresponding part annotations. Based on Part selector and Part generator. The authors demonstrate a multi-stage sketch generation approach - DoodlerGAN, and achieve the STOA compared with other methods in several metrics.\n\nThe topic of creative generation is interesting and gradually becomes the future research direction in the image generation field. There are some pros:\n- The datasets provide a novel perspective to decouple the basic elements of sketch images with fine edge semantic labels and text descriptions. These annotations could be used in the fine-grained domain adaption task, text-to-sketch generation or some reverse tasks. The reviewer thinks this contribution is meaningful and solid.\n- The sketch generation fashion is enlightening. Although the idea of multi-stage generation has been used in many synthesis tasks, it's mainly for different resolutions or granularities (local and global). The partial strokes combine the overall sketch images is a domain-specific technique. The authors reimplement in deep learning way. The reviewer has to say that it may have some limitations on other fields, it's appreciated if the authors can share new possibilities, but that's not the point.\n- The experiments are solid. Quantitative evaluation and human evaluation both outperform other baseline methods. It's convincing.\n\nSome concerns are also proposed:\n- How are the predicted labels added to the generator? It doesn't seem to show up on Figure 3(a). The reviewer wants to know more details.\n- The reviewer has some confusion about the noise range in Figure 3(b), which is N(0,1). However, there is different descriptions in sec 4.2 - N(0,2). Which one is the true sampling distribution?\n- The full loss functions are lack in the mainly body, which should be depicted more clearly.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NtraifPTXTo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper introduces two creative sketch datasets of birds and creatures, segmented into parts, each with ~10k doodles collected from Amazon MTurk workers. In a user study, people tend to favor sketches from their dataset over the similar Gogole QuickDraw sketches. Additionally, the authors propose a GAN architecture for generating novel sketches in an incremental fashion, one part at a time. They provide many qualitative results as well as human studies to validate their approach.\n\n### Explanation of Rating\nWhile the new datasets look nice, I'm not sure that they sufficiently different from or better than existing sketch datasets. Overall, the scale, complexity, and diversity of the sketches are roughly the same as some of the other datasets mentioned in the paper. It isn't obvious to me which novel applications or approaches are made possible with this dataset that were not possible before.\nWith respect to the proposed generative model, I think a user-in-the-loop interface is a reasonable approach, but though the model seems compatible with such a system, the authors do not actually implement it. It would be nice to see this interface in practice, since without it, the results do look a bit better but are not particularly more useful than those from other GAN models.\n\n### Pros\n- The UI and methodology for data collection, in which a user is asked to first draw an eye, followed by semantic parts of their choosing is interesting and appears to be conducive to better sketch quality.\n- The paper is well written and contains many qualitative results and figures.\n- The labeling of sketches by semantic parts presents an advantage over previous datasets.\n- The iterative/incremental design of the generative model is a nice step towards computer-aided user-in-the-loop creative design.\n\n### Cons\n- While the authors claim that Figure 2 shows that their dataset is more diverse and creative, I'm not sure if that's objectively the case and if the marginal improvement is sufficiently important for downstream tasks. \n- I understand the authors' argument that part-level granularity is more reasonable than stroke-based granularity, but I think that one major insight that can be gained from studying sketch datasets is with respect to how humans draw. For this reason, it would be useful to have information about the order of individual strokes, which the dataset does not appear to include.\n- I'm not sure about the argument that raster images are a better format for a generative model than vector. It seems like the long-range relationships and connectivity as well as the sparsity inherent in a vector representation are pretty important in a sketch. Consequently, the some of the results produced by the model appear to suffer from topological artifacts and poorly-defined line segments.\n- It would be useful to provide a citation for the doodling process used for data collection.\n- How were the 100 sketches from Creative Birds and QuickDraw chosen for the user study?\n- The paper claims the importance of certain architecture choices, e..g, that part channels help the model better predict the part locations. Some ablation study to verify this would be appropriate.\n- Page 4: \"subjetcs\" (typo)\n\n---\n\nThank you to the authors for their comments and clarifications.  I still am skeptical that the proposed dataset is a fundamental improvement over Quick Draw---ultimately, both datasets contain compact sketches from a single category containing relatively few strokes. But given that your updates and clarifications have addressed many of my questions, I am raising my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NMS0SnBOUiv", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The contribution of this paper is twofold. First, the authors introduce a pair of manually collected (via AMT) datasets of creative sketches (birds and general creatures) each along with part annotations. Second, they propose a part-based Generative Adversarial Network for the generation of unseen compositions/configurations of novel parts (legs, body, etc.) for creative sketches. The latter is quantitatively and qualitatively (via human studies) evaluated. \n\nStrengths\n\n- Generally, the paper is well introduced and easily understood. Especially, the provided dataset is very useful in promoting the development of the problem addressed: creative sketch generation. Although not being a really “novel” idea/data, providing new benchmarks/datasets/competitions for the AI community is always refreshing.\n\n- The authors also include a generative architecture which is then quantitatively evaluated against several working implementations of the pretrained SOTA baselines. They also provide easy-to-understand evaluation metrics allowing the community to keep on working on the task at hand. The experimental results demonstrated the effectiveness of the proposed method.\n\n- The illustrative experimental setting shows the usefulness of the datasets, the potential of their generative approach and its easiness of use.\n\nWeaknesses\n\n- Of course, this is a general concern any time a diagnostic dataset/tool is introduced. Rarely is an immediately surprising insight offered in the same paper. The value of such datasets/tools is often clear only in hindsight with the benefit of time. If they are useful, they see organic adoption. If they are not, they organically fall to the wayside. \n\n- Furthermore, one can argue that there is no particularly novel insights in this paper (no science behind the collection of the datasets and the proposed GAN is a logic incremental evolution of the SOTA). However, the paper is very well written and structured, the methodology followed is correct and the experimental setting is comprehensive (with promising results), so I'm happy to let the noisy process of science (and reviewing process) figure out the value here.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EJ-dHrve8mc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors collect a large dataset of sketches of imaginative birds and other creatures, then train a GAN to produce similarly creative sketches. They use both quantitative and qualitative methods to evaluate their work and find that the generated sketches are of higher quality than both other models and human sketches, and are novel.\n\nThis topic is delightful and your paper is quite well written. My only concern is fit for the venue. This isn't a paper about a learned representation and doesn't encounter any theoretical issues regarding representations, why should it be published at ICLR rather than another venue, e.g. ICCC (https://computationalcreativity.net/iccc21/)? There are plenty of papers at ICLR that aren't about representation learning these days, so I'm still inclined to accept, but I'd like to hear an argument.\n\nMore specifically, I appreciate the level of detail regarding the dataset collection study and your model training process. I would also like to see more discussion of your \"conditioning perturbation\" trick. Why does it work? Can you characterize the effect the perturbations have on the generated sketches?\n\nI'm a little confused about Figure 5: were the images from each method compared against the same set of images? If not, you may need to do some more sophisticated statistical analysis than the Bernoulli confidence interval to judge the probability that each method is chosen above each competitor. I also find it hard to believe that the human drawings perform significantly worse than the GAN drawings across the board. Does that indicate a problem with the dataset you constructed or some bias among your participants?\n\nVery minor issue, but I think you also have a typo in Table 1, the DS column should probably be labeled GD to match the table caption.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While much of generative modeling is tasked with the goal of generating content within the data distribution, the motivation of this work is to examine whether ML techniques can generate creative content. This work has 2 core contributions:

1) Two new datasets of creative sketches: birds and creatures, that have part annotations (size ~ 10K samples for each set). The way the datasets are structured with the body part annotations will facilitate the creativity aspect of the approach later described in the paper.

2) This paper propose a GAN model that is part-based, which they call DoodlerGAN. It is inspired partly by the human's creative process of sequential drawing. Here, the trained model determines the appropriate order of parts to generate, which makes the model well suited for human-in-the-loop interactive interfaces in creative applications where it can make suggestions based on user drawn partial sketches.

They show that the proposed model, trained on the part-annotated datasets, are able to generate unseen compositions of birds and creatures with novel body part configurations for creative sketches. They conduct human evaluation and also quantitative metrics to show the superiority of their approach (for human preference, and also FID score).

Many reviewers, including R1 and myself observe that the datasets provided, along with the parts-based labeling and modeling approach are a clear advantage over existing datasets and methodology. With ever growing importance of generative models used in real world applications, including the creative industry, I believe this paper provides a much needed fresh take on creative ways of using our generative models besides making them larger, or achieving better log-likelihood scores. Many reviewers, including R3, would think that this work is indeed a "Delightful, well written paper! I have concerns about its fit here." I strongly believe such works in fact definitely *do* belong at ICLR, and I think this work has the potential to get researchers in the generative modeling community to rethink what they are really optimizing for.

I believe this paper will be a great addition to ICLR2021, and I look forward to see their presentation to the community to spark more creativity in our research endeavors. For this reason, I'm strongly recommending an acceptance (Poster).
Decision: Accept (Poster)