SketchEmbedNet: Learning Novel Concepts by Imitating Drawings
Authors: Alexander Wang
Mengye Ren
Richard Zemel
Venue:  (2021)
Keywords: generative
probabilistic
sketch
drawing
few-shot learning
classification
embedding learning

Abstract: Sketch drawings are an intuitive visual domain that appeals to human instinct. Previous work has shown that recurrent neural networks are capable of producing sketch drawings of a single or few classes at a time. In this work we investigate representations developed by training a generative model to produce sketches from pixel images across many classes in a sketch domain. We find that the embeddings learned by this sketching model are extremely informative for visual tasks and infer a unique visual understanding. We then use them to exceed state-of-the-art performance in unsupervised few-shot classification on the Omniglot and mini-ImageNet benchmarks. We also leverage the generative capacity of our model to produce high quality sketches of novel classes based on just a single example. 

Under review as a conference paper at ICLR 2021
SKETCHEMBEDNER: LEARNING NOVEL CONCEPTS BY IMITATING DRAWINGS
Anonymous authors Paper under double-blind review
ABSTRACT
Sketch drawings are an intuitive visual domain that appeals human instinct. Previous work has shown that recurrent neural networks are capable of producing sketch drawings of single or few classes at a time. In this work we investigate representations developed by training a generative model to produce sketches from pixel images across many classes in a sketch domain. We find that the embeddings learned by this sketching model are extremely informative for visual tasks and infer a unique visual understanding. We then use them to exceed state-of-the-arr performance in unsupervised few-shot classification ImageNet benchmarks. We also leverage the generative capacity of our model to produce high quality sketches of novel classes based on just single example.
the Omniglot and mini-
INTRODUCTION
Upon encountering a novel concept, such six-legged turtle, humans can quickly generalize this concept by composing a mental picture. The ability to generate drawings greatly facilitates communicating new ideas. This dates back to the advent of writing, as many ancient written languages are based on logograms, such as Chinese hanzi and Egyptian hieroglyphs, where each character is essentially sketch of the object represents. We often see complex visual concepts summarized by a few simple strokes.
Inspired by the human ability to draw, recent research has explored the potential to generate sketches using a wide variety of machine learning models, ranging from hierarchical Bayesian models (Lake ct al. 2015), to more recent deep autoregressive models (Gregor et al. 2015 Ha & Eck 2018 Chen et al. 2017) and generative adversarial nets (GANs) (Li et al. 2019). is natural question to ask whether we can obtain useful intermediate representations from models that produce sketches in
the output space, as has been shown by other generative models (Ranzato al. [006]] Kingma & Welling 2014: Goodfellow et al. 20144 Donahue et al. [017]] Doersch et al. 2015). Unfortunately, hierarchical Bayesian model suffers from prolonged inference time, while other current sketch models mostly focus on producing drawings in a closed set setting with a few classes (Ha & Eck
2018 Chen et 2017). on improving log likelihood the pixel level (Rezende al. (0066) Leveraging the learned representation from these drawing models remains a rather unexplored topic.
In this paper, we pose the following question: Can learn generalized embedding function that captures salient and compositional features by directly imitating human sketches? The answer is affirmative. In our experiments we develop SketchEmbedNet, an RNN-based sketch model trained to map grayscale and natural image pixels to the sketch domain. It is trained on hundreds of classes without the use of class labels to learn a robust drawing model that can sketch diverse and unseen inputs. We demonstrate salience by achieving state-of-the-an performance on the Omniglot few-shot classification benchmark and visual recognizability in one-shot generations. Then we explore how the embeddings capture image components and their spatial relationships to explore image space compositionality and also show a surprising property of conceptual composition.
We then push the boundary further by applying our sketch model to natural images we are the first to extend stroke-based autoregressive models to produce drawings of open domain natural images. We train model with adapted SVG images from the Sketchy dataset (Sangkloy et al. 2016) and then evaluate the embedding quality directly on unseen classes the mini- -maeeneeeNet task for few-shot classification (Vinyals al. 2016). Our approach is competitive with existing unsupervised few-shot learning methods (Hsu et al. (2019] Khodadadeh et al. 2019 Antoniou & Storkey 2019) on this natural image benchmark. In both the sketch and natural image domain,
knowledge,
show that by learning to draw, our methods generalize well even across different datasets and classes.

---

Under review as a conference paper at ICLR 2021
A. Drawing Imitation
Natural Image
2
T
CNN Encoder
RNN Decoder
Of
SketchEmbedding
Sketch Generations
Sketch
Test time
B. Few-shot Classification
C. Evaluating One-Shot Generation
Using SketchEmbedding
Martini Glass
Classifier
Figure A: A natural or sketch pixel image is passed into the CNN encoder to obtain Gaussian SketchEmbedding which is concatenated with the previous stroke 1/-1 as the decoder input at each timestep to generate yt- B+C: Downstream tasks performed after training is complete.
RELATED WORK
In this section review relevant literature including generating sketch-like images, unsupervised representation learning, unsupervised few-shot classification and "ketch-basee image retrieval (SBIR).
Autoregressive drawing models: Graves (2013) use LSTM to directly output the pen coordinates
to imitate handwriting sequences. SketchRNN (Ha & Eck 2018 builds on this by applying it to general sketches beyond characters. (Song al. (2018); Cao et al.
(2019); Ribeiro al. (2020) all extend SketchRNN through architectural improvements. Chen et al.
(017) change inputs to be pixel images. This and the previous 3 works consider multi-elass sketching, but none handle more than 20 classes. Autoregressive models also generate images directly in the pixel domain. DRAW (Gregor al., 2015) uses recurrent attention to plot pixels; Rezende al. (2016) extends this to one-shot generation and PixeICNN van den Oord et al. 2016) generates image pixels sequentially.
Other works produce sketch-like images based on style transfer or low-level image processing techniques. Classic methods are based on edge detection and image segmentation (Arbelaez al., [ & Tu (017). Zhang al. (2015) use CNN to directly produce sketch-like pixels for face images. Photo-sketch and pix2pix (Li et al., 2019; Isola al., (017)) propose a conditional GAN to generate images across different style domains. Image processing based methods do not acquire high-level image understanding, as all the operations are in terms of low-level filtering: none of the GAN sketching methods are designed to mimic human drawings open domain natural images.
Image processing methods & GANs:
Unsupervised representation learning: the sketch image domain, our method is similar to large category of generative models which learn unsupervised representations the principle of siillsii--b-synthestt Work by Hinton Nair (2005) operates sketch domain and learns to draw by synthesizing interpretable motor program. Bayesian Program Learning (Lake al.) 2015) draws through exact inference of common strokes but learning and inference are computa- tionally challenging. As such, variety of deep generative models aim to perform approximate Bayesian inference by using an encoder structure that directly predicts the embedding, e.g., deep autoencoders (Vincent al. 2010), Helmholtz Machine (Dayan al. 1995), variational autoencoder (VAE) (Kingma & Welling 2014), BiGAN (Donahue et al. 2017). etc. Our method is also related the literature self-supers supervised representation learning (Doersch al. 2015 Noroozi & Favaro 2016 Gidaris et al. 2018 Zhang et al. 2016)), as sketch strokes are part of the input data itself. In
few-shot learning (Vinyals et al. 2016 Snell et al., 2007 Finn et al.) (017), recent work has explored unsupervised meta-training CACTUs, AAL and UMTRA (Hsu et al. [0199] Antoniou & Storkey
2019 Khodadadeh et al. 2019) all operate by generating pseudo-labels for training.
Sketch- image retrieval (SBIR): In SBIR, model is provided sketch-drawing and retrieves a photo of the same class. The area is split into fine-grained (FG-SBIR) (Yu et al., 2016 Sangkloy
et al. 2016 Bhunia et al.] 2020) and zero-shot setting (ZS-SBIR) (Dutta & Akata 2019 Pandey
ct al. 2020: Dey et al. 2019). FG-SBIR considers minute details while ZS-SBIR learns hege-level
cross-domain semantics and joint latent space perform retrieval.
2

---

Under review as a conference paper at ICLR 2021
(a) Sketchy samples
(b) Quickdraw samples
Figure 2: Examples from Sketchy (Sangkloy et al. 2016) and Quickdraw JJngejaan et al. 2016
datasets. Sketchy examples are reshaped and padded to increase image-sketch spatial agreement.
3 LEARNING TO IMITATE DRAWINGS
Here we describe learning to draw through sketch imitation. Our architecture is generative encoder- decoder model with CNN encoder for pixel images and an RNN decoder to output vector sketches as shown in Figure| I Unlike other drawing models that only train on a single or few classes (Ha) & Eeck 2018: Chen et al. 2017), SketchEmbedNet is not limited by class inputs and sketch wide variety of images. We also introduce a differentiable rasterization function for computing an additional pixel-based training loss.
Input & output representation Unlike SketchRNN which encodes drawing sequences, we learn an image embedding by mapping pixels to sketches, similar to Chen et al. (2017). Training data for this task (adopted from Ha & Eck (2018)) consists of tuple (æz, y). where E H xC' is the input image and y E 7 is the stroke target. T is the maximum sequence length of the stroke data y, and each stroke y/: consists of 5 elements, (A., Ay $1, $2, $2). The first 2 elements are horizontal and vertical displacements on the drawing canvas from the endpoint of the previous stroke. The latter 3 elements are mutually exclusive pen states: 81 indicates the pen is on paper for the next stroke, 82 indicates the pen is lifted, and 83 indicates the sketch sequence has ended. Yo is initialized with (0, O, 0) start the generative process. Note that no class information is available while training.
SkeechEmbeddiin as a compositional encoding of images We use CNN to encode the input image and obtain the latent space representation z, as shown in Figure To model intra-class variance, is a Gaussian random variable parameterized by CNN outputs, similar to a VAE (Kingma Welling 2014). Throughout this paper, we refer to z as the SketchEmbedding. representations the embedding is trained to classify object classes, or to reconstruct the input pixels. Here, since the SketchEmbedding is fed into an RNN decoder to produce a sequence actions, is additionally encouraged to have compositional understanding of the object structure, instead of just an unstructured set of pixel features. For example when drawing the legs of a turtle, the model must explicitly generate each leg instance. While pixel-based models suffer from blurriness and in generating the image at once, does not distinguish between individual components such as the legs, body and head. The loss of this component information by pixel models has been observed in GAN literature (Goodfellow) 2017) which we propose is avoided by our sketching task.
typical image
drawing
To accommodate the increased training data complexity by including hundreds of classes, also upscale the size of our model in comparison to work by Chen et al. (2017); Ha & Eck 2018); Song al. (2018). The backbone is either 4-layer CNN (Conv4) (Vinyals et al. (0166) for consistent comparisons in the few-shot setting or a ResNet12 (Oreshkin et al. 2018)) which produces better drawing results. In comparison, Chen et al. (017) only use 2D convolution with maximum of filters.
RNN decoder The RNN decoder used SketchEmbedNet is the same as in SketchRNN (a & Eck 2018). The decoder outputs mixture density which represents the stroke distribution at each timestep. Specifically, the stroke distribution is a mixture some hyperparameter M bivariate Gaussians denoting spatial offsets as well as the probability of the three pen states $1-3. The spatial offsets A = (Ar, Ay) are sampled from the mixture of Gaussians, described by: (1) the normalized
mixture weight Ty; (2) mixture means u (pz, Py);; and (3) covariance matrices Vy. We further reparameterize each D1 with its standard deviation j = (aa, Tylj and correlation coefficient Pay.i- Thus, the stroke offset distribution p) Σド= (A py, Ez).
The RNN is implemented using a HyperLSTM (Ha et al. 2017); LSTM weights are generated at each timestep by smaller recurrent "hypernetwork" to improve training stability. Generation is autoregressive, using RR"' D concatenated with the stroke from the previous timestep yt-1. to form the input to the LSTM. Stroke y/-1 is the ground truth supervision at train time (teacher forcing), or sample v from the mixture distribution output by the model during from timestep / f-11 1.
3

---

Under review as a conference paper at ICLR 2021
Seen Class
Sketch
Seen Class
Sketch
Bugls
Unseen Class
Sketch
Unseen Class
Sketch
Sup
Figure 3: Sampled generated drawings of Quick-
Figure 4: Sampled drawings of mini-ImageNet examples. Weaker drawings boxed in red.
draw examples. Weaker drawings boxed in red.
3.1 LEARNING
We train the drawing model in an end-to-end fashion by jointly optimizing three losses: a pen loss pen for learning pen states, stroke loss Lurke for learning pen offsets, and our proposed pixel loss pps for matching the visual similarity of the predicted and the target sketch:
(I) where a is a loss weighting hyperparameter. Both fpen and Latroke were in SketchRNN, while the Epixel is our novel contribution to stroke-based generative models. Unlike SketchRNN, we do not impose a prior using KL divergence are not interested in unconditional sampling and it worsens quantitative results in later sections.
C = Cpen + (1 a)) Saroke + C
Pen loss The pen-states predictions (5/, 2/. s/) are optimized simple 3-way classification with the softmax cross-entropy loss, Lpon = - Σi E 8m.:log(s'm..!
Stroke loss
The stroke loss maximizes the log- ligllikelhoodd of the spatial offsets of each ground truth stroke A, given the mixture density distribution P: at each timestep: Laroke
7 > log pa A.).
Pixel loss While pixel-level reconstruction objectives are common in generative models (Kingma Welling 2014, Vincent al. 2010 Gregor et al. 2015), we introduce a pixel-based objective for vector sketch generation. After decoding. a differentialle rasterization function frasser is used to map the sketch into a pixel image. fraster transforms stroke sequence y into a set of 2D line segments (lo, 11), (41, 12) ... k.r) where Er= A,. It renders by fixing canvas dimensions, scaling and centering strokes before determining pixel intensity based on the L2 distance between each pixel to lines in the drawing. Further details on fraster can be found Appendix [ fmsterr applied to both the prediction y' and ground truth y. to produce two pixel images. Gaussian blur (.)ur()) is used to reduce strictness before computing the binary cross-entropy loss, Lpisel:
Golur (raster (y)), I'
Ubltur (fruster (z')), Cpisel
ΣΣ logg((()y)
HW
Curriculum training schedule We find that (in Equation [ is an important hyperparameter that impacts both the learned embedding space and the generation quality of SketchEmbedNet. A curriculum training schedule is used, increasing a to prioritize Epise relative Laroke as training progresses; this makes intuitive sense as single drawing can be produced by many different stroke sequences but learning to draw in fixed manner is easier. While Epen promotes reproducing specific drawing sequence, Liisel only requires that the generated drawing visually matches the image. Like a human, the model should learn follow one drawing style (a la paint-by-numberyy before learning to draw frrely.
4 DRAWING IMITATION EXPERIMENTS
In this section, we introduce our experiments on training SkerchEmbedNet using two sketching datasets. The first is based on pure stroke-based drawings, and the second consists of natural image and drawing pairs.
4

---

Under review as a conference paper at ICLR 2021
Table I: Few-shot classification results on Omniglot
Omniglot
(way, shot)
Algorithm
Encoder
Train Data
(5,1)
(5,5)
(20,1)
(20,5)
Training from Scratch (Hsu
2019
N/A
Omniglot
52.50 + 0.84
74.78 + 0.69
24.91 + 0.33
47.62
0.44
CACTUs MAML (Hsu et al..
2019
Cone4
Omniglot
68.84 1: 0.80
87.78 4: 0.50
48.09
4: 0.41
73.36 4: 0.34
PACTUU-ProtoNer el 2079
Conv4
Omniglot
68.12 0.84
83.58
0.61
47.75
0.43
66.27 0.37
AAL-ProtoNet (Antoniou & Storkey
2019
Conv4
Omniglot
84,66 = 0.70
88.41 = 0.27
68.79
= 1.03
74,05
= 0.46
AAL-MAML (Anloaioo & Storkey
2719
Conv4
Onniglot
88.40 0.75
98.00 + 0.32
70.20 1: 0.86
88.30 + 1.22
UMTRA (Khodadateh el
2019
Conv4
Omniglot
83.80
95.43
74.25
92.12
Random CNN
Conv4
N/A
67.96 + 0.44
83.85 + 0.31
44.39 + 0.23
60.87 + 0.22
Conv-VAE
Conv4
Omniglot
77,83 4: 0.41
92,91 0.19
62.59
0.24
84.01 0.15
Cony- VAE
Conv4
Quickdraw
81.49 -: 0.39
94,09 = 0.17
66.24 =: 0.23
86.02 =: 0.14
SkeschEmbeddiing (Ours)
Conv4
Omniglor
94.88 =: 0.22
99.01 = 0.08
86.18 + 0.18
96.69 = 0.07
SketchEmbeddding (Ours)
Conv4
Quickdraw*
96.96 2.17
99.50 0.06
91,67 1: 0.14
98.30 0.05
MAML (Supervisd))) (Finn et al. 2017
Conv4
Omniglot
94.46 0.35
98.83 + 0.12
84.60 + 0.32
96.29 + 0.13
ProtoNet (Supervised) (Snelr et 2017
Conv4
Omniglot
=
99.58 0.09
95.31 0.18
98.81 0.07
Stroke data used for training
Quickdraw: Stroke-based image sketching The Quickdraw (Jongejan et al. 2016) dataset con- sists of 345 classes of each with 70,000 examples, produced by human players participating in the game "Quick, Draw!". Examples from the Quickdraw dataset are shown in Figure 26) The input image æ is a direct rasterization of the drawing data y. 300 of 345 classes are randomly selected for training: æ is rasterized resolution of 28 x 28 and stroke labels y padded up to length T= 64. Any drawing samples exceeding this length were discarded. Note that this an unsupervised representation learning approach, as no class information is used by the system. Data processing procedures and class splits are in Appendix
Sketchy: Open domain natural image sketching We further extend our stroke-based generation model on open domain natural images. Here, the imput is an RGB photo, and the output a human drawing which does not align with the photo precisely and also does not match with the low-level image details. This is novel setting, as prior efforts by Ha & Eck (2018); Chen et al. (2017); /Snn et al. (2018) have only applied their sketch RNN models on the Quickdraw dataset or natural images with only two object classes (shoe/chair) and scrubbed backgrounds (Yu et al. 2016). Learning to sketch open domain natural images is very challenging as it requires the model to identify the subject and filter unnecessary details not present in the sketch. At test time, we further challenge our method by evaluating on unseen data distributions necessitating generalization over natural images.
For this task we use the Sketchy dataset (Sangkloy et al. 2016) which consists of ImageNet images paired with vector sketches for total of 56k examples after processing. Sketches with timestamps preserving their original drawing sequence which we adapt by sampling paths in this order. Images are also centered, padded and resized resolution 84 x 84 (see Figure 2a). We fix the maximum sequence length to T = 100, and use all 125 categories but remove classes that have overlapping child synsets with the test classes of mini- mini-ImageNNt (Vinyals et al. 2016). This enables testing on mini- ImageNe without any alterations to the benchmark. Once again this is an unsupervised learning formulation.
stored as SVGs
4.1 RESULTS AND VISUALIZATIONS
Figure 3 shows drawings conditioned on sketch image inputs. There is little noticeable drop in quality when we sample sketches from unseen classes compared to those it has seen hefore. [ggure]showe examples of sketches generated from natural images. While they are not fine-graine renditions, these sketches clearly demonstrate SketchEmbedNet's ability to capture key components of seen and unseen classes. The model effectively isolates the subject in each natural image and captures the circular and square shapes in the cakes and storefronts respectively. Even with the challenging lion images, it identifies the silhouette of the laying lion despite low contrast and appropriately localizes the one on the mountainside.
Unlike pixel-based auto-encoder models, our sketches do not follow the exact pose of the original strokes, but rather capture a general notion of component groupings. In the basketball example of Figure 3. the lines are not a good pixel-wise match to those in the original image yet they are placed in sensible relative positions. Weaker examples are presented in the last row of Figure 3 4 regardless, even poorer examples still capture some structural aspects of the original image. Implementation details can be found in Appendix B
In later sections we explore the uses of SketchEmbeddings and fix models for all downstream tasks.
5

---

Under review as a conference paper at ICLR 2021
Table 2: Few-shot classification results on mini- Amii-ImaeNNet
mini-ImageNet
(way, shot)
Algorithm
Backbone
Train Data
(5.1)
(5,5)
(5,20)
(5,50)
Training from Scratch Hsu et al.
2019
N/A
mini ImageNet
27.59 = 0.59
38.48 1: 0.66
51.53 =: 0.72
59.63 1: 0.74
MACTL--MAML Hssu et 2019
Cono4
mini Mini--mngeNNet
0.70
63.84
69.64 0.63
CACTUs-ProtoNe Htu 2019
Conv4
mini-ImageNett
39.18 = 0.71
$3.36
61.54 = 0.68
63.55 0.64
AoooProtoNet AAnoiiin Storkey
2019
Conv4
Neti-ImageNet
37.67 0.39
40.29
AAL-MAMI. (Aniomiou & Storkey
2019
Conv4
mini Nmt
34.57 0.74
49.18 0.47
UMTRA Khoddtaadh 2019
Conv4
mini ImageNet
39,93
50.73
61.11
67.15
Random CNN
Conv4
N/A
26.85 0.31
33.37
38.51 = 0.28
41.41
+ 0.28
Conv-VAE
Conv4
mini-ImageNet
0.21
26.22
29.93 + 0.21
32.57 + 0.20
VAE
Sketchy
30.41
Random CNN
ResNet12
NIA
28.59 1: 0.34
35.91 0.334
41.31 1 0.33
44.07
1 0.31
Conv-VAE
ResNot12
mini-ImageNet
28.16
33.64
0.27
Conv-VAE
RosNet12
Sketchy
24.61 = 0.23
28.85 + 0.23
35.72 = 0.27
40.44 + 0.28
SketchEmneeddingt (owrs)
Sketchy*
1:
63.34
0.32
SketchEmbedding (ours)
ReaNet12
Sketchy
40.39 0.44
57.15 + 0,38
67.60 = 0.33
71.99 + 0.3
MAML (supervisee) et 2017
ImageNt
0.72
71.03
75.54 0.62
ProtoNet (aupervised) (Snell et 2017
Conv4
mini ImageNet
46.56 0.76
62.29
1 0.71
70.05 +: 0.65
72.04 1: 0.60
Stroke for training
5 FEW-SHOT CLASSIFICATION USING SKETCHEMBEDDING
We would like to assess the benefits of learning to draw by performing few-shot classification with learned embedding space. Examining performance on discriminative tasks reveals that learning to imitate sketches allows the embeddings to capture salient information of novel object classes. Below describe our few-shot classification procedure and summarize results on the Omniglot (Lake et al. 2013) and mini-ImageNet benchmarks (Vinyals et al. 2016).
Comparison to unsupervised few-shot classification model is not provided with any class labels during meta-training, until it is given few labeled examples (" ("shots") of the novel classes at meta-test time. While our model is provided a "target" - sequence of strokes-due during training, it is not given any class information. Therefore, we propose that the presented sketch imitation training, though it uses sketch sequences, is comparable to other class-label-free representation learning approaches (Berthelot et al., 2019 Donahue et al. 2017 Caron et al.) 2018) and the learned SketchEmbeddlings can be applied to unsupervised few-shot classification methods.
unsupervised few-shot classification, a
In our experiments, we compare to previous unsupervised few-shot learning approaches: CAC- TUs (Hsu et al., 2019). AAL (Antoniou & Storkey 2019), and UMTRA (Khodadadeh et al. 2019). These methods create pseudo-label labels during meta- trainin using either clustering or data augmentation. As additional baselines, Conv- VAE (Kingma Welling 2014) and a random CNN are also included, both using the same Conv4 backbone.
Few-shot experimental setup The CNN encoder of SketchEmbedNet is used as an embedding function combined with a linear classification head to perform few-shot classification. The embedding is made deterministic by taking the mean of the random normal latent space and discarding the variance parameter from the encoder. Otherwise, the conventional episodic setup for few-shot classification is used; each episode consists of a labeled "support" set of N x K (N-way K-shot) embeddings and an unlabeled "query" set. The linear classification head is trained on the labeled support set and evaluated on the query set.
5.1 FEW-SHOT CLASSIFICATION ON OMNIGLOT
The Omniglot (Lake et al., 2015) dataset contains 50 alphabets, 1623 unique character types, each with 20 examples and is presented as both a greyscale image and a stroke drawing. We use the same train-test split as Vinyals et al. (2016) along with randomly sampled episodes. Experiments using the more challenging Lake split where episodes sampled within alphabet, proposed by Lake et al. (2015), are Appendix E and random seed experiments in Appendix F
To ensure a fair comparison with other few-shot classification models, use the same convolutional encoder (Conv4) as Vinyals et al. (0016). Results from training only on Omniglot (Lake et al. 2015 also presented to demonstrate effectiveness without the larger Quickdraw(Jongee) dataset. No significant improvements were observed using the deeper ResNNet12 et al. 2018) architecture; additional results are in Appendix
al. 2016

---

Under review as a conference paper at ICLR 2021
A. Component arrangements
B. Spatial relationships
wia
VAC
SketchEnbedciig
SketchEmlbeddinn
00
+00
O
C. Conceptual composition
8
- +
8.0+ 6
= d
- P --
+
-
VAE
SketchEmbeddiing
Figure 5: Experiments exploring properties SketchEmbeddings. Examples colored for understand-
ability only.
All our methods out- out-perform the previous state-of-the-art on the unsupervised Omniglot benchmark (Table The Quickdraw trained model surpasses supervised MAML (Finn et al. (211)), and is on par with a supervised ProtoNet (Snell et al. 2017) model, especially in the 5-shot settings. Both baselines, Conv-VAE and random CNN, perform well compared to other unsupervised methods.
5.2 FEW-SHOT CLASSTFICATION ON MINI-IMAGENET
We extend our investigation and assess SketchEmbeddings for the classification of natural images in the mini-ImageNet benchmark (Vinyals al.) (0166). The same CNN encoder model from the natural image sketching task is used to match the visual domain of the examples we hope to classify.
The mini-ImageNet (Vinyals et al. 2016) dataset consists of 100 classes each with 600 examples. The setup proposed by Ravi & Larochelle (2017) used, where the classes are split 64- 16-20 for training, validation and test. As noted earlier, any examples in the Sketchy dataset that are also present in the mini-ImageNet test were removed by filtering the synset (and children synset) IDs ensuring train and test classes disjoint.
Natural image classification is a far more challenging problem. Learning to reconstruct pixels of an image actually worsens results; the trained Conv-VAL is outperformed by the VAE with random weights. However, sketch reconstruction still valuable task; our models competitive while our best model out-performs previous state-of the- unsupervised methods on few-shot settings. A full table is in Appendix [ seeding results are Appendix F
Classification results on mini-ImageNet are shown in Table 2
5.3 SKETCHING TO LEARN CLASS-IDENTIRAAHE INFORMATION
Existing sketch works have focused on generating better drawings or unifying sketches with other image domains. We present a new paradigm: using sketching as an auxiliary task to learn visual content. Only by training drawing model that can sketch general image inputs are we able to transfer the learned understanding to new data distributions. By considering the stroke distribution of the Quickdraw dataset, we are able to interpret image inputs from the separate Omniglot dataset and tackle the few-shot classification task with surprising accuracy.
While the natural image sketching task is challenging and does not always produce high-fidelity results, still learns useful visual information. By training on the Sketchy dataset, we learn how to draw other data distributions for which sequential stroke data exists. Then, by knowing how to sketch this mini- ImageNet data we are able to produce distinguishable embeddings that enable competitive few-shot classification performance.
Varying weighting of pixel-loss For both settings we sweep the pixel loss coefficient mmax to ablate its impact model performance the Omniglot task (Table There is substantial improvement in few-shot classification when O'max is non-zero. Ommx = 0.50 achieves the best results, and the trend goes downward when Omax approaches to 1.0, i.e. the weighting for Lamoke goes to 0.0. This is
7

---

Under review as a conference paper at ICLR 2021
Sketch Embeciding
Rezende et al.
Meta PixelCNN
Support
T
-
A
6
7 D
43
3
9 2
v
9 3 2
v
Generated
T
L
7 77 0 t
9 3 v 2 9 1 3 9 7 v
Samples
3 13000
7 n n
3
2 5
Figure 6: One-shot Omniglot generation compared to Rezende et al. (0016): Reed et al. (2017)
reasonable the training of SketchEmbedNNet is more stable under the guidance of ground truth strokes
PROPERTIES OF SKETCHEMBEDDINGS
We hypothesize that reproducing a sketch drawing rather than a pixel-based approach requires the preservation of more structural information due to sequential RNN generation. By learning in this manner, SketchEmbeddings space. We examine this compositionality through several comparisons of SketchEmbeddinng with those generated by Conv-VAL
aware of spatial properties and the composition of elements in image
Component arrangements We construct examples that contain the same set of objects but in different arrangements to test sensitivity to component arrangement and composition in image space. We then embed these examples with both generative models and project into 2D space using UMAP (Meinnes al. (018)) visualize their organization. In the first panels of Figure 55A, we see that the E are easily separated in unsupervised clustering. The rightmost panel of Figure A exhibits non-synthetic classes with duplicated shapes; snowmen with circles and televisions with squares. With these results, we demonstrate the greater component level awareness of SkedchEmeeddings. The rearranged shapes and the nested circle and squares have similar silhouettes that are difficult to differentiate to conventional pixel loss. To SketchEibeeddings the canvas offset and different drawing sequence of each shape make them substantially different in embedding space.
Drawing also builds awareness of relevant underlying variables, such as spatial relationships between components of the image. We examine the degree to which the underlying variables of angle, distance or size are captured by the embedding, by constructing images that vary along each dimension respectively. The embeddings are again grouped by 2D projection in Figure 5 using the UMAP (Mcinnes al.) 2018) algorithm. When clustered, the 2D projection of SkelchEmbeddings arranges the examples along axis corresponding to the latent variable compared to the Conv-VAE embeddings which is visibly non-linear and arranges in clusters. This clear axis alignmett suggests greater level of latent variable disentanglement in the SketchEmbeddings.
Spatial relationships
Conceptual composition Finally, we explore concept space composition using SketchFibeddinng (Figure C) by embedding different Quickdraw examples then performing arithmetic with the latent vectors. By subtracting circle embedding and adding square embedding from snowman composed of stacked circles, we produce stacked boxes. This property of vector arithmetic is reminiscent of language representations, Queen (Ethayarajh et al.) 2019). Our results indicate that this property is captured to a greater degree in SketchEmbedding than in the pixel-based VAE embeddings. Composing SketchEmbeddings produces decoded examples that appeal to our intuitive conceptual understanding while the VAE degrades blurry, fragmented images. We provide more examples the setup in F as well as additional classes in Appendix
evidenced in analogies like King Man Woman
7 ONE-SHOT GENERATION
To evaluate the sketches generated by our model, make qualitative comparisons to other one-shot generative models and quantitatively assess our model through visual classification via a ResNet101
(He et al. 2016). In this section, all models use the ResNet12 (Oreshkin et al. 2018 backbone.
Qualitative comparisons We compare SketchEmbedNet one-shot generations of Omniglot char- acters with examples from other few-shot (Reed et al. 2017) and one-shot (Rezende et al.) 2016)
8

---

Under review as a conference paper at ICLR 2021
Table 3: Effect of pixel loss coefficient on Omniglot few-shot classification
Table 4: ResNet-101 45-way classification score on 1-shot generated sketches of seen and unseen classes.
Ommax
20-way I-shot Acc.
Generation Method
Seen
Unseen
0.00
87.17 0.36
Original Data
97.66
96.09
0.25
87.82 0.36
Conv-VAE
76.28 0.93
75.07
0.84
0,50
91.39 + 0.31
SketchFmmeddNtt
81.44 + 0.95
77.94 + 1.07
0.75
90.59 + 0.32
0.95
89.77 + 0.32
approaches (Figure 66_ In the settings shown, none of the models have seen any examples from the character class, the parent alphabet. Furthermore, the drawer has seen written characters during training and is trained only on the Quickdraw dataset. Visually, our generated images better resemble the support examples and the variations by stretching and shifting strokes better preserves the semantics of each character. Generations in pixel space may disrupt strokes and alter the character human perception. This is especially true for written characters as they are frequently defined by specific set of strokes instead of blurry clusters of pixels.
Quantitative evaluation of generation quality Per-pixel metrics like in Reed et al. (2017); Rezende et al. (2016) may penalize generative variance that still preserves meaning. We propose an Inception Score (Salimans et al. 2016) inspired metric to quantify class-recognizaabliiy and generalization of generated examples. We train two separate ResNet classifiers (He et al., 2016), each on a different of 45 Quiekdraw classes. One set was part of the training set of SketchEmbedNet (referred to "seen") and the other set was held out during training (referred to as "unseen"). We then have SketchEmbedNet generate one-shot sketches from each set and have the corresponding classifier predict class. The accuracy of the classifier on generated examples is compared with its training accuracy in Table [ For a ResNet classifier, SketchEmbedNNtt generations more recognizable for both classes seen and unseen classes.
Evaluating generative models is often challenging.
CONCLUSION
Learning to draw is not only an artistic pursuit but drives a distillation of real-world visual concepts. We present generalized drawing model capable of producing accurate sketches and visual summaries of open-domain natural images. While sketch data may be challenging to source, we show that training to draw either sketch or natural images can generalize for downstream tasks, not only within each domain but also well beyond the training data. More generally research in this direction may lead to more lifelike image understanding inspired by how humans communicate visual concepts.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Antreas Antoniou and Amos Storkey. Assume, augment and learn: Unsupervised few-shot meta-learning via random labels and data augmentation. CoRR, abs/1992.09984 2019.
Pablo Arbelaez, Michael Maire, Charless C. Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 33(5):898 916, 2011.
David Berthelot, Colin Raffel, Aurko Roy, and Ian J. Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. In 7th International Conference on Learning Representations, ICLR, 2019.
Ayan Kumar Bhunia, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, and Yi-Zhe Song, Sketch less for more: On-the-ffy fly fine- mne-grained sketch-base image retrieval. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR, 2020.
Nan Cao, Xin Yan, Yang Shi, and Chaoran Chen. Al-Sketcher: deep generative model for producing high-quality sketches. In The Thirty-Thirr AAAI Conference on Artificial Intelligence, AAAI, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In 15th European Conference on Computer Vision, ECCV, 2018.
Yajing Chen, Shikui Tu, Yuqi Yi, and Lei Xu. Sketch-pix2seq: a model to generate sketches of multiple categories. CoRR, abs/1709.044 709.04121,122 2017.
Peter Dayan, Geoffrey E Hinton, Radford M Neal, and Richard S Zemel. The helmholtz machine. Neural computation, 7(5):889-904, 1995.
Sounak Dey, Pau Riba, Anjan Dutta, Josep Llados, and Yi-Zhe Song. Doodle to search: Practical zero-shot sketch-based image retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
Carl Doersch, Abhinav Gupta, and Alexci A. Efros. Unsupervised visual representation learning by context prediction. In IEEE International Conference on Computer Vision, ICCV, 2015.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In 5th Interna- tional Conference on Learning Representationss ICLR, 2017.
David H. Douglas and Thomas K. Peucker. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. 1973.
Anjan Dutta and Zeynep Akata. Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval. CoRR, 0t//1993.033772 2019.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A bensity-based algorithm for dis- covering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD, 1996.
Kawin Ethayarajh, D. Duvenaud, and Graeme Hirst. Towards understanding linear word analogies. In ACL, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML, 2017.
Dileep George. Wolfgang Lehrach, Ken Kansky, Miguel Marthi, Xinghua Lou, Zhaoshi Meng, Yi Liu, Huayan Wang, Alex Lavin, and D. Scott Phoenix. A generative vision model that trains with high data efficiency and breaks text-based captchas. Science, 358(6368), 2017. ISSN 0036-8075. doi: 10.1 126/science.aag2212.
Christopher Laan, Bhaskara
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In 6th International Conference on Learning Representations, ICLR, 2018.
Ian J. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. CoRR, abs/ 1701 00000006 2017.
10

---

Under review as a conference paper at ICLR 2021
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets, In Advances in Neural Information Processing Systems 27, NIPS, 2014.
Alex Graves. Generating sequences with recurrent neural networks. CORR, abs/1308.0855 2013.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. DRAW: A recurrent neural network for image generation. In Proceedings of the 32nd International Conference on Machine Learning, ICML, 2015.
David Ha and Douglas Eck. A neural representation of sketch drawings. In 6th International Conference on Learning Representations, ICLR, 2018.
David Ha, Andrew M. Dai, and Quoc V. Le. Hyperneworrs., In 5th International Conference on Learning Representations, ICLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Compuler Vision and Patlern Recognition, CVPR, 2016.
Luke B. Hewitt, Maxwell I. Nye, Andreea Gane, Tommi S. Jaakkola, and Joshua B. Tenenbaum. The variational homoencoder: Learning learn high capacity generative models from few examples. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017.
net wtt.t//ooommmttmmm Lttp://oommmmmyyy2//0
Geoffrey H. Hinton and Vinod Nair. Inferring motor programs from images of handwritten digits. In Advances in Neural Information Processing Systems 18, NIPS, 2005.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning. In 7th International Conference Learning Representations, ICLR, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.00116 2015.
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2017.
Jongejan, Rowley, T. Kawashima, Kim, and N. Fox-Gieg. The quick, draw! A.I. experiment., 2016. UtLpttps: http:::://quiooe //tp////uuuccaaa// withgoogle com/
Siavash Khodadadeh, Ladislau Bölöni, and Mubarak Shah. Unsupervised meta-learning for few-shot image classification. In Advances in Neural Information Processing Systems 32, NeurIPS, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Leurning Representations, ICLR, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR, 2014.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 35066266; 2015. doi: 10. 1266 ac3500
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. The omniglot challenge: a 3-year progress report. Current Opinion in Behavioral Sciences, 29:97-104, Oct 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))2278-233 1998.
Mengtian Li, Zhe L. Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan. Photo-sketching: Inferring contour drawings from images. In IEEE Winter Conference on Applications of Computer Vision, WACV, 2019.
11

---

Under review as a conference paper at ICLR 2021
M. Liwicki and H. Bunke. lam-ondb on-line english sentence database acquired from handwritten text on a whiteboard. In Eighth International Conference on Document Analysis and Recognition, ICDAR, 2005.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2 2008.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03422 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. 14th European Conference on Computer Vision, ECCV, 2016.
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems 31, NeurIPS, 2018
Anubha Pandey, Ashish Mishra, Vinay Kumar Verma, Anurag Mittal, and Hema A. Murthy, Stacked adversarial network for zero-shot sketch based image retrieval. In Proceedings of the IEEE Winter Conference on Applications Computer Vision, WACV, 2020.
Marc' Aurelio Ranzato, Christopher S. Poultney, Sumit Chopra, and Yann LeCun. Efficient learning of sparse representations with an energy-based model. In Advances in Neural Information Processing Systems 19, NIPS, 2006.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In 5th International Conference on Learning Representations, ICLR, 2017.
Scott E. Reed, Yutian Chen, Thomas Paine, Aäron van den Oord, S. M. Ali Eslami, Danilo Jimenez Rezende, Oriol Vinyals, and Nando de Freitas. Few-shot autoregressive density estimation: Towards learning to learn distributions. CoRR, (00117100100044 2017.
Danilo Jimenez Rezende, Shakir Mohamed, Ivo Danihelka, Karol Gregor, and Daan Wierstra. One- shot generalization in deep generative models. In Proceedings of the 33nd International Conference on Machine Learning, ICMI., 2016.
Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti. Sketchformer: Transformer- based representation for sketched structure. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR, 2020.
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems 29, NIPS, 2016.
Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchy database: learning to retrieve badly drawn bunnies. ACM Trans. Graph., 35(4): 19:1-119:12, 2016
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems 30, NIPS, 2017.
Jifei Song, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, and Timothy M. Hospedales. Learning to sketch with shortcut cycle consistency. In IEEE Conference Computer Vision and Pattern Recognition, CVPR, 2018
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In Advances in Neural Information Processing Systems 29, NIPS, 2016.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie. Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in deep network with local denoising criterion. J. Mach. Learn. Res., 11:3371 10377--348.,. 2010.
12

---

Under review as a conference paper at ICLR 2021
Oriol Vinyals, Charles Blundell, Tim Lillierap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems 29, NIPS, 2016.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. Int. J. Comput. Vis., 125 2017.
Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales, and Chen Change Loy. Sketch me that shoe. In IEEE Conference Computer Vision and Puttern Recognition, CVPR, 2016.
Liliang Zhang, Liang Lin, Xian Wu, Shengyong Ding, and Lei Zhang. End-to-end photo-sketch generation via fully convolutional representation learning. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, ICMR, 2015.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In 14th European Conference on Computer Vision, ECCV, 2016.
13

---

Under review as a conference paper at ICLR 2021
A RASTERIZATION
The key enabler of our novel pixel loss for sketch drawings is our differentiable rasterization function fraster- Sequence based loss functions such Lumke are sensitive the order of points while in reality, drawings are sequence invariant. Visually, a square is square whether it is drawn clockwise or counterclockwise.
The purpose of sketch representation is to lower the complexity of the data space and decode in more visually intuitive manner. While it is necessary departure point, the sequential generation of drawings is not key to our visual representation and we would like SketchEmbedNet to be agnostic to any specific sequence needed draw the sketch that is representative of the image input.
To facilitate this, we develop our rasterization function fratter which renders an input sequence of strokes pixel image. However, during training, the RNN outputs a mixture Gaussians each timestep. To convert this to stroke sequence, we sample from these Gaussians; this can be repeated to reduce the variance of the pixel loss. We then scale our predicted and ground truth sequences by the properties of the latter before rasterization.
Stroke sampling the end of sequence generation we have N2 x (6M +3) parameters, Gaussian mixture parameters, pen states, N₂ times, one for each stroke. To obtain the actual drawing sample from the mixture of Gaussians:
Arr Aya
Mz,t [yy.t]
Cx,t
0
+
Pry
pry..
e, E, ~ NMO,N (2) 12).
(3)
After sampling we compute the cumulative sum of every stroke over the timestep so that obtain the absolute displacement from the initial position:
xt
Ar
(4)
y/1
A Ayr
Yt,abs
(1r,, Dt. 81, $2 8z).
(5)
Scaling Each sketch generated by our model begins (0,0) and the variance of all strokes in the training set is normalized to 1. On a fixed canvas the image is both very small and localized to the top left corner. We remedy this by computing a scale À and shift Tsiffs Ushhitt using labels y and apply them to both the prediction y' as well as the ground truth y. These parameters are computed as:
W
min
(6)
Temax
Fmin
Imax
Ymin
Tmax + Frii
Umux + Ymni
Xahin
X,
Ushin
X.
2
2
Tmax, Trins ymax, Ymin are the minimum and maximum values 2t from the supervised stroke labels and not the generated strokes. W and H are the width and height pixels of our output canvas.
Calculate pixel intensity our H W canvas.
Finally we are able to calculate the pixel Pt intensity of every pixel in
).
(dis(( (Ir-1, Bt-1), (xx, vt)) + (1 [s1. +11111.111 1010=111000)))))))1))))))) 1110))
Pig = 2-5x
min L.
(8)
where the distance function is the distance between point (i,j) from the line segment defined by the absolute points (Ie-1, 1t-1) and (x,, yt). We also blow up any distances where 81,--1 < 0.5 not render any strokes where the pen is not touching the paper.
14

---

Under review as a conference paper at ICLR 2021
B IMPLEMENTATION DETAILS
We train our model for 300k iterations with batch size of 256 for the Quickdraw dataset and 64 for Sketchy due to memory constraints. The initial learning rate is le-3 which decays by 0.85 every 15k steps. We the Adam (Kingma & Ba 2013) optimizer and clip gradient values at 1.0. used for the Gaussian blur in Lixel For the curriculum learning schedule, the value of a is set 0 initially and increases by 0.05 every 10k training steps with an empirically obtained cap at Qmax = 0.50 for Quickdraw and Omax = 0.75 for Sketchy.
2.0
The ResNet12 (Orehkin et al. 2018) encoder uses 4 ResNet blocks with 64, 128, 256, 512 filters re- spectively and ReLU activations. The Conv4 backbone has 4 blocks of convolution, batch norm (loffe) Szegedy 2015). ReLU and max pool, identical to Vinyals al. (2016). We select the latent space to be 256 dimensions, RNN output size to be 1024, and the hypernetwork embedding size to be 64. We use a mixture of M 30 bivariate Gaussians for the mixture density output of the stroke offset distribution.
C LATENT SPACE INTERPOLATION
Like in many mncoding-decodde models we evaluate the interpolation of our latent space. We select 4 embeddings at random and use bi-linear interpolation to produce new embeddings. Results are in Figures 7a and 7b
88
80
5
OL
(a) Interpolation of classes: power outlet, snowman,
(b) Interpolation of classes: cloud, power outlet, bas- ket. compass
jacket, elbow
Figure 7: Latent space interpolations of randomly selected examples
We observe that compositionality also present these interpolations. In the top row of Figure 7a the model first plots a third small circle when interpolating from the 2-circle power outlet and the 3-circle snowman. This small circle is treated as single component that grows as it transitions between classes until it's final size in the far right snowman drawing.
Some other based sketching models (Ha & Eck 2018 Chen et al. 2017 experience other classes materializing in interpolations between two unrelated classes. Our model does not exhibit this same behaviour as our embedding space is learned from more classes and thus does not contain local groupings of classes.
15

---

Under review as a conference paper at ICLR 2021
D EFFECT OF ON FEW-SHOT CLASSIFICATION
We performed additional experiments exploring the impact of our curriculum training schedule o. The encoding component of our drawing model was evaluated on the few-shot classification task for different values of Omma every 25k iterations during training. A graph shown in Figure g and the full table of all values of max is in Table
atpha, max
0.94
appa_max=o mac.
0.90
Oeniggtt
0.88
Berelions
Figure 8: Few-shot classification accuracy of Ommax values 0.0 and 0.5 over training
Table 5: Few-shot classification accuracy of all O'max values
max
25k
50k
75k
100k
125k
150k
175k
200k
225k
250k
275k
300k
0.00
89.35
87.94
88.73
88.46
88.01
88.04
88.23
87.73
88.03
87.86
87.65
87.17
0.25
89.21
90.39
90.20
89.75
87.78
88.37
88.64
88.05
87.98
88.41
88.15
87.82
0.50
90.48
89.58
89.81
89.02
90.68
91.24
90.26
90.94
91.12
91.30
91.12
91.39
0.75
91.39
89.95
89.56
89.81
89.95
90.79
91.02
91.09
91.82
90.76
91.42
90.59
0.95
90.23
90.15
90.10
89.55
90.27
92.37
92.27
90.29
91.58
91.02
89.73
89.77
E INTRA-ALPHABET LAKE SPLIT
The creators of the Omniglot dataset and one-shot classification benchmark originally proposed an intra-alphabet classification task. This task is more challenging than the common Vinyals split as characters from the same alphabet may exhibit similar stylistics of sub-components that makes visual differentiation more difficult. This benchmark has been less explored by researchers; however, still present the performance of our SketchEmbedding model against evaluations of other few-shot classification models on the benchmark. Results are shown Table 0
Table 6: Few-shot classification results on Omniglot (Lake split)
Omniglot (Lake
shot)
Algorithan
(5,1)
(5,5)
(20,1)
(20,5)
Coev-VAR
Convá
Quickdraw
73.12 + 0.58
88.50 0.39
53.45 + 0.51
73.62 0.48
Sketchhinbeeddnng (Ouns)
Conva
Quickdraw
89.16 0.41 0.41
97.12
0.18
74.24
0.48
89
0.25
SketchFmbeedding (Ouxs
ResNer12
Quickdraw
91.03 0.03--0..3
97.91
0.15
77.94
0.44
92.49
0.21
BPL (Supervised) et 2015 2019
NIA
Omniglat
96.70
ProtuNel (Supersivat Smell d AU Takeet 2019
Cone4
Omnigllt
86 30
RCN (Sspervieed) (iaenn
NAA
Omniglot
92.70
VHE (Suervized)
Omnigllt
16

---

Under review as a conference paper at ICLR 2021
Unsurprisingly, our model is outperformed by supervised models and does fall behind by more substantial margin than in the Vinyals split. However, our SketchEmbedding approach still achieves respectable classification accuracy overall and greatly outperforms Cony-VAE baseline.
F EFFECT OF RANDOM SEEDING ON FEW-SHOT CLASSIFICATION
The training objective for SketchEmbedNet reproduce sketch drawings of the input. This task is unrelated to few-shot classification may perform variably this variance by training our model with 15 unique random seeds and evaluating the performance of the latent space the few-shot classification tasks.
given different initialization. We quantify
We disregard the per (evaluation) episode variance of our model in each test stage and only present the mean accuracy. We then compute new confidence interval over random seeds. Results are presented in Tables
Table 7: Random Seeding on Few-Shot Classification results on Omniglot (Conv4)
(way, shot)
Seed
(5,1)
(5,5)
(20,1)
(20,5)
96.45
99.41
90.84
98.08
96.54
99.48
90.82
98.10
3
96.23
99.40
90.05
97.94
4
96.15
99.46
90.50
97.99
5
96.21
99.40
90.54
98.10
6
96.08
99.43
90.20
97.93
7
96.19
99.39
90.70
98.05
8
96.68
99.44
91.11
98.18
9
96.49
99.42
90.64
98.06
10
96.37
99.47
90.50
97.99
11
96.52
99.40
91.13
98.18
12
96.96
99.50
91.67
98.30
13
96.31
99.38
90.57
98.04
14
96.12
99.45
90.54
98.03
15
96.30
99.48
90.62
98.05
Average
96.37
0.12
99.43 0.02
90.69 + 0.20
98.07 + 0.05
Table 8: Random Seeding on Few-Shot Classification results on Omniglot (ResNet12)
(way, shot)
Seed
(5,1)
(5,5)
(20,1)
(20,5)
96.61
99.58
91.25
98.58
2
96.37
99.52
90.44
98.40
3
96.04
99.58
89.86
98.27
4
96.44
99.50
90.76
98.40
5
95.95
99.52
89.88
98.29
6
95.63
99.45
89,28
98.17
96.24
99.52
89,90
98.34
8 95.41
99.45
88.75
98.05
9
96.04
99.49
89.70
98.24
10
95,40
99.41
88.91
98.05
11
95.82
99.51
89.67
98.24
12
96.25
99.51
90.21
98.28
13
95.84
99.53
89.71
98.18
14
96.04
99.56
89.89
98.31
15
96.04
99.57
89.97
98.32
Average
96.00 0.31
99.51 + 0.04
89.89 0.56
98.27 + 0.12
17

---

Under review as a conference paper at ICLR 2021
Table 9: Random Seeding on Few-Shot Classification results on Imai-emaeeNet
(way, shot)
Seed
(5,1)
(5,5)
(5,20)
(5,50)
1
37.15
52.99
63.92
68.72
2
39.38
55.20
65.60
69.79
39.40
55.47
65.94
70.41
4
40.39
57.15
67.60
71.99
5
38.40
54.08
65.36
70.08
6
37.94
53.98
65.24
69.65
7
38.88
55.71
66.59
71.35
8
37.89
52.65
63.42
68.14
38.25
53.86
65.02
69.82
10
39.11
55.29
65,99
69.98
11
37.39
52.88
63.66
68.33
12
38.24
53,91
65.19
69.82
13
38.62
53.84
63.83
68.69
14
37.73
53.61
64.22
68.41
15
39.50
55.23
65.51
70.25
Average
38.55 + 0.45
54.39 + 0.63
65.14 + 0.59
69.69 + 0.56
G DATA PROCESSING
G.1 QUICKDRAW
We apply the same data processing methods in Ha & Eck (2018) with no additional changes to produce our stroke labels y. When rasterizing for our input æ, we scale, center the strokes then pad the image with 10% the resolution in that dimension rounded to the nearest integer.
The following list of classes were used for training:
clock, ambulance, angel, unimal migration, ant, apple, ams, repongus, bonan, harn, Inschall, boseball but. bothub, beach, hoar. hed, hen, bell, bench, bicycle, hinceulars, hird, blueberry, book, bonmerang. bottlecsp,
Mona
tridge, broccoli,
bulldozer, bush, buttenly.
calculater, calendar,
camoullage, camplire,
candle, cannon, car, carrot, castle, cat, ceiling fan, cell phone, cello, chair, chandelier. church, circle, clarinet, clock, coffee computer, cookie, couch, sow, crayon. crocodile, crown, ship,
diving board. dog. dolphin, fan, feather, fence, finger, thre tydrant, Breplace, firetmik,, fish, flamingo, fishhligh, flip flops, flower, foot, fock, frog. frying garden, garden hose, girffle, goatee, grapes, grass, guitar, hamburger. hand, harp, hat, headphones, hedgshog, helieayeer, helmet, hockey pack, hockey stick, horse, hospital, hourghess, house, house plant,
dragon, dresser,
cur, eye, eyeglasses,
halloon, dog.
key, keyboard, knoe, knife, lnkler, lumien,, leaf, light hull, lighter. lighthouse, lightingg line, lipstick, lobster, muilbox
map, marker, matches, megaphone, mermaid, microyhone, micnoaae, monkey, mosquito, motorbike, mountinn, mouse, monstache, mouth, mushroom, muil, necklace.
octopus,
púintrushh palm parachute, passport, peanur, pezr, pencil. penguin, piano, pickup suck, pig, pinsapple, police
car, pool, popsicle, posicard, purse, rabbit, raccoon, rddio min, minbow, rake, remose control, thinoceros, river, rolkensates, sailboat, sandwich, sixophone, scissoes,
shack, sheep. shoe, shorts, shovel, sink. skull, sleeping bag, smiley face, small, snake, snowflake,
ball, speedbout, square, steak, siithess
stop sign, straweerry, stretlight, string bean, suhmarine, sun swing sct, syring,, t-hhrt, table, teapot, seddy-bear, tennis maquer, test, figer, toe, tooth, toothppste. traffic light, nrangle, troetoone
tumpet, umbrella, underwar,
watermelon, windmill, wine bottle glass, wristwarch,
zigog. blackberry power oulle, peas, hot tub, toothbush, skateboard, cloud, albow. but, pond, compuss, elephant, hurricane, juil, school bus, skystraper, tornado, picture frame, lollipop, spoon, sim, cup, coaster, pants, jacket, rifle, yoga, uoe, television, bowtie, sweater. bockpack, zehra, suitease, stairs, The Great Wall of China
ase, snowmnn
osagon, washing machine.
G.2 OMNIGLOT
We derive our Omniglot tasks from the stroke dataset originally provided by Lake et al. (2015) rather than the image analogues. We translate the Omniglot stroke-by-stroke format to the same one used in Quickdraw. Then we apply the Ramer-Done Douger-oogus-eeeee Peucker (Douglas & Peucker 1973) algorithm with an epsilon value of 2 and normalize variance to 1 to produce y. We also rasterize our images in the same manner as above for our input xc
G.3 SKETCHY
Sketchy data is provided as SVG image composed of line paths that either straight lines Bezier curves. To generate stroke data we sample sequences of points from Bezier curves at a high resolution that then simplify with RDP, 5. We also eliminate continuous strokes with short
18

---

Under review as a conference paper at ICLR 2021
path length small displacement to reduce stroke length and remove small and noisy strokes. Path length and displacement are considered with respect to the scale of the entire sketch.
Once again normalize stroke variance and rasterize for our input image in the same manners as above.
The following classes were use for training after removing overlapping classes with mini- Mini-ImageNet:
hot-air balloon, violin, tiger, eyeglasses, mouse, jack-o-lantert lobster, teddy _beur, teapot, helicopter, duck, wading_birt nibor, penguin, sheep. windmill, pisso, jel- crocodilian, volcano, kndfe, cup. saxogtonee pistol, swan, chicken,
lytish, table, bectie, cabua, scoepion, scissors, bananz, hordog, camel, turkle spider, songhird, rifle, chiir, starfihh, sirplane. bread, bench, rocket, bicycle, owl, squirel, hermit crsb, hoss,
dlant_clock,
seigill, blimp, apple, pryser, rimper, frog. lizadd, me. sen turtle, peatzel, smail, butterfly, hear, my, wine bottle elephant, raceoon, thinscenos, door, snale, upe, Blwer. (sedan), kangaroo, dolphin. hambuger. castle. pincapple, zeben cundle, racket, church, fish, mushroom, strawberry, window, sailbout. hourglass, cat, shoe, hedgehog, couch, ginafi, hammer, motorcylle, shark
H AUTOREGRESSIVE DRAWING MODEL COMPARISONS
We summarize the key components of SketchEmbedNet in comparison to other autoregressive drawing models in Table I0
Table 10: Model comparisons between generative autoregressive models that produce pixel vector
sketch drawings.
slatccinn
Madel
Tatart
Fileus
Eneader
Ticoder
Gigner
IAM Lmill
Desctt
Sige
BMN
Morte UNN
DRAM (Cinroaa
SV12 Notp 2004 punt
Mi MOSTTE ot DOCE
BNN
UNN
Cous
Mouue Deniy
Cesta Coa.
Qxska
CNN
Al Mi-SkekdeeCar Can ca
2017
S. 10. 20
Bi cn-diraciiaal HNN
Asseezodder
Mosure Deaily RNS
Raste Caa.
SuoV:
3115
Re Ri-dinotoral RNN, CNN
Mimer RNX
De. Colamn
(oand
I FEW-SHOT CLASSIFICATION ON OMNIGLOT FULL RESULTS
The full results table for few-shot classification on the Omniglot (Lake et al. 2015) dataset, including the ResNet12 (Oreskkin al. 2018) model.
19

---

Under review as a conference paper at ICLR 2021
Table 11: Few-shot classification results on Omniglot
Omniglot
(way,
Algorithm
Backbone
Train Data
(5.1)
(5.5)
(20,1)
(20,5)
Training from Scratch (Hsu al. 2019
N/A
Omniglot
52.50 + 0.84
74.78 + 0.69
24.91
0.33
47.62 = 0.44
Random
Conv4
N/A
67.96
0.31
60.87
Cony VAE
Conv4
Onuniglat
77.83 1 0.41
92.91
1 0.19
62.59
0.24
84.01
0.15
Cony-VAL VAE
Quickdraw
Conv-AK AE
Conv4
Quickdraw
81.54 1. 0.40
93.57
1 0.19
67.24
0.24
84.15 0.16
3-VAF
(Hiegins
Quickdraw
79.11
0.19
0.24
N.NN (Isu et al. 2019
N/A
Onuniglat
57.46 1. 1.35
81.16
0.57
39.73
0.38
66.38 0.36
Linear Classiher
Hoir
Omniglat
61.08
0.69
MLP
Dropout
[Tuu et a
2019
N/A
Oniniglat
51.95 1. 0,82
77.20
0.65
30.65
0.39
58.62
0.41
Matching
Hsi
2019
Onniglat
0.77
0.40
45.93
CACTUs MAML.
TIS el
2019
Conv4
Omnnglltt
68.84 1. 0,80
87.78
0.50
48.09
0.41
73.36
0.34
CACTUS-ProtoNer
A119
Onniglat
0,43
MAL--ProtoNet (Amlontui &
Strkey
2019
Conv4
Omnigllt
84,66 1. 0,70
88.41
0.27
68.79
1.03
74.05
0.46
AAL-MAMI. (Amimoo & Storkey
A719
Onniglat
0.32
70.20
0.86
88.30
UMTRA (KhodddddS at
2019
Conv4
Omniglot
83.80
95,43
74.25
92.12
SkeichEEmbeokking (Owrs)
Conv4
Oniiglol
94.88 1. 0.22
99.01
0.08
86.18
0.18
96.69 0.07
SketchEmbeedding-e (Ours)
Conv4
Quickdraw
96.37
99,43
90.69
98.07
SkeichEmneeclling best (Qurs)
Conv4
Quickdraw
96.96
1: 0.17
99.50
0.06
91.67
0.14
98.30 0.05
SketchfFmherkling- (Qury)
ResNet12
Quickdraw
96.00
99.51
89.88
98.27
SkeichEimbeekling best (Ours)
RexNet12
Quickdraw
96.61 1: 0.19
99.58
0.06
0.15 91.25
98.58
0.05
SketchhiibbssdinngkLJ) (Ourx)
Quickdraw
97.92
SkotchEmbeodinng(K) KL J-bstt (Ows))
Conv4
Quickdraw
96.60 + 0.18
99.46 =: 0.06
90.84
0,15
98.09
0.06
SketchEEmheoklingg Labels) (Ours)
Quickdraw
88.52
0.24
MAML (Supervized) (Finn
2017
Conv4
Onniglut Onniglot
94.46 1: 0.35
83 1: 0.12
84.60
0.32
96.29
0.13
ProtoNet (Supervized Smell ot 2017
Conv4
98.35 + 0.22
99.58 0.09
95.31 0.18
98.81 = 0.07
Stroke data used training
FEW-SHOT CLASSIFICATION ON MINI-IMAGENET FULL RESULTS
The full results table for few-shot classification on the mini-ImageNet dataset, including the ResNet12 (Oreshkin et al. 2018) model and Conv4 models.
Table 12: Few-shot classification results on mini-ImageNet
mini-ImageNet
(way, shot)
Algorithm
Backbone
Train Data
(5.1)
(5,5)
(5.20)
(5,50)
Training from Scratch Hss ot al.
2019
NA
mini-ImageNNet
27.59 = 0.59
38.48 : 0.66
51.53 = 0.72
59.63 2 0.74
UMTRA Khodadadeh
mini ImageNet
61.11
MAML His et
2019
Conv4
mini- Imii-em
39.90 = 0.74
53.97 0:70
63.84
: 0.70
69.64 1: 0.63
CACTUS-Protos 2
Conv4
mini-ImageNett
39.18 0.71
0.73
61.54
0.68
63.55
0.64
Aet-ProtoNet Ambinin << Markey
2019
Convé
mini-ImageNNer
37.67 0.39
40.29 0.68
AAL-MAMI. Anlomiu Slorkey
2019
Conv4
mini ImiieNe
34.57
0.74
49.18-100.47
Random CNN
Convé
N/A
26.85 1: 0.31
0.332
38.51 1: 0.28
41.41
0.28
Conv-VAE
Conv4
mini-ImageNet
23.30 0.21
26.22
29.93 = 0.21
32.57
0.20
Conv- VAB
Conv4
Sketchy
23.27 -
26.28
0.119
30.41
0.19
Random CNN
ResNet12
NIA
28.59 + 0.34
0.34 35.91
41.31 + 0.33
44.07
0.31
VAE
ResNet12
Iine-emmgeNet
33.64
Con--VAE
ResNer12
Sketchy
24.61 = 0.23
28.85
0.23
35.72 - 0.27
40.44 1: 0.28
SkerchEmbedding-awg (ours)
Skethy
37.01
61.41
SkecchEmbalding-bon (owrs)
Conv4
Sketchy
38.61 - 0.42
53.82 0.41
63.34 = 0.35
67.22
17.22-1000 0.32
SkerchFimheddinggaryy (nwrr)
ResNet12
Sketchy
38.55
54.39
65,14
69.70
SketchEmbedding best (owrs)
ResNet12
Skeichy
40.39
0.44
$7.15 1 0.38
67.60
0.33
71.99 0.3
MAML (supervised) (Finn 2017
Conv4
mini-ImageNet
46.8
0.77
62.
0.72
71.03 + 0.69
75.54
0.62
ProtoNet (supervised) (Snell el 2017
Convé
mini Image-mgeee.
46.56
0.76
62.29 0.71
0.65 70.05
72.04 0.60
Strke for mainning
20

---

Under review as a conference paper at ICLR 2021
K
ADDITIONAL CONCEPTUAL COMPOSITIONALITY
θ
+ +
5
+
0 0
8 O
+ O 6
8
O
+
O à
8
O
0 = =6
8
O
R 0 + O 6
Figure 9: Uncherrypicked conceptual compositionality examples
Conceptual Compositionality Examples
Basketball Circle Square
C
0
D
Baseball Circle Hesagon
R 0 -=
Cup - Circle + Square
0
0.0+3:3
Hexagon
=
D+ C-0 C
Figure 10: Additional conceptual compositionality examples
L EMBEDDING PROPERTIES OF OTHER BASELINE MODELS
Here substantiate the uniqueness of the properties observed in SketchEmbeddings by applying the same experiments to a 3-VAE (Higgins et al. (017)) as well a vamilla autoencoder trained on the same dataset. We also include results of HmbedNe trained with KI. objective.
21

---

Under review as a conference paper at ICLR 2021
L.1 3- V-V
3-VAE (3 = 200)
8 O + + =
8 0
a O
=
8-VAE (B 250)
8
8
G. O =
Figure 11: Section 6 results for 3-VAE
The 3-VAE (Higgins et al. 2017) exhibits similar unsupervised clustering in comparison to the Conv-VAB VAE and is generally incapable of distinguishing input images that have different shape compositions but the same overall silhouette (first two examples from the left). Differently it is better at distinguishing non-synthetic examples that contain multiple squares or circles (3rd figure). However, it utterly fails the latent variable regression task and does not exhibit any significant conceptual composition in latent space.
L.2 AUTOENCODER AND SKETCHEMBEDNETT-KL
AutoEncoder
8 = & 3 - Ø
3- 0=
SketchhEmbedding-K
==
9 [= 0
+ =
O =0
Figure 12: Section results for Autoencoder and SketchEmbedding KL
We show that the performance of SketchEmbedding embeddings in our experiments in Section which focuses on organization latent space is not correlated with the KL term. We present both vanilla autoencoder without the KL objective and a SketchEmbedNet trained with KL objective. We observe a drop in overall generation quality in the Conceptual Composition decoding as is expected with additional constraint but maintained performance in the other tasks. Meanwhile, the autoencoder does not demonstrate any marked improvements over the Conv- VAE in the main paper any other baseline comparison.
22

---

Under review as a conference paper at ICLR 2021
M ADDITIONAL COMPOSITIONALITY MODES
We provide additional clustering methods 1-SNE (Maaten & Hinton 2008 and PCA as well as 2 new experiments that explore the compositionality of our latent SkeichEmbedding.
Additional clustering methods We include additional I-SNE and PCA results of the experiments in the main paper. These are presented in Figures 13 and do not always produce the same visualization while PCA is deterministic and prioritizes the most important dimensions.
14
15
16 17 t-SNE and UMAP are stochastic
StetchEmbeddinggTSS
Sketch ErAbediinggeA
StetchEnbeeddnng-UMM
00 247
OO
A1"
VAE-TSNE
VAE-FCA
VAE LIMP
OO
oO
Figure 13: 2D Embedding visualization of different spatial orientations of circles and squares
SletchEmbedding
SketchEmbeedding
SketchEmbbedingg-nne
CD Clow
95
OO Mediu
VAC-TSNE
VAE-PCA
VAC-UMAP
Capp?
O Far
Figure 14: 2D Embedding visualization of different linear distances between shapes
23

---

Under review as a conference paper at ICLR 2021
SlicchEmboddiigg.ng
Slacch PPC
WasthhEibedddnggukkk
VS 241
86
00
151
ERI
VAE-TINNE
ME--PCA
VNE-UMAP
100
194
28
20g
Figure 15: Latent space visualization squares and circles arranged differently in 2x2 array
SketchEmbedding-TS
Shetchäänbedding.CCC
SkatchhEnbedddin--JMMM
8
13
VAETSSVE
VAE-PCA
VAF-LIMPP
8
0
Figure 16: Latent space visualization of composing circles and squares within one another or outside
SictehEmbaddingTSve
SketchEmbbodd PCA
SkottchEmboodling UMAP
M6
Circle
g
17
Snowman
VAE-TSNE
VAE-FCA
VAE-UMAP
Ba
Square
23
ong o 80
RICO
S Telovisin
Figure 17: Latent space visualization of composing multiple circles and squares in real sketch drawings
24

---

Under review as a conference paper at ICLR 2021
Additional Experiments Here provide different investigations into the compositionality of our
learned embedding space that were not present in our main paper. These results presented in Figure
18 and 19
Sie
Skeethh Eribediing PCA
Sketchhmbeedding UMAP
DO
oD
VAETTSNE
VAE-FCA
VAE-U/AP
28
DO
Figure 18: 2D Embedding visualization of different spatial orientations of circles and squares
In Figure 18 we place a square in the center of the example and place a circle above, below or to the sides of it. Once again we find that our SkebchEmbedding embedding clusters better than the VAE approach.
SleechhEmeoddingSSee
Sketch-ribedding PCA
SketchEmbeddingguuM
00
VAE.TSNE
VAE-PCA
VAE-U/MP
48
Figure 19: Latent space visualization of composing multiple circles and squares in real sketch drawings
New examples are generated where each class has a different numbers of circles. Both the VAE approach and our SketchEmbedding cluster well and neither appear to learn the count manifold.
N HYPERNETWORK ACTIVATIONS
To further explore how our network understands drawings, we examine the relationships between the activations of the hypernetwork of our HyperLSTM (Ha et al. 2017).
The hypernetwork determines the weights of the LSTM that generates the RNN at each decoding timestep. These activations are 512-dimensional vectors. We collect the activations from many
25

---

Under review as a conference paper at ICLR 2021
examples, cluster them in 512-dimensional space and visualize the strokes belonging to each cluster for each example. A full decoding is also rendered where each cluster within an example assigned color.
Single class: snowman First we explore this clustering using only the snowman class from Quick- draw (Jongejan al. 2016). We expect substantial reuse of "circle" both within and over many examples. Clustering of the strokes is done with the DBSCAN (Ester et al. 1996) and parameter = 3.9. Results are in Figure 20 Each row is separate input; the far left column the color-coded, composed image, the second is the noise cluster and every subsequent column is unique cluster.
0
7
Figure 20: Snowman class stroke clustering
While cluster re-use is limited, cluster often contains large, fully enclosed circle. Many other clusters may contain circles or partial strokes with some reuse. Larger, fully composed and coloured sketches are presented in Figure 2T
8 3
8
8
e
Figure 21: Fully composed images with coloured cluster assignments
Many classes: round objects We repeat the above experiment with mixture of classes that generally can be expected to contain circles. These classes were circles, snowmen, clocks and cups. The two former classes frequently composed only of circles while the latter are expected to consistently contain other distinct shapes. Results are presented in Figure 22 and select examples in Figure 23
26

---

Under review as a conference paper at ICLR 2021
3
Figure 22:
Snowman class stroke clustering
We still observe that the model continues to isolate circles in the first column and note it continues to do so for the cup and clock classes which not exclusively circular.
Figure 23: Fully composed images with coloured cluster assignments
Many random classes: Finally, we repeat the above clustering with the 45 randomly selected holdout classes from the Quickdraw training process of SketchEmbedding. Results once again presented in Figure 24 and select examples in Figure 25
1
8
8
Figure 24: Snowman class stroke clustering
Figure 25: Fully composed images with coloured cluster assignments
27


Reviews:
[{"review_id": "tjz8CxSx8iZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper proposes learning embeddings for sketch or natural images by training a network that takes in a raster image and outputs and collection of sketch strokes. The architecture consists of a standard CNN encoder followed by an RNN decoder. The authors evaluate their learned embeddings on few-shot classification tasks and explore the the quality of the latent space. They demonstrate that they outperform unsupervised few-shot classification approaches and seem to obtain a latent space that is more aware of long-range structure than those from methods that operate purely in raster space.\n\n---\n\n### Explanation of rating\n\nInterpreting and synthesizing sketches in a deep learning context is an promising research direction. While the idea of focusing on of sketch-aware embeddings of images is an interesting one, the main technical contribution simply involves taking a standard convolutional encoder with a recurrent decoder, which has already been used for sketch generation (sketch-rnn). In addition to this, some of the claims made in the paper require some clarification or additional experimentation, as I explain below. Thus, I believe that some additions and changes must be made for the paper to be accepted.\n\n---\n\n### Pros\n\n- I like the idea of using the vector structure of sketches to gain more insight into image content.\n- It is difficult to design deep networks that are able to operate on non-raster data. In some sense, this approach sidesteps this issue by allowing the relationship between raster and vector to be learned.\n- The experiments in Section 6 confirm the idea that the embeddings are aware of this high-level long-range relationships that aren't so obvious on the pixel level.\n\n---\n\n### Cons and questions\n\n- I am not sure I follow the intuition behind why the proposed model achieves better semantic awareness than a convolutional VAE. The authors use the example of a six legged turtle and state that the VAE would only retain information about a single \"leg-like feature but not how many legs are present.\" How is having an RNN stroke-based decoder different from a standard convolutional decoder in this sense? In both cases, the final reconstruction must contain the original number of legs, and so the latent vector is encouraged to retain this information.\n- The performance on natural images in Figure 4, especially on unseen classes, is not great. I would be interested to see the nearest neighbor images in the training set for the examples shown. Even in the case of unseen classes, the resulting sketches look like they might match a similar image from the training set.\n- The authors claim that balancing the stroke and and pixel losses via a curriculum mirrors how humans learn to draw (\"paint-by-numbers\"). However, I'm not sure how some of the experiments fit into this methodology. In particular, most of the experiments are done on datasets that have ground truth sketch strokes but not their ordering (e.g., SVG files). In this case, it seems like imposing an order on the strokes (and asking the decoder to replicate it) is a counterintuitive constraint for the model. On the other hand, in the natural image experiments, the pixel loss cannot be used at all. Some discussion of the consequences there would be interesting.\n- I'm not sure that it is fair to compare to fully unsupervised few-shot classification methods. While the proposed method indeed does not use class labels, the ground truth stroke information may provide considerably more information than just the raster image data. Perhaps it would help to have a baseline without stroke loss (i.e., $\\alpha=1$). Even the ablation study in Table 3 does not include this case.\n- How sensitive is the approach to quality of stroke decomposition? For instance, what happens if you subdivide each stroke in the ground truth SVGs?\n\n---\n\nThe authors have addressed many of my concerns in the rebuttal, and so I am increasing my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "sB-PJrmpz_g", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a generalized sketch drawing model named SketchEmbedNet for producing sketches and visual summaries of open-domain natural images. The idea is interesting and the experimental results show SketchEmbedNet is able to do not only few-shot classification but also one-shot generation. \n\nOverall, I vote for rejecting. In my opinion, the main contributions of this paper are not very clear. The introduced model, SketchEmbedNet has limited novelty on neither the methodology nor the network structure. As stated in the title and introduction, the authors aim to capture and generalize the compositional information from sketch/natural images. Section 4 reports the latent variables organization performance, which is directly related to the key motivation I believe. But the authors only compared SketchEmbedNet with VAE, which is not enough to demonstrate their advantages. Moreover, it is not clear why few-shot classification and one-shot generation performance in Section 5 and 6 support their main idea. Thus, this paper needs further improvements.\n\nDetailed comments:\n(1)\tIn the first paragraph of section 2, the authors claimed that the CNN embeddings must preserve a compositional understanding of the input sketches/images to improve the performance in their pix2seq task. So how did you preserve the information? Many sketch synthesis methods, such as [8] in the reference, can reconstruct the sketch with a sketch image input. Do these methods preserve the ``compositional understanding’’?\n(2)\tStill in the same paragraph, I’m pretty sure a vanilla auto-encoder with a CNN encoder containing average pooling layers can reconstruct a six-legs-turtle well as input. Thus, the information about the positions and the turtle legs number must be transported to decoder by the latent embeddings. So why you confirm that a regular CNN embedding cannot preserve that information after an average pooling layer? \n(3)\tThe same CNN encoder is used for both natural images and sketches. As these two types of images are with totally different patterns, many recent studies, such as reference [55], used a two-branched encoder for natural images and sketches, respectively. In this paper, the CNN backbone is a 4-layer CNN or a ResNet12, which are very basic structures. Is it able to extract the image features well?\n(4)\tFigure 5 shows SketchEmbedNet outperforms VAE on latent space organization. In my opinion, the well-organized latent space of SketchEmbedNet is mainly due to getting rid of the KL loss term, which drives the latent distribution to be a uniform distribution. I would like to see the comparison with sketch-pix2seq, which is the reference [8], as both SketchEmbedNet and sketch-pix2seq do not use KL term in training.\n(5)\tAs this paper focuses on sketch drawing, why there is no comparison between SketchEmbedNet and other sketch generative models, such as [8, 11, 15, 26] in the references? \n\n\n\n\nAfter reading the response from the authors, we raise our score by +1.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nERNyKhcwhM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors investigate the possibility to learn a generalized embedding function that captures salient and compositional features of sketches by directly imitating human sketches. The manuscript is written clearly and concisely. Methods have been presented with enough detail and seem accurate. Particularly, the results from the Quickdraw and Omniglot datasets showing generated sketches are rather impressive, and the ones for the natural images seem promising. Overall, I very much enjoyed reading the paper and suggest it for publication without any major changes.\n\nIn my view the results presented in Figure 5, and especially 5C, are the most impressive and interesting ones. These results deserve more space in the manuscript. I was curious to know whether there were also many unsuccessful conceptual composition examples. Are the examples shown in Figure 5C the best ones, or are they representative of performance in general? Does this approach also work with natural images to any extent? Could the authors elaborate on why or why not this may be the case?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ir0EOkagpdo", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A brief summary:\nThis paper shows that the model trained to restore sequential data from images in a supervised manner tends to capture more informative latent representations of the data.\n\nStrengths:\n+ Demonstrates that training a decoder model to reconstruct sequential sketches leads for an encoder to better represent the input image.\n+ Achieves the SOTA result in Omniglot recognition task, compared against existing unsupervised methods.\n\nWeaknesses:\n- Confuses readers with the ambiguous claim. The embedding function is said to produce `salient and compositional features' (p1), but no evaluations on compositions were included in the main paper. \n- Does not include any ablation studies to show the effectiveness of each components of the proposed model.\n- Does not include thorough explanations or analysis on each set of experiments.\n\nInitial recommendation:\nBorderline reject.\n\nReasons:\n- While this paper provides a lot of experimental results, which I very much appreciate, I still found most of them quite irrelevant to support the main claim by the authors, which discusses compositional embeddings. The main contribution of this paper, I believe, is mainly the idea to utilize the sequential sketch data during the supervised training time. This needs to be clearly stated, particularly inside the tables, as most baselines only use sketch images, not their sequential data. \nThat being said, this trick is shown to work well, and if these significant improvement on Omniglot is further verified, this trick will be found useful by the community of researchers who work with sequential data, such as sketches or handwriting.\n- However, I found most of the implementation details quite unclear, and experimental results were often misleading. For instance, the results from Table 4, if we look closely, the authors' results use ResNet12 trained with Sketchy data, which quite differs from other results in the list.\n\nFeedbacks:\n- Authors briefly mention in p.4 that a KL term did negatively effect the performance, but no detailed explanations/experiments were given.\n- What happens when you provide the class labels and train the model with classification errors as well? Will it improve the test result? If so, how much? \n- May have been much more interesting if this paper explores the unsupervised disentanglement in latent space, to support their claims for what they note as structured embeddings. Will other latent disentanglement methods (e.g. beta-, factor-, VQ-VAE, etc.) lead to better representations?\n- Conv-VAE alone may not be the best baseline.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
Description: 
The paper presents a generative model, SketchEmbedNet, for class-agnostic generation of sketch drawings from images. They leverage sequential data in hand-drawn sketches. Results shows this outperforms STOA on few-shot classification tasks, and the model can generate sketches from new classes after one shot.

Strengths:
- Detailed, technically sound, presentation 
- Shows that enforcing the decoder to output sequential data leads to a more informative internal representation, and thus generate better quality sketches 
- Improves over unsupervised STOA methods

Weaknesses:
- Experiments are done against methods that do not use the sequential aspect of sketches. Because ground-truth in this case contains much more data, comparison is not quite fair.
- Will have been useful to see results against a baseline that uses it.
- Quality of sketches generated from natural images is low
Decision: Reject