SketchKnitter: Vectorized Sketch Generation with Diffusion Models
Authors: Qiang Wang
Haoge Deng
Yonggang Qi
Da Li
Yi-Zhe Song
Venue: ICLR 2023 notable top 25% (2023)

Abstract: We show vectorized sketch generation can be identified as a reversal of the stroke deformation process. This relationship was established by means of a diffusion model that learns data distributions over the stroke-point locations and pen states of real human sketches. Given randomly scattered stroke-points, sketch generation becomes a process of deformation-based denoising, where the generator rectifies positions of stroke points at each timestep to converge at a recognizable sketch. A key innovation was to embed recognizability into the reverse time diffusion process. It was observed that the estimated noise during the reversal process is strongly correlated with sketch classification accuracy. An auxiliary recurrent neural network (RNN) was consequently used to quantify recognizability during data sampling. It follows that, based on the recognizability scores, a sampling shortcut function can also be devised that renders better quality sketches with fewer sampling steps. Finally it is shown that the model can be easily extended to a conditional generation framework, where given incomplete and unfaithful sketches, it yields one that is more visually appealing and with higher recognizability.

Published as a conference paper at ICLR 2023
SKETCHKNITTER: VECTORIZED SKETCH GENERA- TION WITH DIFFUSION MODELS
Qiang Wang' Haoge Deng' Yonggang Qi Da Li Yi-Zhe Song?
"Beijing University of Posts and Telccommunication, CN (wanqqiang, denghaogg, qiyg} @ bupt.edu.cn dali dcli.academic gmail.com y.song @ surrey.ac.uk uk
SketchX, CVSSP. University of Surrey, UK
ABSTRACT
We show vectorized sketch generation can be identified as a reversal of the stroke deformation process. This relationship established by means of diffusion model that learns data distributions over the stroke-point locations and pen states of real human sketches. Given randomly scattered stroke-points, sketch generation becomes a process of deformation- based denoising, where the generator rectifies positions of stroke points at each timestep converge recognizable sketch. A key innovation was to embed recognizability into the reverse time diffusion process. was observed that the estimated noise during the reversal process is strongly correlated with sketch classification accuracy. An auxiliary recurrent neu- ral network (RNN) was consequently used to quantify recognizability during data sampling. It follows that, based the recognizability scores, sampling short- cut function can also be devised that renders better quality sketches with fewer sampling steps. Finally it is shown that the model can be easily extended to con- ditional generation framework, where given incomplete and unfaithful sketches, it yields one that is more visually appealing and with higher recognizability.
INTRODUCTION
Free-hand human sketches are abstract concepts which can efficiently express idcas. Generative models for sketches have received increasing attentions in recent years. Compared with producing pixelated sketches (Ge et al.) 2020) Chen et al., [001] Liu et al. 2020). modeling sketches with point trajectories is more reasonable and appealing as it more closely resembles drawing process of humans. Sketch-RNN (Ha & Eck 2018) utilizes a set of discrete stroke points and binary pen states as an approximation
the continuous drawing trajectory. BézierSketch (Das al., 2020) makes use of parametric representation, which fits the stroke trajectory by Bézier curves. Very recently, SketchODE (Das al. 2021a) applies neural ordinary differential equations representing stroke trajectory through continuous-time functions. All said approaches however suffer from the inability model complex vectorized sketches. This is largely attributed to the de-facto RNN backbone that falls short in accommodating large stroke point numbers rule of thumb is anything beyond 200 points will fail (Pascanu et al., 2013 Das et al. 2021b).
In this paper, we attempt to change the status quo in how stroke-point trajectories are modeled. Instead of seeing sketch generation as a process of determining where the next stroke-point lies under each recurrent step (as per RNN), holistically at each time instance as every knitting enthusiast will tell you, is all about having a global plan, never just about the next thread!
attempt to estimate distributions of all stroke-points
Our key novelty lies with the realization that sketch generation can be conceptualized as the reversal of stroke deformation process. Through modeling forward deformation process (i.e., sketch to noise), our diffusion model learns the stroke-point distributions of real human sketches, and thus able to reverse the process generate novel sketches given noisy input. It follows that given this diffusion setup, the sequential information in sketches can be persevered by simply maintaining the temporal ordering of stroke-points during reverse-time diffusion.
"Correspondence to: Yonggang Qi (giyg @ bupt.edu.cn). Code to be found at GitHub page
https::/ htttss://ww..r ravel ry.lyr.co.. comm

---

Published as a conference paper at ICLR 2023
[( I)
Random Noise
Data Sample
Diftusion Models
Sampiiey
steps
(+)
-08
Estimated Notee
0.6
0.6
Start
Intermidiate
End
0.4
0.2
(b)
1000
800
E00 .000
200
Stcp (G)
Long but inefficient steps
shorter but efficient
Figure 1: (a) Built on diffusion models, sketch generation is formulated as stroke deformation rectification problem. Essentially, our model is to reorganize points with fixed adjacency, such that placing them meaningful locations out of a total mess, dubbed sketch knitting. Note that the order of stroke points is pre-determined and unchanged. (b) Generated sketches with different deformation levels associated with timesteps by our model. Early sampling stage is not efficient that the obtained data change little and remain noisy. (c) Early sampling is inefficient as basically the same estimated noise each stroke point is used, led sketch remains noisy and unrecognizable. And different noise patterns can be observed at different timesteps, which motivate us to devise recognizability-basee skipping (blue/)) based on the estimated noise find shortcut sampling. Red solid curve denotes mean and the shade denotes variance. 1000 generated sketches are used for plotting.
We further draw importance the overall quality (recognizability) of the sketches generated. We show that the estimated noise in the sampling stage naturally can reflect the recognizability of the generated sketch at each timestep. It follows that learnable RNN was devised to explicitly model the relation between estimated noise and recognizability. This achieved by introducing an pre- trained image classifier supervision signal. Embedding recognizabilty into the sampling process also yields the added benefit of introducing skip steps that allows for more efficient and effective data generation. This is because early stages for generating sequential data very inefficient using vanilla DDPMs sampling (Ho et al. 2020) as witnessed in Figure [ (b), resulting in minor improvement of recognizabilityy in long period of sampling as unveiled in Figure (c).
Last but not least, we demonstrate the model (without retraining) can be readily used to remedy de- fects in sketches due to unfaithful incomplete drawing, by incorporating instance-aware guidance into data sampling. Motivated by recent works guided diffusion models (Dhariwal & Nichol, 2021
Ho & Salimans 2022), gradients of perceptual similarity (Zhang et al. (218)) between the generated data and the conditional sketch were incorporated during sampling to guide the noise pre- diction, thereby influencing the obtained sample at each timestep. This was done with the goal of enforcing visual similarity to the conditional, flawed sketch, while also being more appealing and recognizable after reverse-tima time diffusion.
Our contributions be summarized as follows: (i) Denoising diffusion models exploited for sketch generation in vector format. The generative model is to learn distribution over stroke points' locations, from deformationn-assel denoising process which starts from noise. (ii) The quality, i.e., recognizablity, of the generated sketches is quantifiable by leveraging the knowledge of the estimated noises during sampling. This is achieved by devising auxiliary RNN, which trained supervised under pre-trained image classifier, to predict the recognizability of generated sketch at timestep / from the corresponding estimated noise. (iii) shortcut sampling path can be discovered through simple skip strategy based on the learned quality measurement net. This allows faster and more effective generation with little trade off in data quality. (iv) Instance-an aware guidance built on perceptual metric is embedded into the reverse-time diffusion. It enables our model to recover distorted or corrupted sketches without retraining.
2 RELATED WORKS
Sketch Generation There is a rich literature of research works related to sketch generation. Early works (Guo et al. 2007 Li et al. 2019) leverage edge maps substitution for sketches. Coupled
2

---

Published as a conference paper at ICLR 2023
with deep learning, much progress has been made recently. Particularly, generative adversarial net- works (GANs) have motivated extensive works in sketch synthesis, involving doodle-sketch genera- tion (Ge et al. 2020), image-to-sketch translation (Liu et al., 2020), colored sketch rendering(Rathod
eet al.
2021) pencil-shading sketch generation (Li et al. 2020b), and face sketch synthesis (Wang 2020 However, those models are all pixel-based generation, which is fundamentally dif-
al.
ferent from how humans sketch objects using pens or brushes. Towards modeling sketches like humans, sketches are preferred to be treated as sequential pen actions, and RNN-based variational autoencoder (VAE) (Ha & Eck] 2018: Zhang et al., 2017 Graves 2013), reinforcement learning (RL) (Xie et al. 2013 Zheng | Ganin al. (018). Transformed- sketch represen- tation (Ribeiro et al. 2020 Lin et al. 2020), learning parametric Bézier curve (Das et al. 2020 2021b), and neural ODE (Das et al. 2021a) are explored for sketch generation. (Aksan et al. 2020p
proposes relational model built on auto-encoder, which can decompose sketch formed by a single temporal sequence into a group of disordered strokes. Particularly, promising generative results on complex structures have been witnessed. Other notable works include generating stylized line draw- ing from 3D shapes (Liu et al., 2021). and intent communication through sketching by referential communication game (Mihai & Hare 2021).
Diffusion Models Recently, approaches with diffusion models have delivered impressive results on several generative tasks, including image generation (Ho et al.) [2020] Dhariwal & Nichol 2021),
shape generation (Cai et al., 2020). 3D shape modelling (Luo & Hul 202T audio synthesis (Kong
iet al. 2020), and cross-domain generation (Popov et al.) [021]
| Nichol et al. (021). Different from
likelihood-based models (variational auto-encoder (VAEs) (Kingma & Welling, (203), normalizing
flow models (Dinh et al. 2014] Papamakarios et al. 2021), energy-based models (EBMs) (LeCun
al. 2006)) and implicit generative models (GANs (Goodfellow et al. 2014). diffusion models can be categorized into score-base generative modeling (SGM) (Song et al.) 2020). which aims to model the gradient of the log probability density function by score matching (Hyvärinen & Dayan 2005). There are two popular sub-classes in SGMs, i.e., score matching with Langevin dynam- ics (SMLD) (Song& & Ermon| 2019) and denoising diffusion probabilistic models (DDPM) (Sohl- Dickstein et al. 2015 Ho et al. 2020). Despite attractive, rare work with diffusion models targets at handling sketches. The most relevant work to ours is Diff-HW (Luhman & Luhman 2020) which also applies diffusion models for sequential data generation, i.e., handwriting. However, Diff-HW adopts the vanilla DDPMs and focuses on text-to-sketch translation. On contrast, we offer (model built-in) quality quantifiable diffusion models, improved sampling strategy and gradients guided conditional sampling based on DDIM.
DIFFUSION MODELS FOR VECTORIZED SKETCH GENERATION
Our objective is to generate sketch stroke sequences from noise by a novel method built on denois- ing diffusion implicit models (DDIMs). DDIMs generalize DDPMs by introducing a Mon-Markovian diffusion process, which yet still has the same forward marginals as DDPMs. Uniquely, we propose a method to find shortcut sampling trajectory based on recognizablity of a generated sketch. More- over, we will present how to rectify a flawed sketch by our trained unconditional sketch generation model on-the-fly during the generative process.
3.1 PROBLEM SETUP
We construct sketch in sequence as 8n { ;-..,,""" 8N) using the representation from Ha & Eek (2018), i.e. stroke-3. Each point s² is represented as a 3-D vector (Ax', Ayy'g"), where (Ax', Ay indicates the offsets at point during the pen's 's moving trajectory, and 9 binary pen state, denoting whether the pen is touching the paper or not. Our goal is to learn the probability distribution of the offsets {(Ax", Ay")} from the training data by diffusion models. Then, sketch
can be drawn given the estimated (Ar', Ayy) for each point and the corresponding pen state p inferred by a pen-state network. We will describe more details the following sections.
3.2 SKETCH DIFFUSION IN FORWARD PROCESS
During the forward process, the noise will be gradually added to each point offsets of an original sketch, resulting in increased stroke distortion over time. Consider we have N ordered stroke points
3

---

Published as a conference paper at ICLR 2023
for given sketch So {s' { ..... s... the offset (Ar', Ay') of each point s² is supposed to be sampled independently from a distribution ((s). To diffuse So into $1...., 8r, the Markov diffusion process introduced in DDPMs is applied here. Formally, the forward process enforces each of the point offsets (Ar', Ay") in 8o to drift along both x and coordinates by gradually adding noises sampled from Gaussain distributions with pre-defined schedules 0T. Therefore, the Markov chain in forward process is defined by:
II (( q(Se|8t-1) N (#ti ax-1
T
at
81-1: (
ar- (1-aa-))
at
11)
((11T
3.3 DDIM BASED GENERATIVE PROCESS
Following (Song et al. 2021), DDIM-based sampling is adopted to generate sketches from noise, ic., the reverse process, as it normally achieves high data quality with significantly small sampling steps. More importantly, their non-Marke non-Markovian sampling process supports in discovering novel shortcut sampling function being more efficient and effective for our sketch generation. Formally, the generative process is defined follows:
(8 8t, 80) = N (var- + V 04
St va,50
(2)
v1 at
With DDIM can predict so through the generative process
( (81).
(t) Pg (s--1
if
(3)
Ya (8t 8t, S0, (se)), otherwise
where (s,, - 600 (st)) Vor is prediction of sg based noise approxi-
mator 60 Then we can sample a data sample 8o from random noise 8T = NO. I) by iteratively repeating the following equation
8 Var-1
81
VI C (8t) +
0--1 oo 100 (s.).
(4)
Var
More detailed derivations about DDPMs and how DDIMs generalize DDPMs by defining non- Markovian forward process and obtain the corresponding generative process described in Eq. caan be found in Appendix A. and Appendix(A.2
3.4 NOISE APPROXIMATOR &
(2) In our case the noise approximator Eg (s:) in Eq (4) is a trainable network to estimate noise e()) R coordinate offsets for 8; Nx2 at timestep An improved UsNet developed to handle the sequence the points from a vectorized sketch. Namely, we add a trainable embedding and a decoding layer into U-Net transform the input s; into embedding e, R x and convert the penultimate feature embedding back to coordinates. Then the rest design is as per the conventional U-Net for dealing with 2D images. Please refer Appendix A.3 for details.
3.5 RECOGNIZARILITY BASED SHORTCUT SAMPLING
As shown in Figure (c), we observe that at the early sampling steps, the recognizability of gener- ated 8, is consistently low, although the amount of the estimated noise is large if we take full reverse steps of length T used in DDPMs. In contrast, denoising gets much more efficient and effective, leading to a noticeable leap of recognizability at the later steps. We then intuitively suppose that the recognizabilityy of a generated sample ought to be inferred from the pattern of denoising sequence for all stroke points, i.e., op E Nx2 Therefore, we incorporate an additional trainable network to predict the recognizability r; given the estimated noise:
A, 1o(e 1).
(5)
4

---

Published as a conference paper at ICLR 2023
And the prediction Ae could be used as signal to find shorteut sampling path can skip m sampling steps if ft < Ç. which implies the period of ineffective sampling. m is a constant indicating the interval steps skip, and is predefined threshold. Therefore the timestep sampling function can be formulated as
l. m, if ho(e)) to 1, otherwise,
(1) (c) <:
tn
where te is the current timestep and t, is the next sampled timestep. Different from the Linear or Quadratic sub-sequence selection proposed in DDIM et al. 22021) our sub-sequence is chosen adaptively, which can more effective and shorter.
In practice, ha(-) is implemented as a bi-directional RNN as Ha & Eck 2018) with the estimated sequence of noise as input. Then the output latent vector E R is fed into a trainable linear layer to predict fz. And the ground truth r could be obtained from an extra pre-trained sketch classifier as done in (Song et al. 2018). Namely, we use the maximum probability of the softmax prediction as Tt- Once trained, ( could be accommodatingly used to inspect how recognizable sketch is during sampling, while no need to render it into image. Please refer to Appendix A.4 for more analysis and insights about the shortcut sampling.
3.6 PEN STATE ESTIMATION
The diffiusion model is to learn the distribution of the coordinate offsets for sketch points. However, it is also required to predict binary pen state for each of the stroke point. Following (Luhman & Luhman (0000) the feature vector, i.e., E R 128 from the penultimate layer of the above U-Net for noise approximation is utilized. We feed this feature sequence into another trainable linear layer followed by sigmoid function to predict each point's 's pen status:
ÿ s siggmmidmssfee
where i are parameters of the trainable linear layer. When g¹ > 0.5, it indicates the pen is touching the canvas at point i. We do this for each timestep in the generative process.
(t) 3.7 TRAINING OBJECTIVE OF 6g
To train noise approximator cf we follow Luhman & Luhman 2020 to jointly minimize the [t) following DDIMs Song et al. (2021)
denoising loss and pen state loss. Specifically, we train Eg
to minimize the 1.2 discrepancy between the estimated noise (!) and generated in the forward process:
(a El (s)
Additionally, at each timestep I, 6g (0)
is also optimized by minimizing the pen state loss
(o( = N Σ --g g' log(g') - (1 - 96) log(1 - g²)],
where ğ is predicted using Eq. 7 In summary, the total training loss is:
L(O) = La( (p)),
(10)
where y is weight to balance two losses.
3.8 CONDITIONAL GENERATION TO RECTIFY BAD SKETCHES
The above has described the details of the unconditional sketch generation. We now further present how to rectify a flawed sketch using our model trained with unconditional generation only. Inspired by the classifier guidance widely used in diffusion models (Dhariwal & Nichol| 2021: Ho & Sali- mans 2022), a sketch generation guidance introduced for the generative process. Different from existing works using gradients of an image classifier as guidance, perceptual metric is employed to
5

---

Published as a conference paper at ICLR 2023
generate guidance- the gradients of the log probability of perceptual distance between generated S; and a condition Sc, denoted as Ve, log p( Sc). Then the estimated noise refined as
(t) e (st) & (st) + +i,e, log p(s,,
(11)
where 17 is control the strength of the guidance. Then the sample 8;-1 will be generated using Eq 4 with the new noise (st). Intuitively. we aim to guide the generation of so by repeatedly sampling 8t, which has similar content as 8c. We follow prior work (Zhang et al. 2018) based on the L.2 distance between two image features measure their perceptual similarity,
it is effective in transferring image content. Neural line rendering (NLR) (Li et al. (020a)) is used to convert vector sketch into image sketch to allow the gradients being propagated the vectorized sketch. After rasterizing sketch into its image version, perceptual metric is thus applied as
p(81; 8c)
Σ 1=1
Σ N.w
Jun O (F (8x) - Fhw (8.)) 11:
(12)
H.W
where (-) E R M1 WiX denotes the feature maps for I-th layer of an ImageNet pre-trained
VGG Simonyan & Zisserman 2014). E RC is adopted to scale the feature activations channel-
wise as per (Zhang et al. 2018 Details about NLR are provided in Appendix A.55
EXPERIMENTS
In this section, we evaluate model in two modes, i.e., unconditional and conditional generation, to verify the quality of the generated data and the ability to mend inferior sketches of our model. Please refer Appendix for implementation details.
4.1 UNCONDITIONAL GENERATION
Dataset. We evaluate our proposed method on QuickDraw Ha & Eck (208)) which contains over 50M sketches in vector format across 345 common categories. A subset from QuickDraw is col- lected for the experiments. Specifically, there are 10 classes chosen adhered to the following princi- ples: (i) Complexity simple, moderate and complex sketch drawings are all included, e.g., fish, umbrella a and lion; (ii) Diversity objects with diverse sub-category variations are involved, e.g., bus and spider; (iii) Ambiguity sketches belong different classes share highly similar ap- pearance, e.g., apple and moon. The original data split is adopted, i..., 70,000 training and 2,500 testing sketches for each class.
Competitors. Three current RNN-based tite--oftheearrr methods, i.e., SketchRNN Ha & Eck (2018), SketchPix2seq Chen et al. (2017) and SketchHealer SS et al. (2020). are included. Diff-HW Luhman & Luhman (2020), which built on DDPMs, is enabled for comparison by tweaking the cross-modal attention layers to be self-attention since only sketches are available our problem.
Additionally, the unconditional mode of SketchODE (Das al. 2021a) is also used for comparison.
Evaluation metrics. We gauge the quality of the generated data resorting to the evaluation metrics for image generation once the vector sketches are rasterized into images. Specifically, Fréchet In- ception Distance (FID) Heusel et al. (2017) measures the distance between the generated (image) data and real ones by comparing the mean and variance of image features, which are obtained from Inception-V33 Szegedy al.) (2016) trained on ImageNet Krizhevsky et al. (2012) for image classi- fication. The Geometry Score (GS) Khrulkov & Oseledets (2018) metric compares the geometrical properties of data manifold between the generated and real data. In addition, the improved preci- sion and recall (Kynkäänniemi al. (019) are used as complementary evaluation metrics following other image generation works (Nichol Dhariwal 2021).
Qualitative results. Figure a) shows some examples of reverse-time diffusion process, i.e., from random noise till reach the data sample, the generated sketch at each step exhibits different (reduced) level of distortion. More results of unconditional sketch generation are demonstrated in Figure (b..
Claas list: moon, a rp1 ane, fish, umbre1 11 a,, trai n, api der, shoe, appl 1i bus. Diff-HW is originally proposed for stylized -attto-handwritinn generation, requiring text and an image condtions control the content and style of the generated handwriting, respectively.

---

Published as a conference paper at ICLR 2023
(a)
(b)
Figure 2: (a) Sketch generated from random noise. (b) More examples of unconditional generation.
Table I: Quantitative comparison results. Testing categories orgnaized in three folds according to the complexity, i.e., the average number of stroke points (ASP). Simple: < 40 ASP, Moderate: 40 100 ASP and Complex: 100 ASP. Speed: second per sketch sampling.
Simple
Moderate
Complex
Model
Speed]
FIDE
GSI
Prect
Rec
FIDI
GSL
Prect
Reef
FID.
GSL Prect
Rec
SketchPizZseq
13.3
7.0
0.40
0.79
16.4
49.7
0.38
0.75
18.0
73.3
0.36
0.72
0.04
SketchHealer
10.3
5.9
0.45
0.81
12.9
9.8
0.39
0.79
25.9
93.2
0.29
0.63
0.03
SketchRNN
10.8
5,4
0.44
0.82
13.0
11.0
0.42
0.77
21.4
97.6
0.35
0.72
0.03
Diff-HW
13.3
6.8
0.42
0.81
15.9
23,4
0.37
0.76
18.3
64.4
0.23
0.64
0.19
SketchODE
11.5
9.4
0.48
0.74
18.8
29.6
0.31
0.66
33.5
68.1
0.20
0.58
0.03
Ours (full 1000 steps)
6.9
3.4
0.52
0.88
8,4
4.7
0.45
0.87
9,4
5.2
0.42
0.85
1.29
Ours (y-Shortuut, S=30)
7.4
3.9
0.47
0.87
8.9
5.2
0.44
0.85
10.5
0.39
0.81
0,08
Ours LLineurDDDMMs, S=30)
11.9
6.4
0.38
0.81
13.3
8.8
0,36
0.78
15.1
9.6
0.33
0.72
0.08
Ours (Quadratic- DDIMs, S=30)
12.3
6.6
0.41
0.79
13.8
8.7
0.35
0.76
15.4
9.9
0.34
0.75
0.09
Ours (Ahs)
20.7
12.1
0.18
0.55
23.4
64.6
0.13
0.48
29.4
98.9
0.10
0.39
0.20
Ours (Point-Shufile
9.5
5.3
0.35
0.72
11.3
7.5
0.31
0.65
12.4
8.1
0.20
0.61
0.18
Ours (Stroko-Shuffle)
8.2
3.8
0.36
0.74
9.6
7.4
0.34
0.66
10.3
0.25
0.62
0.18
Quantitative results. As shown in Table [ our sketch generator clearly outperforms other com- petitors, suggesting better quality of the generated data. Particularly, our model is able to maintain relative stable FID, GS, Precision and Recall scores regardless of the complexity of the generated sketches. On the contrary, obvious performance decline is witnessed when constructing sketches with more complicated structures for other baseline methods.
Sampling trajectory. We compare different approaches for choosing sampling trajectory, including full reverse steps, linear and quadratic in DDIMs and ours using recognizability-ban skip function. Results in ssol that our recognizabillty-bann -based skipping (r-Shorteut) achieves the best results when performing the same total sampling steps, i.e., S = 30, suggesting the superiority of our sampling strategy. In addition, compressing sampling steps from 1000 to 30 using our method deteriorates the data quality very minor, while 16x faster speed is reached.
Effectiveness of h-)-). We further conduct experiments to testify if the learned network h. in Eq can faithfully reflect the recognizability of sketches generated during sampling. Specifically, we compare how well the predicted recognizability can match the probability of assigning the correct class label given by the pre-trained classifier. This experiments are conducted using 10 single class models, thus we can know which class probability in the classifier to be compared. Results in Table show that low and stable error can be achieved along the sampling steps.
Impact of N. To study the impact of point number N, we train our model with different settings. We can see from 3 it is inferior to using too fewer points (N = 24) for modeling relative complex structures, while applying too much points (N = 384) is also sub-optimal. An unique and optimal N is hard to reach as the complexity of sketch structure varies case by case.
Absolute coordinates works? To gain more insights about the importance of modeling relative coordinates by our model, i.c., (Ax, Ay). we train a model to learn from sketches represented by stroke points in absolute coordinates. i.e., yz, with other settings/component unchanged, denoted as "Ours (Abs)" in Table Significant decrease on performances is observed, verifying the crucial role of training model with relative coordinates.
What is learned? We suspect that capturing the implicit drawing structure is the key to success. A simple way to verify the speculation could be destroying the drawing structure. Specifically, we reorganize the original sketch data into a disordered version by randomly shuffling sketch segments
7

---

Published as a conference paper at ICLR 2023
Table 2: Averaged error (10k samples for each class) of the predicted recognizability using h (.).
10
30
50
80
200
700
800
1000
Frror 0.0600
0.0619
0,0723
0.0877
0.1006
0.0928 0.0935 0,0876 010891
0.08
0.19088
0.0972
0.1027
Table 3: Impact of points number n. (full reverse steps performed).
Simple
Moderate
Complex
Speed
FID GS Prec
Rec
FID
GS
Pree
Rec
FiD GS Prec Rec
N=24
7.4
4,4 0.32
0.74
14.7
13.3
0.30
0.76
16.3
18.2 0.26 0.09
0.16
Tw1000 N91
6.9 3.4 0.33
0.76
9.8
5.3
0.31
0.74
10.3
6.3 0.27
0.71
0.18
N=384
8.5 5.0 0.30
0.72
10.6
8.1 0.28 0.69
11.9 12.6 0.23 0.67
0.22
(each sketch segment is formed by connecting two neighbor stroke points). Intuitively, the newly constructed sequential data discards the implicit interdependent relations among stroke points, as the corresponding segments are geometrically far from each other. Then can obtain variant model trained with such structure-broken data, denoted by "Ours (Point-Shuffle)" in Table [ We can see that the performance is clearly harmed compared the model trained with normal data. Furthermore, we also experiment with a model variant trained with a stroke-level shuffle tactic, which means that the structural cues at stroke-level are preserved as each stroke remain complete, however the stroke orders are changed given the random shuffle applied. Less declines are observed ("Ours (Stroke-Shuffie strokes) are of great importance to our model. Note full steps are taken for these two model variants.
in Table [ revealing that the middle-level structural cues (i.e., complete
4.2 CONDITIONAL GENERATION
Experiments on sketch refinement and sketch healing are conducted to verify the effectiveness of the conditional sampling by our model. We show that, given any conditional sketch with defeats due to either stroke distortion or corruption, the generated sketch could be refined version accordingly.
Dataset. We utilize the same data in sec 4. for the sketch refinement task. To synthesis sketches with different degrees of distortion, random Gaussian noise e AO I) is added to each location of stroke points. The deformation degrees could be easily controlled by adding noise using different t timesteps. Because real sketch can be transformed into random noise after T diffusion steps, then a new sample obtained from the intermediate step / would be considered as x% deformed if 1 == .%T. The sketch healing task aims to create new sketches which should resemble the given the partial sketches. For fair comparison, we follow the same experimental setups in SketchHealer (Su) ctt al. (020). 17 categories from QuickDraw and the same data splits (70,000 for training and another 2,500 novel ones for testing per class) are adopted. Sketches are damaged using two different mask ratios: Pmask = 10% or 30,, that is, 10% or 30% key stroke points will be randomly removed to form the corrupted versions from complete sketch.
Sketch classifier. A multi-category classifier built AlexNet is pre pre-trained on the training set of all 345 Dric categories. Intuitively, better recognizability of sketches is manifested by higher recognition accuracy offered by this pre-trained classifier. Following the common practice in (Song] cet al. (2018); Su et al. (2020), acc @ and acc@ 10 are adopted as evaluation metrics. Specifically, acc @ K denotes the accuracy of the true positive is ranked within the top K predictions.
SBIR model. Similarly, sketch-based image retrieval (SBIR) model is pre-trained to verify if the rectified sketch generated by our model could enable better retrieval performances. Specifically, Triplet-SAN Yu et al.) (2016). which is constructed by employing Sketch-A-Net Yu et al. (2017) as backbone network, is trained on the QuickDraw Extended dataset Dey et al. (211)) under the supervision of triplet loss. Built on QuickDraw, QuickDraw Extended dataset is the largest SBIR dataset which contains 330,000 sketches paired with 204,000 photos over 110 categories. In our case, the SBIR model is trained by using the training set of the same selected 10 categories in the section The remaining testing set of the 10 classes is utilized as samples to be deformed. Evaluation metric acc@ I and aco@ 10 measure if a target image could be ranked within the top 1/10 retrieval results. The mean average precision (mAP) is adopted for evaluation well.
Results. As shown in Table 4 we can observe that (i) for all cases involving noisy sketches (i.e., deformation levels from 10% ~ 50%), performance improvements on recognition and retrieval can be achieved by using sketches after rectification, revealing the obvious enhancement benefits gained from our model: (ii) The recognition and retrieval results are stable regardless of the defor- mation level tackled, e.g., recognition accuracy @cc@1 I is kept around 47% ~ 50% for sketches
8

---

Published as a conference paper at ICLR 2023
Condition
Sketch-RNN
BeaterSkeoch
(a)
(c)
(d)
Figure 3: Qualitative comparison on sketch refinement (a)(b) and healing (c)(d) (a) Normal degree of complexity. (b) Complex classes. (c) Corruption Pmosk: 10%. (d) Corruption Pmosk = 30%.
Table 4: Recognition and retrieval results before and after (separated by "I") sketch rectification by our model at different deformation levels (DL). Performance gain in red. "-": without injecting noise, i.e., real data.
Recognition
Retrieval
DL
acc acc@1((%)
acc 10(%)
mAP
acc ((%)
acc 10(%)
51.9
52.4 (+0.90) 87.7
90.2 (+2.50)
0.704
0.789 (+0.045) 67.4
73.3 (+5.90) 91.3
96.2 (+4.90)
10% 45.7
48.9 (+3.20)
82.3
(+0.10)
0.724
0.788 (+0.064) 66.9
73.1 (+6.20) 92.1
96.8 (+4.70)
20% 33,0
47.3 (+14.3) 68.2
81.9 (+13.))
0.607
0.772 (+0.165) 55.8
72.8 (+17.0) 81.8
94.7 (+12.9)
30% 20.6
48.2 (427.6) 51.5
81.9 (433.4)
0.496
0.787 (+0.291) 46.9
72.8 (+25.9) 68.9
95.0 (+26..1)
50% 7.29
50.1 (+42.8)
27.1
84.3 (+57.2)
0.328
0.786 (+0.458) 28.6
74.9 (+46.3) 47.8 96.3 (+48.5)
with deformation 10% ~ 50%. (iii) Interestingly, recognition and SBIR results could be further significantly increased on the original human-drawn sketches after refinement. Exemplar refindd sketches from the original ones are demonstrate in Figure (a), revealing that badly drawn sketches (i.e., missing part, random strokes and line distortions) can be largely rectified by model, while other competitors either not able to faithfully resemble the conditions or unable to model complex structures. Quantitative healing results are shown in Table We forms all the other competitors on sketch recognition in most cases (expect the top 10 result when Pmosk 10%), indicating the superiority of our model recovering incomplete sketches. The advantage of our method is further enlarged when the corruption level increases, i.e., Pmack 30%. Similar situation can be observed for human study results. The healed sketches given by our model are mostly preferred compared against with other baseline methods. Qualitative comparisons against Sketch-RNN, SketchODE and BézierSketch are also provided in Figure 3 (b).
that generator outper-
Table 5: Comparison results sketch healing. Recognition results obtained by classifying generated healed sketches with a pre-trained multi-category sketch classifier. "Human" denotes human's preference of choice among the synthetic outputs by different competitors.
Pmask
Metric
SkeechhNNN SketchPix2seq (Sketcheealer Ours
Top
24.41%
21.88%
49.77%
56.61 %
Recognition
Top
46.23
31.92%
69.92%
71.63% %
10%
Top 10
56. 56.28%
96.91%
80.01 %
79.7
Human
N/A
21.11%
10.82%
33.56%
34.51 %
Top
3.14%
9.51%
41.59%
55.88 %
Recognition
Top
10.25%
16.06%
62.76%
74.57 %
30%
Top 10
15.91%
20.26%
68.12%
80.70% %
Human
N/A
6.14%
6.98%
30.76%
56.12% %
CONCLUSION
We show for the first time sketch generation can be formulated as a process of deformation-based denoising. The key finding is that increased sketch deformation degrees can be monotonically syn- thesized by diffusing stroke points with Gaussian noise, and the demanded probabilistic distribution of the stroke points of sketch objects can thus be effectively learned by diffusion inversion. Im- portantly, the ability of quantifying recognizability of the generated sketch was injected during the sampling. For that, a RNN was developed to predict the recognizability of a sampled sketch based on the estimated noise at each timestep. As result, more efficient sampling can be enabled by skip function. Additionally, our model trained for unconditional generation could be readily extended for conditional generation by incorporating a perceptual similarity based gradients into the sampling. Extensive experiments validated the effectiveness our model. To manage the abstraction level of the generated sketch would be potential future work.
recognizability--onn

---

Published as a conference paper at ICLR 2023
REFERENCES
Emre Aksan, Thomas Deselaers, Andrea Tagliasacchi, and Otmar Hilliges. Cose: stroke embeddings. 2020.
Compositional
Ruojin Cai, Guandao Yang, Hadar Averbuch-Flor, Zekun Hao, Serge Belongie, Noah Snavely, and
Bharath Hariharan. Learning gradient fields for shape generation. In ECCV, 2020.
Hong Chen, Ying-Qing Xu, Heung- Yeung Shum, Song-Chun Zhu, and Nan-Ning Zheng. Example- based facial sketch generation with non con-paamettrc sampling. In ICCV, 2001.
Yajing Chen, Shikui Tu, Yuqi Yi, and Lei Xu. Sketch pixeec:: a model generate sketches of multiple categories. In arXiv preprint vrXiv:/709.04122 2017.
Ayan Das, Yongxin Yang. Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. Béziersketch: A generative model for scalable vector sketches. In ECCV, 2020.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. Sketchode: Learning neural sketch representation in continuous time. In ICLR, 2021a.
Ayan Das, Yongxin Yang, Timothy M Hospedales, Tao Xiang, and Yi-Zhe Song. Cloud2curve: Generation and vectorization of parametric sketches. In CVPR, 2021b.
Sounak Dey, Pau Riba, Anjan Dutta, Josep Llados, and Yi-Zhe Song. Doodle to search: Practical zero-shot sketch-based image retrieval. In CVPR, 2019.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233, 2021.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv: 140.8556, 2014.
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, SM Ali Eslami, and Oriol Vinyals. Synthesizing programs for images using reinforced adversarial learning. In ICML, 2018.
songwei Ge. Vedanuj Goswami. Larry Zitnick, and Devi Parikh. Creative sketch generation. In ICLR, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurlPS, 2014.
Alex Graves. Generating sequences with recurrent neural networks.
arXiv preprint
arXiv:1308.088: 08500 2013.
Cheng-en Guo, Song-Chun Zhu, and Ying Nian Wu. Primal sketch: Integrating structure and texture. CVIU, 2007.
David Ha and Douglas Eck. neural representation of sketch drawings. In ICLR, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
Jonathan Ho and Tim Salimans. 1:Xiv:2207.12590 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Aapo Hyvärinen and Peter Dayan, Estimation of non- normmoomle statistical models by score match- ing. Journal of Machine Learning Research, 2005.
Valentin Khrulkov and Ivan Oseledets. Geometry score: A method for comparing generative adver- sarial networks. In ICML, 2018
10

---

Published as a conference paper at ICLR 2023
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-ennoding variational bayes. arXiv preprint
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In ICLR, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Tuomas Kynkäännieemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In NeurlPS, 2019.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 2006.
Lei Li, Changqing Zou, Youyi Zheng, Qingkun Su, Hongbo Fu, and Chiew-Lan Tai. Sketch-r2enn: An mnn-rasterization-mm architecture for vector sketch recognition. IEEE TVCG, 2020a.
Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan. Photo-sketching: Inferring
contour drawings from images. In WACV, 2019.
SuChang Li, Kan Li, Ilyes Kacher, Yuichiro Taira, Bungo Yanatori, and Imari Sato. Artpdgan: Creating artistic pencil drawing with key map using generative adversarial networks. In ICCS, 2020b.
Hangyu Lin, Yanwei Fu. Xiangyang Xue, and Yu-Gang Jiang. Sketch-bert: Learning sketch bidi- rectional encoder representation from transformers by self-supervised learning of sketch gestalt. In CVPR, 2020.
Difan Liu, Matthew Fisher, Aaron Hertzmann, and Evangelos Kalogerakis. Neural strokes: Stylized line drawing of 3d shapes. In ICCV, 2021.
Runtao Liu, Qian Yu, and Stella X Yu. Unsupervised sketch to photo synthesis. In ECCV, 2020.
Troy Luhman and Eric Luhman. Diffusion models for handwriting generation, 2020.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
Daniela Mihai and Jonathon Hare. Learning to draw: Emergent communication through sketching. In NeurlPS, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 2021.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In ICML, 2013.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. arXiv preprint vrXiv22105.06337, 2021.
Harsh Rathod, Manisimha Varma, Parna Chowdhury, Sameer Saxena, V Manushree, Ankita Ghosh, and Sahil Khose. Xci-sketch: Extraction of color information from images for generation of colored outlines and sketches. arXiv preprint arXiv:2108. 15544, 2021.
Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, and Moacir Ponti. Sketchformer: Transeormer-based representation for sketched structure. In CVPR, 2020.

---

Published as a conference paper at ICLR 2023
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.15500 2014.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Jifei Song. Kaiyue Pang. Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Learning to sketch with shortcut cycle consistency. In CVPR, 2018.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In /CLR, 2020.
Guoyao Su, Yonggang Qi, Kaiyue Pang, Jie Yang. and Yi-Zhe Song. Sketchhealer: A graph-to-
sequence network for recreating partial human sketches. BMVC, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Tianying Wang. Wei Qi Toh, Hao Zhang, Xiuchao Sui, Shaohua Li, Yong Liu, and Wei Jing. Robocodraw: Robotic avatar drawing with gan-based style transfer and time-efficient path op- timization. In AAAI, 2020.
Ning Xie, Hirotaka Hachiya, and Masashi Sugiyama. Artist agent: reinforcement learning ap- proach automatic stroke generation in oriental ink painting. IEICE TRANSACTIONS Infor- mation and Systems, 2013.
Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M Hospedales, and Chen-Change Loy. Sketch that shoe. In CVPR, 2016.
Qian Yu, Yongxin Yang, Feng Liu, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Sketch- a-net: A deep neural network that beats humans. IJCV, 2017.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018.
Xu-Yao Zhang, Fei Yin, Yan-Ming Zhang, Cheng-Lin Liu, and Yoshua Bengio. Drawing and rec- ognizing chinese characters with recurrent neural network. TPAMI, 2017.
Ningyuan Zheng. Yifan Jiang, and Dingjiang Huang. Strokenet: neural painting environment. In ICLR, 2018.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DENOISING DIFFUSION PROBABILISTIC MODELS (DDPM)
To learn the probability distribution over data diffusion models corrupt training data by slowly injecting noise and learn to reverse the corruption, such that the obtained models can gradually transform random noise into sample for data generation.
Forward process. Formally, for each training data X Adota (Xo), discrete Markov chain Xo. ........ Xr is formed by the forward process (also known as diffusion process). This process defined as a Markov chain which slowly adds Gaussian noise to the data according to a variance schedule $1, By....,By:
T (XxiT) = II
a xx-1)
(13)
((xx|xx-1) = (xxe; v B,xx-1,
(14)
If we know ((x:-1(xe), we could sample data from the data distribution g(xo) by first sampling from (xx)) (isotropic Gaussian) and then sampling from (xx,-1 (xx) until we get Xy- However, is difficult to estimate q (xx) since it needs entire dataset to do so. Therefore, pe is proposed to approximate the conditional probabilitiess (xx,,1 (x,) during the backward process.
Backward process. the backward/reverse process, diffusion models have to denoise the perturbed data (starting at random noise p(x.) .((x)) 0, I)) back to the origin data Xo. Mathematicallyy diffusion models is defined as
(x(x))= := f Dpo po (Noor)ddx11
(15)
in which the joint distribution po ():r) defines the reverse process:
(Xo:T) == p(xx) II [=1
pa (xx)
(16)
Po (x = -N(xx11;; (x t). (xx,x))
(17)
The training objective to optimize variational bound on negative log likelihood:
Pa (Xo:T)
E log po (Xo)] S FF,y [- log
9(x1:1 xoo)
Po Po (Xx)
(18)
E log p(T))
Σ
log
(xx; Xr-
121
which is equivalent to optimize the following variational lower-bound Letbi
Lutb == Lo Li LT- + Lr
(19)
Lo = log pe (xo(xx))
(20)
xo)llpo L == DKL Dx (xx11((xx)
(21)
LT = Dxz(g(xr(xx) (p(x))
(22)
Essentially, the above KL terms compare two Gaussian distributions which can be addressed in
closed form Ho al. 2020). The training objective for Eq (17) is to get pso (x;, (). while not
13

---

Published as a conference paper at ICLR 2023
involve [x, t), it is set to time-dependent constants oi=1 Furthermore, instead of predicting ue (xx, t) (forward process posterior mean) by a neural network, Ho et al. (2020) proposed to utilize an approximator eo (x,,,) to predict noise from N,, which is proven to be more effective than optimizing (x((x, t). The simplified training objective is:
L simple (f) == x000 (x t)
(23)
Data sampling. Once trained, we have neural network to estimate noise from sample x, at timestep f, i.., (x), Then (xo(ee,)) t) can be derived from ea (xx,)) t) by the following equation:
Vat _/(( (x:
1 a4 VI- dt
(X;, 1)
En (x.,
(24)
where 04 =1 11-3 and 0g == IT Os- To this end, we can sample data from po (0 122)) according
Eq 17 repeatedly until we reach So.
A.2
DENOISING DIFFUSION IMPLICIT MODELS (DDIMS)
To improve the sampling efficiency, authors of DDIM (Song et al. 2021) have proposed novel non-Markov chain process to reduce the forward and reverse process steps of DDPMs. They found a special property of the forward process of DDPM as
( mo) (a Varao, - (1-a4)
(25)
such that training objective of DDPM is able to not directly based on the joint q(<)1_(20)) they derive that
Then,
01-1 of (( Fc, vo) =N (Var-ino + v
Et Vo
of
(26)
VI
az
where t > and (xx)) = N(Varrao (1 -ar) And based on Bayes' rule, the forward process can derived as
go to(x; (0,1,,1,, = 9o (x [ xo 2o
(27)
4o (zt-1 (2))
i.c. the forward process is no longer Markovian each æ; is dependent on Ø;-1 and Tp.
During the generative process, they train noise approximator (0) estimate Pe (xx-1lax;) based on the probability go (xt 12,, 2o) as follows
(a1). if |a 18 (xx)), otherwise,
(1) Pg (#t-1(0.)) (20.)
(28)
where f/ (ax)) is the prediction of using the noise approximator co
(1) give nt, as
I (xx) = (æ, v1 art (1) (xx)) Vor-
(29)
Therefore as they derived their generative training objective is
(o( Exo-g(=s) [116]] (Varza + v1 onse) cal
(30)
where _N I). Once the noise approximator is trained, one generate a sample ær-1 give a sample from æ, as
V
a (x:) + 11- a:-1 - -o (f
01-1
(31)
Accelerated Generation Process The DDIM authors proposed to non-Markovian chain for the forward procedure as qo (x, (2)) can be directly estimated. Therefore, they propose to use sub- sequence from [1...., to speed up the generative process.
14

---

Published as a conference paper at ICLR 2023
N-06
N-96
N-96
N-96
No96
460
192
672
188
upsample
lincar cenvID
atientiin
downsongie
Figure 4: Architecture of the U-Net for estimating noise eg
(+)
and pen state g.
Table 6: Comparison results of using U-Net and Bi-directional RNN as noise approximator.
Model
Sample
Moderate
Complex
Speed]
FDD
GS}
Prec
Rec
FID,
GS)
Prect
Rec
G8}
Prec
ReeT
Ours-UNet
6.9
3,4
0.52
0.88
84
4.7
0.45
0.87
944
5.2
0.42
0.85
1.29
Ours- birs-biRNN
12.3
0.42
0.78
14.7
18.2
0.39
0.74
22.8
86.6
0.24
0.56
A.3 U-NET ARCHITECTURE
The architecture of our U-Net is shown in Figure 4 average pooling for downsampling, followed by a stack of upsampling convolutions and convolution blocks, with skip connections for concatenating feature maps in the same resolution. Additionally, a single head attention layer is used to inject the timestep embedding into the convolution blocks during downsampling following DDIM Song al. (2021).
It consists of a stack of convolution blocks and
Bi-directional RNN as noise approximator. Instead of using the conventional convolutional U-Net, different network architecture, i.c., bi-directional RNN, is explored to estimate the noise. Specifi- cally, given a sketch represented by a sequence of points 8, {s| s .?....... & where 8; E Nx2 bi-directional RNN is first applied map each point s RR² into hidden state hū RRd. Then all hidden states are concatenated together to construct the overall feature h E IR xd for a sketch. Finally, he is fed into fully-connected layer predict the noises, i.e., e R Similarly, the pen states can be obtained from h2 by another FC-layer. The results are shown in Table 6 We can find out that the obtained performances can not surpass the results using U-Net.
T=1000
T=300
T=0
Figure 5: Examples the process data sampling (one example in a row). We can see that early steps (T=1000 to about T=300) are often not effective that samples in the red box change slowly. In contrast, later steps get much more efficient to rectify sketches as in the blue box.
15

---

Published as a conference paper at ICLR 2023
GOOOOL
pD. 0
Figure 6: The obtained rasterized sketches using neural line rendering (NLR).
A.4 FURTHER INSIGHTS INTO SHORTCUT SAMPLING
Shortcut sampling is cute discovery we made that is specific to sketch data coordinates in sketch sequence are far less robust to added noise than pixel values of an image. This can be seen from Figure [[c) and Figure 5 that early sampling steps are often long and inefficient. Addressing this merely means a speed-up in generation while retaining quality (16x faster, see Table DI.
A.5 NEURAL LINE RENDERING
2020a). neural line rendering (NLR) is performed convert a vectorized sketch to its pixelative image. Specifically, given a sketch represented by sequence of points at timestep a bi-directional Long Short-Term Memory (LSTM) is firstly used to extract per-point features f/ Rd for each point sf:
Following Li
8: [s/.s] ..........".....................
s ........
[bi cl] LSTM st [h :c- fi = s gmo i dwh,; + b)
(32)
where h and c are the hidden states and the optional cell states, w and b are the weights and biases of fully connected layer. Then the point sequence and the features, i.e., {(s{ [.), are fed into the NLR module to produce a d-channel feature map size H x W x And the c-th channel of the feature map I (the timestep and the index for point omitted for clarity) be obtained as follows:
(1-ast) ax) ff + an fit if D(Le.PiPtt)) <Y I = 00
(33)
otherwise,
which means the pixel value IE computed by a linear interpolation of f and fitz (i.e., the c-th feature values of two nearby stroke points p; and p)+1) if J is stroke pixel (the distance from T to
the line segment P:Pc+1 is smaller than threshold, i.e., D(Tk, PiPi+1) < Note that pt and p:+1 absolute coordinates corresponding to points sf and s' And aq 1|p² pelle/1lpi41 pxll2- where d² is the projection point Iş on line segment p.P.+1-
To this end, by rendering the stroke points' features into pixel values Ig. vectorized sketch can be transformed into an image sketch. NLR is differentiable due to the linear interpolation based rendering, thus the gradient w.r.t the perceptual similarity given by 2D-CNN in 12 can be back- propagated to the vectorized sketch. Some examples rendered sketch images are demonstrated in Figure 6 where d = 3, 7 H and W are both set to 256.
16

---

Published as a conference paper at ICLR 2023
A.6 IMPLEMENTATION DETAILS
single Nvidia 3090 GPU is used for model training. The batch size is set to 512. The point number selected 96, which is the average value of stroke points in the QuickDraw dataset. Cosequently, sketches with more than 96 stroke points are excluded during model training. In addition, we will pad the point sequence with zeros to reach N=96 if the actual number of stroke points of any sketch is less than 96. It turns out that N = 96 works well for most cases since about 91% sketches (statistics from all the 345 categories) having stroke points less than 96. The default setting of skipping stride = 50, and the recognizabilityy threshold al,.... QT, 3,,...,,3y 3r is adopted to define the mean and variance of Gaussian noise. And linear noise schedule is leveraged and 3r is defined as:
0.2. Instead of directly using
B 31 T- x (3T - 31)
(34)
where 1001 and B7 0.02 in our case. Then the mean is /1 Be and variance is B, in Eq Adam optimizer (31 = 0.9 and B3 = 0.98) is used for optimization.
17


Reviews:
[{"review_id": "tR46hu3SV_0", "reviewer": "Reviewer_4TYn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to formulate sketch generation as the reversal process of sketch deformation, allowing the use of diffusion process that's more appropriate to stroke based sketch drawing. This is quite similar to [Luhman and Luhman, 2020] which use the same approach but on handwriting generation. The authors also propose recognition guided diffusion to speed up the diffusion process, and experiment on sketch completion based on perceptual metric. ", "summary_of_the_review": "The paper is pretty much a borderline for me. On one hand, it's the first sketch generation that formulate as diffusion process, along with other nice properties, on the other hand there are quite a lot of similarities with previous works that I'm not sure if the current contributions is enough. The choice of model is also somewhat confusing to me; convolutional U-Net with slight modification (mapping to/from 128 embedding) on coordinate type of input/output (delta x, delta y, pen state). RNN, however, is used to learn recognizability of sketch. It just seems a bit odd, I would love some clarification on the motivation behind these choice of architecture.  \n\nOverall, I do think the novelty is not everything. I am somewhat positive about the key ideas of the paper; diffusion for sketch generation, formulate as iteratively deform/de-deform the sketch, recognizability to guide faster steps diffusion, rectifying bad sketch, etc. The evaluation is also quite extensive, with ablation that make sense (love the random stroke/point order). So my initial recommendation is a slightly more toward accept, as long as the authors can provide clarification to my concerns. But I can be convinced either way.\n\nOthers:\nthe reference for Diff-HW seems to be wrong? isn't it out 2020 instead of 2010?", "strength_and_weaknesses": "Strength\n\n- the method and choice of coordinate makes sense for sketch\n\n- the results support the claim including how keeping the order is beneficial \n\n- recognition based step skipping and rectifying bad sketches are quite interesting\n\n- the evaluation seems quite extensive\n\nWeakness\n\n - I think there are a lot of similarities with existing works such as [Luhman and Luhman, 2020] down to the use of diffusion model on stroke-3 coordinate (delta x, delta y, pen state). This means the actual novel contribution of the paper is pretty small. The paper is more about applying existing methods toward a new problem than coming up with an entirely new approach, ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "f9uTGjjUtF", "reviewer": "Reviewer_zdyq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces generating vectorized sketch by modeling the stroke-point locations and pen states via a diffusion model. One major contribution is to embed recognizability of sketch during the sampling. It explores both cases of starting from a random scattered points or an incomplete sketch. Experiments show some appealing results as well as solid quantitative evaluations, ablation study and component analysis.", "summary_of_the_review": "This is a good extension of applying powerful diffusion models to vector domain which is potentially useful for more complex vector design or art regardless of resolution. I do not have too much criticism towards this work. ", "strength_and_weaknesses": "+ A nice addition of introducing diffusion model learning into the vector image generation, compare with tons of pixel image generation\n+ Joint modeling of both point location and pen state\n+ A new adaptive sampling strategy based on sketch recognizability\n+ The ablation in Sec. 4.1 is very impressive, covering every aspect about the proposed method so that readers know how each factor will affect the result\n\nThe diffusion process in Figure 2 (a) somehow could serve as a tutorial-like process to teach people how to draw an object. I have some minor concerns listed below:\n\nIs this diffusion model designed as class-conditioned or not? Basically, starting from a random scattered points, what type of guidance is driving those points to form the final object, or is it a one2many mapping?\n\nInstead of conditioning on an incomplete sketch to refine, one more interesting application is to invert a given sketch and generate its variations, to a simpler or a better version for different needs.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BcBOD87Stc", "reviewer": "Reviewer_ocgD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed using diffusion model to generate vectorized sketches. A vectorized sketch is represented as a sequence of points, and a *pen state* indicating whether the pen is touching the paper. The authors formulated the problem as generating a fixed number of 3D points using a 1D convolution-based DDPM. To accelerate the sampling process, the paper proposed using an RNN network to predict whether the noisy example is already recognizable -- empirically, more reverse steps are needed when the result starts to become recognizable.\nThe proposed model achieved SOTA performance on sketch generation. The version using RNN-based shortcut sampling required significantly less computation compared to Naive DDPM sampling while retaining the generation quality.\nIn addition, similar to image diffusion models, the proposed model is able to perform tasks such as sketch refinement and healing.", "summary_of_the_review": "The paper is well-written and the experiments are comprehensive. Being a domain-specific application of an existing technique (DDPM), the paper has not only thoroughly explored the benefits it provides, but also included some interesting new contributions, such as shortcut sampling. Although it will be better if this finding can be generalized beyond sketch generation.", "strength_and_weaknesses": "## Strength:\n* This is the first paper that applies DDPM to the task of sketch generation.\n* The proposed method significantly outperforms previous methods.\n* The proposed shortcut sampling scheme significantly reduced sampling time while retaining the quality better than DDIM given the same number of steps.\n* The proposed method has all the versatility a DDPM provides -- it can be applied to conditional generation tasks without retraining.\n\n## Weaknesses\n* The shortcut sampling technique, although worked well empirically, is not backed by theory. It will be more useful if the author can share more theoretical insights & generalize it to other tasks.\n* In section 3.8, the paper mentioned using a pretrained 2D CNN to evaluate similarity between two sketches. It is unclear how the gradient from a 2D CNN is propagated to a vectorized sketch. Is there differentiable rendering?\n* The paper used a fixed number of stroke points for each sketch. This might limit its real-world application.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%