Scale-Adaptive Diffusion Model for Complex Sketch Synthesis
Authors: Jijin Hu
Ke Li
Yonggang Qi
Yi-Zhe Song
Venue: ICLR 2024 poster (2024)
Keywords: Generative model
Classifier-based Diffusion Model
Sketch
Primary area: generative models
TL;DR: Our approach achieves a delicate balance between recognizability and complexity in generated sketches through scale-adaptive classifier-guided diffusion models

Abstract: While diffusion models have revolutionized generative AI, their application to human sketch generation, especially in the creation of complex yet concise and recognizable sketches, remains largely unexplored. Existing efforts have primarily focused on vector-based sketches, limiting their ability to handle intricate sketch data. This paper introduces an innovative extension of diffusion models to pixellevel sketch generation, addressing the challenge of dynamically optimizing the guidance scale for classifier-guided diffusion. Our approach achieves a delicate balance between recognizability and complexity in generated sketches through scale-adaptive classifier-guided diffusion models, a scaling indicator, and the concept of a residual sketch. We also propose a three-phase sampling strategy to enhance sketch diversity and quality. Experiments on the QuickDraw dataset showcase the potential of diffusion models to push the boundaries of sketch generation, particularly in complex scenarios unattainable by vector-based methods.

Published as a conference paper at ICLR 2024
SCALE-ADAPTIVE DIFFUSION MODEL FOR COMPLEX SKETCH S SYNTHESIS
Jijin Hu'
Ke Li
Yonggang Qi'
Yi-Zhe Song*
BBijjng University of Posts and Telecommunications CN (jijinhu, likel 990, ygygttb edu. cn
SketchX, CVSSP, University of Surrey, UK y. y.songesurrey.ac.
ABSTRACT
While diffusion models have revolutionized generative AI, their application to hu- man sketch generation, especially the creation of complex yet concise and rec- ognizable sketches, remains largely unexplored. Existing efforts have primarily focused on vector-based sketches, limiting their ability to handle intricate sketch data. This paper introduces innovative extension of diffusion models to pixel- level sketch generation, addressing the challenge of dynamically optimizing the guidance scale for classifier- guided diffusion. Our approach achieves a delicate balance between recognizability and complexity in generated sketches through scale-adaptive classifier-guided diffusion models, a scaling indicator, and the con- cept of residual sketch. We also propose three- three-phase sampling strategy to en- hance sketch diversity and quality. Experiments on the QuickDraw dataset show- case the potential of diffusion models to push the boundaries of sketch generation, particularly in complex scenarios unattainable by vector-based methods.
INTRODUCTION
The field of diffusion models (Ho et al. 2020 Song al.) 2020a bj Dhariwal & Nichol 202 Ho
& Salimans 202I) has seen remarkable progress, pushing the boundaries of generative Al and en- abling the generation of high- high-quality images across diverse domains (Meng al. 2021 Choi al. 2021 Rombach et al. 2022; Poole et al. 2022 Wang et al. 2023). However, this surge of ad- vancements has largely overlooked the unique challenge posed by human sketch generation-a task demanding the creation of complex sketches that maintain a delicate balance between conciseness and recognizability. Recent endeavors in this direction have predominantly centered on vector-based sketches (Ha & Eck [2018] Chen et al., 2017 Zang et al.) (222]). Unfortunately, these vector-base based approaches, while suitable for simpler sketches, grapple with inherent limitations when tackling intricate and complex sketch data (Das et al. 2022 Wang et al., 2022).
In this paper, we embark on an ambitious endeavor harness the full potential of diffusion models by extending their capabilities into the realm of pixel-based sketch generation. Our overarching goal to demonstrate their prowess generating complex sketches that strike the perfect bal- ance-concise yet highly recognizable. Determining the appropriate level of complexity in sketch generation has long been considered a formidable challenge, primarily due to the inherent variabil- ity in line structures within sketches. To address this challenge, we adapt a conventional classifier- guided pipeline (Dhariwal & Nichol] (021) designed specifically for sketch generation. However, this transition is not without its difficulties, as quickly encounter significant hurdle: the con- ventional designs, meticulously fine-tuned for photo generation, do not seamlessly transfer to the intricate realm of sketch creation (as illustrated in Figure I..
As we delve deeper into this problem space, we uncover a compelling revelation-one that diverges from the well-established principles governing photography. In the realm of photos, higher scale values often correlate with increased fidelity (Dhariwal & Nichol, 2021] Ho & Salimans] 2021). However, in the context of sketch generation, we encounter l'accinatig phenomenon that we term "over- sketchikt (as depicted in Figure (a)). When working with larger scale values, we witness the emergence of repetitive strokes, overlaying previously rendered lines and ultimately compromis- ing the quality of the generated sketches. Addressing this issue proves challenging. as there exists
Cooresponndncce Ke Li (like 19ike19900ee (eree@uuuutteen.... Code be found GitHub page

---

Published as a conference paper at ICLR 2024
Poor Fidelly
Over skotshing
0.6
0.0
Guidence Scale
Small Soale
Large Soale
0.0
0.4
Skatches Generated By Using Constant Scale Classffer-Guideed Sampling
Skatches Gonerated By Our Method
(a]
(b)
Figure I: (a) Vanilla constant scale classifier guided sketch sampling suffers from either insufficient recognizabilityy or over-sketching. In addition, there is no universal scale that is suitable for all cate- gories. (b) Sketches generated by our model are highly recognizable and more visually appealing.
no universal scale choice suitable for diverse sketch categories. Moreover, adopting smaller scales may result in insufficient recognizability, a trade-off that is also undesirable.
Our paramount contribution, therefore, revolves around the development of a specialized classifier- guided diffusion model meticulously crafted for the domain of sketches. At core, our model introduces dynamic sampling scale selector. This selector grapples with the intricate task of deter- mining the optimal scale for each distinct sketch while ingeniously sidestepping the issue of over- sketching. This delicate equilibrium ensures that our generated sketches strike the perfect harmony between recognizability and complexity.
This strategy pivots around two integral components that work in tandem: scaling indicator and a residual sketch. The residual sketch offers us a nuanced perspective on the influence of classifier guidance by tracking how the generated sketch evolves, pixel by pixel, under varying scale choices. This empowers us to pinpoint the scale where the residual sketch best aligns with the scaling indi- cator, thereby optimizing the entire generation process.
To further elevate the quality of our generated sketches, we introduce two supplementary design elements. Firstly, demonstrate the effectiveness of commencing the generation process with few unconditional sampling steps. This initial phase allows the rough structure of the sketch to take form, amplifying the diversity of the generated sketches by maximizing mode coverage. Secondly, we address the gradual attenuation of classifier gradients as the sampling deepens. To counteract this, we strategically implement an early-stop mechanism within our scale adaptive sampling, This seamless transition back to unconditional denoising accelerates the process while concurrently re- fining the generation results by eliminating noisy pixels.
Our contributions can be summarized as follows: (i) We introduce scale adaptive classifier classifier-uuieed guided diffusion models tailored for pixel-based sketch generation, replacing the conventional fixed gradient scale approach and achieving high-quality sketch generation. (ii) We present novel scaling indi- cator that optimizes classifier guidance based on recognizability and complexity, complemented by the innovative concept of the residual sketch, enabling fine-tuned control of the generation process in raw pixel space for improved sketch quality. (iii) Our three-phase sampling strategy, comprising shape and structure construction, scale adaptive sampling for class-specific sketches, and denois- ing, significantly enhances sample diversity and quality by removing background clutter. These contributions collectively advance the state-of-the-art in sketch generation with diffusion models.
RELATED WORK
Synthesizing human sketches an appealing task that increasingly received attention in recent years. Early studies (Song et al.] 2018, Wang et al.) 2018 Li et al. 2019b a Yu et al. 2020) and more recent arts (Chan et al.) 2022 Wan et al. 2022 focused on the problem of image-to-sketch generation, to help understand and mimic how humans perceive and represent the visual world using sketches. Another line of work is however concentrating on how to better capture the sequential features in human sketches within the single domain, involving RNN- based (Ha & Eck 2018 SS et al. 2020 Chen et al.] 2017), GAN-based (Ge et al., 2020 Liu et al. 2019), 2021b) approaches. These models typically adopt
Sketch Generation.
and Graph-based (Xu et 2021 Yang al.
2

---

Published as a conference paper at ICLR 2024
sequence decoder, i.e. LSTM Transformer, explicitly capture the geometric structure of the sequential points represented in coordinates or parametric Bézier curve (Das et al. 2020). As result, the sketch generation is formed as autoregressive process. Most recently, diffusion models (Das et al. 2022 Wang et al. 2022) are leveraged to directly learn the distribution of points' coordinates in non- aanaauorrerrsssvve manner, thereby advancing in generating complex sketches. Instead of using the sequential representation of stroke points, we seek to train diffusion models on the raster sketches composed of pixel grids to generate high-quality sketches. An additional property of pixel-basee diffusion modeling is that the classifier gradients (Dhariwal & Nichol (2021) can be conveniently applied as guidance without retraining the unconditional diffusion models or extra differentiable rasterization rendering.
Guided Diffusion Models. There large body of literature on controllable generation using diffusion models. The pioneering work ADM (Dhariwal & Nichol 2021) allows image generation conditioned on a class label by adding the classifier gradients to the frozen unconditional trained diffusion. Later, a Classifier-free approach (Ho & Salimans 202 I) is importantly proposed to avoid separately training the classifier while achieving similar sample quality, thereby triggering plenty of work text-conditional image synthesis, e.g., Stable Diffusion (Rombach et al. 2022). DALL-E 2 (Ramesh et al., 2022), GLIDE (Nichol et al. 2022) and Imagen (Saharia et al. (2022) to name a few. More broadly, latest works expand the scope of conditions to different modalities via cross-attention or adapter with CLIP, such as segmentation mask (Gafni et al.. 2022), sketch (Voynov et al. 2023).
and many others (Zhang & Agrawala, 2023 Mou et al. [2023). Our work different from previous works in that the strength of the classifier guidance is dynamically determined to manage the line complexity, improve the realism of produced sketches.
BACKGROUND
On a high level, diffuuson models can sample data from simple Gaussian distribution by reversing noisy data gradually in multiple steps. It typically consists of two inverse processes, i.e., the forward for diffusion and the backward for denoising.
Diffusion and Denoising The forward process is a predefined diffusion process q, which gradu- ally adds Gaussian noise to a real image zo, resulting noisier versions I1.T- It is formally defined as ([Ir|==t-) (ar;; v1 - -, 3,I). where 0 < 33 < 1, = 1 T is predefined variance schedule to specify the noise levels of x4. The backward process a denoising function po. where a neural network is trained to produce slightly clearer data xT-1 from at each timestep. Given pa. we can sample from pure noise øT and sequentially produce samples -T -T until reaching zo, i.e., produced sample.
Learning Objective As the backward process is also formulated as a Gaussian, i.e.. (((x)-1lxz) = -i(x;-1i t), (x t)), where the mean and variance of the Gaussian parameterized by μi and Eg. respectively. DDPM (Ho et al. 2020) show that the variance can be set to dime-dependent constant, i.e., Eo t) I11, the mean po (12, t) is then reparametrized by vieo(az,()) where ag = 1 - 3g and ā := -T' Os., Consequently, an alternative training objective (x)()),)) - cll2, ie.. MSE loss between the true and the estimated noise, is derived for training the diffusion models.
a noise approximator eg since we have ua (Zt, () 1/ (x, 3./vi
Classifie-guiieed Sampling To generate data conditioned on class labels, a classifer (,yy) can be trained on noisy data Then the gradient of the classifier, i.e., V., log (x)) is leveraged to guide the sampling. Specifically, the predicted noise after classifier guidance is:
é co
VI- ā. m, log (2)0x)
(I)
where s is a gradient scale predefined manually. Then the class conditional sampling is achieved by replacing eg with é. It turns out that the scaling factor s has a significant impact on the generated data, and increasing s will typically trade off the diversity for fidelity (Ho & Salimans 2021
4 METHODOLOGY
Our goal is to generate new sketches in pixel format conditioned on class labels using the learned denoising function, i.e., po (a [2,, y), using DDIM sampler (see Appendix A for more details
3

---

Published as a conference paper at ICLR 2024
Phase#1
Phase#f
Phase#3
Warm-up Sampling
fuss)
Scale Adaptive Sampling
(
42)
Denoising
xta
DM
%
DM
Plae
2,
Adappivv Scale
#p-1
Scale Optimization
DM Unconditional DM
Residual Skatch
Classilier Gradient
Scale
Matching Loss
f Fos
Recognizabiltty
Complexity
Scale
Noley Sketch Classifier
Fufe
SGO
Stochestic Gradient Descent
Figure 2: Schematic overview of our pixel-level sketch generator. There are three sampling phases, ie., warm-up sampling (phase #1), scale adaptive classifier-guided sampling (phase #2), and end-up denoising (phase #3). Core to framework is phase #2 which can adaptively select an optimal classifier guidance scale to encourage better recognizability and avoid over-sketching, thereby boosting the sample quality. Essentially, the scale is dynamically determined by matching scale indicator and residual sketches at each sampling step. The scale indicator is to signal the demand for classifier guidance by predicting the final generation results Tojz- The residual sketch measures whether the chosen scale could activate proper guidance as the indicator suggests.
about DDIM sampling) under the guidance of classifier gradients as described above. Particularly, an adaptive scaling strategy is devised to dynamically determine the level of the gradient scale at each time step to improve the quality of the produced sketches. schematic overview of our model shown in Figure 2 Details described in the following.
4.1 SCALE ADAPTIVE TLASSIFIER-GUIDEE SAMPLING
Following the standard pipeline for handling natural images, first train a DDPM using sketch images, i.e., Zo E R HxW x3 During generation, as shown in Figure constant value of the scale in Eq. ① will lead to sub-optimal quality of the produced sketches, suffering from either insufficient recognizability or over-sketching. Therefore, scale adaptive sampling strategy is developed to overcome the above issues. Specifically, for each intermediate sampling step scaling indicator is used to penalize the guidance strength when the complexity and recognizability of the expected produced sketch Znj+ are already sufficiently high. residual sketch image Tre then to explicitly measure the impact of any gradient scale, by leveraging the visual difference of Zoje before and after performing the guidance. Then the scale optimized by using differentiable matching module to encourage the residual sketch to conform to the scaling indicator, thereby steering the generation accordingly. the following, we will formally define each key module.
Scaling Indicator. High recognizabilityy and proper complexity are two crucial properties to make the produced sketches informative and visually appealing. Therefore, we empirically define scaling indicator by combining these two factors:
s(xr) exp(a (1 - c(zu(:)) - f(xo
where Zoje (xz VI- ai to(xx,t)) va,, c(xon) and f(0oj) are all scalar, denoting the complexity and the recognizability of expected produced sketch Tojs, respectively. Specifically, the stroke complexity c(xop) is heuristically defined as the fraction of stroke pixels to the whole canvas: c(Zaje) = nw Σnw || Ilo- The recognizability f (Faae) is given by the probability of the estimated produced sketch zoje being classified to the conditional class y, i.e., [((oj) = P (uj)) where the classifier Ps is parameterized by ø. and trained using noisy sketches. Intuitively, the scale indicator <(z;) is to signal the demand for classifier guidance. For example, high level of either complexity or recognizability will derive a small <(æ;), suggesting a stop sign for applying the classifier guidance. In contrast, large s(x:) implies the classifier guidance in need. and
4

---

Published as a conference paper at ICLR 2024
used to balance the effects between c(Zore) and ( ). In the following, will show how to optimize the scale s according to the scaling indicator by introducing residual sketch.
Residual Sketch. To measure the impact of classifier guidance Zaja, i.c., the estimated final- step sketch žo based on the current sample 2t, we can compare two versions of zoje before and after performing the classifier guidance, i.e., Fojr and Soj-- Namely, residual sketch Tra is developed to represent the per- per-pixel differences between Zoje and Sott- Formally, Tys is defined as follows:
( 8) (foj) M (zooe)
where (.) E RR is a S Sigmoid function transform sketch (pixel values are firstly averaged across the RGB channels) into a soft binary mask, i.c., most of the pixel values are projected near zeros or ones. And!
ensure the entries in Zra " are all positive.
Scale Optimization. Here, we show how to optimize the gradient scale according to the scaling indicator at each sampling step. Basically, this achieved by enforcing the residual sketch Zys to be synchronized with the scale indicator s(zz). Intuitively, Tys should be an empty mask if s(x)) suggests a stop sign for applying the classifier guidance. Otherwise, Iys should be richly painted if (1x+) is large, indicating guidance in high demand. Therefore, optimization objective can be formulated as follows:
2
(s)
((x(())
(1)
Σ (r)( (
HW
HW
where N is the number of sketches generated within a sampling batch, and Lr(s) is the mean squared error between the scaling indicator s(x;) and the global average pooling of the residual sketch Zya (Z,, s). Stochastic gradient descent (SGD) is employed obtain the optimal value of at each timestep by minimizing Ls))..
4.2 WARM-UP SAMPLING FOR DIVERSITY EXPANSION
Prior works Dhariwal & Nichol (02T): Ho & Salimans 2021 reveal that increasing the strength of the classification guidance can improve the sample precision (i.c., fidelity), while at the cost of re- call, i.e., the degraded diversity of the generation. We show that beginning with a few unconditional sampling steps warm-up can considerably alleviate the issue, i.e., boost the diversity of the gen- crated samples. An empirical principle applied to determine how many unconditional sampling steps are conducted for the warm-up. The idea is to carry out unconditional generation until the overall structure has been shaped. To achieve the goal, we can simply measure if the classification probability of the top-1 class, i.e., p(Cist (roee), exceeds any other classes by pre-defined margin y)- Therefore, the end step to of the warm-up sampling can be set by:
to = t, if
- Ps (C2nd >
(5)
4.3 END-UP UNCONDITIONAL DENOISING
We observe that as the sampling goes on, the classifier gradient will gradually vanish and almost has no effect afterward. Therefore, we early-stop the classifier-gus guidedd sampling when the expected Zoje at timestep la is sufficiently recognizable, i.e.,
(6) where E is a threshold determine the endpoint of the guidance. However, it turns that the produced sketches are still noisy (i.e., lots of clutter pixels scattered across the image) once the dlassifier-guides sampling is ceased at the cut-off point tu. We find out that continuing to progress unconditional denoising till the end (i.e., the pre-defined total number of DDIM sampling steps) will eventually produce sketches with clean backgrounds.
tu = t. if (yll) >
EXPERIMENTS
5.1
EXPERIMENTAL. SETUP
Dataset. The current largest doodle dataset QuickDraw, which has 345 common object cate- gories, is adopted for model training and evaluation. In experiments, small subset, i.e., 30
5

---

Published as a conference paper at ICLR 2024
categories are first randomly chosen to facilitate a thorough yet easier comparison with other base- line methods. Meanwhile, our model is trained on the complete 345 classes and compared with a few top-notch generative models to validate the scalability of our approach.
Competitors. There are two categories of baseline methods based on the representation of the sketches, i.e., vector-based or raster-based. Vectorized sketch generation competitors include
SketchRNNN (Ha & Eck, 2018). SketchHealer (Su et al. 2020), SketchAA (Yang et al.] 2021a),
SketchKnitter (Wang et al. 2022), and ChiroDiff (Das et al. 2022). Due to the lack of existing raster- based sketch generation approaches, three strong image generation models, i.e., StyleGAN2 Karras
al. 2020 DDIM P Song al. 2020a), and classifier-free diffusion guidance (CFDG) (Ho & Salimans 2021), are employed as alternatives for comparison.
Evaluation metries. Several standard metries, including FID, precision, and recall, are leveraged for evaluation. Fréchet Inception Distance (FID) is widely used to measure the fidelity of the RGB images produced by generative models. To tailor same network Inception-V3 is employed but further finetuned on QuickDraw dataset for classifica- tion. Then the obtained customized Inception-V3 is utilized as a feature extractor, which used to calculate the distance between the generated samples and the real data. Precision and recall are typically adopted by diffusion models to validate the quality and mode coverage of the generated samples. We follow (Nichol & Dhariwal] (02T) to employ the improved precision and recall metrics Kynkäänniemi et al. 2019) to assess the generation results.
reasonable measurement for sketches, the
Language Aligned Expressivity; We additionally proposed to utilize CLIP-Score (Radford et al. 2021) to measure the expressiveness of the generated sketches. Intuitively, the produced sketches would express similar visual concepts real sketches. In practice, CLIP-Score is leveraged to measure the distance between the generated sketches and the text descriptions of real ones, where the text descriptions are sourced by manually summarizing the visual content in the QuickDraw dataset. More specifically. five captions are constructed for each category under the template "this a sketch of x", where X is either coarse or fine- uine-griieed linguistic caption, e.g., "a girl's face with two pigtails". A full list of the captions is attached in Appendix E Then pre-trained CLIP model VIT-L/14 (Dosovitskiy et al.) 202T) used to extract the features of the generated sketches and the pre-defined descriptions using the image encoder and text encoder, respectively. The CLIP- Score is then defined as the averaged similarity between the generated sketches and their closest captions. Given CLIP-Score, we further propose CLIP-Fine score which measures whether the retrieved t captions are fine-grained or not.
Implementation details. The same U-Net proposed in ADM (Dhariwal & Nichol 2021 is em- ployed as the noise predictor, and 10k sketches per category (batch size is 64) in the training set used train our model for 200k iterations. The default size of the produced sketches is set to 64x x 64. Four Nvidia 3090 GPUs are used and the learning rate is set to le-4. An EMA rate of 0.9999 adopted to stabilize the training. The default parameters are a 1.0, 8 0.2, and 7 = 0.02 in equation 2 which are determined on a validation set through greedy search. (See Appendix B for more details.) And we set 17 0.2 in equation 5 = 0.5 in equation 6 emppricall. During genera- tion. the DDIM sampler is adopted and the default total steps are set to 250. And we can produce a batch (N=128) of sketches by simply calculating the average values of (ax)) and Tys( s) within the batch, hence updating the loss T. (s) into batch version accordingly. UsingglllyUunnngnn.. shared scale within a batch can dramatically reduce the computation cost and speed up the sampling. Additionally, we use asymmetric reverse process (Kwon et al. (023) to improve the controllability of classifier guidance, i.e., compute xt-1 using the predicted noise before and after performing the guidance together.
5.2 RESULTS
Quantitative Results. As shown in Table our model outperforms other competitors on all met- rics except precision (ours achieves the second best). Interestingly, pixel-based generation methods (i.e.. StyleGAN2, DDIM and ours) can clearly beat vector-based approaches. Additionally, our
fish, umbrella, apple, moon, shoe, cloud, candle, vase, chair, sun, cat, airplane, spider, car, pig, bus, face, yoga, butterfly, mosquito, lion. television, basket, barn, angel. pizza, book, grapes, fireplace, cake
"The guidance scale for DDIM is set to 0.4 determined by greedy search, offering its best FID score. 'The optimal guidance strength of CFDG is set to 2 for sketch generation.

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative comparison results on QuickDraw Dataset. The best and second best are color-coded in red and blue, respectively.
Random Categories
345 Categories
Model
FIDJ
CLIP-Scoref CLIP-Fine($6)) Prect
Reef
FIDI
Prect Reet
SketchRNN
8.15
0.59
52.67
0.37
0.22
10.32
0.26 0.24
SketchHealer
5.85
0.63
51.51
0.67
0.12
SketchAA
5.98
0.59
50.41
0.51
0.17
SketchKnitter
7.05
0,55
43.15
0.41
0.19
ChiroDiff
4.75
0.59
53.16
0.64
0.18
3.17
0.58
0.25
StyleGAN2
4.12
0.67
53.39
0.55
0.24
2.93
0.63
0.27
DDIM
4.08
0.67
54.19
0.30
2.85
0.74
0.31
CFDG
3.75
0.68
54.86
0.68
0.32
2.64
0.73
0.36
Ours
3.08
0.68
55.54
0.68
0.35
2.51
0.72
0.39
model achieves the highest CLIP-Score, indicating that the produced sketches by our model can best align the visual content of real sketches. Notably, higher CLIP-Fine score (i.e., 55.5%) im- plies that our model tends to produce richer visual content. Some examples of the generated sketches and the retrieved captions are shown Figure [4] which showcase how our generated sketches can align with the captions summarized from the real sketches.
SketchRMN
ChiroDM!
Shatchttaeler
BlyleGAN2
SketchAA
DOIM
StetccKnnttter
Ours
Figure 3: (a) Qualitative comparison results. (b) More generation results by our model.
Qualitative Results. Some qualitative results are shown in Figure 3 From Figure Bia), we can observe that: (i) sketches generated by our model are of better quality terms of expressiveness, see the drawn details, see the drawn antennae of butter and the windon of car by our model, while these subtle parts are absent from the sketches obtained by other baseline methods. (iii) the sketches produced by our method are more visually appealing and recognizable, e.g., the eyes and nose on the human face are more vividly portrayed, and the overall visual appearance of 1 oion is better and more identifiable. More samples generated by our approach can be found in Figure B () and F Appendix We also visualize example of the sketch generation process in Figure to better understand the effects different sampling phases. We can observe that the overall shape of the expected sketch is formed during the warm-up sampling. The scale adaptive guidance sampling instantiates the generation according to the classifier guidance, yielding a sketch of a desired class that fits the overall shape formed in the previous stage. The last phase (i.e., end-up denoising) is responsible for further refinement.
of cat. (ii) Our method is also capable of depicting objects with more
5.3 ABLATION STUDY
Computation analysis and sampling acceleration. The number denoising steps during gen- eration is squeezed to 250 using the linear selection procedure following DDIM. To testify how the selection procedure and total sampling steps trade off the overall generation quality and the computa- tional cost, compare the results under different settings. Results in Table revell that the linear procedure is typically better than its quadratic counterpart. More sampling steps lead to improved FID and precision but with slightly reduced recall. However, the computational cost is obviously
7

---

Published as a conference paper at ICLR 2024
8
8
8
8
(a)
20p
Phase#1 Warm-up Sampling (Unconditional)
(b)
Phase#2 :ccae Adaptive Sampling (Conditional)
Phaser3 Ending-up Denoising (Unconditiona)
(c)
21
Figure 4: Visualization of Foje and xt. (a) The estimated final obtained sketches zoit time step t during warm-up sampling (green box). The overall structure has been formed in this phase. (b) Given different class labels, i... cat and car, Toje is gradually transformed into the corresponding sketch object by scale adaptive sampling (red box). The end-up denoising (blue) can further refine the sketches by removing the blur in the background. (c) Sketches generated at different time steps.
Table Ablative studies on applying different (a) skip procedures and (b) sampling phases.
(a) Computation analysis.
Acceleration in gray.
(b) Rffect of each sampling phase.
Procedure Steps|
FID]
Prect
Rect Speed (s)
Model Variant
[FDDL CLIP-Scoref Prec?
Reet
Speed (s)
Quadratic
100
8.85
0.58
0.48
0.86
No warm-up
3.22
0.63
0.69
0.32
2.87
Linear
100
4.95
0.62
0.48
0.90
No Adaptive
3.54
0.62
0.67
0.42
3.07
Quadratic
250
5.54
0.61
0.32
1.87
Full Guidance
4.08
0.67
0.71
0.30
3.27
Linear
250
3.08
0.68
0.35
1,90
No End-up Denoising
3.24
0.62
0.68
0.25
5.74
Shortcut 67
3.30
0.67
0.36
0.98
Our Full Model
3.08
0.68
0.68 0.35
1.90
increased in this case. To accelerate the sampling, we will later show that the end-up denoising can occupy up to about 86% of the total sampling steps that can be dramatically shortened.
Effect of each sampling phase. There are three phases in order during sampling, i.e., the warm- up sampling, the scale adaptive sampling, and the ending-up denoising sampling. To verify the effectiveness of each phase, we evaluate the generation results in different scenarios. Specifically, (i) No Warm-up: Without performing the warm-up sampling, we directly perform scale adaptive sampling at the beginning, followed by the end-up unconditional denoising till the end; (ii) No Adaptive: We keep all sampling phases unchanged (i.e., the start and end of each sampling stage remain unchanged) but switch the scale adaptive sampling to the vanilla classifier sitliierr-gidedddd sampling, i.e. applying constant gradient scale; (ii) Full Guidance: All generation steps are classifier-guidd samplings with a constant gradient scale. (iv) No End-up Denoising: Sketch generation starts with the warm-up up sampling, followed by scale adaptive classifier-guide sampling till the end (i.e., 250 steps are reached).
The results shown in Table 2h Compared with our full model, up: Both the FID and recall are getting worse when without the warm-up sampling, indicating that carrying out unconditional generation the beginning can benefit both fidelity and diversity; (ii) No Adaptive: Applying a constant scale (i.c., 8 = 0.4) to the classifier gradients will clearly harm the quality of generation, i.e., fidelity (FID). (iii) Full Guidance: Merely performing classifier guidance with a constant gradient scale (i.e., (iv) No End-up Denoising: Both the fidelity and mode coverage of the produced sketches are com- promised when maintaining the classifier guidance till the end. This is because too strong classifier guidance can lead to over-sketching, hence resulting in declined sample quality and diversity. More- over, the generation is much more expensive (i.c., 5.74 per sketch) in this case due to the increased sampling steps required for gradient scale optimization.
find out that (i) No Warm-
0.4) will simultancously lower the fidelity and diversity;
Length of each sampling phase. To study the influence of varying the length of each sampling phase, we compare the generation results using different configurations of the parameters and g in Eq. (5) and Eq. (6). Results are shown in Table 3 We can find out that most steps are occupied by the end-up denoising sampling, which can be shortened for acceleration as shown in the last row in Table 3 When increasing & and fixing 17, the length of the scale adaptive sampling becomes longer,
8

---

Published as a conference paper at ICLR 2024
Table 3: Comparison results when varying the parameters 7) and g.
# Steps
Warm-up (%) Adaptive(%) End-up( (6
FID
Prect
Rect
Speed (s),
0.3
6.01
8.26
85.73
3.21
0.65
0.38
1.76
0.2
0.4
6.01
10.02
83.97
3.17
0.66
0.37
1.83
0.5
6.01
11.73
82.25
3.08
0.68
0.35
1.90
T=250
0.1
4.47
12.56
82.97
3.12
0.69
0.32
2.03
0.5
13.27
7.52
79.21
5.37
0.61
0.42
1.78
0.4
18.52
5.44
76.04
8.53
0.57
0.48
1.67
T=67 (shortcut) 0.2
15.22
40.90
43.88
3.30
0.67
0.36
0.98
236
stops218
stepr200
Bay
(D)
Figure 5: Visualization of the residual sketches 2rs and the estimated final sketch Znj. during scale optimization at (a) early (b) middle and (c) late sampling steps. By scale optimization, the residual sketches are getting more organized and cleaner.
leading to improved sample quality (i.e., lower FID score) yet narrowing the mode coverage (i.e., reduced recall). Extending the warm-up sampling will squeeze the scale adaptive sampling, and let the sample quality get worse but achieve better diversity. The optimal balance is reached when warm-up takes about half the number of the adaptive sampling steps.
To better understand the mechanism of utilizing scaling indicator as an explicit signal to obtain the optimized gradient scale, we visualize the optimization process along with the corresponding residual sketches ( 8) and the estimated final sketch Toje. As shown in Figure at the beginning of optimization, the randomly initiated guidance scale often mismatched with the scaling indicator 5(x+). The corresponding residual sketch sy s) looks messy and unstructured, implying less favorable (i.e., too noisy) output sketch Hojz under the classifier guidance. Once the gap between <(x,) and (1r, s) is closed, the residual sketch becomes cleaner and more organized. As result, the expected sketches onjs using the optimized scale are painted in more structured and concise way.
Visualization of scale optimization.
6 CONCLUSION
Raster sketches generated by diffusion models using classifier guidance with a constant scale are sub-optimal, either too sparse to recognize or too densely depicted (i.c., over-sketching). We show that the generation quality can be improved by simply adjusting the guidance scale dynamically at each sampling step, without retraining the model. Concretely, we proposed according to the predictable generation results at each sampling step by using the developed scale indicator and residual sketch. It is observed that the pixel changes, i.e., the residual sketches, dur- ing sampling are more organized and located at critical positions to form sketch object by our model. Injecting unconditional sampling the beginning and the end of generation also benefi- cial. Uniquely, proposed assess the generated sketches in terms of expressiveness by using the CLIP-Score. shows that ours can generate sketches containing richer object details.
optimize the scale
ACKNOWLEDGMENTY
This work was supported by NSFC under No. No.61601042 and the Program for Youth Innovative Re- search Team of BUPT under No. 2023ONTD02.

---

Published as a conference paper at ICLR 2024
REFERENCES
Caroline Chan, Frédo Durand, and Phillip Isola. Learning to generate line drawings that convey geometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7915-7925, 2022.
Yajing Chen, Shikui Tu, Yuqi Yi, and Lei Xu. Sketch-pix2seq: a model to generate sketches of multiple categories. arXiv preprint arXiv:179.00421. 2017.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. In 2021 IEEE/CVE International Conference on Computer Vision (ICCV), pp. 14347-14356. IEEE Computer Society, 2021.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song, Béziersketch: A generative model for scalable vector sketches. In ECCV, 2020.
Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song. Chirodiff: Modelling chirographic data with diffusion models. In The Eleventh International Conference on Learning Representations, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 1148780-8794 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.. /929, 2021.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. European Conference on Computer Vision, pp. 89-106. Springer, 2022.
Songwei Ge, Vedanuj Goswami, Larry Zitnick, and Devi Parikh. Creative sketch generation. In International Conference on Learning Representations, 2020.
David Ha and Douglas Eck. neural representation sketch drawings. In International Conference on Learning Representations, 2018.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11233 2020.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. arXiv preprint 1wX.v:1912.04958, 2020.
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210. 10960, 2023.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances Neural Information Processing Systems, 32, 2019.
Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan. Photo-sketching: Inferring contour drawings from images. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1403 1403-1412, IEEE, 2019a.
Yijun Li, Chen Fang, Aaron Hertzmann, Eli Shechtman, and Ming-Hsuan Yang. Im2pencil: Con- trollable pencil illustration from photographs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1525-15344 2019b.
Fang Liu, Xiaoming Deng, Yu-Kun Lai, Yong-Jin Liu, Cuixia Ma, and Hongan Wang. Sketchgan: Joint sketch completion and recognition with generative adversarial network. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 5830-5839, 2019.
10

---

Published as a conference paper at ICLR 2024
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. International Conference Learning Representations, 2021.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8177 PMLR, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference Machine Learning, pp- 16784-16804. PMLR, 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. The Eleventh International Conference on Learning Representations, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 1768- PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp.
2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-3666 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020a.
Jifei Song, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Learning to sketch with shorteut cycle consistency. In Proceedings of the IEEE conference computer vision and pattem recognition, pp. 801-810, 2018
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020b.
Guoyao Su, Yonggang Qi, Kaiyue Pang. Jie Yang, and Yi-Zhe Song. Sketchhealer a graph-to- sequence network for recreating partial human sketches. In Proceedings of The 31st British Ma- chine Vision Virtual Conference (BMVC 2020), pp. 1-14. British Machine Vision Association, 2020.
Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. keetch-guided extt-to-image diffusion mod- els. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Zhaoyi Wan, Dejia Xu, Zhangyang Wang, Jian Wang, and Jiebo Luo. Cloud2sketch: Augmenting clouds with imaginary sketches. In Proceedings of the 30th ACM International Conference Multimedia, pp. 2441-2451, 2022.
Lidan Wang, Vishwanath Sindagi, and Vishal Patel. High-quality facial photo-sketch synthesis using multi-adversarial networks. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp. 83-90. IEER, 2018

---

Published as a conference paper at ICLR 2024
Qiang Wang, Haoge Deng, Yonggang Qi, Da Li, and Yi-Zhe Song. Sketchknitter: Vectorized sketch generation with diffusion models. In The Eleventh International Conference on Learning Repre- sentations, 2022.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. arXiv preprint vrXiv:2305.16213 2023.
Peng Xu, Chaitanya Joshi, and Xavier Bresson. Multigraph transformer for free-hand sketch recognition. IEEE Transactions Neural Networks and Learning Systems, 33010155100-511, 2021.
Lan Yang. Kaiyue Pang. Honggang Zhang, and Yi-Zhe Song. Sketchaa: Abstract representation for abstract sketches. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10097-10106, 2021a.
Lumin Yang, Jiajie Zhuang, Hongbo Fu, Xiangzhi Wei, Kun Zhou, and Youyi Zheng. Sketchgnn: Semantic sketch segmentation with graph neural networks. ACM Transactions on Graphics (003),,0 2021b.
Jun Yu, Xingxin Xu, Fei Gao, Shengjic Shi, Meng Wang, Dacheng Tao, and Qingming Huang. Toward realistic face photo-sketch synthesis via composition-aides gans. IEEE transactions on cybernetics, 51(9):4350 4362, 2020.
Sicong Zang, Shikui Tu, and Lei Xu. Controllable stroke-based sketch synthesis from self- organized latent space. Neural Networks, 137:138-150, 2021.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to- text-to-image diffusion models. arXiv preprint 200552300..05543 2023.
12

---

Published as a conference paper at ICLR 2024
A DDIM SAMPLING
DDIM (Song et al. 2020a reveals that the learning objective of DDPM (Ho et al., 2020) only depends on the "marginals" ( (2)) rather than the joint ((Z1r=(3))) thus non-markovian forward process is proposed such that the desired marginals are fulfilled, yielding the same learning objective as DDPM. Formally, the forward process of DDIM is formulated as:
(( Fe; 20n (3te
g(0 (T))
where q (2o) and (x0)-1/20) can be obtained by the marginals: g( (mo) N M(zi Vāzto. (1 at)
(8)
and g (2t, mn) is Gaussian defined in the following:
It - v Va
( Fr. mo) = NN 100-1 + â of
o
(9)
v1 ae
where the choice of mean function is to ensure the desired marginals, i.e., ( = (Zx; zā, (I- for a. Then during the generative process, q(2,-1 (2, mn)) can be used to approximate the denoising function 3y at timestep derived from Eq. (8)
without knowing zo. which can be predicted from
2o = (x: VT- ar cg (xt))/
(10)
Consequently, derived from Eq.
9 we can sample data by repeating:
Tt-1 a
37 v1 (34
/ -
u a of < + oret
(11)
Vix
DDIM shows that sampling could be accelerated using much fewer steps when setting o; = for all timesteps t.
B HYPERPARAMETER SETTING IN EQUATION 2
The hyperparameters (i.e.. a, 3, and 7) of Equation 2 for calculating the scaling indicator are deter- mined by greedy search on small validation set, comprising of sketches generated by model. Specifically, we generate 1k sketches per class by conducting classifier guidance on the obtained unconditional DDPM under different hyperparameter configurations. As shown in Figure 6 the
optimal values are a = 1.0, 3 = 0.2, and y = 0.02.
0.02
a= 1.0
y-0.02 B 0-2
3.7
0.5
3.8
y-
1.0
3.5-
-20
y= 0404
5.0
y-
3.4
4.5
3.4
3.2
3.5-
3.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
(a) Searching a, 3 with y fixed.
(b) Scarching 3, y with a fixed.
(c) Searching a with 3 and y fixed.
Figure 6: Greedy search on hyperparameters B, and y based on FID scores.
C Loss CURVE OF L,(s) DURING SCALE OPTIMIZATION
To gain more insights into the process of scale optimization, plot the scaling indicator <(x (2) the fraction per- virepixel difference of the residual sketch, i.e., the Tre term HW Σnw Ire( s), and their resulting loss (s) at six different time steps which are evenly selected during the stage of scale adaptive sampling. As shown in Figure 7 we can see that L, can converge quickly after few SGD optimization steps for various sampling time steps t.
(i)
13

---

Published as a conference paper at ICLR 2024
Step=211
Step=216
Step=221
{(2+)
2.60%
((x))
2.63%
((x))
lemn
2.60%
term
L((s)
2.50%
L-(e)
2.50%
4((+)
2.20%
2.40%
2.53%
2.00%
2.30%
2.53%
2.50%
1.80%
2.20%
2.48%
2
Optimizer_ _,sopp
Optimizer_ _step
Optimizer step
Step=225
Step-229
Step-233
3.60%
(t.)
6.00%
((x))
8.00%
((a)
... Sie
5.50%
I.(e)
5.00%
1.(+)
7.00%
I.(e)
3,20%
4.50%
6.00%
3.00%
4.00%
5.00%
3.50%
2.80%
4.00%
3.00%
0
2
Optimizer step
Optimizer_stoe _ttreerpsppppppee
Optimizer_stoo _tte
Figure 7: Scaling indicator (denoted by the green line) and the 2 term (denoted by the red dashed line), and their corresponding loss (s) (the gap between the red and green lines) at different steps during scale optimization.
Tole
0.7
0.6
Accuracy
0.4
0.3
0.2
0.1
0.0
250
200
150
100
50
0
Step
Figure& 8: Top-1 classification accuracy on noisy sketches in and the estimated sketches Zoje-
D THE EFFECTIVENESS OF CLASSIFICATION ON Toje
To validate the applicability of the classifier (trained using the noisy sketches xx)) to recognize the estimated sketch Zntt. we provide the results of the top-1 classification accuracy both on 2, and Zoje using the same classifier. As shown in Figure g the estimated sketch Zoje be successfully recognized by using the classifier, i.c., achieving the same level top-1 accuracy of It-
14

---

Published as a conference paper at ICLR 2024
E
SUMMARIZED CAPTIONS
Calegory
Plompt
Catagery
Phonpt
Eategary
Prompt
an airplane winge
angel with pair wings
apple
an srpplen with a streamlindd faselage
an angel with a helo abovs his head
an apple with a steem
airpiane
alrplane flying upward an airplane with tall fin airplone with windows
angel
angel
apple
with
an angel with foatherilke wings
an apple with a small pit on the bettom
angel with
apple with two leaves
barn with peaked roof bam with door
bisket with handle
open book
plaki woven basket
closed baok
bam
a barn with double doors
basket
squara besikal
book
book with wavy lines
barn with arched door
a round bottomeddddddeeoooooooo basket
book with a spine
barn with windows
basket a handle
book with pattern
a bus with rectangular body
bu buttorffy with a pair of wings
cyllndricell cake
bus with windous
butterfly with alongated body
with candles
bus
bus with a row tires
buttorfly
buttarfly with antennae
cako
cake with two fiers
with doors
butterfly with honicantal lines its body is butterfly with spots on its wings
takk with wary lines the sides a cake with deconstions on it
bus with an enterns on the roof
rectanguler candle
with two fires a flat
roundi-laced call
cancle with fame
eyes
candle
candle with wax drops on the bottom
car
car with rectangular body
cat
cat with pointed ears
candle draps
side
with
with beard a full bodly cal
a candle with pottaemdd sideo
car with windows
four-lagged chair
fully cloud
a chair with an uprignt back
cloud with wary edges
a smmlling face
chai
shair with oval seat
cloud
dloud with flat bettom
face
face wearing glasses
a chiir with a rectingular seat
cloud
boy's face
chair with rails on the back
a mrarsshaalliw--Bee cloud
girts face with two piglaila
ractangular firaplace
fish with ovall ovall body
a sun with a larga dirtle in tha center
a fireplace with a pile of firewodd
a fish with a triangular tall
sun with dashes around
freplace
freplace with flames
fish eyes
sun
sun with wavvy
an arched fireplace
fish with mouth
sun surrounded by triangular rays
firaplace with chimmey
with fins
sun with smiling face
spldor with many logs
a rectangular or talevision
a bunch of round grapas
spider with an oval body
television with anterna
burch of closely packeed grapess
spider
a spider with segmented body
Ielevision
an television with base
grapes
bunch grapes with leawes
spider with track body
television with a row of buttons
bunch of grapes with glucose
spider with bent logs
tolevision with patterss on the screen
bunch grapes same sizo
llon's face
round pizza
a flat shoe
with thick
frfonguler pizza
round--tee shoe
ton
lion with syea
pirza
a gizza with outer edge
shoe
i boot
lion with eors a full-body lion
pizza round garnish
heal show
pizza with outting lines
shoe with laces
plump
stickman
round moon
a cylincrical vase
a person on a yoga mat person with arms ralsed person standing on one foot person lying the ground
sirll crescent moon
with flat batiom
yage
moon
wakiing moon A waning moon hall-cirdee moon morqutto with spices
a vase with an elangsted neck
vase with flawors
domed umbrella
round faced pig
an umbrella with a curved handle
a pig with eyes pia with oval nose a pig with cars full-body
moequito with wings
umoralla
umorolla with
mosquit
mosquito with slandor logs a mosquito with an alongated body
an umbrolla with wavy odges
an umbrollis with horizontal edges
moequito with longitudinl texture
All captions summarized from the randomly chosen 30 categories. Five captions are collected per category with template "this is a sketch of x" in our experiments, where X is either coarse or fine-grained text description, as shown in the table. The fine-grained and coarse captions are color-coded in red and blue, respectively.
Figure 9:
15

---

Published as a conference paper at ICLR 2024
MORE SAMPLES F
to to
3 / yto
H H
3o
No
4/ x
1 34
b +
* d # 1 to to
t
12
>e 0/ 16 1.
l 14 1o 1 , / 1*
x
86 14 1 1/
x
16
p 1
1
The
C
#
CH
C
03
04
[
34
K
*
che
De
+
-
02
C4
Figure 10: Samples from our model. Classes are yoga, kangaroo, house plant.
16

---

Published as a conference paper at ICLR 2024
Figure 11: Samples from our model. Classes are strawberry, face, car.
17

---

Published as a conference paper at ICLR 2024
Figure 12: Samples from our model. Classes are bird, basket, pineapple.
18

---

Published as a conference paper at ICLR 2024
:
Figure 13: Samples from our model. Classes owl, rain, whale.
19

---

Published as a conference paper at ICLR 2024
G EFFECT OF BATCH STZE
To inspect the robustness of our model under various choices of batchsize, and how the batchsize impact the results, set different batchsize generate 5k samples for evaluation. As shown in the [ the FID score remains relatively stable while the recall increases clearly when the batchsize becomes smaller (but much more time-consuming). The trend of generation results can conform to the expectation, i.e., using smaller batchsize offers improved results.
When setting batchsize=l, other than the known fact that this choice can highly slow down the sam- pling process, we further found that it makes the sampling sometimes unstable due to gradient van- ishing during scale optimization, i.e., the scale gradient becomes zero (achieving local minimum). In contrast, our default setting, i.e., batchsize=128, offers alleviation to the sisue since larger batches provide more samples gradient computation thus slowing down the vanishing of gradients.
To sum up. batchsize=1 should be ideal for scale optimization theoretically, however it is sub- optimal due to issue of scale gradient vanishing in practice. As result, we altered to batch version scale for speeding up and stabilizing the generation.
Table 4: Comparison results when varying batchsize on 30 categories of QuickDraw datascts.
batchsize
FID
Prect
Reet
Speed(s)
3,34
0.67
0.42
274.57
16
3.13
0.68
0.40
17.62
64
3.11
0.68
0.38
3.97
128
3.08
0.68
0.35
1.90
H EXAMPLES AND THE RETRIEVED CAPTIONS USING CLIP-SCORE
this sketch of
sketch
face
a bird with tail
smiling face
with eyas
face wearing glasses
bird winge
boy's face with
flying
girts face with pigtalls
body
Figure 14: Examples of the generated sketches and the retrieved captions (Top-1) using CLIP-Score
20


Reviews:
[{"review_id": "71RO5p92c0", "reviewer": "Reviewer_st3k", "summary": "The paper presents a method for class-guided sketch synthesis. The base model is an unconditional DDIM model operating in pixel space. During testing, the method uses classifier guidance to generate class-guided sketches. Naively using the same scale for all the time steps often produces low-fidelity or over-sketching samples. To address this issue, the paper proposes the scaling indicator which is computed based on stroke complexity and recognizability. At each sampling step, the classifier guidance scale is adaptively optimized to match the residual sketch with the scaling indicator. All the experiments are done on the QuickDraw dataset.", "questions": "What's the effect of \\alpha and \\beta?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The results are good qualitatively.\n- The idea of adaptive scale optimization is interesting.\n- The paper reads well and is easy to follow.", "weaknesses": "The validation of the idea is lacking:\n- The comparison with classifier-free guidance is missing.\n- In Table 1, what's the classifier guidance scale of DDIM? To validate the idea, it will be good to sweep over all possible classifier guidance scales and show that the proposed method works better than any constant guidance scale.\n- Why not just replace the classifier score in Equation-1 with the scaling indicator? In this case, the residual sketch and scale optimization are not needed anymore.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IioQ1DVaTA", "reviewer": "Reviewer_bpbS", "summary": "This paper divides the diffusion denoising process into three phases. The first and the last phases are unconditional generation, the middle phase is conditioned on the classifier guidance. For the middle phase, this paper proposes an extension block for classifier-guided diffusion models to perform scale adaptive classifier-guided sampling on the task of pixel-level sketch generation, addressing the challenge of dynamically optimizing the guidance scale for classifier-guided diffusion. This method achieves a balance between \"recognizability\" and \"complexity\" in the generated sketches. Experiments are on the QuickDraw dataset.", "questions": "1. I wonder whether the SGD process employed to obtain the optimal value of guidance scale s at each timestep t by minimizing Lt(s) is convergent. There is no further exploration.\n\n2. Also about the SGD process. In equation 4,\"N is the number of sketches generated within a sampling batch\". But why should the sketches in the same batch share the same guidance scale $s$? I mean, $s$ should be independent for each sketch in one batch.\n\n3. About the optimization objective Lt(s), which is \"intuitive\" but not from mathematical proof. This optimization objective seems questionable. \n\n4. How to you determine the parameters α = 1.0, β = 0.2, and $\\gamma$ = 0.02?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The three-phase sampling strategy can maintain sketch diversity and quality.\n\n2. The idea of dynamically optimizing the guidance scale is reasonable and prospective.\n\n3. Overall, the presentation and writing are easy to follow, and the experiment is detailed.", "weaknesses": "1. The main weakness is that the proposed block is specialized for classifier-guided diffusion. However, classifier-guided diffusion models are not as popular and powerful as those multimodal diffusion models using cross-attention. This limits the impact and universality of the proposed block.\n\n2. The idea of scale adaptive classifier-guided sampling is sound and this method achieves a balance between \"recognizability\" and \"complexity\". However, this also fixes the \"recognizability\" and \"complexity\" of the results. I mean, in classifier-guided diffusion models, users can adjust the guidance scale to trade off diversity for fidelity. But the proposed adaptive method can not.\n\n3. The implementation details are naive such as the complexity c(x0|t) using $L_0$ norm, and the determination equotion for $t_w$ and $t_d$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GuYu77CW60", "reviewer": "Reviewer_t4As", "summary": "This paper proposes a novel sketch generation method based on classifier-guided diffusion models. Specifically, the authors propose a scale-adaptive classifier-guided diffusion model, which achieves a delicate balance between recognizability and complexity in generated sketches. In addition, the authors also propose a three-phase sampling strategy to enhance sketch diversity and quality.", "questions": "The default size of the produced sketches is set to 64×64. Can this method still generate sketches well at higher resolutions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The authors analyze the impact of guidance scale on diffusion-based sketch generation tasks and propose a scale adaptive classifier-guided sampling method to achieve a delicate balance between recognizability and complexity in generated sketches.\n+ The authors point out the impact of unconditional guidance and classifier guidance on generating sketches in diffusion models and propose a three-phase sampling strategy.\n+ Quantitative and qualitative experiments have shown that the proposed method outperforms existing sketch generation methods.", "weaknesses": "+ From Figure 2 and Section 4.1, we can see that the input of the sketch classifier is a clean sketch estimated from noisy images. However, the author mentioned in Section 4.1 that the classifier is trained by using noisy sketches, which is obviously contradictory.\n\n+ From Table 2, we can see that using unconditional guidance can increase the diversity of generated sketches. However, from Figure 5, it can be seen that under the same random seeds, the images generated by different categories during the warm-up sampling stage are the same, and some strokes will remain in the final generation result. Therefore, it remains to be discussed whether the warm-up sampling is truly effective for sketch generation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GJcFhE7KNw", "reviewer": "Reviewer_SPRZ", "summary": "The authors proposed a strategy to dynamically change the scale of classifier-guidance during the reverse diffusion process for the generation of pixel-based sketches. For this purpose, they empirically designed a scaling indicator and a residual sketch to achieve the optimal guidance scale needed at the current diffusion stage. The scaling indicator is to assess the recognizability and complexity of current generative results. The residual sketch x_rs (x_t, s) is to evaluate the extent to which the guidance scale s will impact the generative process. The guidance scale s is optimized to enforce the residual sketch to be synchronized with the scale indicator. Moreover, they adopted a three-phase sampling strategy to enhance the sketch diversity and quality. Experiments demonstrated the superiority of their approach in the realm of sketch generation.", "questions": "1. The authors mentioned that vector-base approaches are inherently limited when tackling intricate and complex sketches. I would like to understand why pixel-based methods have an advantage over vector-based methods? I hope the authors can provide some intuitive explanations.\n2. In Section 2, the authors claimed that an additional property of pixel-based diffusion modeling is classifier gradient can be used as guidance. So are there any obstacles to using classifier-guidance for vector-based diffusion? It also seems feasible in principle.\n3. In Equ 2, are the settings of the three hyperparameters obtained through experiments or are there some empirical principles?\n4. In Equ. 4, why do different images in the same batch share the same scale?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposal of scaling indicator and residual sketch is novel and can conduct an effective scale adaptive classifier-guided diffusion process.\n2. The experimental part is convincing, and includes both quantitative and qualitative results. And the ablation study shows the effectiveness of their methods.\n3. The organization of this paper is clear and easy to understand.", "weaknesses": "1. The scale adaptive classifier-guided diffusion process seems useful and attractive, however, the scaling indicator and residual sketch are proposed mainly based on their experience in the field of sketch generation. In my opinion, it may rely on the simple structure of sketch. I wonder if similar methods can be applied to other data modalities, such as natural images.\n2. Equation 4 appears to be dubious. I understand the point that the residual sketch should be synchronized with the scale indicator. However, are they on the same numerical scale and is it appropriate to directly apply mean squared error for optimization? I believe that a more rigorous discussion or proof is needed based on their definitions (Formula 2 and 3).\n3. The contribution “three-phase sampling strategy” has been proven to be effective but it is merely a simple technique for diffusion process and the innovation may be limited.\n4. Using the fraction of stroke pixels to the whole canvas to evaluate the sketch complexity is oversimplified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)