DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models
Authors: XiMing Xing
Chuang Wang
Haitao Zhou
Jing Zhang
Qian Yu
Dong Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Sketch; Vector Sketch; Sketch Generation; Diffusion Models
TL;DR: In this paper, we present DiffSketcher, an innovative algorithm that creates \textit{vectorized} free-hand sketches using natural language input.

Abstract: Even though trained mainly on images, we discover that pretrained diffusion models show impressive power in guiding sketch synthesis. In this paper, we present DiffSketcher, an innovative algorithm that creates \textit{vectorized} free-hand sketches using natural language input. DiffSketcher is developed based on a pre-trained text-to-image diffusion model. It performs the task by directly optimizing a set of Bézier curves with an extended version of the score distillation sampling (SDS) loss, which allows us to use a raster-level diffusion model as a prior for optimizing a parametric vectorized sketch generator. Furthermore, we explore attention maps embedded in the diffusion model for effective stroke initialization to speed up the generation process. The generated sketches demonstrate multiple levels of abstraction while maintaining recognizability, underlying structure, and essential visual details of the subject drawn. Our experiments show that DiffSketcher achieves greater quality than prior work. The code and demo of DiffSketcher can be found at https://ximinng.github.io/DiffSketcher-project/.

DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models
Ximing Xing Beihang University ximingxing@buaa. edu. en
Chuang Wang Beihang University chuangvangObuaa. edu. en
Haitao Zhou Beihang University zhouhaitao@buaa. edu cn
Jing Zhang
Qian Yu
Dong Xu
Beihang University zhang-j jingbuua edu. en
Beihang University qianyu@buaa edu. cn
The University of Hong Kong dongxuBcs hku
Abstract
Concrete
deteleed
sustinng
"Coleris
photo
akensching with neightarhood"
painting bolding lightsbeer
like"
100
150
200
250
300
100
150
200
250
300
"Macaw full color, ultra detailed, realistic, insanely beautiful""; blue number indicates the number of irerations
Visualizations of the vector sketches generated by our proposed method, DiffSketcher. Bottom: Visualizations of the drawing process. For each example, we show two sketches with different number of strokes.
Figure 1: Top:
Abstract
Even though trained mainly on images, we discover that pretrained diffusion models show impressive power in guiding sketch synthesis. In this paper, present DiffSketcher, an innovative algorithm that creates vectorized free-hand sketches using natural language input. DiffSketcher is developed based on pre-trained text-to-image diffusion model. It performs the task by directly optimizing a set of Bézier curves with an extended version of the score distillation sampling (SDS) loss, which allows us to use raster-level diffusion model as a prior for optimizing parametric vectorized sketch generator, Furthermore, we explore attention maps embedded in the diffusion model for effective stroke initialization to speed up
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the generation process. The generated sketches demonstrate multiple levels of abstraction while maintaining recognizability, underlying structure, and essential visual details of the subject drawn. Our experiments show that DiftSketcher achieves greater quality than prior work. The code and demo of DiffSketcher can be found at https://ininngggiitbbbd///fffff
1 Introduction
Minimal representations, such as sketches and natural language. are powerful tools for effectively conveying ideas by emphasizing the subject's 's essence. As the carriers of abstract concepts. natural language conveys abstract semantic understanding, whereas sketches embody the human visual abstract presentation. A sketch can provide more visual details than natural language, making the concept more concrete. When a designer discusses the design plan with a client, the designer may sketch a prototype based on the client's description to ensure a full understanding of his/her requirements. If the process of text-to-sketed could be learned automatically, it would significantly lessen associated labor costs.
Unfortunately, the task of text-to-sketch generation remains unexplored. Some studies conducted on generating saaae--onditioned sketches [3. 18, 45, 44]. One such method is Info-drawing [3], which treats a sketch as a photo and uses a generative adversarial network (GAN) to generate a raster sketch based on an input image. It also introduces a style loss to ensure the generated sketches have similar style to the reference sketch used in training. CLIPasso [45] presents a novel pipeline to produce vector sketches through differentiable rasterizer optimized by semantic loss. CLIPasso can only handle the images with single object, so its follow-up work CLIPascene [44] extends to scene-level images. However, all of these methods depend on input images and do not facilitate the generation of sketches directly from text inputs. Moreover, despite their ability to generate realistic sketches from photos, these methods have a limitation in that they cannot generate new content.
Recent breakthroughs in text-to-iman image generation have been driven by diffusion models [23, 28, 30, 31] trained on billions of image-text pairs [34]. These models, conditioned on text, now support high- fidelity, diverse, and controllable image synthesis. Nonetheless, the present text-to-image diffusion models cannot produce highly abstract and vectorized free-hand sketches (as shown in Fig. 2). We draw inspiration from the highly effective text-to-image diffusion models and condiicod sketch generators build a bridge between two fundamental forms of human expression, namely text and free-hand sketches, resulting in the development of our proposed text-to-sketch generator.
In this work, we present DiftSSkttchrr an algorithm that can synthesize novel and high-quality free-hand vector sketches based on natural language text input. DiffSketcher does not require any text-to-sketch training pairs or large datasets of sketches. Instead, with the guidance of a pretrained diffusion model [30], we define sketch as a set of Bézier curves and optimize the curves' parameters through differentiable rasterizer [16]. The key idea behind DiffSketcher to transfer the prior knowledge from text-to-image generation model into the differentiable rasterizer. This allows the optimization of text-guided sketches with semantic consistency, ensuring that the final output sketches are coherent and aligned with their corresponding text inputs. However, it is a non-trivial task to fully leverage the pre-trained diffusion model to efficiently generate -ighquuaitty free-hand sketches for both simple objects and complex scenes.
To this end, we propose three strategies to improve the generation quality and efficiency: (1) Present an extended Score Distillation Sampling (SDS) Loss to guide the optimization of curve parameters. Previous works optimize the parameters of the vector sketch with CLIP loss. We found that an extended version of SDS loss can give more diverse sketch synthesis results, and it can be combined with CLIP loss or LPIPS loss, providing an additional control term. (2) Explore attention maps embedded in the diffusion model for effective stroke initialization. The synthesis process can be very consumnnn if the starting point of each stroke is randomly initialized. Therefore, we explore different initialization strategies and present to initialize the curve control points with a fused product of cross-attention maps and self-attention maps in the U-Net [5] of the diffusion model, which significantly improves the efficiency compared to the random initialization.
(3) Introduce opacity property during the optimization of Bézier curves to better mimic the style of human sketches, achieving the effect of heavy and light brushstrokes.

---

"A dragon flying in the sky, full body"
unicorn running on the grassland"
"There a mug on the plate"
"Photorealistic painting of a fruit basket with fruit"
A
"A detailed portrait Eiffel Tower, incredibly detailed and realistic, 8k, sharp focus"
"Produce an image Batman in battle scene, with smoke and explosions in the background"
"Real photo of sydney opera house
"Electronic board style buildings at New York city
silhouette"
Figure Various free-hand sketches synthesized by DiffSketcher and the corresponding description prompts. DiffSketcher obtains prior information from LDM [30] composite images through score distillation [24] and achieves the same heavy and light drawing styles as human sketches by perform- set of Bézier curves with the opacity property. Our proposed DiffSketcher allows for varying levels of abstraction while matching its corresponding textual semantics. In each example, given the same text prompt and two different random seeds, two sketches with different number of strokes are generated. The red words represent the cross-attention index used to initialize the control points (details about cross-attention
ing gradient descent
covered in Section 4.2).
We propose text-to-sketch diffusion model dubbed DiftSketcher. To the best of our knowledge, is the first diffusion model to generate diverse and high-quality vector sketches with different levels of abstraction the object and scene levels. (2) We present three strategies to improve the generation quality and efficiency, including an extended SDS loss and an aitention-enhaneed stroke initialization strategy. The opacity property of strokes is considered during synthesis to improve the visual effect further. (3) We conduct extensive experiments, and the experimental results show that our model outperforms other models in sketch quality and diversity. Thorough insights are provided for future research.
In summary contributions three-fold: (1)
Related Work
Sketch Synthesis. Free-hand drawings convey abstract concepts through human visual perception with minimal abstraction. Unlike purely edge-map extraction methods [2]. free-hand sketching aims to present sketches that are abstract in terms of structure [3] and semantic interpretation[44.
Therefore, computational sketching methods that aim mimic human drawing consider wide range of sketch representations, ranging from those grounded in the edge map of the input image[46, 17, 15, 42, 18, 4] to those that are more abstract [9, 8, 1, 26, 45], which are normally in vector format. Among the works synthesizing vector sketches, CLIPasso [45] and CLIPascene [44] are conditioned on an input image, while the rest unconditional. Until now, prior work has explored synthesizing sketch based on text.
Vector Graphics. Our work builds upon the differentiable renderer for vector graphics introduced by Li et [16]. While image generation methods that operate over vector images traditionally require a vector-based dataset, recent work has shown how the differentiable renderer can be used to bypass this limitation [36, 14, 41, 29, 19]. Furthermore, recent advances in visual text embedding contrastive language-image pre-training (2LIP)[27] have enabled number of successful methods for synthesizing sketches, such as CLIPDraw [7], StyleCLIPDraw [32], CLIP-CLOP [21], and CliPascene [44]. A very recent work VectorFusion [13] combine differentiable renderer with diffusion

---

model for vector graphies generation, e.g., iconography and pixel art. Our proposed algorithm, DiffSketcher, shares a similar idea with VectorFusion, but our focus is generating object- and scene- level sketches from a natural language prompt.
Diffusion Models. Denoising diffusion probabilistic models (DDPMM) [37, 39, 11, 40), particularly those conditioned on text, have shown promising results in text-to-image synthesis. For example, Classifier-Free Guidance (CFG) [12] has improved sample quality and has been widely used in large-scale diffusion model frameworks, including GLIDE [23], Stable Diffusion [30], DALL-E [28], and Imagen [31]. However, the majority of images available in web-scale datasets are rasterized, and this work follows the framework of synthesis through optimization, in which images are generated through valuation-time optimization against a given metric. Our proposed algorithm. DiffSketcher, uses pre-trained text-to-image diffusion model to synthesize free-hand sketches from natural language input. This is achieved by transferring image synthesis prior information into differentiable renderer.xc
3 Preliminaries
3.1 Diffusion Models
In this section, we provide concise overview of diffusion models, which are a class of generative models that utilize latent variables to gradually transform a sample from a noise distribution to a target data distribution [37, 11]. Diffusion models consist of two components: forward process g and a reverse process or generative model p. The forward process, which is typically modeled as a Gaussian distribution, gradually removes structure from the input data æ by adding noise over time. The reverse process, on the other hand, adds structure to the noise starting from a latent variable 2t- Specifically, the generative model is trained to slowly add structure starting from random noise p(zr)= = (0, I) with transitions Po ( (1) (( (z) = Naaa of I). Training the diffusion model with a (weighted) evidence lower bound (ELBO) simplifies to weighted denoising score matching objective for parameters & [11]
Coit (o. x²) (o0. (u()) ||e (a.z o,e;t) - el
(1)
where we(t) is weighting function that depends on the timestep I. Our work builds on text-to-image
latent diffusion model (LDM) that learn es E2(2;; t, y) conditioned on text embeddings y [30]. LDM uses classificr-free guidance(CFG) [12], which jointly learns an unconditional model enable higher quality generation via guidance scale parameter w & y, t) = + (1+W))&( (Z;; y, t) we (Z;; t)
(&g denotes the guided version of the noise prediction). CFG alters the score function to prefer regions where the ratio of the conditional density to the unconditional density is large. In practice, setting w > 0 improves sample fidelity at the cost of diversity.
3.2 Score Distillation Sampling
Many 3D generative approaches use a frozen image-text joint embedding model (e.g. CLIP) and an optimization-base approach to train Neural Radiance Fields (NeRF) [20]. Such models can be specified as a differentiable image parameterization (DIP) [22], where a differentiable generator transforms parameters 0 create an image æ g(0). In DreamFusion [24], be parameters of 3D volume and g is a volumetric renderer. To learn these parameters, DreamFusion proposed score distillation sampling (SDS) loss that can be applied to Imagen [31]:
0x
V VoLSDS(O, -00)) A Er,e (())(E
00
where the constant I = 0zz/0zz is absorbed into w(t), and the classificr- free-guidedddddde ég is used. In practice, SDS gives access to loss gradients, not scalar loss. Their proposed SDS loss provides a way to assess the similarity between an image and caption:
VoLsDs(ő, x = (()) V. [a, (L (q (z)lg(0): y. t) Ip (zzy, t)]
where Pa is the distribution learned by the frozen, pretrained Imagen model. y is unimodal Gaussian distribution centered at learned mean image g0). DreamFusion [24] proposed an approach to use pretrained pixel-space text-to-image diffusion model (Imagen [31] as a loss function. However,

---

diffusion models trained on pixels have traditionally been used to sample only pixels. We want to create what look like good sketches that match the text prompts when rendered from a set of vector strokes. Such models can be specified as a differentiable image parameterization, where differentiable rasterizer R. transforms parameters 0 to create a sketch S =R()).
Inspired by DreamFusion [24], we extend score distillation sampling (SDS) loss to use a pretrained latent diffusion model [30] as a prior for optimizing curve parameters. Intuitively, score distillation
converts diffusion sampling into an optimization problem that allows the raster image to be represented by a differentiable rasterizer.
3.3 Differentiable Rasterizer
Li et al. [16] introduce a differentiable rasterizer R that bridges the vector graphies and raster image domains. A raster image is a 2D grid sampling over the space of the vector graphics scene f(z, y; O), where O contains the curve parameters, e.g,, coordinates of Bézier control points, opacity, line thickness. Given a 2D location (ær, y) E IR? they first find all the filled curves and strokes that overlap with the location. Then sort them with a user-specified order and compute the color using alpha blending [25]. It enables the gradients of the rasterized curves (image) to be backpropagated to the curve parameters.
4 Methodology
In this section, we present our method for generating sketches using pre-trained text-to-image diffusion models. Let S be sketch that was rendered by differentiable rendering [16] R using the text prompt P. Our goal to optimize a set of parametric strokes to automatically generate a vector sketch that matches the description of the text prompt.
Our pipeline illustrated in Figure set of control points for the strokes, we synthesize the corresponding sketch S while matching the semantic attributes of the text prompt. To initialize the control points of the strokes, extract and fuse the attention map of the U-Net [5] used by the latent diffu- sion model [30]. Our key observation is that the structure and appearance of the gener- ated image depend on the interaction between the pixels and the text embedding through the diffusion process [10]. We provide more details on strokes initialization in Section 4.2. As shown in Figure 4, in each step of the op- timization, we feed the stroke parameters to differentiable rasterizer R to produce the raster sketch. We optimize over parameters such that S = R()) is close sample from the frozen latent diffusion model [30]. To perform this op- timization, we propose variation of the SDS loss function that enhances the texture of hand- sketched strokes and improves the resilience of drawing styles. This results in the raster sketch always being coherent with the text prompt. The resulting sketch, as well as the sample from the frozen latent diffusion model, then defines joint semantic and perceptual loss. We back- propagate the loss through the differentiable rasterizer R. and update the control points and opacity of strokes directly at each step until convergence of the loss function.
Given text prompt y of the desired subject and
Text Pronpt
phoro Biflid Towe
SVG Path S = (s).)))..
Difesestiabe Rastefizrr
Bedigrapagie orto paooeetis
Figure 3:
The overview of the pipeline. DiffSketcher accepts a set of control points (the locations of the strokes) and text prompts as input to generate hand- drawn sketch.
4.1 Synthesis Through Optimization
We define a sketch as set of n strokes (s1..... with the opacity attribute placed on white background. To represent each stroke, we use awo-dimensional Bézier curve with four control points 8; {p] 1 = ((x, ) (Notice that here, æ and y represent the coordinates in the canvas) and one opacity attribute o,, We incorporate the opacity of the strokes into the optimization process and use DiffSketcher semantics understanding to achieve a human-like heavy and light sketch style. The parameters of the strokes are fed to differentiable rasterizer R, which forms the raster
5

---

Latent diffusion model ø
Augmentation
"A photo Eiffel Tower"
Enooces
Raster Image
GySP
Chos Armiiun
Atcanii
Do
U.Net
Differentiable Rasterizer R(0)
Backpropagate Rasteriser Weights
Control Points
Béxier Curves
Fornar: SVG
S = 22 ((s,, 0t)
(Sa. 0w))
Figure 4: Optimization overview. To synthesize sketch that matches the given text prompt, we optimize the parameters of the differentiable rasterizer R. that produces the raster sketch S, such that the resulting sketch is close to a sample from the frozen latent diffusion model (the blue part of the picture). Since the diffusion model directly predicts the update direction, we do not need to backpropagate through the diffusion model; the model simply acts like an efficient, frozen critic that predicts image-space edits.
sketch S R((s1, 01) .... 0w)) = (1) (1p (pp))=1::0)) on)). For simplicity, we define the parameter in R O.
4.1.1 Vanilla Version: Fidelity to Generated Image
We start with a two-stage pipeline: First, we sample an image from the latent diffusion model [30] using text prompt. Next, optimize the control points obtain sketch that is consistent with the text prompt. To preserve the fidelity of the generated sample, we incorporate a Joint Visual Semantic and Perceptual (JVSP) loss optimize the similarity of the synthesized sketches and the instance sampled from the frozen latent diffusion model [30]. We leverage the VAE [6] decoder D to get (y: the RGB pixel representation of Sz,(zz ly: t). Then, we jointly use the LPIPS [47] and CLIP visual
encoders [27, 45] as depth structural similarity and visual semantic similarity metrics, respectively. Specifically, we use the following loss function:
LJVSP LuPrps(D((
Σ CLIP f))) CLIP (LIP)(
(4)
The LJvSP loss function encourages the synthesized sketch match the underlying semantics and perceptual details of the image sampled from LDM, leading to more realistic and visually appealing results. While vectorizing rasterized diffusion sample is lossy, an input Augmentation version of the SDS (ASDS) loss can either finetune the results or optimize random control points from scratch to sample sketch that is consistent with the text prompt. In the Sec 1.2, we introduce the ASDS loss to match the text prompt.
4.1.2 Augmentation SDS Loss: Fidelity to Text Prompt
To synthesize a vector sketch that matches given text prompt, we directly optimize the parameters of the differentiable rasterizer R that produces the raster sketch S. We propose an input augmentation version of the SDS loss function to perform this optimization, which encourages plausible images have low loss and implausible images have high loss. Given a raster sketch Š E # we combine RandomPerspeettive, RandomResizedCCop and RandomAdjjasthharppess to get data augmentation version of S_ E 1512x512x3, These transformations preserve artistic styles, enhance style diversity, emphasize crucial features, and improve the model's 's adaptability to various artistic renditions and size variations commonly found in black-and-white hand-drawn sketches. Then, the LDM uses VAE encoder [6] to encode S. into a latent representation E(S.), where (W/ and f is the encoder downsample factor. In summary, use the following

---

Allcation Map
Latent Diffusion Model
"A photo of Eiffel Tower"
Text Encoder
Phona
Tona
aakelee)
Self-Attention Map
Crosk Atteabbon
Attention
Attertionn
Self
Sell
U-Net
softmax prob
Bézier Curves
Token ét:: Atteation Fasion
Attention Map
Init Points
Init Curve
Figure 5: Strokes Initialization. The blue part of the figure represents the UNet in the LDM, which has two types of attention mechanisms: self-attention and cross-attention. The yellow and green parts respectively depict the visualization results of the cross- attentooon and self-attention. The gray part shows how the initial strokes generated using fused attention map. The dashed box represents the attention fusion, which is composed of the mean of the self-attention map and the cross-attention map corresponding to the 5-th text prompt token ("Tower")) We start 1-th token, because ()-th token taken up by the CLIP starting token.
ASDS loss function:
Oz as.
V S =((B) A E,,e,a
((t)(es(ZE))))) -)
(5)
as, 00
Where the weighting function w(t) is a very high and low noise levels due to numerical instabilities. Like DreamFusion [24], we set w 100 for classifier-fres guidance and higher guidance weights give improved sample quality. Intuitively, this loss perturbs S. with a random amount of noise corresponding to the timestep f, and estimates an update direction that follows the score function of the diffusion model to move to higher density region. The ASDS loss function encourages the synthesized sketch to match the given text prompt, while also preserving the style and structure of the original sketch. At each iteration, backpropagate the loss through the differentiable rasterizer and update the control points and opacity of strokes directly until convergence of the loss function.
And we sample ~ 4(0005, 0.95). avoiding
Loss objectives of our final model. To further enhance the quality of the synthesized sketches, we incorporate the JVSP and ASDS losses. Specifically, we first obtain the initial results with the JVSP loss, then we fine-tune the differentiable rasterizer together with the ASDS loss. We found such design can achieve the best performance. The ASDS loss predicts the gradient update direction directly, thus its loss value does not require weight balancing. of the LPIPS item to 0.2, and the weight of the CLIP visual item to 1. To compute the L2 distance between intermediate level activations of CLIP, we follow the CLIPasso [44] method and use layers 3 and 4 of the ResNer101 CLIP model. As shown in Figure 8, we compare the performance of the vanilla version with that of the version using only the ASDS loss, and the proposed final version. Our experiments show that the final version improves both the generation quality and efficiency.
the JVSP loss, we set the weight
4.2 Joint Attention-based Stroke Initialization
The highly non-convex nature of the ASDS loss function makes the optimization process susceptible to initialization, especially in multi-instance scenarios where strokes must be carefully placed to emphasize the overall semantics conveyed by free-hand sketching. We improve the convergence towards semantic depictions by initializing the curve control points based on the attention map of text prompt conditional latent diffusion model. The UNet in the LDM has two types of attention mechanisms (the blue part), self-attention and cross-attention, as shown in Figure 5 where the yellow part represents the cross-attention visualization result, and the green part represents the self-attention visualization result. The structure and appearance of the image generated by the LDM [30] depend on

---

the interaction between the pixels to the text embedding through the diffusion process [10], which is manifested in the cross-attention layer [43]. The visualization results indicate that the cross-attention layers control the relationship between the spatial layout of the image and each word in the prompt, while the self-attention layer affects the spatial layout and geometric structure of the generated image. With this observation, we linearly combine the probability distributions of the two attention maps to initialize the curve control points. Specifically, we pick a map in the cross-attention maps based on text token and combine it with the averaged self-attention map. This process is formalized as FinalAttn À * CrossAttn; + (1 - A) * (reen(SelfAttm). where is the control coefficient and i indicates the i-th token in the text prompt. Finally, normalize the fused attention map using softmax and use as a distribution map to sample n positions for the first control point pi of each Bézier curve. The other three control points (p: .P.Pp)) .p)) are sampled within small radius (0.05 of image size) around pł define the initial set of Bézier curves {(p } j==-==. Empirical results show that our attentional fusion- based initialization contributes significantly to the quality and rendering speed of the final sketch in comparison to the random initialization.
Results
LITMA's Reczh
Camna
CLIRoann
Dur
Ceeches EXT oemeet BT,
Ameteen Prodetzar cennet 200% CulP vead 2.1%
Arsterte Top Phedkctten:
*Sutiheen cilingts mudlide plaas paweegaphh"
Tep Treckcioue oboed 21.0%
cermet P1,
wesul: 3%
vosei: 0.4
resull 6:thh
Austlatic + Eikel oxmnnt 96 phu sheeriirr 10.%%
Austoaa 18 3136 atop 22p-2200 27.0%
Anatnnt -00807 cesca FLAr 1%%
millde aplilione right tocuad phot dleanee
CommEc 82.7% iam 1.1%
calke 1%s
beii A011 nemeet sd my cna 3%%%
Goatkee tisa cervect at Jome 15.7%
Gonteee comero Faelenkom cernect ahott lna 1..%%
Aurdetiic Preditteen commect 96 ian 0.0%
"Autsai of whar keenii sigll dtuila?
"Phesendiido pREEREAt wat Irai."
2117 Top Prodenton: axplca in
1554 Proikcim 14.1%
001e7 Tay Fremeesoe
Aurbetic 4..00 Top Prodeitenn appk 0L1S
Apple 0L1h
Figure 6: Comparison with existing methods, including edge extraction [2] and vector sketching [45, 44].
5.1 Qualitative Evaluation
As shown in Figure 2, we demonstrate that our approach offers the ability produce object-level and scene-level sketches based on a textual prompt, with the flexibility to manipulate the level of abstraction through the number of strokes. It effective in generating accurate sketches regardless of prompt complexity, including simple or non-existent objects, iconic constructions, and detailed scenes. The utilization of the robust prior of stable diffusion allows for a favorable initialization, promoting the production of high-quality sketches with significantly fewer iterations. Figure 6 qualitatively methods [2]. The Canny edge extraction algorithm extracts an excessive number of edges and produces untidy sketches, as observed in the first example in Fig. 6. CLIPasso uses visual distance metrics to guide gradee optimization. On the task of drawing scene-level sketches, CLIPasso can only draw part of the foreground, and the background part missing.
compared our DiffSketcher with those of CLIP-based [45] and edge extraction-
We also compare our work with the recent work VectorFusion [13]. The results are shown in Fig. 7. This work is highly relevant to ours as it also explores the potential of diffusion model in generating vector graphics. However, there are notable differences between our appraoch and VectorFusion in terms of task setting, model design, and performance. Firstly, our method, DiffSketcher, primarily focuses on generating vector sketches based on text input. In contrast, VectorFusion aims to generate a broader range of vector graphics, including iconography and pixel art. It is important to highlight that our method has the capability to easily extend its functionality to generate other types of vector graphics without changing primitives (as shown in Fig. 10 of Supp. B). Secondly, DiffSketcher follows distinct pipeline compared to VectorFusion. VectorFusion employs different model variants for different types of vector graphics. For generating vector sketches, our approach differs from VectorFusion mainly in two aspects: (I) Initialization: VectorFusion randomly initializes the
8

---

Text Prompt
VectorFusion
DiftSketcher
Text Prompt
VectorFusion
DiffSketcher
watercolor painting
realistic photograph cart
firebreathing dragonf
brightly colored mushroom gi growing logt
elephantf
Figure 7: Qualitative comparison with VectorFussion (VF) [13]. VF's results are based its original paper and project website, with a text prompt suffix of "minimal 2D line drawing trending on ArtStation". Note that our results were optimized from scratch using ASDS without specifically designed text prompt suffix.
rasterizer (i.e., the location of the control points), while our approach utilizes the attention layer of LDM, guided by text prompts. for initialization. This significantly improve both efficiency and synthesis quality. (2) Optimization: While VectorFusion solely optimizes the position of control points, our method also optimize the opacity of stroke. This contributes to enhance the visual quality of the synthesized sketches. As shown in Fig. 7, our approach significantly outperforms VectorFusion in generating vector sketches, and achieves comparable performance in generating other types of vector graphics (see Fig. 10 of Supp. B). worth noting that our model variant shown in Fig. utilizes only ASDS loss for optimization and initializes the strokes randomly. This was done to ensure a fair comparison with VectorFusion. Our full model expected to deliver better performance.
5.2 Quantitative Evaluation
Evaluating text-to-sketch synthesis is challenging due to the absence of ground truth sketches. Therefore, we focus on three indicators: consistency between the generated sketch and text prompt, the aesthetic quality of the sketch, and the recognition accuracy of the sketch. To measure the consistency between the generated sketch and input text, we calculate the mean of cosine similarity of CLIP embeddings for the generated sketches and the text captions used to generate them. Our method achieves a cosine similarity of 0.3494, which higher than the 0.328 achieved by Canny algorithm and the 0.3075 achieved by CLIPasso. Aesthetic appeal is subjective, and being visually appealing is a personal experience and preference. However, when describing the visual appeal of hand-drawe sketch, various factors can be considered, such line quality, texture, and style. As an evaluation of our proposed method, we use CL.IP-based aesthetic indicators [33] to calculate the aesthetic value for samples of multiple categories. Figure 6 compares aesthetic values achieved by various methods. Our method achieves a mean value of 4.8206 for the aesthetic value on a large number of examples, which is higher than the 4.3682 achieved by Canny and the 4.0821 achieved by CLIPasso [45].
5.3 Ablation Study
We conducted a series of experiments demonstrate the effectiveness of the proposed initialization strategy and the effects of ASDS loss and JV loss, respectively. The top row of Fig. 8 compares two initialization strategies used by DiffSketcher and CLIPasso [45], namely LDM Attention and CAM [35] Saliency map. respectively. The text-driven diffusion model produces more precise attention map than the saliency map obtained by the CLIP- method due to LDM's superior generation ability. The cross-attention feature of LDM can efficiently activate relevant regions based on token areas, while the self-attention layer can effectively differentiate foreground from background down to the pixel level. Through combining both mechanisms, the initialization area becomes more precise. As result, superior quality sketches are produced with a reduced number of optimization steps, as illustrated in the second and third rows of Fig. 8. The proposed initialization strategy improves sampling quality and efficiency, which is critical for non-convex objective function optimization.

---

LDM Attn-map
CLIP Saliency mup
LDM Result
Cmo--Att
Self-Arre
Fusion
Irit Points
Init Puints
CLiPasso +opacity +saliency
CLIPasso
+opacity
180
DiffSketcher
w/o ASDS
DiflSketcher
100
250
SDS Only +Random
ASDS Only +Random
ASDS Only
E00 toirs
500 iters
Figure 8: Qualitative results of ablation study. Top: two initialization strategies used by DiflSketcher (i.e., LDM Attention) and CLIPasso [45] (i.e. CAM [35] Saliency map). A result of sampling from LDM The 2nd and 3rd rows: Comparison of the two initialization strategies on the convergence speed. The 4th and 5th rows: The effect of JVSP (Section 4.1.1) and ASDS loss (Section 4.1. The 5th row shows the loss of SDS without data augmentation. The 6th and 7th rows: ASDS loss leads to more diverse results when strokes are randomly initialized. Text prompt used in this example: "Astronaut on Asteroid, galaxy background".
In DiffSketcher, our JVSP loss consists of CLIP loss and LPIPS loss, and it is important to note that ASDS and JVSP do not conflict with each other. As illustrated in Fig. 8, the 4th and 8th rows highlight the effects of JVSP loss and ASDS loss, respectively. When only JVSP loss is used (4th row), the generated sample closely approximates the result of LDM. On the other hand, when only ASDS loss is employed (8th row), the generated sample aligns with the text prompt semantically, but does not follow the attention map of the LDM. Additionally, using ASDS loss results in more diverse sampling outcomes during the synthesis process. For instance, the location of the astronaut's head may change throughout the drawing process when ASDS loss is used, while it remains in the same location when the JVSP loss is employed. We utilize both JVSP loss and ASDS loss for optimization. As shown in the 5th row (labeled as "DiffSketcher') )), the combination of these two losses leads to synthesized sketches with more intricate details and visually more realistic appearance.
Conclusion
In this work, we have proposed novel and effective approach to bridge the gap between natural language and free-hand sketches. By leveraging the power of pretrained text-to-image diffusion models, we have developed a method that can generate diverse and high-quality vector sketches based on textual input, without the need for large-scale datasets of sketches or sketch-text pairs. We have also explored different aspects of the model design, such as the stroke initialization strategies, choice loss functions, and properties valuable insights for future research. With further research and development, our proposed model, DiffSketcher, has the potential to be valuable tool in various domains, such as design and education. Limitation. Our approach has two limitations. Firstly, there is lack of correlation between the
strokes the differentiable rasterizer, which provide
10

---

text prompt and sketch abstractness, which may result in unsatisfactory sketches. To address this, we suggest establishing link between the complexity the text prompt and the number of strokes used. More details can be found in Supp. I. Secondly, the style of the generated sketches is limited. A possible solution is introducing style transfer module during the sketch synthesis process.
Acknowledgement
This work is supported by the National Natural Science Foundation of China (No. 62002012. No. 62006012, and No. 62132001) and CCF-Baidu Open Fund. is also supported in part by The Hong Kong Jockey Club Charities Trust under Grant 2022-0174, in part by the Hong Kong Research Grant Council under General Research Fund (17203023), in part by the Startup Funding and the Seed Funding for Basic Research for New Staff from The University of Hong Kong, and in part by the funding from UBTECH Robotics.

---

References
[1] Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Fahad Shah- baz Khan, Jorma Laaksonen, and Michael Felsberg. Doodleformer: Creative sketch drawing with transformers. In Proceedings of the European conference on computer vision (ECCV), pages 338-355, 2022.
[2] John Canny. A computational approach to edge detection. IEEE Transactions on Pultern Analysis and Machine Intelligence, PAMI-866;679-698, 1986.
[3] Caroline Chan, Frédo Durand, and Phillip Isola. Learning to generate line drawings that convey geometry and semantics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 7915-7925, June 2022.
[4] Shu- Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu. Deepfacedrawing: Deep generation of face images from sketches. In ACM Transactions on Graphics (TOG), volume 39, pages 72-1, ACM New York, NY, USA, 2020.
[5] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems (NIPS), volume 34, pages 8780-8799 8794, 2021.
[6] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for righuuesotution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition (NIPS), pages 12873-12883, 2021.
[7] Kevin Frans, Lisa Soros, and Olaf Witkowski. CLIPDraw: Exploring text-to text-to-drawing synthesis through aanguge-imaag encoders. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems (NIPS), 2022.
[8] songwei Ge, Vedanuj Goswami, Larry Zitnick, and Devi Parikh. Creative sketch generation. In International Conference on Learning Representations (ICLR), 2021.
[9] David Ha and Douglas Eck. A neural representation of sketch drawings. In International Conference on Learning Representations (ICLR), 2018.
[10] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or: Prompt-to-prompt image editing with cross-attention control. In The Eleventh International Conference Learning Representations (ICLR), 2023.
[11] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NIPS), volume 33, pages 6840-6851, 2020.
[12] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.122 12598, 2022.
[13] Ajay Jain, Amber Xie, and Picter Abbeel. Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[14] Dmytro Kotovenko, Matthias Wright, Arthur Heimbrecht, and Bjom Ommer. Rethinking style transfer: From pixels parameterized brushstrokes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12205, 2021.
[15] Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan. Photo-sketching: Inferring contour drawings from images. In 2019 IEEE Winter Conference Applications of Computer Vision (WACV), pages 1403-1412. IEEE, 2019.
[16] Tzu-Mao Li, Michal Lukáč, Gharbi Michaël, and Jonathan Bagan-Kelleyy Differentiable vector graphics rasterization for editing and learning. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 19661:99:11113:15 2020.
[17) Yi Li, Yi-Zhe Song, Timothy M Hospedales, and Shaogang Gong. Free-hand sketch synthesis with deformable stroke models. International Journal of Computer Vision, 122:169-196 190, 2017.
12

---

[18] Runtao Liu, Qian Yu, and Stella X Yu. Unsupervised sketch to photo synthesis. In Com- 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt III 16, pages 36-52. Springer, 2020.
puter
[19] Xu Ma, Yuqian Zhou, Xingqian Xu, Bin Sun, Valerii Filev, Nikita Orlov, Yun Fu, and Humphrey Shi. Towards layer-wise image vectorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 16314-16323, 2022.
[20] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[21] Piotr Mirowski, Dylan Banarse, Mateusz Malinowski, Simon Osindero, and Chrisantha Fer- nando. Clip-clop: Clip-guided collage and photomontage. arXiv preprint arXiv:2205.0314p 2022.
[22] Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah. Differentiable catafferretiabe--paaameeerriiiii
image parameterizations. Distill, 2018.
[23] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 16784-16804, 17-23 Jul 2022.
[24] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[25] Thomas Porter and Tom Duff. Compositing digital images. In Proceedings of Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '84, page 253-259, 1984.
[26] Yonggang Qi, Guoyao Su, Pinaki Nath Chowdhury, Mingkang Li, and Yi-Zhe Song. Sketchlat- tice: Latticed representation for sketch manipulation. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 933-941, 2021.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. 8763. PMLR, 2021.
[28] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint urXiv:2204.06/25 2022.
[29] Pradyumna Reddy, Michael Gharbi, Michal Lukac, and Niloy Mitra. Im2vee: Synthesizing vector graphics without vector supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7342-7351, 2021
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022.
[31] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems (NIPS), volume 35, pages 36479-36494, 2022.
[32] Peter Schaldenbrand, Zhixuan Liu, and Jean Oh. Styleclipdraw: Coupling content and style in text-to-drawing synthesis. arXiv preprint arXiv:2111.03133,00 2021.
[33] Christoph Schuhmann. Improved aesthetic predictor.
https://gitthb. com/
aatpaaaammmmmpeeedaaeeeeeeeeeeee prreetttiicpedeitoorroooo 2022.
13

---

[34] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:22 arXiv:2210.084000
10. 08402, 2022.
[35] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision (ICCV), pages 618-626, 2017.
[36] I-Chao Shen and Bing-Yu Chen. Clipgen: A deep generative model for clipart vectorization and synthesis. IEEE Transactions on Visualization and Computer Graphics, 28 28(12):4211-44 1-2224, 2021.
[37] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Proceedings of the International Conference on Muchine Learning (ICML), volume 37, pages 2256-2265, 2015.
[38] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations (ICLR), 2021.
[39] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NIPS), volume 32, 2019.
[40] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
[41] Yingtao Tian and David Ha. Modern evolution strategies for creativity: Fitting concrete images and abstract concepts. In Artificial Intelligence in Music, Sound, Art and Design, pages 275-291. Springer, 2022.
[42] Zhengyan Tong, Xuanhong Chen, Bingbing Ni, and Xiaohang Wang. Sketch generation with drawing process guided by vector flow and grayscale. In Proceedings of the Conference on Artificial Intelligence (AAAI), volume 35, pages 609-616, 2021.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems (NIPS), 30, 2017.
[44] Yael Vinker, Yuval Alaluf, Daniel Cohen- -r., and Ariel Shamir. Clipascene: Scene sketching with different types and levels of abstraction. arXiv preprint arXiv:2211.17256, 2022.
[45] Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-awar object sketching. ACM Transactions on Graphics (TOG), 11(4):1-11, 2022.
[46] Saining Xie and Zhuowen Tu. Holistically edge detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1395-1403, 2015.
[47] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features a perceptual metric. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
14

---

Supplementary
Overview
This supplementary material is organized into several sections that provide additional details and analysis related work DiffSketcher. Specifically, it will cover the following topies:
In section A, we provide the implementation details of DiffSketcher.
In section B, we present a qualitative comparison of our DiftSketcher with another two text-to-VVG methods, CLIPPraw [7] and VectorFusion [13]. We compare results generated by these methods and analyze the differences in terms of visual quality and semantic consistency.
In section C, we compare sketches generated by our DiffSketcher with those directly sampled from the LDM (i.e., Stable Diffusion [30]) and analyze the differences in their style.
In section D, conducted perceptual study assess the authenticity of the synthesized sketches.
In section E, we compare the results of three different strategies for stroke initialization.
In section F, visualize how DiffSketcher gradually sketches object scene.
In section G, we provide example sketches with different stroke widths.
In section H, we introduce the details of the evaluation metrics used in our experiments.
In section we show several examples of failure cases.
A Implementation Details of DiffSketcher.
We begin by describing the vanilla version of our approach (Section 4.1.1), which involves sampling an image from the latent diffusion model [30] and then automatically sketching it using DiffSketcher. Specifically, given text prompt, we use DDIM solver [38] to sample a raster image from the latent diffusion model in 100 steps with classifier-free guidance [12], using scale of
7.5.
To apply the Augmentation SDS loss (Section 4.1. we sample a noise level / from the uniform distribution (/(005, 0.95). avoiding very high and low values that can cause numerical instabilities. For classifier-fres free guidance, we set w = 100, as we found that higher guidance weight leads to better sample quality. This larger than the scale used in image sampling methods, which likely due to the mode-seeking nature of our objective, leading over-smoothing small guidance weights.
In DiflSketcher, the number of strokes n is defined by the user, and we use 4 duplicates for image augmentation to maintain recognizability under various distortions. For this purpose, RandomPerspectivve, trreh. hettrrrransfossRnnannnkeeseeeed RandomResizedCCrp. and functions in sequence. It is worth noting that data augmen- tation is not the focus of our work, and experimental results show that the choice of augmentation strategies does not affect results significantly.
we apply the torch. torch.transfon tornh.transforms. Rawtasfooms..anndm...o..o
In Section 4, we define a sketch as of strokes (.1.-..,8 8a with the opacity attribute placed on a white background. To optimize the control points and opacity, we use two Adam optimizers. Specifically, we the learning rate of the control point optimizer to 1.0 and the color optimizer to 0.1.
B Comparison to Existing Text-to-SVG Work.
This section presents a qualitative comparison between DiffSketcher and CLIPDraw [7]. CLIPDraw is a based method introduced for text-to-SVG generation. It gradually optimizes the position and colors of the curves by the gradient descents computed by comparing the cosine similarity of the text prompt and the generated drawings. Our method differs from CLIPDraw in two ways, one is the stroke initialization, and more importantly, ours is equipped with the Augmentation SDS (ASDS) loss. As illustrated in Fig. 9, CLIPDraw struggles to synthesize a meaningful and visually pleasing drawing, no matter whether with colors or not. It can be explained by the fact that the CLIP model is not generative model, and it can only provide guidance from highly-semante perspective. In
15

---

Crw
DiffSketcher
Crw
DiftSketcher
RGBA
B&W
RGBA
B&W
"Electronic board style buildings new york city silhouette"
"Astronaut asteroid, galaxy background"
"A photo of the Mr. Bean"
"A photo of Taj Mahal, extremely detailed"
Figure 9: Comparison of the results synthesized by CLIPDraw and DiffSketcher. Specifically, for each example, we compare the results generated by CLIPDraw (Left) and our DiffSketcher (Right) given the same text prompt. We implemented two versions of CLIPDraw: the RGBA version (original version) and the B&W version. The B&W W version forces the stroke color to black to mimic the sketch style. Our results visually more pleasing and meaningful.
Text Prompt
VectorFusion
DiffSketcher
bananas*
the silhouette elephant*
Figure 10: Qualitative comparison with VectorFusion [13]. VF's results were copied from Figure 2 of its original paper, with text prompt suffix of "minimal 2D line drawing trending on
ArtStation". In contrast, our results were optimized from scratch using ASDS without specifically designed text prompt suffix.
contrast, our DiffSketcher can generate sketches that are semantically consistent with the input text, and exhibit high aesthetic quality. This is because the proposed ASDS loss can distill the drawing capability of the latent diffusion model (LDM) into the differentiable rasterizer. These results also suggest the effectiveness of the ASDS loss and the benefits of leveraging the power of the LDM. VectorFusion [13] highly relevant to our work
aims to produce general vector graphies, such iconography. Although our proposed method is designed to generate vector sketches, it can easily extend to generate other types of vector graphics. Fig. 10, we compare results obtained by VectorFusio and our DiffSketcher.
16

---

LDM
Ours
free hand sketch of
astionaut on
Asteroid"
"a free hand
skotch
white bunnies"
frnd sydney opera house"
bee-band sketch of a fruit baskett with
Figure 11: Comparison of sketches generated by sampling from the LDM using the specified text prompt (Left) and ours (Right).
LDM's Results
Output Sketches
"a hand sketch of pure young girl"
"closeup portrait of a pure young girl"
" "closeup portrait of pure young girl by Malcolm T. Liepke, oil paint"
Figure 12: The style of the generated sketches is significantly affected by the keywords used in the text prompt.
C Comparison to Existing Text-to-Image Work.
In this section, we compare the sketches directly generated by LDM using specific text prompts. To encourage the results to be abstract and follow the free-hand sketch style, we append a suffix to the text prompt: "A free-hand sketch of xxx white background, trending ArtStation. Keep abstract." " This prompt was tuned qualitatively to capture the desired style and artistic expression.
The results are shown in Fig. 11. It is clear that the LDM is capable of generating high- raster sketches free-hand sketch style. However, different from the vector sketches generated by our DiffSketcher, LDM's 's sketches have two distinguished characteristics: 1. they are more delicate and like professional sketches. 2. their background exhibits the paper texture. We suppose this because most sketches used for training LDM are photographs of professional sketches drawn paper. By contrast, the sketches synthesized by our DiffSketcher are more like the style of free-hand sketch and exhibit different levels of abstractness.
17

---

Real free hand sketch
VectorFusion
CLiPasso
Ours
Figure 13: Partial sample visualization for conducting user research. The hand-drawn sketches were sourced from the Google. CIIPPsso''s s and our results were sourced from respective paper, VectorFusion's ' results were soureed from their project homepage.
is worth noting that our method does not require indicating "free-hand sketch" in the text prompt. The drawing engine of our model is the rasterizer and it can naturally capture the sketch style. We also conduct experiments to show the results of using different keywords text prompts, such as "sketch", "drawing", and "photo". The results are shown in Fig. 12. We can see that although the style the output images is different, the style the generated sketches is not significantly affected by the keywords.
D User Study.
Table 1: Results of the User Study. The Confusion score of real sketch means only 67% % real sketches recognized real.
Metric/ / Method
CLIPasso [45]
VectorFusion [13]
DiffSketcher (Ours)
Human Sketch
Confusion Score
0.39
0.33
0.65
0.67
To assess the authenticity of the synthesized sketches, we conducted a perceptual study. Specifically, We gathered total of 90 synthesized sketches using three different methods (30 samples per method) and obtained 30 real sketches from Google Image by searching for "free-hand sketch". Figure 13 shows partial sample. We then mixed the real and fake sketches and distributed questionaires to 41 participants. The participants were asked to determine whether each sketch was drawn by a human or not, without any knowledge of its source. We utilized the confusion score the evaluation metric, where higher score indicates a greater likelihood of the generated sketches being recognized as real. The results are presented in Table 1. It is clear to see that our method produced sketches that were more frequently identified as real, highlighting the superior quality our synthesized sketches.
Stroke Initialization.
The highly non-convex nature of the ASDS loss and JVSP loss function makes the optimization process susceptible to stroke initialization, especially for generating scene-level sketches with multi- instances. To address this issue, explore different stroke initialization strategies (as mentioned in Section 5.3 and Fig. 8) and evaluate their impact on the performance of our model.
As shown in Figure. 14, we compare three different stroke initialization methods: random initializa- tion, initialization based on the CLIP saliency map [45], and initialization based on our proposed fusion attention (ours). The experimental results demonstrate that our proposed strategy, initialization based on fusion attention, outperforms the other methods in terms of visual quality and synthesis efficiency. This initialization method can utilize the joint semantic and structural information from the
18

---

0 iter
10 iters
20 iters
50 iters
100 iters
200 iters
300 iters
Random
saliency
Ours
"Colorful hot air balloons high over the mountains"
Figure 14:
Comparison of the (intermediate) results when using different stroke initialization strategies. From top to bottom: (a) random initialization, (b) initialization based on the CLIP saliency map, (c) our proposed fusion attention.
input text and image (i.e., LDM results) to guide the stroke placement, resulting in more semantically meaningful and artistically expressive sketches. However, using a CLIP saliency map for initialization leads to sketch with only salient objects while ignoring the background. Random initialization takes more iterations than ours to synthesize visually pleasing results.
F Visualization of Sketching Process.
In this section, we show the trace of 300 iters of sketching. By visualizing the intermediate out- puts during the generation process, can gain insights into how model sketches object. Specifically, as shown in Fig. 15, we can observe how the strokes are placed and refined over time to gradually form the desired sketch.
Input:
O
10 iters
20 iters
iters
iters
Macaw full color, detailed, realistic, issancly beautiful
300 iters
250 iters
200 iters
150 iters
100 iters
80 iters
Figure 15: The intermediate results throughout the optimization process.
G Effect of the Stroke Width.
In Fig. 16, we compare the results with different stroke widths given the same text prompt. In our implementation, we use a fixed stroke width for all strokes. Such design is to simplify the optimization, making computationally more efficient and less prone to overfitting. can easily extend to include stroke width parameter for optimization, like [2].
19

---

width = 1.5
width
width = 2.5
width =
"Sailboat sailing sea a clear day, realistic photo, professional phorography"
"Very detailed masterpiece painting of baby yoda holding lightsaber"
Figure 16: Different widths of the curves. The width increases from left to right.
Strokes
32
128
Two sparrows perched the branches
A banana the table, and there a glass of water beside it
Figure 17: The failure cases.
H
Evaluation Metrics.
Evaluating text-to-sketch synthesis is challenging due to the absence of ground truth sketches. As we mentioned in Section 5.2 and Fig. 6, we evaluate the models from three aspects: semantic consistency between the generated sketch and text prompt, the aesthetic quality of the sketch, and recognizabilityy of the sketch.
Semantic Consistency Between the Generated Sketch and Text Prompt. To measure the semantic consistency, namely the CLIP score [30, 7]. we calculate the cosine similarity of CLIP ViT-L-14 embeddings of the generated sketches and corresponding input text prompts. Our method achieves a cosine similarity of 0.3494, which is higher than Canny [2] algorithm (0.328) and CLIPasso [45] (0.3075).
The Aesthetic Quality of the Sketch. To measure the aesthetic quality of generated sketches, adopt the CLIP-based aesthetic indicator [33]. This indicator [33] consists of CLIP ViT-L-14 backbone and a multi-layer perception (MLP), which is pre-trained on LAION [34] data. Figure of several examples generated by different methods.
of the main paper compares the aesthetic Sketches generated by our method obtain the highest scores.
The Recognizability of the Sketch. Finally, to measure the recognizability of the generated sketches, we use the CLIP ViT-L-14 model [27] for zero-shot classification. Specifically, we first generate the CLIP model classify these sketches. In Fig. 6 of the main paper, we list the probabilities of the sketch being categorized into different classes. Since Canny edge preserve the object's contours and fine details, it achieves high recognizability. Compare our DiffSketcher with CLIPasso, our DiffSketcher can draw a more complete sketch. Therefore, our sketches easier for CLIP to recognize.
sketches for 34 categories Then,
"The categories include "astronaut", "vessel", "observatory". "needle". "outer space", "earth", "iron man". "batman"; "apple". "sailboat", "ship", "bunny", "castle", "cabin". "inn". "bike", "cat", "dog", "dragon", "snake", "horse", "fruit basket". "Sydney opera house". "lamp". "lighthouse", "mug". "desk", "macaw". "mountain". "river" "eiffel tower". "unicorn". "yoda", "skyscraper".
20

---

I Failure Cases
As shown Fig. 17, our approach mainly has two limitations. Specifically, one limitation is the lack of correlation between the text prompt and sketch abstractness, For instance, if the text prompt describes multiple objects but the number of strokes is set too small, the resulting sketches may be unsatisfactory. A possible solution is to estabilish a link between the complexity of the text prompt (such as the number of described objects) and the number of strokes to be used.
21


Reviews:
[{"review_id": "ECklMN8y59", "reviewer": "Reviewer_iPV2", "summary": "This paper proposes an algorithm to leverage pretrained image diffusion models for text-guided sketch synthesis. The sketch to be optimized is represented as a set of Bezier curves (and their opacities) and optimized with the (augmentation) score-distillation sampling (ASDS) loss. Attention maps from the diffusion model's UNet are fused and used to initialize the Bezier curves to accelerate convergence. A variety of results ranging from abstract to concrete sketch generation are demonstrated.", "questions": "Minor suggestions:\nLine 177: this is good place to define the JVSP abbreviation\nLine 183: the abbreviation ASDS is not defined\n", "limitations": "Limitations are not discussed, it would be good to include those in the revision along with some failure cases.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is easy to understand and makes a first-of-its-kind contribution: using pretrained image diffusion models to optimize vector graphics sketches. While it is thematically similar to DreamFusion and related works, the problem and solution are novel enough and interesting. The results are convincing are significantly better than prior work, and evaluation is comprehensive on a wide range of concepts. The ASDS loss is interesting although a similar augmentation strategy was also used in CLIPDraw. The stroke initialization strategy by analyzing the attention maps is an excellent idea and could potentially be helpful in other future works that have to deal with domain gaps and difficult optimization landscapes. Ablation studies are quite comprehensive. Overall, the paper is technically strong and demonstrates excellent results.", "weaknesses": "This is a strong paper, I do not see any major weaknesses. It would be good to report some failure cases and discuss limitations.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and suggestions. Please see our discussion about limitations in the Common Response.\n\n**R1 - Definition of JVSP and ASDS:** Thank you for pointing it out. In our work, JVSP stands for \"Joint Visual Semantic and Perceptual loss\". This loss is used to optimize the similarity of the synthesized sketches and the instance sampled from the frozen latent diffusion model. ASDS stands for \"Augmentated Score Distillation Sampling loss\". This loss is utilized to ensure that  the generated images align with the text prompt consistently. We have included these denifitions in Line179-189 of Sec.4 in the revised version of the manuscript."}, {"review_id": "9mdPMt0jfR", "reviewer": "Reviewer_PspM", "summary": "This paper proposed an algorithm that creates vectorised free-hand sketches using text as input. Particularly, the authors use a pre-trained text-to-image diffusion model and transfer its prior knowledge into a differentiable rasteriser. It performs the task by directly optimising a set of B\\'ezier curves with an extended version of the score distillation sampling (SDS). The loss is backpropagated through the differentiable rasteriser to update the control points and opacity of b\\'ezier strokes at each step until convergence of the loss function.", "questions": "Apart from my aforementioned weakness, there are a few additional questions/confusions:\n\nFirst, the authors mention in L3 that the \"algorithm that creates vectorized free-hand sketches\" -- I think we should carefully consider the definition of what is a \"free-hand sketch\" -- a sketch that has stroke deformations typically seen in a human drawn sketch.\nAs DiffSketcher was never trained on a human-drawn sketch, so how can the authors assert that it is generating free-hand sketch and not something that looks realistic but follows edge-maps? \nOne way of evaluating if DiffSketcher truly generates free-hand sketch and not some edge-map is to train a Fine-Grained Sketch-based Image Retrieval (FG-SBIR) model on sketches from DiffSketcher and evaluate on real human drawn free-hand sketches. The authors can use some standard FG-SBIR dataset like Sketchy or QMUL-Shoe-V2.\n\nOn a minor note, the authors should provide a clear description of why SDS is better than something like CLIP. This description could be accompanied with some insightful analysis or experiment. I would leave this to the authors to decide and think what experiments or analysis to show that helps to clarify SDS vs CLIP for DiffSketcher. \n\nAlso the authors should better motivate \"LDM for Vector Sketch Synthesis\". Although I know it is not true but the current introduction reads like -- LDM is a hot topic, hence we extended CLIPasso with LDM.\n\nFinally, to make the paper more interesting, the authors can add an interesting discussion as to why something super simple like \"StableDiffusion+CLIPasso\" is not a good alternative for text-to-sketch generation.", "limitations": "Since this work is quite literally a sub-section of another paper \"VectorFusion\", it shares the same as mentioned in the discussion section 6 of the paper. I am simply quoting those limitations (application to this paper as well) for completeness:\n\n\"VectorFusion faces certain limitations. For instance, forward passes through the generative model are more computationally expensive than contrastive approaches due to its increased capacity. VectorFusion is also inherently limited by Stable Diffusion in terms of dataset biases and quality, though we expect that as text-to-image models advance, VectorFusion will likewise continue to improve\"", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is relatively easy to follow and the qualitative results are quite convincing. This line of work -- synthetic sketch generation -- is recently gaining quite some traction and it is nice to see another addition to the community. The authors did a commendable job in writing the paper clearly with nice diagrams that are intuitive.", "weaknesses": "The biggest weakness of this paper is its novelty. While there might not be a paper that ONLY focuses on text-to-sketch using SDS loss, the authors missed an important paper that does the same job -- VectorFusion (https://arxiv.org/pdf/2211.11319.pdf).\n\nParticularly, VectorFusion uses the same differentiable rasteriser (including optimising the opacity parameter) to generate 2D line drawings. See Fig. 2 in VectorFusion and section 5.5\n\nLike in this paper, VectorFusion can also control the level of detail using B\\'ezier controls. I would love to think that DiffSketcher and VectorFusion are parallel works (and the authors did not know about VectorFusion paper). Nonetheless, it should be noted that VectorFusion was uploaded on arXiv on Nov 21st, 2022 and accepted to CVPR'23 long before NeurIPS submission deadline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We address your concerns one-by-one below\n\n**R1 - Definition of \"Free-hand sketch\" and evaluation of these generated sketches:** We appreciate the reviewer for raising this question. However, we respectfully disagree with the reviewer's perspective on the definition of free-hand sketch. Our understanding aligns with the definition provided in CLIPasso that \"using a set of thin, black strokes (Bézier curves) placed on a white background\". It is important to note that sketches can exhibit different styles depending on the drawing skills of the individuals creating them. Datasets such as Sketch or QMUL-Shoe-V2 contain sketches drawn by amateurs, commonly referred to as amateur sketches. These sketches often exhibit significant deformation and a more abstract nature. In contrast, sketches produced by professionals tend to have fewer deformations and are less abstract. The style of sketches generated by our approach closely resembles that of professional sketches. \n\nTo assess the authenticity of the synthesized sketches, we conducted a perceptual study. Specifically, We gathered a total of 90 synthesized sketches using three different methods (30 samples per method) and obtained 30 real sketches from Google Image by searching for \"free-hand sketch\". We then mixed the real and fake sketches and distributed questionaires to 41 participants. The participants were asked to determine whether each sketch was drawn by a human or not, without any knowledge of its source. We utilized the confusion score as the evaluation metric, where a higher score indicates a greater likelihood of the generated sketches being recognized as real. The results are presented in Table R2. It is clear to see that our method produced sketches that were more frequently identified as real, highlighting the superior quality of our synthesized sketches. \n\n**Table R2 Results of the User Study.**\n| Metric \\ Method | CLIPasso |  VectorFusion | DiffSketcher (Ours) | Human Sketch |\n| --- | --- | --- | --- | --- |\n| Confusion Score | 0.39 | 0.33 | 0.65 | 0.67 |\n\n> The Confusion score of real sketch means only 67% real sketches are recognized as real.\n\n**R2 - Comparison of JVSP loss and ASDS loss:** In DiffSketcher,our JVSP loss incorporates both CLIP loss and LPIPS loss, and it is important to note that **ASDS and JVSP do not conflict with each other**. As depicted in Fig. 6 of the manuscript, the 4th and 6th rows highlght the effects of JVSP loss and ASDS loss, respectively. When only JVSP loss is used (4th row), the generated sample closely approximates the result of LDM. On the other hand, when only ASDS loss is employed (6th row), the generated sample aligns with the text prompt semantically, but does not follow the attention map of the LDM. Additionally, using ASDS loss results in more diverse sampling outcomes during the synthesis process. For instance, the location of the astronaut's head may change throughout the drawing process when ASDS loss is used, while it remains in the same location when the JVSP loss is employed.\n\nConsidering these observations, we can summarize the features of JVSP loss and ASDS loss as follows: (1) JVSP loss: It constrains the alignment between the generated sketch and the LDM's result, making it suitable for a two-stage pipeline. JVSP loss relies on the LDM's result and allows modification through adjustments to the text prompt, providing control over the generated sketch. (2) ASDS loss: It does not rely on the LDM's result and can be used for one-stage optimization. ASDS loss offers more flexibility, resulting in diverse generated samples. \n\nIn our full model, we leverage the advantages of both JVSP loss and ASDS loss by employing them for optimization. As demonstrated in the 5th row of our results (labeled \"DiffSketcher\"), when these two losses are combined, the synthesized sketches exhibit more intricate details and appear visually more realistic.\n\n**R3 - Motivation of our approach:** Thanks. The motivation behind that we develop our model based on LDM lies in three aspects: (1) LDM shows great potential in raster image generation. Unfortunately, it cannot generate vector sketches. (2) CLIPasso achieves great performance in image-conditioned sketch generation, inspiring us to utilize differentiable rasterizer for vector sketch generation. (3) DreamFusion shows a way to distill the drawing knowledge from the LDM to Nerf, which inspires us to transfer the knowledge from the LDM to the differentiable rasterizer. We will modify Line 38~39 to provide a clearer motivation.\n\n**R4 - Why \"StableDiffusion+CLIPasso\" is not a good alternative:** A good question! There are primarily two reasons: Firstly, the original CLIPasso does not incorporate opacity for optimization, resulting in relatively low visual quality of the synthesized sketches, as depicted in Fig. 5 of the manuscript. Secondly, the optimization process of CLIPasso is time-consuming. As illustrated in Fig. 6 (labled \"CLIPasso+opacity+saliency\"), approximately 2000 steps are required to achieve a satisfactory result. It is worth noting that the inclusion of opacity in the optimization process enhances the visual quality of the result. For comparison, we replace with our proposed initialization strategy (labeled \"CLIPasso+opacity+LDM Attn\"), the results show that the synthesis process can be accelated by approximately 8 times. "}, {"review_id": "WGBby32mLM", "reviewer": "Reviewer_6RS1", "summary": "The authors introduce DiffSketcher, an algorithm for generating vectorized free-hand sketches from text input. DiffSketcher leverages pretrained text-to-image diffusion models to optimize parametric Bézier curves and generate sketches that align with the text prompt. The method incorporates Score Distillation Sampling (SDS) loss and a differentiable rasterizer to guide the optimization process. Additionally, a joint attention-based stroke initialization strategy improves convergence towards semantic depictions. The paper also demonstrates the effectiveness of DiffSketcher in generating sketches with various levels of abstraction. Overall, this is a promising approach to bridge the gap between natural language and free-hand sketches.", "questions": "- Would appreciate some clarification regarding the VAE decoder\n- How long does it take per sketch?\n- How do you set the number of strokes? Shouldn't complicated prompt need more strokes than simple prompt in order to get the same level of abstraction?\n\npaper that might be related:\nSketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications\n", "limitations": "I might have missed it but I don't see any mention of the limitations of the proposed method in the paper. I think this is quite important to see since the evaluation for this sort of task can be quite subjective (qualitative results could have been cherry pick). So I'd appreciate some discussion about the limitation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is written really well, and the concepts are explain clearly and intuitively for the most part.\n- addressing a relatively underexplored area of text-to-sketch synthesis\n- The overall idea is simple (combining differentiable rasterizer with text-to-image diffusion) and it is well-designed specifically for generating vector sketch from text.\n- the stroke initialization based on fused attention map is also neat\n- interesting idea to incorporate opacity in vector sketch to mimic the way human draw \n- the results are good", "weaknesses": "- the discussion of the results and comparison is somewhat lacking. For example, the ablation study is really short and I'd love to see more insight on the effect of each components\n- The evaluation is also a bit unsatisfying. While the results are impressive on their own, the comparison with existing work seems unfair since the style are quite difference (and if we are using diffusion model to generate image anyway, any image to sketch techniques could have been used?). And I feel like the quantitative evaluation which relies on CLIP embeddings trained on images, may favor sketches with more detailed styles, or more human-like style anyway. So when the baselines are weak, the comparison doesn't really give us much insight into the other strength of the method.\n- I also cannot find much information in the paper about the VAE decoder in the proposed method (that convert image to sketch?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and constructive suggestions. We address your concerns one-by-one below:\n\n**R1 - Insufficient discussion about ablation study:** Thank you for your suggestion. From the Fig.6, we can have the following observations: (1) Our proposed initialization strategy significantly accelerates the synthesis process (”CLIPasso+opacity+saliency“ vs. ”CLIPasso+opacity+LDM Attn“); (2) The 4th and 6th rows highlght the effects of JVSP loss and ASDS loss, respectively. When only JVSP loss is used (4th row), the generated sample closely approximates the result of LDM. On the other hand, when only ASDS loss is employed (6th row), the generated sample aligns with the text prompt semantically, but does not follow the attention map of the LDM. (3) Additionally, using ASDS loss results in more diverse sampling outcomes during the synthesis process. For instance, the location of the astronaut's head may change throughout the drawing process when ASDS loss is used, while it remains in the same location when the JVSP loss is employed. (4) Comparing the results of using SDS loss and ASDS loss (newly added), using ASDS loss leads to higher generation quality.\n\nConsidering these observations, we can summarize the features of JVSP loss and ASDS loss as follows: (1) JVSP loss: It constrains the alignment between the generated sketch and the LDM's result, making it suitable for a two-stage pipeline. JVSP loss relies on the LDM's result and allows modification through adjustments to the text prompt, providing control over the generated sketch. (2) ASDS loss: It does not rely on the LDM's result and can be used for one-stage optimization. ASDS loss offers more flexibility, resulting in diverse generated samples. \n\n**R2 - Insufficient evaluation:** When submitting this paper, the available baseline methods were very limited. Now we have included a comparison with VectorFusion, whcih is a strong baseline method for comparison. Please find more details in Common Response.\n\n**R3 - Explaination of VAE-decoder:** Sorry for the confusion. In this work, the VAE decoder is not intended for image-to-sketch translation. Instead, we use the open-source stable diffusion model, which is a latent space diffusion model that employs a pre-trained VQ-VAE to encode the image from the pixel space into the latent space to perform the diffusion and denoising process. Consequently, a VQ-VAE is required to decode the latent representation back into the pixel space. We strictly adhere to the decoder parameters specified in stable diffusion model and have updated the Fig.6 in the supplementary file to eliminate any ambiguity.\n\n**R4 - Running Efficiency:** We tested the rendering efficiency of DiffSketcher on a single NVIDIA A800 and Intel(R) xeon(R) Platinum 8336C CPUs. When the number of strokes is 32, it takes 29 seconds to iterate 100 steps and 59 seconds to iterate 200 steps. When the number of strokes is 64, it takes 60 seconds to iterate 200 steps. When the number of strokes is 128, it takes 64 seconds to iterate 200 steps.\n\n**R5 - Control of the number of strokes:** \nOur current method requires the user to mannually set the number of strokes. However, it should be noted that the model can implicitly influence the abstractness of generated sketches by setting the opacity of some strokes to 0 (full transparency) if there are too many strokes. As we discussed in the limitations in the common response, we will leave the automatically control the number of strokes or abstractness of generatd sketches in future research.\n\n**R6 - Missing reference**: Thanks. Sketchforme uses bounding boxes and object sketchers to generate sketch strokes based on text conditions, while our approach directly render vector images based on text prompts. This work is related to our work and we have discussed this paper in related work section of the revised version of the manuscript.\n"}, {"review_id": "QXmNNyAaI5", "reviewer": "Reviewer_mvSd", "summary": "The authors introduce DiffSketcher. The model makes use of a frozen pre-trained latent diffusion model (Imagen) to guide the parametrisation of bezier curves and reproduce the generated image in vectorised sketch form. The method works in a similar way to DreamFusion, but instead of using a differentiable 3D renderer (DreamFusion is a text-to-3D model) the authors use a differentiable bezier curves renderer.", "questions": "- The current work is constrained by its direct connection to a model that produces realistic images. Do you see how a future method might be able to achieve true abstraction (e.g. cats with whiskers and a house as a triangle+rectangles)?", "limitations": "The authors do not discuss current limitations of the model.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper shows very \"believable\" results: \"human-like\" sketches.\n- The first text-to-sketch model to work at the scene and object level.\n- They make good use of the methods available in the literature, the combination of the Li et al [15] differentiable rasteriser with the  score distillation sampling loss from DreamFusion being particularly effective\n- The authors additionally improve the random startup of the bezier curves and show how the inclusion of a perceptual (LPIPS) and \"semantic\" (CLIP feature comparison) loss improves fidelity of the sketch to the prompt. ", "weaknesses": "- I was not able to identify what is the benefit of using the Augumentation SDS loss instead of the original SDS loss or why it \"encourages plausible images to have low loss and implausible ones to have high loss\"?\n- The authors claim it is possible to control the abstraction level of the sketches but that seems to be through the number of strokes. There are multiple ways of defining abstraction and because these sketches are always based on image inputs it is improbable that the method would be able to reproduce higher abstraction levels such as the ones found on the QuickDraw dataset.\n- Some of the notation is confusing. Equations 1, 2 and 3 are probably hard to parse if the reader is not familiar with the original references. The use of S_a and S_phi for sketches and images is also particularly confusing on first read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments and constructive suggestions. We address your concerns one-by-one below:\n\n**R1 - Benefits of ASDS loss:** Thank you for your suggestion. To demonstrate the superiority of ASDS loss over SDS loss, we conducted a new ablation experiment and presented the results in Fig.R5 of the supplementary file. We can observe that using ASDS loss leads to higher visual quality compared to the original SDS loss.\n\n**R2 - Control of the abstraction level of the sketches:** A good question! (1) In this work, we follow CLIPasso to use the number of stroke to control the abstraction level. As mentioned in the Common Response, we acknowledge the need for future research to incorporate automatic control of the stroke count. (2) Regarding reproducing sketches with a higher abstraction level, we show several examples in Fig.R3 first row of the supplementary file. However, our method finds it challening to produce sketches similar to those in the QuickDraw dataset. The sketches from the QuickDraw dataset were drawn by amateurs within a limited time, resulting in significant deformations and abstractions. Consequently, they exhibit a distinct style that differs from ours.\n\n**R3 - Confusing notations:** $S_a$ indicates the augmented version of the sketch, while $S_\\phi$ indicates the VAE decoder used to obtain the RGB pixel representation of the sketch.\n\n**R4 - How to achieve true abstraction:** This is a very inspiring question! Firstly, we would like to clarify that when solely using ASDS loss for optimization, our model does not rely on raster images produced by LDM (as explained in Section 4.1.2). However, our model learns how to draw the sketch from the LDM model, which means its drawing ability is limited by the capabilities of the LDM model. Secondly, to achieve the true abstraction, we suppose there are two possible solutions. (1) Treating abstraction, i.e., using basic shapes to represent an object, as a style and introducing a style transfer module to guide the generation model in synthesizing sketches with this specific style. (2) Introducing real-world knowledge to the model, enabling it to understand the composition of each object. This solution is more fundamental. We will explore this problem in our future research."}, {"review_id": "BOjc6Xi6U1", "reviewer": "Reviewer_ZP2A", "summary": "This paper proposes DiffSketcher, a stroke-based rendering model using a pre-trained text-to-image diffusion model. It performs the task by directly optimizing a set of Bézier curves with an extended version of the score distillation sampling (SDS) loss, which allows using a raster-level diffusion model as a prior for optimizing a parametric vectorized sketch generator.", "questions": "1. What is the total stroke number of a sketch? Is it user-specified or automatically determined by some means?\n\n2. Control points (anchors). In line 225, how do you sample the control points from the distribution map? Using rejection sampling or adaptive sampling or other methods? And the stroke width. In the supplementary material, the authors handly set this parameter. Why not make it learnable as the opacity?\n\n3. Running Efficiency. It seems the optimizing process is very costly. What is the running time to generate a sketch? Please give the used image resolution, stroke number, and iteration steps. \n\n4. In figure 3, is the $\\varepsilon$ and $\\epsilon$ are two different things?", "limitations": "I doubt the meaning of this kind of technique route that uses a pre-trained diffusion model to generate a raster image and then translate the raster image into a vector graphic. This technique route is just an engineering combination of a generative model and a raster-to-vector algorithm. I prefer to see the authors proposing improvements to the diffusion models or the raster-to-vector algorithms, but not just combining them.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes a stroke-based rendering method for synthesising vector graphics.\n2. This method does not need to retrain the very cost Diffusion model.\n3. This method could use text to control the painting content.", "weaknesses": "1. Lack of novelty. The model's generation ability essentially origins from the pre-trained diffusion model and the differentiable rasterizer module process is just a fitting process. I will discuss this in the below section \"Limitations\" further.\n\n2. Lack of comparison. Figure 5 shows the raster images generated by the diffusion model and the corresponding sketching results. However, the compared methods Canny and CLIPasso are too weak. The authors should compare their results with more robust image-to-sketching rendering methods such as \"Combining sketch and tone for pencil drawing production\" and image style transfer methods such as StyleGan. In the supplementary material, I have seen comparisons with CLIPDraw. But this kind of comparison could not support anything since their artistic styles are totally different.\n\n3. Lack of a very important citation. This paper is a sketching version of the published CVPR 2023 paper \"Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models\". Both papers follow the same idea that using diffusion models to generate raster images and then using diffvg to optimize stroke parameters. Although the task and technique details are different, no citation of such similar work is unacceptable.\n\n4. Poor writing. The authors should explain their proposed abbreviation the first time they use it. For example, the \"ASDS\" in line 183. When I got to line 185, I saw the \"Augmentation SDS\" invented by the author. And I did not see what the abbreviation \"JVSP\" mean.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive suggestions. We address your concerns one-by-one below:\n\n**R1-Lack of novelty:** Thanks. Firstly, we respectively disagree with your perspective on the *novelty*. Diffvg] was the first to introduce the differentiable rasterizer, while CLIPasso effectively linked Diffvg and CLIP loss, achieving impressive performance in image-conditioned sketch generation. We believe both works are novel contributions. In this work, we further extend the drawing capabilities of LDM to vector sketches. We introduce to use attention map of LDM to initialize SVG path and present ASDS loss and JVSP loss for optimization. We also provide insights into how each loss affects the synthesis process, and eventually achieve significant progress in the target task. Secondly, although vector graphics are widely used in practice, research on vector graphics generation remains limited. Our work demonstrates a new approach to realize text-based vector sketch generation, which we believe will contribute to the advancement of this research field.\n\n**R2-Lack of comparison:** Thanks for your suggestion. While generative models like GANs and Diffusion Models have shown impressive performance in various domains, it is true that most research, including notable works like StyleGAN, has primarily focused on generating raster images. There has been limited research specifically targeting the generation of vector images such as SVG. CLIPasso is the former SOTA method in this research field, CLIPDraw is also a CLIP-based method specifically designed for text-to-SVG generation, thus we compare with these two methods. Additionally, we follow your suggestion to include VectorFusion for comparison. Please find details in our Common Response.  \n\n**R3-Poor writing:** Thank you for pointing it out. JVSP stands for \"Joint Visual Semantic and Perceptual loss\". This loss is used to optimize the similarity of the synthesized sketches and the instance sampled from the frozen latent diffusion model. ASDS stands for \"Augmentated Score Distillation Sampling loss\". This loss is utilized to ensure that  the generated images align with the text prompt consistently. \n\n**R4-How to set the stroke number?** In our work, the number of strokes in a sketch is user-specified.\n\n**R5-Parameters to be optimized:** Firstly, we feed the attention map into a softmax layer to obtain the distribution map, and then sample evenly according to their probabilities. Secondly, in our current implementation, the stroke width is preset, meaning it is fixed throughout the sketch generation process. However, it is possible to treat the stroke width as a learnable parameter and optimize it during training. This would allow the model to adaptively adjust the stroke width based on the text input, providing greater control of the generated sketches.\n\n**R6-Running Efficiency:** Follow your suggestion, we tested the rendering efficiency of DiffSketcher on a single NVIDIA A800 and Intel(R) xeon(R) Platinum 8336C CPUs. The results are shown in Table R3:\n\n**Table-R3 Efficiency of our proposed DiffSketcher.**\n\n| Canvas Size | Stroke Number | Iteration Steps | Time (Sec) |\n| --- | --- | --- | --- |\n| 224 * 224 | 32 | 100 | 29 |\n| 224 * 224 | 32 | 200 | 58 |\n| 224 * 224 | 64 | 200 | 60 |\n| 224 * 224 | 64 | 500 | 150 |\n| 224 * 224 | 128 | 200 | 64 |\n| 224 * 224 | 128 | 500 | 160 |\n\nIn addition, we conducted a comparison of the rendering time required by VectorFusion to generate a sketch with 64 strokes. VectorFusion uses the same layered rendering strategy as LIVE, and it takes 1404 seconds (with LIVE taking 1149 seconds and LSDS fine-tuning taking 255 seconds) to obtain a reasonable result.\n\n**R7-Typo:** Sorry for the confusion. In figure 3,  $eps$ and $eps$  are the same symbol used to represent Gaussian noise."}]

Author rebuttal:
We appreciate the reviewers for their constructive comments. Firstly, we would like to address the common concern raised by Reviewer PspM and Reviewer ZP2A regarding the missing reference of the work VectorFusion, and then discuss the limitations of our approach.

**Comparison with VectorFusion (CVPR'23):** Indeed, we were not aware of this paper prior to our submission, and we only found it when it was listed in   the program on the CVPR 2023 website.

VectorFusion is highly relevant to our work. However, there are notable differences between our appraoch and VectorFusion in terms of task setting, methodology, and performance. Firstly, our method, DiffSketcher, primarily focuses on generating vector sketches based on text input. In contrast, VectorFusion aims to generate a broader range of vector graphics. It is important to highlight that our method has the capability to easily extend its functionality to generate other types of vector graphics, such as icon-style image, without the need to alter primitives, we show the results of our approach in generating general vector graphics in Fig.4 of the supplementary file. 

Secondly, DiffSketcher follows a distinct pipeline compared to VectorFusion. VectorFusion employs different model variants for different types of vector graphics. For generating vector sketches, our approach differs from VectorFusion mainly in two aspects: **1) Initialization:** VectorFusion randomly initializes the rasterizer (i.e., the location of the control points), while our approach utilizes the attention layer of LDM, guided by text prompts, for initialization. This significantly improve both efficiency and syntehsis quality. **2) Optimization:** While VectorFusion solely optimizes the position of control points, our method also optimize the opacity of stroke. This contributes to enhance the visual quality of the synthesized sketches. We present a performance comparison between these two methods in terms of vector sketch generation in Fig.1. 

Furthremore, when generating other types of vector graphics, VectorFusion involves generating raster images using LDM and subsequently converting them into vector images using LIVE. However, our approach does not require the conversion step. Besides, VectorFusion necessitates chaning the primitives and specifically designed prompt suffix string for generating different types of vector graphics, whereas our approach maintains a consistent primitive (i.e., Bézier curve) and has no limitations on text prompt. In Table R1, we provide a comprehensive comparison of these two methods, highlightling their differences and distinctive features. 

**Table-R1 Comparison of our approach and VectorFusion.**

|  | DiffSketcher (ours) | VectorFusion |
| --- | --- | --- |
| Prompt | Any prompt, no limitations | Prompt with specifically designed suffix string |
| Initialization | Fusing cross-attention and self-attention of LDM to initialize control points | Using LIVE to convert the raster image to vector format for initlization |
| Optimization parameters | Control points + Opacity| Control points |
| Optimization | All SVG paths are optimized together | Follow LIVE, using layer-wise optimization (8 paths at a time) |
| Efficiency | 150 seconds per SVG | 1400 seconds per SVG |

**Limitations of the proposed method:** Our approach mainly has two limitations. Specifically, one limitation is the lack of correlation between the text prompt and sketch abstractness. For instance, if the text prompt describes multiple objects but the number of strokes is set too small, the resulting sketches may be unsatisfactory. A possible solution is to estabilish a link between the complexity of the text prompt (such as the number of described objects) and the number of strokes to be used. We show several examples of failure cases (Fig.3) in supplementary file. 

Another limitation of the current approach is the style of the generated sketches is limited. We can address this by introducing a style transfer module during the sketch synthesis process.

Meta-review: The paper proposes a method for generating sketch-like images from text by using Score Distillation Sampling (SDS) on a pretrained text-to-image model to directly optimize a set of Bezier curves. Reviewers were generally favorable on the paper, and appreciated the method and results.

The primary concern raised by reviewers was a missing citation and comparison to VectorFusion which was published at CVPR 2023. In their rebuttal, the authors provided a detailed comparison of their approach with VectorFusion, including both a description of the technical differences between the methods and a user study demonstrating strong performance of the proposed method. Reviewers were largely sympathetic to these arguments, and in the end all reviewers were in favor of acceptance.

The AC agrees that the proposed method is distinct enough from VectorFusion to be of interest, and finds the additional discussion and comparisons provided in the rebuttal to be convincing. The authors are strongly encouraged to include these comparisons and results in the camera-ready version of the paper.
Decision: Accept (poster)