SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Where the weighting function  $w(t)$  is a hyper-parameter. And we sample  $t \sim \mathcal{U}(0.05, 0.95)$ , avoiding very high and low noise levels due to numerical instabilities. Like DreamFusion [24], we set  $\omega = 100$  for classifier-free guidance and higher guidance weights give improved sample quality. Intuitively, this loss perturbs  $\bar{S}_a$  with a random amount of noise corresponding to the timestep  $t$ , and estimates an update direction that follows the score function of the diffusion model to move to a higher density region. The ASDS loss function encourages the synthesized sketch to match the given text prompt, while also preserving the style and structure of the original sketch. At each iteration, we backpropagate the loss through the differentiable rasterizer and update the control points and opacity of strokes directly until convergence of the loss function.
Loss objectives of our final model. To further enhance the quality of the synthesized sketches, we incorporate the JVSP and ASDS losses. Specifically, we first obtain the initial results with the JVSP loss, then we fine-tune the differentiable rasterizer together with the ASDS loss. We found such design can achieve the best performance. The ASDS loss predicts the gradient update direction directly, thus its loss value does not require weight balancing. In the JVSP loss, we set the weight of the LPIPS item to 0.2, and the weight of the CLIP visual item to 1. To compute the L2 distance between intermediate level activations of CLIP, we follow the CLIPasso [44] method and use layers 3 and 4 of the ResNet101 CLIP model. As shown in Figure 8, we compare the performance of the vanilla version with that of the version using only the ASDS loss, and the proposed final version. Our experiments show that the final version improves both the generation quality and efficiency.
4.2 Joint Attention-based Stroke Initialization
The highly non-convex nature of the ASDS loss function makes the optimization process susceptible to initialization, especially in multi-instance scenarios where strokes must be carefully placed to emphasize the overall semantics conveyed by free-hand sketching. We improve the convergence towards semantic depictions by initializing the curve control points based on the attention map of the text prompt conditional latent diffusion model. The UNet in the LDM has two types of attention mechanisms (the blue part), self-attention and cross-attention, as shown in Figure 5 where the yellow part represents the cross-attention visualization result, and the green part represents the self-attention visualization result. The structure and appearance of the image generated by the LDM [30] depend on
the interaction between the pixels to the text embedding through the diffusion process [10], which is manifested in the cross-attention layer [43]. The visualization results indicate that the cross-attention layers control the relationship between the spatial layout of the image and each word in the prompt, while the self-attention layer affects the spatial layout and geometric structure of the generated image. With this observation, we linearly combine the probability distributions of the two attention maps to initialize the curve control points. Specifically, we pick a map in the cross-attention maps based on a text token and combine it with the averaged self-attention map. This process is formalized as FinalAttn = λ * CrossAttni + (1 - λ) * Mean(SelfAttn), where λ is the control coefficient and i indicates the i-th token in the text prompt. Finally, we normalize the fused attention map using softmax and use it as a distribution map to sample n positions for the first control point  $p_i^1$  of each Bézier curve. The other three control points  $(p_i^2, p_i^3, p_i^4)$  are sampled within a small radius (0.05 of image size) around  $p_i^1$  to define the initial set of Bézier curves  $\{\{p_i^j\}_{j=1}^4\}_{i=1}^n$ . Empirical results show that our attentional fusion-based initialization contributes significantly to the quality and rendering speed of the final sketch in comparison to the random initialization.
Figure 6: Comparison with existing methods, including edge extraction [2] and vector sketching [45, 44].
5.1 Qualitative Evaluation
As shown in Figure 2, we demonstrate that our approach offers the ability to produce object-level and scene-level sketches based on a textual prompt, with the flexibility to manipulate the level of abstraction through the number of strokes. It is effective in generating accurate sketches regardless of prompt complexity, including simple or non-existent objects, iconic constructions, and detailed scenes. The utilization of the robust prior of stable diffusion allows for a favorable initialization, promoting the production of high-quality sketches with significantly fewer iterations. Figure 6 qualitatively compared our DiffSketcher with those of CLIP-based [45] and edge extraction-based methods [2]. The Canny edge extraction algorithm extracts an excessive number of edges and produces untidy sketches, as observed in the first example in Fig. 6. CLIPasso uses visual distance metrics to guide gradient-based optimization. On the task of drawing scene-level sketches, CLIPasso can only draw part of the foreground, and the background part is missing.
We also compare our work with the recent work VectorFusion [13]. The results are shown in Fig. 7. This work is highly relevant to ours as it also explores the potential of diffusion model in generating vector graphics. However, there are notable differences between our approach and VectorFusion in terms of task setting, model design, and performance. Firstly, our method, DiffSketcher, primarily focuses on generating vector sketches based on text input. In contrast, VectorFusion aims to generate a broader range of vector graphics, including iconography and pixel art. It is important to highlight that our method has the capability to easily extend its functionality to generate other types of vector graphics without changing primitives (as shown in Fig. 10 of Supp. B). Secondly, DiffSketcher follows a distinct pipeline compared to VectorFusion. VectorFusion employs different model variants for different types of vector graphics. For generating vector sketches, our approach differs from VectorFusion mainly in two aspects: (1) Initialization: VectorFusion randomly initializes the
Figure 7: Qualitative comparison with VectorFusion(VF) [13]. VF's results are based on its original paper and project website, with a text prompt suffix of "minimal 2D line drawing trending on ArtStation". Note that our results were optimized from scratch using ASDS without specifically designed text prompt suffix.
rasterizer (i.e., the location of the control points), while our approach utilizes the attention layer of LDM, guided by text prompts, for initialization. This significantly improves both efficiency and synthesis quality. (2) Optimization: While VectorFusion solely optimizes the position of control points, our method also optimizes the opacity of stroke. This contributes to enhance the visual quality of the synthesized sketches. As shown in Fig. 7, our approach significantly outperforms VectorFusion in generating vector sketches, and achieves comparable performance in generating other types of vector graphics (see Fig. 10 of Supp. B). It is worth noting that our model variant shown in Fig. 7 utilizes only ASDS loss for optimization and initializes the strokes randomly. This was done to ensure a fair comparison with VectorFusion. Our full model is expected to deliver better performance.
5.2 Quantitative Evaluation
Evaluating text-to-sketch synthesis is challenging due to the absence of ground truth sketches. Therefore, we focus on three indicators: consistency between the generated sketch and text prompt, the aesthetic quality of the sketch, and the recognition accuracy of the sketch. To measure the consistency between the generated sketch and input text, we calculate the mean of cosine similarity of CLIP embeddings for the generated sketches and the text captions used to generate them. Our method achieves a cosine similarity of 0.3494, which is higher than the 0.328 achieved by Canny algorithm and the 0.3075 achieved by CLIPasso. Aesthetic appeal is subjective, and being visually appealing is a personal experience and preference. However, when describing the visual appeal of a hand-drawn sketch, various factors can be considered, such as line quality, texture, and style. As an evaluation of our proposed method, we use CLIP-based aesthetic indicators [33] to calculate the aesthetic value for samples of multiple categories. Figure 6 compares aesthetic values achieved by various methods. Our method achieves a mean value of 4.8206 for the aesthetic value on a large number of examples, which is higher than the 4.3682 achieved by Canny and the 4.0821 achieved by CLIPasso [45].
We conducted a series of experiments to demonstrate the effectiveness of the proposed initialization strategy and the effects of ASDS loss and JVSP loss, respectively. The top row of Fig. 8 compares two initialization strategies used by DiffSketcher and CLIPasso [45], namely LDM Attention and CAM [35] Saliency map, respectively. The text-driven diffusion model produces more precise attention map than the saliency map obtained by the CLIP-based method due to LDM's superior generation ability. The cross-attention feature of LDM can efficiently activate relevant regions based on token areas, while the self-attention layer can effectively differentiate foreground from background down to the pixel level. Through combining both mechanisms, the initialization area becomes more precise. As a result, superior quality sketches are produced with a reduced number of optimization steps, as illustrated in the second and third rows of Fig. 8. The proposed initialization strategy improves sampling quality and efficiency, which is critical for non-convex objective function optimization.
Figure 8: Qualitative results of ablation study. Top: two initialization strategies used by DiffSketcher (i.e., LDM Attention) and CLIPasso [45] (i.e., CAM [35] Saliency map). A result of sampling from LDM. The 2nd and 3rd rows: Comparison of the two initialization strategies on the convergence speed. The 4th and 5th rows: The effect of JVSP (Section 4.1.1) and ASDS loss (Section 4.1.2). The 5th row shows the loss of SDS without data augmentation. The 6th and 7th rows: ASDS loss leads to more diverse results when strokes are randomly initialized. Text prompt used in this example: "Astronaut on Asteroid, galaxy background".
In DiffSketcher, our JVSP loss consists of a CLIP loss and a LPIPS loss, and it is important to note that ASDS and JVSP do not conflict with each other. As illustrated in Fig. 8, the 4th and 8th rows highlight the effects of JVSP loss and ASDS loss, respectively. When only JVSP loss is used (4th row), the generated sample closely approximates the result of LDM. On the other hand, when only ASDS loss is employed (8th row), the generated sample aligns with the text prompt semantically, but does not follow the attention map of the LDM. Additionally, using ASDS loss results in more diverse sampling outcomes during the synthesis process. For instance, the location of the astronaut's head may change throughout the drawing process when ASDS loss is used, while it remains in the same location when the JVSP loss is employed. We utilize both JVSP loss and ASDS loss for optimization. As shown in the 5th row (labeled as "DiffSketcher"), the combination of these two losses leads to synthesized sketches with more intricate details and a visually more realistic appearance.
In this work, we have proposed a novel and effective approach to bridge the gap between natural language and free-hand sketches. By leveraging the power of pretrained text-to-image diffusion models, we have developed a method that can generate diverse and high-quality vector sketches based on textual input, without the need for large-scale datasets or sketch-text pairs. We have also explored different aspects of the model design, such as the stroke initialization strategies, choice of loss functions, and properties of strokes in the differentiable rasterizer, which provide valuable insights for future research. With further research and development, our proposed model, DiffSketcher, has the potential to be a valuable tool in various domains, such as design and education. Limitation. Our approach has two limitations. Firstly, there is a lack of correlation between the
text prompt and sketch abstractness, which may result in unsatisfactory sketches. To address this, we suggest establishing a link between the complexity of the text prompt and the number of strokes used. More details can be found in Supp. I. Secondly, the style of the generated sketches is limited. A possible solution is introducing a style transfer module during the sketch synthesis process.
This work is supported by the National Natural Science Foundation of China (No. 62002012, No. 62006012, and No. 62132001) and CCF-Baidu Open Fund. It is also supported in part by The Hong Kong Jockey Club Charities Trust under Grant 2022-0174, in part by the Hong Kong Research Grant Council under General Research Fund (17203023), in part by the Startup Funding and the Seed Funding for Basic Research for New Staff from The University of Hong Kong, and in part by the funding from UBTECH Robotics.
[34] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
[35] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision (ICCV), pages 618-626, 2017.
[36] I-Chao Shen and Bing-Yu Chen. Clipgen: A deep generative model for clipart vectorization and synthesis. IEEE Transactions on Visualization and Computer Graphics, 28(12):4211-4224, 2021.
[37] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the International Conference on Machine Learning (ICML), volume 37, pages 2256-2265, 2015.
[38] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021.
[39] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NIPS), volume 32, 2019.
[40] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
[41] Yingtao Tian and David Ha. Modern evolution strategies for creativity: Fitting concrete images and abstract concepts. In Artificial Intelligence in Music, Sound, Art and Design, pages 275-291. Springer, 2022.
[42] Zhengyan Tong, Xuanhong Chen, Bingbing Ni, and Xiaohang Wang. Sketch generation with drawing process guided by vector flow and grayscale. In Proceedings of the Conference on Artificial Intelligence (AAAI), volume 35, pages 609-616, 2021.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems (NIPS), 30, 2017.
[44] Yael Vinker, Yuval Alaluf, Daniel Cohen-Or, and Ariel Shamir. Clipascene: Scene sketching with different types and levels of abstraction. arXiv preprint arXiv:2211.17256, 2022.
[45] Yael Vinker, Ehsan Pajouheshgar, Jessica Y Bo, Roman Christian Bachmann, Amit Haim Bermano, Daniel Cohen-Or, Amir Zamir, and Ariel Shamir. Clipasso: Semantically-aware object sketching. ACM Transactions on Graphics (TOG), 41(4):1-11, 2022.
[46] Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1395-1403, 2015.
[47] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
This supplementary material is organized into several sections that provide additional details and analysis related to our work on DiffSketcher. Specifically, it will cover the following topics:
- In section A, we provide the implementation details of DiffSketcher.
- In section B, we present a qualitative comparison of our DiffSketcher with another two text-to-SVG methods, CLIPDraw [7] and VectorFusion [13]. We compare results generated by these methods and analyze the differences in terms of visual quality and semantic consistency.
- In section C, we compare sketches generated by our DiffSketcher with those directly sampled from the LDM (i.e., Stable Diffusion [30]) and analyze the differences in their style.
- In section D, we conducted a perceptual study to assess the authenticity of the synthesized sketches.
- In section E, we compare the results of three different strategies for stroke initialization.
- In section F, we visualize how DiffSketcher gradually sketches an object or a scene.
- In section G, we provide example sketches with different stroke widths.
- In section H, we introduce the details of the evaluation metrics used in our experiments.
- In section I, we show several examples of failure cases.
A Implementation Details of DiffSketcher.
We begin by describing the vanilla version of our approach (Section 4.1.1), which involves sampling an image from the latent diffusion model [30] and then automatically sketching it using DiffSketcher. Specifically, given a text prompt, we use a DDIM solver [38] to sample a raster image from the latent diffusion model in 100 steps with classifier-free guidance [12], using a scale of  $\omega = 7.5$ . To apply the Augmentation SDS loss (Section 4.1.2), we sample a noise level  $t$  from the uniform distribution  $\mathcal{U}(0.05, 0.95)$ , avoiding very high and low values that can cause numerical instabilities. For classifier-free guidance, we set  $\omega = 100$ , as we found that a higher guidance weight leads to better sample quality. This is larger than the scale used in image sampling methods, which is likely due to the mode-seeking nature of our objective, leading to over-smoothing at small guidance weights.
In DiffSketcher, the number of strokes  $n$  is defined by the user, and we use 4 duplicates for image augmentation to maintain recognizability under various distortions. For this purpose, we apply the torch.transforms.RandomPerspective, torch.transforms.RandomResizedCrop, and torch.transforms.RandomAdjustSharpness functions in sequence. It is worth noting that data augmentation is not the focus of our work, and experimental results show that the choice of augmentation strategies does not affect results significantly.
In Section 4, we define a sketch as a set of  $n$  strokes  $\{s_1, \dots, s_n\}$  with the opacity attribute placed on a white background. To optimize the control points and opacity, we use two Adam optimizers. Specifically, we set the learning rate of the control point optimizer to 1.0 and the color optimizer to 0.1.
B Comparison to Existing Text-to-SVG Work.
This section presents a qualitative comparison between DiffSketcher and CLIPDraw [7]. CLIPDraw is a CLIP-based method introduced for text-to-SVG generation. It gradually optimizes the position and colors of the curves by the gradient descents computed by comparing the cosine similarity of the text prompt and the generated drawings. Our method differs from CLIPDraw in two ways, one is the stroke initialization, and more importantly, ours is equipped with the Augmentation SDS (ASDS) loss. As illustrated in Fig. 9, CLIPDraw struggles to synthesize a meaningful and visually pleasing drawing, no matter whether with colors or not. It can be explained by the fact that the CLIP model is not a generative model, and it can only provide guidance from a highly-semantic perspective. In
Figure 9: Comparison of the results synthesized by CLIPDraw and DiffSketcher. Specifically, for each example, we compare the results generated by CLIPDraw (Left) and our DiffSketcher (Right) given the same text prompt. We implemented two versions of CLIPDraw: the RGBA version (original version) and the B&W version. The B&W version forces the stroke color to be black to mimic the sketch style. Our results are visually more pleasing and meaningful.
Figure 10: Qualitative comparison with VectorFusion(VF) [13]. VF's results were copied from Figure 2 of its original paper, with a text prompt suffix of "minimal 2D line drawing trending on ArtStation". In contrast, our results were optimized from scratch using ASDS without specifically designed text prompt suffix.
contrast, our DiffSketcher can generate sketches that are semantically consistent with the input text, and exhibit high aesthetic quality. This is because the proposed ASDS loss can distill the drawing capability of the latent diffusion model (LDM) into the differentiable rasterizer. These results also suggest the effectiveness of the ASDS loss and the benefits of leveraging the power of the LDM. VectorFusion [13] is highly relevant to our work but it aims to produce general vector graphics, such as iconography. Although our proposed method is designed to generate vector sketches, it can easily extend to generate other types of vector graphics. In Fig. 10, we compare results obtained by VectorFusion and our DiffSketcher.
Figure 11: Comparison of sketches generated by sampling from the LDM using the specified text prompt (Left) and ours (Right).
Figure 12: The style of the generated sketches is not significantly affected by the keywords used in the text prompt.
C Comparison to Existing Text-to-Image Work.
In this section, we compare the sketches directly generated by LDM using specific text prompts. To encourage the results to be abstract and follow the free-hand sketch style, we append a suffix to the text prompt: "A free-hand sketch of xxx on a white background, trending on ArtStation. Keep abstract." This prompt was tuned qualitatively to capture the desired style and artistic expression.
The results are shown in Fig. 11. It is clear that the LDM is capable of generating high-quality raster sketches in a free-hand sketch style. However, different from the vector sketches generated by our DiffSketcher, LDM's sketches have two distinguished characteristics: 1. they are more delicate and like professional sketches. 2. their background exhibits the paper texture. We suppose this is because most sketches used for training LDM are photographs of professional sketches drawn on paper. By contrast, the sketches synthesized by our DiffSketcher are more like the style of free-hand sketch and exhibit different levels of abstractness.
Figure 13: Partial sample visualization for conducting user research. The hand-drawn sketches were sourced from the Google. CLIPasso's and our results were sourced from respective paper, VectorFusion's results were sourced from their project homepage.
It is worth noting that our method does not require indicating "free-hand sketch" in the text prompt. The drawing engine of our model is the rasterizer and it can naturally capture the sketch style. We also conduct experiments to show the results of using different keywords in text prompts, such as "sketch", "drawing", and "photo". The results are shown in Fig. 12. We can see that although the style of the output images is different, the style of the generated sketches is not significantly affected by the keywords.
D User Study.
Table 1: Results of the User Study. The Confusion score of real sketch means only  $67\%$  real sketches are recognized as real.
Metric / Method	CLIPasso [45]	VectorFusion [13]	DiffSketcher (Ours)	Human Sketch
Confusion Score	0.39	0.33	0.65	0.67