SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ximing Xing
Beihang University ximingxing@buaa.edu.cn
Chuang Wang
Beihang University chuangwang@buaa.edu.cn
Haitao Zhou
Beihang University zhou Haitao@buaa.edu.cn
Beihang University zhang_jing@buaa.edu.cn
Qian Yu*
Beihang University qianyu@buaa.edu.cn
Dong Xu
The University of Hong Kong dongxu@cs.hku.hk
"Very detailed masterpiece painting of baby yoda holding a lightsaber"
"A fox is sitting on the sofa"
"Colorful hot air balloons high over the mountains"
"A cute cat in the style of Pixar animations rides a
"Real photo of Sydney opera house"
"A sketching with watercolors of a modern Athens neighborhood"
Even though trained mainly on images, we discover that pretrained diffusion models show impressive power in guiding sketch synthesis. In this paper, we present DiffSketcher, an innovative algorithm that creates vectorized free-hand sketches using natural language input. DiffSketcher is developed based on a pre-trained text-to-image diffusion model. It performs the task by directly optimizing a set of Bezier curves with an extended version of the score distillation sampling (SDS) loss, which allows us to use a raster-level diffusion model as a prior for optimizing a parametric vectorized sketch generator. Furthermore, we explore attention maps embedded in the diffusion model for effective stroke initialization to speed up
the generation process. The generated sketches demonstrate multiple levels of abstraction while maintaining recognizability, underlying structure, and essential visual details of the subject drawn. Our experiments show that DiffSketcher achieves greater quality than prior work. The code and demo of DiffSketcher can be found at https://ximinng.github.io/DiffSketcher-project/.
Minimal representations, such as sketches and natural language, are powerful tools for effectively conveying ideas by emphasizing the subject's essence. As the carriers of abstract concepts, natural language conveys abstract semantic understanding, whereas sketches embody the human visual abstract presentation. A sketch can provide more visual details than natural language, making the concept more concrete. When a designer discusses the design plan with a client, the designer may sketch a prototype based on the client's description to ensure a full understanding of his/her requirements. If the process of text-to-sketch could be learned automatically, it would significantly lessen associated labor costs.
Unfortunately, the task of text-to-sketch generation remains unexplored. Some studies conducted on generating image-conditioned sketches [3, 18, 45, 44]. One such method is Info-drawing [3], which treats a sketch as a photo and uses a generative adversarial network (GAN) to generate a raster sketch based on an input image. It also introduces a style loss to ensure the generated sketches have a similar style to the reference sketch used in training. CLIPasso [45] presents a novel pipeline to produce vector sketches through a differentiable rasterizer optimized by a semantic loss. CLIPasso can only handle the images with a single object, so its follow-up work CLIPascene [44] extends to scene-level images. However, all of these methods depend on input images and do not facilitate the generation of sketches directly from text inputs. Moreover, despite their ability to generate realistic sketches from photos, these methods have a limitation in that they cannot generate new content.
Recent breakthroughs in text-to-image generation have been driven by diffusion models [23, 28, 30, 31] trained on billions of image-text pairs [34]. These models, conditioned on text, now support high-fidelity, diverse, and controllable image synthesis. Nonetheless, the present text-to-image diffusion models cannot produce highly abstract and vectorized free-hand sketches (as shown in Fig. 2). We draw inspiration from the highly effective text-to-image diffusion models and image-conditioned sketch generators to build a bridge between two fundamental forms of human expression, namely text and free-hand sketches, resulting in the development of our proposed text-to-sketch generator.
In this work, we present DiffSketcher, an algorithm that can synthesize novel and high-quality free-hand vector sketches based on natural language text input. DiffSketcher does not require any text-to-sketch training pairs or large datasets of sketches. Instead, with the guidance of a pretrained diffusion model [30], we define a sketch as a set of Bezier curves and optimize the curves' parameters through a differentiable rasterizer [16]. The key idea behind DiffSketcher is to transfer the prior knowledge from text-to-image generation model into the differentiable rasterizer. This allows the optimization of text-guided sketches with semantic consistency, ensuring that the final output sketches are coherent and aligned with their corresponding text inputs. However, it is a non-trivial task to fully leverage the pre-trained diffusion model to efficiently generate high-quality free-hand sketches for both simple objects and complex scenes.
To this end, we propose three strategies to improve the generation quality and efficiency: (1) Present an extended Score Distillation Sampling (SDS) Loss to guide the optimization of curve parameters. Previous works optimize the parameters of the vector sketch with CLIP loss. We found that an extended version of SDS loss can give more diverse sketch synthesis results, and it can be combined with CLIP loss or LPIPS loss, providing an additional control term. (2) Explore attention maps embedded in the diffusion model for effective stroke initialization. The synthesis process can be very time-consuming if the starting point of each stroke is randomly initialized. Therefore, we explore different initialization strategies and present to initialize the curve control points with a fused product of cross-attention maps and self-attention maps in the U-Net [5] of the diffusion model, which significantly improves the efficiency compared to the random initialization. (3) Introduce opacity property during the optimization of Bezier curves to better mimic the style of human sketches, achieving the effect of heavy and light brushstrokes.
"A dragon flying in the sky, full body"
"There is a mug on the plate"
"A detailed portrait of Eiffel Tower, incredibly detailed and realistic, 8k, sharp focus"
"Real photo of sydney opera house"
Figure 2: Various free-hand sketches synthesized by DiffSketcher and the corresponding description prompts. DiffSketcher obtains prior information from LDM [30] composite images through score distillation [24] and achieves the same heavy and light drawing styles as human sketches by performing gradient descent on a set of Bezier curves with the opacity property. Our proposed DiffSketcher allows for varying levels of abstraction while matching its corresponding textual semantics. In each example, given the same text prompt and two different random seeds, two sketches with a different number of strokes are generated. The red words represent the cross-attention index used to initialize the control points (details about cross-attention are covered in Section 4.2).
"A unicorn is running on the grassland"
"Photorealistic painting of a fruit basket with fruit"
"Produce an image of Batman in a battle scene, with smoke and explosions in the background"
"Electronic board style buildings at New York city silhouette"
In summary our contributions are three-fold: (1) We propose a text-to-sketch diffusion model dubbed DiffSketcher. To the best of our knowledge, it is the first diffusion model to generate diverse and high-quality vector sketches with different levels of abstraction at the object and scene levels. (2) We present three strategies to improve the generation quality and efficiency, including an extended SDS loss and an attention-enhanced stroke initialization strategy. The opacity property of strokes is considered during synthesis to improve the visual effect further. (3) We conduct extensive experiments, and the experimental results show that our model outperforms other models in sketch quality and diversity. Thorough insights are provided for future research.
Sketch Synthesis. Free-hand drawings convey abstract concepts through human visual perception with minimal abstraction. Unlike purely edge-map extraction methods [2], free-hand sketching aims to present sketches that are abstract in terms of structure [3] and semantic interpretation[45]. Therefore, computational sketching methods that aim to mimic human drawing consider a wide range of sketch representations, ranging from those grounded in the edge map of the input image[46, 17, 15, 42, 18, 4] to those that are more abstract [9, 8, 1, 7, 26, 45], which are normally in vector format. Among the works synthesizing vector sketches, CLIPasso [45] and CLIPascene [44] are conditioned on an input image, while the rest are unconditional. Until now, no prior work has explored synthesizing a sketch based on text.
Vector Graphics. Our work builds upon the differentiable renderer for vector graphics introduced by Li et al [16]. While image generation methods that operate over vector images traditionally require a vector-based dataset, recent work has shown how the differentiable renderer can be used to bypass this limitation [36, 14, 41, 29, 19]. Furthermore, recent advances in visual text embedding contrastive language-image pre-training (CLIP)[27] have enabled a number of successful methods for synthesizing sketches, such as CLIPDraw [7], StyleCLIPDraw [32], CLIP-CLOP [21], and CliPascene [44]. A very recent work VectorFusion [13] combine differentiable renderer with diffusion
model for vector graphics generation, e.g., iconography and pixel art. Our proposed algorithm, DiffSketcher, shares a similar idea with VectorFusion, but our focus is generating object- and scene-level sketches from a natural language prompt.
Diffusion Models. Denoising diffusion probabilistic models (DDPMs) [37, 39, 11, 40], particularly those conditioned on text, have shown promising results in text-to-image synthesis. For example, Classifier-Free Guidance (CFG) [12] has improved sample quality and has been widely used in large-scale diffusion model frameworks, including GLIDE [23], Stable Diffusion [30], DALL-E 2 [28], and Imagen [31]. However, the majority of images available in web-scale datasets are rasterized, and this work follows the framework of synthesis through optimization, in which images are generated through evaluation-time optimization against a given metric. Our proposed algorithm, DiffSketcher, uses a pre-trained text-to-image diffusion model to synthesize free-hand sketches from natural language input. This is achieved by transferring image synthesis prior information into a differentiable renderer.xc
3.1 Diffusion Models
In this section, we provide a concise overview of diffusion models, which are a class of generative models that utilize latent variables to gradually transform a sample from a noise distribution to a target data distribution [37, 11]. Diffusion models consist of two components: a forward process  $q$  and a reverse process or generative model  $p$ . The forward process, which is typically modeled as a Gaussian distribution, gradually removes structure from the input data  $\mathbf{x}$  by adding noise over time. The reverse process, on the other hand, adds structure to the noise starting from a latent variable  $\mathbf{z}_t$ . Specifically, the generative model is trained to slowly add structure starting from random noise  $p(\mathbf{z}_T) = \mathcal{N}(\mathbf{0},\mathbf{I})$  with transitions  $p_{\phi}(\mathbf{z}_{t - 1}|\mathbf{z}_t)$ .  $q(\mathbf{z}_t|\mathbf{x}) = \mathcal{N}(\alpha_t\mathbf{x},\sigma_t^2\mathbf{I})$ . Training the diffusion model with a (weighted) evidence lower bound (ELBO) simplifies to a weighted denoising score matching objective for parameters  $\phi$  [11]
$$
\mathcal {L} _ {\text {D i f f}} (\phi , \boldsymbol {x}) = \mathbb {E} _ {t \sim \mathcal {U} (0, 1), \epsilon \sim \mathcal {N} (\boldsymbol {0}, \boldsymbol {I})} [ w (t) \| \epsilon_ {\phi} (\alpha_ {t} \boldsymbol {x} + \sigma_ {t} \epsilon ; t) - \epsilon \| _ {2} ^ {2} ] \tag {1}
$$
where  $w(t)$  is a weighting function that depends on the timestep  $t$ . Our work builds on text-to-image latent diffusion model (LDM) that learn  $\epsilon_{\phi}(\boldsymbol{z}_t; t, y)$  conditioned on text embeddings  $y$  [30]. LDM uses classifier-free guidance(CFG) [12], which jointly learns an unconditional model to enable higher quality generation via a guidance scale parameter  $\omega: \hat{\epsilon}_{\phi}(\boldsymbol{z}_t; y, t) = (1 + w)\epsilon_{\phi}(\boldsymbol{z}_t; y, t) - w\epsilon_{\phi}(\boldsymbol{z}_t; t)$  ( $\hat{\epsilon}_{\phi}$  denotes the guided version of the noise prediction). CFG alters the score function to prefer regions where the ratio of the conditional density to the unconditional density is large. In practice, setting  $w > 0$  improves sample fidelity at the cost of diversity.
3.2 Score Distillation Sampling
Many 3D generative approaches use a frozen image-text joint embedding model (e.g. CLIP) and an optimization-based approach to train a Neural Radiance Fields (NeRF) [20]. Such models can be specified as a differentiable image parameterization (DIP) [22], where a differentiable generator  $g$  transforms parameters  $\theta$  to create an image  $x = g(\theta)$ . In DreamFusion [24],  $\theta$  be parameters of a 3D volume and  $g$  is a volumetric renderer. To learn these parameters, DreamFusion proposed score distillation sampling (SDS) loss that can be applied toImagen [31]:
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {S D S}} (\phi , \boldsymbol {x} = g (\theta)) \triangleq \mathbb {E} _ {t, \epsilon} \left[ \omega (t) \left(\hat {\epsilon} _ {\phi} (\boldsymbol {z}; y, t) - \epsilon\right) \frac {\partial \boldsymbol {x}}{\partial \theta} \right] \tag {2}
$$
where the constant  $\alpha_{t}\pmb{I} = \partial \pmb{z}_{t} / \partial \pmb{x}$  is absorbed into  $w(t)$ , and the classifier-free-guided  $\hat{\epsilon}_{\phi}$  is used. In practice, SDS gives access to loss gradients, not a scalar loss. Their proposed SDS loss provides a way to assess the similarity between an image and a caption:
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {S D S}} (\phi , \boldsymbol {x} = g (\theta)) = \nabla_ {\theta} \mathbb {E} _ {t} \left[ \sigma_ {t} / \alpha_ {t} w (t) \mathrm {K L} \left(q \left(\boldsymbol {z} _ {t} | g (\theta); y, t\right) \mid p _ {\phi} \left(\boldsymbol {z} _ {t}; y, t\right)\right) \right] \tag {3}
$$
where  $p_{\phi}$  is the distribution learned by the frozen, pretrained Imagen model.  $q$  is a unimodal Gaussian distribution centered at a learned mean image  $g(\theta)$ . DreamFusion [24] proposed an approach to use a pretrained pixel-space text-to-image diffusion model (Imagen [31]) as a loss function. However,
diffusion models trained on pixels have traditionally been used to sample only pixels. We want to create what look like good sketches that match the text prompts when rendered from a set of vector strokes. Such models can be specified as a differentiable image parameterization, where a differentiable rasterizer  $\mathcal{R}$  transforms parameters  $\theta$  to create a sketch  $S = \mathcal{R}(\theta)$ .
Inspired by DreamFusion [24], we extend score distillation sampling (SDS) loss to use a pretrained latent diffusion model [30] as a prior for optimizing curve parameters. Intuitively, score distillation converts diffusion sampling into an optimization problem that allows the raster image to be represented by a differentiable rasterizer.
3.3 Differentiable Rasterizer
Li et al. [16] introduce a differentiable rasterizer  $\mathcal{R}$  that bridges the vector graphics and raster image domains. A raster image is a 2D grid sampling over the space of the vector graphics scene  $f(x,y;\Theta)$ , where  $\Theta$  contains the curve parameters, e.g., coordinates of Bezier control points, opacity, line thickness. Given a 2D location  $(x,y)\in \mathbb{R}^2$ , they first find all the filled curves and strokes that overlap with the location. Then sort them with a user-specified order and compute the color using alpha blending [25]. It enables the gradients of the rasterized curves (image) to be backpropagated to the curve parameters.
In this section, we present our method for generating sketches using pre-trained text-to-image diffusion models. Let  $S$  be a sketch that was rendered by a differentiable rendering [16]  $\mathcal{R}$  using the text prompt  $\mathcal{P}$ . Our goal is to optimize a set of parametric strokes to automatically generate a vector sketch that matches the description of the text prompt.
Our pipeline is illustrated in Figure 3. Given a text prompt  $y$  of the desired subject and a set of control points for the strokes, we synthesize the corresponding sketch  $S$  while matching the semantic attributes of the text prompt. To initialize the control points of the strokes, we extract and fuse the attention map of the U-Net [5] used by the latent diffusion model [30]. Our key observation is that the structure and appearance of the generated image depend on the interaction between the pixels and the text embedding through the diffusion process [10]. We provide more details on strokes initialization in Section 4.2.
As shown in Figure 4, in each step of the optimization, we feed the stroke parameters to a differentiable rasterizer  $\mathcal{R}$  to produce the raster sketch. We optimize over parameters  $\theta$  such that  $S = \mathcal{R}(\theta)$  is close to a sample from the frozen latent diffusion model [30]. To perform this optimization, we propose a variation of the SDS loss function that enhances the texture of hand-sketched strokes and improves the resilience of drawing styles. This results in the raster sketch always being coherent with the text prompt. The resulting sketch, as well as the sample from the frozen latent diffusion model, then defines a joint semantic and perceptual loss. We back
Figure 3: The overview of the pipeline. DiffSketcher accepts a set of control points (the locations of the strokes) and text prompts as input to generate a hand-drawn sketch.
propagate the loss through the differentiable rasterizer  $\mathcal{R}$  and update the control points and opacity of strokes directly at each step until convergence of the loss function.
4.1 Synthesis Through Optimization
We define a sketch as a set of  $n$  strokes  $\{s_1, \ldots, s_n\}$  with the opacity attribute placed on a white background. To represent each stroke, we use a two-dimensional Bézier curve with four control points  $\mathbf{s}_i = \{p_i^j\}_{j=1}^4 = \{(x_i, y_i)^j\}_{j=1}^4$  (Notice that here,  $x$  and  $y$  represent the coordinates in the canvas) and one opacity attribute  $o_i$ . We incorporate the opacity of the strokes into the optimization process and use DiffSketcher semantics understanding to achieve a human-like heavy and light sketch style. The parameters of the strokes are fed to a differentiable rasterizer  $\mathcal{R}$ , which forms the raster
Figure 4: Optimization overview. To synthesize a sketch that matches the given text prompt, we optimize the parameters of the differentiable rasterizer  $\mathcal{R}$  that produces the raster sketch  $S$ , such that the resulting sketch is close to a sample from the frozen latent diffusion model (the blue part of the picture). Since the diffusion model directly predicts the update direction, we do not need to backpropagate through the diffusion model; the model simply acts like an efficient, frozen critic that predicts image-space edits.
sketch  $\mathcal{S} = \mathcal{R}((s_1,o_1)\ldots ,(s_n,o_n)) = \mathcal{R}((\{p_1^j\}_{j = 1}^4,o_1),\ldots ,(\{p_n^j\}_{j = 1}^4,o_n))$  . For simplicity, we define the parameter in  $\mathcal{R}$  as  $\theta$
4.1.1 Vanilla Version: Fidelity to Generated Image
We start with a two-stage pipeline: First, we sample an image from the latent diffusion model [30] using a text prompt. Next, we optimize the control points to obtain a sketch that is consistent with the text prompt. To preserve the fidelity of the generated sample, we incorporate a Joint Visual Semantic and Perceptual (JVSP) loss to optimize the similarity of the synthesized sketches and the instance sampled from the frozen latent diffusion model [30]. We leverage the VAE [6] decoder  $\mathcal{D}$  to get the RGB pixel representation of  $\hat{S}_{\phi}(z_t|y;t)$ . Then, we jointly use the LPIPS [47] and CLIP visual encoders [27, 45] as depth structural similarity and visual semantic similarity metrics, respectively. Specifically, we use the following loss function:
$$
\mathcal {L} _ {\mathrm {J V S P}} = \mathcal {L} _ {\mathrm {L P I P S}} \left(\mathcal {D} \left(\hat {\mathcal {S}} _ {\phi} \left(\boldsymbol {z} _ {t} | y; t\right), \mathcal {R} (\theta)\right) + \sum_ {l} \left\| \operatorname {C L I P} _ {l} \left(\mathcal {D} \left(\hat {\mathcal {S}} _ {\phi} \left(\boldsymbol {z} _ {t} | y; t\right)\right)\right) - \operatorname {C L I P} _ {l} \left(\mathcal {R} (\theta)\right) \right\| \quad (4) \right.
$$
The  $\mathcal{L}_{\mathrm{JVSP}}$  loss function encourages the synthesized sketch to match the underlying semantics and perceptual details of the image sampled from LDM, leading to more realistic and visually appealing results. While vectorizing a rasterized diffusion sample is lossy, an input Augmentation version of the SDS (ASDS) loss can either finetune the results or optimize random control points from scratch to sample a sketch that is consistent with the text prompt. In the Sec 4.1.2, we introduce the ASDS loss to match the text prompt.
4.1.2 Augmentation SDS Loss: Fidelity to Text Prompt
To synthesize a vector sketch that matches a given text prompt, we directly optimize the parameters  $\theta$  of the differentiable rasterizer  $\mathcal{R}$  that produces the raster sketch  $S$ . We propose an input augmentation version of the SDS loss function to perform this optimization, which encourages plausible images to have low loss and implausible images to have a high loss. Given a raster sketch  $\tilde{S} \in \mathbb{R}^{H \times W \times 3}$ , we combine RandomPerspective, RandomResizedCrop and RandomAdjustSharpness to get a data augmentation version of  $\tilde{\mathbf{S}}_a \in \mathbb{R}^{512 \times 512 \times 3}$ . These transformations preserve artistic styles, enhance style diversity, emphasize crucial features, and improve the model's adaptability to various artistic renditions and size variations commonly found in black-and-white hand-drawn sketches. Then, the LDM uses a VAE encoder [6] to encode  $\tilde{\mathbf{S}}_a$  into a latent representation  $z = \mathcal{E}(\tilde{\mathbf{S}}_a)$ , where  $z \in \mathbb{R}^{(H / f) \times (W / f) \times 4}$  and  $f$  is the encoder downsample factor. In summary, we use the following
Figure 5: Strokes Initialization. The blue part of the figure represents the UNet in the LDM, which has two types of attention mechanisms: self-attention and cross-attention. The yellow and green parts respectively depict the visualization results of the cross-attention and self-attention. The gray part shows how the initial strokes are generated using a fused attention map. The dashed box represents the attention fusion, which is composed of the mean of the self-attention map and the cross-attention map corresponding to the 5-th text prompt token ("Tower"). We start at 1-th token, because 0-th token is taken up by the CLIP starting token.
ASDS loss function:
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {A S D S}} (\phi , \mathcal {S} = \mathcal {R} (\theta)) \triangleq \mathbb {E} _ {t, \epsilon , a} \left[ w (t) \left(\hat {\epsilon} _ {\phi} \left(\boldsymbol {z} _ {t}; y, t\right) - \epsilon\right) \frac {\partial \boldsymbol {z}}{\partial \tilde {\mathcal {S}} _ {a}} \frac {\partial \tilde {\mathcal {S}} _ {a}}{\partial \theta} \right] \tag {5}
$$