SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To assess the authenticity of the synthesized sketches, we conducted a perceptual study. Specifically, We gathered a total of 90 synthesized sketches using three different methods (30 samples per method) and obtained 30 real sketches from Google Image by searching for "free-hand sketch". Figure 13 shows a partial sample. We then mixed the real and fake sketches and distributed questionnaires to 41 participants. The participants were asked to determine whether each sketch was drawn by a human or not, without any knowledge of its source. We utilized the confusion score as the evaluation metric, where a higher score indicates a greater likelihood of the generated sketches being recognized as real. The results are presented in Table 1. It is clear to see that our method produced sketches that were more frequently identified as real, highlighting the superior quality of our synthesized sketches.
E Stroke Initialization.
The highly non-convex nature of the ASDS loss and JVSP loss function makes the optimization process susceptible to stroke initialization, especially for generating scene-level sketches with multi-instances. To address this issue, we explore different stroke initialization strategies (as mentioned in Section 5.3 and Fig. 8) and evaluate their impact on the performance of our model.
As shown in Figure. 14, we compare three different stroke initialization methods: random initialization, initialization based on the CLIP saliency map [45], and initialization based on our proposed fusion attention (ours). The experimental results demonstrate that our proposed strategy, initialization based on fusion attention, outperforms the other methods in terms of visual quality and synthesis efficiency. This initialization method can utilize the joint semantic and structural information from the
Figure 14: Comparison of the (intermediate) results when using different stroke initialization strategies. From top to bottom: (a) random initialization, (b) initialization based on the CLIP saliency map, (c) our proposed fusion attention.
input text and image (i.e., LDM results) to guide the stroke placement, resulting in more semantically meaningful and artistically expressive sketches. However, using a CLIP saliency map for initialization leads to a sketch with only salient objects while ignoring the background. Random initialization takes more iterations than ours to synthesize visually pleasing results.
F Visualization of Sketching Process.
In this section, we show the trace of 300 iters of sketching. By visualizing the intermediate outputs during the generation process, we can gain insights into how our model sketches an object. Specifically, as shown in Fig. 15, we can observe how the strokes are placed and refined over time to gradually form the desired sketch.
Figure 15: The intermediate results throughout the optimization process.
G Effect of the Stroke Width.
In Fig. 16, we compare the results with different stroke widths given the same text prompt. In our implementation, we use a fixed stroke width for all strokes. Such a design is to simplify the optimization, making it computationally more efficient and less prone to overfitting. It can easily extend to include stroke width as a parameter for optimization, like [7, 32].
Figure 16: Different widths of the curves. The width increases from left to right.
Strokes
Two sparrows perched on the branches
A banana on the table, and there is a glass of water beside it
32
Figure 17: The failure cases.
H Evaluation Metrics.
Evaluating text-to-sketch synthesis is challenging due to the absence of ground truth sketches. As we mentioned in Section 5.2 and Fig. 6, we evaluate the models from three aspects: semantic consistency between the generated sketch and text prompt, the aesthetic quality of the sketch, and the recognizability of the sketch.
Semantic Consistency Between the Generated Sketch and Text Prompt. To measure the semantic consistency, namely the CLIP score [30, 7], we calculate the cosine similarity of CLIP ViT-L-14 embeddings of the generated sketches and corresponding input text prompts. Our method achieves a cosine similarity of 0.3494, which is higher than Canny [2] algorithm (0.328) and CLIPasso [45] (0.3075).
The Aesthetic Quality of the Sketch. To measure the aesthetic quality of generated sketches, we adopt the CLIP-based aesthetic indicator [33]. This indicator [33] consists of a CLIP ViT-L-14 backbone and a multi-layer perception (MLP), which is pre-trained on LAION [34] data. Figure 5 of the main paper compares the aesthetic score of several examples generated by different methods. Sketches generated by our method obtain the highest scores.
The Recognizability of the Sketch. Finally, to measure the recognizability of the generated sketches, we use the CLIP ViT-L-14 model [27] for zero-shot classification. Specifically, we first generate sketches for 34 categories<sup>2</sup>. Then, we use the CLIP model to classify these sketches. In Fig. 6 of the main paper, we list the probabilities of the sketch being categorized into different classes. Since a Canny edge can preserve the object's contours and fine details, it achieves high recognizability. Compare our DiffSketcher with CLIPasso, our DiffSketcher can draw a more complete sketch. Therefore, our sketches are easier for CLIP to recognize.
2The 34 categories include "astronaut", "vessel", "observatory", "needle", "outer space", "earth", "iron man", "batman", "apple", "sailboat", "ship", "bunny", "castle", "cabin", "inn", "bike", "cat", "dog", "dragon", "snake", "horse", "fruit basket", "Sydney opera house", "lamp", "lighthouse", "mug", "desk", "macaw", "mountain", "river", "eiffel tower", "unicorn", "yoda", "skyscraper".
I Failure Cases
As shown in Fig. 17, our approach mainly has two limitations. Specifically, one limitation is the lack of correlation between the text prompt and sketch abstractness. For instance, if the text prompt describes multiple objects but the number of strokes is set too small, the resulting sketches may be unsatisfactory. A possible solution is to establish a link between the complexity of the text prompt (such as the number of described objects) and the number of strokes to be used.