SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Metrics. We evaluate the effect of visual text generation on OCR accuracy. We measure OCR exact match accuracy, denoted as Acc, which assesses the word-level agreement between the OCR recognition results and the ground truth visual text. In other words, it represents the complement of the Word Error Rate (WER), i.e., 1 - WER. As the DeepFloyd model tends to generate visual text in all capital letters, regardless of the original form of the words in the prompt, we introduce the OCR capitalization-insensitive exact match accuracy  $\hat{\mathbf{A}}\hat{\mathbf{C}}\mathbf{C}$ . This measure allows for a fairer comparison by disregarding the case of the text. Additionally, we incorporate character-level OCR accuracy by using the Levenshtein distance for partial matching evaluation. We report the average Levenshtein distance LD for each word, providing insights into the accuracy at the character level.
In addition to the OCR-based metrics mentioned earlier, we evaluate the image-text alignment of the generated visual text images using the CLIP score, as done in previous works [3, 31]. To assess the
Method	Stable Diffusion v2.0	SDXL 1.0	DeepFloyd (IF-I-M)	DeepFloyd (IF-I-L)	DeepFloyd (IF-I-XL)	GlyphControl-100K	GlyphControl-1M	GlyphControl-10M
CLIP Score↑	31.6/33.8	31.9/33.3	32.8/34.3	33.1/34.9	33.5/35.2	33.7/36.2	33.4/36.0	33.9/36.2
FID-10K-LAION-Glyph ↓	34.03	44.77	23.37	30.97	26.58	22.04	22.19	22.22
Table 2: Illustrating the CLIP score and FID comparison results based on the settings described in Table 1. The average results of CLIP scores across four word frequency buckets on both benchmarks are provided.
quality of generated images, we calculate FID scores using 10K samples, which have not been used for training in our LAION-Glyph dataset.
Benchmark. We construct two evaluation benchmarks by incorporating prompt templates from previous works on visual text generation [20, 21] and embedding different words selected by us into these templates.
- SimpleBench: A simple text prompt benchmark following [20]. The format of prompts remains the same: 'A sign that says "<word>"'.
- CreativeBench: A creative text prompt benchmark adapted from GlyphDraw [21]. We adopt diverse English-version prompts in the original benchmark and replace the words inside quotes. As an example, the prompt may look like: 'Little panda holding a sign that says "<word>".' or 'A photographer wears a t-shirt with the word "<word>" printed on it.'
In accordance with [20], we collect a pool of single-word candidates from Wikipedia. These words are then categorized into four buckets based on their frequencies: Bucket $_{\text{top}}$ , Bucket $_{\text{1k}}$ , Bucket $_{\text{10k}}$ , and Bucket $_{\text{10k}}$ . Each bucket contains words with frequencies in the respective range. To form input prompts, we randomly select 100 words from each bucket and insert them into the aforementioned templates. Consequently, we generate four images for each word during the evaluation process.
Inference Details. The scale of classifier-free guidance is set as 9 while we take the empty string as the negative prompt. We use the DDIM[35] sampler with 20 sampling steps.
Table 1 presents the comparison of our method with the most representative generative models, including Stable Diffusion, SDXL, and DeepFloyd. Due to a lack of fully open-sourced codes and checkpoints, we could not conduct fair quantitative comparisons with previous works on visual text generation [21, 20]. Our method achieves the highest OCR accuracy on both benchmarks compared to other methods. Notably, the OCR accuracy of Stable Diffusion is almost zero, indicating that it is unable to generate legible text. While for SDXL, an improved version of Stable Diffusion, although the case-insensitive OCR accuracy is improved due to adopting larger text encoders, the overall OCR performance is still poor. In contrast, our framework, with the addition of glyph control, enables the diffusion models to render relatively accurate visual text while maintaining fewer training parameters to the original Stable Diffusion model. Compared to multiple versions of the pixel-level diffusion model DeepFloyd IF with T5 text encoder, our method could achieve better OCR performance with fewer parameters. Additionally, other compared methods tend to generate capital characters regardless of the original word form, resulting in much lower performance on case-sensitive metrics and reduced flexibility in real-world usage.
The comparison among the DeepFloyd IF models demonstrates that increasing the model parameters can enhance the accuracy of generated visual text. Similarly, for our method, training on a larger dataset can improve text generation performance. For example, the OCR accuracy Acc on SimpleBench increased from  $37\%$  to  $48\%$  when trained on a larger specialized visual text-related dataset. This highlights the importance of leveraging larger datasets specifically focused on visual text in order to achieve further improvements in performance.
In addition to the OCR-related evaluation, we also assess the consistency between prompts and generated images, as shown in Table 2. Our method outperforms other general text-to-image models or achieves comparable results in terms of the CLIP scores. This indicates that our model has the capability to accurately render the specified words in the text prompts within the generated images, while still maintaining the fundamental ability to align image and text. While for FID scores
Figure 6: Qualitative comparison results. The left column presents the text prompt and the other five columns show the images generated by Stable Diffusion XL 1.0 (SDXL), DeepFloyd IF-I-XL (IF), Midjourney, Ideogram AI, and our GlyphControl. The results demonstrate that competitive baseline methods exhibit limitations in generating text, including typos and incomplete text generation.
evaluated on the LAION-Glyph dataset, our method achieves the lowest FID score compared to other text-to-image models, implying that GlyphControl could generate realistic visual text images with high quality. Furthermore, we conduct a comparison of the generation performance across different benchmarks and word buckets (see Figure 10).
Through visualization of generated images (shown in Figure 8 & 6), we compare our GlyphControl with both competitive open-sourced text-to-image generation methods (SD, SDXL, and IF) and some leading billing software (DALL-E 2[29], Midjourney [2], and Ideogram [1]).
As depicted in Figure 8, Stable Diffusion, DALL-E 2, SDXL, and DeepFloyd IF exhibit various types of errors during text rendering, including missing glyphs (the 1st sample), repeated or merged glyphs (the 3rd & 6th samples), and misshapen or wrong glyphs (the 5th & 7th samples). In some challenging cases (the 2nd & 4th samples), these models even fail to generate any text at all. In contrast, our method, which incorporates additional glyph images as structure control, could accurately generate legible text that aligns well with the input text prompts by providing appropriate glyph instructions.
We also employ other compared methods to generate images with the same text prompts in Figure 1 (Figure 6). For the cases of rendering longer phrases or sentences (the 2nd & 3rd samples), SDXL, IF, and Midjourney fail to generate complete readable visual text while Ideogram, a recently published AI-based tool specializing in generating visual text, could generate legible text with fewer errors. By comparison, our method outperforms other methods in terms of the accuracy of text rendering. Furthermore, although current top text-to-image tools like Midjourney could generate high-fidelity images, these models including Ideogram still struggle to handle the cases where multiple groups of visual text need to be rendered at various locations within the images (the 1st & 4th samples of Figure 6). While our method could effectively translate such combinations of visual text.
4.5 Ablation Experiments
Ablation on Font Size. GlyphControl supports users to control the font size of the rendered text by modifying the width property of the text bounding box. We further report the generation results of
Font Size	Acc(%)↑	Acc(%)↑	LD ↓	CLIP Score↑
Small	5/4	10/7	4.85/5.51	31.7/33.4
Medium	30 / 19	37/24	1.77 / 2.58	33.7/36.2
Large	23 / 20	27/23	1.94 / 2.37	33.1/35.7
Table 3: Ablations on font sizes. The font size of the rendered text refers to the width of the text bounding box. We adopt the model trained on LAION-Glyph-100K for ablations. The results under the "Medium" setting are exactly the same as those in Table 1 & 2.
various font sizes using GlyphControl (Table 3). For small font sizes, both the OCR accuracy and CLIP score of the generated images substantially decrease, which could be attributed to the relatively smaller area of rendered glyphs within the entire image. While for large font sizes, the generation performance becomes worse probably because of the limited number of such training samples containing large glyphs. In order to attain better generation performance, choosing appropriate font sizes or other glyph instructions would be critical.
Ablation on a Large Amount of Small Text. We further study the scenarios of generating paragraphs, i.e., numerous groups (OCR boxes) of small text, rather than simple phrases or words. As Figure 9 shows, our method could preserve the global arrangement following glyph instructions, but fail to generate readable small-size text within paragraphs. We also conduct an analysis of other failure cases shown in Appendix C.
While our method offers flexible control over the positions and sizes of rendered text using glyph instructions, it currently lacks the ability to control font style and text color. We only render black text in a single default font style onto whiteboard conditional maps for both training and inference. To enhance our approach, we aim to explore advanced font rendering integration, such as artistic fonts, with GlyphControl and investigate modern font style recognizers to extract style information from training data, incorporating it into text captions during training. For controlling font color, incorporating a color spatial palette adapter[23] into the framework may prove effective.
As illustrated in the ablation experiments in Section 4.5, our method exhibits sub-optimal performance when generating a large amount of text with small font sizes. This issue could be attributed to the removal of samples containing more than five OCR boxes during training. In future work, we aim to relax these restrictions on the filtering of the LAION-Glyph dataset, increasing the threshold from five to ten or twenty bounding boxes. It is worth noting that rendering small text presents challenges for VAE-based pixel-level reconstruction due to the text's relatively diminutive size and the dense distribution of text, which may affect both OCR recognition and glyph rendering. Additionally, we aspire to apply the GlyphControl framework to high-resolution text-to-image methods, such as SDXL at  $1024 \times 1024$ , and explore further possibilities for local editing of visual text within generated images. Last, we observed that BLIP-2 captions often encounter difficulties in consistently representing both image content and OCR text information. Investigating more powerful caption models, such as those presented in [19, 18], is a worthwhile endeavor.
In this paper, we introduced the GlyphControl method, a remarkably simple yet highly effective approach for generating legible and well-formed visual text. The success of our approach is attributed to two key contributions: (i) the employment of the glyph ControlNet, which encodes text shape information based on rendered glyph images, and (ii) the establishment of the LAION-Glyph benchmark, benefiting from large-scale training data. By integrating the GlyphControl method with the LAION-Glyph benchmark, our approach achieves exceptional performance and consistently outperforms recent models such as the DeepFloyd IF in terms of OCR accuracy, FID, and CLIP score. Our work serves as a valuable foundation for future research in developing robust visual text generation models. We would like to further explore font style and color control, as well as address challenges related to generating abundant small text and improving the quality of captions.
Acknowledgement. We are grateful to the anonymous reviewers for their invaluable feedback that greatly improved this work. We also thank Bohan Chen, Farzaneh Rajabi, Ji Li, and Chong Luo for their insightful suggestions post-submission.
A More Visualized Examples
We present additional samples generated by GlyphControl in Figure 7 as the supplementary of Figure 1, demonstrating the capability of GlyphControl to generate legible and well-formed visual text in diverse scenarios. Furthermore, Figure 8 provides more qualitative comparison results with competitive text-to-image models. Additionally, Figure 9 illustrates some GlyphControl-generated images that contain plentiful small text. For a detailed analysis of these results, please refer to Section 4.4 & 4.5.
A photo of two monkeys sitting on a tree. They are holding a wooden board that says "Best Friends", 4K dslr.
A photo of a teddy bear holding a surfboard with the text "GlyphControl".
A photo of a golden retriever  
puppy wearing a green shirt with  
text that says "Microsoft Research". Background office. 4k dslr
A close-up 4k dslr photo of a cartoon man riding a scooter with text "MSRA". There are palm trees in the background.
A photo of a plate at a restaurant table with Ma Po Tofu and with text "Ma Po Tofu" on it. photorealistic, dslr.
Three people stand in a line and their backs with text "5", "2", and "0".
Figure 7: More selected samples at  $512 \times 512$  generated by GlyphControl using different text prompts and glyph conditions.
B Comparisons across Benchmarks and Word Frequency
In Figure 10, we demonstrate the OCR accuracy and CLIP scores based on different test benchmarks and word frequency buckets.
Generally, all methods exhibit higher OCR accuracy on the SimpleBench compared to the more challenging CreativeBench, which features more diverse prompts. However, CLIP scores tested on the SimpleBench are slightly lower than those on the CreativeBench, which could be attributed to the richer descriptions present in the prompts of the latter.
Additionally, as shown in Figure 10, words with high-frequency appearances are generally easier to be rendered onto images compared to rarely-used words. An interesting finding is that low-frequency words consistently exhibit higher CLIP scores. This might be attributed to the CLIP embedding's tendency to overlook low-frequency words in the prompts, potentially leading to overestimation.
Figure 8: Qualitative comparison results among Stable Diffusion v2.0 (SD), DALL-E 2, Stable Diffusion XL 1.0 (SDXL), DeepFloyd IF-I-XL (IF), and our GlyphControl.
Che NewHork Cimes.
A New York Times newspaper.
A well-layed-out book.
A well designed menu templates.
Figure 9: Examples generated by GlyphControl in the scenario of a large amount of small text and OCR boxes. The results show that our method achieves relatively weak capability in generating abundant legible and readable small size text although preserving the global arrangement of glyph images.
Figure 10: Comparison of Stable Diffusion (SD), SDXL, DeepFloyd, and our GlyphControl across different benchmarks and frequency buckets.
C Failure Case Analysis
We show some failure cases generated by GlyphControl in Figure 11. The Rendering Overlap problem happens when the locations of text boxes within glyph instructions overlap. Although the text rendering accuracy has improved significantly compared to other text-to-image generation models (Section 4.3 & 4.4), some "layout issues"[20] like Missing Glyphs, Wrong Glyphs, Duplicate Glyphs, Unexpected Text, and Illegible Text still occur, implying the necessity of improvement of the generative model architecture or conditional control scheme. Bad performance in the scenarios of Excessive Yaws and Small Text may be attributed to a lack of corresponding training samples.
avatar the last airbender the promise
health and wellness articles, recipes, and tips
a for sale by owner sign is irish you would be a drink displayed in front of a keychain house
Rendering Overlap
Illegible Text
Excessive Yaws
MissingGlyphs
bruce & co limited logo
WrongGlyphs
nate certified technician excellence
Unexpected Text
Figure 11: Failure cases generated by GlyphControl using captions and glyph instructions in the LAION-Glyph test dataset. We demonstrate 8 types of errors here. Captions and glyph images are also shown here. And we also mark out the places where the errors occur with red rectangles on the corresponding glyph images.
i will create 500 backlinks for your website
pool party gift set with a bottle opener, bottle opener, and a bottle opener
Small Text
DuplicateGlyphs
Training Dataset	U-Net Decoder	Acc(%)↑	Acc(%)↑	LD ↓	CLIP Score↑
TextCaps 5K	frozen	48/39	55/43	1.21/1.73	33.8/36.3
fine-tuning	61/43	68/49	0.76/1.38	34.2/36.3
(a) Illustrating the effect of unlocking the U-Net decoder: we conducted two experiments, in both cases, the models were fine-tuned for an additional 40 epochs using the checkpoint trained on LAION-Glyph-100K as the starting point (shown in Table 1). During the fine-tuning process, the learning rate of the decoder was set to 1e-4.
Training Dataset	Training Epochs	Acc(%)↑	Acc(%)↑	LD ↓	CLIP Score↑
TextCaps 5K	pre-trained	30/19	37/24	1.77/2.58	33.7/36.2
10	48/28	56/34	1.31/2.32	33.8/35.5
20	50/34	61/41	1.03/2.01	34.3/35.7
40	61/43	68/49	0.76/1.38	34.2/36.3
(b) The comparison between different training epochs. The U-Net decoder is also fine-tuned.
Table 4: Ablation experiments on TextCaps 5K. We report the average results on two benchmarks.
To assess the generalization capability of our approach on different training datasets, we curate a specialized OCR-related training dataset called TextCaps 5K. This dataset consists of images related to signs, books, and posters, which are extracted from the training set of the TextCaps v0.1 Dataset [33]. The original purpose of this dataset was image captioning with an understanding of visual text content. We fine-tune our model on the TextCaps 5K dataset for additional 40 epochs, using the same training settings as those applied to the model trained on LAION-Glyph-100K (as shown in Table 1). This fine-tuning process aims to evaluate how well our model performs on a different training dataset and further validate its robustness and adaptability.
Through our experiments, we discover that unlocking the frozen U-Net decoder in the original Stable Diffusion model can significantly enhance the OCR accuracy of visual text rendering, as demonstrated in Table 4a. This improvement can be attributed to the decoder's improved adaptation to the smaller TextCaps 5K dataset during training. As training progresses, the accuracy of text generation gradually improves, as shown in Table 4b. However, it is worth noting that the generated images tend to resemble the samples in TextCaps 5K.
Figure 12: Illustrating the visual text images generated using the models fine-tuned on TextCaps 5K. The models were trained with the same settings following Table 4b.
As depicted in Figure 12, the visual text regions in the center of the generated images appear more like conventional signs (Figure 12d), without seamlessly merging with the background, while the images generated by the model pre-trained on LAION-Glyph-100K exhibit greater creativity and diversity (Figure 12a). Therefore, in order to generate photo-realistic images that accurately render visual text, it is essential to not only have a training dataset with a large number of high-quality samples containing abundant visual text content, but also to include more realistic images with diverse scenes and creative elements.