SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yukang Yang $^{1\text{串}\dagger}$  Dongnan Gui $^{2\dagger}$  Yuhui Yuan $^{3\dagger\dagger}$
Weicong Liang $^{3}$ ; Haisong Ding $^{3}$ ; Han Hu $^{3}$ ; Kai Chen $^{3}$
<sup>1</sup>Princeton University
https://github.com/AIGText/GlyphControl-release
Newspaper with the headline "Aliens Found in Space" and "Monster Attacks Mars".
A menu of a fast food restaurant that contains "Sandwich Combo", "French Fries", and "Pepsi".
A decorative greeting card that reads "Congratulations on achieving state of the art".
A sign in front of a beautiful village that says "Bear Infested Be Careful".
Dslr portrait of a robot holds a sign that says "StrongAI will Empower The World".
A sign "OpenSource" facing another sign "CloseSource". They point to two completely different paths.
Figure 1: Illustrating selected  $512 \times 512$  GlyphControl samples for different text prompts and glyph conditions. Our GlyphControl can generate coherent images with well-formed visual text.
Recently, there has been an increasing interest in developing diffusion-based text-to-image generative models capable of generating coherent and well-formed visual text. In this paper, we propose a novel and efficient approach called GlyphControl to address this task. Unlike existing methods that rely on character-aware text
†Core contribution. ‡ Interns at Microsoft Research Asia
$^{\ddagger}$ Corresponding author: yuhui.yuan@microsoft.com
encoders like ByT5 and require retraining of text-to-image models, our approach leverages additional glyph conditional information to enhance the performance of the off-the-shelf Stable-Diffusion model in generating accurate visual text. By incorporating glyph instructions, users can customize the content, location, and size of the generated text according to their specific requirements. To facilitate further research in visual text generation, we construct a training benchmark dataset called LAION-Glyph. We evaluate the effectiveness of our approach by measuring OCR-based metrics, CLIP score, and FID of the generated visual text. Our empirical evaluations demonstrate that GlyphControl outperforms the recent DeepFloyd IF approach in terms of OCR accuracy, CLIP score, and FID, highlighting the efficacy of our method.
Denoising diffusion probabilistic models [34, 5, 35, 29, 31, 30, 16] have significantly boosted the development of general text-to-image generation by showing the capability of generating surprisingly high-quality images over the past few years. Although currently plenty of diffusion-based text-to-image generation methods could produce abundant fantastic and photo-realistic images, most existing methods still lack the ability to produce legible and readable text in generated images [29, 30] due to the complex and fine-grained structure within the visual text.
Several very recent efforts have made preliminary attempts to address the task of visual text generation. Motivated by the inspiring analysis in unCLIP [29], the spelling information inside the prompts can not be accurately modeled with the raw CLIP text embedding, the follow-up efforts including eDiff-I [3] and Imagen [31] attempt to leverage the potential of large language models such as T5 [28], which is trained on the text-only corpus, as the text encoder in image generation. With the strength of T5 embedding on encoding individual objects within the prompts [3], eDiff-I produces more accurate visual text. The very recent DeepFloyd IF model further follows the design of Imagen and demonstrates impressive performance in rendering legible text. Besides, [20] found that the text encoders (both CLIP and T5) used in most existing mainstream text-to-image generation models lack sufficient character-level information for spelling due to the usage of BPE tokenizer, thus they verify that adopting the character-aware model ByT5 [36] instead could bring significant improvements.
Despite efforts to modify text encoders used in generation models, layout errors such as missing or merged glyphs still exist in generated images [20], implying that merely relying on textual input prompts would not be sufficient for accurate visual text rendering. To address this problem, we propose to incorporate text glyph information into the off-the-shelf powerful text-to-image generation models for visual text generation. We formulate the task of visual text generation as a glyph-conditional control problem. Specifically, we propose to control the visual text generation with an additional glyph image. The glyph image acts as an explicit spatial layout prior to enforcing the diffusion models generating coherent and well-formed visual text. We show some qualitative results in Figure 1 to show that our method is capable of generating diverse images with well-formed visual text.
In our implementation, we introduce two key innovations including (i) a GlyphControl framework that can augment the off-the-shelf text-to-image generation model by exploiting the shape information encoded in the glyph image with a ControlNet branch, and (ii) a LAION-Glyph benchmark that consists of  $\sim 10$  M text-image pairs augmented with additional OCR detection results that record the presented text information. We further create two evaluation benchmarks, including SimpleBench and CreativeBench, to assess the performance of our method and the other strong methods such as DeepFloyd IF. To demonstrate the effectiveness of our approach, we conduct thorough experiments and show that our approach consistently achieves much higher OCR accuracy than the DeepFloyd IF. For example, on SimpleBench and CreativeBench, our approach gains  $+15\%$  (48% vs. 33%) and  $+13\%$  (34% vs. 21%) than the very recent powerful DeepFloyd (IF-I-XL) while only requiring less than 22% parameters. We summarize our main contributions as follows:
3The glyph image is a whiteboard image where the characters are rendered with a single particular font while keeping the same content, position, and size as the realistic visual text.
- We propose a glyph-conditional text-to-image generation model named GlyphControl for visual text generation, which outperforms DeepFloyd IF, SDXL, and Stable Diffusion in terms of OCR accuracy, CLIP score, and FID while saving the number of parameters by more than  $3 \times$  compared to DeepFloyd IF-I-XL.
- We introduce a visual text generation benchmark named LAION-Glyph by filtering the LAION-2B-en and selecting the images with rich visual text content by using the modern OCR system. We conduct experiments on three different dataset scales: LAION-Glyph100K, LAION-Glyph-1M, and LAION-Glyph-10M.
- We report flexible and customized visual text generation results. We empirically show that the users can control the content, locations, and sizes of generated visual text through the interface of glyph instructions.
Text-to-image Diffusion Models. Denoising Diffusion Probabilistic Model [11] and its successors [24, 29, 3, 30, 31, 27] have demonstrated impressive performance on high-quality image synthesis with text prompts. GLIDE [24] emphasizes the necessity of the classifier-free guidance over CLIP guidance and the usage of cascaded diffusion models [12, 31, 3, 29] for high-fidelity, high-resolution generation. Imagen [31] introduces generic large language models (T5-XXL text encoder) into the text-to-image generation while demonstrating comparable or superior image quality to the CLIP text encoder. Moreover, eDiff-I [3] concatenates the CLIP text embeddings and T5 text embeddings to benefit from the strengths of both two text encoders. Unlike the aforementioned pixel-level diffusion models, Latent Diffusion [30] transforms the image into latent features and applies the diffusion model in the latent space to decrease training and inference costs. Stable Diffusion is an application of the Latent Diffusion method in text-to-image generation but trained with additional data and a powerful CLIP text encoder. SDXL[27] improves Stable Diffusion by using  $3 \times$  larger U-Net and a second text encoder and also introducing another refinement model to further enhance image quality through image-to-image techniques. In this work, we adopt Stable Diffusion as the base model.
Controllable Image Generation. To achieve more customized image synthesis, users could apply additional conditions, such as segmentation maps or depth maps [30], onto diffusion models. Beyond this intuitive approach, multiple diffusion-based methods of image editing [22, 15, 25, 8] demonstrate promising performance in controlling the content of synthesized images. Recently, more related works [37, 23, 14] have focused on flexible and composable control of image synthesis.Composer [14] decomposes the image generation into multiple factors and generates images by re-combining them. Both T2IAdapter [23] and ControlNet [37] can incorporate different conditional maps, such as segmentation maps or depth maps, as additional data into the pre-trained diffusion models, demonstrating accurate structure or color control without dampening the generation ability of the original models. Considering the fact that the glyphs of visual text essentially belong to geometric structures, we adopt ControlNet as the basic framework to generate visual text by controlling the local structure with additional glyphs.
Visual Text Generation. Although diffusion models could generate high-fidelity images, current mainstream text-to-image generation models such as unCLIP [29] and Stable Diffusion have trouble rendering legible and readable text onto images. Several previous works [9, 10] demonstrate that diffusion models have the ability to generate visual text with different fonts but do not extend to general image generation. Due to the findings that CLIP embedding could not precisely perceive the spelling information in the input prompts [29, 3], both Imagen [31] and eDiff-I [3] utilize the large language model T5 [28] to achieve superior visual text generation. The recent open-sourced image generation model DeepFloyd IF [16], inspired by Imagen, takes the T5-XXL as the text encoder as well demonstrating impressive performance in visual text generation. Furthermore, [20] thoroughly exploits the strengths of character-aware language models like ByT5 [36] over character-blind counterparts such as mainly used CLIP and T5. With the usage of ByT5 in the generation, the semantic errors of rendered text decrease while the errors related to the layout of glyphs still exist, implying that the auxiliary information about glyph images would be necessary. The very recent DALL-E 3 [26, 4] not only excels in rendering precise visual text but also showcases exceptional typography quality. GlyphDraw [21] successfully renders Chinese characters onto images by adding glyph images into the input of the diffusion model and also fusing extracted glyph embedding with
(a) GlyphControl.
(b) Training pipeline.
(c) Inference pipeline.
Figure 2: Illustrating the framework of GlyphControl. (a) The GlyphControl architecture comprises a pretrained Stable Diffusion model as a "locked copy" and a randomly initialized ControlNet model as a "trainable copy." (b) During the training process, the input image  $x$  undergoes encoding with a VAE encoder, resulting in a latent embedding  $z_0$ . The diffusion process is then applied to  $z_0$ , generating a noised latent embedding  $z_t$ . Additionally, we utilize an OCR engine (PP-OCR [7]) to extract text from images and employ a glyph render to generate a whiteboard image. This image exclusively represents recognized characters as black regions, forming the glyph image  $g$ . Consequently, both the text embedding (based on text caption  $c$ ) and the noised latent embedding are fed into the U-Net (locked copy) and the Glyph ControlNet (trainable copy). This enables the estimation of the noise term  $\varepsilon(z_t, t)$ , with the crucial step involving passing the glyph image to the Glyph ControlNet to extract vital glyph information for rendering well-formed text. (c) During inference, our method supports diverse user instructions for customizing the rendering of the glyph image  $g$ . Subsequently, we sample a noise latent embedding  $z_T$  from Gaussian noise and employ the DDIM scheme to perform the denoising process, estimating the denoised latent embedding  $z_0$ . Finally,  $z_0$  is sent to the VAE decoder, resulting in the construction of the final output image  $y$ .
text embedding as a condition. Based on similar insights, we utilize glyph images as conditional maps to control image synthesis. Compared to the above methods, we could specify the contents, locations, and sizes of text, which brings more customized and flexible designs.
Figure 3: Illustrating of the generated glyph images based on the glyph render (LAION-Glyph-1M).
Stable Diffusion [30]. We have selected the "stable-diffusion-2-base" (SD 2.0-base $^4$ ) as the foundational model in this work. The Stable Diffusion model is a highly capable and versatile text-to-image generative model that has been meticulously trained from scratch. The training process of basic
<sup>4</sup>https://huggingface.co/stabilityai/stable-diffusion-2-base
A hand-painted wooden "Free Beer" sign hanging out of a bar.
A fancy violet T-shirt decorated with flowers while the message  $[X]$  are written on it.
Figure 4: Illustrating the qualitative results of GlyphControl in terms of flexible user controllability. The whiteboard images depict the corresponding glyph condition maps alongside the generated images on the right side. The "[X]" symbol (in the last example) is used as a replacement for the rendered words on the T-shirt.
models involves 550k steps at resolution  $256 \times 256$ , focusing on a subset, with an aesthetic score of 4.5 or higher, of the LAION-5B dataset. What makes the difference between stable-diffusion-2-base and previous versions is that the model is continuously trained on the same dataset with a resolution of at least  $512 \times 512$  pixels, which contributes to the model's ability to generate more detailed and visually appealing images. The training of stable-diffusion-2-base costs hundreds of hours with  $128 \times \mathrm{A}100$  GPUs. In this work, by employing the off-the-shelf "stable-diffusion-2-base" model and refining it through rigorous training processes, we aim to achieve superior results in the visual text generation domain.
ControlNet [37]. The ControlNet is a powerful network that enhances pre-trained large diffusion models like Stable Diffusion with additional input conditions. It learns task-specific conditions in an end-to-end manner, even with limited training data. The network has a trainable copy and a locked copy of the diffusion model's weights, enabling it to retain general capabilities while fine-tuning for specific tasks. The ControlNet incorporates "zero convolution" layers to connect the trainable and locked components, gradually adapting convolution weights from zeros to optimized parameters. This allows precise control over the model's behavior in various applications.
3.2 GlyphControl
**Framework.** The GlyphControl framework consists of several key components: (i) an OCR engine for detecting text information in the given image, (ii) a Glyph render for rendering the detected text in a whiteboard image at corresponding locations, (iii) an image VAE encoder that projects the input image into a latent code, and an image VAE decoder that reconstructs an output image based on the latent code, (iv) a text encoder (OpenAI CLIP text encoder) that converts the input text into text embedding, (v) a U-Net encoder and decoder that performs the denoising diffusion process, and (vi) a Glyph ControlNet that encodes the conditional glyph information by processing the glyph image rendered by the Glyph render. More details of the GlyphControl framework can be seen in Figure 2. Furthermore, Figure 3 showcases some example images of the rendered glyph images.
To incorporate glyph information, we introduce the concept of glyph input conditions by rendering glyph images and feeding them into the ControlNet branch. Unlike conventional conditions used in the original ControlNet [37], accurate visual text rendering greatly benefits from the use of rendered glyph images. We specifically chose the ControlNet architecture for its proficiency in controlling precise geometric structures.
With our GlyphControl approach, we can successfully generate legible and readable visual text. This is achieved by utilizing pre-rendered glyph images as input condition maps for the ControlNet, allowing us to control the generated glyphs at the layout level. Furthermore, we specify the words in the input text prompts (e.g., "A storefront with "GlyphControl" written on it") and leverage the CLIP text encoder to understand the semantic meaning of the words.
Figure 5: Statistics on LAION-Glyph-10M. Left: Distribution of character counts in each image. Middle: Distribution of word counts in each image. Right: Distribution of detected bounding boxes in each image.
Glyph Instructions. One major advantage of our GlyphControl approach is its ability to support customized glyph instructions  $(i)$ , which enables the specification of various constraints on the rendered text in the final output image. Our GlyphControl framework provides support for three types of text information customization:
Text character information: GlyphControl allows for the specification of not only single words but also phrases or sentences composed of multiple words. As long as the text is intended to be placed within the same area, users can customize the text accordingly.
Text line information: GlyphControl provides the flexibility to assign words to multiple lines by adjusting the number of rows. This feature enhances the visual effects and allows for more versatile text arrangements.
- Text box information: With GlyphControl, users have control over the font size of the rendered text by modifying the width property of the text bounding box. The location of the text on the image can be specified using the coordinates property of the top left corner. Additionally, the yaw rotation angle property of the text box allows for further adjustments. By default, the text is rendered following the optimal width-height ratio, but users can define a specific width-height ratio to precisely control the height of the text box.
We demonstrate the effectiveness of these glyph instructions in Figure 4, where our approach successfully generates legible text according to the specified instructions. For instance, in Figure 4, we showcase examples where users can customize the positions of the rendered text, adjust the font size, or place multiple groups of text at different locations to achieve personalized designs. Additionally, users have the option to split the text into multiple rows or rotate the text boxes for improved arrangement. Our controllable text generation approach opens up possibilities for automated personalized art designs in the future. Moreover, in the experimental section, we provide empirical evidence showcasing that our method achieves significantly higher OCR accuracy compared to the recent DeepFloyd model.
Implementation. We utilize the same architecture and initial weights for the VAE and U-Net as the SD 2.0-base model and adopt the classifier-free guidance[13] as the previous works do. Our training process incorporates PP-OCRv3 [6] as the OCR engine. During inference, users need to provide glyph instructions to generate customized images. For rendering glyphs, we leverage the tools available in the ImageDraw module of the Python library Pillow.
3.3 LAION-Glyph Benchmark
Overview. The training process of Stable Diffusion and DeepFloyd has greatly benefited from the utilization of the extensive multi-modal dataset LAION-5B [32]. However, there is currently a notable absence of openly available datasets specifically tailored for visual text generation tasks. To bridge this gap, we introduce the LAION-Glyph benchmark. To construct this benchmark, we start with LAION-2B-en, which is a subset of LAION-5B [32], and selectively choose specimens that exhibit abundant visual text content using the PP-OCR engine.
Pipeline. Our data construction process consists of two consecutive steps. In the first step, we apply an aesthetic score prediction model to filter out images with an aesthetic score higher than 4.5. Next, we utilize the PP-OCRv3 [6] engine for text detection and recognition. To ensure the quality
Method	#Params	Text Encoder	Training Dataset	Acc(%)↑	Acc(%)↑	LD↓
Stable Diffusion v2.0	865M	CLIP(354M)	LAION 1.2B	0/0	3/2	4.25/5.01
SDXL 1.0	5.8B	CLIP & OpenCLIP(817M)	Internal Dataset (>100M)	0.3/0.5	13/8	6.26/6.30
DeepFloyd (IF-I-M)	2.1B	T5-XXL(4.8B)	LAION 1.2B	0.3/0.1	18/11	2.44/3.86
DeepFloyd (IF-I-L)	2.6B	T5-XXL(4.8B)	LAION 1.2B	0.3/0.7	26/17	1.97/3.37
DeepFloyd (IF-I-XL)	6.0B	T5-XXL(4.8B)	LAION 1.2B	0.6/1	33/21	1.63/3.09
GlyphControl	1.3B	CLIP(354M)	LAION-Glyph-100K	30/19	37/24	1.77/2.58
GlyphControl	1.3B	CLIP(354M)	LAION-Glyph-1M	40/26	45/30	1.59/2.47
GlyphControl	1.3B	CLIP(354M)	LAION-Glyph-10M	42/28	48/34	1.43/2.40
Table 1: Comparison results of OCR-related metrics with prior methods in the field of visual text generation is shown in the table. The results are averaged over four word-frequency buckets. The results on SimpleBench/CreativeBench are presented on the left/right side of the slash, respectively. It is important to note that the total number of parameters reported in the second column of the table does not include the text encoder. The LAION 1.2B dataset comprises image-text pairs with predicted aesthetic scores of 4.5 or higher in LAION 5B. All the DeepFloyd models use IF-II-L (1.2B) and Stable  $\times 4$  as the upscale models to progressively increase the image resolutions from  $64\times 64$  to  $1024\times 1024$ . SDXL generates images with a resolution of  $1024\times 1024$ .
of the data, we discard images where all OCR boxes are located at the image border. Additionally, we remove images that have total OCR areas less than  $5\%$  of the whole image area or contain more than 5 bounding boxes, as these cases may lead to text recognition or image reconstruction failures. To address inaccuracies in the original captions from the LAION dataset, we generate new captions using the BLIP-2 [17] model. As a result, we have curated a high-quality LAION-Glyph dataset consisting of 10 million images. This dataset includes detailed OCR information and captions that are well-formed and accurate.
Statistics. As illustrated in Figure 5, the character count in the images is primarily concentrated within the range of 10 to 50 characters, with the majority of samples containing fewer than 150 characters. In terms of word distribution, the most common cases consist of 3 to 5 words, while instances with more than 15 words are relatively rare. Additionally, the number of bounding boxes is fairly evenly distributed, although images with only one box are less prevalent. To facilitate training and evaluation, we partitioned the LAION-Glyph dataset into three scales: LAION-Glyph-100K, LAION-Glyph-1M, and LAION-Glyph-10M, using a random division approach.
4.1 Training Details
We train our framework on three different dataset scales: LAION-Glyph-100K, LAION-Glyph-1M, and LAION-Glyph-10M for  $60 \times$  epochs,  $20 \times$  epochs, and  $6 \times$  epochs, respectively. The initial weights of both the SD branch and Glyph ControlNet branch are copied from the SD 2.0-base model. For both the Glyph ControlNet and Zero-Conv blocks, we set the base learning rate to  $1\mathrm{e} - 4$ . The U-Net encoder and decoder are both kept frozen during training. The caption dropping rates for the SD branch and Glyph ControlNet branch are set to 0.1 and 0.5, respectively. The input images are maintained at a resolution of  $512 \times 512$ .