GlyphControl: Glyph Conditional Control for Visual Text Generation
Authors: Yukang Yang
Dongnan Gui
Yuhui Yuan
Weicong Liang
Haisong Ding
Han Hu
Kai Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative Models
Visual Text Generation
Diffusion Models

Abstract: Recently, there has been an increasing interest in developing diffusion-based text-to-image generative models capable of generating coherent and well-formed visual text. In this paper, we propose a novel and efficient approach called GlyphControl to address this task. Unlike existing methods that rely on character-aware text encoders like ByT5 and require retraining of text-to-image models, our approach leverages additional glyph conditional information to enhance the performance of the off-the-shelf Stable-Diffusion model in generating accurate visual text. By incorporating glyph instructions, users can customize the content, location, and size of the generated text according to their specific requirements. To facilitate further research in visual text generation, we construct a training benchmark dataset called LAION-Glyph. We evaluate the effectiveness of our approach by measuring OCR-based metrics, CLIP score, and FID of the generated visual text. Our empirical evaluations demonstrate that GlyphControl outperforms the recent DeepFloyd IF approach in terms of OCR accuracy, CLIP score, and FID, highlighting the efficacy of our method.

GlyphControl: Glyph Conditional Control for Visual Text Generation
Yukang Yang Ibf Weicong Liang
Dongnan Gui
Yuhui Yuan 3t1
Haisong Ding Han Hu
Kai Chen
Princeton University
University of Science and Technology of China
Microsoft Research Asia
Aliens Found in Space
Congratulations on achieving state of the art
StrongAl will Empower The World
Monster Attacks Mars
Newspaper with the headline "Aliens Found in Spuce" and "Monster Attacks Mars".
decorative greeting card that reads "Congratulations achieving state of the art".
Dsir portrait of a robot holds sign that says "StrongAl will Empower The World".
Menu
Sandwich Combo
Bear Infested Be Careful
French Fries 44
CloseSource
Pepsi
OpenSource
A sign "OpenSource" " facing another sign "CloseSource". They point completely different paths.
menu of a fast food restaurant that contains "Sandwich Combo", "French Fries", and "Pepsi".
sign front of beautifid village that says "Bear Infested Be Careful".
Figure 1: Illustrating selected 512 x 512 GlyphControl samples for diffferent text prompts and glyph conditions. Our GlyphControl can generate coherent images with well-formed visual text.
Abstract
Recently, there has been increasing interest in developing diffusion-based text- to- co-image generative models capable of generating coherent and well-formed visual text. In this paper, we propose novel and efficient approach called GlyphControl to address this task. Unlike existing methods that rely on character-award text
Core contribution. Interns at Microsoft Research Asia Coorespooning author: yuhui, @microsoft.com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

encoders like ByT5 and require retraining of text-to-image models, our approach leverages additional glyph conditional information to enhance the performance of the off-the-shelff Stable-Diffusion model in generating accurate visual text. By incorporating glyph instructions, users can customize the content, location, and size of the generated text according to their specific requirements. To facilitate further research in visual text generation, we construct a training benchmark dataset called LAION-Glyph. We evaluate the effectiveness of our approach by measuring OCR- based metrics, CLIP score, and FID of the generated visual text. Our empirical evaluations demonstrate that GlyphControl outperforms the recent DeepFloyd IF approach in terms of OCR accuracy, CLIP score, and FID, highlighting the efficacy of our method.
1 Introduction
16 have significantly boosted the development of general text-to-image generation by showing the capability of generating surprisingly high-quality images over the past few years. Although currently plenty of diffusion based text-to- image generation methods could produce abundant fantastic and photo-realistic images. most existing
Denoising diffusion probabilistic models 34 5
35
29
31
30
methods still lack the ability to produce legible and readable text in generated images 29 30 the complex and fine-grained structure within the visual text.
due to
Several very recent efforts have made preliminary attempts to address the task of visual text generation. Motivated by the inspiring analysis in unCLIP [29], the spelling information inside the prompts can not be accurately modeled with the raw CLIP text embedding, the follow-up efforts including eDiff-I 3] and Imagen [1 attempt to leverage the potential of large language models such T5 (28.. which is trained on the text-only corpus, as the text encoder in image generation. With the strength of T5 embedding on encoding individual objects within the prompts B], eDiff-I produces more accurate visual text. The very recent DeepFloyd IF model further follows the design of Imagen and demonstrates impressive performance in rendering legible text. Besides, 20) found that the text encoders (both CLIP and T5) used in most existing mainstream text-to-image generation models lack sufficient character-level information for spelling due the usage of BPE tokenizer, thus they verify that adopting the Character-wware model ByT5 36) instead could bring significant improvements.
Despite efforts to modify text encoders used in generation models, layout errors such as missing or merged glyphs still exist in generated images (20), implying that merely relying on textual input prompts would not be sufficient for accurate visual text rendering. To address this problem, propose to incorporate text glyph information into the off-the-shelf powerful text-to-image generation models for visual text generation. We formulate the task of visual text generation as glyph- conditional control problem. Specifically, we propose to control the visual text generation with an additional glyph image(] The glyph image acts as explicit spatial layout prior to enforcing the diffusion models generating coherent and well-formed visual text. We show some qualitative results in Figure T to show that our method is capable of generating diverse images with well-formed visual text.
In our implementation, we introduce two key innovations including (i) GlyphControl framework that can augment the off-the-shelf text-to-image generation model by exploiting the shape information encoded in the glyph image with a ControlNet branch, and (ii) LAION- Glyph benchmark that consists of ~ 10 M text-image pairs augmented with additional OCR detection results that record the presented text information. We further create evaluation benchmarks, including SimpleBench and CreativeBench, to assess the performance of our method and the other strong methods such as DeepFloyd IF. To demonstrate the effectiveness of our approach, conduct thorough experiments and show that our approach consistently achieves much higher OCR accuracy than the DecpFloyd IF. For example, on SinpheBench and CreativeBench, our approach gains +15% (48% vs. 33%) and +13% (34% vs. 21%) than the very recent powerful DeepFloyd (IF-I-XL) while only requiring less than 22% parameters. We summarize our main contributions as follows:
'The glyph image is a whiteboard image where the characters are rendered with single particular font while keeping the same content, position, and size the realistic visual text.

---

We propose a glyph- conditionn text-to-image generation model named GlyphControl for visual text generation, which outperforms DeepFloyd IF, SDXL, and Stable Diffusion in terms of OCR accuracy, CLIP score, and FID while saving the number of parameters by more than 3x compared DeepFloyd IF-I-XI..
We introduce visual text generation benchmark named LAyoN-Glyph by filtering the LAION-2B-en and selecting the images with rich visual text content by using the modern OCR system. We conduct experiments on three different dataset scales: LAION-Glyph- 100K, LAION-GIyph--N MM, and LAION-GGlyhh-O0M
We report flexible and customized visual text generation results. We empirically show that the users can control the content, locations, and sizes of generated visual text through the interface of glyph instructions.
2 Related Work
Text-to-image Diffusion Models. Denoising Diffusson Probabilistic Model [T] and its successors
31 27 have demonstrated impressive performance on high-quality image synthesis with text prompts. GLIDE [24 emphasizes the necessity of the Mlssifier-free guidance over CLIP guidance and the usage of cascaded diffusion models [2 31 3 29 for high-fidelity, high-resolution generation. Imagen [3] introduces generic large language models (T5-XXL text encoder) into the texi-to-image generation while demonstrating comparable or superior image quality to the CLIP text encoder. Moreover, eDiff-1 3 concatenates the CLIP text embeddings and TS text embeddings to benefit from the strengths of both two text encoders. Unlike the aforementioned pixel-level diffusion models, Latent Diffusion [30] transforms the image into latent features and applies the diffusion model in the latent space to decrease training and inference costs. Stable Diffusion is an application of the Latent Diffusion method in text-to-image generation but trained with additional data and powerful CLIP text encoder. SDXL[27] improves Stable Diffusion by using 3x larger U-Net and a second text encoder and also introducing another refinement model to further enhance image quality through image-to-image techniques. In this work, we adopt Stable Diffusion as the base model.
24 29
3
30
Controllable Image Generation. To achieve more customized image synthesis, users could apply additional conditions, such as segmentation maps or depth maps (30), onto diffusion models. Beyond this intuitive approach, multiple diffusion-basse based methods of image editing [22 [5] 25 g demonstrate promising performance in controlling the content of synthesized images. Recently, more related works have focused flexible and composable control of image synthesis. Composer [14] decomposes the image generation into multiple factors and generates images by re-combining them. Both T21Adapter (23 and ControlNet 377 incorporate different conditional maps, such as segmentation maps or depth maps, as additional data into the pre-trained diffusion models, demonstrating accurate structure color control without dampening the generation ability of the original models. Considering the fact that the glyphs of visual text essentially belong to geometric structures, we adopt ControlNet as the basic framework to generate visual text by controlling the local structure with additional glyphs.
33 23 [4]
Visual Text Generation. Although diffusion models could generate high-fidelity images, current mainstream text-to-image generation models such as unCLIP (29 and Stable Diffusion have trouble rendering legible and readable text onto images. Several previous works (9 10 demonstrate that diffusion models have the ability to generate visual text with different fonts but do not extend to general image generation. Due the findings that CLIP embedding could not precisely perceive the spelling information in the input prompts [29 3 both Imagen [31 and eDiff-I [ utilize the
large language model T5 28 achieve superior visual text generation. The recent open-sourced image generation model DeepFloyd IF (16). inspired by Imagen, takes the T5-XXL as the text encoder as well demonstrating impressive performance in visual text generation. Furthermore, [20] thoroughly exploits the strengths of character aware language models like ByT5 (36] over character- blind counterparts such mainly used CLIP and T5. With the usage of ByT5 in the generation, the semantic errors of rendered text decrease while the emrors related to the layout of glyphs still exist, implying that the auxiliary information about glyph images would necessary. The very recent DALL-E 3 [26 4 not only excels in rendering precise visual text but also showcases exceptional typography quality. GlyphDraw (1 successfully renders Chinese characters onto images by adding glyph images into the input of the diffusion model and also fusing extracted glyph embedding with
3

---

Locked
Text
Copy
the side
buildin
Encodar
Difusiio
Stable Diffusion
U-Net Encoder
U-Net Decodar
Encoder
Zam Zem..com Comi
Zero Corv
Zeno-Oonv Cony
Zg
(F6,
Trainable
Copy
OCR Engine
Glyah Render
Glyph ControlNet
Zero-Conv
ControlNet
(a) GlyphControl.
(b) Training pipeline.
aechnologyy builinn
Text
with that
Encoder
PNNTONTOOO DOY
Glyph Instructions
W
U-Net Encoder
U-Net Decoder
VAE Decoder
Text characfer
DDIM slape
5g
Text line info.
Glyph Render
Glyph ControiNet
Conv
Text bounding box info.
(c) Inference pipeline.
Figure 2: Illustrating the framework GlyphControl. (a) GlyphControl architecture comprises a pre- trained Stable Diffusion model as a "locked copy" and a randomly initialized ControlNet model as a "trainable copy." (b) During the training process, the input image æ undergoes encoding with VAE encoder, resulting in latent emhedding The diffusion process is then applied to 2o, generating noised latent embedding Ze- Additionally, we utilize OCR engine (PP-OCR 7 extract from images and employ a glyph render generate a whiteboard image. This image exclusively represents recognized characters as black regions. forming the glyph image g. Consequently, both the text embedding (based on text caption c) and the noised latent embedding are fed into the U-Net (locked copy) and the Glyph ControlNet (trainable copy). This enables the estimation of the noise term E(2,, t), with the crucial step involving passing the glyph image to the Glyph Cot extract vital glyph information for rendering well-fore well-formed text. (c) During inference, our method supports diverse user instructions for customizing the rendering of the glyph image g. Subsequently, we sample a noise latent embedding zy from Gaussian noise and employ the DDIM scheme to perform the denoising process, estimating the denoised latent embedding 2o. Finally, is sent to the VAE decoder, resulting in the construction of the final output image y.
text embedding as condition. Based on similar insights, utilize glyph images conditional maps to control image synthesis. Compared to the above methods, we could specify the contents, locations, and sizes of text, which brings more customized and flexible designs.
Approach
3.1 Preliminary
Beddpar
FRIAD
TOOM PAR
RESTAURANTI
PORTER
LEAN
Figure 3: Illustrating of the generated glyph images based on the glyph render (LAION-GIyph-IM))
We have selected the Sisole-diffsionn---ae (SD 2.0-basd as the founda- tional model in this work. The Stable Diffusion model is highly capable and versatile text-to-image generative model that has been meticulously trained from scratch. The training process of basic
Stable Diffusion 330]
basa

---

Hello World
Mallo World
CaphConera
GoyphContral
GlyphControl
GlyphControl
Hello World
A portnait of a parrol holding sign with text "Hello World".
A storefront with "GlyphControl" written on centered.
Pasien Poons
Free Boer
Boor
Bear
A hand-painted wooden "Free Beer" sign hanging out of bar.
A fancy violet T-shirt decorated with flowers while the message [X] are written it.
Figure 4: Illustrating the qualitative results of GlyphControl in terms of flexible user controllability. The whiteboard images depict the corresponding glyph condition maps alongside the generated images the right side. The "[X]" symbol (in the last example) used replacement for the rendered words on the T-shirt.
models involves 550k steps resolution 256 256, focusing subset, with aesthetic of 4.5 or higher, of the LAION-5B dataset. What makes the difference between sable-diffusion-2-baal and previous versions is that the model is continuously trained on the same dataset with a resolution of at least 512 x 512 pixels, which contributes to the model's l ability to generate more detailed and visually appealing images. The training of stable-diffusiion22bii 128x A100 GPUs. In this work, by employing the off off-the-shelf -table-iiffusinn-2aaa refining through rigorous training processes, we aim achieve superior results in the visual text generation domain.
costs hundreds of hours with
model and
ControlNet [37]. The ControlNet is a powerful network that enhances pre-trained large diffusion models like Stable Diffusion with additional input conditions. learns task-specific conditions an end-to-end manner, even with limited training data. The network has trainable copy and locked copy of the diffusion model's weights, enabling it to retain general capabilities while fine-tuning for specific tasks. The ControlNet incorporates "zero convolution" layers connect the trainable and locked components, gradually adapting convolution weights from zeros to optimized parameters. This allows precise control over the model's behavior in various applications.
3.2 GlyphControl
Framework. The GlyphControl framework consists of several key components: (i) OCR engine for detecting text information in the given image, (ii) a Glyph render for rendering the detected text in whiteboard image at corresponding locations, (iii) an image VAE encoder that projects the input image into a latent code, and an image VAE decoder that reconstructs an output image based on the latent code, (iv) a text encoder (OpenAl CLIP text encoder) that converts the input text into text embedding, (v) U-Net encoder and decoder that performs the denoising diffusion process, and (vi) Glyph ControlNet that encodes the conditional glyph information by processing the glyph image rendered by the Glyph render. More details of the GlyphControl framework can be seen in Figure 2 example images of the rendered glyph images.
Furthermore, Figure
showcases 3
To incorporate glyph information, we introduce the concept of glyph input conditions by rendering glyph images and feeding them into the ControlNet branch. Unlike conventional conditions used in the original ControlNet (37], accurate visual text rendering greatly benefits from the use of rendered glyph images. We specifically chose the ControlNet architecture for its proficiency in controlling precise geometric structures.
With our GlyphControl approach, we can successfully generate legible and readable visual text. This is achieved by utilizing pre-rendered glyph images as input condition maps for the ControlNet, allowing us to control the generated glyphs at the layout level. Furthermore, we specify the words in the input text prompts (e-g., "A storefront with "GlyphControl" written on it") and leverage the CL.P text encoder to understand the semantic meaning of the words.
5

---

0144
837
24.09
0.55
144
ablooes 18.2
624
22
Figure 5: Statistics on LAION-GIyph--OMM. Left: Distribution of character counts in each image. Middle: Distribution word counts each image. Right: Distribution of detected bounding boxes each image.
Glyph Instructions. One major advantage of our GlyphControl approach is its ability to support customized glyph instructions (i), which enables the specification of various constraints on the rendered text in the final output image. Our GlyphControl framework provides support for three types of text information customization:
Text character information: GlyphControl allows for the specification of not only single words but also phrases or sentences composed of multiple words. As long as the text is intended to be placed within the same area, users can customize the text accordingly.
Text line information: GlyphControl provides the flexibility to assign words to multiple lines by adjusting the number of rows. This feature enhances the visual effects and allows for more versatile text arrangements.
With GlyphControl, users have control over the font size of the rendered text by modifying the width property of the text bounding box. The location of the text on the image can be specified using the coordinates property of the top left corner. Additionally, the yaw rotation angle property of the text box allows for further adjustments. By default, the text is rendered following the optimal width-height ratio, but users can define a specific width haith- ratio to precisely control the height of the text box.
Text box information:
We demonstrate the effectiveness of these glyph instructions in Figure where our approach successfully generates legible text according to the specified instructions. For instance, in Figure 4 showcase examples where users font size, or place multiple groups of text at different locations to achieve personalized designs. Additionally, users have the option to split the text into multiple rows rotate the text boxes for improved arrangement. Our controllable text generation approach opens up possibilities for automated personalized art designs in the future. Moreover, in the experimental section, we provide empirical evidence showcasing that our method achieves significantly higher OCR accuracy compared to the recent DeepFloyd model.
eustomize the positions of the rendered text, adjust the
Implementation. We utilize the same architecture and initial weights for the VAE and U-Net as the SD 2.0-base model and adopt the classifier-fi free guidance [3] as the previous works do. Our training process incorporates PP-OCRv3 (6] as the OCR engine. During inference, users need to provide glyph instructions to generate customized images. For rendering glyphs, we leverage the tools available in the IrageDraw module of the Python library Pillow.
3.3 LAION-Glyph Benchmark
The training process of Stable Diffusion and DeepFloyd has greatly benefited from the utilization of the extensive multi-modal dataset LAION-5B [32]. However, there currently a notable absence of openly available datasets specifically tailored for visual text generation tasks. To bridge this gap, we introduce the LAION-Glyph benchmark. To construct this benchmark, we start with LAION-2B-en, which is a subset of LAION-5B (32), and selectively choose specimens that exhibit abundant visual text content using the PP-OCR engine.
Overview.
Pipeline. Our data construction process consists of two consecutive steps. In the first step, we apply an aesthetic score prediction model filter out images with an aesthetic score higher than 4.5. Next, we utilize the PP-OCRv3 (6) engine for text detection and recognition. To ensure the quality

---

Method
#Params
Encoder
Training Dataset
Acc(Sy)
Acc(S))
Stable Diffusion v2.0
865M
CLIPGGIM)
LAION 1.2B
0/0
4.25/5.01
SDXL 1.0
5.8B
CLIP & OpenCLIP(S17M)
Internal Dalaset (>-00M))
0.3/0.5
13/8
6.26/6.30
DeepPloyd (IF-4-M)
13-XXL(118B)
LAION 1.28
0.3/0.1
2.44/3886
DeepFllyyd (IF-1-1.)
2.6B
T5-XXI((48BB
LAION 1.2R
0.3/0.7
26/17
1.97/3.77
DecpFloyd (IF-I-XL)
6.0B
T5-XXL(188)
LAION 1.2B
0.6/1
33/11
1.63/3.09
GlyphControl
1.3B
CLIRG5IM)
LAION-Glypb-1006
30/19
37/21
1.77/2.58
GlyphControl
1.3B
CLIR3GIM)
LAION-Glyph-IM
40/26
45/30
1.59/2.4T A7
GlyphControl
CLIPG54M)
LAION-Glypb-100
1.43/2.40
Table 1: Comparison results of OCR-related metrics with prior methods in the field of visual text gen- eration shown in the table. The results averaged over four word-frequene, buckets. The results on SimpleBenchuCreatiieeeeee
are presented on left/right side of the slash, respectively, is important note that the total number of parameters reported in the second column of the table does not include the text encoder. The LAION 1.2B dataset comprises image-text pairs with predicted aesthetic scores of 4.5 or higher in LAION 5B. All the DeepFloyd models use IF-II-L (1.2B) and Stable x4 as the upscale models progressively increase the image resolutions from 64 64 1024 1024. SDXL generates images with a resolution of 1024 x 1024.
of the data, we discard images where all OCR boxes are located at the image border. Additionally, remove images that have total OCR areas less than 5% of the whole image than bounding boxes, as these cases may lead to text recognition or image reconstruction failures. To address inaccuracies in the original captions from the LAION dataset, generate new captions using the BLIP-2 [17] model. As a result, we have curated a high-quality LAION-Glyph dataset consisting of 10 million images. This dataset includes detailed OCR information and captions that are well-formed and accurate.
contain more
Statistics. As illustrated in Figure 5] the character count in the images is primarily concentrated within the range of 10 to 50 characters, with the majority of samples containing fewer than 150 characters. In terms of word distribution, the most common cases consist of to words, while instances with more than 15 words are relatively rare. Additionally, the number of bounding boxes is fairly evenly distributed, although images with only one box less prevalent. To facilitate training and evaluation, we partitioned the LAION-Glyph dataset into three scales: LAION-Glyph-100K, LAION-GIyph-IMM and LAION-Glyph- 10M, using random division approach.
4 Experiment
4.1
Training Details
We train framework three different dataset scales: LAION-GLyph- LAION-GGyph-IM and LLAION-Glyph- 10M for 60x epochs, 20x epochs, and 6x epochs, respectively. The initial weights of both the SD branch and Glyph ControlNet branch copied from the SD 2.0-base model. For both the Glyph ControlNet and Zero-Conv blocks, we set the base learning rate to le-4. The U-Net encoder and decoder are both kept frozen during training. The caption dropping rates for the SD branch and Glyph ControlNet branch are set to 0.1 and 0.5,, respectively. The input images are maintained resolution of 512 x 512.
4.2 Evaluation
Metrics. We evaluate the effect of visual text generation on OCR accuracy. We measure OCR exact match accuracy, denoted as Acc, which assesses the word-level agreement between the OCR recognition results and the ground truth visual text. other words, represents the complement of the Word Error Rate (WER), i.c., 1- WER. As the DeepFloyd model tends to generate visual text in all capital letters, regardless of the original form of the words in the prompt, we introduce the OCR exact match accuracy Acc. This measure allows for a fairer comparison incorporate character-level OCR accuracy by
by disregarding the case of the text. Additionally, using the Levenshtein distance for partial matching evaluation. We report the average Levenshtein distance LD for each word, providing insights into the accuracy at the character level.
In addition to the OCR-based metrics mentioned earlier, we evaluate the image-text alignment of the generated visual text images using the CLIP score, as done in previous works [3 31 To assess the
7

---

Maioo
Sosble Diffision v2. MDXI.
DeupPPoogl (IF-I-M) DeopPloga (IF-Ja) (IF-44)) DespFlegd (IF--XI
CLIP Scoret
81.6633.8
31,9/30.3
32.8/34.3
38.1/14.9
39.5415-2
38.423600
33.9/36.2
MA-.IK--LIONNGOE Glyph.
$4.03
44.77
20.37
30.97
26.68
22.04
22.19
Table Illustrating the CL.IP score and FTD comparison results based on the settings described in Table The average results of CLIP scores across four word frequency buckets on hoth benchmarks are provided.
quality of generated images, we calculate FID scores using 10K samples, which have not been used for training in our LAyoh-Glyph dataset.
Benchmark. We construct two evaluation benchmarks by incorporating prompt templates from previous works on visual text generation [20] 21 and embedding different words selected by us into these templates.
SimpleBench: A simple text prompt benchmark following 120 remains the same: 'A sign that says "<word>"!
The format of prompts
CreativeBench:
A creative text prompt benchmark adapted from GlyphDraw 21 We
adopt diverse English- English-version prompts in the original benchmark and replace the words inside quotes. As an example, the prompt may look like: 'Little panda holding sign that says "<word>", ".' or 'A photographer wears a t-shirt with the word "<word> printed on it.
In accordance with 200 we collect a pool of single-word candidates from Wikipedia. These words are 100k then categorized into four buckets based on their frequencies: Bucket Ik Bucket 10k Bucket 10k
Ik
and Bucket plus
100k Each bucket contains words with frequencies in the respective range. To form input prompts, we randomly select 100 words from each bucket and insert them into the aforementioned templates. Consequently, we generate four images for each word during the evaluation process.
The scale of classifier-free guidance is set as 9 while take the empty string as the negative prompt. We use the DDIM (35) sampler with 20 sampling steps.
Inference Details.
4.3 Main Results
Table presents the comparison of our method with the most representative generative models. including Stable Diffusion, SDXL,, and DeepFloyd. Due to a lack of fully open-sourced codes and checkpoints, we could not conduct fair quantitative comparisons with previous works on visual text generation [0). Our method achieves the highest OCR accuracy on both benchmarks compared to other methods. Notably, the OCR accuracy of Stable Diffusion is almost zero, indicating that is unable to generate legible text. While for SDXL, improved version of Stable Diffusion, although the case-insensitive OCR accuracy is improved due to adopting larger text encoders, the overall OCR framework, with the addition of glyph control, enables the
performance is still poor. In contrast, diffusion models to render relatively accurate visual text while maintaining fewer training parameters to the original Stable Diffusion model. Compared to multiple versions of the pixel-level diffusion model DeepFloyd IF with T5 text encoder, our method could achieve better OCR performance with fewer parameters. Additionally, other compared methods tend to generate capital characters regardless the original word form, resulting in much lower performance on case-sensitive metrics and reduced flexibility in real-world usage.
The comparison among the DeepFloyd models demonstrates that increasing the model param- eters can enhance the accuracy of generated visual text. Similarly, for our method, training on larger dataset can improve text generation performance. For example, the OCR accuracy Acc SimpleBench increased from 37% to 48% when trained on a larger specialized visual text-related dataset. This highlights the importance of leveraging larger datasets specifically focused on visual text in order to achieve further improvements in performance.
In addition to the OCR-related evaluation, we also assess the consistency between prompts and generated images, as shown in Table Our method outperforms other general text-to-image models or achieves comparable results in terms of the CLIP scores. This indicates that our model has the capability to accurately render the specified words in the text prompts within the generated images, while still maintaining the fundamental ability to align image and text. While for FID scores
8

---

SDXL
(b) IF
(c)
Midjourney
Ideogram
(e) Ours
Nevspaper with the hradline "Allens Found in Space" and "Monster Attacks Mars".
SPACCE THE NEWISTARS SPACE
wispaer
ALIENS
AIN EELL BESS TAAS OLSTAMBLIEN
ALIEN' FOOND IN SPPACEE MONStra arrern
Allens Found in Space
NLS MAM MR
AL ALINSISSA
Moneler
Space
A deconative goveting that roads "Con- grauuations on achieving shate
Congntulation en achisping state the art
Gong Congeniitiin on Saarerg
STATE OF THE ART
Dalr portrait sign that says
StrongAI will Empawer
"StongAA will Empower World"
STRONGAI WILL
WILL EMPOWED
SFOORT
EMPOWER THE WORLD
ENOWAL
A menu of fust fond that contains "Sandwich
Menu
Pepoi
Seechalhh Coober
Gumbo"
"French Fries", and "Pyya""
Figure 6: Qualitative comparison results. The left column presents the text prompt and the other five columns show the images generated by Stable Diffusion XI. 1.0 (SDXL., DeepFloyd IF-I-XI. (IF), Midjourney, Ideogram AI, and our GlyphControl. The results demonstrate that competitive baseline methods exhibit limitations in generating text, including typos and incomplete text generation.
evaluated on the LAION-Glyph dataset, our method achieves the lowest FID score compared to other text-to-image models, implying that GlyphControl could generate realistic visual text images with high quality. Furthermore, we conduct comparison of the generation performance across different benchmarks and word buckets (see Figure I0p.
4.4 Qualitative Analysis
Through visualization of generated images (shown in Figure g & 6), we compare our GlyphControl with both competitive open-sourced text-to-image generation methods (SD, SDXL, and IF) and some leading billing software (DALL-E 2[2] Midjourney (2). and Ideogram [I).
As depicted in Figure X Stable Diffusion, DALL-E 2, SDXL, and DeepFloyd IF exhibit various types of errors during text rendering, including missing glyphs (the 1st sample), repeated or merged glyphs (the 3rd & 6th samples), and misshapen or wrong glyphs (the 5th & 7th samples). In some challenging cases (the 2nd 4th samples), these models even fail to generate any text at all. contrast, our method, which incorporates additional glyph images structure control, could accurately generate legible text that aligns well with the input text prompts by providing appropriate glyph instructions.
We also employ other compared methods to generate images with the same text prompts in Figure (Figure For the cases of rendering longer phrases or sentences (the 2nd & 3rd samples), SDXL. IF and Midjourney fail to generate complete readable visual text while Ideogram, a recently published Al-based tool specializing in generating visual text, could generate legible text with fewer errors. By comparison, our method outperforms other methods in terms of the accuracy of text rendering. Furthermore, although current top text-to-image tools like Midjourney could generate high-fidelity images, these models including Ideogram still struggle to handle the cases where multiple groups of visual text need to be rendered at various locations within the images (the 1st & 4th samples of Figure C). While our method could effectively translate such combinations of visual text.
4.5 Ablation Experiments
Ablation on Font Size. GlyphControl supports users to control the font size of the rendered text by modifying the width property of the text bounding box. We further report the generation results of

---

Font Size
Acc(
Acc(%))
LD 4
CLIP Scuret
Small
5/4
10/7
4.85/5.51
31.7/33.4
Medium
30/ 19
37/24
1.77/2.58
33. 33.7/36.2
Large
23 20
27/23
1.94 2.37
33.1/35.7
Table Ablations on font sizes. The font size the rendered text refers to width of the text bounding box. We adopt the model trained on LATON-GGlph-100 for ablations. The results under the "Medium" setting are exactly the same as those Table &
various font sizes using GlyphControl (Table 3). For small font sizes, both the OCR accuracy and CLIP score of the generated images substantially decrease, which could attributed to the relatively smaller area of rendered glyphs within the entire image. While for large font sizes, the generation performance becomes worse probably because of the limited number of such training samples containing large glyphs. In order to attain better generation performance, choosing appropriate font sizes or other glyph instructions would be critical.
Ablation Large Amount of Small Text. We further study the scenarios of generating paragraphs, ie.. numerous groups (OCR boxes) of small text, rather than simple phrases or words. As Figure Saerrs,sssww,, our method could preserve the global arrangement following glyph instructions, but fail to generate readable small-size text within paragraphs. We also conduct an analysis of other failure cases shown in Appendix C
5 Discussion and Limitations
While our method offers flexible control over the positions and sizes of rendered text using glyph instructions, it currently lacks the ability control font style and text color. We only render black text in a single default font style onto whiteboard conditional maps for both training and inference. To enhance approach, aim explore advanced font rendering integration, such artistic fonts, with GlyphControl and investigate modern font style recognizers to extract style information from training data, incorporating it into captions during training. For controlling font color, incorporating a color spatial palette adapter 23 into the framework may prove effective.
As illustrated in the ablation experiments in Section 4.5 our method exhibits sub-optimal performance when generating a large amount of text with small font sizes. This issue could be attributed to the removal of samples containing more than five OCR boxes during training. In future work, aim to relax these restrictions on the filtering of the LAION-Glyph dataset, increasing the threshold from five to ten twenty bounding boxes. It is worth noting that rendering small text presents challenges for VAE-based pixel-level reconstruction due to the text's 's relatively diminutive size and the dense distribution of text, which may affect both OCR recognition and glyph rendering. Additionally, we aspire to apply the GlyphControl framework to high-resolution text-to-image methods, such SDXL at 1024 1024, and explore further possibilities for local editing of visual text within generated images. Last, we observed that BLIP-2 captions often encounter difficulties in consistently representing both image content and OCR text information. Investigating more powerful caption models, such as those presented in
19 18
is a worthwhile endeavor.
6 Conclusion
In this paper, we introduced the GlyphControl method, remarkably simple yet highly effective approach for generating legible and well-formed visual text. The success of our approach is attributed to two key contributions: (i) the employment of the glyph ControlNet, which encodes text shape information based on rendered glyph images, and (ii) the establishment of the LAION-Glyph benchmark, benefiting from large-scale training data. By integrating the GlyphControl method with the LAION-Glyph benchmark, our approach achieves exceptional performance and consistently outperforms recent models such as the DeepFloyd IF in terms of OCR accuracy, FID, and CLIP score. Our work serves valuable foundation for future research in developing robust visual text generation models. We would like to further explore font style and color control, as well address challenges related to generating abundant small text and improving the quality of captions.
10

---

Acknowledgement. We are grateful to the anonymous reviewers for their invaluable feedback that greatly improved this work. We also thank Bohan Chen, Farzaneh Rajabi, Ji Li, and Chong Luo for their insightful suggestions post-submission.
References
[1] Ideogram. https://deogram.ai 2023. Accessed: 2023-10-10.
[2] Midjourney v5.2. https//www.mdjouumey.coom
2023. Accessed: 2023-10-10.
[3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0133
2022.
[4] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang. Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions. 2023,
[5] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794 2021.
[6] Yuning Du, Chenxia Li, Ruoyu Guo, Cheng Cui, Weiwei Liu, Jun Zhou, Bin Lu, Yehua Yang, Qiwen Liu, Xiaoguang Hu, et al. Pp-ocrv2: Bag of tricks for ultra lightweight ocr system. arXiv preprint arXiv:2 031444 2021.
[7] Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin
Yu, Yehua Yang. Qingqing Dang, et al. Pp-ocr: A practical ultra lightweight ocr system. arXiv preprint arXiv:2009.099: 09941, 2020.
[8] Rinon Gal. Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2022.
[9] Dongnan Gui, Kai Chen, Haisong Ding, and Qiang Huo. Zero-shot generation of training data with denoising diffusion probabilistic model for handwritten chinese character recognition. Xrviv preprint arXiv:2305. 15660, 2023.
[10] Haibin He, Xinyuan Chen, Chaoyue Wang, Juhua Liu, Bo Du, Dacheng Tao, and Yu Qiao. Diff- font: Diffusion model for robust one-shot font generation. arXiv preprint arXiv:2212.05895, 2022.
[11] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[12] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-228) 2022.
[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop
on Deep Generative Models and Downstream Applications, 2021.
[14] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou, Composer: Cre- ative and controllable image synthesis with composable conditions. In International Conference on Machine Learning. 2023.
[15] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017,
2023.
[16] DeepFloyd Lab. Deepfloyd if.
2023.
[17] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 0rXiv:2301.1255/) 2023.
[18] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuming. arXiv preprint arXiv:2310.03744, 2023.
[19] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, arXiv preprint arXiv:2304. 08488, 2023.
[20] Rosanne Liu, Daniel H Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, R. J. Mical, Mohammad Norouzi, and Noah Constant, Character-aware models improve visual text rendering. In Annual Meeting of the Association for Computational Linguistics, 2022.
11

---

[21] Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang, Di Niu, Haonan Lu, and Xiaodong Lin. Glyphdraw: Learning draw chinese characters in image synthesis models coherently. arXiv
preprint arXiv:2303. 17870, 2023.
[22] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[23] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. arXiv preprini arXiv:2302.008 0.445.0 2023.
[24] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image genera- tion and editing with text-guided diffusion models. In International Conference on Machine Learning, pages 16804. PMLR, 2022.
[25] Yaniv Nikankin, Niv Haim, and Michal Irani. Sinfusion: Training diffusion models on single image or video. In International Conference on Machine Learning, 2022.
[26] OpenAI. Dall-e 3 system card. 2023.
[27]
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint urXiv:2307.0199:
2023.
[28] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5511 2020. [29] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.066 06/5 2022.
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684- 10695, 2022.
[31]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic eoxt-to-image diffusion models with deep language understanding, Advances in Neural Information Processing Systems, 45:36479-36494 2022.
[32] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. L00on-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 (td:::2/11.02111 1.02114, 2021.
[33] Oleksii Sidorov, Ronghang Hu, Mareus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 742-758. Springer, 2020.
[34] Jascha Sohk-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[35] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020.
[36] Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou. Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306, 2022. [37] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to- -mage diffusion models. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
12

---

A More Visualized Examples
We present additional samples generated by GlyphControl in Figure 7 as the supplementary of Figure I demonstrating the capability of GlyphControl to generate legible and well-formed visual text in diverse scenarios. Furthermore, Figure 8 provides more qualitative comparison results with competitive text-to-image models. Additionally, Figure(Jillustraae some GlyphControl- -enerrttd images that contain plentiful small text. For a detailed analysis of these results, please refer to Section
Best Friends
GlyphControl
Microsoft Research
photo of a golden retriever puppy wearing a green shirt with text that says "Microsoft Research". Background office. 4k
A photo of two monkeys sitting tree. They are holding a wooden board that says "Best Friends", ", 4K dslr.
photo of a teddy bear holding surfboard with the text "GlyphControl".
dslr
Ma Po Tofu
MSRA
2
A close-up 4k dsir photo of cartoon man riding a scooter with text "MSRA". There are palm trees in the background.
photo of plate restaurant table with Ma Po Tofu and with text "Ma Po Tofu" on it. photorealistic, dslr.
Three people stand in a line and their backs with text "5", "2", and "0".
Figure 7: More selected samples at 512 x 512 generated by GlyphControl using different text prompts and glyph conditions.
B Comparisons across Benchmarks and Word Frequency
In Figure T0 we demonstrate the OCR accuracy and CLIP scores based on different test benchmarks and word frequency buckets.
Generally, all methods exhibit higher OCR accuracy on the SimpleBench compared to the more challenging CreativeBench, which features more diverse prompts. However, CLIP scores tested on the SimpleBench are slightly lower than those on the CreativeBench, which could be attributed to the richer descriptions present in the prompts of the latter.
Additionally, as shown in in Figure 10 words with high-frequency appearances are generally easier to be rendered onto images compared to rarely-used words. An interesting finding is that low-frequency words consistently exhibit higher CLIP scores. This might be attributed to the CL.IP embedding tendency to overlook low- cow-frequency words in the prompts, potentially leading to overestimation.
13

---

(a) SD
(b) DALL-E 2
(c) SDXL
(d) IF
(e) Ours
Just For Fanm
A blie bantl with "Jurt For
JUST FUN
Fun
JUST TRIENA FU
JUST FURT
FOR
A cloth embbrol- dered with the
Lalon
"laion" and
Ln Lon
embroidered cute baby tion fave.
LATTON
OPEEN
OPPEN
OPEN
Tiso teoliccal intioring chest, left with "Opon" right witk "Clowe"
3
OPEEN
CLOSE
Cose
photo erming wake for named "Playing" 4K.
Playing
dalr
squierel holding a sign that says "Please Protect Enstrommen"
FUSOR
ENVVTTEA
Please Protect Environment
Please Profect
Preitule Erdse
A photo of "Mutery" the boginning of mystic forest.
WA'S TEY
MISTERY
MFTSTRS BEGEKIG
MYYSTER
Mystery
China Town
A photo of street with rexr "China Toun ot
CHINIALLIN
Toun
handlaa made
Hello World
of feathers with "Helio engraved
Warad
plutto of wane and gor- gold ring with wond "Love,,
LOVE
Love
Figure 8: Qualitative comparison results among Stable Diffusion v2.0 (SD), DALL-E 2, Stable Diffusion XL 1.0 (SDXL.), DeepPloyd IF-I-XI. (IF), and our GlyphControl.
14

---

Che NewHork Cimes.
FGRS
PANCARES SOTEEA
WAFFLES
COFFTE
MORE
The York
4 Menou
FGSS
PANCALES
WAFFLES
COFFFE
ILUCE
MORE
A well designed menu templates.
A New York Times newspaper.
A well-layed-out book.
Figure 9: Examples generated by GlyphControl in the scenario of a large amount of small text and OCR boxes. The results show that our method achieves relatively weak capability in generating abundant legible and readable small size text although preserving the global arrangement of glyph images.
Backet
Bycket
Siepleeech
F22
Cretledeecs
Decir
Bucke
0.1
0.0
Modal
Mocea
Frompr Banchmanc
Word Buchett
Word Rushet Guckets
Fronst mardhmar
Bacheij
30/23
50AL-.0
F--0
F-1
F-
Duntae
Outua
Octuu
30.0.0
509.13
F4
PAXL
Figure 10: Comparison of Stable Diffusion (SD), SDXL, DecpFloyd, and our GlyphControl across diffcrent benchmarks and frequency buckets.
C Failure Case Analysis
We show some failure cases generated by GlyphControl in Figure 11 The Rendering Overlap problem happens when the locations of text boxes within glyph instructions overlap. Although the text rendering accuracy has improved significantly compared to other text-to-image generation models (Section 4.3 & 4.4), some "layout issues" (00 like Missing Glyphs, Wrong Glyphs, Duplicate Glyphs, Unexpected Text, and Illegible Text still occur, implying the necessity of improvement of the generative model architecture or conditional control scheme. Bad performance in the scenarios of Excessive Yaws and Small Text may be attributed to a lack of corresponding training samples.
15

---

for sale by owner sign is displayed in front of house
the last airbender the promise
health and wellness articles, recipes, and tips
irish you would be drink keychain
Riggee AVATAR
AVATTAR
IRISH MESSORINE
IRISH NE ABRINN
Rendering Overlap
Illegible Text
Excessive Yaws
Missing Glyphs
pool party gift with bottle opener, bottle opener, and a bottle opener
bruce & co limited logo
nate certified technician will create 500 bucklinks excellence
for your website
CIATE CATTESED
DRUCE &CQ ULIMITED
BRUCE & Co. LIMTED
NATE
NATE
POOL PARTY
POOO PARTY
CENTIFIES
MILLONEF
Wrong Glyphs
Unexpected Text
Small Text
Duplicate Glyphs
Figure 11: Failure cases generated by GlyphControl using captions glyph instructions the LAION-Glyph test dataset. We demonstrate 8 types of errors here. Captions and glyph images are also shown here. And we also mark out the places where the errors occur with rectangles on the corresponding glyph images.
Training Datasel
U-Net Decoder
Acc(%))
Acc(So)
LD J
CLIP Scoref
frozen
48/39
55/43
1.21/1.73
33.8/36.3
TextCaps SK
fine-tuning
61/43
68/49
0.76/1.38
34.2/36.3
(a) Illustrating the effect of unlocking the U-Net decoder: we conducted two experiments, in both cases, the models were fine-tuned for an additional 40 epochs using the checkpoint trained on LAION-Glyph- 100K the starting point (shown Table T.. During the fine-tuning process, the learning rate of the decoder set le-4.
Training Datasct
Training Epochs
Acc(Stf
Acc(%)f
LD
CLIP Scoref
pre-trained
30/19
37/24
1.77/2.58
33.7/36.2
48/28
56/34
1.31/2.32
33.8/35.5
TextCaps 5K
50/34
61/11
1.03/2.01
34.3/35.7
40
61/45 /43
68/49
0.76/1.38
34.2/36.3
(b) The comparison between different training epochs. The U-Net decoder is also fine-tuned
Table 4: Ablation experiments TextCaps 5K. We report the average results on two benchmarks.
D More Ablation Studies
To assess the generalization capability of our approach on different training datasets, we curate specialized OCR-related training dataset called TextCaps 5K. This dataset consists of images related to signs, books, and posters, which are extracted from the training set the TextCaps v0.1 Dataset 33 The original purpose of this dataset was image captioning with understanding of visual text content. We fine-tune our model on the TextCaps 5K dataset for additional 40 epochs, using the same training settings as those applied to the model trained on LAION MALON-Gyyph--i 1000phh000KK-..... (as shown in Table I This fine-tuning process aims to evaluate how well our model performs on a different training dataset and further validate its robustness and adaptability.
Through our experiments, we discover that unlocking the frozen U-Net decoder in the original Stable Diffusion model can significantly enhance the OCR accuracy of visual text rendering, as demonstrated Table 4a This improvement can attributed to the decoder's improved adaptation to the smaller TextCaps 5K dataset during training. As training progresses, the accuracy of text generation gradually improves, as shown in Table [bb] However, worth noting that the generated images tend to resemble the samples in TextCaps 5K.
16

---

(a) pre-trained
(b) 10x epochs
(c) 20 epochs
(d) 40x epochs
A logo for the company "Plant".
where the
Plant
Planl
Plant
Plant
logo is rounded by flowers and butterflies.
An antique
bottle labeled "Energy Passion" "
Energy FoSsIon: U
Energy
Energy
Figure 12: Illustrating the visual text images generated using the models fine-tuned on TextCaps 5K. The models trained with the settings following Table 4b
As depicted in Figure 12 the visual text regions in the center of the generated images appear more like conventional signs (Figure I2(), without seamlessly merging with the background, while the images generated by the model pre-trained on LAION- LALON-Glyp--I00 100K exhibit greater creativity and diversity (Figure 12h). Therefore, in order generate photo-realistic images that accurately render visual text, it is essential to not only have training dataset with large number of high-quality samples containing abundant visual text content, but also include more realistic images with diverse scenes and creative elements.
17


Reviews:
[{"review_id": "algAvlRbsn", "reviewer": "Reviewer_Cb1X", "summary": "This work proposes GlyphControl for visual text generation by augmenting textual prompt with additional glyph conditional information. A benchmark of LAION-OCR is built for evaluating this model.", "questions": "Moreo discussion on technical contribution and more comparison with competitive baselines.", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "++The task of visual text generation is interesting.\n\n++Good results are shown in experiments.\n\n++A new visual text generation benchmark is introduced.\n", "weaknesses": "--The main concern is the limited technical contribution. The whole architecture of GlyphControl is a simple extension of ControlNet by using additional control of glyph images. But the idea of glyph images has been validated in GlyphDraw [13] for the same task.  \n\n--I am curious by the claim of “outperforms DeepFloyd IF and Stable Diffusion in terms of 55 OCR accuracy and CLIP score while saving the number of parameters by more than 3x”. As shown in Table 1, the parameter number of GlyphControl is significantly larger than Stable Diffusion v2.0.  \n\n--Why not train GlyphControl on the common SD 2.1, which has the similar number of parameters and clearly outperforms SD 2.0.  \n\n--More competitive baselines should be included for comparison (GlyphDraw [13], SD XL and Midjourney).  \n\n--As in GlyphDraw [13], it is necessary to report the FID values in performance comparison.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer Cb1X\n\nWe thank the reviewer for the careful reviews and constructive suggestions. We answer the questions as follows.\n\n> \"The main concern is the limited technical contribution. The whole architecture of GlyphControl is a simple extension of ControlNet by using additional control of glyph images. But the idea of glyph images has been validated in GlyphDraw [13] for the same task.\"\n\nA: Please refer to the general response for addressing the concerns regarding the limited technical contribution. To sum up, the main contribution of this work lies in introducing an astonishingly simple yet highly effective approach for rendering legible and visually coherent text.\n\n---\n\n> \"I am curious by the claim of “outperforms DeepFloyd IF and Stable Diffusion in terms of 55 OCR accuracy and CLIP score while saving the number of parameters by more than 3x”. As shown in Table 1, the parameter number of GlyphControl is significantly larger than Stable Diffusion v2.0.\"\n\nA: Thanks for pointing out the typo, we will fix it in the revised paper. \"More than 3x\" refers to the comparison with DeepFloyd IF-I-XL (shown below), which achieves the best OCR accuracy among other baselines. \n\n| Method | \\#Params | Text Encoder | \n| :---------------- | :--------- | :--------| \n| DeepFloyd (IF-I-XL) | $6.0$ B | T5-XXL ($4.8$ B)|\n| GlyphControl  | $1.3$ B |  CLIP ($354$ M) |\n\nThe total number of parameters reported in the second column of the table does not include the text encoder.\n\n---\n\n> \"Why not train GlyphControl on the common SD 2.1, which has the similar number of parameters and clearly outperforms SD 2.0.\"\n\nA: We chose SD 2.0 to demonstrate the effectiveness of our framework and it could be transferred onto SD 2.1. We are training the GlyphControl framework based on SD 2.1 using the LAION-OCR-100k while keeping the same training settings as the former experiments on SD 2.0. The results of OCR accuracy, CLIP Score, and FID are shown below.\n\n| Method           | Epochs |  $\\bf{Acc}$ (%) $\\uparrow$ | $\\bf{\\hat{Acc}}$ (%) $\\uparrow$ |$\\bf{LD}\\downarrow$ | CLIP Score $\\uparrow$ |  FID-10K-LAION-OCR $\\downarrow$ |\n| :---------------- | :---------: | :---------: | :--------:| :---------: | :---------: | :---------: |\n| GlyphControl-SDv2.0        | 60 | $30/19$  | $37/24$ | $1.77/2.58$  | $33.69/36.20$ | $29.13$ |\n| GlyphControl-SDv2.0        |  30 | $20/14$ | $28/19$  | $2.20/3.00$ | $33.73/36.14$ |  $29.18$ |\n| GlyphControl-SDv2.1        |  30  | $17/14$ | $27/19$  | $2.36/3.18$ | $33.85/35.54$ | $28.02$ |\n\nAfter training for 30 epochs, the GlyphControl model trained on SDv2.1 shows slightly worse performance compared with the counterpart trained on SDv2.0 previously in terms of OCR accuracy and CLIP score. While the fidelity of images (FID) has been improved, which may be attributed to the more powerful generation abilities of SDv2.1.\n\nAnd we will report the results of GlyphControl-SDv2.1 in the revised paper after finishing the 60-epoch training.\n\n---\n\n> \"More competitive baselines should be included for comparison (GlyphDraw [13], SD XL and Midjourney).\"\n\nA: As we check in the github repo of GlyphDraw [13] https://github.com/OPPO-Mente-Lab/GlyphDraw, the checkpoints of GlyphDraw are not available. Thus, we could not conduct a fair comparsion on our benchmark. Besides, Midjourney is not a free open-sourced tool, thus we fail to make a comparison with Midjourney.\n\nSDXL was not open-sourced until June, 2023, and the latest version SDXL-1.0 has been released at https://github.com/Stability-AI/generative-models right before the rebuttal. The comparison results with SDXL-1.0 are reported here and will be added to the paper. We use both SDXL-base-1.0 and SDXL-refiner-1.0.\n \n\n| Method           |  $\\bf{Acc}$ (%) $\\uparrow$ | $\\bf{\\hat{Acc}}$ (%) $\\uparrow$ |$\\bf{LD}\\downarrow$ | CLIP Score $\\uparrow$ |  FID-10K-LAION-OCR $\\downarrow$ |\n| :---------------- | :---------: | :--------:| :---------: | :---------: | :---------: |\n| SDXL-1.0 | $0.3/0.5$ |  $13/8$ | $6.26/6.30$ | $31.9/33.3$ |  $44.77$   |  \n| GlyphControl-SDv2.0 (LAION-OCR-100K)          | $30/19$  | $37/24$ | $1.77/2.58$  | $\\bf{33.7}/\\bf{36.2}$ |   $29.13$ |\n| GlyphControl-SDv2.0 (LAION-OCR-1M)          | $\\bf{40}/\\bf{26}$  | $\\bf{45}/\\bf{30}$ | $\\bf{1.59}/\\bf{2.47}$  | $33.4/36.0$ | $28.02$ |\n\n\nAs seen in the above table, our method still significantly outperforms the latest powerful text-to-image generation model SDXL-1.0 in terms of OCR accuracy, CLIP Score, and FID. Moreover, we also include some visualization results in **Figure 1 of the attached PDF** for comparison with competitive baselines, i.e., IF, SDXL, and Midjourney. And we will add the results of SDXL-1.0 to Table 1 & 2 and include visualization results in the revised paper.\n\n---\n\n> \"As in GlyphDraw [13], it is necessary to report the FID values in performance comparison.\"\n\nA: In order to test the visual quality of generated text images, We evaluate the FID on our LAION-OCR after selecting examples which are not used to train GlyphControl framework. \n\n| Method           |   FID-10K-LAION-OCR $\\downarrow$ | \n| :---------------- |  :---------: |\n| SDXL-1.0  | $44.77$ |\n| Stable Diffusion v2.1         | $50.01$     |\n| Stable Diffusion v2.0        |  $39.23$  |\n| DeepFloyd (IF-I-M)           | $23.53$ |\n| DeepFloyd (IF-I-L)          |   $30.85$  | \n| DeepFloyd (IF-I-XL)          |  $26.34$ |\n| GlyphControl-SDv2.0 (LAION-OCR-100K)         | $29.13$   |\n| GlyphControl-SDv2.0 (LAION-OCR-1M)       |   $28.02$ | \n\nOur approaches show comparable performances with DeepFloyd IF models in terms of FID. This demonstrates that the diversity and quality of visual text image generation is preserved by our method. And both SDXL and original SD models perform much worse than ours. \n\nAnd we will add FID values into Table 1."}, {"review_id": "Deg1zMqmQ2", "reviewer": "Reviewer_wGfp", "summary": "In this paper, a glyph-conditional text-to-image generation model named GlyphControl is proposed for visual text generation. In addition, the authors introduce a visual text generation benchmark named LAION-OCR by filtering the LAION-2B-en. The results show that method of this paper outperforms DeepFloyd IF and Stable Diffusion in terms of OCR accuracy and CLIP score.", "questions": "1.\tDoes the glyph render module only support a single bounding box input or can it accommodate multiple bounding box inputs?\n2.\tDoes the method described in this paper utilize classifier-free guidance during sampling? Although caption dropping is mentioned in the experimental details, there is no specific mention of classifier-free guidance.\n", "limitations": "The manageable elements of users are still constrained. Users are unable to select the color or the font.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe visualization results shown in this paper are impressing. The images presented in the paper show that the text of the generated text region is accurate at both the character level and the word level.\n2.\tThe introduction of LAION-OCR dataset is a modest contribution to the realm of NLP and text image generation.\n", "weaknesses": "1.\tThe contribution is limited. The whole model is the same as ControlNet. The LAION-OCR dataset is just filtered out from an open-source dataset LAION-2B-en, which is not collected by the authors.\n2.\tThe results of the quantitative experiment indicate that the accuracy achieved using this method is merely 40%/26%, which to some extent suggests that the visualized outcomes have been carefully selected. More results need to be displayed.\n3.\tThe comparison is unfair. The compared methods do not prioritize text generation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer wGfp\n\nWe thank the reviewer for the careful reviews and constructive suggestions. We answer the questions as follows.\n\n> \"The contribution is limited. The whole model is the same as ControlNet. The LAION-OCR dataset is just filtered out from an open-source dataset LAION-2B-en, which is not collected by the authors.\"\n\nA: Please refer to the general response. In general, the main contribution of this work is  to provide a surprisingly simple yet effective solution to rendering legible (visually coherent) text. We are committed to refining this work further based on any additional valuable suggestions you may provide.\n\n---\n\n> \"The results of the quantitative experiment indicate that the accuracy achieved using this method is merely 40%/26%, which to some extent suggests that the visualized outcomes have been carefully selected. More results need to be displayed.\"\n\nA: Great suggestion! We have followed your valuable comments to visualize more failure cases and illustrate different types of errors by pointing out the places where the errors occur in **Figure 3 of the attached PDF**. We also would like to include these visualization results and add more analysis in the final revision. \n\n---  \n\n> \"The comparison is unfair. The compared methods do not prioritize text generation.\"\n\nA: We disagree with your statement for the following reasons:\n\n-**Training cost**: The key factors contributing to the success of these models are (i) using much stronger text encoders (DeepFloyd IF uses T5-XXXL with 4.8B parameters, SD-XL uses a combination of open CLIP-G and CLIP-L with 817M parameters) and (ii) re-training the entire text-to-image diffusion models from scratch. These factors require thousands of A100 GPUs for training over several months. Given that our method only requires fine-tuning off-the-shelf models, it is unfair to claim that the comparisons are biased considering the substantial training costs.\n\n-**Position of this work**: As reported in Table 1, both DeepFloyd IF[2] and SD-XL[3] still struggle to render accurate legible text. The Midjourney model performs even worse when handling this challenging task. We want to emphasize that our method is not intended to replace these strong models but to improve their accuracy. We are also making significant efforts to integrate our method into these robust baselines, and we will include these results in the revision, such as GlyphControl + DeepFloyd IF and GlyphControl + SD-XL.\n\n[1] https://huggingface.co/stabilityai/stable-diffusion-2-1#limitations\n\n[2] https://www.deepfloyd.ai/deepfloyd-if / https://github.com/deep-floyd/IF\n\n[3] https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0\n\nOverall, it's challenging to definitively claim the comparison as unfair given the noted distinctions. We aspire for our work to play a pivotal role in advancing the accuracy of visual text rendering. We would like to hear your further valuable feedback.\n\n\n---\n\n> \"Does the glyph render module only support a single bounding box input or can it accommodate multiple bounding box inputs?\"\n\nA: Great point! Indeed, we have effectively demonstrated that the glyph render module is capable of handling multiple bounding box inputs. You can find a detailed illustration in the main paper, precisely in text lines 157-158 and the fourth example in Figure 4. Moreover, all four examples showcased in Figure 1 have been generated using multiple bounding box inputs, further substantiating our capabilities in this aspect. \n\n---\n\n> \"Does the method described in this paper utilize classifier-free guidance during sampling? Although caption dropping is mentioned in the experimental details, there is no specific mention of classifier-free guidance.\"\n\nA:  Yes, we adopt the classifier-free guidance following other diffusion models like Stable Diffusion and ControlNet. Thanks for pointing out this important detail we have missed and we will explain how we use classifier-free guidance in the revised paper.\n\n---\n\n> \"The manageable elements of users are still constrained. Users are unable to select the color or the font.\"\n\nA: Thank you for your insightful suggestion! We are committed to delving into the possibilities of controlling colors and fonts in our future endeavors.  We also welcome and appreciate your further valuable suggestions.\n"}, {"review_id": "mFf2HmCHXb", "reviewer": "Reviewer_kTnz", "summary": "The paper add glyphcontrol to diffusion model by adding controlnet that takes rendered whiteboard images as inputs. Texts in the whiteboard images are extracted by OCR engine during training and then rendered by glyph renderer. During inference, glyph renderer renders the whiteboard images based on the instructions and feed the images using glyph controlnet. They also provide LAION-OCR benchmark by filtering LAION dataset using OCR systems.", "questions": "Does finetuning with OCR data harm the generation performance of the original model? (visual quality of non-textual parts after fine-tuning)", "limitations": "The paper does not explicitly describe their limitations. It would be nice the author provides limitations that can be solved for future works.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The incorporation of whiteboard images generated by the glyph renderer into diffusion models for glyph generation, using controlnet, is a reasonable and easy to understand idea. The paper effectively demonstrates the successful synthesis of images with clean glyphs, surpassing the performance of the baselines, and the glyphs can be controlled easily. They also provide LAION-OCR dataset that is useful for OCR-related research.", "weaknesses": "The proposed method requires additional training. The method may be seen as an extension of ControlNet and may appear to involve the combination of multiple modules, which could be seen as engineering work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer kTnz\n\nWe thank the reviewer for the careful reviews and constructive suggestions. We answer the questions as follows.\n\n> \"The proposed method requires additional training. The method may be seen as an extension of ControlNet and may appear to involve the combination of multiple modules, which could be seen as engineering work.\"\n\nA: Good point!\n\nFirst, our method incurs notably lower additional training costs than alternatives like DeepFloyd IF and SDXL. These methods necessitate extensive retraining using thousands of A100 GPUs over months.\n\nSecond, GlyphControl combines modules for coherent text-image synthesis (seem like an engineering work to some degree), yet its implementation is both simple and effective. It offers a valuable baseline to address limitations in existing SD models.\n\nLast, we aspire to create concise, innovative frameworks moving forward.  We also would like to learn from your further valuable suggestions.\n\n---\n\n> \"Does finetuning with OCR data harm the generation performance of the original model? (visual quality of non-textual parts after fine-tuning)\" \n\nA: No, we believe that finetuning with OCR data using our method would not harm the generation performance of the original model heavily. \n\nWhen generating **natural images without specifying the text rendering information** (FID-30K-COCO), our model strictly retains the generation abilities of the original stable diffusion model, i.e., **FID evaluated on the COCO dataset would remain the same**. It is because the Controlnet framework does not alter the original SD model weights and the GlyphControl branch would not be used during inference due to empty glyph image input.\n\nIn order to test the visual qualities of generating **text images**, We evaluate the FID metrics on LAION-OCR. For LAION-OCR, we select examples which are not included in the dataset for GlyphControl training. \n\n| Method           |   FID-10K-LAION-OCR $\\downarrow$ | \n| :---------------- |  :---------: |\n| SDXL-1.0  | $44.77$ |\n| Stable Diffusion v2.1         | $50.01$     |\n| Stable Diffusion v2.0        |  $39.23$  |\n| DeepFloyd (IF-I-M)           | $23.53$ |\n| DeepFloyd (IF-I-L)          |   $30.85$  | \n| DeepFloyd (IF-I-XL)          |  $26.34$ |\n| GlyphControl-SDv2.0 (LAION-OCR-100K)         | $29.13$   |\n| GlyphControl-SDv2.0 (LAION-OCR-1M)       |   $28.02$ | \n\nOur approaches show comparable or slightly worse performances with DeepFloyd IF models in terms of **FID-10K-LAION-OCR**. It implies that the diversity of visual text image generation is preserved by our method and the quality of non-textual parts in the text images would not be harmed heavily.\n\n---\n\n> \"\nThe paper does not explicitly describe their limitations. It would be nice the author provides limitations that can be solved for future works.\"\n\nA: Thanks for your great suggestions. These are some limitations of our method:\n- Lack of controllability on text font, color, and style\n- Poor performance when generating abundant text or characters with small font sizes \n- Using samples with a limited number of OCR boxes for training \n\nWe will also include discussions about the limitations and future work in our revised paper."}, {"review_id": "LJy5rB3DCR", "reviewer": "Reviewer_xxJ1", "summary": "This paper addresses the development of diffusion-based text-to-image generative models for generating coherent visual text. They propose GlyphControl, which augments textual prompts with glyph conditional information to encode shape details and improve accuracy. They introduce the LAION-OCR benchmark dataset and evaluate GlyphControl's effectiveness using OCR-based metrics and CLIP scores, demonstrating its superior performance over the DeepFloyd IF approach in empirical evaluations.", "questions": "See weakness part", "limitations": "The main concern of this paper revolves around the quantity of text. Starting from the database, the authors control the number of OCR boxes, which, to some extent, limits the generation of rich-text images.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper lie in the proposal of GlyphControl, a glyph-conditional text-to-image generation model. Additionally, the introduction of the LAION-OCR benchmark dataset and the ability to customize and control the content, locations, and sizes of generated visual text demonstrate the paper's practical contributions to the field of visual text generation.", "weaknesses": "\nWhen it comes to text generation, information such as font size and style is crucial, and it would be beneficial for the authors to conduct experimental analysis on this aspect.\n\nThe authors' decision to remove images with more than 5 bounding boxes lacks clarity. Considering rich-text images as a valuable corpus and only focusing on a limited number of OCR images may limit the model's ability to generate rich-text images.\n\nWhile the authors utilize BLIP-2 captions, which may not consistently describe both the image content and OCR text information, it would be interesting to see how the authors address the challenge of generating captions that consider both aspects.\n\nAlthough this method generates accurate text, it raises curiosity about its effectiveness when dealing with a large amount of text or small font sizes, such as generating a paragraph rather than a simple phrase or word.\n\nIn addition to text generation, is it possible to modify the text within an image based on given text information, such as changing the color and position of specific words?\n\nWhen dealing with a large amount of text, such as 500 words, ensuring text generation quality becomes crucial, as it requires high-resolution images. Additionally, ensuring efficiency in generating such images is also an important consideration.\n\nMany texts may not need to be generated as they can be obtained through text rendering. Have the authors attempted this approach, based on text rendering?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reviews and suggestions. \n\n>  Q1 \n\nA: Great point! \n\n👉 For font sizes, our GlyphControl supports users to control the font size of the rendered text by modifying the width property of the text bounding box. We further report the generation results of various font sizes using GlyphControl-SDv2.0, trained with LAION-OCR-100K, in **Table 1 of the attached PDF**.\n\n👉 As we have not included the font style during training, our method does not support controlling font style yet. We would like to investigate the modern font style recognizer to include the style information into the text caption during training.\n\n---\n\n> Q2\n\nA: Great point!  We summarize the key reasons as follows:\n\n- First, we want to highlight that there exist more than 30% of images with more than 5 bounding boxes in our dataset. Here we remove them simply due to limited resources. We have further filtered out samples with the total OCR areas less than 5% of the whole image area, considering that these images do not have sufficient text information. Therefore, the remaining samples only comprise around 20% of the entire set with visual text. We believe there still exist lots of opportunities.\n\n- Second, it's pertinent to note that a majority of previous endeavors have predominantly showcased the capability to generate less than two bounding boxes. As a result, this simplifies the complexity of the text-rendering tasks to some degree.\n\n- Third, our analysis has revealed that instances containing a substantial number of OCR boxes (>5) often pertain to scenarios involving books and newspapers. In such cases, there's a dual challenge: the relatively diminutive size of texts poses difficulty for VAE decoder-based pixel-level reconstruction, and the dense text distribution negatively impacts OCR recognition and Glyph Rendering.\n\nWe have visualized some representative cases with more than 5 bounding boxes in **Figure 2 of the attached PDF**. In summary, we wan to relax the restrictions on the Laion-OCR dataset, raising the threshold from 5 to 10 or 20 bounding boxes in the future.\n\n---\n\n> Q3\n\nA: Great point! We've also noticed that BLIP-2 captions often struggle to consistently depict both image content and OCR text information, resulting in considerable noise. We will attempt the following avenues to address this challenge:\n\n- Employing more potent caption models like LLaVA-from-LLaMA-2 (https://github.com/haotian-liu/LLaVA) and Kosmos-2 (https://github.com/microsoft/unilm/tree/master/kosmos-2). Additionally, we aim to craft specialized prompts to guide the model in generating reliable captions that encompass both image content and OCR text information.\n\n- Enhancing BLIP-2 captions using GPT-3.5 or GPT-4 (text-only) based on supplementary OCR recognition outcomes. Our intention is for the rewritten captions to exhibit a higher quality. We can also explore refining the captions based on the above stronger models such as LLaVA-from-LLaMA-2 and Kosmos-2.\n\nGiven the resource-intensive nature of both approaches and the ongoing nature of their implementation, we aspire to incorporate these results into the final revision. We are also keen to receive any further valuable suggestions that could contribute to our progress in this direction.\n\n---\n\n> Q4\n\nA: Great point! We follow your suggestion to generate images that contain a paragraph rather than a simple phrase or word. We visualize the results in **Figure 2 of the attached pdf**. Our method could preserve the global arrangement following glyph instructions, but fail to generate readable small size text within paragraphs. We would like to further explore this inspiring direction.\n\n---\n\n> Q5\n\nA: Great suggestion! We concur that editing glyphs (visual text) within an image using text prompts is an immensely valuable avenue to pursue. While our present solution doesn't currently encompass this feature, we are committed to delving into this direction soon. Our plan involves drawing inspiration from previous notable works like Prompt-to-Prompt and Instruct-Pixel2Pixel. We aim to establish a framework in this exciting direction by constructing relevant training data and frameworks.\n\n---\n\n> Q6\n\nA:  Great point! First, we need to elevate the filtering threshold for the number of boxes and incorporate training data that encompasses a substantial quantity of text. Second, we agree the importance of high-resolution images. Notably, the recently open-sourced SDXL might provide an excellent starting point as it supports resolutions up to 1024x1024. Third, generating high-resolution images surely will bring more computation costs. However, we have observed that the glyph render's efficiency is notably high and only incurs a minor overhead. Furthermore, we are keen to explore various avenues to enhance the efficiency of the diffusion models, a broad and open challenge. This could involve measures like int8 quantization, utilization of advanced samplers like EulerEDMSampler, or even exploring cascade diffusion architectures such as Imagen and its open-source variant, DeepFloyd IF.\n\n---\n\n> Q7\n\nA: Great point! While text generation might not be essential for all cases, coherence issues can arise from current naive glyph rendering methods, which only support rendering black texts of the same font style on the whiteboard. We're keen to explore advanced font rendering integration, like artistic fonts from Adobe Firefly, within GlyphControl. This approach could work well for poster-like images. Your additional feedback is highly appreciated.\n\n---\n\n> Q8\n\nA: Your suggestions have been immensely valuable and have significantly enhanced our work. We intend to enrich the training dataset with more OCR boxes, train the models accordingly, and provide both quantitative and qualitative comparison results. We assure you that despite time constraints during the rebuttal period, we'll include these outcomes in the final revision. Your input has played a pivotal role in refining our work.\n"}, {"review_id": "OSg9S8E3LC", "reviewer": "Reviewer_Yxcw", "summary": "This work proposes an approach to generating images with visual text. The main approach consists of a generalized version of ControlNet with rendered text as control guidance. To facilitate the training and evaluation, a benchmark dataset called LAION-OCR is proposed. Experiments show that the proposed approach outperforms competitors such as DeepFloyd IF and SD. ", "questions": "- Is the text encoder frozen? What about U-Net Encoder? What is the influence of such design choices?", "limitations": "Limitations not well discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Generating visual text accurately has been a challenging issue in the field. This work provides a simple yet effective approach to this challenge by elegantly extending ControlNet with rendered text as control. I am convinced that having rendered text as input makes a lot of sense in this context and would greatly help generate accurate text.\n- In addition, having rendered text also allows control over the positioning, font size, which would be of great help in practice.\n- Experiments show that the proposed approach significantly outperforms competitors while showing compelling visual results.\n- Overall, I believe this work is a nice addition to the current research landscape of text-to-image generation.", "weaknesses": "It would help to add some failure cases analysis in order to better understand when models may fail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer Yxcw\n\nWe thank the reviewer for the careful reviews and constructive suggestions. We answer the questions as follows.\n\n> \"It would help to add some failure cases analysis in order to better understand when models may fail.\"\n\nA: Thanks for your suggestion. We have added additional failure cases of eight types of errors in **Figure 3 of the attached PDF file**. The Rendering Overlap problem happens when the locations of text boxes within glyph instructions overlap. Some layout issues like Missing Characters, Wrong Characters, and Duplicate Characters still occur. Bad performance in the scenarios of Excessive Yaws and Small Text may be attributed to lack of corresponding training samples. \n\nBesides, we also show some cases about generating a large amount of small font size text in **Figure 2 of the attached PDF file**. Although  the arrangement of glyph images is preserved within generated images, our model still struggles in generating readable small size text.\n\nAnd we will add the above failure case analysis to the revised paper.\n\n---\n\n> \"Is the text encoder frozen? What about U-Net Encoder? What is the influence of such design choices?\"\n\nA: Yes, the text encoder CLIP is frozen. As for the U-Net Encoder, the part of the original Stable Diffusion is frozen while Glyph ControlNet (seen in Figure 2), which is essentially an additional copy of the U-Net Encoder, is trainable. Such design choices aim at preserving the original generation ability of SD.  \n\n---\n\n> \"Limitations not well discussed.\"\n\nA: Thanks for your advice. Here are some limitations of our method:\n- Lack of controllability on text font, color, and style\n- Poor performance when generating abundant text or characters with small font sizes \n- Using samples with a limited number of OCR boxes for training \n\nWe will include more discussions about the limitations and potential future work to our revised paper."}]

Author rebuttal:
## To AC and All Reviewers


We would like to express our gratitude to all the reviewers for their careful reviews and constructive suggestions. We appreciate the positive comments, such as "the task of visual text generation is interesting" (Reviewer Cb1X), "the visualization results shown in this paper are impressing" (Reviewer wGfp), "reasonable and easy to understand" (Reviewer kTnz), "demonstrate the paper's practical contributions to the field of visual text generation" (Reviewer xxJ1), and "I believe this work is a nice addition to the current research landscape of text-to-image generation" (Reviewer Yxcw).

We would like to address the major concerns regarding the contribution of this work by focusing on the following aspects:

> **The motivation and potential impact of our work**

First, we would like to emphasize that addressing the well-known limitations[1] of rendering legible (visually coherent) text is a significant challenge for the fundamental Stable-Diffusion model series.

Second, we acknowledge that some groundbreaking works, such as DeepFloyd IF[2] and SD-XL[3], which emphasize their strong capabilities of rendering legible text, were made public shortly before or after our submission. However, we disagree with Reviewer wGfp's statement that "the comparison is unfair, the compared methods do not prioritize text generation," for the following reasons:

-**Training cost**: The key factors contributing to the success of these models are (i) using much stronger text encoders (DeepFloyd IF uses T5-XXXL with 4.8B parameters, SD-XL uses a combination of open CLIP-G and CLIP-L with 817M parameters) and (ii) re-training the entire text-to-image diffusion models from scratch. These factors require thousands of A100 GPUs for training over several months. Given that our method only requires fine-tuning off-the-shelf models, it is unfair to claim that the comparisons are biased considering the substantial training costs.

-**Position of this work**: As reported in Table 1, both DeepFloyd IF[2] and SD-XL[3] still struggle to render accurate legible text. The Midjourney model performs even worse when handling this challenging task. We want to emphasize that our method is not intended to replace these strong models but to improve their accuracy. We are also making significant efforts to integrate our method into these robust baselines, and we will include these results in the revision, such as GlyphControl + DeepFloyd IF and GlyphControl + SD-XL.

[1] https://huggingface.co/stabilityai/stable-diffusion-2-1#limitations

[2] https://www.deepfloyd.ai/deepfloyd-if / https://github.com/deep-floyd/IF

[3] https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0

> **Comparison with GlyphDraw and ControlNet**

First, we would like to clarify that **GlyphDraw is a concurrent work made available on arXiv on 31st March 2023**, which explores the benefits of glyph images in a distinct manner.

Second, we want to emphasize that the ControlNet architecture design is not our contribution. Our contributions lie in (i) demonstrating that using glyph images is a surprisingly simple yet highly effective approach for generating legible text, and (ii) introducing the LAION-Glyph benchmark to facilitate the development of this challenging generation task.

Thirdly, we outline the key differences between our GlyphControl and GlyphDraw:

- Our GlyphControl offers more flexible controllability than GlyphDraw by supporting customized glyph instructions, allowing for control over text line information and text box information (as evidenced by the visual results in Figure 4).
- GlyphDraw requires fine-tuning the cross-attention weights within the U-Net, which may harm the generation capability of the original diffusion model. In contrast, our GlyphControl follows the ControlNet scheme, freezing the well-trained U-Net weights to preserve the original model's capability optimally.
- GlyphDraw necessitates (i) an additional mask prediction module to predict segmentation masks, and (ii) a CLIP image encoder to extract visual representations of the corresponding glyph image. Our GlyphControl, however, only requires an additional Glyph ControlNet to constrain the latent representations within the U-Net.

Lastly, we would like to express our gratitude for the positive and accurate comments from Reviewer Yxcw: **"This work provides a simple yet effective approach to this challenge by elegantly extending ControlNet with rendered text as control. I am convinced that having rendered text as input makes a lot of sense in this context and would greatly help generate accurate text."** We sincerely hope the reviewers will reconsider the original ratings.

PS: **Please refer to the PDF for more visualization results.**

Meta-review: How to generate high-quality and accurate glyphs in the text2image task is challenging. This paper proposed an effective and simple method to handle this problem. The reviews of this paper are slightly mixed. Two reviewers gave “Borderline reject” while three reviewers were very positive on the paper (2 “Weak Accept” and 1 “Accept”). After carefully reading the authors’ rebuttal and all reviewers’ comments. The AC agrees with the majority of reviewers that this paper has made valuable contributions to this challenging task and showed impressive synthesis results. This decision was discussed with and approved by the SAC.
Decision: Accept (poster)