TextDiffuser: Diffusion Models as Text Painters
Authors: Jingye Chen
Yupan Huang
Tengchao Lv
Lei Cui
Qifeng Chen
Furu Wei
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion Model; Text Rendering
TL;DR: Make diffusion model render text accurately and coherently

Abstract: Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue, we introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. TextDiffuser consists of two stages: first, a Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout. Additionally, we contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs with text recognition, detection, and character-level segmentation annotations. We further collect the MARIO-Eval benchmark to serve as a comprehensive tool for evaluating text rendering quality. Through experiments and user studies, we demonstrate that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text. We will make the code, model and dataset publicly available.

TextDiffuser: Diffusion Models as Text Painters
Jingye Chen 13, Yupan Huang 23 Tengchao Lv Lei Cui", Qifeng Chen', Furu Wei 'HKUST Sun Yat-sen University *Microsoft Research
qwerty. cheerty.chen@comecttco ust huangy228mmai12. sysu. eaa.suuduu.co en, cqf@ust. hk (tengchaolv, lecu,
com
Abstract
Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue, introduce TextDiffuser, focusing generating images with visually appealing text that is coherent with backgrounds. Tex:Diffuser consists of two stages: first, Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout. Additionally, contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image- text pairs with text recognition, detection, and character-leve segmentation annotations. We further collect the MARIO-Eval benchmark to serve as a comprehensive tool for evaluating text rendering quality. Through experiments and user studies, show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting reconstruct incomplete images with text. The code, model, and dataset will be available at https // aka. ms wttttdfffffferr..
(a) Text-to-lmage
(b) Text-to-lmage with Template
A cal holds a paper saying "Hello World"
Work
Hello World
Biik
company
layout generation
image generation
Hopgy bunday You
Belly
Ba
Hello World
Hello World
Hello World
cake
pizza
NOT ASE
READ AAT
5O HOT
A dog
An astronaut
dinosaur
template
mask
dress
newspaper
(c) Text Inpainting
Good Morring Ug
Good Nomiig
fusion
Diffusion
Diffusion
a boy draws good morning on a board
a supermarket called diffusion
Figure 1: TextDiffuser generates accurate and coherent images from text prompts together with template images, as well as conducting text inpainting to reconstruct incomplete images.
Equal contribution during internship Microsoft Research.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
The field of image generation has seen tremendous progress with the advent of diffusion models [2, 15, 16, 18, 25, 67, 70, 72, 79, 93] and the availability of large-scale image-text paired datasets [17, 74, 75]. However, existing diffusion models still face challenges in generating visually pleasing text on images, and there currently no specialized large-scale dataset for this purpose. The ability of Al models to generate accurate and coherent text on images is crucial, given the widespread use of text images in various forms (e.g., posters, book covers, memes, etc.) and the difficulty in creating high-quality text images, which typically require professional skills and numerous times of designers.
Traditional solutions creating text images involve using image processing tools like Photoshop to add text onto images directly. However, these often result in unnatural artifacts due to the background's complex texture or lighting variations. Recent efforts have used diffusion models to overcome the limitations of traditional methods and enhance text rendering quality. For instance, Imagen [72], IDiff-I [2], and DeepFloyd [12] observe diffusion models generate text better with T5 series text encoders [61] than the CLIP text encoder [60]. Liu et al. employ character-aware text encoders to improve text rendering [46]. Despite some success, these models only focus on text encoders, lacking control over the generation process. A concurrent work, GryphDraw [49], improves the controllability of models by conditioning on the location and structures of Chinese characters. However, GlyphDraw does not support multiple text bounding-bol box generation, which is not applicable to many text images such as posters and book covers.
In this paper, we propose TextDiffuser, flexible and controllable framework based on diffusion models. The framework consists of two stages. In the first stage, we use a Layout Transformer to locate the coordinates of each keyword in text prompts and obtain character-level segmentation masks. In the second stage, we fine-tune the latent diffusion model by leveraging the generated segmentation masks as conditions for the diffusion process and text prompts. We introduce a character-aware loss in the latent space to further improve the quality of generated text regions. Figure 1 illustrates the application of TextDiffuser in generating accurate and coherent text images using text prompts alone or text template images. Additionally, TextDiffuser is capable of performing text inpainting" to reconstruct incomplete images with text. To train our model, we use OCR tools and design filtering strategies to obtain 10 million bigh-quality image-text pairs with OCR annotations (dubbed as MARIO-10M), each with recognition, detection, and character-level segmentation annotations. Extensive experiments and user studies demonstrate the superiority of the proposed TextDiffuser over existing methods the constructed benchmark MARIO-Eval. The code, model and dataset will be publicly available to promote future research.
2 Related Work
Text Rendering. Image generation has made significant progress with the advent of diffusion models [18, 25, 72, 79, 63, 70, 2, 8, 13, 48, 26, achieving state-of-the-art results compared with previous GAN-based approaches [64, 100, 43, 58]. Despite rapid development, current methods still struggle with rendering accurate and coherent text. To mitigate this, Imagen [72], eDiff-I [2],
and DeepFolyd [12] utilize large-scale language model (large T5 [61]) enhance the text-spelling knowledge. In [46], the authors noticed that existing text encoders are blind to token length and trained character-aware variant to alleviate this problem. concurrent work, GlyphDraw [49], focuses on generating high-quality images with Chinese texts with the guidance of text location and glyph images. Unlike this work, we utilize Transformer [81] to obtain the layouts of keywords, enabling the generation of texts in multiple lines. Besides, we use character-level segmentation masks as prior, which can be easily controlled (e.g., by providing a template image) to meet user needs.
Several papers have put forward benchmarks containing few cases regarding text rendering for evaluation. For example, Imagen [72] introduces DrawBench containing 200 prompts, in which 21 prompts are related to visual text rendering (e.g.. A storefront with 'Hello World' written on it). According to [46], the authors proposed DrawText comprising creative 175 prompts (e.g., letter "c' made from cactus, high-quality photo). GlyphDraw [49] designs 218 prompts in Chinese and
DDifferent from text editing [88, 37, 32), the introduced text inpainting task aims to add or modify text guided by users, ensuring that the inpainted text has a reasonable style (i.e.. no need to match the style of the original text during modification exactly) and is coherent with backgrounds.

---

English (e.g., Logo for a chain of grocery stores with the name 'Grocery"). ) Considering that existing benchmarks only contain limited number of cases, we attempt to collect more prompts and combine them with existing prompts to establish a larger benchmark MARIO-Exal to facilitate comprehensive comparisons for future work.
Image Inpainting. Image inpainting is the task of reconstructing missing areas in images naturally and coherently. Early research focused on leveraging low-level image structure and texture to address this task [3, 6, 5]. Later, deep learning architectures such as auto-encoder [55, 45], GAN [65, 98], VAE [103, 105], and auto-regressive Transformers [57, 83, 96] were applied to tackle this problem. Recently, diffusion models have been used to generate high-quality and diverse results for unconditional image inpainting [71, 48, 67, 11, 99], text-conditional image inpainting [52, 1] and mmage-conditional image inpainting [92]. Our work falls under the category of text-conditional image inpainting using diffusion models. In contrast to prior works that focused on completing images with natural backgrounds or objects, our method focuses completing images with text-related rendering, also named text inpainting, by additional conditioning on a character-level segmentation mask.
Optical Character Recognition. Optical Character Recognition (OCR) is an important task that has been studied in academia for a long period [87, 7J. It has undergone remarkable development in the last decade, contributing many applications like autonomous driving [89, 73], car license plate recognition [101, 53], GPT models [76, 28], etc. Various datasets [31, 19, 90, 91] and downstream tasks are included within this field, such text image recognition [77, 41, 95, 14], detection [106, 51, 42, 84]. segmentation [90, 91, 107, 68]. super-resolution [9, 85, 50, 104], as well as some generation tasks, including text image editing [88, 78, 94, 59, 38), document layout generation [56, 22, 20, 40, 331, font generation [30, 21, 36, 54], etc. Among them, the font generation task is most relevant our task. Font generation aims create high-quality, aesthetically pleasing fonts based on given character images. In contrast, our task is more challenging as it requires the generated text be legible, visually appealing, and coherent with the background in various scenarios.
3 Methodology
As illustrated in Figure 2, Tex:Diffuser consists of two stages: Layout Generation and Image Generation. We will detail the two stages and introduce the inference process next.
3.1 Stagel: Layout Generation
In this stage, the objective is to utilize bounding boxes to determine the layout of keywords (enclosed with quotes specified by user prompts). Inspired by Layout Transformer [20], we utilize the Trans- former architecture to obtain the layout of keywords. Formally, we denote the tokenized prompt as P = (Po. PP........ where I. means the maximum length of tokens. Following LDM [67], we use CLIP [60] and two linear layers to encode the sequence as CLIP(P) E RLxd, where is the dimension of latent space. To distinguish the keywords against others, we design keyword embedding Key(P) E R. with two entries (i.e., keywords and keywodss.dd). Furthermore, encode the width of keywords with embedding layer Width(P) E LR Together with the learnable positional embedding Pos(P) p2 introduced in [81]. we construct the whole embedding as follows:
Embedding(P) = CLIP(P) + Pos(P) + Key(P) + Width(P).
The embedding is further processed with Transformer-basee l-layer encoder &E and decoder & to get the bounding boxes B E xx1 of K key words autoregressively:
B = 4 D & g (Embeedding(P)) (bo, bi...., bk-1).
Specifically, we use positional embedding as the query for the Transformer decoder dp, ensuring that the n-th query corresponds to the n-th keyword in the prompt. The model is optimized with /1 loss, also denoted as Bar B| where Bor is the ground truth. Further, we can utilize some Python packages like Pillow to render the texts and meanwhile obtain the character-level segmentation mask C with |A| channels, where |Al denote the size of alphabet A. To this end, we obtain the layouts of keywords and the image generation process is introduced next.
3

---

box0
box1
Render
a cat holds "Hello World"
Hello World
Transformer Encoder
Transformer
Decoder
Get Mask
Tokenize
Embedding Layer
Embedding
Layer
cat
holds
hello
world
query0
queryl
Stagel Layout Generation
Stage2 Image Generation
Hello
Difflusion
Hello
Wesll
Model
U-Net
Whole-image Generation
Denoising Loss
Charucter aware
holds "Hello World"
Loss
Hello
Hello
Hello
W World
Text Prompt
Part-image Generation
Figure 2: TeftDDffuser consists two stages. the first Layout Generation stage, a Transformer- based encoder-decoder model generates character-level segmentation masks that indicate the layout of keywords images from text prompts. In the second Image Generation stage, diffusion model generates images conditioned on noisy features, segmentation masks, feature masks, and masked features (from left to right) along with text prompts. The feature masks can cover the entire or part of the image, corresponding to whole-image and part-image generation. The diffusion model learns to denoise features progressively with denoising and character-awan loss. Please note that the diffusion model operates in the latent space, but we use the image pixels for better visualization.
3.2 Stage2: Image Generation
In this stage, we aim to generate the image guided by the segmentation masks C produced in the first stage. We use VAE [35] to encode the original image with shape H x W into 4-D latent space features F E IR xW' Then we sample a time step T ~ Uniform Tmxx) and sample Gaussian noise R4 xW' corrupt the original feature, yielding F aar vI- ārr where der is the coefficient of the diffusion process introduced in [25]. Also, we downsample the character- level segmentation mask C with three convolution layers, yielding 8-D Č E 18xH' xw" We also introduce two additional features, called 1-D feature mask M R H' xw" and 4-D masked feature FM E IR H' xW In the process of whole-image generation, M is set to cover all regions of the feature and Fx is the feature of fully masked image. In the process of part-image generation (also called text inpainting), the feature mask M represents the region where the user wants to generate, while the masked feature Fas indicates the region that the user wants to preserve. To simultaneously train two branches, we use masking strategy where sample is fully masked with a probability of and partially masked with probability of 1 σ. We concatenate F, C, M, M in the feature channel 17-D input and denoising loss between the sampled noise and the predicted noise Eg:
Ldenossing = || EØ Č, F, M.F.M.P.T)) P.T)
Furthermore, propose character- loss to help the model focus more on text regions. In detail, we pre-train U-Net [69] that can map latent features to character-level segmentation masks. During training, we fix its parameters and only use it to provide guidance by using cross-entropy loss loher with weight Achar (See more details Appendix A), Overall, the model is optimized with
Leeoisiing
Xchar
lehar-
Finally, the output features are fed into the VAE decoder to obtain the images.

---

LET ME FREES
Homeschoo STLO From Mea
Eloira PAGK VAI SE ACABARI
MILLION DELLAR
KEEP CALM JUMP
CALTAIN
CLEAR
Arabic Alphabet
STMEKA
Putmaan
THE MANAGEMENT
MI ROBOT
David
STEEL
KYE
SOUTH SANTA
GroTs
Horizons
JOAN OF
(a) MARIO-LAION
MARIO-TMDB
(c) MARIO-OpenLibrary
Figure 3: Illustrations of three subsets of MARIO-10M, See more details in Appendix C.
3.3 Inference Stage
TexiDiffuser provides a high degree of controllability and flexibility during inference in the following ways: (1) Generate images from user prompts. Notably, the user can modify the generated layout edit the text to meet their personalized requirements; (2) The user can directly start from the second stage by providing template image (e.g., scene image, handwritten image, or printed image), and a segmentation model is pre-trained to obtain the character-leve segmentation masks (Appendix B); (3) Users can modify the text regions of a given image using text inpainting. Moreover, this operation can be performed multiple times. These experimental results will be presented in the next section.
4 MARIO Dataset and Benchmark
As there is no large-scale dataset designed explicitly for text rendering. to mitigate this issue, we collect 10 million image-text pairs with OCR annotations to construct the MARIO-I0M Dataset. We further collect the MARIO-Eval Benchmark from the subset of the MARIO-1OM test set and other existing sources serve comprehensive tool for evaluating text rendering quality.
4.1 MARIO-10M Dataset
The MARIO-IOM is collection of about 10 million high-quality and diverse image-text pairs from various data sources such as natural images, posters, and book covers. Figure 3 illustrates some examples from the dataset. We design automatic schemes and strict filtering rules construct annotations and clean noisy data (more details in Appendix D and Appendix E). The dataset con- tains comprehensive OCR annotations for each image. including text detection, recognition, and character-level segmentation annotations. Specifically, we use DB [42] for detection, PARSeq [4] for recognition, and manually train a U-Net [69] for segmentation. We analyze the performance of OCR tools in Appendix F. The total size of MARIO-10M is 10,061,720 1,20, from which we randomly chose 10,000,000 samples as the training set and 61,720 as the testing set. MARIO- 10M is collected from three data sources:
MARIO-LAION derives from the large-scale datasets LAION-400M [75]. After filtering, obtained 9, 194,613 high-quality text images with corresponding captions. This dataset comprises a broad range of text images, including advertisements, notes, posters, covers, memes, logos, etc.
MARIO-TMDB derives from The Movic Database (TMDB), which is community-buil database for movies and TV shows with high-quality posters. We filter 343,423 English posters using the TMDB API out of 759,859 collected samples. Since each image has no off-the-sheil captions, we use prompt templates to construct the captions according movie titles.
MARIO-OpenL.ine Library derives from Open Library, which an open, editable library catalog that creates a web page for each published book. We first collect 6,352,989 original-size Open Library covers bulk. Then, we obtained 523,684 higher-quality images after filtering. Like MARIO-TMDB, manually construct captions using titles due to the lack of off-the-shel captions.
5

---

4.2 MARIO-Eval Benchmark
The MARIO-Eval benchmark serves as a comprehensive tool for evaluating text rendering quality collected from the subset of the MARIO- 10M test set and other sources. It comprises 5,414 prompts in total, including 21 prompts from DrawBenchl
[72], 175 prompts from Draw TextCreative [46], 218 prompts from Chinese Text [49] and 5,000 image-text pairs from subset of the MARIO-I0M test set. The 5,000 image-text pairs are divided into three sets of 4,000, 500, and 500 pairs, and are named LA10NEva14000,, TMD0Eva and OpenLibraryEval500 based on their respective data sources. We offer examples in Appendix G to provide a clearer understanding of MARIO-Eval.
Evaluation Criteria: We evaluate text rendering quality with MARIO-Eval from four aspects: (I) Fréchet Inception Distance (FID) [24] compares the distribution of generated images with the distribution of real images. (2) CLIPScore calculates the cosine similarity between the image and text representations from CLIP [29, 60, 23]. (3) OCR Evaluation utilizes existing OCR tools to detect and recognize text regions in the generated images. Accuracy, Precision, Recall, and F-measure metries evaluate whether keywords appear in the generated images. (4) Human Evaluation is conducted by inviting human evaluators to rate the text rendering quality of generated images using questionnaires. More explanations shown in Appendix H.
5
Experiments
5.1
Implementation Details
For the first stage, we utilize the pre-trained CLIP [60] to obtain the embedding of given prompts. The number of Transformer layers is set to 2, and the dimension of latent space d is set to 512. The maximum length of tokens L is set to 77 following CLIP [60]. We leverage a commonly used font "Arial.uff and set the font size to 24 to obtain the width embedding and also use this font for rendering. The alphabet A comprises 95 characters, including 26 uppercase letters, 26 lowercase letters, 10 digits, 32 punctuation marks, and a space character. After tokenization, only the first subtoken is marked as the keyword when several subtokens exist for word.
For the second stage, we implement the diffusion process using Hugging Face Diffusers [82] and load the checkpoint dimension of the input convolution layer (from 4 to 17), allowing our model to have a similar scale of parameters and computational time as the original model. In detail, the height H and W of input and output images are 512. For the diffusion process, the input is with spatial dimension H' 64 and W' = 64. We set the batch size to 768 and trained the model for two epochs, taking four days using Tesla V100 GPUs with 32GB memory. We use the Adam W optimizer [47] and set the learning rate to 1c-5, Additionally, we utilize gradient checkpoint [10] and xformers [39] for computational efficiency. During training, we follow [25] to set the maximum time step Tmaz to 1,000, and the caption is dropped with probability of 10% for classifier- free guidance [27]. When training the part-image generation branch, the detected text box is masked with likelihood of 50%. We use 50 sampling steps during inference and classifier-fres guidance with scale of 7.5 following [67].
Notably, we only need to modify the input
5.2 Ablation Studies
Number of Transformer layers and the effectiveness of width embedding. We conduct ablation studies on the number of Transformer layers and whether to use width embedding in the Layout Transformer. The results are shown Table 1. All ablated models are trained on the training set of MARIO-10M and evaluated on its test set. Results show that adding width embedding improves performance, boosting IoU by 2.1%, 2.9%, and 0.3% when the number of Transformer layers / is set to 1, 2. and 4, respectively. The optimal IoU is achieved using two Transformer layers and the width embedding included. See more visualization results in Appendix
Character-level segmentation masks provide explicit guidance for generating characters. The character-level segmentation masks provide explicit guidance on the position and content of characters during the generation process of TextDiffuser. To validate the effectiveness of using character-level segmentation masks, we train ablated models without using the masks and show results in Appendix

---

Table 1:
Ablation about Lay-
Table 2: Ablation on weight of
Table 3:
Ablation on two-
out Transformer.
couracte-awaar loss.
branch training ratio
#1 Hlyyr
(P)
loUt
Achar
Acct
ratio
Acct Bttp//eeFFt///opoaa
0.268
0.396
0.344 / 0.870 0.663
0.289
0.001
0.486
0.25
0.562 / 0.899 / 0.636
0.269
2
0.01
0.494
0.5
0,552 / 0.881 0.715
0.298
0.1
0.420
0.75
0.524 / 0.921 / 0.695
0.294
4
0.297
0.400
0.494 / 0.380 / 0.218
Table 4: The performance of text-to-image compared with existing methods. TextDiffuser performs the best regarding CLIPScore and OCR evaluation while achieving comparable performance FID.
Metrics
StableDDiffusion [67]
ControlNet [102]
DeepFloyd [12]
TextDiffuser
FID4 CLIPScoref
51,295
51.485
34.902
38.758
0.3015
0.3424
0.3267
0.3436
OCR(Accuracy)
0.0003
0.2390
0.0262
0.5609
(Preciiiii)
0.0173
0.5211
0.1450
0.7846
OCR(Recall)t
0.0280
0.6707
0.2245
0.7802
(CR(F-mmaaurre) 0.0214
0.5865
0.1762
0.7824
The generated texts are inaccurate and not coherent with the background compared with texts generated with TextDiffuser, highlighting the importance of explicit guidance.
The experimental results demonstrated in Table 2, where we conduct experiments with Achor ranging from [O, 0.001, 0.01, 0.1, 1J. We utilize Draw Bench Text [72] for evaluation and use Microsoft Read API to detect and recognize the texts in generated images. We use Accuracy (Acc) as the metric to justify whether the detected words exactly match the keywords. We observe that the optimal performance is achieved when Achar is set 0.01, where the score is increased by 9.8% compared with the baseline (Achar = 0).
The weight of character-aware loss.
The training ratio of whole/part-imast generation branches. We explore the training ratio o ranging from [0, 0.25, 0.5, 0.75, 1] and show results in Table 3. When is set to indicates that only the whole-image branch is trained and vice versa. We evaluate the model using DrawBench Text [72] for the whole-image generation branch. For the part-image generation branch, randomly select 1,000 samples from the test set MARIO- 10M and randomly mask some of the detected text boxes. We utilize Microsoft Read API detect and recognize the reconstructed text boxes in generated images while using the F-measure of text detection results and spotting results as metrics (denoted as Det-F and Spot-F, respectively). The results show that when the training ratio is set 50%, the model performs better on average (0.716).
5.3 Experimental Results
Quantitative Results. For the whole-image generation task, we compare our method with Stable Diffusion (SD) [67]. ControlNet [102], and DeepFloyd [12] in quantitative experiments with the publicly released codes and models detailed in Appendix K. DeepFloyd [12] uses two super-resolution modules to generate higher resolution 1024 x 1024 images compared with 512 512 images generated by other methods. We use the Canny map of printed text images generated with our first stage model as conditions for ControlNet [102]. Please note that we are not able to compare with Imagen [72], eDiff-i [2], and GlyphDraw [49] due to the lack of open-source code, checkpoints or APIs. According to Table 4, we demonstrate the quantitative results of the text-to-image task compared with existing methods. Our TextDiffuser obtains the best CLIPScore while achieving comparable performance terms FID. Besides, TextDiffuser achieves the best performance regarding four OCR-related metrics. TextDiffuser outperforms those methods without explicit text-related guidance by large
7

---

Prompt
DALL-E
SD
SD-XL
Midjourney
ControlNet
DeepFloyd
TextDiffussr
hear holds
NELLNO WORDLO
MELLWORD
hello word!
boand saying
Oi ver D
WEO WILL
Hello World
RAPT
Heppy Burhdda 10z.
Tiagery Birbhlly XOPE
Knopy Sudday XIS
HHppy Birthday XYZ
Sindoy
M. MYCSNC
MONKIC
NOWEY
poster of Menkey Music Festival
Loniers Muric
Narsoy
Nusk FesSe
ARE MEME
DESERR
kloding?
meme Wire kiddingg!
YOUT YSIETS
ARE YOU KIKNG?"
book of Next Century'
BN'S BE BUST
ECTEA
writenn by
: CO
Al Robel
boy holds and
R
girl holds' 'O"
PND
Figure 4: Visualizations of whole-image generation compared with existing methods. The first three cases are generated from prompts and the last three cases are from given printed template images.
margin (e.g., 10% and 60.62% better than Stable Diffusion and DeepFloyd regarding F-measure), highlighting the significance of explicit guidance. As for the part-image generation task, we cannot evaluate our method since no methods are specifically designed for this task to our knowledge.
Qualitative Results. For the whole-image generation task. we further compare with closed-souree DALL-E [63], Stable Diffusion XL (SD-XL), and Midjourney by showing qualitative examples generated with their official API services detailed in Appendix K. Figure 4 shows some images generated from prompts or printed text images by different methods. Notably, our method generates more readable texts, which also coherent with generated backgrounds. On the contrary, although the images generated by SD-XL and Midjourney are visually appealing, some generated text does not contain the desired text contains illegible characters with incorrect strokes. The results also show that despite the strong supervision signals provided to ControlNet, it still struggles to generate images with accurate text consistent with the background. We also initiate a comparison with the Character- Aware Model [46] and the concurrent work GlyphDraw [49] using samples from their papers as their open-source code, checkpoints or APIs are not available. Figure 5 shows that TextDiffuser performs better than these methods. For instance, the Character- Aware Model suffers from misspelling issues (e.g.. 'm' in 'Chimpanzees') due to its lack of explicit control, and GlyphDraw struggles with rendering images containing multiple text lines. For the part-image generation task, we visualize some results in Figure In contrast to text editing tasks [88], we give the model sufficient flexibility to generate texts with reasonable styles. For instance, the image in the second row and first column contains the word "country" green, while the model generates the word "country" yellow. This is reasonable since it follows the style of the nearest word "range". Besides, our method can render realistic text coherent with the background, even in complex cases such as clothing. More qualitative results are shown in Appendix L.
8

---

Characte-AAoee (42)
Midal
TestiaiTuwe
Chumster-kuus Model
(Ours)
CALIFORNIA
CALIFORNIA
Melting Snow
Mellin Snowmer
Please Refrain form Arguing with Chimpanzees
Rehrain Arguing with Chimpanzees
WELCOME
Desive intorfides Eluziri Interuudes
posinge with
1.4 AARa
sige that "Plaase redrain from argning with chimpeanees
The cover for the allomm "Elnsive Intethides" by Melting
welcoane
GlyphDuaw
TestDiflaser (Ours)
GlyphDnsw (45)
Textiiifuerr (Oars)
GlyskDuww (45)
(Ours)
Gus Stasion
Station
nyold habit
nyyold habit
The
planet
There is plneet
Photo of "Gss Station
model 1980s-sttye computer with the texl "my old labal ou the screu
5-shirt message "Thae uu phus!" walliau ou
Figure 5: Comparison with Character-Aware Model [46] and the concurrent GlyphDraw [49].
TECH HOUSE
TECH HOUSE
TECH HOUSE INTERESTING TERRACE PARTY
TOWTOMANTA POSTER?
HOW TOMEA
NEVEPAPTR
NTERESTING
TERRACE PARTY
TERRACE PARTY
Interesting party
How make newspaper
Exylok
Ciiinese
MBSE
FIMINSSE COUNTRY
FIMI
COUNTHY RANGE TEST
COUWTAY RANGE TEST
FIMI X8 SE Range Tost in Country
She teaching Chinese Jesson.
Frequently Askedi Questiro ITA
Sked
Frequently Askan
Handsome
Nor doome
TTCC
Buy
Boy
The Muski Editioy
The Muslim Editio
The Musim Edittoy
man wears a cloth containing handsome bey
Frequently Asked Questions (FAQ): The Muslim Edition image
Figure 6: Visualizations of part-image generation (text inpainting) from given images.
User Studies. For the whole-image generation task, the designed questionnaire consists of 15 cases, each of which includes two multiple-choice questions: (1) Which of the following images has the best text rendering quality? (2) Which of the following images best matches the text description? For the part-image generation task, the questionnaire consists of cases, each of which includes two rating questions: (1) How is the text rendering quality? (2) Does the drawn text harmonize with the unmasked region? The rating scores range from
and indicates the best. Overall, we have collected 30 questionnaires, and the results shown in Figure 8. We can draw two conclusions: (I) The generation performance of TextDiffuser significantly better than existing methods. (2) Users satisfied with the inpainting results in most cases. More details are shown in Appendix M.
Time and Parameter Efficiency For the time efficiency, the first stage of Layout Generation lever- ages auto autorregressive Transformer whose prediction time correlates with the number of keywords. Specifically, we conduct experiments to evaluate the time overhead for different numbers of keywords, including (1.07+0.03s), 2 (1.12=0.09s), 4 (1.23+0.. 13s). (1.57 12s), 16 (1.83++0.1 12s), and 32 (1.95- 0.22ss). Meanwhile, the second stage image generation independent of the number of queries 0..75)). For the parameter efficiency, TextDiffuser builds upon Stable Diffusion 1.5 (859M parameters), adding Layout Transformer in the first stage (+25M parameters) and modifying the second stage (+0.75M parameters), augmenting it only about 3% in terms of parameters.

---

Hello Worll
Hallo Work
Hello World
Hello Work
Hello World
Hello world
bear holding beard with hello world
bear holding board with
Taello workd
bear holding board with purple teello world'
Figure Demonstration of using language descriptions to control text color.
Votes
#Votes
300
294
300
Text rendering quaiity
Q1 Text rendering quality
257
200
QZ Image-lext matiching
Q2 Harmonization
156
136
123
124
100
89
48
67
100
48
10
Midjoumey ControlNet DeepFloyd Teubbiflsser (a) For wholo-image generation, our method clearly outpertorms others in both aspects of text rendering quality and image-text matching.
DALL-E
Score
(b) For part-image generation, our method receives high scores from human evaluntors in these two aspects.
Figure 8: User studies for whole-image generation and part-image generation tasks.
Text Color Controllability In Figure 7, we showcase TexiDiffuser's capability in controlling the color of generated texts through language descriptions. The visualization results show that TextDiffuser can successfully control the color of rendered text, further enhancing its controllability.
Discussion and Conclusion
Discussion. We show that TextDiffuser maintains the capability and generality to create general images without text rendering in Appendix N. Besides, compare method with text editing model in Appendix O, showing that TextDiffuser generates images with better diversity. We also present the potential of TextDiffuser the text removal task in Appendix P. As for the limitations and failure cases, TextDiffuser uses the VAE networks to encode images into low oow-dimensional latent spaces for computational efficiency following latent diffusion models [67, 49, 2], which has limitation in reconstructing images with small characters as shown in Appendix Q. We also observed failure cases when generating images from long text and showed them in Appendix Q. As for the broader impact, TexiDiffuser can be applied to many designing tasks, such as creating posters and book covers. Additionally, the text inpainting task can be used for secondary creation in many applications, such Midjoumey. However, there may be some ethical concerns, such as the misuse of text inpainting for forging documents. Therefore, techniques for detecting text-related tampering [86] need to be applied to enhance security. In conclusion, we propose two-stage diffusion model called TextDiffuser to generate images with visual- pleaaing texts coherent with backgrounds. Using segmentation masks as guidance, the proposed TextDiffuser shows high flexibility and controllability in the generation process. We propose MARIO-10M containing 10 million image-text pairs with OCR annotations. Extensive experiments and user studies validate that our method performs better than existing methods on the proposed benchmark MARIO-Ewe Eval. For future work, we aim to address the limitation of generating small characters by using OCR priors following OCR- OCR-VQGAN [66] and enhance TextDiffuser's s capabilities to generate images with text in multiple languages. Disclaimer Please note that the model presented in this paper intended for academic and research purposes ONLY. Any use of the model for generating inappropriate content strictly prohibited and is not endorsed by this paper. The responsibility for any misuse improper use of the model lies solely with the users who generated such content, and this paper shall not be held liable for any such use.
7 Acknowledgement
This research was supported by the Research Grant Council of the Hong Kong Special Administrative Region under grant number 16203 122.
10

---

References
[1] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
[2] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiamning Song. Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arX1v:2211.01324, 2022.
[3] Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera. Filling-in in by joint interpolation of vector fields and gray levels. IEEE transactions on image processing (TIP), 2001.
[4] Darwin Bautista and Rowel Atienza. Scene text recognition with permuted autoregressive sequence models. In ECCV, 2022.
[5] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpaint- ing. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 2000.
[6] Marcelo Bertalmio, Luminita Vese, Guillermo Sapiro, and Stanley Osher. Simultancous structure and texture image inpainting. IEEE transactions on image processing (TIP), 2003.
[7] Glenn L Cash and Mehdi Hatamian. Optical character recognition by the method of moments. Computer vision, graphics, and image processing, 1987.
[8] Huiwen Chang, Han Zhang. Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Hssgnn Yang, Kevin Murphy, William Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.0007 2023.
[9] Jingye Chen, Bin Li, and Xiangyang Xue. Scene text telescope: Text-focused scene image super-resolution. In CVPR, 2021.
[10] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv: 1604. 06174, 2016.
[1I] Hyungjin Chung, Byeongsu Sim, and Jong-Chul Ye. Come-lloser-diftue-faass Accelerating conditional diffusion models for inverse problems through stochastic contraction. CVPR, 2021.
[12] DeepFloyd. Github link: https:/gittubb.co///epp-to 2023.
[13] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS, 2021.
[14] Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In CVPR, 2021.
[15] Zhida Feng. Zhenyu Zhang. Xintong Yu, Yewei Fang. Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. Ernic-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhane
fxxuue-of-denoosingexxxp
In CVPR, 2023.
[16] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Os Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In NeurlPS, 2022.
[17] Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Minzhe Niu, Hang Xu, Xiaodan Liang, Wei Zhang, Xin Jiang, and Chunjing Xu. Wukong: 100 million large-scale chinese cross-modal pre-training dataset and foundation framework. In NeurIPS Workshop. 2022.
[18] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for sit-to--mmaee synthesis. In CVPR, 2022.
11

---

[19] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In CVPR, 2016.
[20] Kamal Gupta, Alessandro Achille, Justin Lazarow, Larry Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layout generation and completion with self-attention. In ICCV, 2021.
[21] Haibin He, Xinyuan Chen, Chaoyue Wang, Juhua Liu, Bo Du, Dacheng Tao, and Yu Qiao. Diff-font: Diffusion model for robust one-shot font generation. CVPR, 2023.
[22] Liu He, Yijuan Lu, John Corring, Dinei Florencio, and Cha Zhang. Diffusion Diffusion-based document layout generation. arXiv preprint arXiv: 10787, 2023.
[23] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP, 2021.
[24] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
[25] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NewrIPS, 2020.
[26] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research (JMLR), 2022.
[27] Jonathan Ho and Tim Salimans. Classifier-fre free diffusion guidance. NeurIPS Workshop. 2021.
[28] Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, al. Language not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 2023.
[29] Yupan Huang, Bei Liu, and Yutong Lu. Unifying multimodal transformer for bi-directional image and text generation. In ACM MM, 2021.
[30] Shir Iluz, Yael Vinker, Amir Hertz, Daniel Berio, Daniel Cohen-On and Ariel Shamir. Word- as-image for semantic typography. arXiv preprint arXiv:2303.0/8/8, 2023.
[31] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. In NewrIPS Workshop, 2014.
[32] Jiabao Ji, Guanhua Zhang, Zhaowen Wang, Bairu Hou, Zhifei Zhang, Brian Price, and Shiyu Chang. Improving diffusion models for scene text editing with dual encoders. arXiv preprint arXiv:2304.05566 2023.
[33] Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from a label set. In ICCV, 2019.
[34] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, al. Iedar 2015 competition on robust reading. In ICDAR, 2015.
[35] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[36] Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, and Lianwen Jin. Look closer to supervise better: One-shot font generation via component-based discriminator. In CVPR, 2022.
[37] Praveen Krishnan, Rama Kovvuri, Guan Pang, Boris Vassilev, and Tal Hassner. Textstylebrush: Transfer of text aesthetics from single example. IEEE Transactions on Puttern Analysis and Machine Intelligence (T-PAMI), 2023.
12

---

[38] Junyeop Lee, Yoonsik Kim, Seonghyeon Kim, Moonbin Yim, Seung Shin, Gayoung Lee, and Sungrae Park. Rewritenet: Reliable scene text editing with implicit decomposition of text contents and styles. In CVPR Workshop, 2022.
[39] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. https://githi: com/ htt//ffaeooooreeerrrree ch/ xf kformers 2022.
[40] Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generating graphic layouts with wireframe discriminators. In ICLR, 2019.
[41] Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. Trocr: Transformer-be optical character recognition with pre-trained models. AAAI, 2023.
[42] Minghui Liao, Zhisheng Zou, Zhaoyi Wan, Cong Yao, and Xiang Bai. Real-time scene text detection with differentiable binarization and adaptive scale fusion. IEEE Transactions Pattern Analysis and Machine Intelligence (T-PAMI), 2022.
[43] Wentong Liao, Kai Hu, Michael Ying Yang, and Bodo Rosenhahn. Text to image generation with semantic-spatial aware gan. In CVPR, 2022.
[44] Chongyu Liu, Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Yongpan Wang. Erasenet: End-to-end text removal in the wild. IEEE Transactions on Image Processing (TIP), 2020.
[45] Hongyu Liu, Bin Jiang, Yibing Song. Wei Huang, and Chao Yang. Rethinking image inpainting via mutual encoder-decoder with feature equalizations. In ECCV, 2020.
[46] Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant, Character-aware models improve visual text rendering. arXiv preprint 1rXiv:2212.105661 2022.
[47] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
[48] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. RePaint: Inpainting using Denoising Diffusion Probabilistic Models. In CVPR, 2022.
[49] Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang, Di Niu, Haonan Lu, and Xiaodong Lin. Glyphdraw: Learning to draw chinese characters in image synthesis models coherently. arXiv preprint arXiv:2303.17770 2023.
[50] Jianqi Ma, Zhetong Liang, and Lei Zhang. A text attention network for spatial deformation robust scene text image super-resolution. In CVPR, 2022.
[51] Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang, Hong Wang, Yingbin Zheng, and Xiangyang Xue. Arbitrary -rrientry-oreetedd scene text detection via rotation proposals. IEEE transactions on multimedia (TMM), 2018
[52] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mc- Grew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In JCML, 2022.
[53] Safaa S Omran and Jumana A Jarallah. Iraqi car license plate recognition using ocr. In 2017 annual conference (NTICT), 2017.
trends in information & communications technology applications
[54] Song Park, Sanghyuk Chun, Junbum Cha, Bado Lee, and Hyunjung Shim. Multiple heads are better than one: Few-shot font generation with multiple localized experts. In ICCV, 2021.
[55] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
13

---

[56] Akshay Gadi Patil, Omri Ben-Eliezer, Or Perel, and Hadar Averbuch- Elobu Read: Recursive autoencoders for document layout generation. CVPR, 2020.
[57] Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. Generating diverse structure for image inpainting with hierarchical vq-vae. In CVPR, 2021.
[58] Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-to- image generation by redescription. In CVPR, 2019.
[59] Yadong Qu, Qingfeng Tan, Hongtao Xie, Jianjun Xu, Yuxin Wang, and Yongdong Zhang. Exploring stroke-level modifications for scene text editing. In AAAI, 2023.
[60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[61] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research (JMLR), 2020.
[62] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204. Xiiv:2204.06/255 2022.
[63] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and 1lya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[64] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML, 2016.
[65] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. Structureflow: Image inpainting via structure-aware appearance flow. In ICCV, 2019.
[66] Juan A Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli, and Pau Rodriguez. Ocr-vqgan: Taming www-ww-ithhhaeeeee generation. In WACV, 2023.
[67] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[68] Xuejian Rong, Chucai Yi, and Yingli Tian. Unambiguous scene text segmentation with referring expression comprehension. IEEE Transactions on Image Processing (TTP), 2019.
[69] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[70] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aber- man. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. /2242, 2022.
[71] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: image-to-imant diffusion models. In SIGGRAPH, 2022.
[72] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
[73] Markus Schreiber, Fabian Poggenhans, and Christoph Stiller. Detecting symbols on road surface for mapping and localization using ocr. In 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), 2014.
[74] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. NeurlPS,
2022.
14

---

[75] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterd 400 million image-text pairs. arXiv preprint arXiv:2111.02) 14, 2021.
[76] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 2023.
[77] Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for image- based sequence recognition and its application to scene text recognition. IEEE Transactions Pattern Analysis and Machine Intelligence (T-PAMI), 2016.
[78] Wataru Shimoda, Daichi Haraguchi, Seiichi Uchida, and Kota Yamaguchi. De-rendering stylized texts. In ICCV, 2021.
[79] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICML, 2015.
[80] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[81] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[82] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https:/ gi thub. www.coo////aacc homm/huggngg wtw.an/g/aa/fffueeeeeeeeee
2022.
[83] Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. In ICCV, 2021.
[84] Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao. Shape robust text detection with progressive scale expansion network. In CVPR, 2019.
[85] Wenjia Wang, Enze Xie, Xuebo Liu, Wenhai Wang, Ding Liang, Chunhua Shen, and Xiang Bai. Scene text image suser-resolution in the wild. In ECCV, 2020.
[86] Yuxin Wang, Hongtao Xie, Mengting Xing. Jing Wang, Shenggao Zhu, and Yongdong Zhang. Detecting tampered scene text in the wild. In ECCV, 2022.
[87] James M White and Gene D Rohrer. Image thresholding for optical character recognition and other applications requiring character image extraction. IBM Journal of research and development, 1983.
[88] Liang Wu, Chengquan Zhang, Jiaming Liu, Junyu Han, Jingtuo Liu, Errui Ding, and Xiang Bai. Editing text the wild. In ACM MM, 2019.
[89] Zizhang Wu, Xinyuan Chen, Jizheng Wang. Xiaoquan Wang. Yuanzhu Gan, Muqing Fang. and Tianhao Xu. Ocr-rtps: an ocr-based real-time positioning system for the valet parking. Applied Intelligence, 2023.
[90] Xingqian Xu, Zhifei Zhang, Zhaowen Wang, Brian Price, Zhonghao Wang, and Humphrey Shi. Rethinking text segmentation: A novel dataset and text-specific refinement approach. In CVPR, 2021.
[91] Xixi Xu, Zhongang Qi, Jianqi Ma, Honglun Zhang, Ying Shan, and Xiaohu Qie Bts: A bi-lingual benchmark for text segmentation in the wild. In CVPR, 2022.
[92] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. arXiv preprint arXiv:2211.13222 2022.
[93] Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint
2022.
15

---

[94] Boxi Yu, Yong Xu, Yan Huang, Shuai Yang, and Jiaying Liu. Mask-guide gan for robust text editing in the scene. Neurocomputing.. 2021.
[95] Deli Yu, Xuan Li, Chengquan Zhang, Tao Liu, Junyu Han, Jingtuo Liu, and Errui Ding. Towards accurate scene text recognition with semantic reasoning networks. In CVPR, 2020.
[96] Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao. Diverse image inpainting with bidirectional and autoregressive transformers. In ACM MM, 2022.
[97] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint
2012.
[98] Yanhong Zeng. Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid-context encoder network for high-quality image inpainting. CVPR, 2019.
[99] Guanhua Zhang, Jiabao Ji, Yang Zhang, Mo Yu, Tommi Jaakkola, and Shiyu Chang. To- wards coherent image inpainting using denoising diffusion implicit models. arXiv preprint arXiv:2304.03322, 2023.
[100] Han Zhang. Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
[101] Linjiang Zhang, Peng Wang, Hui Li, Zhen Li, Chunhua Shen, and Yanning Zhang, A robust attentional framework for license plate recognition in the wild. IEEE Transactions on Intelligent Transportation Systems (TITS), 2020.
[102] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 2023.
[103] Lei Zhao, Qihang Mo, Sihuan Lin, Zhizhong Wang, Zhiwen Zuo, Haibo Chen, Wei Xing, and Dongming Lu. Uctgan: Diverse image inpainting based unsupervised cross-space space translation. In CVPR, 2020.
[104] Minyi Zhao, Miao Wang, Fan Bai, Bingjia Li, Jie Wang, and Shuigeng Zhou. C3-stisr: Scene text image super-resolution with triple clues. In I/CAI, 2022.
[105] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In CVPR. 2019.
[106] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. East: an efficient and accurate scene text detector. CVPR, 2017.
[107] Xinyan Zu, Haiyang Yu, Bin Li, and Xiangyang Xue. Weakly-supervised text instance segmentation. arXiv preprint arXiv:2303. 10848, 2023.
16

---

Appendix
A Architecture of U-Net and Design of Character-Aware Loss
As shown in Figure 9, the U-Net contains four downsampling operations and four upsampling operations. The input will be downsampled to maximum of 1/16. To provide the character-aware loss, the input feature F 4-D with spatial size x 64, while the output is 96-D (the length of alphabet A plus null symbol indicating the non-character pixel) also with spatial size 64 x 64. Subsequently, a cross-entropy loss calculated between the output feature (need to convert the predicted noise into predicted features) and the resized 64 64 Character-level segmentation mask C². The U-Net is pre-trained using the training set of MARIO- 10M for one epoch. We utilize the Adadelta optimizer [97] and set the learning rate 1. When training the diffusion model, the U-Net frozen and only used to provide character-aware guidance.
H
H/2 x W/2
H/2 x W/2
HI/4 x W/4
H W/4
8/8 x W/s
17 x w/8
W/16
Figure 9: The architecture U-Net contains four downsampling and upsampling operations.
B Character-Level Segmentation Model
We train the character-level segmentation model based on U-Net, whose architecture is similar to the architecture shown in Figure 9. We set the input size 256 x 256, ensuring that most characters readable at this resolution. We train the segmentation model using synthesized scene text images [19], printed text images, and handwritten text images?, totaling about 4M samples. We employ data augmentation strategies (e.g., blurring, rotation, and color enhancement) to make the segmentation model more robust. The segmentation model is trained for epochs using the Adadelta optimizer [97] with a learning rate of 1. Figure 10 shows some samples in the training dataset.
Eleg''s
antiorthodox
innethodccanes
niopoppannaleee
Zipangu yn gladk englie
nett reairectess peate- bredher
wim- *llan loun
fellation
puits- Bonk Mia
Aheemimor
Figure 10: Visualization of some training samples for the character-level segmentation model. The top, middle, and bottom roles are samples from printed, handwritten, and scene datasets.
17

---

C More Details in MARIO-10M
Table 5: Number of texts per image in MARIO-IOM.
#Words
2
3
5
7
8
#Images
592,153
1.148,481
1,508,185
1,610,056
1,549,852
1,430,750
1,229.714
930.809
#Ratio
5.9%
11.5%
15.1%
16.1%
15.5%
14.3%
12.3%
9.3%
More samples are shown in Figure 11. The number of texts per image in MARIO-1OM is shown in Table Also, the MARIO-1OM dataset reveals that about 90% of the text regions maintain horizontal orientation with rotation angles smaller than 5 degrees without perspective changes. Hence, our layout generation model is designed to predict horizontal bounding boxes by detecting the coordinates of their left-top and bottom-right points. Adapting our model to predict more realistic scene text is feasible by detecting enhanced coordinates, such as eight coordinates for four points.
D MARIO-10M Caption Templates
Since TMDB movie/TV posters and Open Library book covers have no off-the-shelf captions, we construct them based on their titles with the following templates. (XXX) is placeholder for title.
For MARIO-TMDB:
Logo (XXX)
A movie poster named (XXX)
Text [XXX]
movie poster with text (XXX) on it
Title (XXX)
A movie poster with logo (XXX) it
Title text (XXX)
A movie poster with title text of (XXX)
An illustration of (XXX) movie
A poster with title text of (XXX)
An photography of (XXX) movie
A poster design with title text of (XXX)
A TV show poster titled (XXX)
A quality movie print with a title text of (XXX)
A TV show poster of (XXX)
A TV show poster with logo (XXX) on it
A film poster of (XXX)
A TV show poster with title text of (XXX)
A movie poster of (XXX)
A TV show poster with text (XXX)
A movie poster titled (XXX)
A TV show poster named (XXX)
For MARIO-OpenLLibrary:
A book with title text of (XXX)
A book design with title text of (XXX)
A book cover with title text of (XXX)
A book of (XXX)
A cover named (XXX)
A cover titled (XXX)
A book with text (XXX) on it
A book cover with logo (XXX) on it
18

---

COLORING NO BOOK
484
184
STANDARD SOP TWARE
The 10
Callandar MOM
Gioetal FOOD Colll
Best Sport
Goitee Lnvie
bil it
DEIVENTO
Wheel Centre Caps Bodge 50mm
thee Bedlin DEccetiion
ARIOENIC LEBANESE JEEEO
PHE TOE DESIGNER TOUCH
Succrssiid Feogle
2000
# Figel Herke
fost
TRADITION II DET DITTMON PROOUCE
SEIE GREEN MAN
KONMMANT
RUN YOU STOLE RM
NOW TV
ALEE TIPES REAER
Just E20
2020
MEDEOORNE
Homeloans
com
CANNABIS
Bizez Pot
Curved Edge Flower
DB KEEP CALM CARRY ON
Life&
CALM LOVE FIM PRODUCE
33 LEEAK
peychothorappee couching
UNTT TПMAROP
(a) Samples in MAION-LAION
FLUSH
DOLLAR
OVerY Chrssous
SHATTER
TEMPUS
RISE
HUDAIKUER
THE
MANNISKA
NOC
THE NURSE
DON'T
Funget Usu
FRONT PAGE
GO
WHINIGG
OUTSIDE
MA HREDS WOMEN
Knives Out
CANAONAA
WINCHE
Gottensell
FOUNDER
AWAKENS
(b) Samples in MARIO-TMDB
Struwberrken Boz
THERR
Hii Grace
Crailcda
Totto Seathh
the Light
YELLOWCARD
BLOW DUST RISIN
and
Dark
Lemiie
Marcus
Farrell
Burbling
meurtribre
COMANCH'
K88
Moust
(c) Samples in MARIO-OpenLibrar
Figure 11: More samples in MARIO-I0M.
19

---

E MARIO-10M Filtering Rules
We clean data with five strict filtering rules to obtain high-quality data with text:
Height and width are larger than 256. Low-resolution samples often contain illegible texts, negatively impacting the training process.
Will not trigger NSFW. For the MARIO-LAION subset, we filter out those samples triggering "not sure for work" flag to mitigate ethical concerns.
The number of detected text boxes should be within [1.8]. We detect texts with DB [42]. Samples with too many texts typically have small areas for each text, which makes them difficult to recognize. Therefore, we remove these samples from the dataset.
Text areas more than 10% % of the whole image. According Appendix B, train a UNet [69] using SexthText [19] to obtain character-level segmentation masks of each sample. This criterion ensures that the text regions will not be small.
At least one detected text appears in the caption. Noticing that the original dataset contains many noisy samples, we add this constraint to increase the relevance between images and captions. We utilize PARSeq [4] for text recognition.
20

---

Analysis of OCR Performance on MARIO-10M
As we rely on OCR tools to annotate MARIO-I0M, it is necessary to evaluate the performance of these tools. Specifically, we manually annotate 100 samples for text recognition, detection, and character-level segmentation masks, then compare them with the annotations given by OCR tools. The results shown in Table 6. We notice that the performance of existing methods is lower than their results on text detection and spotting benchmarks. Taking DB [42] as an example, it can achieve text detection 91.8% precision on ICDAR 2015 dataset [34] while only achieving 76% on MARIO-100 10M. This is because there are many challenging cases in MARIO- 10M, such as blurry and small text. Besides, domain gap may exist since DB trained on scene text detection datasets, while MARIO- comprises text images in various scenarios. Future work may explore more advanced recognition, detection, and segmentation models mitigate the noise in OCR annotations. We demonstrate some OCR results in Figure 12.
Table 6: OCR performance on MARIO- 10U (binary) means we treat each pixel as two classes: characters and non-characters, The evaluation of recognition is included in the spotting task.
Detection
Spotting
Segmentation
Precision
Recall 0.79
F-measure 0.78
Precision 0.73
Recall 0.75
F-measure 0.74
IOU (binary)
IOU
0.76
0.70
0.59
Prite
Chitty
about
Kiocolat
Pps
2012 2015
GREENE GREENE
Pride
Chity
about
hotol
Peppalig
HIGH
HOO
2003
pop
LONDRES
POLISH
Danceshort
RILCY
MonkeYY
WOOL
2003
epop
LONORES
POLISH
Danovsport
FLIES
RELEASE
TEANIENE
DIARY
CAPTURE
LIES
RELEASE
STA
GRAVE
EANIENE
DIARY
FREEI
NI Medium UP VOTES
N Medium UPIVOTES
CUSTOMER SEGMENTATION AND
CUSTOMER PONEERTTTTTT REALITY AND COSMETICS
CAD MPS PLAKER
MOS
M.G.R.
Freeview
Freeview
Tahannam Ae Boyan
Gl Garraaa Bayyn
STREET FRIES
STREET FRIES
HD
HD
Cheaper. Rerrer
Cheaner
EABY Cata BOV!
EABY BOYA
Taster
Laster
wbs
wbs
Mary
TOM CLANK'SS INTO
DRUSAUR
MANACEMENT
Sponish Holidays
POLO
INTEGRIYONE
CAR
CAN BARDER
AAP
Technoldgs
TOM CLANCY
NORTH
COMMAAD AUTHORITY
Figure 12: Visualization of some OCR annotations in MARIO-10M.
21

---

G
Samples in MARIO-Eval
Table Details of each subset in MARIO-Eval.
Subset
Size
Off-the-shelf Captions
GT Images
LAIONEva14000
4,000
TMDBEva1500
500
OpenL1braryEval500
500
Draw-BenchText [72]
Draw DrawText TexeCrravvee [46]
175
X
Chinese [49]
218
X
As illustrated in Table 7, MARIO-Eval contains 5,414 prompts with six subsets. The ground truth images of some samples are shown in Figure 13, and captions for each category are shown below:
LAIONEval4000:
Royal Green' Wristband Set
Are 'Digital Nomads' in Trouble Well Not Exactly
'Sniper Elite' One Way Trip A Novel Audiobook by 'Scott" McEwen Thomas Koloniar Narrated by Brian Hutchison
Travel Artin' Logo
Falls the 'Shadow' Welsh Princes 2
TMDBEva1500:
A movie poster with text 'Heat' on it
A poster design with a title text 'Deadpool 2'
A TV show poster named 'Ira Finkelstein Christmas'
A movie poster with logo 'Playing for Change Songs Around The World Part 2 on
A movic poster titled 'Dreams of a Land'
O OpenLibraaryEval500.
A book cover with title text of *On The Apparel Of Women'
A book with text 'Precalculus' on it
A book design with title of 'Dream master nightmare"
A book cover with logo "Thre Poetical Works of Constance Naden' on it
A book cover with title text of "Discovery'
Traw
A storefront with 'Hello World' written on it.
A storefront with "Text to Image' written on
A sign that says 'Diffusion'.
A sign that says 'NeurlPS'.
New York Skyline with 'Google Research Pizza Cafe" written with fireworks on the sky.
DrawTextCreative:
a grumpy sunflower with a 'no solar panels' sign
A photo of rabbit sipping coffee and reading book. The book title 'The Adventures of Peter Rabbit' is visible.
22

---

graffiti art of the text 'free the pink' on wall
A professionally designed logo for a bakery called 'Just What Kneaded',
scholarly elephant reading newspaper with the headline 'elephants take over the world'
ChineseDrawTest
A street sign on the street reads 'Heaven rewards those who work hard'
There is a book on the table with the title 'Girl in the Garden'
Kitten holding sign that reads want fish'
A robot writes 'Machine Learning' podium
In hospital, sign that says 'Do Not Disturb'
OUTTA COMPTON
RISK EVERYTHING
Adtanture Assile
'Adveurure Awaits'
Advanced
Risk Everything
'50 positive perenting quotes' gentle parenting peacetul percnting gentic discipline
Home Blanket by
Screening Of Strnight
TYGVOLUME DAY BREAKS DAWN LP
Katic Rose Artist Shup
Compron' Courtesy Of 11881987 Aug 11th
LIFE AFTERLIFE
Adom Crossing
A poster with title text Frankk
A show poster 'Burden of Dreams
poster with title of Life to Aflertife Death and Back"
A movie poster with title of "Ster Wans
movie poster logo 'edams Crossing' on
Bufteerit''s Suuprriaa
Marrir Harraker
BOULEZ
cover titled 'Sleoping Beauty'
cover titled Bouuee Coaducting'
design with a title text "Butterllys Surprise'
book title text of 'Expresiooissoo'
book
'The master mummer' ea
Figure 13: We demonstrate five samples for LAIONEva14000 (top), TMDBEva1500 (middle), and OpenLibrary500 (bottom).
23

---

H Implementation Details of Evaluation Criteria
To evaluate the performance of TextDiffuser quantitatively, we utilize three criteria, including FID, CLIPScore, and OCR Evaluation. We detail the calculation of each criterion below.
FID. We calculate the FID score using the pytorch-fid repository. Please note that the proposed MARIO-Eval benchmark's ' three subsets (DrraTexxrreattinn DrawBenchText and Draw Text)) do not contain ground truth images. Therefore, we utilize 5,000 images in the other three subsets (LAIONEval4000, TMDBEva1500, OpenLibraryEvaa5000 the ground truth images. We calculate the FID score using the 5,414 generated images and the 5,000 ground truth images.
CLIP Score. We calculate the CLIP score using the clipscore repository. However, with the FID score, we cannot calculate the CLIP score for the DrawTexttreativnn DrawBench and MhineseDraTTer subsets due the lack of ground truth images. Therefore, we only calculate the score on LAIONEvaL4000, TMDBEva1500, and OpenLibraryEvallee subsets and report the average CLIP score.
OCR Evaluation. For the MARIO-Evar benchmark, we use quotation marks to indicate the keywords that need to be painted on the image. Taking the caption [A cat holds a paper saying 'Hello World'] as example, the keywords are 'Hello' and 'World'. We then use Microsoft Read API to detect and recognize text the image. We evaluate OCR performance using accuracy, precision, recall, and F-measure. If the detected text matches the keywords exactly, it is considered correct. Precision represents the proportion detected text that matches the keywords, while recall represents the proportion of keywords that appear in the image. We report the mean values of precision and recall, and calculate the F-measure using the following formula:
2 Precision Recall
F-measure
(5)
Precision Recall
Visualization of Layouts Generated by Layout Transformer
visualize some generated layouts in Figure 14, showing that the Transformer can produce
reasonable layouts.
Ethics 101
Ethics
Google
Google Resseachh Pizza Cafe
NeurlPS
NeuriPS
10
Cafe
sign that says PPu
robot writing 'Ethies in chalk on a blackboard.
A storefront with GGoogle Research Pizza Cafe' written on it.
shoe for bateey pakey
shoe for
Qusteio North America
Quats of North America
Energy Tonic
Energy S Tonic
antique bottle labeled 'Energy Tonic'
giant
A poster ritled 'Quails of North America', showing diffierent kinds of ququiis..
caption 'shoo for bokey pokey'
Deep Learning
Deep Learning
Hello World
Hello World
Google Brain Toronto
Google Brain Toronto
storefront with 'Deep Teaarning' written it.
storefront with 'Hello World' written it.
sign that says 'Google Brain Toronto'.
Figure 14: Visualization of the generated layouts and images.

---

Experiment without Explicit Guidance of Segmentation Masks
As shown in Figure 15, we try to explore the generation without explicit guidance. For example, according the first row, (i.e., remove the content and only provide the position guidance). We observe that the model can generate words similar to keywords but contain some grammatical errors (e.g., missing "I" in "Hello""). Further, according to the second row, we train TextDiffuser without segmentation masks (i.e., remove both position and content guidance). In this case, the experiment is equivalent to directly fine-tuning pre-trained latent diffusion model on the MARIO- 10M dataset. The results show that the text rendering quality worsens, demonstrating explicit guidance's significance.
the value of character pixels to I and non-character pixels
w/o Con
Helo
Happy
Good Morning
SOSTOTP
World
w/o Pos & Con
Happery HPPPy y
I TOP
Helolo Worl Word
Goood Moniing
w/ Pos & Con
Hello
Happy
Stop
Good Moming
World
A text image of 'Hello World'
A text image of 'Happy'
A board says 'Stop'
board says 'Good Morning'
Figure 15: Visualization of generation without explicit guidance.
25

---

K Baseline Methods Experimental Settings
We introduced all baseline methods and their experimental settings when we used to compare them with the TextDiffuser as follows.
DALL-E [62] utilizes a text encoder to map a given prompt into a corresponding representation space. A prior model is then employed to map the text encoding to an image encoding. Finally, an image decoder generates an image based on the image encoding. Since there is no available code and model, we obtain the results using the provided API"
Stable Difflusion (SD) utilizes CLIP [60] encoder obtain the embedding of user prompts, pre-trained VAE to encode original images and conducts the diffusion process in the latent space for computation efficiency. We use the public pre-trained model "nunwayml/ttabeediffteee wawpuabb--difuiioo//l-aaaa on Hugging Face diffusers [82]. The number of sampling steps is 50, and the scale of classifier free guidance is 7.5.
based
Stable Diffusion XL (SD-XL) is an upgraded version of SD, featuring more parameters and utilizzig a more powerful language model. Consequently, can be expected better understand prompts compared to SD. As the source code and model are not publicly available, we obtained the generation results through web API'.
Midjourney" is commercial project that runs on Discord, allowing users to interact with a bot via the command-ling line interface. We generated images using the default parameters of Midjourney. For example, we can generate an image using the following command: /imagine an image of 'hello world' in Midjourney.
ControlNet [102] aims to control diffusion models by adding conditions using zero-convolution layers. We use the public pre-trained model authors and the implementation from Hugging Face diffusers [82]. For fair comparisons, we use the printed text images generated by our first-stage model to generate Canny maps the condition of ControlNet. We use default parameters for inference, where the low and high thresholds canny map generation set 100 and 200, respectively. The number of inference steps is 20, and the scale of classifier-fret guidance is 7.5.
released by ControltNet
DeepFloyd [12] designs three cascaded pixel-based diffusion modules to generate images of increasing resolution: 64x64, 256x256, and 1024x 0224. All stage modules use frozen text encoders based on T5 Transformer [12]. Compared with CLIP [60], the T5 Transformer is powerful language model that enables more effective text understanding. We use the public pretrained models released by DeepFloyd authors and the implementation from Hugging Face diffusers [82]. We use default models and parameters for inference, where the three pretrained cascaded models are "DeepFloyd/IF- I-XL-vI.0". "DeepFloyd/IF- Flloy//FFF--I1--v//// 11---11.0", and
http:///ttabe-ddiiiiooooooo uutpsiiiiioxxuusullee" upscaler".
"hupss//heta.draamsttiio dreamstudio wwwp:tdddii..iiinnneeeee ai/generate
bttps:/wwww.mddouumey..oom
26

---

Visualization of More Generation Results by Our TextDiffuser
problem solving
Let Your Light Shine
LAIONEva14000
Team
cctvil
steps skills
Fam
'Let Your Light Shime 18k overlay
Teant hat
locksmith 'cev' sign
'prublem solving slepd and 'skills'
Thankogiving 'Pan Mens I Shirt
Cheer new
Lisboa
History
Valle
Settie Hostal
The Witch' 'Tale'
'Cheef the Your
Time Out
Tlistory'
Magazine
cHook Tales of Shupidity Strangeress
Sampo Hostol
Gretet
argentino
Hansel
Society
Persoraiiees
Show
TMDBEva1500
poster ritled Tango argentine!
poster tided 'Geetel Hamsel
pester rilled 'Pensnalitied
poster tiled 'The Kindengatten Show
llihh Society
Wendy
Deep State
The Big LEbRRisk ki
Short 1 Term 12
Dry
poster text "Wndyy and Luzy'
poster logo Deep Stane' on
poster logo The Big Lebowski
muvic puster of 'Soot lemn
TV show logo
Knything
Manster Map
Marco Polo
ceto
Old Harthill
Neu
pogsigle
OpenLibrary Eval500
cover named Anything possible'
A cover named All World 'Monster Mup
book of Marco Polo
A cover tided The world cats
A book with text Old Harthill on
Waye Walker
Green Lantern
Tileland Ireasure
math for and Food
book cover with logo Programming Skills
cover named
cover named
book of 'Tideland
A book with text 'Math for Life
Through
Figure 16: Visualization of more generation results by TextDiffuser in LAIONEva14000, TMD- BEva1500, and OpenLibrary500.
27

---

shioe
Fall is here
free the
own
for
your sole
do toDeh
hckey
pink
Draw TextCreative
poiny
giant shoe, with the caption 'shoe for hokey pokey'
'Fall is here' written in aurumnn floating lake
graffiti of the text 'free the pink wall
showing attacking shoe, with message 1I own your shoe'
plant in fancy pot with 'dd not touch' sign
There no planet
ready for the
Recipes from Peru
magma
diamonds
A I-shirt with the message 'There no planet B
pillow in the shupe words 'ready for the weekend!
volcuno crupting, text 'magma' in red
logo company 'diamondss with diamon shape of heart
large recipe titled 'Recipes from Perul.
NeurIPS
Diffusion
Hello World
Researchh Pizza Cafe
Dee Leamnng
Draw DrawBench Bench
sign says 'NeurlPS..
New York Skyline with 'Diffassion' written with fireworks sky.
storefrent 'Hello World' written on it.
storetiont GGoogee Research Pizza Cafe writton on
storefiont with 'Deap I Learingg written on it.
Google Brain Toronto
Hello World
Google Redearch
Diffusion
NeuripS
Pizza
A storefront with 'Goolle Brain Toronto written
that 'Hello World.
A sign that says 'Diffusion'.
sigm that says 'Google Research
storetiont with 'NeurPSS' written on it.
Pizzu Cafe
pig
Planet Earth
Danger Minefield
HIGHT to learn
No Gambling
prize pumpkin
ChineseDraw Text Text
Prohibition sign "No Gambling" hung on the entrance
A globe with the words "Planet written bold lettens with continents bright colors
photo of roses surrounded by sign the distance that reads "Dunger Minefield"
A newspaper headline read "Local pig cats pumpkin" and photo showed pumpkin
Little raccoon holding sign that reads
casino
want learn"
Do not feed
Fairy Tales
Science
No Smoking
LOVE
"No Smoking" sign placed hotel
A sign saying "Do food" in the aquarium
A pink bottle that says "LOVE"
A little girl is holding book with the words "Fairy Tales" in her hunds
Books with the word "Scienee" printed them
Figure 17: Visualization of more generation results by our TextDiffuser in DrawTexiCreative. DrawBench Text, and Chi
28

---

Frandlng
Dedia
dark
A hand-drawn blueprint for time machine, with caption 'Time Traveling Device'.
a mouse with flashlight saying 'i am afraid of the dark
y anariur
scholarly clephant reading newspaper with the headline 'elephants take over world'
photo of sign with 'having dog named shark at the beach was a mistake'
Sec
rdd
make
if Safe
a picture the earth with the words 'save the carth' in a circle
lizard sitting on a bascball field home plate, with the words 'make it safe' in speech bubble
WAT!
OUD PART
Bam
a model of 1980s-style computer with the text 'my old habit' on the screen
a pencil sketch of tree with the title 'nothing to tree here'
STOP
STOP
Please
Please Adopt Me!
Adopt
dog holds paper saying please adopt me
stop pizza
CUN
SUN CAMQUOTIS
Hello, World.
Hello, World
hello world banner
a stay calm quotes poster
Figure 18: Visualization of more generation results by our TextDiffuser for the text-to-image with template task.
29

---

I ANDIANT
T ANDIANT
TN ANDIANT
EYE ON SECURITY
EYE ON SECURITY PROTECTION
EYE ON SECURITY PROTECTION
VOUCHER
VOUCHER
HARRCUT
VOUCHER HAIROUT C50
c50
E50
Tiecie SMALL SPACE LIVING
Update
Upgraae
Upgrade
SPACE LIVING
SPACE LIVING
Shift
Shift
Gcofie tea proud @B brother!
Ocofie te' proud B510 sister
Ocoffe lea proud 1 sister
CREATE YOUR OWN 2014 CALENDAR!
CREATE YOUR DUVII 2023 CALENDAR!
CREATE YOUR OWN 2023 CALENDAR!
CAUTION
CAUTION
CAUTION
VIDISTON WE ARE PROBLEM
HOUSTOR WE ARE GREAT PEOPLE
HOUSTON ARE THE GREAT PEOPLE
PROTECTIVE
EQUIPMENT
EQUIPMENT
PULS FURn
Pins
HENU
MEM
MEH
SOONDOK
Mono
Figure 19: Visualization of more generation results for text inpainting. The images above the dash lines are from the test set of MARIO-10M, while the images below the dash lines are collected from the Midjoumny community.
30

---

M More Details about User Study
User study on the whole-image generation task. The questionnaire consists of 15 cases, each of which includes two multiple-choice questions:
Which of the following images has the best text rendering quality?
Which of the following images best matches the text description?
In particular, the first question focuses on the text rendering quality. Taking Figure 20 as an example', we expect the model to render the word "EcoGrow" accurately (i.e, without any missing or additional characters). The second question, on the other hand, focuses on whether the overall image matches the given prompt. For example, in Figure meet the requirement in the prompt that the letter looks like plant. We instruct users to select the best option. In cases where multiple good options are difficult to distinguish, they could choose multiple options. If users are unsatisfied with the options, they could decide not to select any.
(G), although the generated
is correct, fails to
AVOVE
ECoGroW
ecoogrow
(A)
(B)
ECCCOY
(G)
((8115) lego fhr the companyy TeaoChrww where the lettere look like plants.
Figure 20: One case the user study for the whole-image generation task.
User study the part-image generation task. We aim to let users vote the quality of text inpainting (from 4 to 1, the higher, the better). We also designed two questions:
How is the text rendering quality?
Does the drawn harmonize with the unmasked region?
Specifically, the first question concentrates the accuracy of the text. The second question focuses on whether the generated part is harmonious with the unmasked part (ie.. whether the background and texture consistent).
TECH HOUSE
TECH HOUSE
TECH HOUSE INTERESTING TERRACE PARTY
INTERESTING TERRACE PARTY
TERRACE PARTY
Figure 21: One case in the user study for the part-image generation task.
(AA TextDiffuser; (B) DALL-E; (C) SD: (D) IF: (E) SD-XL: (F) Midjourney: (G) ControlNet.
31

---

Generating Images without Text
To show the generality of TextDeffuser, we experiment with generating images that do not contain texts and show results in Figure 22. Although TextDiffuser is fine-tuned with MARIO-I0M, still maintains a good generation ability for generating general images. Therefore, users have more options when using TextDiffuser, demonstrating its flexibility. We also provide quantitative evaluations to demonstrate TextDiffuser's generality in generating non-text general images. We compare TextDiffuser with our baseline Stable Diffusion 1.5 as they have the same backbone. For a quantitative evaluation, the FID scores of 5,000 images generated by prompts randomly sampled from MSCOCO are as in Table 8. The results indicate that Tex(Diffuser can maintain the ability to generate natural images even after fine-tuning the domain-specnt dataset.
Table 8: FID scores on MSCOCO compared with Stable Diffusion.
Sampling Steps
Stable Diffusion
TextDiffuser
50
26.47 27.02
27.72 27.04
100
soble Diffinion
TestDimwerr (Oar))
Sabie Diffiiioo 1.5
Restiittuuerr (Uurs)
Diffiiio 1
JestDitlueer (Chrg)
Acem,, orgniied bwihroom with accessories
horse with the sky the backpound. sable Difiiiiii
youg riding skis dowu cuvered slope Sibble Dilloci
sable Difinieu
TeatDiewerr (Oae))
Testiilouerr (Ours)
TestDimuser
A Abble with survings of posiiccs on tays.
Daily Manor fialwing siecies subeuiit stuffed uninads. A Jusu ziding on luy of'a wave surfweed.
Figure 22: Visualizations of general images generated by Stable Diffusion 1.5 and TextDiffuser.
32

---

Comparisons between TextDiffuser and a Text Editing Model
We visualize some results in Figure 23 compared with a text editing model SRNet [88]. Please note that the introduced text inpainting task differs from the text editing task in three aspects: (1) The text editing task usually relies on the synthesized text image dataset for training (synthesizing two images with the different text given a background image and font as pairs). In contrast, the text inpainting task follows the mask-and-recover training scheme and can be trained with any text images. (2) Text editing emphasizes the preservation of the original fonts, while text inpainting allows for greater freedom. For example, we conduct four samplings for each case, and the generated results exhibit diversity and present reasonable font styles. (3) Text editing tasks cannot add text, highlighting the significance of the introduced text inpainting task.
Original Image
Editing Result
Our Result
Our Result 2
Our Result 3
Our Result
PEDESTRIANS
Sessions
Sessions
Sexsions
Seasions
Footpath. To Colchester and Greengteäd
ASSYRIAN on UNFLADGING FRY DEVASTATES
ASSYRIAN UNELAGGING DEVASTATES
ASSTRIAN OR UNFLACGING DEYASTATES
ASSTRIAN ON UNFLAGGING DEVASTATES
ASSYRIAN ON UNTLAGGING DEVASTATES
Campus Shop
Rankin dork
Rankin Dork
Rankin Dork
Rankin Dork
Rankin Dork
RIVERSIDE WALK
1
ALAPENOS Reds
ALAPENOS
ALAPENOS
ALL ALARENDS
ALAPENOS
Rots
* Reds *
MIREER
Rods
Figure 23: Comparison with text editing model. Four cases are obtained from the paper of SRNet.
33

---

P Experimental Results of Text Removal
We demonstrate some results of text removal in Figure 24, and the cases are obtained from the paper of EraseNet [44]. We can easily transform the text inpainting task into text removal task by providing a mask and setting all regions to non-character in the segmentation mask. Experimental results demonstrate that our method achieve results similar the ground truth.
Original Image
Ground Truth
Inpainting Mask
Our Result
YELLOWNDEE
WATER OVER ROAD
DO NOT MAM ENTER
Welcome Porre Stllng- Mark Lengh
Figure 24: Experimental results on text removal. Four cases are obtained from the paper of EraseNet.
Q Limitations and Failure Cases
We observe failure cases when generating images with small characters and from long text.
Generating images with small characters. TextDiffuser uses the VAE networks to encode images into low-dimensional latent spaces for computational efficiency following latent diffusion models [67, 49, 2]. However, the compression process can result in losing details when generating images with small characters. As illustrated Figure 25, we observe that the VAE fails reconstruct small characters, where reconstructed strokes are unclear and reduce the legibility of the text. According the generated images, the small characters appear have vague or disjointed strokes (e.g., the character 'I' in 'World' and character 'r' in 'Morning)), which could impact the readability. As shown in Figure 26, we notice that using more powerful backbone, such as Stable Diffusion 2.1, can mitigate this issue. When the image resolution is enhanced from 512x512 to 768 x 768 using Stable Diffusion 2.1 (instead of 1.5), the latent space resolution also increases from 64x64 to 96x96, enhancing the character-level representation. As the cost, the inference latency rises from 8.5s to 12.0s with a batch size of Therefore, how to render small characters while maintaining the same time cost is worth further study.
Generating images from long text. We observed failure cases when generating images from long text with many keywords, where the generated words in the layouts are disordered and overlapped,
34

---

Original Image
Reconstructed
Generated Image
B Pitg DEx
To BAG Duxi
Motsturkingg
World
Knirlandia
Knirlandia
SEES
8 &
Motning
CLARA PARKES
CLARA PARKES
board Good Morning'
Figure 25; The issue of generating images with small characters.
Wel Novs
Having sharh
having shark
Big
Big Extremely
carned ahork
seneethhat was ninbitoio
Extemmely
Lan Eitaraly Doviee
editrabe
Travsing
Travding
Domaratod by
SDJ.5 board saying having dog named shack the beach was mistake
SD2.1
SD1.5
SD2.1
hand-drawo biseprint for time machine
Figure 26: Pre-trained on high-resolution Stable Diffusion 2.1 enhances the legibility of small text.
as shown in Figure 27. One possible reason could be that training examples containing numerous keywords tend have more noise (i.e.. those images usually contain dense and small text), leading to a higher likelihood of detection and recognition errors. To address this, we could consider enhancing the capability of OCR tools in the future mitigate the noise.
have
My friend
pleasedo
my sport
favorite
Argam durin 17
please
not
eat...
my
ping
is
pong
thaglpt
I hhve
My triend
please dea
my faunite
sport
dkgam
please
17
th7gopt
mmy
ping
pong
cat
an image of") have dream during my 17"
a board of friend please adopt this cat'
meme of 'please not eat my hamburger'
book of 'my favorite
sport is ping pong'
Figure 27: The issue of dealing with large number of keywords.
35


Reviews:
[{"review_id": "ekxvF6a2FY", "reviewer": "Reviewer_cdWT", "summary": "This paper introduces a model for generating visually appealing and coherent text within diffusion models. It also presents the MARIO-10M dataset and the MARIO-Eval benchmark for evaluating text rendering quality. Experimental results demonstrate their method's flexibility and controllability in creating high-quality text images and performing text inpainting.", "questions": "See the weakness", "limitations": "The rationale behind this paper, using seq-2-seq generation, is reasonable. However, there is not a clear explanation of how the generation of rich-text is achieved.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper include the proposal of TextDiffuser, a flexible and controllable framework based on diffusion models, with two stages for layout generation and fine-tuning. ", "weaknesses": "The paper does not explicitly mention how the model handles the generation of rich-text images when the number of queries is limited. However, it raises concerns about the efficiency of the model when dealing with a large number of queries.\n\nWhen there is a sequential relationship between boxes in the context of generating rich-text images, the paper does not specifically address how the model handles the ordering of boxes.\n\nWhile part-image generation seems reasonable, it could be computationally slow when dealing with a large number of texts as it requires predicting numerous positions.\n\nThe paper does not mention the specific number of texts per individual image in the mentioned databases in the main paper, which could be important information to provide.\n\nRegarding image resolution, the paper suggests that it can be low when dealing with single texts. However, when generating a large number of texts, high-resolution generation becomes necessary. It is essential for the authors to address how they ensure the accuracy of each generated text and the efficiency of text generation in such cases.\n\nUsing sequential generation for layout masks is an interesting point. Have the authors considered directly using diffusion to predict layout masks? It would be beneficial to have more comparative analysis between the two methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback. We've taken care to address each of your concerns below.\n\n**Handling of rich-text images with limited queries & efficiency with a large number of queries:**  \nOur first stage of Layout Generation leverages an auto-regressive Transformer whose prediction time correlates with the number of queries (keywords). Meanwhile, the second stage of image generation is independent of the number of queries. The time cost of these two stages, based on experiments conducted three times, are shown below:\n\n| #keywords | Layout Generation (s)|  Image Generation (s) |\n|-----------|--------------|--------------|\n| 1         | 1.07±0.03    | 7.12±0.77    |\n| 2         | 1.12±0.09    | 7.12±0.77    |\n| 4         | 1.23±0.13    | 7.12±0.77    |\n| 8         | 1.57±0.12    | 7.12±0.77    |\n| 16        | 1.83±0.12    | 7.12±0.77    |\n| 32        | 1.95±0.28    | 7.12±0.77    |\n| ...       | ...          | 7.12±0.77    |\n\nAs illustrated in the table, our model's efficiency when handling a large number of queries is commendable, primarily due to our use of a two-layer Layout Transformer. What's more, it's desirable to provide a layout template for image generation, which incurs no time cost in model layout prediction.\n\n**Handling ordering of boxes in rich-text images:**  \nOur Layout Generation process considers text keywords in the order provided by the user and outputs their box positions accordingly. While the box generation is sequential, the resulting image box order is spatial (two-dimensional). This spatial box ordering is learnt by large-scale real-world images (e.g., MARIO-10M) through our Layout Transformer in the context of rich-text images.\n\n**Computational efficiency in part-image generation with abundant text:**  \nTo clarify, our part-image generation occurs during the second stage of image generation, which is guided by a predefined text position mask. Thus, there's no added time cost associated with position prediction. For efficiency regarding a large number of queries in the first Layout Generation stage, plese see our first response point above.\n\n**Specific number of texts per image in databases:**  \nWe appreciate your suggestion. In our revised paper, we will integrate the following table detailing the word count per image in our proposed MARIO-10M dataset:\n\n| #Words |  #Images  | Ratio |\n|--------|-----------|-------|\n| 1      | 592,153   | 5.9%  |\n| 2      | 1,148,481 | 11.5% |\n| 3      | 1,508,185 | 15.1% |\n| 4      | 1,610,056 | 16.1% |\n| 5      | 1,549,852 | 15.5% |\n| 6      | 1,430,750 | 14.3% |\n| 7      | 1,229,714 | 12.3% |\n| 8      |   930,809 |  9.3% |\n\nWe can see from the table that most images contain between 3 to 6 words, while images with one word or eight words are less frequent.\n\n\n**The Accuracy and Efficiency of High-resolution Image Generation:**  \nWe agree that high-resolution generation enhances the quality of rich-text and small-text image generation. Our TextDiffuser can seamlessly transition to high-resolution image generation by deploying a more advanced diffusion backbone. For instance, when we augment the image resolution from 512x512 to 768x768 using Stable Diffusion 2.1 (instead of 1.5), the latent space resolution also increases from 64x64 to 96x96, enhancing our character-level representation. Please refer to Figure(a) in the attached PDF for a visual demonstration of the *accuracy* of our generated text. The inference latency for 512x512 and 768x768 resolutions are 8.5s and 12.0s respectively with a batch size of 1. Thanks to our employment of an *efficient latent* diffusion model, the time increment is insignificant.\n\n**Direct diffusion prediction for layout masks versus sequential generation:**  \nIn our initial investigations, we explored diffusion models for layout generation. However, we encountered overlapping problems during mask generation for *consecutive characters* and *multiple lines of text*, as illustrated in \"Figure(e) in the attached PDF\". This overlap stems from the non-autoregressive nature of diffusion models, which generate all text simultaneously. To address this challenge, we turned to the auto-regressive Transformer for layout generation.\n\nWe believe that our contributions, including the TextDiffuser model, the MARIO-10M dataset and the MARIO-Eval benchmark, push the boundaries of current text image generation. We appreciate your constructive feedback and hope these clarifications address your concerns. We will include these discussions in our revised paper.\n"}, {"review_id": "MsOfqw9u6O", "reviewer": "Reviewer_sG87", "summary": "This work presents an approach to enhance the text rendering ability of a text-to-image diffusion model. The authors introduce a new diffusion model called text-diffuser that leverages image captions and text segmentation masks to generate text images. They also collect a large-scale image dataset MARIO-10M, which includes explicit text information such as segmentation masks and OCR annotations. The experimental results indicate a significant improvement in text generation ability but a slight decrease in image quality.", "questions": "* It would be interesting if we can control the style of generated texts through language descriptions on textdiffuser. For example, the image s generated with this caption: a bear holding a board with a **red/purple/yellow** \"hello world\"? It would be beneficial if the authors could provide some generation examples showcasing this capability.", "limitations": "Authors discussed the limitation of the trained model, including small text generation and multiple-word generation.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The large-scale text image dataset with text annotations MARIO-10M is commendable is valuable for further research in this domain.\n* Incorporating character-level segmentation masks to enhance the text rendering ability of a diffusion model is a novel idea. The results presented in the paper demonstrate the effectiveness.", "weaknesses": "* The paper mentions that the layout generation model is trained to generate text segmentation masks, which might not be suitable for scene texts with significant perspective changes.  Most of the generation examples lack realistic text styles with perspective changes and complicated text layouts.\n* Some GAN-based works [1, 2, 3] exploring scene text editing and a recent diffusion-based scene text editing work [4] could be relevant for part-image generation evaluation. Authors could consider incorporating a discussion and comparison with these related works.\n\n\n[1] Wu, et al . “Editing Text in the Wild.” https://doi.org/10.1145/3343031.3350929.\n\n[2] Qu, et al. “Exploring Stroke-Level Modiﬁcations for Scene Text Editing,” https://ojs.aaai.org/index.php/AAAI/article/view/25305\n\n[3] Krishnan, et al. “TextStyleBrush: Transfer of Text Aesthetics from a Single Example.”  http://arxiv.org/abs/2106.08385.\n\n[4] Ji, et al. “Improving Diffusion Models for Scene Text Editing with Dual Encoders.” http://arxiv.org/abs/2304.05568.\n \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your high recognition of our work, particularly the novelty of our TextDiffuser and the significant contribution of the MARIO-10M dataset. We've responded to your comments as follows:\n\n**Layout generation for scene text with perspective changes:**\nOur MARIO-10M dataset reveals that about 90% of the text regions maintain a horizontal orientation with rotation angles smaller than 5 degrees without perspective changes. Hence, our layout generation model is designed to predict horizontal bounding boxes by detecting the coordinates of their left-top and bottom-right points. Adapting our model to predict more realistic scene text is indeed feasible by detecting enhanced coordinates, such as eight coordinates for four points. We'll clarify this adaptability in our revised paper.\n\n**Discussion and comparison of text-editing and part-image generation:**\nWe appreciate your suggestion of relevant works [1,2,3,4] in the scene text editing domain. We have taken [1] as a representative work to discuss the differences between text editing and text inpainting (part-image generation), as detailed in Footnote 1 and Appendix O. While we have cited [1,2], we will also incorporate references [3,4] in our updated version.\n\n**Control over the style of generated texts through language descriptions:**\nWe are happy to showcase TextDiffuser's innovative capability in controling style of generated texts through language descriptions. As shown in Figure(b) in the attached PDF, TextDiffuser's successfully generates texts in varying colors aligned with the language descriptors.\n\nWe will incorporate these insightful discussions in the final version to inspire future explorations and applications in the field.\n"}, {"review_id": "mVFELjrYv7", "reviewer": "Reviewer_V9VR", "summary": "The paper proposes TextDiffusers, a 2-stage model to generate images with text per the input text prompt. TextDiffusers also supports text inpainting.  The two stages consist of (1) estimation of layout of keywords (inspired by Layout Transformer) to get character-level segmentation masks and (2) image generation conditioned on the character-level segmentation masks.\n\nThe paper also contributes to MARIO-10M, a large data set of image-text pairs along with rich OCR annotations. Benchmark MARIO-Eval with 5414 prompts is created from MARIO-10M and few other existing datasets, in order to evaluate quality of rendering text in the image.\n\nStage 1 consists of novelty such as encoding of width of keywords. Encoding width of keywords improves IoU, especially for shallower Layout Transformer. Stage 2 introduces character-aware loss and extends the denoising loss to include character level segmentation masks.\n\nThe paper contains implementation details for reproducibility, comparison against the states of the art demonstrating large improvements in OCR quality metrics, comprehensive ablations studies, several qualitative samples and limitations.\n", "questions": "Please see list of weakness. \n\nQ1) According to Line 188, the maximum length of tokens (L) is limited to 77. Do all captions in the dataset contain <= 77 tokens?\n\nQ2) Related to Q1, DeepFloyd uses T5-XXL which is much more powerful (as observed by Imagen). Although L<=77, do you have insights on the richness of embedding that may give it a higher edge over TextDiffusers?\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "S1) Novel approach to generate images with text\n\nS2) New data set for MARIO-10M along with benchmark MARIO-Eval\n\nS3) Comprehensive experiments demonstrating large improvements against various states of the art on OCR metrics\n", "weaknesses": "W1) While the methodology introduced by the paper seems solid, the main weakness is comparison against strong baseline which was also trained on training split from MARIO-10M. Appendix J alludes to equivalency of fine-tuning of pre-trained model, but Stable Diffusion model is not the best when it comes to producing visual text. It is understandable that official code for Imagen, Parti are not available. Perhaps DeepFloyd could have been fine-tuned on MARIO-10M. From Table 4, DeepFloyd is also better on Fidelity (lower FID score). It is not clear if it would perform better than TextDiffusers with this additional fine-tuning. \n\nW2) Information and comparison of model size and training/inference latency are missing.\n\nW3) Appendix N / Figure 14 in the supplementary material includes a handful of sample generations without text. However, it is not clear if TextDiffusers loses its capability to generate such images in the general sense. Qualitative and Quantitative evaluations to test this are missing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comprehensive review and feedback on our work.\n\n**Comparison with Strong Baselines Fine-tuning with MARIO-10M:**\nAs mentioned in Appendix K, DeepFloyd is better on Fidelity due to its use of two super-resolution modules to generate higher resolution images (1024×1024) versus our models' (512×512) and a stronger text encoder (T5 versus our CLIP). While the lack of DeepFloyd's publicly available training code and implementation details restricted us from retraining or reimplementing it, we acknowledge the potential benefits of fine-tuning it using MARIO-10M to improve its visual text generation performance. However, we assert that merely changing the dataset cannot improve DeepFloyd over TextDiffuser. We emphasize the importantce of our unique proposal of the explicit supervision for visual text rendering through character-level segmentation masks. As evidenced in our comprehensive ablation studies, this feature significantly improves text rendering accuracy and is missing in DeepFloyd.\n\n**Model Size and Latency Metrics:**\nThank you for highlighting this aspect. Our TextDiffuser builds upon Stable Diffusion 1.5 (859M parameters), adding a Layout Transformer in the first stage (+25M parameters) and modifying the second stage (+0.75M parameters), augmenting it by only about 3% in terms of parameters. It trains within about four days on eight Tesla V100 GPUs, resulting in 6.6 seconds per iteration, with an inference latency of 8.5 seconds per image.\n\n**Capability to Generate General Images:**\nWe provide qualitative and quantitative evaluations to demonstrate TextDiffuser's generality in generating non-text general images. We compare TextDiffuser with our baseline Stable Diffusion 1.5 as they have the same backbone. For a quantitative evaluation, the FID scores of 5,000 images generated by prompts randomly sampled from MSCOCO are as follows:\n\n| Sampling Steps |  Stable Diffusion  | TextDiffuser |\n|--------|-----------|-------|\n| 50     | 26.47     | 27.72 |\n| 100    | 27.02     | 27.04 |\n\nFor a qualitative evaluation, please refer to Figure(c) in the attached PDF, where we show more comparisons. We can see from the table and figure that TextDiffuser is highly competitive. These evaluations demonstrate that TextDiffuser maintains its capability in the general domain, primarily because our training data encompasses large-scale images from diverse real-world scenes.\n\n\n**Token Length in Captions:**\nA significant majority—99% of the captions in our dataset—are are within the 77 tokens limit.\n\n**Richness of DeepFloyd's T5-XXL Embedding:**\nWe agree that the T5-XXL encoder, as leveraged by DeepFloyd, holds the potential for richer textual embeddings as we have mentioned in Appendix K. However, as we analyzed in the answer of **Comparison with Strong Baselines Fine-tuning with MARIO-10M**, the absent of explicit visual text layout supervision in DeepFloyd, remains a deciding factor in the performance of visual text rendering.\n\nIn conclusion, we're grateful for your recognition of our approach's novelty, the dataset contribution, and the rigor of our experiments. We will revise our paper to clearly emphasize these in light of your instructive comments."}, {"review_id": "g8UJG9a3ue", "reviewer": "Reviewer_AzyM", "summary": "This paper proposed TextDiffuser to achieve accurate text rendering for diffusion models. TextDiffuser generates character-level text layouts to guide the text rendering process (image generation). The model is evaluated on 3 tasks including text-to-image, text-to-image with template, and text inpainting to demonstrate its flexibility and controllability. Additionally, the paper contributes the first large-scale text image dataset with OCR annotations, named MARIO-10M.", "questions": "1. How does the model perform on more complex characters such as Chinese characters? A simple layout guidance seems to only constrain the position and order of letters, without any specific improvement on the quality of generation details. I wonder if there is any method to improve the adaptiveness on other language?", "limitations": "Limitations and potential impacts have been discussed.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well motivated, which tries to solve the problem of generating text image with diffusion model. Also, the paper is well organized and easy to follow. \n2.\tThe TextDiffuser model is flexible, which can be adaptive with different conditions as shown in the experiments.\n3.\tA new dataset specifically designed for text rendering is proposed.\n", "weaknesses": "1.\tMismatch of character level layout and the real text. The character level layout is generated from standard fonts (rendered using toolkit such as opencv), however, the real character can be of any style or font, which leads to a mismatch of character level layout. In another word, guided by such a layout, the generated text will tend to have the same character interval and aspect ratio of the standard font, which means it will limit the generation diversity of fonts.\n2.\tGeneration diversity has not been considered. There exist several reasonable text styles for the same background image. As the proposed model cannot explicit control the attribute of texts such as color, layout, font, style, etc., it is important to measure the generation diversity of the model.\n3.\tThe performance comparison seems to be limited to general text-to-image diffusion models with no specific optimization on text painting. A comparison with some state-of-the-art text generation methods is necessary to show the quality of text generation. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback. We aim to address and clarify each point raised.\n\n**Character-Level Layout and Real Character Style:**\nThe primary role of our character-level layout is to inform and guide the position and content of visual text, without restricting it to specific styles or fonts. This design choice empowers TextDiffuser to dynamically adapt to varied text styles, as the model is trained on the diverse MARIO-10M dataset. To illustrate this, Figure 6 in Appendix I of our supplementary material shows the model interprets a standard font layout and coherently translates it into diverse fonts and styles according to the context.\n\n**Generation Diversity:**\nTo clarify, TextDiffuser can explicitly control text layouts by conditioning the second stage of image generation on the first stage's prediction or by user-provided layout templates. As outlined in Line 278 and visualized in Figure 15 of Appendix O, even with the same layout input, TextDiffuser exhibits diversity in text attributes such as color, font and style. Furthermore, Figure(b) in the attached PDF provides additional demonstrations of TextDiffuser's diversity and control capability, like color modulation via prompts. We will emphasize this diversity and capability more clearly in the paper.\n\n**Performance Comparison:**\nTo the best of our knowledge, state-of-the-art text generation methods like Imagen [67], eDiff-i [2], Character-Aware Model [42], and GlyphDraw [45] have not publicly released open-source code, checkpoints or APIs for throughout comparisom, as highlighted in Lines 240-241.\nStill, we initiated a comparison with Character-Aware Model [42] and the concurrent work GlyphDraw [45] using samples from their papers. In Figure(d) of the attached PDF, TextDiffuser performs better than these methods. For instance, [42] suffers from misspell issues (e.g., ‘m’ in ‘Chimpanzees’) due to its lack of explicit control and [45] struggles with rendering images containing multiple text lines. We will release our code to facilitate future comparisons.\n\n**Adaptability to Complex Characters, e.g., Chinese:**\nTextDiffuser is promising to handle complex characters like Chinese characters by collecting a Chinese version of the MARIO dataset and training a character-level segmentation network for Chinese characters. We highlight the effectiveness of our proposed explicit layout guidance in the quality of generation details due to its strong constraint not only on the position but also the content of each characters. The OCR evaluations in Table 4 quantitatively validate the accuracy of our text rendering. Concurrently, our qualitative experiment in Appendix J shows the drastic degradation in quality without explicit guidance. We believe our design of explicit guidance will benefit more complex characters generation.\n\nWe believe that our contributions of comprehensive model, dataset and benchmark are significant for future research and development in this field. We will release code, model, and dataset to advance the research community and broaden the applications of our approach."}]

Author rebuttal:
Thank you for taking the time to review. Enclosed in the attached PDF, we have provided some figures for reference.

* Figure(a): Pre-trained on high-resolution Stable Diffusion 2.1 significantly enhances the legibility of small text.
* Figure(b): Demonstration of using language descriptions to control the style of text.
* Figure(c): Visualizations of general images generated by Stable Diffusion 1.5 and the proposed TextDiffuser.
* Figure(d):  Comparison with Character-Aware Model and the concurrent GlyphDraw. The code is not available
for Character-Aware Model, the checkpoints and datasets are not available for GlyphDraw. So we compare the samples in their papers.
* Figure(e):  Layout generation using diffusion model and Transformer.

Meta-review: This paper presents a new method TextDiffuser to improve diffusion models' visual text generation ability by including the text layout in the diffusion generation process. It presents a large-scale text-rich image dataset MARIO-10M and a benchmark MARIO-Eval to measure text image quality, which could be useful in this field. Compared to other SOTA diffusion models, experiment results demonstrate the superior performance of TextDiffuser in generating correct visual text and high-quality images. All reviewers agree on the effectiveness of the proposed method and the contribution of the proposed text image dataset. During the rebuttal period, reviewers cdWT, sG87, and V9VR were satisfied with the author's response about the design of the text-layout generator (cdWT, sG87), style-text image (sG87), and non-text image generation (V9VR) and recommended acceptance. Reviewer AzyM questioned whether this work can be applied to complex characters such as Chinese characters and still recommended rejection after rebuttal.  However, the AC and other reviewers feel this limitation is beyond the scope of this work. The lack of this experiment does not affect the soundness and contribution of this work.
Decision: Accept (poster)