SlowGuess/NeurIPS_2023_OCR huggingface dataset:
OCR Evaluation. For the MARIO-Eval benchmark, we use quotation marks to indicate the keywords that need to be painted on the image. Taking the caption [A cat holds a paper saying 'Hello World'] as an example, the keywords are 'Hello' and 'World'. We then use Microsoft Read API to detect and recognize text in the image. We evaluate OCR performance using accuracy, precision, recall, and F-measure. If the detected text matches the keywords exactly, it is considered correct. Precision represents the proportion of detected text that matches the keywords, while recall represents the proportion of keywords that appear in the image. We report the mean values of precision and recall, and calculate the F-measure using the following formula:
$$
F - m e a s u r e = \frac {2 \times \text {P r e c i s i o n} \times \text {R e c a l l}}{\text {P r e c i s i o n} + \text {R e c a l l}}. \tag {5}
$$
I Visualization of Layouts Generated by Layout Transformer
We visualize some generated layouts in Figure 14, showing that the Transformer can produce reasonable layouts.
A sign that says 'NeurIPS'.
A robot writing 'Ethics 101' in chalk on a blackboard.
A storefront with 'Google Research Pizza Cafe' written on it.
An antique bottle labeled 'Energy Tonic'
A giant shoe, with the caption 'shoe for hokey pokey'
A poster titled 'Quails of North America', showing different kinds of quails.
A storefront with 'Deep Learning' written on it.
Figure 14: Visualization of the generated layouts and images.
A sign that says 'Google Brain Toronto'.
J Experiment without Explicit Guidance of Segmentation Masks
As shown in Figure 15, we try to explore the generation without explicit guidance. For example, according to the first row, we set the value of character pixels to 1 and non-character pixels to 0 (i.e., remove the content and only provide the position guidance). We observe that the model can generate some words similar to keywords but contain some grammatical errors (e.g., a missing "l" in "Hello"). Further, according to the second row, we train TextDiffuser without segmentation masks (i.e., remove both position and content guidance). In this case, the experiment is equivalent to directly fine-tuning a pre-trained latent diffusion model on the MARIO-10M dataset. The results show that the text rendering quality worsens, demonstrating explicit guidance's significance.
w/o Con
w/o Pos & Con
Pwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww
A text image of 'Hello World'
Figure 15: Visualization of generation without explicit guidance.
A text image of 'Happy'
A board says 'Stop'
A board says 'Good Morning'
K Baseline Methods Experimental Settings
We introduced all baseline methods and their experimental settings when we used to compare them with the TextDiffuser as follows.
DALL-E [62] utilizes a text encoder to map a given prompt into a corresponding representation space. A prior model is then employed to map the text encoding to an image encoding. Finally, an image decoder generates an image based on the image encoding. Since there is no available code and model, we obtain the results using the provided  $\mathrm{API}^4$ .
Stable Diffusion (SD) utilizes CLIP [60] text encoder to obtain the embedding of user prompts, pre-trained VAE to encode original images and conducts the diffusion process in the latent space for computation efficiency. We use the public pre-trained model "runwayml/stable-diffusion-v1-5" based on Hugging Face diffusers [82]. The number of sampling steps is 50, and the scale of classifier-free guidance is 7.5.
Stable Diffusion XL (SD-XL) is an upgraded version of SD, featuring more parameters and utilizing a more powerful language model. Consequently, it can be expected to better understand prompts compared to SD. As the source code and model are not publicly available, we obtained the generation results through a web API<sup>5</sup>.
Midjourney<sup>6</sup> is a commercial project that runs on Discord, allowing users to interact with a bot via the command-line interface. We generated images using the default parameters of Midjourney. For example, we can generate an image using the following command: /imagine an image of 'hello world' in Midjourney.
ControlNet [102] aims to control diffusion models by adding conditions using zero-convolution layers. We use the public pre-trained model "llyasviel/sd-controlnet-canny" released by ControltNet authors and the implementation from Hugging Face diffusers [82]. For fair comparisons, we use the printed text images generated by our first-stage model to generate Canny maps as the condition of ControlNet. We use default parameters for inference, where the low and high thresholds of canny map generation are set to 100 and 200, respectively. The number of inference steps is 20, and the scale of classifier-free guidance is 7.5.
DeepFloyd [12] designs three cascaded pixel-based diffusion modules to generate images of increasing resolution: 64x64, 256x256, and 1024x1024. All stage modules use frozen text encoders based on T5 Transformer [12]. Compared with CLIP [60], the T5 Transformer is a powerful language model that enables more effective text understanding. We use the public pretrained models released by DeepFloyd authors and the implementation from Hugging Face diffusers [82]. We use default models and parameters for inference, where the three pretrained cascaded models are "DeepFloyd/IF-I-XL-v1.0", "DeepFloyd/IF-II-L-v1.0", and "stabilityai/stable-diffusion-x4-upscaler".
4https://openai.com/product/dall-e-2
$^{5}$ https://beta.dreamstudio.ai/generate
<sup>6</sup>https://www.midjourney.com/
L Visualization of More Generation Results by Our TextDiffuser
LAIONEval4000
'Team' hat
locksmith 'ctv' sign
'problem solving steps' and 'skills'
'Let Your Light Shine' 18k overlay
'The Witch' Hunter 'Tale'
'Cheer' the 'new Year'
Time Out  
'Lisboa' Magazine
Stupid 'History'  
eBook Tales of  
Stupidity Strangeness
Thanksgiving 'Fam'Mems T Shirt
Photos of 'Sampa Host'
TMDBEval500
A TV show poster titled 'Tango argentino'
A movie poster titled 'Gretel Hansel'
A movie poster of 'High Society'
A movie poster titled 'Personalities'
A movie poster titled 'The Kindergarten Show'
A TV show poster with a title text of 'Wendy and Lucy'
A TV show poster withlogo ' Deep State' on it
ATV show poster with logo The Big Lebowski' 2
A movie poster of 'Short Term 12'
A TV show poster with logo 'The Dry' on it
OpenLibraryEval500
A book of 'Marco Polo'
A book with text 'Old Harthill' on it
A cover named 'Anything is possible'
A book cover with logo 'Programming Skills Through C' on it
Figure 16: Visualization of more generation results by our TextDiffuser in LAIONEval4000, TMD-BEval500, and OpenLibrary500.
A cover named All World 'Monster Map'
A cover named 'Wavewalker'
A cover named 'Green Lantern'
A cover titled 'The world of cats'
A book of 'Tideland treasure'
A book with text 'Math for Life and Food Service'
DrawTextCreative
a giant shoe, with the caption 'shoe for hokey pokey'
'Fall is here' written in autumn leaves floating on a lake
a graffiti art of the text 'free the pink' on a wall
A meme showing a cat attacking a shoe, with the message 'I own your shoe'
plant in a fancy pot with a 'do not touch' sign
A t-shirt with the message 'There is no planet B'
pillow in the shape of words 'ready for the weekend
a volcano erupting, with the text 'magma' in red
a logo for the company 'diamonds', with a diamond in the shape of a heart
A large recipe book  
titled  
'Recipes from Peru'.
DrawBenchText
A sign that says 'Diffusion'.
A sign that says 'Google Research Pizza Cafe'.
A storefront with 'NeurIPS' written on it.
ChineseDrawText
Prohibition sign "No Gambling" hung on the entrance of the casino
A globe with the words "Planet Earth" written in bold letters with continents in bright colors
A photo of roses surrounded by a sign in the distance that reads "Danger Minefield"
A newspaper headline read "Local pig eats prize pumpkin" and a photo showed a half-eaten pumpkin
Little raccoon holding a sign that reads "I want to learn"
A "No Smoking" sign is placed in the hotel
Figure 17: Visualization of more generation results by our TextDiffuser in DrawTextCreative, DrawBenchText, and ChineseDrawText.
A sign saying "Do not feed" in the aquarium
A pink bottle that says "LOVE"
A little girl is holding a book with the words "Fairy Tales" in her hands
Books with the word "Science" printed on them
A hand-drawn blueprint for a time machine, with the caption 'Time Traveling Device'.
a mouse with a flashlight saying 'i am afraid of the dark'
scholarly elephant reading a newspaper with the headline 'elephants take over the world'
rd sitting on a baseball field home plate, he words 'make it safe' in a speech bubble.
a picture of the earth with the words 'save the earth' in a circle
a 3d model of a 1980s-style computer with the text 'my old habit' on the screen
a pencil sketch of a tree with the title 'nothing to tree here'
a dog holds a paper saying please adopt me
a hello world banner
a stay calm quotes poster
Figure 18: Visualization of more generation results by our TextDiffuser for the text-to-image with template task.
Figure 19: Visualization of more generation results for text inpainting. The images above the dash lines are from the test set of MARIO-10M, while the images below the dash lines are collected from the Midjourney community.
M More Details about User Study
User study on the whole-image generation task. The questionnaire consists of 15 cases, each of which includes two multiple-choice questions:
- Which of the following images has the best text rendering quality?
- Which of the following images best matches the text description?
In particular, the first question focuses on the text rendering quality. Taking Figure 20 as an example<sup>7</sup>, we expect the model to render the word "EcoGrow" accurately (i.e., without any missing or additional characters). The second question, on the other hand, focuses on whether the overall image matches the given prompt. For example, in Figure 20 (G), although the generated text is correct, it fails to meet the requirement in the prompt that the letter looks like a plant. We instruct users to select the best option. In cases where multiple good options are difficult to distinguish, they could choose multiple options. If users are unsatisfied with the options, they could decide not to select any.
(E)
Figure 20: One case in the user study for the whole-image generation task.
(G)
[(8/15)] A logo for the company 'EcoGrow', where the letters look like plants.
User study on the part-image generation task. We aim to let users vote on the quality of text inpainting (from 4 to 1, the higher, the better). We also designed two questions:
- How is the text rendering quality?
- Does the drawn text harmonize with the unmasked region?
Specifically, the first question concentrates on the accuracy of the text. The second question focuses on whether the generated part is harmonious with the unmasked part (i.e., whether the background and texture are consistent).
Figure 21: One case in the user study for the part-image generation task.
$^{7}$  (A) TextDiffuser; (B) DALL-E; (C) SD; (D) IF; (E) SD-XL; (F) Midjourney; (G) ControlNet.
N Generating Images without Text
To show the generality of TextDiffuser, we experiment with generating images that do not contain texts and show results in Figure 22. Although TextDiffuser is fine-tuned with MARIO-10M, it still maintains a good generation ability for generating general images. Therefore, users have more options when using TextDiffuser, demonstrating its flexibility. We also provide quantitative evaluations to demonstrate TextDiffuser's generality in generating non-text general images. We compare TextDiffuser with our baseline Stable Diffusion 1.5 as they have the same backbone. For a quantitative evaluation, the FID scores of 5,000 images generated by prompts randomly sampled from MSCOCO are as in Table 8. The results indicate that TextDiffuser can maintain the ability to generate natural images even after fine-tuning the domain-specific dataset.
Table 8: FID scores on MSCOCO compared with Stable Diffusion.
Sampling Steps	Stable Diffusion	TextDiffuser
50	26.47	27.72
100	27.02	27.04
Figure 22: Visualizations of general images generated by Stable Diffusion 1.5 and TextDiffuser.
O Comparisons between TextDiffuser and a Text Editing Model
We visualize some results in Figure 23 compared with a text editing model SRNet [88]. Please note that the introduced text inpainting task differs from the text editing task in three aspects: (1) The text editing task usually relies on the synthesized text image dataset for training (synthesizing two images with the different text given a background image and font as pairs). In contrast, the text inpainting task follows the mask-and-recover training scheme and can be trained with any text images. (2) Text editing emphasizes the preservation of the original fonts, while text inpainting allows for greater freedom. For example, we conduct four samplings for each case, and the generated results exhibit diversity and present reasonable font styles. (3) Text editing tasks cannot add text, highlighting the significance of the introduced text inpainting task.
Figure 23: Comparison with text editing model. Four cases are obtained from the paper of SRNet.
P Experimental Results of Text Removal
We demonstrate some results of text removal in Figure 24, and the cases are obtained from the paper of EraseNet [44]. We can easily transform the text inpainting task into a text removal task by providing a mask and setting all regions to non-character in the segmentation mask. Experimental results demonstrate that our method can achieve results similar to the ground truth.
Inpainting Mask
Our Result
Figure 24: Experimental results on text removal. Four cases are obtained from the paper of EraseNet.
Q Limitations and Failure Cases
We observe failure cases when generating images with small characters and from long text.
Generating images with small characters. TextDiffuser uses the VAE networks to encode images into low-dimensional latent spaces for computational efficiency following latent diffusion models [67, 49, 2]. However, the compression process can result in losing details when generating images with small characters. As illustrated in Figure 25, we observe that the VAE fails to reconstruct small characters, where reconstructed strokes are unclear and reduce the legibility of the text. According to the generated images, the small characters appear to have vague or disjointed strokes (e.g., the character 'l' in 'World' and character 'r' in 'Morning'), which could impact the readability. As shown in Figure 26, we notice that using a more powerful backbone, such as Stable Diffusion 2.1, can mitigate this issue. When the image resolution is enhanced from  $512 \times 512$  to  $768 \times 768$  using Stable Diffusion 2.1 (instead of 1.5), the latent space resolution also increases from  $64 \times 64$  to  $96 \times 96$ , enhancing the character-level representation. As the cost, the inference latency rises from 8.5s to 12.0s with a batch size of 1. Therefore, how to render small characters while maintaining the same time cost is worth further study.
Generating images from long text. We observed failure cases when generating images from long text with many keywords, where the generated words in the layouts are disordered and overlapped,
Figure 25: The issue of generating images with small characters.
Figure 26: Pre-trained on high-resolution Stable Diffusion 2.1 enhances the legibility of small text.
as shown in Figure 27. One possible reason could be that training examples containing numerous keywords tend to have more noise (i.e., those images usually contain dense and small text), leading to a higher likelihood of detection and recognition errors. To address this, we could consider enhancing the capability of OCR tools in the future to mitigate the noise.
Figure 27: The issue of dealing with a large number of keywords.