SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The weight of character-aware loss. The experimental results are demonstrated in Table 2, where we conduct experiments with  $\lambda_{char}$  ranging from [0, 0.001, 0.01, 0.1, 1]. We utilize DrawBenchText [72] for evaluation and use Microsoft Read API to detect and recognize the texts in generated images. We use Accuracy (Acc) as the metric to justify whether the detected words exactly match the keywords. We observe that the optimal performance is achieved when  $\lambda_{char}$  is set to 0.01, where the score is increased by  $9.8\%$  compared with the baseline ( $\lambda_{char} = 0$ ).
The training ratio of whole/part-image generation branches. We explore the training ratio  $\sigma$  ranging from [0, 0.25, 0.5, 0.75, 1] and show results in Table 3. When  $\sigma$  is set to 1, it indicates that only the whole-image branch is trained and vice versa. We evaluate the model using DrawBenchText [72] for the whole-image generation branch. For the part-image generation branch, we randomly select 1,000 samples from the test set of MARIO-10M and randomly mask some of the detected text boxes. We utilize Microsoft Read API to detect and recognize the reconstructed text boxes in generated images while using the F-measure of text detection results and spotting results as metrics (denoted as Det-F and Spot-F, respectively). The results show that when the training ratio is set to  $50\%$ , the model performs better on average (0.716).
Quantitative Results. For the whole-image generation task, we compare our method with Stable Diffusion (SD) [67], ControlNet [102], and DeepFloyd [12] in quantitative experiments with the publicly released codes and models detailed in Appendix K. DeepFloyd [12] uses two super-resolution modules to generate higher resolution  $1024 \times 1024$  images compared with  $512 \times 512$  images generated by other methods. We use the Canny map of printed text images generated with our first stage model as conditions for ControlNet [102]. Please note that we are not able to compare withImagen [72], eDiff-i [2], and GlyphDraw [49] due to the lack of open-source code, checkpoints or APIs. According to Table 4, we demonstrate the quantitative results of the text-to-image task compared with existing methods. Our TextDiffuser obtains the best CLIPScore while achieving comparable performance in terms of FID. Besides, TextDiffuser achieves the best performance regarding four OCR-related metrics. TextDiffuser outperforms those methods without explicit text-related guidance by a large
Figure 4: Visualizations of whole-image generation compared with existing methods. The first three cases are generated from prompts and the last three cases are from given printed template images.
margin (e.g.,  $76.10\%$  and  $60.62\%$  better than Stable Diffusion and DeepFloyd regarding F-measure), highlighting the significance of explicit guidance. As for the part-image generation task, we cannot evaluate our method since no methods are specifically designed for this task to our knowledge.
Qualitative Results. For the whole-image generation task, we further compare with closed-source DALL-E [63], Stable Diffusion XL (SD-XL), and Midjourney by showing qualitative examples generated with their official API services detailed in Appendix K. Figure 4 shows some images generated from prompts or printed text images by different methods. Notably, our method generates more readable texts, which are also coherent with generated backgrounds. On the contrary, although the images generated by SD-XL and Midjourney are visually appealing, some generated text does not contain the desired text or contains illegible characters with incorrect strokes. The results also show that despite the strong supervision signals provided to ControlNet, it still struggles to generate images with accurate text consistent with the background. We also initiate a comparison with the Character-Aware Model [46] and the concurrent work GlyphDraw [49] using samples from their papers as their open-source code, checkpoints or APIs are not available. Figure 5 shows that TextDiffuser performs better than these methods. For instance, the Character-Aware Model suffers from misspelling issues (e.g., 'm' in 'Chimpanzees') due to its lack of explicit control, and GlyphDraw struggles with rendering images containing multiple text lines. For the part-image generation task, we visualize some results in Figure 6. In contrast to text editing tasks [88], we give the model sufficient flexibility to generate texts with reasonable styles. For instance, the image in the second row and first column contains the word "country" in green, while the model generates the word "country" in yellow. This is reasonable since it follows the style of the nearest word "range". Besides, our method can render realistic text coherent with the background, even in complex cases such as clothing. More qualitative results are shown in Appendix L.
A vintage postage stamp with the message: california all dreams welcome
A sign that says "Please refrain from arguing with the chimpanzees"
The cover for the album "Elusive Interludes" by the band The Melting Snowmen.
Photo of "Gas Station"
a 3d model of a 1980s-style computer with the text "my old habit" on the screen
A t-shirt with the message "There is no planet" written on it
Figure 5: Comparison with Character-Aware Model [46] and the concurrent GlyphDraw [49].
How to make a newspaper
FIMI X8 SE Range Test in Country
She is teaching Chinese lesson.
A man wears a cloth containing handsome boy
Figure 6: Visualizations of part-image generation (text inpainting) from given images.
Frequently Asked Questions (FAQ): The Muslim Edition image
User Studies. For the whole-image generation task, the designed questionnaire consists of 15 cases, each of which includes two multiple-choice questions: (1) Which of the following images has the best text rendering quality? (2) Which of the following images best matches the text description? For the part-image generation task, the questionnaire consists of 15 cases, each of which includes two rating questions: (1) How is the text rendering quality? (2) Does the drawn text harmonize with the unmasked region? The rating scores range from 1 to 4, and 4 indicates the best. Overall, we have collected 30 questionnaires, and the results are shown in Figure 8. We can draw two conclusions: (1) The generation performance of TextDiffuser is significantly better than existing methods. (2) Users are satisfied with the inpainting results in most cases. More details are shown in Appendix M.
Time and Parameter Efficiency For the time efficiency, the first stage of Layout Generation leverages an auto-regressive Transformer whose prediction time correlates with the number of keywords. Specifically, we conduct experiments to evaluate the time overhead for different numbers of keywords, including 1 (1.07±0.03s), 2 (1.12±0.09s), 4 (1.23±0.13s), 8 (1.57±0.12s), 16 (1.83±0.12s), and 32 (1.95±0.28s). Meanwhile, the second stage of image generation is independent of the number of queries (7.12±0.77s). For the parameter efficiency, TextDiffuser builds upon Stable Diffusion 1.5 (859M parameters), adding a Layout Transformer in the first stage (+25M parameters) and modifying the second stage (+0.75M parameters), augmenting it by only about 3% in terms of parameters.
Figure 7: Demonstration of using language descriptions to control text color.
(a) For whole-image generation, our method clearly outperforms others in both aspects of text rendering quality and image-text matching.
(b) For part-image generation, our method receives high scores from human evaluators in these two aspects.
Figure 8: User studies for whole-image generation and part-image generation tasks.
Text Color Controllability In Figure 7, we showcase TextDiffuser's capability in controlling the color of generated texts through language descriptions. The visualization results show that TextDiffuser can successfully control the color of rendered text, further enhancing its controllability.
Discussion. We show that TextDiffuser maintains the capability and generality to create general images without text rendering in Appendix N. Besides, we compare our method with a text editing model in Appendix O, showing that TextDiffuser generates images with better diversity. We also present the potential of TextDiffuser on the text removal task in Appendix P. As for the limitations and failure cases, TextDiffuser uses the VAE networks to encode images into low-dimensional latent spaces for computational efficiency following latent diffusion models [67, 49, 2], which has a limitation in reconstructing images with small characters as shown in Appendix Q. We also observed failure cases when generating images from long text and showed them in Appendix Q. As for the broader impact, TextDiffuser can be applied to many designing tasks, such as creating posters and book covers. Additionally, the text inpainting task can be used for secondary creation in many applications, such as Midjourney. However, there may be some ethical concerns, such as the misuse of text inpainting for forging documents. Therefore, techniques for detecting text-related tampering [86] need to be applied to enhance security. In conclusion, we propose a two-stage diffusion model called TextDiffuser to generate images with visual-pleasing texts coherent with backgrounds. Using segmentation masks as guidance, the proposed TextDiffuser shows high flexibility and controllability in the generation process. We propose MARIO-10M containing 10 million image-text pairs with OCR annotations. Extensive experiments and user studies validate that our method performs better than existing methods on the proposed benchmark MARIO-Eval. For future work, we aim to address the limitation of generating small characters by using OCR priors following OCR-VQGAN [66] and enhance TextDiffuser's capabilities to generate images with text in multiple languages. Disclaimer Please note that the model presented in this paper is intended for academic and research purposes ONLY. Any use of the model for generating inappropriate content is strictly prohibited and is not endorsed by this paper. The responsibility for any misuse or improper use of the model lies solely with the users who generated such content, and this paper shall not be held liable for any such use.
This research was supported by the Research Grant Council of the Hong Kong Special Administrative Region under grant number 16203122.
[94] Boxi Yu, Yong Xu, Yan Huang, Shuai Yang, and Jiaying Liu. Mask-guided gan for robust text editing in the scene. Neurocomputing, 2021.
[95] Deli Yu, Xuan Li, Chengquan Zhang, Tao Liu, Junyu Han, Jingtuo Liu, and Errui Ding. Towards accurate scene text recognition with semantic reasoning networks. In CVPR, 2020.
[96] Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao. Diverse image inpainting with bidirectional and autoregressive transformers. In ACM MM, 2022.
[97] Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
[98] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid-context encoder network for high-quality image inpainting. In CVPR, 2019.
[99] Guanhua Zhang, Jiabao Ji, Yang Zhang, Mo Yu, Tommi Jaakkola, and Shiyu Chang. Towards coherent image inpainting using denoising diffusion implicit models. arXiv preprint arXiv:2304.03322, 2023.
[100] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
[101] Linjiang Zhang, Peng Wang, Hui Li, Zhen Li, Chunhua Shen, and Yanning Zhang. A robust attentional framework for license plate recognition in the wild. IEEE Transactions on Intelligent Transportation Systems (TITS), 2020.
[102] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[103] Lei Zhao, Qihang Mo, Sihuan Lin, Zhizhong Wang, Zhiwen Zuo, Haibo Chen, Wei Xing, and Dongming Lu. Uctgan: Diverse image inpainting based on unsupervised cross-space translation. In CVPR, 2020.
[104] Minyi Zhao, Miao Wang, Fan Bai, Bingjia Li, Jie Wang, and Shuigeng Zhou. C3-stisr: Scene text image super-resolution with triple clues. In IJCAI, 2022.
[105] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In CVPR, 2019.
[106] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. East: an efficient and accurate scene text detector. In CVPR, 2017.
[107] Xinyan Zu, Haiyang Yu, Bin Li, and Xiangyang Xue. Weakly-supervised text instance segmentation. arXiv preprint arXiv:2303.10848, 2023.
A Architecture of U-Net and Design of Character-Aware Loss
As shown in Figure 9, the U-Net contains four downsampling operations and four upsampling operations. The input will be downsampled to a maximum of 1/16. To provide the character-aware loss, the input feature  $\mathbf{F}$  is 4-D with spatial size  $64\times 64$ , while the output is 96-D (the length of alphabet  $\mathcal{A}$  plus a null symbol indicating the non-character pixel) also with spatial size  $64\times 64$ . Subsequently, a cross-entropy loss is calculated between the output feature (need to convert the predicted noise into predicted features) and the resized  $64\times 64$  character-level segmentation mask  $\mathbf{C}'$ . The U-Net is pre-trained using the training set of MARIO-10M for one epoch. We utilize the Adadelta optimizer [97] and set the learning rate to 1. When training the diffusion model, the U-Net is frozen and only used to provide character-aware guidance.
Figure 9: The architecture of U-Net contains four downsampling and upsampling operations.
B Character-Level Segmentation Model
We train the character-level segmentation model based on U-Net, whose architecture is similar to the architecture shown in Figure 9. We set the input size to  $256 \times 256$ , ensuring that most characters are readable at this resolution. We train the segmentation model using synthesized scene text images [19], printed text images, and handwritten text images<sup>3</sup>, totaling about 4M samples. We employ data augmentation strategies (e.g., blurring, rotation, and color enhancement) to make the segmentation model more robust. The segmentation model is trained for ten epochs using the Adadelta optimizer [97] with a learning rate of 1. Figure 10 shows some samples in the training dataset.
Figure 10: Visualization of some training samples for the character-level segmentation model. The top, middle, and bottom roles are samples from printed, handwritten, and scene datasets.
<sup>3</sup>https://github.com/Belval/TextRecognitionDataGenerator
C More Details in MARIO-10M
Table 5: Number of texts per image in MARIO-10M.
#Words	1	2	3	4	5	6	7	8
#Images	592,153	1,148,481	1,508,185	1,610,056	1,549,852	1,430,750	1,229,714	930,809
#Ratio	5.9%	11.5%	15.1%	16.1%	15.5%	14.3%	12.3%	9.3%
More samples are shown in Figure 11. The number of texts per image in MARIO-10M is shown in Table 5. Also, the MARIO-10M dataset reveals that about  $90\%$  of the text regions maintain a horizontal orientation with rotation angles smaller than 5 degrees without perspective changes. Hence, our layout generation model is designed to predict horizontal bounding boxes by detecting the coordinates of their left-top and bottom-right points. Adapting our model to predict more realistic scene text is feasible by detecting enhanced coordinates, such as eight coordinates for four points.
D MARIO-10M Caption Templates
Since TMDB movie/TV posters and Open Library book covers have no off-the-shelf captions, we construct them based on their titles with the following templates. {XXX} is a placeholder for title.
For MARIO-TMDB:
- Logo {XXX}
- Text {XXX}
- Title {XXX}
- Title text {XXX}
- A poster with a title text of {XXX}
- A poster design with a title text of {XXX}
- A quality movie print with a title text of {XXX}
- A film poster of {XXX}
- A movie poster of {XXX}
- A movie poster titled {XXX}
- A movie poster named {XXX}
- A movie poster with text {XXX} on it
- A movie poster with logo {XXX} on it
- A movie poster with a title text of  $\{\mathrm{XXX}\}$
- An illustration of  $\{\mathrm{XXX}\}$  movie
- An photography of  $\{\mathrm{XXX}\}$  movie
- A TV show poster titled {XXX}
- A TV show poster of {XXX}
- A TV show poster with logo {XXX} on it
- A TV show poster with a title text of  $\{\mathrm{XXX}\}$
- A TV show poster with text {XXX}
- A TV show poster named {XXX}
For MARIO-OpenLibrary:
- A book with a title text of  $\{\mathrm{XXX}\}$
- A book design with a title text of {XXX}
- A book cover with a title text of {XXX}
- A book of {XXX}
- A cover named  $\{\mathrm{XXX}\}$
- A cover titled {XXX}
- A book with text {XXX} on it
- A book cover with logo {XXX} on it
(a) Samples in MAION-LAION
(b) Samples in MARIO-TMDB
(c) Samples in MARIO-OpenLibrary
Figure 11: More samples in MARIO-10M.
E MARIO-10M Filtering Rules
We clean data with five strict filtering rules to obtain high-quality data with text:
- Height and width are larger than 256. Low-resolution samples often contain illegible texts, negatively impacting the training process.
- Will not trigger NSFW. For the MARIO-LAION subset, we filter out those samples triggering the "not sure for work" flag to mitigate ethical concerns.
- The number of detected text boxes should be within [1,8]. We detect texts with DB [42]. Samples with too many texts typically have small areas for each text, which makes them difficult to recognize. Therefore, we remove these samples from the dataset.
- Text areas are more than  $10\%$  of the whole image. According to Appendix B, we train a UNet [69] using SynthText [19] to obtain character-level segmentation masks of each sample. This criterion ensures that the text regions will not be too small.
- At least one detected text appears in the caption. Noticing that the original dataset contains many noisy samples, we add this constraint to increase the relevance between images and captions. We utilize PARSeq [4] for text recognition.
F Analysis of OCR Performance on MARIO-10M
As we rely on OCR tools to annotate MARIO-10M, it is necessary to evaluate the performance of these tools. Specifically, we manually annotate 100 samples for text recognition, detection, and character-level segmentation masks, then compare them with the annotations given by OCR tools. The results are shown in Table 6. We notice that the performance of existing methods is lower than their results on text detection and spotting benchmarks. Taking DB [42] as an example, it can achieve text detection  $91.8\%$  precision on ICDAR 2015 dataset [34] while only achieving  $76\%$  on MARIO-10M. This is because there are many challenging cases in MARIO-10M, such as blurry and small text. Besides, a domain gap may exist since DB is trained on scene text detection datasets, while MARIO-10M comprises text images in various scenarios. Future work may explore more advanced recognition, detection, and segmentation models to mitigate the noise in OCR annotations. We demonstrate some OCR results in Figure 12.
Table 6: OCR performance on MARIO-10M. IOU (binary) means we treat each pixel as two classes: characters and non-character. The evaluation of recognition is included in the spotting task.
Detection	Spotting	Segmentation
Precision	Recall	F-measure	Precision	Recall	F-measure	IOU (binary)	IOU
0.76	0.79	0.78	0.73	0.75	0.74	0.70	0.59
Chitty
hocol
Peppalig
GREENE
MonkeY
epop
LONORES
GRAVE
JEANIENE
DIAI
FREE!
Figure 12: Visualization of some OCR annotations in MARIO-10M.
G Samples in MARIO-Eval
Table 7: Details of each subset in MARIO-Eval.
Subset	Size	Off-the-shelf Captions	GT Images
LAIONEval4000	4,000	✓	✓
TMDBEval500	500	✗	✓
OpenLibraryEval500	500	✗	✓
DrawBenchText [72]	21	✓	✗
DrawTextCreative [46]	175	✓	✗
ChineseDrawText [49]	218	✓	✗
As illustrated in Table 7, MARIO-Eval contains 5,414 prompts with six subsets. The ground truth images of some samples are shown in Figure 13, and captions for each category are shown below:
LAIONeval4000:
'Royal Green' Wristband Set
- Are 'Digital Nomads' in Trouble Well Not Exactly
- 'Sniper Elite' One Way Trip A Novel Audiobook by 'Scott' McEwen Thomas Koloniar Narrated by Brian Hutchison
Travel Artin'Logo
- Falls the 'Shadow' Welsh Princes 2
TMDBEval500:
- A movie poster with text 'Heat' on it
- A poster design with a title text of 'Deadpool 2'
- A TV show poster named 'Ira Finkelstein's Christmas'
- A movie poster with logo ‘Playing for Change Songs Around The World Part 2 ’ on it
- A movie poster titled 'Dreams of a Land'
OpenLibraryEval500:
- A book cover with a title text of 'On The Apparel Of Women'
- A book with text 'Precalculus' on it
- A book design with a title text of 'Dream master nightmare'
- A book cover with logo 'Thre Poetical Works of Constance Naden' on it
- A book cover with a title text of 'Discovery'
DrawBenchText:
- A storefront with 'Hello World' written on it.
- A storefront with 'Text to Image' written on it.
- A sign that says 'Diffusion'.
- A sign that says 'NeurIPS'.
- New York Skyline with 'Google Research Pizza Cafe' written with fireworks on the sky.
DrawTextCreative:
- a grumpy sunflower with a 'no solar panels' sign
- A photo of a rabbit sipping coffee and reading a book. The book title 'The Adventures of Peter Rabbit' is visible.
- a graffiti art of the text 'free the pink' on a wall
- A professionally designed logo for a bakery called 'Just What I Kneaded'.
- scholarly elephant reading a newspaper with the headline 'elephants take over the world'
ChineseDrawText:
- A street sign on the street reads 'Heaven rewards those who work hard'
- There is a book on the table with the title 'Girl in the Garden'
- Kitten holding a sign that reads 'I want fish'
- A robot writes 'Machine Learning' on a podium
- In a hospital, a sign that says 'Do Not Disturb'
'Adventure Awaits' Home Blanket by Katie Rose s Artist Shop
Win 2 Tickets To An Advanced Screening Of Straight 'Outta Compton' In Atlanta Courtesy Of HHS1987 Aug 11th
'Risk Everything'
'50 positive parenting quotes gentle parenting peaceful parenting gentle discipline
V A TONY THE TYGVOLUME TWO THE DAY BREAKS AT DAWN LP
A poster with a title text of 'Frank'
A TV show poster of 'Burden of Dreams'
A poster with a title text of 'Life to Afterlife Death and Back'
A movie poster with a title text of 'Star Wars'
A movie poster with logo 'Adams Crossing' on it
A cover titled 'Sleeping Beauty'
A cover titled 'Boulez on Conducting'
A book design with a title text of 'Butterflies Surprise'
A book with a title text of 'Expressionismo'
Figure 13: We demonstrate five samples for LAIONEval4000 (top), TMDBEval500 (middle), and OpenLibrary500 (bottom).
A book with text 'The master mummer' on it
H Implementation Details of Evaluation Criteria
To evaluate the performance of TextDiffuser quantitatively, we utilize three criteria, including FID, CLIPScore, and OCR Evaluation. We detail the calculation of each criterion below.
FID. We calculate the FID score using the pytorch-fid repository. Please note that the proposed MARIO-Eval benchmark's three subsets (DrawTextCreative, DrawBenchText, and ChineseDrawText) do not contain ground truth images. Therefore, we utilize 5,000 images in the other three subsets (LAIONEval4000, TMDBEval500, OpenLibraryEval500) as the ground truth images. We calculate the FID score using the 5,414 generated images and the 5,000 ground truth images.
CLIP Score. We calculate the CLIP score using the clipscore repository. However, as with the FID score, we cannot calculate the CLIP score for the DrawTextCreative, DrawBenchText, and ChineseDrawText subsets due to the lack of ground truth images. Therefore, we only calculate the score on LAIONEval4000, TMDBEval500, and OpenLibraryEval500 subsets and report the average CLIP score.