Title: TextDiffuser: Diffusion Models as Text Painters
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion Model; Text Rendering
Abstract: Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue, we introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. TextDiffuser consists of two stages: first, a Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout. Additionally, we contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs with text recognition, detection, and character-level segmentation annotations. We further collect the MARIO-Eval benchmark to serve as a comprehensive tool for evaluating text rendering quality. Through experiments and user studies, we demonstrate that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text. We will make the code, model and dataset publicly available.

TextDiffuser: Diffusion Models as Text Painters

Abstract

Diffusion models have gained increasing attention for their impressive generation abilities but currently struggle with rendering accurate and coherent text. To address this issue, we introduce TextDiffuser , focusing on generating images with visually appealing text that is coherent with backgrounds. TextDiffuser consists of two stages: first, a Transformer model generates the layout of keywords extracted from text prompts, and then diffusion models generate images conditioned on the text prompt and the generated layout. Additionally, we contribute the first large-scale text images dataset with OCR annotations, MARIO-10M , containing 10 million image-text pairs with text recognition, detection, and character-level segmentation annotations. We further collect the MARIO-Eval benchmark to serve as a comprehensive tool for evaluating text rendering quality. Through experiments and user studies, we show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text. The code, model, and dataset will be available at https://aka.ms/textdiffuser .

(a) Text-to-Image

(b) Text-to-Image with Template

(c) Text Inpainting

A dog ...

An astronaut ...

A cat holds a

paper saying

"Hello World"

A dinosaur ...

layout

generation

image

generation

company

bookmark

cake

pizza

dress

newspaper

mask

template

a boy draws good morning on a board

a supermarket called diffusion

W

OR

WORK

W

OR

Hello

Hello

K

K

World

311'e

HappyBitadaey

Happy I Birthday

Happy

Birthday

ToYou

ToYoul

lou

Hello

Hello

Hello

PUOM

World

World

DONOT

READ

THENEXT

SENTENCE.

DONOT

READ

E

THENEXT

READ

SENTENCE.

DONOT

THENEXT

Hello

poM

Good

Good

fusion

Diffusion

Diffusion

Morning

Moming

1 Introduction

The field of image generation has seen tremendous progress with the advent of diffusion models [2, 15, 16, 18, 25, 67, 70, 72, 79, 93] and the availability of large-scale image-text paired datasets [17, 74, 75]. However, existing diffusion models still face challenges in generating visually pleasing text on images, and there is currently no specialized large-scale dataset for this purpose. The ability of AI models to generate accurate and coherent text on images is crucial, given the widespread use of text images in various forms ( e.g. , posters, book covers, memes, etc.) and the difficulty in creating high-quality text images, which typically require professional skills and numerous times of designers.

Traditional solutions to creating text images involve using image processing tools like Photoshop to add text onto images directly. However, these often result in unnatural artifacts due to the background's complex texture or lighting variations. Recent efforts have used diffusion models to overcome the limitations of traditional methods and enhance text rendering quality. For instance, Imagen [72], eDiff-I [2], and DeepFloyd [12] observe diffusion models generate text better with T5 series text encoders [61] than the CLIP text encoder [60]. Liu et al. employ character-aware text encoders to improve text rendering [46]. Despite some success, these models only focus on text encoders, lacking control over the generation process. A concurrent work, GlyphDraw [49], improves the controllability of models by conditioning on the location and structures of Chinese characters. However, GlyphDraw does not support multiple text bounding-box generation, which is not applicable to many text images such as posters and book covers.

In this paper, we propose TextDiffuser , a flexible and controllable framework based on diffusion models. The framework consists of two stages. In the first stage, we use a Layout Transformer to locate the coordinates of each keyword in text prompts and obtain character-level segmentation masks. In the second stage, we fine-tune the latent diffusion model by leveraging the generated segmentation masks as conditions for the diffusion process and text prompts. We introduce a character-aware loss in the latent space to further improve the quality of generated text regions. Figure 1 illustrates the application of TextDiffuser in generating accurate and coherent text images using text prompts alone or text template images. Additionally, TextDiffuser is capable of performing text inpainting 2 to reconstruct incomplete images with text. To train our model, we use OCR tools and design filtering strategies to obtain 10 million high-quality i ma ge-text pai r s w i th O CR annotations (dubbed as MARIO-10M ), each with recognition, detection, and character-level segmentation annotations. Extensive experiments and user studies demonstrate the superiority of the proposed TextDiffuser over existing methods on the constructed benchmark MARIO-Eval . The code, model and dataset will be publicly available to promote future research.

2 Related Work

Text Rendering. Image generation has made significant progress with the advent of diffusion models [18, 25, 67, 72, 79, 63, 70, 2, 8, 13, 52, 48, 26, 80], achieving state-of-the-art results compared with previous GAN-based approaches [64, 100, 43, 58]. Despite rapid development, current methods still struggle with rendering accurate and coherent text. To mitigate this, Imagen [72], eDiff-I [2], and DeepFolyd [12] utilize a large-scale language model (large T5 [61]) to enhance the text-spelling knowledge. In [46], the authors noticed that existing text encoders are blind to token length and trained a character-aware variant to alleviate this problem. A concurrent work, GlyphDraw [49], focuses on generating high-quality images with Chinese texts with the guidance of text location and glyph images. Unlike this work, we utilize Transformer [81] to obtain the layouts of keywords, enabling the generation of texts in multiple lines. Besides, we use character-level segmentation masks as prior, which can be easily controlled ( e.g. , by providing a template image) to meet user needs.

Several papers have put forward benchmarks containing a few cases regarding text rendering for evaluation. For example, Imagen [72] introduces DrawBench containing 200 prompts, in which 21 prompts are related to visual text rendering ( e.g. , A storefront with 'Hello World' written on it). According to [46], the authors proposed DrawText comprising creative 175 prompts ( e.g. , letter 'c' made from cactus, high-quality photo). GlyphDraw [49] designs 218 prompts in Chinese and

English ( e.g. , Logo for a chain of grocery stores with the name 'Grocery'). Considering that existing benchmarks only contain a limited number of cases, we attempt to collect more prompts and combine them with existing prompts to establish a larger benchmark MARIO-Eval to facilitate comprehensive comparisons for future work.

Image Inpainting. Image inpainting is the task of reconstructing missing areas in images naturally and coherently. Early research focused on leveraging low-level image structure and texture to address this task [3, 6, 5]. Later, deep learning architectures such as auto-encoder [55, 45], GAN [65, 98], VAE [103, 105], and auto-regressive Transformers [57, 83, 96] were applied to tackle this problem. Recently, diffusion models have been used to generate high-quality and diverse results for unconditional image inpainting [71, 48, 67, 11, 99], text-conditional image inpainting [52, 1] and image-conditional image inpainting [92]. Our work falls under the category of text-conditional image inpainting using diffusion models. In contrast to prior works that focused on completing images with natural backgrounds or objects, our method focuses on completing images with text-related rendering, also named text inpainting, by additional conditioning on a character-level segmentation mask.

Optical Character Recognition. Optical Character Recognition (OCR) is an important task that has been studied in academia for a long period [87, 7]. It has undergone a remarkable development in the last decade, contributing to many applications like autonomous driving [89, 73], car license plate recognition [101, 53], GPT models [76, 28], etc. Various datasets [31, 19, 90, 91] and downstream tasks are included within this field, such as text image recognition [77, 41, 95, 14], detection [106, 51, 42, 84], segmentation [90, 91, 107, 68], super-resolution [9, 85, 50, 104], as well as some generation tasks, including text image editing [88, 78, 94, 59, 38], document layout generation [56, 22, 20, 40, 33], font generation [30, 21, 36, 54], etc. Among them, the font generation task is most relevant to our task. Font generation aims to create high-quality, aesthetically pleasing fonts based on given character images. In contrast, our task is more challenging as it requires the generated text to be legible, visually appealing, and coherent with the background in various scenarios.

3 Methodology

As illustrated in Figure 2, TextDiffuser consists of two stages: Layout Generation and Image Generation. We will detail the two stages and introduce the inference process next.

3.1 Stage1: Layout Generation

In this stage, the objective is to utilize bounding boxes to determine the layout of keywords (enclosed with quotes specified by user prompts). Inspired by Layout Transformer [20], we utilize the Transformer architecture to obtain the layout of keywords. Formally, we denote the tokenized prompt as P = ( p 0 , p 1 , ..., p L -1 ) , where L means the maximum length of tokens. Following LDM [67], we use CLIP [60] and two linear layers to encode the sequence as CLIP ( P ) ∈ R L × d , where d is the dimension of latent space. To distinguish the keywords against others, we design a keyword embedding Key ( P ) ∈ R L × d with two entries ( i.e. , keywords and non-keywords). Furthermore, we encode the width of keywords with an embedding layer Width ( P ) ∈ R L × d . Together with the learnable positional embedding Pos ( P ) ∈ R L × d introduced in [81], we construct the whole embedding as follows:

The embedding is further processed with Transformer-based l -layer encoder Φ E and decoder Φ D to get the bounding boxes B ∈ R K × 4 of K key words autoregressively:

Specifically, we use positional embedding as the query for the Transformer decoder Φ D , ensuring that the n -th query corresponds to the n -th keyword in the prompt. The model is optimized with l 1 loss, also denoted as | B GT -B | where B GT is the ground truth. Further, we can utilize some Python packages like Pillow to render the texts and meanwhile obtain the character-level segmentation mask C with |A| channels, where |A| denote the size of alphabet A . To this end, we obtain the layouts of keywords and the image generation process is introduced next.

Transformer

Encoder

a

cat

holds

hello

world

Embedding

Layer

Tokenize

Transformer

Decoder

query0

query1

box0

box1

Hello

World

Render

Get Mask

Stage1 Layout Generation

Stage2 Image Generation

Whole-image Generation

Part-image Generation

Diffusion

Model

Text Prompt

U-Net

Denoising

Loss

Character

aware

Loss

Embedding

Layer

a cat holds

"Hello World"

Hello

World

Hello

Hello

Hello

World

a catholds

"Hello World"

Hello

World

3.2 Stage2: Image Generation

In this stage, we aim to generate the image guided by the segmentation masks C produced in the first stage. We use V AE [35] to encode the original image with shape H × W into 4-D latent space features F ∈ R 4 × H ′ × W ′ . Then we sample a time step T ∼ Uniform (0 , T max ) and sample a Gaussian noise ϵ ∈ R 4 × H ′ × W ′ to corrupt the original feature, yielding ˆ F = √ ¯ α T F + √ 1 -¯ α T ϵ where ¯ α T is the coefficient of the diffusion process introduced in [25]. Also, we downsample the characterlevel segmentation mask C with three convolution layers, yielding 8-D ˆ C ∈ R 8 × H ′ × W ′ . We also introduce two additional features, called 1-D feature mask ˆ M ∈ R 1 × H ′ × W ′ and 4-D masked feature ˆ F M ∈ R 4 × H ′ × W ′ . In the process of whole-image generation , ˆ M is set to cover all regions of the feature and ˆ F M is the feature of a fully masked image. In the process of part-image generation (also called text inpainting), the feature mask ˆ M represents the region where the user wants to generate, while the masked feature ˆ F M indicates the region that the user wants to preserve. To simultaneously train two branches, we use a masking strategy where a sample is fully masked with a probability of σ and partially masked with a probability of 1 -σ . We concatenate ˆ F , ˆ C , ˆ M , ˆ F M in the feature channel as a 17-D input and use denoising loss between the sampled noise ϵ and the predicted noise ϵ θ :

Furthermore, we propose a character-aware loss to help the model focus more on text regions. In detail, we pre-train a U-Net [69] that can map latent features to character-level segmentation masks. During training, we fix its parameters and only use it to provide guidance by using a cross-entropy loss l char with weight λ char (See more details in Appendix A). Overall, the model is optimized with

Finally, the output features are fed into the V AE decoder to obtain the images.

(a) MARIO-LAION

(b) MARIO-TMDB

(c) MARIO-OpenLibrary

LETMEGUESS

Homeschool

MILLION DOLLAR

TIEE

DANGUYMAI

KEEP

Stress-free

he

NAPTAIN

SHOT

CALM

Plan

ACK

HOMEFACTS

THEBALLISSTILLINYOUR

HAND

CLEAR

JUMP

AND

VAISE

ACABARI

HOURS

ULTIMATE HOME

RECORDKEEPER

THE

KSHMR&DALLASK

Arabic

Fun with

STRFKR

Andree

Putman

Alphabet

The

RULESOF

MANAGEMENT

BREEROM

revealed

OR:29

MAGICTRAILS

PONYRUN

MR.ROBOTDaVid

ss0n_4.0

BROOR

STEEL

4

inthoRYE

PARKER

BARBARA

SOUTHoSANTA FE

GODS

MEANT

AFFAIRE

UNESALE

JOANOFARC

RUMSTICKROAD

Lender

xpus

Horizons

Rere/bYJ.DEALINGER

3.3 Inference Stage

TextDiffuser provides a high degree of controllability and flexibility during inference in the following ways: (1) Generate images from user prompts. Notably, the user can modify the generated layout or edit the text to meet their personalized requirements; (2) The user can directly start from the second stage by providing a template image ( e.g. , a scene image, handwritten image, or printed image), and a segmentation model is pre-trained to obtain the character-level segmentation masks (Appendix B); (3) Users can modify the text regions of a given image using text inpainting. Moreover, this operation can be performed multiple times. These experimental results will be presented in the next section.

4 MARIO Dataset and Benchmark

As there is no large-scale dataset designed explicitly for text rendering, to mitigate this issue, we collect 10 million i ma ge-text pai r s w i th O CR annotations to construct the MARIO-10M Dataset . We further collect the MARIO-Eval Benchmark from the subset of the MARIO-10M test set and other existing sources to serve as a comprehensive tool for evaluating text rendering quality.

4.1 MARIO-10M Dataset

The MARIO-10M is a collection of about 10 million high-quality and diverse image-text pairs from various data sources such as natural images, posters, and book covers. Figure 3 illustrates some examples from the dataset. We design automatic schemes and strict filtering rules to construct annotations and clean noisy data (more details in Appendix D and Appendix E). The dataset contains comprehensive OCR annotations for each image, including text detection, recognition, and character-level segmentation annotations. Specifically, we use DB [42] for detection, PARSeq [4] for recognition, and manually train a U-Net [69] for segmentation. We analyze the performance of OCR tools in Appendix F. The total size of MARIO-10M is 10,061,720, from which we randomly chose 10,000,000 samples as the training set and 61,720 as the testing set. MARIO-10M is collected from three data sources:

MARIO-LAION derives from the large-scale datasets LAION-400M [75]. After filtering, we obtained 9,194,613 high-quality text images with corresponding captions. This dataset comprises a broad range of text images, including advertisements, notes, posters, covers, memes, logos, etc.

MARIO-TMDB derives from The Movie Database (TMDB), which is a community-built database for movies and TV shows with high-quality posters. We filter 343,423 English posters using the TMDB API out of 759,859 collected samples. Since each image has no off-the-shelf captions, we use prompt templates to construct the captions according to movie titles.

MARIO-OpenLibrary derives from Open Library, which is an open, editable library catalog that creates a web page for each published book. We first collect 6,352,989 original-size Open Library covers in bulk. Then, we obtained 523,684 higher-quality images after filtering. Like MARIO-TMDB, we manually construct captions using titles due to the lack of off-the-shelf captions.

4.2 MARIO-Eval Benchmark

The MARIO-Eval benchmark serves as a comprehensive tool for evaluating text rendering quality collected from the subset of the MARIO-10M test set and other sources. It comprises 5,414 prompts in total, including 21 prompts from DrawBenchText [72], 175 prompts from DrawTextCreative [46], 218 prompts from ChineseDrawText [49] and 5,000 image-text pairs from a subset of the MARIO-10M test set. The 5,000 image-text pairs are divided into three sets of 4,000, 500, and 500 pairs, and are named LAIONEval4000, TMDBEval500, and OpenLibraryEval500 based on their respective data sources. We offer examples in Appendix G to provide a clearer understanding of MARIO-Eval.

Evaluation Criteria: We evaluate text rendering quality with MARIO-Eval from four aspects: (1) Fréchet Inception Distance (FID) [24] compares the distribution of generated images with the distribution of real images. (2) CLIPScore calculates the cosine similarity between the image and text representations from CLIP [29, 60, 23]. (3) OCR Evaluation utilizes existing OCR tools to detect and recognize text regions in the generated images. Accuracy, Precision, Recall, and F-measure are metrics to evaluate whether keywords appear in the generated images. (4) Human Evaluation is conducted by inviting human evaluators to rate the text rendering quality of generated images using questionnaires. More explanations are shown in Appendix H.

5 Experiments

5.1 Implementation Details

For the first stage, we utilize the pre-trained CLIP [60] to obtain the embedding of given prompts. The number of Transformer layers l is set to 2, and the dimension of latent space d is set to 512. The maximum length of tokens L is set to 77 following CLIP [60]. We leverage a commonly used font 'Arial.ttf' and set the font size to 24 to obtain the width embedding and also use this font for rendering. The alphabet A comprises 95 characters, including 26 uppercase letters, 26 lowercase letters, 10 digits, 32 punctuation marks, and a space character. After tokenization, only the first subtoken is marked as the keyword when several subtokens exist for a word.

For the second stage, we implement the diffusion process using Hugging Face Diffusers [82] and load the checkpoint 'runwayml/stable-diffusion-v1-5' . Notably, we only need to modify the input dimension of the input convolution layer (from 4 to 17), allowing our model to have a similar scale of parameters and computational time as the original model. In detail, the height H and W of input and output images are 512. For the diffusion process, the input is with spatial dimension H ′ = 64 and W ′ = 64 . We set the batch size to 768 and trained the model for two epochs, taking four days using 8 Tesla V100 GPUs with 32GB memory. We use the AdamW optimizer [47] and set the learning rate to 1e-5. Additionally, we utilize gradient checkpoint [10] and xformers [39] for computational efficiency. During training, we follow [25] to set the maximum time step T max to 1,000, and the caption is dropped with a probability of 10% for classifier-free guidance [27]. When training the part-image generation branch, the detected text box is masked with a likelihood of 50%. We use 50 sampling steps during inference and classifier-free guidance with a scale of 7.5 following [67].

5.2 Ablation Studies

Number of Transformer layers and the effectiveness of width embedding. We conduct ablation studies on the number of Transformer layers and whether to use width embedding in the Layout Transformer. The results are shown in Table 1. All ablated models are trained on the training set of MARIO-10M and evaluated on its test set. Results show that adding width embedding improves performance, boosting IoU by 2.1%, 2.9%, and 0.3% when the number of Transformer layers l is set to 1, 2, and 4, respectively. The optimal IoU is achieved using two Transformer layers and the width embedding is included. See more visualization results in Appendix I.

Character-level segmentation masks provide explicit guidance for generating characters. The character-level segmentation masks provide explicit guidance on the position and content of characters during the generation process of TextDiffuser. To validate the effectiveness of using character-level segmentation masks, we train ablated models without using the masks and show results in Appendix

J. The generated texts are inaccurate and not coherent with the background compared with texts generated with TextDiffuser, highlighting the importance of explicit guidance.

The weight of character-aware loss. The experimental results are demonstrated in Table 2, where we conduct experiments with λ char ranging from [0, 0.001, 0.01, 0.1, 1]. We utilize DrawBenchText [72] for evaluation and use Microsoft Read API to detect and recognize the texts in generated images. Weuse Accuracy (Acc) as the metric to justify whether the detected words exactly match the keywords. We observe that the optimal performance is achieved when λ char is set to 0.01, where the score is increased by 9.8% compared with the baseline ( λ char = 0 ).

The training ratio of whole/part-image generation branches. We explore the training ratio σ ranging from [0, 0.25, 0.5, 0.75, 1] and show results in Table 3. When σ is set to 1, it indicates that only the whole-image branch is trained and vice versa. We evaluate the model using DrawBenchText [72] for the whole-image generation branch. For the part-image generation branch, we randomly select 1,000 samples from the test set of MARIO-10M and randomly mask some of the detected text boxes. We utilize Microsoft Read API to detect and recognize the reconstructed text boxes in generated images while using the F-measure of text detection results and spotting results as metrics (denoted as Det-F and Spot-F, respectively). The results show that when the training ratio is set to 50%, the model performs better on average (0.716).

5.3 Experimental Results

Quantitative Results. For the whole-image generation task , we compare our method with Stable Diffusion (SD) [67], ControlNet [102], and DeepFloyd [12] in quantitative experiments with the publicly released codes and models detailed in Appendix K. DeepFloyd [12] uses two super-resolution modules to generate higher resolution 1024 × 1024 images compared with 512 × 512 images generated by other methods. We use the Canny map of printed text images generated with our first stage model as conditions for ControlNet [102]. Please note that we are not able to compare with Imagen [72], eDiff-i [2], and GlyphDraw [49] due to the lack of open-source code, checkpoints or APIs. According to Table 4, we demonstrate the quantitative results of the text-to-image task compared with existing methods. Our TextDiffuser obtains the best CLIPScore while achieving comparable performance in terms of FID. Besides, TextDiffuser achieves the best performance regarding four OCR-related metrics. TextDiffuser outperforms those methods without explicit text-related guidance by a large

Prompt

SD

SD-XL

Midjourney

TextDiffuser

a bear holds

a board saying

'hello world'

a meme of

'Are you kidding'

a cake of

'Happy Birthday

to XYZ'

ControlNet

a poster of

'Monkey Music

Festival'

DALL  E

DeepFloyd

a book of

'AI in Next Century'

written by

'AI Robot'

a boy holds 'P'

and

a girl holds 'Q'

Hello

HEELOO

NELLNO

WEO

MELUWORD

World

HelloWorld

hello

world!

HelloWortd

HLIL OD

WenLDC

ZKAY

Happy

A

*Happy*

HApy

Birthday

toXYz

ToZ

Birthday

toxY2

M..MNCSNC

MONKIC

MONKEY

Monkey

Monkey

MONGSIY

Music

Festivale

MUSI

Music

Festiva

FESTIVAL

AYUYUTANE'AIET

AREMEME

REYEN?HU!

Are you

Areyou

Kidding?

AOTLL

YOUTFOUYEIEIS

AREYOUKIKNG?

INTA BEBAAIN ROBTETY

itblliAI

Ai

RICTTAT

In Next Century

ALIICOINTY

AI RobotA

Ai Robot

PPQ

P

Q

IPID

margin ( e.g. , 76.10% and 60.62% better than Stable Diffusion and DeepFloyd regarding F-measure), highlighting the significance of explicit guidance. As for the part-image generation task , we cannot evaluate our method since no methods are specifically designed for this task to our knowledge.

Qualitative Results. For the whole-image generation task , we further compare with closed-source DALL · E [63], Stable Diffusion XL (SD-XL), and Midjourney by showing qualitative examples generated with their official API services detailed in Appendix K. Figure 4 shows some images generated from prompts or printed text images by different methods. Notably, our method generates more readable texts, which are also coherent with generated backgrounds. On the contrary, although the images generated by SD-XL and Midjourney are visually appealing, some generated text does not contain the desired text or contains illegible characters with incorrect strokes. The results also show that despite the strong supervision signals provided to ControlNet, it still struggles to generate images with accurate text consistent with the background. We also initiate a comparison with the CharacterAware Model [46] and the concurrent work GlyphDraw [49] using samples from their papers as their open-source code, checkpoints or APIs are not available. Figure 5 shows that TextDiffuser performs better than these methods. For instance, the Character-Aware Model suffers from misspelling issues ( e.g. , 'm' in 'Chimpanzees') due to its lack of explicit control, and GlyphDraw struggles with rendering images containing multiple text lines. For the part-image generation task , we visualize some results in Figure 6. In contrast to text editing tasks [88], we give the model sufficient flexibility to generate texts with reasonable styles. For instance, the image in the second row and first column contains the word 'country' in green, while the model generates the word 'country' in yellow. This is reasonable since it follows the style of the nearest word 'range'. Besides, our method can render realistic text coherent with the background, even in complex cases such as clothing. More qualitative results are shown in Appendix L.

Character-Aware Model[42]

TextDiffuser (Ours)

Character-Aware Model[42]

TextDiffuser (Ours)

Character-Aware Model[42]

TextDiffuser (Ours)

CALIFORNIA

CALIFORNIA

The Merting Snowmena

TheMeltingSnowmen

Please

Please

Refrainfrom

Refrain fom

Arguing with

Arguing with

Chimpanzees

Chimpanzees

ALLDREAMSWELCOME

ALLDREAMS WELCOME

EksiveInterludes

Elusive Interludes

A vintage postage stampwith themessage:

A sign that says"Pleaserefrainfrom

The cover for the album"Elusive Interludes"

california alldreamswelcome

arguing with the chimpanzees"

by theband TheMeltingSnowmen.

GlyphDraw[45]

TextDiffuser (Ours)

GlyphDraw [45]

TextDiffuser (Ours)

GlyphDraw[45]

TextDiffuser (Ours)

GasStation

Gas Station

myold habit

There isno

myold habitl

planet

There is ro phanet

Photo of"Gas Station"

a 3d model of a 1980s-style computer with

A t-shirt with the message

the text"my old habit" on the screen

"There is no planet" written on it

COUNTRY

Frequently

FIMI X8 SE Range Test in Country

Frequently Asked Questions (FAQ): The Muslim Edition  image

Handsome

Boy

A man wears a cloth containing handsome boy

NEWSPAPER

How to make a newspaper

Chinese

She is teaching Chinese lesson.

Interesting terrace party

TECH

TECH

TECH

HOUSE

HOUSE

HOUSE

POSTER?

HOWTOMAKEA

VAXVAOLMOH

HOWTOMAKEA

NEWSPAPER

INTERESTING

INTERESTING

TERRACE

TERRACE

TERRACE

PARTY

PARTY

PARTY

Chinese

I1X8SE

FIMIX8SE

→FIMIX8SE

sson

COUNTRY

COUNTRY

RANGE TEST

RANGETEST

RANGETEST

FrequentlyAsked

sked

Frequently Asked

FASHON

Questions

Question

Questions

Handsome

(FAQ):

(FAQ):

(FAQ):

Boy

TheMuslimEdition

TheMuslim Edition

TheMuslimEdition

User Studies. For the whole-image generation task , the designed questionnaire consists of 15 cases, each of which includes two multiple-choice questions: (1) Which of the following images has the best text rendering quality? (2) Which of the following images best matches the text description? For the part-image generation task , the questionnaire consists of 15 cases, each of which includes two rating questions: (1) How is the text rendering quality? (2) Does the drawn text harmonize with the unmasked region? The rating scores range from 1 to 4, and 4 indicates the best. Overall, we have collected 30 questionnaires, and the results are shown in Figure 8. We can draw two conclusions: (1) The generation performance of TextDiffuser is significantly better than existing methods. (2) Users are satisfied with the inpainting results in most cases. More details are shown in Appendix M.

Time and Parameter Efficiency For the time efficiency, the first stage of Layout Generation leverages an auto-regressive Transformer whose prediction time correlates with the number of keywords. Specifically, we conduct experiments to evaluate the time overhead for different numbers of keywords, including 1 (1.07 ± 0.03s), 2 (1.12 ± 0.09s), 4 (1.23 ± 0.13s), 8 (1.57 ± 0.12s), 16 (1.83 ± 0.12s), and 32 (1.95 ± 0.28s). Meanwhile, the second stage of image generation is independent of the number of queries (7.12 ± 0.77s). For the parameter efficiency, TextDiffuser builds upon Stable Diffusion 1.5 (859M parameters), adding a Layout Transformer in the first stage (+25M parameters) and modifying the second stage (+0.75M parameters), augmenting it by only about 3% in terms of parameters.

Hello

Hello

Hello

Hello

Hello

Hello

PJOM

World

World

World

World

a bear holding aboard with a red

1'helloworld'

a bear holding a board with ayellow

'helloworld'

a bear holding a board with apurple

'helloworld'

Q1

9

13

6

9

5

59

10

0

100

200

300

DALL·E

SD

SD-XL

Midjourney

ControlNet

DeepFloyd

TextDiffuser

(a) For whole-image generation, our method clearly outperforms

others in both aspects of text rendering quality and image-text matching.

100

200

300

0

1

2

3

4

(b) For part-image generation, our method receives

high scores from human evaluators in these two aspects.

294 277

274

257

176

156

154 136

123 124

48

89

48

67

41

#Votes

Score

#Votes

Q2

Q1 Text rendering quality

Image-text matching

Text rendering quality

Text Color Controllability In Figure 7, we showcase TextDiffuser's capability in controlling the color of generated texts through language descriptions. The visualization results show that TextDiffuser can successfully control the color of rendered text, further enhancing its controllability.

6 Discussion and Conclusion

Discussion. We show that TextDiffuser maintains the capability and generality to create general images without text rendering in Appendix N. Besides, we compare our method with a text editing model in Appendix O, showing that TextDiffuser generates images with better diversity. We also present the potential of TextDiffuser on the text removal task in Appendix P. As for the limitations and failure cases , TextDiffuser uses the VAE networks to encode images into low-dimensional latent spaces for computational efficiency following latent diffusion models [67, 49, 2], which has a limitation in reconstructing images with small characters as shown in Appendix Q. We also observed failure cases when generating images from long text and showed them in Appendix Q. As for the broader impact , TextDiffuser can be applied to many designing tasks, such as creating posters and book covers. Additionally, the text inpainting task can be used for secondary creation in many applications, such as Midjourney. However, there may be some ethical concerns, such as the misuse of text inpainting for forging documents. Therefore, techniques for detecting text-related tampering [86] need to be applied to enhance security. In conclusion, we propose a two-stage diffusion model called TextDiffuser to generate images with visual-pleasing texts coherent with backgrounds. Using segmentation masks as guidance, the proposed TextDiffuser shows high flexibility and controllability in the generation process. We propose MARIO-10M containing 10 million image-text pairs with OCR annotations. Extensive experiments and user studies validate that our method performs better than existing methods on the proposed benchmark MARIO-Eval. For future work, we aim to address the limitation of generating small characters by using OCR priors following OCR-VQGAN [66] and enhance TextDiffuser's capabilities to generate images with text in multiple languages. Disclaimer Please note that the model presented in this paper is intended for academic and research purposes ONLY . Any use of the model for generating inappropriate content is strictly prohibited and is not endorsed by this paper. The responsibility for any misuse or improper use of the model lies solely with the users who generated such content, and this paper shall not be held liable for any such use.

References

Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR , 2022.

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022.

Coloma Ballester, Marcelo Bertalmio, Vicent Caselles, Guillermo Sapiro, and Joan Verdera. Filling-in by joint interpolation of vector fields and gray levels. IEEE transactions on image processing (TIP) , 2001.

Darwin Bautista and Rowel Atienza. Scene text recognition with permuted autoregressive sequence models. In ECCV , 2022.

Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques , 2000.

Marcelo Bertalmio, Luminita Vese, Guillermo Sapiro, and Stanley Osher. Simultaneous structure and texture image inpainting. IEEE transactions on image processing (TIP) , 2003.

Glenn L Cash and Mehdi Hatamian. Optical character recognition by the method of moments. Computer vision, graphics, and image processing , 1987.

Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704 , 2023.

Jingye Chen, Bin Li, and Xiangyang Xue. Scene text telescope: Text-focused scene image super-resolution. In CVPR , 2021.

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 , 2016.

Hyungjin Chung, Byeongsu Sim, and Jong-Chul Ye. Come-closer-diffuse-faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. In CVPR , 2021.

DeepFloyd. Github link: https://github.com/deep-floyd/if, 2023.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS , 2021.

Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In CVPR , 2021.

Zhida Feng, Zhenyu Zhang, Xintong Yu, Yewei Fang, Lanxin Li, Xuyi Chen, Yuxiang Lu, Jiaxiang Liu, Weichong Yin, Shikun Feng, et al. Ernie-vilg 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts. In CVPR , 2023.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In NeurIPS , 2022.

Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Minzhe Niu, Hang Xu, Xiaodan Liang, Wei Zhang, Xin Jiang, and Chunjing Xu. Wukong: 100 million large-scale chinese cross-modal pre-training dataset and a foundation framework. In NeurIPS Workshop , 2022.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR , 2022.

Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In CVPR , 2016.

Kamal Gupta, Alessandro Achille, Justin Lazarow, Larry Davis, Vijay Mahadevan, and Abhinav Shrivastava. Layout generation and completion with self-attention. In ICCV , 2021.

Haibin He, Xinyuan Chen, Chaoyue Wang, Juhua Liu, Bo Du, Dacheng Tao, and Yu Qiao. Diff-font: Diffusion model for robust one-shot font generation. In CVPR , 2023.

Liu He, Yijuan Lu, John Corring, Dinei Florencio, and Cha Zhang. Diffusion-based document layout generation. arXiv preprint arXiv:2303.10787 , 2023.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP , 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research (JMLR) , 2022.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop , 2021.

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045 , 2023.

Yupan Huang, Bei Liu, and Yutong Lu. Unifying multimodal transformer for bi-directional image and text generation. In ACM MM , 2021.

Shir Iluz, Yael Vinker, Amir Hertz, Daniel Berio, Daniel Cohen-Or, and Ariel Shamir. Wordas-image for semantic typography. arXiv preprint arXiv:2303.01818 , 2023.

Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. In NeurIPS Workshop , 2014.

Jiabao Ji, Guanhua Zhang, Zhaowen Wang, Bairu Hou, Zhifei Zhang, Brian Price, and Shiyu Chang. Improving diffusion models for scene text editing with dual encoders. arXiv preprint arXiv:2304.05568 , 2023.

Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochastic scene layout generation from a label set. In ICCV , 2019.

Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In ICDAR , 2015.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR , 2014.

Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, and Lianwen Jin. Look closer to supervise better: One-shot font generation via component-based discriminator. In CVPR , 2022.

Praveen Krishnan, Rama Kovvuri, Guan Pang, Boris Vassilev, and Tal Hassner. Textstylebrush: Transfer of text aesthetics from a single example. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) , 2023.

Junyeop Lee, Yoonsik Kim, Seonghyeon Kim, Moonbin Yim, Seung Shin, Gayoung Lee, and Sungrae Park. Rewritenet: Reliable scene text editing with implicit decomposition of text contents and styles. In CVPR Workshop , 2022.

Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library. https://github. com/facebookresearch/xformers , 2022.

Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. Layoutgan: Generating graphic layouts with wireframe discriminators. In ICLR , 2019.

Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. Trocr: Transformer-based optical character recognition with pre-trained models. In AAAI , 2023.

Minghui Liao, Zhisheng Zou, Zhaoyi Wan, Cong Yao, and Xiang Bai. Real-time scene text detection with differentiable binarization and adaptive scale fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) , 2022.

Wentong Liao, Kai Hu, Michael Ying Yang, and Bodo Rosenhahn. Text to image generation with semantic-spatial aware gan. In CVPR , 2022.

Chongyu Liu, Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Yongpan Wang. Erasenet: End-to-end text removal in the wild. IEEE Transactions on Image Processing (TIP) , 2020.

Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, and Chao Yang. Rethinking image inpainting via a mutual encoder-decoder with feature equalizations. In ECCV , 2020.

Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering. arXiv preprint arXiv:2212.10562 , 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2019.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. RePaint: Inpainting using Denoising Diffusion Probabilistic Models. In CVPR , 2022.

Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang, Di Niu, Haonan Lu, and Xiaodong Lin. Glyphdraw: Learning to draw chinese characters in image synthesis models coherently. arXiv preprint arXiv:2303.17870 , 2023.

Jianqi Ma, Zhetong Liang, and Lei Zhang. A text attention network for spatial deformation robust scene text image super-resolution. In CVPR , 2022.

Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang, Hong Wang, Yingbin Zheng, and Xiangyang Xue. Arbitrary-oriented scene text detection via rotation proposals. IEEE transactions on multimedia (TMM) , 2018.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML , 2022.

Safaa S Omran and Jumana A Jarallah. Iraqi car license plate recognition using ocr. In 2017 annual conference on new trends in information & communications technology applications (NTICT) , 2017.

Song Park, Sanghyuk Chun, Junbum Cha, Bado Lee, and Hyunjung Shim. Multiple heads are better than one: Few-shot font generation with multiple localized experts. In ICCV , 2021.

Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR , 2016.

Akshay Gadi Patil, Omri Ben-Eliezer, Or Perel, and Hadar Averbuch-Elor. Read: Recursive autoencoders for document layout generation. In CVPR , 2020.

Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. Generating diverse structure for image inpainting with hierarchical vq-vae. In CVPR , 2021.

Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning text-toimage generation by redescription. In CVPR , 2019.

Yadong Qu, Qingfeng Tan, Hongtao Xie, Jianjun Xu, Yuxin Wang, and Yongdong Zhang. Exploring stroke-level modifications for scene text editing. In AAAI , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research (JMLR) , 2020.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML , 2021.

Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML , 2016.

Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li. Structureflow: Image inpainting via structure-aware appearance flow. In ICCV , 2019.

Juan A Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli, and Pau Rodriguez. Ocr-vqgan: Taming text-within-image generation. In WACV , 2023.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Xuejian Rong, Chucai Yi, and Yingli Tian. Unambiguous scene text segmentation with referring expression comprehension. IEEE Transactions on Image Processing (TIP) , 2019.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI , 2015.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208.12242 , 2022.

Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In SIGGRAPH , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , 2022.

Markus Schreiber, Fabian Poggenhans, and Christoph Stiller. Detecting symbols on road surface for mapping and localization using ocr. In 17th International IEEE Conference on Intelligent Transportation Systems (ITSC) , 2014.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. In NeurIPS , 2022.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580 , 2023.

Baoguang Shi, Xiang Bai, and Cong Yao. An end-to-end trainable neural network for imagebased sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) , 2016.

Wataru Shimoda, Daichi Haraguchi, Seiichi Uchida, and Kota Yamaguchi. De-rendering stylized texts. In ICCV , 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML , 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers , 2022.

Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. In ICCV , 2021.

Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao. Shape robust text detection with progressive scale expansion network. In CVPR , 2019.

Wenjia Wang, Enze Xie, Xuebo Liu, Wenhai Wang, Ding Liang, Chunhua Shen, and Xiang Bai. Scene text image super-resolution in the wild. In ECCV , 2020.

Yuxin Wang, Hongtao Xie, Mengting Xing, Jing Wang, Shenggao Zhu, and Yongdong Zhang. Detecting tampered scene text in the wild. In ECCV , 2022.

James M White and Gene D Rohrer. Image thresholding for optical character recognition and other applications requiring character image extraction. IBM Journal of research and development , 1983.

Liang Wu, Chengquan Zhang, Jiaming Liu, Junyu Han, Jingtuo Liu, Errui Ding, and Xiang Bai. Editing text in the wild. In ACM MM , 2019.

Zizhang Wu, Xinyuan Chen, Jizheng Wang, Xiaoquan Wang, Yuanzhu Gan, Muqing Fang, and Tianhao Xu. Ocr-rtps: an ocr-based real-time positioning system for the valet parking. Applied Intelligence , 2023.

Xingqian Xu, Zhifei Zhang, Zhaowen Wang, Brian Price, Zhonghao Wang, and Humphrey Shi. Rethinking text segmentation: A novel dataset and a text-specific refinement approach. In CVPR , 2021.

Xixi Xu, Zhongang Qi, Jianqi Ma, Honglun Zhang, Ying Shan, and Xiaohu Qie. Bts: A bi-lingual benchmark for text segmentation in the wild. In CVPR , 2022.

Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. arXiv preprint arXiv:2211.13227 , 2022.

Ling Yang, Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Yingxia Shao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796 , 2022.

Boxi Yu, Yong Xu, Yan Huang, Shuai Yang, and Jiaying Liu. Mask-guided gan for robust text editing in the scene. Neurocomputing , 2021.

Deli Yu, Xuan Li, Chengquan Zhang, Tao Liu, Junyu Han, Jingtuo Liu, and Errui Ding. Towards accurate scene text recognition with semantic reasoning networks. In CVPR , 2020.

Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao. Diverse image inpainting with bidirectional and autoregressive transformers. In ACM MM , 2022.

Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 , 2012.

Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid-context encoder network for high-quality image inpainting. In CVPR , 2019.

Guanhua Zhang, Jiabao Ji, Yang Zhang, Mo Yu, Tommi Jaakkola, and Shiyu Chang. Towards coherent image inpainting using denoising diffusion implicit models. arXiv preprint arXiv:2304.03322 , 2023.

Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV , 2017.

Linjiang Zhang, Peng Wang, Hui Li, Zhen Li, Chunhua Shen, and Yanning Zhang. A robust attentional framework for license plate recognition in the wild. IEEE Transactions on Intelligent Transportation Systems (TITS) , 2020.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023.

Lei Zhao, Qihang Mo, Sihuan Lin, Zhizhong Wang, Zhiwen Zuo, Haibo Chen, Wei Xing, and Dongming Lu. Uctgan: Diverse image inpainting based on unsupervised cross-space translation. In CVPR , 2020.

Minyi Zhao, Miao Wang, Fan Bai, Bingjia Li, Jie Wang, and Shuigeng Zhou. C3-stisr: Scene text image super-resolution with triple clues. In IJCAI , 2022.

Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In CVPR , 2019.

Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. East: an efficient and accurate scene text detector. In CVPR , 2017.

Xinyan Zu, Haiyang Yu, Bin Li, and Xiangyang Xue. Weakly-supervised text instance segmentation. arXiv preprint arXiv:2303.10848 , 2023.

Appendix

A Architecture of U-Net and Design of Character-Aware Loss

As shown in Figure 9, the U-Net contains four downsampling operations and four upsampling operations. The input will be downsampled to a maximum of 1/16. To provide the character-aware loss, the input feature F is 4-D with spatial size 64 × 64 , while the output is 96-D (the length of alphabet A plus a null symbol indicating the non-character pixel) also with spatial size 64 × 64 . Subsequently, a cross-entropy loss is calculated between the output feature (need to convert the predicted noise into predicted features) and the resized 64 × 64 character-level segmentation mask C ′ . The U-Net is pre-trained using the training set of MARIO-10M for one epoch. We utilize the Adadelta optimizer [97] and set the learning rate to 1. When training the diffusion model, the U-Net is frozen and only used to provide character-aware guidance.

B Character-Level Segmentation Model

We train the character-level segmentation model based on U-Net, whose architecture is similar to the architecture shown in Figure 9. We set the input size to 256 × 256 , ensuring that most characters are readable at this resolution. We train the segmentation model using synthesized scene text images [19], printed text images, and handwritten text images 3 , totaling about 4M samples. We employ data augmentation strategies ( e.g. , blurring, rotation, and color enhancement) to make the segmentation model more robust. The segmentation model is trained for ten epochs using the Adadelta optimizer [97] with a learning rate of 1. Figure 10 shows some samples in the training dataset.

phosphyl

dicalnes

semples

microprogramable

umorosenes

oresetmentoizng

4

Zipangu

fellation

tetremimerar

semi-indirectness

-l中m

Lribromide

peace-breathing

l-ll

Hialf-englin

Got a

Date:

problemswe

ancan

can

uone

C More Details in MARIO-10M

More samples are shown in Figure 11. The number of texts per image in MARIO-10M is shown in Table 5. Also, the MARIO-10M dataset reveals that about 90% of the text regions maintain a horizontal orientation with rotation angles smaller than 5 degrees without perspective changes. Hence, our layout generation model is designed to predict horizontal bounding boxes by detecting the coordinates of their left-top and bottom-right points. Adapting our model to predict more realistic scene text is feasible by detecting enhanced coordinates, such as eight coordinates for four points.

D MARIO-10M Caption Templates

Since TMDB movie/TV posters and Open Library book covers have no off-the-shelf captions, we construct them based on their titles with the following templates. {XXX} is a placeholder for title.

For MARIO-TMDB :

Logo {XXX}

Text {XXX}

Title {XXX}

Title text {XXX}

A poster with a title text of {XXX}

A poster design with a title text of {XXX}

A quality movie print with a title text of {XXX}

A film poster of {XXX}

A movie poster of {XXX}

A movie poster titled {XXX}

For MARIO-OpenLibrary :

A book with a title text of {XXX}

A book design with a title text of {XXX}

A book cover with a title text of {XXX}

A book of {XXX}

A cover named {XXX}

A cover titled {XXX}

A book with text {XXX} on it

A book cover with logo {XXX} on it

A movie poster named {XXX}

A movie poster with text {XXX} on it

A movie poster with logo {XXX} on it

A movie poster with a title text of {XXX}

An illustration of {XXX} movie

An photography of {XXX} movie

A TV show poster titled {XXX}

A TV show poster of {XXX}

A TV show poster with logo {XXX} on it

A TV show poster with a title text of {XXX}

A TV show poster with text {XXX}

A TV show poster named {XXX}

(a) Samples in MAION-LAION

(b) Samples in MARIO-TMDB

8484

COLORiNGBOOK

ONE DOES NOT SIMPLY

STANDARD

mersive

The 10

Ganly

UNLIMITED

FOOD

SOFTWARE

Best Sport!

bovfbale

Clothes Review

She bid It

Merry

the Berlin

Attracting Success &

1Am Successfuly

Successful People

Christmas

DESIGNER

TOUCH

TIOETOE

Badge

90mm

Deception

LEBANESE

BANK

000

around Me

Hapy New

year!

BOORSELVES

4pcs

#1 Favel fectingy

CredtCerd

FILETYPES

HOWTOANALYZE

P

S

RUNLIKE

YOU

NOWV

STOLE

2020

Homeloans

LIFE

SOMETHING

Just

MELBOURNE

.com.au

20

ICHLINEIS

Bije

Curved EdgeFlower

LifetoCome

KEP

4

KEEP

CALM

CALM

THETRAGEDY

EM.FORSTER

LOVEA

AND

FILM PRODUCER

psychotherapic

CARRY

ISNTITMAGICKITTYWARD?

training

coaching

ON

O'DONOGHUES

OPERA

MONANON

DEAD

TRILOGY

FLUSH

DOLLAR

everychristmas

SHATTER

BESTSELLI

TEMPUS

RISE

SURDOU

PROXU

ENVANLIG FUCKING

MANNISKA

STRACONA

NOC

Ciroumgtnntie2

TOMORROW

MAN

TURNER

THE NURSE

L.NOO

DontForget

moh anol T

FRONTPAGE

ANH

Garfunkel and Oates

Teyipeta

ETR

GO

OUTSIDE

FRANKENSTEIN

BurningDreams

WRSNERS

WINCHESTER'73

家药中国

KnivesOut

Gotteszell

FOUNDER

TWAKENS

Shannon

VOYAGE

Strawberries

KSNOERGADTEN

are

ShortCru

red

ALONE

GETHER

Ss01)

EIRWORK

garden

His

Ensalada

Grace

of

faiThGirLz!

the Light

YELLOWCARD

DUST

AFE

Sophie'sEncore

and theDark

LIGHTS-60

RIS

SIN

Life Of Aaron Burr

The

MarcusLyndale

LemLirs

meurtriere

P. D.James

品金

La

fromall

God's

CllffFarrell

AFainfess

shortworksof

arkling

COMANCH'

689

E MARIO-10M Filtering Rules

We clean data with five strict filtering rules to obtain high-quality data with text:

Height and width are larger than 256 . Low-resolution samples often contain illegible texts, negatively impacting the training process.

Will not trigger NSFW . For the MARIO-LAION subset, we filter out those samples triggering the 'not sure for work' flag to mitigate ethical concerns.

The number of detected text boxes should be within [ 1,8 ] . We detect texts with DB [42]. Samples with too many texts typically have small areas for each text, which makes them difficult to recognize. Therefore, we remove these samples from the dataset.

Text areas are more than 10% of the whole image . According to Appendix B, we train a UNet [69] using SynthText [19] to obtain character-level segmentation masks of each sample. This criterion ensures that the text regions will not be too small.

At least one detected text appears in the caption . Noticing that the original dataset contains many noisy samples, we add this constraint to increase the relevance between images and captions. We utilize PARSeq [4] for text recognition.

F Analysis of OCR Performance on MARIO-10M

As we rely on OCR tools to annotate MARIO-10M, it is necessary to evaluate the performance of these tools. Specifically, we manually annotate 100 samples for text recognition, detection, and character-level segmentation masks, then compare them with the annotations given by OCR tools. The results are shown in Table 6. We notice that the performance of existing methods is lower than their results on text detection and spotting benchmarks. Taking DB [42] as an example, it can achieve text detection 91.8% precision on ICDAR 2015 dataset [34] while only achieving 76% on MARIO-10M. This is because there are many challenging cases in MARIO-10M, such as blurry and small text. Besides, a domain gap may exist since DB is trained on scene text detection datasets, while MARIO-10M comprises text images in various scenarios. Future work may explore more advanced recognition, detection, and segmentation models to mitigate the noise in OCR annotations. We demonstrate some OCR results in Figure 12.

Detection

Spotting

Segmentation

Precision

Recall

F-measure

Precision

Recall

F-measure

IOU (binary)

IOU

0.76

0.79

0.78

0.73

0.75

0.74

0.70

0.59

Pride

Chitty

about

hocol

Peppalig

2015

GREENE

HIGH

MonkeY

WOOL

2003

epop

LONORES

POLISH

Dancesport

RILEY

CAPTURE

LIES

RELEASE

STA

GRAVE

IEANIENE

DIARY

FREE!

Prid

Chitty

BeppaPg

2015

GREENE

FESTIVAL

MonkeY

WOOL

2003

epop

LONORES

POLISH

Danceshort

CAPTURE

FLIES

RELEASE

GRAVE

TEANIENE

DIARY

MMedium

UP VOTES

Medium

OTES

SEGMENTATION

CUSTOMER

INBEAUTYAND

CUSTOMER

INBEAUTYAND

SEGMENIATON

TWIN

TWIN

COSMETICS

COSMETICS

CARMP5PLAYER

CARMPSPLAYER

aMcCain

Freeview

Freevlew

Jahannam

Houlnakiyon

KuBayan

Tahannam

Houtnakiyon

STREET

FRIES

STREET

FRIES

00+00

HD

HD

BEE

BT Handsfree

FM/USB/TF/AUX

品

品品

Cheaper.

Cheaper

Beller.

BABY

Faster

Guat hrandna

BOY!

u

BOY!

Brusselmans

Brusselmans

wbs

wbs

MaryHunt

indemode

WARWICK BUSINESS SCHOOL

THE UNIVERSITY OF WARWICK

TOM

DINOSAUR

CLANCY'S

CTIVITY

NATIONAL

SpanishCreative

30

ASSOCIATIONine:

WritingActivity

INTO

Jefernafinal

WinterHolidays

POLO

AVP

INTEGRITYONE

CAR

CENTER FOR

AUTOMOTIVE

SOLUTIONS

RESEARCH

Technology.uc

Thin FIm Eqipment Senices

5ReasonsWhy

TOM

FarmersNeed

Vacations

CLANCY

NORTH

THEI

FACE

COMMAND

AUTHORITY

FREEPRINTABLE

BOOKLET

G Samples in MARIO-Eval

As illustrated in Table 7, MARIO-Eval contains 5,414 prompts with six subsets. The ground truth images of some samples are shown in Figure 13, and captions for each category are shown below:

LAIONEval4000 :

'Royal Green' Wristband Set

Are 'Digital Nomads' in Trouble Well Not Exactly

'Sniper Elite' One Way Trip A Novel Audiobook by 'Scott' McEwen Thomas Koloniar Narrated by Brian Hutchison

'Travel Artin' Logo

Falls the 'Shadow' Welsh Princes 2

TMDBEval500 :

A movie poster with text 'Heat' on it

A poster design with a title text of 'Deadpool 2'

A TV show poster named 'Ira Finkelstein s Christmas'

A movie poster with logo 'Playing for Change Songs Around The World Part 2 ' on it

A movie poster titled 'Dreams of a Land'

OpenLibraryEval500 :

A book cover with a title text of 'On The Apparel Of Women'

A book with text 'Precalculus' on it

A book design with a title text of 'Dream master nightmare'

A book cover with logo 'Thre Poetical Works of Constance Naden' on it

A book cover with a title text of 'Discovery'

DrawBenchText :

A storefront with 'Hello World' written on it.

A storefront with 'Text to Image' written on it.

A sign that says 'Diffusion'.

A sign that says 'NeurIPS'.

New York Skyline with 'Google Research Pizza Cafe' written with fireworks on the sky.

DrawTextCreative :

a grumpy sunflower with a 'no solar panels' sign

A photo of a rabbit sipping coffee and reading a book. The book title 'The Adventures of Peter Rabbit' is visible.

a graffiti art of the text 'free the pink' on a wall

A professionally designed logo for a bakery called 'Just What I Kneaded'.

scholarly elephant reading a newspaper with the headline 'elephants take over the world'

ChineseDrawText :

A street sign on the street reads 'Heaven rewards those who work hard'

There is a book on the table with the title 'Girl in the Garden'

Kitten holding a sign that reads 'I want fish'

A robot writes 'Machine Learning' on a podium

In a hospital, a sign that says 'Do Not Disturb'

RISK

77Hag

Adventure

Auaits

OSITIVE

parenting guotes

DENSATDANM

'Adventure Awaits'

'Risk Everything'

'50 positive parenting quotes'

VATONY THE

Home Blanket by

gentle parenting peaceful

TYGVOLUME

parentinggentle discipline

TWO THE DAY

BREAKSATDAWNLP

BURDEN

DREAMS

LIFEto

Adam'siCrossing

AFTERLIFE

DeathandBack

A poster with a

A TV show poster of

A poster with a title text

A movie poster with

A movie poster with logo

title textof'Frank'

'BurdenofDreams'

of'LifetoAfterlife

a titletextof'StarWars'

Death and Back'

Butterfly's

EXPRESIONISMO

Surprise

The

A cover titled

A cover titled'Boulez

A book design with a

A bookwith a title

A bookwithtext

'SleepingBeauty'

on Conducting'

title text of

text of'Expresionismo'

'The master mummer'on it

H Implementation Details of Evaluation Criteria

To evaluate the performance of TextDiffuser quantitatively, we utilize three criteria, including FID, CLIPScore, and OCR Evaluation. We detail the calculation of each criterion below.

FID. We calculate the FID score using the pytorch-fid repository. Please note that the proposed MARIO-Eval benchmark's three subsets (DrawTextCreative, DrawBenchText, and ChineseDrawText) do not contain ground truth images. Therefore, we utilize 5,000 images in the other three subsets (LAIONEval4000, TMDBEval500, OpenLibraryEval500) as the ground truth images. We calculate the FID score using the 5,414 generated images and the 5,000 ground truth images.

CLIP Score. We calculate the CLIP score using the clipscore repository. However, as with the FID score, we cannot calculate the CLIP score for the DrawTextCreative, DrawBenchText, and ChineseDrawText subsets due to the lack of ground truth images. Therefore, we only calculate the score on LAIONEval4000, TMDBEval500, and OpenLibraryEval500 subsets and report the average CLIP score.

OCR Evaluation. For the MARIO-Eval benchmark, we use quotation marks to indicate the keywords that need to be painted on the image. Taking the caption [A cat holds a paper saying 'Hello World'] as an example, the keywords are 'Hello' and 'World'. We then use Microsoft Read API to detect and recognize text in the image. We evaluate OCR performance using accuracy, precision, recall, and F-measure. If the detected text matches the keywords exactly, it is considered correct. Precision represents the proportion of detected text that matches the keywords, while recall represents the proportion of keywords that appear in the image. We report the mean values of precision and recall, and calculate the F-measure using the following formula:

I Visualization of Layouts Generated by Layout Transformer

We visualize some generated layouts in Figure 14, showing that the Transformer can produce reasonable layouts.

A robot writing 'Ethics 101'

in chalk on a blackboard.

A storefront with 'Google Research

Pizza Cafe' written on it.

An antique bottle labeled 'Energy Tonic'

A giant shoe, with the

caption 'shoe for hokey pokey'

A poster titled 'Quails of North America',

showing different kinds of quails.

A sign that says 'NeurIPS'.

Ethics

Ethics

Google

Google

NeurlPS

NeuriPS

Research

Research

101

101

Pizza

Pizza'

Cafe

Cafe

shoe

shoe

Quails

of

Qualisof

Energy

Energy

for

for

North

North

Tonic

Tonic

hokey

Hokey

America

America

pokey

pokey

Reyporey

Deep

Deep

Hello

HeTlo'

Google

Google

Brain

Brain

earning

Learning

World

World

Toronto

Toronto

A storefront with 'Deep Learning' written on it.

A storefront with 'Hello World' written on it.

A sign that says 'Google Brain Toronto'.

J Experiment without Explicit Guidance of Segmentation Masks

As shown in Figure 15, we try to explore the generation without explicit guidance. For example, according to the first row, we set the value of character pixels to 1 and non-character pixels to 0 ( i.e. , remove the content and only provide the position guidance). We observe that the model can generate some words similar to keywords but contain some grammatical errors ( e.g. , a missing 'l' in 'Hello'). Further, according to the second row, we train TextDiffuser without segmentation masks ( i.e. , remove both position and content guidance). In this case, the experiment is equivalent to directly fine-tuning a pre-trained latent diffusion model on the MARIO-10M dataset. The results show that the text rendering quality worsens, demonstrating explicit guidance's significance.

Happy

SOSTOTP

Cood

World

Morning

w/o Pos & Con

SOP

Helolo

STOP

pooog

Monilng

w/ Pos & Con

Hello

Happy

Stop

Good

om

Morning

A text image of

A text image of

A board says

A board says

'Hello World'

'Happy'

'Stop'

'Good Morning'

K Baseline Methods Experimental Settings

We introduced all baseline methods and their experimental settings when we used to compare them with the TextDiffuser as follows.

DALL · E [62] utilizes a text encoder to map a given prompt into a corresponding representation space. A prior model is then employed to map the text encoding to an image encoding. Finally, an image decoder generates an image based on the image encoding. Since there is no available code and model, we obtain the results using the provided API 4 .

Stable Diffusion (SD) utilizes CLIP [60] text encoder to obtain the embedding of user prompts, pre-trained VAE to encode original images and conducts the diffusion process in the latent space for computation efficiency. We use the public pre-trained model 'runwayml/stable-diffusion-v1-5' based on Hugging Face diffusers [82]. The number of sampling steps is 50, and the scale of classifier-free guidance is 7.5.

Stable Diffusion XL (SD-XL) is an upgraded version of SD, featuring more parameters and utilizing a more powerful language model. Consequently, it can be expected to better understand prompts compared to SD. As the source code and model are not publicly available, we obtained the generation results through a web API 5 .

Midjourney 6 is a commercial project that runs on Discord, allowing users to interact with a bot via the command-line interface. We generated images using the default parameters of Midjourney. For example, we can generate an image using the following command: /imagine an image of 'hello world' in Midjourney.

ControlNet [102] aims to control diffusion models by adding conditions using zero-convolution layers. We use the public pre-trained model 'lllyasviel/sd-controlnet-canny' released by ControltNet authors and the implementation from Hugging Face diffusers [82]. For fair comparisons, we use the printed text images generated by our first-stage model to generate Canny maps as the condition of ControlNet. We use default parameters for inference, where the low and high thresholds of canny map generation are set to 100 and 200, respectively. The number of inference steps is 20, and the scale of classifier-free guidance is 7.5.

DeepFloyd [12] designs three cascaded pixel-based diffusion modules to generate images of increasing resolution: 64x64, 256x256, and 1024x1024. All stage modules use frozen text encoders based on T5 Transformer [12]. Compared with CLIP [60], the T5 Transformer is a powerful language model that enables more effective text understanding. We use the public pretrained models released by DeepFloyd authors and the implementation from Hugging Face diffusers [82]. We use default models and parameters for inference, where the three pretrained cascaded models are 'DeepFloyd/IFI-XL-v1.0' , 'DeepFloyd/IF-II-L-v1.0' , and 'stabilityai/stable-diffusion-x4-upscaler' .

L Visualization of More Generation Results by Our TextDiffuser

TMDBEval500

LAIONEval4000

OpenLibraryEval500

LetYour

problem

solving

Light

Team

cctv

steps

Fam

Shineo

skills

TheWitch

Cheer

History

new

業

Sampa

Hostel

Year

The.

Tango

Gretel

Society

High

Personalities

Show

Kindergarlen

and

State

Deep

LeeBiuski

Big

Tarm

12

Bhort

The

Dry

The

world

Anything

Monster

ofze

cafs

PIO

S

Polo

Harthill

possible

Programming

Through

Skills

Wavewalker

Lantern

Green

treasure

Tideland

Math

Life

and

for

Food

Serv

DrawTextCreative

DrawBenchText

ChineseDrawText

shioe for

hokey pokey

There is

no planet

B

NeuriPs

Google

Brain

Torontd

1

No

Gambling

No

Smoking

da tobeh

Recipesfrom

Peru

Deep

Learning

NeuriPS

1

1

1

1

I

want to

learn

Science

is

Fall here

ready for

the weekend

Diffusion

Hello

World

Planet

Earth

Do not

feed

your owny

sole

diamonds

Coogle

Research:

Pizza

Cafe

Google

Research

Pizza

Cafe

1

1

1

1

Local

pig

eats

prize

pumpkin

Fairy

Tales

the free

pink

gma

Hello

World

Diffusion

1

1

1

1

Danger

Minefield

LOVE

a stop pizza

a hello world banner

a stay calm quotes poster

Time Traveling

Time Traveling

Device

Device

Iamafraidofthedark

Iamafraidofthedark

Having a dog

Havinga dog

Elephant Take Over

zlepbant Take Over

named shark at

the beach

the beach

was a mistake

THE

EARTH

THE

EARIA

make

make

SAVE

rtsafe

ifsafe

nothig

to

MY OLD HASIT

My

OLD

HABIT

true

here

Tree

Fiero

STOP

-STOP

Please

Please

Adopt

Adopt

Me!

Me!

Hello

Hello,

STAYCALMQUOTES

STAYCALMQUOTES

World.

World.

MANDIANT

ANDIANT

MANDIANT

GIFT

GIFT

GIFT

EYE ON

NO 3R3

EYE ON

50

VOUCHER

50

VOUCHER

HAIRCUT

50

VOUCHER HAIRCUT

SECURITY

SECURITY

→

SECURITY

PROTECTION

PROTECTION

De

tipsfors

3

Update

Er

Upgrade

Upgrade

Ei

SMALL

MIDDLE

MIDDLES

→

SPACE

SPACE

SPACE

Shift

Shift

LIVING

LIVING

LIVING

George

George

George

CREATEYOUR

CREATE YOUR

CREATEYOUR

isa proud

BIG

is a proud

BIG

isa proud

BIG

2014

OWN

2023

OWN

2023

OWN

brother!

sister

sister

CALENDAR!

CALENDAR!

CALENDAR!

CAUTION

CAUTION

CAUTION

HOUSTON

HOUSTON

HOUSTON

PROTECTIVE

WEAR

PROTECTIVE

WEAR

PROTECTIVE

WEAR

WEARS

THE

WE ARE

THE

WEARE

THE

EQUIPMENT

EQUIPMENT

EQUIPMENT

PROBLEM

GREAT

GREAT

NECESSARY

NECESSARY

PEOPLE

PEOPLE

TextDiffuserl

SPULSPONNS

POPPins

OPOPPinse

PLEPEP

PLEP6P

MEMIE

MEM

MEM

SOOMOP

Mono

Mono

M More Details about User Study

User study on the whole-image generation task. The questionnaire consists of 15 cases, each of which includes two multiple-choice questions:

Which of the following images has the best text rendering quality?

Which of the following images best matches the text description?

In particular, the first question focuses on the text rendering quality. Taking Figure 20 as an example 7 , we expect the model to render the word 'EcoGrow' accurately ( i.e. , without any missing or additional characters). The second question, on the other hand, focuses on whether the overall image matches the given prompt. For example, in Figure 20 (G), although the generated text is correct, it fails to meet the requirement in the prompt that the letter looks like a plant. We instruct users to select the best option. In cases where multiple good options are difficult to distinguish, they could choose multiple options. If users are unsatisfied with the options, they could decide not to select any.

CIOVE

ECOGFOW

ecoogrow

(A)

(B)

(C)

(D)

EcoGrow

ECCCOV

(E)

(F)

(G)

[(8/15)]A logo for the company'EcoGrow', where the letters look like plants.

User study on the part-image generation task. We aim to let users vote on the quality of text inpainting (from 4 to 1, the higher, the better). We also designed two questions:

How is the text rendering quality?

Does the drawn text harmonize with the unmasked region?

Specifically, the first question concentrates on the accuracy of the text. The second question focuses on whether the generated part is harmonious with the unmasked part ( i.e. , whether the background and texture are consistent).

TECH

TECH

TECH

HOUSE

HOUSE

HOUSE

INTERESTING

INTERESTING

TERRACE

TERRACE

TERRACE

PARTY

PARTY

PARTY

N Generating Images without Text

To show the generality of TextDiffuser, we experiment with generating images that do not contain texts and show results in Figure 22. Although TextDiffuser is fine-tuned with MARIO-10M, it still maintains a good generation ability for generating general images. Therefore, users have more options when using TextDiffuser, demonstrating its flexibility. We also provide quantitative evaluations to demonstrate TextDiffuser's generality in generating non-text general images. We compare TextDiffuser with our baseline Stable Diffusion 1.5 as they have the same backbone. For a quantitative evaluation, the FID scores of 5,000 images generated by prompts randomly sampled from MSCOCO are as in Table 8. The results indicate that TextDiffuser can maintain the ability to generate natural images even after fine-tuning the domain-specific dataset.

Stable Diffusion 1.5

TextDiffuser (Ours)

Stable Diffusion 1.5

TextDiffuser (Ours)

Stable Diffusion 1.5

TextDiffuser (Ours)

A clean, organized bathroom with accessories in it.

A horse with the sky and grass in the background.

A young lady riding skis down a snow covered slope.

Stable Diffusion 1.5

TextDiffuser (Ours)

Stable Diffusion 1.5

TextDiffuser(Ours)

Stable Diffusion 1.5

TextDiffuser (Ours)

A tablewithservings of pastries on trays.

A DailyMirror featuring stories on celebrity stuffed animals.

A man riding on top of a wave on a surfboard.

O Comparisons between TextDiffuser and a Text Editing Model

We visualize some results in Figure 23 compared with a text editing model SRNet [88]. Please note that the introduced text inpainting task differs from the text editing task in three aspects: (1) The text editing task usually relies on the synthesized text image dataset for training (synthesizing two images with the different text given a background image and font as pairs). In contrast, the text inpainting task follows the mask-and-recover training scheme and can be trained with any text images. (2) Text editing emphasizes the preservation of the original fonts, while text inpainting allows for greater freedom. For example, we conduct four samplings for each case, and the generated results exhibit diversity and present reasonable font styles. (3) Text editing tasks cannot add text, highlighting the significance of the introduced text inpainting task.

Original Image

Editing Result

Our Result 1

Our Result 2

Our Result 3

Our Result 4

PEDESTRIANS

Sessions

Sessions

Sessions

Sessions

Sessions-

←Footpath.To.

ASSYRIAN.on

ASSYRIAN ON

ASSYRIAN ON

ASSYRIAN ON

ASSYRIAN ON

Colchester and

UNFLAGGING FRY

UNFLAGGINGFRY

UNFLAGGING FRY

UNFLAGGING FRY

UNFLAGGINGFRY

Greenstead..

DEVASTATES...

DEVASTATES

DEYASTATES

DEVASTATES

DEVASTATES

Campus

Rankin

Rankin

Rankin

Rankin

Rankin

Shop

dork

Dork

Dork

Dork

Dark

RIVERSIDE

ALAPENOS

JALAPENOS

JALAPENOS

JALAPENOS

JALAPENOS

WALK

Reds

iReded

★Reds #

浪Reds中

-Reds

P Experimental Results of Text Removal

We demonstrate some results of text removal in Figure 24, and the cases are obtained from the paper of EraseNet [44]. We can easily transform the text inpainting task into a text removal task by providing a mask and setting all regions to non-character in the segmentation mask. Experimental results demonstrate that our method can achieve results similar to the ground truth.

Original Image

Ground Truth

Inpainting Mask

Our Result

ORNER

WATER

OVER

ROAD

DO NOT

ENTER

★★

Q Limitations and Failure Cases

We observe failure cases when generating images with small characters and from long text.

Generating images with small characters. TextDiffuser uses the V AE networks to encode images into low-dimensional latent spaces for computational efficiency following latent diffusion models [67, 49, 2]. However, the compression process can result in losing details when generating images with small characters. As illustrated in Figure 25, we observe that the V AE fails to reconstruct small characters, where reconstructed strokes are unclear and reduce the legibility of the text. According to the generated images, the small characters appear to have vague or disjointed strokes ( e.g. , the character 'l' in 'World' and character 'r' in 'Morning'), which could impact the readability. As shown in Figure 26, we notice that using a more powerful backbone, such as Stable Diffusion 2.1, can mitigate this issue. When the image resolution is enhanced from 512 × 512 to 768 × 768 using Stable Diffusion 2.1 (instead of 1.5), the latent space resolution also increases from 64 × 64 to 96 × 96, enhancing the character-level representation. As the cost, the inference latency rises from 8.5s to 12.0s with a batch size of 1. Therefore, how to render small characters while maintaining the same time cost is worth further study.

Generating images from long text. We observed failure cases when generating images from long text with many keywords, where the generated words in the layouts are disordered and overlapped,

PARKES

Original Image

Reconstructed Image

a board of 'Hello World'

a board of 'Good Morning'

Generated Image

Bubbling

Fairy

Bubbling Fairy.

#Mofsturising

adbsonine

wWorld

Wold

Knitlandia

Knitlandia

S39S

566S

Morning

Good

Motning

CLARA

PARKES

CLARA

VeTy

Big

News

>

Very

Big

News

A

Having

having

Big

Big

Having

adog

having

bope

named sharh

thebeach

at

sharh

named

the beach

shark

at

shark

An

Time-

Exitemely

Iraveling

Cool

Exitemely

An

Time

Extremely

Traveling

Cool

Extremely

wasamistake

wasamistake

Device

Device

Traveling

Traveling

GeneratedByAl

GeneratedByAl

SD1.5

SD2.1

SD1.5

SD2.1

A board saying having a dog named shark at the beach was a mistake

ahand-drawnblueprintfor a timemachine

as shown in Figure 27. One possible reason could be that training examples containing numerous keywords tend to have more noise ( i.e. , those images usually contain dense and small text), leading to a higher likelihood of detection and recognition errors. To address this, we could consider enhancing the capability of OCR tools in the future to mitigate the noise.

an image of 'I have a

dream during my 17'

a board of 'My friend

please adopt this cat'

a meme of 'please do not eat my hamburger'

a book of 'my favorite

sport is ping pong'

have

durin

a

Ahgam

17

I

Ihave

a

durin

17

My

friend

please

thagopt

cat

My

triend

please

th7gopt

cat

not

eat

my

haniburger

please

do

not

eat

my

DAknuhaoer

my

favorite

sport

is

pong

ping

my

favoiite

sport

ping

IS

pong

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed TextDiffuser to achieve accurate text rendering for diffusion models. TextDiffuser generates character-level text layouts to guide the text rendering process (image generation). The model is evaluated on 3 tasks including text-to-image, text-to-image with template, and text inpainting to demonstrate its flexibility and controllability. Additionally, the paper contributes the first large-scale text image dataset with OCR annotations, named MARIO-10M.\n\nSTRENGTHS:\n1.\tThe paper is well motivated, which tries to solve the problem of generating text image with diffusion model. Also, the paper is well organized and easy to follow. \n2.\tThe TextDiffuser model is flexible, which can be adaptive with different conditions as shown in the experiments.\n3.\tA new dataset specifically designed for text rendering is proposed.\n\nWEAKNESSES:\n1.\tMismatch of character level layout and the real text. The character level layout is generated from standard fonts (rendered using toolkit such as opencv), however, the real character can be of any style or font, which leads to a mismatch of character level layout. In another word, guided by such a layout, the generated text will tend to have the same character interval and aspect ratio of the standard font, which means it will limit the generation diversity of fonts.\n2.\tGeneration diversity has not been considered. There exist several reasonable text styles for the same background image. As the proposed model cannot explicit control the attribute of texts such as color, layout, font, style, etc., it is important to measure the generation diversity of the model.\n3.\tThe performance comparison seems to be limited to general text-to-image diffusion models with no specific optimization on text painting. A comparison with some state-of-the-art text generation methods is necessary to show the quality of text generation.\n\nQUESTIONS:\n1. How does the model perform on more complex characters such as Chinese characters? A simple layout guidance seems to only constrain the position and order of letters, without any specific improvement on the quality of generation details. I wonder if there is any method to improve the adaptiveness on other language?\n\nLIMITATIONS:\nLimitations and potential impacts have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes TextDiffusers, a 2-stage model to generate images with text per the input text prompt. TextDiffusers also supports text inpainting.  The two stages consist of (1) estimation of layout of keywords (inspired by Layout Transformer) to get character-level segmentation masks and (2) image generation conditioned on the character-level segmentation masks.\n\nThe paper also contributes to MARIO-10M, a large data set of image-text pairs along with rich OCR annotations. Benchmark MARIO-Eval with 5414 prompts is created from MARIO-10M and few other existing datasets, in order to evaluate quality of rendering text in the image.\n\nStage 1 consists of novelty such as encoding of width of keywords. Encoding width of keywords improves IoU, especially for shallower Layout Transformer. Stage 2 introduces character-aware loss and extends the denoising loss to include character level segmentation masks.\n\nThe paper contains implementation details for reproducibility, comparison against the states of the art demonstrating large improvements in OCR quality metrics, comprehensive ablations studies, several qualitative samples and limitations.\n\nSTRENGTHS:\nS1) Novel approach to generate images with text\n\nS2) New data set for MARIO-10M along with benchmark MARIO-Eval\n\nS3) Comprehensive experiments demonstrating large improvements against various states of the art on OCR metrics\n\nWEAKNESSES:\nW1) While the methodology introduced by the paper seems solid, the main weakness is comparison against strong baseline which was also trained on training split from MARIO-10M. Appendix J alludes to equivalency of fine-tuning of pre-trained model, but Stable Diffusion model is not the best when it comes to producing visual text. It is understandable that official code for Imagen, Parti are not available. Perhaps DeepFloyd could have been fine-tuned on MARIO-10M. From Table 4, DeepFloyd is also better on Fidelity (lower FID score). It is not clear if it would perform better than TextDiffusers with this additional fine-tuning. \n\nW2) Information and comparison of model size and training/inference latency are missing.\n\nW3) Appendix N / Figure 14 in the supplementary material includes a handful of sample generations without text. However, it is not clear if TextDiffusers loses its capability to generate such images in the general sense. Qualitative and Quantitative evaluations to test this are missing.\n\nQUESTIONS:\nPlease see list of weakness. \n\nQ1) According to Line 188, the maximum length of tokens (L) is limited to 77. Do all captions in the dataset contain <= 77 tokens?\n\nQ2) Related to Q1, DeepFloyd uses T5-XXL which is much more powerful (as observed by Imagen). Although L<=77, do you have insights on the richness of embedding that may give it a higher edge over TextDiffusers?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents an approach to enhance the text rendering ability of a text-to-image diffusion model. The authors introduce a new diffusion model called text-diffuser that leverages image captions and text segmentation masks to generate text images. They also collect a large-scale image dataset MARIO-10M, which includes explicit text information such as segmentation masks and OCR annotations. The experimental results indicate a significant improvement in text generation ability but a slight decrease in image quality.\n\nSTRENGTHS:\n* The large-scale text image dataset with text annotations MARIO-10M is commendable is valuable for further research in this domain.\n* Incorporating character-level segmentation masks to enhance the text rendering ability of a diffusion model is a novel idea. The results presented in the paper demonstrate the effectiveness.\n\nWEAKNESSES:\n* The paper mentions that the layout generation model is trained to generate text segmentation masks, which might not be suitable for scene texts with significant perspective changes.  Most of the generation examples lack realistic text styles with perspective changes and complicated text layouts.\n* Some GAN-based works [1, 2, 3] exploring scene text editing and a recent diffusion-based scene text editing work [4] could be relevant for part-image generation evaluation. Authors could consider incorporating a discussion and comparison with these related works.\n\n\n[1] Wu, et al . “Editing Text in the Wild.” https://doi.org/10.1145/3343031.3350929.\n\n[2] Qu, et al. “Exploring Stroke-Level Modiﬁcations for Scene Text Editing,” https://ojs.aaai.org/index.php/AAAI/article/view/25305\n\n[3] Krishnan, et al. “TextStyleBrush: Transfer of Text Aesthetics from a Single Example.”  http://arxiv.org/abs/2106.08385.\n\n[4] Ji, et al. “Improving Diffusion Models for Scene Text Editing with Dual Encoders.” http://arxiv.org/abs/2304.05568.\n\nQUESTIONS:\n* It would be interesting if we can control the style of generated texts through language descriptions on textdiffuser. For example, the image s generated with this caption: a bear holding a board with a **red/purple/yellow** \"hello world\"? It would be beneficial if the authors could provide some generation examples showcasing this capability.\n\nLIMITATIONS:\nAuthors discussed the limitation of the trained model, including small text generation and multiple-word generation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a model for generating visually appealing and coherent text within diffusion models. It also presents the MARIO-10M dataset and the MARIO-Eval benchmark for evaluating text rendering quality. Experimental results demonstrate their method's flexibility and controllability in creating high-quality text images and performing text inpainting.\n\nSTRENGTHS:\nThe strengths of this paper include the proposal of TextDiffuser, a flexible and controllable framework based on diffusion models, with two stages for layout generation and fine-tuning.\n\nWEAKNESSES:\nThe paper does not explicitly mention how the model handles the generation of rich-text images when the number of queries is limited. However, it raises concerns about the efficiency of the model when dealing with a large number of queries.\n\nWhen there is a sequential relationship between boxes in the context of generating rich-text images, the paper does not specifically address how the model handles the ordering of boxes.\n\nWhile part-image generation seems reasonable, it could be computationally slow when dealing with a large number of texts as it requires predicting numerous positions.\n\nThe paper does not mention the specific number of texts per individual image in the mentioned databases in the main paper, which could be important information to provide.\n\nRegarding image resolution, the paper suggests that it can be low when dealing with single texts. However, when generating a large number of texts, high-resolution generation becomes necessary. It is essential for the authors to address how they ensure the accuracy of each generated text and the efficiency of text generation in such cases.\n\nUsing sequential generation for layout masks is an interesting point. Have the authors considered directly using diffusion to predict layout masks? It would be beneficial to have more comparative analysis between the two methods.\n\nQUESTIONS:\nSee the weakness\n\nLIMITATIONS:\nThe rationale behind this paper, using seq-2-seq generation, is reasonable. However, there is not a clear explanation of how the generation of rich-text is achieved.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}