TextDiffuser: Diffusion Models as Text Painters
Authors: name: Jingye Chen; name: Yupan Huang; name: Tengchao Lv; name: Lei Cui; name: Qifeng Chen; name: Furu Wei
Published: 2023-05-18 10:16:19

Abstract: Diffusion models have gained increasing attention for their impressive
generation abilities but currently struggle with rendering accurate and
coherent text. To address this issue, we introduce TextDiffuser,
focusing on generating images with visually appealing text that is coherent
with backgrounds. TextDiffuser consists of two stages: first, a Transformer
model generates the layout of keywords extracted from text prompts, and then
diffusion models generate images conditioned on the text prompt and the
generated layout. Additionally, we contribute the first large-scale text images
dataset with OCR annotations, MARIO-10M, containing 10 million
image-text pairs with text recognition, detection, and character-level
segmentation annotations. We further collect the MARIO-Eval benchmark
to serve as a comprehensive tool for evaluating text rendering quality. Through
experiments and user studies, we show that TextDiffuser is flexible and
controllable to create high-quality text images using text prompts alone or
together with text template images, and conduct text inpainting to reconstruct
incomplete images with text. The code, model, and dataset will be available at
https://aka.ms/textdiffuser.