Visual Text Generation in the Wild
Authors: name: Yuanzhi Zhu; name: Jiawei Liu; name: Feiyu Gao; name: Wenyu Liu; name: Xinggang Wang; name: Peng Wang; name: Fei Huang; name: Cong Yao; name: Zhibo Yang
Published: 2024-07-19 09:08:20

Abstract: Recently, with the rapid advancements of generative models, the field of
visual text generation has witnessed significant progress. However, it is still
challenging to render high-quality text images in real-world scenarios, as
three critical criteria should be satisfied: (1) Fidelity: the generated text
images should be photo-realistic and the contents are expected to be the same
as specified in the given conditions; (2) Reasonability: the regions and
contents of the generated text should cohere with the scene; (3) Utility: the
generated text images can facilitate related tasks (e.g., text detection and
recognition). Upon investigation, we find that existing methods, either
rendering-based or diffusion-based, can hardly meet all these aspects
simultaneously, limiting their application range. Therefore, we propose in this
paper a visual text generator (termed SceneVTG), which can produce high-quality
text images in the wild. Following a two-stage paradigm, SceneVTG leverages a
Multimodal Large Language Model to recommend reasonable text regions and
contents across multiple scales and levels, which are used by a conditional
diffusion model as conditions to generate text images. Extensive experiments
demonstrate that the proposed SceneVTG significantly outperforms traditional
rendering-based methods and recent diffusion-based methods in terms of fidelity
and reasonability. Besides, the generated images provide superior utility for
tasks involving text detection and text recognition. Code and datasets are
available at AdvancedLiterateMachinery.