Glyph-ByT5: A Customized Text Encoder for Accurate Visual Text Rendering
Authors: name: Zeyu Liu; name: Weicong Liang; name: Zhanhao Liang; name: Chong Luo; name: Ji Li; name: Gao Huang; name: Yuhui Yuan
Published: 2024-03-14 17:55:33

Abstract: Visual text rendering poses a fundamental challenge for contemporary
text-to-image generation models, with the core problem lying in text encoder
deficiencies. To achieve accurate text rendering, we identify two crucial
requirements for text encoders: character awareness and alignment with glyphs.
Our solution involves crafting a series of customized text encoder, Glyph-ByT5,
by fine-tuning the character-aware ByT5 encoder using a meticulously curated
paired glyph-text dataset. We present an effective method for integrating
Glyph-ByT5 with SDXL, resulting in the creation of the Glyph-SDXL model for
design image generation. This significantly enhances text rendering accuracy,
improving it from less than 20% to nearly 90% on our design image
benchmark. Noteworthy is Glyph-SDXL's newfound ability for text paragraph
rendering, achieving high spelling accuracy for tens to hundreds of characters
with automated multi-line layouts. Finally, through fine-tuning Glyph-SDXL with
a small set of high-quality, photorealistic images featuring visual text, we
showcase a substantial improvement in scene text rendering capabilities in
open-domain real images. These compelling outcomes aim to encourage further
exploration in designing customized text encoders for diverse and challenging
tasks.