AnyText: Multilingual Visual Text Generation and Editing
Authors: Yuxiang Tuo
Wangmeng Xiang
Jun-Yan He
Yifeng Geng
Xuansong Xie
Venue: ICLR 2024 spotlight (2024)
Keywords: diffusion model
text-to-image
text generation
Primary area: generative models

Abstract: Diffusion model based Text-to-Image has achieved impressive achievements recently. Although current technology for synthesizing images is highly advanced and capable of generating images with high fidelity, it is still possible to give the show away when focusing on the text area in the generated image, as synthesized text often contains blurred, unreadable, or incorrect characters, making visual text generation one of the most challenging issues in this field. To address this issue, we introduce AnyText, a diffusion-based multilingual visual text generation and editing model, that focuses on rendering accurate and coherent text in the image. AnyText comprises a diffusion pipeline with two primary elements: an auxiliary latent module and a text embedding module. The former uses inputs like text glyph, position, and masked image to generate latent features for text generation or editing. The latter employs an OCR model for encoding stroke data as embeddings, which blend with image caption embeddings from the tokenizer to generate texts that seamlessly integrate with the background. We employed text-control diffusion loss and text perceptual loss for training to further enhance writing accuracy. AnyText can write characters in multiple languages, to the best of our knowledge, this is the first work to address multilingual visual text generation. It is worth mentioning that AnyText can be plugged into existing diffusion models from the community for rendering or editing text accurately. After conducting extensive evaluation experiments, our method has outperformed all other approaches by a significant margin. Additionally, we contribute the first large-scale multilingual text images dataset, AnyWord-3M, containing 3 million image-text pairs with OCR annotations in multiple languages. Based on AnyWord-3M dataset, we propose AnyText-benchmark for the evaluation of visual text generation accuracy and quality. Our project will be open-sourced soon to improve and promote the development of text generation technology.

Published as a conference paper at ICLR 2024
ANYTEXT: MULTILINGUAL VISUAL TEXT GENERA- TION AND EDITING
Yuxiang Tuo, Wangmeng Xiang, Jun-Yan He, Yifeng Geng,, Xuansong Xie Institute for Intelligent Computing, Alibaba Group (yuxi (yuxiang. tyx, wa namegn..... xwm, leyuan hyy, cangyu.nnnc..mmmm gyf, xi xingtong, xxs) @a 1. ibbaabbbaa-a..a inc. com
ABSTRACT
Diffusion model based Text-to-lmage has achieved impressive achievements re- cently. Although current technology for synthesizing images is highly advanced and capable of generating images with high fidelity, it is still possible to give the show away when focusing the text area in the generated image, syn- thesized text often contains blurred, unreadable, or incorrect characters, making visual text generation one of the most challenging issues in this field. To ad- dress this issue, we introduce Any a diffusion-based multilingual visual text generation and editing model, that focuses on rendering accurate and coherent text in the image. AnyText comprises a diffusion pipeline with two primary el- ements: an auxiliary latent module and a text embedding module. The former uses inputs like text glyph. position, and masked image to generate latent features for text generation or editing. The latter employs an OCR model for encoding stroke data as embeddings, which blend with image caption embeddings from the tokenizer to generate texts that seamlessly integrate with the background. We em- ployed text-control diffusion loss and text perceptual loss for training to further enhance writing accuracy. AnyTeext can write characters in multiple languages, the best of our knowledge, this is the first work to address multilingual vi- sual text generation. It is worth mentioning that AnyText can be plugged into existing diffusion models from the community for rendering or editing text accu- rately. After conducting extensive evaluation experiments, our method has outper- formed all other approaches by a significant margin. Additionally, we contribute the first large-scale multilingual text images dataset, Any Word-3M, containing 3 million image-text pairs with OCR annotations in multiple languages. Based on Any Word-3M. dataset, we propose AnyTexte LnyText-benchmark for the evaluation of visual text generation accuracy and quality, Our project will be open-sourced soon on https: / //g/ttttuu gith.b...... com coom/tyxsppa Tex to improve and pro- mote the development of text generation technology.
INTRODUCTION
Diffusion-based generative models Saharia et al. (2022): ; Ramesh et al. (2022); Rombach et al. (2022:: IZhang & Agrawala 2023) demonstrated exceptional outcomes with unparalleled fidelity, adaptability, and versatility. Open-sourced image generation models (for example, Stable Diffu- sion Rombach et al. (2022).
DewplloydddFDDeelllyyl...o. (2033), along with commercial services (Midjourney Inc. (2022), DALL-E 2 Ramesh et al. (2022). etc.) have made substantial impacts in various sectors incl including photography, digital arts, gaming, advertising, and film production. De- spite substantial progress in image quality of generative diffusion models, most current open open-sourced models and commercial services struggle to produce well-formed, legible, and readable visual text. Consequently, this reduces their overall utility and hampers potential applications.
The subpar performance of present open-source diffusion-basse models can be attributed to num- ber of factors. Firstly, there is a deficiency in large-scale image-text paired data which includes comprehensive annotations for textual content. Existing datasets for large-scale image diffusion model training, like LAION-5B Schuhmann et al. (2022), lack manual annotations or OCR results
"corresponding author

---

Published as a conference paper at ICLR 2024
Text Generation
Text Editigg
Any Text of Stable Difusion
CHEERS
HAPPY BIRTHDAY
HAPPY EVERYDAY
Happy
LovE
tottee GRADUATE
loidaae
pol SAtten Fii CR gul cled) valse
coke
from of Bliieaadd lan "AngTaa of dRiMee hiffaaien
be Holl-
and (fuit, wise "CH pec
dage: nalr paots.
the QREACULATE wilton
PA t t ELRE
スウンテ
SCIENEE
# & 03 A & 9
* 4 + 3/ 8 RI 35 13 *
feerball crp wethhettheeeee wonts Mi
Kuggnn build lin thet
(c-leaed" (indeenl and "60HEHOE
with
R
1 1
cilla vita creytin will the vorda
49
A
31 62
A
Figure 1: Selected samples of AnyText. For text generation, AnyText can render the specified text from the prompt onto the designated position, and generate visually appealing images. As for text editing, AnyText can modify the text content the specified position within the input image while maintaining consistency with the surrounding text style. Translations are provided in parentheses for -onglihlish words in prompt, blue boxes indicate positions for text editing. See more in A.6
Table I: Comparison of AnyText with other competitors based on functionality.
Functionality Multi-line Deformed regions Multi-lingual
Text editing Plug-and-play
GlyphDraw
x
TextDiffuser
X
GlyphControl
X
X
AnyText
for text content. Secondly, as pointed out in Liu al. (2023), the text encoder used open-souree diffusion models, such as the CLIP text encoder, employs vocabulary tokenizer that has no direct access to the characters, resulting in a diminished sensitivity to individual characters. Lastly, most diffusion models' loss functions are designed to enhance the overall image generation quality and lack dedicated supervision for the text region.
To address the aforementioned difficulties, present AnyText framework and Any WM dataset. AnyText consists of a text-control diffusion pipeline with two components: auxiliary latent module encodes auxiliary information such text glyph, position, and masked image into latent space to assist text generation and editing; text embedding module employs an OCR model to encode stroke information as embeddings, which is then fused with image caption embeddings from the tokenizer to render texts blended with the background seamlessly; and finally. a text perceptual loss in image space introduced to further enhance writing accuracy.
Regarding the functionality, there are five differentiating factors that set apart us from other com- petitors outlined Table [I a) Multi-line: AnyText can generate text on multiple lines at user- specified positions. b) Deformed regions: it enables writing in horizontally, vertically, and even curved or irregular regions. c) Multi-lingual: our method can generate text in various languages such as Chinese, English, Japanese, Korean, etc. d) Text editing: which provides the capability for modifying the text content within the provided image in consistent font style. e) Plug-and-play: AnyText can be scamlessly integrated with stable diffusion models and empowering them with the ability to generate text. We present some selected examples in Fig. [ and Appendix A.6
RELATED WORKS
Text-to- Imaa-e--Immeeeea Synthesis. In recent years, significant strides has been made in text-to-image synthe- sis using denoising diffusion probabilistic models Ho et al.
a (2020); Ramesh et al. (2021): [Song et al. Dhariwal & Nichol (2021); Nichol & Dhariwal (2021): Saharia et al. (2022); Ramesh et al.
2021
2022 Rombach al. (2022); Chang et al. (2023). These models have advanced beyond simple
image generation and have led to developments in interactive image editing Meng et al. (2022);
Gal et al. 20033: Brooks al. 2022) and techniques incorporating additional conditions, such as
2

---

Published as a conference paper at ICLR 2024
masks and depth maps Rombach et al. (2022). Research is also exploring the area of multi- controllable synthesis Zhang & Agrawala (2023); Mou et al. (2023): ): Huang et al. (2023). Composit- ing subjects into scenes presents more specihic challenges, and approaches like ELITE Wei et al. (2023), UMM-Diffusion Ma et al. (023b), and InstantBooth [Shi et al. (2023) utilize the features from CLIP image encoder to encode the visual concept into textual word embeddings. Similarly, Dreamldentity Chen et al. (2023c) developed a specially designed image encoder to achieve better performance for the word embedding enhancing scheme.
Text Generation. Progress in image synthesis has been substantial, but integrating legible text into images remains challenging RRombach et al.] (2022); Saharia et al. (2022). Recent research has focused on three key aspects of text generation:
Control Condition. Introducing glyph condition in latent space has been predominant approach in many recent methods. GlyphDraw Ma et al. (2023a) originally used an explicit glyph image as condition, with characters rendered at the center. GlyphControl Yang et al. (2023) further extends it by aligning the text based on its location, which also incorporates font size and text box position in an implicit manner. TextDiffuser Chen et al. (2023b) utilizes a character-level segmentation mask as control condition, and in addition, it introduces masked image to simultaneously learn text generation and text in-painting branches. In our work, we adopt a similar way as GlyphControl to render glyph image but incorporate position and masked image as additional conditions. This design enables AnyText to generate text in curved or irregular regions, and handle text generation and editing simultaneously.
Text Encoder. The text encoder plays a crucial role in generating accurate visual text. Recent meth- ods such as Imagen (Saharia et al. (2022), eDiff-I Balaji al. (2022), and Deepfloyd IF DeepFloyd- Lab 2023) achieve impressive results by leveraging large-scale language models (e.g.. TS-XXL). However, most image generation models still rely on character-blind text encoders, and even character-aware text encoders struggle with non-Latin text generation like Chinese, Japanese, and Korean Liu et al. (2023). To address Chinese rendering. GlyphDraw fine-tunes the text encoder on Chinese images and utilizes the CLIP image encoder for glyph embeddings Ma et al. (2023a). Dif- FUTE replaces the text encoder with a pre-trained image encoder to extract glyphs image editing scenarios Chen et al. (2023a). In AnyText, propose novel approach to transform the text en- coder by integrating semantic and glyph information. This aims to achieve seamless integration of generated text with the background and enable multi- lulti-language text generation.
Perceptual Supervision. OCR-VQGAN Rodriguez et al. (2023) employs a pre-trained OCR de- tection model to extract features from images, and supervise text generation by constraining the differences between multiple intermediate layers. In contrast, TextDiffuser Chen et al. (2023b) uti- lizes a character-level segmentation model to supervise the accuracy of each generated character in the latent space. This approach requires a separately trained segmentation model, and the character classes are also limited. In AnyText, we utilize an OCR recognition model that excels in stroke and spelling to supervise the text generation within the designated text region only. This approach provides a more direct and effective form of supervision for ensuring accurate and high-quality text generation.
3 METHODOLOGY
As depicted in Fig. 2 the AnyText framework comprises text-control diffusion pipeline with two primary components (auxiliary latent module and text embedding module). The overall training objective is defined as:
L = Let + x Lip
(1)
where Lia and Lip are text-control diffusion loss and text perceptual loss, and À is used to adjust the weight ratio between two loss functions. In the following sections, we will introduce the text- control diffusion pipeline, auxiliary latent module, text embedding module, and text perceptual loss in detail.
3.1 TEXT-CONTROL DIFFUSION PIPELINE
In the text-control diffusion pipeline, we generate the latent representation 2n E ghxuxx by apply- & Welling (2014) E on the input image 3n E R R x
ing Variational Autoencoder
3

---

Published as a conference paper at ICLR 2024
Cyph Ronder
Prompt (y): Photo of lush estate with reads "BEVERLY" and "HILLS"
Auxiliary Latent Module
Text Embedding Module
Learnable models
G
Poslion-
Masked Image
Encoder Fg
Tokenizer +
Glyp
Frozen models
Replace
Auxiliary Latent
Uiear
Embedding
BEVERLY
OCR Enooder
HILLS
Text Perceptual
Layyer
Blgh Lines
Text Control Net
Za
+
Perceptual Loss
SOYEERLY
VAE
UNet
VAE
SEVERLY
BEVERLY
Zo
Z
ct Time Stop
X
Diffuios
Lee(t.es)
IncadarT
OCR
Encoddrr
DCT
Text-control Diffusion Pipeline
Text Perceptual Loss Lap
Figure 2: The framework of AnyText, which includes text-control diffusion pipeline, auxiliary latent module, text embedding module, and text perceptual loss.
Here, h x w represents the feature resolution downsampled by a factor of f. and c denotes the latent feature dimension. Then latent diffusion algorithms progressively add noise to the Zo and produce noisy latent image zt, where / represents the time step. Given set of conditions including time step auxiliary feature za E Rhxuxe produced by auxiliary latent module, well as text embedding Cte produced by text embedding module, text-control diffusion algorithm applies a network eg to predict the noise added the noisy latent image 2, with objective:
Lid (oo [lle - en (Z:, Cte
(2)
where Lia is the text-control diffusion loss. More specifically, to control the generation of text, we add 2, with 2, and fed them into a trainable copy of UÑet's encoding layers referred to as TextControlNet, and fed 2, into parameter frozen UNet. This enables TexiControlNet to focus text generation while preserving the base model's ability to generate images without text. More- over, through modular binding, wide range of base models can also generate text. Please refer to Appendix A.I for more details.
3.2 AUXILIARY LATENT MODULE
In AnyText, three types of auxiliary conditions are utilized produce latent feature map za: glyph g. position Ip. and masked image km. Glyph lg is generated by rendering texts using uniform font style (i.e., "Arial Unicode") onto an image based on their locations. Accurately rendering characters in curved or irregular regions is considerably challenging. Therefore, we simplify the process by rendering characters based on the enclosing rectangle of the text position. By incorporating special- ized position lp. we can still generate text in non-rectangular regions, as illustrated in Fig. 3 Position 1p is generated by marking text positions on an image. In the training phase, the text positions are obtained either from OCR detection through manual annotation. In the inference phase, 1p is obtained from the user's input, where they specify the desired regions for text generation. Moreover, the position information allows the text perceptual loss to precisely target the text area. Details garding this will be discussed in Sec. 3.4 The last auxiliary information is masked image (m. which indicates what area in image should be preserved during the diffusion process. In the text-to-image mode, Im is set to be fully masked, whereas in the text editing mode, 4, is set to mask the text regions. During training, the text editing mode ratio is randomly switched with probability of a.
To incorporate the image-based conditions, we use glyph block and position block to downsample glyph l, and position lp- and VAE encoder E downsample the masked image I,. respectively. The glyph block G and position block P both contain several stacked convolutional layers. After transforming these image-based conditions into feature maps that match the spatial size of 2t, we utilize convolutional fusion layer f to merge lg. lp. and lms resulting in generated feature map denoted as 2. which can be represented as:
Z, = f(G()) + P(P)) + e(Im)
where z, shares the same number of channels Zt-
4

---

Published as a conference paper at ICLR 2024
BLARBUCKS COFFEE
20P3
CHAMPION
1
arge
AATACCUUCCC "OFFEOO
"CHAMPPOON"
vadtiing! po
Spazinng
Figure 3: Illustration of generating text in curved or irregular regions, left images are text positions provided by the user.
3.3 TEXT EMBEDDING MODULE
Text encoders excel extracting semantic information from caption, but semantic information of the text to be rendered is negligible. Additionally, most pre-trained text encoders are trained on Latin- based data and can't 't understand other languages well. In AnyText, propose novel approach to address the problem of multilingual text generation. Specifically, we render glyph lines into images, encode glyph information, and replace their embeddings from caption tokens. The text embeddings are not learned character by character, but rather utilize a pre-trained visual model, specifically the recognition model of PP-ÓCRv3 Li et al.) (2022). The replaced embeddings are then fed into transformer-base text encoder as tokens to get fused intermediate representation, which will then be mapped to the intermediate layers of the UNet using cross-attention mechanism. Due to the utilization of image rendering for text instead of relying solely on language-specific text encoders, our approach significantly enhances the generation of multilingual text, as depicted in Fig.
lunderstand English
प
Z0ge king
Say on the Bollaing -OARL 8:18 Cigg builainn "416
Eegiish
Figure 4: Illustration of generating text in multiple languages.
Next, we provide a detailed explanation of the text embedding module. The representation Cte that combines both text glyph and caption semantic information is defined
Cte = T ((()'))(((1)) {(19(c)))
(4)
where y' is the processed input caption y that each text line to be generated (enclosed within double quotation marks) is replaced with a special placeholder S.. Then after tokenization and embedding lookup denoted as o(-), the caption embeddings are obtained. Then, each text line is rendered onto an image, denoted as 6gg Note that eg is generated by only rendering single text line onto an image in the center, while lg in Sec. 3.2 is produced by rendering all text lines onto single image on their locations. The image eg is then fed into an OCR recognition model Y to extract the feature before the last fully connected layer as text embedding, then linear transformation & is applied to ensure its size matches caption embeddings, and replace with embedding of S.. Finally, all token embeddings are encoded using CLIP text encoder Tg.
3.4 TEXT PERCEPTUAL LOSS
We propose text perceptual loss to further improve the accuracy of text generation. Assuming E represents the noise predicted by the denoiser network c9, we can combine the time step and noisy latent image 2, to predict 20 as described in Ho et al. (2020). This can be further used with the VAE decoder to obtain an approximate reconstruction of the original input image, denoted as 2/6- By tran- sitioning from latent space to image space, we can further supervise text generation at pixel-wise level. With the help of the position condition 1p. we can accurately locate the region of generated text. We aim to compare this region with the corresponding area in the original image Zo, and focus solely on the writing correctness of the text itself, excluding factors such as background, deviations in character positions, colors, or font styles. Thus, we employ the PP-OCRv3 model, mentioned
5

---

Published as a conference paper at ICLR 2024
in Sec. 3.3 as the image encoder. By processing zy and z/. position p through operations such as cropping, affine transformation, padding, and normalization, we obtain the images Py and P'g to be used imputs for the OCR model. We utilize the feature maps mp, m' Raxxxe before the fully connected layer to represent the textual writing information in the original and predicted image at position p. respectively. The text perceptual loss is expressed as
Σ sp(() hw
Ltp =
Σ h,w
mm
m'p 2l2
(5)
By imposing a Mean Squared Error (MSE) penalty, we attempt to minimize the discrepancies be- tween the predicted and original image in all text regions. As time step is related the text quality in predicted image v we need to design a weight adjustment function (p()). It has been found that
setting (() Ct is a good choice, where dg the coefficient of diffusion process introduced in Ho et al. 2020).
4 DATASET AND BENCHMARK
Currently, there is a lack of publicly available datasets specifically tailored for text generation tasks, especially those involving non-Latin languages. Therefore, we propose Any Word-3M, a large-scale multilingual dataset from publicly available images. The sources of these images include Noah- Wukong Gu et al. 2022). LAION-400M Schuhmann et al. (2021), as well as datasets used for OCR recognition tasks such as ArT Coxo- (RCTWILS LSVT MLT MTWT ReCTS These images cover a diverse range of scenes that contain text, including street views, book covers, advertisements, posters, movie frames, etc. Except for the OCR datasets, where the annotated information used di- rectly, all other images are processed using the PP-OCRv3 Li ct al. (202)) detection and recognition models. Then, captions are regenerated using BLIP-2 Li et al. (023). Please refer to Appendix AA.3 for more details about dataset preparation.
Through strict filtering rules and meticulous post-processing, we obtained a total of 3,034,486 im- ages, with over 9 million lines of text and more than 20 million characters or Latin words. We randomly extracted 1000 images from both Wukong and LAION subsets to create the evaluation set called AnyText--eenhhaa
These two evaluation sets are specifically used to evaluate the accuracy and quality of Chinese and English generation. respectively. The remaining images are used as the training set called Any AnyWord-3M, among which approximately 1.6 million are in Chinese, 1.39 mil- lion are in English, and 10k images in other languages, including Japanese, Korean, Arabic, Bangla, and Hindi. For a detailed statistical analysis and randomly selected example images, please refer to Appendix A.4
For the AnyTexx-benchmark, utilized three evaluation metrics assess the accuracy and quality of text generation. Firstly, we employed the Sentence Accuracy (Sen. Ace) metric, where each generated text line was cropped according the specified position and fed into the OCR model to obtain predicted results. Only when the predicted text completely matched the ground truth was it considered correct. Additionally, we employed another less stringent metric, the Normalized Edit Distance (NED) to measure the similarity between two strings. As we used PP-OCRv3 for feature extraction during training, to ensure fairness, we chose another open-source model called DuGuangOCR ModelScope 2023) for evaluation, which also performs excellently in both Chinese and English recognition. However, relying solely on OCR cannot fully capture the image quality. Therefore, introduced the Frechet Inception Distance (FID) assess the distribution discrepancy between generated images and real-world images.
5 EXPERIMENTS
5.1 IMPLEMENTATION DETAILS
Our training framework is implemented based on ControlNe[] and the model's weights are initial- ized from SDI.$ In comparison to ControlNet, we only increased the parameter size by 0.34% and the inference time by 1.04%, refer to A.2 for more details. Our model was trained on the Any Word-3M dataset for 10 epochs using Tesla A100 GPUs. We employed a progressive fine- tuning strategy, where the editing branch was turned off for the first epochs, then activated with a
https//gthhb..om///llssscCCCooo http://wnggniniaaccccoommmmmmmmp

---

Published as a conference paper at ICLR 2024
Table 2:
Quantitative comparison of AnyText and competing methods. fis trained on LAION-Glyph-IO0 10M,
and ğis fine-tuned on TexiCaps-5k. All competing methods evaluated using their officially released models.
English
Chinese
Methods
Sen. ACC1
NED1
FIDL
Sen. ACC1
NED1
FID
ControlNet
0,3837
0.8015
45,41
0.3620
0.6227
41.86
TextDiffuser
0,5921
0.7951
41.31
0.0605
0.1262
53.37
GlyphControlf
0,3710
0.6680
37.84
0.0327
0.0845
34.36
GlyphControl
0.5262
0.7529
43.10
0.0454
0.1017
49.51
AnyText-v1.0
0,6588
0.8568
35,87
0,6634
0.8264
28.46
AnyText-v1.1 0.7239
0.8760
33.54
0.6923
0.8396
31.58
probability of a 0.5 for the next 3 epochs. In the last 2 epochs, we enabled the perceptual loss with weight coefficient of X = 0.01. Image dimensions of lg and lp are set to be 1024x1024 and 512x512, while Cg. Py. and p'g are all set be 80x512. We use Adam W optimizer with a learning rate of 2e-5 and a batch size of 48. During sampling process, based the statistical information from A.4 a maximum of 5 text lines from each image and 20 characters from each text line were chosen render onto the image, this setting can cover the majority of cases in the dataset.
Recently, we found that the quality of OCR annotations in the training data has a significant impact on the text generation metrics. We made minor improvements to the method and dataset. then fine- tuned the original model, resulting in the Any Tcxt-v1.1, which achieved significant performance improvements. Details of the updated model are provided in Appendix A.7 It should be noted that all example images used in this article are still generated by the v1.0 model.
5.2
COMPARISON RESULTS
5.2.1 QUANTITATIVE RESULTS
We evaluated existing competing methods, including ControlNet Zhang & Agrawala (2023), TextD- iffuser Chen al, (2023b), and GlyphControl Yang et al. (2023), using the benchmark and metrics mentioned in Sec. To ensure fair evaluation, all methods employed the DDIM sampler with 20 steps of sampling, a CFG-scale of 9, fixed random seed 100, a batch size of 4, and the same positive and negative prompt words. The quantitative comparison results can be found in Table Additionally, we provide some generated images from Any TerTTexxbbeehhaae in Appendix A.3
From the results, we can observe that AnyText outperforms competing methods in both Chinese and English text generation by a large margin, in terms of OCR accuracy (Sen.ACC, NED) and realism (FID). It is worth mentioning that training set only consists of 1.39 million English data, while TextDiffuser and GlyphControl are trained on 10 million pure English data. An interesting phenomenon is that ControlNet(with canny control) tends to randomly generate pseudo-text in the background, evaluation methods as in Chen el al. (2023b) that utilize OCR detection and recognition models may yield low evaluation scores. However, our metric, we focus only on the specified text generation areas, and we find that ControlNet performs well in these areas. Nevertheless. the style of the generated text from ControlNet appears rigid and monotonous, as if was pasted onto the background, resulting in a poor FID score. Regarding Chinese text generation, both TextDiffuser and GlyphControl can only generate some Latin characters, punctuation marks or numbers within a Chinese text. In AnyText, our Chinese text generation accuracy surpasses all other methods. In the stringent evaluation metric of Sen. ACC, we achieve an accuracy of over 66%. Additionally, TnyText obtains the lowest FID score, indicating superior realism in the generated text.
5.2.2 QUALITATIVE RESULTS
Regarding the generation of the English text, we compared our model with state-of-the-ee models or APIs in the field of text-to-image generation, such as SD-XL1.0, 0.0 Bing Image Creatof DALL-E2, and DeepFloyd IF, as shown in Fig. 5 These models have shown significant improvements in text generation compared to previous works However, there is still considerable gap between them and professional text generation models. Regarding the generation of Chinese text, the complexity of strokes and the vast number of character categories pose significant challenges. GlyphDraw Ma al. 2023a) is the first method that addresses this task. Due to the lack of open-source models or APIs, we were unable to conduct quantitative evaluation and instead relied on qualitative comparisons using examples from GlyphDraw paper. Meanwhile, since ControlNet can also generate Chinese text effectively. we included it as well. As shown in Fig. 6 AnyText shows superior integration of
hhtps:/wwww.hngg.com/rrrtr
7

---

Published as a conference paper at ICLR 2024
generated text with background. such as text carved into stone, reflections signboards with words, chalk-style text on a blackboard, and slightly distorted text influenced by clothing folds.
SD-XL10
Bing
DALLEE2
DeepRoyd
Original
AnyText
HURY
OF JAN HYEL
FURY WPEA
FURY DENGE
MV MAL oF MA MAN
rhal mein the ukulle
rral meet ukulele
UNGULLE
Awys Al Owe!
owl alwbtse Joun
owd Always lavan
ONL ARLAbs
SHINKY ARTS
SHINKY AWIS
SMRINK ARTS
BARR ARTS
ARRTYS es SNIRIKS
SAARAA SYRRA OI SIIN SARSS
BIRTHDAY
PALID'S KIRTHTES KARY
Birthdays Party Partty
BIRITHDAY PARTEISY FOR KIDS
Birthday Parties for Kids
Birthday
Parties for Kids
Figure 5: Qualitative comparison of Any'exx and state-of-the-art models or APIs in English text generation. All captions are selected from the English evaluation dataset in AnyText-benchmar.
Thore is atond tabtet on the seashive with the Gocoliiiin st.E (beidbi River)
# M
t Si TFC
スラ
saaking +6 #t wont slind but?
riceaon stesdda food of agg (
caaving!?
A T-akirt teat saya "A 4 Dis Vlactt)
GlyphDraw
ControlNet
AnyText
Figure 6: Comparative examples between GlyphDraw, ControlNet, and AnyText in Chinese text generation, all taken from the original GlyphDraw paper.
5.3 ABLATION STUDY
In this part, we extracted 200k images (with 160k in Chinese) from AnyWord-3M as the training set, and used the Chinese evaluation dataset in wwwTTex--beehhmrr to validate the effectiveness of each submodule in AnyText. Each model was trained for 15 epochs, requiring 60 hours 8 Tesla V100 GPUs The training parameters were kept consistent with those mentioned in Sec. 5. By analyzing the experimental results in Table 3 we draw the following conclusions:
8

---

ocr Published as a conference paper at ICLR 2024
Table 3: Ablation experiments of AnyText small-scale dataset from Any AnWWord33M. The results validate the effectiveness each submodule in Any Text.
Chinese
Exp. Ne
Editing
Position
Text Embedding
Perceptual Loss
Sen. ACC1
NED1
0.1552
0.4070
X
X vit conv
X
0,2024
0.4649
3
X
0,1416
0.3809
4
X
0.1864
0.4402
X
ocr X
0.4595
0.7072
X
ocr
0,4472
0.6974
X
0.003
0,4848
0.7353
8
X
Ocr
0.01
0,4996
0.7457
0.03
0.4659
0.7286
Editing: Comparing Exp.1 and Exp.2 we observed slight decrease when editing branch is enabled. This is because text generation and editing are two different tasks, and enabling editing branch increases the difficulty model convergence. To focus on analyzing the text generation part, we disabled editing in all subsequent experiments by setting the probability parameter O.
Text Embedding: Exp.2 ~ validate the effectiveness the text embedding module. Exp.2. cap- tions encoded directly by the CLIP text encoder. Exp.3 employs the CLIP vision encoder(vil) as the feature extractor, but yielded unsatisfactory results, likely due to images with only rendered text falling into the Dusttf-Distributtion data category for the pre-trained CLIP vision model, hinder- ing its text stroke encoding capabilities. Exp.4 experimented with trainable module composed of stacked convolutions and an average pooling layer(conv). which struggled without tailored super- vision. In Exp.5, utilizing the pre-trained OCR model(PP-OCRv3) resulted in a significant 25.7% increase in Sen. Ace metric compared Exp.2.
Comparing Exp.5 and Exp.6, although the rendered text in lg contains positional infor- mation implicitly, the inclusion of more accurate position Lp further improves performance and enables the model to generate text in irregular regions.
Position:
Text Perceptual Loss: Exp.7 validate the effectiveness text perceptual loss. Upon conducting three experiments, we found that À 0.01 yielded the best results, with 4.0% improvement com- pared to Exp.5 Scn. Acc metric. It is worth noting that perceptual loss necessitates transitioning from latent space image space, which slows down the training process. Therefore, for Exp.7 ~ 9, we started training from epoch 13 Exp.5 and trained only for the remaining epochs. Despite this, perceptual loss demonstrated significant improvements.
CONCLUSION AND LIMITATIONS
In this paper, we delve into the extensively researched problem of text generation in the field of text- to-image synthesis. To address this challenge, we propose novel approach called AnyTTex, which a diffusion-
muulti-lingure text generation and editing framework. Our approach incorporates an auxiliary latent module that combines text glyph, position, and masked image into a latent space. Furthermore, in the text embedding module, we leverage an OCR model to extract glyph information and merge with the semantic details of the image caption, thereby enhancing the consistency between the text and the background. To improve writing accuracy, we employ text-control diffusion loss and text perceptual loss during training. In terms of training data, we present the Any Word-3M dataset, comprising 3 million text-image pairs in multiple languages with OCR annotations. To demonstrate the superior performance of AnyText, we conduct comprehensive experiments on the proposed Any Texx-bencchaar showcasing its superiority over existing methods. Moving forward, our future work will focus exploring the generation of extremely small fonts and investigating text generation with controllable attributes.
7 REPRODUCIBILITY STATEMENT
To ensure reproducibility, we have made the following efforts: (1) We will release our code and dataset. (2) We provide implementation details in Sec. 5.1 including the training process and lection syper--arammters. (3) We provide details on dataset preparation and evaluation metric in Sec 4 and Appendix A.3 and the code and data will be made available along with it.

---

Published as a conference paper at ICLR 2024
REFERENCES
ArT. lcdar2019 robust reading challenge on arbitrary-shaped text. http::/ree.vv..uhhes///h11111 2019.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Mïika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Yu Liu. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint, 2022.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint, abs/221 1.09800, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Kr- ishnan. Muse: Text-to-image generation via masked generative transformers. arXiv preprint, abs/2301.000 .00704, 2023.
Haoxing Chen, Zhuoer Xu, Zhangxuan Gu, Jun Lan, Xing Zheng, Yaohui Li, Changhua Meng, Huijia Zhu, and Weiqiang Wang. Diffute: Universal text editing diffusion model. arXiv preprint, abs/2305.10825, 2023a.
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: Diffusion models text painters. arXiv preprint, abs/2305. 10855, 2023b.
Zhuowei Chen, Shancheng Fang, Wei Liu, Qian He, Mengqi Huang, Yongdong Zhang, and Zhen- dong Mao. Dreamidentity: Improved editability for efficient face-identity preserved image gen- cration. arXiv preprint, fbs/2307.00300) 2023c.
COCO-Text A large-scale scene text dataset based on mscoco. https//hgghhinngiithuiioccoooe.. 2016.
DeepFloyd-Lab. Deepfloyd if. https:/gittubb.cm/ddde-fffoo/o/d 2023.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans image synthesis. In NeurlPS, pp. 8780-8794, 2021.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Minzhe Niu, Hang Xu, Xiaodan Liang, Wei Zhang, Xin Jiang, and Chunjing Xu. Wukong: 100 million large-scale chinese cross-modal pre-training dataset and A foundation framework. CoRR, abs/2202.06767, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint, abs/2302.09778. 2023.
Midjoumny Inc. Midjourney. https//wwww.midouueey.comm/ 2022.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Chenxia Li, Weiwei Liu, Ruoyu Guo, Xiaoting Yin, Kaitao Jiang, Yongkun Du, Yuning Du, Lingfeng Zhu, Baohua Lai, Xiaoguang Hu, Dianhai Yu, and Yanjun Ma. Pp-ocrv3: More at- tempts for the improvement of ultra lightweight OCR system. CoRR, abs/2206.03001, 2022.
Junnan Li. Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint, abs/2301.125977 2023.
10

---

Published as a conference paper at ICLR 2024
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi. and Noah Constant. Character-aware models improve visual text rendering. In ACL, pp. 16270-16297, 2023.
LSVT. Icdar2019 robust reading challenge on large-scale street view text with partial labeling. https//tre.ovcc.ua.ee//hh
2019.
Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang, Di Niu, Haonan Lu, and Xiaodong Lin. Glyph- draw: Learning to draw chinese characters image synthesis models coherently. arXiv preprint, abs/2303. 17870, 2023a.
Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation. arXiv preprint, abs/2303.09319, 2023b.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
MLT. Icdar 2019 robust reading challenge multi-lingual scene text detection and recognition. 2019.
ModelScope. Duguangocr. corvnextiiny_oez- wwwp:oooeeeooooioooo-ooeeeeeeeee http://eiioaeeeeeaaaaaaaaaaaaaaa waw.//uumarrry..a..m.mm 2023.
http:://mootllccccc..cc/////ocee /amte//////ao/////oooo/////// /v
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint, abs/2302.088533, 2023.
MTWL. Iepr 2018 challenge on multi-type web images. 2018.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, volume 139, pp. 8162 8171, 2021.
PaddlePaddle. Pp-ocrv4. https://gitthbb release/2. /oc/ doc /t// int raductiion. md. 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In ICML,, volume 139, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint, abs/2204.06125, 2022.
RCTW. lcdar2017 competition on reading chinese text in the wild. http://tctw..liaabneeddaaaa 2017.
ReCTS. Icdar 2019 robust reading challenge on reading chinese on signboard. https://re.cv..uaa.ss/ccc.- 2019.
Juan A. Rodriguez, David Vázquez, Issam H. Laradji, Marco Pedersoli, and Pau Rodríguez. OCR- VQGAN: taming text-within-image generation. In WACV, pp. 3678-3687, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/2111 112111.000 11.2114,, 2021.
11

---

Published as a conference paper at ICLR 2024
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jit- sev. Laion-5b: An open large-scale dataset for training next generation image-text models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), NeurIPS, vol- ume 35, pp. 25278-25222 25294, Curran Associates, Inc., 2022.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instanthooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint, abs/2300.00311,, 2023.
Yang Song. Jascha Sehll-Dicsstei, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. ELITE: en- coding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint, abs/2302.13848, 2023.
Yukang Yang. Dongnan Gui, Yuhui Yuan, Haisong Ding, Han Hu, and Kai Chen. Glyphcontrol: Glyph conditional control for visual text generation. arXiv preprint, abs/2305. abs/2305.18259, 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. arXiv preprint, abs/2302.05543, 2023.
A
APPENDIX
A.1 MORE EXAMPLES ON THE FLEXIBILITY OF ANYTEXT
By adopting the architecture of ControlNet, AnyText exhibits great flexibility. On one hand, the base model's conventional text-to-image generation capability is preserved as shown in Fig. 7 On the other hand, the open-source community, such as HuggingFace and CivitAI, has developed wide range of base models with diverse styles. By modularly binding AnyText, depicted in Fig. S these models(Oil Painting Guoha Diffusion Product Design and Moon Film") can acquire the ability generate text. All of these aspects highlight the highly versatile use cases of AnyText.
Schroenur -ifling o Farid
somgl lortl bourd maide Eel suadi.
volidy en Stpeeceed Separt
slesing jelyfas Rusting fooy forest at twiligee
rollacis Dight
Figure Examples of AnyText generating images without text.
r
POLICE
Wert of Fatuilata cllvy speit cit. product moder diih4, TEGLA
watvinng ppliic woffem and coy, with TOOLDDR co
paintiing of yoing with boani and Cygeeng ang. welttee
wine tradiiiions Delinest geiebing
and aeffriitrii sacle Bingree, with
mastoppest conneatia Ligttigg sligetiy
Figure 8: Examples of community models integrated with AnyText that can generate text.
http:///cvvviaaicoooomeeee////// hhanss//wiinn..coo/mmdee/22223
hbtps//deviaaa.cco/mmoee///////
12

---

Published as a conference paper at ICLR 2024
Table 4: Comparison the parameter sizes modules between ControlNet and AnyText.
Modules
ControlNet
AnyText
UNet
859M
859M
VAE
83.7M
83.7M
CLIP Text Encoder
123M
123M
NontrolNet
361M
TexiControINet
360M
Glyph Block
0.35M
Position Block
0.04M
Fuse Layer
0.93M
OCR Model
2.6M
Lincar Layer
2.0M
Total
1426.7M
1431.6M
A.2 PARAMETER SIZE AND COMPUTATIONAL OVERHEAD OF ANYTEXT
Our framework is implemented based on ControlNet. Despite the addition of some modules, did significantly increase the parameter size, as refered Table We compared the computational overhead of both models using batch size of 4 on single Tesla V100, the inference time for ControlNet is 3476 ms/image, and for AnyText is 3512 ms/image.
A.3 MORE DETAILS ABOUT DATASET PREPARATION
Filtering Rules: We have established strict filtering rules ensure the quality of training data. Taking the Chinese dataset Wukong Gu et al. (2022) as an example, each original image undergoes the following filtering rules:
Width or height of the image should be no less than 256.
Aspect ratio of the image should be between 0.67 and 1.5.
Area of the text region should not be less than 10% of the entire image.
Number of text lines in the image should not exceed 8.
Next, for the remaining images, each text line is filtered based on the following rules:
Height of the text should not be less than 30 pixels.
Score of OCR recognition of the text should be lower than 0.7.
Content of the text should not be empty consist solely of whitespace.
We implemented similar filtering rules on the LAION-400M [Schuhmann et al. (2021) dataset, al- though the rules were more stringent due to the abundance of English data compared to Chinese. By implementing these rigorous filtering rules, we aim to ensure that the training data consists of high-quality images and accurately recognized text lines.
Image Captions: We regenerated caption for each image using BLIP-2 2 (2023) and removed specific placcholders S, (such as **). Then, we randomly selected one the following statements and concatenated it to the caption:
content and position of the texts are
textual material depicted in the image are
texts that say"
captions shown in the snapshot are with the words of
that reads "
the written materials on the picture:
these texts written it:
13

---

Published as a conference paper at ICLR 2024
Table 5: Statistics of dataset size and line count in subsets of Any Word-3M.
Subsets
image count
image w/o text
line count
mean lines/img
#img <=5 lines
Wukong
1.54M
0
3.23M
2.10
I.5IM, 98.1%
LAION
1.39M
0
5.75M
4.13
1.03M, 74.0%
OCR datasets
0.1M
21.7K
203.5K
2.03
93.7K. 93.6%
Total
3.03M
21.7K
9.18M
3.03
2.64M, 86.9%
Table Statistics of characters or words for different languages Any Word-3M.
Lanuages
line count
chars/words count unique chars/words mean chars/line
#linc <=20 chars
Chinese
2.90M
15.09M
5.9K
5.20
2.89M, 99.5%
English
6.27M
6.35M
695.2K
5.42
6.25M, 99.7%
Others
11.7K
59.5K
2.1K
5,06
11.7K, 100%
Total
9.18M
21.50M
703.3K
5.35
9.15M, 99.6%
captions are"
content of the text in the graphic is
Next, based on the number of text lines, the placeholders will be concatenated at the end to form the final textual description, such as: "a button with an orange and white design these texts are written on it: + #" After processing through the text embedding module, the embeddings corresponding to the placeholders will be replaced with embeddings of the text's glyph information.
A.4 STASTIC AND EXAMPLES OF ANY
In Table 5 and Table 6 provide detailed statistics the composition of Any Word-3M dataset. Additionally, in Fig. 9
we present some example images from the dataset.
THELAZYS
HAILON
HEEE PEDER
JADAN FO AADIAE
RUAL TROY
SRR
HE st 49 10
NG
HN TA
WFG
GOOD NIGHT
REVIEW
Kw3 0 2
LIGHT
N2 ?
Darty Karabka
sizzid kaloni
10000 MOVE
CULYBRE
DW
Tha hispee
CRNTS BERIN
MATTAR
Hin Des S
$12
2 50
LAION
Wukong
OCR Datasets
Figure 9: Examples of images from the Wny-Wrd-3M dataset.
A.5 EXAMPLES FROM ANYTEXT-BENCIMARK
We selected some images from the generated image of AnyText-benchmark evaluation set. The English and Chinese examples can be seen in Fig. 10 and Fig. II respectively. All the example images were generated using the same fixed random seed, as well as the same positive prompt ( ("best quality, extremely detailed") and negative prompt ("longbody, lowres. bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, watermark"),
14

---

Published as a conference paper at ICLR 2024
ControlNet
TextDiffuser
GlyphControlt
GlyphControl2
Original
AnyText
FURY WORAI NOR
FURY
BEIL WEPPON BEVTEVIS
FURY WOMAN COPENT
AUEANE HEM OPFEND
WOMAN BFROOWS
neel playee ukulele
Read play slbe ukuleté
real men Blay the vkcatele
real play the ukulele
play
play the ukule
ukulele
owl always
owl always luraa
owl abwys love you.
owl always luve you..
always lavo
foow
SHRINKY SHARTS
SHRINY ARTS
SHRIK YY ART
SELLR ART
SHRINKS ARBS
SHEINNY ARTS
Birthday
Biribday
Sirzdda
Birthday Parties for Kids
Birthday Parties for Kids
Parties
Parties
Porlie Kids
Kids
Kide
Figure 10: Examples images in English from the Any Aextext-Ba SeryTexx-eeehhane benchmark. tis trained on LAION- Glyph-10M, and ii fine-tuned TexiCaps-5k.
ControlNet
TextDiffuser
GlyphControlt
GlyphControlz+
Original
AnyText
# E It
BE 18 C
X ft B
1FC
TiVK
JE IF
+ H
i CHENGUI OI EDUCATION
CHEN GUT EOUCATION
1 WF MMBE
18 03
BE eo
FR 2A
JR J0
1 mano
A
19
3 XX in D 1
12 M
G bu #
+
MANZERR
55 1086
1085
1086
1088
1086
1086
FPER
Figure 11: Examples of images in Chinese from the MnyText--benhmarte Vi trained on LAION- G0yph-10M, and şis fine-tuned on TextCaps-5k.
15

---

Published as a conference paper at ICLR 2024
Table 7: Changes in the data scale of Any AnyWord-3M.
Version
Wukong
LATON
Total
wm s0.re<00.5
v1.0
1.54M
1.39M
3.03M
v1.1
1.71M
1.72M
3.53M
2.95M
A.6 MORE EXAMPLES OF ANYTEXT
We present additional examples the text generation (see Fig. 12 and text editing (see Fig.
Any Text
Any Text
Any Text
Any Text
beautifullly crafted olcorattvee pillow with time words "AnyText"
photo of caravee! masskiato ceffee. top dawn perspeotive, with "Amy"" "Text" written using chosolate
sweeter with lomitted & "AnyText"
luxurious womeen's Leother henabbag, ougraved with the goldten wordss TAnyText"
Text".
ancient chiness copper coler, texts are "A 2 "g (artificial intelligence)
colorful graffiti ow the building with the wordss "Og * & 3J "ddeep learning)
futuristio looking robot, with the words * t "1""""eee "emmkodieo At) ow his body
ov bustiing during night, intow lights spell out *ep
ft. @t # "(neural netwwork))
Super Dog
*
by Alice
Michea Jackson
09-25
wãoe drawingg by orayons, a elg wearing supeereer oppe soars sky. with tine words "superpeg" bby Atios' and 29-255
náse drowing pensil of Mishael Mishael-jackson the words "Micheal" and
drawing by a shild with erayons, castle maddl of lollipops, marchmellows, and canaties, "# NK 2. (candy casele)"
wice Chinase calligraphy work havggng in the middle of the living room, wick words & & (self discipline and social commnitinennt)
Jacksoon'
Figure 12: More examples of AnyText in text generation.
A.7 IMPLEMENTATION DETAILS OF ANYTEXT-VI.1
We randomly sampled some text lines in Any Word-3M and observed that the OCR annotation error rate for the English portion was 2.4%, while for the Chinese portion was significantly higher at 1 1188% So, for the Chinese part, we used the latest PP-OCRv4 PaddleeaddPPPddde (2023) to regenerate the OCR annotations; and for the English part, we found that the annotations from MARIO-LAION by TextDiffuser [Chnn et al. (2023b) performed slightly better on some severely deformed English sentences compared to PP- PP-OCRv4, so replaced them with their annotations. Additionally, slightly increased some data and making the ratio of the English and Chinese was approximately 1:1. This updated dataset was labeled as vl.1, shown in Table 7 In addition, added "wm.score" label for each image to indicate the probability of containing a watermark, which is used to filter images with watermarks in the final stage of model training.
Furthermore, we proposed a method called "inv mask". For each text line in the image, its recog- nition score is too low, the text is too small, if it is not among the 5 randomly chosen text lines,
16

---

Published as a conference paper at ICLR 2024
EDD
THE COFFEE
aaste
I BRITISH BOYS
I BRITISH GIRLS
WHY SO SERIOUS?
WHY SQ NERVOUS?
WORRY TRUIRSES HAPPY!
WORRY IEMER HAPP!!
tt
SEC
A
T THAA o
TBC
>
SERT?!
Forgive
Forgive
Punish
Punish
PHOENIX
PHGENIX
Figure 13: More examples of Any Text in text editing.
Table Improvement watermark and pseudo AnyText, tested on private model and dataset.
English
Chinese
Version
watermark
aseudo-text
watermark
pseudo-text
v1.0
6.9%
119.0%
24.7%
166.8%
v1.1
0.4%
110.9%
2.9%
124.4%
will be marked as invalid. Subsequently, these invalid text lines are combined to form a mask, and during training, the loss in the corresponding area will bring significant improvement in OCR metrics, as illustrated in Table 2 and also notably reduced
O. This straightforward approach can
the ratio of pseudo-text areas in the background, as depicted in Table
The v1.0 model underwent further fine-tuning on the Any Word-3M v1.1 dataset for epochs, with the last epochs using data of wmascore 0.5, filtering out approximately 25% Chinese data and 8% English data. The parameter X for perceptual loss was set to 0.003, as it yielded favorable results for both Chinese and English on the v1.1 dataset. Subsequent
the training process, the original SD1.5 base model was replaced with community model Realistic. Vision which makes generated images more aesthetically appealing. The final model was labeled as version I
hhtn:://wuuggnnifsccccs//iclilef Vision V5.0 VAE
17


Reviews:
[{"review_id": "YFDe9WHpKA", "reviewer": "Reviewer_TuoG", "summary": "This paper presents an adapter-based module that can be plugged into existing diffusion models to perform multilingual visual text generation and editing. It contains a control net to control the text location and text content and a special text embedding module to improve the multilingual text generation ability. This paper also presents a large-scale multilingual text image dataset, AnyWord-3M, with image-text pairs and OCR annotations in multiple languages.", "questions": "I have some questions about the model architecture, dataset construction, and experiment design.\n\nFor the model architecture:\n1. Which feature is extracted from PP-OCR to serve as text embedding?\n2. Is the linear projection layer for the OCR encoder trained?\n3. What is the configuration for the fuse layer? A single convolution layer or a stacked convotion? \n4. What is the input to text controlnet? Based on Figure 2, it seems to be the concatenation of $z_a$ and $z_t$.\n5. Why is the image resolution for glyph image 1024x1024 instead of 512x512? This does not align with the final image resolution.\n\nFor the dataset construction:\n1. How is the tight position mask $l_p$ annotated? As far as I know, the PP-OCR model does not support arbitrary shape text detection.\n2. The glyph image $l_g$ contains rotated text; how is the bounding box annotated from the position mask?\n3. The captions are generated using BLIP-2 model instead of the original captions. Could authors provide some statistics like the length of the captions and show some example captions for images in AnyWord-3M? How does this difference affect the model performance?\n\nFor the experiment:\n1. Could authors provide more information about the input for the visual examples? For example, the layouts for Figures 1, 6, 8, 12, the captions for Figures 5, 10, 11.\n2. The Sen. ACC and NED metric is measured using the same OCR model as the encoder in model training, which might be unsuitable. Could authors evaluate the OCR performance using another OCR model?\n3. Table 3 shows the improvement brought by visual text embedding in Chinese text generation. I wonder if this also improves the English word generation.\n4. In the experiment, the text diffuser model is not trained on the same dataset as GlyphControl and AnyText Model. Is it possible that authors fine-tune them on the same data and compare final performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed adapter-based module is a plug-and-play module that can guide visual text generation of many existing pre-trained diffusion models and can apply to multiple languages, which was not achieved in previous works.\n2. The proposed text adapter and text encoder are proven to be effective in improving the OCR performance.\n3. The proposed AnyWord-3M dataset is the first large-scale text image dataset with multilingual OCR annotations and is useful for future study.", "weaknesses": "1. The method and dataset collection miss a lot of details. For example, is the linear projection layer trained or fixed for ocr encoder in the text embedding module? \n2. A lot of information is not presented in the examples shown in the paper, for example, the layouts for images in Figure 1, and the captions for images in Figure 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xh2KJUPdjy", "reviewer": "Reviewer_xCSM", "summary": "The manuscript unfolds AnyText, a profound diffusion-based multilingual visual text generation and editing model. It meticulously tackles the intricacies involved in precise text portrayal within generated images, deploying auxiliary latent modules and text embedding modules as strategic tools. To augment the training phase, the introduction of text-control diffusion loss and text perceptual loss is articulated, which serves to bolster text generation quality. A formidable performer, AnyText triumphs over existing paradigms, championing improved accuracy and quality in text generation. Furthermore, the introduction of a novel dataset, AnyWord-3M, enriches the existing reservoir of multilingual image-text pairs, reflecting a thoughtful contribution to the scholarly community.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(1) A notable innovation lies in the paper's strategic approach to circumvent challenges, ensuring precise text portrayal within generated images.\n\n(2) The infusion of auxiliary latent modules coupled with text embedding modules acts as a catalyst, promoting enhanced accuracy and coherence in the text generation process.\n\n(3) Strategic incorporation of text-control diffusion loss and text perceptual loss during training heralds improvement in the overall text generation quality.\n\n(4) A commendable addition is the introduction of AnyWord-3M, a robust dataset enriched with 3 million image-text pairs, elaborately annotated with OCR in multiple languages, signifying a valuable asset to the research fraternity.", "weaknesses": "(1) The architecture seems somewhat reliant on pre-established technologies such as Latent/Stable Diffusion and ControlNet, which slightly shadows its novelty.\n\n(2) Encumbered by a complex array of components and a multi-stage training regimen, the model’s re-implementation emerges as a challenging task, compounded further by numerous critical hyperparameters requiring manual assignment.\n\n(3) Certain aspects, such as token replacement, require a more elaborate discourse for clearer comprehension, primarily concerning the identification of corresponding tokens and their subsequent utility in text-image generation.\n\n(4) There exists a potential ambiguity concerning the intermediate generative results (x'_0), where the possible presence of noise or blur could compromise the precision of perceptual loss computation.\n\n(5) A clearer depiction of computational resource demands (GPU Hours), beyond the ambiguity of epochs, would enhance the paper’s practicability and replicability.\n\n(6) A more explicit elucidation on the operational synergy between Z_a from the Auxiliary Latent Module and Z_0 from VAE, as depicted in Fig. 2, alongside their application within Text-ControlNet, would augment the manuscript's clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1TX4efgb0f", "reviewer": "Reviewer_5GTp", "summary": "This paper introduces modules to enhance the text-drawing capabilities of text-to-image diffusion models. The auxiliary latent module embeds glyph and position information obtained from an off-the-shelf OCR module and fuses these latents as diffusion-step invariant conditions through ControlNet. Additionally, the text embedding module encodes a tokenized prompt, replacing the tokens of rendered glyphs with special tokens. Since these special tokens, the components of the auxiliary latent module, and ControlNet are the only trainable parts in the entire model, this method can be readily applied to existing diffusion models without retraining the diffusion UNet. The modules are trained on the AnyWord-3M dataset, also proposed in this paper. The performance of the proposed method surpasses that of previous text-generation-focused text-to-image diffusion models and also offers multilingual text generation capabilities.", "questions": "I'd like to pose questions that can address the weaknesses discussed.\n\n1. Could you elaborate on how the PP-OCRv3's performance specifically influences the results?\n2. Have you considered measuring the computational overhead when additional modules are integrated into the vanilla text-to-image diffusion model?\n3. Why were Korean and Japanese languages included in the qualitative results but not in the quantitative ones?\n4. Is there a reason why FID measurements were not included in the ablation study?\n5. Why were qualitative results not provided for each individual ablation experiment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Generating text glyphs properly in images produced by text-to-image diffusion models has been a longstanding issue. Research has shown that this capability can be improved by increasing data and model size, but this is somewhat obvious or expected. Following ControlNet, which proposes controllability for Diffusion UNet, the text glyph generation problem can be solved; however, as shown in this paper, it would result in a monotonous style. One of the paper's strengths is that the generated glyphs harmonize with the content of the generated images and are not monotonous. Additionally, the paper's ability to handle multilingual text glyph generation with relatively less data is another notable strength.", "weaknesses": "As revealed in the ablation study, the most significant performance improvement of this method occurs upon the introduction of text embedding. This is attributed to the performance of PP-OCRv3. If one were not to use the OCR module's embedding and instead employ a general image encoder like the CLIP visual encoder, it is questionable whether the same level of performance improvement would have been achieved. Additionally, many modules are added to the vanilla text-to-image diffusion model, but the paper fails to mention the computational overhead that arises as a result. Although the paper highlights multilingual capabilities and provides qualitative results for Korean and Japanese in the figures, it is disappointing that these two languages are excluded from the quantitative results, falling under the \"others\" category. Furthermore, it is regrettable that the results of the ablation study are listed only in terms of Chinese sentence accuracy and NED, without any FID measurements. The lack of qualitative results corresponding to each ablation experiment is also a drawback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eXfGexV40S", "reviewer": "Reviewer_LoJV", "summary": "This paper proposed AnyText, a diffusion-based multilingual visual text generation and editing model. It combines auxiliary latent model which is a control net for text condition, and a text embedding module which injects text visual information in the prompt latent space. Text-control diffusion loss and text perceptual loss are using in training. A large-scale multilingual text images dataset, AnyWord-3M, is introduced.", "questions": "- It is not clear whether the improved results come from better training data or the proposed model. It would be best to compare the baseline models trained on the same dataset, or train the proposed model on the previous LAION glyph subset.\n- in the experiments, the ground truth text mask is used as conditional input. It would be interesting to see what if random text position and mask is used, can it still generate reasonable image?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- extended control net for text input condition\n- new visual text token embedded in the prompt \n- introduced OCR related perceptual loss\n- new dataset\n- new state-of-the-art under proposed new evaluation benchmark", "weaknesses": "- using models trained own dataset to compare with previous baselines is not so fair\n- the requirement of user given text mask is not always easy in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)