How Control Information Influences Multilingual Text Image Generation and Editing?
Authors: Boqiang Zhang
Zuan Gao
Yadong Qu
Hongtao Xie
Venue: NeurIPS 2024 poster (2024)
Keywords: Scene Text Generation
Scene Text Editing
Diffusion
Primary area: diffusion_based_models

Abstract: Visual text generation has significantly advanced through diffusion models aimed at producing images with readable and realistic text. Recent works primarily use a ControlNet-based framework, employing standard font text images to control diffusion models. Recognizing the critical role of control information in generating high-quality text, we investigate its influence from three perspectives: input encoding, role at different stages, and output features. Our findings reveal that: 1) Input control information has unique characteristics compared to conventional inputs like Canny edges and depth maps. 2) Control information plays distinct roles at different stages of the denoising process. 3) Output control features significantly differ from the base and skip features of the U-Net decoder in the frequency domain. Based on these insights, we propose TextGen, a novel framework designed to enhance generation quality by optimizing control information. We improve input and output features using Fourier analysis to emphasize relevant information and reduce noise. Additionally, we employ a two-stage generation framework to align the different roles of control information at different stages. Furthermore, we introduce an effective and lightweight dataset for training. Our method achieves state-of-the-art performance in both Chinese and English text generation. The code and dataset are available at https://github.com/CyrilSterling/TextGen.

How Control Information Influences Multilingual Text Image Generation and Editing?
Boqiang Zhang, Zuan Gao, Yadong Qu, Hongtao Xie
University of Science and Technology of China aqgyd))maiil ustc. edu. htxielustc. edu. cn
Abstract
Visual text generation has significantly advanced through diffusion models aimed at producing images with readable and realistic text. Recent works primarily use ControlNet-based framework, employing standard font text images to control diffusion models. Recognizing the critical role of control information in generating high-quality text, we investigate its influence from three perspectives: input encod- ing, role at different stages, and output features. Our findings reveal that: 1) Input control information has unique characteristics compared to conventional inputs like Canny edges and depth maps. 2) Control information plays distinct roles different stages of the denoising process. 3) Output control features significantly differ from the base and skip features of the U-Net decoder in the frequency do- main. Based on these insights, we propose TextGen, a novel framework designed enhance generation quality by optimizing control information. We improve input and output features using Fourier analysis to emphasize relevant information and reduce noise. Additionally, align the different roles of control information at different stages. Furthermore, we introduce effective and lightweight dataset for training. Our method achieves state-of-thee performance in both Chinese and English text generation. The code and dataset are available https///ghhooooommmmyyyeeettttet
employ two-stage generation framework
Introduction
and image-text paired datasets [23] significant improvements have been made in the quality of image generation. Given the prevalence of text in natural scenes (c.g., posters, slides, signs, book covers, etc.), generating images containing text accurately and reasonably crucial.
With the development of diffusion-based generative models 9 26
20
Recently, several methods have been proposed to address the generation of high-quality visual text [4] 4 (28). Among these, ControlNet-based approaches show strong potential [33 28). enabling flexible multilingual visual text generation, text position control, and easy integration into existing pre-trained diffusion models. Current methods directly utilize ControlNet (36) for text generation control, using global glyph image of a standard font as the condition (as shown in Figure I However, achieving accurate and robust control remains challenging due the complex and fine-grad ime-grained structure of characters. Hence, we pose a meaningful question: How does control information influence multilingual text image generation?
images [33]
To address the above issue, we investigate the impact of control information on visual text generation from three perspectives, as illustrated in Figure T For the input of control information, the current model uses a glyph image to guide the generation of accurate text textures. However, unlike general ControlNet inputs such as Canny edges, depth, or M-LSD lines, text glyph images have unique
"Correspondigg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Control Different Time Steps
Control Output
Control Input
X/-1
Figure 1: The overall pipeline of recent text generation works. It utilizes ControlNet for guiding the text generation process, employing a glyph image with standard font as the control information. Control information different stages is generated in the same manner and directly added the skip features in the U-Net decoder.
properties: 1) Glyph images have areas of high information density concentrated within specific regions, with the rest being meaningless backgrounds. 2) Generation in text region is fine-grained, but extracting fine- lineggriined features from glyph images using standard convolution methods is challenging (further discussed in 3.1.I. These properties limit the performance. For the control information in different time steps, current models follow ControlNet [36] by using fixed control, but they often overlook the role of different time steps in the generation process. We further explore the impact of control at various steps in Sec. 3.1.2 Control in early steps influences both text and non-text regions, ensuring that the background reasonably matches the text. Control in late steps still plays significant role modifying mistakes, which different from the general generation 2 15]. For the output of control feature, these features are injected into the U-Net decoder, which receives three types of features: base, skip, and control. Each of these components differs the frequency domain, which explains their respective roles (discussed in Sec. 3.1.3. Balancing these components during inference crucial. Overall, we explore the influence of control information in text generation, raising several critical questions essential for advancements in visual text generation.
Based on the analysis above, we introduce TextGen, a novel framework aimed enhancing the quality of control information. Specifically, for control input, we introduce a Fourier Enhancement Convolution (FEC) block to extract spatial and frequency features from the glyph control image. This operation can enhance the perception of useful regions and edge textures. For the output control feature, we introduce a frequency balancing factor to adjust the frequency information among the features during inference. For the control information in different stages, propose two-stage framework for coarse-to-fine generation. This framework trains the first-stage model for global control and the second-stage model for detail control. Based the two-stage framework, we naturally propose a novel inference paradigm for unifying text generation and editing tasks. Furthermore, as current datasets for visual text generation are large-scald and noisy, construct a lightweight but high-quality dataset for effective training (details provided in Appendix (A). Unlike previous works, were the first delve into control information in the visual text generation task. Our framework enhances the quality of detail generation while elegantly achieving unified generation and editing tasks. To summarize, contributions
follows:
We conducted an in-depth study and discussion on the impact of control information in visual text generation tasks. Our findings can inspire more future research in this area.
We propose a framework for multilingual visual text generation and editing based analysis, which contains two-stage pipeline and a Fourier enhancement in both training and inference. This framework achieves state-of-the-art performance.
We construct an open-source effective and lightweight dataset for the training of visual text generation and editing.
2
Related Works
2.1
Text Generation
With the advancement denoising diffusion probabilistic models 9 26 and text-to-image gen- eration 22 2 20 it has become possible to generate high-quality images. However, visual text
generation remains challenging due to the need for fine-grained alignment and character detail representation. Recent studies, such as Imagen [22 and eDiff-1 [2]. have focused on improving text

---

Hello
Hello
C6 Stelte D0
SATURDAY
SATURDAY
Hello
Helle Helle
Hello
Figure 2: Differences between text control information and general ControlNet control information, including anime line drawings, M-LSD lines, and Canny edges. General controls focus on the overall structure and tolerate localized errors, while text control requires precise detail.
generation from the perspective text encoders. They found that T5-based encoders 19 outperform CLIP text encoders [8]. GlyphDraw [4]] presents robust baseline for visual text generation by incorporating conditions. It utilizes a glyph image representing single word as the content condition and mask image the positional condition. However, GlyphDraw is limited to generating only one text line per image. GlyphControl [33] further introduces a ControlNet- based framework that employs global glyph image the condition, providing both glyph and positional information, achieving outstanding generation performance. Glyph-ByTSs [3] fine-tunes a T5 language model for paragraph-level visual text generation, achieving remarkable performance in dense text generation. However, it is restricted to producing text English. We propose an effective multilingual framework by controlling information enhancement in ContorINet-baseed framework.
2.2
Text Editing
Scene text editing aims replace text in image with new text while preserving the original background and style. Early approaches focused on generating text on cropped images, allowing for more precise text area generation [34 10
24 30 I7I. SRNet (30) the first divide the editing task into three sub-processes; background inpainting, text conversion, and fusion, which inspired subsequent works 117
Although these methods achieved excellent generation performance, integrating the cropped text area into the original scene images proved challenging such as edge inconsistency. Recently, leveraging the diffusion model, some approaches have conducted generation on complete scene images directly, without decomposing the task into sub-procces coooprocesses. DifTUTE 3 proposes concise framework for directly generating the edited global scene image using the diffusion process. However, solely focusing on the complete editing task limits the practicality and generalization of the model.
21 32
2.3 Joint Text Generation and Editing
Due the similarity between visual text generation and editing tasks, developing a unified framework to jointly address these tasks is meaningful. TextDiffuser [5] employs mask to indicate areas requiring editing, ensuring multitasking uniformity. During the generation task, the mask remains empty. while during the editing task, it preserves areas that do not require editing. Additionally, TextDiffuser introduces a layout generator to design the distribution of text lines. Similarly, Any- Text 28 adopts comparable approach to maintain the uniformity of two tasks and further enhances generation quality with a text embedding module and perceptual loss. Building on our explorations, propose two-stage model and design novel inference paradigm to achieve multitasking unity.

---

BIPA DONG
BRAK DAINE
HORLSSAA
PRMPOROAO
PRIEESSOTN
ROA RH
CASH BACK
CASH BACK
AIAMAMIRR
BOME!
BOMB
BOMB
sasiiddg SATHRDAK
C Blatio SATURDAR
CA Stile SATURDAY
Redlo SATURDAW
EAI
THE
malinet May the ukuleie
30%
T00%
30%
1455
16076
Xo
(a)
(b)
Figure 3: Visualization of the impact of control different denoising stages. Control information removed in the gray segments of the color bar during denoising. (a) Since visual text generation requires much detail texture, control information in later stages still plays important role. (b) Even with only glyph and position images as control information, early-stage control influences non-text regions, ensuring the text region is coherent and matches the background.
Method
3.1
Motivations
3.1.1
Control Information Input
In recent works, the inputs of the control module are the glyph image and position image, which provide the texture and position condition for text regions. However, different from general ControlNet conditions (e.g., Canny edge, M-LSD lines, depth, etc.), text conditions have distinct characteristics. As illustrated in Figure 2 general ControlNet conditions typically influence only macroscopic coarse-grained style and global edges, and some incorrect minor texture generation is considered acceptable. However, small differences in texture details can lead to content errors or unrealistic and unreasonable textures in visual text generation. Therefore, using the general ControlNet poses challenges in controlling detailed textures and fine-grained handwriting. Moreover, the text glyph condition concentrates only certain regions, making it sparse condition unlike other general conditions. This characteristic causes the convolution based ControlNet encoder to introduce noise in empty when extracting features, which interferes with the text area's features and affects the correct allocation of attention between edge and background areas. Enhancing the ControlNet encoder's 's perception of locally useful details and edge information is essential for improving text image generation. Meanwhile, the characteristic of having high information density in local areas suggests that we can seek solutions in the frequency domain.
3.1.2 Control Information in Different Denoising Stages
Some studies on general diffusion [5] 2 have suggested that control information in the later stages of the denoising process contributes little to the diffusion model. However, due to the specific nature of text, where text strokes constitute detailed information, we find that control in the later stages remains crucial. As shown in Figure (a), omitting control information in the later stages often results in incorrect text content generation. The control module in these later stages corrects such errors, ultimately leading to high-quality images. This finding indicates that performing the editing task at a late time step reasonable, as it mitigates the performance impact of joint multi-task training.
Furthermore, we investigate the role of early control. As shown in Figure 3 (b), the control information in early steps has a significant impact on the global semantics of the generated image, aiding in the ali alignment of text areas with the global scene. Without the control information the early steps, the generated text regions appear unreasonable and do not match the background. Notably, even though only glyph and position images provide control information, the early stages still strongly influence the generation of non-text regions (the non-text areas may be totally different).

---

Glyph
Global Control Stage
Step
Detail Control Stage
FEC
FEC
Position
Global
Desai ControlNet
Fuse
SC
Position
VAE
VAE
Niñae
U-Net
E(),
Text Embeddingg Module
Inference
Generation
|
|ull|
|col|
Xr
X--1
Xo.sr
Xo.sT-1
Editing
|
Dm
+ Nusion
X
Xo.rr-1
XpST-1
X
X
Figure The pipeline of our TextGen. comprises two stages: the global control stage and the detail control stage. Control information utilizes a Fourier Enhancement Convolution (FEC) block and Spatial Convolution (SC) block extract features. During inference, we introduce a novel denoising paradigm to unify generation and editing based on our framework design. Best shown in color.
Therefore, the control information of different stages should be adapted according to these findings. We divide the control module into global and detail stages (Fig. 4 each with distinct parameters. In affect the entire image, while
the global control stage, we expect that the control information in the detail control stage, we incorporate a mask to guide the model in optimizing local details. Furthermore, the detail stage
refiner in text generation and as editor in text editing.
3.1.3 Control Information Output
During inference, the output of the control module is injected into the base diffusion process. Each layer in the U-Net decoder can be formulated as: F; D, C.-1), where D; is the i-th layer, F; is the output feature of i-th layer, S; and C, represent the skip feature and the control feature of i-th layer, respec- tively. These three parts of the input represent different types of information. Following FreeU (25], we further investigate the balance among these three components, As shown in Fig. 5 we visualize the Fourier relative log am- plitudes of F. S and C. It can be observed that the skip feature contains more tigh-frequency information than the base feature, which may infer denoising (the same conclu- sion with (25). Therefore, there is a need balancing between the base feature and the skip feature. Furthermore, compared with the fusion feature, the control feature has more high and wwwffregencyy information, with greater gap at low frequencies than at high frequencies. How- ever, since only aim to control the texture, which belongs to high-frequency information, the low-frequency information needs to be suppressed.
0-
skip fenture
control Feature
Relative Log. Amlitude
-1
-3-
-4
-5-
0.0x
0.25x
0.5x Frequency
0.75x
1.0g
Figure 5: The relative log amplitude of three parts of features in U-Net decoder.
3.2 Pipeline
Based on the motivations and discussions above, we propose novel framework named TextGen. The pipeline our TextGinn is illustrated in Figure 4 During training, our model comprises two stages: the global stage and the detail stage. The parameters of the pre-trained diffusion U-Net are

---

fixed, and each stage only trains a ControlNet. The global control stage is trained exclusively on larger time steps, while the detail control stage is trained only on smaller time steps. Through such an operation, the global control stage focuses on structure and style construction, and the detail control stage concentrates on detail modification. In this section, we first detail the control design in Sec. [.3 Then, we describe the enhancement of control information output in Sec. 3.4. Finally, we propose novel inference paradigm for task unification using our model in Sec. 3.5
3.3
Model Control
stage receives two pieces of control information: position image indicating the text positions and a glyph image indicating the standard font of the texts. We use Spatial Convolution (SC) block and Fourier Enhance- ment Convolution (FEC) block to extract the feature of position image and glyph image, re- spectively. The structures of SC and FEC are illustrated in Figure G In SC, we employ gen- eral convolution for spatial perception, whereas in FEC, we use two branches for information extraction. The spatial branch similar to SC, while the frequency branch employs 2D Fast Fourier Transform (FFT) algorithm to transform the features into the frequeney domain and per- forms convolution in this domain. Subsequently, an Inverse Fast Fourier Transform (IFFT) algorithm is used to transform the frequency features back. Additionally, the input layer of both blocks is a global perception operation, achieved through convo- lution with large kernel size. This is because the text contains global semantic information, and the general convolution- based encoder has a local receptive field that limits information interaction.
The global control
Ourpur Coev2d
Ourput Crevod
Nx
Nx
SIU
SILU
Conv2d
Conv2d
IFFT
Cunzil ReLU
SILUI
FFT
Convod
Conv2d
Conv BN SILU
Spatial Branch
Frequescy Branch
Global Perception
Global Percepsion
(a)
(b)
Figure 6: (a) The Spatial Convolution Block. (b)
The Frequency Enhancement Convolution Block.
The detail control stage receives three pieces of control information: position image, glyph image, and a masked image. Unlike the global control stage, the detail control stage incorporates the masked image as one of its inputs, aiming to provide background information. This stage is designed to generate specified texts at designated positions while keeping the background consistent with the masked image. The module's 's output is multiplied by position mask, enabling the model to focus on modifying the detailed texture of the text area. It's worth noting that the mask is only applied in the last two layers of the U-Net decoder (for feature sizes greater than or equal to 32). This is because the first two layers (feature sizes less than 32) primarily handle global information, which contributes little the detailed texture. Moreover, by retaining the first two layers without applying the mask, we ensure that the background of the output image remains consistent with the masked image while modifying inconsistent areas.
The frequency enhancement block performs convolution operations both spatial and frequency domains, offering two main benefits: 1) The Fourier transform of visual features provides global information about the glyph image, overcoming the limitations the receptive field in spatial convolution. 2) The glyph image contains both localized detail-rich areas and meaningless backgrounds. Convolution in the frequency domain acts as a frequency filter, allowing for the adjustment of attention different frequency components. This facilitates the extraction of useful information and mitigates the interference of irrelevant information from a global perspective.
Discussion:
3.4 Enhancement for Control Information Output
In our framework, each layer of the U-Net decoder receives three parts of information: the backbone feature F, the skip feature S, and the control feature C. As discussed in Sec. 3.1.3 there is need for balancing among three parts during inference. Therefore, we propose a Fourier enhancement method as formulated as follows:
F = + ( +CC,
(I)
C, = - 07),

---

where & and I represent FFT and IFFT, @ denotes the element-wise multiplication. a and / are balancing factors for the control feature and the base feature. y is modulation factor in the frequency domain. Although enhancing high-frequency information is desired, directly doing so will introduce noise and reduce generation quality. Therefore, we suppress low-frequency information to emphasize fighfrreuency components. This is achieved using scalar to suppress low-frequency information as follows:
{s if < Tthreshs
(())
(2)
(1
otherwise.
3.5 Inference Paradigm for Multi-task
Based on our model and analysis, we propose novel inference paradigm for task unification. For the image generation task, random noise is inputted into the global control stage for the early T/2 steps and the detail control stage for the remaining steps. For the text editing task, the original image is first noise-added to 80% time-step, which maintains most of the global style and texture while destroying the original text content. Then the new text content is first generated using the global control stage until the 7/2 time step, and the remaining time steps use the detail control stage to modify the details. Since the control input of the detail control stage contains the masked original image, it restore the background information that was destroyed during the noise addition, and at the same time optimize the new text content at the specified location.
4 Experiments
4.1 Datasets
Recently, several works have introduced datasets for text generation and editing tasks. TextDiffuser [
4 introduced a dataset named MARIO- 10M, comprising approximately 10 million images annotated
with bounding boxes and content of text regions. AnyText (28) proposed a benchmark named
Wny for evaluation. However, training on 10 million images requires significant computing resources. Therefore, we introduce TG2M, multilingual dataset sourced from publicly available images including MARIO-10M [5). Wukong R,, TextSeg (31], ArT [6]. LSVT [27], MLT [16),
ReCTS (37) Although TG2M contains significantly fewer images, is highly effective for training and achieves superior performance. The dataset will be detailed in the Appendix
4.2 Implementation Details
In our implementation, the diffusion model is initialized from SDI. and our code is based on d The text embedding module follows AnyText (28). We train our model on the TG2M dataset using 8 NVIDIA A40 GPUs with a batch size of 176. Our model converges rapidly and requires only 5 epochs of training. The learning rate is set le-5. Following previous generation and recognition works 35 29 28, 77 we set the maximum length of each text line to 20 characters and the maximum number of lines in each image to During inference, the Fourier balance factors a, B, and s set to 1.4, 1.2, and 0.2, respectively.
We evaluate our model on the Word 28 benchmark. We use DuGuangOCR to recognize the text region and measure performance using sentence accuracy (ACC), Normalized Edit Distance (NED), and Fréchet Inception Distance (FID). During inference, the settings (random seed, control strength, etc.) are consistent across all experiments.
4.3 Ablation Study
Owing to resource constraints, following Any Text 28 we randomly select 200k images (40k English and 160k Chinese) from TG2M the training set for ablation. The results are shown in Tab.

---

Table 1: Ablation of proposed methods. FEC denotes Fourier enhancement convolution, GP signifies
global perception in FEC, TS represents the two-stage generation framework, and IFE indicates inference Fourier enhancement.
English
Chinese
FEC
GP
TS
IFE
ACC1
NED1
ACC1
NED1
49.51
75.99
31.50
60.22
50,90
76.81
0.82
56.98
77.28 17.06
52.24 1,34
77.64
1 0.83
58.60
1,2
78.04 0.76
53.03 0.079
78.14
60.47
78.86
60.18 1.115
82.28
1.1.4
61.42
1 0.95
80.56 1.70
Table 2: Comparison with stte-of-thhear methods. Data denotes the amount of data used in the training process. Our baseline is the AnyText-v (28] model trained on our TG-2M.
English
Chinese
Methods
Data
ACC1
NED1
FID.
ACC1
NED1
FID.
ControlNet
36
58.37
80.15
45.41
36.20
62.27
41.86
GlyphControl
33
10M
52.62
75.29
43.10
4.54
10.17
49.51
TextDiffuser
5
10M
59.21
79.51
41.31
6.05
12.62
53.37
AnyText-v1.0
28
3.5M
65.88
85.68
35.87
66.34
82.64
28.46
Baseline
2.5M
64,26
a usi
84,80 :: 0.10
41.65
2
65.02
6.11
81.95
0.12
30.04
0.79
TextGen
2.5M
73.36
1.15
88.98
0.12
40.37
1.71
67.92
0.28
83.94
0.89
28.90 0.94
From the table, we observe the following: 1) All proposed methods yield performance gains in both Chinese and English text generation. 2) The FEC block enhances edge and texture features through Fourier enhancement, with more significant gains in Chinese due to the greater complexity of Chinese characters. is worth noting that our model trained only on 200k images can achieve 61.42% and 60.18% sentence accuracy on Chinese and English text generation, which is almost as good as the state-of-ther of-thh--rttttttt performance on large-scale training sets.
4.4
Comparison Results
4.4.1
Quantitative Results
Compared to other methods, our approach uses less data while outperforming the state-of-the-art. as shown in Table 2 For a fair comparison, all methods are evaluated under the same settings. The performance of both the bascline and TextGen is assessed using four random seeds, with the final metrics reported as averages and standard deviations. Our method achieves a 9.1% gain in sentence accuracy for English texts and 2.9% gain for Chinese texts compared to baseline trained TG-2M. Notably, other approaches require large amounts of training data and employ perceptual loss enhance performance, which is siiminnginefffciennt Our method, in contrast, does not require additional losses and converges faster, making it easier to train. Besides, we compute the FID on the Any Word-FID 28 benchmark. Our FID scores achieve comparable performance but the best. The higher FID score does not necessarily imply the lower visual quality of the generated images. Our generated images demonstrate greater diversity and there distribution gap between our training set and Any Word. This issue is discussed in more detail in Appendix C
4.4.2 Qualitative Results
The qualitative comparison is shown in Fig Our TextGen produces high-qualigy images with text in various scenarios and excels generating artistic text with a wide range of visually appealing styles. For Chinese text, as demonstrated in Fig TextGen generates more realistic and readable results, particularly in smaller texts. Finally, Fig. 1llustrates the editing capabilities of our model, which can edit various text styles and contents using the proposed inference paradigm.

---

ControlNet
GlyphControl
TextDifluser
TextDiffuse22
AnyText 265 MASKIIIIING nistant POT RECIES
Baseline
GT Fo 25+ Ro THANKSGIVING Trefial Recingg
Ours
25t
THINSCGIIIG
25% THANSGGIING DISTANT OOPPES
THANKSGNING Insiant Pec
HURAWAA WIL
pootecipes
btard pol Recips
Inciant PelPecipes
STAR TREK
STAR TRCN
STAR TarO
STAN FRAY
ICAR
STAR UPE
STAR TREK ICAR
STAP TREX C A
STAR TREK ICAR
MCAR
ICER
TRY
Ponnymoonni
Honoo paosonoay Awchohe Veniog
Home gropning Venice
Honzynosningg Vonice
Harrymoning
Venice
Country in your Heart no Cost
County Helrt.
Cuentyy Heart Close
Crudity Hadaz Chokot
Bolit
Heart pedte SiouE
All Chacee
Chouet
Clicel
Figure 7: Qualitative comparison of generation performance in English texts. Our TextGen can generate more artistic and realistic texts.
ControlNet
AnyText
GT
Ours
Origin Image
Edit
Edi12
Edit3
BESt DE
I Happy Bisthday
i Happip New Year
Happy
Happip
Every dedyy
Foxt Gen
3 X!!
Tetto
MAD
HEAD
SON SCNTTT
CITY
SUOITST
SCOTET
SCOTTT OUSTIMG
FORDD VATGERRR ISTTINATION
LLOBLD WITEERRA
CARRY INTERRAI
WITERRDH ASSOY
AAZM FRRER
ISSI
ASSIO ISSONIATION ANTONN
Figure 8: Comparison of generation in Chinese,
Figure 9: Visualization of editing performance.
5 Limitations
We propose novel diffusion-based network for multilingual text generation and editing, which demonstrates robust performance. Our network excels at generating high-quality scene images with text. However, the framework based on latent diffusion presents certain limitations. 1) The VAE employed in latent diffusion restricts the performance of fine-grained texture generation, particularly for complex texts. Because the diffusion process operates in the latent feature space, the VAE decoder struggles to generate small or complex texts. Consequently, our method is unable to generate such images. This issue can be addressed by generating each local sub-region separately. 2) The text condition controls the content of the generated image. However, the CLIP text encoder has limited ability comprehend text, resulting in performance limitations. To resolve this issue, we pre-train the diffusion model with a large language model serving as the text condition encoder.
Moreover, generating false text can contribute to the spread of misinformation, potentially resulting serious consequences. It hoped that this technology will be used responsibly, fostering a healthy and ethical academic environment.

---

6 Conclusion
Based on ControlNet, current visual text generation has made significant progress, In this work, we build on recent studies using ControlNet to investigate how control information influences visual text generation from three perspectives: control input, control different stages, and control output. Through experiments and discussions, we derive several key conclusions. Based on our analysis, we propose a novel visual text generation framework that improves control information utilization, which surpasses the state-of-the-art performance. We believe the insights we gained about control information can inspire future research the community.
Acknowledgments
This work is supported by the National Key Research and Development Program of China (2022YFB104700) the National Nature Science Foundation of China (U23B2028, 62121002, 62102384). This research is supported by the Supercomputing Center of the USTC. We also acknowl- edge the GPU resource support offered by the MCC Lab of Information Science and Technology Institution, USTC.
References
[1] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading. and beyond. arXiv preprint arXiv:2308. /2966, 2023.
[2] Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint XrXi::2 (0X:::22/10033:: 1.0132,, 2022.
[3] Haoxing Chen, Zhuoer Xu, Zhangxuan Gu, Yaohui Li, Changhua Meng. Huijia Zhu, Weiqiang Wang, et al. Diffute: Universal text editing diffusion model. Advances in Neural Information Processing Systems, 36, 2024.
[4] Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser-2: Unleashing the power of language models for text rendering, arXiv preprint arXiv22311.16655 2023.
[5] Jingye Chen, Yupan Huang, Tengchao Lx, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: Diffusion models as text painters. arXiv preprint arXiv:2305.10855 2023.
[6] Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang. Junyu Han, Errui Ding, et al. lcdar2019 robust reading challenge on arbitrary-shaped text-rre-art. art. In 2019 International Conference Document Analysis and Recognition (ICDAR), pages 1571-1576. JEEE, 2019.
[7] Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7098-7107, 2021.
Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang, Wei Zhang, Xin Jiang, et al. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. Advances in Neurul Information Processing Systems, 35:26418-26431, 2022.
[9] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information pmcessing systems, 33:68-00-8811, 2020.
[10] Qirui Huang, Bin Fu, Aozhong Zhang, and Yu Qiao. Gentext: decoupled font and texture manipulation. arXiv preprint arXiv:2007.0664) 2022.
Unsupervised artistic text generation via
[11] Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, and Lianwen Look closer to supervise better: One-shot font generation via component-based discriminator. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 13482-13491, 2022.
[12] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 22ip-2: Bootstrapping language -mage pre-trainnn training with frozen image encoders and large language models. In International conference on machine learning. PMLR, 2023.
pages
10

---

[13] Zeyu Liu, Weicong Liang. Zhanhao Liang, Chong Luo, Ji Li, Gao Huang, and Yuhui Yuan. Glyph-byt5: A customized text encoder for accurate visual text rendering, arXiv preprini arXiv:2403.09622, 2024.
[14] Jian Ma, Mingjun Zhao, Chen Chen, Ruichen Wang. Di Niu, Haonan Lu, and Xiaodong Lin. Glyph- draw; Learning to draw chinese characters image synthesis models coherently. arXiv preprint arXiv:2303.17870, 2023.
[15] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for texl-to-image diffusion models. In Proceedings of the AAAl Conference Artificial Intelligence, volume 38, pages 2296-4304, 2024.
[16] Nibal Nayet, Yash Patel, Michal Busta, Pinaki Nath Chowdhury. Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal. Jean- Sean-Christophe Burie, Cheng-lin Liu, lcdar2019 robust reading challenge on multi-lingual scene text detection and recognition rro-m1t-2019. In 2019 International conference on document analysis and recognition (ICDAR), pages 1582-1587. IEFE, 2019.
[17] Yadong Qu, Qingfeng Tan, Hongtao Xie, Jianjun Xu, Yuxin Wang, and Yongdong Zhang. Exploring stroke-level modifications for scene text editing. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 2119 2127, 2023.
[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[19] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
[20] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the JEEF/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
[21] Prasun Roy, Saumik Bhattacharya, Subhankar Ghosh, and Umapada Pal. Stefann: scene text editor using font aduptive neural network. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13228-13237, 2020.
[22] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan. Tim Salimans, et al. Photorealistic text-to- image diffussin models with deep language understanding, Advances in neural information processing systems, 35:36479-36494 2022.
[23] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 01225278-52994 2022.
[24] Wataru Shimoda, Daichi Haraguchi, Seüchi Uchida, and Kota Yamaguchi. De-rendering stylized texts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1076-1085, 2021.
[25] Chenyang Si, Ziqi Huang. Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. arXiv preprint Xiv:23009 11497, 2023.
[26] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint urXiv.2010.02502. 2020.
[27] Yipeng Sun, Zihan Ni, Chee-Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, et al. Iedar 2019 competition on large-scale street view text with partial -ssv.... In 2019 International Conference Document Analysis and Recognition (ICDAR). pages 1557-1562. JEEE, 2019.
[28] Yuxiang Tuo, Wangmeng Xiang, Jun- Jun-Yan He, Yifeng Geng, and Xuansong Xie. Anytext: Multilingual visual text generation and editing. arXiv preprint arXtv:2311.03054, 2023.
[29] Zixiao Wang. Hongtao Xie, Yuxin Wang. Jianjun Xu, Boqiang Zhang, and Yongdong Zhang. Symmetrical linguistic feature distillation with clip for scene text recognition. In Proceedings of the 31st ACM International Conference on Multimedia, pages 509-5111 518,1188111 2023.

---

[30] Liang Wu, Chengquan Zhang, Jiaming Liu, Junyu Han, Jingtuo Liu, Errui Ding, and Xiang Bai. Editing text in the wild. In Proceedings of the 27th ACM international conference on multimedia, pages 1500-1508, 2019.
[31] Xinggian Xu, Zhifei Zhang, Zhaowen Wang, Brian Price, Zhonghao Wang. and Humphrey Shi. Rethinking text segmentationn: novel dataset and text-specific refinement approach. In Proceedings of the IEEE/CVF conference on compuder vision and pattern recognition, pages 12045-12055 2021.
[32] Qiangpeng Yang. Jun Huang, and Wei Lin. Swaptext: Image based transfer scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14700-14709, 2020.
[33] Yukang Yang, Dongnan Gui, Yuhui Yuan, Weicong Liang. Haisong Ding, Han Hu, and Kai Chen. Glyph- control: Glyph conditional control for visual text generation. Advances in Neurul Information Processing Systems, 36, 2024.
[34] Boqiang Zhang. Hongtao Xie, Zuan Gao, and Yuxin Wang. Choose what you need: Disentangled representation learning for scene text recognition, removal and editing. arXiv preprint arXiv:2405.04377, 2024.
[35] Boqiang Zhang, Hongtao Xie, Yuxin Wang. Jianjun Xu, and Yongdong Zhang. Linguistic more: Taking further step toward efticient and accurate scene text recognition. arXiv preprint vrXiv:2305.05145 2023.
[36] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-lo-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[37] Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, Mingkun Yang, Icdar 2019 robust reading challenge on reading chinese text on signboard. In 2019 international conference on document analysis and recognition (ICDAR), pages 1577-1581, TEEE,
pages
2019.
12

---

A Details of Dataset
To train the visual text generation model, we constructed dataset called TG-2M, which contains rich textual data. The images in this dataset are sourced from existing open-source datasets, including
MARIO-ION, 10M [. Wukong ( ArT [6, LSVT (27), MLT [T6], ReCTS [37, TextSeg [I., Following AnyText (88), we filter the images with some rules, which can devided into three steps. Specifically,
the filtering rules of the first step include:
The images of width smaller than 256 will be filtered out.
The images with aspect ratio larger than 2.0 or smaller than 0.5 will be filtered out.
After step 1, we use PP-OCK to detect and recognize the texts in these images. Then, we undergo the second filtering step:
The images with more than 10 texts will be filtered out.
The images with more than 3 small texts will be filtered out. The small text refers to horizontal text with a height of less than 30 pixels or vertical text with a width of less than 30 pixels. The orientation the text is determined by the aspect ratio of the text bounding box, with an aspect ratio less than 0.5 considered vertical text.
Images containing more than text instances with recognition scores below 0.7 will be filtered out.
We use BLIP-2 [2] and Qwen-VL. [ to recaption the images. First, we generate initial captions using BLIP-2. Because some initial captions low-quality, then modify these captions using DW. The low-quality captions are defined as those containing meaningless text or having low
CLIP similarity with the reference image. This process is necessary because some captions generated by BLIP-2 are meaningless, as shown in Fig. [0] Additionally, although Qwen-Vi VL's captions are of high quality, many of them are quite lengthy, which can affect the understanding of the CLIP text encoder in the diffusion model.
Welcome Carousel!
BLIP-2 Caption: carouselell carouselell carouselell carouselell carouselell carouselell carouselell carouselell
Qwen-Ve Caption: animated welcome message displayed by website carousell.com
It features several icons arranged around a central circle which contains the text "Carousell".
B1.I-22 Caption: cover for car, car cover for car cover for car, cover for car cover for car.
5 Layer
Qwen-VL. Caption: Free shipping! Car Cover Sun UV Snow Dust Rain Resistant Protection Covers XL XXL size(China (Mainland))
BLIP-2 Caption: barahan fire breathers
FIRE
BREATHE HERE
Qwen-VL Caption: The album artwork of Sarah Burtons' debut release Firebreatherers!.
Inaret
BLIP-2 Caption: pipi pi
pi pi pi
Grtmaal
i
π
Qwen-VL Caption: Two speech bubbles suy be rational and get real.
Figure 10: Comparison of captions by BLIP-2 and Qwen-VL.
Examples from our TG-2M dataset are shown in Fig illustrating the variety image styles. The
dataset statistics are summarized in Tab TG-2M contains total of 2.53 million images with 9.54
13

---

Table 3: The statistics of our proposed TG-2M dataset.
image count
line count
mean chars/line
#line < 20 chars
English
1.3M
5.59M
4.23
5.50M, 98 .%%
Chinese
1.23M
3,95M
5.68
3.83M, 97.0%
Total
2.53M
9.54M
4.83
9.33M, 97.8%
FUNCH MORE NAZIS
SOVE REACE TUT
2017
PRESSURE DROP FOOTO
REFRESH
Year Okas ROGK!
Angles list
YOUR Syy Bullege
TEACHEE PLANNER 20
TRANSIT OF VENUS
CaaLar Shoppet
AWARD
T
$
GGO
Figure 11: Some cases in our proposed TG-2M dataset.
million text lines. On average, each text line contains 4.83 characters. Notably, 97.8% of the text lines have fewer than 20 characters, which facilitates effective training of our model.
B Discussion about the Two-Stage Framework
We propose a two-stage generation framework that achieves detail optimization and task unification. However, this enhancement does not notably improve recognition accuracy our ablation study. This is because: 1) The first stage already allows for some detailed modifications. 2) Our ablation study was conducted on subset of the TG-2M dataset. The second stage enhances texture details, but its performance is limited with insufficient data. On the complete dataset, the two-stage framework demonstrates better performance, as detailed in Table
Table 4: The comparison performance on the complete dataset.
English
Chinese
Methods
ACC1
NED1
ACC1
NED1
w/o TwoStage
71.11 73.36
88.07
66.68
83.16
TwoStage
88.98
67.92
83.94
C Discussion about FID
The Fréchet Inception Distance (FID) metric evaluates the distribution gap between generated images and target images. higher FID indicates a larger distribution gap. We evaluate the FID score on the AnyWord (28) benchmark, which provides a subset of images for this purpose. Since the Any Word FID benchmark derived from the Word training set, is reasonable for AnyText achieve better FID score due to the distribution gap between our TG-2M and Any Word. Additionally, our TextGen can generate more artistic texts, resulting in diversity distribution. Therefore, lower FID score does not necessarily imply lower visual quality of the generated images.
D Discussion about Future Work
Based on ControlNet, current visual text generation has made significant progress. We further investigate the control information ControlNet-based visual text generation tasks and draw several conclusions. Future performance improvements can be achieved through three approaches: 1) Construct high-quality datasets, as current datasets still contain incorrect labels and unreasonable captions. 2) Enhance the text embedding module. Leveraging large language models (LLMs), can
14

---

OSHE 60 COUNTY
NEBRASKA A STATE FAI
HEENEE FURY WORAS
100% Setesaaiie MARRASTED
CRAFT BEER ON TAP
TRLD MAD
STOP
SHERIFF'S OFFICE
GRANDS HIP DETION
LACROSSE GIRL
SEKVII FALL
JUNIORI
Frad eern chuele
Electric Jamilt
CHAOSRANE
ARTURAL
LdS
Wheather Prof
HIR +H F
2019
7C
Figure 12: More qualitative results generated by TextGen. Both Chinese and English are high- quality and realistic.
design more robust text embedding module than the CLIP text encoder, capable of understanding more detailed captions.
E More Qualitative Results
We present additional qualitative results generated TextGen in Fig. 12 images with coherent and readable text. Additionally, TextGen capable of generating artistic text for applications such logos, posters, and clothing design.
TextGen produces realistic
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We discuss the influence of control information in visual text generation. The abstract and the introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
Our investigations are validated by experiments and visualizations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results of the paper in Sections 4. I and 4.2. The details about our datasets are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We will release our code and dataset after accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /otp://ggu..eeeoooouuuuumbbbbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We detailed all of these Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provide the suitable error bar in Table 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide all of the information in Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conform.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work about text generation, which does not have societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: This paper does not existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "IrRSzKjakr", "reviewer": "Reviewer_bu9j", "summary": "This paper analyzes the issue of control in image generation models. Specifically, the article addresses three aspects: input control information, the impact of control information at different stages, and output control information. The model was optimized for two tasks: text-to-image and image editing. Using a method similar to FreeU, the paper conducts Fourier frequency domain analysis on the input and output features and proposes a two-stage generation model based on previous findings.", "questions": "1.Will the dataset be released? If so, please address the privacy, copyright, and consent of the data.\n\n2.The convolution theorem states that the Fourier transform of a function's convolution is the product of the Fourier transforms of the functions. In other words, convolution in one domain corresponds to multiplication in another domain, such as time-domain convolution corresponding to frequency-domain multiplication. Can the FEC network in the article be considered as performing a simple point-wise multiplication operation?\n\n3. I have questions about the effectiveness of FEC. The article only shows that using FEC is better through ablation experiments, but I have two concerns: 1) The improvement for English is not significant, while it is more noticeable for Chinese. Is the main reason for this that the model was fine-tuned on Chinese? 2) When using methods like depth control, the output might not match the control, but the output could still be reasonable. Using glyph images for control requires a strong match between the output and the input text. From a human observation perspective, glyph control might seem more precise, but for model training with MSE loss, both approaches could be similar.\n\n4. Section 4.3 does not have accompanying text?\n\n5. Why does convolution perform poorly in understanding text? Shouldn't there be some small experiments to demonstrate this? Alternatively, could it be explained through some visualization analysis or derivation process?\n\n6. Figure 4, which serves as the overall pipeline diagram, does not seem to reflect the inference structure discussed in Chapter 3.5.", "limitations": "The article does not specifically discuss the potential negative societal impacts. For example, generating high-quality images guided by text might facilitate the creation and spread of fake news. Additionally, using more complex models could lead to increased resource consumption, among other issues.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The writing of the article is excellent, with clear and concise sentences and well-organized structure.\n2.The proposed final model FEC+GP+TS+IFE shows significant improvement.\n3.The method is quite innovative, with a novel approach.", "weaknesses": "1.Some models lack detailed theoretical analysis, which makes the purpose of the model unclear.\n2. It is not clear what the task of this paper issue. Please point it out in a prominent position in the first chapter.\n3. The layout is a bit messy. For example, should Table 2 be placed above Table 1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments and some of the affirmations of our paper. These questions are insightful and deepen my thinking. Below are my rebuttal and discussion for these questions 😊.\n\n**The writing.** We apologize for the writing errors in our paper. In this paper, we primarily studied the impact of control information. Through experiments, we derived some conclusions regarding control information. Based on these conclusions, we designed a novel and effective model to address the issues. We will clear the presentation in the final version of the paper, specifically in the first chapter, according to your suggestions. Furthermore, we believe that the placement of Table 1 and Table 2 is appropriate, as the body of the paper first presents ablation experiments followed by comparative experiments. If you find this sequence unreasonable, we will adjust the order in the final version of the paper.\n\n**Question 1: Privacy and copyright.** We recently employed more stringent filtering methods to process and clean the data, resulting in higher-quality data. Such data can significantly enhance generation performance. We will **fully** release our all dataset and code **without reservation** after the paper is accepted.\n\n**Question 2: Convolution and Fourier transform.** In mathematical theory, convolution in the frequency domain is equivalent to multiplication in the spatial domain. However, in our model, performing convolution in the frequency domain exhibits some differences. This is discussed in Section 3.3 of our paper, but we offer a more detailed discussion here: \n1) In the frequency domain, **additional operations such as activation functions are applied alongside convolutions**, making it not strictly equivalent to point-wise multiplication in the spatial domain.\n2) Direct convolution in the spatial domain is limited by the receptive field. However, each point in the frequency domain represents global information of the same frequency. This allows convolution in the frequency domain to enable **global interactions of similar frequencies**.\n3) Performing convolution in the frequency domain can capture features that are difficult to learn in the spatial domain during gradient descent training. **The training effect of using gradient descent is not equivalent to the point-wise product operation in the spatial domain.**\n\n**Question 3: Concerns in FEC.** \n1. **Concern 1:** The data and data mixture used in the ablation study in Table 1 are entirely consistent with Anytext [1], which randomly selected 200k training samples from the dataset. We did not perform any separate fine-tuning. The noticeable improvement in Chinese data is due to the model's enhanced sensitivity to details under the frequency domain enhancement from FEC. For generating more complex Chinese content, the performance improvement achieved with less data will be even greater.\n2. **Concern 2:** Using MSE loss, the model's training objective remains consistent. However, we propose to enhance the model's ability to perceive details and high frequency. This allows the model to leverage more detailed information, thereby improving the quality of detail generation under the same MSE loss. While such an approach is unnecessary for depth-controlled generation, it is crucial for text generation, which demands fine-grained control.\n\n**Question 4: Section 4.3.**  We apologize for the typographical errors and we will revise the typography in the final version of the paper.\n\n**Question 5: Discussion about convolution.** Our argument is not that convolution performs poorly in understanding text, but that standard convolution faces challenges. Due to the restricted receptive field of convolution, conventional operations struggle to capture global information. **Text often appears in elongated forms**, covering a large area in one direction, which hinders the performance of ordinary convolution in extracting text features and semantics. **Similar observations have been noted in several scene text recognition studies [2] [3]**. Moreover, in glyph images, **most regions are purely black**, serving as noise relative to the text regions during standard spatial convolution operations. We will reference the aforementioned studies to support our perspective in the final version of the paper.\n\n**Question 6: Question about the figure.** The figure 4 contains the inference pipeline we proposed. The lower part of the diagram illustrates the inference strategies. Different inference strategies are used for generation and editing tasks. The color of the UNet corresponds to either the global or detail ControlNets, indicating the use of different control models at various stages. The diagram is consistent with the description in Section 3.5.\n\n[1] Tuo Y, Xiang W, He J Y, et al. Anytext: Multilingual visual text generation and editing[C]//Thirty-seventh Conference on Neural Information Processing Systems. 2023.\n\n[2] Fang S, Xie H, Wang Y, et al. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 7098-7107.\n\n[3] Atienza R. Vision transformer for fast and efficient scene text recognition[C]//International conference on document analysis and recognition. Cham: Springer International Publishing, 2021: 319-334.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NJmyV2oz9v", "reviewer": "Reviewer_JA2v", "summary": "This study explores the advancement of visual text generation using diffusion models within a ControlNet-based framework, focusing on the impact of control information. It examines input encoding, the role of control information during different stages of the denoising process, and the resulting output features. The authors propose TextGen, a method that aims to enhance generation quality by optimizing control information with Fourier analysis and a two-stage generation process.", "questions": "1. Why only consider these three control methods.\n2. What is the reason and necessity for introducing Fourier transform.\n3. Why can't both global and detail control be considered simultaneously during the diffusion process? Wouldn't it be possible to use a learnable module to control the weights of these two controls. The two seperated control stages proposed in this paper is clearly not optimal.\n4. It is recommended to add more ablation experiments, such as without the Fourier module, and considering both global and detail control simultaneously. More visualization results should be provided, such as visualizations of the ablation experiments.\n5. How does the method perform on characters that appear infrequently in the dataset.\n6. It is recommended to include failure cases and analyze the reasons. Additional results for text image editing should also be provided.\n7. Please adequately address the concerns raised in the weaknesses section.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Analyzed the impact of three control factors on text image generation.\n2. Achieved comparable results in text image generation and editing.", "weaknesses": "1. The method lacks novelty, the writing of the paper is unclear, and the experimental results are insufficient. The experimental analysis does not provide new insight into the field.\n2. The paper's analysis of control factors for text image generation is limited to three control conditions, which may not be the most effective. For example, character embeddings could be used as input, sentence prompts through an LLM could be used, or text coordinate encodings could be used as control conditions.\n3. The necessity of the FEC module is not very clear, as it does not specifically target text image scenarios. Additionally, the computational efficiency of FFT and IFFT needs to be considered.\n4. Why are global control and detail control split into two stages? Would it not be possible to consider both global and detail control simultaneously during the whole diffusion process?\n5. The improvement of this method over existing methods is not obvious, especially in terms of visual and perceptual quality. Training on a small dataset also affects the method's generalizability. For example, can it still correctly generate text that appears infrequently or not at all in the proposed dataset?\n6. The results lack demonstrations of “scene” text image generation; most images resemble text printing and have less realism compared to Anytext. This may be due to dataset bias, as shown in Figure 11. Thus, a small dataset is not a contribution of this paper and could be considered a drawback for large models.\n7. The ablation experiments are insufficient, and the effectiveness of the modules is not convincingly demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and have carefully considered your questions and suggestions. Below is our rebuttal and discussion of these questions 😊. Due to rebuttal restrictions, we are unable to include images. We will provide the corresponding visual results in the Appendix of the final version of our paper.\n\n**Question 1: Why only consider these three control methods.** Our research is based on currently common visual text generation networks such as AnyText [1] and TextDiffuser [2]. These methods typically require glyph images for guidance. Since existing methods do not consider some unique properties of text, we investigated the impact of this control information. Additionally, *we examined the influence of the same control information from three different aspects, rather than studying three control methods.*\n\n**Question 2: Fourier transform.** Our motivation for employing FEC is detailed in Sections 1, 3.1.1, and 3.3 of the paper. We analyze it more clearly below. **1)** As shown in Figure 2, general ControlNet conditions primarily affect macroscopic styles and edges and minor texture inaccuracies are acceptable, but *visual text generation requires precise control over textures to avoid content errors and unrealistic details.* **2)** Text glyph conditions are sparse, with most areas being black, acting as noise in standard spatial convolutions. *The Fourier transform filters frequency*, enhancing attention to specific components in detail-rich areas of glyph images. **3)** Direct convolution in the spatial domain is limited by the receptive field, while each point in the frequency domain captures global information of the same frequency. *Convolution in frequency domain enables global interactions among similar frequency components.*\n\n**Overall, while such an approach is unnecessary for general generation, it is crucial for text generation.** Experimental results in Table 1 further confirm the necessity of utilizing the Fourier transform.\n\n**Question 3: Simultaneously global and detail process.** **1)** Our objective is to *investigate the differences between early and late control stages*, as compared to general scene images. This guides us to employ some strategies to promote the model’s learning. **2)** We have implemented a method that combines both local and global control, using MLP to generate control factors that weight both components. However, challenges in balancing these control factors during gradient descent led to noticeable oscillations in training, ultimately resulting in degraded generation quality. **3)** Using both global and local modules would **double the training cost** and significantly decrease the model's speed, thereby limiting its generalization capability. Consequently, the two-stage method proposed in this paper emerges as the optimal solution.\n\n**Question 4: More ablation studies.** **1)** The ablation study without the Fourier block has been conducted in Table 1, lines 1 and 2. Results show that our FEC significantly improves generation quality, especially for Chinese text, due to its complex edges and details. **2)** The ablation study considering both global and detailed control simultaneously is shown below. This model requires a weighting mechanism to balance the two conditions, posing a challenge that we discussed in detail in our response to question 3.\n\n||English|Chinese|\n|-|:-:|:-:|\n|Two stage model|60.18|61.42|\n|One stage model|58.64|59.56|\n\n**Question 5: Characters frequency.** By training with the glyph conditions, the model has developed a capacity to **adhere to the glyph conditions**. For characters that are infrequently represented in the dataset, the model can still generate them effectively when the glyph condition is provided. The model even demonstrates some generation capability for languages that are not present in the training set. We will add relevant visual results in the Appendix.\n\nTo further support this, we calculated the character generation accuracy of our model for both high-frequency and low-frequency characters, as shown in the table below. The difference in accuracy is not significant.\n\n||high-frequency characters|low-frequency characters|\n|-|:-:|:-:|\n| Accuracy|61.95|58.41|\n\n**Question 6: Failure cases.** Our model encounters failure cases in generating small text due to VAE limitations, which is not the focus of this paper (detailed in Section 5). Some text generation errors also stem from poor-quality training data. Despite our data being of the highest quality, it contains some incorrect detections and captions, limiting the model's capabilities. We have recently applied stringent filtering methods to improve data quality, which can significantly enhance performance. We will release the entire dataset and code after the paper is accepted.\n\n**Question 7: Dataset.** Many recent works, such as LLaVA [3], propose that the quality of training data is essential for a model, even if the quantity is small. We believe it is highly significant to construct high-quality datasets, even if they are relatively small. Additionally, our model demonstrates superior performance on small datasets. When the amount of data is increased, the model can achieve even greater performance. **However, scaling up is not our primary goal.**\n\n**Question 8: Scene generation cases.** We list more generated results in the Appendix. As shown in Figure 12, there are many scene cases. We can also generate realistic images.\n\n[1] Tuo Y, Xiang W, He J Y, et al. Anytext: Multilingual visual text generation and editing[C]//Thirty-seventh Conference on Neural Information Processing Systems. 2023.\n\n[2] Chen J, Huang Y, Lv T, et al. TextDiffuser: Diffusion Models as Text Painters[C]//Thirty-seventh Conference on Neural Information Processing Systems. 2023.\n\n[3] Liu H, Li C, Li Y, et al. Improved baselines with visual instruction tuning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 26296-26306.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uBEn8Eno4f", "reviewer": "Reviewer_Z5LF", "summary": "This paper introduces a novel framework to enhance the quality of multilingual text image generation and editing by optimizing control information. The authors investigate the impact of control information from three perspectives: input encoding, the role at different stages, and output features. They find some insight conclusion and propose a framework (TextGen) employs Fourier analysis to emphasize relevant information and reduce noise, uses a two-stage generation framework to align the different roles of control information at various stages. Furthermore, they introduce an effective and lightweight dataset for training. The method achieves state-of-the-art performance in both Chinese and English visual text generation.", "questions": "1. Could TextGen generate other languages, such as Korean?\n2. How about the diversity of textgen's generation of the same prompt? The authors can put some visualization examples.", "limitations": "refer to weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper investigate the influence of control information, which can greatly inspire future works of the community. In summary, this paper is well organized, with reasonable motivation and insights.\n2. The use of Fourier analysis to enhance input and output features, along with the two-stage generation framework, offers innovative methodological approaches in the field of text image generation.\n3. The creation of a new lightweight yet effective dataset, TG2M, provides a valuable resource for training in visual text generation and editing.", "weaknesses": "1. To my best knowledge, there exist other benchmarks such as the benchmark in TextDiffuser[1]. Although these benchmarks may not be of good quality, it would be better to compare TextGen with other works on these benchmarks or construct a high-quality benchmark based on TG-2M.\n2. The data building process is not described in detail. Even though there are some differences (the recaption using Qwen-VL, the data selection), it looks a bit similar to AnyWord-3M and the author needs to describe the differences in paper or Appendix.\n\n[1] Chen, Jingye, et al. \"TextDiffuser: Diffusion Models as Text Painters\" NeurlPS 2023\n[2] Tuo, Yuxiang, et al. \"AnyText: Multilingual Visual Text Generation And Editing\" ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their positive attitude towards our paper and for their valuable comments 😊. Below are our rebuttal and discussion of the questions.\n\n**Question 1: The evaluation.** We conducted evaluations on the AnyWord benchmark, which is the highest quality benchmark. Your suggestion is valuable. We recently employed more stringent filtering methods to process and clean the data, resulting in higher-quality datasets. Such data can significantly enhance generation performance, and we have constructed a new benchmark from it. We will fully release our entire dataset and code without reservation after the paper is accepted.\n\n**Question 2: The data building pipeline.** We recaptioned the data using both BLIP and Qwen-VL. Due to the input length restriction in the CLIP text encoder, the diffusion model requires conditions shorter than 77 tokens. Qwen-VL often generates longer captions and sometimes hallucinates. Therefore, we generated the initial captions using BLIP, identified failure cases, and then recaptioned those using Qwen-VL. Additionally, we filtered the data using aesthetic scores and a watermark detection model.\n\n**Question 3: More languages.** Our training set only contains English and Chinese cases, which enables our model to generate high-quality English and Chinese visual texts. Moreover, by training with the glyph conditions, the model has developed a capacity to adhere to the glyph conditions. For other languages such as Korean and Japanese, the model can still generate them effectively in some cases when the glyph condition is provided. But as for more difficult languages such as Arabic, it is hard for our model to generate. We will try to construct a more comprehensive multi-lingual model in the future.\n\n**Question 4: The diversity.** Similar to other diffusion-based models, our model can generate diverse results when given different random seeds. Due to rebuttal restrictions, we are unable to provide visual examples here. We will include visual results related to diversity in the Appendix of the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MKchkOY6Gq", "reviewer": "Reviewer_dBLt", "summary": "This article investigates an intriguing issue: how controlling information affects performance in the process of text image editing and generation. Through a series of observations and experiments, the authors extract valuable insights. Building upon these findings, they craft a novel framework that demonstrates impressive performance in both the generation and editing of text images.", "questions": "1.\tWithin an equitable comparison framework, the performance improvement in TextGen on the FID metric is less pronounced compared to its advances in recognition accuracy. Are there any further analyses on this issue?\n2.\tThe author notes, \"Although TG2M... it is highly effective for training and achieves superior performance.\" Is there any quantitative evidence available to support this claim?", "limitations": "The authors list the potential limitations and analyze the reasons briefly.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe experimental design, performance analysis, and underlying motivations are presented in a systematic manner with high-quality writing. Some findings offer valuable guidance for the community.\n2.\tThe performance gain in recognition accuracy is substantial.\n3.The dataset introduced holds significant value.", "weaknesses": "1.\tThere is an absence of detailed quantitative analysis for the internal workings of the designed modules.\n2.\tAlthough some perspectives are effectively illustrated through visualizations, they lack the backing of quantitative analysis.\n3.     Typos in references, such as Diffute: Universal text editing diffusion model. Trivial writing of the experimental section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your valuable comments and positive attitude to our paper. Below are my rebuttal and discussion for these questions 😊.\n\n**Question 1: Internal workings.** The pipeline contains two control models and a diffusion process. As shown in Figure 4, the early timestamps in diffusion process using global control model and the latter timestamps in diffusion process using local control model.\n\n**Question 2: Quantitative analysis for perspectives.** For the control input, the first line in Table 1 presents a general diffusion model using ControlNet, which performs worse than our proposed methods. This indicates that general diffusion cannot generate accurate text. More visualizations will be added in the Appendix in the final version of our paper. For control at different stages, we conducted an additional experiment by removing control at various stages. As shown in the table below, removing early control information results in a slight performance decrease, whereas removing late control information leads to a significant performance decline. This supports our conclusion that early control information is crucial for generating coherence and realism, while late control information ensures text detail and accuracy. For the control output, we calculated the frequency distribution shown in Figure 5 and performed an ablation study presented in Table 1 (lines 4 and 5).\n\n|                         |  English  |  Chinese  |\n|-------------------------|:---------:|:---------:|\n| Baseline                |   60.18   |   61.42   |\n| Removing early control  |   54.36   |   53.28   |\n| Removing late control   |   34.84   |   23.42   |\n\n**Question 3: Incorrect citation.** We apologize for the incorrect citation, which may have occurred due to an error during the copying from Google Scholar. We will correct this mistake in the final version of the paper.\n\n**Question 4: The FID score.** The FID score measures the feature similarity between generated images and target datasets. We evaluate the FID score using the AnyWord-FID benchmark. 1) A higher FID score does not necessarily indicate lower visual quality of the generated images. This conclusion has been noted in earlier works [1]. Our generated images exhibit greater diversity, such as word art, which is scarce in the AnyWord-FID dataset, resulting in a higher FID score, as described in [1]. 2) The images in the AnyWord-FID benchmark are selected from the AnyWord training set, while we use a different training set, making it reasonable for AnyText to achieve a better FID score.\n\n**Question 5: The evidence for dataset effectiveness.** Our TG2M dataset contains only 2.5M images, which is significantly smaller than other training sets. As shown in Table 2, our baseline model is a general diffusion model similar to AnyText [2], trained for only 5 epochs, in contrast to other methods trained for 10 epochs. It can be observed that using our TG2M dataset achieves performance comparable to other methods despite using fewer data and fewer epochs. Recently, we employed more stringent filtering methods to process and clean the data, resulting in higher-quality data. Such data can further enhance generation performance. We will fully release our entire dataset and code without reservation after the paper is accepted.\n\n[1] Li T, Chang H, Mishra S, et al. Mage: Masked generative encoder to unify representation learning and image synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 2142-2152.\n\n[2] Tuo Y, Xiang W, He J Y, et al. Anytext: Multilingual visual text generation and editing[C]//Thirty-seventh Conference on Neural Information Processing Systems. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a dataset and approach for dual-lingual image-to-text generation and editing.  This paper was reviewed by three experts, who overall lean towards acceptance.  While some questions as to the novelty of the approach remain, it can at least be seen as an application that may be relevant to some researchers.  The authors are encouraged to take the reviewers comments into account when creating their final version.
Decision: Accept (poster)