How Control Information Influences Multilingual Text Image Generation and Editing?

Authors: boqiang  zhang
zuan  gao
yadong  qu
hongtao  xie
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-27 12:36:00

Abstract: Visual text generation has significantly advanced through diffusion models aimed at producing images with readable and realistic text. Recent works primarily use a ControlNet-based framework, employing standard font text images to control diffusion models. Recognizing the critical role of control information in generating high-quality text, we investigate its influence from three perspectives: input encoding, role at different stages, and output features. Our findings reveal that: 1) Input control information has unique characteristics compared to conventional inputs like Canny edges and depth maps. 2) Control information plays distinct roles at different stages of the denoising process. 3) Output control features significantly differ from the base and skip features of the U-Net decoder in the frequency domain. Based on these insights, we propose TextGen, a novel framework designed to enhance generation quality by optimizing control information. We improve input and output features using Fourier analysis to emphasize relevant information and reduce noise. Additionally, we employ a two-stage generation framework to align the different roles of control information at different stages. Furthermore, we introduce an effective and lightweight dataset for training. Our method achieves state-of-the-art performance in both Chinese and English text generation. The code and dataset are available at https://github.com/CyrilSterling/TextGen.

Main paper: 
 introduction: 
 With the development of diffusion-based generative models [9, 26, 20] and image-text paired datasets [23, 8] , significant improvements have been made in the quality of image generation. Given the prevalence of text in natural scenes (e.g., posters, slides, signs, book covers, etc.), generating images containing text accurately and reasonably is crucial.Recently, several methods have been proposed to address the generation of high-quality visual text images [33, 14, 4, 28] . Among these, ControlNet-based approaches show strong potential [33, 28] , enabling flexible multilingual visual text generation, text position control, and easy integration into existing pre-trained diffusion models. Current methods directly utilize ControlNet [36] for text generation control, using a global glyph image of a standard font as the condition (as shown in Figure 1 ). However, achieving accurate and robust control remains challenging due to the complex and fine-grained structure of characters. Hence, we pose a meaningful question: How does control information influence multilingual text image generation?To address the above issue, we investigate the impact of control information on visual text generation from three perspectives, as illustrated in Figure 1 . For the input of control information, the current model uses a glyph image to guide the generation of accurate text textures. However, unlike general ControlNet inputs such as Canny edges, depth, or M-LSD lines, text glyph images have unique properties: 1) Glyph images have areas of high information density concentrated within specific regions, with the rest being meaningless backgrounds. 2) Generation in text region is fine-grained, but extracting fine-grained features from glyph images using standard convolution methods is challenging (further discussed in Sec. 3.1.1). These properties limit the performance. For the control information in different time steps, current models follow ControlNet [36] by using fixed control, but they often overlook the role of different time steps in the generation process. We further explore the impact of control at various steps in Sec. 3.1.2. Control in early steps influences both text and non-text regions, ensuring that the background reasonably matches the text. Control in late steps still plays a significant role in modifying mistakes, which is different from the general generation [2, 15] . For the output of control feature, these features are injected into the U-Net decoder, which receives three types of features: base, skip, and control. Each of these components differs in the frequency domain, which explains their respective roles (discussed in Sec. 3.1.3) . Balancing these components during inference is crucial. Overall, we explore the influence of control information in text generation, raising several critical questions essential for advancements in visual text generation.Based on the analysis above, we introduce TextGen, a novel framework aimed at enhancing the quality of control information. Specifically, for control input, we introduce a Fourier Enhancement Convolution (FEC) block to extract spatial and frequency features from the glyph control image. This operation can enhance the perception of useful regions and edge textures. For the output control feature, we introduce a frequency balancing factor to adjust the frequency information among the features during inference. For the control information in different stages, we propose a two-stage framework for coarse-to-fine generation. This framework trains the first-stage model for global control and the second-stage model for detail control. Based on the two-stage framework, we naturally propose a novel inference paradigm for unifying text generation and editing tasks. Furthermore, as current datasets for visual text generation are large-scale and noisy, we construct a lightweight but high-quality dataset for effective training (details provided in Appendix A). Unlike previous works, we were the first to delve into control information in the visual text generation task. Our framework enhances the quality of detail generation while elegantly achieving unified generation and editing tasks. To summarize, our contributions are as follows:• We conducted an in-depth study and discussion on the impact of control information in visual text generation tasks. Our findings can inspire more future research in this area. • We propose a framework for multilingual visual text generation and editing based on our analysis, which contains a two-stage pipeline and a Fourier enhancement in both training and inference. This framework achieves state-of-the-art performance. • We construct an open-source effective and lightweight dataset for the training of visual text generation and editing.2 Related Works 
text generation: 
 With the advancement of denoising diffusion probabilistic models [9, 26] and text-to-image generation [22, 2, 20] , it has become possible to generate high-quality images. However, visual text generation remains challenging due to the need for fine-grained alignment and character detail representation. Recent studies, such as Imagen [22] and eDiff-I [2] , have focused on improving text generation from the perspective of text encoders. They found that T5-based encoders [19] outperform CLIP text encoders [18] . GlyphDraw [14] presents a robust baseline for visual text generation by incorporating conditions. It utilizes a glyph image representing a single word as the content condition and a mask image as the positional condition. However, GlyphDraw is limited to generating only one text line per image. GlyphControl [33] further introduces a ControlNet-based framework that employs a global glyph image as the condition, providing both glyph and positional information, achieving outstanding generation performance. Glyph-ByT5 [13] fine-tunes a T5 language model for paragraph-level visual text generation, achieving remarkable performance in dense text generation. However, it is restricted to producing text in English. We propose an effective multilingual framework by controlling information enhancement in a ContorlNet-based framework.Scene text editing aims to replace text in a scene image with new text while preserving the original background and style. Early approaches focused on generating text on cropped images, allowing for more precise text area generation [34, 10, 11, 24, 30, 17] . SRNet [30] was the first to divide the editing task into three sub-processes: background inpainting, text conversion, and fusion, which inspired subsequent works [17, 21, 32] . Although these methods achieved excellent generation performance, integrating the cropped text area into the original scene images proved challenging such as edge inconsistency. Recently, leveraging the diffusion model, some approaches have conducted generation on complete scene images directly, without decomposing the task into sub-processes. DiffUTE [3] proposes a concise framework for directly generating the edited global scene image using the diffusion process. However, solely focusing on the complete editing task limits the practicality and generalization of the model.Due to the similarity between visual text generation and editing tasks, developing a unified framework to jointly address these tasks is meaningful. TextDiffuser [5, 4] employs a mask to indicate areas requiring editing, ensuring multitasking uniformity. During the generation task, the mask remains empty, while during the editing task, it preserves areas that do not require editing. Additionally, TextDiffuser introduces a layout generator to design the distribution of text lines. Similarly, Any-Text [28] adopts a comparable approach to maintain the uniformity of two tasks and further enhances generation quality with a text embedding module and perceptual loss. Building on our explorations, we propose a two-stage model and design a novel inference paradigm to achieve multitasking unity. 3 Method 
control information input: 
 In recent works, the inputs of the control module are the glyph image and position image, which provide the texture and position condition for text regions. However, different from general ControlNet conditions (e.g., Canny edge, M-LSD lines, depth, etc.), text conditions have distinct characteristics. As illustrated in Figure 2 , general ControlNet conditions typically influence only macroscopic coarse-grained style and global edges, and some incorrect minor texture generation is considered acceptable. However, small differences in texture details can lead to content errors or unrealistic and unreasonable textures in visual text generation. Therefore, using the general ControlNet poses challenges in controlling detailed textures and fine-grained handwriting. Moreover, the text glyph condition concentrates only on certain regions, making it a sparse condition unlike other general conditions. This characteristic causes the convolution-based ControlNet encoder to introduce noise in empty areas when extracting features, which interferes with the text area's features and affects the correct allocation of attention between edge and background areas. Enhancing the ControlNet encoder's perception of locally useful details and edge information is essential for improving text image generation. Meanwhile, the characteristic of having high information density in local areas suggests that we can seek solutions in the frequency domain.Some studies on general diffusion [15, 2] have suggested that control information in the later stages of the denoising process contributes little to the diffusion model. However, due to the specific nature of text, where text strokes constitute detailed information, we find that control in the later stages remains crucial. As shown in Figure 3 (a), omitting control information in the later stages often results in incorrect text content generation. The control module in these later stages corrects such errors, ultimately leading to high-quality images. This finding indicates that performing the editing task at a late time step is reasonable, as it mitigates the performance impact of joint multi-task training.Furthermore, we investigate the role of early control. As shown in Figure 3 (b), the control information in early steps has a significant impact on the global semantics of the generated image, aiding in the alignment of text areas with the global scene. Without the control information in the early steps, the generated text regions appear unreasonable and do not match the background. Notably, even though only glyph and position images provide control information, the early stages still strongly influence the generation of non-text regions (the non-text areas may be totally different). Therefore, the control information of different stages should be adapted according to these findings. We divide the control module into global and detail stages (Fig. 4 ), each with distinct parameters. In the global control stage, we expect that the control information can affect the entire image, while in the detail control stage, we incorporate a mask to guide the model in optimizing local details. Furthermore, the detail stage can act as a refiner in text generation and as an editor in text editing. During inference, the output of the control module is injected into the base diffusion process. Each layer in the U-Net decoder can be formulated as:Output )UHTXHQF\ 5HODWLYH/RJ$POLWXGH VNLSIHDWXUH FRQWUROIHDWXUH EDVHIHDWXUH IXVLRQIHDWXUHF i = D i (F i-1 , S i-1 , C i-1 ), where D i is the i-th layer, F i is the output feature of i-th layer, S i and C i represent the skip feature and the control feature of i-th layer, respectively. These three parts of the input represent different types of information. Following FreeU [25] , we further investigate the balance among these three components. As shown in Fig. 5 , we visualize the Fourier relative log amplitudes of F, S and C. It can be observed that the skip feature contains more high-frequency information than the base feature, which may infer denoising (the same conclusion with [25] ). Therefore, there is a need for balancing between the base feature and the skip feature. Furthermore, compared with the fusion feature, the control feature has more high and low-frequency information, with a greater gap at low frequencies than at high frequencies. However, since we only aim to control the texture, which belongs to high-frequency information, the low-frequency information needs to be suppressed.Based on the motivations and discussions above, we propose a novel framework named TextGen. The pipeline of our TextGen is illustrated in Figure 4 Additionally, the input layer of both blocks is a global perception operation, achieved through convolution with a large kernel size. This is because the text contains global semantic information, and the general convolution-based encoder has a local receptive field that limits information interaction.The detail control stage receives three pieces of control information: a position image, a glyph image, and a masked image. Unlike the global control stage, the detail control stage incorporates the masked image as one of its inputs, aiming to provide background information. This stage is designed to generate specified texts at designated positions while keeping the background consistent with the masked image. The module's output is multiplied by a position mask, enabling the model to focus on modifying the detailed texture of the text area. It's worth noting that the mask is only applied in the last two layers of the U-Net decoder (for feature sizes greater than or equal to 32). This is because the first two layers (feature sizes less than 32) primarily handle global information, which contributes little to the detailed texture. Moreover, by retaining the first two layers without applying the mask, we ensure that the background of the output image remains consistent with the masked image while modifying inconsistent areas.Discussion: The frequency enhancement block performs convolution operations in both spatial and frequency domains, offering two main benefits: 1) The Fourier transform of visual features provides global information about the glyph image, overcoming the limitations of the receptive field in spatial convolution.2) The glyph image contains both localized detail-rich areas and meaningless backgrounds. Convolution in the frequency domain acts as a frequency filter, allowing for the adjustment of attention to different frequency components. This facilitates the extraction of useful information and mitigates the interference of irrelevant information from a global perspective.In our framework, each layer of the U-Net decoder receives three parts of information: the backbone feature F, the skip feature S, and the control feature C. As discussed in Sec. 3.1.3, there is a need for balancing among three parts during inference. Therefore, we propose a Fourier enhancement method as formulated as follows:EQUATIONwhere F and F -1 represent FFT and IFFT, ⊙ denotes the element-wise multiplication. α and β are balancing factors for the control feature and the base feature. γ is a modulation factor in the frequency domain. Although enhancing high-frequency information is desired, directly doing so will introduce noise and reduce generation quality. Therefore, we suppress low-frequency information to emphasize high-frequency components. This is achieved using a scalar s to suppress low-frequency information as follows:EQUATION(2)Based on our model and analysis, we propose a novel inference paradigm for task unification. For the image generation task, random noise is inputted into the global control stage for the early T /2 steps and the detail control stage for the remaining steps. For the text editing task, the original image is first noise-added to 80% time-step, which maintains most of the global style and texture while destroying the original text content. Then the new text content is first generated using the global control stage until the T /2 time step, and the remaining time steps use the detail control stage to modify the details. Since the control input of the detail control stage contains the masked original image, it can restore the background information that was destroyed during the noise addition, and at the same time optimize the new text content at the specified location. 
datasets: 
 Recently, several works have introduced datasets for text generation and editing tasks. TextDiffuser [5, 4] introduced a dataset named MARIO-10M, comprising approximately 10 million images annotated with bounding boxes and content of text regions. AnyText [28] proposed a benchmark named AnyWord for evaluation. However, training on 10 million images requires significant computing resources. Therefore, we introduce TG2M, a multilingual dataset sourced from publicly available images including MARIO-10M [5] , Wukong [8] , TextSeg [31] , ArT [6] , LSVT [27] , MLT [16] , ReCTS [37] . Although TG2M contains significantly fewer images, it is highly effective for training and achieves superior performance. The dataset will be detailed in the Appendix A.In our implementation, the diffusion model is initialized from SD1.52 , and our code is based on diffusers 3 . The text embedding module follows AnyText [28] . We train our model on the TG2M dataset using 8 NVIDIA A40 GPUs with a batch size of 176. Our model converges rapidly and requires only 5 epochs of training. The learning rate is set to 1e-5. Following previous generation and recognition works [35, 29, 28, 7] , we set the maximum length of each text line to 20 characters and the maximum number of lines in each image to 5. During inference, the Fourier balance factors α, β, and s are set to 1.4, 1.2, and 0.2, respectively.We evaluate our model on the AnyWord [28] benchmark. We use DuGuangOCR4 to recognize the text region and measure performance using sentence accuracy (ACC), Normalized Edit Distance (NED), and Fréchet Inception Distance (FID). During inference, the settings (random seed, control strength, etc.) are consistent across all experiments.Owing to resource constraints, following AnyText [28] , we randomly select 200k images (40k English and 160k Chinese) from TG2M as the training set for ablation. The results are shown in Tab. 1. Compared to other methods, our approach uses less data while outperforming the state-of-the-art, as shown in Table 2 . For a fair comparison, all methods are evaluated under the same settings. The performance of both the baseline and TextGen is assessed using four random seeds, with the final metrics reported as averages and standard deviations. Our method achieves a 9.1% gain in sentence accuracy for English texts and a 2.9% gain for Chinese texts compared to our baseline trained on TG-2M. Notably, other approaches require large amounts of training data and employ perceptual loss to enhance performance, which is training-inefficient. Our method, in contrast, does not require additional losses and converges faster, making it easier to train. Besides, we compute the FID on the AnyWord-FID [28] benchmark. Our FID scores achieve comparable performance but not the best. The higher FID score does not necessarily imply the lower visual quality of the generated images. Our generated images demonstrate greater diversity and there is a distribution gap between our training set and AnyWord. This issue is discussed in more detail in Appendix C.The qualitative comparison is shown in Fig. 7 . Our TextGen produces high-quality images with text in various scenarios and excels in generating artistic text with a wide range of visually appealing styles.For Chinese text, as demonstrated in Fig. 8 , TextGen generates more realistic and readable results, particularly in smaller texts. Finally, Fig. 9 illustrates the editing capabilities of our model, which can edit various text styles and contents using the proposed inference paradigm.  
limitations: 
 We propose a novel diffusion-based network for multilingual text generation and editing, which demonstrates robust performance. Our network excels at generating high-quality scene images with text. However, the framework based on latent diffusion presents certain limitations. 1) The VAE employed in latent diffusion restricts the performance of fine-grained texture generation, particularly for complex texts. Because the diffusion process operates in the latent feature space, the VAE decoder struggles to generate small or complex texts. Consequently, our method is unable to generate such images. This issue can be addressed by generating each local sub-region separately.2) The text condition controls the content of the generated image. However, the CLIP text encoder has limited ability to comprehend text, resulting in performance limitations. To resolve this issue, we can pre-train the diffusion model with a large language model serving as the text condition encoder.Moreover, generating false text can contribute to the spread of misinformation, potentially resulting in serious consequences. It is hoped that this technology will be used responsibly, fostering a healthy and ethical academic environment. 
conclusion: 
 Based on ControlNet, current visual text generation has made significant progress. In this work, we build on recent studies using ControlNet to investigate how control information influences visual text generation from three perspectives: control input, control at different stages, and control output. Through experiments and discussions, we derive several key conclusions. Based on our analysis, we propose a novel visual text generation framework that improves control information utilization, which surpasses the state-of-the-art performance. We believe the insights we gained about control information can inspire future research in the community. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: The work is about text generation, which does not have societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: This paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: To train the visual text generation model, we constructed a dataset called TG-2M, which contains rich textual data. The images in this dataset are sourced from existing open-source datasets, including MARIO-10M [5] , Wukong [8] , ArT [6] , LSVT [27] , MLT [16] , ReCTS [37] , TextSeg [31] . Following AnyText [28] , we filter the images with some rules, which can devided into three steps. Specifically, the filtering rules of the first step include:• The images of width smaller than 256 will be filtered out.• The images with aspect ratio larger than 2.0 or smaller than 0.5 will be filtered out.After step 1, we use PP-OCR 5 to detect and recognize the texts in these images. Then, we undergo the second filtering step:• The images with more than 10 texts will be filtered out.• The images with more than 3 small texts will be filtered out. The small text refers to horizontal text with a height of less than 30 pixels or vertical text with a width of less than 30 pixels. The orientation of the text is determined by the aspect ratio of the text bounding box, with an aspect ratio less than 0.5 considered vertical text. • Images containing more than 3 text instances with recognition scores below 0.7 will be filtered out.We use BLIP-2 [12] and Qwen-VL [1] to recaption the images. First, we generate initial captions using BLIP-2. Because some initial captions are low-quality, we then modify these captions using Qwen-VL. The low-quality captions are defined as those containing meaningless text or having low CLIP similarity with the reference image. This process is necessary because some captions generated by BLIP-2 are meaningless, as shown in Fig. 10 . Additionally, although Qwen-VL's captions are of high quality, many of them are quite lengthy, which can affect the understanding of the CLIP text encoder in the diffusion model. We propose a two-stage generation framework that achieves detail optimization and task unification. However, this enhancement does not notably improve recognition accuracy in our ablation study. This is because: 1) The first stage already allows for some detailed modifications. 2) Our ablation study was conducted on a subset of the TG-2M dataset. The second stage enhances texture details, but its performance is limited with insufficient data. On the complete dataset, the two-stage framework demonstrates better performance, as detailed in Table 4 . The Fréchet Inception Distance (FID) metric evaluates the distribution gap between generated images and target images. A higher FID indicates a larger distribution gap. We evaluate the FID score on the AnyWord [28] benchmark, which provides a subset of images for this purpose. Since the AnyWord FID benchmark is derived from the AnyWord training set, it is reasonable for AnyText to achieve a better FID score due to the distribution gap between our TG-2M and AnyWord. Additionally, our TextGen can generate more artistic texts, resulting in a diversity of distribution. Therefore, a lower FID score does not necessarily imply a lower visual quality of the generated images.Based on ControlNet, current visual text generation has made significant progress. We further investigate the control information in ControlNet-based visual text generation tasks and draw several conclusions. Future performance improvements can be achieved through three approaches: 1) Construct high-quality datasets, as current datasets still contain incorrect labels and unreasonable captions. 2) Enhance the text embedding module. Leveraging large language models (LLMs), we can We present additional qualitative results generated by TextGen in Fig. 12 . TextGen produces realistic images with coherent and readable text. Additionally, TextGen is capable of generating artistic text for applications such as logos, posters, and clothing design.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We will release our code and dataset after accepted. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: The research conducted in the paper conform.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://huggingface.co/runwayml/stable-diffusion-v1-5https://github.com/huggingface/diffusershttps://modelscope.cn/models/iic/cv_convnextTiny_ocr-recognition-general_damohttps://github.com/PaddlePaddle/PaddleOCR 


Decision: This paper presents a dataset and approach for dual-lingual image-to-text generation and editing.  This paper was reviewed by three experts, who overall lean towards acceptance.  While some questions as to the novelty of the approach remain, it can at least be seen as an application that may be relevant to some researchers.  The authors are encouraged to take the reviewers comments into account when creating their final version.

Reviews: review: paper_summary: paper_summary: This paper analyzes the issue of control in image generation models. Specifically, the article addresses three aspects: input control information, the impact of control information at different stages, and output control information. The model was optimized for two tasks: text-to-image and image editing. Using a method similar to FreeU, the paper conducts Fourier frequency domain analysis on the input and output features and proposes a two-stage generation model based on previous findings.
strength_weakness: strength_weakness: 1.Some models lack detailed theoretical analysis, which makes the purpose of the model unclear.
2. It is not clear what the task of this paper issue. Please point it out in a prominent position in the first chapter.
3. The layout is a bit messy. For example, should Table 2 be placed above Table 1?
questions: questions: 1.Will the dataset be released? If so, please address the privacy, copyright, and consent of the data.

2.The convolution theorem states that the Fourier transform of a function's convolution is the product of the Fourier transforms of the functions. In other words, convolution in one domain corresponds to multiplication in another domain, such as time-domain convolution corresponding to frequency-domain multiplication. Can the FEC network in the article be considered as performing a simple point-wise multiplication operation?

3. I have questions about the effectiveness of FEC. The article only shows that using FEC is better through ablation experiments, but I have two concerns: 1) The improvement for English is not significant, while it is more noticeable for Chinese. Is the main reason for this that the model was fine-tuned on Chinese? 2) When using methods like depth control, the output might not match the control, but the output could still be reasonable. Using glyph images for control requires a strong match between the output and the input text. From a human observation perspective, glyph control might seem more precise, but for model training with MSE loss, both approaches could be similar.

4. Section 4.3 does not have accompanying text?

5. Why does convolution perform poorly in understanding text? Shouldn't there be some small experiments to demonstrate this? Alternatively, could it be explained through some visualization analysis or derivation process?

6. Figure 4, which serves as the overall pipeline diagram, does not seem to reflect the inference structure discussed in Chapter 3.5.
limitations: limitations: The article does not specifically discuss the potential negative societal impacts. For example, generating high-quality images guided by text might facilitate the creation and spread of fake news. Additionally, using more complex models could lead to increased resource consumption, among other issues.
paper_summary: paper_summary: This study explores the advancement of visual text generation using diffusion models within a ControlNet-based framework, focusing on the impact of control information. It examines input encoding, the role of control information during different stages of the denoising process, and the resulting output features. The authors propose TextGen, a method that aims to enhance generation quality by optimizing control information with Fourier analysis and a two-stage generation process.
strength_weakness: strength_weakness: 1. The method lacks novelty, the writing of the paper is unclear, and the experimental results are insufficient. The experimental analysis does not provide new insight into the field.
2. The paper's analysis of control factors for text image generation is limited to three control conditions, which may not be the most effective. For example, character embeddings could be used as input, sentence prompts through an LLM could be used, or text coordinate encodings could be used as control conditions.
3. The necessity of the FEC module is not very clear, as it does not specifically target text image scenarios. Additionally, the computational efficiency of FFT and IFFT needs to be considered.
4. Why are global control and detail control split into two stages? Would it not be possible to consider both global and detail control simultaneously during the whole diffusion process?
5. The improvement of this method over existing methods is not obvious, especially in terms of visual and perceptual quality. Training on a small dataset also affects the method's generalizability. For example, can it still correctly generate text that appears infrequently or not at all in the proposed dataset?
6. The results lack demonstrations of “scene” text image generation; most images resemble text printing and have less realism compared to Anytext. This may be due to dataset bias, as shown in Figure 11. Thus, a small dataset is not a contribution of this paper and could be considered a drawback for large models.
7. The ablation experiments are insufficient, and the effectiveness of the modules is not convincingly demonstrated.
questions: questions: 1. Why only consider these three control methods.
2. What is the reason and necessity for introducing Fourier transform.
3. Why can't both global and detail control be considered simultaneously during the diffusion process? Wouldn't it be possible to use a learnable module to control the weights of these two controls. The two seperated control stages proposed in this paper is clearly not optimal.
4. It is recommended to add more ablation experiments, such as without the Fourier module, and considering both global and detail control simultaneously. More visualization results should be provided, such as visualizations of the ablation experiments.
5. How does the method perform on characters that appear infrequently in the dataset.
6. It is recommended to include failure cases and analyze the reasons. Additional results for text image editing should also be provided.
7. Please adequately address the concerns raised in the weaknesses section.
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a novel framework to enhance the quality of multilingual text image generation and editing by optimizing control information. The authors investigate the impact of control information from three perspectives: input encoding, the role at different stages, and output features. They find some insight conclusion and propose a framework (TextGen) employs Fourier analysis to emphasize relevant information and reduce noise, uses a two-stage generation framework to align the different roles of control information at various stages. Furthermore, they introduce an effective and lightweight dataset for training. The method achieves state-of-the-art performance in both Chinese and English visual text generation.
strength_weakness: strength_weakness: 1. To my best knowledge, there exist other benchmarks such as the benchmark in TextDiffuser[1]. Although these benchmarks may not be of good quality, it would be better to compare TextGen with other works on these benchmarks or construct a high-quality benchmark based on TG-2M.
2. The data building process is not described in detail. Even though there are some differences (the recaption using Qwen-VL, the data selection), it looks a bit similar to AnyWord-3M and the author needs to describe the differences in paper or Appendix.

[1] Chen, Jingye, et al. "TextDiffuser: Diffusion Models as Text Painters" NeurlPS 2023
[2] Tuo, Yuxiang, et al. "AnyText: Multilingual Visual Text Generation And Editing" ICLR 2024
questions: questions: 1. Could TextGen generate other languages, such as Korean?
2. How about the diversity of textgen's generation of the same prompt? The authors can put some visualization examples.
limitations: limitations: refer to weaknesses.
paper_summary: paper_summary: This article investigates an intriguing issue: how controlling information affects performance in the process of text image editing and generation. Through a series of observations and experiments, the authors extract valuable insights. Building upon these findings, they craft a novel framework that demonstrates impressive performance in both the generation and editing of text images.
strength_weakness: strength_weakness: 1.	There is an absence of detailed quantitative analysis for the internal workings of the designed modules.
2.	Although some perspectives are effectively illustrated through visualizations, they lack the backing of quantitative analysis.
3.     Typos in references, such as Diffute: Universal text editing diffusion model. Trivial writing of the experimental section.
questions: questions: 1.	Within an equitable comparison framework, the performance improvement in TextGen on the FID metric is less pronounced compared to its advances in recognition accuracy. Are there any further analyses on this issue?
2.	The author notes, "Although TG2M... it is highly effective for training and achieves superior performance." Is there any quantitative evidence available to support this claim?
limitations: limitations: The authors list the potential limitations and analyze the reasons briefly.

Comments: title: Sincere Invitation to Participate in the Discussion
comment: Dear Reviewer bu9j,

We sincerely appreciate your time and feedback. Given the rush in finalizing the writing, some aspects may have caused confusion or misunderstanding. It is our priority to ensure that the rebuttal aligns with your suggestions, and we are open to further discussions to clarify any remaining questions or concerns. We would be grateful if you could improve the evaluation after reviewing our responses.

Thank you very much for your consideration.

Sincerely,

The Authors
We sincerely appreciate your timely feedback!
We sincerely appreciate your positive feedback and strong support for our work. Thank you also for suggesting the BIQA metrics; we will thoroughly explore this perspective in the future.
Thank you for your appreciation of our work.
We sincerely appreciate your timely feedback and the strong support for our work. We are committed to incorporating all of the clarifications you suggested in the next version of our paper.
Thanks for your recommendation. We will add the discussion about the potential harms arising from visual text generation in the final version of our paper.
Thank you for your response. The author addressed most of my concerns. I also read the questions and responses from other reviewers in detail. I have improved my score, but I still think that this article lacks sufficient novelty.
Thanks for the response. All my concerns have been properly solved. I thus give ACCEPT as my final score.
Thanks for the response. Given the presented aesthetic scoring and previous replies, I will raise the score. (BTW, some of the most recent advancements in Blind Image Quality Assessment (BIQA) metrics can provide new perspectives in evaluating the quality of generated images.)
Thank you for your positive response! **First**, I would like to clarify that the FID metric does assess image generation quality to some extent. However, the evaluation dataset used in my experiments is the FID subset of the publicly available AnyWord dataset, which was selected from the AnyWord training set. Consequently, AnyText naturally yields a lower FID score, so our comparisons are not really fair to us. As shown in Table 2 in our paper, our method achieves a better FID score compared to other models when AnyText is excluded. **Additionally**, since FID is widely considered an inadequate measure of image generation quality, as noted by many scholars, the question you have raised is a very valuable one and well worth thinking about. Personally, We think that aesthetic scoring offers a better evaluation method. Therefore, we employed Laion's publicly available aesthetic scoring model [1] to evaluate the images generated by our model, and we calculated the average aesthetic score of these images, as detailed in the table below.

|                  | English | Chinese |
|------------------|:-------:|:-------:|
| AnyText  | 4.27   | 5.06   |
|Ours | 4.42   | 5.18   |

Lastly, we emphasize that your question is crucial and worth deep consideration by everyone. The solution we provided is one approach, and we hope other researchers will also explore improved methods for evaluating generation quality.

[1] https://laion.ai/blog/laion-aesthetics/
Most of my concerns have been addressed. One remaining issue is that the authors pointed out that FID can not indicate the visual quality of the generated images effectively. Is there any other quantitative metric about image quality that can verify the effectiveness of TexGen?