TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control

Authors: weichao  zeng
yan  shu
zhenhang  li
dongbao  yang
yu  zhou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 05:44:09

Abstract: Centred on content modification and style preservation, Scene Text Editing (STE) remains a challenging task despite considerable progress in text-to-image synthesis and text-driven image manipulation recently. GAN-based STE methods generally encounter a common issue of model generalization, while Diffusion-based STE methods suffer from undesired style deviations. To address these problems, we propose TextCtrl, a diffusion-based method that edits text with prior guidance control. Our method consists of two key components: (i) By constructing fine-grained text style disentanglement and robust text glyph structure representation,  TextCtrl explicitly incorporates Style-Structure guidance into model design and network training, significantly improving text style consistency and rendering accuracy. (ii) To further leverage the style prior, a Glyph-adaptive Mutual Self-attention mechanism is proposed which deconstructs the implicit fine-grained features of the source image to enhance style consistency and vision quality during inference. Furthermore, to fill the vacancy of the real-world STE evaluation benchmark, we create the first real-world image-pair dataset termed ScenePair for fair comparisons. Experiments demonstrate the effectiveness of TextCtrl compared with previous methods concerning both style fidelity and text accuracy. Project page: https://github.com/weichaozeng/TextCtrl.

Main paper: 
 introduction: 
 Scene Text Editing (STE) refers to modifying the text with desired content on an input image while preserving the styles and textures of both the text and the background to maintain a realistic appearance [1] . As a newly emerging task in the field of scene text processing [2] , STE not only possesses distinctive application value [3, 4, 5] but also benefits the text-oriented downstream research in detection [6] , recognition [7, 8] , spotting [9] and reasoning [10, 11] . Recently, increasing attention has been paid to GAN-based and diffusion-based scene text editing methods.Exploiting the Generative Adversarial Networks (GANs) [12] , early works [1, 13] decompose STE into three subtasks: foreground text style transfer, background restoration and fusion. The divide-and-conquer manner significantly reduces the difficulty of pattern learning and enables the pre-training of sub-modules with additional supervision [14] . However, the generalization capabilities of these methods are inevitably limited due to the constrained model capacity of GANs [15] and the challenges in accurately decomposing text styles [4] . Besides, as observed in experiments, the divide- and-conquer design brings about the "bucket effect", wherein the unstable background restoration quality leads to messy fusion artifacts.Recently, large-scale text-to-image diffusion models [16, 17] have convincingly demonstrated strong capabilities in image synthesis and processing. Several methods attempt to realize STE in a conditional synthesis manner, including style image concatenation [18] and one-shot style adaptation [19] . However, these methods are limited to the coarse-grained learning of miscellaneous styles from text images. Other methods [20, 21, 22] tend to resolve STE in a universal framework along with STG (Scene Text Generation) in an inpainting manner conditioned on full images, which enables the leverage of large-scale data for self-supervised learning [23] . Nevertheless, their style guidance predominantly originates from the image's unmasked regions, which can be unreliable in complex scenarios and fail in style consistency. Besides, resulting from the weak correlation between text prompt and glyph structure [24, 25, 26] , diffusion-based STE methods are prone to generating typos, which decreases the text rendering accuracy.For the aforementioned problems, we identify insufficient prior guidance on both style and structure as the primary factor that impedes the previous methods from performing accurate and faithful scene text editing. As depicted in Fig. 1 , we propose a conditional diffusion-based STE model, wherein our method decomposes the prerequisite of STE into two main aspects: text style disentanglement and text glyph representation. The fine-grained disentangled text style features ensure visual coherency, while the robust glyph structure representation improves text rendering accuracy. The dual Style-Structure guidance collectively contributes to significant enhancements in STE performance. Furthermore, undesired color deviation and texture degradation compared with the source text image occasionally occur in the inference of diffusion-based STE methods, which is attributed to the error accumulation in the denoising process [27] as well as the domain gap between training and inference [19] . To overcome this limitation, we introduce a glyph-adaptive mutual self-attention mechanism to improve the generator, which sets up a parallel reconstruction branch to introduce the source image style prior through cross-branch integration. The refined sampling process effectively eliminates visual inconsistency without requiring additional tuning.Additionally, the deficiency of real-world evaluation benchmarks on STE has become a non-negligible problem as increasing methods are proposed. Early assessments [1] , which rely on synthetic data, face significant limitations in practice due to the domain gap. Recent evaluations [14] emphasize text accuracy in edited real images but fail to benchmark visual quality adequately. Based on the observation that scene texts often occur in phrases with the same style and background in real-world scenery, we elaborately collect 1,280 text image pairs in terms of similar style and word length from scene text datasets to build the ScenePair dataset enabling comprehensive evaluation.In summary, we improve STE with the full leverage of Text prior for comprehensive guidance Control throughout the model design, network training and inference control, termed as TextCtrl. Our main contributions are as follows:• For the first time, we decompose the prerequisite of STE into fine-grained style disentanglement as well as glyph structure representation and incorporate the Style-Structure guidance with diffusion models to improve rendering accuracy and style fidelity. • For further style coherency control during sampling, with the leverage of additional prior guidance through the reconstruction of the source image, we introduce a glyph-adaptive mutual self-attention mechanism that effectively eliminates visual inconsistency. • We propose an evaluation benchmark ScenePair consisting of cropped text image pairs along with original full-size images. To the best of our knowledge, it is the first pairwise real-world dataset for STE which enables both visual quality assessment and rendering accuracy evaluation. 
related work: 
 GAN-based Scene Text Editing. SRNet [1] first introduces the word-level editing method built in a divide-and-conquer manner. SwapText [13] further enhances SRNet with Thin Plate Spline Interpolation Network for curved text modification while STRIVE [28] extends the framework into the video domain of scene text replacement. Besides, TextStyleBrush [4] adopts a self-supervised strategy building on StyleGAN2 [29] while MOSTEL [14] designs a semi-supervised training scheme.Diffusion-based Scene Text Editing. Numerous studies have focused on adapting the diffusion model for scene text manipulation. DiffSTE [20] improves pre-trained diffusion models with a dual encoder design, wherein a character encoder for render accuracy and an instruction encoder for style control is used. DiffUTE [23] further utilizes an OCR-based image encoder as an alternative to CLIP Text encoder. Moreover, TextDiffuser [21] and UDiffText [30] leverage character segmentation masks for condition input and supervised labels respectively. To leverage text style, LEG [18] concats the source image as input while DBEST [19] relies on a fine-tuning process during inference. Recently, AnyText [22] adopted a universal framework to resolve STE and STG in multiple languages based on the prevalent ControlNet [31] .Image Editing with Diffusion Models. Image editing aims to manipulate a certain attribute (e.g. color, posture, position) of the target object while keeping the other context unchanged, which can be seen as the parent task of STE. Recent Diffusion-based methods have shown unprecedented potential with a wide variety of designs. Model-tuning methods [32, 33] fine-tune the entire model to enhance subject embedding in the output domain. Leveraging DDIM inversion [34] , prompttuning methods [27] turn to improve identity preservation by optimizing null-text prompts through classifier-free guidance sampling [35] . Recently, [36, 37] explored the self-attention layers in LDMs and demonstrated the rich semantic information preserved in queries, keys and values. Through the cross-frame substitute of keys and values of self-attention, they perform non-rigid editing without additional tuning. [38] further extends the cross-frame interaction to video domain for motion editing. 
method: 
 Based on a conditional synthesis manner, in this work, we define the scene text editing process as I edit = G(C struct , C style )Distinctive from natural objects, scene text possesses a complicated non-convex structure, wherein a minor stroke discrepancy can significantly alter visual perception and lead to misinterpretation [39] , thus presenting unique challenges to editing accuracy. For scene text editing, an ideal text encoder is capable of encoding the target text concerning glyph structure rather than semantic information [ 20, 21] or certain image template [14, 23] . Specifically, for a certain text C text = "Sif ted", encoder T is expected to be aware of the glyph structure of "S", "i", "f ", "t", "e", "d" respectively.To this end, we adopt a character-level text encoder to align the target text feature with its visual glyph structure. As depicted in Fig. 2 (a), the target text embedding in character level is processed with a transformer encoder T to generate glyph structure features C struct ∈ R L×d , which is further aligned to the visual feature of corresponding text image extracted by a frozen pre-trained scene text recognizer with CLIP loss [30, 40] L clip . Differ from [30] , we collect vast quantities of text fonts constructing a cluster {f ont 1 , f ont 2 ...f ont n } to render the corresponding text image with diverse fonts during training. The font-variance augmentation brings continuous glyph structure variation which implicitly enhances the projection from C struct to the cluster centroids of visual features for robust text glyph structure representation.Text styles comprise a variety of aspects, including font, color, spatial transformation and stereoscopic effect, which visually mingle with each other and bring obstacles to disentangle the style features precisely in previous works. To realize the fine-grained disentanglement of the text style, we propose a multi-task pre-training paradigm as illustrated in Fig. 2 (b) , involving text color transfer, text font transfer, text removal and text segmentation.Concretely, given a text image I source ∈ R 3×H×W , we first extract the style feature C style ∈ R N ×d with a ViT [41] backbone S, which is projected to texture feature c texture ∈ R N ×d and spatial feature c spatial ∈ R N ×d respectively. Subsequently, c texture is employed in text color transfer and text font transfer while c spatial is utilized for text removal and text segmentation.Text Color Transfer. Since both intrinsic style and lighting conditions determine the text color, it is challenging to label or classify the holistic color. Instead, we refer to image style transfer and implicitly extract color through colorization training. A light-weight encoder-decoder F c is built to provide colorization on a black and white text image i c in ∈ R 1×h×w with an Adaptive Instance Normalization [42] A for source text color image i c out ∈ R 3×h×w written as:EQUATIONText Font Transfer. With a common intention with color transfer to capture stylized information but focusing on glyph boundary, font transfer is realized through the boundary reshaping process. Another light-weight encoder-decoder F f is employed to transfer a template font text glyph i f in ∈ R 3×h×w to the source font text glyph i f out ∈ R 3×h×w through Pyramid Pooling Module [43] P in latent space as:EQUATIONText Removal and Text Segmentation. Text removal aims at erasing the text pixels and reasoning the background pixels covered by text while text segmentation decouples the spatial relationships between background and text. A residual convolution block with spatial attention mechanism [44] is adopted to construct a removal head F r and a segmentation head F s respectively to generate predicted background i r out ∈ R 3×h×w and predicted mask i s out ∈ R 1×h×w as:EQUATIONMulti-task Loss. With the multi-task pre-training for fostering the text style extraction and disentanglement ability of SE, the whole loss function for style pre-training can be expressed as:EQUATION) wherein we leverage MSE loss for L color , MAE loss for L rem and Dice loss [45] for L f ont and L seg . Synthetic groundtruth is leveraged for fine-grained supervision and the task-oriented pre-training achieves fine-grained textural and spatial disentanglement of stylized text images which fertilizes the style representation for downstream generator.With the robust glyph structure representation C struct and fine-grained style disentanglement C style mentioned above, a diffusion generator G is employed to integrate the prior guidance and generate the edited result as shown in Fig. 2 (c ). For C struct , since the U-Net in latent diffusion models contains both self-attention and cross-attention, wherein the cross-attention focuses on the relation between latent and external conditions [16, 17] , we replace the key-value in cross-attention modules of G with the linear projection of C struct to provide glyph guidance for improving accurate text rendering. For C style , promising results have been shown by additional control injection [31] through the decoder of U-Net, based on which we apply the multi-scale style feature C style to the skip-connections and middle block of the model G to provide a style reference for high-fidelity rendering.With the leverage of pre-trained model [17] , the training is performed under a combined supervision on the synthetic text image data. Please refer to Appendix A for preliminaries of the diffusion model and Appendix B.2 for implementation details of training.During inference of the diffusion-based STE model, undesired color deviation and texture degradation occasionally occur. Such discrepancy can be partly attributed to the error accumulation during the iterative sampling process [27, 36] . Besides, the domain gap between training and inference impedes style consistency in complicated real-world scenery. To control the visual style consistency, we attempt to ameliorate the inference process by injecting style prior from the source image into editing. Specifically, we propose the Glyph-adaptive Mutual Self-Attention mechanism, which seamlessly incorporates the style of source images throughout the deconstruction process.Reconstruction Branch. Rather than transforming random noise samples into an image, our objective is to execute an image-to-image translation, ensuring the preservation of style features. Initially, we perform DDIM inversion [27, 46] to generate an initial latent z T source from the source image I source . The deconstructed inversion process enables a reconstruction branch (z T source , z T -1 source ...z 0 source ) of the source image parallel to the editing branch (z T edit , z T -1 edit ...z 0 edit ), which benefits the proposed integration process, as shown by the arrow in Fig. 2 (d) .Glyph-adaptive Mutual Self-Attention Mechanism (GaMuSa). Diverge from the general image editing, the target text modification of STE can lead to significant changes in the condition representation, which impedes text style preservation through general prompt-tuning methods [27] .Algorithm 1 Glyph-adaptive Mutual Self-attention Input: Inversion latent z T source , reconstruction condition embedding c source , editing condition embedding c edit and target text embedding emb y . Parameters: Time step t, interval τ , intensity parameter λ and µ. Output: Denoised latent z 0 source and z 0 edit .1: t = T, λ = 0, µ = 1, τ = 5; 2: for t = T, T -1...1 do 3: ---------------Reconstruction Branch--------------- 4: z t-1 source , {K s , V s } ← Ĝ(t, z t source , c source ); ▷ Self-Attention. 5: ----------------Editing Branch----------------- 6:if at intervals of τ then 7:emb edit = R(E dec (z t edit )); 8: λ = (emb edit • emb y )/(∥emb edit ∥ * ∥emb y ∥); ▷ Cosine Similarity. 9: end if 10: µ = 1 -λ; 11: {K es , V es } = λ{K s , V s } + µ{K e , V e };▷ Integration. 12:z t-1 edit ← Ĝ(t, z t edit , c edit ; {K es , V es });▷ Glyph-adaptive Mutual Self-attention.13: end for Return z 0 source , z 0 edit Recently, [36, 37] have demonstrated that self-attention layers in the diffusion model focus on latent internal relations. Based on the characteristic, we perform a mutual self-attention process between two branches as shown in Fig. 2(d ), wherein at denoising step t, the Key-Value features {K s , V s } from reconstruction branch are introduced to the self-attention operation in editing branch. Rather than a direct replacement in the previous method [36] , however, we prefer an integration of {K s , V s } and {K e , V e } to mitigate the domain gap between z t source and z t edit . It is worth noting that the original mutual self-attention process is hyper-parameter-sensitive to the starting time step. Premature initialization may introduce ambiguity and lead to spelling errors, whereas late intervention might fail to provide sufficient guidance. Inspired by the gradual deformation of text glyph during the iteration, we design a glyph-adaptive strategy to control the intensity of integration. Specifically, we employed a vision encoder R of the pre-trained text recognizer [47] to construct a glyph-adaptive strategy for the harmonious integration of Key-Value between branches. Specifically, during the iterative process, the intermediate latent z t edit is decoded and processed with R for the cosine similarity calculation with the target text embedding emb y at intervals of τ steps to denote the glyph similarity of the intermediate edited image with target text. The similarity will serve as the intensity parameter λ and µ for controlling the integration between {K s , V s } from reconstruction branch and {K e , V e } from editing branch. The result of integration {K es , V es } is subsequently leveraged in the self-attention modules of the editing branch for style coherency enhancement. The overall sampling pipeline is illustrated in Alg. 1. [20, 21, 22] on TamperScene [14] since it does not contain full-size images.EQUATIONtext labels from ICDAR 2013 [48] , HierText [49] and MLT 2017 [50] , wherein each pair consists of two cropped text images with similar text length, style and background, along with the original full-size images. Collecting methods and dataset details are introduced in Appendix C.Evaluation Dataset. For a fair comparison, we conduct all the evaluations on real-world datasets. ScenePair consists of 1,280 cropped text image pairs along with original full-size images enabling both style fidelity assessment and text rendering accuracy evaluation. TamperScene [14] combines a total of 7,725 cropped text images with predefined target text to provide rendering accuracy evaluation. Nevertheless, it does not involve paired images for style assessment nor full-size images for evaluation on inpainting-based methods, demonstrating the necessity of the proposed ScenePair.Evaluation Metrics. For visual quality assessment, we adopt the commonly used metrics including (i) SSIM, mean structural similarity; (ii) PSNR, the ratio of peak signal to noise; (iii) MSE, the mean squared error on pixel-level; (iv) FID [51] , the statistical difference between feature vectors. For text rendering accuracy comparison, we measure with (i) ACC, word accuracy and (ii) NED, normalized edit distance, using an official text recognition algorithm [52] and corresponding checkpoint.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
performance comparison: 
 Implementation. We conduct the comparison of the proposed TextCtrl with two GAN-based methods: SRNet [1] and MOSTEL [14] as well as three diffusion-based methods: DiffSTE [20] , TextDiffuser [21] and AnyText [22] with their provided checkpoints. The quantitative results are illustrated in Tab. 1 and Tab. 2 while the qualitative results for comparison are shown in Fig. 3 and Fig. 4 . Notably, DiffSTE [20] , TextDiffuser [21] and AnyText [22] conduct STE with an inpainting manner on a full-size image, for which we employed the corresponding full-size image of each pair in ScenePair with the target text area masked as input and crop the generated target text area for style evaluation on text image level. SRNet [1] , MOSTEL [14] and TextCtrl resolve STE in a synthesis manner on a text image, for which we perform a perspective process to paste the generated image back to the full-size image for style evaluation on full image level. Text Rendering Accuracy. Meanwhile, owing to the robust glyph structure representation, TextCtrl achieves superior spelling accuracy among all the methods, with more than 33% improvements in rendering accuracy of paired target text and randomly chosen text in ScenePair. TamperScene [14] contains a number of ambiguous low-resolution text images, which bring obstacles in style disentanglement and therefore impede the rendering accuracy of TextCtrl. Still, TextCtrl achieves a higher normalized edit distance that indicates the explicit mapping constructed between text and glyph. In comparison, GAN-based methods tend to yield ambiguous images, where source text left by imperfect removal blends with target text, resulting in fuzzy visual quality. Besides, due to the limited model capacity, GAN-based methods suffer from weak generalization and show incompetence with unseen style font as shown in Fig. 3 row 4 and 5. Diffusion-based methods achieve a disproportionate NED to their relatively low accuracy, which indicates their struggle with spelling mistakes. Notably, the inpainting manner serves as a primary factor that impedes the editing quality on small text for AnyText, whereas TextCtrl possesses the flexibility to perform editing on arbitrary scale text images.TextCtrl significantly improves STE through the substantial leverage of prior information in proposed (i) glyph structure representation pre-training, (ii) style disentanglement pre-training and (iii) glyphadaptive mutual self-attention. We delve into the efficacy of each module in the following section. on ScenePair. Specifically, we employed a CLIP text encoder [40] for comparison which is generally adopted in generative diffusion models [16, 17] . The contrast between ACC and NED results in Tab.3 confirms that the CLIP text encoder [40] struggles with spelling mistakes which are attributed to the sub-word embedding [24] and sub-optimal alignment between text prompt and text glyph. We further analyze the impact of the proposed font-variance alignment strategy in pre-training and the results indicate the robust representation brought by the augmentation. In Tab. 4, we replace the style encoder with a prevalent module ControlNet following the implementation settings and empirical suggestions in [31] . Inference control with Glyph-adaptive Mutual Selfattention (GaMuSa). In Tab. 5, we assess the style fidelity enhancement of GaMuSa in contrast with direct sampling and a prevalent enhancement method Mas-aCtrl [36] on ScenePair. Quantitative results verify the effectiveness of GaMuSa in enhancing style control during inference on variant real-world text images. Further visualization in Fig. 6 demonstrates the ability to persist style fidelity of GaMuSa when confronted with situations including background color deviation, unseen font and glyph texture degradation. 
limitations and conclusion: 
 Challenging arbitrary shape text editing. Arbitrary shape text editing occurs occasionally when editing with text on a crescent signboard or a circular icon as shown in Appendix Fig. 11 . These texts possess a complicated geometric attribution which is hard to disentangle through the style reference.Early works [13, 14] adopt the Thin-Plate-Spline (TPS) module to capture the accurate geometric distribution of text and perform a transformation on the template image as pre-processing. However, this strategy only takes effect in GAN-based methods which adopt an image-to-image paradigm. It remains a problem to effectively introduce accurate geometric prior guidance to diffusion models.Sub-optimal visual quality assessments metric. Following previous STE methods, we adopt a variety of evaluation metrics for visual quality assessment. However, these metrics either focus on pixel-level discrepancy or concentrate on feature similarity in latent space, which is sub-optimal for assessing text style coherency. Besides, all these metrics rely on the paired data under which a ground-truth image is required. Though we collect a real-world image-pair dataset ScenePair in our work, a large amount of real-world text images remain unpaired and thus fail to provide visual quality assessment in editing. While human evaluation may be a possible solution, a more efficient and objective visual assessment metric is expected for scene text editing.In this paper, we propose a diffusion-based STE method named TextCtrl with the leverage of disentangled text style features and robust glyph structure guidance for high-fidelity text editing.For further coherency control during inference, a glyph-adaptive mutual self-attention mechanism is introduced along with the parallel sampling process. Additionally, an image-pair dataset termed ScenePair is collected to enable the comprehensive assessment on real-world images. Extensive quantitative experiments and qualitative results validate the superiority of TextCtrl.Modules During training of the generator, we follow the Diffusion Denoising Probabilistic Models (DDPM) [54] and perform the forward noising process on the image latent z e 0 = E enc (I e ) with random t ∈ {1, ..., T } to generate noised latent z e t :G E T S R Total Params 859M 83M 66M 118M 90M 1216MEQUATIONwhere α t = t i=1 (1 -β t ) and β t ∈ (0, 1) is defined by a parameter schedule. The noise latent serves as the model input yielding the predicted noise εt along with condition embedding c = {C struct , C style } as: εt = G(t, z e t , c), (11) during which c is randomly replaced by the unconditional embedding c null = {C null , C style } with the probability p uc = 0.1 to jointly train an unconditional model, enabling the leverage of classifier-free guidance [35] .Since the diffusion process is performed in the latent space, to enable further vision and linguistics supervision, we construct the image-level result Ĩe = E dec (z 0 e ) where z0 e is gained through:EQUATIONWe provide a triple-guidance loss for supervision of the diffusion generator, including the denoising loss L dn , the construction loss L cons and the linguistic loss L ocr .For construction loss, we adopt the L per and L style as part of construction loss following [14] , along with MSE loss L reg . The functions are written as:EQUATIONEQUATIONEQUATIONwhere balance factors λ 1 and λ 2 are set to 0.01 and 100 respectively. ϕ i is the activation map from relu1_1 and relu5_1 layer of VGG-19 model [59] and G is the Gram matrix.Words are composed of character sequences that inherently contain linguistic information. pre-trained text recognition models, rich in sequence features prior, can be harnessed as global guidance to enhance text rendering ability. For linguistic loss, a recognition process will be applied to the decoded image Ĩe ∈ R 3×H×W generating recognition result ỹ, which is utilized to calculate the cross-entropy loss CE with the text label y:EQUATIONwhere λ 3 is set to 0.01. The overall objective function in training can be presented as a combination of the denoising loss Eq. 6, the construction loss Eq. 13 and the text recognition loss Eq. 16:EQUATION 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We describe the safeguards that we will put in place in our future release of checkpoints. Please refer to Appendix E.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all relevant existing works and assets which are related/used in our work in References. We also provide license details of the assets used in our work. Please refer to Appendix F.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: Project page: https://github.com/weichaozeng/TextCtrl.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: During inference, for a full-size image, the quadrangle location of the source text region is indicated through user interface or detection with ocr model, which is the same as all the other STE methods to provide clear instruction of the specific text to be edited. Subsequently, we perform a cropping and perspective process on the text region to acquire the input source image. The source image is further employed in an inversion process based on Eq. 8 to construct a reconstruction branch and disentangled in style encoder S to serve as style guidance C style , while the arbitrary target text provided by user is encoder by glyph structure encoder T as C struct . The condition embedding c = {C style , C strcut } is later passed to the generator G guiding the generation process. The sampling step is set to T = 50 and the classifier-free guidance scale is set to ω = 2 with 7 seconds to generate an edited image on a single NVIDIA A6000 GPU. Note that for the reconstruction branch, either a default null text or a source text is suitable for the input of glyph structure encoder T due to the symmetry of the inversion process Eq. 8 and the reconstruction process Eq. 7, which is flexible and will not interface the editing quality. The edited result will be perspective and stitched back to the original region for the full image result. Dataset Introduction. To provide practical assessments on both style consistency and rendering accuracy, we propose the first real-world image-pair dataset in STE termed ScenePair. Specifically, we collect 1,280 image pairs with the text label from ICDAR 2013 [48] , HierText [49] and MLT 2017 [50] . For each pair, we collect the source text image, the target text image, the respective text labels, the respective quadrangle locations in full-size image and the original full-size image.The inspiration for constructing ScenePair dataset comes from the observation that scene texts often occur in phrases with the same style and background in real-world scenery, as depicted in Fig. 7 (a), which serves as a perfect pair sample for evaluation of editing quality. Notably, there isn't a "correct" result for image editing whereas our paired data serves as a reference benchmark for high-fidelity.Collecting Strategy. We design a semi-automatic collecting strategy for ScenePair from several scene text datasets as illustrated in Fig. 7 (b ). Initially, we collect the full-size images from the This paper introduces a novel method for scene text editing which leverages the fine-grained style disentanglement and robust glyph structure representation to achieve high-fidelity editing results. Though we acknowledge that the proposed method has the potential to be misused for image forgery, significant advancement in visual quality and text rendering accuracy would also contribute to textrelated visual art creation. To prevent the high risk of misuse of the proposed method, an additional user commitment will be required for accessing the checkpoint in our forthcoming open release, through which we hope to alleviate the potential misuse while benefiting further research.Here we provide license details of the code and data used in our proposed network and comparison experiments. SRNet [1] is available for use under GNU General Public License v3.0. AnyText [22] ,deep-text-recognition-benchmark [52] , ViTSTR [57] are available for use under Apache License 2.0. DiffSTE [20] , TextDiffuser [21] , CLIP [40] are available for use under MIT License. Stable Diffusion [16] is available for research purposes under CreativeML Open RAIL M License. ABINet is available for non-commercial purposes with license 5 . MLT 2017 [50] and HierText [49] are released under CC BY-SA 4.0 license. There is no known license for ICDAR 2013 6 [48], MOSTEL7 [14] and TamperScene8 [14] , but the data and code are commonly referred to as "public", and so we interpret this to mean they are available for use under research purpose.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.https://huggingface.co/runwayml/stable-diffusion-v1-5https://github.com/roatienza/deep-text-recognition-benchmarkhttps://github.com/FangShancheng/ABINethttps://github.com/FangShancheng/ABINet/blob/main/LICENSEhttps://rrc.cvc.uab.es/?ch=2https://github.com/qqqyd/MOSTELhttps://github.com/qqqyd/MOSTEL 


Decision: All five reviewers recommend accepting this paper (2 “Accept”, 1 “Weak Accept” and 2 “Borderline Accept”). The reviewers mentioned that the key idea of the proposed method is novel, the constructed ScenePair dataset is valuable, and the results of this paper are impressive. The AC agrees with the reviewers that the paper is interesting and deserves to be published in NeurIPS 2024.

Reviews: review: paper_summary: paper_summary: The paper addresses the challenges of Scene Text Editing (STE) by introducing a diffusion-based STE method, TextCtrl. Traditional GAN-based STE methods struggle with generalization, and existing diffusion-based methods face issues with style deviation. TextCtrl overcomes these limitations through two main components: (1) Style-Structure Guidance: By disentangling fine-grained text styles and robust text glyph structures to improve text style consistency and rendering accuracy. (2) Glyph-adaptive Mutual Self-attention Mechanism: it enhances style consistency and visual quality by reconstructing implicit fine-grained features of the source image.
Additionally, the paper introduces ScenePair, the first real-world image-pair dataset designed for STE evaluation.
strength_weakness: strength_weakness: (1) In this paper, the authors mention that ‘their style guidance predominantly originates from the image’s unmasked regions, which can be unreliable in complex scenarios and fail in style consistency.’ In theory, the unmasked region can provide the editing model with more style prior information from the surrounding environment, which is beneficial for style transfer during the editing process.

(2) An important characteristic in text editing is to ensure that the edited word is compatible with the features of the surrounding unmasked region of the original image. Similar approaches have already emerged, such as TextDiffuser and AnyText, as mentioned in your paper. Therefore, the setting of editing only cropped images does not guarantee this compatibility. Additionally, the method proposed in the paper involves relatively complex design choices.

(3) About the ScenePair benchmark: how do authors ensure that the backgrounds of image pairs are consistent? Or are there some examples where the image-pair’s backgrounds are noticeably mismatched after your filtering rules?

(4) About the evaluation: During the evaluation of inpainting-on-full-size-image method DiffSTE, TextDiffuser, and AnyText, do authors replace the unedited region with the origin image when comparing with the cropped-based method.
questions: questions: Please see weaknesses section.
limitations: limitations: Please see weaknesses section.
paper_summary: paper_summary: This manuscript proposes a diffusion-based scene text editing method with prior guidance control. It incorporates style-structure guidance into the model to enhance the text style consistency and rendering accuracy. A Glyph-adaptive mutual self-attention mechanism is designed to deconstruct the implicit fine-grained features to improve the generation quality. Besides, a new real-world image-pair dataset is proposed for fair comparisons. The experimental results show that the proposed method achieves the best results among existing methods.
strength_weakness: strength_weakness: **Limitation in Task Scope**: The proposed method, while highly specialized in the editing domain, is limited to the task of text editing and does not encompass text generation capabilities. This contrasts with previous methods that offer a dual functionality of both generation and editing. The inability to generate new text content as well as edit existing text may be considered a significant limitation, restricting the method's applicability in scenarios that require creative text synthesis in addition to editing.

**Insufficient Ablation Study**: Although the proposed method decomposes text style processing into four distinct tasks—text color transfer, text font transfer, text removal, and text segmentation—the paper does not present a comprehensive ablation study that systematically evaluates the contribution of each individual task. An ablation study is crucial for understanding the impact of each component on the overall performance. Without it, the discussion lacks depth regarding the significance and interplay of these tasks in achieving the method's objectives. Therefore, further research is needed to dissect the individual contributions and optimize the balance between these tasks for improved performance and efficiency.
questions: questions: 1. Can the proposed method deal with generation task directly, as mentioned in Weaknesses?

2. The ablation study for the four tasks is needed, as mentioned in Weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes TextCtrl, which is a new method for high-fidelity scene text editing. The authors identify the primary factor hindering previous methods from achieving accurate and faithful scene text editing to be the absence of prior guidance on stylistic elements and textual organization. By leveraging disentangled text style features and robust glyph structure guidance, TextCtrl achieves superior editing performance. A glyph-adaptive mechanism and a new ScenePair dataset for real-world evaluation further solidify TextCtrl's effectiveness.
strength_weakness: strength_weakness: - The paper misses details and discussions on key contributions (please see the Questions section below), making it hard to evaluate the significance of the contributions of the work.
questions: questions: - Sec. 3.2 on text style disentanglement only provides implementation details but doesn’t give the big picture of why all these components working together would successfully learn to disentangle the text style. Since this seems to be one of the main contributions of the paper, I’d recommend to include more discussions on what each component achieves.
- Line 189 mentions integration of the K, V of the reconstruction branch and K, V of the main branch is preferred over the replacement that is often done in the literature. Could you provide some insights on why this might be more helpful, and in which scenarios?
limitations: limitations: The limitations have been thoroughly discussed in the appendix.
paper_summary: paper_summary: This paper aims to enhance scene text editing performance using a conditional prior-guidance-control diffusion model. It decomposes text style into background, foreground, font glyph, and color features. A text glyph structure representation improves the correlation between the text prompt and glyph structure. For inference, a glyph-adaptive mutual self-attention mechanism with a parallel sampling process is proposed. To evaluate effectiveness, the ScenePair image-pair dataset is created. Experiments on ScenePair and TamperScene datasets, along with ablation studies, are conducted.
strength_weakness: strength_weakness: 1) For the ablation study of text style disentanglement, why replace the style encoder with ControlNet?

2) Some sentences are too long which impacts the clarity. For example, line 235 to line 239, the sentence “GAN-based methods generally … due to the unstable restoration quality” expands 5 lines, and it will be more clear if it is split into two or more sentences.

3) Some figures and tables are not convenient for reading, such as Fig. 4 & 5, Tab. 4 & 5. They should be swapped so that the main parts are near the corresponding figure or table.
questions: questions: See the weaknesses
limitations: limitations: The authors have pointed out the limitations in the Appendix.
paper_summary: paper_summary: Considering the limitations of current GAN-based and Diffusion-based Scene Text Editing (STE) methods, this paper introduces TextCtrl, a diffusion-based method that edits text with prior guidance control. Specifically, TextCtrl incorporates text style disentanglement and text glyph structure representation to control the style and structure of generated scene text images, respectively. During inference, a Glyph-adaptive Mutual Self-attention mechanism is designed to further enhance style consistency and visual quality. Additionally, a new real-world image-pair dataset called ScenePair is crated for fair evaluation. Experiments demonstrate that TextCtrl outperforms previous methods in terms of style fidelity and text accuracy.
strength_weakness: strength_weakness: 1.	Several hyperparameters are included in the TextCtrl. I am not sure is there any experience for users to set up these hyperparameters for different datasets.

2.	Some figures could be improved. For example, Figure 3 is somewhat not easy to follow; highlighting the pretrained parts and the trainable parts would make it clearer. Figure 6 appears chaotic, it may be better to choose fewer texts.
questions: questions: 1.	Can this method be used on Chinese scenes?

2.	The authors reference the way of structure condition insertion in ControlNet to feed style condition into TextCtrl, i.e., zero convolution and add, rather than structure condition. Could the authors explain this modification? BTW, could the authors introduce more experimental settings for ControlNet in Table 3, including inputs?

3.	In Fig.3, why using "sifted" for c_texture but "Despairing" for c_spatial?
limitations: limitations: Yes, the limitations are detailed discussed.

Comments: title: The rebuttal has been read
post-rebuttal
comment: Thank you for your feedback. We will work on refining the mentioned issues and enhancing clarity in the revised paper. Thanks again for the thorough review and valuable suggestions!
Thanks for the responses, which addressed my concerns. Hence, I increase my rating to "Accept". However, the authors should make hyperparameters setting clear, make Fig.3&6 easy to understand, and explain modification for ControlNet in the final version if it is accepted.
We sincerely appreciate your time and effort throughout the review. Thank you again for your valuable comments to improve the quality of our manuscript!
After reviewing the feedback and rebuttal, I would like to maintain my rating.
Thank you for your feedback! We truly appreciate your time and effort in reviewing our paper. We are committed to continuous improvement and value your insights in the revised paper.
After reviewing the feedback and rebuttal, I find that the concerns have been addressed. I will maintain my rating.