TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
Authors: Weichao Zeng
Yan Shu
Zhenhang Li
Dongbao Yang
Yu Zhou
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Image Synthesis
Image Editing
Scene Text Editing.
Primary area: machine_vision
TL;DR: TextCtrl is a novel diffusion-based approach that performs high-fidelity scene text editing through explicit text style disentanglement and glyph structure representation as well as adaptive inference control.

Abstract: Centred on content modification and style preservation, Scene Text Editing (STE) remains a challenging task despite considerable progress in text-to-image synthesis and text-driven image manipulation recently. GAN-based STE methods generally encounter a common issue of model generalization, while Diffusion-based STE methods suffer from undesired style deviations. To address these problems, we propose TextCtrl, a diffusion-based method that edits text with prior guidance control. Our method consists of two key components: (i) By constructing fine-grained text style disentanglement and robust text glyph structure representation,  TextCtrl explicitly incorporates Style-Structure guidance into model design and network training, significantly improving text style consistency and rendering accuracy. (ii) To further leverage the style prior, a Glyph-adaptive Mutual Self-attention mechanism is proposed which deconstructs the implicit fine-grained features of the source image to enhance style consistency and vision quality during inference. Furthermore, to fill the vacancy of the real-world STE evaluation benchmark, we create the first real-world image-pair dataset termed ScenePair for fair comparisons. Experiments demonstrate the effectiveness of TextCtrl compared with previous methods concerning both style fidelity and text accuracy. Project page: https://github.com/weichaozeng/TextCtrl.

TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
Weichao Zeng 1.3
Yan Shu
Zhenhang Li!.3 Yu Zhou
Dongbao Yang!. 1.3
Institute Information Engineering, Chinese Academy of Sciences VCIP & TMCC DISSec, College of Computer Science, Nankai University
School of Cyber Security, University of Chinese Academy of Sciences Ezengweichao, lizhenhang, yangdongbao>@@iie ac. shuyan98120gamil com, yzhou@nankai. edu.
Abstract
Centred content modification and style preservation, Scene Text Editing (STE) remains a challenging task despite considerable progress text-to-image synthesis and text-driven image manipulation recently. GAN-basee STE methods generally encounter a common issue of model generalization, while Diffusion-based STE methods suffer from undesired style deviations. To address these problems, we propose TextCtrl, a diffusion-basse control. Our method consists of two key components: (i) By constructing fine- grained text style disentanglement and robust text glyph structure representation, TextCtrl explicitly incorporates Style-Structure guidance into model design and network training, significantly improving text style consistency and rendering accuracy. (ii) To further leverage the style prior, adyphtadeptive Mutual Self- attention mechanism is proposed which deconstructs the implicit fine-graines features of the source image to enhance style consistency and vision quality during inference. Furthermore, to fill the vacancy of the real-world STE evaluation benchmark, we create the first real-world image-pair dataset termed ScenePair for fair comparisons. Experiments demonstrate the effectiveness of TextCtrl compared with previous methods concerning both style fidelity and text accuracy. Project page: http:://githbb.co///eeaaaoooooo/
method that edits text with prior guidance
Introduction
Scene Text Editing (STE) refers modifying the with desired content on an input image while preserving the styles and textures of both the text and the background to maintain realistic appearance [1]. As newly emerging task in the field of scene text processing [2], STE not only possesses distinctive application value [3, 4, 5] but also benefits the text-oriented downstream research detection [6], recognition [7, 8]. spotting [9] and reasoning [10, 11]. Recently, increasing attention has been paid to GAN-based and diffusion-bassee scene text editing methods.
Exploiting the Generative Adversarial Networks (GANs) [12), early works [1, 13] decompose STE into three subtasks: foreground text style transfer, background restoration and fusion. The crride-and-conque manner significantly reduces the difficulty of pattern learning and enables the pre-training of sub-modules with additional supervision [14]. However, the generalization capabilities of these methods are inevitably limited due the constrained model capacity of GANs [15] and the challenges in accurately decomposing text styles [4]. Besides, as observed in experiments, the divide-
"Coresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ROSE
ROSE
ROSE FORTNAAL
Generator
Style Disentanglement
BASY
DAISY
DERT
EABYY
DAIBY
Text Glyph Structure Representation
(c) Guided Generation
Figure 1: Conceptual illustration of the decomposition of STE by TEXTCTRL. (a) Text style is disentangled into text background, text foreground, text font glyph and text color features. (b) Text glyph structure is represented by the cluster centroid various font text features. (c) The explicit style features and structure features guide the generator to perform scene text editing.
and-conquer design brings about the "bucket effect", wherein the unstable background restoration quality leads to messy fusion artifacts.
Recently, large-scale text-to-image diffusion models [16, 17] have convincingly demonstrated strong capabilities in image synthesis and processing. Several methods attempt to realize STE in a conditional synthesis manner, including style image concatenation [18] and one-shot style adaptation [19]. However, these methods are limited to the coarse-grained learning of miscellaneous styles from text images. Other methods [20, 21, 22] tend resolve STE in a universal framework along with STG (Scene Text Generation) in an inpainting manner conditioned on full images, which enables the leverage of large-scale data for self-supervised learning [23]. Nevertheless, their style guidance predominantly originates from the image's unmasked regions, which can be unreliable in complex scenarios and fail in style consistency. Besides, resulting from the weak correlation between text prompt and glyph structure [24, 25, 26]. diffusion- based STE methods are prone to generating typos, which decreases the text rendering accuracy.
For the aforementioned problems, we identify insufficient prior guidance on both style and structure the primary factor that impedes the previous methods from performing accurate and faithful scene text editing. As depicted in Fig. 1, we propose a conditional diffusion frffusiio-bbsedd STE model, wherein our method decomposes the prerequisite of STE into main aspects: style disentanglement and text glyph representation. The fine-grained disentangled text style features ensure visual coherency, while the robust glyph structure representation improves text rendering accuracy. The dual Style-Structure guidance collectively contributes to significant enhancements in STE performance.
Furthermore, undesired color deviation and texture degradation compared with the source text image occasionally occur the inference diffusion-basee STE methods, which is attributed to the error accumulation in the denoising process [27] as well as the domain gap between training and inference [19]. To overcome this limitation, we introduce a improve the generator, which sets up parallel reconstruction branch introduce the source image style prior through cross-branch integration. The refined sampling process effectively eliminates visual inconsistency without requiring additional tuning.
mutual self-attention mechanism to
Additionally, the deficiency of real-world evaluation benchmarks on STE has become a non-negligible problem increasing methods are proposed. Early assessments [1], which rely synthetic data, face significant limitations in practice due to the domain gap. Recent evaluations [14] emphasize text accuracy in edited real images but fail to benchmark visual quality adequately. Based on the observation that scene texts often occur in phrases with the same style and background in real- world scenery, elaborately collect 1,280 text image pairs in terms of similar style and word length from scene text datasets to build the ScenePair dataset enabling comprehensive evaluation.
In summary, we improve STE with the full leverage of Text prior for comprehensive guidance Control throughout the model design, network training and inference control, termed as TextCtrl. Our main contributions are follows:

---

For the first time, we decompose the prerequisite of STE into fine- tne-grained style disentangle- ment as well as glyph structure representation and incorporate the Style-Structure guidance with diffusion models to improve rendering accuracy and style fidelity.
For further style coherency control during sampling, with the leverage of additional prior guidance through the reconstruction of the source image, introduce a glyph-adaptive mutual self-attention mechanism that effectively eliminates visual inconsistency.
We propose an evaluation benchmark ScenePair consisting of cropped text image pairs along with original full-size images. To the best of our knowledge, it is the first pairwise real-world dataset for STE which enables both visual quality assessment and rendering accuracy evaluation.
2
Related work
GAN-based Scene Text Editing, SRNet [1] first introduces the word-level editing method built in a divide-and-conquer manner. SwapText [13] further enhances SRNet with Thin Plate Spline Interpolation Network for curved text modification while STRIVE [28] extends the framework into the video domain of scene text replacement. Besides, TextStyleBrush [4] adopts a sulf-supe strategy building on StyleGAN2 [29] while MOSTEL [14] designs cun--suprviiee training scheme.
Diffusion-based Scene Text Editing. Numerous studies have focused on adapting the diffusion model for scene text manipulation. DiffSTE [20] improves pre-trained diffusion models with dual encoder design, wherein a character encoder for render accuracy and an instruction encoder for style control is used. DiffUTE [23] further utilizes an OCR- -bsed image encoder as an alternative to CLIP Text encoder. Moreover, TextDiffuser [21] and UDiffText [30] leverage character segmentation masks for condition input and supervised labels respectively. To leverage text style, LEG [18] concats the source image as input while DBEST [19] relies on a fine-tuning process during inference. Recently, AnyText [22] adopted universal framework to resolve STE and STG in multiple languages based on the prevalent ControlNet [31].
Image Editing with Diffusion Models. Image editing aims manipulate certain attribute (e.g. color, posture, position) of the target object while keeping the other context unchanged, which can be seen as the parent task of STE. Recent Diffusion- methods have shown unprecedented potential with a wide variety of designs. Model-tuning methods [32, 33] fine-tune the entire model to enhance subject embedding in the output domain. Leveraging DDIM inversion [34], prompt- tuning methods [27] turn to improve identity preservation by optimizing null-text prompts through classifier-free guidance sampling [35]. Recently, [36, 37] explored the self-attention layers in LDMs and demonstrated the rich semantic information preserved in queries, keys and values. Through the cross-frame substitute of keys and values of self-attention, they perform non-rigid editing without additional tuning. [38] further extends the cross-frame interaction to video domain for motion editing.
3 Method
Based on a conditional synthesis manner, in this work, we define the scene text editing process as Iedit (CSttrutt Cstyle) as shown in Fig. 2 (c). The text glyph structure feature is acquired from character-basse structure encoder as Cstruct = ((Ceeat)
Fig. 2 (a) and the text style feature derived from style encoder Cstyle S( in Fig. 2 (b). The module design and pre-training strategy to enable precise extraction for glyph structure and fine-gr fine-grained disentanglement of text style introduced in section 3.1 and section 3.2 respectively. with the whole model training process illustrated in section 3.3. Furthermore, details of the improved tuning-fite ffree inference control and the proposed glyph-adaptive mutual self-attention mechanism in Fig. 2 (d) are presented in section 3.4.
3.1 Text Glyph Structure Representation Pre-training
Distinetive from natural objects, scene text possesses a complicated non-convex structure, wherein a minor stroke discrepancy can significantly alter visual perception and lead to misinterpretation [39], thus presenting unique challenges to editing accuracy. For scene text editing, an ideal text encoder is capable of encoding the target text concerning glyph structure rather than semantic information
3

---

()) Test Glyph Structure Encoder
(6)
Text Style Encoder
Stat
Lector
SIRRA SiPteo Sifree
Lont
Clip
Lanurce
"Siteo"
Spaciia
Lyem
strratt
Grtyle
Isa
see
Source Text Image Inversion
Groax Adeenton
Zoro Cork Adi
source
Sourcce
Kg
Q
Kg
ldtt
(c) Diffusion
Glyph-adapiive Marual
Figure 2: Decomposed framework of Tex(Curl. (a) Text glyph structure encoder T with corresponding glyph structure representation pre-training. (b) Text style encoder & with corresponding style
disentanglement pre-training. (c) Prior guided diffusion generator g. (d) The improved inference control with the Elyph-adaptive Mutual Self-attention mechanism.
[20, 21] certain image template [14, 23]. Specifically, for certain text Czezt "Si"' encoder T is expected to be aware of the glyph structure of "S", "t", "f", "!", "t","" "d"
"
respectively.
To this end, we adopt character-level text encoder to align the target text feature with its visual glyph structure. As depicted Fig. (a), the target text embedding in character level processed with transformer encoder T generate glyph structure features Cstruct E gLxd which is further aligned to the visual feature of corresponding text image extracted by frozen pre-trained scene text recognizer with CLIP loss [30, 40] Lelip. Differ from [30], collect vast quantities of text fonts constructing a cluster { [font1. fontz... font,) } to render the corresponding text image with diverse augmentation brings continuous glyph structure variation
fonts during training. The which implicitly enhances the projection from Cstruct to the cluster centroids of visual features for robust text glyph structure representation.
3.2
Text Style Disentanglement Pre-training
Text styles comprise variety aspects, including font, color, spatial transformation and stereoscopic effect, which visually mingle with each other and bring obstacles to disentangle the style features precisely in previous works. To realize the fine-grained disentanglement of the text style, we propose multi-task pre-training paradigm as illustrated in Fig. 2 (b), involving text color transfer, text font transfer, text removal and text segmentation.
Concretely, given text image Laource E 3 first extract the style feature Catyle N with VIT [4]] backbone S, which is projected to texture feature Cteature E R x and spatial feature Cspatiat E R N xd respectively. Subsequently, Cteature is employed in text color transfer and text font transfer while Capatial is utilized for text removal and text segmentation.
Text Color Transfer. Since both intrinsic style and lighting conditions determine the text color, it is challenging label classify the holistic color. Instead, refer to image style transfer and implicitly extract color through colorization training. A light-weight encoder-decoder F built to provide colorization a black and white text image f E R x with Adaptive Instance Normalization [42] A for source text color image i 3x3xXx written
(. ene (19 Crezture)),
Text Font Transfer. With a common intention with color transfer to capture stylized information but focusing glyph boundary, font transfer is realized through the boundary reshaping process. Another

---

light-weight macoder-decodedr F/ is employed to transfer a template font text glyph s/ E R to the source font text glyph by 3xxxx through Pyramid Pooling Module [43] P in latent space as:
Text Removal and Text Segmentation. Text removal aims at erasing the text pixels and reasoning if out
F (P(F Crezture)). doe
(2)
the background pixels covered by text while text segmentation decouples the spatial relationships between background and text. A residual convolution block with spatial attention mechanism [44] is adopted to construct removal head F" and segmentation head J respectively to generate predicted background You E 1xxxxx and predicted mask fou E 1x1x as:
Fout F (Cspatts))))))
oout = F (Capatial),
With the multi-task pre-training for fostering the text style extraction and disen- tanglement ability of SE, the whole loss function for style pre- pre-training can be expressed as:
Multi-task Loss.
Ldisentangle Ccolor (oour Ege) + Lyone Cone(iout: i'()) + Crem (Wouts (Te)) + Gseg(tout (g))
(4)
wherein we leverage MSE loss for Lcotor. MAE loss for Crem and Dice loss [45] for Cront and Leeg- Synthetic groundtruth is leveraged for fine- fine-graine grained supervision and the task-oriented pre-training achieves fine-grained textural and spatial disentanglement of stylized text images which fertilizes the style representation for downstream generator.
3.3 Prior Guided Generation
With the robust glyph structure representation Cstruct and fine-grained style disentanglement Cstyle mentioned above, a diffusion generator is employed integrate the prior guidance and generate the edited result shown in Fig. 2 (c). For C' Cstruct. since the U-Net in latent diffusion models contains both self-attention and cross-attention, wherein the cross-attention focuses on the relation between latent and external conditions [16, 17J. we replace the key-value in cross-altention modules of G with the linear projection of Cstract to provide glyph guidance for improving accurate text rendering. For Catyle: promising results have been shown by additional control injection [31] through the decoder of U-Net, based on which we apply the multi-scale style feature Cstyle to the skip- connectonss and middle block of the model G to provide style reference for high-fidelity rendering.
With the leverage of pre-trained model [17], the training is performed under a combined supervision on the synthetic text image data. Please refer to Appendix A for preliminaries of the diffusion model
and Appendix B.2 for implementation details of training.
3.4 Inference Control
During inference of the diffusion-based STE model, undesired color deviation and texture degradation occasionally occur. Such discrepancy can be partly attributed to the егrоr accumulation during the iterative sampling process [27, 36]. Besides, the domain gap between training and inference impedes style consistency in complicated real-world scenery. To control the visual style consistency, we attempt to ameliorate the inference process by injecting style prior from the source image into editing. Specifically, we propose the incorporates the style of source images throughout the deconstruction process.
Mutual self-Attention mechanism, which seamlessly
Reconstruction Branch. Rather than transforming random noise samples into image, objective is to execute an image-to-image translation, ensuring the preservation of style fea- tures. Initially, we perform DDIM inversion [27, 46] to generate initial latent =2 z from the source image Isonree The deconstructed inversion process enables a reconstruction branch (2T (=Tource>> -T-1 "souroe- 20urrel of the source image parallel to the editing branch (2T -T =diit do.)) which benefits the proposed integration process, shown by the arrow in Fig. 2 (d).
Glyph-adaptive Mutual Self-Attention Mechanism (GaMuSa). Diverge from the general im- age editing, the target text modification of STE can lead significant changes the condition representation, which impedes text style preservation through general prompt-tuning methods [27].
5

---

Algorithm Glyph-adaptive Mutual Self-attention
Inversion latent z reconstruction condition embedding Csources editing condition embedding Cedat and target text embedding emby- Parameters:
Input:
Time step t, interval т. intensity parameter À and J..
Output: Denoised latent = Ssource and Eedit
T = 0, T = 5; 2: for = T,T 1...1 do
3:
Reconstruction Branch
4: 211
Ssource: [K,, Vo3 + (t, 2 ssource: Csource); Editing Branch
b. Self-Attention.
5:
6: if at intervals of then
7:
embedsit
8:
X= == =(Embeditt embyy) (embbessel * ||embyll)
Cosine Similarity.
9:
end
10:
1. 1-1-A
11:
(Kesr Vea) = M(Ko, Va} + Me Vel:
D Integration.
12:
eddt t Ôl1, Redar Codit; (Kess Vool);
1-1
> Glyph-adaptive Mutual Self-attention.
end for
Return
2
sourcee edit
Recently, [36, 37] have demonstrated that self-attention layers in the diffusion model focus on latent internal relations. Based on the characteristic, we perform mutual self-attention process between two branches as shown in Fig. 2(d), wherein at denoising step i, the Sey-Value features {K,, V₂} from reconstruction branch are introduced the self-attention operation in editing branch. Rather than direct replacement in the previous method [36], however, we prefer an integration of {K,, V. and (Ke, V2} to mitigate the domain gap between zoorre and Edse
is worth noting that the original mutual self-attention process is syper-paramettr-sensiiii to the starting time step. Premature initialization may introduce ambiguity and lead spelling errors, whereas late intervention might fail to provide sufficient guidance. Inspired by the gradual deformation of text glyph during the iteration, we design a glyph-adaptive strategy control the intensity of integration. Specifically, we employed vision encoder R of the pre-trained text recognizer [47] to construct a -ayphaadaiivve strategy for the harmonious integration of Key-Value between branches. Specifically, during the iterative process, the intermediate latent 2' is decoded and processed with for the cosine similarity calculation with the target text embedding emby at intervals of steps to denote the glyph similarity of the intermediate edited image with target text. The similarity will serve as the intensity parameter and μ for controlling the integration between (Ks, V.) from reconstruction branch and (Ke, KK,1% V. from editing branch. The result of integration (Kes. Ves) subsequently leveraged the self-attention modules of the editing branch for style coherency enhancement. The overall sampling pipeline is illustrated in Alg. 1.
Qe (AK, + +uKe)T (AV, + WVo), 1=1-A
GaMuSa = Softmax
(5)
Vd
Experiments
4.1 Dataset and Metries
Training Data. Based on [1, 3], we synthesize 200k paired text images for style disentanglement pre-training and supervised training of TextCtrl, wherein each paired images are rendered with the same styles (i.e. font, size, colour, spatial transformation and background) and different texts, along with the corresponding segmentation mask and background image. Furthermore, a total of 730 fonts are employed to synthesize the visual text images in text glyph structure pre-training.
ScenePair Benchmark. To provide assessments on both visual quality and rendering accuracy, we propose the first real-world image-pair dataset in STE. Specifically, collect 1,280 image pairs with

---

Metries
ScenePair (Cropped Text Image)
ScenePair (Full-size Image)
Methods
SSIM (x10")
PSNR
MSE (x10")
FID
SSIM t(x10")
FID L
SRNet [1]
26.66 0.00
14.08
0.00
5.61 = 0.0
49.22 0:0
98.91
1.48
MOSTEL [14]
27.45 0.00
14.46 + 0.00
5.19 000
49.19 0:000
98.96
1.49
DifISTE [20]
26.85
= 0.08
13.44 + 0.94
6.11 0004
120.34
# 1.52
98 86 (76.91)
2.37 (96.78)
TextDiffuser [21]
27.02 0.11
13.96
5.75
57.01 0.4
98. (92.76)
1.65 (12.23)
AnyText [22]
30.73 0.55
13.66
0.07
6.19=0144
51.79 0.35
98.99 (82.57)
1.93 (16.92)
TextCul
37,56
0.32
14,99
1 0.15
4.47 0.15
43.78
0.17
99.07
1.17
Table 1: Text style fidelity assessment within text image level and full-size image level, highlighted with best and second best results. For full-size image evaluation, we replace the unedited region with the origin image while values in "()" denote the direct output of inpainting- based methods.
Metries
ScenePair
ScenePair (Random)
TamperScene [ [14]
Methods
ACC(%) t
NED f
ACC(%) f
NED
ACC($)
NED 1
SRNet [1]
17.84 0.00
0..78
0,000
9.61 0.00
0.422
0.000
39.96 0.00
0.776 +: 0.000
MOSTEL [14]
37.69 0,00
0.557
0,000
22.50 = 0.00
0.451
-0000
76.79 0.00
0.858 + 0.000
DifSTTE [20]
31.35 0.35
0.538
0.002
21.56 +: 0.69
0.487
0.002
TexDiffuser [21]
51.48 + 0.19
0.719
0.003
33.99 +: 0.34
0.635
0.004
AnyText [22]
0.21
0.734 0.005
25.05
+ 0.08
0.593
0.005
TexICrrl
84.67 0.34
0.936 0.003
66.95 +: 0.13
0.869 0.007
74.17 0.55
0.909 1 0.011
Table 2:
Text rendering accuracy evaluation with different methods, highlighted with best and second best results. "Random" denotes that we replace the paired target text in SCENEPAIR with randomly chosen text to verify the model robustness. Note that are not able to evaluate inpainting- based STE methods [20, 21, 22] on TamperScene [14] since does not contain full-size images.
text labels from ICDAR 2013 [48], HierText [49] and MLT 2017 [50], wherein each pair consists of two cropped text images with similar text length, style and background, along with the original full-size images. Collecting methods and dataset details are introduced in Appendix C.
Evaluation Dataset. For fair comparison, we conduct all the evaluations real-world datasets. ScenePair consists of 1,280 cropped text image pairs along with original full-size images enabling both style fidelity assessment and text rendering accuracy evaluation. TamperScene [14] combines total of 7,725 cropped text images with predefined target text to provide rendering accuracy evaluation. Nevertheless, it does not involve paired images for style assessment nor full-size images for evaluation on inpainting-base based methods, demonstrating the necessity of the proposed ScenePair.
Evaluation Metrics. For visual quality assessment, adopt the commonly used metrics including (i) SSIM, mean structural similarity; (ii) PSNR, the ratio of peak signal to noise; (iii) MSE, the mean squared error pixel-level; (iv) FID [51], the statistical difference between feature vectors. For text rendering accuracy comparison, we measure with (i) ACC, word accuracy and (ii) NED, normalized edit distance, using official text recognition algorithm [52] and corresponding checkpoint.
4.2 Performance Comparison
Implementation. We conduct the comparison of the proposed TextCtrl with two GAN-base methods: SRNet [1] and MOSTEL [14] as well as three diffusion-based methods: DifISTE [20], TextDiffuser [21] and AnyTeex [22] with their provided checkpoints. The quantitative results are illustrated in Tab. 1 and Tab. 2 while the qualitative results for comparison are shown in Fig. 3 and Fig. 4. Notably, DiASTTE [20], TextDiffuser [21] and Any [22] conduct STE with an inpainting manner full-size image, for which we employed the corresponding full-size image of each pair in ScenePair with the target text area masked as input and crop the generated target text area for style evaluation text image level. SRNet [1], MOSTEL [14] and TextCtrl resolve STE in synthesis manner on a text image, for which we perform a perspective process to paste the generated image back to the full-size image for style evaluation full image level.
Text Style Fidelity. The text style fidelity assessment is performed on both the text image level and the full image level of ScenePair to enable comprehensive comparison among methods. On
7

---

Source Image
Target Text
SRNer
MOSTEL
DifSTE
TexDiffiser
AnyText Text
TextCrr
Council
"County"
Countyin
County
County
County
County
County
Januar
*Fatruar'
Fabruar
Februar
Febu
Februar
Februar
Februar
clear
Kaap*
cleaan
Keep
Keep
Keep
Keep
Keep
Fish
'Chisa"
Phiss
hhads
Chips
Chiss
Chips
Chips
Ware
1Balhroon
Wzoce
Ware
Bathroom
Bathroom
aalhiroom
Bathroow
admittance
"Authorisad"
Auttincise
Authōrised
Authorised
Authorised
Authorised
Authorised
Smooth
"lasts"
Sasts
Slasts
last
Lasts
lasts
last s
Figure 3: Qualitative comparison among different methods. Note that for the inpainting- based methods [20, 21, 22]. we conduct the editing on the full-size images and perform the visualization of the edited text region.
text image level, TextCtrl outperforms other methods by at least 0.07, 0.53, 0.72 and 5.41 in SSIM, PSNR, MSE and FID respectively as represented in Tab. 1. GAN-based methods [1. 14] generally achieve a higher score on pixel-level assessment (i.e., PSNR, MSE). The reason lies in that they method, which contains background restoration process to restrain the background region unaltered. Nevertheless, this may result in generating unsatisfied fuzzy images as shown in Fig. 3 column 3 and 4 due to the artifacts left by the unstable restoration process. Due to the loose style control result from the inpainting manner, undesired text style occurs occasionally in the outcome for diffusion- based methods [20, 21, 22]. On the contrary, TextCtrl benefits from the full leverage of disentangled style prior and the inference control for high-fidelity edited results. Further comparison the full image level demonstrates the superiority of TextCtrl with precise manipulation and less style deviation against the inpainting-baser methods with visualization in Fig. 4. Besides, it not negligible that the inpainting strategy downgrades the image quality of unmasked regions.
adopt
Text Rendering Accuracy. Meanwhile, owing to the robust glyph structure representation, TextCtrl achieves superior spelling accuracy among all the methods, with more than 33% improvements in rendering accuracy of paired target text and randomly chosen text in ScenePair. TamperScene [14] contains number of ambiguous low-resolution text images, which bring obstacles in style disentanglement and therefore impede the rendering accuracy of TextCtrl. Still, TextCtrl achieves higher normalized edit distance that indicates the explicit mapping constructed between text and glyph. In comparison, GAN-based methods tend to yield ambiguous images, where source text left by imperfect removal blends with target text, resulting in fuzzy visual quality. Besides, due to the limited model capacity, GAN-based methods suffer from weak generalization and show incompetence with unseen style font as shown in Fig. row and 5. Diffusion- NED to their relatively low accuracy, which indicates their struggle with spelling mistakes. Notably, the inpainting manner serves as primary factor that impedes the editing quality on small text for Any Text, whereas Tex(Ctrl possesses the flexibility to perform editing on arbitrary scale text images.
methods achieve a disproportionate
4.3 Ablation Study
TextCtrl significantly improves STE through the substantial leverage of prior information in proposed (i) glyph structure representation pre- tre-trainnng (ii) style disentanglement pre- pre-training and (ii) glyph- adaptive mutual self-attention. We delve into the efficacy of each module in the following section.
Text Glyph Structure Representation. Conditional text prompt serves an im- portant role in STE guiding the render- ing of edited text glyphs. Consequently, we conduct the experiments by training TextCtrl with different text encoders to evaluate the text rendering accuracy
ScenePair
ScenePair (Random)
Text Encoder
ACC(S) 1
NED
ACC(S)
NED
CLIP [40]
13.98
0.637
13.47
0.615
T wlo font-variance
76.08
0.875
60.84
0.827
w font-variance
84.67
0.936
66.95
0.869
Table 3: Ablation experiment on glyph structure represen- tation pre-training.
8

---

DijiSTE
TextDiffuser
AnyText
TextCirl
SPRINKLER VALVE CHAMBER
SPRINKL ER VALVE CHAMBER
SPRINKLE VALVE CHAMBER
SPRINKLER VALVE CHAMBER
ESTATE AGENTS
ESTATE AGENTS
ESTAIE AGENTS
ESTATE AGENTS
SAXONS
SAXONS
SAXONS
SAXONS
POLLING STATION
POLLING STATION
POLLING STATION
POLLING STATION
Figure 4: Qualitative comparison with inpainting baased methods [20, 21, 22] on full-size images.
on ScenePair. Specifically, we employed a CLIP text encoder [40] for comparison which is generally adopted in generative diffusion models [16, 17]. The contrast between ACC and NED results in Tab. 3 confirms that the CLIPP text encoder [40] struggles with spelling mistakes which are attributed to the sub-word embedding [24] and sub-optimal alignment between text prompt and text glyph. We further analyze the impact of the proposed font-variance alignment strategy in pre-training and the results indicate the robust representation brought by the augmentation.
Text Style Disentanglement. The explicit text style disentanglement pre-training distinguishes TextCtrl from previous STE methods [14, 21, 22] in fostering the fine-grained feature representation ability on scene text concerning font, color, glyph and background tex- ture. To further verify the style disentangling ability of TextCtrl, as depicted in Fig. 5, we visualize the style feature embedding using t-SNE [53] with text images from ICDAR 2013 [48] encoded by the pre-trained style encoder S. From broad perspective, text images with similar color cluster in different regions of fea- ture space which indicate the style representation of the image entirety. From a micro perspective, as shown by the sample pair pointed out with the magnifier, text images that share the same text style and background adjoin each other regardless of different text content.
work nto
RONTO
PROPER
Figure 5: t-SNE [53] visualization of style features by pre-trained text style encoder.
In Tab. 4, we replace the style encoder with a preva- lent module ControlNet following the implementation settings and empirical suggestions in [31]. Concretely, the style encoder is replaced with vanilla Stable Dif- fusion encoder, serving as the ControlNet module. The module is initialized with the pre-trained weight of SD encoder. As a powerful technique in enhancing genera- tion controllability. ControlNet is prevalently leveraged in image editing for style and structural control. The simple yet effective design enables more meticulous reference from conditional input (c.g., Canny Edge, Depth map), which is also verified through the ablation study against our style encoder S (w/o pre-training) shown in Paper Tab. 4. After performing the style pre-training, however, the style encoder achieves a superiority performance against ControlNet module. Quantitative results demonstrate the fine fine-grained representation ability brought by the explicit style pre- pre-training strat- egy, compared with implicit style learning by ControlNet. Notably, it also improves the parameter efficiency with 1 118M for style encoder S and 332M for ControlNet module.
Injoction Module
S
MSE
FID 4
ControlNet [31]
0.3306
0.0464
58.30
S w/o pre training
0.3130
0.0475
66.10
w pre-training
0.3756
0.0447
43.78
Table 4: Ablation experiment on style dis- entanglement.

---

Buckground Color Regulation
Unseen Fool Imiiation
Source image
The Photo
anyone
Just Ask
PAPER PLEASE PLEASE
GE NE
SA D
Post
starting
FIRST
SACK
Office
WIO GaMinSa
Photo
starting
Ask
FIRST
SACK
Office
# GuldaSa
Glyph Texture Improvement
Soaree Image
CABOT
HIGH TREES
deposit
word HELP
YOU
GOOD
WDO GaldvSn
PLACE PLACE
photos
book
DESK
ARC
FOOD
# GaldaSa
TREES
photos
book
DESK
ARE
FOOD
Figure 6: Visualization of text editing result with and without the proposed GaMuSa during inference, which verifies the improvements in background color, text font and glyph texture.
Inference control with Glyph-adaptive Mutual Self- attention (GaMuSa). In Tab. 5, we assess the style fidelity enhancement of GaMuSa in contrast with direct sampling and a prevalent enhancement method Mas- aCtrl [36] on ScenePair. Quantitative results verify the effectiveness of GaMuSa in enhancing style control dur- ing inference on variant real-world text images. Further visualization in Fig. 6 demonstrates the ability to persist style fidelity of GaMuSa when confronted with situations including background color deviation, unseen font and glyph texture degradation.
Inference
SSIM
MSF
FTD J.
wio
0.3126
0,0609
51.35
w MasaCul [36]
0.3571
0.0468
49.53
GaMuSa
0.3756
0.0447
43.78
Table 5: Ablation experiment on inference enhancement.
5 Limitations and Conclusion
Challenging arbitrary shape text editing. Arbitrary shape text editing occurs occasionally when editing with text on a crescent signboard or circular icon as shown in Appendix Fig. 11. These texts possess complicated geometric attribution which is hard to disentangle through the style reference. Early works [13, 14] adopt the Shin-Plate-Spline (TPS) module to capture the accurate geometric distribution of text and perform transformation on the template image as pre- mre-prcesssing However, this strategy only takes effect in GAN- based methods which adopt an image-to-ime to-image paradigm. remains a problem effectively introduce accurate geometric prior guidance to diffusion models.
Sub-optimal visual quality assessments metric. Following previous STE methods, we adopt variety of evaluation metrics for visual quality assessment. However, these metrics either focus on pixel-level discrepancy or concentrate on feature similarity in latent space, which is sub-optimal for assessing text style coherency. Besides, all these metrics rely on the paired data under which ground-truth image is required. Though we collect a real-world image-pair dataset ScenePair in our work, large amount of real-world text images remain unpaired and thus fail to provide visual quality assessment in editing. While human evaluation may be a possible solution, more efficient and objective visual assessment metric is expected for scene text editing.
In this paper, we propose a diffusion-based STE method named TextCtrl with the leverage of disentangled text style features and robust glyph structure guidance for high-fieeliyy text editing. For further coherency control during inference, ddyphadaptivee mutual self attentttnnion mechanism introduced along with the parallel sampling process. Additionally, an image-pai dataset termed ScenePair is collected to enable the comprehensive assessment real-world images. Extensive quantitative experiments and qualitative results validate the superiority of TextCtrl.
Acknowledgments
This work supported by the National Natural Science Foundation of China (Grant NO 62376266), and by the Key Research Program of Frontier Sciences, CAS (Grant NO ZDBS-LY-7024).
10

---

References
[1] Liang Wu, Chengquan Zhang, Jiaming Liu, Junyu Han, Jingtuo Liu, Errui Ding, and Xiang Bai. Editing Text in the Wild. In Proceedings of the ACM International Conference on Multimedia, 2019.
[2] Yan Shu, Weichao Zeng, Zhenhang Li, Fangmin Zhao, and Yu Zhou. Visual Text Meets Low-level Vision:
A Comprehensive Survey on Visual Text Processing. arXiv preprint (xXiv:2402.03083 2024.
[3] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. Synthetic Data for Text Localisation in Natural Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2315-2324, 2016.
[4] Praveen Krishnan, Rama Kovvuri, Guan Pang, Boris Vassilev, and Tal Hassner. TextStyleBrush: Transfer of Text Aesthetics From Single Example. IEEE Transactions Pattern Analysis and Machine Intelligence, 45:9122-91344 2021.
[5] Victor Fragoso, Steffen Gauglitz, Shane Zamora, Jim Kleban, and Matthew A. Turk. TranslatAR: A Mobile Augmented Reality Translator. In Proceedings of the IEEE Workshop Applicatiioss of Computer Vision, pages 497-502, 2011.
[6] Yan Shu, Wei Wang. Yu Zhou, Shaohui Liu, Aoting Zhang. Dongbao Yang, and Weipinng Wang. Perceiving Ambiguity and Semantics without Recognition: An Efficient and Effective Ambiguous Scene Text Detector. In Proceedings of the ACM International Conference Multimedia, pages 1851-1862, 2023.
[7] Zhi Qiao, Yu Zhou, Dongbao Yang. Yucan Zhou, and Weiping Wang. SEED: Semantics Enhanced Encoder-decoder Framework for Scene Text Recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13528-13537, 2020.
[8] Zhi Qiao. Yu Zhou, Jin Wei, Wei Wang. Yuan Zhang. Ning Jiang, Hongbin Wang, and Weiping Wang. PIMNet: A Parallel, Iterative and Mimicking Network for Scene Text Recognition. In Proceedings of the ACM International Conference on Multimedia, pages 2046-20555 2021.
[9] Wei Wang, Yu Zhou, Jiahao Lv, Dayan Wu, Guoqing Zhao, Ning Jiang, and Weipinng Wang. TPSNet: Reverse Thinking of Thin Plate Splines for Arbitrary Shape Scene Text Representation. In Proceedings of the ACM International Conference on Multimedia, pages 5014-5025, 2022.
[10] Huawen Shen, Xiang Gao, Jin Wei, Liang Qiao, Yu Zhou, Qiang Li, and Zhanzhan Cheng. Divide Rows and Conquer Cells: Towards Structure Recognition Large Tables. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 1369-1377.2000
[11] Gangyan Zeng, Yuan Zhang, Yu Zhou, Xiaomeng Yang. Ning Jiang, Guoqing Zhao, Weiping Wang. and Xu-Cheng Yin. Beyond OCR + VQA: Towards End-to-end Reading and Reasoning for Robust and Accurate TextVQA. Puttern Recognition, 138: 109337, 2023.
[12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. Advances in Neural Information Processing Systems, 27, 2014.
[13] Qiangpeng Yang, Hongsheng Jin, Jun Huang, and Wei Lin. SwapText: Image Based Texts Transfer in Scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 14688 1469-14697,200 2020.
[14] Yadong Qu, Qingfeng Tan, Hongtao Xie, Jianjun Xu, Yuxin Wang, and Yongdong Zhang. Exploring Stroke-Level Modifications for Scene Text Editing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 21 2119-2127, 2023.
[15] Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. arXiv preprint 1rXiv:2105.05233 2021.
[16] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.00 000440066222 /25, 2022.
[17] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10674-10685, 2022.
[18] Changshuo Wang, L. Wu, Xu Chen. Xiang Li, Lei Meng, and Xiangxu Meng. Letter Embedding Guidance Diffusion Model for Scene Text Editing. In Proceedings of the IEEE International Conference on Multimedia and Expo, pages 588-593, 2023.
[19] Joshua Santoso, Christian Simon, et al. On Manipulating Scene Text in the Wild with Diffusion Models. In Proce Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5202-$211, 2024.
[20] Jiabao Ji, Guanhua Zhang, Zhaowen Wang, Bairu Hou, Zhifei Zhang, Brian L Price, and Shiyu Chang Improving Diffuuson Models for Scene Text Editing with Dual Encoders. Transactions on Machine Learning Research, 2023.
11

---

[21]
Jingye Chen. Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Tex:Difuuse:: Diffusion Models Text Painters. Advances in Neural Information Processing Systems, 36, 2024.
[22] Yuxiang Tuo, Wangmeng Xiang, Jun-Yan Yan He, Yifeng Geng, and Xuansong Xie. AnyText: Multilin- gual Visual Text Generation And Editing. In Proceedings of the International Conference on Learning Representations, 2024.
[23] Haoxing Chen, Zhuoer Xu, Zhangxuan Gu, Yaohui Li, Changhua Meng. Hujjia Zhu, Weiqiang Wang. al. DifTUTE: Universal Text Editing Diffusion Model. Advances in Neural Information Processing Systems, 36, 2024.
[24] Rosanne Liu, Daniel H Garrette, Chitwan Saharia, William Chan, Adam Roherts, Sharan Narang, Irina Blok, R. Mical, Mohammad Norouzi, and Noah Constant. Character- Models Improve Visual
Text Rendering. In Proceedings of the Association for Computational Linguisticx, 2022.
[25] Yukang Yang. Dongnan Gui, Yuhui Yuan, Weicong Liang. Haisong Ding, Han Hu, and Kai Chen. Glyph- Control: Glyph Conditional Control for Visual Text Generation, Advances in Neural Information Processing Systems, 36, 2024.
[26] Zhenhang Li, Yan Shu, Weichao Zeng, Dongbao Yang, and Yu Zhou. First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blending. In Proceedings of the European Conference Artificial Intelligence, 2024.
[27] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or: Or. Null-text Inversion for Editing Real Images using Guided Diffusion Models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pottern Recognition, pages 6038-6047, 2023.
[28] Jeyasri Subramanian, Varnith Chordia, Eugene Bart, Shaobo Fang, Kelly Guan, Raja Bala, al. STRIVE: Scene Text Replacement In Vidcos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14549-14558, 2021.
[29] Tero Karras, Samuli Laine, Miik Aittla, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and Improving the Image Quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8107-811 8116, 2019.
[30] Yiming Zhao and Zhouhui Lian. UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character- aware Diffusion Models. arXiv preprint arXiv:2312.04882 2023.
[31] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3813-3824, 2023.
[32] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang. Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text- based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[33] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- Booth: Fine Tuning Text-to-Image Diffusion Models for Subject-Drivee Generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22500-22515 2022.
[34] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In Proceedings of the International Conference Learning Representations, 2021.
[35] Jomathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. In Proceedings of the NewrlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[36] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng, Masactrl: Tuning-free free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22560 22570, 2023.
[37] Yuval Alaluf, Daniel Garibi, Or Patashnik. Hadar Averbuch-Eloe Elor, and Daniel Cohen-Or. Cross-Image Attention for Zero-Shot Appearance Transfer. arXiv preprint arXiv:231 I. 1.03355, 2023.
[38] Shuyuan Tu, Qi Dai, Zhi-Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. MotionFditor: Editing Vidco Motion via Content-Aware Diffusion. In Proceedings of the IEEE/CV Conference on Computer Vision and Puttern Recognition, pages 7882-7891, 2024.
[39] Haibin He, Xinyuan Chen, Chaoyue Wang, Juhua Liu, Bo Du, Dacheng Tao, and Qiao Yu. Diff-Font: Diffusion Model for Robust One-shot Font Generation. International Journal of Computer Vision, pages 1-15, 2024.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the International Conference on Machine Learning, 2021.
12

---

[41] Rinon Gal, Yuval Alaluf. Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. In Proceedings of the International Conference on Learning Representations, 2022.
[42] Xun Huang and Serge J. Belongie. Arbitrary Style Transfer in Real-Time with Adaptive Instance Nor- malization. In Proceedings of the IEEE International Conference Computer Vision, pages 1510-150 1519, 2017.
[43] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid Scene Parsing Network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 0230-6239, 2016.
[44] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In-So Kweon. CBAM: Convolutional Block Attention Module. arXiv preprint 1rXiv:/807.00521, 2018.
[45] Fausto Milletari, Nassir Navab, and BeyeddAhmade Ahmadi, V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In Proceedings of the International Conference on 3D Vision, pages 565-571,
[46] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. DiffEdit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210. 2022.
[47] Shancheng Fang. Hongtao Xie, Yuxin Wang. Zhendong Mao, and Yongdong Zhang. Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7094-7103, 2021.
[48] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, M. Iwamura, Lluís Gómez Bigorda, Sergi Robles Mestre, Joan Mas Romeu, David Fernández Mota, Jon Almazán, and Lluís-Pere de las Heras. ICDAR 2013 Robust Reading Competition, In Proceedings of the International Conference on Docunent Analysis and Recognition, pages 1484-1493, 2013.
[49] Shanghang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. ICDAR 2023 Competition on Hierarchical Text Detection and Recognition. arXiv preprint arXiv:2305.095500 2023.
[50] Nibal Nayef. Fei Yin, Imen Bizid, Hyunsoo Choi, Yuan Feng, Dimosthenis Karatzas, Zhenbo Luo, Umapada Pal, Christophe Rigaud, Joseph Chazalon, Wafa Khlif, Muhammad Muzzamil Luqman, Jean- Christophe Burie, Cheng-Lin Liu, and Jean-Mare Ogier. ICDAR2017 Robust Reading Challenge on Multi- Lingual Scene Text Detection and Script Identification RRC-MLT. In Proceedings of the International Conference on Document Analysis and Recognition, volume 01, pages 1454- 1459, 2017.
[51] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to Local Nash Equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
[52] Jeonghun Back, Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Scong Joon Oh, and Hwalsuk Lec. What Is Wrong With Scene Text Recognition Model Comparisons? Datasct and Model Analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4714-4722, 2019.
[53] Laurens van der Maaten and Geoffrey E. Hinton. Visualizing Data using I-SNE. Journal of Machine Learning Research, 0-2579-26055 2008.
[54] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[55] Diederik P Kingma. Auto-enc variational bayes. arXiv preprint arXiv: 2013.
[56] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Asiisted Intervention, pages 234 244-241,2 22,,,21,2221 2015.
[57] Rowel Atienza. Vision Transformer for Fast and Efficient Scene Text Recognition. In Proceedings of the IEEE International Conference on Document Analysis and Recognition, 2021.
[58] Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An Image is Worth 16x 16 Words: Transformers for Image Recognition at Scale. arXiv preprint 20 1929, 2020.
[59] K Simonyan and A Zisserman. Very Deep Convolutional Networks for Large-scale Image Recognition. In Proceedings of the International Conference on Learning Representations, 2015.
13

---

Appendix
A Preliminaries
Latent Diffusion Models (LDMs). Instead of operating diffusion process [34, 54] in image pixel space, LDMs utilize an autoencoder [55] E to translate the input image to latent space representation 20. Then the denoising network eo built upon a time-conditional UNet [56] is trained to estimate time step t. The condition embedding c, which is c {Cstructy Cstyle) in our work, is integrated through the cross-attention mechanism or the skip-connections and middle block, realizing conditional generation that has the following training objective:
added noise
dnn ( <o 2t, c)
DDIM Sampling. At inference, random Gaussian noised can be gradually denoised to form result 20 through iterative sampling, wherein the deterministic DDIM sampling [34] is adopted in our method:
ar-1 2t + v Var-1
2t-1
1)
(7)
0tt
0t-1
where 11 B:) and Bo and tends to I as increases.
DDIM Inversion. In contrast the stochastic sampling employed in DDPM [54], the sampling process in DDIM [34] is deterministic, allowing for the complete inversion [27, 46] from the original images latent 2p back to initial noised latent zy construct the reconstruction branch in our work.
21+1
0+++1
21 + /C
V
-
-
V
-
ca(1. c),
Qt
a++1
Classifier-Free Guidance (CFG). To improve the visual quality and faithfulness of generated images, [35] introduces the classifier-free guidance technique, which jointly trains a conditional and an unconditional (denoted as Crut)) diffusion model to provide refined result:
(t c) cott, c) + (1 - w) co ze, Cault);
(9)
where w is hyperparameter that controls the strength of guidance.
B Implementation Setting
B.1 Details of Model Architecture and Parameter
CexlCtrl primarily comprises five components: an Encoder-Decoder VAF &, a U-Net backbone G. text glyph structure encoder 'T. text style encoder S and a vision encoder R. For the VAE and U-Net, we employ the pre-trained checkpoint of Stable Diffusson [17] V1-5². For the text glyph structure encoder, utilize a lightweight transformer encoder and perform pre-training proposed glyph structure representation aligning to the visual features captured by frozen vision encoder [57]. For the text style encoder, VIT [58] backbone is employed to perform pre-training on multi-task style disentanglement. For the vision encoder, the vision backbone of ABINet [47] is adopted with pre- traitrd checkpoint.
the
As the model input, the source image resized 256 256 while the max target prompt length set to 24. The training process utilizes a batch size of 256 with a learning rate of x 10-5 and total epoch of 100. TextCtrl trained on NVIDIA A6000 GPU and the parameter sizes of each module provided in Tab. 6.
htnp://gttii..cmmmmmmataaadcccee benchaark
14

---

Modules
E
T
R
Total
Params
X59M
X3M
66M
118M
90M
1216M
Table 6: The parameter sizes of each module TEXTCTRI.
B.2 Details of Training
During training of the generator, follow the Diffusion Denoising Probabilistic Models (DDPM) [54] and perform the forward noising process on the image latent z Eene(Fe) with random
t t {1,...,T T) to generate noised latent of:
2f Varzi + 1 vI- ())
(10)
where a2 = 1T 3)) and Br e (0,1) defined by parameter schedule. The noise latent serves as the model input yielding the predicted noise & along with condition embedding {Cstruct; Cstyle] as:
E = (1,
(11)
during which e is randomly replaced by the unconditional embedding Goall {Crulls Sstyle) with the probability Puc. 0.1 to jointly train unconditional model, enabling the leverage of classifier-free guidance [35].
Since the diffusion process is performed in the latent space, to enable further vision and linguistics supervision, we construct the image-level result Ie ( where =0 is gained through:
20
vl-a,,))
(12)
We provide Hiple-guidance loss for supervision of the diffusion generator, including the denoising loss Lan, the construction loss Leons and the linguistic loss Locr.
For construction loss, we adopt the pee and Estyle as part of construction loss following [14), along with MSE loss Lyeg- The functions are written as:
Leons Xpprer + A2 + Fregs
(13)
Cper E[ - -x (Te)
(14)
Letgle
- (T.)lil.
(15)
where balance factors A and A2 are set to 0.01 and 100 respectively. Φ; is the activation map from relul_] and relu5_ layer of VGG-19 model [59] and is the Gram matrix.
Words are composed of character sequences that inherently contain linguistic information. pre-trained text recognition models, rich in sequence features prior, be harnessed global guidance to enhance text rendering ability. For linguistic loss, a recognition process will be applied to the decoded image T, E 3 xW generating recognition result ÿ. which is utilized to calculate the cross-entropy loss C.E. with the text label y:
Loer = A3 O),
(16)
where A3 is set to 0.01. The overall objective function training can be presented as a combination of the denoising loss Eq. 6, the construction loss Eq. 13 and the text recognition loss Eq. 16:
Lan + Leans + Locr
(17)
B.3 Details of Inference
During inference, for a full-size image, the quadrangle location of the source text region is indicated through user interface or detection with ocr model. which is the same all the other STE methods to provide clear instruction of the specific text to be edited. Subsequently, we perform a cropping and perspective process on the text region to acquire the input source image. The source image is further employed in inversion process based on Eq. 8 to construct a reconstruction branch and disentangled in style encoder serve as style guidance Cstyles while the arbitrary target text provided by user is encoder by glyph structure encoder T as Cstruct- The condition embedding
15

---

== (Cytyle Cstreut] is later passed to the generator 9 guiding the generation process. The sampling step set to T = 50 and the classifier-free guidance scale is set w 2 with 7 seconds generate an edited image on a single NVIDIA A6000 GPU. Note that for the reconstruction branch, either default null text or a source text suitable for the input of glyph structure encoder 7 due to the symmetry of the inversion process Eq. 8 and the reconstruction process Eq. 7, which is flexible and will not interface the editing quality. The edited result will be perspective and stitched back to the original region for the full image result.
C ScenePair Dataset
TIRES
Cropping t images
HAROLD HOLT SWIH centre
ScaneText
Image
CURTIS TIRE CENTER:
ENTRANCE
BL GRAMEE TURE 4R MIMOR REPAIRS ALTONTBUUCK TIKE KERAIS NEW LEED THE4
Automatic with similarity score
Pogal?
Text
WELCOME TO: MARKS & SPENCER
Palred Text Images
Raw Dataset
COLLECTBB:
Manual B
Filturing
ScenePair (b)Pipeline of data collection ScenePuir
Visuuliiizoiio of puir-texts images
Figure 7: Data collection strategy of ScenePair. (a) Texts with the same style and background often occur in real-world scenery. (b) The pipeline for our data collection.
STIRLING
Health "Heath" Centre "Contee"
DENTAL DENTAL SURGERY "SURGERY"
STEBLINE
'STIRLING" LODGINGS 'LODGINGS'
Health Centre Greenwood House
DENTAL SURGERY
STREET STREETT NORRIS WORRIS
M
MADE "MADE' HEAVEN HEAVEN
1406 NORRIS STREET
Gark Muti Millonncairee
Mull"
MATCH MADE HERWEN
Geek
dpple
FOOD "FOOD" BANK "BANK"
Hotel
Tipple Trae Cottage
FOOD BANK ails
Mople Cottag "Cotlage"
Mited de Ville
Wille
'otall"
'Ville
Figure Visualization of paired data along with full-size images in SCENEPAIR.
Dataset Introduction. To provide practical assessments both style consistency and rendering accuracy, we propose the first real-world mmae--pair dataset in STE termed ScenePair. Specifically, we collect 1,280 image pairs with the text label from ICDAR 2013 [48], HierText [49] and MLT 2017 [50]. For each pair, we collect the source text image, the target text image, the respective text labels, the respective quadrangle locations full-size image and the original full-size image.
The inspiration for constructing ScenePair dataset comes from the observation that scene texts often occur in phrases with the same style and background in real-world scenery, depicted in Fig. 7 (a), which serves as a perfect pair sample for evaluation of editing quality. Notably, there isn a "correct" result for image editing whereas our paired data serves as a reference benchmark for high-fidelity.
Collecting Strategy. We design a semi-automatic collecting strategy for ScenePair from several scene text datasets illustrated in Fig. 7 (b). Initially, we collect the full-size images from the
16

---

Source Image
Edited Image
Source Image
Edited Image
Source Image
Edited Image
LOUNGE
HUNTER
canon
Timer
massimo
thicket
value
forly
Reisen
azoxy
bibliotheque
favourably
PROPER
WRONG
twilighh
bicipit'al al
Services
Variant
Terminals
omoplate
Sportif
sedider 12
House
pyuria
2045
bed t table
market
anorectun
SHOGUN
remidd a
Wickfield
retroviral
SPENCER
FINALLY
SECCOR
hedoe
Saisons
mondusiee
Dicembre
upwards
MUSEUM
earnestly
Thohen
interlan
DocMorris
precess
DOLLAR
WEATT
Museum
corbieste
Indian
bustup
Repertory
borgin
Gratta
vvertokk
SCONTIAMO
hypolastic
ALES
ARTS
Enrico
maratials
Krombacher
reinstate
Francesco
fascinating
SHOPS
PLACE
RUSSELL
RECREESS
Feroglobin
fanciest.
IRON-BD
CASK
CAKE
JobCenter
incinderjell
mostra
canthal
Lounge
ramekin
Metisse
whether
with
adjoin
nostro
sidhe
Carrier
Lownsizing
artwork
megiddo
Stand morsi
Attract
ubiety
Ferndale
condol lierg
Figure 9: Visualization of edited result on text image by TEXTCTRL on dataset TAMPERSCENE.
datasets along with the detection and recognition labels. For each full-size image, we perform cropping and perspective with the quadrangle location to acquire all the text images, for which an automatic pairing algorithm is employed to construct paired text images. Specifically, the algorithm calculates the weighted similarity score involving text length, aspect ratio, centre distance in full-size image and SSIM (Structure Similarity Index Measure) between the cropped text images, wherein the paired images with similarity score that surpass a pre-defined threshold would be chosen to form raw dataset. Finally, we manually filter out the unsatisfied pairs to construct the ScenePair dataset, as shown in Fig. 8.
D Visualization
To further demonstrate the editing performance of our TextCtri, we provide various visualizations of edited outcomes. In Fig. 9, variety of images in TamperScene with abundant text style are provided to verify the text style generalization of TextCtrl. In Fig. 10, we provide the edited result on the scene image of ICDAR 2013. Specifically, given a scene photo with an automatically detected text box, we perform scene text editing on box images and stitch back to the original photo. Results demonstrate the ability of TextCtrl to preserve fine-gree ine-graieed detail for high-fidelity editing.
17

---

Sourc Iuuser
Edived Iaag
Siarc Jwage
Edlted Iwage
WARNING
CALLING
Thesgasking
Library Open
Kitchen Close
crininal conviction
rinened conviction
Next
22:
GEME
Last
11:2 31 GENE
Departure FIRST TE
Arrival
FIRST ER
Poed
HARWICH COURT
PROBLEM COUPET
SUPERKINGS
SUPERKINGS
HIGHSS
Coach Stop
Scace SIOp
CAN PARC
HEAIHY CORL BISTER X000
REACIE WE FRENT X0X
hellmann systems
supermar
EAST HOUSE
WEST HOMS
ghosts escape
White Unise
Yellow
Figure 10: Visualization of edited result on scene image by TEXTCTRL on dataset ICDAR 2013.
Source Image
Edited Imagee
Soures Image
Edital hoage
Source Image
Edited Image
PHONOPHIL
metallizer
APPELL ATTION TOURNESOL
C CA
SUPERIOR
soddse
ESPRESSO
ESPRESSI
pipecoline
GROSVENOR
unapplicable
BADGER
Progression
Figure 11: Visualization of fallure cases on image by TEXTCTRL. The problem mainly results
from the insufficient geometric prior guidance control.
E
Additional Considerations
This paper introduces novel method for scene text editing which leverages the fine- tne-grained style disentanglement and robust glyph structure representation to achieve high-fidelity editing results. Though we acknowledge that proposed method has the potential to misused for image forgery, significant advancement in visual quality and text rendering accuracy would also contribute to text- related visual art creation. To prevent the high risk of misuse of the proposed method, an additional user commitment will be required for accessing the checkpoint in our forthcoming open release, through which we hope to alleviate the potential misuse while benefiting further research.
F
Licenses
Here we provide license details of the code and data used in our proposed network and comparison experiments. SRNet [1] is available for use under GNU General Public License v3.0. AnyText [22], deep-text-recoonnitonnbbo iiapooggnitioo--eechmaaakk [52], ViTSTR [57] are available for use under Apache License
2.0. DifISTE [20], TextDiffuser [21], CLIP [40] are available for use under MIT License. Stable
Diffusion [16] is available for research purposes under CreativeML Open RAIL M License. ABINet
18

---

is available for non-commercial purposes with license MLT 2017 [50] and HierText [49] are released under CC BY-SA 4.0 license. There is no known license for ICDAR 2013 [48], MOSTEL" [14] and TamperScene" x [14], but the data and code are commonly referred to as "public", and so we interpret this to mean they are available for use under research purpose.
http:///wwccccccccooooooo/o/ooaa 7 http:///ghhhn.cmmmgggyydoMOOTT
"hutp:///rc.cw http:://rrc.vc..uua.ee///2222
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly reflect the main contributions and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work. Please refer Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification: Our propositions are mainly based on experiments and empirical results which do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided all the implementation details of model design, network training and inference in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Project page: http:///ghhhuu..oom//wccccoooocr
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nwpp...c////////o https/ccppblcccguideeeeooo Stupppuiiiiguuuieeeeeettttutaaia /CooeSuboeeeeeeeee swttuubmisssioiooo wewiissssnniioiiieieoeioo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided all the details including the data and hyperparameter in our paper. Please refer to Sec. 4.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the standard deviation with respect to the random seed after rumning each model times in main experiments. Please refer to Tab. and Tab. 2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided these details in the paper. Please refer to Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
cc/ Sppuulic/tthissuuuie ines?
Answer: [Yes]
Justification: Ethics.
Our research conducted in the paper conforms with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the potential societal impacts of our work. Please refer to Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We describe the safeguards that we will put in place in our future release of checkpoints. Please refer Appendix E.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited all relevant existing works and assets which are related/used in our work in References. We also provide license details of the assets used in our work. Please refer to Appendix F.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Project page: http://gthha..cmmeeecccarrre//et
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "oIN9Kx3nER", "reviewer": "Reviewer_MkTr", "summary": "The paper addresses the challenges of Scene Text Editing (STE) by introducing a diffusion-based STE method, TextCtrl. Traditional GAN-based STE methods struggle with generalization, and existing diffusion-based methods face issues with style deviation. TextCtrl overcomes these limitations through two main components: (1) Style-Structure Guidance: By disentangling fine-grained text styles and robust text glyph structures to improve text style consistency and rendering accuracy. (2) Glyph-adaptive Mutual Self-attention Mechanism: it enhances style consistency and visual quality by reconstructing implicit fine-grained features of the source image.\nAdditionally, the paper introduces ScenePair, the first real-world image-pair dataset designed for STE evaluation.", "questions": "Please see weaknesses section.", "limitations": "Please see weaknesses section.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Style-Structure Guidance: By disentangling fine-grained text styles and robust text glyph structures to improve text style consistency and rendering accuracy.\n\n(2) Glyph-adaptive Mutual Self-attention Mechanism: it enhances style consistency and visual quality by reconstructing implicit fine-grained features of the source image.\n\n(3) This paper propose an image-pair dataset termed ScenePair to enable the comprehensive assessment on real-world images.", "weaknesses": "(1) In this paper, the authors mention that ‘their style guidance predominantly originates from the image’s unmasked regions, which can be unreliable in complex scenarios and fail in style consistency.’ In theory, the unmasked region can provide the editing model with more style prior information from the surrounding environment, which is beneficial for style transfer during the editing process.\n\n(2) An important characteristic in text editing is to ensure that the edited word is compatible with the features of the surrounding unmasked region of the original image. Similar approaches have already emerged, such as TextDiffuser and AnyText, as mentioned in your paper. Therefore, the setting of editing only cropped images does not guarantee this compatibility. Additionally, the method proposed in the paper involves relatively complex design choices.\n\n(3) About the ScenePair benchmark: how do authors ensure that the backgrounds of image pairs are consistent? Or are there some examples where the image-pair’s backgrounds are noticeably mismatched after your filtering rules?\n\n(4) About the evaluation: During the evaluation of inpainting-on-full-size-image method DiffSTE, TextDiffuser, and AnyText, do authors replace the unedited region with the origin image when comparing with the cropped-based method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comprehensive review, which will certainly help improve the revised paper. The concerns are taken care to address point by point in the following.\n\n\n> **[W1]: Discussion of more style prior information from unmasked region.**\n\n[A1]: We agree that surrounding unmasked region could also provide style prior. Nevertheless, how to determine the informative unmasked region for providing accurate style prior is still an unresolved problem. \n\nCurrent inpainting-based methods [*1,2,3*] take the whole masked image as input and reason the text style by surrounding unmasked region. As shown in ***Uploaded Pdf Fig.A***, this implicit and indirect style learning strategy lack controllability and reliability, leading to text style deviation sometimes. In contrast, TextCtrl focuses on cropped text region which provides direct and explicit guidance for maintaining style consistency in editing.\n\n\n\n> **[W2]: Editing on cropped images does not guarantee compatibility; The method involves relatively complex design choices.**\n\n[A2]:  We would like to clarify that while inpainting-based methods possess the intrinsic ability for smoothening, cropped-based methods adopt a background preservation policy to guarantee compatibility. \n\nCompatibility of TextCtrl is ensured through two aspects. Firstly, a text removal task is involved in style pretraining to explicitly foster the ability to preserve backgrounds. Secondly, during inference, we integrates K-V pairs from reconstruction branch to alleviate background degradation. Quantitative results on SSIM in ***Paper Tab.1*** verify the compatibility of TextCtrl comprehensively, along with visualization in ***Paper Appendix Fig.11***.  \n\nThough specialized design has been proposed in TextCtrl, the framework is simple and detachable. It's built upon a vanilla text-to-image diffusion model [*6*] (SD v1-5), with a replacement of text encoder and an increment of style encoder. Our design focuses on scene-text-oriented pretraining and inference control, while following vanilla training and sampling settings in [*6*].  The aforementioned settings jointly contribute to TextCtrl without bringing much complexity.\n\n\n\n> **[W3]: About ScenePair: consistency measurement of backgrounds; mismatched examples after the filtering rules.**\n\n[A3]: The collection pipeline is depicted in ***Paper Appendix Fig.8***. To ensure pair consistency, we design an automatic pairing algorithm with scoring function: \n\n$Score = \\lambda_{length} * S_{length} + \\lambda_{ratio}*S_{ratio} + \\lambda_{distance} * S_{distance} + \\lambda_{SSIM} * S_{SSIM}$, \n\nwherein $S_{length}$ is the similarity of text lengths, $S_{ratio}$ is the similarity of aspect ratio, $S_{distance}$ is the centre distance and $S_{SSIM}$ is the structure similarity of cropped-text regions. The consistency of backgrounds is ensured technically by $S_{SSIM}$ and empirically by $S_{distance}$ (nearer texts are more likely to share same background and lighting condition).\n\nAfter pairing, pairs with $Score$ higher than a threshold are selected and in most cases they share a common background as shown in ***Paper Appendix Fig.9***. Still, we manually filter out a very few number of unsatisfied pairs to guarantee final quality. Examples are provided in ***Uploaded Pdf Fig.B***.\n\n\n> **[W4]: About full-size-image evaluation: do authors replace the unedited region with the origin image for inpainting-based methods?**\n\n[A4]: Official code of inpainting-based methods [*1,2,3*] is adopted and the output is directly used for full-image evaluation without replacing unedited regions, since it is generally considered as the final results which intrinsically distinguishes inpainting-based method.\n\nYet we acknowledge the necessity of evaluation with the unedited region replaced by origin images for a comprehensive comparison. To this end, we extend the full-size evaluation, wherein *(w)* and *(w/o)* denote with and without replacing unedited regions. Comparing within each inpainting-based method, a large margin emerges indicating the inpainting strategy downgrades the image quality of unmasked region, which mainly result from the lossy compression of VAE and the model capacity. Comparison across different methods verifies the compatibility achieved by TextCtrl. Besides, it is worth noting that cropped-image evaluation is regarded as the main indicator of text style consistency since full-image evaluation spends most calculation on unedited regions.    \n\n| Methods          |SSIM $\\uparrow(\\times10^{-2})$               | FID $\\downarrow$ |\n| ---      | ---                                | ---      |\n| SRNet [*4*]               |98.91                              |1.48                  |\n| MOSTEL [*5*]              | 98.96                             |1.49                  |\n| DiffSTE [*1*] *(w/o)*     | 76.91                             |96.78                  |\n| DiffSTE [*1*] *(w)*       | 98.86                             |2.37                  |\n| TextDiffuser [*2*] *(w/o)*| 92.76                             |12.23                  |\n| TextDiffuser [*2*] *(w)*  | 98.97                             |1.65                  |\n| AnyText [*3*] *(w/o)*     | 82.57                             |16.92                  |\n| AnyText [*3*] *(w)*       | 98.99                             |1.93                  |\n| TextCtrl                  | **99.07**                             |**1.17**                  |\n\n\n> **References**\n\n[1] Ji et al. Improving Diffusion Models for Scene Text Editing with Dual Encoders. TMLR, 2024.\n\n[2] Chen et al. TextDiffuser: Diffusion Models as Text Painters. NeurIPS, 2023.\n\n[3] Tuo et al. AnyText: Multilingual Visual Text Generation And Editing. ICLR, 2024.\n\n[4] Wu et al. Editing Text in the Wild. ACM MM, 2019.\n\n[5] Qu et al. Exploring Stroke-Level Modifications for Scene Text Editing. AAAI, 2023.\n\n[6] Rombach et al. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JAWUd6QL3s", "reviewer": "Reviewer_Ktb6", "summary": "This manuscript proposes a diffusion-based scene text editing method with prior guidance control. It incorporates style-structure guidance into the model to enhance the text style consistency and rendering accuracy. A Glyph-adaptive mutual self-attention mechanism is designed to deconstruct the implicit fine-grained features to improve the generation quality. Besides, a new real-world image-pair dataset is proposed for fair comparisons. The experimental results show that the proposed method achieves the best results among existing methods.", "questions": "1. Can the proposed method deal with generation task directly, as mentioned in Weaknesses?\n\n2. The ablation study for the four tasks is needed, as mentioned in Weaknesses.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Innovative Approach**: The paper introduces a novel diffusion-based method that leverages prior guidance control for text editing, addressing the limitations of previous GAN-based and diffusion-based methods.\n\n**Fine-grained Style Disentanglement**: By constructing a fine-grained text style disentanglement, the method improves text style consistency, which is crucial for maintaining the original style and texture during editing.\n\n**Robust Glyph Structure Representation**: The system incorporates a robust representation of text glyph structures, enhancing the accuracy of text rendering.\n\n**Comprehensive Evaluation**: The authors have created the ScenePair dataset to evaluate both visual quality and rendering accuracy, providing a more holistic assessment of STE methods.", "weaknesses": "**Limitation in Task Scope**: The proposed method, while highly specialized in the editing domain, is limited to the task of text editing and does not encompass text generation capabilities. This contrasts with previous methods that offer a dual functionality of both generation and editing. The inability to generate new text content as well as edit existing text may be considered a significant limitation, restricting the method's applicability in scenarios that require creative text synthesis in addition to editing.\n\n**Insufficient Ablation Study**: Although the proposed method decomposes text style processing into four distinct tasks—text color transfer, text font transfer, text removal, and text segmentation—the paper does not present a comprehensive ablation study that systematically evaluates the contribution of each individual task. An ablation study is crucial for understanding the impact of each component on the overall performance. Without it, the discussion lacks depth regarding the significance and interplay of these tasks in achieving the method's objectives. Therefore, further research is needed to dissect the individual contributions and optimize the balance between these tasks for improved performance and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review. We are encouraged by the comments that TextCtrl serves as an innovative approach and ScenePair provides a more holistic assessment to STE methods. The concerns are taken care to address point by point in the following.\n\n\n> **[W1/Q1]: Limitation in task scope to offer a dual functionality of both generation and editing.**\n\n[A1]: Thanks for bringing the question and concern. To start off, we would like to highlight that the main focus of our work is to address scene text editing with throughly disentangled text style and glyph structure feature as the prior guidance. Differ from scene text generation, scene text editing possesses the unique challenge in faithfully representing text styles (e.g., font, color, serifs) as well as detail texture (e.g., grids, shadow). To this end, we constructe a style encoder for explict text style extraction and propose the glyph-adaptive mutual self-attention machanism for implicit consistency control, with the concentration on preserving fidelity in editing.\n\nSome inpainting-based methods offer a dual functionality of both generation and editing, yet their editing ability mainly inherits from reasoning the text style according to surrounding unmasked context, which would be uncontrollable and unreliable sometimes and lead to style deviation, as shown in the ***Uploaded Pdf Fig.A***. \n\nAlthough our current model focuses on high-fidelity text editing, it can also be easily modified to enable both high-fidelity text editing and unconditional creative text generation. Specifically, leveraging the classifier-free guidance (CFG) technique [*1*] which enables a joint training of conditional and unconditional model\n\n $\\widehat\\epsilon_{\\theta}(t, z_{t}, c) = \\omega ·\\epsilon_{\\theta}(t,z_t, c) + (1-\\omega)·\\epsilon_{\\theta}(t, z_t, \\emptyset)$,\n \n we could encode either the pure background image or the text removal output of style encoder, serving as an additional guidance $c_{bg}$.  Along with the text style guidance $c_{style}$, we have \n \n $\\widehat\\epsilon_{\\theta}(t, z_{t}, c_{bg}, c_{style}) = \\omega ·\\epsilon_{\\theta}(t,z_t, c_{bg}, c_{style}) + (1-\\omega)·\\epsilon_{\\theta}(t, z_t, c_{bg}, \\emptyset)$,\n \n based on which our model could generate without $c_{style}$, which offers a dual functionality for generation and editing. This multi-condition approach has been proven to be feasible in other tasks [*2,3*]. We appreciate the reviewer's suggestion on model versatility and we will further the research in future work.\n\n\n\n\n\n> **[W2/Q2] Further ablation study is needed to evaluate the contribution of each task in text style pretraining.**\n\n[A2]: We have performed an ablation study of the text style pretraining from a broad perspective in  ***Paper Tab.3*** to verify its joint contribution to foster style representation. Yet we agree that a further discussion on the contribution of each task would benefit the understanding of each component and optimization in implementation. \n\nThe text style pretraining is decomposed into four sub-tasks, wherein *Text Font Transfer* and *Text Color Transfer* focus on textural style (e.g., glyph style, text color), while *Text Removal* and *Text Segmentation* concentrate on capturing spatial style (e.g.,  background, size). Due to the time constraint of rebuttal, we augment our style ablations with further experiments on two groups of sub-tasks, namely texture group (font & color) and spatial group (removal & segmentation). Results show that texture group mainly achieves growth in fidelity reflected on FID, while spatial group improves the overall quality of editing reflected on all metrics. The reason lies in the fact that sub-tasks in texture group concentrate on detail texture transfer, while sub-tasks in spatial group reckon with both foreground text and background pixel, thus achieving better statistical similarity. Meanwhile, the two groups jointly contribute to the mutual collaboration in training and achieve finer performance.\n\n\n|Font|Color|Removal|Seg|SSIM $\\uparrow$|PSNR $\\uparrow$|MSE $\\downarrow$|FID  $\\downarrow$|\n|:------:|:------:|:------:|:------:|------|------|------|------|\n|&#10008;|&#10008;|&#10008;|&#10008;|0.3130|14.78|0.0475|66.10|\n|&#10004;|&#10004;|&#10008;|&#10008;|0.3197|14.79|0.0470|58.81|\n|&#10008;|&#10008;|&#10004;|&#10004;|0.3652|14.91|0.0453|47.19|\n|&#10004;|&#10004;|&#10004;|&#10004;|0.3756|14.99|0.0447|43.78|\n\n\n\n> **References**\n\n[1] Ho et al. Classifier-Free Diffusion Guidance. NeurIPS Workshop, 2021.\n\n[2] Sharma et al. Alchemist: Parametric Control of Material Properties with Diffusion Models. CVPR, 2024.\n\n[3] Song et al. Arbitrary Motion Style Transfer with Multi-condition Motion Latent Diffusion Model. CVPR, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1n1SS0Ef8V", "reviewer": "Reviewer_rTPt", "summary": "This paper proposes TextCtrl, which is a new method for high-fidelity scene text editing. The authors identify the primary factor hindering previous methods from achieving accurate and faithful scene text editing to be the absence of prior guidance on stylistic elements and textual organization. By leveraging disentangled text style features and robust glyph structure guidance, TextCtrl achieves superior editing performance. A glyph-adaptive mechanism and a new ScenePair dataset for real-world evaluation further solidify TextCtrl's effectiveness.", "questions": "- Sec. 3.2 on text style disentanglement only provides implementation details but doesn’t give the big picture of why all these components working together would successfully learn to disentangle the text style. Since this seems to be one of the main contributions of the paper, I’d recommend to include more discussions on what each component achieves.\n- Line 189 mentions integration of the K, V of the reconstruction branch and K, V of the main branch is preferred over the replacement that is often done in the literature. Could you provide some insights on why this might be more helpful, and in which scenarios?", "limitations": "The limitations have been thoroughly discussed in the appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written.\n- The experiments are extensive and well-discussed.\n- The dataset is valuable for the research community.", "weaknesses": "- The paper misses details and discussions on key contributions (please see the Questions section below), making it hard to evaluate the significance of the contributions of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. We are encouraged by the positive response to the paper presentation and ScenePair dataset. The remaining concerns are taken care to address point by point in the following.\n\n> **[Q1]: A big picture of why all the components working together in style pretraining would successfully learn to disentangle the text style.** \n\n[A1]: \nText style comprises a variety of aspects (e.g., font, color, perspective), which visually mingle with each other. This characteristic poses great challenge to the direct apply of general representation learning methods (e.g., vqvae [*1*]) on stylized text images. While it is intuitive to decompose text style into isolated components for object-oriented learning, some components (e.g., size, spatial relation) are comparatively difficult to depict or synthesize. \n\nEventually, we turn to resolve the problem with task-oriented pretraining on the stylized text image, which effectively enables the leverage and representation of implicit text style during learning. Notably, we unify several tasks on the same model to promote mutual collaboration from joint training and enhance the style feature extraction, as suggested in [*2*].\n\nFour tasks are included in style pretraining as shown in ***Paper Fig.3(b)***, wherein \n\n- *(i) Text Color Transfer* extracts text color from the stylized text image. Since the text color is determined by both intrinsic style and lighting conditions, it is challenging to label or classify the holistic color (e.g., RGB, Captions). Instead, we refer to image style transfer and implicitly extract color through colorization training.\n\n- *(ii) Text Font Transfer* shares a common intention with *Text Color Transfer* to capture stylized information but focuses on glyph boundary that shapes the font style, for which we construct a similar architecture and learn from the boundary reshaping.\n\n- *(iii) Text Removal* has already been explored in considerable works [*3,4*], which aims at erasing the text pixels and reasoning the background pixels covered by text. We package it as one of the pretraining tasks to benefit the background preservation in editing.\n\n- *(iv)* *Text Segmentation* not only facilitates the precise removal but also decouples the spatial relationships between background and text, as well as extracting an internal description of text (e.g., size, interval of characters).\n\nThe implementation details are provided in ***Paper Sec.3.2***. The task-oriented pretraining achieves fine-grained textural and spatial disentanglement of stylized text image and fertilizes the style representation for downstream generator. \n\n\n\n\n> **[Q2]: Why prefer the integration of K-V of reconstruction branch and main branch.**\n\n[A2]: This is indeed an interesting question concerning the semantic correspondence of diffusion latents. \n\nTo start off, leveraging the internal representations, numerous works [*5,6*] have revealed the generative diffusion model's ability on establishing reasonable semantic correspondence across different images, even when exhibiting significant differences in categories, shapes, and poses. \n\nA step forward, [*7*] utilizes the aforementioned correspondence in self-attention module through a mask-guided replacement of K-V to perform non-rigid editing of foreground objects (e.g., adjust the posture of a dog with the appearance and background maintained). [*8*] further explores the roles played by the Q-K-V in encoding semantic information and observes that Q determines the semantic meaning of each spatial location while K-V offer the context of different parts of the image for weighted aggregation.\n\nScene Text Editing shares a common motivation with aforementioned non-rigid editing task [*7,8*] but preserves differences in its complicated non-convex structure, wherein a minor stroke discrepancy can significantly alter visual perception and lead to misinterpretation. The complicated text structure brings obstacles to the replacement since it's difficult to either obtain a text mask or mask the text region precisely on the intermediate feature. In addition, without the mask, performing replacement on the whole K-V often results in sparse attention maps during self-attention which may lead to inaccurate transfers and artifacts on final result. \n\nTo deal with the problem, we prefer a heuristic integration of K-V from two branches to avoid losing concentration and enable a flexible weighted aggregation for style consistency control. Experience in ***Paper Tab.4*** verifies that our integration strategy surpasses replacement strategy [*7*] on maintaining text style consistency. We believe the integration strategy would also be helpful when performing editing without a precise mask for the complex structural object.\n\n> **References**\n\n[1] Van et al. Neural discrete representation learning. NeurIPS, 2017.\n\n[2] Peng et al. UPOCR: Towards Unified Pixel-Level OCR Interface. ICML, 2024.\n\n[3] Wang et al. What is the Real Need for Scene Text Removal? Exploring the Background Integrity and Erasure Exhaustivity Properties. TIP, 2023.\n\n[4] Peng et al. Viteraser: Harnessing the power of vision transformers for scene text removal with segmim pretraining. AAAI, 2024.\n\n[5] Zhang et al. A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence. NeurIPS, 2023.\n\n[6] Tang et al. Emergent Correspondence from Image Diffusion. NeurIPS, 2023.\n\n[7] Cao et al. MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing. CVPR, 2023.\n\n[8] Alaluf et al. Cross-Image Attention for Zero-Shot Appearance Transfer. ACM SIGGRAPH, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ACRovAfzr", "reviewer": "Reviewer_mqKG", "summary": "This paper aims to enhance scene text editing performance using a conditional prior-guidance-control diffusion model. It decomposes text style into background, foreground, font glyph, and color features. A text glyph structure representation improves the correlation between the text prompt and glyph structure. For inference, a glyph-adaptive mutual self-attention mechanism with a parallel sampling process is proposed. To evaluate effectiveness, the ScenePair image-pair dataset is created. Experiments on ScenePair and TamperScene datasets, along with ablation studies, are conducted.", "questions": "See the weaknesses", "limitations": "The authors have pointed out the limitations in the Appendix.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1) Text glyph structure representation addresses the weak correlation between text prompts and glyph structures in text-to-image diffusion models.\n\n2) The proposed glyph-adaptive mutual self-attention ensures coherent control during inference, guided by source image reconstruction, which is novel.\n\n3) The ScenePair dataset is valuable for research on STE tasks.\n\n4) Both quantitative and qualitative experimental results demonstrate notable performance improvement across multiple datasets.", "weaknesses": "1) For the ablation study of text style disentanglement, why replace the style encoder with ControlNet?\n\n2) Some sentences are too long which impacts the clarity. For example, line 235 to line 239, the sentence “GAN-based methods generally … due to the unstable restoration quality” expands 5 lines, and it will be more clear if it is split into two or more sentences.\n\n3) Some figures and tables are not convenient for reading, such as Fig. 4 & 5, Tab. 4 & 5. They should be swapped so that the main parts are near the corresponding figure or table.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments. Your detailed review will certainly help improve the revised paper. The remaining concerns are taken care to address point by point in the following.\n\n\n> **[W1]: For the ablation study of text style disentanglement, why replace the style encoder with ControlNet?**\n\n[A1]: Thanks for bringing the question. We will first detail the experiment and later discuss the purpose. \n\nIn ***Paper Tab.3***, the ablation study of ControlNet [*1*] is conducted by replacing our style encoder with a vanilla Stable Diffusion encoder, serving as the ControlNet module. Following the implementation settings and empirical suggestion in [*1*], we initialize the parameters of ControlNet module with the copy of pretrained SD encoder weight and enable training on both ControlNet module and SD generator to fit in the specialized text data.  \n\nAs a powerful technique in enhancing generation controllability, ControlNet is prevalently leveraged in image editing for style and structural control. The simple yet effective design enables a more meticulous reference from conditional input (e.g., Canny Edge, Depth map), which is also verified through the ablation study against our style encoder (w/o style pretraining) shown in ***Paper Tab.3***. After performing the style pretraining, however, the style encoder achieves a superiority performance against ControlNet module. \n\nThe ablation study is leveraged to demonstrate the fine-grained representation ability brought by the explicit style pretraining strategy, compared with implicit style learning by ControlNet. Notably, it also improves the parameter efficiency with 118M for style encoder and 332M for ControlNet module.\n\n\n\n> **[W2]: Some sentences are too long which impacts the clarity. For example, line 235 to line 239, the sentence “GAN-based methods generally … due to the unstable restoration quality” expands 5 lines, and it will be more clear if it is split into two or more sentences.**\n\n[A2]: We appreciate your detailed review. In our revised paper, we will rewrite this kind of sentences with a clearer presentation.\n\n*\"GAN-based methods generally achieve a higher score on pixel-level assessment (i.e., PSNR,MSE). The reason lies in that they adopt a divide-and-conquer method, which contains a background restoration process to restrain the background region unaltered. Nevertheless, this may result in generating unsatisfied fuzzy images as shown in Fig.5 column 3 and 4 due to the artifacts left by the unstable restoration process.\"*\n\n\n\n> **[W3]: Some figures and tables are not convenient for reading, such as Fig. 4 & 5, Tab. 4 & 5. They should be swapped so that the main parts are near the corresponding figure or table.**\n\n[A3]: Thanks for the suggestion. We will rearrange the order of the figures and tables in our revised paper for better readability. Please don't hesitate to let us know if you have further questions or suggestions.\n\n> **References**\n\n[1] Zhang et al. Adding Conditional Control to Text-to-Image Diffusion Models. ICCV, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WIYTfL7VO6", "reviewer": "Reviewer_25Re", "summary": "Considering the limitations of current GAN-based and Diffusion-based Scene Text Editing (STE) methods, this paper introduces TextCtrl, a diffusion-based method that edits text with prior guidance control. Specifically, TextCtrl incorporates text style disentanglement and text glyph structure representation to control the style and structure of generated scene text images, respectively. During inference, a Glyph-adaptive Mutual Self-attention mechanism is designed to further enhance style consistency and visual quality. Additionally, a new real-world image-pair dataset called ScenePair is crated for fair evaluation. Experiments demonstrate that TextCtrl outperforms previous methods in terms of style fidelity and text accuracy.", "questions": "1.\tCan this method be used on Chinese scenes?\n\n2.\tThe authors reference the way of structure condition insertion in ControlNet to feed style condition into TextCtrl, i.e., zero convolution and add, rather than structure condition. Could the authors explain this modification? BTW, could the authors introduce more experimental settings for ControlNet in Table 3, including inputs?\n\n3.\tIn Fig.3, why using \"sifted\" for c_texture but \"Despairing\" for c_spatial?", "limitations": "Yes, the limitations are detailed discussed.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well written, and the presentation is clear\n2.\tIn the field of STE, the proposed method is reasonable in targeting structural integrity and stylistic consistency. By 3 prior guidance control of text style disentanglement, text glyph structure representation and source image reconstruction, TextCtrl can generate edited image with both high style fidelity and high recognition accuracy, which is novel.\n3.\tThe paper constructs the first real-world image-pair dataset for STE, which is contributed to the community.\n4.\tThe experimental results show that the proposed method surpasses previous methods with large margins in most cases.", "weaknesses": "1.\tSeveral hyperparameters are included in the TextCtrl. I am not sure is there any experience for users to set up these hyperparameters for different datasets.\n\n2.\tSome figures could be improved. For example, Figure 3 is somewhat not easy to follow; highlighting the pretrained parts and the trainable parts would make it clearer. Figure 6 appears chaotic, it may be better to choose fewer texts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review. We are encouraged by the positive comments on novelty of the method and the contribution of the proposed dataset. The concerns are taken care to address point by point in the following.\n\n\n> **[W1]: Discussion about the hyperparameters in TextCtrl.**\n\n[A1]: Thanks for the question. Briefly, we have made an effort to reduce the demand for manually setting hyperparameters and aligned the rest to regular settings. Specifically, \n\n-  In pretraining, we revise the max-length for text structure encoder to 24 since it focuses on character-level representation of words rather than long context. \n\n-  For training and sampling, we follow the regular settings according to [*1*]. \n\n-  For integration in inference, as shown in ***Paper Fig.3(d)*** and ***Paper Appendix Alg.1***, we set the initial intensity parameter $\\lambda=0$ and $\\mu=1$ to avoid the distraction of self-attention in early sampling stages, facilitating text structure initialization. Subsequently, $\\lambda$ is automatically calculated by the structural similarity of intermediate latent and target text with $\\mu=1-\\lambda$, which allows an adaptive integration and avoids manual settings.\n\nThrough aforementioned settings, the hyperparameters are determined and they are set consistently to perform stable and robust editing, irrespective of different datasets. \n\n\n\n> **[W2]: Figure 3 and figure 6 could be improved.**\n\n[A2]: Thanks for the suggestion on improving ***Paper Fig.3/6*** for a more intuitive presentation. Briefly, ***Paper Fig.3(a)(b)*** depicts the pretraining of text encoder *TE* and the style encoder *SE* respectively, while ***Paper Fig.3(c)(d)*** presents the training (with *TE* frozen) and inference of the whole framework. We will improve the figures in the revised paper.\n\n\n\n> **[Q1]: Can this method be used on Chinese scenes?**\n\n[A3]: Although the discussion in the paper mainly focuses on English scenes, our method is promising to realize editing on Chinese scenes by simply replacing the dataset. We highlight the effectiveness of the proposed text structure pretraining on aligning the character-level representation and the visual glyph structure, as well as the text style pretraining on disentangling text style. Since both modules are trained solely on synthetic data, they can be easily modified to fit in another language scenes. In addition, the proposed inference control effectively leverages the style information from real text image, which further enhances the model's adaptability to different scenes.\n\n\n\n> **[Q2]:The authors reference the way of structure condition insertion in ControlNet to feed style condition into TextCtrl, i.e., zero convolution and add, rather than structure condition. Could the authors explain this modification? BTW, could the authors introduce more experimental settings for ControlNet in Table 3, including inputs?**\n\n\\[A4\\]: Sure. We will first explain the reason for the modification and later detail the ControlNet in ***Paper Tab.3***.\n\nAs stated in ***Paper Sec.3***, TextCtrl is built based on a synthesis manner, wherein text is the essential product of synthesis/generation while style (e.g., font, background) serves as the additional reference. As a result, text structure is introduced through cross-attention to enable the basic generation of text image while style is inserted through ''zero convolution and add'' in decoder for reference. In fact, rather than modified, the design of TextCtrl coincides with ControlNet [2] which introduces additional reference to a foundational generative model.\n\nIn the ablation study of ControlNet, we follow the implementation settings and empirical suggestions in [*2*]. Concretely, the style encoder is replaced with a vanilla Stable Diffusion encoder, serving as the ControlNet module. The module is initialized with the pretrained weight of SD encoder and the input includes the style text image $I_{source}$, the noised latent $z^{t}$, timestep $t$ and text structure embedding $C_{struct}$. During training and sampling, $I_{source}$ is first encoded and integrated with $z^{t}$, which subsequently interacts with $t$ and $C_{struct}$ in the ControlNet module and finally inserts into diffusion generator through the ''zero convolution & add'' process.\n\n\n\n> **[Q3]: In Fig.3, why use \"sifted\" for c_texture but \"Despairing\" for c_spatial?**\n\n[A5]: Thanks for your detailed review of ***Paper Fig.3(b)***. \n\nBriefly, both $c_{texture}$ and $c_{spatial}$ are encoded from $I_{source}$ *(\"Despairing\")*. The image of *\"Sifted\"* is rather integrated with $c_{texture}$ than used for $c_{texture}$ in respective task.\n\nSpecifically, for an input $I_{source}$ *(\"Despairing\")*,  \n\n- $c_{spatial}$ is leveraged in *Text Removal* and *Text Segmentation* for capturing the spatial style. As a result, a pure background and a segmentation mask of *\"Despairing\"* are expected as the direct output. \n\n- $c_{texture}$ is leveraged in *Color Transfer* and *Font Transfer* for capturing the texture style. With $c_{texture}$ encoded from the stylized image of \"Despairing\", to avoid degeneration of the transfer model into an identity mapping network, we turn to a different text image *\"Sifted\"* (synthesized in the same color/font) as the sub-models input/output with $c_{texture}$ as conditional style guidance. The substitution enforces the disentanglement of color/font from the source image context and the implementation details are given in ***Paper Sec.3.2***.\n\nThe aforementioned processes ensure the encoded features of style encoder and the objectives of each task are closely associated with the text style disentanglement ability. \n\n> **References**\n\n[1] Rombach et al. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR, 2022.\n\n[2] Zhang et al. Adding Conditional Control to Text-to-Image Diffusion Models. ICCV, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their thoughtful and constructive feedback. It's encouraged to hear from the reviewers that

- The Model TextCtrl: *"Innovative Approach; addresses weak correlation between text prompts and glyph structures; generate edited image with both high style fidelity and high recognition accuracy;"* [Reviewer Ktb6, mqKG, 25Re]

- The Benchmark ScenePair: *"enable the comprehensive assessment on real-world images; providing a more holistic assessment of STE methods; is contributed to the community;"* [Reviewer MkTr, Ktb6, rTPt, mqKG, 25Re]

- The Evaluation: *"Comprehensive; extensive and well-discussed; demonstrate notable performance improvement;"* [Reviewer Ktb6, rTPt, mqKG]

- The Paper: *"is well written; the presentation is clear;"* [Reviewer rTPt, 25Re]

In response to the review, we provide individual replies below to address the remaining concerns from each reviewer. Notably, some concerns share a common topic yet are raised from different perspectives, to which our respective responses could serve as a mutual reference. 

- Research and discussion of text style disentanglement pretraining; [A2 for Reviewer Ktb6, A1 for Reviewer rTPt, A5 for Reviewer 25Re]  

- Reason and detail for ControlNet module involved in ablation study; [A1 for Reviewer mqKG, A4 for Reviewer 25Re]


Besides, an additional uploaded one-page pdf, some references and the main paper are used in our responses for clarity and precision. 

- Figures from uploaded one-page pdf are denoted as ***Uploaded Pdf Fig.X***; 

- Figures and tables from the main paper are denoted as ***Paper Fig.X/Tab.X***;

- Figures from the appendix of the main paper are denoted as ***Paper Appendix Fig.X***;

- Reference paper is denoted as [*X*];

We hope our responses could resolve your concerns. Please do not hesitate to let us know if you have further questions.

Meta-review: All five reviewers recommend accepting this paper (2 “Accept”, 1 “Weak Accept” and 2 “Borderline Accept”). The reviewers mentioned that the key idea of the proposed method is novel, the constructed ScenePair dataset is valuable, and the results of this paper are impressive. The AC agrees with the reviewers that the paper is interesting and deserves to be published in NeurIPS 2024.
Decision: Accept (spotlight)