DiffUTE: Universal Text Editing Diffusion Model
Authors: Haoxing Chen
Zhuoer Xu
Zhangxuan Gu
jun lan
Xing Zheng
Yaohui Li
Changhua Meng
Huijia Zhu
Weiqiang Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion model
text editing
self-supervied learning
TL;DR: This paper propose a unified model to edit any texts in images.

Abstract: Diffusion model based language-guided image editing has achieved great success recently. However, existing state-of-the-art diffusion models struggle with rendering correct text and text style during generation. To tackle this problem, we propose a universal self-supervised text editing diffusion model (DiffUTE), which aims to replace or modify words in the source image with another one while maintaining its realistic appearance. Specifically, we build our model on a diffusion model and carefully modify the network structure to enable the model for drawing multilingual characters with the help of glyph and position information. Moreover, we design a self-supervised learning framework to leverage large amounts of web data to improve the representation ability of the model. Experimental results show that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity. Our code will be avaliable in \url{https://github.com/chenhaoxing/DiffUTE}.

DiffUTE: Universal Text Editing Diffusion Model
Haoxing Chen 12, Zhuoer Xu' Zhangxuan Gul Jun Lan', Xing Zheng',
Yaohui Li, Changhua Meng', Huijia Zhu', Weiqiang Wang'
1 AAnt Group *Nanjing University
hx www.hhhhhtaaalll..mmmmom.. com, {xuzhuoer guzhangxuan (aaliaannggx)aannttege gaxx)xxttgrru.ccoo
Abstract
Diffusion model based language-guided image editing has achieved great suc- cess recently. However, existing state-of-the-art diffusion models struggle with rendering correct text and text style during generation. To tackle this problem, we propose universal self-supervised text editing diffusion model (DiffUTE), which aims to replace or modify words in the source image with another one while maintaining its realistic appearance. Specifically, we build our model on a diffusion model and carefully modify the network structure to enable the model for drawing multilingual characters with the help of glyph and position information. Moreover, we design self-supervised learning framework to leverage large amounts of web data improve the representation ability of the model. Experimental results show that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity. Our code will be avaliable in https://githi: com/ http:/chxxii//iiffff...
1 Introduction
Due to the significant progress of social media platforms and artificial intelligence Xu et al. [2022a] Gu al. [2023), Zhang et al. [2022a]. image editing technology has become common demand. Specifically, Al-based technology Niu et al. (2023), Chen et al. 2023 has significantly lowered threshold for fancy image editing, which traditionally required professional software and labor- intensive manual operations. Deep neural networks can now achieve remarkable results in various image editing tasks, such as image inpainting Feng et al. [2022], image colorization Zhang et al. 2022b). and object replacement Kwon and Ye [2022] by learning from rich paired data. Futhermore, recent advances in diffusion models Brack et al. 2023), J. Brooks et al. [2023] Saharia et al. [202aa] enable precise control over generation quality and diversity during the diffusion process. By incor- porating a text encoder, diffusion models can be adapted to generate natural images following text instructions, making them well-suited for image editing.
Despite the impressive results, existing image editing methods still encounter numerous challenges. As typical task, scene text editing is widely used in practical applications such as text-image synthesis, advertising photo editing, text-image correction
augmented reality translation. aims to replace text instances (i.c., the foreground) in image without compromising the background. However, the fine-graines and complex structures of text instances raise two major challenges: (i) How to transfer text style and retain background texture. Specifically, text style includes factors such as font, color, orientation, stroke size, and spatial perspective. It difficult to precisely capture the complete text style in the source image due the complexity of the background. (ii) How to maintain the consistency of the edited background especially for complex scenes, e.g., menus and street store signs.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Image
Gmmm Tagg
Grrm Bey arden studios
Gr Bay cordan studiog
Grru dey 8o0 Re studios
Grr Tey taadaa stadio
Tert boy gardem studios
'rr in Doy glar dlem studios
studtes
Tea
garden
Imsge
998:: REP. NO.
SE: REP NO.
2 REP. NO: X AMOUNT:
S REP NO.
BE REP NO.
3E: REP. NO.
AMOUUTT
20 AMOUNT:
2- AMOUNT:
AMOUNT
AMOUNT:
AMOUNT:
Text
ACAF
4-R TIPS:
4-1
TIPS
TIPS: MOSTEL
TIPS:
TIPS
TIPS.
laput
SRNett
ControlNet
SD2
SD2-FT
Ours
Figure 1: Examples of text editing. DiffUTE achieves the best result among existing models.
Numerous studies formulate scene text editing style transfer task and approach it by generative models like GANs Wu et al. [2019), Qu et al. [2023]. Typically, a cropped text region with the target style is needed as the reference image. Such methods then transfer rendered text in the desired spelling to match the reference image's 's style and the source image's background. However, the two major challenges for scene text editing remains. (i) These methods are currently constrained to editing English and fail to accurately generate complex text style (c.g.. Chinese). (ii) The process of cropping, transferring style and blending results in less natural-looking outcomes. End-to-end pipelines are needed for the consistency and harmony.
To address the above issues, we present DifFUTE, general diffusion model designed to tackle high-quality multilingual text editing tasks. DifTUTE utilizes character glyphs and text locations in source images auxiliary information to provide better control during character generation. As shown in Figure T our model can generate very realistic text. The generated text intelligently matched to the most contextually appropriate text style and seamlessly integrated with the background while maintaining high quality.
The major contribution of this paper is the universal text edit diffusion model proposed to edit scene text images. DiffUTE possesses obvious advantages over existing methods in several folds:
We present DiffUTE, novel universal text editing diffusion model that can edit any text in any image. DiffUTE generates high-quality text through fine tne-grained control of glyph and position information. DiffUTE capable of seamlessly integrating various styles of text characters into the image context, resulting in realistic and visually pleasing outputs.
2. We design a self-supervised learning framework that enables the model to be trained with large amounts of scene text images. The framework allows the model to learn from the data without annotation, making it highly efficient and scalable solution for scene text editing.
3. We conduct extensive experiments to evaluate the performance of DifTUTE. Our method performs favorably over prior arts for text image editing, as measured by quantitative metrics and visualization.
2 Preliminaries
In this paper, we adopt Stable Diffusion (SD) Rombach et al. [2022] as our baseline method to design our network architecture. SD utilizes a variational auto-encoder (VAE) to enhance computation efficiency. Through VAE, SD performs the diffusion process in low-dimensional latent space. Specificallyy, given an input image B HxWx3 the encoder 8, of VAE transforms it into latent representation 2 E R hxwxe where a h
H W is the downsampling factor and is the latent feature dimension. The diffusion process is then executed in the latent space, where a conditional UNet denoiser benneberrger et al. [2015] cp co(2, (z,, f, is employed to predict the noise with noisy latent z,, generation condition input and current time step 1. The condition information y may encompass various modalities, e.g., natural language, semantic segmentation maps and canny edge maps. To pre- pre-processing y from various modalities, SD employs a speecfc encoder To to project y into an intermediate representation Tæ (v)) B Mxd, which is then mapped to the intermediate layers of the

---

V. where Frozen models
Leamable models
BOiSe
U-Net
OCR detector
select
1OTAL
112
Xym
Zt
2r-1
100 0.30
Glyph encader
RM 30.00
X
(a) Training process
Optional:
you change cash armouni to 29.00?
Large
Diffussio
Language
Model
(LLM)
C
detector
Glyph
RM 29.00
(b) Inference process
Figure 2: Training and inference process of our proposed universal text editing diffusion model. (a) Given an image, we first extracted all the text and corresponding bounding boxes by the OCR detector. Then, a random area selected and the corresponding mask and glyph image are generated. We use the embedding of the glyph image extracted by the glyph encoder as the condition, and concatenate the masked image latent vector Xms mask m, and noisy image latent vector z, as the input of the model. (b) Users can directly input the content they want to edit, and the large language model will understand their needs and provide the areas to be edited and the target text to DiffUTE, which then completes the text editing.
UNet via cross-attention mechanism implementing Attention( Q, K, softmax OK* v
Q = w d1(2,), K w To(y). w To(y). w(()) w(()) w(() are learnable projection matrices, d denotes the output dimension of key (K) and query (0) features, and Φ;(2,) Nxd denotes flattened intermediate representation of the UNet implementing Eg. In the scenario of text-to-image generation, the condition C = Ty( is produced by encoding the text prompts y with a pre-trained CLIP text encoder Tg- The overall training objective of SD is defined
Lsd = Bg(s).... [[|e - Eo(z,, To(y))113
Therefore, Ta and can be jointly optimized via Equation (I.
3 Universal Text Editing Diffusion Model
3.1 Model Overview
The overall training process of our proposed DiffUTE method is illustrated in Figure 2
(a). Based on the attention mechanism in SD Rombach et al. 2022] the original input latent vector z, is replaced by the concatenation of latent image vector 2, masked image latent vector xm. and text mask m. The condition C is also equipped with glyph encoder for encoding glyph image Xg- Introducing text masks and glyph information enables fine-grained diffusion control throughout the training process, resulting in the improved generative performance of the model.
3.2 Perceptual Image Compression
Following Rombach et al. 2022], we utilize a VAE to reduce the computational complexity of diffusion models. The model learns perceptually equivalent space to the image space but with

---

significantly reduced computational complexity. Since the VAE in SD is trained on natural images, its ability to restore text regions is limited. Moreover, compressing the original image directly through the VAE encoder causes the loss of dense text texture information, leading to blurry decoded images by the VAE decoder. To improve the reconstruction performance of text images, we further fine-tune the VAE on text image datasets. As shown in our experiments (Section 4.4), training VAE directly on the original image size lead to bad reconstruction results, i.c., unwanted patterns and incomplete strokes. We propose a progressive training strategy (PTT) in which the size of the images used for training increases as the training proceeds. Specifically, in the first three stages of training, we randomly crop images of sizes S/8, S/4 and S/2 and resize them to S for training, where S is the resolution of the model input image and S = H = W. Thus, the tuned VAE can learn different sizes of stroke details and text recovery. In the fourth stage, we train with images of the same size as the VAE input to ensure that the VAE can predict accurately when inferring.
3.3 Fine-grained Conditional Guidance
The pixel-level representation of text images differs greatly from the representation of natural objects. Although textual information consists of just multiple strokes of wwo-dimensional structure, it has fine-grained features, and even slight movement or distortion lead to unrealistic image generation. In contrast, natural images have a much higher tolerance level as long as the semantic representation of the object is accurate. To ensure the generation of perfect text representations, we introduce two types of fine-graing iinegrraieed guidance: positional and glyph.
Positional guidance. Unlike the small differences between natural images, the latent feature distri- butions of character pixels differ dramatically. Text generation requires attention to specific local regions instead of the existing global control conditions for natural images Zhang and Agrawala [2023), Mou et al. [2023), Cheng et al. [023] (e.g., segmentation maps, depth maps, sketch and grayscale images). To prevent model collapse, we introduce position control to decouple the distribu- tion between different regions and make the model focus on the region for text generation. As shown in Figure
(a). binary mask is concatenated to the original image latent features.
Glyph guidance. Another important issue is to precisely control the generation of character strokes. Language characters are diverse and complex. For example, a Chinese character may consist of more than 20 strokes, while there are more than 10,000 common Chinese characters. Learning directly from large-scale image-text datasets without explicit knowledge guidance is complicated. [iu et al. [202aa] proposes that the character-blinded can induce robust spelling knowledge for English words only when the model parameters are larger than 100B and cannot generalize well beyond Latin scripts such as Chinese and Korean. Therefore, we heuristically incorporate explicit character images as additional conditional information to generate text accurately into the model diffusion process. As shown in Figure 2 (a), we extract the latent feature of the character image as a control condition.
3.4 Self-supervised Training Framework for Text Editing
It is impossible to collect and annotate large-scale paired data for text image editing, i.e., {(x,.xg. It may take great expense and huge labor to manually paint reasonable editing results. Thus, we perform self-supervised training. Specifically, given an image and the OCR bounding box of a sentence in the image, our training data composed of {(xm. Xg. m)
For diffusion-based inpainting models, the condition C' is usually text, which is usually processed by a pre-trained CLIP text encoder. Similarly, a naive solution is directly replacing with an image encoder. To better represent glyph images, we utilize the pre-trained OCR encoder Li et al. [2023] as the glyph encoder. Such naive solution converges well on the training set. However, the generated quality is far from satisfactory for test images. We argue that the main reason is that the model learns a mundane mapping function under the naive training scheme: Xg + X, - (1 m) = Xy- impedes the network from understanding text style and layout information in the image, resulting in poor generalization. To alleviate such issue, we use a uniform font style (i.e., "arialuni") and regenerate the corresponding text image, as shown in Figure 2 (a) with the example of "RM 30.00". Thus, we prevent the model from learning such a trivial mapping function and facilitate model understanding in self-supervised training manner.

---

Table 1: Quantitative comparison across four datasets. indicates the second best method.
1 means the higher the better, underline
Web
ArT
TextOR
ICDARI3
Average
Model
OCRT
Corf
OCRT
Cort
OCRT
Cort
OCRI
Cort
OCRT
Corl
Pix2Pix
17.24
16
13.52
11
15.74
14
15.48
15
15.50
14
SRNet
30.87
42
31.22
44
32.09
41
30.85
44
31.26
42.8
MOSTEL
48.93
61
60.73
68
45.97
53
53.76
59
52.35
60.3
SDI
4.32
5
5.98
7.43
7
3.64
5.34
6.3
SD2
5.88
7
6.94
9.29
11
5.32
8
6.86
8.8
SD1-FT
33.53
45
33.25
49.72
46
28.76
32
36.32
42.5
SD2-FT
46.34
51
49.69
44
62.89
59
46.87
46
51.45
50
DifiSTE
48.55
50
82.72
84
84.85
85
81.48
81
74.30
75
84.83
85
85.98
87
87.32
88
83.49
82
85.41
85.5
DifTUEE
+35.90
+24
+3.26
+3
+2.47
+3
+2.01
+1
+11.11
+10.5
Our self-supervised training process summarized as follows: (1) An ocr region randomly selected from the image and the corresponding text image is regenerated with a uniform font style. (2) The regenerated character image Xg is fed into glyph encoder to get condition glyph embedding eg. (3) The masked image latent vector mask m and noisy image latent vector is concatenated to form a new latent vector z; Concat(Xm: m, 2,). After dimension adjustment through convolution layer, the feature vector & = Conv(zz) is fed into the UNet the query component. Consequently, the training objective of DifTUTE is:
DDMUTE Bg, (g.. -........--N...... [[le - Eg Xg. Xm, m m)
3.5 Interactive Scene Text Editing with LLM
To enhance the interaction capability of the model, introduced the large language model (LLM), iee., ChatGLM Zeng et al. (2023). Moreover, we fine-tuned ChatGLM using the extracted OCR data to facilitate better understanding of structured information by ChatGLM, The inference process of DiffUTE is show in Figure (b). We first provide the OCR information extracted by the OCR detector and the target that the user wants edit with LLM, which will return the target and its corresponding bounding box. Then, we use bounding boxes to generate mask and masked images, and generate images through complete diffusion process ( .... 0}) by DDIM Song et al. 2020) sampling strategy. By using ChatGLM to understand natural language instruction, we avoid requiring users provide masks for the areas they want edit, making our model more convenient.
4 Experiments
4.1 Data Preparation
Due the lack of large-scale datasets for generating text image compositions, collect 5M images
by combining the web-crawled data and publicly available text image datasets, including CLDA Li]
XFUND Xu al. [0022)) Publ
PubLay ayNet /Zhong al. [2019) and ICDAR series competitions Zhang
et al. [2019] Nayel et al. (2019), Karatzas et al. ]. to prepare our training dataset. To verify
the effectiveness of our model. we randomly selected 1000 images from ArT Chng et (0119]
Tex1OCR Singh et al. 202T) ICDARI3 Karatzas et al. (2013) and web data collected by oursel ves to form the set, respectively. All the images are cropped/resized to 512 512 resolution as model
inputs.
4.2 Implementation Details and Evaluation
Implementation details. Our DiffUTE consists of VAF, glyph encoder and UNet. To obtain better reconstruction ability for text images, we first fine-tuned the VAE, which is initialized from the
5

---

Salesperson
Campu
Visa Platinum
56
Input
Description
HB)
FOAMBOARD
BURREN
1234
DESIGN
plected
2023-05-20
LAKERS
Miracle
Venue
8888
Selesperson
Campu
SRNNE
8
Mni adia BURREN
TCHe
FOAMBOARD
MILLER 16/22
ptected
Salesparson
Campt
Visa Platinum
ControlNet MOSTHL
66
Description
IMicaalié BURREN
029-0:20
COMAC
Varée
123%
FOAMBOARD
DESIGN
otected
Salesperson
Campt
Description
+Miac Ho BURREN
Hi)
LAKERS
Yeute
1234
FOAMBOARD
DESIGN
ptected
Salesperson
Campu
Visa Platinum
56
SD2
Description
Mitaacié! BURREN
FOAMBOARD
Vmut
5988
1234
DESIGN
olected
Salesperson
Campu
Visa Platinum
SD2-FT
Description LAKERS
M'rade BURREN
003-20
VaneG
8858
1234
DESIGN
otected
Salesperson
Campt
Visa Platinum
55.00 00
Ours
Descrigtion
Miracle BURREN
A))
Venue
1334
FOAMBOARI
DESIGN
otected
Figure 3: Visualization comparison. Our DiffUTTE beats other methods with significant improve- ment.
Table 2: Ablation study results. (Pos.: position control, Gly.: Glyph control.)
Web
ArT
Tex1OC
ICDARI3
Average
Model
OCR1
Cort
OCRT
Corf
OCR1
Corf
OCRT
Cort
OCR1
Cort
w/o PTT
44.73
47
45.29
41
60.83
52
41.22
39
48.02
44.8
w/o Pos.
49.84
50.89
47
65.72
49.72
47
54.04
52.5
w/o Gly.
46.34
51
49.69
44
62.89
59
46.87
46
51.45
50.0
84.83
85
85.98
87
87.32
88
83.49
82
85.41
85.5
DiffUTE
+34.99
+32
+35.09
+40
+21.60
+25
+33.77
+35
+31.37
+33
checkpoint of stableddiftsion-2-inaantte of 48 and a learning rate of 5e-6. We use a pre-trained OCR encoder as our glyph encoder, i.c., TROCR Li et al. 2023). During the training of DiffUTE, set the batch size to 256, the learning rate to le-3, and the batch size to 5. Note that the weights the glyph encoder and VAE were frozen during the training of DiffUTE.
The VAE is trained for three epochs with a batch size
Evaluation and metrics. In our evaluation, we evaluate the accuracy of the generated text. We report OCR accuracy, calculated separately using pre-trained recognition model Fang et al. [2021] and human evaluation of the correctness between the generated text and the target text, denoted as OCR and Cor, respectively.
hhtg::/hwnggigggaaooooaaaiiiiiih i-ipaanniig

---

FARNSCROUGGH AIRSHOW
FARNBOROUGH AIRSHOW
FARNBOROUGH AIRSHOW
FARNBOROUGH AIRSHOW
FARNBOROUGH AIRSHOW
Image
Elying
daily glow programme
daily resy programme
daily glou programme
daily programme
Text
rosy glow
Image
EAPTY
ROPNY
MAPDO
Happy
BEEN FREN FIERCE
IMPORTED BEER
IMPORTED BEER FROM MENCO
IMPPORTED BEER FROM IEEIIOO
IMPORTED BEER FROM IEEICO
Text
Happy
Image
Firstry Eastern
Bus Timess
young
Eastemn National
young
Eastern National
YOUr
Eastern National
young Eastem National
Sumno
Bus Tirees
uss Times
Bss Eirres
Bus Tirres
Text
Image
you Know
you Know anyone starting Paraselize this year? w/o Pos.
Do you Know anyone starting Parasdize this year?
you anyone starting Pizzolize
Do you Know anyone starting Parasolize this year?
Text
Parasolize
this year?
Input
w/o Gly.
PTT
DinUTE
Figure Sample results of ablation study.
Baseline methods. We compare DiffUTE with state-of-the-art scene text editing methods and diffusion models, i.e., Pix2Pix Isola et al. [017], SRNet Wu al. (019), MOSTEL. Qu al. [2023] Rombach et al. [2022]. Control Net Zhang and Agrawala 2023 and DifSSTE et al. 20023] Pix2Pix is an image translation network. To make Pix2Pix network implement multiple style translations, we concatenate the style image and the target text in depth as the network input. Training of SRNet requires different texts to appear in the same position and background, which does not exist in real-world datasets. Therefore, images for fine-tuning. For MOSTEL, we fine-tuned it on our dataset. For SD, we selected two baseline methods, i.e., gabblediffusion-iiee impainting (SD1) and stableddiffssion 2-inaanntin (SD2). For fair comparison, we fine-tuned SD1 and SD2 by instruction tuning. The resulting models are termed as SD1-FT and SD2-FT. In the NLP field, instruction tuning techniques used to train models to perform tasks based on task instructions. We aim to accurately map text instructions to corresponding text edits using the SD model. To achieve this, constructed dataset for fine-tuning. Each sample in the dataset consists of a language instruction describing the target text, mask, and the ground truth. ControlNet is image synthesis method that achieves excellent controllability by incorporating additional conditions to guide the diffusion process. To adapt this method to our text editing problem, we take the glyph image as the input to the ControlNet network. DifiSTE introduces an instruction tuning framework to train the model to learn the mapping from textual instruction to the corresponding image, and improves the pre-trained diffusion model with a dual encoder design. We followed the original setup to train DiffSTE.
use synthtiger Yim al. [2021] synthesize
4.3 Comparison results
The quantitative results for text generation are shown in Table 1 We can find that our DiffUTE achieves state-of-the-art results all datasets. For example, DiffUTE improves average OCR

---

PHAM LANIIC ALN BHO
VIAMA AAML AGU BHO
VIAM MAMI AL BHO
SLE
MOROONES
ELEW
0100
ORT 8:01
2010 6O 195
CO .90
SD2-VAE
Ours-w/o PTT
Ours
SD2-VAE
Ours-w/o PIT
Ours
Figure 5: Examples of image reconstruction with our method DiffUTE.
accuracy and human evaluatee text correctness by 14.95% and 14.0% compared with the second best method DifISTE. Moreover, our method achieves better results than the diffusion model and the fine- tuned diffusion model because our fine-grained control can provide the model with prior knowledge of glyph and position. Furthermore, the poor performance of the diffusion model for instruction fine-tuning also demonstrates the superiority of our inference approach combining ChatGLM, which can achieve better editing effects.
We further conducted visualization experiment. As shown in Figure + our method successfully achieved the transfer of foreground text and background texture, resulting in a regular textual structure and consistent font with the original text. Moreover, the background texture was clearer, and the overall similarity with real images was improved. In contrast, the results edited using the diffusion model often deviated from the target text, further validating the effectiveness of the glyph condition we introduced. Furthermore, other methods perform poorly when faced with more challenging Chinese text generation tasks, whereas DiffUTE still achieves good generation results.
4.4 Ablation results
The Ablation studies examine three main aspects, namely 1) the effectiveness of the progressive training strategy of VAE, and 2) the impact of position control and glyph control on the image generation performance of DiffUTE. The experimental results are shown in Table Figure 4 and Figure
Progressive training strategy. Without using the progressive training strategy, the editing results become distorted and the accuracy of text generation significantly decreases. The reason for such poor results is the complexity of the local structure of the text, whereby the VAE may need to learn the reconstruction ability of local details efficiently by focusing when there are too many characters in the image. And using proposed progressive training strategy, the reconstruction ability of the model is significantly improved and more realistic results are obtained. The experimental results validate the effectiveness of this strategy and highlight the pivotal role of VAE in the diffusion model.
Fine-grained control. When position control is not used, the mask and masked images at the input of the UNet are removed. When glyph control is not used, the latent code obtained from the text through the CLIP text encoder is used as the condition. When position control and glyph control are not used, there is a significant drop in performance. For example, when position control is not used, the OCR accuracy of the model drops by 36.7%, and the Cor drops by 38.6%. When glyph control is not used, the model cannot generate accurate text and the OCR accuracy of the model drops by 39.8%. and the Cor drops by 41.5%. These results show that position control can help the model focus on the area where text is to be generated, while glyph control can provide prior knowledge of the shape of the characters to help the model generate text more accurately.
Visualisation. In Figure 6 we provide some examples edited by DifFUTE. DifTUTE consistently generates correct visual text, and the texts naturally follow the same text style, i.e. font, and color, with other surrounding texts. We can see from the experiment that Dif has a strong generative power. (i) In sample NI, DiffUTE can automatically generate slanted text based on the surrounding text. (ii) As shown in sample N2, the input is 234, and DiffUTE can automatically add the decimal point according to the context, which shows that DiffUTE has some document context understanding ability. (ii) In the sample CN4, DifTUTE can generate even artistic characters very well.
8

---

27
665
38.90
34
0.90
46.20
Imagee
Toxt Imagge
110
42
100
140
61.34
110
53
42
1.00
1.25
66.00
16.70
53
800
50.40
66
180
61.34
00
1335
125
39
M1
2.34
685
5..40
50.40
12.50
Tout
1234567890
234
125
79.60
NO
N2
8 23456789522
HEC. ILEN
Imnage
00077035
AS 00077035
OIOOAO-- JRHEINER
5
76.08844 M8031
* 00070355
Text
12345
1234567895
74.088ml 07228944
123456
N3
N4
Fatel
Farkol AD
SCTV4
7/50
Intgee
Lingvin Templeg
Caroga Pharma
Imagee
BALL
hending
7:25
Laborex Theornis Limitd
Laborzx Tanzanis Limited
BRLL
7/56
AlBuel
Ant Group
Lingyin Temple AlRazi
Nanjing
7/42
8O
Tetal Bart
35,000
0000
Cartechiieocc cut ailile 4 &
West Lake noksow Ders
37506-2061
Iniage
Txtal Fas 4i
Discurt
Discount
AS 10 Wi
C-0
0.00
Inc 8t
19
15.00
0.00
1.80
Ad
real Tota
CACH OHASS
Total
Round Ad,
Tota
70 With
Tonpe
GRR PARBAISS
CASH CHANGE
Lumi
Lake
Confucius Tempe
E3
E MH
:: 8013089
5 8213059
Imiige
* #
Hi, BIRK 29
FM115
BQ13069
19999 Mddeelll-
Text
BQ13059
ELA
El Ei #8.2 billbbll
Tood
CNN
CNI
CN2
RP __088
PP0998
AR SI +
#88: gaana
10223
E
10223
Imarge
XB00.00
BM
MARK
EET
FR.
V800.00
RINO
R
10.000
TE
Texi
CN3
CN4
CN5
Figure 6: More visualization results of text editing.
5
Related Works
5.1 Scene Text Editing
Style transfer techniques based on Generative Adversarial Networks (GANs) have gained widespread popularity for scene text editing tasks Roy et al. [2020), Huang et al. [2022]. Kong et al.) [2022). Lee et al. [202T], (Simoda et al. [2021] J. Yang et al. [2020] Zhan et al. [211]] These methods typically involve transferring text style from a reference image to a target text image. STEFANN Roy et al. [2020], for instance, leverages font-adaptive neural network and a color- pollo-preserving model
to edit scene text at the character level. Meanwhile, SRNet Wu et al. [2019) employs two-step
approach that involves foreground bacooundu--bagoouudd separation and text spatial alignment, followed by

---

fusion model that generates the target text. Mostel Qu et al. [2023] improves upon these methods by incorporating stroke-level information enhance the editing performance. However. despite their reasonable performance, these methods are often constrained in their ability to generate text in arbitrary styles and locations and can result in less natural-looking images.
5.2 Image Editing
Text-guided image editing has attracted increasing attention in recent years among various semantic image editing methods. Early works utilized pretrained GAN generators and text encoders to progressively optimize images based on textual prompts Bau et al. [2021], Gal et al. (2021). Pérez manipulation methods encounter difficulties in editing images with complex scenes or diverse objects, owing to the limited modeling capability of GANs.
al.] [003]. However, these
The rapid rise and development of diffusion models Rombach al. [2022], Saharia al.) 2022b).
Ruiz et al. [2023] have demonstrated powerful abilities in synthesizing high-quality and diverse images. Many studiesBrack et al. (2023), Brooks et al. [2023] have employed diffusion models for text-driven image editing. Among various diffusion models, Stable Diffusion Rombach et al. [0022] one of the state-of-the-an models, which compresses images into low-dimensional space using an auto-encoder and implements effective text-based image generation through cross-aie coss-attention layers. This model can easily adapt to various tasks, such as text-based image inpainting and image editing.
However, it has been observed that diffusion models exhibit poor visual text generation performance and are often prone to incorrect text generation. Only few studies have focused on improving the text generation capability of diffusion models. Recently, one study trained a model to generate images containing specific text based on a large number of image-text pairs Liu al. [2022b). However, this work differs from ours in terms of application, as they focus text-to-image generation, while ours concentrates on editing text in images. Another ongoing work, ControlNet Zhang and Agrawala 2023). has demonstrated remarkable performance in image editing by providing reference images such as Canny edge images and segmentation maps. While ControlNet achieves remarkably impressive results, it performs poorly on text editing tasks. To obtain better editing results, incorporate auxiliary glyph information into the conditional generation process and emphasize local control in all diffusion steps.
5.3 Large Language Model
Large language models (LLMs) refer to language models that contain billions (or more) of parameters,
which are trained on massive amounts of text data, such as models like GPT-3 Brown et al. [2020).
Galactica [Talorr et al. [2022). I.LaMA Touvron et al. [2023] and ChatGL.M Zeng et al. [0023
Among them, ChatGLM billion-scale language model with rudimentary question -nwsoornnnnwwiing and
conversational capabilities. differs from BERT Devlin et al. [2018), GPT-3 and T5 Xue et al.) [202]]
architectures and is self-regressive pre-training model that inc includes multiple objective functions. In this paper, we use ChatGLM to enhance the interaction capability of our model.
6 Conclusion and Limitations
In this paper, we argue that the current diffusion model can not generate realistic text in images. To tackle this problem, we present a novel method DifTUTE, a diffusion-basse universal text editing model. DifTUT generates high-quality text through fine-grained control of glyph and position information, and benefits from massive amounts of text images through a self-supervised training approach. Moreover. by integrating large language model (i.e., ChatGLM), we can use natural language to edit the text in images, enhancing the editing usability and convenience of model. Extensive experiments have shown that DiffUTH excels textual correctness and image naturalness.
The main limitation of our method is that the accuracy of generated text will decrease as the number of characters be edited in the image increases. This is due to the fact that the number of characters increase, the spatial complexity of the characters will also increase, making the generation process more challenging. Therefore, our future work will focus on improving the generation quality and solving the problem of rendering long texts.
10

---

References
Zhuoer Xu, Guanghui Zhu, Changhua Meng. Zhenzhe Ying, Weiqiang Wang, Ming Gu, Yihua Huang. et al. A2: Efficient automated attacker for boosting adversarial training. Advances in Neural Information Processing Systems, 35:22844-22855, 2022a.
Zhangxuan Gu, Zhuoer Xu, Haoxing Chen, Jun Lan, Changhua Meng, and Weiqiang Wang. Mobile user interface element detection via adaptively prompt tuning. In CVPR, pages 11155-11164, 2023.
Chao Zhang, Huaxiong Li, Yang Gao, and Chunlin Chen. Weakly-supervised enhanced semantic- aware hashing for cross-modal retrieval. JEEE Trans. Knowl. Data Eng., 35(6))6475-6488, 2022a.
Li Niu, Junyan Cao, Wenyan Cong. and Liqing Zhang. Deep image harmonization with learnable augmentation. In ICCV, pages 7482-7491, 2023.
Haoxing Chen, Zhangxuan Gu, Yaohui Li, Jun Lan, Changhua Meng. Weiqiang Wang, and Huaxiong Li. Hierarchical dynamic image harmonization. ACM MM, 2023.
Tingliang Feng, Wei Feng, Weiqi Li, and Di Lin. Cross-image context for single image inpainting. In NeurlPS, 2022.
Jiangning Zhang, Chao Xu, Jian Li, Yue Han, Yabiao Wang, Ying Tai, and Yong Liu. Sesnet: An efficient paradigm for learning simultaneously image colorization and super-resolution. In AAAI, pages 3271-3277 2022b.
Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with a single text condition. In CVPR, pages 18041-18050. 2022.
Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. Sega: Instructing diffusion using semantic dimensions. arXiv preprins arXiv:2301.12247. 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, volume 35, pages 36479-36494, 2022a.
Liang Wu, Chengquan Zhang, Jiaming Liu, Junyu Han, Jingtuo Liu, Errui Ding, and Xiang Bai. Editing text in the wild. In ACM MM, pages 1500-1508, 2019.
Yadong Qu, Qingfeng Tan, Hongtao Xie, Jianjun Xu, Yuxin Wang, and Yongdong Zhang. Exploring stroke-level modifications for scene text editing. In AAAI, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234 241, 2015.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. 2023.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08455. 2023.
S Cheng, Yu-Jic Chen, Wei-Chen Chiu, Hung- Yu Tseng, and Hsin-Yine Ying Lee. Adaptively-rcalistie image generation from stroke and sketch with diffusion model. In WACV, pages 4054 4062, 2023.
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant, Character-aware models improve visual text rendering. arXiv preprint 2rXv:22212.10562 2022a.

---

Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li and Furu Wei. Trocr: Transformer-baae based optical character recognition with pre-trained models. In AAAI, 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GL.M-130b: An open bilingual pre-trained model. In ICLR, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020.
Hang Li. Cdla: A chinese document layout analysis (cdla) dataset. [Online]. Available: 7/ Ngithub. com/buptlihang//Oa CDLA Accessed 2021.
https
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. Xfund: A benchmark dataset for multilingual visually rich form understanding. In Findings of ACL, pages 3214-3224, 2022b.
Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. In ICDAR, pages 1015-1022, 2019.
Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, Mingkun Yang, et al. Icdar 2019 robust reading challenge on reading chinese text on signboard. In ICDAR, pages 1577-1581, 2019.
Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowdhury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, Christoohh Burie, Cheng- lin Liu, et al. lcdar2019 robust reading challenge munti-lingual scene text detection and recognition mrc-mtt-209. In ICDAR, pages 1582-1587,20 2019.
Dimosthenis Karatzas, Lluis GomezzBigorda: Anguelos Nicolaou, Suman K. Ghosh, Andrew D. Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, Faisal Shafait, Seüchhi Uchida, and Ernest Valveny. ICDAR 2015 competition on robust reading. ICDAR, pages 1156-1160, 2015.
Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang. Shuaitao Zhang, Junyu Han, Errui Ding. et al. lcdar2019 robust reading challenge on arbitrary-shaped text-rre-art. In ICDAR, pages 1571 1571-1576, 2019.
Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojeiech Galuba, and Tal Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In CVPR, pages 8802- 8812, 2021.
Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In CVPR, pages 7098-7107, 2021.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, pages 1125-1134, 2017.
Jiabao Ji, Guanhua Zhang, Zhaowen Wang, Bairu Hou, Zhifei Zhang, Brian Price, and Shiyu Chang. Improving diffusion models for scene text editing with dual encoders. arXiv preprint arXiv:2304.05568, 2023.
Moonbin Yim, Yoonsik Kim, Han-Cheol Cho, and Sungrae Park. Synthtiger: Synthetic text image generator towards better text recognition models. In ICDAR, pages 104, 2021.
Prasun Roy, Saumik Bhattacharya, Subhankar Ghosh, and Umapada Pal. Stefann: scene text editor using font adaptive neural network. In CVPR, pages 13228-132377 2020.
Qirui Huang, Bin Fu, Yu Qiao, et al. Gentext: Unsupervised artistic text generation via decoupled font and texture manipulation. arXiv preprint arXiv:2207.006443 2022.
12

---

Yuxin Kong, Canjie Luo, Weihong Ma. Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, and Lianwen Jin. Look closer to supervise better: One-shot font generation via component-bes based discriminator, In CVPR, pages 13482-13491,
Junyeop Lee, Yoonsik Kim, Seonghyeon Kim, Moonbin Yim, Seung Shin, Gayoung Lee, and Sungrae Park. Rewritenet: Reliable seene text editing with implicit decomposition of text contents and styles. arXiv preprint arXiv:2107. 10041, 2021.
Wataru Shimoda, Daichi Haraguchi, Seiichi Uchida, and Kota Yamaguchi. De-rendering stylized texts. In ICCV, pages 1076-1085, 2021.
Qiangpeng Yang. Jun Huang, and Wei Lin. Swaptext: Image based texts transfer in scenes, In CVPR, pages 14700-14709, 2020.
Fangneng Zhan, Hongyuan Zhu, and Shijian Lu. Spatial fusion gan for image synthesis. CVPR, pages 3653-3662, 2019.
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba. Paint by word. arXiv preprint 1rXiv:2103.10951, 2021.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nadee Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108. 2021.
Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing. In ACM SIGGRAPH, pages 313-318. 2003.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. ACM SIGGRAPH, pages 1-10, 2022b.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2023.
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant, Character-aware models improve visual text rendering. arXiv preprint XrXiv:2212 10562, 2022b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In NeurlPS, volume 33, pages 1877-1901, 2020.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:22 2..02088,.0 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
Jacob Devlin, Ming-Woi Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2018.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al- Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: massively multilingual pre-trained text-to-text transformer. In ACI., 2021.
13


Reviews:
[{"review_id": "vH8xFkesOo", "reviewer": "Reviewer_8YaX", "summary": "This paper introduces DiffUTE, an innovative diffusion-based text editing framework designed to seamlessly fill in missing words in an image with user-specified text. By employing a self-supervised training framework, the model effectively learns from an extensive collection of synthetic data pairs, enabling it to infer accurate text styles and generate images that seamlessly incorporate the desired text. Experimental results showcase remarkable qualitative text editing performance from both the model's precision in both text and style accuracy. Additionally, quantitative analysis shows that the proposed method surpasses the performance of baseline approaches.", "questions": "1. Are there any metrics available for assessing the accuracy/consistency of text style? Or potentially this can be validate through a human study similar to the Cor metrics present in paper.\n2. Performance comparison with other diffusion-based text-editing method, DiffSTE?\n3. An intriguing aspect of this work is its remarkable ability to accurately infer text styles, even when multiple possible texts are present within an image. For instance, in Figure 4, column 1, we can observe that the imprinted time exhibits the correct style (black), despite the presence of additional red texts. It raises the question of how robust this achievement is. Specifically, does the model consistently succeed when confronted with multiple text categories? Furthermore, can this success be attributed to different types of characters, such as numerical values versus Chinese characters, as depicted in this example?", "limitations": "This paper clearly discusses the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The proposed method exhibits impressive editing performance, as demonstrated through extensive experiments. It displays a strong ability to accurately infer text styles and generate corresponding images.\n\n+ Leveraging LLM, the model offers broad applicability across many possible application scenarios.\n\n+ The paper is organized clearly and is easy to read.", "weaknesses": "- Quantitative metrics for style: Although the paper effectively showcases the model's ability to generate text that is stylistically consistent with the rest of the images, it does not provide a quantitative analysis or specific metrics to support this claim.\n- Alternative diffusion-based baselines: While ControlNet is a powerful diffusion-based editing framework, it is not specifically designed for text editing tasks. Another recent diffusion-based editing approach, DiffSTE[1], shares similarities with this work as it focuses on specialized text editing and exhibits commendable performance. How does the proposed method compare to DiffSTE in terms of performance?\n\n[1] Improving Diffusion Models for Scene Text Editing with Dual Encoders.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and detailed review. We are encouraged that the reviewer find that our DiffUTE 'exhibits impressive editing performance, as demonstrated through extensive experiments' and 'leveraging LLM, the model offers broad applicability across many possible application scenario'. We would like to response as below to adress your remaining concerns.\n\n> [W1&Q1] Quantitative metrics for style: Although the paper effectively showcases the model's ability to generate text that is stylistically consistent with the rest of the images, it does not provide a quantitative analysis or specific metrics to support this claim. & Are there any metrics available for assessing the accuracy/consistency of text style? Or potentially this can be validate through a human study similar to the Cor metrics present in paper.\n\nIt is difficult to quantify the effect of the style directly, but we can provide the results of a user study. Specifically, we randomly select 100 images from our Web dataset. Given each image, we can obtain 4 edited results including 3 baselines and our method. We invited 50 users to identify the edited text style in each group that they felt was most similar to the original image.\nFinally 20,000 comparison results are collected, followed by using the Bradley-Terry (B-T) model [1] to calculate an overall ranking of all methods. As presented in the following Table, our DiffUTE achieves the highest B-T score.\n\n| Method | B-T Score |\n| :---: | :---: | \n| SRNet | 0.1140 | \n| SD2-FT | 0.1545 | \n| DiffSTE | 0.3378 | \n| DiffUTE | 0.3937 | \n\n> [W2&Q2] Alternative diffusion-based baselines: While ControlNet is a powerful diffusion-based editing framework, it is not specifically designed for text editing tasks. Another recent diffusion-based editing approach, DiffSTE[1], shares similarities with this work as it focuses on specialized text editing and exhibits commendable performance. & How does the proposed method compare to DiffSTE in terms of performance? Performance comparison with other diffusion-based text-editing method, DiffSTE?\n\nThank you for your professional comments. We have made a detailed comparison with DiffSTE on the validation set. As shown in the table below, our method performs better than DiffSTE on all datasets, which may be due to the use of instructions to control image editing in DiffSTE. Obviously, glyph-based control conditions can provide more spatial information. In addition, DiffSTE only supports English text editing and does not perform well on more difficult Chinese text editing. (The Web dataset contains text in various languages, mainly English and Chinese.)\n\n| Model | Avg.-OCR | Avg.-Cor |Web-OCR | Web-Cor | ArT-OCR | ArT-Cor | TextOCR-OCR | TextOCR-Cor | ICDAR13-OCR | ICDAR13-Cor |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| DiffSTE | 74.30 | 75 | 48.55 | 50 | 82.72 | 84 | 84.85 | 85 | 81.48 | 81 |\n| DiffUTE | **85.41 (+11.11)** | **85.5 (+10.5)**| **84.83 (+36.28)** | **85 (+35)** | **85.98 (+3.26)** | **87 (+3)** | **87.32 (+2.47)** | **88 (+3)** | **83.49 (+2.01)** | **82 (+1)** |\n\n> [Q3] An intriguing aspect of this work is its remarkable ability to accurately infer text styles, even when multiple possible texts are present within an image. For instance, in Figure 4, column 1, we can observe that the imprinted time exhibits the correct style (black), despite the presence of additional red texts. It raises the question of how robust this achievement is. Specifically, does the model consistently succeed when confronted with multiple text categories? Furthermore, can this success be attributed to different types of characters, such as numerical values versus Chinese characters, as depicted in this example?\n\nThis is indeed an interesting question. In order to further understand the generation ability of DiffUTE, we have provided more examples in this global response PDF. As shown in the figure S4, when the image is filled with a single Chinese character, DiffUTE can also infer the text style based on the surrounding text. Furthermore, as shown in the figure S6 , the target we input for modification is \"13\", but DiffUTE generated a result with the addition of \"元\" based on its understanding of surrounding information. This demonstrates that DiffUTE has a certain degree of document understanding ability, and can infer the required text style based on contextual information in document data. This reasoning and learning ability can also be observed from the image S4. DiffUTE infers the angle of the text to be filled based on the posture of the surrounding text, so that the inclination angle of the text is consistent with other relevant fonts around it. We believe that the reasoning ability of DiffUTE comes from the training on a large amount of data, from which it learns some structured information. Of course, it is not always able to accurately infer and there may be situations where the style does not match the expectation, but there is no perfect model after all.\n\n\nReferences\n\n[1] A comparative study for single image blind deblurring. In CVPR, 2016."}, {"review_id": "8xYRKFCQoE", "reviewer": "Reviewer_sxrD", "summary": "The paper proposes DiffUTE for general text editing. \n\nDiffUTE utilizes Stable Diffusion model with several specific model designs, progressive training strategy, positional and glyph guidance, and a self-supervised training framework. \n\nEquipped with these designs, DiffUTE achieves remarkable results compared to other baselines on several public datasets. \nMoreover, the authors also provide a chat-based interface which enables an easier manipulation for the users. ", "questions": "Writing:\n- Line 129: it is better explicitly to explain what $x_m$ is at the first place.\n ", "limitations": "The authors emit some ethical discussions in the paper. For example, the authors should discuss the misusage of the technique for misinformation spread. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality, motivation and significance:\n- The paper shades an interesting perspective to edit text using pre-trained Stable Diffusion model. Two motivations raised in Line 29 and Line 32 are intuitive. \n- The interaction module is interesting and easy to use. \n\nTechnical approach: \n- Finetuning VAE with a progressive training strategy  (PTT) with different image sizes in different stages is a good choice to overcome blurry outputs. As shown in Table 2, with PTT, DiffUTE has a noticeable gain. \n- The insight into generating fine-grained texts makes sense. With positional and glyph guidance, DiffUTE generates texts with natural shapes.\n- Proposed self-supervised training strategy is straight-forward and useful. It also reduces the need of human annotations. \n\nClarity: the paper offers a smooth writing and is easy to follow.  ", "weaknesses": "- The motivation of using diffusion models v.s. GANs is not clearly stated. Why would the authors prefer to use diffusion model (e.g., Stable Diffusion)? \n- The paper lacks some failure case analysis. For example, DiffUTE relies on pretrained OCR detector. What if the OCR detector compromises in some cases? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and detailed review. We are encouraged that the reviewer find that our DiffUTE 'shades an interesting perspective to edit text using pre-trained Stable Diffusion model' and 'interaction module is interesting and easy to use'. We would like to response as below to adress your remaining concerns.\n\n> [W1] The motivation of using diffusion models v.s. GANs is not clearly stated. Why would the authors prefer to use diffusion model (e.g., Stable Diffusion)?\n\nPreviously, most text editing work used GAN or simple CNN as network structures, which focused mostly on low-resolution, simple background English images and performed poorly in editing texts while maintaining text style and background consistency. Recently, diffusion models have achieved remarkable results, performing well in style preservation and detail texture. We conducted a feasibility analysis of zero-shot text editing based on GAN models and diffusion models and found that the diffusion model has great potential and can almost restore the shape of simple text in some editing tasks. Therefore, we chose the diffusion model because of its powerful generation ability as well as its controllability and scalability.\n\n> [W2] The paper lacks some failure case analysis. For example, DiffUTE relies on pretrained OCR detector. What if the OCR detector compromises in some cases?\n\nThank you for your professional comments. Here we provide an analysis of some failed examples. Since DiffUTE relies on OCR-extracted boxes for text editing, there is no way to edit the text when the box is inaccurate, as shown in global response PDF (Figure S1-S3). However, when the OCR box can cover the text well, DiffUTE can usually perform editing work well. DiffUTE itself may also fail to generate text accurately in some scenarios. For example, when there are too many Chinese characters to be edited (more than 6), it is difficult to generate them accurately due to the complexity of Chinese character generation. Some studies that focus on font style generation only compare the performance of generating individual characters. In the future, we will consider how to edit long text sequences.\n\n> [Q1] Line 129: it is better explicitly to explain what x_m is at the first place. \n\nWe would like to clarify that we have stated in line 83: \"by the concatenation of latent image vector $z_t$, masked image latent vector $x_m$, and text mask $m$.\", and also visualized the corresponding image in Figure 3.\n\n> [L1] The authors emit some ethical discussions in the paper. For example, the authors should discuss the misusage of the technique for misinformation spread.\n\nFirst, our model aims to improve users' efficiency and accuracy in image processing, especially when editing large amounts of text quickly and with high quality. Our goal is to provide users with better tools, not to spread false information. Secondly, we recognize that any technology has the risk of being misused, including our model. We will discuss this issue in our paper and propose some suggestions to minimize this risk. For example, we may recommend that regulatory agencies and social media platforms take measures to identify and combat false information, as well as provide necessary education and training to the public. At the same time, using the data we generate, we can also help improve the detection performance of false information detection models. In fact, we are also committed to developing an AI-generated content detection model for regulating generated content. Finally, we hope that users and society can be aware of the potential risks of this technology and exercise responsibility and caution when using it. We hope that our technology can bring more benefits to society rather than harm."}, {"review_id": "74ZXr9ka2I", "reviewer": "Reviewer_jbcg", "summary": "The authors propose a method of fine-tuning Stable Diffusion to modify words in images, while maintaining the original font style and the background region.\nSpecifically, they first fine-tune the VAE with text images from several datasets.\nThen, utilizing an off-the-shelf OCR detector, they randomly mask out one text box, and tune the denoising network asking it to fill the region with original text that is given as the condition in cross-attention layers.\nWith this simple and intuitive method, they achieve improved performance in various evaluations.", "questions": "- not sure whether this method is self-supervised or not, since it requires the utilization of an OCR detector in training\n- could you elaborate more on a glyph image? for example, what is the output format of the glyph encoder?\n- could you compute the accuracy of ChatGLM’s predictions? and how much cost does it take to fine-tune it?\n- it seems that the figure 3 is redundant. maybe you can incorporate it into the figure 2?", "limitations": "please refer to the Weaknesses section", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the authors tackle a meaningful task\n- the proposed method is simple and easy to reproduce\n- the proposed method demonstrates improved performance on various evaluation metrics", "weaknesses": "- would only work for texts that can be detected by off-the-shelf OCR detectors\n- there are missing details on some parts of the method", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and detailed review. We are encouraged that the reviewer find that our DiffUTE 'is simple and easy to reproduce' and 'demonstrates improved performance on various evaluation metrics'. We would like to response as below to adress your remaining concerns.\n\n> [W1] would only work for texts that can be detected by off-the-shelf OCR detectors\n\nAs DiffUTE relies on the text boxes extracted by OCR, the generated results will not be very good when the boxes are inaccurate. Here we provide an analysis of some failed examples in this global response PDF (Figure S1-S3). However, when the OCR box can cover the text well, DiffUTE can usually perform editing work well. Our next step is to design a dynamic mask to compensate for the problem of inaccurate mask regions. We also tried an OCR-free method, fine-tuned SD1 and SD2 by instruction tuning. However, the experimental results (Table 1 and Figure 4) show that controlling text editing through instructions cannot achieve good results. This may be because natural language can only provide the semantics of the text to be modified, while glyphs can provide more detailed spatial structural information.\n\n> [W2] there are missing details on some parts of the method.\n\nConsidering the limited space, if there are any unclear parts, we will provide detailed explanations in the appendix. In addition, we have provided a detailed description of the experiment's details and our training and inference code in the appendix.\n\n> [Q1] Not sure whether this method is self-supervised or not, since it requires the utilization of an OCR detector in training.\n\nNowadays, OCR technology is quite mature and can be used for preprocessing in many downstream tasks. In fact, we only provide the position information of the text to the model, without any additional artificially defined information or labels. At the same time, OCR information is a basic requirement for scene text editing. Therefore, compared with artificially constructing an instruct-tuning dataset, our model can be regarded as a self-supervised task.\n\n> [Q2] Could you elaborate more on a glyph image? for example, what is the output format of the glyph encoder?\n\nThe glyph image is created by using a universal font to write the text to be edited on a white background image. We have submitted the complete code in the support material. For easy understanding, here is a snippet of the code used to generate the glyph image. And we used trocr as the glyph encoder. Specifically, trocr consists of an encoder and a decoder. The encoder encodes the text in the image, and the decoder interprets the encoding to obtain the recognition result. In our model, we only use the encoder of trocr to extract the encoding features. Therefore, the output of the glyph encoder is a deep feature vector.\n```python\ndef draw_text(im_shape, text): \n    # text size\n    font_size = 40\n    # text font, you can download it from the internet\n    font_file = 'arialuni.ttf'\n    # create a pure white background\n    img = Image.new('RGB', ((len_text+2)*font_size, 60), color='white')  \n    # define the font object\n    font = ImageFont.truetype(font_file, font_size)\n    # define the text and position\n    pos = (40, 10)\n    # write the text on background\n    draw = ImageDraw.Draw(img)\n    draw.text(pos, text, font=font, fill='black')\n    img = np.array(img)\n\n    return img\n```\n\n\n> [Q3] Could you compute the accuracy of ChatGLM’s predictions? and how much cost does it take to fine-tune it?\n\nWhen ChatGLM is not fine-tuned, it is difficult to return correct results and understand human commands due to the lack of learned structured data such as OCR results. We tested it on 100 commands, and the accuracy before fine-tuning was 5%, while after fine-tuning, it reached 98%. We used an A100 for fine-tuning.\n\n\n> [Q4] it seems that the figure 3 is redundant. maybe you can incorporate it into the figure 2?\n\nThank you for your professional comments. We will update our images in the revised version.\n"}, {"review_id": "wxGGunsjSC", "reviewer": "Reviewer_Rgqn", "summary": "In this paper, the authors present DiffUTE, a universal self-supervised text editing diffusion model for language-guided image editing. They address the limitations of existing diffusion models by focusing on rendering accurate text and text style during image generation. DiffUTE incorporates modifications to the network structure, allowing it to handle multilingual character drawing using glyph and position information. Furthermore, a self-supervised learning framework leverages a large amount of web data to enhance the model's representation ability. The experimental results showcase the impressive performance of DiffUTE, demonstrating its ability to achieve high-fidelity and controllable editing on diverse real-world images. Overall, this paper presents a significant advancement in language-guided image editing and offers a promising approach for rendering realistic and customizable text in generated images.", "questions": "As Above", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem addressed in this paper is a realistic problem that current diffusion models struggle to handle effectively.\n\n- The incorporation of LLM  into the inference process is a compelling and intriguing approach.", "weaknesses": "- The paper claims significantly better results than other baselines in Table 1. However, it would be helpful to clarify if there are other baselines that have not been adequately considered.\n\n- A simple baseline is missing. Have the authors considered directly replacing the \"source text\" with the \"target text\" and calculating the FID (Fréchet Inception Distance)?\n\n- In Table 1, the results for SD1-FT and SD2-FT appear to be poor. It would be valuable to explain the main differences between your method and these baselines.\n\n- There is limited mention of other methods that fine-tune the encoder-decoder. How important is this step? Additionally, could you provide details on the difference in parameter numbers shown in Table 1?\n\n- The discussion regarding self-guidance is absent, despite the proposal of a self-supervised approach for achieving text editing in diffusion. The related papers are:\n\n- Self-Guided Diffusion Models\n- Why Are Conditional Generative Models Better Than Unconditional Ones?\n- Visual Chain-of-Thought Diffusion Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and detailed review. We would like to response as below to adress your remaining concerns.\n\n> [W1] The paper claims significantly better results than other baselines in Table 1. However, it would be helpful to clarify if there are other baselines that have not been adequately considered.\n\nThank you for your professional comments. According to the opinions of other reviewers, we have added a comparison with the latest DiffSTE[1]. As shown in the table below, our method performs better than DiffSTE on all datasets, which may be due to the use of instructions to control image editing in DiffSTE. Obviously, glyph-based control conditions can provide more spatial information. In addition, DiffSTE only supports English text editing and does not perform well on more difficult Chinese text editing. \n\n| Model | Avg.-OCR | Avg.-Cor |Web-OCR | Web-Cor | ArT-OCR | ArT-Cor | TextOCR-OCR | TextOCR-Cor | ICDAR13-OCR | ICDAR13-Cor |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| DiffSTE | 74.30 | 75 | 48.55 | 50 | 82.72 | 84 | 84.85 | 85 | 81.48 | 81 |\n| DiffUTE | **85.41 (+11.11)** | **85.5 (+10.5)**| **84.83 (+36.28)** | **85 (+35)** | **85.98 (+3.26)** | **87 (+3)** | **87.32 (+2.47)** | **88 (+3)** | **83.49 (+2.01)** | **82 (+1)** |\n\n> [W2] A simple baseline is missing. Have the authors considered directly replacing the \"source text\" with the \"target text\" and calculating the FID?\n\nThank you for your suggestion. We added a simple method as a baseline, as shown in the table below, and compared different methods in terms of FID. Specifically, our baseline is divided into two steps. In the first step, we use traditional inpainting algorithms[2] to restore the areas that need to be tampered with, and then directly write the desired text in those areas.\n\n| Method | MSE | PSNR | SSIM |FID |\n| :---: | :---: |  :---: |  :---: |  :---: | \n| Baseline | 0.0894 | 11.04 | 0.3523 | 90.82 | \n| SRNet | 0.0352 | 17.62| 0.6232 | 40.88 | \n| SD2-FT | 0.0132| 20.84 | 0.7472 | 32.52 | \n| DiffSTE | 0.0114 | 21.94 | 0.7895 | 29.84 | \n| DiffUTE | **0.0094** | **23.72** | **0.8323** |**28.22**| \n\n> [W3] In Table 1, the results for SD1-FT and SD2-FT appear to be poor. It would be valuable to explain the main differences between your method and these baselines.\n\nThe reason why SD1-FT and SD2-FT perform poorly is because they perform editing through instruction. It is difficult for instruction to provide information about the shape of the text, and there are also errors in understanding the specific text to be edited from the instruction. Our method uses glyph image to provide information about the shape of the text, and adds position control to strengthen the generation target, which provides richer control information than SD and generates high-quality text. In addition, since we use glyph image as the condition, when facing the editing task of a new language, only the image of the text in the new language needs to be provided for fine-tuning our model. In contrast, SD needs to prepare corresponding instructions for fine-tuning.\n\n> [W4] There is limited mention of other methods that fine-tune the encoder-decoder. How important is this step? Additionally, could you provide details on the difference in parameter numbers shown in Table 1?\n\nIn our model, there are two fine-tuning processes. In fine-tuning the VAE, there are two options: training directly using the input image size of the model, and training using the progressive training strategy. We compare these two methods in Figure 5 and Figure 6. In fine-tuning the entire SD, there are no other options for fine-tuning methods. I have listed the detailed parameters of the models in Table 1 below. Unfortunately, many models did not provide parameter information, which may be because the editing effect in this field is currently not satisfactory enough. When the effect becomes good enough, people will consider lightweight.\n\n\n| Method | Params |\n| :---: | :---: | \n| Pix2Pix | Unknown|\n| SRNet | Unknown | \n| MOSTEL | Unknow |\n| SD| 1070 MB | \n| DiffSTE | Unknown | \n| DiffUTE | 1500 MB | \n\n\n> [W5] The discussion regarding self-guidance is absent, despite the proposal of a self-supervised approach for achieving text editing in diffusion. The related papers are: Self-Guided Diffusion Models; Why Are Conditional Generative Models Better Than Unconditional Ones?;Visual Chain-of-Thought Diffusion Models.\n\nThank you for your professional comments. We will add the following discussion about other self-supervised diffusion models in the revised paper. \n\nHu et al. proposed self-guided diffusion models, which use the flexibility of self-supervised signals to design the framework of self-guided diffusion models to eliminate the need for annotations. By leveraging a feature extraction function and a selfannotation function, they provides guidance signals at various image granularities: from the level of holistic images to object boxes and even segmentation masks. Bao et al. train a conditional diffusion model by taking the cluster indices as conditions. And Harvey et al. propose to close the gap between conditional and unconditional models using a two-stage sampling procedure. The above methods mostly train by obtaining pseudo-labels of the image, while DiffUTE is essentially a task similar to MAE. It is hoped that in this process, the model can learn the mapping relationship between glyph, surrounding background, and image representation.\n\nReferences\n\n[1] Improving Diffusion Models for Scene Text Editing with Dual Encoders. arXiv preprint arXiv:2304.05568.\n\n[2] Navier-stokes, fluid dynamics, and image and video inpainting. In CVPR, 2001.\n\n[3] Self-Guided Diffusion Models. arXiv preprint arXiv:2210.06462.\n\n[4] Why Are Conditional Generative Models Better Than Unconditional Ones? arXiv preprint arXiv:2212.00362.\n\n[5] Visual Chain-of-Thought Diffusion Models. arXiv preprint arXiv:2303.16187."}, {"review_id": "WEVa6FFF86", "reviewer": "Reviewer_FkEt", "summary": "This paper describes an application of diffusion models (Sohl- Dickstein et al., 2015; Ho et al., 2020) to text editing. Methodologically, this work differs from previous text diffusion (Li et al., 2022) by leveraging insights on glyph encoder and OCR detector. Empirically, this work advances the state of the art for text editing by scaling these methods to larger datasets. The paper also proposes to use self-supervised training to train the diffusion model and further explore diffusion guidance.", "questions": "The author mentioned that \"in the first three stages of training, we randomly crop images of sizes 𝑆/8, 𝑆/4 and 𝑆/2 and resize them to 𝑆 for training\", however, which are the three stages and where is the fourth one? Moreover, an ablation that would be of interest is to train with different resolutions at different stages.", "limitations": "Beyond the weaknesses I listed, the authors were good at addressing several limitations of this work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This is the work on diffusion LMs that shows results on a text editing baseline and seems to have positive results in terms of the metrics used.\n2. The writing is clear and the motivations seem sound.", "weaknesses": "1. The main weakness is the novelty. The core idea of this paper, i.e. latent diffusion, has been demonstrated to be successful in many generation tasks. Thus it is not surprising that it works on scene text editing. Most of the techniques used in the paper have been proposed perviously.\n2. The author did not provide any details regarding the position control module, thus the ablation study of this part is not convincing.\n3. The authors did not evaluate a variety of evaluation measures that prior work has done such as SSIM, MSE, PSNR, and many more. These metrics should be computed to a get better idea of the quality and diversity of the output. Please see these this paper for the description of these metrics: “ Krishnan P, Kovvuri R, Pang G, et al. Textstylebrush: transfer of text aesthetics from a single example[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.”.\n4. There are missing comparisions such as Krishnan et al 2023's TextStyleBrush[1] and Ji’s 2023’s DiffSTE [2]. \n5. The model rely on a pretrained OCR encoder which just seems like an arbitrary choice. An ablation should be provided with different pretrained encoders to understand the impact of this choice.\n\n[1] Krishnan P, Kovvuri R, Pang G, et al. Textstylebrush: transfer of text aesthetics from a single example[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.\n[2] Ji, Jiabao, et al. \"Improving Diffusion Models for Scene Text Editing with Dual Encoders.\" arXiv preprint arXiv:2304.05568 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback and detailed review. We would like to response as below to adress your remaining concerns.\n\n> [W1] Novelty Concern. The core idea of this paper, i.e. latent diffusion, has been demonstrated to be successful in many generation tasks. Thus it is not surprising that it works on scene text editing. Most of the techniques used in the paper have been proposed perviously.\n\nDiffusion has good performance in other tasks, but its text scene performance is unverified, and zero-shot and direct fine-tuning performance is poor (\"Achilles' heel\"). Achieving good results in text editing is important. DiffSTE uses instruction tuning but only supports English text editing and lacks scalability. Our multilingual text editing model prepares only corresponding images for fine-tuning new languages, due to glyph image support.\n\n> [W2] The author did not provide any details regarding the position control module, thus the ablation study of this part is not convincing.\n\nWe need to clarify that we have already elaborated on the position control in Line 109-115. It is not a module, but a mask of the area to be edited. Therefore, in the ablation experiment, we compared the results without adding the mask at the input end to verify the effectiveness of the position control.\n\n> [W3] The authors did not evaluate a variety of evaluation measures that prior work has done such as SSIM, MSE, PSNR, and many more. These metrics should be computed to a get better idea of the quality and diversity of the output. Please see these this paper for the description of these metrics.\n\nWe need to clarify that \"diversity\" is unnecessary in the scene text editing task, and the closer the generated text style is to the original text style, the better. Therefore, we conducted a user study to verify the superiority of our model in generating text styles. Specifically, we randomly select 100 images from our Web dataset. Given each image, we can obtain 4 edited results including 3 baselines and our method. We invited 50 users to identify the edited text style in each group that they felt was most similar to the original image. Finally 20,000 comparison results are collected, followed by using the Bradley-Terry (B-T) model [1] to calculate an overall ranking of all methods. As presented in the following Table, our DiffUTE achieves the highest B-T score.\n\n| Method | B-T Score |\n| :---: | :---: | \n| SRNet | 0.1140 | \n| SD2-FT | 0.1545 | \n| DiffSTE | 0.3378 | \n| DiffUTE | 0.3937 | \n\nIn addition, we provide some other metrics for comparison here as well. Since Textstylebrush does not provide open source code, we do not compare with it. We compared MSE, PSNR, and SSIM on the TextOCR validation set. It should be noted that the task of natural image editing is to make the text style of the edited image similar to that of the original image and the overall comparison is harmonious. Therefore, we calculated the metrics for the entire edited image. As shown in the table below, our method achieved the best results in all metrics.\n\n| Method | MSE | PSNR | SSIM |FID |\n| :---: | :---: |  :---: |  :---: |  :---: | \n| SRNet | 0.0352 | 17.62| 0.6232 | 40.88 | \n| SD2-FT | 0.0132| 20.84 | 0.7472 | 32.52 | \n| DiffSTE | 0.0114 | 21.94 | 0.7895 | 29.84 | \n| DiffUTE | **0.0094** | **23.72** | **0.8323** |**28.22**| \n\n\n> [W4] There are missing comparisions such as Krishnan et al 2023's TextStyleBrushand Ji’s 2023’s DiffSTE.\n\nWe compared our model with DiffSTE [1] on the validation set, as TextStyleBrush was not available in open-source code. The available reproduction code (https://github.com/grenlayk/text-deep-fake) differed from the original paper. As shown in the table, our method outperforms DiffSTE on all datasets, possibly due to glyph-based control conditions providing more spatial information. Also, DiffSTE only supports English text editing and struggles with more difficult Chinese text editing (Web dataset).\n\n| Model | Avg.-OCR | Avg.-Cor |Web-OCR | Web-Cor | ArT-OCR | ArT-Cor | TextOCR-OCR | TextOCR-Cor | ICDAR13-OCR | ICDAR13-Cor |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| DiffSTE | 74.30 | 75 | 48.55 | 50 | 82.72 | 84 | 84.85 | 85 | 81.48 | 81 |\n| DiffUTE | **85.41 (+11.11)** | **85.5 (+10.5)**| **84.83 (+36.28)** | **85 (+35)** | **85.98 (+3.26)** | **87 (+3)** | **87.32 (+2.47)** | **88 (+3)** | **83.49 (+2.01)** | **82 (+1)** |\n\n> [W5] The model rely on a pretrained OCR encoder which just seems like an arbitrary choice. An ablation should be provided with different pretrained encoders to understand the impact of this choice.\n\nDue to high training costs, we conducted a feasibility analysis and chose a multilingual OCR encoder, trocr, for our OCR encoder. It can support 100+ languages with image data collection and model training. Unlike DiffSTE, our model only requires OCR models to fine-tune text in different languages, which is convenient for extension.\n\n> [Q1] The author mentioned that \"in the first three stages of training, we randomly crop images of sizes S/8, S/4 and S/2 and resize them to S for training\", however, which are the three stages and where is the fourth one? Moreover, an ablation that would be of interest is to train with different resolutions at different stages.\n\nThese trainings are specifically for VAE. As the VAE of SD is designed for natural images, its ability to restore text is weak and requires targeted fine-tuning. Our training is divided into 20 epochs, using 512-size input images and progressively increasing crop sizes (64/128/256/512) for difficulty. Starting with very large images can make VAE training difficult (with noise-like dots). Thanks for your suggestion; we'll add ablation experiments later due to high training costs.\n\nReferences\n\n[1] A comparative study for single image blind deblurring. In CVPR, 2016.\n\n[2] Improving Diffusion Models for Scene Text Editing with Dual Encoders. arXiv preprint arXiv:2304.05568.\n"}]

Author rebuttal:
We thank the reviewers for the positive reviews and constructive feedback. We thank the AC, SAC and PC for facilitating the review process.

It is very encouraging to hear from the reviewers that:

- Performance of DiffUTE: “exhibits impressive editing performance; has strong ability to accurately infer text styles and generate corresponding images; demonstrates improved performance on various evaluation metrics; ” [8YaX, jbcg]

- Usability of DiffUTE: "Leveraging LLM, the model offers broad applicability across many possible application scenarios; the interaction module is interesting and easy to use; the incorporation of LLM into the inference process is a compelling and intriguing approach; " [8YaX, sxrD, Rgqn]

- Motivation of this work: "tackle a meaningful task; addressed  a realistic problem that current diffusion models struggle to handle effectively” [jbcg, Rgqn]

- Paper writing: "writing is clear and the motivations seem sound; paper is organized clearly and is easy to read" [FkEt, 8YaX]


We provide clarifications to each of the queries from the reviewers as response to each of the reviews. We sincerely hope that our DiffUTE would be positively received, considering its value addition to the community. We provide in the PDF attachment examples that the reviewers requested to be supplemented.

Meta-review: This paper presents DiffUTE, a diffusion-based scene-text editing framework. Leveraging several proposed modules on top of the Stable Diffusion, DiffUTE is able to modify the words in images while preserving the original font style. Reviewers agree that the proposed model designs are intuitive and effective. Furthermore, reviewers also agree that DiffUTE shows remarkable performance compared with prior baselines in terms of content generation and style preservation. In the rebuttal phase, the authors actively respond to the reviewers and address the raised concerns including comparison with more baselines and metrics.
Decision: Accept (poster)