Title: DiffUTE: Universal Text Editing Diffusion Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: Diffusion model

text editing

self-supervied learning
Abstract: Diffusion model based language-guided image editing has achieved great success recently. However, existing state-of-the-art diffusion models struggle with rendering correct text and text style during generation. To tackle this problem, we propose a universal self-supervised text editing diffusion model (DiffUTE), which aims to replace or modify words in the source image with another one while maintaining its realistic appearance. Specifically, we build our model on a diffusion model and carefully modify the network structure to enable the model for drawing multilingual characters with the help of glyph and position information. Moreover, we design a self-supervised learning framework to leverage large amounts of web data to improve the representation ability of the model. Experimental results show that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity. Our code will be avaliable in \url{https://github.com/chenhaoxing/DiffUTE}.

DiffUTE: Universal Text Editing Diffusion Model

Abstract

Diffusion model based language-guided image editing has achieved great success recently. However, existing state-of-the-art diffusion models struggle with rendering correct text and text style during generation. To tackle this problem, we propose a universal self-supervised text editing diffusion model (DiffUTE), which aims to replace or modify words in the source image with another one while maintaining its realistic appearance. Specifically, we build our model on a diffusion model and carefully modify the network structure to enable the model for drawing multilingual characters with the help of glyph and position information. Moreover, we design a self-supervised learning framework to leverage large amounts of web data to improve the representation ability of the model. Experimental results show that our method achieves an impressive performance and enables controllable editing on in-the-wild images with high fidelity. Our code will be avaliable in https://github.com/chenhaoxing/DiffUTE .

1 Introduction

Due to the significant progress of social media platforms and artificial intelligence Xu et al. [2022a], Gu et al. [2023], Zhang et al. [2022a], image editing technology has become a common demand. Specifically, AI-based technology Niu et al. [2023], Chen et al. [2023] has significantly lowered the threshold for fancy image editing, which traditionally required professional software and laborintensive manual operations. Deep neural networks can now achieve remarkable results in various image editing tasks, such as image inpainting Feng et al. [2022], image colorization Zhang et al. [2022b], and object replacement Kwon and Ye [2022], by learning from rich paired data. Futhermore, recent advances in diffusion models Brack et al. [2023], Brooks et al. [2023], Saharia et al. [2022a] enable precise control over generation quality and diversity during the diffusion process. By incorporating a text encoder, diffusion models can be adapted to generate natural images following text instructions, making them well-suited for image editing.

Despite the impressive results, existing image editing methods still encounter numerous challenges. As a typical task, scene text editing is widely used in practical applications such as text-image synthesis, advertising photo editing, text-image correction and augmented reality translation. It aims to replace text instances (i.e., the foreground) in an image without compromising the background. However, the fine-grained and complex structures of text instances raise two major challenges: (i) How to transfer text style and retain background texture. Specifically, text style includes factors such as font, color, orientation, stroke size, and spatial perspective. It is difficult to precisely capture the complete text style in the source image due to the complexity of the background. (ii) How to maintain the consistency of the edited background especially for complex scenes, e.g., menus and street store signs.

欠費

Image

Text

garden

Image

Text

SRNet

MOSTEL

SD2

ControlNet

SD2-FT

Ours

Input

trinity

trinity

trinity

trinity

trinity

trinity

trinity

Sardrn

garden

Sarran

Matden

garden

garden

Studios

studios

studios

studios

studios

studios

studios

参考號：

参考號：

参考號：

参考號：

参考號：

参考號：

参考號：

REF.NO.:

REF.NO.

REF.NO.

REF.NO.:

REF.NO.:

REF.NO.:

AMOUNT:

久翼

欠

欠费

欠：

欠費：

小费：

AMOUNT:

AMOUNT:

AMOUNT:

AMOUNT:

AMOUNT:

AMOUNT:

TIPS:

小费：

小费：

小费：

小费：

小费：

小费：

TIPS:

TIPS:

TIPS:

TIPS:

TIPS:

TIPS:

Numerous studies formulate scene text editing as a style transfer task and approach it by generative models like GANs Wu et al. [2019], Qu et al. [2023]. Typically, a cropped text region with the target style is needed as the reference image. Such methods then transfer a rendered text in the desired spelling to match the reference image's style and the source image's background. However, the two major challenges for scene text editing remains. (i) These methods are currently constrained to editing English and fail to accurately generate complex text style (e.g., Chinese). (ii) The process of cropping, transferring style and blending results in less natural-looking outcomes. End-to-end pipelines are needed for the consistency and harmony.

To address the above issues, we present DiffUTE, a general diffusion model designed to tackle high-quality multilingual text editing tasks. DiffUTE utilizes character glyphs and text locations in source images as auxiliary information to provide better control during character generation. As shown in Figure 1, our model can generate very realistic text. The generated text is intelligently matched to the most contextually appropriate text style and seamlessly integrated with the background while maintaining high quality.

The major contribution of this paper is the universal text edit diffusion model proposed to edit scene text images. DiffUTE possesses obvious advantages over existing methods in several folds:

We present DiffUTE, a novel universal text editing diffusion model that can edit any text in any image. DiffUTE generates high-quality text through fine-grained control of glyph and position information. DiffUTE is capable of seamlessly integrating various styles of text characters into the image context, resulting in realistic and visually pleasing outputs.

We design a self-supervised learning framework that enables the model to be trained with large amounts of scene text images. The framework allows the model to learn from the data without annotation, making it a highly efficient and scalable solution for scene text editing.

We conduct extensive experiments to evaluate the performance of DiffUTE. Our method performs favorably over prior arts for text image editing, as measured by quantitative metrics and visualization.

2 Preliminaries

In this paper, we adopt Stable Diffusion (SD) Rombach et al. [2022] as our baseline method to design our network architecture. SD utilizes a variational auto-encoder (VAE) to enhance computation efficiency. Through VAE, SD performs the diffusion process in low-dimensional latent space. Specifically, given an input image 𝑥 ∈ R 𝐻 × 𝑊 × 3 , the encoder E 𝑣 of VAE transforms it into a latent representation 𝑧 ∈ R ℎ × 𝑤 × 𝑐 , where 𝛼 = 𝐻 ℎ = 𝑊 𝑤 is the downsampling factor and 𝑐 is the latent feature dimension. The diffusion process is then executed in the latent space, where a conditional UNet denoiser Ronneberger et al. [2015] 𝜖 𝜃 ( 𝑧 𝑡 , 𝑡, 𝑦 ) is employed to predict the noise with noisy latent 𝑧 𝑡 , generation condition input 𝑦 and current time step 𝑡 . The condition information 𝑦 may encompass various modalities, e.g., natural language, semantic segmentation maps and canny edge maps. To pre-processing 𝑦 from various modalities, SD employs a domain-specific encoder 𝜏 𝜃 to project 𝑦 into an intermediate representation 𝜏 𝜃 ( 𝑦 ) ∈ R 𝑀 × 𝑑 𝜏 which is then mapped to the intermediate layers of the

VAE

🔥

Random

select

OCR

detector

VAE

❄

❄

Add

noise

Glyph

encoder

𝑪

U-Net

𝑚

𝑥

!

𝑧

"

𝑧

"#$

𝑥

%

𝑥

&

❄

❄

Frozen models

Learnable models

🔥

OCR

detector

❄

Glyph

encoder

𝑪

Diffusion Model

𝑚

𝑧

!

𝑧

'

𝑥

"

𝑥

&

❄

Can you change the

cash amount to 29.00?

❄

Large

Language

Model

(LLM)

🤖

❄

Optional

(a) Training process

(b) Inference process

KIKU JAPANESE RESTAURANT

53100 SETAPAK, KUALA LUNPUR

TK DIVISION KITCHEN SON BHD

TEL: 03-4161 8936

1202152-P

INV01CE

Date : 17/1/2018 12:12:21

Pax :0

lnveice :9722

Table

！

8311

A10

U-P (R)

PI$G(%)

Art (R8)

BEN02

Set10

TS02 Kat

Sub TotalBefore Discount

Rounding Ad justnent

Service Charge

Total Sales

1

1

12.90

6.90

9. 90

0.00

0.00

0.00

RM 29.70

RM29.70

000

RM 0. 00

12. 90

6. 90

9. 90

TOTAL

RM29.70

CASH

RM 30.00

CHANGE

RM 0.30

TOTAL

CASH

CHAN'E

RW30. 00

RWI 29. 70

RM 0.30

KIR

TOTAL

CASH

CNANE

RM 29. 70

RMI 0. 30

TOTAL

CACH

CHANTE

RW29.

RW30.

RM0.

70

00

30

RM30.00

i   -61

52100 SCTAPWL KOALA LUARUR

TE DTY1SI0W KITGKK SDN BP0

TIL: 00-4561 8936

1202152-P

INV010E

ITIN

Chicken

BNG3

Katsu_Ben (Fish)Set

let/10

TOTAL

CASH

CHANGE

TtRa/1_a/4

17/1/3018 12:12:21

9723

Berto

vF08%

￥.购

.99

Gashiar

RM 29. 70

RM 30. 00

BH80 [%)

9.90

0.00

9.90

RM 0.30

RW 29.7)

R8 29.70

00:00

R 0.00

0671

9.90

4.90

1P-1, JALRN BRL/INUA 2, TAGABE BUNGA RRYA

KIKU JAPANESE RESTAURANT

5I100_SCTAPAKKIALA LUWPUR

9H9 NIS NDOLIX NDISIA10 31

9068V9-0T11

1202152-P

INV010E

Date_: 17/1/2018 12:12:21

Pa.0

Inwgice :9722

P0S 1o:1

3

Dhicken Katau

Set10

BEN02

T902 Kat

TOTAL

Service Dargt

Total Sales

3rivssr[py Ipuo)

Sub Tatal Befere Piecourt

9TY

giNiSet

1

1

Berto

u.+380

12.90

6.90

06′6

FISG(%)

RM 29.70

0.00

9.00

9.00

RI 29.70

829.70

RE 0.00

RI 0.00

12. 90

&. 90

9. 90

CASH

RM 29.00

CHANGE

RM 0.30

TOTAL

CACN

DKANE

90.30

CACH

CHAN'E

TOTAL

RM29.00

UNet via a cross-attention mechanism implementing Attention ( 𝑄, 𝐾,𝑉 ) = softmax ( 𝑄𝐾 ⊤ √ 𝑑 ) · 𝑉 , where 𝑄 = 𝑊 ( 𝑖 ) 𝑄 · 𝜙 𝑖 ( 𝑧 𝑡 ) , 𝐾 = 𝑊 ( 𝑖 ) 𝐾 · 𝜏 𝜃 ( 𝑦 ) , 𝑉 = 𝑊 ( 𝑖 ) 𝑉 · 𝜏 𝜃 ( 𝑦 ) . 𝑊 ( 𝑖 ) 𝑄 , 𝑊 ( 𝑖 ) 𝐾 , 𝑊 ( 𝑖 ) 𝑉 are learnable projection matrices, 𝑑 denotes the output dimension of key ( 𝐾 ) and query ( 𝑄 ) features, and 𝜙 𝑖 ( 𝑧 𝑡 ) ∈ R 𝑁 × 𝑑 𝑖 𝜖 denotes a flattened intermediate representation of the UNet implementing 𝜖 𝜃 . In the scenario of text-to-image generation, the condition 𝐶 = 𝜏 𝜃 ( 𝑦 ) is produced by encoding the text prompts 𝑦 with a pre-trained CLIP text encoder 𝜏 𝜃 . The overall training objective of SD is defined as

Therefore, 𝜏 𝜃 and 𝜖 𝜃 can be jointly optimized via Equation (1).

3 Universal Text Editing Diffusion Model

3.1 Model Overview

The overall training process of our proposed DiffUTE method is illustrated in Figure 2 (a). Based on the cross attention mechanism in SD Rombach et al. [2022], the original input latent vector 𝑧 𝑡 is replaced by the concatenation of latent image vector 𝑧 𝑡 , masked image latent vector 𝑥 𝑚 , and text mask 𝑚 . The condition 𝐶 is also equipped with a glyph encoder for encoding glyph image 𝑥 𝑔 . Introducing text masks and glyph information enables fine-grained diffusion control throughout the training process, resulting in the improved generative performance of the model.

3.2 Perceptual Image Compression

Following Rombach et al. [2022], we utilize a VAE to reduce the computational complexity of diffusion models. The model learns a perceptually equivalent space to the image space but with

significantly reduced computational complexity. Since the V AE in SD is trained on natural images, its ability to restore text regions is limited. Moreover, compressing the original image directly through the VAE encoder causes the loss of dense text texture information, leading to blurry decoded images by the VAE decoder. To improve the reconstruction performance of text images, we further fine-tune the VAE on text image datasets. As shown in our experiments (Section 4.4), training VAE directly on the original image size lead to bad reconstruction results, i.e., unwanted patterns and incomplete strokes. We propose a progressive training strategy (PTT) in which the size of the images used for training increases as the training proceeds. Specifically, in the first three stages of training, we randomly crop images of sizes 𝑆 / 8 , 𝑆 / 4 and 𝑆 / 2 and resize them to 𝑆 for training, where 𝑆 is the resolution of the model input image and 𝑆 = 𝐻 = 𝑊 . Thus, the tuned VAE can learn different sizes of stroke details and text recovery. In the fourth stage, we train with images of the same size as the VAE input to ensure that the VAE can predict accurately when inferring.

3.3 Fine-grained Conditional Guidance

The pixel-level representation of text images differs greatly from the representation of natural objects. Although textual information consists of just multiple strokes of a two-dimensional structure, it has fine-grained features, and even slight movement or distortion lead to unrealistic image generation. In contrast, natural images have a much higher tolerance level as long as the semantic representation of the object is accurate. To ensure the generation of perfect text representations, we introduce two types of fine-grained guidance: positional and glyph.

Positional guidance. Unlike the small differences between natural images, the latent feature distributions of character pixels differ dramatically. Text generation requires attention to specific local regions instead of the existing global control conditions for natural images Zhang and Agrawala [2023], Mou et al. [2023], Cheng et al. [2023] (e.g., segmentation maps, depth maps, sketch and grayscale images). To prevent model collapse, we introduce position control to decouple the distribution between different regions and make the model focus on the region for text generation. As shown in Figure 2 (a), a binary mask is concatenated to the original image latent features.

Glyph guidance. Another important issue is to precisely control the generation of character strokes. Language characters are diverse and complex. For example, a Chinese character may consist of more than 20 strokes, while there are more than 10,000 common Chinese characters. Learning directly from large-scale image-text datasets without explicit knowledge guidance is complicated. Liu et al. [2022a] proposes that the character-blinded can induce robust spelling knowledge for English words only when the model parameters are larger than 100B and cannot generalize well beyond Latin scripts such as Chinese and Korean. Therefore, we heuristically incorporate explicit character images as additional conditional information to generate text accurately into the model diffusion process. As shown in Figure 2 (a), we extract the latent feature of the character image as a control condition.

3.4 Self-supervised Training Framework for Text Editing

It is impossible to collect and annotate large-scale paired data for text image editing, i.e., GLYPH<8> ( 𝑥 𝑠 , 𝑥 𝑔 , 𝑚 ) , 𝑦 GLYPH<9> . It may take great expense and huge labor to manually paint reasonable editing results. Thus, we perform self-supervised training. Specifically, given an image and the OCR bounding box of a sentence in the image, our training data is composed of GLYPH<8> ( 𝑥 𝑚 , 𝑥 𝑔 , 𝑚 ) , 𝑥 𝑠 GLYPH<9> .

For diffusion-based inpainting models, the condition 𝐶 is usually text, which is usually processed by a pre-trained CLIP text encoder. Similarly, a naive solution is directly replacing it with an image encoder. To better represent glyph images, we utilize the pre-trained OCR encoder Li et al. [2023] as the glyph encoder. Such naive solution converges well on the training set. However, the generated quality is far from satisfactory for test images. We argue that the main reason is that the model learns a mundane mapping function under the naive training scheme: 𝑥 𝑔 + 𝑥 𝑠 · ( 1 -𝑚 ) = 𝑥 𝑠 . It impedes the network from understanding text style and layout information in the image, resulting in poor generalization. To alleviate such issue, we use a uniform font style (i.e., "arialuni") and regenerate the corresponding text image, as shown in Figure 2 (a) with the example of "RM 30.00". Thus, we prevent the model from learning such a trivial mapping function and facilitate model understanding in a self-supervised training manner.

Our self-supervised training process is summarized as follows: (1) An ocr region is randomly selected from the image and the corresponding text image is regenerated with a uniform font style. (2) The regenerated character image 𝑥 𝑔 is fed into glyph encoder to get condition glyph embedding 𝑒 𝑔 . (3) The masked image latent vector 𝑥 𝑚 , mask 𝑚 and noisy image latent vector 𝑧 𝑡 is concatenated to form a new latent vector 𝑧 ′ 𝑡 = Concat ( 𝑥 𝑚 , 𝑚, 𝑧 𝑡 ) . After dimension adjustment through a convolution layer, the feature vector ˆ 𝑧 𝑡 = Conv ( 𝑧 ′ 𝑡 ) is fed into the UNet as the query component. Consequently, the training objective of DiffUTE is:

3.5 Interactive Scene Text Editing with LLM

To enhance the interaction capability of the model, we introduced the large language model (LLM), i.e., ChatGLM Zeng et al. [2023]. Moreover, we fine-tuned ChatGLM using the extracted OCR data to facilitate a better understanding of structured information by ChatGLM, The inference process of DiffUTE is show in Figure 2 (b). We first provide the OCR information extracted by the OCR detector and the target that the user wants to edit with to LLM, which will return the target text and its corresponding bounding box. Then, we use bounding boxes to generate mask and masked images, and generate images through a complete diffusion process ( 𝑡 = { 𝑇,𝑇 -1 , ..., 0 } ) by DDIM Song et al. [2020] sampling strategy. By using ChatGLM to understand natural language instruction, we avoid requiring users to provide masks for the areas they want to edit, making our model more convenient.

4 Experiments

4.1 Data Preparation

Due to the lack of large-scale datasets for generating text image compositions, we collect 5M images by combining the web-crawled data and publicly available text image datasets, including CLDA Li, XFUND Xu et al. [2022b], PubLayNet Zhong et al. [2019] and ICDAR series competitions Zhang et al. [2019], Nayef et al. [2019], Karatzas et al. [2015], to prepare our training dataset. To verify the effectiveness of our model, we randomly selected 1000 images from ArT Chng et al. [2019], TextOCR Singh et al. [2021], ICDAR13 Karatzas et al. [2015] and web data collected by ourselves to form the test set, respectively. All the images are cropped/resized to 512 × 512 resolution as model inputs.

4.2 Implementation Details and Evaluation

Implementation details. Our DiffUTE consists of VAE, glyph encoder and UNet. To obtain better reconstruction ability for text images, we first fine-tuned the V AE, which is initialized from the

SRNet

MOSTEL

SD2

ControlNet

SD2-FT

Ours

Input

2023-05-20

LAKERS

自习室

Miracle

Venue

8888

肥

55.00

Campu

VisaPlatinum

座

Description

时间

BURREN

12345

全票

FOAMBOARD

12/20

DESIGN

otected

A.HILLER

交通建设专项资金

肥

55.00

5.00至月阳

Campu

三时间8025051-20

座

Description

LARBERS

自到量

BURREN

-Mmrade

888112345

全票

工

FOAMBOARD

DESIGIN

otected

A.HILLER

4000

12/20

交通建设专项资金

过路

肥

55.00

至青阳

Campu

丰时间

023-05.20

座信

Description

自到

e-IMlcaale

BURREN

888812345

》

全票

FOAMBOARD

4000

12/20

DESIGN

otected

A. HILLER

交通建设专项资金

肥

55.00

Campu

时间

2023-05-20

座信

Description

LAKERS

白习幸

-+Miac1o

BURREN

eute

889812345

全票

FOAMBOARD

s000

12/20

DESIGN

otected

A.HILLER

交通建设专项资金

肥

55.00

5.00至月阳

Campu

VisaPlatinum

三时间2023-05-20

座信

LAKERS

Description

自习

BURREN

-Mltaacie！

Vmue

8988.12345

全票

FOAMBOARD

DESIGIN

otected

交通建设专项资金

肥

55.00

Campu

三时间2023-055-20

座

LAKERS

Description

自习室

e-Mirade

BURREN

Vanee

888812345

全票

FOAMBOARD

12/20

DESIGIN

otected

A.HILLER

交通建设专项资金

肥

55.00

500至月阳

Campu

VisaPlatinum

三时间2023-05-20

座

Description

LAKERS

自习室

e-Miracle

BURREN

Venue

8888123451

全票

FOAMBOARD

12/20

DESIGIN

otected

A. HILLER

交通建设专项资金。过路。

checkpoint of stable-diffusion-2-inpainting 2 . The VAE is trained for three epochs with a batch size of 48 and a learning rate of 5e-6. We use a pre-trained OCR encoder as our glyph encoder, i.e., TROCR Li et al. [2023]. During the training of DiffUTE, we set the batch size to 256, the learning rate to 1e-5, and the batch size to 5. Note that the weights of the glyph encoder and V AE were frozen during the training of DiffUTE.

Evaluation and metrics. In our evaluation, we evaluate the accuracy of the generated text. We report OCR accuracy, calculated separately using pre-trained recognition model Fang et al. [2021] and human evaluation of the correctness between the generated text and the target text, denoted as OCR and Cor, respectively.

Image

Text

rosy glow

Happy

Image

Text

Input

w/o Pos.

young

DiffUTE

Parasolize

Image

Text

Image

Text

w/o PTT

AIRSHOW

FARNBOROUGH

FARNBOROUGH

FARNBOROUGH

FARNBOROUGH

FARNBOROUGH

AIRSHOW

AIRSHOW

AIRSHOW

AIRSHOW

28JULY2002

dailyflying

28JULY2002

28 JULY 2002

28 JULY 2002

28JULY 2002

programme

daily rosy glow

daily rosy low

daily rosy glow

daily rosy glow

programme

programme

programme

programme

IMPORTED

PaphY

hopny

MAPOB

Happy

BEERFROM

IMPORTEO

IMPORTED

IMPORTEO

IMPORTEO

MEXICO

BEER FROM

BEER FROM

BEER FROM

BEER FROM

MEXICO

MEXICO

MEXICO

MEXICO

First Eastern ational

Bus Times

young Eastern National

young Eastern National

youw Eastern National

young Eastern National

Bus Times

Bus Times

Bus Times

Bus Times

Do you Know anyone

udentAccounts

dentAccounts

StudentAccounts

tudent Accounts

startinguniversity]

Do you Know anyone

Do you Know anyone

Do you Know anyone

Do you Know anyone

this year?

starting Paraselize

starting Parasdize

starting Plzaolize

starting Parasolize

this year?

this year?

this year?

this year?

Baseline methods. We compare DiffUTE with state-of-the-art scene text editing methods and diffusion models, i.e., Pix2Pix Isola et al. [2017], SRNet Wu et al. [2019], MOSTEL Qu et al. [2023], SD Rombach et al. [2022], ControlNet Zhang and Agrawala [2023] and DiffSTE Ji et al. [2023]. Pix2Pix is an image translation network. To make Pix2Pix network implement multiple style translations, we concatenate the style image and the target text in depth as the network input. Training of SRNet requires different texts to appear in the same position and background, which does not exist in real-world datasets. Therefore, we use synthtiger Yim et al. [2021] to synthesize images for fine-tuning. For MOSTEL, we fine-tuned it on our dataset. For SD, we selected two baseline methods, i.e., stable-diffusion-inpainting 3 (SD1) and stable-diffusion-2-inpainting (SD2). For fair comparison, we fine-tuned SD1 and SD2 by instruction tuning. The resulting models are termed as SD1-FT and SD2-FT. In the NLP field, instruction tuning techniques are used to train models to perform tasks based on task instructions. We aim to accurately map text instructions to corresponding text edits using the SD model. To achieve this, we constructed a dataset for fine-tuning. Each sample in the dataset consists of a language instruction describing the target text, a mask, and the ground truth. ControlNet is an image synthesis method that achieves excellent controllability by incorporating additional conditions to guide the diffusion process. To adapt this method to our text editing problem, we take the glyph image as the input to the ControlNet network. DiffSTE introduces an instruction tuning framework to train the model to learn the mapping from textual instruction to the corresponding image, and improves the pre-trained diffusion model with a dual encoder design. We followed the original setup to train DiffSTE.

4.3 Comparison results

The quantitative results for text generation are shown in Table 1. We can find that our DiffUTE achieves state-of-the-art results on all datasets. For example, DiffUTE improves average OCR

SD2-VAE

Ours-w/o PTT

Ours

SD2-VAE

Ours-w/o PTT

Ours

Doc No.

Cestwer

DESION

FOAMDOAKD D1X40CM

DESIGN

ROLLUP BUNTINO

CKCSK2F

FLYERS

PRINT EXPERT SDN BHD

40OOO SSAH ALAM, BELANOOR

Total Sases (Excluding OsT)

SO00040264

UNEN

FAX

ATHIRAH

D3002S-A

0355107200

110

OHL

OEPT

100

1

3

2

2

1

1

Dete:

Time

Total OST

Rouncting

Deseount

647 30

176 00

Priee

03 60

51 60

31 80

31.00

073

12/01/2018

16 32.00

Ameun

100 8

350 1

310

05 0

100.

73 1

94 :

01

5)

0

Costwer

NO 16,20,22.JALAN BUNGATANJONG 2/16

DESIGN

ROLLUP OUNTING

FLYERS

CESKN

DEPO8T(CB00490783)

PRINTEXPERT SDN BHD

Tocal Sales (Exctμdng OST)

SO00046204

USEN

092000000001150

FAX0395107900

ATHRAH

T6L09-65190105

003625-A

110

OY

DEPT

100

2

2

1

Date:

Time

Roundhing

Drseount

176.03

64730

6360

Priee.

3160

31 80

31 00

0.73

12/01/2018

10 32.00

Amoun

-047.3

180 9

386 1

656

31.0

100.

73 1

94.:

01

5)

0

CashN

uodusea

DESIGN

FOAMDOARDD1X46CM

DESIGN

ROLLUP BUNTING

FLYERS

DEPO9IT (CS00469793)

PRINTEXPERTSDNBHD

40OOO SHAH ALAM,SELANGOR

Total Gabes (Excluding GST)

O00046284

USER

GST1D000680077504

Total Qty

FAX 03-55107309

ATHIRAH

0B9025-A

110

OIY

DEPT

100

1

3

2

2

1

Date:

Time

Total GST

Rounding

Discount

-647.30

178.08

Ptice

31.80

63 60

31.80

31.60

0.73

12/01/2018

16:32.00

Amoun

-647.3

190.8

3561

318

63.6

100.

31.6

73.1

94.

0.1

5)

授权号：088786

622166******2965

有效期：2016/01

卡行建设银行

：2014/08/25

卡人存程

次号：000071

机水：366342

中国建银行

号：52201202

号：

：105290059990193

RMB

步考号：15132762300

收单行：建设银行

换作号：01

S

间：15:13:27

69.00

:CUP

005747

授权号：088786

小要（TIPS）：

622166******2965

有效期：2016/01

收馨机流水：366342

发卡行建设银行

沃号

持卡人存模

：2814/08/25

电编号：S2201202

：000071

中国建认银行

号：

：105290059990193

RMB

梦考号：1513276230

插作员：01

收单行：建设银行

卡组织：CIP

满算日期：08/25

5S

定号：005747

间：15:13:27

69.00

授权号：088786

小费（TPS）：

批次号：00071

622166******2965

有效期：2016/01

发卡行：建设银行

收银机流水：366342

下

期：2014/68/25

持卡人存根

中国建设银行

号：

：5221282

：105290059990193

RMB

梦考号：1513276230

收单行：建设银行

卡组织：CUP

证号：005747

S9

：15:13:27

69.00

accuracy and human-evaluated text correctness by 14.95% and 14.0% compared with the second best method DiffSTE. Moreover, our method achieves better results than the diffusion model and the finetuned diffusion model because our fine-grained control can provide the model with prior knowledge of glyph and position. Furthermore, the poor performance of the diffusion model for instruction fine-tuning also demonstrates the superiority of our inference approach combining ChatGLM, which can achieve better editing effects.

We further conducted a visualization experiment. As shown in Figure 3, our method successfully achieved the transfer of foreground text and background texture, resulting in a regular textual structure and consistent font with the original text. Moreover, the background texture was clearer, and the overall similarity with real images was improved. In contrast, the results edited using the diffusion model often deviated from the target text, further validating the effectiveness of the glyph condition we introduced. Furthermore, other methods perform poorly when faced with more challenging Chinese text generation tasks, whereas DiffUTE still achieves good generation results.

4.4 Ablation results

The Ablation studies examine three main aspects, namely 1) the effectiveness of the progressive training strategy of VAE, and 2) the impact of position control and glyph control on the image generation performance of DiffUTE. The experimental results are shown in Table 2, Figure 4 and Figure 5.

Progressive training strategy. Without using the progressive training strategy, the editing results become distorted and the accuracy of text generation significantly decreases. The reason for such poor results is the complexity of the local structure of the text, whereby the V AE may need to learn the reconstruction ability of local details efficiently by focusing when there are too many characters in the image. And using our proposed progressive training strategy, the reconstruction ability of the model is significantly improved and more realistic results are obtained. The experimental results validate the effectiveness of this strategy and highlight the pivotal role of V AE in the diffusion model.

Fine-grained control. When position control is not used, the mask and masked images at the input of the UNet are removed. When glyph control is not used, the latent code obtained from the text through the CLIP text encoder is used as the condition. When position control and glyph control are not used, there is a significant drop in performance. For example, when position control is not used, the OCR accuracy of the model drops by 36.7%, and the Cor drops by 38.6%. When glyph control is not used, the model cannot generate accurate text and the OCR accuracy of the model drops by 39.8%, and the Cor drops by 41.5%. These results show that position control can help the model focus on the area where text is to be generated, while glyph control can provide prior knowledge of the shape of the characters to help the model generate text more accurately.

Visualisation. In Figure 6, we provide some examples edited by DiffUTE. DiffUTE consistently generates correct visual text, and the texts naturally follow the same text style, i.e. font, and color, with other surrounding texts. We can see from the experiment that DiffUTE has a strong generative power. (i) In sample N1, DiffUTE can automatically generate slanted text based on the surrounding text. (ii) As shown in sample N2, the input is 234, and DiffUTE can automatically add the decimal point according to the context, which shows that DiffUTE has some document context understanding ability. (iii) In the sample CN4, DiffUTE can generate even artistic characters very well.

Image

1

Text

N0

Image

1234567890

Text

N1

Image

234

Text

N2

Image

12345

Text

N3

Image

1234567895

Text

N4

Image

123456

Text

N5

Image

Ant Group

Text

E0

Image

Text

E1

Image

Nanjing

Text

E2

Image

Lumi

Text

E3

Image

West Lake

Text

E4

Image

Text

E5

Image

南京大学

Text

CN0

Image

天上人间

Text

CN1

Image

万事如意

Text

CN2

Image

千里共婵

Text

CN3

Image

南京

Text

CN4

Image

天筭

Text

CN5

14:15:

数量

1

哈啰助力车骑行

交通出行

哈啰

哈啰助力车骑行

交通出行

1234567890

110

3.40.8046.20

4.2

2.70.6338.80

5.3

1.0056.70

1.2566.40

110

4.2

2.7

3.4

0.63

1.00

0.80

38.80

46.20

56.70

115.00

115.00

100.00

15.00

0

115.00

1.00

115.00

0

餐饮美食

07-15 14:12

艾溪湖东站--彭家桥站

交通出行

张记老上海馄饨铺（火炬...

07-15 14:12

张记老上海馄饨铺（火炬...

餐饮美食

125

7.41.6096.54

4.8

3.9

6.6

3.1

6.0

1.6081.34

1.25

1.00

0.63

79.60

59.40

50.40

72.50

125

4.8

5.3

3.9

6.6

3.1

1.00

1.60

0.63

2.34

1.25

66.40

81.34

59.40

72.50

50.40

100.00

15.00

艾溪湖东站--彭家桥站

交通出行

7.4

6.0

1.60

1.25

96.54

79.60

企业际志及宜节用品

酒序、排水、阳围

安全警示标龙房

五第二

绿化

60

80

3000

60

30

企业标志及宜传用品

道路、排水、地面

现场围挡、大门

安全警示标志牌

五牌二图

8000

6000

1245

3000

600

山西省汽车客票

山站

码00077035

山西省汽车客票

山站

码

00077035

123456789522

律。

网：【解析】5

选。

图①②③分为一组，2条对称轴重合：图

形组成，且均为“等膜“图形，分别面出对

称轴。

但这种思考不严谨，两组更

【选0】

每

幅图均面2个图

形组成，且均为“等腰“图形，分别画出对

图①②③分为一组，2条对称轴重合：图

有学员考虑图①②③对称，图③

称轴。

③④@分为一组，2条对称轴垂直，B项当

选。

助粒酱阿：【解析】5.每幅图均由2个图

123456

四

8000

500

材料堆放设施及仑库（易燃、易爆物品）

鼎化

8000

3000

74.0五台山晋H28944华

*0077035*

*0077035*

这位就是小

长短边应读会比较明显

就是想太多

律。【选B】

天街小雨润如舒：111

这位就是小王吧：下一题吧

现场防火及消防器材配置

000

74.0五台山晋H28944华

王大头potato：长短边应该会比较明显

这位就是小王吧：就是想太多

<10

12SCTV4

ve5555046

7/50

12SCTV4

V85555046

7/50

即购买

CarogaPharma

CarogaPharma

双流电视台

7/3513

双流电视台

首单保障计划

LaborexTanzania Limited

最内江

ejang0832

7/56

14最内江

Nanjing

7/35

包邮吗

为确保您资金安全，请遵守疑范一定在平台内成交。

了解闲鱼安全交易指南

22:40

AlRazi

LaborexTanzania Limited

15达州新闻

dztow

7/42

15达州新闻

zuinejang0832

7/56

包邮吗

dztvxw

7/42

SubTotal

RH

RH

15.00

15.00

0.00

Total ExCl.6GSI RH

Sub Total

Discount

RM

RM

15.00

0.00

1X35.0000

Discount

@6%

WestLakenclusiveosi@6%

1X35.0000

生活饮用水标准检验方法金属指标生活饮用水标准检验方法金属指标

电感耦合等离子体发射光谱法

GB/T 5750.6-2006 1.4

电感耦合等离子体发射光谱法

GB/T 5750.6-2006 1.4

Total Incl.6GST

Rounding

CHANGE

15.00

15.00

0.00

000

0.00

GST6#

Total Incl.6GST

Rounding

CHANGE

Lumi

RM

RM

RH

RM

RM

15.00

15.00

15.00

0.00

0.00

0.00

CHANGE

Total

Round Ad,

Final Total

CASH

Total

Round Adj

CASH

CHANGE

Final Total

生活饮用水标准检验方法金属指标

电感耦合等离子体发射光谱法

电感耦合等离子体发射光谱法

GB/T 5750.6-2006 3.5

2.3

生活饮用水标准检验方法金属指标

生活饮用水标准检验方法金属指标

电感耦合等离子体发射光谱法

电感耦合等离子体发射光谱法

GB/T 5750.6-2006 3.5

Confucius Tempe6 2.3

斯盾

品

台州

台州

布龙路

布龙路

货号：BQ13059

货号：BQ13059

du

地级市

天上人间地级市

正坑

材料：进口头层牛皮

积9411平方公里，领海和内；

台州市地处浙江省沿海中部，

州，西连丽水、金华，紧接纟

清

平

万事如意

®

BQ13059

里，台州市的地理位置得天独

丘陵相间，形成“七山一水二：

积9411平方公里，领海和内

高

里，台州市的地理位置得天

BQ13059

口

?

口

3

用户_098

用户_098

国务院服务充值中心

医保

刷医保

公安交管疫情疫苗

一

国务院服务充值中心千里共婵

刷医保

会员名：g***4

￥800.00

南京

会员名：g***4

￥800.00

10223

天第10223

医保

公安交管

疫情疫苗

全新

含运费0.00元

为你推荐

深圳热门就学

一

交易前聊一聊

全新

含运费0.00元

（元）

为你推荐

深圳热门

就学

交易前聊一聊

（元）

5 Related Works

5.1 Scene Text Editing

Style transfer techniques based on Generative Adversarial Networks (GANs) have gained widespread popularity for scene text editing tasks Roy et al. [2020], Huang et al. [2022], Kong et al. [2022], Lee et al. [2021], Shimoda et al. [2021], Yang et al. [2020], Zhan et al. [2019]. These methods typically involve transferring text style from a reference image to a target text image. STEFANN Roy et al. [2020], for instance, leverages a font-adaptive neural network and a color-preserving model to edit scene text at the character level. Meanwhile, SRNet Wu et al. [2019] employs a two-step approach that involves foreground-background separation and text spatial alignment, followed by a

fusion model that generates the target text. Mostel Qu et al. [2023] improves upon these methods by incorporating stroke-level information to enhance the editing performance. However, despite their reasonable performance, these methods are often constrained in their ability to generate text in arbitrary styles and locations and can result in less natural-looking images.

5.2 Image Editing

Text-guided image editing has attracted increasing attention in recent years among various semantic image editing methods. Early works utilized pretrained GAN generators and text encoders to progressively optimize images based on textual prompts Bau et al. [2021], Gal et al. [2021], Pérez et al. [2003]. However, these GAN-based manipulation methods encounter difficulties in editing images with complex scenes or diverse objects, owing to the limited modeling capability of GANs. The rapid rise and development of diffusion models Rombach et al. [2022], Saharia et al. [2022b], Ruiz et al. [2023] have demonstrated powerful abilities in synthesizing high-quality and diverse images. Many studiesBrack et al. [2023], Brooks et al. [2023] have employed diffusion models for text-driven image editing. Among various diffusion models, Stable Diffusion Rombach et al. [2022] is one of the state-of-the-art models, which compresses images into low-dimensional space using an auto-encoder and implements effective text-based image generation through cross-attention layers. This model can easily adapt to various tasks, such as text-based image inpainting and image editing.

However, it has been observed that diffusion models exhibit poor visual text generation performance and are often prone to incorrect text generation. Only a few studies have focused on improving the text generation capability of diffusion models. Recently, one study trained a model to generate images containing specific text based on a large number of image-text pairs Liu et al. [2022b]. However, this work differs from ours in terms of application, as they focus on text-to-image generation, while ours concentrates on editing text in images. Another ongoing work, ControlNet Zhang and Agrawala [2023], has demonstrated remarkable performance in image editing by providing reference images such as Canny edge images and segmentation maps. While ControlNet achieves remarkably impressive results, it performs poorly on text editing tasks. To obtain better editing results, we incorporate auxiliary glyph information into the conditional generation process and emphasize local control in all diffusion steps.

5.3 Large Language Model

Large language models (LLMs) refer to language models that contain billions (or more) of parameters, which are trained on massive amounts of text data, such as models like GPT-3 Brown et al. [2020], Galactica Taylor et al. [2022], LLaMA Touvron et al. [2023] and ChatGLM Zeng et al. [2023]. Among them, ChatGLM is a billion-scale language model with rudimentary question-answering and conversational capabilities. It differs from BERT Devlin et al. [2018], GPT-3 and T5 Xue et al. [2021] architectures and is a self-regressive pre-training model that includes multiple objective functions. In this paper, we use ChatGLM to enhance the interaction capability of our model.

6 Conclusion and Limitations

In this paper, we argue that the current diffusion model can not generate realistic text in images. To tackle this problem, we present a novel method DiffUTE, a diffusion-based universal text editing model. DiffUTE generates high-quality text through fine-grained control of glyph and position information, and benefits from massive amounts of text images through a self-supervised training approach. Moreover, by integrating a large language model (i.e., ChatGLM), we can use natural language to edit the text in images, enhancing the editing usability and convenience of model. Extensive experiments have shown that DiffUTE excels in textual correctness and image naturalness.

The main limitation of our method is that the accuracy of generated text will decrease as the number of characters to be edited in the image increases. This is due to the fact that as the number of characters increase, the spatial complexity of the characters will also increase, making the generation process more challenging. Therefore, our future work will focus on improving the generation quality and solving the problem of rendering long texts.

References

Zhuoer Xu, Guanghui Zhu, Changhua Meng, Zhenzhe Ying, Weiqiang Wang, Ming Gu, Yihua Huang, et al. A2: Efficient automated attacker for boosting adversarial training. Advances in Neural Information Processing Systems , 35:22844-22855, 2022a.

Zhangxuan Gu, Zhuoer Xu, Haoxing Chen, Jun Lan, Changhua Meng, and Weiqiang Wang. Mobile user interface element detection via adaptively prompt tuning. In CVPR , pages 11155-11164, 2023.

Chao Zhang, Huaxiong Li, Yang Gao, and Chunlin Chen. Weakly-supervised enhanced semanticaware hashing for cross-modal retrieval. IEEE Trans. Knowl. Data Eng. , 35(6):6475-6488, 2022a.

Li Niu, Junyan Cao, Wenyan Cong, and Liqing Zhang. Deep image harmonization with learnable augmentation. In ICCV , pages 7482-7491, 2023.

Haoxing Chen, Zhangxuan Gu, Yaohui Li, Jun Lan, Changhua Meng, Weiqiang Wang, and Huaxiong Li. Hierarchical dynamic image harmonization. In ACM MM , 2023.

Tingliang Feng, Wei Feng, Weiqi Li, and Di Lin. Cross-image context for single image inpainting. In NeurIPS , 2022.

Jiangning Zhang, Chao Xu, Jian Li, Yue Han, Yabiao Wang, Ying Tai, and Yong Liu. Scsnet: An efficient paradigm for learning simultaneously image colorization and super-resolution. In AAAI , pages 3271-3279, 2022b.

Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with a single text condition. In CVPR , pages 18041-18050, 2022.

Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting. Sega: Instructing diffusion using semantic dimensions. arXiv preprint arXiv:2301.12247 , 2023.

Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR , 2023.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , volume 35, pages 36479-36494, 2022a.

Liang Wu, Chengquan Zhang, Jiaming Liu, Junyu Han, Jingtuo Liu, Errui Ding, and Xiang Bai. Editing text in the wild. In ACM MM , pages 1500-1508, 2019.

Yadong Qu, Qingfeng Tan, Hongtao Xie, Jianjun Xu, Yuxin Wang, and Yongdong Zhang. Exploring stroke-level modifications for scene text editing. In AAAI , 2023.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In CVPR , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI , pages 234-241, 2015.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. 2023.

Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 , 2023.

Shin-I Cheng, Yu-Jie Chen, Wei-Chen Chiu, Hung-Yu Tseng, and Hsin-Ying Lee. Adaptively-realistic image generation from stroke and sketch with diffusion model. In WACV , pages 4054-4062, 2023.

Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering. arXiv preprint arXiv:2212.10562 , 2022a.

Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. Trocr: Transformer-based optical character recognition with pre-trained models. In AAAI , 2023.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In ICLR , 2023.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2020.

Hang Li. Cdla: A chinese document layout analysis (cdla) dataset. [Online]. Available: https: //github.com/buptlihang/CDLA . Accessed 2021.

Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, and Furu Wei. Xfund: A benchmark dataset for multilingual visually rich form understanding. In Findings of ACL , pages 3214-3224, 2022b.

Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Publaynet: largest dataset ever for document layout analysis. In ICDAR , pages 1015-1022, 2019.

Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, Mingkun Yang, et al. Icdar 2019 robust reading challenge on reading chinese text on signboard. In ICDAR , pages 1577-1581, 2019.

Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowdhury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, Jean-Christophe Burie, Cheng-lin Liu, et al. Icdar2019 robust reading challenge on multi-lingual scene text detection and recognition-rrc-mlt-2019. In ICDAR , pages 1582-1587, 2019.

Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman K. Ghosh, Andrew D. Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, Faisal Shafait, Seiichi Uchida, and Ernest Valveny. ICDAR 2015 competition on robust reading. In ICDAR , pages 1156-1160, 2015.

Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al. Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art. In ICDAR , pages 1571-1576, 2019.

Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In CVPR , pages 88028812, 2021.

Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition. In CVPR , pages 7098-7107, 2021.

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR , pages 1125-1134, 2017.

Jiabao Ji, Guanhua Zhang, Zhaowen Wang, Bairu Hou, Zhifei Zhang, Brian Price, and Shiyu Chang. Improving diffusion models for scene text editing with dual encoders. arXiv preprint arXiv:2304.05568 , 2023.

Moonbin Yim, Yoonsik Kim, Han-Cheol Cho, and Sungrae Park. Synthtiger: Synthetic text image generator towards better text recognition models. In ICDAR , pages 109-124, 2021.

Prasun Roy, Saumik Bhattacharya, Subhankar Ghosh, and Umapada Pal. Stefann: scene text editor using font adaptive neural network. In CVPR , pages 13228-13237, 2020.

Qirui Huang, Bin Fu, Yu Qiao, et al. Gentext: Unsupervised artistic text generation via decoupled font and texture manipulation. arXiv preprint arXiv:2207.09649 , 2022.

Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, and Lianwen Jin. Look closer to supervise better: One-shot font generation via component-based discriminator. In CVPR , pages 13482-13491, 2022.

Junyeop Lee, Yoonsik Kim, Seonghyeon Kim, Moonbin Yim, Seung Shin, Gayoung Lee, and Sungrae Park. Rewritenet: Reliable scene text editing with implicit decomposition of text contents and styles. arXiv preprint arXiv:2107.11041 , 2021.

Wataru Shimoda, Daichi Haraguchi, Seiichi Uchida, and Kota Yamaguchi. De-rendering stylized texts. In ICCV , pages 1076-1085, 2021.

Qiangpeng Yang, Jun Huang, and Wei Lin. Swaptext: Image based texts transfer in scenes. In CVPR , pages 14700-14709, 2020.

Fangneng Zhan, Hongyuan Zhu, and Shijian Lu. Spatial fusion gan for image synthesis. In CVPR , pages 3653-3662, 2019.

David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba. Paint by word. arXiv preprint arXiv:2103.10951 , 2021.

Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108.00946 , 2021.

Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing. In ACM SIGGRAPH , pages 313-318. 2003.

Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH , pages 1-10, 2022b.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2023.

Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering. arXiv preprint arXiv:2212.10562 , 2022b.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS , volume 33, pages 1877-1901, 2020.

Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 , 2022.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT , 2018.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In ACL , 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes an application of diffusion models (Sohl- Dickstein et al., 2015; Ho et al., 2020) to text editing. Methodologically, this work differs from previous text diffusion (Li et al., 2022) by leveraging insights on glyph encoder and OCR detector. Empirically, this work advances the state of the art for text editing by scaling these methods to larger datasets. The paper also proposes to use self-supervised training to train the diffusion model and further explore diffusion guidance.\n\nSTRENGTHS:\n1. This is the work on diffusion LMs that shows results on a text editing baseline and seems to have positive results in terms of the metrics used.\n2. The writing is clear and the motivations seem sound.\n\nWEAKNESSES:\n1. The main weakness is the novelty. The core idea of this paper, i.e. latent diffusion, has been demonstrated to be successful in many generation tasks. Thus it is not surprising that it works on scene text editing. Most of the techniques used in the paper have been proposed perviously.\n2. The author did not provide any details regarding the position control module, thus the ablation study of this part is not convincing.\n3. The authors did not evaluate a variety of evaluation measures that prior work has done such as SSIM, MSE, PSNR, and many more. These metrics should be computed to a get better idea of the quality and diversity of the output. Please see these this paper for the description of these metrics: “ Krishnan P, Kovvuri R, Pang G, et al. Textstylebrush: transfer of text aesthetics from a single example[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.”.\n4. There are missing comparisions such as Krishnan et al 2023's TextStyleBrush[1] and Ji’s 2023’s DiffSTE [2]. \n5. The model rely on a pretrained OCR encoder which just seems like an arbitrary choice. An ablation should be provided with different pretrained encoders to understand the impact of this choice.\n\n[1] Krishnan P, Kovvuri R, Pang G, et al. Textstylebrush: transfer of text aesthetics from a single example[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.\n[2] Ji, Jiabao, et al. \"Improving Diffusion Models for Scene Text Editing with Dual Encoders.\" arXiv preprint arXiv:2304.05568 (2023).\n\nQUESTIONS:\nThe author mentioned that \"in the first three stages of training, we randomly crop images of sizes 𝑆/8, 𝑆/4 and 𝑆/2 and resize them to 𝑆 for training\", however, which are the three stages and where is the fourth one? Moreover, an ablation that would be of interest is to train with different resolutions at different stages.\n\nLIMITATIONS:\nBeyond the weaknesses I listed, the authors were good at addressing several limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present DiffUTE, a universal self-supervised text editing diffusion model for language-guided image editing. They address the limitations of existing diffusion models by focusing on rendering accurate text and text style during image generation. DiffUTE incorporates modifications to the network structure, allowing it to handle multilingual character drawing using glyph and position information. Furthermore, a self-supervised learning framework leverages a large amount of web data to enhance the model's representation ability. The experimental results showcase the impressive performance of DiffUTE, demonstrating its ability to achieve high-fidelity and controllable editing on diverse real-world images. Overall, this paper presents a significant advancement in language-guided image editing and offers a promising approach for rendering realistic and customizable text in generated images.\n\nSTRENGTHS:\n- The problem addressed in this paper is a realistic problem that current diffusion models struggle to handle effectively.\n\n- The incorporation of LLM  into the inference process is a compelling and intriguing approach.\n\nWEAKNESSES:\n- The paper claims significantly better results than other baselines in Table 1. However, it would be helpful to clarify if there are other baselines that have not been adequately considered.\n\n- A simple baseline is missing. Have the authors considered directly replacing the \"source text\" with the \"target text\" and calculating the FID (Fréchet Inception Distance)?\n\n- In Table 1, the results for SD1-FT and SD2-FT appear to be poor. It would be valuable to explain the main differences between your method and these baselines.\n\n- There is limited mention of other methods that fine-tune the encoder-decoder. How important is this step? Additionally, could you provide details on the difference in parameter numbers shown in Table 1?\n\n- The discussion regarding self-guidance is absent, despite the proposal of a self-supervised approach for achieving text editing in diffusion. The related papers are:\n\n- Self-Guided Diffusion Models\n- Why Are Conditional Generative Models Better Than Unconditional Ones?\n- Visual Chain-of-Thought Diffusion Models\n\nQUESTIONS:\nAs Above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
The authors propose a method of fine-tuning Stable Diffusion to modify words in images, while maintaining the original font style and the background region.
Specifically, they first fine-tune the VAE with text images from several datasets.
Then, utilizing an off-the-shelf OCR detector, they randomly mask out one text box, and tune the denoising network asking it to fill the region with original text that is given as the condition in cross-attention layers.
With this simple and intuitive method, they achieve improved performance in various evaluations.

STRENGTHS:
- the authors tackle a meaningful task
- the proposed method is simple and easy to reproduce
- the proposed method demonstrates improved performance on various evaluation metrics

WEAKNESSES:
- would only work for texts that can be detected by off-the-shelf OCR detectors
- there are missing details on some parts of the method

QUESTIONS:
- not sure whether this method is self-supervised or not, since it requires the utilization of an OCR detector in training
- could you elaborate more on a glyph image? for example, what is the output format of the glyph encoder?
- could you compute the accuracy of ChatGLM’s predictions? and how much cost does it take to fine-tune it?
- it seems that the figure 3 is redundant. maybe you can incorporate it into the figure 2?

LIMITATIONS:
please refer to the Weaknesses section
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DiffUTE for general text editing. \n\nDiffUTE utilizes Stable Diffusion model with several specific model designs, progressive training strategy, positional and glyph guidance, and a self-supervised training framework. \n\nEquipped with these designs, DiffUTE achieves remarkable results compared to other baselines on several public datasets. \nMoreover, the authors also provide a chat-based interface which enables an easier manipulation for the users.\n\nSTRENGTHS:\nOriginality, motivation and significance:\n- The paper shades an interesting perspective to edit text using pre-trained Stable Diffusion model. Two motivations raised in Line 29 and Line 32 are intuitive. \n- The interaction module is interesting and easy to use. \n\nTechnical approach: \n- Finetuning VAE with a progressive training strategy  (PTT) with different image sizes in different stages is a good choice to overcome blurry outputs. As shown in Table 2, with PTT, DiffUTE has a noticeable gain. \n- The insight into generating fine-grained texts makes sense. With positional and glyph guidance, DiffUTE generates texts with natural shapes.\n- Proposed self-supervised training strategy is straight-forward and useful. It also reduces the need of human annotations. \n\nClarity: the paper offers a smooth writing and is easy to follow.\n\nWEAKNESSES:\n- The motivation of using diffusion models v.s. GANs is not clearly stated. Why would the authors prefer to use diffusion model (e.g., Stable Diffusion)? \n- The paper lacks some failure case analysis. For example, DiffUTE relies on pretrained OCR detector. What if the OCR detector compromises in some cases?\n\nQUESTIONS:\nWriting:\n- Line 129: it is better explicitly to explain what $x_m$ is at the first place.\n\nLIMITATIONS:\nThe authors emit some ethical discussions in the paper. For example, the authors should discuss the misusage of the technique for misinformation spread.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DiffUTE, an innovative diffusion-based text editing framework designed to seamlessly fill in missing words in an image with user-specified text. By employing a self-supervised training framework, the model effectively learns from an extensive collection of synthetic data pairs, enabling it to infer accurate text styles and generate images that seamlessly incorporate the desired text. Experimental results showcase remarkable qualitative text editing performance from both the model's precision in both text and style accuracy. Additionally, quantitative analysis shows that the proposed method surpasses the performance of baseline approaches.\n\nSTRENGTHS:\n+ The proposed method exhibits impressive editing performance, as demonstrated through extensive experiments. It displays a strong ability to accurately infer text styles and generate corresponding images.\n\n+ Leveraging LLM, the model offers broad applicability across many possible application scenarios.\n\n+ The paper is organized clearly and is easy to read.\n\nWEAKNESSES:\n- Quantitative metrics for style: Although the paper effectively showcases the model's ability to generate text that is stylistically consistent with the rest of the images, it does not provide a quantitative analysis or specific metrics to support this claim.\n- Alternative diffusion-based baselines: While ControlNet is a powerful diffusion-based editing framework, it is not specifically designed for text editing tasks. Another recent diffusion-based editing approach, DiffSTE[1], shares similarities with this work as it focuses on specialized text editing and exhibits commendable performance. How does the proposed method compare to DiffSTE in terms of performance?\n\n[1] Improving Diffusion Models for Scene Text Editing with Dual Encoders.\n\nQUESTIONS:\n1. Are there any metrics available for assessing the accuracy/consistency of text style? Or potentially this can be validate through a human study similar to the Cor metrics present in paper.\n2. Performance comparison with other diffusion-based text-editing method, DiffSTE?\n3. An intriguing aspect of this work is its remarkable ability to accurately infer text styles, even when multiple possible texts are present within an image. For instance, in Figure 4, column 1, we can observe that the imprinted time exhibits the correct style (black), despite the presence of additional red texts. It raises the question of how robust this achievement is. Specifically, does the model consistently succeed when confronted with multiple text categories? Furthermore, can this success be attributed to different types of characters, such as numerical values versus Chinese characters, as depicted in this example?\n\nLIMITATIONS:\nThis paper clearly discusses the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}