Improving Diffusion Models for Scene Text Editing with Dual Encoders
Authors: Jiabao Ji
Guanhua Zhang
Zhaowen Wang
Bairu Hou
Zhifei Zhang
Brian L. Price
Shiyu Chang
Venue: Accepted by TMLR ()

Abstract: Scene text editing is a challenging task that involves modifying or inserting specified texts in an image while maintaining its natural and realistic appearance. Most previous approaches to this task rely on style-transfer models that crop out text regions and feed them into image transfer models, such as GANs. However, these methods are limited in their ability to change text style and are unable to insert texts into images. Recent advances in diffusion models have shown promise in overcoming these limitations with text-conditional image editing. However, our empirical analysis reveals that state-of-the-art diffusion models struggle with rendering correct text and controlling text style. To address these problems, we propose DIFFSTE to improve pre-trained diffusion models with a dual encoder design, which includes a character encoder for better text legibility and an instruction encoder for better style control. An instruction tuning framework is introduced to train our model to learn the mapping from the text instruction to the corresponding image with either the specified style or the style of the surrounding texts in the background. Such a training method further brings our method the zero-shot generalization ability to the following three scenarios: generating text with unseen font variation, e.g., italic and bold, mixing different fonts to construct a new font, and using more relaxed forms of natural language as the instructions to guide the generation task. We evaluate our approach on five datasets and demonstrate its superior performance in terms of text correctness, image naturalness, and style controllability.

Published in Transactions on Machine Learning Research (Feb/2024)
Improving Diffusion Models for Scene Text Editing with Dual Encoders
Jiabao Ji Guanhua Zhangl*
Zhaowen Wang? Bairu Hou Zhifei Zhang²
Brian Price²
Shiyu Chang
University of California, Santa Barbara Adobe Research
Equal contribution. Work done while Guanhua was affiliated with UCSB.
Corvespondance to Kjabaojj@uussb. edu> hhw.eeeneeenattttegnneeeee me...i.n.ggnm............
Reviewed on OpenReview: https: // openreview. net/ por/fo id- yL1555555
Abstract
Scene text editing is a challenging task that involves modifying or inserting specified texts in an image while maintaining its natural and realistic appearance. Most previous approaches to this lask rely style-transfer models that crop out text regions and feed them into image transfer models, such as GANs, However, these methods are limited in their ability to change text style and are unable to insert texts into images. Recent advances in diffusion models have shown promise in overcoming these limitations with text-conditional image editing. However, our empirical analysis reveals that state-of-the-art diffusson models struggle with rendering correct text and controlling text style. To address these problems, we propose DIFFSTE to improve pro-trained diffusion models with dual encoder design, which includes character encoder for better lext legibility and an instruction encoder for better style con- trol. An instruction tuning framework is introduced to train our model to learn the mapping from the text instruction to the corresponding image with either the specified style or the style of the surrounding texts in the background. Such a training method further brings our method the zero-shot generalization ability to the following three scenarios: generating text with unseen font variation, e.g., italic and bold, mixing different fonts to construct new font, and using more relaxed forms of natural language the instructions to guide the generation task. We evaluate our approach five datasets and demonstrate its superior performance in terms of text correctness, image naturalness, and style controllability. Our code is publicly available al httpp://githutt CCC/ CCtp//UCSSS-LLP-CCaaa/111111
1 Introduction
Scene text editing has gained significant attention in recent years due to its practical applications in various
fields, including text image synthesis Karacan et al. (2016) Qin et al. (2021) | Zhan et al. (2018). styled text transfer Atarsaikhan et al. (2017): Azadi et al. 2018 Zhang et al. (2018). and augmented reality translation Cao et al. (2023); Du et al.
b Fragoso et al. (001)). The task involves modifying or inserting specified text in an image while maintaining its natural and realistic appearance Huang et al. (0022): ; Shi
ct al. p Wu al. (2019). Most previous approachos have formulated the problem as style transfer
Lee fask using generative models such GANSs Lee et al. (2021) ; Park al. (2021); Qu el. al. 2022
Roy et al.)
(2020): Wu et al. (2019): Yang et al. (2020)
Specifically, these methods rely on reference image with the target style, e.g., cropped out text region that needs to be modified. The mothod then transfers rendered text in the desired spelling to match the style and the background of the reference. However, these methods are limited in their ability to generate text in arbitrary styles (e.g., font and color) or at arbitrary locations. Additionally, the process of cropping, transferring style, and then replacing back often leads less natural-looking results. Figure illustrates image generated by MOSTEL Qu et. al. (2222) a state-of-the-art GAN-based scene text editing model.

---

Published in Transactions on Machine Learning Research (Feb/2024)
The edited part of the image exhibits discordance with the surrounding areas with distinct boundaries and messy distortions.
On the other hand, recent advances in diffusion models have shown promise for overcoming these limitations and creating photo-realistic images Bal- aji et al. (2022); Ho et al. (2020); [Saharia et al. (2022); ; Song et al. (2020). Furthermore, by incor- porating text encoder, diffusion models can be adapted to generate natural images following text instructions, making them well-suited for the task of image editing Ramesh et al. (2022); Rombach et al. 2022). Despite the remarkable success of diffusion models in general image editing, our empirical study reveals that they stilll struggle with generating accu- rate texts with specified styles for scene text editing. For instance, Figure (d) shows an example of the text generated by stable-diffusion-inaaitii inpainting model. We feed the model with the masked image and a text instruction, requiring the model to complete the masked region. Although the resulting image shows better naturalness, neither the spelling of the generated text is correct nor the specified style is well followed by the model.
DifSSTE
COSSTE
DIffeTE
INPUT
RENDERED
MOSTEL
SD
DIFFSTE
Text editing results by GAN-based Mos- TEL, diffusion model SD, and our method DIFFSTE. A reference RENDERED image with desired content and style is fed to MOSTEL as style guidance. SD and our method are prompted by instruction Write a grass- colored word "DiffSTE" in BadScript font.
Figure 1:
(SD) Rombach et al. (2022), a statc-of-thhear text-conditional
Motivated by this, in this paper, we aim to improve the pre-trained diffusion models, especially SD, in their scene text editing ability. In particular, we identify two main issues as the cause of the diffusion model's failure: @ although the target text is fed the character-level information of the target text, making it challenging to accurately map the spelling to appropriate visual appearance; @ the models are unable to understand the language instructions for a stylc-specified generation, which lcads to the incorrect font or color generation. To address these challenges, we propose a novel model for scene lext editing called DIFFSTE, which incorporates a dual encoder design comprising of an instruction encoder and a character encoder. Specifically, DIFFSTE is built upon SD, and the instruction encoder is inherited from the original SD's text encoder. The input to this encoder describes the target text and its desired style, which is then utilized in cross-attention to guide the generation process. target
part of the input to the text encoder of SD, it cannot encode
On the other hand, the character encoder is an add-on module that only takes the character- text. The encoded character embeddings are then attended by the image encodings and further aggregated together with the attended result from the instruction encoder side. Having direct access to the characters equips the model with the capability of explicit spelling checks and the awareness of larget text length.
To make both encoders understand the scene text editing task, we further introduce an instruction-tuning framework for training model. Instruction tuning is a popular technique in the natural language process-
ing (NLP) domain Gupta et al. (2022); Mishra et al. (2022): Sanh et al. (2021): ): Wei et al. (2021): h; Honovich
ct al. [022]; Wang et al. (2022); Xu et al. (2022) : Zhong et al. (2022); Zhou et al. (2022). It aims to teach foundation language models to perform specific lasks based on the corresponding lask instructions. By doing so, models can achieve better performance on both training and unseen tasks with appropriate instructions Chung et al. (2022): Chen et al.] (022);; Gu et al. (2022); ; Jang et al.] (2023); Muennighoff et al. (2022): 9: Brooks et al. (2022). Similarly, our goal is to teach DIFFSTE to capture the mapping from the text instruction to scene text generation. To achieve this, we create a synthetic training datasel, where each training example includes three components: ① a text instruction that describes the desired target text, @ a masked region surrounded by other scene texts in similar style indicating the location in the image where the text should be generated, and @ a ground truth reference image. The text instructions are generated using a fixed natural language rule that can be grouped into four main categories: specifying both color and font, specifying color only, specifying font only, and providing no style specification. If a text instruction is missing either color or font information or both, the scene text in the surrounding area of the image can be used to infer the missing information.
Our model is trained to generate the ground-truth scene text by minimizing mean square error (MSE) loss conditioned on the provided instructions and the unmasked image region using both synthetic and real-world datasets. This enables our model to achieve both spelling accuracy and good style control. An example of
2

---

Published in Transactions on Machine Learning Research (Feb/2024)
the generated image can be found in Figure [ Even more surprisingyy, our model performs well on test instructions that were significantly different from the ones used for training, demonstrating its zero-shot generalization ability. Specifically, our model generalizes well to the following three scenarios: generating text with unseen font variation, e.g., italic and bold, mixing different fonts to construct a new font, and using more relaxed forms of natural language as the instructions to guide the generation task. Figure 2 showcases the new font style created by DIFESTE, which mixes the fonts PressStart2P and MoonDance.
We compared our approach with state-of-the-art baselines on five datasets, considering both style-free and style-conditional text editing scenarios. Our method showed superior performance in terms of text correctness, image naturalness, and style con- trol ability, as evaluated quantitatively and sub- jectively. For instance, under the style-frec text editing setting, DIFFSTE achieves an average im- provement of 64.0% on OCR accuracy compared to the most competitive bascline. Additionally, our approach maintained high image naturalness, as demonstrated by 28.0% more preference over other baseline methods in human evaluations. In the style- conditional scenario, our method shows 69.2% and 26.8% more preference compared to other diffusson basclines in terms of font correctness and color correctness, with consistent improvements in text correctnoss and naturalness observed in the style-free case.
Greetings Greetings
Greelings
Greetings
Greetings
Font1
Font2
+ont1+Font22
Figure 2: Generating text in new style by blending two existing fonts. The first row shows the texts rendered with real fonts, and the second row shows our gen- eration results with instructions Wrile "Greetings" in font: FONT, where FONT is PressStart2P, MoonDance and PressStart2P and MoonDance, respectively.
2 Related Work
Scene text editing GAN-based style transfer methods have been widely used for the task of scene text editing Huang et al. (2022): Kong et al. (202bb); Lee et al. (2021); : [oy et al. (2020); Shimoda et al. (2021): Yang et al. (2020); Zhan et al. [2019]- These works accomplish the task by transferring the text style in a reference image to the rendered target text image. STEFANN Roy ct al. (2020) edits scene text at the character level with a font-adaplive neural network for extracting font structure and a color-preserving model for extracting text color. SRNet Wu et al.] (2019) breaks down scene text editing into two sub-tasks, which involve extracting the image background and text spatial alignment, and fusion model is then used to generate the target text with the extracted information. Mostel Qu et al. (2022) further incorporates additional stroke-level information to improve scene text editing performance. To reduce the dependence of paired synthetic data with source style images, and target style images, TextstyleBrush Krishnan et al. (2023) proposes to disentangle the text appearance into content and style vectors in a self-supervised manner, which allows the utilization of real-world text images. Another recent work, SSTE [S et al. (2023), proposes to embed the text styles in the latent feature space, thus allowing users to control the text style, such as
text rotation, text color, and font via latent space editing similar to StyleGAN Karras al. (2019) Despite the reasonable performances, these methods are limited in generating text in arbitrary styles and locations and often result. in less natural-looking images.
Diffusion models Recent advances diffusion models have achieved significant success image edit- ing Balaji et al. 2022]; ; Brack et al. (2023); Ho al. (2020); Li al. (2023): Nichol et al.] (2021); Nichol & Dhariwal 2202 Ryu & Ye (2022) Wu & De la Torre (202)); Wu et al. (2022); Xie et al. 2022. Stable diffusion Rombach et al. (2022) is one of the state-of-tin ahe diffusion models, which enables effective text- conditional image generation by first compressing images into low-dimensional space with an autocncoder and then leveraging text. encodings with a cross-atiention layer. The model could be easily adapled for var- ious tasks like image inpainting and image editing conditioned on texts. However, it has been observed that the diffusion models exhibit poor visual text generation performance and are often susceptible to incorrect text generation Ramesh et al. (2022): [Saharia et al. (2022). There is a limited amount of research focusing improving the text-generation capabilities of diflusion models. A recent study has sought to improve
3

---

Published in Transactions on Machine Learning Research (Feb/2024)
the generation of visual text images by training character-aware diffusion model conditioned large character-aware text encoder Liu et al. (2022). However, this work differs from our method in the application, as they concentrate on text-to-image generation, while our work focuses on editing texts within an image. In contrast to their single-cncoder architecture, our approach incorporates dual encoders to provide character and style encoding for the target lext, respectively. Some other concurrent works, such as ControlNet [hhang & Agrawala (203), DiffUE Chen et al. (2023a), and TextDiffuser Chen et al. (2023b), have demonstrated exceptional performance in image editing by providing the model with references such canny edge images and character segmentation maps. However, such fine- fine-grained text reference is challenging to apply for scene text editing. Moreover, the reference can only be used to control the text content, while the text style is not considered. In contrast, our method only needs natural language instruction to control the generated text and style, enjoying better scalability and flexibility.
3 Methodology
The proposed DIFFSTE aims to improve the pre-trained diffusion models in their scene text editing ability, We specifically use the SD, state-of-the-ar diffusion model, as our backbone, but the proposed methodology is general to other text-to-image diffusion model architectures. The key architectural innovation of DIFFSTE is to augment SD with an additional character-level encoder to form a dual-oncoder structure. We will first elaborate on our model architecture with the dual-encoder design. An illust can be seen Figure 3 Next, we will describe our instruction tuning framework for training our model.
3.1 The Dual-encoder Design
Our goal is to equip pre-trained diffusion model SD with the ability to accurately capture the spelling of the tar- get text while understanding the provided instructions for generating the target text in the desired style and font. To achieve this, we consider dual-encoder architec- ture in DIFFSTE, which consists of an instruction encoder ENCinst and a character encoder ENCehar. The instruction encoder ENCinst is inherited from the text encoder of SD, namely, a CLIP Radford et al. (2021) text encoder. It is expected to encode the desired style information for tar- get text generation with the proper instruction as input. In contrast, the character encoder ENCobar is a new add- module to SD that takes only the target text to be generated tokenized at the character level as input, pro- viding DIFFSTE the direct access to the correct spelling,
Tokesization
Dual analiatttionn layer
gulling
Drosa-Atbentiion
Character
Cross Atcorition
C.3P Test Tokenizer
austertio
"Damm"" gres
Incomndiaaee
Figure 3: Model structure of DIFFSTE with dual-encoder design. The input text instruc- tion and the target text spelling are processed by instruction encoder ENC and character en- coder ENCehar: respectively. Both encodings are attended to the intermediate hidden represen- tation z from the previous layer through cross- attention. The results are averaged to guide im- age generation.
To incorporate the information from both encoders, we modify only the cross-attention module from SD. Specifi- cally, unlike SD, which only computes the cross-attention from the instruction encoder side, construct the cross- attention layer by first separately attending to the em- beddings from the instruction and character levels and then aggregating the attended results using simple average. As shown in Figure 3 both encodings are used to atlend the intermediate hidden representation z (output of the previous layer) in the UNet components during visual denoising. The averaged result then fod into the following UNet layer. By applying this modified cross-attention at every denoising layer, the generated clean image at the end of the diffusion process is expected to be natural, with correct spelling, and consistent with the language instruction.
4

---

Published in Transactions on Machine Learning Research (Feb/2024)
3.2 Instruction Tuning
We adopt an instruction tuning framework to train our model to understand the scene text editing task. Similar to the instruction tuning technique in the NLP arca, where foundation models are taught to follow task instructions to perform the corresponding task, our goal is to teach our dual-encoder DIFFSTE model to accurately map text. instructions to the corresponding visual appearance. To achieve this goal, we first introduce a synthetic dataset using an automatic rendering engine. Each constructed example contains @ a language instruction that describes the desired target text, @ a masked image indicating where scene text editing should be performed, and @ a corresponding ground truth image. We have also included existing real-world scene text editing datasets in our instruction tuning process enable DIFFSTE to generalize wide range of real-world backgrounds and scenarios. In the following, we provide a detailed explanation of how generate the synthetic dataset and how we convert the existing dataset to similar format it can be utilized in our instruction tuning pipeline. Once these datasets are appropriately prepared, DIFFSTE is trained to minimize the MSE loss between the ground truth image and the generated image conditioned on the text instruction and the masked image.
Synthetic dataset creation We begin by randomly selecting few English words or numerical numbers and pairing each of them with color and font style. We then utilize publicly available synthesizing engine, Synthtiger Yim et al. [2021), to render each text using its corresponding specified style. Afterward, we apply rotation and perspoctive transformation to these rendered scene texts Qu et al. (2022) and place them in non-overlapping locations on a background image selected at random from the SynthText [uupta et al. (2016) project. After constructing the image, one random word in the image is masked out to create a masked input, and the corresponding text and style information is utilized to generate task instructions. In particular, consider two settings to construct the instructions, which are style-free and style-specific. As the name suggests, for the former case, the instruction only provides the specific text want the model to generate without specifying the style. In this scenario, we ensure that the style of the surrounding seene texts to the masked one is the same and consistent with the masked ground truth. Here, our model is required to follow the style of other texts in the surrounding area to generate the target text correctly and naturally. For the stylo-specific setting, we consider three different cases, namely generating the scene text with font specification only, color specification only, and both font and color specification. Similarly, in the first two partial style instruction cases, the surrounding texts in the masked image always conform to the missing style, the model could infer the correct style from the background to recover the ground truth image. On the other hand, in the last case, the surrounding texts are not necessarily consistent with the ground truth style of the target text, so the model would have to follow text instructions to generate the correct style.
There are four categories of instructions in total, and we generated these instructions for our synthetic datasct using a simple fixed rule. For the style-free case, our instruction created simply by using the rule Write "TEXT", where TEXT is the ground truth masked word. For the case where the style is specified, we construct the instruction following the rule Write "TEXT" in color: COLOR and font: FONT. Furthermore, for the partial style-missing case, we modify the previous full instruction rule and remove the corresponding missing information. Although our instructions are created following a very simple rule, we will later show that the model tuned with these instructions can generalize well to other free-form language instructions.
Real-world dataset utilization Commonly used real-world scene editing datasets such as ArT Chnng et al. (2019), COCOText Gomez et al. (2017), and TextOCR, Singh et al. (2021) only provide the bounding box locations of seene texts in images without text style information. Therefore, we only generate style-free instructions to pair with real-world images for instruction tuning.
4 Experiments
4.1
Experiment Setup
Datascts As described Section 3.2, we collect 1.3M examples by combining the synthotic dataset (Synthetic) and three real-world datasets (AriChng et al. (2019), cocoTextGomez et al. (2017). and TextOCR [Singh et. al. (2021) for instruction tuning. For thewSynthetic dataset, we randomly pick up
5

---

Published in Transactions on Machine Learning Research (Feb/2024)
100 font families from the google fonts library and 954 XKCD colors" for text rendering. We randomly select 200 images from each dataset for validation and 1000 images for testing. Additionally, we test our method on another real-world dataset, ICDAR13 Karatzas et al. (2015), to assess the model's generalization ability. Different from the training examples described in Section 3.2 the language instruction for cach test example is constructed with a random larget lext, which different from the text in the original image. In this way, we focus on the more realistic text editing task and measure the result quality with human evaluation. All the images are cropped/resized to 256x256 resolution as model inputs. More details can be found in Appendix B
Basclines We compare DIFFSTE with stato-of-the-art GAN-based style-transfer methods and diffusion models described below. The GAN-based methods include SRNET Wu et al. (2019) and MOSTEL Kong et al. 2022a)). The training of SRNET requires "parallel" image pairs with different texts appearing at the same location and background, which is not available for real-world datasets, so we fine-tune the released model of SRNET only on our synthetic dataset. We use the original model for MOSTEL as we empirically find that fine-tuning on our data does not improve its performance. For the diffussion model baselines, we include pre-trained
(SD) and suableediitusioon--inaiin models are further fine-tuned by instruction tuning in the same way as our method, as described in Sec- tion 3.2 The resulting models are termed as SD-FT and SD2-FT. More implementation details could be found in Appendix B
(SD2). These two
Evaluation and metrics Our evaluations are conducted under style-free and style-conditional settings. Stylo-free generation focuses on the correctness and naturalness of the generated texts, with the language instruction only describing the target text content. All iiffusion-based methods are fed with the text in- struction and the masked image. On the other hand, GAN- models SRNET and MOSTEL cannot take text instructions as input, we feed them with the cropped-out text region for editing in the original image and a rendered text image with the target text. Their outputs are filled into the masked image as the final generation. Note that the GAN-based methods enjoy extra ad advantages compared with ours as they have access to the style of the original text. This makes it casier for them to generate natural-looking images by simply following the original text style. In contrast, our method has to infer the style of the target text from other surrounding texts in the masked image for natural generation.
For the stylo-conditional generation, we require all methods to generate the target text following a specified style. In this setting, the text instruction describes the target text with a random color and font style specified. The diffusion-based methods are fed with the style-specified text instruction and the masked image. However, the GAN-based style transfer models, SRNET and MOSTEL, do not support style specification by texts. Thus, we syuthesize a style reference image of a random text using the specified font and color as well as a rendered image with larget text. Both images are fed into GAN-based models as input, and the resulting output is filled into the masked image as the final generation. Note that the style reference images can be different in synthetic and real-world testing sets. For the synthetic testing set, the synthesized reference image contains the same background as the original image. For the real-world testing sets, a pure-colored background is used as we can not synthesize an image with texts in the specified target style on the same background as the original image.
We evaluate the effectiveness of our method from three aspects: ① text correctness, @ image naturalness, and @ style correctness. For @ text correctness, we report the OCR accuracy, which is calculated as the exact match rate between the target text and the recognized text from the generation by a pre-trained recognition model Fang et al. (2021). Human evaluation of the text accuracy is also reported, denoted as Cor. For @ image naturalness, we ask the human evaluator to compare the generated images by a certain baseline and our method DIFFSTE, and the vote rate gap (vote of bascline minus vote of DIFFSTE) is reported as naturalness score Nal. For @ style correctness, under the style-specific evaluation setting, huraa evaluators are presented with a rendered reference image of the target text in the specified style and are asked to vote between a bascline and our method w.r.t. font correctness and color correctness of the generated images. The resulting vote gaps are reported as Font and Color, respectively. All human evaluation results are conducted 50 randomly selected images for each of the five datasets. More details could be found in Appendix A
https://tonts.ggnt www.ttts..ooggec.ooooooo com https://xked. http:////ddccccc/oo/a//aaacaa//a com/color/rgb/
6

---

Published in Transactions on Machine Learning Research (Feb/2024)
AFT
COcOText
TextoCR
ICDAR13
CNSTITUTIONA
ABUSE
THE THEN
DUANO
CUANS
JAPAN:
trin tey
Input
D.C.C. OR
CR
Black Cols
Tokyo Game Gear
LEGAL PRO
BY ORDER:
PLEIKU
MIENS-
"diffrrecce"
"1459992*
"BADMEN"
"VORACIOUS"
"yellow"
"One"
"Polley" "
"Blude"
MOSTEL SINETT
GELD
CINTITTUIONN
MERIDA
Ballnng
THLk
QUANS
JAPAN:
trim Jey
GOVT,
Black Gats
FOWRCIOUS
KONTUA
Tokyo
S. SPoLI @
459992
Lon
Dne Gear
Btunding
LEA
JAPANT
CONSTITUTIONA
ONLD
trin 1 yyy
deterance
RADMEN
yellow
GOVT,
Black futs 1459998
VORACIOUS
KON
Tokyo
Policy
Lo
One Gear
stundos
GELD
CUANSTB TBI
JAPAN'
Gria Myy
CONSTITUTION
SD
TORS
GOVT.
Black futs
BADDAEN
VORTOBUUES
KONTUA
WATLL
Tokyo OhNE Gear
edy @UrL
1e0 4 44445.555 sh-
Lon
sfandios
RLLEDS
JAPANT
CONSTIUUTION
GILD
THES
trin t toy
SD2
DIREFENCE
BADIEN
SOLOIS
GOVT.
Black Gals
VORAUCUUS
KONTUA
Tokyo
DILG co
1107a9
Lak
Onee Gear
sdauddio
LUDDE
JAPANT
SD-FT
GUILD
CONSTITUTIONA
BADDANN
YOURA
THLs
trim Jyy
GOVT,
Black fuls
VORACUUICS
KDN
Tokyo
polyighy 1
141121599333M
Lan
Onne Gear
saudfos
elluuda
JAPAN
SD2-FT
GUILD
CONSTITUTIONA
DIFFEENCE
BADDMENS
YEU DH
TaLe
QUENS
trăm Tty
OR
Black fals 145599529
VORAICU-UN
KON
Tokyo Game
POLICY
Lak
ONE Gear
studllo
LLUDE
Ours
GUILD
CONSTITUTIONA
dffference
GOVT,
Black futs 1459992
BADMEN
VORACIOUS
vellow KONTUA
Tack
18)
Tokyo Game
JAPANT
Pell e standlios
trim 1ty
Lo
One Gear f
ORDER:
Figure Examples of style-free text editing results on four real-world datascts.
4.2 Experiment Results
Style-free generation The quantitative results for style-frec generation are shown in Table I We observe that our method DIFFSTE consistently achieves better text correctness and image naturalness than other baselines over all datasets. For example, DIFFSTE improves average OCR accuracy and human-evaluated text correctness (Cor) by 64.0% and 21.7% compared with the most competitive bascline MOSTEL. For human evaluated naturalness (Nat), DIFFSTE outperforms all other baselines by at least 28% more vote percentage average. Besides, the superiority of our method over other fine-tuned diffuusion-based methods SD-FT and SD2-FT have demonstrated the effectiveness of our newly introduced character encoder.
We emphasize the effectiveness of DIFFSTE as follows. First, DIPFSTE generates texts strictly following the given instruction, which consistent with the superior performance of DIFFSTE in terms of text correctness in quantitative evaluation. In comparison, other diffusion- based baselines are prone to generating wrong spellings, while the GAN-based methods tend to generate blurry and ambiguous texts and thus get low text correctness. For example, in the ArT dataset with the target text "difference", with our method perfectly generating the correct text, SD-FT generates "DIFEFRERE" while the generation of MOSTEL is hard to read due to the blur. Second, our method generates more natural images than other baselines. For example, in the TextDCR dataset with "One" target text, the "One" generated by DIFFSTE conforms to the font and color of other texts in the context. By contrast, GAN-based methods SRNET and MOSTEL also copy the text style from context and generate red "One", but the edited region is incoherent with the background with clear boundary.
We present the qualitative results Figuregant Appendix D.1
7

---

Published in Transactions on Machine Learning Research (Feb/2024)
Style-free generation
Method
Synthecic
ArT
COCOText
TextOCR
ICDAR13
Averago
OCR1
Cort
Nott
OCRT
Cort
Natt
OCR1
Cort
Naft
OCR
Curf
Nost
OCRT
Cort
Not
OCRT
Cort
Notf
SRNET
50.74
44
-28
30.45
-16
26.87
84
-68
31.14
-40
30.13
46
-52
33.87
51.6
-40.5
MOSTRI
71.51
50
-12
60.90
70
-20
24.11
84
-40
46.11
56
-52
52.23
00
-20
50.99
70.0
-28.5
SD
3.12
6
-40
6.39
12.46
10
-16
8.22
-40
4.56
6
-44
6.75
6-4
-28.0
SD2
4.61
8
-32
7.28
-20
19.34
14
-36
11.88
-30
9.13
-48
10.45
12.0
-33.2
SD-FT
29.51
34
-64
32.08
34
-40
51.02
60
48.91
44
-48
26.29
30
-24
37.48
40.4
-36.0
SD2-FT
37.50
46
46.17
56.13
50
60.82
-24
43.32
48.79
48.4
-35.2
OURS
83.79
86
83.08
86
84.05
88
851,48
82
81.79
84
83.04
85.2
Style-condit onnal ganaration
SINNET
67.58
54
71.51
-48
52.91
84
62.29
-24
64.04
65.2
-30.4
MOSTRI
72.54
74
-16
76.21
80
-44
65.72
80
-40
67:89
72
-20
68.23
776
8
70.12
75.2
-22.4
SD
2.42
4
-28
4.13
16
9.69
8
-12
7.91
-16
3.81
2
-20
5.50
4.8
-12.0
SD2
4.82
6
7.64
-8
18.13
12
-12
12.52
9.28
10.46
8.0
-5.6
SD-FT
21.52
16
-52
26,22
18
-20
47.16
42
-28
43.38
40
-36
22.77 77
18
-34
32.61
26.8
-34.D
SD2-FT
28.56
32
34.87
-36
55.84
60
54.79
36.94
-36
42.20
42.8
-43.2
Ouns
72.39
82
74,82
80
73.38
84
73.67
79.26
84
74.70
80.4
Table 1: Quantitative ovaluation on five datasets with stylc-frec generation (top) and style-conditional generalion (bottom). Text. correctness rale evaluated by both aulomatic OCR model (OCR) and human labelling (Cor). The image naturalness (Nat) is evaluated by human, with the difference of vote percentage between each bascline and our method reported. A positive Nat value indicates the corresponding bascline is belter than ours. All the numbers are in percentage.
Style-conditional generation We present the quantitative results of style-conditional generation in Ta- bles and Our key observations are as follows. First, similar to style-frce setting, our method achieves the best average text correctness and image naturalness, with 6.5% and 6.9% improvement in OCR accuracy and Cor, and 5.6% more votes in Nat. Second, DIFESTE achieves the best font and color correctness compared with other diffusion baselines over all datasets. Different from the diffusion models that take text instructions for specifying the style, the C.AN-baeed methods like MOSTEL and SRNET take reference image of rendered texts with the specified font and color. This unfair advantage explains why the GAN-based methods achieve better average color correctness. However, our method DIFFSTE is still better font correctness over four of the five datasets with 13.2% more vote percentage in average.
We present qualitative results in Figure 5 and Appendix D.2 It is shown that our model consistently gen- crates correct texts that conform to specified text styles. Although GAN-based methods also gencrate texts following the specified color, they suffer from poor font correctness. An example could be seen for "survey" text in ArT dataset. Besides, GAN-based methods cannot capture the orientation of surrounding texts.
With oblique surrounding texts in TextOCR dalaset over "better" larget text, SRNET and MOSTEL. still tend to generate horizontal texts, resulting in un- naturalness. On the other hand, pre- trained diffusion models SD and SD2 completely ignore the styles specified in the instruction, and the problem is miti- gated in SD-FT and SD2-FT after fine- tuning, demonstrating that our instruc- tion tuning framework could improve the style controllability. Despite the im- provement in style correctness, the fine- tuned diffusion models still cannot fully capture the specified style compared to our method DIFFSTE and struggle with incorrect lext spelling, demonstrating the effectiveness of our dual-encoder model architec- ture. For example, SD2-FT generates "JPG" in Art dataset when the target text is lowercase "jpg". We present additional experiment results with an ablation study of the proposed character encoder, instruction tuming, and data mixture in Appendix C and quantitative comparison with other diffusion models such DIPPUSR and SD-XT, in Appendix
Method
Syathetic Foutt Colort
ArT Foat Colort
COCTTaxt Fout Colort
TextDCR Foalt Colort
ICDAR13
Average Foalt Colort
Foult Colort
SRNET MOSTEL
-8
0
-16
-10
8
4
28
-36
20
113.2
12.4
-18
-24
-23.2
SD
-86
-72
-82
-72
-76
-54
-82
-54
-80
-76
-81.2
-65.6
SD2
-86
-70
-84
-68
-62
-48
-76
-64
-70
-68
-75.6
-63.6
SD-FT
-74
-36
-76
-44
-34
-70
-38
-38
-00.2
-38.0
SD2-FT -72
-16
-78
-28
-74
-34
-68
-24
-32
-70.4
-26.8
Figure 6: Human evaluation results of font and color correctness for stylo-conditional gencration. We report the difference of votes in percentage between each baseline and our method. A positive value indicates the method is better than ours.
8

---

Published in Transactions on Machine Learning Research (Feb/2024)
ArT
COCOText
TextOCR
ICDAR13
ALL P
If you
you be off ip
VESTE
INPUT
UN
legsage. Onee your own..
BAR
0
NG PL
Lei HNES
NATURAL
dinero a
TOTYPE &
RENDERRD
jpg
survey
boundaries
amazing
kenneth
better
retailer
CZECH
ALL P
you Bouundanies di with the light- by Haattssnn not bensigs, Onee your
SRNETT
Certro
WESTE UN
survey
retailer
BAR
POOL
NG ipc
Lo
kenneth
HUNA AURCYO
TOTYPE
ALL PO
MOSTEL
Cenio
your off up with the light- Hartesans-
WESTE UN
survey
retailer
POOL
NG BeG
Lo better mMsS
N Kenneth NATURAL
mRIr fo
TOTYPE
ALL PO
Centie
upp Hartesans-
you Borekelisuz
WESTE
Fēulexte
SD
SUIVR
UN
nst begsage. Once
BAR
NG JugE PL
your owe..
APZINI
Beteine mus dinero
KCH Z
KIMNT EN NATURAL
TOTYPE
ALL PO
you Burledion
WESTE
SD2
SURVNRT
with the light- Hartmmnn-- begsngs, Onee your
Ralley
BAR POOL
UN
mole
NG PGE
AAAZING
Ia beteroot nuis
ZCCEHI
mnkar dinero
TOTYPP
ALL PO
SD-FT
Cenbral
you off to up with the light-
WESTE
Serven verain
retlielerr
UN
Onse
POOL ARZZING
HHHHH
NG JPG
La betteer toiis
KENNETH NATURAL
milor Alieero a Ro
TOTYPE
SID2-FT FT
ALL PO
Cerdne
you borsiades upp Harttsanss nt begsige. Onee your owr..
WESTE
Retelmler
Survery
UN
BAR
NG JPG PL
AMZZING
bebttttezzuss
L. L. D L TOTYPE &
KENNT
diiero
ALL PO
loonderiig ip with the light- Hlrtteant- hagsnge. Onee
Conina
WEST UN
Ours
survey
retailer
BAR POOL
NG jpg
your
anazLnng
La better sunds
ZECH TOTYPE
waler maber dinerb
Figure 5: Examples of style-conditional text editing results. The RENDERED row shows the larget text and
desired style.
4.3 Zero-shot Style Combination
In this section, we show that our method can create new text styles by composing basic font attributes in a zero-shot man- ner benefitting from instruction tuning, We consider Iwo settings: font extrapo- lation and font interpolation.
BREAK FOR
BREAK GLASS FOR
BREAK GLASS
BREAK GLASS FOR
BREAK GLASS FOR
Input
Lancelof
Lencelot+-italic
Lanosiot+boll
Lanceiot both
intelligent
intelligent
Intelligent
Intelligent
Intelligent
Unioersitg
Unloersits
University
University
Font extrapolation In the font ex- trapolation experiment, model to generate texts in a certain font style but with an unseen format such as italic, bold, or both of them. None of these formats were seen during training for the given font. We append format keyword ilatic, bold to those fonts with
Input
NovaOvol
NosoOval++Balle
NocaOval+
NovaOval+botth
require the
Examples of font extrapolation with our method DIFF- STE. The last three columns are generated with the text instruc- tion template Write "TEXT" in font FONT-FORMAT, where the FORMAT is Bold, Halic and Bold Ialic,, respectively.
Figure 7:
9

---

Published in Transactions on Machine Learning Research (Feb/2024)
such format in training data and give the model the extrapolated font name. As shown in Figure 7 the generated texts in the image appear inclined or bolded when append the format keyword, showing that our method can successfully generalize to unseen font variations while maintaining correctness and naturalness.
Font interpolation In the font interpolation experiment, require the model to generate texts with mix of two certain font styles, which does not correspond to particular font style in the real world and was not seen during training. We feed the model with the following instruction template: Write "TEXT" in font FoNTl and FONT2, where the FONTl and FoNT2 are the names of two fonts seen in model training. The qualitative results can be seen in Figure 2 where a new font generated by mixing MoonDance and PressStart2P. More results are in Appendix D.3
4.4 Editing with Natural Language Instruction
Although we only use static text instruction templates during instruction tuning, we show that our method still shows good scene text editing ability with natu- ral language instructions. The qualitative results can be seen in Figure 8 where feed the model with in- structions such as "The word "HOTEI." in the color of sky" instcad of the fixed templated instruction scen in training. We show that the model has learned the mapping from instructions to visual text appearances in the generated image following the specified text style. This demonstrates that our instruction tuning frame- work brings the model the gexeralization ability seen text instructions, which is consistent with previ- ous findings in NLP area Sanh et al. (2021): Wei et al. (2021). The compatibility with natural language in- structions makes our model casier to use for non-expert users and enables our model to work in vast down- stream applications like chat robots. More results could be seen in Appendix D.4
Fancy
Fancy
Fancy
KPORT.
KPORT.
KPORT.
KPORT.
STQUE
STQQUE HOTEL
STQQUE
ATIQQE
HOTEL
HOTEL
Input
grasa
cherry
sky
Figure 8: Control text color using natural language instructions with our method DIFFSTE. The color is described with words "grass colored". "add cherry color to and "in the color of sky" for the right three columns, respectively.
5 Conclusion
In this paper, we present. novel method DIFFSTE to improve pre-trained diffusion models for text editing with dual encoder design that overcomes the limitations of existing diffusion models in terms of text correctness and style control. Our proposed method consists of an instruction encoder for better style control and a character encoder for improved text legibility. Empirical evaluations on five datasets demonstrate the superior performance of DIFFSTE in terms of text correctness, image naturalness, and style controllability. DIFFSTE also shows zero-shot generalization ability to unseen text styles and natural language instructions.
6 Broader Impacts
In this paper, propose novel method DIFFSTE to adapt pre-trained diffusion models for scene text editing. Our proposed method consists of an instruction encoder for better style control and character encoder for improved text legibility. DIFFSTE improves text correctness for the pro-trained diffusion model and further allows users to control the generated text styles. However, we also admit that our method can be used to forge signatures or spread misinformation, which might have a significant impact on society. In practice, our method should be appropriately used with careful checks on potential risks. For example, we can adopt recently proposed diffusion-watermann 2023) to identify whether certain text. images are generated by our model to identify forged signatures.
methods Zhao al. (2023); Liu al. (2023); Wen al.
10

---

Published in Transactions on Machine Learning Research (Feb/2024)
7 Acknowledgement
The work of Jiabao Ji, Bairu Hou, and Shiyu Chang was partially supported by the Adobe Research Award and the Cisco Research Award.
References
Gantugs Atarsaikhan, Brian Kenji Iwana, Atsushi Narusawa, Keiji Yanai, and Seiichi Uchida. Neural font style transfer. In 2017 14th IAPR international conference on document analysis and recognattio (ICDAR), volume 5, pp. 51-56. IEEE, 2017.
Samaneh Azadi, Matthew Fisher, Vladimir G. Kim, Zhaowen Wang, Eli Shechtman, and Trevor Darrell. Multi- content gan for few-shot font style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recogrition (CVPR), June 2018.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint or Xix::2211.01324 2022.
Manuel Brack, Felix Friedrich, Dominik Hintersdorf, Lukas Struppek, Patrick Schramowski, and Kristian Kersting, Sega: Instructing diffusion using semantic dimensions. arXie preprint arXiv:2301. 122224771.. 2023.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions.
arXi preprint ar 2rXiw:2211. 1.09800, 2022.
Jacky Cao, Kit- Kit-Yung Lam, Lik- Hang Lee, Xiaoli Liu, Pan Hui, and Xiang Su. Mobile augmented reality: User interfaces, frameworks, and intelligence. ACM Computing Surveys, 55(9):1 36, 2023.
Haoxing Chen, Zhuoer Xu, Zhangxuan Gu, Jun Lan, Xing Zheng, Yaohui Lá, Changhua Meng, Huijia Zhu, and Weiqiang Wang. Diffute: Universal text editing diffusion model. arXiv preprint arXiv: 2305. 10825, 2023a.
Jifan Chen, Yuhao Zhang, Lan Liu, Rui Dong, Xinchi Chen, Patrick Ng, William Yang Wang, and Zhiheng Huang. Improving trossttakk generalization unified table-to-text models with compositional task configuratioss. arXiv preprint Xrx:2222 08780, 2022.
Jingye Chen, Yupan Huang, Tengchao Lx. Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: Diffusion models as text painters. arXin preprint Nix:2330 10855, 2023b.
Chee Kheng Chnn, Yuliang Liu, Yipeng Sun, Chun Chet Ng. Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al. ledar2019 robust reading challenge on arbitrary-shaped text-rre-art. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1571 1576. IEEE, 2019.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Ti, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et Scaling instruction-fietuue language models. ArXio, abs/2211 2b1//2210.1146, 2022.
Jun Du, Qiang Huo, Lei Sun, and Jian Sun. Snap and translate using windows phone. In 2011 International Conference on Document Analysis and Recognition, pp. 809 813. IEEE, 2011.
Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang. Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recoguition. In Proceedings of the IEEE/CVE Confer- ence Computer Vision and Pattern Recomnition, pp. 7098-7107, 2021.
Victor Fragoso, Steffen Gauglitz, Shane Zamora, Jim Kleban, and Matthew Turk. Translatar: A mobile augmented reality translator. Tn 2011 IEFE workshop on applications of computer vision (WACV), pp. 497-502. IFF.., 2011.
Raul Gomez, Baoguang Shi, Lluis Gomez, Lukas Numann, Andreas Veit, Jiri Matas, Serge Belongie, and Dimosthenis Karatzas. ledar2017 robust reading challenge coco-text. In 2017 14th IAPR International Conference Document Analysis and Recognition (ICDAR), volume 1, pp. 1435-1443. IEEE, 2017.
Yuxian Gu, Pei Ke, Xiaoyan Zhu, and Minlie Huang. Learning instructions with unlabeled data for zero-shol cross- task generalization. Xixiv preprint Xtm:2210.0011 2022.
11

---

Published in Transactions on Machine Learning Research (Feb/2024)
Ankush Gupla, Andrea Vedaldi, and Andrew Zisserman. Synthetic data for text localisation in natural images. In IEEE Conference on Computer Vision and Pattern Recogrition, 2016.
Prakhar Cupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey P Bigham. Improving zero and few-shot generalization in dialogue through instruction tuning. arXiv preprint arXiv:220. 12673, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neurol Information Processing Systems, 33:6840 6851, 2020.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning langnage models with (almost) no human labor. arXw preprint arXim:222 12.::222200969,, 2022.
Qirui Huang, Bin Fu, Yu Qiao, et al. Gentext: Unsupervised artistic lext generation via decoupled font and texture manipulation. Xiv preprinl orXiv:2007.0960 09644, 2022.
Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning, arXiv preprint ar arXix:2002.00 002202 2023.
Levent Karacan, Zeynep Akata, Aykut Erdem, and Erkut Erdem. Learning to generate images of outdoor scenes from attributes and semantic layouts. arXiv preprint ar arXis:1112.00215 2016.
Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Twa- mura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. lodar 2015 competition robust reading, In 2015 13th international conference on document analysis and recognition (ICDAR), pp. 1156 1160. IEEF, 2015.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognátion (CVPR), June 2019.
Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zlu, Shenggao Zhu, Nicholas Yuan, and Lianwen Jin. Look closer to supervise better: One-shot font generation via component- based discriminator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1M882-1491,, 2022a.
Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, and Lianwen Jin. Look closer to supervise better: One-shot font generation via component- based discriminator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19482-13491, 2022b.
Praveen Krishnan, Rama Kovvuri, Guan Pang, Boris Vassilev, and Tal Hassner. Textstylebrush: Transfer of text aesthetics from single example. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(7):9122- 9134, 2023. doi: R..1109/TPAMI.CM 202.33333....
Junyeop Lee, Yoonsik Kim, Seonghyeon Kim, Moonbin Yim, Seung Shin, Gayoung Lee, and Sungrae Park. Rewritenel: Reliable scene text editing with implicit decomposition text contents and styles. Xix preprial ar 2111:21107.1041, 2021.
Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Guiding text-to-image diffusion model towards grounded generation. arXiv preprint arXiv:2301 05221, 2023.
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering. arXiv preprint 221:2221 10562, 2022.
Yugeng Liu, Zheng Li, Michael Backes, Yun Shen, and Yang Zhang, Watermarking diffusion model. arXiv preprint arXiv: 12502, 2023.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-Lask generalization via natural lan- guage crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470-3453 3487, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 18.18653/ 10////1/2222/ acl- song 244. URL htttps://aclatthollls
acl-onoc. 244
Niklas Muennighoff, Thomas Wang, Lintang Sulawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xoe Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. aixiv preprint ar Xim:22 111. 1.0 1.01786, 2022.
12

---

Published in Transactions on Machine Learning Research (Feb/2024)
Alex Nichol, Prafulla Dhariwal, Adilya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arxi preprint 21. 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Song Park, Sanghyuk Chun, Junbum Cha, Bado Lee, and Hyunjung Shim. Multiple heads are better than one: Few- shot font generation with multiple localized experts. In International Conferencee on Computer Vision (ICCV), 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint or 2307.01952, 2023.
Xugong Qin, Yu Zhou, Youhui Guo, Dayan Wu, Zhihong Tian, Ning Jiang, Hongbin Wang, and Weiping Wang. Mask is all you need: Rethinking mask r-cmn for dense and arbilrary-shaped scene text detection. In Proceedings of the 29th ACM International Conference on Mullimediaa, pp. 414-414444 414-423,4 2021.
Yadong Qu, Qingfeng Tan, Hongtao Xie, Jianjun Xu, Yuxin Wang, and Yongdong Zhang. Exploring stroke-level modifications for scene text editing. arXiv preprint or Xit:22122001982. 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning pp. 8748 8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprind erXw:2204.06125 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognation, pp. 10684-10695, 2022.
Prasun Roy, Saumik Bhattacharya, Subhankar Ghosh, and Umapada Pal. Stefann: scene text editor using font adap- tive neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 13228 13237, 2020.
Dohoon Ryu and Jong Chul Ye. Pyramidal denoising diffusion probabilistic models. Xi preprint Xi:::22008.01 0188. 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXio preprint arXi::205. 11400. 2022.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint Xrxi:22110. 08207, 2021.
Yangming Shi, Haisong Ding, Kai Chen, and Qiang Huo. Aprnet: Attention Aateetion-based pixel-wise rendering network for reoto-realistic text image generation. erXiv preprint or orXiv:2203.07700 2022.
Wataru Shimoda, Daichi Haraguchi, Seiichi Uchida, and Kota Yamaguchi. De-rendering stylized texts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1076 1085, 2021.
Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojeiech Galuba, and Tal Hassner. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. Tn Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR), pp. 8802-8812, June 2021.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.
arXiv preprint
arXiv:2010.00 08502, 2020.
Tonghua Su, Fuxiang Yang, Xiang Zhou, Donglin Di, Zhongjie Wang, and Songze Li. Scene style text editing, ariio preprint arXio: 2304. 10097, 2023.
13

---

Published in Transactions on Machine Learning Research (Feb/2024)
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Ha- jishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprinl arXiv: 10560, 2022.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint or arxiv:2109. 01652, 2021.
Yuxin Wen, John Kirchenbauer, Jonas Geiping, and Tom Goldstein. Tree-ring watermarks: Fingerprints for diffusion images that are invisible and robust. arViv preprint arXi:: 2305.20030, 2023.
Chen Henry Wu and Fernando De la Torre Unifying diffusion models' latent space, with applications to cyclediffusion and guidance. arXiv preprint or orX1:2210000533 2022.
Liang Wu, Chengquan Zhang, Jiaming Liu, Junyu Han, Jingtuo Lin, Errui Ding, and Xiang Bai. Editing text in the wild. In Proceedings of the 27th ACM international conference on multimedia, pp. 1500-1508, 2019.
Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang, and Shiyu Chang, Uncovering the disentanglement capability in text-to-image diffusion models. arXiv preprint Xio:2221 086888, 2022.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object impainting with diffussio model. arXie prepriul arXiv:2212.0500 2022.
Zhiyang Xu, Ying Shen, and Lifu Huang, Multinstruct: Improving multi- modal zero-shot learning via instruction tuning. Xrviv preprint Xiv:22212 10773, 2022.
Qiangpeng Yang, Jun Huang, and Wei Lin. Swaptext: Image based texts transfer scenes. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 14700- 14709, 2020.
Moonbin Yim, Yoonsik Kim, Han-Cheol Cho, and Sungrae Park. Syuthtiger: Syuthetic text image generator towards better text recognition models. In International Conference on Document Analysis and Recognition, pp. 109-124. Springer, 2021.
Fangneng Zhan, Shijian Lu, and Chuhui Xue. Verisimilar image synthesis for accurate detection and recognition of texts in scenes. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 2018
Fangneng Zhan, Hongyuan Zhu, and Shijian Lu. Spatial fusion gan for image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3653-3662, 2019.
Lvmin Zhang and Mancesh Agrawala. Adding conditional control to text-to to-image diffusion models, 2023.
Yexun Zhang, Ya Zhang, and Wenbin Cai. Separating style and content for generalized style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai-Man Cheung, and Min Lin. A recipe for watermarking diffusson models. aix preprint arXiv: 1913,, 2023.
Wanjun Zhong, Yifan Gao, Ning Ding, Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan. Improving task generalization via unified schema prompt. arXiv preprint Xrxi:22008 08229, 2022.
Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Birg-tirk,,trick, and Graham Neubig. Prompt consistency for zero-shot task generalization. ArXio, abs/2205.000 2022.
14

---

Published in Transactions on Machine Learning Research (Feb/2024)
A Human Evaluation Details
In this section, we provide a summary of the human evaluations conducted for both style-free and style- conditional generation settings. As outlined in Section 3 we perform human evaluations on Amazon Mtur for both style-free and style-conditional generation settings. In total, we design six human evaluation tasks to assess the performance of our method from three aspects: the accuracy of the generated text in the image; @ the naturalness of the generated image; the correctness of text style in terms of font and color.
Human evaluation for style-free generation To evaluate style-free image generation, we assess the generated images from two aspects: @ Cor, text correctness, where evaluators check whether the generated image matches the exact target text. The user interface is shown in Figure g
@ Nol, image naturalness, where evaluators select the more natural and visually coherent image between those generated by our method and another baseline method. In other words, the chosen image should look more like natural image, and the completed missing part should be visually coherent with the surrounding region. The user interface is shown in Figure 10
Human evaluation for style-conditional generation To evaluate style-conditional generation, we as- the generated images from three aspects: @ Cor, text correctness, which the same as the style-free generation setting. @ Not, image naturalness, where evaluators select more natural image between those generated by our method and other baseline methods. We note that in the style-conditional generation setting, the specified color in instructions may not be coherent with the surrounding region. Therefore, we exclude the coherence critcrion for image naturalness in this evaluation. The user interface for depicted in Figure 11 @ Font and Color, style correctness, where evaluators compare images generated by our met and other baseline methods and select better one based on how similar the generated texts are given reference image in terms of font color, respectively. The user interfaces shown in Figure 12 and
Figure 13
Instructions:
Please read the instructions carefully. Failure follow the instructions will lead to rejection of your results.
this task, you will see an image which certain area is highlighted with a red box. Your task to determine whether the text's spelling in this area matches the target text shown above the image. It important that the spelling need to exactly the same as the target text. Specifically, the following cases are considered incorrect:
Additional characters, e.g. the generated text is 'apples" but the target text is 'apple'
Missing characters, e.g. the generated text is 'appe' but the target text is 'apple'.
Mismatched capitalization, e.g. the generated text is 'Apple' but the target text is 'apple'
We provide an example help you understand the criteria.
hosting
PA
this example, you should choose Yes. It is important that we care about the correctness of text spelling within the highlighted area. Although the text in this region is blurry, it has the correct target text 'hosting:
Figure 9: Instructions of human evaluation on text correctness.
hhttps://www.cc 9https//www.mturk.com
15

---

Published in Transactions on Machine Learning Research (Feb/2024)
Instructions:
Please read the instructions carefully. Failure to follow the instructions will lead to the rejection of your results. In this task, you will be asked judge and compare the quality of two Al-edited images. Specifically, you will first see one source image with some parts missing. Then, you will see two candidate images, where the missing part completed by Al algorithms. Your task to determine which of the two candidate images is more visually natural and coherent. It is important that we only care about the visual naturalness and coherence of the completed missing part regardless of anything else, such as the correctness of the text spelling.
There are two criteria for this task:
First, the edited image should look like a natural image. should not contain a lot artifacts, distortion, or non-commonsensical scenes. The completed part should not be blurry and there should be no clear boundary between this completed part and the surrounding region.
Second, the completed missing part should be visually coherent with the surrounding region. If there is any other lext in the surroundings, the text in the completed region should match the style of these texts in terms shape and color.
provide an example help you understand the criteria.
Source
Candidate
Candidate 2
JAPAN'
JAPANT
JAPANT
Tokyo Game Gear g
Tokyo Game
Tokyo Game
One Gear
One Gear
In this example, you should choose Candidate the better one. is important that the completed text in the missing region matches the style of surrounding texts. Candidate 2, however, has a black text, which does align well with other red surrounding texts, which does not meet our criteria on coherence.
Figure 10: Instructions of human evaluation based image naturalness and coherence.
16

---

Published in Transactions on Machine Learning Research (Feb/2024)
Instructions:
Please read the instructions carefully. Failure to follow the instructions will lead to rejection of your results. this task, you will be asked to judge and compare the quality two AT-edited images. Specifically, you will first where the missing part is completed by two Al algorithms. Your task is to determine which the two candidate images more visually natural. It is important that only care about the visual naturalness of the completed missing part regardless of anything else, such as the correctuess of the text spelling. criteria for this task:
source image with
parts missing. Then, you will see candidate images,
There
The edited image should look like natural image. It should not contain lot of artifacts, distortion, or non-commonnennsice. scenes. The completed part should not be blurry and there should be no clear boundary between this completed part and the surrounding region.
We provide example help you understand the criteria.
Source
Candidate
Candidate 2
KEI
KEIO
hook
hook
In this example, you should choose Candidate 1 as the better one. It is important that the completed part the image is not blurry and there is no clear boundary between completed part and surrounding image. Candiate 2, however, has a clear boundary between the completed part and the surrounding region, which does not meet our creteria on naturalness.
Figure 11: Instructions of human evaluation on image naturalness.
Instructions:
Please read the instructions carefully. Failure to follow instructions will lead to rejection of your results. In this task, you will be asked to judge and compare the quality of two Al-edited images. Specifically, you will first see a reference image with a certain text that specifies the desired font style we want the Al model generate. Then, you will with a red box. Your task is to determine which of the two candidate images appears more similar to the reference image in terms of font style. More specifically, font style refers the shape of characters and whether the text is bold or italic. It is important to note that we only care about the font style of the text, regardless of anything else, such as the correctness of the text spelling or the color of the text.
candidate images, both of which have some parts highlighted
There is "hard-to-tell" option available if you cannot decide which is better, but please try your best to avoid using this option.
We provide example to help you understand the criteria.
Reference Candidate 1
Candidate 2
JAPAN'
JAPAN':
One
Tokyo Game Onee Geor
Tokyo Game
One Gear ç
In this example, you should choose Candidate 1 as the better one. It is important that all we care about whether the font style of text is simiar to the reference image. You can see that the shape of characters in Candidate2 is not the same as the reference image, e.g. the chracter O is not similar. Therefore, you should choose Candidatel as the better one despite that the text spelling Candidate 1 not the same the reference image.
Figure 12: Instructions of human evaluation on font correctness.
17

---

Published in Transactions on Machine Learning Research (Feb/2024)
Instructions:
Please read the instructions carefully. Failure to follow the instructions will lead to rejection of your results. In this lask, you will be asked to judge and compare the quality of two Al-edited images. Specifically, you will first reference image with certain text that specifies the color of the lext that want the AI model to generate. Then, you will see two candidate images, both of which have some parts highlighted with red box. Your task is determine which of the two candidate images appears the reference image in terms of the text color. Tt is important to note that we only care about the color of the text, regardless of the spelling correctness or font type. There 'hard-to-tell' option for you to choose if you cannot decide which is better, but please try your best to avoid using this option. We provide an example help you understand the criteria.
similar to
Reference
Candidate
Candidate 2
JAPAN'
JAPANT
One
Tokyo Gome
Tokyo Game
One Gear
Onee Gear
In this example, you should choose Candidate 2 the better Tt. is important that all we about is whether the color of text is simiar to the reference image. Candidate 2 is better since the text in the highlighted area looks more purple, which similar to the reference image, despite that the text content is not the same.
Figure 13: Instructions human evaluation on color correctness.
18

---

Published in Transactions on Machine Learning Research (Feb/2024)
B Implementation Details
B.1 Dataset Details
As mentioned in Section we construct the synthetic dataset using a publicly available engine Syn- thtiger Yim et al. 20211) We randomly selected 100 font families in the google fonts library and 954 named XKCD colors to render the texts. During the creation process, we first randomly sample several English words or numerical numbers (1-5) and render them using a random color and font style. Then we apply the same rotation and perspective Lransformation to these rendered scene lexts and place them on 256 x 256 background image non-overlappingly. In training time, random word in the image is masked out and the model is trained to restore the full image including the text in the masked part. Figure [4] shows examples of the created synthetic data and the masked input for model training. Besides the synthetic data, we also include real-world OCR datasets in the training set, where the word in image is masked out for training. total, we collect 1.3M training examples with 600K synthetic examples and 700K real-world examples.
Syathetic example
Augmentation example
rodoen
Synthitc
RTS
Lag
voolle
1ZXDN
N
Masked
928
ucolics
RED
Figure 14: Example images of synthetic data(first row) and the masked input(second row) we use to fine-tune the model.
B.2 Model Details
GAN-based methods We implement the GAN-based method, SRNET and MOSTEL using their publicly released code. We created additional synthetic examples where different texts of the same style are put at the same location the background image to construct paired images for style-transfer training, which is only used in fine-tuning SRNET. Real-world datasets are not included in training SRNET since it requires paired images with same background but different texts, which is not given in real-world datasets. For SRNET, we follow the iyperparameters reported in their paper to fine-tune the released model. For MOSTEL, fine-tuning the released model on the created synthetic data does not improve its performance on real-world datasets. Therefore the released checkpoint. is used for comparison.
basee methods All diffusion- based methods are built upon diffusers For the pre-trained dif- fusion models, we use saable-diifusiion-npaannnggn ((D) and iibble-ddiiuusion--inpaiinnn (Snn).nnnnnnnn)))....))..)..s.") We further fine-tune SD and SD2 using our instruction tuning framework, where the resulting models are termed as SD-FT and SD2-FT. For our method DIPFSTE, we use SD as the backbond and randomly initialize the characker-embeedingg layer and corresponding cross-attention weights. The dimension of character embed- ding is 32, and the number of cross-attention heads is 8. We use the same hyperparameters to train SD-FT, SD2-FT, and our method DIFFSTE. The batch size is set to 256. We use the AdamW optimizer with fixed learning rate 5c - 5 to train the model for 15 epochs. In total, the training has 80k steps, which requires approximately two days of training time using eight Nvidia-V100 gpus.
Data augmentation In our initial experiment, we observed strong correlation between the mask shape and the visual layout of text in the generated image. Specifically, the model tends to duplicate characters to fill the entire masked region even if there are only a few characters to generate. To mitigate this bias, we augment the masks to be larger than the text in the original image. Specifically, we extend the mask region
htttps:/hugginggen http::///pcccaaacccccco/cctttttt
co//ocs//iifusers
"stabilityal/ stable-diifnsion 2-mpaaanin
7 SDD is not released by the time of our implementation.
19

---

Published in Transactions on Machine Learning Research (Feb/2024)
along the direction of the masked word while keeping sure that other texts included in the mask. Examples can be seen on the right of Figure 14
C Ablation Study
Our method DIFFSTE improves upon the pre-trained SD model from two aspects: C the dual-encoder design with the newly added character encoder; 2 the instruction tuning framework that helps model understand the task of scene text editing. In this section, we study the effectiveness of these two designs. To obtain a deeper understanding of the effectiveness of synthetic and real data, we also conduct an ablation study on the data mixture ratio.
Character encoder We evaluate the effectiveness of the character encoder by comparing the pre-traiend backbone model SD, instruction-tuned SD without character encoder, and our method DIFFSTE, the instruction-tuned SD with character encoder. Table 2 shows the quantitative evaluation results on five datasets, where these three models are termed as SD, +INST, and +INST+ CHaR respectively. Although instruction tuning (+INST) improves the ability of the pre-trained model SD to generate correct texts with an average of 30% increase OCR accuracy, the text correctness is way lower than the +NNST+ CHAR, which is additionally trained with the character encoder. This demonstrates that the dual-encoder design could effectively improve the model's ability to capture the spelling of target text.
Style-free generation
Syathetic
ArT
COC0Text
TextOCR
ICDAR13
Average
Method
OCRt
Cort
OCR1
Cort
OCR1
Cart
OCRT
Cort
OCE1
Cort
OCR1
Cort
SD)
3.12
5.39
6
12.46
10
8.22
4.56
6.75
6.4
+ INST
29.51
32.08
34
37.48
INST+CHHAR
83.79
86
83.08
86
84.05
88
85.18
82
81.79
84
83.64
85.2
Style-conee Style--conaational generation
SD
3.12
6
5.39
6
12.
10
8.22
4
4.56
6
6.75
6.4
+INST
23.52
16
26.22
18
17.16
12
13.38
22.77
32
26
CHAR
72.39
82
74.82
80
73.38
84
73.67
79.26
74.70
80.4
Table 2: Quantitative evaluation on five datasets with style-free generation (top) and style-conditional generation (bottom) of the pre-trained backbone model SD, the fine-tuned SD without character encoder (+INST) and the fine-tuned model with character encoder, i.e. are percentage.
All the numbers
Instruction tuning We evaluate the effectiveness of the instruction tuning framework by comparing DIFFSTE by comparing DIFFSTE with three other models that share the same backbone and character encoder but are trained using different instructions. The models evaluated are trained using instructions without font information, color information, and neither of them, respectively. We denote them as -FONT, -COLOR, and -BOTH.
Synthetic Fonit Colort
ArT Fonlt
COCOText Fondh Color
TextDCR Fontt Colort
ICDAR13 Font? Colort
Average Foxit Color
Meshod
Colort
DIFISTE
FOOT
-22
64
-76
-70
68
-70.0
COLOR
54
.66
-72
-84
-88
-8.88
-BOTH
-78
Table 3: Human evaluation results of font and color correctness for cooiisstruction/paatee wawtr/aaaaaaii-sstttioo tuning. The difference of votes in percentage between the baseline method and our method is reported. A positive value indicates the method is better than ours.
To evaluate their ability in understanding style specifications the instructions, we compare the three models with DIFFSTE under the style-conditional generation setting, Similar to Section 4.2 we ask human evaluators make pairwise comparison between images generated by these models and DIFESTE, which is
20

---

Published in Transactions on Machine Learning Research (Feb/2024)
trained on full instructions with font and color information on 50 randomly picked images for each dataset. The quantitative results are shown in Table 3 We observe that removing certain style information in the instructions would lead to a decrease in corresponding style correctness dramatically. For example, without font instructions, -FNNT receives 70.0% fewer votes in terms of font correctness, while a 68. 15.8% average drop is observed for color correctness for -COLORR. Without both font and color, -BOTH, descend 74.8% and 70.8% in terms of font and color correctness, even more than -FONT and -COLOR. This further demonstrates the effectiveness of our instruction tuning framework in teaching the model to align text instruction with visual appearances.
Synthetic v.s. Real data mixture ratio To comprehensively study the mixture ratio between synthetic and real-world data, we train two additional versions of our method. Specifically, we train one model on synthetic data only and another on real-world text images only. We compare these two trained models to the original DIFFSTE in both style-free and style-conditional settings, as described in the main paper. The
OCR results are presented in Table | and the generalion examples can be found in Figure 15 and Figure 16 We highlight the following observations:
First, all three models achieve significantly better text correctness compared to the vanilla stable-diffusion 1/2 model and even the fine- tuned SD1/2. This demonstrates that our dual encoder design effectively helps the base diffu- sion model understand correct word spelling and generate accurate visual texts.
Style-free generation
Model
ArT
COCOText
TextOCR
ICDARI3
Average
DifITTE
83.08
84.05
85.48
81.70
83.64
-REAL
46.34
64.17
61.83
55.47
57.73
-SYN.
52.31
70.31
68.27
62.11
63.25
SD
5.39
12.46
8.22
4.56
7.67
SD2
7.28
19.34
11.88
9.13
11.91
Second, training real synthetic data only hurts the text's correctness. The combined DiffSTE model achieves more than 20% abso- lute percentage improvement.
SD-FT
32.08
51.62
48.91
25.29
39.48
SD2-FT
46.17
56.13
60.82
43.32
51.61
Style-cond generation
Model
ArT
COCOText
TextOCR ICDAR13
Average
Third, training on synthetic images only hurts the image quality. As can be seen in the fourth row of Figure [5 the visual texts gen- erated by the model trained on synthetic data only do not align well with other surround- ing texts. This issue arises from the limited text layouts and styles in the synthetic dataset we use. Therefore, we include real data in DIFFSTE training to provide the variability of background scenes, enrich the text styles, and enable the model to better generalize to test-time real-world text images.
DiNSTE
74.82
73.38
73.67
79.26
75.33
-REAL
32.35
45.12
43.28
34.21
38.74
-SYN.
44.21
58.93
60.42
54.32
54.47
SD
4.13
9.69
7.91
3.81
6.39
SD2
7.64
18.13
12.52
9.28
SD-FT
26.22
47.16
43.38
22.77
34.88
SD2-FT
34.87
55.84
54.79
36.94
45.61
Table 4: OCR correctness on real-world datasets with style- free generation (top) and style-conditional generation (bot- tom) for three versions DifITTE The correctness is eval- uated by the automatic OCR model and all numbers are reported in percentage. -REAL and -SYN. represent model trained real-world text images only and synthetic images only respectively using our method.
Finally, We also want to highlight that syn- thetic data builds the foundation for the edit- ing ability of DIFESTE. We explicitly con- struct the data teach the model to follow editing instructions and infer style from surrounding texts (soe See 3.2 in the main paper), which is the main focus of our paper. As can be seen in the fourth row in Figure 16 the model trained on real data only does not understand the style control instructions, thus the generated texts are always plain text style, either matching the surrounding text or the background similar to style-free generation. The text correctness is also affected as the model never sees style specification instructions in training (failed generation in row 4 column of Figure 16
21

---

Published in Transactions on Machine Learning Research (Feb/2024)
TER THE
Input
Books, In
457
"Bulf"
*Domps"
OATHS"
"LATINER"
DIFSTE
Dampe 7
TRR OATHS
Books, In
Bufr
457 LATINER
REAL ONLY
Damps Tr
TER THE OATHS
Books, In
Buff
457 LATIINER
Dam P. Te
ER THE OATHS.
SYNTH
Books, In
Buff
457 LATIVER
Figure 15: Examples of style-free text editing results with three versions of DIFFSTE. REAL ONLY and SYNTH
ONLY denotes the model trained on real-world text images and synthetic text images only using our method.
APEI
FAMOUS
COS
Input
OF
ANNED.
THE
taffie Factor
RENDERED
LSD
BRITON
THINE
BRITON
ADEN
FAMOUS
DIFSTT
COS
OF LSD ANNED.
BRITON
THE THINE
EN Bostion
aaffe Factorb
SYNTH ONLY REAL ONLY
ADEN
FAMUUS
COS
OF LSD ANNED.
THINE EN Basstios
BRITORNI
afffe Factora
FAMUUS
COS
OF LSD ANNED.
BRLTON
THE THINE
NN B Bastion
aile Factord
Figure 16: Examples of style-condition text editing results with three versions of DIFTSTE. REAL ONLY and
SYNTH ONLY denotes the model trained on real-world text images and synthetic text images only using our method.
22

---

Published in Transactions on Machine Learning Research (Feb/2024)
D Additional Qualitative Results
D.1 Additional Style-free Qualitative Results
We provide additional generated images for editing different texts in the same image by our method DIFFSTE in Figure 17 The first row shows the target lext lo gemerate, and each column shows the different generated images on the target text. DIFFSTE consistently generates correct visual text, and the texts naturally follow the same text style, Lee. font, and color, with other surrounding texts. The orientation of generated texts also aligus with other lexts properly. In addition the normally shaped masks shown in Figure 17 DIFFSTE is able to generate texts within arbitrarily shaped masks. We provide the qualitative results in Section D.5
Iuput
"The" "
"five"
"bozing" "
"wizards""
"jump""
"quickly" "
oad now and oad now an oad now an oad now a oad now an oad now an
oad now a
to The
five to the
boxing to wizards
jumpa to t
quickly to
ArT
more and pl
more and pl more and p more and pl more and pl more and pl
more and pl
to 10% FREE to 10% FREE 10% FREE to 10% FREE to 10% FREE 10% FREE to 10% FREE
EK THR SK THR CK THR TITRR
THIR
HITHR
N
WE
BEST
BEST
BEST
BEST
BEST
BEST
BEST
FT
S FT The S FT five S
FT boxing S FT wizards S
FTJump S
FT quickly S
S
WD S
SQROMIY
S
Wib Specuity
Specirty
Speciary
SQUARE PO
SQUARE PI
SQUARE PD
SQUARE PD
SQUARE PI
SQUARE PI
SQUARE PD
Wib 3
WES Specialy
Win Specialy F
With Specialy
Win Specialty
With Specialt D
Wib Specialty T
COCOText
Cheese
Choese
Choose
Cheese
Choose
Cheese
Cheese
Cheese wf
Cheese w/
Cheese w/
Cheese of
Cheese o/
Cheese w/
Cheese w/
The
five
boxing
wizards
1 jump
quickly
ayward.c c ayward.c ayward.c
c ayward.co c ayward.c ayward.c c ayward.c C
PRo
PR
five
Pho
boxing
PRo
wizards
PRo
jump
Pho
quickly
PRO
Oex
NEW
NEH five
NEW
NEW
The
boxing
wizards
Tump
quickly
SORBET
SORBET LIGHT
SORBET LIGHT
SORBET LIGHT
SOREET FLIGHT
PSORBET FLIGHT
SORBET LIGHT
LIGHT
v5l
v5
15
155
15
1l
WETT NER ET
NERR ET The
EIE ET five
BETNER ET boxing
PEETERR ET vecerda
EEEERRR ET jump
BET ET quickyy
ICDAR13
Stor
The Stor
five Stor
boxing Stor
wizards Stor
jump Stor
quickly Stor
NATIONAL
NATIONAL EXPRESS
NATIONAL EXPRESS
NATIONAL EXPRESS
NATIONAL EXPRESS
NATIONAL EXPRESS
NATIONAL
EXPRESS
EXPRESS
UATED
UNTED
UNTEL
UNTED
FROM
The FROM
five FROM
boxing FROM
wizards FROM
jump FRON
quicky FROM
MEXICO
MEXICO
MEXICO
MEXICO
MEXICO
MEXICO
MEXICO
Figure 17: Examples of style-free text editing results on four real-world datasets.
23

---

Published in Transactions on Machine Learning Research (Feb/2024)
D.2 Additional Style-conditional Qualitative Results
We provide additional examples of editing different texts with specified styles in the same image by our method DIFFSTE in Figure 18] The first row shows the target visual text style in the generated image, Our method DIFFSTE generate correct texts following the specified text styles consistently. The generated texts naturally fit the surrounding region with suitable size and orientation. For example, texts in the last row in are placed in the same orientation with surrounding texts.
Figure 18
Input
The
five
boxing
wizards
jump
quickly
oad now a oad now an oad now an oad now a oad now a oad now a oad now an
to t
The to five to bexing to wizards to jump to quickly to
ArT
more and pl
more and pl more and pl more and pl more and pl more and pl more and pl
10% FREE 10% FREE to 10% FREE 10% FREE to 10% FREE 10% FREE to 10% FREE
TITTHR IK THR CK THR EK TITHR CK THR GK THR CK THI
BEST
BEST
BEST
BEST
BEST
BEST
S
FT The S
FiDe S
F S
wizarac S FT jump S
quickkyy S
WED SOOCRRYY
WDb WaOaa
Specury
SQUARE PD
SQUARE PET
SQUARE PD
SQUARE PD
SQUARE PD
SQUARE PE
SQUARE PD
WD Specait
Specialy
We Speiaaty
Specialy
NE
With Specially
COCOText
Choose
Choese
Choese
Cheese
Cheese
Choose
Cheese
Choose w/
Choese
Choese
Cheese w/
Cheese w/
Cheese we
Cheese w/
The
1 fine
I boxing
1 wizarass
I jump
1 quickly
ayward.c c ayward.c ayward.c
ayward.c C ayward.c
ayward.cc
ayward.co C
Pho
Pho
Pho
Pho
Pho
Pho
quickly Pho
TextOCR
bexing
SORBET
SCRBET
LIGHT
v55m
25
YER ET
WETTER NER ET The
HER Exfi
TER ET Eounn
BETTER ET
WETHER ET jump
WETTER reuckdy
ICDAR13
Stop
The Stor
five
Stor
boxing Stop
wizer de
Stor
jump Stor
quickly Stor
NATIONAL
NATIONAL
NATIONAL
NATIONAL
NATIONAL EXPRESS
NATIONAL
NATIONAL
EXPRESS
EXPRESS
EXPRESS
EXPRESS
EXPRESS
EXPRESS
UATEL
UNTEE ED
UATEL
UAT ED
UNTEL
UNTEE
UNTED
FROM
FROM
Five FROM
baxing FROM
WidAMAE FROM
jump FROM
quicly FROM
MEXICO
MEXICO
MEXICO
MEXICO
MEXICO
MEXICO
MEXICO
Figure 18: Examples of style-conditional text editing results on four real-world datasets.
24

---

Published in Transactions on Machine Learning Research (Feb/2024)
D.3 Additional Qualitative Results with Zero-Shot Style Combination
Font extropolation As mentioned in Section 4.2 our method DIFESTE is able to extend seen font style to unseen formats such as italic, bold and both of them. We provide additional examples for font extrapolation in Figure 19 Each row shows the input masked image and generated texts using the specifiod font and the font with other format combinations. The instruction for font combination is generated in this template Write "TExT" font FONT-FORMAY. As seen Figure 19 the generated texts are bolded and inclined following the given format keyword, showing the zero-shot generalizability of DIFFSTE.
andd play ard play and piay ara piay and play
% FREE Local % FREE Local FREE Local FREE Local FREE Local
FRE
FRE
FRE
FRE
FRE
C
Local C Local C
Local C
Local C
Imput
Icebery
lebergg + italic Loeberg bbold Ioebery+boo both
ght,
ght, About gh, About ght, About p ght, About
at they car at they car at they car at they car at they car
BBARY
BRARY Gluten
BRARY
BRARY
BRARY
Imput
Gluten- -italic
Gluten bold
Gluten+ booh
rts
rts CENTRE
rts CENTRE
rts CENTRE
rts CENTRE
ennooe - ark ennoo ark
ennoe ar enuiee ark
enuiee ark
ference C ference C ference Ce ference Ce ference Ce
Irpuut
SixCaps
SiaCops + italic StrCaps bold SinCaps +
RF
RF STATION
RF STATION
RF STATION
RF STATION
Imput
NovaMono
Mova + italic
NovaMono +bold
NovaMono + both
CE
CE CREAM
CE CREAM
CE CREAM
CE CREAM
Imput
Mallanna
Mallanna italic
Mallanna++oold
Mallanna
Figure 19: Examples of font extrapolation with our method DIFFSTE.
25

---

Published in Transactions on Machine Learning Research (Feb/2024)
Font interpolatoin As mentioned in Section 4.2 method DIFFSTE is able to create new font style by mixing two seen font styles. We provide additional examples for font interpolation in Figure 20 To generate instructions for mixing font styles, we use the template Write TEXT in font PONTI and FONT2, where FONT1 and FONT2 are two font names seen in training. As seen in Figure 20 DIFFSTE is able to mix the two font styles by merging the glyphs of the same character in different fonts using a simple natural language instruction.
FontI-render
Font1
FontΩ-render
Font2
Font1 + Font2
THE
THE
THE
THE
THE
Mews
Mews.
Mews
Mews
Mews
Reload
Reload
Reload
Reload
Reload
Plant
Plant
Plant
Plant
Flant
LEGAL
LEGAL
regac
Legac
LEGAL
Figure 20: Examples of font interpolation with our method DIFFSTE.
26

---

Published in Transactions on Machine Learning Research (Feb/2024)
D.4 Additional Qualitative Results with Natural Language Instructions
We provide additional examples for controlling text color with natural language instructions. In Figure 21 each row shows examples generated using the following instructions generated by ChalGP @ A mango- colored word "TEXT". @ The word "TEXT" resembles the color of an eggplant. @ The word "TEXT" is colored in a delicate, ladylike shade of lilac. @ The color of word "TEXT" is like a blazing inferno. @ The color of word "TEXT" is deep, oceamic, reminiscent of the deep sea. The color of word "TEXT" used is the fresh, vibrant color of a springtime tree. As seen in Figure 21] DIFFSTE is able to generalize to unseen natural language instructions, which shows the effectiveness of our instruction learning framework in helping the model learn the mapping from natural language instructions to visual text appearances in the generated images.
Input
mango
eggplant
blac
inferno
tree
COLCHESTER
COLCHESTER
COLCHESTER
COLCHESTER
COLCHESTER
COLCHESTER
SPECI
SPECI
SPECI
SPECI
SPECI
SPEOI
SPECI
RE
PESTIVE BE
PESTIVE BE
FESTIVE HE
FESTIVE GE
FESTIVE BE
FESTIVE BE
ILLUSTR
I
ILLUSYR
ILLUSYI
ILLUSYR
ILLUSTR
I
London
London
London
London
London
London
London
welcome
welcome
A222
welcome
Relcome
(222
welcome
A22
welcome
A22
VERNM
VERNM
VERNM
VERNM
VERNM
VERNM
VERNM
STAFF
STAFF
STAFF
STAFF
STAFF
STAFF
F
black
black F
black
black
black F
black
ICK
ICK
ICK
ICK
ICK
ICK
ICK
EN STREET
EN STREET
EN STREET
EN STREET
EN STREET
EN STREET
EN STREET
XX 20 HEATE
RAIT NI
2:0 HOATE
ST SMONTGATE ROUTR
areal
areal
areal
areal
areal
area!
areal
to
Permit to
Permit to
Permit Permit to
Permit to
Permit to
system system
system o
system system in system
system
Managam
Managom
Managem
Managam
Do you Kn
Do you Kn Do you Kn Do you Kn Do you Kn Do you Kn Do you Kn
u
starting u
starting u starting u starting u starting u starting u
this year?
this year? this year? this year? this year? this year? this year?
Figure 21: Control text color using different natural language instructions with DIPFSTE.
We provide this instruction ChatGPT: Show some sentences describing the color of a text simslar this 'The text is in the color of grass" and mamually exclude unnecessary details from the generated sentences.
27

---

Published in Transactions on Machine Learning Research (Feb/2024)
D.5 Additional Qualitative Results with Arbitrarily-Shaper Mask
We provide additional examples for our method DIFFSTE editing texts in arbilrarily shaped masks in Fig- ure 22 As seen in Figure 22 our method is able to generate visually appealing text layouts within highly curved and lengthy masks, even when the text to generate is short. Notably, in the fourth row Figure [22] DIFFSTE successfully generates the words "The" and "five" in an appropriate layout within the curved mask. Fach character is placed a suitable orientation based on the geometry of the given mask region. In other examples, this is also observed for long texts such as "bozing".
Input
"The"
"fuve" "
"boxring"
"wizards" *
"jump"
"qusickly"
he MILE
f five MILE
f
e
boxing MILE
wizards MILE
j ump
quickly MILE
MILE
MILE
FOR READERS
FOR The READERS
FOR five READERS
FOR boxing READERS
FOR wizards READERS
FOR Jump READERS
FOR quickly READERS
The
five
toxing
wizards
jump
quickly
boxing
wizards
Jump
quickly
STOU
TOUU
STOU
STOU
STOU
TTOU
TTOU
S SABOR
ABOR ABOR ABOR S ABOR
ABOF S ABBOR
The DI
five DI
boxing DI cheese
wizards DI cheese .cheese
jump DI
quickly DI
DIE
.cheese
cceess
cheese
cheese
Figure 22: Generate texts within arbitrarily shaped mask.
28

---

Published in Transactions on Machine Learning Research (Feb/2024)
E More discussions
E.1 Comparison with other diffusion-based text editing model.
We conduct a comparison with two more diffusion-based text editing models, Text-difuuser Chen et al. (203bb) and SDXL Podell et al. (2023) on our test dataset in both style-free and style-cond settings. The quantitative results are presented in Table and example generations are listed in Figure 23 and Figure We highlight the following observations:
Significant improvements over SDXL. Compared to the backbone model we use, Stable- (SD) (865M), SDXL has much more than twice the parameters (2.7B). Nevertheless, the proposed dual-encoder design and instruction tuning framework can significantly improve SD's scene text editing capability. Equipped with our method, the SD/SD2 outper- forms the much larger model, SDXL, with 70% absolute improvement in text correctness rate.
Comparable performance as T-xtD- iffuser with less input informa- tion. Our method achieves stato-of-the- art performance and is comparable to TEXIDIPPUSER. This is impressive, es- pecially considering TEXTDIFFUSER has an advantage in the imput. Particularly, TEXT first renders a layout im- age that already contains the ground- truth content, font, color, and other in- formation about the scene text be added or edited. After that, the diffu- sion model will take the user prompt and the layout image for generation. In com- parison, our method only relies on the user prompt but still achioves good per- formance on lext correctness. We addi- tionally discuss the divergence in input information in point 3 below.
Style-free generation
Model
ArT
COCOText
TextOCR
ICDAR13
Average
DIFESTE
83.08
84.05
85.48
81.70
83.64
TEXTDIFFUSER
84.29
84.86
88,20
82.14
84.87
SDXL 5.41
9.74
8.67
6.20
7.51
Style-cond generation
Model
ArT
COCOText
TextOCR
ICDAR13
Average
DIFFSTE TEXTDIFFUSER
74.82
73.38
73.67
79.26
75.33
72.77
74.34
80.67
75.43
75.80
SDXL 4.83
9.75
6.89
5.13
6.65
Table OCR correctness on real-world datasets with style-free generation (top) and style-conditional generation (bottom) for DIFFSTE, TEXTDIFFUSER and SDXL. The correctness is evalu- ated by the automatic OCR model, and all numbers are reported in percentages.
Divergence between DiftDeffuser and our method. In our paper, mainly consider more flexible scene text editing, where users only need to specify the style information (e.g., color and font) and the content of texts through prompts. The diffusion model can also infer style that is coherent with the background and other parts of the image. In comparison, TEXFDIFFUSER focuses on specifying most text information via the layout image. While achieving better control of the generation, flexibility can sometimes be hurt. For example, we list two scenarios where the layout image in TEXTDIFFUSER cannot accurately and naturally specify the larget style follows:
Scene text editing on curved regions or perspective changes: the first and second column in Figure 23 and Figure 24 we illustrate examples where the target region highly curved or has perspective changes. TEXTDIFFUSER cannot provide high-quality and visually appealing text layouts within such curved regions. This is mainly because TEXTDIPFUSER highly depends the rendered layout image. Once the layout image for target scene texts cannot be generated perfectly (which is exactly the case in those examples), the generation quality would be largely decreasod. In comparison, our method provides a more flexible and robust. scene text editing for such scenarios and can generate significantly better images.
Tiffuuffuser cannot support style editing. As can be seen in the fourth row of Figure 24| TExtDifFuser cannot understand the style specification instructions and simply conducts style- free generation by generating texts following the color of other surronning texts (columns 1, 3, 4). In comparison, our instruction-tuned DifSTE model generates text faithful to the specified text style shown in the third row in Figure 24 which offers users more flexibility to edit the text style.
29

---

Published in Transactions on Machine Learning Research (Feb/2024)
8-19
FICIAL AM
Inpor
P
"Voueer" "
"Powered
"DBA" M
RETRACT"
Valuer
DUFSTE
Powered
8-19 DBA
FICIAL AM
RETRACT
PA
AKEN BY
CORPS
TEXT DIFFUSIR
Valuer
8 1-
FICIAL AM
Powered
RETRACT
DBA
PA
AKEN BY
CORPS AND
VALUER
8-19 aDd
SDXL
PIWLED
FICIAL AM
RERTIAT
PA
AKEN BY
CORPS AND
Figure 23: Examples of style-free text editing results with DIFESTE, TEXTDIFFUSER and SDXL.
Inpot
TI MA
From
PHIL
RENDUADD
AFB
BADMEN GAFF
malignant
SDXL TEXIDIPRUSI DIFSTTE
Cendoa
malignant
AFB delie
BADMEN
IP MA
CAFE
From
PHIL
malignant
MAD
BADMEN
GAFF
From
PHIL
Corhha
Madglannt
MAD
GGAR
From
PHIL
Figure 24: Examples of style-conditionall text editing results with DIFESTE, TEXTDIFUUEER and SDXL.
E.2 Limitation and Future work
Although DIFFSTE generally produces natural scene texts, evidenced by the human evaluations detailed in the main paper. Sometimes, the generation results may not be satisfiable. We attribute the unnaturalness to three main reasons: @ lack of surrounding scene texts, which makes inferring the correct lext style very challenging. It's evident that when more surrounding texts are present, DIFFSTE is better able to generate natural-looking texts; see examples columns 1 and 6 of Figure image in our dataset, which is limited by our computational resources. ① the synthetic data included in our dataset, where rendered text is simply placed onto background image. Enhancing the quality of training image dataset is potential avenue for improving overall image quality, We regard this key area for future work.
@ low resolution (256x256) training
30


Reviews:
[{"review_id": "YnAy3B1Mga", "reviewer": "Reviewer_KBmj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- In this work, editing is conducted by giving natural language instructions, rather than reference images as previous works, which is more convenient. \n- The performance gain over baseline methods is validated by both objective and subjective metrics.\n- The paper is clearly written and easy to follow.\n\nWeaknesses:\n\n- In Figure 6 (actually it is a table?), the advantage over baseline methods is not significant / consistent. The authors argue that this can be explained by the difference in instruction format, for which the baseline methods use a reference image, while this work uses natural language. This is a reasonable explanation, and I appreciate the author's honesty. However, I feel the reference image is easily available in real world scenarios. If you can specify a desired text by its characters, font, and color, you can directly render it as an example. Using a reference image as an extra condition is very likely to make this method more appealing.\n- Some recent works [1-2] are not covered.\n\n[1] TextStyleBrush: Transfer of Text Aesthetics From a Single Example\n\n[2] Scene Style Text Editing", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a diffusion model-based text editing technique that supports style-free and style-conditioned scene text-editing task with natural language instruction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- (optional) Add image instruction.\n- (obliged) Discuss the missing related works, and compare with them if applicable.", "broader_impact_concerns": "Generative models can be misused for spreading misinformation and strengthen existing bias in the data by its nature. Besides, this work can be misused for faking documents. The authors should at least discuss them.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wHRqaTX58v", "reviewer": "Reviewer_X8wx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The text correctness of DiffSTE is significantly better than SD/SD2.\n2. The instruction tuning design enables DiffSTE to perform zero-shot style combination and generalization. For example, the font extrapolation results in Figure 17 is interesting.\n\nWeaknesses\n\n1. According to Table 2, adopting the character-level encoder is the key to rendering correct texts. However, the character-level encoder design is largely inspired by the `Character-Aware Models Improve Visual Text Rendering` paper and is not novel. \n2. The author proposed to train the model on the combination of real-world dataset and synthetic dataset. The ratio for mixing the synthetic samples and real-world samples can be an important design choice. However, the author has not conducted ablation study about the mixture ratio. I think the paper just mentioned getting a 1.3M dataset that combines 600K synthetic samples and 700K real-world samples. The ablation study on synthetic v.s. real ratio is important because it can help quantify the contribution of synthetic samples for scene-text generation.\n3. According to Table 1, DiffSTE is on par with SD/SD2 in terms of image naturalness. In fact, the naturalness is worse than SD2 in TextOCR and is worse than SD in ArT. Samples in Figure 4 also show that the text edits generated by DiffSTE does not align with the image style very well, and seems to align too well with the fonts. This glimpse of unnaturalness is a limitation of DiffSTE.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Lacking the capability of generating correct text is a well-known limitation of stable-diffusion-like text-to-image generative models. The paper proposed DiffSTE that significantly outperforms SD / SD2 in terms of text correctness rate. DiffSTE adopts two encoders (a character-level encoder and a sentence-level) to encode the instruction. Later experiments (Table 2) show that the character-level encoder is crucial in boosting the text correctness. To enrich the training data, the author generates a synthetic dataset by rendering random text with random color and fonts in images and trains DiffSTE on the combination of synthetic samples and real-world text-image samples. Experiments show that DiffSTE is better than SD/SD2/Mostel in text correctness and is on par with SD2 in image naturalness. In addition, the author shows that DiffSTE is able to perform zero-shot style combination, such as interpolate between two fonts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The author needs to conduct ablation study in synthetic v.s. real mixture ratio.\n2. Is there a way to make the image more natural? Sometimes, it is okay to sacrifice a little bit of text correctness for more natural image.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eTmhTaZ84r", "reviewer": "Reviewer_ZirC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Dual-encoder architecture is design for input conditions of text character and font style. The char encoder extract the text shape and spatial information and fuse such information though cross-attention. The inst encoder extract the style information given the instruction.\n \n- Instruction tuning technology is applied to train these modules to equip the model with the ability of font stylization. \n\n- Through style instruction, the trained model is able to do style blending.\n\nWeakness:\n\n- Comparisons with diffusion-based related works are necessary, such as Text-diffuser  (code released) and  DiffUTE (if released) or text2image method such as SDXL  (code & model released). Although these methods are not explicitly conditioned on input font style condition, they capture font styles from the input image or input prompt (e.g. SDXL).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work propose a model DiffSte built upon pre-trained SD model together with an instruct tuning method. A dual encoder architecture is designed  for better text legibility and style control. Instruction tuning framework is enable font style control and blending. The generated results show the superiority of the proposed method over other baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Comparisons with diffusion-based methods such as Text-diffuser  (code & model released)  or  DiffUTE (if released), or text2image method such as SDXL  (code & model released)  can be presented.", "broader_impact_concerns": "No broader impact concerns are required.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Lacking the capability of  is a well-known limitation of stable-diffusion-like text-to-image generative models. The paper proposed DiffSTE for generating correct text, which is shown to significantly outperform SD / SD2 in terms of text correctness rate. DiffSTE adopts two encoders (a character-level encoder and a sentence-level) to encode the instruction. Experiments also show that the character-level encoder is crucial in boosting the text correctness. The authors proposed a synthetic dataset by rendering random text with random color and fonts in images and trains DiffSTE on the combination of synthetic samples and real-world text-image samples. In addition, the author shows that DiffSTE is able to perform zero-shot style combination, such as interpolate between two fonts.

After reading author's responses, all reviewers are leaning to accept this paper.
Decision: Accept (poster)