SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fine-grained control. When position control is not used, the mask and masked images at the input of the UNet are removed. When glyph control is not used, the latent code obtained from the text through the CLIP text encoder is used as the condition. When position control and glyph control are not used, there is a significant drop in performance. For example, when position control is not used, the OCR accuracy of the model drops by  $36.7\%$ , and the Cor drops by  $38.6\%$ . When glyph control is not used, the model cannot generate accurate text and the OCR accuracy of the model drops by  $39.8\%$ , and the Cor drops by  $41.5\%$ . These results show that position control can help the model focus on the area where text is to be generated, while glyph control can provide prior knowledge of the shape of the characters to help the model generate text more accurately.
Visualisation. In Figure 6, we provide some examples edited by DiffUTE. DiffUTE consistently generates correct visual text, and the texts naturally follow the same text style, i.e. font, and color, with other surrounding texts. We can see from the experiment that DiffUTE has a strong generative power. (i) In sample N1, DiffUTE can automatically generate slanted text based on the surrounding text. (ii) As shown in sample N2, the input is 234, and DiffUTE can automatically add the decimal point according to the context, which shows that DiffUTE has some document context understanding ability. (iii) In the sample CN4, DiffUTE can generate even artistic characters very well.
缪助力车骑行
张记者上海饮食
艾溪湖东站
交通出行。
哈啰助力车骑行 交通出行 1234567890
张记老上海馄饨铺（火炬）餐饮美食07-1514:12
艾溪湖东站一彭家桥站  
交通出行
2.7 0.63 38.0   
3.4 0.80 46.20   
4.2 1.00 56.70   
5.3 1.25 66.40   
6.6 1.60 81.34   
3.1 0.63 50.40   
3.9 1.00 59.40   
4.8 1.00 72.50   
6.0 1.25 79.60   
7.4 1.60 96.54
6.6 1.60 81.34   
3.1 0.63 50.40   
3.9 1.1 59.40   
4.8 1.00 72.50   
6.0 1.25 79.60   
7.4 1.60 96.54
这真是一首《风华颂》的曲调，虽然唱起来很  
像歌，却显得太单调了。  
这真是一首《风华颂》的曲调，虽然唱起来很  
像歌，却显得太单调了。  
这真是一首《风华颂》的曲调，虽然唱起来很  
像歌，却显得太单调了。
$⑤$  ①②③为一组，2对雌性雌蝇，如图13-4所示  
 $④$  只有条件满足  $⑤$  对雌性雌蝇，如图13-5所示，但这种雌蝇不严重，两组要作对照。【练习】  
几天间假鼠的111  
这几天假鼠小结一下题型  
王大成：假鼠的这一段路程会比较明显  
这段路程小结 跟想象多  
123456
123456  
图  
对 系列【解析】5每幅图均由2个图形组成，且均为“等腰”图形，分别测出对称轴。  
图 ①  $②③④$  分为一组，2条对称轴重合；图 ③④⑤⑥分组为一组，2条对称轴重合，8项当选。  
答 有学者考虑图③④⑤⑥对称，图③④⑤对称，但这种题作不严谨，两组要各取规律。[38]  
当天小组内如图：111  
这就是王小虎“下吧一走吧”  
王大头 potato：短边应设会比较明显  
这就是王小虎“就是想太多
E0
E3
E4
台州
E2
E5
地级市  
Image  
台州市地处浙江省沿海中部，  
州，西连丽水、金华，紧接  
积9411平方公里，领海和内；  
里，台州市的地理位置得天独厚  
丘陵相间，形成“七山一水二
天上人间地级市
台州市地处浙江省沿海中部，州，西连丽水、金华，紧接织9411平方公里，领海和内7里，台州市的地理位置得天独厚丘陵相间，形成“七山一水二：
CNO
CN3
0会员
¥8
含
347 交特
云贝石：
CN4
¥800.00
含运费0.00元。
父易前哪一聊
CN2
CN5
5.1 Scene Text Editing
Style transfer techniques based on Generative Adversarial Networks (GANs) have gained widespread popularity for scene text editing tasks Roy et al. [2020], Huang et al. [2022], Kong et al. [2022], Lee et al. [2021], Shimoda et al. [2021], Yang et al. [2020], Zhan et al. [2019]. These methods typically involve transferring text style from a reference image to a target text image. STEFANN Roy et al. [2020], for instance, leverages a font-adaptive neural network and a color-preserving model to edit scene text at the character level. Meanwhile, SRNet Wu et al. [2019] employs a two-step approach that involves foreground-background separation and text spatial alignment, followed by a
fusion model that generates the target text. Mostel Qu et al. [2023] improves upon these methods by incorporating stroke-level information to enhance the editing performance. However, despite their reasonable performance, these methods are often constrained in their ability to generate text in arbitrary styles and locations and can result in less natural-looking images.
5.2 Image Editing
Text-guided image editing has attracted increasing attention in recent years among various semantic image editing methods. Early works utilized pretrained GAN generators and text encoders to progressively optimize images based on textual prompts Bau et al. [2021], Gal et al. [2021], Pérez et al. [2003]. However, these GAN-based manipulation methods encounter difficulties in editing images with complex scenes or diverse objects, owing to the limited modeling capability of GANs. The rapid rise and development of diffusion models Rombach et al. [2022], Sahara et al. [2022b], Ruiz et al. [2023] have demonstrated powerful abilities in synthesizing high-quality and diverse images. Many studiesBrack et al. [2023], Brooks et al. [2023] have employed diffusion models for text-driven image editing. Among various diffusion models, Stable Diffusion Rombach et al. [2022] is one of the state-of-the-art models, which compresses images into low-dimensional space using an auto-encoder and implements effective text-based image generation through cross-attention layers. This model can easily adapt to various tasks, such as text-based image inpainting and image editing.
However, it has been observed that diffusion models exhibit poor visual text generation performance and are often prone to incorrect text generation. Only a few studies have focused on improving the text generation capability of diffusion models. Recently, one study trained a model to generate images containing specific text based on a large number of image-text pairs Liu et al. [2022b]. However, this work differs from ours in terms of application, as they focus on text-to-image generation, while ours concentrates on editing text in images. Another ongoing work, ControlNet Zhang and Agrawala [2023], has demonstrated remarkable performance in image editing by providing reference images such as Canny edge images and segmentation maps. While ControlNet achieves remarkably impressive results, it performs poorly on text editing tasks. To obtain better editing results, we incorporate auxiliary glyph information into the conditional generation process and emphasize local control in all diffusion steps.
5.3 Large Language Model
Large language models (LLMs) refer to language models that contain billions (or more) of parameters, which are trained on massive amounts of text data, such as models like GPT-3 Brown et al. [2020], Galactica Taylor et al. [2022], LLaMA Touvron et al. [2023] and ChatGLM Zeng et al. [2023]. Among them, ChatGLM is a billion-scale language model with rudimentary question-answering and conversational capabilities. It differs from BERT Devlin et al. [2018], GPT-3 and T5 Xue et al. [2021] architectures and is a self-regressive pre-training model that includes multiple objective functions. In this paper, we use ChatGLM to enhance the interaction capability of our model.
In this paper, we argue that the current diffusion model can not generate realistic text in images. To tackle this problem, we present a novel method DiffUTE, a diffusion-based universal text editing model. DiffUTE generates high-quality text through fine-grained control of glyph and position information, and benefits from massive amounts of text images through a self-supervised training approach. Moreover, by integrating a large language model (i.e., ChatGLM), we can use natural language to edit the text in images, enhancing the editing usability and convenience of model. Extensive experiments have shown that DiffUTE excels in textual correctness and image naturalness.
The main limitation of our method is that the accuracy of generated text will decrease as the number of characters to be edited in the image increases. This is due to the fact that as the number of characters increase, the spatial complexity of the characters will also increase, making the generation process more challenging. Therefore, our future work will focus on improving the generation quality and solving the problem of rendering long texts.
Yuxin Kong, Canjie Luo, Weihong Ma, Qiyuan Zhu, Shenggao Zhu, Nicholas Yuan, and Lianwen Jin. Look closer to supervise better: One-shot font generation via component-based discriminator. In CVPR, pages 13482-13491, 2022.
Junyeop Lee, Yoonsik Kim, Seonghyeon Kim, Moonbin Yim, Seung Shin, Gayoung Lee, and Sungrae Park. Rewritenet: Reliable scene text editing with implicit decomposition of text contents and styles. arXiv preprint arXiv:2107.11041, 2021.
Wataru Shimoda, Daichi Haraguchi, Seiichi Uchida, and Kota Yamaguchi. De-rendering stylized texts. In ICCV, pages 1076-1085, 2021.
Qiangpeng Yang, Jun Huang, and Wei Lin. Swaptext: Image based texts transfer in scenes. In CVPR, pages 14700-14709, 2020.
Fangneng Zhan, Hongyuan Zhu, and Shijian Lu. Spatial fusion gan for image synthesis. In CVPR, pages 3653-3662, 2019.
David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba. Paint by word. arXiv preprint arXiv:2103.10951, 2021.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108.00946, 2021.
Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing. In ACM SIGGRAPH, pages 313-318. 2003.
Chitwan Sahara, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH, pages 1-10, 2022b.
Nataniel Ruiz, Yuzhhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2023.
Rosanne Liu, Dan Garrette, Chitwan Sahara, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering. arXiv preprint arXiv:2212.10562, 2022b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, volume 33, pages 1877-1901, 2020.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*, 2018.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In ACL, 2021.