Harmonizing Visual Text Comprehension and Generation
Authors: Zhen Zhao
Jingqun Tang
Binghong Wu
Chunhui Lin
Shu Wei
Hao Liu
Xin Tan
zhizhong zhang
Can Huang
Yuan Xie
Venue: NeurIPS 2024 poster (2024)
Keywords: multimodal generative model
visual text comprehension and generation
LoRA expert
Primary area: generative_models

Abstract: In this work, we present TextHarmony, a unified and versatile multimodal generative model proficient in comprehending and generating visual text. Simultaneously generating images and texts typically results in performance degradation due to the inherent inconsistency between vision and language modalities. To overcome this challenge, existing approaches resort to modality-specific data for supervised fine-tuning, necessitating distinct model instances. We propose Slide-LoRA, which dynamically aggregates modality-specific and modality-agnostic LoRA experts, partially decoupling the multimodal generation space. Slide-LoRA harmonizes the generation of vision and language within a singular model instance, thereby facilitating a more unified generative process. Additionally, we develop a high-quality image caption dataset, DetailedTextCaps-100K, synthesized with a sophisticated closed-source MLLM to enhance visual text generation capabilities further. Comprehensive experiments across various benchmarks demonstrate the effectiveness of the proposed approach. Empowered by Slide-LoRA, TextHarmony achieves comparable performance to modality-specific fine-tuning results with only a 2% increase in parameters and shows an average improvement of 2.5% in visual text comprehension tasks and 4.0% in visual text generation tasks. Our work delineates the viability of an integrated approach to multimodal generation within the visual text domain, setting a foundation for subsequent inquiries. Code is available at https://github.com/bytedance/TextHarmony.

Harmonizing Visual Text Comprehension and Generation
Zhen !2 Jingqun Tang2M; Binghong Wu?, Chunhui Lin?, Shu Wei?, Hao Liu",
Xin Tan', Zhizhong Zhang" Can Huang", Yuan Xie
Fast China Normal University ByteDance
Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China 51255901056@ www..edd.cc...... tangjingqun @ bytedance.com, yxie@ cww.cc..ddu.cc
Abstract
In this work, we present TextHarmony, a unified and versatile multimodal genera- tive model proficient in comprehending and generating visual text. Simultaneously generating images and texts typically results in performance degradation due to the inherent inconsistency between vision and language modalities. To overcome this challenge, existing approaches resort to modality-specific data for supervised fine-tuning, necessitating distinct model instances. We propose Slide-LoRA, which dynamically aggregates modality-specie and modality-agnose agnossic LoRA experts, partially decoupling the multimodal generation space. Slide-LoRA harmonizes the generation of vision and language within a singular model instance, thereby facili- tating a more unified generative process. Additionally, we develop high-quality image caption dataset, DetailedTexxrCapp 100K, synthesized with sophisticated closed-source MI..M to enhance visual text generation capabilities further. Com- prehensive experiments across various benchmarks demonstrate the effectiveness the proposed approach. Empowered by Slide-LoRA, TextHarmony achieves comparable performance to modality-specife fine-tuning results with only 2% increase in parameters and shows an average improvement 2.5% in visual text comprehension tasks and 4.0% in visual text generation tasks. Our work delineates the viability of an integrated approach to multimodal generation within the visual text domain, setting a foundation for subsequent inquiries. Code is available https: //gi gigthu........ com/ weeeeeednneeeettaaaeee
Introduction
Visual text comprehension and generation tasks such scene text detection and recognition [79, 63, 40, 75, 62, 61, 24, 58), document understanding [64, 23), visual question answering (VQA) [26, 15, 27, 39, 59], key information extraction (KIE) [64, 23], multi-modal retrieval[33, 34, 30, 31, 36, 29, 32, 35, 28), visual text generation, editing, and crasure [66, 6, 5] are consistently of significant value for both academic research and practical applications. Recently, remarkable advancements have been achieved in visual text comprehension and generation, driven by the evolution of Multimodal Large Language Models (MILMs) and diffusion models. Foremost text-centric MILMS [73, 20, 27, 39] utilize a cohesive framework to comprehend text-rich images comprehensively, whereas diffusion-basse approaches [66, 6, 5] introduce innovative modifications to enhance visual text generation capabilities. As depicted in Figure 1, text-centric MILMs and diffusion models are capable of handling language and vision modalities adeptly, with MILMS generating texts and diffusion models producing images. However, integrating language and vision generation capabilities within a large multimodal model for visual text scenarios remains unexplored. This paper focuses on
Work done when Zhen Zhao intern ByteDance.
X
Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the simultaneous manipulation of language and vision generations to further streamline the processing of diverse text-centric multimodal tasks.
image visual text comprehension
text
(2) visual text generation
(3) comprehension and generation
TextMonkey
TextDiffusor
TextHarmony
(a) Comparison of model pipelines
Visual Text Comprehension
Visual Text Generation
Give a detailed description of this image.
Generate image: heur holding board suying "Hello World".
DREAM
WEATH
HELLO WORLD
The image shows a book cover with the fitle "DREAM HEALTH" and the subtitle "How Live Healthy, Balanced Life an Unbalanced World". The book written by Dr. Brian Wilmovski. The left of the cover stands cheerful man and the right boat on the lake. The whole image presents a positive and calm atmosphere under the sun.
BRIAN WILMOVSKY
Visual Text Perception
Visual Text Editing
Locate and recognise all the text in this picture.
mashed part of the image with "play"
SOUTHERNMOST
real men play the ukulel
SOUTHERNMOST [140, 114, 428, 148]
real men
HOTEL RECEPTION
HOTEL [208, 196, 331, 196]
the ukulek
THE [228, 200, 331, 220]
RECEPTION [188,
306]
(b) Visual text-related multimodal generation tasks
Figure 1: Figure (a) illustrates the different types of image-text generation models: visual text comprehension models can only generate text, visual text generation models can only generate images, and TextHarmony can generate both text and images. Figure (b) illustrates the versatility of TextHarmony in generating different modalities for various text-centric tasks.
In the general multimodal domain, some pioneering efforts [57, 16, 80, 65) empower MLLMs with the ability to generate images beyond texts, vastly extending the versatility of multimodal models. Such advancements inspire us to develop text-centric multimodal generative model. Our foundational model follows these approaches, incorporating a VIT-based image encoder, text tokenizer, an LLM, a text detokenizer, and a diffusion-based image decoder.
Previous works [57, 80, 65] and our pilot experiments (Figure 2) have shown that multimodal generation often leads a notable decline performance due to the substantial inconsistency between language and vision modalities in the generation space. Prior studies [57, 16, 80, 65] commonly rely on modality-seecific supervised fine tinettniig to bolster generative capacities. A nuanced challenge involves boosting generative capabilities across modalities using a singular model instance. Bixed-of-Expeett (MoE) [54] architecture is widely adopted in LLMs because it improves the model's scalability while keeping the cost of inference similar to that of smaller models. Impressed by MOE-based models [18, 9] that set up lask-specific experts to handle different tasks efficiently, propose adapting modality-speciff experts to partially decouple the generation of images and texts. Transforming a dense multimodal generative framework into an MoE-based sparse model poses significant challenges due to the high computational demand and extensive training data requirements.
To tackle these challenges, we incorporate Low-Rank Adaptation(Loke [22] experts instead. Specif- ically. we integrate multiple LoRA experts into the vision encoder and LLM components, encompass- ing modality-agnostic experts, vision modality generation experts, and language modality generation experts. Modality-specine experts are instrumental in refining and integrating modality-specinf generative representations, while modality-agnosti experts enhance certain general representations. A dynamic gating network then amalgamates these modality-speciin and general generative rep-

---

resentations, assigning precise expert weights. Consequently, with minimal parameter increase, we enhance image comprehension and generation capabilities within singular model instance. Slide-LoRA A achieves results comparable to those obtained through separate modality-specifft fine- tuning, demonstrating the efficacy our approach in bridging the gap between language and vision modalities in multimodal generation.
Uni-Modal Output
Multi-Modal Output TextHarmony
64.5
62.4
61.1
0.86
59.5
57.6
60.2
0.75
47.1
0.68
0.36
0.33
0.35
DocVQA
TabFact
TexVVQ
AnyText-Bench (NED) MARIO-Eval (CLIP score)
Text Generation
Image Generation
Figure 2: Comparison single-modal and multi-modal output performance in text generation and image generation Tasks. "Uni-Modal Output" represents the results achieved by modality supervised fine-tuning. "Multi-Modal Output" represents the results achieved by modal-independent supervised fine-tuning. Compared to the multi-modal output, major performance degradation in the uni-modal output observed for both text generation and image generation tasks.
Despite the strides made with Sli harmonizing comprehension and generation, the image quality produced by TextHarmony requires improvement. High-quality, detailed image caption data crucial for visual text-centric image generation tasks. Thus, a detailed image caption dataset, DetailedTextCaps- 100K, is created using an advanced closed-source MLLM with prompt engineering. By incorporating Detailed/exxtapss 100K in the supervised mne-tuning stage, TextHarmony 's image generation quality significantly improves over using original simple captions alone.
Utilizing the above approaches, TextHarmony versatile in various visual text-centric tasks. In visual text perception tasks, TextHarmony is able to perform well in text detection and recognition, achieving state-of-the-an performance text grounding tasks. In visual text comprehension tasks, HextHarmony achieves comparable performance to dedicated text comprehension models. In image generation tasks, TextHarmony matches performance dedicated visual text generation models. The contribution of this paper be summarised in three folds:
We introduce TextHarmony, a versatile large multimodal that allows for the unification of diverse visual text perception, comprehension, and generation tasks. TextHarmony performs comparably to specialized models in visual text perception, comprehension, generation, and editing.
To mitigate the inconsistency between vision and language modalities in the generative space, we propose Slide-LoRA. Slide-LoRA dynamically aggregates modality-specific and modality- agnostic LoRA experts, partially decoupling the multimodal generative space. With Slide-LoRA, TextHarmony achieves comparable performance to modality-specific fine-tuning results by only adding 2% parameters in a singular model instance.
A high-quality dataset of detailed visual text image captions (DetailedTeex(aps-I0000 100K) is con-
structed with closed-source MILM to enhance the performance of visual text generation.
2
Related Work
2.1 Visual Text Comprehension
Recent multi-modal large language models have increasingly focused on comprehending images with textual information [26, 39, 15, 14, 72, 69, 41, 76, 60, 53, 67]. Among them, UniDoc [15] and mPL.U-DocOwl [72] creates noval text-oriented instruction-follooinng datasets. mPLUG-DocOwl 1.5 [21] builds parsing system for visual text and employs large-scale pre-training. Monkey [26], DocPedia [14], and HRYDA [27] comprehend dense text by supporting higher image resolution. TextMonkey [39] adopts shifted window attention and filters out significant tokens.

---

2.2 Visual Text Generation
Eeffusion-baseed lext-to-image generation has recently achieved impressive progress [19, 50, 77, 52,
48, 7J. while the capability of rendering accurate, coherent text in images remains an open problem. DiffUTE [4] designs a text editing model through fine- line-grained glyph and position information control. GlyphControl [71] generates visual text images by first rendering the glyph and then performing the denoising process. TextDiffuer [6] builds a large-scale text images dataset with OCR annotations and generates visual text conditioned
character-level layouts. Further, TextDiffuer-2 [5] leverages language model as the layout planner, relieving the character-level guidance. Anytext [66] integrates the text-control diffusion pipeline with auxiliary latent module and a text embedding module, which has achieved remarkable success in multilingual visual text generation.
2.3 Unified Multi-Modal Comprehension and Generation
Current multi-modal large language models (M-LLMs) [82, 2, 11] largely use predicting the next text token as the training objective but exert no supervision for visual data [57]. Recent researches [57, 56, 65, 80, 16, 17. 70] have been attempting empower M-LLMs to generate visual elements. The Emu family [57. 56] learns with seeeict-the-nnet-eeeeee
objective in multi-modality
and decodes the regressed visual embeddings by a visual decoder. MiniGPT-5 [80] introduces fixed number of special tokens into the LLM's vocabulary as the generative tokens for images. SEED-LLaMA [16] proposes the SEED tokenizer, which produces discrete visual codes with causal dependency and high-level semantics. MM-Interleaved [65] extracts fine tne-grained visual details from multiple images' multi-scale feature maps, proving effective in generating interleaved image-text sequences. They all focus generic multimodal generation, whereas no such work exists in the visual text domain.
3 Methodology
3.1 Model Architecture
Figure 3 presents overview of TextHarmony. The backbone network follows the paradigm established by MM-Interleaved [65], where Vision Encoder, an LLM, and an Image Decoder are internally integrated to empower the model to generate both visual and textual content. Specifically, the image embedding extracted by the Vision Encoder is abstracted by Q-Former [25] and aligns with text tokens. Image and text tokens then concatenated and forwarded through the LLM. The output token of the [.LM either predicts text content or serves as the conditional input of image detokenization. The image decoder perceives the conditional input from LLM and generates images based on the denoising diffusion process [19].
Given the multi-mon modal input X, the multi-moda generation task involves generating interleaved token sequences Y, which can be detokenized into both image and text contents. The token generation stage is achieved by maximizing the conditional probability under the classic auto-regressive paradigm as follows:
P(Y x) = II XY)) II P(Y))XYY)))
II X(Y,,,,Y=)) (I)
where Ny and NG refer to the index sets of the text and image tokens, respectively. Y, is the I-th predicted token and <1 is the set preceding tokens. After that, the text tokens are classified with linear layer g(-) to produce text content, while the image tokens serve as the condition input in the denoising diffusion process to produce image content. The loss function in the detokenization stage consists of the above two parts:
((X,Y) (f1eNx)"
E.,llc DM 1 211
text gencration
image generation
where Y is the ground-truth token sequence and DM is the diffusion model for image denoising.

---

Image
Hidden stute
Demoising Diflusion
SOUTHERNNGST
special that the hatel is the sourthernmost in the USA.
Slide-LoRA
Multi-Modal LLM
Llidc-LoRA
Merge
R
Image Generation
Text Generation
Modal Agnostic
Vision
What makes hotel special from the signage?
Modal-Aware Gating
SOUTHERNNAGT
Change the word "RECEPTION" to "W "Welcome" in this picture.
RECEPTIO JUN
Text Prompt
Figure 3: Pipeline of TextHarmony. TextHarmony generates both textual and visual content by concatenating vision encoder, LLM, and an image decoder. The proposed Slide-LoRA module mitigates the problem of inconsistency multi-modal generation by partially separating the parameter space.
On the supervision of Equation the optimization of TextHarmony is tremendously difficult due to inconsistent training objectives. Firstly, text generation aims to generate text from images, while image generation aims generate images from text, which are mutually exclusive. Secondly, text generation is a classifier problem with cross-entropy loss function, while image generation (i.e., the denoising process)
regression problem with a mean square error loss function. To this end, we propose to mitigate the training inconsistency problem by adaptively adjusting the forward pass according to the input tokens.
3.2 Slide-LoRA: Enhancing Image and Text Generation Consistency
To harmonize multi- generation tasks in a single model, the parameter space would be optimized for inconsistent or conflict training objectives, stated before. We propose Slide-LoRA (which is shaped like a "slide" between different LoRA experts as shown in Figure 3), a novel module that be conveniently inserted into Transformer layers Low-Rank Adaptation (LoRA) [22] and introduces limited parameter increase. As such, Slide-LoRA spontaneously processes text generation and image generation in separate parameter spaces, thus relieving the inconsistent training problem.
As shown Figure Slide-LoRA is composed (R" R'. R³). R and R' serve as the separate parameter space for text and image generation, respectively, while R$ aims to learn the knowledge shared by both text and image generation. Specifically, Given the input token sequence established using either MLPs or mused discriminators) determines whether the processing of the input token sequence requires knowledge of text generation image generation, and produces scalar = ((x) e [0, 1]. The output of the Slide-LoRA layer can formulated as
gating network G and three LoRA modules
LxD, the gating network G (which can be
0 (17 0.5 RT(x)+ 0.5] R R (x)))
where [:] equals 1 if the condition inside is true and ( otherwise. Slide-LoRA incorporates task- specific and task-shared knowledge from input tokens, thus separating the inconsistent training objective and learning the shared knowledge of text and image generation.
5

---

MARIO-LAION
MARIO-LAION
Dreann Stringe Plano
PRINCESS BRIDE
DetailedTextCape
DetailedTextCaps
The image is a cover of an album of dreamy relaxing music. The bright
The image a poster of the movie "The Princess Bride.. The poster features man and woman standing castle. The man is wearing a cape and holding is looking
Dreamy
features a staryy night sky moon. There is a violin in the foreground. reads "Dreamy', Bed".
Strings & Piano
Time For Beul
The text
dress. They
"Sirings
Figure 4:
Captions from DetailedTeet(aps--000 and MARIO-LAION for the same image. Detailed/exxCCps-1000 100K can better depict the textual elements in the image.
3.3 Multi-Modal Pre- Pre-Training and Comprehensive Fine- TineiTunigg
TextHarmony training process consists of two stages. In the multi-modal pre-training stage, TextHar- mony is trained on text-rich image-text corpus and learns to produce multi-modal outputs. In the comprehensive fine-tuning stage, we concurrently cultivate the text and image generation capabilities of TextHarmony by training on series of text-centric tasks.
3.3.1 Stage I: Multi-Modal Pre-Tree Pre-Training
TextHarmony is pre-trained based on the pre-training weight of MM-Interleaved [65], with extra text-rich datasets including MARIO-LAION [6] and DocStruct4M [21]. MARIO-LAION contains 9M web images with brief captions and the according OCR results. DocStruct4M consists of 2M documents and IM natural images with text-oriented structure annotations. We use MARIO-LAION for both text and image generation i.e., either predict the caption of the image generate the image based on the caption), and we use DocStruct4M for text generation only. In this stage, we freeze the vision encoder and the LLM, training only the Q-Former and the image decoder to obtain basic image understanding and generation capabilities.
3.3.2 Stage 2: Comprehensive Fine-Tuiee TeeeeTunnng
We integrate various text-centric datasets and employ uniform instructions for all tasks. In this stage, the vision encoder, Q-former, image decoder, and the proposed Slide-LoRA are trained to enhance the multi-modal generation and human-instrrctioon-flloivviiiiii
capabilities of TextHarmony.
Visual Text Generation. In this task, TextHarmony generates images according the text description and is required to render accurate and coherent text. Although MARIO-LAION contains captions of text-rich images, the description oversimplified and lacks concentration on the textual elements within the image. To this end, we sample 100K images from MARIO-LAION and generate detailed captions about them, termed Detailed/TexCCas- elements in the images. This is achieved by prompting Gemini Pro [12], pioneer multi-modal large language model, to generate detailed descriptions based on the sampled image and the OCR results. As shown in Figure 4, the image description from DetailedTextCaps- 100K is more comprehensive compared with MARIO-LAION and can better depict the textual elements in the image.
The captions focus on both visual and textual
Visual Text Editing. In this task, TextHarmony substitutes or renders text in the given location of the image and keeps the background consistent. We randomly mask the image with the help of MARIO-LAION's OCR results and fine-tune TextHarmony in self-supervised manner.
Visual Text Comprehension. We employ the training set collected Monkey [26] for the text- centric VQA fine-tuning. The training set involves 1.4M QA pairs and covers various text-rich scenarios.
Visual Text Perception. For the basic OCR capabilities, we randomly sample 1M images from MARIO-LAION and leverage the OCR annotations.

---

Experiment
4.1 Experimental Setup
Implementation details. We use CLIP-ViT-L/44 [49], Vicuna-13B [81], and Stable Diffusion v2.1 [51] as the vision encoder, the LLM and the image decoder, following MMM-Iterreaaeed [65]. The image resolution is increased to 896 to capture fine-grained features better. A Q-Former with 12 blocks adopted reduce the number of visual tokens 512. the multi-modal pre-training stage, the initial learning rate is set to le 5, while in the fine-tuning stage. it is reduced to 5e - 6. The pre-training stage takes 3264 A-100 hours with a batch size of 256; While the fine-tumii fine-tuning stage takes 2352 A-100 hours with a batch size of 64.
Datasets and Metrics. We evaluate TextHarmony on broad range of vision-language tasks. Visual Text Comprehension includes Document-Orienter VQA (InfoVQA [43], Doc VQA [44], ChartQA [42]). Table VQA (TabFact [8), WTQ [47]), Scene Text-Centric VQA (TextVQA [55], OCRVQA [45], STVQA [3]) and OCRBench [38]. We adopt the Accuracy metric following TextMonkey [39]. Visual Text Generation includes Any www.eeeennchmrk--eeee
[66] and MARIOEval [6], where the metric of NED, FID, and CLIP Score are used following AnyText [66] and TextDiffuser [6].
4.2 Quantitative Analysis
Table 1: Results of visual text comprehension. TextHarmony is compared with both uni uni-modal generation models and multi-moue modal generation models. We employ the Accuracy metric for all methods. TextHarmony" is trained without Slide-LoRA.
Method
VQA ChariQA
Table VQA
Scene Text-Centric VQA TexIVQA OCRVQA STVQA
OCRBench
InfoVQA DocVQA
TabFact WTQ
Modela for text generation only
LLaVAR [78]
16.5
12.3
41.8
39.2
346
UniDoc [15]
14.7
7.7
10.9
46.2
36.8
35.2
DocPedia [14]
15.2
47.1
46.9
60.2
$7.2
45.5
mPL.GG-ww12 [74]
18.9
17.9
19.4
53.9
58.7
49.8
366
LL [37]
14.7
8.5
9.3
38.7
58.1
38.1
297
Monkey [26]
25.8
50.1
54.0
49.8
25.3
64.3
64.4
54.7
514
InternVI. [10]
23.6
28.7
45.6
59.8
30.5
62.2
517
InterL. M-XXommpeerrr [13]
28.6
39.7
51.6
62.3
28.7
62.2
49.6
59.6
511
Models for both text and image generution
SEED-LLaMA-14B [16]
23.5
6.5
13.8
49.2
13.2
14.4
16.3
20.1
357
MiniGPTS [80]
2.1
1.6
1.4
4.9
0.9
2.8
2.3
2.4
68
MM-Interieaved [65]
17.0
8.1
11.9
40
15.1
37,2
11.7
26.4
197
"extHarmony"
26.1
44
36.2
59.5
26.1
57.6
51.9
47.2
397
TexIHarmony-CHe
28.9
61.1
57.6
448
TextHarmony
28.5
47.1
38.0
62.4
27.1
60.2
55.3
49.7
440
Table 2: Text grounding performance on MARIO-Eval. The Acc@0.5 metric is employed.
TGDoc [68]
DocOwl 1.5 [21]
TextHarmony
82.5
84.3
88.7
4.2.1 Visual Text Comprehension and Perception
Comparison to Text-Generatin and Multi-Modal Generation Methods. As shown in Table 1, we evaluate TextHarmony on a broad range of visual text comprehension tasks following [39]. As we can see, the performance of TextHarmony is comparable that of SOTA methods specialized for visual comprehension overall. Specifically, on Info VQA and TabFact, TextHarmony achieves one of the top performances, with an accuracy of 28.5% and 62.4%. On Doc WTQ and TextVQA, TextHarmony scores 47.1 %, 27.1% and 60.2%, which is competitive against top-tier methods like Monkey (50.1%, 25.3% and 64.3%) and 62.2%). On ChartQA, STVQA, and OCRBench, our model lags behind Monkey, Intern VL, and MiternlM-XCComoseee which can be attributed to pre-training weights or training data variations. However, TextHarmony surpasses other methods like mPLUG-Ow12 and LLaVA 1.5VA..5-7B. Furthermore, the performance state-of-the-art multimodal generative models has a significant performance gap compared to TextHarmony.
(39.7%, 28.7% and
7

---

thus lowered from Comparison to TextHarmony" and TextHarmony-Chat To further validate the effectiveness of the proposed method, we train two copies of TextHarmony. TextHarmony" trained without the proposed Slide-LoRA module, while TextHarmony- Chat is trained with only Visual Comprehension data and forms the upper bound of TextHarmony on visual text comprehension tasks. As shown the bottom of Table 1, the performance of Texttarmony" is strictly lower than that of TextHarmony. For example, Tex scores 3.1% higher on Doe AQ,, 2.9% higher on TabFact, and 3.4% higher on OCRVQA. Overall, the performance increase brought by Slide-LoRA on comprehension tasks is 2.5%. And the performance gap between TextHarmony and TextHarmony- 3.96% to 1.5%.
Comparison on Text Grounding and Recognition. As shown in Table we evaluate the text grounding capabilities by sampling 300 visual text-position pairs from MARIO-Ever TextHarmony achieves 88.7% and surpasses current MI.LMs with text grounding capabilities (i.e., TGDoc and DocOwl 1 1.5). The performance on OCRBench (Table 1) reflects the text recognition capabilities of MLLMs. Specifically, TextHarmony surpasses LLa VAR, mPLUG-Ow12, and LLa by 94, 74, and 143, while the gap between TextHarmony and top-tier MLLMs remains.
Table 3: Results of visual text editing and generation. TextHarmony is compared with both uni- modal generation models and multi-modal generation models. TextHarmony" is trained without Slide-LoRA.
NED (f)
FID (1)
CLIP Score (t)
Models for image generation only
TextDiffuser-2 [6]
0.81
336
0.35
GlyphControl [71]
345
0.36
AnyTexx [66]
0.88
352
0.36
Models for both text and image generation
SEED-LLaMA- [16]
0.11
348
0.27
MiniGPTS [80]
0.02
380
0.25
MM-Interleaved [65]
0.04
412
0.29
TextHarmony"
0.68
356
0.33
TexxHarmony-Gen
0.86
330
0.36
TextHarmony
0.75
342
0.35
4.2.2 Visual Text Generation and Editing
As shown in Table 3, we compare TextHarmony with diffusson model-based text-to-image generation methods and multi-modal generation models. We evaluate the performance of visual text editing by sampling 200 images from AnyText-benchmark, ben.hmannch-----eeeeee ENN and randomly choosing one available text polygon in each image. We use the NED metric to evaluate visual text editing. The performance of Image Generation is evaluated by sampling 100 text-image pairs from MARIO-Eval using the FID and CLIP score metrics. TextHarmony achieves 0.75 NED and 0.35 CLIP score, which is comparable to GlyphControl and TextDiffuser. Note that TextHarmony is a multi-modal generation model that is not specialized for image generation and editing, while other multi-modi generation models can hardly generate visual text.
We further train image generation version of HarmonyText with only image generation/editing data, termed HarmonyText--Gr As we can see, CeermonnTText-Co achieves 0.86 on NED and 0.36 CLIP score, surpassing most image-generation methods and achieving comparable results to Anytext. Morcover, the application of Slide-LoRA brings an improvement 0.07 on NED (from 0.68 to 0.75) and 0.02 on CLIP score (from 0.33 to 0.35), further validating the effectiveness of Slide-LoRA.
4.3 Ablation Studies
Impact of the Config Choice of Slide-LoRA. As shown in Table 4, we compare the performance of TextHnrmony by varying the total number of LoRA modules used in Slide-LoRA. As we can see, the varied number of LoRA modules has little influence on comprehension and generation performance. For example, the accuracy on TextVQA is 60.2%. 60.4%, and 60.4% when increased from 3 to 9,

---

Table 4: Ablation studies of the config choices of Slide-LoRA and the places to insert Sli.e-LoRA.
Image to Text
Text to Image
TeX
Info
OCRBench
NED
CLIP Score
w/o Slide-LoRA
57.6
26.1
426
0.68
0.33
Config Choice Slide-LoRA
n=3, s=l
60.2
28.5
440
0.75
0.35
n=6, s=2
60.4
28.2
440
0.73
0.35
n=9, s=3
60.4
28.3
442
0.74
0.36
Vision Encoder
26.7
432
0.69
0.34
Place Insert Slide-LoRA
LLM Both
59.9
28.1
434
0.73
0.35
60.2
28.5
440
0.75
0.35
Table 5: Ablation studies of DetailedTextCaps.
Detaleed(ee((Cas-10000
NED
FID (4)
CLIP Score
w/o
0.70
368
0.32
w/
0.75
342
0.35
Impact of varied places to insert Slide-LoRA. As shown in Table 4, simply inserting Slide-LoRA into the vision encoder would strongly decrease the performance. On the contrary, simply inserting Slide-LoRA into the LLM brings limited performance degradation. The LLM processes multi-modal inputs and generates both visual and textual tokens, while the vision encoder receives only image inputs and generates visual tokens. As result, when inserted only in the vision encoder, Slide-LoRA would unable to separate parameter spaces for text generation and image generation.
Impact of TexailedTextCaas--1oe 100K dataset in Table 5. By removing DetailedTexxCaps 100K from the training set and using the original caption from MARIO-LAION, the performance of both the image editing and the image generation decreased. Specifically, the NED decreased from 0.75 0.70, and the CLIP score decreased from 0.35 to 0.32. This further validates the effectiveness of detailing the image caption in visual text when training TextHarmony.
We validate the effectiveness of the proposed DetailedTextCaps-
Prompts
SD-XL
PixArt-a
Glyph
AnyText
TextDiffuser-2
TextHarmony
Good
Good Time.
cake "Good Time".
Good
GOV
Keep Facused
KEEP FOCUSED
T-shirt "Keep Focused"..
KEEP FOCUSED
Photo book cover of "Summer
SUMER SUMMER + Love
Summmr
Love
Summer Love
SUTDRer Low
Figure 5: Visualisation of visual text generation.
4.4 Qualitative Analysis
We present examples of visual text generation in Figure 5 and examples of visual text editing of TextHarmony in Figure
Limitation
Despite the strides made by TextHarmony in unifying visual text generation and comprehension within single model instance, it is important acknowledge certain limitations of our approach.

---

Origin
AnyText
TextDiffuer-2
TextHarmony
THE LAST TIME SAW HER
THE Good TIME SAW HER KATHLEEN KNOWLES
THE GOOD TIME I SAW HER
THE GOOa TIME I SAW HER KATHLEEN KNOWLES
KATHLEEN KHOWIES
KATHLETN KNOWLES
real Imen play the ukulek
real Boy play the ukulek
real BOy play the ukulek
real Boy play the ukulek
STORY PART
STORY PARTS
STORY PART
STORY
PART
GOOD DEEDS
GOOD STORY
GOOD STORY
GOOD STORY
Figure 6: Visualisation of visual text editing.
Firstly, the performance of Taxmonn in visual text perception and comprehension tasks marginally lags behind some of the state-of-the-art open-source models. This discrepancy could be attributed to variations in the training data the foundational model itself. Secondly, the model's proficiency in generating images is less robust in scenarios characterized by dense text, indicating an area that necessitates further investigation and enhancement. These limitations highlight the need for ongoing research to refine the capabilities of TextHarmony and similar multimodal generative models.
6 Conclusion
In summary, this work presents TextHarmony, a versatile multimodal generative model adept at reconciling the disparate tasks of visual text comprehension and generation. Utilizing the proposed Slide-LoRA mechanism, TextHarmony synchronizes the generation process of both vision and lan- guage modalities within singular model instance, effectively addressing the inherent inconsistencies between different modalities. The model architecture is proficient in executing tasks that involve processing and generating images, masks, texts, and layouts, particularly within the realms of Optical Character Recognition (OCR) and document analysis. The accomplishments of TextHarmony herald the significant potential for comprehensive multimodal generative models within the visual text do- main. The adaptability of TextHarmony indicates that models of a similar nature could be effectively employed across diverse array of applications, offering the prospect of revolutionizing sectors that depend on the intricate interplay of visual text comprehension and generation.
Acknowledgement. This work is partially supported by National Natural Science Foundation of China (No.62222602, No.6210601 16.62176092, No.62302167, No.62476090, U23A20343), Natural Science Foundation of Shanghai (23ZR 1420400). Shanghai Sailing Program (23YF1410500) Natural Science Foundation of Chongging, China (CSTB2023SCQQ-QX0007 CSTB2023NSCQ- MSX0137). Chenguang Program of Shanghai Education Development Foundation and Shanghai Municipal Education Commission (23CGA34).
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch. Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 15:23716-237535 2022.
[2] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023.
[3] Ali Furkan Biten, Ruben Tito, Andres Maila, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar. and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291 4301, 2019.
[4] Haoxing Chen, Zhuoer Xu, Zhangxuan Gu. Yaohui Li, Changhua Meng. Huijia Zhu, Weiqiang Wang, Diffute: Universal text editing diffusion model. Advances in Neural Information Processing Systems, 36, 2024.
[5] Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser-2: Unleashing the power of language models for text rendering, arXiv preprint ar arXiv:2311.1666,, 2023.
[6] Jingye Chen, Yupan Huang, Tengchao Lv. Lei Cui, Qifeng Chen, and Furu Wei. Texidiffuser: Diffusion models text painters. Advances in Neural Information Processing Systems, 36, 2024.
[7] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xic, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-imaae synthesis, arXiv preprint arXiv:2310.00420 2023.
[8] Wenhu Chen. Hongmin Wang. Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li. Xiyou Zhou, and William Yang Wang. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint MeXXv:1909.02160 2019.
[9] Zeren Chen, Ziqin Wang, Zhen Wang, Huayang Liu, Zhenfei Yin, Si Liu, Lu Sheng, Wanli Ouyang. Yu Qiao, and Jing Shao. Octavius: Mitigating task interference in milms via moe. arXiv preprint arXiv:2311.02665 2023.
[10] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su. Guo Chen, Sen Xing. Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, el al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguitti tasks. arXiv preprint 2rXiv22312.14238, 2023.
[11] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hai. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[12] DeepMind. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. 2023.
[13] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang. Haodong Duan, Maosong Cao. et al. Internlm-xcopposeee xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16422 2024.
[14] Hao Feng, Liu, Hao Liu, Wengang Zhou, Houqiang Li, and Can Huang. Docpedia: Unleashing the power of large multimodal model in the frequency domain for versatile document understanding. arXiv preprint Xiv: 1. 1810, 2023.
[15] Hao Feng Zijian Wang, Jingqun Tang, Jinghui Lu, Wengang Zhou, Houqiang Li. and Can Huang. Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding. arXiv preprint arXiv:2308.1 11592, 2023.
[16] Yuying Ge, Sijie Zhao, Ziyun Zeng. Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama and draw with seed tokenizer. arXiv preprint 3rXiv:2310.01218, 2023.
[17] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified muati-granullrity comprehension and generation. arXiv preprint arXiv:2404. 14396., 2024.
[18] Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit- Yan Yeung, James T Kwok, and Yu Zhang. Mixture of cluster conditiona lora experts for vision-languse instruction tuning. arXiv preprint arXiv:2312.12379 2023.
11

---

[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[20] Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914 2023.
[21] Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understanding. arXiv preprint arXiv:2403. 12895, 2024.
[22] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, Lora: Low-rank adaptation large language models. arXiv preprint arXiv:2 09685, 2021.
[23] Yupan Huang. Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layouthmv3: Pre-training for document ai with unified text and image masking. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4083-4091, 2022.
[24] Qing Jiang, Jiapeng Wang, Dezhi Peng, Chongyu Liu, and Lianwen Jin. Revisiting scene text recognition: A data perspective. In Proceedings of the IEEE/CVF international conference on computer vision, pages 20543-2055 20554, 2023.
[25] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- -mage pre-training with frozen image encoders and large language models. In Internutional conference on machine learning. pages 19730-19742. PMLR, 2023.
[26] Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint ar arXiv:2311.06607 2023.
[27] Chaohu Liu, Kun Yin, Haoyu Cao, Xinghua Jiang, Xin Li, Yinsong Liu, Deqiang Jiang, Xing Sun, and Linli Xu. Hrvda: High-resolution visual document assistant. arXiv preprint 2444,6180, 2024.
[28] Daizong Liu, Xiang Fang, Pan Zhou, Xing Di, Weining Lu, and Yu Cheng. Hypotheses tree building for one-shot temporal sentence Incalization. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 1640-1648, 2023.
[29] Daizong Liu, Yang Liu, Wencan Huang, and Wei Hu. A survey on text-guided 3d visual grounding: Elements, recent advances, and future directions. arXiv preprint 240.v22406.55785, 2024.
[30] Daizong Liu, Xi Ouyang, Shuangjie Xu, Pan Zhou, Kun He, and Shiping Wen. Saanet: Siamese action- units attention network for improving dynamic facial expression recognition. Neurcomputing, 413:145-157, 2020.
[31] Daizong Liu, Xiaoye Qu, Xing Di, Yu Cheng, Zichuan Xu, and Pan Zhou. Memory- guioed semantic learning network for temporal sentence grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1665-1673, 2022.
[32] Daizong Liu, Xiaoye Qu, Jianfeng Dong, and Pan Zhou Adaptive proposal generation network for temporal sentence localization in videos. arXiv preprint arXiv:2 2rxiv:2109.000 (09,0998,, 2021.
[33] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng. Wei Wei, Zichuan Xu, and Yulai Xie. Context-aware biaffine localizing network for temporal sentence grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11244, 2021.
[34] Daizong Liu, Xiaoye Qu, Xiao Yang Liu, Jianfeng Dong, Pan Zhou, and Zichuan Xu. Jointly cross-and modal graph attention network for based moment localization. In Proceedings of the 28th ACM 2020.
International Conference on Multimedia, pages
[35] Daizong Liu, Xiaoye Qu, Yinzhen Wang, Xing Di, Kai Zou, Yu Cheng. Zichuan Xu, and Pan Zhou. Unsupervised temporal video grounding with deep semantic clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1683-1691, 2022.
[36] Daizong Liu, Pan Zhou, Zichuan Xu, Haozhao Wang, and Ruixuan Li. Few-shot temporal sentence grounding via memory-guided guided semantic learning. IEEE Transactions on Cireuits and Systems for Video Technology, (3(5)22491 -2505,,,--5,,,.--- 2022.
12

---

[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[38] Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin. et al. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv: 07895, 2023.
[39] Yuliang Liu, Biao Yang. Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.0444 04400004443,.,. 2024.
[40] Yuliang Liu, Jiaxin Zhang, Dezhi Peng, Mingxin Huang, Xinyu Wang, Jingqun Tang, Can Huang, Dahua Lin, Chunhua Shen, Xiang Bai, and Lianwen Jin. Spts v2: Single-point scene text spotting. IEEE Transactions on Pattern Analysis and Machine Intelligence,
2023.
[41] Chuwei Luo. Yufan Shen, Zhaoqing Zhu, Qi Zheng, Zhi Yu, and Cong Yao. Layouillm: Layout instruction tuning with large language models for document understanding. arXiv preprint arXiv:2404.0555 242255525, 2024.
[42] Ahmed Masry, Do Xuan Long, Jia Qing Tam, Shafiq Joty, and Enamul Hoque. Chartqu: A henchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
[43] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographievqa. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pages 1697-1706. 2022.
[44] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEF/CVF winter conference on applications of computer vision, pages 2200-2209,
2021.
[45] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages
IEEE, 2019.
[46] OpenAL. Gpt-4v(ision) system card. 2023.
[47] Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. arXiv preprint Xrx:: 050,0000550 2015.
[48] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXi::3307.01952 2023.
[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMIR, 2021.
[S0] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821 1831. Pmir, 2021.
[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
[52] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to- image difffusion models with deep language understanding, Advances in neural information processing systems, 35:36479-36433 2022.
[53] Bin Shan, Xiang Fei, Wei Shi, An-Lan Wang, Guozhi Tang, Lei Liao, Jingqun Tang, Xiang Bai, and Can Huang. Metbench: Multimodal cognition towards text-rich visual scenes benchmark. arXiv preprint arXiv:2410.1 1538, 2024.
[54] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-exxert layer. arXiv preprint arXiv: (00010000.6838 2017.
[55] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 8317 8326, 2019.
13

---

[56] Quan Sun. Yufeng Cui. Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, al. Generative multimodal models in-context learners. arXiv preprint 1ixi::3312.32266 2023.
[57] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, Yueze Wang, Hongeheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, 2023.
[58] Jingqun Tang. Weidong Du, Bin Wang, Wenyang Zhou, Shuqi Mei, Tao Xue, Xing Xu, and Hai Zhang. Character recognition competition for street view shop signs. National Science Review, m0(6) 2023.
[59] Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Hao Feng, Yang Li, Siqi Wang. Lci Liao, et al. Textsquare: Scaling up text-centric visual instruction tuning. arXiv preprint arXiv:2404. 12803, 2024.
[60] Jingqun Tang. Qi Liu, Yongjie Ye, Jinghui Lu, Shu Wei, Chunhui Lin, Wanqing Li, Mohamad Fitri Faiz Bin Mahmood, Hao Feng, Zhen Zhao, et al. Mtvqa: Benchmarking multilingual text-centric visual question answering, arXiv preprint arXiv:2405. 11955, 2024.
[61] Jingqun Tang, Wenming Qian, Luchuan Song, Xiena Dong, Lan Li, and Xiang Bai. Optimal boxes: boosting end-to-end scene text recognition by adjusting annotated bounding boxes via reinforcement learning. In Eumpean Conference Computer Vision, pages 233-248. Springer, 2022.
[62] Jingqun Tang, Su Qiao, Benlei Cui, Yuhang Ma, Sheng Zhang, and Dimitrios Kanoulas. You can even annotate text with voice: Transcription oolaaaooooooooooonoeeoo supervise text spotting. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4154 4163, 2022.
[63] Jingqun Tang, Wenqing Zhang, Hongye Liu, MingKun Yang, Bo Jiang, Guanglong Hu, and Xiang Bai. Few could be better than all: Feature sampling and grouping for scene text detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4563-4572, June 2022.
[64] Zineng Tang. Ziyi Yang. Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu. Michael Zeng, Cha Zhang. and Mohit Bansal, Unifying vision. text, and layout for universal document processing. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9924-19264,, 2023.
[65] Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yumtao Chen, Lewei Lu, Tong La, Jie Zhou, et al. Mm-interlcaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. arXiv preprint arXiv:2401. 10208, 2024.
[66] Yuxiang Tuo, Wangmeng Xiang, Jun Yan He, Yifeng Geng, and Xuansong Xie. Anytext: Multilingual visual text generation and editing. arXiv preprint arX1v:3311.03054, 2023.
[67] An-Lan Wang, Bin Shan, Wei Shi, Kun- Yu Lin, Xiang Fei, Guozhi Tang, Lei Liao, Jingqun Tang, Can Huang, and Wei-Shi Zheng. Pargo: Bridging vision-language with partial and global views. arXiv preprint arXiv:2408. 12928, 2024.
[68] Yonghui Wang. Wengang Zhou. Hao Feng, Keyi Zhou, and Houqiang Li. Towards improving document understanding: An exploration on lext-grounding via milms. arXiv preprint arXiv:2311. 2023.
[69] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge. Jinrong Yang, Jianjian Sun. Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-language models, arXiv preprint Xrx: 10,,0000000000 2023.
[70] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm. arXiv preprint 23xxi::30090051999 2023.
[71] Yukang Yang, Dongnan Gui, Yuhui Yuan, Weicong Liang. Haisong Ding, Han Hu, and Kai Chen. Glyph- control: Glyph conditional control for visual text generation. Advances in Neural Information Processing Systems, 36, 2024.
[72] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Junfeng Tian, et al. mplug-docowld Modularized multimodal large language model for document understanding. arXiv preprini urXiv:2307.024999 2023.
[73] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan. Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually situaate language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126, 2023.
14

---

[74] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXi::231..02257 2023.
[75] Wenwen Yu, Yuliang Liu, Wei Hua, Deqiang Jiang. Bo Ren, and Xiang Bai. Turning clip model into scene text detector. In IEEE Conference Computer Vision and Puttern Recognition, 2023.
[76] Yuechen Yu, Yulin Li, Chengquan Zhang, Xiaoqiang Zhang, Zengyuan Guo, Xiameng Qin, Kun Yao, Junyu Han, Errui Ding, Jingdong Wang. Structextv2: Masked visual-textual prediction document image pre- vre-training. arXiv preprint arXiv:2303. 00289, 2023.
[77] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[78] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang. and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023.
[79] Zhen Zhao, Jingqun Tang, Chunhui Lin, Binghong Wu, Can Huang, Hao Liu, Xin Tan, Zhizhong Zhang, and Yuan Xie. Multi-modal in-context learning makes ego-evolving scene text recognizer. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15567 15576, June 2024.
[80] Kaizhi Zheng, Xuchai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and- lann-uugee generation via generative vokens. arXiv preprint arXiv:2310.022.39 2023.
[81] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Fric Xing, et al. Judging Ilm-as-a-judge with mt-bench and chathot arena. Advances in Neural Information Processing Systems, 36, 2024.
[82] Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXi:2304..15922
2023.
15

---

A Appendix supplemental material
A.1 Experiments and Visualisation
A.1.1 Construction of Detailed/eetCaps-1000
DetailedTexx(aps-IO000 is constructed by re-generating the captions about the images from MARIO- LAION through prompting Gemini Pro. Given a sampled image, Gemini Pro is required generate a detailed caption about the image through the following prompt:
Play an image content analysis expert. First, analyze all the image contents in a comprehensive manner and pay special attention to the textual elements in this image. Then, describe the image in as much detail as you can.
After that, we eliminate the sample if its generated caption is longer than 100 or if the caption contains non-ASCII characters. Further, Advanced MLLMs, including Gemini, are utilized to check the quality of the generated captions through the following prompt:
Play image content analysis expert. First, analyze all the image contents comprehensively and pay special attention to the textual elements in this image. Then, judge whether the caption consistent with the image and whether the caption can thoroughly describe the image.
After the above filtering system, we obtained 102, 421 images with high-quality text-oriented captions.
To validate the effectiveness of DetailedTextCaps- 100K before merging into the training set, we let GPT-4V [46] determine which is better, the captions from DetailedTextCaps CocCC-sssSKOK- 100K or the captions from MARIO-LAION, Specifically, we randomly sample 88 image-caption pairs from DetailedTexxCaps, and GPT-4V is asked to answer the following question for each image:
100K
Which is better description for this picture? A:<Caption from DetailedTextCaps>> B: B:<Caption from
Please only answer or B without any other choices.
The evaluation result is 82 6, indicating that GPT-4V chooses captions from our DetailedTextCaps 100K as the better image description in most cases. The above evaluation protocol may contain biases from GPT-4V. Thus, we present this result for reference. Nevertheless, the ablation studies conducted in Section 4.3 prove the effectiveness of DetailedTTxtCCas- 100K in favoring visual text generation. More examples of DetailedTextCaps- 100K are shown in Figure? 7.
MARIO-LAION
MARIO-LAION
Merriam Webster Intermediate
No Distance Toa Far (Home to Blessing Book d2) cover
Thesaurus Hardcover
Merriam- Webster's Intermediate Thesaurus
DetailedTextCaps
DetailedTextCaps
The image shows the cover book titled "No Distance Too Far" by Lauraine Smelling. the second in the Home to features
The image contains book titled "Merriam- Webster's intermediate Thesaurus The book has grean with yellow and blue circle on the frant. The circle contains the worde Merriam-Webstee Internediate Thesaurus",
No Distance
book
Jea Thir
Blessing series. The a young woman in a white dress with pink sash standing fleld of wheat. The background sunset.
LAURAINE
SNELLING
MARIO-LAION
VELTERETTEE HOODY CATISFACTION GUARANTEED
MARIO-LAION
Pax Edwards LLC
30 Day Money Back Guarantee DetailedTextCCn
DetailedTextCaps
The image contains a blue and white circular seal. The seal a thick outer border with the words 1100% SATISFACTION GUARANTEED written in white. Inside the barder is a thinner words
The image contains a logo of Pax Edwards. The logo is compased two blue gray overlapping mountains and the company *Pax Edwards" below The logo is enclosed black frame.
30-DAY
PAX EDWARDS
MONEY-BACK BACK GUARANTEE
ring DAY" and "MONEY-BACK GUARANTEE".
Figure 7: More Examples of DetailedTextCaap-1000.
16

---

What the tnlo of this book?
DREAM
HEALTH
Dream Health: How to Live Balanced Life Unbalanced World
Scaett booil S.D.
Aun
DR. WILMOVSKY
Wimovski.
When is the momorandum dated?
Where 'HEALTH" this image?
November 8,
[[84, 175), [84, 118), [350, 118 (850, 175g
Locale and extract the text this image.
Who has sent the memorandum?
[|88 4171, 198, 390), 1332, 390), [3s2, 417]] WILMONSKY,
385), 362), [260, 362), (280, 365J
Mary Winston, Ed. D.
Figure 8: Visualisation of TextHarmony's visual text comprehension and perception capabilities.
A.1.2 Visualisation of visual text comprehension and perception
Figure 8 presents examples showing the visual text comprehension and perception capabilities of TextHarmony.
Table 6: Prompt design in comprehensive fine-tuning.
Task
Prompt
What the text in <mask> this image?
Where is <text> in this image?
Visual Text Perception
Extract all the text in this image.
Locate all the text in this image.
Locate and extract all the text in this image.
Visual Text Generation
Generate image according to the caption.
Visual Text Editing
Fill the masked part in this image with <text>
A.1.3 Prompt design and data formatting in comprchensive fine-tuning
Table 6 presents the prompt design of TextHarmony in the stage of comprehensive fine-tuning, For visual text comprehension and perception, the input data sequence in the training stage formulated as:
Answer following question based on the image. <Image> Question: <Question> Answer: <Answer>.
For visual text generation and editing, the input data sequence in the training stage is formulated as follows:
<lmage> <Instruction>> <Target Image>.
We employ an all-black image the input <Image> for visual text generation make the data formatting consistent with visual text editing. The input data processed into token sequences by image and text tokenizer, and TextHarmony is trained in the classical auto-regressive manner.
17

---

Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. We summarize these contributions in the conclusion of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper conducts critical evaluation of methodology and acknowledges
its inherent limitations. We summarize the limitations at the end of the main text.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper provides all necessary details reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We provide open access the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
hhblli//guiiee//eeeeStuissssss coiiliiisiooooocccr for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: ctsps.cc/public//eeeee guwdee/eee/ee// /CodeSubmia sgionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all the necessary details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
20

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources required reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http///eurrrp.eeeee Spppulic/ttticsuuiiiii
Answer: [Yes]
We conduct in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: This paper discusses both potential positive socictal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators and original owners of assets used in the paper properly credited. The license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "nGsQV6chA8", "reviewer": "Reviewer_SovZ", "summary": "This work presents TextHarmony, a unified and versatile multimodal generative model proficient in comprehending and generating visual text. Simultaneously generating images and texts typically results in performance degradation due to the inherent inconsistency between vision and language modalities.", "questions": "see weakness", "limitations": "see weakness", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-  This work introduces TextHarmony, a versatile large multimodal that allows for the unification of diverse visual text perception, comprehension, and generation tasks. TextHarmony performs comparably to specialized models in visual text perception, comprehension, generation, and editing\n\n- The proposed Slide-LoRA dynamically aggregates modality-specific and modality agnostic LoRA experts, partially decoupling the multimodal generative space.\n\n-  A high-quality dataset of detailed visual text image captions (DetailedTextCaps-100K) is constructed with a closed-source MLLM to enhance the performance of visual text generation", "weaknesses": "- Details of the Modal-Aware Gating are not given?\n\n-  Results of visual text editing and generation in Table 2 demonstrate that Anytext achieves better performance.\n\n-  The case in Fig. 5, \"Good Time\" and  \"Summer Love\", are not correctly synthesized, and the performance is worse than Textdiffuser2 and Anytext.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. If you have any further comments or suggestions, please let us know. Your concerns are addressed as follows:\n\n**W1: Details of the Modal-Aware Gating are not given?**  \n\n\nAs stated in **Line 123-127**,  the Modal-Aware Gating is an MLP module containing two linear layers. It determines whether the processing of the input token sequence requires knowledge of text generation or image generation. In the multi-modal pretraining stage, the Modal-Aware Gating is trained with $\\gamma$=1 in text generation and $\\gamma$=0 in image generation, according to the **Equation (3)** in the manuscript. We will update this section in the revised version.\n\n**W2: Results of visual text editing and generation in Table 2 demonstrate that AnyText achieves better performance.**\n\nPlease refer to the **Global Author Rebuttal**,  in which we address your concern in detail in three points. \n\n**W3: The case in Fig. 5, \"Good Time\" and \"Summer Love\", are not correctly synthesized, and the performance is worse than Textdiffuser2 and AnyText.**\n\n\nThanks for the careful review. The incorrectly synthesized character cases (i.e., \"e\" to \"E\" and \"r\" to \"R\") are case-sensitive text generation issues. As observed from Figure 5 of the manuscript, TextDiffuser-2, which uses the same training set to TextHarmony in visual text generation, also fails to generate the correct character cases (i.e., \"Keep Focused\" to \"KEEP FOCUSED\"). Given that the same training data (i.e. Mario-Laion) as TextDiffuser-2 is used, we check the training data and find that **some of the training data is case-insensitive**. Besides, aligned with AnyText's training set, TextHarmony is able to **generate correctly case-sensitive text**, as illustrated in Figure A of the PDF file submitted during rebuttal.  For the comparison with AnyText's performance, please also refer to the response in the **Global Author Rebuttal**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E0LUMsnhUC", "reviewer": "Reviewer_2avc", "summary": "TextHarmony is a versatile multimodal generative model designed to comprehend and generate visual text. Traditional methods struggle with the inconsistency between vision and language modalities, leading to performance issues. TextHarmony overcomes this with Slide-LoRA, which combines modality-specific and modality-agnostic LoRA experts, ensuring a unified generative process. The model is enhanced by a high-quality image caption dataset, DetailedTextCaps-100K. Experiments show that TextHarmony, with only a 2% increase in parameters, matches modality-specific fine-tuning performance and improves visual text comprehension and generation tasks.", "questions": "Please refer to the above weaknesses.", "limitations": "The interleaved generation ability of the proposed approach is unknown.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Explore the possibility of integrating visual text comprehension and generation\n+ The proposed Slide-LoRA method is effective to harmonize the training of different modalities and tasks\n+ Experimental results show that the combination is possible and the effectiveness of the proposed model", "weaknesses": "- The motivation of combing visual text comprehension and generation is not clear. Only the first try (maybe) to integrate the two tasks may not be convincing enough. \n\n- As shown in Table 1 and Table 3, the performance of the proposed method is not superior over existing baselines, e.g., TextHarmony vs. Monkey for comprehension and TextHarmony vs. AnyText for generation. It may be not easy to identify the advantages of combining visual text comprehension and generation. From this point of view, this work seems to simply try combing these two aspects, and not provides valuable research insights. \n\n- Due the limited performance improvement, the human evaluation becomes more necessary to discriminative the proposed method and existing baseline models. \n\n- More explanations or evidence are expected to support certain arguments, e.g., 1) “the optimization of TextHarmony is tremendously difficult due to inconsistent training objectives” in line 106, 2) “mutually exclusive” in line 108, and 3) why classifier and denoising problems are inconsistent as discussed in line 111? \n\n- Some unclear experimental settings, such as “w/o Slide-LoRA”, n and s in Table 4. \n\n- From the qualitative results in Figure 6, it seems that there is not any evident improvement of TextHarmony compared with AnyText.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful review and valuable comments. We are looking forward to further discussions with you. Your concerns are addressed as follows:\n\n**W1: The motivation ... is not clear ... first try ... may not be convincing ...**\n\n\nOur motivation is more than just addressing gaps in unified visual text comprehension and generation.\nA unified model combining visual  text comprehension and generation is indispensable in many aspects:\n- In many scenarios like multi-modal story generation [1], multi-modal document generation [2], the model is required to **generate coherent multimodal outputs**. Separate models cannot guarantee the contextual consistency of multimodal content. \n- Using separate models **increases deployment and maintenance costs**.\n- Recent works like GPT-4o[3] and Chameleon[2] show that the unification of visual and text generation expands the scope of MLLMs and enables a more unified process of multimodal data. Please also refer to reviewer LmXP \"S1: The unification ... visual text domain is a novel approach that **broadens the scope of LMM applications**\".\n\nConcerning the unsatisfactory performance of previous multi-modal generation models in visual text (refer to Table 1 and 3 in the manuscript), we focus on perception, comprehension and generation of visual text and **address the issue of multimodal generation inconsistency**.\n\n\n**W2 (1): The performance ... is not superior ...**\n\nPlease refer to the response to **Common Issue 1** in our Global Author Rebuttal.\n\n\n**W2 (2): ... not easy to identify the advantages of combining ...**\n\n- A unified generation model is indispensable (detailed in response to W1). \n- The modality inconsistence brings performance degradation. We relieves the degradation and \"achieves comparable performance to modal-specific models\" (refer to S1 of Reviewer r95i). \n- Aligning the settings with Monkey and AnyText, TextHarmony-Align **outperforms Monkey and AnyText** (refer to **Response (2),  Table B and C** in the Global Author Rebuttal).\n- Interleaved multimodal generation abilities(Figure B in the PDF file). \n\n**W2 (3): ... seems to simply try ... not provides valuable research insights.**\n\nOur work is **not** simply trying to combine visual text comprehension and generation.\n- Besides unifying visual text comprehension and generation, we demonstrates that the **inconsistency in the multimodal generative space** is a cause of underperformance. \n- The proposed Slide-LoRA shows 2.5% gains in comprehension and 4.0% in generation tasks, demonstrating its effectiveness against modal inconsistency.\n- Our contribution is also acknowledged by other reviewers (\"The analysis of the modality inconsistency problem ... is **reasonable**  ... SlideLoRA is **well motivated, novel and effective**\" by Reviewer r95i,  \"The results ... **demonstrates the effectiveness** of Slide-LoRA\" by Reviewer LmXP).\n\n**W3: Due to limited performance improvement, human evaluation ... necessary ...**\n\nSlide-LoRA achieves 2.5% gains in comprehension and 4.0% in generation tasks compared to baseline. TextHarmony-Align outperforms Monkey and AnyText(**Table B and C** in the **Global Author Rebuttal**). As a result, we think the improvement is **not limited**. Regardless, we agree with you about the importance of human evaluation(**Table D**). \n\n**W4: More explanations or evidence are expected to support certain arguments**\n\n(1): \"the optimization ... difficult due to inconsistent training objectives\" in line 106\n\nWe draw this conclusion mainly from our pilot experiment (Figure 2 in the manuscript). The performance declines a lot (4%~8%) when simultaneously optimizing comprehension and generation, which is also observed in studies like [2] (i.e., the performance of Chameleon-MultiTask is much worse than Chameleon-SFT).\n\n(2): \"mutually exclusive\" in line 108\n\nHere \"mutually exclusive\" refers to the fact that text and images are naturally inconsistent in many aspects (information density, data structure, information granularity, etc), thus text generation and image generation require different (exclusive) feature and generation space. Thus, modality alignment has been a fundamental issue in multimodal learning[1, 2, 4]. \n\n(3): why classifier and denoising ... inconsistent ... line 111\n\nPrevious studies [5,6] show the inconsistency of classification and regression tasks in deep learning. For example, in object detection, a double-head architecture splitting classification and regression has better performance than single-head[5]. In our work, the optimization of TextHarmony contains classification (text generation) and regression (image generation), which are also inconsistent and more difficult given that it is a multi-modal generation issue.\n\n**W5: Some unclear experimental settings, such as \"w/o Slide-LoRA\", n and s in Table 4.**\n\n\"w/o Slide-LoRA\" refers to training TextHarmony without Slide-LoRA module. \"n\" refers to the total number of lora experts in Slide-LoRA. \"s\" represents RT, RI and RS (Line 120-122), each contains \"s\" lora experts, i.e., s=n/3. We will add them in the revised version.\n\n**W6: ... there is not any evident improvement of TextHarmony compared with AnyText.**\n\nWe would like to clarify that we do not claim the performance of TextHarmony is better than AnyText.  On the issue of comparisons between model performances, please refer to **Response (2),  Table B and C**.\n\n**L1: The interleaved generation ability ... is unknown.**\n\nTextHarmony supports generate interleaved sequences, as showcased in Figure B in the PDF file.\n\n---\n\n>[1] SEED-Story: Multimodal Long Story Generation with Large Language Model. Arxiv 2024\n\n>[2] Chameleon: Mixed-Modal Early-Fusion Foundation Models.  Arxiv 2024\n\n>[3] Hello GPT-4o. OpenAI, 2024\n\n>[4] Learning transferable visual models from natural language supervision. ICML 2021\n\n>[5] Revisiting the Sibling Head in Object Detector. CVPR 2020\n\n>[6] D2det: Towards high quality object detection and instance segmentation. CVPR 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "16u3Try7Qa", "reviewer": "Reviewer_r95i", "summary": "This paper introduces a multimodal generative model (TextHarmony) for unified   comprehension and generation of visual text. To overcome the performance degradation brought by modality inconsistency, the authors propose the slide-lora, which partially decouples the multimodal generation space. An image-text caption dataset, DetailedTextCaps-100K, is also developed to enhance visual text generation capabilities.", "questions": "Line 74 ‘a versatile large multimodal’. Do you mean ‘a versatile large multimodal model’ ?", "limitations": "See above", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. TextHarmony involves visual text coprehension and generation in a single model for the first time. It achieves comparable performance to modal-specific models. It is a solid step forward for multimodal task unification in visual texts.  \n2. The analysis of the modality inconsistency problem in the multi-modal generation is reasonable and the proposed solution(SlideLoRA) is well motivated, novel and effective.", "weaknesses": "a) The connection between this work and visual text is not clearly stated. I understand the focus of this work is to construct a multimodal generative model, but it would be helpful to elaborate specifically how this work achieves multimodal generation in the field of visual text.\n\nb) It would be helpful to report the model size and the inference speed of TextHarmony.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time to review our paper,  the valuable comments and the approval of contributions to our work. And we are looking forward to further discussions with you. Your concerns are addressed as follows:\n\n\n**W1: The connection between this work and visual text is not clearly stated. ...**\n\nTo improve the performance of visual text comprehension, we increase the resolution of the input images (specifically from 448 to 896). Then, in the pre-training phase, we use images with rich OCR annotations (e.g., DocStruct-4M) to enhancing the text perception abilities of the model. For visual text generation, we randomly mask the text portions of the image and force the model to generate these portions in order to focus training on the generation of text elements.\n\n**W2: It would be helpful to report the model size and the inference speed of TextHarmony.**\n\nThanks for the suggestion. TextHarmony has **15.4 billion parameters** in total. On AnyText-Bench, the generation of each image costs **1340ms** on average. On DocVQA, the inference time for each output text token is **92ms** on average.\n\n**W3: Line 74 ‘a versatile large multimodal’. Do you mean ‘a versatile large multimodal model’ ?**\n\nYes. We will modify it in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jUy4txlM2P", "reviewer": "Reviewer_LmXP", "summary": "This work presents TextHarmony to simultaneously comprehending and generating visual text. The paper performs theoretical and experimental analysis about the performance degradation due to the inherent inconsistency between vision and language modalities. A MoE and LoRA-based module, Slide-LoRA, is then proposed to solve this problem by applying modal-specific and modal-independent LoRA experts dynamically. The experimental results indicate TextHarmony achieves comparable performance to modality-specific fine-tuning results.", "questions": "1. Could TextHarmony generate more complex glyphs such as Chinese characters?\n2. Have you tried using OCR tools to further polish the captions of DetailedCaps-100K? For example, removing the images whose captions are inconsistent with the OCR results.", "limitations": "Please refer to weaknesses.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The unification of image understanding and image generation into the visual text domain is a novel approach that broadens the scope of LMM applications.\n2. The results presented in this paper looks good and demonstrates the effectiveness of Slide-LoRA.\n3. The paper is well organized, with reasonable motivation and insights.", "weaknesses": "1. To my knowledge, there exist other multimodal generative models like DreamLLM[1] and Emu[2]. It would be better to compare with these methods.\n2. The authors put related work into the supplementary material. It would be better to briefly summarize the background and representitive works in the main text.\n\n[1] Dong, Runpei, et al. \"Dreamllm: Synergistic multimodal comprehension and creation.\" ICLR2024\n\n[2] Sun, Quan, et al. \"Emu: Generative pretraining in multimodality.\" ICLR2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments and the approval of contributions to our work. Your concerns are addressed as follows:\n\n\n**W1: Compare TextHarmony with DreamLLM and Emu.**\n\nIn Table E, we compare TextHarmony with DreamLLM and Emu in terms of both visual comprehension and visual generation. As we can see, TextHarmony **performs better** to DreamLLM and Emu in terms of visual text comprehension and generation.\n\n\n> #### Table E: Comparison between TextHarmony, DreamLLM and Emu.\n|    | DocVQA | TabFact | TextVQA | AnyText-Bench | Mario-Eval\n| :---  |   :----:   |  :---: |   :---: |   :---: |   :---: |\n| Emu      | 13.2       | 56.3  | 22.1 |  0.13 |  0.31 |\n| DreamLLM   | 32.7   | 60.4  | 41.8 |  0.18 |  0.30 |\n| TextHarmony   | **47.1** | **62.4**  | **60.2** | **0.75** |  **0.35** |\n\n**W2: Briefly summarize the background and representative works in the main text.**\n\nThanks for your advice. We will briefly summarize the background and representative works of our paper in the main text of the revised version.\n\n**Q1: Could TextHarmony generate more complex glyphs such as Chinese characters?**\n\nYes. The training data of TextHarmony in the manuscript only involves English, thus it could not generate Chinese characters. However, the training data of TextHarmony-Align (please refer to Global Author Rebuttal) contains Chinese characters, which makes it capable of generating Chinese characters. We show some examples of Chinese character generation in **Figure C** of the PDF file submitted during rebuttal.\n\n**Q2: Using OCR tools to further polish the captions of DetailedCaps-100K**\n\nThanks for the suggestion. We use DBNet [1] for text detection and Parseq[2] for text recognition. We filter out the captions that have over 50% mis-matching rate to the OCR results. Specifically, a detected text line is called \"mis-matching\" if it cannot be found in the caption. After the above procedure, a total of 1839 captions are filtered out from DetailedCaps-100K, suggesting that there is room for improvement in the dataset. Restricted by the rebuttal time, we would like to leave for future work the specific impacts of this strategy to the performance of TextHarmony.\n\n---\n\n> [1] Real-time Scene Text Detection with Differentiable Binarization. AAAI 2020.\n\n> [2] Scene Text Recognition with Permuted Autoregressive Sequence Models. ECCV 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks to the ACs and reviewers for taking time and effort to review our manuscript. And we are also looking forward to further discussion. Below, we would like to address the common issues raised by reviewers. 


**Common Issue 1: Performance compared to unimodal generation models, such as TextHarmony *vs.* Monkey in image comprehension and TextHarmony *vs.* AnyText in image generation. (W2 and W3 by Reviewer SovZ,  W2, W3, and W6 by Reviewer 2avc)**

- **Response (1): Unfair comparison between generic models and specialized models** 
    - Please kindly note that AnyText and Monkey are unimodal generation specialized models while TextHarmony is a multi-modal generation model. It is **not particularly fair** to simply compare the performance of a specific task **between specialized models and general models**.  For instance, Monkey's performance on scene text recognition, text grounding and text-centric VQA is much worse than specialized models, as shown in Table A.  Besides, in the field of multi-modal generation, the performance of multi-modal generation models (SEED-LLaMA [1], Emu [2], DreamLLM [3], MM-Interleaved [4], Chameleon [5]) is also inferior to unimodal generation specialized models.  
    - Thus, when a generic model is compared with a specialized model, the focus should be on overall performance, not just on the performance of a particular task. The overall performance of TextHarmony is **unanimously approved by all the reviewers** (Reviewer SovZ "TextHarmony performs comparably to specialized models", 2avc "Experimental results show...the effectiveness of the proposed model", r95i "achieves comparable performance to modal-specific models”, and LmXP  "The results presented in this paper looks good"). In addition, among multi-modal generation models, TextHarmony achieves much better performance than other models (Table 1 and 3 in the manuscript) in visual text comprehension and generation.

> #### Table A: Comparison of the performance of Monkey and specialized models.
|    | Scene Text Recognition (Union-14M) | Text Grounding (MSRA-TD500)   | Text-centric VQA (DocVQA) |
| :---  |   :----:   |  :---: |   :---: |
| Monkey      | 32.7       | 13.6  | 66.5 |
| Specialized Models   | **85.2** (MAERec [6]) |  **84.9** (DBNet [7]) | **88.4** (ERNIE-Layout [8]) |

---

- **Response (2): Aligning the settings with specialized models and human evaluation** 
    - Monkey and AnyText have different settings with TextHarmony in terms of model architecture, training data, etc. To address this case, we conduct ablation experiments **aligning the settings** of training data (AnyWord-3M in AnyText), image resolution (1344*896 in Monkey), LLM (Qwen 7B in Monkey), and model pipeline (two-stage visual text rendering in AnyText). As shown in Table B and C, our model (TextHarmony-Align) slightly **outperforms Monkey and AnyText** in image comprehension and image generation.  
   - What‘s more, following the constructive suggestion by Reviewer 2avc (many thanks), we conduct **human evaluation** in visual text editing following the setting established by TextDiffuser. Specifically, the questionnaire consists of 100 cases, which includes two multiple-choice questions: (1) Which of the following images has the best text rendering quality?  (2) Which of the drawn text best harmonizes with the unmasked region? We have collected 10 questionnaires, and the results are shown in Table D. TextHarmony-Align also outperforms AnyText and TextDiffuser2 in human evaluation.

>  #### Table B:  Comparison of the performance on visual text comprehension.  
|   | DocVQA| TextVQA | OCRBench |
| :---  |   :----:   |  :---: |   :---: |
| Monkey      | 50.1      | 64.3  | 514 |
| TextHarmony  | 47.1 |  60.2 | 440 |
| TextHarmony-Align | **52.9** |  **64.5** | **523** |


> #### Table C:  Comparison of the performance on visual text generation.
|    | NED| CLIP Score | 
| :---  |   :----:   |  :---: |  
| AnyText      |  **0.88**   | 0.36  | 
| TextHarmony  | 0.75 |  0.35 | 
| TextHarmony-Align | **0.88** |  **0.38** | 

> #### Table D: Human evaluation of visual text editing
| |TextDiffuser2| Anytext|TextHarmony|TextHarmony-Align
| :---|:---:|:---:|:---:|:---:|
|Q1|699|740|721|**765**|
|Q2|645|691|685|**698**|

---

- **Response (3): Comparison to the actual baseline**  
    - Our actual baseline model is the multimodal generative model without the addition of Slide-LoRA (TextHarmony* in Table 1 and 3 in the manuscript). TextHarmony equipped with Slide-LoRA shows an average improvement of **2.5%** in visual text comprehension and **4.0%** in visual text generation tasks. It demonstrates that TextHarmony considerably mitigates the issue of modal inconsistency in multimodal generation, which is what our paper focuses on.

Hopefully, our reply would address the concerns about model performance. 

---

>  [1] Making LLaMA SEE and Draw with SEED Tokenizer.  ICLR 2024. 

> [2] Emu: Generative Pretraining in Multimodality.  ICLR 2024. 

> [3] DreamLLM: Synergistic Multimodal Comprehension and Creation.  ICLR 2024. 

> [4] MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer. Arxiv 2024.

> [5] Chameleon: Mixed-Modal Early-Fusion Foundation Models. Arxiv 2024

> [6] Revisiting Scene Text Recognition: A Data Perspective. ICCV 2023.

> [7] Real-time Scene Text Detection with Differentiable Binarization. AAAI 2020. 

> [8] ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding. EMNLP 2022 Findings.

Meta-review: This paper proposed TextHarmony, a new model for visual text understanding and generation, aka, understanding and generating text in images, which is different from prior work for interleaved text and image generation. In addition to the method, the authors also introduce a new dataset particularly for this task. The results seem quite promising. Reviewers had some general concerns about inferior results on unimodal generation compared to other methods, but the AC agrees with the authors that is not the focus of this study, and the reviewers all stated their concerns are well addressed after the rebuttal. As for the connection and discussion with works like DreamLLM, it is a rather minor issue and can be addressed in the camera ready version. The AC thinks this is a solid work for visual text generation and recommends acceptance for this work.
Decision: Accept (poster)