Harmonizing Visual Text Comprehension and Generation

Authors: zhen  zhao
jingqun  tang
binghong  wu
chunhui  lin
shu  wei
hao  liu
xin  tan
zhizhong  zhang
can  huang
yuan  xie
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 11:14:57

Abstract: In this work, we present TextHarmony, a unified and versatile multimodal generative model proficient in comprehending and generating visual text. Simultaneously generating images and texts typically results in performance degradation due to the inherent inconsistency between vision and language modalities. To overcome this challenge, existing approaches resort to modality-specific data for supervised fine-tuning, necessitating distinct model instances. We propose Slide-LoRA, which dynamically aggregates modality-specific and modality-agnostic LoRA experts, partially decoupling the multimodal generation space. Slide-LoRA harmonizes the generation of vision and language within a singular model instance, thereby facilitating a more unified generative process. Additionally, we develop a high-quality image caption dataset, DetailedTextCaps-100K, synthesized with a sophisticated closed-source MLLM to enhance visual text generation capabilities further. Comprehensive experiments across various benchmarks demonstrate the effectiveness of the proposed approach. Empowered by Slide-LoRA, TextHarmony achieves comparable performance to modality-specific fine-tuning results with only a 2% increase in parameters and shows an average improvement of 2.5% in visual text comprehension tasks and 4.0% in visual text generation tasks. Our work delineates the viability of an integrated approach to multimodal generation within the visual text domain, setting a foundation for subsequent inquiries. Code is available at https://github.com/bytedance/TextHarmony.

Main paper: 
 introduction: 
 Visual text comprehension and generation tasks such as scene text detection and recognition [70, 54, 31, 66, 53, 52, 24, 49] , document understanding [55, 23] , visual question answering (VQA) [26, 15, 27, 30, 50] , key information extraction (KIE) [55, 23] , visual text generation, editing, and erasure [57, 6, 5] are consistently of significant value for both academic research and practical applications. Recently, remarkable advancements have been achieved in visual text comprehension and generation, driven by the evolution of Multimodal Large Language Models (MLLMs) and diffusion models. Foremost text-centric MLLMs [64, 20, 27, 30] utilize a cohesive framework to comprehend text-rich images comprehensively, whereas diffusion-based approaches [57, 6, 5] introduce innovative modifications to enhance visual text generation capabilities. As depicted in Figure 1 , text-centric MLLMs and diffusion models are capable of handling language and vision modalities adeptly, with MLLMs generating texts and diffusion models producing images. However, integrating language and vision generation capabilities within a large multimodal model for visual text scenarios remains unexplored. This paper focuses on the simultaneous manipulation of language and vision generations to further streamline the processing of diverse text-centric multimodal tasks. In the general multimodal domain, some pioneering efforts [48, 16, 71, 56] empower MLLMs with the ability to generate images beyond texts, vastly extending the versatility of multimodal models. Such advancements inspire us to develop a text-centric multimodal generative model. Our foundational model follows these approaches, incorporating a VIT-based image encoder, a text tokenizer, an LLM, a text detokenizer, and a diffusion-based image decoder.Previous works [48, 71, 56] and our pilot experiments (Figure 2 ) have shown that multimodal generation often leads to a notable decline in performance due to the substantial inconsistency between language and vision modalities in the generation space. Prior studies [48, 16, 71, 56] commonly rely on modality-specific supervised fine-tuning to bolster generative capacities. A nuanced challenge involves boosting generative capabilities across modalities using a singular model instance. Mixed-of-Experts (MoE) [45] architecture is widely adopted in LLMs because it improves the model's scalability while keeping the cost of inference similar to that of smaller models. Impressed by MOE-based models [18, 9] that set up task-specific experts to handle different tasks efficiently, we propose adapting modality-specific experts to partially decouple the generation of images and texts. Transforming a dense multimodal generative framework into an MoE-based sparse model poses significant challenges due to the high computational demand and extensive training data requirements.To tackle these challenges, we incorporate Low-Rank Adaptation(LoRA) [22] experts instead. Specifically, we integrate multiple LoRA experts into the vision encoder and LLM components, encompassing modality-agnostic experts, vision modality generation experts, and language modality generation experts. Modality-specific experts are instrumental in refining and integrating modality-specific generative representations, while modality-agnostic experts enhance certain general representations. A dynamic gating network then amalgamates these modality-specific and general generative representations, assigning precise expert weights. Consequently, with minimal parameter increase, we enhance image comprehension and generation capabilities within a singular model instance.Slide-LoRA achieves results comparable to those obtained through separate modality-specific finetuning, demonstrating the efficacy of our approach in bridging the gap between language and vision modalities in multimodal generation. Uni-Modal Output Multi-Modal Output TextHar mony Despite the strides made with Slide-LoRA in harmonizing comprehension and generation, the image quality produced by TextHarmony requires improvement. High-quality, detailed image caption data is crucial for visual text-centric image generation tasks. Thus, a detailed image caption dataset, DetailedTextCaps-100K, is created using an advanced closed-source MLLM with prompt engineering. By incorporating DetailedTextCaps-100K in the supervised fine-tuning stage, TextHarmony's image generation quality significantly improves over using original simple captions alone.Utilizing the above approaches, TextHarmony is versatile in various visual text-centric tasks. In visual text perception tasks, TextHarmony is able to perform well in text detection and recognition, achieving state-of-the-art performance in text grounding tasks. In visual text comprehension tasks, TextHarmony achieves comparable performance to dedicated text comprehension models. In image generation tasks, TextHarmony matches the performance to dedicated visual text generation models. The contribution of this paper can be summarised in three folds:• We introduce TextHarmony, a versatile large multimodal that allows for the unification of diverse visual text perception, comprehension, and generation tasks. TextHarmony performs comparably to specialized models in visual text perception, comprehension, generation, and editing.• To mitigate the inconsistency between vision and language modalities in the generative space, we propose Slide-LoRA. Slide-LoRA dynamically aggregates modality-specific and modalityagnostic LoRA experts, partially decoupling the multimodal generative space. With Slide-LoRA, TextHarmony achieves comparable performance to modality-specific fine-tuning results by only adding 2% parameters in a singular model instance.• A high-quality dataset of detailed visual text image captions (DetailedTextCaps-100K) is constructed with a closed-source MLLM to enhance the performance of visual text generation.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. We summarize these contributions in the conclusion of the introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
visual text comprehension: 
 Recent multi-modal large language models have increasingly focused on comprehending images with textual information [26, 30, 15, 14, 63, 60, 32, 67, 51, 44, 58] . Among them, UniDoc [15] and mPLUG-DocOwl [63] creates noval text-oriented instruction-following datasets. mPLUG-DocOwl 1.5 [21] builds a parsing system for visual text and employs large-scale pre-training. Monkey [26] , DocPedia [14] , and HRVDA [27] comprehend dense text by supporting higher image resolution. TextMonkey [30] adopts shifted window attention and filters out significant tokens.Diffusion-based text-to-image generation has recently achieved impressive progress [19, 41, 68, 43, 39, 7] , while the capability of rendering accurate, coherent text in images remains an open problem. DiffUTE [4] designs a text editing model through fine-grained glyph and position information control. GlyphControl [62] generates visual text images by first rendering the glyph and then performing the denoising process. TextDiffuer [6] builds a large-scale text images dataset with OCR annotations and generates visual text conditioned on character-level layouts. Further, TextDiffuer-2 [5] leverages a language model as the layout planner, relieving the character-level guidance. Anytext [57] integrates the text-control diffusion pipeline with an auxiliary latent module and a text embedding module, which has achieved remarkable success in multilingual visual text generation.Current multi-modal large language models (M-LLMs) [73, 2, 1, 11] largely use predicting the next text token as the training objective but exert no supervision for visual data [48] . Recent researches [48, 47, 56, 71, 16, 17, 61] have been attempting to empower M-LLMs to generate visual elements. The Emu family [48, 47] learns with a predict-the-next-element objective in multi-modality and decodes the regressed visual embeddings by a visual decoder. MiniGPT-5 [71] introduces a fixed number of special tokens into the LLM's vocabulary as the generative tokens for images. SEED-LLaMA [16] proposes the SEED tokenizer, which produces discrete visual codes with causal dependency and high-level semantics. MM-Interleaved [56] extracts fine-grained visual details from multiple images' multi-scale feature maps, proving effective in generating interleaved image-text sequences. They all focus on generic multimodal generation, whereas no such work exists yet in the visual text domain.3 MethodologyQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper conducts a critical evaluation of its methodology and acknowledges its inherent limitations. We summarize the limitations at the end of the main text. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
model architecture: 
 Figure 3 presents an overview of TextHarmony. The backbone network follows the paradigm established by MM-Interleaved [56] , where a Vision Encoder, an LLM, and an Image Decoder are internally integrated to empower the model to generate both visual and textual content. Specifically, the image embedding extracted by the Vision Encoder is abstracted by a Q-Former [25] and aligns with text tokens. Image and text tokens are then concatenated and forwarded through the LLM. The output token of the LLM either predicts text content or serves as the conditional input of image detokenization. The image decoder perceives the conditional input from LLM and generates images based on the denoising diffusion process [19] .Given the multi-modal input X, the multi-modal generation task involves generating interleaved token sequences Y , which can be detokenized into both image and text contents. The token generation stage is achieved by maximizing the conditional probability under the classic auto-regressive paradigm as follows:EQUATIONwhere N T and N I refer to the index sets of the text and image tokens, respectively. Y l is the l-th predicted token and Y <l is the set of preceding tokens. After that, the text tokens are classified with a linear layer q(•) to produce text content, while the image tokens serve as the condition input in the denoising diffusion process to produce image content. The loss function in the detokenization stage consists of the above two parts:EQUATION)where Ŷ is the ground-truth token sequence and DM is the diffusion model for image denoising. ...It is special that the hotel is the sourthernmost in the USA.Text Prompt On the supervision of Equation 2, the optimization of TextHarmony is tremendously difficult due to inconsistent training objectives. Firstly, text generation aims to generate text from images, while image generation aims to generate images from text, which are mutually exclusive. Secondly, text generation is a classifier problem with a cross-entropy loss function, while image generation (i.e., the denoising process) is a regression problem with a mean square error loss function. To this end, we propose to mitigate the training inconsistency problem by adaptively adjusting the forward pass according to the input tokens.To harmonize multi-modal generation tasks in a single model, the parameter space would be optimized for inconsistent or conflict training objectives, as stated before. We propose Slide-LoRA (which is shaped like a "slide" between different LoRA experts as shown in Figure 3 ), a novel module that can be conveniently inserted into Transformer layers as Low-Rank Adaptation (LoRA) [22] and introduces limited parameter increase. As such, Slide-LoRA spontaneously processes text generation and image generation in separate parameter spaces, thus relieving the inconsistent training problem.As shown in Figure 3 , Slide-LoRA is composed of a gating network G and three LoRA modules (R T , R I , R S ). R T and R I serve as the separate parameter space for text and image generation, respectively, while R S aims to learn the knowledge shared by both text and image generation. Specifically, Given the input token sequence x ∈ R L×D , the gating network G (concatenation of two linear layers in specific) determines whether the processing of the input token sequence requires knowledge of text generation or image generation, and produces a scalar γ = G(x) ∈ [0, 1]. The output of the Slide-LoRA layer can be formulated asEQUATIONwhere [•] equals 1 if the condition inside is true and 0 otherwise. Slide-LoRA incorporates taskspecific and task-shared knowledge from input tokens, thus separating the inconsistent training objective and learning the shared knowledge of text and image generation.The image is a cover of an album of dreamy and relaxing music. The cover features a starry night sky with a bright moon. There is a violin in the foreground. The text on the cover reads "Dreamy", "Strings & Piano", and "Time For Bed".Dreamy Strings & Piano Time For Sleep.Princess Bride.The image is a poster of the movie "The Princess Bride". The poster features a man and a woman standing in front of a castle. The man is wearing a cape and the woman is wearing a dress. They are holding hands and looking at each other. TextHarmony is pre-trained based on the pre-training weight of MM-Interleaved [56] , with extra text-rich datasets including MARIO-LAION [6] and DocStruct4M [21] . MARIO-LAION contains 9M web images with brief captions and the according OCR results. DocStruct4M consists of 2M documents and 1M natural images with text-oriented structure annotations. We use MARIO-LAION for both text and image generation (, i.e., either predict the caption of the image or generate the image based on the caption), and we use DocStruct4M for text generation only. In this stage, we freeze the vision encoder and the LLM, training only the Q-Former and the image decoder to obtain basic image understanding and generation capabilities.We integrate various text-centric datasets and employ uniform instructions for all tasks. In this stage, the vision encoder, Q-former, image decoder, and the proposed Slide-LoRA are trained to enhance the multi-modal generation and human-instruction-following capabilities of TextHarmony.Visual Text Generation. In this task, TextHarmony generates images according to the text description and is required to render accurate and coherent text. Although MARIO-LAION contains captions of text-rich images, the description is oversimplified and lacks concentration on the textual elements within the image. To this end, we sample 100K images from MARIO-LAION and generate detailed captions about them, termed DetailedTextCaps-100K. The captions focus on both visual and textual elements in the images. This is achieved by prompting Gemini Pro [12] , a pioneer multi-modal large language model, to generate detailed descriptions based on the sampled image and the OCR results. As shown in Figure 4 , the image description from DetailedTextCaps-100K is more comprehensive compared with MARIO-LAION and can better depict the textual elements in the image.Visual Text Editing. In this task, TextHarmony substitutes or renders text in the given location of the image and keeps the background consistent. We randomly mask the image with the help of MARIO-LAION's OCR results and fine-tune TextHarmony in a self-supervised manner.Visual Text Comprehension. We employ the training set collected by Monkey [26] for the textcentric VQA fine-tuning. The training set involves 1.4M QA pairs and covers various text-rich scenarios.Visual Text Perception. For the basic OCR capabilities, we randomly sample 1M images from MARIO-LAION and leverage the OCR annotations.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experimental setup: 
 Implementation details. We use CLIP-ViT-L/14 [40] , Vicuna-13B [72] , and Stable Diffusion v2.1 [42] as the vision encoder, the LLM and the image decoder, following MM-Interleaved [56] . The image resolution is increased to 896 to capture fine-grained features better. A Q-Former with 12 blocks is adopted to reduce the number of visual tokens to 512. In the multi-modal pre-training stage, the initial learning rate is set to 1e -5, while in the fine-tuning stage, it is reduced to 5e -6. The pre-training stage takes 3264 A-100 hours with a batch size of 256; While the fine-tuning stage takes 2352 A-100 hours with a batch size of 64.Datasets and Metrics. We evaluate TextHarmony on a broad range of vision-language tasks. Visual Text Comprehension includes Document-Oriented VQA (InfoVQA [34] , DocVQA [35] , ChartQA [33] ), Table VQA (TabFact [8] , WTQ [38] ), Scene Text-Centric VQA (TextVQA [46] , OCRVQA [36] , STVQA [3] ) and OCRBench [29] . We adopt the Accuracy metric following TextMonkey [30] .Visual Text Generation includes AnyText-benchmark-EN [57] and MARIOEval [6] , where the metric of NED, FID, and CLIP Score are used following AnyText [57] and TextDiffuser [6] . Impact of varied places to insert Slide-LoRA. As shown in Table 4 , simply inserting Slide-LoRA into the vision encoder would strongly decrease the performance. On the contrary, simply inserting Slide-LoRA into the LLM brings limited performance degradation. The LLM processes multi-modal inputs and generates both visual and textual tokens, while the vision encoder receives only image inputs and generates visual tokens. As a result, when inserted only in the vision encoder, Slide-LoRA would be unable to separate parameter spaces for text generation and image generation.Impact of DetailedTextCaps-100K. We validate the effectiveness of the proposed DetailedTextCaps-100K dataset in Table 5 . By removing DetailedTextCaps-100K from the training set and using the original caption from MARIO-LAION, the performance of both the image editing and the image generation decreased. Specifically, the NED decreased from 0.75 to 0. We present examples of visual text generation in Figure 5 and examples of visual text editing of TextHarmony in Figure 6 . 
limitation: 
 Despite the strides made by TextHarmony in unifying visual text generation and comprehension within a single model instance, it is important to acknowledge certain limitations of our approach.  
conclusion: 
 In summary, this work presents TextHarmony, a versatile multimodal generative model adept at reconciling the disparate tasks of visual text comprehension and generation. Utilizing the proposed Slide-LoRA mechanism, TextHarmony synchronizes the generation process of both vision and language modalities within a singular model instance, effectively addressing the inherent inconsistencies between different modalities. The model architecture is proficient in executing tasks that involve processing and generating images, masks, texts, and layouts, particularly within the realms of Optical Character Recognition (OCR) and document analysis. The accomplishments of TextHarmony herald the significant potential for comprehensive multimodal generative models within the visual text domain. The adaptability of TextHarmony indicates that models of a similar nature could be effectively employed across a diverse array of applications, offering the prospect of revolutionizing sectors that depend on the intricate interplay of visual text comprehension and generation.A Appendix / supplemental material A.1 Experiments and Visualisation 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: This paper discusses both potential positive societal impacts and negative societal impacts of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators and original owners of assets used in the paper are properly credited. The license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: New assets introduced in the paper are well documented.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: DetailedTextCaps-100K is constructed by re-generating the captions about the images from MARIO-LAION through prompting Gemini Pro. Given a sampled image, Gemini Pro is required to generate a detailed caption about the image through the following prompt:Play an image content analysis expert. First, analyze all the image contents in a comprehensive manner and pay special attention to the textual elements in this image. Then, describe the image in as much detail as you can.After that, we eliminate the sample if its generated caption is longer than 100 or if the caption contains non-ASCII characters. Further, Advanced MLLMs, including Gemini, are utilized to check the quality of the generated captions through the following prompt:Play an image content analysis expert. First, analyze all the image contents comprehensively and pay special attention to the textual elements in this image. Then, judge whether the caption is consistent with the image and whether the caption can thoroughly describe the image.After the above filtering system, we obtained 102, 421 images with high-quality text-oriented captions.To validate the effectiveness of DetailedTextCaps-100K before merging it into the training set, we let GPT-4V [37] determine which is better, the captions from DetailedTextCaps-100K or the captions from MARIO-LAION. Specifically, we randomly sample 88 image-caption pairs from DetailedTextCaps, and GPT-4V is asked to answer the following question for each image: The image contains a book titled "Merriam-Webster's Intermediate Thesaurus". The book has a green cover with a yellow and blue circle on the front. The circle contains the words "Merriam-Webster's Intermediate Thesaurus".The image shows the cover of a book titled "No Distance Too Far" by Lauraine Snelling. The book is the second in the Home to Blessing series. The cover features a young woman in a white dress with a pink sash standing in a field of wheat. The background is a sunset.No Distance Too Far (Home to Blessing Book #2) -coverThe image contains a logo of Pax Edwards. The logo is composed of two blue and gray overlapping mountains and the company name "Pax Edwards" below it.The logo is enclosed in a black frame.Pax Edwards LLCThe image contains a blue and white circular seal. The seal has a thick outer border with the words "100% SATISFACTION GUARANTEED" written in white.Inside the border is a thinner blue ring with the words "30-DAY" and "MONEY-BACK GUARANTEE".30 Day Money Back Guarantee Paper Checklist 


Decision: This paper proposed TextHarmony, a new model for visual text understanding and generation, aka, understanding and generating text in images, which is different from prior work for interleaved text and image generation. In addition to the method, the authors also introduce a new dataset particularly for this task. The results seem quite promising. Reviewers had some general concerns about inferior results on unimodal generation compared to other methods, but the AC agrees with the authors that is not the focus of this study, and the reviewers all stated their concerns are well addressed after the rebuttal. As for the connection and discussion with works like DreamLLM, it is a rather minor issue and can be addressed in the camera ready version. The AC thinks this is a solid work for visual text generation and recommends acceptance for this work.

Reviews: review: paper_summary: paper_summary: This work presents TextHarmony, a unified and versatile multimodal generative model proficient in comprehending and generating visual text. Simultaneously generating images and texts typically results in performance degradation due to the inherent inconsistency between vision and language modalities.
strength_weakness: strength_weakness: - Details of the Modal-Aware Gating are not given?

-  Results of visual text editing and generation in Table 2 demonstrate that Anytext achieves better performance.

-  The case in Fig. 5, "Good Time" and  "Summer Love", are not correctly synthesized, and the performance is worse than Textdiffuser2 and Anytext.
questions: questions: see weakness
limitations: limitations: see weakness
paper_summary: paper_summary: TextHarmony is a versatile multimodal generative model designed to comprehend and generate visual text. Traditional methods struggle with the inconsistency between vision and language modalities, leading to performance issues. TextHarmony overcomes this with Slide-LoRA, which combines modality-specific and modality-agnostic LoRA experts, ensuring a unified generative process. The model is enhanced by a high-quality image caption dataset, DetailedTextCaps-100K. Experiments show that TextHarmony, with only a 2% increase in parameters, matches modality-specific fine-tuning performance and improves visual text comprehension and generation tasks.
strength_weakness: strength_weakness: - The motivation of combing visual text comprehension and generation is not clear. Only the first try (maybe) to integrate the two tasks may not be convincing enough. 

- As shown in Table 1 and Table 3, the performance of the proposed method is not superior over existing baselines, e.g., TextHarmony vs. Monkey for comprehension and TextHarmony vs. AnyText for generation. It may be not easy to identify the advantages of combining visual text comprehension and generation. From this point of view, this work seems to simply try combing these two aspects, and not provides valuable research insights. 

- Due the limited performance improvement, the human evaluation becomes more necessary to discriminative the proposed method and existing baseline models. 

- More explanations or evidence are expected to support certain arguments, e.g., 1) “the optimization of TextHarmony is tremendously difficult due to inconsistent training objectives” in line 106, 2) “mutually exclusive” in line 108, and 3) why classifier and denoising problems are inconsistent as discussed in line 111? 

- Some unclear experimental settings, such as “w/o Slide-LoRA”, n and s in Table 4. 

- From the qualitative results in Figure 6, it seems that there is not any evident improvement of TextHarmony compared with AnyText.
questions: questions: Please refer to the above weaknesses.
limitations: limitations: The interleaved generation ability of the proposed approach is unknown.
paper_summary: paper_summary: This paper introduces a multimodal generative model (TextHarmony) for unified   comprehension and generation of visual text. To overcome the performance degradation brought by modality inconsistency, the authors propose the slide-lora, which partially decouples the multimodal generation space. An image-text caption dataset, DetailedTextCaps-100K, is also developed to enhance visual text generation capabilities.
strength_weakness: strength_weakness: a) The connection between this work and visual text is not clearly stated. I understand the focus of this work is to construct a multimodal generative model, but it would be helpful to elaborate specifically how this work achieves multimodal generation in the field of visual text.

b) It would be helpful to report the model size and the inference speed of TextHarmony.
questions: questions: Line 74 ‘a versatile large multimodal’. Do you mean ‘a versatile large multimodal model’ ?
limitations: limitations: See above
paper_summary: paper_summary: This work presents TextHarmony to simultaneously comprehending and generating visual text. The paper performs theoretical and experimental analysis about the performance degradation due to the inherent inconsistency between vision and language modalities. A MoE and LoRA-based module, Slide-LoRA, is then proposed to solve this problem by applying modal-specific and modal-independent LoRA experts dynamically. The experimental results indicate TextHarmony achieves comparable performance to modality-specific fine-tuning results.
strength_weakness: strength_weakness: 1. To my knowledge, there exist other multimodal generative models like DreamLLM[1] and Emu[2]. It would be better to compare with these methods.
2. The authors put related work into the supplementary material. It would be better to briefly summarize the background and representitive works in the main text.

[1] Dong, Runpei, et al. "Dreamllm: Synergistic multimodal comprehension and creation." ICLR2024

[2] Sun, Quan, et al. "Emu: Generative pretraining in multimodality." ICLR2024
questions: questions: 1. Could TextHarmony generate more complex glyphs such as Chinese characters?
2. Have you tried using OCR tools to further polish the captions of DetailedCaps-100K? For example, removing the images whose captions are inconsistent with the OCR results.
limitations: limitations: Please refer to weaknesses.

Comments: title: RE: Sincere Invitation to Participate in the Discussion
Thanks to the Reviewers
Sincere Invitation to Participate in the Discussion
Thanks for the authors' rebuttal
comment: We sincerely appreciate your timely feedback and raising your score. We are committed to incorporating all of the clarifications you suggested in the revised version.
All my concerns have been addressed, so I raise the rating to WA (6)
Dear Reviewers,

Thanks for your valuable comments and suggestions during the review and discussion period. These feedbacks play a vital role in improving the quality of our work.

As the discussion session is coming to an end, please do not hesitate to reach out to us if you have any additional comments or questions. We are more than willing to clarify your concerns. We greatly value and appreciate having this opportunity to discuss with you.

Yours sincerely,

The Authors
We sincerely appreciate your timely feedback and the strong support for our work. We are committed to incorporating all of the clarifications you suggested  in the next version of our paper.
Dear Reviewer SovZ,

We would like to extend our appreciation for your time and comments.  Due to the rush in finalizing the writing, some aspects may cause confusion and misunderstanding. Ensuring that the rebuttal aligns with your suggestions is of utmost importance. We are open to further discussions to clarify any remaining questions or concerns. We would greatly appreciate it if you could consider improving the evaluation after reviewing our responses. 

Thank you very much for your consideration.

Sincerely, 

The Authors
We sincerely appreciate your timely feedback and raising your score.  We would like to clarify our research insights.

Our valuable research insights mainly lie in the word "Harmonizing". In addition to unifying visual text perception, comprehension and generation in a single model, we reveal the issue of modal inconsistency in multimodal generation through comprehensive observation (Emu2 [1], Chameleon [2], MM-Interleaved [3] ) and experiments ( Figure 2 in the manuscript ). The different performance of DreamLLM may be due to the optimization of the model structure and training strategy. We then propose an effective solution, Slide-LoRA, which uses multiple LoRA experts to partially decouple the generative space.  That's why "Harmonizing" rather than "Unifying" is chosen as the title of the work.

We are open to different perspectives and further discussion. Thank you again.


> [1] Generative Multimodal Models are In-Context Learners. CVPR 2024

> [2] Chameleon: Mixed-Modal Early-Fusion Foundation Models. Arxiv 2024

> [3] MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer. Arxiv 2024.
Thanks for carefully considering my comments and adding the human evaluation results. Some of my concerns have been addressed, e.g., the performance comparison and certain unclear arguments. Therefore, I raise my score to 5. 

However, compared with the existing work combining comprehension and generation, this work seems not to provide valuable insights (e.g., the synergetic relations in DreamLLM) in terms of research, even though some applications may be promoted by unifying these two.
Thanks to the clarification, my concerns are nicely addressed. Like I mentioned within the Strengths, TextHarmony is a solid work with reasonable motivation and impressive experimental results, and I am inclined to accept it.
Thank you for the appreciation of our work.
TextHarmony incorporates multimodal generation tasks in visual text, with an effective method called Slide-LoRA proposed towards the modality inconsistency problem. My corncens are well solved. As for the main concerns raised by other reviewers, namely performance comparison to specialized models like AnyText and Monkey, the authors have demonstrated the promising performance of TextHarmony-Align in their Author Rebuttal. Additionally, I agree with their response regarding the comparison baseline. Overall, I believe this work is rather solid and may make a big impact in the field of visual text. I am increasing my rating to 8 consequently.
Thanks for your timely reply. Your concerns are addressed as follows:

**Q3: Did you reproduce them yourselves, and which version of Emu did you use?**

Yes. We reproduce the results of DreamLLM and Emu based on the official repositories. We use the instruction-tuned Emu, i.e., Emu-I for fair comparison, since TextHarmony has also gone through the instruction tuning stage.

**Q4: Was the training data changed from Mario-Laion to Anytext-3m or was it all used?**

The training data was changed from Mario-Laion to Anytext-3m for fair comparison.
I have read the authors' rebuttals and other reviewers' comments, and have a few questions for the authors. The DreamLLM and Emu papers don't include some of the benchmark results in Table E. Did you reproduce them yourselves, and which version of Emu did you use? Besides, in the global rebuttal, TextHarmony-Align outperforms Monkey and Anytext , so was the training data changed from  Mario-Laion to Anytext-3m or was it all used?