Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
Authors: Zhiyang Xu
Minqian Liu
Ying Shen
Joy Rimchala
Jiaxin Zhang
Qifan Wang
Yu Cheng
Lifu Huang
Venue: ICLR 2025 Poster (2025)
Keywords: vision-language generation
interleaved vision-language instruction tuning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent advancements in Vision-Language Models (VLMs) have led to the emergence of Vision-Language Generalists (VLGs) capable of understanding and generating both text and images. However, seamlessly generating an arbitrary sequence of text and images remains a challenging task for the current VLGs. One primary limitation lies in applying a unified architecture and the same set of parameters to simultaneously model discrete text tokens and continuous image features. Recent works attempt to tackle this fundamental problem by introducing modality-aware expert models. However, they employ identical architectures to process both text and images, disregarding the intrinsic inductive biases in these two modalities. In this work, we introduce Modality-Specialized Synergizers (MoSS), a novel design that efficiently optimizes existing unified architectures of VLGs with modality-specialized adaptation layers, i.e., a Convolutional LoRA for modeling the local priors of image patches and a Linear LoRA for processing sequential text. This design enables more effective modeling of modality-specific features while maintaining the strong cross-modal integration gained from pretraining. In addition, to improve the instruction-following capability on interleaved text-and-image generation, we introduce LeafInstruct, the first open-sourced interleaved instruction tuning dataset comprising 184,982 high-quality instances on more than 10 diverse domains. Extensive experiments show that VLGs integrated with MoSS achieve state-of-the-art performance, significantly surpassing baseline VLGs in complex interleaved generation tasks. Furthermore, our method exhibits strong generalizability on different VLGs.

Published as a conference paper at ICLR 2025
MODALITY-SPECIALIZED SYNERGIZERS FOR INTER- LEAVED VISION-LANGUAGE GENERALISTS
Zhiyang Xu"
Minqian Liu
Ying Shen'
Joy Rimchala 2
Jiaxin Zhang
Qifan Wang"
Yu Cheng" 4
Lifu Huang 1,5
Virginia Tech
Intuit Al Research
Meta AI
"The Chinese University of Hong Kong
University of California, Davis
(zhiyangx, mingianliuj@rt.ede edu, Lfuhuangducdaviss.cr
ABSTRACT
Recent advancements in Vision-Language Models (VLMs) have led to the emer- gence of Vision-Language Generalists (VLGs) capable of understanding and gen- erating both text and images. However, seamlessly generating an arbitrary se- quence of text and images remains challenging task for the current VLGs. One primary limitation lies applying unified architecture and the same set of pa- rameters to simultaneously model discrete text tokens and continuous image fea- tures. Recent works attempt to tackle this fundamental problem by introducing modality-aware expert models. However, they employ identical architectures process both text and images, disregarding the intrinsic inductive biases in these two modalities. In this work, we introduce MODALITY-SPECIAIIZE SYNER- GIZERS (MoSS), novel design that efficiently optimizes existing unified archi- tectures of VLGs with tional LoRA for modeling the local priors of image patches and Linear LoRA for processing sequential text. This design enables more effective modeling of modality-specific features while maintaining the strong cross- modal integration gained from pretraining. In addition, to improve the instruction-followiin capa- bility interleaved set-and-image generation, we introduce LEAFINSTRUCT. the first open-sourced interleaved instruction tuning dataset comprising 184,982 high-quality instances more than 10 diverse domains. Extensive experiments show that VLGs integrated with MOSS achieve state-of-the-art performance, sig- nificantly surpassing baseline VLGs in complex interleaved generation tasks. Fur- thermore, our method exhibits strong generalizability on different VI.Gs.
adaptation layers, i.e., a Convolu-
INTRODUCTION
As multimodal learning research advances, there is growing trend of building Vision-Language Generalists (VLGs) (Sun et al., 2023b; 2024; Koh et al., 2023; Aghajanyan et al., 2022: Li et al., 2023b; Dong al., 2024; Team, 2024) that can comprehend and generate interleaved text and im- ages, where multiple text segments and images are presented in arbitrary sequences. Compared with previous LinguanLan Models (VLMs) (Alayrac et al., 2022; Li al.. 2023c; Liu al., 2023c; Fu et al., 2024; Qi et al., 2024b) that can only generate text and diffusion models (Ramesh et al., 2021; Rombach et al., 2022) that can only produce images, such VLGG enable a wider ar- ray of applications that require the simultaneous generation of both images and text, such as script generation (Qi et al., 2024a), visual storytelling (Huang et al., 2016), and many others.
Despite these recent advancements, one notable issue of existing VLGs is that they often fail to pro- duce coherent and high-quality interleaved text and images. As shown in the top example in Figure 1, current state-of-the-ar VLG, e.g., Emu2 (Sun et al., 2024), still suffers from poor text and image quality, including heavy repetition in text and unnatural distortions in the image. We attribute this issue to a fundamental challenge: existing VLGs the same architecture (i.e., transformer back-
"Zhiyang Xu and Minqian Liu contributed equally to this work. We publicly released the code, model checkpoints, and dataset httph:tt///ee http:://ggtttb.co///o. wtg.thubbccom/VTTMMMMMMS...

---

Published as a conference paper at ICLR 2025
Inferior Text and Image Quality
Deseribe the new features of the Intest mountain bike. Context: Last year's 's top sales now has an alaminum model <IMG>. The Slash 9.9 bike has an upgraded drivetrain <IMG>
Instruction:
Output The Slash
<IMG>
9.9 is available
27. S-nchhh and 29-inch
wheel sizes. The Slash 9.2 available in Swinch and 29-inch wheel sizes <IMG>
Weak Iostruction-Followeee Capability
need to clean cloudy glasses. Context: have finished these steps: Acquire soft, clean cloth <IMG>. Use cleaner <IMG> Clean with dish soup and warm water <IMG>, What should I do next?
Instruction:
Output: how elean plasses with eleth and drop of dish soap -IMGs
Frres
CHERRA
Figure 1: Failure cases of existing VLGs (Emu2 at the top and GILL the bottom). The output text with inferior quality is highlighted with underline. The regions that impede output images' quality are highlighted with red bounding boxes.
bone) with the same set of parameters to process both text and images, which may not be sufficient to model the distinct inductive biases in each modality given their intrinsic discrepancy. For ex- ample, text follows linear, left-to-right sequence, whereas images are inherently two-dimensional, composed of local priors in adjacent patches. Previous studies show that the transformer architec- ture (Vaswani et al., 2017) predominantly employed in current LLMs and VLGs excels at sequence modeling. But compared with convolutional architectures, transformer is less effective at modeling local priors of adjacent image patches, which is crucial for various vision tasks (Zhong et al., 2024; Chen et al., 2023d). Thus, applying unified architecture with the same set of parameters can result in poorer performance in mixed-modal generation, such as producing images with local inconsis- tency and distortion among adjacent patches. Recently, several works (Akbari et al., 2023; Ye et al., 2024) have proposed modality-aware expert models an attempt tackle this problem. However, they still apply the same architecture to process both text and images, ignoring the inherent discrep- ancies between these two modalities. These challenges underscore the need for careful architectural design and specialized allocation of model parameters tailored to each modality.
Another critical challenge is existing VLGs (Sun et al., 2023b; 2024) often fail to adhere to human instructions to perform interleaved generation tasks. In the bottom example in Figure 1, GILL (Koh al., 2023) failed to accurately follow the instruction and the context to complete the next step for "clear cloudy glasses". Instead, GILL produces unhelpful text that is repetitive with the input and an image irrelevant to the task. While existing VLGs are often pretrained on interleaved docu- ments (Zhu et al., 2023c), they only instruction-tur
for dingle-modalin generation, e.g., either text or image generation, leading to a weak instruction instruction-followinng capability of interleaved gener- ation. Moreover, there is a lack of large-scale instruction-following data specifically designed for interleaved generation, making the interleaved instruction tuning not scalable and less feasible.
To address these fundamental challenges, first propose PODALIITY-SPCIIAIIZEE SYNERGIZ- ERS (MoSS), a novel framework that introduces modality-speciilized parameters to seamlessly handle the inductive biases of different modalities within the unified architectures of VLGs. As lightweight adaptation layers, our proposed MOSS is generic and can be integrated into most, if not all, existing VLGs without requiring expensive pre-training. Specifically, for images, we introduce Convolutional Low-Rank Adaptation (Convolutional LoRA) layers to better model the local prior of image patches. For text, we employ a separate set Linear Low-Rank Adaptation (LoRA) lay- ers, acknowledging the distinct sequential modeling process of text compared to images. During finetuning. both modality-specialzze architectures are zero-initialized and progressively fine-tuned to learn their modality features, while the VLG's parameters remain frozen, maintaining strong cross- modal integration gained from massive pre- training. Our design allows each modality to have better representation of modality-specine features with its own specialized parameters and optimal architectural design. Additionally, to improve VLG's Instruction-followin capabilities un- der diverse interleaved generation scenarios, we introduce LEAFINSTRUCT, the first open-sourced
2

---

Published as a conference paper at ICLR 2025
high-quality interleaved instruction tuning data with 184,982 instances spanning more than 10 do- mains. To obtain high-quality instruction data at scale, we develop a rigorous automatic pipeline.
To validate the effectiveness and generalizability of our method and dataset, adopt our method on two different VLG backbones with discrete and continuous image token spaces, and conduct exten- sive experiments on multiple datasets. The results demonstrate that the VLGs instruction our method achieves state-of-the-ar performance across most evaluation aspects. Particularly, our method can produce interleaved content with better quality, including text quality, image coherence, text-image consistency, and helpfulness. In summary, our contributions are threefold. First, we introduce MoSS, novel design that enhances VLGs to generate interleaved content with modality- specialized parameters and adaptation architectures. To the best of our knowledge, we are the first to apply different adaptation architectures within an autoregressive generative model to improve interleaved generation. Second, to fill the blank in existing resources and improve the instruction- following capability of VLGGs, we introduce the first open-sourced large-scale instruction-tuning dataset across diverse domains. Third, by instruction-tuning existing VLGs with small num- ber of parameters, we achieve significant performance improvement on most aspects of evaluation benchmarks, outperforming existing open-source baselines by 34.7% InterleavedBench. We also demonstrate that our approach can effectively generalize to different VLG backbones.
with
RELATED WORK
Interleaved Vision-Language Models There are two popular formulations for VLGs: The first leverages VQGAN (Esser et al., 2021) to quantize an image into a long sequence of discrete tokens and add the vocabulary in VQGAN's codebook into the vocabulary of LLMs (Aghajanyan et al., 2022: Yu et al.. 2023; Yasunaga et al., 2023; Team, 2024; Jin et al., 2023). In this way, the LLMs are trained with unified autoregressive objective to predict image tokens or text tokens. The predicted image tokens are fed into VQGAN decoder to reconstruct images. The second formulation em- ploys the CLIP image encoder to transform images into sequences of continuous embeddings (Koh et al., 2023; Tang et al., 2023; Zhu et al., 2023b; Sun et al., 2023b; 2024; Li et al., 2024b; Wu al., 2023; Tian et al., 2024), which are then concatenated with text embeddings in their original order. Compared to the first approach, this formulation often requires shorter sequences to represent an im- age and generally yields superior performance. Our proposed method requires minimal assumptions on VLG's architectures and can be applied to many of the existing transformer-base VLGs.
Visual Instruction Tuning Xu et al. (2023) propose Multilnstruct, the first human-label visual instruction tuning dataset to improve the generalizability of VLMs. L.LVA (Liu et al., 2023c) leverages GPT-4 to convert image captions from existing annotations into three tasks, including visual dialogues, visual question answering, and detail captions. Following studies either utilize proprietary LLMs (Dai et al., 2023; Ye al., 2023; Yin al., 2023; Liu et al., 2023b; Li et al., 2023a; Lyu et al., 2023; Zhu et al., 2023a; Wang et al., 2023; Chen et al., 2023b; Zhang et al., 2024) or human efforts (Liu et al., 2023b; Xu et al., 2024) to augment visual instruction tuning tasks. Several studies target specific aspects of VLMs' capability, such as domain and instruction bias (Avrahami al., 2022; Liu et al., 2023a), object grounding (Chen et al., 2023a), and OCR (Zhang et al., 2023b; Hu et al., 2023). Instruction tuning has also been widely applied to other vision-language tasks, such as image editing (Brooks et al., 2023a) and interlcaved text-image understanding (Jiang et al., 2024). Hu et al. (2024) finetune a model that can follow multimodal instructions to generate desired images. However, most existing instruction-tune -tuing datasets only consider the tasks where the outputs are in a single modality, i.e., either text or image. To facilitate the training and enhance the instruction- dataset
following capabilitiess for VLGs, we curated LEAFINSTRUCT, the first instruction-tume tailored for interleaved text-image generation across diverse domains, where the inputs and oulputs can contain interleaved text and multiple images.
Parameter-Enie Efficient Finetuning (PEFT) PEFT methods (Hu et al., 2021; Li & Liang, 2021; Karimi Mahabadi et al., 2021; Zaken et al., 2022; Jia et al., 2022; Lian et al., 2022; Jie & Deng, 2022; Liu & Huang, 2023; Liu et al., 2022; Chen et al., 2023d; Zhong et al., 2024) aim to adapt pre- trained large models to various downstream tasks and have become prevalent in instruction tuning. Typically, these methods involve freezing the pretrained large models while finetuning a minimal set of newly introduced parameters. Recent studies (Wang et al., 2022; Zadouri et al., 2023; Lin et al., 2024: Shen et al., 2024) propose to combine PEFT methods with Mfxture-of-Expeee task interference and enhance performance, particularly in visual instruction tuning where models need to process inputs from two modalities. Our proposed MOSS is the first PEFT method that
mitigate
3

---

Published as a conference paper at ICLR 2025
Token
Image
Image Encoder
dog grass
Iuput Image(s)
2D Reshape Padding
Comuggx
Linear Layer
Linear Lora
ConvL.ora
Iapur Image Features with Padding I (M13) Rys6.
Output Image
Features E gellWrr
dog gruss
Flatten
Image Decoder
Output Image(s)
An autoregressive VLG with our proposed MoSS added to its linear layers. The linear LoRA the left side is specialized to generate text tokens and the Convolutional LoRA the right side is specialized to generate image patches. On the right handside, we show the details of convolutional operation applied autoregressively generate image tokens. Best viewed in color.
Figure 2:
utilizes two distinct LoRA architectures -ineee and comvolutional-for text and image generation within autoregressive VLGs.
BACKGROUND: AUTOREGRESSIVE VISION-LANGUAG GENERALISTS
Existing autoregressive VLGs can be broadly classified into two categories: those that represent each image sequence of discrete tokens (Yasunaga et al., 2023; Aghajanyan et al., 2022; Team, 2024), and those that represent each image as a sequence of continuous vectors (Sun et al., 2023b; 2024). However, despite these differences image representation, their underlying model architectures and formulations for vision-language generation remain largely similar. Thus, we do not differentiate them in the following formulation.
Model Architecture Autoregressive VLGs typically comprise three components: image en- coder (e.g., CLIP (Sun al., 2023a) or VQ-VAE encoder (Gafni et al., 2022)), a decoder-only large language model (LLM), and image decoder (e.g.. diffusion model (Podell et al., 2023) VQ- VAE decoder (Gafni et al., 2022)). Given a sequence of interleaved text segments and images, image encoder processes each image into sequence of image tokens. These image tokens then concatenated with the text tokens in their original order and input into the LLM. The LLM autore- gressively predicts the next token, which could be either text image. Finally, the image decoder takes in the predicted image tokens and reconstructs the target image.
Training Objective The training objective of VLGs can be loosely defined in the following unified autoregressive manner.
N
arg I1aX
Σ
Po $1,8,,,-00 S2, Sn-1)
(I)
where & denotes the model parameters, N denotes the input sequence length, 'D denotes the training dataset, and s; denotes text token or an mmage-patch embedding. This unified objective is optimized through two types of losses: (1) If the image is represented discrete tokens, the CrossEntropy loss employed to minimize the divergence between the predicted probability distribution of the image or text tokens and the ground truth distribution;
(2) If the image is encoded as continuous vectors, the mean- seraan--uuuedd-ere -eror (MSE) loss is used to minimize the difference between the predicted and actual image embeddings.
4

---

Published as a conference paper at ICLR 2025
4 MODALTY-SPECIALIZER SYNERGIZERS (MoSS)
In this section, we first detail the two modality--peciilzzed adaptations in MoSS: Linear LoRA for text tokens and Convolutional LoRA for image tokens. We then describe the process of synergisti- cally integrating these adaptations into autoregressive VLGs to perform interleaved generation.
4.1 LINEAR LOW-RANK ADAPTATION (LoRA)
LoRA (Hu et al., 2021) a parameter-efficint finetuning method that freezes the pretrained model parameters and injects low-rank decomposable matrices into the layers of transformers, Formally, given the weights in linear layer W Rdoueedin LoRA modifies the weights by adding decom- posable weight matrix AW to W. Thus, for a vector R""" the modified linear transformation T
Rd Rd becomes:
(h) = h(W + AW) -W + hAW!
(2)
A W is decomposed into two low-rank matrices, i.e., LORA A: Wa R rxd:.. and LoRA B:
W B Rouurrr satisfying the low-rank constraint r < min(dout: d.a). The final expression is
T(h) = hwT + +ohWl W
where a IR a hyper-parameter.
4.2 CONVOLUUIIONAL LOW-RANK ADAPTATION (CONVOLUTTIONAL. LORA)
We propose Convolutional LoRA, variant of LoRA specifically designed for modeling the local structure of image hidden states during image generation, by improving the architecture proposed in Zhong et al. (2024). Detailed empirical comparison between two Convolutional LoRAs be found in Table 10 in Appendix C.3. The previous approach first reduces the dimension of input features and then performs the convolution operation within lower-dimension space. Since dimension reduction can cause information loss, the convolution within a reduced dimension can be less effective at modeling the local priors of image patches. On the contrary, our method performs convolution in the original input feature space and the dimension is deducted during the convolution process, which alleviates the information loss issue in the previous design.
Specifically, our approach consists of convolutional LoRA A layer, i.e., ConV where the kernel size is & k the number of input channels is Cin and the number of output channels is r, as well as
LoRA B: Wn E X Given the 2D feature E IR #xW*X1.. of an image, where H denotes the height, W denotes the width, and C₁,, denotes the number of channels of I, the convolutional LoRA projects down its number of channels to and simultaneously performs convolution operation. Then the LoRA B projects its number of channels up to Cout. The equation becomes:
Í(I)= IWT + (oConvexa (I)W]
(4)
where a is paparera
4.3 INTEGRATING MOSS INTO VLGS
As shown in Figure 2, we propose to integrate two types of adaptations into VLGs, i.e., ing Linear LoRA for text generation and Convolutional LoRA for image generation. For- mally, let W E I2 x be the weights of any linear layer in a LLM, and let H = [hi..., hi f b bin......, (HxW)) h E R Nxd_a denotes the hidden states of se- quence of interleaved text and images, where a subscript indicates position a hidden state and the superscript indicate if hidden state is decoded into text token (t) or decoded into an image-patch embedding (i). We untie H into text hidden states H' [hh h h h ( hly and image hidden states H' [[h h Hx)] J. [ ...., [ where m + 1 and 1 denote the starting positions of two subsequences of image hidden states. Each subsequence
of single image has a fixed length of Hl x W and we reshape the hidden states of each image in H' into 2D structure. Hence, the dimension of H' becomes B X H X W x Cans where B denotes
5

---

Published as a conference paper at ICLR 2025
the number of images in the sequence H. We feed H' into the Equation 3 to get H' T(H') and H' into Equation 4 to get H' (H').
It is non-trivial to integrate convolutional operation in auto-regressive model and to the best of our knowledge, we are the first incorporate the convolutional architecture improve interleaved generation. The right part of Figure 2 visualizes the convolutional operation applied to a sequence of image patches. The squares on the left denote the reshaped 2-dimensional input image patches and the larger blue squares denote the 2 2x2 convolution kernels. The number on each square denotes the original positions of patch in the image sequence. For demonstration purposes, we draw image patches with H 3 and W 3. Note that the current hidden state of an image patch can only depend on previous hidden states since we use the autoregressive architecture. Thus, when applying the convolution operation and left sides of a patch. For example, the new hidden state of patch 9 is computed from patches: 5, 6, 8, and 9. To preserve the shape (H x W) of the input image patches, hidden states with zero vectors on the top and left sides, as shown by the grey squares in Figure 2. Finally, we assemble H' and H' back to their original sequence to form H.
image patch, the kernel only covers neighboring patches on the top
pad the reshaped image
5 INTERLEAVED INSTRUCTION TUNING WITH LEAFINSTRUCT
Existing interleaved vision-language models (Sun et al., 2023b; 2024; Dong et al., 2024) predom- inantly follow the training procedures that they are first pretrained on massive corpora of inter- leaved data such as MMC4 (Zhu et al., 2023c) and other resources and then finetuned on mix of high-quality datasets, such as visual instruction tuning data in Liu et al. (2023c) and Instruct- Pix2Pix (Brooks et al., 2023b). However, one significant limitation of these instruction-tune -tuning datasets that the outputs are typically a single modality, e.g., either text or image, which hinders capability of VLGs especially in generating interleaved text and images
the instruction-followe specified by the given instructions.
5.1 DATASET: LEAFINSTRUCT
To bridge the gap between limited existing resources and the practical need for improving interleaved generation models, we curated LEAFINSTRUCT, the first comprehensive instruction tuning dataset for interleaved text-and-image generation. Each instance in dataset consists of (1) detailed instruction, (2) an input context with interleaved text and images, and (3) ground-truth output also with interleaved text and images. We show an example of LeafInstruct in Figure 6, and compare with other representative instruction-based datasets.
Dataset construction We construct a diverse cnstruction-tuning data collection from large-scale web resources and academic datasets, including MMDialog (Feng et al., 2023), VIST (Huang et al., 2016), WikiWeb2M (Burns et al., 2023) and YouCook2 (Zhou et al., 2018). Since the original data sources can be noisy, we meticulously devised automatic data annotation pipeline to ensure the high quality of our curated data. include the details on dataset construction Appendix B. We also conducted rigorous human assessment of our dataset (see Section 7).
Dataset Statistics After applying our rigorous data processing pipeline, we totally obtain 184,982 high-quality instances out of more than 7 million source samples. Our dataset covers wide range of realistic instruction-tuning tasks, including multimodal document completion, multimodal dia- logue, visual storytelling, multimodal script generation, and knowledge-intensive generation. In Appendix B, we show the domain distribution of LEAFINSTRUCT in Figure 7 and compare our dataset with existing datasets Table 4. These analyses effectively demonstrate the diversity and the novelty of our dataset.
5.2 INSTRUCTION TUNING FOR INTERLEAVED GENERATION
With our curated LEAFINSTRUCT, we enable large-scale interleaved instruction tuning so that the model can learn how to follow human instructions to generate desired interleaved text and images. To preserve the VLG's capability obtained from pre-training, we only fine-tuned the modality- specialized adaptation layers, and the remaining parameters in the VL.Gs kept frozen.

---

Published as a conference paper at ICLR 2025
Specifically, shown on the right side of Figure 6, given the task instruction and the interleaved context as imputs, the model is trained to autoregressively generate interleaved text tokens and images with two alternative generation modes for text and images, respectively. We use special token <IMG> to indicate where an image occurs in the interleaved sequence. The training process is as follows: (1) The model is set to the text generation mode by default. During this mode, the hidden states of newly generated tokens are always routed to linear LoRA, and only the parameters in the linear LoRA are optimized. (2) After the <IMG> token is generated, the model switches to image generation mode. The VLG takes in the updated context ended with <IMG> and is trained to generate a fixed-length (H x W) sequence of image patch embeddings autoregressively. All the hidden states of generated image embeddings are routed to Convolutional LoRA and only the parameters in the Convolutional LoRA are fine-tuned. (3) When the generation of an image finished, the model is trained to predict an end-of-imaae token </IMG>, and the model will resume the text generation mode. This process will be iterated until the training on a sequence is finished.
Interleaved Inference The inference procedure of our framework is largely identical the in- struction tuning, where two generation processes iterate alternatively. The only key difference is that the fine-tuned VLGG will automatically determine when to generate text segment an image at their own discretion. The iterative generation process terminates when the model produces the -od-o--generation token 
 at the end of a response. Note that although our inference process is designed for interleaved generation, we can also handle the cases where the outputs only contain text or images, enabling a wide range of applications.
EXPERIMENTS
6.1 EXPERIMENT SETUP
Evaluation Benchmarks We evaluate the interleaved generation capability of our method on In- terceavedBench (Liu et al., 2024b). InterleavedBench is a comprehensive dataset specifically tai- lored for interleaved evaluation. InterleavedBench covers a diverse array of tasks, where the eval- uation data are either curated by the authors (e.g., document completion), or re-annotated based on subsets of well-established academic evaluation benchmarks, including visual storytelling from VIST (Huang et al., 2016). activity generation from ActivityNet (Krishna et al., 2017), script gener- ation from WikiHow (Yang et al., 2021), image editing from MagicBrush (Zhang et al., 2023a), and multi-concept image composition from CustomDiffusion (Kumari et al., 2023). We include more details on evaluation benchmarks in Appendix C.I.
Evaluation Metries We adopt InterleavedEval (Liu et al., 2024b). a strong reference-free evalua- tion metric to conduct a holistic assessment of the quality of interleaved generation. InterleavedEval prompts GPT-4o to score an interleaved output from five aspects, including Text Quality, Percep- tual Quality, Image Coherence, Text-Image Coherence (TIC), and Helpfulness. For each aspect, the GPT-4o outputs a discrete score from (0, 1, 2, 3, 4, 5), where 0 is the worst and 5 is the best. We re- fer to the original paper (Liu et al., 2024b) for detailed definition of each score and each evaluation aspect. We also have an additional evaluation on image editing on the full test set of MagicBrush using well-established metrics, including CLIPScore (Hessel et al., 2021) and DINO (Caron et al., 2021) in Table 5 in Appendix C.2.
Implementation Details To demonstrate the generalizability of our method, we adopt our MoSS to two representative autoregressive VLG backbones, i.e., Emu2 (Sun et al., 2024) and Chamelcon (Team, 2024), and fine-tune them on our LEAFINSTRUCT dataset. The rank number of all the LoRA is set to 256 by default. Note that for the Chameleon model, we adopt the implementa- tion in Chern et al. (2024) since the original model and checkpoints are not publicly available. More implementation details including hyperparameters and GPU setups can be found in Appendix A.
Baselines For fair comparisons, we primarily compare our methods with current -tate-of-the-art open-source VLGs, including GILL (Koh et al., 2023), MiniGPT-5 (Zheng et al., 2023a), Pre- trained Emu2, and Chameleon. We also report the performance of pipelines based on proprictary models, including Gemini 1.5 (Reid et al., 2024)+SDXL (Podell et al., 2023) and GPT-4o (Ope- nAI, 2024)+DALLE 3 (Betker et al.). For these baselines. first prompt the VLMs (e.g., GPT-4o) to generate text along with image captions, and then feed the image captions to a separate image generation model (e.g., DALLE). We report these performances only for reference purposes.
7

---

Published as a conference paper at ICLR 2025
Table 1: Main results of interleaved generation on InterleavedBench. We show the performance of pipelines based on proprietary models (Top), open-source VLGs (Middle), and the VLGs trained with our MoSS and LEAFINSTRUCT (Bottom), respectively. Note that the scale is from 0 to 5 (5 is the best). We also report the percentage of improvement in our method over the original VLG backbone in the parentheses. The best results are highlighted in bold.
Model
Text Quality
Perceptual Quality
Image Coherence
TIC
Helpfulness
Proprietary Models
Geminil.5 SDXL
3.37
4.34
3.34
3.98
3.28
GPT-40 DALLE
3.16
4.44
3,13
4.39
3.46
Open-Source Models
MiniGPT-5
1.31
3.44
2.06
2.66
1.76
GILL
1.44
4.02
2.12
2.69
1.53
Emu2
1.33
2.29
1.71
1.22
1.87
Chumeleon
3.33
0.67
0.28
0.47
1.43
Kmu2 + MOSS (Ours)
2.61 (+96.2%)
3.62 (+58, 19%)
3.41 (499.4%%
3.54 (+190.2%)
2.71 (+44.,9%)
Chameloon MOSS (Ours)
2.98 (-10.5%)
2.25 (+235.8%)
1.05 (+275%)
1.7 (+261.7%)
1.82 (+27.39%)
6.2
MAIN RESULTS
Quantitative Results Table presents the main results of our method in comparison to the base- lines. We have the following findings. Firstly, our approach is highly effective and efficient when it is adapted to existing VLGGs. Applying over their original performance on all evaluation aspects. For example, compared with the original Emu2 model. Emu2+MoSS achieved a performance gain of up 190.2% (on Text-Image Coher- ence) and 97.76% on the average of 5 aspects, almost doubling the overall performance. Secondly, our method beats the previous open-sourced state-of-the-art (i.e., GILL) by a large margin, i.e., 44.7% on the average of 5 aspects. Particularly, the outputs of our method have better coherence across images (w/ 37.8% improvement in Image Coherence) and between text and images (w/ 31.6% improvement in Text-Image Coherence). Our method also exhibits better mnstruction-folloviii ca- pability and is able to generate more helpful content given the 11.5% improvement in Helpfulness. Thirdly, it is worth noting that the Chameleon baseline achieves good performance on Text Quality but extremely poor performance on image-related aspects. We observed that Chameleon usually generates long and comprehensive text responses with no image output, thus leading to poor perfor- mance on image-related aspects. We hypothesize the reason lies in the lack of instruction tuning on interleaved generation with both text and images. From Table our approach improves the original Chameleon by a significant margin, especially on image-related aspects. This shows that our inter- leaved instruction tuning can effectively enhance a VLG that was previously poor at mixed-modal generation. We noticed that adding MOSS in Chameleon can cause a slight performance drop in text quality. We discuss the details of this problem in Appendix C.3, conduct an additional human evaluation of text quality, and present the results and findings in Table 11. Fourthly, there remains notable gap between open-sourced VLGs and the pipeline approaches based on proprietary models, indicating building powerful and general-purpose open-sourced VLGs is still challenging.
MoSS to VLGs achieved significant improvement
Per-task Performance We also show the average performance each task on InterleavedBench in Figure 3. Specifically, our method (i.c., Emu2- MoSS) outperforms the baselines most tasks, of- ten by a large margin. For subject-driven genera- tion, the slightly lower performance of our approach compared to other baselines is due to its poorer per- ceptual quality. We included more detailed justifi- cations for this result in the following section. We report the per-task performance on all aspects in Fig- ure 9 in Appendix C.5. In addition, we include more results on well-established benchmarks of image- understanding, text-to-image generation, and image- editing in Table 5, 6, 7, 8, and in Appendix C.2.
Activity Generation
U1A
Subject-driven
Multimodal Dacument Generation
Generation
Image Editing
Completion
Visul Story
Mukiiooda Scricc Ceneration
Figure 3: Per-task performance averaged on 5 aspects on InterleavedBench. sourced baselines and our MOSS in Figure 4. Our findings are follows. Firstly, method
Qualitative Results To better interpret the results, we conducted a qualitative analysis on several open-
8

---

Published as a conference paper at ICLR 2025
Input
MiniGPT-S
GILL
Emu2
EmuZ MoSS (Ours)
Instruction: Complete travel story. Context visited the countryside the
Output: excited viail the country side the weekend -IMGs was excited country side the weekend.
countryside und saw many beautiful flowrs -IMG>
Output:
Output: <IMG>
Output: the gardens
-IMG-
many besutiful
buildings <IMG>
Instruction: Deseribe teants in Univenity Rover Challenge. Context: During the challeage, the PCZ Team placed the socond -IMGS
Output: The team rover is an autenemous valiele of the moon. <IMG>
Output: Describo a team's suceess in Univensty Challenge <IMG>
Output: 16 souns from countries compered in the University Rover Challinge. <MGG>
Output: -IMG>
Instruction: Tell should plan vegan healthy
Output: you eut becakfist make to have dinser <IMGs
mials. Context: I've done: I had
Output: the tofls semmble -IMG
Output Eat vegi cerval with non-dairy milk for quick breakfas:
Output good vegan meal for lunch <IMG>
outeeal with fruit <IMG>
tots
moming <TMG
Instruction: Heepp modify picture <IMG> change the bus into train.
Output: <IMG>
bus the HIMG>
Outpat:
Output: The trick <IMG>
Figure 4: Qualitative results of MOSS based on Emu2 and open-source baselines. The <IMG> tokens denote the images' positions in the interleaved sequences.
demonstrates better helpfulness and instruction-following capabilities. For example, in the first row in Figure 4, method generates more coherent visual story with more diverse content given the input. In the third row, our approach provides more natural and reasonable next step for the user, i.e., have good vegan lunch, while other baselines either jump dinner (MiniGPT-5) or stick to breakfast (Fmu2). Secondly. we observed that poor text quality is common issue for many baselines. For instance, MiniGPT-5 often fails to generate explanatory text while GILL usually generates short caption, c.g., the gardens in the first row and the tofu scramble in the third row, instead generating useful content to solve the task. Thirdly, neither GILL nor MiniGPT-5 preserve the visual appearance of the entities and scenes in the input images. Our approach, on the contrary, faithfully retains most visual characteristics, leading to significantly better Image Coherence. Finally, shown in the fourth composition, although MiniGPT-5 and GILL can sometimes generate images with better perceptual quality, the image contents often irrelevant to the task, ignoring input instructions and context. In contrast, our method strives to adhere to instructions and can better condition its generation on the provided image. Due to the complexity of the task, our model may produce images with lower perceptual quality and noticeable distortions. However, when taking Helpfulness into account, the images generated by our model can be considered as the better ones compared with the baselines. We present additional qualitative results of Chameleon MoSS in Figure 8 in Appendix C.4.
in Figure 4, for tasks such image editing image
7 DISCUSSIONS
Comparison between MOSS and other PEFT Methods To directly validate the performance improvement brought by our proposed MoSS, fine-tuned Emu2 using (1) traditional linear LoRA (Hu et al., 2021) and (2) Mixture-of- (MoE) LoRA (Shen et al., 2024), with the re- sults presented in Table 2. In traditional linear LoRA, text and images share the same low-rank

---

Published as a conference paper at ICLR 2025
Table Comparison between MOSS and existing PEFT methods, i.c., traditional linear LoRA, and Mixture-of- (MoE) LoRA. fixture-of-EEppe LoRA uses two different sets of linear LoRA for images and text, respectively. The rank number set to 256 for all methods in this table.
Model
Text Quality
Image Quality
Image Coherence
TIC
Helpfulness
Emu2
1,33
2.29
1.71
1.22
1.87
LoRA
1.77
2.38
1.99
2.04
1.64
MoB-LoRA
1.98
3.28
2.66
2.62
2.01
MOSS (Ours)
2.61
3.62
3.41
3.54
2.71
adaptation parameters, while in MoF-LoRA, two different sets of linear LoRA are used for images and text respectively. The routing strategy in MoE-LoRA is based on the output modality of each hidden state, i.e., whether the hidden state is used generate text image. From Table 2, effectively verify the benefits of using separate parameters for image and text. MoSS significantly outperforms the MoE-LoRA across all aspects, especially the image-related aspects such as Image Coherence and Text-Image Coherence (TIC). The conclusions from the results are two-fold. First, it shows that introducing interleaved text-and-image generation. Second, it verifies that convolutional LoRA can improve image generation by better modeling the local priors of images. Additionally, we compare the com- putational cost of using MoSS and LoRA in Appendix C.3.
architecture and parameters can effectively improve
Effect of Rank Number To investigate how the number of rank can affect the performance, we show the performance averaged on 5 aspects on InterleavedBench with the rank number equals (32, 64, 128, 256) comparing LoRA, MoE-LoRA, and our MoSS in Figure 5. Our approach consis- tently outperforms LoRA and MoE-LoRA across all rank numbers, and as the rank number increases, the gap between MoSS and previous methods consis- tently grows larger. This proves the effectiveness and generalizability of MoSS across different rank sizes. Based on this experiment, we set the rank number of our approach to 256 by default. We in- clude more results on the effect of rank in LoRA in Table 13, 14, and 15 in Appendix C.6.
LoRA
3.0
MoE-LoRA
MoSS
Average
2.0
1.5
32
128
256
Rank
Figure 5: Performance averaged on aspects with different rank numbers.
To verify our LEAFINSTRUCT dataset is of high quality, we conduct a rigorous human evaluation using the multi-aspect evaluation criteria in InterleavedE-- val (Liu et al., 2024b). Specifically, we use scale of ( to 3 in the evaluation, where O is the lowest score while 3 the highest. We randomly sampled 200 instances from LEAFINSTRUCT and asked two human annotators with expertise in NLP and multimodal research to rate each instance from 5 aspects. We report the averaged scores from two annotators in Table 3. We show that the sampled in- stances consistently achieved almost full scores across all 5 aspects, which effectively demonstrated that our curated dataset is high quality.
Quality Assessment of INSRRCT
Table 3: Human evalution of randomly sampled instances from LeafInstruct. Note that the scale is from 0 to 3 (Score 3 is the best), which is different from the scale used in Table 1 and Table 2.
Text Quality
Perceptual Quality
Image Coherence
TIC
Helpfulness
Score
2.89
2.96
2.77
2.87
2.71
8 CONCLUSION
We propose PEDALIIT-SPECIALIIIE SYNERGIZERS (MoSS), novel modality adaptation framework tailored for VLGs. MOSS dedicates a set of linear LoRA for processing text and set of Convolutional LoRA for images, allowing each modality to have its own optimal adaptation design. Besides, we propose the first interleaved instruction tuning dataset LEAFIN- STRUCT and verify the dataset quality via rigorous human evaluation. Extensive experiments InterleavedBench showcase that our proposed method and dataset are highly effective, establishing the new of-t--of-hheart among open-sourced VLGS in interleaved text-and-image generation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research is partially supported by research award from Intuit AI Research and the award No. #2238940 from the Faculty Early Career Development Program (CAREER) of the National Science Foundation (NSF). The views and conclusions contained herein are those of the authors and should be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin. Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A causal masked multimodal model of the internet. CoRR, hos/2201.075222 2022. URL https:
//arxiv. org/ /aa///b//22// 07520. 3,4,
Hassan Akbari, Dan Kondratyuk, Yin Cui, Rachel Hornung. Huisheng Wang, and Hartwig Adam. Alternating gradient descent and mixpure-of-experts for integrated multimodal perception. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:// 2 Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Mari- anne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zis- serman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. 2022. URL http://papers. paappers..iiip www.niiippsaaaaaee cwtpcccr_pppiiieeeeeeeeeeeeeeeee faaperr_flles fiw.fiell//eeee.. htse/pppr//22/// / 38001772..7bf0117ccccc444243
openreview
html.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 18187-18197 IEEE, 2022. doi: 10.1109 CVVR52668.2022.01172 URL https:///o/ org/10. RAL.009/VVPPR22880 2022 01767. 3
James Betker, Gabriel Goh, Li Jing, + Tiimrroos, Jianfeng Wang, Linjie Li, t LongOuyang, JuntangZhuang, i JoyceLee, & YufeiGuo, WesamManassra, PrafullaDhariwal, I CaseyChu, YunxinJiao, and Aditya Ramesh. Improving image generation with better captions. URL https:
//api. i escholar ar prrg/Corpusllii 264403242.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. IEEF/CVE Conference on Computer Vision and Puttern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 18392-18402. IEEF, 2023a. doi: 10. WWWVCVPR52722.002.0000
URL httph:t////// //tps::///o...
111.:/11111//1////1100 2023. 01764.
3, 20
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning follow image editing instructions. In CVPR, 2023b. 6, 20
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, and Mandy Guo. Wikiweh2m: page-level multimodal wikipedia dataset. CoRR, doi: 1048550/ARX1V22355055333 48550/ arXiv 2305 05432.
abs/2305.05433 05432, 2023.
URL attps://dol http////oo.oo//...../ org/10.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 9650-9660, 2021. 7
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 11m's 's referential dialogue magic. CoRR, abs/2306. 15195, 2023a. doi: 10.48550/ ARXIV.2306.155 1.1...55999. 15195. URL https: //doi. org/10..or 48550/ arr.vv 2306. 15195.

---

Published as a conference paper at ICLR 2025
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. CoRR,
abs/2311.127993 2023b. doi: 1048500/ARX1.331.22299 URL https://doi.onn 48550/ame arXiv. 2311.12793.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming- Wei Chang, and William W. Cohen. Subject-driven text-to-image generation via apprenticeship learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023c. URL http:/ /ttp:://paaaa http://papers pwwprsppppssesee nips wc/paper_pppeeeeeceeeeeee saarr_fieespppppp wateee/pppe///eeeee paaper/2022 22t3//a22/////////// 111.1111542211222744440888888880
Neu-
httpppaeeeeeeee//aaeppppppaaaeea Abstract Conf erence. html. 20
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023d. URL 2,
net wttp://ewweeeeeeeeeaadddeddee..e
Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. ANOLE: an open, autoregressive, native large multimodal models for interleaved image-text generation. CoRR, abs/2407 006135,...6,, 2024. doi: 00448550/RRIIV.24077 066335. URL https://doi org/10. 48550/ arriv 2407. 06135.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. CoRR, abs/2305.06500 2023. doi: 10.48550/RRXI. 2305.06500. URL https:///dot http//iioorrgr///o//////oo org/ 48544/5555/111 2305. 06500. 3
Rumpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. DreamL.LM: Synergistic multimodal comprehension and creation. In The Twelfth International Conference on Learning Representations, 2024. URL https: / ///p//ppeeeeeeeeeeeeee net/ f fd
y01KGvd9Bw. 1, 6
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high- resolution image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2021, virtual, June 19.25, 2021, pp. 12873-12883. Computer Vi- sion Foundation / IEFE, 2021. doi: 10.1109/CVPR46437.220 .01268.
URL https: //openaccess. thecff com/ wow.teen///VPP222//tm//////eesee Taming. Transformers. for MiigH Resolution
for_High
html.
Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang, Chongyang Tao, Dongyan Zhao, and Qingwei Lin. MMDialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6l Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7348-7363, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. 118653//1/222..acl-aaeo acl acl-long.400 405. 6
URL https: //attp:://aaaaat http://aaaaaaooogyyoyyoo org/2023.
Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollic Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. Isobench: Benchmarking multimodal foundation mod- els on isomorphic representations. In First Conference on Language Modeling, 2024. URL wttp:///ooooom//////d//
https://openrevieu. http///errrviee.eeeettrrrmm
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision
- ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV. volume 13675 of Lecture Notes in Computer Science, pp. 89-889--66 Springer, 2022. doi: 001- 0_6.
10.1007/978- 5--31-199844 6. URL https: .rr...og/gooo. 2tgg/10.1007/900
12

---

Published as a conference paper at ICLR 2025
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song. Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi multi-granulariin comprehension and generation. arXiv preprint arXhv:2404.14399 2024. 22
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024. 22
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP, 2021. 20
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021. URL https: / arxiv. org/ abs hetg/abs/21000 /2066 09685. 5,
Hexiang Hu, Kelvin C. K. Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William W. Cohen, Ming-We Wei Chang, and Xuhui Jia. Instruct- imagen: Image generation with multi-modal instruction. CoRR, abs/2401.01952, 2024. doi: (0-4550/ARXXIV240000022 URL https: /ttps:://oii. www.oii.rgg..... wtt.ooo.//.///////...///.../// 485500/55arxii art.v.riiii/..////// 240. 01952. 3,
20
Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. BLIVA: A simple multi- modal LLM for better handling of text-rich visual questions. CoRR, abs/2308 abs/2308.09936. 2023, doi: 10.48550/ARXIV22308 09936. URL https: //doi org/10. 48550/ arXiv 2308. 09936.999.
Ting-Hau Hao (Kenneth) Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross B. Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. Visual storytelling. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), NAACL HLT 2016. The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pp- 1233-1239) The Association for Computational Linguistics. 2016. doi: 10.18653/V1/N 1147. URL https: //dos. org/10..00./... 11w.3//1//16-111111
1, 6,
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariha- ran, and Ser-Nam Lim. Visual prompt tuning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 17th European Conference. Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXXIII, vol- ume 1 13693 of Lecture Notes in Computer Science, pp. 709-727. Springer, 2022. doi: 10.1007/ 978-3-031- 1980-19927-444 41. URL https: org/ 10. 1007 3- 19827-49444444111
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. MANTIS: interleaved multi-image instruction tuning. CoRR, abs/2405.01483. 2024. doi: 10.48550/ARXIV. 2405.01483. URL https:tt//// / http//oii.o/gg////////// org/10 485500/rXiiv 2405. . 01483. 3, 20
Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. CoRR, 2otp/2207.0739, 2022. doi: 10.48550/ARXIV22077.73333 URL https://doi.
10. 21.44855//rr/0 arxvv 2207. 07039. 3
Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, Di Zhang, Wenwu Ou, Kun Gai, and Yadong Mu. Unified language-vision pretraining LLM with dynamic discrete visual tok- enization. CoRR, abs/2309 04666, 2023. doi: 10.48550/ARXXI2330900666 URL https: //doi. //doi.org/10. 11.10.4550/880 arXiv 2309. 04669.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances Neural Information Processing Systems, 34:1022- 1035, 2021.
Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. Generating images with multimodal language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on
13

---

Published as a conference paper at ICLR 2025
Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA. Decem- ber 10 16, 2023, 2023, URL http://papers: http:/ppppr...ipppp..... .ips http:papprrfees//eeeeep Cesstract-Confrrennne html. 1, 2, 3,7
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In International Conference on Computer Vision (ICCV), 2017. 7
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-to-image diffusion. 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. MIMIC-IT: multi-modal in-context instruction tuning. CoRR, 0abs/2366.05425, 2023a. doi: 10448500/ARXIV.23660555
URL
https:////do http::::///ooo////////// /1/g//0./// 48550/arXiv www............. 2302.05425. 05425.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan
Li. Llava-next-imterlcave: Tackling multi-image, video, and 3d in large multimodal models. CoRR, abs/2407.07895, 2024a. doi: I048550ARRXX..247.0079999 URL attps://doi.org:
10. 18550//ar1x1 2407 07895. 20
Huayang Li, Siheng Li, Deng Cai, Longyue Wang, Lemao Liu, Taro Watanabe, Yujiu Yang, and Shuming Shi. Textbind: Multi-turn interleaved multimodal instruction-folloving in the wild. CoRR, 0bs/2309.08673 2023b. doi: 10.48550/ARXIV223090000 086637... URL /ttps:://do..oo
10. 48550/arXiv 2309 2
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language- image pre-training with frozen image encoders and large language models. 202:19730-19742, 2023c. URL https://procceddnng mlr, aemmm..rrreee wwwppprrsssseeeeee
html.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1tth International Joint Conference Natural Language Processing (Volume 1: Long Papers),
pp. 4582-4597, 2021.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modalit modality vision language models. CoRR, abs/2403. 18814, 2024b. doi: 10.48550/ARXIV.2400: 18814. URL https: org/ 10. 48550/ 2403 18814.
Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Scaling & shifting your features: A new baseline for efficient model tuning.
Sanmi Koyejo. S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 Decem- ber 9, 2022, 2022. URL http:://pppppae http://ppprr..i......... wwterrsiiin...cpaapeeeee cttpcccr_pppiiaeceeceeee feaper_filest www.ffiee//eeeeeeff Leeee//e willse/ppper//22// /hash/ Abstract- Conf erence .ttml 3
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. CoRR, ab$/2401.15947, 2024. doi: 1044850/ARXIV.200..555 15947.
URL hattps://dol org/10. http//..8550///// arxiv arXiv.2411. 2401.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014. 22
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. CoRR, abs/2306. 14565, 2023a. doi: 10.48550/ ARXIV.2306. 14523000..14...... URL https:: //ooi. org/ 10. 48550/ arviv. 2306. 14565.
14

---

Published as a conference paper at ICLR 2025
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficiem fine-tuning is better and cheaper than in-context
learning. CoRR, abs/2205. 056388, 2022. doi: 10.4850/ARXXV.220.00666 //doi.org/10. org/10.48550/ 48550/ arXiv. 2205. 05638.
URL https:
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CoRR, abs/2310.03744, 2023b. doi: 1044550/ARXXW.3300037444,, URL / org/10. 48550/ 48550/arXiv wtt..i...11.... 2wt.2.21.....2. 03744. 3
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023c. URL attps://doi. org/10 10. 48550/ arXiv 2304. 08485. 1,3, 6, 20
Minqian Liu and Lifu Huang. Teamwork is not always good: An empirical study of classifier drift in inasminnrreentll information extraction. In Anna Rogers, Jordan
and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 2241- 2257, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 1023.findings-acl.44 URL https: : /acaaannantoooy org/ oogg/022 2023 findings-acl.144
Minqian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eunah Cho, Vaibhav Kumar, Reza Ghanadan, and Lifu Huang. X-eval: Generalizable multi-aspect text evaluation via augmented instruction tun- ing with auxiliary evaluation aspects. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 8560-8577 Mexico City, Mexico, June 2024a. Association for Computational Linguistics. doi: 10. 18653/v1/ 473.. URL https: / /ataa/aaaantoooooo org/2024 maaacl-long 473/. 23
Minqian Liu, Zhiyang Xu, Zihao Lin, Trevor Ashby, Joy Rimchala, Jiaxin Zhang, and Lifu Huang. Holistic evaluation for interleaved text-and-image generation. In Proceedings of the 2024 Con- ference Empirical Methods in Natural Language Processing, 2024b. 7, 10, 23
Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. CoRR, abs/2306.09093, 2023. doi: 10.48550/ARXIV.2306.0000 URL https://doi. org/10. otg/10.4855///...... 408.44880a/iii.../ 29000.
OpenAl. Hello gpt-4o.
gpttlloo-ppto///..o 2024. Accessed: 2024-
05-26. 7
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joc Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, abs/2307.01952,, 2023. doi: 10.44550/ARXXV.233..00999
URL https:
//doi. otg//0.48550 48550/ arXiv 2007.0072220 01952. 4, 7, 20
Jingyuan Qi, Minqian Liu, Ying Shen, Zhiyang Xu, and Lifu Huang. MULTISCRIPT: multimodal script learning for supporting open domain everyday tasks. In Michael Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (eds.), Thirty-Eighth AAAI Conference Artificial Intelligence, AAAI
2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 18888-18896 AAAI Press, 2024a. doi: 10.1609/AAAL 10///110.00 1609/ /aaai v38i 17. 29854.
V38117 298544. URL UtRLhtts:://dot
Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jin Di, Yu Cheng, Qifan Wang, and Lifu Huang. Rora-vim: Robust aeureeeal-ugmmented vision language models, 2024b. URL https:///rxiiv org/ prrg/abs/2410 08876.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot cext-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021. 1
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap. Jean- Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer,
15

---

Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal, Codi- 2: In-context, interleaved, and interactive any-to-any generation. CoRR, abs/2311. 2023. doi: 10.48550/ARXIV23311.88 Published as a conference paper at ICLR 2025
Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mcllroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ay- oub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang. Henryk Michalewski, Za- heer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR, abs/2403.05530, 2024. doi: 10.48550/ARXIV.240..00 https://doi. org/10. 10. 48544/ aar.0o/rrii...o 2403. 05530.
URL
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10674-10685. IEEE, 2022. doi: 10.1 (0.1109/CVPP5222882222220200 10. N0.1119/CVPR55288 2022 01042.
URL httpp:://dot
Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models. arXiv preprint arXiv:2310. 10076, 2023, 23
Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang. Multimodal instruction tuning with conditional mixture of lora. CoRR, abs/2402.15896, 2024. doi: 10.48550/ 1RXIV.2402.158%. URL httpp://ddae
10. 48550/arXiv. 2402. 15896. 3,
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: improved training techniques for CLIP scale. CoRR, abs/2203..1338, 2023a. doi: 10.485550/AXX1.2300.. URL aattps://dos org/10. 48550/arXiv 2303 15389. 4, 20
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. CoRR, abs/2307.05222. 2023b. doi: 10445500AR3312230.0022222 48550/ arwiv. 2307.05222 05222. 1, 3,
URL https://doi. /tpp:////o.rg/10...
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 14398-14409, June 2024. 1,2, 3, 4, 6, 7, 20, 22
URL sttps:://ol: .rg/ 10. a8550//rXiv.23111 18775.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024. 1, 3,4,,7 7
Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yun- Chen, Lewei Lu, Tong Lu, Jie Zhou, Hongsheng Li, Yu Qiao, and Jifeng Dai. Mm- interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer. CORR, abs/2401.10208, 2024. doi: 10.485500ARXIV2240111. 10208. URL https://doi..oo org/ 10.48550/arXiv 1.001.000000 3
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, Abs/2302.13971, 2023. doi: 10.48550/ARXXI.22302.11 13971.. URL https://doi. org/10. 2.00048550/888 2a.50//axxiv..m 2302. 13971. 20
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In L. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.). Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL 2ffeee/255//eeee111bbccccccaaaa .df.aaaar..aaaaaaaaaaaeapaa.aeep
wtwp:eeeeddinnnne.iiiieeeeeeeeee neurips...... wtwppppp.cppppp..pppp httpp/pppp..llll//pppeeeeeeeee /iapperrftlle//ppppe////////
16

---

Published as a conference paper at ICLR 2025
Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting GPT-4V for better visual instruction tuning. CoRR, abs/2311.07574, 2023. doi: 10.44500/ARXII2331.00555 URL https:h//oi..ooo org/ 10. 18550/aaxivy http://..iiiv........... 2311.07574.
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-adaapatiion for parameter-efficient model tuning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 5744-5760. Association for Computational Linguistics, 2022. doi: 10.18653/V 1/20222/1/22222222 WEMNLP-MAIN.S. URL https:://dooo http:///d/oooo//////// 18653/v1/2022. emm2pemmllppp www.p-mmimlpmnme 388.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal LLM. CoRR, 0092339900559, 2023. doi: 104850ARRKI2230.05511 URL
https://doi. org/10. wrg//..48550/////.....
2309. 3
Zhiyang Xu, Ying Shen, and Lifu Huang. Multinntruuct Improving multi-modal zero-shot learning via instruction tuning. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6 Annual Meeting the Association for Computational Linguisties (Volume I: Long Papers), ACL. 2023, Toronto, Canada, July 9-14, 2023, pp. 11445- 1146.. Association for Computational Linguisties, 2023. doi: 10. 10.11865//11 A URL https: //doi. org/10. 10. 1ww111861/v/a
acl-long. 641. 3, 20
Zhiyang Xu, Chao Feng. Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled tasks in visual instruction tuning, 2024. 3, 20,
22
Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar, and Chris Callison-Buree Visual goal-step inference using wikiHow. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2167-2179, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 00.18653/v1/2211. 20tS53//1/221.mmmlppp.. main. 165. URL https: h/tp:/acllaaaa h/alaanthoollllyoooooooooo or2/2021 emnlp-main. 165. 7
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-Tau Yih. Reerieeal-augmmeneed multimodal language modeling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 39755-39769. PMLR, 2023. URL yasunaga23a, html. 3.4
mlr, www.mmwww..pm.............m....e press/v202/
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality. CoRR, abs/2304.14178. 2023. doi: 1048550/ARXIV2200411177 URL attps::/doil org/10. 100 48550/ arxiv. 2304. 14178.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collabo- ration. In Proceedings the the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13040-13051, June 2024.
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, Jing Shao, and Wanli Ouyang. LAMM: language-assisted multi- modal instruction-tuningg dataset, framework, and benchmark. CoRR, abs/2306.06688 2023. doi: 10.488500ARRII.23060000 06687. URL //attp:://oo org/10. 10. 48550/ arrvv 2306. 06687.. 3
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-War Li, Susan
17

---

Published as a conference paper at ICLR 2025
Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Fazel- Zaral-Zar Asli Celikyilmaz, Luke Zert, and Armen Aghajanyan. Scaling autoregressive multi-modal models: Pretrain- ing and instruction tuning. CoRR, abs/2309.02555 02591, 2023. doi: 10.4850/AR21..200.02259 URL https: : //to////..../////// org/10. prg//..48550///.... 48550/arXiv 2309. 02591.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Push- ing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. CoRR, abs/2309.05444 05444, 2023, doi: 10448550ARRXI.2330005445 URL https://doi.c
10. 4..45550/aaii.... 2309. 200...0499944444.. 05444. 3
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter--fficient fine-tuning for transformer-base based masked language-moder In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pp. 1-9, 2022.
Kai Zhang, Lingho Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manu- ally annotated dataset for instruction-guider image editing. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA. December 10 16, 2023, 2023a. URL http://papers: iores..nng nips. httpppcecpppppcceeeeeeee saapeerffies/ppaaa htts/paapx/22233/3 hatp:///dt---sstttt--aaaaaaaaeee and Benchmarks html. 7,
20
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018. doi: 10.1109/CVPR 8.0066.. 21
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. CoRR, 0ls/2306.171091 2023b. doi: 10.48550ARXIV23366.771 48550/ arxiv. 17107.
URL
Yue Zhang, Ziqiao Ma, Jialu Li, Yanyuan Qiao, Zun Wang, Joyce Chai, Qi Wu, Mohit Bansal, and Parisa Kordjamshidi. Vision-anddlanguagge navigation today and tomorrow: survey in the era of foundation models, 2024. URL https://arxti org/abs.....................e..
3
Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-language gener- ation via generative vokens. CoRR, abs/2310.022333 2023a. doi: 10.46550/ARXIV.310002333 URL https: //do:. /ttps://ddo.orr/11... 48550/arXiv. wtt..//.............//////.... 2310. 02239. 7
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- benc and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023b. 23
Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. Convolution meets lora: Parameter efficient finetuning for segment anything model. CoRR, abs/2401.17866 2024. doi: 00.48550/ARXI.2441.11 2,
URL https: //doi. org/10. /ee//1.4855555 48550/ arXiv 2... 17868. 2, 3,
Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 7590-7598. AAAI Press, 2018. doi: (0.1609/AAALV3211..22 10. 1w19/699aaaa. 1t..0//aaa.v32....... 12342. 6
URL https: //dg..or////////////////ooo
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing Dision-language understanding with advanced large language models. CoRR, abs/2304.10592, 2023a. doi: 10.48550/ARXIV2230..105 URL https: ///oii http://tioooo....///// 2r0vv. 10592. 3
18

---

Published as a conference paper at ICLR 2025
Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan. VL-GPT: A generative pre-trained transformer for vision and language under- standing and generation. CoRR, abs/2312.09251, 2023b. doi: 1D48550/ARXIV2312.9935 48550//rxxi/ 2312. 09251. 3
URL https: / /doii
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. In Thirty-eeventh Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2023c. URL https: //operrviee net/forum?id=toeee
2,
19

---

Published as a conference paper at ICLR 2025
A MORE DETAILS OF MOSS
A.1 IMPLEMENTATION DETAILS
We leverage the Emu2 model (Sun al., 2024), consisting of the EVA-O2-CLIP-E-piie (Sun al., 2023a) as the image encoder, the LLaMA-33B (Touvron et al., 2023), and the SDXL (Podell et al., 2023) as the image decoder, as our base model. The EVA-02-CLIP-E-plle and the LLaMA-33B is connected by a linear project-up layer and the LLaMA-33B and the SDXL is connected by linear project-down layer. All the variants of LoRA in Section 7, including our MoSS are trained with LEAFINSTRUCT for one epoch on A100 GPUs with learning rate 2e batch size per GPU, and gradient accumulation step of 16. All the LoRA have rank of 256, dropout rate of 0.05, and the LoRA a in Section 4 set to 2 x 128. The kernel size of MoSS x 2, the stride is set to 1. During training, all parameters of the Emu2 model are kept frozen and only the LoRA parameters are updated.
B MORE DETAILS OF LEAFINSTRUCT
M4-Instruet
LeafInstruct (Ours)
Instruction: Here are 5 images <IMG> -IMG> <IMG> <IMG>, which image shows following content: (cuption of image 2]
Deseribe a travel log during hiking. Highlight the socne of plants, insect, and emplasizc the beauty of
Instruction:
Output: The colors of nature are truly stunning <IMG> A wildflower plant of some sort, possibly native species, landscape <IMG?,
Input Context: That beauty all glory caplivating. with vibrant colers and delicati <IMG> be trying to cat the plant -IMG>
There was little Indy Indybug in nature, its contrasting beantifully with surrounding foliage <IMGs
Output: Image
InstruetPix2Pix
Instruction:
Output:
Swap sunflowers
with roses
-IMG>
Figure 6: Comparison between existing benchmarks and our LEAFINSTRUCT. In existing datasets such as InstructPix2Pix (Brooks et al., 2023b) and Mantis-Instrut (Li et al., 2024a), the outputs are in single modality, either text or image. On the contrary, the inputs and outputs of our LEAFIN- STRUCT cover multiple modalities.
Table 4: Comparison between our LEAFINSTRUCT and existing instruction tuning datasets.
Dataset Name
Input Text
Input Images
Output Text
Output Images
Publicly Availahle
LLaVA (Liu al., 2023c)
Single
Single
No
Yes
Multilnstruct (Xu et al., 2023)
Yes
Single
Single
No
Yes
Vision-Flan (Xu et al., 2024)
Yes
Single
Single
No
Yes
InstructPix2Pixx (Brooks ct al., 2023a)
Yes
Single
No
Single
Yos
MagicBrush (Zhang et al., 2023a)
Yes
Single
No
Single
Yes
SuTI (Chen el al., 2023c)
Multiple
No
Single
No
Instruct-Imagen (Hlu al., 2024)
Yes
Multiple
No
Single
No
Mantis-Instrus (Jiang al., 2024)
Yes
Multiple
Yes
No
Yes
LEAFINSTRUCT (Ours)
Yes
Multiplc Yes
Multiplc
Yes
B. MORE DETAILS DATASET CONSTRUCTION
We elaborate on the details of our dataset construction pipeline follows. Firstly, we filter the samples based on the text length, number of images, and the coherence between text and images (measured by CLIPScore (Hessel et al., 2021)). We only keep the instances with 3 to 6 images in total. We also discard the instances with more than 12 sentences to ensure a balanced ratio between the number of textual sentences and images. Secondly, leverage state-of-the-an open-sourced LLM (i.e.. Llama-8u -B-aa-BB-rn 1e---BBInnee Instruct) as a text filter to discard the instances with poor text quality. Thirdly, remove the instances with duplicate perceptually highly similar images ensure
20

---

Published as a conference paper at ICLR 2025
the diversity of the images. Finally, also apply Llama3 annotate the task instruction for each instance based on the text content and rewrite the text it's too verbose to prevent the context length from being too long.
Details of Text Quality Filter We use -aamam---Instrae model rate the text quality of an instance with the following prompt: "Imagine you are an expert data an- notator. You are given text material and you need evaluate its quality terms of whether it is coherent, fluent, easy to understand, and helpful to humans. Please critical and rate the quality as good only when the text quality is good in all four aspects. Output if you think the material is good after you consider all four aspects. Outpul O if you think the material is not good enough. Here is the text material to be evaluated: {TEXT) Only output 0 or and do not output anything else. Your evaluation is:" We discard the instances if the output from Llama
Art
Business & E-commerce
Nature
8%
24
Automotive
Food & Cooking
7.0%
7. 7.4%
13 13.2%
Travel
Technology
8%
15.2%
Fashion
Events
Figure 7: Domain distribution in Leaflnstruct.
Details of Image Filter We empirically found that if the images are too identical in the training instances, the trained models tend to find a shortcut to simply copy the image during generation. To this end, design filter discard the instances with duplicate images to improve data quality. Specifically, we leverage the LPIPS score (Zhang et al., 2018) that measures the perceptual similarity between the images. Specifically, for each instance, their LPIPS score. there is one pair with score higher than 0.6, we discard the instance. We determine the threshold of 0.6 by empirical trial.
enumerate each pair of images and compute
Details of Instruction Annotation We also adopt Baama--8-Innteeen to annotate the task in- struction for each instance. We devise instructions prompt the Llama3 model rewrite the original text material in the pretraining dataset MMC4 into instruction-tuning instances. The in- put context length 2048 and the output context length is 1024. We set the temperature as 1 to encourage the diversity of instructions. We use the following prompt: "Imagine you are expert instruction annotator. You are given material. You need to read its content and output a brief task instruction with sentence such that another person recover the given the material given the instruction. The instruction you predict should be specifically tailored for creative interleaved content generation that consists of both text and images. Now you need to annotate a concise, ac- curate instruction for the following instance. Please only predict the instruction and do not output anything else. Please design the instruction for the multi-modal generation task interleaved with both text and images. Text: {TEXT) Instruction:".
C MORE EXPERIMENT RESULTS
C.I MORE DETAILS ON EVALUATION BENCHMARKS
InterleavedBench has two splits: context-based split which the input of each instance is equipped with interleaved text and images; and a context-free split with text-only inputs. The context-basse based split contains 465 instances and the text-only split contains 350 instances. We only use the context- based split as the testing set since we mainly focus on tasks with interleaved inputs and outputs.
C.2 MORE RESULTS ON ESTABLISHED BENCHMARKS
Interleaved Generation and Image Editing Although InterleavedBench is new evaluation bench- mark, it also consists of testing instances from 3 well-established benchmarks including (1) visual story completion from VIST, (2) MagicBrush, and (3) multi-concept image composition from Cus- tomDiffusion. Below, directly report the performance of MoSS and baselines these three
21

---

Published as a conference paper at ICLR 2025
well-established benchmarks in Table 6, 7, 8, respectively. We also report the performance of Mag- ieBrush using established metrics Table 5.
Table Results of image editing the full test set of MagicBrush. We show the performance of open-source VLGs (Top), and the VLGs trained with our proposed method (Bottom), respectively.
Model
CLIP-
DINO
CLIP-T
AVG
MiniGPT-5
72.04
41.66
25.45
46.38
GILL
72.95
43.32
24.81
47.03
SEED-X-Edit (Ge et al., 2024)
85.56
68.74
27.28
60.53
Emu2 + MOSS (Ours)
85.88
74.92
25.98
62.26
Multimodal Understanding and Text-to-Image Generation To show that our MoSS framework also excel tasks requiring single modality outputs i.e., the output only contains text image, we evaluate its performance on widely adopted image understanding benchmarks including MMBench, MME, MMMU, Pope, and MM-Vet, and text-to-image generation benchmarks includ- ing MSCOCO 30K (Lin et al., 2014), and GenEval (Ghosh et al., 2024). For MSCOCO-23 following the previous evaluation protocol (Sun et al., 2024), we randomly sample 30,000 captions from the validation set of MSCOCO and generate 30,000 images. We report the FID between the 30,000 generated images and real images from the validation set MSCOCO (Note for FID, lower the better). For other benchmarks, we adopt their official implementation of the evaluation. Since LeafInstruct mainly targets tasks with interleaved outputs, we augmented it with 500,000 instances from Vision-Flait (Xu et al., 2024), popular image understanding, and 500,000 instances from LAION-COCO", standard training dataset for rext-to-image generation. We finetune Emu2 with LoRA, MoE-LoRA, and MoSS on the mixed dataset. We report their performance Table 9.
tuning dataset targeting
Table 6: Performance of our MoSS, traditional linear LoRA, and of (MoE) LoRA using Emu2 as the backbone model on the VIST subset in InterleavedBench.
PEFT
Text Quality
Image Quality
Image Coherence
TIC
Helpfulness
LoRA
0.3
0.52
0.46
0.68
0.43
MoF-LoRA
1.76
2.19
2.52
2.91
1.92
MoSS
1.73
2.87
2.54
3.30
2.26
Table Performance of our MoSS, traditional linear LoRA, and Mixture-of-Expert (MoE) LoRA using Emu2 as the backbone model on the MagicBrush subset in InterleavedBench.
PEFT
Text Quality
Image Quality
Image Coherence
TIC
Helpfulness
LoRA
N/A
2.33
1.48
N/A
0.93
MoE-LoRA
N/A
2.85
1.54
N/A
1.08
MoSS
N/A
3.43
2.03
N/A
1.27
Table 8: Performance of our MoSS, traditional linear LoRA, and Mixture-of-Expert (MoE) LoRA using Emu2 as the backbone model the CustomDiffusion subset in InterleavedBench.
PEFT
Text Quality
Image Quality
Image Coherence
TIC
Helpfulness
LoRA
N/A
2.95
1.73
N/A
1.56
MoF-LoRA
N/A
3.36
1.67
N/A
1.39
MOSS
N/A
3.24
2.04
N/A
1.83
22

---

Published as a conference paper at ICLR 2025
Table Results on widely adopted multimodal understanding and text-to-image generation bench- marks. Note that the FID metric on MSCOCO is the lower the better.
Model
MMBench
MME
MMMU
Pope
MM- Vet
MSCOCO-30K FID (U)
GenEval
Chameleon
32.7
604.5
38.8
59.8
9.7
26.7
39.0
Emu2+LoRA
54.1
1148.0
33.7
87.3
31.3
23.4
26.8
Lmu2+MoB-LoRR
54.6
1170.3
34.1
88.1
31.9
22.7
28.1
Emu2+MoSS(Ous)
56.0
1278.4
35.8
87.6
34.1
18.2
28,9
From Table our MoSS outperforms previous LoRA and MoE-LoRA most of the multimodal understanding benchmarks by a notable margin, which demonstrates that MoSS can well general- ized to diverse multimodal comprehension tasks. For text-to-image generation, MoSS achieves better performance on both benchmarks, showing the effectiveness and generalizability of our ap- proach. Notably, our MoSS achieves significantly better FID MSCOCO-30K, which validates that our ConvLoRA can effectively improve the quality of generated images.
C.3 ADDITIONAL ANALYSIS
Comparison between previous and our ConvLoRA To show the benefits of our modified Con- vLoRA architecture compared to the ConvLoRA proposed in Zhong al. (2024) denoted as SAM- ConyLoRA, we replace the ConvLoRA in MoSS with SAM-ConVLokE rank of project-down and project-up matrices in SAM- BAM-ConvLoRA to 256 which is the number of ranks our proposed MoSS-CCnvLoRA, and adopt the multi-scale convolution kernels to the size of 2x2 and 4x4. As shown in Table 10, vious SAM-ConVLoRR proposed ConvLoRA architecture. Particularly, sual qualities, including perceptual quality and image coherence, thanks to our novel design that our convolution operation applied the full-rank original image features instead of low-rank image features as in SAM-ConvLoRA.
Specifically, we set
MoSS-ConyLoRI consistently outperforms the pre- on all other evaluation aspects, which demonstrates the superiority of our
MoSS-ConvLofe achieves notably better vi-
Table 10: Comparison of two types of CoRYLoRA
Model
Text Quality
Image Quality
Image Coherence
TIC
Helpfulness
MoSS w/ SAM-ConVL.RA MoSS w/ MoSS CoSS-CCnVLLoRA (Ours)
2.50
3.33
3.17
3.50
2.41
2.61
3.62
3.41
3.54
2.71
Human Evaluation on Text Quality of Chameleon and Chameleon MoSS We noticed that adding MoSS in Chameleon cause slight performance drop in text quality. This is because the original Chameleon usually generates long and verbose text responses but with no image output. On concise to allow for including
the contrary, the text responses in LeafInstruct dataset more images, after interlcaved instruction tuning, our model learns to generate more concise text re- sponses. Specifically, the average generated word length of the original Chameleon 653, whereas that of Chamelcon-Moss is 166. The verbose responses from the original Chameleon are preferred by the LLM judge due to their verbosity bias (Zheng al., 2023b; Saito et al., 2023), leading to slight drop in the text quality of Chameleon- MoSS in LLM-based based evaluation.
To better support this analysis, we further conduct human evaluation of the text quality of the two models by randomly sampling 100 instances from InterleavedBench. We ask human annotator to select the preferred text responses given the system outputs from two models. We report the Win- Min-Tie-Loss results in Table 11. Win means our Chamelcon-MoSS is better than the original Chamelcon, Tie means the quality of two responses is equally good, and Loss means the original Chameleon is better.
From Table 11, the text quality of our Chameleon-Ne Chameleon. One issue frequently observed in the original Chameleon is the text responses are overly verbose and sometimes even severely repetitive. In our evaluation protocol of text quality adopted from InterleavedEvvl (Liu et al., 2024b;a), such verbosity and repetitiveness
actually better than the original
penal-
23

---

Published as a conference paper at ICLR 2025
Table 11: Human evaluation results on text quality the original Chameleon and our Chameleon-
MoSS. "Win" indicates our Chameleon-MoSS responses are preferred by humans.
Wins
Ties
Losses
28
54
18
ized, making the automatic evaluation results heavily biased towards the longer responses from the original Chameleon.
Performance of full-fnettninng We compare the performance of full-parameter fine-tuning using LEAFINSTRUCT in Table 12. The first row represents the results of fully fine-tuning Emu2 proposed LEAFINSTRUCT dataset. The second row shows the results of parameter-efficien fine- tuning Emu2 using our proposed MoSS framework. As observed, while full fine-tuning allows Emu2 to achieve better performance on text generation, the model demonstrates inferior perfor- mance on image generation due its lack of inductive bias. In contrast, tuning with MoSS, which incorporates ConyLoRA, significantly improves image generation performance, even though the number of trained parameters full fine-tuning substantially larger than that of MoSS. These results clearly highlight the advantages of integrating ConvLoRA into the transformer architecture for processing visual information.
Table 12: Comparison between full finctuning and parameter-efficient tuning with MoSS based on Emu2.
Model
Text Quality
Perceptual Quality
Image Coherence
TIC
Helpfulness
Full Finetuning
3.20
3.21
2.98
3.60
3.23
MoSS (Ours)
2.61
3.62
3.41
3.54
2.71
Computation cost of ConALoRA We compared the computational cost of using linear LoRA and our ConyLoRA, respectively. We compute the inference time for generating 1,000 images for each model. The total inference times for linear LoRA and ConvLoRA are 4,380 seconds and 5,910 seconds, respectively. The difference between the two models is around 1.5 seconds image, indicating the computational cost increased by ConvLoRA is not significant.
C.4 ADDITIONAL QUALITATIVE RESULTS OF CHAMELEON
We show the additional qualitative comparison between the original Chameleon and Chamelcon+MoSS in Figure From the results, most of the examples of the original Chameleon only contain text without any image, which indicates the original Chameleon often fails to follow the instructions to generate desired mixed-modal content. Our Chameleon+MoSS, on the contrary, achieves much better performance as is able generate high-quality and coherent text and images. These results further validate the effectiveness of our approach and its strong generalizability and robustness when applying to different VLGs, including Emu2 based on continuous image tokens and Chameleon based on discrete image tokens.
C.5 PER-TASK PERFORMANCE ON EACH EVALUATION ASPECT
We report the performance on all aspects for each task InterleavedBench in Figure
C.6 MORE RESULTS ON THE EFFECT OF RANKS
We conducted experiments show the performance of LoRA baselines and Lateralization LoRA on InterleavedBench with different ranks (64, 128, 256) in Table 13, 14, 15, respectively.
24

---

Published as a conference paper at ICLR 2025
Input
Chameleon
Chamelcon MoSS (Ours)
Instruction: Complete travel story. Context: visited the countryside for the weekend <IMG>
Output: I've
Output: There are many popular activities to do the countryside, including hiking. biking, camping, and visiting local attractions
wanted visit the the ocean. I've heard great about the of Mainc, so would would like to go hiking, some lighthouses, and enjoy fresh scafoud...
like museums and castles. including cycling the ridgeway Trail
<IMG>, camping in yorkshire dales <IMGs laking bost tour the Lake District <MG.
Instruction: Describe a team's Univensity
Output: The event challenges teams to design, build, and operate that can navigate across challenging Martian-Hi like termiin, while performing tasks such as recomaiseence, exploration, and scientific rescarch....
Challenge. Context: During
Output: This competition encourages univensties design, build, and test rovers that can operate on the Martian surfhce replicating real-worll scenarios that ongineers scientists may encounter during future Mars missions, <JMG>
challenge,
placed the sccond <IMG>
PALA
Instruction: Tell me what should plan vegan healthy meals, Context: I've done: had oatmeal fruit
Output: Hore are few more ideas: Smeothic bowls made with fruit, nuss, <IMG> tofu <IMGs Overnight cats made with rolled nats, milk, and Chin pudding cakes made with whole grains and plant-hased milk <IMGs.
scrambled
Output: Have a healthy snack breakfast, such as apple un orange, almonds or cushews.
nats <IMG>, Muffins
tofu scramble for breakfastt the
morning <IMG>
Instruction: Help modify this picture -IMG> and change bus
Outputs:
Outputs: course! I'll make sure do hest to assist <IMG>
version of picture with the replaced by
Figure Qualitative results MoSS based on Chameleon and the original Chameleon. The <IMG> tokens denote the images' positions in the interleaved sequences.
Table 13: Performance of our MoSS, traditional linear LoRA, and Mixture-of -ff-eeertttttttt (MoE) LoRA using Emu2 as the backbone model with rank r 64.
PEFT
Text Quality
Image Quality
Image Coherence
TIC
Helpfulness
LoRA
1.7
1.81
1.90
1.39
MoE-LoRA
1.86
2.17
2.17
2.15
1.66
MOSS
1.46
2.34
2.20
2.13
1.58
Table 14: Performance of our MoSS, traditional linear LoRA, and Mixture-of offuurre-f--xperrttt (MoE) LoRA
using Emu2 as the backbone model with rank 128.
PEFT
Text Quality
Image Quality
Image Coherence
TIC
Helpfulness
LoRA
1.25
1.43
1.61
1.79
1.30
MoE-LoRA
1.94
2.22
2.42
2.54
1.90
MoSS
1.95
2.41
2.64
2.81
2.05
25

---

Published as a conference paper at ICLR 2025
Text Autety
Perceptual Quality wTy
Acuay
Subt cut pirins
Vedittooal Document
Sia direer
10y1 mrobu Docamnat. Coppptttion
Yu Bareda
Wodloade Document Breant
Uoall Stoy
Whaa Stary
Wesk Strry
Must lack Softn
Complttion
Image Coherence Actulty
Image Coherence
Siuect-srion
Inktmmebll Decument
Sugee offes
best preio Documerr
Mu Briidh
MLEOOAEE Decumert
Irage Bgsing
Voull Strey
Image Fetro
Wilay Suay
thd vieder
Woug Strr
Emu2-MatS
Emu2
Chamekoo-MoSS
Chameleon
GILL
MiHIGPT-S
Figure 9: Per-task performance each evaluation aspect InterleavedBench.
Table 15: Performance of our MoSS, traditional linear LoRA, and Mixture-of-Exper (MoB) LoRA using Emu2 as the backbone model with rank 256.
PEFT
Text Quality
Image Quality
Image Coherence
TIC
Helpfulness
LoRA
1.77
2.38
1.99
2.04
1.64
MoE-LoRA
1.98
3.28
2.66
2.62
2.01
MoSS
2.61
3.62
3.41
3.54
2.71
26


Reviews:
[{"review_id": "ntVG9J8dJK", "reviewer": "Reviewer_pC3r", "summary": "This paper aims to improve the understanding ability of existing Vision-Language Generalists (VLGs) and propose the Modality-Specialized Synergizers (MoSS). Moss focuses on two things: 1) To build high-quality interleaved text and images, Moss modifies the connectors (such as the Linear layers between the image encoder and LLMs) in VLGs) by introducing a linear LoRA (for textual inputs) and Convolutional LoRA (For image inputs). 2) To improve the understanding of perform interleaved tasks. MoSS develops a high-quality instruction dataset (with 184,982 instances). Experiment results show the improvements of the introduced two LoRAs and ablations show the efficiency of the convolutional LoRA.", "questions": "Please see the Weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper is written well with clear figures and tables, which make the readers easy to follow the story.\n\n2) The ideas that utilize model-specific adapter to process the text and image make sense to me. Such adapter may capture inherent semantics of corresponding inputs.\n\n3) This paper develops a high-quality interleaved instruction dataset, which will benefit to the VLG community.\n\n4) Experiments and ablations show the improvements and efficiency of the proposed modules", "weaknesses": "1) One of the core concerns is the novelty of the two LoRA types. Given the facts that both Linear LoRA and convolutional LoRA are not new to recent vision-language models. I think the contributions are limited.\n\n2) Technically, MoSS beliefs that the linear layer could lose the visual information and adopts the convolution LoRA to capture local patch features. However, the ViT-based image encoder in EMU (ViT are often used in VLMs) already flatten an image into a token sequence and employs self-attention  to model their dependents. That is that the ViT outputs inherently contain the local information, and the linear adapter act as a semantic translator that map the visual features into LLM space. Thus, I do not think the convolutional LoRA is necessary for VLG, mathmetically.\n\n3) MoSS argues that previous methods use the same framework for both text and image. If we think the model-specific encoder and the shared adapter as a whole module, we find the they use different frameworks to process the text and image.\n\n4) Can the authors apply MoSS to more powerful models, such as Emu 3 to test it robustness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MVGpPBJniL", "reviewer": "Reviewer_izYW", "summary": "This paper proposes a new modality-specialized training method called MOSS and an instruction-turning dataset for interleaved text-and-image generation. By adopting MOSS on two existing frameworks, they show improvements on an interleaved evaluation benchmark (InterleavedBench).", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposed a novel design that enhances VLGs to generate interleaved content with modality-specialized parameters and adaptation architectures.\n2. This paper introduces an open-sourced large-scale instruction-tuning dataset that allows interleaved multi-image and text input and output.", "weaknesses": "1. The proposed convolutional LoRA (Equation 4) is similar to the LoRA proposed in [1]. The authors claim that their new LoRA could alleviate the information loss, yet no experimental comparison between the two kinds of Conv LoRAs is provided.\n2. The evaluation is limited. \n    1. They only evaluate on InterleavedBench and an image editing benchmark called MagicBrush. The coverage of the evaluation is relatively small. \n    2. Since the proposed model can do both image and text generation, more benchmarks could be included to measure the model performance from different perspective. E.g., multimodal understanding benchmarks like MMMU, MathVista, VQAv2, POPE. image generation benchmarks such as GenEval and T2I-CompBench. \n3. For the proposed data:\n    1. The automatic data annotation pipeline  mentioned in Line 298 is not elaborated. How did the authors acquire LeafInstruct from existing academic dataset?  \n    2. According to Line 299, the details of dataset construction are shown in Table 3. I cannot find any construction details in Table 3.\n4. This paper is not well organized and written. Some reference (such as the above table reference) is not accurate. This may cause it is not easy to read and understand.\n5. The improvements are relatively limited. Adding MOSS only gets the open-source state-of-the-art performance on 3/5 metric. Adding MOSS onto Chameleon even causes text quality drop. Did the authors have hypnosis or investigate these results?\n6. No qualitative results of Chameleon-MOSS.\n[1] Convolution meets lora: Parameter efficient finetuning for segment anything model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IuJFGbWjFv", "reviewer": "Reviewer_YfET", "summary": "This paper examines the intrinsic inductive biases present in vision and language modalities in VLGs. It introduces MoSS, a novel approach that optimizes existing unified VLG architectures through modality-specialized adaptation layers—ConvLora for capturing local priors in image patches and LinearLora for handling sequential text. Additionally, the paper presents LeafInstruct, an open-source interleaved instruction tuning dataset. Experimental results demonstrate that MoSS enhances VLG model performance.", "questions": "I wonder whether the results for the four middle rows in Table 1 reflect the model's original performance or its performance after full-parameter fine-tuning on LeafInstruct. Did you try full-parameter fine-tuning using your constructed data? Or full-parameter fine-tuning with two different sets of parameters for image and text tokens?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes a novel idea that parameters for processing information of different modalities in VLGs should be trained with different strategies.\n2. The proposed MoSS method brings promising enhancement in model performance of VLGs.", "weaknesses": "1. The performance improvement shown in Table 1 is inconsistent, with Chameleon displaying a decline in text quality after training with MoSS.\n\n2. It remains unclear whether the observed performance enhancements are attributable to the MoSS training method or the LeafInstruct dataset (see Questions).\n\n3. The parameters of ConvLora cannot be merged into the original parameters, as convolution is not linear. This limitation may lead to increased computational costs during inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4QPYcAmFNX", "reviewer": "Reviewer_ccUt", "summary": "This work aims to enhance the interleaved text-image generation capabilities of VLGs. The authors note that current VLGs use the same architecture for processing and generating both text and images, which may not adequately capture the distinct inductive biases inherent to each modality. To address this, they propose the Modality-Specialized Synergizers (MOSS), introducing modality-specific parameters within a unified model architecture. Specifically, they integrate convolutional LoRA for image processing and Linear LoRA for text processing.", "questions": "1. Could the authors elaborate on why, even after fine-tuning with interleaved text-image data, a unified model is still unable to capture modality-specific inductive biases?\n\n2. What is the relationship between the capability for interleaved image-text generation and the use of discrete tokens versus continuous embeddings? Which approach holds a distinct advantage?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work proposes to address the distinct inductive biases of each modality by using specialized LoRA parameters.\n2. The proposed method is intuitive and straightforward.\n3. A new high-quality interleaved instruction tuning dataset with 184,982 instances covering over 10 domains is introduced.\n4. The study conducts experiments on two different VLG backbones with both discrete and continuous image token spaces.", "weaknesses": "1. There are existing works focusing on interleaved image-text generation that the authors have overlooked, such as [1-3]. These works are not included in the experimental comparisons.\n\n     [1] MM-Interleaved: Interleaved Image-Text Generation via Multi-modal Feature Synchronizer    \n     [2] OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation    \n     [3] Anole: An Open, Autoregressive and Native Multimodal Models for Interleaved Image-Text Generation    \n\n2. In Figure 1, the authors only show the performance of continuous embedding-based VLGs for interleaved text-image generation. What about discrete-based VLGs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)