Wings: Learning Multimodal LLMs without Text-only Forgetting
Authors: Yi-Kai Zhang
Shiyin Lu
Yang Li
YanQing Ma
Qing-Guo Chen
Zhao Xu
Weihua Luo
Kaifu Zhang
De-Chuan Zhan
Han-Jia Ye
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual and Textual Learners
Large Language Models
Multimodal Large Language Models
Primary area: generative_models
TL;DR: A new Multimodal LLM: constructing visual and textual "wings" (i.e., learners) to extend visual comprehension without sacrificing text-only instruction capabilities.

Abstract: Multimodal large language models (MLLMs), initiated with a trained LLM, first align images with text and then fine-tune on multimodal mixed inputs. However, during the continued training, the MLLM catastrophically forgets the text-only instructions that the initial LLM masters. In this paper, we present Wings, a novel MLLM that excels in both text-only and multimodal instructions. By examining attention across layers of MLLM, we find that *text-only forgetting* is related to the attention shifts from pre-image to post-image text. From that, we construct an additional Low-Rank Residual Attention (LoRRA) block that acts as the "modality learner" to expand the learnable space and compensate for the attention shift. The complementary learners, like "wings" on either side, are connected in parallel to each layer's attention block. The LoRRA mirrors the structure of attention but utilizes low-rank connections to ensure efficiency. Initially, image and text inputs are aligned with visual learners operating alongside the main attention, balancing focus on visual elements. Later, textual learners are integrated with token-wise routing, blending the outputs of both modality learners collaboratively. Our experimental results demonstrate that Wings outperforms equally-scaled MLLMs in both text-only and visual question-answering tasks. Wings with *compensation of learners* addresses text-only forgetting during visual modality expansion in general MLLMs.

WINGS: Learning Multimodal LLMs
without Text-only Forgetting
Yi-Kai Zhang 1.233
Shiyin Lu
Yang Li
Yanqing Ma
Qing-Guo Chen
Zhao Xu
Weihua Luo
Kaifu Zhang De-Chuan Zhan
Han-Jia Ye
School of Artificial Intelligence. Nanjing University
National Key Laboratory for Novel Software Technology. Nanjing University Alibaba International Digital Commerce
Abstract
Multimodal large language models (MLLMs), initiated with trained LLM, first align images with text and then fine-tune on multimodal mixed inputs. However, during the continued training, the MLLM catastrophically forgets the text-only instructions that the initial LLM masters. In this paper, we present WINGS, novel MILM that excels in both text-only and multimodal instructions. By examining attention across layers of MILM, we find that rext-only forgetting is related to the attention shifts from pre-image post-image text. From that, construct an additional Low-Rank Residual Attention LoRRA) block that acts as the "modality learner" expand the learnable space and compensate for the attention shift, The complementary learners. like "wings" on either side, are connected in parallel each layer's attention block. The LoRRA mirrors the structure of attention but utilizes low-rank connections to ensure efficiency. Initially, image and text inputs are aligned with visual learners operating alongside the main attention, balancing focus on visual elements. Later, textual learners are integrated with token-wise routing, blending the outputs of both modality learners collaboratively. Our experimental results demonstrate that WINGS outperforms equally-scaled MLLMs both text-only and visual
tasks. WINGS with compensation of learners addresses text-only forgetting during visual modality expansion in general MLLMs.
Introduction
are making significant strides toward Artificial General Intelligence (AGI) systems. Multimodal Large Language Models (MLLMs), visual expansion of L.Mss, have demonstrated astonishing performance in vision-related captioning [14]
Large Language Models (LLMs) 34 54 93 115
16
68
127 133). Common MILMs build upon understanding 7 33 122] and reasoning [117]
powerful pre-trained LLMs that take mixed textual and visual tokens inputs. The visual ones are acquired using an image encoder and a projector. We describe instructions processed by the LLM without images as text-only instructions. In comparison, multimodal instructions incorporate visual feature tokens into text-only sequences. Modality fusing the token level provides flexible and effective pipeline for training MILMs to comprehend visual information [77] 80 81. However, training on multimodal instructions seems to impair the pre-existing profound knowledge, especially making MILM forget how to respond to text-only instructions like the initial LLM [86] 90). MLLM experiences a drastic performance decline on text-only evaluation. We term as the text-only forgetting of MILMM
"Work done during the internship at Alibaba International Digital Commerce. Cooresponding author, email: yehj @ Lunda.nju.edu.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Text-only Instruction
Text-only QA Multimodal QA
User: Let's pick out a better watermelon.
User: What are some refieshing fruits
heat summer hear?
MMUU
HallesionBench
WiNGS (Ours); Berh watermelons appear good quality, but The left une has a smoother rind and more splited oolor, indicating potential sweetness and freshness the intact green stem, And... ...
WINGS (Ours): watermeion, strowberries. blucberries, raspberies, peaches, and oranges.
CMNILLI
A120
RACE-M
SEED-Bench
SclenceG4
Interleaved Image-Tex Text Contexi
RACE-H
Multimodal Instruction
User: wow! just found
MME
User: You make good point, but I haven't
watermelons in the gurden:
ARC
tasted yet. You know
They'e te gonna
MMMBench CN
thing is? Besutiful.
be super rofreshing.
HellsSwag
MMBench-EN
WINGS (Ours): The object the image q waterrelon which has been cerved and decornted to resemble the heart. The heast shape fomsed by watermelon's natural rind and the roses are made from the watenmelon's 's [lesh inserihed
(Ours):
WinoGrande
image shows [wo waternelons,, one labeled as "I" and other "2. " They placod next each other, and both are ripe. The watermelons are green and have a striped pattern...
MMMMU-ES5T
MEPP
MMMUNAL
LLaVA Nest
DeepSeck VL
WINGS (Oars)
The initials "K.
Figure I: Examples of text-only and multimodal conversations. From left to right: Interacting with MLLM through text-only and interleaved instructions; Performance radar charts for W WINGS, LLaVA-Newx [8]] and DeepSeek- VL [86] in text-only and multimodal QA tasks, with dark green indicating WINGS with the comprehensive performance; Interacting with multimodal instructions.
In practical applications, MLLMs also require engaging in text-only interleaved conversations. As demonstrated in [Figure I users often start with text-only inquiries and then, if not fully satisfied with the response, proceed to supplement questions with visual content. For multimodal instructions, MLLMs still rely on text to capture critical elements, as images may offer redundant information [15]
17 155]
The first existing approaches replay extensive text-only interleaved [61 [5]] training data to mitigate catastrophic forgetting in MLL.Ms [72] 86, 90]. However, increasing training data incurs additional computational overhead and data collection challenges. Secondly, some applications [40) switch between LLM and MLLM based on whether images are included. This intuitive solution inevitably demands more deployment memory ( and is less frche-friendly in long vision-and- language interleaved conversations 42 76 [O1] Therefore, it is crucial to train MLLM while preserving the text-only performance efficiently.
Given that the visual input tokens can be inserted at any position within the text sequence, we begin by examining the text before and after the inserted position to mark the impact of the visual part. Considering that MLLM's s attention weights reflect the focus on tokens and influence the decision- making process, we first analyze the attention weights across each layer of the MLLM. Specifically, for each layer, we compute the attention weight proportion on all text tokens before and after the inserted image, termed as Layer-level Attention Weights (LAWS) of the before and after image text. From this, we examine the dynamic of attention across all layers as MLLM-Laws. Through training and sampling over 100 diverse MLLMs, we find that a performance shows positive correlation of MLLM-LAW: between the text segments before and after the image. This suggests that in a well-structured feature space, the main branch attention on text exhibits similar trends across layers, which is statistically linked to the semantic similarity of the text around the visual part. A closer similarity indicates minor disruption to MLLM's s core attention, while a negative correlation shows that excessive focus on visual tokens shifts attention away from the text, significantly impacting MLLM-Lawe
model with superior text-only
Based on this observation, we propose WINGS, which introduces an extra module that acts as the boosted learner to compensate for the attention shift. We integrate complementary visual and textual learners in parallel at each layer's attention block, with visual learners enhancing focus on visual tokens and textual Iearners on text, respectively. In the first stage, visual features align with textual feature tokens, with all visual learners operating parallel to the main branch attention. The visual learners allocate some attention to visual tokens, mitigating the attention shift in the main branch. Subsequently, textual leamners are integrated in parallel. We implement token-wiin wise soft-routing based on shifted attention weights to harmonize the learning on visual and textual tokens. We design the Low-Rank Residual Attention (LoRRA) as the architecture for learners to ensure high efficiency. Figure 3 shows that the visual and textual learners on either side, like light feathers woven into "wings". Experiments show that our WINGS comprehensively achieves superior performance in text-only under the same training condition and exceeds other equal-level MI.LMss on multimodal benchmarks. In addition, construct the Interleaved Image- TTxt (IIT) benchmark with multi-turn
2

---

evaluations towards a general modality scenario. The samples are from text-only questions to strongly image-related conversations. WINGS achieve leading performance across various vision- relevance partitions. Overall, our contributions are as follows: (1) We claim and verify the text-only forgetting phenomenon of MILLM related to the attention shift of cross-layer MI.LM-LAWS before and after the image.
(2) WINGS construct the visual and textual learners and introduce a router based on shifted attention weights for collaborative learning to compensate for attention shifts. (3) Experiments on text-only, essullquestioon-asseiiii and newly constructed Interleaved Image-Text (IIT) benchmarks demonstrate the comprehensive and versatile performance of WINGS.
2 A Closer Look at Attention Shift in Multimodal LLMs
In this section, we introduce the development from initialized LLM to MLLM. Next, we devise the MLLM-LAWS metric for representing attention shift and discuss the insights in building WINGS.
2.1
Granting Sight to Large Language Models
Large Language Models (LLMs). Even though existing Transformer-basee 120 models 20
83
100, 134
like BERT [88 and OPT 142 have demonstrated profound language understanding
capabilities, there has been recent surge in powerful Generative Pre-trained Transformers (GPT) [[O]
under the auto-regressive language modeling paradigm. Both public (55 [166 and private [
93
95
[12] solutions show remarkable progress in language comprehension and generation 99
126.
These LLMs generally exceed billion parameters, including pre-training [22] 32] 50 56
supervised fine-tuning with instructions [26] 104 1TO, 125], and reinforcement learning from human
feedback [2
96 107
152)
massive training data.
Multimodal LLMs (MLLMs). Integrating visual inputs into foundational LLMs to create MLLMs is becoming increasingly popular [18
[36]. Unlike vision-centric mul- 137 such as CLIP series 99 MLLMs aim align new modality features as the input of LLM with an additional encoder and perform multimodal question- answering 775 80
19
63
72
timodal frameworks [69]
[4] [50]. As illustrated in Figure 2 (a), it enables the com- bined training of mixed multimodal tokens, facilitating rapid deployment across various appli- 145). One example this pipeline is the LLAVA
81
128
[8) series, which integrates CL vision encoder with a linear projection to L.M Vicuna 21 and innovatively intro- duces cnstructionfoolloviii
cations 24 25
122
training data. Following this, some methods consider the richness of the vision-related training context [14. 46 62). the scaled visual backbone [52 73 79), the enhanced connectors [T4) to boost the visual effectiveness of MLLMs. Additionally, some works introduce monolithic multimodal solutions (30 88 MTI [23] Recently, some work has focused on the general capabilities of MILM, specifically their performance on new modalities without suffering catas- trophic forgetting of the text-only question-answenn
skills initially mastered by LLM For example, DeepSeek-Vi VL [86] suggests that supplementing additional text-only training data can mitigate this forgetting. Others [78 90, 114 try to incorporate interleaved visual-textual data into training to retain language knowledge. However, these methods are limited by training resources and data collection costs. We aim preserve even boost performance with text-related training data as little as possible. Some studies [66] 77) [06 113 135 143] also consider expanding the scalability of LLM, such using Mixture-of-Expert (MoE) with numerous parallel FFNs in the Transformer
38 74] 90).
block alongside a sparse gating network for efficient selection 108 148 149]. There are some
methods to configure effective information and feedback examples to enhance in-context learning abilities [31 132). These methods, however, require a massive increase in training parameters or inference costs. In WINGS, the newly designed parallel learners of Low-Rank Residual Attention (LoRRA) are similar to MoE on attention block, but with at least three orders of magnitude less in resource consumption. Compared to some LoRA-related methods [44] 47), WINGS focuses on parallel processing within the attention block rather than in certain in-block linear mappings [89] [21] particularly addressing the issue of capability forgetting in existing architectures.
2.2 Capturing the Attention Shift with MILM-LAWS
The significant decline in text-only performance is closely linked to the observed related shift during the training process. Research on mooss-modal learning [35 67 74] shows that transferring to new modalities affects feature distribution, output values, and activation levels. Considering attention
3

---

VLLM Layers
Correlation of
Before Image
After Image
Ingut Features
Ex [
Whht
Types of
E. |p[ !
0.255
watermelon7
In
+00.48
More positive correlation sugtests better text-only performance.
Tex -ealllllyy Performince
0.531
Viaar
Attiction
R.p.
proportion X-avis Loyors from shollww to deep
what
w/ Better Port
Vaner
VILM vo Worse Iect-Criy Perf
Correlation of MLW Refore and After image
(a) The mixed visual and textual fenture inputs.
(b) Multimodal LLMss
Layer-level Attention Weights (MEWS-LAWWS).
MLLM-LAWS Correlation Iusttation
Figure 2: Ilusttaatinn of mixed visual-and-textual inputs and the Layer-level Attention Weights (LAWS) with its properties. (a) The visual feature tokens from the visual encoder and projector are inserted into the textual feature sequence. (b) The attention weight proportion on textual tokens before-image, image-itself, and after-image across layers. The red curve is from the superior text-only MLLM, while the blue curve is from the inferior one. (c) Experiments on over 100 MLLMs show a positive correlation from the for MLLM-LAWS before and after the visual tokens (æ-axis) to the text-only performance of the MILM (y-axis).
weights highlight where MLLM's focus depends on visual or textual tokens for decision-makinn [8] we investigate how attention shifts among different parts of the sequences, mainly where divided by the visual feature tokens. Specifically, we study over 100 diverse MLLMs to uncover how attention is allocated to each part for text-only better MILM. We take closer look at the cross-layer dynamic curve of attention proportion on all text tokens before and after the inserted image.
For instruction x and its hidden states in MILM as h [h1 h2. hg] consisting of mixed visual and textual tokens. Let at, represent the attention weight between the 7th and jth tokens in the 1th of the L-layers MILM. We have, for Vi, Σラ= aty (b'-1) 1. As shown in Figure (a), since the sequence of flattened visual tokens is continuously interleaved with the textual sequence, we denote the index set of the visual tokens as Viseer = {Ustart, Ustant + 1.. Bend }. We refer to the textual sequence before the visual tokens as Voetore. and similarly, after the visual part as Vater. For an MLLM with L layers, we define the Layer-level Attention Weights MLLM-LAWS) as:
LAWS v, = =[ab- ab.
at.] ale. = - Σ a (b'-1)
(1)
i JEV.
where token index set V. be Visetf, Vhefone: Vaner mentioned above, and for simplicity, omit h in ahy, (-) of LAWS v.. In practice. LAWS characterizes the MLLM's attention on the current sequence Vitsetr. Vicfore- or Vatter regarding the dynamic curve over all MLLM-layers. As shown in Figure (b), the attention to the textual part initially increases and then decreases as the layers progress, while the trend for the visual one is often the opposite. We find that when the MLLM forgets the text-only instructions, the LAWS of the textual sequence after the visual ones show deviation from the initial trend of rising and then declining. This implies a shift of layer- attention in the text following the image Vaner compared to that preceding the image Viefore The dynamics labeled as 3 in Figure 2 (b) show the red curve for better text-only performance towards the worse blue one. To quantify this, we compute the Pearson Correlation Coefficient [92 between LAWS before and after the visual sequence. Formally,
Attention Shift Fx [-p (L.WSS Vietow? LAWS Vanw )] +
Studying the attention shift of over 100 diverse MI.LMs, we find positive correlation between the shift and the text-only performance degradation. In Figure 2 (c), each point represents trained

---

MLLM. We find that one reason for the poor text-only performance of MLLMs is the misalignment of textual LAws before and after the visual sequence, which largely stems from the main branch's attention block lacking sufficient capacity for continued fine-tuning [30 147 Next, We focus on how to mitigate the shifted attention weights. Starting with LAWS we give the MILM "wings".
3 WINGS: Flying to Generality with Low-Rank Residual Attention Learners
In this section, we explore a sufficiently reliable and convenient mechanism to alleviate attention shifts. Specifically, we introduce an additional attention extension structure to assist in learning the main branch's 's attention. The WINGS architecture operates intuitively by incorporating visual and textual learners designed to mitigate shifts in attention. A dynamic attention-weighte router, guided by negative feedback from biased attention weights, adjusts the outputs of these visual and textual learners. WINGS aims to excel in text-only and visual question-answering tasks with high generality. We start with the typical training pipeline for MLLM like LLaVA [80] (subsection 3.T). Following this, explore the motivation behind employing parallel modality learners and explain its implementation (subsection 3.2p. Finally, we describe the training process for WINGS (subsection 3.3).
3.1 Revisit the Training Pipeline of the MILM
Following the mainstream architecture of visual-encoder- -asdee--edddd MLLM, we take mixed visual and textual features as inputs. For a one-turn conversation, the sequence of the visual feature tokens may appear at any position in the input We represent the feature tokens as:
Xxy, xr)
b Liextual features
visual features
textual features
where we omit the superscript of layer-index 1 for the 0th layer. The Ustrr and Vend represent the starting and ending indices of the visual feature tokens, usually obtained through the vision encoder ab and projector W proj. Xy = "Wpoo proj & (Ximnge). Correspondingly. x; = the remaining 0 to Veant and vend to length s denote features the textual system prompt or user instructions. We consider the posterior of the ground-truth answer as:
Pr (Xa x) = II ((m))r)) 6p (h: [h), 1=1
(3)
Here, & represents the main branch LLM, which consists of Transformer decoder layers [19
Considering the interleaved image tokens, we omit the loss calculation for the "next visual token."
3.2 Visual and Textual Learners Weave WINGS
Motivation: Learning to mitigate the attention shift with modality-speciffi auxiliary structures. As mentioned in (subsection 2.2] MLLM-Laws demonstrates the attention shift in the sequence following the visual features. The shift results from excessive dependency on visual features. This issue may stem from the insufficient alignment within mixed inputs [5 130,, or the main branch struggles to accommodate capacity expansion during continual learning, where new modalities can obscure existing knowledge. It suggests adding small, adjustable factor to the shifted mixed modality features and regulating unnecessary fluctuations in MLLM-LAWS, Consequently, we aim to adopt an efficient, Iearnable module as the visual "wing". Compared to the image-text mixed feature inputs of the main branch, it should focus specifically on extracting visual information to share the burden overly shifted attention. The interaction between the current hidden state and visual features is conducted within this module. Similarly, to balance the auxiliary function of the visual learner, we also construct a symmetrical textual learner. Moreover, is crucial to appropriately distribute the two leamners across both modalities ensure they function collaboratively.
Structure:
parallel learner of attention & token-wise router of attention outputs. To capture key information in shifted modalities while ensuring efficiency, we design a multihead Low-Rank Residual Attention (LoRRA) learner at every layer. It takes input from the hidden state and interacts with the initial visual or text-only feature. The learner facilitates cross-cascading with the initial
5

---

types of wetermelon include: Red Yellow, Seeded Seedless,
Norm FPN
Visual
Textual
Multi-Head Self-Attention
Learner
Learner
Weighted Elementtwisse Add
Multi-Head Selt Aet-Aittenionn
Learner
earnnr
Norm
Welghiodd Slementoonee Add
Multi-Head Sett Attenttnn
Learn
carnnr
Visual Features
Language Features
000000
Mixed Feature Inputs
What the types of watermelon? In this <image>, what kind it
Figure 3: The WINGS model architecture. We introduce extra modules parallel to the main attention, serving as boosted learners to compensate for the attention shift. We train the visual learners side, alleviating some shifted attention. Then, collaboratively learn visual and textual learners based on routing shifted attention weights. They are like light feathers woven "wings".
projected information. Specifically, for the layer, the visual/text-only learner is formulated as: h1 (1+ (1++9) ( (x. (1 + (L++"))) (1+ WY) wo
Learner" (Q=h', K.V=x. .E(V.T)
Softmax
Vdhead
(4) where the matrix Wº WK WV and Wo is low-rank and obtained by the dot product of W. E Bddd and W, E Rddd and is relatively small enough. The symbol is represented as the identity matrix. The structure of multihead LoRRA preserves the effectiveness of the cross-attention structure and employs efficient low-rank mapping to reduce computational demands. Following LoRA 47.. LoRRA learners also employ random Gaussian initialization for Wa and sets W, to zero. Since W lacks a residual, the output of LoRRA is zero at the beginning of training, As shown in Figure 3 the visual and textual features are fed into their respective side learners, like two "wings" woven together. The outputs of two learners from each layer are then weighted sum the attention of the main branch. As illustrated in the left part Figure 4] a router receives attention weights to generate the balance weights of visual and textual learners for each token. In summary, formulate the WINGS block as:
Att main
Att
Router (a) Learner* (h', x.)
(5)
(VEVV
where a 8 represents the attention weights of the current main branch. The router is formalized as Router (a) W|:: s)-aT, which is implemented by single-layer dynamic MLP, W 2 It receives the attention weights a and processes them using Softmax on two modality learners.
3.3 Stable Training Recipe
The architecture of WINGS comprises four elements: vision encoder, projector, initialized LLM,, and the learners with router. During the training process, the vision encoder is consistently fixed. Firstly,

---

Attention
W:
Intulization:
Router
Add
Routter
W. fandom
Wi:Den
Learner
Learner
Scaled mot-Frodutt Attention
LLM
LLM
weggtt
Learner
Learnr
Attention
11
Projectio Encoder
Projector Encodder
Xxy Nog
Key
Query
(b) Training Paradigm
Xy
Stage-1: I: align projector and learn visual learners Stage:2: bune L.M with rouding visual and textual learners
Visual Features
Hidden States
(a) Low-Rank Residual Attention
Figure 4: Illustrations of the detailed WINGS structure, and training strategies. WINGS is constructed by the Low-Rank Residual Attention (LoRRA) module where the previous hidden state acts as the query and the visual/textual features serve as the key and value. Training starts with visual learners and projectors, followed by the dynamic attention-based routing.
we only fine-tune the projector and visual learners. We primarily employ image-text pairs for visual alignment, while the outputs of visual learners are directly added to the main branch. For this part, the visual learners primarily handle the visual focus, minimizing disturbances to the main branch during continued learning. Subsequently, textual learners are paralleled with visual learners on the attention block of LLMs. The router begins by learning to allocate visual and textual learners from the attention weights of the main branch, At this stage, both types of learners work more effectively together to focus attention on key tokens. To summarize, WINGS prioritizes enhancing visual learners first. Subsequently, "spreads its wings" by concurrently learning and routing visual and textual learners based on shifted attention weights. During inference, the routed weights of the visual wings branch are deactivated for text-only instructions, while multimodal instructions activate both wings.
Experiments
In this section, first introduce the benchmarks for evaluating WINGS, including Table text-only forgetting on the same multimodal training data, Table 2) comparison with general MLLMs, and Figure analysis the Interleaved mmate-Text (IIT) benchmark with varying levels of vision-related conversation. Following that, we outline the training details and configurations of the WINGS, and delve into experimental analysis across each benchmark. Following that, we perform ablation study on various learning rates with different training parts. Finally, we provide supplementary descriptions regarding WINGS' overhead compared general MILMs and how its innovative compensatory learners help effectively mitigate attention issues.
Evaluation Setups. We aim to assess through MLLM how much visual information is required for evaluation. For example, generic multimodal instructions require MLLMs to strongly capture image aspects, whereas text-only instructions focus the text. We introduce three types of benchmarks:
Standard text-only benchmarks. We are particularly interested in the text-only performance improvement of WINGS under the same training data and resource conditions. Different datasets including interdisciplinary exams like MMLU [43], CMMLU (65), ARC-Easy, ARC-
Challenge (27), language understanding and knowledge such as WinoGrande [03]. Open-
bookQA [8] Race-Middle, Race-High [60], WSC [129], CHID [144), reasoning such as Hel-
laSwag 139 SIQA [055. PIQA D. OCNLI [48], and math and code-related tasks such as
GSM8K 28 and MBPP 4 are comprehensively evaluated.
General multimodal benchmarks. We evaluate on MMMU [38., MME [37]. MMBench
(MMB) in English (EN) and Chinese (CN), ScienceQA 87 for test (SciQA), SEED-Bend for image part (SEED), A12D (57) for test, and HallusionBench 41 (HalIB).
64
Our Interleaved Image- Text (IIT) benchmark with diverse text-only, interleaved, and image- ralated multi-ture conversations. includes sampling for MMLU, CMMLU, OpenbookQA, HellaSwag, MMMU, MMBench, SEED-Bench, and AI2D datasets.
7

---

Vicuma
LoRAyto
Quen
Qwen
LoRAg
Qwen
WINGS (Ours)
Text-anly Forgetting
Our
Dataset
LLM
CLIP
CLIP
SigLIP
LLM
CLIP
CLIP
SigLIP
Impro
MMLU
51.18
51.12
48.89
50.63
60.86
50.83
59.67
51.16
60.53
9.70
9.37
CMMLL
38.60
38.29
37.24
38.73
69.37
67.87
60.46
69.82
8.91
9.36
Exam
ARC-E
57.62
53.63
55.82
53.95
59,96
56,93
59.35
55.87
54,29
4.09
-1.58
ARC-C
33.75
34.60
34,68
35.17
38.90
39.14
38,6 64
39.50
43.39
-0.60
3.89
Winogrande
68.01
64.97
67.83
65.21
71.38
69.82
71.03
69.05
69.28
2.33
0.23
OpenbookQA
77.10
73.28
77.15
72.12
81.73
78.31
81 29
77.51
81.05
4.22
3.54
Under-
Race Middlee
63.99
60.10
62.84
59.45
74.82
68.25
72.06
68.34
74.24
6.48
5.90
standing
Race-High
58.74
53.24
54.91
52.69
71.05
59.20
65.67
57.72
69.62
13.33
11.90
WSC
51,30
47.21
51,06
47.72
56.17
54,18
$7.30
55,23
66.35
0.94
11.12
CHID
39.05
49.66
45,26
53.49
71.94
71,82
72.92
74.29
74.06
-2.35
.0.33
HellaSwag
63.11
63.08
62.58
63.02
65.70
61.90
64.32
63.24
65.12
2.46
1.88
SIQA
42.37
44.06
43.27
44,52
45.57
50.20
46.83
51.71
49.64
-6.14
-2007
Reasoning
PIQA
71.92
71.95
70.35
71.84
76.59
74.60
73.77
75.19
78.06
1.40
2.87
OCNLI
33.89
37.74
39.41
40.46
49.73
48.31
48.07
50.29
50.39
-0.56
0.10
Math
GSMSK
25.19
23.72
22.68
23.05
56.77
50,10
54.25
51.37
52.08
5,40
0.71
Code
MBPP
13.80
11.29
13,92
10.80
37,50
34,82
36.72
33,20
38.92
4.30
5.72
MMMU-VAL
35.67
30.78
35.56
34.56
32.33
35.11
39.89
4.78
MMMU TEST
34.40
30.90
35.33
34,90
31.80
35.10
37.30
2.20
Multimodal
MMBench
63.18
59.83
65.14
66.05
62.84
70.94
70.53
-0.41
ScienceQA
71.50
69.09
74.89
78.76
3.87
Table 1: Performance comparisons of WINGS and the baseline MLLMs under the same training data. We consider 8 baseline MILMs, including LLMs as Vicuma.1.5 & Qwen1.5, visual encoders as CLIP [99] & SigLIP [140), and training strategies as full-parameter & LoRA fine-tuning. The first entry represents the initial LLM, upon which each MLLM is trained. Our evaluation spans 6 domains with 20 datasets. WINGS is based on the Qwen 1.5 and SigLIP, and the column "Our Improvement" highlights how much W WINGS surpasses its baseline with the same backbones.
Model Summaries & Implementation Details. We release the WINGS and WINGSpro- with Qwenl.5-7B LLM (6) and SigLIP [14) visual encoder the foundations. We also introduce the WINGS1 8B version, adapted to DDwel1.5-1.8 LLM for edge device compatibility. As illustrated in Figure 4 we only optimize the projector and the image learners of W WINGS for the first alignment stage. The LLM branch adaptation is incorporated during the second instruction tuning stage. We train for 1 epoch with the AdamW optimizer and the Cosine learning schedule. Typically, the learning rates for the first and second stages are set at le-3 and 2e (with the projector part as le 1e-"), respectively. For WINGSpase> approximately 1m training data align image learners and about 0.6m supervised fine-tuning instructions for the next stage (the same as LLaVA [80). In the WINGSpro- we use the same aligned data and approximately 2m training data for learning image-text learners. These two types of MLLM require about 1.5 and 6 days of training on 8x A100 GPUs, respectively. The training datasets for WINGSmini are consistent with the WINGSpro- takes approximately 5 days to run on 4x A100 GPUs
Details in F Figure 2 We adopt various multimodal to text-only sample ratios (25: 1, 20: 10: 5: 5, 2: 1:1, 1:2,
1:25) plus an all:0 : setup (12 combinations total) to ensure sufficient scenarios. The learning rate is kept consistent with the setup described above. We sample 5 models per epoch, excluding 12 failed ones due to issues like gradient explosion, resulting in 108 for analysis.
4.1 Toward Comprehensive Text-only and Multimodal Performance
Text-only Comparison in Fair Data and Resource Environments. As shown in Table 1 "Vicuna- v1.5 + CLIP" corresponds to LLa LL and "Qwen 1.5 + SigLIP" serves as the foundation for WINGS. When comparing LLM itself and the rest of MLLMs, we observe that fine- -eeittnnngg with multimodal instructions, compared to the "Qwen LLM", there is text-only forgetting in 12 out of 16 datasets, with notable decreases of up to 9.70, 8.91, and 13.33 in MMLU, CMMLU, and R RACE-High, respectively. WINGS significantly improve performance on datasets such as MMLU, CMMLU, RACE- and WSC, despite the potential for severe text-only forgetting on basclines. Additionally, we find that
8

---

Dataset
Text-Only QAs
Multimodal QAs
Method
MMLAUC RACE-MH ARC HellaSwang Winog: GSM8K MBPP
MMMUU-VT MMB-ENNCN MME SeiQA SEED AIDD
Hall
Equal-Scale Open-Source TB Multimodal LLMs
O O.Plmming 26.9/27.1
40.3 / 32.6
31.0
55.4
58.3
10.2
9.1
29.1 28.7
10.9/13. 13.3
803.9
55.8
30.2
32.6
30.4
IDEFICS
36.0/
36.9
11.7
17.67 20.2
24.6
InsinccBLIIP
A0
43.2/35.7
52.8/49.7
39.5
55.7
54,9
18.3
10.3
32.7 /3.1
38.5 26.8
1425.6
61.3
45.7
41.1
33.1
ShareGPT4V
47.6736.9
55.9/510
41.6
18.0
35.5
67.47 63.1
1915.3
68.9
26.6
Qwen-VL 7
/8.3
6
34.4
58.2
61.0
49.0
34.6
35..9
1574
1806.2
69.6
62.0
61.9
34.1
Monkey
66.8/66.9
62.6
38.2
51.8
37.1
40.3
7
1815.
78.3
62.5
42.1
1LaVA,15
80
51.1/38.3
60.1 53.2
34.6
63.1
65.0
23.7
11.3
35.7 34.4
63.2 57.7
1518.6
67.7
63.7
56.4
29.7
LLaVANe
81
50.2 39.7
65.1 25.1//58.1
36.0
63.7
68.9
30.3
23.0
37.6/35.8
67.8
61.8
1760.3
70.1
69.1
66.4
29.6
DeepSeck VL 86
53.9 64.0
70.6 63.8
39.2
65.1
67.2
55.3
43.1
72.7 37.6
72.5
1716.8
80.6
70.0
66.5
36.2
WINGS (Ours)
60.5 69.8
74.2/6691
43.4
65.1
69.3
52.1
38.9
3379
70.5
68.3
1753.8
78.8
69.5
62.7
45.8
(ONGS)ro(Ouus)
61.3/ 68.5
/6.3
46.3
69.2
70.9
56.3
39.3
38.2 36.9
73.1 / 69.0
1786.1
83.1
70.2
65.8
47.3
Advanced Private Multimodal LLMs
GPT-4 95
83.5/
93.2 87.8
93.6
88.4
75.6
91.6
56.21
GPT-VV 3+
79.3 69.4
93.7/
89.2
92,9
84.7
76.1
88.4
72.4
58.9/56.8
77.0 74.4
2153.6
68.4
73.7
75.5
46.5
Geminigo vises
102
$5.9 73.7
88.9/83.2 83.2
85.0
78.8
71.5
86.4
61.5
60.6
62.2
74.6
2193.2
58.3
70.8
70.2
45.2
Efficient Multimodal LLMs with WINGSLA8
DeepSock-VL,
31.7 38.2
63.6 58.4
35.8
52.9
45.7
17.6
16.3
33.8/32.3
65.1 60.7
1483.4
65.4
63.3
50.1
25.0
MincCPM-V3.48 2m
42.4
62.6
37.0
51.7
32.5
24.2
37.2
31.8
WINGS Ours)
4 50.9
68.5
63.2
37.1
50.5
53.0
40.6
33.7/33.9
64.2/ 61.2 1527.3
67.5
55.2
30.2
Table 2: Performance comparisons of the equal-scale MLLMs and the efficient multimodal LLMs on text-only and multimodal datasets. We evaluate the open-source, efficient, and private API MLLMs. We select 18 representative evaluation datasets. C* represents the CMMLU dataset.
the forgetting effects of CLIP and SigLIP are similar. In contrast, parameter-efficient fine-tuning methods like LoRA result in less text-only forgetting but underperform on multimodal questions. Overall, WINGS' visual and textual learners are credibly demonstrated to retain performance text-only tasks while also performing well on visual-related questions. In datasets like CHID, OCNLI, and SIQA, MLLMs show improved text-only performance due to increased language diversity (e.g.. Chinese context) or semantic similarity in their fine-tumin tie-ttning data.
General Evaluation in Text-Only and Multimodal Tasks. We present the performance of 9, roughly open-source MILMs, 2 roughly 2B, and 2 private API ones evaluated the general text-only and multimodal tasks. Table 2 shows that WINGS series can perform better on text-only and multimodal question-answenting datasets. achieves wtate-of-the-an performance on out of 18 datasets, significantly surpassing LLaVAv1. with the same architecture. We find that WINGS is equally effective for more efficient foundations, as shown in the "Efficient Multimodal LLMs" parts. WINGS can still capture key elements and demonstrate good scalability the parameter increases. Although WINGStase does not receive additional training for the text-only component, it still able to achieve comparable performance.
4.2 Interleaved Image-Text (IIT) Benchmark
To finely evaluate MILMs, we construct a series of text-only and multimodal mixed multi-turn conversations. We extract instructions from MMLU, CMMLU, OpenbookQA, HellaSwag, MMMU, MMBench, SEED-Benche and A12D datasets with similar semantics by chroma [39]. We then polish connection between some instructions using GPT-3.5 Turbo to make them closer to real-world conversations. We set up 6 vision-content configurations, categorized by the multi-turn content as: (T), (T, T), (T, T, T), (T, T, V), (T, V), and (V). For instance, (T, T, V) indicates two consecutive text-only queries followed by visual question requiring response.

---

75
71.8 TIL6
68.4
6K4
67.4 67.1
65
65
61,9
655
68
68
60
53.0
51.8
53.8
51.4
0.1
$8.1
50
4K
49.
47.3
45
42.
45
44.
40
(2.21
(2.2.1) (3.2.8)
(8.01
(1)
(Qwen SIgLIP)
Winns (Ours)
Smiller
Larger
Partitioned
Oaly Visual Winzs
Ouly Wings
(Oura)
Comparison of LL Series and WINGS
Companion
Vario Fasming Rate
WINGs
Companson of Differant Triinin Parts WixGs
Figure 5: Performance comparison the newly constructed Interleaved Image and Text (ITT) Benchmark of the L.a.V. VA series, different learning rate and fine-tuning parts. The horizontal axis represents different multimodal question settings. The horizontal axis shows different multimodal setups, e.g., (T, T, I) represents a visual question after two text-only QAs. The three subfigures represent different ablation settings, with the violet color representing our WINGS.
4.3 Ablation Studies
Referencing Figure 5 we address three questions to comprehensively analyse WINGS:
Can WINGS sustain performance with interleaved evaluation? We find that part (a) highlights WINGS surpassing LLaVAy1.5 and the same backboe as LLaVA (Qwen- (QgLi-S for each multi-turn setting, especially text-centric dialogues.
How do WINGS fare with different learning rate settings? Part (b) demonstrates that using a lower learning rate maintains proficiency in text-only tasks but falls short multimodal questions, while higher rate boosts multimodal abilities but not text-only. Applying higher learning rate to the projector and lower one the others achieves the optimal.
Are all components of WINGS equally effective? In part (c), we examine that incorporating visual learners alone slightly preserves text-only abilitics, likely by minimizing disruption the LLM, but diminishes performance on multimodal tasks.
In the diverse IIT bench, which ranges from text-rich to multimodal contexts, the effectiveness of WINGS is particularly evident. As shown in Figure I] within real-world applications, textual content offers insights for following visual tasks. WINGS excels in handling text-only tasks while improving performance on visual-related instructions.
5 Conclusion
We propose WINGS, which includes visual and textual learners, to alleviate text-only forgetting. The learner composed of efficient Low-Rank Residual Attention (LoRRA). We start by considering the shifted attention weights in MLLM and, in the first stage, focus on learning the visual learner. Then, co-train the visual and textual learners with routing based the shifted attention weights. WINGS demonstrates remarkable performance on text-only, constructed Interleaved Image-Text (IIT) benchmarks. WINGS allows for maintaining text-only performance with limited resources and further enhances performance in well-resourced settings.
and newly
Acknowledgments and Disclosure of Funding
This research was supported by National Science and Technology Major Project (2022ZD01 14805), NSFC (61773198, 623761 (.61921006), Collaborative Innovation Center of Novel Software Tech- nology and Industrialization, CCF-Tencent Rhino Chino-Bird Open Research Fund (RAGR20240101)
10

---

References
[1] Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko, Karen Khatamifard. Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mchrdad Farajtabar. LLM in a flash: Efficient large language model inference with limited memory. CoRR, abs/2312. 11514, 2023.
[2] Reza Yazdami Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. Deepspeed- inference: Enabling efficient inference of transformer models at unprecedented scale. In SC22, pages 46:1 46:1-46:155 2022.
[3] Anthropic. Introducing Claude, 2023.
URL
introducing-claude
[4] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021.
[5] Anas Awadalla. Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy. Wanrong Zhu, Kalyani Marathe. Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa. Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openftamingo: An open-source framework for training large autoregressive vision-language models. CoRR, abs/2308.01 390, 2023.
[6] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:23009 16609, 2023.
[7] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A fronticr large vision- lision-language model with versatile abilitics. arXiv preprint a30iv:2308. 12966, 2023.
[8] Pratyay Banerjee, Kuntal Kumar Pal, Arindam Mitra, and Chitta Baral, Careful selection of knowledge to solve open book question answering. In ACI. pages 6120-6129. Association for Computational Linguistics, 2019. doi: 10. 1158///1//P19-1611 /ttpss//::///oi org/ 10. 11653/v1//19-- 1615
[9] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In AAAL, pages 7432-7439, 2020.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 33:1877-1900 1901, 2020.
[11] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced projector for multimodal 1lm. arXiv preprint arXiv:2312.06642. 2023.
[12) Yingshan Chang, Guihong Cao. Mridu Narang, Jianfeng Gao, Hisami Suzuki, and Yonatan Bisk. Webqa: Multihop and multimodal QA. In CVPR, pages
IEEE, 2022.
[13] Guiming Hardy Chen, Shunian Chen, Ruifci Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Hamessing gpt4v-synthesized data for lite vision- language model. arXiv preprint arXiv:2402.11688 2024.
[14] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. SharegntAv: Improving large multi-modal models with better captions. arXiv preprint 1xXi:23311.1255 2023.
[15] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vasion-language models? arXiv preprint wwXiv:2403.20334 2024.
[16] Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and
C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015.
[17] Yunkai Chen. Qimeng Wang, Shiwei Wu, Yan Gao, Tong Xu, and Yao Hu. Tomgpt: Reliable text-only training approach for cost-effective multi-modal large language model. ACM Trans. Knowl. Discow, Data, 2024.
[18] Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Zhong Muyan, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual linguisic tasks. arXiv preprint 2322 2023.

---

[19] Zhe Chen, Weiyun Wang, Hao Tian, Shengiong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far we gpt-4v? closing the gap to commercial multimodal models with open-source suites, arXiv preprint arXiv:2404. 16821, 2024.
[20] Po-Han Chi, Pei-Hung Chung. Tsung-Han Wu, Chun-Cheng Hsieh. Yen-Hao Chen, Shang-Wes Wen Li, and Hung-yi Lee. Audio albert: A lite bert for self-supervised learning of audio representation. In IEEE SLT. pages 344-350, 2021.
[21] Wei-Lin Chiang. Zhuohan Li. Zi Lin. Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez. lon Stoica. and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. httt::tttt::mmmm org/ org/blog/ /log/ 2023-03- 30---iivviaaa
[22] Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240: 14:240:1-220:::1 1-240: 113, 2023.
[23] Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforeement learning from human preferences. In NewrIPS, pages 4299 03079-43722220 2017.
[24] Xiangxiang Chu. Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang. Bo Zhang, Xiaolin Wei, et al. Mobilevim: A fast, reproducible and strong vision language assistant for 2023.
mobile devices. arXiv preprint arXiv:2312.
[25] Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun. Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint xxiv:2440..0376, 2024.
[26] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction -innctuin-finetuuee language models. CoRR, abs/2210.11416, 2022.
[27] Peter Clark. Isaac Cowhey, Oren Etzioni. Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the A12 reasoning challenge. CokR, abs/1 abs/1803.05457, 2018.
[28] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
[29] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose purpose vision-language models with instruction tuning. NeurlPS, 36, 2024.
[30] Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling ercoder-free vasion-language models. CoRR, abs/2406. 1 11832, 2024.
[31] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. /4233, 2023.
[32] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baohao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey for in-context learning. CoRR, abs/23011.022 2023.
[33] Xiaoyi Dong. Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm- -xcopposxx Mastering form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:240I. 16420, 2024.
[34] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In ACL, pages 00-33,,,555 2022.
[35] Haoyi Duan, Yan Xia, Mingze Zhou, Li Tang, Jieming Zhu, and Zhou Zhao. mross-modal prompts: Adapting large pre-trained models for audio-visual downstream tasks. NeurlPS, 2023.
[36] Wenfeng Feng. Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mfxture-of-loras: An efficient multitask tuning for large language models. CoRR, abs/2403.0344 2024.
[37] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang. Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint 231v:2306.1394. 2023.
12

---

[38] Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan A. Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-inage pretraining. In NeurIPS, 2022.
[39] Chroma Group. Chroma the open-source embedding database. htpps//gghhuubcoomm/cmommooooooo
2017.
[40] Poe Group. Poe- fast, helpful ai chat. https://poe.com, 2024.
[41] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang. Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: advanced diagnostic suite for entangled language hallucination & visual illusion in large vision- laiiual models. arXiv preprint arXiv:2310. 14566,, 2023.
[42] Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmente large multimodal model for long-term video understanding. In CVPR, 2024.
[43] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint wrXiv2209.03300 2020.
[44] Zhang Hengmin, Yang Jian, Du Wenli, Zhang Bob, Zha Zhiyuan, and Wen Bihan. Enhanced acceleration for generalized nonconvex low-rank matrix learning. Chinese Journal of Electronics, 34(1):1-15 2025. doi:
[45] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang. Yuxiao Dong Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.000 089114 2023.
[46] Anwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye, Qinghao Ye, Ming Yan, Chenliang Li, Qi Qian, Ji Zhang, and Fei Huang. arXiv preprint arXiv:2311.18244 2023.
Scientific diagram analysis with the multimodal large language model.
[47] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[48] Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence S. Moss. OCNLI: original chinese natural language inference. EMNLP, volume EMNLP 2020 Findings of ACI. pages 3512-3526, 2020.
[49] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang. Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.006 2404,06395 2024.
[50] Jie Huang and Kevin Chen-Chuan Chang. Towards rcasoning in large language models: A survey, In ACL, 2023.
[51] IDEFICS. Introducing idefics: An open reproduction of state-of ofate-oftthe-art visual language model. https / //huggingface haaggginggaaeecooboo///eeeeeeeee ht///bllogidefcccc/ 2023.
[52] Jitesh Jain, Jianwei Yang, and Humphrey Shi. Veoder: Versatile vision encoders for multimodal large language models. CoRR, abs/2312. 14233, 2023.
[53] Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Lei Zhang, Baochang Ma, and Xiangang Li. Exploring the impact of instruction data scaling on large language models: An empirical study on real-world use cases, arXiv preprint 233v 14742, 2023.
[54] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[55] Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401 040888 2024.
[56] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray. Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, Abs/2001.08361, 2020.
13

---

[57] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo. Hannaneh Hajishirzi, and Ali Farhadi. A diagram worth dozen images. In ECCV, pages 235-251, 2016.
[58] Jacob Devlin Ming-Woi Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep hidirec- tional transformers for language understanding. In NAACI, pages 4171-4186, 2019.
[59] Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Won- seok Hwang, Sangdoo Yun. Dongyoon Han, and Seunghyun Park. Der-free document understanding transformer. In ECCV, 2022.
[60] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. RACE: large-scale reading comprehension dataset from examinations, In EMNLP, pages 785-794, 2017.
[61] Hugo Laurençon. Lucile Saulnier. Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kicla, Matthicu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
[62] Bo Li. Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang. Fanyi Pu, and Ziwei Liu. Otterhd: A high- resolution multi-modality model. arXiv preprint arXiv:2311.04211 2023.
[63] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang, and Ziwei Liu. Otter: A multi- modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03722 2023.
[64] Bohao Li, Rui Wang. Guangzhi Wang. Yuying Ge, Yixiao Ge, and Ying Shan. benchbench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv: 16125, 2023.
[65] Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin, CMMLU: Measuring massive multitask language understanding in Chinese. arXiv preprint arXiv:2306.092/1 2023.
[66] Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. Cumo: Scaling multimodal Ilm with co-upeycled mixture-of-experts. arXiv:, 2024.
[67] Jumnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu- Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, pages 9694 9705, 2021.
[68] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICML, pages 2288-112000 2022.
[69] Junnan Li. Silvio Savarese. and Steven C. H. Hoi. Masked unsupervised self self-training for zero-shot image classification. CoRR, abs/2206. 022977 2022.
[70] Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. CoRR, abs/2403.00231, 2024.
[71] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL/JUNNP,, pages 4582 4597, 2021.
[72] Yanwei Li, Yuechen Zhang, Chengyao Wang. Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 11.400.8884. 18814, 2024.
[73] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yaho Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311. 1..2231110066.... 2023.
[74] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung. and James Y. Zou. Mind the gap: Under- standing the modality gap in multi-modal contrastive representation leaming. In NeurlPS, 2022.
[75] Bin Lin, Bin Zhu, Yang Ye. Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.1011 2023.
[76] Bin Lin, Tao Peng, Chen Zhang, Minmin Sun, Lambo Li, Hanyu Zhao, Wencong Xiao, Qi Xu, Xiafei Qiu, Shen Li, Zhigang Ji, Yong Li, and Wei Lin. Infinite-lIm: Efficient LLM service for long context with distattention and distributed kvcache. CoRR, abs/2401.02669 2024.
[77] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Mlvaal Mixture of experts for large vision- models. arXiv preprint 401 2024.
14

---

[78] Ji Lin, Hongxu Yin, Wei Ping. Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao. Jan Kautz. Mohammad Shoeybi, and Song Han. VILA: on pre-training for visual language models. CoRR, abs/2312.07533. 2023.
[79] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wengi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:231 1.07755, 2023.
[80] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2023.
[81] Haotian Liu, Chunyuan Li. Yuheng Li, Bo Li. Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URI. httpps://llaaa-vll.. gith io/ blog/2024- 202//0222000000 001 301-30-11 aalvv-nnett///
[82] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, and Chunyuan Li. Llava-plus: Learning to use tools for creating multimodal agents. 1eXiv:2311.05433 2023.
[83] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zetteemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907 1692, 2019.
[84] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model all-around player? arXiv preprint arXiv:2307.0m 06281, 2023.
[85] Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr- free large multimodal model for understanding document. arXiv preprint arXiv:2403.044 04473, 2024.
[86] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: Towards real-world vision-language understanding. arXiv preprint 2xXii224330052555 2024.
[87] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurlPS, 35:2507-255 2521, 2022.
[88] Shiyin Lu, Yang I., Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. CoRR, 0bs/2405.207997 2024.
[89] Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono- internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-trainin training, 2024.
[90] Brandon McKinzie, Zhe Gan, Jcan-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal I1m pre-training. arXiv preprint 24100,,00 2024.
[91] Shervin Minaee, Tomás Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey. CoRR, abs/2402.06116, 2024.
[92] Cuong V Nguyen, Tal Hassner, Cedrie Archambeau, and Matthias Seeger. Leep: new measure to evaluate transfcrability of learned representations. /CML, 2020.
[93] OpenAI, Chatgpt.
2022.
[94] OpenAI. Gpt-4v(ision) system card. httt::://cch.ooenaa..o.... http://pw.........po/e/t///////a pdf 2023.
[95] OpenAI, GPT-4 technical report. CoRR, wtw.aaa3..23077....... 2023.
[96] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instruc- tions with human feedback. In NewrIPS, 2022.
[97] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao, Instruction tuning with g. arXiv preprint 2rX1v:2304.03277 2023.
15

---

[98] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, pages 3942-3951, 2018.
[99] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages
2021.
[100] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The J. Mach. Learn. Res., 21(():5485- $551, 2020.
[101] Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero-infinity: breaking the GPU memory wall for extreme scale deep learning. In International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, November 14-19, 2021, page 59, ACM, 2021.
[102] Machel Reid, Nikolay Savinov. Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jcan- caan-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieserr, el. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.00 05553000 2024.
[103] Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019.
[104] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai. An- toine Chaffin, Arnaud Stiegler, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In ICLR, 2022.
[105] Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. SociallQA: Commonsense reasoning about social interactions. CoRR, abs/1904.09728, 2019.
[106] Noam Shazeer, Azalia Mirhoseini. Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff' Dean. Outrageously large neural networks: The sparsely- gated mixture-of experts layer, 2017.
[107] Nisan Stiemmon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[108] Zai-Hao Tan, Jian-Dong Liu, Xiao-Dong Bi, Peng Tan, Qin-Cheng Zheng, Hai-Tian Liu, Xie, Xiao- Chuan Zou, Yang Yu, and Zhi- -ua Zhou. Beimingwu: A learnware dock system. In KDD, 2024.
[109] Rohan Taori, Ishaan Gulrajani. Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followine Ilama model. httts:///githbb com/tatsu 1ab/ sttsssanfoddaaaaaaaa ppaca 2023.
[110] Rohan Taori, Ishaan Gulrajani. Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-follovingg model. Stanford Center for Research on Foundation Models, https:/erffm. stanford. edu/2023/03// 3///2022/33///aappcaa html, 3(6)7, 2023.
[111] Chameleon Team. Chameleon: Mixed-modal early- ffsiin foundation models. CoRR, abs/24455009811 2024.
[112] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 11800, 2023.
[113] The Mosaic Research Team. Introducing dbrx: A new state-of-the open 1lm. March 2024. URL https://vww. hatp:/ddaaabiicck.ck
www.aaabr..c...ooooooooooo..o.o. www.///oooo/cocecceceecooooeeeee inw.oduciooodnnnoooooo dbrx- new staussstae-aa state-art-tttteeeeeee open- 11m
[114] Fu Tianhao, Yang Zehua, Ye Zhisheng, Ma Chenxiang, Han Yang, Luo Yingwei, Wang Xiaolin, and Wang Zhenlin. A survey on the scheduling of dl and IIm training jobs in gpu clusters. Chinese Journal of Electronics, 34:1-25, 2024, doi: (o.23919/cj..224 2004.00.7...
[115] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anee Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[116] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 207.0200 2023.
16

---

[117] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals. and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34: 200-212, 2021.
[118] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter effi- cient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558, 2022.
[119] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017.
[120] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NewrIPS, 30, 2017.
[121] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong. Ming Ding, and Jie Tang, Cogylm: Visual expert for pretrained language models. CoRR, abs/231 .03079, 2023.
[122] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, et al. Cogvim: Visual expert for pretrained language models. arXiv preprint vrXiv2311.00079 2023.
[123] Xinlong Wang. Xiaosong Zhang, Zhengxiong Luo. Quan Sun. Yufeng Cui, Jinsheng Wang. Fan Zhang, Yueze Wang. Zhen Li. Qiying Yu. Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, I Liangdong Wang, Guang Liu, Zheqi Hc, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need. CoRR, abs/2409. 18869, 2024.
[124] Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge Jinrong Yang, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. Vary: Scaling up the vision vocabulary for large vision-languee models. arXiv preprint arXiv:2312.06109, 2023.
[125] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In ICLR, 2022.
[126] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022.
[127] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang. Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models, arXiv preprint arXiv:2303.04671, 2023.
[128] Shengqiong Wu, Hao Leigang Qu. Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. arXiv preprint arXiv:2309.055/9, 2023.
[129] Liang Xu, Hai Hu, Xuanwei Zhang. Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang. He Zhou, Shaoweihua Liu, Zhe Zhao. Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. CLUE: A chinese language understanding evaluation benchmark. In COLING, pages 1762-4772 2020.
[130] Shipeng Yan, Jiangwei Xie, and Xuming He. DER: dynamically expandable representation for class incremental learning. In CVPR, pages 3014-3023, 2021.
[131] Xu Yang, Yongliang Wu, Mingzhuo Yang. Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. In NeurIPS, 2023.
[132] Xu Yang. Yingzhe Peng, Haoxuan Ma, Shuo Xu, Chi Zhang, Yucheng Han, and Hanwang Zhang. Lever Im: Configuring in-context sequence to lever large vision language models, 2024.
[133] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 20..2.333..88 11381, 2023.
[134] Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019.
17

---

[135] Chao Yi, De-Chuan Zhan, and Han-Jia Ye. Bridge the modality and capacity gaps in vision-language model selection. CoRR. abs/2403.13799, 2024.
[136] Alex Young, Bei Chen, Chao Li, Chengen Huang. Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, al. Yi: Open foundation models by 01. ai. arXiv preprint 44X.4:2403.04652. 2024.
[137] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 2021.
[138] Xiang Yue, Yuansheng Ni. Kai Zhang. Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311 16502, 2023.
[139] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In ACL, pages
2019.
[140] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- pre-training. In ICCV, pages 11975 11966, 2023.
[141] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li. et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv preprint ar0iv:2402. / 12226, 2024.
[142] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv:2205.0100
2022.
[143] Yi-Kai Zhang, Ting-Ji Huang, YianXi Ding, De-Chuan Zhan, and Han-Jia Yc. Model spider: Learning rank pre-trained models efficiently. In NeurlPS, 2023.
[144] Chujie Zheng, Minlie Huang. and Aixin Sun. Chid: A large-scale chinese idiom dataset for cloze test. In ACL. pages 778-787, 2019.
[145] WANG Zhifang, ZHEN Jiaqi. LI Yanchao, LI Guoqiang, and HAN Qi. Multi feature multimodal biometric recognition based on quaternion locality preserving projection. Chinese Journal of Electronics, 28(4):789-78) 796. 2019. doi: 10. 10.1049/cje.2119.00
[146] Chunting Zhou, Pengfei Liu, Puxin Xu. Srinivasan Iyer, Jiao Sun. Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less is more for alignment. In NeurlPS, 2023.
[147] Da- Da-Wei Zhou, Fu- Yun Wang. Han-Jia Ye, Liang Ma. Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In CVPR, pages 9036-9046. IEEE, 2022.
[148] Zhi-Hur Zhou. Leamware: on the future of machine learning. Frontiers Computer Science, 10(4), 2016.
[149] Zhi Zui-Hua Zhou and Zhi-Hao Tan. Learnware: Small models do big. CoRR, abs/2210.03647,, 2022.
[150] Bin Zhu, Bin Lin, Munan Ning. Yang Yan, Jiaxi Cui, Hongfa Wang. Yatian Pang. Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, Li Yuan. Languagebind: Extending lideo-language pretraining to n-modality by language-bass based semantic alignment. CoRR, abs/2310.01852. 2023.
[151] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal C4: an open, billion-scale corpus of images interleaved with text. In NeurIPS, 2023.
[152] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Chris- tiano. and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, 190.10909999999 2019.
18

---

Supplementary Material
A Experimental Setups and Implementation Details
Training Datasets. The training datasets for the first and second stage of WINGSpase are consistent with LLa LLaVA.1. [80]. For the second stage, WINGSpro extends the training dataset to include some visual QA datasets as ALLaVA [13], SonthDog [59]. and ArXivQA [70], and text-only QA datasets
Stanford Alpaca [109], Alpaca GPT-4 [97]. LIMA [46]. UltraChat [TI. WebQA [2], and
BELLE-0.5M 53 WINGS shares the same training set as WINGSpro-
Model Structures. We employ Qwen1.5 .5 6 and SigLIP [40] as our foundations.
Training Hyperparameeers. We utilize a batch size 32, along with the AdamW optimizer and cosine schedule. For all WINGS-series, the learning rate is set at le for the first stage and adjusts 2e for the second stage, except for the projector 1e-5
Training Environment. WINGSpase and WINGSpro are trained over approximately 1.5 or 6 days on 8x A100 GPUs. WINGS1.8 require approximately 5 days of training on 4x A100 GPUs.
B Additional Experimental Results
70.5
70.3
70
67.7
68.4
05.0
6.31
55
51.8
50
45.3
46.3
47.5
46.5
45
40
(70
(7,19
(T.7.1)
(T.7.V)
(T.9)
(V)
LLRRA
Wings (Ours)
Comparison of Parameter Efficient Modules and WINGS
Figure 6: Performance comparison on the newly constructed Interleaved Image and Text (IIT) Benchmark of the Parameter Efficient Modules. The horizontal axis represents different mul- timodal question settings. The horizontal axis shows different multimodal setups, e,g., (T, T, I) represents a visual question after two text-only QAs.
Should we only add additional modules on top of an LLM branch or, like WINGS, create two distinct learners for visual and textual modalities? We delve into the low-rank adaptation (LoRA) 47 and Prefix-tuning [71] for minimally adapt to the LLM component. These techniques introduce optimization parameters beyond the primary branch. These lightweight adjustments align with extensive modifications, effectively minimizing text-only forgetting but concurrentyy curbing cross- modal positive transfer.
C Discussion
WINGS is universal plugin that be integrated with any multimodal mixed-input MLLMs. Notably, it introduces a new concept of competitive reuse among multiple expert groups: we may not require the experts to the Transformer block's s MLP layer scale three orders of magnitude larger; instead, a minor update in the attention for better allocation may suffice. This idea also found in some variants of LoRA 36 [ 118). In the future, will gradually explore the future of MLLMs.
Regarding the extent to which WINGS alleviates attention shifts, we acknowledge that the main branch of WINGS (without visual and textual learners) still exhibits attention shifts. However, since the outputs from the visual textual learners compensate within the hidden state, we extract portions of the attention weight matrices from both learners and add them to the main branch's 's weights. Results
19

---

show that throughout the training process, the attention shift phenomenon is continually mitigated under the influence of these learners. Essentially, the primary branch's 's attention ideally retains its original text-only capabilities, while the visual solutions are implemented within the auxiliary learner.
D Limitation & Broader Impact
Despite WINGS' strong adaptability for embedding auxiliary attention learners in various MLLMs, integrating visual learners requires restarting the feature alignment training, incurring extra costs. Additionally, its deployment on edge devices faces limitations, with WINGSJ,st ss offering a solution at the expense of performance. Furthermore, WINGS still requires some text-only data replay and enhance overall performance, aiming for integration into more generic Al systems in the future.
Figure Dynamies of Attention Weights from Shallow Deep Layers. We calculate the proportion of attention weights for the image-before (yellow), the image-itself (red), and the image- after (green) in each layer. From left right, top bottom, from shallow deep layers.
QAI: Text-only Instruction (from MMLU)
User:
There single choice question about Sociology.
Answer question by replying Question: Which the following did the post-war welfare state of 1948 not aim provide:
free health care and education for all
B. minimum wage
C. full employment
D. universal felfare
QA2: Multimodal Instruction (from MMMU)
User: Sociology studies <image> governmental
relationships
Figure 8: An Example of an Interleaved Image-Text Benchmark. This dialogue is represented as (T, V), consisting of text-only QA from MMLU 43 and visual QA from MMMU (138). It can be observed that, due to the sampling, both include questions from the Sociology category.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract reflects the contribution and scope of our work and illustrates model and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We show the architecture in Figure and the experimental details in section 4
Figure
Additionally, we provide
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [No]
Justification: The datasets and benchmarks used in our work are open source. The code of our proposed method will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /mips.ccc/pullc//guu guwde//ddee/eee wwt.sSuumisssispoooooooooooooooe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide training and test details in Section 4
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: During the training process and evaluation, we use methods such fixed random seeds to obtain certain and consistent results, so our work do not focus on the significance of the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
23

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources for the experiments is reported in section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read and strictly followed the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: discuss broader impacts in the Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risk in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have listed and cited relevant datasets and models in detail in the paper to ensure that the license and terms met.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: Our work uses public datasets for training and evaluation, and does not involve or open source new datasets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "mXA7JxHa5W", "reviewer": "Reviewer_GnKS", "summary": "The paper addresses how to solve the well-known problem of multimodal large language models (MLLMs), text-only forgetting referring to the phenomenon of MLLMs showing drastic performance drops on text-only instructions. The paper first observes based on the analysis over 100 MLLMs that the performance drop is related to attention shifts in text tokens before and after visual tokens within mixed visual-and-textual inputs. To compensate for the attention shift, the paper adds separate visual and textual learners with a router to each of the LLM blocks. The visual and textual learners are implemented with a newly designed Lor-Rank Residual Attention (LoRRA). The proposed method, Wings, achieves improvements over baselines on text-only benchmarks while achieving competitive or better performance on multimodal benchmarks, including a newly collected interleaved Image-Text (IIT) benchmark.", "questions": "- Please provide the details of 100 MLLMs explored for the attention shift analysis.\n- How can we get a, the attention weights of shape sxs from the LLM main branch? And please elaborate more on how the router operates on a, that is, more details about computing weighted sums of visual and textual learner outputs.\n- In Figure 5 (b), did the authors ablate learning rates in the second stage of training? in other words, did the authors use the same learning rate of 1e-3 in the first training stage?\n- Does the authors plan to release the second-stage training data for Wings_pro, and the new IIT benchmark?\n\nMinor comment: If the indicator variable becomes zero for any i in equation (3), the probability is zero. Is the equation (3) correct, or there are some typos?", "limitations": "The paper addressed the limitations of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The overall presentation of the paper is quite good. The analysis that explores over 100 MLLMs is really impressive. Especially, the Figure 2 effectively shows the observation that is the key to the proposed method, Wings.\n- The proposed visual and textual learners make sense based on the observation.\n- Wings achieves superior performance on extensive text-only and multimodal benchmarks.", "weaknesses": "This work is really intriguing. However, the rationale behind how the authors designed the proposed Low-Rank Residual Attention (LoRRA) as well as the ablation studies proving some critical design choices are missing.\nFor example,\n- why did the authors add the residual terms (the identity matrices in Equation 4)?\n- please provide the results without the router, if possible.\n\nAlso, please provide computation overheads of Wings in FLOPs, and how much does Wings increase training and inference time (comparison between Wings_base and the Qwen + SigLIP model)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GnKS,\n\nWe appreciate Reviewer GnKS's thoughtful feedback and support, especially regarding the motivation, structure, and performance of our Wings. In response to these valuable insights, we have conducted additional experiments and enriched our descriptions to reinforce our approach. All modifications will be highlighted in the final version.\n\n* **Q1:** \"Wings design rationale and ablation studies\",\n   * **Q1.1**: the residual terms.\n   * **Q1.2**: w/o the router.\n\n* **A1:** The primary rationale for Low-Rank Residual Attention (LoRRA) is to mitigate the attention shift seen in MLLM when integrating visual inputs (as noted in lines 147-149). The main features include:\n\n   * **Lightweight, Independent Attention Module:** LoRRA improves multimodal attention by adding **an independent learner module** that manages visual interactions with minimal parameters. It generates keys and values from initial visual features and queries from previous hidden states. This reduces the visual attention load on the primary branch, allowing for more focused processing of text parts.\n\n   * **Pre-alignment Process in the First Stage:** In the initial training stage, LoRRA is learned **on inter-modal adaptation using the learner's visual attention modules**, without the router's involvement.\n\n   * Regarding the ablation study of structural design: the construction of the LoRRA structure underwent several unreliable iterations. **In PDF Table 6**, We conducted **ablation studies focusing on the attention component**, as training MLP mappings is resource-intensive. Our results indicate that incorporating **linear mappings, learnable prompts, or dynamic mixed LoRA structures** can improve multimodal perception. However, these methods often suffer from limited learning, resulting in a trade-off where less forgetting leads to diminished learning outcomes.\n\n   **Responses to Q1.1 and Q1.2:**\n\n   * **A1.1: Table 7 in PDF** demonstrates that the **text-only input performs slightly worse than the LoRRA structure with residuals**, which enhances gradient propagation and promotes faster learning. The residual connections create a more coherent overall structure by simplifying the fully connected mappings to an identity mapping.\n\n   * **A1.2: In Table 8 of PDF**, the results indicate that random allocation (w/o a router) significantly disrupts attention and impairs performance, while 1:1 allocation reduces the effects of scale transformation. **As noted in lines 281-283 of the main paper**, the router weights are also crucial for multimodal capabilities.\n\n* **Q2:** \"details of computation overheads\"\n\n* **A2: Table 9 of PDF** summarizes the computational overheads of Wings in (Tera) FLOPS, **alongside parameter counts and TFLOPS costs compared to the Qwen + SigLIP baseline**, using measurements from the [`calculate-flops.pytorch`](https://github.com/sovrasov/flops-counter.pytorch) library with a batch size of 1 and a maximum sequence length of 2048.\n\n* **Q3:** \"details of 100 MLLMs for the attention shift analysis.\"\n\n* **A3:** We analyzed **108 models derived from** 12 multimodal training configurations (`25:1`, `20:1`, `10:1`, `5:1`, `2:1`, `1:1`, `1:2`, etc., up to `1:25` and `all:0`) with varying multimodal:text-only data ratios and 2 learning rates (1e-3 for the first, and either 2e-6 or 1e-5 for the second stage), assessing their MMLU, 5-shot performance differences in text-only and multimodal contexts, **as illustrated in Figure 2 of the main paper**.\n\n* **Q4:** \"details of attention weights\"\n\n* **A4:** Wings leverages attention weights from transformers by **enabling access during inference through `output_attentions=True`**. The attention weights are generated by performing a query-key-value operation, where the query from the LLM's main attention module multiplies with the key, is divided by the square root of the head dimension, and **is processed with a softmax function—without any dropout**. These attention weights, **shape [`number of heads`, `sequence length`, `sequence length`], are summed over the last dimension to yield values that equal 1**. For image tokens, wings uses a visual learner where the router maps attention weights by first summing along the head dimension and then **applying an MLP for transformation into a two-dimensional output**. Consequently, this reshapes the attention weights to [`sequence length` x `2`], allowing the router to assign specific attention weights to each image token.\n\nIn the second stage, we incorporate **textual learners alongside masked visual learners to create weighted hidden states** from both image and textual tokens, with a detailed process description and pseudocode provided in the final version.\n\n* **Q5:** \"learning rate of the second stage\"\n\n* **A5:** We used a learning rate of 1e-3 for the first training stage and 2e-6 for the second, while the visual projectors’ alignment modules were set to 1e-5. **Our comprehensive ablation**, as detailed **in Table 10 of PDF**, tested learning rates of 2e-6, 6e-6, and 1e-5, applying larger fine-tuning steps for the visual component to **adapt better to multimodal inputs** and **reduce text-only forgetting**, similar to models like Qwen-VL and DeepSeek-VL.\n\n* **A6:** We are committed to **releasing all data, model weights, and code, along with the new Interleaved Image and Text (IIT) benchmark**.\n\n* **Q7:** \"i in equation (3)\"\n\n* **A7:** In the main paper, we adapted equation 3 from LLaVA to account for interleaved image tokens within the text, omitting loss computation for the \"next image token\" **since the MLLM cannot generate image tokens.** This adjustment results in a text-only indicator variable interval of `[1, v_start)` U `(v_end, s]`. We will elaborate on this point in detail in the final version.\n\nThank you once again for your constructive feedback, which will greatly enhance the clarity of our paper.\n\nBest regards,\n\nAll Authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U5ctDTNNZj", "reviewer": "Reviewer_3jFi", "summary": "This paper addresses a significant challenge that when Multimodal Large Language Models (MLLMs) as they expand LLMs’ capabilities to include vision tasks. Specifically, it highlights the issue of \"text-only forgetting,\" which occurs when MLLMs trained with visual inputs struggle to effectively process text-only instructions. The problem is attributed to inconsistent attention shifts between text and visual inputs before and after training. To solve this, this paper proposes Wings, a method adding visual and textual learners to balance attention shifts. Their approach shows improved performance on both text-only and multimodal tasks, including a newly constructed Interleaved Image-Text (IIT) benchmark for mixed-modality interactions.", "questions": "1. Wings note that the text-only performance of MLLMs and the performance of the initial LLMs. This makes a compelling argument for a more general model. Could you provide examples illustrating the phenomenon of performance degradation on text-only in existing models?\n2. In line 131, how is the activation value for each token computed for each layer? Is the averaging done along the columns or rows of the attention matrix, i.e., which dimension is averaged?\n3. There are other methods for adding additional modules to MLLMs, such as CogVLM [1]. Please clarify the similarities and differences, particularly why Wings-architecture can alleviate the forgetting problem in text-only data.\n4. Provide more specific implementation details of the model structure mentioned in line 161?\n5. Please include more ablation studies on training Wings on the base LLM, such as the effect when the initial LLM is base-version and does not have a system prompt. If the images are always positioned at the beginning, does Wings still work effectively?\n6. Please provide more experimental results regarding the efficiency gains with LoRRA. Compared to LoRA, it specifically handles modalities. How does its efficiency compare to LoRA, and what are its advantages?\n7. Please provide more inference details, particularly how the data flows through the Wings modules (the learners for each modality) and the routing mechanism.\n8. In Figure 4, the second stage requires fine-tuning the LLM parts of the MLLM. How much resource overhead does this require? Is it significantly more than methods like LLaVA?\n9. Please provide the detailed parameter counts for the 1.8B and 7B model versions and include detailed descriptions of each dataset in subsequent versions.\n10. In the IIT benchmark, why does the (T, T, T) configuration not show performance improvement compared to (T, T)? Does it indicate the presence of noisy text-only few-shot samples?\n\nSome Tips:\nPlease add a description of $\\mathbb{1}$ and its subscript in Eq. 3.\nAdd details to the weighted sum description in line 189, specifically how the attention weight matrix is applied in the router module.\nThe ARC-Easy dataset mentioned in line 217 is missing a citation.\nWhat is referred to by \"Efficient Multimodal LLMs\" in line 260?\nProvide a more detailed explanation of Partitioned LR in Figure 5.\nPlease include the prompts used to generate data with GPT-3.5 Turbo mentioned in line 268 in subsequent versions.\n\n[1] CogVLM: Visual Expert for Pretrained Language Models.", "limitations": "The Limitation section in the appendix states that a relatively small amount of text-only training data is still needed to activate the MLLM's capabilities, which is reasonable. Additionally, Wings is a multimodal framework trained from a general LLM. The generalization capabilities and other limitations of Wings can be comprehensively evaluated in the future.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing in the article is clear.\n2. Noticing the text-only ability of MLLMs for a more general model is important, this ability can be lost when turning LLMs into MLLMs.\n3. Existing methods need a lot of text-only training samples, which limits improving MLLM performance. The proposed model shows good results.", "weaknesses": "The paper provides a detailed analysis of the impact of image tokens on the distribution of existing tokens and introduces the Wings structure. However, to ensure that the Wings structure effectively minimizes the overhead of text-only training data while maintaining model performance, more time-based quantitative analyses are necessary to demonstrate the module's effectiveness.\nIt is observed that methods like DeepSeek [1], which have published their performance on text-only datasets, do not show significant performance degradation. This might be due to their extensive use of text-only training data. Thus, there is a trade-off between the overhead of text-only training data (i.e., excessive training costs) and model optimization strategies (or training methods). Please show why Wings is efficient in this trade-off. Specifically, to what extent can text-only data be reduced in Wings training process while still maintaining text-only performance?\nIn Figure 2, it is shown that image tokens are inserted among text tokens. Is this insertion specific to chat models with system prompts, or does it also occur in base-version models?\nConcerning the IIT benchmark, when constructing multimodal samples (e.g., (T,T,V) as shown in Figure 5), how are these samples constructed, and how is dataset balance maintained? Please provide additional implementation details.\n\n[1] DeepSeek-VL: Towards Real-World Vision-Language Understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3jFi,\n\nThank you very much for Reviewer 3jFi's detailed feedback. We appreciate the reviewer's recognition of Wings' motivation, writing, and overall performance. All updates will be incorporated into the final version. Below are our responses to the clear comments and questions raised:\n\n* **Q1:** \"trade-off between training cost and performance\"\n\n* **A1:** Wings shows **a direct improvement in training efficiency**: when adding an equivalent amount of Text-only data to the training set, Wings achieves better Text-only performance and improved multimodal performance while maintaining a training cost nearly identical to the baseline LLaVA. To achieve comparable performance, **the baseline requires more Text-only training data.** Table 1 in the main paper uses entirely consistent training data, demonstrating that Wings can maximize overall general performance within a balanced trade-off. Additionally, **we compared the same-architecture baseline using Wings_pro training data with Wings_base** (which utilized less training data, as elaborated in line 675 of the paper). We found that Wings can deliver stronger overall Text-only and multimodal capabilities with a smaller dataset.\n\n* **Q2:** \"Wings for LLM_base\"\n\n* **A2:** Wings proves effective for LLM_base as well: the presence of image tokens in any position of the training data aids the model in understanding the context of images and text. The placement of image tokens **does not restrict** Wings' expression and **does not affect** the attention weight extraction in Wings' LLM main branch. Results from training on the Qwen1.5-7B base LLM are presented **in Table 13 of PDF**, showing sustained excellent performance.\n\n* **Q3:** \"details of Interleaved Image and Text (IIT) Bench\"\n\n* **A3:** As discussed in line 264 of the paper, we first sampled multimodal examples **from MMMU, MMBench, SEED-Bench, and AI2D, then collected semantically similar text-only samples from MMLU, CMMLU, OpenbookQA, and HellaSwag.** For some questions, **we used GPT-3.5 Turbo for polishing**. We performed clustering to sample different semantic clusters, aiming for a diverse dataset, and we will open-source the entire dataset after the paper is accepted.\n\n* **Q4:** \"case of text-only forgetting\"\n\n* **A4:** In Table 2 of the main paper, we present comparisons with other methods, such as Qwen-VL and the LLaVA series models. **These models did not specifically integrate text-only data into their training or address the issue of text-only forgetting**, which led them to underperform compared to the initial text-only LLM (as shown in Table 2 of the main paper).\n\n* **Q5:** \"router details (with attention inputs)\"\n\n* **A5:** We begin by performing linear mapping on the softmax dimension. Below, we detail how the Wings structure extracts attention weights for routing, collaborating with the learner for inference:\n   \n   1. First, we obtain the LLM attention weights from the main branch: during inference, by setting `output_attentions=True`, **the attention weights are derived by multiplying the query and key, dividing by the square root of head_dim, and applying softmax** (note that dropout is not applied). Consequently, the shape of the attention weights is [`number of heads` x `sequence length` x `sequence length`], ignoring the batch size dimension, and the sum along the last dimension equals 1.\n   \n   2. For the image tokens, Wings introduces a visual learner. **The router maps the attention weights to a weight ratio between the image learner and the main LLM**: first, we sum the weights along the head dimension and then apply an MLP (linear mapping followed by an activation layer) to produce a shape of [`sequence length` x `2`], assigning weights to each token. The visual learners for text tokens will be masked, but in the second stage, textual learners will also undergo training in a similar manner. **This weighted combination forms a new set of hidden states, which are then processed in the next layer.** We will provide a detailed description and pseudocode in the final submission.\n\n* **Q6:** \"learner of Wings *v.s.* CogVLM visual expert\"\n\n* **A6:** \n   1. **Positioning differs**: The visual expert in CogVLM is a parallel module through linear mapping, while the LoRRA in Wings operates independently from the entire attention block.\n   2. **The motivation is similar but slightly distinct**: Similar to Mix of Experts and P-tuning, visual experts enhance the attention module's ability to **learn cross-modal interactions and alignments**. Wings also boosts multimodal learning capabilities, specifically addressing compensatory attention for text-only forgetting.\n   3. **Utility varies**: Visual experts increase training overhead significantly and, without low-rank optimization, **will substantially raise network parameters** (though inference overhead remains nearly unchanged). In contrast, Wings adds only a minimal number of parameters (**refer to Table 9 of PDF**).\n\n* **Q7:** \"details on line 161\"\n\n* **A7:** Line 161 refers to a visual encoder, specifically the `vit_so400m_patch14_siglip_384` and a linear mapping with a single-layer MLP.\n\n* **Q8:** \"more studies on Wings with base-version LLM\"\n\n* **A8:** **In Table 12 of the PDF**, we evaluate the MLLM based on the Qwen-base model (the image is at the beginning of instructions and lacks a system prompt). **The results show good performance**; however, because the base model has weaker instruction capabilities, **a standard chat LLM is generally more suitable**. We will include related experiments in the final version.\n\nDue to space constraints, **subsequent responses will be presented in the comments**.\n\nBest regards,\n\nAll Authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F6ZrhCVdq8", "reviewer": "Reviewer_TqCY", "summary": "Multimodal large language models (MLLMs) are initiated with a trained vision encoder and LLM, then fine-tuned on multimodal mixed inputs. In this process, the LLM catastrophically forgets the text-only instructions. This paper first reveals that text-only forgetting is related to the attention shifts from pre-image to post-image texts. Based on this analysis, the authors proposed Wings, the complementary visual and textual learners. The experiment results show that Wings outperforms similar-scaled MLLMs in both text-only and visual question-answering tasks.", "questions": "- The most intuitive and direct approach to increase the attention correlation can be simply adding correlation loss between the attentions. What would the results be if we applied the correlation loss directly?\n- The proposed method looks somewhat similar to P-LoRA [1]. How does it compare with this?\n- For ablation studies (Figure 5), which model is used between Wings-base and Wings-pro?\n\n[1] Dong, Xiaoyi, et al. \"Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model.\" arXiv preprint arXiv:2401.16420 (2024).", "limitations": "The paper addresses limitations in the paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The analysis of attention shifts is novel and interesting.\n- The experiment results demonstrate the effectiveness of the proposed method in both text-only and multimodal benchmarks.", "weaknesses": "- The discrepancy seems to exist between the motivation/analysis and the method. There is a lack of detailed explanation on how the Wings module improves the correlation between pre-image and post-image attentions.\n- The proposed Low-Rank Residual Attention (LoRRA) module appears to be a variant of LoRA, but there is no detailed motivation or comparative analysis provided.\n- The ablation study is insufficient. Ablation experiments are only compared on the Interleaved Image and Text (IIT) benchmark. It would be better to include comparisons across various benchmarks used in the main results as well.\n- The simple baseline to prevent forgetting is simply utilizing text-only data, yet there are no comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TqCY,\n\nWe sincerely appreciate reviewer TqCY for acknowledging our analysis of attention shift and the effectiveness of Wings. All updates will be included in the final version. Thank you!\n\n* **Q1:** \"how the Wings module improves the correlation\"\n\n* **A1:**\n\n   * **From a structural perspective:** the Wings module operates **independently of the attention block**, with its outputs combined through weighted addition with the main branch LLM's attention to compensate for attention weight distribution.\n\n   * **In terms of the forward process:** the Wings module uses image features as keys and values, while the query derives from previous hidden states. This allows it to **focus more on the attention between the image and the pre-image and post-image**, compensating for the main branch LLM's attention weights.\n\n   * **From an optimization standpoint:** A key step in the Wings module's compensation for attention shift is aligning with the main branch **in the initial stage**. As shown in Figure 4(b) of the main paper, learners **first allocate attention between the pre-image, post-image, and image in the first stage**. Following this, the main branch LLM completes alignment under the \"guidance\" of the Wings module in the second stage, enhancing attention relevance for pre-images and post-images.\n\n   In the experiment described **in Figure 5 (C)**, we observe that compensation with the visual wings improves performance for both text-only and multimodal cases. We also conducted **additional experiments**:\n\n   * We hypothesize that models with significant pre-image and post-image attention shifts may have **a more dispersed** attention distribution for text-only tasks, with relevant analysis provided **in Table 1 of the supplemental PDF**.\n\n   * **In Table 2 of the PDF**, we compare the allocation weights of the Wings module and the main branch of the two stages. The weights from the router indicate that the Wings module **effectively compensates for pre-image and post-image attention** in the first stage.\n\n   * Additionally, **Figure 1 in the PDF** shows that the attention distribution **over words varies** with different models exhibiting attention shifts.\n\n\n* **Q2:** \"detailed comparative analysis between LoRRA and LoRA\" **and \"compare to P-LoRA\"**\n\n* **A2:** While the Low-Rank Residual Attention (LoRRA) structure also utilizes low-rank mappings, it fundamentally differs from Low-Rank Adaptation (LoRA) in key aspects:\n\n   1. **Motivation Design**: LoRRA functions **independently as an auxiliary module** to compensate for attention shifts in the main branch, while **LoRA** is **an efficient parameter tuning method** for adapting pre-trained models to target tasks with minimal training overhead.\n\n   2. **Structural Position and Training Strategy**: LoRRA operates as a separate module **that aligns prior to** attention distribution, whereas **LoRA and P-LoRA** are integrated within the attention module, **parallel to the internal attention linear mapping**, allowing the main branch mapping to remain unchanged while learning low-rank decomposition matrices. P-LoRA processes only forward image features on the low-rank mapping.\n\n   3. **Effectiveness**: Table 1 in our paper compares LoRA and Wings **under identical training data and base model architecture**. In multimodal scenarios, LoRA retains text-only content better but demonstrates weaker multimodal capabilities. This suggests its limitations in handling the complexity of multimodal tasks, as low-rank perturbations may be insufficient [1]. Results for P-LoRA with InternLM-XComposer2 7B can be found in Table 12 of the PDF.\n\n   [1] LoRA Learns Less and Forgets Less.\n\n\n* **Q3:** \"ablation comparisons across various benchmarks in the main results\"\n\n* **A3:**\n\n   * **Table 1 in the main paper** presents a comprehensive benchmark result from **16 text-only datasets** across **5 major domains**, including mathematics and coding, along with multimodal instructions (see Table 2, line 242). All methods in Table 1 used **the same training data and model architecture**, **serving as an ablation study**. The Wings architecture shows significant improvements over the baseline and LoRA with equal parameters and consistent training data.\n\n   * We added ablation experiments. **In Table 5 of the supplement PDF**, we compare ablation results across various benchmarks from the main results. Similar to IIT Bench, we found that **a lower learning rate or the inclusion of visual Wings** can maintain strong performance for text-only tasks (e.g., MMLU, WinoGrande).\n\n* **Q4:** \"the baseline simply utilizing text-only (training) data\"\n\n* **A4:**\n\n   1. Incorporating text-only training data during continuous training of MLLMs **helps prevent forgetting is common**, although it increases labeling and training costs. For instance, DeepSeek-VL used over 50% text-only data in its multimodal training.\n\n   2. The large parameter counts of LLMs complicate regularization and knowledge distillation, **making them less feasible**. Additionally, heightened expert supervision can be costly compared to collecting labeled text-only data.\n\n   3. While the MLLM community is evolving, some methods **lack open training data and may not align with publishers' hyperparameter settings**. Nonetheless, we've tested and compared most MLLMs in Tables 1 and 2 of the main paper.\n\n   4. In Figure 5 of the main paper, we introduced the Interleaved Image and Text (IIT) Bench for evaluations in continuous multimodal environments.\n\n* **Q5:** \"add attention correlation loss\".\n\n* **A5:** We attempted to impose KL divergence as a constraint, but this **made the training process unstable**. This is due to correlation **being a statistical phenomenon**, making it hard to control the scale of each sample. Results can be found in Table 12 (line 5) of the PDF.\n\n* **A6:** For ablation studies (in Figure 5 of the main paper), we used **Wings_base**.\n\nBest regards,\n\nAll Authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2SpcFMBtli", "reviewer": "Reviewer_LUvX", "summary": "This paper introduces the text-only forgetting phenomenon, where multimodal large language models (MLLMs) experience a significant decline in performance on text-only evaluations. The authors claim that this phenomenon is related to the attention shift of cross-layer MLLM-LAWS (Layer-level Attention Weights) before and after processing images. To address this, the authors propose WINGS, which utilizes visual and textual learners to compensate for the attention shift. Experiments across text-only, VQA, and IIT benchmarks demonstrate the effectiveness of WINGS.", "questions": "1. What are the 100 diverse MLLMs visualized in Figure 2(c)? Are these models sampled from the same type of MLLMs? Does the correlation occur in MLLMs trained on interleaved image-text datasets?\n\n2. Is the relationship between text-only performance and MLLM-LAWS consistent across various scales of LLM backbones?", "limitations": "The major limitation of this paper is the lack of a deep understanding of the phenomenon shown in Figure 2. While the paper identifies a correlation between text-only forgetting and MLLM-LAWS, it does not explore the underlying mechanisms behind this relationship.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The observation that text-only forgetting is related to MLLM-LAWS is intriguing.\n2. WINGS shows significant improvements in text-only QA and multimodal QA tasks.\n3. The paper is well-organized and easy to follow.", "weaknesses": "1. Although the paper provides a valuable observation about the correlation between text-only forgetting and MLLM-LAWS correlation in Figure 2, it lacks a deep discussion on the underlying reasons for this phenomenon. MLLM-LAWS correlation measures the attention states before and after the image, but in the text-only evaluation, no image involved. Thus, it remains unclear why text-only forgetting is related to MLLM-LAWS correlation.\n\n2. Figure 2 shows that text-only performance is correlated with MLLM-LAWS. However, it would be interesting to know if the multimodal performance is also correlated with MLLM-LAWS. If so, the correlation score could be a valuable metric for MLLM model selection or evaluation.\n\n3. Table 1 shows that LoRa efficiently mitigates the text-only forgetting problem but degrades multimodal performance. The authors do not provide any explanation for this observation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer LUvX,\n\nWe sincerely thank Reviewer LUvX for the keen observations and suggestions, as well as the recognition of the attention shift and Wings effectiveness, and the overall flow of the writing. We will update all modifications in the final version. Thank you.\n\n* **Q1:** \"MLLM-Laws w/ image *v.s.* text-only evaluation w/o image, why text-only forgetting is related to MLLM-Laws correlation\"\n\n* **A1:** In Figure 2 of the main paper, the attention shift phenomenon guided by MLLM-Laws is shown to be strongly correlated with text-only forgetting, **following are the two main factors:**\n\n    * The attention shift in MLLM occurs because the model's LLM main branch **excessively focuses on image features**, causing a deviation in attention toward the images (especially post-image part, line 138). **This \"over-focus\" prevents effective information gathering in text-only scenarios**, leading to potential attention dispersion. We discussed this in details at the final version.\n\n    * Additionally, we have conducted supplementary experiments to support this hypothesis.\n\n    1. **In Figure 1 of the supplement PDF**, our analysis shows that a reduced attention shift results in more accurate and focused text-only attention distributions.\n\n    2. **In Table 1 of the PDF**, we propose a new metric for the variance of inter-word probabilities in text-only tasks, linking attention shift (MLLM-Laws induced) and text-only performance. **We find that attention shift can diminish MLLM's information aggregation capabilities in text-only data**; further experimental results will be published in the final version. \n\n    Furthermore, as described around line 231 in Figure 4(b) of the paper, a key step in compensating for attention shift is aligning the Learner with the main LLM branch during the initial training stage. We conducted additional experiments:\n\n    * **In Table 2 of the PDF**, we compared the Learners' attention weights after the first training stage with **the main branch's scale and the routing allocation weights after the second stage**. The Router's distribution weights indicate effective compensation for attention by the Learner structure.\n\n    * **In Table 3 of PDF**, comparing the training outcomes of Wings with baseline methods, we observed that Wings effectively mitigates attention shift.\n\n    * Finally, **In Figure 1 of PDF**, further analysis of the attention distribution in cases within Wings shows reduced shifts due to dominant segments.\n\n\n* **Q2:** \"more analysis of attention shift *v.s.* multimodal performance\"\n\n* **A2:** **In Table 11 of the PDF**, we found only a weak correlation in multimodal performance. For instance, on MMMU-VAL and ScienceQA, but it wasn't significant on MME.\n\n    Following LEEP and LogME's benchmark, **we developed a MLLM library for task-level model selection**, achieving improvements over random selection. Establishing a connection between attention shift and the generalized lower bound is essential for this process. Further analysis and results will be included in the final version.\n\n* **Q3:** \"more explanation for LoRA observation\"\n\n* **A3: LoRA reduces forgetting in text-only tasks but develops less multimodal capability.** Early LoRA studies often focused on simpler models and tasks, whereas multimodal tasks typically show high-rank weight variations. We demonstrate this **in Table 1 of the main paper under the same training data and network parameters.**\n\n    To verify that full parameter fine-tuning in multimodal scenarios **is not a low-rank perturbation, in Table 4 of the PDF**, we sampled weight transformations and performed singular value decomposition, **revealing significant changes in the weight matrix spectra from full fine-tuning**. It's important to note that our findings pertain to full fine-tuning, while Wings uses an independent module that interacts with cross-modal inputs. Therefore, the ranks identified do not directly indicate the rank needed in the Wings learner's LoRRA.\n\n    [1] LoRA Learns Less and Forgets Less.\n\n* **Q4:** \"details of 100 MLLMs in Figure 2\"\n\n* **A4:** In our training data, we adopted **various ratios of multimodal to text-only samples**: `25:1`, `20:1`, `10:1`, `5:1`, `2:1`, `1:1`, `1:2`, ..., `1:25`, along with an `all:0` (12 combinations, ensuring a sufficient amount of multimodal samples). We used learning rate 1e-3 for the first stage and 2e-6 or 1e-5 for the second. **We sampled 5 models in one epoch**, excluding 12 failed models due to issues like gradient explosion, resulting in 108 models for analysis.\n\n    Although these models share the same architecture (Qwen1.5 7B + SigLIP), performance varies significantly between text-only and multimodal scenarios. **The results reflect different training epochs, hyperparameters, and data, revealing potential correlations.** Nonetheless, they are classic and general, with a sufficient sample size.\n\n* **Q5:** \"different scales of MLLMs for attention shifts\"\n\n* **A5:** Generally, scaling laws suggest that model patterns can be transferred across different scales. **We conducted experiments using the 1.8B Qwen-Chat backbone**, training models with varying ratios of **text-only instruction data to multimodal data**: `10:1`, `2:1`, `1:1`, `1:2`, and `1:10`, alongside fully multimodal data for a total of six variations.\n\n    In the first training stage, we set the learning rate to 1e-3, adjusting it to 2e-6 in the second stage. We averaged 5 models per epoch, discarding 4 due to issues like gradient explosion, **resulting in 26 distinct models**. All models started with the same initialization and were trained on 4 A100 GPUs.\n\n    The results **in Table 12 of PDF** demonstrated that MLLM-Laws and text-only correlations were also evident in smaller-scale models.\n\nPlease let us know if you have any further suggestions or questions! Thank you!\n\n\nBest regards,\n\nAll Authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

**Thank you for your meticulous observations and analyses.** We are thrilled that you have recognized our work, Wings. We appreciate your acknowledgment of Wings’ **effectiveness on** text-only, multimodal, and Interleaved Image and Text (IIT) benchmarks. We are particularly pleased that you found the concept of **attention shift** intriguing and that you described the paper as clear and easy to read.

We observed **the attention shift phenomenon** in MLLMs that have forgotten their text-only capabilities. To address this issue, we designed Wings with **the Low Rank Residual Attention (LoRRA) structure**, which significantly **enhances performance without** adding high costs. During this time, we've been continuously improving Wings to achieve even better results.

Once again, thanks to each reviewer and everyone involved! We have responded to all your questions in detail and **included additional tables and figures in the PDF**. **Should you have further inquiries, please feel free to reach out. We appreciate your support and will continue to work diligently to enhance Wings.**

Best regards,

All Authors.

Meta-review: This paper unanimously receives positive rates thanks to clear representation and convincing experiments. Although overall reviews are positive, the additional description in the rebuttal should be reflected in the final draft.
Decision: Accept (poster)