LLMs Can Evolve Continually on Modality for $\mathbb{X}$-Modal Reasoning
Authors: Jiazuo Yu
Haomiao Xiong
Lu Zhang
Haiwen Diao
Yunzhi Zhuge
Lanqing HONG
Dong Wang
Huchuan Lu
You He
Long Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: MLLMs
Transfer Learning
Continual Learning
Primary area: machine_vision
TL;DR: The paper proposes a more challenging Continual Learning on Modality (MCL) to enable MLLMs to extend to new modalities without retraining.

Abstract: Multimodal Large Language Models (MLLMs) have gained significant attention due to their impressive capabilities in multimodal understanding. However, existing methods rely heavily on extensive modal-specific pretraining and joint-modal tuning, leading to significant computational burdens when expanding to new modalities. In this paper, we propose \textbf{PathWeave}, a flexible and scalable framework with modal-\textbf{path} s\textbf{w}itching and \textbf{e}xp\textbf{a}nsion abilities that enables MLLMs to continually \textbf{ev}olve on modalities for $\mathbb{X}$-modal reasoning. We leverage the concept of Continual Learning and develop an incremental training strategy atop pre-trained MLLMs, enabling their expansion to new modalities using uni-modal data, without executing joint-modal pretraining. In detail, a novel Adapter-in-Adapter (AnA) framework is introduced, in which uni-modal and cross-modal adapters are seamlessly integrated to facilitate efficient modality alignment and collaboration. Additionally, an MoE-based gating module is applied between two types of adapters to further enhance the multimodal interaction. To investigate the proposed method, we establish a challenging benchmark called \textbf{C}ontinual \textbf{L}earning of \textbf{M}odality (MCL), which consists of high-quality QA data from five distinct modalities: image, video, \textcolor{black}{audio, depth} and point cloud. Extensive experiments demonstrate the effectiveness of the proposed AnA framework on learning plasticity and memory stability during continual learning. Furthermore, PathWeave performs comparably to state-of-the-art MLLMs while concurrently reducing parameter training burdens by 98.73\%. Our code locates at \url{https://github.com/JiazuoYu/PathWeave}.

LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Jiazuo Yu', Haomiao Xiong', Lu Zhang Haiwen Diao', Yunzhi Zhuge', Lanqing Hong". Dong Wang', Huchuan Lu², You He Long Chen
Dalian University of Technology. Huuawei Noah's 's Ark Lab Tsinghua University, "The Hong Kong University of Science and Technology yujiazuodmail. dlut .du....du.cco en, zhangluuddlut. edu.cn
Abstract
Multimodal Large Language Models (MLLMs) have gained significant attention due to their impressive capabilities in multimodal understanding, However, exist- ing methods rely heavily on extensive modal-speciff pretraining and joint-modal tuning, leading to significant computational burdens when expanding to new modal- ities. In this paper, we propose Path Weave, flexible and scalable framework with modal-path switching and expansion abilities that enables MI.LMs continually evolve modalities for X-modal reasoning. We leverage the concept of Continual Learning and develop an incremental training strategy atop pre-trained MLLMs, enabling their expansion to new modalities using uni-modal data, without executing joint-modal pretraining. In detail, a novel Alapter-in-Adayye introduced, in which uni-modal and cross-moda adapters are seamlessly integrated to facilitate efficient modality alignment and collaboration. Additionally, an MoE- based gating module is applied between two types of adapters to further enhance the multimodal interaction. To investigate the proposed method, we establish a challenging benchmark called Continual Learning of Modality (MCL), which consists of high-quality QA data from five distinct modalities: image, video, audio, depth and point cloud. Extensive experiments demonstrate the effectiveness of the proposed AnA framework on learning plasticity and memory stability during con- tinual learning. Furthermore, PathWeave performs comparably state-of-the-art MLLMs while concurrently reducing parameter training burdens by 98. 98.73%. Our code locates at https: //git gig..u..... com/ JiazuoYus Yu Patpaaaheeee...
(AnA) framework is
Introduction
With recent advances in artificial intelligence, Large Language Models (LLMs) have demonstrated impressive capacities in language understanding and reasoning. The success of LLMs [1, 2. 3. 4] has spurred researchers to develop Multimodal LLMs (MLLMs) by integrating additional input for multimodal tasks, such image-text understanding [5, 7], audio recognition [8, 9] and 3D question answering [10, 11]. Aided by large-scale image-text paired data from the Internet [12, 13, 6, 14, 5], vision LLMs have become thriving
in the research community. The typical framework comprises a visual encoder, a frozen or trainable LLM, and projection module for vision-language alignment. Through stepwisely pretraining on large-scale image-text pairs and instruction tuning on specific datasets, vision LLMs exhibit promising generalization abilities on downstream applications such detection [15], grounding [16, 17], and captioning [6, 14]. Subsequently, the LLM-bbsed framework and training pipeline of vision LLMs serve as the basis and drive the extension to other modalities, including video [18, 19]. audio [9, 8]. and point cloud [11, 10]. However, these modal-
"Correspondngg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Image
Video
Andio
Deptt
Point
Imoge
Video
Andio
Depth
Point
Unified Sumpling
Modality Incremental
Ques fion
Normal MLLMs
Answering
Question
Incremental
Answering
MLLMs
(a) Normal Multi-modal LLMs
(b) Our Incremental MILMS
Figure I: Comparisons of Different Multimodal LLMs: (a) The normal multimodal methods [21, 23, 22] require unified sampling across multi-modal. (h) Our proposed incremental MILMS learns each modality sequentially without joint-modal datasets.
specific LLMs that inject single-modal data into language models struggle to tackle the challenge of perceiving different modalities like us humans.
To address this issue, recent approaches [20, 21, 22, 23] extend the architecture and training strategies of modal-specifte MLLMs, and try to integrate multiple modalities into unified system. Some early attempts [22, 23] utilize specitic projection modules to align image, video, and audio encoders into frozen LLM. However, complex training process is usually required to enhance cross-modal alignment, involving separate pretraining on uni-modal data and joint fine-tuning on multimodal data. Subsequent attempts try enhance the scalability of MLLMs by unifying the architecture and simplifying the training process. For instance, X-InstructBLIP [20] proposes a unified projection architecture for all modalities and constructs high-quality instruction tuning data to simplify modal- specific customization and pretraining. Onel..M [21] leverages a unified encoder and projection module and introduces incremental pretraining strategy achieve parameter unification for wide range of modalities. While effective, most approaches still rely on joint-modal optimization that is high-resouree demanding (see Figure (a)). When expanded to new modalities, the models have to re-access all the historical data and repeat the complete training process, limiting the continual extension of MILMs.
In this paper, we propose Path Weave & a flexible and scalable framework with modal- switching and expansion capabilities that enables MLLMs to continually evolve modality for X-modal reasoning. Path Weave leverages the concept of Continual Learning (CL) and forms an incremental training pipeline uni-modal data, eliminating the necessity for joint-modal pretraining finetuning. To this end, we employ a pre-trained vision L.LM [20] as the interface and propose a novel Adapter- in-Adapter (AnA) framework, allowing efficient extension and alignment for other modalities. We set two types of adapters in AnA, uni-modal and cross-modal, and seamlessly incorporate them to boost modality alignment and collaboration during incremental learning. Specifically, the uni-modal adapters are progressively added to the interface and optimized on the corresponding modality data, which will be frozen once trained. Meanwhile, we insert in-adapters into the previous uni- modal adapters to form cross- adapters, allowing the effective integration between historical knowledge and ongoing modality. Additionally, an MoE-based gating module is implemented between uni-modal and cross-modal adapters to further enhance multimodal collaboration. As shown in Figure (b), our Path Weave can be flexibly implemented on the pretrained MILMSs and efficiently expand to more modalities in an incremental manner.
To evaluate the proposed PathWeave we establish a challenging benchmark, namely Continual Learning of multi-Modality (MCL). It consists of data from five distinct modalities: image, video, depth, audio, and point cloud. In our setting, the modalities data are incrementally fed to the MLLMs. Thus, we leverage the commonly-used metrics from [20, 21] to investigate the precision on newly learned modalities. Furthermore, we introduce a metric to measure the forgetting rate in MCL to demonstrate the effectiveness of the proposed AnA strategy on historical modality memorization. Finally, we conduct extensive experiments to compare with state-of-the-an continual learning approaches, demonstrating that PathWeave is effective at incorporating multimodal data in an incremental manner. Moreover, our method achieves comparable performance with state-of-the-art MLLMs without requiring modat pretraining fine-tuning.
In summary, our contributions are summarized as follows:
We present an efficient and scalable framework, Path Weave, which enables MILLM to progressively expand on multiple modalities, without the need for modal pretraining.

---

We introduce a novel shapter-in-adatie framework that seamlessly integrates uni-modal and cross cross-modal modal adapters enhance modality alignment and interaction during incremental learning. We establish a challenging MCL. benchmark with well-defined evaluation metrics. Extensive results demonstrate the effectiveness of Path Weave on modality plasticity and memorization during continual learning. Furthermore, PathWeave performs on par with state-of-the-art MLLMs while reducing parameter training burdens by at least 98. 98.73%.
2 Related Work
Multimodal Large Language Models. In recent years, researchers have been exploring the potential of LLMs in multimodal perceptions, such as visual question answering [5. 14] and captioning [6. 24].
This leads to the rapid development of Multimodal LLMs [6, 5, 21, 22]. For example, LLaV VA [5] utilizes simple linear layer to project visual information into language space, enduing L.MSs the ability to perceive natural scenes. Subsequently, several methods attempt to expand the supported modalities of LLMs by modifying architecture designs or training strategies. For instance, X- LLM [22] and Chatbridge [23] use modal-specific modules to extract features for multiple modalities and exploit modal-specific projection layers for multimodal alignment on a frozen LLM. However, complex training process is usually required to enhance cross- modal alignment, which involves separate pretraining on uni-modal data and joint instruction tuning on multimodal data. Later, X- InstructBLIP [20] proposes unified projection architecture (Q-former) for all modalities and collects large-scale, high-quality instruction tuning data to eliminate the need for uni-modal pretraining. OneLLM [21] explores parameter unification by introducing unified encoder and projection module for a wide range of modalities. Although an incremental pretraining strategy is proposed to alleviate the high resource demand of cross-modal alignment, OneLLM still relies cross-modal finetuning on large-scale instruction datasets. In contrast to these methods, we incorporate the continual learning concept into MLLMs and propose an incremental training strategy to allow MLLMs' modal expansion by finetuning on uni-modal data, without requiring joint-modal pretraining or finetuning. Among these approaches, X-InstructBLIP [20] is highly related to our method, as it separately tunes Q-former to align multimodal into a uniform system. However, our method designs an adaptee-bbsse expansible framework that significantly reduces the parameter training burdens by least 98.73%.
Continual Learning in Foundational Models. Continual Learning (CL) has been applied to large foundational models [25, 26, 27, 28], allowing them to continually acquire new knowledge. To address the forgetting issue in CL, significant efforts [29] have been made, including data replay, regularization constraints, and dynamic frameworks. Data replay- based methods [30, 31, 32, 33, 34] retain the historical data in a memory bank and mix them with new data to execute the general training process. However, the redundant historical data would incur increasing resource demand during lifelong learning. Regularization-basee methods add explicit regularization terms on weights [35, 36, 37] or data [38, 39, 40, 41] to achieve a balance between historical and new tasks, which are usually used as an auxiliary trick in data-replay or dynamic methods. In contrast, dynamic methods [28, 26, 42, 43, 44] exhibit impressive expansible abilities by inerementally adding new parameters into a shared interface. Recently, the dynamic frameworks have been combined with efficient tuning techniques to achieve efficient, cost-friendly continual learning on visual- textual domain [28, 27, 26]. This inspires us to eliminate joint-modal pertaining from MLLMSs by developing an efficient, scalable framework where new modalities are incrementally involved by accessing uni- modal data. To this end, we propose an Stapter-in-adaper framework, which incorporates uni-modal and pross-modal adapters for efficient modality alignment and collaboration.
Transfer learning. In the realm of Natural Language Progressing (NLP), fine-tuning large-scale models (e.g., 175B GPT-3 [4]) imposes significant burdens in both parameter complexity and time consumption. As result, transfer learning methods [45, 46, 47, 48] have gained significant attention to facilitate the efficient adaption of LLMs on downstream applications. The techniques usually activate a small set of parameters on the frozen models while achieving comparable performance with hally-finetuned approaches. Among these methods, LoRA [45] reduces the trainable parameters through low-rank matrix decomposition, leading to the generalization of the pre-trained model on diverse downstream tasks. The success of LoRA further promotes the development of parameter- efficient transfer learning of MILMs [5, 49, 50] and uni-modal continual learning approaches [27, 26, 51]. However, these methods cannot be directly applied to fix the proposed MCL task due to the significant variations in modality spaces. In this paper, we propose modality continual learning
3

---

"In the depth image, group of young playing game of ultimate frisbee in grassy field two players are jump."
Frezen
Trainable
Image
LLM
Testing audío modality
Enender
Image
1.M.M Projection
ABA
A
Modality Incremental
Vidro Encoder
Q-Former
A²
Villew
G
Cross Attention
Audio Encoder
(b) Training on depth modality
Audio
AnA
A'
A
A
Attention
Depth Encoder
A
A
Depth
AA'
A²
AmA
K
AmA
AnA
Point
Encode
"Please describe this depth in detail."
Figure 2: Overall framework of Path Weave. We start from pretrained vision LLM [20] and progressively expand new modalities on it without acquiring historical data. Given input samples from modality m, first exploit frozen encoder (Em) for feature extraction and leverage Q- Former to achieve multimodal alignment with LLMs. Then, the Adapter-in-Adapter (AnA) module implemented in Q-Former to achieve flexible modal-path switching and expansion. In detail, the uni-modal adapters (AATh) are implemented in parallel to facilitate new modal plasticity, which will be frozen once trained. While the cross-modal adapters (Á²")) are formed by inserting a set of in in-adapters (fF.T" M==1) into the learned uni uni-adapters to enhance the collaboration of historical knowledge. Additionaily, an MoE-based gating module (G implemented among uni-adapters adaptively multimodal integration in input space.
method that incorporates adapter-based dynamic architecture on a frozen LLM, allowing efficient adaption and flexible expansion new modalities in incremental manner.
Path Weave
3.1 Preliminaries
Continual learning can empower large-scale foundation modals constantly acquire new knowledge without accessing the entire historical data. We introduce this concept into MLLMs to form an incremental training strategy on uni-modal data called Continual Learning on Modality (MCL), eliminating the necessity of speccic pertaining and joint-modal datasets. Given set of M modalities (M 1M m= we enforce L.MSs sequentially access and learn on each modality for question answering. Here, each modality r contains Nm datasets, which can be represented as Mm = [D.m Ji=1 More specifically, D = , " denotes the i-th data of the m-th modality Mm which s and o are instruction, modality samples, and answering, respectively.
3.2 Framework Overview
This work presents Path Weave, an efficient and extensible framework that empowers MLLMs to constantly evolve modalities, without requiring modal-specific pretraining. Considering the complicity of training MLLMs from scratch, we start from pretrained vision LLM and align other modalities in an incremental manner. The overall framework of Path Weave is illustrated in Figure Specifically, we build the PathWeave on X-InstructBI.IP [20], providing a unified Q-Former architecture for various modalities. Given the samples from m-th modality, modal-specific encoder Fm pretrained on the corresponding modality is first exploited for feature extraction. Then, the Q- Former Q takes the input of modality feature, Iearnable query Jn, and instruction embedding Im, for

---

multimodal alignment on a frozen LLM. It is worth noting that the initial modality M is predefined as images, as we leverage the pretrained I-InstructBLIP to facilitate the alignment of subsequent modalities. As result, the entire parameter of the encoder, Q-Former, and LLM will be frozen during continual learning. To achieve continual learning on modalities, we propose Adapter-in-Adaple (AnA), a dynamically expansible framework atop MILMs, enabling the efficient integration of new modalities by executing uni-modal instruction tuning. The AnA consists of uni-modal and cross- modal adapters to boost modality alignment and collaboration along the modality sequence. In detail, the uni-modal adapters (A are implemented in parallel in Q-Former to efficiently adapt to new modalities, which will be frozen once trained to "memorize" the historical modalities. Meanwhile, the cross-modal adapters (A"T)) are constructed by inserting a set of in-adapters (fF,") 1m- into previously learned uni-adapters acquire their knowledge for ongoing modality, which will be removed accordingly when testing former modalities. Furthermore, an MoE-bbaeed gating module is implemented between uni-adapter and cross- pooss-dd for further multimodal integration.
3.3 Adapter-in-Adapter
X-InstructBLIP [20] utilizes Q-Former as a unified framework to extend MLLMs' capabilities on more diverse modality reasoning, eliminating the need for modal-specific pretraining. However, instruction tuning on uni-modal data is implemented on separated Q-Formers, which leads to signifi- cant computational costs and parameter burdens when integrating more modalities. Recently, some attempts [26, 27] have demonstrated that adapters with few parameters can enhance the adaption of foundation modal on downstream tasks. Inspired by this, we leverage effective transfer learning technique, LoRA [45], to serve as the basic unit of our AnA framework, enabling the efficient adaption of subsequent modalities during incremental learning.
Uni-modal Adapters. Given the current modality M we implement uni-modal adapters AA" in the pretrained Q-Former for new modal alignment. The adapters A are inserted into different linear layers of pretrained model in parallel. The output of layer with adapters A can be expressed as:
yi (x"x""))))))))))x))))())))) At" (x!"),
(1)
where x7" and y!" are the input and output embedding of (-th layer when aligning m-th modality. At" is the adapter of m-th modality in layer, and AA'' F.m (F)" where F, and Fa are the up and down projection of adapter. The uni-modal adapters are effective at acquiring modal-specific knowledge. Besides, the parallel architecture of adapters endows our system with the capabilities to flexibly switch and expand to diverse modalities.
Cross-modal Adapters. The uni-modal adapters are effective preserving the uni-m uni-modal knowledge and alleviating the forgetting issue in long-term learning. Based on it, we introduce a modal-special in-adapter module (F?") to form a cross-modal adapter (AÂT)), which can help the ongoing modality learn previous knowledge and encourage mnter-modality collaboration. Specifically, the in-adapters are inserted into the previously learned uni-modal adapters to effectively acquire the learned knowledge without reactivating their parameters. Then, the output of 1-th layer y?" after adding In-Adapter J. can be reformulated as:
yi" = Q1(x)))) + Σ
1 + (x?",,
1=1
where (xxx) (FF))( [L, m m-mm1 1] represents the cross-modal adapters for current modality M F.' is the in-adapter that is inserted into i-th frozen uni-adapters A', which is a single linear layer with the dimension of adapters' low rank. The uni-modal and cross-modal adapters collaborate to facilitate the new modality alignment and cross moddll integration during incremental learning. Furthermore, the proposed in-adapter serves plug-and-ple and-play module that will not affect the performance of previously learned adapters, thereby effectively alleviating the modality forgetting.
MoE-based Gating. Cross-modal adapters rely on in-adapters to effectively leverage historical knowledge to boost the aligmment of ongoing modality. However, the output of cross-modal and uni-modal adapters are treated equally in the original Q-Former. Considering the significant gap between distinct modalities, this simple integration strategy might pose performance degradation affected by the interfering information from other modalities. To address this issue, we propose an MoE-based gating module between cross-modal and uni-modal adapters for adaptive multimodal integration. Our MoE-based gating Gmd automatically assigns weights of paths pm of different
5

---

cross-modal adapters and uni-modal adapter to produce outcomes tailored to each modality MM". include the previous cross-modal adapters with the current in-adapter and
The paths {PSH current mni-modal adapter. Therefore, each linear's output yTh after adding MoB-based gating G in AnA module can be computed as:
m== "p"
m
yi = Qi(x") + Σ px
where W = (W? Ne of each adapter's path p The gating weights then computed follows:
Ji= represents the gating weights assigned by g", dictating the contribution
T Softmar((" (eatt(((""(x""))))
where 9TT projects each token of embeddings x to I-D vector indicating each modality's 's likelihood of functioning. It is worth noting that do not set the Topk hyper-parameter here. By default, the knowledge of each modality will provide a reference for the current modality. The Softmarfs function normalizes these weights to emphasize the modality- -rrnchh contribution. Finally, the output y"" of AnA with MoE-based gating can be expressed as:
372
y = Q1" (x+")
W A''''x"" + WT A A" (x"""")
(5)
4 Continual Learning on Modality
MCL Benchmark. We establish challenging benchmark, Continual Learning on Modality (MCL), which consists of multimodal high-quality QA data to evaluate the effectiveness of our method on continual uni-modal finetuning. These datasets are collected from five distinct modalities: image, video, depth, audio and point cloud. Based on this benchmark, our Path Weave is trained and tested along the multimodal sequence without requiring modal-specific pretraining or joint-modal finetuning. More details of the dataset list and size for each modality are illustrated in Table A6 of the Appendix.
MCL Metrics. We formulate the metrics from two aspects to evaluate the proposed MCL strategy on multimodal reasoning. On the one hand, we use the general metries from MLLMs [20, 21] to investigate the model's overall performance on learned new modalities. On the other hand, we modify the conventional metrics of continual learning to verify the performance of our method on "catastrophic forgetting". Specifically, for each modality and dataset, suppose S represents the evaluation score on n-th datasets of modality M after training on modality MMM We redefine the forgetting rate [28] to measure the degree of forgetting F,, on all old modalities after each modality stage m:
F.N
m,
(6)
77
where FIN, is the average forgetting across N, datasets of modality after modality m training, and N₁ is the number of datasets in modality i. And the ENN; are defined: m,
N:
FN
Σ
max (S),)) -S
(7)
2
O5i<m
n=1
In addition, we define the forgetting F for the n-th dataset in modality during the training of all modalities:
M
F
Σ
max (5
S m,
(8)
M
USj<m
m=i+1
To measure the overall performance on learned modalities, we further report the average scores of across N,, datasets of modality m after training on m. modality, it can be expressed as:
N
T
Σ
S
Smm.
N
And the performance on learned modalities 7.T for the n-th dataset modality can be expressed as 7. = ST

---

Image
Video
AAudo
Audio DDptt
Depth -DD
Method
Ti
Fi
Ta
F3
Ta
Fa
T4
F4
Continual-FT
51.33
25:50
60.97
57.74
68.19
149.9
WISF-FTTS4)
37.50
1.30
15.70
5.18
67.60
10.94
4.75
13.18
Reg&WWE [25]
39.05
EPPojj288 281
47.60
0.00
17.67
0.00
20.75
0.00
7.75
0.00
Oars
45.08
56.63
83.35
73.45
0.00
Table 1: Comparison with other CL methods on each modalities of in-domain datasets. We label the best and second methods with bold and underline styles. The top gray block indicates the upper-bound scores Tm of transfer learning capability to adapt the new modality.
COCO Tex 1Sg
ASSRVA 150
ASSRVV QA (A [56]
Audio Val (:7]
Audio Caps TESt
1571
AudioCaps QA
CC3M [88]
LLAVASOK 131
Cap QA (89)
Cap3D Co3DDCA Cap [59]
Method
Average
Continual-FT
59.4
74.7
45.8
41.7
108.2
WiSE-FT[ss]
40.5
34.5
9.5
10.5
27.1
84.9
50.3
4.2
5.3
29.64
17
L2 Reg&WE
43.8
34.3
14.4
3.4
4.2
S7.4
52.6
3.2
10.3
28.20
EProi[
55.1
40.1
17.7
10.0
25.3
86.1
55.4
4.9
10.6
33.91
Ours
52.8
37.4
64.0
59.4
46.5
96.5
70.2
39.3
107.6
63.74/+29.85)
Continual-FT FT
80.1
39.0
57.2
40.7
WiSE-FT[sS]
10.3
16.1
5.4
11.5
4.8
7.0
17.0
8.5
3.00
9.29
ir
L2 Reg&&WE [25]
0.5
0.0
8.6
6.3
10.3
0.4
0.0
0.6
3,00
EProj[28]
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.00
Ours
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.00k-300)
Table 2: Comparison with other CL methods on the performance of each in-domain datasets. We label the best and second methods with bold and underline styles. The top gray block indicates the upper-bound scores T.ne of transfer learning capability adapt the new modality.
5
Experiments
5.1
Implementation Details
Our method is built on the LAVIS library's framework [52] atop the Vicuna v1.1 7b [3]. The input preprocessing method remains consistent with X- X-InstructBLIP [20]. We optimize our model on A880 GPUs (80GB) using A [53] with 31 = 0.9, 82 0.999, and weight decay of 0.05. Our initial pre-trained model the image modality model of X-InstructBLIP [20]. During training, the unified incremental module, consisting of Q-former and L.M projection, is continuously trained in the order of image, video, audio, depth, and 3D modalities. During testing, the learnable query and modality encoder are kept modality-specific. The CL methods compared below maintain consistent settings with our method. More details are provided in the Appendix A.2.
5.2 Comparison with State-of-the-art Methods
Transfer Learning on New Modality. As shown in Table 1 and 2, we conduct experiments on existing traditional CL methods under our proposed MCL setting. We report the average expansion capability for each modality, which is represented as Tm and indicates the scalability in the new modality. The inference datasets are in-domain, which involved in model training, and additional results of out-of-domain are provided in the supplementary material. Continual-FT, which refers to continuous learning of each modality without incorporating anti-forgetting strategies, exhibits the best expansion ability due to fine-tuning all parameters but inevitably leads to catastrophic forgetting. In contrast, the methods L2 Reg& WE [25]. WISE-FT [54] and EProj [28] effectively alleviate forgetting by parameter regularization and ensemble, but it is difficult for them to transfer new modality. As shown in Table I, when performing transfer learning on new modalities with significant data distribution gaps from the images, these methods under-perforn ours by at least 38 points on the Audio modality and 66 points on the 3D modality. Furthermore, as shown in Table 2, our method surpasses the current best methods by over 29 points in the average transfer learning metrics across in-domain datasets. This demonstrates that our approach can effectively prevent forgetting while flexibly extending to modalities with substantial data distribution differences.
Alleviate Forgetting Previous Knowledge. We also present the average forgetting rate Fm of historical modality knowledge after training each modality m, shown in the F,, columns of Table I
7

---

Method
Params
All Modal
Data Size
Timest
GPUt
MSVD QA
Clotho Caps
Modelnet Cls
InstrueBBLIP [20]
189
X
27.78M+
0.34x/it
28.7G
51.7
29.4
62.8
OneI.I [2]]
7B+
1007M+
0.83</i
64.8G
56.5
29.1
X-LLM [22]
189.91M+
17.2M+
0.34chi
28.7G
ChatBridge [23]
7B+
4.4++
0.34
28.7G
45.3
26.2
Ours
0.8-2.4M
X
23.2M+
0.236/
13.1G
48.2
28.6
59.5
Table 3: Comparison with state-of-the-art methods on training parameters, data requirements and some performance. "All Modal" indicates whether fine-tuning on all modality datasets is included. "I" represents the same hyperparameters and training settings of different methods for fair comparison.
Method
Vidoo-Audio
Audio Audio-Deptth
Depth-33D
Tiie
Tyoutt
Tatert
Tgweot
Tyeet
Tower
Continual Continual-Adaper
51.17
75.75
49.10
68.00
$1.05
w/o MoE-based gating
43.77
39.35
76.40
49,80
69.50
49.70
w/o In An-Adapeer
52.47
40.78
79.50
50.25
71.35
52.60)
Ours
56.63
42.90
83.35
52.20
73.45
53.70
Table Ablation study of different parts for the influence of the each modalities' performance. We
label the best and second methods with bold and underline styles.
and 2. The results show that continually full finetuning pre-trained modal suffers from catastrophic forgetting. WiSE-FT [54] and 1.2 Reg& WE [25] achieve some effectiveness combating forgetting via parameter regularization and ensemble. However, the constraint of parameters limits their transfer learning on new modalities. In contrast, the EProj [28] and our method achieve forgeffrgettigg by freezing model parameters. However, the scalability of the EProj [28] is significantly lower than our method, especially in the audio and 3D modes. It indicates that our method achieves an optimal balance between anti-forgetting and effective expansion compared to other methods.
Comparison with Existing MLLMs. Table shows the comparison between our approach and state-of-thee -frtthrartt multimodal QA methods in terms of training parameters, required data, training times, GPU usage, and relevant multimodal QA metrics. Among these methods, we unify the settings to ensure fairness in the Times and GPU metrics by only training the instruction tuning stage, setting all batchsize to 4, and keeping the L.LMs BL.IP-based X-LLM and ChatBridge frozen. can be seen that our method demonstrates significant advantage in parameter efficiency compared to X-InsructBLIP [20] and OneLLM [21], reducing parameter training burdens by at least 98. 98.73%. Moreover, compared with OneLLM [21], X-LLM [22], and ChatBridge [23], our approach does not necessitate pre-training and instruction tuning with all joint-modal datasets to adapt to multimodal language reasoning tasks. Our method offers flexible scalability and requires considerably less training data than other methods. The results of the three QA tasks involving video, audio, and 3D, as shown in Table 3, indicate that our approach maintains flexibility without significantly compromising model performance. More experiments are provided in the Table A1l 1 of Appendix.
5.3 Ablation Study
Ablation Study of the In-Adapter and MoE-based gating. We conduct detailed ablation studies on different parts of the proposed method, as shown in Table and 5. Table 4 shows the average performance Túm of transfer learning in each modality. It can be seen that our final method demon- strates increasingly significant performance improvements compared to others when faced with continual modality changes. For instance, as we further extend to depth and 3D modalities, the collaborative synergy between MoE-based gating and In-Adapter becomes increasingly apparent. In addition, Table demonstrates that compared to directly using the incremental adapter method, our approach improves the average performance of transfer learning across all datasets by 4.3 points. When removing the In-Adapter or MoE-based gating, the model's 's transfer learning performance of transfer learning across all datasets decreases by at least 1.1 points and 4.0 points. It indicates the effectiveness of our proposed In- In-Adapter and MoE-based gating, which enhance inter-modal interactions and modulate cross-modal knowledge.
Analysis of the Benefit from Previous Modalities. Figure presents the ablation study on the ability to transfer learning based on different knowledge of modalities. As shown in Figure 3 (a). our method enhances the scalability of audio modality after incorporating additional video modality training. It indicates that our designed method can extract knowledge from the other adapter to enhance the learning of the current modality. addition, when more than one modality is additionally introduced, method can still enhance new generalization by modulating inter-modal knowledge and fine-

---

Austin Cpe Val (57)
Text (57)
Audio QA
1SC000 Cls (00)
ESC50 Open (60)
ClothoQA (6))
Clatbo Caps [62]
CCOM [88)
LLAVA [S]
NYU V2 1631
SUN RGBB D (6-1)
Modelnes CIS [65]
Modilnes Open (65)
Cap3D QA (59).
Cop3D Cap
Method
Audoo MaahuCaa
Average
Continaal-Adapter
61.0
60.9
11.6
66.3
1688
30.2
28.8
90.7
60.8
58.7
39.5
56.2
45.9
35.8
100.2
$3.49
MoE-based gating
52.1
$1.0
28.2
34.1
28.7
27.5
92.5
60.3
58.4
41.2
55.8
43.6
36.1
102.9
$1.97
Adapter
71.4
93.9
65.1
59.1
414
58.5
46.2
37.9
55.15
Ours
64.0
59.4
46.5
72.6
36.9
33.5
28.6
96.5
70.2
62.2
422
39.5
47.9
39.3
107.6 87.79(02.44)
Table 5: Ablation study of different parts for the influence of the each dataset's performance. We
label the best and second methods with bold and underline styles.
78
90
180
48
Rased
Reedd
Andioape
E:SC50
BSCC90
Clatke
Cheto
CCIM
NVT
SUN
Maddle Madeleet CapdD
Capii
Ope
ag
Caps
RGBD
Open Q4
Captin
(a) The performance of audio
(b) The performance of depth (e) The performance of point
Figure 3: Ablation study of the TV" performance for the n-th dataset in modality which benefits from knowledge of different modalities. "Based on I-V-A-D" represents training point modality based on pre-trained Path Weave that is trained in the sequence of image, video, audio, and depth.
tuning frozen knowledge with In-Adapter shown in Figure 3 (b) and (c). It demonstrates that our method can enhance the adapting new modalities by knowledge learned from other modalities.
5.4
Qualitative Analysis
Figure shows the qualitative results of our method for inference on each modality after continual training is completed. We show our final model (a) understand visual content in images, (b) leverage temporal information in videos, (c) scene understanding using depth maps, (d) do creative writing based on audio content, and (e) understand the details of 3D shapes. More qualitative results are provided in Figure A5 of the Appendix.
detal
What doing?
How bear the audi?
What purtt?
What the mede?
This image
dip fooe
bench warlcciing
proparing
When sudio, beach, with the
conan. The aceas doe bike there ore woves breaking active
seen stering stove, edding dngrec pauring
soud the wones creahing and the wind blowing
playing gane of atteute friiee in groaty
This is
(a) Image modality Figure 4: Qualitative results of our method on each modality after continuous training.
(b) Video modality
(c) Audio modality
(d) Depth modality
(c) 3D modality
Conclusion and Discussion
We propose a flexible and scalable framework for multi-modal language reasoning that enables MLLMs to continually expand on multiple modalities without joint- datascts. We introduce an incremental Adapter-in-Adapter (AnA) strategy, incorporating two types of adapters to enhance modality plasticity and collaboration during expanding on other modalities. Moreover, we design an MoE-based gating module to further enhance multi-modal integration by modulating the output space of different modalities. Extensive experimental results in our proposed benchmark demonstrate the superiority of method over previous arts in terms of modality alignment and memorization.
limitation of this paper is that we only explored the extension of five modalities and do not cover all modal information in real-world scenarios. Furthermore, the implicit interaction between the modalities in our method cannot accomplish cross-modal joint language reasoning tasks in an incremental manner.

---

Acknowledgements
This work was supported by National Natural Science Foundation of China under Grant 62206039, 62293544, and the Fundamental Research Funds for the Central Universities (DUT24RC(3)025),
References
[I] Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, (140))1167, 2020.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[3] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph Gonzalez, et al. Vicuna: open- chatbot impressing gpt-4 with 90% * chatgpt quality. See https://vicun. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subhiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advancex in neural information processing systems, 33: 13:1877-1901, 2020.
[5] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[6] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- Inguage-image image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[7] Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems, 36, 2024.
[8] Xingjian Du, Zhesong Yu. Jiaju Lin, Bilei Zhu, and Qiuqiang Kong. Joint music and language attention models for zero-shot music tagging. In ICASSP 2024-2024 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 1126-11 11300 IEEE, 2024.
[9] Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. Pengi: An audio language model for audio tasks. Advances in Neurol Information Processing Systems, 36:1869:0 18900-188188 2023.
[10] Dingning Liu, Xiaoshui Huang, Yuenan Hou, Zhihui Wang, Zhenfei Yin. Yongshun Gong, Peng Gao, and Wanli Ouyang. Uni3d-llm: Unifying point cloud perception, generation and editing with large language models. arxiv preprint arXiv:2402.03.: 033,27377 2024.
[11] Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36:20482-20094, 2023.
[12] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks, arXiv preprint arXiv:2312. 14238, 2023.
[13] Wenhai Wang. Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36. 2024.
[14] Wenliang Dai. Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- vision- lision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[15] Junwen Hc, Yifan Wang, Lijun Wang, Huchuan Lu, Jun-Yan He, Jin-Peng Lan, Bin Luo, and Xuansong Xie. modti-modal instruction tuned Imss with fine-grained visual perception. arXiv preprint arXiv:2403.0296 2024.
[16] Jianing Yang. Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. Lim groundde:rr:: Open-vo vocabulary 3d visual grounding with large language model as an agent. arXiv preprint 230.3331331111111,.
10

---

[17] Chan Hee Song. Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. LIm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998-3009, 2023.
[18] Jiawen Zhu, Zhi-Qi Cheng. Jun He, Chenyang Li, Bin Luo, Huchuan Lu, Yifeng Geng, and Xuansong Xie. Tracking with human-intent reasoning. arXiv preprint arXiv:2312. 17448, 2023.
[19] Hui Yang, Lekha Chaisom, Yunlong Zhao, Shi- Shi-Yong& Neo, and Tat Tat-Seng Chua. Videoqa: question answering on news video. In Proceedings of the eleventh ACM international conference on Multimedia, pages 632-641, 2003.
[20] Artemis Panagopoulou. Le Xue, Ning Yu, Jumnan Li, Dongxu Li, Shafiq Joly, Ran Xu, Silvio Savarese. Caiming Xiong, and Juan Carlos Niebles. X-instructblip: framework for aligning x-modal instruction- aware representations to Ilms and emergent cross-modal reasoning. arXiv preprint arXiv:2311.8775 2023.
[21] Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yuc. Onellm: One framework to align all modalitics with language. arXiv preprint arXiv:2312.037000 2023.
[22] Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-IIm: Bootstrapping advanced large language models by treating multi-modalities foreign languages. arXiv preprint arXiv:2305. 1600, 2023.
[23] Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model language catalyst. arXiv preprint 16100 2023.
[24] Peipei Song, Dan Guo, Xun Yang, Shengeng Tang, and Meng Wang. Emotional vidco captioning with vision-based emotion interpretation network. IEEE Transactions on Image Processing, 2024.
[25] Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation continual learning of vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19125-19138 2023.
[26] Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang. Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixuureoof-experts adapters. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23219-23230, 2024.
[27] Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang, Multimodal instruction tuning with conditional mixture of lora. arXiv preprint arXiv:2402.15880 2024.
[28] Jinghan He, Haiyun Guo. Ming Tang, and Jingiao Wang. Continual instruction tuning for large multimodal models, arviv preprint arXiv:231 1..223311.16066, 2023.
[29] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7)3366633885, 2021.
[30] David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[31] Frantzeska Lavda, Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Continual classification learning using generative models. arXiv preprint 1800 10612, 2018.
[32] David Lopez-Paz Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[33] Oleksiy Ostapenko, Timothee Lesort, Pau Rodríguez, Md Rifat Arefin, Arthur Douillard, Irina Rish, and Laurent Charlin. Continual learning with foundation models: An empirical study of latent replay. In Conference on lifelong learning agents, pages 60-91. PMLR, 2022.
[34] Sylvestre-Alvise Rehuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 2001-2010, 2017.
[35] Sang-Was Woo Lee, Jin- -wa Kim, Jachyun Jun, Jung-War Woo Ha, and Byoung Tak Zhang. Overcoming catas- trophic forgetting by incremental moment matching. Advances in neural information processing systems. 30, 2017.
11

---

[36] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference machine learning. pages 3987-3995. PMLR, 2017.
[37] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan. Tiago Ramalho, Agnieszka Grabska- Barssk-ka et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 14((13):521-3356, 2017.
[38] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5138-5146, 2019.
[39] Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer vision-ECCV 2020: 16th European conference, Glasgow; UK, August 23-28, 2020, proceedings, part XX 16, pages 86-102. Springer, 2020.
[40] Saihui Hou, Xinyu Pan, Chen Change Loy. Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 831-839, 2019.
[41] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40 20:12)22335 2017.
[42] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate; Lifelong learning with a network of experts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3366-3375, 2017.
[43] Arthur Douillard, Alexandre Ramé, Guillaume Couairon, and Matthieu Cord. Dytox: Transformers for continual learning with dynamic token expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 0285-92955 2022.
[44] Zhiyuan Hu, Yunsheng Li, Jiancheng Lyu, Dashan Gao, and Nuno Vasconcelos. Dense network expansion for class incremental learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 11858-1 1867, 2023.
[45] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 200.0000...00 096655 2021.
[46] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning, Advances in Neural Information Processing Systems, 35:12991-13005. 2022.
[47] Haiwen Diao, Bo Wan, Ying Zhang, Xu Jia, Huchuan Lu, and Long Chen. Unipt: Universal parallel tuning for transfer leaming with efficient parameter and memory. arXiv preprins arXiv:2308. 14316, 2023.
[48] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709 727. Springer, 2022.
[49] Нaoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang. and Yinfei Yang. Ferret: Refer and ground anything wnyweere at any granularity. arXiv preprint arXiv:2310.07704, 2023.
[50] Jinjin Xu, Liwu Xu, Yuzhe Yang, Xiang Li, Yanchun Xie, Yi-Jie Huang, and Yaqian Li. u-llava: Unifying multi-modal tasks via large language model. arXiv preprint trXiv:2311.05344 2023.
[51] Yukun Li, Guansong Pang, Wei Suo, Chenchen Jing, Yuling Xi, Lingqiao Liu, Hao Chen, Guoqiang Liang, and Peng Wang. Coleclip: Open-domain continual learning via joint task prompt and vocabulary learning. arXiv preprint arXiv:2403. 10245, 2024.
[52] Dongxu Li, Junnan Li, Hung Lc, Guangsen Wang, Silvio Savarese, and Steven CH Hoi. Lavis: A library for language-vision intelligence. arXiv preprint arXiv:2209.0000 2022.
[53] Ilya Loshchilov and Frank Hutter. a0Xiv:171..050012200
Decoupled
weight decay regularization. arXiv preprint
[54] Mitchell Wortsman, Gabriel Ilbarco, Jong Wook Kim, Mike Li, Simon Kornblith. Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine- tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7959-7971, 2022.
12

---

[55] Soravit Changpinyo, Piyush Sharma. Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558-3568, 2021.
[56] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Mst-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288-5296, 2016.
[57] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 119 132, 2019.
[58] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernmmed, image alt-text datasct for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 2556-2565, 2018.
[59] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models. Advances in Neural Information Processing Systems, 36, 2024.
[60] Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015--018, 2015.
[61] Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho-age aqa: A crowdsoureed dataset for audio question answering. In 2022 30th European Signal Processing Conference (EUSIPCO), pages 1140-1 144. IEEE 2022.
[62] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736-740. IEEE, 2020.
[63] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision-ECCV 2012: 12th European Conference Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746-760. Springer, 2012.
[64] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgh-d: rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567-576, 2015.
[65] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[66] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10786-10796, 2021.
[67] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pages 740-755. Springer, 2014.
[68] Vicente Ordonez, Girish Kulkami, and Tamara Berg, Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
[69] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32 73, 2017.
[70] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. European Conference on Computer Vision, pages 146-162. Springer, 2022.
[71] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/eyf conference on computer vision and pattern recognition, pages 1194-3204, 2019.
13

---

[72] Anand Mishra. Shashank Shekhar. Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. 2019 international conference on document analysis and recognition (ICDAR). pages 947-952. IEEE, 2019.
[73] Yash Goyal, Tejas Khot, Douglas Summers- Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 6904 6913, 6017.2017.
[74] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. Proceedings of the 25th 2017.
ACM international conference on Multimedia, pages
[75] Xinhao Mei, Chutong Meng, Haobe Liu, Qiugiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A Chatgpt-assiste wcakly-labelled audio captioning dataset for audio-language multimodal research. arXiv preprint arXiv:2303. 17395, 2023.
[76] Max Bain, Arsha Nagrani, Gül Varol. and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 1728-1738, 2021.
[77] Yuxin Fang. Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang. Tiejun Huang, Xinlong Wang. and Yue Cao. Eva: Exploring limits masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358-19369, 2023.
[78] Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09055. 2022.
[79] Drew Hudson and Christopher D Manning. Gqa: A new dataset real-world visual reasoning and compositional question answering. Procedingss of the IEEF/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[80] David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190-200, 201 1.
14

---

A Appendix
A.1 Dataset Details
We summarize the multimodal-text dataset in Table A6 for modality continue learning. For depth-text pairs, adopt the DPT model pre-trained on ominidata [66] to generate depth maps. The source dataset a subset of CC3M [58], around 0.5M image-text pains and 50K image-text pairs random sampled from VLaVA-150K [5].
LLaVA data includes multiple rounds of dialogue. To align with our training process, we randomly select one round as a training sample. This selection method also applies when creating the validation set, where these samples remain fixed and do not change during testing.
Modality
Size
Dataset
MS COCO [67], CapFilt14M [6], CC12M [55]
SBU Captions [68], Visual Genome [69], AOK VQA [70]
Image
21.3M
OK VQA [7 OCR VQA [72]. Visual Genome QA [69]
VQAV2 [73], LLaV [5]
Video
0.2M
MSRVTT [56], MSRVTT-QA [74]
Audio
0.3M
WavCaps [75], AudioCaps [57], AudioCaps-QA [57]
3D
0.9M
Cap3D [59], Cap3D-QA
Depth*
0.5M
CC3M [58], LLa VA-5OK [5]
Total
23.2M+
All Datasets
Table A6: Datasets for continually uni-modal finetuning. Our datasets extensions of X-
InstructBLIP [20], in contrast, we additionally included depth data and removed inaccessible video
data WebVid2M [76]. represent data generated ourselves.
A.2 Training Evaluation Details
Table A7 records the detailed hyper- pepee--prameters we used during the training and testing process. It worth noting that the training of our method on each modal data is continuous. The encoders for image, video, and depth are set WVA-CLIP-ViT-G/I [77]. The audio and 3D encoders are BEATS [78] and ULIP-2, respectively.
When using WiSE-FT [54] and L2 Reg& [25] methods for training, in order to be as consistent as possible with the original approach, we update the weights of the Q-Former and LLM projection layer in each inner epoch (for WiSE-FT, we set update coefficient a as 0.8). For example, when we train on Audio modal data, the total training iteration set to 65000, and 5000 iterations per inner epoch, then the number of weight updates is 13 times in the current situation.
During modality backward testing for methods in Table 1, we keep the encoder and Q-Former queries consistent with the test modality. We utilize the same instruct prompts as X-InstructBL.IP [20] during training and testing.
Modality
Iteration
Batch Size (Taai
Learning Rate
Video
15K
16/8
lc-5
Audio
65K
16/8
le-5
Depth
35K
4/8
le-5
3D
65K
16/16
le-5
Table A7: Hyper-parametert for modality continue learning. We keep all the learning rate decrease from 1e-5 and cosine annealing strategy with 0.5 decay weight. The warm-up phase starts from 1e-8 and lasts for 1000 iterations for all modality training.
A.3 Complete Raw Data
Table A8 records all the original data of the methods compared in Table We highlight the transfer learning performance in new modality of each method with green color.
15

---

DISE-T
Continual FT
Ours
[Pro]
1. Reg
WE WB
Point
Depth
Audio
Video
Image
Poinl
Depth
Audio
Video
Image
Point
Depth
Audio
Video
Image
Point
Depth
Audio
Video
Image
Point
Depth
Audio
Video
Image
46.1
4K
18
46.1
48
88.1
47.9
47.9
47.7
42.5
45.8
47.1
40.2
31.8
46.1
GQA [
Image modality
137
137.9
137.3
196.
136.5
131.2
150.5
50.5
112.8
137.7
COCO Va [55]
1802
L382
L352
1308
L30.
61.0
112.1
138.2
CO T ( [55]
W3
43.1
S
233
MSVD QA [74]
Video modality
1065
1055
125
125
45.7
MSVD ASAD Cap (80)
Table A8: Raw data records of all compared CL methods in all modalities.
53.8
23
MSRVTT [56]
MSRVYT QA [50]
37.4
27.4
X74
40
10.
AudioCaps Val [57]
53.1
AudioCaps Test [57]
AudioCaps QA (57)
Audio modality
30
36.1
ESC50 Cls [60]
2
2.8
ESC50 Open [60]
88
12.5
ClathoAQA [61]
Clotho Caps [62]
CC3M [58]
Depth modality
LLAVA50K [5]
NYU v2 [63]
SUN RGB-D [64]
Modelnet Cls [65]
504
Modelnet Open [65]
modality
Cap3D QA [59]
Cap3D Cap [59]
16

---

A.4 Additional Experiments
As shown in Table A9 and Table A10, we conduct experiments analyze the performance on out- of-domain data in addition to the in in-domain experiments. Our method shows robust generalization while maintaining anti- forgettngg performance on out-of-domain data. Specifically, compared with the full-finetune method, our average accuracy only decrease 0.33 points, while achieving 31.34 points forgeftgg capability. At the same time, with the same powerful anti-forgetiing ability as EProj [28], the generalization of our method between different modalities improves 18.95 points.
Image
Video
Audio
D3pth-3D
Method
Ti 1
Fi 4
T2
Fg 4
Ta
Fs L
T4 1
F4 J
Continual-FT
93.60
16.30
33.75
34,63
53.35
45.30
56.45
49.67
WiSE-FTT[54]
69.95
1,00
5.88
50.15
4.60
11.00
7.87
1.2 We [25]
73.75
0.40
4.45
0.00
52.65
3.99
1.15
5.20
EProj] [28]
87.15
0.00
10.9
0.00
51.3
0.00
14.25
0.00
Ours
27.54
0.00
42.9
0.00
52.2
0.00
53.7
0.00
Table A9: Comparison with other CL methods on each modality of out-of-domain datasets. We label the best and second methods with bold and underline styles, The top block indicates the upper-bound scores Tm of transfer learning capability adapt the new modality.
GQA [7)]
AAWADA Cup 1aW
ESCSO [60]
ESCS0 Openn Open
ClotbAQA 1611
Clotbo Caps [62]
NYU 1263
SUN ROB-D
Modslaxt Cls [65]
Modelnet Open (65)
Method
Averuge
Continual-FT WiSE-FT[S44
50.7
136.5
66.2
18.4
24.3
26.1
62.1
44.6
62.5
50.4
54.18
45.7
94.2
5.2
2.1
14.4
1.8
60.7
39.6
13.5
8.5
28.57
1/ 1
Reg&WE [25]
47.1
100.4
2.0
13.4
1.6
61.9
43.4
1.8
0.5
27.29
EProjj 28]
48.7
125.6
12.9
2.8
18.6
9.3
61.9
40.7
15.1
13.2
34.90
106.9
(3.85)
Continual-FT FT
22.8
36.5
96.1
46.9
7.15
20.7
21.3
17.3
13.3
31.34
WiSE-PT[S4]
5.1
33.4
1,9
7.0
0.6
7.9
8.05
Fr 4
L2 Reg&WE [25]
0.2
3.1
8.7
0.4
0.7
1.1
0.0
0.5
1.62
EProo[
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
01.00
Ours
0.00k-1662)
Table A10: Comparison with other CL methods on the performance of each out-of -ott---- dataset. We label the best and second methods with bold and underline styles. The top block indicates the upper-bound scores 7T of transfer learning capability to adapt the new modality.
We quantitatively compare the results our method and other multi-modal large language models that support multiple modalities in Table A11. 1. Compared with other MILMs, achieve better trade-off between model performance and the number of supported modalities with fewer learnable parameters and less training data.
Method
MSVD MSVD Cap ESC30
ESC30 Open
ClothoAQA
Clotha Caps
NYU SUN
Modelnet Modeliet Open
X-InstructBLIPP [2]]
48.1
52.5
118.2
75.9
38.2
15.4
29.4
62.8
46.7
OneLLM
56.8
ChatBridge [25]
45.3
26.2
Ours
47.8
48.2
106.9
72.6
35.9
33,5
28.5
62.2
42.2
59,5
47.9
C cOCO. MSRVIE MSRVITUAAAdioCCun AudioCCaps AndioCapsQAA LLaVA QA Csps3U
InstructBLIP [20]
OneLLLM [21]
565
CharBrddge [23]
45.3
26.2
Ours
137.
138.7
52.8
37.4
61.0
39.4
16.5
96.5
70.2
39.3
107.
Table A11: Comparison with state-of-the-an methods metrics of different datasets.
In addition, we provide more qualitative results on each modality in Figure A5.
A.5 More training details
All modalities are trained by an Autoregressive CE loss. The detailed hyperparameter settings for each modality are shown in Table A12 of the attached PDF. We will provide further details and descriptions of the loss and hyperparameters in the paper to ensure better clarity and flow.
17

---

Please describe this image in detail.
What does this video describe?
OF ST
The image shows blue double-decker bus on city street. The bus has a digital display on the front that reads "Ashington," indicating route or destination. The word "ARRIVA" is prominently displayed on the front of the bus, suggesting is part of Arriva bus network.
This video shows cartoon of witch chasing cat away with broom and then reading book to cat.
What does sound like?
What the mon doing ?
What is this 3D model used for?
An audio clip from the forest
sounds like bird singing, with leaves rustling the background. These sounds very pleasont and make people wontt get close to nature.
The might using both wii controller and balance board while playing the game because the wii balance board.
This 3D model of potted plant, which be used to cultivate different plants. The specific situation must be judged according to its volume.
Please describe image detail.
What this 3D model showing?
The image shows two large teddy bears pasitioned side by side, with third smaller teddy bear in the background. (The teddy bear in foreground on the is wearing a blue Hawaiian-stee sttaliann-stleeeeoo shirt with floral pattern and a brown bow tie with the text "Teddy Bear Museum" printed on
This model Rhinos, short rhinoceroses, are large, herbivorous mammals for their characteristic horns their snoutsore. known their poor eyesight but have excellent senses smell and hearing.
Figure A5: More qualitative results of our method on each modality after continuous training.
18

---

Modality
Dataset
Prompt
Len. Penalty
Min Len.
Max Len.
based the given the image respond to
GQA [79]
-1.
10
Image
{}
COCO Val [55]
short description
1.
10
80
COCO Test [55]
short description
1.
10
80
based on the given
MSVD QA [74]
vidco respond to {
MSVD Cap [80]
short description
10
80
Video
MSRVTT [56]
short description
1.
10
80
based the given
MSRVTT QA [56]
-1.
10
video respond {
AudioCaps Val [57]
short description.
1.
10
80
AudioCaps Test [57]
short description.
1.
10
80
Question: ()
AudioCaps QA [57]
-1.
10
Answer:
Audio
ESC50 Cls [60]
describe the audio.
80
ESC50 Open [60]
describe the audio,
10
80
Question: ()
ClothoAQA [61]
-1,
10
Answer:
Clotho Caps [62]
a short description.
1.
10
80
A short description of the depth:
CC3M [58]
1.
30
Question: ()
LLAVASOK [5]
30
Answer:
(class} What is the
category this
Depth
NYU v2 [63]
scene? Choice one
30
class from the class sets.
(class) What is the category of this scene? Choice one class from the class sets.
SUN RGB-D [64]
1.
8
30
describe the model.
Modelnet Cls [65]
0.
10
80
based on the given
Modelnet Open [65]
80
inpul respond to [).
Point
describe the 3d
Cap3D QA [59]
1.
3
model.
describe the 3d model.
Cap3D Cap [59]
1.
3
Table A12: More details of hyperparameters used each of the datasets.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have claimed the contributions in both abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in Conclusion and Discussion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We have explained the methodology.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided the implementation details including training and testing.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
We will release the code once acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt / ssssssoliccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provided the details accprdingly.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: We set the parameters following previous methods without strictly adjusting the parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We claimed the resource demand.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the csQuidel NeurIPS Code of Ethics https: httpar///////u/..eeeee//ee cc cc/ ppublic/Ethi t ines?
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No social impacts exist in this work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We used the public benchmarks as previous methods.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have credited the related benchmark.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We used published benchmarks.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our research didn't contain this issue.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "BWq8qydoWr", "reviewer": "Reviewer_PRBt", "summary": "This paper proposes a flexible and scalable framework, PathWeave, which enables MLLMs to allow MLLMs gradually to involve reasoning ability on diverse modalities. The introduction of the adapter-in-adapter structure effectively alleviates the heavy burdens of the joint training or data replay strategies in previous MLLM methods. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance while significantly reducing training parameters by 98.73% compared to OneLLM and X-InstructBLIP.", "questions": "This method is based on the X-InstructBLIP. If the LLaVA-based method is employed, utilizing a projection layer for modality encoders and LLMs, will this method still be effective? Besides, did the authors verify the generalizability of this method?", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper proposes an interesting framework to remedy the parameter and data burdens existed in training MLLMs. The proposed adapter-in-adapter framework exists novelties in multi-modal interaction and single-modal learning. Besides, they also provide a new benchmark to support the continual training and evaluation.\t\n+Contribution&Results: This paper proposes a novel PathWeave, which combines the transfer learning and continual learning method to progressively expand LLMs on multiple modalities. And the paper proposes a new benchmark MCL to evaluate the model’s overall performance on learned modalities and the performance on “catastrophic forgetting”. The extensive experimental results provide sufficient experimental details and verification, which validates the effectiveness and superiority of the proposed methods.\n\n+Inspiration: The proposed method exhibits promising experimental results for inspiring the subsequent work. Specifically, Figure 3 shows the positive effect of different modalities’ knowledge on special modal expansion, which is promising for using old parameters to incrementally learn new knowledge. \n\n+Presentation: The paper writing is good and the idea of paper is easy to follow. Both the figures and tables are easy to understand.", "weaknesses": "-In Tables 4 and 5, there are no experimental results for Image-Video, which should be added. Furthermore, I think that the red subscript for the Average results in Table 5 may be unnecessary.\n\n-Tables 4 and 5 show that the final method reduces the $T_2$ performance compared to the method “w/o In-Adapter”. However, the explanation in lines 287-289 is ambiguous, which requires further analysis.\n\n-In Table 3, the metrics of XLLM[22] are missing. Please explain the reasons.\n\n-The authors only provided the comparison on Params and data size, it is better to provide more sufficient analysis to show the efficiency of this continual learning on modalities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for all your comments. We respond to each of the weaknesses and questions you raised to address your concerns and make the necessary revisions to improve the quality of the paper. Our point-by-point summary and responses to your comments are as follows. (Notes: the mentioned Tables 1-5 are in the original paper, and Table R#PRBt 1-4 are in the attached PDF)\n\n- **Weakness 1: No results for Image-Video and suggestion about removing subscript.**\n\n  - The reason for not presenting the Image-Video results in Tables 4 and 5 is that the results of all ablation methods are identical, **as shown in Table R\\#PRBt-1**. Specifically, when extending to the first modality (video), we only add the uni-adapters $\\mathcal{A}^1$ without the In-Adapter $\\mathcal{F}_{i}^m$ and MoE-based gating modules $\\mathcal{G}^m$. The cross-adapters $\\hat{\\mathcal{A}}^m$ with gating and In-Adapter modules are only employed when extending to the second modality and other modalities. We will provide a more detailed explanation in the revised version to clarify this issue.\n\n  - Furthermore, we have removed the red subscript of Table 5 from the original text.\n\n- **Weakness 2: The performance of the method \"w/o In-Adapter\" in Tables 4 and 5.**\n\n  $\\quad$ In response to this comment, we provide replies and analyses based on Table 4 and Table 5, respectively:\n\n  - **For the $T_{2(in)}$ in Table 4**, the reported score is an average of the AudioCaps Val, AudioCaps Test and AudioCaps QA in Table 5. According to these results of the original paper, we find that there is a small calculation typo in the $T_{2(in)}$, which should be 64.85 $\\rightarrow$ 52.47. The revised table **is shown in Table R\\#PRBt-2(a)** and the results are corrected in the revised paper. The results indicate that our final model performs best in Table 4.\n\n  - **For the results of AudioCaps Val in Table 5**, we visualize some cases where the final model's scores are much lower than those of the \"w/o In-Adapter\" model. **In Table R\\#PRBt-2(b)**, we show the CIDEr score of these cases obtained by the final model and the \"w/o In-Adapter\" model, as well as a comparison of their caption results. It can be seen that although the final model has a lower CIDEr score, the answer quality is comparable to the \"w/o In-Adapter\" model. The final model can describe more objects ```pigeons``` in \"Case id: wIvYjuR3nrg\" and object ```engine``` in \"Case id: yZmhM1HcsyE\". In addition, excluding this dataset, the final model's performance on the other 14 datasets and the average result is higher than the \"w/o In-Adapter\" model.\n\n  $\\quad$ Therefore, we believe that these situations do not affect our experimental conclusions. We will add the above related analysis in the revised version to further improve the quality of the paper.\n\n\n\n- **Weakness 3: The metrics of XLLM[22] are missing.**\n\n  $\\quad$ There are **two main reasons** why the metrics of X-LLM are missing here:\n\n  - The X-LLM paper does not provide any relevant test results.\n\n  - We attempt to use the released code to test inference results on various datasets, but the model weights are not available.\n\n  $\\quad$ Therefore, we mainly want to highlight the trainable parameters and the requirement for joint-modal datasets of X-LLM to showcase the strengths of our method, as illustrated in Table 3. We are trying to rerun the code to supplement the missing results, and we will add the results in the revised version if it can be completed. It may take some time, as the computational resources and time required could exceed 11 days on 8 GPUs. The resource consumption details from the X-LLM paper are listed **in Table R\\#PRBt-3**.\n\n\n- **Weakness 4: More analysis about the efficiency of the method.**\n\n  - We further analyze the **Times Cost and GPU Memory** during the training stage between the comparison methods and ours. The relevant results are **shown in Table R\\#PRBt-4**. The experiments demonstrate that our method is not only more flexible in extending dataset modalities but also more efficient in terms of training time and memory usage. \n\n \n  - Different methods involve different numbers of modalities and training settings. To ensure fairness, we unify the settings in our evaluation process by using the common modality dataset MSRVTT, the same hyperparameters and training settings: only training on the instruction tuning stage, setting all Batchsize to 4, and keeping the LLMs of BLIP-based X-LLM and ChatBridge frozen.\n\n- **Question: Effectiveness of the proposed modules for LLaVA-based methods and verification of generalizability.**\n\n  $\\quad$ Thanks for your inspiring question.\n\n  - For LLaVA-based methods, the proposed modules is effective in terms of anti-forgetting and flexibly expanding modalities for LLaVA-based methods, but it suffers from a significant decline in transfer learning performance. The main reason is that under our setting, the only learnable parameters in LLaVA-based methods are from a linear projection, which is too limited to achieve good transfer performance across multiple modalities. The EProj results of Tables 1 and 2 in the main paper also indirectly proves this situation.\n\n  - Therefore, we mainly focus on the Q-Former-based methods under our setting and do not further explore the our modules' generalizability for other methods, whose training strategies and learnable parameters mostly not be suitable for our setting.\n\n  $\\quad$ We believe that this is an interesting question to inspire our future work on how to expand existing pre-trained models to new modalities or tasks with fewer trainable parameters, such as a single linear projection. And we will add a related discussion about this future work in the revised paper.\n\n> [22] Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.04160, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J5PNykRJyd", "reviewer": "Reviewer_odey", "summary": "Due to a serious illness I have been experiencing recently, I deeply regret to inform you that I am unable to complete the review as scheduled. I kindly request the Chair to consider the opinions of the other reviewers", "questions": "-", "limitations": "-", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-", "weaknesses": "-", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C43HNbFJE0", "reviewer": "Reviewer_fRFW", "summary": "The authors introduce continual learning into MLLMs to explore the ability of pre-trained LLMs to evolve continually on multiple modalities while keeping knowledge from being forgotten. A novel PathWeave with Adapter-in-Adapter (AnA) is proposed, in which uni-modal and cross-modal adapters are seamlessly integrated to facilitate efficient modality alignment and collaboration. The authors establish a challenging benchmark, MCL, to investigate the proposed method’s performance on the new modality and all previous knowledge. The experimental results are encouraging, significantly reducing training parameter burdens.", "questions": "See weakness.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is logical, fluent, and easy to understand.\n\n2. The proposed method enables existing pre-trained large models to progressively expand on multiple modalities without requiring joint training on all modalities. This idea of continually learning knowledge from pre-trained models to expand the modality is novel and could inspire the further exploration of multimodal works.\n\n3. This paper establishes a challenging MCL benchmark to explore the generalization and anti-forgetting of cross-modal continual learning of pre-trained models. It is a promising benchmark that evaluates the performance of modality expansion of pre-trained MLLMs. \n\n4. This paper conducted sufficient experiments on five modalities and more than 20 datasets. The performance is comparable to the joint training MLLMs while significantly reducing parameter burden. In addition, the comparison with other continual learning methods shows the state-of-the-art generalization performance. More experiments in the supplementary materials further demonstrate the effectiveness of this method.", "weaknesses": "1. Compared with fine-tuning all parameters, the performance of the adapter fine-tuning method still has room for improvement. I think some necessary discussion, analysis, or experimentation should be conducted. \n\n2. What are the similarities and differences between this method and VPGTrans[1]? It is a highly efficient Visual Prompt Generator Transfer across LLMs with less training data and even task improvements. I believe that a related analysis of these two works is needed.\n\n3. Some minor issues. 1) It seems that the modality sequence in Fig. 2 is unmatched with the experiments in Table 1. 2) In Fig. 2, the upper and lower trapezoids of A1, A2, etc. of AnA are represented by the same symbol. Do they share the parameters? 3) Some training details, including loss functions and hyperparameters for each modality, are not clear.\n\n[1] Vpgtrans: Transfer visual prompt generator across llms. NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **Weakness 1: The performance gap between the PEFT and fully finetuning method.**\n\n    $\\quad$ Thanks for your comments. Our method does fall slightly behind in performance metrics compared to fully fine-tuning approaches. This is primarily because that the Parameter-Efficient Fine-Tuning (PEFT) method naturally affects performance compared to fully fine-tune all parameters, as shown in the Continual-Adapter of Tables 4 and 5 of the main paper. **More importantly**, the main advantage of our method is the ability to flexibly and efficiently extend to multiple modalities sequentially while reducing the forgetting of previous modalities. Specifically, \n\n  - Our method reduces the number of trainable parameters by 98.73\\% compared to fully fine-tuning methods. \n  - In contrast, our method demonstrates significantly superior anti-forgetting capabilities compared to fully fine-tuning methods.\n  - Compared to other anti-forgetting methods, our approach achieves state-of-the-art performance on transfer learning metrics $T_n$ and $\\hat{T}_i^n$ and anti-forgetting $F_n$ and $\\hat{F}_i^n$, as shown in Tables 1 and 2 of the main paper. \n\n  $\\quad$ Therefore, these comparisons and results demonstrate that our method achieves the optimal balance between flexible modality extension and anti-forgetting. To further highlight our contribution, we have added related discussion and analysis in the revised version.\n\n- **Weakness 2: Related analysis between this paper and VPGTrans [1].**\n\n  $\\quad$ Thank you for bringing this related work to our attention. We summarize the differences and similarities between the related work and ours as follows:\n\n  $\\quad$ **Similarities**: \n\n  - Both works emphasize leveraging previous knowledge to facilitate new transfer tasks.\n  - Both works have studied the transfer learning performance based on the BLIPs architecture, with LLMs kept frozen.\n\n  $\\quad$ **Differences**:\n  - Our work focuses on transfer learning across different modality datasets, while VPGTrans focuses on transfer learning across different language models.\n  - We address the issue of forgetting for previous modalities by continual learning strategy, whereas VPGTrans does not consider the problem of forgetting for previous models.\n  - VPGTrans has utilized the warming-up process on the projection layer to prevent performance drop and expedite VPG training.\n\n  $\\quad$ The similarities between the two works indicate that transfer learning for new modalities or tasks can benefit from existing modalities or models. The comparison with VPGTrans can highlight our advantages in modality expansion and alleviate forgetting of previous knowledge, while also inspiring us to further optimize transfer learning performance in new modalities through warming-up strategies on some network layers. We will add related analysis and discussion in the revised version. \n\n- **Weakness 3: Some minor issues and more hyperparameters.**\n\n  $\\quad$ Thanks for your comments and reminders. we have revised and added related content in the following three aspects.\n\n  - **Modality order correction.** Thanks for your reminder. we have corrected the order of modality in Figures 1 and 2 of the main paper.\n\n  - **Further descriptions of the notation in adapters.** The upper and lower trapezoids of A1, A2, etc. of AnA represent the up linear projection and down linear projection of LoRA, whose parameters are not shared. The relevant descriptions are on lines 181 and 182 of the main paper. We will add the above parameter settings in the revised version.\n\n  - **More training details.** All modalities are trained by an Autoregressive CE loss. The detailed hyperparameter settings for each modality are shown in Table R\\#fRFW-1 of the attached PDF. We will provide further details and descriptions of the loss and hyperparameters in the paper to ensure better clarity and flow.\n\n>  [1] Vpgtrans: Transfer visual prompt generator across llms. NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
To all reviewers:

We are grateful to all the reviewers for their valuable comments. We hope that our responses have effectively addressed your previous concerns. We have revised our paper according to your comments. **The major changes are summarized as follows:**

- **According to Reviewer#PRBt Comments:**
  - Experimental Results. We add further analysis and results about the missing Image-Video results in Tables 4 and 5, as shown in Table R#PRBt-1 of attached PDF. 

  -  Metrics of XLLM. We provide the explanation for the missing metrics of XLLM and add it in Sec 5.2 of the revised paper.

  - Ablation Study. We correct the results of the $T_{2(in)}$ of Table 4, as shown in Table R#PRBt-2(a) of attached PDF. We give more Q&A visualization of some failure cases in our final model to demonstrate the effectiveness of our final model, as shown in Table R#PRBt-2(b) of attached PDF.  

  - Efficiency. We conduct more experiments on the Times Cost and GPU Memory, as shown in Table R#PRBt-4 of attached PDF.


- **According to Reviewer#fRFW Comments:**

  -  Performance Gap. We add more analysis about the advantages of our method compared with fully finetuning method.

  - Related Work "VPGTrans". We summarize the similarities and differences between the VPGTrans and our method, and we add related analysis and discussion in the revised version.

  - Some Issues and Implementation Details. We correct these minor issues and provided more details (Table R#fRFW-1 of attached PDF) about our method in the revised version.

We take this as a great opportunity to improve our work and would appreciate any further feedback you can provide.

Sincerely yours,

Authors.

Meta-review: Thank you for the detailed rebuttal. After reviewing the reviewers' comments and the authors' responses, I recognize the contributions of this paper in introducing continual learning into multimodal large language models (MLLMs). The novel PathWeave with Adapter-in-Adapter (AnA) method presents an innovative approach to expanding the capabilities of pre-trained models across multiple modalities while effectively addressing the challenge of knowledge retention. Given the logical presentation, novel contributions, and comprehensive validation, I recommend accepting this paper.
Decision: Accept (poster)