LLMs Can Evolve Continually on Modality for $\mathbb{X}$-Modal Reasoning

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 08:37:30

Abstract: Multimodal Large Language Models (MLLMs) have gained significant attention due to their impressive capabilities in multimodal understanding. However, existing methods rely heavily on extensive modal-specific pretraining and joint-modal tuning, leading to significant computational burdens when expanding to new modalities. In this paper, we propose \textbf{PathWeave}, a flexible and scalable framework with modal-\textbf{path} s\textbf{w}itching and \textbf{e}xp\textbf{a}nsion abilities that enables MLLMs to continually \textbf{ev}olve on modalities for $\mathbb{X}$-modal reasoning. We leverage the concept of Continual Learning and develop an incremental training strategy atop pre-trained MLLMs, enabling their expansion to new modalities using uni-modal data, without executing joint-modal pretraining. In detail, a novel Adapter-in-Adapter (AnA) framework is introduced, in which uni-modal and cross-modal adapters are seamlessly integrated to facilitate efficient modality alignment and collaboration. Additionally, an MoE-based gating module is applied between two types of adapters to further enhance the multimodal interaction. To investigate the proposed method, we establish a challenging benchmark called \textbf{C}ontinual \textbf{L}earning of \textbf{M}odality (MCL), which consists of high-quality QA data from five distinct modalities: image, video, \textcolor{black}{audio, depth} and point cloud. Extensive experiments demonstrate the effectiveness of the proposed AnA framework on learning plasticity and memory stability during continual learning. Furthermore, PathWeave performs comparably to state-of-the-art MLLMs while concurrently reducing parameter training burdens by 98.73\%. Our code locates at \url{https://github.com/JiazuoYu/PathWeave}.

Main paper: 
  
 Appendix:  


Decision: Thank you for the detailed rebuttal. After reviewing the reviewers' comments and the authors' responses, I recognize the contributions of this paper in introducing continual learning into multimodal large language models (MLLMs). The novel PathWeave with Adapter-in-Adapter (AnA) method presents an innovative approach to expanding the capabilities of pre-trained models across multiple modalities while effectively addressing the challenge of knowledge retention. Given the logical presentation, novel contributions, and comprehensive validation, I recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a flexible and scalable framework, PathWeave, which enables MLLMs to allow MLLMs gradually to involve reasoning ability on diverse modalities. The introduction of the adapter-in-adapter structure effectively alleviates the heavy burdens of the joint training or data replay strategies in previous MLLM methods. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance while significantly reducing training parameters by 98.73% compared to OneLLM and X-InstructBLIP.
strength_weakness: strength_weakness: -In Tables 4 and 5, there are no experimental results for Image-Video, which should be added. Furthermore, I think that the red subscript for the Average results in Table 5 may be unnecessary.

-Tables 4 and 5 show that the final method reduces the $T_2$ performance compared to the method “w/o In-Adapter”. However, the explanation in lines 287-289 is ambiguous, which requires further analysis.

-In Table 3, the metrics of XLLM[22] are missing. Please explain the reasons.

-The authors only provided the comparison on Params and data size, it is better to provide more sufficient analysis to show the efficiency of this continual learning on modalities.
questions: questions: This method is based on the X-InstructBLIP. If the LLaVA-based method is employed, utilizing a projection layer for modality encoders and LLMs, will this method still be effective? Besides, did the authors verify the generalizability of this method?
limitations: limitations: Yes.
paper_summary: paper_summary: Due to a serious illness I have been experiencing recently, I deeply regret to inform you that I am unable to complete the review as scheduled. I kindly request the Chair to consider the opinions of the other reviewers
strength_weakness: strength_weakness: -
questions: questions: -
limitations: limitations: -
paper_summary: paper_summary: The authors introduce continual learning into MLLMs to explore the ability of pre-trained LLMs to evolve continually on multiple modalities while keeping knowledge from being forgotten. A novel PathWeave with Adapter-in-Adapter (AnA) is proposed, in which uni-modal and cross-modal adapters are seamlessly integrated to facilitate efficient modality alignment and collaboration. The authors establish a challenging benchmark, MCL, to investigate the proposed method’s performance on the new modality and all previous knowledge. The experimental results are encouraging, significantly reducing training parameter burdens.
strength_weakness: strength_weakness: 1. Compared with fine-tuning all parameters, the performance of the adapter fine-tuning method still has room for improvement. I think some necessary discussion, analysis, or experimentation should be conducted. 

2. What are the similarities and differences between this method and VPGTrans[1]? It is a highly efficient Visual Prompt Generator Transfer across LLMs with less training data and even task improvements. I believe that a related analysis of these two works is needed.

3. Some minor issues. 1) It seems that the modality sequence in Fig. 2 is unmatched with the experiments in Table 1. 2) In Fig. 2, the upper and lower trapezoids of A1, A2, etc. of AnA are represented by the same symbol. Do they share the parameters? 3) Some training details, including loss functions and hyperparameters for each modality, are not clear.

[1] Vpgtrans: Transfer visual prompt generator across llms. NeurIPS 2023.
questions: questions: See weakness.
limitations: limitations: N/A

Comments: title: Looking forward to your post-rebuttal feedback
comment: Dear Reviewer PRBt:

We sincerely appreciate your time and efforts in reviewing our paper. We have provided corresponding responses and results, which we believe have covered your follow-up concerns. We hope to further discuss with you whether or not your concerns have been addressed. Please let us know if you still have any unclear parts of our work. 

Best,

Authors
Thank you for reading the response and your support of our work! We are glad that we have addressed your concerns.
The rebuttal has addressed my initial concerns. Overall, I believe the proposed contributions are clearly demonstrated, and the paper is well-organized. Therefore, I will keep my rating as acceptance.