Model Composition for Multimodal Large Language Models (2024)

Abstract: Recent developments in Multimodal Large Language Models (MLLMs) have shown rapid progress, moving towards the goal of creating versatile MLLMs that understand inputs from various modalities. However, existing methods typically rely on joint training with paired multimodal instruction data, which is resource-intensive and challenging to extend to new modalities. In this paper, we propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model. Our basic implementation, NaiveMC, demonstrates the effectiveness of this paradigm by reusing modality encoders and merging LLM parameters. Furthermore, we introduce DAMC to address parameter interference and mismatch issues during the merging process, thereby enhancing the model performance. To facilitate research in this area, we propose MCUB, a benchmark for assessing ability of MLLMs to understand inputs from diverse modalities. Experiments on this benchmark and four other multimodal understanding tasks show significant improvements over baselines, proving that model composition can create a versatile model capable of processing inputs from multiple modalities.1

heading: 1 Introduction
text: Recent advancements in Multimodal Large Language Models (MLLMs) have established them as the forefront of multimodal learning paradigms (Liu et al., 2023b,a). The prevalent approach involves aligning modality encoders with large language models (LLMs) through extensive modality-text paired data and then fine-tuning with modality-specific instruction data. This paradigm
*Equal contribution. BCorresponding authors: Peng Li (lipeng@air.tsinghua.
edu.cn) and Yang Liu (liuyang2011@tsinghua.edu.cn). 1Code is available at https://github.com/THUNLP-MT/ ModelCompose
has been successfully applied to a wide range of modalities such as image (Liu et al., 2023b; Dai et al., 2023), audio (Gong et al., 2023; Deshmukh et al., 2023; Tang et al., 2024), video (Zhang et al., 2023; Lin et al., 2023), and point cloud (Hong et al., 2023; Xu et al., 2023), resulting in the emergence of a diverse array of MLLMs with unique modal capabilities.
There are also some efforts to enable a single MLLM to handle multiple modalities. One way to achieve this is to align the LLM with a multimodal encoder such as ImageBind (Girdhar et al., 2023) with only image-text data (Figure 1(a)) (Su et al., 2023; Han et al., 2023). This approach leverages the inherent alignment of different modalities within the multimodal encoder, allowing the MLLM to comprehend various modalities to a certain degree. However, the absence of modalityspecific instruction data often results in suboptimal performance. Another approach entails the concurrent training of the MLLM with multiple
11246
modality encoders (Figure 1(b)). For example, ChatBridge (Zhao et al., 2023) connects image, video and audio encoders with the LLM through a joint training process with multimodal instruction data (i.e., video-audio chats). This kind of methods show potential but faces two major challenges. First, it is resource-heavy to collect paired data across multiple modalities. Second, adapting these models to new modalities requires additional training, adding to the complexity and resource demands of the development process.
Given the limitations of current approaches, we propose and study a more practical setting: model composition for MLLMs (Figure 1(c)). Our primary research question is simple yet fundamental: Can we create a new MLLM by combining existing MLLMs to inherit their understanding of different modalities without training? Model composition for MLLMs is advantageous for two key reasons: (1) it eliminates the need for the resource-heavy process of training and gathering multimodal data, and (2) it promises enhanced adaptability, facilitating seamless incorporation of new modalities.
Some recent studies, such as X-InstructBLIP (Panagopoulou et al., 2023), serve as pioneering efforts in model composition for MLLMs. These works primarily train projectors to align different encoders with a single LLM and demonstrate the ability to process multiple modalities concurrently. However, a critical limitation is their applicability only to MLLMs with frozen language model weights. This constraint restricts the range of models that can be utilized, and impairs the overall performance of the MLLMs (Zeng et al., 2023).
In this paper, we first propose a framework for model composition for MLLMs. Our implementation, named NaiveMC, is elegantly simple: for the MLLMs to be composed, we directly reuse their modality-specific encoders and merge their LLM parameters. We demonstrate that MLLMs, as long as initialized from the same LLM, can achieve zeroshot multi-modality expansion through this model composition framework, regardless of whether the parameters of the LLM have been fine-tuned.
Furthermore, to mitigate parameter interference in the composition process and optimize the performance of the composite model, we propose DAMC, an advanced framework with parameter Decoupling and Adjustment for Model Composition. By separating modality-specific parameters from language model parameters during initial MLLM training, DAMC allows for the selec-
tive merging of textual parameters, reducing crossmodal interference. Moreover, DAMC introduces an adaptive parameter adjustment mechanism to ensure optimal compatibility and effectiveness of the composite model, achieving a balanced and efficient multi-modality expansion.
To assess the efficacy of our proposed frameworks, we conduct comprehensive experiments on tasks that require an integrated understanding of inputs from diverse combinations of four prevalent modalities: image, audio, video, and point cloud. To facilitate the research in model composition for MLLMs, we also build MCUB, a benchmark specifically designed for evaluating the capability to concurrently comprehend multiple modalities by identifying commonalities across inputs from various modalities. Experimental results indicate that our frameworks enable the composition of existing MLLMs from different modalities without requiring further training, yielding a versatile and high-performing multimodal model adept at handling any combination of these modalities.
Our contributions are three-fold:
• We propose the concept of model composition for MLLMs, realized through the NaiveMC framework, which allows for seamless integration of different MLLMs without additional training, enabling zero-shot multi-modality expansion. • We introduce DAMC, an advanced model composition framework that employs parameter decoupling and adaptive adjustment to mitigate parameter interference and optimize composite model performance across multiple modalities. • We create MCUB, a benchmark designed to evaluate the unified understanding of diverse modalities, and demonstrate the efficacy of our model composition frameworks via extensive experiments on various multimodal understanding tasks and MCUB.

heading: 2 Related Work

heading: 2.1 Multimodal Large Language Models
text: Recent advancements have integrated Large Language Models (LLMs) with multiple modalities such as image, video, audio, and point cloud. Approaches like X-LLM (Chen et al., 2023) utilize modality-specific adapters, while ChatBridge (Zhao et al., 2023) employs a Perceiver for each modality. Macaw-LLM (Lyu et al., 2023) uses a
unified alignment module, and NeXT-GPT (Wu et al., 2023) relies on linear projection for multimodal integration. However, these methods often necessitate joint multimodal dataset training, posing scalability and extension challenges. While PandaGPT (Su et al., 2023) and ImageBind-LLM (Han et al., 2023) bypass joint training with ImageBind (Girdhar et al., 2023) as a unified encoder, their performance is limited by a lack of modality-specific instruction data. In contrast, our approach benefits from instruction-following training for each MLLM and integrates their capabilities through a training-free process.

heading: 2.2 Model Composition
text: Model composition integrates different models’ capabilities, primarily through weight space merging strategies (Ilharco et al., 2022; Matena and Raffel, 2022; Jin et al., 2023; Huang et al., 2023; Yu et al., 2024; Wortsman et al., 2022), enhancing fine-tuned models from the same initialization. However, existing research has not fully explored model composition for multimodal LLMs. Prior works on multimodal tasks use linear interpolation for tasks like speech recognition (Sundar et al., 2023). Another work converts model processing images and text separately into one using same parameters for simultaneous processing (Sung et al., 2023). But they do not expand the processing modalities of the model. X-InstructBlip (Panagopoulou et al., 2023) incorporates new modalities by training separate projections for modal encoders and LLM, but its performance is limited by the frozen LLM. Our work advances multimodal capabilities by enabling LLM training and applying model composition techniques to LLM weights, significantly improving the effect of multimodal integration.

heading: 3 Methodology

heading: 3.1 Task Definition
text: In this work, we examine the composition of a set of Multimodal Large Language Models (MLLMs), denoted as {M1,M2, . . . ,Mn}, each capable of responding to textual queries with different modalities m = {m1,m2, . . . ,mn}. The core objective is to develop a composition method C that effectively integrates these MLLMs into a singular, more versatile model Mcompose = C(M1,M2, . . . ,Mn). This integration aims to enable the model to process and understand inputs from any combination of modalities in m. For example, by integrating two
specialized MLLMs—a vision LLM and an audio LLM—the resulting composite model should not only preserve the individual proficiencies of these models in processing images and audio, respectively, but should also acquire a zero-shot capacity for handling inputs that encompass both visual and auditory information simultaneously.
Algorithm 1 Model Composition for MLLMs Input: Set of MLLMs {M1,M2, . . . ,Mn}, each
with a set of parameters Θi
Output: Integrated parameters Θcompose 1: Initialize Θcommon and Θcompose as empty sets 2: Define a mapping f : θ 7→ G that maps the
parameter θ to group G based on functionality 3: for i = 1 to n do 4: for each parameter θ in Θi do 5: if ∀j ̸= i, θ /∈ Θj then 6: Add θ to Θcompose 7: else 8: Assign θ to group f(θ) in Θcommon 9: end if
10: end for 11: end for 12: for each group G in Θcommon do 13: θmerge = average(parameters in G) 14: Add θmerge to Θcompose 15: end for 16: return Θcompose

heading: 3.2 A Model Composition Framework
text: When composing models, two critical elements must be taken into account: the components and their weights. The prevalent MLLMs typically feature two component types: (1) modal-specific components, like modal encoders and projectors, which adapt modality inputs to the language embedding space and (2) modal-agnostic components that exist in each MLLM, primarily the underlying LLM itself. In our composition framework, we retain all modal-specific components (and their weights) from different MLLMs to handle respective modal inputs, and connect them to the same LLM. In cases where the LLMs have not been adapted during the training of MLLMs (as illustrated in Figure 2(a)), we employ the pre-trained weights of the LLM directly. Conversely, if the LLMs have undergone adaptation in the MLLM training process (Figure 2(b)), we simply average their weights. We name this composition framework NaiveMC and provide a formal procedure of it in Algorithm 1.

heading: 3.3 Parameter Decoupling
text: In the previously described framework, there arises an unaddressed issue, specifically the potential for parameter interference when merging fine-tuned LLM parameters. During the training of MLLMs, the LLM parameters are optimized for specific modal inputs. This specialized optimization could lead to variations in the LLM parameters of different MLLMs. When these parameters are merged, conflicts may arise, potentially impacting the ability of the model to understand modal inputs.
To alleviate the issue of parameter interference during model composition, we advocate for initially training the MLLMs with a parameter decoupling strategy in the first place. As shown in Figure 2(c) and Figure 2(d), the main idea is to separate the modality processing parameters from those of the language model within MLLMs. For example, in an MLLM M that processes modality m, the input for each attention layer is denoted as X = [Xm, Xt] where Xm and Xt represent the modality-specific and text sequences, respectively. The attention components are computed as follows:
Q = [XmW Q m , XtW Q t ] (1)
K = [XmW K m , XtW K t ] (2)
V = [XmW V m , XtW V t ] (3)
where Q, K, and V represent the queries, keys, and values in the attention mechanism, respectively. Note that two sets of weights W (·)m and W (·) t are employed, tailored to the modality-specific and tex-
tual inputs. The attention operation is then applied:
[XOm, X O t ] = split(Attention(Q,K, V )) (4)
which splits the attention output into modalityspecific and textual outputs. The final unified output representation Xo is obtained by:
Xo = [X O mW O m , X O t W O t ] (5)
ensuring separate processing streams for each modality within the model. Similarly, given input X for each feed-forward layer, the output is:
FFN(X) = [FFNm(Xm),FFNt(Xt)] (6)
where FFNm and FFNt are feed-forward layers for modality-specific and text inputs, respectively.
When composing MLLMs that are trained through parameter decoupling, we merge only the text-related parameters, maintaining distinct modality-specific parameters as depicted in Figure 2(d). The composite model functions as a natural extension of the methodology previously described, guaranteeing that inputs from each modality are processed independently with their respective parameters. By doing so, it effectively mitigates the risk of interference from other modalities, ensuring that the composite model maintains high fidelity in processing multimodal data. Please note that after the MLLMs are trained, the composition phase remains emphatically training-free.

heading: 3.4 Adaptive Parameter Adjustment
text: Due to variations in data quality and training strategies among different MLLMs, their performance can significantly differ. Thus, when composing these models, employing a simplistic averaging strategy often falls short of achieving optimal results. To enhance model effectiveness, it is crucial to implement adaptive adjustments to the model parameters during the composition process, allowing for better compatibility and flexibility among the different models. Specifically, for N distinct MLLMs {M1,M2, . . . ,MN} and any parameter θi common across these models, where i = 1...N , the merged parameter is defined as:
θmerge = N∑
i=1
λiθi (7)
where λi represents the adjustment coefficient. For simplicity, we adopt a uniform adjustment coefficient λi for all θi in Mi. For models trained using parameter decoupling, we can additionally adjust their modality-specific parameters if needed. The values of these coefficients can be determined with a validation set from target tasks requiring various modal inputs. If such a validation set is not available, a practical alternative is to select the coefficients based on general performance of the model on tasks of each modality. We refer to the updated model composition framework with parameter decoupling and adjustment as DAMC.

heading: 3.5 Multimodal Commonality Understanding Benchmark
text: As previously discussed, DAMC inherently possesses the advantage of scaling up the number of modalities. However, there are few benchmarks available to evaluate the performance across various modalities. To demonstrate the effectiveness of our approach on tasks involving numerous modalities, inspired by Panagopoulou et al. (2023), we introduce a new benchmark called the Multimodal Commonality Understanding Benchmark (MCUB). We provide an example of MCUB in Figure 3. The task of MCUB is to measure the ability of the model to identify commonalities among input entities from diverse modalities and select the most appropriate answer from four given candidates.
We leverage GPT-4 (Achiam et al., 2023) to create MCUB from existing captioning datasets across various modalities. To be specific, we begin by randomly selecting groups of inputs from each modal-
ity, and retain the groups with the highest semantic similarity. This similarity is quantified by averaging the similarities of their respective captions. Subsequently, GPT-4 is prompted with in-context examples to generate questions, options, and correct answers for each group. In this study, we develop two variants of MCUB: MCUB-3 and MCUB-4. MCUB-4 comprises data entries that include inputs from four modalities—image, video, audio, and point cloud. In contrast, MCUB-3 consists of four subsets, each representing combinations of inputs from any three of these modalities. For more details, please refer to Appendix C.

heading: 4 Experiments

heading: 4.1 Implementation
text: In our experimental setup, we explore four modalities: image, audio, video and point cloud. To ensure a comprehensive and comparative analysis, we reimplement MLLMs for each modality following previous works (Liu et al., 2023a; Lin et al., 2023; Panagopoulou et al., 2023; Xu et al., 2023). For each of these modalities, we train three versions of MLLMs with same data and hyperparameters but vary the trainablity of LLM parameters: a model with a frozen LLM, a fully trainable LLM, and a trainable LLM with parameter decoupling, as illustrated in Figure 2. We employ the LoRA (Hu et al., 2021) technique for efficient LLM training. All models are based on Vicuna-7B-v1.5 (Zheng et al., 2023). More details for training the MLLMs are in Appendix A. We apply parameter adjustment by conducting a search for the optimal λi values within the set [1/N, 2/N, . . . , N/N ] for composition of N modalities based on the performance of the validation set for each corresponding task. The results for the best-performing parameters are documented in Appendix B.
of video (V), image (I) and audio (A) inputs. Methods developed in this study are distinguished with a grey background for clarity.

heading: 4.2 Evaluation
text: Datasets and Benchmarks. Our aim is to evaluate the performace of the composite MLLM to process inputs from multiple modalities. In addition to the MCUB benchmark described in Section 3.5, we also include datasets from two key areas for evaluation: (1) audio-visual question answering including MUSIC-AVQA (Li et al., 2022) and AVQA (Yang et al., 2022) where image, video and audio inputs are available and (2) 3D object classification on ModelNet40 (Wu et al., 2015) and Objaverse (Deitke et al., 2023) where image and point cloud inputs are considered.
Baselines. Our evaluation strategy involves comparison with baselines primarily focusing on models that do not leverage training data from multiple modalities. We consider two representative baselines for this purpose: ImageBind-LLM that aligns a multimodal encoder, ImageBind, to the LLM with image-text paired data and XInstructBLIP that aligns individual modal encoders to a frozen LLM using instruction data specific to each modality. Regarding model composition, we further compare our DAMC approach against two alternative intermediate baselines as discussed before: Proj-only where the LLM is frozen and only the modal encoders and connectors are composed and NaiveMC that directly composes MLLMs without adopting our proposed parameter decoupling and adjustment techniques.

heading: 4.3 Main Results
text: The main experimental results on audio-visual question answering, 3D object classification and MCUB are presented in Table 1, Table 2 and Table 3, respectively. In the tables, the notation “X+Y” represents the combination of different types of modal inputs used in evaluating the models. For example, “V + A” refers to the combination of video (V) and audio (A) inputs being used together as part of the input to the model. For model composition, we compose MLLMs based on the specific modal inputs required. Generally, our proposed DAMC achieves the highest performance across all input combinations and tasks, demonstrating
its effectiveness of adeptly managing inputs from multiple modalities even without any training on these modal combinations. In addition, we make the following observations:
Model composition brings performance improvements with additional modal inputs. The results reveal a generally consistent trend where the application of model composition methods leads to improved performance as more modal inputs are integrated. For example, in Table 1 in the MUSIC-AVQA task, the transition from solely video (V) inputs to combinations that include image (I) and audio (A) inputs leads to noteworthy performance boosts. Specifically, when combining these three modalities, the performance for our DAMC reaches a peak at 57.32, marking a notable increase compared to V+I/A. In contrast, models like ImageBind-LLM and X-InstructBLIP do not show similar improvements when transitioning from V+I/A to the V+I+A combination. This demonstrates that models developed through model composition can handle inputs from different modalities more effectively, showcasing their ability to improve multimodal understanding.
DAMC outperforms strong baselines. Compared to previous methods, DAMC achieves superior performance, even surpassing methods that utilize paired multimodal data, such as ChatBridge and 3D-LLM (Hong et al., 2023). When comparing the performance across various model composition methods, DAMC significantly outperforms Proj-only and NaiveMC, particularly evident in scenarios involving the integration of more than two modalities. On the MCUB benchmark, for instance, DAMC achieves a significant performance enhancement, with an improvement over the second best NaiveMC of +5.10 points in scenarios combining three modalities and +6.05 points in the subset involving four modalities. This dis-
tinction proves the capability of DAMC to mitigate interference effectively when merging multiple MLLMs, demonstrating its robustness and efficiency for model composition.

heading: 4.4 Ablation Study
text: Our ablation study, summarized in Table 4, evaluates the effects of parameter decoupling and adjustment on DAMC. The findings reveal that employing neither strategy yields an average performance of 62.79 across benchmarks. Introducing parameter decoupling alone enhances the average to 65.05, while adjustment alone improves it to 63.55. Remarkably, combining both strategies boosts the average performance to 66.24, with significant improvements noted in all benchmarks. This highlights the synergistic impact of parameter decoupling and adjustment in optimizing multimodal model composition and achieving superior performance across varied tasks involving different types of modality inputs.

heading: 4.5 Qualitative Results
text: Figure 4 demonstrate the capability of the composition model to understand and reason over multimodality inputs. Additional qualitative results are in Appendix F.

heading: 5 Discussion and Analysis
text: Composition of MLLMs with different training strategies. In Section 3.3, we advocate training MLLMs with parameter decoupling. Intriguingly, this decoupling can also be implemented at inference time, by replicating the parameters to mimicking identical parameters on modality-specific and textual inputs. In this way, we can compose models with different training strategies. We conduct experiments on the MUSIC-AVQA dataset with V + I + A inputs. We vary the training strategy of video and image MLLMs, and compose them with an audio MLLMs trained with parameter decoupling. The results in Table 6 indicate that decoupling at inference time is also effective, which is beneficial when aiming to integrate existing models that were not trained with parameter decoupling.
Composite model performance on the singlemodality task. To explore the effectiveness of the composite model on single-modality tasks, we conducted experiments on the vision-language understanding evaluation benchmark MME (Fu et al., 2023). The results are shown in Table 5. Each composite model processes only images and textual questions. The ∆ values denote the relative improvements of DAMC over NaiveMC. As more models are integrated, the performance of
NaiveMC tends to decline, while DAMC consistently shows improvement. This highlights the capability of our proposed methods to mitigate parameter interference effectively when composing models. Moreover, it is noteworthy that model composition enhances performance even in tasks limited to a single modality. We attribute this to the possibility that our approach could integrate knowledge from different models during the combination process, which we will leave for future research.
Impact of trainable parameters. We investigate whether the benefits of parameter decoupling arise from the augmentation of trainable parameters, given that two distinct sets of parameters are allocated for varying inputs. To this end, we train a new vision LLM without decoupling but increase the rank of the LoRA modules from 128 to 256, which doubles the trainable parameters. The findings, as shown in Table 7, reveal that merely expanding the pool of trainable parameters does not enhance performance. This suggests that the advantage of parameter decoupling extends beyond simple parameter quantity increase, highlighting its effectiveness in mitigating parameter interference.

heading: 6 Conclusion
text: In this paper, we introduce and explore the concept of model composition for MLLMs, showcasing its ability to seamlessly integrate diverse modal capabilities without additional training. Initially, we introduce a foundational model composition framework, referred to as NaiveMC. Advancing
further, we design DAMC that employs parameter decoupling and adjustment to reduce crossmodal interference and optimize performance of the composite model. In addition, we construct a benchmark MCUB for multimodal commonality understanding evaluation to facilitate the relative research. Extensive experiments demonstrate the effectiveness of our approaches. We hope that our work will inspire further exploration into model composition for multimodal models.
Limitations
Our exploration is restricted to four commonly used modalities, omitting a comprehensive examination across the entire spectrum of potential modalities. This limitation may result in missed opportunities to identify further benefits or challenges associated with model composition. Additionally, our proposed approach has been tested on models of specific sizes, leaving the applicability of the mehods on larger-scale MLLMs an open question for future research.

heading: Acknowledgments
text: This work is supported by the National Key R&D Program of China (2022ZD0160502) and the National Natural Science Foundation of China (No. 61925601, 62276152).

heading: B Parameter Adjustment
text: In preliminary experiments, we find that composing MLLMs rom two modalities typically achieved optimal results through a direct average, specifically a 1/2 + 1/2 combination. Due to time and resource constraints, we only conducted parameter adjustments on three types of tasks. This process involved conducting a search for the optimal λi values within the set [1/N, 2/N, . . . , N/N ] for the composition of N modalities, based on the validation set performance for each corresponding task. The results are showcased in Table 8. We assume that the variance in coefficients comes from the differing demands of each task on the understanding capabilities across modalities. For MCUB-4, which requires a comprehensive grasp of content from all four modalities, an average coefficient emerged as the best result. Based on these findings, we also applied average coefficients for all MCUB-3 tasks.

heading: C Details of Multimodal Commonality Understanding Benchmark
text: Table 11 presents the captioning datasets we use to generate MCUB task. For point cloud modality, we reserve 3000 point clouds from Objaverse (Deitke et al., 2023) dataset following Xu et al. (2023), and obtain their captions in Cap3D (Luo et al., 2023) dataset. Note that this part of the point clouds are not used in training and 3D object classification.
The semantic similarity of a group of entities is obtained by averaging the similarities of the captions of all two-entities combination, which are calculated by the all-MiniLM-L6-v2 model (Reimers and Gurevych, 2019). For example, if a group of entities with captions pair (A,B,C) is provided, the group similarity of it will be the average of the similarities between (A,B) , (A,C) and (B,C).
We report the detailed results on subtasks of MCUB-3 benchmark in Table 12. The final result of MCUB-3 is reported as the average of the four sub-tasks.
Prompt template for generate questions, options
and correct answers:
Prompt Template
Given entity A with caption "A cat meowing and humans speaking on the background." with properties: agile, independent, and domesticated, entity B with caption "Loud barking and traffic" with properties aggressive, loud, and high energy, entity C with caption "Birds chirping in a quiet forest" and properties peaceful, wild, and vocal, and entity D with caption "A bustling city street with honking cars" and properties busy, noisy, and chaotic, you can generate a set of instruction answer pairs to find the commond point of the entities as follows: Example: Question: Which of the followings are the common point of the four entities.
A. Rhythmic ocean waves crashing on the shore. B. Gentle rustling of leaves in a serene garden. C. Audible environmental sounds. D. Soft crackling of a campfire under a starry sky. Answer: C. Explanation: The maximum common point among the four entities is the presence of ambient environmental sounds, which can be perceived audibly. Generate three such Question, Answer, Explanation triplets for entity A with caption "<Caption A>" and properties "<Properties A>", entity B with caption "<Caption B>" and properties "<Properties B>", entity C with caption "<Caption C>" and properties "<Properties C>", and entity D with caption "<Caption D>" and properties "<Properties D>" Examples:

heading: D Prompt for Evaluation
text: We list the evaluation prompts for each dataset and modality combination in Table 13. In the prompts, we use “<image>”, “<audio>”, “<video>” and “<point>” to represent image, audio, video and point cloud modality inputs.

heading: E Additional Point Cloud Results
text: In Table 2, we report exclusively on the zero-shot 3D object classification performance using P + I (Point cloud + Image) inputs, with comprehensive results detailed in Table 14. We find that ImageBind-LLM, X-InstructBLIP, and Proj-only struggle with point cloud inputs alone, particularly in adhering to open-ended generation instructions, leading to poor results. We assume that this issue likely stems from the necessity for point MLLM to undergo training on point-text instruction data with trainable LLM parameters to enhance performance.

heading: F Additional Qualitative Results
text: We provide additional qualitative results about in
Figure 5-7.

heading: Future Work
text: Additionally, our proposed approach has been tested on models of specific sizes, leaving the applicability of the mehods on larger-scale MLLMs an open question for future research