Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE
Authors: Zeren Chen
Ziqin Wang
Zhen Wang
Huayang Liu
Zhenfei Yin
Si Liu
Lu Sheng
Wanli Ouyang
Jing Shao
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Model (LLM)
Multi-task learning
Multi-modal learning
Mixture-of-Experts (MoE)
PEFT
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Recent studies have demonstrated Large Language Models (LLMs) can extend their zero-shot generalization capabilities to multimodal learning through instruction tuning. As more modalities and downstream tasks are introduced, negative conflicts and interference may have a worse impact on performance. While this phenomenon has been overlooked in previous work, we propose a novel and extensible framework, called Octavius, for comprehensive studies and experimentation on multimodal learning with Multimodal Large Language Models (MLLMs). Specifically, to mitigate the interference, we combine the concept of Mixture-of-Experts (MoE) with LoRA and design a multimodal LoRA-MoE decoder for task- and modality-specific learning. To the best of our knowledge, we are one of the pioneering efforts to introduce MoE into MLLMs to address this problem. The experimental results (about 20% improvement) have shown the effectiveness and versatility of our design in various 2D and 3D downstream tasks. Code and corresponding dataset will be available
soon.

Published as a conference paper at ICLR 2024
OCTAVIUS: MITIGATING TASK INTERFERENCE IN MLLMS VIA LORA-MOE
Zeren Chen' 2+ Ziqin Wang Zhen Wang', Huayang Liu" Zhenfei Yin 144, Si Liu', , Lu Sheng", Wanli Ouyang Jing Shao
Shanghai Al Laboratory, School of Software, Beihang University,
3 Institute of Artificial Intelligence, Beihang University, University of Sydney (czr1604, wzqin, wz1234, hy43, liusi, 1sheng)0buaa edu. cn, (yinzhenfei, shaojing) @p wwl.bborg.cc..... org.en, wanli. owyangesydeyyyynccooooooo org.a.
ABSTRACT
Recent studies have demonstrated Large Language Models (LLMs) can extend their zero-shot generalization capabilities to multimodal learning through instruc- tion tuning. As more modalities and downstream tasks are introduced, nega- tive conflicts and interference may have worse impact performance. While this phenomenon has been overlooked in previous work, we propose a novel and extensible framework, called Octavius, for comprehensive studies and ex- perimentation on multimodal learning with Multimodal Large Language Mod- els (MLLMs). Specifically, we combine the well-known Mixture-of-Expert (MoE) and one of the representative PEFT techniques. i.e., LoRA, designing novel LLM--bssed decoder, called LoRA-MoE, for multimodal learning. To the best of our knowledge, we are one of the pioneering efforts to introduce MoE into MLLMs to address this problem. The experimental results (about 20% improvement) have shown the effectiveness and versatility of our design in various 2D and 3D downstream tasks. Code and datasets are available at https://open qithub. ho/paper List/Octavius
INTRODUCTION
Multimodal Large Language Models (MLLMs) (Alayrac et al. 2022 Huang et al. 2023 Liu et al.
2023 Li et al., (203a; Zhu et al., (023) have been considered promising general-purpose inter- faces that can perform various multimodal tasks under few-/zero-shot settings. Apart from lever- aging the powerful Large Language Models (LLMs) (OpenAl 2023 Touvron et 2023a) as the universal interfaces that unify the responses to different types of tasks as task-specified textual se- quences, the keys to the success of MLLMs are to reliably perceive more modalities and effi- ciently fine-tuned to adapt more downstream tasks.
To achieve this goal, MILMs rely on the instruction-tunin scheme (Ouyang et al. 2022) where model is fine-tuned based on multimodal eostruction-follwviii dialogues orchestrated from various multimodal tasks. Morcover, thanks the Parameter-Efficient Fine- Tunnng (PEFT) techniques (e.g., LoRA (Hu et al. 2021) and Adapter (Houlsby et al. 2019)) where only small trainable components are injected in the model and updated during fine-tuning, recent MILMS (Zhang et al., 2023 Yin efficiently learn to solve downstream tasks with small scale of annotated data, while preserve the language proficiency and generalizability to novel situations. Re- markably, these models achieve comparable performance low costs in comparison LLa VA (Liu
let al. 2023 Ye et al., 2023)
et al. 2023), KOSMOS series (Huang et al., 2023 Peng et al. 2023) and Shikra (Chen et al. 2023 which are learned by full model fine-tuning with large amount of multimodal data.
However, PEFT has to address the crucial tug-of-war problem (Hadsell et al. 2020), where simulta- neously learning different tasks may cancel each task-specific optimization out, and ultimately com-
"Equal contribution.
I Project leader.
1 "orresponniig author.

---

Published as a conference paper at ICLR 2024
Octavius Architecture
23 Captioing
RSB image
Spene-lovel Point Cloud
Trainabie
Frozen
Q: What the caption this image?
System Prompt You are an Al sosistatt, and required to
Encocer
Encoder
Identify at the objects the mmage At coordinates of
Query
Response
R: The Image depicts construction worker weesinn a hard hist and high viaibiity clothing. stainaing on radlaay frack
Large Language Model
+
LoRA
LoRA
LoRA.
LoRA
Cetection
What objects
Q: Detect all objedt the image.
Langugee toiks
Q1: What thes object?
R: At the posttion of
R1: brown desk egainnal mod and neud to a bockshelf
Rosseting
JolG, 0.62 0.97.
What objects coffee tabie, white bedt
0.40), more ensts a cer There a bicycle at the (0.0, 0.72 0.98, G 944..
02: What this object?
R2: 7here or lamp bellswe two chwirs
More makk
Figure 1: Octavius is unified, multimodal large language model with a novel capability to com- prehend various tasks across different modalities, including but not limited to 2D captioning. 2D detection, 3D VQA, and 3D dense captioning.
promise the performance of each downstream task. This problem is much more in MLLMs, especially when more modalities and tasks are involved, but only a few well-annotated data are available. First, the features from modalities are not easy to be aligned with each other, not to mention compatible with the LLM--ased language decoders. Second, simultaneously learning to acquire knowledge at distinct granularities, such as the instance-level perception (e.g., object de- tection) and logical reasoning (e.g., VQA), may lead to significant interference. Third, it is more complicated by using an LLM-based decoder to generate textual responses that meet the special re- quirements of tasks in modalities other than natural language, such as the bounding box coordinates detection tasks, or action sequences for robotics.
To resolve this issue, we propose LoRA-MoF, which combines the well-known Mixture-of-Fxpper (MoE) (Jacobs al.) 1991 Jordan & Jacobs 1994) and one of the representative PEFT techniques, i.., LoRA (Hu et al. (021). Based on LoR MoBF an I.L.M-bseed decoder can efficiently be in- volved in more downstream tasks and more modalities by learning more LoRA modules. Different from conventional MoE models (Shazeer et al. 2017: Lepikhin al. 2020 Fedus et al. 2022 Du et al. 2022), we adopt a simple yet effective instance- bastacce-baedd gate routing scheme, sparsely activat- ing independent LoRA experts with instance-level instructions and further acquiring task task-specific knowledge for better aligning different tasks. Notably, to the best of our knowledge, we are one of the pioneering efforts to introduce MoE into MILMs address the tug-of-war problem.
To validate the effectiveness of LoRA-MoE, in this work, we investigate a more complicated sce- nario, where the MLLMs should simultancously learn downstreaming tasks from more additional modalities, such as 2D images and 3D point clouds. This scenario is especially useful for embodied agents (Duan al. 2022 Mu al. 2023 Driess et al. 2023). Specifically, in addition the off- the-shelf image encoder, we design point cloud encoder called Object-As-Scene, which provides language-aligee aligned scene-level point cloud representations. This encoder at first gathers language- aligned point cloud features of each instance (Xue et al. (023) in a scene, which are then aggregated into a scene-level feature based on the attention operation guided by the input instructions.
Based on the aforementioned contributions, we introduce a novel and extensive framework called Octavius, which learns the MLLMs upon the instruction-follooinng datasets adapted from LAMM (Yin et al. 2023) and ScanNet (Dai et al.) 2017). As shown in Figure Octavius can successfully address various 2D/3D vision and language tasks, including but not limited to 2D de- tection, 2D captioning, 3D VQA, and 3D dense captioning. We conduct various experiments to validate the effectiveness and versatility of our design. improving multiple downstream tasks by about 20% while increasing only few trainable parameters.
2

---

Published as a conference paper at ICLR 2024
APd
Trainable
Frozen
Image
20 Projector
Gup
System
Modallty Embedding
Question n
Multimodal Instruction
Gatligg G
Language Model with LoRA-MoE
0.8
0.
0.2
3D Projector
Delectoo
Pusion
LoRA-MoE
L0
LoRA
LoRA
LORA
Learnabl
g
d1.
Point Cloud Queries
Caption
Detection
Classification
Multimodal Encoder
Tasksspecific Response
Figure 2: Overall pipeline of Octavius. We design corresponding encoders for different modalities, with the primary objective of empowering the LLMs to gain a deeper understanding of visual fea- tures. Additionally, propose a dynamic gating network that selects distinct LoRA experts based on input instructions, thereby proficiently mitigating interference arising from multimodal learning.
RELATED WORKS
Large Language Models (LLMs)
PEFT. Recently, Large Language Models (LLMs) (Brown et al. 2020; Chowdhery et al. 2022 Chiang al. Touvron al.) 20236 have gained signif- icant attention due to their impressive capabilities in language generation (Zhang et al. 2022a), in-context learning (Wei al. 2022), and reasoning (Touvron et al., 2023a). For both data- and compute-efficient adaptation on certain downstream tasks, several PEFT (Parameter-Efffiiien Fine- Tuning) (Li Liang 2021 Houlsby al. 2019:] Karimi Mahabadi et al. 2021| Hu al. [021] are proposed. For instance, LoRA (Hu et al. 2021) represents weight updates using two smaller matri- ces through low-rank decomposition, where original weights are kept frozen while the new update matrices are trained. In this work, we adopt LoRA for efficient MLLMs fine-tuning.
Multimodal Large Language Models (MLLMs), Several recent studies have attempted to extend the capability of LLMs to multimodal tasks. Alayrac et al. (2022); Li et al.) (2023b); Liu et al. (2023 : /Zhang et al. (0023 Yin al. (2023); Chen et al. ) Peng a (2023) introduce image modality in L.MSs for comprehending 2D visual content. Hong et al. (2023) combines LLMs with 3D modality by rendering point clouds into 2D images and utilizing them to represent 3D visual features. Driess et al. ( Mu et al. y; BBrohan et al. (2023) establish connections between visual inputs and embodied controls for robotic tasks. Despite its wide range of multimodal appli- cations, the performance degradation caused by interference between tasks and modalities during fine-tuning in MILMS receives inadequate attention.
Mixture-of-Expee Ex-exts (MoE). Deep MoE. models are proposed to increase the number of model pa- rameters without adding computational overhead in the field of computer vision (Riquelme et al. 2021 Mustafa et al. 2022 Shen et al.) 2023) and natural language processing (Shazeer al.] 2017 Lepikhin et al.) 2020 Fedus et al. 2022). Different from these approaches, we aim to address con- flicts between tasks with MoE. Adamix (Wang al. 2022a), an approach related but distinct from ours, randomly selects experts during training and uses the average weights of experts in inference, which may be analogous to dropout (Srivastava et al. 2014) in certain cases. In this paper, de- sire for dynamic gate routing strategy to automatically calculate the weights of each LoRA expert according to the input instructions, adapting MI.LMs for broader multimodal applications.
3 METHODOLOGY
As illustrated in Figure we propose an extensible framework called Octavius for multimodal instruction tuning. In Section..1 we first elaborate on the tug-of- problem LoRA-MoE decoder to break through the bottleneck caused by interference between different tasks and modalities. We then verify our design using both image and point cloud modality in this work and describe corresponding encoders in Section 3.2
probblem and propose a unified
3

---

Published as a conference paper at ICLR 2024
w/ MMM, T1.2x
System Prompt:
50
w/ MoE, 15.1x
47.0
40.9
You Al assistant, and required
40
39.0
38.7
30
24.5
Human: z
20
<img> imn </img>
In
10
6.7-7.6
Question:
0
0
Please describe the Object in this picture.
VOC/Rec@0.5
SQA/Aco@1
Tuned on MS-COCO only
Tuned "LAMM-v1"
Response:
The image describes a construction worker wearing a hard hat and high visibility clothing, standing railroad track
Tuned "LAMM-v2"
Tuned on "LAMM-v2" with MoE
We conduct simple pilot study PASCAL VOC and ScienceQA to demonstrate the tug-of-war problem and the effectiveness of our proposed LoRA-MoE. Recall@0.5 denotes recall an loU threshold of 0.5, respectively.
Figure 3:
Figure 4: We follow previous works, e.g., LAMM (Yin et al.] 2023), LLaVA (Liu et al., 2023). apply an inslruction-followingg train- ing pipeline.
3.1 MULTIMODAL DECODER
3.1.1 THE TUG-OF-WAR PROBLEM
Interference among different modalities and tasks is common and critical issue (Zhao al. 2018 Vandenhende et al., 202T) multimodal and multitask learning. While MILMS can alleviate this problem by adopting the same learning objective, i.e., next-token prediction loss, for all tasks, there still exists task-specific divergences that limit their potential in various downstream tasks. Since previous works have yet delve into the tug-of-war phenomenon in MLLMs, we conduct simple pilot study on image modality to reveal this problem.
The pipeline image encoders in MLLMs are simple and similar in previous works. Here, we select LAMM YYn et al. 2023) as our model due to its rich benchmarks on downstream tasks. We fine- tune LoRA and projector LAMM following (Yin et al. 2023) and validate zero-shot performance on PASCAL VOC (Everingham et al. 2010) and Science (Lu et al. (2022) datasets. We report the recall and precision at an Intersection over Union (IoU) threshold of 0.5 on PASCAL VOC and the accuracy of multiple-choice questions ScienceQA.
The results are shown in Figure 3 Although the original dataset of LAMM, referred to as "LAMM vI", contains numerous images from MS-COCO (Lin et al.] 22144), the lack of sufficient detection instructions results in poor performance on PASCAL VOC. To overcome this problem, we lever- age the entire COCO detection annotations and GPT-API (OpenAl 2023) to generate additional detection instructions as supplementation, constructing a new dataset called "LAMM v2" for better generalization of detection tasks. After verifying the detection ability of LAMM by using COCO detection instructions alone, we find using a mixed dataset does not lead to a huge improvement in detection performance. Also, there is decline in the VQA tasks. Moreover, same results on another dataset used in LLaVA (Liu et al. 2023). It can be concluded that MLLMs suffer from severe tug-of-war problem on image modality, not to mention incorporating more modalities for training simultaneously.
can achieve the
3.1.2 LORA-MOE
Some prior works (Kendall et al. 2018 Chen et al. 2018) have attempted to balance the magnitudes of losses or gradients across different tasks to address the ofg-of-war issue. However, considering that different objectives are defined for each task in multitask learning, it is challenging directly
extend existing methods to MLLMs that adopt a unified optimization objective for tasks. In this section, introduce the concept of Mixture-of-Expert (Jacobs et al. 1991 Jordan & Jacobs
4

---

Published as a conference paper at ICLR 2024
1994 Shazeer et al. 2017), proposing unified LoRA-MoE decoder based instance- based gate routing strategy.
Revisiting MoE Models. A typical MoE Model injects multiple MoE layers into LLM to accom- modate greater number of parameters. The MoE layer consists of a group N expert networks E1, E2, ...: En and gating network G, taking the previous tokens as input and producing the prob- ability of the next token:
N
tok; = (ttok...--))eeeatttk....
where tok; denotes i-th token. We refer to this kind of gating network, which based on token-level input, token-based gate. Furthermore, prevent G from consistently producing imbalanced weights that favor only few experts, an auxiliary loss Lialance (Shazeer et al. 2017 Zhou et al. 2022 Wu al., 2022) is introduced to balance gating routing. For example, [Shazeer al. (2017) minimize the coefficient of variation of the gate values for each token, encouraging all experts to have equal importance:
Lbalance = a CV Σ [((toko..-))))
where is a hyyer--praamed and CV(-) the coefficient of variation, which is the ratio of the standard deviation to the mean.
LoRA-MoE and Instance-based Gate Routing. Different from token-based gate in LLM, design simple but effective routing strategy for MLLMs, assigning downstream tasks to indepen- dent experts for specific knowledge based on individual instances, called instance-based gate. is motivated that the input questions 2 applied in multimodal instructions will substantially affect the responses generated by MI.LMs, we take the questions as input to predict routing scores for each expert. Then, we select sparsely-activatd experts based on routing scores for each individual instance to generate the entire sentence. In this work, the LoRA module is treated as an expert in MLLMs, combining instance-based gate with it to alleviate interference arise from multimodal learning, named LoRA-MoE. By replacing LoRA in each projection layer of language model fLLM with group of independent LoRA experts {EL.ORA)
} Y,, we can predict the i-th token value as follow:
N Σ G(2 )x ELORA (toko..a-i).
tok, = (toko.......
(3)
Additionally, we find that Ltalance is incompatible and infeasible with LoRA-MoE in an instance- based gate scenario. For example, it is more reasonable to assign detection samples LoRA expert proficient in localizing than the other experts for the purpose of balancing. Therefore, we can observe some imbalance phenomenon in experiments (see Section 4.3 for details), unless the amount of data for each task in the whole dataset is balanced.
Compared with previous MoE models, LoRA-MoE allows for efficient fine-tuning on small datasets and faster convergence with instance-based gate routing. During the inference phase, the down- stream tasks and input questions are specified, LoRA-MoE can also merge parameter weights with language model like vanilla LoRA to reduce storage requirements and inference costs.
Instruction Tuning with LoRA-MoE. Given the target modal features zimg or zPel, we construct image-text conversation pairs in an instructiio-follloingg format based on previous works (Zhang al. 2023 Yin et al., 2023 Peng et al. 2023). as shown in Figure 4 The language model with LoRA-MoE is then trained to predict corresponding responses based on the system prompts, target modal features and questions.
3.2
MODALITY ENCODER
3.2.1
IMAGE ENCODER
Benefit from the pioneer vision-language model (Radford et al. [02I that bridges the gap between the image and language modality, [i al. 2023a) Liu al. (2023); ( Chen al. (2023): Zhang al.)
5

---

Published as a conference paper at ICLR 2024
Trainable
Frozen
Linear fore
MutliHead SelfAttn
Point in
Point-Bert
MutliHead CrossArtn
Detector
Linear frs
Leamable Queries Eq
(F'snst
Step
Step
Step 3
Figure 5: Structure of Object-As-Scene. To acquire scene-level features, we follow a three-step process. Firstly, obtain Rols from a given point cloud using pre-trained detector. Next, pre-train a Point-Bert model following a ULIP-like pipeline and employ it to extract instance-level 3D features. Finally, by aggregating features from visual embedding, derive the final scene-level feature.
2023); ; Yin et al. 2023) achieve impressive results. We follow their pipeline to extract features for use the pre-trained CLIP visual encoder ViT-L/14 FCLPP (Rafford et al. 2021) to extract the language-aligned visual feature h img following by trainable linear layer fro)) match the dimension of img with the word embedding space in language model: h img
image modality. Specifically, for image input I E H
CLIP (I);
z 2 forms (h img)
where z iing is the output features of image modality for further instruction tuning.
3.2.2 POINT CLOUD ENCODER
Conventional 3D methods typically apply 3D CNNs Yan et al. 2018 Shi et al. 2020) Qi et al. 2017) Transformers (Zhao et al. 2021: Fan et al. 2022) as feature extractors process sparse point cloud data. However, they still retain numerous background points with low-density informa- tion, which may confuse the subsequent language models in MI.LMs, ignoring the pivotal elements in the scene. Besides, the unavailability of encoders capable of aligning scene-level 3D features with language may pose significant challenges for LLMs in comprehending the semantic information of the entire scene. To address these issues, we propose Object-As-Seen As-Scene as our point cloud encoder dedicated to iangugee-aligned scene-level 3D representation generation, illustrated in Figure
Step 1: Locating Regional Rols as Candidates. An intuitive way to avoid excessive background points to identify specific regions in the scene that may contain instances relevant semantic information and encode entire scene with these regions. Specifically, given 3D point cloud scene, employ pre-trained object detector, i.e., FCAF3D (Rukhovich al. (022). locate candidate Rols (Region-of-Interee such captioning classification that primarily foeus instances mentioned in the conversations, we directly use regional features associated with these instances as input.
{r) New Note that NROT denotes the number of Rols. Besides, for tasks
Step 2: Extracting Rol Features Aligned with Language and Image. Inspired by recent work (Xue et al., 2023), we pre-train a Point-Bert YYu et al. 2022) encoder pPoit-BBct aligned with both language and image modalities following ULIP-like pre-training pipeline, allowing us ex- tract instance-level 3D visual features from points P E R. x6 in candidate RoIs {r) Nuw during instruction tuning:
[he New = Point-Bent (P, Nraa))
(5)
More details about improved ULIP-like pre-training pipeline can be found in the Appendix.
Step 3: Aggregating Rol Features as Scene. Next, we adopt fusion module with two stacked transformer layers to fuse scene-level features with Rol features. Specifically, we utilize multi-head cross-attention mechanism (Vaswani et al. 2017) (denoted MHCA) to attend group of trainable

---

Published as a conference paper at ICLR 2024
Table Comparisons on image modality. We investigate the zero-shot performance of our method on multimodal (2D) scenarios. We use the following abbreviations in this table and subsequent ex- periments for ease: "Det." for detection tasks, "Cap." for captioning task, and "Cls." for classifica- tion tasks.
Det. (loU=0.5)
VQA
Cap.
Cls.
Facial Attr
Models
MoE
FT. Dataset
Avg.
Recall
Prec
Aocé9 1
CIDEr
Acc@1 1
Hair Acc I
Smile Acc
7.61
5,95
40.31
0.21
73.50
58.04
50.15
LAMM
LAMM
39.04
35.21
46.95
5.66
65.40
60.93
59.82
20.89%
52.35
30.75
2.89
12.50
50.23
LLaVA-LORA
LLaVA
55.58
23.08
41.00
3.93
52.17
18.36%
queries Eg with Rol features (hPel) Naa:
h MHCA(q Eqp kv (h + [Pp(()) Nea)
(6)
fPE is used to transform 3D bbox coordinates into positional embedding to enrich spatial informa- tion. And q, & and u denote query, key, and value in attention. As in image encoder (Section 3.2.1 trainable linear layer is also applied for final 3D features pel.
prod (hDe))
3D Instruction Data. We construct 3D instruction tuning dataset called Scan2Inst using Sean- Net (Dai et al. 2017) as our 3D instruction tuning dataset due to its diverse tasks and annotated categories. Following Wang et al. (2022b), we use GPT-API generate total of 80k data pairs comprising instructions and responses based on original dataset.
4 EXPERIMENT
4.1 EXPERIMENT SETUP
To explore the effectiivnness of our framework in multimodal learning, fine-tune Octavius in three modality setups: i.) image modality, ii.) point cloud modality and iii.) both image and point cloud modalities. We then evaluate the zero-shot performance using these three fine-tuned models on various downstream tasks. More details about architecture and training scheme are provided in Appendix.
Instruction Datasets. For image modality, we follow Yin et al. (2033) to construct "LAMM
v2", instruction dataset consisting of MS-COCO (Lin et al. 2014) and Bamboo (Zhang et al. 2022b), which includes object detection, classification, captioning, and other common 2D tasks. For point cloud modality, we utilize ScanNet (Dai et al. (017) to generate an instruction dataset called "Scan2Inst" which contains VQA, captioning, and classification tasks. In the multimodal learning (2D&3D) setup, we merge the image and point cloud instruction dataset fine-tune the entire framework simultaneously.
Quantitative Zero-shot Evaluation. We perform zero-shot evaluation on various downstream tasks for both image and point cloud modalities. For image modality, we perform Visual Question An- swering (VQA) on ScienceQA (Lu al. 2022). classification on CIFAR-I0 (Krizhevsky et al. 2009). captioning on Flickr30K (Young et al. 2014 and facial attribute recognition on CelebA (Liu al. 2018). Note that we evaluate performance
VQA tasks through multiple-chote selection. For point cloud modality, we perform classification on ShapeNet (Chang et al. (015)) (55 classes) and captioning on NR3D (Achlioptas et al.] 2020). We also evaluate the performance of classi- fication, captioning, and QA the test split of ScanNet verify the proposed Object-As-Scene encoder.
4.2 QUANTITATIVE RESULTS.
All results are provided in Table I 3 for different modality setups. Severe interference can be found in the experimental results, especially between the localizing tasks and semantic understand- ing tasks like VQA and captioning. After equipping with LoRA-MoF, we can observe remarkable
7

---

Published as a conference paper at ICLR 2024
Table 2: Comparisons on point cloud modality. We report both Fine-Tuning (FT.) Results and Zero-shot (ZS.) Results on the 3D tasks. We also compare our model with 3D-LLM (Hong et al., 2023). Here, 1 indicates the results of Scan2Cap are evaluated on custom test set regenerated by 3D-LLM, which is different from ours.
FT. Results
ZS. Results
Models
Cap. (Scan2Cap)
VQA (ScanQA)
Cls. (ScanNet)
Cls. (ShapeNet)
Cap. (Nrád)
ZS. Avg.
BL.UU-1
BLEU-I
CIDEr
Acc@1
Acc@1
BLEU-I
CIDEr
3D-LLM (Flamingo)
36.10f
30.30
59.20
Ours
33,58
35.11
43.21
168.21
47
19.75
20.02
16.19
Ours w/ MoE
35.94
39.38
44.24
167.31
48.80
24.85
21.16
17.22
17.06%
Comparison on image & point cloud modalities. While there are some performance gaps compared to the model fine-tuned single modality, our model with the inclusion of MoE exhibits a superior performance (~20%% compared to its counterpart.
Table 3:
2D Results (ZS.)
3D Results (FT)
Results (ZS.)
FT. Dataset
MoE
Det.
VQA
Cap.
Cls.
Facial
Cap.
VQA
Cls.
Cls.
Cap.
Avg.
Rcc@0.5
CIDEr
Smile
CIDE
CIDEr
CIDEr
LAMM v2
7.61
40.31
13.28
73.50
58.04
50.15
39.04
46.95
26.71
65.40
60.93
59.82
Scam2Inst
39.56
162.14
47.60)
19.75
16.19
39.38
167.31
43.40
24.85
17.22
LAMM v2+Scan2Inst
2.64
39.71
0.04
71,66
42.47
50.66
19.76
182.00
38.80
14.85
8.26
34.30
38.80
10.06
56.86
51,52
54.22
33.29
181.44
47.20
21.10
17.22
21.40%
improvement of approximately 20% in setups, demonstrating the effectiveness of our design in resolving the tug-of-wa war issue. Additionally, we compare our proposed point cloud encoder Object- As-Scene with a recent work, 3D-LLM (Hong et al. 2023) in Table We achieve a comparable performance on Scan2Cap, and outperform 3D-LLM ScanQA by a significant margin, suggesting Besides, as shown in Table as the complexity of interference among tasks increases, especially when tasks of different modalities are introduced, we can observe huge performance drop when training the model with different modalities like image and point cloud simultaneously compared to separate training. Our proposed MoE- based decoder partially alleviates the degradation and even achieves comparable performance with separate training some tasks.
a better scene-level understanding capability of Object-As-Scent
4.3 ABLATION AND ANALYSIS
The results are shown Table 4 We first ablate on MoE architecture by individually employing dedicated LoRAs for each tasks (denoted as "Individual" in the table). While the indi- vidual gate exhibits performance merits in specific tasks, its primary challenge is the difficulty in assigning suitable experts for tasks that are not encountered in the instruction dataset, thereby com- promising the model's generalizability. Another observation is the superior efficacy of the sparse gate relative to the dense gate (denoted as "Weighted Sum" in the table). which is intuitive when considering that the dense gate can essentially be regarded as a singluar LoRA with additional pa- rameters. Furthermore, we compare the performance of the sparse gate against the baseline model (single LoRA) under conditions of parameter-consistency during inference (ie., sparse top-2 gate only uses half of rank in LoRA compared with baseline model). The enhanced performance of sparse gate demonstrate that MoE transcends mere aggregation of parameters.
LoRA-MoE.
Gate Routing in MoE and Load Balancing, As shown Figure there a huge discrepancy in expert selection between detection and VQA tasks, which demonstrates the tug-of- -ar phenomenon, and explains why using a single LoRA yields poor performance on both tasks simultaneously. Addi- tionally, it is found that the routing weights of experts assigned by gate network tend to concentrate on a subset of specific experts. In particular, in a 4-expert model, despite the superior performance compared to 3-expert model, the final converged model ends up utilizing only 3 of 4 available experts. To further explore this imbalance issue, we conduct several experiments with load bal- ancing loss (Equation 2 in Table 5 As result, improvements and better routing results
8

---

Published as a conference paper at ICLR 2024
Table Ablation studies on MoE architecture on 2D tasks. "Sparse Top-2" gate picks out top-2 ranked experts based on routing scores. "Weighted Sum" gate uses the weighted sum of all experts as output. "Individual" gate employs different experts for each 2D tasks individually. "#Trainable Param." denotes the proportion of trainable parameters to total parameters.
Det. (VOC, IoU=0.5)
VQA
Gate Type
LoRA-Rank
#Trainable Param.
Recall
Prec.
Acc@1
(Baseline)
32
7.61
5,95
40.31
0.4%
Sparse Top-2
32
39.04
35.21
46.95
1.6%
Weighted Sum Individual
32
9.78
5.33
44.71
1.6%
32
28.38
25.64
48.54
2.4%
Sparse Top-2
16
32.81
24.46
39.11
0.8%
Sparse Top-2
25.44
21.87
37.65
0.4%
Gate routing on different tasks
LoRAs
Table 5: Ablation studies on load balancing in MoE. "LB" means apply load balance loss. Token means using token-based gate.
Det.
AD
VQA
Cls.
Det.
VQA
#Exeetts
LB
Token
Cap.
Recall
Prec.
Acc@i
FA Hair
39.04
35.21
46.95
/
33.21
26.80
45.26
Smile
22.30
11.01
39.91
v
21.52
12.10
37.53
Figure 6: Gate routing different 2D tasks. We use different color to represent i-th LoRA. The pro- portion represents the score of each expert.
Fail
v
Fail
observed, which is intuitive because directly employing load balancing strategies is incompatible and infeasible in an instance-based gate scenario (see Section 3.1.2). We also attempt replace the instance-based gate with token-based gate used in conventional MoE methods (Shazeer et al.) 2017 Lepikhin al. 2020). However, since only the LoRA modules trained in approach, the number of trainable parameters differs from conventional MoE models where all parameters of the entire foundation model are trained, resulting poor convergence.
Besides, we also provide more ablations and analysis on point cloud encoder, LoRA-MoE, gate routing and qualitative results in Appendix.
CONCLUSION AND LIMITATIONS
In this paper, propose Octavius, unified multimodal framework, effectively address the critical challenge of task interference in complex learning scenarios. By integrating the Mixture- of of-Experts (MoE) with LoRA, we present LoRA-MoE decoder, which delivers specialized learning paths for different tasks and modalities. After the validation across multiple modalities and tasks, Octavius alleviates the severe tug-of-war issue and achieves significant performance boost both 2D and 3D tasks.
Limitations. Compared to separate training on a single modality, introducing simultaneously mul- tiple modalities for joint training may result in performance degradation, posing challenge for future research. The combination of MLLMs and MoE still has great potential in addressing this problem, especially for more complicated real-world system like embodied AI scenarios that quire more modalities as input. Besides, we will further explore the token- based gate with load balancing strategies, especially when the number of downstream tasks increases.

---

Published as a conference paper at ICLR 2024
Acknowledgement. This work is supported in part by National Key Research and Development Program of China (NO. 2021 (FBI7117))))00) and National Natural Science Foundation of China (62132001).
REFERENCES
Panos Achlioptas, Ahmed Abdelrcheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification in real-world scenes. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Pro- ceedings, Part 16, pp. 422 440. Springer, 2020.
Jean-Ban Saan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision lasioonlanguage-actto models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15811 2023.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012.c 2015.
Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint 230.. 2023.
Zhao Chen, Vijay Badrinarayanan, Chen-Ye Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pp. 794-803. PMLR, 2018
WL Chiang, Z Li, 7. Lin, Y Sheng, Z Wu, H Zhang, I. Zheng. S Zhuang. Y Zhuang, JE Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, mar. 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02133 2022.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießnner Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2017.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint arXiv:2303.08: 03378, 2023.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-exper In International Conference Machine Learning, pp. 5547- 5569. PMLR. 2022.
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230-244, 2022.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. ijev, 88(2):303-338, 2010.
10

---

Published as a conference paper at ICLR 2024
Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang. Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8458-
8468, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1): 2232-5270, 2022.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, pp. 1-15, 2023.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12): 2020.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lIm: Injecting the 3d world into large language models. arXiv preprint urXiv:2307.12988 2023.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffcient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.099 09685, 2021.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, al. Language not you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the algorithm. Neural computation, 6(2):181-214, 1994.
Rabech Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022- 1035, 2021.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7482-7491, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Taku Kudo and John Richardson. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.0622 2018.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 10006616660 16668, 2020.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03776, 2023a.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023b.
11

---

Published as a conference paper at ICLR 2024
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101 00000,0 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr
Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 08485, 2023.
Ziwei Liu,, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba)
dataset. Retrieved August, 15(2018):11, 2018.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems, 35:2507-2521, 2022.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. arXiv preprint arXiv: 15021, 2023.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Mul- timodal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Processing Systems, 15:9564-9576,, 2022.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8763.. PMLR, 2021.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-885 8595, 2021.
Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Feaf3d: Fully convolutional anchor free 3d object detection. European Conference on Computer Vision, pp. 477-493 Springer, 2022.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 701.06533.. 2017.
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226 2023.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang. and Hongsheng Li. Pv-renn: Poxnt-voxel feature set abstraction for 3d object detection. Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. (0529-10533) 2020.
12

---

Published as a conference paper at ICLR 2024
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, 1lya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-195, 2014.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint wwXiv:2307099266 2023b.
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Lue Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7)33614-3633, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Yaqing Wang. Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-addapations for parameter-efficient model tuning. Xrviv preprint arXiv:2210.1744 17451, 2022a.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi. and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.0056/ 2022b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of- oh-tht prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-2488 2022.
Lemeng Wu, Mengehen Liu, Yinpeng Chen, Dongdong Chen, Xiyang Dai, and Lu Yuan. Residual mixture of experts. arXiv preprint MXXx:22244006336. 2022.
Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martin Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1179-1189, 2023.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 2304:22304.41778 2023.
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, et al. Lamm: saoguugge-assister multi modal instruction-tuning dataset, framework, and benchmark. arXiv preprint arXiv:2306. 06687, 2023.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Compulational Linguistics, 2:67-78, 2014.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19313-19322, 2022.
Renrui Zhang. Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient finettuning of language models with zero-init atten- tion. arXiv preprint arXiv:2303. /6/99, 2023.
13

---

Published as a conference paper at ICLR 2024
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 2r05v:200..01068, 2022a.
Yuanhan Zhang, Qinghong Sun, Yichun Zhou, Zexin He, Zhenfei Yin, Kun Wang, Lu Sheng, Yu Qiao, Jing Shao, and Ziwei Liu. Bamboo: Building mega-scale vision dataset continually with muchn- synergy. arXiv preprint arXiv:2203.07844/ 2022b.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 16259-16268, 2021.
Xiangyun Zhao, Haoxiang Li, Xiaohui Shen, Xiaodan Liang, and Ying Wu. A modulation module for multi-task learning with applications in image retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 401-416, 2018.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. of with expert choice routing. Advances in Neural Information Processing Systems, 35:7 35:7103-77144 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2024
A ADDITIONAL IMPLEMENTATION DETAILS
Pre-training a language- and image-aligned Point-Bert following ULIP-like Pipeline. To im- prove the generalization performance of instance-level point cloud encoder, we select ScanNet (Dai et al. 2017) as our dataset due to its diverse object categories instead of the original ULIP dataset. Besides, we devised memory bank in our pre-training framework for fast convergence and better representative capabilities.
For specific, given the instance-level point cloud P. RN within each candidate Rols rss we first retrieve images I, from related regions using its camera intrinsic matrix, and generate a simple prompt template L;, e.g., "a photo of {CLASS)", obtaining multimodal triplet (P,, I, L,)- We then extract corresponding features with a pre-trained CLIP (Gao et al. (023) and trainable Point-Bert encoder
(Yu et al. 2
he Point Ben (P:); h JELIP (CLIP(I);; h
CLIP (L)),
We then contrast h with h img and h of images and language:
lang bringing 3D representation closer to semantic information
Coontrast w1 (pol, + (pelipellang)))
where Cipel.img) , C (pel. are respective contrastive loss and W1, W2 are corresponding loss weights. As mentioned before, the memory bank M is introduced to accommodate more negative samples for better feature alignment. For example, given i-th associated multimodal feature pair (1e. h lang
contrastive loss is given by
pel cxp(h1. h1
/T)
D ((pll.ang)
Σ Llog
(10)
E exp(h1") -1 lane /T)
Eventually, the pre-trained Point-Bert is used for extracting instance-level 3D visual features that align with language and image in Object-As-Scene.
LLM Architecture and Training Scheme. We choose Vicuna- 13B (Chiang et al.) as our LLM. Instructions are tokenized by SentencePiece (Kudo & Richardson 2018). We apply LoRA-MoE on the language model for efficient fine-tuning and task-speeific learning in all three setups. The number of experts in the above three setups is 4, 3, and respectively. The rank of each LoRA expert is set to 32. During fine-tuning, we use an Adam (Kingma & Ba (014)) optimizer with a total batch size of 64, learning rate of 5 10-4, and an epoch of on all setups. All experiments are conducted on 4 NVIDIA A100 80GB GPUs.
all resized 224x224 and split into 256 feature patches using CLIP VIT- L/14 (Radford et al. (222I). For point cloud data, we sample 1024 points from each Rol extracted by FCAF3D (Rukhovich al., 2022) and generate corresponding features by pre-trained Point- Bert Yu et al. 2022). Then we select NRæi instances with bbox confidence larger than threshold = 0.3 for each seene. Next, we use queries in the fusion module to obtain aligned 3D visual features. Furthermore, in the multimodal setup, we pad the output 3D visual features to 256 with masks for aligning with image patches.
Input images
B ADDITIONAL EXPERIMENTS AND ABLATIONS
-fg-oo-War The Tug-of- War Issues in Multimodal Learning. We attempt investigate the tug-of-war war issues within the realm of multimodal learning. The results, delineated in Table reveal that the tug-of-war war issues not only prevail in multimodal learning, but also can more severe. Here, we mainly focuses on 2D and 3D captioning tasks. When introducing more modalities during instruction tuning, a huge performance degradation can be observed, especially in 3D captioning tasks. After applying LoRA- MoE, the performance of 3D captioning tasks is enhanced, aligning with the levels achieved when fine-tuned on the single 3D modality. Meanwhile, the performance of 2D captioning is also greatly improved, underscoring the effectiveness of LoRA-MoE.
Point Cloud Encoder. As shown in Table 7 positional embedding (PE) improves the overall per- formance, since the position and scale of objects in PE can help the model better understand the
15

---

Published as a conference paper at ICLR 2024
Table 6: We conduct another pilot study to reveal the tug-of-war issues in multimodal learning.
Captioning
FT. Dataset
MoF
Avg.
Flickr30k (2D ZS.)
Scan2Cap (3D FT.)
NR3D (3D ZS.)
LAMM v2
0.21
Scan2Inst
35.10
16.19
LAMM v24Scan2Inst
0.04
19.76
8.26
LAMM V2+Scan2Inst
10.06
33.29
17.22
43.91% f
Table 7: Ablation studies on point cloud encoder. "PE" means positional embedding.
Cap. (Scan2Cap)
VQA (ScanQA)
Cls. (ScanNet)
#Queries
PE
Fused Modality
Avg.
CIDEr
CIDEr
Acc
16
Add
Lang.
35,11
168.21
47. .4
83,57
16
Add
Lang. + Image
45.00
160.33
61.60
88.97
64
Add
Lang- Lang.
41.45
161.69
48.80
83,98
256
Add
19.36
164.55
48.40
77.44
16
X
Lang.
29.39
168.98
42,60
80.32
16
Concat
Lang.
26.65
174.86
47.40
82,97
Gate routing on different fasks
LoRAs
Table 8: Additional ablation studies MoE architecture. "Ques." and "Sys." refer to using question or system prompt in the instruction gate input, respectively.
2D Det.
2D VQA
Cls.
A1
2D Cap.
FA Hair
Gate Imput
Det. (VOC, IoU=0.5)
VQA
Gate Type
Ques.
Sys.
Recall
Acc@1
FA Smile
(Baseline)
7.61
5:95
40.31
3D VQA.
Sparse Top-2
39.04
35.21
46.95
A4
3D Cap.
Sparse Top-1
22.42
21.23
36.88
Sparse T
38.57
36.02
43,89
Cls.
Sparse Top-2
34.23
Figure 7: Gate routing 2D and 3D tasks.
semantic information of the scene and instances. We select "Add" in model due to its more balanced downstream results. We also ablate different numbers of learnable queries. Considering that extract about 50 Rols in the scene, if we far more queries than this number, the overall performance will decrease. 16~66 is a reasonable range for the number of queries. Furthermore, attempt enhance the semantic information by introducing image features corresponding 3D Rols using the cross-attention mechanism. Despite the significant improvement in captioning and classification tasks, the additional cost of rendering images based on point cloud limits its practical usage, making it only supplementary method.
Additional Ablations LoRA-MoE. The results shown in Table g By incorporating the "System Prompt" as an adjunctive input alongside the question, performance decline can be found possibly due to the redundant global information embedded within the system prompt (e.g.. "You are an AI assistant, which introduces ambiguity and hampers the efficacy of the gate prediction network.
Furthermore, we also explore the impact of using gates of different top-k selection. Specifically, the top-1 gate exhibits poor performance due to its limited flexibility in selection, offering only four combinations of different experts, in contrast to the more versatile top-2 and top-3 gate. In scenarios that employing the top-3 gate, we find that the contribution of the third choice is relatively small across most tasks. For instance, in classification tasks, the distribution of routing weights often resembles 1, 0, 0.4, 0.5]", which is closely similar to the scores in top-2 gate. Consequently. the top-3 gate yields comparable performance achieved by the top-2 gate.
16

---

Published as a conference paper at ICLR 2024
Table 9: Ablation studies on OOD generaliza- tion of query in VQA evaluation. "Ctx." " and "Ques." denotes context and question, respec- tively. We report top accuarcy on ScienceQA Lu al. 2022) test dataset.
LoRA Expert
Gate routing scores
wo/ Enrich
Enriched Ques.
Query Pattern
VQA
Enriched Ctx.
Enriched Ctx.
Enriched Ques.
Enriched Q+C
46,95
47.58
47.43
Figure 8: Gate routing on different query pattern. "Enriched Q+C" means using both enriched context and question as input.
47.03
More Analysis and Discussion on Gate Routing. As depicted in Figure 6 tasks such as classifi- cation, captioning and facial attribute recognition exhibit tendency engage similar experts with different routing scores. can be attributed to underlying similarities among these tasks in training instructions and reponses, despite their varied downstream applications. For instance:
Captioning
Q: What the caption of the image?
A: The image depicts an older man and a young man in a close embrace, The older man, who bald and wearing a red shirt, appears be hugging the younger man. They are standing in front of black wall, which enhances the contrast of their positions.
Classification
Q: What is the most accurate category label for this image? The image depicts a cute and adorable small white fluffy dog.
Facial Attribute
Q: What color is the person's 's hair in the image?
A: The image shows a young woman with dark, long, and curly hair.
These examples indicate that in tasks like classification and facial attribute recognition, LLMs tend to offer comprehensive descriptions of the target object (including attributes like color, shape, and descriptive adjectives) rather than mere categorical labels, which are very similar in captioning tasks. Therefore it is reasonable that the gating network makes similar expert selections for these tasks. Conversely, in very different tasks like detection, gating networks generate distinct expert selections.
Moreover, we jointly fine-tune Octavius on both LAMM v2 and Scan2Inst datasets, supplementing our analysis with an additional illustration of gate routing on 2D and 3D tasks, as presented in Fig- ure Similar to the observation in Figure 6 load balancing issues still occur within the distribution of gating scores, notably with experts #2 and For the 3D modality, 3D captioning and 3D) classi- fication tasks mainly focus on instance-level perception, such as the caption category of a specific object, while 3D VQA focuses more on inter-relations among multiple objects in the scene and the understanding of the entire scene. This divergence leads to two different pattern of routing weights between 3D captioning//lassifcattiin observation the emergence of knowledge sharing across different modalities certain experts (e.g., expert #2), while others perfer for specilized modality (e.g., experts #1 and #5).
and 3D VQA in Figure 7 Additionally, another interesting
Generalizability of Instance-based Gate Routing, To assess the generalizability of the pro- posed instance-basee gate, we conduct several comprehensive ablation studies on the ScienceQA dataset [u al. (2022). The queries in SeicenceQA datasets comprise of distinct problem state-
17

---

Published as a conference paper at ICLR 2024
toothpaste
soccer shorts
Queetion: (Nhtch peopary thesee frre cijeecll forew in common??
Choice: bllw (8) fand (C) slick
Arsawer: Allar souminig piduue dossely a concluded that blue the arnpone Prix quealan.
Enriched Queetion What is the common propeety smonng these three objects?
lolloing engatienna primary in thke faod oNe?
Context Select beet answen
hand (1)) siick
queediao..
Inke esneystem Wassonain food web models matter catan by organisin moves thoough eccsystem. The anows i food seeb represent how matter mover betowen organieme in on ccosystom.
Queetion WWathh property these frrm objeett lorw commor??
Enriched Contact: /lessos choose the moad is Endll seoweer. Cholce:: (A) blue (B) hard (0) sticky
Arswer: clearving the image chonely, becomees diese that t'e umrooer to this quandion blas.
Choloe: (1) copopod black diopple (C) banteria that oopeppdd the erseect to this question
property wmonng
objerte?
Enriched Contect Pleaos choose the moat
Choice: (A) ti (f) hand (E) sticky
Answer irepecting the pkdiire doowly one can cancklow thei the sensseer to thés quaedion is biow
Enriched Question:
orpanisms fommismm listed this food web?
food from Unto Rock Lake, frestwator lake ecceysiem Wilsooniin A food web modele how the materr eaton by organtems moves through an ecceystem The anows a food web represent how mateer moves between onganis me on ecceyster
Context
Ooea
Choice (A) smpappod (8) sleck crappiw (G) borderis
to concluded that oopepod the correct answer to this question.
Answer anslyctng
Quesstonn Which The lolowing organiama primary coroumer in food wab?
Beriched Contect Heme is food one oevwn from Litla Rock sake, which a freehwweer lake accsystem located in Whecorein food web serves as represoncation of Tow the mator consumce organ lurre flow welhon an escagsdeen: The smoce in finsd web depicl movemmnnt of mater between organisms the
Question: Whitch of these statess futthest wwaf?
NMA
Choice: (A) copopad black orappie bactoria
Cholce: Missouri Foeids (C) Montana (D) Oeegon
Answer: Upon corcful absorvation of the skouu,, becomes
Answor. After Barrug examinaton of the pkoure, becomes apparent that the correct answer (D) Oregon.
evident that coperped the correc seseoar this queedian
Enniched Queeation Which stoie among thew is the hutteed the wexd?
Contest N.A
Enriched Qusstion: Whs is the primary corearmer arring the expertenm bated lhee lood wab?
(1) Oragann
pkcture
acquenent cornect Dregan.
Bnriched di Contect Here a food seb beben from Litle Rock skkn which a freerwaterr lake woneystem located Whecorein food serves as represontation The matirr consumcd within matter
Question WWich of those states is Tarthest west?
Enniched Contact NAA
Choice: (A) Missoun (B) Flarida (C) Mentana (D) Oragan
organis the
Answer correct
queetion
by conly eoamining image, Ohegann
ceooystem.
Enriched Question: Wrich state among Phese the furtest the west?
capaped crappie
Enriched Contivt: N/A
Answer answer question can be dotemiined by ylssey
Choice: (A) Mitsmin (1ly Flariw (C) Mientrrs (9) Oregin
observing ls copopod.
Answer: /ftor esamiiing picture disslly can condluded that Orogon the answor cuestion.
Figure 9: The response of Octavius given different query pattern in downstream VQA evaluation.
ments as well as contextual information. We employ GPT-3.5-turbe Brown et al. (2020) to enrich both the questions and the contexts separately, and ensure the enriched contents maintain consistency with the original semantics. Subsequently, we validate the proposed instance- based gate using these enriched questions and contexts. As detailed Table Octavius achieve stable performance across all enriched data, highlighting the strong generalization capacity of cestance-based gate in process- ing input queries of different patterns. We also present several examples in Figure P Furthermore, provide comparative analysis of the routing weights between the default and enriched queries in Figure [ Remarkably, the model consistently selects similar gates with comparable weights, regardless of the modifications in the data. This consistency demonstrates the robustness of Oc- tavius in effectively managing VQA tasks, proficiently navigating questions and contexts of diverse structures and complexities.
Complete Results on Downstream Tasks. We provide complete experimental results for detec- tion. captioning, and VQA tasks in all setups, as shown in Table I0 and 12 We report recall and precision at IoU thresholds of 0,5 and 0.25 in detection tasks, and BLEO-T/2/3/44 "CIDEr", "METEOR" and "ROUGE-L"" in both captioning and VQA tasks.
18

---

Published as a conference paper at ICLR 2024
Table 10: Complete results on 2D downstream tasks. In the detection task, we also provide recall and precision of the predicted bounding box without categories.
Detection (PASCAL VOC)
MoB
w/ cls (loU=0.5)
wo/ cls (loU=0.5)
w/ cls (loU=0.25)
wo/ cls (IoU=0.5)
Recall
Prec.
Recall
Prec.
Recall
Prec.
Recall
Prec.
7.61
5.95
10.1
7.91
20.96 51.38
16.41 46.12
27.14 59.19
21.24
39.04
35.21
44.16
39.63
53.13
Captioning (Plickr30K)
MoB
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
METEOR
ROUGE-L
13.283
7.328
3.733
1.883
0.21
12.482
17.707
/
26.705
15.163
8.296
4.566
5.66
16.979
26.849
Table 11: Complete results on 3D downstream tasks. Here, indicates the results of Scan2Cap is evaluated on a custom test set regenerated by 3D-LLM, which is different from ours.
Captioning (Scan2Cap)
Models
MoE
BLEU--
BLBU-2
BLEU-3
BLEU-4
CIDEr
METEOR
ROUGE--L
3D-LLM' (Flamingo)
36.10
24.50
18.70
15.60
17.60
35.80
Ours
34.16
20.92
12.45
7.56
39.56
13.03
32,66
Ours
35.93
21.66
12.79
7.75
39.38
13.34
32.36
VQA (ScanQA)
Models
MoE
BLEU-]
BLBU-2
BLEU-3
BLEU-4
CIDEr
METBOR
ROUGE-L
3D-LLM (Flamingo)
30.30
17.80
16.00
7.20
59.20
12.20
32.30
Ours
43.07
32.69
25.17
19.26
162.14
21.44
45.08
Ours
44.24
33.16
25.24
19.16
167.31
21.44
44.87
Captioning (Nr3d)
Models
MoE
BLEU--
BLBU-2
BL.U--3
BL.U--
CIDH
METHOR
ROUGHE-
Ours
20.02
8.95
3.63
1.66
16.19
9.71
20.45
Ours
V
21.16
10.00
4.38
2.07
17.22
11.06
22.37
Table 12: Complete results on multimodal learning (2D & 3D).
Detection (PASCAL VOC)
MoR
cls (lolI=0.5)
wn' clx (InUI=0.5)
w/ (laU=00225)
wel als (InU=0.55
Recall
Prec.
Recall
Prec.
Recall
Prec.
Recall
Prec.
161
3.62
8.15
4,95
34,3
25.07
38.97
28.48
47.11
34.43
$4.72
39.99
Captioning (Flick:30K)
MoB
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
METEOR
ROUGE-L
14,335
8.132
4.288
2.274
0.038
13,673
17.083
Captioning (Scan2Cap)
MoB
BLEU-1
BLEU-2
BLEU-3
BLEU4
CIDE
METEOR
ROUGE-L
26.16
14.79
7.91
4.36
13.76
29.26
19.76
36.62
21.91
12.56
7.29
13.30
31,69
33,29
VQA (ScanQA)
MoBi
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
METEOR
ROUGE-L
45.63
35.11
27.30
21.01
22.73
46.56
182.00
44.48
34.20
26.76
21.04
22.23
46.22
181.44
Captioning (Nr3d)
BLEU-I
BLBU-2
BLEU-3
BLEU-4
CIDEr
METBOR
ROUGE-L
13.60
6.34
2.72
1,20
10.72
20.77
8.26
9.95
4.27
11.13
22.29
19

---

Published as a conference paper at ICLR 2024
C ADDITIONAL VISUALIZATION
In this section, we provide several responses of Octavius in Figure TO 11 and
Question: caption image
Question: caption the image Answer: The image shows man wearing yellow safety vest. sitting with extanded.
Answer:
The image shows man and woman holding hands in a
caption image Answer: The image depicts scene where a group of people are gathered with clock face.
caption image Anewer: The image depicts bustling street scene with
vendors.
Pair 1
S
N
N
J- 16 mm
Pair 2
N
N
30
Think about magnetic between the magnets in each pair. Which of the following siatements is true?
Question:
Context: The images below show two pairs of magnets. The magnets different pairs
magnets shown are made of the same material.
colony shown?
(B) magnetic force is stronger in Pair 1. (C) The strangth of the magnetic force is same both peirs.
Choice:
magnetic
stronger
Context: N/A
Choice: New Hampshire Vermant New York (D)
Delaware
Answer: By observing the image closely. bacomes clear the answer to this quesiion B. The magnetic force is stronger
Answer: Analysis of the image loads to the conclusion that New
York is correct response to this question.
Figure 10: The response of Octavius on 2D captioning and VQA.
20

---

Published as a conference paper at ICLR 2024
Question: Identify all objects in the image and provide their positions.
Answer: Positioned at [0.0, 0.24, 0.68, 0.86] within the image is an object that can be classified as belonging to the cat category, characterized by its unique texture and distinctive silhouette that differentiates from other elements the scene. This cat appears to the focal point the image, drawing the viewer's attention with its relaxed posture. At the [0.0, 0.21, 1.0, 0.99) position of the image, there is an object categorized as bed. The cat seems to be lying on the bed, adding sense of tranquility and homeliness to the image.
Identify all objects in the image and provide their positions.
Question:
Answer: Positioned at [0. 0.89, 0.7, 0.98) within the image is an object that can classified
belonging to the keyboard category, and this object also the only one in the image that has a specific
type of texture and a distinctive shape that sets it apart from the other objects. At the coordinates of [0.41, 0.62, 0.52, 0.95] position of the image, there exists object categorized bottle, characterized by unique shape. Upon close inspection of the image, it can be observed that there is an object positioned at [0.14, 0.15, 1.0, 0.77] that belongs to the tv category.
007/08/0Z 16:10"
Question: Identify all objects in the image and provide their positions.
The object present at the [0.75, 0.42, 0.88, 0.73] coordinate the image is classified
Answer:
motorcycle. Positioned at [0.72. 0.3, 0.84, 0.67]
within the image an object that can be classified as belonging to the person category, and this object is also the only one in the image that has a specific type of texture and a distinctive shape that sets it apart from the other objects. At the coordinates of [0.07, 0.29, 0.52, 0.67] position of the image, there exists object categorized
Question:
Identify all objects in the image and provide their positions.
Answer: The [0.62, 0.39, 0.73, 0.7] position of the image contains an object that can be classified as person. the [0.23, 0.33, 0.37, 0.81] position the image, there is an object that can be categorized as person. The [0.37, 0.26, 0.51, 0.71] position of the image contains an object that can be identified as belonging to the person category. At the exact coordinates of [0.34, 0.72, 0.59, 0.87] in the image, there an object that can be identified as belonging to the dog category.
Figure 11: The response of Octavius on 2D detection.
21

---

Published as a conference paper at ICLR 2024
What is the caption of the point cloud?
Question:
What is the shape of the object that is in contact with wall surface? Answer: The object in the scene is rectangular shape.
Question:
Answer:
This is trash can. it sets in the comer.
Question: Which side the monitor the shelf positioned
Question: What located to the left the display case?
Answer: The shelf is on the rightsside of the monitor.
Answer: brown chair can be found in that position.
Question: What is the color of the door located in the comne??
Question: What is the caption of the point cloud?
Answer: The door situated in the corner is brown color.
Answer: This is black coffee table. is front of the couch.
What is the caption of the point cloud?
Question:
What is the number of white chairs that encircle the brown end table? Answer: There are four white chairs arranged in circle around the brown table.
Question:
Answer:
The door is northwest of the stove. The door is white and rectangular.
Figure 12: The response of Octavius on 3D captioning and VQA.
22


Reviews:
[{"review_id": "XxemJdLl39", "reviewer": "Reviewer_vGo5", "summary": "This paper proposes to use a combination of MoE with the LoRA technique to address the challenge of incorporating additional tasks in the MLLM. The MoE components are chosen using a sparsely gating network. The outcomes demonstrate the efficacy of this approach.", "questions": "How can you attain top-2 sparsity gates while ensuring compatibility with gradient flow?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This method is straightforward but powerful, capable of integrating numerous vision tasks within this framework. Its adaptability is showcased as it seamlessly operates with both 2D images and 3D point clouds. Ultimately, the integration of LoRA with MoE for PEFT proves to be highly efficient.", "weaknesses": "Given our awareness of each example's task, an important baseline involves employing a dedicated LoRA for each task individually. Additionally, conducting an ablation study on the impact of top-k would be informative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CfVkOUtLd7", "reviewer": "Reviewer_WqWj", "summary": "The paper mainly introduced 1) LoRA-MoE (Mixture-of-Experts) decoder to mitigate tug-of-war (interference) problem between different tasks/modalities, and 2) point cloud encoder called Object-As-Scene to extract language-aligned scene-level 3D features. \nExperiments on both 2D and 3D tasks show LoRA-MoE improves performance by ~20% over strong baselines like LAMM and LLaVA-LoRA. The model is also verified on multimodal learning with both images and point clouds.", "questions": "- Can you show some qualitative study of the predictions from the sample-based gating network as responses to the input task? It can help us understand how network routing works.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The idea of MoE with sample routing to mitigate task interference for MLLMs is novel.\n- The author conducted thorough experiments to validate the framework on a diverse set of 2D and 3D tasks. The gains are substantial.\n- The framework is modular and extensible to incorporate more modalities and tasks.", "weaknesses": "- There is no analysis on how the routing among experts actually works. It would be great if the authors can provide some qualitative study of the predictions from sample-based gating network as responses to the input task, to show how the routing mechanism work. I wonder whether the gating network will simply act like a task classifier, or it's not the case.\n- The scaling behavior as more modalities and tasks are added is not studied. There may be limitations in very high multi-task settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gMKs5GShFT", "reviewer": "Reviewer_6dGg", "summary": "This paper contributes a method for mitigating task interference in instruction tuning by learning LoRA-based Mixture-of-Experts. By using a sparse gate routing scheme, different LoRA experts can learn task- and modality-specific knowledge. In experiments, they do instruction tuning for MLLMs on both 2D image tasks and 3D point cloud tasks, each having individual vision encoders.", "questions": "- What is the actual LLM decoder that you use? it's never mentioned throughout the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The paper is well-motivated -- MoEs have been shown to be useful for distributing the different types of knowledge that are required for multi-task learning, and VL instruction tuning is a good application of this insight.\n\n- The experiments are performed on both 2D image and 3D point cloud tasks, both individually and with the two datasets combined.", "weaknesses": "- I am primarily concerned by the analysis in Figure 5 -- it seems that all the 2D tasks are using only two experts! This makes me skeptical about the utility of MoE at all.  Could you run the ablation in Table 5 with 2 experts? so these two experts will get selected each time and there in routing involved, but the model has the capacity to learn with 2 LoRAs at once instead of one (which is what it seems to be doing in Fig5)\n\n- An additional analysis that is needed is how the gate routing is distributed between 2D and 3D tasks, for the model that is trained on the combined LAMMv2+ScanNet instruction tuning dataset.\n\nGiven the result in Figure 5, I am not convinced about the utility of MoE with routing, when it doesn't seem that different experts are even used.\n\n- I am not sure what the non-MoE baseline in Tables 2-3 is -- is it merely training the frozen model minus the LoRA parameters?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rxV7rmhbzf", "reviewer": "Reviewer_3GST", "summary": "Observing the tug-of-war problem between 2D detection and VQA, this paper proposes a Mixture-of-Expert (MoE) style architecture with LoRA for efficient optimization of multi-modality and multi-task. To further support 3D instruction-following tasks, the authors propose a novel point cloud encoding architecture called Object-as-scene. The simple yet effective architecture demonstrate strong experimental performance.", "questions": "- Implementation details:\n  - In Fig. 5, do you select a single layer to do the visualization or take an average of all layers?\n  - In Tab. 6, the authors claim that loading balancing loss has negative effect. Do you use loading balancing afterwards? If not, how would we prevent the MoE architecture from collapsing to always using a single expert?\n  - The 2D results in Tab. 4 have a significant performance gap with state-of-the-art VLLM like InstructBLIP (e.g., ScienceQA and Flickr30K captioning).\n  - The authors should have utilized stronger baseline methods. There is a significant performance gap between LAMN and state-of-the-art methods including InstructBLIP and Shikra.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper constructs Octavius by collecting better detection annotation, combing multiple modality data and propose a novel MoE architecture.\n- The authors provide a throughout discussion between related works.\n- The proposed method is relatively simple but with strong performance.", "weaknesses": "- About the tug-of-war problem:\n  - The authors demonstrate the existence of this problem simply by optimization 2D detection and VQA simultaneously.\n  - Considering the authors also include multi-modality, it would be better to include preliminary also in this direction (e.g., 2D & 3D captioning).\n- About the router input:\n  - As shown in Fig. 2, the input consists of the system prompt, modality embedding and the question embedding, which contradicts wit Tab. 5.  Would you mind clarifying how exactly to construct the router input?\n  - Also how to get the question embedding? Do you utilize other embedding model or just do that on-the-fly?\n\n- About the MoE deployment:\n  - Do you adopt LoRA-MoE in each Transformer block?\n  - Do you adopt LoRA only for the MLP, similarly with the original MoE?\n- About zero-shot evaluation:\n  - According to Sec. 4.1, the zero-shot evaluation conducted in this paper is only about zero-shot evaluation on novel datasets of the training tasks (e.g., ScienceQA for evaluation and VQA for training).\n  - Therefore, I wonder how it works if you have not seen any QA tasks during training, since it is difficult to understand a specialized architecture like MoE can generalize well to totally unseen tasks during pre-training.\n  - Moreover, if you still train with VQA, but change the prompt template of ScienceQA during testing, will the MoE router be robust to this kind of OoD generalization?\n- Overall, I think this is an interesting paper, but still with some problems to convince me about the effectiveness of the proposed method. I would consider increasing the score if my questions are well addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wuGqdG5Qlh", "reviewer": "Reviewer_7dhr", "summary": "The work proposes a multimodal LoRA-MoE decoder for task- and modality-specific learning. The experimental results (about 20% improvement) have shown the effectiveness and versatility of our design in various 2D and 3D downstream tasks.", "questions": "1. In Figure 5, Loras2 has a probability of 0 on 6 tasks. What could be the possible reasons for this phenomenon? Could you provide more visualizations of experts and provide some insightful analysis?\n2. Can you analyze the impact of k in the top-k sparse gate?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The work adopts a simple but effective gate routing scheme allowing sparsely-activated LoRA modules to learn task- and modality-specific knowledge as an independent expert. \n2. The work can address various 2D/3D vision and language tasks, and conduct various experiments to validate the effectiveness and versatility with a few trainable parameters.", "weaknesses": "1. The work focuses on SFT, whether it can be generalized to pre-training on massive data.\n2. Lack of evaluation on some of the latest MLLM Chat evaluation, such as MMBench, MME, SEED-Bench, etc.\n3. Lack of comparison with some classic and latest methods, such as MiniGPT4, mPLUG-Owl, LLAVA-1.5, Qianwen-VL, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)