Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of
  Experts
Authors: name: Jinqiang Long; name: Yanqi Dai; name: Guoxing Yang; name: Hongpeng Lin; name: Nanyi Fei; name: Yizhao Gao; name: Zhiwu Lu
Published: 2024-11-16 02:10:14

Abstract: As the research of Multimodal Large Language Models (MLLMs) becomes popular,
an advancing MLLM model is typically required to handle various textual and
visual tasks (e.g., VQA, Detection, OCR, and ChartQA) simultaneously for
real-world applications. However, due to the significant differences in
representation and distribution among data from various tasks, simply mixing
data of all tasks together leads to the well-known``multi-task conflict" issue,
resulting in performance degradation across various tasks. To address this
issue, we propose Awaker2.5-VL, a Mixture of Experts~(MoE) architecture
suitable for MLLM, which acquires the multi-task capabilities through multiple
sparsely activated experts. To speed up the training and inference of
Awaker2.5-VL, each expert in our model is devised as a low-rank adaptation
(LoRA) structure. Extensive experiments on multiple latest benchmarks
demonstrate the effectiveness of Awaker2.5-VL. The code and model weight are
released in our Project Page: https://github.com/MetabrainAGI/Awaker.