HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal
  Large Language Models
Authors: name: Wenqiao Zhang; name: Tianwei Lin; name: Jiang Liu; name: Fangxun Shu; name: Haoyuan Li; name: Lei Zhang; name: He Wanggui; name: Hao Zhou; name: Zheqi Lv; name: Hao Jiang; name: Juncheng Li; name: Siliang Tang; name: Yueting Zhuang
Published: 2024-03-20 09:42:43

Abstract: Recent advancements indicate that scaling up Multimodal Large Language Models
(MLLMs) effectively enhances performance on downstream multimodal tasks. The
prevailing MLLM paradigm, e.g., LLaVA, transforms visual features into
text-like tokens using a static vision-language mapper, thereby enabling
static LLMs to develop the capability to comprehend visual information
through visual instruction tuning. Although promising, the static tuning
strategy~The static tuning refers to the trained model with static
parameters. that shares the same parameters may constrain performance across
different downstream multimodal tasks. In light of this, we introduce
HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters,
in conjunction with a dynamic visual expert and language expert, respectively.
These experts are derived from HyperNetworks, which generates adaptive
parameter shifts through visual and language guidance, enabling dynamic
projector and LLM modeling in two-stage training.
  Our experiments demonstrate that our solution significantly surpasses LLaVA
on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and
LLaVA-Bench. ~Our project is available on the link
https://github.com/DCDmllm/HyperLLaVA.