HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models

## Abstract

Abstract Recent advancements indicate that scaling up Multimodal Large Language Models (MLLMs) effectively enhances performance on downstream multimodal tasks.
The prevailing MLLM paradigm, e.g. , LLaVA, transforms visual features into text-like tokens using a static vision-language mapper, thereby enabling static LLMs to develop the capability to comprehend visual information through visual instruction tuning.
Although promising, the static tuning strategy 1 1 1 The static tuning refers to the trained model with static parameters. that shares the same parameters may constrain performance across different downstream multimodal tasks.
In light of this, we introduce HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters, in conjunction with a dynamic visual expert and language expert, respectively.
These experts are derived from HyperNetworks, which generates adaptive parameter shifts through
visual and language guidance, enabling dynamic projector and LLM modeling in two-stage training.
Our experiments demonstrate that our solution significantly surpasses LLaVA on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and LLaVA-Bench. 2 2 2 Our project is available on the link https://github.com/DCDmllm/HyperLLaVA .

## 1 Introduction

The landscape of Large Language Models (LLMs) Devlin et al. (); Radford et al. (); Ouyang et al. () has undergone significant evolution, highlighting their exceptional versatility in managing a wide variety of language-centric applications. To extend the capabilities of LLMs to a wider array of modal inputs, Multimodal Large Language Models (MLLMs) have garnered increasing attention Radford et al. (); Li et al. (); Huang et al. (); Achiam et al. (); Li et al. (). MLLMs are crucial for the development of flexible, general-purpose assistants, as everyday interactions encompass information from various modalities (*e.g.*, videos, audio, 3D environments, point clouds) in addition to text.

Figure: Figure 1: (a) is the overview of LLaVA. (b) describes the simplified version of our HyperLLaVA. (c) shows that compared to LLaVA, our method achieves superior performance across different MLLM benchmarks.
Refer to caption: x1.png

Contemporary MLLMs (*e.g.*, LLaVA Liu et al. (, )) typically adhere to a two-stage training protocol:
(i) Vision-Language Alignment: A static projector is trained by leveraging image-text pairs to synchronize visual features with the language model’s word embedding space.
The projector with static parameters connects the vision and language modalities by translating visual features into visual tokens, enabling the LLM to understand the visual content. The quality of conveyed visual tokens directly influences the MLLM’s performance Zhou et al. ().
(ii) Multimodal Insturction Tuning. Following vision-language alignment, multimodal instruction data are used to refine the LLM, enabling it to respond to users’ varied requests involving visual content. This step is crucial for augmenting the capabilities and controllability of MLLM to address different downstream multimodal tasks.

Despite two-stages’ critical importance, the projector’s structure and LLM tuning strategy have been relatively underexplored, most of the pieces of literature focus on scaling up the pretraining data Bai et al. (); Dai et al. (), instruction-following data Li et al. (); Zhang et al. (); Zhao et al. (), visual encoders Bai et al. () or language models Lu et al. () to facilitate vision-language understanding. What’s more, further quantitative analyses show that the learned model with static parameters may limit their potential for multi-downstream tasks Mahabadi et al. (); Zhang et al. ().
Based on the aforementioned insights, our investigation focuses on the two-stage training process, transitioning from static to dynamic tuning—that is, tuning both the projector and LLM with dynamic parameters to provide flexible design choices that bolster the MLLM’s reasoning abilities across diverse multimodal tasks.

In this paper, we propose HyperLLaVA (Figure 1(b)), its dynamic characterization benefits from a carefully designed expert module, which is derived from HyperNetworks Ha et al. () to generate the dynamic parameters based on input information.
Our bootstrapping philosophy is to dynamically generate strongly correlated features according to visual and language guidance, thereby dynamically modeling the projector and LLM layers, respectively. In detail, HyperLLaVA is learned following the two steps: (i) In vision-language alignment, we divide the projector into static layers (original MLP in LLaVA Liu et al. ()) and dynamic layers (visual expert), where the parameters of static layers are fixed, while the parameters of dynamic
layers are dynamically generated based on visual input.
The visual expert leverages the Hypernetwork to assist the static projector learn a visual-specific projector that adaptively models the
visual features according to the visual guidance.
By doing so, the projector can deliver the adaptative visual tokens to the language semantic space.
(2) In the multimodal instruction tuning stage, we equip the LLM with a language expert, modeling dynamic parameters for LLM blocks. We regard the intermediate output of LLM as language guidance that guides the language expert to provide an improved instruction-specific comprehension of the user’s request.
By doing so, the MLLM increases the flexibility by instead generating unique parameters for every input, allowing the MLLM to make
use of similarities between samples across datasets and avoid potential interference between samples within the same dataset.
Notably, the proposed language expert serves as a parameter-efficient fine-tuning approach for the MLLMs, yielding a comparable performance according to the original LLaVA.

In summary, our contributions are three-fold as follows:

- •
We study the under-explored dynamic tuning strategy for MLLMs and introduce HyperLLaVA, leveraging the visual and language-guided dynamic tuning for projector and LLM;
- •
The proposed visual and language expert serves as a parameter-efficient method of multitask fine-tuning;
- •
We conducted comprehensive and detailed experiments across multiple MLLM benchmarks. The rich experimental results prove the effectiveness and universality of our proposed method.

## 2 Related Work

Large Language Model.
The proliferation of Large Language Models (LLMs) has dramatically reshaped the landscape of natural language processing.
Pioneering models such as encoder-centric model BERT Devlin et al. () and decoder-centric model GPT Radford et al. () have led this charge, showcasing that enhancements in model size and the expansiveness of training datasets can result in unprecedented improvements in performance.
Building on the achievements of their predecessors, subsequent models have brought forth substantial innovations that further advance the prowess of LLMs. PaLM Chowdhery et al. () highlighted the benefits of increasing model parameters for enhanced language comprehension. Meanwhile, InstructGPT Ouyang et al. () and ChatGPT utilized fine-tuning and reinforcement learning strategies to refine their performance in conversational interaction Chen et al. ().
However, the reliance on textual data as the sole source of learning has been a limiting factor, as it constrains the models’ ability to engage with the richly interconnected world of multimodal information.
Multimodal Large Language Model. In recent years, the development of deep learning has brought prosperity to the field of multimodal intelligence Baltrušaitis et al. (); Li et al. (); Zhang et al. (, , ).
Multimodal Large Language Models (MLLMs) leverage the power of LLMs, mitigating extra computational cost and enhancing the efficacy of multimodal pre-training Zhang et al. (), to bridge the gap between textual and multimodal data(*e.g.*, images, videos, and audios). A prominent attempt is CLIP Radford et al. (), demonstrating the alignment of visual and textual modalities via contrastive learning across a broad dataset of image-text pairs. Li et al. () and  Li et al. () follow this trend, proposing BLIP and BLIP-2 improved upon CLIP, and gain remarkable performance in basic visual tasks.
Flamingo Alayrac et al. () led the way in merging vision and language models by utilizing vast amounts of intertwined image-text dataset, revealing unparalleled zero-shot capabilities in processing image-text content within conversational contexts for the first time.
LLaVA Liu et al. () distinctively incorporates short captions annotated by humans and bounding boxes into the GPT4 language model. In the realm of audio processing, there are also some brilliant works, such as SpeechT5 Ao et al. (), MMS Pratap et al. (), PandaGPT Su et al. (), etc.
Hypernetworks.
The original HyperNetwork Ha et al. () is designed to reduce the number of parameters, *i.e*, a small neural network generates parameters for another big neural network, thereby obtaining the model compression for different tasks. Subsequently, HyperNetwork is developed to various domain tasks, including few-shot learning Brock et al. (), graph modeling Zhang et al. (), domain adaptation Zhang et al. (), device-cloud collaboration Lv et al. (, ), etc.

## 3 Methodology

This section describes the proposed MLLM framework HyperLLaVA. We shall present each module and its training strategy.

### 3.1 Problem Formulation

The primary goal is to effectively leverage the capabilities of both the pre-trained LLM and visual model.
The network architecture is illustrated in Figure 2. Given an RGB image $x\in R^{H\times W\times 3}$, where $H$ and $W$ are the origin resolution.
The vision encoder processes input images to obtain a visual token sequence $\mathcal{V}=[v_{1},v_{2},\cdots,v_{N_{v}}]$, where ${N_{v}}$ represents the sequence length of text tokens.
Subsequently, we concatenate the visual tokens and text tokens $\mathcal{T}=[t_{1},t_{2},\cdots,t_{N_{t}}]$, together and feed them into a LLM $\mathcal{M}_{l}lm$, then generate the language response $\mathcal{R}=[r_{1},r_{2},\cdots,t_{N_{r}}]$, where $N_{t}$ and $N_{r}$ indicate the length of text tokens and textual response.
In general, MLLM model $\mathcal{M}(\cdot)$ consists of two functions as below:

$$ $\displaystyle\underbrace{\mathcal{M}(\cdot)}_{\rm{MLLM}}:\underbrace{\mathcal{ M}_{p}((\mathcal{T}|\mathcal{V});\Theta_{p})}_{\rm{Projector}}\rightarrow \underbrace{\mathcal{M}_{l}((\mathcal{R}|\mathcal{V},\mathcal{T});\Theta_{{l}} )}_{\rm{LLM}}\,,$ (1) $$

where $\mathcal{M}_{p}(\cdot;\Theta_{p})$ is the projector and $\mathcal{M}_{t}(\cdot;\Theta_{l})$ LLM tuning with multi-modal instructions with parameters $\Theta_{p}$ and $\Theta_{l}$, respectively.

### 3.2 Preliminaries

LLaVA.
*LLaVA* Liu et al. () is trained following two steps: (i) First,
a two-layer MLP is employed as vision-language projector $\mathcal{M}_{p}(\cdot)$ to convert visual features into
visual tokens $V$, which have the same dimensionality as the word embedding space
in the language model. (ii) Then LLaVA performs instruction-tuning with visual tokens $V$ and language tokens $T$ for the LLM (Llama) $\mathcal{M}_{l}(\cdot)$, generating response tokens $\mathcal{R}$ by optimizing its auto-regressive training objective.

HyperNetwork.
Hypernetwork Ha et al. () is a neural network that generates the weights for another neural network.
Specifically, HyperNetwork treats the parameters of the multi-layer perception (MLP) as a matrix $K^{(n)}\in{R}^{N_{in}\times N_{out}}$, where $N_{in}$ and $N_{out}$ represent the number of input and output neurons of the $n^{th}$ layer of MLP, respectively. $N_{in}$ and $N_{out}$ portray the structure of the MLP layers together. The generation process of $K^{(n)}$ can be regarded as a matrix factorization:

$$ $\displaystyle K^{(n)}=\xi(z^{(n)};\Theta_{p}),\forall n=1,\cdots,N_{l}\,.$ (2) $$

In the *training procedure*, $z^{(n)}$ and $\xi(\cdot)$ are randomly initialized. The gradients are backpropagated to $z^{(n)}$ and $\xi(\cdot)$, which can help to update them. $z^{(n)}$ and $\xi(\cdot)$ will be saved instead of $K^{(n)}$.

### 3.3 Vision-language Guided Expert Module

Original LLaVA’s projector and LLM are trained with static parameters. We argue that the static tuning paradigm may limit the flexible visual token delivery and appropriate expression for different downstream multi-modal tasks. Thus, we propose to equip the original’s LLaVA projector and LLM with a visual expert $\mathcal{E}_{V}$ and a language expert $\mathcal{E}_{L}$: (i) the visual expert adaptively fits the projector’s output according to the specific visual guidance (*e.g*, visual features); (ii) the language expert dynamically modeling the posterior blocks of LLM through anterior LLM’s block output.

Figure: Figure 2: Overview of proposed HyperLLaVA. (a) describes how the proposed visual expert assists the static projector that dynamically converts the image features to adaptive visual tokens, yielding an augmented visual expression for subsequent instruction tuning. (b) is the proposed language expert-integrated tuning, which uses the output of the intermediate layer as language guidance to generate dynamic instruction-specific feature, increasing the flexibility for processing different multimodal tasks.
Refer to caption: x2.png

The expert module is derived from Hypernetorks, which is a neural network that generates its parameters for another neural network. As HyperNetwork dynamically generates a network conditioned on the input embeddings, *i.e.*, the “dynamic characterization” can be modeled by HyperNetwork.
However, directly utilizing the HyperNetwork may not satisfactorily model dynamic learning for two key reasons:

- •
Weak Correlation. The original HyperNetwork learns the latent vector to generate another model’s parameters. This lacks a strong correlation between parameter generation and input guidance.
- •
Unstable Optimization. Using HyperNetwork generate the parameters of the projector or LLM block is large ($D_{x}\times N_{in}\times N_{out}$), *i.e.*, it is hard to optimize the such the numerous parameters, the optimization process is intuitively unstable.

To this end, we carefully tailor the HyperNetwork with the following adjustments:

Input Prior Guidance. We first propose to model the dynamic layers by replacing the learned latent vector $z$ with specific input.
Specifically, given the feature $f_{x^{(i)}}$ extracted from backbone of sample $x^{(i)}$, we first develop a layer-specific encoder $E^{n}(\cdot)$ that encode the $f_{x^{(i)}}$ as $\boldsymbol{e}^{(n)}$. This vector
represents the $n^{th}$ layer parameters.

$$ $\displaystyle\begin{gathered}\boldsymbol{e}^{(n)}={E}^{n}(f_{x^{(i)}}),\forall n =1,\cdots,N_{l}\,,\end{gathered}$ (4) $$

where $N_{l}$ is the number of the modeled layers.

Then the HyperNetwork is used to convert the embedding $\textbf{e}^{(n)}$ into parameters, *i.e.*, we input $\textbf{e}^{(n)}$ into the following two MLP layers to generate parameters of dynamic layers.

$$ $\displaystyle\begin{gathered}\mathcal{\boldsymbol{w}}^{(n)}=(W_{1}\boldsymbol{ e}^{(n)}+B_{1})W_{2}+B_{2},\\ K^{(n)}=\mathcal{\boldsymbol{w}}^{(n)}+\mathcal{\boldsymbol{b}}^{(n)},\end{gathered}$ (7) $$

where $K^{(n)}$ denotes the $n^{th}$ layer parameters of dynamic layers. Two MLP layers’s weight are denoted by $W_{1}$ and $W_{2}$, respectively. $\mathcal{\boldsymbol{b}}^{(n)}$, $B_{1}$ and $B_{2}$ are the biases.

HyperNetwork-aware Adapter. Adapters are sub-networks with small parameters that are inserted after every attention and feed-forward layer in a model Houlsby et al. (). The original adapter is a parameter-efficient learning approach that learns downstream tasks by updating only a small number of parameters. The adapters consist of a pair of downsampling and upsampling layers, and a residual connection. We found that using downsampling and upsampling strategies, the HyperNetwork-generated parameters can be substantially reduced.

Given the visual guidance $x_{V}$ and language guidance $x_{L}$, the vision-language guided expert can be defined as:

$$ $\displaystyle\begin{gathered}\mathcal{E}_{M}(x_{M})=W_{M}^{u}({\rm SwiGLU}(W_{ M}^{d}(x_{M})))\\ W_{M}^{u},W_{M}^{d}=\mathcal{H}_{M}(x_{M}),{\rm where}M\in{V,L}\end{gathered}$ (10) $$

where $M$ indicate the modality, $W_{M}^{u},W_{M}^{d}$ respectively denote the weights for upsampling and downsampling. SwiGLU Ramachandran et al. () is the activation function, Gaussian Error Linear Unit. $\mathcal{H}_{M}$ is the HyperNetwork.

### 3.4 Visual Expert-Assisted Projector

In this stage, our objective is to adapt the image tokens to LLM, allowing the LLM to comprehend the instances in the images. As shown in Figure 2, we divide the projector as static layers and dynamic layers. Following LLaVA1.5 Liu et al. (), we employ an two-layer MLP as the static layers. To empower the projector’s expression, we develop a visual expert that learning the projector shift to model the dynamic text tokens. Specifically, given the visual feature $f_{V}$ extracted from visual encoder, the visual expert will adaptively convert $f_{V}$ to dynamic visual embeddings. We show three alternatives for the dynamic vision-language alignment, the visual tokens $V$ can be calculated as:

$$ $V=\begin{cases}\underbrace{\mathcal{L}_{2}(\mathcal{L}_{1}(f_{V})+\mathcal{E}_ {V_{1}}(f_{V}))}_{\mbox{Use 1st Visual Expert}}\\ \underbrace{\mathcal{L}_{2}(\mathcal{L}_{1}(f_{V}))+\mathcal{E}_{V_{2}}( \mathcal{L}_{1}(f_{V}))}_{\mbox{Use 2nd Visual Expert}}\\ \underbrace{\mathcal{L}_{2}(\mathcal{L}_{1}(f_{V})+\mathcal{E}_{V_{1}}(f_{V})) +\mathcal{E}_{V_{2}}(\mathcal{L}_{1}(f_{V}))}_{\mbox{Use 1st\&2nd Visual Expert}}\end{cases}$ (11) $$

where $\mathcal{L}_{1}$ and $\mathcal{L}_{2}$ denotes two MLPs, $\mathcal{E}_{V_{1}}$ and $\mathcal{E}_{V_{2}}$ are the visual expert for first and second MLPs. We give a details comparison in the Sec. experiments.

**Table 1: Comparison with SoTA methods on 12 benchmarks. Res, PT, IT indicate input image resolution, the number of samples in the pretraining and instruction tuning stage, respectively. We color each row as the best and second best. Improv. $\uparrow$ indicates performance improvement compared with LLaVA.**
| Method | LLM | Res. | PT | IT | VQA Datasets | Benchmark Toolkits |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 10 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 17 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  |  |  |  |  | VQA^v2 | GQA | VizWiz | SQA^I | VQA^T | POPE | MME | MMB | MMB^CN | SEED | LLaVA^W | MM-Vet |
| BLIP-2 Li et al. () | Vicuna-13B | 224 | 129M | - | 41.0 | 41 | 19.6 | 61 | 42.5 | 85.3 | 1293.8 | - | - | 46.4 | 38.1 | 22.4 |
| InstructBLIP Dai et al. () | Vicuna-7B | 224 | 129M | 1.2M | - | 49.2 | 34.5 | 60.5 | 50.1 | - | - | 36 | 23.7 | 53.4 | 60.9 | 26.2 |
| InstructBLIP Dai et al. () | Vicuna-13B | 224 | 129M | 1.2M | - | 49.5 | 33.4 | 63.1 | 50.7 | 78.9 | 1212.8 | - | - | 58.2 | - | 25.6 |
| Shikra Chen et al. () | Vicuna-13B | 224 | 600K | 5.5M | 77.4 | - | - | - | - | - | 58.8 | - | - | - | - | - |
| IDEFICS-9B Laurençon et al. () | LLama-7B | 224 | 353M | 1M | 50.9 | 38.4 | 35.5 | - | 25.9 | - | - | 48.2 | 25.2 | - | - | - |
| IDEFICS-80B Laurençon et al. () | LLama-65B | 224 | 353M | 1M | 60.0 | 45.2 | 36.0 | - | 30.9 | - | - | 54.5 | 38.1 | - | - | - |
| Qwen-VL Bai et al. () | Qwen-7B | 448 | 1.4B | 50M | 78.8 | 59.3 | 35.2 | 67.1 | 63.8 | - | - | 38.2 | 7.4 | 56.3 | - | - |
| Qwen-VL-Chat Bai et al. () | Qwen-7B | 448 | 1.4B | 50M | 78.2 | 57.5 | 38.9 | 68.2 | 61.5 | - | 1487.5 | 60.6 | 56.7 | 58.2 | - | - |
| HyperLLaVA w/o $\mathcal{E}_{V}$ (Ours) | Vicuna-7B | 336 | 558K | 665K | 79.0 | 62.5 | 50.3 | 70.4 | 58.1 | 85.9 | 1486.0 | 65.9 | 59.7 | 61.0 | 63.7 | 32.8 |
| HyperLLaVA w/o $\mathcal{E}_{L}$ (Ours) | Vicuna-7B | 336 | 558K | 665K | 78.8 | 61.9 | 52.1 | 70.7 | 57.5 | 85.6 | 1492.0 | 66.7 | 58.6 | 60.8 | 62.6 | 30.9 |
| LLaVA-1.5 Liu et al. () | Vicuna-7B | 336 | 558K | 665K | 78.5 | 62.0 | 50.0 | 66.8 | 58.2 | 85.9 | 1510.7 | 64.3 | 58.3 | 58.6 | 63.4 | 30.5 |
| HyperLLaVA (Ours) | Vicuna-7B | 336 | 558K | 665K | 79.1 | 62.7 | 51.9 | 70.4 | 58.5 | 86.3 | 1481.2 | 65.9 | 60.6 | 61.4 | 64.0 | 31.0 |
| Improv. $\uparrow$ | - | - | - | - | 0.6 | +0.7 | +1.9 | +3.6 | +0.3 | +0.4 | - | +1.6 | +2.3 | +2.8 | +0.6 | +0.5 |
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |

**Table 2: Three Alternatives for Dynamic Vision-language Alignment. $\mathcal{E}_{V_{1}}$ and $\mathcal{E}_{V_{2}}$ denote visual expert for first and second MLP layer.**
| Methods | VQA Datasets | Benchmark Toolkits |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 4 |  |  |  |  |  |
| 6 |  |  |  |  |  |
|  | GQA | SQA-I | VQA-T | POPE | MME |
| w/o $\mathcal{E}_{V}$ | 62.5 | 70.4 | 58.1 | 85.9 | 1486.0 |
| $\mathcal{E}_{V_{2}}$ | 62.0 | 69.8 | 58.0 | 86.4 | 1442.6 |
| $\mathcal{E}_{V_{1}}\&\mathcal{E}_{V_{2}}$ | 60.1 | 69.5 | 54.4 | 86.1 | 1449.8 |
| $\boldsymbol{\mathcal{E}_{V_{1}}}$ | 62.7 | 70.4 | 58.5 | 86.3 | 1481.2 |
|  |  |  |  |  |  |

Such visual experts learn the projector shift to model the dynamic text tokens, and thus empower the projector’s expression for downstream tasks.

### 3.5 Language Expert-integrated Tuning

In this stage, LLM is adjusted to become an LVLM with multi-modal understanding. We use more complex instructions, including tasks such as image logical reasoning and text recognition, which require the model to have a stronger multi-modal understanding. Different
Previous studies have shown that features provided by the intermediate layer may suffice to preliminarily understand the given input samples Xin et al. ()and can serve as guidance hints to improve
training Romero et al. (). Thus, generating guidance in the intermediate LLM layer allows the model to form a preliminary understanding of the given instruction. Therefore, we regard the output of the intermediate LLM layer as language guidance that generates adaptive instruction-specific features that enhance the generation accuracy. As shown in Figure 2, given the language guidance $f_{L}$, the adapter’s parameters $\{W_{L}^{u},W_{L}^{d}\}$ are generated by $\mathcal{H}_{L}(f_{L})$. By doing so, the instruction-specific features can be calculated as below:

$$ $\displaystyle\begin{gathered}\hat{x}_{L}=\mathcal{E}_{L}(x_{L})+x_{L}+{\rm FFN }({\rm SwiGLU}(x_{l}))\end{gathered}$ (13) $$

where $x_{L}$ is the features generated from RMS normalization and self-attention in LLM’s block.

## 4 Experiments

We verify HyperLLaVA’s effectiveness on multiple datasets and then discuss HyperLLaVA’s properties with controlled studies.

### 4.1 Dataset and Setting

Benchmark Datasets. We evaluate our proposed HyperLLaVA on five VQA datasets:
VQA-v2 Goyal et al. (); GQA Hudson and Manning (); VizWiz Gurari et al. (); SQA^I: ScienceQA-IMG Lu et al. (); VQA^T Singh et al. (): TextVQA and seven Benchmark Toolkits:
POPE Li et al. (); MME Fu et al. (); MMB: MMBench Liu et al. (); MMB^CN: MMBench-Chinese Liu et al. (); SEED: SEED-Bench Li et al. (); LLaVA^W: LLaVA-Bench(In-the-Wild) Liu et al. (); MM-Vet Yu et al. ().

Implementation Details. The model was trained on an 8-A100 machine in one day. The implementation details refer to the Appendix.
In the training of the HyperLLaVA, we utilize the ADAMW Loshchilov and Hutter () optimizer, adapting hyperparameters to cater to the specific requirements of each phase. For the feature alignment stage, parameters are set as $B=32$, $Lr=0.001$, while for visual instruction tuning stage, we adjust the parameters to $B=16$, $Lr=0.00002$. The configuration for the ADAMW optimizer incorporates the following settings: $\boldsymbol{\beta}=(0.9,0.999)$, $\varepsilon=1\times 10^{-8}$, and $W_{d}$ = 0.0, ensuring a bespoke optimization strategy that effectively addresses the unique demands of each training phase.

Besides,
We train our model following the same training process as LLaVA-1.5. The process includes two stages: (1) feature alignment stage: use 558K subset of the LAION-CC-SBU dataset to connect a frozen pretrained vision encoder to a frozen LLM; (2) visual instruction tuning stage: use 150K GPT-generated multimodal instruction-following data, plus around 515K VQA data from academic-oriented tasks, to teach the model to follow multimodal instructions.

Comparison of Methods. For quantifying the efficacy of the proposed framework, we compare HyperLLaVA with previous SOTA approaches. We choose BLIP-2Li et al. (), InstructBLIPDai et al. () based on Vicuna-7B, InstructBLIPDai et al. () based on Vicuna-13B, Shikra Chen et al. (), IDEFICS-9BLaurençon et al. (), IDEFICS-80B Laurençon et al. (), Qwen-VL Bai et al. (), Qwen-VL-Chat Bai et al. () and LLaVA-1.5 Liu et al. (). More details of baselines are in the Appendix.

**Table 3: Analysis of Language Expert Integration for Different LLM Layers.**
| Method | VQA Datasets | Benchmark Toolkits |  |  |  |
| --- | --- | --- | --- | --- | --- |
| 4 |  |  |  |  |  |
| 6 |  |  |  |  |  |
|  | GQA | SQA-I | VQA-T | POPE | MME |
| w/o $\mathcal{E}_{L}$ | 61.9 | 70.7 | 57.5 | 85.6 | 1492.0 |
| Anterior 16 Blocks | 62.5 | 69.4 | 58.5 | 85.9 | 1481.4 |
| All 32 Blocks | 62.7 | 69.5 | 58.6 | 86.0 | 1460.3 |
| Posterior 16 Blocks | 62.7 | 70.4 | 58.5 | 86.3 | 1481.2 |
|  |  |  |  |  |  |

**Table 4: Zero-shot object hallucination evaluation results on POPE dataset. "Yes" indicates the proportion of positive responses to the given question.**
| Method | LLM | Activated | Adersarial | Popular | Random |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 6 |  |  |  |  |  |  |  |  |  |  |  |
| 9 |  |  |  |  |  |  |  |  |  |  |  |
| 12 |  |  |  |  |  |  |  |  |  |  |  |
|  |  |  | Acc | F1-Score | Yes | Acc | F1-Score | Yes | Acc | F1-Score | Yes |
| mPLUG-Owl Ye et al. () | LLaMA-7B | 6.7B | 82.4 | 81.6 | 45.2 | 85.5 | 84.3 | 42.1 | 86.3 | 85.3 | 42.3 |
| MM-GPT Gong et al. () | LLaMA-7B | 6.7B | 50.0 | 66.7 | 100.0 | 50.0 | 66.7 | 100.0 | 50.0 | 66.7 | 100.0 |
| LLaVA-1.5 Liu et al. () | Vicuna-7B | 7B | 85.1 | 84.2 | 44.0 | 87.2 | 86.1 | 41.9 | 50.3 | 45.9 | 41.9 |
| HyperLLaVA | Vicuna-7B | 7B | 85.6 | 84.7 | 44.1 | 87.3 | 86.2 | 42.4 | 50.7 | 46.5 | 42.1 |
|  |  |  |  |  |  |  |  |  |  |  |  |

### 4.2 Overall Performance

We benchmark HyperLLaVA on a wide range of academic VQA benchmarks and recent benchmarks specifically proposed for instruction-following LMMs, totaling 12 benchmarks.
Table 1 summarizes the quantitative results of our framework and baselines on five VQA datasets and five Benchmark Toolkits. We make the following observations:
1) In general, irrespective of the different scenarios, compared to LLaVA, HyperLLaVA achieves the best performance on almost all the multimodal scenarios across both datasets (Except for the MME benchmark), which
strongly demonstrates the generalizability of the proposed HyperLLaVA.
2) HyperLLaVA (both 7B and 13B) outperforms bigger MLLMs with billions of trainable parameters for cross-modal connection (*e.g.*, 80B IDEFICS Laurençon et al. ()). This further indicates the effectiveness of the proposed MLLM structure. 3) Compared with the original LLaVA, we show that HyperLLaVA achieves the best performance across 11 out of 12 benchmarks. Such results benefit from the carefully designed lightweight visual and language expert, which empowers the static projector and LLM to facilitate different multimodal tasks.

### 4.3 Ablation Study

Effectiveness of Each Component.
Table 1 also illustrate the effectiveness of each component, *i.e.*, visual expert $\mathcal{E}_{V}$ and language expert $\mathcal{E}_{L}$. Comparing HyperLLaVA and HyperLLaVA(-$\mathcal{E}_{V}$) (Row 11 *v.s* Row 13), the $\mathcal{E}_{V}$ contributes 2.61% improvement on mean accuracy.
Meanwhile, Row 11 indicates that it suffers from 0.94%, a noticeable performance degradation without the $\mathcal{E}_{L}$.
To sum up, we can observe that the improvement of using each module alone is distinguishable. Combining all the
components, our HyperLLaVA exhibits steady improvement over the baselines.

### 4.4 In-Depth Analysis

We validate the effectiveness of the proposed two modules through the experiments on GQA, SQA-I, VQA-T, POPE and MME benchmarks.

Three Alternatives for Vision-language Alignment. To build
insights on the visual expert-assisted projector in HyperLLaVA, we perform
an in-depth analysis of three alternatives for dynamic vision-language alignment.
Table 2 exhibits the three results. According to our observation, using one visual expert to access the dynamic projection yields the best results. Besides, the other two plans also obtained comparable results, indicating the effectiveness of dynamic projection.

Analysis of Language Expert Integration for Different Blocks. To deeply analyze the effectiveness of language experts, we study the language expert integration for different blocks in Table 3, including anterior 16 blocks (before 1/2 LLM layers), all 32 blocks (all LLM layers) and posterior 16 blocks (after 1/2 LMM layers). Generally speaking, leveraging the language expert integration for the posterior 16 blocks obtained almost the best performance. Besides,
Row 2 and Row 3 utilize the initial language input as language guidance, obtaining suboptimal results compared with language expert integration for the posterior 16 blocks. Our intuition is that the language guidance might not have gathered sufficient contextual information for subsequent dynamic LLM layer modeling.

Figure: Figure 3: Selected blocks for language guidance.
Refer to caption: x3.png

Analysis on the Inserted Blocks for Language Guidance. We investigate the impact of inserting language guidance into different layers
of LLMs.
We report the evaluation score of GQA and POPE datasets in Figure 4. We observe that the performance is low when we insert language guidance too early (*i.e.*, 4, 8) as the model might not have gathered sufficient contextual information to generate effective guidance. Meanwhile, inserting language guidance too late (*i.e.*, 24, 28) degenerates
the performance. We speculate this is due to the generated guidance being too concentrated and there not being enough layers to integrate the language-aware details.

Analysis of Expert’s Structure. We systematically present the explicit benefits from the carefully designed expert’s structure in Table LABEL:tab:expert. The adapter-based structure surpasses MLP-based structure across all datasets, mainly due to the generated MLP is no longer a lightweight network to optimize, producing unstable performance. Compared with HyperNetwork+Adapter (Row 3 *vs* Row 4), our proposed vision-language guided expert structure obtained the best performance. The results correspond with our assumption of the original HyperNetworks, which lacks a strong correlation between input and parameter generation. Our method, allows the model to make use of similarities between samples across datasets
and avoid potential interference between samples
within the same dataset.

Effect of Dimension of Expert Input and Downsampling. Figure 4 empirically provides an appropriate dimension of input and downsampling, *i.e*, 64 and 16, respectively, either increasing or decreasing this value results in a performance decay. According to our analysis, a bigger dimension may result in an unstable HyperNetwork optimization and a smaller value contains less language-guided information for dynamic learning, and thus yielding performance decay.

Parameter-efficient Fine-tuning. Our proposed language expert also can serve as a parameter-efficient fine-tuning function. The structure is similar to the HyperNetwork+Adapter.
However, original hypernetwork-based approaches generally condition their parameters on a learned latent embedding, implying the model is the same for every example, yield performance decay. Summing up, the proposed language expert is an effective and parameter-efficient way to share information across multiple adapters to enable
positive transfer to low-resource and related tasks.

Object Hallucination Evaluation.
We adopt the evaluation pipeline of POPE  Li et al. (), a polling-based query method, to evaluate object hallucination in HyperLLaVA. The results are presented in Table  4,
where HyperLLaVA exhibits the best performance, indicating that HyperLLaVA tends to generate objects consistent with the given image. Additionally,
we observe that the “yes” ratio of HyperLLaVA remains relatively balanced, indicating that our model is capable of providing accurate feedback based on the questions.

Figure: Figure 4: Performance with respect to the different input and downsampling dimension in expert.
Refer to caption: x4.png

## 5 Conclusion

Building upon HyperLLaVA’s innovative dynamic tuning strategy, our work paves the way for groundbreaking advancements in multimodal learning systems. By adaptively tuning both projector and LLM parameters, and integrating dynamical visual and language experts, we not only surpass the performance benchmarks set by LLaVA but also introduce a parameter-efficient methodology. This approach offers a new horizon for enhancing multimodal task performances through personalized, dynamic adjustments. Future research could further explore the scalability of dynamic tuning mechanisms, potentially unlocking new avenues for understanding and integrating multimodal information more seamlessly.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the weaknesses above.
soundness: 3
strengths: - This paper is clearly written and easy to follow
- This paper has conducted detailed experiments that show how each part works and validated that the overall method is effective.
summary: This paper addresses potential issues of task interference or negative transfer that may arise in the multi-task instruction tuning stage of MLLMs SFT training. Based on the concept of HyperNetworks, the authors propose dynamic visual and language experts that can adjust the corresponding projectors and LLM weights based on the semantics of the instruction. This approach aims to balance the potential negative effects of different tasks on each other during multi-task instruction tuning stage. Additionally, the authors introduce a new benchmark to comprehensively evaluate various aspects of MLLM capabilities. The effectiveness of their method is validated on both existing benchmarks and the new benchmark they proposed.
weaknesses: - **Limited Scalability of the Method**: According to Table 1 in the paper, as the model size increases from 7B to 13B, the improvements provided by the proposed method over the baseline decrease, with most accuracy gains staying within a range of 0.1 to 1.0. This suggests that the method may lack scalability.
- **Limited Innovation of the CMT Benchmark**: The CMT Benchmark proposed by the authors seems to primarily integrate existing open-source benchmarks, lacking significant innovation compared to other established benchmarks. Additionally, the models tested on this benchmark are quite limited, and no evidence is provided to show that it offers any insightful conclusions.
- **More Ablation Study on Each Component**: Tables 3 and 4 focus on validating the effectiveness of the different experts proposed by the authors. In some benchmarks, such as SQA, SEED, and MMB, there are hardly any significant improvements. The authors seem to lack an ablation study to evaluate the effectiveness of combining these two modules. Additionally, Table 5 indicates that the performance improvements of the proposed method are quite limited.
- **Some Content Is Unclear**: The authors should specify the number of parameters introduced by their method. In a supervised training setting, it is difficult to determine whether the performance gains are due to the increase in parameters rather than the method itself.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: See weakness
soundness: 2
strengths: 1. The writing is good and easy for understanding.
2. Experiments are extensive to validate performance gains of the proposed method on some benchmarks.
summary: The paper proposes HyperLLaVA, which generates dynamic parameter shifts through visual and language guidance, enabling
dynamic vision-language alignment and instruction tuning in two-stage training.  Experiments validate its effectiveness on some benchmarks.
weaknesses: 1. MoE is not new in LLM and MLLM, which can also be a dynamic way for task transferring. Why not comparing with these methods, such as MoE-LLaVA, LLaVA-MoLE.
2.  Performance gains seem minor from experiments.  In Tab4, performance gains on SQA and MMB are almost ignoble.
3. Line 300-301, why authors claim that CMT is zero-shot, since it’s actually a supervised setting. 
4. I also suspect the value of CMT benchmark, which is simply combined with existing datasets.  Actually, existing benchmarks like MMVet and MMBench contain more complex and diverse multi-modal instructions, which are strictly zero-shot.
5. All figures could be more clear for reading. It’s very difficult to understand the method from figures.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Is the training process of hypernetworks similar to the bilevel optimization seen in NAS (Neural Architecture Search)? What are the advantages and characteristics of parameters generated by hypernetworks compared to directly fine-tuned parameters? Could you possibly provide quantitative or visualized results?

In Table 5, the bolded section for MME accuracy seems incorrect. Typically, "LoRA" is the preferred capitalization rather than "LoRa." Additionally, the use of "L" in Equation 6 may lead to misunderstanding, like "Loss".
soundness: 2
strengths: The writing is fluent. The paper effectively identifies two challenges in applying hypernetworks to MLLM: weak correlation and unstable optimization.
summary: Recent advancements in scaling Multimodal Large Language Models (MLLMs) show improved performance on multimodal tasks, though static visual-language mappers like those in LLaVA limit potential across various tasks. To overcome this, This paper with HyperLLaVA introduces adaptive adjustments to both the projector and language model parameters through dynamic visual and language experts. These experts, generated by a hypernetwork, produce adaptive parameter offsets that respond to visual and language cues. This dynamic modeling is achieved through a two-stage training process. Experiments confirm that HyperLLaVA significantly outperforms LLaVA on multiple MLLM benchmarks, addressing limitations of static model adaptation.
weaknesses: The paper discusses the issue of unstable optimization in large language models and suggests a potential trade-off between the number of parameters generated by hyperparameters and optimization stability. However, the authors have only considered a minimal number of additional parameters (it would be helpful if the paper specifies what percentage these parameters represent of the original network). For instance, the addition of the visual expert is limited to the projector, which seems similar to adding a joint fine-tuning LoRA. The actual performance enhancement from the visual expert is unconvincing since it is only integrated at the input layer, resembling an enhanced connector, while many MLLM studies have shown that connectors contribute minimally to model performance.

Furthermore, in the language expert, the use of half-layer language guidance as input for the hypernetwork is quite unusual. Using the sample x itself, particularly the hidden layer embedding of its own sample, is rare (relevant literature would be appreciated if available). This self-generated network approach might suggest that it merely adds an extra structure rather than generating effective parameters, akin to LoRA, requiring many additional tuning parameters. It is unclear how the choice of using half of the layers was made. Will generating parameters for the second half of the network from the first half's output create a highly coupled network structure? Is there any pruning of structures like the hypernetwork during inference?

Lastly, the paper lacks clear motivation for the method used, and it is not clear what exactly constitutes the "unstable" performance mentioned in Table 5. Is this instability due to the simplicity of the projector structure or other optimization issues, such as drastic parameter changes? If adding experts to the projector laterally can improve these issues, why not utilize a multi-head approach in the projector? Considering inference costs in MoE and the unclear independence of the visual expert, these questions remain. Moreover, the experiments compare with LLaVA v1.5 NeurIPS 23, but it would be beneficial to compare with more recent works like CogVLM, Wings, and Mono InternVL. The performance improvements in Table 4 are almost all below 0.5%, which seems unconvincing.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: (1) The symbols should be defined systematically and consistently.    
(2) The comparisons with baselines should be fair.
soundness: 3
strengths: (1) The method is simple to implement.      
(2) Sound experimental results and ablations.
summary: The static paradigm in current MLLM, like LLaVA, can introduce potential task interference, ie, the improvements for one task can cause performance degradation on other tasks.
Thus， the paper introduces HyperLLaVA to adjust the parameters of the vision projector and LLM conditioned on instruction semantics.

Experimental results are reported on commonly used benchmarks, like VQAv2, GQA, VizWiz, SQA, POPE, MME, MMB, SEED, and LLaVA-bench. Reasonable improvements are achieved.
weaknesses: (1) The symbols are not consistent. What are the relationships between $\xi$, $E^{n}()$, and $H_M$ in Eqs (2), (3), (5)?

(2) What's the "L1" in line of 250?

(3) Why does the last input token $h^{L/2}$ at L/2-th layer can fully perceive the whole multimodal context?

(4) The comparison with LLaVA seems unfair because the paper uses an extra 505K CMT training data.     
      For the HyperLLaVA, is LLM fully fine-tuned or trained with LoRA?

(5) Indicated by the formulation of $\mathcal{V}$ in line of 246 for vision projector and Eq. (6) for LLM, it seems not to be dynamic but an extra residual output.     
Moreover, what's the network structure of $\xi$ in Eq. (6)?

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: My concerns and questions are given in the weakness section.
soundness: 1
strengths: Subjectively speaking, given an ill-defined problem, the other efforts made by the authors can hardly be regarded as the merits of this paper.
summary: This paper focus on the problem of the static mapping of existing MLLMs,  and propose a design of dynamic vision and language token experts to handle the claimed ``negative transfer'' of MLLMs. However, the reviewer think that the problem of this paper focuses is ill defined, which greatly contradicts the main principle of existing MLLMs. Meanwhile, the experimental results can not well support the argument of this paper as well as the design proposed in this paper.
weaknesses: 1. The main problem of this paper is its motivation. Task transfer learning is important for MLLMs, but it is hard to say negative transfer really matters in MLLMs. Above all, the most recognized principle of MLLMs is to use the giant LLM to learn massive multimodal data at the same time, thus generalizing to all downstream tasks. In this case, how can the authors argue that the static parameters are not good for large-scale MLLMs, just depending on the observation of [Want et al. 2019] experimented on the small-scale models and benchmarks. 

More theoretical analyses and recent literature review are required to validate the motivation of this paper. 

2. In addition, the design focus of this paper is also very problematic, i.e., learning dynamic projection for visual and text tokens. The main pricinple of most MLLMs is to project tokens of different modalities onto the same semantic, i.e. ,the one of LLM. Why the authors think the task-specific projection is better than this mainstream paradigm? Moreover, the text tokens have been well trained with the LLM architecture, thus why task-specific text embedding is needed? 

Similarly, More theoretical analyses and recent literature review are required to validate this design . 

3. The experiments do not support the argument of this paper. The overall improvement of the proposed Hyper-LLaVA is very marginal, which can be easily achieved by just increasing the training steps of LLaVA-1.5. Meanwhile, other experiments can also not strongly prove the motivation about the dynamic token design of this paper, as well as the qualitative analyses.

Some detailed suggestions:
1. Provide a direct comparison with LLaVA-1.5 trained for additional steps.
2. Include more rigorous statistical analyses  to demonstrate significance.
3. Design additional experiments that more directly test the benefits of the dynamic token approach.

Besides,  the complexity and additional steps of the proposed dynamic token projection are better to highlighted.

Overall, this paper states an argument against common research paradigm of existing MLLMs, but are not sufficiently supported in terms of either principle, methodology or experimental results.