Phantom of Latent for Large Language and Vision Models

## Abstract

Abstract The success of visual instruction tuning has accelerated the development of large language and vision models (LLVMs). Following the scaling laws of instruction-tuned large language models (LLMs), LLVMs either have further increased their sizes, reaching 26B, 34B, and even 80B parameters. While this increase in model size has yielded significant performance gains, it demands substantially more hardware resources for both training and inference. Consequently, there naturally exists a strong need for efficient LLVMs that achieve the performance of larger models while being smaller in size. To achieve this need, we present a new efficient LLVM family with model sizes of 0.5B, 1.8B, 3.8B, and 7B parameters, Phantom , which significantly enhances learning capabilities within limited structures. By temporarily increasing the latent hidden dimension during multi-head self-attention (MHSA), we make LLVMs prepare to look and understand much more vision-language knowledge on the latent, without substantially increasing physical model sizes. To maximize its advantage, we introduce P hantom O ptimization ( PO ) using both autoregressive supervised fine-tuning (SFT) and direct preference optimization (DPO)-like concept, which effectively follows correct answers while eliminating incorrect and ambiguous ones. Phantom outperforms numerous larger open- and closed-source LLVMs, positioning itself as a leading solution in the landscape of efficient LLVMs. Code is available in https://github.com/ByungKwanLee/Phantom .

## 1 Introduction

In recent years, artificial general intelligence (AGI) has increasingly become a part of daily life, significantly enhancing our convenience. This trend is largely attributed to technical advancements of large language models (LLMs) and their impressive generalization performance, facilitated by instruction tuning (Wei et al., ; Chung et al., ). Building on this momentum, instruction tuning has expanded its realm into visual instruction tuning (Liu et al., ), integrating both language and vision as a format of text and image, under the use of pretrained LLMs. Based on them, numerous large language and vision models (LLVMs) have continuously emerged as multimodal LLMs and they have shown outstanding vision-language performances.

In terms of open-to-public regarding model architectures and their trained parameters, LLVMs can be categorized into open-source and closed-source models. For example, there are representative closed ones: GPT-4V (OpenAI, ), Gemini-Pro (Team et al., ), and Qwen-VL-Plus (Bai et al., ; ), all of which are renowned for their remarkable vision-language performances, large model sizes, and extensive number of dataset samples. In response, open-source LLVMs have tried to narrow the performance gap with their closed-source performances, by following the similar strategies the closed ones used, such as scaling up model sizes (Liu et al., ; McKinzie et al., ; Li et al., ) (e.g., 26B, 34B, and 80B) and curating larger number of visual instruction tuning samples (Hu et al., ; Fang et al., ; Tong et al., ) (e.g., 4M, 6M, and 10M).

Along with them, several modules have focused on image-level understanding by leveraging numerous types of vision encoders (Kar et al., ; Lu et al., ; Goncharova et al., ; Ranzinger et al., ; Zhao et al., ; Li et al., ) and multiple computer vision models (Chen et al., ; Wang et al., ; Jiao et al., ; Lee et al., ; ). Additionally, a series of projectors have been employed alongside various vision encoders to improve fine-grained understanding (Li et al., ; Tong et al., ; Ge et al., ; Chen et al., ; Yao et al., ) through partitioning the image. Besides, a multifaceted rationale-embedded projector (Lee et al., ) has been used to enhance real-world knowledge such as document, chart, and math.

Figure: Figure 1: Overview of performances compared with Phantom and closed-source LLVMs
Refer to caption: /html/2409.14713/assets/x4.png

However, these efforts — summarized as (a) scaling up model size, (b) curating larger datasets, and (c) incorporating additional modules and projectors — may not be regarded as a primary key to basically improve their own learning capabilities of LLVMs. In other words, there remains unexplored potential in fully utilizing LLVMs to align vision knowledge with language one and embed much more vision-language knowledge within limited structures, without relying on external modules and projectors. Beyond their limited learning capabilities, specifically, (a) and (b) bring in striking computational burdens during training, necessitating high-end GPUs with substantial VRAM. This (a) more becomes a critical drawback in devices with limited GPU resources, such as mobile phones and embedded boards. Furthermore, the high computational inference costs, associated with larger model sizes, exacerbate these issues, particularly for real-time applications such as augmented reality (AR) systems. As a result, deploying and operating LLVMs in such resource-constrained on-device environments becomes a major challenge.

To meet the two needs of maintaining model sizes while achieving superior performance, we present an efficient LLVM family, Phantom, which stimulates enlarging vision-language learning capabilities within limited structures. When conducting multi-head self-attention (MHSA), Phantom temporarily increases the latent hidden dimension and prepares to look and understand much more vision-language knowledge. Without significantly increasing the physical model size, we get an effect of increasing the dimension in query, key, and value, which we now call as Phantom Dimension. In order to maximally boost this advantage, we introduce Phantom Optimization (PO), inspired by RLHF and DPO (Christiano et al., ; Stiennon et al., ; Ouyang et al., ; Rafailov et al., ; Hong et al., ; Meng et al., ). Unlike traditional preference-based methods, PO is designed to minimize the generation of incorrect and ambiguous answers. Since autoregressive supervised fine-tuning (SFT) primarily focuses on producing correct answers, PO provides Phantom with additional guidance to avoid confusing answers by borrowing the recent DPO formulation (Meng et al., ).

To do so, we first need a collection of incorrect and ambiguous answers. These are generated and filtered through GPT-4o(-mini) and human review from 2.8M visual instruction tuning samples covering diverse capabilities (details in Section [3](#S3)). This process resulted in the curation of 2M Phantom triples including question, its correct answer, and the corresponding incorrect and ambiguous answers (see Appendix [A](#A1)). By using the triple, Phantom is trained with the two training steps, where we train vision projector and Phantom Dimension in the first step with the pretrained LLM frozen. In the second step, all components are trained together. Notably, PO utilizes SFT together with DPO-like concept throughout first training step, making Phantom have an ability that follows correct answers while eliminating incorrect and ambiguous ones. In the experiment section, we demonstrate that handling the latent hidden dimension and using PO enhances vision-language performances by a large margin. As a result, we release an efficient LLVM family Phantom with 0.5B, 1.8B, 3.8B, and 7B model sizes, which outperform open- and closed-source LLVMs, establishing a leading solution in the realm of efficient LLVMs.

Figure: (a) 7B$\sim$80B, and Unknown
Refer to caption: /html/2409.14713/assets/x13.png

Our contribution can be summarized into two main aspects:

- •
We present a new efficient large language and vision model (LLVM) Family, Phantom, which temporarily increases the latent hidden dimension during multi-head self-attention (MHSA) to enhance vision-language learning capabilities within limited structures.
- •
Curating efficient size 2M number of Phantom triples, we introduce a training strategy of Phantom Optimization (PO) which avoids incorrect and ambiguous answers, showcasing more advancements across numerous evaluation benchmarks.

## 2 Related Works

### Large Language and Vision Models.

To bridge the performance gap with closed-source LLVMs, open-source LLVMs have adopted three primary strategies: scaling up model size, curating larger datasets, and incorporating additional modules or projectors. For instance, LLaVA-NeXT  (Liu et al., ), MM1 (McKinzie et al., ), Yi-VL (Young et al., ) and MiniGemini (Li et al., ) build model variants with parameters up to 34B. Concurrent to these efforts, mPLUG-Owl (Hu et al., ), VILA${}^{2}$ (Fang et al., ), and Cambrian-1 (Tong et al., ) curate high-quality visual instruction tuning datasets specialized for diverse visual capabilities. Lastly, recent works have leveraged various vision encoders  (Kar et al., ; Lu et al., ; Goncharova et al., ; Ranzinger et al., ; Zhao et al., ; Li et al., ) and integrated external computer vision modules (Chen et al., ; Wang et al., ; Jiao et al., ; Lee et al., ; ) to expand LLVMs’ perception capabilities. Alongside using extra vision encoders, several works utilize projectors to extract hierarchical features of images (Li et al., ; Tong et al., ; Ge et al., ; Chen et al., ; Yao et al., ) or to improve real-world knowledge comprehension such as document analysis, chart interpretation, and mathematical reasoning (Lee et al., ).

While these approaches enhance downstream task performance, they do not address the core challenge of improving the intrinsic learning capabilities of LLVMs. Scaling up model size or employing larger instruction tuning datasets leads to substantial computational burdens. In addition, relying on extra visual encoders or computer vision modules brings in external visual knowledge, but they mainly focus on visual perception-related capabilities and their additional parameters may also lead the burden. This underscores the need for developing more efficient LLVMs with enhanced inherent capabilities that do not depend on such resource-intensive strategies.

### Efficient Modeling.

In an effort to enhance the fundamental capabilities of LLMs while maintaining model size, several works for natural language processing has increasingly focused on developing smaller model sizes (Thawakar et al., ; Mehta et al., ; Liu et al., ), network pruning (Ma et al., ; Men et al., ; Ashkboos et al., ), and quantization (Li et al., ; Shao et al., ; Park et al., ). These approaches primarily aim to accelerate training speed and reduce inference time while retaining performance, rather than boosting performances or improving LLVMs’ embedding capabilities of vision-language knowledge within the limited structures. While efficient modeling has been extensively explored for LLMs, the design of efficient vision-language models (LLVMs) remains underexplored. A recent work, TroL (Lee et al., ), uniquely introduces a layer traversing technique that reuses layers in a token-wise manner to potentially embed more vision-language knowledge. However, it faces significant challenges, such as increased inference time due to doubling layer propagation and critical issues with key-value cache storage, preventing it from fully realizing its potential for efficient LLVMs.

In response to the need for efficient yet high-performing LLVMs, we introduce a new efficient LLVM family, Phantom, which enhances the embedding capability of vision-language knowledge by temporarily increasing the latent hidden dimension during multi-head self-attention (MHSA). This innovation, combined with 2M Phantom triples to guide LLVMs towards correct answers while avoiding confusion, is expected to pave the way for more efficient LLVMs in both training and inference and to represent a crucial first step in advancing the field.

## 3 Phantom

### Overview of Model Architecture.

As shown in Figure [3](#S3.F3)(a), the architecture of Phantom model consists of vision encoder, vision projector, and a multimodal language model including word embedding and language model head, which follows a common configuration used in open-source LLVMs (Liu et al., ; ; Bai et al., ; Chen et al., ; McKinzie et al., ). Specifically, we utilize InternViT-300M (Chen et al., ) as the vision encoder instead of CLIP-L-428M (Radford et al., ), due to its superior ability to align text-to-image representations through contrastive learning with large language models (LLMs). The vision projector is constructed using two fully connected layers, where GELU (Hendrycks & Gimpel, ) activation function is interleaved with each layer. For multimodal LLM component, we initialize it using pretrained LLMs across various sizes, selected for their state-of-the-art performance within their respective size: Qwen2-0.5B (Yang et al., ), InternLM2-1.8B (Cai et al., ), Phi3-mini-3.8B Abdin et al. (), and InternLM2.5-7B (Cai et al., ).

### Gathered Visual Instruction Tuning Sample Configuration.

To cover a broad range of capabilities, we compile 2.8M visual instruction tuning samples across multiple datasets, encompassing various domains such as fundamental image understanding, real-world common-sense knowledge, non-object visual concepts (e.g., documents, charts, diagrams, symbols, and signs), and general mathematical problems. Our dataset collection includes basic image understanding samples from ShareGPT4o-Images (57K) (Erfei Cui, ), ShareGPT4V (755K) (Chen et al., ), ALLaVA-VFLAN/Text (548K) (Chen et al., ), and MiniGemini (27K) (Li et al., ) targeting tasks for DocVQA (Mathew et al., ), ChartQA (Masry et al., ), DVQA (Kafle et al., ), and AI2D (Kembhavi et al., ). Additionally, we collect LLaVA-HD (116K) (Zhang et al., ) for Science and Mathematical Reasoning (SMR), supporting ArXivQA (Li et al., ) and TextbookQA (Kembhavi et al., ), and we further integrate document understanding samples from mPLUG-DocOwl1.5-Downstream/Reasoning (599K) (Hu et al., ) and general mathematical problems from GLLaVA (177K) (Gao et al., ), MathVision (3K) (Wang et al., ), MathInstruct (262K) (Yue et al., ), and MathPlus (304K) (Yue et al., ).

### Curation of Phantom Triples.

From the gathered 2.8M visual instruction tuning samples, we generate incorrect and ambiguous answers based on the existing question-answer pairs. To reduce data generation costs, we utilize GPT-4o-mini with the following prompt: “Question: {}. Answer: {}. Based on the question and the answer, make an incorrect and ambiguous answer compared to the original one. The length of the original answer should be maintained. Do not include any additional text.”. Here, {} serves as a placeholder. Next, we employ GPT-4o to validate the generated responses using the prompt: “Original Answer : {}. Incorrect and Ambiguous Answer: {}. Provide ‘Yes’ or ‘No’, where ‘Yes’ means it is incorrect and ambiguous answer compared to the original one, ‘No’ means it is correct answer compared to the original one. Do not include any additional text.”. All samples labeled ‘No’ are discarded, while the ‘Yes’-labeled samples undergo human review to verify if they are genuinely confusing. Through this process, we curate 2M Phantom Triples, consisting of a question, its correct answer, and a corresponding confusing answer.

Figure: (a) Model Architecture
Refer to caption: /html/2409.14713/assets/x21.png

### Realization of Phantom Dimension.

For better understanding, Figure [3](#S3.F3)(b) represents the simple overview of how Phantom Dimension works. We utilize start of sequence (sos) token that will serve as a key in enhancing the latent hidden dimension for the query, key, and value components in multi-head self-attention (MHSA) layers. The latent feature on the location of sos token is propagated into QKV linear function, and we denote its outputs as $Q^{*}_{l}\in\mathbb{R}^{d_{q}}$, $K^{*}_{l}\in\mathbb{R}^{d_{kv}}$, and $V^{*}_{l}\in\mathbb{R}^{d_{kv}}$ at each layer $l$. Note that $d$ denotes the latent hidden dimension. $Q^{*}_{l}$, $K^{*}_{l}$, and $V^{*}_{l}\in\mathbb{R}^{d_{kv}}$ are supposed to inject into the multi-head cross-attention (MHCA) module. A natural question arises: Why inject these features into the cross-attention module? The reason lies in the dynamic length $N$ of user input tokens, which varies with the question length. Therefore, these features need to have dimension $Q^{*}_{l}\in\mathbb{R}^{N\times d_{q}}$, $K^{*}_{l}\in\mathbb{R}^{N\times d_{kv}}$, and $V^{*}_{l}\in\mathbb{R}^{N\times d_{kv}}$ since sos token only represents a single token. Therefore, it must be expanded to match the $N$ tokens of the input sequence, and the cross-attention module make these features expanded into input sequence token number $N$, as follows:

$$ $\begin{split}Q^{*}_{l}&\leftarrow\text{MHCA}(q=Q_{l},k/v=Q^{*}_{l}),\\ K^{*}_{l}&\leftarrow\text{MHCA}(q=K_{l},k/v=K^{*}_{l}),\\ V^{*}_{l}&\leftarrow\text{MHCA}(q=V_{l},k/v=V^{*}_{l}),\end{split}$ (1) $$

where we change their dimension into $Q_{l}$: $\mathbb{R}^{N\times h_{q}\times\frac{d_{q}}{h_{q}}}$ and $K_{l},V_{l}$: $\mathbb{R}^{h_{kv}\times\frac{d_{kv}}{h_{kv}}}$ for conducting multi-head cross attention with head number $h_{q}$ and $h_{kv}$. Next, in order to make LLVMs embed much more vision-language knowledge, we enlarge the latent hidden dimension by concatenating the original query, key, and value matrices with the cross-attended outputs dimension-wise, yielding $\begin{bmatrix}Q_{l}&Q^{*}_{l}\end{bmatrix}\in\mathbb{R}^{N\times h_{q}\times\frac{2d_{q}}{h_{q}}}$, $\begin{bmatrix}K_{l}&K^{*}_{l}\end{bmatrix}\in\mathbb{R}^{N\times h_{kv}\times\frac{2d_{kv}}{h_{kv}}}$, and $\begin{bmatrix}V_{l}&V^{*}_{l}\end{bmatrix}\in\mathbb{R}^{N\times h_{kv}\times\frac{2d_{kv}}{h_{kv}}}$. We then apply multi-head self-attention (MHSA) used in multimodal LLM to these concatenated ones:

$$ $O_{l}=\text{Softmax}\left(\lambda\left(\frac{2d_{q}}{h_{q}}\right)^{-\frac{1}{2}}\begin{bmatrix}Q_{l}&Q^{*}_{l}\end{bmatrix}\begin{bmatrix}K_{l}&K^{*}_{l}\end{bmatrix}^{\top}\right)\begin{bmatrix}V_{l}&V^{*}_{l}\end{bmatrix},$ (2) $$

where $\lambda$ denotes a regularization parameter, and $O_{l}\in\mathbb{R}^{N\times h_{q}\times\frac{2d_{q}}{h_{q}}}$ represents the output features of MHSA. After its computation, the output features should return to the original hidden dimension, as they will be propagated through the remaining transformer modules, such as feed-forward network (FFN). At this stage, we aim to compress the output features while minimizing information loss as much as possible. To achieve this, we split the output $O_{l}$ into two halves: $O_{l}$[:, :, :$\frac{d_{q}}{h_{q}}$] and $O_{l}$[:, :, $\frac{d_{q}}{h_{q}}$:] (Python slicing format), denoted as $\bar{O_{l}}\in\mathbb{R}^{N\times h_{q}\times\frac{d_{q}}{h_{q}}}$ and $\tilde{O_{l}}\in\mathbb{R}^{N\times h_{q}\times\frac{d_{q}}{h_{q}}}$, respectively. To flexibly mix them, weighted-average operation is employed, and then finally we can get the compressed outputs $O_{l}\leftarrow\bar{w}\odot\bar{O_{l}}+\tilde{w}\odot\tilde{O_{l}}$ where $\odot$ is element-wise multiplication, and

$$ $\bar{w}=\frac{e^{f(\bar{O_{l}})}}{e^{f(\bar{O_{l}})}+e^{g(\tilde{O_{l}})}},\quad\tilde{w}=\frac{e^{f(\tilde{O_{l}})}}{e^{f(\bar{O_{l}})}+e^{g(\tilde{O_{l}})}},$ (3) $$

where $f$ and $g$ comprise each one fully-connected layer: $\mathbb{R}^{N\times h_{q}\times\frac{d_{q}}{h_{q}}}\rightarrow\mathbb{R}^{N\times h_{q}}$, and the compressed outputs are then propagated into remaining modules with root mean square (RMS) layer normalization (Ba et al., ; Zhang & Sennrich, ) and Add&Norm operation.

**Table 1: Comparison with the current existing standard model size open-source LLVMs, evaluating vision-language performances of Phantom on numerous general evaluation benchmarks: SQA${}^{\text{I}}$ (Lu et al., ), AI2D (Kembhavi et al., ), ChartQA (Masry et al., ), SEED${}^{\text{I}}$ (Li et al., ), POPE (Li et al., ), HallB (Liu et al., ), MME (Fu et al., ), MathVista (Lu et al., ), MMB (Liu et al., ), MMB${}^{\text{CN}}$ (Liu et al., ), MM-Vet (Yu et al., ), and LLaVA${}^{\text{W}}$ (Liu et al., ). Bold and Underline represent the top and the second, each.**
| LLVMs | SQA${}^{\text{I}}$ | AI2D | ChartQA | SEED${}^{\text{I}}$ | POPE | HallB | MME | MathVista | MMB | MMB${}^{\text{CN}}$ | MM-Vet | LLaVA${}^{\text{W}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ShareGPT4V-7B (Chen et al., ) | 68.4 | - | - | 69.7 | - | 49.8 | 1944 | 25.8 | 68.8 | 62.2 | 37.6 | - |
| InternLM-XC-7B (Zhang et al., ) | - | - | - | 66.1 | - | 57.0 | 1919 | 29.5 | 74.4 | 72.4 | 35.2 | - |
| Monkey-10B (Li et al., ) | 69.4 | - | - | 68.9 | - | 58.4 | 1924 | 34.8 | 72.4 | 67.5 | 33.0 | - |
| VILA-7B (Lin et al., ) | 68.2 | - | - | 61.1 | 85.5 | - | - | - | 68.9 | 61.7 | 34.9 | - |
| VILA-13B (Lin et al., ) | 73.7 | - | - | 62.8 | 84.2 | - | - | - | 70.3 | 64.3 | 38.8 | - |
| SPHINX-7B (Lin et al., ) | 70.6 | - | - | 71.6 | 86.9 | - | 1797 | 27.8 | 65.9 | 57.9 | 40.2 | - |
| SPHINX-MoE-7B$\times$8 (Gao et al., ) | 70.6 | - | - | 73.0 | 89.6 | - | 1852 | 42.7 | 71.3 | - | 40.9 | - |
| SPHINX-Plus-13B (Gao et al., ) | 70.6 | - | - | 74.8 | 89.1 | 52.1 | 1741 | 36.8 | 71.0 | - | 47.9 | - |
| LLaVA-NeXT-7B (Liu et al., ) | 70.1 | - | - | 70.2 | 86.5 | - | 1851 | 34.6 | 69.6 | 63.3 | 43.9 | 72.3 |
| LLaVA-NeXT-8B (Liu et al., ) | - | 71.6 | 69.5 | - | - | - | 1972 | 37.5 | 72.1 | - | - | 80.1 |
| LLaVA-NeXT-13B (Liu et al., ) | 73.6 | 70.0 | 62.2 | 72.2 | 86.7 | - | 1892 | 35.1 | 70.0 | 68.5 | 47.3 | 72.3 |
| MM1-7B (McKinzie et al., ) | 72.6 | - | - | 69.9 | 86.6 | - | 1858 | 35.9 | 72.3 | - | 42.1 | - |
| MM1-MoE-7B$\times$32 (McKinzie et al., ) | 74.4 | - | - | 70.9 | 87.8 | - | 1992 | 40.9 | 72.7 | - | 45.2 | - |
| MiniGemini-HD-7B (Li et al., ) | - | - | - | - | - | - | 1865 | 32.2 | 65.8 | - | 41.3 | - |
| MiniGemini-HD-13B (Li et al., ) | - | - | - | - | - | - | 1917 | 37.0 | 68.6 | - | 50.5 | - |
| Cambrian-1-8B (Tong et al., ) | 80.4 | 73.0 | 73.3 | 74.7 | - | - | - | 49.0 | 75.9 | - | - | - |
| Cambrian-1-13B (Tong et al., ) | 79.3 | 73.6 | 73.8 | 74.4 | - | - | - | 48.0 | 75.7 | - | - | - |
| Eagle-8B (Shi et al., ) | 84.3 | 76.1 | 80.1 | 76.3 | - | - | - | 52.7 | 75.9 | - | - | - |
| Eagle-13B (Shi et al., ) | 82.0 | 74.0 | 77.6 | 74.8 | - | - | - | 54.4 | 75.7 | - | - | - |
| VILA1.5-8B (Lin et al., ) | 82.0 | - | - | 73.8 | 85.6 | - | - | - | 75.3 | 69.9 | 43.2 | 71.9 |
| VILA1.5-13B (Lin et al., ) | 80.1 | - | - | 72.6 | 86.3 | - | - | - | 74.9 | 66.3 | 44.3 | 80.8 |
| VILA${}^{2}$-8B (Fang et al., ) | 87.6 | - | - | 66.1 | 86.7 | - | - | - | 76.6 | 71.7 | 50.0 | 86.6 |
| MiniCPM-V-2.5-8B (Yao et al., ) | - | - | - | - | - | - | 2025 | 54.3 | 77.2 | 74.2 | - | 86.7 |
| CogVLM2-8B (Hong et al., ) | - | 73.4 | 81.0 | - | - | - | 1870 | - | 80.5 | - | 60.4 | - |
| TroL-7B (Lee et al., ) | 92.8 | 78.5 | 71.2 | 75.3 | 87.8 | 65.3 | 2308 | 51.8 | 83.5 | 81.2 | 54.7 | 92.8 |
| LLaVA-OneVision-8B (Li et al., ) | 96.0 | 81.4 | 80.0 | 75.4 | - | - | 1998 | 63.2 | 80.8 | - | 57.5 | 90.7 |
| Phantom-7B | 95.6 | 79.5 | 87.8 | 75.3 | 87.7 | 65.4 | 2126 | 70.9 | 84.8 | 84.7 | 70.8 | 84.9 |

### Implementation of Phantom Optimization.

To fully leverage the enhanced learning capability provided by Phantom Dimension, we introduce Phantom Optimization (PO), which is heavily inspired by Direct Preference Optimization (DPO) (Rafailov et al., ). While methods such as RLHF (Christiano et al., ) and DPO are designed to optimize towards human or AI-driven preferences, PO is tailored to follow correct answer and reduce incorrect and ambiguous answers during training. To reduce the computational complexity of incorporating an additional reference model, we adopt the loss formulation from SimPO (Meng et al., ). Similar to ORPO (Hong et al., ), we simultaneously use autoregressive supervised fine-tuning (SFT). This enables Phantom to effectively reinforce correct answers $y^{+}$ while eliminating incorrect and ambiguous ones $y^{-}$ in response to a given prompt $x$. This formulation can be expressed as follows:

$$ $\min_{\theta}\mathcal{L}_{\text{PO}}=\mathcal{L}_{\text{SFT}}-\mathbb{E}_{\mathcal{D}}\left[\log\sigma\left(\frac{\beta}{|y^{+}|}\log{\pi_{\theta}(y^{+}|x)}-\frac{\beta}{|y^{-}|}\log{\pi_{\theta}(y^{-}|x)}-\gamma\right)\right],$ (4) $$

where $\theta$ represents the trainable parameters and $\mathcal{L}_{\text{SFT}}$ denotes the supervised fine-tuning loss for question-answer pairs. We implement a two-step training strategy. In the first step, which focuses on vision and language alignment, the parameters of the pretrained LLM are frozen. We then train the parameters of vision projector and the components related to Phantom Dimension (MHCA and the functions $f$ and $g$). In the second step, we unfreeze all parameters and train them all at once. We apply PO throughout the first training step only, not to interrupt multimodal LLM’s own text generation ability because the positive and negative answers $y^{+}/y^{-}$ are mostly generated by closed-source LLVMs instead of instruction fine-tuned self model, which is totally different strategy from RLHF and DPO. For verification, we show the performance degradation in experiment section when using PO in the second training step.

## 4 Experiments

**Table 2: Comparison with the current existing smaller open-source LLVMs across 0.5B$\sim$4B model sizes, evaluating vision-language performances of Phantom on numerous evaluation benchmarks equally used in Table [1](#S3.T1).**
| LLVMs | SQA${}^{\text{I}}$ | AI2D | ChartQA | SEED${}^{\text{I}}$ | POPE | HallB | MME | MathVista | MMB | MMB${}^{\text{CN}}$ | MM-Vet | LLaVA${}^{\text{W}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MobileVLM-3B (Chu et al., ) | 61.2 | - | - | - | 84.9 | - | - | - | 59.6 | - | - | - |
| MobileVLM-V2-3B (Chu et al., ) | 70.0 | - | - | - | 84.7 | - | - | - | 63.2 | - | - | - |
| MoE-LLaVA-2.7B$\times$4 (Lin et al., ) | 70.3 | - | - | - | 85.7 | - | - | - | 68.0 | - | 35.9 | - |
| LLaVA-Phi-2.7B (Zhu et al., ) | 68.4 | - | - | - | 85.0 | - | - | - | 59.8 | - | 28.9 | - |
| Imp-v1-3B (Shao et al., ) | 70.0 | - | - | - | 88.0 | - | - | - | 66.5 | - | 33.1 | - |
| TinyLLaVA-3.1B (Zhou et al., ) | 69.1 | - | - | - | 86.4 | - | - | - | 66.9 | - | 32.0 | - |
| TinyLLaVA-Sig-Phi-3.1B (Zhou et al., ) | 69.1 | - | - | - | 86.4 | - | - | - | 66.9 | - | 32.0 | - |
| Bunny-3B (He et al., ) | 70.9 | 38.2 | - | 62.5 | 86.8 | - | 1778 | - | 68.6 | - | - | - |
| MiniCPM-2.4B (Hu et al., ) | - | 56.3 | - | - | - | - | 1650 | 28.9 | 64.1 | 62.6 | 31.1 | - |
| MiniCPM-V2-2.8B (Hu et al., ) | - | 62.9 | - | - | - | - | 1809 | 38.7 | 69.1 | 66.5 | 41.0 | - |
| MM1-3B (McKinzie et al., ) | 69.4 | - | - | 68.8 | 87.4 | - | 1762 | 32.0 | 67.8 | - | 43.7 | - |
| MM1-MoE-3B$\times$64 (McKinzie et al., ) | 76.1 | - | - | 69.4 | 87.6 | - | 1773 | 32.6 | 70.8 | - | 42.2 | - |
| ALLaVA-3B (Chen et al., ) | - | - | - | 65.2 | - | - | 1623 | - | 64.0 | - | 32.2 | - |
| ALLaVA-3B-Longer (Chen et al., ) | - | - | - | 65.6 | - | - | 1564 | - | 64.6 | - | 35.5 | - |
| VILA1.5-3B (Chen et al., ) | 69.6 | - | - | 66.4 | 85.3 | - | - | - | 62.8 | 52.2 | 38.6 | 76.7 |
| TroL-3.8B (Lee et al., ) | 90.8 | 73.6 | 73.8 | 70.5 | 86.5 | 62.2 | 1980 | 55.1 | 79.2 | 77.1 | 51.1 | 76.6 |
| Phantom-3.8B | 94.2 | 71.7 | 87.3 | 72.8 | 87.1 | 60.8 | 2046 | 60.6 | 80.4 | 77.1 | 54.4 | 76.2 |
| DeepSeek-VL-1.3B (Lu et al., ) | - | - | - | 66.7 | 87.6 | - | - | 31.1 | 64.6 | 62.9 | 34.8 | - |
| MobileVLM-1.7B (Chu et al., ) | 57.3 | - | - | - | 84.5 | - | - | - | 53.2 | - | - | - |
| MobileVLM-V2-1.7B (Chu et al., ) | 66.7 | - | - | - | 84.3 | - | - | - | 57.7 | - | - | - |
| MoE-LLaVA-1.8B$\times$4 (Lin et al., ) | 63.1 | - | - | - | 87.0 | - | - | - | 59.7 | - | 25.3 | - |
| Mini-Gemini-2B (Li et al., ) | - | - | - | - | - | - | 1653 | 29.4 | 59.8 | - | - | - |
| TroL-1.8B (Lee et al., ) | 87.5 | 68.9 | 64.0 | 69.0 | 88.6 | 60.1 | 2038 | 45.4 | 76.1 | 74.1 | 45.1 | 69.7 |
| Phantom-1.8B | 91.9 | 62.3 | 87.0 | 68.6 | 89.6 | 62.2 | 1885 | 60.9 | 76.6 | 75.1 | 54.1 | 68.6 |
| LLaVA-OneVision-0.5B (Li et al., ) | 67.2 | 57.1 | 61.4 | 65.5 | - | - | 1478 | 34.8 | 52.1 | - | 29.1 | 74.2 |
| Phantom-0.5B | 83.2 | 54.1 | 78.0 | 60.6 | 86.0 | 54.6 | 1743 | 51.7 | 72.7 | 70.1 | 45.7 | 69.6 |

### Implementation Details.

To ensure successful reproducibility, we outline four key technical aspects of Phantom: (a) the detailed architecture of the backbone multimodal LLMs, vision encoder, and vision projector, (b) the structure of the multi-head cross-attention (MHCA) module in Phantom Dimension, (c) the computing environments and bit quantization configurations, and (d) the procedures for training and inference.

### (a)

We utilize Qwen2 (Yang et al., ), Phi-3-mini (Abdin et al., ), and InternLM2/2.5 (Cai et al., ) as the backbone multimodal LLMs. Specifically, Qwen2-0.5B is configured with $h_{q}=14$, $h_{kv}=2$, a hidden dimension of $d_{q}=896$, and 24 layers; InternLM2-1.8B with $h_{q}=16$, $h_{kv}=8$, a hidden dimension of $d_{q}=2048$, and 24 layers; Phi-3-mini-3.8B with $h_{q}=32$, $h_{kv}=32$, a hidden dimension of $d_{q}=3072$, and 32 layers; and InternLM2.5-7B with $h_{q}=32$, $h_{kv}=8$, a hidden dimension of $d_{q}=4096$, and 32 layers. For the vision encoder, we employ InternViT-300M (Chen et al., ), which has a hidden dimension of 1024 and 24 layers. The vision projector is designed as MLP that adjusts the hidden dimension from 1024 to match the corresponding multimodal LLM’s latent hidden dimension.

### (b)

In each layer, MHCA consists of four linear modules for the query, key, value, and output of the multi-head self-attention operation, where MHCA has similar head dimension for MHSA. For the 0.5B model, the number of parameters required for MHCA module is approximately 1.2M, calculated as $(\frac{896\text{ (hidden dimension)}}{14\text{ (number of heads)}})^{2}\times 4\text{ (linear modules)}\times 24\text{ (layers)}\times 3\text{ (}qkv\text{)}$. Similarly, the required parameters for the 1.8B, 3.8B, and 7B models are 4.8M, 3.7M, and 6.2M, respectively. These additional parameters do not significantly impact the overall model size compared with 0.5B, 1.8B, 3.8B, and 7B. Note that, the regularization parameter $\lambda$ during MHSA is set to $\sqrt{2}$.

**Table 3: Detailed comparison for challenging evaluation benchmarks. Sub-benchmark category names in (c), (d), and (g) are represented in Appendix [B](#A2). For (f), LLaVA-Wilder (Zhang et al., ) is a more advanced challenging evaluation benchmark over LLaVA${}^{\text{W}}$ (Liu et al., ).**
| Benchmarks | OmniFusion-7B | DeepSeek-VL-7B | MoVA-7B | Eagle-8B | CoLLaVO-7B | MoAI-7B | Meteor-7B | Phantom-7B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SQA${}^{\text{I}}$ (Lu et al., ) | 69.7 | 57.7 | 74.4 | 84.3 | 80.7 | 83.5 | 87.5 | 95.6 |
| \cdashline1-9<br>MMB (Liu et al., ) | 69.0 | 73.2 | 81.3 | 75.9 | 83.0 | 79.3 | 82.9 | 84.8 |
| \cdashline1-9<br>MM-Vet (Yu et al., ) | 39.4 | 41.5 | - | - | 40.3 | 43.7 | 57.3 | 70.8 |
| \cdashline1-9<br>MathVista (Lu et al., ) | - | - | 44.3 | 52.7 | 57.6 | 56.2 | 53.4 | 70.9 |
| \cdashline1-9<br>MMStar (Chen et al., ) | - | - | - | - | 42.1 | 48.7 | 45.5 | 57.7 |

### (c)

In a computing environment utilizing 8$\times$NVIDIA RTX A6000 48GB GPUs and 8$\times$NVIDIA RTX 3090 24GB GPUs, Phantom’s training and inference processes take place. To conduct efficient training, each step undergoes a single epoch of training using 8-bit quantization and bfloat16 data format (Kalamkar et al., ) for every backbone multimodal LLM. Following bit quantization, we apply QLoRA (Hu et al., ; Dettmers et al., ) to both vision encoders and backbone multimodal LLMs across all linear layers, using 256 rank and 256 alpha parameters.

### (d)

For Phantom Optimization, we choose equal hyperparameters used in SimPO (Meng et al., ): $\beta=2$ and $\gamma=0.5$. For training, AdamW optimizer (Loshchilov & Hutter, ) is applied, and cosine annealing adjusts the learning rate from 1e-5 to 1e-6 throughout each training step. For multimodal LLM, gradient checkpointing (Sohoni et al., ) is employed to manage memory efficiently. A gradient accumulation of 4 leads to batch sizes totaling 128 for each training step, with each step taking roughly two to five days depending on model size. For inference efficiency, Phantom is validated using the same quantization level in training, and we make Phantom Dimension cache: $Q^{*}_{l}$, $K^{*}_{l}$, and $V^{*}_{l}$ in each layer to get speedy inference like kv-cache technique, where we use deterministic beam search (Freitag & Al-Onaizan, ) ($n=3$). Memory-efficient scaled dot product attention (SDPA) and FlashAttention2 (Dao et al., ; Dao, ) accelerates multi-head self-attention (MHSA) computation for Phantom Dimension, benefiting from its hardware-aware ability to mitigate the overhead from the increased latent hidden dimension.

**Table 4: Identifying the effectiveness of Phantom by controlling the three factors: Weighted-Average (WA) operation, Phantom Dimension (PD), and Phantom Optimization (PO). If we do not use WA, we then use simple element-wise summation or averaging. In this case, we pick the better performances. Note that, PO-Step1 and -Step2 mean PO is applied in Step1 or Step2.**
|  | WA | PD | PO-Step1 | PO-Step2 | CV-Bench | BLINK | MMB | SEED-Bench-2-Plus | VisualWebBench | MM-Vet | MM-Vet-v2 | LLaVA-Wilder | MathVista |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Phantom-0.5B | ✗ | ✗ | ✗ | ✗ | 28.2 | 21.4 | 60.7 | 35.7 | 34.7 | 26.6 | 22.0 | 60.8 | 33.8 |
| ✗ | ✓ | ✗ | ✗ | 29.8 | 21.9 | 62.4 | 39.9 | 37.2 | 27.4 | 22.3 | 64.9 | 36.7 |  |
| ✓ | ✓ | ✗ | ✗ | 38.1 | 27.4 | 70.1 | 43.7 | 42.3 | 31.8 | 29.7 | 69.7 | 40.0 |  |
| \cdashline2-14 | ✓ | ✓ | ✓ | ✗ | 41.5 | 39.3 | 72.7 | 47.9 | 51.8 | 45.7 | 41.5 | 72.2 | 51.7 |
|  | ✓ | ✓ | ✗ | ✓ | 36.2 | 36.7 | 68.8 | 40.4 | 47.1 | 39.9 | 36.6 | 67.4 | 48.2 |
|  | ✓ | ✓ | ✓ | ✓ | 38.5 | 38.0 | 69.1 | 45.5 | 47.2 | 42.3 | 36.2 | 71.0 | 47.3 |
| \cdashline2-14 | ✗ | ✗ | ✓ | ✗ | 32.0 | 24.2 | 64.2 | 39.0 | 36.4 | 31.2 | 24.1 | 63.4 | 36.8 |
| Phantom-1.8B | ✗ | ✗ | ✗ | ✗ | 44.7 | 28.9 | 60.2 | 43.3 | 45.4 | 35.1 | 26.1 | 63.2 | 42.3 |
| ✗ | ✓ | ✗ | ✗ | 47.0 | 32.6 | 64.7 | 44.9 | 46.5 | 36.0 | 27.4 | 68.7 | 46.4 |  |
| ✓ | ✓ | ✗ | ✗ | 52.6 | 35.2 | 69.8 | 50.0 | 53.5 | 41.8 | 32.5 | 71.1 | 49.1 |  |
| \cdashline2-14 | ✓ | ✓ | ✓ | ✗ | 63.1 | 44.2 | 76.6 | 57.1 | 55.9 | 54.1 | 46.3 | 78.5 | 60.9 |
|  | ✓ | ✓ | ✗ | ✓ | 59.9 | 39.9 | 72.2 | 49.7 | 48.4 | 50.5 | 37.0 | 77.1 | 55.9 |
|  | ✓ | ✓ | ✓ | ✓ | 59.6 | 40.6 | 73.7 | 54.5 | 55.2 | 53.3 | 41.7 | 76.0 | 58.8 |
| \cdashline2-14 | ✗ | ✗ | ✓ | ✗ | 48.2 | 30.7 | 61.2 | 47.0 | 49.7 | 37.1 | 29.5 | 68.2 | 44.4 |
| Phantom-3.8B | ✗ | ✗ | ✗ | ✗ | 63.7 | 34.4 | 62.6 | 42.9 | 45.6 | 38.1 | 32.6 | 73.5 | 45.3 |
| ✗ | ✓ | ✗ | ✗ | 66.6 | 37.9 | 65.9 | 44.5 | 46.1 | 40.9 | 34.1 | 78.0 | 49.8 |  |
| ✓ | ✓ | ✗ | ✗ | 69.1 | 44.1 | 68.9 | 51.8 | 51.8 | 46.9 | 37.0 | 83.4 | 50.5 |  |
| \cdashline2-14 | ✓ | ✓ | ✓ | ✗ | 73.8 | 51.5 | 80.4 | 61.8 | 59.8 | 54.4 | 48.5 | 85.7 | 60.6 |
|  | ✓ | ✓ | ✗ | ✓ | 67.9 | 45.9 | 76.2 | 54.4 | 56.8 | 50.6 | 42.0 | 84.5 | 53.7 |
|  | ✓ | ✓ | ✓ | ✓ | 69.2 | 47.8 | 79.2 | 58.6 | 54.8 | 49.9 | 42.9 | 85.0 | 58.1 |
| \cdashline2-14 | ✗ | ✗ | ✓ | ✗ | 68.6 | 37.6 | 65.5 | 47.0 | 46.8 | 41.3 | 36.3 | 76.1 | 49.7 |
| Phantom-7B | ✗ | ✗ | ✗ | ✗ | 59.1 | 41.9 | 71.9 | 50.2 | 51.9 | 50.2 | 44.2 | 69.5 | 56.2 |
| ✗ | ✓ | ✗ | ✗ | 59.8 | 45.9 | 72.5 | 54.2 | 53.6 | 53.7 | 46.3 | 74.4 | 60.9 |  |
| ✓ | ✓ | ✗ | ✗ | 69.0 | 47.7 | 81.7 | 59.3 | 57.1 | 62.1 | 53.2 | 77.2 | 64.5 |  |
| \cdashline2-14 | ✓ | ✓ | ✓ | ✗ | 74.9 | 58.9 | 84.8 | 65.5 | 67.7 | 70.8 | 60.6 | 82.9 | 70.9 |
|  | ✓ | ✓ | ✗ | ✓ | 71.7 | 52.2 | 77.9 | 59.1 | 64.1 | 68.2 | 53.1 | 78.7 | 68.1 |
|  | ✓ | ✓ | ✓ | ✓ | 70.8 | 54.4 | 82.9 | 60.5 | 66.6 | 69.0 | 54.5 | 82.6 | 68.3 |
| \cdashline2-14 | ✗ | ✗ | ✓ | ✗ | 61.1 | 43.0 | 75.9 | 52.4 | 53.4 | 54.8 | 47.1 | 73.3 | 59.3 |

### Validation and Ablation Studies.

We present an overview of Phantom’s vision-language performance in Figure [1](#S1.F1)-[2](#S1.F2), and evaluate it on generally used standard evaluation benchmarks as shown in Table [1](#S3.T1)-[2](#S4.T2). In the table, LLaVA-OneVision-8B (Li et al., ) uses significant number of image tokens up to 7290 with three training steps on 558K+4M+3.2M datasets. To highlight the benefits of Phantom, Table [4](#S4.SS0.SSS0.Px3) reports performance on more challenging evaluation benchmarks. These results demonstrate that Phantom offers a significant advantage on tasks requiring reasoning abilities and densely learned knowledge. Descriptions of the evaluation benchmarks can be found in Appendix [B](#A2), and Phantom’s text generation quality is illuminated in Appendix [C](#A3). In conclusion, Phantom achieves outstanding performance across numerous vision-language tasks, with a large margin over competing LLVMs, despite having a smaller model size and fewer instruction tuning samples. To better understand the source of this effectiveness, Table [4](#S4.SS0.SSS0.Px5) presents an ablation study focusing on three key factors: (a) Weighted-Average (WA), (b) Phantom Dimension (PD), and (c) Phantom Optimization (PO). The results reveal several insights: (1) PD significantly enhances vision-language performance, as increasing the latent hidden dimension improves the embedding of vision-language knowledge; (2) WA is more effective than simple summation or averaging for compressing output features; and (3) PO yields greater performance gains when combined with PD and when applied only during the first training step with a frozen pretrained LLM. Besides, we investigated the effect of replacing the sos token with alternative tokens. We observed using the token that appears earlier in the user question prompt, before the question, is more effective. Regarding inference speed, we measured computation time and found only a marginal 10% difference in tokens-per-second between the settings with and without PD. It is definitely attributed to hardware-level computed operation using SDPA and FlashAttention2 (Dao et al., ; Dao, ).

### Discussion and Limitation.

The development of high-performing LLVMs increasingly depends on combining diverse models (Lu et al., ; Lee et al., ; ; ; Zong et al., ; Shi et al., ) and refining existing architectures (Liu et al., ; Lee et al., ), as many aspects of these systems remain unexplored. However, such structural modifications often leads to substantial low-level programming when addressing both development and production-level demands. In response, we will do comprehensive exploration of significantly larger open-source LLVMs, without additional architectural changes. Although there has been a growing trend toward open-source LLVMs, much of the research continues to focus on closed-source LLVMs such as GPT-4V and Gemini-Pro. We either had used GPT-4o-mini and GPT-4o. Therefore, we believe there is untapped potential not only in utilizing the textual outputs of larger open-source LLVMs but also in accessing deeper insights, such as layer-wise features or full parameter sets across layers. Moving forward, we plan to investigate layer-wise distillation methods, which go beyond traditional distillation, to transfer knowledge into models with entirely different architectures using human-understandable language. This direction promises to open up exciting possibilities in a more easier way to develop efficient LLVMs, such as transferring knowledge across heterogeneous structures.

## 5 Conclusion

We present an efficient LLVM family Phantom with significantly enhanced learning capabilities within limited model sizes. By introducing Phantom Optimization (PO) that leverages both autoregressive supervised fine-tuning (SFT) and DPO-like concept, it effectively learns and boosts vision-language performances. Remarkably, despite being smaller than many high-performing LLVMs with larger model sizes, Phantom demonstrates comparable or even superior performance, making it a promising solution for resource-constrained environments. Our results underscore the power of latent space optimization in boosting both efficiency and performance, offering a pathway toward more efficient LLVMs for various applications.

## Appendix A Phantom Triples

[Uncaptioned image]: /html/2409.14713/assets/x46.png

[Uncaptioned image]: /html/2409.14713/assets/x47.png

[Uncaptioned image]: /html/2409.14713/assets/x48.png

[Uncaptioned image]: /html/2409.14713/assets/x49.png

[Uncaptioned image]: /html/2409.14713/assets/x50.png

[Uncaptioned image]: /html/2409.14713/assets/x51.png

[Uncaptioned image]: /html/2409.14713/assets/x52.png

## Appendix B Description of Evaluation Benchmarks

- •
SQA-IMG (SQA${}^{\text{I}}$) (Lu et al., ) is part of the broader ScienceQA (SQA) dataset, which aims to improve reasoning and interpretability in AI systems through science-based question answering. This dataset covers a wide range of science disciplines, featuring 26 different topics in natural, social, and language sciences, all accompanied by annotated answers, lectures, and explanations. SQA-IMG includes image-related samples, amounting to 10,332 question-answer pairs.
- •
AI2D (Kembhavi et al., ) or AI2 Diagrams, addresses diagram interpretation and reasoning challenges, focusing on syntactic parsing and semantic understanding. It supports research into diagram structure and element relationships, critical for tasks like diagram-based question answering. This collection includes over 5,000 diagrams from elementary science topics, along with over 15,000 multiple-choice questions.
- •
ChartQA (Masry et al., ) develops to challenge and improve question answering systems that deal with data visualizations like bar charts, line charts, and pie charts. This dataset tests systems on questions requiring arithmetic and logical reasoning and includes both human-generated and machine-created question-answer pairs. It comprises 32,719 samples in total.
- •
SEED-IMG (SEED${}^{\text{I}}$) (Li et al., ), a subset of SEED-Bench, evaluates the generative comprehension skills of multimodal large language models (MLLMs) with a focus on spatial and temporal understanding. It offers several subsets mapped to 12 evaluation dimensions across image and video modalities, with SEED-IMG specifically concentrating on images.
- •
SEED-Bench-2-Plus (Li et al., ) evaluates multimodal large language models in their ability to understand text-rich visual content, common in real-world settings like charts, maps, and website interfaces. This benchmark specifically measures how effectively MLLMs can interpret these complex, text-rich scenarios that require simultaneous comprehension of visual and textual information. The benchmark is divided into three main categories—Charts, Maps, and Webs, and further subdivided into 63 unique data types with 2.3k multiple-choice questions.
- •
POPE (Li et al., ) introduces a method to systematically assess the tendency of LLVMs to falsely generate nonexistent objects in images. This method turns the evaluation into a binary classification task using polling questions, providing a fair and adaptable approach.
- •
HallusionBench (HallB) (Liu et al., ) is crafted to evaluate and explore visual illusions and knowledge hallucinations in large language and vision models (LLVMs). This benchmark uses carefully crafted example pairs to identify model failures, featuring diverse visual-question pairs including subsets focused on illusions, math, charts, tables, maps, and OCR. It includes 346 images and 1,129 questions.
- •
MME (Fu et al., ) serves as a comprehensive evaluation framework for Multimodal Large Language Models (MLLMs), focusing on various perception and cognition tasks through 14 sub-tasks like coarse and fine-grained recognition, OCR, and commonsense reasoning. This benchmark aims to address existing evaluation gaps and ensures a thorough testing environment for MLLMs.
- •
MathVista (Lu et al., ) is an extensive benchmark designed to test visual-based mathematical reasoning in AI models. It integrates visual understanding in evaluating models’ abilities to solve math problems that involve visuals. The dataset consists of three subsets: IQTest, FunctionQA, and PaperQA, totaling 6,141 examples.
- •
MMB, MMB-Chinese (MMB${}^{\text{CN}}$) (Liu et al., ) aims to establish a robust evaluation standard for vision language models by covering a broad spectrum of necessary multimodal comprehension skills (20 fine-grained abilities) in both English and Chinese. This benchmark consists of 3,217 questions gathered from various sources to challenge different facets of LLVMs.
- •
MM-Vet (Yu et al., ) is designed to systematically evaluate LMMs on complex tasks requiring multiple vision language (VL) capabilities. It tests recognition, knowledge, OCR, spatial awareness, language generation, and math, integrating these abilities into 16 different task combinations. The dataset includes 200 images and 218 questions, each requiring the integration of multiple capabilities.
- •
MM-Vet-v2 (Yu et al., ) evaluates a wide range of integrated abilities in large multimodal models, such as Recognition, Knowledge, Optical Character Recognition (OCR), Spatial Awareness, Language Generation, Math, and Image-Text Sequence Understanding. This version builds upon the original MM-Vet benchmark by adding tasks that involve comprehending sequential information from both images and text, which is essential for real-world scenarios. MM-Vet-v2 places a strong focus on assessing the model’s capacity to interpret and reason through intricate image-text sequences. The benchmark includes 517 evaluation samples, a notable increase from the 217 samples in the original MM-Vet.
- •
LLaVA Bench in the Wild(er) (LLaVA${}^{\text{W}}$ and LLaVA-Wilder ) (Liu et al., ; Zhang et al., ) assesses large multimodal models (LMM) on complex tasks and new domains through a collection of 24 images with 60 questions for ‘wild’ and its more advanced version of ‘wilder’. This dataset features diverse settings, including indoor, outdoor, artworks, and memes, with each image accompanied by detailed descriptions and curated questions.
- •
MMStar (Chen et al., ) is crafted to precisely evaluate the true multimodal capabilities of LLVMs by ensuring that each sample critically relies on visual content for accurate answers while minimizing data leakage. It comprises 1,500 meticulously selected samples and is organized into six primary sub-benchmarks as follows:
–
Coarse perception (CP), which pertains to the ability to grasp and interpret the overarching features and themes of an image without focusing on minute details,
–
Fine-grained perception (FP), which denotes a detailed level of image comprehension that emphasizes the intricate and nuanced aspects of visual content,
–
Instance reasoning (IR), which encompasses advanced cognitive abilities aimed at understanding and interpreting individual and collective object attributes and their interrelations within an image,
–
Logical reasoning (LR), which involves a sophisticated framework of cognitive processes designed to interpret, deduce, and infer conclusions from visual content through a structured approach to logic and reasoning,
–
Science & technology (ST), which includes a comprehensive framework for the application and integration of knowledge across a wide range of scientific and technological domains,
–
Math (MA), which is a fundamental pillar of logical and analytical reasoning and includes a broad spectrum of skills essential for understanding, applying, and interpreting quantitative and spatial information.
- •
MathVerse (Zhang et al., ) assesses the capabilities of Multi-modal Large Language Models (MLLMs) in visual mathematical reasoning, particularly their ability to understand visual diagrams and mathematical expressions. This dataset is categorized into three primary areas: plane geometry, solid geometry, and functions, and further detailed into twelve types like length and area, encompassing 2,612 visual mathematical challenges.
To investigate how MLLMs process visual diagrams in mathematical reasoning, the creators of MathVerse developed six distinct versions of each problem, each version presenting different levels of multi-modal information. They initially established three specific classifications for the text content within the problems:
–
Descriptive Information, which includes content that is directly visible and explicitly depicted in the diagrams,
–
Implicit Property, which encompasses details that demand a more advanced visual perception yet less mathematical knowledge to interpret from the diagram,
–
Essential Condition, which pertains to crucial numerical or algebraic data necessary for solving the problem that cannot be inferred solely from the visual diagram.
Based on these categories, to thoroughly assess the true visual understanding capabilities of MLLMs and their utility in multi-modal mathematical contexts, the researchers created six versions or sub-benchmarks of each problem in MathVerse, described as follows:
–
Text dominant (TD) version, which preserves all textual elements, including the three textual categories and the main question, prompting MLLMs to primarily depend on textual information.
–
Text lite (TL) version reduces the Descriptive Information from the Text dominant version, promoting reliance on the diagram for elementary data.
–
Text only (TO) version removes the visual elements entirely, focusing on textual content to discern where MLLMs predominantly derive contextual information for problem solving.
–
Vision intensive (VI) further excludes Implicit Property from the Text lite version, urging MLLMs to intensify their visual analysis to gather essential cues for mathematical reasoning.
–
Vision dominant (VD), evolving from the Text lite version, omits Essential Condition from the textual information and instead visually annotates these details in diagrams, compelling MLLMs to identify and accurately link these essential conditions solely through visual examination.
–
Vision only (VO) eliminates all textual descriptions, presenting the problem exclusively through visual means and challenging MLLMs to decode and identify mathematical queries purely based on visual data, serving as the ultimate test of their visual reasoning skills in mathematics.
- •
VisualWebBench (Liu et al., ) assesses the capabilities of multimodal large language models (MLLMs) specifically in the web domain. It is designed to address the lack of a comprehensive benchmark that evaluates the unique characteristics of web pages and measures fine-grained abilities such as OCR, understanding, and grounding (Grd) in text-rich and interactive web environments. It covers a wide range of domains, including science, travel, sports, engineering, and government, and tasks such as captioning (Cap), WebQA (QA), heading OCR, element grounding (Grd), and action prediction (Pred), containing a total of 1,534 instances.
- •
CV-Bench (Tong et al., ) is designed for vision-focused evaluation in multimodal large language models. This benchmark aims to fill the gaps in traditional benchmarks, which often fall short in thoroughly assessing visual grounding in real-world contexts. CV-Bench assesses the model’s abilities in both 2D and 3D visual tasks using natural language questions. The evaluation is split into 2D tasks (such as spatial relationships and object counting) and 3D tasks (like depth order and relative distance), providing a well-rounded test of the model’s visual comprehension with 2,638 carefully inspected examples.
- •
BLINK (Fu et al., ) is created to assess the visual perception capabilities of multimodal large language models. It features 14 key visual perception tasks, which are based on traditional computer vision challenges but restructured into 3,807 multiple-choice questions that involve one or more images. These tasks address difficulties such as relative depth estimation, visual correspondence, forensic detection, and multi-view reasoning.

Additionally, we will continue to explore more challenging evaluation benchmarks to uncover previously unaddressed issues such as Yu et al. (), advancing Phantom through ongoing technical development. By leveraging a wide range of methods established over the years (Lee, ; Lee et al., ; Kim et al., ; Lee et al., ; Kim et al., ; Lee et al., ; Kim et al., ; ; Park et al., ; ; Kim et al., ), we aim to drive innovative breakthroughs across both general and specialized tasks.

## Appendix C Phantom Generation Quality

[Uncaptioned image]: /html/2409.14713/assets/x56.png

[Uncaptioned image]: /html/2409.14713/assets/x57.png

[Uncaptioned image]: /html/2409.14713/assets/x58.png

[Uncaptioned image]: /html/2409.14713/assets/x59.png

[Uncaptioned image]: /html/2409.14713/assets/x60.png

[Uncaptioned image]: /html/2409.14713/assets/x61.png

[Uncaptioned image]: /html/2409.14713/assets/x62.png

[Uncaptioned image]: /html/2409.14713/assets/x63.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why do you use sequence (sos) token to enhance the latent hidden dimension?

2. Is the MHSA-based method specific for MLLMs? If not, why don't you consider using it in LLMs first?

3. Why are there so many blanks in Table 1, 2, 3?

4. Why don't you conduct experiments based on open-source datasets and models to ensure fair comparison?
soundness: 2
strengths: The authors present an efficient LLVM family Phantom with enhanced learning capabilities within limited model sizes. They introduce Phantom Optimization (PO) that seems interesting. Phantom demonstrates good performance in their evaluations.
summary: The authors propose a method that can temporarily increase the latent hidden dimension during multi-head self-attention (MHSA), without substantially increasing physical model sizes. The authors also introduce Phantom Optimization (PO) using both autoregressive supervised fine-tuning (SFT) and direct preference optimization (DPO)-like concept. Using these methods, they present a new efficient LLVM family with different model sizes with good performance.
weaknesses: 1. Authors only compare with other open-sourced Multimodal LLMs (MLLMs) using their checkpoints. They lack comparisons with related baseline methods using the same pre-trained models, datasets, and training configurations. There are several baseline methods [1,2,3,4,5] that contribute to the training algorithms of MLLMs. Besides, there are also huge amounts of works talking about how to modify the attention mechanism in Transformer models that need to be discussed.


2. It seems that the authors have not conducted experiments to compare Phantom Optimization with other RLHF methods (such as DPO, SimPO, ORPO).

3. It's not a good habit to use benchmark (evaluation) dataset for training, e.g., the MathVision benchmark.


[1] Jie, Shibo, et al. "Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning." arXiv preprint arXiv:2405.05615 (2024).

[2] Luo, Gen, et al. "Cheap and quick: Efficient vision-language instruction tuning for large language models." Advances in Neural Information Processing Systems 36 (2024).

[3] Gao, Peng, et al. "Llama-adapter v2: Parameter-efficient visual instruction model." arXiv preprint arXiv:2304.15010 (2023).

[4] Wang, Weihan, et al. "Cogvlm: Visual expert for pretrained language models." arXiv preprint arXiv:2311.03079 (2023).

[5] Jia, Ding, et al. "GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer." arXiv preprint arXiv:2406.01210 (2024).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See the questions raised in Weakness above.
soundness: 2
strengths: **Comprehensive evaluation**  
The authors evaluate their VLM architecture + dataset + training strategy on a variety of vision-language benchmarks, and show good performance compared to various baselines (both open and closed models) 

**Method ablations**  
I appreciated how the authors ablate several components of their proposed contribution and report performance over multiple tasks to study how these components add to the overall performance: (1) the weighted-average mechanism for combining self- and cross-attention outputs, (2) the concatenation of self- and cross-attention features in their Phantom dimension, and (3) the use of an SimPO optimization over their training (Phantom Optimization).
summary: The paper introduces Phantom, a new family of efficient large language and vision models (LLVMs) designed to improve performance within limited model sizes. To do so, the authors (1) curate a vision-language instruction dataset featuring both correct and incorrect / ambiguous answers, (2) propose a new attention scheme that computes attention with larger query, key, and value dimensions by concatenating the outputs of a first round of attention within query, key, and value projections to the queries, keys, and values, before performing attention once again on these concatenated features (Phantom Dimension), and (3) use a SimPO training objective over their curated dataset while training in a parameter-efficient scheme (Phantom Optimization). They show across a wide range of vision-language tasks that Phantom models are able to achieve competitive or state-of-the-art quality against various closed and open-source VLMs, and provide additional ablations to study the contribution of different Phantom components.
weaknesses: **Method motivation / understanding**  
While the authors say that increased feature dimension size in the attention interactions is important for improved quality (“to make LLVMs embed much more vision-language knowledge”; L244-245), I still don’t see the motivation for concatenating the cross-attention and self-attention projections. To increase dimensionality while preserving parameter-count, we could also arbitrarily repeat or expand the features from `head_dimension` to `2 * head_dimension`. It would be good to see some kind of ablation or more careful study motivating the concatenation of these features. 

**Method clarity**  
Some details and justification about the method are left out. For example, why use the softmax terms in Eq. 3 to mix between the outputs as opposed to simple learnable weights that can directly be element-wise multiplied with $\bar{O}_l$ and $\tilde{O}_l$? i.e., 
```python
w_bar   = nn.Parameter(d/h)  # d is model_dim, h is num_heads
w_tilde = nn.Parameter(d/h)  
# Combine, following L260
o  = torch.einsum('d,nhd -> nh', w_bar, o_bar)
o += torch.einsum('d,nhd -> nh', w_tilde, o_tilde)
```

In Equation 4, what is $\mathbb{E}_{\mathcal{D}}$ the expectation over? What are $\sigma$, $\beta$ and $\gamma$? Although these may be taken from SimPO, they should still be defined in a standalone manner. On a similar note, how does “Phantom Optimization” differ from SimPO? 

In Figure 3, how do the SoS tokens and User Prompt tokens correspond to the vision and language attention layers? Apologies if I missed this detail, but it seems important to clarify in regards to the overall architecture understanding. Related to the point on method motivation above, why does it make sense to do a first round of attention within each query, key, and value projection between the SoS token position output and the remaining tokens? 

**Component-specific study**   
Because of the introduction of dataset curation, it is unclear to me how much of the Phantom model performance is due to the data (collection over 2.8B visual instruction tuning samples, + phantom triples duration) versus the proposed architecture and training optimization components. It would be nice to see a data-controlled comparison, perhaps by applying the Phantom architecture and optimization over prior work’s training sets. Otherwise, with the current setup, although there is comprehensive comparison to different vision-language models, it’s hard to say what exactly is driving the boost in performance.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: * Will the models be made open source? Please explain if not.
* In line 076 the authors say state that "curating larger datasets" "brings in striking computational burdens during training". Could you please elaborate on this - what kind of dataset size increase is meant there in practice such that the resulting computational burden can be considered striking?
* What is the role of the regularisation parameter introduced in (2)? Is there a citation for this mechanism? If not, is the chosen value being ablated?
* In line 367 the authors mention that a single training step (of 128 examples) takes 2-5 days. Is this correct? Based on these numbers, it looks like a single epoch over the "Phantom triples" dataset would take more than 85 years.
soundness: 3
strengths: * Strong results on a range of benchmarks.
* A set of ablations that clearly demonstrate which of the introduced components is responsible for the improved performance.
* The proposed architectural modification to the MHSA module highlights the importance of further exploring VLLM architectures beyond the native transformer.
summary: The paper develops a new set of efficient VLLMs named "Phantom" that strong outperform models of comparable size on a range of VLLM benchmarks. The models are developed using open source components rely on a two-part solution - an architectural modification of the self-attention component of transformers; and a DPO-like optimisation method that utilises it.
weaknesses: **Major**
* "Phantom dimension" - the main of the two contributions of the paper is not described in an accessible manner in the paper, which significantly limits the ability to evaluate the paper and its potential impact. Consider adding pseudocode to describe what exact this modification does.
* It appears that the authors (primarily) used open source components to develop their models, however, it's unclear from the write up whether their models will similarly be made open source.

**Minor**
* It's not entirely clear "limited structures" used throughout the paper refers to.
* Is "Team et al" the correct citation for "Gemini Pro" (line 043)?
* Extra word in line 072 "... as a primary key to basically improve ..."?
* In line 086 and elsewhere - it's unclear what is meant by the phantom dimension "prepares to look and understand much more vision-language knowledge". It appears to be a subjective and strong statement - could you please elaborate on whether this is an intuition that the authors have, or it it can be concluded from some of the experiments presented in the paper? In any case claiming understanding due to an architectural modification might not be justified.
* Very minor, and admittedly subjective, but I find that the model name (Phantom) is overused in the text, e.g. "phantom dimension", "phantom optimisation", "phantom triplets".
* The added value of performing ablations for all model sizes in Table 4 is not that high in my opinion. Consider moving results for larger models to the appendix,

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: - Do you plan to share weights, code and data for training Phantom?
soundness: 3
strengths: - Introduced techniques are novel and offer a potentially valuable new direction for efficient LLVM design.
- The reported results demonstrate impressive performance compared to other LLVMs, especially in the lower parameter regime (0.5B, 1.8B), suggesting effectiveness.
- The ablation study provides insights into the contribution of each proposed component, although some aspects could be further explored.
summary: The paper proposes Phantom, a new family of Large Language and Vision Models (LLVMs) that achieve comparable or superior performance to larger models while maintaining a smaller model size. They achieve this through a new cross attention architecture "Phantom Dimension", which temporarily expands the latent hidden dimension during multi-head self-attention, and "Phantom Optimization" (PO), a training strategy that minimizes the generation of incorrect and ambiguous answers. They claim Phantom outperforms existing open- and closed-source LLVMs in its size category across multiple benchmarks.
weaknesses: ## Major

The paper's writing quality hinders clear communication of the results and techniques, making evaluation difficult.  For example, the abstract states:  "...LLMs either have further increased their sizes, reaching 26B, 34B, and even 80B parameters." This sentence is awkwardly phrased. The authors should prioritize improving the writing throughout the paper for clarity.  Using an LLM grammar checker would be beneficial.

The explanation of the proposed "Phantom Dimension" attention mechanism on page 5 lacks clarity.  A clearer presentation would first describe the attention mechanism used in the baseline architecture and then highlight the novel aspects introduced by Phantom. Figure 3 does not adequately illustrate these distinctions.

The paper introduces multiple novel elements: a new dataset, a potentially novel attention mechanism, and a new loss function. However, the ablation study omits the dataset. This omission prevents a proper assessment of whether the state-of-the-art results stem from the data itself or the specific combination of the Phantom architecture and optimization techniques.

The paper lacks a comparison to Malmo, a relevant work found at: https://arxiv.org/abs/2409.17146. This comparison is crucial for contextualizing the contributions.

## Minor 

The term "projectors" is used without clear definition.  If these are Multilayer Perceptron (MLP) layers, that term should be used for clarity and consistency.

Meta-review: The authors introduce a method to temporarily increase latent dimension during MHSA without inflating model size and Phantom Optimization (PO) using SFT and DPO-like concept, yielding an efficient LLVM family with good performance.  Overall, the techniques are novel, the experimental results are impressive especially for lower parameter models, and the ablation study offers some insights. However, reviewers have two major concerns on the presentation and experimental setting. The presentation requires significant enhancement and further justification for the phantom dimension design. Also, comparisons with related baselines in similar settings are lacking. AC agrees that addressing these issues would strengthen the work.

Decision: Reject