HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding
Authors: Rui Yang
Lin Song
Yicheng Xiao
Runhui Huang
Yixiao Ge
Ying Shan
Hengshuang Zhao
Venue: ICML 2025 poster (2025)
Keywords: Vision language models
Multi-modal large language model
Primary area: deep_learning->large_language_models

Abstract: Recent advancements in large language models (LLMs) have significantly propelled the development of large multi-modal models (LMMs), highlighting the potential for general and intelligent assistants. However, most LMMs model visual and textual modalities separately, leading to recent efforts to develop native LMMs using a single transformer. Despite the promise, these native models are resource-intensive and often exhibit performance gaps compared to their compositional counterparts. To alleviate this issue, we propose a simple yet efficient method to construct a baseline for the native and end-to-end large multi-modal model in a single transformer. First, we propose a new early-fusion LMM that can fuse multi-modal inputs in the early stage and respond to visual instructions in an auto-regressive manner. Second, we devise an efficient training recipe for the proposed model, which harnesses the prior knowledge of the pre-trained models, addressing both the performance limitations and the challenge of resource consumption. The proposed model demonstrates superior performance compared to other LMMs using one transformer and significantly narrows the performance gap with compositional LMMs.

HaploVL: A Single- Transformer Baseline for Multi-Modal Understanding
Rui Yang' Lin Song?*
2+
Yicheng Xiao
Runhui Huang'
Yixiao Ge
Ying Shan
Hengshuang Zhao 1*
Abstract
RWOA
A12D
Recent advancements in large language models (L.LMs) have significantly propelled the develop- ment of large multi-modal models (LMMs), high- lighting the potential for general and intelligent as- sistants. However, most LMMs model visual and textual modalities separately, leading to recent ef- forts develop native LMMs using single trans- former. Despite the promise, these native mod- els are resource-intensing and often exhibit per- formance gaps compared to their compositional counterparts. To alleviate this issue, we propose a simple yet efficient method to construct a baseline for the native and end-to-end large multi-modal model in single transformer. First, we propose a early-fusion LMM that can fuse multi-modal inputs in the early stage and respond to visual instructions in an auto-regressive manner. Sec- ond, we devise an efficient training recipe for the proposed model, which harnesses the prior knowl- edge of the pre-trained models, addressing both the performance limitations and the challenge of resource consumption. The proposed model demonstrates superior performance compared to other LMMs using one transformer and signifi- cantly narrows the performance gap with com- positional LMMs. Code is available at https: //github. com/Tencent /HaploVIM.
MMMU
POPE
MMB
SEED
MMS
MMVP
LLaVA-2 2A-2--5-7
66.0
LlaM6- 1.---.44
HapioVL
Figure 1: Performance comparison with single-transfore models on multi-modal understanding benchmarks. Our HaploVL demonstrates superiority over other counterparts.
et al., 2023a; Liu al., 2024a). These models facilitate complex vision-language dialogues and interactions. The majority of open-source models (Liu et al., 2024c; Dai et al., 2023) leverage one or more separate vision components to model the visual modality, thus equipping LLMs with visual understanding and reasoning capabilities. For instance, the LLa VA series (Liu et al., 2024c;a) directly harnesses the pretrained CLIP vision encoder (Radford et al., 2021) to extract high-level vision embeddings and uses projector connect these embeddings with LLMs.
1. Introduction
Large language models (LLMs) (OpenAI, 2023; Dubey et al., 2024; Yang et al., 2024a) have recently made sig- nificant strides in the realm of artificial intelligence. This progress has substantially accelerated the development of large multi-modal models (LMMs), which include both pro- prietary commercial models (Achiam et al., 2023; Team et al., 2024) and open-source models (Dai et al., 2023; Zhu
Since language is human-gene generated signals that have al- ready been abstracted (He et al., 2022), the text embeddings produced by the word embedding layer contain semantic information and are high-level. Therefore, is reason- able combine text embeddings with vision embeddings from pre-trained vision encoder, as both types of em- beddings semantic. However, the off-the thethhe-seef vision encoder (Radford et al., 2021) tends to produce highly com- pressed global semantics and neglect fine-grained visual information. Thus, may fail to extract effective visual cues required by the text, leading to difficulties for LMMs in handling fine- grained tasks (Tong al., 2024).
'The University of Hong Kong FARC Lab, Tencent PCG 'Tsinghua University. Team leader: Lin Song <ron- nysong tencent.com........ Correspondence to: Hengshuang Zhao Chszhao@cs hku.hk>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

HaploVL: VL: A Single-Trannnn Transs-Tranerrrmrr Baseline for Multi-Modal Understanding
Cat
VT
Cttp
Post-decoder
Text krowloage
LLM Decoder
Projectos
cap
LLM Decoder
Pre-decoder
Vissan krowlegge
Projector
Tokenizer& Cokemizee&&mmboociinng
VIT
This is image of
Patch Embedding
Patch Embedding
Bokenizce&Emmeeddiue
This an image of
This is image of
Patch Embedding
(4) The compositional LMM
(b) EVE
(c) HaploVL (ours)
Figure 2: Architecture comparison with the compositional LMM (Liu et al., 2024a), EVE (Diao et al., 2024). In our VI, the pre-decoder dynamically extracts vision cues based on the input text, and the post-decoder further fuses the multi-modal embeddings. Our model inherits the prior knowledge from vision and language models, thus requiring less data than EVE.
Data Early Seamntic (M) fusion
To this end, our Ha utilizes a pre-decoder to au- tonomously acquire the necessary vision cues according to text information, and a post-decoder further process the extracted high-level multi-modal embeddings. Since training such model from scratch is very expensive, e.g. the energy consumption required to pre-train the Chameleon- 30B (Team, 2024) is equivalent to what is needed to power a Tesla Model 3 to travel around the equator for about 225 times 1, we propose to leverage prior knowledge acquired from pre-trained models. This because the pre-trained models have gained extensive knowledge by training on massive data, e.g.. the CLIP vision encoder (Radford et al., 2021) obtained vision-based knowledge by seeing billions of images, and Llama (Dubey et al., 2024) gained text-based knowledge by seeing trillions of text tokens. Specifically, the pre-decoder inherits prior vision knowledge from a vi- sion encoder while simultaneously processing text and vi- sion modalities to perform modal expansion. Plus, the LLM retains its prior text knowledge and learns to take vision beddings a condition. In this way, we significantly reduce the required data and training costs in comparison to other early-fusion and single-transformer LMMs (Bavishi et al., 2023; Team, 2024; Diao et al., 2024; Wang et al., 2024b), and bridge the performance gap between unified and com- positional LMMs. As shown Table 1, Haplo VL achieves significant performance improvement over LLaVA and EVE (Diao al., 2024) on fine-grained perception bench- marks (Chen et al., 2024a; Tong et al., 2024). This demon- strates promising potential for developing multi-modal mod- els with a single transformer efficiently.
Method
SEED MMStarmp MMVP
align
LLaVA-1.5-7B
1.2
66.1
24.0
21.3
EVE-7B
35.0
V
x
54.3
24.2
19.3
HaploVL-7B 1.2
V
v
67.5
28.9
24.7
Table I: Comparison with compositional LLaVA (Liu et al., 2024a) and unified EVE (Diao et al., 2024) on multi- modal benchmarks: SEED-Bench (Li et al., 2023a), the fine-grained split of MMStar (Chen et al., 2024a), and MMVP (Tong al., 2024).
To address this issue, we propose an early-fusion LMM named HaploVL. Our model fuses the vision and text em- beddings at an early stage, enabling text embeddings to autonomously acquire the necessary vision cues. Specifi- cally, Va uses lightweight patch embedding layer, single linear layer, to embed visual input and a text embed- ding layer to process textual inputs. Subsequently, the trans- former backbone extracts the necessary vision information based on the text input and generates language responses according to the resulting fused representations.
Some recent studies (Bavishi et al.. 2023; Team, 2024; Diao al., 2024) also fall under the category of early-fusion LMMs and have endeavored to develop unified multi- modal transformer with a concise inference process. For ex- ample, Fuyu (Bavishi et al., 2023) directly utilizes a simple linear layer instead a vision encoder to embed the input image and leaves the mixed modality sequence to the subse- quent transformer. EVE (Diao al., 2024) aims to replicate Fuyu by being distilled from fixed vision encoder, thus reducing the training data. However, it forces alignment be- tween large language model (7B) and a small ViT (300M) without allowing the LMM to learn from high-level vision features. Therefore, there a significant performance gap between it and compositional LMMs on laigual benchmarks, despite using 35M training data.
Our contributions can be summarized as follows:
We develop new early-fusion LMM with single transformer that acquires the necessary vision cues
Estimated by Chameleon-30B (Team, 2024)'s GPU hours, 1100 GPU's power, and Tesla Model 3's power consumption.
2

---

HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding
in the early stage and generates language responses conditioned fused multi-modal embeddings.
tokens. Compared with models with the "ENcoder-MI.P LLM" configuration, Fuyu directly fuses low-level vision embeddings with text embeddings instead of high-level vision embeddings (hidden states of the vision encoder). Besides, Chameleon (Team, 2024) employs a VQ code- book (Van Den Oord et al., 2017) to discretize the image to a set of discrete visual tokens, akin to the process of the text tokenizer. Thus, the vision and text embedding can be extracted from the same embedding layer and processed by decoder-only transformer. Emu3 (Wang et al., 2024b) has extended this streamlined pipeline to generate high-quality images and videos. Since these methods are trained from scratch, they consume substantial computing resources and necessitate significant amounts of data. To adapt an off-the- shelf decoder-only language model a multi-modal model, EVE (Diao et al., 2024) introduces meticulously designed patch embedding layer and training strategies. However, they still exhibit a significant performance gap compared to encoder-decoder multi-modal language models, despite utilizing 35 M images.
We design an efficient training recipe for the proposed model, which leverages the prior knowledge from pre- trained models. This approach not only reduces the need for large-scale data and computational resources but also bridges the performance gap between the uni- fied and compositional LMMs.
2. Related Work
Encoder-decoder large multi-modal models as exempli- fied by LLaVA (Liu et al., 2024c), employ a pre-trained vision encoder like CLIP (Radford et al., 2021) to extract visual embeddings and an MLP layer to align the visual embeddings with large language models (LLMs). Then, these models with the "ELCLLer-MI.--LLMM configuration are fine-tuned on tailored instruction data to obtain the ca- pability of image understanding and reasoning. Numerous innovations have sought to improve the performance of this method by utilizing more powerful vision encoders such as SigLIP (Zhai et al., 2023) and InternViT-61 (Chen et al., 2024b), expanding the input size to any resolution (Liu et al., 2024a), and synthesizing high-quality data (L.i et al., 2024a). At the same time, inspired by this straightforward architecture, numerous studies have replaced the vision en- coder with domain-specific encoder to develop a modality- specific multi-modal model (Chu et al., 2023; Qi et al., 2024). Plus, others (Lu et al., 2022a; Zhan et al., 2024) inte- grate multiple modality-specific encoders with the language model to enable to accommodate more additional modali- ties. However, a significant limitation of this method is the lengthy visual sequences. To alleviate this issue, BLIP-2 (Li et al., 2023b) develops Q-former replace the long vi- sual features with fixed number of learnable queries. This "Encoder-Q Q-fo-eer--ooraae LLM" configuration has been replicated by many studies (Zhu et al., 2023a; Dai et al., 2023; Ye et al., 2023; Li et al., 2024c), MiniGPT-4 (Zhu et al., 2023a), In- structBLIP (Dai al., 2023), mPLUG-OWl (Ye et al., 2023), and MGM (Li et al., 2024c). Apart from those architectures, Flamingo (Alayrac et al., 2022) incorporates gated cross- attention layers into LLMs for cross-modality alignment; CogVLM (Wang et al., 2023) equips extra vision experts in each block of L.MSs to enable visual-language feature alignment; and mPLUG-OWI-2 (Ye et al., 2024) introduces a modality-adaptive module to decouple vision-language representations.
3. Method
Our HaploVI. is a single-transformer multi-modal model. Like popular LMMs (Liu et al., 2024c; Dai et al., 2023; Bav- ishi et al., 2023), it maps visual and textual input to the same latent space and takes them as conditions for text generation in an auto-regressive manner. Unlike other LMMs that al- ways rely on highly compressed vision embeddings from a fixed vision encoder, our Haplo fused the visual and tex- tual input in the early stage and extracts the necessary vision information based on the text input. Compared to previous carly-fusion and tralf--trnnffrrmer LMMs (Bavishi et al., 2023; Diao et al., 2024; Team, 2024), our HaploVL is more efficient in training, as it absorbs the prior knowledge that the model learned. the subsequent section, we begin by presenting a detailed description of HaploVL's 's architecture, followed by a receipt of the efficient training procedure.
3.1. Architecture
From holistic perspective, as illustrated on the right side of Figure 3, our HaploVI. adopts multi-modal end-to- end transformer architecture. Most of its parameters are at- tributed to the transformer decoder that processes sequences, regardless of the modality. Ha can generate the lan- guage response X₂ conditioned on the visual input X., and textual input X, in an auto-regressive manner. This genera- tion process is as clear as that of language models (Radford et al., 2019; Dubey et al., 2024). Given sequence of length L, Ha calculates the probability of X, by:
Single-transformer multi-modal models aim to discard the vision encoder and merely allow the language model to process text embeddings and vision embeddings that are not fully compressed Fuyu (Bavishi et al., 2023) utilizes a lin- ear projector to patchify the raw image wherein the obtained low-level vision patch embeddings are treated as contiguous
|X X.) = II po (2 Xy, Xx, Xa,ee),
(1)
3

---

VaploVL: A Single- Transforrre Baseline for Multi-Modal Understanding
Vision Encoder
Tokemizen& Embosiing
Language Resporso X,
Language Response X
T
T.
br
ir
Bi
Post-decoder
Vision Head
Text Head H.
Transformer Decoder
AE
Pre-decoder
Pre-decoder
Patch Finheldiig
Tokenizen& Eibaddiing
Patch Fimbeddiig
Patch Embeeding
Tokenizer& Fimbedding
X
X
Xx
X,
Ingun Teat
Imput Test
Imput Test
(a) Stage
Pro-tmining
(b) Stage 2: Fully finetuning
(c) Architecture
Figure 3: The diagram of HaploVL.. It includes a transformer decoder made up of pre-decoder and a post-decoder. During the pre-training stage (a), the pre-decoder is trained by distilling knowledge from the pre-trained vision encoder and the text embeddings of the LLM. Heads and teacher models are dropped after pre-training. In the full fine-tuning stage (b), the entire model is fine-tuned using visual instruction data.
where Xa,<s denotes the answer tokens before the current prediction token x;- 0 is the parameter of components that model conditional probability. Thus, 0 of our HaploVI VL is from the whole model, while 9 of compositional LMMs us- ing separated vision encoders (Liu al., 2024c; Karamcheti al., 2024) is the parameter of the LLM.
ing visual cues based on text embeddings. Then, it yields a multi-modal hidden state Hm- Each block of the pre- decoder consists of multi-head self-attention layer and 2-layer MLP with GELU (Hendrycks & Gimpel, 2016) nonlinearity in between. Its configuration, such as depth and width, mirrors that of the vision transformer, it is re- quired to inherit the prior vision knowledge of a pre-trained vision model. In practice, we default to leveraging CLIP- ViT-L (Radford et al., 2021) which has 24 blocks and an embedding dimension of 1024. Notably, although the pre- decoder inherits prior knowledge from a vision encoder, it differs from the vision encoder. For one thing, the pre- decoder can process both visual and textual input, whereas the vision encoder only processes visual input. Furthermore, the text embeddings in the pre-decoder utilize causal mask strategy, allowing the pre-decoder to predict the next token in an rugo-segressive manner.
From detailed perspective, as depicted in Figure 3 (b), Ha can be decomposed into three primary compo- nents: (1) multi-modal embedding layers, (2) a pre-decoder, and (3) a post-decoder. These bottom-up modules work together to facilitate efficient training and enhance visual understanding and reasoning performance, especially in the fine-grained scene.
Multi-modal embedding layers. Regarding the input data, we use lightweight and modality-specific components with unshared parameters to map them into a shared latent space Rd. Specifically, for the input RGB image X., we apply simple patch embedding layer, single linear layer, to compress local windows (k of pixels into vision embedding Z, within the shared latent space Rd This approach differs from existing compositional LMMs (Liu et al., 2024a; Karamcheti et al., 2024), which typically rely on one or more separate vision encoders to embed visual input. For the input text X,, we leverage the pre-trained LLM's embedding matrix W to convert each text token into vector within LLM's space R!. These text vectors are then projected into text embeddings Z, within the shared latent space IR by text projector, also single linear layer. The resulting vision and text embeddings, Zy and Z,, are combined to form a mixed multi-modality embedding sequence Zm. which fed into the subsequent transformer.
Post-decoder, Based on the multi-moda. hidden state Hi.- the post-decoder further processes it and outputs language response. Each block of the post-decoder mirrors the Llama block (Dubey et al., 2024) it needs to acquire prior tex- tual knowledge the Llama model. Leveraging the inherited knowledge from extensive text data, the post-decoder can swiftly learn multi-modal knowledge and generate language responses based on multi-modal hidden states.
Masking strategy. HaploVL employs mixed masking strategy within its self-attention layers. In mixed multi- modal sequence, causal mask is utilized for the textual part. This is consistent with GiT-like language models (Radford et al., 2019). For the visual part, a bidirectional mask is applied to embeddings from single image, as correlations exist between image tokens regardless of their positional order. In addition, causal mask is employed between
Pre-decoder: Upon the modti sequence Zm, the pre- decoder fuses it in the initial stage of Ha,loVI., extract-

---

HaploVL: A Single-Transforme Baseline for Multi-Modal Understanding
multiple images, reflecting the temporal causal relationships in sequential data, This modeling approach aligns with prevalent vision models (Dosovitskiy et al., 2021).
(a) The first type is feature loss, which is formulated as:
S 2 Lieat = 1+ 1 S 1=1
(HHl,,-Te,n, -..l--T-oo---s Te,)]. (3)
3.2. Training
Here, S denotes the length of text tokens in the input se- quence; and T, represents the text embedding directly ob- tained from the embedding matrix W using indices of text tokens, which also serves as the input of the pre-decoder. Equation (3) involves the L2 distance to align the magnitude and the cosine loss function to align the direction. This is because the magnitude and direction of text embeddings are crucial for alignment with the post-decoder, which has been retained in certain input mode when inheriting knowledge from pre-trained LLM.
We use two-stage training recipe for HaploVL, illus- trated in Figure 3. In the first stage, the pre-decoder is trained through feature distillation. This enables it to effec- tively process both visual and textual inputs simultaneously, laying the foundation for subsequent stages. In the sec- ond stage, the model is trained to follow visual instruction, which equals LLa'A's visual instruction tuning.
Pre-training. As mentioned before, the pre- decoder inherits prior vision knowledge from the pre-trained ViT and can fuse visual and textual inputs. This stage mainly endows the pre-decoder to support vision and text modality. As illustrated in Figure (a), the knowledge distillation ap- proach (Hinton, 2015) is employed to train the pre-decoder, prompting the model to learn new text knowledge and avoid- ing the model forgetting the inherited vision knowledge. Given the visual input X, and textual input Xt, the output of the pre-decoder is the hidden state Hm. which can be de- composed into the visual hidden state Hy and textual hidden state H. based on their respective token positions.
Stage 1:
(b) The second type of loss function is a current token pre- diction loss which can be formulated as:
5
C
Lap
Vicclog
(4)
Σ5
e FL
In this loss function, C' refers to the vocabulary size of the tokenizer; yi is the one-hot label of the i-th token; and Vi. is the label of e-th word in the vocabulary. Iy H., W is the logit of A.., A learnable temperature T is used to adjust the distribution of the logits as CLIP (Radford et al., 2021). This very effective in reducing the magnitude of the output text embeddings, as the cross-entropy loss function minimizes the total loss by enlarging the logit mag- nitude. The difference between Equation (4) and the next token prediction loss (Radford et al., 2019) lies in that the target in Equation (4) is derived from the current token instead of the next token.
To preserve the image processing capabilities of the pre- decoder, we adopt the pre-trained CLIP vision encoder (Rad- ford et al., 2021) as a teacher model to guide the expansion process. This approach enables the pre-decoder to retain its inherited knowledge, ensuring its image abilities are not compromised. This vision loss can be formulated as:
So far, we have introduced two types of loss functions, and the total text loss function used in the modal expansion stage is the sum of them: L. = f + Letp- We combine interleaved image-text data and pure text data to train the pre- decoder. After modal expansion, we keep the pre-decoder, while discarding the heads.
how
1
C.
cos (o,, Tv,),
hw
where H. is the projected H. by vision head; T, is the fea- ture extracted from CLIP vision encoder; and hwo represents the number of vision embedding after patch partition.
Stage 2: Fully fine-tuning. This training stage mainly for multi-modal learning. As illustrated in Figure 3 (b), we fine-tune all components of Haplo in this stage. The next token prediction loss (Radford et al., 2019) is still adopted to maximize the log-likelihood of Equation (1). After tun- ing, our HL performs capabilities in following human visual instructions.
For the textual input, the pre-decoder performs simple identity mapping. This training target enables the pre- decoder to leverage the strengths of the post-decoder in handling complex generation tasks, thus alleviating the chal- lenging multi-modal learning. More importantly, language semantic (He et al., 2022). When the text and image are jointly input into the pre-decoder in mixed way, semantic text embeddings can autonomously acquire the necessary vision cues from raw vision embeddings. For the split tex- tual hidden state H,, we utilize a learnable text head to align it with the teacher embedding. resulting in H.. We employ two types of loss functions to encourage this distillation.
4. Experiment
In this section, we first outline the experimental setup in- cluding training settings and dataset. Then, we compare our Haplo VL with leading methods on various benchmarks. Finally, an analysis of training procedures and some qualita- tive results are given at the end of this section.
5

---

Hap A Single-Transformer Baseline for Multi-Modal Understanding
Method
Base LLM SEED POPE Al2D RWQA MMMU MMB MMS VQAv2 GQA
SQA MMVP
Compositional LMM
InstructBLIP (Dai et al., 2023)
Vicuna-7B 7B
58.8
33.8
37.4
30.6
36.0
49.2
60.5
16.7
1.. (L.uu et al., 2024a)
Vicuna-7B 66.1
85.9
54.8
54.8
35.3
64.3
30.3
78.5*
62.0*
66.8
21.3
LLaVA-1.6 (Liu et al., 2024b)
Vicuna-7B
70.2
86.5
66.6*
57.8
35.8
67.4
81.8
64.2*
70.1
37.3
ShareGPT4V (Chen et al., 2023)
Vicuna-7B 7B
58.0
54.9
37.2
68.8
33.0
80.6*
63.3*
68.4
VILA (Lin et al., 2024)
Llama-2-7
61.1
85.5
68.9
80.8*
63.3*
73.7
LLaAA-OV (Li et al., 2024a)
Qwen2-7B 75.4
81.4*
66.3
48.8
80.8
61.7
96.0*
Single-Transfore Tramsformer LMM
Fuyu-8B (Bavishi et al., 2023)
Persimmon-88
74.1
64.5
27.9
10.7
74.2
Chameleon- 300 (Team, 2024)
37,6
69.6
EVE-7B (Diao al., 2024)
Vicuna-7B
54.3
83.6
49.5
28.2
75.4*
60.8*
63.0
19.3
Emu3-8B (Wang et al., 2024b)
68.2
85.2
70.0
57.4
31.6
58.5
75.1*
60.3*
89.2*
HaploVT.-8B (ours)
Llama-3-8B
75.1
88.6
79.2*
61.4
37.4
73.6
57.2
81.0*
65.5*
95.3*
45.3
HaploVL-8B-Aa (ours)
Llama-3-8B
75.5
88.2
79.6*
62.0
41.7
75.0
57.6
80.7*
65.0*
94.4*
46.0
HaploVL-7B- Pro (ours)
Qwen2.5-7B
75.0
88.7
80.6
64.3
48.7
80.5
61.4
81.1*
64.6*
96,9*
50.1
Table 2: Comparison on multi-modal benchmarks, including SEED (Li et al., 2023a), POPE (Li et al., 2023c), A12D (Kemb- havi et al., 2016), RWQA (x.ai, 2024), MMMU (Yue al., 2024), MMB (Liu al., 2024d), MMStar (Chen et al., 2024a), VQAv2 (Goyal et al., 2017), GQA (Hudson & Manning, 2019), and SQA (Lu al., 2022b). *** denotes images of related training datasets are observed during training. HaploVL-8B-MM is the model further fine-tuned on multi-image datasets.
4.1. Experiment Setup
memory. More details are recorded in the Appendix.
Implementation details. In this study, we instantiate HaploVL by allowing pre-decoder to receive images and texts at the same time. The pre- inherited the vi- sion knowledge of the CLIP-ViT-L (Radford et al., 2021). The post-decoder inherits the text knowledge from Vicuna- 7B (Chiang et al., 2023) and Llama-3-8B (Dubey et al., 2024), resulting in HaploVL-7B and HaploVL-8B, respec- tively. During the pre-training stage, we optimize the post- decoder for 40 K steps with 1 x e learning rate, batch size of 256, and 2 K warm-up steps. In terms of the data, all models are trained on 665 K plus 558 K multi-modal sam- ples from LLaVA-1.. 1.5 (Liu et al., 2024a) if there is no other statement. During the fully fine-tuning stage, the learning rate is set to 2 - and batch size to 128. Regarding the data, our best model is optimized on the 4 M visual instruc- tion data for epoch 30K steps). For HaploVL-7BB we align it with LLaVA (Liu et al., 2024c). Thus, we first tune the connector between the pre-decoder and post-decoder us- ing 558 K caption data and then fully tune the model using 665 K instruction data. For HaploVL-8B with the ability to input any resolution, we first tune the whole model using 1.2 M caption data (Chen et al., 2023) and then tune the model using 4 M instruction data (Li al., 2024a). For the models that support the multi-image and video input, we continue training the single-image model using the mix of interleaved data and single-image data. For the ablation experiments, the models are optimized on the 0.6 M visual instruction data for 5 K steps. All models are optimized using the AdamW (Loshchilov & Hutter, 2019) optimizer and cosine scheduler on 32 GPUs with 64GB per-device
The data is mainly from LLaVA (Liu et al., 2024a; Li et al., 2024a), dolphin (Computations, 2023), CC3M (Changpinyo et al., 2021), and COCO (Lin et al., 2014).
Pre-training data. The main samples include 665 K vi- sual instruction data (Liu et al., 2024c) and 665 K tex- tual instruction data from dophin (471 K) (Computations, 2023), Alpaca (51 K) (Taori et al., 2023), and ShareGPT (143 K) (ShareGPT, 2023). The auxiliary samples involve 558 K image caption data (Liu et al., 2024c) and M pure text data (Computations, 2023). The auxiliary samples are randomly combined with the main samples during train- ing. Assume main sample is Sm and a auxiliary sample is Sa, the combined samples can be one of < SaySm and < Sm.SS Sg >. In this way, we obtain interleaved samples.
Single-image imgge- instruction data, 665 K instruction data is from LLa 1.5 (Liu et al., 2024a). M instruction data is partly from OeeA--OeViiiin (Li et al., 2024a), where we filter some error samples.
Multi-image instruction data. We use the interleaved data collected by (Li et al., 2024b) to endow the model (HaploVL. 8B-MI) with the ability to process multiple images.
Evaluation data. Moreover, HaploVL is evaluated on widely adopted image-based benchmarks including GQA (Hudson & Manning, 2019), VQAv2 (Goyal et al., 2017), ScienceQA-IMG (SQA) (Lu al., 2022b), AI2D (Kembhavi et al., 2016), MMBench-EN-dew (MMB) (Liu et al., 2024d), MMMU (Yue et al., 2024), Real- WorldQA (x.ai, 2024), MMStar (MMS) (Chen et al., 2024a),
6

---

HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding
POPE (Li et al., 2023e), SEED-Bench-IMG (SEED) (Li
Base LLM
Res.
VID
Avg
GQA
POPE
MMS
MMVP
et al., 2023a), and MMVP (Tong et al., 2024). Among these benchmarks, MMVP mainly focuses on fine-grained perception.
Vicnua-7B
336 665K
5188
62.5
85.4
34.5
24.7
LJama-3-88 336
665K
54.3
63.1
84.8
39.4
30.0
Llama-3-8B
672
665K
57.6
65.7
88.2
42.0
34.3
LJama-3-8B 672
4M 64.2
65.5
88.6
57.2
45.3
4.2. Main Results
Table 3: Ablation for different LLMs, resolution (Res.), and visual instruction data (VID).
We compare our model with existing multi-modal models, including both separate models and unified models with a single transformer, in Table 2. Notably, our model achieves superior performance compared to other unified models. Specifically, we outperform Emu3 (Wang et al., 2024b) by 15.1% on the MMBench (Liu et al., 2024d) and by 5.5% on the MMMU (Yue et al., 2024). Additionally, our model significantly surpasses EVE (Diao et al., 2024), a model that uses pre-trained weights, with a lead of 24.1% on MM- Bench (Liu et al., 2024d) and 20.8% on SEED-Boc BEED-Bench (Li et al., 2023a). These results demonstrate the promising potential of our model in multi-modal capabilities. Further- more, we also compare our model with separate models and find that model has significant advantage over previous separate models (Dai et al., 2023; Liu et al., 2024a; Chen et al., 2023; Lin et al., 2024). However, our performance still falls short of the state-of-the-an separate open-source models LLaVA-One Vision (Li et al., 2024a). We attribute this to the input resolution and context length. LLaVA- One Vision (Li et al., 2024a) uses 7290 tokens to represent an input image, while our model only uses up to 2304 tokens. Due to computational resouree constraints, we can only set the context length to 6144, which affects the model's s effec- tiveness to some extent. Nevertheless, the performance of HaploVL--BB-Pro is nearly comparable to that of LLaVA- One Vision. Plus, we achieve a simple and efficient baseline for one modal transformer, which outperforms other native LMMs using fewer resources. We expect to further improve the performance of such models based on this foun- dation.
MMS
Method
ST MMVP
Avg
CP
FP
IR
LR
ST
MA
LLaVA-1.5-7B EVE-7B
X
21.3
30.3 58.8 24.0 38.8 24.0 13.6 22.8
19.3
28.2 40.6 24,2 32.7 27.5 20.4 23.8
HaploVL-7B
24.7
34.5 63.8 28.9 38.5 33.6 17.6 24.6
Table 4: In-depth comparison with LLaVA-1.5-7B (Liu et al., 2024a) and EVE (Diao et al., 2024) on MMVP (Tong et al., 2024) and MMStar (Chen et al., 2024a). 'ST' denotes whether the model belongs to the single-transee tranf--rrnnformme LMM. CP: coarse perception, FP: fine-grained perception, IR: in- stance reasoning, LR: logical reasoning, ST: science and technology, and MA: mathematics.
resolution, the average performance improves by 6.6% since LMM's s knowledge is enriched. These gains are particularly pronounced on benchmarks such MMStar (Chen et al., 2024a) and MMVP (Tong et al., 2024), suggesting that fine- grained perception ability can be enhanced after expanding LMM's vision knowledge. However, a slight performance decline is observed on GQA (Hudson & Manning, 2019). This discrepancy may stem from differences in the distri- bution of the 4M instruction data compared to the GQA dataset.
Compared with the compositional LMM using the same LLM and training data. We compare the performance of our method with that of LLaVA-1.5-7B (Liu et al., 2024a), a typical compositional LMM, using the same LLM (Vicuna- 7B) and instruction data (665K) from (Liu et al., 2024a). Since the LLM and instruction data primarily affect per- formance on different benchmarks, we restrict the data to ensure a fair comparison between our method and VA- 1.5-7B. This allows us to verify whether the LMM using one single transformer has advantages over separate models. As shown in Table 4, on the MMVP (Tong et al., 2024) bench- mark, our model obtains 3.4% and 5.4% gains than LLaVA- 1.5-7B and EVE-7B (Diao et al., 2024), respectielyy and, on the MMStar (Chen et al., 2024a) benchmark, our model out- performs LLL respectively. We further analyze the detailed scores on MM- Star (Chen et al., 2024a), including coarse perception (CP), fine- mme-grained perception (FP), instance reasoning (IR), logi- cal reasoning (LR), science and technology (ST), and math- ematics (MA). Notably, our HaploVL.-78 model exhibits a
4.3. Ablation Study
Ablation for different LLMs, resolution, and visual in- struction data. As shown in Table 3. we achieve enhanced performance by upgrading the language model, input reso- lution, and instruction data. Specifically, employing a more advanced language model (Llama-3 (Dubey et al., 2024)) yields an average performance gain of 2.5%. This highlights that multi-modal understanding capabilities are correlated with the capabilities of the language model.
Increasing the resolution from 336 x 336 to 672 x 672 results in an average performance improvement of 3.3% using the same 665K dataset, especially showing a notable 3.7% gain on POPE (Li et al., 2023c). This underscores the importance of enabling the LMM to perceive finer- greinnd visual details. When expanding the visual instruction data at the 672 x 672
and EVE-7B by 4.2% and 6.3%,
7

---

HaploVL: VL: A Single-Transtone Transformer Baseline for Multi-Modal Understanding
Happy
birthday
N BEVIEW
CAVS
SG
CRV:
0
ALL THE
KING'S MEN MEN
DE
Kaaaf Simdlor
Free: Thailladd
SPALL
Q: many colors the
What the
color of the socks detected in the attribute detection of the image?
Q:
Q: many drawers can Q: many people are
eyes of the depicted animals?
letter the red circle?
be the white kitchen'visible the image?
A: Two, B: One,
A: Black, B: White,
A: Three, B: Two,
A: One, B: Two,
C: Three, D: Four
C: Red. D: Yellow
A: Black. B: Red.
C: One. D: Four
Four, D: Three
LLaVA: D
LLaVA: C
C: White, D: Striped
LLaVA: A
LLaVA: A
Haplo: B
Haplo: B V
LLaVA: A
Haplo: D
Haplo: B
Haplo: C.V
What basketball player the left doing the image? A: Dribbling the ball, B: Throwing the ball to a teammate,
Subtract yellow things. Subtract all blue cylinders. How many objects are left?
Q: Subtract small green shiny balls. Subtract all small metallic things. How many objects are lett?
Which country is highlighted??
Q: Which doesn't any food?
A: Kiribati,
A: top-right,
(A) 0 (B) 2
B: Nauru,
B: top-left,
C: Trying to tackle the other
(C) (D)
(A) 0 (B)
C: the Marshall Islands,
C: bottom-left.
player, D: Standing still
LLaVA: A
(C) 3 (D)
D: Solomon Islunds
D: bottom-right
LLaVA:
Haplo:
LLaVA:
LLaVA: A
LLaVA: A
Haplo: C.V
Haplo:
Haplo: CN
Haplo: D
Figure 4: Qualitative comparison of LLLV (Liu et al.. 2024a) and our HaploVL-7B. The first line involves cases
about fine-grained perception. The second line includes cases of logical reasoning that depend on fine-grained perception.
Metheds
Person
Age
Chris
38
dlaly lning
Dannis
45
Sarah
29
Karen
47
Ecstooicil Methods
Person
Age
Ocooghhe to uni sange ledt
Sat ah
Karen
47
Q: Hew many people are there the image? HaploVL:
Q: Where the drum located in the image!
Q: According to this picture, how old are Denns.
Q: where the sal?? HaploVL: bottom-right
Q: What the position of grape relation the plate of walnuls?
HaploYL: On the right side
HaploVL:
HaploVL: The grape on top the plate of walnuts
Figure 5: Visualization for the early fusion mechanism of our single transformer. The second row illustrates the attention map of the gray words concerning the vision embeddings after the pre-decoder.
4.9% improvement in fine-grained perception and a 9.6% improvement in logical reasoning over LLaVA-11557B. This
may obscure fine-grained image information, thereby im- pairing the model to perform tasks that rely on image details. This consistent with previous study (Tong al., 2024).
suggests that fusing raw image and text embeddings sin-
gle transformer is beneficial for fine-gre -re-grained perception and subsequently enhances image-basee based logical reasoning. In contrast, separate models using high-level semantic embed- dings from CLIP-ViT ViT encoder (Radford et al., 2021) directly
To further illustrate the differences in fine-graines percep- tion and logical reasoning, provide qualitative results in Figure 4. The first row shows cases of fine-grained per-
8

---

Vap A Single- Transs-Tranrrrrmr Baseline for Multi-Modal Understanding
ements within the image. These findings suggest that the early fusion mechanism of our single-transforee model is effective for fine-grained perception tasks, thereby corrobo- rating the results presented in Table 4.
10-
wio sage
Loas Value
5. Conclusion and Discussion
5000
10000
15000
20000
25000
30000
This work presents simple baseline for the multi-modal model with single transformer architecture and corre- sponding efficient training approach. By fusing raw vision and text embeddings in the early stages, our model enhances its fine-grained perception capabilities, enabling it to cap- ture subtle relationships in the image better. Furthermore, our model builds upon prior knowledge from pre-trained single-modal models. This allows to achieve superior per- formance with relatively few training tokens and bridge the performance gap between single-transformer multi-modal models and compositional models. Consequently, it demon- strates the potential of single-transfomms architectures for multi-modal tasks.
Tranning Step
Figure 6: Loss curve of the full fine-tuning stage. We use ema to smooth the actual loss value for better visualization.
Base LLM Res. S-1
Avg GQA
POPE
MMB
MMS MMVP
Llama-3-8B 336
X
55.5
56.6
84.0
64.3
46.0
26.7
Llama-3-8B 336
60.5
60.7
84.5
72.9
51.3
33.3
Table 5: Ablation for the stage one (S-1). Both models are trained using 4M instruction tuning data. The model with the pre-training stage shows faster convergence and superior performance.
We anticipate that future work can further improve the per- formance of our model by leveraging more data, setting longer context lengths, and adopting dynamic resolution as representative studies (Wang et al., 2024a). In addition, future work can employ such unified multi- models to act as agents (Yang et al., 2024b) for tool use. Furthermore, our qualitative results (Table 10 and Table 11) reveal that models trained on the same 665K instruction data exhibit similar response templates. This suggests that the patterns present in the data can impact the model's behavior. There- fore, future research can focus developing more diverse formats to enhance the flexibility of LMMs. Plus, the align- ment technique (Ouyang et al., 2022: Liao et al., 2024; Rafailov et al., 2024) can be used to align the LMMs with human preference. Finally, we envision that our model can extended to generation tasks using a similar approach, thereby achieving a unified understanding and generation framework without the need for separate encoders (Radford et al., 2021; Zhai et al., 2023) and decoders (Podell et al., 2023) in training efficient way.
ception, where LLaVA-1.5-7B fails to recognize the color of small objects and the number of objects outside the im- age center. For instance, LLaVA-1.5-7BB incorrectly identi- fied the color of the NBA player's socks. The second row shows examples of logical reasoning, where the lack of fine- grained perception ability leads LLaVA- 1.5-7B failure in tasks that rely it, such edge object perception and rea- soning, and highlighting regions in images. In contrast, our VLploVL, fusing raw image embeddings after the patch em- bedding layer, enhances its ability to perceive fine-grained image information. Therefore, it shows better performance on tasks relying on the capability of fine-grained perception.
Is it possible to use the next token prediction loss di- rectly? To validate the effectiveness of the modal expansion, we directly optimized the model using next-token predic- tion loss without the first stage. As shown in Figure 6, the model converged slowly when optimized directly, as it had to perform both modality fusion and text generation simultaneously. contrast, the model with the modal ex- pansion stage converged significantly faster. Furthermore, as shown in Table 5, we found that the model without the modal expansion stage exhibits 4.3% performance drop.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (No. 62422606, 62201484).
4.4. Visualization study
Impact Statement
In order to investigate whether text embeddings can dynam- ically capture visual clues, we visualize the attention map between text embeddings and visual embeddings after the pre-decoder, as illustrated in Figure 5. It is observable that the text exhibits an automatic response to regions of higher relevance. For example, it demonstrates responsiveness to objects located at the image edges as well as to textual el-
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential socictal consequences of our work, none which we feel must be specifically highlighted here.
9

---

HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding
References
Diao, H., Cui, Y., Li, X., Wang, Y., Lu, H., and Wang, X. Unveiling encoder-free vision-language models. wrXiv:240.111832 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Alt- man, S., Anadkat, S., et al. Gpt-4 technical report. arXiv:2303.0877 2023.
Dosovitskiy, A., Beyer. L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M.. Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I,, Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. NeurlPS, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv:2407.21783, 2024.
Bavishi, R., Elsen, E., Hawthorne, C., Nye, M., Odena, A., Somani, A., and Taşılla,, S. Introducing our mul- timodal models, 2023. URL https::///// http://wwwwwppepe ai/blog/ fuyu-8b.
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
He, K., Chen, X., Xie, S., Li, Y.. Dollár, P., and Girshick, R. B. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal 2023.
models with better captions. 2rXiv:2311.12777
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus), arXiv: 11X6:06600604411 2016.
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, 7.,, Duan, H., Wang, J.. Qiao, Y., Lin, D., et al. Are we on the right way for evaluating large vision- langua-a models? arXiv:2403.20338 2024a.
Hinton, G. Distilling the knowledge in neural network. drXiv:/503.02555 2015.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong. M., Zhang, Q. Zhu, X., Lu, L., al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguise linguistic tasks. In CVPR, 2024b.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., and Sadigh, Prismatic vims: Investigating the design space of visually-conditions language models. arXiv:2402.07865. 2024.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: /tts////mss/////////////ss///sl/ httpm/mssssssoooggall///oo org/blog/ cuna/.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. diagram is worth a dozen images. ECCV, 2016.
Chu, Y., Xu, J., Zhou, X., Yang, Q. Zhang. S., Yan, Z., Zhou, C., and Zhou, J. Qwen-audio: Advancing uni- versal audio understanding via unified large-scale audio- language models. 1xXiv:2311.07911 2023.
Li, B., Wang, R., Wang. G., Ge, Y., Ge, Y., and Shan, Y. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv:3307.16125 2023a.
Computations, C. https: httgs//g//ggiiggace....
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava-onevision: Easy visual task transfer. arXiv:2408. 03326, 2024a.
co co/datasets dolphin, 2023.
Dai, W., Li, Li, D.. Tiong, A. M. H., Zhao, J., Wang. W., Li, B., Fung, P., and Hoi, S. C. H. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurlPS, 2023.
Li, F., Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., and Li, C. Llava-next-interlee multi-image, video, and 3d large multimodal models. arXiv:240..07895, 2024b.
Tackling
Deng, J., Dong, W., Socher, R., L.-J,, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Li, Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, In ICMI., 2023b.
10

---

HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding
Li, Y., Du, Y., Zhou, K., Wang. J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. 2tt.:2305.10355 2023c.
Simens, M., Askell, A., Welinder, P., Christiano, F. Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
Li, Y., Zhang, Y., Wang, C., Zhong. Z., Chen, Y. Chu, R., Liu, S., and Jia, Mini-geming Mining the potential of multi-modality vision language models. arXiv:2403. 18814, 2024c.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv:2307.0/952, 2023.
Liao, H., Han, H., Yang, K., Du, T., Yang, R., Xu, Q. Xu, Z.,, Liu, J., Lu, J., and Li, X. Baton: Aligning text-to-audio model using human preference feedback. In IJCAI, 2024.
Qi, Z., Fang, Y., Sun, Z., Wu, X., Wu, T., Wang, J., Lin, D., and Zhao, H. Gpt4point: A unified framework for point-language understanding and generation. In CVPR, 2024.
Lin, J., Yin, H.,, Ping. W., Molchanov, P., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In CVPR, 2024.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Lin, T.-Y.. Maire, M.. Belongie, S.. Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zimick, C. L. Microsoft coco:
Common objects context. ECCV, 2014.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Liu, H., Li, C., Li, Y., and Lee, J. Improved baselines with visual instruction tuning. In CVPR, 2024a.
Liu, H., Li, C., Li. Y., Li, B., Zhang. Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, oer, and world knowledge, 2024b. URL
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In NeurIPS, 2024.
github. 1
Liu, H., Li, C., Wu, Q.. and Lee, J. Visual instruction tuning. In NeurlPS, 2024c.
ShareGPT. https: htttpp://sharepp.... com, 2023.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang. S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z.,, et al. Mmbench: Is your multi-mode modal model an all-around player? In ECCV, 2024d.
Su, J., Ahmed, M. H. M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024.
Loshchilov, I. and Hutter, F. SGDR: stochastic gradient descent with warm restarts. In ICLR. OpenReview.net, 2017.
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li. X., Guestrin, C., Liang. P., and Hashimoto, T. B. Stanford alpaca: An instruction-follloiing llama model.
http::///gthuub/ wtteee.cccmmaaaaeeeeeeeeeeeee cow/tatsu-lalala s stanford_alpacae 2023.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In ICLR, 2019.
Team, C. Chameleon: Mixed-moda early-fusion foundation models. arXiv:2405.09818 2024.
Lu, J., Clark, C.,, Zellers, R., Mottaghi, R., and Kembhavi, A. Unified-io: A unified model for vision, language, and mudti-modal tasks. In ICLR, 2022a.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang. S., et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.05550 2024.
Lu, P., Mishra, S., Xia, T., Qiu, L.. Chang, K.-.., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurlPS, 2022b.
OpenAI. Chatgpt. https://openal com/blog/ chatgpt/,200 2023.
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. CVPR, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, I..,
Van Den Oord, A., Vinyals, O., et al. Neural discrete repre- sentation learning. NewrlPS, 2017.

---

Hap A Single-Transforme Baseline for Multi-Modal Understanding
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Attention is all you need. In NeurlPS, 2017.
Zhou, H., Tang, L.., Yang, R., Qin, G., Zhang, Y., Hu, R., and Li, X. Uniqa: Unified vision-language pre-training for im- age quality and aesthetic assessment, arXiv:2406.01069, 2024.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, Ki, X., Wang, J., Ge, W., al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. MrXiv:2409.12193 2024a.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models, arXiv:2304. 2023a.
Wang, W., Lx, Q. Yu, W., Hong, W., Qi, J., Wang, J., Yang, Z., Zhao, L., Song, X., et al. Cogvlm: Visual expert for pretrained language models. arXiv:2311.03073 2023.
Zhu, W., Hessel, J., Awadalla, A., Gadre, S. Y., Dodge, J., Fang, A., Yu, Y., Schmidt, L., Wang, W. Y., and Choi, Y. Multimodal C4: An open, billion-scale corpus of images interleaved with text. arXiv: 2tte::0930099 2023b.
Wang, X., Zhang, X., Luo, Z., Sun, Q. Cui, Y., Wang, J. Zhang, F. Wang, Y., Li, Z., Yu, Q. et al. Emu3: Next-token prediction is all you need. arXiv:2409.1886; 2024b.
x.ai. Grok-1.5 1.5 vision preview, 2024. URL https://x. ai/blog/ /grok- 1.5v.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, Li, C., Liu, D., Huang, F. et al. Qwen2 technical report. arXiv:240.110771, 2024a.
Yang. R., Ma, H., Wu, J., Tang, Y., Xiao, X., Zheng, M., and Li, X. Scalablevit: Rethinking the context-oriented generalization of vision transformer. In ECCV, 2022.
Yang, R., Song, L., Li, Y., Zhao, S., Ge, Y., Li, X., and Shan, Y. Gpt4tools: Teaching large language model to use tools via self-instruction. In NeurlPS, 2024b.
Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et al. mplug-owl. Modulariza- tion empowers large language models with multimodality. a3Xiv:2304. 14178, 2023.
Ye. Q., Xu, H. Ye, J., Yan, M.. Hu, A., Liu, H., Qian, Q. Zhang, J., and Huang, F. mplug-ow12: Revolution- izing multi-modal large language model with modality collaboration. In CVPR, 2024.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. Mmmu: A massive multi- conti--issciplie multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In ICCV, 2023.
Zhan, J., Dai, J., Yc, J., Zhou, Y., Zhang, D., Liu, Z., Zhang, X., Yuan, R. Zhang, G., Li, L., et al. Anygpt: Uni- fied multimodal Ilm with discrete sequence modeling. arXiv:2402. /2226, 2024.
12

---

HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding
In the appendix, we first provide more details in Appendix A and more results in Appendix B. Secondly, implementation details are presented in Appendix C. Finally, more qualitative results showcased in Appendix D.
A. Model Details
Current token prediction loss. The PyTorch-like pseudo code for the proposed current token prediction loss is presented in Algorithm 1. The target for a text embedding is derived from the current token index instead of the next token index, which a key distinction from the next token prediction loss (Radford et al., 2019).
Algorithm Current Token Prediction Loss
def loss (hidden_state, tarqet_ids, embed_tokena)
Siiden_sstates [X, C] targat_ids- [N] embed_ tokens: [K, C]
N: the sequence length
K: the vocabulazy size
C: the dimension of hidden_state
gez logits
logits hädden_state embod_tokens, traaoostt.nnposseeeee....... (-2, -1) [N, K]
scale logits
logics logit_scall logits
calculate loss
loss w.cross_entronn (legita, target_ids)
return loss
embed tckens
frozen.
The positional embedding. Positional embedding plays a crucial role in models (Dosovitskiy et al., 2021; Yang et al., 2022; Zhou et al., 2024) based on the attention mechanism (Vaswani et al., 2017), as it enables the self-attention module to capture the spatial relationship between embeddings. Our pre-decoder adopts similar architecture to CLIP-VIT-L (Radford et al., 2021) but with the extra capability of accepting both image and text inputs. Therefore, pre-decoder must consider text position information in addition to image position information. To address this, we retain the learnable position embedding for image embeddings in the patch embedding layer and incorporate Rotary Position Embedding (RoPE) (Su et al., 2024) in each self-attention layer to inject positional information into the multi-modal embeddings.
B. More Results
Data in the pre-training stage. The pre-training stage is critical step in training reccipt since it enables the pre-decoder to acquire new text knowledge while retaining its inherited vision knowledge. The data composition in this stage can impact the model in processing multi-moda inputs, as it influences the model to learn effective representations of text and images. Therefore, we conducted an exploratory analysis of the training data used in this stage. Specifically, we omitted the alignment stage and proceeded directly to instruction tuning after completing the pre-training stage. Initially, we employed Vicuna- (Chiang et al., 2023) as the language model and CLIP-ViT-L. 14 (Radford al., 2021) with an input resolution of 224 the vision teacher. We directly mix 665 K instruction data and 558 K pre-train data of L.La VA-1.5,,, termed 'mix-vl', When training with this data, the average result was 59.0%. In this setup, the image position was fixed in the sequence, which may lead the model to learn shortcuts related to the position. This can hinder multi-modal fusion since the model may not learn to effectively integrate image and text embeddings. Therefore, we created an interleaved sequence by randomly combining 665 K instruction data and 558 K pre-train data, denoted as 'mix-v2', where each sequence may have multiple images. This data resulted in an average performance of 59.2% across multiple benchmarks. To further investigate the effectiveness of interleaved data in improving the performance on subsequent understanding tasks, we collected 730 samples from MMC4 (Zhu et al., 2023b) for training. Our findings indicate that using interleaved data for modal expansion improves the performance downstream tasks, particularly on ScienceQA (Lu et al., 2022b).
Next, we expanded the input resolution and used CLIP-ViT-L--1 (Radford et al., 2021) with an input resolution of 336
as the vision teacher for pre-training. When training with the 'mix-v2' data, the increased resolution resulted in a 1.5% improvement in average performance, with notable gains on GQA (Hudson & Manning, 2019), MMBench (Liu et al., 2024d), and MMStar (Chen et al., 2024a). At this resolution, we found that replacing MMC4 resulted in a similar improvement to that observed at the 224 resolution. Furthermore, we combined 730 data from MMC4, 665 K instruction data, and 558
13

---

HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding
LLM
Res.
Data-S1
Avg GQA
SQA
POPE
MMB
MMS
Vicuna-7B
224
mix-vl
59.0 59.0
61.7
84.6
56,9
32,8
Vicuna-7B
224
mix-v2
59.2
60.7
63.7
83.6
55.2
32.8
Vicuna-7B
224
mmc4
59.4
59.1
65.5
82.9
57.3
32.5
Vicuna-7B
336
mix-v2 60.7
61.8
63.8
84.6
59.5
33.6
Vicuna-7B
336
mmc4 61.1 61.9
67.0
84.4
58.3
33.7
Vicuna-7B
336
mix-v3
61.2
61.2
67.5
84.8
59.0
33.6
Vicuna-7B 336
mix-v4
61.3 62.0
67.3
84.7
59.5
33.3
Llama-3-83 336
mix-v4
64.2 63.1 70.5
84.8
63.0
39.4
Table 6: Ablation for data used in the first stage (Data-S1). The quantity of instruction data used in the second stage is 665
K. 'Res.' denotes the resolution of the vision teacher.
pre-train data to create 'mix-v3'. This mixture of data only brings a marginal improvement. Because our goal is to teach the model new text knowledge, we deemed it necessary to incorporate pure text data into the training process. Therefore, we mixed 665 K instruction data, 558 pre-train data, and 600 K pure text data (Computations, 2023; Taori et al., 2023; ShareGPT, 2023) to create 'mix-v4'. Training with this data resulted in a slight performance improvement. Nevertheless, confirm the importance of using pure text data teach the model new knowledge. Building this data, replaced Vicuna-7B with Llama-3-8B (Dubey et al., 2024) as the LLM, which brings significant improvement in performance from 61 .1.3% to 64.2%. This suggests that the ability conditioned on multi-modal sequences for effective reasoning is crucial in multi-modal understanding and reasoning.
Zero-shot accuracy on ImageNet. In the pre-trainin pre-training stage, we leverage the vision encoder of CLIP-ViT-L (Radford et al., 2021) as a teacher model to teach the visual knowledge for pre-decoder. evaluate the effectiveness of this stage, we assess the zero-shot accuracy on ImageNet (Deng et al., 2009) in Table 7. Specifically, when optimized directly using the next token prediction loss, pre-decoder fails to retain image classification capabilities as shown in the first line. In contrast, with the first stage. pre-decoder exhibits minimal performance drop compared to its teacher model. This outcome confirms the efficacy of the first stage in our training receipt.
LLM
Res. w/ S1 Data-S1 Avg
INIK INTK-Tcacher
Llama-3-88
8B
336
x
0.1
76.6
Vicuna-7B
224
mix-v2
59.2
71.7
75.5
Vicuna-7B
224
mme4
59.4
71.6
75.5
Vicuna-7B
336
v
mix-v2
60.7
72.3
76.6
Vicuna-7B
336
mme4
61.1
73.6
76.6
Vicuna-7B
336
mix-v3
61.2
73.6
76.6
Vicuna-7B 336
mix-vd
61.3
71.9
76.6
Llama-3-8B
336
V
mix-v4 64.2
72.4
76.6
Table 7: Zero-shot accuracy on ImageNet-IK IK (INIK) (Deng et al., 2009) after the pre-training stage. 'w/ S1' denotes the model tuned by the pre-training stage. 'Avg' refers to the average results on GQA (Hudson & Manning, 2019), ScienceQA- IMG (SQA) (Lu et al., 2022b), POPE (Li et al., 2023c), MMBench (MMB) (Liu et al., 2024d), and MMStar (Chen et al., 2024a) benchmark. Vision teacher is the CLIP-ViT-L (Radford et al., 2021), and KNIK-Teacher denotes its zero-shot accuracy ImageNet-1K. The first line is the results of the model optimized directly by the next-token prediction loss.
C. Implementation Details
C.1. Experiment Settings
We summarize the training settings of each stage in Table and Table 9, To maintain the training stability of the pre-training stage, we reduce the value of 32 as Siglip (Zhai et al., 2023) rather than change the model structure as Chameleon (Team, 2024). For HaploVL-7B, we train it as LLaVA (Liu et al., 2024c). After pre- pre-training, we first tune the connector between the pre-decoder and dest-decoder using 558 caption data (Liu et al., 2024a) and then fully tune the model using 665
14

---

HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding
instruction data (Liu al., 2024a). For HaploVL-SB with the ability input any resolution, first the whole model using 1.2 M caption data (Chen al., 2023) because the prior knowledge of the pre-decoder doesn't have any-resolution vision knowledge. Then, we tune the model using M instruction data. For HaploVL-8B-MI that supports the multi- image and video input, we continue training the single-image model (HaploVL-8B) using the mix of interleaved data and single-image data.
config
value
config
value
optimizer
AdamW (Loshchilov & Hutter. 2019)
optimizer
Adam (Loshchilov & Hutter, 2019)
base learning rate
x 10-4
base learning rate
2.0
10-6
weight decay
1.0 x 10=4
weight decay
0.0
optimizer momentum
81 0.9, 82 0.95 256
optimizer momentum
91
0.9, B2 0.98
batch size
batch size
128
learning rate schedule cosine decay (Loshchilov & Hutter, 2017)
learning rate schedule cosine decay (Loshchilov & Hutter, 2017)
Table 8: The first stage setting.
Table 9: The second stage setting.
D. Qualitative Results
We provide additional examples that demonstrate the capabilities of HaploVL in various tasks, including visual fine-grained perception, logistic reasoning, and image captioning. As shown in Table 10, HaploVL. excels in fine- fineggrinedd visual perception, including recognizing subtle color differences, object locations, and object quantities. Furthermore, HaploVL demonstrates correct logistic reasoning based on visual information, illustrated in Table 11. This is attributed to its exceptional fine-graine tme-grained perception ability, which is facilitated by early fusion. In terms image captioning, VL. generally produces accurate descriptions, as shown in Table 12. Notably, HaploVL-8B pays closer attention to image details compared VL- due to the higher quality and quantity of instruction data. Interestingly, models trained on 665 K instructions from LLaVA (Liu et al., 2024a) exhibit similar response pattern, suggesting that the inherent patterns in the data influence the behaviors of the model.
15

---

HaploVL: A Single-Transforme Baseline for Multi-Modal Understanding
Visual fine-grained perception examples:
User
What color is the ribbon that the the right holding?
LLaVA
The man on the right holding gold ribbon. (X)
Chameleon
The man on the right holding white and ribbon. (X)
EVE
The ribbon that the man right is holding green.
HaploVL-7BB
The the right holding green ribbon. (V)
User
Where is cat? Options: top-right, top-left, bottom-left, bottom-right
LLaVA
Top-left. (V)
Chameleon
The cat is on top of the blanket.(X)
EVE
Top-right. (X)
HaploVL-77
Top-left. (V)
User
How many balls are on the table? Options: A: 6, B: 4, C: D.
LLaVA
A. (X)
Chameleon
The answer is B: There are four balls on the table.(v)
EVE
A. (X)
Hap1oVL-7B
B. (V)
Table 10: Visual fine- fine-grained perception examples. HaploVL excels in recognizing subtle colors, object locations, and object quantities.
16

---

HaploVL: A Single-Transformee Baseline for Multi-Modal Understanding
Visual logistic reasoning examples:
User
Subtract all large gray things. Subtract all small brown metallic balls. How many objects left?
LLaVA
After subtracting large gray objects and small brown metallic balls, there are three objects left on the table: two blue balls and one gray ball. (x)
Chameleon
There are 12 objects left. (x)
EVE
There are three objects left: a blue sphere, green cube, and gray sphere. (X)
HaploVL-77B
After subtracting large gray things and small brown metallic balls, there are two objects left: blue sphere and green cube. (v)
User
What the basketball player the left doing in the image?
LLaVA
The basketball player on the left is trying to steal the ball from the player on the right. (v)
Chameleon
The basketball player on the is dribbling the ball with left hand and is about to shoot the with his right hand. (X)
EVE
The basketball player on left holding the ball and dribbling while standing on the court. (X)
Haplo VL-7B
The basketball player on left attempting steal the basketball from the player on the right. (V)
User
What will happen next?
LLaVA
image, a young boy with a green shirt crying and has his eyes closed. It not possible to predict exact next event, but likely that the boy's emotional state will continue, and may eventually open his eyes and explain the reason for his crying. Alternatively, he might be comforted by someone or something, which could help him stop crying. The situation highlights the importance of empathy and support for children during their emotional moments. (V, Hallucinations)
Chameleon
The image shows young boy with his mouth wide open. He has light brown hair and is wearing green striped t-shirt. There black background. He likely blow out some candles cake. (X, Hallucinations)
EVE
In the image, young boy with blond hair is making funny face, possibly sticking his tongue. It not possible to predict exactly what will happen next, but is likely that the boy will continue make funny face or gesture, possibly to entertain himself or others around him. The boy's facial expression suggests that he is in a playful mood, and he might engage in some lighthearted antics or jokes. (X)
VL-7oVL-7B
In the image, a young boy with blonde hair is shown crying with his eyes wide open. is difficult to predict exactly what will happen next, but considering the context of the image, it is likely that the boy's emotions will continue to be expressed through his tears. He might be expressing his feelings about something that has upset him causing him distress. Alternatively, the boy could be reacting a recent event or situation that affected him deeply. The image captures moment vulnerability and emotionality, and is important to he empathetic and supportive when encountering such situations. (V)
Table 11: Visual logistic reasoning examples. Haplo VL shows correct reasoning for the open open-ended questions.
17

---

Vap A Single-Transformer Baseline for Multi-Modal Understanding
Visual caption example:
NBA PREVIEW
LOS
SLCEH
CAVS
CAVS 0
2
CAVS
WW WANT
ALL THE KING'SMEN
SSURE
FOCUS.
LEBRON JAMES TAKES
TALENTS
BACK TO CLEVELAND
KANNI LEDNARD
THE AREN'T FINESHED
THE TEAM- TE-AMU ERTAINIENTT INDEX
THE SMALL BALL WORLD SERIES
User
Describe this image in detail.
LLaVA
The image features three professional basketball players posing for a photo on the cover of an NRA Sports magazine. They are all wearing yellow sports jerseys, showcasing their team spirit. The players are standing close to cach other, with player the left, another in the middle, and the third the right side of the image. In the background, there is sports ball, likely a basketball, placed near the center of the scene. The players all smiling and appear be enjoying the moment, capturing their camaraderie and dedication to the sport. (Distracted by text in images)
Chameleon
This image features two posters advertising the "All AtarStar Weekend" of the National Basketball Association (NBA). The image is portrait orientation and features two advertisements in the center of the image.
The advertisement on the left is titled "KINGS" white text with a hold, black outline. The text is placed above black-and-gold baskethall that is shown from the side, with the black side facing the viewer. The advertisement below the title features two athletes, one with their back to the viewer and one facing forward. The athlete on the left is wearing yellow Nike jersey with black sleeves and black number "20" on the back. They are holding black basketball in their right hand and have their left raised in the air. The athlete the right wearing gold Nike jersey with black trim and a black number "35" on the back. They are also holding a black basketball in their right hand and have their left arm raised in the air. The athletes are both wearing black shorts and white socks. The background of the advertisement is a black wall with a large, yellow text that reads "ALL-STAR WEEKEND" in bold, white text.
Both advertisements feature the Nike logo in the top left corerr of the advertisement, and the text "Nike" is printed in white in the top right corner of each advertisement. (Hallucinations)
EVE
The image features three men dressed in yellow basketball uniforms, standing close together and posing for picture. They are all wearing shorts and sports shoes, and one of them is holding a basketball. The men appear to be athletes, likely part of basketball team.
The image also includes few other elements, such as sports ball located near the center of the scene, and couple of books placed on the ground. The books are positioned on the left side of the image, and they seem to be unrelated to the main focus of the seene, which is the three men in their basketball uniforms. (Hallucinations)
HL-7oVL-7B
The image is magazine cover featuring three basketball players from the Cavaliers teamn The cover is titled "Sports Illustrated" and has subtitle "All the King's 's Men" with tagline "Lebron James takes his team-building talents to Cleveland." " The three players are standing next to each other, with the player in the middle holding a basketball. The cover also includes a note that the Cavaliers have won the NBA championship and mention the by-mm entertainment index. The magazine cover is designed to attract readers interested in basketball and the Cavaliers team. (More consistent with the image)
Table 12: Visual caption example. Ha provides a description that is more consistent with the image.
18

---

Vap A Single- Transforrre Baseline for Multi-Modal Understanding
Visual question answering example:
User
Do you know who drew these paintings? Please explain each image.
HaploVL-8B-
The first painting famous work by Vincent van Gogh, titled "The Starry Night". depicts swirling night sky with bright stars and bright creseent moon, set against backdrop of dark, rolling hills and a small village with lit windows. The painting is characterized by its bold and expressive brushstrokes, and vibrant use of color. It is widely regarded one of the most iconic and influential works of post-impressionist art. The second painting is also by Vincent pipe, wearing a blue hat, a white collar, and a green coat. The background a solid orange color, which creates strong contrast with the man's features. The painting is characterized by its bold and expressive use of color, as well as its swirling brushstrokes. which give it dynamic and vibrant quality.
MI
Gogh, titled "Self-PPrtrait with Pipe". It shows
with beard and
The third painting is also by Vincent van Gogh, titled "Self- Portratt... It shows
with beard and pipe, wearing blue suit and white shirt. The background is swirling pattern blues and greens, which gives the impression of a dynamic and vibrant energy. The painting is characterized by its bold and expressive use color, as well as its swirling brushstrokes, which give it a dynamic and vibrant quality.
In summary, the first painting is "The Starry Night" by Vincent van Gogh, which depicts swirling night sky with bright stars and bright crescent which shows a man with heard and pipe, wearing a blue hat, white collar, and green coat. The third painting is "Self-Portrai Portrait" by Vincent van Gogh, which shows man with a beard and pipe, wearing a blue suit and a white shirt, set against a swirling pattern of blues and greens.
The second painting is "Self-Portrai with Pipe" by Vincent van Gogh,
Table 13: Visual question answering example. Va able to process multiple images.
19


Reviews:
[{"review_id": "sJxvRsd7Gk", "reviewer": "Reviewer_yfeb", "summary": "The paper proposes an early-fusion method for vision-language reasoning. They claim to have pre-decoder that extracts visual information from raw vision embeddings based on text input and a post-decoder to process fused multi-modal embeddings and generate text responses. The experiments suggest that the method is better than many state of the art methods, including EVE.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions and feedback. We address the primary concerns as follows.\n\n* Q1: Clarify more beyond stating that early fusion leads to efficient data usage.\n\n**Response:** We propose utilizing a pre-decoder to fuse image and text data in the early stages of processing. **This pre-decoder is designed to inherit prior vision knowledge from a visual model (like ViT), thereby requiring minimal additional data.** The inherited prior vision knowledge enables the pre-decoder to effectively integrate visual information with textual data using fewer samples. Moreover, empirical results demonstrate that this method achieves superior performance with the same LLM and vision teacher, as evidenced by the MMStar benchmark scores (HaploVL-Vicuna-7B: 34.5 vs EVE-Vicuna-7B: 28.2).\n\n* Q2: Used for image generation.\n\n**Response:** Our current work verifies that this method is feasible for image understanding. In order to explore image generation, we introduce a vision loss by adding an image decoder during the second stage of our method. This approach is similar to the Masked Autoencoder (MAE), where image embeddings are decoded into RGB images. To assess the impact, we conducted experiments using the LLaVA-665K instruction data.\n\n| W/ visual loss | GQA  | MMStar |\n|----------------|------|--------|\n| True           | 60.8 | 34.0   |\n| False          | 62.5 | 34.5   |\n\nModels trained with vision loss perform worse, with GQA and MMStar scores dropping from 62.5 to 60.8 and from 34.5 to 34.0, respectively. While the symmetrical usage of visual and text modalities seems plausible, **the introduction of vision loss for image generation currently degrades the performance of our model in image understanding tasks. This is because additional vision loss conflicts with the textual loss used in multimodal understanding.** We plan future experiments to further explore and possibly mitigate these conflicts to successfully integrate image generation capabilities.", "claims_and_evidence": "The experiments correctly reflect the claims of the paper.", "methods_and_evaluation_criteria": "Yes, the proposed method seems reasonable and correct for the application.", "theoretical_claims": "No, I did not check theoretical claims. I believe the paper does not propose any theoretical claims either.", "experimental_designs_or_analyses": "The experiments look valid to the best of my knowledge.", "relation_to_broader_scientific_literature": "Overall, vision-language learning is important for various applications. The proposed paper fits correctly in the overall literature.", "essential_references_not_discussed": "None that I know of.", "other_strengths_and_weaknesses": "Please see questions for authors.", "other_comments_or_suggestions": "I write the overall strength and weakness of the paper here\n\nPositives\n\n- The proposed method is simple and effective.\n- The overall idea of early fusion is very interesting (though I am not very familiar with related work in this area and would seek help from other reviewers as well)\n- The experiments show good gains compared to the competitive baseline\n\nNegatives\n\n- It is not clear why this method would take less data compared to EVE. Can the authors clarify more beyond stating that early fusion leads to efficient data usage.\n- The limitations of the method are not clearly stated. Why can this method not be used for image generation if the setup is reversed, since the usage of visual and text modality looks symmetrical?\n\nOverall, I like the paper and want it to be accepted. Please clarify on the negatives in the rebuttal phase.", "questions_for_authors": "Please see comments or suggestions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "cOC6UcLouL", "reviewer": "Reviewer_6jcN", "summary": "The paper introduces HaploVL, an early-fusion multi-modal model (LMM) that processes visual and textual inputs through a single-transformer architecture. Unlike traditional compositional LMMs that handle modalities separately, HaploVL integrates raw visual and textual embeddings at an early stage, leveraging a pre-decoder to extract visual cues from text-guided attention and a post-decoder for deeper multi-modal fusion. By inheriting prior knowledge from pre-trained vision and language models (e.g., CLIP-ViT and Llama-3), HaploVL achieves competitive performance on fine-grained perception and reasoning tasks (e.g., MMVP and MMStar benchmarks) while requiring significantly less training data and computational resources. The model outperforms existing single-transformer LMMs (e.g., Fuyu-8B, EVE-7B) and rivals compositional LMMs (e.g., LLaVA-1.5) in specific benchmarks.\n\nStrengthens\n\n- Native Multi-Modal Architecture: HaploVL eliminates the need for separate vision/text encoders (e.g., ViT + LLM pipelines), simplifying the design and reducing computational overhead.\n\n- Efficient Early Fusion: By fusing visual and textual embeddings early, the model retains fine-grained visual details, enhancing performance on perception-heavy tasks (e.g., 4.9% improvement in fine-grained perception over LLaVA-1.5).\nData and Resource Efficiency: Leveraging pre-trained models significantly cuts training costs. For example, HaploVL-7B achieves superior results using only 1.2M training samples vs. EVE-7B’s 35M.\n\n- Clear Methodology: The two-stage training (pre-training for vision-text alignment and fine-tuning for instruction following) is well-structured, and the ablation studies (e.g., resolution scaling, data mixtures) validate design choices.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful suggestions and feedback. We have responded to the key concerns in the details below.\n\n* Q1: Limited Benchmark Comparisons.\n\n**Response:** While recent state-of-the-art LMMs such as QwenVL2-7B and InternVL2.5-7B achieve higher MMBench scores, it is important to note that these models use closed data despite releasing their model weights. According to their technical reports, QwenVL2-7B utilized 1.4 trillion tokens during the pre-training phase, and InternVL2.5-7B used 142 billion tokens. In contrast, our HaploVL-7B model relies solely on open-sourced data, comprising only 1.2 million samples (~7 billion tokens). This significantly lower data usage underscores the efficiency of our approach. Moreover, **our reliance on open-sourced data ensures that our model can be easily reproduced by the research community to build their single transformer models.** Nevertheless, we acknowledge the importance of more recent benchmarks and will incorporate the results of these state-of-the-art models into our main table for comprehensive comparison.\n\n* Q2: Context-Length Limitations\n\n**Response:** The context length of HaploVL is indeed scalable. Specifically, with an input resolution of 336, the context length is 2048 tokens, and it extends to 6144 tokens when the maximum image size is 672x672. Even at a context length of 2048 tokens, HaploVL-Vicuna-7B (HaploVL-7B) demonstrates superior performance compared to LLaVA-1.5-7B on the MMVP, MMStar, and SEED benchmarks.\n\n| Method         | context-length | MMVP | MMStar | SEED |\n|----------------|----------------|------|--------|------|\n| LLaVA-1.5-7B   | 2048           | 21.3 | 30.3   | 66.1 |\n| HaploVL-7B     | 2048           | 24.7 | 34.5   | 67.5 |\n\n* Q3: Impact of Retaining Visual Supervision in Stage 2\n\n**Response:** We retain a vision loss by adding an image decoder during the second stage of our method. This approach is similar to the Masked Autoencoder (MAE), where image embeddings are decoded into RGB images. To assess the impact, we conducted experiments using the LLaVA-665K instruction data.\n\n| W/ visual loss | GQA  | MMStar |\n|----------------|------|--------|\n| True           | 60.8 | 34.0   |\n| False          | 62.5 | 34.5   |\n\n**Models trained with vision loss perform worse**, with GQA and MMStar scores dropping from 62.5 to 60.8 and from 34.5 to 34.0, respectively. This is because additional vision loss conflicts with the textual loss used in multimodal understanding. We will include this result in the latest version.\n\n* Q4: Baseline Fairness\n\n**Response:** Due to the high workload involved in reproducing the EVE model (35 million data samples and 2 A100-80G GPUs running for 9 days), **we compared the performance of HaploVL utilizing the same Vicuna-7B language model.** The results demonstrate that HaploVL-Vicuna-7B outperforms EVE-7B. Specifically, on the SEED benchmark, HaploVL-Vicuna-7B achieves 67.5%, while EVE-7B (based on Vicuna-7B) only scores 54.3%.\n\nFurthermore, when utilizing the same Qwen2.5-7B language model, our HaploVL-7B-Pro also surpasses the performance of the improved EVE-2.0 (Qwen2.5-7B).\n\n| Model         | LLM         | SEED  | POPE  |\n|---------------|-------------|-------|-------|\n| EVE-7B        | Vicuna-7B   | 54.3  | 83.6  |\n| HaploVL-7B    | Vicuna-7B   | 67.5  | 85.4  |\n| EVE-2.0-7B    | Qwen2.5-7B  | 71.4  | 87.6  |\n| HaploVL-7B-Pro| Qwen2.5-7B  | 75.0  | 88.7  |\n\nThese results confirm that **the performance improvements are attributable to HaploVL’s architectural innovations rather than just the superiority of the LLM backbone.**", "claims_and_evidence": "Weakness：\n- Limited Benchmark Comparisons:\nOutdated Baselines: The paper focuses on older models (e.g., LLaVA-1.5, InstructBLIP) and lacks comparisons with recent state-of-the-art LMMs like InternVL2.5-7B or QwenVL2-7B, which achieve superior MMBench scores (>80).\nContext-Length Limitations: The authors acknowledge that HaploVL underperforms LLaVA-OV due to restricted tokenization (2,304 vs. 7,290 tokens), suggesting scalability challenges.\n\n- Unclear Impact of Training Strategy:\nThe decision to discard visual supervision in Stage 2 (post-decoder fine-tuning) lacks justification. Retaining visual losses might improve multi-modal alignment but risks overfitting.\nBaseline Fairness:\n\n- HaploVL-8B uses Llama-3-8B, while EVE-7B uses weaker Vicuna-7B. Performance gains could stem from Llama-3’s superior language capabilities rather than architectural innovations.\n\nQuestions：\n1. Impact of Retaining Visual Supervision in Stage 2\n -If visual losses are retained during Stage 2, the model might better preserve fine-grained visual-text alignment. However, this could also:\n\n- Improve Performance: By preventing catastrophic forgetting of visual features.\n- Harm Performance: If textual instruction tuning dominates the loss, visual signals could introduce noise.\n\n- The paper’s current approach (discarding visual losses) prioritizes language-focused instruction following. To resolve this, ablation experiments comparing both strategies are needed.\n\n2. Missing Comparisons with Recent Models\nThe authors should include benchmarks against InternVL2.5-7B and QwenVL2-7B, which excel in MMBench (scores >80). HaploVL’s MMBench score of 75.0 (HaploVL-8B-MI) falls short, suggesting architectural or scalability limitations. Potential solutions:\n- Expand tokenization capacity (e.g., longer context windows).\n- Incorporate high-resolution training (beyond 672×672).\n\n3. Fair Baseline Comparison\n- To isolate the impact of the proposed architecture (vs. Llama-3’s superiority), the authors should: Re-train EVE-7B using Llama-3-8B under identical settings.\n- Compare the revised EVE-7B (Llama-3) with HaploVL-8B. This ensures gains are attributable to HaploVL’s early fusion, not the LLM backbone.\n\n\n4.Minor Issues\n- Typo: Line 412: “Figure Figure 5.” → Correct to “Figure 5.”\n- Clarity: Clarify resolution scales (e.g., 336 vs. 672 in Table 3) and tokenization limits in the main text.", "methods_and_evaluation_criteria": "SEE Claims And Evidence", "theoretical_claims": "SEE Claims And Evidence", "experimental_designs_or_analyses": "SEE Claims And Evidence", "relation_to_broader_scientific_literature": "SEE Claims And Evidence", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "SEE Claims And Evidence", "other_comments_or_suggestions": "SEE Claims And Evidence", "questions_for_authors": "SEE Claims And Evidence", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0pO8PGwIBV", "reviewer": "Reviewer_y5EY", "summary": "HaploVL is a large, single-transformer multi-modal model designed to overcome the limitations of existing models by integrating visual and textual inputs early on for efficient multi-modal comprehension.\n\nThey introduce an innovative pre-decoder model that merges visual patches with text embeddings at the initial stage.\n\nTheir two-stage approach utilizes knowledge distillation to maintain vision capabilities while fine-tuning with visual instruction data. \n\nThey outperform previous single-transformer models in their results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions and feedback. We respond to the major concerns in the following.\n\n* Q1: The comparison method is not detailed enough.\n\n**Response:** Due to page limitations, we have compared our model against several widely-recognized methods, which include very mainstream approaches. For single-transformer LMMs, we have also included comparisons with the latest models.\nTo provide a more comprehensive comparison, we will add the results of more state-of-the-art models to our main table.\n\nHowever, regarding recent state-of-the-art separate LMMs, such as  QwenVL-2 and InternVL-2.5, it's important to note that these models used closed datasets, although their model weights were released. According to their technical reports, QwenVL-2 utilized 1.4 trillion tokens during pre-training, and InternVL-2.5-7B used 142 billion tokens. \n**In contrast, our HaploVL-7B relies solely on open-sourced data, using only 1.2 million samples (~7 billion tokens). This ensures that our model can be easily reproduced by the research community to build their single transformer models.**\n\n* Q2: The necessity of this design.\n\n**Response:** \n**Our design aims to explore a novel architecture that utilizes a single transformer during inference.** Building on the well-established vision knowledge acquired from web-scale image data by Vision Transformer, we leverage this by initializing our pre-decoder with ViT and employ it as the teacher model. This allows us to inherit its vision knowledge and significantly reduce the required data and training costs compared to other early-fusion and single-transformer models.\nFor example, while single-transformer LMMs like EVE and Emu3 have notable performance gaps when compared to separated LMMs such as LLaVA, **our approach strives to narrow this performance gap**. By integrating the strengths of ViT in the initial stages, we can achieve enhanced efficiency and performance, making our model a compelling candidate despite the apparent complexity.\n\n* Q3: The strong language model may narrow the performance gap.\n\n**Response:**\nWe conducted comparisons using the same LLM (Vicuna-7B) and equivalent data with LLaVA-1.5-7B. This can eliminate the effect of the strong language model. Our findings demonstrate that HaploVL-Vicuna-7B exhibits superior performance compared to EVE-7B and surpasses LLaVA-1.5-7B on fine-grained perception tasks.\n\n| Model        | LLM       | SEED | MMStar | MMVP |\n|--------------|-----------|------|--------|------|\n| EVE-7B       | Vicuna-7B | 54.3 | 28.2   | 19.3 |\n| LLaVA1.5–7B  | Vicuna-7B | 66.1 | 30.2   | 21.3 |\n| HaploVL-7B   | Vicuna-7B | 67.5 | 34.5   | 24.7 |\n\n**These results validate our architectural innovations rather than relying solely on the superior capabilities of a strong language model.** This demonstrates that our design contributes significantly to performance enhancement.", "claims_and_evidence": "This paper provides detailed experiment results and these results are convincing.", "methods_and_evaluation_criteria": "The authors propose a new model design using a single transformer and claim its practical significance.", "theoretical_claims": "There are no proofs in this paper that should be checked for correctness.", "experimental_designs_or_analyses": "Part of the experimental design is reasonable.\n\n1. The author proposes using a unique mask strategy and does not verify the effectiveness of this design.\n\n2. The author needs to verify the effectiveness of the distillation module, particularly when initializing the pre-decoder with a well-trained vit.", "relation_to_broader_scientific_literature": "Earlier models like LLaVA and BLIP2 use separate vision encoders and language models. Subsequently, EVE and FUYU propose using only a single transformer to process multimodal input simultaneously. \n\nThis work also proposes a single transformer model design which consists of a pre-decoder and post-decoder.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "1. The comparison method is not detailed enough and lacks performance comparison with mainstream methods.\n\n2. Is this design necessary? This model still includes a vit and a large language model and an even more complex projector. Even compared to the early LLaVA 1.5, the performance improvement of the model is not very significant.\n\n\n3. Can the effectiveness be validated on a smaller model? The strong language model may narrow the performance gap.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cPPKGnwUrd", "reviewer": "Reviewer_TtK1", "summary": "- This paper introduces HaploVL, a multimodal model with a dual Transformer decoder for joint vision-language processing.\n- The proposed two-stage training distills knowledge from a pre-trained model into the first decoder, integrating text and vision.\n- HaploVL extends the EVE approach by reusing aligned vision tokens in a second decoder module.\n- Built on CLIP-ViT-L and Llama-3-8B, HaploVL outperforms baselines across multiple multimodal benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your suggestions and feedback. In the following, we respond to the major concerns.\n\n* Q1: This early-fusion design is conceptually similar to related work such as EVE, offering only minor innovations over EVE when combined with a customized training recipe.\n\n**Response:** We distinguish our HaploVL from EVE through the following differences:\n\n| Difference      | EVE                                                           | Ours                                                                                        |\n|-----------------|---------------------------------------------------------------|---------------------------------------------------------------------------------------------|\n| Architecture    | Uses multiple attention layers to tokenize images (see Figure 3 in EVE's paper) | Utilizes a simple MLP to tokenize images, ensuring simplicity in inference                  |\n| Methodology     | Does not inherit vision knowledge, requiring 35M data for training                | Inherits vision knowledge, needing only 1.2M samples to achieve performance surpassing EVE |\n| Performance     | Shows a significant performance gap with separate models, e.g., 28.2 on MMStar                         | Achieves performance comparable to other separate models, e.g., 34.5 on MMStar                                    |\n\nThese differences highlight our architectural innovations and efficiency in training, which contribute to the superior performance of HaploVL compared to EVE.\n\n* Q2: The claim about \"end-to-end large multi-modal model in a single transformer\"\n\n**Response:** We acknowledge that the model employs a dual-decoder architecture during training. However, this dual-decoder setup is used exclusively for the training phase. **During inference, both the image and text inputs pass through the same single transformer model. This aligns our approach more closely with end-to-end processing.** Other models like EVE also utilize multi-stage training strategies.\n\nFurthermore, we have experimented with jointly training all components from the start using a unified objective. The results indicate that **\"training all components from the start using a unified objective\" yields lower performance compared to our two-stage training approach.** Specifically, the average score across 5 benchmarks was 60.5 for the two-stage method, versus 55.5 for the one-stage joint training.\n\n* Q3: The subsequent explanation in lines 210–215 does not sufficiently justify the claim that the architecture is encoder-free.\n\n**Response:** First, we do not claim that HaploVL is encoder-free. Second, we outline the differences between our pre-decoder and Vision Transformer (ViT) architecture:\n\n| Aspect                | ViT                          | Our Pre-decoder                                |\n|-----------------------|------------------------------|------------------------------------------------|\n| Input                 | Image-only                   | Image and text                                 |\n| Positional embeddings | Learnable positional embeddings | Rotary positional embeddings (RoPE)             |\n| Attention block       | Differs from LLM attention block | Same attention block as post-decoder           |\n| Performance           | Shows inductive bias, low fine-grained perception | Better fine-grained perception due to early fusion |\n\n* Q4: There should be Llama-3-8B based VLMs, e.g. MiniCPM-V2.5 and Llama-3.2.\n\n**Response:** We will include the results of MiniCPM-V2.5 and Llama-3.2 in our main table for comparison. However, it should be noted that MiniCPM-V2.5 utilizes 778M training data, whereas our best model is trained on less than 6M samples. Additionally, **we fully leverage open-source data, which ensures our model's ease of reproducibility.**\n\n* Q5: The teacher model of the text embedding.\n\n**Response:** It is the text embeddings of the language model. We will clarify this in the final version.\n\n* Q6: Ablation study for using the same pre-trained encoders and LLM.\n\n**Response:** First, we report the total parameters of the two decoders. The parameters of our two decoders are almost equivalent to those of EVE (7.3B vs 7B).\nSecond, we conducted comparisons using the same LLM (Vicuna-7B) and equivalent data with LLaVA-1.5-7B. Our findings demonstrate that HaploVL-Vicuna-7B exhibits superior performance compared to EVE-7B and surpasses LLaVA-1.5-7B on fine-grained perception tasks.\n\n| Model        | LLM       | SEED | MMStar | MMVP |\n|--------------|-----------|------|--------|------|\n| EVE-7B       | Vicuna-7B | 54.3 | 28.2   | 19.3 |\n| LLaVA1.5–7B  | Vicuna-7B | 66.1 | 30.2   | 21.3 |\n| HaploVL-7B   | Vicuna-7B | 67.5 | 34.5   | 24.7 |\n\nThese ablation results validate our architectural innovations and demonstrate significant performance enhancements.", "claims_and_evidence": "- The first main claim in lines 24–27 and 96–99 is overclaimed. The authors state, “First, we propose a new early-fusion LMM that can fuse multi-modal inputs in the early stage and respond to visual instructions in an autoregressive manner.” However, this early-fusion design is conceptually similar to related work such as EVE, and although this work employs a different teacher encoder to incorporate prior knowledge and reuse aligned tokens in a second decoder, the core idea of the first decoder remains the same.\n- The second main claim is also inaccurate. The claim in lines 21–24 that the model is an “end-to-end large multi-modal model in a single transformer” is misleading. In reality, the proposed system is a dual-decoder architecture trained in two stages with different objective functions at each stage. In a truly end-to-end system, all components would be jointly trained from the start with a unified objective, and calling the dual decoder system a “single transformer” is imprecise.\n- Additionally, the authors argue that the method is encoder-free. However, they do make use of pre-trained encoder parameters. In specific, l.188-189 ‘For the input text $Xt$, we leverage the pre-trained LLM’s embedding matrix W to convert each text token into a vector within LLM’s space $R^l$.’, and l.207-210 ‘Notably, although the pre-decoder inherits prior knowledge from a vision encoder, it differs from the vision encoder.’ The subsequent explanation in lines 210–215 does not sufficiently justify the claim that the architecture is encoder-free.", "methods_and_evaluation_criteria": "Given the proposed method use Llama-3-8B as base LLM, in Table 2 there should be Llama-3-8B based VLMs, e.g. MiniCPM-V2.5 [1] and Llama-3.2.", "theoretical_claims": "- Many arguments in this work are speculations and lack theoretical or empirical support.\n    - l.47-l.50 ‘Our model fuses the vision and text embeddings at an early stage, enabling text embeddings to **autonomously acquire the necessary vision cues**.’\n    - l.117-120 ‘our HaploVL fused the visual and textual input in the early stage and **extracts the necessary vision information based on the text input.**’\n    - l.244-547 ‘When the text and image are jointly input into the pre-decoder in a mixed way, **semantic text embeddings can autonomously acquire the necessary vision cues from raw vision embeddings**.’\n- The teacher model of the text embedding from pre-deocder seems not introduced in l.248-l.274.", "experimental_designs_or_analyses": "- An ablation study is needed using a single decoder with a size equal to the combined size of the two decoder layers—that is, an EVE-style setup where the decoder's capacity matches that of the two decoders used in this work.\n- An ablation study is also required for an encoder–decoder–based approach that utilizes the same pre-trained encoders employed as teachers in this work (e.g., CLIP for vision and Llama-3 for text).\n- In Table 4 and lines 358–359, the experiment “to verify whether the LMM using one single transformer has advantages over separate models” is important, but it lacks setup details. Specifically, clarify whether the single transformer in EVE-7B has the same number of decoder layers and/or parameters as the combined decoders in HaploVL-7B, rather than just matching the second decoder (the LLM).", "relation_to_broader_scientific_literature": "See below the ‘Essential References Noe Discussed’ section", "essential_references_not_discussed": "[1] Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z. and Chen, Q., 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.", "other_strengths_and_weaknesses": "- The presentation and experiments are sufficient.\n- The technique’s novelty is modest, offering only minor innovations over EVE when combined with a customized training recipe.\n- The writing—particularly the claims about key contributions—requires more careful revision. There are numerous overclaims throughout the manuscript, the authors should ensure that their claims accurately reflect their contributions.", "other_comments_or_suggestions": "l.185-186: ‘it needs to acquire prior textual knowledge the Llama model’ → ‘it needs to acquire prior textual knowledge from the Llama model’", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces HaploVL, an early-fusion multimodal LLM that processes visual and textual inputs through a single-transformer architecture. After rebuttal, it received scores of 3344, and one reviewer increased their score from 2 to 3.  Overall, the reviewers are happy about the paper after rebuttal, commenting that the overall idea of early fusion is interesting, and the experiments show good gains compared to competitive baselines.

On the other hand, I also share some concerns with the reviewers. That is, (1) the paper mainly compared with relatively old baselines, and results on more challenging benchmarks, such as those involving text-rich image understanding tasks that demand high-resolution image encoders, are not included to more comprehensively benchmark the model. (2) The writing—particularly the claims about key contributions—requires more careful revision. There are some overclaims in the manuscript that needs revision.

Overall, the AC thinks this paper still presents some interesting ideas, and the strengths outweigh the weaknesses, therefore, would like to recommend acceptance by the end.
Decision: Accept (poster)