Unveiling Encoder-Free Vision-Language Models
Authors: Haiwen Diao
Yufeng Cui
Xiaotong Li
Yueze Wang
Huchuan Lu
Xinlong Wang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Large Vision-Language Model
Encoder-Free Multi-Modality Model
Pure Decoder-only Architecture
Primary area: machine_vision
TL;DR: We explore an encoder-free vision-language model to untangle the bottlenecks of decoder-only architectures in visual perception and understanding.

Abstract: Existing vision-language models (VLMs) mostly rely on vision encoders to extract visual features followed by large language models (LLMs) for visual-language tasks. However, the vision encoders set a strong inductive bias in abstracting visual representation, e.g., resolution, aspect ratio, and semantic priors, which could impede the flexibility and efficiency of the VLMs. Training pure VLMs that accept the seamless vision and language inputs, i.e., without vision encoders, remains challenging and rarely explored. Empirical observations reveal that direct training without encoders results in slow convergence and large performance gaps. In this work, we bridge the gap between encoder-based and encoder-free models, and present a simple yet effective training recipe towards pure VLMs. Specifically, we unveil the key aspects of training encoder-free VLMs efficiently via thorough experiments: (1) Bridging vision-language representation inside one unified decoder; (2) Enhancing visual recognition capability via extra supervision. With these strategies, we launch EVE, an encoder-free vision-language model that can be trained and forwarded efficiently. Notably, solely utilizing 35M publicly accessible data, EVE can impressively rival the encoder-based VLMs of similar capacities across multiple vision-language benchmarks. It significantly outperforms the counterpart Fuyu-8B with mysterious training procedures and undisclosed training data. We believe that EVE provides a transparent and efficient route for developing pure decoder-only architecture across modalities.

Unveiling Encoder-Free Vision-Language Models
Haiwen Di Yufeng Cui Xiaotong Li Yueze Wang' Huchuan Lu Xinlong Wang
Dalian University of Technology "Bejjig Academy of Artificial Intelligence Peeingg University diachw@mail dlut. alwt.edu.cc.. en, yfcuiebaai. en, lixiaotong@atu. www.settttuuuuuuuu...uuu yzvangubaai. ac. en, Ihchuan&dlut. eew.aalllttuuuueeeoo en, vangzinlong@baai.cc ac. Code & Models: https: sttps://githhn github. comc baaivision/EVE
Abstract
Existing vision-language models (VLMs) mostly rely on vision encoders to extract visual features followed by large language models (LLMs) for visual-language tasks. However, the vision encoders set strong inductive bias in abstracting visual representation, egg., resolution, aspect ratio, and semantic priors, which could impede the flexibility and efficiency of the VLMMs. Training pure VLMMs that accept the seamless vision and language inputs, ie.. without vision encoders, remains challenging and rarely explored. Empirical observations reveal that direct training without encoders results in slow convergence and large performance gaps. In this work, we bridge the gap between encoder-based and encoder-free models, and present simple yet effective training recipe towards pure VLMs. Specifically, we unveil the key aspects of training mncoder-free VLMs efficiently via thorough experiments: (1) Bridging vision-language representation inside one unified decoder; (2) Enhancing visual recognition capability via extra supervision. With these strategies, we launch EVE, an encoder-free vision-language model that can be trained and forwarded efficiently. Notably, solely utilizing 35M publicly accessible data, EVE can impressively rival the encoder-based VLMSs of similar capacities across multiple vision- vision-language benchmarks. significantly outperforms the counterpart Fuyu-8B [3] with mysterious training procedures and undisclosed training data. We believe that EVE provides a transparent and efficient route for developing pure decoder-only architecture across modalities.
Introduction
Recently, significant advancements in Large Language Models (LLMs) have catalyzed the emergence of Vision-Language Models (VLMs), showcasing powerful visual perception and cognition capability in visual question answering, image captioning, world knowledge understanding, etc. Typically, vision encoders (e.g., CLIP [61] and EVA [22]) focus on extracting highly-compressed visual-semantic representations, succeeded by adaptable language models (e.g., LLaMA [75] and Vicuna [10]) to handle ausion-language alignments and instruction instruction-following requirements. Nevertheless, these encoder-based VLMs have several potential drawbacks shown in Figure
(1) Image Resolution/ / Aspect Ratio. Existing LVMs are pre-trained with square and fixed-size images. However, this restriction forces VLMs to resize, pad, partition images of varying shapes [48, 84, 51, 20], resulting in large layout distortion, fragmented connection between image slices, and extra computational burden [8 especially in terms of high-resolution images.
(2) Deployment Overhead. Generally, LVMs and LLMs are executed successively. The growing scale of LVMs [8, 73, 13] severely undermines computational efficiency in real-world deployment, especially when high-resolution images are divided and processed through LVMs multiple times.
'Equal contribution. Correspondence Ihchuan @ eluttedu..c,, wangxinlong @ bagi.ac.cm.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(A) Encoder- Large Vision- Language Model
(B) Encoder-free Large Vision-Language Model
Large Language Model
Existing issues
Large Language Model
dowed:
PalM
Chade
ERNDE
Resolution/ Aayeot Ratio
JRE
(HI Reasnning
Upen:
LAMA
GLM
Prealyging
Encoder- free
VB
Projector
Whin AuI FBo......
WEN DN AAE
(00 Fullyakgringg
AED
& Permer
Vision Enander
Vatton
Percalsing
Lanpunge Ressering
Replogement Overthead
LLM
Effisienyy 2
Notivatlen:
Cla We Recaove Vusie Roooder tooss L oas We Counaler 12.2 Rooodde thes LVVLad can We Eitlccestty Buod is Revals Booodder beaod LILL55
Encoder+ LVLMS
VS parameters
Property:
Capacity Proportion
Decadeeenntyy timpler Model Architieture
Axpect
Multi-mrdality bridging
LLM
VI
Hice 2o Ratsince?
2 Vision Blerrgtition entionted by Viaton Rrcester Supervaccon
Kay core:
LLM controo Pro daiging Stage Dood Couns Ree Canvengenze
Figure 1: Overview of large (A) encoder-based and (B) encoder-free vision-language models. Encoder-based VLMs contain vision encoders (VE) and large language models (LLM), with projector as vision- rision-language bridge, while encoder-free VLMs exclude vision encoders and handle vision perception and linguistic instruction simultaneously with one unified architecture.
(3) Model Capacity between LVMs and LLMs. Existing LVMs and LLMs are compartmentalized during the pre-training stage, raising uncertainty about how to match their capacities and capabilities. As the scale of LLMs rises from 1.3B to more than 540B, how to strike corresponding vision encoders to maximize their respective abilities remains tricky and elusive.
Given the above considerations, we wonder that: Is it possible to bypass the constraints of vision encoders and integrate perception and reasoning capabilities into single unified architecture? Nevertheless, we empirically discover that even with enormous training data and large model sizes, this conceive remains highly challenging, suffering from greatly slow convergence and large performance gaps compared to encoder-based VLMs (e.g., Fuyu-8B [3] vs. LLa [49]). The essential problems of constructing encoder-free VLMs from scratch stem from:
(1) Representation Unity and Emergent Ability. Due to lacking high-quality image-text data, directly training encoder-free VLMs from scratch is impractical and expensive to learn abundant knowledge and universal representations across modalities. In contrast, the industry has capitalized on the wealth of language data and continuously updated various LLMs [75, 60, 5] with impressive capabilities. Consequently, we position LLMs as a central pivot and strive to compel LLMs per se to develop visual perception while preserving original linguistic proficiency. The process is not a light breeze. We observe that before scaling up pre-trained data, vision- lision-language pre-aligning from an LLM-centric perspective does count. This dramatically prevents model collapse and optimization interference, leading to rapid yet stable model convergence and great performance gains.
(2) Visual Recognition Capability. Contrastive learning [61, 62]. masked image modeling [45, 86] and auto-reeressive generation [21, 16, 19] essentially attempt to prompt visual backbones to produce highly compressed holistic semantics and frequently neglect fine-grained visual clues [14, 15, 17, 18]. On the contrary, we transmit visual signals almost losslessly into encoder-free VLMs. This operation can allow VLMs to autonomously acquire the necessary visual-semantic information. By aligning with patch features captured from vision backbones and text labels predicted by encoder- based VLMs, we implicitly embed visual encoders with input-restricted patterns into encoder-free VLMS for comprehensive visual perception and cognition. This also sidesteps the expensive re-training process of visual encoders for arbitrary image shapes inside encoder-based VLMS.
From the above perspective, we launch EVE-7B, an encoder-free VLM evolved from Vicuna-7B [10] and trained with two 8-A A-00 (40G) nodes in -9 days. The encoder-free property naturally supports high-resolution images with arbitrary aspect ratios. Notably, solely utilizing 35M publicly accessible data, we efficiently construct decoder-oni) VLM that can impressively rival the ancoder-based VLMs of similar capacities across multiple languaeee benchmarks. More importantly, we are pioneering a transparent, efficient, and practical route for subsequent VLM research. Our current version significantly outperforms the counterpart Fuyu-8B [3], which relies on mysterious training procedures and undisclosed training data. Beyond facilitating the transition from L.LMs to VLMS, EVE holds promising potential for developing scalable and efficient training strategies for encoder-free freee multi- modti-modality models, requiring fewer training data and device resources.
2

---

2 Related Work
2.1 Encoder-based Vision-Language Model
With the remarkable advancements in large language models (LLLMs) [75, 60, 1, 24, 76] and large vision models (LVMss) [61, 22, 73, 8, 13], integrating L.MSs with LVMs has become mainstream for building vision-language models (VLMs) effectively. Commercial VLMs extend the capabilities of their proprictary LLM to incorporate images, texts, audio, and videos, including Anthropic's
Claude-3V series [1]. StepFun's ss Step-IV [68], xAI's Grok- 1.5V [79], Apple's ' MMI [58], Google's Gemini series [74], and OpenAI's GPT-4V [83]. In terms of open-source VLMs, existing methods (e.g., BLIP series [42, 43, 121, LLaVA series [50, 49, 51], Emu series [72, 70], Intern-VL. VL [8, 9], and
etc) have demonstrated promising performance by employing simple intermediate layers to bridge the gap between LVMs and L.Mss. Recently, some studies [48, 49, 20, 28] have recognized the significance of input image resolution and aspect ratio for visual perception and cognition, particularly in the interpretation of document, chart, table, and infographic data. However, limited by pre-trained resolution, vision encoders are compelled to partition images into multiple slices or explore a -ual-ppth architecture for low-resolution and high-resolution images respectively, resulting in significant image distortion, fragmented relationship between image slices, and additional computational consumption. For model deployment, some popular open-source libraries, e.g., SGLang [91] or vLLM [39] already support inference acceleration for rugo-regressive prediction of encoder- VLMs. However, as the capacity of vision encoders scales up, the deployment efficiency of vision models would increasingly become a bottleneck for encoder-based VLMs. Meanwhile, how to match the capacities and capabilities between LVMs and L.LMs for masdd VI.MS remains highly debatable problem with no definitive conclusion. Some studies [49, 51] highlight the notable benefits via substituting CLIP-ViT-B with stronger CLIP-VVTT--3336x in enhancing multimodal models alongside Vicuna-7B [10]. Conversely, other findings [65] indicate that larger vision encoders may not be necessary, as features of multi-scale smaller ones can approximate their performance. Moreover, recent state-of-the-arr approaches [9, 70] exhibit significant performance improvements by introducing extremely LVMs. Based on these observations, we attempt to explore pure decoder-only VLM excluding vision encoders and integrate language understanding and reasoning capabilities into one unified architecture. This could effectively bypass the inherent problems inside encoder-biic based VLMs, including input constraints of pre-trained vision encoders, inefficiency issues of application deployment, and tricky capacity trade-offs between LVMs and L.LMs.
2.2 Encoder-free Vision-Language Model
Fuyu-8B [3] stands out as a pure decoder-only network that processes image inputs without relying on an image encoder. Fuyu-8B's design naturally handles high-resolution images with arbitrary aspect ratios, because image patches are fed directly into the model through simple linear projection layer. However, it demonstrates only average performance across lasgun-language benchmarks and lacks transparency in training strategies and data sources. This straightforward architecture has inspired further research [40, 31, 44], which focuses on developing powerful supervised instruction datasets to further enhance application capabilities. In response, we explore practical and promising direction toward developing pure VLMs and breaking the obstacles between encoder-base based and encoder free VLMs. We reveal two crucial lessons: (1) Before scaling up pre- mre-trained data, it is essential to prioritize vision-language alegmnetment from an LLM-centric perspective. This foundational step stabilizes the training process and alleviates optimization interference for integrating visual and linguistic information. (2) Enhancing image recognition capability via visual representation supervision and language conceptual alignment generates stronger visual representations for various vision-language tasks. With publicly available web data, our strategies efficiently accelerate model convergence and achieve performance that can surpass Fuyu-8B and match encoder-based VLMs.
3 Methodology
3.1 Model Architecture
Figure 2 illustrates the overall framework of our proposed EVE. Firstly, we initialize decoder-only EVE by Vicuna-7B [10] to acquire rich linguistic knowledge and strong instruction-followie ability. Under the premise of encoder-free VLMs, we then construct lightweight patch embedding layer and
3

---

(b)
Patch Aligning Layer
Imsaee
Image Spiit
AOT
AGT
work
Large Vision Language Model
Caussi Decoder-only Architecture)
What
(8) Patch Embedding Layer
Input Imagee:
Input Text:
What to AGI AGI car work as well as Humann AGI would nos be limited to apecific tacki or dounatne like existing Al cystemee but would have strong stility the (eceraltied inteiltgenee.
<CLS>
<BPL
Figure 2: Overview of our proposed EVE. We start by encoding images using concise (a) patch embedding layer, and then concatenate the patch and word features into decoder-only network. Next, we facilitate image perception through visual representation supervision using (b) patch aligning layer, and achieve linguistic conceptual alignment using next-word prediction pretext task.
(a) Patch Embedding Layer
(b)
Aligning Layer
Patch Embed
Pool
Cross Cross-Attentionn Attention
<LS>>
y
Feed Porweay
Delete <CLS SPL>
<cL/ssp>>>
Reshape Patch
Adaptive Pool
CA3
Patch
Delete /CLL/PADD
Vision Encoder
Flatten
Layer-wir
Plattem
MSE Loss
Copp
<SPL>
Figure 3: Overview of (a) patch embedding and (b) patch aligning layer. The former layer encodes images into patch features and employs cross-attention (CA1) within a limited receptive field to enhance representations. Meanwhile, a special <CLS> token provides a holistic view each patch feature in the subsequent backbone. The latter layer removes all meaningless tokens and adjusts the size patch features to align with the semantics of the same positional features in the vision encoder.
encode image and text inputs efficiently into the above network. After obtaining the network outputs, we attempt to align patch features with pair-wise ones from the vision encoder (VE) [61] through hierarchical patch aligning layer. Meanwhile, EVE predicts next-word labels that are generated by multi-source encoder-based VLMSs via Cross-Entropy (CE) loss.
Patch Embedding Layer (PEL). We introduce a succinct and trainable structure in Figure 3(a) to transmit images almost losslessly, rather than using deep encoders or tokenizers compress image information into high-level semantic representations. Given an image with (H, W) resolution, we first apply convolution layer obtain 2-D feature map with (h, w) sizes. To flexibly control computational overhead, we then implement an average pooling layer within each uncrossed feature slice. To further enhance these downsampled features. a Cross-Attention (CA1) layer employed in limited receptive field between each resulting feature and its corresponding slice. Besides, we employ a Cross-Attentinn (CA2) layer between a special <CLSS> token and all patch features. The obtained feature serves as the starting symbol of the image and provides holistic information for patch features in the subsequent backbone. Considering the varying aspect ratios of image inputs, insert learnable newline <SPL> token the end of each row of patch features. This helps the network understand the 2-D spatial structure and dependencies of the image. Finally, we flatten these features and pass them through two-layer Feed Forward layer, which embeddings into one unified decoder-only architecture. Note that the aspect ratios input images be arbitrary, requiring predefined sets, absolute position embedding, partitioning operations to accommodate a pre- pre-trainnd vision encoder.
then concatenated with text
Patch Aligning Layer (PAL). Besides moarse-grainee text supervision, we further facilitate fine- grained representations by learning from the pre-trained vision encoder. Moreover, challenging to establish shared space for aligning visual features with the vision encoder's 's output and mapping

---

Stage 1: L-Luudddd Pre-aligning
Stage 2: Generative Pre-training
Stage 3: Supervised Fine-tuning
PAL
PAL
PAL
E VE
E VE
E VE
PEL
PRL
PEL
Training:
T> Stage is the key essence for efficient training in Stage 2
Inference: Delete PAL supervision
w/ LLMM-gudedd Pre-slighing
w/o L..M-guided
w/o Pre-aligning
Robust! !
EVE
Unstable
Accelerate!
Collapse
PEL
Figure 4: Overview of training procedure with three successive stages. We perform initial vision- language alignment guided by a frozen LLM in Stage I, and then update the entire backbone for Stage 2 and Stage 3. We empirically find that Stage is quite crucial to avoid collapse and accelerate convergence, thereby enhancing training efficiency. Notably, PAL is removed during inference.
text features to language vocabulary simultancously. Hence, we explore hierarchical aggregation strategy in Figure 3(b) to integrate intermediate features across layers from entire layers (Interval ()) Specifically for the vision encoder, we discard meaningless <CLS><<PAD> tokens from its final output and record the 2-D shape of valid patch fields. For EVE, we first exclude <CLS>/<SPL> tokens from intermediate features of selected layers, and reshape the sequential features back into their original 2-D shape, each of which aligns with the previously recorded shape from the vision encoder via an adaptive pooling layer. We then implement& layer-wise tokens from the last layer as the Query and corresponding positional tokens from other layers as the Key and Value. We normalize the token features obtained from multi- aggregation to better match the normalized one from vision encoder one-to-one, utilizing Mean Squared Error (MSE) loss. Such an operation "implicitly" compresses a vision encoder with absolute position embedding (small resolution, fixed aspect ratio) into the decoder-only EVE framework (flexible resolution, arbitrary aspect ratio), enhancing the visual perception ignored by overly simplistic captions.
(CA3) function, using
3.2 Training Procedure
In Figure 4, the entire training process is categorized into three successive stages, where train EVE with publicly available image data captioned by existing VLMs, diverse QA data, and multi mudti-modality dialogue datasets. Notably, after finishing training, remove PAL supervision during inference.
LLM-guided Pre-training. The primary goal of this stage to establish an initial connection between vision and language modalities. thereby driving LLM to understand the visual concepts and entities in the images from scratch. We introduce publicly available web-scale data in Table including image-only: SA-1B [36]. OpenImages [38]; and image-text: LAION [63]. We remove noisy text captions and reproduce 33M high-quality descriptions via Emu2 (17B) and LL- (13B) as EVE-capssar. Here, only patch embedding and aligning layers are trainable to align with frozen Vicuna-7B [10] for better initializing an encoder- free VLM. Considering that these layers have limited parameter capacities and capabilities, we only adopt 16M of 33M image-text data (EVE-cap1e/ssh) 3a in this stage. We minimize CE loss with text labels and patch-wise MSE loss between EVE and vision encoder. We empirically discover that Stage does count for efficient training, as prevents collapse and accelerates convergence throughout the entire process.
Generative Pre-training. In this phase, we explore in-depth pretraining strategies by unfreezing patch embedding and aligning layers, and the full LLM modules for comprehending vision-language contents. Here, we make use of all 33M image-text pairs (EVE-capssar), and keep both text CE loss and image MSE loss as training objectives. Nevertheless, we discover that though multi-modality performance gradually increases, language capability suffers from significant downtrend. To strike
5

---

Table 1: Details of pre-training datasets. We only extract 16/33M data for Stage 1, and utilize the full 33M data for Stage 2
Table 2: Details of fine-tuning datasets. We only adopt LLaVA-mix-6655 for standard EVE-7B, and further train EVE (HD) with high resolution and all datasets
Dataset
Captioner
#Samples
Task
Dataset
#Samples
SA-IB [36]
LLaVA-I55 (13B)
11M
A12D [33D Syuthdog [34], DVQA [32]
OCR-VQA
270K
Openlmages [38]
Emu2 (17B)
7M
CharQA [57]. Doc [1]]
LAION [63]
Emu2 (17B)
15M
Jisoon-Flan [82]. Bunny-695K [27]
Multi-Task
1.5M
EVB-capssM
Total
33M
LJaVA- [49]
a compromise between enhancing vision-language capacities and maintaining linguistic competency, we opt for a relatively lower learning rate for stable optimization. Alternatively, involving language- only data remains a prospective avenue for further exploration [53]. Stage 2 serves as a catalyst, driving EVE per se to develop vision- -aaiuuge encoding and aligning capabilittess
Supervised Fine-tuning. After obtaining a well-aligned state, EVE is expected to possess further capabilities for following linguistic instructions and learning dialogue patterns. The entire architecture updated using the same loss functions in Stage 2. Here, we utilize -LaVA-mix-6665 [49] (LLaVA- mixe65x) with both VL and pure NLP dialogues as supervised fine-tuning (SFT) data to obtain the standard version of EVE-7B. Besides, we also attempt to enlarge the limitation of maximum resolution only in the SFT stage. To bridge the resolution gap between pre-training and fine-tuning stages, we further involve 1.2M SFT conversation data, as listed in Table 2 (including A12D [33], VoA [11], and etc) to develop high-resolution version, dubbed EVE-7B (HD).
4 Experiments
4.1 Training Settings
Data Preparation. (1) Pre-training Datasets. We train EVE using 33M publicly accessible samples from SA- IB- [36], OpenImages [38], and LAION [63]. Following pre-procsssing pipelines [61, 46], we retain only samples with image resolutions higher than 448 448. In particular, the text data from LAION [63] are noisy and simplistic, lacking linguistic complexity. Besides, it suffers from duplication problems [78], which heavily hinder image diversity and quality. To extract more representative images, we apply K-means clustering technology on image features extracted by EVA-CLIP [71]. This process results in 50,000 clusters, from which we selected the 300 images closest to each cluster center, yielding a carefully curated subset of 15M image samples from LAION [63]. Subsequently, we utilize Emu2 (17B) and VLL (13B) to regenerate high-quality image descriptions for the above three datasets and eliminate image-text samples with repetitive text or incomplete sentences. (2) Supervised Fine-tuning Datasets. We import LLa 1aa-mmiix6655 [49] for the standard EVE-7B, and collect a blended set of A12D [33], Synthdog [34], DVQA [32], ChartQA [57], VOA [11]. Vision-Flan [82]. and Bunny- [27] for high-resolution version.
Implementation Details. EVE-7B is developed from Vicuna-7B [10] for vision-language domains. To control complexity, we limit the longest image edge to 672 for EVE-7B and 1344 for EVE-7B (HD), maintaining image aspect ratios unless otherwise specified. For visual supervision, we use CLIP-ViT-L/14 [61] as visual encoder and follow LLa [49] protocols to add padding pixels and resize images to 336x 336.336. Besides, the head is 8 for all cross-attention layers. The stride of the convolution layer and average pooling layer in PEL are 14 and 2, while the interval factor in PAL is 4. The patch features from EVE and VE are normalized by fo-norm before MSE loss.
We adopt the Adam optimizer [35], warm-up strategy, and cosine scheduler for training EVE. The maximum learning rates for Stage 1, 2, 3 are 4 x 10 4 x 10 10-5, 5, 2 10 10-5, 5, while the number of batch size and training samples 512, 512, 128 and 16M, 33M, 665K for EVE-7B, which spends -9 days using two 8-A100 (40G) nodes. Notably, we only implement a high-resolution training strategy in supervised fine-tuning stage to obtain EVE-7B (HD) by involving an extra 1.2M SFT data.
4.2
Main Results
We evaluate EVE on a series of public vasual-language benchmarks, including Seademic-task-oriente benchmarks (VQA-v2 [25], GQA [29], VizWiz [26], and TextVQA [67]), hallucination benchmarks

---

Table 3: Comparison with state-of-the-art VLMs on vision-language benchmarks. #Samples:
the number of samples in the pretraining/supervieed fine-tuning stage. AR: image aspect ratio. HD:
high image resolution. We evaluate VLMs on VQA VQA-v2 [25]: GQA [29]; VizWiz [26]; SQA!:
ScienceQA-IME [54]; VQA": TextVQA [67]] POPE [47]; MME [23]; MMB: MMBench [52];
SEED: SEED-Bench [41]; Vet [89]. Includes in-house data that is not publicly accessible
Method
LLM
ASamples
AR
V GQA
VizWiz SOA!
VOA
POPE
MME
MMB
SEED
MM-V
Encoudr-buused Langon-Lngggge Models
InstructRI
Vicuna-7IB
129M/1
Fix
49.2
34.5
60.5
36.0
53.4
IDEFICS-98
LLaMA-7B
355M/1M
Fix
50.9
38.4
35,5
25.9
-88.2
QwenVL
Qwen-7H
1.48 /50M)
Fix
78.8
59.3
35.2
67.1
38.2
56.3
QweaVL-Chat
Owen-7B
1.48 150M
Fix
78.2
57.5
38.9
68.2
61.5
1487.5
60.6
58.2
LLaVA-1.5
Vicuna-7B
558K/665K
Fix
78.5
62.0
50.0
66.8
58.2
85.9
1510.7
64.3
58.6
30.5
InterVI--CMe -haat
Vicuna-7B
4.98886K
Fix
79.3
62.9
86.4
1525.1
mPLUG-OW12
LLaMA2-7B
400M/1 2M
Fix
79.4
56.1
54,5
68.7
58.2
86.2
1450.2
64.5
57.8
36.5
LVIS-4V
55SK/B85K
Fix
79.6
62.6
51,8
68.3
58.7
86.0
1528.2
66.2
60.6
31.5
ShareGPTaV
Vicuna-7B
L.2M/66K
Fix
80.6
63.3
57.2
68.4
Monkey (HD)
Qwen-7H
NA/1 44M
Enom
80.3
60.77
61.2
69.4
67.6
LLaVA-16 (HD)
Vicuna-7B
558K/790K
Enum
$1.8
64.2
57.6
70.1
64.9
86.5
1519.3
67.4
64.7
43.9
Pncoder-free Zagunng Models
EVE-78
Vicuna-7B
33M/665K
Any
75.4
60.8
63.0
83.6
1217.3
49.5
54.3
FByu-8B (HD)
Persimmon-8B
-1/-1
Any
74.2
74,1
728.6
10.7
21.4
EVE-78 (HD)
Vicuna-78
33M/1
Any
78.6
62.6
64.9
1305.7
56.8
(POPE [47]), open-world multi-modal understanding benchmarks (MME [23], MMBench [52], SEED-Bench [41], and MM-Vet [89]), scientific problem benchmarks (ScienceQA-IMe) [54]).
(1) In Table 3, EVE demonstrates superior performance compared to Fuyu-8B [3]. an encoder-free counterpart, across various Laiiual benchmarks, despite its smaller size. The incorporation of diverse SFT datasets and larger image sizes in EVE (HD) significantly enhances its image recognition capabilities. This highlights how our adaptable image processing approach benefits from higher resolution inputs and more comprehensive instructional data. (2) EVE (HD) shows the competitive performance when compared to encoder-based based VLMs, e-g., VI--Chtl [8], mPL.UG-Ow12 [85], LVIS-4V [77], ShareGPT4V [7], and Monkey [48], without requiring additional visual encoders for complex image encoding. Notably, EVE (HD) outperforms several VL.Ms (InstructBIIIP [12], IDEFICS-9B [30], QwenVL-CCI [2]), and performs par with the well-regarded VLa [49]. Notably, EVE faces challenges in accurately responding to specific instructions like option letters and binary questions. Besides, extensive training steps with only vision-language data have some- what diminished its language competency and instruction -ooouwttooooolooiinng capability. Consequently, the performance of EVE-7B MME, MMB, SQA, and MM- Vet benchmarks is relatively subpar.
The fact that EVE-7B can match encoder-based VLMs using a simpler architecture and publicly available million-scale data is quite encouraging. This discovery suggests that encoder-free VI.Ms, in virtue of appropriate training strategies and high-quality image-text data, can efficiently achieve performance on par with or even surpass that of encoder-based VLMs. This opens up promising avenues for addressing the challenges typically associated with encoder-based VLMs, such as inflexible input modes, inefficient deployment, and inconsistent capacity across modalities.
4.3 Ablation Studies
Configurationss of patch embedding and aligning layer. To validate the effectiveness of the proposed PEI and PAI., we conduct experiments with different configurations. oVE...M. EVE,M and EVEsa represent models trained by -LaVA-pretrain-5588 [49] (LLaVA-capssss) ), and subsets of 4M and 8M from the overall 33M EVE pre-training datasets (EVE-capa /EECCap//3M... EVE-caps/saa. in in Stage Besides, MSE means aligning features between the current EVE output and the next tokens from the vision encoder. From Table 4, (1) we observe that removing any cross-attention layer in PEL results marginal performance degradation; (2) Pairwise patch alignments perform slightly better than next-patch prediction, possibly because the vision-specific layer is shallow, causing the network to prefer the current patch encoding over predicting the next patch in auto-regressive manner; (3) Whether using small-scale or large-scale training data, visual supervision via PAI. effectively enhances fine fine-grained image representations, thereby facilitating visual perception and accelerating model convergence.
Stages 1-2. All models use LLa VA-mix 665K [49]
7

---

Table 4: Configurations of PEL and PAL. EVE EVELM, EVE&M utilize LLaVA- capsssk, EVE-cap4/33Mr EVE-caps/331 in
Table 5: Configurations of training procedure. The top half all use LLaVA- Vaiiiiix for fine-tuning. The bottom half all use EVE-caps3M for pre-training
Stage 1-2 with LLa LLaVA-mixesst P denotes the next-patch alignment
Stage
Model
VQA
GQA
MMB
SEED
EVE w/o Stage I
Model
VQA*2
GQA
MMB
SEED
Stage (4M)
64.6
54.1
40.6
45.4
Stage (8M)
50.2
42.5
26.8
36.2
EVEN.5
58.5
49.8
35,3
39.3
PEL CA1
58.2
49.4
34.4
40.1
EVE w/ Stage
59.3
51.0
36.5
39.7
PEL CA2
58.4
49.8
35.8
39.3
Stage 2 (4M)
69.4
56.5
42.0
48.7
+
MSENP
58.0
50.0
34.8
39.6
Stage 2 (8M)
71.2
58.9
44.0
50.3
PAI.
55.6
47.5
34.5
37.8
EVE w/ Stage 1-2
EVEI
69.4
56.5
42.0
48.7
Stage (665K)
75.4
60.8
49.5
54.3
PAL
66.4
55.3
41.2
47.5
+ HD t
77.5
62.6
47.7
55.2
EVERM
71.2
58.9
44.0
50.3
Stage (1.8M)
76.7
61.8
51.2
54.8
PAL
69.4
57.3
42.7
49.2
+ HD
78.6
62.6
52.3
56.8
8
From Scratch
w/o Stage
w// Stage (Standard)
Layer Interval=8
Accuracy (%)
555
40600
4077
Interval=2
Interval=4 (Standard)
Training
4
45
2922
353
2-
35
Goootaming!
Collappn
15k 25
Sk
10k
GQA Datasets
Training Step
Figure 5: Training loss in Stage using various strate-
Figure 6: Configurations of interval factor.
gies. Optimization remains unstable and prone to col- We observe that only the last vision output is lapse, despite searching learning rate from 2c-5 to le-3. insufficient and interval works the best.
Stage 1: LLM-guided pre-aligning stage does count for efficient training. In Table 5, display the influence of our training strategies, including pre-training and supervised fine-tuning stages. We discover that (1) without LLM- pre- allgiingn between vision and language in Stage 1, the entire pre-training process becomes unstable and suffers from sudden collapse. This results in an unusual situation where performance rapidly decreases data volume increases beyond certain point, as shown in Figure 5. We assume that the randomly initialized PEL introduces significant noise and negative interference into network optimization for severely turning LLM into VLM, exacerbated by accumulation problems. Hence, Stage stabilizes the training process and scamlessly transitions into Stage 2, emphasizing its indispensable role in efficient training construct an encoder-fie free VLM. This stage prevents collapses and expedites convergence, guaranteeing a smooth and robust training route. (2) Increasing image resolutions and incorporating diverse instruction data in Stage bring extreme performance gains after obtaining well-aligned state across modalities in Stage 1-2. The high-resolution property greatly enhances fine- tine-grained visual recognition, particularly for charts, tables, OCR information, and high-definition samples. Moreover, due to the extensive pre-training data, there a pressing need for more supervised fine-tuning data to improve inherent linguistic proficiency, commonsense knowledge, and custruction-folloviii patterns forgotten by EVE. (3) Training VLMs from scratch in Figure often leads to instability and poses significant challenges for model optimization. Therefore, LLMs serve an effective initialization for developing VLMs.
Merging patch features across layers outperforms the last layer output. Figure illustrates the impact of varying interval factors for selecting cross-layer features in PAL. Optimal performance emerges when the interval factor set to 4, yielding peak scores across VQA-v2, GQA, and SEED benchmarks. Our findings solidify the superiority of multi-layer aggregation over singularly relying on the last layer output. That is because competitive interplay unfolds within the dynamic mapping space between vision and text, resulting in projection coupling and mutual interference within the last layer of L.L.M. Furthermore, hierarchical integration for vision features empowers multi-level representations, facilitating adaptive alignments with abundant characteristics from the vision encoder.

---

GQA Accuracy (%)
SEED Accuracy (%)
70
65
60
55
45
EYE (10)
16
24
32
40
8
16 24
32
40
Training Step with Stage2 Data (M)
Training Step with Stage2 Data (M)
GQA and SEED using LLa as encoder-based bascline. We first train its projector in Stage with EVE-capio/as 6/33MM Here, (VE)-(LLM) indicates training Vision Encoder in Stage 2 and LLM in Stage 3, where we train the projector across all stages.
Figure 7: Scaling performance
GQA Accuracy (%)
SEED Accuracy (%)
70
65
55
DVE-7BB-1
1778-73-110 payo vt
35
0
16
24
32
40
8
16
24
32
40
Training Step with Stage2 Data (M)
Training Step with Stage2 Data (M)
Figure 8: Scaling performance on GQA and SEED w/ or w/o vision encoder supervision (PAL).
EVE consistently improves with more pre-training data, nearing encoder-based performance. In Figure we first train LLaVA-1.5's projector via EVE-cap16/// 6/33/ in Stage and then attempt various training strategies in Stage 2-3 following [85, 2, 8, 9, 53, 7]. For example, (VE)-(ALL) denotes training vision encoder in Stage 2 and all model parameters in Stage 3. We discover that:
(1) Using synthetic captions, encoder- based model often suffers from collapse [88] due to simplified language structure and lack of knowledge details. Only the (VE)-(ALL) strategy avoids this issue and outperforms the baseline by freezing LLM weights during pre- vee-training to preserve language proficiency and world knowledge. (2) EVE shows consistent performance gains with scaled-up data, gradually approaching the encoder-based model's ss performance. This may be because encoding and aligning vision and text unified network is more challenging and requires more training data, making encoder-free free models less prone to overfitting compared to encoder- based ones. (3) For the vision encoder (CLIP) within the PAL of EVE, we discover that unfreezing it at stages results in model collapse, which does not yield any improvements and instead increases computational costs.
Vision supervision aids in early convergence but becomes less critical during large data scale-up. Vision encoder supervision improves training efficiency, especially with relatively limited data scale compared the billion scale of CLIP image-text pairs. Actually, this component can be simplified with sufficient data resources as shown in Figure 8. Our findings indicate that its influence diminishes over large data scales, and by the 24M mark, the difference in performance with or without this supervision is about 0.3-0.8% This may be because large amounts of high-quality and detailed captions greatly enhance the understanding of visual information, thus gradually reducing the need for visual supervision. We emphasize that vision supervision (PAL) is not the crucial factor for training stability and scaling efficiency. Even with vision supervision, performance without ALM-guided pre-aligning in Stage 1 rapidly decreases as data volume increases beyond certain point.
Efficient deployment and lower latency. Table demonstrates several inference metrics, including floating point operations per second (FLOPs) and time delay. By eliminating the deep pre-trained vi- sion encoder, EVE significantly accelerates the im- age encoding process, achieving an order of magni- tude speed improvement over its counterparts. The CA1 component in PEL serves to control computa- tional complexity. Even with a minimal downsam- pling rate of 2, EVE (HD) surpasses LLaVA-1.6 1.6 in both efficient deployment and reduced inference
Table 6: Model FLOPSs and inference latency
Vision Part
LLM Part
Model
FLOPs(G) Time(s) FLOPs(T) Time(s)
LLaVA-1.5
372
0.033
15.2
0.48
EVE-7B
42
0.003
15.2
0.48
LLaVA-1.6 (HD)
1860
0.13
76.1
2.07
EVE-7B (HD)
170
0.013
60.8
1.52

---

delay. By optimizing FLOPs and minimizing inference latency, EVE ensures rapid and efficient processing. This leads a more responsive and resource-efficient system, positioning encoder- free VLMMs as superior to encoder-based VL.Ms in practical applications.
5 Limitation and Discussion
EVE demonstrates a desired and powerful encoder-free decoder-only architecture that essentially solves various issues of existing encoder-based VLMs. Despite its promising results, key limitation the performance gap that still exists between EVE and state-of-the encoder- based VLMs. Additionally, due to time constraints, several questions and considerations remain:
(1) Further performance gain. We empirically find that training solely with vision- usion-language data significantly reduces language capability (SQA score drops from 65.3% to 63.0%) while gradually improving multimodal performance. This indicates catastrophic forgetting of linguistic knowledge within the LLM. To address this, suggest merging language and multimodal data appropriately or employing mixture-of -e-eeessss (MoE) strategies to mitigate vision- Aasion-aanuuag interference.
(2) Encoder-free expectation. We validate that encoder-free VLMs can rival encoder-based VLMs but require larger training samples. How about the performance under the same LLM capacity and training data? We are working on the assumption that scaling up LLM capacity and training data would enable encoder-free VLMs to match or even surpass encoder-based VLMs, due the nearly lossless image inputs that bypass pre-procsssing by vision encoders.
(3) Multi-modality inspiration. As bonus, we gain inspiration for integrating additional modalities (e.g., audio, video, thermal, depth, etc.). The core idea is to pre-align these modalities with frozen LLM from LLM-ccntric perspective before introducing large-scale unified training, supervised by corresponding singe-modality encoders and linguistic conceptual alignments.
6 Conclusion
In this paper, we introduce EVE, a simple yet effective encoder-free model with decoder-only architecture, designed to revolutionize vision-language understanding. Rather than merely presenting another VIM,, we aim to unveil the under-explored bottlenecks towards constructing an encoder- free VLM. Our key findings reveal that: (1) An LLM-centric perspective, where vision and language modalities are pre-aligned, delivers robust, efficient, and consistent improvements as the pre-trained data scale up. (2) Incorporating fine-grained alignment with pre-trained vision encoders and linguistic conceptual supervision significantly enhances vision recognition, resulting in faster convergence and substantial performance gains. Remarkably, EVE performs par with mainstream encoder- based VLMs solely relying on 35M publicly available data, and dramatically outperforms the counterpart Fuyu-8B, trained by undisclosed strategies and private datasets. As pioneer, EVE provides transparent and efficient roadmap for future VLM development, essentially tackling the challenges of multi-modal input processing, deployment efficiency, and model capacity across modalities. In the future, we are devoted to scaling up the model capacity with high-quality training data to explore the limits of encoder-free free VLMs. Besides, we are working on translating the single-modality LLMs to large multimodal models with more modalities through our proposed training strategies.
10

---

Acknowledgments
This project is supported by the National Key R&D Program of China (2022D01166002), the National Natural Science Foundation of China under grant No. 62293540 and 62293542, Liao Ning Province Science and Technology Plan No. 20022222////10000016 and Dalian City Science and Technology Innovation Fund No. (023JJ1ICO011
References
[1] Al Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma. Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan. Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang. Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu, Qwen technical report. Xrviv: 23009, /6609, 2023.
[3] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşurla. Introducing our multimodal models, 2023. URL https: / /www. adept. ai/ blog/fuyu 8b.
[4] Shikha Bordia and Samuel R. Bowman. Identifying and reducing gender bias in word-level language models. In NAACL-HLT, pages 7-15, 2019.
[5] Zheng Cai. Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong. Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui Hc, Yingfan Hu, Ting Huang. Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning. Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song. Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang. Jiaqi Wang. Jiayu Wang, Rui Wang. Yudong Wang, Ziyi Wang. Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, and et al. Internim2 technical report. arXiv: 2403.17297 2024.
[6] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voos Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. USENIX Security Symposium, pages 2063-2250,,26 2021.
[7] Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao. and Dahua Lin. Sharegpt4v: Improving large mudti-modal models with better captions. arXiv: 2311. 2023.
[8] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv: 23 03/22.1233 /4238, 2023.
[9] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we 4p? closing the gap commercial multimodal models with open-source suites. arXiv:2404. 16821, 2024.
[10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: h/tma/s//////////////////smsss org/blog/ 2023-03- 30- vi vicuna/
[11] Christopher Clark and Matt Gardner. Simple and effective multi-paragraph reading comprehension. In ACL, pages 145-855,222 2018.
[12] Wenliang Dai. Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung. and Steven C. H. Hoi. Instructblip: Towards general-purpose -urpose vision-languse models with instruction tuning. In NeurlPS, 2023.
[13] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin. Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang. Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver. Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Ma- hendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko.
11

---

Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf. Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen. and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In JCML, volume 202, pages 1480-7512,22 2023.
[14] Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu, Similarity reasoning and filtration for image-text matching, In AAAI, pages
2021.
[15] Haiwen Diao, Ying Zhang, Wei Liu, Xiang Ruan, and Huchuan Lu. Plug-and-play regulators for image-text matching. TIP, 12:2322-2334, 2023.
[16] Haiwen Diao, Bo Wan, Ying Zhang, Xu Jia, Huchuan Lu, and Long Chen. Unipt: Universal parallel tuning for transfer learning with efficient parameter and memory. CVPR, 2024.
[17] Haiwen Diao, Ying Zhang, Shang Gao, Xiang Ruan, and Huchuan Lu. Deep boosting learning: A brand-new cooperative approach for image-text matching. 77P, 2024.
[18] Haiwen Diao, Ying Zhang. Shang Gao, Jiawen Zhu, Long Chen, and Huchuan Lu. Gssf: Generalized structural sparse function for deep cross- moosl metric learning. arXiv preprint arXiv:2410.1526 2024.
[19] Haiwen Diao, Bo Wan, Xu Jia, Yunzhi Zhuge, Ying Zhang, Huchuan Lu, and Long Chen. Sherl: Synthesizing high accuracy and efficient memory for resource-limited transfer learning. In ECCV, pages 75-95, 2025.
[20] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang. Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internim- mmemimm-xcompose22 xcomposer2-44Ma
A pioneering large vision-language model handling resolutions from 336 pixels to 4k HD. arXiv: 2404.06512, 2024.
[21] Alaaeldin El-Nouby. Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. arXiv preprint arXiv:2401.00 088541, 2024.
[22] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: exploring the limits masked visual representation learning at scale. CVPR, pages 19358- 19369, 2023.
[23] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv: 13394, 2023.
[24] Google. Google bard. httpp::/bbate httpa://bardggooglee.om 2023.
[25] Yash Goyal. Tejas Khot, Douglas Summers- Summers-Stay, Dhruv Batra, and Devi Parikh. Making the in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6325-6334, 2017.
[26] Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, pages 3608 3617, 2018
[27] Muyang He. Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv: 2102. 2024.
[28] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wemmeng Yu, Junhui Ji, Yan Wang. Zihan Wang. Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. 4rXiv:2312.08914, 2023.
[29] Drew A. Hudson and Christopher D. Manning. GQA: new dataset for world visual reasoning and compositional question answering. CVPR, pages 6700-6709, 2019.
[30] IDEFICS Research Team. Introducing idefics: An open reproduction of state-of ofte--of-hhe-rr visual language model. http:///ggnngggae...//////////ea
2023.
[31] Dongfu Jiang. Xuan He, Huaye Zeng, Con Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv:2405.01488 483, 2024.
12

---

[32] Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. DVQA: understanding data visualiza- tions via question answering. In CVPR, pages 5648-5656, 2018.
[33] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, pages 235 235-251, 2016.
[34] Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeong Ycon Nam, Jinyoung Park, Jinyeong Yim, Wonscok Hwang. Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In ECCV, 2022.
[35] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[36] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tote Xiao, Speneer Whitehcad, Alexander C. Berg, Wan-Yon Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. arXiv: 2304.02643, 2023.
[37] Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata. Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. I/CV, 123(1):32-73 2017.
[38] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. Uijlings, Ivan Krasin, Jordi Pont- Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. arXiv: 181 1.00982, 2018.
[39] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Liammin Zheng, Cody Hao Yu, Joseph Gonzalez. Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, pages 611-626 2023.
[40] Bo Li, Peiyuan Zhang, Jingkang Yang. Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: high- resolution multi-modality model. arXiv: 2311,04219, 2023.
[41] Bohao Li, Rui Wang. Guangzhi Wang. Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal ilms with generative comprehension. arXiv: 1312,1 2023.
[42] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICL.R, volume 162, pages 12888-1288 12880- 2022.
[43] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre- training with frozen image encoders and large language models. In ICML, volume 202. pages 19730-19742, 2023.
[44] Lin Li, Guikun Chen, Hanrong Shi, Jun Xiao, and Long Chen. A survey on multimodal benchmarks: In the era of large ai models, 2024. URL https: / wrw./ba//////eeeeeeoooo 18142.
[45] Xiaotong Li, Yixiao Ge, Kun Yi, Zixuan Hu, Ying Shan, and Ling- Yu Duan. me-beit: Multi-choice discretization for image bert pre-training. In ECCV, pages 231-246, 2022.
[46] Xiaotong Li, Fan Zhang, Haiwen Diao. Yueze Wang, Xinlong Wang, and Ling-Yu Yu Duan. Densefusion- Im: Merging vision experts for comprehensive multimodal perception. arXiv preprint vrXiv:2407.08303 2024.
[47] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large vision- lsiounlannuuag models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors. EMNLP, pages 292-305, 2023.
[48] Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun. Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv: 231 .06607077720 2023.
[49] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv: 2310,03344, 2023.
[50] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[51] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge. 2024. URL https: //llava-- vl, github. bbog/ 2024-01- 01-30- next/.
13

---

[52] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model all-around player? arXiv: 0307,06281, 2023.
[53] Haoyu La, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek- vl: Towards real-world vision-language understanding. arXiv: 2403.05525, 2024.
[54] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurlPS, 2022.
[55] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan I. Yuille, and Kevin Murphy, Generation and comprehension of unambiguous object descriptions. In CVPR, pages 11-20, 2016.
[56] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, pages 3195 3204, 2019.
[57] Ahmed Masry, Xuan Long Do, Jia Qing Tam, Shafiq Joty, and Enamul Hoque. Chartqa: A henchmark for question answering about charts with visual and logical reasoning. In ACL, pages 2263-2279, 2022.
[58] Brandon McKinzie, Zhe Gan. Jean- Phillpipp Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang. Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hc, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang. Peter Grasch, Alexander Toshev, and Yinfei Yang. MM1: methods, analysis & insights from multimodal LLM pre-training. arXiv: 2403.09611. 2024.
[59] Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty, Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pages 947-952. 2019.
[60] OpenAL. GPT-4 technical report. arXiv: 2303.08774, 2023.
[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretehen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139, pages 8748 8763, 2021.
[62] Yongming Rao, Wenliang Zhao, Guangyi Chen. Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language- dense prediction with context-aware prompting. In CVPR, pages 18061-18070, 2022.
[63] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5h: An open large-scale dataset for training next generation image-text models. NeurlPS, 35:25278-25294, 2022.
[64] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In ECCV, pages 162, 2022.
[65] Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? arXiv: 13043, 2024.
[66] Oleksii Sidorov, Ronghang Hu, Mareus Rohrbach, and Amanprect Singh. Texteaps: A dataset for image captioning with reading comprehension. In ECCV, volume 12347, pages 742 4527758,77 2020.
[67] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In CVPR, 2019.
[68] StepFun Research Team. Step-Iv: A hundred billion parameter multimodal large model. https:// platform wwwrrrssteepiii stepfun........m com, 2024.
[69] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In ACL, pages 3645-3650, 2019.
[70] Quan Sun. Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in context learners. arXiv: 2312.13286, 2023.
14

---

[71] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: improved training techniques for CLIP at scale. arXiv: 2303. 15389, 2023.
[72] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao. Jingjing Liu, Tiejun Huang, and Xinlong Wang, Generative pretraining in multimodality. arXiv: 2307.05222, 2023.
[73] Quan Sun, Jinsheng Wang. Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, and Xinlong Wang. EVA-CLIP-ISB: scaling CLIP to billion parameters. arXiv: 2402.04252, 2024.
[74] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptise Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv: 2312. 1805, 2023.
[75] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, arXiv: 2307.13971, 2023.
[76] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et Llama Open foundation and fine-tuned chat models. arXiv: 2307 337700988, 2023.
(77] Junke Wang. Lingehen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting GPT-4V for better visual instruction tuning. arXiv: 2311.07574, 2023.
[78] Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. On the de-duplication of laion-2b. arXiv preprint 2rXiv:2303, 12733, 2023.
[79] X.ai. Grok-1.5 vision preview.
2024.
[80] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm:
Empowering large language models to follow complex instructions. arXiv: 2304.12244, 2023.
[81] Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an LMM perceiving any aspect ratio and high-resolution images. arXiv: 2403. / 1703, 2024.
[82] Zhiyang Xu, Trevor Ashby, Chao Feng, Rulin Shao, Ying Shen, Di Jin, Qifan Wang, and Lifu Huang. flanssainnn visual instruction tuning. 2023. URL httttt////viiiooo flan. ..e..ggttt github. io/.
[83] Zhengyuan Yang. Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Chine Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv: 17421, 9, 2023.
[84] Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal free visually-situtted language understanding with multimodal large language model. arXiv: 2310.05 2023.
[85] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou, mplug-ow12: Revolutionizing multi-modal large language model with modality collahoration. arXiv: 23/1 2023.
[86] Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. In /CLR, 2023.
[87] Licheng Yu, Patrick Poirson. Shan Yang. Alexander C. Berg, and Tamara L. Berg, Modeling context in referring expressions. In ECCV, volume 9906, pages 69 85, 2016.
[88] Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capsfusion: Rethinking image-text data at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14022-14032. 2024.
[89] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinehao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv: 2308.02490. 2023.
[90] Bo Zhao, Boya Wu, and Tiejun Huang. SVIT: scaling up visual instruction tuning. arXiv: 2307.04087. 2023.
[91] Lianmin Zheng, Liangsheng Yin. Zhiqiang Xie, Jeff Huang, Chuyue Sun, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, lon Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Efficiently programming large language models using sglang, arXiv: 2312.07104, 2023.
15

---

A Appendix
A.1 Ethics Statement
We introduce EVE, pioneering encoder-free VLM. along with efficient training strategies for developing an encoder-free VLM. important to address the ethical considerations surrounding large pre-trained models, such as potential bias or discrimination [6] and privacy concerns [4] in the extensive training data. Additionally, the computational cost and environmental impact have become significant topics of discussion [69]. To our knowledge, previous VLMs have not thoroughly investigated their impact on issues like bias or information leakage, which warrants further exploration. Unlike closed-soures VLMs, we train EVE using publicly available data, thereby raising fewer ethical concerns. Furthermore, our approach reduces model deployment costs by lowering computational consumption and server resource requirements. Notably, our training experiments are conducted in data center powered entirely by renewable energy sources.
A.2 Dataset Details
In Table 7, we provide a detailed description of SFT datasets, along with their sampling rates during Stage The majority are derived from VLi 665K [49] and Bunny-695K [27]. Besides, we compile blended set from A12D [33], Synthdog [34], DVQA [32], ChartQA [57], DocVQA [11] and Vision-Flan [82] to leverage the advantages of high-resolution inputs.
Table 7: Supervised Fine-tuning Data Mixture
Table 8: Hyperparameter settings
Data
Sample Raito
Size
Task
Hyperparameter
Stage-1
Stage-2 Stage-3
LLaVA [49]
8.7%
158K
Batch Size
512
512
128
Conversation
SVIT-core-I50K [90]
8.7%
158K
Learning Rate (Ir)
4e-4
4e-5
2e-5
LR Schedule
cosine decay
ShareGPT [7]
2.2%
40K
Text-only
LR Warmup Ratio
0.03
0.01
0.01
W-Z [80]
3.9%
70K
Weight Decay
0
VQAv2 [25]
9.1%
83K
Epoch
GQA [29]
7.9%
72K
General QA
Optimizer
AdamW
OKVQA [56]
1.0%
DeepSpeed stage
3
3
3
A-OKVQA [64]
7.3%
66K
Knowledge QA
TexICaps [66]
2.4%
22K
Synthdog [34]
1.7%
30K
OCR QA
OCRVQA [59]
8.8%
80K
RefCOCO [57
5.3%
48K
Grounding
VG [37]
9.5%
86K
Vision-Flan [82]
10.2%
186K
Multi-task
ChartQA [57]
1.0%
18K
Chart QA
DVQA [32]
11.0%
200K
A12D [33]
0.7%
12K
Science QA
DoeQA [11]
0.6%
10K
Document
Total
100.0%
1.35M
A.3
Training Details
We report the detailed training hyper-parameter settings of EVE during the three stages in Table 8.
A.4
Qualitative Results
The qualitative results are demonstrated for comparisons. showcase the ability EVE in the aspects of OCR, reasoning, and image captioning, respectively.
16

---

Visual input example
$55
User
Pleuse describe image contents in detail.
LLaVA-1.5
The image features man with an unusual appearance, standing in front of a mirror, He is wearing purple shirt and appcars to robot or a character from comic. The man taking a selfic with his cell phone, bolding it up to face while capturing moment. Two other people the main focus of the seene. Additionally, bottle is visible at the far left of the image, potentially decorative item part of the room's 's design.
scene, potted plant is positioned nearby, adding a touch of greenery to the setting. be in the background, one situated to the left and another towards the right, but they are not
EVE
The image features man taking picture of himself bathroom mirror. He standing in front of a sink, which is located on the left side of the room. The man is holding a cell phone in his hand, capturing the moment. The bathroom is decorated with few potted plants, one of which is placed near the sink, and another one is located closer to the right side of room. There also a vase in the scene, positioned near the center of the room.
EVE (HD)
The image depicts a hathroom scene. A man is taking selfie in the mirror, which mounted on the wall. He wearing a purple shirt and leopard print pants. is holding camera, capturing reflection. The mirror is large and spans significant portion of the wall. The bathroom features white sink with silver faucet, which is located the left side of the image. The sink is positioned and has white tile hacksplash. To the right of the sink, there is a white tiled wall. A white tiled floor is visihle in the lower part of the image. The bathroom also contains a plant with white flowers, which is placed in a vase. The plant is located near the sink and the mirror. The flowers are white and are arranged in a bouquet. There is also a poster on the right
countertop. Above the sink, there mirror. The countertop made of granite
wall. which pesitioned of remote control and some text. The image also includes white tile wall, which is located behind the man, The wall is clean and well-maintained. Overall, the image captures moment in bathroom, with man taking selfie in mirror. a plant with white flowers, a sink, mirror, poster, and uled wall.
mirror. poster black and white and is quite large. It features a picture
Visual input example
Neanderthal
Years and 8 Wor the Dav
Friday January 2015
User
Please describe image contents in detail.
LLaVA-1.5
The image depicts au unconventional scene of man wearing a costume jaw, mose, and chin made unique character appears be humanoid figure made from with gold background that highlights the costume.. The man's face, with its interesting rock-formed facial features, seems to the focal point of the image.
This
EVE
The image features appcurs to be an older individual with beard. He is standing in front of wall, which could be separate structure. The scone is set outdoors, with the man being the main focus of the image.
holding wooden stick, possibly staff walking stick, in front of building. The
of the bailding
EVE (HI)
The image features man with long beard and a bald head. He is holding wooden stick in his hand. The man's 's face partially covered by large yellow text that reads "Neanderthal'. The man's 's face is also partially covered by large yellow text that reads "Years and 8", The man's face is also partially covered by large yellow text that reads "Worst of the Day". The man's face is also partially covered by large yellow text that reads "Friday January 2015", The man's face also partially covered by a large yellow that reads Years and 8", The face also partially covered by large yellow text that reads "Neunderthal..
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims are made in abstract and introduction (Section 1).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
This paper discusses the limitations in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This paper does not include theory assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The training setting listed in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
The codes will be publicly available and the training data is described in Table and Table 2.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https:
ccpps.cc/pulli///u guwdee/iee///ee CCodeSubeesiaceecceee simiisssinn ssssonoliccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and testing details are listed in Section 3.2 and Section 4.1, respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: For fair comparsion, we follow the standard and commonly used setting for experiment evaluations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
20

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources are listed in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ttp///eurrpp.ee/ee//ee cc/ cc/public/Ethi 1 iiieeessddliee??
Answer: [Yes]
Justification:
This research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
The discussion conducted Appendix A.I.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification:
The discussion is conducted in Appendix A.I.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original owners of the code, data, models are properly credited this paper (Section 4.1).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "JzpLgNWrpy", "reviewer": "Reviewer_gQSV", "summary": "This paper explores the topic of encoder-free vision language model. It proposes to directly input the image patches into the decoder network together with the language tokens, without the use of a separate visual encoder during inference time. The benefits are mainly two-fold: simpler architecture and more flexible image resolution and aspect ratio. In experiments, they utilized 33M/1.8M data for pretraining/SFT and demonstrated superior performance to Fuyu-8B which is not open sourced.", "questions": "1. In Fig.6, what does “from scratch” mean exactly? Is the language model also randomly initialized?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The topic of encoder-free VLMs is very interesting and has valuable potential benefits. \n2. As an empirical paper, the experimental results look solid. The performance almost matches the best of encoder-based VLMs . It is a plus that the proposed model outperforms the counterpart Fuyu-8B.\n3. Experiments to verify the necessity of stage-1 training is properly done and provides some insight of training VLMs with added components.", "weaknesses": "1. The training is using an existing visual encoder as teacher, which kind of defeats the title of \"encoder-free\". Is this necessary? The contribution would be more valuable if it can be completely encoder-free (during both training and inference).\n2. Related to 1., if there is a need to train the model on some ouf-of-domain datasets, does it need to first train the separate visual encoder, and then redo the 3 training stages described in this work? This makes the training even more complex than encoder-based VLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We respond to all questions you raise to address your concerns and make the necessary revisions to improve the quality of the paper.\n\n`Q1: The training is using an existing visual encoder as teacher, which kind of defeats the title of \"encoder-free\". Is this necessary? The contribution would be more valuable if it can be completely encoder-free (during both training and inference)`\n\nAs discussed in ALL-Q1, we found that while the vision encoder aids in early convergence, it becomes less crucial as the data scale increases. Inspired by this, we empirically found that pre-training EVE with 4M high-resolution images (1344 longest edge) without vision supervision can match the performance of our original version, which used 12M data points and vision supervision. This indicates that eliminating the vision encoder for a fully encoder-free VLM is practical during both training and inference.\n\n`Q2: Related to 1., if there is a need to train the model on some out-of-domain datasets, does it need to first train the separate visual encoder, and then redo the 3 training stages described in this work? This makes the training even more complex than encoder-based VLMs.`\n\nGood question. \nTransferring the visual encoder initially may be beneficial due to its strong inductive bias and susceptibility to out-of-domain issues. Fortunately in our experiments, we discovered that after 24M pre-training data, vision encoder supervision becomes less essential for EVE to develop visual perception from scratch. This enables EVE to circumvent these issues by abandoning the vision encoder supervision and preserving the original three stages during the training process.\n\n`Q3: In Fig.6, what does “from scratch” mean exactly? Is the language model also randomly initialized?`\n\nYes. \"From scratch\" denotes that the entire model is initialized randomly. We empirically discovered that training EVE from scratch faces extreme optimization challenges and usually suffers from model collapse.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FGn4Dua1Rr", "reviewer": "Reviewer_VqDP", "summary": "Authors bridge the gap between encoder-based and encoder-free models, and present a simple yet effective training recipe towards pure VLMs. They unveil the key aspects of training encoder-free VLMs efficiently via thorough experiments: (1) Bridging vision-language representation inside one unified decoder; (2) Enhancing visual recognition capability via extra supervision", "questions": "see weakness", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Authors propose to find the key aspects of training encoder-free VLMs\n2. EVE outperforms the counterpart Fuyu-8B (encoder-free)", "weaknesses": "I am concerned with the motivation and experiments.\n\nFrom Tab. 3, although it outperforms Fuyu-8B, it is inferior to many encoder-based Large Vision-Language Models. I would except EVE had additional benefits, however, it is unclear why we need it. \n\nThe AR is 'Any' for encoder-free VLMs, but I do not see the benefits of it. At least I cannot have it from the tables. Could authors provide any explanations? From Tab. 6, EVE-7B (HD), which had acceptable performance, does not gain enough benefits in terms of FLOPs(G) and Time(s) if we consider the vision part and LLM part together. At current stage, I could not see enough motivation of EVE-7B (HD).\n\nOne minor question (not urgent, authors could simply ignore it if time does not allow): the benchmarks presented are normally simple VQA benchmarks, how would the model perform on detailed description tasks?\n\nOverall, I am concerned with the motivation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful remarks. We have responded to all your queries and made the necessary changes to enhance the paper.\n\n`Q1:  (((1))) EVE outperforms Fuyu-8B, but lags behind many encoder-based VLMs. why we need it?  (((2))) Any explanations for the benefits of 'Any image ratio'?  (((3))) No enough benefits of FLOPs(G) and Time(s) when considering the vision part and LLM part together.`\n\n**(1) We highlight that EVE shows potentially more promising scaling with pretraining duration, which is the key motivation behind our efforts to build encoder-free VLMs.** \nEVE attempts to remove strong vision inductive bias and transmit visual signals almost losslessly for better scaling properties.\nIn Figure 2 of the rebuttal PDF, we observe that encoder-based models often suffer from collapse. Only the (VE)-(ALL) training strategy avoids this issue by freezing LLM weights during pre-training and unfreezing them during the SFT stage. In contrast, EVE shows better scaling properties and gradually approaches the performance of well-developed encoder-based VLMs with only 33M data scale.\n\n**(2) Encoder-free VLMs are promising for scaling but require enormous training data to develop vision perception from scratch.**\nHere, with only 33M pre-training data, our pioneering exploration currently lags behind but performs comparably to popular encoder-based methods. **Interestingly, subsequent PaliGemma [a]** also explores an encoder-free version via 1B pre-training image-text pairs, showing promising early results alongside its encoder-based counterpart across 37 validation datasets (see Figure 3 of the rebuttal PDF). They particularly mention that only the separate vision encoder i.e. SigLIP, has been trained with 40B image-text pairs, far greater than 1B data of encoder-free version. They also indicate that decoder-only VLMs may be a promising direction although currently suffering in training efficiency due to building vision perception from scratch.\n\n**(3) The 'Any' image ratio, simple architecture, and efficient deployment are bonuses of encoder-free VLMs.** \nRecent studies on encoder-based VLMs reveal that \n**(i)** Due to the limitations of pre-trained encoders, existing VLMs exhibit vulnerabilities in basic capabilities rooted in visual encoding trade-off [b, c]. \n**(ii)** Various vision encoders show uneven levels of capability due to pretext pretraining tasks, relying heavily on the corrective capabilities of LLMs for multimodal understanding [d, e].\nIn contrast, encoder-free VLMs remove semantic priors in abstracting visual representation, theoretically allowing VLMs to autonomously acquire all available information. **While 'any image ratio' and 'FLOPS gains' are natural benefits of the encoder-free approach, the primary reason for exploring an encoder-free model is its scaling efficiency with less inductive bias.** \nIn this premise, removing the vision encoder provides only a modest bonus in terms of flexible image input and deployment efficiency. Notably, the encoder-free track is still in early development and has a long way to explore its limits.\n\n[a] PaliGemma: A versatile 3B VLM for transfer. Google DeepMind. arXiv 2407.07726.\n\n[b] LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images. Xu et al. arXiv 2403.11703.\n\n[c] HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models. arXiv 2407.08706.\n\n[d] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. Tong et al. CVPR2024.\n\n[e] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. Tong et al. arXiv 2406.16860.\n\n`Q2: One minor question (not urgent, authors could simply ignore it if time does not allow): the benchmarks presented are normally simple VQA benchmarks, how would the model perform on detailed description tasks?`\n\nFrom the LMMs-Eval leaderboard [a], we observe that LLaVA-1.6 **surprisingly** achieves lower CIDEr scores than LLaVA-1.5 on COCO-Cap and Flickr-cap under the same model capacity. These description tasks do not well reflect the capability of multimodal models due to out-of-distribution issues and evaluation metrics, which are seldom evaluated by most existing VLMs.\n\n[a] LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models. Zhang et al. arXiv 2407.12772", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CVtIpvZYas", "reviewer": "Reviewer_nP6r", "summary": "This paper revisits the vision-encoder-free MLLM direction, which is not a popular choice in the community at present. A new method EVE is proposed to reduce the gap between encoder-free and encoder-based MLLMs, demonstrating a large improvement against Fuyu-8B (the best encoder-free MLLM so far). EVE introduce additional modules to distill visual features from an existing vision encoder, then align image-text in the consequential stage.", "questions": "It’s unclear whether 35m data is enough (likely not?). Have you tried data scaling ablations?\n\nIn line 234, it should be Table 3?", "limitations": "It’s worth mentioning EVE inherits the same limitations from the resued LLM, i.e., Vicuna-7B (likely also including CLIP-ViT-L-336px).", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The recipe (EVE) shared by this paper is simple yet effective, which could be insightful to the VLM community to develop better encoder-free MLLMs, which have many benefits, such as flexible resolution and aspect ratio, efficient deployment and lower latency etc.\n\nThe experiments are comprehensive, with a large number of baselines.\n\nThe paper is well written and easy to read.", "weaknesses": "IIUC, Fuyu learns visual features from scratch, while EVE actually distillates from  an existing CLIP-ViT-L-336px vision encoder, so it’s not fair to claim that EVE is better than Fuyu under the same “encoder free” setting. It’s more like “encoder distilled” vs. “encoder free”. Therefore, it would be more insightful if we could ablate encoder free vs. distilled in 4.3 (if the training stability issue can be solved), i.e., to investigate why EVE is much better than Fuyu.\n\nAs mentioned in the paper, the language ability is largely affected in stage 2. It’s very common to add an additional LM task and there have been many open-sourced corpuses already. It’s worth explaining why this common approach was not tried (though the paper suggests to solve this in future work).\n\nEven though EVE-7B is comparable with the selected encoder-based baseline models of the same size, it is still largely behind some recent models with even half size and lower resolution, such as PaliGemma-3B [1].\n\n[1] https://ai.google.dev/gemma/docs/paligemma/model-card", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your constructive comments are much appreciated. We have addressed all your points and revised the paper accordingly to ensure its improvement.\n\n`Q1: IIUC, Fuyu learns visual features from scratch, while EVE actually distillates from an existing CLIP-ViT-L-336px vision encoder, so it’s not fair to claim that EVE is better than Fuyu under the same “encoder free” setting. It’s more like “encoder distilled” vs. “encoder free”. Therefore, it would be more insightful if we could ablate encoder free vs. distilled in 4.3 (if the training stability issue can be solved), i.e., to investigate why EVE is much better than Fuyu.`\n\n**(1) As discussed in ALL-Q1, while the vision encoder aids in early convergence, it becomes less crucial as the data scale increases.** \nWe introduce vision encoder distillation to improve training efficiency, especially with a limited data scale, which is less significant after 24M pre-training data in our experiments. This indicates that eliminating the vision encoder for a fully encoder-free VLM is practical during both training and inference.\n\n**(2) The LLM-guided pre-aligning stage is one crucial factor for training stability and scaling efficiency.**\nWithout this stage, even with enormous training data and vision distillation, training decoder-only VLMs remains highly challenging in Table 5 and Figure 6, suffering from model collapse and large performance gaps.\n\n**(3) Compared with Fuyu-8B, EVE adopts image resolution with 672 image longest edge during pre-training due to device constraints and efficiency consideration.**\nFor fairness, we attempt to pre-train EVE with 4M high-resolution data (1344 longest edge) without vision supervision, which can match the performance of our original version, using 12M data and vision supervision. This indicates that EVE has room for further improvement and could potentially widen the gap beyond Fuyu-8B.\n\n`Q2: As mentioned in the paper, the language ability is largely affected in stage 2. It’s very common to add an additional LM task and there have been many open-sourced corpuses already. It’s worth explaining why this common approach was not tried (though the paper suggests solving this in future work).`\n\nWhile maintaining language ability in stage 2 is a concern, it is not the primary challenge in developing encoder-free VLMs effectively. The main issue is addressing optimization problems and bridging the gap in multi-modality capabilities compared to encoder-based VLMs. We spent a long time exploring and finally discovered that an LLM-guided pre-aligning stage and vision encoder supervision are essential for training stability and early convergence, respectively. Considering text-only data is less critical at this stage, we decided to solve it in the follow-up study.\n\n`Q3: Even though EVE-7B is comparable with the selected encoder-based baseline models of the same size, it is still largely behind some recent models with even half size and lower resolution, such as PaliGemma-3B [1].`\n\n**(1) Actually, such a comparison is unfair due to many notable differences.**\n- [Data Magnitude]: As mentioned before, EVE learns vision perception from scratch, where the vision encoder could be removed using the entire 33M pretraining data. However, PaliGemma’s SigLIP encoder has seen 40B image-text pairs during Stage 0, with Stage 1 and Stage 2 further processing about 350B and 90B tokens, respectively.\n- [LLM Capability]: Despite being smaller, Gemma-2B exhibits similar capabilities to our Vicuna-7B.\n- [Finetuning Stategy]: Table 1 in PaliGemma shows that they train a 'specialist' model for each task with specific hyperparameters, while EVE serves as a 'generalist' model for downstream tasks.\n\n**(2) We highlight that EVE shows potentially more promising scaling with pretraining duration.**\n\nIn Figure 2 of the rebuttal PDF, we observe that EVE shows better scaling properties and gradually approaches the performance of well-developed encoder-based VLMs with only 33M data scale. **Interestingly, PaliGemma** also explores an encoder-free version via 1B pre-training image-text pairs, showing promising early results alongside its encoder-based counterpart across 37 validation datasets (see Figure 3 of the rebuttal PDF). They particularly mention that their vision encoder i.e. SigLIP, was trained with 40B image-text pairs, and indicate that decoder-only VLMs may be a promising direction although currently suffering in training efficiency due to building vision perception from scratch.\n\n`Q4: It’s unclear whether 35m data is enough (likely not?). Have you tried data scaling ablations?`\n\n35M data points provide valuable insights for constructing encoder-free VLMs but fall short of peak performance. Figure 1-2 in the rebuttal PDF show that performance is not yet saturated, and EVE offers better training stability and scaling efficiency. Due to budget and time constraints, we will scale the pre-training corpus to a billion scale in future work.\n\n`Q5: In line 234, it should be Table 3?`\n\nThanks for the kind reminder. We will revise it.\n\n`Q6: It’s worth mentioning EVE inherits the same limitations from the resued LLM, i.e., Vicuna-7B (likely also including CLIP-ViT-L-336px).`\n\nThis is a common problem for both encoder-based and encoder-free VLMs. We believe that as existing LLMs continue to advance impressively, these limitations, we think, would be properly addressed. Besides, encoder-free VLMs escape the limitations of pre-trained vision encoders, unlike their encoder-based counterparts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7i2jJRA9Ik", "reviewer": "Reviewer_Z41h", "summary": "This paper introduces EVE, a novel paradigm for Vision-Language Models (VLMs) designed to eliminate the need for a preceding visual encoder in the LLM decoder. This approach aims to create a more flexible and lightweight vision-language framework. EVE incorporates a Patch Embedding Layer and a Patch Aligning Layer to align image tokens with the language model. Additionally, it proposes a new training recipe for robust encoder-free model learning. The result is an efficient VLM that achieves performance (somewhat) comparable to existing encoder-based VLMs across several benchmarks.", "questions": "I love the concept and the exploration of encoder-free VLMs. However, as mentioned in the weaknesses, several points significantly undermine the significance of this work and the key claims of it. Here are several important questions.\n1. Given that the Patch Embedding Layer (PEL) functions similarly to an image encoder, how do you justify classifying EVE as an \"encoder-free\" model? \n2. The manuscript claims that the PEL converts images into embeddings \"almost losslessly.\" How do you elaborate this claim with the use of average pooling, which is inherently a lossy operation?\n3. Despite using more training data, EVE's performance is still lower than some encoder-based VLMs like LVIS-4V, ShareGPT4V, and LLaVA-1.6. What factors do you believe contribute to this performance gap, and what strategies are you considering to address it?\n4. While Table 6 shows significant reductions in FLOPs and inference time for visual encoding, the LLM block remains the computational bottleneck. How significant do you consider the efficiency gains from the encoder-free approach in the context of the overall model performance?\n5. The paper emphasizes the advantages of flexible Image Resolution (IR) and Aspect Ratio (AR) support. Could you provide specific examples or use cases where this flexibility offers significant practical benefits over traditional encoder-based models?", "limitations": "The discussion on limitations and societal impact about the paper is relatively thorough. The points raised in the weaknesses section further articulate limitations that should be considered.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "[**Writing**] The paper exhibits exceptional clarity and coherence in presenting its concepts. The accompanying figures serve as effective visual aids, significantly enhancing the comprehensibility of the discussed ideas.\n\n[**Motivation**] The rationale behind removing off-the-shelf encoders in current VLMs is compelling and innovative. While most existing works adhere to the well-established encoder-based VLM paradigm, this exploration into a new direction is commendable. Such investigations are invaluable to the research community.\n\n[**Methodology Transparency**] The authors provide comprehensive details on dataset curation and implementation specifics. \n\n[**Ablation Studies**] The inclusion of detailed ablation experiments is a significant strength of this paper. These studies collectively offer a clear demonstration of the proposed model's capabilities and distinctive features, providing crucial insights into the model's performance and design choices.", "weaknesses": "[**Encoder-Free?**] The model's positioning as \"encoder-free\" is somewhat misleading, given the presence of a Patch Embedding Layer (PEL) that functions very similarly to an image encoder. Despite its light-weight and compact nature, the PEL's architecture closely resembles a ViT encoder. Furthermore, the proposed Patch Aligning Layer (PAL) encourages CLIP-like token generation, further blurring the line between this approach and traditional encoder-based models.\n\n[**Almost Lossless?**] The manuscript repeatedly claims that the PEL converts images into embeddings \"almost losslessly.\" However, the use of average pooling for feature map downsampling is inherently a lossy operation, undermining this claim.\n\n[**Performance Gap**] The model's performance, while promising, still lags behind encoder-based VLMs. It's notable that models like LVIS-4V, ShareGPT4V, and LLaVA-1.6 achieve superior results despite utilizing substantially less training data than EVE.\n\n[**Complexity Analysis**] Table 6 demonstrates significant reductions in FLOPs and inference time for the visual encoding procedure. However, this improvement should be contextualized within the overall model architecture, where the LLM block dominates computational demands. This raises questions about the relative impact of optimizing the visual encoder in terms of overall efficiency gains. \n\n[**Flexibility Benefits**] The manuscript emphasizes the advantages of supporting flexible Image Resolution (IR) and Aspect Ratio (AR) as key benefits over encoder-based VLMs. However, the practical significance and impact of this flexibility are not comprehensively demonstrated or evaluated or quantified in the current study.\n\n[**Structural Issues**]\n* Figure 1 is never referenced in the main text.\n* Figure 5 is mentioned after Figure 6 in the text, disrupting the logical flow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your meticulous and insightful review. We have carefully considered your questions and polished our paper.\n\n`Q1: [Encoder-Free?] How do you justify classifying EVE as an \"encoder-free\" model?`\n\n**(1) A more essential difference between PEL and image encoders is whether they involve strong inductive bias and semantic priors in abstracting visual representation by pre-trained models via pretext tasks.** We introduce CA1 and CA2 mainly due to efficiency consideration and holistic information for raster-order token inputs respectively. Note that Table 4 shows that CA1 and CA2 are optional and have a less significant impact on performance.\n\n**(2) As discussed in ALL-Q1, we found that while PAL aids in early convergence and training efficiency, it becomes less crucial as the data scale significantly increases.**\nBesides, we can completely remove it during inference, allowing EVE to function as a pure encoder-free architecture like Fuyu-8B. Hence, we attribute our work to the ‘encoder-free’ track. \n\n`Q2: [Almost Lossless?] The use of average pooling, which is inherently a lossy operation?`\n\nGood question. CA1 may not deviate from the \"almost\" lossless concept due to its dense feature aggregation strategy in a restricted receptive field, only using pooling features as the query. We adopt this design, mainly considering the trade-off between large token numbers and training efficiency, especially with limited resources.\n\n`Q3: [Performance Gap?] [Complexity Analysis?] [Flexibility Benefits?]`\n\n**(1) We highlight that EVE shows potentially more promising scaling with pretraining duration, which is the key motivation behind our efforts to build encoder-free VLMs.**\nEVE attempts to remove strong vision inductive bias and transmit visual signals almost losslessly for better scaling properties.\nIn Figure 2 of the rebuttal PDF, we observe that encoder-based models often suffer from collapse. Only the (VE)-(ALL) training strategy avoids this issue by freezing LLM weights during pre-training and unfreezing them during the SFT stage. In contrast, EVE shows better scaling properties and gradually approaches the performance of well-developed encoder-based VLMs with only 33M data scale.\n\n**(2) Encoder-free VLMs are promising for scaling but require enormous training data to develop vision perception from scratch.** \nHere, with only 33M pre-training data, our pioneering exploration currently lags behind but performs comparably to popular encoder-based methods. **Interestingly, subsequent PaliGemma [a]** also explores an encoder-free version via 1B pre-training image-text pairs, showing promising early results alongside its encoder-based counterpart across 37 validation datasets (see Figure 3 of the rebuttal PDF). They particularly mention that only the separate vision encoder i.e. SigLIP, has been trained with 40B image-text pairs, far greater than 1B data of encoder-free version. They also indicate that decoder-only VLMs may be a promising direction although currently suffering in training efficiency due to building vision perception from scratch.\n\n**(3) The 'Any' image ratio, simple architecture, and efficient deployment are bonuses of encoder-free VLMs.** \nRecent studies on encoder-based VLMs reveal that \n**(i)** Due to the limitations of pre-trained encoders, existing VLMs exhibit vulnerabilities in basic capabilities rooted in visual encoding trade-off [b, c]. \n**(ii)** Various vision encoders show uneven levels of capability due to pretext pretraining tasks, relying heavily on the corrective capabilities of LLMs for multimodal understanding [d, e].\nIn contrast, encoder-free VLMs remove semantic priors in abstracting visual representation, theoretically allowing VLMs to autonomously acquire all available information. **While 'any image ratio' and 'FLOPS gains' are natural benefits of the encoder-free approach, the primary reason for exploring an encoder-free model is its scaling efficiency with less inductive bias.** \nIn this premise, removing the vision encoder provides only a modest bonus in terms of flexible image input and deployment efficiency. Notably, the encoder-free track is still in early development and has a long way to explore its limits.\n\n[a] PaliGemma: A versatile 3B VLM for transfer. Google DeepMind. arXiv 2407.07726.\n\n[b] LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images. Xu et al. arXiv 2403.11703.\n\n[c] HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models. arXiv 2407.08706.\n\n[d] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. Tong et al. CVPR2024.\n\n[e] Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. Tong et al. arXiv 2406.16860.\n\n`Q4: The performance gap with some encoder-based VLMs? How to address it?`\n\n**(1) The key issue with EVE is the massive data needed to build vision perception from scratch.** \nEncoder-based models first train a separate visual encoder with billions of data points, while EVE constructs vision perception from scratch using only 33 million data points. Figures 2-3 show performance is far from saturated. Larger data scales and more computing resources are required for further exploration.\n\n**(2) Besides, several important factors deserve to be explored**. We found image resolution, reducing modality perturbations, the base LLM’s size and capability, data size and quality, etc. also impact the scaling efficiency of encoder-free VLMs. For instance, pre-training EVE with 4M high-resolution data (1344 longest edge) without vision supervision, can match the performance of the original version, using 12M data and vision supervision. We would polish these factors and develop a fully and stronger encoder-free VLM in the follow-up study.\n\n`Q5: Figure 1 is never referenced in the main text. Figure 5 is mentioned after Figure 6 in the text, disrupting the logical flow.`\n\nThanks for your kind reminder and we will polish them in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
To all reviewers:

We thank all reviewers for your constructive comments. We are encouraged by approvals on an **interesting and novel idea** (gQSV, nP6r, Z41h), **simple yet effective** (VqDP, nP6r, Z41h), **comprehensive experiment and solid results** (gQSV, nP6r, Z41h), and **insights for the VLM community** (gQSV, nP6r, Z41h). We conducted more experiments and analyses to address some reviewers’ concerns. The supplements in the attached PDF are summarized as follows:

- Scaling performance of EVE with or without vision encoder supervision in Figure 1 of the attached PDF.

- Scaling performance of EVE vs. the encoder-based baseline in Figure 2 of the attached PDF;

- Scaling performance of PaliGemma with or without an image encoder from 100M to 1B data in Figure 3 of the attached PDF;

`All Reply Q1: Is 'encoder-free' accurate? How does vision encoder distillation/supervision work in EVE? Is it necessary?`

**(1) Vision encoder supervision does help with early convergence by 1-3% gains shown in Table 4, but is not very crucial during large data scale-up.** 
**(i)**  We introduce vision encoder supervision to improve training efficiency, especially with a limited data scale.
**(ii)**  Actually, vision supervision from a pre-trained encoder is less significant when using sufficient data resources. 
Figure 1 of the rebuttal PDF indicates that the influence of vision encoder supervision diminishes over large data scales, and by the 24M mark, the difference in performance with or without this supervision is negligible by less than 0.3-0.8\%. 
This may be because large amounts of high-quality and detailed captions greatly enhance the understanding of visual information, thus gradually reducing the need for visual encoders.

**(2) More importantly, vision supervision is not the crucial factor for training stability and scaling efficiency.**
Table 5 and Figure 6 show that even with vision supervision, performance without LLM-guided Pre-aligning in Stage 1 rapidly decreases as data volume increases beyond a certain point. This indicates that vision supervision is not essential for the scaling efficiency of EVE.

**(3) We came up with the ‘encoder-free’ concept because EVE can work like Fuyu-series without visual encoders during the inference and deployment.**
We can completely remove it during inference, allowing EVE to function as a pure encoder-free architecture like Fuyu-8B. 
Besides, though involving vision supervision helps with training efficiency, it becomes less necessary as the pre-training data scale significantly increases. In other words, developing a fully encoder-free VLM during training or inference is practical with larger data and computing resources. 

We hope that the point-by-point responses below have effectively addressed your previous concerns, and would appreciate any further feedback you can provide.

Sincerely yours,

Authors.

Meta-review: The paper proposes to train LVLM without using pretrained vision encoders as an essential building block. It shows promising performance and the key aspects are simple and effective. This can be insightful to the research community to build better encoder-free MLLMs. The encoder free model can  offer flexible resolution and aspect ratio, improved efficiency. Concerns raised by reviewers were addressed during rebuttal. We encourage the authors to include the clarifications and discussion of main rebuttal points, and clarify the purpose of encoder free and lossless terms.
Decision: Accept (spotlight)