From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
Authors: Haiwen Diao
Mingxuan Li
Silei Wu
Linjun Dai
Xiaohua Wang
Hanming Deng
Lewei Lu
Dahua Lin
Ziwei Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Native Vision-Language Models
Vision-Language Primitive
Holistic Vision-Language Buffer
Primary area: foundation or frontier models, including LLMs
TL;DR: A novel family of native VLMs built from first principles, capable of rivaling top-tier modular counterparts across diverse real-world scenarios.

Abstract: The edifice of native Vision-Language Models (VLMs) has emerged as a rising contender to typical modular VLMs, shaped by evolving model architectures and training paradigms. Yet, two lingering clouds cast shadows over its widespread exploration and promotion: (-) What fundamental constraints set native VLMs apart from modular ones, and to what extent can these barriers be overcome?
(-) How to make research in native VLMs more accessible and democratized, thereby accelerating progress in the field. In this paper, we clarify these challenges and outline guiding principles for constructing native VLMs. Specifically, one native VLM primitive should: 
(i) effectively align pixel and word representations within a shared semantic space; (ii) seamlessly integrate the strengths of formerly separate vision and language modules; (iii) inherently embody various cross-modal properties that support unified vision-language encoding, aligning, and reasoning. Hence, we launch NEO, a novel family of native VLMs built from first principles, greatly narrowing the gap with top-tier modular counterparts across diverse real-world scenarios. With 390M image-text examples, NEO efficiently develops visual perception from scratch while mitigating vision-language conflicts inside a dense and monolithic model crafted from our elaborate primitives. We position NEO as a cornerstone for scalable and powerful native VLM development, paired with a rich set of reusable components that foster a cost-effective and extensible ecosystem.

Published as a conference paper at ICLR 2026
FROM PIXELS TO WORDS TOWARDS NATIVE VISION- LANGUAGE PRIMITIVES AT SCALE
Haiwen Diao! Mingxuan Li
Silei Wu²
Linjun Dai
Xiaohua Wang² Hanming Deng² Lewei Lu
Dahua Lin Ziwei Liu't
S-Laa,, Nanyang Technological University Time Research 'i''an Jiaotong University
Website: https:/ http//gittte http:/tttt.......eeeeeeeeeeee www./oowwwing..ee........ Labb/NO
ABSTRACT
The edifice of native Vision- Vision-Language Models (VLMs) has emerged as a rising contender to typical modular VLMs, shaped by evolving model architectures and training paradigms. Yet, two lingering clouds cast shadows over its widespread exploration and promotion: (-) What fundamental constraints set native VLMs apart from modular ones, and to what extent can these barriers be overcome?
(-) How to make research in native VLMs more accessible and democratized, thereby accelerating progress in the field. In this paper, we clarify these challenges and outline guiding principles for constructing native VL.Ms. Specifically, one native VI.M primitive should: (i) effectively align pixel and word representations within shared semantic space; (ii) seamlessly integrate the strengths of formerly separate vision and language modules; (iii) inherently embody various tross-moda properties that support unified vision-language encoding, aligning, and reasoning. Hence, we launch NEO, a novel family of native VLMs built from first principles, greatly narrowing the gap with top-tier modular counterparts across diverse real- world scenarios. With 390M image-text examples, NEO efficiently develops visual perception from scratch while mitigating vision-language conflicts inside a dense and monolithic model crafted from our elaborate primitives. We position NEO cornerstone for scalable and powerful native VLM development, paired with a rich set of reusable components that foster cost-effective and extensible ecosystem.
INTRODUCTION
Recently, Vision-Language Models (VLMs) (Bai al., 2025; Zhu al., 2025; Wang et al., 2025b; xAI, 2025; Anthropic, 2025; DeepMind, 2025; Hurst et al., 2024; OpenAl, 2025) have emerged as a major breakthrough, extending the strong linguistic capabilities of Large Language Models (LLMs) to multimodal understanding. They typically follow a modular design that integrates pre-trained Visual Encoder (VE) (Radford al., 2021: Chen al., 2024f; Fang al., 2023; Tschannen et al., 2025), Projector (Alayrac et al., 2022; Liu et al., 2024a; Dai et al., 2024), and an LLM (Touvron al., 2023; Yang al., 2025: DeepSeek-AI scale, they incrementally overcome limitations in image resolution, aspect ratio, and visual encoding flexibility. Yet, modular designs still contend with strong inductive biases in pre-trained visual semantics, complex infrastructure. and scaling laws needed to harmonize their components.
al., 2025). Through multi-stage post-training at
Against this backdrop, native VLMs have arisen as a new avenue of exploration, with Fuyu (Bavishi et al., 2023) and EVE (Diao et al., 2024) pioneering a promising route towards monolithic VLMs. Subsequent efforts seek to learn vision perception from scratch and mitigate vision-language confficts via visual encoder distillation (Diao al.. 2024; et al., 2025b; Wang et al., 2025a; Li et al., 2025a), mixed training data (Lei et al., 2025; Li et al., 2025a), and modality-speciiff decomposition (Diao al., 2025; Luo et al., 2024; 2025; Li al., 2025a). Nonetheless, constructing visual representations via mapping functions inside pre-trained L.LMs often hinders efficiency (Chen et al., 2024d; Luo al., 2024), destabilizes optimization (Team, 2024; Wang et al., 2024b), and disrupts original linguistic knowledge (Diao et al., 2024; Chen et al., 2024d), even under decoupled designs or large budgets (Beyer al., 2024). Besides, HoVLE (Tao et al., 2025) and HaploVL (Yan et al., 2025) Work was done during Haiwen's 's remote collaboration with SenseTime Research. 'Corresponding author.

---

Published as a conference paper at ICLR 2026
Discrete V5. Continuous
Nouneer 1UM eutt 2018
Modular VLMs
10425
Takma
Kathee VLM Primitives
Alskkas Prddon Bocaddingg
(Laveee of Denvvntaall
Viaton
Language Model
Hakes Authoalic SILE
20h Sh-aatkss 2i44
R.FF af Nat
Fribed
Reffie 1800
BArEO 189403
FPM
Layw Naan 6, 11,24 27
Propertios
91
Tile Native Resolution
1.Seamleedyyy Lood LLMe
Vieuaa Leamingg frale
Ffficiiee Flok Miadd Alignnsnnt
IT
L.MM
L.M
Fel--ite A Raase
VIM
Fext
Sodii inkiiaaaiio
Zacolz
2.28 Paooess
Dense Mode
Mulll Studaal Nivea Allentiies
Mhulta Descrokonal Rueld for Mulli Stediiity lctetions
Slevosid
Late vs.. Early Pussion
Mixed ansetio
Natire RoPE
Mlserti
LLM
VIM
IMA (AA BAGA
Channal Alkcorton T.B..
1000 Hud Iuol Hole Houl PAil
BB
Resshhi Pra-iluttir
Tackle 12:32
Moddahe
Manstihi VUN
8600enteen Coanal Antertion
Teloppiii
Figure 1: Overview of our native vision-language frameworks, which project abbirray--esoluttin images into a continuous latent space, integrating the virtues of modular VLM architectures and enabling efficient vision-language encoding, alignment, and interaction in an early-fusion manner.
address this by first mapping vision- vision-language inputs into a shared space. Yet, their modality-sharinn modules, whether derived from LLM or VE layers, neglect intrinsic discrepancy in encoding and interaction across modalities, compromising VLM's capacity to unify visual-linguistic properties.
Figure 1 outlines central question: What properties must native VLMs possess compete with modular ones? Modular VLMs decouple vision encoders from language models, allowing each to exploit modality-specific characteristics, e-g-, bi-directional versus causal attention, distinct positional embeddings, and varied network configurations. This separation accelerates the development of visual and linguistic competencies and permits flexible combinations of individual components. However, it fragments the training procedure, increases alignment costs, and leaves the intermodal balance unresolved. Motivated by these analyses, we formulate the following strategies accordingly:
(1) Native VLM Primitive. Native VLMs should embody a unified vision-language language primitive that simultaneously integrates encoding. aligmment, and reasoning across modalities in one single module. Its design should encompass three principles: (i) a Flexible Position Encoding scheme that generalizes effectively to dynamic spatial structures; (ii) a Multi-Head Native Attention (MHNA) that jointly processes issull-txttul connectivity; (iii) Native Rotary Position Embedding (Native- RoPE) that preserves compatibility with pretrained LLM while absorbing VE's interaction patterns. Guided by these tenets, we evolve the LLM blocks into native VLM primitives with brand-new RoPE designs and modality- aware interaction patterns, thereby capturing mimeniddimesioonll relationships for fine-grained and comprehensive correspondence from intrinsically multimodal perspective.
(2) Pre- BrefBuffer and Pust-LLM. The next crucial issue is to efficiently scale visual training while securing consistent pixel-word alignment. Here, we partition the monolithic backbone into pre-Buffer and post-L.LM layers during pre-training, each rooted in identical native primitive architectures. This transient stage enables pretrained LLMs to steer visual learning and establish coherent relevance with later stages. As mid-training and supervised fine-tuning advance, the partition dissolves, yielding a unified architecture that autonomously allocates the VLM's capacities their respective functions. This end-to-end training reduces semantic biases of separate pretraining and large overheads of post-stage alignment, effectively bridging native and modular VLMs. Crucially, pre-Buffer Buffer persists as reusable pretrained asset, facilitating sustainable resources for native VLM development.
We launch NEO, an innovative native VLM that reimagines multi-modal integration from first principles. Unlike typical modular designs, NEO rests on unified primitives that natively encode, align, and reason across modalities, forming coherent pixel-word correspondences from the outset. Through streamlined end-to-end training on 390M image-text samples, NEO acquires strong visual perception and approaches leading modular VLMs of comparable scale across diverse benchmarks. Beyond these results, NEO offers reusable components that simplify subsequent development and reduce barriers to promoting native exploration. This reveals that next-generation multimodal systems could also originate from architectures that native, unified, and intrinsically multimodal.
2

---

Published as a conference paper at ICLR 2026
tbc
red
pill
Pleet Aligning sta Primuttive pre dugior
Image Reasaningg
Prinntvvss post
MHNNA
Native Vision Language Model
Patch Embedding Layer
Word Embedding Layer
the
red
pm
Figure 2: Overview of our proposed NEO architecture. We begin with lightweight patch and word embedding layers that encode images and text into token sequences, which are subsequently processed by monolithic decoder-only architecture. The pre-Buffer and post-LLM components, each stacked with multiple native primitives, facilitate efficient and precise pixel-wort alignment and reasoning.
2 RELATED WORKS
2.1 MODULAR VISION-LANGUAGE MODELS
Current Vision-Langume Models (VLMs) have converged modular paradigm, where pretrained Vision Encoder (VE) is paired with a Large Language Model (LLM) via lightweight adapters, e.g., projection layers (Li et al., 2024a;b) gross-attention mechanisms (Alayrac al., 2022; Dai et al., 2024). This encoder-basse systems, including VL (Zhu al., 2025; Wang et al., 2025b), Qwen-VL (Wang et al., 2024a; Bai et al., 2025), Seed-VL (Guo et al., 2025), GLM-V (Hong et al., 2025), and Grok (xAI, 2024; 2025). By harnessing the complementary strengths of vision and language components, modular architectures, adhering to the "ViT-MLP- LLM" pipeline, achieve unprecedented performance across diverse multimodal benchmarks and have emerged as the dominant design principle in the field.
architecture underlies various popular and leading vision- lasoon-aanguage
Despite empirical successes, modular VLMs remain constrained by multi-stage training and heteroge- neous structures. Extensive test-training interventions are often required to mitigate rigid inductive biases in pretrained VEs (Wang et al., 2024a), which limit resolution flexibility, erode fine-grained de- tails, and blunt sensitivity to features across scales. Besides, pretraining semantic biases and capacity trade-offs between VEs and LLMs collectively impede design simplicity, deployment efficiency, and seamless integration of vision and language, underscoring the urgent need for a monolithic backbone.
2.2 NATIVE VISION-LANGUAGG MODELS
Native VLMs embrace furliffssion integration rather than grafting VEs onto LLMs. Early Fuyu (Bav- ishi et al., 2023), EVE (Diao et al., 2024), and SOLO (Chen et al., 2024d), embed image patches via linear projections, whereas Chameleon (Team, 2024), MoMA (Lin al., 2024), and MoT (Liang et al., 2024) transform images into symbolic sequences via discrete tokenizers. Later studies (Luo al., 2024; Diao al., 2025; Li al., 2025b; Luo et al., 2025; Li al., 2025a) leverage Mixture-of- Experts (MoE) or Divide-and- -nda--uu-- (DaC) strategies to suppress vision-language interference, while others (Diao al., 2024; Li et al., 2025b; Wang et al., 2025a: Li et al., 2025a) upgrade visual encoder supervision to accelerate the acquisition of visual concepts. Empirical evidence (Beyer al., 2024; Luo et al., 2024; Lei al., 2025) reveals that, with sufficient data and progressive training, native VLMs rapidly approach modular counterparts, corroborating recent scaling-law insights (Shukor et al., 2025b;a). Besides, recent methods (Tao et al., 2025; Yan et al., 2025; Xiao al., 2025) indicate that multi-modality encoding modules with the LLM VE style slightly resolve linguaelanguage misalignment, yet fail to fully integrate the distinct properties of each modality.
Notably, NEO redefines native VLMs as a unibody system built from first-principle primitives. Every network component, from native rotary position embeddings to multi- modality interaction patterns, ensures full compatibility with the intrinsic modeling patterns of VEs and LLMs. Meanwhile, NEO differs from existing modular VLMs via modality-agnostic pre-Buffer and end-to-end training, dramatically enhancing mixel-word alignment and pushing the frontier of native VLM research.
3

---

Published as a conference paper at ICLR 2026
Primitive
(3) Native Rutary Pocition I E'mbeediig
latu G.K for H. W
(lk New FCNam
<ES+: Enc Systel
Multi Head Nation Attention
(6 Q Hoad
Hark Rase Kreperadts
Chanmilade
Ch. Native Frame-si Malio Malic Medalty Altenlies
(861
RMSNeem
Pre Batfer & Pos-LLLM Formalion
Pre Buffer
-Madde Japar
Post LLM
-adda tapat:
Roud-Furward
Network
Part: Toat Tokees
Ward Supanee
DPV
Pragert:
Preperts
Tall
Bego
Figure 3: Overview of our native primitive, which integrates native attention with bi-directional dependencies within images and word/ frame-wise causal interactions, together with native rotary po- sition embeddings parameterized by modality-specific frequency, channel, and index allocation. It is inherently unified and intrinsically multimodal, substantially enhancing pixel-word correspondence.
3 METHODOLOGGY
3.1 MODEL ARCHITECTRRE
Figure 2 illustrates the proposed NEO framework, which comprises lightweight patch and word embedding layers, pre- grfferffer, and post-LLM, built upon stacked native VLM primitives.
Patch and Word Embeddings. Given an image we convert into token sequences via a lightweight Patch Embedding Layer (PEL) with two Convolutional layers (Conv1-2) (Krizhevsky et al., 2012) and Gaussian Error Linear Unit (GELU) (Hendrycks & Gimpel, 2016). For input text T, we encode it into word tokens using the original LLM Tokenizer Word Embedding Layer (WEL):
Tp = 2 (GELL(())))( + PE), Tt = Tokenizer(T),,
(I)
where To E (bxw)xd / xt E n denote visual / textual tokens, and PE is 2D Sinusoidal Positional Encoding (Dosovitskiy et al., 2021). The stride of Conv1 / Conv2 is 16 / 2, ie., each visual token corresponds pixel unshuffle (Chen et al., 2024e), with the special <img> and </img> tokens inserted at the boundaries of visual tokens. while mapping position and patch embeddings into a unified space. Afterward, visual and textual tokens are merged and propagated through the unified backbone.
32 32 image patch. Notably, Conv2 performs token folding like
Native VLM Primitive. It adopts RMSNorm (Zhang Sennrich, 2019) and SwiGLU (Dauphin et al., 2017) consistent with LLM layers. Unlike prior methods that collapse visual tokens into ID representations (Zhu et al., 2025; Wang et al., 2025b) or merely reallocate pre-trained LLM head dimensions across temporal (T), height (H), and width (W) (Wang al., 2024a; Bai et al., 2025), enlarge Query (Q) and Key (K) head dimensions and decouple H, W, and relations in Figure 3(1), adding 10% more parameters over the raw Transformer block. The T dimension retained, and new H and W dimensions are added with their respective QK normalization (Yang et al., 2025).
This philosophy aligns with our Native Rotary Position Embedding (Native-RoPE) in Figure 3(2). (a) Index Allocation: For text, T index is retained while H / W indexes are zeroed. For images, each visual token has a constant T index, with unique H/ / W indexes encoding spatial location. Videos, treated as sequences of frames, increment T index per frame. while H / W indexes follow the same spatial scheme as images. In multimodal inputs, each modality's 's index starts from the maximum ID of the preceding modality, ensuring continuous and unambiguous positional encoding across modalities. This serves two purposes: (-) For image pairs, H W indexes start independently from (0,0), and tokens at corresponding positions share identical dependency, strongly reinforcing correlations and interactions across matching regions (Liao et al., 2025; Wu et al., 2025); (-) For image-text text pairs, H/ / W indexes are decoupled from T index and bounded within (0,0) and (H.W), preventing large index growth from disproportionately affecting H/ / W indexes (Wang et al., 2024a; Bai et al., 2025) and thereby keeping spatial dependencies between long-range text and images.
4

---

Published as a conference paper at ICLR 2026
Another key aspect is (b) Channel and (c) Frequency Allocation. Unlike recent 3D-Re 3D-RoPE meth- ods (Bai et al., 2025; Wei et al., 2025; Yuan et al., 2025; Liao et al., 2025), we fully decompose the channel and frequency allocation of H. W. and T, equipped with additional Q/K head dimensions for H and W. This resolves two issues: (-) Zeroing H / W indexes for pure text would disrupt the modeling patterns and linguistic capacity of the L.MM if restricted to its original channels. Repairing this disruption requires substantial resources; (-) Even with interleaved or segmented reallocation, H and W are theoretically equívalent but are assigned different frequencies. Meanwhile, the RoPE frequency in LLMs is far lower than that of visual encoders in Figure 3(2c). This mismatch limits the modeling of relative distances and local semantics. The problem exacerbated by the disparity in scales, with temporal ranges spanning up to one million and spatial ranges only a few hundred.
Specifically, Native-RoPE assigns distinct base frequencies to T, H. and W within their own dimen- sions, i.e., original LLM head dimension for T and new head dimension for H / W as follows:
Qr = ( BT K [0. d 2
3H # i 10, }. θw =
{ S {
4 ijE 10, })} (2)
0
where 33 and θ indicate the base and rotation frequency across H, W. and T. Notably, temporal dimension captures both local and long-range relations, whereas spatial H / W dimensions emphasize local dependencies. This also opens avenues for broader applications, e.g., video understanding (Wei et al., 2025), multimodal generation (Deng al., 2025b), and editing (Deng et al., 2025a).
Inspired by prior works (Lei et al., 2025; Deng et al., 2025b; Li et al., 2025a; Beyer et al., 2024), we also treat one single image as a unified meta-unit for autoregressive modeling, denoted as Native Multi-Modal A Attention with mixed masking in Figure 3(3). Text tokens adhere to standard causal attention, attending only to preceding tokens to maintain autoregressive generation. In contrast, image tokens employ full bidirectional attention, enabling exhaustive interactions among all visual tokens, akin to a visual encoder. This design captures rich spatial and contextual dependencies within images and facilitates vision- vision-language correspondences, thereby supporting complex multimodal reasoning. We use FlexAttention (Dong et al., 2024) to minimize memory overhead and increase throughput, as variable-Iength block-wits wise attention is fully optimized through CUDA kernel modifications.
Pre-Buffer and Post-LLM. In Figure 3(4), we develop a modality-shared pre-Buffer to translate mixel-word inputs into unified representation with minimal disturbance to the post-LLM, which inherits the linguistic proficiency and reasoning capabilities of pre-trained LLM. The layer depths L1 and L2 primarily refer to parameter counts and scaling properties (Tian al., 2025) of existing VEs and LLMs to balance accuracy and efficiency. Here, we formulate one primitive T' as follows:
2tx = (HSM x = ==="" FFN(RMSNOm(((""))
(3)
where m E {v.t) indicates input modality. Besides, (Φ"...,, 111-1) and (TL1,... -b
denotes Brf-Bu and post-LLLM, respectively. Notably, we randomly initialize the entire pre- Buffer, while the post-LLM inherits RMSNorm, Ferward Network (FFN), and Q/K/QK-Norm parameters along the temporal dimension from a pretrained LLM. The temporal Q is reused to initialize Q for the H and W dimensions, their K weights are sero-initialized and the corresponding QK-Norm is initialized with = 0 and y = 1. We further match the attention scaling to that of the pretrained LLM, thereby preserving its pre-training paradigm from the outset and enabling a progressive emergence of multimodal spatial reasoning within the post-LLLM. Crucially, this separation exists only during pre-training. After that, these components are merged into monolithic backbone that autonomously allocates capacity for encoding, alignment, and reasoning.
3.2 TRAINING PROCEDURE
Figure illustrates the whole training pipeline, where the entire model is optimized end-to-end.
Pre- Training Stage. In this phase, NEO acquires fundamental visual concepts and contextual dependencies from scratch, guided by pre-trained patterns from LLMs. Training leverages 345M web-scale and synthetic image-caption pairs, including 100M English and 20M Chinese pairs from LAION- 4400M (Schuhmann et al., 2021), 150M English pairs from COYO-700M (Byeon et al., 2022), 20M long-caption examples from BLIP3o (Chen et al., 2025), and 5M short-caption pairs from OpenImages (Kuznetsova et al., 2018), recaptioned with a pre-trained InternVL2-88 VL2-8B model. The dataset further enriched with 30M samples from LAION-COCO (Schuhmann et al., 2022)
5

---

Published as a conference paper at ICLR 2026
Stage 1: Pre-Training
Stage 2: Mid-Training
Stage 3: Supervised Fine-Tuning
NEO
NEO
NEO
Pre- Buffer, QK Post-I.L.M
PRL.
WEL
PEL
WRL
PEL
WEL
Insage-Text
256 1024" Any Reselution
40M Capcion
256 2048" Any Resolution
High-Quality
256 2048 Any Resollution
Caption Data Pairs
OCR / Detection
Instruction Data
Figure 4: Overview of the entire training recipe. During pre-training, NEO learns visual perception from massive web-scale and synthetic image-caption pairs with frozen LLM weights to preserve linguistic knowledge. In mid-training and supervised finettuning, the full model is progressively optimized end-to-end using caption, conversation, OCR, detection, and high-quality instruction data.
and 20M examples from Wukong (Gu et al., 2022) with rich Optical Character Recognition (OCR) annotations. A 3:7 ratio of language to multi-modal data is incorporated to reconstruct text projections in the pre- guffBrfer. Only the patch embedding layer, the pre- mrefBuffer, and extra QK linear weights and normalization in LLM, along with H and W, are optimized with a simple next-token prediction objective. Notably, the new QK heads not only counteract the LLM's strong language bias that limits visual specialization but also safeguard its capabilities against the effects of low-quality data.
The objective at this stage is to strengthen the alignment between visual and linguistic capabilities while progressively enhancing recognition of high- ressuucso images, complex scenes, object scales, spatial grounding, and compact OCR content. The training data is drawn from the pre-training corpus of InternVL-1.3 (Chen et al., 2024f), comprising 40M samples across image captioning, conversation, detection, and OCR data, which account for approximately 66%, 1 1,, 8%, and 15% of the total, respectively. A 3:7 ratio of language to multi-modal data is again applied. The entire architecture is updated with the same loss functions to consolidate vision-language alignment, thereby equipping NEO with the foundational abilities required for various visual scenarios.
Mid-Traime Training Stage.
Supervised Fine- TamiTTn Stage. During the SFT stage, NEO's ability to follow complex linguistic instructions and varied dialogue patterns is further enhanced, a critical step towards real-world deployment. The full network is optimized across diverse high-quality, multi-source instruction datasets. Following Mono-InternVL. (Luo al., 2024), we employ about 4M bilingual instructions for supervised learning, covering tasks such as visual question answering, multimodal dialogue, mathematics, and knowledge reasoning. Details of the instruction data are provided in the Appendix.
EXPERIMENTS
4.1 TRAINING SETTINGS
Our NEO models are built on Qwen3-1.7B and Qwen3-8B (Yang et al., 2025) as the LLMs. The
pre-Buffer employs L1 = 12 primitive layers for NEO-2.2B and L1 = for NEO-9B. We extend only the QK head dimension in transformer layers, introducing roughly 10% extra parameters over the original design. The base RoPE frequencies 3r. 3H, and ßw are set to x 10 x 10 and 1 x 10 respectively. NEO is trained sixteen 8-GPU (80G) nodes using the AdamW optimizer (Loshchilov & Hutter, 2019). The maximum learning rates for pre-training, mid-training, and SFT are 8 x 10-4, 4 x 10-5 and 5 x 1 with warm-up ratio of 0.01 and cosine decay scheduler across all stages.
4.2 MAIN RESULTS
We conduct standard evaluations with VLMEvalKit (Duan et al., 2024) on diverse benchmarks, covering chart, diagram, and document understanding tasks, e.g., AI2D (Kembhavi et al., 2016),
Doc (Clark& & Gardner, 2018), ChariQA (Masry et al., 2022), InfoVQA (Mathew et al., 2022), TextVQA (Singh et al., 2019), and OCRBench (Liu et al., 2023e); visual perception and challeng-
ing reasoning tasks, e.g., MMMU (Yue et al., 2024), MMBench-EN (MMB) (Liu et al., 2024b), MMVet (Yu et al., 2024), MMStar (Chen et al., 2024c), SEEDBench- (SEED-I) (Li et al., 2023a); hallucination tasks, e.g., POPE (Li et al., 2023b) and HallusionBBench (HalIB) (Guan et al., 2024).

---

Published as a conference paper at ICLR 2026
Table Comparison with modular and native VLMs on general vision-language benchmarks. "# Data" denotes the dataset scale during pre-training, mid-training, and supervised fine-tuning. indicates models that employ reinforcement learning (RL). Bold highlights the highest performance.
Model
LLM
# Data
MMMU MMB MMVet MMStar SEED-1 POPE HallB
Modular Vision-Language Models (2B)
Qwen2-VL VL
Qwen2-1 1.5B
-/-/-
41.1
74.9
49.5
48.0
41.7
Intern VL2.5
InterL.M2.551.888
>6B/
100M
43.6
74.7
42.6
InterVV331
Qwcn2.5-1.50 >6B 100M /2M
48.6
81.1
62.2
60.7
89.6
42.5
Qwen2.5-VL
Qwen2.5-3B
-1-1-
51.2
79.1
61.8
55.9
46.3
Incoder-Based
Qwen3-1.. 1.7B
>B
40M 4M
47.1
75.8
37.4
52.7
73.6
44.4
Native Vision-Language Models (28)
Mono-IntemvVL
IntemLM2-11.m
143M 7M
33.7
65.5
67.4
34.8
Limo-IntemVL-13.
IntemL.M2111
400M 150M / 7M
39.1
64.0
54.0
66.9
32.5
HoVLE
IntemL.M2-1.8
550M / 50M 7M
32.2
73.3
43.8
70.9
87.4
38.4
OneCAT
Qwen2.5-1.50
436M 70M 13M
72.4
70.9
NEO
Owen3-1.78
345M / 40M 4M
48.6
76.0
49.6
54.2
74.2
87.5
43.1
Modular Vision-Language Models (8B)
Qwen2-Vi VI
Qwen2-7B
-1---
54.1
83
62.0
60.7
88.1
50.6
Intern VL22.5
IntemL.M2.5-7B
6B/
50M
56.0
84.6
64.4
50.1
Qwen2.5-VL
Qwen2.5-772
-1---
55.0
83.5
67.1
63.9
86.4
52,9
InternVI.3!
Qwen2.5-7B
>B/
100M 22M
62.7
83.4
81.3
68.2
91.1
49.9
Encoder- Based
Qwen3-8B
>B 40M 4M
54.1
84
63.5
76.2
87.8
51.4
Native Vision-Language Models (8B)
Fuyu
Persimmon-8B
-1-/-
27.9
10.7
59.3
Chamelcon
from scratch
1.4B OM 18M
25.4
31.1
8.3
30.6
19.4
17.1
EVE
Vicuna-7B
33M OM 1.8M
32.6
52.3
25.7
64.6
85.0
26.4
SOLO
Mistral-7B
44M /OM////
67.7
30.4
64.4
Emu3
from scratch
-/-/-
31.6
58.5
37.2
68.2
85.2
EVEv2
Qwen2,5-7B
77M 15M 7M
39.3
66.3
45.0
71.4
87.6
BREEN
Qwen2.5-7B
13M OM/
42.7
71.4
37.0
VoRA
Qwen2.5-7B
30M / OM 0.6M
32.0
61.3
33.7
68.9
85.5
SAIL
Mistral-7B
512M / 86M / 6M
70.1
46.3
53.1
72.9
85.8
54.2
NEO
Qwen3-8B
345M 40M 4M
82.1
62.4
76.3
46.4
Following InterVVL3 (Zhu et al., 2025), we construct the Encoder-Based by combining Qwen3 (Yang et al., 2025) and IntermiTT-300M (Zhu et al., 2025). In the mid-training stage, we first train the projector 10M samples, and further unfreeze the vision encoder utilizing another 30M samples.
Comparison with Modular VLMs. Table and Table 2, NEO achieves highly competitive performance against narrows the performance gap with top-tics modular VLMs, e.g.. Qwen2-VE (Wang et al., 2024a), Intern VL.2.5 (Chen al., 2024e), Qwen2.5-VL. (Bai et al., 2025), and Intern (Zhu al., 2025) across multiple benchmarks, despite using relatively limited training data and without reinforcement learning. These results highlight the effectiveness of end-to-end training strategy and unified model design with Native-RoPE and multi-modality interaction patterns. Moreover, the performance gap between Encoder- Eneoder-Based variants and state-of-the-art methods on MMMU, MMVet, TextVQA, and etc, indicates that NEO still suffers from the limitation in training data scale and quality.
counterparts the 2B and 8B scales. Impressively, NEO largely
Comparison with Native VLMs. From Table and Table 2, NEO delivers substantial gains on visual-centric benchmarks over the best competitors, e,g., Mono-Intee Inteo- (Luo et al., 2024; 2025), HoVLE (Tao al., 2025), OneCAT
al., 2025a), EVE (Diao et al., 2024; 2025), Emu3 (Wang et al., 2024b,, BREEN (Li et al., 2025b), VoRA (Wang et al., 2025a), and SAIL (Lei et al., 2025). By seamlessly integrating post-LLM components with the pre-Buffer for large-scale visual learning, NEO aligns visual inputs with textual features from scratch and supports complex visual reasoning, even without visual encoder supervision (Diao et al., 2024: Tao et al., 2025; Li et al., 2025a; Wang al., 2025a; Li et al., 2025b), highlighting the strengths of its native primitive designs and training strategies. These design choices allow NEO to surpass many native VLMs using fewer training resources, demonstrating the advantages of our primitives with efficient data-scaling capability.
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison with modular and native VLMs visual question answering benchmarks. Any Res., Tile-wise, Any Rat., and Fix Res. refer to any resolution, image tile splitting, any aspect ratio, and fixed resolution. MoE and DaC are Mixture-of- lixture-of-Expertt Experts and Divide-and-Cone ann-Conu models.
Model
Input
RoPE
Backbone AT2D DocVQA ChartQA InfoVQA TextVQA OCRBench
Modudar Vision-Languae Models (2B)
Qwen2-Vr VL
Any Res.
M-RoPE
Dense
74.7
90.1
73.5
65.5
79.7
80.9
InterVL2.55
Tile-wise
ID-RoPE
Dense
74.9
88.7
79.2
60.9
74.3
80.4
InterVLLS
Tile-wise
ID-RoPE
Dense
78.7
88.3
80.2
66.1
77.0
83.5
Qwen2.5-VLI
Any Res.
M-RoPE
Dense
81.6
93.9
84.0
77.1
79.3
79.7
Encoder-Baad Based
Tile-wise
ID-RoPE
Dense
77.4
89.9
78.4
65.9
73.3
83.5
Native Vision-Language Models (2B)
Mono-InternVL.
Tile-wise,
ID-RoPE
MoE
68.6
80.0
73.7
43.0
72.6
76.7
Mono-Intern Ve-o--ntteVV--.555 VL-1.5
Tile-wise.
ID-RoPE
DaC
67.4
81.7
72.2
47.9
73.7
80.1
HoVLE
Tile-wise.
ID- RoPE
Dense
73.0
78.6
55.7
70.9
74.0
OneCAT
Any Res.
M-RoPE
Dense
72.4
87.1
76.2
56.3
67.0
NEO
Any Res.
Native-Ron RoPE
Dense
80.1
89.9
81.2
63.2
74.0
77.1
Moduaar Vision- anguage Models (8B)
Qwen2-W VL
Any Res.
M-RoPE
Dense
83.0
94.5
84.3
V12.
Tile-wise
ID-RoPE
Dense
84.5
93.0
84.8
77.6
79.1
82.2
Qwen2.5-VT.t
Any Res.
M-RoPE
Dense
83.9
95.7
87.3
82.6
84.9
86.4
InternVL3'
Tile-wise
ID-RoPE
Dense
85.2
92.7
86.6
76.8
80.2
Sacoder-Bassed
Tile-wise
ID-RoPE
Dense
82.9
92.1
83.5
7.5
77.1
85.3
Native Vision-Language Models (8B)
Fuyu
Any Res.
ID- RoPE
Dense
64.5
36.6
Chameleon
Fix Res.
ID-RoPE
Dense
46.0
1.5
2.9
5.0
4.8
0.7
EVE
Any Rat.
ID-RoPE
Dense
61.0
53.0
59.1
25.0
56.8
39.8
SOLO
Any Res.
ID-RoPE
Dense
61.4
12.6
Emu3
Fix Res.
ID-RoPE
Dense
70
76.3
68.6
43.8
64.7
68.7
EVEv2
Any Rat.
ID-RoPE
DaC
74.8
73.9
71.1
70.2
BREEN
Any Res.
ID-RoPE
MoEi
76.4
65.7
VORA
Any
ID-RoPE
Dense
61.1
58.7
SAIL
Any Res.
M-RoPE
Dense
76.7
77.1
78.3
NEO
Any Res.
NaPP
Dense
83.1
88.6
82.1
60.9
75.0
77.7
Despite strong results, NEO lags Lnowledge-/OCR-heau tasks, e.g., MMMU, InfoVQA, and TextVQA. Interestingly, NE0-9B does not surpass NEO-2B on DocVQA and InfoVQA, indicating limitations in our current training corpus. Even so, NEO performs well under constraints, highlighting the native VLM as a scalable paradigm. Larger datasets and resources can unlock its full potential.
4.3 ABLATION STUDIES
Unless otherwise specified, we report the average evaluation results, denoted as Avg. across ten vision-language benchmark datasets Table 3. The pre-Buffer and new head dimensions in the post-LLM are trained on 20M pre-training samples, followed by full-backbone fine-tuning on 2M SFT instruction data. These constitute the standard training settings for our ablation studies.
Hyperparameters of the Pre-Buffer Layer. Figure 5 illustrates the relationship between the number of pre-
Avg. Accuracy (%)
60
Buffer layers and the model's ' average accuracy, using Qwen3-1.7B as the L...- Performance improves consistently as the layer count increases, but gains begin to saturate beyond eight layers. Inspired by the parameter counts of popular VEs (Chen et al., 2024f; Radford et al., 2021; Zhai et al., 2023) and the scaling properties (Tian et al., 2025) between existing VEs and LLMs, select 12 and 6 layers for NEO-2.2B and NEO-9B balance the trade-off between performance and efficiency.
46.8
47.2
48.8
39.7
40
30
20
2
6
10
12
Figure 5: Configurations of pre-Buffer.
8

---

Published as a conference paper at ICLR 2026
Table 3: Configurations of attention and RoPE. MMS, CQA, IVQA, and OCRB denote MMStar, ChartQA, Info and OCRBench. indicates that the base RoPE frequencies for height and width are set IM. To ensure fairness, we add new head dimensions of equal size across all models.
Model Attention
RoPE
MMMU MMB MMS SEED-I A12D CQA
IVQA
TVQA
OCRB POPE Avg.
Causal
ID-RoPE
40.2
36.1
55.3
16.1
22.5
13.9
78.6
B
Mixed
ID-RoPE
40.8
48.8
36.4
57.3
63.7
16.0
21.9
17.4
16.0
79.2
39.8
Mixed
IL-RoPE
36.3
57.6
62.0
18.8
23.4
13.2
39.5
D
Mixed
M-RoPE
40.3
49.6
37.2
57.8
64.2
23.7
25.2
20.4
18.8
79.3
41.7
Mixed
MM-RoPE
40.5
50.8
37.6
58.2
65.8
25.7
26.3
22.1
18.2
78.8
42.4
Mixed
Vidco-RoPE
40.6
51.3
37.8
58.8
64.3
27.4
26.1
23.7
21.3
81.0
43.2
Causal
Native-Roft
40.2
49.2
36.3
57.1
63.7
1972
23.5
19.5
16.7
77.8
40.3
H
Mixed
Native-RoPE
40.7
51.9
38.2
58.9
65.8
30.6
26.9
24.1
23.2
80.0
44.0
I
Mixed
Native-RoPEx
40.4
50.4
36.9
57.0
64.1
25.6
25.2
21.7
20.1
78.7
42.0
Avg. Accuracy (%)
Avg. Accuracy (%)
Avg. Accuracy (%)
80
80
BSage BSagee 2 TSagee
75
Dense
MoE
40
70
65
35
Dense
65
PR
CLIP
VVT
SigLUP
EVEVI
EVRvL.
EVR2
NRO
NEO-2.28
NEO-9R
Figure 6: Pre-Buffer VEs.
Figure 7: Neo EVE series.
Figure 8: Three training processes.
Configurations of Native Primitives. Table 3 compares various attention and RoPE designs. The pre-Buffer depth is 4, and the post-LLM is initialized with Qwen3-1.7B. All models share the same new QK head dimensions and normalization. (1) Attention mode. Comparing models A/B and G/H reveals consistent gains of mixed attention over causal one, reflecting its stronger capacity to model comprehensive dependencies and cross mooss alignment. (2) RoPE mode. Native-RoPP outperforms ID-RoPE (Zhu al., 2025), IL-RoPE (Liao al., 2025), M-RoPE (Bai al., 2025), MM-RoPE (Yuan et al., 2025), and Video-RoPP (Wei et al., 2025), with at least a 0.8% gain. This validates the importance of disentangling height, width, and temporal components in RoPE to enhance spatial-temporal representations and fine mnainne interactions. By contrast, setting the base RoPE frequency to IM for height and width severely impairs the ability to perceive local semantics.
Comparison between Pre-Buffer and Vision Encoders. In Figure 6, PB 1-3 denotes the Pre-Buffer obtained from stage 1-3. For fairness. all post-LLMs are initialized by Qwen3-1.7B (Yang et al., 2025) combined by pre-trained pre Buffer, CLIP-vit- -it--vit--rrge--- 300M (Chen et al., 2024e), and SigL.IP-s400m-ppal lit--laree-pcccc patch 14 (Radford et al., 2021). Intern ViT- 4-384 (Zhai et al., 2023). During pre-training, train the projector for meoder-hased methods and the newly added QK parameters for all models. During SFT, we unfreeze the entire backbone. After two-stage re-training, PB3 delivers strong performance. trailing CLIP/ / Intern VIT SigLIP by only 1.7 2.4 3.7% on average across diverse benchmarks. Notably, despite using only 22M training samples, PB3 is just 2.5% below the full NEO model, substantially reducing the training cost of developing native VLMs for future research.
Comparison between NEO and Native VLM variants. Existing native VLMs in Table and differ dramatically training pipelines, data volume/quality, and base LLM choice. To isolate model architectural effects, we compare our NEO with representative EVEv1.0 (Dense), EVEv1.5 Mfixture-of-Fxpeeee and EVEx2.0 DDivideannd-Conque
built on Qwen3-1.7B. From Figure 7, EVEv1.0, EVEv1.5, and EVEv2.0 achieves 33.4%. 40.2%. and 41.5%, respectively, compared with 44.0% for our NEO. This confirms that the gains stem from our pre- pre-Buffer. native RoPE design, and interaction patterns, rather than from newer backbone or larger and higher-quality data alone.
Performance Gains across Stages. Figure 8 presents the result evolution across training stages. In Stages and 2, the model is fine-tuned 2M SFT examples. Performance improves consistently as training data scales increase across 2.2B and 9B model sizes. Following progressive training, NEO shows strong multimodal capabilities, enabling robust performance across diverse real-world tasks.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We introduce NEO, a native VLM that seamlessly integrates vision and language into a single unified framework, eliminating the need for separate visual encoders ad-hoc alignment modules. By leveraging hybrid attention and modality-aware rotary position embeddings. NEO captures rich, fine-grained interactions between pixels and words from the outset. Its pre-Buffer and post-LLM training paradigm ensures efficient convergence and robust alignment while maintaining end-to-end learning. Experiments show that this unified design not only advances multimodal understanding and reasoning but also lays the foundation for reusable, scalable components. Our native primitives highlight promising path toward intrinsically multimodal, unified, and adaptable architectures.
ETHICS STATEMENT
All resources drawn from open-access datasets with explicitly defined usage policies. Our work seeks to advance multimodal learning capabilities without introducing ethical or safety concerns beyond those already associated with existing models. Nevertheless, risks such as dataset biases and potential misuse cannot be entirely ruled out. We emphasize the importance of careful data curation, responsible deployment, and transparent reporting as essential practices to mitigate these challenges.
REPRODUCIBILITY STATEMENT
We place strong emphasis reproducibility, providing detailed descriptions to facilitate replication and validation. Information about dataset selection, training strategies, and evaluation settings is provided in Sec. 3.2 and Sec. 4.1. We commit releasing the code, model weights, and detailed documentation to allow the community to reproduce our findings in future research.
ACKNOWLEDGEMENT
This research is supported by cash and in-kind funding from NTU S-Lab and industry partner(s). This study is also supported by the Ministry of Education, Singapore, under its MOE AcRF Tier (M0E-12EEP20221-0012 MOE-222220223-000).
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski. Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In Advances of Neural Information Processing Systems, New Orleans, LA, USA, 2022.
Anthropic. Claude 3.7 sonnet: A hybrid reasoning ai model, 2025. URL https: //www.. anthropic ctt.:i...c...ceooooooomommo.oo hwwpe//wwweee-e-eeeeeeeeee
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li. Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. CoRR, abs/2502. 139233..1392, 2025.
Yuelin Bai, Xinrun Du, Yiming Liang, Yonggang Jin, Junting Zhou, Ziqiang Liu, Feiteng Fang, Mingshan Chang, Tianyu Zheng, Xincheng Zhang, et al. Coig-cqia: Quality is all you need for chinese instruction fine-tuning. CoRR, abs/2403. 18058, 2024.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Taşırlar Introducing our multimodal models, 2023. URL httpa: wwwpt//wwwwppp..//////////// a bl blog/ fuyu-8b. 8b.
10

---

Published as a conference paper at ICLR 2026
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Gryener, Alexey A. Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong. Julian Fisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bosnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinclopi Papalampidi, Olivier J. Hénaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiaohua Zhai. Paligemma: a versatile 3b vlm for transfer. CoRR, 0ins/2407.077266 2024.
Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluis Gómez i Bigorda, Marçal Rusiñol. C. V. Jawahar. Ernest Valveny, and Dimosthenis Karatzas. Scene text visual question answering. In IEEE International Conference on Computer Vision, pp. 4290-4300, Scoul, Korea (South), 2019.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sachoon Kim. Coyo-700m: Image-text pair dataset, 2022. URL https: / /ggthtt.thhhgggggggg com/ kakaobrain coyo-datasee dataset.
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding, In International Conference on Computational Linguistics, pp. 1511- 1520, 2022.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang. Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: harnessing spr4v-synthesized data for a lite vision-language model. CoRR, abs/2402. 1684, 2024a.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang. Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal aachitsarchiteectuee, training and dataset. CoRR, abs/2505 ...5550,,66, 2025.
I Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: improving large multi-modal models with better captions. In European Conference on Computer Vision, volume 15075, pp. 370-387, Milan, Italy, 2024b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are the right way for evaluating large vision- language models? In Advances of Neural Information Processing Systems, Vancouver, BC, Canada, 2024c.
Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. A single transformer for scalable vision- language modeling. CoRR, abs/2407. 06438, 2024d.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv. Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge. Kai Chen, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. CoRR, abs/2412.05271, 2024e.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ma. Jiaqi Wang. Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang. Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. CoRR, abs/2404.16821. 2024f.
Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al. lcdar2019 robust reading challenge arbitrary-shaped text-rre-art. In International Conference on Document Analysis and Recognition. pp. 1571-1576, 2019.
Christopher Clark and Matt Gardner. Simple and effective multi multi- paragrapp reading comprehension. In Annual Meeting of the Association for Computational Linguistics, pp. 845-855, Melbourne, Australia, 2018
11

---

Published as a conference paper at ICLR 2026
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NVLM: open frontier-class multimodal Ilms. CoRR, (bs/2409.11402, 2024.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1080-1089, Honolulu, HI, USA, 2017.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International Conference on Machine Learning, volume 70, pp. 933-941, Sydney, NSW, Australia, 2017.
Google DeepMind. Gemini 2.5 pro: Google's most advanced reasoning model, 2025. URL https: // //deppinnd g modeeooesssssoeooo s gemini/miii// /ptini///////////
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang. Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs/2501.12945 2025.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song. Shi Guang, and Haoqi Fan. Emerging properties in unified multimodal pretraining. CoRR, abs/2505. 100005..4666 14683, 2025a.
Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. In International Conference on Learning Representations, Singapore, 2025b.
Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models. CoRR, abs/2406.1 1832, 2024.
Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. Evev2: Improved baselines for encoder-free lisoon-language models. CoRR, abs/2502.06788, 2025.
Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. CoRR, abs/2412.05446,, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: transformers for image recognition at scale. In International Conference on Learning Representations, Austria, 2021.
Haodong Duan, Junming Yang. Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. VImevalkit: An open-source toolkit for evaluating large multi modaliyy models. In ACM International Conference on Multimedia, pp. 11198-11201, Melbourne, VIC, Australia, 2024.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. EVA: exploring the limits of masked visual representation learning at scale. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 19358-19369 Vancouver, BC, Canada, 2023.
12

---

Published as a conference paper at ICLR 2026
Yash Goyal, Tejas Khot, Douglas Summers-Stee Dhruv Batra, and Devi Parikh. Making the in vqa matter: elevating the role of image understanding in visual question answering. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 6325-6334, Honolulu, HI, USA, 2017.
Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Niu Minzhe, Xiaodan Liang, Lewei Yao, Runhui Huang, Wei Zhang, Xin Jiang, Chunjing Xu, and Hang Xu. Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. In Advances of Neural Information Processing Systems, New Orleans, LA, USA., 2022.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 14375- 14385, Seattle, WA, USA, 2024.
Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seedl. 5-vl technical report. CoRR, abs/2505.00 070662, 2025.
Conghui He, Zhenjiang Jin, Chao Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan, Jiaqi Wang, and Dahua Lin. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models. CoRR, abs/2308. 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). CoRR, abs/ /bbs/1606.08415, 2016.
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 I v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. CoRR, abs/2507.01000 2025.
Drew A. Hudson and Christopher D. Manning. GQA: new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Putlern Recognition, pp. 6700-670 6709, Long Beach, CA, USA, 2019.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker- Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Back, Angela Jiang, An- toine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig. Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi. Carroll L. Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger. Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, and Dane Sherburn. Gpt-4o system card. CoRR, abs/2410.21276. 2024.
Jimmycarter. Textocr gpt-4v dataset, 2023. URL https: https//ttttttt/////gg//ttggee http:///giinacc.c///oaaaaaaaaaaa jimmycrteer/eextoca gptr...
Kushal Kafle, Brian L. Price, Scott Cohen, and Christopher Kanan. DVQA: understanding data visualizations via question answering. In IEEE Conference Computer Vision and Pattern Recognition, pp. 5648-5656, Salt Lake City, UT, USA, 2018
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European Conference on Computer Vision, volume 9908, pp. 235-251, Amsterdam, The Netherlands, 2016.
13

---

Published as a conference paper at ICLR 2026
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprchension. In JEEE Conference Computer Vision and Pattern Recognition, pp. 4999-5007, 2017.
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeong Yeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision, volume 13688, pp. 498-517, Tel Aviv, Israel, 2022.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances of Neural Information Processing Systems, pp. 1106-11 1006-1114 1114, Lake Tahoe, Nevada, US, 2012.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: unified image classification, object detection, and visual relationship detection at scale.
CoRR, abs/1811 1.00922,, 2018.
LAION. Gpt-4v dataset, 2023. URL http::///hgggggggaaaaeeceaaaaaae
gpt 4v-aatsse.
Weixian Lei, Jiacong Wang, Haochen Wang, Xiangtai Li, Jun Hao Liew, Jiashi Feng, and Zilong Huang. The scalability of simplicity: Empirical analysis of vision-language learning with a single transformer. CoRR, abs/2504.1046t 2025.
Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, a dataset for knowledge-basee based visual question answering about named entities. In ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 3108-3120, 2022.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: stronger lims supercharge multi- modal capabilities in the wild, 2024a. URL https://llava htt.:/awwwww.ggabb..bb//.//oee wtithub..obbllo... 2024-05- 105-10-111 10ll11vlla salvv---ett--.. next stronger- Eerronner-llms/...
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: easy visual task transfer. CoRR, abs/2408.03326. 2024b.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: bench- marking multimodal Iims with generative comprehension. CORR, abs/2307.16125. 2023a.
Han Li, Xinyu Peng, Yaoming Wang, Zelin Peng, Xin Chen, Rongxiang Weng, Jingang Wang, Xunliang Cai, Wenrui Dai, and Hongkai Xiong. Onecat: Decoder-only auto-regressive model for unified understanding and generation. CoRR, abs/2509.03498, 2025a.
Tianle Li, Yongming Rao, Winston Hu, and Yu Cheng. BREEN: bridge data-efficient encoder-free multimodal learning with learnable queries. CoRR, abs/2503. 12466, 2025b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- languala models. In Conference on Empirical Methods in Natural Language Processing, pp. 292-305, Singapore, 2023b.
Zhuowan Li, Xingrui Wang. Elias Stengel-Esking Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robust- ness in visual reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 14963-14973, 2023c.
14

---

Published as a conference paper at ICLR 2026
Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, and Xi Victoria Lin. ofeture-of-tramsfoone A sparse and scalable architecture for
foundation models. CoRR, abs/241 2024.
Chao Liao, Liyang Liu, Xun Wang. Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation. CoRR, abs/2505.05472, 2025.
Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer, and Armen Aghajanyan. Moma: efficient early-fusion pre-training with mixture of modality-aware experts. CoRR, abs/2407.21170, 2024.
Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. CoRR, abs/2208.05358, 2022.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 1 11:65565. 651, 2023a.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. CoRR, 2023b.
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mme: Advancing multimodal chart understanding with large-scale instruction tuning. CoRR, abs/2311.10777 2023c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances of Neural Information Processing Systems, New Orleans, LA, USA, 2023d.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 26286-26296.
Seattle, WA, USA, 2024a.
Xi Liu, Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, et al. Icdar 2019 robust reading challenge on reading chinese text on signboard. CORR, 1bs/1991.00641, 2019.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: is your multi-moda model an all-around player? In European Conference on Computer Vision, volume 15064, pp. 216-233, Milan, Italy, 2024b.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. CoRR, abs/2305.078 007955 2023e.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, New Orleans, LA, USA, 2019.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chin Long-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. CoRR, abs/2105.04415. 2021.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Chang, Gong-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: multimodal reasoning via thought chains for science question answering. Advances of Neural Information Processing Systems, volume 35, pp. 2507-2521, New Orleans, LA, USA, 2022a.
Pan Lu, Liang Qiu, Wei-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. CoRR, abs/2209, 14610, 2022b.
Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jifeng Dai, Yu Qiao, and Xizhou Zhu. Mono- internvl: pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. CoRR, abs/2410.08202, 2024.
15

---

Published as a conference paper at ICLR 2026
Gen Luo, Wenhan Dou, Wenhao Li, Zhaokai Wang. Xue Yang. Changyao Tian, Hao Li, Weiyun Wang, Wenhai Wang, Xizhou Zhu, Yu Qiao, and Jifeng Dai. Mono-intermvi-1.5: Towards cheaper and faster monolithic multimodal large language models. CoRR, abs/2507.12666 2025.
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In IEEE Conference Computer Vision and Puttern Recognition, pp. 11 11-20, Las Vegas, NV, USA, 2016.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: a visual question answering benchmark requiring external knowledge. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3195-3204, Vienna, Austria, 2019.
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqu: a benchmark for question answering about charts with visual and logical reasoning. In Annual Meeting of the Association for Computational Linguistics, pp. 2263-2279, Dublin, Ireland, 2022.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. Infographicvqa. In IEEE Winter Conference on Applications of Computer Vision, pp. 2582-2591, Waikoloa, HI, USA, 2022.
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In IEEE Winter Conference Applications of Computer Vision, pp. 1527-1536, 2020.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: visual question answering by reading text in images. In International Conference on Document Analysis and Recognition, pp. 947-952, Sydney, Australia, 2019.
OpenAl. Gpt-5: A unified multimodal model, 2025. URL wwtpt:tt//ppeaaaaeeeeeee com/ gpt- -
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, volume 139, pp. 9748-8763, virtual, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Muliis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-40MM: open dataset of clip-filterdd 400 million image-text pairs. CoRR, abs/2111.02114 2021.
Christoph Schuhmann, Andreas Köpf, Richard Vencu, Theo Coombes, and Romain Beaumont, Laion coco: 600m synthetic captions from laion2b-en, 2022. URL https: alblogg
1aion-coco/.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-OKVQA: a benchmark for visual question answering using world knowledge. In European Conference Computer Vision, volume 13668, pp. 146-162, Tel Aviv, Israel, 2022.
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar, Kvqa: Knowledge- aware visual question answering. In AAAI Conference Artificial Intelligence, volume 33, pp. 8876-8884, 2019.
Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge Belongie, Shijian Lu, and Xiang Bai. lcdar2017 competition on reading chinese text in the wild (retw-17). In International Conference on Document Analysis and Recognition, volume 1, pp. 1429-1434. IEBE, 2017.
Mustafa Shukor, Louis Béthune, Dan Busbridge, David Grangier, Enrico Fini. Alaaeldin El-Nouby, and Pierre Ablin. Scaling laws for optimal data mixtures. CoRR, abs/2507.09 .09404, 2025a.
Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua M. Susskind, and Alaaeldin El-Nouby. Scaling laws for native multimodal models. CoRR, abs/2504.07951, 2025b.
16

---

Published as a conference paper at ICLR 2026
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In European Conference on Computer Vision, volume 12347, pp. 742-758, Glasgow, UK, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruy Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In IEEE Conference Computer Vision and Pattern Recognition, pp. 8317-8326, Long Beach, CA, USA, 2019.
Yipeng Sun, Zihan Ni, Chee-Kheng Chng. Yuliang Liu, Canjie Luo, Chun Chet Ng, Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, et al. Iedar 2019 competition on large-scale street view text with partial bebeliin--rr-lsvt In International Conference on Document Analysis and Recognition, pp. 1557-1562, 2019.
Chenxin Tao, Shiqian Su, Xizhou Zhu, Chenyu Zhang, Zhe Chen, Jiawen Liu, Wenhai Wang, Lewei Lu, Gao Huang, Yu Qiao, and Jifeng Dai. Hovle: Unleashing the power of monolithic vision- language models with holistic vision-language embedding. In JEEE Conference on Computer Vision and Pattern Recognition, pp.
Nashville, TN, USA, 2025.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. hllps://crfm. stanford. edu/20233033/1 html, 3(6):7, 2023.
Chameleon Team. Chameleon: mixed-modas early-fusion foundation models. CoRR, abs/2405.098 2024.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023. URL hwt.h:ttt////tt. ng
cw/datttttoooooooooooo deaatateeeesaeeeee..eaaaaaeeeete teett///ekkre teknin um/ htt//Oppeeeemess-e 2.5.
Changyao Tian, Hao Li, Gen Luo, Xizhou Zhu, Weijie Su, Hanming Deng, Jinguo Zhu, Jie Shao, Ziran Zhu, Yunpeng Liu, et al. Navil: Rethinking scaling properties of native multimodal large language models under data constraints. CoRR, abs/25 10.08665, 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: open foundation and fine-tuned chat models. CoRR, Abs/2307.09288. 2023.
Michael Tschannen, Alexey A. Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier J. Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision- language encoders with improved semantic understanding, localization, and dense features. CoRR, 147886 2025.
Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. CoRR, abs/1601.077 .07140, 2016.
Han Wang, Yongjie Ye, Bingru Li, Yuxiang Nie, Jinghui Lu, Jingqun Tang, Yanjie Wang, and Can Huang. Vision as lora. CoRR, abb/25533.2062. 2025a.
Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: prompting GPT-4V for better visual instruction tuning. CoRR, 11755774 2023.
17

---

Published as a conference paper at ICLR 2026
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: vl: enhancing vision- lsiounlanguage model's perception of the world at any resolution. CoRR, abs/2409. 12191, 2024a.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. CoRR, abs/2508. 182655 2025b.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang. Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: next-token prediction is all you need. CoRR, abs/2409. 188629 2024b.
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo. Jiaqi Wang, Xipeng Qiu, and Dahua Lin. Vidcorope: What makes for good video rotary position embedding? CoRR, abs/2502.05117. 2025.
Chenyuan Wu, Pengfei Zheng. Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang. Yexin Liu, Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng, Jiahao Wang, Kun Luo, Bo Zhang. Defu Lian, Xinlong Wang. Zhongyuan Wang. Tiejun Huang, and Zheng Liu. Omnigen2: Exploration to advanced multimodal generation. CoRR, abs/2506. 18877, 2025.
XAI. Grok-1.5 vision preview, 2024. URL htt.htss//x..aaibll////ttt
xAI. Grok 3: xAI's flagship ai model, 2025. URL /tLLtttts////xaa////e//rrro
Yicheng Xiao, Lin Song, Rui Yang, Cheng Cheng, Zunnan Xu, Zhaoyang Zhang, Yixiao Ge, Xiu Li, and Ying Shan. Haploomni: Unified single transformer for multimodal video understanding and generation. CoRR, abs/2506.02975. 2025.
Rui Yan, Lin Song, Yicheng Xiao, Runhui Huang, Yixiao Ge, Ying Shan, and Hengshuang Zhao. Haplovl: A single-transformer baseline for multi-modal understanding. CoRR, abs/2503. 14694, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li. Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report.
CoRR, abs/2505.0925 003888, 2025.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions. In European Conference on Computer Vision, volume 9906, pp. 69-85, Amsterdam, The Netherlands, 2016.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. CoRR, abs/2309. 12284, 2023.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: evaluating large multimodal models for integrated capabilities. In International Conference on Machine Learning, Vienna, Austria, 2024.
Hangjie Yuan, Weihua Chen, Jun Cen, Hu Yu, Jingyun Liang, Shuning Chang, Zhihui Lin, Tao Feng, Pengwei Liu, Jiazheng Xing, Hao Luo, Jiasheng Tang, Fan Wang, and Yi Yang. Lumos-I: On autoregressive video generation from a unified model perspective. CoRR, abs/2507.0884 2025.
Tai-Ling Yuan, Zhe Zhu, Kun Xu, Cheng- Jun Li, Tai-Jiang Mu, and Shi-Min Hu. A large chinese text dataset in the wild. Journal of Computer Science and Technology, 34(3)500-551, 2019.
18

---

Published as a conference paper at ICLR 2026
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu. Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. MMMU: a massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In IEEE Conference Computer Vision and Pattern Recognition, pp. 9556-9567, Seattle, WA, USA, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In IEEE International Conference on Computer Vision, pp. 11941-11952, Paris, France, 2023.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances of Neural Information Processing Systems, pp. 22360-12377 Vancouver, BC, Canada, 2019.
Bo Zhao, Boya Wu, and Tiejun Huang, SVIT: scaling up visual instruction tuning. CoRR, abs/2307.04087, 2023.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as -ln-as-a-judg with mt-bench and chatbot arena. Advances of Neural Information Prucessing Systems, 36:46595-46623, 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang. Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang. Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang. Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu. Kaipeng Zhang, Huipeng Deng, Jiaye Ge, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-tims recipes for open-source multimodal models. CoRR. abs/2504. 10479, 2025.
19

---

Published as a conference paper at ICLR 2026
A
APPENDIX
USAGE OF LARGE LANGUAGE MODELS
During manuscript preparation, large language models were used solely writing assistants. They helped to check grammar, refine sentence structure, and provide style alternatives. All content related methodology, experiments, and conclusions was developed entirely by the authors. LLM outputs were reviewed critically, and only human-verified edits were incorporated into the final text.
A.1
SUPERVISED FINE-TUNING DATASETS
Table 4: Dataset summary in supervised fine-tuning stage.
Task
Dataset
Captioning
TextCaps (en) (Sidorov et al., 2020), ShareGPT4V (enkzzh (Chen et al., 2024b)
VQAv2 (en) (Goyal al., 2017), GQA (en) (Hudson & Manning, 2019), OKVQA (en) (Marino al., 2019),
General QA
VSR (en) (Liu et al., 2023a), VisualDialog (en) (Das et al., 2017)
Science
AI2D (en) (Kembhavi al., 2016). ScienceQA (en) (I. et al., 2022a). TQA (en) (Kemhhavi al., 2017)
ChanQA (en) (Masry et al., 2022), MMC-Inst (en) (Liu al., 2023c). DVQA (en) (Kalle al., 2018).
Chart
PlatQA (en) (Methani al., 2020), LRV Instruction (en) (I.iu et al., 20235)
GeoQA+ (en) (Cao & Xiao, 2022), TabMWP (Lu et al., 2022b), MatiQA (en) (Yu el., 2023).
Mathematics
CLEVR Math/Super (en) (Lindström & Abraham, 2022; Li et al., 2023c), Geometry3K (en) (Lu et al., 2021)
KVQA (en) (Shah et al.. 2019), A-OKVQA (en) (Schwenk et al., 2022), VIQuAE (en) (Lerner et al., 2022),
Knowledge
Wikipedia (en&zth (He et al., 2023)
OCRVQA (cn) (Mishra et al., 2019), InfoVQA (cn) (Mathew et al., 2022), TexVVQ (en) (Singh et al., 2019),
ArT (en&zh) (Chng
2019), COCO-Text (en) (Veit al., 2016), CTW (2h) (Yuan et al., 2019),
OCR
LSVT (zh) (Sun et al., 2019), RCTW-17 (2h) (Shi et al., 2017). RcCTs (2h) (Liu ot al., 2019).
SynthDoG (en&zzh) (Kim et al., 2022), ST-VQA (en) (Biten et al., 2019)
Document
Doc (en) (Clark & Gardner, 2018), Common Crawl PDF (cn&zh)
Grounding
Re(COCO/+/g (en) (Yu et al., 2016; Mao et al., 2016). Visual Genome (en) (Krishna et al., 2017)
LLaVA-150K (en&tzh) (Liu et al., 2023d), LVIS-Instues (en) (Wang al., 2023).
Conversation
ALLaVA (en&zh) (Chen et al. 2024a), Laion-GPT4V (en) (LAION, 2023).
Tex1OCR-GPTAV (en) (Jimmycarter, 2023), SVIT (endtzh) (Zhao ul., 2023)
OpenHermes2.5 (en) (Teknium, 2023), Alpaca-GPT4 (en) (Taori al., 2023), COIG-CQIA (zh) (Bai et., 2024).
Text-only
ShareGPT (en& (Zheng et al., 2023)
A.2 IMPLEMENTATION DETAILS
Table 5: Implementation details in the pre- pre-training, mid-training and supervise fine-tuning.
Configuration
Pre- Pre-Training
Mid-Training
Supervised Fine-Tuning
Resolution
256 - 0242
256 - 2, 0482 AdamW
256 2, 0482
Optimizer
Optimizer hyperparameters
31 0.9, B/ 0.999,
eps 1
Learning rate schedule
cosine with min Ir 8e-4
cosine with min Ir -
cosine decay 5e
Peak learning rate
Min learning rate ratio
0.05
0.1
Weight decay
0.01
Training steps
190k 2k
50k
6k
Warm-up steps
200
200
Max sample length
8, 192
8, 192
8, 192
Global batch size
2, 560
1, 200
650
Text-only ratio
0.3
0.3
Numerical precision
bfloat16
20

---

Published as a conference paper at ICLR 2026
A.3 LIMITATION AND DISCUSSION
In this study, we innovate network architectures and training strategies for efficiently building native vision-language models. The full promise of NEO has remained largely untapped, hindered by scarce training data and limited computational resources, especially in knowledge-intensiv and OCR-focused domains. Yet, strikingly, our NEO rivals state-of- the-art VLMs despite these severe constraints. We envision subsequent directions of NEO for the native VLM community as follows:
Contextual relevance recent advancements. Recent models such as Qwen3VL QwiniVLL concepts that resonate with our design choices, including dense linking of visual-language features, relative positional encodings, and architectural details like patch embedding and bias. In particular, the DeepStack approach underscores the importance of establishing strong pixel-word associations from the earliest stages, reinforeing the significance of densely integrated visual-language representations.
Maximizing the potential via large investment. It is in great demand for continuously investing substantial resources, especially during the pre-training stage, to fully unlock NEO's performance and approach the upper bound of the native model. At the same time, selectively open-soureing key components during intermediate development can reduce follow-up training costs for future researchers and attract models from this work provide a valuable baseline for advancing reinforcement learning research.
research to native visual-language models. Moreover, the fundamental
Explorations of full-spectrum model capacities. Expanding the full model sizes remains a critical factor in advancing various real-world applications. Even with limited resources, NEO-2.2B closely matches those of modular tisual-languag models with equivalent capacity, suggesting that the design philosophy of models in the 0.6 to 8 billion parameter range has matured. Such architectures not only achieve high performance but also facilitate the deployment of lightweight models the edge, which is crucial for scenarios with limited computational resources or strict real-time time requirements.
Upgrading architectures and applications. To date, our work has focused on dense models for mage-text understanding, while sparse devide-and-conqee active development. Notably, we regard NEO not merely as an autoregressive VL.M but new paradigm for visual-language intelligence. Its principle is to leverage end-to-end training within unified architecture, eliminating manually imposed biases and scaling-up complexities by allowing data and models dictate the learning process. Besides, our efforts are designed not merely to improve performance but to establish a definitive baseline for visual-language generation, long video understanding, and embodied AI. Crucially, NEO's architecture systematically integrates the demands of video generation and related tasks, including attention mechanisms and rotary positional encodings, from the ground up. Although currently focused text and images, NEO is poised to push the boundaries of what is possible across a wide spectrum of application scenarios and input modalities.
architecture is simultaneously under
21


Reviews:
[{"review_id": "IWfOG1IJtI", "reviewer": "Reviewer_QS4o", "summary": "The authors present Neo, a family of native Vision-Language Models built on top of Qwen3-1.7B and Qwen3-8B. The key elements of the Neo architecture are (1) an attention block that decouples H, W and T dimensions for Query and Key computation; (2) RoPE position embeddings that use separate frequencies for H, W and T dimensions; (3) bidirectional attention for images; (4) added transformer layers (\"Pre-Buffer\") to project vision and text embeddings to the same embedding space. The authors train Neo models on 390M image-text examples in a process containing three stages and compare against prior modular VLMs using pretrained vision encoders and native VLMs. The trained models obtain competitive performance on various datasets against modular VLMs, while outperforming prior work on native VLMs.", "questions": "- A majority of the citations should be changed to parentheticals, rather than in-text citations with \\citet.\n- As a follow-up to my point regarding the Pre-Buffer, I also struggled at understanding what exactly was done in the \"Comparison between Pre-Buffer and Vision Encoders\" section. Would it be correct to say that training was repeated here with InternViT/CLIP/SigLIP used in place of the Pre-Buffer?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Each architectural component is motivated with the incorporation of inductive biases that make the processing of images in native VLMs more similar to modular VLMs, which I found to be intuitive.\n- The decoupling of H, W and T dimensions in both the Query and Key computation as well for RoPE computation is novel to the best of my knowledge.\n- The training process does not assume access to any pretrained vision encoders.\n- The various ablation studies in Section 4.3 are very valuable in arguing for the importance of various design choices in the attention blocks, particularly of the suggested adjustment to RoPE. This is especially the case when all native VLM models compared against make use of different datasets or pre-trained LLMs.", "weaknesses": "While I believe the contribution is valid, in large part due to the ablation studies, there are various flaws to the paper:\n- Firstly, there are clear errors in the related works section. We do not know the underlying architecture for multimodal GPT models and it is therefore incorrect to claim that they are modular or native. It is also important to note that GPT-4o, being able to both condition on and generate images, is more similar to a model like Chamelon (included in the native model section) than standard modular model counterparts. It is likewise an issue to make claims regarding Claude or Gemini.\n- I would argue that saying Neo \"rivals top-tier modular counterparts\" in the abstract and introduction is overclaiming. For general vision benchmarks, Neo approaches modular model performance, but nonetheless falls short in each case. This is particularly an issue for MMMU and MMVet, where 8B model performance is at least ~20% poorer. This and the bullet point above are the rationale for the soundness score.\n- Likewise, any comparison made to prior native VLMs has the confounder of Neo making use of the newer and stronger Qwen3 backbones. This is mitigated by the ablation studies, which are much welcome.\n- I found the writing to be unclear, particularly for sections dealing with the \"Pre-Buffer\" and \"Post-LLM.\" Some added formalism about what computation each of these components perform and which components are initialized from the Qwen3 backbone and which are not would make understanding easier. Figure 2 sadly is unclear as both the Pre-Buffer and Post-LLM components make use of the same primitives but just differ in color.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "03JmK9fhjO", "reviewer": "Reviewer_JqhJ", "summary": "The paper proposes a novel approach to train autoregressive, monolithic vision-language models that omit domain-specific vision encoders in favor of light-weight encoders and a native multi-modal training resulting in the *NEO* models.  The model consists of a small two-layer convolution encoder, a multimodal pre-buffer, and a pretrained LLM (Qwen3). The attention and position encoding procedure is optimized for multi-modality. The model training comprises of 3 stages with 390M image-text examples. The resulting 2.2B/9B models are thoroughly benchmarked against prior modular and monolithic VLMs, and outperform all prior models in the latter category.", "questions": "- Are there any insights why the performance on HallusionBench and \"knowledge-heavy\" tasks suffers? Fundamentally, this does not seem like a multi-modal problem to me.\n- Please review the LLM written parts for clarity.\n- Please consider using \\citep to improve legibility", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper proposes improvements in native, monolithic multimodal LLM training by introducing a pre-Buffer for better alignment (trained separately in the first stage), native rotary position embeddings with modality-specific base frequencies, native multimodal attention (causal for text, full-bidirectional for vision) with decoupled H, W, T processing. \n- Using 390M image-text examples, NEO reaches a high performance and outperforms all previous native VLMs.\n- NEO is built on top of a modern LLM (Qwen3) and supports flexible resolution\n- The authors provide NEO-2B and 9B intermediate and final checkpoints\n- The models are thoroughly evaluated against other models, including modular and native models in 2B and 8/9B categories\n- Some design choices (number of layers in the pre-Buffer and attention/embedding methods) are ablated", "weaknesses": "- I keep wondering why NEO-9B uses a 50% smaller pre-Buffer than 2.2B. The paper mentions \"mainly due to the good trade-off between performance and efficiency.\" (L431) but does not provide evidence for that. I am not convinced that the results in Fig. 5 extrapolate to a larger post-LLM.\n- Some systematic ablations of design choices are often missing. The number of layers in the pre-Buffer and attention/embedding methods are ablated but nothing else. This left wondering which design choices in NEO actually impact performance: e.g., is it the data (quality/quantity)? The stages? The more modern LLM (related work uses older LLMs)? Or, is it actually the proposed combination of design choices. I understand that providing such controlled ablation experiments might not be economically feasible, but they obfuscate the contribution nonetheless. \n- NEO still significantly lags behind modular VLMs, even older ones like Qwen2-VL (e.g., 16% on InfoVQA). Given the relative improvements in its category and the lower amount of data this is not a big issue itself, however the phrasing in \"Comparison with Modular VLMs.\" (L357ff) is a bit overselling.\n- The scaling improvements between 2B and 9B seem modest compared to modular VLMs, \"casting shadows\" over the scalability of NEO.\n- Some parts of the paper feel LLM-generated by overusing (sometimes nonsensical) synonyms, making it hard to follow the paper. I would encourage the authors to manually revise the paper.\n- Fig. 1/3 are densely packed and hard to comprehend.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uGzW0Rn6R7", "reviewer": "Reviewer_EYZR", "summary": "This work proposes a novel monolithic large vision-language model, called NEO, supported by an improved rotary positional embeddings (RoPE) mechanism and multi-stage training. NEO is also backed by several existing techniques, such as hybrid attention masking, and shared FFN, attention and norm layers for vision and language modalities. With evaluations reported on several well-established benchmarks, the work aims to demonstrate the performance of the proposed method.", "questions": "- Given the similarities in performance with Video-RoPE [E] and the proposed RoPE mechanism in this work (Table 3), how well the proposed RoPE mechanism compare against it under smaller training budgets, such as those utilized for [A, B]? \n\n- Can you comment on the fairness of evaluations raised from above? How do you think the differences in training and architectural settings could be effecting the evaluation results and how do you think you could address these?\n\n---\n[A] Diao, H., Cui, Y., Li, X., Wang, Y., Lu, H., & Wang, X. (2024). Unveiling encoder-free vision-language models. Advances in Neural Information Processing Systems, 37, 52545-52567.\n\n[B] Diao, H., Li, X., Cui, Y., Wang, Y., Deng, H., Pan, T., ... & Wang, X. (2025). Evev2: Improved baselines for encoder-free vision-language models. arXiv preprint arXiv:2502.06788.\n\n[E] Wei, X., Liu, X., Zang, Y., Dong, X., Zhang, P., Cao, Y., ... & Lin, D. (2025). VideoRoPE: What Makes for Good Video Rotary Position Embedding?. arXiv preprint arXiv:2502.05173.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The primary strengths of the work could be listed as the following:\n\n- The particular idea of leveraging an improved RoPE mechanism (called \"Native RoPE\" in the work) in the context of monolithic vision-language models is novel.\n- The work includes a very thorough literature review with more than sufficient citations to contemporary works, even those released within the same year, which is much appreciated.\n- Evaluation is performed on a decent number of benchmarks, and there are a number of ablations on the positional encoding strategy, which is also a great to have.\n\nRelatively minor strengths of the work could be listed as the following:\n\n- Overall structure and flow of thought presented in the work is decent.\n- The figures are of very high quality and are visually appealing, though a bit crowded (see the minor weaknesses below).", "weaknesses": "The primary weaknesses of the work could be listed as the following:\n\n**W1: Architectural and Training-time Adjustments Similarities with Existing Works:** The work borrows heavily from two existing works, EVE [A] and EVEv2 [B] in both of its architectural and training-time adjustments. In particular, sharing the norm layers, attention blocks and FFN blocks have been explored in [A], patch embedding and word embedding layers are nearly identical to [B], and the _native multi-modal attention_ is the standard practice in large VLMs [C, D]. Furthermore, the overall training strategy is nearly identical to [B], with the Stage 1 pretraining corresponding to [B]'s Stage 1 & 2.1, Stage 2 mid-training corresponding to [B]'s Stage 2.2. and Stage 3s matching. Normally, having these similarities would not be a major weakness if it was not for the narrative of the work, which, in its current form, appears to present these as novelties of the proposed framework.\n\n**W2: Ambiguities in the Narrative:** There are several ambiguities in the narrative of the work. Most importantly, due to the aforementioned W1 the exact contributions of the work are not very clear to read from the text, as the current narrative renames a few well-established practices in the field in a rather ad-hoc manner. To exemplify, these include renaming the standard hybrid attention masking in the literature to \"Native Multi-modal Attention\" or renaming the monolithic blocks of [A, B] to \"Native VLM primite\" while the only architectural difference from [A, B]'s blocks is the improved RoPE mechanism and the added Q and K parameters that go along with it. Finally, I found it a bit hard to grasp the exact changes introduced over existing works in Section 3.1 in general, with several terms like \"Pre-Buffer\" not being well-defined.\n\n**W3: Fairness of Evaluations:** The work indeed includes a good number of benchmarks and a good number of ablations trying out different RoPE variants within the same framework. However, one critical thing that is lacking is a fair comparison between [A, B] and this work. Notably, NEO utilizes a much better LLM compared to the baselines considered (Qwen 3 versus older Qwen 2.5/Vicuna variants) in Table 1 and it was also trained with much more data than many of them. Given NEO does not add much beyond the improved RoPE mechanism over [A, B] architecturally, a more fairer comparison would demand them trained with similar budgets or at least with similar performing LLMs.\n\nRelatively minor weaknesses of the work could be listed as the following:\n\n- In many parts of the text the usage of \\citep and \\citet commands were used incorrectly. This hinders the readability of the text for the broader audience and fixing them would greatly improve the reading experience.\n\n- Some figures are very crowded with many details, creating potential confusions in grasping their main message. To exemplify, Figure 1 includes many different details regarding the full LVLM pipeline. Although its quality is very high and I can clearly see that much effort went into constructing it, I believe that pruning it greatly would make it easier for the reader to grasp its main message.\n\n*Finally , although I am leaning towards rejection for the work in its current form, I would like to encourage the authors to clarify any potential misunderstandings I might have had.*\n\n---\n[A] Diao, H., Cui, Y., Li, X., Wang, Y., Lu, H., & Wang, X. (2024). Unveiling encoder-free vision-language models. Advances in Neural Information Processing Systems, 37, 52545-52567.\n\n[B] Diao, H., Li, X., Cui, Y., Wang, Y., Deng, H., Pan, T., ... & Wang, X. (2025). Evev2: Improved baselines for encoder-free vision-language models. arXiv preprint arXiv:2502.06788.\n\n[C] Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D., ... & Zhai, X. (2024). Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726.\n\n[D] Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., ... & Dai, J. (2024). Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 24185-24198).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)