Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
Authors: Junlin Han
Shengbang Tong
David Fan
Yufan Ren
Koustuv Sinha
Philip Torr
Filippos Kokkinos
Venue: ICLR 2026 Oral (2026)
Keywords: LLM pre-training
MLLMs
multi-modality
Primary area: foundation or frontier models, including LLMs
TL;DR: Explore and understand the visual priors within LLMs and thus build better MLLMs.

Abstract: Large Language Models (LLMs), despite being trained on text alone, surprisingly develop rich visual priors. These priors allow latent visual capabilities to be unlocked for vision tasks with a relatively small amount of multimodal data, and to perform symbolic visual generation tasks without ever having seen an image. Through systematic analysis, we reveal that visual priors—the implicit, emergent knowledge about the visual world acquired during language pre-training—are composed of separable perception and reasoning priors with unique scaling trends and origins. We show that an LLM's latent visual reasoning ability is predominantly developed by pre-training on reasoning-centric data (\eg, code, math, academia) and scales progressively. This reasoning prior acquired from language pre-training is transferable and universally applicable to visual reasoning. In contrast, the perception prior emerges more diffusely from broad corpora, and perception ability is more sensitive to the vision encoder and visual instruction tuning data. In parallel, text describing the visual world proves crucial, though its performance impact saturates rapidly. Leveraging these insights, we propose a data-centric recipe for pre-training vision-aware LLMs and verify it in 1T token scale pre-training.  Our findings are grounded in over 100 controlled experiments consuming 500,000 GPU-hours, spanning the full MLLM construction pipeline—from LLM pre-training to visual alignment and supervised multimodal fine-tuning—across five model scales, a wide range of data categories and mixtures, and multiple adaptation setups. Along with our main findings, we also propose and investigate several hypotheses, and introduce a Multi-Level Existence Bench (MLE-Bench) to facilitate future research. Together, this work provides a new way of deliberately cultivating visual priors from language pre-training, paving the way for the next generation of multimodal LLMs.

We recommend a visit to our project page (https://junlinhan.github.io/projects/lsbs/) for an interactive reading.

Published as a conference paper at ICLR 2026
LEARNING TO SEE BEFORE SEEING: DEMYSTIFY- ING LLM VISUAL PRIORS FROM LANGUAGE PRE- TRAINING
Junlin Han Shengbang Tong', David Fan', Yufan Ren', Koustuv Sinha',
Philip Torr², Filippos Kokkinos
Meta Superintelligence Labs UUniversity of Oxford
Junl inhann mattanmmmetaccom com
ABSTRACT
Large Language Models (LLMs), despite being trained on text alone, surprisingly develop rich visual priors. These priors allow latent visual capabilities to be unlocked for vision tasks with a relatively small amount of multimodal data, andto perform symbolic visual generation tasks without ever having seen image. Through systematic analysis, we reveal that visual priors- the implicit, emergent knowledge about the visual world acquired during language pre-training are composed of separable perception and reasoning priors with unique scaling trends and origins. We show that LLM's latent visual reasoning ability is predominantly developed by pre- rre-training on reasoning centrri data (e.g., code, math, academia) and scales progressively. This reasoning prior acquired from language pre-tree pre-training is transferable and universally applicable to visual reasoning. In contrast, the perception prior emerges more diffusely from broad corpora, and perception ability is more sensitive to the vision encoder and visual instruction tuning data. In parallel, text describing the visual world proves crucial, though its performance impact saturates rapidly. Leveraging these insights, we propose data-centric recipe for pre-training vision-awarc LLMs and verify it in IT token scale pre- training. Our findings grounded in over 100 controlled experiments consuming 500,000 GPU-hours, spanning the full MILM construction pipeline-from LLM pre-training visual alignment and supervised multimodal fine-tuning-acrenn five model scales, a wide range of data categories and mixtures, and multiple adaptation setups. Along with our main findings, we also propose and investigate several hypotheses, and introduce a Multi-Level Existence Bench (MLE-Bench) to facilitate future research. Together, this work provides a new way of deliberately cultivating visual priors from language pre-training, paving the way for the next generation of multimodal LLMs.
We recommend a visit to our pro ject page for an interactive reading.
INTRODUCTION
compelling phenomenon has emerged at the forefront of Al research: Large Language Models (LLMs), despite being trained exclusively on vast corpora text, appear to develop profound priors about the visual world. This latent capability is paradoxical, suggesting that the statistical patterns within language might be rich enough to encode fundamental principles of vision, from object properties to spatial relationships, without ever observing single image. This emergent visual prior presents in several surprising and powerful ways:
(1) Programmatic visual knowledge: LLMs possess a rich visual knowledge, enabling them to gen- erate executable code that renders complex 2D and 3D scenes, from objects to spatial layouts (Sharma al., 2024; Ge et al., 2025; Ashutosh al., 2025). This demonstrates a grasp of visual concepts without ever seeing single image. The resulting synthetic data is of sufficient quality to pre-train standard vision models for successful generalization to real-world images (Sharma et al., 2024).
Project lead

---

Published as a conference paper at ICLR 2026
(2) Data-efficient visual adaptation: LLMs are highly efficient for visual adaptation. With vision encoder, high-level reasoning emerges from instruction tuning on a small scale of image-text pairs, bypassing the need for massive multimodal pretraining (Alayrac al., 2022; Liu et al., 2023a; Li et al., 2023; Grattafiori et al., 2024; Tong et al., 2024a; Bai et al., 2025b). This data-efficient instruction tuning extends to unified models, where visual generation unlocked with minimal data (Tong et al., 2024b). Furthermore, this efficiency enables adaptation to low-level visual tasks using vision-only data (Zheng et al., 2024; Du al., 2025), proving that an LLM's reasoning framework can function independently of cross-modal alignment.
(3) LLMs as strong vision encoders:
The learned representations of LLMs can directly benefit pure vision tasks without language (Kumar et al., 2024; Pang et al., 2024; Lai et al., 2024; Bai et al., 2025a). When repurposed as visual encoders, the transformer layers of LLMs offer competitive performance on image classification, segmentation, and video understanding, even surpassing vision- specific backbones (Pang et al., 2024). These findings suggest that the hierarchical abstraction and long-range dependency modeling intrinsic to LLMs are not modality-specific, but rather capture general-purpose computational motifs that are well-suited to processing visual signals. This is also shown in neuron-level studies, which have identified multimodal neurons within LLMs that respond to the same abstract concept regardless of whether it is presented through text or vision (Schwettmann al., 2023; Pan et al., 2023; Verma al., 2024).
Collectively, these phenomena are not isolated curiosities; they point toward a deeper principle of representation learning. They lend strong empirical support to the Platonic Representation Hypothesis (Huh et al., 2024b; Jha et al., 2025), which posits that as models scale across diverse data and tasks, their latent appresentationsswheeee underlying statistical model of reality. In this view, text and images are different "projections" or "shadows" of the world, and a powerful enough model can learn the structure of the world itself from any single projection. The visual priors in LLMs, therefore, may be a direct consequence of them recovering this unified internal world model from text alone.
trained on text or saaages-converge toward shared,
These observations motivate a systematic investigation into the visual priors that LLMs acquire from language pre-training. Rather than a hand-crafted bias or Bayesian prior distribution, we frame visual priors as implicit knowledge or prior vision capabilities encoded in LLMs, whose primary effect is to grant both enhanced capability for vision tasks and greater ease of transfer to vision. We seek to determine their origins, dissect whether they form uniform block of knowledge or are composed of distinct, separable abilities, and explore how they can be leveraged to build more capable MILMMs.
Our work presents the first systematic investigation into the nature and origins of visual priors in the pre-training of LLMs and shows three key contributions: (1): Structure of visual priors. We establish that visual priors can be decomposed into perceptual and reasoning components. (2):
Source of visual priors. We identify that the model's latent visual reasoning is predominantly cultivated by and scales progressively with reasoning-centric data, whereas its perception ability emerges more diffusely from broad, diverse data. (3): Vision-aware language pre-training, We propose a pre-training data-mixing strategy that strategically balances ceasoning-centre and visually descriptive text deliberately cultivate powerful visual priors for training L.LSs that can result in stronger multimodal performance.
2 PROBLEM FORMULATION
In this section, we very briefly introduce our default training and evaluation settings. More details are available in Appendix B.
Training protocol.
We pre-train Alama-3-styyle language models (340M to 13B parameters) with
default 3B parameter model trained on 30B tokens. Training data is composed of 16 sources. We then adapt these LLMs into MLLMs using a two-stage process: 1) visual alignment with an MLP projector, and 2) supervised fine-tuning on a curated mix vision-language and language-only instruction data.
Evaluation protocol. Our evaluation is twofold. For LLMs, measure perplexity for language modeling and average accuracy across eight benchmarks. For MLLMs, we follow Cambrian-1 and establish a diverse evaluation suite comprising 16 public benchmarks, we assess performance
2

---

Published as a conference paper at ICLR 2026
General
Knowledge
Gloud
When lightning flash of current and time occurs, the electric field strength between cloud and Earth reduced by how much?
dirty ground made of plastic?
OCR Chart
Vislon-Centric
?
Healthy choice
Rot Ma
What brand name is visible in the white text inside the green square?
Which ane picture follows the same pattern established by the previous pictures?
Figure 1: VQA evaluation question examples for each category.
AVG VQA
General VQA
Knowledge VQA
OCR Chart VQA
Vision
Vision-Gentric Centric VQA
32
Data (1)
Data
Dathh 53)
Eath 13)
Dats (10
Model dizes
300M 1.00
38 70
330
Figure Impact of model and data sizes. The plots illustrate the performance of MLLMs, built upon LLMs of five different sizes (340M to 13B parameters), pre-training data (OB to 100B tokens). The general trend shows that performance improves with both increasing model size and data volume, but the scaling behavior differs across task categories.
function of the amount of web crawl
16 benchmarks grouped into four categories: General, Knowledge, OCR & Chart VQA, and Vision-Centric. Figure presents example questions representing each category.
Additionally, measure the representational alignment between language and vision modalities using a kernel-based similarity metric (Huh et al., 2024a).
3 DEMYSTIFYING LLM VISUAL PRIORS: STUDIES AND FINDINGS
This section presents our main results and findings. We first conduct a series of controlled experiments to systematically deconstruct the origins of LLM visual priors. Each subsection details its specific experimental setup or analytical approach, followed by the results and key findings.
3.1 IMPACT OF MODEL AND DATA SIZES.
Finding VQA performance scales positively with model and data size. However, this scaling is not uniform across all visual abilities.

---

Published as a conference paper at ICLR 2026
AVG VQA
General VOA
Knowledge VOA
32.1
24-
Rccuracy
16.7
31.3
20.2
Vision-Certric
303
30
con ar
Figure 3: Impact of pre-training data sources. The bar charts illustrate the downstream VQA performance of MILMs built upon 3B parameter LLM, where each LLM was pre-trained on 30B tokens from a single, specific data source. The plots show that performance varies significantly depending on the pre-training sources.
We begin our analysis by investigating the fundamental impact of scale. To study how model size and pre-training data volume influence downstream multimodal capabilities, we perform set of experiments to pre-train five L.MMs of varying sizes (340M, 1B, 3B, 7B, and 13B parameters). Each model size was trained on eight different scales of data, ranging from OB 100B tokens. The training dataset is web crawl for all experiments.
As illustrated in Figure 2, both model sizes and pre-training data sizes generally lead stronger downstream multimodal performance. This holds true for the overall average VQA. However, a closer look at the different VQA categories reveals significant nuances. Performance on General VQA and Knowledge VQA demonstrates a similar scaling trend, consistently improving with both model and data size. In sharp contrast, OCR & Chart VQA is far more sensitive to model size than data volume; the performance gap between models is significantly wider. Meanwhile, Vision-Centric VQA also presents a unique pattern where the largest models benefit disproportionately from more data, while smaller models plateau much earlier. These divergent scaling patterns across different abilities demonstrate different visual abilities do not scale uniformly, but instead possess different properties that govern how they benefit from increased model and data size.
3.2 IMPACT OF PRE-TRAINING DATA SOURCES.
Finding 2:
Specific categories of language pre-training data can enhance certain visual capabilities in the resulting MIL.M.
Having characterized the effects of data and model scales, we now transition our analysis to the composition of the data itself. To investigate the role of different pre-training sources, we fix the model size to 3B parameters and the total training data volume to 30B tokens. We then pre-train 16 distinct models, each trained exclusively pre-training sources (e.g., academia, biology, code, etc.). This setup allows us to attribute performance variations directly to the specific data source used for pre-training.
data from one of the 16 sources outlined in our
As illustrated in Figure 3, the results reveal a significant variance in downstream multimodal perfor- mance depending the pre-training data source. This divergence suggests that different categories of text data contribute to distinct and non-uniform visual priors. Notably, strong performance on Vision-Centric VQA tasks is highly correlated with two types of data: reasoning-centrie (e.g., code,
4

---

Published as a conference paper at ICLR 2026
AVC VQA (Reasoning)
Visian Centric VQA (Reasoning)
AVG VQA (Visual)
Vision Centric VQA (Miual)
21.
32.5
44
22.5
33.5
323
35.0
75%
75%
Proportional 150
Prosortors
Proportiona (5)
002e
SCancc
opomal
viuua
Vinuel
Figure 4: Impact of reasoning-centric and visual data categories and proportions. The plots illustrate how varying the proportion of specific data categories in the pre-training mix affects downstream VQA performance.
hathemat ics, academia) a) and corpora rich in visual world descriptions (e.g., arts, food). The top-performing models in Vision-Centric VQA, all scoring above 42%, are trained on these specific sources. This finding motivates a more granular analysis in the next section.
3.3 IMPACT OF REASONING AND VISUAL DATA CATEGORIES AND PROPORTIONS.
Finding small amount of data about the visual world is crucial, but its contribution saturates quickly; in contrast, increasing the proportion of reasoning data in the pre-training mix progressively enhances visual abilities, with gains observed up to 75% ratio.
Findings in the previous section show that reasoning-centric categories and categories related to the visual world were the most potent drivers of downstream visual capabilities. To dissect this phenomenon further, we focus our next set experiments specifically on these two domains.
The reasoning-centrie data science reasoning, and reasoning combination category, which aggregates the three aforementioned categories. Concurrently, define four categories for data related the visual world: visual concept: Text naming visual entities like objects, people, places, and scenes. visual attribute: Descriptions of visual properties such as color, shape, texture, and style. visual relationship: Language detailing spatial arrangements or part-whole connections. visual combination! io A combination of all three visual categories.
partitioned into code reasoning, math reasoning,
We begin by creating data pool of approximately 300B tokens. comprising all sources used in Section 3.2. To partition this corpus into these categories, we employ 32-B LLM (Yang al., 2025a) to classify the text into finer- finer-grained visual world and reasoning categories, Detailed classification settings and results are presented in the Appendix
With this fine-level categorization, we conduct series of controlled mixing experiments to study how varying the proportion (mixing ratio) of these data types affects the final MLLM's performance. For each category, we train five separate models, systematically varying its proportion in the data mixture 0%. 1,, 50%. 75%, and 100%. The remainder of the data for each run is drawn from proportional mix of all other data within the 300B token pool, ensuring the total training volume is held constant 30B tokens.
As shown in Figure 4, the results reveal critical divergence in how visual world and reasoning data categories contribute to visual priors. The impact of reasoning-centrie data is profound and progressive, with performance scaling steadily up to 75% proportion. The contribution from data explicitly describing the visual world saturates quickly; small initial amount seems to be crucial, but further increases yield diminishing returns.
5

---

Published as a conference paper at ICLR 2026
3.4 DERIVING A DATA MIXTURE FOR MORE VISION-AWARE LLMS.
Finding 4: Maximizing MILM VQA performance is best achieved by pre-training on data mixture heavily skewed towards reasoning-centrr content but with necessary vision world knowledge. The balance point between language and vision proficiency is reached via calibrated data mixture between pagguage-favorabie and vision-favorable.
Building on these findings, our objective is to de- rive a single, practical data mixture that not only excels on language tasks but also serves powerful foundation for MLLMs. Our approach proceeds in three stages. First, we determine blend using our 300B token pool to establish a target. For the subsequent, more practical stages of our analysis, we then narrow our focus six primary data categories: web-crawl, encyclopedia, academia. literature, math, and code. Within this practical set of sources, then identify language-favorable mixture (second), and nally, derive balanced mixture by interpolating between these two optima (third), Since our goal also involves downstream VQA performance,
Data Ratio
Avg VQA
Data Ratio
Avg VQA
rea
vis
rea
vis
5
30.7
5
30.9
50
31.3
31.7
15
31.8
15
32.2
31.9
5
32.0
10
32.4
10
32.2
15
32.7
15
32.5
60
65
20
32.5
20
32.1
25
32,4
31,9
30
31.6
30
31,4
31.9
5
31.6
70
10
32.3
75
10
31.5
15
32.6
15
32.4
we adapt the most direct approach of using grid search over mixture ratios.
5
31,5
5
31.2
80
10
32.4
85
10
31.6
32.2
31.8
mixture. We first aim to iden- tify data mixture that excels at visual tasks. To do conduct grid search over the propor- tions of reasoning- and visual-world data drawn from our 300B token pool. Specifically, we perform a grid search across 24 data blends constructed by sampling from a space where the reasoning combinat ion ranges from 50% 85% and the visual combinat ranges from 5% to 30%, following the conclusions drawn from Section 3.3. The comprehensive results of this search are presented Table 1.
Table 1: Grid Search for vision-favorable data mixture. Results from pre-training a 3B parameter LLM on 30 distinct data blends, each totaling 30B tokens. The table explores how varying the propor- tions of reasoning centric (rea) and visual-world (vis) data affects various capabilities.
From this search, we find that the best-performing models for downstream MLLM tasks emerge from mixture containing approximately 60% reasoning and 15% visual content. Results show powerful visual foundation is not built by simply maximizing exposure to visual descriptions, but by establishing a strong reasoning faculty, which is then grounded by a smaller amount visual world knowledge. This experiment is performed on our 300B token pool, provides target ratio for maximizing VQA performance. the next part, we proceed to more practical experiment at the data source level, guided by this result.
lanoubeeefavorable mixture. We begin by establishing a language-favorable mixture that achieves the best performance our language task suite. Guided by recent literature (Shukor et al., 2025a; Ge et al., 2024; Ye et al., 2024) and empirical testing over 10 experiments, we identify this as a mix of 50% web- web-crawl. 2.5% encyclopedia, 2.5% academia a, 20% literat ure, 5% math. and 20% code. This blend, designated as mix0 in Table 2, serves as our baseline for strong language proficiency, achieving the highest text accuracy (53.0%) and the best perplexity (13.46).
Balanced mixture. To reconcile these two objectives, we seek single, balanced mixture that offers strong performance across both modalities. We achieve this by performing a series of interpolation experiments, detailed as mix0 through mix10. We shift the data composition from our language- favorable baseline (mix0) towards an endpoint representing the rision-favoral blend (approximated by mix9 and mix 10). To ensure stabilized results, each model in this series trained for 50B tokens.

---

Published as a conference paper at ICLR 2026
Recipe
vab-cravl aroyclopedia scadenic
litaracure
rassoning ViDu0l
ppl
min0
32.4
mix1
48.3
17.0
38
22.3
36.2
20.6
52.8
13.48
32.4
mix
46.7
3.3
14.0
6.1
25.0
39.6
19.4
52.6
13:51
32.6
min1
45.0
5.2
1.8
11.0
1,5
273
41.6
18.2
52.5
13.56
32.
9
mind
43.3
6.1
42
80
8,3
300
45.7
17.1
524
13,62
32.7
10
mis5
41.7
7.1
45
5.0
9.2
32.5
48.9
16.0
$2.6
13:57
330
6
mixs
40.0
3.0
50
2.0
10.0
35.0
52,0
14,8
52.7
13.52
33.3
mix7
36.5
7.0
7.5
2.0
11.5
35.5
55.5
14,4
52.5
13.56
35.1
mixs
33.0
6.5
9.5
2.0
12.0
37.0
57.2
14,0
52.7
13.52
33.2
mix9
29.5
6.1
11.5
2.0
12.5
36.5
990
13.6
52.3
13.71
33.2
26.0
12.5
2.0
13.0
41.0
61.3
13.3
S2.1
13.8
33.
11
Table Deriving a data mixture for more vision-aware LLMs. This table details a series of 11 data mixtures, from mix0 (languaee-favorable blend) to mix (approximating the vision-favorable blend), all trained on a 3B- 18-parameter LLM with 50B tokens. The results highlight a trade-off, with mix6 emerging the most balanced mixture, achieving top-ranked overall performance by improving visual capabilities without a significant drop in language proficiency.
The performance metries in Table 2 reveal the expected trade-off: the mixture becomes more reasoning-centric, vision accuracy (v-acc) generally improve, while language proficiency (I-acc and ppl) shows slight decline. Our analysis identifies mix6 the balanced mixture, achieving the highest overall rank. Mixtures in its vicinity (e.g., mix5, mix7, mix8) also achieve high rankings. This demonstrates that carefully calibrated data mixture can cultivate powerful visual priors without substantially compromising core language abilities.
3.5 THE STRUCTURE AND ORIGIN OF LEARNED VISUAL PRIORS.
The learned visual prior is not a single entity but decomposes into at least perception prior and reasoning prior with different origins.
Finding 5:
We now synthesize our previous results to investigate the internal structure the visual prior. We conceptualize the visual prior as collection of distinct abilities, each measured by one of our four VQA categories.
Internal structure of visual priors. We aggregate the performance data across all 105 3B models from our previous experiments, encompassing variations in data sources, mixing ratios, and training scales. We then compute the Spearman correlation matrix across the four VQA performance categories to identify which abilities scale together and which diverge.
The results in Figure 5 suggest potential inter- nal structure within the visual prior, hinting at separation into at least two distinct types of abil- ities. We observe moderate correlation (0.37) between General and OCR performance. This connection
Correlation matrix VOA performances
General
0.03
0.11
Knowled ledge-
6:31
point towards perception prior, as success in both categories relies heavily on the model's perceptual acuity- the ability to accurately process raw visual input-rather than complex, multi-step reasoning.
0.08
0.67
Mision
0.30
0.33
0.07
In contrast, we find another moderate correla- tion (0.33) between the Knowledge and Vision- Centric tasks. This link appears to emerge be- cause both categories often require abstract infer- ence that goes beyond simple perception. For in- stance, the Knowledge category demands multi- step reasoning solve complex scientific or mathematical problems, while Vision-Centric object counting, and correspondence matching,
Knowtedge
Vidao
Figure 5: Correlation matrix for VQA per- formances. The matrix reveals two loosely- coupled skill clusters: one axis for percep- tion (General/OCR) and another for reasoning KKnowledgeVvision-Ceee
tasks include challenges like visual IQ puzzles, necessitate a blend of perception and reasoning, often with a heavier reliance on the latter. The correlation matrix also reveals very weak, even slightly negative, correlations between these two
7

---

Published as a conference paper at ICLR 2026
AVG VQA
General VQA
Knowledge VOA
OGR & Chart VOA
Vision-Centric VQA
44
20
42-
40:
336%
50% Propersions (59)
185%
Propattionn.c (%)
Proportions (1)
Propertions (8))
Piopertonne (8)
Vision chcodors
MetaCLiP
DINO
MAE
Figure 6: Universality of the learned visual priors. The plots show the VQA performance of MLLMs built using three distinct vision encoders based on the proportion of reasoning-centric data used in the LLM's pre-training mix. Despite differences in their absolute performance, all three configurations show consistent improvement pre-training proportion increases, similar to trends observed before.
crasoning-heay tasks the LLM's 's reasoning
groups pprception-heayy vs. reasoning-heavy) ! This lack of a strong positive correlation raises the possibility that these are largely independent abilities, potentially stemming from loosely-coupled priors within the LLM's representation. Our observations on the separability of these visual priors in MLLMs align with and extend the findings of recent research Chen et al. (2025), which identified similar dissociation through parameter merging.
Different origins of priors. The statistical independence of these two priors implies they cultivated through different mechanisms. As our analysis in Section 3.2 and Section 3.3 demonstrated, the reasoning prior is from reasoning-centric data and can be predictably enhanced by increasing the proportion of reasoning-centrie data.
In contrast, the origins of the perception prior appear more diffuse. signal comes from single- source experiments (Section 3.2), where web- ccbacrral data yields the best performance on General and OCR tasks. However, web-craw is an extremely general category, and no other, more specific data category consistently boosts perceptual abilities. This mixed effect suggests the perception prior may be general byproduct of large-scaly scale language modeling, emerging from the sheer diversity of language rather than specific category.
To further investigate this perception prior and characterize its properties more directly, we introduce multi-level existence benchmark (MLE-Bench) designed to assess pure perception abilities across multiple levels. In Section A. we use this benchmark test our hypothesis that the perception prior scale-dependent and is most pronounced for small and medium-sized objects. Further details on the benchmark's ' construction are presented Appendix I. We also present an evaluation of some MLLMs MLE-Bende revealing that they exhibit varying levels of ability in perceiving objects of different sizes. Detailed results and analysis are available in Appendix
3.6 DECONSTRUCTING MULTIMODAL ABILITIES: VISION OR LANGUAGE.
Finding 6: Visual reasoning is primarily shaped by reasoning prior acquired from language pre-training: perception is more dependent post-training (visual instruction tuning).
Here, we conduct further analysis to first verify the universality of learned visual priors and then deconstruct the source of different multimodal abilities, distinguishing between those inherited more from the LLM and those acquired more from the visual instruction tuning.
Universality of the learned visual priors. To test the general influence of the visual prior, we apply two more vision encoders (DINOv2-G (Oquab et al., 2023) and MAE-H (He et al., 2022)) other than our default MetaCLIP-B/16. We pair these with LLMs pre-trained on varying proportions of our reasoning combi ion data category, from 0% to 100%.
TThis categorization into perception-heavy and reasoning-heavy tasks is conceptual simplification intended facilitate analysis. The boundary between perception and reasoning is always clear.
8

---

Published as a conference paper at ICLR 2026
Relative performance after data removal
100
20.79
$0.59
Remaining perforance (%6
90
20 62
26
95.23
$0.34
1.0% Baseline
Porco ption romaining
Perce (0%s romaining datal
Reasoning (50% remaining data)
Reasoning (0% ning
JVG
General
Knowledge
OCN Chart
Vislon-Centiic
Figure 7: Step-wise effects of perception and reasoning visual-instructon tuning data removal. The chart shows the remaining performance (%) on Avg VQA and per-category VQA (x-axis) relative to 100% baseline. The bars show performance after ablating perception reasoning instruction data in stages (removing 50% and then 100% of the data).
As illustrated in Figure 6, the results reveal a dual-faceted pattern. Firstly, they confirm the universality of the reasoning prior. For reasoning-heavy tasks, all three vision encoder configurations exhibit nearly identical, strong upward trend in performance as the proportion of reasoning data in the LLM's pre-training increases. This demonstrates that the visual reasoning prior cultivated in the LLM is foundational, modality-agnostic prior that benefits the multimodal system regardless of the specific vision encoder used. In contrast, the perception prior lacks this universality. The performance trends for perception-oriented tasks are more inconsistent across the different vision encoders.
Source of abilities, from visual priors or visual instruction tuning. Second, we conduct targeted studies to determine whether perception and reasoning skills originate primarily from the LLM's visual priors or the subsequent visual instruction tuning stage. We use an MLLM to classify our Cambrian-7M dataset that contains 5M text-image pairs into these categories, resulting in 1.8M perception and 0.6M reasoning data, and the remaining 2.6M data as others. Further classification details provided in the Appendix G.
We partition our instruction-tuning data into perception, reasoning, and other categories and trained
five tuning configurations that ablate perception and reasoning data stages (100% - 50% - 0%)
while leaving other data unchanged. The results, presented in Figure 7, show two observations: (1) reducing prrceptiio-targerte tuning produces the largest performance drops on perception-heant benchmarks and modest drops on reasoning tasks; (2) removing massoning-targeter tuning causes only small incremental drops on perception tasks and modest drops on reasoning tasks.
Together, results in this section show two mechanisms. First, the LLM encodes robust, transferable visual reasoning prior primarily via language pre-training; this prior benefits reasoning-centern VQA across different vision encoders. Second, perception performance depends more vision-encoder characteristics and on subsequent supervised visual instruction tuning.
In addition our main findings, we investigate the universality of reasoning priors acquired during pre- mre-training. Our experiments show that this reasoning ability is highly transferable and modality- agnostic, detailed in Appendix A.2. Furthermore, in Appendix A.3, study whether language data structures drive representational alignment with vision, finding that they can indeed partially drive this alignment.

---

Published as a conference paper at ICLR 2026
4 SCALING UP AND TRAINING A VISION-AWARE LLM
4.1 SETTINGS AND MODELS
Building upon our findings, we scale up our approach to validate our findings and develop vision- aware LLM on larger-scale. The goal is to test whether the principles identified in our controlled, smaller-scale studies hold true when applied to larger training runs. To this end, we pre-train two 7B parameter LLMs, each on IT tokens, based on the two data mixtures identified previously: Language-favorable model: Following the mix0 mixture, which is the best-perforning blend for pure language tasks. Balanced model: Based on the mix6 recipe, our proposed balanced mixture is designed to deliberately cultivate strong visual priors without compromising language proficiency.
Language
Vision
Model
ppl
avg acc
General
Knowledge
OCR& QA Vision-Centric
Overall
Sunguage-Favoralle Balanced
8.72
0.647
46.92
28.35
21.49
46.31
37.32
7.49
0.655
49.59
29.02
23.63
46.59
38.64
Table 3: Performance comparisons of the Language-Favorea Favorable and Balanced models across
both language and vision- laiion-language benchmarks. The table summarizes key language metrics (perplexity and accuracy) and provides average scores for a suite of vision tasks.
4.2 RESULTS
As shown in Table 3, the Balanced model, pre-trained with balanced recipe, exhibits competitive language proficiency. Notably, it achieves lower (better) average perplexity of 7.49 compared the favorale model's 8.72, while also maintaining a slightly higher average accuracy (0.655 vs. 0.647). An interesting dynamic observed during pre-training was that the Balanced model's language performance initially lagged behind the aanguage-favorabl model, beginning to surpass after approximately 600B tokens. This may suggest that when the pre-training token volume is sufficiently large, the benefits from reasoning-related tokens can be more effectively unleashed when grounded a substantial amount of world knowledge.
On VQA benchmarks detailed, the Balanced model consistently outperforms the Language-Favorable model in most of the benchmarks, achieving a higher overall VQA average (38.64 68..64 vs. 37.32). This confirms that the deliberate pre-training successfully imbues the L.M with stronger visual priors in a larger scale.
data mixture rich in reasoning and visual world text
We also introduce Blind Visual Instruction Tuning, trick that serves the MILM community both a practical tool for visual adaptation and a probe for revealing how models "hack" visual tasks with language (see Appendix K). Together with this trick, find that many MILMs, even state-of-the-an ones, fail to recognize the absence of an image input. This leads to hallucinations in visual question answering scenarios without visual context. A study of this behavior is presented in Appendix L..
CONCLUSION
This work has undertaken systematic deconstruction of the visual priors that LLMs acquire from text-only pre-training. Through series of controlled experiments manipulating data composition, we moved beyond observing the phenomenon of vision priors to interrogating its fundamental drivers. Our investigation provides data-centrie roadmap for developing multimodal systems, shifting the paradigm from serendipitous emergence to the deliberate cultivation visual capabilities.
Looking forward, we hope this research encourages paradigm where LLM development is more considerate of vision and multimodality, prompting the cultivation of visual priors from the earliest stages of pre- training.. We also hope it inspires deeper investigation into the fundamental correlations between cross- modal representations, contributing to more unified understanding how knowledge is structured across modalities.
10

---

Published as a conference paper at ICLR 2026
THE USE OF LARGE LANGUAGE MODELS (LLMs)
LLMs were used to aid and polish paper writing.
ETHICS STATEMENT
This research was conducted with commitment to responsible practices. The ethical implications of this research are primarily related to the societal biases present in large-scale text corpora. A more detailed discussion of these considerations and the broader impact of our work is provided in the Broader Impact section in Appendix E.
REPRODUCIBILITY STATEMENT
To support the reproducibility of our findings, we will publicly release our newly proposed Multi- Level Existence Bench (MLE-Bench). The detailed methodology for classifying our pre-training data described in Appendix G and Appendix F. Furthermore, details regarding training protocols and evaluation procedures are provided in Appendix B.
REFERENCES
AI@ Meta. Llama model card. 2024. URL URL.nttp:://githbb..om/// Blob/main/MODEL. CARD md.
/
Ban--Batistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. In NeurlPS, 2022.
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training, arXiv preprint arXiv:2312.02206, 2023.
Zeyuan Allen-Zhi Ieml 2024 tutorial: Physics of language models. Project page: https://physics. allen-zhu. com, 2024.
Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. To code, or not to code? exploring impact of code in pre-training. arXiv preprint arXiv:2408. 10914, 2024.
Kumar Ashutosh, Yossi Gandelsman, Xinlei Chen, Ishan Misra, and Rohit Girdhar. Llms can see and hear without any training. arXiv preprint arXiv:2501.18091 2025.
Sören Auer, Dante A. C. Barone, Cassiano Bartz, Eduardo G. Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mourontsev, Dmitrii Pliukhin, Daniil Radyush, Ivan Shilin, Markus Stocker, and Eleni Tsalapati. The seiqa scientific question answering benchmark for scholarly knowledge. Scientific Reports, 13(1):7240, May 2023. ISSN 2045-2322. doi: 10.1038/s41598-002 322-67----
Lichen Bai, Zixuan Xiong, Hai Lin, Guangwei Xu, Xiangjin Xie, Ruijie Guo, Zhanhui Kang, Hai-Tao Zheng, and Hong-Gee Kim. Frozen language models are gradient coherence rectifiers in vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 1817-1825, 2025a.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2rXiv 13923, 2025b.
Tianyi Bai. Ling Yang, Zhen Hao Wong, Jiahui Peng, Xinlin Zhuang, Chi Zhang, Lijun Wu, Jiantao Qiu, Wentao Zhang, Binhang Yuan, et Multi-agent collaborative data selection for efficient 11m pretraining, arXiv preprint 1000:2240.0 2024.

---

Published as a conference paper at ICLR 2026
Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yu- val Noah Harari, Ya-Qin Zhang. Lan Xue, Shai Shalev-Shwarke et al. Managing extreme ai risks amid rapid progress. Science, 384(6698):842-8883 845, 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. In CVPR, 2023.
Mayee F Chen, Michael Y Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher Ré. Aioli: A unified optimization framework for language model data mixing. arXiv preprint arX1v:2411.05735, 2024.
Shiqi Chen, Jinghan Zhang, Tongyao Zhu, Wei Liu, Siyang Gao, Miao Xiong, Manling Li, and Junxian He. Bring reason to vision: Understanding perception and reasoning through model merging. arXiv preprint arXiv:2005.05464, 2025.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating Iims by human preference. arXiv preprint arXiv:2403.041333 2024.
Christopher Clark, Kenton Lee, Ming-Wat Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint UrXiiv/905. 10044 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.088 2018.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie. Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 146633 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free free vision- langunlanguage models. arXiv preprint (tt::2400.118322, 2024.
Haiwen Diao, Xiaotong Li, Yufeng Cui, Yueze Wang, Haoge Deng, Ting Pan, Wenxuan Wang, Huchuan Lu, and Xinlong Wang. Evev2: Improved baselines for encoder-free vision-language models. arXiv preprint arXiv:2502.06788, 2025.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Junhao Du, Chuqin Zhou, Ning Cao, Gang Chen, Yunuo Chen, Zhengxue Cheng, Li Song, Guo Lu, and Wenjun Zhang. Large language model for lossless image compression with visual prompts. arXiv preprint 2502. 2025.
David Fan, Shengbang Tong. Jiachen Zhu, Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representation learning. arXiv preprint arXiv:2504.01017 2025.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang. Xiawu Zhengg, et al. Mme: a comprchensive evaluation benchmark for multimodal large language models. coir abs/2306. 13394 (2023), 2023.
Xingyu Fu, Yushi Hu, Bangzheng Li. Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:2404 /2390, 2024.
12

---

Published as a conference paper at ICLR 2026
Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. Data mixing made efficient: A bivariate scaling law for language model pretraining. arXiv e-prints, pp. arXiv-2405, 2024.
Jiaxin Ge, Zora Zhiruo Wang, Xuhui Zhou, Yi-Hao Peng, Sanjay Subramanian, Qinyue Tan, Maarten Sap, Alane Suhr, Daniel Fried, Graham Neubig, et al. Autopresent: Designing structured visuals from scratch. arXiv preprint arXiv:2501.000 100912, 2025.
Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08441 2023.
Google. Gemini, 2023.
URL https://bblog.goolle/eteeee
google-gemini-an
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXiv:2407.21788 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint ar arXiv:23/2.007/22 2023.
Junlin Han, Huangying Zhan, Jie Hong, Pengfei Fang. Hongdong Li, Lars Petersson, and Ian Reid. What images are more memorable to machines? arXiv preprint arXiv:221 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
William Held, Bhargavi Paranjape, Punit Singh Koura, Mike Lewis, Frank Zhang, and Todor Mihaylov. Optimizing pretraining data mixtures with llm-estimated utility. arXiv preprint arXiv:2501.11747, 2025.
Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams. Language Resources and Evaluation, 35:661-685 2021.
Drew A. Hudson and Christopher D. Manning, Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. In ICML, 2024a.
Minyoung Huh, Brian Cheung, Tongzhou Wang. and Phillip Isola. Position: The platonic representa- tion hypothesis. Forty-first International Conference on Machine Learning, 2024b.
Rishi Jha, Collin Zhang, Vitaly Shmatikov, and John X Morris. Harnessing the universal geometry of embeddings. arXiv preprint arXiv:2505. /2540, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wtX.i:20011.8861,, 2020.
Gurucharan Marthi Krishna Kumar, Aman Chadha, Janine Mendola, and Amir Shmuel. Medvisionl- lama: Leveraging pre-trained large language model layers to enhance medical image segmentation. arXiv preprint arXi::2410.02458 2024.
Zhixin Lai, Jing Wu, Suiyao Chen, Yucheng Zhou, and Naira Hovakimyan. Residual-based language models are free boosters for biomedical imaging tasks. In Proceedings the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 5086-5096, 2024.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02244 2024.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICMI. 2023.
13

---

Published as a conference paper at ICLR 2026
Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, and Qi Liu. Temporal reasoning transfer from text to video. arXiv preprint arXiv:2410.066 2024.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next; Improved reasoning, ocr, and world knowledge, 2024b. URL https: //1lava-vll github. wtt.bic////o////.. -0---------11111 1eoo-lava---e Lavaaaaaaa.a....aa
Qianchu Liu, Sheng Zhang. Guanghui Qin, Timothy Ossowski, Yu Gu, Ying Jin, Sid Kiblawi, Sam Preston, Mu Wei, Paul Vozila, et al. X-reasoner: Towards generalizable reasoning across modalities and domains. arXiv preprint arXiv:2505. 03981, 2025.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2024c.
Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, al. On the hidden mystery of ocr in large multimodal models. arXiv preprint S.Kiv:2305.0789 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05011, 2017.
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. In Proceedings of the AAAI conference artificial intelligence. volume 36, pp. 7628-7636, 2022a.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2023.
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help llms reasoning? arXiv preprint arXiv:2309. 2023.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022.
Prasanna Mayilvahanan, Thaddäus Wiedemer, Sayak Mallick, Matthias Bethge, and Wieland Brendel. Lims on the line: Data determines loss-to-loss scaling laws. arXiv preprint arXiv:2502.12120, 2025.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.0784/ 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: wet.i18099000000 02789, 2018.
14

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 16:50358-50370 2023.
OpenAl. gpt4o, 2024. URL https: http://aaa..com//eee/aaaalll---/
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et Dinov2: Learning robust visual features without supervision. In TMLR, 2023.
Haowen Pan, Yixin Cao, Xiaozhi Wang, Xun Yang, and Meng Wang. Finding and editing multi-modal neurons in pre- pre-trained transformers. arXiv preprint urXiv:2311.074700 2023.
Ziqi Pang, Ziyang Xie, Yunze Man, and Yu-X lu-Xiong Wang. Frozen transformers language models are effective visual encoder layers. ICLR, 2024.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv: /60iv 06031, 2016.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon IIm: outperforming curated corpora with web data, and web data only, arXiv preprint arXiv:2306.01 116, 2023.
Yiting Qu Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models. In Proceedings of the 2023 ACM SIGSAC conference computer and communications security, pp. 3403-3417, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Jack W Rae, Sebastian Borgcaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint ar 2021.
Abhinav Rastogi, Albert Q Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, et al. Magistral. arXiv preprint arXiv:2506.1091 2025.
Siva Reddy, Danqi Chen, and Christopher Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics,
2019.
Laura Ruis, Maximilian Mozes, Juhan Bac, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. Procedural knowl- edge in pretraining drives reasoning in large language models. arXiv preprint arXiv:2411.12580 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106.
2021.
Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, and Antonio Torralba. Multimodal neurons in pretrained text-only transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2862-2867 2023.
Pratyusha Sharma, Tamar Rott Shaham, Manel Baradad, Stephanie Fu, Adrian Rodriguez- Modriguez-Munoz Shivam Duggal, Phillip Isola, and Antonio Torralba. A vision check-up for language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2024.
15

---

Published as a conference paper at ICLR 2026
Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Anton van den Hengel, and Damien Teney. Transformers pretrained on procedural data contain modular structures for algorithmic reasoning. arXiv preprint arXhv:2505.22::000 22.300, 2025.
Mustafa Shukor, Louis Bethune, Dan Busbridge, David Grangier, Enrico Fini, Alaaeldin El-Nouby, and Pierre Ablin, Scaling laws for optimal data mixtures, 2025a. URL httpt:tttaatii.aaa///aaaaaaaaatt abs/2501 09404.
Mustafa Shukor, Enrico Fini, Victor Guilherme Turrisi da Costa, Matthieu Cord, Joshua Susskind, and Alaaeldin El-Nouby. Scaling laws for native multimodal models. arXiv preprint arXiv:2504007951, 2025b.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension, 2020.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Wikipedia-baset image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp. 2443-2449 2021.
Chenxin Tao, Shiqian Su, Xizhou Zhu, Chenyu Zhang, Zhe Chen, Jiawen Liu, Wenhai Wang, Lewei Lu, Gao Huang. Yu Qiao, al. Hovle: Unleashing the power of monolithic vision-language models with holistic vision-language embedding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14559-14569, 2025.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint 00X1::2405.09818, 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang. Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. Advances in Neural Information Processing Systems, 10--3356,,,, 2024a.
Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412. 14164, 2024b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. LLaMA 2: Open foundation and fine-tuned chat models. 2023.
Gaurav Verma, Minje Choi, Kartik Sharma, Jamelle Watson-Daniels Sejoon Oh, and Srijan Kumar. Cross-modal projection in multimodal Ilms doesn't really project visual attributes to textual space. arXiv preprini 20X 2024.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv 18869, 2024.
Yana Wei, Liang Zhao, Jianjian Sun, Kangheng Lin, Jisheng Yin, Jingcheng Hu, Yinmin Zhang, En Yu, Haoran Lv, Zejia Weng, et al. Open vision reasoner: Transferring linguistic cognitive behavior for visual reasoning. arXiv preprint arXiv:2507.05255, 2025.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410.13848, 2024.
XAI. grok, 2024. URL htt.ittss//...a//blgggggre
16

---

Published as a conference paper at ICLR 2026
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zenlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309. 16671, 2023.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505. 09388, 2025a.
Songlin Yang. Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with harfware-efficien training. arXiv preprint wrXiv:23/2.06633 2023.
Yi Yang, Xiaoxuan He, Hongkun Pan. Xiyan Jiang, Yan Deng. Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cooss- formalization. arXiv preprint arXiv:2503. 10615, 2025b.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. arXiv preprint 1rX1v:2403.16952. 2024.
Zheng-Xin Yong, M Farid Adilazuarda, Jonibek Mansurov. Ruochen Zhang, Niklas Muennighoff, Carsten Eickhoff, Genta Indra Winata, Julia Kreutzer, Stephen Bach, and Alham Fikri Aji. Crosslingual reasoning through test-time scaling. arXiv preprint arXiv:2505.055. 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.0000 2019.
Chi Zhang, Huaping Zhong, Kuan Zhang, Chengliang Chai, Rui Wang, Xinlin Zhuang, Tianyi Bai, Jiantao Qiu, Lei Cao, Ju Fan, et al. Harnessing diversity for important data selection in pretraining large language models. arXiv preprint arXiv:2409. 16986, 2024.
Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang. Lichang Chen, William Yang Wang, and Linda Ruth Petzold. Unveiling the impact of coding data instruction fine-tuning on large language models reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 25949-25957, 2025.
Boyang Zheng, Jinjin Gu, Shijun Li, and Chao Dong. Lm4lv: A frozen large language model for low-level vision tasks. arXiv preprint arXiv:2405. 15734, 2024.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJCV, 2019.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. In NeurlPS, 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 15440499 2025.
17

---

Published as a conference paper at ICLR 2026
Oedel MLCO Acceaay
Aounyy Seart
Acarosy Notknn
Accunicy Large
Figure 8: Performance of MILMs on the Multi-Level Existence Bench (MLE-Bench). The left plot shows the overall accuracy for models pre-trained on 16 different single-source data types. Other plots detail performance on objects of varying relative sizes, from small (0-30% of image pixels) to medium (30-60%) to large (60-100) 100%). The results demonstrate that pre-training on the broad and diverse web-crawl corpus most effective gaining perception prior, with its advantage being particularly pronounced for perceiving smaller objects.
A DISCUSSION AND HYPOTHESES
This section transitions from empirical findings to more speculative exploration of the underlying mechanisms of visual priors. The following subsections present three key hypotheses about the structure of the perception prior, the universal nature of reasoning, and the role of data structure in modal alignment. These hypotheses are not presented as definitive conclusions but as frameworks for interpreting the results and for future research.
A.1 IS THE PERCEPTION PRIOR MULTI-LEVEL? AN EVALUATION USING THE
Hypothesis The perception prior derived from diverse data exhibits scale-dependency, with its benefits being most pronounced for the perception of small and medium-sized objects.
Our previous analyses show that the perception prior is diffuse in origin, emerging most strongly from diverse data. This leads to question about its internal structure: is this prior uniform ability, or does it possess finer-grained characteristies?
To study this question, we introduce the Multi-Level Existence Bench (MLE-Bench), a benchmark designed to probe perception with greater precision. MLE-Bench consists of 4-choice questions about the existence of objects or scenes within an image. We categorize questions based on the target object's s relative size, measured by the percentage of pixels it occupies. In total, MLE-Ban E-Bench comprises 1,861 images, with a distribution of 732 questions for small objects (0-30%), )) 698 for medium objects deconstruct "perception"
(30-60%), and 431 for large objects (60-100%). This structure allows into distinct, scale-dependent components. Further details on the benchmark 's construction are presented in Appendix We also present an evaluation of common MLLMs on MLE-Bench, with detailed results and analysis available in Appendix J.
We evaluate our 16 single-source pre-trained models (from Section 3.2) on MLE-Bench, with the results presented in Figure 8. The 3B LLM model trained on web-crawl remains the top performer overall, confirming that data diversity is key for perception prior. Its advantage most pronounced for small-to-medium objects (0-60% pixel range), where it establishes a clear lead over models trained on other data sources. In contrast, for large objects that dominate the visual scene, this performance gap diminishes significantly.
These results indicate that the perception prior is indeed scale-dependent. A possible explanation is that diverse, unstructured text like web-crawl contains a vast vocabulary describing wide array of entities, including smaller, often overlooked details within a larger scene. This textual richness forces the model to learn representations sensitive to fine- fine-grained visual concepts, capability less critical when identifying large, obvious objects. This finding refines our understanding of the perception prior, revealing that it is not uniform faculty.
18

---

Published as a conference paper at ICLR 2026
Question: A bounding box an annotated rectangle surrounding object. The edges of bounding boxes should touch the outermost pixels of the object that is being labeled.
two bounding boxes image, labeled by B, bounding accurately localizes and encloses fork? Select from the following options.
(A) Box A (B) Box B
Anawer the question with detailed thinking process and explanations.
Answers
The correct answer (B) Box. The image clearly shows the fork resting on the surface of the fork, which is the only object that clearly
The correct Box The bounding box encompasses the outermost pixels of the object being labeled.
The correct (B) Box. The bounding box encloses the fork, as it the only rectangle
answer
The correct answer is (B) Box. The box container that holds fork.
the question The image shows box with the label A enclosing the object.
encompassing the entire object in.
0% code reasoning
25% code reasoning
50% code reasoning
75% code reasoning
100% code reasoning
Figure 9: Qualitative impact of reasoning-centric data on visual reasoning. The figure displays the answers from five prdeel--pre-traineed with 0% 100% code reason ng data-to visual question requiring the application of a specific rule. Answers show a clear improvement in reasoning quality: the model with 0% code reasoning provides a simplistic justification, while the models with 75% and 100% code reasoning produce more detailed reasoning that correctly applies the definition from the prompt.
A.2 Is REASONING UNIVERSAL, CROSS-MODAL SKILL ALREADY ACQUIRED DURING PRE-TRAINING?
Hypothesis 2: The reasoning capabilities LLM acquires from text fundamentally modality-agnostin Language reasoning skills can be directly transferred to solve visual problems.
Our findings suggest hypothesis: the reasoning capabilities an LLM acquires from text are not bound to the linguistic domain. We posit that by pre-training on reasoning-centrie data, model learns abstract, generalizable principles of logic, structure, and compositionality. This foundation largely
allowing the model to apply this faculty to other domains, including vision, since the reasoning process likely occurs within the language domain.
Verity this, we propose an exper- iment that directly probes the qual- ity of the models' visual reasoning processes. For reasoning-focused VQA tasks (Knowledge and Vision- Centric), switch the evaluation from prompting for a direct answer to answering with detailed explana- tions. This will require each model produce detailed explanation of its reasoning. We then use a separate LLM (Yang et al., 2025a) as judge to evaluate the quality of these rea- soning traces based on clear rubric, assessing criteria such as: (1) Logical Soundness: The percentage reason- ing traces that are coherent and reason- able; and (2) Reasoning Depth: The average length of the reasoning trace measured by text count taken to reach the conclusion.
Logical Soundness (96)
Roasoning Depth
Volio
30
10
01%
27%
10%
75%
101%
055
25%
135
10%
Proportions 1%61
Proportions (%)
code Neasoingg
science reasoning
Figure 10: Qualitative impact of crasoning-centree data on visual reasoning tasks. The plot shows how varying data categories
the proportion of different reasoning-centee in the pre-training mix impacts metrics of visual reasoning quality. Results indicate that more reasoning data leads to more coherent and detailed visual reasoning.
19

---

Published as a conference paper at ICLR 2026
The results, presented in Figure 10, strongly support our hypothesis that the reasoning capabiliities an LLM acquires from text are transferable to vision. We observe a clear trend: as the proportion of reasoning centrri data increases, the models generate visual reasoning that is both more logically sound and significantly longer. For instance, increasing the proportion of code reason ng data from 0% 100% boosts Logical Soundness from 4.52% to 9.52% and more than sextuples the Reasoning Depth from 8.31 to 53.25. This demonstrates that the model is applying a general, abstract reasoning framework, learned from text, solve visual problems. The particularly dramatic increase in Reasoning Depth for rode-traineds models may also reflect a stylistic transfer; pre-training code reason which is often structured in long, logically coherent sequences, likely predisposes the model to generate longer, more structured step-by-step explanations.
Figure provides a qualitative example of this phenomenon. showcases how models trained with more code reason i ng data produce increasingly sophisticated and reasonable reasoning for visual task. While the model with 0% code reasoni ng offers a simplistie justification, the model trained on 100% code reasoning provides a detailed, step-by-step explanation that correctly applies the abstract rule given the prompt. This demonstrates that the model is applying a general, abstract reasoning framework, learned from text, to solve visual problems. Our conclusions here also reflect the results shown in prior work showing cross lingull reasoning transfer (Yong et al., 2025), improvements in multimodal reasoning via post-training (Rastogi et al., 2025; Wei et al., 2025; Yang et al., 2025b; Liu et al., 2025; Chen et al., 2025), transfer of structured procedural knowledge from text to -inguuiuiitic tasks (Ruis et al., 2024), and temporal reasoning transfer from text to video understanding (Li et al., 2024). We further demonstrate that reasoning abilities are highly modality-agnostic, the extent that training solely on code can yield strong multimodal reasoning. Moreover, we show this transferability is not confined to post-training phases.
A.3 DOES LANGUAGE DATA STRUCTURE DRIVE REPRESENTATIONAL ALIGNMENT WITH VISION?
Hypothesis 3: The structural properties of language data can partially drive representational alignment with visual data.
An alternative, perhaps complementary, hypothesis centers on the structural similarities between the data modalities themselves.
Data from domains like code and mat hetat cs in- herently highly structured. It is governed by strict syntax, logical dependencies, and hierarchical compositions. Sim- ilarly, visual data is far from being random collection of pixels. It rich with its own structure: spatial relationships between objects, part-whole hierarchies, and the implicit rules physics and geometry. We thus hypothesize that this shared structural foundation means that representa- tions learned from structured text are intrinsically more similar to, and thus more readily transferable to, the visual domain.
LLM-vision Score
0.0520
0.0518
0.0516
Mattic
0.0512
0.0510
0.058
0%
25%
50% Proportions (%6)
75%
100%
To test this hypothesis, we analyze the LLM-vision align- ment score across models trained with varying proportions of structured reasoning data. Our analysis, presented in Figure 11, reveals clear but non-monotonic trend. As we increase the proportion of structured reasoning data, the alignment score generally improves, suggesting that learning from abstract structure fosters more congruent latent space. However, this trend peaks at 75% ratio before declining at 100%. This might be due to a model trained purely on reasoning data learning abstract struc- ture but lacking the necessary vocabulary from other text types to effectively map it onto diverse visual concepts, thus hindering the final alignment. While the relationship is not linear, this analysis provides some signals for
code reasoring
science reasoning
Figure 11: Qualitative impact of reasoning-centric data on represen- tation alignment. The plot show how varying the proportion of differ- ent ceasoning-centric data categories in the pre-training mix impacts metries of cross-modal alignment. Results re- veal that with the LL---vision alignment score showing a generally positive but non-monotonic trend.
20

---

Published as a conference paper at ICLR 2026
structural similarity hypothesis. It remains compelling direction for future research to untangle the precise interplay between abstract structure and semantic grounding in forming cross- modal representations.
B PROBLEM FORMULATION DETAILS
B.I TRAINING PROTOCOL.
LLM pre-training setup. We follow standard practices and pre-train a suite of decoder-only Transformer models that closely adhere to the Llama-3 architecture (Grattafiori et al., 2024), spanning five model scales: 340M. IB. 3B, 7B, and 13B parameters. These models are trained for varying numbers of tokens at OB, 5B, 1OB, 20B, 30B, 50B. 70B, 100B, and up to IT tokens. We use tokenizer with vocabulary size of approximately 32000. Training is performed using the Adam optimizer (Loshchilov & Hutter, 2017) with peak learning rate of 3 x 10-4, following cosine decay schedule and a warm-up over the first 1024 steps. All models are trained with a context length of 2048 tokens and an effective global batch size of 1024. We fix the model size to 3B parameters and the total training data volume to 30B tokens as our default setting.
LLM pre-training data. Our training data is composed of diverse mixture of 16 sources, including academic, c, arts, biology, code, computer science, economics. cs, encyclopedia, food, law, literature, ma themat ica, medi cine, philosophy, politics, q forum, and wew-wrawt Each source contains at least 50B tokens.
MILMM adaptation setting. We adopt a two-stage adaptation strategy following Cambrian-1 (Tong et al., 2024a) and Web-SSI. (Fan et al., 2025), consisting of visual alignment and supervised fine- tuning. In the first stage, we train an ML.P-based projector on top of frozen vision encoder and language model align visual features with the LLM. Unless otherwise specified, we use MetaCLIP- B/16 (Xu et al., 2023) as the default vision encoder. Extracted visual features are uniformly resized mixture
to fixed length of 576 tokens. In the second stage, we perform supervised fine-tuning of rision-language and language-only instruction data to enhance the model's multimodal instruction- following ability. Both the alignment and instruction tuning stages use the Adam optimizer with cosine learning rate schedule and linear warm-up, and models are trained for a single epoch. During alignment, we use learning rate of 1 10-3 with a warm-up ratio of 6%. For instruction tuning, the learning rate is set to 4 x 10-5 with 3% warm-up. Training is conducted with an effective global batch size of 512.
MLLM adaptation data. We adopt the Cambrian-1 and Web-SSL data pipeline, but with strategic data reductions to highlight the effect of vision priors. The initial alignment stage utilizes a 1M image-caption dataset, which is roughly 40% of the original dataset's size. This is followed by supervised fine-tuning on a curated 3.5M subset of the Cambrian- data. This subset is balanced with approximately 1.5M language-only and 2M vision vision-language paired instructions, resulting in higher percentage of language-only instruction data than the original curation, as our models learn to follow language instructions during this phase.
For all experiments, we fix the random seed as 42. For MILM adaptation, we use the same order for data loading in both alignment and instruction tuning stage get stable results.
B.2 EVALUATION PROTOCOL.
LLM evaluation. We conduct comprehensive evaluation of our pre-trained models' language understanding and reasoning abilities. Following the benchmark suite used in Mamba (Gu & Dao, 2023) and GLA (Yang et al., 2023), we assess performance on two main fronts. For raw language modeling quality, we report the averaged perplexity (ppl) across Wikitext (Merity et al., 2016) and LAMBADA (Paperno et al., 2016). For reasoning, we evaluate zero-shot performance on a diverse suite of commonsense and question-ansse answering tasks, including PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), 1). ARC (Clark et al., 2018), Copa (Reddy et al., 2019), SciQA (Auer et al., 2023), OpenbookQA (Mihaylov al., 2018), and
21

---

Published as a conference paper at ICLR 2026
BooIQA (Clark et al., 2019). For concise comparison, report the unweighted averaged accuracy over all these benchmarks.
MILM evaluation. To comprehensively assess the multimodal capabilities of models, we follow 1 and establish a diverse evaluation suite comprising 16 public benchmarks. We group these benchmarks into four key categories isolate and probe the distinct components of the learned visual prior, ranging from fine-grained perception to abstract reasoning, and provide holistic view of model performance:
General: This category probes the model's ability perform visual perception and connect it with commonsense knowledge, rather than complex, multi-step reasoning. It includes GQA (Hudson & Manning, 2019), MME (Fu al., 2023), MMBench (Liu et al., 2024c), and SEED (Ge et al., 2023).
This category evaluates the model's capacity to connect visual information with the world and perform multi-step reasoning to solve complex scientific or mathematical prob- lems. It covers ScienceQA (Lu et al., 2022b), MMMU (Yue et al., 2024), A12D (Hiippala et al., 10, and Math Vista (Lu et al., 2023).
Knowledge:
This category focuses on fine-graine perception, specifically the ability to accurately read and interpret dense textual and structured data within images. It comprises TexiVQA (Sidorov et al., 2020), ChartQA (Masry et al., 2022), and OCRBench (Liu et al., 2023b).
OCR & Chart VQA:
Vision-Centric: This category mainly probes abstract visual reasoning and rough perception skills, requiring the model to perform tasks such as spatial and 3D understanding, object counting. and IQ tests. It uses benchmarks including RealWorldQA (xAI, 2024), Blink (Fu et al., 2024), COCO, ADE, and Omni3D. COCO (Lin et al., 2014), ADE (Zhou et al., 2019), and Omni3D (Brazil et al., 2023) are proposed as CV-Bench from Cambrian-1 (Tong et al., 2024a).
The overall average result is based all benchmarks. We also report the averaged multimodal evaluation accuracy for each category. To fairly assess a model's core vision ability, independent capabilities,
of its instruction-followiii often embed correct answers within conversational text rather than providing a direct response. Our evaluation uses robust parsing strategy extract the intended answer from this free-form text. This approach ensures a reliable assessment of all models, including those without language pre-training but only visual instruction tuning, making our results resilient to variations in response formatting. More details about this parsing strategy are presented in Appendix H.
address challenge: models, especially smaller ones,
LLM-vision alignment. To quantify the representational convergence and similarity between language and vision modalities, we measure the alignment between the feature spaces of LLMs and pretrained vision models, following the methodology the Platonic Representation Hypothesis (Huh et al., 2024a). For this analysis, (WIT) dataset (Srinivasan et al., 2021). Given an image I; and its caption we compute vision and language kernels, Kvision and Kiang from their respective model representations, fvision and flung:
image-caption pairs from the Wikipedia-baseee based Image Text
vission (i, = (.fvisioa (Is). fxs)io Kinng (i, = (fung (Jnny(Vi), Stang (Vz))
We then assess the alignment between these kernels using the mutual necrest- (m metric, which calculates the average overlap of the k-nearest neighbor sets (with k: = 20) for each pair. The final alignment score for given LLM is reported the average of its m.v. scores against three strong vision backbones: ViT-Large (Dosovitskiy et al., 2021) (trained on ImageNet-21K (Deng et al., 2009)), DINOv2- Mnantt-Giant (Oquab al., 2023), and CLIP-Huge (Radford et al., 2021).
For all evaluations, we fix the random seed as 42 and use temperature for testing to get consistent results.
22

---

Published as a conference paper at ICLR 2026
C RELATED WORK
C.I. FROM LLMS TO MLLMS.
With the rapid development of L.MMs (Radford et al., 2021; Google, 2023; Touvron et al., 2023), a direction of work extends L.LMs to Multimodal I.LMs. Pioneering works like Flamingo (Alayrac et al., 2022) and BLIP-2 (Li et al., 2023) connected pre-trained vision encoders to LLMs using connectors like tross-attention modules. Later models such as LLaVA (Liu et al., 2023a) demonstrate that even with a projection layer, LLMs can be extended MLLM with visual instruction tuning. This adapter-style architecture has been widely explored in numerous subsequent works (Liu al., 2024a; Tong et al., 2024a; Laurençon et al., 2024; Liu et al., 2024b; AI@ Meta, 2024; Bai et al., 2025b; Zhu et al., 2025; Team al., 2025; Lin et al., 2024).
The success of visual instruction tuning has enabled open-source multimodal models to achieve performance even comparable to proprictary counterparts (Google, 2023; xAI, 2024; OpenAI, 2024). This success underscores that multimodal capabilities in adapted LLMs largely emerge through instruction tuning, effectively unlocking knowledge already embedded in pretrained language mod- els (Zhou et al., 2024). Furthermore, recent studies (Tong al., 2024a; Laurençon et al., 2024) highlight that improvements in the underlying language model remain the most impactful means to improve multimodal performance. Inspired by these insights, our work investigates the visual priors and inherent multimodal potential embedded within pretrained L.MSs.
Though there are different ways of connecting vision to LLMs like the of discrete tokeniza- tion ( (Wang et al., 2024; Deng et al., 2025; Team, 2024), we focus on adapter-style architectures, which are most widely used and permit clean analysis of visual priors from language pre-training.
C.2 THE ROLE OF DATA IN SHAPING FOUNDATION MODEL CAPABILITIES.
Pretrained LLMs encode rich latent knowledge even across modalities depending heavily on the nature of their training data (Kaplan et al., 2020: Grattafiori et al., 2024; Han et al., 2022; Rae et al., 2021; Penedo et al., 2023; Lu et al., 2022a; Mayilvahanan et al., 2025). This has shifted the focus of research from simply scaling data to understanding the role of data and then strategically curating it to unlock specific, powerful abilities (Allen-Zhu, 2024; Aryabumi et al., 2024; Ye et al., 2024; Shinnick et al., 2025).
A prominent example, and one highly relevant our findings, is the strategic inclusion of reasoning- centric data like code. Research has consistently shown that pretraining on a mix of text and code does more than just improve coding skills; it significantly enhances a model's 's foundational reasoning and ability to understand abstract, structural patterns (Muennighoff et al., 2023; Aryabumi et al., 2024; Ma al., 2023; Zhang al., 2025). This suggests that the pre-training data mixture may endows the model with latent, generalizable structures that can be activated for tasks beyond their original domain.
This has established a central challenge in the field: determining the optimal data mixture to cultivate these desired foundational abilities (Chen al., 2024; Ma et al., 2023; Xie et al., 2023; Touvron et al., 2023; Grattafiori et al., 2024; Zhang et al., 2024; Held et al., 2025; Bai et al., 2024; Albalak et al., 2023; Shukor et al., 2025a). This has spurred a move beyond simple heuristics toward quantitative frameworks which aim to predict model's performance based on different data blends, thereby guiding the search for an optimal mixture.
However, much of this prior work has focused on optimizing data mixtures for core language proficiency. As we transition from LLMs to MLLMs, a critical question emerges: how do these text-only pre-training choices influence the model's visual priors and the potential for multimodal capabilities? Our work directly addresses this gap. We extend the investigation of language data composition's 's impact from the purely linguistic to the visual domain, systematically analyzing how different text sources contribute to the emergent visual priors in LLMs and seeking data mixture to help them "lear to see" more effectively from text pre-tnnin
23

---

Published as a conference paper at ICLR 2026
D LIMITATIONS AND FUTURE RESEARCH DIRECTIONS
While this work provides a systematic analysis of visual priors in LLMs, it is subject to several limitations that open avenues for future research.
First, our investigation primarily centers adapter-style MILM architectures. While this is prevalent and effective paradigm, our findings may not fully generalize to other approaches, such as those that employ discrete visual tokenization (Team, 2024; Wang et al., 2024; Deng al., 2025; Wu et al., 2024) or involve end-to-end joint training of vision and language components (Diao et al., 2024; Tao al., 2025; Diao et al., 2025; Shukor et al., 2025b). In these latter cases, language and vision data are co-trained, making it hard to identify the priors originating solely from language. The dynamics of how visual priors are formed and utilized could differ in these models, which leaves promising future direction.
Second, significant area our study does not address is the safety and ethical implications of these learned visual priors. Language corpora are known to contain societal biases, stereotypes, and potentially harmful content (Bengio et al., 2024; Qu al., 2023). Our analysis focused on capability, but did not investigate whether these text-based priors encode biased visual associations (e.g.. linking certain objects or roles to specific genders or races) that could manifest as harmful generation or classification behavior in downstream MLLM. A thorough audit of the faimesss and safety of these emergent priors a critical next step.
Finally, our study is confined to the domain of static images, leaving the exploration of visual priors for dynamic modalities, such as video understanding, as an open question. For example, the temporal knowledge important for video understanding might be learned more from story-related data like 1 diterattuu ure. Investigating how different textual sources contribute to priors for temporal reasoning, action recognition, and causality in video rich area for future work.
E BROADER IMPACT
Our research provides a systematic analysis of how prior visual capabilities emerge in LLMs from language-only pre-training, shifting the paradigm from accidental discovery to deliberate cultivation. While our work focuses capability, the textual data used for pre-training contains societal biases. significant risk is that these models could learn and reinforce harmful visual stereotypes, which could then manifest in downstream multimodal systems.
Nevertheless, our findings primarily help rescarchers and developers understand the nature and origins of these visual priors. We demonstrate that these priors are not single, uniform block but are composed of separable perception and reasoning components, each cultivated by different types of text.
This deeper understanding provides a clear, actionable path for more efficiently cultivating these abilities. Instead of relying serendipity scaling, teams can now strategically curate their text-only pre-training data to deliberately build a stronger foundation for vision tasks before multimodal training even begins. This targeted approach not only improves the final model's multimodal performance but also reduces the computational resources required, offering a more sustainable methodology for creating the next generation of vision- lisiuanlanguage models.
F VISUAL WORLD AND REASONING-CENTRIC LANGUAGE DATA CLASSIFICATION
This section provides the detailed classification setting and results for the visual world and reasoning- centric pre-training data sources discussed in the main paper. We use a 32B dense LLM (Yang et al., 2025a) to perform multi-label classification on 1024-token segments from each data source. Below is the full prompt provided to the LLM for the visual world and reasoning-centee data classification task. The prompt instructs the model to perform a multi-labed label classification on text segments, assigning one or more predefined categories that describe the content.
24

---

Published as a conference paper at ICLR 2026
Reasoning Categories
Vibual Categories (%)
maahh
sciance
Viseal concept
visual
visual reletio
nig reesoning
wab-craul
encycl opedia
12.2
scademic
21.0
68.2
74.4
83.3
5.2
5.1
Literature
0.3
1.3
8.9
9.1
33.4
8,4
27.9
33.5
mach
31.1
81.7
83.2
92.9
7.5
2.8
6.5
8.0
bode
96.7
13.3
6.8
97.3
3.3
2,0
2,4
3.7
Table 4: Conceptual categories of key pre-training data corpus ( (%). The table shows the
percentage of text segments from each data corpus classified into one of the conceptual categories.
Prompt for LLM-based Visual World and Reasoning-centric Data Classification
Analyze the provided text paragraph. Classify its content by identifying the primary concepts and domains using only the categories listed below. Select categories that represent the text's significant content.
visual concept: Language for naming visual entities (e.g., objects, people, places, actions, scenes).
visual attribute: Language describing visual properties (e.g., color, size, shape, texture, style).
vi visull relationships Language describing spatial or part-whole relations between entities.
Content centered on algorithmic problem-solving, logical coding implementation, and software engineering challenges.
code reasoning:
math reasoning: Content focused on logical math reasoning, proof construc- tion, and the application of mathematical principles to solve problems.
science reasoning: Focuses on scientific reasoning, including hypothesis testing, data analysis, and modeling of complex systems.
If none of the above categories apply, output None.
The percentages in Table 4 represent the proportion of text segments within each data source that were assigned a given label.
G VISUAL INSTRUCTION TUNING DATA CLASSIFICATION
We use a 7B VLM (Yang et al., 2025a) to perform a multi-label classification on the Cambrian- 7M data. We prompt the model classify visual instruction tuning data into three categories: perception-oriented reasoning-orient orienntd and neither.
Prompt for VLM-based Perception and Reasoning-centric Data Classification
Please classify the following into three categories: OTHERS, PERCEPTION, or REASON- ING. OTHERS covers most standard questions including visual analysis, general knowledge, and basic inquiries. PERCEPTION for questions that require visual speculation or inter- pretation. REASONING is for complex analytical tasks that require advanced theoretical frameworks or sophisticated analysis. When classifying, consider what type of cognitive process would be most relevant to answering the question effectively.
H ROBUST PARSING FOR VQA EVALUATIONS
significant challenge in the automated evaluation of VQA is that models often generate conversa- tional or free-form text instead of single-letter answer. A naive parsing strategy that only checks for
25

---

Published as a conference paper at ICLR 2026
exact match to the ground-truth letter (e.g., "B") would unfairly penalize models that provide correct but differently formatted response.
To illustrate, consider simple VQA task:
Question: "What is the primary object in the image?"
Options: (A) A bicycle, (B) car, (C) tree
Ground Truth:
A model could correctly answer in multiple ways, such as "The answer is (B)", "A car", "The image shows car". To capture all these valid responses, our evaluation protocol employs robust, hierarchical parsing strategy. The logic is executed as sequence of prioritized steps, stopping as soon as a valid answer is found:
1. Explicit letter extraction: The parser first searches for high- high-connidence patterns that directly indicate the chosen option letter. It uses regular expressions find formats like:
"The correct answer is (B)"
"Answrr: B"
Outputs starting ending with 'B)' output that is simply 'B' 'B.
2. Exact option text matching: If the first step fails, the parser extracts the text associated with each option from the prompt (e.g.. "A bicycle", "A car", "A tree"). It then checks if the model's generated text is an exact, case-insensitive match for any of these option strings.
Example caught: model output of "A car" would correctly mapped to option B.
3. Substring matching: As a final fallback, the parser checks if the text of any option appears as substring within the model's generated output. This handles more verbose, conversational answers.
Example caught: A model output of "The image features a car driving down the street" would be correctly mapped option because "a car" is present.
To prevent ambiguity (e.g.. if one option was "car" and another was "race car"), this step returns the longest matching option text found in the response.
This multi-tiered parsing strategy ensures a comprehensive and fair evaluation across all models, regardless of their verbosity or adherence to specific formatting instructions. It allows us to more ac- curately measure the model's underlying visual capabilities rather than its ability to follow formatting rules.
MULTI-LEVEL EXISTENCE BENCHMARK CONSTRUCTION
This section describes how we constructed our benchmark using publicly available SA-1B and ADE20K datasets. We selected images with ground-truth segmentation masks and calculated the proportion the image area each object occupied. Based on this, we created three splits: 0-30 for small objects, 30-60 for medium, and 60-100 for large, dominant objects. For each image, created a multiple-chore choice query to test object existence, sampling distractors from the dataset vocabulary and filtering them to exclude objects present in the ground truth. As this process is opea-vocabulary, use an LLM to filter out distractors that correspond to objects already present in the image but under different names. This ensures the distractors remain plausible but incorrect, providing granular evaluation of a model's ability to identify objects across a wide range of sizes.
J MULTI-LEVEL EXISTENCE BENCHMARK RESULTS
crucial tool for pushing models toward more universal visual understanding. This section presents the performance of top models on our Multi-Level Existence (MLE) benchmark, which evaluates their ability to identify objects of varying sizes. The results, detailed in Table 5, reveal
26

---

Published as a conference paper at ICLR 2026
Image
Ref seg map
Query
Options
Which of the following appears in this image?
Wardrobe X
0-30
Fireplace
Ottoman X
Barrel X
Counter X
30-60
Which of the following is in the image?
Hill
Boat x
Building v
Which the following can seen image
River
60-100
Escalator Grass
Conveyer belt X
Figure 12: MLE Benchmark Examples. The figure provides examples from the MLE-Bench, illustrating how the dataset is partitioned based on the ground-truth object size from reference segmentation maps. For instance, in the 0-30 split, the target object (a fireplace) constitutes a small fraction of the image. In contrast, the 60-90 split features correct object (grass) that covers substantial portion the image.
Model
0-30
30-60
60-100
Overall
gpt-5
73.63
90.69
86.31
82.97
gpt-4o-mini
68.72
85,10
87.94
79.32
gemini-2.5-flam claude-opue-4-202500
58.88
80.66
80.74
72.11
61.20
76.36
59.86
66.58
Table 5: Model performance on the MLE-Bench. Results are reported in three splits based on
object size in percentage (0-30, 30-60, 60-100), along with weighted overall accuracy, evaluating the ability of different models to identify objects of varying sizes.
distinct performance profiles and highlight that robust perception across all object scales remains challenge.
gpt-5 achieves the highest overall performance with weighted accuracy of 82.97% demonstrating strong capabilities across all categories. It particularly excels at identifying medium-sized objects (90.69%), which form the largest portion of the visual scene, while still maintaining competent, albeit lower, performance on small objects (73.6%). In contrast, other models exhibit more pronounced trade-offs. gemini-2.5-flar
struggles significantly with small objects, scoring only 58 88888 which is over 20 percentage points lower than its performance on medium and large objects. This indicates potential weakness in fine-gra Ene-grained perception. Similarly, claude-opus-4-2022 4-22550--- shows notable drop in performance on large, dominant objects (99. compared to smaller ones.
These findings underscore the utility of the MLE benchmark in diagnosing model weaknesses. While some models demonstrate strong general performance, universal recognition across different scales is not guaranteed. The benchmark serves as a tool for driving progress toward more comprehensive and reliable visual understanding.
K BLIND VISUAL INSTRUCTION TUNING
We also introduce "blind visual instruction tuning" trick that provides more effective starting point for visual adaptation. This trick involves an initial instruction tuning phase using only the textual data while withholding the corresponding images. This initial "blind" stage allows the model to first focus on learning the instruction-fol followiinn format of the task. Consequently, the subsequent standard tuning phase with images simultancously learning how to follow instructions. Furthermore, this process enables the model to effectively leverage its pre-existing language priors solve VQA questions that may strictly
be more dedicated to learning the core vision capabilities, rather than
27

---

Published as a conference paper at ICLR 2026
Language
Vision
Model
ppl
avg
General
Knowledge
OCR&Chart QA
Vision-Centric
Overall
Lang-Favorable
8.72
0.647
46.92
28.35
21.49
46.31
37.32
Lang- Fang-aaveebble (+Blind)
20.77
38.20
Balanced
7.49
0.655
49.59
29.02
23.63
46.59
38.64
Balanced (+Blind)
50.90
31.25
22.60
47.32
39.56
Table 6: Performance comparisons of the Lang-Favorable and Balanced models across both language and vision-language benchmarks with blind visual instruction tuning trick. The table summarizes key language metrics (perplexity and accuracy) and provides average scores for suite of vision tasks, categorized as General, Knowledge, OCR & Chart QA, and ceiionCenttri. It also shows the impact of applying our blind visual instruction tuning trick (+Blind). The blind tuning method provides an additional performance boost for both models.
require visual input, a known phenomenon and potential "shortcut" on some benchmarks (Tong et al., 2024a). This trick can lead to broad performance improvements in most of the tasks.
This trick should not be a standard in practice, since when no images are provided. models should identify their absence rather than encouraging more hallucination. This type of hallucination may persist even in frontier models including GPT-5 thinking, Gemini 2.5 Pro, and Claude Opus 4.1. Models will exhibit such hallucinatory behaviors when answering VQA questions without actual visual context given. Some examples are presented Appendix L. This highlights a systemic issue that warrants more future investigation. We leave this an optional trick and introduce this phenomenon to the community for future investigation. This also shows that "vision" can be "learned" from the language side through an unconventional, shortcut-based mechanism.
We apply this trick to Language- favorable model and Balanced model. The trick yields additional overall performance gains for both models by clear margins, as shown in Table 6. The performance gains are most pronounced in the Knowledge categories, while tasks in OCR & Chart VQA, con- versely, suffer performance drop. This pattern suggests that while the blind tuning phase provides more effective initialization for the subsequent visual instruction tuning, it gains more from strength- ening the model's s ability to leverage its internal knowledge and reasoning priors to "shortcut" the problem. This is highly beneficial for Knowledge-intensing tasks where answers can be inferred or reasoned from textual context and the LLM's pre-existing knowledge.
L HALLUCINATIONS BLIND VQA
To test for hallucinations, we present two randomly selected questions (with slight rephrasing to increase diversity) from our MLE-Bench to various models without the corresponding images. The examples below are from models also chosen randomly, primarily via LMArena (Chiang et al., 2024). We observe that stronger models do not necessarily guarantee fewer hallucinations in this blind VQA setting.
28

---

Published as a conference paper at ICLR 2026
Question
Which of the following visible in the image: Choose from: (A) purple and white striped wall, (B) person's arm, (C) a purple triangle, (D) building, Please answer with single phrase.
Question:
Answers: gemini 2.5 pro:
A purple and white striped wall
gpt-5 thinking:
persson's 's arm
qwen-max-2025-00--E5
Red and white striped wall
o4-mini-2025-04-16:
person's 's arm
mistrrl-medium-- (D) building
gemma 3 27B: person's arm.
o4-mini-2025-04-16:
person's arm
kini-k2-0711-previeew:
(B) a person's arm
qwq-32b:
(C) purple triangle
apt-oss-120br
I'm sorry, but can't see images. Could you describe what's in the picture?
kimi-k2--011-previeer:
(B) person's 's arm
mistral-meddimm25500
(D) building
llama-4scoutt17hbn 17----oouutt77h-------nrrrttttt
03-2025-04-16:
building
44aud--5omett-42225555444
I don't see any image attached to your message. Could you please share the image you't 'd like
me to analyze?
p8antom-0807-1:
(D) building
2hudd=--pus441-225000055
(B) a person's arm
29

---

Published as a conference paper at ICLR 2026
Question
Question:
What can you see in the image: Choose one from: (A) blue jersey, (B) the street, (C) araff logo, (D) blue and white computer screen. Please answer directly.
Answers:
D) blue and white computer screen.
gwwn3-235-a22b-isstuutttt-
(C) araff logo
qwen3-235b-a22b:
(A) blue jersey
amaazon.ovv--po-v11:0:
(A) blue jersey
phantom-0807-3:
Without access the image, is impossible to determine the correct answer definitively. However, based on the options provided, D) blue and white computer screen is common element in images and could plausible choice.
mistral-medium-25500
(D) building
gpt-oss-20hn 20b:
I'm sorry, but can't see the image.
(ween--3b-a33--isstuut---
(D) a blue and white computer screen.
gpt-4.1-2025-04-14.
(D) a blue and white computer screen.
phantom-0807-1:
(C) araff logo
I don't see any image attached to your message. Could you please share the image you'd like
to analyze? Once you upload it, I'll be happy choose from the options you 've provided.
gpt-5-chat:
(D) blue and white computer screen.
mistral--eedium-2500
(D) building
clade-opus-4-11222500000
Looking the image, can (B) the street. The image shows an urban street scene with buildings, trees, and what appears to be road or pathway.
folsom-08114
I cannot view or analyze images directly. However, you describe the image or provide more context, I can help determine the correct answer. Since must choose one option blindly, I'll note that (D) a blue and white computer screen is common object that might align with typical visual descriptions, but this is guess. For accuracy, please describe the image!
M ADDITIONAL QUALITATIVE EXAMPLES
To illustrate the differences between models, provide additional chat examples. These in- clude one from reasoning combinate ion (Figure 13) and one from visual combination (Figure 14). Each figure shows five different proportions of pre-training data.
30

---

Published as a conference paper at ICLR 2026
Question: Two points are circled on the image, labeled by and B beside each circle. Which point is closer to the camera?
Select from the following choices.
(A) closer
(B) is closer
Anawer the question with detailed thinking process and explanations.
Answers
The correct is
The image shows bird that is closer the camera. The bird positioned to the left of center of the frame.
The point interest for the points the image is (B). B closer to the camera. The point of interest is located at (B).
The correct B closer. The image shows the bird perched on the edge of rock, which is closer to the camera compared to the rock.
Point Bis located the body of the bird, which is the main subject in the foreground of the image. Point on the background.
0% reasoning
25% reasoning
50% reasoning
75% reasoning
100% reasoning
Figure 13: Qualitative impact of reasoning-centric data on visual spatial reasoning. The figure shows answers from five models--pee-trainnn tra-nee with 0% 100% reasoning combination data-to a visual question requiring depth perception. The answers demonstrate a clear improvement in reasoning quality: the model with 0% reasoning data gives a blunt answer, while the model with 100% reasoning data provides detailed explanation correctly applying concepts of foreground and background.
Question: Two points annotsted on the image, labeled by A and B. Consider the surface color of the points (the albedo of the surface, without the effect shading). Which point has darker surface color, or the colors is about the same? Select from the following choices.
(A) A darker
(B) (C) About
Answer the question with detailed thinking process and explanations.
Answers
question points the image. The correct answer (C) About the same.
The question asks identify the color the points. Point is on chair which is black. The answer is (B) Bis darker.
The image has darker surface color, which is the result the surface color being lighter. lighter shades
The correct answer the question is (C) About the same. The image shows kitchen with white countertop, refrigerator.
The correct answer (C) About the same. The image shows a kitchen with a white color scheme, which is choice kitchens.
the laft side of the image. The answer (A). 50% visual
X
visual
25% visual
75%
100% visual
Figure 14: Qualitative impact of visual world data on complex visual perception. The figure displays answers from five models prratie-pre-rraieeed with 0% to 100% visual combination data- question requiring an understanding of color constancy. The results show that while the model with 25% visual data provides the correct answer with reasoning relevant to the core visual principle, models trained on more visual data offer incorrect answers and flawed explanations. This suggests that simply increasing descriptive visual text does not necessarily cultivate a deeper perceptual understanding.
31


Reviews:
[{"review_id": "7EDepmnnj3", "reviewer": "Reviewer_Zsbw", "summary": "This paper investigates the emergence of visual priors in Large Language Models (LLMs) that are trained solely on text. these text-only models encode surprisingly rich visual understanding, which can be unlocked through lightweight multimodal adaptation. The authors conduct over controlled experiments that systematically vary model scale, pre-training data composition, and adaptation strategies to dissect the structure and origin of such priors. Their key findings include:\n\n1. Visual priors decompose into perception and reasoning components\n2. Reasoning priors are predominantly acquired from text sources involving code, math, and academic language\n3. Perception priors emerge diffusely from broad web-crawl text and depend heavily on visual instruction tuning\n\nLastly, the paper introduces MLE-Bench, a benchmark focused on perception-specific evaluation.", "questions": "1. The perception vs reasoning tasks showed weak or negative correlation (Figure 4). Can the authors clarify why such inverse relationships might occur?\n2. since hallucination \"stronger models do not necessarily guarantee fewer hallucinations in this blind VQA\", did the authors measure hallucination rates systematically?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The decomposition of visual priors into perception and reasoning is novel and theoretically impactful; it challenges assumptions of monolithic visual semantics in LLMs.\n2. Extensive and well-structured experimental setup across five model sizes, varying data domains, and different visual encoders (MetaCLIP, DINOv2, MAE).\n3. Offers practical insights for data-centric pre-training of future MLLMs—highly relevant to both industry and academic development.\n4. The paper conducts over 100 controlled pretraining runs, systematically varying data mixtures, encoder choice, and instruction-tuning strategies.\n5. The paper is well-written and structured logically.", "weaknesses": "1. The paper’s split of pretraining text into “reasoning” vs. “visual” buckets hinges on an automated pipeline: a 32B dense LLM is prompted to multi-label 1024-token segments into categories such as visual concept/attribute/relationship and code/math/science reasoning; those labels then drive mixture sweeps and downstream claims about where “priors” come from. That setup introduces several risks. \nFirst, it is model-dependent: is there consistency across other models? i.e. other than Qwen\nSecond, the error characteristics are unknown: there was no report of inter-annotator (human) checks are reported for the classifier, so mislabeling rates—especially on ambiguous text that mixes descriptive visual language with abstract reasoning—are unconstrained\nMy main concern here is whether the LLM-based labeling truly distinguishes “visual” from “reasoning” content, or if the observed gains simply reflect differences in data quality—i.e., one category inadvertently contains cleaner or more informative text, which could artificially boost general VQA performance.\n\n2. The paper’s validation at scale is constrained and, in some cases, internally inconsistent, which makes it difficult to confidently generalize the findings beyond the mid-scale (30–50B token) regime. While the main conclusions are derived from extensive 30–50B-token experiments, the authors extend only two mixtures to 1T tokens, both using a 7B LLM: one “language-favorable” (mix 0) and one “balanced” (mix 6). Here, the results in Table 2 (mid-scale) show that mix 0 outperforms mix 6 on pure language benchmarks, as expected from its language-heavy composition. However, in Table 3 (1T-token scale), the trend reverses on several language tasks—mix 6 overtakes mix 0 despite having proportionally less “language” text. This reversal is puzzling and suggests a possible interaction between model scale, token budget, and mixture composition that is not fully explored. Could the authors elaborate more on this?\n\nCompounding this, the 1T-token budget far exceeds the Chinchilla optimal region for a 7B model (≈150–200B tokens), placing both scaled-up models deep into the over-training regime where diminishing return are likely. That makes it difficult to attribute performance shifts solely to mixture content, rather than to scaling artifacts or excessive token exposure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0uYExuDTaY", "reviewer": "Reviewer_fgou", "summary": "## Summary\n\nThis paper is a pioneering work that uses a proxy visual task framework (P-SET) to quantify and dissect the **visual priors** that emerge in Large Language Models (LLMs) from **text-only** pre-training, decomposing them into separable perception and reasoning components. The study reveals that these priors follow unique scaling laws, where the reasoning prior scales poorly, providing the MLLM community with critical **descriptive insights** to guide future model alignment and pre-training strategies.", "questions": "## Questions\n\n1.  **Q1: Mechanistic Validation of Prior \"Separability.\"**\n    The core claim is the operational separation of visual priors (perception vs. reasoning) based on P-SET tasks. Can the authors provide evidence at the level of the LLM’s internal representations (e.g., via Representation Similarity Analysis, CCA, or causal intervention) to prove that these priors are **orthogonal or uncorrelated** in the latent space? Lacking this mechanistic evidence, how do we rule out that the separation is merely an artifact of the **proxy task design** (P-SET)?\n\n2.  **Q2: Rigorously Excluding Textual Knowledge Confounding.**\n    The paper asserts that visual priors emerge from text-only training, yet fails to **strictly exclude** that this is simply the encoding of **structured, descriptive visual language** found in the massive text corpus. Have the authors attempted to train a small LLM on a corpus **devoid of all relational and spatial language** to see if the reasoning prior still emerges? If this **linguistic confounding** is not ruled out, the causal foundation of the central thesis—that \"LLMs learn to see before seeing\"—is a fatal weakness.\n\n3.  **Q3: Transferability Challenge of the P-SET Proxy Tasks.**\n    The P-SET tasks are synthetic. While the paper shows a **correlation** between the prior and downstream MLLM performance, how **robust** is this correlation? Have the authors performed an ablation showing that an LLM with a higher P-SET score *consistently* yields superior downstream performance even when subjected to *different* minimal fine-tuning techniques (e.g., LoRA vs. full tuning)? If a high P-SET score does not reliably predict downstream gains, its value as an **efficient MLLM pre-training metric** is questionable.\n\n4.  **Q4: Influence of Specific LLM Family on Scaling Laws.**\n    The scaling laws are predominantly based on the Llama family or similar Transformer architectures. Have the authors attempted to replicate the scaling trends of these priors on **non-Transformer architectures** (e.g., Mamba or RetNet)? If these laws are **architecture-dependent** rather than a **universal property of language models**, doesn't the demystification of visual priors only serve as a description of a **specific model family** instead of a **general law of learning**?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "## Strengths\n\n1.  **S1: Pioneering Dissection of Emergent Visual Knowledge.**\n    This work represents a **pioneering effort** to systematically dissect and analyze the implicit visual knowledge (priors) that emerges purely from **text-only** pre-training in Large Language Models. By demonstrating the existence and growth of these priors, the paper provides a crucial, non-trivial insight into the mechanisms by which LLMs acquire world knowledge, laying the foundation for **mechanistic interpretability** studies in the multimodal domain.\n\n2.  **S2: Novel Framework for Operationalizing and Separating Priors.**\n    The paper introduces a **novel and comprehensive framework** using proxy visual tasks (perception vs. reasoning) to quantitatively *operationalize* and *separate* the distinct components of the visual prior. This methodology allows for the first time a clear, quantitative distinction between different types of emergent visual capabilities, moving beyond anecdotal evidence to **systematic, scalable measurement**.\n\n3.  **S3: Key Scaling and Developmental Insights for MLLM Construction.**\n    The work yields **critical, quantitative insights** into how the perception and reasoning priors scale with model size. The key finding that the **reasoning prior scales poorly** compared to the perception prior provides a valuable, data-driven hypothesis for MLLM researchers, guiding future efforts to strategically adjust data composition or architecture to unlock superior visual reasoning capabilities.\n\n4.  **S4: Rigorous Construction of Proxy Visual Tasks (P-SET).**\n    The paper's construction of the P-SET benchmark, specifically designed to isolate and measure perception and reasoning capabilities within the text modality, is highly rigorous. This set of tasks effectively serves as a crucial tool for **probing** the latent knowledge structure of LLMs, and the benchmark itself is a valuable public contribution to the field of MLLM analysis.\n\n5.  **S5: Strong Empirical Evidence via Causal Intervention (Minimal Fine-Tuning).**\n    The central hypothesis—that these priors exist and are beneficial—is supported by strong empirical evidence showing that LLMs with higher measured priors achieve better performance in MLLM tasks (e.g., VQA) after **minimal multimodal fine-tuning**. This suggests that the latent knowledge is indeed functional and ready to be \"unlocked,\" validating the study's relevance to the efficiency of MLLM alignment.", "weaknesses": "## Weaknesses\n\n1.  **W1: Lack of Mechanistic Evidence for the Separation of Priors.**\n    The central claim is the separation of visual priors into \"perception\" and \"reasoning\" components. This distinction relies heavily on an **operational definition** (performance on specific proxy tasks) rather than **mechanistic evidence**.\n    * The authors **fail to demonstrate** that these priors are truly **orthogonal or separable** in the LLM's internal representations (e.g., via representation similarity analysis, or targeted causal interventions on specific latent dimensions).\n    * This lack of a solid **mechanistic foundation** suggests that the claimed separation might be an artifact of the task design itself rather than an inherent property of the emergent knowledge within the LLM.\n\n2.  **W2: Fatal Causal Confounding of Textual Knowledge and Visual Priors.**\n    The core, high-impact thesis is that visual priors emerge from **text-only** pre-training. However, the pre-training data contains vast amounts of **structured, descriptive visual language** (e.g., \"the red square is to the left of the blue circle\").\n    * The paper fails to **rigorously rule out** that the LLM is merely encoding **linguistic knowledge about visual relationships** (i.e., the language semantics of relational phrases) rather than truly emergent **visual spatial knowledge**.\n    * This **causal confounding** between linguistic encoding and visual priors significantly weakens the claim that LLMs learn to \"see\" before seeing an image, as the identified \"prior\" could simply be complex language fluency.\n\n3.  **W3: Lack of Prescriptive Guidance for MLLM Design (The Analysis-Synthesis Gap).**\n    The work is primarily **descriptive**, successfully identifying the existence and scaling trends of these priors (e.g., reasoning prior scales poorly). However, it fails to offer **actionable, prescriptive guidance**.\n    * For instance, given the finding that the reasoning prior scales poorly, can the authors propose a **concrete, quantifiable text pre-training data optimization strategy** (e.g., adjusting the ratio of relational reasoning text) and empirically prove that this modification **more efficiently** boosts the reasoning capability?\n    * This lack of **analysis-to-synthesis** guidance limits the practical value of this research for constructing or optimizing the pre-training stage of next-generation MLLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lMNUL9aWUZ", "reviewer": "Reviewer_1T9p", "summary": "Based on the common practice of finetuning Multimodal Large Language Models (MLLMs) from pretrained Large Language Models (LLMs), this paper analyzes how LLM pretraining influences downstream MLLM finetuning performance. It focus primarily on how LLM pretraining data construction affects the downstream MLLM performance. The results reveal several interesting findings including aspects of\n- Scaling law of LLM pretraining on MLLM finetuning\n- Source of visual prior\n- Data arrangements\n- Structure of visual priors\n\nGuided by these findings, this paper pretrains a new LLM using a reorganized language dataset, achieving improved downstream performance for MLLM finetuning.", "questions": "How is model trained with \"0B\" data obtained in Figure 1? Why a model with no training data can also get non-trivial performance?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Well-motivated problem with clear relevance to current MLLM construction practices.\n- Sufficient evaluation breadth and depth, with extensive controlled studies across data mixtures and scales.\n- In-depth findings that are coherent with the presented experiments and lead to a concrete, data-centric pretraining recipe.", "weaknesses": "- Overclaim in the abstract that \"perform visual tasks without ever having seen an image\". The paper does not actually demonstrate an MLLM that performs visual tasks without any visual training.\n- Unclear vision-input processing in the default settings. It is not clear whether the default setup uses any vision encoders or just a MLP adapter. And how to process the vision inputs.\n- Since stronger LLMs typically yield stronger MLLMs after adaptation, it remains ambiguous how much of the reported gains come from claimed \"visual priors\" versus general language capacity. The paper only reports perfomance on vision tasks, but lacks evaluation of the language capacity of each mdoel. So it is not very clear whether it is really benefits from visual prior or just stronger langauge capacities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cpumblq34Y", "reviewer": "Reviewer_JSpZ", "summary": "This paper presents a comprehensive study on how the vision-related capabilities of MLLMs—including visual perception and reasoning—emerge during text-only pretraining. To investigate this, the authors conduct extensive controlled experiments by training LLMs of different sizes under various data mixture settings, carefully varying the proportion of image-caption pairs of different types.\n\nThese experiments yield a set of valuable insights, revealing where the model’s perception and reasoning abilities originate from even in the absence of real visual input. The results highlight the importance of descriptive language data and reasoning data in shaping vision-related capabilities, and identify the optimal data mixture ratios for strong downstream VQA performance.\n\nLeveraging these findings, the authors scale up the training to approximately 1 trillion tokens and train a larger model following the recommended data composition. The improved performance of this scaled model further validates the effectiveness of the insights derived from the ablation studies.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The problem investigated in this paper is highly important and relevant to the MLLM community, yet has been largely overlooked in prior work. Understanding how vision-related capabilities emerge from text-only pretraining provides valuable insight into the foundations of multimodal learning.\n\n* The conclusions drawn from the extensive experiments are both instructive and insightful. They offer practical guidance for future model development, particularly in designing effective data mixtures and scaling strategies for training more capable MLLMs.\n\n* The paper is well-organized, clearly written, and easy to follow. The presentation is informative, with a logical flow that effectively conveys the key ideas and findings.", "weaknesses": "* All VQA questions are divided into several categories, such as General VQA, Knowledge-based VQA, and OCR & Chart VQA. However, these category definitions may appear abstract to readers, especially without concrete examples. Including one or two representative examples for each type in the main text or a table would significantly improve clarity and help readers better understand the distinctions between the tasks.\n\n* I understand that the 9-page limit constrains the space available for presentation. Nevertheless, some conclusions in the supplementary materials also contain valuable insights that support the paper’s findings. While they may not fit in full, briefly referring to them in the main text would help draw readers’ attention to these contents and enhance the overall impact of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)