Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training
Authors: Yangyi Chen
Hao Peng
Tong Zhang
Heng Ji
Venue: Accepted by TMLR ()

Abstract: In standard large vision-language models (LVLMs) pre-training, the model typically maximizes the joint probability of the caption conditioned on the image via next-token prediction (NTP); however, since only a small subset of caption tokens directly relates to the visual content, this naive NTP unintentionally fits the model to noise and increases the risk of hallucination. We present PRIOR, a simple vision-language pre-training approach that addresses this issue by prioritizing image-related tokens through differential weighting in the NTP loss, drawing from the importance sampling framework. PRIOR introduces a reference model—a text-only large language model (LLM) trained on the captions without image inputs, to weight each token based on its probability for LVLMs training. Intuitively, tokens that are directly related to the visual inputs are harder to predict without the image and thus receive lower probabilities from the text-only reference LLM. During training, we implement a token-specific re-weighting term based on the importance scores to adjust each token's loss. We implement PRIOR in two distinct settings: LVLMs with visual encoders and LVLMs without visual encoders. We observe 19% and 8% average relative improvement, respectively, on several vision-language benchmarks compared to NTP. In addition, PRIOR exhibits superior scaling properties, as demonstrated by significantly higher scaling coefficients, indicating greater potential for performance gains compared to NTP given increasing compute and data. The code is available at https://github.com/Yangyi-Chen/PRIOR.

Published in Transactions on Machine Learning Research (03/2026)
Prioritizing Image-Related Tokens Enhances Vision-Language Pre- Prainingn
Yangyi Chen
yongyic3@illioiis.ed edu
University of Ilinnis Urbana- Champpignmnn
Hao Peng
haopengthillinois.coo edu
University of Illinois Artana-Champatie
Tong Zhang
toshang@illinois.coo edu
University of Alinois Jybana-Champpln
Heng Ji
hengji@illinois.. edu
University of Alinois Prbuna-Chammpeen
Reviewed on OpenReview: https: // openreview. net/ forum? id- jDenL1hB12
Abstract
In standard large vision-language models (LVLMs) pre-training, the model typically maxi- mizes the joint probability of the caption conditioned the image via next-token prediction (NTP); however, since only small subset of caption tokens directly relates to the visual content, this naive NTP unintentionally fits the model to noise and increases the risk of hal- Iucination. We present PRIOR, a simple vision-language pre-training approach that addresses this issue by prioritizing image-related tokens through differential weighting the NTP loss, drawing from the importance sampling framework. PRIOR introduces reference model-a text-only large language model (LLM) trained the captions without image inputs, to weight each token based on its probability for LVLMs training. Intuitively, tokens that are directly related to the visual inputs are harder to predict without the image and thus receive lower probabilities from the text-only reference LLM. During training, we implement a token- specific re-weighting term based on the importance scores to adjust each token's loss. We implement PRIOR in two distinct settings: LVLMs with visual encoders and LVLMs without visual encoders. We observe 19% and 8% average relative improvement, respectively, on several vision-language benchmarks compared to NTP. In addition, PRIOR exhibits superior scaling properties, demonstrated by significantly higher scaling coefficients, indicating greater potential for performance gains compared to NTP given increasing compute and data. The code is available at https: //tp//ggttttttt//// Cat.uu..coo/aanyiiiii Saaryiii-heePPPI...
1 Introduction
Vision-language pre-training enhances both visual perception and visual-textual association capabilities in large lasion-language models (LVLMs) (Zhang et al., 2021; Wu et al., 2024). However, our preliminary human annotations 100 examples from Capsfusion (Yu et al., 2024) reveal that only 31.3% of words in web-scale image-caption pairs directly relate to the associated images, with the remainder containing irrelevant information, stylistic elements, or websile-specific conlent. For example, considering the second example in Fig. (Top), only "house", "front yard", and "lawn" are image-related, while the remaining tokens lack visual correspondence, such as the location and price information of the house. The standard next-token prediction (NTP) objective treats all tokens equally, regardless of their relevance to visual content. Thus, besides learning visual perception and vision-language alignment, LVL.Ms inevitably model the entire

---

Published in Transactions on Machine Learning Research (03/2026)
Image Description:
Kyle Casey, player
basketball team, contributed points their
77:70 overtime victory on the at Columbia.
Loss (Original):
0.68 0.47
0.47
0.65
0.08
LLM Probability:
0.01 0.11
0.47
0.05
0.48
0.88
HARAR 30
Loss (PRIOR):
0.56
0.51
0.01
Image Description:
A small house with front yard and lawn located at 7338 24th Ave NE, Seattle was recently sold for $365,500.
Loss (Original):
0.72
0.13
0.17
0.32
0.08
LLM Probobility:
0.03
0.88
0.83
0.77
0.93
Loss (PRIOR):
2.24
0.05
0.09
0.24
0.02
NTP (Next-Token Prediction)
Average
202
NTP (Nest-Tuken Prediction)
PRIOR
CNTP = Σ log p (te v,t,i)
220
Performance
202
PRIOR (Ours)
182
CPRICA
W/
log 1og((410,,e)
162
Dj=1 Wy
==1
542
W/ 1 (ti tci))
600
1000
1000
2000
2000
2000
3000
4000
4003
6000
Pre-Training Step
Figure 1: (Top) Synthetic examples to highlight the motivation of PRIOR. Only few tokens in the captions (highlighted in blue, word-level for better visualization) related to the associated images. PRIOR utilizes probability scores from text-only LLM to recalibrate the original loss function at the token level, prioritizing image-related tokens that receive lower probability scores from the LLM. (Bottom Left.) PRIOR formulation. Given an image paired with a caption f..',',.,''', PRIOR enhances vision- language pre-training by assigning a normalized weight to each token loss, which is computed based on the LLM probability (1g(ke)). (Bottom Right) Performance of PRIOR. PRIOR demonstrales consistent. performance improvement (average over several languae benchmarks) and better training stability compared to the widely used next-token prediction objective in vision-language pre-training. In addition, PRIOR shows superior scaling behaviors in both performance predictability and potential improvement with increased compute and data ($4).
caption distribution, potentially overfitting to noise and contributing to hallucination (Sharma et al., 2018; Changpinyo et al., 2021; Liu et al., 2023b).
In this work, we present PRIOR, a simple approach that enhances vision-language pre-training by prioritizing optimization on image-related tokens. PRIOR addresses the above challenge by effectively distinguishing between imago-relatod tokens and other tokens in the training corpus. We train text-only reference model capture the caption distribution without visual context the vasion-language corpus. This allows identify which tokens likely to be image-related and establish importance distribution over the token
2

---

Published in Transactions on Machine Learning Research (03/2026)
space. The key insight is that tokens casily predicted by the reference model likely contain minimal visual information, while those difficult for text-only models to predict more likely convey image-specific content. We formalize this intuition by using the text-only model's predictive probability to reweight the NTP loss for each token, as shown in Fig. (Bottom left). This training algorithm, inspired by importance sampling principles, prioritizes the optimization on tokens containing image-related information while preserving general language generation capabilities.
PRIOR addresses several crucial limitations in previous vision-language pro-training approaches. Existing methods, detailed in $6, primarily fall into two categories, each with their own challenges. The first category involves supplementing NTP with additional training objectives such as contrastive loss (Li et al., 2019) or distillation loss (Liao et al., 2025), but these approaches sacrifice NTP's simplicity and create challenges for large-scale, efficient implementation with existing frameworks (Rasley et al., 2020). In contrast, PRIOR can be casily integrated with existing pre-training frameworks through minimal code changes for the loss computation and ofline-computed token importance scores. The second category focuses on optimizing training data through distillation (Chen et al., 2024a); filtering pipelines (Guo al., 2024), and related approaches. While conceptually promising, these methods face inherent scalability constraints due to their dependence on teacher models or human-designed hcuristics. In contrast, PRIOR's text-only reference model offers a scalable alternative that dynamically extracts useful knowledge from noisy web-scale datasets, ensuring broad compatibility with advancmeents across vision-language datasets.
To evaluate PRIOR's broad applicability across different LVLMs architectures, we conduct experiments with both vision encoder-base based LVLMs (Zhu et al., 2023; Li et al., 2023b) and end-to-end unified LVLMs with simple linear projector for image processing (Chen et al., 2024f; Diao et al., 2024; Tao et al., 2024). Our results demonstrate that PRIOR consistently outperforms the naive NTP pre-training baseline, achieving 19% and 8% average relative improvement across these two architectures respectively, while also enhancing training stability. The average performance comparisons are visualized in Fig. (Bottom Right) and Fig. 4. In $3.4, we compare PRIOR with additional basclines and alternative token selection methods, and show the unique advantages of PRIOR. Further analysis in $4 reveals that PRIOR exhibits superior scaling behaviors in terms of performance predictability (Fig. 6) and potential improvement with increased computational resources (Fig. 7). Morcover, as detailed in $5.2, PRIOR accelerates pro-training by more effectively optimizing loss both image-related and image-unrelated tokens.
2 PRIOR
2.1
Problem Formulation: Vision-Language Pre-training with Image-Caption Pairs
We review the general lasion-language pre-training that leverages a web-scale dataset comprising image-caption pairs (v, c) to develop models capable of generating relevant textual descriptions based on visual inputs. v represents the image, and c= 4,, 11.,2....,4 .....,,,,, 4s represents the corresponding caption consisting of k tokens. The widely used vision-language pre-training objective, next-token prediction (Lute), trains the model to predict each token in the caption based on the image and all previous tokens (Liu et al., 2023d; Dai et al., 2023; Lu et al., 2024), formally expressed as:
k Σ
CuP = logp(t, v,tes).
(1)
where 4 is the current token to predict, ber represents the prefix.
2.2 PRIOR: Prioritizing Image-Relate Related Tokens for Vision-Langune Pre- Pre-Training
Eq. outlines a model that conditions Loken prediction on both images and text but lacks a mechanism to verify whether visual information is actually being effectively utilized. The model can optimize this objective by relying exclusively on textual context, potentially resulting LVLMs that overfit supplementary text that docsn't correspond to visible image content. To address this, we introduce PRIOR, simple method to advance the original vision-language pre-training by prioritizing image-related tokens, which are
3

---

Published in Transactions on Machine Learning Research (03/2026)
automatically identified by a text-only reference LLM. Drawing from the importance sampling framework ($2.3), we use the reference model to construct target distribution that assigns higher probability to tokens likely requiring visual information. This approach effectively concentrates optimization on image-related content while reducing emphasis on tokens prodictable from text alone.
Text-Only LLM for Modeling Text Distribution PRIOR introduces a text-only LLM as a reference model the caption distribution without the image inputs. Specifically, we apply NTP loss exclusively on the text tokens, and the training objective is:
logpr (4: lei) LTEXT Σ
(2)
Vision-Language Pre-Training with Reweighted Tokens Loss PRIOR utilizes the reference model's token probability calculate the importance score uy for each token:
Wh: (1 (1 (=)))
where (empirically set to 1) modulates the impact of w, during pre-training. Tokens with higher importance scores are those that the text-only LLM finds difficult to predict, suggesting they are more likely to be image- related. We implement this scoring process oflline computing and storing the importance score for each loken beforehand- -hich eliminates the need for reference model inference during rision-langual pre-training.
PRIOR applics these token-specific importance scores to reweight the NTP loss. The training objective is:
LPRIOR
k
Wy
log t,ts)
The normalization term w// E/= wy ensures that the importance scores form proper distribution across all I tokens, while the multiplication by preserves the overall scale the loss. This algorithm strategically upweights tokens that the text-only model struggles to predict, which typically correspond to image-specific information, while maintaining sufficient weight distribution across contextual tokens to ensure coherent. language generation.
2.3 PRIOR as Importance Sampling
We present the theoretical framework based on importance sampling that underpins and motivates PRIOR. In general, PRIOR can be interpreted as form of importance sampling where we draw more samples from regions where the reference model is uncertain. In classical importance sampling, we estimate expectation under target distribution p(x) using samples from proposal distribution q(z):
Ep(=)]((x]]= = [(x))())d ((x))(o0) (()) lgg(s) f(x) p(x) g(x))
(5)
g(x)
where the term plx) q(æ) represents the importance weight that corrects for the mismatch between the distributions. For PRIOR, we can formulate our problem as follows:
Function f(x): The NTP loss (a.k.a, negative log-likelihood loss) log Pnodes (4//0, tee).
Sampling Distribution q(z): The empirical distribution in data Patta (t//v, tei).
Target Distribution p(z): We aim to define new target distribution Ptargat (tile, tei) that assigns higher probability to tokens that are difficult for the reference model to predict:
Ptarget (1s/u, bei) x (La (e)) - (1-p(((()e)))
(6)
where pt(tittes) the reference model's probability estimate. Intuitively, we upweight tokens that have low reference probability (surprising tokens).
4

---

Published in Transactions on Machine Learning Research (03/2026)
Importance Sampling Formulation: Since only have samples from Pasta (tyle, tes), we use importance sampling to estimate the expectation under Ptargat hes):
E Perrge (t.),.tes) log Prodel (Life,, (cxi)]
Prargot (4/ (k/)))
(7)
E
log Pnodel Pnodei (1,),,)))))))
Peata (1e 0, bee))
Thus, the importance weight for each token loss (i.e., log Phodal (4:/u, (ei) in Eq. 7), with the target distribution defined in Eq. 6, is:
w(1iuubee)) = Prarget PtargarttilU;te tes) x (1 pr(((()))))))
(8)
Paata (li,,,
To make this a proper probability distribution over the sequence, normalize:
w(t,lu,,es)
((tllutte) ΣS=1 w(tzlo,tes)
(9)
While this self-normalization introduces bias into our estimation the expectation over the target distribution, it substantially reduces variance, yielding improved training stability well-established tradeoff in previous work (Metelli et al., 2018; Korbak et al., 2022). Our importance-samplet loss thus becomes:
Lis Σ wr (t,lu, log Phodel (t: (u,
(10)
This weighted average represents the expected loss under our target distribution that emphasizes difficult tokens. Note that this formulation consistent with Eq. 4, with the difference in the multiplication by k, which preserves the expected loss magnitude when transitioning from NTP to importance- the normalized weights sum to I, omitting & reduces the loss to a weighted average rather than a weighted sum, effectively shrinking the gradients by factor of approximately k. Multiplying by & restores the original scale. This design choice is important from two perspectives. First, ensures gradient consistency: gradient. magnitudes are preserved, enabling direct reuse of existing learning rates and optimizer configurations without retuning. Second, it provides theoretical grounding by aligning with self-normalized importance sampling, where normalization reduces variance at the cost of introducing small bias (Metelli et al,, 2018). In addition, this importance sampling framework can be easily extended to consider the a term. We also provide a theoretical justification for the key intuition of PRIOR through mutual information analysis, as detailed $A.
loss. Since
3
Experiments
3.1
Implementation Details of PRIOR
We adopt CapsFusion (Yu al., 2024), which comprises 120M image-text pairs for vision-language pre-training experiments. We first sample a subset (~5M) and use only the caption to pre-train a text-only reference LLM (initialized with Llama-3-8B (Dubey et al., 2024)). Since the captions in vision-language datasets are typically shorter, we pack multiple samples within the context length (8192) for elficient pre-training.
For vision-language pre-training, we sample another subset (~3M) and compute token-level reference probabilities pr (t.lee))) offline using the reference LLM. We store this data as (image, caption, token-level reference probability list) tuples. To measure the extensive applicability of PRIOR across diverse LVLMs architectures, investigate two types of LVLMs:
H-LVLMs: We implement LVLMs with pre-trained visual encoders Heteroggooosss architcctures). Following the typical VA-Sllle LVLMs design (Liu et al., 2023d), we adopt a straightforward ViT- MLP-LLA architecture, employing Llama-3.2-3B as the LLM backbone for efficient experimentation and SLIP-ViT-Large only train the MLP component during pre-training learn the lision-language alignment and maintain consistent language capabilities across all LVLMs for controlled study.
(Radford al., 2021) as the vision encoder. Following Liu et al. (2023c), we
5

---

Published in Transactions on Machine Learning Research (03/2026)
MME-Porcaption
MME-Reasoning
MMStar
27433
Peeformance
peeformmnce
S1AT
TEE
Pre-Sraining
Pre-Tralaing
Pre-Training
POPE
MMBench
SEEDBench
38.33
Performance
Performunce
GOLDS
300
100
2000
200
10
Pre-Eratning Step
Pre-tratning
Pre-Iraining
NTP (Next-Token Prediction)
PRIOR
Figure Main experimental results of LVLMs with pre-trained visual encoders. We compare PRIOR with the NTP vision-language pre-training across various training steps on LVLMs with pre-trained visual encoders, and we annotate the highest performance for each method, respectively. PRIOR demonstrates both superior performance and greater stability throughout the entire training.
We implement IVLMs with Unified architectures, which drop the pro-trained visual encoders and adopt single end-to-end Transformer architecture for vision-language modeling (Chen et al., 2024f; Lei et al., 2025; Zhang et al., 2025). Following Chen et al. (2024f), we initialize with Llama-8B and pre-train it on ImageNet for 500 steps before further pre-training on imagc-caption data.
U-LVLMs:
For the two types of LVLMs, we set the total training steps as 5,000, record every 500 steps, with the batch size as 512. During U-LVLM pre-training, we also incorporate an equal proportion of DCLM language dataset (Li al., 2024b) maintain general language capabilities.
3.2 Experimental Setting
Evaluation Benchmarks We select the following benchmarks for evaluation: (1) MME (Fu et al., 2024), which measures both the perception and reasoning capabilities. Thus, we report MMF-Perccptioo and LLME-Reasoning respectively. (2) MMStar (Chen al., 2024b), which measures advanced vision-language skills. (3) POPE (Li et al., 2023d), benchmark for object hallucination evaluation. (4) MMBench (Liu
et al., 2024c), a benchmark designed for general vision-language capacities evaluation. (5) SEEDBench (Li et al., 2024a), which covers 12 evaluation dimensions including diverse aspects of LVLMs.
Evaluation Setting To elicit meaningful responses beyond image captions during evaluation, we conduct 20 steps for instruction fine-tuning on all pre-trained checkpoints using the same data subset from the post- training set in Chen et al. (2024f). We include a controlled sensitivity study over the number of fine-tuning Steps in $B. We employ VLMEvalKit (Duan et al., 2024) for unified evaluation and comparison across models.
3.3 Experimental Results
The main experimental results for cach dataset are shown in Fig. 2 (H-LVLMs) and Fig. (U-LVLMs). Average performance across all datasets is summarized in Fig. 1, bottom right (H-LVLMs) and Fig. (U-LVLMs). We have the following findings:

---

Published in Transactions on Machine Learning Research (03/2026)
MME-Percaption
MMEE-Rasoning
MMStar
7B
206-1
Peeformance
Prefformenee
Performance
20.77
Pre-Iraining
Pro-Tralning
Pre-Training
POPE
66-76
64.81
MMBench
SEEDBench
15.00
$1.00
Performonce
Porformance
23.54
30C
100
2000
201
100
100
Pre-Eratning Step
Pre-tratning
Pre-Iraining
NTP (Next-Token Prediction)
PRIOR
Figure 3: Main experimental results of LVLMs with unified architectures. We compare PRIOR with the NTP lasion-language pre- treiniiin on LVLMs with unified architectures.
PRIOR generally outperforms the naive NTP pre-training objective by a large margin. We observe that PRIOR enhances performance across the entire training Irajectory for both H-LVLMs and U-IVLMs, demonstrating improvements compared the NTP objective at most intermediate training steps. For fully converged models (i.e., 5,000 steps), we observe average relative improvements across all datasets of 18.61% for H-LVLMs and 7.93% for U-LVLMs.
PRIOR demonstrates better training stability. We observe performance fluctuations during training across both H-IVLMs and U-LVLMs. The NTP objective causes significant performance drops MMBench and SEEDBench for H-LVLMs, while produeing sharp performance fluctuations in POPE and MMBench for U-IVLMs. PRIOR significantly reduces these fluctuations, demonstrating superior training stability compared to the naive NTP pre-training objective.
PRIOR exhibits higher potential in performance. The highest performance scores are highlighted in Fig. 2 and Fig. for each dataset. Our results show PRIOR outperforms the NTP objective in peak performance for both LVLM types. For H-IVTMs specifically, PRIOR achieves optimal performance in final checkpoints, while NTP peaks carlier in training. This suggests PRIOR benefits from extended training, likely due to ils comprehensive optimization approach that continues refining multimodal representations in later stages. PRIOR's sustained improvement trajectory indicates greater performance potential.
3.4 Comparison with Additional Baselines, Token Selection Methods, and Selected Reference LLMs
We compare PRIOR with the following approaches specifically designed for H-LVLMs: (1) Image-text matching: The training objective that pre-trains the MLP connector to map the [CLS] token embedding close to the caption embedding (Li et al., 2023b; Chen el al., 2024g). (2) Image-text contrastive learning: The training objective that pre-trains the MLP connector to align projected image representations with text representations (Chen et al., 2024g: Radford al., 2021). (3) Reconstructive tuning: The training objective that supervises LVLMs reconstruct images, focusing on the inherent richness and detail within input images (Wang et al., 2024). All these methods combined with NTP for 5,000 training steps.
7

---

Published in Transactions on Machine Learning Research (03/2026)
Table
A comparison of PRIOR with additional baselines and alternative token-selection approaches implemented on H-LVLMs. The evaluation is the average of runs. PRIOR yields consistently better results throughout the evaluation.
Category
Method
MME-P
MME-R
MMStar
POPE
MMBench
SEEDBench
Naive
Next-Token Prediction
576.2
207.5
30.9
68.7
36.5
Image- Text Matching
Baseline
Image-Text Contrastive
635.6
218.6
26.3
67.5
38.2
11.2
Reconstructive Tuning
618.3
202.1
27.3
69.3
38.0
11.3
Token
Atlention Weighting
653.4
215.8
31.2
73.4
36.9
39.2
Selection
Rare Word Weighting
(8B off-the-sheif LLM)
703.4
203.6
29.3
68.6
3144
Selected
PRIOR (31 finestuned L.M)
824.2
222.9
32.1
71.0
38.0
44.6
Reference LLMs
FRIOR (On-the-fly inference)
856.0
210.7
31.7
67.0
38.3
42.5
(8B fine-tuned LLM)
Average
2.0
NTP (Nest-loken Prediction)
PRIOR
170
130
130
900
1000
2000
<065
Pre-Training Step
Figure 4: The average performance comparison on LVLMs with unified architectures. PRIOR demonstrates better performance and stability across the entire training process.
Figure 5: The ablation study of PRIOR regarding and k on H-LVLMs. Results show optimal performance with a=1 and scaling factor k; justifying the design choices in PRIOR.
The results presented Tab. demonstrate that PRIOR consistently outperforms these basclines across benchmarks, with particularly significant gains observed on complex reasoning tasks (i.e., MME-R, MMStar). Additionally, PRIOR integrates more seamlessly with existing pre-training frameworks, requiring only modifications to the loss computation when data stored offline. This enables scalable pre-training of LVLMs on large-scale datasets across compute clusters.
To further validate the effectiveness of our approach, we conduct another study comparing PRIOR against. alternative weightneg strategies. Since PRIOR operates by prioritizing optimization on important tokens within LVLMs, we examine two additional heuristic methods for computing token importance scores: (1) Attention weighting, which leverages cross-attention scores from a pre-trained CLIP model (Radford et al., 2021) to assign weights text tokens based on their visual relevance. (2) Rare word weighting, which compules importance
using inverse token frequency statistics from large-scale text corpora (Das el al., 2023), under the assumption that rare tokens carry more semantic significance.
As shown in Tab. 1, PRIOR substantially outperforms these two loken-weighting methods across all evaluation benchmarks, validating the superiority of our principled approach to token imporlance estimation, with gains of +222.2 and +329.5 on MME-P, a critical benchmark that examines the visual perception capabilities of LVLMs, respectively. This substantial margin highlights that our importance sampling formulation of token importance, grounded in the PRIOR objective, more effectively identifies semantically critical tokens than attention-based or frequency-based heuristies.
8

---

Published in Transactions on Machine Learning Research (03/2026)
H-LVLM
U-IVLM
70-
Relative Prodiction Error (%)
Relative Prediction Error (%)
70-
50-
34.00
7.11
20-
4.65
14.17
33.66
1.48
1.12
6.9
4.56
1.94
(Next-Token Prediction)
Figure 6: The relative prediction error of NTP and PRIOR. We observe that the performance of LVLMs trained via PRIOR is more predictable at scale.
We further conduct ablation studics to examine the effect reference LLM configuration (í.e., fino-tuned vs. off-the-shelf, 3B vs. 8B) as well as an on-the-fly inference alternative, with results on H-LVLMs after 5,000 pretraining steps reported in Tab. 1. We observe that using an off off-the-shelf 8B LLM without fine-tuning leads substantial degradation across all benchmarks (e.g., MME-Peroeption drops from 917.6 to 703.4), demonstrating that task-specific adaptation is critical for generating reliable importance scores. Among fine-tuned variants, scaling the reference model from 3B to 8B yields consistent improvements, with notable gains on MME- LiME-Perceetion (+93.4) and POPE (+4.0). Nonetheless, even the smaller 3B fine-tuned model provides meaningful guidance for larger LVLMs, indicating that the proposed framework is robust to scale mismatch between the reference and target models, though a scale-matched configuration remains preferable. We also compare against an on-the-fly baseline that estimates text-only predictability during training by running caption-only forward pass (with images masked) to compute importance weights, followed by an captton forward pass optimizing the reweighted loss. PRIOR with offline precomputed scores from the 8B fine-tuned LLM substantially outperforms this approach across all benchmarks, with notable improvements on MME-Perceptte (+61.6), POPE (+8.0), and SEEDBench (+3.0). The reason is that the caption-domain fine-tuned reference LLM is specifically optimized for modeling caplion distributions, yielding more reliable estimates of text-only predictability than the LVLM's masked forward pass.
3.5 Ablation Study
We conduct an ablation study to understand the influence a and & in Eq (4). While PRIOR sets a by default, experiment with 0.5, 2, to test sensitivity. Higher increases focus on optimizing loss for image-related tokens identified by the reference model. Additionally, evaluate the necessity of k. which main- tains the token loss scale, by comparing performance with and without k while fixing 1. All experiments are conducted on H-LVLMs. Results in Fig. 5 demonstrate that performance degrades a increases beyond (to 2 or 4), yet remains stable within the moderate range of 0.5-1. Furthermore, removing k negatively impacts performance on several benchmarks, validating our design choice to preserve the original loss scale. These findings suggest that a balanced approach token weighting is crucial, as excessive emphasis on imago-relatod tokens can impede the model's overall learning dynamics. Our empirical selection of and inclusion of scal- ing factor k represents optimal trade-off between focusing on visual content and maintaining stable training.

---

Published in Transactions on Machine Learning Research (03/2026)
H-LVLM
U-LVLM
5.004
Estimated do
0.028
Estimated do
0.0479.000
0L01E
2,030
0.000
9.084
0.,02
10.003
3.044
00.10
MMEEP
MME-I
MMstta
POPE
MMIlench
SE1UBench
MME-F
MNE-R
MMBlnnh SEEDBench
(Next-Token Prediction)
Figure 7: The scaling behavior comparison of NTP and PRIOR. PRIOR shows better scaling coefficients, indicating higher efficiency in translating increased resources into performance gains.
4 Scaling Laws: PRIOR Scales Predictably and Reliably with Increasing Compute
A fundamental distinction exists between LLMs and LVLMs regarding pre-training objectives. While NTP has proven to be an effective proxy for downstream task performance in LLMs (Chen et al., 2024e; Huang et al., 2024), this correlation does not extend to LVLMs (Chen al., 2024f). Specifically, optimizing LVLMs better predict caption given images through NTP does not reliably improve performance on downstream benchmarks. In this section, we conduct critical analysis of the scaling behaviors of NTP and PRIOR. We train both H-LVLMs and U-LVLMs using different amounts of data (ranging from 7M to 70M training tokens, and including sampling models), and use the following analytical form to estimate the scaling laws of NTP and PRIOR (Kaplan et al., 2020):
D. (D.)" oD
L(D)
(11)
where ap and De are constants to be estimated, D is the amounts of token used in training, and L is the predictive performance. This analytical form differs slightly from Kaplan et al. (2020) since we are largeting the benchmark performance rather than the pre-training loss. A larger ap indicates superior scaling behavior, reflecting greater performance improvement for equivalent amount of training tokens. With the fitted analylical function, we investigate two problems:
Predictability of model performance: We use the fitted function to estimate the performance of LVLMs trained on 100M tokens, and mcasure the relative prediction error:
Predictive Performance Actual Performancel
Relative Prediction Error
(12)
Actual Performance
The results are presented in Fig. 6. The downstream performance of both H-LVLMs and U-LVLMs exhibits significantly higher predictability when trained using PRIOR. This property facilitates more reliable performance estimation for larger-scale deployments, addressing a critical challenge in production-level LVLM implementation
Fig. 7 illustrates the scaling factor (i.e., un) comparison for both methods. PRIOR demonstrates better scaling properties across two LVLMs architectures, consistently achieving higher ap than NTP. This indicates that PRIOR more efficiently converts additional training data and compute into improved downstream performance.
Scaling behavior:
10

---

Published in Transactions on Machine Learning Research (03/2026)
Text Length vs. Average Importance Score
Importance Score Distribution by Token Position
temple Trest x 2.00
Maar
10-
Importance Score
Average Importance
0.7
0.5
02-
0.2
Text Length (Number of Tokens)
Token Position Percentile
Figure 8: Quantitative analysis of the importance score distribution. We find that the average importance score for each caption decreases (linearly) with the text length. Within each caption, the importance score decreases in later positions.
Image-Related Tokens Loss
Image-Unrelated Tokens Loss
Loss Differsnce
Maage-Rellted Tokens %
Purcentage
Asaring
30
13
Importance Thresholdd
Importance Threshold
Importance Threshold
Importance Throshuld
NTP (Next-Token Prediction)
PRIOR
Figure 9: The comparison of NTP and PRIOR regarding the achieved loss on image-related and unreeeedd tokens. These two token groups dynamically categorized based varying the importance threshold w/ (Eq. 3). We find that PRIOR accelerates the LVLMs training, consistently achieving lower loss on both image-related and image-unrelated tokens.
5
Further Analysis
5.1 Importance Score Distribution
PRIOR adjusts the loss for each token based on the importance score w; assigned to that token. We conduct quantitative analysis to understand the distribution of the importance score. The results are presented in Fig. 8. We examine the relation between text length (f.e., number of tokens) and the average importance score across all lokens within the caption, finding that the average importance score decreases as text length increases, with linear correlation of r=20.243 This suggests longer caption introduces more information absent from images, potentially including background content. Analysis of importance scores by token position reveals that initial tokens receive high importance scores, with scores progressively declining for later tokens. This indicates that later tokens can be more readily predicted from previous tokens without image reference.
5.2 Comparative Loss Analysis of NTP and PRIOR
We compare NTP and PRIOR from the loss perspective. By applying various importance thresholds, we catego- rize tokens into imago-related and image-unrclated subsets basod on their assigned importance scores w;. We measure the average NTP loss H-LVLMs trained via two methods on these two distinct subsets. The results in Fig. 9 reveal significant, pattern: PRIOR deliberately guides models to prioritize image-related tokens, achiev-
11

---

Published in Transactions on Machine Learning Research (03/2026)
ing consistently lower loss these information-rich elements. Interestingly, PRIOR also optimizes learning image-unrelated tokens, though this performance difference is less pronounced than for image-related tokens. This suggests that PRIOR introduces general model improvement and effectively accelerates LVLMs training overall. Furthermore, our analysis reveals a compelling trend: as we progressively increase the importance threshold, the performance gap wetween methods widens substantially. This monotonic relationship confirms that PRIOR delivers increasingly significant improvements for tokens with higher image relevance, validating its fundamental design principle of prioritizing tokens that carry the most visually related information.
6 Related Work
6.1 Large Vision-Language Models
Existing research advances the development of LVLMs capable of addressing diverse tasks via a unified interface
that can directly generate natural language, thus avoiding task-specific modifications (Wang et al., 2021; 2022; Li et al., 2023b; Agrawal el al., 2024; Luo el al., 2025). Utilizing advanced pre-trained LLMs (Brown el al., 2020; Bubeck et al., 2023; Dubey et al., 2024; Yang et al., 2024) as the language component (Liu et al., 2023d; Zhu et al., 2023), the instruction-followin and complex reasoming abilitics of LVLMs are significantly improved (Du et al., 2022; Ghosh et al., 2024). Typically, LVLMs leverage exlensive image-caption pair datasets (Lin el al., 2014; Schuhmann et al., 2021; 2022) to train a projector that maps image features into the embedding space of LLMs, thereby aligning the two modalities (Liu et al., 2023d; Zhu et al., 2023; Alayrac et al., 2022; Li et al., 2023b; Yu et al., 2023; Awadalla et al., 2024). Furthermore, large-scale vision-language instruction tuning datasets (Su et al., 2023; Wei et al., 2023; Tiu et al., 2023a; Gong et al., 2023; Gao et al., 2023; Li al., 2023a; 2025) and feedback datasets (Chen al., 2023a; Li et al., 2023c; Sun et al., 2023; Chou et al., 2024; Zhao et al., 2025) are utilized to align LVLMs with human preferences, cnsuring their ability to comprehend instructions and generate responses that are user- usee-friendly. In this work, we present simple vision- language pre-training algorithm applicable to existing datasets that enhances visual-related training outcomes.
6.2 Vision-Language Training Objective
NTP loss on language tokens serves as the predominant pre-training objective for LVLMs (Wang et al., 2022; Dai et al., 2023; Bai et al., 2023; Lu et al., 2024; Dai el al., 2024; Chen et al., 2024f). Researchers have explored complementary approaches, including but not limited to: (1) Contrastive or matching loss to align the vision-language modalities (Li et al., 2019; Lu et al., 2019; Li et al., 2022). (2) Distillation loss to expedite the training (Diao et al., 2024; Liao et al., 2025). (3) Grounding objective to more effectively align the text tokens to the corresponding image regions (Koh et al., 2023; Rasheed et al., 2024). (4) Reconstructing the image based on the pre-defined codebook or external decoders (Sun et al., 2024; Zou et al., 2023; Go et al., 2024). (5) External constrains to improve the visual perception in LVLMs (Wang et al., 2024; Luo el al., 2024a; Chen et al., 2023b). These supplementary objectives, though effective, compromise the simplicity of NTP in the original pre-training approach, complicating large-scale, efficient implementation (Shocybi et al., 2019; Rasley et al., 2020; Liang et al., 2024; Zheng et al., 2023). PRIOR prioritizes maintaining NTP simplicity by merely adding a regularization term (i.e., weighting factor) to each token during training.
6.3 Vision-Language Training Corpus
The quality of training data is a critical determinant of LVLMs' ultimate performance. The established efforts on improving the training data include but not limited to: (1) Distilling knowledge from advanced closed-source LVLMs (Chen et al., 2024a;c; Liu et al,, 2024b), like GPT-4o (Hurst et al., 2024). (2) Scaling the size of the dataset with scalable data collection pipelines (Li et al., 2024d; Awadalla et al., 2024; Chen et al., 2024d; Wang et al., 2025). (3) Incorporating the human-written filtering rules or pipelines (Guo et al., 2024; Gohari et al., 2025). (4) Curating domain-specific corpus to facilitate certain abilities in LVLMs (Li et al., 2024c; Yun et al., 2024; Fan et al., 2024; Han et al., 2024; Yang et al., 2025). (5) Relying on the self-cvolving ability in LVLMs (Liu et al., 2024a; Luo et al., 2024b). While existing work makes valuable progress toward higher-quality training data, these approaches face fundamental scalability constraints imposed by their data creation components, including the inherent. capabilities of teacher models, pre-defined
12

---

Published in Transactions on Machine Learning Research (03/2026)
filtering criteria, and established pipelines all of which create practical upper bounds on quality improvement. PRIOR avoids imposing human priors on the training dataset and remains compatible with advancements across all vision-langual datasets.
7
Conclusion
This work introduces PRIOR, advanced vision-language pre-training method that prioritizes the loss optimization on image-related tokens that a text-only reference model struggles to predict. Our experiments demonstrate that PRIOR significantly outperforms NTP, achieving 19% and 8% average relative improvement when implemented on LVLMs with and without pre-trained visual encoders, respectively. Furthermore, PRIOR exhibits more predictable and reliable scaling behaviors given increasing compute, indicating that PRIOR represents a promising pre-training algorithm.
Limitations and Broader Impacts
Limitations PRIOR requires the LVLMs and the reference LLM to share the same tokenizer, preventing the creation of a universal dataset for all LVLMs pre-training. This const.rain on the tokenizer necessitates specific implementations and processing, increasing computational overhead when deploying across multiple model architcctures. However, this tokenizer constraint is common practice in knowledge distillation and model training pipelines, and many modern model families (e.g., LLaMA series (Dubey et al., 2024), Qwen series (Yang et al., 2024)) already share tokenizers across their variants of different sizes, making this limitation less restrictive in practice. We also discuss the limitations of PRIOR' token importance heuristic in $C.
Broader impacts PRIOR'S token prioritization approach enhances I.T.ss performance across benchmarks, potentially accelerating progress building advanced general-purpose LVs Additionally, by reducing hallucination risk through better grounding of language in visual content, PRIOR contributes to developing more trustworthy LVLMs models. However, there remains the possibility of misuse, and the differential weighting approach may unintentionally amplify existing biases in training data. We encourage continued rescarch into responsible deployment practices and bias mitigation techniques alongside performance improvements.
References
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Singh Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Théophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego de Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang, and Sophia Yang. Pixtral 12b. CoRR, abs/ /bs/241.00007,, 2024. doi: 10.48550 AAR550 2410.07073.. URL https: :/dpsi:///..../ org/ 100 48550/arXXt 2410.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katic Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for fow-shot learning. arxiv preprint arXiv:2204.12 141988 2022.
Anas Awadalla, Le Xue, Oscar Lo. Manli Shu, Hannah Lee, Etash Guha, Sheng Shen, Mohamed Awadalla, Silvio Savarese, Caiming Xiong, Ran Xu, Yejin Choi, and Ludwig Schmidt. MINT-IT: scaling open-source multimodal data by 10x: A multimodal datasct with one trillion tokens. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquel, Jakub M. Tom- czak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15. 2024, 2024. URL http: /tttprr/ppprrs.... nips. cc/paper_files/ppeeeeeeeeeeeeeee wwwes//pppee//22///4/ 4099199622ffe114448//aaaaaaaa
Abssract--atassee and Benchmarks, Track. Trank html.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, 2023. doi:
ARR.I..230..220002 URL https://doi.org: 11t.///.///////.....2.5
12966.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tílio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 3558- 3568. Computer Vision Foundation / LEEE, 2021. doi:
2021.00336. URL Stcproyoooocccppaa_____a 12M BaaaaPPhhin____
attpss:/openaccess. thecvf. cwm/ Web-Scale, Image-Textt Text Pre-Training To Recognize RecognizzeLLongTTalll Visual __PR____________________ 10121_ppper_._ hhmpl...mm..
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, oditors, Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29- 19-Oerober 4, 2024, Proceedings, Part XVII, volume 15075 of Lecture Notes in Computer Science, pages 978-3-031-72643-9 22. URL https: http:///ooooog....../ ott.:///r...0000//7-------------
Springer, 2024a. doi: 10.1007/
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint Niiv:2403.20330 2024b.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Lin Bin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. SharegptAvideo: Improving vidco understanding and generation with better captions. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024c. URL http:// nips cc/paper satpaaper_ileeepppppeeee filles/paper/205 22022/222///////// _nndeettt_______eeeeeeeeeeeee Benchmarks htm..
Xiaohui Chen, Satya Narayan Shukla, Mahmoud Azab, Aashu Singh, Qifan Wang, David Yang, Sheng Peng, Hanchao Yu, Shen Yan, Xuewen Zhang, and Baosheng He. Compcap: Improving multimodal large language models with composite captions. CoRR, abs/2412.05243,, 2024d. doi: 10.48550/ARRa 2412.05243. URL https: atttps://dot ://oii. org/100 48550/ 2412.05244
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. arXiv preprint. arXiv:2311.10081, 2023a.
Yangyi Chen, Xingyao Wang, Manling Li, Derek Hoiem, and Heng Ji. Vistruct: Visual structural knowledge extraction via curriculum guided code-vision representation. arXiv preprint arXiv:2311.13258 2023b.
Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance in llms. arXiv preprint arXiv:2410.08527, 2024e.
Yangyi Chen, Xingyao Wang, Hao Peng, and Heng Ji. A single transformer for scalable lanionelanguage modeling. CoRR, abs/ 4447066488, 2024f. doi: 10.48550/ARXIV 240.066438. URL https://doi.org/// 48550/arXiv 06608.06438.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24199,,4198, 2024g.
Christopher Chou, Lisa Dunlap, Koki Mashita, Krishna Mandal, Trevor Darrell, lon Stoica, Joseph E. Gonzalez, and Wei-Lin Chiang. Visionarena: 230k real world user-vim conversations with preference labels. CoRR, abs/2411 2as/2442.08687, 2024. doi: 10.48550/ 10.48550/ARXIV..1 AAX55///AXX..21220666. URL attps://doi.org// 48550/arXiv 2412. 2412.08887..
Wenliang Dai, Junnan Li, Dongxu Li, Authony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. CoRR, 2023.
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NVLM: open frontier-class multimodal llms. CoRR, abs/2409.11 11402, 2024. doi: 10.48550/ARXIV ARXIV 2409.11402. URL ttttps:///ol wwodddi..////oooooooeoee org/10 100 48550/arXii 2409. 11402.
Mamata Das, PJA Alphonse, et al. A comparative study on (fiidf feature weighting method and its analysis using unstructured dataset. arXiv preprint arXiv:2308.04031 2023.
Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http::/ .ips. cc/paper 011.7778287755661177000666608800
html.
Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A survey of vision-language pre-trained models. In Luc De Raedt, editor, Proceedings of the Thirty-Firs International Joint Conference on Artificial Intelligence, IJCAT 2022, Vienna, Austria, 23-29 July 2022, pages 5436-5443. ijcai.org, 2022. doi: 10. 24963 (1963/IJCAL2222/
URL https://doi. http://doo.org///// wtt.oo/.////////...// 24963/1jc6iiii... http:/jjaa..22//62....
Haodong Duan, Jumming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, al. VImevalkit: An open-source toolkit for evaluating large multi-modality models. arXiv preprint arXiv:2407.11691, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodrigues, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jacwon Lec, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Ject Shah, Jelmer van der Linde, Jennifer Billock, Jemny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and al. The Ilama 3 herd of models. CoRR, abs/2407 21783, 2024. doi: 10.48550/ P..4855/AAXIIV ARXIV 2407.21783. URL https: //doi. htt////iio/////////////o 48550/ 48550/arXiv arrvv. 2407. 21783.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Wan-Cyuan Fan, Yen-Chun Chen, Mengehen Liu, Lu Yuan, and Leonid Sigal. On pre-training of multimodal language models customized for chart understanding. CoRR, /2///4407.14006 2024. doi: 10.48550/ ARXIV. 2407.14506. URL attps://doi.org// org//0 arXiv. 2407.14506.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojum Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficint visual instruction model. arXiv preprint 15X00:230.115010, 2023.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama SEE and draw with SEED tokenizer. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. SpenReview.net, net, 2024. URL https://openreviexx net/ Sarum?id--tn 2orumm7dd-MM11111000.
Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. Exploring the frontier of vision-language models: A survey of current methodologies and future directions. CoRR, abs/ abs/2204.00 /2444402244 2024. doi: 10.48550/ 0148550/ARXIV 22004.02214. URL https://doi.org 10. 001560/arxiv.244..00
Hajar Emami Gohari, Swanand Ravindra Kadhe, Syed Yousaf Shah Adam, Abdulhamid Adebayo, Praneet Adusumilli, Farhan Ahmed, Nathalie Baracaldo Angel, Santosh Borse, Yuan-Chi Chang, Xuan-Hong Dang, ct al. Gneissweb: Preparing high quality data for Ilms at scale. arXiv preprint arXiv:250..19907, 2025.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint MxXiv:2305.04790, 2023.
Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yuc. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. CoRR, abs abs/2211.0 2024. doi: (8855//ARXIV.22 22112 URL https :/tpsi///o...// weaaii..r//.ioeeeeeeeeeoo org/o0..///0//. 23650//rXiv.2412. 05237.
Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, and Quanzeng You. Infinm-webmathe Advancing multimodal pre-training for enhanced mathematical reasoning. CoRR, abs/2409. 12568, 2024. doi: 10.48550/ A1855V/ARXIV.... AW....240....... URL t/tps:///oi....
10. 48550/ 2409. 12568.
Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. arXiv preprint arXiv:2404.09933, 2024.
Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Back, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samie, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll L. Wainwright, Cary Bassin, Cary Hudson, Cascy Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnocki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, and Dane Sherburn. Gpt-40 system card. CoRR, abs/2410 22102210.2 2024. doi: 10.48550/ ARXIV 2410.21277. URL https:// //toi. org/.o..g1///. 48550/arXivv 211.22271...
Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex
16

---

Published in Transactions on Machine Learning Research (03/2026)
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint. arXiv:2001.08361 2020.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 17283 17300. PMLR, 2023. URL https::/prooceedingg wwt.mmll.peeeeee press//202/koo228233 html.
Tomasz Korbak, Ethan Perez, and Christopher I. Buckley. RI with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275 2022.
Weixian Lei, Jiacong Wang, Haochen Wang, Xiangtai Li, Jun Hao Liew, Jiashi Feng, and Zilong Huang. The scalability of simplicity: Empirical analysis of vision-language learning with a single transformer. arXiv preprint arXiv:2504. 10462, 2025.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03720 2023a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299-13308, 2024a.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sodrick Scott Kch, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37:14200- 14282, 2024b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, 1CML 2022, 17-23 July 2022, Baltimore, Maryland, USA, Pmlr, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: pre-training with frozen image encoders and large language models. CoRR, 2023b.
bootstrapping language-image
Lei Li, Zhihui Xie, Mukai Li, Shumian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.1666 2023c.
Lei Li, Yuqi Wang, Runxin Xu, Pciyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 14369 14369-14387, Association for Computational Linguistics, 2024c. doi: 10..863/11/002AAA1-1100 775. URL https: //doi. org/10. 18653/v v222//4./2222..//// acl- 775.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant bascline for vision and language. CoRR, 2019.
Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhangwei Gao, Erfci Cui, Jiashno Yu, Hao Tian, Jiasheng Zhou, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Zhenxiang Li, Pei Chu, Yi Wang, Min Dou, Changyao Tian, Xizhou Zhu, Lewei Lu, Yushi Chen, Junjun He, Zhongying Tu, Tong Lu, Yali Wang, Limin Wang, Dahua Lin, Yu Qiao, Botian Shi, Conghui He, and Jifeng Dai. Omnicorpus: A unified multimodal corpus of 10 billion-level images interleaved with text. CoRR, abs/2406 0as//240600444 2024d. doi: 10.48550/AA (0555//AXX// ARXIV 08418.. URL /ttpps://do:: org/10 10. 48550/ arxiv
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.1083 10355, 2023d.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, Nadine Chang, Karan Sapra, Amala Sanjay Deshmukh, Tuomas Rintamaki, Matthieu Le, Ilia Karmanov, Lukas Voegtle, Philipp Fischer, De-An Huang, Timo Roman, Tong Lu, José M. Álvarez, Bryan Catanzaro, Jan Kautz, Andrew Tao, Guilin Liu, and Zhiding Yu. Eagle 2: Building post-training data strategies from scratch for frontier vision-language models. CoRR, abs/ 2ass/2501.1481. 2025. 28550/arXiv arriiv 2501 14818.
doi: 10.48550/AM 148550/AAXTVV 2501.14818. URL https: /ttp://////o..o//////// 10. 48550/
Wanchao Liang, Tianyu Liu, Less Wright, Will Constable, Andrew Gu, Chien-Chin Huang, Tris Zhang, Wei Feng, Howard Huang, Junjie Wang, Sanket Purandare, Gokul Nadathur, and Stratos Idreos. Torchtitan: One-stop pytorch native solution for production ready LLM pre-training. CoRR, abs/2410.06611, 2024.
doi: 10.48550/ (15550/ARXTV/ ARXIV 2410.06511. URL https://doi. org/10 48550/arXiv arriv 2410 06511..
Bencheng Liao, Hongyuan Tao, Qian Zhang, Tianheng Cheng, Yingyue Li, Haoran Yin, Wenyu Liu, and Xinggang Wang. Multimodal mamba: Decoder-only multimodal state space model via quadratic to linear distillation. arXiv preprint arXiv:2502. 13145, 2025.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, 2014.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 10066.14.8 14565, 2023a.
Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. Examining Nmss uncertainly expression towards questions outside parametric knowledge. arXiv preprint 2XXv:2311.09733) 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec, Visual instruction tuning. CoRR, 2023d.
Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, and Junxian He. Diving into self-evolving training for multimodal reasoning, CoRR, abs abs/2412.17451, 2024a. doi: 10445500/RRXIV.2112..777 https://doi.. org/ 10. 48550/arXiv 2412 17451.
URL
Yangzhou Liu, Yue Cao, Zhangwei Gao, Weiyun Wang, Zhe Chen, Wenhai Wang, Hao Tian, Lewei Lu, Xizhou Zhu, Tong Lu, Yu Qiao, and Jifeng Dai. Mminstruct: A high-quality multi-modal instruction tuning dataset with extensive diversity. CoRR, abs/2407. 255./2087.1888, 15838, 2024b. doi: URL https://doi.orr 10. 48550/ 04407.1838..
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui Hc, Ziwci Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216 233. Springer, 2024c.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang. Kai Dong, Bo Lin, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision- language understanding. CoRR, 0es/2403305552 2024. doi: 10.48550/ARXX 2403.05525.
URL https: /tw..//oo.oooo/o org/10 100 48550/ 2402.05525.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for rision-and-langueee language tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'AlchéBBuc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 841, 2019, Vancouver, BC, Canada, pages 13 13-23, 2019. URL httpps:/proceeddnngs. Abstract .hmml.
neurips. wttppppcccppppppepeeeceeeee http//aahhhhaaaaa http://hcc.ccc./oooeaaaeaaeaaaaa
Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, Min Yang, and Binyuan Hui. DEEM: diffusion models serve as the eyes of large language models for image perception. CoRR, /2//2.21552/2 2024a. doi: 10.48550/ARXIV R.X.V.2400.15232.. URL https: / /tt.//i.oooo adw.i.or//..00... httg//0.4850/arrt... 2405. 15232.
18

---

Published in Transactions on Machine Learning Research (03/2026)
Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Xiaobo Xia, Fei Huang, Jingkuan Song, and Yongbin Li. Mmevol: Empowering multimodal large language models with evol-instruct. CoRR, 2024b. doi: 00.4550/ARRXI 2409.05840. URL https:
Run Luo, Ting-En En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, et al. Openomni: Large language models pivot zero-shot omnimodal alignment across language with real-time self-aware emotional speech synthesis. arXiv preprint arXiv:2501.045 2025.
Alberto Maria Metelli, Matteo Papini, Francesco Faccio, and Marcello Restelli. Policy optimization via importance sampling. Advances in Neural Information Processing Systems, 31, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amauda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Hanoona Abdul Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Fric P. Xing, Ming- Hisug-Hsuan Yang, and Fahad Shahbaz Khan. Glamm: Pixel grounding large multimodal model. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 13009 13018. IEEE, 2024. doi: 10.1109/CCV CBB00//CVPP22733220222 2024.01236. URL https: : //doi. org/ 10. (LL09/CVPR5277 2024. 01236.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 3505-3506. ACM, 2020. URL https://dol .ereloorg/g/ 10. 1145/3394488 34067733.
doi: 10. 10.1145/332 015/3344486.3406770
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterdd 400 million image-text pairs. arxiv preprint arXiv:2111.02114 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large- scale dataset for training next generation image-text modcls. Advances in Neural Information Processing Systems, 35:25278- 01222277222944, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, Association for Computational Linguistics, 2018.
Mohammad Shoeybi, Mostofa Patwary Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019. URL
http//aaxxvooorrrrrraaaa/a hrrg/abs/1909 08053.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction- follow them all. arXiv preprint arXiv:2305.166: 16355, 2023.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Tiu, Tiejun Huang, and Xinlong Wang- Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.nel, 2024. URL httts::/ http://wwww.e....nuueee htt.:orummmdmmmmm000000mmll
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint 1rXiv:2309.14522 2023.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Chenxin Tao, Shiqian Su, Xizhou Zhu, Chenyu Zhang, Zhe Chen, Jiawen Liu, Wenhai Wang, Lewei Lu, Gao Huang, Yu Qiao, et al. Hovle: Unleashing the power of monolithic vision-language models with holistic vision-language embedding. arXiv preprint arXiv:2412.16155, 2024.
Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge, Xiangyu Zhang, and Zhaoxiang Zhang. Reconstructive visual instruction luning. CoRR, abss/241000957. 2024. doi: 10.48550/ ARXIV 24100.0911. URL https://doi. org/10. 100 48550/arXiv. 2410.09575.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA. Pmir, 2022.
Xiao Wang, Tbrahim Alabdulmohsin, Daniel Salz, Zhe Li, Keran Rong, and Xiaohua Zhai. Scaling pre- pre-training one hundred billion data for vision language models. arXiv preprint arXiv:2502.07617, 2025.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint 1it::2108.109044 2021.
Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4: A 200-instruction paradign for fine-tuning minigpt-4. arXiv preprint Xiv::233 12067, 2023.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xic, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12: advanced multimodal understanding. CoRR, URL https: //doi. org/10 10. 48550/ 48550/arXiv.... arxiv.iiiivvviia//aaaaaaaaaeleaa 2412. 2412.10302.
vision vision-language models for 2412..000.
2024. doi:
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, (bss/2412.1111 2024. doi: AA8855/AAXIV..20 2412.11155. URL 2412. 15115.
https://doi. org/ 10.
Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, et al. Scaling text-rich image understanding via code-guided synthetic multimodal data generation. arXiv preprint arXiv:2502.14846, 2025.
Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capslusion: Rethinking image-text data at scale. In Proceedings of the IEEE/CVT Conference on Computer Vision and Pattern Recognition, pages
2024.
Tianyn Yu, Yangning Li, Jiaoyan Chen, Yinghui Li, Hai- Tao Zheng, Xi Chen, Qingbin Liu, Wenqiang Liu, Dongxiao Huang, Bei Wu, and Yexin Wang. Knowledge-augmented few-shot visual relation detection. CoRR, 2023.
Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang. Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, Timothy Baldwin, Zhengzhong Liu, Eric P. Xing. Xiaodan Liang, and Zhiqiang Shen. Web2code: A large-scale webpage-to-code datasct and eval- uation framework for multimodal llms. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Urrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024. URL http://paperss wwtppprrr..ii...oooo............ cc ct/cccccpp/oo/ccccccp/. heeppaper_ftte www.fllllpeleeeee aates//aapr//22//. haal://6cbbe22278777777812822822
and Benchmarks Track. html.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinel: Revisiting visual representations in vision-language models. In IREF Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 5579-5588. Computer Vision Foundation / IEEE, 2021. doi: D0.109//VPR446437..0 2021. 00553. URL
wtarscccssss..tt..eeeeee thecvf. www.onntenn///n211////eo 1at00021hhtm//aaama /2aang.//ang..... Visual Representations in_ Lissoon-anguage__oone LaseieeMoeells__sseee Becpilss__________ _021
2aa___aae2pr_______________ paperr html.
Tao Zhang, Xiangtai Li, Zilong Huang, Yanwei Li, Weixian Lei, Xueqing Deng, Shihao Chen, Shumping Ji, and Jiashi Feng. Pixcl-sail: Single transformer for pixel-grounded understanding, arXiv preprint arXiv:2504.10465, 2025.
Xiangyu Zhao, Shengyuan Ding, Zicheng Zhang, Haian Huang, Maosong Cao, Weiyun Wang, Jiaqi Wang, Xinyu Fang, Wenhai Wang, Guangtao Zhai, et al. Omnialign-v: Towards enhanced alignment of milms with human preference. arXiv preprint ar arXiv:2502. 18411, 2025.
Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and- language generation 2023. doi: 10.48550/ARXII 2310.02229. URL https:
via generative vokens. CoRR, 2310..2239.
//doi. wwg//o..//.oo.o. 18550/arXis 1tt.0/arrxv.221.0222222
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. CoRR, 2023.
Xucyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang. Lu Yuan, Nanyun Peng. Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language. In IEEE/CV /CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 15116-15127. IEEE, 2023. doi: 111109/CCPR522222222..00
URL https: fttps://dor //doi org/10. 1119/C/PR5200
00451.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Sensitivity study over the number SFT steps on H-LVLMs. We report performance NTP and PRIOR along with their gap (A) across six benchmarks. PRIOR consistently outperforms NTP regardless of the SFT duration, confirming that its gains stem from improved pre-training representations.
SFT Steps
Method
MME-P
MME-R
MMStar
POPE
MMBench SEEDBench
NTP
576.2
207.5
30.9
68.7
38.3
36.5
Step-20
PRIOR
917.6
220.4
33.3
75.0
40.1
45.5
A
+341.4
+12.9
+2.4
+6.3
+1.8
+9.0
NTP
871.0
224.7
32.7
68.9
42.7
42.8
Step-50
PRIOR
995.8
235.2
37.0
75.5
48.1
47.6
A
+124.8
+10.5
+4.3
+6.6
+5.4
+4.8
NTP
992.9
237.8
34.5
69.3
52.6
44.3
Step-500
PRIOR
1024.7
243.9
37.3
76.3
59.7
50.4
A
+31.8
+6.1
+2.8
+7.0
+7.1
+6.1
NTP
947.0
241.6
37.3
70.3
54.1
47.0
Step-1000
PRIOR
1037.9
254.3
38.6
77.1
60.0
54.0
A
+90.9
+12.7
+1.3
+6.8
+5.9
+7.0
A Theoretical Justification via Mutual Information
The efficacy of PRIOR can be further understood through the lens of mutual information theory. When optimizing vision-language models, we aim to maximize the prodictive power of visual information v with respect to lext l. This can be formalized by maximizing the mutual information between v and I, quantilying the uncertainty reduction about when is observed. For data sampled from the joint distribution (u, t), the mutual information is expressed as:
Iwu; () = [E.e,)) In p(u,t)
= [(ve)) In p(tlu)
(13)
p(u)p(t)
p(t)
Decomposing this expression the token level, we obtain:
I(u; = E(v,0)
Σ Inp((al,, - In ((p(4)))))
(14)
This formulation shows that mutual information is maximized when there's large discrepancy between token probability given both visual and textual context p(t,/us, txe)) versus textual context alone p(trlter)) Tokens exhibiting this difference benefit most from visual inputs.
The weighting mechanism in PRIOR, defined as = (1 -1- (L)e((t) implicitly aligns with this mutual information objective. By assigning higher importance scores to tokons that are difficult to predict from text alone, PRIOR effectively prioritizes tokens where visual information potentially provides the greatest reduction in uncertainty. This approach creates natural emphasis on tokens where p(t/// u, bee)) - In p(t:) likely be large, thus indirectly promoting higher mutual information between vision and language representations.
This intuitively corresponds to focusing loss optimization the tokens where the LVLMs have the greatest opportunity to outperform text-only predictions by leveraging visual information. In essence, PRIOR adaptively modulates the learning signal based on the potential information gain from incorporating visual context, leading to more efficient and effective vision-language pre-training.
Controlled Sensitivity Study over the Number of SFT Steps
In the main experiments (53), all pre-trained checkpoints are evaluated after lightweight 20-step instruction fine-tuning (SFT) stage. A natural question arises: does PRIOR's advantage originate from the pre-training
22

---

Published in Transactions on Machine Learning Research (03/2026)
objective itself, or is it amplified by this particular fine-tuning regime? To answer this, conduct controlled sensitivity study by varying the number of SFT steps E {20, 50, 500, 1000) while keeping the SFT data, optimizer, and evaluation protocol fixed. All SFT runs start from the fully converged 5,000-step pre-trained H-LVLMs checkpoints. The results are presented in Tab. 2. We have the following observations: (1) The PRIOR-NO NTP gap persists across all SFT regimes. Across all six benchmarks and all four SFT configurations, PRIOR consistently outperforms NTP. This demonstrates that PRIOR's advantage is not an artifact of a particular fine-tuning duration. (2) PRIOR maintains advantages even with extended SFT. At Step-1000, (c.g., +90.9 on MME-Perception +6.8 on MMBench, +7.0 SEEDBench), confirming that the gains are not merely amplified by limited fine-tuning but reflect gemuinely improved representations learned during pre-training.
PRIOR still outperforms NTP on all benchmarks
C Limitations of the Token Importance Heuristic
While PRIOR's text-only probability heuristic effectively identifies image-related tokens in the majority of cases, we acknowledge scenarios where low text-only probability does not correspond to visual relevance. We provide a qualitative analysis of such failure cases to better characterize the limitations of our approach.
Rare Proper Nouns and Specific Identifiers Consider the following caption from our curated study datasct: "A hand-painted ceramic vase crafted by artisan Mikhail Vorontsov, displayed at the Hermitage Museum.. In this case, visually verifiable tokens like "hand-painted" and "ceramic" receive moderate importance scores (e.g., w, 0.65 and 0.55, respectively). appropriately reflecting their image relevance. However, the artisan's first name "Mikhail" and the specific museum "Hermitage" receive very high importance scores (w; = 0.98 and 0.92) because they are rare and difficult to predict from textual context alone. These tokens represent metadata or provenance information that cannot be verified from the image, yet. they receive disproportionately high weights in the PRIOR loss.
Invisible Attributes and Non- Visual Metadata A related failure mode arises with attribules that are fundamentally non-visual. For example, given the caption "An authentic antique mahogany chair, appraised at $47,500 and previously owned by aristocrats," " the token "aristocrats" receives high importance score (w: = 0.91) due to ils unpredictability from context. Similarly, "authentic" (w; = 0.62) describes a property that requires expert verification rather than visual inspection. While visually informative tokens like "mahogany" (w; 0.85) are also appropriately upweighted, the heuristic cannot distinguish between tokens that are hard to predict because they describe visual content versus those that are hard to predict because they encode non-visual metadata.
Numerical Specificity Exact quantities such as prices, dates, and measurements are inherently unpre- dictable from preceding text, leading to high importance scores. For instance, a price tag like "$365,500" specific address like "7338 24th Ave NE" in real estate caption will receive elevated importance scores despite being metadata that cannot be inferred from the associated image.
Mitigating Factors Despite these cases, several aspects of PRIOR's design help mitigate their impact. First, the normalization term wt/ E wy bounds the maximum weight any single token can receive, preventing rare entities from dominating the loss. Second, these failure cases represent a relatively small fraction of all high-importance tokens in practice- the majority of tokens with low text-only probability genuinely correspond to visual content, as evidenced by the consistent performance improvements across all benchmarks.
23


Reviews:
[{"review_id": "CXkmNwo5AZ", "reviewer": "Reviewer_Ez6s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes PRIOR, a simple pre-training scheme for LVLMs that uses a caption-only reference LLM to estimate token probabilities and assign offline importance weights to reweight the standard NTP loss so that image-related tokens are prioritized. Experiments on both H-LVLMs and U-LVLMs show that PRIOR yields consistent gains over NTP with smoother training and more predictable scaling. It also outperforms ITM/contrastive/reconstructive and heuristic weighting baselines.\n\nStrength:\n1. The paper is well-written and easy to follow.\n2. The proposed method is very intuitive, simple yet effective.\n3. The theoretical perspective regarding informative mutual information is interesting.\n4. The experiments are comprehensive. The results show significant performance gain across architectures and diverse benchmarks\n\nWeaknesses:\n1. The proposed method requires training a separate LLM to guide the VLM training, which is computationally expensive and requires additional training data to avoid overfitting.\n2. Some experiments are missing. See requested changes for more details.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The numerical results  show broad, consistent gains across multiple benchmarks (MME-P/R, MMStar, POPE, MMBench, SEEDBench) and two LVLM settings. The authors also provide interesting theoretic perspective to illustrate the idea.", "audience_yn": "Yes", "audience_explanation": "PRIOR offers a simple, drop-in modification to NTP with clear gains across benchmarks, improved stability, and better scaling predictability, which are practical concerns for large LVLM training.", "requested_changes": "1. LLM ablation is insufficient. Please clarify the reference LLM requirements: how much caption-only data is needed, how performance scales with LLM size, and whether an off-the-shelf pretrained LLM (no fine-tuning) suffices. Also test mismatched scales (e.g., a small 3B reference LLM guiding a larger LVLM).\n2. Since PRIOR leverages extra data to train the reference LLM, it is necessary to compare against an NTP baseline where the LVLM is trained on the same total data budget (i.e., include the reference-LLM data in the LVLM’s training) to control for data advantages.\n3. For the CLIP-based weighting baseline, the authors should fine-tune CLIP on the same training data used by PRIOR to ensure a fair comparison of methods under matched adaptation.\n4. It would be interesting to provide attention visualizations over image tokens (or cross-attention maps) to show whether PRIOR improves vision–language alignment qualitatively.\n5. Since PRIOR emphasizes image-related tokens over other conjunctions, the authors should assess whether language fluency degrades. E.g., including open-ended VQA or free-form benchmarks (e.g., LLaVA-Bench) and language quality metrics to evaluate coherence.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QYTipHt9R8", "reviewer": "Reviewer_qJ9q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper identifies a fundamental limitation of the standard next-token prediction (NTP) objective in vision–language pre-training: most caption tokens are weakly or entirely unrelated to visual content, causing LVLMs to over-optimize language-only signals and undermining visual grounding. To address this issue, the authors propose PRIOR, a lightweight modification to the standard next-token prediction (NTP) objective for vision–language pre-training. The key idea is to prioritize image-related tokens by reweighting the token-level NTP loss using probabilities from a text-only reference language model. Tokens that are difficult to predict without visual input receive higher loss weights, encouraging the model to rely more strongly on visual information rather than spurious textual correlations.\n\nStrengths:\n\n* It provides a lightweight yet effective mechanism for better aligning visual representations with language generation by reshaping the optimization focus toward genuinely vision-dependent tokens.\n* It modifies only the loss computation and does not require additional training objectives, architectural changes or online teacher inference. This makes the method easy to integrate into existing large-scale pre-training pipelines, which is particularly appealing in LVLM.\n\nWeakness:\nThe core assumption is that low probability under a text-only LLM implies visual relevance, which is reasonable but not universally true.\nBecause some tokens may be rare or context-specific in text but still weakly grounded in the image, while some visually grounded tokens may be common in text. While empirical results mitigate this concern, the paper could benefit from a deeper analysis of failure cases. The current paper does not provide qualitative or quantitative analysis of scenarios where the importance scores may be misleading (e.g., visually irrelevant but textually surprising tokens), nor does it analyze how such cases affect downstream behavior.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. The main claims of the paper are generally supported by accurate, convincing, and clearly presented evidence.\nThe authors claim that prioritizing image-related tokens during vision–language pre-training leads to improved performance, better training stability, and more predictable scaling behavior compared to standard next-token prediction (NTP).\n\nThese claims made by authors are substantiated through a carefully designed experimental evaluation. The authors validate PRIOR across two substantially different LVLM architectures: (1) heterogeneous LVLMs with frozen visual encoders and (2) unified end-to-end LVLMs, demonstrating consistent gains in both settings. The improvements are reported on a diverse set of established benchmarks such as POPE, MMBench, SEEDBench covering perception, reasoning, and hallucination, which strengthens the generality of the conclusions.\nImportantly, the paper goes beyond reporting final performance numbers. The training dynamics are analyzed in detail, showing reduced performance fluctuations and sustained improvement throughout training, which convincingly supports the claim of improved stability.\nThe proposed method is also supported by ablation studies on key design choices (e.g., the weighting exponent α and the loss scaling factor k), as well as comparisons with alternative token-weighting baselines, which helps isolate the effect of the proposed token prioritization mechanism.\nThe scaling law analysis further supports the claim that PRIOR exhibits better predictability and efficiency when increasing training data or compute, which is a non-trivial and well-motivated evaluation for large-scale vision–language pre-training. While some assumptions (e.g., using text-only surprisal as a proxy for visual relevance) could benefit from deeper theoretical or empirical validation, the experimental evidence provided is overall coherent and convincing.", "audience_yn": "Yes", "audience_explanation": "Yes. The findings of this paper are likely to be of interest to a broad segment of TMLR’s audience.\n\nThe paper addresses a fundamental and timely problem in vision–language model pre-training: the mismatch between token-level training objectives and actual visual grounding in noisy, web-scale image–caption datasets. This issue is directly relevant to researchers working on multimodal learning, large language models, training objectives, and scaling laws. The proposed method introduces a principled modification to the standard next-token prediction objective. Moreover, the work appeals to readers interested in scalable and architecture-agnostic methods, as PRIOR operates purely at the loss level without introducing additional supervision, model components, or complex training stages.", "requested_changes": "* (Critical) Provide a deeper qualitative analysis of failure cases where low text-only probability does not correspond to visual relevance, to better characterize the limitations of the token importance heuristic.\n\n* (Critical) Clarify the role of the scaling factor k in the loss formulation. The loss function multiplies the normalized token weights by a factor k, but the rationale for explicitly introducing k is only briefly mentioned as “maintaining the loss scale.” A more formal explanation is needed. Whether k is theoretically motivated or purely empirical. While an ablation study is provided, a clearer conceptual explanation would significantly improve interpretability and reviewer confidence in the method’s design.\n\n* (Non-critical) In the H-LVLM setup, the paper adopts the original CLIP ViT-L/336 vision encoder. Given the availability of more recent and stronger variants such as OpenCLIP/ SigLIP trained on larger and more diverse datasets, it would be useful to clarify the rationale behind this choice. In particular, the authors could discuss whether the use of an older vision encoder was motivated by fairness, reproducibility, and whether PRIOR is expected to yield similar gains when paired with more advanced vision encoders. Addressing this point would improve the generality and practical relevance of the experimental evaluation.\n\n* (Non-critical) Clarify the computational and storage overhead introduced by offline importance score computation, especially at very large dataset scales.", "broader_impact_concerns": "The paper does not raise significant new ethical concerns beyond those commonly associated with large-scale vision–language models. While prioritizing certain tokens during training could potentially amplify biases present in the underlying data, this risk is acknowledged at a high level and does not appear unique to the proposed method.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qy7KJvL8WA", "reviewer": "Reviewer_wW5f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper argues that standard next-token prediction (NTP) for vision-language pre-training overfits to caption tokens that are weakly related to the paired image.\nTo address this problem, authors propose PRIOR, a token-reweighting variant of NTP: a text-only reference LLM is trained on captions without images, and each caption token is assigned an importance score based on the reference probability, yielding a normalized reweighted NTP loss. \n\nTo verify the effectiveness of PRIOR, authors perform experiments in two LVLM settings, i.e., (a) H-LVLMs with a pretrained vision encoder where only an MLP connector is trained, and U-LVLMs with unified architectures. Experimental results suggest that PRIOR improves average benchmark performance by 18.61% for H-LVLMs and 7.93% for U-LVLMs, and displays better training stability. \nFinally, the paper presents a scaling-law analysis and claims that PRIOR shows improved predictability and better scaling coefficients compared to NTP.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The effectiveness of core contribution in the paper, PRIOR, i.e., token reweighting based on text-only predictability,  is supported by a clearly specified loss with equations and implementation details.\nPRIOR also demonstrates consistent improvements across multiple benchmarks underder two LVLM scenarios, including reported average relative gains at convergence and improved stability claims. \nThe paper also provides a reasonable baseline comparison in the H-LVLM setting.\nHowever, some claims would be more convincing with more comprehensive ablation studies and broader-scale validation beyond the current training regime.", "audience_yn": "Yes", "audience_explanation": "The paper targets a central and widely used LVLM NTP and proposes an effective alternative that is easy to integrate while reporting meaningful improvements and improved scaling predictability.", "requested_changes": "1. Since all checkpoints are evaluated only after a lightweight 20-step instruction fine-tuning stage, it is currently unclear whether PRIOR’s gains originate from the pre-training objective itself or are amplified/attenuated by this post-hoc alignment procedure. Please include a controlled sensitivity study over the number of SFT steps (e.g., 0, 20, 100, 500, 2k), keeping the SFT data, optimizer, and evaluation protocol fixed, and report both absolute performance and the PRIOR–NTP gap as a function of SFT steps. Ideally, repeat key points with multiple SFT seeds and/or explicitly report the delta induced by SFT for each method. This would substantially strengthen the causal interpretation that PRIOR improves representations learned during pre-training rather than benefiting disproportionately from a particular fine-tuning regime.\n\n2. For H-LVLMs, only the MLP is trained. Please add at least one setting where more parameters are trained (e.g., partial LLM unfreezing, LoRA, or end-to-end) to establish that PRIOR remains beneficial when the model has more freedom to adapt.\n\n3. The approach involves training a reference LLM and precomputing per-token probabilities; please provide compute estimates (GPU hours, storage footprint for token-prob tuples, preprocessing time) so readers can judge cost/benefit.\n\n4. The current method relies on training a separate caption-domain reference LLM and precomputing token-level importance scores offline, which is presented as a pragmatic way to avoid reference inference during LVLM training. Explore how PRIOR depends on the reference LLM (size, training duration, or using an off-the-shelf LLM vs caption-domain trained) is an important yet missing baseline. \nEspecially, it remains unclear whether the performance gains fundamentally require a separately trained reference model, or whether a simpler on-the-fly estimate of text-only predictability would suffice. I recommend adding an online self-referenced baseline: during LVLM training, for each batch, run: \n   - a caption-only forward pass (image removed/masked) to obtain $p_{\\text{text}}(t_i \\mid t_{<i})$ and compute weights $w_i$ (with stop-gradient/detach through $w_i$);\n   - an image+caption forward pass to optimize the reweighted NTP loss using those weights.\n\n   Reporting both accuracy/hallucination metrics and the additional compute cost would clarify whether the offline proxy/reference training is necessary or primarily an engineering choice.", "broader_impact_concerns": "n/a", "additional_comments": "n/a", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)