Demystifying Supervision Data Generalization in Multimodal LMs
Authors: Xuan Qi
Luxi He
Dan Roth
Xingyu Fu
Venue: ICLR 2026 Poster (2026)
Keywords: MLLM
Primary area: generative models

Abstract: Conventional wisdom in selecting supervision data for multimodal large language models (MLLMs) is to prioritize datasets that are intuitively similar to the target task (e.g. text-rich v.s. vision-centric). However, it remains unclear how reliably such similarity translates into improved performance on the test benchmarks. 
In this paper, we take the first step to study the problem in MLLMs: can we predict a training data's influence on a target benchmark even before any training takes place?
To answer this question, we first conduct an in-depth analysis using 14 vision-language datasets covering 7 diverse tasks. Our analysis shows that intuitive task
similarity is unreliable in predicting task generalizability, and that transfer depends on the specific dataset rather than the broader task category. 
We propose DATAPROPHET, a training-free, simple yet effective metric based on multimodal perplexity, similarity, and data diversity. Our experiments demonstrate that the influence rankings for different supervision datasets derived from DATAPROPHET is strongly-correlated with rankings based on the actual performance increase after training, with a Kendall’s $\tau$ correlation coefficient of 86.0\%. Moreover, we show that DATAPROPHET can help select better supervision data, achieving up to 6.9\% improvement in average over uniform selection, 1.4\% over SoTA training-based baseline, and 0.2\% higher than oracle experiment performance-based selection. Our code and data will be released.

Published as a conference paper at ICLR 2026
MN 600
DATAPROPHET: DEMYSTIFYING SUPERVISION
DATA GENERALIZATION IN MULTIMODAL LLMS
Xuan Q
Luxi He
Dan Roth"
Xingyu Fu
University Pennsylvania
"Tsinghua University "Princeton University
Website: http:://ddc..ppchheee....gtiabbb .i//
Code
Dataset
ABSTRACT
Conventional wisdom in selecting supervision data for multimodal large language models (MLLMs) is to prioritize datasets that are intuitively similar the target task (e.g. text-rich v.s. vision-centric). However, it remains unclear how reliably such similarity translates into improved performance this paper, we take the first step to study the problem in MLLMs: can we pre- dict training data's influence takes place? To answer this question, we first conduct an in-depth analysis us- ing 14 vision-language datasets covering diverse tasks. Our analysís shows that intuitive task similarity is unreliable in predicting task generalizability, and that transfer depends on the specific dataset rather than the broader task category. We propose DATAPROPHET, a training free, simple yet effective metric based on multimodal perplexity, similarity, and data diversity. Our experiments demonstrate that the influence rankings for different supervision datasets derived from DAT- APROPHET is strongly-correlated with rankings based on the actual performance increase after training, with a Kendall's correlation coefficient of 86.0%. More- over, we show that DATAPROPHET can help select better supervision data, achiev- ing up 6.9% improvement in average over uniform selection, 1.4% over SoTA training-basse -baedd baseline, and 0.2% higher than oracle experiment performance- based selection. Our code and data will be released.
the test benchmarks. In
target benchmark even before any training
INTRODUCTION
Training data one of the most important deciding factors for Multimodal Large Language Model (MLLM) performance (Li et al., 2023; Albalak al., 2024; Sachdeva et al., 2024; Bai et al., 2023; Wang et al., 2024b; Bai et al., 2025; Zhu et al., 2025; OpenAI, 2023; Team et al., 2023). Given the vast amount of multimodal supervision datasets, how should effectively utilize them towards certain training targets? Some prior work has explored using high-quality data and similar tasks' data during MLLM supervised fine-tuning stage (Xia et al., 2024; Wu al., 2025). However, these methods still require some training and mainly improve supervision performance on target tasks by removing irrelevant, redundant, or low-quality data from the training mixture. this work, we further demystify how training data affects a test benchmark's performance. We pose the following research question in an MLLM setting:
Given a training dataset D., can we predict the influence (measured by relative performance change) on target benchmark T, even before any training takes place?
To answer this question, we first conduct a comprehensive experiment utilizing 14 vision-language fine-tuning datasets covering 7 diverse tasks (2 for each task): OCR, chart understanding, spatial reasoning, counting, kmowledge-based QA, document understanding, and map understanding. We
"This work was completed during Xuan Qi's summer research internship at the University of Pennsylvania. Correspond to <Xuan Qi: qi-2220ma 1 tsi aaninghaaa... edu. cn?, <xingyu Fu: ngyufu@priiin wwt.:/irrrrinntteeeeee..

---

Published as a conference paper at ICLR 2026
Intuition: (s) Similar task data help similar task more X (b) Similar Influence for some task X
(c) Selecting better data needs training sigrale X
Tost
Multimodal LIM
OOKYOA
9.50
6:24
0:51
texining
OCR
Sosen QA
5:01
6:13
13.86
7.20
Cherie Chansieed
Open-Spanall
GLEVR
DCK
improves which morc
Spotial
Inter relative improvement after trainin
Calcduso
Not awaysi OCR help Spatial
Chart
Darioetslovel influence
Train-free free DataPhophet previous data selection E
Figure 1: Three major takeaways in similarity between training data and test data turns out to be a poor indicator of model performance influence. In contrast, our designed DATAPROPHET metric more reliably predicts the influence of training data on test benchmarks; (b) the impact of multimodal supervision is decided by specific individual datasets, rather than by broad task categories: datasets from the same task category do not necessarily help each other the most, and do not share similar influence on the same target bench- mark. Here, OCR-VQA (Mishra et al., 2019) and Screen- QA (Hsiao et al., 2022) are both OCR data, ChartQA (Masry et al., 2022) and Chart2Text (Kantharaj et al., 2022) are both chart question answering benchmarks, and Open-Spatial Cheng et al. (2024) and CLEVR spatial (Johnson al., 2017) are both spatial reasoning benchmarks: (c) DATA PROPPET provides an effective approach for training-free data selection under fixed compute budgets fi.e.fixed number of total training samples). We compute influence of each supervision dataset based on a combined set of lesting benchmarks and select data according to the influence ratio, yielding consistent improvements across 14 tasks, with average gains of +3.4% and +6.9% under real and synthetic data settings.
(a) Surprisingly, human intuition about
surprisingly find that different training datasets' influence on a target benchmark may contradict human intuition. For example, humans may intuitively assume that training the model OCR task data will help its performance on chart tasks more than spatial reasoning tasks, since OCR and chart both require extracting text and numbers an image. However, experiment results as in Figure 1 (a) and (b) indicate the reverse: training on OCR improves spatial reasoning tasks more than chart tasks. This counter-intuition finding suggests that model's generalization across different tasks depends on more than surface-level similarity. We also find that performance improvement are not decided by task category but are dependent on individual datasets: datasets from the same task category do not necessarily help each other the most, and do not share same influence on the same target benchmark.
Based on our analysis, we extract factors that truly matter for predicting such influence. We in- troduce PATAPROPHET, and visual similarity, multimodal perplexity, and data diversity to predict the data influence (Equa- tion (3)). We evaluate the ranking of 14 training datasets' influence compared to ranking using real training effect, measured by the benchmark's 's relative improvement over baseline. Our experiments show strong correlation between the two rankings (86.0% Kendall's r), demonstrating the useful- ness of PROPHET in predicting data influence rankings in scale. Notably, the most important factors for prediction are multimodal perplexity and visual similarity, increasing 37.3% and 23.5% % of Kendall's r respectively.
extremely simple yet effective, training-free metric which uses text
In addition to predicting source data usefulness, we also show that given target benchmarks, DAT- APROPHET can be applied to curate good instruction-tuning data under fixed compute and data budget. We conduct experiments under two different settings: selecting instruction-tuning data from (1) real data pool and (2) synthetic data pool. For real data, PRTAPROPHET performs 3.4% better than uniform selection and is 1.4% better than ICONS (Wu et al., 2025), a SoTA selection requir- ing training. We observe even larger performance gain in synthetic data selection, reaching almost 6.9% performance increase compared to the uniform selection baseline and 1.2% improvement over ICONS. These show promise for DATAPROPHET as a light-weight yet effective measure for choos- ing supervision data for MLLMs.
Overall, this work performs a deep analysis of different source datasets' influence on target bench- marks' performance, with counter- intuitive findings. Our analysis motivates DATAPROPHET, a sim- ple training-free yet effective metric to predict the usefulness of different training datasets on the target. We further demonstrate the promising effect of DATAPROPHET to guide MLLM instruction- tuning data selection that outperforms SoTA training-based selection.
2

---

Published as a conference paper at ICLR 2026
2 DATA INFLUENCE ANALYSIS
In this section, we demonstrate how we quantitatively analyze data influence across multiple visual question answering (VQA) tasks. Specifically, first select base Multimodal Large Language Model (MLLM): VL3 (Zhu et al., 2025), and 14 diverse VQA datasets that comes with both train (source) and test (target) sets. Then, conduct supervised fine-tuning with each source dataset individually, and evaluate relative performance gains on all target sets. The goal is demystify data influence from real experiment results. We will expand this section by introducing the experiment setup, followed with results and observations.
base model
2.1 ANALYSIS SETUPS
Benchmark and Task Selection. We try to include as diverse benchmarks as possible and mainly select three types of tasks: text-rich tasks such as OCR, chart, and document; general VQA tasks; and vision-centric tasks such as spatial reasoning, counting, and map understanding. This ends in 14 vision-language datasets spanning seven task families (2 datasets for each task), as detailed below:
OCR: OCR-VQA (~1002K QA pairs) (Mishra et al., 2019); ScreenQA (~86K QA pairs) (Hsiao et al., 2022). This task requires extracting specific text from text-rich im- ages and reasoning about its context.
Chart understanding: ChartQA (~33K QA pairs) (Masty et al., 2022); Chart2Text (~44K QA pairs) (Kantharaj et al., 2022). Models must interpret chart data, identify trends, and answer questions or generate textual descriptions of charts.
Document understanding: DocVQA (~50K QA pairs) (Mathew et al., 2021); Sujet- Finance-QA QA (~107K QA pairs) (Sujet AI, 2024). This task requires extracting relevant information from structured and unstructured text in documents to answer questions.
General VQA: A-OKVQA (~~2.99K QA pairs) (Schwenk et al., 2022); VC-GVQA ex- tracted from Visual-CoT (~438K QA pairs) (Shao al., 2024). This task focuses on general domain question-answering over natural images.
Spatial reasoning: Open-Spatial (~8.7MM 78. QA pairs) (Cheng et al., 2024): CLEVR- Relation (~160K QA pairs) (Johnson et al., 2017). This task tests model's ability to under- stand spatial relationships between objects, and perform relational reasoning.
Counting: CLLEVR-Counting (~ 160K QA pairs) (Johnson et al., 2017); TallyQA (~287K QA pairs) (Acharya et al., 2019). This task requires accurately counting objects in images identifying quantities based visual cues.
Map reasoning: Geom Verse (~~23K QA pairs) (Kazemi et al.. 2023); MapQA (~800K QA pairs) (Chang et al., 2022). This task focuses on understanding maps, spatial navigation, and reasoning about geographic or structural layouts.
Unless otherwise noted, we use the officially released splits or collections. For CLEVR-based subsets (CLEVR-Relatee and (LEEVR-Coutting) we follow the question-family filters described in Johnson et al. (2017) and train/evaluate on their corresponding target sets.
Experiment Setup. For each benchmark D, we uniformly sample 20k examples from their train- ing splits for supervised fine-tuning, and 1 from test splits for evaluation. We denote the training (source) data as and testing (target) data t. All experiments are conducted with fixed compute setting same number of training data (20K), same base model, and same hyper-parameter choices (optimizer, scheduler, epoch count, and batch size). The base model 2B (Zhu et al., 2025). The implementation details can be found in subsection A.2.
here is Intern VL.3-
Relative Performance Gain. Here, we introduce how measure the data inffuence in real periment results. Let A, denote the base model's performance score on test set / (e.g., accuracy). and A the score after fine-tuning source dataset 8. We quantify the source-to-target influence via
A Similarly, define A,,, Ai-A, where A, is the base A
relative improvement A.-t
score on s and A is the score on after fine-tuning on t.
hhttp::/huuggi www.hngggggggaaooooo wttp:aaa.....cc....aaaaaaa 1tt.eenvviiaa//eeeei....--3 lntp///.nnenn//.3222..
3

---

Published as a conference paper at ICLR 2026
OCR-VQA
19:25
17.00
9.50
10.35
15:25
3.07
0.87
9.55
17:17
21
13.50
OCR
50
ScreanQA
ChartQA
6:37
3.03
24
12.16
14
95
2.02
5.24
16:40
6.99
7.14
10.76
4.62
3.42
9.63
40
Chart
Chart2Text
9.02
17.42
4.07
1.63
10.39
30
DocVQA
Document
Rolative Improvement (%)
Train Dataset
Finance-QA
24.24
22:47
5.75
8.58
23.41
52.12
4.03
1.87
21:18
17.52
14.03
18:02
10.87
5.46
11.56
A-OKVQA
2.48
5.57
5.24
20.00
26.24
7.32
2.33
General
VC-GVQA
9.55
7.64
open-spatial
11.70
10.22
13.45
24.24
15.02
12.02
9.24
0.27
53.24
22.53
13.93
27:24
15.75
4.42
56.22
-10
Spatial
0.32
3.29
15.00
2.35
1:34
21.05
15.00
CLEVR(C)
8.51
12.77
14.35
10.66
19.35
13.28
15.35
19.75
17.54
20
5.60
16:44
Counting
TailyQA
6:7
0.22
2.29
9.49
9.22
2.21
1.18
1.28
13.50
22.69
21:03
+0.02
12.77
2.35
23.24
7.68
Map
MapQA
5.2
4.50
9.18
100
11:02
3.22
0.29
25
21.97
9:35
Average
15.00
1124
1300
20.3
15
1450
7.54
338
17.32
22.00
24.77
15:33
10
5.87
ScreenOA OCR-VQA
ChartQA
Chart2Text
DoCVGA
Finance nen- QA
O-OKYAA
VC-GV0A
open- open-spatial
CLEVR(R)
CLEVR(C)
TallygA
GoamVerse
MapdA
Average
Test Dataset
Figure 2: Data influence analysis under fixed compute. We conduct supervised fine-tuning on base model (InternVL. (Zhu et al., 2025)) with each training (source) dataset individually, and evaluate relative performance gains all test (target) sets. See details in Section 2. We highlight several observations: (1) Data influence is not symmetric. (2) Data sources from the same task type do not always influence each other most. E.g., OCR-VOR VQA improves ScreenQA (OCR task) by 17.88% vs. Geom Verse (map understanding task) by 21.74% (3) Data influence is not decided by task type, but by the individual dataset. E.g., text-rich tasks (ScreenQA) influence vision-cent centric ones POpen-Spatial with gains of 12.86%) more than text-rich ones (ChartQA with gains of 5.61 %).
2.2 ANALYSIS RESULTS
We finetune our base model with each source dataset individually, and evaluate relative performance gains on all target sets, to demystify data influence through controlled real experiment results. We report the full experiment results as in Figure We highlight the below observations:
Data influence is not symmetric. The diagonal gains are still the largest, which align with intuition that data coming from the same distribution is still the most influential. However, we find that the data influence is not symmetric, i.e. Ag-xt * A For instance, the influence of Open- Spatial on DoA (train on Open-Spatial and test on DocVQA) is 15.92% while the reverse is high 26.14%
Data sources from the same task type do not always influence each other most. Data influence decided by task type, but by the individual dataset. For instance, after training on OCR-VQA data, the relative performance gain achieved on ScreenQA (which is also OCR) is 17.88%, lower than that achieved on Geom (which is on map understanding) 21.74%.
Text-rich tasks can influence vision-centric ones more than text-rich ones. As demonstrated in Figures and 2, after training on OCR data (ScreenQA), the relative performance gain model achieves on chart tasks 5.61% for ChartQA and 6.13% for Chart2Text is lower than that on spatial reasoning tasks 12. for Open-Spatial and 7.96% for CLEVR-Relation).
DATAPROPHET: DEMYSTIFY DATA INFLUENCE
Section 2 shows that intuitive similarity is unreliable in predicting task generalizability. In this section, investigate what the true underlying factors for predicting data influence could be. Our
4

---

Published as a conference paper at ICLR 2026
goal is to identify reliable data influence predictor based on these factors. We begin with defining a evaluation for such dataset influence prediction metrics ($3.1). We then introduce details of our metric ($3.2). Our metric is motivated by some heuristic functions that people have studied in Large Language Models (LLMs)) research, such as data perplexity, task difficulty, and answer length (Marion al., 2023; Xie et al., 2023b; Wang et al., 2024a), and try study their impact in multimodal settings. Building on these heuristics, we identify a training-free metric for MLLMs that reliably predicts the actual data influence results, not only for A but also for At-s.. We conclude the section with results of our metric and analysis including different components' effect, and ineffective but heuristic ones that we explored($3.3)
3.1 PREDICTOR EVALUATION SETUP
Considering the asymmetric nature of data influence discussed previously, good data influence predictor should be one that (1) produces valuable insights on how different datasets compare in contributing to some target task, and (2) offers understanding on how one dataset could affect various downstream targets differently. The ground-truth for these can be approximated by the actual relative performance gain after finetuning and evaluation, following the setup Section 2.
Two Evaluation protocol. For each pair of (source dataset target data t), we compute the influence prediction score between them using the data influence predictor. Since we have 14 datasets, get 256 (14x14; scores in total. Given one target dataset, rank all 14 source datasets using the influence prediction score (estimate of training data's helpfulness), and compare that with the real ranking using the relative improvement scores in Figure 2 (true helpfulness). To quantify the correlation between two rankings, we deploy Kendall's т. We denote the correlation as Te for measuring A, and the average across all target datasets is TTg--
Conversely, given one source 8 dataset, we rank all target datasets based on the influence score between them. Then we similarly compare with the real rankings as in Figure 2. We denote the Kendall's correlation as Ts in this setting for measuring A. and the average across all source datasets Sse- We use
TTp Tte
as the final evaluation metric for influence predictor. Here, the two-way protocol allows for a more comprehensive evaluation of the data influence predictor itself. However, when using the influence predictor to select better training data as in Section 4, we only need to select by Figt-
3.2 THE PROAPEO METRIC
There are three major components in our prediction metric, building on previous work in LLM- setting and incorporating MLLM-specific considerations:
Multimodal perplexity. Previous studies (Marion al., 2023: Thrush et al., 2024a) have explored to use perplexity as data quality estimator in language model pretraining, ranking and pruning pre- training corpora accordingly to achieve better model performance. Especially Thrush al. (2024a) points out the timodal perplexity is the best factor for this predictor. We hypothesize that intrinsically more chal- lenging source data can provide greater value for enhancing model capability. We define multimodal perplexity as below:
-eeeeiitt--eehmmakkk correlations in LLMs. Inspired by them, we examine if mul-
Let A (a1...., oT) be the answer tokens, and pe the base MLLM. We define multimodal per- plexity on a dataset as
PPL(D) = exp( B(IQ.A))D Ea-TT) log po(e 1, TQ()), az))
We evaluate the data influence rankings using only multi-modal perplexity as predictor, and find that the resulting Tıg is 0.274, which is not high enough, indicating that additional components needed for a better predictor.
Cross-dataset Similarity. If the source dataset contains images and text similar to the target in the model's 's embedding space. then heuristically it should more closely resemble learning in the target domain's distribution. We measure alignment along questions, answers, and images using the output
5

---

Published as a conference paper at ICLR 2026
OCR
Chart
Spatial
Counting
General YQA
Document
Math
Axg.
ChanQA: 0911
Opeai-Spatal: 0.736
CLEVRIC): 0.846
A-OKVQA:
DocVQA: 0.390
Geam Verse:
7hp.
ScreentJA: 0.869
CharzText 0.869
CLEVRIR); 0,824
TailyAA 0,822
VU-GVQA: 0.542
PinancelAA 0,590
MapQA:: 0,953
ACCR-VQA08840
ChariQ: 0 868
Open- Spacia:: 0,822
CLEVR(C): 0.869
A-OKVOA: 0.846
Do<VQA: 0 0,568
GeomVere: 0.846
SercorQA: 0911
ChanZTest 0822
CLEVR(R):
TallyQA
VC-GVQQ:
FinanceQA:
MapQA: 0.86P
0,857
Table 1: For each target dataset under the 7 task categories, we compute Kendall's 7 between the ranking of source datasets by observed relative improvements (As-rrss and the ranking induced by the training-frre DATAPROPIET scores {M(s + )).. The high correlation coefficients show the effectiveness of DATAPROPHET in predicting the contribution ranking of different source datasets.
of the (frozen) encoders of the base MILM,, and caleulate the similarity for each of them separately. For each field of question, answer and image, we encode samples with appropriate templates, then do the fg-normalization for the embeddings, and take the expected cosine similarity between inde- pendently drawn items from D and T. We denote these expectations by QSim, ASim, and ISim. This factor rewards sources whose question and answer formats and visual layouts resemble the target. Interesting, we first tried to encode question and answer together as QASim, but it showed worse than separate version by 6% on F score. We evaluate the data influence rankings using only multi-modal perplexity times similarity as predictor, and find that the resulting Fry is 0.658, still not satisfactory.
Source dataset question diversity. Heuristically, a source dataset contains a diverse set of ques- tions, fine-tuning on it may encourage the model to acquire more generalizable skills, as previous papers have pointed out in LLMs (Wang et al., 2024a). To quantify this diversity, we construct an embedding representation z for each source data's 's example question We then cluster the embed- dings into K groups by certain clustering algorithms such as K-means, and compute the silhouette coefficient (Rousseeuw, 1987) given by
b(u) a(u) max(a(u), b(u)]
Sil En-zl
where auu) is the mean intra-cluster distance and b(u) is the minimum mean distance from u to any other cluster (both measured in the embedding space). We also quantify cluster balance via the normalized entropy H = - -(log x>--E Ta log Te with Ta the empirical cluster proportion. The final diversity is thus + H. which larger value indicating greater coverage and more balanced, well-separated clusters. In experiments, we construct Z, by averaging field embeddings and apply k-means algorithm for clustering with K 10.
DATA PROPHET Metric. Combining the three components above, we introduce our training-free, simple yet effective, data influence predictor DATAPROPHES
M(s+1)- = QSim ASim ISim PPL(s) (Sil + +H)
(3)
PPL
where s the training (source) data and is the testing (target) data. DATA PROPHET implies that successful transfer requires simultaneous alignment in text, vision, difficulty to base model, and question coverage. This thus motivates the product form in our metric, which down-weights the estimates for all expectations; when
score when any single factor is weak. We use subsampling is required, we select uniformly from the full set.
3.3 PREDICTOR RESULTS AND ANALYSIS
DATAPROPHET is simple but effective. While heuristically based and training-free our results in Table 1 suggests that DATAPROPHET can successfully predict the data reflectance as in real training results. The Tig is consistantly high across all target datasets and task types, with an average value of 0.863. And the Tam also reaches 0.857 across all source datasets, indicating the effectiveness of our metric.
Importance of individual components. We further investigate the contribution of each compo- nent to our metric. Removing each factor in Equation (3) degrades rank accuracy (Table 2). The most pronounced drop oceurs when maltimodal perplexity removed (0.863 0.491), highlighting the

---

Published as a conference paper at ICLR 2026
Variant
Kendall's 7 (Avg)
Full DATAPROPHET
0.860
w/o Answer Similarity
0.810
w/o Question Similarity
0.778
w/o Diversity (Silhouette Entropy)
0.659
w/o Image Similarity
0.625
w/o Perplexity
0.487
Table 2: Ablation of DATA PROPHET components Each row indicates removing one factor from Equation equation and recompute Kendall's т. Different components have varying effects on the correlation coefficient.
value of modeling souree headroom and target difficulty. Image similarity (0.627) and diversity (0.658) are also crucial components, suggesting that visual alignment and coverage are essential for transfer. Textual similarities from questions and answers are useful, but their contributions are more marginal compared to the other stronger signals.
Additional trials. We conduct multiple additional heuristic-basee trials but they all fail to show improvement: question difficulty, model's familiarity with the image, model's familiarity with the question, and answer length. For example, after incorporating answer length, the v decreased by 0.15. Only the three major factors in DATA metric remains effective.
DATAPROPHET FOR DATA SELECTION
Besides predicting the usefulness ranking of different source datasets, DATAPROPHET can also be used to guide data selection under a fixed compute budget. We first introduce two selection setups for supervised instruction tuning: real-data reweighting ($4.1) and synthetic-data ranking and selection ($4.2). We then present the outcomes and analysis in $4.3. Finally, we include an additional study on RL post-training data selection (Table 4).
4.1 REAL-DATA REWEIGHTING
We fix the total training budget to be N=14 x 20K = 280K samples. Using the fourteen vi- sion-language datasets as described in Section as candidate sources, we determine each dataset's relative weighting in the mixture using their DATAPROPHET score. Intuitively, datasets assigned higher DATAPROPHET scores are expected to contribute more positively to target performance, and are therefore allocated a larger proportion of samples in the mixture. For more details reweighting algorithm, please see Appendix A.3.
Note that the relative weighting would sometimes require more than 20K items from given dataset, which is beyond the size of our original 14 datasets. In such scenarios, we sample new data from the original full data source to obtain the desired amount.
For the real-data reweighting setup, we compare the following baseline methods of obtaining the training set of 280K training samples for given target set.
Uniform: combine the 14 source datasets with 20K examples uniformly sampled from each.
Oracle: observed relative improvements from Figure reweigh the source datasets.
ICONS (Wu et al., 2025):: applies the gradient- influence consensus approach to score all data in the original real data pool, then use the top 280K datapoints as the training This training-
required method is current state-of-the-art data selection method for MLLM instruction tuning.
D.P. DATA PROPHET reweigh the source datasets.

---

Published as a conference paper at ICLR 2026
Real Data Reweighting
Synthetic Data Sclection
Baseline
Gold
Ours
Baseline
Ours
Benchmarks/ (Sampling Methods
Uniform
ICONS
Oracie
D.P.
Uniform
ICONS
D.P.
OCR-VQA (Mishra al., 2019)
85.2
86.5
87.5
87.4
65.4
73.5
74.6
ScreenQA (Hsiao et al., 2022)
83.1
83.6
84.8
85.0
67.3
76.5
78.4
ChartQA (Masry et al., 2022)
83.7
84.8
84.9
85.6
68.2
77.2
78.3
Char12Text (Kantharaj al., 2022)
84.0
84.5
86.0
86.2
68.7
75.9
77.5
Open- Open-Spatial (Cheng et al., 2024)
37.9
40.7
41.2
41.1
26.7
27.3
28.5
CLEVR(R) (Johnson et al., 2017)
41.5
42.2
45.0
45.6
33.8
35.2
37.6
CLEVR(C) (Johnson et al., 2017)
44.5
45.3
47.2
48.1
35.6
36.8
38.6
TallyQA (Acharya et al., 2019)
41.8
42.8
45.2
45.1
30.4
35.5
36.2
A-OKVQA (Schwenk al., 2022)
83.7
84.9
85.8
86.0
68.2
74.4
74.8
VC-GVQA (Shao al., 2024)
81.5
82.3
83.7
83.5
67.8
75.9
76.8
VQA (Mathew et al., 2021)
76.9
79.4
80.4
81.4
58.6
66.4
67.2
FinanceQA (Sujet AI, 2024)
73.8
74.1
75.1
75.2
57.2
64.8
65.9
Geom (Kazemi al., 2023)
60.1
61.2
62.4
63.0
47.1
53.1
54.4
MapQA (Chang et al., 2022)
82,9
83.3
83.5
84.8
75.2
78.2
79.8
Average
67.6
69.6
70.8
71.0
55.1
60.8
62.0
Improve over Uniform
+2.0
+3.2
+3.4
+5.7
+6.9
Table 3: Data selection results. Keeping fixed compute budget of 280K datapoints, we present the data selection results on 14 target benchmarks. Sampling methods: Un form is the uniform sampling baseline, ICONS is the sota training-based baseline, and Orac is the gold baseline that selects samples based on oracle real experiment performance improvement as Figure 2. Average reports the mean performance across all benchmarks, and Improve over Uniform shows the absolute performance gain relative to uniform sampling. From the results, our DATAPROPHET--aal pling achieves the best average performance both real and synthetic data, surprisingly reaching up to +0.2% improvement over the gold oracle selection.
sam-
4.2 SYNTHETIC-DATT SELECTION
Given a target dataset and a large pool of synthetic training candidates, DATAPROPHET can also be used to identify the subset of data expected to be most beneficial for improving target perfor- mance. We score each individual datapoint syn.d by simplified version the original metric which removes the diversity term:
M(syn +T) QSim ASim ISim PPL(Sym D)
PPL(T)
In our experiments, we construct the synthetic data pool via the following: For each of the 14 datasets, we split its 20k images into two disjoint halves of 10k images each. One half is sent to GPT-5 and the other half Gemini 2.5 Pro. For every image, the generator asked produce five VQA-style crestton-answer pairs conditioned only on the image. Both GPT and Gemini 2.5 Pro receive the same prompt template (Appendix A.4). This yields 10k x 2 x = 100k QA pairs per dataset and ~ 1.4M pairs in total across the 14 datasets. We retain all pairs that satisfy basic validity checks (non-empty strings, short answers without special tokens).
Using this synthetic data pool, compare the following baseline data selection methods: Uniform, which uniformly samples 280k data from the full synthetic data pool and ICONS (Wu et al., 2025), our method D.P. for DATA PROPHET.
8

---

Published as a conference paper at ICLR 2026
No RL
Real RL Data Selection
Syuthetic RL Data Selection
Baseline
Baseline
Ours
Baseline
Ours
Benchmarks/ / Methods
Baseline
Equal
D.P.
Random
D.P.
OCR-VQA (Mishra et al., 2019)
72
75
76
71
73
ScreenQA (Hsiao et al., 2022)
66
68
69
68
67
CharlQA (Masry et al., 2022)
70
71
73
69
72
Chart2Text (Kantharaj et al., 2022)
69
Open- (Cheng et al., 2024)
35
39
41
36
37
CLEVR(R) (Johnson et al., 2017)
41
44
46
42
42
CLEVR(C) (Johnson al., 2017)
49
52
52
50
52
TAllyQA (Acharya et al., 2019)
33
35
A-OKVQA (Schwenk et al., 2022)
68
69
62
65
VC-GVQA (Shao et ul., 2024)
65
67
69
64
65
DocVQA (Mathew al., 2021)
59
62
60
62
63
FinanceQA (Sujet AI, 2024)
55
56
54
Geom Verse (Kazemi et al., 2023)
42
44
47
43
45
MapQA (Chang et ul., 2022)
64
67
66
64
68
Average
55.7
58.3
59.5
56.4
57.7
Improve over Baseline
+2.6
+3.8
+0.7
+2.0
Table 4: RL post-training data selection results. We fix a prompt budget of 300 14 and evaluate on the same 44-benchmark test suite. For real RL data, we compare equal allocation (Equal) to DATAPROPHET-gunE --gdddddd allocation (D.P.). For synthetic RL data, we compare random sampling (Random) DAIAPROPHET-guiee selection. Average reports the mean score across benchmarks, and Improve over Baseline shows the absolute gain relative to the no-RI. baseline.
4.3 RESULTS AND ANALYSIS
DATAPROPHETS provides effective training-free guidance for data selection. For each of the 14 target benchmark datasets, we conduct real-data reweighting and synthetic-data selection using methods above. Table summarizes the results for both selection paradigms. Overall, DAT- APROPHET-guides selection achieves the best macro-average performance across tasks in both the real and synthetic setups, yielding 3,4% and 6.9% improvements, respectively. Notably, reweight- ing guided by DATAPROPHET matches exceeds the Oracle reweighting constructed from ob- served performance gains. Moreover, our training-fres metric outperforms the training- based selec- tor ICONS, demonstrating that DATAPROPHET both effective and computationally efficient for data selection.
Does Gemini 2.5 Pro or GPT provide higher-quality supervision data? Interestingly, turns out that Gemini produces higher-quality synthetic data than GPT. In the synthetic data setup, about 38% of selected items come from GPT-5 and 62% from Gemini 2.5 Pro, indicating that Gemini contributes larger share of items that score highly under the instance-level DATAPROPHET metric. Manual inspection suggests that these items are more challenging, reflected by higher per-token perplexity, and better aligned with the target's distributions in layout, entities, and answer forms.
PROPHRT also improves RI post-training data selection. We further evaluate whether PATAPEO can guide data selection for RL post-training. We perform GRPO training using the verl implementation on Dwwen.55VL-3B-110 Insttrutt with default hyperparameters, and fix the RL prompt budget to 300 14. We compare equal allocation across the datasets to DATA PROPHET- guided allocation for real RL data, and random sampling to synthetic RL data. As shown in Table 4, DATAPROPHET improves the average score from 0.583 to 0.595 for real RI. data and from 0.564 to 0.577 for synthetic RI data on the 14-benchmark test suite.
selection for

---

Published as a conference paper at ICLR 2026
5 RELATED WORKS
Data Mixtures for LMs Data Mixture has long been an interesting problem in the research com- munity. Xie et al. (2023a) propose DoReMi, which trains a small proxy model with GroupDRO to learn domain weights and then resamples the corpus for full-scale training. A complementary line fits predictive mixture laws (Ye al., 2025; Ge et al., 2025; Kang et al., 2025; Li et al., 2025). Beyond domain-level mixtures, example-level selection has been studied for both pretrain- ing and instruction tuning, such as DSTR (Xie et al., 2023b) and LESS (Xia et al., 2024). Simple perplexity-basee pruning and perplexity correlations offer strong, training-free selectors at pretrain- ing scale (Marion et al., 2023; Thrush et al., 2024b; Qi et al., 2025). Our work differs in being training-frree instead of proxy runs, gradients, or online adaptation, propose simple metric that scores training sets for multi-task utility before any finetuning.
Data Mixtures for Multimodal LMs In instruction tuning of VLMs, ICONS aggregates first- order influence estimates across tasks to identify broadly useful examples, attaining pear-parity with full-data training using compact subsets (Wu et al., 2025). Related selection frameworks estimate task- and instance-level value using gradient features (Liu et al., 2024) score with the model itself to filter hard/diverse instructions (Chen et al., 2024). On the curation side, DATACOMP standardizes dataset filtering problem, enabling fair comparison across
large-scale image-text pretraining selection strategies and scales (Gadre et al., 2023). At the post-training/RL. stage, MoDoMoDo formulates multimodal RLVR mnituure-pptmmizaten of-distribution generalization (Liang et al., 2025). In contrast to these maining-dependee paradigms (proxy sweeps, gradient stores, or RI. rollouts), target training-free pre-finetuning decision rule that is compute-light and directly applicable to both real and synthetic multimodal data.
problem, learning policies that improve out-
Domain and dataset generalization for LMs In the LM context, existing work explores the gen- eralizability of SFT or RL training from one domain to another. Reinforcement post-training (RPT) can generalize beyond the fine-tumed domain under certain conditions (Hu et al., 2025); comparative studies suggest SFT tends memorize while RL-style post-training generalizes more robustly (Chu et al., 2025); and math-specialized training shows mixed transfer unless optimized via RL rather than SFT (Huan al., 2025). However, these works do not offer metrics for analyzing the contri- bution of different datasets to improving performance on the target domain. Our work seeks to both deepen understanding and offer prescriptive guidance.
CONCLUSIONS
In this paper, we investigate whether one can predict before any training how much given source dataset will benefit a target dataset in MILMMs. Across seven task families, we find that intuitive task similarity an unreliable guide and that transfer is -ataset-specific rather than task-specific. To address this, we introduce DATAPROPHET, simple, training-frree and interpretable metric that inte- grates cross-modal similarity, multimodal perplexity, and source dataset diversity. DATAPROPHET closely aligns with realized training outcomes when ranking data influence, and further serves useful guidance for both real and synthetic data selection. We believe DATAPROPHES provides a principled signal for data curation, enabling more effective MLLM training.
REFERENCES
Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp- 8076-8084, 2019.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint 1.X.:22402.162777 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A versatile vision- laiiunl model for understanding, local- ization, text reading, and beyond, 2023.
10

---

Published as a conference paper at ICLR 2026
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
Shuaichen Chang, David Palzer, Jialin Li, Eric Losser-Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.08455, 2022.
Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, and Heng Huang. Your vision-language model itself is strong filter: Towards high-quality instruction tuning with data selection. arXiv preprint 2XXi:2442.12201 2024. URL https:
//arxiv. www.orrgaaoo... abs org./v.orr..ooooooo
2202.4222..22 .1001.22011
An-Chieh Cheng, Hongxu Yin. Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. Advances in Neural Information Processing Systems, 37: 15002-133593,, 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang. Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sft memorizes, r1 generalizes: A comparative study of foundation model post-training arXiv preprint arXiv:2501.77661 2025. URL https: http://xxxvvv.o/.//..
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim En- tezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Olga Saukh, Alexander Ratner, Shu- Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Alexandros Dimakis, Jenia Jit- sev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search the next gener- ation of multimodal datasets. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https:ttp//. / www.i.org...o.............. abs 22304 4108.
Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. Bimix: Bivariate data mixing law for language model pretraining. OpenReview preprint, 2025. URL //pprr///eeerreeew.. net / /fotum/umm Jd= 602ix7. Submitted to ICLR 2025.
Yao-Hung Hubert Hsiao, Yue Wu, Xiaojuan Chen, and Tak-Sang Lee. Screenqa: Large-scale guestion-answer pairs over mobile app screenshots. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 8384 8395, 2022.
Chuxuan Hu, Yuxuan Zhu, Antony Kellermann, Caleb Biddulph, Suppakit Waiwitlikhit, Jason Benn, and Daniel Kang. Breaking barriers: Do reinforcement post training gains transfer to un- seen domains? arXiv preprint arXiv:2506.19733 2025. URL https: org/ /bs/ 2506.19733
Mingzhen Huan et al. Does math reasoning improve general Ilm capabilities? a controlled study of transfer and forgetting. arXiv preprint arXiv:2507.00432, 2025. URL /ttttts::/arxi......
abs/2507. 00432.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2-01-22900 2017.
Feiyang Kang, Yifan Sun, Bingbing Wen, Si Chen, Dawn Song, Rafid Mahmood, and Ruoxi Jia. Autoscale: Scale-aware data mixing for pre-training 1ims. Conference on Language Modeling (COLM), 2025. URL https: //tp///eeeeeeeeeeeeeeeeeeeeeeeee net/ Forum?id=ruf v
Shankar Kantharaj, Rixie Tian Qi Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. Chart-to-text: A large-scale benchmark for chart summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4005-4023, 2022.
Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Ge- omverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312. 12241, 2023.
11

---

Published as a conference paper at ICLR 2026
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting Ilm performance with self-guided data selection for instruction tuning. arXiv preprint vrXiv2338..10022 2023.
Yuan Li, Zhengzhong Liu, and Eric Xing. Data mixing optimization for supervised fine-tuning of large language models. In Proceedings of the 42nd International Conference Machine Learn- ing (ICML). PMLR, 2025. URL https: / opwnrevieeeee ew net. /dftt d== 9 9kgoNoc2N.
Yiqing Liang, Jielin Qiu, Wenhao Ding, Zuxin Liu, James Tompkin, Mengdi Xu, Mengzhou Xia, Zhengzhong Tu, Laixi Shi, and Jiacheng Zhu. Modomodo: Multi-domain data mixtures for multimodal IIm reinforcement learning. arXiv preprint (rXiv:2505.2487/ 2025. URL https: //arxiv. org/ abs/ 2505 24871.
Zikang Liu, Kun Zhou, Wayne Xin Zhao, Dawei Gao, Yaliang Li, and Ji-Rong Wen, Less is more: High-value data selection for visual instruction tuning. arXiv preprint arXiv:2403.09559, 2024.
URL htt.t::tp//............t / http://axxvv..roo.o
abs/2403. 09559.
Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining ilms at scale. arXiv preprint arXiv:2309.0455 2023.
Ahmed Masry, Xuan Long Do, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279, 2022.
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2200-2209, January 2021.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Oer-vqa: Visual question answering by reading text in images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4492-4501, 2019.
OpenAl. Gpt-4 technical report, 2023.
Xuan Qi, Rongwu Xu, and Zhijing Jin. Difficulty-basee preference data selection by dpo implicit reward gap. arxiv preprint 2rXiv:2508.04144 2025.
Peter J. Rousseeuw. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20(1-2):53-65, 1987. doi: 10. 1016/0377-04477679991555
Noveen Sachdeva, Benjamin Coleman, Wang- Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient llms. arXiv preprint rxXiv:2402.09668 2024.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: benchmark for visual question answering using world knowledge. arXiv preprint 2.XY::2006.01718. 2022.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Liu, and Hong- sheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and benchmark for chain-of of thoutht reasoning. arXiv preprint arXiv:2403. 2024. doi: 10.485550arxxi.24403.66
Sujet AI. Sujet-
Hugging Face Dataset, 2024. URL https:
/tw.ggiigfacc..eeecc/// Accessed: 2025-09-03.
Gemini Team, Rohan Anil, Sebastian Borgeaud. Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 2r1.v:2312./1800 2023.
12

---

Published as a conference paper at ICLR 2026
Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. Improving pretraining data using per- plexity correlations. arXiv preprint arXiv:2409.05816, 2024a.
Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. Improving pretraining data using per- plexity correlations. arXiv preprint arXiv:2409.05816, 2024b. URL /ttppp://axiiy org/ abs/2409 2409. 05816. ICLR 2025.
Peiqi Wang, Yikang Shen, Zhen Guo, Matthew Stallone, Yoon Kim, Polina Golland, and Rameswar Panda. Diversity measurement and subset selection for instruction tuning datasets. arXiv preprint arXiv:2402.02311. 2024a. /tRLtttts///axxi...rrrte
C2318.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12/91, 2024b.
Xindi Wu, Mengzhou Xia, Rulin Shao, Zhiwei Deng, Pang Wei Koh, and Olga Rus- sakovsky. ICONS: Influence consensus for vision-language data selection. arXiv preprint arXiv:2501.00654 2025. URL https: http:::/arxxv..oot http:/vv..rr/ab//////00/ 00654.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning, arXiv preprint arXiv:2402.04333, 2024.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Advances in Neural Information Processing Systems (NeurIPS), 2023a. URL https: /awiiv.iivvv/.ia////v//aaa//aaaa org/ abs 2305 10429.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma. and Percy Liang. Data selection for language models via importance resampling. In Advances in Neural Information Processing Systems (NeurIPS), 2023b. URL htLhhtts:///arr
org/ atsp//bbs//222222 222/223220000 03169.
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In International Conference on Learning Representations (ICLR), 2025. URL https: / http:p::///ppreeeeee wpwpreveeeeeeeeeeeeeeeeeeereeeee .eett
forum??dd=jjR227TMMM
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403. 13372, 2024. doi: 10.48550/a2Xxv.2403..33 ACL 2024 System Demonstration Track.
URL https: /tappss//arxiv...
org/abs 2403. 13372.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint 250::240444449 2025.
A APPENDIX
A.1 THE USE OF LLMS
LLMs did not play important role in this paper's research ideation writing the extent that they should be regarded as a contributor. the experiments, LLMs are the main experimental object.
A.2 IMPLEMENTATION DETAILS FOR SECTION
All runs fine-tune the base model for epoch using full-parameter supervised fine-tuning (Full SFT) implemented in LLaMA-Factors (Zheng et al., 2024). The learning rate set to be le 5. All evaluations follow the benchmarks' task-appropriate metrics.
13

---

Published as a conference paper at ICLR 2026
Algorithm Data Reweighting based on the metric M
Require: target dataset D; source datasets {D{ Jimi total budget NN
Ensure: training set $ for SFT
1: for i < 1 M do
2:
My f M DD,)
compute metric between source and target
3: end for
4: Z + f=1 my
5: for + to M do
6: N; + round N z
7: S, + SAMPLEUNIFORM (D!,, N.)
8: end for
9: S+ -- S
10: return 9
Algorithm Metric-guided Synthetic Data Selection
Require: target dataset Dr; synthetic pool S; either Top-K or threshold т Ensure: selected synthetic subset Syel
1: for each u E S do
2: QSim(u, D.), ASim(u, D:), ISim(u, D:)
3: PPL (fu)). PPL (D;)
4: m(u)) Sai--SQimmittt ASSm-IIim- 1-im--PpmPPP Siim-PPLaain(( ((+)) PPLics (D;)
5: end for
6: if USEToPK then
7: Sael + ToPK Ba_iBBBVVIIII VALUAAL m. K)
8: else
9: Sael +- {uES m(u)= 2 }
10: end if
11: return Sael
A.3 DETAILED ALGORITHMS
A.4 PROMPT TEMPLATES
The following template is used for both GPT-5 and Gemini 2.5 Pro. The calling code injects the image place <IMAGE>.
are generatinn ng visual question answering data strictly grounded in the given image. Use only what is visible in the image. Do not rely on outside knowledge, These questions should be inference questions about what is in the picture.
Image: <IMAGE>
Task: Create exactly five diverse VQA pairs about this image.
Constraints:
questions must be answerable solely from the image; answers should be short (several words or a number); avoid ambiguous
subjective wording; avoid near-duplicate questions; avoid requiring reading tiny illegible text; if no text is legible, do not ask OCR questions. Output format: return a single JSON array of five objects. Each object has the following fields
H "question"; one string
[ "answer": one string (lavercase for each word and numerals for numbers) Do not include any additional text before after the JSON.
14


Reviews:
[{"review_id": "iff3RaPJFi", "reviewer": "Reviewer_aw8L", "summary": "The paper explores the effectiveness of supervised data selection for training multimodal large language models (MLLMs). It introduces a novel metric, DATAPROPHET, which predicts the impact of training datasets on benchmark performance without prior training. Through empirical research on 14 diverse visual-language datasets, the authors find that intuitive task similarity does not correlate with model generalization. Instead, dataset characteristics significantly influence performance. The findings highlight the importance of careful data selection in enhancing MLLM efficacy.", "questions": "How is the target task selected? I don't fully understand data selection. Is the ultimate goal of data selection simply to improve accuracy on the target task? How does it contribute to enhancing the model's generalization ability on new datasets?\n\nThe evaluation metric used is too singular.  We also need to consider efficiency, the sensitivity of the task data, and the sensitivity of different components of the new methods.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strength:\n\nNovel Perspective on Data Selection: The article presents a new viewpoint on data selection, highlighting factors beyond mere similarity that influence model performance.\n\nEffective Task Selection Method: The introduction of the DATAPROPHET metric has shown excellent experimental results.\n\nComprehensive Empirical Validation: The study includes extensive empirical validation across diverse visual-language datasets, providing robust evidence for the proposed method's effectiveness.", "weaknesses": "Insufficient Innovation: The proposed method lacks substantial innovation, as it appears to be merely a combination of existing metrics rather than offering a fundamentally new approach to data selection. This may limit its perceived contribution to the field.\n\nLimited Exploration of Alternatives: The focus on the DATAPROPHET metric may overshadow other potentially valuable data selection strategies. The study doesn’t thoroughly explore how different combinations of metrics or alternative methodologies might yield even better results.\n\nThe experiment relies solely on one evaluation metric, which may lead to an incomplete assessment of the model's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TKKhAHPP9r", "reviewer": "Reviewer_KhQ8", "summary": "This paper tackles the critical, yet poorly understood, problem of data selection for training Multimodal Large Language Models (MLLMs). The authors first conduct a large-scale experiment (14 datasets, 7 tasks) to \"demystify\" data influence, finding that human intuition about \"task similarity\" is an unreliable predictor of which training datasets will improve performance on a target benchmark.Based on these counter-intuitive findings, the paper proposes DataProphet, a simple, interpretable, and training-free metric to predict a source dataset's influence on a target benchmark before any training occurs. The metric is a product of three components:Multimodal Perplexity: How \"difficult\" the source and target data are for a base model.Cross-dataset Similarity: Cosine similarity of text (question, answer) and visual (image) embeddings.Data Diversity: The coverage and balance of questions in the source data.The paper shows that DataProphet's predicted influence rankings are strongly correlated (86.0% Kendall's $\\tau$) with the actual performance rankings obtained after SFT. Finally, it demonstrates that this metric can be used for data selection, creating training mixtures that outperform both uniform sampling and a state-of-the-art training-based selection method (ICONS).", "questions": "Major: \n- Please explain the result in Table 3 where DataProphet (D.P.) selection outperforms the \"Oracle\" baseline. The Oracle is defined as using the ground-truth relative improvements from the single-dataset experiments (Fig 2). How is it possible for a predictive metric to beat the ground-truth oracle? Does this suggest that the optimal data mixture is non-linear, and that the single-dataset-training oracle is not, in fact, the true \"oracle\" for this task? This is a key point that needs clarification.\n\nMinor\n- How sensitive are the DataProphet rankings to the choice of the base model? The experiments use InternVL3-2B. If all components (perplexity, embeddings for similarity) were calculated using a different, popular MLLM (e.g., LLaVA-1.5), would the resulting Kendall's $\\tau$ correlation still be as high?\n\n- For the synthetic data selection (Sec 4.2), the diversity term ($Sil+H$) was removed from the metric (Eq 4). What was the reason for this? Was it computationally too expensive to calculate at the instance level, or did it simply not improve the results in this setting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses one of the most important and practical problems in modern ML: data selection.\n\n- The fact that the DataProphet metric requires no training is its greatest strength, making it universally applicable, cheap, and fast.\n\n- An 86.0% Kendall's $\\tau$ correlation with ground-truth training outcomes is extremely high and validates the metric's effectiveness.\n\n- The paper successfully translates the predictive metric into a practical data selection algorithm (Sec 4) and demonstrates its superiority. It outperforms uniform selection by a large margin (e.g., +6.9% on synthetic data) and, impressively, beats the computationally expensive training-based SoTA (ICONS).\n\n- The initial analysis (Sec 2) is a valuable contribution in itself, providing concrete evidence that \"intuitive task similarity\" is not a reliable guide for data selection.", "weaknesses": "- The paper reports in Table 3 that DataProphet-guided selection (D.P.) outperforms the \"Oracle\" baseline on average (e.g., 71.0% vs 70.8% on real data). The \"Oracle\" is defined as reweighting based on the observed single-dataset improvements from Figure 2. This result is confusing and potentially counter-intuitive. How can a predictive metric beat an oracle based on the ground-truth outcomes? This implies that the optimal mixture is non-linear and that the single-dataset oracle is not the \"true\" oracle for a mixed-data SFT. This is a fascinating result but is not explained at all, leaving the reader to wonder if it's a profound insight or a methodological quirk. This must be clarified.\n\n- The metric (Eq 3) is a simple product of its components. While the ablation (Table 2) shows all components are important, it's not clear why a product is the optimal combination. This feels heuristic. The paper's strength is its empirical results, but it's lighter on the theory of why this specific combination works so well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vVXZRDqmrb", "reviewer": "Reviewer_SyPJ", "summary": "This paper investigates the challenge of predicting a multimodal large language model’s (MLLM’s) generalization benefit from a given supervision dataset, even before training. The authors develop DataProphet, a training-free metric combining multimodal perplexity, cross-modality similarity, and dataset diversity to estimate the influence of candidate training data on downstream benchmarks. They show that traditional intuition about task similarity fails to predict which training data is most helpful, and demonstrates that its influence ranking correlates with performance gains.", "questions": "Around Table3, the discussion lacks qualitative analysis or concrete case studies of failures: When does DataProphet select harmful or unhelpful examples? How robust is the metric to data imbalance, annotation quality, or label noise in the source datasets?\n\nAnd also see my embedded question in \"weaknesses\".", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a systematic, large-scale analysis of data generalization in MLLMs, involving 14 datasets over diverse VQA-related vision-language tasks. This scope (detailed in Section 2 and Figure 2) provides a valuable resource and a robust empirical foundation for the claims made.\n- By quantitatively demonstrating (Figure 1, Figure 2) that surface-level or even task-category similarity does not reliably predict data transfer utility, the work provides a corrective to widespread \"intuitive data selection\" assumptions in the field.\n- DataProphet leverages a mix of complexity (perplexity), alignment (similarity), and diversity factors, all easily computable without requiring costly pretraining or proxy tasks. This increases applicability in compute-limited scenarios.\n- The authors use strong evaluation protocols (employing Kendall's \\tau between predicted influence rankings and the ground truth) and ablation studies to show the contribution of each metric component.\n- They have shown DataProphet is competitive or superior to state-of-the-art, more expensive approaches, boosting both real and synthetic data selection, and even outperforming performance-based oracle mixtures in average (Table 3).", "weaknesses": "- All experiments are performed using InternVL3-2B. The generality of findings to other competitive models (e.g., Gemini, GPT-4V, Qwen-VL) is not established. Since DataProphet calculations depend on the base model’s embedding space and perplexity, there is a risk of model-specific artifacts, especially when transferred to different backbone architectures.\n\n- Critical equations such as the definition of dataset diversity (Page 6), silhouette score, and their combination with entropy lack detailed motivation for the additivity, choice of $K=10$, cluster methodology, and scaling. The rationale for normalizing perplexities when transferring across datasets is not justified (e.g., differing answer lengths, vocabulary distributions, or answer spaces could bias comparisons. There are also minor ambiguities in notation (e.g., use of $\\Delta_{s \\rightarrow t}$ notation on Page 3, 4) should clarify whether this strictly measures test accuracy delta or considers area under curve for training budget). Additionally, the metric for synthetic data selection drops the diversity term (Page 8) without justification or discussion of why this is valid—an omission that may have side effects for coverage in less-redundant pools.\n\n- While the procedure is generally clear, some details are underspecified. For instance, random seeds, batch sizes, number of epochs, hardware, and exact computation of metrics like Kendall’s \\tau and confidence intervals are missing. There is no assessment of the variability of DataProphet scores under resampling, nor is there a discussion of selection stability over different random seeds.\n\n- While the DataProphet metric is intuitive and empirically effective, the paper advances no rigorous theoretical analysis of why the product formulation (Equation 3), or the selection of factors, is optimal or robust. What are the theoretical regimes or failure cases? Why use a product rather than, e.g., weighted sum, normalized version, etc.? The absence of any formal support or negative result is especially glaring given the field’s growing interest in principled data mixture theory (see, e.g., [1]).\n\n- Minor: suggestion include some recent related works:\n  - [2] related for benchmarking MLLMs under data ambiguity and generalization; should be referenced in Section 5 (Related Works) and discussed in terms of differences in experimental scope and generalization perspective.\n  - [3] gives metrics for bias/generalization in model evaluation; should be mentioned when discussing selection criteria and effects of selection (Section 4, Section 5).\n  - [4] is related for ongoing tuning/data mixture effects; discuss in Related Works and comparison section.\n\n[1] Jiasheng, Ye et al. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In International Conference on Learning Representations (ICLR), 2025.\n\n[2] Wang, Ru; Song, Selena; Ding, Liang (2025): MMA: Benchmarking Multi-Modal Large Language Model in Ambiguity Contexts\n\n[3] Vo, An; Taesiri, Mohammad Reza; Kim, Daeyoung (2025): B-score: Detecting biases in large language models using response history\n\n[4] Chen, Cheng; Zhu, Junchen; Luo, Xu (2024): CoIN: A Benchmark of Continual Instruction Tuning for Multimodal Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xLwwTz7BmE", "reviewer": "Reviewer_LeY5", "summary": "This paper addresses the fundamental question of predicting training data influence on target benchmarks in multimodal large language models (MLLMs) before any training takes place. The authors conduct a comprehensive empirical analysis using 14 vision-language datasets across 7 diverse tasks and make several counter-intuitive discoveries that challenge conventional wisdom about data selection.\nKey Contributions\n1. Counter-intuitive Empirical Findings: The paper argues that intuitive task similarity is an unreliable predictor of cross-task generalization. \n2. Dataset-level vs. Task-level Influence: The authors demonstrate that data influence operates at the individual dataset level rather than broad task categories.\n3. DATAPROPHET Metric: The paper introduces a training-free, interpretable metric that combines three components:\n    - Multimodal Perplexity: Measures source data difficulty relative to target\n    - Cross-dataset Similarity: Captures alignment in questions, answers, and images using MLLM embeddings\n    - Source Dataset Diversity: Quantifies question coverage using clustering-based silhouette coefficient and entropy", "questions": "**Contradictory Evidence in Task Category Claims**\nThere appears to be a contradiction between your claim that \"datasets from the same task category do not necessarily help each other the most\" (Figure 1b) and the actual results in Figure 2. Taking ChartQA as an example from the Chart understanding task family: the highest improvement comes from Chart2Text (+16.04%), which is indeed from the same task category as defined in Section 2.1. Spatial reasoning tasks like Open-Spatial (+12.46%) and CLEVR(R) (+9.19%) show lower improvements than the same-category Chart2Text. This pattern appears to support intuitive same-task-category transfer rather than contradicting it. Could you:\n  - Clarify how you define \"task category\" boundaries for this specific analysis?\n  - Explain why the ChartQA example doesn't contradict your main claims about counter-intuitive transfer patterns?\n  - Provide a more systematic analysis of when same-category transfer does vs. doesn't dominate?\n\n**Fundamental Flaws in Cross-Task Transfer Analysis**\n\nYour Observations 2 and 3 in Section 2.2 are based on comparing relative improvements across different target datasets from the same source dataset (same row, different columns in Figure 2). However, this analytical approach suffers from critical confounding factors that invalidate your conclusions:\n\nCore Problem: You compare relative improvements across tasks with fundamentally different baseline difficulties and improvement potentials. The tasks themselves have different intrinsic characteristics that affect their \"improvability,\" making cross-task comparisons of relative gains meaningless.\n\n Specific Evidence from Your Data: For Observation 2, you claim: \"after training on OCR-VQA data, the relative performance gain achieved on ScreenQA (OCR, 17.88%) is lower than that achieved on GeomVerse (map understanding,21.74%).\"\n   However, examining Figure 2's data reveals confounding factors:\n  - GeomVerse shows consistently higher average relative gains (16.89%) compared to ScreenQA (11.14%) across ALL source datasets\n  - GeomVerse's self-improvement ($\\Delta_{s→s}$) is 55.71% vs ScreenQA's 30.38%\n  - This suggests GeomVerse is simply more \"improvable\" as a benchmark, regardless of source dataset\n\n Questions:\n\n  1. Confounding Control: How do you distinguish between genuine cross-task transfer effects and task-intrinsic improvability differences? Your current analysis cannot separate these factors.\n  2. Baseline Normalization: Have you considered normalizing improvements by task-specific baselines or maximum achievable gains? Without such normalization, comparing raw relative\n  improvements across different tasks is scientifically invalid.\n  3. Alternative Explanations: How do you rule out that the observed patterns are due to:\n    - Different evaluation metric sensitivities\n    - Varying task saturation points\n    - Benchmark design artifacts\n    - Different training dynamics needed by task types\n  4. Causal Claims: Your claim that transfer depends on \"individual datasets\" rather than \"task categories\" requires showing that dataset-specific factors (beyond task characteristics) drive\n  the observed patterns. How do you establish this causal relationship?\n\n  The Same Issue Applies to Observation 3: Your examples of \"text-rich tasks influencing vision-centric ones more than text-rich ones\" likely reflect the same confounding - vision-centric\n  tasks may simply have more room for improvement rather than indicating genuine cross-modal transfer superiority.\n\n  Impact on Paper's Validity: This analytical flaw undermines your central claims about dataset-specific vs. task-specific influence. Without proper controls for task-intrinsic factors, your\n  conclusions about \"counter-intuitive\" transfer patterns may be statistical artifacts rather than genuine insights.\n\n  Suggested Resolution: Re-analyze your data using improvement metrics that account for task-specific characteristics, or restrict comparisons to tasks with matched baseline difficulties and\n  improvement potentials.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Originality**\n\n- The paper introduces a research question - predicting cross-dataset influence in MLLMs before any training occurs. While data selection has been extensively studied, the specific focus on training-free prediction of multimodal data influence represents a clear departure from existing gradient-based or proxy-model approaches.\n\n- Systematic Empirical Investigation: The comprehensive 14×14 influence matrix analysis is unprecedented in scope for multimodal settings. This systematic approach to mapping cross-task transfer patterns fills an important gap in understanding MLLM behavior.\n\n**Quality**\n- The paper demonstrates experimental breadth by selecting 14 diverse vision-language datasets spanning 7 task families (OCR, chart understanding, document understanding, general VQA, spatial reasoning, counting, and map reasoning). This coverage, with 2 datasets per task type, enables robust cross-task transfer analysis. The systematic 14×14 experimental matrix (196 training-evaluation combinations) provides empirical evidence for understanding how different source datasets influence performance across various target benchmarks.\n- Rigorous Experimental Design: The controlled experimental setup is well-designed with consistent hyperparameters, fixed compute budgets (20K samples each), and standardized evaluation protocols across all 14 datasets. \n\n**Clarity**\n- Well-Structured Presentation: The paper follows a logical progression from motivation → empirical analysis → method development → validation. The three-part takeaway in Figure 1 effectively communicates the main idea.\n\n- Effective Visualization: Figure 2's heatmap clearly illustrates the nature of cross-dataset influence. The color-coding and task groupings make patterns easily interpretable.\n\n**Significance**\nThe paper challenges the widespread assumption that \"similar tasks help similar tasks more.\" The systematic influence analysis framework and the 14-dataset benchmark provide a reference for comparative studies.", "weaknesses": "**Single Model Architecture**: All experiments are conducted exclusively on InternVL3-2B, which severely limits the generalizability of findings. Different MLLM architectures (e.g., GPT-4V, LLaVA, BLIP families) may exhibit fundamentally different cross-task transfer patterns due to varying pretraining objectives, data distributions, and architectural choices. The authors provide no evidence that DATAPROPHET's effectiveness extends beyond this single model family, making it unclear whether the discovered \"counter-intuitive\" patterns are universal phenomena or model-specific artifacts.\n\n**Ill-Defined Problem Formulation and Claims**\n- Vague \"Counter-Intuitive\" Claims: The paper's central claims about counter-intuitive findings are problematic due to poorly defined baselines. In the Introduction, the authors state that \"humans may intuitively assume that training the model on OCR task data will help its performance on chart tasks more than spatial reasoning tasks, since OCR and chart both require extracting text and numbers in an image.\" However, \"intuitive\" is not a well-defined, measurable concept. The authors provide no systematic survey of expert opinions, formal definition of intuitive similarity, or principled baseline for what constitutes \"expected\" transfer patterns. This makes their counter-intuitive claims essentially unfalsifiable and scientifically questionable.\n\n- Undefined Task Categories: The paper repeatedly refers to findings being \"dependent on individual datasets\" rather than \"task category,\" but \"task category\" itself lacks clear definition. The authors don't explain what constitutes a task boundary, how fine-grained the categorization should be, or what criteria distinguish one task category from another. This conceptual ambiguity undermines the central thesis about dataset-level vs. task-level influence. This subjective categorization makes it impossible to assess whether the reported transfer patterns reflect genuine task relationships or merely artifacts of the chosen taxonomy.\n\n**Limited Long-term Training Analysis**: All experiments use single-epoch training, but production MLLM training typically involves multiple epochs and complex scheduling. The influence patterns observed in short training runs may not persist during extended training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)