Can We Predict Performance of Large Models across Vision-Language Tasks?
Authors: Qinyu Zhao
Ming Xu
Kartik Gupta
Akshay Asthana
Liang Zheng
Stephen Gould
Venue: ICML 2025 poster (2025)
Keywords: Large Vision-Language Models (LVLMs)
Benchmarking
Probabilistic Matrix Factorization (PMF)
Markov Chain Monte Carlo (MCMC)
Active Evaluation
Primary area: deep_learning->large_language_models
TL;DR: A comprehensive evaluation of 108 LVLMs on 36 benchmarks and a framework for predicting LVLM performances using PMF with MCMC, showing interesting applications and observations.

Abstract: Evaluating large vision-language models (LVLMs) is very expensive, due to high computational cost and the wide variety of tasks. The good news is that if we already have some observed performance scores, we may be able to infer unknown ones. In this study, we propose a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. We first formulate the performance prediction as a matrix completion task. Specifically, we construct a sparse performance matrix $\boldsymbol{R}$, where each entry $R_{mn}$ represents the performance score of the $m$-th model on the $n$-th dataset. By applying probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC), we can complete the performance matrix, i.e., predict unknown scores. Additionally, we estimate the uncertainty of performance prediction based on MCMC. Practitioners can evaluate their models on untested tasks with higher uncertainty first, which quickly reduces the prediction errors. We further introduce several improvements to enhance PMF for scenarios with sparse observed performance scores. Our experiments demonstrate the accuracy of PMF in predicting unknown scores, the reliability of uncertainty estimates in ordering evaluations, and the effectiveness of our enhancements for handling sparse data. Our code is available at https://github.com/Qinyu-Allen-Zhao/CrossPred-LVLM.

Can We Predict Performance of Large Models across Vision-Language Tasks?
Qinyu Zhao
Ming Xu
Kartik Gupta? Akshay Asthana
Liang Zheng'
Stephen Gould
Abstract
(A) Performance Matrix
Normal Evaluaaion
Run LLaVA on Celebeiit Recogaition
Evaluating large vision-language models (LVLMs) very expensive, due to high com- putational cost and the wide variety of tasks. The good news is that if already have may
LLaVA
65
50
Math Problem Solving Run GPT-4V on 0 bjjec Recognition Cambrian Scene U Understanding
GPT-4V
70
32
71
Cambrian
64
42
Tiny Benchmariss
Gemini
75
59
Run L1&VA VA Celebrity Recognitinn deliiioo--in
some observed performance scores, be able to infer unknown ones. In this study, we propose a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. We first formulate the performance prediction as a matrix completion task. Specifically, construct sparse performance matrix R, where each entry Rmn represents the performance score of the m-th model on the n-th dataset. By applying probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC), we can complete the performance matrix, i.e., predict unknown scores. Additionally, we estimate the uncertainty of performance prediction based on MCMC. Practitioners can evaluate their models on untested tasks with higher uncertainty first, which quickly reduces the prediction errors. We further introduce several improvements to enhance PMF for scenarios with sparse observed performance scores. Our experiments demon- strate the accuracy of PMF in predicting unknown scores, the reliability of uncertainty estimates in ordering evaluations, and the effectiveness of our enhancements for handling sparse data. Our code is available at https:://// http:::://gtttttttcc.c Qinyu- Zhac wwt.ahha/roossseeeeacooo
estimate the original performance
Scroe Understanding
Object Recagnition
Celebrity Recognition
Math Problem Salving
Run LL&VA Math Problem Solving: Tany,
estimate the original performance
Run GPT-41 4VT-V Object Recognition
estimate the original performance
(D) Ours
High uncectainty.
Do this experiment first.
Leaminng
Predicting
Ranking
Figure Framework. (A) Given sparse matrix of performance scores of LVLMs various tasks, the goal is estimate the miss- ing entries. (B) normal way is to evaluate untested model-datasct pairs one-hy-one.
(C) TinyBenchmarks (Polo et al., 2024) runs models on smaller test sets and reproduce the original performance. (D) We use PMF to predict missing entries, reducing unnecessary evaluations, and rank new experiments based on uncertainty.
tasks (Liu et al., 2024; 2023a; Wang et al., 2024b; Bai et al.,
2025; Lu et al., 2024; Chen et al., 2024; Achiam et al., 2023; Reid et al., 2024). However, due to their large scale and versatility, evaluating these models leads to high cost.
First, large-scale models result in significant computational and memory costs. Second, since single LVLM can handle various tasks, hundreds of benchmarks have been proposed to comprehensively assess the strengths and weaknesses of models (L.i & Lu, 2024; Li et al., 2023a; Fu et al., 2023). Last, increase in model size, the development video LVLMs (Bai et al., 2025), and test-time scaling (OpenAl, 2024) techniques further raise evaluation cost. In our estima- tion, evaluating 7B model on 50 benchmarks takes around 20 hours on one A100 GPU, and even three times longer when using test-time scaling et al., 2024; Wu et al., 2024) (Details in Section A).
1. Introduction
Large vision-language models (LVLMs) have shown markable performance across wide range of multimodal
'Australian National University, Canberra, Australia Seeing Machines Ltd, Canberra, Australia. Correspondence to: Qinyu Zhao <qinyu.zhao@e anu.aeuuuuuuuuuuu.......
video benchmarks (Fu
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Fortunately, we have already observed performance scores from some of these models on some tasks, for instance, from

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
the official reports of released models and datasets. For new models, scores also be readily obtained with limited compute by running a small number of tasks. If these observed scores can be used to predict unknown ones, we could avoid unnecessary evaluations and effectively reduce costs. Recent works (Polo et al., 2024; Zhang et al., 2024b) require running the same model on the same task to predict model performance, and most of them ignore the potential of observed performance data from other models tasks.
eral VQA (Li et al., 2023a), knowledge-dense VQA (Yue et et.,, 2024), hallucination (Li al., 2023b), medicine (He et al., 2020), emotion recognition (Goodfellow et al., 2013), and others. To reduce computational and API costs, we subsample some datasets, following Liang et al. (2024).
Using the results from 108 LVLMs across 176 datasets, we construct a 108 176 performance matrix, with some entries masked for testing. First, experiments demonstrate that PMF accurately predicts masked scores and consistently outperforms baselines as long as more than 10% entries in the performance matrix are observed. Second, we also show that selecting high-uncertainty model-dataset pairs for evaluation significantly reduces prediction errors compared to random selection. third, our improvements effectively alleviate the sparse data issue of PMF. Last, we also expand the performance matrix by introducing new models and datasets, showing the generalization ability of our method.
In this study, we propose a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. We first formulate this as a matrix completion problem. Specifically, we construct a sparse performance matrix R. where each entry Rynn represents the performance score of the m-th model on the n-th dataset. By applying probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC), we can predict un- known performance scores based on observed entries in the matrix. A summary of the framework is shown in Fig. 1.
In summary, this paper covers three main points. First, we formulate a problem of predicting the unknown per- formance of LVLMs across tasks. Second, we apply the well-established PMF algorithm to this problem, show the application of active evaluation, and propose several strate- gies to mitigate the sparse data issue. Third, we conduct comprehensive evaluation of LVLMs, constructing training and testing sets for further experiments.
A bonus of our framework is active evaluation, which aims to select subset of model-dataset pairs to evaluate in order to minimize prediction errors across the entire performance matrix. Given a PMF model very sparse performance matrix, we calculate prediction uncertainty from MCMC and prioritize evaluating model-dataset pairs with high un- certainty. Our experiments will confirm the effectiveness of this strategy for active evaluation.
2. Related Works
A challenge is that PMF tends to predict the average score for models and datasets with very few observed scores, re- sulting in poor prediction results (Mnih & Salakhutdinov, 2007). To address this, we introduce several improvements to enhance PMF for scenarios with sparse observed data. First, we extend PMF to a simple tensor factorization ap- proach, which can handle multiple performance metrics across different vision- language tasks. Second, we utilize Bayesian PMF (Salakhutdinov & Mnih, 2008) with an LKJ prior (Lewandowski al., 2009) on the variance. Third, we also incorporate specific information from LVLMs and datasets to improve performance prediction. For example, if we know a model uses CLIP as a vision encoder, the information may help predict the model's performance, es- pecially when we observe only a few performance scores.
2.1. Benchmarking LVLMs
In recent years, an increasing number of LVLMs demon- strate impressive capabilities on a wide variety of tasks, including GPT-4 (Achiam et al., 2023), Gemini (Team et al., 2023), LLaVA (Liu et al., 2024; 2023a), Instruct- BLIP (Dai et al., 2023), Intern (Chen et al., 2023; 2024), and Prismatic VLMM (Karamcheti et al., 2024). Their versa- tility requires various benchmarks to fully understand their strengths and weaknesses. Existing datasets can be repur- posed for assessing these models, such as Flickr30k (Young et al., 2014), GQA (Hudson & Manning, 2019), and OKVQA (Marino et al., 2019). Recent works also propose new benchmarks to evaluate LVLMs in handling complex reasoning tasks, including MMMU (Yue et al., 2024), and MME (Fu et al., 2023). Ad- ditionally, as LVLMs become more integrated into everyday applications, benchmarks (Li et al., 2023b) have been in- troduced assess trustworthy issues like hallucination in these models. The variety of LVLMs and benchmarks leads to substantial computational demands and memory usage.
(Li et al., 2023a),
In experiments, we conduct a systematic evaluation of 108 LVLMs across 176 distinct datasets derived from 36 exist- ing benchmarks, based on prior works (Duan et al., 2024; Zhang et al., 2024b; Liang et al., 2024; Karamcheti et al., 2024; Schaeffer et al., 2024). We evaluate open-source models such as LLaVA-v1.5 (Liu et al., 2023a), Instruct- BLIP (Dai et al., 2023), mPLUG-OWl (Ye al., 2023), and MiniGPT-4 (Zhu al., 2023), well as closed-source models including GPT-4o, GPT-4 (Achiam al., 2023), Gemini-1.5 (Reid et al., 2024). The benchmarks cover gen-
2.2. Improve Evaluation Efficiency
Recent works introduce unified frameworks to assess models across multiple benchmarks using a single code-
2

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
base, such as VLMEvalKit (Duan et al., 2024), LMMs- Eval (Zhang et al., 2024b), and HEMM (Liang et al., 2024). Our study builds on these efforts by consolidating their frameworks and integrating models in Prismatic VLMs se-
similarity constraint matrix to address the problem.
3. Modeling LVLM Performance
ries (Karamcheti et al., 2024; Schaeffer et al., 2024).
In this section, we first describe the application of PMF to model the performance score matrix of LVLMs across datasets. Then, we discuss active evaluation for LVLMs. Last, three techniques are introduced to enhance PMF: sup- porting multiple metrics, incorporating Bayesian PMF, and integrating model and dataset profiles in modeling.
Predicting unknown model performance can reduce the evaluation cost. Prior works select coreset of samples from a large benchmark, for evaluating large language mod- els (LLMs) (Polo et al., 2024; Perlitz al., 2023) and LVLMs (Zhang et al., 2024b; Zhu et al., 2024). The perfor- mance of a specific model on the coreset is used to estimate its performance on the full benchmark. Besides, existing studies estimate model performance on an unlabeled test set based on distribution shift (Deng& & Zheng, 2021), con- fidence scores (Guillory et al., 2021; Yang et al., 2024), or LLM feedback (Zheng al., 2023). Instead of running models on coreset or an unlabeled set, our framework predicts unknown performance by utilizing the correlation between model performances across benchmarks, and is complementary to coreset- based ones (see Section 5.3).
3.1. Revisit PMF
Let R be M x N matrix representing model performance scores on datasets, where M is the number of models and N is the number of datasets. For simplicity, we initially assume single performance metric. In reality, benchmarks often employ multiple metrics, so R becomes an M x N x $ tensor, where S represents the total number of metrics. We will extend our framework to this in Section 3.3.
In practice, only a subset of the elements in R are observed, meaning we evaluate only portion of the model-dataset pairs and aim to estimate the remaining performance scores. Specifically, we define a matrix O E {0, 1),MX,, where Omn = if Rm is observed, and otherwise.
concurrent study by Zhang et al. (2024c) applies deep matrix factorization to predict LLM performance. Their goal and methodology differ from ours. First, their work aims to explore whether L.M performance can be predicted except using scaling laws, while we focus on improving evaluation efficiency. Second, they use neural networks for prediction, whereas we adopt PMF with MCMC, which allows us to quantify uncertainty in performance prediction.
To model the observed matrix and estimate the unknown values, we employ PMF (Mnih & Salakhutdinov, 2007), as illustrated by the probabilistic graphical model in Fig. 2(A). PMF decomposes R into two cow-dimensional matrices, U E R M and V E R RND,, where D is the latent di- mension. Here, Un.: and Vn,: are the latent feature vectors for the m-th model and the n-th dataset, respectively, and we refer to them as Um and V,- These latent vectors are modeled as multivariate Gaussian distributions, and the ob- served ratings are assumed to follow Gaussian distribution centered at the dot product of the latent feature vectors:
Another related direction is adaptive testing (Rodriguez et al., 2021; Prabhu et al., 2024), i.e., adaptively selecting subset of samples for evaluation. While prior work re- lies on statistically inferred sample difficulty, we propose a method to rank model-dataset pairs for evaluation based on uncertainty in performance prediction from MCMC.
2.3. Probabilistic Matrix Factorization
p(R M. N
| U.V. V.)
PMF (Mnih& & Salakhutdinov, 2007) is widely applied in recommender systems. Given part of user ratings for items, the goal is to model the observed ratings and predict the missing ones. PMF achieves this by decomposing the ob- served rating matrix into two lower-dimensional matrices, representing the latent features of users and items. A rating is modeled via a Gaussian distribution centered around the dot product of the user's 's and item's 's feature vectors.
II
II
[W (Rmn VnVn,
(1)
m== n=1
P(U 100)= NNU 1n,
(2)
m=
N
p(V | (ot) = II N(V, Op. ob
(3)
One major challenge with PMF is that, if users rate very few items, their predicted ratings will be near the average for those items. Bayesian PMF (BPMF) (Salakhutdinov & Mnih, 2008) addresses this by placing distributions over the priors of the latent user and item features, making it more effective in handling sparse data. Additionally, Constrained PMF (Mnih& & Salakhutdinov, 2007) introduces a latent
where ID is D D identity matrix, and N(x |1, 02,02) represents the probability density function of a Gaussian distribution with mean f and variance o². For simplicity, we set ou = ov = 1.
Rather than using Maximum A Posteriori estimation to ob- tain point estimates of the unknown performance scores in
3

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
Ao
than 10% model-dataset pairs are observed). To address this, we enhance our model with several techniques, with new graphical model shown in Fig. 2(B).
di
Uie the Muy li Me + Gal
Previously, we assumed that cach dataset has only one scor- ing metric, but this is usually not the case in practice. For example, yes-or-no questions can be evaluated using ac- curacy, precision, recall, and F1 score, while open- open-ended questions may use metrics like BART score (Yuan al., 2021) and BERT score (Zhang et al., 2019). Model perfor- mances are represented by a tensor R E R. xNXS where S is the total number of metries. Empirically, we find that using PMF to model and predict each metric independently works well when sufficient data is available. However, when observed data is sparse, incorporating relationships between metrics will be helpful to performance prediction.
= L.nd
(0)
Figure Graphical Models PMF (A) and the enhanced model (B). (A) is adapted from the original paper (Mnih & Salakhutdinov, 2007). In (B), we set the mean to 0 and the covariance to the identity matrix. thus omitting most of the hyper-parameters for the random variable distributions.
To address this, extend our PMF model into simple Probabilistic Tensor Factorization (PTF), where we decom- pose the 3D tensor R into the product of two low-rank ma- trices and 1D vector. This can be interpreted as applying a linear transformation to the original PMF output, translating it into multiple metrics. Specifically, we define:
R. we apply MCMC to obtain distributions over the esti- mated scores and quantify prediction uncertainty.
We compare PMF and deep learning-based matrix com- pletion methods, such as Deep Matrix Factorization (DMF) (Arora et al., 2019) and Graph Convolutional Ma- trix Completion (GCMC) (Berg al., 2017). The results indicate that PMF is more suitable to our problem than deep learning methods, probably because deep learning is more data hungry. Thus, this study will focus on PMF The details are shown in the supplementary materials (Section C.I).
p(R U,V, w. b, = N
II II [N (Riens) (U,.V w., + b,, bz, Omne m= 1=1==1
(4)
p(w (²)) = =Nww Os, Is), p(b of) Nb Os, (i
(5)
3.2. Active Evaluation
(6)
MCMC allows to estimate score distributions and readily obtain uncertainty estimates for each unknown score, en- abling us to prioritize evaluation experiments. For example, we are uncertain about GPT-4's performance on 3D un- derstanding but confident about LLaVA's performance on object recognition, we can prioritize evaluating GPT-4 on the 3D task when our resources are limited.
where set ow σ. = 1 for simplicity.
This approach implicitly assumes linear relationship be- tween scoring metrics, which may not exactly hold in reality. However, we usually observe some linear correlation be- tween the metrics on the same task. More sophisticated techniques, such as advanced tensor factorization methods are left as future work further improve the model.
In our method, we begin by applying PMF to model sparse performance matrix. Using MCMC, we get hundreds of estimations of each unknown score and calculate the stan- dard deviation of estimations as a measure of uncertainty. The unobserved scores are ranked by their uncertainties. High-uncertainty scores are replaced with ground truth, sim- ulating evaluation process in practice. We rerun PMF with updated observed data, calculate uncertainty, and determine the next set of evaluations. This process is repeated until our resource budget is exhausted or all scores are observed.
Note that some metries may be irrelevant for certain datasets, e.g., accuracy is not meaningful for long-answer questions. While our model can predict these scores, we discard the predicted results because there is no ground truth.
3.4. Bayesian PMF
Instead of using fixed priors for the feature vectors, we model the priors using probabilistic distributions, as pro- posed by Salakhutdinov & Mnih (2008). Unlike the original paper, which employs a Wishart distribution for the vari- ance, we use the LKJ correlation prior (Lewandowski et al., 2009) and an Exponential prior model the variance, as suggested by the PyMC official documentation,
3.3. Multiple Metrics
Our experiments show that standard PMF performs well with sufficient observed data. But its performance degrades significantly and is even worse than predicting the average score, when the observed data is very sparse (i.e., fewer
A 5 (diag (az) b (ar)) LT)
(7)

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
where Lo nw)= = Ly L my) and p Au) = Ila=1 Exp(aa) | ). Then, latent feature vectors are mod- eled as follows:
explore three different approaches generate these latent representations: D1. using MPNet (Song et al., 2020) to encode short description of each dataset. D2. using CLIP to encode images and BGE-M3 to encode questions in a dataset (following Zhang et al. (2024b)), then averaging the embeddings on the dataset; and D3. using VA-7B to en- code both images and text, then averaging the embeddings as the profile for the dataset.
plue I AT")) = =N(HU 0OD, A
(8)
M
PUU AU II
Nu mne A
m==1
A similar formulation applies to V. which we omit here.
4. Experiments
3.5. Model and Dataset Profiles
We first construct a large performance matrix by evaluating LVLMs on different benchmarks, and then, we present key experiments to validate our framework.
The final enbancement is the incorporation of additional information about the models and datasets. For example, knowing that two LVLMs use CLIP as the vision encoder, or that L.L.A-v and L.L.V--NeT are developed by the same team, suggests potential relationships in their perfor- mances. Inspired by Constrained PMF (Mnih & Salakhutdi- nov, 2007), we incorporate extra information model and dataset profiles, to improve performance prediction.
4.1. Evaluating Models on Benchmarks
Based on prior code repositories (Duan et al., 2024; Zhang et al., 2024b; Liang et al., 2024), we evaluate 108 LVLMs on 176 tasks in 36 benchmarks and build a 108 x 176 per- formance matrix, which is used for our main experiments PMF and MCMC in Section 4.2 and 4.3.
Let HEEH E R M and G E RR NJJ represent the model and dataset profiles, where Hm,; encodes K properties of the m-th model (e.g., vision encoder type), and G.,; encodes J properties of the n-th dataset. We introduce Gaussian- distributed variables Y KxD and X E /XD to learn the effects of these profiles. The latent feature vectors are now the sum of the original vectors and the profile features, following Constrained PMF (Mnih& & Salakhutdinov, 2007).
The open-source models we cover include I.La (Liu et al., 2023a), LLa Ve- (Liu et al., 2023a), Instruct- BLIP (Dai et al., 2023), mPLUG-Owi (Ye et al., 2023), and Prismatic VL (Karamcheti et al., 2024). We also evaluate closed-source models like GPT-4 (Achiam et al., 2023) and Gemini-1.5 (Reid al., 2024).
K
The benchmarks span variety of domains, including gen- (10) eral VQA (SEED-2), knowledge-dense VQA (MMMU), hal- lucination (POPE), medical question answering (Path and emotion recognition (FaceEmotion). Some large- (11) scale benchmarks, such as SEED-2 (Li al., 2023a) and MMMU (Yue et al., 2024), cover multiple tasks. To con- (12) duct a fine-grain analysis, we split these benchmarks into task-specific datasets, resulting in 176 datasets in total. Fol- lowing HEMM (Liang et al., 2024), we subsample some datasets to reduce computational and API calling costs of LVLMs. For each dataset, we calculate a main metric for PMF (either accuracy or BARTScore), and several other metrics, leading a total of six metrics for PTF modeling.
1 o3 P
II
N(Yx 0p, ob.))1D),
k=1
p(X xxx) II N(X; xb
U - U + HY, V + GX.
Oracle Profiles. To explore the upper bound of model and dataset similarities, we use the full R matrix to cluster models and datasets. For each model, we take R. (its performance across all datasets) as vector and apply the K- Means algorithm to cluster all models. We select the optimal number of clusters using the elbow method. Similarly, for each dataset, we cluster R., in the same way. We convert the cluster assignments into one-hot vectors as profiles.
To evaluate the generalization ability of framework, we expand the model and dataset pool by introducing new models: WVen2-VL-Inntrru 2B/ / 7B (Wang et al., 2024b), Qwen2.5- 3B/ / 7B / 32B (Bai et al., 2025), and DeepSeek- VL tiny / small (Lu et al., 2024), as well as new benchmarks: Math Vision (Wang et al., 2024a), EMMA (Hao et al., 2025), Video-MME (Fu al., 2024), and Bice (Wu et al., 2024). This expansion in- creases the performance matrix cover 115 LVLMs across 40 benchmarks. We present the results in Section 4.5.
Custom Profiles. Since oracle profiles rely on complete per- formance data, they are not feasible for real-world use. To overcome this, we define custom profiles that can be applied in practice. For models, we include features such the num- ber of parameters in the LLM backbone, vision encoder type (one-hot), and the LVLM family (one-hot), illustrated in the supplementary material (Table 8). Additionally, we cluster datasets based on latent representations obtained from var- ious models and get one-hot encoded dataset profiles. We
Full details of datasets and models are provided in the sup-
5

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
Global Mean
Mean Of Means
PMF
Overall IRMSE
0.4
ACC RMSE
0.20
BART| RMSE
1.0
0.8
0.15
0.3
0.10
0.6
0.2
20
40
60
80
95
20
60
80
90.95
95
60
80
90.95
Test Ratio (A)
Test Ratio (BB
Test Ratio (C)
1.0
1.0
1.0
Ground Truth
0.5
0.5
0.5
0,0
0.0
0.0
-5.0
-2.
-5.0 Predicted BART
-7.5
-5.0
-2.5
Prodicted ACC
Predicted BART
Predicted ACC
Predicted ACC
Predicted BART
Figure 3. Performance of PME. (A-C) PMF consistently outperforms both baselines when the test ratio is below 90% for estimating all unobserved scores (A), accuracy scores (B), and BART scores (C). with particularly strong performance at lower test ratios. (D-F) The predicted scores exhibit correlations with the ground truth at test ratios of 20% (D), 60% (E), and 90% (F). Gray dashed lines represent perfect prediction i.e., y = x. We subsampled 200 scores (D-F) for visualization.
plementary material (Section B).
Table I. Comparison of PMF and PTE Superior results are high- lighted. PMF (Sep) models cach score separately, while PMF
4.2. Estimating Unknown Performance
(OneMat) combines accuracy and BART scores into single ma- trix, as each dataset contains either accuracy or BART scores. that supports multiple scoring metrics. which outperforms PMF at high test ratio.
We mask P% of the elements in the score matrix R, use the observed portion to normalize R. and train the PMF model using MCMC sampling. The model reconstructs the matrix R. and we evaluate the performance by comparing the esti- mated values with the ground truth for the masked elements. For MCMC, we the No-U-Turn Sampler (NUTS) (Hoff- man et al., 2014), an advanced Hamiltonian Monte Carlo method (Neal, 2011), tuning with 500 samples in the burn-in stage and drawing 100 samples. Empirical results show that 100 samples are sufficient for stable estimation. The mean prediction from MCMC is the final reconstructed matrix R.
PTF is the enhanced model
Method
Overall
Acc
Precision
Recall
F1
BART
BERT
Test Ratio: 20%
PMF (Sep)
0.175
0.073
0.135
0.166
0.134
0.463
0.068
PMF (OneMat)
0.193
0.074
0.461
0,205
0.078
0.129
0.176
0.108
Test Ratio: 90%
PMF (Sep)
0.327
0.159
0.238
0.262
0.227
0.864
0.096
PMF (OneMat)
0.317
0.156
0.723
PTF
0.290
0.159
0.186
0.230
0.180
0.754
0.094
We use Root Mean Squared Error (RMSE) as the primary metric. Additional metrics such Mean Absolute Error (MAF) and the coefficient of determination (R²), as well as ranking-basee based metrics including Spearman's 's rank corre- lation, Kendall rank correlation, Precision and Recall are reported in the supplementary material (Section C).
estimated scores strongly correlate with the actual scores, indicating reliable predictions from our framework.
However, as the amount of observed data decreases, PMF's performance declines as can be expected. extreme cases where the test ratio exceeds 90%, with limited information about model or dataset performance, PMF can perform worse than predicting the means.
We compare our method against two baselines: (1) Global Mean: predicting the global mean for unobserved scores; (2) Mean of Mcans: for each unobserved score, average the mean performance of the model, the mean performance on the dataset, and the global mean.
4.3. Active Evaluation for LVLMS
We compare our uncertainty- approach against two baselines: (1) random selection of model-dataset pairs, and (2) an oracle approach that selects the pairs with the highest actual errors. In the experiment, we start by 20% data for initial training, 60% as the pool set, and 20% for testing.
Results. As shown in Fig. 3(A-C), PMF significantly out- performs the baselines when the test ratio lower than 90%. This suggests that when only a portion of the scores is available, PMF infer the unobserved scores with high accuracy. Additionally, as demonstrated in Fig. 3(D-F), the
Then, we progressively include more data from the pool set for training using three different strategies, and calculate
6

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
0.26
Random
Table 2. Generalization to New Models Datasets. Average
Active
RMSEs of different methods are reported.
0.24
Oraele
Method
New Model
New Dataset
Both New
RMSE
If only know 20% performance
models and new datasets
0.22
Global Mcan
0,390
0.043
0.106
Mean Means
0.303
0.037
0.084
PMF
0.326
0.032
0.047
0.20
BCPMF
0.297
0.033
0.039
If we imows 50% performance of new models and new datuaets
0
10
20
30
40
50
60
Global Mean
0.389
0.045
0.090
Pereenlage Extra
Mean of Means
0.311
0.039
0.073
(A)
0.265
0.031
0.034
BCPMF
0.228
0.030
0.034
0.4
Moin Abolure
Fig. 5 illustrates the impact of the other two enhancement techniques. As shown, Bayesian PTF offers only negligi- ble improvements over standard PTF when there is enough observed data, but it is particularly beneficial in sparse con- ditions. In Fig. 5(B), our custom profiles also show im- provements when data is limited, though there remains a gap between custom profiles and the oracle profiles. Ad- ditionally, Fig. 5(C) highlights that adding profiles not only enhances PTF's overall performance but also reduces insta- bility, as seen by smaller error bars. Model profiles show significant performance gains, whereas dataset profiles con- tribute only marginally. Better methods for encoding and utilizing dataset information need further exploration.
0.2
0.0
0.05
0.10
0.15
0.20
Uncertainty
(B)
Figure 4. Comparison of Active Evaluation Methods. Starting with 20% of the data observed, we progressively conduct additional LVLM evaluations using three different strategies. (A) RMSE im- provement demonstrate the advantage of our method compared to random evaluation. (B) Uncertaintics from MCMC are correlated with the actual absolute errors.
4.5. Generalization to New Models and Datasets
To validate the generalization ability of the framework, we expand the performance matrix by adding new models and datasets. Then, we provide the complete performance data old models and datasets, along with partial observations for: (1) old models new datasets, (2) new models on old datasets, and (3) new models on new datasets. Based on this setup, we predict the missing performance scores and calculate the average RMSES for different methods.
the improvement in performance prediction on the test data. The experiment is repeated with 10 different random seeds, and we report the average improvement.
Results. As shown in Fig. 4, our uncertainty-aware method consistently outperforms the random baseline for a fixed budget of evaluations, especially when the amount of extra data is lower than 30%. Besides, uncertainties from MCMC are correlated with the actual absolute errors.
As shown in Table our method shows better generalization compared the baselines. We observe that generalizing to datasets is relatively easier than to models. This is probably because new datasets are often very challenging, leading to generally lower model performance and smaller RMSEs. In contrast, new models often have novel designs and remarkable improvements. This makes generalization to unseen models more difficult and needs further exploration.
4.4. Enhancing PMF
We apply three enhancement techniques to our PMF model and evaluate their effectiveness across different test ratios. To minimize experimental variance, we perform each exper- iment 10 times with different random seeds and report the average performance at each test ratio.
Results. As seen in Table 1, the multi-scon method PTF can get better performance when the matrix is very sparse. When there is enough data, separately modeling PMF with each score works very well and is comparable to PTF. For BART and BERT scores, PMF even outperforms PTF. This is likely because PTF assumes a linear relationship between scores. When this assumption does not hold, such as in the case of BART and BERT scores, it degrades performance.
5. Discussion
5.1. Predictability of Model Performance
There are two possible perspectives to understand why the model performance can be well predicted.
Benchmarks. There is redundancy in existing benchmarks, as also reported by Zhang et al. (2025). Multiple bench-

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
0.40
Menn Of Means Baseline
0.40
PTF
PTF
CPTF (Oraele)
RMSE
0.35
Overall RMSE
CPTF
BCPTP (Omdee
0.2
0.50
0.30
BCPTF
0.25
0.25
0.1
0.20
0.20
20
PTF Custem Random
Custom (DI
Custom D22
Custom DJ Cuarom Ocadl
Gracke Oracle
Test Ratio
Test Ratio
(A)
(B)
Figure 5. Performance of Enhanced PTE. (A) BPTF shows minimal improvement over standard PTF when data sufficient but proves particularly beneficial under sparse conditions. (B) Custom profiles improve performance when data is limited, though a gap remains compared to oracle profiles. (C) Ablation study on model and dataset profiles. "A B" represents using A for the model profile and B for the dataset profile. Custom model profiles lead to significant performance gains, while dataset profiles contribute only marginally. BPTF, Bayesian PTF; CPTF, Constrained PTF.
AldMA-Adapter- V2-L LORA--RBBBLL MARAA--A--B--- B-v21
Instances Counting (SBE1) (SBb)) 2)
InvernL M-XC crmposer2-78
Chart Understanding (SBED 2)
Opent lamin WT-pp----VV-------
VTL-MPT7R
Action Prediction (SEED 2)
MiniGP LaMMA777
Visual Mathematics (SbbD
Open Open Flamingo- 4B- VTL RP Langlastruct
Velocity, Acceleration.
(ScienceQA)
Owen- VL.Cha-
Structuralized Image-Text Understanding (MMBench
Cambrian-348
Instance Location (SBED 2)
Opan Hi-i-ooo---VVLLLPPP0
Genes to Trails (ScienceUA)
Min Vicuna0-130
DECIMER
Internl M-XXCompase2--vil
10 8b
Muterials (MMMU)
OpenFlamingo-333
Colebrity Recognition (SBED 2)
MiniGPT4 Vicumma0
Text Translation (MME)
OpenF lamingo- ----------I--PI11111
Music (MMMU)
Gemini-1
Image Style (MMBench
IntemtLM-XC M-onLLMXCCoppsste Co-0
Bocrgy and Power (MMMU)
GPT-4Turbo (2024-04-09)
Global Video Understandigg (SBED 2)
GPI-4e-Mini (2024-07-18)
Business (CMMUU
Gemini-1.3 P-PoPo
Magncis (ScienceOA)
GPT-4o (2024-08-0) 08--888000 06)
Numerical Calculation (MML)
1-4 (2024-05-13)
Text-to-Image Generation (Sbbb 2)
RMSE Improvement (36)
Improvement (%)
(A)
(B)
Figure Which Models and Datasets Are Informative for Performance Estimation. Given a PMF model train on 20% data of the performance matrix, We measure the improvement in RMSE of PMF when adding the entire results of model (A) or dataset (B).
marks may test similar skills like math. Besides, there might be correlation across different tasks. For instance, Liu al. (2023a) reports that an LVLM with a stronger LLM can achieve consistently better performance on different tasks.
5.2. Modeling Varying Settings of LVLM Evaluation.
Our framework is also extensible to different evaluation settings, such various prompts decoding strategies. There are two possible methods. Additional models: treat model under different evaluation settings as different mod- els, such as "LLaVA (Chain-of- Thouughh"-" and "LLaVA (Beam Search)". Additional profile: encode evaluation settings as extra profiles into PMF.
Models. Similarity in model architectures and training may also contribute to correlation between performance. First, if models use the same vision encoder, e.g., CLIP, they may share similar failure cases (Tong al., 2024b). Besides, many models use training data generated from GPT-4V, possibly resulting in similar strengths and weaknesses.
Specifically, we evaluate LLLVA--v.55777 on the 27 tasks in SEED-2, with the following various evaluation settings.
Predictability is valuable during model development. For example, when exploring optimal video LLM design, Zohar al. (2024) evaluate their models on reduced set of bench- marks to guide design decisions. Only at the final stage, they assess the model the full benchmarks to report actual performance. In such cases, they could use our method to reduce unnecessary evaluation when developing models.
Image input. (1) Default: use the clean images, or (2) add Gaussian noise into the images.
prompt the model to choose option ("Answer with the option's letter from the given choices directly:"), (2) provide no hint, (3) the Chain-of-
Prompt. (1) Default:
Thought (CoT) prompt ("Let's think step by step, and then
8

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
embeddings for images and BGE-M3 for text, and concate- nates them get the samples embeddings. Based on sample embeddings, we use random, Herding, and K-Center Greedy to select 10% core samples from benchmarks.
Table Prediction Accuracy under Different Evaluating Set-
tings. Average RMSEs of different methods are reported.
Method
Default
Input
Prompe
Decoding
Gaussian Noise
Sumpling (t=0.2)
Sampling (t=0,5)
No Hint
Col
Beam
Test Ratio: 20%
Table 4 demonstrates the effectiveness of our approach.
Global Mean
0.112
0.105
0.090
0.127
0.111
More experimental results can be found in the supplemen- tary materials (Section C.3).
Mean of Means
0.090
0.088
0.090
0.102
0,105
0.092
0.088
Ours (Profiles) 0.041
0.055
0.075
0.064
0.045
0.055
0.052
Ours (Models)
0.043
0.045
0.073
0.050
0,040
0.046
0.041
Test Rotio: 80%
Global Mean
0.140
0.115
0.093
0.115
0.131
0.132
0.139
5.4. What Can We Tell Based on Vision Encoders?
Mean of Means
0.119
0.097
0.094
0.099
0.109
0.114
0.123
Ours (Profiles)
0.089
0.099
0.090
0.096
0.082
0.111
0.117
Constrained PMF can capture the impact of model and dataset profiles. Here, we present a showcase analysis fo- cusing on the impact of vision encoder type. Specifically, we calculate the dot product between the feature vector of the vision encoder type, Hm. and the feature vector of the dataset, V. The calculation result measures the influence of a vision encoder on a task. In experiments, CLIP and DINO show improvements on a few datasets, while FNet, SigLIP, and ViT are less effective in comparison. The details are shown in the supplementary material (Section C.4).
Ours (Models)
0.094
0.081
0.092
0.088
0,075
0.092
0.095
Table Performance of Combining Coreset Methods with Our Approach. As seen, coreset methods and our approach are com- plementary. Their combination achieves better performance than either method individually.
Overall
Acc
BART
Method
RMSF.
MAET
RMSE
MAE
RMSF
MAF
Ours
0.193
0.090
0.074
0.052
0.459
0.299
Random Solection
0.224
0.141
0.152
0.107
0.444
0.326
Ourr (Unee)
Ouurs (Avg)
0.139
0.149
0.076
0.088
0.069
0,085
0.061
0.049
0.322
0,313
0.224
0.237
Which Models or Benchmarks are Most 5.5.
Herding
0.2166
0.140
0,155
0.112
0.410
0.297
Informative to Performance Prediction?
Ours (Avg)
0,144
0.088
0.087
0.064
0.305
0.220
.uus (Une)
0,140
0.076
0.070
0.049
0.315
0.220
We assess how representative model is and how informa- tive a benchmark is, by measuring the RMSE improvements of PMF when we add the full results of model or dataset. The most informative models and tasks are shown in Fig. 6. As observed, strong models like GPT-4, Gemini, and In- ternIM are more representative than weaker models. This is likely because their performance tends to deviate from the average and, being more general. they reliably reflect the difficulty level of various datasets. Interestingly, the text-to-image generation task is particularly informative. In this task, models must select the correct generated image from four candidates, and we observe that strong models, such as GPT-4, perform significantly better than others. This performance gap leads to larger errors in PMF, so including this dataset can significantly improve the PMF model.
Center Greedy
0.142
0.154
0.109
0.437
0.322
Ourr (Avg)
0,144
0.088
0.086
0.063
0.306
0.236
Ours (Une)
0.138
0.077
0.070
0.049
0.313
0.224
answer with the option's 's letter!").
Model decoding. (1) Default: greedy decoding, (2) sam- pling with temperature = 0.2, (3) sampling with temperature = 0.5, or (4) beam search with temperature = 0.2 and the number of beams= 10.
We add the results under different evaluation settings into framework and simply use PMF for prediction. Table 3 reports RMSE of different methods and indicate that our framework can handle different evaluation settings.
5.3. Combine Coreset Method with Our Approach
6. Conclusion and Future Work
Previous studies (Polo et al., 2024; Zhang et al., 2024b) select small set of representative samples as a coreset in a benchmark and evaluate models on the coreset. Our work utilizes known model performance from different bench- marks or models and is complementary to existing methods.
Our framework estimates unknown LVLM performances across tasks using PMF, prioritizes evaluations based on uncertainty, and introduces some enhancements to address the sparse data issue. Our study could result in significant savings in development time and computation costs. We highlight two limitations. First, modeling different L.V.M evaluation settings, such 5-shot and chain-of-thought could extend our framework. While some preliminary study is shown in Section 5.2, more exploration is left as future work. Second, it is challenging to apply our framework to new models or datasets without any performance data. We discuss this in the supplementary material (Section C.5).
In experiments, we explore two combination approaches. (1) Avg: simply average predictions of coreset methods and PMF; (2) Une: use uncertainties from MCMC to combine the coreset and PMF predictions. In short, when PMF is confident, we mainly rely on using known performance for prediction, and otherwise, rely on the coreset methods.
Following Zhang et al. (2024b), we use CLIP to generate
9

---

Can We Predict Performance of Large Models across Vision-Languee Tasks?
Acknowledgments
Brinkhaus, H. O., Zielesny, A., Steinbeck, C., and Rajan, molecule images dataset. Journal of Cheminformatics, 14(1):36, 2022.
K.
We would like to extend our deepest appreciation to Qiyuan Zhang, Caixia Zhou, Weijian Deng, Xingjian Leng, Evan Markou, Yunzhong Hou, Yicong Hong. Taojun Lin, Jiahao Zhang, Sam Bahrami, Shu Zou, Zeyu Zhang, Yang Yang. Jiahao Ma, and all our other lab colleagues for their invalu- able support throughout this project. Their collaborative efforts, insightful discussions, and constructive feedback have been crucial in shaping and improving our paper.
Chen, Z., Wu, J., Wang, W., Su, W. Chen, G., Xing. S.. Zhong, M., Zhang, Q., Zhu, X., Lu, L.., Li, B., Luo, P., Lu, T., Qiao, Y., and Dai, J. InternVL: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. arXiv preprint arXiv:2312. 14238, 2023.
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. How far are we to GPT-4V? closing the gap to commercial multi- modal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024.
This work was supported by an Australian Research Coun- cil (ARC) Linkage grant (project number LP21020093],, Qinyu Zhao supported by OpenAl Researcher Access Program (0000005453) and Google Cloud Research Credits Program (353970026).
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 105(10):1865-1883, 2017.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500 2023.
References
Deng, W. and Zheng, L. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15009-15078, 2021.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303.0873 087744 2023.
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, I.,, Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al. VLMEvalKir: An open-source toolkit for evaluating large multi multi-modality models. arXiv preprint 41X1 2024.
Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., John- son, M., Batra, D., Parikh, D., Lee, S., and Anderson, P. NoCaps: Novel object captioning at scale. In Proceed- ings of the IEEE International Conference on Computer Vision, pp. 8948 8948-8957, 2019.
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., al. MME: A comprehen- sive evaluation benchmark for multimodal large language models. arXiv preprint 23Xx:22306.13344 2023.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zimick, C. I.., and Parikh, D. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2425-2423 2015.
Fu, C., Dai, Y., Luo, Y., Li, L.., Ren, S., Zhang, R., Wang. Z.. Zhou, C., Shen, Y.. Zhang, M., et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075. 2024.
Arevalo, J., Solorio, T., Montes-y Gómez, M., and González, F. A. Gated multimodal units for information fusion. 0170720...992...... 2017.
arXiv preprint
Arora, S., Cohen, N., Hu, W. and Luo, Y. Implicit regular- ization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. Challenges in representation learn- ing: A report on three machine learning contests. In Neural information processing: 20th international con- ference, ICONIP 2013, daegu, korea, november 3-7, 2013. Proceedings, Part 20, pp. 117-124, Springer, 2013.
Bai, S.. Chen, K., Liu, X., Wang. J., Ge, W., Song. S., Dang, K. Wang, P. Wang, S., Tang, J., et al. Qwen2.5-VL. technical report. arXiv preprint 25X2 2025.
Berg, R. v. d., Kipf, T. N., and Welling, M. Graph convolutional matrix completion. arXiv preprint arXiv:/706.000 02263, 2017.
Guillory, D., Shankar, V., Ebrahimi, S., Darrell, T., and Schmidt, L. Predicting with confidence on unseen dis- tributions. In Proceedings of the IEEE/CVF Conference
10

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
on Computer Vision and Pattern Recognition, pp. 1134- 1144, 2021.
Leiva, L. A., Hota, A., and Oulasvirta, A. ENRICO: A high- quality dataset for topic modeling of mobile ui designs. Proc. MobileHICI extended abstracts, 2020.
Hao, Y., Gu, J., Wang, H. W., Li, L.,, Yang, Z., Wang, L., and Cheng, Y. Can MLLMs reason in multimodality? EMMA: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444, 2025.
Lewandowski, D., Kurowicka, D., and Joe, H. Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100(9); 0889-2001, 2009.
He, X., Zhang, Y., Mou, L., Xing. E., and Xie, P. PatVVQ: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020.
Li, B., Ge, Y., Ge, Y., Wang, G., Wang, R., Zhang, R., and Shan, Y. SEED-Bench-2 Benchmarking multimodal large language models. arXiv preprint arXiv:231.17099,, 2023a.
Hessel, J., Marasović, A., Hwang, J. D., Lee, L., Da, J., Zellers, R., Mankoff, R., and Choi, Y. Do androids laugh at electric sheep? humor "undeestandinng" benchmarks from the new yorker caption contest. arXiv preprint 2xXiv:2209.06293, 2022.
Li, J. and Lu, W. A survey on benchmarks of multimodal large language models. arXiv preprint urXiv:2408.008 08632, 2024.
Hoffman, M. D., Gelman, A., al. The No-U-Turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(1):
Li Y., Du, Y., Zhou, K., Wang, J., Zhao. W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305.10355 2023b.
2014.
Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T., and
Zou, J. Leveraging medical twitter to build a visual- language foundation model for pathology AI. bioRxiv, pp. 6023-03,20 2023.
Liang, P. P., Goindani, A., Chafckar, T., Mathur, L., Yu, H., Salakhutdinov, R., and Morency, L.-P. HEMM: Holis- tic evaluation of multimodal foundation models. arXiv preprint ar arXiv:4407.03418 2024.
Hudson, D. A. and Manning, C. D. GQA: A new dataset for worldworld visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700- 2019.
Liu, B., Zhan, L.-M., Xu, L., Ma, L.,, Yang. Y., and Wu, X.- M. SLAKE: A semantically- dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imag-
knowledge-enhanced
ing (ISBI), pp. 1650-1654. IEEE, 2021.
Hwang, E. and Shwartz, V. MemeCap: A dataset for captioning and interpreting memes. arXiv preprint arXiv:2305. 13703, 2023.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved base- lines with visual instruction tuning. arXiv preprint arXiv:2310003744, 2023a.
Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., and Sadigh, D. Prismatic VLMs: Investigating the design space of visually-condittinne language models. arXiv preprint arXiv:2402.07865 2024.
Liu, H., Li, C.,, Wu, Q., and Lee, Y.J. Visual instruction tun- ing. Advances in Neural Information Processing Systems, 36, 2024.
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., and Testuggine, D. The hateful memes challenge: Detecting hate speech in multi-modal memes. Advances in Neural Information Processing Systems, 33: 2020.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., Hc, C., Liu, Z., et al. MMBench: Is your multi modal model an all-around player? arXiv preprint arXiv:2307.062.
2023b.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. Inter- national Journal of Computer Vision, 123:32-73, 2017.
Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Yang, H., et al. DeepSeek-Ve VL: towards real-world lision-language understanding. arXiv preprint arXiv:2403.05525, 2024.
Lu, P., Mishra, S., Xia, T., Qiu, I.., Chang, K.-W., Zhu, S.-C., Tafford, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 35:2507-2521, 2022.
Lau, J. J., Gayen, S., Ben Abacha, A., and Demner- Fushman, D. A dataset clinically generated visual questions and answers about radiology images. Scientific Data, 5(1):1-10, 2018.

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. p 3195-3204, 2019.
Metaphor! In Proceedings of the 14th International Work- shop Semantic Evaluation (SemEval-200) Barcelona, Spain, Sep 2020. Association for Computational Linguis- tics.
Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.-Y. MPNet: Masked and permuted pre-training for language under- standing. Advances in Neural Information Processing Systems, 33:16857-16867 2020.
Mnih, A. and Salakhutdinov, R. R. Probabilistic matrix factorization. Advances in Neural Information Processing Systems, 20, 2007.
Neal, Handbook of Markov chain Monte Carlo, volume 2, chapter MCMC using hamiltonian dynamics, 2011.
Suhr, A., Lewis, M., Yeh, J., and Artzi, Y. corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pp. 217-223, 2017.
OpenAL. GPT-4o. https: /pwp/appnnao///////e///.e/...... www.co////////// 02/, 2024. Accessed: 2025-05-29.
Perlitz, Y., Bandel, E., Gera, A., Arviv, O., Ein-Dor, L.. Shnarch, E., Slonim, N., Shmueli-Scheuer, M., and
Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., and Artzi, Y. A corpus for reasoning about natural language grounded in photographs. arXiv preprint 1.XY::8811.0041,, 2018.
Choshen, L. Efficient benchmarking (of language mod- els). arXiv preprint arXiv:2308. 1696, 2023.
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., and Yurochkin, M. Benchmaakss Evaluating LLMs with fewer examples. arXiv preprint arXiv:2402. 14992, 2024.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu,
J., Soricut, R., Schalkwyk, Dai, A. M., Hauth, A., et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:23/2.11805 2023.
Prabhu, A., Udandarao, V., Torr, P., Bethge, M., Bibi, A., and Albanie, S. Lifelong benchmarks: Efficient model evaluation in an era of rapid progress. arXiv preprint arXiv:2402.19472. 2024.
Thrush, T., Jiang, R., Bartolo, M.. Singh, A., Williams, A.. Kiela, D., and Ross, C. Winoground: Probing vision and language models for visio- linguiiig compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5238-5248, 2022.
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T., Alayrac. J.-b., Soricut, R., Lazaridou, A., Firat, O., S.hhiitwiieeer, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint ar arXiv:2433.055330 2024.
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. Cambrian-1: A fully open, vision-centric exploration of multimodal LLMs. arXiv preprint arXiv:2406. 168600, 2024a
Rodriguez, P., Barrow, J., Hoyle, A. M. Lalor, J. P., Jia, R., and Boyd- Grader, J. Evaluation examples are not equally informative: How should that change NLP leader- boards? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pp. 4486-4503. 2021.
Tong, S., Liu, Z., Zhai Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal LLMs. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9068-558999999 9578, 2024b.
Salakhutdinov, R. and Mnih, A. Bayesian probabilistic matrix factorization using Markov chain Monte Carlo. In Proceedings of the 25th International Conference on Machine Learning, pp. 880-888 887, 2008
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C.. Shepard, A., Adam, H., Perona, P., and Belongie, S. The iNaturalist species classification and detection dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8769-8778, 2018.
Schaeffer, R., Valentine, D., Bailey, L., Chua, J., Eyzaguirre, C., Durante, Z., Benton, J., Miranda, B., Sleight, H., Hughes, J., et When do universal image jailbreaks transfer between lasoun-language models? arXiv preprint arXiv:2407.15211, 2024.
Wang, B., Li, G., Zhou, X., Chen, Z.,, Grossman, T., and Li, Y. Screen2Worde Automatic mobile UI summarization with multimodal learning. In The 34th Annual ACM Symposium on User Interface Software and Technology. pp. 498-510, 2021.
Sharma, C., Paka, Scott, W., Bhageria, D., Das, A., Poria, S., Chakraborty, T., and Gambäck, B. Task Report: Mem- otion Analysis 1.0 SemEval 2020: The Visuo-Lingual
12

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
Wang, K., Pan, J., Shi, W., Lu, Z., Ren, H., Zhou, A., Zhan, M., and Li, H. Measuring multimodal mathematical reasoning with Math Vision dataset. Advances in Neu- ral Information Processing Systems, 37:95095-95169 2024a.
Zhang, G., Du, X., Chen, B., Liang, Y., Luo, T., Zheng, T., Zhu, K., Cheng, Y., Xu, C., Guo, S., al. CMMMU: A chinese massive multi-discipline multi-modal under- standing benchmark. arXiv preprint arXiv:2401.11944, 2024a
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., al. Qwen2-VL: Enhanc- ing rision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Zhang, K., Li, B., Zhang, P., Pu, F., Cahyono, J. A., Hu, K., Liu, S., Zhang, Y., Yang, J., Li, C., et al. LMMs- Eval: Reality check on the evaluation of large multimodal models. arXiv preprint urXiv:2407. 12772, 2024b.
Wu, H., Li, D., Chen, B., and Li, J. benchmark for long-context interleaved -atouaan understanding. Advances in Neural Information Process- ing Systems, 37:28828 28857, 2024.
A
Zhang, Q. Lyu, F., Liu, X., and Ma, C. Collaborative performance prediction for large language models. arXiv preprint MxXiv:2407.01300 2024c.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. BERTScore: Evaluating text generation with BERT, arXiv preprint arXiv:1904.0961
Yang. Y. and Newsam, S. Bag-of- viss-vl-vooltte and spatial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL international conference on ad- vances in geographic information systems. pp. 270-279, 2010.
2019.
Zhang, Z., Zhao, X., Fang. X., Li, C., Liu, X., Min, X., Duan, H., Chen, K., and Zhai, G. Redundancy principles for MILMs benchmarks. arXiv preprint 25X1v:2501.13953, 2025.
Yang, Y., Wang, W., Chen, Z.., Dai, J., and Zheng, L. Bound- ing box stability against feature dropout reflects detec- tor generalization across environments. arXiv preprinl arXiv:2403.13803 2024.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing. E., al. Judg- ing LLM-as-a-judge with MT-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Ye, Q.. Xu, H., Ye, J., Yan, M., Liu, H., Qian, Q., Zhang, J., Huang. F., and Zhou, J. mPLUG-Ow12: Revolution- izing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04255 2023.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. MiniGPT-4: Enhancing lannu-l understanding with advanced large language models. arXiv preprint 2304. 2304:2090.. 2023.
Yosef. R., Bitton, Y., and Shahaf, D. IRFL: Image recognition of figurative language. arXiv preprint arXiv:2303. 15445, 2023.
Zhu, K., Zang, Q., Jia, S., Wu, S., Fang, F., Li, Y., Guo, S., Zheng, T., Li, B., Wu, H., et al. LIME-M: Less is more for evaluation of MILMs. arXiv preprint ar axXiv:2409.068 2024.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Lin- guistics, 2:67-78, 2014.
Zohar, O., Wang, X., Dubois, Y., Mehta, N., Xiao, T. Hansen-Estruch, P., Yu, L., Wang, X., Juefei-Xu, F. Zhang, N., et al. Apollo: An exploration of video un- derstanding in large multimodal models. arXiv preprint 241 10360, 2024.
Yuan, W., Neubig, G., and Liu, P. BARTScore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 02272633272777 2021.
Yue, X., Ni, Y.. Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. MMMU: A massive multi- dilli-dissipliee multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Zellers, R., Bisk, Y., Farhadi, A., and Choi, Y. From recog- nition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2019.
13

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
A. Estimation of Evaluation Cost
learning rate is le-3 and the batch size is 256. The embed- ding dimension of each user item is 10, which is the same for PMF. We train DMF for 200 epochs and the result of the best epoch is reported.
To estimate evaluation cost of LVLMs, we evaluate a rep- resentative model, -ren2..-VV-IInstuuce 7B (Wang al., 2024b) based on LMls-Eval (Zhang et al., 2024b) with one A100 GPU. The results are shown in Table 5.
For GCMC, we refer to the GitHub implementation 0.7, learning rate is 0.01, hidden units are [500, 75] in 1st and 2nd layers, accumulation method is "stack", the number of basis functions is 2, and the model is trained for 200 epochs. We note two main issues when using GCMC:
Dropout ratio is
As shown in the first and second columns, although vllm significantly accelerate the evaluation, the computational cost is still high. Assuming that we are comparing 5 7B models on 10 similar-scale benchmarks, the evaluation will take around 21.7 hours. Besides, in our experiments, vllm usually leads to slight performance decrease.
First, GCMC handles discrete rating levels and treats each rating level as a separate class (see Section 2.3 in Berg et al. (2017)), which is not suitable for our setting, because we also use continuous ratings like the BART scores. To address this, we only use LVLM benchmarks with accuracy as the main metrie, and discretize accuracy into 101 classes, i.e., O, .... 100. Second, when training data is sparse, some classes (for example, accuracy = 67) do not occur in training set, leading to running errors in the code.
LMMs-Lite (a coreset method) significantly reduces evalua- tion cost, indicating reducing evaluation cost is still valuable in practice. As shown in Section 5.3, coreset methods may get inaccurate results, while our method improves them.
B. Comprehensive Evaluation of LVLMs
We provide a comprehensive overview of the datasets and LVLMs used in our study. Detailed dataset information can be found in Table 6 and 7, while the model profiles are presented in Tables 8 and 9.
Table 10 summarizes the average results from 10 experi- ments with different random seeds. As shown, PMF demon- strates superior performance on our dataset compared to DMF and GCMC. We notice that deep matrix completion methods are commonly applied in recommender systems, where there are usually thousands of users and items, with millions of samples. But in our setting, we have 108 models, 176 datasets, and 19K samples. Thus, we build our method on a simple but strong baseline, PMF, rather than neural networks, which are possibly more data-hungry.
A heatmap illustrating the model ranking across datasets is shown in Fig. 7. Additionally, the correlation analysis of performance scores is illustrated in Fig. 8 and 9. Notably, even within the same model family, such as the LLaVA series, the rankings between models do not exhibit strong correlation. Datasets tend to have much more consistent ranking correlations, suggesting that models performing well on one dataset are likely to rank highly on others.
C.2. Detailed Performance Evaluations of PMF and PTF
Besides, we investigate the impact of different latent dimen- sions in the PMF models and find that relatively small latent dimension, around 10, is sufficient. As shown in Fig. 10, increasing the latent dimension reduces the RMSE on the training data to zero due to overfitting, but it does not lead to significant improvements in RMSE on the testing data. Additionally, when we extract the singular values of the score matrix, we observe that the top singular values are much larger than the rest, indicating that most of the information is captured by few dimensions. This suggests a high degree of similarity in performance scores.
We present detailed performance evaluations of PMF in Table 11 and PTF in Table 12. As shown, our methods consistently outperform the baselines. In scenarios where performance data is sparse, our enhancements significantly improves the prediction accuracy of PMF.
We also include the following ranking- metrics.
Spearman's rank correlation.
Kendall rank correlation.
Precision @ K. The proportion of the predicted top I model that fall within the top K positions of the ground-truth rank- ing. For example, our method predicts LLa is the best on the task and it will be correct if VL. within top 5 of the fround-truth ranking for Precision@5.
C. Further Experimental Results
C.1. Comparing PMF and Deep Learning-Bass Methods
We evaluate Deep Matrix Factorization (DMF) (Arora al., 2019) and Graph Convolutional Matrix Completion (GCMC) (Berg et al., 2017) for comparison.
Recall @K. The proportion of the ground- -ooun best ones that is correctly retrieved within our top K predicted results. For instance, if LLaVA-1.5 is the best model, it must be within the top 3 predictions of our method for Recall@3.
For DMF, we use MSE loss and the Adam optimizer. The
14

---

Can We Predict Performance of Large Models across Vision-Languen Tasks?
As shown in Table 13, ranking-based metrics also supports the advantage of our methods. Baseline methods typically achieve very high precision but low recall, while our meth- ods provide more balanced precision and recall, as well as improved rank correlation.
"Une 50%" and "Unc 30%" mean keeping the 50% or 30% most confident predictions based on our estimated uncer- tainty, which further improve the estimation accuracy. But note that this may lead to fewer predicted results.
C.3. Combining Coreset Methods with Our Approach
We sample different ratios of coreset samples from the benchmarks, and the results are shown in Table 14.
C.4. What Can We Tell Based on Vision Encoders?
Detailed results shown in Fig. 11.
C.5. Generalization New Models and Datasets
We also investigate the models' ability to generalize to new models and datasets without any performance scores for training. As illustrated in Fig. 12, using model and dataset profiles provides slight improvement for new models or datasets. However, when both the model and dataset are entirely new, performance falls below the Global Mean baseline. But we argue that this situation is rare in practice. Some initial performance scores are usually available when model dataset is released, and the community usually reports performance scores in subsequent works.
15

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
Table 5. Evaluation Cost in different settings.
Setting
Basic
vilm
L.Ms-Lite
Larger model
Video benchmarks
Test-time scaling
Use vllm
N
Y
Y
N (no meemory) 32B
Y
Y
Model Size
7B
7B
7B
Dulaset
MMBench En
MMBench En
MMBench En Lite
MMBennch Ein
VideoMME 2.7K
MathVissin
Num of Samples
6.72K
6.72K
500
6.72K
5.14K
Real Performance?
Y
Possibly lower
Estimated
Y
Y
Time
24min 0.21s
13min
Imin
45min
2-
Ih
Time per sample
0.11s
0.11s
0.40s
>1s
0,70s
Vidco loading and proprocessing are the botteneck.
Table Dataset Information. Our study utilizes 36 benchmarks. For larger benchmarks such as SEED-2,, we divide them into sub- datascts based on task categories. To reduce computational costs, subsample certain benchmarks.
Benchmazk
Datasets For GPT Gemini
No.
No. of Samples
No. of Samples
Other Models
Downlead URL
SEED 2 (13 at. 20238)
27
2606
24371
hitps
2023)
2374
http:///uggingtiie...o wta.gn/laee..ollaatsssseeee lab/ME
MMBench CN (1ib et al. 2023)
20
1994
4329
higs: hhip:///gginggseee www.laoweeh/mmmmm-a-MMBBa
MMBeacc EN (Liu al., 2023b)
20
1994
4329
http://hogggtn /tt.g//ggggaaeeecooooooo/o// www.eettttttteeeeeemsssm lab/MMBeacch
MMMU (Yur 2024)
30
900
900
hegpsl/ligggggaan www..oalsss..eeeemmmm. lab/MMMU
(Zlang
2024x)
573
hipp www.://lggginggaano www..//wwwba..ssssosmoo... MMMMMMMMMMMM
ScienceQA (Lu al.
1467
2017
hups Whigging www.nngatee...ooo woddaassssssssssssssss Jab/ScieaceeAA
CVBench (Tong et al., 202
400
2638
htps /fh www.ggggggaaee...mm wwwignaaw.colaaateeesssseeeeesee wwwltutttlttttttteetoooo wisteesssssoooiiiiuCCCC Dench
POPE (1i et 20235)
980
900
higss gilhih com Domp/AiDDgeeePPPP
DECIMER (Brinkhaus
2022)
100
hute wwwww/ kagale com www..//aoo..yyllaaaaaaaaaaa whw.s//wwhigggabboosssssaaem.m
Entioo (Leiva el al., 2020)
100
100
hups: Weseerweeercct en facess anloo fieerice!
FaceEmotion (Goodflloww et al., 2013)
100
100
higps wwwww kaggle comelaustt-mmmmmarreete
Fiskk20k (Young 2014)
100
100
hups hupp///www Laggle wwu.daaaeeet/aaamssrrraaa image datuset
GQA dloudoo&& Manning, 2019)
hitp stamford
HatefulMemes (Kiela et al., 2020)
100
100
hups wwww. kaggle com daaastteppptttiiccacc.... hateful meme dataset hups //ml maat competiion datasets s3 amazonaws coma/ 2 taal hips Wwwggigggnaeeeeee wwwlas.eess//lmmme. herFpe////
INAT (Van Horr et al., 2018)
100
100
IRFL. al.. 2023)
100
100
MameLaps (Hwang Shwartz, 2012)
100
100
hutge
www.///llwwwnggg/nn..o meme capitrelmmin
Memotion (Shazma al., 2020)
100
100
hupe: We w kaggle com womddiiaeee//illll www.a///wwwwlllm........o Ammemttio dataset 7kk
MMIMD6 (Aresalo et ol., 2017)
100
100
htips //t www.ggggaaaeee..ooo www.itoeeettsssoooooooo.oo www.shhyyliimmmii...o
New Yorkerreeratooo (Hessel 2022)
100
100
//github wtt.iteee///iiiinn contest data
NLVR al.
100
NNVV22 et al., 2018)
100
100
hing /f githhb comniill labiutiv
NoCapp lAggrwaa et al., 2019))
100
100
hups hhw.ggiggtae.e....oomoooo endutteeralssaayttei
Jest
OKYQA (Manio al. 2019)
100
100
hups Hokk allenai orghdownioon hcml
OpenPatt (Huang al., 2023)
100
100
Path/QA (He 2020)
100
100
hups I/ g/ghhhub cc//USSDD AUSH/atlVQA
Resisc15 (Cheng et al., 201
100
100
hups htwp:///wwww..o.. laggle wwwpoeteeeehappyyygalalloo data set
ScremnZWords (Wing
100
100
/twww/wwwwoo..o Kaggle com-dataeees 199aric
al., 2021)
100
UCMerced (Yang & Newsam, 2010))
100
100
lings /www luggle com/endehpalll Seetanndd-eeeddeee weas---ee-ssssiiiiteeee classffictionn
VCR iZellers et 2019)
100
100
linps wowmalooemooossee ense, com/downlood/
VisualGenome (Krishna
2017)
100
100
Whomees.s......... wwshingtun.cco runass/isalleemmme
VQA (Amoll al., 2015)
100
100)
lmng Wwisnalgg org lgga Adwulladd hrai
VQARAD (Laa et al., 2018)
100
100
huns Whwgginggaaeee....ooooooooooom.e htw.:e/pw........eeeoeeeaaaaaaan rad
Winoground (Thrush et al, 2022)
100
100
hups hwt.:///ggggggggg..c.. www.://www.............ooooeeone
MutiVisiaa (Wang
202461
Not evalianed
3040
Whagginggha htt.://www...iooaaeeeaaaaaaaaaaa
EMMA (Hao et 2025)
Not evaluated
2788
Vidas-MME: (Fi al., 2014)
Not evaluatal
2700
laps Whegging www.ae/wwww.........o...ooaaaa..
Long VidenBBanch (Wu et al. 2024)
Not evalnated
1337
linss whg.giiggaae...oom www..//www....eee.....eeeeeeeeee VidaaBesch
16

---

Can We Predict Performance of Large Models across Vision-Languen Tasks?
Table Dataset Metrics. PMF models the main metric the datasets, while PTF utilizes the main and other metries (six in total) in modeling. BARTScore is proposed by Yuan et al. (2021), while BERTScore is introduced by Zhang et al. (2019).
Benchmark
Main Metric
Other Metries
SEED (Li 2023g)
Accunacy
MMF al., 2003)
Accunacy
Precision, Recall, Fl
MMBench CN (Liu et al, 20236)
Accuncy
20236)
(Yue et 2024)
Accuncy
CMMMU (Zhamg al. 2024a)
Aucuracy
ScienceAA (Iu 2022)
Accuncy
20244)
Accuracy
eL al.,
Ascuracy
Precision, Recall,
DECTMER (Brinkhams ol., 2022)
BARTScure
BERT
Farico (Tatva 2030)
BARTScore
BERTSove
(Grodfellow
BERTS
Flickr30k (Young et 2014)
BARTScore
BERTSuvee
GQA (LEadeen & Manning. 2019)
BARTScore
BERT
Hateffuleeess (Kiela 2020)
BARTScure
BARTScore
BERTS
ct 2023)
BARTScore
Sooe
MemeCaps (1wangg & Shwartz, 2023)
BARTScure
Seore
Memotion (Slurna al. 2020)
BARTScore
BERTSove
at.
SERe
YorkeeCnntoon (Nnoee et al., 2022)
BARTScure
BERTS
NLVR (Sube al., 2017)
BAKTScore
BEKI Score
NLVR2 (Subr 2018)
BARTScure
(Agrowal
BERTS
OXVQA (Mizino at., 2019)
BARTScore
OpenPuth (llungg al., 20233
BARTScure
BERTScONE
2020) Resisct5 (Cheng
PuhhVA (Te et al., 2020)
BARTScore
BERTS
BARTScore
Scmen2WWrlss (Wag
2021)
BARTScore
BERTS
Slake (Liu et al., 2021)
BAKTScore
BEKI Score
UCMerced (Yang Newsum,
BARTScore
BERTS
(Zellons
BARTScure
BERTS
VisualGienome (Knssina all, 2017)
BARTScore
BEOES
VQA et 20 2015)
BARTScore
BERT Score
VQARAD (Lau 2018)
BARTScure
BERTS
Winoground (1laush
2022)
BAKTScore
Math Vision (Wang al... 2024gg
Accuracy
EMMA (Hao 2025)
Accuracy
Video
2024)
Accuracy
Long (Wu
Accuracy
17

---

Can We Predict Performance of Large Models across Vision-Languean Tasks?
Table Model Information. Our study evaluates 108 models. each model, we report the number parameters the LLM backbone, the vision encoder, and the model family that define.
Model
Checkpoint R1J12-cpt-2.78
No. Param, in LLM (B)
Vision Encoddr
Model Family
VIT
HI
BLIP22-ann4-ed
BLIP2- ULLI22-qg--6.76-2000
6.7
VII
BLIP
1P2-opt-6677
6.7
VIT
RI.IP
BLIP22-0n--5--
Vicuna-7B
13
VIT
HI.IP
InstmetLLB RIP dan-t-xl
VII
BLIP
MiniGiTT-Vicuall--
VII
MiniGPT4
MiniGPT4 Vicumut--
13
VI
MiniGPT4
MPLUGGOWI
HPL.15-D122.1MM22-77
MiniGPP4
mPLUG-OWi2_1
LLAV
HLVA--7B
C1IPP
ELaVA
LLaVA-13B
13
CLIP
LLaVA
Vicunaa
Vioann-
LLaVAs1 6-Misral-78
CLIP
LLaVA
[LVA-vl 6-34B
34
CLIP
[LaVA
Cambrian-1
CambrianPPe11-38
DINO2
Cambrian-BR
SigLIP,
ComNeXX, DINOw2
Cumbrian11 13B
13
CLIP. SigL.P.
ComNNeXi DINOw2
Cambrian
Cambrian-4R
34
CLIP Sigl ComNeXL, DINOW
Cambrian
Riyu-SB
M.MMA. Adayter
ILMMA-Adape----BBAA--
ULMA-AApper
CLIP
CLIP
Audper
OpenFlamingo
OpenFaaingg-BB--illlmmg
OpenFuaninga
Ope
OpenFlaminga
Donnflaingg--BB-iill
NTNel
OpenFuaming
Don
NPNal
OpenFaminga
OpenFlaiinga
Qwen-VL.
Oomm-VL-Chi.
VIT
Qwen
Qum22-V1-22-Imiiin
VI
Qwen
Owen2 VL-7B Instruct
VIT
Qwen
Qwen
Qwen2.3 5 VL-32B Instruel
32
VI
Qwen
DeepSeck-V VI.
DuupSeck-V..2-ing V1.2-iny
10
CLIP
Qwen
IniemL IniemLM..Compuuet
InieraLM XComposer
CLIP
IntemLM
Iniernl M-XXMMXXXMeeee XCoppose-
CLIIP
IntenE
InteraLM XCompeser2
CLIP
IntenLM
Internl O0-
CLIP
GPTI
gpl-4o 2024 05 13
Unknown
Unkaown
GPT4
gpi-lo-2024 200--022----000
Unknown
Unknown
GPT4
gol-4e-mini-204--00
Unknown
Unknown
GPT4
gpt-4 turbo 2024-05-04
Uaknowa
Unksown
GPT4
Gemini
gemin: 1,5
Unkmown
Unknown
Gemini
1.5 1.5-flshh
Unknown
Unknowa
Gemini
18

---

Can We Predict Performance of Large Models across Vision-Languee Tasks?
Table 9, Model Information. This is continued table Table
Model Prismatic
Checkpuini
No. Param. in LLM
Visien Encuder
Model Family
reproduction Bava-v15577
CLIP
prism
eppoddettio--laaae one sluge+7b ine-sigge+1 3b
primm
CLIP
prism
13
CLIP
prism
full ft multi suge+?b
CLIP
[ull-fl-oee stgge77
prism
fnlk-224px+76
prism
dinov2 22
DINO<2
prism
clip 2ppp222/px+7b
CLIP
224pt-77
prism
clip 3.ppx resize crupebb
CLIP
prism
clip-336pp 36ps resixe na sighip 154pt-leterbot-
CLIP
prism
Sigl
prism
Higlip 381p resize chop+76
prism
sighip 384pt resize muive+?h
SigLIP
prism
dinectip 336px lettecbox+7
CLJP, DINOV2
prism
dinoclip 356ps
naive+7b
CLJF, DINOv2
dinusiglip- 38dpx
DINOv2
prism
dinosiglip- 384px resize naivelib
SigLIP DINO12
pram
Lama2277
CLIP
prism
[aama=]366
CLIP
6o-cotraining+76
priam
Laamal no cocraning+7
CLIP
prism
train 1.20-11.55... spochs778
CLIP
prism
epochs+bb
CLIP
train: 2-epochs+7b
prism
tmai---epoche+7bb
CLIP
prism
Lava Iv
CLIP
prism
laaa--rve Jawa-lrv+bb
CLIP
prism
prism clip controllde?bb
CLIP
prism
friim-clip-comtllldeee
CLIP
prism
clipe7b priom
prism
SigLIP
prism
prism siglip 103
SigLIP
prism
siplip+bb
SigLIP
prism-sgglip1130
prism
prism dinosigli-ccottollee----
SigLIP. DINOV2
prism
prism-dinosiglip-oae friim-diosiglpp-oeenoolll.
SigL DINOW2
prism
prism-duoosigii--7
prism
priam dinestglip-116
SigLIF,
prim
SigL DINO12
prism
prism www.diiggggii22pprr..
SigLIP DINOV2
prism
Liama2 chatel
CLIP
mistral-B.147b
prism
CLIP
prism
phi-243h
CL.IP
prism
prism
prism
gomma-incmmis+2hhllee dindisiilipp
Sigl IP. DINOW
prism
gemma- instees--tt Shlp
CLIP
prism
Whb.snnttthhhiiigllllle sisiig
gemmo-incimc--gha didinsiiipp
primm
[lmmz-chus Pbecllp 7bsclip
CLIP
prism
samal2chaat1 7bbssgglip
SigLIP
prism
[1ama3-chut diddnossiip
DINOW2
[lamaT-insireett
prism
llann -intsintttt----seiipp
Sigl TP
prism
SigT TP, DINOW2
prism
mistml-asstmee--l
prism
mistml--nsmmc--dl Thssipip
Sigl
primm
misrml-assrass-dl 24 dinnsiig
Sigl DINOW2
prism
Table 10. Comparison of PMF and Deep Learning Based Methods.
Overall
Ace
BART
Method
RMSEJ
MAEJ
RMSE
MAE
RMSE
MAE
ratio: 20%
DMF [1]
0.225
0.105
0.086
0.060
0.538
0.353
[2]
0.139
PMF
0.193
0.090
0.074
0.052
0.461
0.303
Test ratio: 80%
DMF [1]
0.561
0.314
0.289
0.209
1.26
0.896
[2]
PMF
0.317
0.174
0.156
0.115
0.723
0.504
19

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
Figure 7. Heatmap of Model Rankings on Each Dataset.
20

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
Figure 8, Heatmap of Correlation in Model Ranking.
21

---

Can We Predict Performance of Large Models across Vision-Languege Tasks?
Figure 9. Heatmap of Ranking Correlation on Datasets.
22

---

Can We Predict Performance of Large Models across Vision-Languaee Tasks?
100
10
0.20
80
SNE Composent
RMSS
0.15
40-
0.10
Train
20-
Test
0.05
-10
20
20
10
Number Lazent Dimensions
Indox (B)
t-SNE Component
(A)
(C)
Figure 10. Low-Rank Property of the Score Matrix. (A) RMSE on the test set for PMF stabilizes when the latent dimension exceeds 15. (B) The top singular values of performance matrix are significantly larger others. (C) I-SNE visualization of dataset clusters.
Table 11. Detailed Performance of PME Superior results are highlighted.
Overall
Acc
BART
Method
RMSEL
MAES
RMSE
MAE
R²
RMSE
MAE
R²
Tost Ratio: 20%
Global Mean
0.367
0,220
0.190
0.148
0.475
0.823
0.611
0.549
PMF
0.192
0.090
0.073
0.051
0.922
0.458
0.303
0,860
Test Ratio: 40%
Glohal Mean
0.368
0.220
0.190
0.149
0.477
0.829
0.614
0.551
Mcan Of Mcans
0.320
0.186
0.161
0.125
0.623
0.725
0.527
0.657
PME
0.199
0.005
0.078
0.056
0.911
0.474
0.314
0.853
Test Ratio: 60%
Global Mean
0.370
0.220
0.190
0.149
0.474
0.831
0.613
0.551
Mean Of Means
0.322
0.188
0.162
0.126
0.618
0.729
0.529
0.654
PMP
0.220
0.106
0.089
0.063
0.886
0.521
0.348
0.823
Test Ratio: 80%
Global Mean
0.373
0.221
0.837
0.612
0.546
Mean Of Means PMF
0.329
0.191
0.166
0.128
0.601
0.742
0.533
0.643
0.258
0.131
0.114
0.812
0.600
0.407
0.766
'Test Ratio: 90%
Global Mean
0.381
0.226
0.198
0.153
0.430
0.852
0.630
0.529
Mcan Of Means
0.339
0.197
0.174
0.133
0.564
0.765
0.555
0.621
Ratio:
Global Mean
0.458
0.278
0.227
0,182
0.254
1.041
0.807
0.297
Mean Of Means
0.385
0.230
0.191
0.150
0,474
0.875
0.672
0.504
PMF
0.462
0.276
0.228
0.180
0.248
0.805
0.282
23

---

Can We Predict Performance of Large Models across Vision-Language Tasks?
Table 12. Detailed Performance of PTE. Superior results highlighted.
Overall
Ace
Precision
Recall
F1
BART
BERT
Method
RMSEL
MAEL
RMSB
MAE
RMSE
MAE
RMSE
RMSE
RMSE
MAE
Test Ratio: 20%
Global Mean
0.320
0.190
0.190
0.149
0.223
0.167
0.245
0.186
0.205
0.156
0.812
0.603
0.088
0.044
Mean Or Means
0.260
0.150
0.149
0.115
0.181
0.131
0.205
0.152
0.169
0.123
0,664
0.482
0.074
0.036
PMF
0.206
0.196
0.081
0.057
0.130
0.086
0.175
0.126
0.108
0.070
0.563
0.378
0.077
0.039
CPTF
0.208
0.099
0.087
0.060
0.134
0.089
0.173
0.123
0.107
0.070
0.564
0.579
0.076
0.039
BPTF
0.202
0,055
0.079
0,056
0.129
0.084
0.177
0.127
0.109
0.070
0.553
0.372
0.077
0.039
BCPTF
0.207
0.096
0.079
0.129
0.085
0.178
0.127
0.113
0.072
0.568
0.378
0.076
0.039
Test Ratio:
Global Mean
0.323
0.192
0.190
0.149
0.223
0.167
0.247
0.189
0.213
0,160
0,818
0.609
0.091
0.045
Mean OF Means
0.262
0.151
0.150
0.116
0.182
0.132
0.209
0.156
0.174
0.126
0.667
0.486
0.078
0.038
PMF
0.210
0.100
0.083
0.059
0.132
0.087
0.181
0.130
0.112
0.073
0.572
0.388
0.081
0,040
CPTF
0.209
0.101
0.087
0.062
0.134
0.089
0,180
0.128
0.113
0.074
0,566
0.385
0.081
0,040
0.060
0.132
0.086
0.185
0.133
0.117
0.558
0.379
0,082
BCPTF
0.209
0,100
0.082
0.059
0.131
0.086
0.184
0.131
0.117
0.076
0.568
0.384
0.081
0.040
Test Ratio: 60%
Global Mean
0.325
0.192
0.191
0.149
0.227
0.170
0.248
0.189
0.214
0.160
0.825
0.611
0.092
0.045
Mean Of Mcans
0.265
0.153
0.151
0.116
0.186
0.135
0.212
0.157
0.178
0.128
0.676
0.490
0.080
0.038
PMF
0.218
0,107
0.093
0.067
0.136
0.090
0.188
0.134
0.123
0.081
0,588
0.400
0.084
0.041
CPTF
0.098
0.070
0.141
0.094
0.187
0,133
0.125
0.040
BPTF
0.217
0.108
0.096
0.068
0.138
0.092
0.194
0.139
0.130
0.087
0.582
0.397
0.085
0.042
BCPTF
0.216
0.105
0.089
0.064
0.135
0.090
0.191
0.136
0.127
0.083
0.584
0.394
0.083
0.041
Test Ratio: 80%
Global Mean
0.330
0.194
0.193
0.150
0.230
0.171
0.253
0.191
0.217
0.162
0.839
0.619
0.092
0,046
Msan Of Mcans
0.277
0.158
0.155
0,119
0.198
0.141
0.225
0.165
0.186
0.134
0.709
0.510
0.084
0.041
PMF
0.120
0.087
0.151
0.103
0,207
0.148
0,098
CPTF
0.240
0.123
0.115
0.083
0.151
0.104
0,208
0,148
0.147
0.099
0.637
0.437
0.088
0.043
BPTF
0.239
0.123
0.116
0,083
0.152
0.103
0.212
0.151
0.151
0.102
0.630
0.433
0.050
0.044
BCPTF
0.236
0.119
0.108
0.077
0.147
0.099
0,208
0.149
0.147
0.099
0.627
0.427
0.089
0,043
Test Ratio: 90%
Global Mcan
0.338
0.198
0.199
0.154
0.237
0.174
0.258
0.195
0.224
0.166
0.858
0.629
0.095
0.047
Means
0.298
0.166
0.125
0.216
0.153
0.239
0.176
0.090
0.043
PMF
0.294
0.161
0.161
0.119
0.194
0.135
0.235
0.171
0.187
0,131
0,761
0.535
0.094
0.045
CPTF
0.274
0.147
0.145
0.105
0,190
0.133
0.233
0.168
0.184
0.128
0.710
0.492
0.092
0.043
BPTF
0.267
0.143
0.142
0.103
0.179
0.123
0,252
0,167
0.178
0.122
0.690
0.480
0.093
0.044
BCPTF
0.268
0.141
0.138
0.099
0.179
0.124
0.228
0.166
0.176
0.120
0.698
0.481
0.093
0.045
Test Ratio: 95%
Global Mean
0.404
0.228
0.182
0.251
0.194
0.270
0.209
0.240
0.183
0.057
Mean OF Means
0.387
0.217
0.202
0.158
0.241
0.182
0.261
0.198
0.230
0,172
1.027
0.772
0.097
0.056
PMF
0.403
0.233
0.223
0.177
0.244
0.185
0.269
0.204
0.236
0.175
1.059
0.801
0.101
0.057
CPTF
0.364
0.199
0.188
0.142
0,216
0.161
0.252
0.188
0.216
0.157
0,970
0.712
0,097
0.054
BPTF
0.355
0.196
0.189
0.144
0.208
0.153
0,251
0.188
0.206
0.148
0.940
0.687
0.098
0.055
BCPTF
0.360
0.196
0.186
0.141
0.211
0.156
0.251
0.186
0.205
0.148
0.959
0.702
0.100
0.056
Table 13, Ranking- -aasid Metrics. Superior results are highlighted.
Method
Spearmant
Kendall
Precision@: I1
Precision P
51
Recall@ 11
Recall@: Recall@5
Global Mean
0.76
0.75
0.96
1.00
0.75
0.50
Mean Of Means
0.90
0,80
0.75
0.97
1.00
0.75
0.79
0.80
PMF
0.87
0.69
0.87
0.91
0.69
0.82
0.87
BCPMF
0.95
0,87
0.69
0,87
0.90
0.69
0.81
0.87
BCPMF (Une 50%)
0.96
0.89
0.76
0.92
0.95
0.76
0.86
0.90
BCPMF (Une 30%)
0.97
0.91
0.87
0.98
0.99
0.87
0.91
0.93
Global Mean
0.19
0.20
0.48
0.66
0.20
0.21
0.21
Mean Of Means
0.65
0.48
0.20
0.49
0.66
0.20
0.21
0.22
PMF
0.73
0.55
0.19
0.44
0.60
0.19
0.29
0.40
BCPMF
0.75
0.57
0.21
0.47
0.62
0.21
0.36
0.46
RCPME (Unee 50%)
0.75
0.61
0.40
0.64
0.75
0.40
0.50
0.58
BCPMF (Une 30%)
0.82
0.71
0.60
0.78
0.86
0.60
0.67
0.73
24

---

Can We Predict Performance of Large Models across Vision-Languag, Tasks?
Humanities and Social Sciences (CMMMU)
Pharmacy (MMMU)
Action Prediction (SEED 2)
Biology (MMMU)
Memo Comprehension
(SFED
Celebrity Recognition (MMBench CN)
Visual Referring Expression (SEED
Object Localization (MMBench CN)
Image Creation (SEED
Identity Reasoning (MMBench CN
Landmark Kecognition (SEED
Image Style (MMBench CN)
Arts and Design (CMMMU)
Procedure Understanding (SEED 2
Astronomy (
Accounting (MMMU
Celebrity Recognition (SEED
Image Quality (MMBench EN)
Science Knowledge (SEED
Image Quality (MMBench CN)
Impact (A)
Impact (B)
(MME)
Count (MME)
Genes Traits (ScienceQA))
Weather and Climate (ScienceQA)
Public Health (MMMU)
Distance (CVBenchh)
In-Context Captioning (SEED 2
Interleaved Image- Text Analysis (SEE)
Text- Text-to-Imape lo--e Generation (SEEE 2)
Physical Relation (MMBench EN]
Chemistry (MMMU)
Chart Understanding (SEED
Architecture and Engineering (MMMU)
Difference Spetting (SEED
Mechanical
Engineering (MMMU)
Particle Motion Energy IScienceOA
Electronics (MMMU
Visual Mathematics (SEED
Finance (MMM)
Procedure Understanding (SEED
Marketing (MMMU)
Impact
Impact
(C)
(D)
Image Topic (MMBench CN)
Designing Experiments (SejenceQA)
2D Count (CVBcach)
PathVQ&
Color (MMF)
Random (POPE)
Spatial Relation (SEED
Posters (MME)
Instance Location (SEED
VQA
Instance Interaction (SEED
VisualGenome
Instances Counting (SEED
VQARAD
Instance Att Attributes (SEED
In-Context Captioning (SEED 2
Instance Identity (SEED
Existence (MME)
Visual Reasoning (SEED
Scene (MME)
Impact
Impact
(E)
(F)
Figure 11. Effect Analysis of Vision Encoders on Downstream Tasks. We evaluate the impact of each vision encoder on downstream tasks by calculating the dot product between the feature vector of the vision encoder and the feature vector of the dataset.
Global Mean
PMF
0.30
CPMF
CPMF (Oracle)
0.25
Overall RMSE
0.20
0.15
0.10
0.05
0.00
New Model
New Dataset
Both New
Figure 12. Results Purely New Models and Datasets.
25

---

Can We Predict Performance of Large Models across Vision-Languag, Tasks?
Table 14. Combining Coresct Methods with Our Approach with Different Sampling Ratios.
Overall
Ace
BART
Method
RMSEL
MAEL
RMSE
MAE
RMSE
MAE
Select 5% Samples
Ours
0.193
0.090
0.074
0.052
0,459
0.299
Random Selection
0.345
0.224
0.250
0.175
0.652
0.494
+ Ours (Avg)
0.199 (40.146)
0.126
0.131
0.093
0.404
0.306
Ours (Unee
0.157 (-0. 188)
0.083
0.070
0.050
0.365
0.261
Herding
0.326
0,220
0.252
0.177
0,582
0.458
(Avg)
(-0.134)
0.124
Ours (Une)
0.155 (-0.171)
0.081
0.070
0.050
0.358
0,252
K-Center Greedy
0.353
0,231
0.262
0.182
0,656
0.498
Ours (Avg)
0,200 (-0.153)
0.128
0.137
0.096
0.394
0.302
+ Ours (Uncc)
D.0.4
0.082
0.070
0.050
0.356
0.258
Select Samples
Ours
0.193
0,090
0.074
0.052
0.459
0.299
Random Selection
0.224
0,141
0.152
0.107
0,444
0.326
Ours (Avg)
0.149 (-0.055)
0,088
0.085
0.061
0.322
0.237
+ Ours (Unee)
0.139 (-0.085)
0.076
0.069
0.049
0.313
0.224
Herding
0.216
0.140
0.155
0.112
0,410
0.297
+ -uurs (Avg)
0.144 (-0.072)
0,088
0.087
0.064
0,305
0.220
(Une)
(-0.076)
0.049
K-Center Greedy
0.223
0,142
0.154
0.109
0.437
0.322
+ Ours (Avg)
(-0.079)
0.088
0.086
0.063
0,306
0,226
Ours (Unce
0.138 (40.855)
0.077
0.070
0.049
0,313
0.224
Select 15% Samples
0.193
Random Selection
0.100
0.1
0.125
0.087
0.352
0.261
+ Ours (Avg)
0.133 (-0.047)
0.078
0.073
0.053
0,291
0.212
Ours (Unc)
0.132 (-0.048)
0.074
0.068
0.049
0.295
0.210
Herding
0.177
0.117
0.130
0.093
0.332
0.245
(Avg)
0.131 (-0.046)
+ Ours (Une)
0.135 (-0.042)
0,074
0.069
0.049
0,302
0,210
Center Greedy
0.087
+ Ours (Avg)
0.129 (-0.043)
0.076
0.073
0.053
0.281
0,203
+ Ours (Unc)
0.131 (-0.042)
0.074
0.069
0.049
0.291
0.208
26


Reviews:
[{"review_id": "ftfsP95zDv", "reviewer": "Reviewer_Yhor", "summary": "The paper proposes a new framework for predicting the performance of large vision-language models across various tasks using probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC). The framework formulates performance prediction as a matrix completion task, constructs a sparse performance matrix, and predicts unknown scores based on observed ones. The authors introduce enhancements to PMF to handle sparse data, including Bayesian PMF and incorporating model and dataset profiles. Experiments demonstrate the accuracy of PMF in predicting unknown scores and the effectiveness of active evaluation based on uncertainty.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Explanation to our method**\n\n*Q: Why to formulate it as a matrix prediction problem? What is the underlying principle or intuition?*\n\nWe are inspired by recommender systems. Imagine we are recommending movies to users: there are many users and many movies, but each user only rates a few movies. We can build a matrix where each row represents a user, each column is a movie, and each element is a rating from a user to a movie. We observe some elements in the matrix and want to estimate the unknown ones for making recommendation. This suits a matrix completion task and PMF is an effective way to solve that. In our case, we can see LVLMs as “users” while benchmarks as “movies”. We observe some performance scores of LVLMs on some benchmarks, and want to predict the unobserved ones. Thus, we formulate our problem similarly and solve it with PMF.\n\n*Q: Is it specifically applicable to LVLM?*\n\nWhile our paper focuses on LVLM evaluation, the methods are general and can be applied to other models and tasks as well.\n\n*Q: Do LVLM models and benchmarks satisfy the Gaussian distribution condition?*\n\nThe Gaussian distribution assumption primarily serves as quadratic regularization terms in the objective function [1], which can alleviate the overfitting problem. It does not require that the latent distributions must be Gaussian. To verify this, we apply Kolmogorov-Smirnov test to check if the learned latents follow a Gaussian distribution. The p-values are all below 0.05, indicating they are not actually Gaussian.\n\n> **How can the framework be extended to handle different evaluation settings**\n\nThank you for your question! We show some preliminary experiments in the supplementary material (Section B.5.). Specifically, we explore two ways to extend our methods.\n\n**Additional profile.** The other method is to encode evaluation settings as extra profile into PMF.\n\n**Additional models.** A straightforward way is to treat a model under different evaluation settings as different models, such as \"LLaVA (Chain-of-Thought)\" and \"LLaVA (Beam Search)\".\n\nWe evaluate LLaVA-v1.5-7B on the 27 tasks in SEED-2, with the following various evaluation settings.\n\n**Image input.** (1) Default: use clean images, or (2) add Gaussian noise into the images.\n\n**Prompt.** (1) Default: prompt the model to choose option directly, (2) provide no hint, or (3) use the Chain-of-Thought (CoT) prompt.\n\n**Model decoding.** (1) Default: greedy decoding, (2) sampling with temperature = 0.2, (3) sampling with temperature = 0.5, or (4) beam search with temperature = 0.2 and the number of beams = 10.\n\nWe add the results under different evaluation settings into our framework and apply PMF. The estimation accuracy is shown as follows, indicating that our framework can predict model performance under different evaluation settings.\n\n*Test Ratio: 20%* (More results are in Section B.5.)\n|**Method**|**Overall**|**Default**|**Gaussian Noise**|**No Hint**|**CoT**|**Sampling (t=0.2)**|**Sampling (t=0.5)**|**Beam Search** |\n|-|-|-|-|-|-|-|-|-|\n|Global Mean|0.119|0.112|0.105|0.090|0.117|0.127|0.109|0.111|\n|Mean of Means|0.103|0.090|0.088|0.090|0.102|0.105|0.092|0.088|\n|Ours (Profiles)|0.062|0.041|0.055|0.075|0.064|0.045|0.055|0.052|\n|Ours (Models)|0.053|0.043|0.045|0.073|0.050|0.040|0.046|0.041|\n\n> **The generalization capabilities of the framework**\n\nThank you! The question is also raised by Reviewers dJrE and mCcw. To validate the generalization ability of the framework, we add new models and new datasets into our pool.\n\n**New models.** Qwen2-VL-Instruct (2B, 7B), Qwen2.5-VL-Instruct (3B, 7B, 32B), DeepSeek-VL (tiny, small)\n\n**New datasets.** MathVision, EMMA, Video-MME, LongVideoBench\n\n**It took several days to conduct experiments, because we need to evaluate old models on new datasets and new models on old datasets. At the end, 11 of 108 old models are not tested on the new datasets, and those results will not be included in the following test.*\n\nThe averaged RMSEs are shown as follows.\n\n*If we only know 20% performance of new models and new datasets*\n|**Method**|**New Model**|**New Dataset**|**Both New**|\n|-|-|-|-|\n|Global Mean|0.390|0.043|0.106|\n|Mean of Means|0.303|0.037|0.084|\n|PMF|0.326|0.032|0.047|\n|BCPMF|0.297|0.033|0.039|\n\n*If we know 50% performance of new models and new datasets*\n\n|**Method**|**New Model**|**New Dataset**|**Both New**|\n|-|-|-|-|\n|Global Mean|0.389|0.045|0.090|\n|Mean of Means|0.311|0.039|0.073|\n|PMF|0.265|0.031|0.034|\n|BCPMF|0.228|0.030|0.034|\n\nOur method shows better generalization compared to the baselines. We observe that generalizing to new datasets is relatively easier than to new models. This is probably because new datasets are often very challenging, leading to generally lower model performance and smaller RMSEs. In contrast, new models often have novel designs and remarkable improvements. This makes generalization to unseen models more difficult and needs further exploration.\n\n---\n\n[1] Probabilistic matrix factorization. NeurIPS 2007.", "claims_and_evidence": "Most claims made in the paper are generally supported by clear and convincing evidence. However, why can the performance prediction of LVLM be formulated as a matrix prediction problem? What is the underlying principle or intuition behind this approach? Is it specifically applicable to LVLM, or can it be applied to other models such as LLMs? Additionally, PMF requires that the two low-dimensional latent distributions obtained from the factorization follow a Gaussian distribution. In this context, we are dealing with different LVLM models and various benchmarks. Do these two aspects satisfy the Gaussian distribution condition?", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem at hand.", "theoretical_claims": "No formal theoretical proofs are provided.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper provides a practical solution to the expensive evaluation process.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The active evaluation strategy based on uncertainty is innovative and practical, potentially saving substantial computational resources.\n- The comprehensive experiments and detailed analysis provide strong empirical support for the proposed methods.\n\nWeaknesses:\n- The paper does not explore the impact of different evaluation settings (e.g., varying prompts or decoding strategies) in depth.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- How can the framework be extended to handle different evaluation settings, such as varying prompts or decoding strategies? Would incorporating these settings as additional profiles improve prediction accuracy?\n- Can the authors provide more insights into the generalization capabilities of the framework for new models and datasets? Are there any preliminary results or ideas on how to address this limitation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iyl1EZO4tf", "reviewer": "Reviewer_dJrE", "summary": "This paper formulates the problem of predicting Large Vision-Language Model (LVLM) performance on unseen benchmarks as a sparse matrix completion task. The authors propose using Probabilistic Matrix Factorization (PMF) to predict model performance across datasets that haven't been evaluated yet. The paper introduces three key contributions: (1) a PMF-based approach to predict LVLM performance, (2) an active evaluation strategy to efficiently select which model-dataset combinations to evaluate first, and (3) an extension of PMF to Probabilistic Tensor Factorization (PTF) to handle multiple evaluation metrics simultaneously.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Metric Validity**\n\nThank you for your suggestion! We include the following ranking-based metrics.\n\n**Spearman’s rank correlation.** \n\n**Kendall rank correlation.**\n\n**Precision@K.** The proportion of the predicted top 1 model that fall within the top K positions of the ground-truth ranking. For example, our method predicts LLaVA-1.5 is the best on the task and it will be correct if LLaVA-1.5 is within top 5 of the ground-truth ranking for Precision@5.\n\n**Recall@K.** The proportion of the ground-truth best ones that is correctly retrieved within our top K predicted results. For instance, if LLaVA-1.5 is the best model, it must be within the top 3 predictions of our method for Recall@3.\n\n*If we know 80% performance data for training*\n\n|Method|Spearman|Kendall|P@1|P@3|P@5|R@1|R@3|R@5|\n|-|-|-|-|-|-|-|-|-|\n|Global Mean|0.86|0.76|0.75|0.96|1.00|0.75|0.79|0.80|\n|Mean Of Means|0.90|0.80|0.75|0.97|1.00|0.75|0.79|0.80|\n|PMF|0.95|0.87|0.69|0.87|0.91|0.69|0.82|0.87|\n|BCPMF|0.95|0.87|0.69|0.87|0.90|0.69|0.81|0.87|\n|BCPMF (Unc 50%)|0.96|0.89|0.76|0.92|0.95|0.76|0.86|0.90|\n|BCPMF (Unc 30%)|0.97|0.91|0.87|0.98|0.99|0.87|0.91|0.93|\n\nBaseline methods typically achieve very high precision but low recall, while our methods provide more balanced precision and recall, as well as improved rank correlation.\n\n\"Unc 50%\" and \"Unc 30%\" mean keeping the 50% or 30% most confident predictions based on our estimated uncertainty, which further improve the estimation accuracy. But note that this may lead to fewer predicted results.\n\n*If we know 20% performance data for training*\n\n| Method|Spearman|Kendall|P@1|P@3|P@5|R@1|R@3|R@5 |\n| --|--|--|--|--|--|--|--|-- |\n| Global Mean|0.27|0.19|0.20|0.48|0.66|0.20|0.21|0.21 |\n| Mean Of Means|0.65|0.48|0.20|0.49|0.66|0.20|0.21|0.22 |\n| PMF|0.73|0.55|0.19|0.44|0.60|0.19|0.29|0.40 |\n| BCPMF|0.75|0.57|0.21|0.47|0.62|0.21|0.36|0.46 |\n| BCPMF (Unc 50%)|0.75|0.61|0.40|0.64|0.75|0.40|0.50|0.58 |\n| BCPMF (Unc 30%)|0.82|0.71|0.60|0.78|0.86|0.60|0.67|0.73 |\n\nOur method still outperforms baselines.\n\nDue to the space constraints, we omit additional results. We will update our paper accordingly.\n\n> **Generalizability to New Benchmarks**\n\nThank you! The question is also raised by Reviewers Yhor and mCcw. \n\nWe add new models and new datasets.\n\n**New models.** Qwen2-VL-Instruct (2B, 7B), Qwen2.5-VL-Instruct (3B, 7B, 32B), DeepSeek-VL (tiny, small)\n\n**New datasets.** MathVision, EMMA, Video-MME, LongVideoBench\n\nThe averaged RMSEs are shown as follows.\n\n*If we only know 20% performance of new models and new datasets*\n|**Method**|**New Model**|**New Dataset**|**Both New**|\n|-|-|-|-|\n|Global Mean|0.390|0.043|0.106|\n|Mean of Means|0.303|0.037|0.084|\n|PMF|0.326|0.032|0.047|\n|BCPMF|0.297|0.033|0.039|\n\n*If we know 50% performance of new models and new datasets*\n\n|**Method**|**New Model**|**New Dataset**|**Both New**|\n|-|-|-|-|\n|Global Mean|0.389|0.045|0.090|\n|Mean of Means|0.311|0.039|0.073|\n|PMF|0.265|0.031|0.034|\n|BCPMF|0.228|0.030|0.034|\n\nOur method shows better generalization compared to the baselines. We find that generalizing to new datasets is easier than to new models. This is probably because new datasets are often very challenging, leading to generally low model performance and small RMSEs. In contrast, new models often have novel designs and remarkable improvements. This makes generalization to unseen models more difficult and needs further exploration.\n\n> **How to interpret our results?**\n\nThere are two possible perspectives.\n\n**Benchmarks.** There is redundency in existing benchmarks, as also reported by [1]. Multiple benchmarks may test similar skills like math. Besides, there might be some correlation across different tasks. For instance, [2] reports that an LVLM with a stronger LLM can achieve consistently better performance on different benchmarks.\n\n**Models.** The similarity in model architectures and training may also contribute to the correlation between performance. For example, if two models use the same vision encoder, e.g., CLIP, they may have similar failure cases [3]. Besides, as commented by Reviewer mCcw, many models use training data generated from GPT-4V, possibly resulting in similar strengths and weaknesses.\n\nWe will update our paper to include the discussion.\n\n> **How do users rely on our predictions?**\n\nThank you! As shown above, the model rankings from our methods have high correlation to the ground truth. Even with very sparse training data, our methods remain more reliable than guessing based on average scores, a strategy that mimicking what a human might do. Moreover, our estimated uncertainty can further support users to make decisions. Users can focus on predictions with higher uncertainty, conduct additional evaluations, and make better model selections.\n\n---\n\n[1] Redundancy Principles for MLLMs Benchmarks. ArXiv.\n\n[2] LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data?. Blog.\n\n[3] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. CVPR2024.", "claims_and_evidence": "The paper's main claims about predicting LVLM performance using matrix factorization techniques are generally supported by empirical evidence. However, the reported MAE values (exceeding 5% in many cases) raise questions about the practical utility of these predictions for real-world evaluation scenarios. While the authors demonstrate that their approach outperforms baselines, the absolute performance may not be sufficient for confident decision-making in model selection or evaluation planning.", "methods_and_evaluation_criteria": "The paper primarily uses RMSE/MAE as evaluation metrics, but it may be more insightful to analyze whether the predicted model ranking is consistent with actual rankings. \nSimilarly, we are uncertain about how low the RMSE/MAE metrics need to be in order to provide sufficient guidance value for the model evaluation process.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- The experimental setup correctly partitions observed and unobserved model-dataset pairs to simulate sparse evaluations.\n- The effectiveness of active evaluation is demonstrated, but further clarification is required on the robustness of the method across diverse benchmark sets.\n- One potential concern is the generalizability to newer datasets. The majority of benchmarks studied are relatively early releases in 2024 (CMMMU in January, CVBench in June), while more recent ones are not tested.", "relation_to_broader_scientific_literature": "The paper builds upon Probabilistic Matrix Factorization, which is a well-established technique in collaborative filtering. \nIt also connects with Bayesian approaches commonly used in uncertainty estimation and low-data learning scenarios. \nThe work is related to efficient language model benchmarking studies such as tinyBenchmarks and Lifelong Benchmarks, \nwhich emphasize cost-effective model evaluations.", "essential_references_not_discussed": "It is recommended to include more latest LVLM benchmarks in the study, to demonstrate the prediction capability of the proposed approach.", "other_strengths_and_weaknesses": "Strengths: \n- The paper proposes an efficient strategy for reducing evaluation costs, which is particularly useful given the rapid development of LVLMs. The use of active evaluation is a practical addition that aligns with real-world model selection needs.\n\nWeaknesses:\n- The generalizability of predictions across benchmarks is not fully addressed—if accurate predictions indicate redundancy, does this mean newer benchmarks aren't offering meaningful novel insights? Additionally, the high MAE values suggest that the approach might not yet be sufficiently reliable for direct deployment in performance prediction tasks.", "other_comments_or_suggestions": "- More empirical justification for metric selection (why RMSE/MAE instead of ranking-based metrics?).\n- Evaluating the method on newer benchmarks (MathVision, EMMA) would provide stronger validation.\n- Discussion on the implications of predictive benchmark redundancy would strengthen the paper’s impact.", "questions_for_authors": "1. **Metric Validity**: Have you considered ranking-based evaluation metrics instead of RMSE/MAE? Would ranking consistency be more informative for practitioners?\n2. **Generalizability to New Benchmarks**: How well does your approach work on latest benchmarks, such as MathVision and EMMA?\n3. **Benchmark Redundancy**: If a benchmark’s performance can be accurately predicted from previous evaluations, does this signal that the benchmark itself is redundant? How should LVLM developers interpret such cases?\n4. **Prediction Accuracy**: Given the relatively high MAE values (often exceeding 5%), how confident should users be in relying on these predictions for model selection?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7cR4Nqfsof", "reviewer": "Reviewer_mCcw", "summary": "This paper introduces a novel framework to predict unknown vlm  benchmark scores based on partial observation, from other LVLMs or tasks.\n\nThe problem is formulated as a matrix completion task, and the author proposes to apply probabilistic matrix factorization (PMF) with MCMC for this.\n\nThe key challenge of the vanilla baseline is the requirement of sufficient observed data, The authors augment PMF via:\n\n- add more scores than a single accuracy (e.g., BERT score) -> PTF\n- Model and dataset profiles are complied as extra information to the model.\n\nFor evaluation, the authors collect a large score matrix (108 LVLMs x 36 benchmarks), mask P% of the matrix and use the rest part to predict the masked scores. Experiment results show tthat he proposal is effective, achieving lower prediction error regarding the RMSE scores.\n\n\nFurther results suggest that the proposal could combine with coreset evaluation to reduce the computational costs while improving the accuracy.\n\n\n## After rebuttal\n\nThe detailed response during rebuttal regarding the utility of the proposed method is somewhat convincing, and the OOD results are interesting.  Therefore, I increased my score from 2 -> 3 .", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **The computational cost**\n\n*Q: Evaluation cost with the newest LMMs-Eval implementation.*\n\nThank you for your suggestion! We would like to clarify vllm was integrated into LMMs-Eval after our submission. We did not intend to exaggerate anything.\n\nIn the rebuttal, we do not have enough time to rerun all our evaluations. Instead, we evaluate a representative model, Qwen2.5-VL-Instruct based on LMMs-Eval with 1 A100 GPU.\n\n|Setting|Basic|vllm|LMMs-Lite|Larger model|Video benchmarks|Test-time scaling|\n|-|-|-|-|-|-|-|\n|Use vllm|N|Y|Y|N (no memory) |Y|Y|\n|Model Size|7B|7B|7B|32B|7B|7B|\n|Dataset|MMBench En|MMBench En|MMBench En Lite|MMBench En|VideoMME|MathVision|\n|Num of Samples|6.72K|6.72K|500|6.72K|2.7K|5.14K|\n|Real Performance?|Y|Possibly lower|Estimated|Y|Y|Y|\n|Time|24min|13min|1min|45min|>2h*|1h|\n|Time per sample|0.21s|0.11s|0.11s|0.40s|1.3s|0.70s|\n\n**Video loading and preprocessing are the botteneck.*\n\nAs shown in the first columns, although vllm significantly accelerate the evaluation, the computational cost is still high. Assuming that we are comparing 10 7B models on 10 similar-scale benchmarks, the evaluation will take around 21.7 hours. Besides, in our experiments, vllm usually leads to slight performance decrease.\n\nLMMs-Lite (a coreset method) significantly reduces the evaluation cost, indicating reducing evaluation cost is still valuable in practice. As shown in Section 5.1, coreset methods may get inaccurate results, while our method improves them.\n\n*Q. The cost will be less significant in the future*\n\nWe agree that, with the development of software and hardware, evaluating old models on the old benchmarks is becoming cheaper. However, evaluation remains costly for the two main reasons. \n\nFirst, the growing number of models and benchmarks significantly increases the overall evaluation burden. After our submission, just from January to March, there are already several new models (e.g., the DeepSeek series, Qwen2.5 variants, MM-Eureka) and benchmarks (e.g., MathGlance, Mobile-MMLU, MathFlow, MMDT, MapBench).\n\nSecond, the trend toward larger models, video LLMs, and test-time scaling techniques further raise evaluation costs, as illustrated in our table.\n\n> **We need real performance numbers to write a paper/report instead of an estimated number.**\n\nThank you! We totally agree. Not only our paper, previous studies on predicting model performances, e.g., TinyBenchmarks and model auto-evaluation, also suffer from this problem. \n\nHowever, our method is still valuable during model development. For instance, when exploring the optimal video LLM design [1], the researchers evaluate their models on a reduced set of benchmarks to guide design decisions. Only at the final stage, they evaluate the model on the full benchmarks to report real performance numbers. In this case, the reseachers could use our method to reduce the evaluation cost in model development.\n\n> **OOD generalization of the method**\n\nThank you! The question is also raised by Reviewers Yhor and dJrE. We add new models and new datasets into our pool.\n\n**New models.** Qwen2-VL-Instruct (2B, 7B), Qwen2.5-VL-Instruct (3B, 7B, 32B), DeepSeek-VL (tiny, small)\n\n**New datasets.** MathVision, EMMA, Video-MME, LongVideoBench\n\nThe averaged RMSEs are shown as follows.\n\n*If we only know 20% performance of new models and new datasets*\n|Method|New Model|New Dataset|Both New|\n|-|-|-|-|\n|Global Mean|0.390|0.043|0.106|\n|Mean of Means|0.303|0.037|0.084|\n|PMF|0.326|0.032|0.047|\n|BCPMF|0.297|0.033|0.039|\n\n*If we know 50% performance of new models and new datasets*\n\n|Method|New Model|New Dataset|Both New|\n|-|-|-|-|\n|Global Mean|0.389|0.045|0.090|\n|Mean of Means|0.311|0.039|0.073|\n|PMF|0.265|0.031|0.034|\n|BCPMF|0.228|0.030|0.034|\n\nOur method shows better generalization compared to the baselines.\n\n> **Different models may require different optimal prompts**\n\nIn the supplementary material (Section B.5.), we apply different evaluation settings to LLaVA and explore two ways to extend our methods. The results show that our framework can predict model performance under different evaluation settings.\n\n> **The clear gain of the GPT-4 series**\n\nThank you! We separate models to two groups: (1) Train w/ GPT, such as LLaVA and Cambrain. (2) Train w/o GPT, such as BLIP-2 and InstructBLIP. We run 10 experiments to get the averaged results. For \"Train w GPT\", adding evaluation results of GPT-4o decreases 0.029 (2.48%) RMSE of PMF, while for \"Train w/o GPT\", it decreases 0.022 (2.04%) RMSE.  Thus, distilling knowledge from GPT may lead to similar strengths and weaknesses, which needs further exploration.\n\n> **Typos**\n\nThank you!\n\n> **Explanation to Figure (6)B**\n\nIn Fig. 6(B), the tasks in tail do not have the same RMSE improvements, but their values are relatively close. We observe that, these tasks show smaller performance gaps across models, so adding them brings limited RMSE improvement.\n\n---\n\n[1] Apollo: An Exploration of Video Understanding in Large Multimodal Models. ArXiv.", "claims_and_evidence": "The claim that PMF is effective is well-supported by the comprehensive results such as Figure 3 and Table 1.\n\nHowever, I feel the computational cost of the evaluation is somewhat exaggerated in the first paragraph, since\n- We do not need to evaluate on all 50 tasks of LMMs-eval since many of them are redundant and highly correlated (as TinyBenchmark did);\n- the original implementation of LMMs-eval is not very efficient (e.g., querying one sample per GPU)\n\nI would recommend that the author report a real number with the newest LMMs-eval (with vLLM support)  on TinyBenchmark to justify the significance of this computational cost.", "methods_and_evaluation_criteria": "The paper evaluates a large score matrix collected by the authors, covering diverse tasks and models. \n\nHowever, as different models may require different prompts to elicit their performance, using standards templates in LMMs-eval may lead to inaccurate scores.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments are generally solid.", "relation_to_broader_scientific_literature": "This paper proposes a new problem aiming to predict the performance scores using partially observed data points, which is novel.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Pros: \n- Overall, the paper is well-written and easy to follow.\n\nCons:\n\n1. The actual computational cost may not be that high as stated, given the evolving software and hardware stack, rendering the problem less significant in the future;\n2. From a practical perspective, we need real performance numbers to write a paper/report instead of an estimated number. \n3. Most of the current benchmarks are sourced from similar original image datasets (such as COCO), and I am curious about the OOD generalization. For example, could the scores on image benchmarks using COCO images generalize to video benchmarks such as Video-MME (youtube videos)? \n\nPoints 1 and 2 are bigger ones, I am happy to discuss them with the authors.", "other_comments_or_suggestions": "Sec 5.3 is interesting. Regarding the clear gain of the GPT-4 series, I am curious whether this is because many models are trained using distilled datasets from GPT-4V, such as ShareGPT-4V. The authors could separate models (trained w/ and w/o ShareGPT4V ) if see if similar conclusions could be drawn.\n\nTypos: \nLine 418: We -> we", "questions_for_authors": "In Figure (6)B,  why do many tasks lead to the same RMSE improvements?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d4XOrFvaPm", "reviewer": "Reviewer_6d7s", "summary": "The paper discusses a very interesting problem, i.e., predicting the performance of MLLMs. This task is very practical because evaluating LLMs is expensive. Specifically, the authors focuses on the problem that, given a performance matrix with missing value, whether we can fill these missing ones.\n\n## Update after rebuttal\n\nDear authors, I would like to express my honor to have an opportunity to review your paper and apologize for forgetting to file some areas, such as “Claims and Evidence”. Here, as the update after the rebuttal, I want to summarize my opinion and give the final recommendations.\n\nOverall, \n(1) the paper makes claims such as we can predict the performance of MLLM and these claims are well-supported by the experiments in the paper and rebuttal; \n(2) after the rebuttal, I agree with that there is novelty in the proposed method, and the evaluation criterias are reasonable; \n(3) I do not find any theory contributions of this paper although ICML is a conference that prefers the ML theory; but it is OK for no theoretical claims from my opinion; \n(4) the experiments designs and analyses are comprehensive and detailed; \n(5) the supplementary material is not required to be read by the reviewers and therefore I do not check it; and \n(6) the paper mentions a strong relationshio to current literatures and all the essential references are discuessed.\n\nIn the future, I would like to recommend some extensions: (1) Exploring **why** there is a relationship between different tasks, could there be a theoratical explanations? (2) Will the proposed method can be generalizable to new released models, for example, the latest NVIDA Nemotron? (3) The degree of **needed** benchmarks, i.e., with your framework, how many benchmarks are needed in the future? If the number is limited, many new benchmarks will not be useful. (4) Including some failure cases is recommended and exploring why such cases fail. In this case, we will get clearer relationships between different MLLMs.\n\nFinally, congratulations for such interesting work! Well done!", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Novelty of the paper**\n\nThank you for your question! We would like to highlight three main contributions that reflect the novelty of our work.\n\nFirst, we propose and formulate the problem of LVLM performance prediction based on known performance scores. Previous works on efficient evaluation need to select a coreset within each dataset or use an unlabeled test set to estimate model performance. Instead, we explore a new direction and we show that our approach is complementary to coreset-based methods in Section 5.1. \n\nSecond, we introduce model profiles and dataset profiles in Section 3.5 to enhance the prediction accuracy. For model profiles, we include features such as the number of parameters in the LLM backbone, the vision encoder type, and the LVLM family. For dataset profiles, we cluster datasets based on latent representations obtained from various models and get one-hot encoded dataset profiles. We validate three different approaches to generate these latent representations: \n\nThird, we propose an uncertainty-aware active evaluation pipeline for efficient model evaluation. As shown in Fig. 4, our uncertainty-aware evaluation consistently outperforms the random baseline under a fixed evaluation budget, especially when the budget is limited. Besides, the estimated uncertainties are correlated with the actual absolute errors, indicating the reliability of our confidence estimates.\n\nWe hope that our paper can inspire future works, reduce redundancy in evaluation of large models, and contribute to more efficient development of LLMs and LVLMs.\n\n> **Relationship between the proposed method with the evaluation of LLMs. Why do you focus on the evaluation of LLMs?**\n\nThank you for your question! We focus on LLMs and LVLMs for two main reasons. First, the evaluation of these large models is very expensive, and the number of models and datasets are growing rapidly. Second, LLMs can handle a wide variety of tasks within a single model, making it very interesting to explore the correlations of their ablities across different tasks.\n\nTo reduce the evaluation costs, our paper explores a question: Can we predict unknown model performance scores based on known ones? \n\nTo answer this, we draw insipration from recommender systems and adpot PMF to our question. PMF is a well-established algorithm for matrix completion. In our case, we construct a sparse performance matrix R, where each entry represents the performance score of a model on a dataset. By applying PMF, we aim to complete the performance matrix, that is, predict unknown scores. This makes PMF a natural choice for our problem.\n\nWhile our paper focuses on LVLMs, the proposed idea and methods are general and can be applied to other types of models and tasks as well.\n\nAs an extension, we reference a related work [1], which ranks samples for efficient evaluation in image classification models. They introduce the Lifelong-ImageNet benchmark, which contains 1.98M test samples and results from 167 models. \n\nTo extend our method, we validate PMF with a Sigmoid output layer to predict sample-level model accuracy (0 or 1) on Lifelong-ImageNet. Specifically, we use 10% of the performance data for training and 90% for testing. Due to the large scale of this dataset, MCMC becomes pretty slow. Thus, we use the L-BFGS-B optimization algorithm to get a maximum a posteriori estimate. The results are shown as follows.\n\n| **Method** | **Global Mean** | **Mean of Means** | **[1]*** | **PMF** | \n| -- | -- | -- | -- | -- |\n| MAE | 0.500 | 0.399 | 0.128 | 0.205 |\n\n*[1] explores a different setting to ours, so the numbers here are only for reference.\n\nMore meticulous design may futher improve the performance of PMF. However, such improvements are beyond the current scope of our paper and are left for future work.\n\n\n> **Typos and mistakes**\n\nThank you! We will carefully refine our paper and correct the typos.\n\n---\n\n[1] Efficient Lifelong Model Evaluation in an Era of Rapid Progress. NeurIPS 2024.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Pros:\n1. I am really excited about the proposed problem. If the solutions become mature, it will save a lot of money.\n2. The paper indeed tries to unveils a in-depth relationship betwwen the evaluation tasks and MLLMs.\n3. As shown in Fig. 3, the proposed method is effective. \n\n\nCons:\n1. The novelty. As far as my concern, the paper is lack of novelty. The paper is indeed a naive extension of Probabilistic Matrix Factorization and some related things. Please clarify the novelty in the rebuttal. I will increae the score if the novelty is clarified.\n2. I do not see a strong relationship between the proposed method with the target task. Why do you focus on the evaluation of LLMs? Is it just because it is hot? It seems the proposed method is generalizable but there is no experimental results on other tasks.\n3. Small mistakes: (1) the citation should be consistent: (Li & Lu, 2024). Zhang et al. (2024b) ; (2) “with limited compute” should be “with limited compution”", "other_comments_or_suggestions": "Please focus on the novelty and the relationship problem. I will raise the score to 4 if the novelty is clarified and to 5 if the relationship is justified.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper tackles the pressing challenge of efficiently evaluating the proliferation of LVLMs by formulating performance prediction as a matrix completion task using probabilistic matrix factorization. The introduction of model and dataset profiles, together with an uncertainty‑aware active evaluation pipeline, yields consistently strong predictive accuracy across 108 models and 36 benchmarks, and generalizes well to new models and datasets. The authors have thoroughly addressed reviewer concerns—validating statistical assumptions, incorporating ranking metrics, and providing detailed cost analyses. The method promises substantial computational savings while maintaining reliability, making it a valuable contribution to the field. I recommend acceptance.
Decision: Accept (poster)