Can We Predict Performance of Large Models across Vision-Language Tasks?

## Abstract

Abstract Evaluating large vision-language models (LVLMs) is very expensive, due to the high computational costs and the wide variety of tasks. The good news is that if we already have some observed performance scores, we may be able to infer unknown ones. In this study, we propose a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. We first formulate the performance prediction as a matrix completion task. Specifically, we construct a sparse performance matrix 𝑹 𝑹 \bm{R} bold_italic_R , where each entry R m ⁢ n subscript 𝑅 𝑚 𝑛 R_{mn} italic_R start_POSTSUBSCRIPT italic_m italic_n end_POSTSUBSCRIPT represents the performance score of the m 𝑚 m italic_m -th model on the n 𝑛 n italic_n -th dataset. By applying probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC), we can complete the performance matrix, that is, predict unknown scores. Additionally, we estimate the uncertainty of performance prediction based on MCMC. Practitioners can evaluate their models on untested tasks with higher uncertainty first, quickly reducing errors in performance prediction. We further introduce several improvements to enhance PMF for scenarios with sparse observed performance scores. In experiments, we systematically evaluate 108 LVLMs on 176 datasets from 36 benchmarks, constructing training and testing sets for validating our framework. Our experiments demonstrate the accuracy of PMF in predicting unknown scores, the reliability of uncertainty estimates in ordering evaluations, and the effectiveness of our enhancements for handling sparse data. Our code is available at https://github.com/Qinyu-Allen-Zhao/CrossPred-LVLM .

## 1 Introduction

It is expensive to evaluate large vision-language models (LVLMs). First, large-scale models result in significant computational or API calling costs and memory usage. Additionally, since a single LVLM can handle a wide range of tasks, comprehensively understanding model performance on different tasks becomes more challenging. As a result, hundreds of benchmarks have been proposed to assess the strengths and weaknesses of LVLMs (Li & Lu, ). Zhang et al. () report that it takes hundreds of hours to evaluate one model on around 50 tasks in LMMs-Eval, and evaluation even exceeds 1,400 hours on models of 100B parameters or more.

Fortunately, we have already observed performance scores from some of these models on some tasks, for instance, from the official reports of released models and datasets. For new models, scores can also be readily obtained with limited compute by running on a small number of tasks. If these observed scores can be used to predict unknown ones, we could avoid unnecessary evaluations and effectively reduce costs. Recent works (Polo et al., ; Zhang et al., ) require running the same model on the same task to predict model performance, and most of them ignore the potential of leveraging observed performance data from other models or tasks.

In this study, we propose a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. We first formulate this as a matrix completion problem. Specifically, we construct a sparse performance matrix ${\bm{R}}$ where each entry $R_{mn}$ represents the performance score of the $m$-th model on the $n$-th dataset. By applying probabilistic matrix factorization (PMF) with Markov chain Monte Carlo (MCMC), we can predict unknown performance scores based on observed entries in the matrix. A summary of the framework is shown in Fig. 1.

A bonus of our framework is active evaluation, which aims to select a subset of model-dataset pairs to evaluate in order to minimize prediction errors across the entire performance matrix. Given a PMF model on a very sparse performance matrix, we calculate prediction uncertainty from MCMC and prioritize evaluating model-dataset pairs with high uncertainty. Our experiments will confirm the effectiveness of this strategy for active evaluation.

A challenge is that PMF tends to predict the average score for models and datasets with very few observed scores, resulting in poor prediction results (Mnih & Salakhutdinov, ). To address this, we introduce several improvements to enhance PMF for scenarios with sparse observed data. First, we extend PMF to a simple tensor factorization approach, which can handle multiple performance metrics across different vision-language tasks. Second, we utilize Bayesian PMF (Salakhutdinov & Mnih, ) with an LKJ prior (Lewandowski et al., ) on the variance. Third, we also incorporate extra information as model and dataset profiles to improve performance prediction. For example, if we know a model uses CLIP as a vision encoder, the information may help predict the model’s performance, especially when we observe only a few performance scores of the model.

Figure: Figure 1: Framework. (A) Given a sparse matrix of performance scores of LVLMs on various tasks, the goal is to estimate the missing entries. (B) A normal way is to evaluate untested model-dataset pairs one-by-one. (C) TinyBenchmarks (Polo et al., ) runs models on smaller test sets and reproduce the original performance. (D) We use Probabilistic Matrix Factorization (PMF) to predict missing entries, reducing unnecessary evaluations, and rank new experiments based on uncertainty.
Refer to caption: x1.png

In experiments, we conduct a systematic evaluation of 108 LVLMs across 176 distinct datasets derived from 36 existing benchmarks, based on prior works (Duan et al., ; Zhang et al., ; Liang et al., ; Karamcheti et al., ; Schaeffer et al., ). We evaluate open-source models such as LLaVA-v1.5 (Liu et al., ), InstructBLIP (Dai et al., ), mPLUG-Owl (Ye et al., ), and MiniGPT-4 (Zhu et al., ), as well as closed-source models including GPT-4o, GPT-4 (Achiam et al., ), Gemini-1.5 (Reid et al., ). The benchmarks cover general VQA (Li et al., ), knowledge-dense VQA (Yue et al., ), hallucination (Li et al., ), medicine (He et al., ), emotion recognition (Goodfellow et al., ), and others. To reduce computational and API costs, we subsample some datasets, following Liang et al. ().

Using the results from 108 LVLMs across 176 datasets, we construct a $108\times 176$ performance matrix, with some entries masked for testing. We empirically demonstrate that PMF accurately predicts masked scores and consistently outperforms baselines as long as more than 10% entries in the performance matrix are observed. We also show that selecting high-uncertainty model-dataset pairs for evaluation significantly reduces prediction errors compared to random selection. Additionally, our improvements effectively alleviate the sparse data issue of PMF.

In summary, this paper covers three main points. First, we formulate a problem of predicting the unknown performance of LVLMs across tasks. Second, we apply the well-established PMF algorithm to this problem, show the application of active evaluation, and propose several strategies to mitigate the sparse data issue. Third, we conduct a comprehensive evaluation of 108 LVLMs across 176 datasets, constructing training and testing sets for further experiments.

## 2 Related Works

### 2.1 Recent LVLMs and Benchmarks

In recent years, there has been increasing growth in LVLMs, with many new models demonstrating impressive capabilities. Notable closed-source models include GPT-4 (Achiam et al., ) and Gemini (Team et al., ), while open-source models such as LLaVA (Liu et al., ; ), InstructBLIP (Dai et al., ), and InternVL (Chen et al., ; ) have also gained widespread attention. Karamcheti et al. () explore the design of LVLMs and have released a series of models (i.e., Prismatic VLMs) featuring different architectures and training strategies.

These LVLMs can handle a wide variety of tasks within a single model, but this versatility also requires more various benchmarks to fully understand their strengths and weaknesses. Some existing benchmarks can be repurposed for assessing these models, such as Flickr30k (Young et al., ), GQA (Hudson & Manning, ), and OKVQA (Marino et al., ). Recent works also propose new benchmarks to evaluate LVLMs in handling dense knowledge, complex reasoning, and decision-making tasks. Examples of novel benchmarks include SEED-Bench-2 (Li et al., ), MMMU (Yue et al., ), and MME (Fu et al., ). Additionally, as LVLMs become more integrated into everyday applications, benchmarks like POPE (Li et al., ) have been introduced to assess trustworthy issues like hallucination in these models. The variety of LVLMs and benchmarks leads to substantial computational
demands and memory usage.

### 2.2 Improve Evaluation Efficiency

Recent works introduce unified frameworks to assess models across multiple benchmarks using a single codebase, such as VLMEvalKit (Duan et al., ), LMMs-Eval (Zhang et al., ), and HEMM (Liang et al., ). Our study builds on these efforts by consolidating their frameworks and integrating models in Prismatic VLMs series (Karamcheti et al., ; Schaeffer et al., ).

Predicting unknown model performance can reduce the evaluation cost. Recent works select a coreset of samples from a large benchmark, for evaluating LLMs (Polo et al., ; Perlitz et al., ) and LVLMs (Zhang et al., ; Zhu et al., ). The performance of a specific model on the coreset is used to estimate its performance on the full benchmark. Besides, prior studies estimate model performance on an unlabeled test set based on distribution shift (Deng & Zheng, ), confidence scores (Guillory et al., ; Yang et al., ), or LLM feedback (Zheng et al., ). Instead of running models on a coreset or an unlabeled set, our framework predicts unknown performance by utilizing the correlation between model performances across benchmarks.

Another related direction is adaptive testing (Rodriguez et al., ; Prabhu et al., ). Given a new model, only a subset of samples is selected based on sample difficulty for evaluating the new model. While their work focuses on sample-level testing with a single metric, our approach operates at the dataset level, using six different metrics. Furthermore, instead of relying on statistically inferred sample difficulty, we propose a method to rank model-dataset pairs for evaluation based on uncertainty in performance prediction from MCMC.

### 2.3 Probabilistic Matrix Factorization

PMF (Mnih & Salakhutdinov, ) is a technique widely applied in recommender systems. Given part of the ratings that users provide for items, the goal is to model the observed ratings and predict the missing ones. PMF achieves this by decomposing the observed rating matrix into two lower-dimensional matrices, representing the latent features of users and items. A rating is modeled as a Gaussian distribution centered around the dot product of the user’s and item’s feature vectors.

One major challenge with PMF is that, if users rate very few items, their predicted ratings will be near the average for those items. Bayesian PMF (BPMF) (Salakhutdinov & Mnih, ) addresses this by placing distributions over the priors of the latent user and item features, making it more effective in handling sparse data. Additionally, Constrained PMF (Mnih & Salakhutdinov, ) introduces a latent similarity constraint matrix to further refine the user feature vectors.

## 3 Modelling LVLM Performance

In this section, we first describe the application of PMF to model the performance score matrix of LVLMs across datasets. Then, we discuss active evaluation for LVLMs. Last, three techniques are introduced to enhance PMF: supporting multiple metrics, incorporating Bayesian PMF, and integrating model and dataset profiles in modeling.

Figure: Figure 2: Graphical Models of PMF (A) and the enhanced model (B). (A) is adapted from the original paper (Mnih & Salakhutdinov, ). In (B), we set the mean to $\bm{0}$ and the covariance to the identity matrix, thus omitting most of the hyper-parameters for the random variable distributions.
Refer to caption: x2.png

### 3.1 Revisit Probabilistic Matrix Factorization

Let ${\bm{R}}$ be an $M\times N$ matrix representing model performance scores on datasets, where $M$ is the number of models and $N$ is the number of datasets. For simplicity, we initially assume a single performance metric, though in reality, benchmarks often employ multiple metrics. In such cases, ${\bm{R}}$ becomes an $M\times N\times S$ tensor, where $S$ represents the total number of metrics. We will address this more complex scenario in the following sections.

In practice, only a subset of the elements in ${\bm{R}}$ are observed, meaning we evaluate only a portion of the model-dataset pairs and aim to estimate the remaining performance scores. Specifically, we define a matrix ${\bm{O}}\in\{0,1\}^{M\times N}$, where $O_{mn}=1$ if $R_{mn}$ is observed, and 0 0 otherwise.

To model the observed matrix and estimate the unknown values, we employ PMF (Mnih & Salakhutdinov, ), as illustrated by the probabilistic graphical model in Fig. 2(A). PMF decomposes ${\bm{R}}$ into two low-dimensional matrices, ${\bm{U}}\in\mathbb{R}^{M\times D}$ and ${\bm{V}}\in\mathbb{R}^{N\times D}$, where $D$ is the latent dimension. Here, ${\bm{U}}_{m,:}$ and ${\bm{V}}_{n,:}$ are the latent feature vectors for the $m$-th model and the $n$-th dataset, respectively, and we refer to them as ${\bm{U}}_{m}$ and ${\bm{V}}_{n}$. These latent vectors are modeled as multivariate Gaussian distributions, and the observed ratings are assumed to follow a Gaussian distribution centered at the dot product of the latent feature vectors:

$$ $\displaystyle p({\bm{R}}\mid{\bm{U}},{\bm{V}},\sigma^{2})=\prod_{m=1}^{M}\prod _{n=1}^{N}\left[\mathcal{N}\left(R_{mn}\mid{\bm{U}}_{m}^{T}{\bm{V}}_{n},\sigma ^{2}\right)\right]^{O_{mn}},$ (1) $\displaystyle p({\bm{U}}\mid\sigma_{U}^{2})=\prod_{m=1}^{M}\mathcal{N}({\bm{U} }_{m}\mid\mathbf{0}_{D},\sigma_{U}^{2}{\bm{I}}_{D}),\quad p({\bm{V}}\mid\sigma _{V}^{2})=\prod_{n=1}^{N}\mathcal{N}({\bm{V}}_{n}\mid\mathbf{0}_{D},\sigma_{V} ^{2}{\bm{I}}_{D}),$ (2) $$

where ${\bm{I}}_{D}$ is a $D\times D$ identity matrix, and $\mathcal{N}(x\mid\mu,\sigma^{2})$ represents the probability density function of a Gaussian distribution with mean $\mu$ and variance $\sigma^{2}$. We simply set $\sigma_{U}=\sigma_{V}=1$.

Rather than using Maximum A Posteriori estimation to obtain point estimates of the unknown performance scores in ${\bm{R}}$, we apply MCMC to obtain distributions over the estimated scores and quantify the uncertainties in our predictions. Specifically, we use the No-U-Turn Sampler (NUTS) (Hoffman et al., ), an advanced Hamiltonian Monte Carlo method (Neal, ).

Our experiments show that standard PMF performs well with sufficient observed data. But its performance degrades significantly and is even worse than predicting the mean, when the observed data is very sparse (i.e., fewer than 10% model-dataset pairs are observed). To address this, we enhance our model with several techniques, with a new graphical model shown in Fig. 2(B).

### 3.2 Active Evaluation

MCMC allows us to estimate score distributions and readily obtain uncertainty estimates for each unknown score, enabling us to prioritize evaluation experiments. For example, if we are uncertain about GPT-4’s performance on a 3D understanding but confident about LLaVA’s performance on object recognition, we can prioritize evaluating GPT-4 on the 3D task when our resources are limited.

In our method, we begin by applying PMF to model a sparse performance matrix. Using MCMC, we get hundreds of estimations of each unknown score and calculate the standard deviation of estimations as a measure of uncertainty. The unobserved scores are ranked by their uncertainties. High-uncertainty scores are replaced with ground truth, simulating evaluation process in practice. We rerun PMF with updated observed data, calculate uncertainty, and determine the next set of evaluations. This process is repeated until our resource budget is exhausted or all scores are observed.

### 3.3 Multiple Metrics

Previously, we assumed that each dataset has only one scoring metric, but this is not the case in practice. For example, yes-or-no questions can be evaluated using accuracy, precision, recall, and F1 score, while open-ended questions may use metrics like BART score (Yuan et al., ) and BERT score (Zhang et al., ). Model performances are represented by a tensor ${\bm{\mathsfit{R}}}\in\mathbb{R}^{M\times N\times S}$, where $S$ is the total number of metrics. Empirically, we find that using PMF to model and predict each metric independently works well when sufficient data is available. However, when observed data is sparse, incorporating relationships between metrics will be helpful.

To address this, we extend our PMF model into a simple Probabilistic Tensor Factorization (PTF), where we decompose the 3D tensor ${\bm{\mathsfit{R}}}$ into the product of two low-rank matrices and a 1D vector. This can be interpreted as applying a linear transformation to the original PMF output, translating it into multiple metrics. Specifically, we define:

$$ $\displaystyle p({\bm{\mathsfit{R}}}\mid{\bm{U}},{\bm{V}},{\bm{w}},\bm{b}, \sigma^{2})=\prod_{m=1}^{M}\prod_{n=1}^{N}\prod_{s=1}^{S}\left[\mathcal{N} \left({\mathsfit{R}}_{mns}\mid({\bm{U}}_{m}^{T}{\bm{V}}_{n})w_{s}+b_{s},\sigma ^{2}\right)\right]^{O_{mns}},$ (3) $\displaystyle p({\bm{w}}\mid\sigma_{w}^{2})=\mathcal{N}({\bm{w}}\mid\mathbf{0} _{S},\sigma_{w}^{2}{\bm{I}}_{S}),\quad p(\bm{b}\mid\sigma_{b}^{2})=\mathcal{N} (\bm{b}\mid\mathbf{0}_{S},\sigma_{b}^{2}{\bm{I}}_{S}),$ (4) $$

where we set $\sigma_{w}=\sigma_{b}=1$ for simplicity.

This approach implicitly assumes a linear relationship between scoring metrics, which may not exactly hold in reality. However, we usually observe some linear correlation between the metrics on the same task. Moreover, more sophisticated techniques, such as advanced tensor factorization methods, modeling non-linear metric relationships with neural networks, or using manually defined transformation functions for specific metrics, can be explored to further improve the model.

Note that some metrics may be irrelevant for certain datasets, e.g., accuracy is not meaningful for long-answer questions. While our model can predict these scores, we discard the predicted results.

### 3.4 Bayesian PMF

Instead of using fixed priors for the feature vectors, we model the priors using probabilistic distributions, as proposed by Salakhutdinov & Mnih (). Unlike the original paper, which employs a Wishart distribution for the variance, we use the LKJ correlation prior (Lewandowski et al., ) and an Exponential prior to model the variance, as suggested by the PyMC documentation,

$$ $\bm{\Lambda}^{-1}_{U}=(\textbf{diag}\left(\bm{\sigma}_{L}\right){\bm{L}}_{U})( \textbf{diag}\left(\bm{\sigma}_{L}\right){\bm{L}}_{U})^{T},$ (5) $$

where $p({\bm{L}}_{U}\mid\eta_{U})=\text{LKJ}({\bm{L}}_{U}{\bm{L}}_{U}^{T}\mid\eta_{U})$ and $p(\bm{\sigma}_{L}\mid\lambda_{U})=\prod_{d=1}^{D}\text{Exp}(\sigma_{d}\mid
\lambda_{U})$.

Latent feature vectors are then modeled as:

$$ $\displaystyle p(\bm{\mu}_{U}\mid\bm{\Lambda}^{-1}_{U})=\mathcal{N}(\bm{\mu}_{U }\mid\mathbf{0}_{D},\bm{\Lambda}^{-1}_{U}),$ (6) $\displaystyle p({\bm{U}}\mid\bm{\mu}_{U},\bm{\Lambda}^{-1}_{U})=\prod_{m=1}^{M }\mathcal{N}({\bm{U}}_{m}\mid\bm{\mu}_{U},\bm{\Lambda}^{-1}_{U}).$ (7) $$

A similar formulation applies to ${\bm{V}}$, which we omit here for brevity.

### 3.5 Model and Dataset Profiles

The final enhancement to our framework is the incorporation of additional information about the models and datasets. For example, knowing that two LVLMs use CLIP as the vision encoder, or that LLaVA-v1.5 and LLaVA-NeXT are developed by the same team, suggests potential relationships in their performances. Inspired by Constrained PMF (Mnih & Salakhutdinov, ), we incorporate extra information as model and dataset profiles, to improve performance prediction.

Let ${\bm{H}}\in\mathbb{R}^{M\times K}$ and ${\bm{G}}\in\mathbb{R}^{N\times J}$ represent the model and dataset profiles, where ${\bm{H}}_{m,:}$ encodes $K$ properties of the $m$-th model (e.g., vision encoder type), and ${\bm{G}}_{n,:}$ encodes $J$ properties of the $n$-th dataset. We introduce Gaussian-distributed variables ${\bm{Y}}\in\mathbb{R}^{K\times D}$ and ${\bm{X}}\in\mathbb{R}^{J\times D}$ to learn the effects of these profiles. The latent feature vectors are now the sum of the original vectors and the profile features, following Constrained PMF (Mnih & Salakhutdinov, ).

$$ $\displaystyle p({\bm{Y}}\mid\sigma_{Y}^{2})=\prod_{k=1}^{K}\mathcal{N}({\bm{Y} }_{k}\mid\mathbf{0}_{D},\sigma_{Y}^{2}{\bm{I}}_{D}),\quad p({\bm{X}}\mid\sigma _{X}^{2})=\prod_{j=1}^{J}\mathcal{N}({\bm{X}}_{j}\mid\mathbf{0}_{D},\sigma_{X} ^{2}{\bm{I}}_{D}),$ (8) $\displaystyle{\bm{U}}^{\prime}={\bm{U}}+{\bm{H}}{\bm{Y}},\quad{\bm{V}}^{\prime }={\bm{V}}+{\bm{G}}{\bm{X}}.$ (9) $$

Oracle Profiles. To explore the upper bound of model and dataset similarities, we use the full ${\bm{R}}$ matrix to cluster models and datasets. For each model, we take ${\bm{R}}_{i,:}$ (its performance across all datasets) as a vector and apply the K-Means algorithm to cluster all models. We select the optimal number of clusters using the elbow method. Similarly, for each dataset, we cluster ${\bm{R}}_{:,j}$ in the same way. We convert the cluster assignments into one-hot vectors to serve as profiles.

Custom Profiles. Since oracle profiles rely on complete performance data, they are not practical for real-world use. To overcome this, we define custom profiles that can be applied in practice. For models, we include features such as the number of parameters in the LLM backbone, vision encoder type (one-hot), and the LVLM family (one-hot), illustrated in the supplementary material (Table 4). Additionally, we cluster datasets based on latent representations obtained from various models and get one-hot encoded dataset profiles. We explore three different approaches to generate these latent representations: D1. using MPNet (Song et al., ) to encode a short description of each dataset. D2. using CLIP to encode images and BGE-M3 to encode questions in a dataset (following Zhang et al. ()), then averaging the embeddings on the dataset; and D3. using LLaVA-7B to encode both images and text, then averaging the embeddings for the dataset.

## 4 Experiments

In this section, we construct a performance matrix and present key experiments for our framework.

Figure: Figure 3: Performance of PMF. (A-C) PMF consistently outperforms both baselines when the test ratio is below 90% for estimating all unobserved scores (A), accuracy scores (B), and BART scores (C), with particularly strong performance at lower test ratios. (D-F) The predicted scores exhibit correlations with the ground truth at test ratios of 20% (D), 60% (E), and 90% (F). Gray dashed lines represent perfect prediction i.e., $y=x$. We subsampled 200 scores in (D-F) for visualization.
Refer to caption: x3.png

### 4.1 Evaluating Models on Benchmarks

Prior works have developed general pipelines for evaluating LVLMs across a wide range of benchmarks (Duan et al., ; Zhang et al., ; Liang et al., ). Building on these code repositories, we evaluate 108 LVLMs on 36 benchmarks. The open-source models we cover include LLaVA-v1.5 (Liu et al., ), LLaVA-NeXT (Liu et al., ), InstructBLIP (Dai et al., ), mPLUG-Owl (Ye et al., ), and Prismatic VLMs (Karamcheti et al., ). We also evaluate closed-source models such as GPT-4 (Achiam et al., ) and Gemini-1.5 (Reid et al., ).

The benchmarks span a variety of domains, including general VQA (SEED-2), knowledge-dense VQA (MMMU), hallucination (POPE), medical question answering (PathVQA), and emotion recognition (FaceEmotion). Some large-scale benchmarks, such as SEED-2 (Li et al., ) and MMMU (Yue et al., ), cover multiple tasks. To conduct a fine-grain analysis, we split these benchmarks into task-specific datasets, resulting in 176 datasets in total. Following HEMM (Liang et al., ), we subsample some datasets to reduce computational and API calling costs of LVLMs. For each dataset, we calculate a main metric for PMF (either accuracy or BARTScore), and several other metrics, leading to a total of six metrics for PTF modeling. Full details of datasets and models are provided in the supplementary material (Section A).

### 4.2 Estimating Unknown Performances

We mask $P\%$ of the elements in the score matrix ${\bm{R}}$, use the observed portion to normalize ${\bm{R}}$, and train the PMF model using MCMC sampling. The model reconstructs the matrix $\hat{{\bm{R}}}$, and we evaluate the performance by comparing the estimated values with the ground truth for the masked elements. For MCMC, we employ the NUTS sampling method, tuning with 500 samples in the burn-in stage and drawing 100 samples. Empirical results show that 100 samples are sufficient for stable estimation. The reconstructed matrix $\hat{{\bm{R}}}$ is taken as the mean prediction from MCMC.

We use Root Mean Squared Error (RMSE) as the primary metric to evaluate PMF performance. Additional metrics such as Mean Absolute Error (MAE) and the coefficient of determination ($R^{2}$) are reported in the supplementary material (Section B).

We compare our method against two baselines: (1) Global Mean: predicting the global mean for unobserved scores; (2) Mean of Means: for each unobserved score, we average the mean performance of the model, the mean performance on the dataset, and the global mean.

Results. As shown in Fig. 3(A-C), PMF significantly outperforms the baselines when the test ratio is lower than 90%. This suggests that when only a portion of the scores is available, PMF can infer the unobserved scores with high accuracy. Additionally, as demonstrated in Fig. 3(D-F), the estimated scores strongly correlate with the actual scores.

However, as the amount of observed data decreases, PMF’s performance declines as can be expected. In extreme cases where the test ratio exceeds 90%, with limited information about model or dataset performance, PMF can perform worse than predicting the means. We will address this issue in the following sections with our proposed enhancement techniques.

Figure: Figure 4: Comparison of Active Evaluation Methods. Starting with 20% of the data observed, we progressively conduct additional LVLM evaluations using three different strategies. (A) RMSE and (B) MAE improvement demonstrate the advantage of our method compared to random evaluation. (C) Uncertainties from MCMC are correlated with the actual absolute errors.
Refer to caption: x4.png

### 4.3 Active Evaluation for LVLMs

**Table 1: Comparison of PMF and PTF. Superior results are highlighted. PMF (Sep) models each score separately, while PMF (OneMat) combines accuracy and BART scores into a single matrix, as each dataset contains either accuracy or BART scores. PTF is the enhanced model that supports multiple scoring metrics, which outperforms PMF at a high test ratio.**
| Method | Overall | Acc | Precision | Recall | F1 | BART | BERT |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RMSE$\downarrow$ | MAE$\downarrow$ | RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE |  |
| Test Ratio: 20% |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| PMF (Sep) | 0.175 | 0.086 | 0.073 | 0.051 | 0.135 | 0.086 | 0.166 | 0.115 | 0.134 | 0.087 | 0.463 | 0.318 | 0.068 | 0.031 |
| PMF (OneMat) | 0.193 | 0.090 | 0.074 | 0.052 | - | - | - | - | - | - | 0.461 | 0.303 | - | - |
| PTF | 0.205 | 0.096 | 0.078 | 0.055 | 0.129 | 0.085 | 0.176 | 0.126 | 0.108 | 0.070 | 0.563 | 0.378 | 0.077 | 0.039 |
| Test Ratio: 90% |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| PMF (Sep) | 0.327 | 0.177 | 0.159 | 0.118 | 0.238 | 0.174 | 0.262 | 0.197 | 0.227 | 0.167 | 0.864 | 0.628 | 0.096 | 0.047 |
| PMF (OneMat) | 0.317 | 0.174 | 0.156 | 0.115 | - | - | - | - | - | - | 0.723 | 0.504 | - | - |
| PTF | 0.290 | 0.158 | 0.159 | 0.118 | 0.186 | 0.129 | 0.230 | 0.167 | 0.180 | 0.124 | 0.754 | 0.529 | 0.094 | 0.045 |

We compare our uncertainty-based approach against two baselines: (1) Random selection of model-dataset pairs, and (2) an oracle approach that selects the pairs with the highest actual errors. In the experiment, we start by masking 80% performance data in the performance matrix. Then, we progressively conduct more LVLM evaluations using three different strategies, and calculate the improvement in performance prediction of PMF with the updated observed data. The experiment is repeated with 10 different random seeds, and we report the averaged improvement.

Results. As shown in Fig. 4, our uncertainty-aware method consistently outperforms the random baseline for a fixed budget of evaluations, especially when the amount of extra data is lower than 30%. However, there remains a gap between our method and the oracle approach.

Figure: Figure 5: Performance of Enhanced PTF. (A) BPTF shows minimal improvement over standard PTF when data is sufficient but proves particularly beneficial under sparse conditions. (B) Custom profiles improve performance when data is limited, though a gap remains compared to oracle profiles. (C) Ablation study on model and dataset profiles. “A $\mid$ B” represents using A for the model profile and B for the dataset profile. Custom model profiles lead to significant performance gains, while dataset profiles contribute only marginally. BPTF, Bayesian PTF; CPTF, Constrained PTF.
Refer to caption: x5.png

### 4.4 Enhancing PMF

We apply three enhancement techniques to our PMF model and evaluate their effectiveness across different test ratios. To minimize experimental variance, we perform each experiment 10 times with different random seeds and report the average performance at each test ratio.

Results. As seen in Table 1, the multi-score method PTF can get better performance when the matrix is very sparse. When there is enough data, separately modeling PMF with each score works very well and is comparable to PTF. For BART and BERT scores, PMF even outperforms PTF. This is likely because PTF assumes a linear relationship between scores. When this assumption does not hold, such as in the case of BART and BERT scores, it can negatively impact model performance. When the test ratio is high, PTF demonstrates better performance.

Fig. 5 illustrates the impact of the other two enhancement techniques. As shown, Bayesian PTF offers only negligible improvements over standard PTF when there is enough observed data, but it is particularly beneficial in sparse conditions. In Fig. 5(B), our custom profiles also show improvements when data is limited, though there remains a gap between our custom profiles and the oracle profiles. Additionally, Fig. 5(C) highlights that adding profiles not only enhances PTF’s overall performance but also reduces instability, as seen by smaller error bars. Model profiles show significant performance gains, whereas dataset profiles contribute only marginally. Better methods for encoding and utilizing dataset information need further exploration.

Figure: Figure 6: Low-Rank Property of the Score Matrix. (A) RMSE on the test set for PMF stabilizes when the latent dimension exceeds 15. (B) The top singular values of the performance matrix are significantly larger than the others. (C) t-SNE visualization of dataset clusters.
Refer to caption: x6.png

## 5 Discussion

### 5.1 Low-Rank Properties of the Performance Matrix

We investigate the impact of different latent dimensions in the PMF models and find that a relatively small latent dimension, around 10, is sufficient. As shown in Fig. 6, increasing the latent dimension reduces the RMSE on the training data to zero due to overfitting, but it does not lead to significant improvements in RMSE on the testing data. Additionally, when we extract the singular values of the score matrix, we observe that the top singular values are much larger than the rest, indicating that most of the information is captured by a few dimensions. This suggests a high degree of similarity in performance scores across benchmarks. A detailed correlation analysis of these performance scores is provided in the supplementary material (Section A).

### 5.2 What can we tell based on vision encoders?

The Constrained PMF model can capture the impact of model and dataset profiles. Here, we present a showcase analysis focusing on the vision encoder type from the model profiles. Specifically, we calculate the dot product between the feature vector of the vision encoder type, ${\bm{H}}_{m}$, and the feature vector of the dataset, ${\bm{V}}^{\prime}_{n}$. The calculation result measures the influence of a vision encoder on a task. As shown in Fig. 7, DINO shows improvements on a few datasets compared to CLIP, while FNet, SigLIP, and ViT are less effective in comparison.

Figure: Figure 7: Effect Analysis of Vision Encoders on Downstream Tasks. We evaluate the impact of each vision encoder on downstream tasks by calculating the dot product between the feature vector of the vision encoder and the feature vector of the dataset.
Refer to caption: x7.png

### 5.3 Which Models or Benchmarks are Most Informative?

We assess how representative a model is and how informative a benchmark is, by measuring the RMSE improvements of PMF when we add the full results of a model or dataset. The most informative models and tasks are shown in Fig. 8. As observed, strong models like GPT-4, Gemini, and InterLM are more representative than weaker models. This is likely because their performance tends to deviate from the average and, being more general, they reliably reflect the difficulty level of various datasets. Interestingly, the text-to-image generation task is particularly informative. In this task, models must select the correct generated image from four candidates, and we observe that strong models, such as GPT-4, perform significantly better than others. This performance gap leads to larger errors in PMF, so including this dataset can significantly improve the PMF model.

Figure: Figure 8: Which Models and Datasets Are Informative for Performance Estimation. Given a PMF model train on 20% data of the performance matrix, We measure the improvement in RMSE of PMF when adding the entire results of a model (A) or a dataset (B).
Refer to caption: x8.png

## 6 Conclusion and Future Work

In this study, we evaluate 108 models on 176 datasets across 36 benchmarks. Our framework estimates unknown LVLM performances across tasks using PMF, prioritizes evaluations based on uncertainty, and introduce some enhancements to address the sparse data issue. Our study could lead to significant savings in development time and computation costs. We highlight several limitations. First, recent advances show that in-context learning or generating multiple responses can improve LVLM performance on the same dataset. Modeling these different evaluation settings (e.g., 5-shot) could extend our framework. Second, some model-dataset pairs with high uncertainties might offer limited value for improving performance prediction on other datasets, so better heuristics for active evaluation could be developed. Third, our method cannot answer what new benchmarks are needed, which we believe is an interesting future direction.

## Acknowledgments

We would like to extend our deepest appreciation to Caixia Zhou, Weijian Deng, Xingjian Leng, Evan Markou, Yunzhong Hou, Yicong Hong, Taojun Lin, Jiahao Zhang, Sam Bahrami, Shu Zou, Zeyu Zhang, Yang Yang, Jiahao Ma, and all our other lab colleagues for their invaluable support throughout this project. Their collaborative efforts, insightful discussions, and constructive feedback have been crucial in shaping and improving our paper. This work was supported by an Australian Research Council (ARC) Linkage grant (project number LP210200931).

## Appendix A Comprehensive Evaluation of LVLMs

We provide a comprehensive overview of the datasets and LVLMs used in our study. Detailed dataset information can be found in Table 2 and 3, while the model profiles are presented in Tables 4 and 5.

A heatmap illustrating the model ranking across datasets is shown in Fig. 9. Additionally, the correlation analysis of performance scores is illustrated in Fig. 10 and 11. Notably, even within the same model family, such as the LLaVA series, the rankings between models do not exhibit a strong correlation. Datasets tend to have much more consistent ranking correlations, suggesting that models performing well on one dataset are likely to rank highly on others as well.

## Appendix B Further Experimental Results

We present detailed performance evaluations of PMF in Table 6 and PTF in Table 7. As shown, our methods consistently outperform the baselines. In scenarios where performance data is sparse, our enhancements significantly improves the prediction accuracy of PMF.

We also investigate the models’ ability to generalize to new models and datasets without any performance scores for training. As illustrated in Fig. 12, using model and dataset profiles provides slight improvement for new models or datasets. However, when both the model and dataset are entirely new, performance falls below the Global Mean baseline. But we argue that this situation is rare in practice. Some initial performance scores are usually available when a model or dataset is released, and the community usually reports more performance scores in subsequent works.

**Table 2: Dataset information. Our study utilizes 36 benchmarks. For larger benchmarks such as SEED-2, we divide them into sub-datasets based on task categories. To reduce computational costs, we subsample certain benchmarks. Download URLs for all benchmarks are provided.**
| Benchmark | No. of<br>Datasets | No. of Samples<br>for GPT and Gemini | No. of Samples<br>for Other Models | Download URL |
| --- | --- | --- | --- | --- |
| SEED 2 (Li et al., ) | 27 | 2606 | 24371 | https://huggingface.co/datasets/lmms-lab/SEED-Bench-2 |
| MME (Fu et al., ) | 14 | 1000 | 2374 | https://huggingface.co/datasets/lmms-lab/MME |
| MMBench CN (Liu et al., ) | 20 | 1994 | 4329 | https://huggingface.co/datasets/lmms-lab/MMBench |
| MMBench EN (Liu et al., ) | 20 | 1994 | 4329 | https://huggingface.co/datasets/lmms-lab/MMBench |
| MMMU (Yue et al., ) | 30 | 900 | 900 | https://huggingface.co/datasets/lmms-lab/MMMU |
| CMMMU (Zhang et al., ) | 6 | 573 | 900 | https://huggingface.co/datasets/lmms-lab/CMMMU |
| ScienceQA (Lu et al., ) | 25 | 1467 | 2017 | https://huggingface.co/datasets/lmms-lab/ScienceQA |
| CVBench (Tong et al., ) | 4 | 400 | 2638 | https://huggingface.co/datasets/nyu-visionx/CV-Bench |
| POPE (Li et al., ) | 3 | 900 | 900 | https://github.com/AoiDragon/POPE |
| DECIMER (Brinkhaus et al., ) | 1 | 100 | 100 | https://www.kaggle.com/datasets/juliajakubowska/decimer |
| Enrico (Leiva et al., ) | 1 | 100 | 100 | https://userinterfaces.aalto.fi/enrico/ |
| FaceEmotion (Goodfellow et al., ) | 1 | 100 | 100 | https://www.kaggle.com/datasets/msambare/fer2013 |
| Flickr30k (Young et al., ) | 1 | 100 | 100 | https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset |
| GQA (Hudson & Manning, ) | 1 | 100 | 100 | https://cs.stanford.edu/people/dorarad/gqa/download.html |
| HatefulMemes (Kiela et al., ) | 1 | 100 | 100 | https://www.kaggle.com/datasets/parthplc/facebook-hateful-meme-dataset |
| INAT (Van Horn et al., ) | 1 | 100 | 100 | https://ml-inat-competition-datasets.s3.amazonaws.com/2021/val.tar.gz |
| IRFL (Yosef et al., ) | 1 | 100 | 100 | https://huggingface.co/datasets/lampent/IRFL |
| MemeCaps (Hwang & Shwartz, ) | 1 | 100 | 100 | https://github.com/eujhwang/meme-cap/tree/main |
| Memotion (Sharma et al., ) | 1 | 100 | 100 | https://www.kaggle.com/datasets/williamscott701/memotion-dataset-7k |
| MMIMDB (Arevalo et al., ) | 1 | 100 | 100 | https://huggingface.co/datasets/akshayg08/mmimdb_test |
| NewYorkerCartoon (Hessel et al., ) | 1 | 100 | 100 | https://github.com/nextml/caption-contest-data |
| NLVR (Suhr et al., ) | 1 | 100 | 100 | https://github.com/lil-lab/nlvr.git |
| NLVR2 (Suhr et al., ) | 1 | 100 | 100 | https://github.com/lil-lab/nlvr.git |
| NoCaps (Agrawal et al., ) | 1 | 100 | 100 | https://huggingface.co/datasets/akshayg08/NocapsTest |
| OKVQA (Marino et al., ) | 1 | 100 | 100 | https://okvqa.allenai.org/download.html |
| OpenPath (Huang et al., ) | 1 | 100 | 100 | https://huggingface.co/datasets/akshayg08/OpenPath |
| PathVQA (He et al., ) | 1 | 100 | 100 | https://github.com/UCSD-AI4H/PathVQA |
| Resisc45 (Cheng et al., ) | 1 | 100 | 100 | https://www.kaggle.com/datasets/happyyang/nwpu-data-set |
| Screen2Words (Wang et al., ) | 1 | 100 | 100 | https://www.kaggle.com/datasets/onurgunes1993/rico-dataset |
| Slake (Liu et al., ) | 1 | 100 | 100 | https://huggingface.co/datasets/BoKelvin/SLAKE/ |
| UCMerced (Yang & Newsam, ) | 1 | 100 | 100 | https://www.kaggle.com/code/apollo2506/land-scene-classification |
| VCR (Zellers et al., ) | 1 | 100 | 100 | https://visualcommonsense.com/download/ |
| VisualGenome (Krishna et al., ) | 1 | 100 | 100 | https://homes.cs.washington.edu/ ranjay/visualgenome/ |
| VQA (Antol et al., ) | 1 | 100 | 100 | https://visualqa.org/vqa_v1_download.html |
| VQARAD (Lau et al., ) | 1 | 100 | 100 | https://huggingface.co/datasets/flaviagiammarino/vqa-rad |
| Winoground (Thrush et al., ) | 1 | 100 | 100 | https://huggingface.co/datasets/facebook/winoground |

**Table 3: Dataset Metrics. PMF models the main metric on the datasets, while PTF utilizes the main and other metrics (six in total) in modeling. BARTScore is proposed by Yuan et al. (), while BERTScore is introduced by Zhang et al. ().**
| Benchmark | Main Metric | Other Metrics |
| --- | --- | --- |
| SEED 2 (Li et al., ) | Accuracy | - |
| MME (Fu et al., ) | Accuracy | Precision, Recall, F1 |
| MMBench CN (Liu et al., ) | Accuracy | - |
| MMBench EN (Liu et al., ) | Accuracy | - |
| MMMU (Yue et al., ) | Accuracy | - |
| CMMMU (Zhang et al., ) | Accuracy | - |
| ScienceQA (Lu et al., ) | Accuracy | - |
| CVBench (Tong et al., ) | Accuracy | - |
| POPE (Li et al., ) | Accuracy | Precision, Recall, F1 |
| DECIMER (Brinkhaus et al., ) | BARTScore | BERTScore |
| Enrico (Leiva et al., ) | BARTScore | BERTScore |
| FaceEmotion (Goodfellow et al., ) | BARTScore | BERTScore |
| Flickr30k (Young et al., ) | BARTScore | BERTScore |
| GQA (Hudson & Manning, ) | BARTScore | BERTScore |
| HatefulMemes (Kiela et al., ) | BARTScore | BERTScore |
| INAT (Van Horn et al., ) | BARTScore | BERTScore |
| IRFL (Yosef et al., ) | BARTScore | BERTScore |
| MemeCaps (Hwang & Shwartz, ) | BARTScore | BERTScore |
| Memotion (Sharma et al., ) | BARTScore | BERTScore |
| MMIMDB (Arevalo et al., ) | BARTScore | BERTScore |
| NewYorkerCartoon (Hessel et al., ) | BARTScore | BERTScore |
| NLVR (Suhr et al., ) | BARTScore | BERTScore |
| NLVR2 (Suhr et al., ) | BARTScore | BERTScore |
| NoCaps (Agrawal et al., ) | BARTScore | BERTScore |
| OKVQA (Marino et al., ) | BARTScore | BERTScore |
| OpenPath (Huang et al., ) | BARTScore | BERTScore |
| PathVQA (He et al., ) | BARTScore | BERTScore |
| Resisc45 (Cheng et al., ) | BARTScore | BERTScore |
| Screen2Words (Wang et al., ) | BARTScore | BERTScore |
| Slake (Liu et al., ) | BARTScore | BERTScore |
| UCMerced (Yang & Newsam, ) | BARTScore | BERTScore |
| VCR (Zellers et al., ) | BARTScore | BERTScore |
| VisualGenome (Krishna et al., ) | BARTScore | BERTScore |
| VQA (Antol et al., ) | BARTScore | BERTScore |
| VQARAD (Lau et al., ) | BARTScore | BERTScore |
| Winoground (Thrush et al., ) | BARTScore | BERTScore |

**Table 4: Model Information. Our study evaluates 108 models. For each model, we report the number of parameters in the LLM backbone, the vision encoder, and the model family that we define.**
| Model | Checkpoint | No. Param. in LLM | Vision Encoder | Model Family |
| --- | --- | --- | --- | --- |
| BLIP2 | BLIP2-opt-2.7B | 2.7 | ViT | BLIP |
|  | BLIP2-flan-t5-xxl | 11 | ViT | BLIP |
|  | BLIP2-opt-6.7b-coco | 6.7 | ViT | BLIP |
|  | BLIP2-opt-6.7b | 6.7 | ViT | BLIP |
|  | BLIP2-flan-t5-xl | 3 | ViT | BLIP |
| InstructBLIP | InstructBLIP-Vicuna-7B | 7 | ViT | BLIP |
|  | InstructBLIP-Vicuna-13B | 13 | ViT | BLIP |
|  | InstructBLIP-flan-t5-xl | 3 | ViT | BLIP |
|  | InstructBLIP-flan-t5-xxl | 11 | ViT | BLIP |
| MiniGPT4 | MiniGPT4-LLaMA2-7B | 7 | ViT | MiniGPT4 |
|  | MiniGPT4-Vicuna0-7B | 7 | ViT | MiniGPT4 |
|  | MiniGPT4-Vicuna0-13B | 13 | ViT | MiniGPT4 |
| mPLUG-Owl | mPLUG-Owl2-LLaMA2-7B | 7 | ViT | MiniGPT4 |
|  | mPLUG-Owl2_1 | 7 | ViT | mPLUG-Owl |
| LLaVA | LLaVA-7B | 7 | CLIP | LLaVA |
|  | LLaVA-13B | 13 | CLIP | LLaVA |
|  | LLaVA-v1.6-Vicuna-7B | 7 | CLIP | LLaVA |
|  | LLaVA-v1.6-Vicuna-13B | 13 | CLIP | LLaVA |
|  | LLaVA-v1.6-Mistral-7B | 7 | CLIP | LLaVA |
|  | LLaVA-v1.6-34B | 34 | CLIP | LLaVA |
| Cambrian-1 | Cambrian-Phi3-3B | 3 | CLIP, SigLIP, ConvNeXt, DINOv2 | Cambrian |
|  | Cambrian-8B | 8 | CLIP, SigLIP, ConvNeXt, DINOv2 | Cambrian |
|  | Cambrian-13B | 13 | CLIP, SigLIP, ConvNeXt, DINOv2 | Cambrian |
|  | Cambrian-34B | 34 | CLIP, SigLIP, ConvNeXt, DINOv2 | Cambrian |
| Fuyu | Fuyu-8B | 8 | - | Fuyu |
| LLaMA_Adapter | LLaMA-Adapter-V2-BIAS-7B | 7 | CLIP | LLaMA-Adapter |
|  | LLaMA-Adapter-V2-LORA-BIAS-7B | 7 | CLIP | LLaMA-Adapter |
|  | LLaMA-Adapter-V2-LORA-BIAS-7B-v21 | 7 | CLIP | LLaMA-Adapter |
| OpenFlamingo | OpenFlamingo-3B-vitl-mpt1b | 1 | NFNet | OpenFlamingo |
|  | OpenFlamingo-3B-vitl-mpt1b-langinstruct | 1 | NFNet | OpenFlamingo |
|  | OpenFlamingo-4B-vitl-rpj3b | 3 | NFNet | OpenFlamingo |
|  | OpenFlamingo-4B-vitl-rpj3b-langinstruct | 3 | NFNet | OpenFlamingo |
|  | OpenFlamingo-9B-vitl-mpt7b | 7 | NFNet | OpenFlamingo |
| Qwen-VL | Qwen-VL-Chat | 7 | ViT | Qwen |
| InternLM_XComposer | InternLM-XComposer-7B | 7 | CLIP | InternLM |
|  | InternLM-XComposer-vl-7B | 7 | CLIP | InternLM |
|  | InternLM-XComposer2-7B | 7 | CLIP | InternLM |
|  | InternLM-XComposer2-vl-1_8b | 1.8 | CLIP | InternLM |
|  | InternLM-XComposer2-vl-7B | 7 | CLIP | InternLM |
| GPT4 | gpt-4o-2024-05-13 | Unknown | Unknown | GPT4 |
|  | gpt-4o-2024-08-06 | Unknown | Unknown | GPT4 |
|  | gpt-4o-mini-2024-07-18 | Unknown | Unknown | GPT4 |
|  | gpt-4-turbo-2024-04-09 | Unknown | Unknown | GPT4 |
| Gemini | gemini-1.5-pro | Unknown | Unknown | Gemini |
|  | gemini-1.5-flash | Unknown | Unknown | Gemini |

**Table 5: Model information. This is the continued table of Table 4**
| Model | Checkpoint | No. Param. in LLM | Vision Encoder | Model Family |
| --- | --- | --- | --- | --- |
| Prismatic | reproduction-llava-v15+7b | 7 | CLIP | prism |
|  | reproduction-llava-v15+13b | 13 | CLIP | prism |
|  | one-stage+7b | 7 | CLIP | prism |
|  | one-stage+13b | 13 | CLIP | prism |
|  | full-ft-multi-stage+7b | 7 | CLIP | prism |
|  | full-ft-one-stage+7b | 7 | CLIP | prism |
|  | in1k-224px+7b | 7 | ViT | prism |
|  | dinov2-224px+7b | 7 | DINOv2 | prism |
|  | clip-224px+7b | 7 | CLIP | prism |
|  | siglip-224px+7b | 7 | SigLIP | prism |
|  | clip-336px-resize-crop+7b | 7 | CLIP | prism |
|  | clip-336px-resize-naive+7b | 7 | CLIP | prism |
|  | siglip-384px-letterbox+7b | 7 | SigLIP | prism |
|  | siglip-384px-resize-crop+7b | 7 | SigLIP | prism |
|  | siglip-384px-resize-naive+7b | 7 | SigLIP | prism |
|  | dinoclip-336px-letterbox+7b | 7 | CLIP, DINOv2 | prism |
|  | dinoclip-336px-resize-naive+7b | 7 | CLIP, DINOv2 | prism |
|  | dinosiglip-384px-letterbox+7b | 7 | SigLIP, DINOv2 | prism |
|  | dinosiglip-384px-resize-naive+7b | 7 | SigLIP, DINOv2 | prism |
|  | llama2+7b | 7 | CLIP | prism |
|  | llama2+13b | 13 | CLIP | prism |
|  | vicuna-no-cotraining+7b | 7 | CLIP | prism |
|  | llama2-no-cotraining+7b | 7 | CLIP | prism |
|  | train-1.25-epochs+7b | 7 | CLIP | prism |
|  | train-1.5-epochs+7b | 7 | CLIP | prism |
|  | train-2-epochs+7b | 7 | CLIP | prism |
|  | train-3-epochs+7b | 7 | CLIP | prism |
|  | llava-lvis4v+7b | 7 | CLIP | prism |
|  | llava-lrv+7b | 7 | CLIP | prism |
|  | llava-lvis4v-lrv+7b | 7 | CLIP | prism |
|  | prism-clip-controlled+7b | 7 | CLIP | prism |
|  | prism-clip-controlled+13b | 13 | CLIP | prism |
|  | prism-clip+7b | 7 | CLIP | prism |
|  | prism-clip+13b | 13 | CLIP | prism |
|  | prism-siglip-controlled+7b | 7 | SigLIP | prism |
|  | prism-siglip-controlled+13b | 13 | SigLIP | prism |
|  | prism-siglip+7b | 7 | SigLIP | prism |
|  | prism-siglip+13b | 13 | SigLIP | prism |
|  | prism-dinosiglip-controlled+7b | 7 | SigLIP, DINOv2 | prism |
|  | prism-dinosiglip-controlled+13b | 13 | SigLIP, DINOv2 | prism |
|  | prism-dinosiglip+7b | 7 | SigLIP, DINOv2 | prism |
|  | prism-dinosiglip+13b | 13 | SigLIP, DINOv2 | prism |
|  | prism-dinosiglip-224px-controlled+7b | 7 | SigLIP, DINOv2 | prism |
|  | prism-dinosiglip-224px+7b | 7 | SigLIP, DINOv2 | prism |
|  | llama2-chat+13b | 13 | CLIP | prism |
|  | mistral-v0.1+7b | 7 | CLIP | prism |
|  | mistral-instruct-v0.1+7b | 7 | CLIP | prism |
|  | phi-2+3b | 3 | CLIP | prism |
|  | gemma-instruct+2b+clip | 2 | CLIP | prism |
|  | gemma-instruct+2b+siglip | 2 | SigLIP | prism |
|  | gemma-instruct+2b+dinosiglip | 2 | SigLIP, DINOv2 | prism |
|  | gemma-instruct+8b+clip | 8 | CLIP | prism |
|  | gemma-instruct+8b+siglip | 8 | SigLIP | prism |
|  | gemma-instruct+8b+dinosiglip | 8 | SigLIP, DINOv2 | prism |
|  | llama2-chat+7b+clip | 7 | CLIP | prism |
|  | llama2-chat+7b+siglip | 7 | SigLIP | prism |
|  | llama2-chat+7b+dinosiglip | 7 | SigLIP, DINOv2 | prism |
|  | llama3-instruct+8b+clip | 8 | CLIP | prism |
|  | llama3-instruct+8b+siglip | 8 | SigLIP | prism |
|  | llama3-instruct+8b+dinosiglip | 8 | SigLIP, DINOv2 | prism |
|  | mistral-instruct-v0.2+7b+clip | 7 | CLIP | prism |
|  | mistral-instruct-v0.2+7b+siglip | 7 | SigLIP | prism |
|  | mistral-instruct-v0.2+7b+dinosiglip | 7 | SigLIP, DINOv2 | prism |

Figure: Figure 9: Heatmap of Model Rankings on Each Dataset.
Refer to caption: x9.png

Figure: Figure 10: Heatmap of Correlation in Model Ranking.

Figure: Figure 11: Heatmap of Ranking Correlation on Datasets.

**Table 6: Detailed performance of PMF. Superior results are highlighted.**
| Method | Overall | Acc | BART |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RMSE$\downarrow$ | MAE$\downarrow$ | RMSE | MAE | $R^{2}\uparrow$ | RMSE | MAE | $R^{2}$ |  |
| Test Ratio: 20% |  |  |  |  |  |  |  |  |
| Global Mean | 0.367 | 0.220 | 0.190 | 0.148 | 0.475 | 0.823 | 0.611 | 0.549 |
| Mean Of Means | 0.319 | 0.186 | 0.161 | 0.125 | 0.622 | 0.719 | 0.525 | 0.656 |
| PMF | 0.192 | 0.090 | 0.073 | 0.051 | 0.922 | 0.458 | 0.303 | 0.860 |
| Test Ratio: 40% |  |  |  |  |  |  |  |  |
| Global Mean | 0.368 | 0.220 | 0.190 | 0.149 | 0.477 | 0.829 | 0.614 | 0.551 |
| Mean Of Means | 0.320 | 0.186 | 0.161 | 0.125 | 0.623 | 0.725 | 0.527 | 0.657 |
| PMF | 0.199 | 0.095 | 0.078 | 0.056 | 0.911 | 0.474 | 0.314 | 0.853 |
| Test Ratio: 60% |  |  |  |  |  |  |  |  |
| Global Mean | 0.370 | 0.220 | 0.190 | 0.149 | 0.474 | 0.831 | 0.613 | 0.551 |
| Mean Of Means | 0.322 | 0.188 | 0.162 | 0.126 | 0.618 | 0.729 | 0.529 | 0.654 |
| PMF | 0.220 | 0.106 | 0.089 | 0.063 | 0.886 | 0.521 | 0.348 | 0.823 |
| Test Ratio: 80% |  |  |  |  |  |  |  |  |
| Global Mean | 0.373 | 0.221 | 0.193 | 0.150 | 0.462 | 0.837 | 0.612 | 0.546 |
| Mean Of Means | 0.329 | 0.191 | 0.166 | 0.128 | 0.601 | 0.742 | 0.533 | 0.643 |
| PMF | 0.258 | 0.131 | 0.114 | 0.081 | 0.812 | 0.600 | 0.407 | 0.766 |
| Test Ratio: 90% |  |  |  |  |  |  |  |  |
| Global Mean | 0.381 | 0.226 | 0.198 | 0.153 | 0.430 | 0.852 | 0.630 | 0.529 |
| Mean Of Means | 0.339 | 0.197 | 0.174 | 0.133 | 0.564 | 0.765 | 0.555 | 0.621 |
| PMF | 0.313 | 0.172 | 0.153 | 0.113 | 0.660 | 0.714 | 0.502 | 0.669 |
| Test Ratio: 95% |  |  |  |  |  |  |  |  |
| Global Mean | 0.458 | 0.278 | 0.227 | 0.182 | 0.254 | 1.041 | 0.807 | 0.297 |
| Mean Of Means | 0.385 | 0.230 | 0.191 | 0.150 | 0.474 | 0.875 | 0.672 | 0.504 |
| PMF | 0.462 | 0.276 | 0.228 | 0.180 | 0.248 | 1.052 | 0.805 | 0.282 |

**Table 7: Detailed performance of PTF. Superior results are highlighted.**
| Method | Overall | Acc | Precision | Recall | F1 | BART | BERT |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RMSE$\downarrow$ | MAE$\downarrow$ | RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE |  |
| Test Ratio: 20% |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Global Mean | 0.320 | 0.190 | 0.190 | 0.149 | 0.223 | 0.167 | 0.245 | 0.186 | 0.205 | 0.156 | 0.812 | 0.603 | 0.088 | 0.044 |
| Mean Of Means | 0.260 | 0.150 | 0.149 | 0.115 | 0.181 | 0.131 | 0.205 | 0.152 | 0.169 | 0.123 | 0.664 | 0.482 | 0.074 | 0.036 |
| PMF | 0.206 | 0.096 | 0.081 | 0.057 | 0.130 | 0.086 | 0.175 | 0.126 | 0.108 | 0.070 | 0.563 | 0.378 | 0.077 | 0.039 |
| CPTF | 0.208 | 0.099 | 0.087 | 0.060 | 0.134 | 0.089 | 0.173 | 0.123 | 0.107 | 0.070 | 0.564 | 0.379 | 0.076 | 0.039 |
| BPTF | 0.202 | 0.095 | 0.079 | 0.056 | 0.129 | 0.084 | 0.177 | 0.127 | 0.109 | 0.070 | 0.553 | 0.372 | 0.077 | 0.039 |
| BCPTF | 0.207 | 0.096 | 0.079 | 0.056 | 0.129 | 0.085 | 0.178 | 0.127 | 0.113 | 0.072 | 0.568 | 0.378 | 0.076 | 0.039 |
| Test Ratio: 40% |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Global Mean | 0.323 | 0.192 | 0.190 | 0.149 | 0.223 | 0.167 | 0.247 | 0.189 | 0.213 | 0.160 | 0.818 | 0.609 | 0.091 | 0.045 |
| Mean Of Means | 0.262 | 0.151 | 0.150 | 0.116 | 0.182 | 0.132 | 0.209 | 0.156 | 0.174 | 0.126 | 0.667 | 0.486 | 0.078 | 0.038 |
| PMF | 0.210 | 0.100 | 0.083 | 0.059 | 0.132 | 0.087 | 0.181 | 0.130 | 0.112 | 0.073 | 0.572 | 0.388 | 0.081 | 0.040 |
| CPTF | 0.209 | 0.101 | 0.087 | 0.062 | 0.134 | 0.089 | 0.180 | 0.128 | 0.113 | 0.074 | 0.566 | 0.385 | 0.081 | 0.040 |
| BPTF | 0.206 | 0.100 | 0.084 | 0.060 | 0.132 | 0.086 | 0.185 | 0.133 | 0.117 | 0.077 | 0.558 | 0.379 | 0.082 | 0.041 |
| BCPTF | 0.209 | 0.100 | 0.082 | 0.059 | 0.131 | 0.086 | 0.184 | 0.131 | 0.117 | 0.076 | 0.568 | 0.384 | 0.081 | 0.040 |
| Test Ratio: 60% |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Global Mean | 0.325 | 0.192 | 0.191 | 0.149 | 0.227 | 0.170 | 0.248 | 0.189 | 0.214 | 0.160 | 0.825 | 0.611 | 0.092 | 0.045 |
| Mean Of Means | 0.265 | 0.153 | 0.151 | 0.116 | 0.186 | 0.135 | 0.212 | 0.157 | 0.178 | 0.128 | 0.676 | 0.490 | 0.080 | 0.038 |
| PMF | 0.218 | 0.107 | 0.093 | 0.067 | 0.136 | 0.090 | 0.188 | 0.134 | 0.123 | 0.081 | 0.588 | 0.400 | 0.084 | 0.041 |
| CPTF | 0.219 | 0.109 | 0.098 | 0.070 | 0.141 | 0.094 | 0.187 | 0.133 | 0.125 | 0.082 | 0.588 | 0.398 | 0.083 | 0.040 |
| BPTF | 0.217 | 0.108 | 0.096 | 0.068 | 0.138 | 0.092 | 0.194 | 0.139 | 0.130 | 0.087 | 0.582 | 0.397 | 0.085 | 0.042 |
| BCPTF | 0.216 | 0.105 | 0.089 | 0.064 | 0.135 | 0.090 | 0.191 | 0.136 | 0.127 | 0.083 | 0.584 | 0.394 | 0.083 | 0.041 |
| Test Ratio: 80% |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Global Mean | 0.330 | 0.194 | 0.193 | 0.150 | 0.230 | 0.171 | 0.253 | 0.191 | 0.217 | 0.162 | 0.839 | 0.619 | 0.092 | 0.046 |
| Mean Of Means | 0.277 | 0.158 | 0.155 | 0.119 | 0.198 | 0.141 | 0.225 | 0.165 | 0.186 | 0.134 | 0.709 | 0.510 | 0.084 | 0.041 |
| PMF | 0.249 | 0.128 | 0.120 | 0.087 | 0.151 | 0.103 | 0.207 | 0.148 | 0.145 | 0.098 | 0.661 | 0.457 | 0.091 | 0.044 |
| CPTF | 0.240 | 0.123 | 0.115 | 0.083 | 0.151 | 0.104 | 0.208 | 0.148 | 0.147 | 0.099 | 0.637 | 0.437 | 0.088 | 0.043 |
| BPTF | 0.239 | 0.123 | 0.116 | 0.083 | 0.152 | 0.103 | 0.212 | 0.151 | 0.151 | 0.102 | 0.630 | 0.433 | 0.090 | 0.044 |
| BCPTF | 0.236 | 0.119 | 0.108 | 0.077 | 0.147 | 0.099 | 0.208 | 0.149 | 0.147 | 0.099 | 0.627 | 0.427 | 0.089 | 0.043 |
| Test Ratio: 90% |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Global Mean | 0.338 | 0.198 | 0.199 | 0.154 | 0.237 | 0.174 | 0.258 | 0.195 | 0.224 | 0.166 | 0.858 | 0.629 | 0.095 | 0.047 |
| Mean Of Means | 0.298 | 0.168 | 0.166 | 0.125 | 0.216 | 0.153 | 0.239 | 0.176 | 0.204 | 0.147 | 0.764 | 0.547 | 0.090 | 0.043 |
| PMF | 0.294 | 0.161 | 0.161 | 0.119 | 0.194 | 0.135 | 0.235 | 0.171 | 0.187 | 0.131 | 0.761 | 0.535 | 0.094 | 0.045 |
| CPTF | 0.274 | 0.147 | 0.145 | 0.105 | 0.190 | 0.133 | 0.233 | 0.168 | 0.184 | 0.128 | 0.710 | 0.492 | 0.092 | 0.043 |
| BPTF | 0.267 | 0.143 | 0.142 | 0.103 | 0.179 | 0.123 | 0.232 | 0.167 | 0.178 | 0.122 | 0.690 | 0.480 | 0.093 | 0.044 |
| BCPTF | 0.268 | 0.141 | 0.138 | 0.099 | 0.179 | 0.124 | 0.228 | 0.164 | 0.176 | 0.120 | 0.698 | 0.481 | 0.093 | 0.045 |
| Test Ratio: 95% |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Global Mean | 0.404 | 0.238 | 0.228 | 0.182 | 0.251 | 0.194 | 0.270 | 0.209 | 0.240 | 0.183 | 1.058 | 0.805 | 0.101 | 0.057 |
| Mean Of Means | 0.387 | 0.217 | 0.202 | 0.158 | 0.241 | 0.182 | 0.261 | 0.198 | 0.230 | 0.172 | 1.027 | 0.772 | 0.097 | 0.056 |
| PMF | 0.403 | 0.233 | 0.223 | 0.177 | 0.244 | 0.185 | 0.269 | 0.204 | 0.236 | 0.175 | 1.059 | 0.801 | 0.101 | 0.057 |
| CPTF | 0.364 | 0.199 | 0.188 | 0.142 | 0.216 | 0.161 | 0.252 | 0.188 | 0.216 | 0.157 | 0.970 | 0.712 | 0.097 | 0.054 |
| BPTF | 0.355 | 0.196 | 0.189 | 0.144 | 0.208 | 0.153 | 0.251 | 0.188 | 0.206 | 0.148 | 0.940 | 0.687 | 0.098 | 0.055 |
| BCPTF | 0.360 | 0.196 | 0.186 | 0.141 | 0.211 | 0.156 | 0.251 | 0.186 | 0.205 | 0.148 | 0.959 | 0.702 | 0.100 | 0.056 |

Figure: Figure 12: Results on Purely New Models and Datasets.
Refer to caption: x12.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: * My main concern is the limited set of baseline matrix completion methods (mentioned above).
* Evaluation of active evaluation: could you consider a more canonical active learning evaluation setup? i.e., randomly partition elements of the matrix into an initial training set, an "unlabeled pool set" (in the active learning nomenclature), and a test set, and report active learning-style curves: for each acquisition method (oracle, random, uncertainty), plot RMSE on the test set versus the number of acquisition steps, as you acquire evals in the pool set? e.g., what is done in [3].
* Comment for possible future work: because the indices of the unobserved (model, dataset) elements are known a priori (and you also have access to side information such as which image encoder was used, etc.), this setting seems to fit naturally with some transductive active learning methods, such as [4].

[3] Gal et al., 2017. Deep Bayesian Active Learning with Image Data. https://arxiv.org/abs/1703.02910

[4] Bickford-Smith et al., 2023. Prediction-Oriented Bayesian Active Learning. In AISTATS. https://proceedings.mlr.press/v206/bickfordsmith23a/bickfordsmith23a.pdf
soundness: 3
strengths: - Strong motivation: VLM evaluation is very expensive, so being able to accurately predict downstream evaluation performance from a limited set of evaluations is very valuable.
- The method is elegant and appears to work well, e.g., the correlation plots in Figure 3 look clean. It is also surprisingly effective in active evaluation, which is a very practical and exciting direction for this line of work.
- The paper is exceptionally well-written and clear.
- The evaluation uses a large set of (model, dataset) evaluations on a variety of open- and closed-source models.
summary: This paper provides a framework for predicting the performance of large vision-language models on held-out downstream tasks using a small set of observed task performances, i.e., evaluations for a small set of (model, dataset) tuples. They formulate this as a matrix completion problem and demonstrate that probabilistic matrix factorization (PMF) with MCMC is surprisingly effective, using a large set of 108 VLM evaluations on 176 datasets. Further, the authors demonstrate that the uncertainty estimates of PMF can be used in active evaluation to prioritize which evaluation to conduct next, outperforming random selection. Lastly, the work explores extensions of the naive Bayesian PMF model: tensor factorization to handle multiple metrics, and incorporating side information for better performance under extreme sparsity.
weaknesses: The authors only consider a limited set of naive baselines for the main experiments in Figure 3. Could the authors benchmark other more sophisticated (neural) matrix completion methods, such as deep matrix factorization [1] or Graph Convolutional Matrix Completion [2]?

[1] Arora et al., 2019. Implicit Regularization in Deep Matrix Factorization. In NeurIPS. https://arxiv.org/abs/1905.13655

[2] van den Berg et al., 2018. Graph Convolutional Matrix Completion. In KDD. https://www.kdd.org/kdd2018/files/deep-learning-day/DLDay18_paper_32.pdf

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Could you clarify how Bayesian PMF differs from standard PMF in practical terms? Specifically, how does incorporating an LKJ prior (Lewandowski et al., 2009) impact the predictions in practice?

2. Including an ablation study to better quantify the contribution of each component—such as tensor factorization, Bayesian PMF, and the use of profiles—would help clarify their respective impacts on performance.

3. There's a noticeable gap between the uncertainty-based active evaluation and the oracle method. Have you considered alternative heuristics for prioritizing evaluations that might close this gap?
soundness: 3
strengths: 1. The proposed method is grounded in well-established techniques of matrix factorization and probabilistic modeling. The mathematical foundation of PMF is solid, and using MCMC for uncertainty estimation is a sensible approach to prioritize evaluations. The paper demonstrates that the method can effectively predict unknown performance scores, especially when more than 10% of the data is available. 

2. Evaluating 108 LVLMs across 176 datasets demonstrates the practicality and scalability of the proposed method across a wide range of tasks.
summary: This paper introduces a framework for predicting the performance of large vision-language models (LVLMs) across multiple tasks. The main idea is to employ probabilistic matrix factorization (PMF) to estimate unknown performance scores based on a sparse set of observed scores. By formulating performance prediction as a matrix completion problem and leveraging MCMC methods to estimate prediction uncertainty, the authors aim to reduce the computational cost of evaluating large models across diverse tasks. In addition, the authors propose several enhancements to handle data sparsity, including tensor factorization for multiple performance metrics and Bayesian PMF.
weaknesses: 1. The paper tackles an important problem: efficiently evaluating large-scale models as they grow in size and complexity. The idea of using matrix completion and active evaluation is interesting and, if successful, could lead to significant computational savings. However, the novelty is somewhat limited since the approach mainly builds on existing techniques like PMF, Bayesian modeling, and MCMC.

2.  Several parts of the paper lack clear explanations. For example, the differences between PMF, PTF, and Bayesian PMF are densely presented, and their respective impacts on performance are not sufficiently disentangled in the experiments. An explicit ablation study would help understand each enhancement's individual contributions. 

3. While using uncertainty to prioritize evaluations is compelling, the results show a gap between the uncertainty-based approach and the oracle method. The paper could explore why this gap exists and whether alternative heuristics could narrow it.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weakness section
soundness: 3
strengths: - An interesting application of existing statistical method for the problem of estimating performance on benchmarks. 
- The work has potential for impact and being useful for the community, especially developers. 
- Easy to implement, nice and thorough empirical analysis with sufficient ablations and insights/discussions.
summary: Evaluating VLMs across various number of tasks is costly (as the number of benchmarks can be huge) and the model sizes can be very large as well. The paper tries to propose an approach to estimate the performance on some datasets, by converting the problem to that of sparse matrix factorization, a well studied statistical approach for matrix completion. They assume a M x N matrix, where M is the different models and N is the various tasks. Given some entries of this matrix, one can estimate the rest using matrix factorization. The paper proposes some trivial modifications to the standard PMF to fit this specific use case. While the proposed work is an application of existing techniques to this problem, it is unique and has not been done previously in this setting. The empirical results are great, and the proposed idea can be useful to the community as such, especially while practitioners are developing models and need to frequently evaluate a lot of checkpoints/variations/finetuned versions of VLMs.
weaknesses: - In the active evaluation, the authors order the priority of the task for evaluation based on its estimation uncertainty/deviation. But this doesn’t factor in the cost of evaluation (time) or the model size for that entry. It can be possible that estimating 2 other entries with lower uncertainty initially, and a lower combined evaluation cost turns out to be better than evaluating the entry with highest uncertainty. Curious to know if the authors explored multi-objective optimization, or tried to incorporate evaluation cost in other versions of there proposed approach. 
- As such, the work is basically applying an existing statistical technique (matrix completion) to the problem of estimating performance on benchmarks. Authors do propose some small modifications over standard matrix factorization. One can say that using matrix completion for various applications in the real world is not a novel contribution.
- It would have been much more compelling work, if the approach incorporated VLM specific ideas or benchmark specific stuff over an above the standard matrix factorization techniques.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: A more detailed analysis is needed. Conducting PMF based on different learning methodologies, evaluation pipelines, and promptings could improve the quality of the paper.
soundness: 2
strengths: 1. This paper evaluates 108 models on 176 datasets, covering a wide range of tasks and benchmarks. This systematic evaluation can provide a foundation for many future research.
2. PMF for handling sparse data, such as tensor factorization, Bayesian PMF, and the use of model and dataset profiles, seems a more robust approach to mitigate potential weaknesses in the matrix completion task.
summary: This paper presents a framework for predicting unknown performance scores of LVLMs by formulating it as a matrix completion task using probabilistic matrix factorization with MCMC. The paper addresses the challenge of high computational costs in evaluating LVLMs and aims to reduce unnecessary evaluations by predicting performance scores based on observed ones from other models or tasks.
weaknesses: Of course, it is statistically possible to make more robust predictions, but even humans can predict the performance level of a model to some extent just by observing certain patterns in the results. However, the reasons we still need to directly evaluate are:
1. The learning methodology may show significant weaknesses or strengths on specific benchmarks, and such frameworks cannot analyze these.
2. K-shot, certain promptings, or new evaluation methods could lead to changes in results across benchmarks, but this framework lacks insights into these aspects.

Therefore, although we can statistically predict the results to some extent without directly evaluating a new model, we still confirm the actual performance through evaluation. Moreover, even testing just 10% less in a setting where only a subset of the test set is used can significantly undermine the reliability, making it even harder to trust this framework. In short, using this framework to predict scientific conclusions presents a risk that far outweighs the cost savings.

Meta-review: This paper proposes a new framework for predicting unknown performance scores based on observed ones from other LVLMs or tasks. This paper formulates the performance prediction as a matrix completion task and applies probabilistic matrix factorization with Markov chain Monte Carlo to solve this problem. This paper also introduces several improvements to enhance probabilistic matrix factorization for scenarios with sparse observed performance scores. Experiments are conducted to demonstrate the effectiveness of the proposed method.

Pros:

- The studied problem is interesting and practically important.
- This paper covers a comprehensive list of vision-language models/tasks.

Reasons to reject:

- Simply formulating the problem of LVLM performance prediction as a matrix completion problem seems not reasonable. It is a purely statistical problem if we do so, without any additional information (i.e., a small part of the test dataset).

- To solve this matrix completion problem, this paper did not propose any new method or involve the features or properties of large vision-language models (LVLMs) or vision-language tasks. I feel that even the model is a linear model or other deep models, with general tasks (instead of vision-language tasks), the matrix completion problem can be still formulated. So I think it is important to indicate why the proposed method of this paper is suitable for vision-language scenarios. From my side, I did not find any vision-language information leveraged in this paper.

Decision: Reject