Ranked from Within: Ranking Large Multimodal Models Without Labels
Authors: Weijie Tu
Weijian Deng
Dylan Campbell
Yu Yao
Jiyang Zheng
Tom Gedeon
Tongliang Liu
Venue: ICML 2025 poster (2025)
Keywords: Large Multimodal Models
visual question answering
Primary area: social_aspects->robustness

Abstract: Can the relative performance of a pre-trained large multimodal model (LMM) be predicted without access to labels? As LMMs proliferate, it becomes increasingly important to develop efficient ways to choose between them when faced with new data or tasks. The usual approach does the equivalent of giving the models an exam and marking them. We opt to avoid marking and the associated labor of determining the ground-truth answers. Instead, we explore other signals elicited and ascertain how well the models know their own limits, evaluating the effectiveness of these signals at unsupervised model ranking. We evaluate 47 state-of-the-art LMMs (e.g., LLaVA) across 9 visual question answering benchmarks, analyzing how well uncertainty-based metrics can predict relative model performance. Our findings show that uncertainty scores derived from softmax distributions provide a robust and consistent basis for ranking models across various tasks. This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.

Ranked from Within: Ranking Large Multimodal Models Without Labels
Weijie Tu
Weijian Deng' Dylan Campbell
Yu Yao? Jiyang Zheng²
Tom Gedeon 134 Tongliang Liu"
Abstract
LMMs
Can the relative performance of pre-trained large multimodal model (LMM) be predicted without access to labels? As LMMs proliferate, it be- comes increasingly important to develop efficient ways to choose between them when faced with new data or tasks. The usual approach does the equivalent of giving the models an exam and marking them. We opt avoid marking and the associated labor of determining the ground- -ruth answers. Instead, we explore other signals elicited and ascertain how well the models know their own limits, evaluating the effectiveness of these sig- nals at unsupervised model ranking. We evaluate 47 of-te-of-thh-arr art LMMs VA) across 9 visual question answering benchmarks, analyzing how well uncertainty- based metries can predict relative model performance. Our findings show that uncertainty scores derived from softmax dis- tributions provide a robust and consistent basis for ranking models across various tasks. This fa- cilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting mod- els for diverse target domains without requiring manual annotation.
Real performance
0.92
What written the image?
CLOSE
Label-free ranking score
Figure I. Overview of unsupervised ranking for LMMs. In sce- narios where labeled data is scarce, selecting the best-performing model can be challenging. Our approach introduces label-free proxy ranking score designed to reflect true performance, achiev- ing high correlation (p 0.92) with actual metries. This enables unsupervised comparison of LMMs, allowing users to identify the most suitable model without needing labeled data.
et al., 2022; Mathew et al., 2021) to identifying the spatial position of objects (Tong et al., 2024; x.ai, 2024), LMMs are increasingly widespread in practical settings. As LMMs proliferate, the need for rigorous evaluation metrics that accurately capture their capabilities and limitations has be- come more urgent. Thus, numerous benchmarks have been developed (Lu al., 2022; Hiippala et al., 2021; Masry et al., 2022; Singh et al., 2019; Mathew et al., 2021; x.ai, 2024; Yue et al., 2024), aiming to provide reliable rankings and guide users in selecting models best suited for specific deployment scenarios.
1. Introduction
Large multimodal models (LMMs), such as LLaVA (Liu al., 2024c) and InstructBLIP (Dai et al., 2023), have demonstrated remarkable capabilities in handling wide ar- ray of complex multimodal tasks, proving highly adaptable across diverse real-world applications (Liu et al., 2024a; Dai et al., 2023; Wang et al., 2024a; Huang et al., 2024; 2025; Zheng et al., 2025). From addressing scientific chal- lenges (Lu et al., 2022; Hiippala et al., 2021) and performing optical character recognition (Mishra et al., 2019: Masry
However, these benchmarks are based on carefully curated datasets that require substantial resources to develop and label. For many users who may not have access to these resources, assessing model performance can be challenging. Additionally, standard evaluations are often dataset-centric depending on fixed, human-labeled metrics that may not capture the full range of model capabilities necessary for diverse applications. As LMM tasks continue to diversify and expand, evaluating them effectively has become increas- ingly complex, with new applications frequently needing additional data curation and specialized capabilities.
Australian National University "Sydney AI Centre, The University of Sydney *Curtin University "University of ÓBuda. Correspondence to: Tom Gedeon <tom, foo..gedooneeee @ aaw..cdu.uus Tongliang Liu <tongliang. liu @iiiin.llu@@yeeeee @aud@dyyycccuuu....
Addressing the challenge of efficiently evaluating a set of LMMs is important for their usability and effectiveness in deployment. As illustrated in Figure 1, selecting the most suitable LMM from a range of available options is
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
challenging in the absence of annotations. To address this, our work investigates label-free proxy ranking scores that closely align with the true performance ranking, enabling effective model comparison in label-scarce settings.
We discuss the most relevant two lines of research as follows. Miller el al. (2021) introduce an accuracy-on-tee -ourrac-oo--th-line (AoL.) phenomenon where strong linear correlation between probit- scaled in in-distribution (ID) accuracy and (OOD) accuracy across a variety of ML models. This means ID accuracy serves as a good indicator of model perfor- mance for a target domain. Shi el al. (2024a) revisit the established concept of lowest common ancestor (LCA) dis- tance, which measures the hierarchical distance between labels and predictions within predefined class hierarchy. By the observed linear correlation between ID I.CA dis- tance and OOD accuracy, it is viable to select a model based LCA distance of predictions. This work differs from prior study that we focus on ranking LMMs. This group of mod- els makes predictions in a significantly different way from conventional classification models, which also results in different evaluation protocols.
Our first finding is that the naïve approach of using the model's performance on an existing benchmark to rank its performance new target environment could be unstable. We further report that measures of prediction uncertainty more effective ranking indicators. LMMs generate an- swers in an open-ended form by producing token sequences, with each token selected from the vocabulary based on the probability. This enables model uncertainty to be assessed using the logits at each token position.
To this end, we evaluate 45 different LMMs that have dif- ferent training frameworks, e.g., LLaVA-V1.5 (Liu et al., 2024a) and InstructBLIP (Dai et al., 2023), different visual encoders, e.g., CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023), and language models, e.g., Vicuna (Team, 2023) and LLaMA (Touvron et al., 2023). We investigate three categories of model uncertainty approaches: softmax probabilities, self-consistency, and labeled proxy sets. We evaluate the ranking performance on widely-adoptes mul- timodal benchmarks, which span diverse domains, including reasoning scientific questions, recognizing optical charac- ters, and identifying objects' spatial positions. Our main findings are that
Uncertainty Estimation for LLMs and LMMs. Uncer- tainty estimation seeks to quantify an MI. model's confi- dence in its predictions (Guo et al., 2017). Recent stud- ies have been dedicated to exploring uncertainty estima- tion specifically for LLMs (Kuhn et al., 2023; Malinin & Gales, 2021; Xiao & Wang, 2021; Huang et al., 2023; Lin et al., 2023; Kadavath et al., 2022; Azaria & Mitchell, 2023; Gottesman & Geva, 2024). For instance, Xiao et al. (2021) utilize ensemble methods to evaluate uncertainty in natural language generation models. Malinin el al. (2021) simi- larly introduce a unified approach to uncertainty estimation, leveraging ensemble methods, for autoregressive structured prediction tasks. To deal with the challenge of capturing "se- mantic equivalence" in natural language, Kuhn et al. (2023) propose semantic entropy, a method that utilizes linguistic invariances derived from shared meanings. Additionally, in- ternal states of LMMs can be leveraged for uncertainty quan- tification error detection by training a classifier (Azaria & Mitchell, 2023; Gottesman & Geva, 2024). This paper does not propose new way to estimate uncertainty. Instead, it offers the novel insight that the existing uncertainty in LMM-generated outputs effectively reflects their relative performance across benchmarks without manual labels.
the performance of models on one dataset may not accurately reflect the ranking of the same models on a different dataset (Section 4);
the effectiveness of ranking methods is influenced by task characteristics (e.g.. closed-form or free-form gen- eration), but probability-based variants are typically quite robust and predictive (Section 5); and when examining correlations in model performance across different dataset pairs, we observe that text prompt similarity better correlates with model perfor- mance across datasets than image feature similarity (Section 6).
2. Related Work
Unsupervised Model Ranking. The goal of this task is to rank and select a best performant models without the access to the data annotations of target environments. The research on this task can date back to Forster et al. (Forster, 2000), and was further investigated in various tasks: (1) outlier detection (Zhao et al., 2022; 2021); (2) image classifica- tion (Kotary et al., 2022; Tu et al., 2024a; Zohar et al., 2024; Back et al., 2022; Miller et al., 2021; Shi et al., 2024a); (3) time series anomaly detection (Ying et al., 2020); (4) multi- variate anomaly detection in manufacturing systems (Eng- bers& & Freitag, 2024), etc.
Evaluation & Benchmarking LMMs. The rapid devel- opment of LMMs has greatly propelled advancements in multimodal models, showcasing significant improvements in their perception and reasoning abilities. This shift has rendered traditional benchmarks, which focus solely on isolated task performance (Karpathy & Fei-Fei, 2015; An- tol et al., 2015). Researchers have introduced new bench- marks to evaluate LMM in a broad spectrum of multimodal tasks (Goyal et al., 2017: Lin et al., 2014; Russakovsky et al., 2015). Recent studies (Yue et al., 2024; x.ai, 2024; Fu et al., 2023) highlight the need for more comprehen- sive benchmarks to effectively evaluate the reasoning and
2

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Optical Character Recognitton
Science
Vision
General
Performence (5) on AUD0
0.74
Performance (35) on OCRVOA
80
P
-0.09
TextVQA
0.31
1.0
70-
70
ChartQA
0.94
0.96
0.27
0.66
0.85
0.77
0.51
0.58
-0.8
60
60-
50
DocVQA
0.97
0.96
0.21
0.72
0.86
0.74
0.5
0.51
40
40
OCRVOA
0.31
0.21
1
-0.091
0.53
0.52
0.33
50
60
70
80
90
100
50
60
70
100
Performance (%) SQA-
Performance (%6 on SQA-
SQA-1
0.66
0.66
0.72
-0.091
1
0.74
0.56
0.38
0.53
-0.4
189) on AIDD
80-
D
85
Performance (96 on OCRVOA
8o-
0:27
0.65
70-
70-
RWQA
0.76
0.77
0.74
0.53
0.56
0.9
0.67
0.68
-0.2
GQA
0.54
0.51
0.5
0.52
0.38
0.65
0.67
1
0.51
0.0
50-
50-
MMMU
0.49
0.51
0.33
0.53
0.68
0.51
40-
40
10
30
50
90
10
30
50
70
TextVOA
ChartDA
DacVOA
OCRVOA
SOA-1
A120
RWOA
GQA
MMMU
Performance (56) on ChartQAA
Performance (96 on ChartQA
(a) Correlation study on model performance between benchmarks
(b) Correlation matrix for 9 benchmarks
Figure 2. Correlation analysis of model performance across benchmarks. (a) Scatter plots illustrating the Spearman's rank correlation coefficients (p) hetween performance on selected henchmarks, indicating how well performance on one henchmark predicts performance on another. Fach point represents model. The straight lines are fit with robust linear regression (Huber, 201 1). (b) Heatmap of
the correlation matrix for performance across eight henchmarks, with color intensity representing the strength of correlation. Higher correlations (closer to 1) appear red, while weaker correlations approach blue. The varying correlation strength indicates that using performance on one benchmark to rank LMMs in a target deployment environment may be inconsistent or unreliable.
understanding abilities of LMMs. For instance,
ground-truth answers y:-
Several benchmarks (x.ai, 2024; Ainslie et al., 2023; Tong et al., 2024) have been developed to assess multimodal mod- els' real-world spatial understanding. Lu es al. (2024) and Zhang el al. (2024) introduce benchmarks specifically de- signed to evaluate MILMs' mathematical reasoning, focus- ing on their ability comprehend and reason about visual mathematical figures. Yue et al. (2024) carefully curate di- verse set of multi-discipling tasks that require college-level subject knowledge and complex reasoning. Additionally, nu- merous benchmarks (Mishra et al., 2019; Masry et al., 2022; Mathew et al., 2021; Liu et al., 2023) assess the performance of LMMs in optical character recognition.
The objective of this paper is to develop methods for com- puting a score Sm for each LMM without requiring access to task-specific data annotations. Ideally, these computed scores should closely correlate with ground-truth perfor- mance, allowing us rank and select LMMs based on their performance using only these scores.
Evaluation Metric. We use Spearman's rank correlation coefficient p (Kendall, 1948) evaluate the monotonic relationship between scores and model performance. Addi- tionally, we calculate Kendall's weighted rank correlation (Shich, 1998), which effectively highlights top-ranked items (You et al., 2021). Both coefficients range from [-1, 1]. with values near or indicating strong nega- tive or positive correlations, and indicating no correlation.
3. Task Formulation
Task Definition. Let multimodal task be represented by a dataset 7 = {x;, y: where x; and y: denote the input prompt and the corresponding answer for the i-th sample, respectively. We have access to M large multimodal models (LMMs), denoted as { Each LMM fm. with pre-trained weights I m- generates sequence of tokens UK {=x j=- from the input prompt via decoding process: 2z = (m,,(x2111 21, 2..... _ ), where 2д represents the k-th generated token. To assess the performance of each LMM, this task employs an evaluation metric (e.g., accuracy) that determines the ground-truth performance {gm
4. Uncertainty for Ranking LMMs
This section discusses the unique characteristics of rank- ing various LMMs compared to conventional ML models. We then introduce three distinct approaches that leverage uncertainty in model predictions for ranking.
4.1. What Makes Ranking LMMS Interesting?
Unique Challenges for Ranking LMMs. While LMMs can be considered subset of machine learning (ML. mod-
}M m= by comparing the generated sequences with the
3

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
els, ranking them introduces unique challenges not present in traditional MIL models. Below, we discuss the unique characteristics of LMMs and the challenges they bring to the model-ranking process. First, LMMs often have bil- lions of pre-trained parameters, which presents significant challenges for traditional "white-box" analysis for ranking various LMMs. Second, these models are typically trained on large datasets of instruction fine-tuning samples, which may be proprietary data. As a result, risk assessment meth- ods that require access to training data are either unsuitable or should be adjusted. Additionally, although some LMMs may disclose their final model weights, other information, such as training loss and intermediate checkpoints, often remains undisclosed. The lack of the access to training de- tails limits the use of techniques in unsupervised accuracy estimation (Deng& & Zheng, 2021; Tu et al., 2023).
first
penultimate
Input prompt:
word Close written the image [EOS]
What is written in the image?
produde that
Utd adan's
uncensiitt measure:
CLOSE
perultmaie tchan's ML7
close
CEEZZA
+ max NLL over the soqucnee?
TZZZA
original
entrpy??
Irage
average enpoppy over the sequence?
00
The word close is written the image. [EOS]
stochastic
The word close is on the Image. [EOS]
Word clock is written in the image. [EOS]
LMM
Figure 3. An example of running one LMM for a VQA task. We also present different token positions, methods to compute token- level uncertainty and the generation of stochastic predictions.
niques that utilize these readily accessible LMM outputs.
Does Accuracy-on-the-LLin (AoL) Suffice? Given these challenges, one may consider leveraging the AoL phe- nomenon (Miller et al., 2021) as a potential method for ranking LMMs. AoL refers to the strong linear correlation between probit-scaled in di-drriibuutonn (ID) accuracy and out- of-distribution (OOD) accuracy across various MI. models. This suggests that ID accuracy could serve as reliable pre- dictor of OOD performance, making AoL suitable for select- ing LMMs in target testing environments. However, there are several reasons why this is not the case. First, obtain- ing ID performance data is often impractical, as LMMs are typically trained on large, sometimes proprictary datasets, limiting direct access to ID metrics. Second, while perfor- mance on existing benchmarks is more readily available, using this data to rank models for new deployment environ- ments is problematic. Figure 2 illustrates the correlation between proxy benchmark performance and target testing environments, revealing extreme variability in correlation strength. This highlights the unreliability of using bench- mark performance as the sole ranking criterion. Third, rely- ing solely on proxy benchmark performance fails to capture the unique statistical characteristics of target testing datasets. AoL tends to rank models identically across different envi- ronments, regardless of the actual deployment context.
4.2, Assessing Uncertainty by Softmax Probabilities
LMMs generate answers in an open-form manner by produc- ing sequences of tokens. The selection of each token can be viewed as a classification process, where the model selects the token with, e.g., the highest probability over its entire vo- cabulary. This allows us to assess model uncertainty based on the logits at each token position.
For token-level uncertainty (Huang et al., 2023), we can focus on two specific positions: the first token and the penul- timate token (i.e., the token preceding the end-of-sequenee token), as illustrated in Figure 3. The logit of the first to- ken reflects the model's initial response to the input prompt, while the logit of the penultimate token captures the model's understanding of both the prompt and the generated re- sponse. The uncertainty associated with these two positions may provide insight into the model's overall confidence in answering the question. Beyond token-level uncertainty, sentence-level uncertainty can be calculated by aggregat- ing uncertainty across all tokens in the sequence (Manakul et al., 2023; Huang et al., 2023). Specifically, sentence-level uncertainty can be quantified using the mean or the maxi- mum negative log-likelihood (NLL) values across the entire generated sequence:
What can we use for ranking LMMs? Our ap- proach leverages the readily available outputs of LMMss-pecifiicllyy, token prediction logits and generated tokens-without requiring intricate architectural analysis or complex extraction techniques. Inspired by recent work on uncertainty scores for classifier ranking (Hu et al., 2024; Tu et al., 2024b), we introduce novel adaptation of these tech- niques tailored to the specific characteristics of LMMs. By analyznng the distribution of prediction logits and the vari- ability in generated outputs, we aim to assess each model's self-awareness of its limitations. This enables an unsuper- vised model ranking method, focusing primarily on tech-
NLL max (- (-log
(I)
J
NLL - Σ logPij Pij
-
(2)
where Pty is the likelihood of the word generated by the LMM at the j-th token of the i-th sentence and J is the number of tokens generated in the sentence. Equation (I) quantifies a sentence's 's uncertainty via the least likely to- ken, while Equation (2) uses the average per-token log- likelihood, allowing for length-independen comparisons
4

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
of uncertainty (Malinin & Gales, 2020; Murray & Chi- ang, 2018). Moreover, Equation (2) relates perplexity, exp (org(-logp) (Jelinek et al., 1977; Manning & Schütze, 1999), a standard measure of model quality.
generation while keeping compute overhead manageable. We adopt unigram BLEU and denote the methods using BLEU and BERTScore as SampleaLEu and SampleBERT, respectively. Furthermore, BERTScore tends to assign high similarity scores between single letters, such as "A" and "B", which should be considered very different in the contexts of MCVQ. For example, BERTScore gives a high similar- ity score of 0.998 for "A" and "B", making it ineffective for distinguishing model performance. To address this, we modified the response to include the full answer text (e.g.. "A. North America"), denoted as Sample BERT.
Alternatively, entropy H can be used instead of the negative log-likelihood to assess uncertainty. In the context of LMMs ranking, we adopt normalized entropy to account for varying vocabulary sizes across models, thus scaling entropy to the interval [0, 1] and making it comparable across different models. The normalized entropy is given by:
1 Hij log [
(is(w) (o) Σ WEW
4.4. Assessing Uncertainty With Labeled Proxy Data
where Pij( is the likelihood of the word w being gener- ated at the j-th token of the i-th sentence, and W is the set of all possible words in the vocabulary.
Beyond the uncertainty score calculated directly the tar- get dataset, we also explore scores obtained using proxy labeled dataset. Garg et al. (2022) proposed average thresh- olded confidence (ATC), which calculates a threshold on a validation set (e.g.. CIFAR- 10 (Krizhevsky & Hinton, 2009) validation set) and considers an image correctly classified new dataset with the same task if its confidence score exceeds the threshold. With the derived threshold, model performance on the target domain is estimated by the pro- portion of samples with confidence scores higher than the threshold. For LMMs, we use an existing benchmark as a proxy set to calculate 8. The calculation of ATC is
There are eight variants of output probability methods, denoted as NLLF, NLLp, NLLmax. NLLavg Ente. Enty, Entmax: and Entayy- "NLL" and "Ent" represent negative log-likelihood and entropy, respectively, while "F" and "p" refer the first and penultimate tokens.
4.3. Assessing Uncertainty by Self-Consistency
Another approach involves examining the non-deterministic generations produced by models. The core intuition is that more accurate model will produce predictions closely aligned with the original answer, while less accurate model may yield more divergent responses with each inference. In the case of LMMs, the temperature parameter t controls the randomness of predictions: a temperature of zero forces deterministic predictions, where the model selects only the token with the highest probability. When / is larger than O, the model generates tokens stochastically, selecting tokens based on probabilities above threshold. As t increases, tokens are sampled from an increasingly uniform distribu- tion (Chen et al., 2023b; Cobbe et al., 2021).
ATC Ezer (()(()) > 8]]
(4)
where I is binary indicator function, and u(-) represents an uncertainty estimation method, for which we use NLL ATC provides both ranking of model performance and an estimate of the expected performance.
5. Experiments
In this section, we first introduce the experimental setup including the evaluated datasets and models we considered. Then, we show the results of ranking different large multi- modal models (LMMs).
To analyze the consistency in these stochastic predictions, we explore two common methods: BLEU (Papineni al., 2002) and BERTScore (Zhang al., 2019). BLEU is gram-based metric that evaluates the similarity of generated sequences to the reference answer, while BERTScore uses pre-trained language model to embed answers and mea- sures similarity in embedding space. Then, we use the mean value of similarities to represent the consisteney for the in- put sample, which can be denoted as + Σ sim(P,, Por). where T is the number of stochastic inferences, sim(-) is the similarity function (e.g., BLEU), and P, and Pos are the i-th stochastic prediction and the original answer, re- spectively. Following the practice outlined in (Chen et al., 2023b; Cobbe et al., 2021; Huang et al., 2023), we collect five stochastic inferences per sample and set t = 0.7 to maintain relatively high degree of stochasticity in LMM
5.1. Experiment Setup
Benchmarks. We choose the task of visual question an- swering, which is common way to evaluate LMMs (Liu et al., 2024a; Dai et al., 2023; Beyer et al., 2024; Lu et al., 2022; Yue et al., 2024). We evaluate on multiple choice visual question (MCVQ) and visual question answering (VQA) benchmarks. MCQ and VQA are both types of question-answertnn formats for evaluating LMMs. For the former, the model is provided with several answer op- tions, out of which the correct subset is to be selected. In contrast, the latter is usually open-ended and the mod- els may generate answers freely. We consider 8 widely- adopted MCVQ and VQA benchmarks. They are (1) the
5

---

Ranked from Within: Ranking Large Multimodal Models Without Labels
MCYO
VQA
Method
SQUJ
A12D
RWQA
MMMU
GQA
ChartiA
OURVOA
TexTVQA
Doc
Average
P
Ts
P
T
Te
P
Te:
Tv
P
Tv
Tol
P
Ty
To
Aut.
0.52
0:45
0.73
0.61
0.70
0.57
0.54
0.45
0.53
0.32
0.69
0.50
0.30
0.20
0.69
0.57
0.68
0.60
0.60
0.48
NLL
0.83
0.78
0.84
0.776
0.56
1.58
[0.00
00.59
0.71
0.57
1.68
0.41
0.78
0.66
0.74
0.63
0.54
0.74
0.77
[.63
NLL.
0.64
0.60
0.61
0.58
0.25
0.37
0.15
0.18
0.58
0.52
0.79
0.67
00.81
0.68
0.71
0.67
0.88
0.78
0.60
0.56
NLL...
10:00
0.76
0.91
0.81
0.63
0.63
0.49
0.44
0.72
0.50
0.94
0.80
0.66
0.63
0.83
0.69
0.92
0.80
0.76
0.60
NLL..,
0.72
0.64
0.88
0.73
0.64
0.56
0.50
0.40
0.67
0.55
0.02
0.75
00.81
0.65
0.81
0.72
0.93
0.82
0.70
0.05
Ents
0.59
0.51
0.82
0.50
0.65
0.56
0.64
0.52
0.54
0.20
0.64
0.45
0.69
0.57
0.71
0.56
0.50
0.70
0.68
0.52
0.49
0.20
0.69
0.48
0.48
3.43
0.34
0.24
0.46
0.21
11.22
0.64
0.801
0.164
0.70
0.54
0.86
0.08
0.02
0.47
Entea
0.58
0.39
0.82
0.50
0.67
0.80
00.85
0.54
0.21
0.88
0.06
0.53
0.52
0.76
0.90
0.87
0.73
0.70
0.54
0.80
0.50
0.50
0.91
0.08
0.52
Samplexaru
0.78
0.36
0.37
0.89
0.61
0.60
0.51
Samplegaa
0.4G
0.52
0.70
0.55
0.52
0.28
0.45
0.44
0.51
0:47
0.300
0.61
0.75
0.72
0.77
0.30
0.89
0.64
0.06
0.54
Sample LEETT
0.67
0.56
0.78
0.62
0.07
0.48
0.59
0.39
0.51
0.47
0.90
0.61
0.75
0.72
0.77
0.30
0.69
0.64
0.73
0.58
ATC
0.73
0.74
0.40
0.41
0.95
0.60 0.64
0.54
Table 1. Method comparison across eight multimodal tasks. The table presents a comparison of four groups of methods: accuracy-based, prttt-pobbbbbbbbbbaeeeea sample-based, and unsupervised model evaluation methods. We evaluate these methods using Spearman's is rank correlation (p) and weighted Kendall's correlation (7w). Both coefficients range from -1 to 1, where values close to 11 or indicate strong negative or positive correlations, respectively, and 0 indicates no correlation. The AoL and ATC performance is calculated as the average correlation when using the scores computed on the other seven domains and the model performance on the target domain. The highest correlation values for each task are highlighted in green while the second highest values are marked in blue. All methods are ranked three decimal places. Note that, we use the absolute value of correlation strength in the table for NLL and Ent. The results indicate that NL. and Nw are often preferable, as they show greater stability and stronger correlation with model performance.
subset of ScienceQA (Lu et al., 2022) with images (SQA- I) and A12D (Hiippala et al., 2021) which assess LMMs'
consistently high correlation with model performance in seven out of the eight benchmarks. Although AoL shows strong results on RealWor, where it performs best, uncertainty- methods also demonstrate high correla- tion strength. For instance, NLL exhibits a Spearman's 's P 0.07 lower but a weighted Kendall's Tue 0.06 higher than AoL. These findings suggest that relying on existing bench- marks alone to select models for deployment could be risky and unstable, as they may not well capture the statistical characteristics of the new target domain.
scientific knowledge; (2) ChartQA (Masry et al.. 2022), OCRVQA (Mishra et al., 2019), TextVQA (Singh et al., 2019) and DocVQA (Mathew et al., 2021) to examine their ability to recognize optical character; (3) W (RWQA) (x.ai, 2024) and GQA (Ainslie al., 2023) which evaluate LMMs' vision-centric capability; (4) MMMU (Yue et al., 2024) which assays LMMs on multi-disciplinary tasks that demand college-level subject knowledge. Note that SQA-I, A12D, RWQA and MMMU are MCVQ datasets, while the others are VQA.
The choice of token matters for output probability- ranking. We studied four ways of using tokens for estimat- ing model uncertainty. For the two on specific positions, the first and penultimate tokens, their predictive performance depends on the task type (i.e., MCVQ vs. VQA). The un- certainty associated with generating the first token is more indicative for MCVQ tasks, while the penultimate token generally proves more predictive for VQA tasks. This dif- ference is due the nature of MCVQ tasks, which often prompt models to generate only single option letter (e.g.. "A"), making the first token crucial for evaluating response accuracy. In contrast, VQA tasks require forn re- sponses consisting of multiple tokens, making the penulti- mate token-reflecting the model's understanding of both the question and the answer-more informative.
Models. The goal is to choose the best LMM over all different series of LMMs. We include LLaVA- 1.5 (Liu et al., 2024a), ShareGPT4V (Chen et al., 2023a), LLa NeXT (Liu et al., 2024b), InstructBI.II (Dai et al., 2023), LLa LaaT---aeT-lmttrllave (Li et al., 2024b), LLaVA- One (Li et al., 2024a), Eagle (Shi et al., 2024b),
mPLUG-OWl (Li et al., 2022), InternVI. (Chen et al., 2024),
PaliGemma (Beyer et al., 2024). Mantis (Jiang et al., 2024), DeepSeek-VL2 (Wu et al., 2024) and Qwen2-VL (Wang et al., 2024b). In total, we collect 32 different models, which all can be accessed on Hugging Face ( (Wolf et al., 2020).
5.2. Key Findings
Accuracy-on-the-Line is unreliable for ranking LMMs in new domains. Table 1 summarizes the ranking capability of all methods across eight benchmarks. The AoL perfor- mance is calculated as the average correlation strength when using the other seven domains to predict model rankings in the target domain. We observe that Aol, does not achieve
For the two that consider every token in the generated output, ie.. NLI max and NLL we find that they are more stable and less task-specific compared to variants that consider individual tokens. While NLL and Ent,ax (reflecting the least confident token) is more effective for MCVQ tasks, both methods perform similarly on VQA tasks. The higher
6

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
ranking correlation of using the first token and the least con- fident token as ranking indicators for MCVQ suggests that single token (typically the option letter) holds significant meaning. Considering all potential tokens can sometimes result in an unexpectedly higher confidence level, as LMMs may generate the complete answer alongside the option let- ter (e.g., "B. Columbia"). The tokens following the option letter often have high confidence levels, leading to an overall increase in estimated uncertainty. This tendency results in models generating entire answers with lower uncertainty scores, yielding higher ranks. These findings underscore the importance of further exploration into the optimal token positions for uncertainty estimation in LMMs. One poten- tial approach is leveraging a language model to identify the position of the option letter and use its softmax probability as the uncertainty measure for the complete response.
MCVQ
VQA
Method
Average
A12D
MMMU
TextVQA CharQA
AoL
0.42
0.43
0.57
0.64
0.52
NLL
0.29
0.59
0.97
0.99
0.71
NL
0.26
0.28
0.98
0.98
0.63
Entmax
0.37
0.62
0.97
0.96
0.73
Entag
0.11
0.38
0.98
0.98
0.63
SamplesLEu
0.63
0.73
0.76
0.64
0.69
ATC
0.12
0.59
0.85
0.84
0.60
Table 2. Method comparison for ranking different LLaVA- prismatic models on A12D and TexIVQA. We use Spearman's 's rank correlation (p) as the metric. We observe that uncertainty- based method still exhibit moderately high correlation strength, which indicate their effectiveness in ranking aAV models.
model architecture internal states. This highlights their potential rank API-based or closed-source LMMs (e.g.. GPT-4V (Achiam et al., 2023)).
The negative log-likelihood (NLL) is more stable and predictive for LMMs ranking than normalized entropy. For seven out of eight benchmarks, NLL consistently shows stronger correlation with model performance than entropy. Both NLL and entropy make use of the softmax proba- bility distribution predicted for each token during genera- tion. However, the NLL-based approaches only consider the maximum probability in this distribution, while the en- tropy considers all entries. The lower correlation strength of normalized entropy may therefore stem from the signif- icantly different vocabulary sizes of various LMMs. For instance, LLaVA-VI.5-7B has vocabulary of 32k tokens, while PaliGemma-3B-nixs has much larger vocabulary of 257k tokens. Despite being normalized by vocabulary size, entropy may still be more susceptible to noise than NLL. Future work could include applying dimension reduc- tion techniques or limiting consideration to the top-k most probable tokens or a pre-defined set of tokens.
ATC can be used for LMMs ranking, but the correla- tion strength is influenced by the choice of proxy dataset. We compute ATC performance as the average correlation when using the other seven datasets as proxy datasets. Our analysis reveals the choice of proxy dataset is critical, as the scale of uncertainty calculated on different datasets can vary. This variation can lead to an inaccurately estimated threshold for determining instance correctness. A poten- tial improvement involves adopting uncertainty calibration methods, such as temperature scaling (Guo et al., 2017), to calibrate model uncertainty onto a consistent scale.
6. Analysis
This section includes three distinct analyses. The first specif- ically investigates whether the considered methods are ef- fective for ranking models within the same series. We use LLaVA models as a case study because they are widely adopted and representative of LMM architectures. The other two analyses consider models from different series, consis- tent with the broader evaluation in Section 5.
Sample-based methods are strong candidates for model ranking without intrinsic access to LMMs. The corre- lation strength of cample-based methods is influenced by the nature of the task. They yield higher correlation scores on VQA tasks compared to MCVQ tasks. In VQA, models typically produce more varied responses, making BLBU and BERTScore effective for capturing uncertainty. However, MCVQ tasks constrain models to select from a pre-defined set answers. While non-zero temperature introduces some randomness, the variation between stochastic inferences is limited. Additionally, we also find that Sample BERT has higher correlation scores than SampleBERT on four MCVQ benchmarks and suggests that developing more advanced algorithms to capture uncertainty from multiple stochastic inferences could be beneficial.
Ranking LMMs from the same series. So far, all ex- periments have focused on selecting LMMs from different model series. However, in some scenarios, the objective is to choose a training recipe that yields better-perforning LMMs within the same model series. These models may be trained with additional finettunig steps, varied data aug- mentations, different training sources, or different language models (e.g., Vicuna and Mistral (Jiang et al., 2023)) and visual encoders, such as CLIP and DINOv2 (Oquab et al., 2023). For our analysis, we use the LLaVA series due to its widespread adoption. Specifically, we employ 15 dif- ferent LL prismatic (A.LaVA-rri) models (Karamcheti
Although sample-basee based methods show weaker overall corre- lations, they remain competitive without requiring access to
7

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Corresttion of mooe performnnee
MCVQ
VQA
Correasoon of mooe pertor
Method
A12D
MMMU
TextVQA
ChartQA
#Samples
3088
1050
5000
2500
50 samples
0.89
0.35
0.92
0.98
+2-
NLLmu
0.92
0.56
0.82
0.93
Fröchet distances boxt features
100
125 iDe
175
200
136
Fréchet distaness of Image festures
340
140
183
200
100
240
200
Table 3. Labelling a subset of target domain to rank models on AI2D, MMMU, TexiVQA and ChariQA. We report the Spear- man's rank correlation (p). While a small labeled set provides a reasonable ranking, it may not fully capture the overall order. In contrast, NLLmax offers more stable correlations, highlighting its potential for label-free model ranking.
Figure 4. Correlation Analysis of Fréehet Distances and Model Performance Correlation Across Datasets. Orange stars indi- cate the dataset pairs with the highest similarity for each dataset. Observations reveal that variations in text prompt similarity are more closely aligned with changes in performance correlation than variations in image feature similarity.
influence the correlation strength of model performance, we utilize CLIP-L-14-333 (Radford et al., 2021) to extract im- age and text embeddings. We then use the Fréchet distance (FD) (Fréchet, 1957) to measure the similarity between datasets based on these features.
et al., 2024). We report results on four datasets in Table 2: AI2D, MMMU, Tex,VQA, and ChartQA, with AI2D and Tex being used to evaluate the performance of differ- ent VA variants in the original paper. The AoL and ATC performance metries are computed in the same manner described in Section 5, using the average correlation across other three datasets.
Figure 4 presents a correlation study between the FD of dataset pairs and the correlation strength of model perfor- mance on those datasets. We observe strong correlation for FDs computed using text input features, while FDs mea- sured by image features show a weaker correlation. This suggests that text input similarity is likely more influen- tial factor for model performance correlation than image similarity. Additionally, orange stars are used to label the points representing the lowest FD for each dataset. More- over, OCRVQA shows a low correlation strength with other datasets (Figure 2). The closest dataset in prompt feature space is TextVQA, with FD of 124, which is considerably higher than the lowest FDs observed between other dataset pairs, typically around 70 or lower. This difference sheds light on the weak model performance correlation between OCRVQA and other OCR-foc datasets.
Our findings align with those from ranking different series of LMMs. First, using model performance on a single ex- isting dataset may not accurately reflect LMMs rankings on different domain, even when the models are trained with a similar pipeline and minor variations. Second, we observe that ranking different LLaVA-pri models in MCVQ presents a significant challenge, as the variance in model performance on MCVQ tasks lower compared to VQA tasks. For instance, the performance gap on A12D is only 4%, whereas the gap on TextVQA exceeds 20%. This indi- cates that ranking methods must capture subtle differences between models. We find that SampleBLEE remains effec- tive, while NLL and Entropy may not capture these nuances accurately. Additionally, the decrease in correlation between uncertainty estimated by softmax probability suggests that although modifications to the training pipeline may have slight effect on performance, they can lead significant variations in the confidence levels for generation. This find- ing underscores the importance of assessing LMMs more comprehensively, beyond accuracy alone.
Ranking LMMs by a labeled subset of the target domain. Table 3 presents the correlation strength between model performance on 50 labeled instances in the target domain and the overall performance across the entire dataset. We observe that small labeled set can provide good ranking. However, such approach may not fully capture the model ranking across the entire dataset, since the sampled data may not be representative (Polo et al., 2024). In contrast, NLL gives a more stable correlation, highlighting the methods for effective model ranking without data annotation.
Analysis of the weak correlation of AoL. Figure 2(b) il- lustrates the correlation model performance across differ- ent benchmarks. We observe that while TextVQA, CharlQA, VQA, and OCRVQA all aim assess the capability of LMMs to recognize optical characters, the correlation be- tween them varies significantly. Specifically, model perfor- mance on TexIVQA, ChartQA, and Doe VQA shows strong correlations, whereas performance on OCRVQA consis- tently exhibits low correlation with the other three datasets. To explore whether the images or texts within these datasets
potential of uncertainty-
7. Conclusion and Discussion
This work studied whether the performance of large multi- modal models in a new target domain can be ranked without the access to target domain labels. Our analysis identi- fied only a weak correlation in model performance across
8

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
different domains. This motivated the investigation and evaluation of alternative approaches based on uncertainty estimates obtained from model predictions. We evaluate 45 LMMs on closed and open-ended visual question answering tasks, testing various training frameworks, visual encoders, and language models. Our experiments reveal that scores based on the negative log-likelihood of generated tokens serve as highly effective performance indicators for target domains. We also find that while stochastic sampling can be helpful, it is less effective for multiple-choice tasks, where it requires many repetitions of inference and careful tempera- ture tuning. By establishing a baseline for uncertainty-base -based LMMs ranking, this study aims to motivate and inspire further research into this important area.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Ainslie, J., TeerTh J., De Jong, M., Zemlyanskiy, Y., Lebrón, F, and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points. In EMNLP, 2023.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. Vqa: Visual question answering. In ICCV, pp. 2425-2433, 2015.
Azaria, A. and Mitchell, T. The internal state of an LLM knows when it's lying. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: http:///pppeeeieeeeee net/ forum? id Ld-y2V6YgLaW7
Potential future directions. Beyond uncertainty scores, several promising directions remain for future research. Test-time augmentation and semantic entropy present natural next steps. Additionally, the internal states of LMMs (Azaria & Mitchell, 2023; Gottesman & Geva, 2024) offer opportunities for uncertainty estimation or eгror de- tection, for example, by training a classifier on these repre- sentations. However, in unsupervised LMM ranking, this approach requires training a separate classifier per model, which can be computationally expensive. An alternative is to measure the statistical distance between a model's inter- nal state for given response and the distribution of internal states from multiple inference passes. A larger average distance may signal greater uncertainty and potentially in- dicate a lower model rank. Exploring how internal states can inform LMM ranking an open direction. Finally, the observed asymmetry in the impact of text and image fea- ture dissimilarity on cross-dataset correlations highlights valuable area for improving multimodal benchmark design.
Back, C., Jiang, Y., Raghunathan, A., and Kolter, J. Z. peremenn--nn-he--iee Predicting the performance of neural networks under distribution shift. In NeurlPS, pp. 19274-19289, 2022.
Beyer, L., Steiner, A., Pinto, A. S., Kolesnikov, A., Wang, X., Salz, D., Neumann, M., Alabdulmohsin, I,, Tschan- nen, M., Bugliarello, E., et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024.
Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang. J., Zhao, F., and Lin, D. Sharegpt4v: Improving large models with better captions. In ECCV, 2023a.
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models to self-debug. In ICLR, 2023b.
Acknowledgments
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing. S., Zhong, M., Zhang, Q., Zhu, X., Lu, I.., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pp. 24185- 24198, 2024.
We thank all anonymous reviewers for their con- structive feedback, which helped improve the qual- ity of this paper. Tongliang Liu partially sup- ported by the following Australian Research Council projects: FT220100318, DP220102121, LP220100527, LP220200949, and IC190100031.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1466,, 2021.
Impact Statement
This paper aims to advance machine learning while acknowl- edging potential societal impacts. Our findings could be misused: adversarial researchers may train models that con- sistently get ranked higher than other models, despite per- forming poorly on data. To mitigate this, incorporating robust calibration methods would be helpful, making model confidence accurately reflects uncertainty. This would help promote safe and responsible deployment of our findings.
Dai, W., Li, J. Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose lanoua-anguuae models with instruction tuning. In NeurlPS, 2023.
Deng, W. and Zheng, L. Are labels always necessary for classifier accuracy evaluation? In CVPR, pp. 15069 15078, June 2021.

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong. X., Zang, Y., Zhang, P., Wang, J., Lin, D., and Chen, K. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024. URL https: / aaa.iivv...// /ttpaa/wogg//////a//////../ooo/t abs/2407 11691.
Hu, D., Luo, M., Liang, J., and Foo, C.-S. Towards reliable model selection for unsupervised domain adaptation: An empirical study and certified baseline. In Adv. Neural Inform. Process. Syst. Datasets and Benchmarks Track, 2024. URL https::/t///...... htw.:pp://ppeeeeeiieeeeee wpwpeeeeeeeeeeeeeeeeeeeeeee net/ forum? id=rI7kbFISpr.
Engbers, H. and Freitag, M. Automated model selection for multivariate anomaly detection in manufacturing systems. Journal of Intelligent Manufacturing, 2024, doi: 10.1007/ $10845-024-004479.. URL https: /dw.//do...///// wwl.do..rroo org/10. 1007/s10845 024 02479-2.
Huang, Y., Song. J., Wang, Z., Zhao, S., Chen, H., Juefei- Xu, F., and Ma, L. Look before you leap: An exploratory study of uncertainty measurement for large language mod- els. In ICSE, 2023.
Huang, Z., Liu, C., Dong, Y., Su, H., Zheng, S., and Liu, T. Machine vision therapy: Multimodal large language models can enhance visual robustness via denoising in- context learning. In ICML, 2024.
Forster, M. R. Key concepts in model selection: Perfor- mance and generalizability. Journal of mathematical psychology, 44(1):205-233 2000.
Fréchet, M. Sur la distance de deux lois de proba- bilité. Comptes Rendus Hebdomadaires des Seances de L. Academie des Sciences, 244(6):689-669 1957.
Huang, Z., Niu, G., Han, B., Sugiyama, M., and Liu, T. To- wards out-of-modal generalization without instance-level modal correspondence. In ICLR, 2025, URL https:
/ /pperevieew net/ wwrumm/rrummmmmm net/:orumiiddLaVWane Lid-LaulfPLLW..
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang. M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A comprehen- sive evaluation benchmark for multimodal large language models. arXiv preprint 20w::3006.13944 2023.
Huber, P. Robust statistics. In International Encyclopedia of Statistical Science, pp. 1248-1251 Springer, 2011.
Jelinek, F., Mercer, R. L., Bahl, L.. R., and Baker, J. M. Perplexity-a measure of the difficulty of speech recogni- tion tasks. Journal of the Acoustical Society of America, 62, 1977. URL httts:::: //app/ppppppp/// semant. ieraanttiiccie escholas org htts/CorpusDD::: :21 121618873. 680873
Garg, S., Balakrishnan, S., Lipton, Z. C., Neyshabur, B., and Sedghi. H. Leveraging unlabeled data to predict out-of-distribution performance. In ICLR, 2022.
Gottesman, D. and Geva, M. Estimating knowledge in large language models without generating single token. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Pro- ceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 3994-4019, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10.18653w1120244emmmm. 232. URL ht https:tpp/ /tl//acaanntt //aaaaaaanoolooooo.oooooo 020.22244.../ 2024. emn 1-mpppmait in. 232/.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b, arXiv preprint arXiv:2310.0682. 2023.
Jiang, D., He, X., Zeng, H., Wei, C., Ku, M., Liu, Q. and Chen, W. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483 2024.
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Slatfield-Doddss Z., DasSarma, N., Tran-Johnson, E, Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernande7, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. Language models (mostly) know what they know. arXiv preprint arXiv:2207.0522. 2022.
Goyal, Y., Khot, T., Summers-Star D., Batra, D., and Parikh, D. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pp. 6904-6913, 2017.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In ICML, pp. 1321-1330, 2017.
Hiippala, T., Alikhani, M., Haverinen, J., Kalliokoski, T., Logacheva, E., Orekhova, S.. Tuomainen, A., Stone, M., and Bateman, J. A. Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams. Language Re- sources and Evaluation, 55:661-688, 2021.
Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., and Sadigh, D. Prismatic vlms: Investigating the design space of visually-condittooed language models. In ICML, 2024.
Karpathy, A. and Fei-Fei, L. Deep visual-semantic align- ments for generating image descriptions. In CVPR, pp. 3128-3137, 2015.
Hinton, G. E. and Roweis, S. Stochastic neighbor embed- ding. In NeurlPS, 2002.
10

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Kendall, M. G. Rank correlation methods. Griffin, 1948.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Math- vista: Evaluating mathematical reasoning of foundation models in visual contexts, In ICLR, 2024.
Kotary, J., Di Vito, V., and Fioretto, F. Differentiable model selection for ensemble learning. In IJCAI, 2022.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. 2009.
Malinin, A. and Gales, M. Uncertainty estimation in autore- gressive structured prediction. In ICLR, 2020.
Kuhn, L.,, Gal, Y., and Farquhar, S. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natu- ral language generation. In ICLR, 2023.
Malinin, A. and Gales, M. Uncertainty estimation in autore-
gressive structured prediction. In ICLR, 2021.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a.
Manakul, P., Liusie, A., and Gales, M. J. Selfcheckgpt: Zero- resource black box hallucination detection for generative large language models. In ACI., 2023.
Manning, C. D. and Schütze, H. Foundations of Statistical Natural Language Processing. MIT press, 1999.
Li, C., Xu, H., Tian, J., Wang, W., Yan, Bi, B., Ye, J., Chen, H., Xu, G., Cao, Z., et al. mplug: Effective and efficient vision- languaee learning by cross-modie skip- connections. arXiv preprint arXiv:2205. 12005, 2022.
Masry, A., Long, D. X., Tan, J. Q.. Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022.
Li, F, Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., and Li, C. Llava-next: Tackling multi-image video, and 3d in large multimodal models, June 2024b. URL https:
Mathew, M., Karatzas, D., and Jawahar, C. Docvqa: A dataset for vqa on document images. WACV, pp. 2200- 2209, 2021.
gitttuu.. io/blog/ 1aa-16661lvv--exxttee -nee--aer--eet----eeelleeeeee
Miller, P., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L. Accuracy on the line: the strong correlation between out-of-distribun and in-distribution generalization. In ICML, pp. 7721-7735. 2021.
Lin, T--Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects context. In ECCV, pp. 740-755. Springer, 2014.
Lin, Z., Trivedi, S.. and Sun, Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXhv:2305.19188 2023.
Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, pp. 947-952, 2019. doi: 10.1109/ ICDAR.2019.00156
Liu, H., Li, Li, Y., and Lee, J. Improved baselines with visual instruction tuning. In CVPR, pp. 26296-26306, 2024a.
Murray, K. and Chiang, D. Correcting length bias in neural machine translation. In Bojar, O., Chatterjee, R., Feder- mann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Yepes, J., Koehn, P., Monz, C., Negri, M., Névéol, A., Neves, M., Post, M., Specia, L., Turchi, M., and Verspoor, K. (eds.), Proceedings of the Third Conference on Ma- chine Translation: Research Papers, pp. 212-223, Brus- sels, Belgium, October 2018. 8. Association for Computa- tional Linguistics. doi: 10.1 11/86///1//11-6633 https: / ac www.aannchhooooooooooooooooooc hww org / 1 8- 6322.
Liu, H. C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024b. URL
wwwtwww.ww.......i..... io/blog/ 2024-01-30-11ava-
Liu, H., Li, C., Wu, Q. and Lee, Y. J. Visual instruction tuning. In NeurlPS, 2024c.
URL
Liu, Y., Li, Z., Yang, B., Li, C., Yin, X., Liu, C.-I,, Jin, L., and Bai, X. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07895 2023.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, EI- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. TMLR, 2023.
Lu, P., Mishra, S. Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurlPS, pp. 2507-2521, 2022.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: A method for automatic evaluation machine translation. ACL, pp. 311-318. Association for Computational Linguistics, 2002.

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Polo, F. M., Weber, L., Choshen, L.,, Sun, Y., Xu, G., and Yurochkin, M. tinybenchmarks: evaluating Ilms with fewer examples. arXiv preprint arXiv:2402.14992 2024.
Wang, H., Huang. Z., Lin, Z., and Liu, T. NoiseGPT: Label noise detection and rectification through proba- bility curvature. In NeurIPS, 2024a. URL https: / //openreview. net htt./oorummmm/ www.mmpdd-/vvvvnge....
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML., pp. 8748-8763 2021.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang. J., Ge, W., et al. Qwen2-vl: Enhancing lasoun-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. IJV, 115:211-201 2015.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., and Brew, Transformers: State-of-the-ar natural language processing. EMNLP, pp. 38-45, 2020.
Shi, J., Gare, G., Tian, J., Chai, S., Lin, Z., Vasudevan, A., Feng, D., Ferroni, F., and Kong, S. Lea-on-the- line: Benchmarking out-of-distribution generalization with class taxonomies. In ICML, 2024a.
Wu, Z., Chen, X., Pan, Z.,, Liu, X., Liu, W., Dai, D., Gao, H., Ma. Y.. Wu, C., Wang. B., Xie, Z., Wu, Y., Hu, K., Wang, J., Sun, Y., Li, Y., Piao, Y., Guan, K., Liu, A., Xie, X., You, Y., Dong, K., Yu, X., Zhang, H., Zhao, L., Wang. Y., and Ruan, C. Deepseek-v12. Mixture-of- experts Assion-language models for advanced multimodal understanding, 2024. URL https: /tt//viivvvvoro../eeeeee........ abs 22s22 10302.
Shi, M., Liu, F., Wang, S., Liao, S., Radhakrishnan, S., Huang, D.-A., Yin, H., Sapra, K., Yacoob, Y., Shi, H., Catanzaro, B., Tao, A., Kautz, J., Yu, Z.,, and Liu, G. Eagle: Exploring the design space for multimodal Ilms with mixture of encoders. arXiv:2408. 15998, 2024b.
Shich, G. S. A weighted kendall's tau statistic. Statistics & Probability Letters, 39(1):17-24, 1998.
x.ai. Grok 1.5v: The future of ai models, 2024. URL https://xe https:///.a//////////oo gttp///rook15500
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In CVPR, pp. 8317-8326, 2019.
Xiao, Y. and Wang, W. Y. On hallucination and predictive uncertainty in conditional language generation. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2734 2734-2744, 2021.
Team, V. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023, URL https: //Imsys. htt//blog///2/ 3012233022 3--3-300------- -3-a3-vviuna// cuna/. *Based on evaluations comparing Vicuna's responses to those of ChatGPT.
Ying, Y., Duan, J., Wang, C., Wang, Y., Huang, C., and Xu, B. Automated model selection for time-series anomaly detection. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3428-3222 01884444.4444 ACM, 2020.
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In CVPR, pp. 9568-9578, 2024.
You, K., Liu, Y., Wang. J., and Long, M. Logme: Practical assessment of pre-trained models for transfer learning. In ICML, 2021.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv: 13971 2023.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang. D.,, Ren, W., Sun, Y., et al. Mmmu: A massive multi multi-dis multimodal understanding and reasoning benchmark for expert agi. In CVPR, pp. 9556- 9567, 2024.
Tu, W., Deng, W., Gedeon, T., and Zheng, L. A bag-of- prototypes representation for dataset-level applications. In CVPR, pp. 2881 2892, June 2023.
Tu. W., Deng, W., Zheng, L., and Gedeon, T. What does softmax probability tell us about classifiers ranking across diverse test conditions? TMLR, 2024a.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In CVPR, 2023.
Tu, W., Deng, W., Zheng. L.,, and Gedeon, T. What does softmax probability tell us about classifiers ranking across diverse test conditions? Transactions on Machine Learn- ing Research, 2024b. ISSN / //openreview net/ forum? ? ddvt iDggGjxx
Zhang, R., Jiang, D.,, Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.W.,, Qiao, Y., et al. Mathverse: Does your multi-modal I1m truly see the diagrams in visual math problems? In ECCV, pp. 169-186, 2024.
URL ht https:
12

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Zhang, T., Das, R., Sima'an, an, K., and Sutskever, Bertscore: Evaluating text generation with bert. In ICLR, 2019.
Zhao, Y., Rossi, R., and Akoglu, L. Automatic unsupervised outlier model selection. NeurlPS, pp. 0489-4502, 2021.
Zhao, Y., Zhang, S., and Akoglu, L. Toward unsupervised outlier model selection. In ICDM, pp. 773-782. IEEE, 2022.
Zheng, J., Shen, J., Yao, Y., Wang, M., Yang. Y., Wang, D., and Liu, T. Chain-of-focus prompting: Leverag- ing sequential visual cues to prompt large autoregres- sive vision models. ICLR, 2025. URL https: //openreview net/ noidwwBB.....
Zohar, O., Huang, S.-C., Wang. K.-.,, and Yeung, S. Lovm: Language-only vision model selection. In Ad- vances Neural Information Processing Systems Track on Datasets and Benchmarks, volume 36, 2024.
13

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
A. Appendix
In this supplementary material, we first introduce the experimental details including the models, datasets and compute in Appendix B. Next, we visualize the image and text features using t-SNE plots. Last, we show the full results of the correlation study all datasets.
B. Experiment Details
B.1. Datasets
All experiments are conducted on VLMEvalKit (Duan et al., 2024). We consider 8 datasets and their corresponding links to download TSV files via the toolkit:
ScienceQA (Lu al., 2022) histpp::/opprrccopasss www.xlab.bbcc.o..ooomm.......... httpp///........a..oo//////////r
TEST. tsv);
A12D (Hiippala et al., 2021) http:///waaaaaoooo..././////////
tsv);
ChartQA (Masry et al., 2022) http::///..pooooe..........aaats
tsv);
OCRVQA (Mishra al., 2019)
FESTCORE.tsv);
TextVQA (Singh et al., 2019) http:///who.aa/aoa/a/////a/////a
tsv);
DocVQA (Mathew et al., 2021)
=sv);
RealWorldQA (x.ai, 2024)
MMMU (Yue al., 2024) http::///g...cooa......../e/b/es
tsv)
GQA (Ainslie et al., 2023) http::///aaaoooos....eeaa..aaaaa
Bal l aellanddddee.... ttv)
B.2. Models
We include a diverse array of large multimodal models from 12 series:
VL.
1lava.v1. .5.bb vl.va.v1.5.133
LLaVA-NeXT
1lava next mistral 7b [aava_next.vicune..7 Llavanexxt.vicuaa.1S3
LLaVe-NeXT-mmtrleuwe
ilava.exttinterrleve.. 1lava next interleave 7b dpo
14

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Vision
llwaa_oneeiiioon.qveeeoo 5b.ow
Llava.ooreiisioo...eeee.o.oooo
[lava_onevisioon.ween2].. 5b.si
ilava.onevissioo.ovenn..bbba
ShareGPT4V
sharegpt4v.7b sharegpt4v.13b
InstructBLIP
InstructBLIP-N _7b
InstructBLIP 13b
Eagle
Eagle-X5-7B
fagle-X5-138
InternVL
Mini--nnterm2---hha-------55
InternVI2-1B
InternVL2-2B
InternVL2-4B
InternV12-88
PaliGemma
paligemma- 3-b--bb---- 3b-mixx222
paligemma-333 -3x-1xx444888
Mantis
Mantis Ideffcce
Vantis-BB-olip-llmma) Mantis- siglip- 11ama3
mPLUU-Ow12
MPLUG-Owl.
Qwen2-VL
Qwen2-VL-2B ruct
15

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Qwen- VL
Seepseek_v12..t
LLaVA Prismatic
reproodcctoor-1lavvvv11BB9
one-stage+7b
full-ft-mulii-sttee-f
full-ft-oone-sttgee+
inlk-224px+7b
dinov2-224pxt7b
clip-224px+7b
siglip-224pxtTb
cll--35px-restee-cro0-b
clip-B3ffx--eesiee-nnoee+++
aiglip=-84p--1eeteerbxxf1
1lama22-1o--cob.raiinnnng11
1lava-1iis4++7b
1lava-1rv+7b
-lava-lvis4-1rvvt7
B.3. Compute and Library
PyTorch version is 2.01.0+cul17. All experiment is run on four A6000 GPUs. All 45 models can be downloaded via Huggingface with different versions of transformer library:
cransformers==4.33 3 for mPLUG-OW|2 (Li al., 2022) and InstructBLIP (Dai et al., 2023);
trans ftamerormess=44. 37.0 for LLaVA-V1.5 (Liu et al., 2024a), ShareGPT4V (Chen et al., 2023a), InternVL (Chen et al., 2024) series;
cranssorreerg:= for LLaVA-NeXT (Liu et al., 2024b), LLaVA-OneVision (Li et al., 2024a), LLaVA-NeXT- Interleave (Li et al., 2024b) PaliGemma-3B (Beyer et al., 2024), Mantis (Jiang et al., 2024), Eagle (Shi et al., 2024b) and LLaVA Prismatic (Karamcheti et al., 2024) series.
C. Visualization of Image and Text Features
In the main paper, we demonstrate that distances in text features contribute more significantly to the weak correlation of model performance across datasets than image features. To visualize this finding, we utilize t-distributed stochastic neighbor embedding (t-SNE) (Hinton & Roweis, 2002). Our results show that CLIP-LL-14-336 (Radford et al., 2021) effectively captures distinctions between images from different datasets, with images from the same dataset forming distinct clusters. Additionally, the features of visual question answering (VQA) and multiple-choice visual questioning (MCVQ) tasks separated by the text encoder of CLIP. Notably, the text features of OCRVQA distant from those of ChartQA, TextVQA, and VQA, despite all being VQA tasks. This finding supports observation in the main paper that the effectiveness of ranking methods is influenced by dataset characteristics (i.e., VQA vs. MCVQ).
D. Full Results of Correlation Study
In the following, we present the full results correlation study for ranking different series of large multimodal models. We only show NLLp, NLLp, NLL NLLmean- Ente, Entp, Entmax Entmean SampleBLEU. SampleBERT. Sample BERT. ATC and Accuracy on the line included because their performance computed by the average correlation strength across eight datasets.
16

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
2D t-SNE by image features
100
1:5NE
-50
ScienceQA
TextvQA
ChartQA
OCRVQA
-100
RealWoridQA
AI2D
DocVQA
MMMU
GQA
-150
100
50
50
100
t-SNE
2D t-SNE by text features
100
t-SNE
ScienceQA
550-
TextVQA
ChartQA
OCRVQA
RealWorldOA
A12D
DocVQA
-100
MMMU
GQA
-100
100
t-SNE 1
Figure 5. Two t-SNE plots are presented: one using image features (Top) and the other using text features (Bottom) of the datasets. We observe that the text features of OCRVQA more scattered and significantly distant from those other datasets.
17

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
ScienceQA
AI2D
Rea WorldQA
MMMU
GQA
Model performance
100
0.78
076
0.58
50
Tw
80
60
60
10
30
5
0.00
0.25
0.50
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
ChartQA
OCRVQA
TextVQA
DocVQA
Model performance
-3
70
0.78
0.74
-.84
Tw 0.41
75
0.63
To 0.74
60
50
50
50
40
25
0.4
0.5
0.5
All x-axes: NLL
Figure 6. Correlation study between NLL; and model performance. Spearman's 's correlation (p) and weighted Kendall's correlation (Tw) are metrics. Each point denotes a model. Straight lines are fit with robust lincar regression (Huber, 2011).
ScienceQA
A12D
RealOOAI
MMMU
GQA
Model performance
50
15
80
Tw
80
60
86
40
50
60
40
5
0.00
0.25
0.50
0.0
0.5
0.5
0.0
0.5
0.00
0.25
0.50
ChartQA
OCRVQA
TextVQA
DocVQA
Model performance
1100
70
0.67
60
50
50
50
40
25
20
0.5
1.0
01
0.2
0.25
0.50
0.75
0.5
1.0
All x-axes: NLL,
Figure 7. Correlation study between NLLp and model performance. Spearman's correlation (p) and weighted Kendall's correlation (Tw) are metrics. Each point denotes a model. Straight lines are fit with robust linear regression (Huber, 2011).
18

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
ScienceQA
AI2D
RealWorldQA
MMMU
GQA
Model performance
70
0.63
50
15
02
0.76
0.81
0.63
0.44
0.59
80
60
10
50
60
30
40
0.25
0.75
0.25
0.50
0.75
0.50
0.75
1.00
0.25
0.75
ChartQA
OCRVQA
TextVQA
DocVQA
100
Model performance
0.94
0.64
0.83 0.69
0.92
0.80
0.63
75
0.82
60
50
50
50
25
40
0.5
1.0
1.5
0.4
0.5
All x-axes: NLL
Figure 8. Correlation study between NLLmin and model performance. Spearman's 's correlation (p) and weighted Kendall's correlation (Tw) are metrics. Each point denotes model. Straight lines fit with robust lincar regression (Huber, 2011).
Model performance
ScienceQA
RealWoridQA
MMMU
GQA
80
0.64
0.56
50
0.50
0.67
60
10
40
40
30
5
0.0
0.2
0.1
0.2
0.3
0.2
0.4
0.2
0.4
0.2
0.4
ChartQA
OCRVQA
TextVQA
DocVQA
Model performance
70
80
75
50
60
50
40
25
40
25
20
0.25
0.50
0.75
0.1
0.2
0.2
0.4
0.6
0.5
1.0
All x-axes: NLL
Figure Correlation study between NLLmsnn and model performance. Spearman's conrllation (p) and weighted Kendall's correlation (Tw) are metrics. Each point denotes a model. Straight lines are fit with robust linear regression (Huber, 2011).
19

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Model performance
ScienceQA
AI2D
RealWoridQA
MMMU
GQA
0.59
0.51
Ty 0.39
0.82
70
0.65
0.56
50
0.64
0.52
12.5
Tx 0.20
0.54
80
40
10.0
60
50
30
7.5
60
40
20
0.05
0.15
0.15
0.10
0.15
ChartQA
OCRVQA
TextVQA
DocVQA
Model performance
0.64
0
0.69
0.71
0.84
75
05
60
0.57
75
0 56
75
0.70
50
50
50
40
25
25
0.2
0.1
0.2
0.1
0.2
0.3
0.2
0.4
All x-axes: Ent
Figure Correlation study between Enty model performance. Spearman's correlation (p) and weighted Kendall's correlation (Tw) are metrics. Each point denotes a model. Straight lines are fit with robust linear regression (Huber, 2011).
Model performance
ScienceQA
AI2D
RealWoridQA
MMMU
GQA
0.49
0.39
80
0.69
0.48
70
0.43
0.43
50
0.34
0.24
0.46
0.21
40
10
60
50
0.
60
40
30
0.00
0.05
0.10
0.1
0.0
0.1
0.2
0.0
0.1
ChartQA
OCRVQA
TextVQA
DocVQA
Model performance
0.
0.80 T. 0.64
0.70 0.34
- 0
0.64
60
75
75
0.68
50
50
25
20
25
25
0.0
0.1
0.2
0.05
0.10
0.15
0.1
0.2
0.1
0.2
All x-axes: Ent
Figure 11. Correlation study between Entp and model performance. Spearman's 's correlation (p) and weighted Kendall's correlation (Tw) are metrics. Each point denotes a model. Straight lines are fit with robust linear regression (Huber, 2011).
20

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
ScienceQA
AI2D
RealWoridQA
MMMU
GQA
Model performancc
80
50
12.5
60
40
10.0
60
40
50
30
7.5
20
5.0
0.1
0.1
0.2
0.1
0.2
0.05
0.10
0.15
ChartQA
OCRVQA
TextVQA
DocVQA
Model performance
0.76
100
75
0.66
0.52
0.60
0.73
60
50
50
50
25
40
0.
0
0.1
0.2
0.4
All x-axes: Ent
Figure Correlation study between Entmax and model performance. Spearman's correlation (p) and weighted Kendall's correlation (Tw) are metrics. Each point denotes a model. Straight lines are fit with robust linear regression (Huber, 2011).
ScienceQA
A12D
RealWoridQA
MMMU
GQA
Model performance
100
0.50
-.62
0.57
75
0.33
Tu 0.56
60
Ty. 0.50
40
0.41
10
Tu 0.26
50
50
40
5
20
0.10
0.15
ChartQA
OCRVQA
TextvQA
DocVQA
100
Model performance
75
0.91
0.68
60
f 0.62
0.77
75
0.74
0.58
0.88
0.74
50
50
20
25
0.10
0.15
All x-axes: Entmean
Figure 13. Correlation study between Entmean and model performance. Spearman's comrelation (p) and weighted Kendall's correlation (Tw) are metrics. Each point denotes a model. Straight lines are fit with robust linear regression (Huber, 2011).
21

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Model performance
ScienceQA
AI2D
RealWorldQA
MMMU
GQA
P
80
P
0.76
70
50
6.68
f+ 0.59
T+ 36
Fy 0.12
0.41
80
80
60
60
40
10.0
50
7.5
60
30
40
5.0
0.8 ChartQA
1.0
0.6
0.6
0.6
0.5
1.0
OCRVQA
TextVQA
DocVQA
70
Model performance
0.889
0.47
00881 Ty 0.60
0.90 Ty 0.63
75
0.61
60
Tu 0.58
75
100
50
50
50
50
25
40
25
0.25
0.50
0.75
0.6
0.8
0.8 All x-axes: Sample
0.25
0.50
0.5
Figure 14, Correlation study between SamplesLEU and model performance. Spearman's correlation (p) and weighted Kendall's s correlation (Tw) metries. Each point denotes model. Straight lines fit with robust linear regression (Huber, 201
ScienceQA
AI2D
Rea WorldQA
MMMU
GQA
Model performanos
100
0.46
80
75
50
0.45
0.51
80
10
40
60
8
30
40
40
0.950
0.975
1.000
0.98
1.00
0.9
1.0
0.950 0.975
0.95
1.00
ChartQA
OCRVQA
TextvQA
DocVQA
Model performance
70
100
100
60
75
50
50
25
40
0.96
0.98
0.96
0.98
0.950
0.975
0 950
0.975
All x-axes: SamplegERT
Figure 15. Correlation study between SampleBERT and model performance. Spearman's 's correlation (p) and weighted Kendall's correlation (Tw) are metrics. Each point denotes model. Straight lines are fit with robust linear regression (Huber, 2011).
22

---

Ranked from Within; Ranking Large Multimodal Models Without Labels
Model pertormang mance
ScienceQA
AI2D
WealWorIdQA
MMMU
GQA
1000
80
50
0.59
12
0.56
Gy 62
10
60
50
20
30
40
0
0.950
0.975
1.000
1.0
0.950
0.975
0.95
1.00
ChartQA
OCRVQA
TextVQA
DocVQA
Model performance
75
To 0.61
0.90
70
Ty 0.72
0.75
100
Tw 0.59
0.77
100
00.89 Tu 0 64
60
50
50
50
25
25
0.98
0.96
0.950
0.975
0.950
0.975
All x-axes: Sample
Figure 16. Correlation study between Sample BERT
BERT and model performance. Spearman's correlation (p) and weighted Kendall's s correlation (Tw) are metrics. Each point denotes model. Straight lines are with robust linear regression (Huber, 2011).
23


Reviews:
[{"review_id": "olrj1rjdVH", "reviewer": "Reviewer_HFDC", "summary": "This paper explores how to effectively evaluate the performance of LMMs without requiring task-specific labels, and systematically validates three types of model uncertainty-based approaches, including softmax probabilities, self-consistency, and labeled proxy sets. \nComprehensive experiments across various LMMs and benchmarks show that uncertainty scores derived from softmax distributions provide a robust and consistent basis for ranking models across various tasks, facilitating the ranking of LMMs on unlabeled data.\n\n### update after rebuttal\nThe author's rebuttal addressed several of my concerns. I keep my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. Authors did not propose new methods or metrics based on these findings for selecting the optimal LMM for a given task in an unlabeled scenario.\n\n**A**: We acknowledge that our work does not introduce new methods or metrics. However, it addresses a previously under-explored yet practically important problem: how to evaluate and select LMMs in unlabeled scenarios.  We systematically analyze existing uncertainty quantification techniques across diverse models and datasets, offering insights into their effectiveness and failure cases. These indicators serve as practical tools for model selection without labels, which we believe is a critical step toward reliable LMM evaluation. Our findings also highlight the promise of uncertainty-based approaches, motivating future research into new metrics for this setting.\n\n> Q2. What is the core finding of this paper?\n\n**A**: Our key findings are: (1) model performance on one dataset does not reliably predict its ranking on another; (2) uncertainty in model predictions can be predictive of model rank, **with $NLL_{max}$ being the most accurate technique in general**; and (3) text prompt similarity better correlate with model performance across datasets than image feature similarity. We have clarified this in the revised paper, making these observations more crisply than in the initial submission.\n\n> Q3. How does this finding contribute to the LMM community?\n\n**A:** Our findings highlight three key contributions to the LMM community. First, the weak correlation of model performance across datasets underscores the need for diverse multimodal benchmarks to comprehensively assess LMM capabilities. Second, we find that text prompt similarity has a stronger influence on model performance correlations across datasets than image feature similarity.  This finding emphasizes the importance of evaluating LMMs with diverse textual prompts rather than focusing solely on image variation. Third, the effectiveness of baseline uncertainty quantification methods suggests uncertainty estimation as a promising approach for unsupervised LMM ranking.\n\n> Q4. How can the readers employ this finding to evaluate LMMs without label annotations?\n\n**A:** Given a test dataset without labels and multiple LMMs to evaluate, our findings provide a practical framework for ranking and selecting the most suitable model. Specifically, there are two approaches:\n  - For **models with access to internal statistics (*e.g*., logits)**, their $NLL_{max}$ scores can be directly used to rank model performance.\n  - For **closed-source models**, where internal statistics are unavailable, $Sample_{BERT}^*$ serves as a viable alternative for effective performance ranking.", "claims_and_evidence": "Strengths: \n- This paper presents an important and interesting problem regarding how to evaluate LMMs without requiring task-specific labels. \n- The authors perform extensive experiments to analyze how well uncertainty-based metrics can measure relative model performance, resulting in some empirical findings. \n\nWeaknesses: \n- Although the authors derived a series of empirical conclusions through extensive experimental analysis, \n    - 1) they did not propose new methods or metrics based on these findings for selecting the optimal LMM for a given task in an unlabeled scenario. \n    - 2) The findings in this paper lack universality, are somewhat diffuse, and would benefit from a more concise formulation. As a reader, these findings cannot directly guide me to quickly and effectively select the optimal LMM for a given task without manual labels.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "This paper does not involve the claim and proof of novel theories.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "While the paper focuses on an intriguing and valuable problem and conducts extensive experimental analysis, its contribution and impact on the broader literature are not fully clear. Because the authors did not introduce novel methods and all implemented uncertainty-based methods are pre-existing.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Other Strengths: \n- The paper additionally provides detailed correlation analysis of model performance.", "other_comments_or_suggestions": "- making the findings more concise and instructive.", "questions_for_authors": "- What is the core finding of this paper? \n- How does this finding contribute to the LMM community? \n- How can the readers employ this finding to evaluate LMMs without label annotations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iDFWf4fq5G", "reviewer": "Reviewer_V6Xj", "summary": "The paper investigates alternative ways to rank large multimodal models on new domains in absence of ground truth annotations. The authors compare 3 types of approaches: (1) labeled proxy datasets (AoL: where the performance on N-1 datasets is used to predict the performance on the Nth dataset; ATC: where proxy datasets are used to extract a confidence threshold to be used for the new unlabeled dataset), probability-based (NLL, Entropy), and self-consistency (sample-based).  By running an extensive analysis with 45 models (30 different models and their versions) on 8 multiple-choice or open-ended visual QAs, the authors conclude that probability-based metrics tend to be more reliable and show strong correlations with actual performance, compared to AoL or ATC. An additional analysis is provided for AoL, to shed light on its low correlation with performance, showing that text similarities across datasets is more indicative of strong correlation with performance than image similarities.\n\n## update after rebuttal ##\nI read the rebuttal and the other reviews and I maintain my initial score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. These works should be discussed as well: (1) The Internal State of an LLM Knows When It’s Lying  (2) Estimating Knowledge in Large Language Models Without Generating a Single Token\n\n**A**: Thank you for your valuable suggestions. We will include these two studies, along with other relevant works on the internal states of LLMs and LMMs, in the related work section. Additionally, we will discuss the limitations of using internal states for model ranking and potential directions for future research as below:\n\n*\"Internal states of LMMs can be leveraged for uncertainty quantification or error detection by training a classifier. However, in the context of unsupervised LMM ranking, a separate classifier must be trained for each LMM, making the approach computationally expensive. One potential alternative is to measure the statistical distance between the internal state of the original answer and the distribution of internal states of multiple inference processes. A larger average distance may indicate higher uncertainty in model predictions, suggesting a lower model rank. It would be interesting to investigate the use of internal states for LMM ranking is left as future work.”*\n\n> Q2. The impact section could include a statement about the possibility of training models to elicit high confidence when producing deceiving answers.\n\n**A**: Thank you for the suggestion. We will incorporate this discussion into the broader impact section:\n\n*\"Our findings could be misused: adversarial researchers may train models that consistently get ranked higher than other models, despite performing poorly or adversarially on data. To mitigate this risk, incorporating robust calibration methods is helpful, ensuring that model confidence accurately reflects uncertainty. This would help promote safer and more responsible deployment of our findings.\"*\n\n> Q3. A few typos: L295: $Prob_{min}$, $Prob_{avg}$ are not used in the table, they should be $NLL_{max}$, $NLL_{avg}$? L247-248, some citations are incorrect and appear as “?”\n\n**A**: Thank you for your meticulous comment. The $Prob_{min}$ and $Prob_{avg}$ should be $NLL_{max}$ and $NLL_{avg}$. We will carefully review the manuscripts and correct all typos.", "claims_and_evidence": "This is a compelling submission, with potential for impact in how benchmarking of large multimodal models is approached.\nThe claims are clearly presented and backed by an extensive set of experiments.", "methods_and_evaluation_criteria": "The proposed analysis is comprehensive and robust, including multiple metrics (probabilistic, sample-based, proxy-labelled).", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The empirical analysis is extensive and covers well the space of multimodal models.", "relation_to_broader_scientific_literature": "The related work section covers relevant works, but some are missing, see below.", "essential_references_not_discussed": "These works should be discussed as well\n\nThe Internal State of an LLM Knows When It’s Lying https://aclanthology.org/2023.findings-emnlp.68.pdf \nEstimating Knowledge in Large Language Models Without Generating a Single Token https://aclanthology.org/2024.emnlp-main.232.pdf", "other_strengths_and_weaknesses": "The impact section could include a statement about the possibility of training models to elicit high confidence when producing deceiving answers.", "other_comments_or_suggestions": "The paper is well written.\nA few typos:\nL295: Prob_min, Prob_avg are not used in the table, they should be NLL_max, NLL_avg?\nL247-248, some citations are incorrect and appear as “?”", "questions_for_authors": "1. Would it be possible to train models to elicit high confidence when producing deceiving answers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U9gRWh3eWY", "reviewer": "Reviewer_isqi", "summary": "This paper presents a study on ranking methods for evaluating large language models (LLMs) without accessing labels. The authors main findings are that Accuracy-on-the-Line is unreliable for ranking LLMs in new domains, the choice of token significantly impacts output probability-based ranking, and the negative log-likelihood (NLL) is more stable and predictive for LLM ranking than other evaluation methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. In the section \"Related Work: Evaluation & Benchmarking LMMs,\" the paper mentions RealWorldQA and several other benchmarks for evaluation purposes. However, it fails to address some commonly used benchmarks for LLMs, such as CV-benchmarks (e.g., GQA, MMVP, OCRBench) and math-benchmarks (e.g., MathVerse, MathVista).\n\n**A**: Thank you for pointing out these benchmarks. We will include them in the related work. The following text will be used in Section 2:\n\n*“Several benchmarks (x.ai, 2024; Ainslie et al., 2023; Tong et al., 2024) have been developed to assess multimodal models' real-world spatial understanding. Lu et al. (2024) and Zhang et al. (2024) introduce benchmarks specifically designed to evaluate MLLMs' mathematical reasoning, focusing on their ability to comprehend and reason visual mathematical figures. Additionally, numerous benchmarks (Mishra et al., 2019; Masry et al., 2022; Mathew et al., 2021; Liu et al., 2024) assess the performance of LMMs in optical character recognition.”*\n\n *[1] x.ai. Grok 1.5v: The future of ai models, 2024. URL https://x.ai/blog/grok-1.5v.*\n\n *[2] Ainslie, J., et al. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In EMNLP, 2023.*\n\n *[3] Tong, S., et al. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In CVPR, 2024.*\n\n *[4] Lu, P., et al. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In ICLR, 2024.*\n     \n *[5] Zhang, R., et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In ECCV, 2024.*\n\n *[6] Liu et al., Ocrbench: on the hidden mystery of ocr in large multi-modal models. Science China Information Sciences, 67(12), December 2024d.*\n\n> Q2. Consider incorporating more vision and spatial understanding-related benchmarks, such as GQA and MMVP.  They should also update their model selection to include recently developed LLMs like the Qwen series and DeepSeek series to validate the universality of their proposed criteria.\n  \n**A**: Thank you for your valuable suggestion. Following your recommendation, we incorporated four new models—QwenVL, QwenVL2, DeepSeek-VL, and DeepSeek-VL2—and added GQA as an additional dataset for evaluation. We summarize the results in the table below, where the metric is rank correlation, and AoL performance represents the average correlation strength when using the other eight domains to predict rankings in the target domain.\n\n- Generalizability to new models: The uncertainty-based ranking methods (*e.g.*, $NLL_{max}$) successfully extend to newly introduced models.\n\n- Performance on new dataset GQA: We observe consistent trends: (1) AoL does not reliably rank model performance, and (2) $NLL_{max}$ provides a more stable and indicative ranking signal compared to other methods.\n\nThese new results further support the applicability and robustness of the evaluated uncertainty-based methods. We will incorporate them into the revised version.\n\n| |  AoL  | $NLL_{F}$ | $NLL_{P}$ | $NLL_{max}$ | $NLL_{mean}$ | $Ent_{F}$ | $Ent_{P}$ | $Ent_{max}$ | $Ent_{mean}$ | $Sample_{BLEU}$ | $Sample_{BERT}$ | $Sample_{BERT}^*$ |\n|--------------|:----:|:-----:|:-----:|:-------:|:--------:|:-----:|:-----:|:-------:|:--------:|:-----------:|:-----------:|:------------:|\n| **SQA-I**   | 0.54 | 0.84  | 0.66  | **0.81**    | 0.74     | 0.63  | 0.50  | 0.62    | 0.59     | 0.68        | 0.51        | 0.70         |\n| **ChartQA** | 0.68 | 0.62  | 0.79  | **0.94**    | 0.92     | 0.64  | 0.82  | 0.89    | 0.91     | 0.86        |  0.87     | 0.87         |\n| **GQA**     | 0.59 | 0.64  | 0.55  | **0.70**   | 0.63     | 0.47  | 0.46  | 0.52    | 0.52     |   0.59    |    0.61      |  0.61      |\n\n\n> Q3. Finally, the authors should elaborate on the applicability of their evaluation framework to closed-source models to ensure its relevance across different model types.\n\n**A**: We appreciate the reviewer’s insightful suggestion. While our experiments focus on models with accessible internal outputs, we note that **our evaluation framework can be extended to closed-source models as well**—particularly through methods that rely solely on final predictions, such as $Sample_{BERT}$. These approaches operate without access to logits or internal statistics and can support **label-free model selection** even in restricted-access settings.\n\nWe agree this distinction is important and will clarify in the manuscript that our framework can be **applied to closed-source models**, depending on the availability of output-level information, without requiring structural modifications or access to internals.", "claims_and_evidence": "Yes, the claims are well-supported and make sense in light of the experimental results.", "methods_and_evaluation_criteria": "Yes, this manuscript introduces a ranking method that effectively evaluates LLMs without relying on labeled data. The NLL (Negative Log-Likelihood) metric demonstrates highly stable performance indicators for target domains.", "theoretical_claims": "This work does not include proofs for its theoretical claims. However, based on the results provided by the authors, the findings appear to be reasonable.", "experimental_designs_or_analyses": "Yes, the experimental designs and analyses appear to be sound based on the reported correlation studies.", "relation_to_broader_scientific_literature": "The paper points out the unreliability of evaluating and ranking LLMs across different datasets, and provides a method for assessing the ranking of LMMs in the absence of labeled data. This method yields more stable evaluation results across different domains compared to previous approaches. It holds significant importance for further research into the performance of LLMs and issues related to their deployment.", "essential_references_not_discussed": "In the section \"Related Work: Evaluation & Benchmarking LMMs,\" the paper mentions RealWorldQA and several other benchmarks for evaluation purposes. However, it fails to address some commonly used benchmarks for LLMs, such as CV-benchmarks (e.g., GQA, MMVP, OCRBench) and math-benchmarks (e.g., MathVerse, MathVista).", "other_strengths_and_weaknesses": "Strengths:\nThe authors have investigated the important issue of evaluating models using unlabeled data and proposed a corresponding ranking evaluation method. The results make sense and have yielded many findings beneficial to the development of the community. The authors evaluated the performance on 45 LLMs, demonstrating good breadth.\nWeaknesses:\nHowever, \n1.the benchmarks used by the authors for evaluation seem limited and unbalanced. The paper employs four types of benchmarks: Optical Character Recognition, Science, Vision, and General. But the authors appear to focus mainly on document OCR-related benchmarks (such as TextVQA, ChartVQA, DocVQA, OCRVQA), with very little attention to vision-related benchmarks (only one RWQA). There seems to be a lack of evaluation in areas that are also very important for LLM assessment, such as the evaluation of LLMs' spatial understanding capabilities. \n2.In addition, the evaluation models used by the authors also seem limited, involving only some LLaVA series models or relatively older models like InstructBLIP and mPLUG-Owl2. There is a lack of assessment of newly developed models, such as the Qwen series and DeepSeek series. It needs further examination whether these models still follow the evaluation criteria proposed by the authors. The authors should also clarify whether the evaluation of closed-source models follows the proposed pattern.", "other_comments_or_suggestions": "To enhance the comprehensiveness of the evaluation, the authors should \n1.consider incorporating more vision and spatial understanding-related benchmarks, such as GQA and MMVP. \n2.They should also update their model selection to include recently developed LLMs like the Qwen series and DeepSeek series to validate the universality of their proposed criteria. \n3.Finally, the authors should elaborate on the applicability of their evaluation framework to closed-source models to ensure its relevance across different model types.", "questions_for_authors": "Please see weakness and suggestions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7FHp8PAgQH", "reviewer": "Reviewer_Fjnt", "summary": "This paper addresses the challenge of ranking large multimodal models (LMMs) without access to labeled data. Specifically, the authors propose uncertainty-based ranking methods that utilize softmax probabilities, self-consistency, and labeled proxy sets to estimate model performance. They evaluate 45 LMMs on eight benchmarks and find that uncertainty-based metrics, particularly negative log-likelihood (NLL), provide a robust ranking mechanism. The study suggests that relying solely on benchmark performance for ranking models may be unreliable due to domain shifts.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1. The novelty of this paper seems limited. The authors only test many existing ranking methods (NLL loss, Entropy, BLEU, and BERTScores) without providing new insights.\n\n**A**: While NLL, Entropy, BLEU, and BERTScore have been widely used for uncertainty quantification in LLMs, to the best of our knowledge they have not been explored for unsupervised LMM ranking. Our work is the first to systematically investigate their applicability in this context. Through extensive experiments across diverse models and benchmarks, we demonstrate that $NLL_{max}$ is the most reliable and accurate ranking proxy measure. Additionally, we introduce a simple yet effective modification (*i.e.*, $Sample_{BERT}^*$) to $Sample_{BERT}$, improving its consistency. Furthermore, we observe that self-consistency methods perform better for LMM ranking in VQA than MCQA, and that $Sample_{BERT}^*$ outperforms BLEU.\n\nIn the revised version, we will highlight our core findings: (1) model performance on one dataset does not reliably reflect its ranking on another; (2) uncertainty in model predictions can be predictive of model rank, **with $NLL_{max}$ being the most accurate technique in general**; and (3) text prompt similarity better correlate with model performance across datasets than image feature similarity.\n\n> Q2. They do not provide clear guidance on selecting appropriate ranking methods for different scenarios.\n\n**A**: We agree this would strengthen the paper. In the revision, we provide guidance on selecting appropriate ranking methods based on model accessibility and computational constraints, as follows.\n  - $NLL_{max}$ is generally the most accurate, efficient and consistent method for LMM performance ranking. \n  - While self-consistency methods are also competitive for VQA, they require K-times the compute, where K is the number of unique inference outputs per prompt. However, for closed-source models where internal statistics (*e.g*., logits) are unavailable, self-consistency-based methods are more suitable. Among these, $Sample_{BERT}^*$ is the recommended approach.\n\n> Q3. In Models of Section 5.1, the authors claim to provide two settings. However, they only explain one, while the other setting is missing.\n\n**A**: Thank you for picking up this error - it is a typo. Section 5 only contains one setting, where different series of models are evaluated. We have corrected this in the revision.\n\n> Q4. In Section 6, the authors only evaluate the series of LLaVA models, raising concerns about the generality of the analysis.\n\n**A**: Section 6 includes three distinct analyses. The first specifically investigates whether the considered methods are effective for ranking models within the same series. We use LLaVA models as a case study because they are widely adopted and representative of LMM architectures. Furthermore, the diversity within the LLaVA series is ensured by incorporating LLaVA-prismatic models trained with varying visual encoders and large language models, making the analysis more comprehensive. We will clarify this in the paper.\n\nThe other two analyses in Section 6 consider models from different series, consistent with the broader evaluation in Section 5. We have rewritten this section to make this clearer.\n\n> Q5. In Section 5.1, some citations appear as question marks, indicating missing or improperly formatted references.\n\n**A**: Thank you for your meticulous comment. we will double-check and correct the improperly formatted references.", "claims_and_evidence": "Yes. The study tests 45 LMMs across eight diverse benchmarks, demonstrating strong empirical rigour.", "methods_and_evaluation_criteria": "The analysis encompasses multiple ranking methods, ablation studies, and cross-domain correlation assessments, ensuring a comprehensive evaluation. However, the authors do not provide clear guidance on selecting appropriate ranking methods for different scenarios.", "theoretical_claims": "N/A. This paper does not include any theoretical claims.", "experimental_designs_or_analyses": "Yes. The analysis encompasses multiple ranking methods, ablation studies, and cross-domain correlation assessments, ensuring a comprehensive evaluation.", "relation_to_broader_scientific_literature": "This paper estimates model performance using ranking methods, such as NLL loss, Entropy, BLEU, and BERTScores.", "essential_references_not_discussed": "This paper contains enough references.", "other_strengths_and_weaknesses": "Weaknesses:\n1. Although the authors conduct many experiments testing the model performance across many ranking methods(NLL loss, Entropy, BLEU, and BERTScores), they do not provide clear guidance on selecting appropriate ranking methods for different scenarios.\n2. In Models of Section 5.1, the authors claim to provide two settings. However, they only explain one, while the other setting is missing.\n3. The novelty of this paper seems limited. The authors only test many existing ranking methods (NLL loss, Entropy, BLEU, and BERTScores) without providing new insights.\n4. In Section 6, the authors only evaluate the series of LLaVA models, raising concerns about the generality of the analysis.\n5. In Section 5.1, some citations appear as question marks, indicating missing or improperly formatted references.", "other_comments_or_suggestions": "Please refer to the weaknesses.", "questions_for_authors": "Please refer to the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Other expertise"]}}]

Author rebuttal:

Meta-review: ## Summary
This paper explores how to effectively evaluate the performance of LMMs without requiring task-specific labels and systematically validates three types of model uncertainty-based approaches, including softmax probabilities, self-consistency, and labeled proxy sets. Comprehensive experiments across various LMMs and benchmarks show that uncertainty scores derived from softmax distributions provide a robust and consistent basis for ranking models across various tasks, facilitating the ranking of LMMs on unlabeled data.

## Reviewer Consensus and Review Quality
The submission received four reviews. Reviewer HFDC and V6Xj provided positive rating, highlighting the good novelty and experiments. Reviewer isqi and Fjnt expressed concerns about the experiment design. In the rebuttal, the authors partially address these concerns by **adding the experiments of QwenVL, DeepSeek-VL, and so on**.

## Strengths
- This paper presents an important and interesting problem regarding how to evaluate LMMs without requiring task-specific labels.
- The authors perform extensive experiments to analyze how well uncertainty-based metrics can measure relative model performance, resulting in some empirical findings.

## Weaknesses
- They did not propose new methods or metrics based on these findings for selecting the optimal LMM for a given task in an unlabeled scenario.
- The findings in this paper lack universality, are somewhat diffuse, and would benefit from a more concise formulation.

## Justification
Based on the reviewers’ comments and the author’s response, I believe this paper has potential but needs more work. While the paper did not present a concise suggestion to select the optimal LLM, the strengths of the paper outweigh its weaknesses. In addition, it ranks 2 among 12 papers. Therefore, I recommend **weak accept**.
Decision: Accept (poster)