K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge
Authors: Zhikai Li
jiatong li
Xuewen Liu
Wangbo Zhao
Pan Du
Kaicheng Zhou
Qingyi Gu
Yang You
Zhen Dong
Kurt Keutzer
Venue: ICLR 2026 Poster (2026)
Keywords: Visual generation evaluation
preference alignment
corrected VLM-as-a-Judge
efficient evaluation
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: The rapid development of visual generative models raises the need for more scalable and human-aligned evaluation methods. While the crowdsourced Arena platforms offer human preference assessments by collecting human votes, they are costly and time-consuming, inherently limiting their scalability. Leveraging vision-language model (VLMs) as substitutes for manual judgments presents a promising solution. However, the inherent hallucinations and biases of VLMs hinder alignment with human preferences, thus compromising evaluation reliability. Additionally, the static evaluation approach lead to low efficiency. In this paper, we propose K-Sort Eval, a reliable and efficient VLM-based evaluation framework that integrates posterior correction and dynamic matching. Specifically, we curate a high-quality dataset from thousands of human votes in K-Sort Arena, with each instance containing the outputs and rankings of K models. When evaluating a new model, it undergoes (K+1)-wise free-for-all comparisons with existing models, and the VLM provide the rankings. To enhance alignment and reliability, we propose a posterior correction method, which adaptively corrects the posterior probability in Bayesian updating based on the consistency between the VLM prediction and human supervision. Moreover, we propose a dynamic matching strategy, which balances uncertainty and diversity to maximize the expected benefit of each comparison, thus ensuring more efficient evaluation. Extensive experiments show that K-Sort Eval delivers evaluation results consistent with K-Sort Arena, typically requiring fewer than 90 model runs, demonstrating both its efficiency and reliability. The dataset and code are publicly available.

Published as a conference paper at ICLR 2026
K-SORT EVAL: EFFICIENT PREFERENCE EVALUATION FOR VISUAL GENERATION VIA CORRECTED VLM-AS- A A-JUDGE
Zhikai Li Jiatong Li Xuewen Liu', Wangbo Zhao', Pan Du', Kaicheng Zhou', Qingyi Gu!-t, Yang You--1, Zhen Dong Kurt K
Institute of Automation, Chinese Academy of Sciences
Universityy of California, Berkeley 3 University of California, Santa Barbara
Nattooal University of Singapore Nannynn Technological University "Collov Labs
Dataset: https: httpa::/huugiingfaee http:://hcggggccccccc/caaaaa/a// http:///ee...aaaatee//e//////e/
Code: https: : //gi thub. 2ttom/kkk1ii/ -Srrt-vvva
ABSTRACT
The rapid development of visual generative models raises the need for more scal- able and human-aligned evaluation methods. While the crowdsourced Arena platforms offer human preference assessments by collecting human votes, they are costly and time-consuming, inherently limiting their scalability. Leveraging vision-language model (VLMs) as substitutes for manual judgments presents a promising solution. However, the inherent hallucinations and biases of VLMs hin- der alignment with human preferences. thus compromising evaluation reliability. Additionally, the static evaluation approach lead to low efficiency. In this paper, we propose K-Sort Eval, a reliable and efficient VI.M-based evaluation frame- work that integrates posterior correction and dynamic matching. Specifically, we curate a high-quality dataset from thousands of human votes in K-Sort Arena, with each instance containing the outputs and rankings of K models. When evaluat- ing a new model, it undergoes (K+1)-wise free-for-all comparisons with existing models, and the VLM provide the rankings. To enhance alignment and reliability, we propose a posterior correction method, which adaptively corrects the posterior probability in Bayesian updating based the consistency between the VLM pre- diction and human supervision. Moreover, we propose a dynamic matching strat- egy, which balances uncertainty and diversity to maximize the expected benefit of each comparison, thus ensuring more efficient evaluation. Extensive experiments show that K-Sort Eval delivers evaluation results consistent with K-Sort Arena, typically requiring fewer than 90 model runs, demonstrating both its efficiency and reliability. The dataset and code are publicly available.
INTRODUCTION
Visual generative models have achieved remarkable progress, enabling high-quality outputs in tasks such as text-to-image (Betker et al., 2023; Podell et al., 2023; Rombach et al., 2022) and text-to- video (Esser et al., 2023; He al., 2022: Zhou et al., 2022) generation. This rapid advancement has fueled growing interest in the field, driving the continuous emergence of new models (Zhao et al., 2025a;b; 2026) and applications (Du et al.; Li al., 2023h; 2025a). However, the evaluation methods fail to keep pace with the model development, struggling to offer a fair and comprehensive assessment of generated outputs. Traditional metrics such as IS (Salimans et al., 2016), FID (Heusel et al., 2017), and FVD (Unterthiner et al., 2018), while widely used, are criticized for their inability to capture human preference judgements in the real world. In response, several efforts attempt to construct static datasets for human preference evaluation (Kirstain et al., 2023; Wu al., 2023; Xu et al., 2023). However, these datasets are inherently limited by their closed-ended nature, lack of user interaction, and inability to stay up-to-date (Li et al., 2025b; Chiang et al., 2024).

---

Published as a conference paper at ICLR 2026
Dataset Curation
K-Sort Eval: Preference Evaluation Corrected VLM-as-e-Judge
Crowdsourced voting
Step 1: Dynamic matching
Step 3: Bayesian updating with correction
& & &
New model
Uncertainty
Bayesian updating
A>CaBDD
P(AD) xr M(D))))))) =p
n kMM at Var MD
9~ ((p-0")))
Diversity
sin (Pit, 1P0
VLM judgement
Posterior correction
supervision
8
Consistency filtering
AxCSi>o
Step 2: VLM judgement
Observation noise
Swapping operation
Rule augmentation
P(D()) APIDW) M)P
&&&& & Shuffled
&
A
Rank Images/videos by given criteria
Correction
Arena leaderboard
de =
A/W
&f A + +1 X7
Figure I: Overview of the proposed K-Sort Eval. First, a high-quality dataset is curated through consistency filtering. When evaluating a new model, we begin with dynamic matching to select the most informative instance. Then, two prompt strategies employed to effectively guide the VLM and mitigate hallucinations. Finally, Bayesian updating with correction is performed. where the discrepancy between VLM prediction and dataset supervision is treated as observation noise to correct the posterior estimation of model capability.
In contrast, the Arena method, which is an open and live benchmark platform, is a more effective approach for human preference evaluation. It captures real human feedback by collecting crowd- sourced manual voting on model comparisons, allowing for a better reflection of real-world prefer- ences. This approach is initially used for evaluating large language models (LLMs) (Chiang et al., 2024) and is later extended to apply to visual generative models (Jiang et al., 2024; Li et al., 2025b) and multi-modaa models (Lu et al., 2024; Chou et al., 2024). Notably, for visual generative models, K-Sort Arena (Li et al., 2025b) significantly improves the efficiency and reliability of Arena eval- uation by incorporating an optimized model matching strategy. Nevertheless, due to its heavy reliance on human voting, it still faces significant cost and time-consuming challenges. The excessive human involvement can also lead to potential leaderboard illusion issues (Singh et al., 2025), and the potential delays in crowdsourcing may hinder the timely evaluation of new models, thus limiting its scalability.
improved comparison mode, probabilistic modeling and updating, and
Therefore, leveraging powerful vision-language models (VLMs) to replace manual judgements, known as VLM-as-a-Judge (Chen et al., 2024; Liu & Zhang, 2025), presents a promising solu- tion. For instance, T2I-CompBench (Huang et al., 2023) investigates the potential of VLMs for compositionality evaluation of text-to-image models, and VIEScore (Ku et al., 2023) demonstrates that VLMs can provide results that have certain relevance to human evaluations. However, VLMs are inherently prone hallucinations, inconsistencies, and biases, raising concerns about their reli- ability trustworthy substitute for human judgements (Li et al., 2024b; Gu et al., 2024). While certain techniques from LLM-aas-aJuuge,, such as swapping operation (Zheng et al., 2023), rule augmentation (Bai et al., 2022), and multi-agent collaboration (Li et al., 2023a) strategies, can be adapted, they fail to addressing these inherent issues, thus still hindering reliable alignment with hu- man preferences. In addition, existing methods follow the static evaluation style, which necessitates processing the entire large-scale dataset, making the efficiency fall short of expectations.
In this paper, we propose K-Sort Eval, built top of K-Sort Arena (Li et al., 2025b), to enable efficient and reliable visual preference evaluation via VLM-as-a-Judge. Specifically, we curate a high-quality dataset from thousands of human votes in K-Sort Arena, where each instance consists of the outputs of K models along with their rankings. First, we use Spearman's 's rank correlation coefficient (Spearman. 1961) align local rankings within each instance with the overall leader- board, filtering out contaminated votes that significantly deviate from typical preference patterns. Then, Llama Guard (Inan et al., 2023) applied to screen out potentially harmful or offensive user prompts, ultimately resulting in a widely applicable and representative dataset. With the above dataset, the model to be evaluated can form a (K+l)-wise free-for-all comparisons with the K models in each instance. Notably, we propose optimized algorithms to ensure the reliability and efficiency of the evaluation. I For reliability, we propose a posterior correction method, which utilizes the human preference outcomes in the dataset supervision to correct the update of model capability. Following K-Sort Arena, we employ probabilistic modeling to represent model capa- bilities, and use VLM judgements as observations to update the posterior probability via Bayesian
2

---

Published as a conference paper at ICLR 2026
Table I: Comparison with existing evaluation methods. The proposed K-Sort Eval demonstrates
advantages in terms of scalability, alignment, efficiency, and generalizability.
Pipeline (Scalabilitt)
Judgemeut (Aligument)
Data Selection
Target Model
Evaluation Method
(Efliciency)
(Generalizabilit)
K-Sort Arena (Li et at.,, 2025b)
Manual
Human Predelined metric
No dataset
Image & Video
TIFA (Hu et al., 2023), 121-CompBench (Huang et al., 2023)
Automatic
Statie
Imuge
VBench (Iuung el al., 2024), EvalCrafter (Liu 2024)
Automatic
Predefined metric
Static
Videa
ImageReward (Xu et al., 2023), HPD (Wa et al., 2023)
Automaric
Reward model
Static
Image
GenAl-Bench (Li al, 2024s)
Automatic
Reward model
Statie
Image & Video
VIFScore (Ku al., 2023), MiniGPT4 CoT (Huang 2023)
Automaric
VIM judge
Static
Image
VidcoPhy (Bansal et al., 2024), VideoScore (He et al,, 2024)
Automatic
VLM judge
Static
Video
K-Sort Eval (Ours)
Automatic
Corrected VLM judge
Dynamic
Image Video
inference. Here, we treat the misalignment between VLM results and supervision as observation noise, and thus derive an adaptive correction policy for the posterior probability. @ For efficiency, we propose dynamic matching strategy, which leverages both uncertainty and diversity to promote the comparison of maximum expected gains. It avoids worthless comparisons, enabling the evalua- tion to be accomplished using only a subset of the dataset, without traversing the entire dataset. The overview of K-Sort Eval is illustrated in Figure 1.
Table compares K-Sort Eval with existing evaluation methods across various categories, high- lighting its advantages in scalability, alignment, efficiency, and generalizability. Furthermore, conduct extensive experiments to validate the effectiveness of K-Sort Eval, and the results show that it achieves results consistent with K-Sort Arena, while requiring fewer than 90 model runs in most cases, demonstrating its strong potential for reliable preference evaluation of generative models.
2 RELATED WORK
Visual Generation Evaluation. Traditional metrics assess the quality of generated content by mea- suring its divergence from real data, with FID (Heusel et al., 2017) and IS (Salimans et al., 2016) commonly used for images, and FVD (Unterthiner et al., 2018) for videos. To enable more com- prehensive evaluations, various benchmarks have been proposed, including image benchmarks such as TIFA (Hu et al., 2023) and T21-CompBende (Huang et al., 2023), as well as video benchmarks like VBench (Huang et al.. 2024) and EvalCrafter (Liu et al., 2024). However, these benchmarks still rely on predefined metrics, which typically fail to reflect human preferences. Several efforts focus on developing reward models, such as ImageReward (Xu et al., 2023), HPD (Wu al., 2023), Pick-a-Pic (Kirstain et al., 2023), and GenAl-Bence (Li et al., 2024a), which finetune the CLIP model (Radford et al., 2021) to achieve better alignment. However, CLIP's limited ability to capture high-level semantics continues to hinder alignment and fairness in evaluation.
Arena Evaluation with Human Preferences. To enable evaluations that better align with human preferences, Chatbot Arena (Chiang et al., 2024) builds platform for anonymized pairwise com- parisons of language models, and collects user judgements on the outputs to obtain an overall model ranking. This approach also inspires efforts in other domains, such WildVision (Lu et al., 2024) for multi-modal models and GenAl Arena (Jiang et al., 2024) for visual generative models. Further- more, K-Sort Arena (Li al., 2025b) introduces K-wise comparisons (K >2), leveraging proba- bilistic modeling and matching strategies to enable more efficient and reliable evaluation of visual generative models. Despite their success, these methods are resource-intensive and time-consuming. leading to evaluation delays and potential issues such as leaderboard overfitting or illusion (Singh al., 2025), which inherently limit their scalability.
Large Model as a Judge. In addition to generation, the judgement capabilities of LLMs, called auude, have also been explored for scoring and ranking tasks (Li et al., 2024b). To address the hallucinations and biases issues, various strategies have been developed, including swapping operations (Zheng et al., 2023), rule augmentation (Bai et al., 2022), multi-agent collaboration (Li et al., 2023a), demonstrations (Jain et al., 2023), and multi-turn interactions (Bai et al., 2023b). Likewise, leveraging VLMs as judge models to harness their visual understanding capabilities has shown great promise (Chen et al., 2024; Liu & Zhang, 2025). VLMs have been employed to evaluate the quality of generated images (Ku et al., 2023; Huang et al., 2023) and videos (Bansal et al., 2024; He et al., 2024). However, VLMSs still exhibit inherent hallucinations and biases, which limit their
3

---

Published as a conference paper at ICLR 2026
ability to make judgments fully aligned with human preferences. Thus, how to utilize VLMs for reliable human preference evaluations remains an open issue.
METHODOLOGY
In this paper, propose K-Sort Eval, an efficient VLM-as-a--Jude evaluation framework that liably aligns human preferences. K-Sort Eval benefits from both new datasets and novel evaluation strategies. The dataset curation is presented in Section 3.1, followed by the proposed methods for improving evaluation reliability and efficiency in Sections 3.2 and 3.3, respectively, with the overall evaluation pipeline ultimately formed in Section 3.4.
3.1 HUMAN PREFERENCE DATASET CURATION
K-Sort Arena (Li et al., 2025b), as a precursor to this work, serves as the platform for collecting data on human preferences. K-Sort Arena organizes free-for-all comparisons among K visual generative models, including text-to-image models and text-to-video models. Here, K >> 2 and is set to 4 in practice. Leveraging the intuitive nature of visual perception, users can confidently vote to rank the outputs based on their preferences. Each data instance D = (Pi, O+ R²), which consists of one prompt f along with the outputs O' {0*) of the K models M {Mj] and the user-voted rankings R {R ) becomes preliminary candidate for the dataset D... Here,
i == 1,2, ,N,, and Ne is the number of candidate instances.
K-Sort Arena makes efforts in terms of data diversity and voting consistency, thus providing fun- damental assurance of data quality. K-Sort Arena supports input prompts sampled from existing datasets as well as fresh prompts customized by users, which facilitates prompts from diverse do- mains and varying complexity levels. To ensure the voting quality, all crowdsourced participants are professors and graduate students specializing in visual generation. They all complete pre-voting training, particularly on the evaluation criteria, which is detailed in Appendix A. Additionally, as an open-source project, K-Sort Arena actively encourages contributions from the public community, with the criteria serving guiding reference for their voting well.
To date, K-Sort Arena has collected thousands of votes from both crowdsourced participants and the public community. For text-to-image generation, we have gathered over 1,800 human votes across 35 models, resulting in more than 10,800 pairwise comparisons. For text-to-vides generation, we have collected more than 700 human votes across 14 models, which are equivalent to more than 4,200 pairwise comparisons. However, despite training and provided guidelines, inherent subjective differences among individuals can lead inconsistencies in voting, with some votes deviating from typical preference patterns. In some cases, unintended operational errors may further introduce inaccuracies, posing the risk of preference data contamination.
To this end, we apply a careful filtering for each instance to ensure representative dataset. Due to probabilistic modeling and Bayesian updating, the leaderboard constructed by K-Sort Arena demon- strates strong robustness to preference noise, i.e., the leaderboard is sufficiently reliable. Thus, we use the consistency between the local ranking R within each instance and the overall ranking R(L) in the leaderboard as the filtering criterion. Specifically, we quantify this consistency by calculating Spearman's 's rank correlation coefficient as follows:
Σ
K
R
Rd
(R(
R((L)
Pi =
(1)
K 1k=1
R
Ri
2
E (1E( -R(L))
where Rj denotes the local ranking assigned to model Mi, and R denotes the corresponding ranking in the global leaderboard R(4) of the same model M in R'. Rt and R(L) are their respec- tive mean rankings. With the coefficient p. the filtered dataset obtained follows:
= (D' p. > T, D i 1,2, ........... NN> ...
(2)
where T is the filtering threshold. Threshold selection is presented in Appendix B. Furthermore, we apply Llama Guard (Inan al., 2023) to identify and filter out user prompts that are poten- tially harmful or offensive, which ensures the exclusion of inappropriate content, contributing to the creation of a dataset that is broadly applicable and ethically sound.
4

---

Published as a conference paper at ICLR 2026
Following the curation and filtering pro- cesses outlined above, the K-Sort Eval dataset is ultimately established. The dataset description, including size and for- mar, is presented in Table 2.
Table Description of the curated dataset.
Model
#Instance
#Visual Data
Visual Format
Annotation
Text to Image
500
2,000
512x512
[1.2,3,4]
Text-to- Video
300
1,200
512x512, 8 FPS, 58 [1,2,3.1
3.2 POSTERIOR CORRECTION FOR EVALUATION RELJARILITY
In order to align with K-Sort Arena (Li et al., 2025b), we follow the probabilistic modeling approach for model capability 0 as follows:
0 ~ N(,
where μ and a are the model's expected capability and uncertainty, respectively, and (/))) denotes the normal distribution. In each round of voting, the probability density of the model's current capa- bility P(0) taken as the prior probability, while the voting result P(D serves as the likelihood function for the observation D conditioned 0. The posterior distribution of the capability P(0ID) is then computed using Bayes' theorem as follows:
P(D)O)P(O) S_ P (DI#') P (0)) de
P(DIO)P(0)
D)
C
With the posterior probability, the posterior mean and variance of the model capability are updated as follows:
E SC 0 D)
(5)
σ² Var = S P(
The derivation and results of the above equations are detailed in Appendix C.
Posterior Correction. The above is the updating process under the ideal condition of unbiased human preferences. However, when employing a VLM-as-a-Judge, the results inherently contain hallucinations and biases, which cannot ensure alignment with true human preferences. We define the misalignment between VLM predictions and human preferences as observation noise, and ac- cordingly model the conditional distribution of the observation mixture distribution, resulting in the following noise-aware likelihood function:
P(DI0) = XP(D (1-A)
(6)
where (D) is the noise distribution of observation D. and À E [0, 1] is the confidence coefficient of the observation, with X = indicating perfect reliability and no noise.
Assumption Assume that (D)D). reprexenting a non-informative noise distribution over the observation D, is statistically independent of the parameter 0.
Lemma 1. Under Assumption 1, when the observation is subject to contamination by the noise distribution Pa (D), the resulting posterior distribution P(0) D) can be represented as mixture of the noise-free posterior distribution and the prior distribution. Specifically, it holds that:
P(0 D) XP(0)D) (1-X))(0))
(7)
where X' E [0, reflects the relative credibility of the posterior distribution induced by the obser- vation with respect to the prior.
Proof. According to Bayes' theorem, when the likelihood function is computed in Eq. 6, the
posterior probability is given by:
P(D)))P(()
+ +1 (- + (1-))) (0') do'
P(OD) D) 1 P(D)O)) P (0') do'
(8)
Based on the additivity of integration, the expression in the denominator can be split into two sep- arate terms, f XP(D)0') (0') d0' and (1 -1 (D)) d0') Base on the homogencity, the first term can be simplified as:
1_ XP(D)O')P (0') do = = P(DIO') AC
5

---

Published as a conference paper at ICLR 2026
where C is the normalizing constant in Eq. 4. For the second item, with Assumption Pa(D) is independent of f, we have:
since
- (1- ()P do' (1 (I P
(1--)))) (10)
Substituting the simplified terms into the expression for the posterior, we obtain:
D) (P(D)0) + (1 (D))())
AC X)P,(D)
(11)
AC
P(D())(())
-1- ()),(
XC (1-)),,,,, P(O)
AC + -1-((1)))
C
When the actual observation is D", we define X' (1- This completes the proof of Lemma 1.
According to Lemma 1, the posterior under noise can be viewed as a weighted combination between the noise-free posterior and the prior. To derive the weighting factor, we treat human preferences R' in the dataset as supervision, and quantify the noise level in the VLM outputs by computing Spearman's rank correlation coefficient p' as follows:
Σx (RVLLM RIIVLM)
[ - R8.
(12)
(RALLM) PRIVMO) 2 E
(Rİ. R2:
where R (VLM) denotes the VLM's ranking result of the same model Mj in R Here, the range of p' is [-1,1]. To further normalize it and constrain the values to [0,1] as in Eq. 7, we apply the sigmoid
function as follows:
X Sigmoid 1 1ee e
(13)
where x is the coefficient that controls the slope.
Given X, we proceed to derive the posterior mean and variance under the presence of noise. In Eq. 7, the weighted posterior D) follows normal distribution A X²22), and the weighted prior (1 - X(0) follows normal distribution NN((1 X)μ, (1 X))o²)) Since P(O)D) and P(0) are independently distributed, the additive property of normal distributions applies. Therefore, their weighted sum P(O) D) also follows a normal distribution, with its mean and variance given by:
A = f + X)μ {) a² = x² + X
(14)
where tee and a² are the corrected posterior mean and variance, respectively.
3.3 DYNAMIC MATCHING FOR EVALUATION EFFICIENCY
Modern datasets tend to establish their authority through increasingly large scales. However, tra- ditional evaluation methods predominantly rely on static evaluation, which requires exhaustively traversing all instances in the dataset, regardless of the model characteristics the task complexity. This uniform strategy typically incurs numerous low-gain and unnecessary processes, potentially leading to redundant computation and inefficient evaluation (Kossen et al., 2021; Polo et al., 2024).
Therefore, we are motivated to adaptively select representative subset based model-specific traits, enabling more efficient evaluation process. Thanks the prohabilistic modeling of model capabilities, the evaluation process is equipped with clear stopping criterion, i.c.. the capability uncertainty reaches the predefined threshold. To this end, we propose a dynamic matching strategy, aiming dynamically select the dataset instance that is expected to make the largest reduction in the current uncertainty. Specifically, we introduce an uncertainty criterion and a diversity criterion to jointly guide the selection process, thereby maximizing the benefit of each comparison.
Uncertainty Criterion. Traditional approaches, such as exhaustive or random matching, can lead uninformative comparisons. For instance, when the current model has high confidence in

---

Published as a conference paper at ICLR 2026
achieving strong capability score, it may still be matched against significantly weaker model. Such comparisons yield limited gains for updating the model capability. Therefore, our goal is to promote matchups between models of comparable strength. In this way, the model maintains approximately a 50% win rate, indicating maximum uncertainty in the comparison outcome. Based on this insight, we define the uncertainty criterion Unn as follows:
K
U unc
Σ
u
(15)
K
where z is the current capability mean of the new model being evaluated, and nt is the capability mean of the k-th model in the i-th dataset instance.
Diversity Criterion. In the proposed dataset, there K models in each instance, making the evaluation of new model essentially a one-to-many matching process. This requires not only considering the relationship between the new model and each model within the instance, but also accounting for the interrelations among the K models themselves. To this end, we aim to ensure that the group covers as diverse a set of opponents as possible, thereby avoiding homogeneous matchups and reducing information redundancy. Here, we quantify the diversity among models within an instance by measuring the degree of overlap between their Gaussian-modest capability distributions, and the diversity criterion Uaw is defined follows:
U
Σ
min ((((),),
(16)
IskicksSK
where P((() is the probability density of the k-th model's capability in the i-th dataset instance. Based on the above two criterions, we can dynamically match the next dataset instance by maximiz- ing the expected gain with respect the current model status follows:
arg max (U + aUjiv)
(17)
where a is a balancing coefficient.
3.4 OVERALL PIPELINE OF K-SORT EVAL
In this section, present the overall pipeline of K-Sort Eval in evaluating model. Specifi- cally, we first initialize its capability (14, a) and then put it into the following procedures:
We select a dataset instance using Eq. 17, and form group of size K by combining the new model with those in the selected instance.
Dynamic Matching:
VLM Judgement: To mitigate hallucinations of the VLM, adopt two prompt design strategies: swapping operation and rule augmentation. Specifically, we first randomly shuffle the K+1 models eliminate potential positional biases. Then, following the voting criteria in K-Sort Arena (Li al., 2025b), we provide the VLM with identical judgement instructions, as presented in Appendix D.
Updating with Correction: We compute the posterior mean and variance under noise-free assumption via Eq. 5, and subsequently correct the results using Eq. 14.
The above procedures are iteratively executed until the value of a falls below predefined stop- ping criterion. Finally, the model capability is estimated using the conservative score (Phillips & Edwards, 1966) defined as i - 1j0, where 1) is a coefficient typically set 3.0.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
K-Sort Eval enables automated preference evaluation of new models. For the VLM selection, GPT- 4o (OpenAI, 2024) is used as the judge for images, while Owen-VL-Man (Bai et al., 2023a) used for videos!. ! It quantifies model capability using both absolute and relative metrics: the absolute met- ric is the conservative score, while the relative metric the model's ranking K-Sort Arena. The
'GPT-40 API does natively support video input.

---

Published as a conference paper at ICLR 2026
Table 3: Validation of evaluation reliability of K-Sort Eval for text-to-image/video models. The
model scores and rankings produced by K-Sort Eval are highly consistent with K-Sort Arena.
FLUX.1-dev
Midjourney-15.0
Realvisal-33.0
Dalle-2
Text-lo-Image
Rank
Rank
Score
Score
K-Sort Arema
5
28.83
27.44
23,93
24
21.74
29
20.10
K-Sort Eval (Ours)
28.86
27.50
21.79
29
20.03
Runway-Gen
CogVideoX-Sb
KLing-v1.0
Pika-v10
VideoCrafter2
Text-to-Vidie
Rank
Rank
Score
Score
K-Sort Arena
33.93
33.60
32,80
29.17
12
23.65
K-Sort Eval (Ours)
2
33.98
33.63
32.90
7
29.10
12
23.72
Model Capability
50
K-Sort Eval (Durs)
K-Sort Arena
Model Capability
g
K-Sort Arena
K-Sort Evel (Ours)
Model Capability
50
K-Sort Eval (Ours)
K-Sort Arena
40-
w/o Correction
40
w/o Correction
40-
wio Correction
50-
30-
20
100
10
10
Number Runs
Number
Number
(a) FLUX.I-dev
(b) Cog VidcoX-5b
(c) KLing-v1.0
Figure 2: Visualization of the evaluation processes. With posterior correction, K-Sort Eval achieves a smoother trajectory and produces more accurate results that are consistent with K-Sort Arena.
referenced Arena leaderboards are the version updated on Sep 15, 2025, as presented in Appendix E. In addition to reliability, K-Sort Eval also offers a notable efficiency advantage, as measured by the number of model runs required for evaluation, which equals the number of VLM calls. For dataset curation, we set the filtering threshold to 0.75. The threshold in the stopping criterion is set to 0.75. The coefficients and are set 5.0 and 0.5, respectively, after a simple grid search.
4.2 VALIDATION OF EVALUATION RELIABILITY AND EFFICIENCY
Evaluation Reliability of K-Sort Eval. We select models from the K-Sort Arena (Li al., 2025b) leaderboard and evaluate them using K-Sort Eval, including text-to-image and text-to-video models. These models span different positions the leaderboard to demonstrate the generalizability of our dataset and method. The results, including both rankings and scores, are compared with those in K-Sort Arena, as shown in Table 3. The evaluation results of K-Sort Eval consistent with those of K-Sort Arena, which is entirely based on human preferences. For instance, in the evaluation of FLUX.1-dew the score produced by K-Sort Eval differs by only 0.03 from that in K-Sort Arena, with both methods assigning it the same rank of 5, which highlights the effectiveness of K Sort Eval.
Evaluation Efficiency of K-Sort Eval. Thanks to the proposed dynamic matching strategy, the evaluation process does not requires traversing the entire dataset, which significantly improves efficiency. Figure illustrates the number of runs required for the new model, with data from 100 tries covering all models. The vast majority (91 for images, 93% for videos) complete the eval- uation in less than 90 runs, which signifi- cant efficiency gain
18%
7%
13%
Image
Video
73%
70 Runs
<0 Runs
70 -090 Runs
70 -00 Runs
existing methods such FID (typically 50,000 runs) (Heusel et al., 2017), GenAl-Bench (1,600 runs) (Li et al., 2024a).
> 90 Runs
> 90 Runs
Figure Number of runs required for the new model in the evaluation.
8

---

Published as a conference paper at ICLR 2026
4.3 COMPARISON WITH PREFERENCE SCORING METHODS
We select 100 instance groups to compare the correlations between different methods and ac- tual human preferences, including CLIP-basee scoring methods (ImageReward (Xu et al., 2023),
1.00
Spearmas Correlation
CLiP-based
VLM-based
0.75
0.73
0.62
0.55
0.57
PickScore (Kirstain et al., 2023), HPS (Wu et al.,
0.50
0.43
2023), and VQAScore (Li et al., 2024a)) and VLM-based methods using GPT-4o (OpenAI, 2024), as shown in Figure 4. We also report the result with correction obtained through X' weight- ing. GPT-4o consistently outperforms CLIP- based methods, and introducing correction signif- icantly improves overall correlation, as it reduces the influence of noisy observations.
0.38
0.25
0.00
Image PickScore
HPS
VQAScore GPT-4o
GPT-4o
Figure 4: Correlations of different methods with actual human preferences.
4.4 APPLICATION OF EVALUATION ON COMPRESSED MODELS
The evaluation of compressed models also crucial real-world applications (Li Gu, 2023; Li et al., 2023c; 2022). K-Sort Eval provides both absolute scores and relative rankings, thus it not only assesses the performance degradation after compression, but also identifies which standard model the compressed model is functionally comparable to.
Table 4: Application evaluation on compressed models, which reveals not only changes in abso- lute scores, but also the results of relative rankings. The model size is calculated in FP16 by default.
(a) Distilled models
(b) Quantized models
Model
Rank
Score
Size (GB)
Step
Model
Rank
Score
Size (GB)
Step
SD-x3.5-large
3
28.95
16.2
40
F.U
5
28.83
24
28
Dulle-3
28.25
Dalle-3
28.27
SD-v3.5 large-turho
9
27.71
16.2
NT4 (RNB)
27.93
6
PLUX.--schnel
10
27.69
SD-v3.5-large--uuie
27.73
16.2
SDXL
32
18.85
5.2
25
I-schnell
27.71
W4A4 (SVDQuant)
11
27.66
6
28
SD.v1.5
20.11
Midjourncy-v5.0
27.44
SDXL-SSD-IB
30
19.40
2.6
25
SD-2..1
31
18.95
1.73
Distilled Models. Table 4a gives examples of distilled models, with reduced step and model size, the number of inference steps is reduced from 40 to 4, result- ing in score drop 1.24 and a ranking shift from to 9. Based on its ranking, we casily conclude that its performance is comparable to that of Dalle-3 and FLUX.. B-U.X.1-schae 1-schnell.
respectively. For 3D-v3.5-large-tune
Quantized Models. Table 4b reports the quantization results FLUX.1-dev, including BNB (Dettmers al., 2023) and SVDQuant (Li et al., 2024c). When quantizing in NF4 format, the model size is reduced by 4x, while delivering score decrease of 0.90. Crucially, it offers an intuitive measure of relative capability and directly points to benchmark model of similar strength.
4.5 ABLATION STUDIES
Effeec of the Proposed Modules. We verify the validity of posterior correction and dynamic match- ing, shown in Table 5. When evaluating FLUX.I-dev. without posterior correction, every VLM judgment fully accepted, even when misaligned with human preferences. This leads to reduced evaluation accuracy, with score deviation of 0.49 and rank discrepancy of 2 compared to Arena. Additionally, the absence of dynamic matching, the entire dataset needs to be traversed, leading to increased costs. The grid search of coefficients and
shown in Appendix F.
Prompt Designs for VLM. Table 5 further illustrates the impact of prompt designs. In the case of FLUX.11dev for example, the model ranking is shifted when the wrapping operation is removed.

---

Published as a conference paper at ICLR 2026
Table 5: Ablation studies on effect of the proposed modules and prompt designs. We report the
results of text-to-image model FLUX.1-de and text-to-video model Cog VideoX-bb.
(a) 1-Ux.
(b) CogVideoX-Sb
Score
#Runs
Rank
Score
#Runs
Sort Arena
28.83
K Sort Arena
33,60
K-Sort Eval (Ours)
28.86
81
K-Sort Eval (Ours)
33.63
Pusterior Correction
Posterior Correction
w/o Dynamic Matching
28.79
500
woo Dynamic Matching
33.65
300
Wapping Operation w/o Rule Augmentation
28.93
79
Wapping Operation w/o Rule Augmentation
33.55
28.13
119
33.10
130
Moreover, without rule augmentation, the VLM lacks clear and uniform principles in the judgment, resulting in a substantial score difference 0.70.
5 CONCLUSION
In this work, we propose K-Sort Eval, scalable and reliable evaluation framework that leverages vision-language models (VLMs) with posterior correction and dynamic matching strategies to ap- proximate human preferences in generative model assessment. By utilizing high-quality dataset from K-Sort Arena and introducing Bayesian correction based on Eval significantly improves alignment with human judgemenss.. Furthermore, the proposed dynamic matching enhances evaluation efficiency by selecting instances with maximum expected gains. Ex- perimental results show that K-Sort Eval achieves alignment with human-voted scores and rankings, while substantially reducing evaluation costs, highlighting its reliability and efficiency.
consistency, K-Sort
ACKNOWLEDGMENTS
This work is supported part by the Strategic Priority Research Program of Chinese Academy of Sciences under Grant Number XDB1100000; in part by the National Natural Science Founda- tion of China under Grant Number 62276255; in part by the Postdoctoral Fellowship Program of CPSF under Grant Number GZC20251175. Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant, NUS ARTIC grant, Apple grant, Alibaba grant and Adobe gift.
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A versatile languge model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint 2xXiv:22/2.0807: 2022.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang. Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, et al. Benchmarking foundation models with -aapee---ooo-------eee--eeee Advances in Neural Information Processing Systems, 36:78142-78 1677, 2023b.
Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical common- sense for video generation. arXiv preprint arXiv:2406.03520. 2024.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dalll--3, pdf. 2(3):8, 2023.
10

---

Published as a conference paper at ICLR 2026
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Milm-as-an as-aa-jddeee: Assessing multimodal im-as-a-judge with vision- vision-language benchmark. In Forty-first International Conference on Machine Learning, 2024.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating IIms by human preference. In Forty-first International Conference on Machine Learning, 2024.
Christopher Chou, Lisa Dunlap, Koki Mashita, Krishna Mandal, Trevor Darrell, Ion Stoica, Joseph Gonzalez, and Wei-Lin Chiang. Visionarena: 230k real world user-vim conversations with pref- erence labels. arXiv preprint arXiv:2412.08687, 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems,
2023.
Pan Du, Wangbo Zhao, Xinai Lu, Nian Liu, Zhikai Li, Chaoyu Gong, Suyun Zhao, Hong Chen, Cuiping Li, Kai Wang, et al. Unsupervised learning for class distribution mismatch. In Forty- second International Conference on Machine Learning.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germani- dis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7346-7356, 2023.
Jiawei Gu, Xuhui Jiang. Zhichao Shi, Hexiang Tan, Xuehao Zhai Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, et al. survey on llm-as-a-judge. arXiv preprint arXiv:2411.15500 2024.
Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, al. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. arXiv preprint arXiv:2406. 15252, 2024.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.132211 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp- 20406-20415 2023.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-oompbench: compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78733-7877777777 2023.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06677 2023.
Sameer Jain, Vaishakh Keshava, Swarnashree Mysore Sathyendra. Patrick Fernandes, Pengfei Liu, Graham Neubig, and Chunting Zhou. Multi-dimenssonal evaluation of text summarization with in-context learning. arXiv preprint arXiv:2306.01200. 2023.

---

Published as a conference paper at ICLR 2026
Dongfu Jiang, Max Ku, Tianle Li, Yuansheng Ni, Shizhuo Sun, Rongqi Fan, and Wenhu Chen. Genai arena: An open evaluation platform for generative models. Advances in Neural Information Processing Systems, 37:79889-79908 2024.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
Jannik Kossen, Scbastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In International Conference on Machine Learning, pp. 5753-5763, PMLR, 2021.
Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312.18867 2023.
Baiqi Li, Zhiqiu Lin, Deepak Pathak, Jiayao Li, Yixin Fei, Kewen Wu, Tiffany Ling, Xide Xia, Pengchuan Zhang, Graham Neubig, al. Genai-bench: Evaluating and improving compositional text-to-visual generation. oennerationnnnnnninnnn arXiv preprint arXiv:2406.13743 2024a.
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al. From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594 2024b.
Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun- Yan Zhu, and Song Han. Svdqunat: Absorbing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.0550
2024c.
Ruosen Li, Teerth Patel, and Xinya Du. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02776, 2023a.
Zhikai Li and Qingyi Gu. I-vit: Integer-only quantization for efficient vision transformer infer- ence. Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 17065- 17075, 2023.
Zhikai Li, Liping Ma, Mengjuan Chen, Junrui Xiao, and Qingyi Gu. Patch similarity aware data-free quantization for vision transformers. In European conference on computer vision, pp. 154-170. Springer, 2022.
Zhikai Li, Mengjuan Chen, Junrui Xiao, and Qingyi Gu. Psaq-vit v2: Toward accurate and gen- cral data-free quantization for vision transformers. IEEE Transactions on Neural Networks and Learning Systems, 2023b.
Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameterizzaioo for post-training quantization of vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 17227-17236, 2023c.
Zhikai Li, Mengjuan Chen, Junrui Xiao, and Qingyi Gu. Coleq: Improving data-free quantization via contrastive learning. IEEE Transactions on Multimedia, 2025a.
Zhikai Li, Xuewen Liu, Dongrong Fu, Jianquan Li, Qingyi Gu, Kurt Keutzer, and Zhen Dong. K-sort arena: Efficient and reliable benchmarking for generative models via k-wise human preferences. In Proceedings the
the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025b.
Ming Liu and Wensheng Zhang. Is your video language model a reliable judge? arXiv preprint arXiv:2503.05977 2025.
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22139-22144 2024.
Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wild- vision: Evaluating vision- languaee models in the wild with human preferences. arXiv preprint arXiv:2406. 1069, 2024.
12

---

Published as a conference paper at ICLR 2026
OpenAl. Hello http://p////ooaaaa.oo/////////// 2024. Accessed: 2024-05-26.
Lawrence D Phillips and Ward Edwards. Conservatism in a simple probability inference task. Jour- nal of experimental psychology, 72(3):346, 1966.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. XrXiv preprint arXiv:2307.01952, 2023.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuckai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating Ilms with fewer examples. arXiv preprint arXiv:2402.14992. 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763. PmLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D'souza, Sayash Kapoor, A. Ustun, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah Smith, Beyza Hilal Ermíş. Marzieh Fadaee, and Sara Hooker. The leaderboard illusion. arXiv preprint arXiv:2504.20879, 2025.
Charles Spearman. The proof and measurement of association between two things. 1961.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprini arXiv: 1812.0117, 2018
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. arXiv preprint arXiv: /4420, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 11.903-11933, 2023.
Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer. 2025a.
Wangbo Zhao, Yizeng Han. Zhiwei Tang, Jiasheng Tang, Pengfei Zhou, Kai Wang, Bohan Zhuang, Zhangyang Wang, Fan Wang, and Yang You. Rapid 3: Tri-level reinforced acceleration policies for diffusion transformer. arXiv preprint 2509:22500222.222. 2025b.
Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang. Hao Luo, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dydit++: Diffusion transformers with timestep and spatial dynamics for efficient visual generation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2026.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging aud-ee with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46622 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint 21X1122211.11018, 2022.
13

---

Published as a conference paper at ICLR 2026
A EVALUATION CRITERIA IN K-SORT ARENA
In K-Sort Arena, all crowdsourced participants are professors and graduate students specializing in visual generation, affiliated with institutions such as University of Califoria Berkeley, Chinese Academy of Sciences, National University of Singapore, and Nanyang Technological University, etc. They all complete pre-voting training, particularly on the following evaluation criteria:
text-to-Image Models The evaluation is based on alignment (50%) and aestheties (50%). Align- ment encompasses entity (30%) and style (20%), while aesthetics includes photorealism (30%), light and shadow rendering (10%), and the absence of artifacts (10%).
> Text-to- Video Models The models are also evaluated based on alignment (50%) and aesthet- ies (50%). Alignment is assessed based on video content matching (20%), movement matching (15%), and inter-frame consistency (15%), while aesthetics considers photorealism (30%), physical correctness (10%), and the absence of artifacts (10%).
Additionally, as open-source project, K-Sort Arena actively encourages contributions from the public community, with the criteria serving as a guiding reference for their voting as well.
B FILTERING THRESHOLD IN DATASET CURATION
We filter the data by the Spearman's ' rank correlation coefficient between the local rankings within the dataset and the corresponding model's ranking in the overall leaderboard to prevent preference contamination. Due to performance fluctuations relative to a model's 's true capability and the presence of ties, even preference-aligeee aligned data cannot always guarantee correlation of 1.0. Therefore, is necessary to determine sufficiently reliable selection threshold.
Table 6 lists spearman's 's rank correlation coeffi- cients in different cases, including tie and mis- ordering cases. In our dataset curation, we con- sider the cases of tie between two models and misordering between two models to be valid samples, while the cases of misordering among three models is invalid samples. As a result, in order to balance validity and diversity, the filtering threshold to 0.75.
Table 6: Spearman's 's rank correlation coefficients different cases, including tie and misordering.
Case
Rank
Spearman
Ground Truth
[0.1,2.3]
Fully consistent
[0,1,2.3]
1.00
Tie between two models Tie among three models Misordering between two models
[0.1,1.2]
0.95
[0.1.1.]]
0.77
[0,2,1.3]
0.80
Misordering among three
[0.3.1.2]
C DERIVATION OF BAYESIAN UPDATING
We begin by analyzing the case of two competing models, Mj and M2, before generalizing to the comparison among K models. Suppose the observation /) indicates that model M1 outperforms model M2. The likelihood of this event, conditioned on the latent performance parameters 01 and 02 is given by:
( 110-0
(1)
01 01-02 02
P(D 02) P(X, X2) =
(18)
VBJ 32
where (xx) denotes the cumulative distribution function (CDF) of the standard normal distribution, and ((x) is the corresponding probability density function (PDF):
(xx) = J slu) du, (xx)
12/2
(19)
v 2m
Using Bayes' theorem, we can then derive the joint posterior distribution of (0,, 0z) given the ob- servation D as follows:
P(),,021D) & PO)PP()))) = ( 0
02
U2
0 02
(20)
02
14

---

Published as a conference paper at ICLR 2026
The marginal posterior distribution of 01 can be obtained by integrating out 0₂ from the joint poste- nor::
P(0iDD) - [ P(01,02|D) d0g x
( 141
(1
12 $2 02
+
(21)
31
Given the marginal posterior distribution, the posterior expectation 0₃ can then be computed as:
A E _ (]P(0]
v (p{+))
l41 + VE (B)
S_ P(0)
+
#1 HI-H2
(22)
VE(+++o?)
= f1 + of
11
42
V
C12
C12
where V(z) (xx)/((z) and c Σ based on the observed outcome. In similar manner, the posterior variance of 2 is updated using the following expression:
of) The mean Pu of 02 is updated accordingly
of W +
di =Var Var(0/1D) = E(0 (E(O
-1-
P2 VE (B. + o/)
(Bg
of)
of (1-
of W
12
c22
C12
(23)
where W(x) (Vx) + z). The aforementioned process completes the Bayesian updating for a pairwise comparison between two models. We now extend this framework to free-for-all comparison among K models. In this case, the update rules for the performance parameters of the i-th model are given by the following equations:
tus = =4=4+ +1+
Σ qirq>:n
My
Σ
v
My 14
(24)
Cig
Ciq
Cig
Cig
giT's qeristo
di 2 = (1-
Mi
Mlg
Hi
W
(25)
Cig
Q:Ty <Pg
D RULE AUGMENTATION FOR VLM PROMPT
We adopt the rule augmentation strategy to provide clear and effective guidance for VLM judge- ments. To ensure consistency with K-Sort Arena, these rules are aligned with the manual voting criteria used by human annotators. This enhances the interpretability of VLM outputs and improves their comparability with human preferences. The complete prompt design is illustrated in Figure 5.
VLM Prompt for Text-to-Image Models
VLM Prompt for Text-to-Viine Video Models
You will be given images. Your task is rank them from best to worst based on visual aesthetics (50%) and alignment with the given description (50%): [prompt].
You will be given 5 videos. Your task to rank them from best to worst based visual aesthetics (50%) and alignment with the given description (50%): [prompt].
Aesthetics includes photorealism (30%). light and
Aesthetics includes photorealism (30%). physical
shadow (10%), and absence of artifacts (10%);
correctness (10%), and absence of artifacts (10%);
Alignment includes entity matching (30%) and style matching (20%):
Alignment includes content matching (20%), movement matching (15%), and inter-frame consistency (15%);
You MUST respond with only the sorted images in the strict
You MUST respond with only the sorted videas in the strict format: "video 1, video 2, video 3, video 4, video 5*.
format: "Image Image 2, Image 3, Image 4, Image 5*.
Figure 5: Prompt design that provides voting criteria consistent with human voting in K-Sort Arena, serving as guidance for the VLM Judgement and helping reduce hallucinations.
15

---

Published as a conference paper at ICLR 2026
E K-SORT ARENA LEADERBOARD
We use the leaderboard provided by K-Sort Arena (Li et al., 2025b) as the ground-truth baseline for human preferences. In this work, all referenced leaderboards are based on the version updated on Sep 15, 2025, as shown in Table 7.
Table 7: K-Sort Arena leaderboards updated on Sep 15, 2025.
(a) Text-to- Text-to-Image Models
(b) Text-to-Video Models
Rank
Model
Organization
Score (2//or
Rank
Model
Organization
Score (p//)
GPT-4o
OpenAT
30.86 (33.20) (3..8)
Sora (official)
OpenAI
34.66 (37.42 0.92)
FLUX-ILI-pro
Black Forest Labs
29.52 (31.57 37/0.68)
Runway-Gen3
Runway
33.93 (35,94 0.67)
SD-v3,5-larg
Stubility Al
28.97 (31.13/0) (..72)
Cog CogVidexX-5t 5b
Tsinghua
33.60 (35.63 0.68)
FLUX.I-pro
Black Forest Labs
28.90 (30.89 0.66))
Sora (release)
OpenAI
33.53 (35.61 0.69)
FLUX.II-A
Black Forest Labs
28.83 (30.81 /0,66)
Kling-vl.0
Kanishou
32.80 (34.84 (0.68)
Aurora
xAI
28.72 (31.05 0.78)
Runway-Gen2
Runway
29.57 (31.63 (0.69)
Midjourney-v6.0
Midjoumey
(.67)
Pika-v100
(31.27
Dallo-3
OpenAl
28.27 (30.26 (.67)
LaVie OpenSora
Shanghai Al Lab
28.68 (30.67 (
SD-v3.5-Jarg -.-y..--laroo-turrbo
Stability Al
27.23 (00.94
HPC-AL
27.39 (29.41 0.67)
FLUX.I-schmed
Black Forest Labs
27.71 (29.72/0.61 (.677)
10
Piku beta
Pika
27.38 (29.49 0.70)
Midjourey-vv...
Midjoumey
27.44 477.9///// (..6))
11
AnimateDiff
CUHK etc.
26.46 (28.49 0.68)
SD-x3.0
Stability Al
27.13 (29.10/000 0.66)
12
VideoCrafter2
Tencent
23.65 (25,70 /
13
Pixart-Sigma
PixArt-Alpha
26.38 (28.3 0.67)
13
SubblViisoDDituiionn
Stability Al
23.01 (25.09 0,70)
Proteus-v0.2
DataAuloGPP3
Zenseope-v2-xl
Cerspense
0.79)
Opea-Dalle-v1.1
DataAutoGPT3
24.65 (2.665/0.67) 0.67)
Realvisxt-v3.0
Realistic Vision
23.93 (25.94/0.67)
Dreamshuper xl
Lykon
23.89 (25.85 (0.66)
Renlvisal-v20
Realistic Vision
23.87 (25.87 (0.67)
Kandinsky-v2.22
Al-Forever
23.57 (25.56 0.66)
20
Deepfloyd-IF
DoepFloyd
23.47 (25.47 125.47/0..5 (.677)
Meissonic
Alibaba, Skywork
(24.93/0.3)
Kandinsy-vv.00
Al-Forever
22.51 (24.48 0.65)
SDXL-turbo
Stubiiity Al
21.83 (23.93 (070))
Dalle:2
OpenAl
21.74 (23.72/0.66)
Playground-v2.5
Playground AT
21 660 (23.55 (0.65)
Openjouncy-v4
Promptbero
21.41 (23.39 0.66)
27
LCM-v1.5
Tsinghua
20.89 (22.90 (0677)
Stabiiity
(22.36 0.70)
SD-v1.5
Stablity
20.10 (22.12/060 0.677
SSD-1b
Segmind
19.40 (21.41 /0.67)
SD-v2.1
Staiility Al
18.94 (20.93 (.666)
32
SDXL
Stability Al
18.85 (20.8 / (.66)
Playground-v200
Playground
34
SUXL-Lighunng
ByteDance
18.06 (20.05 (.67)
Stable-casznte
Staillity NUS
0.700
36
SDXL DDX--Despccche
16. 16 (18.15/0.66)
F GRID SEARCH OF COEFFICIENTS
We perform a simple grid search of coefficients and a, as reported in Table 8. Based on the grid search results, we select к 5 and a 0.5 as the optimal hyperparameters. This setting achieves strong alignment with the K-Sort Arena benchmark (rank = 6), while maintaining competitive performance score (28.86) Notably, also results the lowest number of model runs (81), indicating high evaluation efficiency. Compared to other settings, this combination provides the best balance between ranking consistency and evaluation cost.
Table 8: Grid search of coefficients k and a. We report results for text-to-image model FLUX.1 1-dev and they hold for other models.
x
K-Sort Arena
3
5
7
Rank Score #Runs
6
6
6
5
28.83
28.58
28.80
28.86
28.90
28.22
110
90
81
74
K Sort Arena
0.1
0.3
0.5
0.7
0.9
Rank Score
6
28.82
28.77
28. 29,50
28.86
28.72
28.93
16


Reviews:
[{"review_id": "UeVPZ2SBUa", "reviewer": "Reviewer_mWxy", "summary": "This paper proposes K-Sort Eval, an automated evaluation framework for visual generative models that uses vision-language models (VLMs) as judges to approximate human preferences. The method builds upon K-Sort Arena by curating a high-quality dataset from human votes and introducing two key innovations: (1) posterior correction using Bayesian updating that accounts for VLM-human alignment consistency, and (2) dynamic matching strategy that selects informative instances based on uncertainty and diversity criteria. The framework aims to provide scalable, efficient, and reliable evaluation while maintaining alignment with human preferences from K-Sort Arena.", "questions": "1. Can you be more clear about how the evlauation is set up? You select a model from the K-sort arena leaderboard and predicts its rank, using the data in the K-sort arena. But this model is already been compared with multiple models in the arena, will this cause circular information leakink?\n2. In Table 3, why there are only 5 models for the text-to-image and text-to-video tasks, while the total rank can be as big as 12. I think you should run K-eval for all of the models in the leaderboard and summarize the prediction result, instead of picking some of them for presentation?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper addresses a real scalability challenge with Arena-style human evaluations, which are costly and time-consuming. The proposed VLM-as-judge approach is timely and practical.\n2. The posterior correction method (Section 3.2) is theoretically grounded in Bayesian inference, with formal derivations provided. The treatment of VLM-human misalignment as observation noise is elegant and well-justified through Lemma 1.\n3. The paper demonstrates consistency with K-Sort Arena across multiple models while requiring significantly fewer evaluations (<90 runs vs. traversing entire datasets). This is quite helpful for people who wants to quickly evaluate a new model without access of huge amounts of human annotation resources.", "weaknesses": "1. The K-sort Eval still an existing K-Sort Arena data as supervision, which limits where this can be applied and cannot produce a score independently. Therefore, the quality of the evaluation also depends on the quality of K-sort arena data.\n2. The evaluation setting more clear about how they define the evaluation task. The K-sort eval selects a model already in the K-sort arena and then predicts its rank using exisitng information from the K-sort arena, there might be information leaking during the prediction that affects the evaluation, thus making the evaluation unfair. This authors should be more clear about this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4YH82cEoBU", "reviewer": "Reviewer_Jr9d", "summary": "This paper proposed K-Sort Eval, a novel framework for efficiently evaluating visual generative models by combining VLM judgments with statistical correction mechanisms informed by large-scale human preference rankings. The system dynamically selects the most informative model comparisons and applies Bayesian posterior correction to align VLM-based scores with true human preferences. This approach dramatically reduces evaluation cost and time compared to human-only leaderboards, while maintaining high agreement with actual human rankings.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1) This paper tickles the pain-point that Its difficult to evaluating visual generative models both reliably and efficiently. Human annotations are highly accurate but expensive and infeasible to scale for every model update or new dataset. Meanwhile, fast automatic scoring using VLMs is scalable, but suffers from biases. K-Sort Eval aims to deliver a solution that preserves human-grade reliability while radically improving cost-efficiency and scalability by statistically correcting VLM judgments using robust human data.​ This is a highly original work and can be generalized to apply on other Arena works.\n\nS2) It proposed a rigorous data cleaning using Spearman correlation filtering and prompt safety screening to ensure high data quality and annotation reliability.\n\nS3) The experiments are well designed and showing high reliably of the proposed method.", "weaknesses": "W1) When this is applied in cases where there are not enough votings, the rankings might not be fully reflecting the performances. Will such behavior propagate to the new models? Also would it be possible to find an estimate on how much collected human/ VLM annotated data are enough to make sure that K-sort eval will works effectively?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ECLK6UIIsp", "reviewer": "Reviewer_2oDh", "summary": "This paper introduces K-Sort Eval, an efficient framework for evaluating visual generative models. The method addresses the limitations of existing approaches: human evaluation is costly and time-consuming, while using Vision-Language Models (VLMs) as judges suffers from inherent bias and hallucinations .", "questions": "1. Have you tested the framework's robustness using different VLM judges? How much does the correction mechanism's effectiveness depend on using a top-tier model like GPT-4o?\n2. Can you provide more justification for Assumption 1? Did you test for systematic VLM biases (e.g., errors correlating with model rank), which would violate this assumption?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Reliable VLM Correction: The core strength is the posterior correction method. By modeling the VLM-human preference gap as observation noise and correcting the Bayesian update , the framework significantly improves alignment with human judgments .\n2. Strong Empirical Validation: The method demonstrates high consistency with the human-voted K-Sort Arena leaderboard on both image and video tasks. Its utility is also shown in practical use cases, like evaluating compressed models .", "weaknesses": "1. Simplified Noise Model: Assumption 1—that VLM noise is statistically independent of the true model capability—is a strong simplification. It's plausible that VLM biases are systematic (e.g., favoring certain aesthetics or failing to spot specific artifacts), which this noise model would not capture.\n2. Incomplete Literature Review: The \"Large Model as a Judge\" related work section overlooks several recent and relevant works exploring VLMs as human-aligned evaluators. For instance, it misses applications in other domains like text-to-3D generation (e.g., **\"GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation\"**) or specialized tasks like personalized image generation (e.g., \"DreamBench++\"), which could offer broader context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2lwdwqntkU", "reviewer": "Reviewer_9sHD", "summary": "This paper proposes K-Sort Eval, a framework for evaluating visual generative models that uses a Vision-Language Model (VLM) as a judge. The goal is to create a more scalable and efficient alternative to human-based \"Arena\" evaluations. The method leverages a curated, high-quality dataset of human preference rankings from the existing K-Sort Arena. It introduces two key technical contributions: 1) a \"posterior correction\" method to align VLM judgments with human supervision by modeling VLM-human discrepancies as observation noise, and 2) a \"dynamic matching\" strategy to efficiently select the most informative comparisons. Experiments show that K-Sort Eval's rankings are consistent with the human-based K-Sort Arena leaderboard, while requiring significantly fewer (under 90) model runs.", "questions": "1. Could you please clarify the workflow for evaluating a new model on a novel prompt that is not in the curated dataset? How is the posterior correction (Eq. 12) applied in that scenario, given that there is no \"dataset supervision\" $R^i$ to compare against? Or, is the framework strictly limited to the 800 prompts in the dataset?\n\n2. The dataset curation (Section 3.1) uses a correlation threshold ($\\tau=0.75$) to filter out human votes that deviate from the global leaderboard. How did you ensure that this process doesn't just \"bake in\" the specific biases of the K-Sort Arena leaderboard and remove genuinely diverse or high-variance (but still valid) human preferences?\n\n3. The ablation study (Table 5) shows very large rank changes when posterior correction is removed. This suggests the VLM's \"raw\" judgment is significantly different from the human-aligned one. Could you provide more analysis on the nature of this discrepancy? For example, is the VLM consistently biased on certain criteria (e.g., aesthetics vs. alignment) or for certain models?\n\n4. How sensitive is the dynamic matching strategy (Eq. 17) to the hyperparameters, particularly the balancing coefficient $\\alpha$? The paper states it was set after a simple grid search, but it seems crucial for balancing the exploration of diverse model matchups versus challenging the new model against its closest competitors.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Addresses a Critical Problem: The paper tackles the highly relevant and important challenge of scalable, efficient, and human-aligned evaluation for generative models, which is a major bottleneck in the field.\n\n2. Novel Correction Mechanism: The \"posterior correction\" method (Section 3.2) is a novel and clever approach to mitigate the known biases and unreliability of VLM-as-a-Judge. Modeling the discrepancy between VLM and human votes as observation noise in a Bayesian framework is a solid technical contribution.\n\n3. Demonstrated Efficiency: The dynamic matching strategy (Section 3.3) and VLM-based approach demonstrably reduce the evaluation cost to fewer than 90 model runs. This is a significant practical improvement over the thousands of comparisons often required in large-scale human evaluations.", "weaknesses": "1. Dependency on Curated Dataset: The entire framework seems to be a \"closed-world\" system. The posterior correction (Eq. 12) appears to require a pre-existing human-voted ranking ($R^i$) for the exact same prompt to calculate the correction factor $\\lambda'$. This suggests the method can only be used to evaluate new models on the static, pre-existing prompts (500 for images, 300 for videos) from the curated dataset, which severely limits its generalizability for open-ended evaluation on new, unseen prompts.\n\n2. Potential for \"Overfitting\" to the Leaderboard: The dataset curation process itself (Section 3.1) filters out human votes that are inconsistent with the overall K-Sort Arena leaderboard. This creates a risk of circularity: the \"ground truth\" used to build the evaluator is already a product of the system it's meant to replicate. This could cause the evaluator to simply \"overfit\" to the specific biases of the K-Sort Arena, rather than capturing a more general or robust notion of human preference.\n\n3. Unclear Scalability of Supervision: The reliance on dataset supervision for the correction mechanism is a key weakness. If the method does require human ranking data for every new prompt to be evaluated, it doesn't solve the scalability problem; it just shifts it. If it doesn't require this, the paper fails to explain how the correction $\\lambda'$ is calculated for a truly new prompt that isn't in the curated dataset.\n\n4. Limited Ablation on VLM Choice: The choice of VLM (GPT-4o, Qwen-VL-Max) is critical, yet its impact is not deeply explored. The ablation study (Table 5) shows that removing posterior correction dramatically changes the model rankings (e.g., from 3 to 6 for CogVideoX-5b), proving the VLM's raw judgment is unreliable. This makes the system highly dependent on the (unexplored) interaction between the specific VLM and the specific correction dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)