VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understanding
Authors: Abdul Waheed
Zhen Wu
Dareen Safar Alharthi
Seungone Kim
Bhiksha Raj
Venue: ICLR 2026 Poster (2026)
Keywords: Meta-evaluation
llm-as-judge
synthetic data
self-refinement
video understanding
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understanding

Abstract: Precisely evaluating video understanding models remains challenging: commonly used metrics such as BLEU, ROUGE, and BERTScore fail to capture the nuances of human judgment, while obtaining such judgments through manual evaluation is costly. Recent work has explored using large language models (LLMs) or multimodal LLMs (MLLMs) as evaluators, but their extension to video understanding remains relatively unexplored. In this work, we introduce VideoJudge, a 3B and 7B-sized MLLM judge specialized to evaluate outputs from video understanding models (\textit{i.e.}, text responses conditioned on videos). To train VideoJudge, our recipe builds on the interplay between a generator and an evaluator: the generator is prompted to produce responses conditioned on a target rating, and responses not matching the evaluator's rating are discarded. Across three out of four meta-evaluation benchmarks, VideoJudge-7B outperforms or is on par with larger MLLM judge baselines such as Qwen2.5-VL (32B and 72B). Notably, we find that LLM judges (Qwen3) models perform worse than MLLM judges (Qwen2.5-VL), and long chain-of-thought reasoning does not improve performance, indicating that providing video inputs is crucial for the evaluation of video understanding tasks.

Published as a conference paper at ICLR 2026
VIDEOJUDGE: BOOTSTRAPPING ENABLES SCALABLE SUPERVISION OF MLLM-AS-A-JUDGE FOR VIDEO UNDERSTANDING
Abdul Wahced Zhen Wu Dareen Alharthi
Seungone Kim Bhiksha Raj
Carnegi Mellon University {abdulw, zhenwu, dalharth, seungonk, bhi cmu. edu
?
Code
Models & Data
ABSTRACT
Precisely evaluating video understanding models remains challenging: commonly used metries such as BLEU, ROUGE, and BERTScore fail to capture the nuances hu- man judgment, while obtaining such judgments through manual evaluation is costly. Recent work has explored using large language models (LLMs) or multimodal LLMs (MLLMs) evaluators, but their extension video understanding remains relatively unexplored. In this work, we introduce VideoJudge, a 3B and 7B-sized MLLM judge specialized to evaluate outputs from video understanding models (i.e., text responses conditioned on videos). To train VideoJudge, our recipe builds on the interplay be- tween a generator and an evaluator: the generator is prompted to produce responses conditioned on a target rating, and responses not matching the evaluator's 's rating are discarded. Across three out of four meta-evaluation benchmarks, VideoJudge-77 out- performs or is on par with larger MLLM judge baselines such as Qwen2.5-VL. (32B and 72B). Notably, we find that LLM judges (Qwen3) models perform worse than MLLM judges (Qwen2.5-ViL VLL,, and long chain-of-thought reasoning does not improve perfor- mance, indicating that providing video inputs is crucial for the evaluation of video un- derstanding tasks.
INTRODUCTION
Recent advances in multimodal large language models (MLLMs) have significantly improved video captioning, question answering. and long-form video understanding across various do- mains (Nguyen et al. 2024). However, their progress poses critical challenge: how evaluate their outputs with reliability, interpretability. and at scale? Traditional reference-based metrics such BLEU, ROUGE (Lin 2004), and BERTScore (Zhang al. 2020) struggle to capture seman- tic fidelity, contextual grounding, or task-specific reasoning. Moreover, in open-ended tasks where multiple valid answers exist, simple reference overlap be misleading. Human evaluation is of- ten considered the gold standard, but is expensive, slow to scale, and suffers from inter-annotator variability (Liang al., (2025). A promising alternative is LLM-as-a-Judge. By prompting or fine- tuning language models to assess responses, this paradigm has improved evaluation in text gener-
ation (Zheng al., 2023 Kim et al. 2024a Gu al. 2025 Li al. 2024b) and more recently
in vision- vision-language tasks via MLLM-as -aM---as----aa Judge (Chen et al. 2024a 2024b).
Xiong et al. 2025 Lee et al.
Applying judee to video understanding remains underexplored, largely due to the tem- poral and multimodal complexity of the video. Beyond this inherent difficulty, two broader limi- tations persist. First, the field lacks large-scale evaluation resources: there are no comprehensive datasets with human preference signals or standardized benchmarks for verifying alignment with human judgments. As a result, existing work either relies on proprietary models such as GPT-4 or GPT-4o (Pu et al. (025a)), which lack transparency and reproducibility, or on small open-source MLLMs zero-shot settings, which fall short of human-level reliability. Second, principled eval- uation criteria are missing. Current (M)LLM MMLLMM-as---ju au-a--uddee methods depend either on generic rubrics, which are often vague and brittle, manually authored rubries, which cannot scale across tasks.

---

Published as a conference paper at ICLR 2026
To address this gap, we introduce framework to bootstrap data to train video understanding evalua- tors. The framework has two key pillars. First, it automatically generates training data by producing candidate responses across a 1-5 rating scale, validating them with evaluator model. and refin- ing cases where predicted ratings diverge from expectations. These bootstrapped examples are then used train both pointwise and pairwise judge models. Second, the same process enables the struction of new pointwise and pairwise meta-evaluation benchmarks, providing large-scale, high- quality resources for systematic comparison. For evaluation, video-instruction pairs are sourced from datasets distinct from those used in training to avoid distribution overlap. In this way, our ap- proach eliminates the need for costly human annotation while yielding both robust training data and standardized evaluation suites, Beyond these bootstrapped benchmarks, we evaluate our judge mod- els on existing independent, auman-annnoatee benchmarks both pointwise and pairwise settings, demonstrating that trained judge models learn generalizable evaluation capabilities that align with human judgment rather than merely fitting to the preferences of the generator-evan evaluuto pipeline.
Further, we train MLLM judge models not only to predict ratings with explanations, but also to generate instance-specific rubrics at test time. This enables fine-grained evaluation that is both interpretable and anchored in explicit evaluation guidelines. Experiments in both pointwise and pairwise settings show that our models match or surpass much larger models while correlating more strongly with human ratings and demonstrating higher sample efficiency.
In summary, our contributions are:
We introduce VideoJudge, the first bootstrapped framework for training scalable MILM- based evaluators across diverse video understanding tasks.
Train judge models that can not only assign ratings but also generate high-quality, instance- specific rubrics at inference time.
We demonstrate that fine-tuned small models on the bootstrapped data can match or outper- form much larger models in accuracy and alignment with human-specifted ratings.
We provide suite of trained pointwise and pairwise judge models, meta-evaluation bench- marks, bootstrapped datasets, and other artifacts to support reproducible research in video understanding evaluation.
2 RELATED WORKS
Video Understanding Models and Evaluation Advances in large language models have led to the development of multimodal models that jointly process and generate outputs across text, image, au- dio, and video modalities (Bai et al. 2025b; Chen et al. 2024c Wu et al.) 2024b] Xu et al. 2025 Zhao et al.] 2025 Chen al. [025 Wu al. (025). growing line of work explores video un-
derstanding specifically, either by pre-training multimodal models with video-text corpora (Zhang jet al. 2024b) 2023) Cheng et al., 2024 Zhang al., 2025a] Wang et al.
2025) or by instruction- 2024c These models are often evaluated using conventional, automatic metrics such as BLEU (Papineni et al.] 2002), ROUGE (Lin 2004), and BERTScore (Zhang al. 2020), which all assume the existence of ref- erence answers. Human evaluation is also widely used, but it is costly and inconsistent. These limitations motivate more principled automatic and semi-automatic approaches.
tuning to align video representations with downstream tasks (Zhang al.
LLM-as-Judge An alternative paradigm for evaluation uses LLMs themselves as evaluators. Sev- eral works have investigated the viability of prompting powerful models such as GPT-4 act as judges on text generation tasks (Zheng et al., 2023; Liu et al. 2023) Ye et al. 2023 Kim et al.) 2025ah. Beyond prompting, other efforts fine-tune peigh models such as Llama-2 (Touvron et al.) 2023) and Mistral (Jiang et al., (023) to serve as reliable evaluators by distilling from GPT- 4's assessment trajectories (Kim et al. [2033] 2024b 2025a). More recently, researchers have ex- tended this line of work to multimodal settings. For example, Chen et al.) (2024b) examines whether multimodal LLMs can act as reliable judges, while Lee et al. (2024a) explores fine-tuning open- weight models such as LLa (Liu et al. 2024) to mimic the evaluation capability of propri- etary MLLMs. Similarly, He et al. (2024) and Ku et al. (2024) investigate the use MLLMs as judges for text-to-image and text-to-video tasks. Together, these works highlight the promise of LLM-as-a- a-Jdge for scalable evaluation, while underscoring the need to further test its robustness in video understanding.
2

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
Our bootstrapping framework consists of a generator-evaluutoo pipeline that jointly synthesizes data and enforces quality control. The design draws inspiration from self-refinement approaches, where self-consistency (Mitchell et al.) 2022, Wang et al.] 2023; Chen et al.} (023) and self- verification (Weng al. 2023) enhance LLM performance, and models adapt through verbal feed- back (Madaan el al.) 2023). Our overall framework has two stages: (1) iterative bootstrapping to construct large-scale, fine tine-grained training data, and (2) fine-tuning judge models which are evalu- ated under both pointwise and pairwise settings. The proposed framework is shown in Figure [
Generator
Ratting
Response
Evalustor
Rating
Hesporsullt
I-l
Response N-1
Rating
Response
Sold Responce
Reeprrmee
Reeporte
Generation Prompt
Whempt genern.ing
Evaluation Prompt
(Ailettngg Billttngl 3'hneabald ?
resporses video
Generator
(rromp for eossating generated reparssee
Evaluator
Rootstrappad
Input: Eins crussin,,, [vides], [rexpunse]
evaluate
finetune
Understanding Eval Stite
Outgut:
creasontegic/reen scorans/sccees
Small Evaluator Model
Figure 1: Overview of our bootstrapping framework for training scalable video understanding eval- uators. A generator first produces candidate responses for a to N-1 rating scale (N 5) for each video-instruction pair. These responses are then scored by an evaluator, and only candidates whose ratings align with expectations are retained. Through an iterative refinement loop. mismatched re- sponses are revised until they satisfy the acceptance criterion. The resulting bootstrapped dataset provides high-quality supervision signals, which we to fine-tune small VideoJudge models.
3.1 BOOTSTRAPPING PROCESS
We begin with seed data sourced from three large-scale video estructtionrrsponnee datasets: VideoInstruct-IUn 100K (Muhammad Maaz & Khan 2023), VCG-Plus-112K (Maaz al. (024)) and VideoChat2-IT (Li et al. 2023). For VideoChat2-IT, which contains multi-turn dialogues, we retain only the first human assistant exchange. The three corpora are merged and de-duplicated the instruction level for each video using a MINHASHLSH index (128 permutations, Jaccard threshold 0.9). From this de-duplicated pool, we randomly sample 25K examples, resulting in a corpus of triplets (v, x. y). where v denotes the video, x the instruction, and the gold-standard response. During bootstrapping. we generate dense video descriptions using strong vision-language models A.2) and them semantic context for both the generator and evaluator. This provides richer grounding while reducing the need for repeated inference over raw video, making the pipeline more cost efficient.
To transform this seed corpus into a training dataset for the evaluator, we iteratively generate and refine candidate responses for each (ū,z,y") triplet. The process follows three stages: Initial Gen- eration, Feedback, and Refinement, described formally below.
Initial Generation:
For each instruction-vide pair (a,, ū) with gold response y", generator model G produces candidate responses, each intended to correspond to a rating E {1,.... -1] shown in
The gold response y" is included the highest-rated response with rating N.
(r) yo G(Pgenllib ||||>|y"" .r).
Feedback; Each candidate response yi is evaluated by an evaluator model E, which assigns a rating & and provides reasoning n(()) We then compute the deviation between the intended rating
3

---

Published as a conference paper at ICLR 2026
and the evaluator's rating f to determine whether the candidate should be accepted refined. Candidates for which A S a are accepted directly into the dataset. The evaluation process can be formalized as:
F Eppeval (| ||0||=||y""l IIv(")
(2)
A (r) - -r
Refinement: For candidates with a rating deviation A a. the generator is prompted again using the evaluator's 's feedback to improve the response. This iterative refinement continues until the candidate meets the acceptance criterion or a maximum of iterations reached, refinement step is formalized as:
(r) 1e+1
Acceptance Criterion: A candidate response y/ is added to the bootstrapped dataset if |r- < a. The final dataset, therefore, consists of {(v, y/ r)} triplets with aligned ratings.
The complete process is outlined Algorithm [ Using this pipeline, we bootstrap pointwise data with N 5, where each instruction is paired with five responses rated from 1. Representative examples are shown in Table 6 Appendix A.4
3.2 MODEL TRAINING
We use the bootstrapped dataset to train pointwise and pairwise evaluator models. The dataset is defined as D {(v,, Fi, Mh; (>) where nt denotes the video, æ; the instruction, Y/ candidate response (or response pair in the pairwise setting), and f; the associated target annotation, such as a rating or preference label. The evaluator model is trained end-to-end to autoregressively generate the target sequence conditioned on (Vi, 24, 34), with the standard negative log- log-likelihood over tokens:
M M
((0) = M
ΣΣ
log Po (tas t4,ss, (i,
where 4ij denotes the j-th token of 4. This loss is applied to both pointwise and pair- wise models. In the pointwise setting, the model produces intermediate reasoning within <thinkings<<thhink..n
followed by scalar rating in <score><<scores> and it can op- tionally generate task-specific rubrics in soubric></rubbr In the pairwise setting, the model outputs its decision within <answer></ annserr> based on pair of candidate responses.
before reasoning and evaluation.
4 EXPERIMENTS
We bootstrap pointwise data starting from 25K seed video instruction-response pairs. After the bootstrapping process, we retain only instructions with at least five responses (one for each rating), resulting in 103,825 examples across 70,765 unique indoo-instruction pairs.
We construct pairwise supervision by forming response pairs where the higher-rated output is chosen as preferred, Due to computational limitations and while keeping the setting identical, we randomly sample 50% of all possible pairs, resulting in 103,825 pairwise training examples. Both pointwise and pairwise judge models are trained on these bootstrapped datasets. In the pointwise setting, the model takes video, instruction, and candidate response input, and is trained to produce reason- ing trace followed by a score. We further train a judge model to first generate an instruction-specific evaluation rubric, which is then applied when scoring, ensuring that evaluations context-specifie criteria. In the pairwise setting, each instruction is paired with two candidate re- sponse and the model trained identify the preferred response.
grounded in
"o avoid positional bias, the order of responses is randomized during hoth training and evaluation.
4

---

Published as a conference paper at ICLR 2026
4.1 BASELINES
We evaluate broad range of models, including unimodal language models and multimodal video- language models, to evaluate their ability as judges. Unimodal models are tested using detailed video descriptions ($A.2) proxies for visual input, while video models directly process the video content. This setup allows us to compare how different model classes handle the judging task.
Unimodal Models Each unimodal model is provided with the description, instruction, and candi- date response, and is prompted to generate a reasoning trace followed by a score. We consider Qwen3 (Yang et al. 2025a) family of models from 0.6B to 14B, and also enable the "thinking mode" of smaller models (up to 4B) to test whether extended reasoning sequences enhance judging ability (Chan al. 2025 Kim et al. 2025b Zhou et al. 2025).
(Bai et al. 2025a along with other recent video- language models, including LLaVA-Next (7B) (Zhang et al. 2024b), VideoR1 (7B) (Feng) itt al,, 2025). and LLaVA-OneVision (Li et al.] (024a). In our preliminary experiments we find that several models-such
Video Models We evaluate Qwen2.5-VLL.3B-725
VideoLL (Zhang et al. 2025b), VideoChat-Flash (Li et al. 2024d), Keye-VL. VL (Yang et al., 2025b), and MLM2 (Marafioti et al. 2025) often failed to fol- low instructions or produce valid scores under the same evaluation setup. Consequently, we exclude them from our main results.
4.2 EVALUATION
Each videa-instruction-rrspsss triplet evaluated independently, with the model pro- ducing reasoning trace followed by rating 1-5 scale. We construct two meta-evaluation benchmarks, MitoolugeLLLaA-Meeie struction data from LLaVA- (Zhang et al. 2024c) and VideoChatGPT 2 then generating ad- ditional responses via our bootstrapping pipeline (Algorithm [ with threshold O. We report corre- lation and error metrics, as well as divergence error. We further evaluate on Vatex-Eval (Shi al.) 2022). which contains multiple human judgments aggregated into continuous ground-truth scores, emphasizing ranking- and separation- based measures to capture preference consistency. We also use Long VideoBench (Wu et al.) 2024a) for long-form multiple-choice evaluation by rating correct versus distractor answers, reporting both the average score gap (Delta) and pairwise superiority.
Pointwise
and VydelludgVCGG-Metttvy
by sourcing seed in-
Pairwise The judge models compare two candidate responses for the same instroiisttuuttin and select the preferred one. We use AudeoAAuoAArena (Luo et al. 2025), where human preferences serve as ground truth. From our pointwise evaluation data, we also construct pairing responses with different ratings and treating the higher-rated response as correct, measuring accuracy against this derived ground truth. To probe more subtle distinctions, we create VideoJudge- Pairwise-H, focusing on challenging 2-vs.-3 cases: we sample 250 such pairs, collect annotations from two human evaluators, and retain only those with full agreement, yielding over 200 pairs with human preference ground truth. We report the accuracy score for all pairwise evaluations.
BessoJuddePPaiivvi by
Experimental Setup All models are trained and evaluated under identical hyperparameter settings ensure fairness. We perform fidl fine-tuning in BF16 precision with maximum sequence length of 128K tokens with fps rate of with maximum number of frames 60 for training and 180 during evaluation. We train all models for 2 epochs with batch size of 16. The learning rate is set to 2 x 10 with cosine decay, a warmup ratio of 0.03, weight decay of O, and gradient clipping at 1. We provide key hyperparameters and other implementation details in Appendix
5 DATA EVALUATION
We evaluate the bootstrapped data ensure the quality and that it provides meaningful and reliable supervision. Our data evaluation has two parts: automatic checks to assess the relative quality of the responses, and human evaluation to validate correctness and preference alignment. Together, these evaluations confirm that the generated data is of sufficient quality for training and benchmarking.
httts::/huugiinggfaee.coom
mmms aab.iiabvvvii.aaa deoChat.PP
5

---

Published as a conference paper at ICLR 2026
5.1 AUTOMATIC EVALUATION
During our bootstrapping process, we prompt the generator model produce candidate responses for different ratings by progressively degrading the quality ac- cording to the specified score. A natural proxy to verify that the generated dataset adheres to this design is assess whether response quality indeed declines as we move from higher lower ratings. To this end, we compute BERTScore and BLEU using the gold response reference and the generated candidates (ratings 4-1) as hypotheses. The results presented in Fig- ure 2 show a clear monotonic degrada- tion: BERTScore decreases from 91.1 (5- 4) to 86.9 (5-1), while BLEU drops from 11.0 to 3.0. This consistent downward trend shows that the generator reliably produces responses of progressively lower quality, validating effectiveness of our controlled response generation process. To further incorporate video context
SERT
BLEU
91
-15
90
BERTScore
BLEU (t)
100
68
87
85
5-3
5-1
Figure Comparison of BERTScore (left axis, blue) and BLEU (right axis, orange) across rating pairs. BERTScore remains consistently high while BLEU drops sharply with larger rating gaps.
into the evaluation, also calculate VQAScore (Lin et al. 2024) ($ A.5 with results shown in Figure and observe similar trend.
5.2 HUMAN EVALUATION
We construct our pairwise data by sampling response pairs with different ratings for the same in- struction, choosing the higher-rated one preferred. In practice, found that generator-evaluator disagreements were most frequent around ratings and 3, even after incorporating the feedback loop. To focus these harder cases, restricted human evaluation to pairs with ratings vs. 3. We sample 250 such examples, each containing the video, detailed description, the instruction, and candidate responses. We then ask annotators select the preferred response. Results Agreement between annotators is high (94.8% with Cohen's 's k of 89.5), and
are shown in Table 7 both annotators achieved over 92% correctness relative to the gold preference. The preference dis- tribution shows a mild bias toward response b, but етгог analysis indicates only 4.4% cases where both annotators agreed on the wrong response and 5.2% where they disagreed on a correct one. Overall, human evaluation validates that the generated pairwise data is consistent and reliable, even in the most ambiguous rating regions. We provide detailed metrics of our human evaluation study in Table 7 and examples in Table 8 Appendix A.6
RESULTS AND DISCUSSION
We train Qwen2.5-VL (3B, 7B) models for both pointwise and pairwise evaluation under identi- cal settings. We evaluate various baselines and our trained judge models across suite of meta- evaluation benchmarks. We report pointwise evaluation results Table ! and pairwise results in Table 3 Our findings show that bootstrapped supervision enables smaller models to reach, and in some cases surpass, the judgment reliability and accuracy of much larger (~10x) general-purpose models. We discuss our findings in subsequent sections.
6.1 POINTWISE EVALUATION
We evaluate all models in the pointwise setup, where each system is required to produce scalar score. We use the identical prompt decoding parameter across the different models. Unimodal models provide a useful reference point. Across Qwen3 variants, we find that performance on VideoJudgeLLaVA and VideoJudgeVCG is reasonably strong, but VATEX remains challenging with consistently high error and poor calibration. Long VideoBench is considerably more difficult. Al- though unimodal models achieve non-trivi triviaivial PSup scores, the gap between correct and distractor responses remains small, suggesting that temporal dependencies hard to capture from text-only

---

Published as a conference paper at ICLR 2026
signals. Thinking mode further improves the 0.6B model, showing that explicit reasoning steps are beneficial. However, enabling unimodal models to perform judgment requires high-quality, de- tailed descriptions, often generated by powerful models such Qwen2.5-VL-72B GPT-4o-mini. As a result, the computational cost of generating these descriptions should be accounted for when considering the overall efficiency of the approach.
Video-language model baselines such as LLaVA-NeXT, Vision, and Video-R1 perform compet- itively on VideoJudgeLLa VA and VideoJudge VCG, achieving correlations in the 0.66-0.77 range with relatively low error values, on par with or better than several unimodal Qwen3 baselines. How- ever, their performance degrades substantially on Long Benn,,c where both PSup and ACC-D) drop sharply (e.g., LLaVA-NeXT: 0.59 / 0.45, Video-RI: 0.60 / 0.54), underscoring the difficulty of long-context temporal reasoning. In contrast, the Qwen2.5-VI. series scales more robustly: larger variants (32B, 72B) show consistent improvements across all four benchmarks, achieving higher correlations and stronger ACC-D) values on
Our trained VideoJudge models demonstrate consistently strong performance across most evaluation settings. On VideoJudgeLLaVA and VideoJudgeVCG, both VideoJudge-3B and VideoJudge-7B achieve correlations that not only match but in several cases surpass those of substantially larger baselines such as Dwwn2.5-VL-32255 language systems including LLaVA-NeXT Vision, and Fideo-R1. Beyond short-context bench- marks, VATEX shows similar patterns: feedhack-guides training reduces error and improves cali- bration, leading more accurate and getter-grounder predictions. The effect is even clearer on LongVideo VideoBencn,, where many existing models degrade substantially, while VideoJudge maintains high PSup and ACC-D) scores. Together, these findings indicate that feedback supervision sup- ports more consistent and temporally coherent evaluation, particularly for longer and more complex videos. Overall, cubric-supervinn judges perform on par with or better than larger lidoo-language models, while remaining comparatively scalable.
while also outperforming post-trained baseline video-
Table 1: Benchmark results across VideoJudgeLLaVa. VideoJudgeVCG. VATEX, and Long VidB. Metrics: RMSE/MAE (error), S/P (Spearman/Pearson correlation), ECE (calibration), PSup/ ACC- D) (preference)
Model
VideeJudgel.LaNi
VideaJudgeVCG
VATEX
LongVidB
RMSF, MAR.
St Pt
RMSF, MAF 8+ Pt
RMSF, ECE,
PSup- ACCD))
Qwen3-0.6B
1.20
0.92
0.64
0.63
1.20
0.92
0.66 0.65
1.44
0.84
0.58
0,17
Unimodal
Qwen3-1.7B
0.97
0.60
0,78 0.78
1.33
0.85
0.61 0.60
2.18
0.86
0.64
0,71
Qwen3-4B
0.96
0.61
0.72
0.74
1.22
0.79
0.58
0.59
2.42
0.97
0.67
0.69
Qwen3-883
0.97
0.61
0.73
0.74
1.19
0.76
0.61
0.61
2.06
0.94
0.64
0.53
Qwen3-
1.09
0.65
0.69
0.69
1.29
0.83
0.58
0.58
2.38
0.96
0.63
0.57
Unimodal
(thndking)
Qwen3-0.6B
1.12
0.81
0.64
0.64
1.31
0.98
0.59 0.59
1.98
0.89
0.62
0.63
Qwen3-1.7B
1.00
0.70
0.73
0.75
1.34
0.94
0.56 0.58
1.99
0.82
0.65
0.86
Qwen3 4B
1.02
0.69
0.70
1.33
0.52
0.92
0.69
LLaVA-NeX1-7B
1.06
0.72
0.67
0.66
1.06
0.67
0.70
0.69
1.65
0.84
0.59
U.45
Video Models
LLaVA-OneVisioon
1.01
0.71
0.77
0.75
1.00
0.69
0.77
0.76
1.52
0.78
0.64
0.83
Vidco-RI-7B
1.07
0.67
0.73
0.73
1.74
1.21
0.46
0.47
1.87
0.72
0.6
0.54
Qwea2.5-VL-3B
1.31
0.94
0.63
0.63
1.58
0.52
0.85
0.20
Qwen2.5-VL--B
0.92
0.61
0.77 0.76
1.22
0.76
0.65
0.65
2.36
0.88
0.57
0.35
Qwcn2.5-VL-32B
0.87
0.59
0.80
0.79
1.05
0.75
0.69
0.70
1.43
0.81
0.73
1.08
Qwen2.5-VL-728
0.87
0.61
0.80
0.81
0.98
0.69
0.76
0.77
1.40
0.79
0.71
1.06
Ourt
VideoJadge-3B
1.07
0.61
0.82
1.59
1.06
0.59
0.63
0.70
Videaladge:77
0.96
0.52
0.78
0.80
1.20
0.72
0.74
0.76
1.46
0.64
0.66
1.16
Training Judge Models to Generate Instance-Specifiic Rubrics at Test Time In our setup, we first synthesize training rubrics and then train the model to (i) generate rubric for each instance, (ii) reason with the rubric, and (iii) output integer score. This approach enables scalable, rubric-driven evaluation tailored to individual examples. For computational feasibility, we train Qwen2.5.VVZ VL-3B on 10% of total pointwise data and evaluate on 1,000 examples sampled from VideoJudgeLLav and VideoJudgeVCG: We report the results in Table 2 The rubric generation prompt is shown in Figure TO and the training/evaluation prompt is provided in Figure T2 in Appendix A.3
7

---

Published as a conference paper at ICLR 2026
Our results show that VideoJudgeR-333 trained to generate instance-specifts rubrics, substantially improves over the 3B and 7B baselines. It reduces error (MAE 0.59 vs. 1.15, RMSE 1.05 vs. 1.56) and achieves correlations above 73, comparable to the much larger 32B and 72B base models. This demonstrates that rubric-driven supervision can close most of the performance gap without scaling model size, yielding evaluations that are both more reliable and more interpretable.
Evaluating the Quality of Generated Rubrics While rubric-driven supervision improves model performance, is also important to verify whether the rubrics themselves are meaningful and useful for evaluation. High-quality rubrics should spec- ify explicit, context- context-specific criteria, whereas poor ones risk being vague or generic. To assess rubric quality,
Table Divergence errors and correlation metrics (P Pearson, Spcarman) zero-shot models and the VideoJudgeR 3B model trained generate sppliic--peccilic rubrics
prompled to produce ruhrics logether with reasoning and score.
Model
MAE
RMSE
P-
St
Qwen2.5-VL. VL
1.15
1.56
37.85
37.96
two methods: LLMMas-Judge, where GPT-4o-mini (lemperalure = 0) selects the better rubric between candidates, and human evalua- tion, where 300 rubric pairs per model are judged by three annotators, with outcomes aggregated by unan- imous (as shown in Figure 3) or majority vote (Figure 17 both automated LLM judgments and human preferences.
Owan2.5-VL-7B
0.86
1.22
57.09
57.26
Owen2.5-VL. VL--33B
78.59
Qwen2.5.VI.nn VI..722
0.54
0.87
78.10
78.61
VideoJudgeR-33
0.59
1.05
73.96
74.16
This dual setup measures alignment with
Our results show that VideoludgeR-33 pro- duces substantially higher-quality rubrics than the 3B, 7B, and 32B baselines across all settings, with large margins under both unanimous and majority human judgments and even stronger gains LLM-as-Judge evaluation. Against stronger models, it mains competitive: in the LLM-as-Judge setup, VideoJudgeR-3B achieves 92.7% win rate against GPT-4o-mini and 71.3% against Qwen-72B, consistently maintaining above 50% win rate across all evaluation settings. These findings demonstrate that instance-speciffy rubric generation enables a compact 3B model to outperform much larger models while producing rubrics preferred by both humans and strong LLM judges. We provide example rubries generated by different models in Table 5 in Appendix A.3.2
Win
mini
53.4%
46.6%
Qwen-728
63.9%
36.1%
vs Qwen-328
74.2%
25.8%
Qwen-3B
96.8%
Qwen-7B
98. 3%
1.7%
Videojudge-38
Rate
Figure 3: Win rates from human evaluations com- paring VideoJudge-38 against other models.
6.2 PAIRWISE EVALUATION
Table 3: Accuracy scores (f) of zero-shot base models and
VideoJudge on pairwise meta-evaluation benchmarks. Abbre- viations: VAA Aideo, VJ VideoJudge, VJ-H VideoJudge- w/ FB with feedback, w/o FB = with- out feedback.
We next train and evaluate mod- els in the pairwise setting, where the task to prefer the better two responses the same video- instruction pair. This setup directly captures relative quality and aligns closely with human preference judgments. As before, we assess both base models and our trained Videoludge models, with and with- out feedback, across VideoAu- toArena (VAA), VideoJudge (VJ), and VideoJudge-HHuan (VJ-H). Results are summarized in Ta- ble 3 VideoJudge models con- sistently outperform their backbone baselines across all benchmarks. Notably, VideoJudge-3B achieves 94.0 on VJ and 89.45 on VJ-H
VAA
VJ
VJ-H
Model
w/
w/o FB
w/
w/o
FB
w/
w/o FB
Qwen2,5-VL-31
54.90
52.16
82.60
75.00
85.23
81.01
Qwen2.5-VL-7B
75.29
71.37
89.00
84.60
89.03
82.28
Qwen2.5-VL-32B
80.78
90.59
91.20
91.20
92.83
90.72
Qwen2.5-VIL-728
89.80
89.80
94.00
93.20
94.51
93.25
VideoJudge-3B
7..76
64.71
94.00
95.80
89.45
90.72
Videoludge-7B
85.49
87.45
95.60
98.60
93.67
93.25
8

---

Published as a conference paper at ICLR 2026
(w/ feedback), far surpassing Qwen2.5- VL-3B (82.6/ 85.23) and even outperforming much larger models such as Qwen2.5-VL-325 and 72B in several cases. VideoJJudge-7B further improves per- formance, achieving 98.6 on VJ and 93.67 on VJ-H. These results highlight that our bootstrapping process enables smaller models to match or exceed the reliability of much larger video-language systems. Feedback consistently improves the 3B and 7B baselines. For larger models, however, the gains are smaller. In the VideoJudge variants, the effect is more mixed and depends on the benchmark.
How Many Frames Are Enough for an Effective Video Judge We study the effect of maxframes on judgment performance, as controls the temporal context available for evalua- tion. Too few frames risk omitting critical context, while excessively large values increase compu- tation without proportional benefit. To analyze this tradeoff, we vary maxframes during training (30-500, evaluation fixed at 180; Figure (20a) and separately during evaluation (30-180, training fixed at 60; Figure 20b). This design isolates the role of temporal coverage both training and inference.
When varied during training, VideoJudge shows consistent gains from larger ma x Frames. Corre- lations with ground truth rating increase steadily up to ~24 frames (exceeding 0.7), while RMSE and MAE decline. Beyond this point, improvements plateau, suggesting diminishing returns despite higher cost. In evaluation, increasing maxf aames at inference improves correlation and reduces error up to 120 frames, after which performance saturates.
Overall, these results indicate that moderate to large temporal context is crucial for effective judg- ment. Training benefits from up to 240 frames, while at evaluation, modest values (around 120) suffice capture most relevant evidence. Thus, carefully chosen max rames balance accu- racy and efficiency, strengthening temporal grounding without unnecessary cost.
Decoding Temperature We study the effect of de- coding temperature on sampling reliability, as it di- rectly controls the trade-off between determinism and diversity. This is particularly important for eval- uation models, where stochastic decoding can lead to inconsistent judgments.
Base
Videojudge
0.8
Figure 4 (other metrics in Figure 19 in Ap-
pendix A.8 reports the pointwise performance of the base Owen2.5-VI--3B and its VideoJuuge-trainedd counterpart across different temperatures. The base model degrades steadily as temperature increases, with Spearman correlation dropping from 0.56 = 0.0 to 0.42 at rates and more invalid outputs. In contrast, the VideoJudge model remains stable and even improves moderate to high temperatures, achieving peak correlation of 0.73 and the lowest MAE of 0.69.
Speaman
0.6
0.5
1.0, alongside higher er-
0.4
0.0
0.2
0.4
Temperature
Figure 4: Spearman correlation across tem- peratures for base and VideoJudge models.
Overall, these results suggest that stochastic decod- ing introduces variability in judge model outputs, but rubric- guiidegg training significantly improves robustness to temperature changes. This robustness is practically important, as non-deterministic decoding is often used to encourage diverse and well- calibrated evaluations.
Error Analysis To better understand where Videoludge fails, we analyze model evaluations and find a consistent overestimation bias. For example, our judge models overestimate the score by 22 points in about 14.8% of cases but underestimate by the same margin in only 1.5%. Calibration in the mid-to-high rating range is also poor: only 36.9% of rating-3 responses get the correct score, with 46.6% inflated perfect 5, and 81.3% of rating-4 responses are incorrectly rated as 5, Together, these findings point to the need for training data that emphasizes hard negatives and finer- finer-grained distinctions near the top of the rating scale.
grained

---

Published as a conference paper at ICLR 2026
7 LIMITATIONS AND FUTURE WORK
While Videoludge demonstrates strong performance across multiple benchmarks, both the train- ing supervision and portion of the meta-evaluation benchmarks are constructed using the same generator- pipeline, which can introduce partial "closed-loop" effects. Although we evalu- ate, to the best of our knowledge, the available human-annotann benchmarks, fully independent large-scale pointwise meta-evaluatee evaluation to relatively small set of benchmarks. Expanding evaluation to larger and more diverse human-annotated datasets would further strengthen the reliability and generalizability of our conclu- sions. As detailed in our error analysis ($6.2). VideoJudge also exhibits consistent overestimation bias and poor calibration in the mid-to-high rating range.
datasets for video understanding remain limited, constraining
Looking ahead, incorporating harder negatives and finer-grained supervision could directly ad- dress these calibration gaps, while broader coverage of specialized tasks and domain shifts would strengthen generalization. More broadly, the bootstrapping methodology introduced here is not spe- cific to video understanding and could naturally extend to other multimodal settings where human- annotated evaluation data is scarce.
CONCLUSION
We introduce Videoludge, bootstrapping framework for training MLLM-based evaluators spe- cialized for video understanding. Our approach addresses the lack of evaluation resources with human preference signals and principled evaluation criteria for video understanding. The core con- tribution lies in an iterative generator-ewe evalluato pipeline that synthesizes training data and enforces quality control, creating over 100,000 training examples without costly human annotation. We then fine-tune judge models to generate both ratings and instance-specine rubrics at test time, enabling interpretable evaluations anchored in explicit criteria grounded in the specific instruction and video content. Our experiments demonstrate that fine-tuned 3B and 7B VideoJudge models match or outperform much larger baselines accuracy and alignment with human ratings. VideoJudge-3B achieves comparable performance to models up to 10x larger, while VideoJudge-7B consistently outperforms larger video-language models across multiple benchmarks. VideoJudgeR-3B produces rubrics preferred by both human annotators and LLM judges while maintaining performance compa- rable to much larger base models. By releasing curated meta-evaluation benchmarks, bootstrapped datasets, and trained models, we provide essential resources for reproducible multimodal evaluation research. The bootstrapping methodology is general and could extend to other modalities beyond video understanding.
ACKNOWLEDGEMENT
We would like to express our sincere thanks to Shaily Bhatt and Kashu Yamazaki for their valuable contributions during the early stages of this project, whose discussions and inputs helped shape its initial direction. We also thank Sagnik Mukherjee and Arif Ahmad for their helpful feedback on the manuscript, which improved the clarity, presentation, and overall quality of the write-up.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025a. URL https: /ttp///axvvv./. / abs/2502 2502000.22 13923
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 2025b.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
10

---

Published as a conference paper at ICLR 2026
Chi-Min Chan, Chumpu Xu, Jiaming Ji, Zhen Ye, Pengcheng Wen, Chunyang Jiang, Yaodong Yang, Wei Xue, Sirui Han, and Yike Guo. J1: Exploring simple test-time scaling for as---- 2025. URL
arxiv. org abs httgg//ab///50.222 11875
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yinuo Liu, Yaochen Wang, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Milm-as-a-judge: Assessing multimodal 11m-ase im-as-a-judge with vision-language benchmark, 2024a.
org. abs /t.pb///4400 2402 04788
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Milm-as-a-jjuge: Assessing multimodal llmas--ajudge with vision-language benchmark. In Forty-first International Conference on Machine Learning, 2024b.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501. /7811, 2025.
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin. Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal selff-consitenncy for large language model generation, 2023. URLhhttps://axe http://xxiv...r/a/a//1...1...
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao. Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024c.
Zesen Cheng, Sicong Leng, Hang Zhang, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing patiaal-empooal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.07476 2024. URL https:
//arxiv. //arx//.orrg/as//2066.0777 07476
Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms, 2025. URL /ttt/t/ts///aat http:///xxvvvoooo org/ abs/
Jiawei Gu, Xuhui Jiang, Zhichao Shi. Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge, 2025. URL httptttt///////t http///xxvvvorrr ort.absrg/bb.. 2411. 15594
Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, al. Videoscore: Building automatic metrics simulate fine-grained human feedback for video generation. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 2105-2123, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https: //arxiv.org/ abs 2310 06825
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome,, et al. Prometheus: Inducing fine-grained evalua- tion capability in language models. In The Twelfth International Conference Learning Repre- sentations, 2023.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, and Minjoon Sco. Prometheus: Inducing fine- grained evaluation capability in language models, 2024a. URL https: / /taxivvvivv.. .rrg abs/ 2310 0. 08491
11

---

Published as a conference paper at ICLR 2026
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4333-4355, 2024b.
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang. Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko. Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models, 2025a.
2406. 05761
Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Kiril Gashteovski, Carolin Lawrence, Julia Hockenmaier, Graham Neubig, and Sean Welleck. Scaling evaluation- time compute with reasoning models as process evaluators, 2025b. URL https://arxiv.
Btg/abs/2502 19877
Max Ku, Dongfu Jiang. Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12268-12290, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL http://axxvvvvooo abs 2309. 06180
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. /3787, 2024.
Seongyun Lee, Seungone Kim, Sue Park, Geewook Kim, and Minjoon Seo. Prometheus-vision: Vision-language model as a judge for fine-grained evaluation. In Findings of the Association for Computational Linguistics ACL 2024, pp. 11286-11315, 2024a.
Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, and Minjoon Seo. Prometheus- vision: Vision-language model as judge for fine-grained evaluation, 2024b. URL https:
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.011 01696, 2018.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024a.
//apxiv wrg. 2408... 03326
Haitao Li, Qian Dong. Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lundeas-judges: A comprehensive survey on llm-based evaluation methods, 2024b. URL
https: org/ abs /212. 05579
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 2023.
Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang. Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, et al. Vlrewardbench: A challenging benchmark for vision- language generative reward models. arXiv preprint arXiv:2411.17455 2024c.
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flashi Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00577 2024d.
Hao Liang, Zirong Chen, Hejun Dong, and Wentao Zhang. Evgascore: A fine- grained metric for video question answering data quality evaluation, 2025. URL htthh:://aaaxttt 2411 06908
//t::///rxiii org/ abs/
12

---

Published as a conference paper at ICLR 2026
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out. pp. 74 81, Barcelona, Spain. July 2004, Association for Computational Linguis- tics. URL httpttss:: http:///ccaaatttttogoeo wwt.wog..wwooooo..ooooooo. 1013/
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:240.001291, 2024.
Haotian Liu, Chunyuan Li. Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 10296-26300 2024.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511-2222222 2023.
Ziyang Luo, Haoning Wu, Dongxu Li, Jing Ma, Mohan Kankanhalli, and Junnan Li. Videoau- toarena: An automated arena for evaluating large multimodal models in video analysis through user simulation, 2025. (ttthtts:/// //arxiv/arvvv..///aa wweeiiivorrr/oeeo ///g///aa 241../1281111
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Videogpt+: In- tegrating image and video encoders for enhanced video understanding. arxiv, 2024. URL
https::/aart /arxiv//rxvv.voo org/ /wwp///a/////eo.eee 2406.0000000 09418
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback, 2023. URL
2363.1765I
Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Za- kka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Smolvim: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504. 05299, 2025.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. Howto 100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 2630-2660 2640,226000000 2019.
Eric Mitchell, Joseph J. Noh, Siyan Li, William S. Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher D. Manning. Enhancing self-consistency and performance of pre-trained language models through natural language inference, 2022. URL https://arxiv.org
abbs/2211.. 2211.
Salman Khan Muhammad Maaz, Hanoona Rasheed and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. ArXiv 2306.05424, 2023.
Thong Nguyen, Yi Bin, Junbin Xiao, Leigang Qu, Yicong Li, Jay Zhangjie Wu, Cong-Duy Nguyen, See-Kiong Ng. and Anh Tuan Luu. Video-language understanding: A survey from model architecture, model training, and data perspectives. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 3636-3657, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. (0..18653/v12022 Sindings-acl.2177
www.:022...nnnn----ne indings-acl.227.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguisties, pp- 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis- tics. doi: 10.3115/1073083,1 1073135. URL https: / a wawaacaatttnn ant o
13

---

Published as a conference paper at ICLR 2026
Shu Pu, Yaochen Wang. Dongping Chen, Yuhang Chen, Guohao Wang. Qi Qin. Zhongyi Zhang, Zhiyuan Zhang. Zetong Zhou, Shuang Gong, Yi Gui, Yao Wan. and Philip S. Yu. Judge any- thing: Milm as a judge across any modality, 2025a. URLhttps://axxi.. wwt.aawwwwwwg..//////3.aa 17489
Shu Pu, Yaochen Wang, Dongping Chen, Yuhang Chen, Guohao Wang. Qi Qin, Zhongyi Zhang, Zhiyuam Zhang, Zetong Zhou, Shuang Gong, et al. Judge anything: Mllm as a judge across any modality. arXiv preprint arXiv: /789,, 2025b.
Kate Sanders and Benjamin Van Durme. survey of video datasets for grounded event understand- ing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7314-3377, 2024.
Yaya Shi, Xu Yang, Haiyang Xu, Chunfeng Yuan, Bing Li, Weiming Hu, and Zheng-Jun Zha. Emscore: Evaluating video captioning via coarse-grained and fine- mne-grained embedding matching, 2022. URL ptttps://ariiy wwtp://gggaaaaaaaa/ee 08919
Guijin Son, Dongkeun Yoon, Juyoung Suk, Javier Aula-Blasco, Mano Aslan, Vu Trong Kim, Shayekh Bin Islam, Jaume Prats-Cristià, Lucía Haamo-Bauuelos, and Seungone Kim. Mm-eval: A multilingual meta-evaluation benchmark for Im-aas-a-judge and reward models. arXiv preprint DrXiv:2410.17572 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.0922 09.0888888 2023.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and lann-uagee research, 2020. URL
https: wtt./as/////ooooooooooo/eeeeereo 1904 03493
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models, 2023. httlh: wwwpsiii.rrr//aaae abwpg//bb//22// 11171
Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma. Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Yu Qiao, Yali Wang. and Limin Wang. Internvideo2.5: Empowering video milms with long and rich context modeling. arXiv preprint urXiv:2501. 12366, 2025.
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with verfiiccii,,,,oon, 2023. URL https:
//arxiv.orq///ooo.....ao.... wttpp//gaa////22aa 09561
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma. Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12966-12999 12977, 2025.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding, 2024a. [RLLLtpps///xxiv.orc///////////
15754
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal 1mm. In Forty-first International Conference on Machine Learning, 2024b.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 9777-9786, 2021.
Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models, 2025. URL https::/
wtt.:oog//bbb//1///// 027222
14

---

Published as a conference paper at ICLR 2026
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, Bin Zhang, Xiong Wang. Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025. URL httppp:://axxxiii /wwxivvvvvv..////ia/...////.vtia org/obg///a.... abs//////////// / 2503. 20215
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Kegin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL https: //tp:v/./aaivv.veaa org/ abs 2505 09388
Biao Yang, Bin Wen, Boyang Ding, Changyi Liu, Chenglong Chu, Chengru Song. Chongling Rao, Chuan Yi, Da Li, Dunju Zang, Fan Yang. Guorui Zhou, Guowang Zhang, Han Shen, Hao Peng, Haojie Ding, Hao Wang, Haonan Fan, Hengrui Ju, Jiaming Huang, Jiangxia Cao, Jiankang Chen, Jingyun Hua, Kaibing Chen, Kaiyu Jiang, Kaiyu Tang, Kun Gai, Muhao Wei, Qiang Wang, Ruitao Wang, Sen Na Shengnan Zhang, Siyang Mao, Sui Huang. Tianke Zhang, Tingting Gao. Wei Chen, Wei Yuan, Xiangyu Wu, Xiao Hu, Xingyu Lu, Yi-Fan Zhang, Yiping Yang, Yulong Chen, Zeyi Lu, Zhenhua Wu, Zhixin Ling, Zhuoran Yang. Ziming Li, Di Xu, Haixuan Gao, Hang Li, Jing Wang, Lejian Ren, Qigen Hu, Qianqian Wang, Shiyao Wang. Xinchen Luo, Yan Li, Yuhang Hu, and Zixing Zhang. Kwai keye-vl 1.5 technical report, 2025b. URL https:
orr/ htg//bb//a 2509. 01563
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928, 2023.
Boqiang Zhang, Kehan Li. Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang, Hang Zhang, Xin Li. Peng Jin, Wengi Zhang, Fan Wang. Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video un- derstanding. arXiv preprint 1rXiv:2501.13166, 2025a. URL
2551. 13106
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang. Xin Li. Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video under- standing, 2025b. URL httpt::///// /ttttss:/arxiiie org/ abs/ 22001. 13106
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tume audio-visual lan- guage model for video understanding. arXiv preprint arXiv:2306.00 038808880. 2023. URL https:
02858
Peiyuan Zhang. Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint urXiv:2406. /6852, 2024a.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evalu- ating text generation with bert, 2020. URL htttt:ttt///tttt
httpg//bs/19900 09675
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng. Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024b. URL /tttps//llavaal wwwtaawww.ggthhhbbb.. 10 www.:///ooo////200 -oo wew.aaaava--tiiieeeeeeee wedeetietetttt/eeeeeie
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruc- tion tuning with synthetic data, 2024c. URL https::/arxiv.corm
02713
15

---

Published as a conference paper at ICLR 2026
Jiaxing Zhao, Xihan Wei, and Liefeng Bo. R1-omni: Explainable muni-mlltimodal emotion recog- nition with reinforcement learning, 2025. URL httph: /tttppss//rxxv...o orq/ abs/2503, 05379
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang. Joseph E. Gonzalez, and lon Stoica. Judging llm-as-at im-as-a-judge juddee with mt-bench and chatbot arena, 2023. URLpttts://aaxx orq/
abs/ 220..2206.co0 05685
Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, and Shafiq Joty. Evaluating judges as evaluators: The jetts benchmark of llm-as-judges as test-time scaling evaluators, 2025. URL
https://arae org/ /ws.//50000 15253
A APPENDIX
A.1 RELATED WORK
Evaluation Benchmarks Video understanding models are evaluated on wide range of bench- marks spanning different tasks (Sanders & Van Durme 2024). For captioning, datasets such as MSR-VTT (Xu et al., 2016), VATEX (Wang al. I (020), and HowTol00k (Miech et al. (219)) provide large-scale paired video-text data, with evaluation often relying on reference-based metrics correlation with human judgments (Shi al., 2022). For action recognition, datasets like Activi- tyNet offer a large-scale benchmark covering hundreds of activity categories with temporal annota- tions (Caba Heilbron et al. 2015). For video question answering, datasets such as TVQA (Lei al., 2018) and NEXT-QA (Xiao et al., 2021) require models to integrate visual content with natural language reasoning over semantically complex or temporally extended video segments. In parallel, meta-evaluation benchmarks have been proposed to measure the reliability of evaluators themselves, both in unimodal and multimodal settings. Examples include RewardBench (Lambert et al.) 2024) and MM-EVAL (Son et al. 2024) in the unimodal domain, and multimodal resources such as VA- TEX EVAL (Shi et al. 2022), ) VLRewardBench (Li al. 2024c). Judge Anything (Pu et al. 2025b),
and ALaVA-Critie (Xiong et al. 2025
A.2 DATASET
Table Video duration statistics (in seconds) across evaluation datasets, sorted by number of unique videos. Count indicates unique videos considered after deduplication.
Duration/Datuset
Count
Min
Max
Mean
Median
Remark
VideoJudge-RS-20
9469
2.1
745.4
117.6
114,4
-caee Videoludge preference dataset
MaeohudgeLLLVVAMeettee
3038
5.0
341.8
18.3
9.9
VideoJudge benchmark with LLaVA prompts
Vatex Meaa
2340
1.9
7180.0
167.5
81.4
VATEX evaluation split (multilingual captions)
VideoJudgeVCG-MMttaeee
Evall
499
3.0
238.0
108.0
98.7
VideoJudge benchmark with VCG prompts
280
8.0
297.1
56.0
16.8
Long-context video reusoning benchmark
MibooAutoAreennPPPrreeene.
241
8.0
3291.4 /4
433.6
60.7
Preference pairs from VideoAutoArena
We provide details of the videos along with the nature and source of instruction data in Table 4 The datasets span a wide range of video durations, from short-clip benchmarks to long-form videos extending to several minutes, reflecting the diversity of evaluation settings covered in our work.
Video Description Generation We generate dense video descriptions using two vision-language models: Qwen2.55VV-3325 -nn--L-------ntttttaaaaa and GPT-4o-mini (frames are provided as a sequence of images) to use as proxy during bootstraping process. We sample 2 frames per second (FPS) with maximum of 180 frames for each video. We use temperature of 0.6, with all other generation parameters kept at their default settings. We initially use -o-T--ommnni for description generation due to its strong multimodal description quality; however, to scale annotation under practical budget constraints, the remaining descriptions generated using Qwen2.5-VL-320 1---------B--nstrrue Instruct as cost-effective alterna- tive. Overall, approximately 33% of the descriptions are generated using GPT-4o-min and 67% by Qwen2.5-VL-32B -I-nsuuttttt We provide the prompt for generating dense video description in for Long VideoBench and VatexEval, and GPT--4-mini for the remaining benchmarks to balance quality, cost, and efficiency.
Figure 5 For unimodal evaluation, we use Qwen2.5- VL-72B--
16

---

Published as a conference paper at ICLR 2026
A.3 PROMPTS
In this section, we provide a comprehensive list of prompts that we use in our study.
A.3.1
BOOTSTRAPPING
Dense Video Description Generation in Figure 5
We provide the prompt generate dense video description
Video Description Generation Prompt
You are an assistant specialized in generating concise, factual video descriptions hased solely on provided visual information. produce coherent and continuons description in natural, buman-generated paragraph style that accurately captures all visible elements, settings, and actions. Do not refer to the fact that the information comes from video frames, avoid using bullet points numbered heyond what is explicidy shown in the input. Prioritize clarity, precision, and comprehensive coverage of the visual information.
When given the
introduce any
details, assumptions, interpretatioss
Input:
Using the video content provided, generate detailed and factual video description that strictly reflects the visible content.
Figure 5: Prompt used for generating factual video descriptions from visual content. The prompt is used with both GPT-4o-mini and Qwen2.5-VL-320 ----2.55-V--32-Insttutt with frames provided sequence of images at 2 FPS and maximum of I 80 frames per video.
Response Generation Prompt The prompt used to generate candidate responses is provided in Figure 6
Response Evaluation Prompt The prompt for evaluating candidate responses during the boot-
strapping stage is shown in Figure 6
Response Regeneration from Feedback After the initial round of response generation and eval- uation, measure the difference between the generator's self-assigned rating and the evaluator's rating. This numerical gap is then incorporated into feedback, which guides response regeneration. The corresponding prompt shown in Figure g
A.3.2 TRAINING AND EVALUATION
Pointwise For pointwise evaluation, we prompt the model produce reasoning sequence fol- lowed by a scalar score, as illustrated in Figure
Rubric Generation We employ GPT-4o-mini to construct evaluation rubrics conditioned on the instruction, the video description (serving as a proxy for the video content), and the gold standard response.
We provide example rubrics generated by different models in Table
Pointwise Training, Evaluation, and Rubric Generation We use the prompt in Figure train Videoludge models and to evaluate responses in pointwise setup. The prompt in Figure 12 extends this by enabling models both train and evaluate in a pointwise setting while also generating rubrics at test time. Finally, Figure 13 shows the prompt used to evaluate rubrics produced by different models.
Pairwise training and evaluation. We provide the prompt to train and evaluate models in a pair- wise setup without feedback in Figure 14 and with feedback generation in Figure 15
17

---

video description ground the Published as a conference paper at ICLR 2026
Candidate Response Generation Prompt
You are provided with a detailed video description. gold standard response rated (perfectly accurate, highest quality), and corresponding instruction a video understanding task. This task may include video captioning, video question answering. video instruction following, temporal action localization, any other open-ended video understanding scenario.
progressively lower-quality outputs same instruction. Each generated response should corresund to quality rating from 4 where Rating 5 the provided gold standard and Rutings 4 through represent decrensing quality.
generate additional responses
As rating decreases, the respouses should reflect increusing levels degradation, including hallucinations, omissions, irrelevant information, logical inconsistencies, grammatical issues. All generated responses to the gold standard and maintain the expected task format (o.g.. caption, answer, instruction). Do not simply truncate the gold response simulate realistic and meaningful degradation quality across correctness of the response content.
similar length
Task:
Generate lour degraded responses corresponding to quality ratings 4 through 1, based on the gold standard response (Rating S). Follow the rating guidelines strictly.
Rating Guidelines:
Rating 4:
Mostly accurate with minor issues. Preserves the key meaning with small lapses in detail or precision.
RRtinng Partially correct. Conveys the general idea includes noticcable errors, omissions, or misinterpretations.
Rating 2: Poor aligmment. Contains serious flaws and only loosely relates to the instruction or video.
Rating I: Unrelated or incorrec.. Fails to reflect the video or follow the instraction meaningfully.
Example:
Instruction: "Can you give summary Video Description: "A large group of people participating in acrobics class inside spacious indoor venue...
video content?"
Gold Standard Response (Rating 5); "The video is about group of people doing stop excrcise class, with some confusion smong participants. The locus
the main
who stops the end and walks towards the camera."
Ruting 4: "The video shows a group of people a step exercise class, with some participants appearing confused during the workout. The woman stops exercising the Ruting 3: "The video features people duing an aerobics class with stepping movemnnts and some coondination issues. There's ' woman seems
toward the camera."
leading the group,
approaches
some point during session."
Rating "The vidco shows several people what appears be dance or fitness routine taking place indoors. woman in the group walks around and comes closer to where the video is being filmed, while others continue moving in the background." Ruting "The video depicts cooking demonstration where chel weuring warkout clothes explains different healthy recipes a small audience in what looks like kitchen or dining area, with people standing around tables."
Output Format:
Your output must a valid ISON object with exactly the following keys: "rat "rat ing.3", "rat ing 2 "rat ing_1* P, Each value should the generated response corresponding that quality rating. Do not include any additional commentary, formatting, or explanation cutside the JSON object.
Input:
Instruction:
('nstrucction
Video Description:
sriido_descripter
Gold Standard Response (Rating 5);
(gold da_ttttaaaa andaaddd response)
Output (according the JSON scheuaa):
"rating_4"" [Generazed response; "
*rating_3" *Generated response:" "rating_2"* [Generated responae,
"rating_11 "Generated response
Figure 6: LuMdge-judge prompt for generating degraded responses at different quality levels based gold standard.
18

---

Published as a conference paper at ICLR 2026
Candidate Response Evaluation Prompt
You are provided with a detailed video description, instruction, gold standard response rated (perfectly accurate, highest quality). and set of candidate responses for a video understanding task. This task may include video captioning, question answering, instruction following. temporal action localization, other open-ended video understanding scenario.
Task:
Your task is to evaluate each candidate response and ussign evaluation rating from 1 to While the gold standard response can reference
response looks like, your evaluation should primarily focus how each candidate response fultills the task defined by the instruction, givon the provided video description. Focus reasoning identifying what is incorrect, missing, or misleading in the response itself.
Evaluation Steps:
1. Read the instruction carefully understand the intended task.
2. Refer to the gold standard response (rated 5) reference for correctness and completeness.
provided video description
factual basis evaluating responses.
4. For each candidate response: identify inaccuracies, omissions, ballucinations, irrelevant content, and evaluate align- ment with the instruction and video description.
5. Assign an integer score from 1-4 to each camdidate response.
6. Provide reasoning for each rating, focusing on specific aspects that affect alignment with instruction and video de- scription.
Rating Guidelines for Evaluation:
Kating 4: Mostly accurate with minor issues. Preserves key meaning with small lapses.
Rating 3: Partially correct. General idea cunveyed but with noticeable emveviomissions.
Rating 2: Poor alignment. Serious flaws, loosely related to task or video.
RRating Unrelated incorrect. Fails reflect video follow the instruction.
Example:
Instruction:
"Can give hrief summary of the video content?"
"A large group of people are participating in an acrobics class insäde a spacious indoor venue..." Gold Standard Response (Rating 5): "The video is about group of people doing step exercise class, with confusion among the participants. The focus on the main woman, who stops at the end and walks towards the camera." Rating "The vidzo shows group people workout The main woman stops exercising at the end and moves toward the camera." Rating
Video Description:
step exercise class,
some participants appearing confused during the
stepping movements some coondination issues. There's woman who seems to he leading the group, and she approaches the camem at some point during the session."
video features people doing
Ruting "The video shows several people what appears be a dance or fitness routine taking place indours, woman in the group walks around and comes closer where the video is heing filmed, while others continue maving in the hackgrond." Rating I: "The vidco depicts cooking demonstration where chef wearing workout clothes explains different healthy recipes to small
looks
dining with people standing
tables."
Output Format:
Your final output must follow the exact JSON schema provided. Do not include additional formatting, comments, text outside ISON object.
Input:
Instruction:
(*rstrruct:
Video Description:
(video_deds crroo__derrrreeee ptt
Gold Standard Response (Rating 5);
(gold_es settandarddrrseeeee
Generated Responses:
[genezated_ respddeeeppeeeeeeee
Output (according to the JSON schema)):
[output__Formel
Figure 7: LLM-as-judge prompt for evaluating candidate responses against a gold standard using a 1-4 quality scale.
A.4
BOOSTRAPPING PROCESS
A.5
EVALUATION DATA
Representative examples of the generated data are shown in Table Figure ?? further illustrates the evaluation results, presenting the BERTScore with the gold-standard response as reference along-
19

---

Published as a conference paper at ICLR 2026
Candidate Response Regeneration with Feedback Prompt
You are provided with detailed video description, an instruction, a gold standard response for video inderstanding task (such as captioning, question auswering. instruction following. event description), and set of generated respouses. cach intended match specific quality rating from 4 to However, generated responses evaluated and found deviate from their inteaded quality levels.
Task:
responses only for the evaluation rating is greater than zero (ic.. lint ended_rr a Lin eva..ral ing > 0), For each such cntry. produce revised response that strictly conforms to the intended quality rating, based on definitions in the rating guidelines below. Use the video description the factual grounding for determining what is valid. Adjust the response that evaluation rating would now exactly match the intended rating.
Your regenerate
absolute difference between intended rating
If the eval, _rating is higher than the intended rating, degrade the response by introducing errors such as hallucinations, factual distortions, vagueness, grammar issues. the eval rating is lower than the intended rating, improve the response by clarifying actions, reducing errees, restoring key context from the video description.
Rating Guidelines:
Ruting Mostly accurate minor Preserves key meaning with small lapses.
Rating 3: Partially correct. Conveys the general idea but with noticeable ermrslomissions.
Rating Poor alignment. Serious flaws, loosely related to or vidco.
Rating I: Unrelated incorrect. Fails reflect the video follow the instruction.
Example:
Instruction: "Caan you give brief summary of the video content?"
aerobics inside spacious indoor venuie.." Gold Standard Response (Rating 5): "The video is ahout group of people doing step exercise class, with some confusion among participants. The focus on the main woman, who stops at the end and walks towards the camera." Rating 4: "The video shows group of people
Video Description: large group of people participating
step exercise class, with workout. The main woman stops exercising at the end and moves toward the camera."
participants appearing confused during the
Rating "The video features people doing acrobics class stepping movements some coordination issues. There's woman who seemss to be leading the group, and she approaches the carnert al some point during the sessiun." Rating
appears be dance or group walks around and comes closer to where the video is being filmed, while others continue moving in the background"
video shows several people
routine taking place indoors. woman the
Rating 1: "The video depicts cooking demonstration where chef wearing workout clothes explains different healthy recipes to small authence in what looks like kitchen dining with peuple standing around tubles."
Output Format:
Your final output must he a valid ISON object. For each entry where lintanded. rat rating eval irotingl > 0. include key "rating {1)" and update value with newly revised response adheres precisely to intended qual- ity rating. If the eval watiraatingi equal to the intended do not modify include that entr. Ouly output the revised responses.
Input:
Instruction:
(instruct: on
Video Description:
Uvideo_descriptten
Gold Standard Response (Rating 5);
[gold_al Leeddddddeespppeeeee
Feedback Data (ISON):
[Leedback_datee
Output (according to the JSON schema)::
[output__ format)
Figure 8: LLMMaasjudge prompt for regenerating responses to align with intended quality ratings.
side the absolute VQAScore across different ratings. For BERTScore, we use the evaluate Flibrary with its default configuration, which uses roberta-larg model. The VQAScore is computed using
20

---

errons Published as a conference paper at ICLR 2026
Evaluating Model-Generated Responses Prompt
You are provided with video, corresponding instruction. and response generated by model. The instruction defines video understanding task, which may take any form incloding not limited open-ended question answering. captioning. instruction following, temporal reasoning, multi-step inference grounded in the video. These tasks open-ended and often require complex or muanced reasoning over the visual and temporal content.
Your task is evaluute the quality of the response, considering how well it satisfies the task defined by the instruction, based on correctness, relevance, completeness,
content grounding of the response.
judgment and shoold be
Task:
For each response:
Assess well
context video.
Consider whether the response is accurate, relevant, complete, and grounded in the video.
Provide a rationale explaining the overall quality and alignment response with instruction <thinking> </ /thinkkng>
Outpul a score from (worst) to 5 (best) indicating the overall quality inside <score> </ </corr>>.
Rating Guidelines:
Rating Fully accurate, complete, well grounded. response precisely
instruction
omissions.
RRtting Mostly accurate with minor issues. The response preserves key meaning relevance, with only small lapses
detail or precision.
Rating Purtially correct. The respanse conveys the general idea bui includes noticeable emons, omissions, or misinterprelaaionss. and only loesely relates instruction video. Rating I: Unrelated or incurrect The response fails to reflect the video or follow the instruction meaningfully.
Rating aligmment. The response
Input:
Instruction:
(instructionn]
Response:
(response]
Output (Strict Format);
<think (your reasoning explanation rating) //thinking> <score> (integer score from to 5) s/score
Figure 9: LLM-as-judge prompt for evaluating model-generated responses to video understanding tasks.
T2V Metric with the video model Qwenn.5-VL-7B-nnstrue For BLEU, the sacreblet implementation with default settings.
A.6 HUMAN EVALUATION
A.6.1 PAIRWISE
A.7 HYPERPARAMETERS
In Table provide a detailed list of hyperparameters we use in our experiments. We use Qwen2.5- vL as training framework vLLM (Kwon al. 2023) for evaluation. We keep all other parameters as default until stated otherwise.
A.8 RESULTS
Rubric Generation
httees//4ihuub.cco///pot////oo0 Instrutt
21

---

Published as a conference paper at ICLR 2026
Instruction-Specita Rubries Generation Prompt
You are provided with video (or a detailed description). conreponning instruction, and reference response. The instraction detines a video understanding task. which may invoive open-ended question answering, captioning. instruction following. temporal reasoning, multi-step inference grounded in the video. Such tasks open-ended and often require complex nuanced reasoning over the visual and tomporal conteut.
Your task is to generate an instruction-preiifi evaluation rubric. mubric will
rate quality any response to instruction on scale. The reference response is provided only to help you define what perfect answer (Rating 5) looks like, but during evaluation the rubric must stand slone evaluators will not given the reference response.
Important Note:
the provided video description you watching video yourself.
Pay close attention to all details in the video description and the reference response. Use these construct procise, example-specific scoring rubrics can be applied withour needing reference later.
TTisk:
For given video, instruction, reference response:
Generate a single 1-5 scoring rubric tallrred to this instruction.
Each score level through 5) include
example specific description of a response would like. Use the reference response anchor what counts as a "5 (Excellent)" answer.
Ensure that the rubric self-contained so that it can be applicd without access the reference response.
Ruting Guidelines:
Completely wrong. inrelevant, missing. Rating (Poor]: Vague, incomplete, or largely inaccurate with minimnal grounding.
Rating (Very
Rating (Fairi: Partially correct, captures some aspects
key
Rating (Good); Mostly correct and grounded, covers most important aspects but not fully comprehensive.
Rating (Excellent); Fully correct, detailed, coherent, and well-grounded aligned with the reference response.
Input:
Video Content detailed description):
(video_gr ceeeoddderrpptioo..
Instruction:
(instruction
Gold Standard Reference Response:
(reference responsee__pps
Output (Strict Format):
** Rubric 1Scale 1451 s*a
>ARating
(Very cAs [instruction specific description]
**Rating (Pcor) s* "Laszructionsspeclee description]
>aRatang
(Fair):*x
deseription]
RnRating
(Good) instrructoonsspen desorigtion]
ReRting
(Excellent):* (Enstruction-spent description)
Figure 10: LLM-as-judge prompt for generating instruction-speciif rubrics for video understanding tasks.
Human Evaluation of Generated Rubrics We conduct a human evaluation study on Ama- zon Mechanical Turk to compare the quality of rubrics generated for video-instruction-rrspom triplets. The evaluation dataset consists of 300 randomly sampled rubrics for 6 models, including VideoJudge-3B (which we train). For each example, we generated rubrics using VideoJudge as one option and compared them against rubrics produced by five other models: GPT-4-mini, Qwen-3B, Qwen-7B, Qwen-32B, and Qwen-72B. Annotators were given the video along with its description, the instructions the models, reference response illustrating good answer, and candidate rubrics (A and B), each defined on a 1-5 scale. Their task was to compare the two rubrics and select the they considered more effective for evaluating Al-generated responses to the given instruction. Each rubric pair was assessed independently by three annotators. The full annotation framework, including the instructions provided and a representative Human Intelligence Task (HIT) example, is shown in Figure 18
Decoding Temperature
We provide other metrics for decoding at different temperature Ta-
ble 19
22

---

Published as a conference paper at ICLR 2026
Table 5: Rubrics generated by different models for the instruction "What is the man wearing while climbing the rock?".
Instruction
What the man wearing while climbing the rock?
Reference Response
The man is wearing gney west while climbing the rock.
Model
Rubric
Qwen2.5-VL-3B
*Rubric (Scale 1-5):hs , **Rating (Very Poor): The respouse does address instruction provides imelevant information, **Rating 2 (Poor):** The response partially addresses the instruction but contains significant inaccuracies or irrelevant details RRainng 3 (Fair):** The response addresses instruc- tion but lacks detail or is not entirely accurate. **Rating 4 (Good);** The response addresses the instruction adoquately and provides relevant information, but there are some inaccuracies or omissions. "Rating 5 (Ex- celleni):** 46 'The response adiresses the instraction thorogghly and accuralely, providing all necessary details and information.
Qwen2.5-VI.-7B
Ruubric (Scale 1-5):*. - **Rating I (Very Poor):** response does not mention the man climbing the rock and instead describes a different scene or activity - **Raiing 2 (Poor):** The response mentions the man climbing the rock but includes irrelvvnnt details or descriptions that do not match the video content. ""Rating 3 (Fair):** 40 The response accuralely describes the man climbing the ruck but may include minor inaccuracies omissions. RaRinng 4 (Good); The response accurately deserihes the man climbing the mek and includes relevant details, but there might some minur insecuracies or omissions. *RRaiing (Excellent):* The response accurately descrihes the man climbing the mek, includes all relevant details, and does not contain inaccuracies or omissions.
VL. VL.32B
*-RRhhric (Scale 1 1-5 _ **Raiing (Very Four). The response is completely untelated the video content and does not mention any clothing or climhing gear. It may describe an entirely different scene or activity. **Rating (Poor):*s The response mentions some clothing or gear,
incurrect or irrelevant to the
video. For example, it might describe bright red helmet or a heach setting, which are not present in the vidco. **Rating (Fair):** The response partially describes the clothing or gear worn by the man climbing the rock, but it is incomplete or contains minor inaccuraciss. For example, might mention shirt and shorts but fail to note the climbing harness or shoes. **Raiing 4 (Good):** The response accurately describes most of the clothing and year shoes, However, it may miss minor details or fail to mention the climbing rope. **Rating 5 (Excellent):** The response provides complete and accurate description all the clothing and gear worn by the man climbing the mokk, including a shirt, shorts, climhing hameess, shoes, and the climhing rope. It is precise and fully grounded in the video content.
the man climbing the rock, incluting shirt, shorts, climbing hamess, and
Qwen2.5 VIL-72B
R-Ruhrii (Scale 1-5): .*Rating (Very Poor):* The response does not mention any clothing omeettent the man is wearing while climbing the rock. fails to describe the climbing activity and instead provides irrel- evant details about beach scene. **Rating (Poor):** The response mentions some clathing or equipment but includes significant inaccumcies or irrelevant details. It may deseribe the wrong nctivity or setting, such as sitting on a rock or being on a beach. RRtinn 3 (Fair):** ** The response partially deseribes the man's attire while climbing but misses key details. It may mention some correct elements but lacks completeness accuracy in describing the climhing gear and selting. +*Rating (Goud):** The response accurately describes the man's attire, inchuding tank top, shorts, and climbing shoes, while climbing the rock. It may also mention the climbing harness and rope but could be slightly lacking in detail or context. **Rating 5 (Excellent): The response comprehensively and accurately describes the man's attire, including a tank top, shorts, elimbing shoes, a harness, and rope, while climbing the rock. It provides a clear and detailed account of the climbing activity and setting. matching the video content perfectly.
GPT-4o-mini
R*Ruhric (Scale 1-5):*. *Rating I (Very Poor):** The response completely fails address the instruction, providing irrelevant information about the climbing scene or omitting any mention of the man's 's clothing.
*Rating 2 (Paor):** The response offers vague or inaccurate information about man's clothing such
general description (c.g.. "he is wearing something") but fails specify or properly identify the clothing item. ** "Rating 3 (Fair):** The response provides a partially correct description of the man's 's's clothing but may miss critical details, such the specilic color or type clothing, leading
incomplete answer (e.g.. "the is wearing a shirt"). *Raaing (Good):"* The response accurately identifies the man's is clothing with most important details provided, such the color and type of the elothing, but might lack or minor dctails (c.g. "the man is wearing grey tank top") - -* **Rating 5 (Excellent)):** The response is fully coreet, detailed, and coherent, elearly stating that man is wearing grey tank top while climbing rock, and aligns perfectly with the instruction.
additional context
VideoJudge 3B
Ruhrii (Scale 1-5):40- **Rating 1 (Very Poor):* The response completely misses the question about what the man wearing while climbing, providing irelevvnt information no answer at all. =Roting (Poor):** The respouse gives a vague or incomplete answer regarding the man's 's clothing. such as mcutioning is "dressed" or "wearing something." but lacks specific details or accurncy. **Rating (Fair):** + The re- sponse provides some correct information about the man's elothing but misses key details, such as stating he "wearing shorts" without specitying the color or type of clothing. *"Raing (Goodd):** The response accu- rately descrihes the man's may lack additional context or detail about his overall appearance. fully and accuratcly deseribes the man's clothing while climbing, elearly stating "the man wearing tank top and shorts." and may also include additional relevant details about his appearance or actions.
while climbing. menliuning specific items like lank top and shorts," but *Raainn (Fxceelent)) The responce
Number of Frames
We provide other metrics for max frames ablation in Figure 20
23

---

Published as a conference paper at ICLR 2026
Evaluating Model-Generated Responses Prompt
You are provided with video, corresponding instruction, and a response generated by model. The instruction defines video understanding task, which may take any form incloding not limited open-ended question answering. captioning, instruction following. temporal reasoning. multi-step inference grounded in the video. These tasks open-ended and oflen require complex or nunnced reasoning over the visual and lemporal content.
Your task is to evaluate the quality of the response, considering how well it satisfies the task delined by the instruction, based on judyment and should be based overall correctness, relevance, completeness,
content
grounding of the response.
Task:
For each response:
Assess how well it addresses the task instraction the context the video.
Consider whether the response is accurale, relevani, complete, und grounded in the video. quality and alignment
Provide rationale explaining
response
<t hirkiigg?
//thinking
Output a score from (worst) to 5 (best) indicating the overall quality inside <acore> s/oore.
Ruting Guidelines:
Rating 5: Fully accurate, complete, and well-grounded. The response precisely follows instruction with notable omissions.
Rating Mostly accurate minor
response preserves key meaning relevance, only lapses
detail or precision.
|aating Partially correct. The response conveys the general idea but includes noticeable errors omissions, or misinterpreiations.
Rating 2: Pour aligument. The response has serious flaws and only lousely relates the instruction video.
Rating I: Unrelated or incorrect. The response fails to reflect the video or follow the instraction meaningfully.
Input:
Instruction:
(instruction]
Response:
(xesponse]
Output (Strict Format):
<thinking>> (your reasoning anation (tthiinki//htikingg <score> Hinteger acore from Lo 5; s//cocc
Figure 11: LLM-as-judge prompt for evaluating model-generated responses to video understanding tasks.
Algorithm 1: Bootstrapping Training Data with Self-Refinement
Input: Video v, video description ö, instruction x, gold response y", generator G, evaluator E, threshold a, max iterations T
Output: Bootstrapped dataset
Initialize D < {(e, y" y","""". NN) // gold response with max rating
for r {1, (1....,
do
yo + G(Pgen |||>> rr // initial generation.
for (O...., do
// feedback and refinement
f. f + ||
if
5 then
+ DU ((0,0. x. (..))
break
else
Me41 G(Prer G(pre//0/alll 115
return D
24

---

Published as a conference paper at ICLR 2026
Rubric Generation and Response Evaluation Prompt
You are provided with video, corresponding instruction. and a response generated by model. The instruction delines a video understanding task, which may take any form incloding not limited open-ended question answering. captioning, instruction following. temporal reasoning. multi-step inference grounded in the video. These tasks open-ended and oflen require complex or nunnced reasoning over visual and lemporal content.
Your task has two parts:
Part 1 Rubric Generation:
Generate instruction-speclfic evaluation rubric on scale for example.
"The rubric must deline what Rating through Rating response would look like in the context of this exact video and instruction.
rubric should tailored to this instruction and grounded in the video content, generic.
Each rating should describe conorete aspects the vidco/inttmction that would or would not appear in response at that level. rubric should self-contained re lerenzes.
be applied any response
instruction, without needing additional
Output your rubric within <rubric></rabbet
Part 2 - Response Evaluation:
Using the rubric
generated, evaluate the provided response. Assess how well it addresses the task in the instruction in the centext of the video.
complete, and grounded the Provide a hrief rationale explaining the overall quality and aligament of the response with the instraction inside <thinking> /tthinking>.
Consider whether response accurate,
Oulput from (worst) to 5 (best) indicating the overall quality insäde <azore> </score>.
Input:
Instruction:
cunstruction
Response:
[response]
Output (Strict Format);
<zubric>
Rubrii IScale 1(5)
>xRating (Very Poor): :ax [=nae:rctionn-paciii description referencing video detaila
.xRating (Poor) inntrrutiionsspeee description referencing video detaila]
description rafercncing video detaila] **Rating (Good) ['netruct ton-spectt deseription referencing ng video detafla]
**Rating (Fair):** [inseruction-spedd
>xRating (Excellent) tax [nstruction specifis description reCerencing video detaila)
</rubric>
<thinking> (your reasoning and explanation for the <score> [ (integer acore from to 1//soree
Figure 12: LL-M-as-judge prompt for generating rubrics and evaluating responses to video under- standing tasks.
25

---

Published as a conference paper at ICLR 2026
Rubric Comparison Prompt
You ure provdeed with viden, a conesponding instruction, and reference response The instructtonn defines video understanding task, which may take any form including limited open-ended question answering. captioning, instruction following, temporal reasoning. or multi-step inference grounded in the video. These tasks are open-eaded and require complex or nuanced reasoning over visual temporal
are also provided two rubrics (Rubric Rubric B), each generated by model. These rubrics are intended evaluate responses
instruction 1-5 scale.
Your task decide which rubric better for ovaluating responses the instruction. This bolistic judgment and shoold be based on the rubric's 's specificity. clarity, coverage. and usefulness for evaluation.
Task:
For cach pair of rubrics:
Assess which rubric hetter reflects the instruction and video (instrationn cetemaccioo-scccfciittt
Consider whether the rubric self-contained and usable without the reference response. rating
Evaluste clarity, distinctness,
logical progression Judge which rubric provides better coverage of key uspects required to evaluate respomess.
brief rationale explaining your choice inside <th: mking>> </thinking Output the preferred rubric A inside <answer> </anawer>.
Guidelines for Comparison:
Prefer the rubric that is more specific to the given instruction and vidco.
well-structured, Prefer the rubric that captures all important uspects of what makes good or bad response. strong, choose one that slightly more precise comprehensive. Do not outpur tic always select either A B.
easy apply.
Input:
Instruction:
(imstruction
Reference Response:
(xef_respcnse)
Rubric A:
(rubric_s)
Rubric B:
(rubric_b)
Output (Strict Format):
<thinking> I (your reasoning and explanation for why one rubric is better) //thinkiggs canswer> (AA oz </aswee
Figure 13: LLLM-as-judge prompt for pairwise comparison of two instruction-specifi rubries.
26

---

Published as a conference paper at ICLR 2026
Pairwise Response Comparison Prompt
You provided with viden, comresonning instruction, and candidate responses yenerated models. The instruction defines video understanding task, which may involve open-ended question answering captioning. instruction following. temporal reasoning. multi-step inference grounded the video. Such open-ended and require complex or nuanced reasoning over the visual and temporal content.
Your task is compare the responses and decide which better addresses the instruction, based the content of the video. Make a holistic judgment that considers correciness, relevance, completeness, and grounding.
Task:
For given pair of responses:
Judge which response better addresses the instruction in the context of the video.
Consider whether each response is accurate, relevant, complete, and grounded video.
Output only wrapped strictly inside <answer><</ answer> lags.
Evaluation Guidelines:
Accuracy: Prefer responses that are factually correct and consistent with the video.
Relevance: responses directly answer instruction
digression.
Completenes: Prefer responses that capture all key aspects needed for a full answer.
Grounding: Prefer responses clearly supported by the video, avoiding ballucinations. response contains ballucinations, irrelevant content, amissions, prefer the other. both responses are strong, choose the one that more precise and detailed. flawed.
both responses weak, choose the
Input:
Instruction:
(inst runtionntion]
Response A:
(response_s]
Response
(response_h)
Output (Strict Format):
(f
Figure 14: LLM-as-judge prompt for pairwise comparison cenerl-generated responses to video understanding tasks.
27

---

Published as a conference paper at ICLR 2026
Pairwise Response Evaluation with Reasoning Prompt
You provided with video, corresponding instruction, and candidate responses generated models. The instruction defines video understanding task, which may involve open-euded question answering, captioning, instruction following. temporal reasoning. multi-step inference grounded the video. Such are open-ended and require complex or nuanced reasoning over the visual and temporal content.
Your task is compare the two responses and decide which one better addresses the instraction, based on the content of the video. Make a holistic judgment that considers correctness, relevance, completeness, and grounding.
Task:
For given pair of responses:
First, evaluate Response A (strengths and weaknesies).
Then, evaluate Response (strengths weaknesses).
Finally, compare them and decide which better satisfies the instruction the context of the video.
Ground your reasoning in the video content treat as if you are directly watching the video.
Write the reasoning inside <think ingo e/thinkiin
After the reasoning. output only & or wrapped strictly inside <answe>></ <atseee>></nnwee>>>>> answer>> tags.
Evaluation Guidelines:
Accuracy: Prefer resonses are factually correct and consistent
video.
Relevance: Prefer responses that directly answer the instruction without digression.
Completenes: Prefer responses that cupture all key aspects needed for a full unswer.
Grounding: Prefer responses clearly supported
video, avoiding hallucinations.
If one response contains ballucinations, irrelevant content, or onissions, prefer the other. more precise detailed.
both responses strong. choose the If both responses are weak, choose the one that less flawed.
Input:
Instruction:
(instruction))
Response
[response_a]
Response B:
(response_b]
Output (Strict Format):
<thinking>
([Bvaluate Responae -> Evaluate Response B -> Compare them concisely, explaining why one better, grounded in video. </thinking>
<answer> (A_or_ </answer>>
Figure 15: L.L--as-judge prompt for pairwise evaluation of responses with explicit stepwise rea- soning.
28

---

Published as a conference paper at ICLR 2026
VQAScore
0.20
VQAScore (1)
0.15
0.10
0.05
0.00
5
4
3 Rating
2
1
Absolute VQAScores across specified ratings (1-5). Higher-rated responses achieve higher scores, showing alignment between VQA score and specified ratings.
Figure 16:
Wn
Loss
Win
Loss
$1.0%
Qwen-728
71.5%
vs Qwen-728
vs Qwen-328
90.0%
Qwen-32B
62.3%
37.7%
GPF-4o-mini
93.1%
7.3%
Qwen-3B
63.0%%
18.0%
Qwen-7B
96.3%
3.7%
86.0%
14.0%
98.3%
100
20
40
10
in
100
Videojudge-38 Win Rate
Videojudge-38 Win Rate
(a) Majority vote with 3 annotators
(b) LLM-as-Judge based evaluation
Figure 17: Win rate of VideoJudge-3B compared other models under two evaluation settings.
17a: majority voting of human annotator, where the most common annotators' choice determines the label. 17b: LLM-as-Judge preference with deterministic decoding = 0). Across both settings, VideoJJudge-23 consistently produces high-quality rubrics and achieves performance competitive with, or surpassing, models up to x larger, including proprietary systems such as GPT-4o-mini.
29

---

Published as a conference paper at ICLR 2026
Table 6: Representative examples of video frames paired with instructions and bootstrapped re- sponses at different rating levels (RI-R5) generated by our pipeline.
Video Frames
Instruction and Responses
Instruction: What the mam wearing while ironing the dress shirt?
white building from outside. the inside ballet studio where children practice, girl speaking from inside well-lit stadio, children walking through the school, boy and girl dancing in a studio as other children watch.
R5:
The vidco
R4: The video features exterior view of grand building. followed by seenes insido ballet studio where young dancers practice. There are moments showing girl speaking and other children walking through the school, along with and girl performing ballet moves while others observe.
R3: The video features a beautiful building the start, transitions ballet class where children are practicing their routines. A girl talks one point, and there are some glimpses kids walking around.
focus
hallet practice, with some
children watching and learning.
The video shows building and then cuts to some kids who are doing ballet in a studio. There girl who talks brielly, and kids are just wandering around. The dancing is mixed with scenes children doing other activities, which makes hard tell what's happening.
R2:
RI: The video features a besutiful building at the start, then transitions intoo ballet class where children practicing their routines. girl talks point, glimpses of kids walking around. The main focus is on the ballet practice, with some children watching and learning.
some
Instruction: Can you describe in detail what happens in the vidoo?
vidco depicts ship the sea during the daytime, with hill visible in the distance al the shore. The ship dangerously tilted towards the left and appears to be on verge capsizing. while its be seen approaching the ship in motorhoar. The video ends with the camera panning around the motorboal and the surrounding area. There are no visible signs of distress ar emergency response. Overall, the video captures dramatic and potentially hazardous situation on the high seas.
R5:
fully deployed. Another group people can
R4: The video shows sailing yacht navigating through rough waters during with distant coastline visible. The yacht is leaning significantly to ese side, giving impression that it might capsize, and the yacht, but there no signs of distres. The seene captures an adveaturous sailing experience in open waters.
up. motorbost approaches
R3: The video features yacht sailing the ocean sunny day, with hills visible far in the bac background. The yachi leaning to ome side, which could look dangerous. All fully deployed,
thrilling experience. There nu immediate danger, and the everall atmosphere leels calm despite the tilt of the yachi, leisurely outing rather
appears
making it
emergency.
R2: The video presents a hoat moving across the water, and it's bright outside. There some distant landforms, possibly hills. bost seems leaning, which might suggest it's having trouble. A small hoat is around, but it's unclear what the situation The video feels like it's capturing sailing adventure.
RI: The video shows large ship flonting calm waters, with people enjoying pianic on the dock. In the background, thore's is a beautiful sunset, and the ship appears to be stationary. focus people laughing eating, signs of sailing any movement.
30

---

Published as a conference paper at ICLR 2026
Table 7: Pairwise human evaluation results across two annotators. The table reports overall inter- annotator agreement and Cohen's Kappa as measures of reliability. Preference distributions show the proportion times each annotator selected response b versus response a, indicating a slight bias toward b. Correctness is computed as the fraction of instances where the annotator's preferred response matches the higher-rated (gold) response, with both annotators showing high accuracy over 250 samples each. The last two rows capture error analysis: in 11 cases (4.4%), both annotators agreed on an incorrect answer, while in 13 cases (5.2%) the annotators disagreed on correct answer, highlighting residual uncertainty.
Metric
Value
Agreement
94.80
Cohen's 's Kappa
89.54
Annotatorl Preference (b a)
53.2 46.8
Annotator2 Preference (b aa)
54.4 45.6
Annotator Correctness
92.40 (250 samples)
Annotator2 Conrectness
93.60 (250 samples)
Both agreed on wrong answers
11 / 250 (4.4)
Both disagreed on correct answers
13 250 (5.2)
Table 8: Comparison of annotator decisions across instruction-ressonse pairs.
Instruction
Response
Response
What is huppening in the videa?
The video shows couple dancing out- with some people watching them. Children are playing in the background, and there might
In the video, a couple is moving around A A looks like backyard gather- ing. There are kids playing, and it seems like a party. A few adults are standing around, hut it's hard to what's really hoppening.
someone cooking food. Another couple appears later, hut the details are a bit unclear.
Can you describe the
camoutlage shirt seen doing nail care for woman. He appears to be using some kind of prod- uct, and there people sitting nearby. The lighting seems good, and the focus the nails, there are moments where it shifts away from the main ac- tion.
video, a man
The video features a casual setting where person doing something with nails. There is man in camouflage shirt, and it looks like he applying of treatment. Other people are around, the
video in detail?
kind
about the process are quite vague and unclear.
Can you deseribe competition the man is participating in!
man participating sports competition where he throws round object in circular area. He al- tempts throw long distance, and there are some spectators watching, but the focus the throwing tochnique not very clear, and the details about the event vague.
video, man seen throwing discus in competition. Hc spins gain speed and then throws it. The footage captures him from different perspectives, and there are some people in the crowd, not mony details about them.
What is woman wear- ing and what is behind her?
has on green shirt, and there seems to be clothing item behind not specific. bright, and there might be some products nearby. but details are unclear.
wearing green top, behind ber is a denim jucket. She is in well. with window, are some products on table, but it's not clear what specific items they are.
The
31

---

Published as a conference paper at ICLR 2026
Table 9: Training and evaluation hyperparameters.
Hyperparameter
Value
Training
Learning rate
2c-7
Batch size (per device)
16
Gradient accumulation steps
Num. train epochs
Warmup ratio
0.03
Weight decay
Max grad norm
1.0
LR scheduler
Cosine
Precision
b6loat16
Max sequence length
128,000
Video max frames
60
Video max frame pixels
25,088
Vidco min frame pixels Gradient checkpointing tune_mm, unnemmm_vsiion tune.mi mom mmmmlp tune-mm.lln
3,136
True
False
True
True
Evaluation
wew.new.tokenn.
1024
fps
frames
180
max pixels
20480 x 28 28
min pixels
16 x 28 x 28
Tale's
Rorpes Veli
Figure 18: Example of the Human Evaluation MTurk Interface. Annotators were provided with video, its description, the instruction, reference response, and candidate rubrics. They com- pared the rubrics and selected the one they considered more effective for evaluating Al-generated responses.
32

---

Published as a conference paper at ICLR 2026
videojudge
Videojudge
Videojudge
2.0
1.1
0.7
1.0
Pearson
0.6
0.9
1.4
0.8
0.5
0.4
0.2
0.8
1.0
1.0
0.0
0.4
1/0
0.6
0.2
0.4
0.8
1.0
Temperature
Temperature
Temperature
(a) Pcarson
(b) RMSE
(c) MAE
Figure 19: Effect of decoding temperature on pointwise judgment reliability for the base Qwen2.5- VL-3B and the Videoludge-33B model, evaluated using Pearson correlation, RMSE, and MAE. As temperature increases, the base model shows steady degradation in correlation and higher error rates, indicating sensitivity to stochastic decoding. In contrast, the VideoJudge model remains stable and even improves at moderate to high temperatures, achieving higher correlation and lower error This suggests that rubric-guided training yields substantially more temperature-robut and consistent judgments under non-deterministic decoding.
Videojudge Yoe
Base
Videojudge
Base
Videojutge
Base
0.7
0.6
1.50
1.29
ANSE
MAE
0.5
1.35
0.50
1.20
0.75
0.3
120 180 240 200
360 420
abo
sie
60
120
180
240 non
Mi sōn Anp sia
150
1no 240 nön ni
450 4no sio
Max Frames
Max Frames
Max Frames
(a) Training: Effect of varying maxfr
aness ames (30-500) during training while fixing evaluation at 180 frames.
Base Videojudge
Base
Videojudge
Base
Videojudge
1.20
1.61
1.38
0.60
1.50
Perront
1.56
0.45
1.30
1.07
50
120
150
30
60
150
160
90
120
150
160
Max Frames
Max Promes
Max Frames
(b) Evaluation: Effect of varying maxfssmes (30-180) at inference while fixing training at 60 frames.
Figure 20: Effect of temporal context (maxframes) on VideoJudge performance across Pearson, RMSE, and MAE. Increasing temporal coverage during training improves correlation and reduces error up to 244 frames, after which gains plateau. At evaluation time, performance improves up to ~ 120 frames and then saturates, indicating diminishing returns beyond moderate temporal context.
33


Reviews:
[{"review_id": "SxXL7OCIo0", "reviewer": "Reviewer_Q1YP", "summary": "This paper addresses the challenge of evaluating video MLLMs, noting that traditional metrics (e.g., BLEU) are insufficient and human evaluation is costly. The authors propose VideoJudge, a specialized 3B/7B MLLM designed to act as an evaluator. The core method is a \"bootstrapping\" pipeline: A \"Generator\" model creates candidate answers for 1-5 star ratings, and an \"Evaluator\" model validates if these answers match the intended rating. This bootstrapped dataset (over 100k examples) is then used to fine-tune the small VideoJudge models. The central claim is that the fine-tuned VideoJudge-7B outperforms much larger models, such as Qwen2.5-VL-32B and 72B, on several meta-evaluation benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles the critical and unsolved problem of scalable, reliable evaluation for open-ended video MLLMs.\n- Good writing and easy to read.", "weaknesses": "- Core Logical Flaw (Circular Evaluation): The paper's primary claim—that its 7B model \"outperforms\" a 72B model —is built on a foundation of circular logic. The training data for VideoJudge is created by a \"Generator-Evaluator\" (G-E) pipeline. The key meta-evaluation benchmarks used to prove this (e.g., VideoJudgeLLaVA-MetaEval, VideoJudgeVCG-MetaEval) were created using the exact same bootstrapping pipeline. This means the 7B model (the \"student\") is fine-tuned to mimic the preferences of the G-E pipeline (the \"teacher\"). The evaluation then tests this student against the teacher (in a zero-shot setting) on a test set created by the teacher itself. The student outperforming the zero-shot teacher on the teacher's own test is not surprising. It merely proves that fine-tuning on a specific data distribution works (i.e., distillation is effective). It does not prove the 7B model is a \"better\" or more reliable judge in a general sense. \n- Given the circular evaluation, the paper requires strong validation on a fully independent, human-annotated preference benchmark. While VideoAutoArena and a small 200+ sample set (VJ-H) are used, the main conclusions from Table 1, which support the paper's primary claim, are drawn from the self-generated benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "29136GIHcI", "reviewer": "Reviewer_PNxS", "summary": "The paper introduces VideoJudge, a framework that uses multi-language large language models (MLLMs) as judges for video understanding tasks. It employs bootstrapping to iteratively improve the model’s performance by generating scalable supervision signals.", "questions": "please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The use of bootstrapping to create scalable supervision for MLLM judges is novel. This method provides a unique solution to the challenge of limited labeled data in video understanding tasks.\n- The bootstrapping process is well-designed, with clear iterations and refinement steps that enhance model performance.\n- The paper is well-structured, with clear explanations of the methodology, experiments, and results.", "weaknesses": "- The resulting VideoJudge model might learn to perfectly mimic the idiosyncrasies of evaluator model rather than learning a generalized, human-aligned evaluation function. This could create an artificial performance ceiling, preventing VideoJudge from truly exceeding the quality of the initial MLLM judge.\n- These general instruction-following datasets likely share a common distribution of tasks (e.g., simple QA, short-form captioning). This raises a question about the framework's performance when applied to video tasks with fundamentally different reasoning requirements, such as long-form temporal reasoning, dense video grounding,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yPMdF8bD5D", "reviewer": "Reviewer_Lbto", "summary": "This paper addresses the challenge of scalable and reliable evaluation for video understanding tasks. The authors introduce VideoJudge, a 3B and 7B MLLM-based evaluator model trained via a bootstrapped generator-evaluator pipeline, where candidate responses at different quality levels are iteratively refined and filtered before being used as supervision. The system supports both pointwise rating and pairwise comparison, and the authors also construct meta-evaluation benchmarks to verify alignment with human preferences. Experiments show that VideoJudge models achieve comparable or even superior judgment quality to significantly larger MLLMs.", "questions": "Please refer to the weakness section for my questions. My major concerns are about the performance and the very limited baselines. Strong justifications shall be provided to demonstrate the significance of this submission.\n\nBesides, the authors claim that \"we have anonymously released the model checkpoints and datasets used for training and evaluation on HuggingFace\", but the links are not found.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Overall, the paper is generally well-organized and clearly motivates the need for scalable video evaluation beyond standard reference-based metrics.\n2. The proposed bootstrapping framework is reasonable and avoids costly manual annotation, which is an important practical benefit.\n3. Experiments include multiple settings (pointwise, pairwise) and span several baseline models, showing consistent improvements.", "weaknesses": "1. While the paper claims this is \"the first bootstrapped framework for training scalable MLLM-based evaluators across diverse video understanding tasks\", it would benefit from clarifying how it differs from existing MLLM-as-a-Judge work that also uses distillation from other evaluators (e.g., VideoScore and Prometheus-Vision). The novelty statement could be toned down to avoid overclaim.\n2. The performance gains over baseline on multiple datasets seem to be limited. According to Table 1, VideoJudge-7B performs worse than LLaVA-NeXT-7B (which shall be considered as a weak baseline given the weaker general video understanding capabilities than Qwen-2.5-VL-7B) under several metrics, yielding concerns regarding the actual effectiveness.\n3. Besides, Table 1 should include more MLLM-as-a-Judge works to clearly identify the potential effectiveness and significance of the proposed scheme.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CzEwT16fQo", "reviewer": "Reviewer_zfmP", "summary": "This paper proposes VideoJudge, a bootstrapped framework for training multimodal large language models to serve as automatic evaluators for video understanding tasks. By generating synthetic data through an iterative generator–evaluator process, the method enables scalable supervision without human labels. The resulting small models outperform larger MLLM judges in correlation with human judgments across multiple benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe generator-evaluator cycle to produce synthetic training data for the judge is meaningful and well-motivated. It reduces reliance on human annotations and enables larger scale.\n\n2.\tThe authors show that a relatively small model (3B) can outperform much larger baselines when trained appropriately, which is a strong practical result.", "weaknesses": "1.\tThe bootstrapping relies on LLM-based generator and evaluator and that LLM may carry bias and errors. If the evaluator is weak, the whole pipeline could propagate flawed judgments. The paper could more deeply analyze potential bias or drift in this synthetic supervised signal.\n\n2.\tWhile the benchmarks used show strong correlation results, it is not entirely clear how robust the judge will be to entirely new video domains, tasks, or instruction types. The paper would benefit from a “domain shift” evaluation (unseen video types).\n\n3.\tWhile the correlation metrics are good, more detailed breakdowns of failure cases where the judge disagrees with humans would help understand the limitations. Are there types of errors the judge misses (e.g., subtle temporal reasoning, common sense)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)