VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding
Authors: Zongxia Li
Xiyang Wu
Guangyao Shi
Yubin Qin
Hongyang Du
Tianyi Zhou
Dinesh Manocha
Jordan Lee Boyd-Graber
Venue: NeurIPS 2025 poster ()
Keywords: Vision-language Model
Video Understanding
Video Generation
Hallucination
Diffusion Models Monitoring
Primary area: deep_learning
TL;DR: We present VideoHallu, a benchmark of over 3,000 synthetic videos with expert-crafted counterintuitive QA pairs, evaluating MLLMs' ability to detect perceptually obvious abnormalities often missed due to language priors.

Abstract: Vision Language models (VLMs) have achieved remarkable success in video understanding tasks. Yet, a key question remains: Do they comprehend visual information or merely learn superficial mappings between visual and textual patterns? 
Understanding visual cues, particularly those related to physics and common sense, is crucial for AI systems interacting with the physical world. However, existing VLM evaluations primarily rely on positive-control tests using real-world videos that resemble training distributions. While VLMs perform well on such benchmarks, it is unclear whether they grasp underlying visual and contextual signals or simply exploit visual-language correlations. To fill this gap, we propose incorporating negative-control tests, i.e., videos depicting physically impossible or logically inconsistent scenarios, and evaluating whether models can recognize these violations. True visual understanding should evince comparable performance across both positive and negative tests. Since such content is rare in the real world, we introduce VideoHallu, a synthetic video dataset featuring physics- and commonsense-violating scenes generated using state-of-the-art tools such as Veo2, Sora, and Kling. The dataset includes expert-annotated question-answer pairs spanning four categories of physical and commonsense violations, designed to be straightforward for human reasoning. We evaluate several leading VLMs, including Qwen-2.5-VL, Video-R1, and VideoChat-R1. Despite their strong performance on real-world benchmarks (e.g., MVBench, MMVU), these models hallucinate or fail to detect physical or logical violations, revealing fundamental weaknesses in visual understanding. Finally, we explore reinforcement learning-based post-training on our negative dataset: fine-tuning improves performance on VideoHallu without degrading results on standard benchmarks, indicating enhanced visual reasoning in VLMs. Our data is available at https://github.com/zli12321/VideoHallu.git.

VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understanding
Zongxia Lit
Xiyang Wu
Guangyao Shi
Yubin Qin
Hongyang Du
Tianyi Zhou
Dinesh Manocha
Jordan Lee Boyd-Graber'
UUniversity of Maryland, College Park Uniiversity of Southern California
{z1i12321, wuxiyang, Yubinq, hydu, zhou, dmanocha, ying) edu, shig@usc. edu
Abstract
Vision Language Models (VLMs) have achieved remarkable success in video understanding tasks. Yet, key question remains: they comprehend visual information, merely learn superficial mappings hetween visual and textual patterss? Understanding visual cues, particularly those related to physics and common sense, is crucial for AT systems interacting with the physical world. However, existing VI.M evaluations primarily rely positive- control tests using real-world videos that resemble training distributions. While VLMs perform well such benchmarks, is unclear whether they grasp underlying visual contextual signals or simply exploit visual-language correlations. To fill this gap, we propose incorporating negative-come contrre tests, Le., vidcos depicting physically impossible or logi- cally inconsistent scenarios, and evaluating whether models can recognize these violations. True visual understanding should evince comparable performance across both positive and negative tests. Since such content rare the real world, we introduce VideoHallu, a syn- thetic video dataset featuring physics- and commonsense-violate state-of-the-art tools such Veo2, Sora, and Kling. The dataset includes expert-annotated question answer pairs spanning four categories of physical and commonsense violations. designed straightforward
scenes generated using
human reasoning. We evaluate several leading VLMs. including Qwen-2.5-VL VL, Video-RI. RI, and VideoChat-RI. Despite their strong performance on real-world benchmarks (e.g., MVBench, MMVU), these models hallucinate or fail to detect physical or logical violations, revealing fundamental weaknesses in visual understanding. Finally, we explore reinforcement lcarning-based post-training on negative dataset: tne-ttnnng improves performance on VideoHallu without degrading results on standard benchmarks benchnarks-inniiatinng enhanced visual reasoning in VLMs. Our data is available at
https: github. com/ (1412321/Videeoo luu. git
Introduction
Vision-Language Models (VLMs) have made remarkable progress in video understanding. However, they remain prone to hallucinations and shallow visual reasoning [H3] Prior works mitigate these issues across various domains, including chart interpretation 4.. video understanding [3]. and visual question answering (VQA) (6), primarily through supervised fine-tuning (SFT) R1-style chain-of- thought training (reinforcement learning) 7 []. However, most of these VLM evaluations rely on posifive-comtro! test, that is, real-world data drawn from distributions closely aligned with training data. Consequently, remains unclear whether current VLMS genuinely reason about visual cues or merely exploit prior visual-language correlations within familiar distributions 9
To truly evaluate visual understanding, mwe test VLMs under negative-conte conditions, i.e., videos outside their training distribution that depict physically impossible or logically inconsistent events.
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Positive-Contral Tests
Negative- Control Tests
The proceso that bulless shooos inbo
Video
Videoe
Ruman Vieual Desoription:
Human Vouel Description:
Sioee nanumelia plice elruphh change Vas auncaral skaper
Ricomahi cr folle veverne ieseet
Human Visual Description:
Comment:
balee alor untra natronin and
Comment
Countetivv phwnomento polentally conticiing with ooremo rootis plapicaa Rogical tomporal events
Harain optinkr
M Mosliggn wilh the pronst urusual Tonsiloon no bullet prosent
Counterintultive Question Answering
Critical Thinking
the viñeo (Sora), the watemelon breaks the niddiz viico. it intect broken the and?
[Sora]
Oaiiton
Queliece
At the the video, the vaterne is clearly braken been
Bullet interaction Looks Artificiat travcling toware the
viable entering
stuttored into pioces...
any frams.
X
anusual lspest vido
toy shoot
the end of the video, the wotermelon broken
watermeton, which object typical not
Owan S-AL
The unvoss and coumenamatta thing abour the viheo that tt
Therdlors watermslon broken.
playing in reverse. Reforming watormetonn The video starts by showing pleces of waterme on sceniing gly Tiving back tegetricr to frui....
Geri
Gembii250r
Questiors
inquitive
ceee-ended
criiial
spocting visual abnormalities.
Figure 1: Illustrative examples of designed negative-control tests evaluate the critical thinking abilities of VLMs. Unlike real-worke videos, synthetic videos can contain counterfactual or commonsense-vinn vinioooinnne-iioatinng contexts misaligned with reality. VideoHallu includes such synthetic videos with perceptually obvious abnormalities, paired with crafted questions that probe counterintuitive phenomena or test VLMs' critical thinking detecting such abnormalities. When SOTA VLMs are evaluated on VideoHallu, they frequently hallucinate, which suggests that these models rely on language priors and commonsense knowledge rather than truly understand the vidcos.
These tests reveal whether models detect violations of physics, causality, or commonsense instead of relying on memorized language knowledge. However, constructing such out-of tnttof-disttibution (OOD) videos in the real world is costly and impractical [10].
can produce photorealistic but physically impossible scenes. Such models provide alternative option to generate test videos for probing VLMs' visual understanding. By careful design, these synthetic videos can be systematically introduced to include violations of gravity, causality, and commonsense interactions [6), enabling controlled OOD evaluations where models need to rely on visual cues. Current VLMs, predominantly trained on videos conforming to physical laws, may thus overfit to statistical regularities rather than learning genuine causal reasoning [77. Figure illustrates that even state-of-the-art VL.Ms such Gemini-2.5-PP (18), GPT-4o [19) and Qwen2.5-VI. (20) hallucinate when confronted with counterintuitive scenarios. For instance, when a watermelon reassembles after an explosion, models rely on linguistic priors (e.g., "a watermelon should break when shot") rather than actual visual cues, exposing their limited physical reasoning.
Modern video generation models such Vco2, Sora, and Runway ||
To rigorously evaluate such limitations, we introduce VideoHallu, a dataset of expert-curated question- answer pairs over synthetic videos featuring controlled violations of alignment, consistency, commonsense, and physics. We benchmark several leading VLMs and analyze their failure modes on these physically and logically inconsistent videos. Finally, we explore two post- training strategies, supervised fine-tuning (SFT) and reinforcement learning (RL) via Group Relative Policy Optimization (GRPO) (1), using both real-world data from Video-RI I 77 and synthetic data from VidcoHallu. GRPO enhances generalization synthetic video reasoning without degrading real-world performance.
Contributions.
1) We introduce VideoHallu, dataset of 3K expert-annotated QA pairs on synthetic videos that include violations spanning alignment, consistency, commonsense, and physical reasoning.
2) We evaluate state-of-the-art VLMs and find that even top-performing models (e.g., GPT-4o, 2.mini-2.5-Proo) can achieve only ~50 accuracy our dataset, exhibiting frequent hallucination in counterintuitive scenarios.

---

Physies Movement
XL
eenergy conservation under external forces
Physics: Can model
Common Sense & Phyaics
Gravity, Linear/Crrule Motion
detectt plysies violations?
Constraints & Properties Rigid bodies, Reflections
State Transition Heal, Phoise transitions
Reasoning: mouel apply common sense to inter- prer prompt visuals?
Reasoning with Knowledge
General Knowledge
Reasoning
Geomenryy Leyout, State Changes
Que interpretation Chain Thouughtsss
Temporal Dynamics Tracks Object Changes Over Time
Spatial Dynamics
Consistent: model detecr smooth changes overr time abrupt shifis?
Motion Dynamics Consistency
Enniity motion, Intoractions, Abrupt Changes
Dynamics
Perislions in Movementi Angle.
and Viewpoint
Recegnition & Classification
Entiyy Counting
Alignment: modlll identiff entities and match prompt?
Text Visual Alignment
Idemifiee categorizes antitiaa attribures
301
Quantifies Enthtles Present
Scane
Spetial Relationshipa
Color, Shape, Texture
Entity Properties
positions of static entities
Fxamines relative
Figure 2: Question Categorization of VideoHallu. We design our benchmark, VideoHallu, with four question categories to probe limitations in synthetic video understanding, covering perceptual understanding to abstract reasoning: (a) Physics assesses if the model applies physical laws to entity motions and procedural understanding. (b) Common Sense Reasoning tests if the model can reason based on its knowledge. (c) Spatial-temporal Consistency examines whether the model can track entity motion across frames. (d) A Alignment checks if the model correctly identifies and understands entities using visual and textual cues.
3) GRPO post-training with synthetic data improves visual reasoning on VideoHallu while maintain- ing real-world performance, providing path toward more physically grounded VLMs.
2 VideoHallu: Evaluating VLMs' Synthetic Video Understanding
Preliminary. Our objective is to evaluate whether VLMs can effectively reason about and answer questions concerning synthetic videos that fall outside the distribution of their training data. To construct such evaluation data, we synthesize videos from text prompts using generative models. For these synthetic test videos to meaningful, they should incorporate deliberate visual abnormalities that elicit responses contradicting common-sente expectations situations in which a model relying solely on linguistic reasoning would produce one answer, but where careful visual observation reveals an alternative, visually grounded truth. To see how VLMMs handle synthetic videos with such abnormalities, we categorize our evaluation questions into counter-intuitive and critical thinking types. Counter-intuitive questions focus on implausible shattered watermelon reassembling itself), while critical thinking questions evaluate the model's ability to detect visual inconsistencies logical contradictions (e.g., unnatural object breakage).
physically impossible events (e.g.,
Data Collection. Our data collection pipeline consists of two main stages: The first stage generates synthetic videos V with common sense physics abnormalities, i.e., videos that satisfy constraints (5). where the LLM backbone possesses human-aligned knowledge but VLMs overlook abnormalities, resulting in answers that disagree with human perception. We recruited five human experts to review the defined abnormality categorics (detailed in Table 2 and Appendix (C) and craft prompts that reproduce such abnormalities in generated synthetic videos. In total, they created 141 adversarial prompts, used to generate 987 videos across seven models: Sora (22). Veo2 11], Runway Gen 2 [3]
Kling 23 Pixverse 24] Lavie 25). and Video (26).
In the second stage, we craft adversarial video QA pairs to evaluate VLMs' understanding of synthetic videos. Human experts manually review each video to identify counterintuitive contexts that lead to significant discrepancies between VLM outputs and human perception, i.e., video QA pairs maximizing the objective function They then construct natural language questions- along with
3

---

the ground truth answer-baase based on the context. These QA pairs are categorized into sub-categories (Table 2) Each annotator writes QA pairs highlighting visually clear but semantically abnormal content, difficult for VL.Ms to detect. These questions are not designed to trick models but rather to probe their ability to catch subtle violations of common sense, physies, or prompt-viden mismatches, critical for robust, interpretable video evaluation (Figure 3).
Dataset Metadata. Our dataset comprises 3,233 video question-answw pairs with no video overlap across splits: 800 pairs for training, 908 for validation, and I 1,525 for testing. The videos average 96.0 frames per video corresponding to approximately 5.3 seconds average framerate of 23 FPS. Frame resolution averages 1042 x 588 pixels across all videos.
3 Experiment and Results
Given the collected adversarial QA pairs. we evaluate 17 SOTA VLMs (Table For models not trained with RL or chain-of-thouu (CoT) generation, we use standard prompting to generate direct answers. For those trained with RL or CoT supervised finetuning (e.g.. Video-RI-CoT 7 and VideoChat-PP R1-think [),, we prompt them to generate step-by-step critical thinking and reasoning before generating final answer (Appendix, D). Figure highlights hallucinations produced by SoTA models across all four categories in synthetic video understanding tasks, with the hallucinated contexts marked within each answer (additional examples in Appendix AA).
Answer Evaluation:
We adopt LLM-as-a-Judge 27 as our evaluation method. GPT-4o-mini
evaluates the correctness of model responses ($
6
Model
Alignment
Physics
Consistency
Commonsense
Overall
VLMs: <7B
SmolVLM-38
15.94
13.44
22.49
8.75
16.13
Quen2. 5- VL 3B 20
41.53
27.21
26.91
26.25
35.48
InternVL33 2B
31
47.36
32.79
42.17
32.50
42.82
VLMs: >7B
MLVA--DeeVision
32
44.22
32.46
32.13
45.00
39.93
Video-LLaVA 33
46.58
40.00
43.37
31.25
43.93
LLaVA-NeXT 34
50,95
36.07
38.96
31.25
44.98
Video-LLaMA 35
55,67
38.69
50.20
32.50
50.16
InternVL3-99
81
53.54
43.61
47.79
38.75
49.84
InteraVL3 14B
31
53.65
45.90
46.18
31.25
49.70
InternVI3
31
55.78
38.69
50.20
38.75
50.56
Twen2.5-VL-333
20
58.81
42.95
46.59
40.00
$2.66
Quen2.5-VL-7B
20
58.02
44.59
46.99
47.50
52.98
VLMs: RI Rinfetntde
VideoChat-PP R1
53.31
40.33
44.58
45.00
48.85
Mideo-R1-Sti
58.14
47.21
48.19
41.25
53.44
Video-R1
58.14
48.20
49.00
38.75
53.64
VLMs: Close-Source
Gemini-2. -2.5ir2
18
58.36 56.57
33.11
40.16
36.25
49.18
Gemini-2. 2..0-a.... 36
39.02
42.97
40.00
49.97
GPT-4o-mini
54.88
41.97
48.19
38.75
50.36
Table Video Model Evaluation Results. We evaluate diverse VLMs across different sizes on our test set, reporting alignment, physics understanding, spatial-temporal consistency, and commonsense reasoning. Video-R1 is best overall.
For CoT generations, we extract the final answer as responses to evaluate. To validate reliability on our dataset, we manually annotated 200 randomly sampled answer pairs, achieving 97% agreement with GPT-4o- mini.

---

Alignment
Spatial-tamporal Consistency
Ayunng othicte & lilinng dritzlligg
cout, porbrring
ditbo
Can you moving?
qual hunndlsiing
No, the mon's fect are not visible Broughout the video
Yes, the qual subteriy tranafomms into a rocater, afach vistace the tarpponal
consiltoney rike vedece.
cleatyy
No, do
tranetorming into roceter viceo...
Daeed on the tragg,, the menns feet arw not cieary staible ths to the ainle
No. the image thoes twe rootee csed sie by sid or factue backgrrund There it appear
the mation brur....
Seginkiig
Yes, you con 300 his foot moving.....
So the giil fone not kondoum into roceser roceler i male (hi3kon) insend partilgge is quickly replcadd a sacond mostter
Reasoning
Physics
A <worher and sabiyy Fork art rnteored Anmm beigh and begin to hel
Tee ground Earti. (Voo22)
Generate
being
from fouther the 9697
by
whish reathes wond fig
vidao?
feothr
(hrddos
glound
The woternaion and video.
The rods drearyy maches the ground Hirst in the sideo, contining the oxpected offect guanity and nor piace
video
chartroes
Fro pleces
video, the of goudty and si ree covoos
reach
fist. Th
and video, watermaton la broken.... ...
Ussad
diseraaions vise,, the rock machas the ground firs., The feathare rull much more simely and land aterthe roet. Ground Truth
Inneionn the end of the action sequerce depicied, the watermalon la broken.
Video Prompt
Questian
GPT-4o
N-W1-2.5-VLL-228
Caminn-2.Sspro
Figure 3: Example Synthetic Videos in VideoHallu. Example hallucination cases observed during SOTA VLM evaluations on synthetic video tasks. Each example includes the generation prompt, key frames, questions, human-annotated ground truth, and hallucinated answers from GPT-4o, Qwen2.5- VL, and Gemini-2.5-Prr with ballucinations marked in Red.
3.1 Limitations of VLMs in OOD Data
VLMs struggle with counterintuitive phenomena and abnormalities in generated videos. State- of-the-art VLMs achieve below 55% accuracy on our synthetic video QA dataset, only slightly above the 50% random baseline (Table They particularly falter in commonsense and physical reasoning. often failing to detect abnormalities or relying on linguistic shortcuts instead of visual evidence. As shown in Figure 3 none of the models recognize the implausibility of shattered watermelon reassembling, nor do they notice abrupt, counterfactual entity changes, such as quail suddenly turning into a rooster. These failures highlight VLMs' limited capacity for abnormality reasoning and critical visual understanding beyond text priors.
Chain-of-though reasoning learned from real-world videos provides limited benefit for derstanding synthetic videos. VLMs trained with reinforcement learning, such as GRPO (37 (RI-finetuned) used in the DeepSeek series (21]. show potential for improving reasoning and crit- ical thinking abilities in reasoning-heav
tasks like mathematics, real-world video understand- ing [38 3 19.. This raises a question: does RL truly improve visual reasoning in VLMs, or does only optimize for correct answers without enhancing actual visual understanding' Table [ two RI-finetuned models, Video-RI 7 and VideoChat-RI R, using chain-of- of-thougtttttt prompts. Both models show limited improvement compared to their base models (0wen2.5-VL-77 on synthetic video understanding, with minimal or worse alignment and commonsense, suggesting that training on real-world videos only inculcates real-world reasoning patterns.
evaluates
Solely pre-training on real-world data biases visual grounding. While RL improves reasoning on math and real-world videos, it does not help with counterintuitive synthetic content that contradicts real-world norms and cannot elicit video-grounded critical thinking. In such cases, chain-of- of-thutght prompting can bias the LLM backbone to rely too heavily on prior commonsense knowledge, neglecting synthetic visual cues and leading to hallucinated responses (). For instance, in the third case in Figure 3] the video shows the feather dropping to the ground before the rock when falling from the same height. When asked which reaches the ground first the feather or the rock, VideoChat-Be R1 -think responds: "The video shows a feather and a rock being dropped... This is classic demonstration of Galileo's principle; therefore the ruck drops before the feather... While this language explanation alone is grounded in correct physical principles, it directly contradicts what actually occurs in the video. The model generates an incorrect conclusion based on prior language reasoning, showing how chain-of of-thought prompting can amplify reliance on language priors and
5

---

increase hallucination risk when understanding synthetic videos that do not align with real-world expectations.
3.2
Visual Learning or Pattern Matching? Questioning RL's True Impact on Vision Models
Current VLMs struggle with counter-intuitie questions and critical visual thinking in synthetic videos They frequently hallucinate and dearth of critical thinking leads them to overlook abnormality examples in VideoHallu, relying on the model's ' language knowledge instead of reasoning directly from the visual input. This raises crucial question: Can VLMs learn counter-intuifive commonsense knowledge and improve their critical thinking abilities for detecting abnormalities through training with synthetic video data?
(Section 3.1
While supervised fine-tuning (SFT) or RL from human feedback (RLHF) are natural approaches for improving VLMs, the distributional gap between standard alignment-based video QA tasks used in pre-training and the specialized critical reasoning required for synthetic videos poses an impediment. Since synthetic videos are scarce in typical pre- pre-training corpora, models lack sufficient exposure to develop robust reasoning capabilities for such content.
To investigate whether incorporating synthetic data alongside real data can improve VLM performance on synthetic videos, we pose two primary research questions:
Between SFT and GRPO training, which approach more effectively enables VLMs to develop genuine understanding of synthetic videos?
Is synthetic data in the training mixture necessary for improving model reasoning abilities on synthetic videos, can training on real data alone suffice?
Method Overview: SFT vs. GRPO. We compare two training paradigms- SFT and GRPO 21 to compare their effectiveness and generalization.
Supervised Fine-Tuning (SFT). SFT directly optimizes the model predict ground-truth responses using maximum likelihood estimation. For a dataset D = {(T). y:) }ir1 containing video-question pairs F; and corresponding answers yi, the SFT objective minimizes the negative log-likelihood:
bd
(1) y
(Ct)
logpe pe
I4 Vi
N
where 0 represents model parameters, yi (t) the 1-th token in sequence y4 and (<) denotes all preceding tokens.
Group Relative Policy Optimization (GRPO). GRPO, a variant of reinforcement learning from human feedback, optimizes the model using meference-basee learning without requiring explicit reward models. Given preference pairs (yw, 31) where yo is preferred over y for prompt x, GRPO maximizes the likelihood of preferred responses while penalizing less preferred ones:
(B (x) (GRPO(O) - -. WD log (8log
pe ( x)) Pref (r)
3 log
Prer(Mio (Moo x))
where Prer is a reference model (typically the pre-trained checkpoint), 3 is a temperature parameter controlling the strength of the KL penalty, and the sigmoid function. This approach encourages the model to generate responses that align better with the training data distribution while maintaining proximity the reference policy.
The key distinction lies in their learning signals: SFT learns from direct supervision with ground- truth labels, while GRPO learns from comparative preferences, potentially enabling more internal reasoning from the model for video understanding.
Experimental Setup and Results. Both research questions require training data. We combine our 800 synthetic video training data with 2,000 video QA pairs sampled from Video-RI training data derived from LLaVA- Video 44).

---

To keep a fair comparison across different finetuning methods while reducing the training resources needed, we use 15 frames during training with learning rate 1e to train the model for one epoch using the Open-R1 (88 framework on eight A100 80G GPUs. For GRPO training, since our answers are free-form answers, we use the average ROUGE-I, ROUGH-I, ROUGE-L score as the reward:
(Apred((ppred; (gold) =
Σ (1.2L))))
ROUGE-i agold):
where ROUGE captures n-gram overlap F-score between expected answers and generated responses.
Result: SFT VS. GRPO. To address our first research question, we use both SFT and GRPO to train models on the mixed dataset and evaluate their performance on out-of-distribution synthetic video understanding. To validate the generalization of our findings, we run experiments on two architectures: Qwen2.5-VL-77 and LLaVA-One- Onviionn- [32].
GRPO outperforms SFT on out-of-distribution and critical visual understanding tasks (Table 2 consistent with Feng et al. [4]. who demonstrated GRPO's superior generalization. Because our dataset contains genuinely unseen during pre-training. these results offer stronger evidence of the two paradigms' differences. GRPO's advantage indicates that SFT tends to memorize surface-level patterns, whereas GRPO cultivates reasoning skills that better transfer to novel scenarios, a key capability for synthetic video understanding, where visual and temporal dynamics diverge markedly from natural videos.
synthetic videos generated by diffusion models
Model
Alignment Physics
Consistency
Commonsense
Overall
Previous Base Models
DOMA--nOeViiiion 32
44.22
32.46
32.13
45,00
39.93
Qwen2.5-VV 5-BL-7B (20
58.02
44.59
46.99
47.50
$2.98
Video-R1 7
58.14
48.20
49.00
38.75
53.64
SFT vs.
GRPO
(wen2.5-VLL. SFT
55.22
45.90
47.39
35.00
51.02
Tren2.55VL-777 GRPO
62.18
53.77
56.63
45.00
57.69
LLaVA-OneWision SFT
44.67
26.23
33.33
38.75
38.82
LLaVA-OneVisiio GRPO
46.24
30.82
34.54
48.75
41.38
Real Data vs. Synthetic Data (GRPO)
Qwen2. 5-VLBBB Real Only Quen2. 5-V Synthetic Only Swen2.5-VL-77 Combined
57.35
46.89
51.41
33.75
53.05
60.16
48.20
48,19
52.50
55,41
62.18
53.77
56.63
45.00
57.69
Table 2:
Fine-tuning results for SFT and GRPO. GRPO training leads to better improvement than SFT; augmenting the small synthetic video data leads higher accuracy than training on just real videos or limited synthetic videos.
Result: Effect of Training with Synthetic Videos. To address second research question regarding the relative contributions of real-world and synthetic video data to GRPO training performance, ablate the training for models via three different data configurations: (1) combined dataset mixing both data types, (2) synthetic videos only, and (3) real-world videos only. Training only on real-world videos leads to minimal improvement (0.07%) over the base model on synthetic video understanding tasks (Table 2): real-world video training alone does not transfer effectively to the reasoning required for synthetic video analysis. In contrast, synthetic videos improve the model's detection of abnormalities in synthetic content. However, the limited size of our synthetic video training set necessitates data augmentation: combining real-world and synthetic videos in the training mixture produces the most effective results. The mixed dataset setting enables VLMs to better adapt their reasoning capabillitess to synthetic videos, outperforming both single- single-domain training approaches. While synthetic data is crucial for developing domain-spectinn world videos helps stabilize training and improve overall robustness. Thus, VLMs require exposure to synthetic video data during training to develop effective reasoning abilities for synthetic content, and a balanced mixture of real and synthetic data optimizes duttt--dist synthetic video understanding tasks.
reasoning skills, the additional diversity provided by real-
7

---

Result: Performance on Real-world Benchmark. Incorporating synthetic video data along- side real-world videos can improve VLMs' understanding of synthetic videos. But does synthetic video training come at the cost of degraded real-world video comprehension?
To investigate this, we evaluate our trained
Qwen models on two real-world benchmark
Model
MMVU
MVBench
datasets: MVBench [ a comprchensive benchmark for evaluating temporal understand- ing and reasoning in videos, and MMVU [44). which tests expert-level multidisciplinary video understanding across diverse domains. Syn- thetic and real-world video understanding abil- ities can coexist.
2wen-2.5VL- (base)
58.7
69.6
+ Real Only
61.3
70.9
SSnthhttic Only
60.1
70.1
Commincd
61.3
70.1
Table 3: Post-training performance on real-world video understanding benchmarks.
Discussion. Throughout the evaluations over our benchmark and the fine-tuning over pre- trained VLMs, we gather essential insights accelerate further improvement over future VLMs for synthetic video understanding. We list them as follows:
1. VLMs hallucinate on Synthetic Data due to Neglect of Actual Visual Content. As shown in Table and Figure 17 all tested SOTA VLMs, including large models like Qwen2.5-VL. VL (7B/32B), GPT-40, and Gemimi-2.55 2.5-Proo as well as smaller models (<7B), struggle with counterintuitive QA on synthetic videos in VideoHallu. One reason is that VLMs often solely rely on their embedded commonsense and physics priors to answer questions, even when prompted to rely on video content (Figure 3 These hallucinations, caused by misalignment between video context and real-world norms, are rare in real-world QA but prevalent in synthetic settings, particularly for counterfactual reasoning. Although VLMs are exposed to some synthetic data during training, the vast majority of their training consists of real-world videos that follow physical laws and commonsense principles. Consequently, VLMs treat such rules as universal priors that override visual evidence, leading to hallucinations when synthetic videos contradict learned physical principles.
Critical thinking may be biased by language priors in synthetic visual abnormality detection. As discussed in Section 3.1] while RL enhances critical thinking in real-world video QA, all R1- trained VLMs we evaluated, such as Video-R1 1-oT and VideoChat-RI-thee underperform their base model (Qwen2.5- VL-BB) on VideoHallu, showing no clear improvement on commonsense or physics-oriented questions. We attribute this to flawed reasoning patterns in Rl-trained VLMs. Although chain-of-thought reasoning can elicit more structured inference in real-world contexts, it proves ineffective in synthetic video settings, where detecting abnormalities demands grounded, fine-grained visual understanding. R1-trained models often excel in language- only reasoning tasks [3]]]] yet when extended multimodal domains, their reasoning becomes heavily rely linguistic priors. Consequently, their CoT responses tend to reflect superficial comprehension of visual content and are more susceptible to hallucinations in counterintuitive or visually deceptive scenarios
in Table consistently
9 46
The high-quality negative control examples matter for model improvement. Given the need to improve VLMs' performance in synthetic video abnormality detection, as shown in Section 3.1 we run RL training experiments over Qwen2.5-VL. and LLaVa-OOe- Vision with mixture of real-world and synthetic videos. Our results show that, after training models with some synthetic videos, VLMs show improvements in critical thinking and their ability to handle counterintuitive scenarios. Our results suggest that it is the quality and coverage of the data, not just the fine-tuning method, that drive gains. With a small but rell-annotated dataset containing both positive and negative examples, detailed reasoning steps, and reasoning oriented training like GRPO, even small models like Qwen2.5- VL-7B show improved QA accuracy. This highlights the importance of high niigh-qualty, reasoning- -ich data in helping VLMs internalize and apply commonsense and physics knowledge, even with limited post-training resources.
4 Related Work
Hallucinations in VLMs. Hallucinations refer to the persistent challenge of generating outputs that contradict or misrepresent the target texts, images, or videos 47 48 It arises from conflicts
between the language priors of VLMs and the actual visual inputs 49 which is more severe in
8

---

video understanding than in static image understanding due to the complex entanglement of spatial- temporal information across the timeline and the contextual cues associated with entities within frames. A line of prior work, such as VideoHallucer [50], EventHallusion [51], and HAVEN [52]. established benchmarks for evaluating model hallucination on both entities and events within videos,
while also proposing methods to enhance the video understanding capabilities of VLMs A 53 54]
However, most prior works on hallucination, particularly in the video domain, rely on real-world factual data, rather than synthetic data generated by generative models. Hallucination in generative video understanding models remains an open and largely unexplored research area.
Reinforcement Learning for Post-training of Vision-Languaal Models. Inspired by the techniques from DeepSeek-R2 [2]] there an increasing body of research that leverages reinforcement learning
in post-training to enhance the general general-purpose purpose multimodal reasoning capabilities of VLMs [55] [6] Most recent efforts have focused on using GRPO and its variants to fine-tune VLMS to elicit more
robust reasoning and perception skills [57 56 3 58 The representative work Video-R1 7 collects
a large-scale corpus of 260K video and image samples and performs GRPO with data sype-specific reward engincering. It applies gegression-baset approximations for numeric answer types, ROUGE- based metries for free-form textual responses, and exact match rewards for multiple-choice questions, enhancing models' temporal reasoning for real-world video understanding. VideoChat-RI [59] extends this paradigm to interactive multimodal dialogue, combining video-centric instruction tuning with reinforcement learning from human feedback (RLHF). Nonetheless, prior research has pre- dominantly focus on visual understanding in real-world imagery and videos, with generated videos receiving comparatively little attention.
Video Generation Models and Synthetic Content Monitoring. Recent advances video generation models have enabled the creation of highly realistic and aesthetic videos from text prompts, reference images, or conditioning frames 04 23 12 25 13 (6). These models are increasingly applied in content creation, simulation for robotics and autonomous driving [60]. Since the release of Veo3 [14] Sora 2 [6.. Wan (62] the volume of generated content has exploded. These vast generated videos present major challenges for content monitoring, quality evaluation, and content verification. Manual annotation and evaluation are increasingly infeasible given the scale and variability of generated outputs, thus motivating the need for automated, scalable evaluation and reasoning frameworks that are specifically tuned for synthetic video understanding. To date, a few works have explored using VLMS to evaluate generated images (for example, detection of synthetic images assessing image generation quality). . For example, [63] presents a method named Bi-LORA that uses a VLM to detect synthetic images. However, the domain of synthetie videos is still largely under-explored: lack systematic methods, datasets and evaluation protocols for using VLMs to judge and understand synthetic videos
5 Conclusion and Limitation
Conclusion. We introduce VideoHallu, a dataset designed evaluate VLMs' visual common- sense and physics reasoning through synthetic videos with counterfactual scenarios. It features expert-annotated, reasoning QA pairs spanning alignment, spatial-temporal consistency, commonsense, and physics categories to assess VLMs' ability to detect abnormalities and violations of physical laws. Evaluation of SOTA VLMs on VideoHallu shows hallucinations and critical thinking failures. Fine-tuning with GRPO with both real and synthetic videos leads to accuracy improvements on VideoHallu, showing the value of incorporating structured physics and commonsense reasoning data to improve VIMM performance on synthetic video tasks. However, scalability remains limitation, as generating high-quality annotations and fine-tuning VLMs at scale is costly, and limited access to data and compute constrains further progress. Future work will focus on expanding synthetic video datasets with abnormality QA pairs to train VLMs for critical, -gouull-goounded reasoning. Scaling with adversarial QA pairs can enhance robustness and enable automatic video evaluation via prompt decomposition, reducing reliance on human annotations.
Limitations. Despite enabling controlled evaluation of visual commonsense and physics reasoning, VideoHallu has three key limitations: (i) a domain gap between synthetic videos and real-world visual complexity, which may reduce transfer; (ii) limited coverage of rare physical edge cases and long-horizon causal interactions; and (iii) high scalability cost for expert annotation, counterfactual generation, and GRPO fine-tuninn fine-tuning, constrained by data and compute access. Morcover, QA-basca

---

scoring may not fully reflect grounding or uncertainty calibration, motivating complementary metrics and automated adversarial QA generation.
Acknowledgment
The work done with the computer resources and support of Lambda Lab. Boyd-Graber and Li are supported by National Science Foundation Grant No. IIS-2403436. Any opinions, findings, and conclusions recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
References
[1] Zechen Bai, Pichao Wang. Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey, arXiv preprint, 2024.
arXiv:2404.1880 18930..
[2] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375 14385, 2024.
[3] Zongxia Li, Wenhao Yu, Chengsong Huang, Rui Liu, Zhenwen Liang, Fuxiao Liu, Jingxi Che, Dian Yu, Jordan Boyd-Graber, Haitao Mi, and Dong Yu. Self-rewarding vision-language model via reasoning decomposition, 2025. URL URL https //axxiv. org/ abs abs/2508....///////// 2508 19652
[4] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigat- ing hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023.
[5] Wenyi Hong. Yean Cheng, Zhuoyi Yang, Weihan Wang, Lefan Wang, Xiaotao Gu, Shiyu Huang, Yuxiao Dong, and Jie Tang. Motionbench: Benchmarking and improving fine-grained video motion understanding for vision language models, 2025. URL htt.hhttp/////c //arxiv. org/abs/
2501.02955
[6] Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Dhruv Batra, and Devi Parikh. Vqa: Visual question answering, 2016. URL https://arxi::
org/ org/abs/ was.bb//..... 00468
[7] Kaituo Feng. Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-rl: Reinforcing video reasoning in milms. arXiv preprint 25X3::2503. 2025.
[8] Xinhao Li, Ziang Yan, Desen Meng. Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-rl: Enhancing spatio-temporal perception via reinforce- ment fin-tuning. arXiv preprint 250: 06958, 2025.
[9] Chengzhi Liu, Zhongxing Xu, Qingyue Wei, Juncheng Wu, James Zou, Xin Eric Wang, Yuyin Zhou, and Sheng Liu. More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models, 2025. URL wtttthtt://et :ttps:///rrxii /ttp/vaxvvvv.. .rgg htgg///bs/5055.5 21523
[10] Yihong Dong, Xue Jiang, Huanyu I. Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge L.. Generalization or memorization: Data contamination and trustworthy evaluation for large language models,
2024. U https:://///. //tp///axvvvv./////// wwgg/bs/////e/ abs 2402. 15938.
[11] Veo-Team. Veo 2. DeepMind Blog, 2024.
URL. https://deepmind. google
technologies. wttp//ee/veoveo-222
[12] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, al. Video generation models as world simulators. OpenAl Blog, 1:8, 2024.
10

---

[13] Introducing Gen-3 Alpha: A New Frontier for Video Generation. Runway ml. Imagine.Art, alpah
2024. http//////////aaaaaooooo///////a
[14] Google DeepMind. Veo 3 model card, 2025. URL stoos:: www.geeppooogepissso.ooooooo com/ deepmind deepmind-mediaeee 3atpoodee--aardae//e3333-----
pdf Model card, version published
May 23 2025.
[15] Bingyi Kang. Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang. and Jiashi Feng. How far is video generation from world model: A physical law perspective, 2025. URL www..bs.///ooo....oooo...o... https: fitps://arxiv/ http://wwwvvi..r..r.////// abs /411 11111.000 02385
[16] Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. Video models are zero-shot learners and reasoners, 2025. URL Utt.tttss:://// h //axiv.. org/ 2b50 20328,
[17] Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning, 2024. URL https
//arxiv. www.//rgg//////ee..eeooo
03570
(18) Google DeepMind. DeepMind, 2025.
Gemini 2.5: Our most intelligent ai model.
Google
URL https: howpl////eeoeooooo/gggo/oo////eo /ttp//////////oogoo/////////aaaa
2025
[19] OpenAL. Gpt-4o mini: advancing cost-efficient intelligence. OpenA/, 2024. URL https
[20] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[21] DeepSeek-AI. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
12948
[22] Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao. Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, Lifang He, and Lichao Sun. Sora: review on background, technology, limitations, and opportunities of large vision models, 2024. URL
https://arxix: org. hogg/abs/2400 2402. 17177
[23] Kuaishou. Kling ai. Kling AI, 2024. URL wttph:t//////tt www.:wwwiia...ooo/aaea
[24]
Vixverse Team. Pixverse: Ai-powered image generation platform. https://app: http//pppppixveeeeeeeeee
ai/home 2025. Online; accessed April 25, 2025.
[25] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang. Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. International Journal of Computer Vision, pages 1-20, 2024.
[26] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.1555 2022.
[27] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging -lm-as-a-judge with mt-bench and chatbot arena, 2023. URL htttps////xxx abs/ 2as6/230...0 006685558
[28] Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thore,, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability language models, 2024. URI.sttps: //arxivv.. .rgg/i.ooorraaeee abs/
2310. 08491
11

---

[29] Zongxia Li, Ishani Mondal, Yijun Liang, Huy Nghiem, and Jordan Lee Boyd-Graber. Pedants: Cheap but effective and interpretable answer equivalence, 2024. /tttss////aax abs/2402, 21102.1111111
[30] Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril 7akka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvim: Redefining small and efficient multimodal models. arXiv preprint arXiv:2504. 05299, 2025.
[31] Jinguo Zhu, Weiyun Wang, and Zhe Chen et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URI.https: h //axxvv..vv////////////x///aaa org/ abs/ 2504.
[32] Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. URL https: / //a::////xii /wwpvv..vvvv..////////aa//aaaa wrw.iiioorrraoooooeeo htt//bbb/24008 03326.
[33] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.1012, 2023.
[34] Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng. Ziwei Liu, and Chunyuan Li. Llava-next: strong zero-shot video understanding model, April 2024. URL
https://llava: vl. github wtt.hbu..o/olog//000 1000-00000---11a---------------- -extaaaet----iiee
[35] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction audio-visual language model for video understanding. arXiv preprint arXiv:2306.008888 2023.
[36] Google
DeepMind. Introducing gemini 2.0:
our new ai model for the agen- URL https://blog google/tecchology. /oogg/////cooooggggooloeeeeeeeee goople-deepmind/ 2024 #ceo
tic era. 2024.
google-gemini hatggaa--aaa-------aaaaeeeeeeeee
[37] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
[38] Hugging Face Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https://g/ htt.:/bb.t.b......//./ com wwwm http://hgggggggggeee ace/ open-rl
[39] Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generaliza- tion ability in vision-language models with less than $3.
R1-V V 2025. Accessed: 2025-02-02.
[40] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava- video: Video instruction tuning with synthetic data, 2025. URL TRLLhttps: http:::///xxxvv///e///aa
2410.02713
[41] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74 81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL httpps: haaittpps//clllnhooooyyyy hrw.ooloogy.oog///e... wrg//04------0 1013//
[42] Zihao Feng, Xiaoxue Wang, Ziwei Bai, Donghang Su, Bowen Wu, Qun Yu, and Baoxun Wang. Improving generalization in intent detection: Grpo with 2025. URL //arxiv org/abs/////// 250/ 13592.
curriculum sampling,
[43] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video understanding benchmark, 2024. URL htttpss/aarii..oor/a////111.1 17005
[44] Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, Zhijian Xu, Chengye Wang, Weifeng Pan, Ziyao Shangguan, Xiangru Tang, Zhenwen Liang, Yixin Liu, Chen Zhao, and Arman Cohan. Mmvu: Measuring eeeert- multi-discipline video understanding, 2025. URL htttss//aaxxi
org/ /bs//ab///aa
12

---

[45] Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang. and Haitao Mi. Self-evolving reasoning Ilm from zero data, arXiv preprint arXiv:2508. 05004, 2025.
[46] Tong Zheng, Hongming Zhang. Wenhao Yu, Xiaoyang Wang, Xinyu Yang, Runpeng Dai, Rui Liu, Huiwen Bao, Chengsong Huang, Heng Huang, et al. Parallel-rl: Towards parallel thinking via reinforcement learning. arXiv preprini arXiv:2509.00 079900790, 2025.
[47] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint 1iXiv:180.002/1 2018.
[48] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[49] Xiyang Wu, Tianrui Guan, Dianqi Li, Shuaiyi Huang, Xiaoyu Liu, Xijun Wang, Ruiqi Xian, Abhinav Shrivastava, Furong Huang, Jordan Lee Boyd-Graber, et al. Autohallusion: Auto- matic generation of hallucination benchmarks for vision-language models. arXiv preprint WxXi:22466.10900 2024.
[50] Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models. arXiv preprint arXiv:2406.16336 2024.
[51] Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Na Zhao, and Jingjing Chen. Eventhallusion: Diagnosing event hallucinations in video llms. arXiv preprint arXiv:2409. 16597, 2024.
[52] Hongcheng Gao, Jiashu Qu, Jingyi Tang, Baolong Bi, Yue Liu, Hongyu Chen, Li Liang, Li Su, and Qingming Huang. Exploring hallucination of large multimodal models in video understanding: Benchmark, analysis and mitigation. arXiv preprint 25.3v:22003196222 2025.
[53] Zongxia Li, Xiyang Wu, Guangyao Shi, Yubin Qin, Hongyang Du, Tianyi Zhou, Dinesh Manocha, and Jordan Lee Boyd-Graber. Videohallu: Evaluating and mitigating modti-modal hallucinations synthetic video understanding. 2025. URL.ttpss:// //axxiv./ org/ haagg//b///// /205
01481
[54] Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations. arXiv preprint 22.0022410.02762. 2024.
[55] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-24198, 2024.
[56] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models, 2025. URL httppp::/aaxivvoooo /aogg//ab///// 2200//33a/////a..a/// 06749
[57] Rui Liu, Dian Yu, Tong Zheng, Runpeng Dai, Zongxia Li, Wenhao Yu, Zhenwen Liang, Linfeng Song, Haitao Mi, Pratap Tokekar, and Dong Yu. Vogue: Guiding exploration with visual uncertainty improves multimodal reasoning, 2025. URL http:://axxvvv.rg//e// 01444
[58] Jingyi Zhang, Jiaxing Huang. Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization, 2025. URL /tttts:/t//aattil org/ herg//b//aa 220//550331 12937
[59] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.00 06355, 2023.
[60] Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, and Guangyao Shi. A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges, 2025. URL htthttps::// // arxiv org/oos//////oooooooooooooaes abs/aaas//51. 02189
13

---

[61] OpenAI. Sora 2 system card. Technical report, OpenAL, September 2025. URL
https: :ttpss:/cddis openai com/pdtt c4ff 4c2d- 986f www.ooo0f0099/0orro....o /orr/066///rr___ sora__.
system_ ceastem_card.oe pdf Version released on September 30, 2025.
[62] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang. Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang. Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng. Tong Shen, Wei Lin, Wei Wang, Wei Wang. Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng. Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models, 2025. URL
20314
[63] Mamadou Keita, Wassim Hamidouche, Hessen Bougueffa Eutamene, Abdenour Hadid, and Abdelmalik Taleb-Ahmed. Bi-lora: A vision-language approach for synthetic image detection, 2024. http:htss://aat
org httg//bs/224444 .095/
[64] Dacheng Li, Yunhao Fang, Yukang Chen, Shuo Yang, Shiyi Cao, Justin Wong, Michael Luo, Xiaolong Wang, Hongxu Yin, Joseph E Gonzalez, et al. Worldmodelbench: Judging video generation models as world models. arXiv preprint arXiv:2502.20094 2025.
14

---

A More Synthetie Video Examples
We present selected cases from SOTA MLLM evaluations across each VideoHallu sub-category. Hallucinations model answers, common sense or physics violations videos, and other notable cues in the video, questions, or ground truth are highlighted to assist the reader's s understanding.
statue, plastic statue, and glass beer
many statues made from distinct materials appear throughout video?
Only one distinct matertal is present: bronze, Although two bear statues appear, both are made of bronze.
observed: One bronze bear statue; One plastic bear statue; One glass bear statue; Total: bear statues of distinct materials.
There are two beer statues. Both appear to be made of the same material, which looks like metallic bronze-like substance...
There one pair bear statues with distinct materials present the
Alignment Entity Counting
Figure 4: Hallucination Case from Alignment Entity Counting (A-EC). We show hallucination examples from SOTA MLLM evaluations under the A-EC category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VLL (Purple), and Gemini-2.5-Pro (Blue), with hallucinations and critical context highlighted in Red.
Generste one marble ball with a coarse surface and another with smooth surface.
Which marble bigger the one on left or one right?
They appear to the same size.
It seems that only one marble ball was clearly detected in the first frame, couldn't confidently compare the
Both marble balls appear
the video.
The one the right.
Alignment Entity Properties
Figure 5: Hallucination Case from Alignment Entity Properties (A-EP). We show hallucination examples from SOTA MLLM evaluations under the A-EP category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VI (Purple), and Gemini-2.55PPi (Blue), with hallucinations and critical context highlighted in Red.
15

---

A young athlete playing baskatball on court. performing impressive oribbling and siam dunks.
The man dribóling basketball on the court. Can you see feet moving?
No, man's feet visible throughout video
Yes, based on the sequence of frames extracted from the video, the man's feel are clearly maving...
Based the image, men's feel not clearly visible due the angle and the motion blur...
Yes, you can see feet moving...
Alignment Entily Recognition and Classification
Figure Hallucination Case from Alignment - Entity Recognition and Classification (A-ERAC). We show hallucination examples from SOTA MLLM evaluations under the A-ERAC category. Each case includes the video generation prompt (Gray), key frames from synthetic videos questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VL (Purple), and Gemini-2.5-Prr (Blue), with hallucinations and critical context highlighted Red.
Generste rose, sunflower, and delay.
Where each sunflower facing?
One facing the screen, facing the left, and one facing left but angled toward screen.
Most sunflowers facing to the right.
The one the left is facing forward. The one in the middle is facing more towards the right. one on right also facing towards right.
sunflowers facing towards
right.
Alignment Spatial Relationships
Figure 7: Hallucination Case from Aligmment Spatial Relationships (A-SR). We show hallucina- tion examples from SOTA MLLM evaluations under the A-SR category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VI. (Purple), and Gemini-2.5-PPro (Blue), with hallucinations and critical context highlighted in Red.
16

---

Dew drops on leaves in the morning.
What the direction of viewpoint's movement?
From right left.
The viewpoint moving right slightly upward
The viewpoint in video moving slightly the right.
The viewpoint appears be static.
Spatial-temporal Consistency Camera Dynamics
Figure 8: Hallucination Case from Spatial-temporal Consistency Camera Dynamics (SC- CD). We show hallucination examples from SOTA MLLM evaluations under the SC-TD category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VL (Purple), and Gemini-2.5-Pre P.5-Pro (Blue), with hallucinations and critical context highlighted Red.
Generste quall end a rooster celebrating New Year.
beginning the quail slowly glass its right hand
left hand, correct?
No, the transitions abruptly from its left hand its right hand.
Yes, does appear that the quall-Ike figure is gradually transferring the glass from Its right
No, that's not correct The quail is holding the glass in its right hand throughout the sequence.
Based Birds (lkee qualls) don't hands. glass remains that position relative to the quail throughout the first few seconds.
Spatial-temporal Consistency Spatial Dynamics
Figure Hallucination Case from Spatial-temporal Consistency Spatial Dynamics (SC- SD). We show hallucination examples from SOTA MLLM evaluations under the SC-SD category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VI. (Purple), and Gemini-2 Semmini-255-Pa 2.5-Pro (Blue), with hallucinations and critical context highlighted Red.
17

---

Generate a quail and a rooster celebrating New Year:
Do you see quail transforming into rooster in the video?
Yes, the quail suddenly transforms into rooster, which violates the temporal consisiency rule videos.
No, not quall transforming Into rooster the video....
No, the image shows two roosters standing side side festiee background There is no transformation; both birds appear to be roosters from beginning to end.
the quail does transform rooster rooster partridge is quickly replaced by second rooster.
chicken). Instead,
Spatial-temporal Consistency Temporal Dynamics
Figure 10: Hallucination Case from Spatial-temporal Consistency Temporal Dynamics (SC- TD). We show hallucination examples from SOTA MLLM evaluations under the SC-TD category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VL (Purple), and Gemini-2.5- P.r (Blue), with hallucinations and critical context highlighted Red.
Generate a video showing the Earth revolving around the Sun.
We know Sun does not have rings. Sun present this video?
No, the Sun only has planets surrounding it.
We can confidently say this not the Sun the video. more closely resembles planet Saturn, or artistic representation celestial body with rings.
The image likey depicts an imaginative or conceptual scene rather than realistic depiction sun.
Yes, there is sun in this video.
Common Sense Reasoning Knowledge
Figure 11: Hallucination Case from Common Sense Reasoning Knowledge (CS-K). We show hallucination examples from SOTA MLLM evaluations under the CS-K K category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VL. (Purple), and Gemini- 2.5-Pro (Blue), with hallucinations and critical context highlighted in Red.
18

---

feather heavy are same height begin fall ground on Earth.
the feathers reaches the ground first-one one of the feather or the rock based on THIS VIDEO?
start falling
The feather on the left falls to the ground before the rock, which violates the laws of physics.
The rock clearly reaches the ground first in the video, confiring the expected gravity and resistance.
the video, the rock will reach the ground first. This due to the principle of gravily and air resistance...
visual observations feathers fail much more slowly and land after the rock.
rock ground first.
Common Sense Reasoning Reasoning
Figure 12: Hallucination Case from Common Sense Reasoning Reasoning (CS-R). We show hallucination examples from SOTA MILM evaluations under the CS-R category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VL (Purple), and Gemini- 2.5-Pro (Blue), with hallucinations and critical context highlighted Red.
Generste man drinking cup of wine.
Does the amount of wine the glass change?
No, which violates physics, the is drinking the wine.
Visually inspecting the wine level the glass appears to gradually decrease.
No, glees change.
Yes, of glass decreases.
Physics Conservation
Figure 13: Hallucination Case from Physics Conservation (P-C). We show hallucination examples from SOTA MLLM evaluations under the P-C category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VI. (Purple), and Gemini-2.5-PP Prr (Blue), with hallucinations and critical context highlighted Red.
19

---

Generate a video showing robot collidig with an obstacl from top-down view.
the robot colliding with its surroundings?
No, the robot's passes directly through the wall without any collision, rebound, obstruction, which indicates a violation of physical laws.
To confirm whether collision occurs, we'r need more detailed malysis--espccially focusing
trajectory
interaction abjects.
While the robot does collide with its surroundings, these collisions do not block its motion but rather contribute to its overall movement...
Yes, robot collides with Its surroundings.
Physics Constraints and Properties
Figure 14: Hallucination Case from Physics Constraints and Properties (P-CAP), We show hallucination examples from SOTA MLLM evaluations under the P-CAP category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VL (Purple), and Gemini- 2.5-Pro (Blue), with hallucinations and critical context highlighted Red.
Generste free-fall sequence of an Iron ball dropped from tower.
What the mation the ball?
making pendulum like mation.
The plot shows that the ball descending smoothly. likely under the influence of gravity.
The ball moving circular motion around the stone tower. It appears to be orbiting the tower.
The stationary.
Physics Motion
Figure 15: Hallucination Case from Physics Motion (P-M). We show hallucination examples from SOTA MLLM evaluations under the P-M category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VI. (Purple), and Gemini-2.5-Pa S-Pro (Blue), with hallucinations and critical context highlighted Red.
20

---

Generatc the sequence showing a builet being shot into watermelan.
watermelon breaks video?
intact by
watermelon clearly has shattered
pieces...
video, watermelon broken...
Therefore, at the and of the action sequence depicted, the watermelon broken.
Physics State Transition
Figure 16: Hallucination Case from Physics State Transition (P-ST), We show hallucination examples from SOTA MLLM evaluations under the P-ST category. Each case includes the video generation prompt (Gray), key frames from synthetic videos (Gray), questions (Orange), ground truth (Green), and model answers from GPT-4o (Black), Qwen2.5-VI. (Purple), and Gemini-2.5-Pro (Blue), with hallucinations and critical context highlighted in Red.
21

---

B Theoretical Problem Formulation
The motivation of our work comes from the assumption that the language priors within the LLM backbone of the VLMs may interfere with their understanding of synthetic videos. Our goal is to craft a dataset of synthetic videos featuring perceptually obvious violations of common sense and physical laws that require true visual recognition to detect. Let fVLN, fLIN denote the VLM and its LLM backbone, respectively, and /uunan denote the human expert providing ground truth understanding. fvun(video, query) can take video-query pair as input, JLum(conted query) can take a text-only context-query pair as input, and (context, query) can take multi-modal inputs paired with queries. We denote V as the set of all contexts within the synthetic video V. The context denotes the context being probed during the video understanding process, where Q denotes the query probing this context C. We define a mapping function (-)) that transforms set of contextual elements into a natural language-forma formulated text for both the query Q and context C. This mapping be performed by either humans LLMs. We introduce the contextual distance di -I to quantify the semantic divergence between two contexts or texts [49]. When two contexts convey similar or mutually consistent information, d is small; otherwise, it is large. This metric captures the degree of contextual alignment and can be estimated using LLM-as-a-Judge approaches (97 or other model- based evaluators. In the post-training human preference aligument setting, regard [Human (-:)) as the ground truth and expect both fVLM and SLI to align with human perception and understanding of the real world. The objective formulated as:
d[fvanVV T())), frunan (V, T(Q))
(4)
v.Q.c s.t. d[funn T(C), (())), Muumnn T(C), (())) Se 6, d[Sunaan (C))), T(Q)), Human (V, T()) 8, CCV,
where Equation (4) maximizes the contextual distance between the VL.M's output and the human- annotated ground truth for a given synthetic video V and query Q. The constraints in (5) ensure that the language-only model fux. given the same query Q and context C, remains aligned with human judgment within a tolerance e, while the video V introduces human-detectable inconsistencies relative to C, yielding contextual distance exceeding threshold d. The context C is embedded within V to preserve coherence.
22

---

C Video Understanding and Evaluation Categorization/ooivvttonn
We provide details specific categorizations of errors video generation models make. We draw inspiration from basic video quality evaluation definitions from MVBench [43] and WorldModel- Bench [64] to first organize the current challenges of video generations and evaluations in four basic categories (Figure 2 Given the probing target of each question-answen
pair and the demand for reasoning abilities or prior knowledge of the LLM backbone solve the question provided, we divide the question-answent cub-categories.
pairs for testing MLLM-as-evaluators into four major categories with
The categorization to go beyond superficial metrics like frame consistency or resolution by enabling rigorous evaluation through the identification of visual abnormalities across predefined categories. To achieve this, we design targeted adversarial questions that expose these anomalies. This allows us to assess whether current SOTA MILMs effectively detect and interpret such issues, which is an essential step toward scalable and interpretable video evaluation. We further extend these principles define our video understanding criteria benchmark.
Alignment checks whether the model accurately identifies basic entity details and ensures the video content fully aligns with the prompt without omissions or discrepancies.
Entity Counting (A-EC): Quantifies how many entities are present in the scene.
Entity Properties (A-EP): Focuses on visual features such as color, shape, and texture that define an entity's 's appearance.
Entity Recognition and Classification (A-ERAC) Identifies and categorizes entities based on attributes like shape, color, and texture.
Spatial Relationships (A-SR): Examines the relative positions of mostly static entities as described in the prompt.
Spatial-Temporey Consistency evaluates whether the model can detect smooth, consistent changes objects, actions, and viewpoints over time, without abrupt or abnormal transitions in space or time.
Camera Dynamics (SC-CD): Covers variations in camera movement, angle, and viewpoint.
Spatial Dynamics (SC-SD): Focuses entity motion, changing positions, and interactions, identifying any inconsistencies or abrupt spatial changes.
Temporal Dynamics (SC-TD): Tracks changes in entities or scenes over time, including appearance shifts, transformations, and abnormal appearances or disappearances.
Common Sense Reasoning assesses the model's ability to apply general knowledge and reasoning to detect conflicts between common sense and the visual context, ensuring interprets the prompt correctly without hallucinating entities actions.
Knowledge (CS-K): Assesses the model's ability apply general knowledge of everyday phenomena, including object geometry, layout, and state transitions.
Reasoning (CS-R): Tests the model's ability to interpret problem cues including emotional or environmental hints, and solve them through reflection and thain-of-thought.
Physics assesses the model's ability to detect physical inconsistencies, such as violations of gravity, motion dynamics, or conservation laws, requiring careful reasoning about object properties and movements even if not explicitly stated.
Conservation (P-C): Assesses understanding of mass and energy conservation, ensuring entity quantities remain constant unless acted upon by external forces.
Constraints and Properties (P-CAP): Checks understanding of physical constraints and properties, such rigid bodies blocking motion or light behavior like reflection.
Motion (P-M): Evaluates the model's grasp of motion-related physics (like gravity, lin- ear/circular motion, relative movement, and fluid dynamics), spotting inconsistencies or abrupt changes.
State Transition (P-ST): Tests knowledge of physics-driver -rivve state changes, including heat effects, phase transitions, and dynamic interactions.
23

---

D Prompt Templates
We provide the prompt templates we use for CoT prompt (Table then generate the final answer (Video-R1-oe RI-CoT and VideoChat-RR - -thinkking) and prompt templates for generating answers directly (Table 5
CoT Prompt Template
System Prompt: conversation batvean User and Assistant. The acks a question, and the Assistant solves it. The assistant first thinks sbout the ressoning procees in the nind and then provides the user with the answer. The reasoning process and ansver are enclosed vithin <think </think> and <anewer> </ansver> tage, respectively, i.e., <think? reasoning process here </thinto<anaver> ansser here </anaver>
Input: Please
you were pendering deeply. Engage in an internal dialogue using expressions such 'let think', 'wait', 'lam', 'ch, see', 'let's break dovn', etc, or other naturel Language thought expressions. It is encouraged include self-reflection or verification the reasoning process. Provide your detailed ressoning between the <think> </think> tags, and giva your final
this queetion
batuaen the <ansver> </answer> tags.
Quastion: (Quastion)
Table 4: The prompt template for Video-RI-Col and VideoChat-RR Machhat-R-thhiee -tthinking to generate answers. This prompt encourages them to first think critically about the video and the question then generate final answer.
Direct Answer Prumpl Template
System Proupt: conversation betveen User and Aseistant. The question, and Assiatant solves it. The asstatant provide ansvers within the <answer> </ansuer> tags: <ansver> ansuer here </ </answer>
be given video question. Please provide anaver to the question based on the video enclosed by <ansver> your ansver tage.
Input:
Quastion:
(Quastion)
Ansoer:
Table 5: Direct answer directly prompts a model generate the answer without generating additional thain-of-thoughts.
LLM-as-A-Jugg' Prompt Template
You will be given a question, a reference answer, and a predicted judge correctnesa
response. You task reaponse. If the predicted reeponse is correct, please anaver predictad response incorrest, please "incorrect", Please strictly follow the output format below.
predicted
"carract".
OUTPUT PORMAT:
Judgnant:
JUDGVENT
Question: (Question)
Referance Ansver:
(Reference Ansuer)
Predicted Answer:
Predicted Reeponse)
YOUR QUTPUT:
Table 6: LLM--ssa-judde prompt template.
24

---

E Categorization Breakdown Results
We provide qualitative breakdown of results in multiple radar charts across fine-grained categories for the evaluated baselines, serving as supplementary analysis to Table
Between
Finetuned R1
Large-scale
50-C3
A-R
ASR
SC-cn
ASR
A-S
AERAC
AERACC
AERAC
AERAC
ATP
A-E
M-
MEP
SCTD
SCTTC
SCTD
SC-TD
ABC
C5.6
CSR
C5.8
C5R
PM
PC
P-C
PC
vilechal en He
Wem
0V812556-33
0v812.54-23
dac-li-CaT
Figure 17: SOTA VLM Evaluation on VideoHallu Across Sub- Sub-Categories. We evaluate SOTA VLMs on VideoHallu, with results broken down by sub-category. From left to right, we show: (a) models under 7B parameters; (b) models between 7B-38B; (c) RI fine-tuned models; and (d) large black-box VLMs. While many perform well on alignment tasks, they remain prone to hallucinations in reasvning-heavy tasks, with notably weaker performance physics and commonsense reasoning.
Previous SOTA MLLMS
Taining Separately
Curriculum Learning
100
SC-D
ASR
SCOD
A-R
AERNC
AERAC
MERAC
SC-50
SC-50
SC-5D
SC-TD
SCIT
SC-TD
PST
P-S
P-T
P-CAP
P.CAP
P-CAP
516-75
GRW-31-11 1--10
56-33-10
GRAi 23-61-11
04112.500.7
PRA4-3-70
0402-541-18
PRN 10--58-111-3
Figure 18: Evaluation Breakdown of Fine-Tuned Models. We show results for (a) previous SOTA VLMs, (b) models fine-tuned sub-datasets, and (c) models fine-tuned the full dataset via curriculum learning. Compared to the baseline (Qwen2.5-VL-7B), RFT on commonsense and physics data improves models' reasoning and overall performance in synthetic video understanding.
25

---

F Common Sense and Video-dependent Question-Answering
Our benchmark, VideoHallu, is designed evaluate MLLMs' abilities to detect abnormalities in synthetic videos-a task often confounded by hallucinations stemming from commonsense or physical knowledge embedded in their language priors. This section breaks down model performance across question types in VideoHallu, including:
Common Sense-only Questions: These can be answered using language priors alone, without relying video input. egg, What typically happens when bullet hits watermelon? (Answer: It explodes into pieces.)
Target counterfactual contexts in synthetic videos, testing whether MLLMs can recognize visually implausible phenomena. e.g. In the video (Sora), the watermelon breaks in the middle of the video. intact broken at the end? (Answer: It's intact.) (Figure
Counterintuitive Questions:
Critical Thinking Questions: Open-ended questions that ask whether MLLMs can identify abnormalities in synthetic videos, evaluating their visual reasoning. e.g- What is unusual in this video (Sora)? (Answer: The watermelon explodes, then reassembles.) (Figure
while the latter two types of questions must be answered with video inputs, so that we denote them as video-dependent questions.
Video-dependent
Model
Common Senve-unly
Ovrrlll
Counterituitive
Criticl Thinking
100,0
46.8
15.0
45.5
100.0
46.7
Denda.122 2, 5-
100.0
50.2
23.3
49.8
Video-Ba 81
100.0
52.3
16.7
50.8
Queu2. 5-VL78
100.0
53,1
10.0
51.0
Quea2. 5-V--22
100.0
52.5
51.4
Table 7: Common Sense and Video-dependent QA over VideoHallu. We divide VideoHallu into multiple categories over the question types: (a) Common Sense-only Questions, answerable via language priors without video inputs; (b) Counterintuitive Questions, probing MLLMs' abilities in detecting counterintuitive phenomena; and (c) Critical Thinking Questions, assessing MLLMs' ability detect abnormalities in synthetic videos.
In Table 7 we show the evaluation breakdown by question type for six SOTA MLLMs. All models reach 100.0% accuracy commonsense-onlk questions, indicating strong grounding in pre- pre-trained knowledge. However, performance drops on counterintuitive questions (all below 55%) and further on critical thinking questions, where model exceeds 25% accuracy, revealing major limitations in detecting and reasoning about abnormalities based on physics and commonsense.
Gemini-2. 2.5-Pro performs best critical thinking (23.3%), followed by Video-R1 (16.7%), suggesting some benefit from CoT prompting. However, CoT remains unreliable under language prior bias and does not consistently improve abnormality detection. Enhancing MLLMs' critical thinking for such tasks remains an open challenge.
Counterintuitive questions typically include contextual hints, helping models locate anomalies. In contrast, critical thinking questions are open-ended, requiring models to identify and reason about abnormalities unaided, making them more vulnerable to hallucinations when their video understanding incomplete.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state our main claim in the abstract and the introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitation in our work in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

We provide rigorous technical proof with clearly stated assumptions in our problem formulation and data collection sections.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: provide the technical detail of our work, and we will release our code and the dataset we use.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release our code and the dataset we use, with sufficient instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ips.cc//pill wttp:ccppppcccceeeeeeeeee..c.ett httpe//guideee/eeeuubbsssssssss /Cc////oddiiisssseeeeeee ssiiissiiciiiyyyycc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide sufficient details over the experiment details and data collection methods in the benchmark and experiment sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[NA]
Justification:
Our experiment does not involve statistical significance analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide necessary information in the data collection and experiment sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hattp::/neeurps: httparppp..cppublc///eettee htt.oliie//hheesuuieellllll...
Answer: [Yes]
We have read and understood the code of ethics; and have done our best to conform.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work focuses the benchmark and its potential application, which does not conduce societal impacts beyond current knowledge scope.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work centers the benchmark and its potential applications, primarily concerning generated content and its intended use, which does not pose a risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We proposely cited all the models and data we use in our work.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We will release the associated code and data, with proper instructions of its usage.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We have released the prompts used in our experiments and will also release the associated code and data.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our data collection process does not involve crowdsourcing research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We clearly state our use of LLMs in the paper, as they constitute the core methodology of our research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) hbtte:///eerrrppcccc//ooeeeeeeee /cte:///e/eeeeeecc//22/////1/. for what should or should not be described.
33


Reviews:
[{"review_id": "kqqygEEt3o", "reviewer": "Reviewer_tQNG", "summary": "- This work introduces VideoHallu, a video hallucination benchmark containing 3K QA pairs for synthetic videos created by modern video generation models. The goal is\n    \n- They define four evaluation categories, then employ human experts to create 141 adversarial prompts with which they generate 987 videos across seven video generation models.\n    \n- Question answer pairs are then created for each video by human experts and categorized into sub-categories\n    \n- They use LLM as a judge for evaluation, finding that frontier MLLMs struggle with counterintuitive questions and counterfactual entity changes\n    \n- Reasoning models do not fare better on these synthetic videos, having been trained only on real-world videos, so treating real-world rules as universal\n    \n- Propose to improve reasoning on synthetic videos via a three-stage curriculum which trains on two real-world datasets followed by the synthetic VideoHallu. They use GRPO with Answer-Equivalence BERT to assess accuracy and a BERTScore-like cosine similarity for the reward signal when training Qwen 2.5 VL\n    \n- The proposed curriculum training, especially the physics subset, improves performance on their benchmark", "questions": "Please see the weaknesses. If these evaluation limitations are addressed I would be happy to raise my score", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n- Strong dataset contribution with human-crafted prompts and adversarial QA pairs organized into a taxonomy\n    \n- Important contribution in noting MLLMs’ deficiencies on synthetic videos, despite strong performance on real-world videos\n    \n- Evaluations show that the proposed training regime improves performance over the baseline model\n    \n\n# Weaknesses\n\n- The benefits of the proposed curriculum and use of reinforcement finetuning are unclear. Therefore, one is unsure whether the gains should be attributed to the training data mixture or the training strategy\n    \n    - Curriculum learning is compared to training on single subsets. However, a better comparison would be to train the base model on a mixture of the three datasets to evaluate the usefulness of curriculum learning in this setting\n        \n    - There appears to be no SFT baseline, so GRPO’s usefulness here is unclear.\n        \n- Without knowing whether the training strategy makes a significant difference over batched SFT, the takeaways of Table 3 and Figure 5 seem to be reduced to “training on synthetic video data improves performance on a synthetic video benchmark over the base models”\n\n- No evaluations on real-world video datasets to ensure regression does not occur after training on only synthetic video data", "was_revised": true, "final_justification": "The authors have addressed my key concerns by training SFT models for comparison with RFT, training on a full data mixture to compare to their proposed curriculum, and by evaluating on real-world (non-synthetic) datasets. With these additional experiments the paper is more well-rounded. Therefore, I change my score to accept.", "rebuttal": "Thank you for the thoughtful feedback. We appreciate the recognition of our human-crafted dataset with adversarial QA pairs and taxonomy, as well as our effort to expose MLLM weaknesses on synthetic videos. We are also glad our training approach was noted to improve performance over the baseline.\n\nGiven the limitation on the rebuttal length, we include our all our supplementary experiment results in tables below, and we will cite these tables throught our rebuttal:\n\n**Table A. Ablation Study on Curriculum RFT**\n| Metric      | GRW-PRW-SR-R1-7B  | GRW+SR-R1-7B | PRW+SR-R1-7B | Qwen2.5-VL-7B | Qwen2.5-VL-7B-RFT-Mixture-Data-Shuffle | Qwen2.5-VL-7B SFT |\n|-------------|-----------------------------|--------------|--------------|---------------|----------------------------------------|-------------------|\n| A-EC        | **58.8**                    | 51.3         | 50.7         | 53.5          | 52.4                                   | 57.7              |\n| A-EP        | 62.4                        | 59.8         | 58.7         | **63.1**      | 58.9                                   | 62.2              |\n| A-ERAC      | **69.8**                    | 68.9         | 67.7         | 65.5          | 66.8                                   | 61.0              |\n| A-SR        | 59.6                        | 51.2         | 53.5         | 54.8          | 61.5                                   | **62.5**          |\n| SC-CD       | 57.5                        | 53.5         | 55.2         | 47.6          | **61.1**                               | 51.3              |\n| SC-SD       | **58.5**                    | 42.1         | 47.4         | 42.3          | 50.7                                   | 50.7              |\n| SC-TD       | 61.1                        | 47.6         | **62.0**     | 43.9          | 56.1                                   | 53.1              |\n| CS-K        | 54.2                        | 57.7         | **61.6**     | 50.8          | 57.0                                   | 55.1              |\n| CS-R        | 58.8                        | 56.4         | **62.1**     | 42.0          | 53.9                                   | 52.9              |\n| P-C         | **61.9**                    | 56.8         | 56.8         | 42.9          | 47.6                                   | 45.2              |\n| P-CAP       | **60.0**                    | **60.0**     | 56.9         | 58.8          | 55.7                                   | 55.7              |\n| P-M         | **59.3**                    | 46.2         | 44.8         | 45.6          | 57.7                                   | 48.5              |\n| P-ST        | 61.6                        | 60.0         | **80.0**     | 40.0          | 55.0                                   | 45.0              |\n| Overall     | **58.7**                    | 52.1     | 54.2     | 51.0          | 52.3                                   | 50.9              |\n\n\n**Table B. Real World Data Evaluation**\n| Model             | MMVU  | MVBench |\n|------------------|-------|---------|\n| Qwen-2.5VL-7B     | 58.7 | 69.6    |\n| SFT (synthetic)   | 57.9  | 69.6    |\n| SFT (GRW)         | 60.1  | 70.1    |\n| SFT (PRW)         | 57.9  | 69.9   |\n| GRW-R1-7B         | **61.3**  | 70.7    |\n| PRW-R1-7B         | 58.3  | 69.7    |\n| SR-R1-7B          | 59.1  | 69.3    |\n| GRW-SR-R1-7B      | **61.3**  | **70.9**    |\n| PRW-SR-R1-7B      | 59.1  | 69.7    |\n| GRW-PRW-SR-R1-7B (Qwen2.5-VL-7B-R1-CL)      | **61.3**  | 70.0    |\n\n\n\n**Response to Weakness 1:** \nThe gain of the model improvement come from curricumum reinforcement fine-tuning strategy and augmenting the right training data. We provide more clear reasons and explaination below:\n1. We present new results on non-curriculum learning by mixing physics and synthetic video data in Qwen2.5-VL-7B-RFT-Mixture-Data-Shuffle, yielding 52.3% accuracy, only 0.1% better than physics-only training, 1.1% lower than synthetic-only, and 1.9% below curriculum learning (PRW+SR-R1-7B). This suggests that just 0.8% of the gain comes from training strategy. In contrast, organizing the data using curriculum learning (GRW-PRW-SR-R1-7B) leads to a 7.7% improvement over the base model after RFT.\n\n2. Our findings show that data mixing without curriculum learning achieves 52.3% accuracy, which is lower than the 54.2% from curriculum learning (PRW+SR-R1-7B).\n\n3. From the GRW+SR-R1-7B results, we observe that mixing general world and synthetic data slightly degrades performance compared to synthetic-only training. However, integrating physics and synthetic data via curriculum learning yields superior results. Since the synthetic dataset only has 800 samples, augmenting it with physics data significantly boosts performance, highlighting the importance of strategic data combinations in training.\n\n\n**Response to Weakness 1.1**\nWe present the results on training on training on the mixed dataset with shuffling in the result of Qwen2.5-VL-7B-RFT-Mixture-Data-Shuffle. Training on mixed data has an overall accuracy of 52.3%, where training on physics only data has 52.2%, and 53.4% training on synthetic data. Combining them without curriculum learning leads to minimal performance improvement or even degradation than learning first from easy real world physics data then transition to the harder synthetic video data (54.2%). Learning merely from mixed data without increasing difficulty level would cause the model to generate wrong responses for the more challenging data thus not backpropagating on the synthetic videos.\n\n**Response to Weakness 1.2:** There appears to be no SFT baseline, so GRPO’s usefulness here is unclear.\n\nWe train the 7B model with SFT on our synthetic video understanding datasets and get the following results, with no significant improvement on our benchmark data. Since this is a challenging dataset, training the model with SFT does not lead to improvement (50.9% compared to base model 51%) because SFT simply maximizes likelihood of the ground-truth answers and does not expose the model to its own chain-of-thought. Synthetic video understanding tasks require temporal reasoning—tracking objects and causal events across frames—and SFT’s teacher-forcing objective doesn’t train the model to perform multi-step inference at test time. In addition, even the SoTA model has only 51% accuracy on the test data, if we use distilled CoT data to train the model, there will not be improvement since the model we are performing SFT/RFT is the SoTA model. Using RFT can fully leverage model's CoT process to generalize and improve temoral and physics reasoning on synthetic videos. \n\n**Response to weakness 2**: We present new results in our table and highlight the following key takeaways:\n\n**SFT vs. RFT:** Supervised fine-tuning (SFT) only optimizes the likelihood of ground-truth answers and does not involve the model's own chain-of-thought (CoT), limiting its ability to learn physics rules. In contrast, reinforcement fine-tuning (RFT) leverages the model’s CoT process, resulting in stronger performance gains on the synthetic benchmark.\n\n**Why Not Distill CoT from SoTA:** Distilling CoT data from a state-of-the-art (SoTA) model for SFT is ineffective in our case, as the SoTA model is Qwen2.5-VL-7B itself—the one being trained. This would mean training the model on answers it already knows, offering little benefit.\n\nRFT is more effective than SFT for improving synthetic video understanding in our setup, and we therefore adopt it as our primary training strategy.\n\n**Response to weakness 3**: We provided real-world video benchmark results (We used VLMEvalkit). We show that there is no big difference or degradation on real world datasets when training with synthetic data, where a very small fluctuation of accuracy (<0.5%) is expected after finetuning."}, {"review_id": "IZC1qPam0b", "reviewer": "Reviewer_bcUY", "summary": "The paper presents VideoHallu which is a benchmark to evaluate the critical (physics-based) thinking abilities of the multi-modal large language models (MLLMs). By using 3000 synthetic videos paired with counterintuitive QA, it evaluates and mitigate hallucinations in multi-modal large language models (MLLMs) where the generations can show abnormal or abnormal or counterintuitive visual phenomena (such as an object exploding and reassembling back). The paper presents an exhaustive evaluation strategy into multiple categories such as spatial-temporal inconsistency, physics violations, and commonsense inconsistencies. Multiple state-of-the-art MLLMs are evaluated based on these metrics and the paper discusses several insights such as the MLLMs fail to answer counterintuitive questions, chain of thought also underperforms, and solely based on real-world data can create a bias of how the real-world works and limiting the understanding of counterintuitive physical phenomenon. \n\nTo address this, the paper proposes a curriculum learning-based finetuning strategy using general videos, real-world physics videos, and synthetic video QA data to improve physical understanding of the models. The evaluation shows modest gains in performance through this approach.", "questions": "Will the VideoHallu dataset be publicly released with all associated QA pairs and metadata?", "limitations": "Not sure. The question 10: broader impact of the checklist has answer as [NA].", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of using counterintuitive examples (such as bullet being shot into a watermelon) is interesting.\n2. The paper is well-written and clearly explains how synthetic data can be helpful to provide counterintuitive examples and thus ensure physical plausibility.\n3. The paper presents an exhaustive set of categories and subcategories for evaluation of the physics on the basis of alignment, spatial-temporal consistency, common-sense reasoning, and physics.\n\nWeakness:\nWhile the idea of the paper of using counterintuitive questions/examples via synthetic data is sound, the results from the proposed contributions of curriculum training with general videos+real-world videos+synthetic videos and using GRPO in Table 3 doesn't seem to add much strength to the contributions and don't show much performance improvement on the proposed evaluation categories of alignment, spatial-temporal consistency, and physics. While the overall score is in favor of curriculum training using physics-real world and synthetic data, if we look at the results for each category in Table 3, for example in physics, for some metrics, training separately works better and for some metrics, curriculum training works better. The inconsistency raises some doubt as to whether curriculum learning is helpful or can training separately can be helpful in general. Similarly in alignment, the previous SoTA methods seem to show better performance majorly. Can the authors discuss why the performance is still on lower end when compared to previous SoTA methods?", "was_revised": true, "final_justification": "During the rebuttal phase, the authors addressed my concern of whether the curriculum training is helpful on all evaluation axes (alignment, spatio-temporal consistency, and physics) by showing exhaustive results. I still have some concern about the data coverage and diversity which I understand wouldn't have been possible to address during the rebuttal phase. Therefore, I would prefer to maintain my rating to borderline accept. The authors are encouraged to explore diversity in their dataset.", "rebuttal": "Thank you for the thoughtful feedback. We're glad you enjoyed the paper and found the use of counterintuitive examples compelling. We truly appreciate your recognition of the writing clarity and the role of synthetic data in promoting physical plausibility.\n\n**Response to Weakness 1:** \nThank you for your insightful doubts. Aas the results shown in Table A, We train Qwen2.5-VL-7B on mixed data without performing curriculum learning on physics + synthetic video data (Qwen2.5-VL-7B-RFT-Mixture-Data-Shuffle) and show that no curiculum learning has an overall accuracy of 52.3%, where training on physics only data has 52.2%, and 53.4% training on synthetic data. Combining them without curriculum learning leads to minimal performance improvement or even degradation than learning first from easy real world physics data then transition to the harder synthetic video data (54.2%). Learning merely from mixed data without increasing difficulty level would cause the model to generate wrong responses for the more challenging data thus not backpropagating on the synthetic videos. \n\n**Response to Weakness 2:**  Thank you for pointing out this limitation. We agree that in the current version, our reinforcement fine-tuning (RFT) with limited data improves performance on physics and commonsense reasoning but at the cost of reduced performance on alignment-related QA in VideoHallu. We attribute this trade-off to the narrow focus of our RFT dataset, which emphasizes video understanding (real and synthetic) while lacking coverage on alignment tasks. In future work, we plan to incorporate more diverse and balanced training data to enhance MLLMs’ physics/commonsense reasoning without compromising performance on alignment QA. This includes expanding data coverage and improving training efficiency.\n\n**Response to Question 1:** \nWe will release our dataset and code in the final version as one of the core contribution of our paper. Also, we are happy to provide the metadata of our dataset as below. Here are more details of the metadata that will be included in the script:\n* Avg. frames per video: 96.0\n* Avg. video length: ~5.3 seconds\n* Avg. framerate (FPS): 23\n* Avg. frame resolution: 1042 × 588\n* Number of video generation models: 7\n* Avg. videos per model: 132\n* Maximum number of frames to sample for training: 18\n* Number of training iterations: 3 epochs\n* Maximun number of frames to sample for testing: 128\n\nAll video question-answer (QA) pairs used in our benchmark will be released alongside the final version of our paper.\n\n\n**Response to Limitaion 1:** According to NeurIPS guidelines, Question 10 (potential negative societal impacts) is out of scope for our work. We propose a benchmark focused on physics and commonsense violations in synthetic video QA tasks and demonstrate its utility in improving foundation models’ video understanding through reinforcement learning fine-tuning. Our work does not involve negative societal impacts, as it aims to mitigate such issues in other systems."}, {"review_id": "dG7zvBDFu8", "reviewer": "Reviewer_DNqt", "summary": "This paper evaluates counterintuitive, commonsense, and physics reasoning in videos. It introduces a benchmark of syntetically generated videos with commonsense and physical-law violations that are perceptually clear to humans but hallucinated by MLLMs with video understanding capabilities. It then evaluates state-of-the-art MLLMs on the proposed benchmark and proposes a GRPO-based reinforcement fine-tuning to cope with this problem.", "questions": "**Q1:** The authors claim that: *after curriculum learning, MLLMs exhibit notable improvements in critical thinking and their ability to handle counterintuitive scenarios*, as stated based on Table 3. However, this conclusion is not sufficiently supported by the experimental evidence. \nSpecifically, the study evaluates RFT with curriculum learning on only a single model (Qwen2.5-VL-7B) and a single dataset (the authors' proposed benchmark).\nAs such, the **RTF results cannot be generalized to multiple MLLMs** without further validation. The authors should rephrase the claim or provide further experiments.  \n\n**Q2:** The authors claim that: *it is the quality and coverage of the data, not just the fine-tuning method, that drive gains* (L233-234). However, based on results in Table 3, it seems to me that Qwen2.5-VL-7B benefits from Curriculum Learning with composite datasets primarily in cases where **improvements were already evident under the Training Separately setting**. Specifically, the gains from Qwen2.5-VL-7B to the central part of the table are higer than those observed toward the end of it. Can the authors elaborate on this?\n\n**Q3:** Implementation details are completely omitted. Key information such as the **duration of the generated videos, the number of input frames provided to MLLMs for evaluation**, and whether frame sampling is needed (and if so, which one) is entirely missing.  These are not just details but important to understand the solidity of this work. \n\n**Q4:** The **data collection** is formulated as an optimization problem, yet the procedure seems mostly human-curated. It is not clear how the mapping T(.) is formulated. What does it mean that can be performed by either humans or LMMs? What is that exactly? Moreover, the actual hyperparameters of the constraints ($\\epsilon$, $\\delta$)are not stated, and it is not clear how they have been decided. Finally, what is the exact MLLM used in this step? I am not able to find this information, but I may be missing it. \n\n**Q5:** Why did the authors not send this paper to the **Datasets and Benchmarks** track? This paper explicitly proposes a benchmarking suite (VideoHallu) and evaluates state-of-the-art models on it. Given the existence of a dedicated track with clear guidelines, including requirements for code documentation, dataset formatting, and open-sourcing by the main deadline, this raises the question of whether this submission is best suited to the current track. At a minimum, the inclusion of code and at least a sample of the dataset in the supplementary materials would have been expected. It would be helpful to understand the authors' perspective on this decision.\n\n**Q6:** There are no information on **computing resources** used. Although in the checklist the answer to the question 8 is 'Yes', it seems that there is no information on the number and type of GPU needed for the evaluation and for the reinforcement fine-tuning. This input is needed.", "limitations": "The authors include limitations in the conclusions, however, this is a 2-line comment that could be further extended at least in the Appendix.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 1, "originality": 3, "strengths_and_weaknesses": "**Strengths:** The motivation of studying counterintuitive, commonsense, and physics reasoning in MLLMs with video understanding capabilities is interesting and well motivated. The paper evaluates several recent MLLMs, including less tested models trained with reinforcement learning or chain-of-thought supervised fine-tuning. Lastly, it proposes an approach to improve the poor performance of MLLMs on the proposed benchmark. \n\n**Weaknesses:** While the idea is interesting, several parts of the execution need, in my opinion,  improvement. Some parts of the data collection and the conducted evaluation are not clear, as detailed in the part dedicated to questions below. Moreover, although the final section of the paper introduces a methodology aimed at enhancing MLLM capabilities, it is evaluated on a single model with results that do not convincingly support the claims made.\nBoth main Tables 1 and 2 are complemented with Figures that, in my opinion, are useless. Figure 3 shows exactly the same numbers in Table 2, yet without the y-axis range, so it contains even less information. The same applies to Figure 5 and Table 3. I suggest removing these figures and using the space more cleverly. \n\n**Minor Weaknesses:** \n- *Black Box* in Table 2 may be an inappropriate name. I suggest changing it to closed-source or proprietary. \n- Unify LVLM and MLLM inconsistency: in the problem formulation in L87-99 LVLM is used while in the rest of the paper MLLM. \n- SOTA is missing the acronym, and sometimes it is written as SoTA (L133).", "was_revised": true, "final_justification": "During the rebuttal phase, the authors addressed my concerns and provided additional results using new MLLMs. However, I remain unconvinced that the benefits of curriculum learning are broadly demonstrated; improvements appear mainly in cases where gains were already observed under the ‘training separately’ setting. This issue was also raised by reviewer bcUY. Considering the new results, I will raise my score, as the current version no longer warrants a 2. Nevertheless, I believe the paper requires further development and is not yet ready for full acceptance.", "rebuttal": "**Response to Weakness 1 :** Please see our response to the questions below.\nWe run an additional different model (internVL 9B) and show the same conclusion that curriculum learning can improve MLLMs' reasoning abilities on synthetic video understanding. See Table I for reference.\n\n**Response to Weakness 2 (Figures Layout):** \nFigures 3 and 5 (radar charts) are intended to provide a qualitative comparison across sub-categories for each MLLM, as the corresponding tables (Tables 2 and 3) present complex numerical results that are not easy to interpret. We agree to move these figures to the appendix in the final version to make space for additional content in the main paper.\n\n**Response to Weakness 3-5:** Thank you for pointing out the typos and term misuses in our paper (Black Box => Closed-Source, LVLM => MLLM, and SOTA => SoTA). We will correct them in the final version.\n\n**Response to Question 1:** \nOur task involves using MLLMs for video understanding and question answering, specifically targeting physics and commonsense abnormalities. As a novel task, there are no existing datasets addressing this specific setting, video QA on physics violations in synthetic videos. Our paper is the first to introduce this problem, release a benchmark, and conduct both evaluation and fine-tuning experiments. Thus, it's not possible for us to provide evaluation on other datasets over exactly the same task.\n\nImportantly, our fine-tuning experiments aim to demonstrate the feasibility of the benchmark and point toward future directions for improving MLLMs. We are also happy to conduct additional experiments on  **relevant** benchmarks, such as MMVU [1] and MVBench [2], using our fine-tuned models to show that our training approach maintains generalization on broader video QA tasks. The evaluation results are presented below, using model names consistent with those in the original paper. **Refer to Table B in reviewer tQNG's console**\n\nResults on real-world datasets show that models fine-tuned with curriculum learning, whether using a composed or single training dataset for RFT, show little performance difference. Notably, the GRW-PRW-SR-R1-7B model, which is fully fine-tuned with all three datasets in a curriculum learning setup, achieves performance close to the model trained solely on real-world videos. Given its state-of-the-art results on synthetic video understanding and physics comprehension, this highlights the importance of incorporating real-world videos in RFT post-training to avoid performance degradation on general video tasks when using physics-focused data.\n\nTo evaluate model feasibility beyond our current SoTA open-source model (Qwen2.5-VL-7B), we also present overall results on InternVL3-9B below, using our latest synthetic video dataset.\n\n| Metric   | InternVL3-9B-R1-CL | Qwen2.5-VL-7B-R1-CL | InternVL3-9B | Qwen2.5-VL-7B |\n|--|-|-|--|--|\n| Overall  | 58.2  | **58.7**     | 46.4         | 51.0           |\n\nAccording to the table, both RFT and curriculum learning yield notable improvements across all VideoHallu subcategories when performing RFT on both base MLLMs, demonstrating the generalizability of our training methods across MLLMs.\n\n[1] Zhao, Yilun, et al. \"Mmvu: Measuring expert-level multi-discipline video understanding.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Li, Kunchang, et al. \"Mvbench: A comprehensive multi-modal video understanding benchmark.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n**Response to Question 2:**\n\nWe present comprehensive results on our curriculum learning setup and training data compositions. Due to rebuttal word limits, detailed evaluations are provided in our response in **reviewer tQNG's console**. Our findings show that curriculum learning significantly outperforms baselines using only one or two datasets for RFT. While RFT on a single dataset can improve performance on specific tasks, it often harms performance on others. Moreover, for complex tasks requiring high-level comprehension, direct RFT without intermediate steps may be ineffective. Curriculum learning enables MLLMs to tackle such tasks more efficiently through gradual, step-by-step learning.\n\nWe also run mixed data training without curriculum learning and show that it has worse results (52.3%) than training just on synthetic data (53.4%) or training them with curriculum learning (58.7%).\n\nMeanwhile, we are actively running more experiments comparing the MLLM's performance after RFT by composed data across all three datasets and each identical dataset under Training Separately setting. Please stay tuned for more results over the discussion period.\n\n\n**Response to Question 3:** \nThank you for highlighting this weakness. Below are the implementation details:\n* Avg. frames per video: 96.0\n* Avg. video length: ~5.3 seconds\n* Avg. framerate (FPS): 23\n* Avg. frame resolution: 1042 × 588\n* Number of video generation models: 7\n* Avg. videos per model: 132\n* Maximum number of frames to sample for training: 18\n* Number of training iterations: 3 epochs\n* Maximun number of frames to sample for testing: 128\n\n**Response to Question 4:** The problem formulation section models the hallucination triggering mechanism in MLLMs, not just data collection. It outlines how both human annotators and MLLMs can generate QA pairs for synthetic videos with physics or commonsense violations, following the setting in Figure 1.\n\nThe mapping function $T(\\cdot)$ represents a question-generation mechanism that, given an entity $C$ (e.g., a watermelon in Figure 1), produces either a natural language question $T(Q)$ (e.g., In the video, the watermelon breaks in the middle. Is it intact or broken at the end?) or a textual description $T(C)$ of the event involving that entity (e.g., A bullet shot into the watermelon), possibly involving physics or commonsense violations.\n\nIn practice, $T(\\cdot)$ can be executed by human annotators, who identify a context $C$ in video $V$, then formulate both $T(Q)$ and $T(C)$. Alternatively, MLLMs can generate these when provided with a known violation. **While MLLMs cannot yet fully automate this process, they show promise in crafting QA pairs for synthetic videos when supplied with abnormal contexts, effectively serving as $T(\\cdot)$.**\n\nThe first constraint, $d[f_\\mathtt{LLM}(\\mathit{T}({\\mathcal C}), \\mathit{T}({\\mathcal Q})), f_\\mathtt{Human}(\\mathit{T}({\\mathcal C}), \\mathit{T}({\\mathcal Q}))]\\leq \\epsilon$, ensures that when only textual context $T(C)$ and question $T(Q)$ are given, the LLM’s response aligns semantically with human commonsense or physics knowledge. The second constraint, $d[f_\\mathtt{Human}(\\mathit{T}({\\mathcal C}), \\mathit{T}({\\mathcal Q})), f_\\mathtt{Human}(V, \\mathit{T}({\\mathcal Q}))]\\geq \\delta$, requires that hallucination cases lead to a significant difference in responses when humans view the actual video $V$, indicating a semantic gap of at least $\\delta$. We measure these similarities using GPT-4 as an LLM-based evaluator. When framed as a binary classifier (same vs. different), both thresholds are implicit but can be estimated with high confidence given sufficient data.\n\n\nExamples are provided in Appendix C. Overall, our goal is to rigorously model hallucination in synthetic video understanding and guide future MLLM improvements.\n\n\n**Response to Question 5:** \nOur contribution goes beyond merely proposing a benchmark. We uncover a novel problem in synthetic video understanding: hallucinations in detecting physics and commonsense violations. In addition to introducing the benchmark and evaluating state-of-the-art MLLMs, we propose a curriculum learning-based training method to mitigate this issue. Our results show that even a small amount of synthetic video QA data can effectively help models improve in synthetic video understanding, suggesting a promising direction for future synthetic video hallucination mitigation.\n\nMoreover, our curriculum pipeline offers an alternative approach to fine-tuning MLLMs on complex tasks using limited, high-cost data by leveraging more accessible, simpler datasets to guide training. These contributions align with the Deep Learning sub-track of NeurIPS (e.g., architectures, generative models, optimization, foundation models, LLMs), which we selected upon submission.\n\nRegarding NeurIPS data/code disclosure policy, we were finalizing cleanup to prevent identification or double-blind policy violations at submission time. We will published our dataset and code for full reproducibility and include the link in the camera-ready version, **as we are not allowed to including any link in our NeurIPS rebuttal.** All necessary implementation details are provided.\n\n**Response to Question 6:** We are using 8 A100 (80GB) for our SFT, GRPO fine-tuning and evaluation inference. We are using Video-R1 codebase [1]. We will include the computation resource in the final version. Thank you for pointing out this issue.\n\n[1] Feng, Kaituo, et al. \"Video-r1: Reinforcing video reasoning in mllms.\" arXiv preprint arXiv:2503.21776 (2025).\n\n\n**Response to Limitation 1:**  Our main limitation lies in the benchmark’s scalability, as both generating high-quality annotations and fine-tuning MLLMs at scale are expensive. The benchmark data were collected via paid APIs and expert-level human annotations, which are costly and time-consuming. Once more data gather and annotated with either human or MLLM automatic pipeline, future MLLMs are less likely hallucinated when probing physics/commonsense violations in synthetic videos. Another limitation, as noted by Reviewer DNqt, is the limited coverage of MLLMs used for fine-tuning. While we included InternVL3-8B fine-tuning results in our rebuttal, we are happy to expand the experiments over both currrent open-source video understaning MLLMs in the final version.\n\nWe will also elaborate further on these limitations in the Appendix."}]

Author rebuttal:

Meta-review: This paper presents a video-QA benchmark to evaluate MLLM's capabilities to identify hallucinations in generated videos, as well as a training strategy to improve MLLM performance in such a setting.  The final reviews are mixed, with one accept, one borderline accept and one borderline reject. The main source of contention seems to be the experiments (or lack thereof) to strongly support the idea, especially with respect to the curriculum learning.

Having read the reviews, responses and reviewer discussion, the AC recommends the paper to be accepted. The benefits outweigh the weaknesses and the work will make for an interesting addition to the program. That having been said, the authors are kindly requested to incorporate their additional experiments and explanations throughout the discussion into the camera-ready.
Decision: Accept (poster)