Impossible Videos
Authors: Zechen Bai
Hai Ci
Mike Zheng Shou
Venue: ICML 2025 poster (2025)
Keywords: Videos
Benchmark
Evaluation
Impossible Videos
Counterfactual
Anti-reality
Primary area: applications->computer_vision
TL;DR: This work present a comprehensive benchmark of impossible videos and text prompts, aiming to evaluate sota video generation and understanding models.

Abstract: Synthetic videos nowadays is widely used to complement data scarcity and diversity of real-world videos.
Current synthetic datasets primarily replicate real-world scenarios, leaving impossible, counterfactual and anti-reality video concepts underexplored. This work aims to answer two questions: 1) Can today's video generation models effectively follow prompts to create impossible video content? 2) Are today's video understanding models good enough for understanding impossible videos?
To this end, we introduce *IPV-Bench*, a novel benchmark designed to evaluate and foster progress in video understanding and generation. *IPV-Bench* is underpinned by a comprehensive taxonomy, encompassing 4 domains, 14 categories.
It features diverse scenes that defy physical, biological, geographical, or social laws. Based on the taxonomy, a prompt suite is constructed to evaluate video generation models, challenging their prompt following and creativity capabilities. In addition, a video benchmark is curated to assess Video-LLMs on their ability of understanding impossible videos, which particularly requires reasoning on temporal dynamics and world knowledge. Comprehensive evaluations reveal limitations and insights for future directions of video models, paving the way for next-generation video models.

Impossible Videos
Zechen Bai"
Hai Ci
Mike Zheng Shou
Generated
Impossible Type:
Physical Laws Conservation Laws
Explanation:
The incomplete cookie time, which is impossible.
Generated by Kling
Impossible Type:
Physical Laws Material Properties
Explanation:
The knife cuts towards o'clock, but the actually
o'clock. Such Inconsistency impossible.
Collected Community
Impossible Type:
Blological Anthropomorphism
Explanation:
A video of fried egg speaks, We cannot tell video impossible looking single frame, but tell watching the video.
Generated HunyuanVideo
Impossible Type: Biological Laws -> Morphology
Explanation:
grows the sunflower. Such specle fusion is impossible.
Generated by HailuoAI
Impossible Type:
Social Laws Commonsense
Explanation:
A truck emerges from underground hole and then the hole closed, which goes against social commonsense.
Generated by Sora
Impossible Type:
Geographical
Explanation:
The Merlion Park in Singapore experienced snowfall, which impossible tropical country equator.
Figure I: Impossible Video Examples with Impossible Type and Explanation.
Abstract
sible, counterfactual and anti-reality video con- cepts underexplored. This work aims answer two questions: 1) Can today's 's video generation models effectively follow prompts to create im- possible video content? 2) Are today's video understanding models good enough for under- standing impossible videos? To this end, we introduce IPV-BENCH, novel benchmark de- signed evaluate and foster progress in video understanding and generation. IPV-BENCH is underpinned by comprehensive taxonomy, en-
Synthetic videos nowadays widely used complement data scarcity and diversity of real- world videos. Current synthetic datasets primar- ily replicate real-world scenarios, leaving impos-
"Equal contribution 'Show Lab, National University of Sin- gapore, Singapore. Correspondence to: Mike Zheng Shou <mike.zheng shou @ @ew.mmlllmmm........
Proceedings of the n2nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Impossible Vidcos
compassing 4 domains, 14 categories. It features diverse scenes that defy physical, biological, geo- graphical, or social laws. Based on the taxonomy, a prompt suite is constructed to evaluate video generation models, challenging their prompt fol- lowing and creativity capabilities. In addition, a video benchmark is curated to assess Video- LLMs on their ability of understanding impos- sible videos, which particularly requires reason- ing temporal dynamics and world knowledge. Comprehensive evaluations reveal limitations and insights for future directions of video models, paving the way for next-generation video models. Project page: https: / hho.la.......................ab. github. io/ /ts/Imoossinn sitppsssibleeiieess...
aesthetic quality, motion smoothness) advanced semantic quality (e.g.. physical laws, subject consistency) (Bansal et al., 2024). State-of-the-art models have been positioned as world simulator (OpenAI, 2025). Ideally, the model should be able to generate either physically coherent or anti-reality videos with detailed control of the text prompt, enabling wider range of applications, such as filming, ad- vertising, etc. Prompting the model to generate impossible videos challenges the model to "break the rule" yet faithfully following the prompt.
Second, are today's video understanding models good enough for understanding impossible videos? Advanced video understanding models, especially video large lan- guage models (Video-I.L.Ms), are mostly trained on large- scale video datasets, demonstrating remarkable performance popular benchmarks with real-world videos. Impossi- ble videos pose specific challenges on reasoning temporal dynamics and world knowledge.
1. Introduction
Video data has been a long-standing focus in the research community, offering the potential to capture richer and more structured information compared to text (Yang et al., 2024a; Liu et al., 2024c). Over time, this domain has expanded from early tasks like action recognition (Zhu et al., 2020) to more advanced applications such as video captioning (Bai & An, 2018), question answering, video generation (Ning et al., 2023), and video-based world modeling (Liu al., 2024c), showcasing its versatility and growing importance in advancing AI capabilities.
To achieve these goals, we propose a benchmark, IPV- BENCH, focusing on ImPossible Videos. To the best of our knowledge, this is the first work focusing on this topic. We start with building comprehensive taxonomy covering diverse aspects of impossible videos, including scenes vio- lating physical laws, biological laws, geographical laws, and social laws. Based on the taxonomy, we construct IPV-TXT, a prompt suite that consists of 260 text prompts; IPV- VID, a video set with 902 high-quality vidcos. Both the text prompts and videos are organized following the structure of the taxonomy, describing or displaying impossible scenes with particular consideration on temporal dynamics. Exam- ples of impossible videos are shown in Fig. 1.
Although video data seems to be abundant on the Inter- net, it suffers from issues of low quality and diversity, etc. Recent efforts have been trying to alleviate these issues by generating videos either from neural generation mod- els or simulation engines (Agarwal et al., 2025). However, the primary goal is still to replicate real-world scenes in controlled way, which severely limits the broader imagina- tion space and applications. In this work, we introduce the concept of impossible videos, which particularly captures counterfactual and anti-reality scenes that are impossible in real world. We argue that impossible videos can form an effective testbed assess video models. As an out-of-reat world-distributtin data, it requires the model not simply memorize real-world data and retrieve similar information based on the input, but to genuinely learn from real-world data and reason upon the input.
Based on this benchmark, we conduct comprehensive evalu- ations for mainstream video understanding models and gen- eration models, suggesting that most video models fall short on impossible videos. Extensive analysis further reveals several insights of certain limitation and future direction. We will make the data public to inspire future research. In summary, our contributions includes:
To the best of our knowledge, this is the first work to identify and investigate impossible videos, which explores a critical yet absent space for advanced video understand- ing and generation research. To this end, we construct a benchmark IPV-BENCII.
Currently, mainstream video models can be categorized into two majoy categories based on tasks: understanding and generation (Xie et al., 2024; Wu et al., 2024). Thus, this work aims probe the boundary of the two types of models by answering the following questions.
IPV-BENCH is underpinned by a comprehensive taxon- omy. Based on the taxonomy, a prompt suite and a set of high-quality videos is collected and carefully annotated, supporting downstream evaluations.
Based on DV--BNCHH we conduct extensive evaluation and analysis on mainstream video understanding and gen- eration models, unveiling current limitations and revealing future directions.
First, can today's video generation models effectively follow text prompts to create impossible video content? Recent research focus of video generation has been evoly- ing from fundamental video quality (Ning et al., 2023) (e.g.,
2

---

Impossible Vidcos
Table I: Comparison of IPV-BENCII and Existing Benchmarks.
Tasks
Video Data
Text Data
Benchmark
AIGC Detection
Video Understanding
Video Generation
Real World Videos
Generated Videos
Impossible Videos
Text Prompts
Text Descriptions
Impossible
Text
GenVideo (Chen et al., 2024a)
X
X
GenVsdBBenh al., 2025)
d
LOKI (Ye et al., 2024)
/
X
/
/
x
V
K
VBench (Huang et al., 2024)
X
X
/
x
x
X
X
x
VideoPhy (Bansal et al., 2024)
X
x
/
X
PhyGenBeach (Mong
2024g)
X
SFFDDBBench (1.i al. 2023)
K
8
Video Bench (Ning el al., 2023)
X
X
X
X
X
MV Bench (1. el al., 2024)
X
X
TempCompass
2024d)
IPV-Bench (Ours)
2. Related Work
ers. Proprictary models include Kling (KLING, 2025), Sora (OpenAI, 2025), and Hailuo (Hailuo, 2025), among others. One of the primary challenges video generation is achieving high visual quality, including factors such as resolution, realism, and temporal consistency (Huang et al., 2024). To address this, benchmarks such VBench (Ning et al., 2023) comprehensively evaluate various aspects of visual quality. However, with advancements in video gen- eration models, research focus has shifted towards ensur- ing semantic coherence, particularly in maintaining adher- ence to physical laws. Recent benchmarks such as PhyGen- Bench (Meng al., 2024a) and VideoPhy (Bansal al., 2024) have emerged to evaluate models' ability to generate physically plausible videos. Beyond physics-constrained generation, an ovten-overlooked aspect is physics-defying content, or more broadly, the generation of impossible scenes, which plays a significant role in creative domains such as film and advertising. Creativity is tangentially re- lated to this challenge: however, it is only briefly considered in some comprehensive benchmarks (Zeng et al., 2024) and has yet to be systematically studied.
This work evaluates video models across two key domains: video understanding and video generation. Tab. 1 presents comprehensive comparison of [PV-BENCH and existing benchmarks, highlighting their relationships and key differ- ences. We then outline the primary objectives of existing models and benchmarks in each domain.
Video Understanding remains a fundamental challenge in computer vision, encompassing tasks such as action recogni- tion (Zhu al., 2020), object localization (Fan al., 2023; Bai et al., 2025), tracking (Zhao et al., 2023b), temporal grounding (Lin et al., 2023b), captioning (Wang et al., 2020; Bai et al., 2021), 1), and, more recently, Al-Generated video detection (Ye et al., 2024). Video Large Language Mod- els (Video-LLMs) (Tang et al., 2023), powered by Large Language Models (LLMs) (Zhao et al., 2023a), leverage language as a universal interface to facilitate wide range of video-related tasks. Most open-source Video-LL.Ms extend from multimodal large language models (MLLMs) origi- nally designed for images, such as LLaVA (Liu et al., 2024b). Some closed-source models, such as GPT-4o (Hurst et al., 2024), although initially designed for images, can also func- tion as video models by processing multiple frames as input. Popular benchmarks such as VideoMME (Fu et al., 2024), MV-Bench (Li et al., 2024), assess models range of tasks (e.g., multiple-choice and open-ended questions) and scenarios (e.g.. daily life, sports, and films). These datasets primarily consist of publicly available videos sourced from the internet. To our knowledge, no existing benchmark explicitly includes dedicated set of impossible or counter- factual videos for evaluation, which are crucial for assessing model generalization, robustness and reasoning abilities.
3. IPV
We first develop a taxonomy that systematically categorizes various types of impossible scenes. This taxonomy serves as the foundation for two critical components of the bench- mark: 1) IPV-TXT, suite of high-quality text prompts describing impossible scenes that cannot occur the real world. 2) IPV-VID, a curated collection of high-quality videos depicting impossible scenes, each with correspond- ing annotations. An overview of the taxonomy and the roles of its components is presented in Fig. 2.
Video Generation has gameerdd significant attention in both academia and industry, with text-to-video genera- tion serving as a foundational task (Wu et al., 2023).
3.1. IPV Taxonomy and Prompt Suite
Overview. As illustrated in Fig. 2. the taxonomy is struc- tured around four major categories: "Physical Laws", "Bio- logical Laws", "Geographical Laws". and "Social Laws". Each category is divided into multiple subcategories, which are further subdivided, forming a detailed hierarchical struc-
Notable ooen-source models include Stable Video Diffu- sion (Blattmann al., 2023), CogVidX (Yang et al., 2024b), Open-Sora (Zheng et al., 2024a), Show-1 (Zhang et al., 2024), and Hunyuan Video (Kong et al., 2024), among oth-
3

---

Impossible Vidcos
Conservation Lowe
Impossible Video Generation
Mechanics
Physical Laws
Oytics
IPV-Txt Prompt Suite
T2V Models
Lluman Annotation
Fluid
Auto-Eival
Material Properties
Binlogical Cxpubility
IPV Taxonomy
Biological Laws
Morphology
Anthopomurphiio
Climate and Weather
IPV-Vid Videos
Video- LLMs
Judgment Multi-choice Open-end QA
Geographical Laws
Terrain and Envinonment
Gemini
w
Open-imutee
Commoneense
Social Laws
Magioal Effeets
Impossible Video Understanding
Historieal
Benchmark. IPV-BENCH is structured with a comprehensive taxonomy, enabling the creation of diverse prompt suite (IPV-TXT) and a high-quality video dataset VID). These components facilitate the evaluation of popular video generation and understanding models.
Figure 2: Overview of the
ture. Building upon this hierarchy, our IPV-TXT benchmark incorporates 260 high-quality text prompts that describe var- ious counterfactual and other unusual scenarios.
and the cup immediately breaks into pieces." " 3) Histor- ical anomalies highlight a interesting type of scene that displays impossible combinations of human-object or human hummnn across different historical periods. For example, "Albert Einstein and Donald Trump are shak- ing hands at the White House".
1. Physical Laws" covers common laws: Mechanics, Thermal, Optics, Fluid, Material Properties, and Con- servation Laws. This categorization considers most physical phenomenons. We instantiate text prompts to explicitly describe scene defying a specific physical law. For example, "A person pours milk into glass cup half filled with milk, but the amount of milk in the glass cup does not change at all" describes a video that violates the law of conservation of mass.
Construction.
We propose a comprehensive framework for developing both the taxonomy and the IPV-TXT bench- mark. During the development process, the taxonomy struc- ture and text prompts evolve together, informing and refin- ing each other.
1) Initialization. We establish the initial taxonomy structure by reviewing relevant literature (Bansal et al., 2024; Meng et al., 2024a;b; Zheng et al., 2024b) and leveraging large language models (LLMs), such as GPT-4o, for additional insights. Beyond the widely studied domain of physical laws, we expand our taxonomy include broader categories, incorporating everyday scenarios and intriguing effects that can be applied in downstream applications.
2. "Biological Laws" categorizes 3 sub-categories: 1) Bio- logical Capability covers scenes that exceeds human or animals capabilities. For example, "A person flapped his arms like wings and successfully flew into the sky." ; 2) Morphology consider impossible body composition. E.g.. "A horse walks on the grassland, gradually grow- ing from four legs to eight legs"; and 3) Anthropomor- phism includes non-living objects exhibit anthropomor- phic behavior, e.g., "A fried egg with a face on it is opening its mouth and speaking something".
2) Iterative Refinement. We refine the taxonomy and prompts through an itcrative process. At each itcration, we collect a set of text prompts to expand the existing tax- onomy structure. If certain text prompts do not fit within the structure, this suggests the need for adjustments the taxonomy, ensuring that both the taxonomy and prompts evolve together. New prompts drive the taxonomy toward greater comprehensiveness. while the updated taxonomy encourages human to explore more intriguing examples.
3. Geographical Laws" " considers the impossible phe- nomenons in natural environment, including Climate and Weather anomalies, and Terrain and Environmen- tal anomalies. For example, "A mountain flattens into perfect plateau, leaving all its trees and snow intact on the new flat surface".
4. " "Social Laws" " includes types of counterfactual phe- nomenons violating social laws: 1) Commonsense de- fines unusual scenes that violates our daily routine or customs. For example, "The programmer in front of the computer suddenly sturted eating the keyboard and quickly ate half of if". 2) Magical Effects emphasize creative content and effects that cannot be easily inter- preted by certain scientific law, For example, "e.g., A hand turns on a flashlight and shines it on a glass cup,
LLM Refinement. We carefully design prompts to guide various LLMs in generating a broader range of example scenarios. We encourage the models to generate original and creative examples rather than merely substituting nouns in existing examples. We employ three LIMS L.LMMsnamlly GPT- 4o, Claude-3.5, and Gemini-1.5-a maximize diversity.
Crowdsourcing Refinement. To further improve the taxon- omy, we aim to collect more "outlier" samples to challenge

---

Impossible Vidcos
Mase
Enseug
Porisrette
This methodology yields a robust and comprehensive taxon- omy, and prompt suite, which can be used for assessing T2V models' capability on generating impossible videos, providing a valuable tool for advancing this domain. The distribution of the prompt suite is shown in Fig. 3.
Chjest
W eather
Cimate man
Tereain & Pavrrn cereenmmn
Conseratte moti
Loues
Genvity
Burnancy
Magional
Priction
Special Geo
Dinarale chongee
Andiro
Lone
Mechanics
Mespholous
3.2. VIV
Kinematiio
Ansmalies
Loods
hnlogioon
Plysirad
Lores
We construct IPV- VID, novel video benchmark designed to assess the capabilities of popular VideoLLMs in rea- soning about temporal dynamics and world knowledge us- ing impossible videos. IPV-VID is structured according to the established taxonomy. Given the unique nature of video data, IPV-VID places particular emphasis on temporal anomalies, such motion and stage changes, which are challenging to identify from individual frames alone.
Binlagionl Copalallay
Themnoi
Non-humon
Reffection de Reffrcctin
Oypias
Human
Materaa Propertes
Transprracy
Fund Firak
Ofject Internaient
Calor & Speatrum
Alnormal
Phoye caniiiion
sidi & emectto
Voltomi
Share
bntencced
Figure 3: Distribution of the Prompt Suite Across the Tax- onomy.
3.2.1. VIDEO COLLECTION.
and refine the current structure until it accommodates the majority of samples. To this end, employ crowdsourcing to collect data samples from diverse set of contributors. We employ a questionnaire (see Appendix Fig. 10) to collect data from 15 participants from diverse academic and profes- sional backgrounds, including computer science, economics, arts, and education. The participants have acknowledged that the collected data will be used for research purposes. On average. participants required 31 minutes to complete the questionnaire. This time commitment demonstrates the complexity and value of the collected text prompts. Finally, we have collected 150 impossible text prompts, which con- tribute to refining the taxonomy. It is essential to explicitly specify "LLMM are not allowed" in the questionnaire to guarantee the authenticity of collected data samples.
T2V Generation. We prompt 10 ofate-of-the-art T2V models to generate a comprehensive set of videos, in- cluding open-sourced models (Open-Sora (Zheng al., 2024a), Hunyuan Video (Kong et al., 2024), CogVidX (Yang et al., 2024b), Mochi (GenmoAI, 2025), LTX (GenmoAI, 2025), and Pyramid-Flow (Jin et al., 2024)) and closed- sourced models (Sora (OpenAI, 2025), Kling (KLING, 2025), Luma (LumaLabs, 2025), and Hailuo (Hailuo, 2025)), Using 260 high-quality text prompts from the IPV-TXT suite, we generate total of 2,600 synthetic videos.
Web Video Collection. To enhance the scale and diversity of the benchmark, we supplement our dataset with videos collected from the Internet. The primary sources include community websites for commercial video generation mod- els, such as Sora and Hailuo, among others. Additionally, we gather videos from Twitter (X) shared by users, explicitly mentioning that they were generated by specific Al mod- els. During the manual collection process, we adhere an implicit criterion: videos must exhibit phenomena that are impossible in the real world. From these sources, we collect total of 155 videos.
We conduct thorough review of the prompts across multiple dimensions. Clarity. The text prompts should be expressed in a clear and comprehensible manner. Complicated or confusing prompts will be manu- ally revised or removed. Accessibility. The described scene should be accessible in daily life. We exclude some obscure or esoteric samples. Relevance. The text prompt should be relevant its belong category (and sub-category) in the tax- onomy. We also merge some similar duplicated samples at this step. Visualizability. The described scene should be able to be displayed by a video. Some non-visual scenes, such as sound-related descriptions, will be excluded.
3) Quality Control.
Real-world Videos. To mitigate evaluation bias and ensure balanced distribution, we incorporate real-world videos into the benchmark. These videos are used alongside syn- thetic videos to evaluate the performance of VideoLLMs. We utilize Open Vid (Nan et al., 2024) as the foundational dataset for real-world videos. Using Al-generated videos as queries, we leverage the CLIP (Radford et al., 2021) model to retrieve content-consistent videos from Open Vid. During retrieval, we filter out unsuitable videos based on video length, aspect ratio, and aesthetic score. Additionally, we exclude videos with cartoon-style visuals, conspicuous logos, subtitles, and similar features. Totally, we collect 650 real-world videos to include in the benchmark.
4) Linguistic Enhancement. We further refine the linguis- tic clarity and expressiveness of the text prompts to enhance their suitability for video generation models. Specifically, we reference popular prompt rewriting strategies (Kong et al., 2024; Yang et al., 2024b) and instruct GPT-4o to enhance the text prompts. Compared to the original de- scriptions, the rewritten prompts feature relatively longer sequences, greater detail, and a more structured format.
5

---

Impossible Vidcos
Number of Videos from Different Sources
and exclude real-world videos with cartoon-style visuals, conspicuous logos, subtitles, or similar features. To ensure balanced evaluation, the dataset maintains a 1 : ratio of synthetic to real-world videos. This task is framed as a binary classification problem and evaluated using average Accuracy and Fl-score. Additionally, we report the 'yes rate" in Appendix B. to facilitate model diagnosis.
160
140
of Videos
120
100
80
60
Multi Choice Task (MCQA) task requires models iden- tify the description that best captures the impossible phe- nomenon depicted in the video. The question is formulated as follows: "Select the best answer to the following multiple- choice question based on the video'. To create effective distractors that challenge the model, we carefully design an instructional prompt and leverage the GPT-4o model for distractor generation.
40
Interett
Sora
Kinng
Hailuo
HunyuanVidis Vidao
Luma
Mochi 1
PyramidFlow PFow
Open- Sora
CogViix
LTx
Figure 4: Sources Impossible Videos.
3.2.2. HUMAN ANNOTATION.
The instructional prompt is designed with the following con- siderations: 1) Ensure all options, including both correct answer and distractors, are similar in length, style, detail degree, and complexity; 2) Ensure distractors also present specific impossible phenomenon; 3) Ensure the impossi- ble phenomenon in distractor shall involve visual elements shown in the given video frame, avoid the model solve the problem with simple visual element grounding. By incorpo- rating detailed annotations and visual content as references, we mitigate hallucination in GPT-4o, ensuring high- distractor generation. A qualitative example of distractors is provided in Fig. 6. The instructional prompt is included in Appendix C. The MCQA task is treated as a multi-class classification problem and evaluated using mean Accuracy.
Step I: Video Filtering. Our objective is to curate collec- tion of high-quality videos that depict impossible or counter- factual scenes. To ensure data quality, we develop custom annotation tool and perform human annotation on the col- lected videos. A detailed description of the annotation tool provided in the Appendix A. The main criteria of IPV- VID contains two aspects: 1) Visual Quality. Videos of low visual quality are excluded, including but not limited to spatial blurring, temporal flickering, poor aesthetic quality, insufficient motion, and inconsistent style. 2) Semantic Plau- sibility. In line with the benchmark's 's objective, we retain videos with semantically counterfactual content- scenes that are impossible in the real world.
Open-ended QA Task (OpenQA). We introduce open- ended impossible explanation task, which requires models to independently and correctly identify the impossible phe- nomenon depicted in the video without any hints. The task is framed with the following question: "Based on your ob- servation of the video, what content or event makes the video impossible or unusual in the real world?". Compared to the MCQA task, the open-ended explanation task is more challenging, as models must generate responses without reference to candidate options. This task more accurately as- sesses whether models can genuinely perceive and articulate detailed anomalies. rather than relying on guesswork.
This ensures that the selected videos exhibit high visual quality while maintaining low semantic plausibility, thereby requiring models to reason based on semantic content rather than low-level visual features. Fig. 4 illustrates the distribu- tion of retained impossible videos after filtering.
Step 2: Detailed Annotation. For videos satisfying the aforementioned criteria, we perform detailed annotation with the following labels: 1) Spatial or Temporal Anomaly. This field requires annotators to determine whether the im- possibility can be identified through spatial semantic infor- mation or necessitates temporal reasoning; 2) Taxonomy Category. Annotators assign a category label based on the IPV taxonomy; 3) Explanation. Annotators provide brief textual description of the specific impossible phenomenon depicted in the video.
For evaluation, we employ an LLM as an evaluator to score model responses by comparing them to the annotated text observe
explanations in the benchmark. Empirically, that directly instructing the L.MM to assign scores results in instability. To address this, we propose justification-tiee score approach. In this approach, the evaluator first provides a justification by identifying key matches or mismatches be- tween the model's prediction and the ground truth. Based on this justification, the evaluator assigns semantic alignment score a scale from 0 to 1, where 1.0 indicates perfect alignment, 0.8-0.9 indicates good alignment, 0.5-0.7 indi-
3.2.3. TASK DESIGN.
Judgment Task requires models to classify the input video as either synthetic or real by answering the question, "Is the provided video generated by AI?" To minimize the influence of visual elements and style, ensuring models focus on se- mantic content, we use synthetic videos without watermarks
6

---

Impossible Vidcos
Table 2: Evaluation Results of IPV-TXT Across Dimensions. This table compares the performance of state-of-the-art video generation models using the IXV-TXT benchmark as text prompts in the T2V setting. A higher score indicates better performance in given dimension. Bold denotes best, underline denotes second.
Physical
Biological
Social
Geugraphical
Overall
Model
Visual Quality
Prompe Following
Visual Quality
Prompt Following
Visual Quality
Prompt Following
Visual Quality
Prompt Following
Visual Quality
Procupt Following
IPV
Score
Open-source Models
LTX (HaCohen et al., 2024)
58.3
14.1
35.3
21.6
44.0
16.0
57.1
23.8
52.3
16.5
10.0
Open-Sora (Zheng
2024a)
63.8
22.1
25.5
29.4
20.0
32.0
57.1
26.5
15.8
Pyramid-Flow (Jin ot al., 2024)
88.3
15.3
92.2
21.6
72.0
28.0
100.0
52.4
88.5
20.8
19.6
Cog Cog\idX-1.55 (Yang al. 2024b)
29.4
35.3
52.9
40.0
56.0
39.2
16.9
Mochi (GenmoAl, 2025)
68.7
44.2
64.7
56.6
80.0
60.0
71.4
76.2
69.2
50.8
37.3
HunyuanVid (Kong et ol., 2024)
95.1
23.9
88.2
35.3
88.0
40.0
90.5
92.7
29.2
26.2
Proprictary Models
Luma (Lamsalabs, 2025)
11.7
90.2
19.6
82.6
17.4
85.7
17.1
14.3
Sora (OpenAL, 2025)
98.8
15.3
98.0
43.1
100.0
30.4
95.2
61.9
98.4
26.0
25.2
Kling (KLING, 2025)
21.5
94.1
33.3
95.2
43.5
81.0
96.1
27.5
26.7
Hailuo (Hailua, 2025)
96.3
30.1
94.1
45.1
100.0
52.0
90.5
61.9
95.8
37.7
36.2
cates partial alignment, 0.1-0.4 indicates weak alignment, and 0.0 indicates no alignment. The justification step crit- ical to ensure fair and stable score assignment. By default, we employ GPT-40 the primary evaluator. To avoid self- evaluation bias, as GPT-4o is also evaluated as a VideoL..M, we additionally utilize Claude-3.5 3.5nonett as an evaluator.
tomated evaluation method to assess a model's capability in generating impossible videos. A detailed analysis is pro- vided in the Appendix B.2.
4.1. Results and Analysis
Can today's 's video generation models effectively follow prompts to create impossible video content? As shown in Tab. 2, the cop-performing model, Mochi 1, generates high- quality impossible videos in only 37.3% of cases. Other models perform even worse, suggesting that current video generation models remain far from achieving satisfactory performance in generating high-quality impossible videos.
Quality Assessment, Since the task of MCQA and OpenQA constructed or evaluated with the help of GPT-4o, which risks hallucination, we conduct a human assessment. Specif- ically, for each task, randomly select subset of 100 samples and ask human to answer the questions, imitating the video understanding models. After that, the accuracy or score serve as a golden reference in models' evaluation.
Unbalanced capability between visual quality and im- possible prompt following. For instance, Luma demon- strates remarkable visual quality, surpassing most open- source models, but its prompt-following score is signifi- cantly lower. In contrast, some open-source models, such as Mochi 1, exhibit superior prompt-following capabilities, even outperforming many proprictary models. An ideal model should excel in both dimensions, achieving balance that is quantified by our IPV-Score metric.
4. Evaluate Impossible Video Generation
Setup. We evaluate mainstream video generation models, including open-source (Open-Sora 1.2 (Zheng et al., 2024a), Video Hunyuan Video (Kong et al., 2024), CogVidX (Yang et al.,
2024b), Mochi (GenmoAI, 2025), LTX (HaCohen et al., 2024), Pyramid-Flow (Jin et al., 2024)) and closed-source models (Sora (OpenAI, 2025), Kling 1.5 (KLING, 2025), Luma (LumaLabs, 2025), and Hailuo (Hailuo, 2025)).
What limits the the model on impossible video gener- ation? Beyond basic prompt comprehension and follow- ing, we identify two unique challenges posed by impossi- ble text prompts, as illustrated in Fig. 5. First, impossible prompts may trigger low visual quality. While the model attempts to follow the prompt, the unusual nature of impos- sible prompts often results in visual artifacts or generation failures. This is likely because impossible prompts rep- resent out-of-distribution data for the model. Second, an overemphasis on adhering to physical laws may constrain the model's 's creative freedom. In many failure cases, videos accurately capture the semantic elements of the prompt but fail to depict the critical impossible phenomenon. Instead, they depict normal scenes that conform to real-world rules. Future video generation models may consider the above factors to achieve a better model. Additional examples are included in the Appendix E.
IPV-Score Metric. To evaluate model, we first generate a set of videos using the IPV-TXT prompt suite. Human annotators are then employed to label two aspects of each video: 1) Visual quality, assessing whether the video meets high-quality standards, and 2) Imposxible prompt following, determining whether the video accurately depicts the impos- sible event described in the text prompt. Annotators provide binary labels for each dimension. Using these labels, we in- troduce the IPV-Score, a novel metric to evaluate model's 'ss ability to generate high-quality videos that faithfully depict impossible events. The IPV-Score is calculated as the per- centage of videos that are both high-quality and faithful to the prompt within the entire generated set.
Human- and Auto-eval. Tab. 2 presents the performance of the evaluated models. Additionally, we propose an au-
7

---

Impossible Vidcos
Video Gen. Fallure Type 1: Low Visual Quality
Prompt: solitary in in stari, empty room suddenly and ineupitcblly duplicates itsell, defying the laws of phrpics.....
Mochis
Longli: The video shows cat eutlin from a boul of food that is placed on the floor. This is normal and expected behavior a cal, so there no content or event that makes the video imposs bie or ununual in real world.
Oowen--VVL witeo
as because cats typically large
food time do
video Additionally amounts shown enting from bowi
lorger than the cat's head, which unusuat.
NVILA: video depicts cat catingg large amount of food short period of time, which
Unussual courter mulliv
wor sconarios.
Video Gen. Failure Type 2: Failed Prompt Following
LaaVi--Nett The video impoos bie in restsvort because the cot is not eating the food, but indward pouring the food
baod.
Prompt: Someone pour water into half-fited glass rup, hut remarkabblyy the woster level
remains unchanged despite continuuus
Inter--V-2.3 The video depitss a cat defying the los physics by littnng barge quantity
khole with its mouth and thenr pouring it bowl. This impossible the real worrld
cats do not have the physical ability to lift and pour such large amount of kibble.
Hampaan
GPT-4o: video shmoe cat seemiigly pourtng an endiess stream of kibble from mouth onto plate, creating an unuxallly lere pll,, This countetntuttioe and imposoitle the real world Becouse:
1. Endless ess Supply: A cat cannot store or produce such a large amount of kiiblle mouth,
2. Gravity and Prysics The kibble appears to flow contt nuously without any vsible source,
detying normal cepertations of how food & dispensed.
3. Cat Behaeaar: Cats do not typically dispense food in this manner.
King
Human Annotation: A cat unnaturally generstes cat food from its head, which is physicelly imposuible in reality.
Figure 5: Failure case of impossible video generation.
Figure 7: Example of the OpenQA task. We ask state-of- the-art video understanding models to analyze whether the video is impossible or not. We highlight the correct analysis in red and the incorrect analysis in green.
What is counterintuiivee in the video?
achieves only 26.8% accuracy, which is close to the random baseline. Most open-source models exhibit substantial room for improvement. Fig. 6 presents an example of the MCQA task, where most models fail to select the correct option.
A. The metal ball creates riples that move in reverse direction.
B. The metal bal changes from so to iquid state.
C. The wate level detreases as the bell floats on
The video plausible without any countertacual content
The motal ball continuously expands in volume.
Video A
NVILA:
LLaVA-Next
Drycc D
LongVU: C
Geriini-..-fleshi
Open-ended QA Task. Among the three tasks, the OpenQA task is the most conceptually straightforward yet practically challenging (Fig. 7). The last two column of Tab. 3 indicate that most current models struggle to independently identify counterfactual phenomena in videos. Among open-source models, LLaVA- achieves superior scores with both GPT-4o and Claude-3.5 evaluators, aligning with its strong performance on the MCQA task. GPT-4o attains the highest performance with both evaluators, underscoring its robust visual understanding and reasoning capabilities. The overall scores suggest that most models struggle to comprehend impossible videos, highlighting a promising direction for future research.
Intem-VL-25:: A
Owen2-VL: A
GPT-4u:
Figure 6: Example of the MCQA task. We highlight the correct option in red and the incorrect option in green.
5. Evaluate Impossible Video Understanding
We evaluate a diverse set of popular Video-LLMs, including both open-souree and proprietary models. The three tasks are organized hierarchically, with increasing difficulty, to comprehensively assess model capabilities. Table reports the evaluation results for impossible video understanding.
5.1. Results
Judgment Task. Most models achieve comparable Ac- curacy and F1-scores on this task. Qwen2-VL achieves the highest accuracy at 76.2%, even outperforming Gem- ini by 3.1 percentage points. Empirically, we observe that some models exhibit bias, predominantly answering 'Yes" (or 'No') for the majority of videos. To account for this bias, we also report Fi-scores, with Video-LLaVA achiev- ing the highest performance. Detailed results and additional analysis are provided in the Appendix B. GPT-4o reject answering "Yes" "No" for this task. To ensure fair com- parison, we do not further tune the text prompt for it.
5.2. Analysis
Are today's video understanding models good enough for understanding impossible videos? The results from the MCQA and OpenQA tasks collectively provide insights into this question. Overall, proprietary models show promis- ing potential, outperforming most open-source models on this task. However, their ability to independently identify impossible videos remains suboptimal, as evidenced by the OpenQA scores. Most open-source models perform poorly on this task, indicating significant room for improvement.
Multi Choice Task. Model performance on this task ex- hibits significant variation. The top-performing model, LLaVA-Next, achieves an accuracy of 86.4%, surpass- ing both GPT-4o and Gemini. In contrast, Video-LLaVA
Unbalanced model capabilities across domains and tasks. Tab. reveals that many models exhibit unbalanced capabil- ities across domains and tasks. Across domains, "Physical"
8

---

Impossible Vidcos
Table 3: Evaluation Results for Impossible Video Understanding. This table compares the performance of sota LMss using the -PV-Vid benchmark. A higher score indicates better performance in a given dimension. Bold denotes best, underline denotes second, Open(C) denotes using Claude as the evaluator.
Judgement
Physical
Biological
Social
Geographical
Overall
Model
MC
Open
MC
Open
MC
Open
MC
Open
MC
Open
Open(C)
Open-source Models
Random
50.0
50.0
20.0
20.0
20.0
20.0
20.0
Human
94,0
82.7
Video-LLaVA (Lin al., 2023a)
72.7
72.9
23.0
13.3
34.6
29.7
31.8
16.9
24.1
25.9
26,8
14.2
18.7
Oryx (Liu et ul,, 2024e)
58.6
44.6
52.0
16.7
68.8
33,9
70.3
32.6
83,9
35.2
60.4
18.7
22.7
Intem-VL-225 (Chen et al., 2024b)
56.5
69.7
61.9
34,9
64.2
59.6
65.5
49.6
77.0
48.4
62.4
33.0
32,5
NVILAA (1.in et al., 20241)
72.6
64.0
60.2
26.4
63.3
52.3
68.9
36.6
69.0
42.9
62.6
26.8
30.6
LongVU (Shen et el., 2024)
70.3
68.0
69.3
21.8
79.6
38.3
77.0
35.6
77.0
35.6
73.3
21.9
25.4
Qwen2-V VL (Wang et al., 2024)
76.2
21.1
32.8
75.8
56.3
75.0
48.3
75.9
53.0
71.4
33.7
V.xt (L al., 2024a)
73.2
70.4
82.8
37.7
92.9
57.2
90.5
51.3
90.8
51.4
86.4
34.4
38.6
Proprictary Models
Gemini- Flash (Team et al., 2024)
73.1
64.0
80.5
48.5
90.8
66.4
89.2
59.3
93,1
65.3
84.4
42.5
36.8
GPT-4o (Hurst al., 2024)
76.7
58.2
83.8
75.5
84.5
64.9
92.0
71.1
79.7
49.1
45.1
Table 4: Video understanding evaluation results on two cat- egories of videos: 1) videos that can be understood through spatial scene understanding and world knowledge, and 2) videos that require temporal reasoning for comprehension.
the understand- ing that Singapore is a tropical country. Due to space limit, additional case studies are provided in the Appendix F.
knowledge for adentification-speciiie
Challenge and opportunity on temporal reasoning. Of the two key factors, world knowledge is primarily governed by the L.M,, whereas temporal reasoning offers greater design flexibility but remains more challenging. Tab. 4 pro- vides a separate evaluation of spatial- and temporal- -occused videos. For all models, scores on temporal-focused videos are consistently lower than those on spatial-focused videos. This clearly demonstrates that temporal dynamic reason- ing poses significant challenges for most current models. Video expert models with high frame rates (e.g.. Long VU) do not exhibit significant advantage. Interestingly, the top-performing models (e.g., I.LVVA-Next and GPT-4o) are all image- It is worth noting that GPT-4o is evalu- ated using only 1 FPS. This observation suggests that more sophisticated temporal modules, rather than simply expand- ing the context window, may be key to understanding and reasoning about impossible videos.
Spatial
Temporal
Model
MC
Open
MC
Open
Open-source Models
Random
20.0
20.0
Vidco-LLaVA (Lin et al., 2023a)
34.6
30.3
23.8
13.7
Oryx (i et al., 2024e)
83.2
41.0
51.4
17.5
Intern VL-2.5 (Chen et al., 2024b)
72.4
56.6
60.6
37.2
NVILA (1.u et 2024f)
75.2
51.6
57.6
28.0
LUngVU (Shen al. 20224
85.7
42.6
68.5
22.7
Qwen2-VL VL (Wang al., 2024)
79.7
57.6
68.2
34.4
LLaVA-Nest (Liu et al.,, 2024a)
95.5
55.8
82.7
39.8
Proprietory Models
Gemini-1 J.5 Plash (Teum et al., 2024)
93.0
67.5
81.1
50.0
GPT-4o (Hurst et 2024)
90.6
75,2
75.6
59.1
emerges as the most challenging, with most models achiev- ing the lowest scores in this category. In contrast, the re- maining domains exhibit relatively balanced performance across models. We hypothesize that "Physical" contains more challenging samples that necessitates temporal dy- namic reasoning. Across tasks, performance on the MCQA and OpenQA tasks is closely correlated, whereas the Judg- ment task appears to be distinct. For instance, Video-LLaVA excels in the Judgment task but underperforms significantly in the other tasks. Discrepancies also exist between MCQA and OpenQA performance. LLaVA-Next shows strong per- formance on MCQA but underperforms on OpenQA. In con- trast, GPT-40 achieves the highest performance on OpenQA but lags slightly behind on MCQA.
6. Conclusion
In this work, we introduce the concept of Impossible Videos as a novel testbed to challenge and advance video under- standing and generation models. Unlike real-world videos, impossible videos depict counterfactual or physically im- plausible scenarios, requiring models to move beyond mem- orization and retrieval toward deeper reasoning and gener- alization. To support research in this direction, we propose IPV-BENCH, a comprehensive benchmark consisting of a structured taxonomy, a diverse prompt suite (IPV-TxT), and a high-quality video dataset (ID).VID). Extensive eval- uations show that current video models struggle with impos- sible videos, revealing critical limitations in their reasoning abilities under non-realistic conditions. These findings shed light on existing gaps and point to promising directions for future research.
What makes good model for impossible video under- standing? We identify two critical factors: temporal dy- namic reasoning and world knowledge reasoning. For in- stance, in the first example from Fig. 1, the cookie appears plausible in individual frames. However, only by analyzing and reasoning across frames can one detect the self- growing phenomenon an impossible event. Conversely, the last example in Fig. 1 ("snowing in Singapore") requires world

---

Impossible Vidcos
Impact Statement
Ai-generated video detection on million-scale genvideo benchmark. arXiv preprint arXiv:2405. 19707, 2024a.
This work introduces Impossible Videos, a benchmark for evaluating video models on counterfactual and anti-reality scenarios. By focusing on violations of physical, biological, geographical, and social laws, it pushes video generation and understanding models beyond imitation toward deeper reasoning and generalization. Our findings reveal current limitations in temporal dynamics and world knowledge rea- soning, and offer a new testbed for safer, more robust video AI. However, benchmarks involving implausible or surreal content may be misused to deliberately generate deceptive or misleading media. To mitigate this risk, our benchmark is designed for research purposes, with focus transparency, educational value, and responsible model development.
Chen, Z., Wang, W., Cao. Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z.,, et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint (..Xv:2242.05555 2024b.
Fan, K., Bai, Z.,, Xiao, T., Zietlow, D., Horn, M., Zhao, Z.., Simon- Mamoo-Gabriel, C.-J., Shou, M. Z., Locatello, F., Schiele, B., et al. Unsupervised open-vocabulary object localiza- tion in videos. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 13747-13755 2023.
Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.210755 2024.
Acknowledgment
This research is supported by the National Research Founda- tion, Singapore under its AI Singapore Programme (ATSG Award No: AISG3-RP-2022-030)
GenmoAL Mochi 1: A new sota in open-source video gen- eration models. https: /www//wwoooooo genmo /i/ble /bllg,
References
2025. Accessed: 2025-01-28.
Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025.
HaCohen, Y., Chiprut, N., Brazowski, B., Shalem, D., Moshe, D., Richardson, E., Levin, E., Shiran, G., Zabari, N., Gordon, O., et al. Ltx-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00/00 2024.
Bai, S. and An, S. A survey on automatic image caption generation. Neurocomputing, 311:291-304, 2018.
Hailuo. Hailuo ai: Transform idea visual with ai. https: / //hat meaiiuced lucal video/ 2025. Accessed: 2025-01-28.
Bai, Z., Nakashima, Y., and Garcia, N. Explain me the painting: Multi-topic knowledgeable art description gen- eration. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5422 54322,542224 2021.
Huang, 7., He, Y., Yu, Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVV Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Bai, Z., He, T., Mei, H., Wang, P., Gao, Z., Chen, J.. Zhang, 7., and Shou, M. Z. One token to seg them all: Language instructed reasoning segmentation in videas. Advances in Neural Information Processing Systems, 37:6833-6859, 2025.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv: 20..22270. 2024.
Bansal, H., Lin, Z., Xic, T., Zong, Z., Yarom, M., Bitton, Y., Jiang, C., Sun, Y., Chang, K.-W., and Grover, A. Videophy: Evaluating physical commonsense for video generation. arXiv preprint 14Xi::240060352200 2024.
Jin, Y., Sun, Z., Li, N., Xu, K., Jiang, H., Zhuang, N., Huang, Q., Song, Y., Mu, Y., and Lin, Z. Pyramidal flow matching for efficient video generative modeling. arXiv preprint arXiv:2410.0595: 2024.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
KLING. Kling ai: Next-generation ai creative studio. https:/ / illingii..... com/, 2025. Accessed: 2025- 01-28.
Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong. J., Li, X., Wu, B., Zhang, J., et al. Hunyuan- video: A systematic framework for large video generative models. arXiv preprint 24xx::2412.03000 2024.
Chen, H., Hong. Y., Huang, Z., Xu, Z., Gu, Z., Li, Y., Lan, J., Zhu, H., Zhang, J., Wang, W., et al. Demamba:
10

---

Impossible Vidcos
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Meng, F., Liao, J., Tan, X., Shao, W., Lu, Q. Zhang, K., Cheng, Y., Li, D., Qiao, Y., and Luo, P. To- wards world simulator: Crafting physical commonsense- based benchmark for video generation. arXiv preprint arXiv: 241.0533, 2024a.
Li, B., Wang, R., Wang, G., Ge, Y.. Gc, Y., and Shan, Y. beed-bencte Benchmarking multimodal llms with gener- ative comprehension. arXiv preprint urXiv22307.16125 2023.
Meng, E. Shao, W., Luo, I.,, Wang, Y., Chen, Y., Lu, Q., Yang, Y., Yang, T., Zhang, K., Qiao, Y., et al. Phybench: A physical commonsense benchmark for evaluating text- to-image models. arXiv preprint arXi:2240..11800 2024b.
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al. Mvbench: A compre- hensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22195-22206, 2024.
Nan, K., Xie, R., Zhou, P., Fan, T., Yang. Z., Chen, Z., Li, X., Yang, J., and Tai, Y. Openvid-Img mm: A large-scale nigh-quality dataset for text-to-video generation. arXiv preprint 24Xi:2407.0237/// 2024.
Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-Ilava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:2311.1012 2023a.
Ni, Z., Yan, Q., Huang, M., Yuan, T., Tang, Y., Hu, H., Chen, X., and Wang, Y. Genvidbench: A challenging bench- mark for detecting ai-generated video. arXiv preprint arXiv:2501. 1340, 2025.
Lin, K. Q., Zhang, P., Chen, J., Pramanick, S., Gao, D., Wang, A. J., Yan, R., and Shou, M. Z. Univtg: To- wards unified video- ladeu-language temporal grounding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2794-2804, 2023b.
Ning, M., Zhu, B., Xie, Y., Lin, B., Cui, J., Yuan, L.,, Chen, D., and Yuan, L. Video-bench: A comprehensive bench- mark and toolkit for evaluating video-based large lan- guage models. arXiv preprint arXiv:2311.16100 2023.
OpenAl. Sora. https: http///paaacc///o.aaaeeeeeeeeeee com/sora/ 2025. Accessed: 2025-01-28.
Liu, H., Li, C., Li, Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Lexta-next: Improved reasoning, ocr, and world knowledge, 2024a.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Liu, H., Yan, W., Zaharia, M., and Abbeel, P. World model on million-length video and language with ringattention. arXiv e-prints, pp. arXiv-2402, 2024c.
Shen, X., Xiong, Y., Zhao, C., Wu, L., Chen, J., Zhu, C., Liu, Z., Xiao, F. Varadarajan, B., Bordes, F., et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410. /7434, 2024.
Liu, Y., Li. S., Liu, Y., Wang, Y., Ren. S., Li, L., Chen, S., Sun, X., and Hou, L. Tempcompass: Do video lims really understand videos? arXiv preprint arXiv:2403.00476 2024d.
Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang, T., Zhang, D., An, J., Lin, J., Zhu, R., et al. Video understanding with large language models: A survey. arXiv preprint arXiv: 2312. 17432, 2023.
Liu, Z. Dong, Y., Liu, Z., Hu, W., Lu, J., and Rao, Y. Oryx milm: On-demand spatial- understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961. 2024e.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05533 2024.
Liu, Z., Zhu, L., Shi, B., Zhang, Z., Lou, Y., Yang, S., Xi, H., Cao, S., Gu, Y., Li, D., et al. Nvila: Efficient frontier vi- sual language models. arXiv preprint arXiv.2412.04466, 2024f.
Wang, L., Bai, Z., Zhang, Y., and Lu, H. Show, recall, and tell: Image captioning with recall mechanism. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 22176-12188 2020.
LubaLabs. Luma dream machine. htt/mm/mmaaaaaaa//// a /1//ream--ee mach i ne, 2025. Accessed: 2025-01-28.

---

Impossible Vidcos
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint (tw.ii::249912299900 2024.
Zheng, Z., Peng, X., Yang, T., Shen, C., Li, S., Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratiz- ing efficient video production for all. arXiv preprint arXiv:2412.20404 2024a.
Wu, C., Chen, X., Wu, Z.,, Ma, Y., Liu, X., Pan, Z.., Liu, W., Xie, Z., Yu, X., Ruan, C., et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024.
Zheng, Z., Yan, X., Chen, Z., Wang, J., Lim, Q. B., Tenen- baum, J. B., and Gan, C. Contphy: Continuum physi- cal concept learning and reasoning from videos. arXiv preprint arXiv:2402.06/ 19, 2024b.
Wu, J. Z., Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7623- 7633, 2023.
Zhu, Y., Li, X., Liu, C., Zolfaghari, M., Xiong, Y., Wu, C., Zhang, Z., Tighe, J., Manmatha, R., and Li, M. A comprehensive study of deep video action recognition. arXiv preprint arXiv:20/..06562 2020.
Xie, J., Mao, W., Bai, Z., Zhang, J.J., Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
Yang, S., Walker, J., Parker-Holder, J., Du, Y., Bruce, J., Barreto, A., Abbeel, P., and Schuurmans, D. Video as the new language for real-world decision making. arXiv preprint ar arXiv:2402. 17139, 2024a.
Yang. Z., Teng. J.. Zheng, W., Ding, M., Huang, S., Xu, J., Yang. Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint 4rXiv:2408.06072, 2024b.
Ye. J., Zhou, B., Huang, Z., Zhang, J., Bai, T., Kang, H., He, J., Lin, H., Wang, Z., Wu, T., al. Loki: A compre- hensive synthetic data detection benchmark using large multimodal models. arXiv preprint 10.09:2410.097322, 2024.
Zeng, A., Yang, Y., Chen, W. and Liu, W. The dawn of video generation: Preliminary explorations with sora-like models. arXiv preprint arXiv:2410.052277, 2024.
Zhang, J. Wu, J. Z., Liu, J.,W., Zhao, R., Ran, L., Gu, Y.,
Gao, D., and Shou, M. Z. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. In- ternational Journal of Computer Vision, pp. 1-15, 2024.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y.,
Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023a.
Zhao, Z., Wang, J., Horn, M., Ding, Y., He, T., Bai, Z.,, Zietlow, D., Simon-Gabriel C.-J., Shuai, B., Tu, Z., et al. Object-centric multiple object tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16601-1661 1, 2023b.
12

---

Impossible Videos
A. Additional Details of Benchmark Construction
A.1. Video Annotation Tool
Fig. illustrates a screenshot of our data annotation tool. It has been divided into two zones: Display Zone showing the essential information for annotation, including original text prompt, the laxonomy label of the prompt, and the video. Annotation Zone provides several fields for annotations, including data for both IPV-VID curation and labels for video generation models evaluation. This all-in-one evaluation tool greatly eases the annotation efforts, supporting multiple ways for downstream usage. Specifically, the questions for each filed is carefully designed to fit human customs, For example, although we aim to annotate "if this video is impossible", we find it is more intuitive to answer "if this video is reasonable" in practice.
Display
Wenttll
piote ruritte sers, somsoee pours wuter into al filed glos oup nanuhhul wale sanutoo anc enchonad duppte contitmnat poaring The ohenomee normal Plydes sn actitonal
dluappann, pertoaallly
appeaatt
Paykal Lows Cosuervatan Lne Mess
Annotation Zone
Mouslg wit.listssiie the be dodie High drosett
bypeioer (9rinag))
plvn
Proaical
folovv
Fig. mici poound foar perto however the lend of them Bnin in cup doen increnee
Yebeze Type una- 2i0m20
Dampors Flusia la ziso mase mal worl?
pessilic toiii
(rimmey)
Hols
Mi naed wotth the whole do make jaggement.
Intiaaas
Submit
End
Figure Screenshot of the annotation tool.
B. Additional Results and Analysis
B.1. Al-Generated Video Judgment
Tab. presents more detailed results on the task of Al-Generated video judgment, helping understanding unique properties of each model. Video-LLaVA is representative balanced model, with similar accuracy both fake and real videos, and Yes Rate around 50%. In contrast, Intern-Vi VL, NVILA and Gemini significantly biased. Intern-Ve VL is biased to answer "Yes" with high Yes Rate 93.5%. NVILA, and Gemini are biased toward the other direction, preferring answer "No" with lower Yes Rate. The trends can also be reflected in the huge accuracy difference between fake and real videos.
B.2. Evaluating Impossible Video Generation An Automatic Strategy
In this section, we introduce an automatic evaluation strategy as surrogate to human score presented in the main text. We first generate collection of videos based on the IPV-TxI prompt suite for each model. Then, we compute the the following metrics to evaluate the visual quality and impossible prompt following capability, respectively.
!) Visual Quality. We use the popular video quality assessment suite, VBench (Huang al., 2024), to build a compound
13

---

Impossible Vidcos
Table 5: Detailed Evaluation Results for Al-Generated Video Judgment Task.
Model
Overall Acc.
Fake Acc.
Real Acc.
Fl-Score
Yes Rate
Video-LLaVA (Ning al., 2023)
72.7
72.9
72.5
72.9
50.3
Oryx (Liu et al., 2024e)
58.6
33.2
84.3
44.6
24.5
Intem-VL-2.5 (Chen et al., 2024b)
56.5
99.7
12.8
69.7
93.5
NVILA (Ni et al., 2025)
72.6
48.4
97.1
64.0
25.8
LongVU (Shen al., 2024)
70.3
62.7
78.0
68.0
42.5
Qwen2-VL (Wang et al., 2024)
76.2
58.3
71.1
32.1
LLaVA-Next (Liu et al., 2024a)
73.2
63.3
83.2
70.4
40.2
Gemini- I.S-Flash (Team al., 2024)
73.1
47.6
64.0
24.6
metric that measures overall video quality without considering the prompt. Specifically, we combine the six factors-Subject Consistency, Background Consistency, Motion Smoothness, Aesthetic Quality, Imaging Quality, and Dynamic Degree-from from VBench to form our final metric. Similar to VBench, calculate the weighted average of these six factors the final evaluation score. We tailor the weights of these factors better suit our video domain, such as reducing the weight of Aesthetic Quality since our impossible videos mostly follow realistic style. The weights we use for each factor are: 2.0, 2.0, 0.2, 0.2, 2.0, 1.0.
2) Impossible Prompt Following. To assess whether the impossible event described in the text prompt is faithfully represented the generated video. We utilize GPT-4o to provide a binary judgment for each video and calculate the following ratio as the final score. To achieve accurate judgment, we propose three-step strategy break down the task. Specifically, prompt GPT-4o with the text prompt and frames sampled FPS. We instruct GPT-4o to: 1) identify and summarize the impossible event in the text prompt, 2) ground the impossible event in the video, and 3) confirm the visibility of all key elements that constitute a violation of common sense, reason, and conclude with a "Yes" or "No." We also provide two additional chain-of-thoughe examples as demo cases. Please see Fig. 12 for qualitative examples.
We compare the baseline prompting strategy with our prompting strategy on videos generated by Kling, where our approach better aligns with human annotations. See Tab.
Table 6: Comparison between prompt strategy and baseline prompt strategy for impossible prompt following evaluation.
Human Alignment
GPT-40
0.72
GPT-4o + our prompt strategy
0.80
A
11.1%
3) IPV-Score. We calculate the IPV-score the product of the visual quality score and the impossible prompt following score, designed to assess the model's ability to generate high-quality, impossible videos. Since both scores evaluated independently, their product effectively models the joint distribution of these factors. Before performing the multiplication, we further scale the ranges of both to better align them.
B.2.1. HUMAN ALIGNMENT
calculate the Spearman's 's correlation coefficient (p) between our automatic evaluation score and human annotation score. Fig. 9 shows the corresponding results. We can see that the Spearman's 's correlation coefficient between the visual quality score, prompt following score, and IPV-score with human annotations remains above 0.8, demonstrating consistency with human annotators.
C. Instructional Prompt of MCQA Task
In Fig. 11, we present the complete instructional prompt used in the MCQA task. has particularly designed rules for constructing distractors that challenge the reasoning capability of video understanding models.
14

---

Impossible Vidcos
Cofficient. 0.9268
Lincr Cofficint:
Human Qualtty Soore
Humar PV
0.59
Quality Score
Following Score
Auto Score
Figure 9: Spearman's 's correlation coefficient between automatic evaluation score and human annotation score on 10 different video generation models.
D. Qualitative Examples
Fig. 13 illustrates more video examples from the IPV-BENCIL
E. Case Study of Impossible Video Generation
In this section, we present three common failure examples of impossible video generation in Fig. 14.
Rows 1-2 show cases with high visual quality but that do not faithfully follow the video prompt; the generated videos adhere to the physical laws of the real world.
Rows 3-4 display cases with high visual quality, but they do not faithfully follow the video prompt to generate the designated impossible event, instead introducing other common-senss violations. For example, in the third row, the airplane engine asymmetrical, and in the fourth row, one corner of the cookie moves independently.
Rows 5-8 show videos that follow the video prompt but suffer from poor visual quality, with issues like blurriness animated style.
F. Case Study of Impossible Video Understanding
In this section, we present a series of case studies of impossible video understanding in Fig. 15, Fig. 16, Fig. 17, Fig. 18, Fig. 19, Fig. 20. We observe that either strong open-source model, L.LaVA-Next (Liuu er al., 2024a) or close-source model, GPT-4o (Hurst al., 2024) suffer from difficulty on understanding impossible phenomenon, particularly on temporal dynamics.
15

---

Impossible Vidcos
Scenario Brainstorming Questionnaire
Please come up with 10 dynamic video scenarios that meet the following requirements:
1. The scenario should be impossible (or extremely difficult) to achieve in the real world.
2. The scenario should be dynamic video scene.
3. The scenario may seem reasonable in a static state but becomes unreasonable when in motion. For example, "a cat growing wings" is already highly unrealistic in a static scene, so such cases should be avoided.
4. Tip: Prioritize using common, everyday objects to create creative and unexpected scenarios. Example: A face in a cookie starts talking.
Consider scenarios that violate social norms or expectations. Example: It snows in Singapore.
5. Tip:
6. Tip: You may also consider scenarios that violate physical laws. Example: Water flows backward.
7. Do not use ChatGPT or other large language models use your own creativity!
8. Avoid overly abstract scenarios the scene should have a reasonable degree of visual feasibility.
Figure 10: Questionnaire used for collecting impossible text prompts for IPV-TXT.
16

---

Impossible Vidcos
You tasked with assisting in the creation of multiple-choice QA data for a video benchmark evallating Video- Video-Language Models. The goal is to generate correct answer and a set of plausible distractors for each video based on the provided single description that emphasizes the counterfactual aspect.
Task Overview
each video:
description is provided, focusing on the counterfactual or impossible content.
An additional tag specifying the "impossible type" label (e.g.. physical laws mechanics kinematics) will also be included. there any inconsistency between description and the type tags, always prioritize the text description. The type tags only for reference.
A single frame from the video will also be provided aid in curating realistic distractors. The frame only used construct distractors and infer the correct
You must generste:
One correct answer: precise, accurate explanation of the counterfactual aspect. Three distractors: Plausible but incorrect options, designed to test nuanced reasoning.
Guidelines
Correct Answer:
Must align perfectly with the specific counterfactual event highlighted the video, based solely on the description.
Should be concise and match the length, style, and level of detail of the distractors. Reduce unnecessary details or specificity ensure the correct answer not conspicuously longer or more descriptive than distractors.
Distractors:
distractors have similar description length and, more importantly, detail degree to correct answer!
Diverse Objects and Attributes (Prioritized):
on various aspects of scene, including different objects attributes, such the motion of objects, colors, sizes, or environmental elements (e.g.. "the ball's motion" "the grass's color").
Leverage the provided video frame identify objects not directly involved in the counterfactual phenomenon and create distractors around these objects their attributes.
Impossible Phenomenon Option (Prioritized):
Describes alternative impossible phenomenon not present in the video. Ensure that the object is detectable in the video frame, but the described phenomenon does not occur. This type of distractor ensures that models rely on both video and text reasoning.
Prioritize temporal dynamics, state changes, and motions, etc.
General Rules for All Options (including both Correct Answer and Distractors):
Ensure all options, including both correct answer and distractors. are similar in length, style, detail degree, and complexity.
Use the provided video frame effectively to curate distractors based on realistic details visible in the scene. Ensure distractors challenge reasoning rather than simple visual-text matching. This can be achieved by describing motions or state changes.
Input Format
single description the video, focusing on counterintuitive aspect(s). tag specifying the lype impossibility (e.g.. physical mechanics kinematics). The tag is for your reference. A single frame from the video.
Output Format
Produce dictionary where each key represents counterfactual event (e.g., 'event_1. 1", 'event_ 2'), and the value is another dictionary with:
correct answer"
distractors'
NOTE: You directly output the result without explanation or other words.
Input: (actual input)
Figure 11: Instructional Prompt of MCQA Task.
17

---

fraves] [New prompt] Impossible Vidcos
Systen pronot: You are a helpful video analyzer that can determine 1f counter intuttive or commonsense violation event in text proept is fattifflly followed in the provided video fraves. Your analyze video fromes and if event described text proept occurs, providing detailed explonation for your answer.
User:
Proept: nelted candle unexpectedly defies nature by regenerating itself upword.
Here video generated by Al fros the provided text proupt. Please check whether the counten-intuttive consonsense violation event the generated video and explatn the reason. You should first identify and brief the counter- intuitive or commonsense violation event from the pronpt. Then, curefully check if this event occurs video. Finally, sursorize reason [No]. ""Do not analyze other factors*s such as video quality, etc. eonly facus on the counter-intvitive commonsense viotation event**, If the key elements constituténg violotion of common sense connot directly observed confirned in video, violation of comson sense is deexed not to have occurred.
text proxpt is faithfully fellowed by
conclude your onsaer with: [Yes] or
Assistant:
The proept describes scenorio where a relted condle regenerates itself upword, defying grovity and appearing reconstruct
time is flowing backward.
In the franes provided, the condle appears to grow taller, with the wax extending upward, be reconstructing
which oliges with the descrihed counter-fnte contte-intuttive itself, suggesting a reverse process of selting.
Thenefore, video deconstrate the commnssense violatian descrtbed the proupt.
[Yos]
User:
Proept: Larp unexpectedly illueinates grows brighter after soneone unplugs power cord, defying narwal electrical, behavior.
Here is video generated by AI from the provided text proept. Please check whether the counter-intuitiie commonsense violation event in the text proupt is foithfully followed by the generated video and explatn the intuitive or cowsnnsense vtolation event from the proept. Then, carefutly check if this event occurs in the video. Finally, sumsorize the reason and conclude your onswer with: [Yes] or [No]. "Tao counter-tnts intuštíve consonsense vialation event"*, If key elesents constituring viotation of comso sense cannot be directly observed and confireed in the video, the violation of
You should first identify and brief
onalyze other factors** such video quality,
*onny focus the
is deesed to have occurred.
Assistant: The proupt describes on event where a desk Lomp grows brighter ofter being unplugged, defying nomall electrical behavtor.
In the fromes provided, the Lomp indeed glowing und appears to get brighter, but there is no visible indication of the Laxp being unplugged without power source. This crucial that the losp resains illuxinated without heing connected Ced to power not evident the imuges.
aspect of
intuitive
sithhut visuat conftreation of the disconnection and the continued operation the Lamp, video does not fuithfully follow the comonsense-deffinn scenarto described in the text proupt.
answer (s: [No]
User:
Here a video generated by AI from the provided text proust. Please check whether the counter- countnn-nntutrive or consonsense violation event in the text prospt is foithfully followed by generuted video and explain intustive or consonsense violation event frae the proept. Then, corefully check if thts event occurs in the video. Finolly, samorize the reason and conclude [No]. not enalyze other factors** such as counter- counter-tntutrive or cousonsense vialatton event*s, If the key elemens constituting a viotation of conson sense connot be directly observed and confirsed in the video, the violation of conton sense deeved not have occurred.
reason.
ident
brief
with: [Yes] quality, etc. vonly focus the
Figure 12: CoT examples we use to prompt GPT-4o for impossible prompt following evaluation. Videos may contain impossible events that are outside the scope of the prompt. We ignore such events when evaluating impossible prompt following.
18

---

Impossible Vidcos
[Physical Laws -> Material Properties) A paper transforms into smoke when being touched.
[Physical Laws -> Mechanics] car flies up into sky, defying gravity.
[Physical Laws Optics] A short-hair boy's reflection in the mirror shows long-hair girl.
[Physical Laws -> Material Properties] White chalk writes pink words on the blackboard.
[Biological Laws Anthropomorphism] toy dog come to life, playing with the toy bone.
[Blological Laws Capability Violation] A dog writes the table.
[Biological Laws Morphology] crab on the beach grows two wings.
[Social Laws Magical Effects] man takes out food from of computer screen.
(Geographical Laws >> Terrain and Environment] A fountain appears in the desert.
Figure 13: More examples of impossible videos.
19

---

Impossible Vidcos
[Physical Laws Conservation Laws] solitary chair suddenly and inexplicably duplicates itself.
[Physical Laws Optics] A girl sits motionless in front of a mirror, her mirror reflection moves independently.
[Biological Laws >> Anthropomorphism] The rigid metal wings of commercial airliner flap like bird's wings.
[Physical Laws >> Conservation Laws] partially eaten cookie magically regenerates its missing.
portion.
[Physical Laws-> Optics] A yellow changes its color to
[Physical Laws >> Material Properties) A commercial aircraft mysteriously transforms into luxury car.
[Physical Laws-> Material Properties] A ball drops to the floor and sinks halfway into the ground.
(Socill Laws >> Magical Transformations] Massive buildings suddenly begin rotating.
Figure 14: Failures in Generating Impossible Videos.
20

---

Impossible Vidcos
Based on your observation of the video, what content or event that makes the video impossible or unusual real-world?
Question:
The video depicts woman eating piece of bread, which is common and normal activity. However, the unusual aspect of the video is that the woman is eating the bread with her eyes closed, which is not typical way to eat. This could be interpreted as playful or humorous gesture, but it is not common way eat bread.
NVILA:
The video depicts person eating a slice of bread, which is common and everyday activity. There nothing the video that makes the content impossible unusual in the real world. The actions shown consistent with normal human behavior and the environment typical for such an activity.
Intern-VL-2 VL-2.5:
LongVU: The video content or event is impossible unusual in real-world because the person is holding a piece of bread and taking a bite out of it.
The video depicts a woman biting into a piece of bread and then inmediately it out. This action unusual real-world scenarios because biting into piece bread typically involves chewing and swallowing, not spitting it out immediately. The woman's reaction suggests that the bread might have a bitter or unpleasant taste, which is not common occurrence with bread.
Qwen2-VL:
LLaVA-Next: The video impossible in real-world because the person holding slice of bread with no visible filling topping, and there is no context provided for why they are eating it.
The sequence images shows a person taking bite bread, but the unusual aspect that the bite mark appears before the person actually bites into the bread. This suggests that the video might be edited or reversed, making it impossible real-world scenario where the bite mark should only appear after the person bites the bread.
GPT-4o:
Human Annotation: piece toast mysteriously disappears as bitten, despite no one actually taking bite
Figure 15: Case study of impossible video understanding.
21

---

Impossible Vidcos
Based on observation of the video, what content event that makes the video impossible unusual in real-world?
Question:
The video is impossible or unusual in real-world because shows two water bottles with same amount af water, but bottle is frozen while the other is not.
NVILA:
Intern-VL-2.55:
The video impossible the real world because shows two plastic bottles filled with water that are frozen solid, which is not common or physically possible state for water in a bottle. Water freezes at degrees Celsius, and if the bottles were filled with water, they would not remain solid without being in controlled environment where the temperature is kept below freezing for an extended period. The video suggests that the water has frozen, but this is not typical realistic scenario for simple demonstration.
LongVU: The video shows two identical plastic bottles of water placed side by side, but one of them is frozen while the other not. This is unusual event in the real world.
The video shows two plastic bottles filled water shaking violently on a table. This is unusual because water is liquid and should not shake violently unless there external force acting it, such earthquake strong wind. Additionally, the bottles are not attached anything, so there is no reason them to shake violently on their own. This makes the video impossible, unusual, or counterintuitive in real-world scenarios.
Qwen2-VL:
LLaVA-Next: The video is impossible in real-world because the water level in both bottles rises simultaneously, which not possible reality.
The images depict two water bottles with liquid inside that appears be frozen or turning phenomenon is known as "supercooling." where water remains in liquid state below its freezing point and then rapidly crystallizes when disturbed. real-world context, this is unusual not impossible. Supercooled water can instantly freeze when shaken disturbed, which might be what's happening here. This effect be counterintuitive because expect water freeze O'C (32'F), but under certain conditions, it remain liquid below this temperature until disturbance triggers freezing.
GPT-4o:
This
Human Annotation:
The water level in the bottles spontaneously increases without any water source, which violates
the law of conservation of mass.
Figure 16: Case study of impossible video understanding.
22

---

Impossible Vidcos
Based on your observation of the video, what content or event that makes the video impossible or unusual real-world?
Question:
The video depicts a person pouring water container filled with sand. This an unusual event because sand is dry substance and does not absorb water.
NVILA:
Intern-VL-2.5: The video depicts hand pouring liquid into container filled with sand, which is common and expected action. However, the unusual aspect is the presence of holes in the sand, which suggests that the sand might have been pre-drilled or has some other unusual property. This could be counterfactual in real-world scenario where sand is typically poured into a container without pre-existing holes. The video does not provide an explanation for the holes, which adds an element of mystery or could imply specific purpose experiment that is not immediately clear.
LongVU: The video shows person pouring a clear liquid into container sand, which is an unusual event in the real world.
Qwen2-VL:
where person pouring water into container filled with sand. This action unusual in real-world scenarios because sand is not suitable medium for holding water. Sand is composed of small particles that not dense enough to retain water, and the water would simply seep through the gaps between the particles. Additionally, the container appears to be plastic container, which is not typically used for storing or pouring water due to the risk of water damage.
The video depicts
LLaVA-Next:
The video impossible in real-world because the liquid poured into the sand creates swirling motion and forms vortex, which is not possible in real-world physics.
GPT-4o: The images depict a scenario where liquid is being poured onto sand, but the sand remains dry and unaffected
by the liquid. In the real world, pouring liquid onto sand would typically result in the sand becoming wet and clumping together. This unusual behavior suggests that the sand might treated with hydrophobic substance, making it repel water. This is counterintuitive because, under normal circumstances, sand would absorb the liquid.
Human Annotation: The water completely disappears into the sand without leaving any wet marks darkening the surface, which defies how water normally interacts with sand.
Figure 17: Case study of impossible video understanding.
23

---

Impossible Vidcos
Based on your observation of the video, what content or event that makes the video impossible or unusual in real-world?
Question:
The video depicts a tomato being cut in half, but the tomato not cut half. This is an impossible and unusual event in the real world.
NVILA:
Intern-VL-2.5: The video depicts tomato being sliced half, which a common and expected action. However, the unusual aspect is that the tomato is already sliced in half, and the video shows the process of slicing it again. This repetition of the same action might be considered counterintuitive unusual real-world context, does not add new information or progress the scene.
The video shows tomato being cut in half, but the tomato is already sliced in half before the knife is used. This creates a paradoxical situation where the knife used to cut tomato that already cut in half.
LongVU:
Qwen2-VL: The video shows a tomato being sliced in half with knife, but the tomato is already cut in half. This is an impossible event because the tomato cannot be cut half while is still whole.
LLaVA-Next: The video shows tomato being cut in half, but the tomato is not actually being cut. This creates an illusion tomato being cut, which is unusual real-world scenarios.
The video depicts a knife cutting into a tomato, but the tomato appears be unusually large, almost the size small watermelon. reality, tomatoes do not grow to this size, making the scenario impossible counterintuitive in the real world.
GPT-4o:
The tomato splits horizontally despite being cut vertically with a knife, creating an impossible mismatch between the cutting direction and splitting result.
Human Annotation:
Figure 18: Case study of impossible video understanding.
24

---

Impossible Vidcos
Based on your observation of video, what content or event that makes video impossible or unusual real-world?
Question:
The video depicts book that is impossible to open, unusual in that is not real book, and counterintuitive in that is not a real book.
NVILA:
Intern-VL-2.5; The video is impossible in real-world terms because it shows a book opening and closing on its own without any visible human interaction or external force. This defies the laws physics and the natural behavior objects, as books do not have the ability to open and close autonomously.
LongVU: The video shows book being placed on top of another book, which is normal action.
The video depicts a sequence of events that are physically impossible in the real world. Specifically, the video shows book being stacked on top of another book, which then causes first book to flip over and land on the second book. This sequence of events impossible because books are not capable of flipping over in this manner. reality, if a book placed on top of another book, the top book would simply rest on the bottom book without flipping over. The video's depiction of the books flipping over is therefore a violation of the laws physics and is not possible in the real world.
Qwen2-VL:
LLaVA-Next:
The video shows a book being placed on top of another book, which is an impossible event in real-world
physics.
The sequence images shows books appearing to multiply or move without any visible external force or interaction, which is impossible in the real world. In reality, books cannot move duplicate themselves without someone or something physically interacting with them. This scenario defies the laws of physics, making it unusual and counterintuitive.
GPT-4o:
Human Annotation: The book spontaneously splits and replicates itself into three copies, violating the laws of physics and conservation of matter.
Figure 19; Case study of impossible video understanding.
25

---

Impossible Vidcos
Based on your observation of the video, what content event that makes the video impossible unusual in real-world?
Question:
NVILA:
The video depicts a person holding a bowl of food that is on fire, which is impossible in real-world scenarios.
Intern-VL-2.55 The video presents a counterfactual scenario where laptop screen displays a bowl of food that is steaming, and the person interacts with the steaming food it were physical object. In the real world, laptop screen flat, two-dimensional display and cannot physically hold emit steam from bowl food. interaction with the steaming food the screen is unusual and impossible activity in reality.
LongVU: The video shows laptop screen displaying bowl of food that on fire, which unusual and potentially dangerous situation.
The video depicts person interacting with laptop that displays bowl of food fire. This scenario impossible the real world because it is not physically possible for laptop screen to emit flames or smoke. The image the screen is likely digital virtual representation, but the interaction with the laptop and the bowl of food on fire is a surreal and unrealistic concept.
Qwen2-VL: VL:
LLaVA-Next:
The video shows bowl of food on the laptop screen with flames and smoke, which is not possible in real
life.
The video depicts a scenario where food and steam appear to coming out of a laptop screen, which impossible in the real world. Screens cannot produce physical objects substances like food or steam. This is digitally manipulated or staged effect, making unusual and counterintuitive.
GPT-4o:
Human Annotation: A person reaches through a computer screen to grab food from a video, defying physical boundaries between digital and real worlds.
Figure 20: Case study of impossible video understanding.
26


Reviews:
[{"review_id": "lwYWYetHTD", "reviewer": "Reviewer_YfiS", "summary": "The paper introduces IPV-BENCH, a benchmark for evaluating video understanding and generation models using \"impossible videos\". It includes a taxonomy, a prompt suite (IPV-TXT), and a video dataset (IPV-VID). Evaluations reveal limitations in current models, highlighting the need for improved reasoning and generalization in non-real-world scenarios.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We greatly appreciate your insights on our paper. Below, we outline our responses to each point.\n\n**Q1: Reproducibility: Code/Dataset Release**\n\nWe appreciate the reviewer’s concern regarding reproducibility. Upon acceptance, we will publicly release all **code and data**, including the IPV-Bench taxonomy, IPV-TXT prompts, IPV-VID videos, and evaluation protocols. Additionally, we will provide a detailed **Data Usage Instruction** in the Appendix to enhance accessibility. We believe that this comprehensive release will maximize the impact of Impossible Videos and foster further research in this area.\n\n\n**Q2: Significance and Applications of Impossible Videos.**\n\nWe appreciate the reviewer’s valuable suggestion, as it will further enhance the impact of Impossible Videos.\n\n**Significance:**\n- As highlighted in our paper, this benchmark fills a critical gap in evaluating counterfactual video generation and understanding—an area currently absent in the community. This importance has also been **recognized by Reviewers F9mi and AP4m**.\n- Evaluating **AI robustness and generalization** in out-of-distribution scenarios is a well-established challenge, particularly for autonomous systems encountering rare events. As noted by **Reviewers F9mi and JPJB**, \"Impossible Videos\" serves as a **robustness and generalization benchmark** for video understanding and generation models, addressing an overlooked yet crucial aspect of AI evaluation.\n- As **Reviewers F9mi and AP4m** acknowledged, our work is closely related to broader topics in **counterfactual and causal reasoning in AI**. \"Impossible Videos\" provides a valuable case study for counterfactual reasoning in the video domain, encompassing both video understanding and generation.\n\n**Applications:**\n- \"Impossible Videos\" can be leveraged to enhance video understanding and generation models by improving their robustness and generalization capabilities.\n- Real-world applications include:\n\t- **Creative industries** – Enhancing special effects, game design, advertising, and filmmaking, etc.\n\t- **Industrial safety** – Assisting in anomaly detection and risk assessment.\n\t- **Advanced AI assistants** – Equipping AI systems with stronger reasoning capabilities for more intelligent decision-making.\n\n\n**Q3: Experimental Details: IPV-Score and Evaluation**\n\nWe appreciate the reviewer’s feedback on this issue.\nThe computation of the IPV-Score is based on the statistics of **Visual Quality** and **Prompt Following** and is defined as:\n$$\nIPV Score=\\frac{\\text{Num. of High Visual Quality}\\cap \\text{Num. of Good Prompt Following}}{\\text{Num. of All Vid.}}\n$$\nThis metric intuitively aligns with our design philosophy: it measures the percentage of videos that satisfy both high visual quality and strong prompt adherence.\nTo improve clarity, we will include this equation along with a detailed textual explanation in the revised paper.\n\n\n**Q4: How to Encourage Broader Adoption?**\n\nWe appreciate this constructive question and will take the following steps to encourage broader adoption:\n-   **Public Release** – Make all data and evaluation code openly available to enhance accessibility.\n-   **Comprehensive Documentation** – Provide detailed instructions for data usage to facilitate easy adoption.\n-   **Community Engagement** – Organize competitions and workshops on Impossible Videos to attract more researchers to this domain.\n-   **Integration with Existing Toolkits** – Collaborate with established toolkits to incorporate IPV-Bench into widely used evaluation suites.\n-   **Ongoing Benchmark Maintenance** – Maintain a leaderboard and regularly update the benchmark to ensure its relevance.\n\nWe sincerely hope that these efforts will inspire further research and drive innovation in the video understanding and generation community.", "claims_and_evidence": "The claims in the submission are not fully supported by clear and convincing evidence. The paper lacks a detailed release of code and benchmark datasets, which are crucial for reproducibility and validation. Additionally, focusing on \"impossible videos\" is niche and may not attract widespread adoption, limiting the benchmark's impact. These issues weaken the overall credibility and practical utility of the claims.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria, including the IPV-BENCH benchmark, are relevant for assessing video models on impossible videos.", "theoretical_claims": "The paper does not present any theoretical claims or proofs that require verification. It focuses on empirical evaluation and benchmarking of video understanding and generation models.", "experimental_designs_or_analyses": "The paper lacks detailed experimental designs and analyses, particularly in the evaluation of video generation models. The IPV-Score metric is introduced but not thoroughly explained or validated. Additionally, the absence of released code and datasets undermines the reproducibility and soundness of the experiments.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. Introduces a novel concept of \"impossible videos\" to challenge video models.\n2. Constructs a comprehensive taxonomy and benchmark (IPV-BENCH) for evaluation.\n\nWeaknesses:\n1. Lacks release of code and benchmark datasets, limiting reproducibility.\n2. Focuses on non-mainstream scenarios, potentially reducing broad interest and adoption.", "other_comments_or_suggestions": "1. Provide detailed supplementary materials, including code and benchmark datasets, to enhance reproducibility.\n2. Highlight practical applications of impossible videos to justify their significance in the research community.", "questions_for_authors": "How do you plan to encourage broader adoption of IPV-BENCH as a standard benchmark in the video understanding and generation community?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QKQG0dz8WK", "reviewer": "Reviewer_F9mi", "summary": "The paper introduces IPV-BENCH, a novel benchmark designed to evaluate video understanding and generation models from the perspective of impossible videos. It categorizes scenarios that violate physical, biological, geographical, and social laws. The main experimental results reveal that current models have difficulty understanding and generating such videos, pointing potential developments in video models.\n\n## update after rebuttal\n\nI appreciate the author's response and I maintain positive.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and constructive suggestions. We are grateful for your recognition of our work’s novelty and will incorporate your recommendations to further strengthen the paper. Below, we address your comments in detail.\n\n**Q1: Claims about model limitations of understanding impossible videos could be strengthened with category-specific quantitative metrics.**\n\nWe appreciate the reviewer’s suggestion.\n\nTable 2 of the paper presents impossible video understanding metrics (Multi-choice QA and Open-ended QA) **across the four categories of IPV-Taxonomy**: Physical, Biological, Social, and Geographical. Notably, the **Physical** category proves to be the most challenging, yielding the lowest scores across both tasks.\n\nUpon further analysis, we observed that videos in the **Physical** category often exhibit complex **impossible temporal dynamics**, requiring sophisticated temporal reasoning. In contrast, videos in other categories can be largely addressed through **world knowledge reasoning**, which aligns well with the capabilities of large language models (LLMs).\n\nTo further investigate this, **Table 3 reports an experiment** where videos are classified into two groups via human annotation:\n- **Spatial** – Impossible phenomena identifiable from a single frame.\n- **Temporal** – Impossible phenomena requiring cross-frame temporal reasoning.\n\nResults indicate that models perform significantly worse on Temporal videos than on Spatial ones, highlighting temporal reasoning as a major bottleneck in understanding impossible videos.\n\n**Q2: It is better to provide more information about the video filtering criteria.**\n\nWe appreciate the reviewer’s valuable suggestion. The goal of video filtering is to ensure that the selected videos: 1) Maintain high visual quality; 2) Clearly demonstrate impossible phenomena.\n\n**Visual Quality Criteria:**\n- **Accepted:** Clear, sharp videos with high aesthetic value and smooth temporal motion.\n- **Rejected:**\n\t- Videos with jitter, flicker, blur, large-scale artifacts, or indistinct/distorted foreground objects.\n\t- Completely static videos with no visible changes.\n\t- Videos that lack logical coherence and appear visually chaotic.\n\n**Impossible Semantics Criteria:**\n- The video must clearly depict an impossible, counterfactual phenomenon that cannot occur in the real world. The impossibility should be a salient event, rather than minor visual details that are difficult to perceive.\n- The video should be in a photo-realistic style. Non-realistic styles (e.g., cartoon-style videos) are excluded to avoid confusion in video understanding.\n\nWe will include these detailed filtering criteria in the revised paper.\nThank you again for your insightful review. Your feedback is helpful on enhancing the rigor and clarity of our work. We are happy to address any further questions.", "claims_and_evidence": "The claims made in the paper are largely supported by the empirical results. However, some claims regarding the models' limitations in understanding impossible videos could be enhanced by more detailed quantitative metrics. For example, the authors claim most video models fall short on impossible videos, specific metrics across categories would strengthen this statement.", "methods_and_evaluation_criteria": "The methods, including the construction of the IPV-BENCH benchmark and the associated taxonomy, make sense. The evaluation criteria, encompassing the Judgment, Multi Choice, and Open-ended QA tasks, effectively assess the model's capabilities in understanding impossible scenarios. The diverse sources of video data, including synthetic, real, and community-generated content, enhance the robustness of the evaluation.", "theoretical_claims": "The paper does not present formal proofs for any theoretical claims but relies on empirical evaluations.", "experimental_designs_or_analyses": "The experimental designs are generally thorough, with well-defined tasks and clear criteria for measuring model performance. Whereas, providing more information about the filtering criteria for selecting videos would enhance the dataset's integrity.", "relation_to_broader_scientific_literature": "The paper highlights the gaps in existing benchmarks that do not address impossible or counterfactual videos. It may be related to some literatures that study how humans react with the impossible or counterfactual information.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper provides an interesting perspective of evaluating video model's ability.\nThe experimental results and analysis provide insights in how to develop video understanding and generation models.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NA6poexduX", "reviewer": "Reviewer_JPJB", "summary": "This paper introduces the novel concept of \"impossible videos\" as a challenging testbed for advancing video understanding and generation models. It proposes the IPV-Text benchmark composing of many anti-reality videos for evaluating the video LLM models in the understanding task. Results in the paper reveal that although the video LLM models excel at processing real-world scenarios, they struggle with anti-reality content which need deep reasoning rather than simple memorization. Besides, it also propose a IPV-VID benchmark, which has many anti-reality text prompts. These prompts can be used to prompt the T2V models to generate the corresponding videos. Results in the paper find that today's T2V models also struggle to generate the aligned videos, highlighting their reliance on pattern matching rather than true understanding. The benchmarks and findings not only identify crucial shortcomings in existing approaches but also establish promising directions for future research aimed at developing more robust and generalizable video AI systems.\n\n## update after rebuttal\nI appreciate that the authors take time to explain the implementation details and the metrics. These responses have addressed my concerns. I think the benchmarks proposed by the paper is promising on the future research for more robust and generalized video AI systems. I will increase my score and support its acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging review and valuable suggestions. We appreciate your acknowledgment of the paper’s novelty and will carefully address your concerns to improve clarity and rigor. Below are our detailed responses:\n\n**Q1: Better to explain more clearly about how to calculate the score in the Open-ended QA task.**\n\nWe appreciate the reviewer’s suggestion and provide a more detailed explanation below.\n\nFor evaluating the Open-ended QA task, we employ an LLM-based evaluator that compares model responses against the annotated text explanations in the benchmark. However, we empirically observed that directly instructing the LLM to assign scores led to instability. To address this, we propose a **justification-then-score** approach:\n1. **Justification Step:** The evaluator first analyzes key matches or mismatches between the model’s response and the ground truth, providing a textual justification.\n2. **Scoring Step:** Based on this justification, the evaluator assigns a semantic alignment score on a scale from 0 to 1:\n\t- **1.0** – Perfect alignment\n\t- **0.8-0.9** – Good alignment\n\t- **0.5-0.7** – Partial alignment\n\t- **0.1-0.4** – Weak alignment\n\t- **0.0** – No alignment\n\nThis justification step is crucial for ensuring fair and stable score assignment. In this work, we employ GPT-4o as the evaluator.\nIn the current version of the paper, we briefly mention the use of GPT-4o in **Section 4, line 306**. We will include this detailed explanation in the revised version (or supplementary materials) to improve clarity.\n\nWe sincerely thank the reviewer for highlighting this point.\n\n**Q2: Are there any method to calculate the Prompt Following metric without human?**\n\nThis is an insightful question, and we fully agree that an **automatic evaluation strategy** would enhance the scalability of our benchmark.\n\nIn our **main paper (Table 4)**, we report human-annotated results to provide reliable insights into the performance of current T2V models. Additionally, in **Appendix B.2**, we introduce an **automatic evaluation strategy** for impossible video generation. Specifically, the Prompt Following metric can be assessed using state-of-the-art Vision-Language Models (e.g., GPT-4o) in conjunction with a carefully designed prompting strategy. Experiment result clearly demonstrate the consistency between human evaluation and auto-evaluation, as revealed in Table 6 and Figure 6 in the Appendix.\n\nWe appreciate the reviewer’s insightful question and will further emphasize this discussion in the revised version.\n\n\n**Q3: This paper should discuss some other benchmark regarding both the video understanding and generation.**\n\nWe appreciate the reviewer’s comment. In the Related Work section and Table 1, we have comprehensively discussed the relationships and distinctions between IPV-Bench and existing benchmarks across three key areas: video understanding, video generation, and AIGC video detection. This comparison highlights the unique contributions of IPV-Bench and its role in bridging gaps that are not addressed by prior benchmarks.\n\nWe will ensure that this discussion is clearly emphasized in the revised version. Thank you for your valuable feedback.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A, not proofs in this paper.", "experimental_designs_or_analyses": "Yes, the experimental designs are validity.", "relation_to_broader_scientific_literature": "This paper proposes two benchmarks for evaluating the understanding and generation model in video domain. These benchmarks focus on the impossible videos, a field previous benchmarks ignored. They are very valuable for the practitioner to probe whether their model truly understand the real world law or simple memory the training set.", "essential_references_not_discussed": "This paper should discuss some other benchmark regarding both the video understanding and generation.", "other_strengths_and_weaknesses": "Strength:\n1. The motivation of this paper is compelling and thoughtful.\n2. The paper is easy to follow.\n3. Results in the paper is valuable for the research community to better understand current video understanding and generation task.\nWeakness:\n1. How to calculate the score in the Open-ended QA task?\n2. When calculating the Prompt Following metric, it needs the human involve, which is time consuming and cost unfriendly. Are there any method to calculate this metric without the human?", "other_comments_or_suggestions": "No", "questions_for_authors": "See the weakness in the Other Strengths And Weaknesses section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8vAcvxFtb2", "reviewer": "Reviewer_AP4m", "summary": "The paper introduces IPV-VID, a dataset designed to evaluate video understanding models on \"impossible videos\", which depict scenarios that violate commonsense. The study evaluates from two perspectives: video understanding and video generation. For video understanding, benchmark tasks such as VideoQA and video-text alignment reveal that state-of-the-art models struggle significantly on \"impossible videos,\" exposing their limitations in temporal reasoning and commonsense knowledge. For video generation, the paper assesses the ability of text-to-video (T2V) models to generate high-quality \"impossible videos\" and proposes IPV-Score to evaluate their semantic consistency and visual quality. The findings highlight the challenges and opportunities for advancing both video understanding and generation in counterfactual and out-of-distribution scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the encouraging review and valuable suggestions. We appreciate your acknowledgment of the paper’s novelty and will carefully address your concerns to improve clarity and rigor.\n\n**Q1: Disentangling Impossibility vs. Synthetic Data Quality**\n\nDisentangling impossibility from synthetic data quality is crucial for strengthening the rigor of our study. We leverage the Multi-choice QA task to investigate this. Due to the limited time during rebuttal, we were unable to generate and annotate **a large-scale set** of synthetic videos. We conducted a preliminary study on a smaller dataset:\n1. Data Collection: We collect 420 synthetic videos from an existing work [1], generated using HunyuanVideo.\n2. Annotation: We label each video with an action description.\n3. Filtering: Videos were excluded if they:\n\t- Lacked an explicit event (e.g., landscape scenes).\n\t- Contained counterfactual content (ensure distinction from impossible videos).  \n        After filtering, 200 videos remained.\n4. Task Construction: We instructed GPT-4o to generate multi-choice questions and answers, adapting prompt from the impossible video setting.\n5. Evaluation: Several popular VideoLLMs were evaluated and compared to impossible videos.\n\n|Model|Acc. (Normal Vid)|Acc. (Impossible Vid)|\n|-|-|-|\n|Video-LLaVA|70.0|25.8|\n|NVILA|89.5|62.2|\n|LLaVA-NEXT|90.0|83.4|\n\nThe results show that **models consistently perform better on normal synthetic videos** than on impossible videos, indicating that **\"impossibility\" introduces a distinct challenge** for video understanding.\n\nRecent studies [2] suggest that the visual quality gap between synthetic and real-world videos is rapidly shrinking. This further underscores the unique challenges posed by impossible videos, which are independent of synthetic data quality.\n\nWe appreciate this valuable suggestion. We will conduct a more comprehensive study in the revised version.\n\n**Q2: Code/Data Release**\n\nUpon acceptance, we will publicly release all **code and data**, including the IPV-Bench taxonomy, IPV-TXT prompts, IPV-VID videos, and evaluation protocols. To ensure accessibility, we will also include a detailed **Data Usage Instruction** in the Appendix.\n\nRegarding concerns about video quality, e.g., action coherence, our human annotation filters out low-quality videos:\n- If the action forms a semantically meaningful impossible phenomenon, we consider it a valid sample.\n- If the action exhibits inconsistencies, artifacts, or unnatural distortions, we classify it as low-quality data and exclude it.\n\nFor a detailed data filtering criteria, please refer to **Response 2 of Reviewer F9mi**. Besides, we provide sufficient video examples on the anonymous website in the paper abstract for further reference.\n\n**Q3: For T2V, any difference in visual quality and prompt-following between normal and impossible videos?**\n\nCurrent T2V models are primarily optimized for normal videos, while impossible videos is often overlooked. Our assumption: creating impossible videos is more challenging than creating normal ones. To explicitly verify this, we conducted a human evaluation on normal text prompts:\n1. We collected 420 synthetic videos from an existing work [1], generated using the HunyuanVideo model.\n2. We annotated the visual quality and prompt-following of these videos.\n3. During annotation, we excluded 83 prompts that describe impossible phenomena and evaluated the remaining 337 normal prompts.\n\n||Visual Quality|Prompt Following|\n|-|-|-|\n|Normal|92.6|66.5|\n|Impossible|88.9|37.2|\n\nWe observe that 1) normal visual quality is slightly better than impossible ones; 2) normal prompt following is significantly better than impossible prompt following, which further underscores the unique challenges posed by impossible videos.\n\nWe appreciate this insightful suggestion. In the revised paper, we will conduct a more comprehensive study to further strengthen our findings.\n\n**Q4: Can impossible videos serve as negative samples to improve Video-LLM in reasoning tasks or enhance T2V generation quality?**\n\nWe recognize the potential in these directions:\n\n-   For Video-LLMs, impossible videos can serve as high-quality training samples to enhance reasoning capabilities. Since these videos introduce \"novel\" counterfactual knowledge beyond real-world data, they may help models develop stronger reasoning and generalization skills.\n-   For T2V models, while the community has focused heavily on physical law adherence, there is limited understanding of how to explicitly improve this. Impossible videos could serve as negative samples, reinforcing a more structured comprehension of physical plausibility.\n\nBoth directions present exciting and meaningful research opportunities. We hope the release of Impossible Videos will inspire further exploration in these areas.\n\n[1] The Dawn of Video Generation: Preliminary Explorations with SORA-like Models. arXiv 2024.\n\n[2] Cosmos World Foundation Model Platform for Physical AI. arXiv 2025.", "claims_and_evidence": "**Claim1: Existing video understanding models struggle with \"impossible videos\" due to their reliance on commonsense and reasoning beyond real-world scenarios.**\n\n**Evidence**:\nExperimental results show that state-of-the-art video understanding models perform significantly poor on the IPV-VID dataset, suggesting their limitations in understanding \"impossible videos.\"\n\n**Comment**:\nWhile the results indicate poor performance on \"impossible videos,\" this may not necessarily be attributed to the \"impossibility\" itself. The performance drop could also stem from the gap between synthetic and real-world data, such as differences in visual quality or semantic consistency.\nThe lack of comparative experiments on similarly scaled \"normal synthetic videos\" makes it difficult to isolate the effect of \"impossibility\" as the primary cause of the performance drop. Drawing such a conclusion without addressing this gap might be risky.\n\n**Suggestions for Improvement**:\nTo better validate the unique impact of \"impossibility\" on model performance, future studies could include a control group with normal synthetic videos. This would help disentangle the influence of data quality from that of \"impossibility.\"\n\n**Claim2: The IPV-VID dataset provides high-quality \"impossible videos\" that are semantically consistent and visually realistic**.\n\n**Evidence**:\nThe dataset was generated using T2V models and manually filtered to ensure semantic consistency and visual quality. The authors also proposed the IPV-Score as a metric to evaluate the quality of generated videos.\n\n**Comment**:\nDespite the reported quality control, generating high-quality videos remains a significant challenge for current T2V models. For instance, many models struggle with action coherence (e.g., unnatural movements and violations of physical laws), and only a few models, such as Hunyuan and Hailuo, perform reasonably well in this regard.\nSince the dataset has not been fully released, it is unclear whether all \"impossible videos\" in the dataset strictly meet the claimed quality standards. Issues such as motion coherence and physical consistency may directly affect the validity of downstream tasks, but these aspects were not thoroughly addressed in the paper.\n\n**Suggestions for Improvement**:\nIncreasing the dataset's transparency by fully opening sources would enhance its credibility.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Based on the provided document, there is no explicit mention of formal mathematical proofs or theoretical validations for the claims made. The paper primarily focuses on the construction of the IPV-BENCH benchmark, categorization of impossible video types, and empirical evaluations of existing video understanding and generation models. These aspects are more experimental and empirical in nature rather than relying on theoretical proofs.", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper are well-constructed and align with the goal of evaluating video understanding and generation models using the IPV-BENCH benchmark. For video understanding, the results demonstrate that existing models struggle with \"impossible videos,\" as evidenced by their poor performance on the IPV-VID dataset. This supports the claim that these models face challenges when reasoning beyond real-world scenarios. However, while the findings highlight a significant performance drop, it is worth noting that this may not solely be attributed to the \"impossibility\" of the videos. The gap between synthetic and real-world data, such as differences in visual quality or semantic consistency, could also play a role. Without comparative experiments on similarly scaled \"normal synthetic videos,\" it remains difficult to isolate the unique impact of \"impossibility\" on model performance. This limitation in the experimental design underscores the need for a control group to disentangle these factors in future studies.\n\nFor video generation, the IPV-VID dataset is presented as a high-quality resource of \"impossible videos,\" with semantic consistency and visual realism ensured through manual filtering and evaluation metrics like the IPV-Score. While these efforts are commendable, challenges inherent to current T2V models, such as issues with action coherence and physical consistency, suggest that the dataset's quality may not fully meet the claimed standards. Additionally, since the dataset has not been fully released, it is difficult to independently verify the quality of the videos or their impact on downstream tasks. A more transparent approach, including full dataset release and quantitative analyses of video quality, would strengthen the dataset's credibility and its utility for benchmarking.", "relation_to_broader_scientific_literature": "The paper’s contributions align with ongoing research in video understanding and generation, extending prior work on real-world datasets like Kinetics and Something-Something by introducing \"impossible videos\" that challenge models to reason beyond physical laws. This builds on ideas from benchmarks like CLEVRER but applies them to dynamic video data. In video generation, the paper advances text-to-video (T2V) research by focusing on generating semantically consistent \"impossible videos,\" addressing limitations of current T2V models in temporal coherence and complex prompts. The proposed IPV-Score complements existing evaluation metrics like FVD and CLIP alignment. Additionally, the IPV-BENCH benchmark fills a gap in AI evaluation by targeting counterfactual reasoning, contributing to the trend of specialized benchmarks. These efforts position the paper as a significant step in addressing underexplored challenges in video reasoning and generation.", "essential_references_not_discussed": "Up to me, it seems to be sufficient.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. Easy to follow for the readers\n2. Interesting.\n\n**Weakness**\n1. The experiment design and analyses can be imporved if the authors compare the performance between synthetic normal and impossible videos. More analysis could be offered on the difference between synthetic videos and the real ones.\n2.  Not fully open-source and the \"impossible videos\" data may not reach the standard that supposed to be.", "other_comments_or_suggestions": "1. To better validate the unique impact of \"impossibility\" on model performance, future studies could include a control group with normal synthetic videos. This would help disentangle the influence of data quality from that of \"impossibility.\"\n2. Increasing the dataset's transparency by fully opening sources would enhance its credibility.", "questions_for_authors": "1. For the synthetic data generated by current T2V models, is there a significant performance difference in downstream video understanding tasks between \"normal\" videos and \"impossible\" videos? Similarly, for text-guided video generation, are there notable differences in visual quality and prompt-following between \"normal\" and \"impossible\" videos?  \n  Clarifying this would help determine whether the challenges observed are truly due to the \"impossibility\" of the videos or are influenced by general limitations in synthetic data quality. This could validate or challenge the paper's claims about the unique challenges of \"impossible videos.\"  \n\n2. Can \"impossible videos\" serve as negative sample guidance to significantly improve the performance of Video-LLM models in reasoning tasks or enhance T2V video generation quality?  \n  If \"impossible videos\" can effectively guide reasoning or generation improvements, it would strengthen the practical value of the dataset and benchmark, highlighting its utility beyond evaluation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper analyses generated videos that defy reality. The analysis consists of two aspects: from the perspective of video understanding models, and from the perspective of video generators, whether they can follow unrealistic prompts. The reviewers overall suggest acceptance, as the results are interesting. The paper doesn't propose novel ideas or theoretical analysis. It proposes a new benchmark generated using existing state-of-the-art video generators. AC believes that the paper has value as of now, but very likely it'll need to be revisited in the future as video generators, as well as understanding methods will change. 

The decision to accept the manuscript is conditional upon the authors’ commitment to release the benchmark code and data.
Decision: Accept (poster)