Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation
Authors: Agneet Chatterjee
Rahim Entezari
Maksym Zhuravinskyi
Maksim Lapin
Reshinth Adithyan
Amit Raj
Chitta Baral
Yezhou Yang
Varun Jampani
Venue: NeurIPS 2025 poster ()
Keywords: Video Generation
Professional Controls
Evaluations
Primary area: applications
TL;DR: Structured Taxonomy and Evaluation for Professional Video Generation

Abstract: Recent advances in video generation have enabled high-fidelity video synthesis from user provided prompts. However, existing models and benchmarks fail to capture the complexity and requirements of professional video generation. Towards that goal, we introduce Stable Cinemetrics, a structured evaluation framework that formalizes filmmaking controls into four disentangled, hierarchical taxonomies: Setup, Event, Lighting, and Camera. Together, these taxonomies define 76 fine-grained control nodes grounded in industry practices. Using these taxonomies, we construct a benchmark of prompts aligned with professional use cases and develop an automated pipeline for prompt categorization and question generation, enabling independent evaluation of each control dimension. We conduct a large-scale human study spanning 10+ models and 20K videos, annotated by a pool of 80+ film professionals. Our analysis, both coarse and fine-grained reveal that even the strongest current models exhibit significant gaps, particularly in Events and Camera-related controls. To enable scalable evaluation, we train an automatic evaluator, a vision-language model aligned with expert annotations that outperforms existing zero-shot baselines. SCINE is the first approach to situate professional video generation within the landscape of video generative models, introducing taxonomies centered around cinematic controls and supporting them with structured evaluation pipelines and detailed analyses to guide future research.

Stable Cinemetrics: Structured Taxonomy and Evaluation for Professional Video Generation
Agneet Chatterjee 2,,
Rahim Entezari* Reshinth Adithyan' Yezhou Yang"
Maksym Zhuravinskyi*
Max Lapin' Chitta Baral?
Amit Raj² Varun Jampani'
Stabiiitt AI
2 Ariiona State University
Googge DeepMind
https://stabe--cneeeettttt github..hhi... io/
Abstract
Recent advances in video generation have enabled high-fidelity video synthesis from user provided prompts. However, existing models and benchmarks fail to capture the complexity and requirements of professional video generation. Towards that goal, we introduce Stable Cinemetrics, structured evaluation framework that formalizes filmmaking controls into four disentangled, hierarchical taxonomies: Setup, Event, Lighting, and Camera. Together, these taxonomies define 76 fine- grained control nodes grounded industry practices. Using these taxonomies, construct benchmark of prompts aligned with professional use cases and develop an automated pipeline for prompt categorization and question generation, enabling independent evaluation of each control dimension. We conduct large- scale human study spanning 10+ models and 20K videos, annotated by a pool of 80+ film professional.. Our analysis, both coarse and fine-grained reveal that even the strongest current models exhibit significant gaps, particularly in Events and Camera-related controls. To enable scalable evaluation, we train an automatic evaluator, a vision-language model aligned with expert annotations that outperforms existing zero-shot baselines. SCINE is the first approach to situate professional video generation within the landscape of video generative models, introducing taxonomies centered around cinematic controls and supporting them with structured evaluation pipelines and detailed analyses to guide future research.
Introduction
The field of video generative models has made significant progress in recent years [37], drawing substantial interest from both academia and industry. This can be evidenced by the growing number of benchmarks (25 32 35), datasets [41 58], and both open- B 56 14 65 and closed- [6 54 36)
source models that have collectively driven the field forward. The foundational nature of these models makes them useful for several downstream tasks, including video editing (26), 3D generation (55) and roboties (69] This widespread adoption of video generative models underpins the growing assertion that they represent a revolution for professional video generation.
Generative vision offers tremendous potential for media creation, but fundamental question remains: how can we shift generative video from casual, exploratory synthesis to a medium that supports professional-grate controllable cinematic outputs? The important distinction between casual and professional generative video lies in the critical gap of cinematic control [7]: while today's 's models generate videos of "an astronaut riding horse", professional creation necessitates granular control over cinematic elements such as the framing of the shot, position of the key light, and even whether the astronaut smiles before after the horse gallops away truly professional video generation system must put every one of those cinematic choices back in the creator's 's hands. The need for this exact control over every cinematic element, from the timing of smile to the quality of
Work done during an internship at Stability AI.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

VM6.2 PM4
3099
3214
Setup
Camera
Lighting Events
Heryaw
900
1030
1218
3333
Dry
996
110
1630
540 AaMUs
800
8/2
A07
UCVVe
LLM
+ Categories
0000
Taxonomy
Prompts
Questions
10+
80+
20K
248K
Models Evaluated
Annotators
Videos
Annolations
Figure 1: Stable Cinemetries introduces structured taxonomies grounded in the controls required for professional video generation. These taxonomies form the foundation of our prompt based benchmark that mirrors real-world shot creation, progressing from scriptwriting to on-screen visuals. Every control element in prompt is automatically categorized back to the taxonomy, enabling the generation of isolated evaluation questions for independent investigation into each element. This supports large scale human evaluation enabling both coarse and fine-grained insights into the capabilities current models for professional video generation. To drive scalable annotations, we develop own VLMs that outperform existing models in alignment with human judgements.
light, is the very reason filmmakers shoot multiple takes, selecting only the frames where everything comes together to tell the story most effectively [10].
In this work, we investigate the intersection of video generative models and the nuanced control mechanisms associated with professional video production. Despite rapid advancements in video generative modeling, the field still lacks both a clear definition of essential cinematic controls and standardized evaluation protocols for benchmarking progress at this crucial intersection. To bridge this gap. we present SCINE (Stable Cinemetrics) an evaluation suite specifically designed to characterize this intersection, enabling us to directly address the question: "Are Current Video Generative Models Ready for Professional Use?"
The core focus of SCINE is to develop a taxonomy, at the intersection of generative vision and professional video generation, built on the principles of control and specificity 28 43 These principles are important in film-making, as every decision carries cinematic meaning; a low angle conveys power while low-key lighting evokes drama. SCINE captures these principles by orga- nizing control knobs into four cinematic pillars- Setup, Events, Lighting, and Camera enabling the evaluation of video generative models along the same axes that a professional would. This disentanglemen also allows evaluation around personalization: which aligns with the collaborative nature of real productions, in comparison the monolithic nature of current video generation models. Our taxonomy is hierarchical, branching from coarse cinematic concepts to leaf-level controls that naturally map onto computer vision concepts such as object semantics and scene geometry (49
Leveraging our taxonomy, we generate two prompt types: story-driven and visual exposition, to mirror professional workflows. Story-driven prompts act as mini-screenplays [17, specifying characters. dialogue, actions, and emotions. We enrich these with visual exposition cues, by sampling control nodes from our taxonomy, emulating the transition from script to shot filmmaking [46). Sampling control nodes allows automated (a) prompt categorization: mapping each control element to the taxonomy, and (b) generation of targeted evaluation questions for each element, allowing disentagled evaluation of each cinematic control. The structured nature of our taxonomy supports scalable human annotation: we evaluate 10+ models across 20K generated videos with feedback from 80+ professionals. This enables analysis across control dimensions where we observe substantial variance performance across taxonomy pillars, even for rop-performing models such as WAN-14B and Minimax. Our taxonomy facilitates both coarse insights, showing that models struggle most with

---

Events and Camera and fine tme-grained comparisons, such as better performance on shot size over camera framing, and on natural over artificial lighting. To support automatic evaluation of fine- grained cinematic controls in generated videos, we train vision-language model (VI.M) that aligns with the large-scale human annotations. Our model outperforms existing baselines, achieving an overall accuracy of 72.36% with human annotators. An overview of SCINE, outlining its contributions is shown in Figure [
2 Related Work
Video Generative Models and Evaluations, Video generative models can broadly be classified into categories based on their input conditioning: image-to-video (I2V) and text-to-video (T2V). While 12V approaches such as Stable Video Diffusion (SVD) K have been widely adopted by the community, the focus of our work to evaluate T2V models for professional use. We choose text as an input modality, since it is an effective and free-form way of describing the controls defined in our taxonomy. The Sora Preview 6 served as a catalyst for wave of T2V model releases across the closed [54 11 and open-source communities. State of the art open-source models include Wan [56], Hunyuan Video [4] and VtepVideo [2]. Several T2V evaluation benchmarks have also emerged, gaining broad adoption. VBench evaluates T2V models by developing text prompts that evaluate generated videos across dimensions such as temporal flickering. aesthetic quality and, motion smoothness, while employing automatic metrics. Additional T2V benchmarks include VideoPhy (2), which evaluates physical plausibility by measuring adherence of generated physics, and T2V-CompBench [48]. which studies compositional consistency [23 in video generation. Prior work lacks the shot-level structure and cinematic detail needed for professional control; motivating taxonomy and benchmark to capture the nuanced elements of filmmaking. Existing benchmarks lack cinematic depth; for example, a prompt such as "A man is walking" from VBench-2 [67] omits key details like character appearance camera movement; all essential to setup a cinematic shot. Existing benchmarks are static, relying on fixed prompt sets that limit extensibility. In contrast, SCINE's taxonomy-guu guidded prompt generation enables future-proof evaluation, allowing prompt complexity to scale with model capabilities.
with VBench 25 67
videos to
industry-standee
Structured Video Generation and Shot-Level Control. MovieNet (4 provides large-scale movie annotations including scene boundaries, cinematic styles, and character metadata. Storyboard approaches like VDS 44 and VAST (66] generate structured video via intermediate pose/layout representations. Multi-shot generation has been addressed through systems like MffcoGGn--ff-hhuueee [68] and MovieAgent [62], which use hierarchical reasoning to plan and synthesize sequences. However, these efforts overlook the fine-grained structure of a single shot which core unit of cinematic composition. They often emphasize isolated factors, without modeling the full set of interdependent creative controls. In contrast, SCINE introduces a unified taxonomy that captures the complete spectrum of shot-level cinematic elements.
Stable Cinemetrics
The subsequent sections detail our proposed tax- onomy (Section 3.1.. and its underlying design principles. Next, we develop our benchmark comprising of prompts designed for professional use (Section 3.2) Section 3.3 describes how prompt categorization and question generation are performed, enabling large-scale evaluations of video generative models for professional use.
Arkm
Teglest
Pertyyd
Baliges
3.1 Taxonomy Design
Diggr Evirercy
Sim Stukker
Regelerr
Our taxonomies are developed in iteration with industry professionals, including organizations established by the Big Five Studios [7]. in- dependent cinematographers and screenwriters, and an Academy Award winning Visual Effects
Figure 3: The Events taxonomy captures the nar- rative dimension of a shot which includes actions, emotions, and their fine grained portrayal as they evolve over time within shot.
3

---

Fare
Dopth oieil
Aperiazc
Expasaze
Seater Speed
Ieeas
BS0
Angle
Eaki Faler
Supes
Carern
Statiz
Nezz
Cursraa Miversca)
Sunz
Lireory
Spasil Learse
Trijetary
Shet Sine
Forrlig
Nattrnal
(b) The Camera taxonomy defines all controls related to camera confi onfiguration during a shot setup.
De
Azvecco
Oxpatiation
of
Negrlit
Overr Tregerane
Seucc
Aniifcal Prasicck)
Narui
Hall
Mikee
ligriay
Uyting
Stadawe
Suli
Eflab
Harl
Raligened
Refkezies
Foetion
(a) The Setup taxonomy outlines the visual compo- nents within the frame, including subjects, props, and environmental context.
(c) The Lighting laxonomy specifies the illumination of shot, through light sources, their properties, and their interaction with the scene.
Figure 2: Setup, Camera and Lighting Taxonomies that structure the visual elements of shot.
Artist. The central guiding question in our taxonomy development was: "What controls do profes- sionals require when setting up shot?".
A shot is the atomic unit of filmmaking; uninterrupted sequence without cuts in which cinematic meaning emerges from the coordination of multiple cinematic choices. The average shot length (ASL) in feature films is 5-10 seconds [8), which closely aligns with the temporal limits of current video generation models. This duration is not a limitation; it is a compact canvas where rich and enough narrative and visual complexity emotion, and story. This motivates our decision to design the taxonomy at the shot level, where fine- grained control is paramount. Control is the key distinction between casual and professional video creation. In casual settings, users often accept model outputs with minimal intervention, delegating the key creative decisions to the model. In contrast, professional demands precise, deliberate control at every stage of the generative process. In fact, pixel-level control is common in film-making 29 6I], underscoring the importance of fine-grained adjustments achieving the desired visual effect. Professionals such as cinematographers and directors are primarily responsible for defining shot's creative intent. While filmmaking collaborative, these roles offer practical abstraction for modeling control. A key insight is that, despite overlap. they are sufficiently disentangled to support distinct control dimensions: screenwriters rarely specify lighting or camera movement, and production designers are typically not associated with emotional tone or narrative pacing. These factors motivate the development of our control pillars, each contributing to the composition of shot. Our taxonomies are structured as hierarchical trees, where leaf nodes correspond to the most granular control parameters, each associated with set of values. We describe each pillar below:
unfold. A single shot entails numerous controls to convey intent,
Setup. Setup (Figure 2ap encompasses all visible elements within the frame. We organize into three top-level groups: (I) Scene aggregates environmental controls, including Texture, Geometry and Set Design. Texture covers aspects such as contrast and color palette, which govern the surface feel of the shot. Geometry captures dominant shapes and the spatial arrangement of elements within the scene. Set Design comprises Props and their attributes; the Backdrop, which establishes the macro context of the set; and Environment, which defines micro-level elements contributing to the "feel" of the shot. (2) Subjects refer to the focal characters within shot, defined by attributes such as costumes and accessories.
(3) Text Generation refers to on-screen typography such as titles or lettering, designed to appear as integrated graphical elements. Each node in Setup has cinematic meaning: a dawn (Time of Day) setup combined with mist (Elements) can suggest danger, while clean (Organization), symmetrical hallway (Balance) conveys order.

---

Table 1: Structured prompt upsampling with SCINE taxonomies. We show how control nodes from our taxonomies enable the generation of visually expressive (SCINE Visuals) prompts from narrative scripts (SCINE Scripts). The table also demonstrates how single script can yield multiple visual interpretations, enabled by our taxonomy guided prompt generation pipeline. This aligns with filmmaking principles, where a script can be Visually realized in diverse ways depending on the creative choices made by the filmmakers.
Baseline script: man serves dinner his family.
Taxonomy Branch
Baseline choice - narrative impact
Aliernative choice - narrative impact
Depth of Field
Shallow - isolates food/serving hand, ro-
Deep - every family member equally sharp.
(Camera)
mantic warmth
ensemble clarity
Camera
Movement
Statie tripod gentle dolly-in -> calm locus
Handheld tracking > urgency, energetic fam-
(Camera)
gesture; subtle emphasis
ily chans
Lighting
Source
Warm tungsten practicals + cozy, inviring domestic glow
Cool morning daylight through windows
(Lighting)
brisk freshness and emotional distance
Backdrop Time of Day (Setup)
Evening interior
nostalgic comfort,
Bright morning interior + optimism and up- beat tempo
winding-down mood
Props (Setup)
Farth-tone wooden utensils + homely
Silver cutlery + formal, upscale
Upsampled prompt A nun serves dinner his Jamuly with shollow depih of field on static tripod with a genile dolly-in, under warm tungsten interior lighting in the evening, in cory earth-tone kitchen with wooden utensils.
Lighting. "Lighting is the key to turning amateur footage into professional stories and presentation" Jay Holben [21]. Motivated by this principle, we define the following groups for Lighting (Figure (b): (1) Source, the origin of illumination within the shot: (2) Color Temperature, which controls the warmth of the light; (3) Lighting Conditions, preset configurations describing scene-wide illumination; (4) Effects, visual outcomes resulting from light interacting with the scene; (5) Position, the spatial relation of the light source to the subject; and (6) Advanced Controls such as flickering modulation and the use of color gels to adjust lighting hue. Each control knob corresponds to distinct cinematic expressions: shot with only back light (Position evokes mystery, while hard shadows are often used to amplify tension.
Camera. The camera taxonomy (Figure 2cb encompasses all camera-related control dimensions involved in a shot. We organize these into high-level groups: (1) Intrinsics: optical and exposure parameters governing the light captured by the camera; (2) Extrinsics: position and orientation of the camera relative to the subject; (3) Trajectory: motion of the camera and the supporting gear that enables it; and (4) Creative Intent: compositional choices that shape the narrative or emotional tone of shot. Prior works have primarily focused on camera motion control [99 22]; however, we show that a much broader range of camera parameters can be independently manipulated while setting up a shot. Each parameter has tangible cinematic impact; for example, a shallow depth of field can isolate the subject from the background to direct emotional focus while, an insert framing spotlights narrative details with precision.
Events. Events (Figure 3 encodes the narrative substance of a shot, namely the depicted actions, emot emotions, and dialogues which are further decomposed into dependent nodes for fine-grained control. These dependent nodes represent attributes that cannot exist independently of their parent categories; for instance, these nodes can specify the type of interaction or the delivery mode of a dialogue. Emotions may appear explicitly (visible tears) or implicitly (a clenched jaw), while actions can be stand-alone or interactive. The Portrayed As category captures aspects such as: Temporal, which refers to the unfolding pattern of the event (e.g., laughter erupting simul taneously vs. sequentially), and Contextual, which indicates whether the event occurs the foreground or background. Advanced Controls refines pacing and the story structure of the shot, such as a turning point or climax. While recent works [59] [6] evaluate T2V models on sequential event generation, we show that, from a professional standpoint, Events encompass a much broader and richer evaluative space.
The taxonomies define total of 76 leaf-level controls that can be independently adjusted when crafting shot. We structure our taxonomies as hierarchical trees to enable disentanglement and multi-level abstraction of cinematic controls. Attributes within each branch are highly correlated, while branches remain independent, ensuring, for example, adjusting Depth of Field does not affect Camera Movement. The tree structure naturally supports multi-level abstraction, aligning with how filmmakers conceptualize scenes, starting from high-level intent and refining toward specific implementation. For example, directive such "set tense alley at night" can be decomposed
5

---

into an EXT Setting, cool Color Temperature and a Deep Depth of Field. This structure also allows easier scalability; adding a new detail like floating ember sparks. fits cleanly under (Environment Elements) without disrupting the rest of the taxonomy. An alternate structure such as flat, linear list would not support such extensibility.
Developing the taxonomy is a non-trivial task. This required multiple iterations with experts since professionals interpret and prioritize controls differently. Furthermore, shot creation is multi-stage process, starting from script-writing to setup design to camera blocking, making unification under single structured framework challenging. While some taxonomies [40] focus only on filmmaking aspects, they lack structure and are not aligned with generative modeling. Our goal instead was to impose structure, and develope a taxonomy that bridges professional filmmaking and generative video models. Details of each control node and their corresponding values are provided in Appendix A.1
3.2 Designing Prompts for Professional Use
The taxonomies form the foundation for con- structing prompts tailored to professional use. Our core approach in creating prompts involves sampling values from the control nodes and creating prompts that reflect realistic cinematic intent. Mirroring the filmmaking process. we first generate narrative scripts and then inject visual elements into these scripts:
Scripts. These prompts, referred to as SCINE- Scripts, contain the narrative content of individ- ual shots. We collaborate with professional screenwriter to create seed prompts that meet strict constraints: single shot, under 10 sec- onds, with no reliance on off-screen elements. These seed prompts, along with sampled nodes from the Events taxonomy are provided input to LLM, for prompt generation. We use the Events taxonomy for these prompts because directly encodes narrative beats i.e., what hap- pens in shot. It covers nodes such as physical dynamics (actions) and verbal interactions (dialogues), which are crucial to the story conveyed in a shot. ensure prompt diversity, we vary parameters across multiple LLM invocations, sampling emotions from Plutchik's 's model [42). alternating actions, dialogue structures, genres, and subject composition. We use LLMs, as prior work [39 50 have shown their effectiveness in screenplay gen- eration. t-SNE visualization (Figure of SCINE-Scripe Scripts embeddings [57) shows substantial overlap with ground-truth screenplays [45] whereas prompts from VBench-2. 220 (67] and MovieGenBench
Original Scoerplays
SCIE
VBeecc220
Figure t-SNE visualization showing substan- tial overlap between ground truth screenplays and prompts in SCINE Scripts, in comparison to exist- ing prompt based benchmarks such as VBench-2.00
exhibit minimal overlap.
Visual Exposition. We refer to these set of prompts as SCINE- Visuals, which enrich SCINE-Scripts with visual elements from the Camera, Lighting, and Setup taxonomies. In contrast to Events, these taxonomies offer fine-grained control over the visual style and composition of a shot. For each base prompt in SCINE-Scripts, we sample values from one or more control nodes and inject them to expand the prompt with structured visual specifications. SCINE-Visuals highlight a key advantage of taxonomy: structured prompt upsampling. Unlike existing prompt upsampling techniques that delegate all creative decisions to the LLM, our method constrains generation within the taxonomy. enabling more controlled and interpretable prompt expansion. Table I SCINE-Scripts is subsequently upsampled via the taxonomy to generate SCINE- Visuals.
provides breakdown of how
3.3 Category and Question Generation
Next, we extract categories and generate questions for each prompt. Categories link prompts back to the taxonomy, allowing fine-grained evaluations across different levels of abstraction; single prompt can map to multiple categories across different taxonomies. For each category, we generate targeted questions that are shown to human annotators during video evaluations. These questions are specific

---

(a) Overall results on SCINE Scripts, across genres. Minimax and Wan 14B emerge as the strongest mod- cls. The strongest genre is Biography while all models consistently struggle Comedy.
(b) Overall results on SCINE Visuals across four pil- lars of professional control. Events emerges as the most challenging category across models, while Serup yields the strongest performance.
Figure 5: Overall results on SCINE Scripts and Visuals.
in nature and target only single control node, enabling its isolated evaluation. Unlike high-level prompt adherence questions, which lack fine-gr gra-graindd attribution, our framework supports per-control annotations. A minimal example is shown below:
Prompt: A tight close-up focuses fireplace, its embers flickering brightly.
Category: Lighting - Advanced Controls Motion Question Does the exhibit dynamic flickering effects its lighting that align with the description?
Category: Camera + Creative Intent + Shot Size Question Does the video include a tight close-up shot that captures the detailed framing?
Additional details are presented in Appendix A.3
4 Are current Video Generative Models Ready for Professional Use?
We now evaluate state of the text-to-video (T2V) models against the professional standards defined in our taxonomy pillars. Our analysis reveals both strengths and persistent challenges of current models, offering an overview of how current models align with professional quality expectations.
4.1 Experimental Setup
Prompts. The SCINE benchmark comprises two prompt categories, Scripts and Visuals, each aligned with distinct professional roles (Table The Visuals prompts are created by systematically upsampling Scripts using our taxonomies leading to a total of 2,089 prompts. categorize prompts by difficulty: create basic prompts by limiting the number of sampled control nodes, while in advanced prompts, we do not impose any restriction.
Models. We evaluate 13 state-of-the-ar T2V models, both open-source (WAN IB/14B [66. Hunyuan-
Video 14 Step Video [2], Cog Video 5B
65), -iddoo 18 Pyramid Flow 27 Easy Animate
5.1 63 Mochi [52), VChitect-2.0 [5] and closed source (Minimax 38 Luma Ray 2 [30]. Pika
2.2 B1). Our goal to assess each model's 's suitability for role-specific professional tasks, like
evaluating narrative fidelity from screenwriter's perspective. Unless otherwise noted, we use default sampling parameters and maintain a consistent seed per prompt, across models for fair comparison.
Human Annotation Setup. To ensure
Table 2: The SCINE benchmark includes prompts tai- lored professional roles, where each prompt is paired with multiple, fine grained evaluation questions.
high-quality evaluation, we work with a pool of 84 expert annotators with an av- erage of 6.5 years of experience in film production, across roles such as cinematog- raphers, film editors, screenwriters, visual communication designers, and directors. Annotators were shown prompt along with two generated video samples. For each prompt, they were presented with tax- onomy derived evaluation categories and
Target Role
Target Taxonomies
# Prompis
Avg. Questions per
Prompi
SCINE-Scripts
Screemwriters
Events
1133
2.57 0.98
SCIVE: Visuals
Cinemattorapherrs Production Designers Directors
Lighting
355
Sctup
298
4.00 + 3.35
All
7

---

corresponding questions. Each video was rated independently on a 1-5 scale, where indicated com- plete misalignment with the category and indicated perfect match. Although the evaluation was non-comparative, our UX ablation studies showed that displaying two videos side by side improved annotator calibration, especially when selecting middle range scores. To promote consistency and reduce subjectivity, we developed a comprehensive annotation guide covering each control node in the taxonomy. We collect votes for every video-question pair across 13,457 unique questions, collecting a total of 248,536 pairwise annotations. We observe intra-class correlation coefficient (ICC) [51] of 80.4% for the 1-5 ratings at the model-pair level, and 95.5% when the models are considered individually. Further, conduct Wilcoxon signed-rank tests [60] across 45 model pairs where observe statistically sigmificant preferences in 37 of them, which highlights that the annotators agree on model preferences.
4.2 Results and Analysis
SCINE Scripts We first evaluate models narrative event generation, i.e. the story di- mension of a shot. Figure [5 compares model performance across different genres, focus- ing on the accuracy and coherence of generated events. Minimax and WAN-14B emerge as the overall top performers, while Lidevideo consistently underperforms. We observe that models generally perform better on the Biography genre, whereas Comedy proves challeng- ing. In Figure [ we zoom into sub-categories within Events + Types. Minimax leads in nearly all categories, showing the largest margins in Dialogues and Change in Environment, but falls short in Advanced Controls, where WAN-14B outperforms. Further, models are bet- ter at stand-alone actions compared to interac- tive actions and portray implicit emotions better than explicit ones. Within Event Types, Actions exhibit lowest variance across models. Perfor- mance on Causal and Sequential events is highly correlated (p = 0.94), as is performance on Concurrent and Overlapping events (p 0.86) across models. Despite variation across models, all show limitations in multiple Event aspects, highlighting opportunities for improvement.
Adwiic Cortra
Figure 6: Fine-grained evaluation on Events. Mod- els handle environmental changes well but struggle with dialogues and shot pacing. Standalone actions outperform interactive, and implicit emotions are easier than explicit.
SCINE Visuals Figure 5b demonstrates over- all results on SCINE Visuals. Consistent with the pair-wise rankings in Figure | WAN-14B
and Minimax emerge as top performers across all pillars. We find that current models struggle most with Events and Camera, while elements of Semu and Lighting are comparatively easier to capture. Only the top three models- WAN-14B, WAN-IB, and Minimax reliably depict Events, with a substantial performance gap from the rest. While Camera scores are low across the board, the narrow spread suggests that all models face similar limitations. Lighting shows the most consistent performance, with most models achieving relatively high scores whereas Setup yields the highest absolute scores for the top- nop-performing models.
Cinematographer: We evaluate this role by cre- ating prompts that inject control nodes from the Camera and Lighting taxonomy. Within Cam- era, Extrinsics and Trajectory have the lowest average performance and the narrowest inter- model spread. For Lighting, the primary bottle- neck is Lighting Position. We further present results, split by prompt difficulty in Figure 7 Across all models, performance degrades on ad- vanced prompts, indicating that under conditions resembling professional workflows where cin- ematographers have large amount of control, current models struggle. The biggest perfor- mance drops occur in Lighting Source, Color Temperature, and Creative Intent. Lighting Position and
Desic Protz3
Aceereed
Figure 7: Split results on basic vs. advanced prompts for Camera and Lighting. All models show performance drops on advanced prompts, with the largest decline in Lighting Source.
8

---

34
Van
Horean
Caty Axmon
(a) Fine sene-grained results on Subjects, Models perform well hair and accessories, but struggle with person- ality and makeup.
(b) Fine- srineggrined results form better Backdrop in comparison Environment, and struggle most with styling the shot appropriately.
Set Design. Models per-
Figure 8: Fine-graines results Setup across Subjects and Set Design.
Blector
Stastalore
Lighting
Setup
Camera
Figure 9: Director results: Joint specification of all controls, mirroring weal-world shot creation leads to performance drop on all models, compared to evaluation in standalone manner.
Advanced Controls show the smallest performance drops, but remain the weakest categories overall, highlighting persistent challenges regardless of prompt complexity. Hunyuan and WAN-1B exhibit consistent performance across complexity levels.
Production Designer. Models perform strongest on the Setup pillar; within Setu, models achieve comparable performance on Subject and Scene generation, but show a drop Text Generation. In Subjects (Figure Sa). model performances largely vary. with the highest scores for Hair and Accessories, and the weakest in Personality and Make-up. In Set Design (Figure (8b), performance trends follow the order: Backdrop >Props >Environment. Within Props. models perform well in Material but struggle at generating intricate Patterns. For Environment, current limitations lic in adhering to coherent Style and Backgrounds, whereas better performance is seen in organizing Space within a frame.
Director. Prompts targeting this role differ from prior categories as they evaluate models across all taxonomies simultaneously. Model performance declines on average when all controls are defined jointly (Figure 9.. The largest performance drop is observed in Camera, followed by Setup and Lighting. Wan T4B is the only model to show improved performance on Director prompts for Lighting and Setup, compared to its Standalone results.
Our evaluation identifies three-tier hicrarchy among current T2V models: Minimax and WAN-14B at the top, followed by Luma Ray 2, Hunyuan, and WAN-IB, with the remaining models forming the third tier. While overall performance varies, most models struggle with the fine- fine-grained elements critical to professional video generation. For example, atomic events are handled reasonably well, but models falter on concurrent and causal events, which demand deeper temporal reasoning. Similarly, high-level cues like lighting conditions are better captured than nuanced aspects like precise light positioning. In summary, even top performing models exhibit substantial room for improvement across all dimensions of our taxonomy. No model achieves consistently strong performance across all aspects of shot composition, underscoring the challenge of aligning generative video models with professional standards. Additional results and analysis presented in Appendix A.2

---

5 Scalable Evaluation of Professional Videos
In the previous section, we evaluated video generative models using expert annotations across 76 control nodes defined by our taxonomy. While human evaluation remains the gold standard, is costly and difficult to scale, and defining reliable automatic metrics for each control node is non-trivial. Recent advances in vision-language models (VLMs) [B] [T offer a scalable alternative, showing strong performance in video understanding tasks. In this section, we leverage these models to perform automatic evaluation of professional video generation.
Zero-shot VLM Evaluations. The rise of multimodal VLMs has enabled progress on vision-languan tasks, including video understanding, making them natural candidates for evaluating professional video generation. We use expert annotations as ground truth and measure VLM alignment by prompt- ing models with a video, its associated prompt, and specific question tied to a taxonomy node, asking for 1-5 rating similar to our user study. We explicitly instruct the VLM to ignore factors unrelated to the specified category when evaluating the video. We determine VLM preferences by indepen- dently scoring each video and selecting the higher-scoring one. This design mitigates hallucination and order-sensitivg issues commonly observed when prompting with both videos simultancously. We use --L-Intr----sstrucc models due to their strong video understanding capabilities. To study the effect of model scale, we evaluate 3 sizes: 7,32
(ward 541770
Rurse Prtfrrre Ph3
and 72B. Figure 10 shows that increasing model size does not yield significant improvements in
alignment with human judgment. Our results re-
veal overall poor agreement, consistent with prior findings 20 64] that highlight the need for fine- data.
tuning on diiiittriuution,
Aligning Human and VLM ratings. We adopt Owen-2.5-VL-78 [33 as the base model for fine- tuning: our training and validation dataset consist of 44,062 and 12,763 samples, respectively. We aggregate annotator scores (3/video pair) into bi- nary preferences, excluding ties. Videos are pre- processed at fps at their native resolution. Each sample consists of prompt, two videos, and a binary label. The trained model acts classifier; we modify the model architecture output scalar score using a linear projection over the final layer's ' last token. The model takes as input single video, its prompt, and the evaluation question as input. For training, we use the Bradlcy-Terry objective (5) due to its sample efficiency over regression (34). The model is trained for I epoch with batch size of 8 and learning rate 6e-5, Similar to zero-shot evaluation, we compute pairwise preference accuracy against the average of the annotators as the target metric. Our fine-tuned model achieves an overall accuracy of 72.36%, outperforming all zero-shot VLM baselines. This represents an absolute improvement of ~20% over the baseline 7B model. Our model (Figure [0) shows con- sistent performance across videos generated by different models, highlighting its ability to generalize across different video qualities. Additional VLM results are presented in Appendix A.7
Figure 10: Our trained VLM shows consis- tent alignment with human annotations across video generation models, outperforming base- lines, most notably on WAN-14B.
6 Conclusion and Future Work
Stable Cinemetries probes the intersection of professional video generation and generative video models, grounding prompts, evaluations, and analysis in our structured taxonomies. Our findings reveal where current state-of-the-art models perform well, and where substantial improvements are needed. Our prompt suite offers strong testbed for future video generative models and can be casily extended as models improve, owing to the flexibility of our taxonomy. We envision several extensions of our work; while our current focus is on evaluation, the taxonomy can also support analyzing video datasets for cinematic diversity or serve as structure for video captioning. While today's 's text-to-video models are not yet usable in fully zero-shot capacity, our findings identify the main challenging pillars for professional filmmaking, illuminating the need for potential solutions like fine-tuning and customization that can bring these models closer to real production use. We hope SCINE encourages deeper exploration at the intersection of filmmaking and video generative models, fostering closer collaboration between artists and models.
10

---

7 Acknowledgements
We thank Robert Legato, Hanno Basse and Heather Ferreira for their valuable input on our work. We are also grateful to the team at MovieLabs for their feedback on our taxonomies. A special thanks to Cedric Wagrez for his invaluable assistance with the human annotations!
Limitations
Although our laxonomy was developed in consultation with domain experts, it is limited by the scope of our collaborator network. Filmmaking terminology and interpretive nuance vary across regions and cultures, greater expert diversity would enable broader incorporation of global cinematic controls into the taxonomy. Some taxonomy nodes (e.g., Color Temperature, ISO) were abstracted for evaluation, found it difficult for annotators consistently perceive fine-grained values (such as 2000K ISO 800). Prompt generation is based on LLMs, whose proprietary nature and potential biases can influence the language and structure of the prompts. Our zero-shot VLM evaluations bounded by compute and data resources, limiting the scale and scope of the experiments.
References
[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. ArXiv, abs/2502. 13923, 2025. URL https: //api. semant commanticscholarr org/CorpusII http///rpusS::2744411111 276449796
[2] Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical commonsense for video generation. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttpttts://openreiiee net/Forumm1d-=922000ww
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023.
HtLLhttps::/// //ap://///xvv org/ abs/2311. 15127
[4] Bruce Block. The visual story: Creating the visual structure film, TV, and digital media. Routledge, 2020.
[5] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[6] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URI. httptss:///ppaiii wttpaaaaaccceeeeeecccai as- wor simulssamtorrssss
widdo--gnneration-noeee as-
[7] Blain Brown. Cinematography: theory and practice: image making for cinematographers and directors. Routledge, 2016.
[8] James E Cutting and Ayse Candan. Shot durations, shot classes, and the increased pace of popular movies, 2015.
[9] Google DeepMind. Veo tech report. Technical report, DeepMind, 2025. Veo: text-to-video generation system. Available at Veo T-ch ReapchhhReoor..ppf.
[10] Edward Dmytryk, Andrew Lund, and Mick Hurbis-Cherrier. On film editing: an introduction to the art of film construction. Routledge. 2018.
[11] Adam Polyak et al. Movie gen: A cast of media foundation models, 2025. URL. https //axxiv http:/.org/abs////100 13720

---

[12] Guoqing Ma et al. Step-video- technical report: The practice, challenges, and future of video 10248
foundation model, 2025. URL
[13] OpenAl Josh Achiam et al. Gpt-4 technical report. 2023. semanticscholar... ortprioa.....ooooooooousi/a////1
URL https://apt
[14] Weijie Kong et al. Hunyuanvideo: A systematic framework for large video generative models, 03603
2025. URL BtRLittss///xxxi..rrraaa///////
[15] Weichen Fan, Chenyang Si, Junhao Song. Zhenyu Yang, Yinan He, Long Zhuo, Ziqi Huang, Ziyue Dong, Jingwen He, Dongwei Pan, Yi Wang, Yuming Jiang, Yaohui Wang, Peng Gao, Xinyuan Chen, Hengjie Li, Dahua Lin, Yu Qiao, and Ziwei Liu. Vchitect-2.0: Parallel trans- former for scaling up video diffusion models, 2025. URL httpp::/arxxi..org//bbb.
08453
[16] Weixi Feng. Jiachen Li, Michael Saxon, Tsu jui Fu, Wenhu Chen, and William Yang Wang. Tc- bench: Benchmarking temporal compositionality in text-to-video and image-to-vided videeo generation, 2024.
08656
[17] Syd Field. Screenplay: The foundations of screenwriting. Delta, 2005.
[18] Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richard- son. Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon. Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofrr Bibi. Ltx-video: Realtime video latent diffusion, 2024. URL htttttp:://a...oggg///.o//o 2521 .00003
[19] Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for video diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:// net/forum?1d=Z4ev00ng
[20] Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, et al. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. arXiv preprint arXiv:2406. 2522:2220..1555,, 2024.
[21] Jay Holben. A Shot in the Dark: A Creative DIY Guide to Digital Video Lighting on (almost) No Budget. Course Technology Press, 2011.
[22] Chen Hou and Zhibo Chen. Training-free camera control for video generation. In The Thirteenth International Conference on Learning Representations, 2025. URL htttss:/ het/foormm)ddKI121d0077
[23] Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i- compbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation, 2025. /ttps::://// http://aawxxxvovaaaaaae 2307.06350
[24] Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. Computer ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedingg, Part IV 16, pages 709-727. Springer, 2020.
[25] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEF/CVE Conference on Computer Vision and June 2024.
Pattern Recognition (CVPR), pages 21807
[26] Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu, Vace: All-in-one video creation and editing, 2025. URL htttpp://aaxxv.orgee
[27] Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling, 2025. TRLLhttps://aree org/ aaaaa/bs////// 2410. 05954
12

---

[28] Steven Douglas Katz. Film directing shot by shot: visualizing from concept to screen. Gulf Professional Publishing, 1991.
[29] MediaBee
Color Lab.
The art and impact of color grading in cin-
ema,
July 2024. -nw--adddmmaatt---ooooo----
URL https: /wwww mediabeecollrlab. com/ com/post/
cinema Blog post.
[30] Luma Labs. Ray2, 2025. URL https::/lumalaas.ai////
Accessed: 2025-05-02.
[31] Pika Labs. Pika 2.2, 2025. URL httpss///iikllab.orrggggoo
Accessed: 2025-05-02.
[32] Mingxiang Liao, Hannan Lu, Xinyu Zhang, Fang Wan, Tianyu Wang, Yuzhong Zhao, Wangmeng Zuo, Qixiang Ye, and Jingdong Wang. Evaluation of text-to-video gen- eration models: A dynamics perspective. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 109790-109816. Curran Associates, Inc., 2024. URL htttps://proceeteteee
neurips httppppscccpppppeeeeeeeeeeeeeeee httperrrriieeseeeeee/////// pdf]
[33] Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, abs/2304.08485, 2023. URL
[34] Jie Liu, Gongye Liu, Jiajun Liang. Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback. arXiv preprint urXiv:2501. 13918, 2025.
[35] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22139 200-2221499 June 2024.
[36] Luma Labs. Luma ray2 video model. https://lumalabs. ai ai/ray. 2025. Accessed: 2025-04- 23.
[37] Nestor Maslej. Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Capstick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, Tobi Walsh, Armin Hamrah, Lapo Santarlasci, Julia Betts Lotufo, Alexandra Rome, Andrew Shi, and Suknut Oak. Artificial intelligence index report 2025, 2025. URL httpss://arviv.orr/bbb/////////
07139
[38]
Video generation api, 2024. vi video- gpp Accessed: 2025-05-02.
MiniMax.
URL https://www.miinng www.miiiiax.ii/new......
[39]
Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. Co-writing screen- plays and theatre scripts with language models: An evaluation by industry professionals, 2022.
httlhttts///rxitin http://wwwwogg///////0/0 14958
[40] Motion Picture Laboratories. Ontology for media creation: Part 3f:
Images. Technical com//occs//ootooogg/
Report v2.6, MovieLabs, May 2024. URL
besets-images// images/ Accessed: 2025-05-15.
[41] Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid- Im: A large-scale high-quality dataset for text-to-video generation. In The Thirteenth International Conference on Learning Representations, 2025.
URL httphh::tt//p/eeee//e/
net/ mett/forum2dd:7kkkkkr1.
[42] Robert Plutchik. The emotions. University Press of America, 1991.
[43]
Michael Rabiger. Directing: Film techniques and aesthetics. Routledge, 2013.
[44] Anyi Rao, Xuekun Jiang, Yuwei Guo, Linning Xu, Lei Yang. Libiao Jin, Dahua Lin, and Bo Dai. Dynamic storyboard generation in an engine-based virtual environment for video production.
In ACM SIGGRAPH 2023 Posters, pages 1-2. 2023.
13

---

[45] Rohit Saxena and Frank Keller. Moviesum: An abstractive summarization dataset for movie screenplays. 2024. URL
66281
[46] Linda Seger and Edward Jay Whetmore. From Script to Screen: The collaborative art of filmmakingg Lone Eagle Publishing Company, LLC, 2004.
[47] StudioBinder. What are the major film studios?, n.d. URL https: petps://wer wttp:/ssuudibbdddr.coomm com/
blog/what-aree are- tee-ttheeemaaooo major- film- studios/ Accessed: 2025-05-14.
[48] Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2v- compbench: A comprehensive benchmark for compositional text-to-video generation, 2025.
hkI.hhttps//arxiiii http:ppppp//aa..voo.r///a/////// 14505
[49] Richard Szeliski. Computer vision: algorithms and applications. Springer Nature, 2022.
[50] Yuying Tang, Haotian Li, Minghe Lan, Xiaojuan Ma, and Huamin Qu. Understanding screen- writers' practices, attitudes, and future expectations in human-ai co-creation, 2025. URL
https://arxiv http:/awiivo.og//aaae awtgg//bb/222..... 16153
[51] Judith M. Tanur, George Casella, Richard Dykstra, Mark P. Finster, Donald P. Gaver, Joel Greenhouse, Gudmund R. Iversen, guillermina Jasso, Jan Kmenta, S. James Press, Seymour Sudman, Luke Tierney, Jessica Utts, Katherine K. Wallman, Stanley Wasserman, and Mark Watson. Journal of the American Statistical Association, 84(407):8308833 834, 1989. ISSN 01621459, 1537274X. wtt.hhtttp//ww..soorrrr/////////
[52] Genmo Team.
Mochi http:::/ggthhb...mm//gmm/aaa//// 2024.
[53] Qwen Team. Qwen2.5-vl, January 2025. URL http://pp////ee/m.g..t..oobbbbbo
5-v17
[54] Veo-Team. Veo 2. 2024. URL htt.A//er//eeemmnooooo/////////p
[55] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion, 2024. URL https
//arxiv. wtwpiv..rrr/b///240333 120088
[56] Team Wan. Wan: Open and advanced large-scale video generative models, 2025. URL https://arxiv http:/iiw.org/bb//555000 20314
[57] Liang Wang, Nan Yang. Xiaolong Huang, Linjun Yang. Rangan Majumder, and Furu Wei. Multilingual e5 text embeddings: technical report, 2024. URL/https://arii orgp/////.rr//////////////b
1402.05672
(58) Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation, 2024.
/tt/htttss//opperreee
net//foru?2ddddUWWW3BMMM
[59] Yiping Wang. Xuehai He, Kuan Wang, Luyao Ma, Jianwei Yang, Shuohang Wang, Si- mon Shaolei Du, and Yelong Shen. Is your world simulator a good story presenter? consecutive events-based benchmark for future long video generation, 2024. URL https
//arxiv.org/abs/2412 16211
[60] Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs statistics: Methodology and distribution, pages 196-202. Springer, 1992.
[61] Aylish Wood. Pixel visions: Digital intermediates and micromanipulations of the image. Film Criticism, 32(1):22-94 2007.
[62] Weijia Wu, Zeyu Zhu, and Mike Zheng Shou. Automated movie generation via multi-agent cot planning, 2025.
[63] Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: A high-performance long video generation method based on transformer architecture, 2024. URL https://arxiiv wttpaawwwwwi...rr/////////////// 18991
14

---

[64] Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang. Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, al. Visionreward: Fine-gra aine-grineed multi-dimensionall human preference learning for image and video generation. arXiv preprint arXiv:24/2.2 10529 2024.
[65]
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong. Xiaohan Zhang. Guanyu Feng, Da Yin, Yuxuan. Zhang, Weihan Wang. Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-vider diffusion models with an expert transformer. In The Thirteenth Interationnal Conference on Learning Representations, 2025. URL https::/openneviiew htte:://ooeerriiiii.eeererrrrtrr
[66] Chi Zhang, Yuanzhi Liang, Xi Qiu, Fangqiu Yi, and Xuelong Li. Vast 1.0: A unified framework for controllable and consistent video generation, 2024. URL //tpps://axxi....m www. arggabs/2422 2412
16677
[67] Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness, 2025. bttps:tss://axxtt wttp:/wwww..gg/////....... 21755
[68] Mingzhe Zheng, Yongqi Xu, Haojian Huang. Xuran Ma, Yexin Liu, Wenjie Shu, Yatian Pang, Feilong Tang, Qifeng Chen, Harry Yang, and Ser-Nam Lim. Videogen-of-thoughh Step-by-step generating multi-shot video with minimal manual intervention, 2025. URL
https://arxiv org
werrg/bss/244 2412. 02259
[69] Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit- Yan Yeung, and Chuang Gan. Robo- Dreamer: Learning compositional world models for robot imagination. In Ruslan Salakhutdi- nov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 61885-61 61896. PMLR, 21-27 Jul 2024. URL https::///// httt::://prooeedinggs.oo mlr. prwss/rsssssppp. wttres///////a/ 221123////hhh22 html
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction accurately capture the paper's contribution and scope by clearly stating the claims made and outlining primary contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification:
We discuss the limitations of our current work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not present theoretical results, assumptions, or proofs. Instead, it proposes taxonomy for professional movie elements and provides statistical analysis on how current state-of-the-ee text-to-video (T2V) models perform across different pillars of the taxonomy.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
17

---

The paper provides all necessary information to reproduce the main experi- mental results. Details for creating the taxonomy are available in Section The setup for VLM evaluation is described in Section
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release our proposed taxonomy upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
public/guidh saaaid/s/oodeSuumissiooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https? / //iiss httpss.cc/pbblcee http//gguide//oeeuuumii ssionPolicy: for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
18

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper provides all necessary information to reproduce the main experi- mental results. Details for creating the taxonomy are available in Section The setup for VLM evaluation is described in Section
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail
that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report statistically sigmificant annotator agreements within the annotators. Additional details are present in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
19

---

Answer: [Yes]
Justification: Yes, we report these numbers in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:///bburrrssccc/ooooo/oottd
Answer: [Yes]
Justification: The paper complies with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we report the impacts of our current work the Supplementary.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
20

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do anticipate any high risk misuse of our current work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite and credit papers that introduce the pre-trained models we train on, as well as the LLM used for generating prompts, and the metrics we use to compare across models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppaper swi thcode. thccode com coom/daasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release our taxonomy which is well documented.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
21

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We discuss the full text of instructions given to participants and include screenshots in the Supplementary.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: The user study involves participants score/rank videos generated with pre- determined list of prompts and quetions in the professional pillars. As such, we believe the study does not pose any risk harm to participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: provide information on using LLMs in helping us creating prompt set and questions in Section and 3.3
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer LLM policy htttep///eeerrooscccc/ooeerreees for what should or should not be described.
22

---

A Appendix
Taxonomy Details
23
Additional Results and Analysis
24
Details on Prompt Generation
30
Distribution of Taxonomy Categories SCINE Prompts
Annotation Details
Statistical Tests
37
Additional VLM Results
40
Additional Results on Recent Models
Broader Impact
A.1
Taxonomy Details
We provide additional details control nodes and their associated values in the taxonomies. Some nodes accept open-ended values, for example, a range of stand-alone actions. To simplify evaluation, abstract certain values that be one-grained in future works. For instance, into wide/medium/naroow though exact f-stop values can also be studied in the future. Similarly, color palette is treated discrete value in our current work, but can be decomposed into hue, brightness, and saturation. Table 3 6 details the control nodes and their values of the Camera, Lighting, Setup and Events Taxonomies, respectively.
group Aperture
Table 3: Camera Taxonomy Control Nodes and Values
Name
Description
Potential Values
Lens Size
Defines the Focal length and field of view of the camera lens.
Standard, Fisheye. Wide, Medium, Long
Lens, Telephoto
Depth Field
Controls range of in the image, affecting subject isolation.
Deep. Shallow, Solt, Rack, Split Diopter,
Tilt Shift
Aperture
The camera lens opening that controls the amount of light
Wile, Medium, Narrow
propagated through the camera.
Shutter Speed
The duration for which the camera sensor is exposed to light.
Slow, Medium, Fast
ISO
Sensitivity the camera sensor to light.
Low, Modium, High
Angle
Detines the camerr''s viewpoint relation to the subject.
Low, High, Aerial, Overhead, Dutch, Lye- level. Shoulder, Hip, Knee, Ground, Con- tinuous Values
Static 2D
A lixed camera pusition without any movement, Camera movements restricted to horizontal or vertical axes.
Pan left, Pan right, Till up, Tilr down, Zoom
in, Zoom out
3D
Camera movements that incorporate spatial depth and multi-axis motion.
Push In, Pull Out, Dolly Zoom, Camera Roll, Tracking, Trucking, Are, Crane Handheld, Tripod, Pedestal, Cranes, Over-
Gear
Specifies the support systems and stabilization equipment used to facilitate camera movement.
head Rigs. Dolly, Stabilizer, Snorricam, Vehicle Mount, Drones, Motion Control,
Steudicam
Shot Size
Deternines how much of the subject and surroundings are visible frame.
Establishing, Master, Wide, Full, Medium-Full, Medium, Medium Close- up. Close-up. Extreme Close-up
Framing
Placements and composition of subjects within the frame.
Single, Two Shot, Crowd, OTS, PoV, Insert
23

---

Table 4: Lighting Taxonomy Control Nodes and Values
Name
Description
Potential Values
Natural Light
Natural sources light, lielight.
sunlight, moonlight.
Sunlight, Moonlight, Firelight
AriiicialPracticals Light
Man-made light sources that illuminate scenc.
LED. HMI. Tungsten, Fluoreseent, HID
Color Temperature
Defines the huc of the light, typically measured in Kelvin,
Warm, Cool, Cold
affecting the secne's ' mood.
Lighting Conditions
Describes various lighting scenarios or ambient conditions
Candlelight, Golden Hour, White cent, Clear Daylight, Overcast
present in scene.
Soft Shadows
Subtle and diffisee shadows resulting from indirect or scattered light.
Diffised Light, High Key Lighting, Reflec-
Hard Shadows
Sharp. well defined shadows generated by direct light source.
Direct Light, Low Key Lighting
Rellection
The effect light bouncing off surfaces
rellec-
tive appearance.
Lighting Position
Specities the placement or direction of light source
Back Light, Light, Top Light, Side Light. Key Light Plickering, Pulsing
relative to the subject.
Motion
Dynamic changes or movement in the lighting effect.
Color
Colored filters applied lights modify
color of the illumination.
Causal
Atomic
Concurentt
Sequential
Overlapping
Cyclic
3.5
Score
2.5
1.5
Hinimax
Wisn 148
Hinynan
Hochi
Luma 2
StepViiti
2.
Easy Arimase 3.1
Cogrideo 58
Paramid Piez
Vohttet 2.0
Win
ITx
Model
Figure 11: Model performance on Events across temporal portryal of Actions. Atomic actions handled well, whereas models struggle with causal and overlapping Events.
A.2
Additional Results and Analysis
A.2.1 Events
Figure [ ssowws Events performance across models and 6 Temporal portrayal of Actions. Models handle atomic and concurrent actions well, but struggle with causal, overlapping, and cyclic events. Figure 1 shows Events performance across 13 models and 12 genres. Biography and Adventure are strongest whereas Comedy and Horror are the weakest. Minimax leads in 6/12 genres, Luma Ray tops Action and Drama, and WAN-14B is the most consistent, with the lowest standard deviation.
Figure 13 shows performance of 10 open-source models across 19 emotion classes. Models perform best on remorse and ecstasy, but fare poorly on aggressiveness and rage. As shown in Figure [4] dialogue performance is weaker in comparison to Emotions and Actions. Models particularly struggle with multi-turn dialogues or when non-verbal reactions follow. Since T2V models do not generate audio, we evaluate whether the correct character delivers the line and/or with appropriate visual expression. Most models fail localize the speaker, often attributing a single dialogue multiple characters.
24

---

Action
Animation Biography
Comedy Crime
Drama Fantasy
Horror Romance
Science Fiction Thriller
Adventure
Score
Minmmx
van 148
Hunysan
18
Pike 22
Modhi
Caprideo 5g
StepViden
Easy Animate 5.1
Pyramid Flow
vittect 2.0
Valen
Lona
Model
Figure 12: Model performance Events across genres, Across 13 models and 12 genres, portrayal of Events Biography and Adventure are the strongest, while Comedy and Horror are the weakest.
Admiration
Anticipation
Disapproval
Grief
Optimism
Sadness
Aggressivenest eness
Awe
Disgust
Joy
Rage
Surprise
Anger
Contempt
Ecstasy
3.0
Score
2.5
2.0
1.5
1.0
Wan 4B
Humyysnn
Stepsideo
Modsl
Essy Animats 8.1
Capeeto 38
Vchitecs 2.0
On Video
Pyramud
Model
Figure 13: Model performance on Emotions. Among 10 models and 19 emotions, Remorse is best portrayed, while Aggressiveness is the weakest.
25

---

Dialouues
Diviogues With Reaction
25-
15.
10-
65
Ray
Wenn 34B
PBa 2d
Han
Exp anmate 5.2
Caplatee SM
Pyramic
Vontact 2.
07 Viaa
Figure 14: Model performance on Dialogues. Compared Actions and Emotions, models struggle at Dialogues. Within Dialogues, performance drop is seen during multi-turn conversations.
Firelight
Fluorescent
HMI
LED
Moonlight
Strobes
Sunlight
Tungsten
4.0
3.0
2.5
2.0
Masimax
Wan 148
Humpuan
Luma Ray
Adimate 5.1
LTk Viteo
Stopitda
CogVilee 58
Model
Figure 15: Model performances across Lighting Source. Strobes and Sunlight emerge strongest, whereas HMI and Fluorescent are points of weaknesses.
Aeraa
Ground
Low Overhead
Shoulder
Dutch
High
Knee
level
3.5
Score
3.0
2.5
Minimac
Wan IB
Humyaan
Steviide
Lume Ray
Lix Video
Caprideo 5p
Wan
Easy Animate
Model
Figure 16: Model performances across Camera Angles. The Dutch angle poses a common
challenge to current video generative models
26

---

Table 5: Setup Taxonomy Control Nodes and Values
Name
Description
Potential Values Low, High
Contrast
Determines the difference hetween light and dark areas enhance visual impact.
Blur
Introduces softness to parts of the image to guide Tocus or
Gaussian, Radial, Motion
create mood.
Noise
Adds random variations brightness or color, mimicking
Gaussian, Salt Pepper, Poisson
film grain or digital sensor noise.
Film Grain
Emulates the granular texture of traditional fil photogra-
phy for a classic look.
Color Pulette
Delines the overall range harmony of colors the
Open Ser
scene, influencing its mood.
Lines
Directional elements that guide the viewer's is gaze within the
Horizontal, Vertical, Diugonal
Regular Shapes
Structured, geometric forms such us squares, circles, and triangles that add order to the design.
Square, Circle, Triangle
Natural Shapes
Unstructured shapes that naturally emerge in the scene,
Water-like, Cloud-like
without any geometric constraints.
Frame Balance
Refers the distribation of visual weight across the com-
Rule of Thirds, Symmetry, Right Heavy, Left Heavy
position, casuring a harmonious layout.
Positional Accuracy
The absolute position of an object or subject in a scene.
Open Ser
Relative Positioning
relative positioning of an object in relationship
Open Set
other objects in the seene.
Depth
Controls the perception of distance between elements,
Deep, Flat, Limited, Ambiguous
enhancing the three-dimensional feel the scene.
Setting
Defines the scenc happening indoors ouidoors.
NIIEXT
Time of Day
The time of day the scene set
DAY. NIGHT, MORNING, EVENING.
DAWN, DUSK, LATE NIGHT. MIDDAY.
SUNRISE, SUNSET, AFTERNOON
Location
The specilic place or setting the scene.
Open Ser
Negutive Space
Defines if there a lot of empty space.
Positive
Detines how the space is occupied in the environment.
Clean, Cluttered
Mood
The emotional atmosphere or feeling created by the envi- ronment.
Open Set
Scale
The relative size extent of the environment.
Open Set
Style
The artistic or visual style of the backdrop.
Open Set
Background
The part the scene that is behind the main subject and does not need exactly described,
Open Set
Elements
The natural artificial components of the backdrop.
Rain, Snow, Fog, Wind, Thunder, Smoke,
Dust, Ash. Fire
Prop Description
A general deseription of the prop.
Open Ser
Prop Class
prop. The substance(s) the prop made of.
category or type
Open Set
Prop Material
Wood, Glass, Gold, Paper. Plastic
Prop Pattern
The design on the prop.
Grid, Checker, Stripes, Zigzag, Dots,
Bricks, Metal. Hexagons
Prop Utility
The purpose or function the prop. whether it just exists
Decorative, Functional
in the scene or will be used by the subject.
Subject Class
The category or type of the subjects.
Open Ser
Subject Accessories
ftems worn or carried by the subjects that enhance their
Open Ser
appeurance functionality.
Subject Costume
The clothing worn by the subjects, especially for perfor-
Open Ser
mance or to create a specific character.
Subject Hair
The style and uppearance of the subjects hair.
Open Ser
Subject Makeup
Cosmeties applied the subjects' face or body to enhance
Open Ser
alter their appearance.
Subject Pose
The position or stance of the subjects. especially for a
Open Ser
photograph or portrait.
Subject Silhouetie
The outline shape the subjects against light back- ground.
Open Ser
Subject Proportions
The relative size and scale of the subjects' body parts or features.
Open Ser
Text Generation
The process of creating written content displayed
Open Set
the video.
A.2.2 Camera and Lighting
On Lighting Source (Figure 15). Sunlight, Strobes, and Firelight are handled more reliably, while HMI, Fluorescent, and Tungsten lighting show lower performance. As shown Figure [6] Aerial and Knee level camera angles are depicted better, while the Dutch and Shoulder-level angles show lower performance. On Shot Sizes (Figure 17). Medium-Wide and Master shots have stronger performance in comparison to Full and Extreme Close-Up shots.
27

---

Table Events Taxonomy Control Nodes and Values
Name
Description
Potential Values
Standalone Actions
If the action is stand-alone
Open
Interactive Actions
If the action involves subject-subjece or object-subject
Open Set
interaction
Temporal (Actions)
How actions unfold across line.
Atomic, Concureet, Sequential, Causal,
Overlapping, Cyelic, Reverse
Foreground (Actions)
Describes
laking place foreground
Global, Focal
Background (Actions)
Descrihes it the taking place in the background
Uncertainty
The probabilittc nature of the action outcome
Probabilistic, Deterministic, Mixed
Implicit Emotions
Emotions that suggested implied rother than directly
Open
stated.
Explicit Emotions
Emotions that are clearly directly shown or stated
Open
within the scene.
Temporal (Emotions)
How emotions evolve across time.
Atomic, Concurret, Sequential, Overlap- ping. Causal Local, Global, Focal
Foreground (Emotions)
Describes if the emotion is taking place in the foreground
Background (Emotions)
Deseribes emotion is taking place in the buckground
Tyne of Dialogue Delivery
How the dialogue delivered
Dash, Ellipsis, Monalogue
Foreground (Emotions)
Deseribes the dialogue is being spoken in the foreground
Local, Global, Focal
Background (Emotions)
Describes
the dialogue heing spoken back
ground
Change in Environment
Change ol enviromment or occurrences within a shot
Open Set
Story Structure
Key barrative elements shape the progression.
Turning Point, Climax Foreshadowing.
Conflict
Pace
How fast the events are happening in shot happening in shot
Slow, Fast
Regularity
How regularly the
Regular, Imegular
Close-up Establishing
Extreme Close-up
Master
Medium-Full
Medium Wide
Medium
Medi um-Clese-Up
Wide
4.0
3.5
Score
2.0
Midima
Wan 140
Haryaan
Easy animune 5.1
Win
CouVidioo
Model
Figure 17: Model performances across Camera Shot Size. Models perform well on Master and
Establishing shots and struggle at medium- and extreme-close-up shots.
A.2.3 Setup
For the Setup taxonomy, we also analyze performance at the value level. In Balance (Figure 18). models handle rule of thirds framing more effectively but struggle with symmetrical compositions. For Time of Day (Figure 19), among 11 categories, Sunrise and Morning are portrayed well, while Afternoon remains challenging.
28

---

Right Heavy
Rule of Thirds
Symmetry
3.-
3.0-
2.5
2.0
1.0-
Humyaan
Sepiideo
Luma 2
Van 18
Easy Animato 5.1
Capiidto
Video
Wen
Figure 18: Between different frame compositions, models are better at Rule of Thirds but struggle at maintaining Symmetry.
Attemoon
Day
Evening
Midday Morning
Night Sunrise
Sunset
Dawn
Dusk
Late Night
4.0
Score
3.0
2.0
Alnmax
van 148
Luma 2
Hanyuan
Wan 18
SteyVidteo
Copparee 58
Easy Animate 5.1
DTX Vidto
Model
Figure 19; Across Time of Day setups, Sunrise shots handled better, while Afternoon remains more challenging for models.
29

---

Taxonomy
Seed Prompts
Instruction
SCINE
(Events)
Scripts
Juir
LLM J
Taxonomy
SCINE Scripts
Instruction
SCINE
CCamera/l Biapierr//ightinggiee Setup
Visuals
2
Categories
Taxonomy
Prompt
Instruction
Questions
Figure 20: 1. Prompt Generation Pipeline SCINE Scripts are created by passing seed prompts and sampled Events taxonomy nodes to an LLM, forming the narrative component of our benchmark. SCINE Visuals are then generated through structured upsampling, where nodes from the Camera, Lighting, and Setup taxonomies are sampled and injected into each SCINE Script to create prompts that capture visual exposition. 2. Automatic Categorization and Question Generation Given SCINE prompt and taxonomy, we categorize each taxonomy element present in the prompt and generate a corresponding question enable isolated evaluation of each control node.
Table 7: A working example of prompt with its corresponding categories and questions. Each question targets a single control node from the taxonomy, enabling human annotators to perform fine-grained, independent evaluations node.
Prompt
Final Category
Question
In stark white laboratory illuminated by cool LEDs casting clinical precision,
Camera Creative Intent - Shot Size
Does the generated video clearly exhibit well-executed close-up up shot that captures the subject with the intended intimacy and detail?
scientist carefully drops a single blue chemical into beuker, the cameru fruming an intimate close-up as soft depth of hield blurs sterile environment behind. A back light carves a subtle halo around the glassware moments before the liquid erppts bright green, intensified by strategic neon- moon-tinted color gel that makes the reaction glow like bottled lightning.
Camera Intrinsics + Depth Field
Does the video effectively showease soft depth of field that isolates the subject while smoothly blurring the hack ground?? effect artificial LED source clearly visible and does emulate the clinical, cool lighting effect as descrihed in the scene? Does the video convey a cool color temperature in lighting setup that reinforces the clinical precision suggested the prompt?
Lighting + Sources - Artiffcial/ructicals Light
Lighting + Color Temperature
Lighting Lighting Position
back lighting effect evident in the video, such that it effectively carves hulo or outline around subject as descrihed? Does the video incorporate acon-tinted color gel effect that intensifies the lighting during the chemical reaction as detailed led in the prompt?
Lighting Advanced Controls Color Gels
A.3 Details on Prompt Generation
The overall evaluation pipeline, depicting prompt generation, categorization and question generation, is presented in Figure 20
Table 7 presents example of prompt and the corresponding categories and generated questions. Below, we show an example of the instruction given to an LLM to upsample SCINE-Script into SCINE-Visuale by incorporating control nodes from the Camera and Lighting taxonomies.
30

---

SCINE Scripts Cinematographer
System Prompt
You world-class cinematographer known for your visionary storytelling, mastery of light, and camera. You have decades of experience working on award-winning films across genres, collaborating with top directors and production teams. Your insights blend technical expertise with artistic sensibility. When describing scenes or advising visual storytelling, you cinematic terminology with clarity and inspiration. Think like Roger Deakins, Emmanuel Lubezki, and Greig Fraser-your visual choices always elevate the emotional tone and narrative are project.
User Prompt
GOAL
You will given a prompt and taxonomies that define camera and lighting controls commonly used by cinematic professionals. Your objective is to enrich the given prompt by sampling relevant nodes from both the taxonomies. cinematographer, your role is to "shoot" this scene using the best possible cinematic expression, utilizing camera and lighting control options provided in the taxonomy.
As
PROMPT: [prompt}
MOST IMPORTANT INFORMATION
1. Only Use Nodes from the Provided Taxonomies You must never introduce nodes that are not present in the given taxonomies. While the values within each node can be flexible-allowing for creativity and imagination grounded in your professional experience. For example, the node "Color Gel" is defined, but has no values. It is upto you to define these values. The structure must strictly adhere the nodes defined in the taxonomy. Think expansively within the bounds of each node, but never go beyond them.
2. Preserve the Original Prompt Content Do NOT remove add of the original content from the input prompt. Your only task is enrich the prompt by layering camera and lighting related information. prompt must remain entirely intact.
core semantics narrative the
3. Do NOT include the path through which you sample the nodes in the prompt. That is, do NOT add the paths from the taxonomy using '->".
GUIDELINES 1. Input Prompt The input prompt describes a single continuous event, intended to occur within one uninterrupted shot. Therefore, do not include any cuts multiple camera setups. Assume this is one-shot sequence.
2. Each node in the taxonomies contains: Description: definition of what the node represents.
Example: An example of how the node may appear in a prompt.
Values: A non-exhaustive list of possible values for the node. Some notation: a. predefined and
OPEN SET Indicates the node supports a wide range of possible values.
b. Indicates the node may have multiple values, which should selected based on your reasoning and cinematic knowledge. 3. Enriched Prompt Your enriched version will serve as input to text-to-video model. It must be fluent, natural, and interpretable by the model, while incorporating cinematic elements effectively.
CAMERA TAXONOMY The Camera Taxonomy defines elements related to the camera's intrinsics, extrinsics, and its cinematic use (camera_taxonome
LIGHTING TAXONOMY The Lighting taxonomy broadly defines all elements of lighting, including source, position lighting, along with effects such as shadows and reflections, along with color temperature, lighting motion such as flickering etc : (lighting_taxonm
When incorporating lighting into your enriched prompt, remember that cinematographer can shape the look and feel of shot selectively illuminating different depth planes scene. Lighting can be applied to foreground, mid-ground, background, and the subject itself-either individually or in combination. Your choices should support the emotional tone, visual focus, and narrative intent of the shot.
Below, we show an example the instruction given to an LLM to categorize and generate evaluation questions for input prompt using the Camera taxonomy.
31

---

Camera Categorization and Question Generation
GOAL
You expert prompt evaluator Your task is analyze video generation prompt and categorize it based on predefined taxonomy.
PROMPT: (prompt)
Available Categories (with Examples)
The category presented to you that of Camera. The Camera taxonomy broadly defines everything related to the camera the intrinsics, the extrinsics and the cinematic use of camera. (camera_taxonon
Notes about the Taxonomy
Each node in the taxonomy contains
1. Description Definition of what that node represents.
2. Example An example of the presence of node in the form of prompt.
non-exhaustive list of values of these nodes. Values are a list values that this node can have. Some nomenclature
3. Values
a. OPEN SET indicates that this node contains a large number of values.
b. [] indicates that this node may have multiple values, but are not defined explicitly and it is upto your reasoning and knowledge.
Examples of Categorization
1. Static Mediun-Close-Up of David's face showing quiet devastation. Quick Push In tears well up in his eyes. Shot with a medium ISO to capture the dim apartment lighting.
Static Camera -> Trajectory -> Camera Movement -> Static Push Camera Trajectory -> Camera Movement 3D Sedium-Closs up Camera Creative Intent -> Shot Size Medium ISO Camera -> Intrinsics -> Exposure -> ISO
2. Wide shot of bustling city street night. The lights of the shops and restaurants cast a colorful glow on wet pavement. People walk by, their faces illuminated by the bright signs. The camera pans up to reveal the towering skyscrapers that loom overhead, their windows reflecting the city lights.
Wide shot Camera Creative Intent Shot Size
Pans Up Camera -> Trajectory -> Camera Movement -> 20
TASK
Analyze the given prompt and return following structured output in a valid JSON
format:
Words: Extract important keywords or key phrases from the prompt using the following guidance:
Identify named entities related to a camera in professional use as you would in NER (Named Entity Recognition).
Extract noun phrases or descriptive terms that relate to camera.
Prefer multi-word expressions where meaningful related to camera. Avoid generic or uninformative words "a", "video", "the", etc.
Categories: For each word phrase, assign the most appropriate category from the taxonomy. A dictionary of relevant categories from the taxonomy.
For each relevant category, assign
between '8' and '1" representing how
strongly the prompt matches category.
Provide a reason for each score, referring to the words or phrases extracted and how they relate to the category.
Generate question that helps human evaluator determine whether this category is generated video. Use your reasoning to guide the question. The
visually present
32

---

Table 8: Lexical Diversity of SCINE Scripts. Compared to existing prompt-based based benchmarks, SCINE-Scripte demonstrate higher lexical diversity across multiple metrics.
Benchmark
TTR f
Distinct Bi-Grams 1
Jaccard Distance 1
VBench [25
0.1489
0.4605
0.9384
MovieGenBench
0.1660
0.5311
0.9285
EvalCrafter [35]
0.2270
0.6038
0.9413
T2V-CompBench [48
0.1435
0.4781
0.9350
SCINE Scripts
0.1760
0.6177
0.9445
evaluator will this question rate the video scale from (not all) to (strongly represented).
The generated question should evaluate quality, consistency and presence the node in the video.
Important Guidelines:
The camera information should be explicitly mentioned assume or derive anything. Only consider word or a phrase a match, it is explicitly mentioned in the prompt.
prompt. Do NOT imply,
Each prompt can have multiple nodes of the Camera taxonomy. You should capture all nodes the prompt and map it back the taxonomy.
You must always traverse from the root node, which is Camera in this case. That is, the 'category' should always start as (Camera -> ..)
- You will never create node that is not in the taxonomy. These nodes can have multiple values, previously explained and you the values.
expected to be imaginative about the nodes, should always come from the given taxonomy.
Since the taxonomy is of Camera, we do not care about objects, subjects, lighting, events, actions or emotions. Your sole focus should be about camera terms that are present in the prompt in accordance with the taxonomy. You will NOT ask any question related to objects, subjects, lighting, events, actions or emotions.
Table 8 compares SCINE-Scripts with existing prompt-based video generation benchmarks. We compute token level metrics: Type-Token Ratio (TTR), Distinct Bi-Grams, and average pairwise Jaccard Distance, and find that SCINE-Scripts exhibits strong lexical diversity.
A.4 Distribution of Taxonomy Categories in SCINE Prompts
Figures 21 22 and 23 show the distribution of activated nodes in SCINE Visuals, aggregated the node level, across the roles of Cinematographer, Production Designer, and Director, respectively. As shown, our prompts cover broad distribution of nodes across all the taxonomies.
A.5
Annotation Details
shows the annotation interface used by human annotators during evaluation. We also present the distribution of annotators' years of experience in film production in Figure 25 While annotations for cinematic controls can be subjective, especially given the large number of control nodes, we try our best to mitigate this by providing clear rating guidelines annotators for each control node. Table presents a minimal example of the rating guidelines shared with the annotators.
Figure 24
33

---

Arieve brggete banvary te
Ailiote bequesy Sweres Nutsattist
(a) Level Activations
(b) Level Activations
Aalvstian
chees Haadoy canas accebee
Settu luet
(c) Level 3 Activations
(d) Level 4 Activations
Figure 21: Node activations in Camera and Lighting taxonomies for the Cinematographer role in SCINE Visuals.
Actustion Frepancs Baaeary Nadis
Acoustice Irggancy looramy Nade lovel
Level Activations
(b) Level Activations
Achvelen requanyy el Scoree Rode Level
Jchvolea Deguancy ol Sccreee Ride clve!
Badiow
lahe
bike
Gre Core comerr cone
Level Activations
(d) Level Activations
Figure 22: Node activations in Setup taxonomy for the Production Designer role in SCINE Visuals.
34

---

Smart Narevce Contee
(a) Level Activations
(b) Level 2 Activations
Stmp into
Gever th
Tam
Erees
Level Activations
(d) Level 4 Activations
Figure 23: Node activations in All taxonomies for the Director role in SCINE Visuals.
Video A
Prompt
ciyy rosttap right, munic people dance under stoing quents stand the railing, behind
Events Types Change in Environment
Does the video clearly depict pulsating music as a dynamic environmental element that contributes to the overall atmosphere?
(wnlect 1
Cselect
Types Actions
effectively show people dancing lively, coordinated that capture the intended energy of the scene?
())
())lect
Figure 24: User Interface used by annotators to perform evaluations.
35

---

Number of Annotators
25
15
5
5-8
9-12
13-16
17-20
21-24
Years Experience
Figure 25: Distribution of the years of film production experience amongst human annotators in evaluation setup.
Table Examples rating guidelines provided human annotators for different control nodes, across all taxonomies.
Dimension
Score
What Look For
Camera is at or above eye level, not low angle at all.
Slight upward tilt, but still feels neutral.
Low Angle
Below subject, mild upward view, light impact.
Clear low angle, subject looks larger imposing,
Strong low angle, subject dominates, towering presence.
action is present. Actions are isolated unrelated. Timing is off, they start or end awkwardly.
No action
Overlapping Actions
Some overlap, but hard to follow. Fluid overlap, actions feel natural and dynamic together.
Light clearly comes from front or side, no rim light or background separation.
Back Light Position
Some cdge lighting, but not consistent or strong, subject may still blend into background.
Back light is partially visible, outline is hinted but not clear full subject.
Back light clearly present, rim light separates subject from back- ground.
Strong back light effect, glowing edges around hair or shoulders. Sub- ject clearly pops against the background. Perfect match.
Composition is clearly asymmetrical.
Some repeating elements, but no visual mirror.
Symmetrical Frame Balance
Partial symmetry or mirrored clutter that's 's not clean.
4 Almost perfect symmetry, small inconsistencies exist. Clear and precise symmetry, mirrored subjects, reflections, centered framing. Strong and intentional.
36

---

A.6 Statistical tests
Pairwise t-tests show that the vast majority of model comparisons in our human evaluation, across all laxonomies are statistically significant at the 5% level (p <0.05). Figure 26 28 presents the t-test results of Events, Lighting and Camera, and Sctup, respectively.
How masit oatter than Courme mocel -.065)
Copa 20
medel tren medd 0.952
model sere than modelte
Evsy Anirate 1
Nut cxtitiiaal Mapitouu 1.00
Humwen
Haryaan
UX Video
ITX Vire
lume
Luma Ray
Mirire
Nea 2.
Ster
Steptieee
VO
Won 18
Pyramu Aloo
Eas Anteaaee Su
Hemouar
shupideo
Vice udd
Firamal Howe
Lanoe Besca
phayiden
poreent 2.0
Coppites
tanna
Bay Are
the
(a) Pairwise I-tests across models on emotions
(b) Pairwise t-fests across models on dialogues
Row modal is bettr then Column model (p + 0.351
CogVideo 50
model than Colum model 0.05)
Easy Animate 5.1
saatbiially signiticant (o
Hunyyan
ITX Video
Luma Ray
6-6
Mirmax
en
Mcchi
2331
17822
StapVideo
10
19:00
111
VChibect
9.4 10:0
0.1
148
1200
2359)
10
Wan 18
Pyremid Flow
Cogridde
Easy Arimate 5.
Humpaas
Video
Luma Ray
Menimar
Machi
Pike 2.
voweect
Wan 140
(e) Pairwise t-tests across models actions
Figure 26: Statistical comparison matrices for Events: Emotions, Dialogue, and Actions.
37

---

Row ruslt is bete than coken mvdt la 0.331
Susy
Not san siailly signticart la suss
Fow mosi la than noda
Fow modd beber than modd 0.351
mesid O
Easy EazyAnmatie mate
slatsttuu atpiituumi 0.251
UX deg
lama Ray
tuma Ray
M M4
Stagt dey-
Stepl dec
Wan 240
Wan 140
Wan 38
Copeldte
Exsy Asinase
Kay
Stapsitte
Van AB
mil.
Oaginder 18
Ammele 9
Hamynan
Meomar
Beenddes
We SxB
Lama
(a) Pairwisc t-tests across models Color Tempera-
ture
(b) Pairwise t-tests across models on Lighting Effects
Piis eoste better than Eok are mati (p- sos; for star oiin dayalli tp e9t
Fow model butter than Colurie madil < 3.651
Easy
stat stiaallyy saye toreet 251
Hungann
ITX des
sama Ray
Lunra Ray
M Miiima
Meimax
Stopl/
SoadWides
Wen 140
Wan
Cagisden SB
Easy Asimate
Humwitsn
UiA Mileo
Dhopliate
Capridis 38
Gasy Ammreto 3.3
Hempaan
Lok Vilao
Lama Rey
Repiole
Lano
(c) Pairwise t-tests across models Extrinsies
(d) Pairwise t-fests across models Intrinsics
Row model bettor than Column model (p 0
Now midelt than model t.0%)
Nr. www.sttteeeeeeeoooooo cagricantii 0,051
Huryuer
UX Video
Mirimus
Slaphides
414
Won 18
Copiibo 19
Easy Asimute $1
Hampaat
Li bate
Luma Ray
Stapiitt
Wan 148
(e) Pairwise t-tesss across models Trajectory
Figure 27: Statistical comparison matrices for Camera and Lighting
38

---

Hew made is soter than Colare musil sc1,90
mudd botter Cokem medd
Now racke it wone one mocitip R-00
Kow moudd in ware than ća use model (o B000)
sy skgotoott in
Easy Artnala
Kir stan aay dkyalicar In 0:33
Hanysan
Mice
VTXvidee
Lime
Lana May
Nin max
Nelmac
Sue
Suathideg
Whn 163
Wan 140
Na11 15
Cognsdie SR
Eany Asiimate
ute video
Loma Key
Neiina
Steiridee
Wen A4B
Copgssiee St
Easy Adimnate
vider
Nilima at
Stopedee
Won 14b
Lama
(a) Pairwise (-tests across models Subject Class
(b) Pairwise t-fests across models on Subjet Costume
now node beterr then Oolume mode 3.051
Row noce is verea than Colurn rocul ip 6:331 Not mttiec y sig aopiiiaant up- i 6G)
modil Calame modeltp
Evey Salmans
Not stas stici ripnificanc lp 6:35)
Huryynn
UX
UX deu-
Lurn Ray
z4me Bay
M mirmo
Minmax
Stugh/ dox-
Stape deu-
War. 148
Wan 542
Wan 28
Wan
Captibee 19
Eae mniatt
Kzy
Hoiima
Stephidee
Wam An
Copioder 3t
Easy Animate 8.
Hempate
LDe adiee
Lomn Ray
Shantter
(c) Pairwise t-tests across models on Elements
(d) Pairwise I-tests across models on Time of Day
Fowr buttar than Dalumm modd O
Now eddee wiree than ese model ip ic 601)
Eayy Inimate 5.1
Not sta ca signtcaattpp 0.65)
Manyuan
Lumr Ray 2-
Wirinne
Replicee 11911
War 148
Capissse Mp
Homplen
Vddeo
Mesmms
Stapidde
Bay Arimate
Lama
Wan
(c) Pairwise t-tests across models on Location
Figure 28: Statistical comparison matrices for Setup
39

---

Table 10: Top 10 nodes where the VLM shows the strongest performance. Score indicates the
accuracy of the VLM's choice with the most common preference among human annotators.
Node
Score
0.88
Eietts--AdiiCCooooo----yyt------
0.82
0,82
0.80
0.79
0.78
Lighting-+Lighting
0.76
Seeel--Tpps--AActooo------------ Rel.
0.75
0.75
Events Events->Types -nm--yppe---mtioo-----E- -Eppoiioonn-xxxp-pp---EEEEeeeee
0.75
Table 11: Bottom 10 nodes where the VLM shows the weakest performance. Score indicates the accuracy of the VLM's choice with the most common preference among human annotators.
Node
Score
Setup Suup--Subjects
0.33
Setu==SScene-SSe
0.33
Evetts-Types--Accio--Pooyyyy as Coctootttettull-Baaaaaooooo
0.46
Seunp=SSubiccs--Acccssiiiss
0.53
0.55
Ssung=Seenn-Teectuu---CCooorrr
Palette
0.55
Cumes=-aarrisse--Exssuu-----hhre Speed
0.57
Seup--ASeen--TTexunnBBBBBr Gels
Lightiig--Addv.CCooto---Crrr
0.57
0.57
Lighting->Color Temperature
0.58
A.7 Additional VLM results
Node-specific results of VLM evaluator Table and 11 list the set of nodes on which our VLM
evaluator has the strongest and weakest performance, respectively.
Comparison with Closed Source Models We extend our validation to closed-source, flagship SOTA models. Specifically, we evaluate two recent models from the Gemini family with distinct purposes: Gemini-2.0-Flassh optimized for fast inference, and Siaiii-2---PPo--reveew--- complex reasoning. We use the same human-aligned preference accuracy metric as with open- source models. Due to the lack of public details on model sizes, we cannot draw conclusions about scaling effects. However, Gemini-2.5-F VL-2552.-.-- across all categories. Notably, as shown in Figure 29] our 7B model outperforms Gemini-Flash across all categories and performs competitively with 2e5ini-2.5-Pro.. This highlights the strength and scalability of our approach for professional video evaluation.
optimized for
consistently outperforms open- source models, including
should produce consistent scores when given the same video, prompt, and focus aspect. In this analysis, we evaluate the raw scores generated by VLMs rather than preference rankings, and measure their stability under Best-of-5 sampling. Since VLMs are probabilistic, we evaluate reliability via the standard deviation of scores across runs. We use temperature=0 to sample to make ensure that the highest probability is selected at each sampling step. We exclude our model from this analysis, as its architecture includes a dedicated value head, unlike zero-shot VLMs that produce rewards as text. Our results show that Qwen-2.5VL-33B exhibits high variance, making it unreliable under repeated sampling. In contrast, the flagship models and the strongest open-source model, QwenVI.-2..5722) demonstrate high reliability, with consistently low variance (Table (2).
Reliability in VLMs A reliable VLM-as-a-Juee
40

---

Zero-Shot Model's Preference Accuracy
100
Oward 3-t 220
Gener 2.0 Rack
(70)
180
60
12ta
Events
Scdup
Camera
Lighiin
Catogory
Figure 29: Preference Accuracy of open and closed-sourced VLMs in rating videos generated for Professional Use
Table 12: Measuring VLM Reliability across best-of-5 sampling
Model
Standard-Deviation + Krippendorff-alpha
Owen2.5-VL-33B
2.34
0.36
Qwen2.5-VL-7B
0.37
0.84
Qwen2.5-VL-32B
0,47
0.65
Qwen2.5-VL-72B
0.23
0.95
Gemini2.5-Flash
0.20
0.90
Gemini2.5-Pro
0.14
0.95
Table 13: Average scores across taxonomy categories recently released video generative models.
Model
Camera
Events
Lighting
Setup
Overall Avg.
Veo 3 Fast
3.686
3.382
3.974
4.078
3.780
Wan 2.2 14B
3.584
3.060
4.009
4.114
3.692
Wan 2.1 14B
3.239
2.821
3.827
3.975
3,466
Wan 2.2 5B
3.194
2.705
3,838
3.913
3,412
Wan 2.1 IB
3.240
2.579
3.698
3.742
3.315
A.8
Additional Results on Recent Models
We also perform small-scale evaluations recently released models, specifically Veo (Fast) 9 and the Wan 2.2 family of models. Results are presented in Table 13
A.9 Broader Impact
We hope SCINE encourages the generative Al and computer vision communities to engage more deeply with the elements required to produce a professional cinematic shot. While our taxonomy is currently used for evaluation, also offers structured foundation for broader tasks such captioning, creating control aware video datasets, and guiding model training toward explicit cinematic intent. We also envision that our fine-grat fine-grained control nodes drive new directions in open computer vision problems, such as estimating camera intrinsics from video, inferring lighting position, and understanding frame compositionality. In conclusion, systematically understanding cinematic controls generative models, and we are hopeful it will be meaningfully adopted and extended across both generative AI and filmmaking communities.
taxonomy
first step toward
41


Reviews:
[{"review_id": "oDnPu8jbkr", "reviewer": "Reviewer_FGjZ", "summary": "The paper introduces Stable Cinemetrics (SCINE), an evaluation framework designed to assess the readiness of current video generative models for professional use. SCINE organizes filmmaking principles into four hierarchical taxonomies—Setup, Events, Lighting, and Camera—covering 76 fine-grained control nodes based on industry practices. It creates a benchmark with story-driven and visually rich prompts to mirror professional workflows, enabling automated prompt categorization and targeted question generation for detailed evaluation. A human study involving 10+ models, 20K videos, and 80+ film professionals reveals significant performance gaps, particularly in Events and Camera controls. To support scalable evaluation, the authors developed a vision-language model (VLM) aligned with expert annotations, outperforming existing baselines with a 72.36% accuracy.", "questions": "1. Given that SCINE's current evaluation operates at the shot level, how might the framework be extended to evaluate multi-shot sequences? \n\n2. What are the most common failure modes of the VLM evaluator? Would its computational cost be a burden?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. Existing benchmarks lack the detailed, shot-level structure and cinematic depth required for professional filmmaking, as exemplified by a simple VBench prompt (\"A man is walking\") that omits crucial details like character appearance or camera movement. In contrast, SCINE's taxonomy-guided approach enables scalable and future-proof evaluation, adapting as model capabilities evolve beyond the limitations of static, fixed-prompt benchmarks.\n\n2. The study involves over 80 film professionals, including independent cinematographers, screenwriters, and an Academy Award-winning Visual Effects Artist. The taxonomy development is driven by the central question: \"What controls do professionals require when setting up a shot?\" ensuring a practical, industry-relevant framework.\n\n3. The taxonomy provides both coarse insights, revealing that models struggle most with Events and Camera, and fine-grained comparisons, such as better performance on shot size over camera framing and natural lighting over artificial lighting.\n\n4. Detailed information is provided, including Taxonomy Details, the User Interface used by annotators for evaluations, and instructions given to the LLM for prompt generation, enhancing the transparency and reproducibility of the methodology.\n\n### Weaknesses\n1. Not enough evaluated generated videos with evaluation results are presented. The supplemental material contains some ranked videos, but it is not organized in a way that is easy for readers to interpret. Side-by-side comparisons of different T2V models using the same prompts would significantly improve clarity.\n\n2.  Certain metrics, such as camera parameters, movement, depth, and contrast, are not subjected to rigorous, objective quantitative assessments. Incorporating precise numerical analysis for camera movement, depth, and contrast, etc. alongside human ratings would create a more balanced and reliable evaluation.", "was_revised": true, "final_justification": "In the author's rebuttal, the pilot study on supplementing our human and VLM‑based assessments with quantitative, objective-driven metrics should improve this manuscript. Other additional information and discussion provided are also helpful. After reading the rebuttal and other reviewers' opinions, my concerns are addressed. Therefore, I am increasing my rating accordingly. I am also looking forward to seeing the additional visual results, which should be available to ACs when they are making their decisions.", "rebuttal": "We thank the reviewer for their constructive feedback of our work. Please find below our responses to your questions: \n\n---\n> Quantitative Metrics\n\nWe appreciate the reviewer’s suggestion to supplement our human and VLM‑based assessments with quantitative, objective driven metrics. \n\nBelow we first explain why such metrics remain challenging to be directly applied to state‑of‑the‑art video‑generation systems : \n\n* **Out‑of‑distribution content** — Monocular depth and camera‑pose estimators are trained and evaluated almost exclusively on real‑world data (e.g. **KITTI**, **RealEstate10K**). Their behaviour on *synthetic* videos has not been systematically benchmarked.\n* **Poor alignment with human annotators** — Widely‑used automated video metrics (e.g. **FVD**) have been shown to poorly correlate with human judgments \\[1], prompting a shift towards **human** and **VLM‑assisted** evaluations \\[2,3].\n                                                                            \nTo unify evaluations of all of our 76 control nodes, and because professionals are the ultimate judge of the video's quality, we treat professional human raters as the gold standard and train a VLM proxy to approximate them.\n\nIn response to the reviewer’s suggestion, we carried out baseline experiments that demonstrate how geometry‑based computer‑vision metrics can serve as an additional, objective signal for future evaluations : \n\n**Camera Pose and Depth Estimation**\n\nWe generate two 81‑frame videos (Wan 1 B vs. Wan 13 B) for the below prompt : \n\n*\"A figure walks along the beach shore at sunset as the sun hangs low, ... The camera ***pushes in smoothly on a dolly*** from an eye-level perspective, ... drama in sharp relief.\"*\n\nUsing CUT3R [4] we recover per‑frame camera parameters and depth maps, then derive the baseline metrics below (ideal dolly path = best‑fit straight line through the 81 camera centres) : \n\n|Metric|Wan 1 B |Wan 13 B |Interpretation |\n|- |- |- |- |\n|Path RMS / max (m) |0.022 / 0.042 |0.046 / 0.126|\\~2 × straighter path for *1 B*. |\n|Total push (m) |1.705 |0.596|*1 B* travels further. |\n|Linearity (1 = perfect)|0.965 |0.846|Smoother progression for *1 B*.|\n|Speed μ ± σ (m / f)|0.026 ± 0.018 |0.024 ± 0.014|Similar mean speed; *13 B* slightly steadier.|\n|Jerk (max) (m / f²)|0.103 |0.056|Larger instantaneous spike in *1 B*. |\n|Drift roll / pitch / yaw (°) |2.3 / 4.3 / 2.6 |2.0 / 15.0 / 1.1 |*13 B* tilts up ≈ 15 °, deviating from a true dolly. |\n|Depth median μ ± σ (m) |3.42 ± 0.32 |4.02 ± 0.96|*13 B* generates a deeper scene. |\n|Depth IQR (m)|0.84|2.74 |Narrower depth spread in *1 B*.\n\n**Insight**: Wan 1 B adheres more closely to the intended dolly motion and maintains a tighter depth distribution, whereas 13 B introduces significant pitch drift.\n\n**Global and Temporal Contrast**\n\nAcross 9 models, we compute standard contrast metrics to 50 videos per model, each video generated from prompts activating the `Setup → Scene → Texture → Contrast` branch of our taxonomy.-\n\n|Model|Mean RMS|Patch (32x32) Mean|Patch Variance|Dynamic Mean RMS|Dynamic Max RMS\n| - | -: | -: | -: | -: | -: |\n|CogVideo5B|55.64|21.50|357.44|0.64|4.64|\n|Easyanimate5.1|60.83|32.65|469.87|0.76|3.86|\n|Hunyuan|58.81|15.43|267.16|0.21|1.92|\n|LTXvideo|50.39|22.02|264.17|0.06|0.30|\n|LumaRay2|50.86|16.88|234.12|0.28|2.00|\n|Minimax|55.35|18.04|297.49|0.19|1.18|\n|Wan14B|56.89|21.38|373.43|0.15|1.11|\n|Wan1B|60.10|25.30|410.62|0.24|1.63|\n\nIn summary, our current work benchmarks model capabilities through expert human evaluations, which remain the gold standard, and trains a vision–language model (VLM) to approximate them. Additionally, our initial experiments reveal potential directions for grounding geometry-based metrics in the evaluation of video generative models.\n\n**\\[1]** Ge Songwei et al., \"On the Content Bias in Fréchet Video Distance\"\n\n**\\[2]** Wu, Jay Zhangjie, et al. \"Towards a better metric for text-to-video generation.\"\n\n**\\[3]** Bansal, Hritik, et al. \"VideoPhy 2: Challenging Action-Centric Physical Commonsense Evaluation of Video Generation\"\n\n**\\[4]** Wang, Qianqian et al., \"CUT3R: Continuous 3D Perception Model with Persistent State\"\n\n---\n\n> Extension to multi-shot sequences\n\nEvaluating multi-shot sequences are an important future direction. Multi-shot setups will add new dimensionalities to our taxonomy, where two axes become relevant:\n\n1. **Intra-shot correctness** : the per‑shot controls defined and evaluated in our current work;\n2. **Inter-shot consistency** : coherency of shots across sequences.\n\nOur taxonomy already provides the foundation and structure for evaluating both. In a multi-shot setup, each node remains relevant while additional factors such as **consistency** will link identical nodes across shots. **Consistency** of nodes such as **character**, **style**, **story**, and **color** become important in a multi-shot professional filmmaking setup; all of which are already defined in our taxonomies : \n\n* Attributes such as **Costume** and **Hair**, associated with character identity, are under the `Setup → Subjects` axis.\n* **Style** and **Color** are captured under `Set Design` and `Scene Texture`, respectively.\n\nMultiple baseline approaches can then be adopted to measure inter-shot consistency, across different control elements:\n\n* **Facial consistency**: calculated using facial embedding similarity [1]\ncomputed with models like RetinaFace \\[2] and FaceNet \\[3].\n* **Costume consistency**: measured using visual feature similarity methods\nsuch as DINO embeddings as shown in \\[4].\n* **Hair consistency**: evaluated using perceptual and semantic metrics\n  e.g., SSIM, and CLIP-I, as shown in \\[5].\n* **Color consistency**: evaluated using sliced Wasserstein distances \\[6] or histogram comparisons \\[7] in the CIELAB color space. Discrete color palettes can also be compared using palette-based photo recoloring and color transfer methods \\[8].\n* **Style consistency**: measured using the Contextual Style Descriptor (CSD) \\[9], which evaluates style similarity between reference and target sets.\n\n**In conclusion**,  our current work concentrates on defining a structure of controls in a single shot setting; an understanding of which is still missing from the generative video literature. By formalising this space with our structured taxonomy, we lay a solid foundation for future research on multi‑shot filmmaking; our framework can be extended with additional dimensions such as \"consistency\", that are crucial when moving beyond single shots.\n\n\\[1] Wang, Mengyu, et al. \"CharaConsist: Fine-Grained Consistent Character Generation.\"\n\n\\[2] Serengil, Sefik et al. \"A Benchmark of Facial Recognition Pipelines and Co-Usability Performances of Modules\"\n\n\\[3] Schroff, Florian, et al. \"Facenet: A unified embedding for face recognition and clustering.\"\n\n\\[4] He, Zijian, et al. \"VTON 360: High-fidelity virtual try-on from any viewing direction.\"\n\n\\[5] Zhang, Yuxuan, et al. \"Stable-hair: Real-world hair transfer via diffusion model.\"\n\n\\[6] He, Jiaqi, et al. \"Multiscale sliced wasserstein distances as perceptual color difference measures.\"\n\n\\[7] Liu, Yihao, et al. \"Temporally consistent video colorization with deep feature propagation and self-regularization learning.\"\n\n\\[8] Chang, Huiwen, et al. \"Palette-based photo recoloring.\"\n\n\\[9] Somepalli, Gowthami, et al. \"Measuring style similarity in diffusion models.\"\n\n---\n> Failure modes of VLM Evaluator\n\nPlease find the nodes on which the VLM has the strongest and weakest performance, respectively. `Score` indicate the accuracy of the VLM's choice with most common preference among human annotators. \n\n**Top 10 Nodes**\n\n|Node|Score|\n|-|-|\n|Setup→Scene→Set Design→Environment→Mood|0.88|\n|Events→Adv.Controls→Rhythm→Pace|0.82|\n|Setup→Scene→Set Design→Environment→Style|0.82|\n|Setup→Scene→Set Design→Props→Utility|0.80|\n|Events→Types→Emotions→Exp.Types→Explicit|0.79|\n|Setup→Scene→Geometry→Frame→Shapes→Regular|0.78|\n|Lighting→Lighting Effects→Shadows→Soft|0.76|\n|Setup→Scene→Geometry→Space→Spatial Loc.→Rel.Pos.|0.75|\n|Events→Types→Actions→Int.Types→Standalone|0.75|\n|Events→Types→Emotions→Exp.Types→Explicit|0.75|\n\n**Bottom 10 Nodes**\n\n|Node|Score|\n|-|-|\n|Setup→Subjects→Makeup|0.33|\n|Setup→Scene→Set Design→Environment→Background|0.33|\n|Events→Types→Actions→Portrayed as→Contextual→Background|0.46|\n|Setup→Subjects→Accessories|0.53|\n|Lighting→Lighting Effects→Reflection|0.55|\n|Setup→Scene→Texture→Color Palette|0.55|\n|Camera→Intrinsics→Exposure→Shutter Speed|0.57|\n|Lighting→Adv.Controls→Color Gels|0.57|\n|Setup→Scene→Texture→Blur|0.57|\n|Lighting→Color Temperature|0.58|\n\n---\n> Computational cost of VLM\n\nOur VLM, trained as a Bradley-Terry style evaluator starting from Qwen2.5-VL-7B adds a regression head in place of a LM head. It needs just one forward pass instead of  `seq_len` passes for a traditional autoregressive generative model. This makes our model roughly 100 ×  than the 72 B baseline, while also offering better performance (72.36 % vs 62.50 %) since it's trained for the objective to work like a preference model.\n\n|Model|Avg FLOPs (in petaFLOPs)|Accuracy|\n|---|----|---|\n|Qwen-7B|17.738|59.86 %|\n|Qwen2.5‑VL‑32B|81.087|59.93 %|\n|Qwen2.5‑VL‑72B|182.44|62.50 %|\n|**Ours**|**1.82**|**72.36 %**|\n\n---\n> Additional Visual Results\n\nWe apologize for this. We were initially restricted by the 100 MB NeurIPS supplementary size limit; the size of each video is quite large, for example a video generated by Wan 14B, is on average ~7 MB.\n\nThus, as requested by the reviewer, we have now created a fully anonymous web-page with 200+ videos, presenting side-by-side comparisons across models on SCINE Scripts and Visuals, and detailed pairwise comparisons with categories, questions, and scores.  \n\nHowever, we are unable to share this link with the reviewer, due to the NeurIPS rebuttal policy. Therefore, we have shared this fully anonymized link with the AC; we hope you are able to consider the web-page during your internal discussions. Thank you!"}, {"review_id": "lZqYp5FFFs", "reviewer": "Reviewer_r8DJ", "summary": "This paper introduces Stable Cinemetrics (SCINE), a structured evaluation framework for professional video generation that organizes filmmaking principles into four hierarchical taxonomies (Setup, Events, Lighting, and Camera) with 76 fine-grained control nodes. The authors conduct extensive human evaluation with 80+ film professionals assessing 20,000 videos generated by 10+ text-to-video models, revealing that current models struggle most with Events and Camera controls while performing better on Setup and Lighting elements. To enable scalable evaluation, they fine-tune a vision-language model that achieves 72.36% alignment with human judgments, though its testing on same-distribution data raises questions about generalization capabilities. Despite limitations including the absence of visual examples and limited cross-domain testing, SCINE provides a valuable foundation for aligning generative video models with professional production standards.", "questions": "1. Could additional experiments be conducted to verify the generalization ability of the automatic evaluator on videos generated by other models or on real-world videos?\n2. The paper does not explicitly state whether the data and models will be open-sourced. Given that this is an evaluation benchmark, open-sourcing is crucial for community reproduction and further research. Do the authors have plans for open-sourcing?\n3. Some videos labeled as 'good cases' in the supplementary materials (e.g., ltx_00569.mp4) clearly do not conform to real-world principles, which raises questions about the rigor of the evaluation criteria. Could the authors provide a more detailed explanation of the judgment criteria for these cases, or consider re-evaluating them?", "limitations": "See Weakness above", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Pioneering Professional Video Evaluation Benchmark: Introduces SCINE, an innovative evaluation framework that formalizes filmmaking principles into four disentangled taxonomies (Setup, Event, Lighting, Camera), addressing a critical gap in current T2V model evaluation for professional use.\n2. Large-Scale Expert Human Annotation: Conducts a comprehensive human study with over 80 film professionals annotating 20K videos, providing high-quality, industry-aligned ground truth for model performance.\n\nWeaknesses:\n\n1. Questionable Generalization of Automatic Evaluator: The trained VLM evaluator's generalization is unproven, as it's not validated on real-world videos or new, unseen generative models, which is crucial for a benchmark's universality.\n2. Quality Issues in Supplementary Examples: \"Good case\" examples in the supplementary material (e.g., ltx_00569.mp4) are unrealistic, undermining the claimed alignment with professional standards and raising further doubts about the evaluation method's generalization.\n3. Lack of Open-Source Commitment: The absence of a clear commitment to open-source the proposed data and models significantly hinders reproducibility and wider community adoption of the proposed dataset, thereby diminishing the paper's overall practical impact.", "was_revised": true, "final_justification": "Most of my concerns are solved.", "rebuttal": "We thank the reviewer for their constructive feedback of our work. Please find below our responses to your questions: \n\n---\n> Generalization to unseen videos\n\nThank you for your suggestion. We have now evaluated our VLM‑based evaluator on previously unseen data from Veo 3 Fast, a model released *after* our submission.\n\nWe sample 27 Veo 3 Fast Videos (restricted due to high rate limits), evaluating a total of 100 questions, ensuring equal distribution across categories. We follow a similar setting to our previous experiments and use human annotation as the ground truth. Below we present VLM accuracy, on *unseen* Veo 3 Fast videos : \n\n| Model         |    Events |    Setup |   Camera |    Lighting |   Overall |\n| :------------ | ----: | ----: | ----: | ----: | ----: |\n| Qwen2.5-VL-7B       |  53.6 |  42.5 |  51.0 |  60.0 |  51.8 |\n| Qwen2.5-VL-32B       |  55.4 |  55.4 |  72.0 |  65.0 |  62.0 |\n| Qwen2.5-VL-72B       |  64.3 |  60.0 |  82.0 |  74.6 |  70.2 |\n| **Ours**      |  79.5 |  78.9 |  78.9 |  74.9 |  78.1 |\n\nAs shown above, our model *still* outperforms existing baselines and generalizes well to out-of-distribution data not seen during training.\n\n---\n> Open-Source commitment\n\nWe are fully committed to open-sourcing the data and models associated with our work. Post legal approvals and additional safety checks, we will release all the artifacts associated with Stable Cinemetrics.\n\n---\n> Quality issues in Supplementary \n\nThank you for this question. Our evaluation framework assesses **each cinematic dimension independently** using our 4 taxonomies, **Camera**, **Lighting**, **Setup**, and **Events**.  And a generated video can be `good` in one aspect, while being `poor` in another.\n\nThe presented example, `ltx_00569.mp4` is considered a `good` example **only** for `Camera Movement`, which is expected to be `static` (as mentioned in the prompt : *A sun-dappled camping ground framed by towering pine trees....The camera **static** on a tripod, shallow depth of field isolating....mirroring the vertical trunks rising behind them* | full prompt in `metadata.csv`), in this case. \n\n| Taxonomy Node | Evaluation Question | Annotator&nbsp;1 | Annotator&nbsp;2 | Annotator&nbsp;3 | **Avg.** |\n|--------------|---------------------|-------------|-------------|-------------|----------|\n| **Camera Movement (Static)** | Is the camera maintained in a fixed, static position throughout the scene as specified in the prompt? | 5 | 5 | 4 | **4.67** |\n\nAs shown above by the annotator ratings, and as indeed seen in the generated video (present in `Camera__Trajectory__Camera Movement folder` in the `good_examples` folder of the `Supplementary`), the video correctly depicts a video generated with a `static` camera movement. This is the *only* reason the video is listed as a `good` example, but only for `Camera Movement`.\n\nLike you correctly pointed out, the generated video has issues related to the movement of the axe and its corresponding effects. These are evaluated and captured under the `Events` taxonomy, in this case. Below, are the evaluation scores : \n\n| Taxonomy Node        | Evaluation Question                                                                                                                     | Annotator&nbsp;1 | Annotator&nbsp;2 | Annotator&nbsp;3 | **Avg.** |\n|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------|-------------|-------------|-------------|----------|\n| **Events (Actions)** | Does the video clearly display the physical action of gripping the axe and the dynamic motion of the swing as described? | 1           | 2           | 3           | **2.0**  |\n| **Events (Causal)**  |Is the causal sequence from the axe swing leading to the wood splitting, splinters erupting, and the halves falling depicted clearly and coherently in the video? | 1           | 3           | 3           | **2.33** |\n\nThese low scores align with your observation that the physical realism of the axe swing is inadequate.\n\nIn conclusion, \n\n* `ltx_00569.mp4` is **only** a positive example for **Camera‑Movement (Static)**, in this case.\n* Under **Events**, the same video would be a **negative example**, and is treated as such during evaluation.\n\n---\n> Concerns about visual examples\n\nWe apologize for this. We were initially restricted by the 100 MB NeurIPS supplementary size limit; the size of each video is quite large, for example a video generated by Wan 14B, is on average ~7 MB.\n\nThus, as requested by the reviewer, we have now created a fully anonymous web-page with 200+ videos, presenting side-by-side comparisons across models on SCINE Scripts and Visuals, and detailed pairwise comparisons with categories, questions, and scores.  \n\nHowever, we are unable to share this link with the reviewer, due to the NeurIPS rebuttal policy. Therefore, we have shared this fully anonymized link with the AC; we hope you are able to consider the web-page during your internal discussions. Thank you!"}, {"review_id": "cWebpgAUsx", "reviewer": "Reviewer_hh2a", "summary": "The paper introduces Stable Cinemetrics (SCINE), a structured evaluation framework based on professional filmmaking principles and standards. SCINE includes 4 hierarchical taxonomies (Setup, Event, Lighting, and Camera) to evaluate text-to-video (T2V) models. This paper evaluates 10+ models across 20K generated 60 videos with feedback from 80+ professionals.", "questions": "see weaknesses", "limitations": "The authors adequately addressed the limitations and potential negative societal impact.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Quality and Clarity: This paper, espically its taxonomies, is clearly written and technically sound.\n2. This paper formally introduces professional filmmaking standards into the T2V evaluation process, a aspect that has been previously explored (e.g., in Wan) but not systematically assessed. The proposed taxonomies and evaluation framework, based on professional feedback, are original contributions.\n\nWeaknesses:\nThe scope of this paper is relatively limited to the following aspects:\n1. Evaluation is limited to controllability (L106): The evaluation taxonomies only evaluate controllability but neglect the overall video quality. There remain fundamental flaws in the generated outputs, such as distortions in human bodies or hands, object movements that violate physical laws, and poor imagery quality. These aspects are critical for professional video generation but are largely overlooked in this paper.\n2. Evaluation is limited to an independent shot: filmmaking involves more than generating a single independent shot; it requires character consistency across different scenes. Character identity is nearly impossible to control through text alone, meaning that aspects such as character consistency are also overlooked in the evaluation. In application, this issue may be even more important than the aspects evaluated in the paper.\n3. Evaluation is limited to text-to-video: this paper only evaluates the text-to-video (T2V) aspect because T2V is more suitable to be evaluated by the proposed taxonomies (L73). It remains unclear whether T2V models are the primary workflow used in professional settings. This questions the significance of this work.\n4. I2V models are also widely adopted by the community due to their greater flexibility in control through reference images. Can the evaluation framework proposed be adapted to evaluate I2V model and pipelines?", "was_revised": true, "final_justification": "I sincerely appreciate the rebuttal and the effort during the rebuttal period. The new clarification and experimental results has addressed most of my concern, and I choose to maintain my current recommendation of 4: Borderline accept.", "rebuttal": "We thank the reviewer for their constructive feedback of our work. Please find our responses to your questions:\n\n---\n> Clarification on Controllability and Visual Quality.\n\nThank you for raising this point. We considered this extensively as well. It's important to clarify that cinematic controls represent the \"input\" dimensions specific knobs or dials that professionals prioritize, while video quality represents the \"output\" fidelity of the model. Correctness in elements like human anatomy and physical laws is inherently assumed in real-world professional videos. Our initial evaluations indicated a strong correlation between control metrics and video quality. Specifically, we observed that human annotators *implicitly* factor in visual quality while rating for controls. For example, if a video shows an \"interactive\" action, but the \"interaction\" is not grounded in physical laws, humans inherently tend to rate the video low. To validate this empirically, we performed the following experiment, at scale : \n1. Augmented each SCINE Scripts prompt with extra questions evaluating **visual quality (VQ)** and **temporal smoothness (TS)**.\n   * Example prompt:\n     > \"In an airport during the day, passengers form a line, checking tickets as they prepare to board. Outside the large window, ground crew load suitcases onto a baggage cart, while another airplane taxis slowly across the tarmac in the distance.\"\n   * VQ question: \"How smooth and artifact‑free is the rendering of the airplane taxiing in the distance?\"\n   * TS question: \"Is the motion of the ground crew loading suitcases and the taxiing airplane fluid and natural, with consistent transitions between frames?\"\n2. Collected independent ratings from annotators on all control, plus VQ and TS questions.\n3. Computed per‑model averages and compared Control vs. VQ and TS, with results shown below : \n\n|Model|Ctrl|VQ|TS|Δ(VQ−Ctrl)|Δ(TS−Ctrl)|\n|--|-:|-:|-:|-:|-:|\n|Wan 13B|3.158|3.319|3.113|+0.160|–0.045|\n|Minimax|3.110|3.437|3.181|+0.327|+0.071|\n|Hunyuan|2.973|3.138|2.900|+0.165|–0.074|\n|Luma Ray 2|2.897|3.282|2.980|+0.385|+0.083|\n|Wan 1B|2.877|2.974|2.659|+0.097|–0.218|\n|Pika 2.2|2.759|2.976|2.690|+0.217|–0.069|\n|Mochi|2.704|2.708|2.396|+0.004|–0.307|\n|StepVideo|2.610|3.078|2.788|+0.467|+0.178|\n|EasyAnimate 5.1|2.573|2.748|2.474|+0.175|–0.099|\n|Cogvideo 5B|2.566|2.653|2.380|+0.087|–0.186|\n|Pyramid Flow|2.339|2.528|2.234|+0.189|–0.105|\n|Vchitect 2.0|2.144|2.089|1.930|–0.055|–0.215|\n|LTX Video|2.056|2.271|2.045|+0.215|–0.011|\n\nThis denotes a Spearman correlation of 0.93 (p < 1e-5), confirming that VQ and TS are implicitly embedded within control evaluations. \n\nAlthough video quality and controllability are fundamentally different problems, our findings demonstrate that our primary metric for controllability is an effective proxy for overall model performance, as it implicitly factors in the visual quality of a generated video. However, as models evolve and video quality approaches a plateau, we anticipate that fine-grained controllability will increasingly become the decisive differentiator for professional applications. Our focus on controllability is thus a forward looking evaluation of this critical frontier.\n\n--- \n> Limitation to Single Shot Evaluation\n\nWe agree that filmmaking involves more than generating a single independent shot. That said, our current work specifically focuses on a **single** shot control because even a **single** shot involves coordinating multiple elements to translate a creative vision to screen.\n\nWe identify a critical gap in current literature: despite the growing capabilities of generative video models, a principled understanding of what constitutes a well-controlled **single** shot in a professional setting, remains underexplored. Our contribution lies in filling this gap by proposing taxonomies that define and evaluate these elements in a structured way. Further, we believe that if we cannot reliably control a **single** shot using video generative models, building coherent sequences or shots becomes an even greater challenge. As our evaluations show, current models struggle in setting up a **single** shot.\n\nWe agree that extending to multi-shot setups is an important future direction. Multi-shot setups will add new dimensionalities to our taxonomy, where two axes become relevant:\n\n1. **Intra-shot correctness**\n2. **Inter-shot consistency**\n\nOur taxonomy already provides the foundation and structure for evaluating both. In a multi-shot setup, each node remains relevant while additional factors such as **consistency** will link identical nodes across shots. \n\n**Consistency** of nodes such as **character**, **style**, **story**, and **color** become important in a multi-shot professional filmmaking setup; all of which are already defined in our taxonomies : \n\n* Attributes such as **Costume** and **Hair**, associated with **Character** identity, are under the `Setup → Subjects` axis.\n* **Style** and **Color** are captured under `Set Design` and `Scene Texture`, respectively.\n\nMultiple baseline approaches can then be explored to measure inter-shot consistency, across different controls:\n\n* **Facial consistency**: via facial‐embedding similarity (RetinaFace, FaceNet) [1–3]\n* **Costume consistency**: via DINO feature similarity [4]\n* **Hair consistency**: via perceptual (SSIM, PSNR) + semantic (CLIP-I) metrics [5]\n* **Color consistency**: via sliced Wasserstein/histogram in CIELAB space + palette recoloring and transfer methods [6–8]\n* **Style consistency**: via Contextual Style Descriptor (CSD) [9]\n\n**In conclusion**,  our current work concentrates on defining a structure of controls in a single shot setting; an understanding of which is still missing from the generative video literature. By formalising this space with our structured taxonomy, we lay a solid foundation for future research on multi‑shot filmmaking; our framework can be extended with additional dimensions such as \"consistency\", that are crucial when moving beyond single shots.\n\n[1] Wang et al. \"CharaConsist: Fine-Grained Consistent Character Generation.\"\n\n[2] Serengil et al. \"A Benchmark of Facial Recognition Pipelines and Co-Usability Performances of Modules\"\n\n[3] Schroff et al. \"Facenet: A unified embedding for face recognition and clustering.\"\n\n[4] He et al. \"VTON 360: High-fidelity virtual try-on from any viewing direction.\"\n\n[5] Zhang et al. \"Stable-hair: Real-world hair transfer via diffusion model.\"\n\n[6] He et al. \"Multiscale sliced wasserstein distances as perceptual color difference measures.\"\n\n[7] Liu et al. \"Temporally consistent video colorization with deep feature propagation and self-regularization learning.\"\n\n[8] Chang et al. \"Palette-based photo recoloring.\"\n\n[9] Somepalli et al. \"Measuring style similarity in diffusion models.\"\n\n--- \n> T2V and its scope in professional applications\n\nWe use text as our input modality as it is the most common source of control across video models today. Further, a single text prompt can combine coarse ideas (\"dolly zoom at dusk\") with exact parameters (35 mm, 4000K). T2V models enables us to cover the breadth of the entire taxonomy in a single, unified workflow. Furthermore, our control taxonomy is input‑agnostic; it defines what a creator needs to steer, regardless of how those controls are fed in.\n\nWhile T2V models are not the only tools in a professional workflow, they have largely been integrated into creative pipelines. A recent report titled, \"Darren Aronofsky Launches AI-Driven Storytelling Venture, As Google Unveils New Gen AI Video Tool\" - states that T2V models have been widely used by highly creative professionals. Furthemore, the \"AI in Content Creation 2025\" report by Wondercraft, states that the rise of T2V workflows is opening new doors; platforms that allow users to start with a written script and quickly generate narrated videos or animated explainers are in high demand. A recent film, titled \"We Tested Google Veo and Runway to Create This AI Film\" by the Wall Street Journal describes using text prompts to create specific shots. Short films titled, \"THE CLEANER\" and \"SWAT\", have been created entirely by using text prompts as input. \n\nIn conclusion, T2V models allow complete traversal of our control taxonomy and align with professional pipelines. Evaluating through this lens therefore covers the taxonomy’s full scope without constraining future extensions to other input modalities.\n\n--- \n> Extensions to I2V\n\nYes, our evaluation framework can be extended to I2V pipelines. Below, we propose and perform a simple experiment to validate the same.\n\n1. Reference Image: Since reference images have widely been used to control `character` appearance, we base our current experiment on them. \n2. Build Prompts for respective models : \n* T2I prompt – contains only the character description, conditioned on (`Setup -> Subjects`) from our taxonomy. e.g. \"Male punk with a neon-green mohawk (`Subjects -> Hair`), nose ring, tattoos (`Subjects -> Accessories`), black tank-top (`Subjects -> Costume`), arms crossed.\"\n* I2V prompt – Adds more controls from our taxonomies. \"A punk man leans forward (`Events`) while filmed from a static low-angle shot (`Camera`) under warm light (`Lighting`).\"\n3. Evaluate : \n* Control: \"Is the clip low-angle while the punk leans forward under warm light?\"\n* Character Consistency: \"Do the mohawk, nose ring and tattoos remain identical to the reference image?\"\n\nWe create 20 prompts and generate reference images with Stable Diffusion 3.5 Large and generate videos with two I2V models - **Hunyuan** and **Wan-13B** - and rate the models on consistency and control. We present their win rates : \n\n|Metric|Hunyuan|Wan|Tie|\n|-|-|-|-|\n|Consistency|3|13|4|\n|Control|4|11|5|\n\nAs shown, our taxonomy and evaluation framework are inherently flexible and can be naturally adapted to additional modalities.\n\n---\n> Space below Figure 4.\n\nThank you for the suggestion; we will improve our presentation in the camera-ready version."}, {"review_id": "yefI3i919P", "reviewer": "Reviewer_H89U", "summary": "This article introduces SCINE (Stable Cinemetrics) - an evaluation suite specifically designed to assess the quality of video generation from the perspective of professional video production. The evaluation is divided into four major categories: Setup, Event, Lighting, and Camera, each of which contains numerous subcategories, totaling 76 subcategories. Each subcategory was discussed and determined by professional video creators, making it more professional compared to other video evaluation benchmarks.", "questions": "See Weaknesses.", "limitations": "Although a relatively detailed classification is constructed, the prompts for each category do not reflect professionalism and uniqueness. Comparing the professionalism claimed by the author in the main text with the examples in the supplementary materials, there is a suspicion of overclaim.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**  \n1. The evaluation system was established through a relatively professional process, which is a task with a high barrier to entry.\n\n**Weaknesses:**  \n1. Although the article claims that the construction process of SCINE is highly professional, the data provided in the supplementary materials suggests otherwise. First, the `ranked_videos` folder contains two subfolders, `good_examples` and `bad_examples`, but the videos in these folders are not paired, and their prompts are not the same, which contradicts the claims made in the paper. Additionally, the evaluation prompts constructed are not professional. For instance, in the `camera trajectory` category of prompts in the supplementary materials, most prompts describe a static camera, which is overly simplistic for assessing camera movement.  \n2. The supplementary materials do not provide the scores for the human annotation videos, making it impossible to determine the accuracy of the scoring.  \n3. The training process of the evaluation model is simple and entirely based on existing work.  \n4. The main text does not include any visualized presentations of videos, nor does it use visualized examples to explain the necessity of the 76 categories.", "was_revised": true, "final_justification": "Weak Accept.", "rebuttal": "We thank the reviewer for their constructive feedback of our work. Please find below our responses to your questions: \n\n---\n> Professional Nature of Prompts\n\nThe videos and prompts shown in the *Camera Trajectory* section of the supplementary material were selected as illustrative **positive/negative examples**, not as a representative sample of the full prompt set. To clarify, we report below the actual distribution of values from all prompts labeled `cinematographer` in our `SCINE_VISUALS.json`.\n\n| Taxonomy Node | Distribution (%) |\n| -| -|\n| **Camera Movement** | Push In (33.38), Tracking (20.79), Static (11.90), Zoom In (6.70), Dolly Zoom (7.52), Crane (4.79), Camera Roll (4.79), Arc (3.01), Tilt Down (1.92), Pan Right (1.50), Trucking (1.50), Zoom Out (1.23), Tilt Up (0.96) |\n| **Camera Angle**    | Low (30.67), Dutch (21.24), Eye-level (16.96), Ground (11.25), Shoulder (6.02), High (4.12), Hip (3.01), Overhead (2.46), Knee (2.30), Aerial (1.98)|\n| **Camera Shot Size**       | Medium-close-up (28.69), Wide (20.47), Close-up (17.45), Medium-Full (13.34), Medium (12.93), Establishing (3.61), Extreme Close-up (2.51), Full (0.50), Medium-wide (0.27), Master (0.23)|\n\nAs this shows, our prompt set spans a wide range of values, across different nodes. Importantly, these values do not occur in isolation; controls often **co-occur**, e.g., \"push in + low angle\" or \"push in + dutch angle\", yielding a rich and combinatorially diverse testbed for video generative models.\n\nAdditionally, as demonstrated in Figure 4, our prompts closely align with real-world screenplays. Figures 25–27 further show that our prompt set offers broad and diverse coverage across all four taxonomies. These findingss collectively support that our prompts faithfully reflect the professional intent of our taxonomy.\n\n---\n> Additional visual examples \n\nWe apologize for this. We were initially restricted by the 100 MB NeurIPS supplementary size limit; the size of each video is quite large, for example a video generated by Wan 14B, is on average ~7 MB.\n\nThus, as requested by the reviewer, we have now created a fully anonymous web-page with 200+ videos, presenting side-by-side comparisons across models on SCINE Scripts and Visuals, and detailed pairwise comparisons with categories, questions, and scores.  \n\nHowever, we are unable to share this link with the reviewer, due to the NeurIPS rebuttal policy. Therefore, we have shared this fully anonymized link with the AC; we hope you are able to consider the web-page during your internal discussions. Thank you! \n\n---\n> Organization of ranked_videos in Supplementary\n\nThank you for pointing this out. The *ranked\\_videos* directory in the supplementary material was not intended to contain paired examples generated from the same prompts. Rather, it provides representative samples of high- and low-scoring videos across different categories. For example, videos located at\n`good_examples/all_models/Setup__Text Generation` are videos that received high scores in the `Text Generation` node of our taxonomy.\n\nTo eliminate any ambiguity, we have now added both : (i) sets of videos produced from the same prompt and (ii) explicit pairwise comparisons to the anonymous link.\n\n---\n> Scores of Human annotation videos\n\nFor each video  in the Supplementary, we present the individual annotations and the reason of it being a positive or negative case. \n\n|No.|Folder|Taxonomy Node|Video|Annotator 1|Annotator 2|Annotator 3|Avg|Reason\n|-|-|-|-|-|-|-|-|-\n|1 |bad_examples |Camera\\_\\_Trajectory\\_\\_Camera Movement|wan1b_00748|1|2 |1 |1.33|Zoom out movement not depicted|\n|2 |bad_examples |Camera\\_\\_Trajectory\\_\\_Camera Movement |wan14b_00025|1 |2 | 1| 1.33 |Generated video does not feature a push-in|\n|3 |bad_examples |Camera\\_\\_Trajectory\\_\\_Camera Movement|wan14b_00748|1 |2 |2 |1.67|Zoom out movement not depicted|\n|4 |bad_examples |Lighting\\_\\_Sources\\_\\_Artificial\\_Practicals Light|hunyuan_00501|1| 3|1|1.67|Video does not capture the single practical light|\n|5 |bad_examples |Lighting\\_\\_Sources\\_\\_Artificial\\_Practicals Light|stepvideo_00791|1|2| 2|1.67|Video does not clearly incorporate fluorescent lights as an artificial source of illumination|\n|6 |bad_examples |Setup\\_\\_Scene\\_\\_Geometry |easyanimate5.1_00410 |1| 2|2 |1.67|Video does not incorporate strong diagonal lines in its composition.|\n|7 |bad_examples |Setup\\_\\_Scene\\_\\_Geometry |wan1b_00570|2| 1| 3|2 |Diagonal elements like the light trails  are not effectively portrayed in the video|\n|8 |bad_examples |Events\\_\\_Types\\_\\_Actions |pyramidflow_00834|1 | 1|1|1|Video does not show the vines interacting with and gripping the tombstone|\n|9 |bad_examples |Events\\_\\_Types\\_\\_Actions |pyramidflow_01100|1 |1 |1|1|The warriors do not enter the holographic arena|\n|**—**|**—**|**—** |**—**|**—**|**—**|**—**|**—**|**—**|\n|10 |good_examples|Camera\\_\\_Trajectory\\_\\_Camera Movement|ltx_00569| 5|5| 4|4.67|Camera maintains a static position|\n|11|good_examples|Camera\\_\\_Trajectory\\_\\_Camera Movement|ltx_01076| 5|4|5|4.67|Camera maintains a static position|\n|12|good_examples|Camera\\_\\_Trajectory\\_\\_Camera Movement|stepvideo_00452 |5 |5|3|4.33|Slow zoom in movement is depicted in the video|\n|13|good_examples|Camera\\_\\_Trajectory\\_\\_Camera Movement|wan1b_00569 |5 |5 |4 |4.67|Camera maintains a static position.|\n|14|good_examples|Events\\_\\_Types\\_\\_Emotions|easyanimate5.1_00722| 5|4| 5|4.67|Video visually captures the elderly woman's subtle, soft smile.|\n|15|good_examples|Events\\_\\_Types\\_\\_Emotions|wan14b\\_00704|4 | 5|5 |4.67| Explicit sense of urgency through the children’s expressions is depicted|\n|16|good_examples|Events\\_\\_Types\\_\\_Emotions|wan14b\\_01011| 5| 3|5|4.33|Gardener's startled expression visibly conveyed in the video.|\n|17|good_examples|Events\\_\\_Types\\_\\_Emotions|wan14b\\_01131| 5|4 | 5|4.67|Subtle expression of nervousness and wariness is portrayed in the video|\n|18|good_examples|Lighting\\_\\_Lighting Effects\\_\\_Reflection|Ray2\\_00813 | 3| 4|5|4|Long reflections on surfaces is well captured by the video|\n|19|good_examples|Lighting\\_\\_Lighting Effects\\_\\_Reflection|wan14b\\_00055|4 |5 |5 |4.67|Video showcase hard geometric reflections|\n|20|good_examples|Setup\\_\\_Scene\\_\\_Geometry |wan1b_00473 |3 | 5| 5|4.33|Video showcases a structured arrangement with clear horizontal lines.|\n|21|good_examples|Setup\\_\\_Scene\\_\\_Geometry |wan1b_00623|5 | 5|4 |4.67|Vertical and diagonal lines clearly rendered in the composition.|\n|22|good_examples|Setup\\_\\_Scene\\_\\_Geometry |wan1b_01076| 5| 3|5| 4.33 |Subjects are symmetrically framed|\n|23|good_examples|Setup\\_\\_Text Generation|Minimax_00115|5|3 |5|4.33|On-screen text 'The Grand Finale' is rendered well|\n|24|good_examples|Setup\\_\\_Text Generation|Minimax\\_00127 |3 |5 | 4|4|On-screen text 'Guiding Light' is rendered well|\n|25|good_examples|Setup\\_\\_Text Generation|Minimax\\_00481 | 5| 5|5 |5|Neon signage with text 'EAT' is clearly integrated into the scene.|\n\nAdditional annotation scores for more videos across multiple categories are also available at the anonymous webpage link. \n\n---\n> Visual examples justifying Taxonomy Categories\n\nWe apologize for this. The webpage link now contains visual examples justifying the categories introduced in our taxonomy.\n\nAdditionally, the taxonomy is grounded in well-established references from professional filmmaking and cinematography literature:\n\n* *A Shot in the Dark* by Jay Holben offers a deep dive into lighting practice in film. It covers concepts such as light quality (`hard` vs. `soft`), `light position`, the use of `practical lights` on set, and the role of the `exposure` triangle in audience perception.\n* *Film Directing: Shot by Shot* by Steven Katz outlines key visual storytelling principles, including `props`, `makeup`, `shot sizes`, and `camera movements`, which directly map to our `Setup` and `Camera` categories.\n* *Cinematography: Theory and Practice* by Blain Brown discusses visual language elements such as `lines`, `space`, `contrast`, `blur`, and `noise`.\n* *The Visual Story* by Bruce Block explores how `depth`, `shapes` `story structure`, and `composition` contribute to narrative meaning in film-making.\n* *The Tools of Screenwriting* by Howard and Mabley analyzes how `actions`, `dialogues`, and `emotions` are intentionally crafted during the writing process, informing our `Events` taxonomy and its sub-nodes.\n\nIn addition to these references, we also conducted multiple iterative feedback rounds with professionals to validate the taxonomy’s relevance and completeness. We hope these revisions clarify both the motivation behind the taxonomy and the real-world grounding of each of its 76 categories.\n\n\n---\n> Training process of VLM\n\nWhile our training setup is simple, the design goal was not novelty, but practicality and scalability. Human evaluation is costly and does not scale to the thousands of generations required for rigorous benchmarking. Our lightweight VLM enables scalable, high quality evaluation with a single forward pass and outperforms much larger baselines.\n\nSpecifically, we fine-tune Qwen2.5-VL-7B as a Bradley-Terry–style evaluator, replacing the LM head with a regression head. Unlike traditional autoregressive models that require `seq_len` forward passes per evaluation, ours requires only 1, making it roughly 100× more efficient than a 72B model, while also being more accurate.\n\n|Model|Avg FLOPs (in petaFLOPs)|Accuracy|\n|---|----|---|\n|Qwen-7B|17.738|59.86 %|\n|Qwen2.5‑VL‑32B|81.087|59.93 %|\n|Qwen2.5‑VL‑72B|182.44|62.50 %|\n|**Ours**|**1.82**|**72.36 %**|\n\nOur results show that even state of the art models perform poorly when used directly for evaluation, underscoring the need for task specific tuning. Additionally, this design enables downstream applications. For example, our evaluator can be plugged in as a learned critic during video model training, offering a path towards fine-grained, control optimization."}]

Author rebuttal:

Meta-review: This paper introduces Structured Taxonomy and Evaluation for Professional Video Generation. Initially, the reviewers raised concerns about side-by-side comparisons of different T2V models using the same prompts and the generalization of automatic evaluators. However, after the rebuttal, the authors successfully addressed most of these concerns, including clarifications regarding the novelty of the work, and all reviewers provided positive feedback. The AC has carefully reviewed the paper, the reviewer comments, and the rebuttal, and agrees that the paper is well-motivated, clearly written, and supported by thorough experiments. Therefore, the AC recommends acceptance. It is encouraged that the authors incorporate all additional experiments and discussions from the rebuttal into the final version.
Decision: Accept (poster)