Title: Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generation

Professional Controls

Evaluations
Abstract: Recent advances in video generation have enabled high-fidelity video synthesis from user provided prompts. However, existing models and benchmarks fail to capture the complexity and requirements of professional video generation. Towards that goal, we introduce Stable Cinemetrics, a structured evaluation framework that formalizes filmmaking controls into four disentangled, hierarchical taxonomies: Setup, Event, Lighting, and Camera. Together, these taxonomies define 76 fine-grained control nodes grounded in industry practices. Using these taxonomies, we construct a benchmark of prompts aligned with professional use cases and develop an automated pipeline for prompt categorization and question generation, enabling independent evaluation of each control dimension. We conduct a large-scale human study spanning 10+ models and 20K videos, annotated by a pool of 80+ film professionals. Our analysis, both coarse and fine-grained reveal that even the strongest current models exhibit significant gaps, particularly in Events and Camera-related controls. To enable scalable evaluation, we train an automatic evaluator, a vision-language model aligned with expert annotations that outperforms existing zero-shot baselines. SCINE is the first approach to situate professional video generation within the landscape of video generative models, introducing taxonomies centered around cinematic controls and supporting them with structured evaluation pipelines and detailed analyses to guide future research.

Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation

Abstract

Recent advances in video generation have enabled high-fidelity video synthesis from user provided prompts. However, existing models and benchmarks fail to capture the complexity and requirements of professional video generation. Towards that goal, we introduce S table Cine metrics, a structured evaluation framework that formalizes filmmaking controls into four disentangled, hierarchical taxonomies: Setup, Event, Lighting, and Camera . Together, these taxonomies define 76 finegrained control nodes grounded in industry practices. Using these taxonomies, we construct a benchmark of prompts aligned with professional use cases and develop an automated pipeline for prompt categorization and question generation, enabling independent evaluation of each control dimension. We conduct a largescale human study spanning 10+ models and 20K videos, annotated by a pool of 80+ film professionals. Our analysis, both coarse and fine-grained reveal that even the strongest current models exhibit significant gaps, particularly in Events and Camera-related controls. To enable scalable evaluation, we train an automatic evaluator, a vision-language model aligned with expert annotations that outperforms existing zero-shot baselines. SCINE is the first approach to situate professional video generation within the landscape of video generative models, introducing taxonomies centered around cinematic controls and supporting them with structured evaluation pipelines and detailed analyses to guide future research.

1 Introduction

The field of video generative models has made significant progress in recent years [37], drawing substantial interest from both academia and industry. This can be evidenced by the growing number of benchmarks [25, 32, 2, 35], datasets [41, 58], and both open- [3, 56, 14, 65] and closed- [6, 54, 36] source models that have collectively driven the field forward. The foundational nature of these models makes them useful for several downstream tasks, including video editing [26], 3D generation [55] and robotics [69]. This widespread adoption of video generative models underpins the growing assertion that they represent a revolution for professional video generation .

Generative vision offers tremendous potential for media creation, but a fundamental question remains: how can we shift generative video from casual, exploratory synthesis to a medium that supports professional-grade, controllable cinematic outputs? The important distinction between casual and professional generative video lies in the critical gap of cinematic control [7]: while today's models can generate videos of " an astronaut riding a horse ", professional creation necessitates granular control over cinematic elements such as the framing of the shot, position of the key light, and even whether the astronaut smiles before or after the horse gallops away - a truly professional video generation system must put every one of those cinematic choices back in the creator's hands. The need for this exact control over every cinematic element, from the timing of a smile to the quality of

Taxonomy

Prompts

LLM

10+

80+

Camera

Lighting

Events

Setup

Models

Evaluated

Annotators

Videos

Annotations

Wan 14B

1079

1135

1099

1114

1200

Minimax

1057

1075

1067

1084

1150

1021

1052

1001

1048

1100

uenAunH

1028

1022

1018

1013

1050

score

Wan 1B

1012

1030

1027

1020

1000

Step Video

1008

1000

1006

950

950

952

86

980

900

CogVideo 5B

932

932

913

188

850

LTX Video

884

820

855

815

800

口O

Camera

Events

Uighting

Setup

◇+

100

Qwen2.5-VL-328

wen2.5-VL-7B

Models

80

Ours

(%) Aseuny

60

40

Total

Events

Category

Setup

Camera

Lighting

√

light, is the very reason filmmakers shoot multiple takes, selecting only the frames where everything comes together to tell the story most effectively [10].

In this work, we investigate the intersection of video generative models and the nuanced control mechanisms associated with professional video production. Despite rapid advancements in video generative modeling, the field still lacks both a clear definition of essential cinematic controls and standardized evaluation protocols for benchmarking progress at this crucial intersection. To bridge this gap, we present SCINE ( S table Cine metrics) - an evaluation suite specifically designed to characterize this intersection, enabling us to directly address the question: " Are Current Video Generative Models Ready for Professional Use? "

The core focus of SCINE is to develop a taxonomy, at the intersection of generative vision and professional video generation, built on the principles of control and specificity [4, 28, 43]. These principles are important in film-making, as every decision carries cinematic meaning; a low angle conveys power while low-key lighting evokes drama. SCINE captures these principles by organizing control knobs into four cinematic pillars - Setup, Events, Lighting, and Camera - enabling the evaluation of video generative models along the same axes that a professional would. This disentanglement also allows evaluation around personalization : which aligns with the collaborative nature of real productions, in comparison to the monolithic nature of current video generation models. Our taxonomy is hierarchical, branching from coarse cinematic concepts to leaf-level controls that naturally map onto computer vision concepts such as object semantics and scene geometry [49].

Leveraging our taxonomy, we generate two prompt types: story-driven and visual exposition, to mirror professional workflows. Story-driven prompts act as mini-screenplays [17], specifying characters, dialogue, actions, and emotions. We enrich these with visual exposition cues, by sampling control nodes from our taxonomy, emulating the transition from script to shot in filmmaking [46]. Sampling control nodes allows automated (a) prompt categorization: mapping each control element to the taxonomy, and (b) generation of targeted evaluation questions for each element, allowing disentagled evaluation of each cinematic control. The structured nature of our taxonomy supports scalable human annotation: we evaluate 10+ models across 20K generated videos with feedback from 80+ professionals. This enables analysis across control dimensions where we observe substantial variance in performance across taxonomy pillars, even for top-performing models such as WAN-14B and Minimax. Our taxonomy facilitates both coarse insights, showing that models struggle most with

Events and Camera and fine-grained comparisons, such as better performance on shot size over camera framing, and on natural over artificial lighting. To support automatic evaluation of finegrained cinematic controls in generated videos, we train a vision-language model (VLM) that aligns with the large-scale human annotations. Our model outperforms existing baselines, achieving an overall accuracy of 72.36% with human annotators. An overview of SCINE, outlining its contributions is shown in Figure 1.

2 Related Work

Video Generative Models and Evaluations. Video generative models can broadly be classified into two categories based on their input conditioning: image-to-video (I2V) and text-to-video (T2V). While I2V approaches such as Stable Video Diffusion (SVD) [3] have been widely adopted by the community, the focus of our work is to evaluate T2V models for professional use. We choose text as an input modality, since it is an effective and free-form way of describing the controls defined in our taxonomy. The Sora Preview [6] served as a catalyst for a wave of T2V model releases across the closed [54, 11] and open-source communities. State of the art open-source models include Wan [56], HunyuanVideo [14] and StepVideo [12]. Several T2V evaluation benchmarks have also emerged, with VBench [25, 67] gaining broad adoption. VBench evaluates T2V models by developing text prompts that evaluate generated videos across dimensions such as temporal flickering, aesthetic quality and, motion smoothness, while employing automatic metrics. Additional T2V benchmarks include - VideoPhy [2], which evaluates physical plausibility by measuring adherence of generated videos to real-world physics, and T2V-CompBench [48], which studies compositional consistency [23] in video generation. Prior work lacks the shot-level structure and cinematic detail needed for professional control; motivating our taxonomy and benchmark to capture the nuanced elements of industry-standard filmmaking. Existing benchmarks lack cinematic depth; for example, a prompt such as " A man is walking " from VBench-2 [67] omits key details like character appearance or camera movement; all essential to setup a cinematic shot. Existing benchmarks are static, relying on fixed prompt sets that limit extensibility. In contrast, SCINE's taxonomy-guided prompt generation enables future-proof evaluation, allowing prompt complexity to scale with model capabilities.

Structured Video Generation and Shot-Level Control. MovieNet [24] provides large-scale movie annotations including scene boundaries, cinematic styles, and character metadata. Storyboard-driven approaches like VDS [44] and VAST [66] generate structured video via intermediate pose/layout representations. Multi-shot generation has been addressed through systems like VideoGen-of-Thought [68] and MovieAgent [62], which use hierarchical reasoning to plan and synthesize sequences. However, these efforts overlook the fine-grained structure of a single shot which is core unit of cinematic composition. They often emphasize isolated factors, without modeling the full set of interdependent creative controls. In contrast, SCINE introduces a unified taxonomy that captures the complete spectrum of shot-level cinematic elements.

3 Stable Cinemetrics

The subsequent sections detail our proposed taxonomy (Section 3.1), and its underlying design principles. Next, we develop our benchmark comprising of prompts designed for professional use (Section 3.2). Section 3.3 describes how prompt categorization and question generation are performed, enabling large-scale evaluations of video generative models for professional use.

3.1 Taxonomy Design

Our taxonomies are developed in iteration with industry professionals, including organizations established by the Big Five Studios [47], independent cinematographers and screenwriters, and an Academy Award winning Visual Effects

Events

Change in Environment

Advanced Controls

Types

Actions

Emotions

Story Structure

Dialogues

Type of Interaction

Portrayed as

Portrayed as

Type of Expression

Type of Delivery

Portrayed as

Regularity

Pace

Explicit

Implicit

Contextual

Temporal

Contextual

Foreground

Background

Background

Foreground

Background

Foreground

Temporal

Uncertainty

Contextual

Interactive

Standalone

Setup

Class

Text Generation

Subjects

Scene

Texture

Geometry

Hair

Accessories

Costume

Set Design

Makeup

Pose

Personality

Contrast

Film Grain

Noise

Blur

Space

Frame

Props

Environment

Backdrop

Proportions

Silhouette

Lines

Shapes

Balance

Depth

Spatial Locations

Utility

Pattern

Material

Class

Description

Location

Time of Day

Setting

Elements

Background

Style

Scale

Mood

Organization

Natural

Regular

Relative Positioning

Positional Accuracy

Positive

Negative

(a) The Setup taxonomy outlines the visual components within the frame, including subjects, props, and environmental context.

Camera

Creative Intent

Trajectory

Extrinsics

Intrinsics

Depth of Field

Exposure

Angle

Camera Movement

Gear

2D

Aperture

3D

ISO

Framing

Static

(b) The Camera taxonomy defines all controls related to camera configuration during a shot setup.

Lighting

Effects

Lighting Conditions

Color Temperature

Sources

Natural

Artificial / Practicals

Soft

Reflection

Shadows

Position

Advanced Controls

Hard

Color Gels

Motion

(c) The Lighting taxonomy specifies the illumination of shot, through light sources, their properties, and their interaction with the scene.

Artist. The central guiding question in our taxonomy development was: " What controls do professionals require when setting up a shot ? ".

A shot is the atomic unit of filmmaking; an uninterrupted sequence without cuts in which cinematic meaning emerges from the coordination of multiple cinematic choices. The average shot length (ASL) in feature films is 5-10 seconds [8], which closely aligns with the temporal limits of current video generation models. This duration is not a limitation; it is a compact canvas where rich and enough narrative and visual complexity can unfold. A single shot entails numerous controls to convey intent, emotion, and story. This motivates our decision to design the taxonomy at the shot level , where finegrained control is paramount. Control is the key distinction between casual and professional video creation. In casual settings, users often accept model outputs with minimal intervention, delegating the key creative decisions to the model. In contrast, professional use demands precise, deliberate control at every stage of the generative process. In fact, pixel-level control is common in film-making [29, 61], underscoring the importance of fine-grained adjustments in achieving the desired visual effect. Professionals such as cinematographers and directors are primarily responsible for defining a shot's creative intent. While filmmaking is collaborative, these roles offer a practical abstraction for modeling control. A key insight is that, despite overlap, they are sufficiently disentangled to support distinct control dimensions: screenwriters rarely specify lighting or camera movement, and production designers are typically not associated with emotional tone or narrative pacing. These factors motivate the development of our 4 control pillars, each contributing to the composition of a shot. Our taxonomies are structured as hierarchical trees, where leaf nodes correspond to the most granular control parameters, each associated with a set of values. We describe each pillar below:

Setup . Setup (Figure 2a) encompasses all visible elements within the frame. We organize it into three top-level groups: (1) Scene aggregates environmental controls, including Texture , Geometry , and Set Design . Texture covers aspects such as contrast and color palette , which govern the surface feel of the shot. Geometry captures dominant shapes and the spatial arrangement of elements within the scene. Set Design comprises Props and their attributes ; the Backdrop , which establishes the macro context of the set; and Environment , which defines micro-level elements contributing to the "feel" of the shot. (2) Subjects refer to the focal characters within a shot, defined by attributes such as costumes and accessories . (3) Text Generation refers to on-screen typography such as titles or lettering, designed to appear as integrated graphical elements. Each node in Setup has cinematic meaning: a dawn ( Time of Day ) setup combined with mist ( Elements ) can suggest danger, while a clean ( Organization ), symmetrical hallway ( Balance ) conveys order.

Lighting . ' Lighting is the key to turning amateur footage into professional stories and presentation ' -Jay Holben [21]. Motivated by this principle, we define the following groups for Lighting (Figure 2b): (1) Source , the origin of illumination within the shot; (2) Color Temperature , which controls the warmth of the light; (3) Lighting Conditions , preset configurations describing scene-wide illumination; (4) Effects , visual outcomes resulting from light interacting with the scene; (5) Position , the spatial relation of the light source to the subject; and (6) Advanced Controls such as flickering modulation and the use of color gels to adjust lighting hue. Each control knob corresponds to distinct cinematic expressions: a shot with only a back light ( Position ) evokes mystery, while hard shadows are often used to amplify tension.

Camera . The camera taxonomy (Figure 2c) encompasses all camera-related control dimensions involved in a shot. We organize these into 4 high-level groups: (1) Intrinsics : optical and exposure parameters governing the light captured by the camera; (2) Extrinsics : position and orientation of the camera relative to the subject; (3) Trajectory : motion of the camera and the supporting gear that enables it; and (4) Creative Intent : compositional choices that shape the narrative or emotional tone of a shot. Prior works have primarily focused on camera motion control [19, 22]; however, we show that a much broader range of camera parameters can be independently manipulated while setting up a shot. Each parameter has tangible cinematic impact; for example, a shallow depth of field can isolate the subject from the background to direct emotional focus while, an insert framing spotlights narrative details with precision.

Events . Events (Figure 3) encodes the narrative substance of a shot, namely the depicted actions , emotions , and dialogues - which are further decomposed into dependent nodes for fine-grained control. These dependent nodes represent attributes that cannot exist independently of their parent categories; for instance, these nodes can specify the type of interaction or the delivery mode of a dialogue. Emotions may appear explicitly (visible tears) or implicitly (a clenched jaw), while actions can be stand-alone or interactive . The Portrayed As category captures aspects such as: Temporal , which refers to the unfolding pattern of the event (e.g., laughter erupting simultaneously vs. sequentially ), and Contextual , which indicates whether the event occurs in the foreground or background . Advanced Controls refines pacing and the story structure of the shot, such as a turning point or climax. While recent works [59, 16] evaluate T2V models on sequential event generation, we show that, from a professional standpoint, Events encompass a much broader and richer evaluative space.

The taxonomies define a total of 76 leaf-level controls that can be independently adjusted when crafting a shot. We structure our taxonomies as hierarchical trees to enable disentanglement and multi-level abstraction of cinematic controls. Attributes within each branch are highly correlated, while branches remain independent, ensuring, for example, adjusting Depth of Field does not affect Camera Movement . The tree structure naturally supports multi-level abstraction, aligning with how filmmakers conceptualize scenes, starting from high-level intent and refining toward specific implementation. For example, a directive such as 'set a tense alley at night' can be decomposed

into an EXT Setting , a cool Color Temperature and a Deep Depth of Field . This structure also allows easier scalability; adding a new detail like floating ember sparks, fits cleanly under ( Environment → Elements ) without disrupting the rest of the taxonomy. An alternate structure such as a flat, linear list would not support such extensibility.

Developing the taxonomy is a non-trivial task. This required multiple iterations with experts since professionals interpret and prioritize controls differently. Furthermore, shot creation is a multi-stage process, starting from script-writing to setup design to camera blocking, making unification under a single structured framework challenging. While some taxonomies [40] focus only on filmmaking aspects, they lack structure and are not aligned with generative modeling. Our goal instead was to impose structure, and develope a taxonomy that bridges professional filmmaking and generative video models. Details of each control node and their corresponding values are provided in Appendix A.1.

3.2 Designing Prompts for Professional Use

The taxonomies form the foundation for constructing prompts tailored to professional use. Our core approach in creating prompts involves sampling values from the control nodes and creating prompts that reflect realistic cinematic intent. Mirroring the filmmaking process, we first generate narrative scripts and then inject visual elements into these scripts :

Scripts. These prompts, referred to as SCINEScripts , contain the narrative content of individual shots. We collaborate with a professional screenwriter to create seed prompts that meet strict constraints: a single shot, under 10 seconds, with no reliance on off-screen elements. These seed prompts, along with sampled nodes from the Events taxonomy are provided as input to a LLM, for prompt generation. We use the Events taxonomy for these prompts because it directly encodes narrative beats i.e., what happens in a shot. It covers nodes such as physical

Original Screenplays

SCINE

MovieGenBench

VBench 2.0

dynamics ( actions ) and verbal interactions ( dialogues ), which are crucial to the story conveyed in a shot. To ensure prompt diversity, we vary parameters across multiple LLM invocations, sampling emotions from Plutchik's model [42], alternating actions, dialogue structures, genres, and subject composition. We use LLMs, as prior work [39, 50] have shown their effectiveness in screenplay generation. t-SNE visualization (Figure 4) of SCINE-Scripts embeddings [57] shows substantial overlap with ground-truth screenplays [45], whereas prompts from VBench-2.0 [67] and MovieGenBench [11] exhibit minimal overlap.

Visual Exposition. We refer to these set of prompts as SCINE-Visuals , which enrich SCINE-Scripts with visual elements from the Camera, Lighting, and Setup taxonomies. In contrast to Events , these taxonomies offer fine-grained control over the visual style and composition of a shot. For each base prompt in SCINE-Scripts , we sample values from one or more control nodes and inject them to expand the prompt with structured visual specifications. SCINE-Visuals highlight a key advantage of our taxonomy: structured prompt upsampling. Unlike existing prompt upsampling techniques that delegate all creative decisions to the LLM, our method constrains generation within the taxonomy, enabling more controlled and interpretable prompt expansion. Table 1 provides a breakdown of how SCINE-Scripts is subsequently upsampled via the taxonomy to generate SCINE-Visuals .

3.3 Category and Question Generation

Next, we extract categories and generate questions for each prompt. Categories link prompts back to the taxonomy, allowing fine-grained evaluations across different levels of abstraction; a single prompt can map to multiple categories across different taxonomies. For each category, we generate targeted questions that are shown to human annotators during video evaluations. These questions are specific

Overall

Biography

Comedy

Action

Drama

Crime

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Model

Wan 14B

Minimax

Hunyuan

Wan 1B

StepVideo

CogVideo 5B

LTX Video

(a) Overall results on SCINE Scripts , across genres. Minimax and Wan 14B emerge as the strongest models. The strongest genre is Biography while all models consistently struggle at Comedy.

Camera

Lighting

Setup

Events

overall

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Model

Wan 14B

Minimax

Hunyuan

Wan 1B

StepVideo

CogVideo 5B

LTX Video

in nature and target only a single control node, enabling its isolated evaluation. Unlike high-level prompt adherence questions, which lack fine-grained attribution, our framework supports per-control annotations. A minimal example is shown below :

Prompt: A tight close-up focuses on a fireplace, its embers flickering brightly.

Category : Lighting → Advanced Controls → Motion | Question : Does the scene exhibit dynamic flickering effects in its lighting that align with the description?

Category : Camera → Creative Intent → Shot Size | Question : Does the video include a tight close-up shot that captures the detailed framing?

Additional details are presented in Appendix A.3.

4 Are current Video Generative Models Ready for Professional Use?

We now evaluate state of the art text-to-video (T2V) models against the professional standards defined in our taxonomy pillars. Our analysis reveals both strengths and persistent challenges of current models, offering an overview of how current models align with professional quality expectations.

4.1 Experimental Setup

Prompts. The SCINE benchmark comprises two prompt categories, Scripts and Visuals , each aligned with distinct professional roles (Table 2). The Visuals prompts are created by systematically upsampling Scripts using our taxonomies leading to a total of 2,089 prompts. We categorize prompts by difficulty: we create basic prompts by limiting the number of sampled control nodes, while in advanced prompts, we do not impose any restriction.

Models. Weevaluate 13 state-of-the-art T2V models, both open-source (WAN 1B/14B [56], HunyuanVideo [14], Step Video [12], CogVideoX 5B [65], LTX-Video [18], Pyramid Flow [27], Easy Animate 5.1 [63], Mochi [52]), VChitect-2.0 [15] and closed source (Minimax [38], Luma Ray 2 [30], Pika 2.2 [31]). Our goal is to assess each model's suitability for role-specific professional tasks, like evaluating narrative fidelity from a screenwriter's perspective. Unless otherwise noted, we use default sampling parameters and maintain a consistent seed per prompt, across models for fair comparison.

Human Annotation Setup. To ensure high-quality evaluation, we work with a pool of 84 expert annotators with an average of 6.5 years of experience in film production, across roles such as cinematographers, film editors, screenwriters, visual communication designers, and directors. Annotators were shown a prompt along with two generated video samples. For each prompt, they were presented with taxonomy derived evaluation categories and

corresponding questions. Each video was rated independently on a 1-5 scale, where 1 indicated complete misalignment with the category and 5 indicated a perfect match. Although the evaluation was non-comparative, our UX ablation studies showed that displaying two videos side by side improved annotator calibration, especially when selecting middle range scores. To promote consistency and reduce subjectivity, we developed a comprehensive annotation guide covering each control node in the taxonomy. We collect 3 votes for every video-question pair across 13,457 unique questions, collecting a total of 248,536 pairwise annotations. We observe an intra-class correlation coefficient (ICC) [51] of 80.4% for the 1-5 ratings at the model-pair level, and 95.5% when the models are considered individually. Further, we conduct Wilcoxon signed-rank tests [60] across 45 model pairs where we observe statistically significant preferences in 37 of them, which highlights that the annotators agree on model preferences.

4.2 Results and Analysis

SCINE Scripts We first evaluate models on narrative event generation, i.e. the story dimension of a shot. Figure 5a compares model performance across different genres, focusing on the accuracy and coherence of generated events. Minimax and WAN-14B emerge as the overall top performers, while LTX-Video consistently underperforms. We observe that models generally perform better on the Biography genre, whereas Comedy proves challenging. In Figure 6, we zoom into sub-categories within Events → Types . Minimax leads in nearly all categories, showing the largest margins in Dialogues and Change in Environment , but falls short in Advanced Controls , where WAN-14B outperforms. Further, models are better at stand-alone actions compared to interactive actions and portray implicit emotions better than explicit ones. Within Event Types, Actions exhibit lowest variance across models. Performance on Causal and Sequential events is highly correlated ( ρ = 0 . 94 ), as is performance on Concurrent and Overlapping events ( ρ = 0 . 86 ) across models. Despite variation across models, all show limitations in multiple Event aspects, highlighting opportunities for improvement.

SCINE Visuals Figure 5b demonstrates overall results on SCINE Visuals. Consistent with the pair-wise rankings in Figure 1, WAN-14B and Minimax emerge as top performers across

Actions

Emotions

Dialogues

Change in

Environment

Advanced Controls

2.0

2.5

3.0

3.5

Event Types

Standalone

Actions

Interactive Actions

Implicit

Emotions

Explicit

Emotions

2.0

2.5

3.0

3.5

Interaction / Expression Types

Minimax

Wan 14B

Hunyuan

Wan 1B

Pika 2.2

Mochi

all pillars. We find that current models struggle most with Events and Camera , while elements of Setup and Lighting are comparatively easier to capture. Only the top three models- WAN-14B, WAN-1B, and Minimax - reliably depict Events , with a substantial performance gap from the rest. While Camera scores are low across the board, the narrow spread suggests that all models face similar limitations. Lighting shows the most consistent performance, with most models achieving relatively high scores whereas Setup yields the highest absolute scores for the top-performing models.

Cinematographer. We evaluate this role by creating prompts that inject control nodes from the Camera and Lighting taxonomy. Within Camera , Extrinsics and Trajectory have the lowest average performance and the narrowest intermodel spread. For Lighting , the primary bottleneck is Lighting Position . We further present results, split by prompt difficulty in Figure 7. Across all models, performance degrades on advanced prompts, indicating that under conditions resembling professional workflows where cinematographers have a large amount of control, current models struggle. The biggest perfor-

Intrinsics

Extrinsics

Trajectory

Creative Intent

Color

Temperature

Source

Lighting

Conditions

Effects

Position

Advanced

Controls

2.4

2.7

3.0

3.3

3.6

3.9

Basic Prompts

Intrinsics

Extrinsics

Trajectory

Creative Intent

Color

Temperature

Source

Lighting

Conditions

Effects

Position

Advanced

Controls

2.4

2.7

3.0

3.3

3.6

3.9

Advanced Prompts

Wan 14B

Minimax

Wan 1B

StepVideo

Hunyuan

CogVideo 5B

mance drops occur in Lighting Source , Color Temperature , and Creative Intent . Lighting Position and

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Model

Wan 14B

Minimax

Hunyuan

Wan 1B

StepVideo

CogVideo 5B

LTX Video

Class

Accessories

Costume

Hair

Makeup

Pose

Personality

(a) Fine-grained results on Subjects . Models perform well on hair and accessories, but struggle with personality and makeup.

Class

Material

Pattern

Setting

Time of Day

Location

Organization

Style

Elements

Background

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Props

Backdrop

Environment

Model

Wan 14B

Minimax

Hunyuan

Wan 1B

StepVideo

CogVideo 5B

LTX Video

(b) Fine-grained results on Set Design . Models perform better at Backdrop in comparison to Environment, and struggle most with styling the shot appropriately.

Hunyuan

Minimax

Wan 14B

Wan 1B

Lighting

Hunyuan

Minimax

Wan 14B

Wan 1B

Setup

Hunyuan

Minimax

Wan 14B

Wan 1B

Camera

Director

Standalone

Advanced Controls show the smallest performance drops, but remain the weakest categories overall, highlighting persistent challenges regardless of prompt complexity. Hunyuan and WAN-1B exhibit consistent performance across complexity levels.

Production Designer. Models perform strongest on the Setup pillar; within Setup , models achieve comparable performance on Subject and Scene generation, but show a drop in Text Generation . In Subjects (Figure 8a), model performances largely vary, with the highest scores for Hair and Accessories , and the weakest in Personality and Make-up . In Set Design (Figure 8b), performance trends follow the order: Backdrop > Props > Environment . Within Props , models perform well in Material but struggle at generating intricate Patterns . For Environment , current limitations lie in adhering to a coherent Style and Backgrounds , whereas better performance is seen in organizing Space within a frame.

Director. Prompts targeting this role differ from prior categories as they evaluate models across all taxonomies simultaneously. Model performance declines on average when all controls are defined jointly (Figure 9). The largest performance drop is observed in Camera , followed by Setup and Lighting . Wan 14B is the only model to show improved performance on Director prompts for Lighting and Setup, compared to its Standalone results.

Our evaluation identifies a three-tier hierarchy among current T2V models: Minimax and WAN-14B at the top, followed by Luma Ray 2, Hunyuan, and WAN-1B, with the remaining models forming the third tier. While overall performance varies, most models struggle with the fine-grained elements critical to professional video generation. For example, atomic events are handled reasonably well, but models falter on concurrent and causal events, which demand deeper temporal reasoning. Similarly, high-level cues like lighting conditions are better captured than nuanced aspects like precise light positioning. In summary, even top performing models exhibit substantial room for improvement across all dimensions of our taxonomy. No model achieves consistently strong performance across all aspects of shot composition, underscoring the challenge of aligning generative video models with professional standards. Additional results and analysis are presented in Appendix A.2.

5 Scalable Evaluation of Professional Videos

In the previous section, we evaluated video generative models using expert annotations across 76 control nodes defined by our taxonomy. While human evaluation remains the gold standard, it is costly and difficult to scale, and defining reliable automatic metrics for each control node is non-trivial. Recent advances in vision-language models (VLMs) [13, 33, 1] offer a scalable alternative, showing strong performance in video understanding tasks. In this section, we leverage these models to perform automatic evaluation of professional video generation.

Zero-shot VLM Evaluations. The rise of multimodal VLMs has enabled progress on vision-language tasks, including video understanding, making them natural candidates for evaluating professional video generation. We use expert annotations as ground truth and measure VLM alignment by prompting models with a video, its associated prompt, and a specific question tied to a taxonomy node, asking for a 1-5 rating similar to our user study. We explicitly instruct the VLM to ignore factors unrelated to the specified category when evaluating the video. We determine VLM preferences by independently scoring each video and selecting the higher-scoring one. This design mitigates hallucination and order-sensitivity issues commonly observed when prompting with both videos simultaneously.

We use Qwen2.5-VL-Instruct models due to their strong video understanding capabilities. To study the effect of model scale, we evaluate 3 sizes: 7, 32 and 72B. Figure 10 shows that increasing model size does not yield significant improvements in alignment with human judgment. Our results reveal overall poor agreement, consistent with prior findings [20, 64] that highlight the need for finetuning on in-distribution, human-labeled data.

Aligning Human and VLM ratings. We adopt Qwen-2.5-VL-7B [53] as the base model for finetuning; our training and validation dataset consist of 44,062 and 12,763 samples, respectively. We aggregate annotator scores (3/video pair) into binary preferences, excluding ties. Videos are pre-

Hunyuan

Minimax

Wan 14B

Wan 1B

Video Models

0

20

40

60

80

100

Accuracy of Zero-shot VLMs and Ours across Different Video Models

Qwen2.5-VL-7b

Qwen2.5-VL-32b

Qwen2.5-VL-72b

Ours (7B)

processed at 2 fps at their native resolution. Each sample consists of a prompt, two videos, and a binary label. The trained model acts as a classifier; we modify the model architecture to output a scalar score using a linear projection over the final layer's last token. The model takes as input a single video, its prompt, and the evaluation question as input. For training, we use the Bradley-Terry objective [5] due to its sample efficiency over regression [34]. The model is trained for 1 epoch with a batch size of 8 and learning rate 6e-5. Similar to zero-shot evaluation, we compute pairwise preference accuracy against the average of the annotators as the target metric. Our fine-tuned model achieves an overall accuracy of 72.36%, outperforming all zero-shot VLM baselines. This represents an absolute improvement of ∼ 20%, over the baseline 7B model. Our model (Figure 10) shows consistent performance across videos generated by different models, highlighting its ability to generalize across different video qualities. Additional VLM results are presented in Appendix A.7.

6 Conclusion and Future Work

Stable Cinemetrics probes at the intersection of professional video generation and generative video models, grounding prompts, evaluations, and analysis in our structured taxonomies. Our findings reveal where current state-of-the-art models perform well, and where substantial improvements are needed. Our prompt suite offers a strong testbed for future video generative models and can be easily extended as models improve, owing to the flexibility of our taxonomy. We envision several extensions of our work; while our current focus is on evaluation, the taxonomy can also support analyzing video datasets for cinematic diversity or serve as a structure for video captioning. While today's text-to-video models are not yet usable in a fully zero-shot capacity, our findings identify the main challenging pillars for professional filmmaking, illuminating the need for potential solutions like fine-tuning and customization that can bring these models closer to real production use. We hope SCINE encourages deeper exploration at the intersection of filmmaking and video generative models, fostering closer collaboration between artists and models.

8 Limitations

Although our taxonomy was developed in consultation with domain experts, it is limited by the scope of our collaborator network. Filmmaking terminology and interpretive nuance vary across regions and cultures, greater expert diversity would enable broader incorporation of global cinematic controls into the taxonomy. Some taxonomy nodes (e.g., Color Temperature, ISO) were abstracted for evaluation, as we found it difficult for annotators to consistently perceive fine-grained values (such as 2000K or ISO 800). Prompt generation is based on LLMs, whose proprietary nature and potential biases can influence the language and structure of the prompts. Our zero-shot VLM evaluations were bounded by compute and data resources, limiting the scale and scope of the experiments.

References

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. ArXiv , abs/2502.13923, 2025. URL https://api.semanticscholar.org/CorpusID:276449796 .

Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical commonsense for video generation. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=9D2QvO1uWj .

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023. URL https://arxiv.org/abs/2311.15127 .

Bruce Block. The visual story: Creating the visual structure of film, TV, and digital media . Routledge, 2020.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openai.com/research/ video-generation-models-as-world-simulators .

Blain Brown. Cinematography: theory and practice: image making for cinematographers and directors . Routledge, 2016.

James E Cutting and Ayse Candan. Shot durations, shot classes, and the increased pace of popular movies, 2015.

Google DeepMind. Veo 3 tech report. Technical report, DeepMind, 2025. Veo: a text-to-video generation system. Available at https://storage.googleapis.com/deepmind-media/veo/ Veo-3-Tech-Report.pdf .

Edward Dmytryk, Andrew Lund, and Mick Hurbis-Cherrier. On film editing: an introduction to the art of film construction . Routledge, 2018.

Adam Polyak et al. Movie gen: A cast of media foundation models, 2025. URL https: //arxiv.org/abs/2410.13720 .

Guoqing Ma et al. Step-video-t2v technical report: The practice, challenges, and future of video foundation model, 2025. URL https://arxiv.org/abs/2502.10248 .

OpenAI Josh Achiam et al. Gpt-4 technical report. 2023. URL https://api. semanticscholar.org/CorpusID:257532815 .

Weijie Kong et al. Hunyuanvideo: A systematic framework for large video generative models, 2025. URL https://arxiv.org/abs/2412.03603 .

Weichen Fan, Chenyang Si, Junhao Song, Zhenyu Yang, Yinan He, Long Zhuo, Ziqi Huang, Ziyue Dong, Jingwen He, Dongwei Pan, Yi Wang, Yuming Jiang, Yaohui Wang, Peng Gao, Xinyuan Chen, Hengjie Li, Dahua Lin, Yu Qiao, and Ziwei Liu. Vchitect-2.0: Parallel transformer for scaling up video diffusion models, 2025. URL https://arxiv.org/abs/2501. 08453 .

Weixi Feng, Jiachen Li, Michael Saxon, Tsu jui Fu, Wenhu Chen, and William Yang Wang. Tcbench: Benchmarking temporal compositionality in text-to-video and image-to-video generation, 2024. URL https://arxiv.org/abs/2406.08656 .

Syd Field. Screenplay: The foundations of screenwriting . Delta, 2005.

Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitterman, Zeev Melumian, and Ofir Bibi. Ltx-video: Realtime video latent diffusion, 2024. URL https://arxiv.org/abs/2501.00103 .

Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for video diffusion models. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview. net/forum?id=Z4evOUYrk7 .

Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, et al. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. arXiv preprint arXiv:2406.15252 , 2024.

Jay Holben. A Shot in the Dark: A Creative DIY Guide to Digital Video Lighting on (almost) No Budget . Course Technology Press, 2011.

Chen Hou and Zhibo Chen. Training-free camera control for video generation. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview. net/forum?id=KI1zldOFz9 .

Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2icompbench++: An enhanced and comprehensive benchmark for compositional text-to-image generation, 2025. URL https://arxiv.org/abs/2307.06350 .

Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16 , pages 709-727. Springer, 2020.

Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 21807-21818, June 2024.

Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing, 2025. URL https://arxiv.org/abs/2503.07598 .

Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling, 2025. URL https://arxiv.org/abs/2410.05954 .

Steven Douglas Katz. Film directing shot by shot: visualizing from concept to screen . Gulf Professional Publishing, 1991.

MediaBee Color Lab. The art and impact of color grading in cinema, July 2024. URL https://www.mediabeecolorlab.com/post/ the-art-and-impact-of-color-grading-in-cinema . Blog post.

Luma Labs. Ray2, 2025. URL https://lumalabs.ai/ray . Accessed: 2025-05-02.

Pika Labs. Pika 2.2, 2025. URL https://pikalabs.org/pika-2-2/ . Accessed: 2025-05-02.

Mingxiang Liao, Hannan Lu, Xinyu Zhang, Fang Wan, Tianyu Wang, Yuzhong Zhao, Wangmeng Zuo, Qixiang Ye, and Jingdong Wang. Evaluation of text-to-video generation models: A dynamics perspective. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems , volume 37, pages 109790-109816. Curran Associates, Inc., 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ c6483c8a68083af3383f91ee0dc6db95-Paper-Conference.pdf .

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv , abs/2304.08485, 2023. URL https://api.semanticscholar.org/CorpusID:258179774 .

Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback. arXiv preprint arXiv:2501.13918 , 2025.

Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 22139-22149, June 2024.

Luma Labs. Luma ray2 video model. https://lumalabs.ai/ray , 2025. Accessed: 2025-0423.

Nestor Maslej, Loredana Fattorini, Raymond Perrault, Yolanda Gil, Vanessa Parli, Njenga Kariuki, Emily Capstick, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, Tobi Walsh, Armin Hamrah, Lapo Santarlasci, Julia Betts Lotufo, Alexandra Rome, Andrew Shi, and Sukrut Oak. Artificial intelligence index report 2025, 2025. URL https://arxiv.org/abs/2504. 07139 .

MiniMax. Video generation api, 2024. URL https://www.minimax.io/news/ video-generation-api . Accessed: 2025-05-02.

Piotr Mirowski, Kory W. Mathewson, Jaylen Pittman, and Richard Evans. Co-writing screenplays and theatre scripts with language models: An evaluation by industry professionals, 2022. URL https://arxiv.org/abs/2209.14958 .

Motion Picture Laboratories. Ontology for media creation: Part 3f: Images. Technical Report v2.6, MovieLabs, May 2024. URL https://mc.movielabs.com/docs/ontology/ assets-images/images/ . Accessed: 2025-05-15.

Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang, Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1m: A large-scale high-quality dataset for text-to-video generation. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=j7kdXSrISM .

Robert Plutchik. The emotions . University Press of America, 1991.

Michael Rabiger. Directing: Film techniques and aesthetics . Routledge, 2013.

Anyi Rao, Xuekun Jiang, Yuwei Guo, Linning Xu, Lei Yang, Libiao Jin, Dahua Lin, and Bo Dai. Dynamic storyboard generation in an engine-based virtual environment for video production. In ACM SIGGRAPH 2023 Posters , pages 1-2. 2023.

Rohit Saxena and Frank Keller. Moviesum: An abstractive summarization dataset for movie screenplays, 2024. URL https://arxiv.org/abs/2408.06281 .

Linda Seger and Edward Jay Whetmore. From Script to Screen: The collaborative art of filmmaking . Lone Eagle Publishing Company, LLC, 2004.

StudioBinder. What are the major film studios?, n.d. URL https://www.studiobinder.com/ blog/what-are-the-major-film-studios/ . Accessed: 2025-05-14.

Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2vcompbench: A comprehensive benchmark for compositional text-to-video generation, 2025. URL https://arxiv.org/abs/2407.14505 .

Richard Szeliski. Computer vision: algorithms and applications . Springer Nature, 2022.

Yuying Tang, Haotian Li, Minghe Lan, Xiaojuan Ma, and Huamin Qu. Understanding screenwriters' practices, attitudes, and future expectations in human-ai co-creation, 2025. URL https://arxiv.org/abs/2502.16153 .

Judith M. Tanur, George Casella, Richard Dykstra, Mark P. Finster, Donald P. Gaver, Joel Greenhouse, Gudmund R. Iversen, guillermina Jasso, Jan Kmenta, S. James Press, Seymour Sudman, Luke Tierney, Jessica Utts, Katherine K. Wallman, Stanley Wasserman, and Mark Watson. Journal of the American Statistical Association , 84(407):830-834, 1989. ISSN 01621459, 1537274X. URL http://www.jstor.org/stable/2289675 .

Genmo Team. Mochi 1. https://github.com/genmoai/models , 2024.

Qwen Team. Qwen2.5-vl, January 2025. URL https://qwenlm.github.io/blog/qwen2. 5-vl/ .

Veo-Team. Veo 2. 2024. URL https://deepmind.google/technologies/veo/veo-2/ .

Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion, 2024. URL https: //arxiv.org/abs/2403.12008 .

Team Wan. Wan: Open and advanced large-scale video generative models, 2025. URL https://arxiv.org/abs/2503.20314 .

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual e5 text embeddings: A technical report, 2024. URL https://arxiv.org/abs/ 2402.05672 .

Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation, 2024. URL https://openreview.net/forum?id=dUDwK38MVC .

Yiping Wang, Xuehai He, Kuan Wang, Luyao Ma, Jianwei Yang, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Is your world simulator a good story presenter? a consecutive events-based benchmark for future long video generation, 2024. URL https: //arxiv.org/abs/2412.16211 .

Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution , pages 196-202. Springer, 1992.

Aylish Wood. Pixel visions: Digital intermediates and micromanipulations of the image. Film Criticism , 32(1):72-94, 2007.

Weijia Wu, Zeyu Zhu, and Mike Zheng Shou. Automated movie generation via multi-agent cot planning, 2025. URL https://arxiv.org/abs/2503.07314 .

Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkuo Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang. Easyanimate: A high-performance long video generation method based on transformer architecture, 2024. URL https://arxiv.org/abs/2405.18991 .

Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation. arXiv preprint arXiv:2412.21059 , 2024.

Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan.Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with an expert transformer. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=LQzN6TRFg9 .

Chi Zhang, Yuanzhi Liang, Xi Qiu, Fangqiu Yi, and Xuelong Li. Vast 1.0: A unified framework for controllable and consistent video generation, 2024. URL https://arxiv.org/abs/2412. 16677 .

Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness, 2025. URL https://arxiv.org/abs/2503.21755 .

Mingzhe Zheng, Yongqi Xu, Haojian Huang, Xuran Ma, Yexin Liu, Wenjie Shu, Yatian Pang, Feilong Tang, Qifeng Chen, Harry Yang, and Ser-Nam Lim. Videogen-of-thought: Step-by-step generating multi-shot video with minimal manual intervention, 2025. URL https://arxiv.org/abs/2412.02259 .

Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. RoboDreamer: Learning compositional world models for robot imagination. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 61885-61896. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/zhou24f.html .

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately capture the paper's contribution and scope by clearly stating the claims made and outlining primary contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our current work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not present theoretical results, assumptions, or proofs. Instead, it proposes a taxonomy for professional movie elements and provides statistical analysis on how current state-of-the-art text-to-video (T2V) models perform across different pillars of the taxonomy.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides all necessary information to reproduce the main experimental results. Details for creating the taxonomy are available in Section 3. The setup for VLM evaluation is described in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release our proposed taxonomy upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper provides all necessary information to reproduce the main experimental results. Details for creating the taxonomy are available in Section 3. The setup for VLM evaluation is described in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report statistically significant annotator agreements within the annotators. Additional details are present in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, we report these numbers in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper complies with NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Yes, we report the impacts of our current work in the Supplementary.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not anticipate any high risk misuse of our current work.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite and credit papers that introduce the pre-trained models we train on, as well as the LLM used for generating prompts, and the metrics we use to compare across models.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will release our taxonomy which is well documented.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: We discuss the full text of instructions given to participants and include screenshots in the Supplementary.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [No]

Justification: The user study involves participants score/rank videos generated with a predetermined list of prompts and quetions in the professional pillars. As such, we believe the study does not pose any risk or harm to participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: We provide information on using LLMs in helping us creating prompt set and questions in Section 3.2 and 3.3.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Appendix

A.1 Taxonomy Details

We provide additional details on control nodes and their associated values in the taxonomies. Some nodes accept open-ended values, for example, a range of stand-alone actions. To simplify evaluation, we abstract certain values that can be fine-grained in future works. For instance, we group Aperture into wide/medium/narrow, though exact f-stop values can also be studied in the future. Similarly, color palette is treated as a discrete value in our current work, but can be decomposed into hue, brightness, and saturation. Table 3 - 6 details the control nodes and their values of the Camera, Lighting, Setup and Events Taxonomies, respectively.

Minimax

Wan 14B

Wan 1B

Hunyuan

Mochi

StepVideo

Pika 2.2

CogVideo 5B

Pyramid Flow

LTX Video

VChitect 2.0

Model

1.5

2.0

2.5

3.0

3.5

Score

Causal

Atomic

Concurrent

Sequential

Overlapping

Cyclic

A.2 Additional Results and Analysis

A.2.1 Events

Figure 11 shows Events performance across 13 models and 6 Temporal portrayal of Actions. Models handle atomic and concurrent actions well, but struggle with causal, overlapping, and cyclic events. Figure 12 shows Events performance across 13 models and 12 genres. Biography and Adventure are strongest whereas Comedy and Horror are the weakest. Minimax leads in 6/12 genres, Luma Ray 2 tops Action and Drama, and WAN-14B is the most consistent, with the lowest standard deviation.

Figure 13 shows performance of 10 open-source models across 19 emotion classes. Models perform best on remorse and ecstasy, but fare poorly on aggressiveness and rage. As shown in Figure 14, dialogue performance is weaker in comparison to Emotions and Actions. Models particularly struggle with multi-turn dialogues or when non-verbal reactions follow. Since T2V models do not generate audio, we evaluate whether the correct character delivers the line and/or with appropriate visual expression. Most models fail to localize the speaker, often attributing a single dialogue to multiple characters.

Minimax

Wan 14B

Hunyuan

Wan 1B

Pika 2.2

Mochi

CogVideo 5B

StepVideo

Pyramid Flow

VChitect 2.0

LTX Video

Model

1.5

2.0

2.5

3.0

3.5

4.0

Score

Action

Adventure

Animation

Biography

Comedy

Crime

Drama

Fantasy

Horror

Romance

Science Fiction

Thriller

Wan 14B

Hunyuan

Wan 1B

StepVideo

Mochi

CogVideo 5B

Pyramid Flow

VChitect 2.0

LTX Video

Model

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Score

Admiration

Aggressiveness

Anger

Anticipation

Awe

Contempt

Disapproval

Disgust

Ecstasy

Grief

Joy

Love

Optimism

Rage

Remorse

Sadness

Surprise

Terror

Minimax

Wan 14B

Hunyuan

Pika 2.2

Wan 1B

Mochi

StepVideo

CogVideo 5B

Pyramid Flow

VChitect 2.0

LTX Video

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Score

Dialogues

Dialogues With Reaction

Minimax

Wan 1B

Wan 14B

Hunyuan

LTX Video

StepVideo

CogVideo 5B

Model

2.0

2.5

3.0

3.5

4.0

Score

Firelight

Fluorescent

HMI

LED

Moonlight

Strobes

Sunlight

Tungsten

Wan 14B

Minimax

Wan 1B

Hunyuan

StepVideo

LTX Video

CogVideo 5B

Model

2.0

2.5

3.0

3.5

4.0

Score

Aerial

Dutch

Ground

High

Hip

Knee

Low

Overhead

Shoulder

Eye-level

A.2.2 Camera and Lighting

On Lighting Source (Figure 15), Sunlight, Strobes, and Firelight are handled more reliably, while HMI, Fluorescent, and Tungsten lighting show lower performance. As shown in Figure 16, Aerial and Knee level camera angles are depicted better, while the Dutch and Shoulder-level angles show lower performance. On Shot Sizes (Figure 17), Medium-Wide and Master shots have stronger performance in comparison to Full and Extreme Close-Up shots.

Minimax

Wan 14B

Hunyuan

Wan 1B

LTX Video

CogVideo 5B

Model

2.0

2.5

3.0

3.5

4.0

Score

Close-up

Establishing

Extreme Close-up

Full

Master

Medium

Medium-Full

Medium-Close-Up

Medium-Wide

Wide

A.2.3 Setup

For the Setup taxonomy, we also analyze performance at the value level. In Balance (Figure 18), models handle rule of thirds framing more effectively but struggle with symmetrical compositions. For Time of Day (Figure 19), among 11 categories, Sunrise and Morning are portrayed well, while Afternoon remains challenging.

Minimax

Wan 14B

Hunyuan

StepVideo

Wan 1B

CogVideo 5B

LTX Video

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Score

Right Heavy

Rule Of Thirds

Symmetry

Afternoon

Dawn

Day

Evening

Late Night

Midday

Morning

Night

Sunrise

Sunset

Minimax

Wan 14B

Hunyuan

Wan 1B

StepVideo

CogVideo 5B

LTX Video

Model

2.0

2.5

3.0

3.5

4.0

Score

Taxonomy

(Camera/Lighting/Setup)

Instruction

Taxonomy

Prompt

Seed

Prompts

Instruction

SCINE

Scripts

Taxonomy

(Events)

SCINE

Scripts

Instruction

2

1

Categories

Questions

SCINE

Visuals

首

自

自

X

SALLME

三

A.3 Details on Prompt Generation

The overall evaluation pipeline, depicting prompt generation, categorization and question generation, is presented in Figure 20.

Table 7 presents an example of a prompt and the corresponding categories and generated questions.

Below, we show an example of the instruction given to an LLM to upsample a SCINE-Script into SCINE-Visuals by incorporating control nodes from the Camera and Lighting taxonomies.

SCINE Scripts - Cinematographer

System Prompt :

You are a world-class cinematographer known for your visionary storytelling, mastery of light, and camera. You have decades of experience working on award-winning films across genres, collaborating with top directors and production teams. Your insights blend technical expertise with artistic sensibility. When describing scenes or advising on visual storytelling, you use cinematic terminology with clarity and inspiration. Think like Roger Deakins, Emmanuel Lubezki, and Greig Fraser-your visual choices always elevate the emotional tone and narrative arc of a project.

User

Prompt

:

GOAL

You will be given a prompt and 2 taxonomies that define camera and lighting controls commonly used by cinematic professionals. Your objective is to enrich the given prompt by sampling relevant nodes from both the taxonomies. As a cinematographer, your role is to "shoot" this scene using the best possible cinematic expression, utilizing the camera and lighting control options provided in the taxonomy.

PROMPT : {prompt}

MOST IMPORTANT INFORMATION

Only Use Nodes from the Provided Taxonomies : - You must never introduce nodes that are not present in the given taxonomies. - While the values within each node can be flexible-allowing for creativity and imagination grounded in your professional experience. For example, the node "Color Gel" is defined, but has no values. It is upto you to define these values. - The structure must strictly adhere to the nodes defined in the taxonomy. Think expansively within the bounds of each node, but never go beyond them.

Preserve the Original Prompt Content : - Do NOT remove or add any of the original content from the input prompt. - Your only task is to enrich the prompt by layering in camera and lighting related information. The core semantics and narrative of the prompt must remain entirely intact.

Do NOT include the path through which you sample the nodes in the prompt. That is, do NOT add the paths from the taxonomy using '->'.

GUIDELINES 1. Input Prompt - The input prompt describes a single continuous event, intended to occur within one uninterrupted shot. Therefore, do not include any cuts or multiple camera setups. Assume this is a one-shot sequence.

2. Each node in the taxonomies contains: - Description: A definition of what the node represents.

Example: An example of how the node may appear in a prompt.

Values: A non-exhaustive list of possible values for the node. Some notation: a. OPEN SET - Indicates the node supports a wide range of possible values.

[] - Indicates the node may have multiple values, which are not predefined and should be selected based on your reasoning and cinematic knowledge.

3. Enriched Prompt - Your enriched version will serve as input to a text-to-video model. It must be fluent, natural, and interpretable by the model, while incorporating cinematic elements effectively.

CAMERA TAXONOMY The Camera Taxonomy defines elements related to the camera's intrinsics, extrinsics, and its cinematic use : {camera_taxonomy}

LIGHTING TAXONOMY The Lighting taxonomy broadly defines all elements of lighting, including source, position of lighting, along with its effects such as shadows and reflections, along with color temperature, lighting motion such as flickering etc : {lighting_taxonomy}

When incorporating lighting into your enriched prompt, remember that a cinematographer can shape the look and feel of a shot by selectively illuminating different depth planes of the scene. Lighting can be applied to the foreground, mid-ground, background, and the subject itself-either individually or in combination. Your choices should support the emotional tone, visual focus, and narrative intent of the shot.

Below, we show an example of the instruction given to an LLM to categorize and generate evaluation questions for an input prompt using the Camera taxonomy.

Camera Categorization and Question Generation

GOAL

You are an expert prompt evaluator. Your task is to analyze a video generation prompt and categorize it based on a predefined taxonomy.

PROMPT : {prompt}

Available Categories (with Examples)

The category presented to you is that of Camera. The Camera taxonomy broadly defines everything related to the camera -the intrinsics, the extrinsics and the cinematic use of camera. {camera_taxonomy}

Notes about the Taxonomy

Each node in the taxonomy contains :

Description : Definition of what that node represents.

Example : An example of the presence of a node in the form of a prompt.

Values : A non-exhaustive list of values of these nodes. Values are a list of values that this node can have. Some nomenclature :

OPEN SET indicates that this node contains a large number of values.

[] indicates that this node may have multiple values, but are not defined explicitly and it is upto your reasoning and knowledge.

Examples of Categorization

Static Medium-Close-Up of David's face showing quiet devastation. Quick Push In as tears well up in his eyes. Shot with a medium ISO to capture the dim apartment lighting.

Static -Camera -> Trajectory -> Camera Movement -> Static

Push In -Camera -> Trajectory -> Camera Movement ->

Medium-Close up -Camera -> Creative Intent -> Shot

Medium ISO -Camera -> Intrinsics -> Exposure -> ISO

Wide shot of a bustling city street at night. The neon lights of the shops and restaurants cast a colorful glow on the wet pavement. People walk by, their faces illuminated by the bright signs. The camera pans up to reveal the towering skyscrapers that loom overhead, their windows reflecting the city lights.

Pans Up -Camera -> Trajectory -> Camera Movement -> 2D

TASK

Analyze the given prompt and return the following structured output in a valid JSON format:

Words: Extract important keywords or key phrases from the prompt using the following guidance:

-Identify named entities related to a camera in professional use as you would in NER (Named Entity Recognition).

Extract noun phrases or descriptive terms that relate to a camera.

Prefer multi-word expressions where meaningful related to a camera.

Avoid generic or uninformative words like 'a', 'video', 'the', etc.

Categories: For each word or phrase, assign the most appropriate category from the taxonomy. A dictionary of relevant categories from the taxonomy.

-For each relevant category, assign a score between '0' and '1' representing how strongly the prompt matches the category.

Provide a reason for each score, referring to the words or phrases extracted and how they relate to the category.

-Generate a question that helps a human evaluator determine whether this category is visually present in the generated video. Use your reasoning to guide the question. The

evaluator will use this question to rate the video on a scale from 1 (not at all) to 5 (strongly represented).

The generated question should evaluate quality, consistency and presence of the node in the video.

Important Guidelines :

-The camera information should be explicitly mentioned in the prompt. Do NOT imply, assume or derive anything. Only consider a word or a phrase a match, if it is explicitly mentioned in the prompt.

-Each prompt can have multiple nodes of the Camera taxonomy. You should capture all of the nodes in the prompt and map it back to the taxonomy.

-You must always traverse from the root node, which is Camera in this case. That is, the 'category' should always start as (Camera -> ..)

-You will never create a node that is not in the taxonomy. These nodes can have multiple values, as previously explained and you are expected to be imaginative about the values. But the nodes, should always come from the given taxonomy.

-Since the taxonomy is of Camera, we do not care about objects, subjects, lighting, events, actions or emotions. Your sole focus should be about camera terms that are present in the prompt in accordance with the taxonomy. You will NOT ask any question related to objects, subjects, lighting, events, actions or emotions.

Table 8 compares SCINE-Scripts with existing prompt-based video generation benchmarks. We compute token level metrics: Type-Token Ratio (TTR), Distinct Bi-Grams, and average pairwise Jaccard Distance, and find that SCINE-Scripts exhibits strong lexical diversity.

A.4 Distribution of Taxonomy Categories in SCINE Prompts

Figures 21, 22, and 23 show the distribution of activated nodes in SCINE Visuals, aggregated at the node level, across the roles of Cinematographer, Production Designer, and Director, respectively. As shown, our prompts cover a broad distribution of nodes across all the taxonomies.

A.5 Annotation Details

Figure 24 shows the annotation interface used by human annotators during evaluation. We also present the distribution of annotators' years of experience in film production in Figure 25. While annotations for cinematic controls can be subjective, especially given the large number of control nodes, we try our best to mitigate this by providing clear rating guidelines to annotators for each control node. Table 9 presents a minimal example of the rating guidelines shared with the annotators.

0

250

500

750

1000

1250

1500

1750

2000

Frequency

Camera

Lighting

Activation Frequency of Taxonomy Nodes at Level 1

(a) Level 1 Activations

0

100

200

300

400

500

600

700

800

Frequency

Camera -> Intrinsics

Camera -> Trajectory

Camera -> Creative Intent

Lighting -> Sources

Camera -> Extrinsics

Lighting -> Lighting Position

Lighting -> Color T emperature

Lighting -> Advanced Controls

Lighting -> Lighting Conditions

Lighting -> Lighting Effects

Activation Frequency of Taxonomy Nodes at Level 2

(b) Level 2 Activations

0

50

100

150

200

250

300

Frequency

Camera -> Trajectory -> Camera Movement

Camera -> Intrinsics -> Exposure

Camera -> Creative Intent -> Shot Size

Camera -> Intrinsics -> Depth of Field

Camera -> Intrinsics -> Lens Size

Camera -> Trajectory -> Gear

Camera -> Extrinsics -> Angle

Lighting -> Sources -> Natural Light

Lighting -> Sources -> Artificial/Practicals Light

Lighting -> Advanced Controls -> Motion

Camera -> Creative Intent -> Framing

Lighting -> Advanced Controls -> Color Gels

Lighting -> Lighting Effects -> Shadows

Lighting -> Lighting Effects -> Reflection

Activation Frequency of Taxonomy Nodes at Level 3

(c) Level 3 Activations

0

25

50

75

100

125

150

175

200

Frequency

Camera -> Trajectory -> Camera Movement -> 3D

Camera -> Intrinsics -> Exposure -> Aperture

Camera -> Intrinsics -> Exposure -> ISO

Camera -> Trajectory -> Camera Movement -> 2D

Camera -> Intrinsics -> Exposure -> Shutter Speed

Camera -> Trajectory -> Camera Movement -> Static

Lighting -> Lighting Effects -> Shadows -> Soft

Lighting -> Lighting Effects -> Shadows -> Hard

Activation Frequency of Taxonomy Nodes at Level 4

(d) Level 4 Activations

0

200

400

600

800

1000

1200

1400

Frequency

Setup -> Scene

Setup -> Subjects

Setup -> Text Generation

Activation Frequency of Taxonomy Nodes at Level 2

(a) Level 2 Activations

0

200

400

600

800

Frequency

Setup -> Scene -> Set Design

Setup -> Subjects -> Class

Setup -> Scene -> Texture

Setup -> Scene -> Geometry

Setup -> Subjects -> Costume

Setup -> Subjects -> Hair

Setup -> Subjects -> Accessories

Setup -> Subjects -> Pose

Setup -> Subjects -> Makeup

Setup -> Subjects -> Personality

Activation Frequency of Taxonomy Nodes at Level 3

(b) Level 3 Activations

0

100

200

300

400

Frequency

Setup -> Scene -> Set Design -> Backdrop

Setup -> Scene -> Set Design -> Props

Setup -> Scene -> Set Design -> Environment

Setup -> Scene -> Geometry -> Frame

Setup -> Scene -> Texture -> Color Palette

Setup -> Scene -> Texture -> Contrast

Setup -> Scene -> Geometry -> Space

Setup -> Scene -> Texture -> Film Grain

Setup -> Scene -> Texture -> Blur

Setup -> Scene -> Texture -> Noise

Activation Frequency of Taxonomy Nodes at Level 4

(c) Level 4 Activations

0

25

50

75

100

125

150

175

Frequency

Setup -> Scene -> Set Design -> Backdrop -> Setting

Setup -> Scene -> Set Design -> Backdrop -> Time of Day

Setup -> Scene -> Set Design -> Backdrop -> Location

Setup -> Scene -> Geometry -> Frame -> Lines

Setup -> Scene -> Set Design -> Environment -> Elements

Setup -> Scene -> Geometry -> Space -> Depth

Setup -> Scene -> Set Design -> Props -> Class

Setup -> Scene -> Geometry -> Frame -> Balance

Setup -> Scene -> Set Design -> Environment -> Organization

Setup -> Scene -> Set Design -> Props -> Description

Setup -> Scene -> Set Design -> Props -> Material

Setup -> Scene -> Geometry -> Frame -> Shapes

Setup -> Scene -> Set Design -> Props -> Pattern

Setup -> Scene -> Set Design -> Environment -> Style

Setup -> Scene -> Geometry -> Space -> Spatial Locations

Setup -> Scene -> Set Design -> Environment -> Background

Setup -> Scene -> Set Design -> Environment -> Mood

Activation Frequency of Taxonomy Nodes at Level 5

(d) Level 5 Activations

0

250

500

750

1000

1250

1500

1750

Frequency

Setup

Lighting

Camera

Events

Activation Frequency of Taxonomy Nodes at Level 1

(a) Level 1 Activations

0

200

400

600

800

1000

1200

1400

Frequency

Setup -> Scene

Events -> Types

Setup -> Subjects

Camera -> Trajectory

Camera -> Intrinsics

Lighting -> Sources

Camera -> Creative Intent

Lighting -> Color T emperature

Lighting -> Lighting Conditions

Camera -> Extrinsics

Lighting -> Lighting Position

Lighting -> Lighting Effects

Lighting -> Advanced Controls

Setup -> Text Generation

Events -> Advanced Controls

Activation Frequency of Taxonomy Nodes at Level 2

(b) Level 2 Activations

0

200

400

600

800

Frequency

Setup -> Scene -> Set Design

Events -> Types -> Actions

Setup -> Scene -> Geometry

Setup -> Subjects -> Class

Camera -> Trajectory -> Camera Movement

Camera -> Creative Intent -> Shot Size

Camera -> Intrinsics -> Depth of Field

Lighting -> Sources -> Natural Light

Setup -> Scene -> Texture

Lighting -> Sources -> Artificial/Practicals Light

Camera -> Extrinsics -> Angle

Camera -> Intrinsics -> Lens Size

Camera -> Trajectory -> Gear

Setup -> Subjects -> Costume

Events -> Types -> Emotions

Lighting -> Lighting Effects -> Shadows

Lighting -> Advanced Controls -> Motion

Events -> Types -> Dialogues

Camera -> Creative Intent -> Framing

Setup -> Subjects -> Accessories

Camera -> Intrinsics -> Exposure

Lighting -> Lighting Effects -> Reflection

Setup -> Subjects -> Hair

Lighting -> Advanced Controls -> Color Gels

Setup -> Subjects -> Personality

Events -> Advanced Controls -> Rhythm

Activation Frequency of Taxonomy Nodes at Level 3

(c) Level 3 Activations

0

100

200

300

400

500

Frequency

Setup -> Scene -> Set Design -> Backdrop

Setup -> Scene -> Set Design -> Props

Setup -> Scene -> Geometry -> Frame

Events -> Types -> Actions -> Interaction Types

Setup -> Scene -> Set Design -> Environment

Camera -> Trajectory -> Camera Movement -> 3D

Events -> Types -> Actions -> Portrayed as

Events -> Types -> Emotions -> Expression Types

Setup -> Scene -> Geometry -> Space

Setup -> Scene -> Texture -> Contrast

Camera -> Trajectory -> Camera Movement -> Static

Camera -> Trajectory -> Camera Movement -> 2D

Setup -> Scene -> Texture -> Color Palette

Lighting -> Lighting Effects -> Shadows -> Hard

Setup -> Scene -> Texture -> Blur

Lighting -> Lighting Effects -> Shadows -> Soft

Setup -> Scene -> Texture -> Film Grain

Camera -> Intrinsics -> Exposure -> Aperture

Setup -> Scene -> Texture -> Noise

Camera -> Intrinsics -> Exposure -> Shutter Speed

Setup -> Subjects -> Personality -> Silhouette

Camera -> Intrinsics -> Exposure -> ISO

Events -> Advanced Controls -> Rhythm -> Regularity

Events -> Types -> Emotions -> Portrayed as

Activation Frequency of Taxonomy Nodes at Level 4

(d) Level 4 Activations

VideoA

Prompt

Ona cityrooftopatnight,music pulsesaspeople dance under stringlights.Other guestsstand in smallgroups neartherailing,

talking and admiring theglittering cityview,while abartendermixes drinks behind aportablebar.

Events→Types→ChangeinEnvironment

Does the video clearly depict pulsating music as a dynamic environmental element that

contributestotheoverallatmosphere?

1.Score A(select1)*

1

2

VideoB

1.Score B(select1)*

1

2

Events→Types→Actions

Doesthevideoeffectivelyshowpeopledancingwithlively,coordinatedmovements

thatcapturetheintendedenergyofthescene?

1.Score A(select1)*

1

2

1.Score B(select1)*

1

2

5

1-4

5-8

9-12

13-16

17-20

21-24

Years of Experience

0

5

10

15

20

25

30

35

Number of Annotators

A.6 Statistical tests

Pairwise t-tests show that the vast majority of model comparisons in our human evaluation, across all taxonomies are statistically significant at the 5% level (p <0.05). Figure (26 - 28) presents the t-test results of Events, Lighting and Camera, and Setup, respectively.

Pairwise Statistical Comparison

Emotions

Pairwise Statistical Comparison

Dialogues

Pyramid Flow

CogVideo 5B

Hunyuan

LTX Video

Minimax

Mochi

Pika 2.2

StepVideo

VChitect 2.0

Wan 14B

Model B

CogVideo 5B

Hunyuan

LTX Video

Minimax

Mochi

Pika 2.2

StepVideo

VChitect 2.0

Wan 14B

Wan 1B

t=9.72

p=3.1e-22

t=10.91

p=1.5e-27

t=1.02

p=0.31

t=29.58

p=1.7e-183

t=19.46

p=1.2e-82

t=18.72

p=9.6e-77

t=-12.48

p=1.8e-35

t=-22.22

p=2e-106

t=-23.65

p=5.9e-120

t=-42.67

p=0

t=30.35

p=9.8e-193

t=20.09

p=8.2e-88

t=19.35

p=9.3e-82

t=0.39

p=0.7

t=43.66

p=0

t=35.27

p=4.1e-254

t=25.10

p=4.3e-134

t=24.44

t=5.74

p=9.7e-09

t=48.44

p=0

t=5.43

p=5.9e-08

t=13.94

p=1.1e-43

t=3.98

p=7e-05

t=3.00

p=0.0027

t=-15.78

p=2.5e-55

t=26.79

p=4.9e-152

t=-16.37

p=2.3e-59

t=-21.52

p=5.5e-100

t=21.02

p=1.1e-95

t=10.90

p=1.7e-27

t=10.03

p=1.5e-23

t=-8.87

p=8.4e-19

t=34.10

p=4.9e-239

t=-9.38

p=8.3e-21

t=-14.66

p=4.9e-48

t=7.03

p=2.3e-12

t=14.04

p=2.8e-44

t=4.49

p=7.1e-06

t=3.56

p=0.00037

t=-14.47

p=6.8e-47

t=26.28

p=1.7e-146

t=-15.01

p=2.6e-50

t=-19.99

p=7.1e-87

t=0.69

p=0.49

t=-6.05

p=1.5e-09

t=-7.37

p=1.8e-13

t=-17.24

p=1.7e-65

t=-18.59

p=1.1e-75

t=-37.71

p=2.5e-288

t=5.30

p=1.2e-07

t=-38.63

p=9.4e-302

t=-43.51

p=0

t=-21.72

p=5.8e-102

t=-29.04

p=7.8e-177

t=-21.45

p=1.8e-99

t=34.64

p=1.4e-246

t=24.48

p=4.4e-128

t=23.82

p=1.7e-121

t=5.12

p=3.2e-07

t=47.80

p=0

t=4.79

p=1.7e-06

t=-0.62

p=0.53

t=20.89

p=1.3e-94

t=14.03

p=3.3e-44

t=19.39

p=5.3e-82

t=42.87

p=0

t=24.99

p=2.8e-133

t=14.83

p=3.7e-49

t=14.01

p=3.7e-44

t=-4.87

p=1.1e-06

t=38.13

p=7.8e-295

t=-5.33

t=-10.66

p=2.4e-26

t=11.03

p=4.2e-28

t=4.03

p=5.6e-05

t=9.89

p=6.2e-23

t=33.10

p=1.5e-226

t=-10.03

p=1.5e-23

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(a) Pairwise t-tests across models on emotions

Pyramid Flow

CogVideo 5B

Hunyuan

LTX Video

Minimax

Mochi

Pika 2.2

StepVideo

VChitect 2.0

Wan 14B

Model B

CogVideo 5B

Hunyuan

LTX Video

Minimax

Mochi

Pika 2.2

StepVideo

VChitect 2.0

Wan 14B

Wan 1B

Model A

t=6.59

p=5e-11

t=7.98

p=1.8e-15

t=1.24

p=0.21

t=20.04

p=1.3e-85

t=13.23

p=3.2e-39

t=12.21

p=9.2e-34

t=-8.53

p=2e-17

t=-14.89

p=5.4e-49

t=-16.44

p=4e-59

t=-28.26

p=1.6e-161

t=25.78

p=4.5e-137

t=18.50

p=9.2e-74

p=1.2e-66

t=4.65

p=3.5e-06

t=34.43

t=29.25

p=8.2e-172

t=22.09

p=1.3e-102

t=21.19

p=5.8e-95

t=8.52

p=2.2e-17

t=37.66

p=3.1e-268

t=4.13

p=3.6e-05

t=9.24

p=3.6e-20

t=2.54

p=0.011

p=0.18

t=-10.82

t=17.62

p=2.5e-67

t=-16.02

p=2.8e-56

t=-19.69

p=8.1e-83

t=16.50

p=2e-59

t=9.46

t=8.34

p=9.8e-17

t=-4.25

p=2.1e-05

t=25.10

p=3.4e-130

p=2.7e-20

t=-13.18

p=6.7e-39

t=6.92

p=5.2e-12

t=7.83

p=6e-15

t=1.34

p=0.18

t=0.14

p=0.89

t=-11.68

p=4.4e-31

t=15.96

p=7.3e-56

t=-16.78

p=2.1e-61

t=-20.36

p=3.7e-88

t=-1.14

p=0.25

t=-7.90

p=3.4e-15

p=1.5e-13

t=-13.87

p=7.3e-43

t=-15.42

p=2.2e-52

t=-27.39

p=2.2e-152

t=1.23

p=0.22

t=-33.59

p=6e-221

t=-36.87

p=5.4e-258

t=-16.62

p=2.6e-60

t=-24.16

p=2.9e-121

t=-14.97

p=1.7e-49

t=23.63

p=2e-116

t=16.70

p=8.4e-61

t=15.73

p=2.6e-54

t=3.37

p=0.00077

t=31.90

p=5e-201

t=-1.17

p=0.24

t=-5.13

p=3e-07

t=14.30

p=2.1e-45

t=7.77

p=9.8e-15

t=15.09

p=3.3e-50

t=31.06

p=2.3e-191

t=15.53

p=4.4e-53

t=8.55

p=1.7e-17

t=7.42

p=1.4e-13

t=-5.09

p=3.8e-07

t=24.08

p=6e-121

t=-10.12

p=7.8e-24

t=-13.99

p=1.5e-43

t=6.02

p=1.9e-09

p=0.37

t=7.02

p=2.5e-12

t=23.14

p=3.3e-112

t=-8.59

p=1.2e-17

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(b) Pairwise t-tests across models on dialogues

Pyramid Flow

CogVideo 5B

Hunyuan

LTX Video

Minimax

Mochi

Pika 2.2

StepVideo

VChitect 2.0

Wan 14B

Model B

CogVideo 5B

Hunyuan

LTX Video

Minimax

Mochi

Pika 2.2

StepVideo

VChitect 2.0

Wan 14B

Wan 1B

Model A

t=13.07

p=6.5e-39

t=14.79

p=2.9e-49

t=1.38

p=0.17

t=38.34

p=0

t=24.58

p=1.2e-131

t=23.73

p=6.5e-123

t=-17.02

p=1.6e-64

t=-29.91

p=1.3e-192

t=-32.08

t=-55.81

p=0

t=40.58

p=0

t=27.33

p=1.9e-161

t=26.54

p=1.5e-152

t=3.68

t=57.33

p=0

t=52.78

p=0

t=38.95

p=0

t=38.39

p=0

t=14.87

p=1e-49

t=70.18

p=0

t=10.71

p=1.1e-26

t=21.47

p=3e-101

t=7.89

p=3.2e-15

t=6.66

p=2.9e-11

t=-17.19

p=8.1e-66

t=38.92

p=0

t=-20.27

p=2.2e-90

t=-31.98

p=7.7e-219

t=28.53

p=1.7e-175

t=15.07

p=4.6e-51

t=14.02

p=2e-44

t=-9.45

p=3.8e-21

t=45.70

p=0

t=-12.76

p=3.8e-37

t=-24.09

p=3e-126

t=7.51

p=6.2e-14

t=18.50

p=8.3e-76

t=5.45

p=5e-08

t=4.21

p=2.6e-05

t=-18.82

p=2.4e-78

t=35.22

p=3.7e-264

t=-21.76

p=1e-103

t=-33.13

t=-2.22

p=0.026

t=-9.44

p=4.1e-21

t=-13.77

p=5.9e-43

t=-26.75

p=5.1e-155

t=-28.84

p=1.5e-179

t=-52.62

p=0

t=3.32

p=0.0009

t=-54.28

p=0

t=-67.05

p=0

t=-35.68

p=6.7e-271

t=-42.53

p=0

t=-32.10

p=9.8e-221

t=49.67

p=0

t=35.63

p=3.3e-270

t=35.01

p=2.8e-261

t=11.08

p=1.9e-28

t=67.41

p=0

t=6.98

t=-3.99

p=6.5e-05

t=28.47

p=4.5e-175

t=20.48

p=2.6e-92

t=29.73

p=2.3e-190

t=64.21

p=0

t=32.60

p=1e-227

t=18.75

p=7.5e-78

t=17.76

p=4.1e-70

t=-6.24

p=4.4e-10

t=50.27

p=0

t=-9.74

p=2.4e-22

t=-21.22

p=8.7e-99

t=11.12

p=1.2e-28

t=3.38

p=0.00073

t=12.96

p=2.8e-38

t=47.03

p=0

p=4.5e-68

Pairwise Statistical Comparison

Actions

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(c) Pairwise t-tests across models on actions

Model A

Model A

Model A

Pairwise Statistical Comparison

Color T emperature

Pairwise Statistical Comparison

Effects

CogVideo 5B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Model B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Wan 1B

t=9.43

p=4.7e-21

t=20.36

p=1.3e-90

t=11.00

p=5e-28

t=9.05

p=1.7e-19

t=-0.37

p=0.71

t=-11.35

p=1.1e-29

t=19.14

t=9.97

p=2.7e-23

t=-0.80

p=0.42

t=10.30

p=8.6e-25

t=25.13

p=6.2e-136

t=15.78

p=1.6e-55

t=4.73

p=2.3e-06

t=16.11

p=8.5e-58

t=5.43

p=5.7e-08

t=13.91

p=1.2e-43

t=4.84

p=1.3e-06

t=-5.76

p=8.5e-09

t=5.18

p=2.2e-07

t=-4.87

p=1.1e-06

t=-10.33

p=6.3e-25

t=23.76

p=4.7e-122

t=14.44

p=6.6e-47

t=3.45

t=14.78

p=5.4e-49

t=4.18

p=2.9e-05

t=-1.25

p=0.21

t=9.08

p=1.3e-19

t=22.63

p=4.3e-111

t=13.20

p=1.5e-39

t=2.10

p=0.036

t=13.55

p=1.7e-41

t=2.86

p=0.0042

t=-2.67

p=0.0076

t=7.83

p=5.3e-15

t=-1.39

p=0.16

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(a) Pairwise t-tests across models on Color Tempera- ture Pairwise Statistical Comparison Extrinsics

CogVideo 5B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Model B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Wan 1B

Model A

t=5.94

p=3e-09

t=17.98

p=2.6e-71

t=12.38

p=5.8e-35

t=2.93

p=0.0034

p=0.0031

t=-15.04

p=1.1e-50

t=14.17

p=3.2e-45

t=8.41

p=4.7e-17

t=-4.11

p=4e-05

t=11.19

p=6.3e-29

t=20.64

p=5e-93

t=15.06

p=8.7e-51

t=2.55

p=0.011

t=17.69

p=4.1e-69

t=6.73

p=1.7e-11

t=10.14

p=4.4e-24

t=4.50

p=7e-06

t=-7.51

p=6.2e-14

t=7.26

p=4e-13

t=-3.61

t=-10.07

p=9.7e-24

t=19.11

p=3e-80

t=13.48

p=4.1e-41

t=0.95

t=16.15

p=4.8e-58

t=5.12

p=3.1e-07

t=-1.63

p=0.1

t=8.52

p=1.8e-17

t=17.82

p=3.9e-70

t=12.07

p=2.2e-33

t=-0.63

p=0.53

t=14.81

p=3.5e-49

t=3.58

p=0.00034

t=-3.26

p=0.0011

t=7.09

p=1.4e-12

t=-1.61

p=0.11

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(b) Pairwise t-tests across models on Lighting Effects

CogVideo 5B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Model B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Wan 1B

t=5.12

p=3.1e-07

t=17.56

p=3e-68

t=12.89

p=8.5e-38

t=2.46

p=0.014

t=-2.71

p=0.0067

t=-15.48

p=1.3e-53

t=16.46

p=2.7e-60

t=11.67

p=2.6e-31

t=-1.49

p=0.14

t=14.32

p=3.5e-46

t=22.24

t=17.70

p=2.8e-69

t=4.65

p=3.3e-06

t=20.27

p=5.4e-90

t=6.26

p=4e-10

t=12.99

p=2.5e-38

t=8.31

p=1e-16

t=-4.10

p=4.1e-05

t=10.85

p=2.5e-27

t=-2.75

p=0.006

t=-8.62

p=7.2e-18

t=21.56

p=2.3e-101

t=17.04

p=1.9e-64

t=4.16

p=3.1e-05

t=19.60

p=2.7e-84

t=5.74

p=9.8e-09

t=-0.43

p=0.67

t=8.11

p=5.5e-16

t=18.48

p=2.8e-75

t=13.76

p=8.6e-43

t=0.60

p=0.55

t=16.39

p=8.1e-60

t=2.14

p=0.033

t=-4.14

p=3.5e-05

t=4.76

p=1.9e-06

t=-3.65

p=0.00026

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(c) Pairwise t-tests across models on Extrinsics

CogVideo 5B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Model B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Wan 1B

Model A

t=8.92

p=5e-19

t=25.14

p=2.3e-137

t=16.48

p=1.2e-60

t=5.05

p=4.5e-07

t=-3.85

p=0.00012

t=-20.17

p=1.3e-89

t=23.38

p=2.1e-119

t=14.66

p=2.2e-48

t=-1.92

p=0.054

t=18.38

p=7.5e-75

t=28.89

p=5.6e-180

t=20.22

p=5e-90

t=3.56

p=0.00037

t=23.90

p=1.5e-124

t=5.53

p=3.3e-08

t=17.14

p=2.1e-65

t=8.72

p=2.9e-18

t=-7.04

p=1.9e-12

t=12.33

p=8.1e-35

t=-5.24

p=1.6e-07

t=-10.51

p=9e-26

t=29.79

p=4.9e-191

t=21.18

p=1.7e-98

t=4.61

p=4e-06

t=24.83

p=3.7e-134

t=6.57

p=5.1e-11

t=1.08

p=0.28

t=11.48

p=2e-30

t=25.63

p=1.3e-142

t=16.88

p=1.6e-63

t=0.20

p=0.84

t=20.61

p=2.1e-93

t=2.15

p=0.032

t=-3.40

p=0.00066

t=7.31

p=2.8e-13

t=-4.47

p=7.9e-06

Pairwise Statistical Comparison

Intrinsics

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(d) Pairwise t-tests across models on Intrinsics

CogVideo 5B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Model B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Wan 1B

Model A

t=6.66

p=2.9e-11

t=23.75

p=4.1e-123

t=17.37

p=4.2e-67

t=3.35

p=0.00082

p=0.0009

t=-20.58

p=3.7e-93

t=21.38

p=2.5e-100

t=14.94

p=3.4e-50

t=-2.48

p=0.013

t=18.17

p=3.3e-73

t=28.50

p=2.4e-175

t=22.15

t=4.67

p=3.1e-06

t=25.35

p=1.1e-139

t=7.17

p=7.6e-13

t=16.22

p=8e-59

p=2.6e-23

t=-6.72

p=1.9e-11

t=13.09

p=5.2e-39

t=-4.36

p=1.3e-05

t=-11.24

p=3.3e-29

t=28.77

p=1.1e-178

t=22.47

t=5.12

p=3.1e-07

t=25.65

t=7.61

p=2.9e-14

t=0.49

p=0.62

t=11.63

p=3.7e-31

t=24.21

p=1e-127

t=17.77

p=4.1e-70

t=0.21

p=0.83

t=21.00

p=5.9e-97

t=2.72

p=0.0066

t=-4.51

p=6.7e-06

t=6.99

p=2.9e-12

t=-4.97

p=6.9e-07

Pairwise Statistical Comparison

Trajectory

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(e) Pairwise t-tests across models on Trajectory

Model A

Model A

Pairwise Statistical Comparison

Class

Pairwise Statistical Comparison

Costume

CogVideo 5B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Model B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Wan 1B

p=1.3e-14

t=18.77

p=2.1e-77

t=10.86

p=2.4e-27

t=-11.08

p=2.1e-28

t=-18.78

p=1.7e-77

t=-30.07

p=3.9e-191

t=19.62

p=2.8e-84

t=11.87

p=2.6e-32

t=1.26

p=0.21

t=30.71

p=8.7e-199

t=26.16

t=18.26

p=2.1e-73

t=7.59

p=3.4e-14

t=37.46

p=2.5e-289

t=6.17

p=7.2e-10

t=14.37

p=2.1e-46

t=6.64

p=3.3e-11

t=-4.09

p=4.4e-05

t=25.45

p=6.7e-139

t=-5.23

p=1.7e-07

t=-11.50

p=2e-30

t=24.40

p=3.7e-128

t=16.44

p=4.8e-60

t=5.66

p=1.6e-08

t=35.76

p=2.5e-265

t=4.26

p=2e-05

t=-1.99

p=0.047

t=9.63

p=7e-22

t=17.17

p=2.9e-65

t=9.27

p=2.2e-20

t=-1.63

p=0.1

t=28.48

p=2.6e-172

t=-2.86

p=0.0043

t=-9.22

p=3.5e-20

t=2.49

p=0.013

t=-7.30

p=3.1e-13

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(a) Pairwise t-tests across models on Subject Class

CogVideo 5B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Model B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Wan 1B

Model A

t=2.25

p=0.025

t=10.52

p=1.2e-25

t=8.27

t=-11.51

p=2.5e-30

t=-13.86

p=4.7e-43

t=-22.57

p=2.1e-108

t=10.90

p=2.2e-27

t=8.70

p=4.1e-18

t=0.66

p=0.51

t=22.65

p=5e-109

t=17.98

p=2.2e-70

t=15.77

p=6.4e-55

t=7.67

p=2e-14

t=30.23

p=5.2e-187

t=6.79

p=1.2e-11

t=8.56

p=1.4e-17

t=6.35

p=2.3e-10

p=0.081

t=20.27

p=2.4e-88

t=-2.34

p=0.019

t=-9.20

p=4.7e-20

t=14.73

p=2.8e-48

t=12.52

p=1.7e-35

t=4.38

p=1.2e-05

t=26.85

p=4.4e-150

t=3.60

p=0.00032

t=-3.25

p=0.0012

p=2.1e-09

t=9.87

p=8.3e-23

t=7.64

p=2.6e-14

t=-0.58

p=0.56

t=21.82

p=1.1e-101

p=0.22

t=-8.19

t=1.16

p=0.24

t=-4.92

p=8.8e-07

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(b) Pairwise t-tests across models on Subjet Costume

CogVideo 5B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Model B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Wan 1B

t=5.90

p=3.9e-09

t=9.11

p=1.3e-19

t=3.00

p=0.0027

p=1.8e-08

t=-11.52

p=3.3e-30

t=-14.90

p=7.4e-49

t=13.82

p=2e-42

t=7.70

p=1.7e-14

t=4.86

p=1.2e-06

t=19.60

p=1.1e-81

t=16.99

p=1.7e-62

t=10.76

p=1.3e-26

t=7.97

p=2e-15

t=22.87

p=9.1e-109

t=3.02

p=0.0026

t=11.43

p=9.3e-30

t=5.42

p=6.2e-08

t=2.55

p=0.011

t=17.11

p=2.5e-63

t=-2.22

p=0.026

t=-5.23

p=1.8e-07

t=16.88

p=9.1e-62

t=10.61

p=5.7e-26

t=7.81

p=7.3e-15

t=22.79

p=3.8e-108

t=2.83

p=0.0047

t=-0.21

p=0.83

t=5.05

p=4.5e-07

t=10.31

p=1.3e-24

t=4.24

p=2.3e-05

t=1.29

p=0.2

t=16.06

p=2.9e-56

t=-3.54

p=0.00041

p=4.5e-11

t=-1.27

p=0.21

t=-6.44

p=1.4e-10

Pairwise Statistical Comparison

Elements

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(c) Pairwise t-tests across models on Elements

CogVideo 5B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Model B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Wan 1B

Model A

t=6.77

p=1.4e-11

t=12.30

p=1.8e-34

t=5.35

p=9.1e-08

t=-10.33

p=7.2e-25

t=-16.90

p=5.1e-63

t=-22.53

p=5.2e-109

t=13.46

p=7.2e-41

t=6.51

p=8e-11

t=1.19

p=0.23

t=23.65

p=1.4e-119

t=20.29

p=3e-89

t=13.22

p=1.8e-39

t=8.01

p=1.3e-15

t=30.43

p=3.1e-192

t=6.79

p=1.2e-11

t=10.97

p=8.9e-28

t=4.18

p=3e-05

t=-1.08

p=0.28

t=21.01

p=2.3e-95

t=-2.24

p=0.025

t=-8.90

p=6.7e-19

t=18.07

p=1.3e-71

t=10.91

p=1.5e-27

t=5.57

p=2.6e-08

t=28.40

p=5.7e-169

t=4.34

t=-2.59

p=0.0096

t=6.54

p=6.7e-11

t=11.23

p=4.5e-29

t=4.21

p=2.6e-05

t=-1.23

p=0.22

t=21.59

p=1.6e-100

t=-2.44

p=0.015

t=-9.35

p=1.2e-20

t=-0.11

p=0.91

t=-6.90

p=5.5e-12

Pairwise Statistical Comparison

Time of Day

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(d) Pairwise t-tests across models on Time of Day

CogVideo 5B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Model B

Hunyuan

LTX Video

Minimax

StepVideo

Wan 14B

Wan 1B

Model A

t=4.41

p=1.1e-05

t=13.63

p=1.2e-41

t=9.16

p=7.3e-20

t=-5.88

p=4.3e-09

t=-10.24

p=2.2e-24

t=-19.43

p=1.9e-81

t=14.48

p=9.7e-47

t=10.03

p=1.7e-23

t=0.94

p=0.35

t=20.24

p=4.1e-88

t=21.25

p=1.8e-96

t=16.73

p=2e-61

t=7.62

p=2.9e-14

t=26.99

p=5.4e-151

t=6.63

p=3.6e-11

t=9.56

p=1.8e-21

t=5.14

p=2.8e-07

t=-3.95

p=7.9e-05

t=15.33

p=5.1e-52

t=-4.86

p=1.2e-06

t=-11.50

p=3e-30

t=18.76

p=2.8e-76

t=14.26

p=2.3e-45

t=5.12

p=3.2e-07

t=24.53

p=2.2e-126

t=4.14

p=3.5e-05

t=-2.51

p=0.012

t=9.02

p=2.5e-19

t=13.68

p=6.5e-42

t=9.19

p=5.4e-20

t=0.00

p=1

t=19.49

p=6.3e-82

t=-0.94

p=0.35

t=-7.64

p=2.5e-14

t=3.97

p=7.4e-05

t=-5.13

p=3e-07

Pairwise Statistical Comparison

Location

Row model is better than Column model (p < 0.05)

Row model is worse than Column model (p < 0.05)

Not statistically significant (p

0.05)

(e) Pairwise t-tests across models on Location

A.7 Additional VLM results

Node-specific results of VLM evaluator Table 10 and 11 list the set of nodes on which our VLM evaluator has the strongest and weakest performance, respectively.

Comparison with Closed Source Models We extend our validation to closed-source, flagship SOTA models. Specifically, we evaluate two recent models from the Gemini family with distinct purposes: Gemini-2.0-Flash, optimized for fast inference, and Gemini-2.5-Pro-Preview-05-06, optimized for complex reasoning. We use the same human-aligned preference accuracy metric as with opensource models. Due to the lack of public details on model sizes, we cannot draw conclusions about scaling effects. However, Gemini-2.5-Pro consistently outperforms open-source models, including QwenVL-2.5-72B, across all categories. Notably, as shown in Figure 29, our 7B model outperforms Gemini-Flash across all categories and performs competitively with Gemini-2.5-Pro. This highlights the strength and scalability of our approach for professional video evaluation.

Reliability in VLMs A reliable VLM-as-a-Judge should produce consistent scores when given the same video, prompt, and focus aspect. In this analysis, we evaluate the raw scores generated by VLMs rather than preference rankings, and measure their stability under Best-of-5 sampling. Since VLMs are probabilistic, we evaluate reliability via the standard deviation of scores across runs. We use temperature=0 to sample to make ensure that the highest probability is selected at each sampling step. We exclude our model from this analysis, as its architecture includes a dedicated value head, unlike zero-shot VLMs that produce rewards as text. Our results show that Qwen-2.5VL-3B exhibits a high variance, making it unreliable under repeated sampling. In contrast, the flagship models and the strongest open-source model, QwenVL-2.5-72B, demonstrate high reliability, with consistently low variance (Table 12).

Total

Events

Setup

Camera

Lighting

0

20

40

60

80

100

Accuracy (%)

Zero-Shot VLMs vs Our Model's Human Preference Accuracy

Models

Qwen2.5-VL-7B

Qwen2.5-VL-32B

Gemini 2.0 Flash

Qwen2.5-VL-72B

Ours (7B)

Gemini-2.5-Pro-Preview-05-06

Category

A.8 Additional Results on Recent Models

We also perform small-scale evaluations on recently released models, specifically Veo 3 (Fast) [9] and the Wan 2.2 family of models. Results are presented in Table 13.

A.9 Broader Impact

We hope SCINE encourages the generative AI and computer vision communities to engage more deeply with the elements required to produce a professional cinematic shot. While our taxonomy is currently used for evaluation, it also offers a structured foundation for broader tasks such as captioning, creating control aware video datasets, and guiding model training toward explicit cinematic intent. We also envision that our fine-grained control nodes can drive new directions in open computer vision problems, such as estimating camera intrinsics from video, inferring lighting position, and understanding frame compositionality. In conclusion, we see our taxonomy as a first step toward systematically understanding cinematic controls in generative models, and we are hopeful it will be meaningfully adopted and extended across both generative AI and filmmaking communities.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article introduces SCINE (Stable Cinemetrics) - an evaluation suite specifically designed to assess the quality of video generation from the perspective of professional video production. The evaluation is divided into four major categories: Setup, Event, Lighting, and Camera, each of which contains numerous subcategories, totaling 76 subcategories. Each subcategory was discussed and determined by professional video creators, making it more professional compared to other video evaluation benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**  \n1. The evaluation system was established through a relatively professional process, which is a task with a high barrier to entry.\n\n**Weaknesses:**  \n1. Although the article claims that the construction process of SCINE is highly professional, the data provided in the supplementary materials suggests otherwise. First, the `ranked_videos` folder contains two subfolders, `good_examples` and `bad_examples`, but the videos in these folders are not paired, and their prompts are not the same, which contradicts the claims made in the paper. Additionally, the evaluation prompts constructed are not professional. For instance, in the `camera trajectory` category of prompts in the supplementary materials, most prompts describe a static camera, which is overly simplistic for assessing camera movement.  \n2. The supplementary materials do not provide the scores for the human annotation videos, making it impossible to determine the accuracy of the scoring.  \n3. The training process of the evaluation model is simple and entirely based on existing work.  \n4. The main text does not include any visualized presentations of videos, nor does it use visualized examples to explain the necessity of the 76 categories.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nAlthough a relatively detailed classification is constructed, the prompts for each category do not reflect professionalism and uniqueness. Comparing the professionalism claimed by the author in the main text with the examples in the supplementary materials, there is a suspicion of overclaim.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Stable Cinemetrics (SCINE), a structured evaluation framework based on professional filmmaking principles and standards. SCINE includes 4 hierarchical taxonomies (Setup, Event, Lighting, and Camera) to evaluate text-to-video (T2V) models. This paper evaluates 10+ models across 20K generated 60 videos with feedback from 80+ professionals.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Quality and Clarity: This paper, espically its taxonomies, is clearly written and technically sound.\n2. This paper formally introduces professional filmmaking standards into the T2V evaluation process, a aspect that has been previously explored (e.g., in Wan) but not systematically assessed. The proposed taxonomies and evaluation framework, based on professional feedback, are original contributions.\n\nWeaknesses:\nThe scope of this paper is relatively limited to the following aspects:\n1. Evaluation is limited to controllability (L106): The evaluation taxonomies only evaluate controllability but neglect the overall video quality. There remain fundamental flaws in the generated outputs, such as distortions in human bodies or hands, object movements that violate physical laws, and poor imagery quality. These aspects are critical for professional video generation but are largely overlooked in this paper.\n2. Evaluation is limited to an independent shot: filmmaking involves more than generating a single independent shot; it requires character consistency across different scenes. Character identity is nearly impossible to control through text alone, meaning that aspects such as character consistency are also overlooked in the evaluation. In application, this issue may be even more important than the aspects evaluated in the paper.\n3. Evaluation is limited to text-to-video: this paper only evaluates the text-to-video (T2V) aspect because T2V is more suitable to be evaluated by the proposed taxonomies (L73). It remains unclear whether T2V models are the primary workflow used in professional settings. This questions the significance of this work.\n4. I2V models are also widely adopted by the community due to their greater flexibility in control through reference images. Can the evaluation framework proposed be adapted to evaluate I2V model and pipelines?\n\nQUESTIONS:\nsee weaknesses\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Stable Cinemetrics (SCINE), a structured evaluation framework for professional video generation that organizes filmmaking principles into four hierarchical taxonomies (Setup, Events, Lighting, and Camera) with 76 fine-grained control nodes. The authors conduct extensive human evaluation with 80+ film professionals assessing 20,000 videos generated by 10+ text-to-video models, revealing that current models struggle most with Events and Camera controls while performing better on Setup and Lighting elements. To enable scalable evaluation, they fine-tune a vision-language model that achieves 72.36% alignment with human judgments, though its testing on same-distribution data raises questions about generalization capabilities. Despite limitations including the absence of visual examples and limited cross-domain testing, SCINE provides a valuable foundation for aligning generative video models with professional production standards.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Pioneering Professional Video Evaluation Benchmark: Introduces SCINE, an innovative evaluation framework that formalizes filmmaking principles into four disentangled taxonomies (Setup, Event, Lighting, Camera), addressing a critical gap in current T2V model evaluation for professional use.\n2. Large-Scale Expert Human Annotation: Conducts a comprehensive human study with over 80 film professionals annotating 20K videos, providing high-quality, industry-aligned ground truth for model performance.\n\nWeaknesses:\n\n1. Questionable Generalization of Automatic Evaluator: The trained VLM evaluator's generalization is unproven, as it's not validated on real-world videos or new, unseen generative models, which is crucial for a benchmark's universality.\n2. Quality Issues in Supplementary Examples: \"Good case\" examples in the supplementary material (e.g., ltx_00569.mp4) are unrealistic, undermining the claimed alignment with professional standards and raising further doubts about the evaluation method's generalization.\n3. Lack of Open-Source Commitment: The absence of a clear commitment to open-source the proposed data and models significantly hinders reproducibility and wider community adoption of the proposed dataset, thereby diminishing the paper's overall practical impact.\n\nQUESTIONS:\n1. Could additional experiments be conducted to verify the generalization ability of the automatic evaluator on videos generated by other models or on real-world videos?\n2. The paper does not explicitly state whether the data and models will be open-sourced. Given that this is an evaluation benchmark, open-sourcing is crucial for community reproduction and further research. Do the authors have plans for open-sourcing?\n3. Some videos labeled as 'good cases' in the supplementary materials (e.g., ltx_00569.mp4) clearly do not conform to real-world principles, which raises questions about the rigor of the evaluation criteria. Could the authors provide a more detailed explanation of the judgment criteria for these cases, or consider re-evaluating them?\n\nLIMITATIONS:\nSee Weakness above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Stable Cinemetrics (SCINE), an evaluation framework designed to assess the readiness of current video generative models for professional use. SCINE organizes filmmaking principles into four hierarchical taxonomies—Setup, Events, Lighting, and Camera—covering 76 fine-grained control nodes based on industry practices. It creates a benchmark with story-driven and visually rich prompts to mirror professional workflows, enabling automated prompt categorization and targeted question generation for detailed evaluation. A human study involving 10+ models, 20K videos, and 80+ film professionals reveals significant performance gaps, particularly in Events and Camera controls. To support scalable evaluation, the authors developed a vision-language model (VLM) aligned with expert annotations, outperforming existing baselines with a 72.36% accuracy.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. Existing benchmarks lack the detailed, shot-level structure and cinematic depth required for professional filmmaking, as exemplified by a simple VBench prompt (\"A man is walking\") that omits crucial details like character appearance or camera movement. In contrast, SCINE's taxonomy-guided approach enables scalable and future-proof evaluation, adapting as model capabilities evolve beyond the limitations of static, fixed-prompt benchmarks.\n\n2. The study involves over 80 film professionals, including independent cinematographers, screenwriters, and an Academy Award-winning Visual Effects Artist. The taxonomy development is driven by the central question: \"What controls do professionals require when setting up a shot?\" ensuring a practical, industry-relevant framework.\n\n3. The taxonomy provides both coarse insights, revealing that models struggle most with Events and Camera, and fine-grained comparisons, such as better performance on shot size over camera framing and natural lighting over artificial lighting.\n\n4. Detailed information is provided, including Taxonomy Details, the User Interface used by annotators for evaluations, and instructions given to the LLM for prompt generation, enhancing the transparency and reproducibility of the methodology.\n\n### Weaknesses\n1. Not enough evaluated generated videos with evaluation results are presented. The supplemental material contains some ranked videos, but it is not organized in a way that is easy for readers to interpret. Side-by-side comparisons of different T2V models using the same prompts would significantly improve clarity.\n\n2.  Certain metrics, such as camera parameters, movement, depth, and contrast, are not subjected to rigorous, objective quantitative assessments. Incorporating precise numerical analysis for camera movement, depth, and contrast, etc. alongside human ratings would create a more balanced and reliable evaluation.\n\nQUESTIONS:\n1. Given that SCINE's current evaluation operates at the shot level, how might the framework be extended to evaluate multi-shot sequences? \n\n2. What are the most common failure modes of the VLM evaluator? Would its computational cost be a burden?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}