ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models
Authors: Hongbo Liu
Jingwen He
Yi Jinn
Dian Zheng
Yuhao Dong
Fan Zhang
Ziqi Huang
Yinan He
Weichao Chen
Yu Qiao
Wanli Ouyang
Shengjie Zhao
Ziwei Liu
Venue: NeurIPS 2025 poster ()
Keywords: MLLM
cinematography understanding
VLM
Primary area: deep_learning
TL;DR: We present ShotBench, a new benchmark for evaluating VLMs' cinematography understanding, along with the ShotQA dataset and our ShotVL model, which achieves state-of-the-art performance over both strong open-source and proprietary baselines.

Abstract: Recent Vision-Language Models (VLMs) have shown strong performance in general-purpose visual understanding and reasoning, but their ability to comprehend the visual grammar of movie shots remains underexplored and insufficiently evaluated. To bridge this gap, we present \textbf{ShotBench}, a dedicated benchmark for assessing VLMs’ understanding of cinematic language. ShotBench includes 3,049 still images and 500 video clips drawn from more than 200 films, with each sample annotated by trained annotators or curated from professional cinematography resources, resulting in 3,608 high-quality question-answer pairs. We conduct a comprehensive evaluation of over 20 state-of-the-art VLMs across eight core cinematography dimensions. Our analysis reveals clear limitations in fine-grained perception and cinematic reasoning of current VLMs. To improve VLMs capability in cinematography understanding, we construct a large-scale multimodal dataset, named ShotQA, which contains about 70k Question-Answer pairs derived from movie shots. 
Besides, we propose ShotVL and train this VLM model with a two-stage training strategy, integrating both supervised fine-tuning and Group Relative Policy Optimization (GRPO). Experimental results demonstrate that our model achieves substantial improvements, surpassing all existing strongest open-source and proprietary models evaluated on ShotBench, establishing a new state-of-the-art performance.

ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models
Hongbo Liu Jingwen H Yi Jin' Dian Zheng?
Yuhao Dong' Fan Zhang? Ziqi Huang Yinan He
Weichao Chen Yu Qiao² Wanli Ouyang Shengjie Zhao Ziwei Liu"t
TTngji University, Shanghai Artificial Intelligence Laboratory. The Chinese University of Hong Kong, *S-Lab, Nanyang Technological University 'Equal contribution. 'Correpponing authors.
Project Page: httpps//vchitectt. wecpiittettt..tteee. github. io/ Sww.SStececcpoojeeeeeeeee
Abstract
Cinematography, the fundamental visual language of film, is essential for conveying narrative, emotion, and aesthetic quality. While recent Vision-Languaae Models (VLMs) demonstrate strong general visual understanding, their proficiency in comprehending the nuanced cinematic grammar embedded within individual shots remains largely unexplored and lacks robust evaluation. This critical gap limits both fine-grained visual comprehension and the precision of Al-assisted video generation. To address this, introduce ShotBench, comprehensive benchmark specifically designed for cinematic language understanding. It features over 3.5k expert-annotated QA pairs from images and video clips, meticulously curated from over 200 acclaimed (predominantly Oscar- (ecar-nominate) films and spanning eight key cinematography dimensions. Our evaluation of 24 leading VLMs on ShotBench reveals their substantial limitations: even the top cop-performing model achieves less than 60% average accuracy, particularly struggling with fine-gres grained visual cues and complex spatial reasoning. To catalyze advancement in this domain, we construct ShotQA, a large-scale multimodal dataset comprising approximately 70k cinematic QA pairs. Leveraging ShotQA, we develop Sh through supervised fine-tuning and Group Relative Policy Optimization. ShotVL significantly outperforms all existing open-source and proprietary models ShotBench, establishing new state- of-the-art performance. We open- open-source our models, data, and code to foster rapid progress in this crucial area of Al-driven cinematic understanding and generation.
Introduction
Cinematography, the art crafting visual narratives through meticulously designed shots 44 forms the bedrock of high-quality filmmaking. Each shot, from framing and lens choice to light- ing and camera movement, deliberately composed to convey narrative meaning, emotional tone, and aesthetic impact. For text-to-iman -itt-too-mme//vdeoo generation
40 59 to achieve similar cinematic quality, it requires a mechanism capable of understanding these cinematographic princi- the primary candidates for developing such understanding. Thus, the core challenge is whether current VLMS can genuinely grasp the nuanced language of cinematography and its artistic intent, moving beyond literal scene interpretation. This deep cinematographic comprehension remains significantly underexplored. Ex- isting VLM benchmarks, while diverse [8 31 typically lack the necessary focus for robust cinematographic evaluation, gap exacerbated by a scarcity of specialized models, datasets with rich cinematic annotations, and consequently, rigorous benchmarks for this specific type of understanding.
2
23
24
ples. Vision-Langura
Models (VLMs) 3 26
33
35
48
61
65
To bridge this critical gap, we introduce ShotBench, a comprehensive benchmark specifically designed to assess VLMs understanding of cinematic language. ShotBench comprises over 3.5k
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

T10
Lagaties 1ope
Lighter Cxnditin
Camposton
Stat Sise
Bas tack
Midkar Since
Shat
Size
Stado ode Bu darddee
Lov
Ulex vidu Falicys
Overview ShotBench. The benchmark covers eight core dimensions cinematography: shot size, framing, camera angle, lens size, lighting type, lighting condition, composition, and camera movement.
Figure 1:
expert-annotated multiple-choice QA examples, meticulously curated from both images and video clips across over 200 films, predominantly those that have received Oscar nominations for Best Cinematography! It rigorously spans eight fundamental cinematography dimensions: shot size, shot framing, camera angle, lens size, lighting type, lighting condition, composition, and camera movement. Our rigorous annotation pipeline, combining trained annotators with expert oversight, ensures sigh-quality evaluation set grounded in professional cinematic knowledge.
[
We conduct an extensive evaluation of 24 leading open-source and proprietary VLMs on ShotBench. Our results reveal that even the strongest VLM (GPT-4o (38) in our evaluation averages below 60% accuracy, clearly indicating a considerable gap between current VL. capabilities and genuine cinematographic comprehension. In-depth analysis further highlights specific weaknesses: advanced models such as GPT-4o (38) and Qwen2.5-VL (3, despite grasping core cinematic concepts, often struggle map subtle visual details to precise professional terminology (c.g.. distinguishing medium shot from a medium close-up). They also demonstrate constrained spatial reasoning, especially regarding camera position and angle. Strikingly, the camera movement dimension proved exceptionally challenging, with over half of the models failing to surpass 40% accuracy.
To further advance cinematography understanding in VLMs, construct ShotQA, the first large- scale multimodal dataset for cinematic language understanding, consisting of approximately 70k high-quality QA pairs derived from movie images and video clips. Leveraging ShorQA, we develop ShotVL., an optimized VI.M series based on Qwen2.5-VI-333 and Qwen2.5-VL.-2 [3, trained with supervised fine-tuning and Group Relative Policy Optimization (GRPO) 446 enhance its
alignment of visual features with cinematography knowledge and strengthen its reasoning capabilities. Experimental results demonstrate that ShotVL achieves consistent and substantial improvements across all ShotBench dimensions, establishing new state-of-the-art performance and decisively surpassing both the best perforrergg open-source (Qwen2.5-VL-72B Instruct and proprietary (GPT-4o (38) models.
Our contributions are summarized follows:
We introduce ShotBench, a comprehensive benchmark for evaluating VLMs' understanding of cinematic language. It comprises over 3.5k expert-annotated QA pairs derived from images and video clips of over 200 critically acclaimed films (predominantly Dscar-nominated), covering eight distinct cinematography dimensions. This provides rigorous new standard for assessing fine-grate imeeggraieed visual comprehension in film.
'http:://en htagss//ee.wkkiieciiaggggwktllee
for Best_ Cinematography

---

We conducted an extensive evaluation of 24 leading VLMs, including prominent open source and proprietary models, on ShotBench. Our results reveal critical performance gap: even the most capable model, GPT-4o, achieves less than 60% average accuracy. This systematically quantifies the current limitations of VL.Ms genuine cinematographic comprehension.
To address the identified limitations and facilitate future research, constructed ShotQA, the first large-scale multimodal dataset for cinematography understanding. containing approximately 70k high-quality QA pairs. Leveraging ShotQA, developed ShofVL, novel VLM series trained using Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). ShotVL significantly surpasses all tested open-source and proprictary models, establishing new state-of-the-ar on ShotBench.
2 Related Work
2.1 Benchmarking Vision-Language Models
61 65 are large-scale models designed to integrate visual perception with natural language understanding. In recent years, VLMs have demon- strated strong capabilities across perception, reasoning, and a wide range of multi-disciplinary 57 Recently, researchers have proposed a variety of benchmarks to assess VLMs' capability. For example, MMBench [3] evaluates VLMs across 20 dis- tinct ability dimensions, and MMVU 663 focuses on video understanding across four core academic disciplines. Other benchmarks target specific cognitive or reasoning capacities: Logic Vista [54] assesses visual logical reasoning multi-choic format, and SPACE [41] systematically compare spatial reasoning abilities between VL.Ms and animals. Additional efforts, EgoSchema (37], and VSI-Bench [55], evaluate egocentric video understanding. Moreover, some works introduce tasks with specific domains, such as scientific and mathematical figure interpretation [4 [33 knowledge acquisition 21 and visual coding (60]
Vision-Language Models (VLMs) 3
26 33
35
48
applications [
Cinematography Understanding
Early work on automatic film analysis includes many sub-tasks such as shot type classification scene segmentation 43
42
and cut type recognition [39]] For example, MovieShots
categories shots into five scale types and four camera movements types, providing early taxonomy for cinematography understanding. With the rapid progress image and video generation, film-level
generation has begun to attract increasing attention 11
23
24
40
59 Many recent works rely
on VLMs to synthesise large training corpora 23 [5]] However, they often introduce additional
classifiers to identify camera movements shot sizes. For example, Hunyuan Video [23] trains camera movement classifier capable of predicting distinct camera movement types, introducing additional training and data annotation overhead.
Table 1: Cinematography Understanding Benchmark Comparison
Dimeusions
MovieShots 42
MovieNet 22
CineSeale2 45
CameraBench 29
Cine TechBench
52
ShotBench
Shot size
V
Shot froming
X
Comera angle
X
X
Lens size
X
X
Lighting type
Lighting condition
Composition
Comera movement
3 ShotBench
To evaluate the capabilities of VLMs on cinematography understanding, we first define the concept of cinematography understanding and introduce ShotBench in 3.]] Next, we provide a detailed description of the data collection process in 3.2 Using ShotBench, we then perform evaluations to assess whether VI.Ms can effectively comprehend cinematic conventions and analyze potential
causes of their performance limitations in 3.3
3

---

3.1 Overview
Data Curation Pre-processing
Annotator Traiinn
QA Annotation
Verification
Quality Auaeeciioaiyy
Colletiay Hish
Marinn Crassig and Soymeniing
Consorbessiee refirncce motwisk & Profzstrrl Iutonab
Malioun/pak cenerotioe
Ao luuge N00
For video doto
of/val C4
Conttrutiigg Jazt bondhwarrt
chamsst alots
Authall Cnpy Kilei Yarr
G24 vaaly Sughi righ quiling
Figure 2: An overview of the ShotBench construction pipeline.
Understanding cinematography involves not only identifying visual elements like framing, lighting, and camera movement, but also interpreting how they work together to convey narrative and mood. While recent VLMs show some ability to recognize cinematic language, their deeper understanding of cinematic conventions remains underexplored. Here, we introduce ShotBench, a dedicated benchmark designed to evaluate VLMs' understanding of cinematography language in comprehensive and structured manner. ShotBench covers eight core dimensions commonly used in cinematic analysis: shot size, shot framing, camera angle, lens size, lighring rype, lighting conditions, composition, and camera movement. These dimensions reflect key principles of visual storytelling in film production and serve as the foundation for evaluating model comprehension.
Each sample ShotBench is paired with a multiple-choice question targeting specific cinematogra- phy aspect, requiring the model to not only perceive the scene holistically, but also extract fine-grained visual cues to reason about the underlying cinematic techniques. An overview of the benchmark framework is illustrated in Figure I
3.2 Data Construction Process
To construct. ShotBench, we design systematic data collection and processing pipeline, as illustrated in Figure 2 The process consists of four key stages: Data Curation & Pre-processing, Annotator Training, QA Annotation, and Verification.
Data Curation & Pre-processing We collect the dataset primarily from films that won or were nominated for the Academy Award for Best Cinematography, ensuring high-quality and professionally crafted shot. Data are sourced from public websites and include high-resolution images and video clips. To ensure quality and safety, we apply the L.AION aesthetic predictor (25) for filtering low- quality samples, NSFW detection [50) to remove inappropriate content, and FFmpeg [18) to crop black bars. For video processing, we use TransNetV2 [47] to segment footage into individual shots.
The full list of collected movies is provided in Appendix
Annotator Training To ensure high-quality annotations, we first curated comprehensive refer- ence materials from publicly available cinematography tutorials covering all eight dimensions in ShotBench. Annotators were required to study these materials before labeling. We then conducted multi-round pilot annotations, supported by expert audits and daily discussions resolve ambiguities. All issues and resolutions were documented to guide the final annotation phase.
QA Annotation
Based on ShotBench's predefined dimensions, automatically generated question prompts using templated formats (e.g.. "Whal is the shot size of this movie shot?"). We ensured an even distribution of questions across the eight dimensions, illustrated in Appendix C (Figuree 15b). For image data, we extracted candidate labels from Shotdeck 3) professional cinematography reference platform, where metadata had been curated by experienced photographers. Annotators verified these labels against ShotBench guidelines and corrected any discrepancies. All label modifications were reviewed by experts. For videos, annotators identified all valid camera movement intervals by marking start and end timestamps.
Verification All question-answer pairs were reviewed through multiple expert audits, with batches revised iteratively until reaching satisfactory quality. Through this rigorous pipeline, we further sampled from the validated data to construct the final benchmark, consisting of 3,049 images and
hntpss//shotdeck.com

---

Camera Angle
InterMM13789 388
Lens
Shot tramiig
Size
InterMM33146 148
IntemVL3 88
51.4
Intemvi3 29
5wen2.5 VL-229
Lightin Type
Shot
0wen2.5 5 32.
5.
53.8
Qwen2.5 39
InternVL3
Condittian
Movement
VILA-155 88
35.1
Qwen2.554 VL
VILA-15-31
14.4
VILA-1.5
Composition
s6 Overall performance
Owen25
Gerire 2faehhharrssiwwwae
Figure 3: Overall performance comparison of In- tern VL.3, Qwen2.5-VL, and VILA-1.5 model families, highlighting variations by model size. results con- sistently show that larger models within each series gen- erally yield superior performance outcomes.
Figure 4: Performance evaluation of six Vision Language Models (VLMs) cinematographic derstanding, visualized across several dimensions. Stronger models perform well uniformly, without spe- cific dimensional weaknesses.
464 video clips, resulting in 3,572 high-quality question-answer pairs across all eight ShotBench dimensions.
3.3 Evaluation
Setup To provide comprehensive assessment of the challenges posed by ShotBench and establish reference baselines for future research, we evaluate diverse set of state-of-the-art multimodal
evaluate total of 24 foundation models, including both open open-source and proprictary models: Qwen2.5-VL [3, LLaVA
foundation models that support video multi-image inputs. Specifically,
Video 62
ALaVA-Onee Vision [26]. Intern VL-2.5 & 77 65]. IntemLM-XCCopose
[4].
Ovis2
36
VILA1.5 (28), InstructBI.I 9 and Gemini-2.0 & 2.5
122 13 All ShotBench questions
designed four-option single-choice questions. For questions involving multiple keywords (e.g., lighting condition), each option contains the same number of keywords to maintain balance. Only the correct option includes all the correct keywords, while the distractors may contain mixture of correct and incorrect keywords to enhance the challenge and maintain fairness. Specifically, To ensure fairness and reproducibility, we adopt the VLMEvalKit [16] framework for standardized evaluation. We report accuracy as the primary metric to quantify model performance on ShotBench. Additional implementation details and evaluation prompts are provided in the Appendix
Results and Findings The evaluation results reported in Table yield several key findings: (1) Approximately half of the evaluated models attain an overall accuracy below 50%. Even the leading model, GPT-4o, fails to reach 60% accuracy, underscoring the significant gap between current VLMs and a true understanding of cinematography. (2) The overall performance differences between open- source and proprictary models marginal. Notably, Dwen2.5-VL-722 S-5-VL-72B-Instrruee In-tructtt (59.1% 1)) achieves almost the same performance as GPT-40 (59.3%) (3) The camera movement dimension represents partieular area of weakness across current models, with achieved accuracy often approximating random selection (around 25%). (4) Within each series, larger models generally achieve higher accuracy (as shown in Figure 3. suggesting a potential scaling effect with respect to model size in cinematography language understanding.
To better understand the limitations of current VLMs in cinematic language understanding, conduct extensive quantitative and qualitative analyses on the prediction results of representative models. Our analysis reveals significant challenges for current models across three core aspects: (1) fine-graine grainer visual-terminole terminology alignment, (2) spatial perception of camera position and orientation, and (3) visual reasoning in cinematography.
Fine-Grained Visual-Terminology Alignment Through extensive case studies, we find that cur- rent VLMs frequently fail to precisely align visual cues with specific cinematic terms, particularly when the task requires expert-level distinctions. Such shortcomings are especially evident in di- mensions like shot size and lens size, where categories are defined by fine-grained framing or focal length conventions. For example, Medium Wide Shot (MWS) typically frames the subject from
5

---

Table Evaluation results for VLMs. Abbreviations adopted: SS for Shot Size; SF for Shot Framing: for Camera Angie; LS for Lens Size; LT for Lighting Type; LC for Lighting Conditions; SC for Shot Composition: CM for Camera Movement. Bold indicates the best result, and underline indicates the second best in cach group.
Models
8s
SF
CA
L.
LTT
L.C
SC
CM
Avg
Open-Sourced VLMs
Qwen2.5-VI 1-12-------B.. Instimac
54.6
56.6
43.1
36.6
59.3
45.1
41.5
31.9
46.
Owen225-VL-7B-Instee
69.1
73.5
53.2
47.0
60.5
47.4
49.9
30.2
53.8
LLaVA-NeXT-Vidoon
02
35.9
37.1
32.5
27.8
50.9
31.7
28.0
31.3
34.4
VLavA-Vidoo-75 Qwen2
37.4
LLaVA Onevision D0ven2-7B-Ov-hhal 26
58.4
71.0
52.3
38.7
59.5
44,9
50.9
39.7
51.9
IntemWV.2.5-88 X
56.3
70.3
60.2
50.1
33.6
LntemVL3-2B 05
56.3
56.0
44.4
34.6
56.8
44.6
43.0
38.1
46.7
IntemV113-8B 61
62.1
65.8
46.8
42.9
58.0
44.3
46.8
44.2
51.4
IntemVL33148 65
59.6
82.2
44.6
51.1
38.2
Incrminnxcooppseedd--7 14
51.1
71.0
39.8
32.7
59.3
35.7
35.7
38.8
45.5
Ovis2-8B [6
35,9
28.0
VII.AI.5-BB AX
33.4
44.9
32.1
28.6
50.6
35.7
28.4
21.5
34.4
VILAI.5-8B
40.6
44.5
39.1
29.7
48.9
32,9
34.4
36.9
VILAII5-13B 28
36.7
54.6
40.7
34.8
52.8
35.4
34,2
31.3
40.1
Instmaablip-rrccua-778
0
27.0
27.9
34,5
29.4
44.4
29.7
27.1
25.0
30.6
Insttuctbippxiicun-138
26.8
39.0
IntemnVL...53883 7
67.8
85.4
55.4
41.7
61.7
48,9
52.4
44.0
$7.2
65
84.0
46,9
54.7
Qwea2.5-VL-S2B--nntnn
62.3
76.6
51.0
48.3
61.7
44.0
52.2
43.8
55.0
Owen2.5-VL-722-Immee
Intern VB378V 65
69.7
80.0
54.5
44.0
65.5
47.4
51.8
44.4
$7.2
Proprictury VLMS
Gemini-2.0-Baa [2]
48.9
75.5
44.6
31.9
62.2
48.9
52.4
47.4
51.5
Gemini 2.5 S---laab---eveeeee 1review-04-17 04-17
57.7
82.9
51.4
43.8
65.2
45.7
45.9
43.5
54.5
GPT-4o
69.3
83.1
63.2
48.0
55.2
48.
Confesion Mutric: Shot Bize (%)
Confusion
68.9%
17.2%
11.5%
2.5%
9.1%
67.2%
26.1%
-00
Trak L2b4 (27)
0.0%
3.0%
57
Mad
33.9%
25.4%
39.8%
0.8%
Toue Labal
0.0%
0.0%
36.2
53.8%
10.1%
0.0%
0.0%
-00
M005
0.0%
0.0%
2.0%
10.3%
79.4%
4.4%
2.0%
11.2%
10.3%
72.4%
6.0%
200
0.0%
0.0%
1.5%
4,4%
23.5%
70.6%
2.8%
5.7%
4.9%
61.5%
27.9%
6998
0.0%
1.4%
0.0%
0.0%
2.9%
26.7%
70.0%
Predicted
Prodicted Label
Figure Confusion matrices GPT-4o" predictions on shot (left) and lens (right).
the knees up, while a Medium Shot (MS) frames from the waist up. Regarding lens size, U1 tra Wide offer broader field of view and often introduce edge distortion, whereas Long Lens compress spatial depth, making the foreground and background elements appear closer. We draw the confusion matrices based results of GPT-4o, shown Figure It reveals that most misclassifications occur between visually adjacent categories. For instance, MS is frequently confused with MCU (36.2%) or MWS (10.1%), and Medium lens is often misclassified as Wide or Long lens.
These findings suggest that current VLMs lack fine- ome-grained alignment needed to reliably distinguish between visually similar but semantically distinct categories. A plausible explanation is that the training data used for these models may lack sufficient annotation granularity or consistency in cinematography labeling, limiting their ability to internalize professionalllevel distinctions.
Spatial Perception Camera Position and Orientation ShotBench systematically evaluates this ability at the level of cinematic language, covering concepts such camera angle, position, and focal

---

Canpollir
Shartsak
Long
Modiam Wile
GFC-4o
Lorr pagl
GPC-4e
Wider
GBT-46
Cuase
VVL
Qweel X3
Queeli 339
Cleraay
VILAL--3
VILA15-0
Wide
VILAI 5:30
Right bury
Lighting by
Liktnng
anlicua
Puubie
GPT
Oceredt
CPT
Buckide
VILAE 118
Dayligh
STLAI 5: 13B
Buckiiht
Doem
Figure 6: Examples of failure cases where VLMs struggle with fine-grained visual teruinoormmnolgg alignment. spatial perception, and visual reasoning.
length changes. ShotBench evaluates both static and dynamic camera attributes. For static scenarios, models are tested on fixed-angle concepts such as low angle and high angle. For dynamic cases, the camera movement dimension probes a model's i ability to recognize changes in position (e.g., pull out), angle (e.g., tilt up), and focal length (e.g., zoom in). Results show that even the best-performing model, GPT-4o, achieves only 58.2% accuracy on static camera angle recognition, indicating struggle in perceiving and reasoning the camera orientation in the space. The situation is worse for camera movements, where more than half of the evaluated models fall below 40%. substantially lower than their performance on other ShotBench dimensions.
Our case study analysis reveals that frontier models often show solid textual understanding of camera- related terms. However, they often fail make correct predictions in practice. For instance, almost no existing model successfully distinguishes between position change (push in) and focal length change (zoom in), task that requires perceiving parallax (6 (sccond row, third column). Besides, even identifying a high angle may result in incorrect predictions due to misperception of camera height and tilt (Figure 6 top row, second case), not to mention the change of orientation (b (second row, third column).
Visual Reasoning in Cinematography We observe that understanding some dimensions might need VLM to reason like a cinematography expert. For example, recognizing a short side composition (Figure second row, third column) requires the model infer the subject's gaze direction relative to their frame position-a subtle yet important cuc. Similarly, identifying fluorescent lighting may involve reasoning the light source based on the subject's color tone, apparent color temperature, and the direction and softness of shadows in the scene (Figure O second row, first column). We hypothesize that reasoning processes can help VLMs attend to critical visual details relevant to cinematic semantics such as spatial reasoning for determining camera angle or lens size, identifying camera movement from the motion of elements within the frame, and even discerning the director's intent in guiding the viewer's 's attention through compositional choices. We provide quantitative and qualitative analyses Section 55. and Appendix A.2 A.2 Our findings suggest that encouraging VLMMs to engage in structured reasoning provides noticeable improvements in their ability to understand cinematic language.
ShotQA & ShotVL: Advancing Cinematography Understanding via Targeted Training
To address the nuanced challenge enabling Visual Language Models (VLMs) to perceive and reason about cinematic elements, we introduce ShotQA, a novel large-scale dataset, and ShotVL, VLM series specifically designed for cinematography understanding. ShotVL. employs a strategic two-stage training pipeline: initial large-scale Supervised Fine-tuning (SFT) for broad knowledge
7

---

acquisition, followed by Group Relative Policy Optimization (GRPO) 46 for fine- tineggrined reasoning refinmeent on a curated subset.
ShotQA: Dedicated Dataset for Cinematography Comprehension. ShotQA stands as the first large-scale dataset meticulously designed to henchmark and enhance VLMSs' grasp of cinematographic techniques. It comprises 58k images and 1.2k video clips. These resources are sourced from 243 diverse films ensure broad coverage of cinematic styles. All samples are formatted as multi-choice QA pairs, facilitating structured evaluation and targeted training. Each entry is enriched with metadata, including film title and source clip timestamp, allowing for contextual understanding. Table 9 details the sample distribution, revealing a noteworthy balance across most cinematic dimensions. The scale and specificity of ShotQA provide a critical resource for advancing research in this domain.
Stage I: Large-scale Supervised Fine-tuning for Foundational Alignment. In the foundational first stage, ShotVL. undergoes SFT using approximately 70k QA pairs sampled from the ShotQA dataset. We utilize Qwen-2.5-VI. or video alongside a question and multiple-choice options, and is trained directly predict the correct answer via cross-entropy loss. This SFT phase is crucial for establishing strong alignment between visual features and specific cinematic terminology, equipping the model with broad understanding of cinematographic concepts.
3 as the base model. The model processes an image
Stage 2: Reinforcement Learning with GRPO for Enhanced Reasoning, Building upon the SFT- initialized model, the second stage employs GRPO to further elevate ShotVL's reasoning capabilities and prediction accuracy.
Given multimodal input x (an image/video and textual query), GRPO generates G distinct responses {o,,..., oG) from the current policy Te.a- These are evaluated using rule-based binary reward function, inspired by prior work [20 32 51::
11, 0, otherwise.
if o is correct (matches the ground truth).
r(o, x)
(I)
Following DeepSeek-R (20), our reward incorporates two components: (1) a format reward to ensure outputs adhere to structured pattern (<think>.. </think>> and <answer>.. </answer> tags), and (2) an accuracy reward comparing the extracted answer from the <answer> block with the ground truth.
The advantage A, for the i-th response is calculated by normalizing its reward within the group:
A r: mean (ri....,,. rG)) (where 8 a small constant for numerical stability, e.g., le 8). Finally, GRPO optimizes the policy Ty by maximizing the objective:
(2)
std (ri,, ,G]]. r + 8
COROO Σ min
(o:(2))) Tea (otlr)
A, clip
To(0)|0)) Ta (o,(ar)
e, +e) A
Here, is a hyperparameter controlling the policy update step size, and the clipping mechanism stabilizes training. For this RL phase, we utilize a focused subset of approximately 8k high-quality multiple-choice QA instances from ShotQA to refine the model's ability to select the correct option with higher confidence and precision.
5 Experiments
5.1
Implementation Details
Our implementation is based on ms-swift [64]. We initialize 1---2-5-----B-----e- model. We use around 60k samples for SFT and approximately 8k samples for GRPO. We use Flash Attention-2 [10] as the model's 's attention implementation and 6 precision for both training and inference to reduce memory consumption. In SFT stage, the global batch size is set to 4, and the model is trained for I epoch with learning rate of le-5. In GRPO stage, we set the group size G to 12 and the global batch size to 24. The clipping parameter e set to 0.2. The model is trained for 10 epochs with learning rate of 1 x 10-6, Detailed typer-parameterr are provided in the Appendix B
3 as our base

---

5.2 Main Results
Table 3: Quantitative comparison of GPT-4o 38,
and ShotVL (3B,
7B) on ShotBench. Underline indicates previous SOTA in each group.
Models
SF
CA
L$
LI
LC
SC
CM
Avg
Qwen2.5.VL.-7 Vareel--V--72--nstmmae
75.1
82.9
56.7
46.8
59,0
49.4
54,1
48.9
59.1
GPT-4o (8
69.3
83.1
58.2
48.9
63.2
48.0
55.2
48.3
59.3
ShotVL (3B)
77.9
85.6
68.8
59.3
65.7
53.1
57.4
51.7
65.1
ShotVL. (7B)
82.5
88.8
74.1
63.8
68.1
58.6
62.6
60.6
70.1
For comparison, we include results from the strongest open-source model (Qwen2.5-VVL-72B-mmnl and the leading proprictary model (GPT-4o) from Table alongside the baseline Qwen2.5-VL-3B-
Instruct, as reported in Table 3 Compared to the bascline Qwen2.5- VL-3--Inttruce Shot-VL (3B) achieves substantial improvements across all dimensions, with average gain of 19.0 points, demonstrating the effectiveness of our dataset and training methodology. Furthermore, despite having only 3B parameters, model surpasses both GPT-4o and the strongest open-source model, setting a new state of the art in cinematography language understanding while offering significantly lower deployment and usage costs. We further conduct experiments the 7B variant of our model and observed even stronger performance, which reinforces the robustness of our dataset and training strategy. We present further experiments and analysis in the following section, with visualizations of representative model outputs included in the Appendix A.2
5.3 Ablation Study
this section, we investigate the effectiveness of ShotVL's two-stage training strategy. In particular, compare five training strategies: SFT, CoT-SFT, GRPO, SFT->GPPO, and CoT-SFT-
For fast exploration, we sample approximately 4k images for the SFT stage and around Ik for GRPO. Besides, we reduce the batch size for SFT to 2, and set the group size and batch size for GRPO to The number of training epochs for GRPO is also reduced to 5, while all other settings remain unchanged. To generate reasoning process for CoT-SFT, knowledge base containing definitions and identification methods for all cinematic terms covered in ShotBench. For each training sample, we retrieve the relevant entries corresponding the question and candidate choices from the knowledge base, and prompt Gemini-2.0-ffan to produce reasoning process grounded in cinematic knowledge. More details of the knowledge base are provided in the Appendix B
first construct formattrdt
Table 4: Performance comparison of different training strategies. Bold indicates the best result, and
underline indicates the second best in each group.
Method
SFT
CoT
GRPO
SS
CA
LI
LC
Avg
54.6
56.6
43.1
36.6
59.3
45.1
41.5
25.8
45.3
68.2
78.6
53.6
47.2
63.2
44.9
53.0
25.8
54.3
52.6
64.3
47.3
36.4
54.8
38.0
42.2
27.4
45.4
Choices
69.3
75.5
52.1
46.0
63.0
47.4
48.2
26.2
53.5
66.8
78.2
52.1
46.4
60.0
44.9
51.4
30.4
53.8
69.1
79.3
56.7
51.1
60.5
45.4
53.2
28.6
55.5
We report the performance of each training method in Table It is observed that all training strategies yield notable improvements over the bascline, demonstrating the high quality and effectiveness of our constructed dataset. Comparing SFT with CoT-SFT, find that the latter yields very small gains. This may be due to the low quality of reasoning chains generated by Gemini-2, 2.0-flash, which fail to provide effecctive supervision and may introduce noise. This further highlights the advantage of GRPO, which focuses solely on outcome reward supervision.
Another observation is that reasoning augmented training consistently improves performance in the camera movement dimension (ranging from +0.4% to +4.6%), despite the ablation experiments being conducted solely on static images and containing no camera movement related questions. This may indicate that reasoning chain generation may implicitly enhance VL.Ms' capability to recognize dynamic motion. From Figure]]
GRPO consistently improves performance across most dimensions

---

Basaline vs GRPO
SFT vs 8FT-ORPO
COT-SFT COT-SFT-GGRPO
700
Kxurtry (84
Figure 7: Performance comparison across dimensions before and after applying GRPO under three training setups: baseline, SFT, and CoT-SFT.
under all training settings. Among all configurations, the SFT-+GRPO setup achieves the best overall performance, confirming its effectiveness for enhancing cinematography understanding, More case studies are provided in Appendix A.2
Conclusion
In this work, we introduce ShotBench, the first comprehensive benchmark designed evaluate VLMs on cinematography understanding. Through extensive evaluations, we identify notable limitations in current VLMs' capabilities. To tackle these problems, construct ShotQA, the first large-scale dataset dedicated to this area. We propose ShotVL series with SFT and GRPO training, successfully enhancing the model's (Qwen2.5-VL-3B-Im achieving new state-of-the-ar performance. We hope our work will contribute to future progress in image/video understanding and generation.
and Qwen2.5-VL-7B --IB-Inntrrutt) capability and
References
[1] Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37:12461-12499 2024.
[2] Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera-controlled generative rendering from single video. arXiv preprint arXiv:2503. 1647, 2025.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13925 2025.
[4] Blain Brown. Cinematography: theory and practice: image making for cinematographers and directors. Routledge, 2016.
[5] Mathilde Caron, Alireza Fathi, Cordelia Schmid, and Ahmet Iscen. Web-scale visual entity recognition: An Ilm driven data approach. Advances in Neural Information Processing Systems. 37:34533-34560, 2024.
[6] Mathilde Caron, Ahmet Iscen, Alireza Fathi, and Cordelia Schmid. generative approach for wikipedia-scabe visual entity recognition. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 17313-17322, 2024.
[7] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271. 2024.
10

---

[8] Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. Physbench: Benchmarking and enhancing -anguage models for physical world understanding, arXiv preprint arXiv:2501.166 16411, 2025.
[9] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
[10] Tri Dao. FlashAttentiinn2:: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
[11] Google Deepmind. Veo2. Accessed: 2025-05-02.
[12] Google agentic era.
DeepMind.
Introducing gemini 2.0: our new ai model for the
https://blog hoogg////eeemooooo/gooo/o/ededee google Laaiee-eemii-- -ai ai- -ww--aaat-eeeeeebbeerrrre dwcember- 2024/ 2024.
[13]
Google Deep.indd Gemini 2.5: Our most intelligent ai model. https::/blog.
www.:/gggggoogeggeooooooe ceeepmiindggmiii--nooe -wweniimoell-thiaeeeeee thinking -pwates-march--22200
2025.
[14] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. xicoppen-xcompsse2:: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401. 2024.
[15] Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9072, 2025.
[16] Haodong Duan, Junming Yang. Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang. Pan Zhang, Jiaqi Wang, et Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11 11198-11201 2024.
[17] Columbia faculty. Shot, scene, and sequence. Accessed: 2025-05-02.
[18] FFmpeg Developers. complete, cross cross-platform solution to record, convert and stream audio and video. https:///ffpeg. org/ 2006. Accessed: 2025-05-02.
[19] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36:27092-27112, 2023.
[20] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint MtXiv:2501.12945 2025.
[21] Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501. 11.00.....6.... 18266, 2025.
[22] Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 23-28, 2020, Proceedings, Part IV 16, pages 709-727. Springer, 2020.
[23] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412 2024.
[24] Black Forest Labs.
http://tttbb..cmmmllaooffrrrrr labs/flus 2024.

---

[25] LAION-AI. http:////p//////.../////////00/0 2022.
[26] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint 2Xiv:2408.03326,, 2024.
[27] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13299-13300 2024.
[28] Ji Lin, Hongxu Yin, Wei Ping. Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023,
[29] Zhiqiu Lin, Siyuan Cen, Daniel Jiang, Jay Karhade, Hewei Wang, Chancharik Mitra, Tiffany Ling, Yuhan Huang, Sifan Liu, Mingyu Chen, et al. Towards understanding camera motions in any video. arXiv preprint wxXv:2250.153766 2025.
[30] Benlin Liu, Yuhao Dong, Yiqin Wang. Yongming Rao, Yansong Tang, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondence elicit 3d spacetime understanding in multimodal language model. arXiv preprint 2007544408.00754 2024.
[31] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model an all-around player? In European conference computer vision, pages 216-233, Springer, 2024.
[32] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025.
[33] Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx milm: On-demand spatial-temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409. 12961, 2024.
[34] Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models. arXiv preprint arXiv:2403. 12966, 2024.
[35] Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment. arXiv preprint arXiv:2502.04322 2025.
[36] Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. 240,:22700.20797, 2024.
[37] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding, 2023.
[38] OpenAl. Hello gpt-4o, 2024.
[39] Alejandro Pardo, Fabian Caba Heilbron, Juan León Alcázar, Ali Thabet, and Bernard Ghanem. Moviecuts: A new dataset and benchmark for cut type recognition. In European Conference on Computer Vision, pages 668 668.685. Springer, 2022.
[40] Quynh Phung, Long Mai, Fabian David Caba Heilbron, Feng Liu, Jia-Bin Huang, and Cusuh Ham. Cineverse: Consistent keyframe synthesis for cinematic scene composition. arXiv preprint arXiv:2504. 19894, 2025.
[41] Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, and Vladlen Koltun. Does spatial cognition emerge in frontier models? arXiv preprint 1.Xi::44100064666 2024.
12

---

[42] Anyi Rao, Jiaze Wang, Linning Xu, Xuekun Jiang. Qingqiu Huang, Bolei Zhou, and Dahua Lin. A unified framework for shot type classification based on subject centric lens. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XI 16, pages 17-34. Springer, 2020.
[43] Anyi Rao, Linning Xu, Yu Xiong, Guodong Xu, Qingqiu Huang, Bolei Z.hou, and Dahua Lin. local-to-global approach to multi-modal movie scene segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10146-10155, 2020.
[44] Jonathan Roberts, Kai Han, Neil Houlsby, and Samuel Albanie. Scifibench: Benchmarking large multimodal models for scientific figure interpretation. arXiv preprint arXiv:2405.08807, 2024.
[45] Mattia Savardi, András Bálint Kovács, Alberto Signoroni, and Sergio Benini. Cinescale2: dataset of cinematic camera features in movies. Data in Brief, 51:109627, 2023.
[46] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 0r33002402.03300, 2024.
[47] Tomáš Souček and Jakub Lokoč. Transnet v2: An effective deep network architecture for fast shot transition detection. arXiv preprint arXiv:2008. 048888 2020.
[48] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiss Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[49] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568-9578, 2024.
[50] Tost-AI. nsfw- detaaage-ddeeetiio---aaag 2024.
nsfw-detector.
https://muggiingfacc.co///
[51] Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing self-reflection of lisiuaelanguage models with reinforcement learning. arXiv preprint arXiv:2504.08837, 2025.
[52] Xinran Wang, Songyu Xu, Xiangxuan Shan, Yuxuan Zhang, Muxi Diao, Xueyan Duan, Yanhua Huang, Kongming Liang, and Zhanyu Ma. Cinetechbench: A benchmark for cinematographic technique understanding and generation, 2025.
[53] Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal 1lms. Advances in Neural Information Processing Systems, 37:113569-113697, 2024.
[54] Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logievista: Multimodal Ilm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973 2024.
[55] Jihan Yang, Shusheng Yang. Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412. 14171, 2024.
[56] Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang, Jiamu Kang. Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-language programmer from environmental feedback. In European Conference on Computer Vision, pages 20-38. Springer, 2024.
[57] Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, et al. Egolife: Towards egocentric life assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 28885-28900, 2025.
13

---

[58] Yang Yang. Yurui Huang, Weili Guo, Baohua Xu, and Dingyin Xia. Towards global video scene segmentation with context-aware transformer. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 3206-3213, 2023.
[59] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.00 06078 2024.
[60] Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, and Jacky Keung. Humaneval-v: Evaluating visual understanding and reasoning abilities of large multimodal models through coding tasks. arXiv preprint arXiv:2410.12381 2024.
[61] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li. Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024.
[62] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024.
[63] Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, al. Mmvu: Measuring expert-level multi-disciplin video understanding. arXiv preprint arXiv:2501 123800 2025.
[64] Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swifta scalable lightweight infrastructure for fine-tuning, 2024.
[65] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504 10479, 2025.
Acknowledgments and Disclosure of Funding
Acknowledgments. The authors would like to thank the anonymous reviewers for their valuable comments.
Disclosure of Funding. This work was funded in part by the National Key R&D Program of China (Grant No. 2023 (033YF3300))0) It was also funded in part by the Ministry of Education, Singapore, under its MOE AcRF Tier (MOE-T2FP2021-000) in-kind funding from NTU S-Lab and industry partner(s). It was also funded in part by the MTR Research Funding (MRF) Scheme (CHU-24003), and Shanghai Artificial Intelligence Laboratory.
MOE-T2EP20223-000
and by cash and
A Discussions
A.1 Limitations
(1) Both ShotBench and ShotQA are constructed from real-world movie data. However, cinematic shots are not always with standard and clearly defined terminology. Besides, the data distribution is imbalanced across some dimensions (e.g., camera movements like dolly zoom is rare in real data). Moreover, high-quality video annotation is labor-intensive. To improve scalability, future work may explore synthetic data for more robust performance.
(2) We primarily validate the effectiveness of our dataset and training approach using Qwen2.5-VL. 3B 3B-Instruct, which has limited capability due to a relatively fewer parameters. Further studies may focus on larger base model to further improve performance.
A.2 ShotVL: Reasoning Like a Cinematographer
ShotVL Beats GPT-4o Our experimental results show that our model outperforms previous SOTA
GPT-4o, we visualize some cases and compare the outputs between ShotVL and GPT-4o in Figure g
14

---

oagle
CASPOR WYDMING
A. Low angle
Overkead
CrArrial
D. High aeglt
Mediura
Lltra Wakeiidkee
Lang
Wale
Hilty smoric
The mant Liety The tackgroand and forggornd, chamstetion line
The caraen
roctreus loding diveradd virck chanetertteee
high- high-ange dux.
ShotVI.
<heck-Tie
level of
steefoor roredl
ShotVL.
conehow lngg vitoonrrry any devond propaction Theerone, fhr comortt reower
"Toebeetiess Bariselle" This saggess out the someno is pasficedd goord lovel caysurng
<toak image pvage-ppaar capar repaeciv arppet ranecy sothour <abasco Dww.orn--booeee
Quentis (Wher's
(Weeess thés
Opdexe Lan conmart
Siburette
C. Edge light
D. ligh contaer
A. Sharr ddd
B. Conter
C. Balineed
D. bany
The saai likey gane
Di High cnared inage sarna shabeo hightigho. avning lighoustra lightiz
The foore primottly on the nons urend ce laft of forna, viill night eddeis relsiv city angtls ciaiing kiillarre baknec.
ShotVL.
<hiek--ihe The irige shee chooup vicw as opes plac will wikos ihees inile. inchading
papan pomilty and palkauing aubona appoars lighting liggh volbursly inseuity bet and dfficeed Thereronn comeot
escna
image C--ahiic cansoeen caessee
earnly spoced
spally
Thersoo,
A wwwkooeeeee lelesseee
Figure 8: Comparison between GPT-4o and ShotVL.
train/reward
1.2
25000
50000
75000
100000
125000
150000
175000
Step
Figure 9: Progression average reward during GRPO training.
15

---

Reasoning process improves performance As illustrated in Figure the average reward increases throughout the GRPO process, indicating that the generation of resoning process helps ShotVL to better understand and recognize cinematic language within movie shot. More intuitively, we visualize the outputs between ShotVI. and the pure SFT variant in Figure TO.
Qveates WWher's the conrgsstions thi shat!
Quediox Wher'ss lee dee ofthis
MELCOMETO
WATERLOO IOHA
Opdexc
Optinnc Falise
Lelly
bany
D.
SFT
ShotVL
ShotVL.
olisic
visile Leails
<deak-The uach dieritatton elmmnns
indicas a close ap ormacn porpective The eagaora tua the lane une in awerad in caram dends maher wade-agge vioe encorppa the entir without
walharian any dea pober his aoaadd sile.
Queates :Wihars the lighting type thks
Qseaios WWbur'tte lighting type
Optinn
Optiacc
C. A.Aniiiaa bade, Ebrreat
HML, Masulight
SFT
SET
ShotVL
ShotVL
<beak-1 bh slena inse anbiunt
casdice
other surco, nhsd tuaical be trienas sertaue lhe bure or rostaneto. There ioso indiation
chise The image tinns aloovup peecss is whhe appwers to indior wrilng pasiiity got mannng aoves zencet
artiical wareo zach HDd ii Baascscct libting. poseeees ansdiceaggora
backgreand -MMiieorasssen Cassorenn
stroehher it cothebooee www.oooaaaeeeiiieeeeoooo
which wreie anficce commenly
Figure 10: Comparison between GPT-4o and SL
A.3 Current VLMs' Cinematography Understanding Needs Further Enhancement
We further visualize failure cases from the strongest open-source model, Qwen2.55-VL-72B-Inntt well the strongest proprietary model, GPT-40, shown in Figure [1
Witch de
Crille
Via
C-lebon
Leggen
B-
Tre Aumee
Tim AnEVET
gr-b
Figure 11: Visualization of failed cases.
16

---

A.4 More Visualizations
More output cases with thinking process from ShotVL are provided in Figure
Qrectie Ahhur's
Optinec:
Durch sugl
B.Serbi
Hlihh Aagl
D. Overhied
Dasligtr. Taagaen
ILBtammmeet, Mitend light Diylight, Sunny
Aniiical ligh, LED
ShotVL
ShotVL
enkcatoon of peopactive.
irageslono Socsing un object being lekd
<deak The image thos to people sveding an beach wat derr Abarices andi refecing wan Exdrading the they condenn during daylight hore.
<tlise
Queccias /Whrr's the oospariiiaa thi the??
Quoikeen: What's thcaamm nevarmiine anndc
Optnne
Oplone right
ShotVL
contovel eldett sun appars brxitinng ripeiiiaa derwns other wald
1Daak/The ixge anwaad wonsaaiiiiiiia vicn blccs inccee vilourany
Figure 12: Output visualization of ShotVL.
B Implementation Details for Evaluation and Experiments
Details of Evaluation During evaluation, we first attempt to extract each model's final answer using template-based based matching. If valid match is found, follow the previous works [3 63 to use GPT-4o as an automatic answer extractor. For open-source models, we densely sample video frames 12 FPS with a maximum resolution of 360x640 pixels. For image-based samples, follow the default input configurations of each model. We apply greedy decoding during inference for reproducibility. For proprietary models, evaluate them via their official APIs, setting the temperature to to produce deterministic outputs. We use GPT-4o (2024-08 0024-08-06) 08-06) to extract final answers, based on the prompt Figure
Details of Experiments Our training implementation is based on ms-swift framework [64]. All hyper-parameters use for the main experiments are reported Table 5 and Table The training process of SFT is performed on 4 Nvidia A100 GPUs and the GRPO process is performed on Nvidia A 100 GPUs.
In our ablation study, we construct a JSON formatted knowledge base on cinematography and use prompt Gemini-2.0-flash generate reasoning process. We visualize some examples in Figure
C Dataset Statistics for ShotBench and ShotQA
ShotBench Below is the list of titles used in constructing the benchmark.
Title
Year
IMDb ID
Manchester by the Sea
2016
t10034228
17

---

Title
Year
IMDb ID
The Kids Are All Right
2010
u0842926
Little Women
2019
tt3281548
Flamin Hot
2023
tr1105234
A Quiet Place
2018
u6644220
Belfast
2021
u12789558
Green Book
2018
u6966692
Phantom Thread
2017
t15776858
Bridge of Spies
2015
tt3682448
10
20th Century Women
2016
t14385888
11
Passengers
2016
tt 355644
12
The Fahelmans
2022
tt 4208870
13
Moonlight
2016
t4995722
14
Licorice Pizza
2021
ttl 1111180388
15
BARDO, False Chronicle of a Handful of Truths
2022
u114176522
16
Women Talking
2022
tt 3669038
Foxcatcher
2014
100089
18
Christopher Robin
2018
tt4575566
19
The Great Gatsby
2013
tt 343092
20
Blade Runner 2049
2017
ut 856101
21
Marriage Story
2019
L17653254
22 Tinker Tailor Soldier Spy
2011
LL 340800
23 Nebraska
2013
ttl
24 Black Swan
2010
10947798
25 Youth
2015
113312830
26
The Batman
2022
tt 87783330
27
Max: Fury Road
2015
tt 392190
28 Minari
2020
tt 0633456
29 Sicario
2015
tt3397884
30
Knives Out
2019
L18946378
31
Ma Raineys Black Bottom
2020
LL105 14222
32
Amour
2012
LL] 602620
33
Lincoln
2012
t0443272
34
Judas and the Black Messiah
2021
t19784798
35
Life of
2012
t10554876
36
Jojo Rabbit
2019
L12584384
37
Inside Llewyn Davis
2013
t12042568
38
The Banshees of Inisherin
2022
ut 1
39
Barbie
2023
tt1517268
40
The Favourite
2018
L15083738
41 Whiplash
2014
112582802
42
Straight Outta Compton
2015
tt 398426
43
The Revenant
2015
It 663202
44
Top Gun: Maverick
2022
tt 745960
45 Nomadland
2020
L19770150
46 Carol
2015
L12402927
47 Ad Astra
2019
L2935550 10
48
RRR
2022
tt8 18644
49
Midnight Paris
2011
t156738
50
A Separation
2011
tt 71833382
51
The Hobbit: The Desolation of Smaug
2013
tt 70358
52
The Worst Person in the World
2021
tt 0370710
53
The Power of the Dog
2021
tt 0293406
54
Alice in Wonderland
2010
t11014759
55
TÁR
2022
tt 444442726
56
Can You Ever Forgive Me?
2018
t15595822
Ted
2012
L1163725
58
Hugo
2011
10970179
59
Cold War
2018
U6543652
60
May December
2023
ttl 3636511 3651199
61
Her
2013
11799700
62
Unbroken
2014
tt 11889398
63
Ida
2013
11718899
64
Ex Machina
2014
U0470752
18

---

Title
Year
IMDb ID
65
Beyond the Lights
2014
tt3125324
66
The Kings Speech
2010
t1150330
67
American Sniper
2014
tt2179136
68
The Imitation Game
2014
(12084970
69
Before Midnight
2013
u2209418
70
Promising Young Woman
2020
u1920292
71
Baby Driver
2017
tt3890160
72
Indiana Jones and the Dial of Destiny
2023
tt1462764
73
Captain Phillips
2013
tt1535009
74
Glass Onion: A Knives Out Mystery
2022
u24734444
75
The Disaster Artist
2017
tt3521 126
76
Never Look Away
2018
ut531
77
Hail, Caesar!
2016
t10475290
78
Star Trek Into Darkness
2013
11408801
79
Nightmare Alley
2021
L17740966
80
All Quict on the Western Front
2022
t11016150
81 Fences
2016
tt267 1706
82 Harrict
2019
t14648786
83
Zero Dark Thirty
2012
uI 790885
84
No Time Die
2021
L1282330
Get Out
2017
lL5052448
86 Moneyball
2011
tt1210166
87 Skyfall
2012
t11074388
88
Living
2022
11905 1908
89
The Lobster
2015
tt3464902
The Big Sick
2017
t15462602
91 Spectre
2015
tr2379713
92 Napoleon
2023
tt 3287846
93
El Conde
2023
L121 13540
Lion
2016
Lt3741
95
Arrival
2016
U12543 164
96 Parasite
2019
tt675 1668
97
The Lost Daughter
2021
10000
98 Gravity
2013
tt1454468
99
The White Tiger
2021
L15771548
100
Mank
2020
tt 1110618286
101
The Trial of the Chicago 7
2020
ut 07007774
102
Maestro
2023
L15535276
103
Silence
2016
L10990215
104
Drive My Car
2021
t1L4039582
105
Silver Linings Playbook
2012
tt 0456588
106
Logan
2017
13315342
107
The Hobbit: The Battle of the Five Armies
2014
tt2310332
108
Moonrise Kingdom
2012
u1748822
109
Room
2015
U3170832
110
Triangle of Sadness
2022
u7322224
111
Real Steel
2011
t10333035
112
The Post
2017
116294822
113
Roma
2018
tt6155172
114
If Beale Street Could Talk
2018
u7125860
115
The Ballad of Buster Scruggs
2018
u6412452
116 Django Unchained
2012
u1855328
117 The Lighthouse
2019
(17984734
118
A Star Is Born
2018
tt1517451
119
The Descendants
2011
t11033575
120 Babylon
2022
tt 0640346
121 Once Upon Time in Hollywood
2016
tt4010844
122
The Shape of Water
2017
tt5580390
123
King Richard
2021
U1920288
124
Lady Bird
2017
U4925292
125
Joker
2019
u72864566
126
The Danish Girl
2015
10810819
127
Winters Bone
2010
111399683
19

---

Title
Year
IMDb ID
128
La La Land
2016
tt3783958
129
Beasts of the Southern Wild
2012
t21125435
130
Bloods
2020
t19776644
131
The Irishman
2019
u11302006
132
Darkest Hour
2017
u4555422
133
The Father
2020
u10272386
134
BlackkKlansman
2018
tt7349662
135
12 Years Slave
2013
tt2024544
136 Adaptation.
2002
t0268 126
137
Bridesmaids
2011
t1178838
138
Vice
2018
t16266538
139
The Girl with the Dragon Tattoo
2011
tt 568346
140
The Hobbit: An Unexpected Journey
2012
t10903624
141
Into the Woods
2014
11218041 I
142
Boyhood
2014
t11065033
143
First Man
2018
tt 213641
144
Paraliel Mothers
2021
tt 261 211268826
145
The Martian
2015
t33659388
146
The Social Network
2010
uL 285016
147
First Reformed
2017
U1055438
148
Deepwater Horizon
2016
LL 860357
149
The Wolf of Wall Street
2013
110933466
150
Dallas Buyers Club
2013
117990336
151
Hacksaw Ridge
2016
tt21 19532
152
Dunkirk
2017
t1501 3055
153
Selma
2014
tt 1200722
154
The Theory of Everything
2014
t12980516
155
Nighterawler
2014
tr2872718
156
Everything Everywhere All Once
2022
L16710474
157
True Grit
2010
LLL403865
158
Jackie
2016
U11619029
159
Killers of the Flower Moon
2023
tt5537002
160
One Night in Miami...
2020
119212922
161
Ford Ferrari Fall
2019
111950186
162
Anatomy
2023
LL 7009710
163
The Midnight Sky
2020
tt 053099668
164
The Tree of Life
2011
L0478304
165
Brooklyn
2015
112381111
166
American Hustle
2013
tt 11002441
167
The Lone Ranger
2013
tt1210819
168
Mudbound
2017
tr2396589
169
Oppenheimer
2023
It 5398776
170
Another Round
2020
tt 0288566
171
Fifty Shades Grey
2015
L12322441
172
Argo
2012
L11024648
173
The Two Popes
2019
L18404614 14
174 Elvis
2022
tt3704428
175
Hell or High Water
2016
t12582782
176
The Hateful Eight
2015
IT3460252
177
Molly's Game
2017
tt4209788
178
News of the World
2020
L16878306
179
The Adventures of Tintin
2011
10983193
180
The Greatest Showman
2017
LL L1L485796
181
Empire of Light
2022
tt 410444146
182
Interstellar
2014
1O0 1 16692
183
Extremely Loud & Incredibly Close
2011
t10477302
184
1917
2019
t18579674
185
127 Hours
2010
ttl 5t2344
186
Spotlight
2015
U1889588
187
Rocketman
2019
12066051
188
Marshall
2017
US301662
189
Drive
2011
10780504
190
Inherent Vice
2014
tt1791528
20

---

Title
Year
IMDb ID
191
Tangled
2010
u0398286
192
Captain America: The Winter Soldier
2014
tt 1438666
193
The Big Short
2015
tt (1656363
194
Tenet
2020
U6723392
195
Sound of Metal
2019
U13663188
196
The Holdovers
2023
ut (114849] 194
197
Les Misérables
2012
711707386
198
Call Me by Your Name
2017
tt5726616
199
Hidden Figures
2016
t14846340
200
The Grand Budapest Hotel
2014
u2278888
201
Past Lives
2023
tt 3233346
202
Dont Look Up
2018
34232
203
Poor Things
2023
11L4230458
204
Mr. Turner
2014
u12473794
205
Prisoners
2013
ttl 1922114
206
Casino Royalc
2006
t1038 106
207
Polytechnique
2009
tt 194238
208
Gladiator
2000
to1 72495
209
47 Ronin
2013
Ut 335997
210
Mindhunters
2004
u0297284
211
The Curious Case of Benjamin Button
2008
u0421715
212
Mission: Impossible
2011
tt1229238
213
Wednesday
2007
t11024676
214
No Country for Old Men
2007
t04773488
21.5
The Last Duel
2021
tt4244994
216
Rebel Moon
2023
tt 4998742
217
Thor: Love and Thunder
2022
tt 0648342
218
One Piece
2018
tt10109772
219
Star Wars
1977
U0076759
220
The Witcher
2017
Lt735 1402
221
The Lord of the Rings: The Rings of Power
2022
Lt763 1058
222
There Will Be Blood
2007
t10669944
223
Bullet Train
2022
tt 2593682
224
Quantum Solace
2008
t10800515
225
Dune: Part Two
2024
5239678
226
Forrest Gump
1994
t01199:30
227
Loki season 2
2023
ut (11827133 1346
228
Cruella
2021
1(3228774
229
Fight Club
1999
t00137523
230
The Fall Guy
2024
111684562
231
James Bond
2015
tt4896340
232
Mission: Impossible
2011
tt 229238
233
Scott Pilgrim vs. the World
2010
t10446029
234
John Wick
2014
[1291 1666
235
The Suicide Squad
2021
U1634454
236
The Killer
2023
uL 136617
237
Superman
1978
t10778466
238
Inception
2010
tt1375666
239
World of Warcraft
2014
tt4191810
240
The Raid
1954
u0047388
241
Barry Lyndon
1975
u0072684
242
Captain America: Civil War
2016
tt3498820
243
Marvels Jessica Jones
2015
u12355577
An overview of cinematic terms used in ShotBench and the distribution of QA pairs are visualized in Figure I5a and Figure 15b] Details of ShotBench are provided in Table g We adopted an aesthetic score threshold of 3.0, samples with low aesthetic scores often characterized by poor composition, motion blur, or chaotic visual content. Such samples typically lack well-defined cinematic attributes and were excluded maintain the overall quality our dataset. And a total of 20 trained annotators participated in the annotation process during datasct construction.
21

---

'You are un Al assistant who will help me mutch an answer with several options single-choice question. 'You provided with question, several options, and answer, 'and you need to find which option most similar to the answer. "If the answer says things like refuse to answer, I'm sorry cannot help, etc., output Z."
'I meaning of all options are significantly different from the answer,
'or answer does not select any option, output Z.
'You should output one the choices,
(if they are valid options), or Z.'n'
'Example I: 'nn'
QQestion: Which point is closer to the Ammera/nSelec from the following choices. 'nn'
'Options: A. Point A/mB. Point Bin( Failed'n'
'Answer: Point B, where child sitting. is closer to cameru. 'our output: (B)n'
'Example 2: 'un'
QQuestio: Which point closer to the camera?inSelect from the following choices.
'Options: (A) Point A/n( Point Bnn(Z) Failedn'
"Answer: I'm sorry, but can't assist with that request. 'n Your output: (Z)'n"
'Example 3: 'nn'
'Question: Which point corresponding the reference point?nSSeect from following choices. 'nn'
'Options: (A) Point (B) Point Ba(ZZ Failedin'
A reference point (REF) on the first image is at the tip of the pot.
which is the part used to Poke if the pots were used for that action. Looking at the second image,"
'e need to find the part of the object that would correspond to poking, 'n'
"(A) Point A at the tip the spoon's handle, which is not used poking 'nn
(B) Point B the bottom of the spoon, which not used for poking 'm'
(C) Point C is the side of the pspoonot, which is not used for poking, In
(D) Point D is at the tip of the spoon, which is not used for poking. 'n''
'eehhrrfore, there is no correct answer in the choices'm in Your output: (2)n'
"Example 4:
'Question: ()?nOption:: ! (Z) Failed'nAnswer: { Your output:"
Figure 13: Prompt format used for answer extraction from GPT-4o.
Table 5: Hyper- parameters for SFT.
Table 6: Hyper- parameters for GRPO.
Parameter
Value
Parameter
Value
model
Qwen2.5 S.ee2...VL... V.. Insiruct flash_
model rihí thf__tpe
Qwen2.5. VL After SFT
grpo
train type
full
use usc_vilm
truc
tosch_dtype
bitear16
vllm device
auto
mum, train epochs
vllm Bllm__pu__mmee mppu_menory...o uilization
0.6
per_ deldieviceeeee train,
jape
sii_deiice_evee bat____al__achhiiiiiee batch_size
terch_dlype
b6loat16
learning rate
le.5
max length
2048
gradient_ accumulation steps
completionn Jength
1024
oval steps
100
num, www__train__oo caiins
10
save_steps
100
per device. batch,
save, Jotal limit
3
per deldddeeeeeeeeeeeeee eval batch size
logging steps
learning
max_length
3072
gradient wwwumulationnnnooom
system
"You are a helpful assistamt""
save strategy
steps
warmup
0.05
strarey
steps
dataloader_numa num, workers
steps save sseess save sotal limit
500
bogging_steps
warmup ratio dataloader, num, workers num, gencrations temperature
0.01
12
12
1.0
repetition penalty deepspeed
zero3
num. itcrations
infer workers seyne_gencrate
false
0.001
max grad
norm norm
0.5
22

---

Table 8: Distribution of cinematic terms used in ShotBench
Dimension
Term
Wide
13.3
Close
Shot Size
Medium Close Up
12.6
Medium Wide
12.1
Medium
Up
15.4
Insen
2 shot
14.5
Shot Framing
Group shot
14.2
Establishigg
Over the shoulder
13.6
3 shot
13.5
Acrial
20.7
Overhead
20. 1
Angle
angle
High angle
19.7
Dutch angle
19.7
Long Leus
25.5
Lens Size
Ulima Widc&Fisheye
Wide
25.2
Daylight
Artifiiaal light
11.9
Mixed light
10.5
Lightng Type
Practical light
9.4
Sunny
9.3
Moonlight
Fluoreseent
HMT
7.7
Tungstes
1.2
LED
0.8
Side light
10.8
Backlight
High contrett
10.3
Stlhouete
10.2
Lightin Condition
Edge light
Usderiaght
Top light
10.0
Hard light
10.0
Balanced
17.1
Composition
Right heavy
Symmetrical
16.4
16.7
heavy
Shori side
16.2
Push in
10.4
Pullicut out
9.1
Boom
8.5
right
7.4
Till down
7.1
688
Camera Movement
Zoomi in
6.4
Static
6.3
Move the right
5.9
Zoom
4.5
Art
3.9
Camera roll
3.7
Dolly
23

---

"name": "Wide Shot Long Shot", "abbreviation" : "WS / LS" "meaning": "Shows
although not necessarily filling but the subject is more prominent than in an EWS, It shows the subjoct within their surroundings, providing context"" the "dentifieation" "Identify the full body of the subject within the frame, usually with considerable space above their head and below their impoctan,
subject
focus largely on the enviromment,
feet. The setting elearly
identifialle their full figure shown."
"name": "Close Up". "abbreviation" : "CU",
"meaning"" "Frames specific part of the subject. typically the head and lace. from the neck up, It tightly frames the character's face, coppbasizing their emotional state and rcactions. Littic to background visible.", most the below the chin, often showing the neck or very top of the shoulders, Facial details and emotions are the primary focus."
"identifictiion" subject's head/face
top frume usually just hairline,
"name": "Extreme Close Up". "abbreviation" : "ECU"
detail the subject, such eyes, mouth, specific object. used to intensity, tension.", "ddetiiication: "The shot magnifies a single feature (like eyes, lips) or small object, filling the entire screen with You cannol see the whole face the broader context. often foels intimate intense."
"meaning" "Frames only small highlight a cnucial detaill, convey strong
Figure 14: Examples constructed knowledge base on cinematic language.
Single
Medium
1
lens
Dem
Wide
lens
Establishing shot
14%
Movement
heavy
O
ng
Le
ns
Mltedd fite
Shat 13%
Stop
Backlign
Top lignt
Fraet
Shot Franing
Compusition 13%
Right
Balanced
Center
Overhead
Shart sice Buspercart
Daylight
Medium
light Up
Close
Proticall Haghs
Wide
I Ultra
m
Low
Hil
content
ano
e
13%
Lighting Condition
Size Lighting Type
Symmetrica
Wide
shot
Fisheye
Aerial
14%
Dutch angle
Group shat
LoW
Over
shoulder
Extrome
angle
Close
light
Left heavy Medium Wide
Medium close Up
Insert
(a) Overview of cinematic terms used in ShotBench.
(b) Distribution of questions.
Figure 15: Statistics of ShotBench across different dimensions.
More details of ShotQA ShotQA is constructed in similar manner to ShotBench (Section 3.2) except that only video samples are totally manually annotated and verified by trained annotators and experts. For large-scale image annotations sourced from expert cinematography websites, we conducted random sampling checks and found their quality adequate for training use. We adopt a two-stage filtering strategy to ensure overlap between training and evaluation sets: first remove duplicate samples based on IMDb IDs and timestamp as a coarse-level filtering step. Then, we extract CLIP features for all samples and exclude samples from the training set whose feature has cosine similarity greater than 0.95 (following 5 6 19
with any sample in ShotBench.
The GRPO sub-dataset consists of a combination of mid-difficulty samples and uniformly sampled QA pairs across all eight dimensions. We identify mid-difficulty samples by prompting Qwen2.5- VL-3B to answer subset of the data multiple times and selecting those for which both correct and incorrect answers are observed across different runs. The remaining QA pairs were uniformly sampled again and used for SFT.
24

---

Table 9: Sample distribution the ShotQA
Dimension
#Samples
Camera Angle (CA)
9,405
Shot Composition (SC)
9,597
Lens Size (I.S)
8,324
Lighting Condition (I.C)
8,778
Lighting Type (LT)
6,811
Shot Framing (SF)
8,298
Shot Size (SS)
8,579
Camera Movement (CM)
1,200
D Reference Materials on Cinematography Used in ShotBench Construction.
During the construction of ShotBench, trained annotators through professional teaching websites and teaching videos publicly available on the Internet. We provide some representive materials in Table I0
Table 10: Representative reference materials used to train annotators.
Dimension
Website
Video
Size
httpe: http://wwwwwdddiiideeeee www.ccoo/oooo.o..... blog//
kttpe: //vvx youtube coutuube.co/vvttc//v
types of camera chots sizes in film/
AyNLBruKfoc
Framing
http:///oos..tttoooooo con/ blog/
https://90s wwutubo..uuuooooooo com.stthhymm///
oypes-of-canere shot Tranes- in- film/
90ViqauX [M
Camera Angle
attps://oww.c http:///www..ttdoooodeeeeee blog
https://90n 900
youtube conbubbe.an//atchh/>
types-of- -hotl-nnlts--e-----eaa
Lens Size
attpe://vkk wtw.:///ddddooodeeeeee com/ blog/
https://wv/ http///wwwoouubbe.eeeeeee youtube. con/vatccby-
focal Dccll-ennthh-o
ceneettb--eerr-a-...eaaaaaaaaaaa camera-
lined/
Lighting
https:// oww www..attdoobbrr..... con/ blog/
https://9og 90% www.you..ub.ccooo com220
1ighting//
knakrri
Shot Composition
nttpp://vxx.22 atudiobinder con/bloo/
Bttps:/ cwuutub..coo/vatcclyy.
rules- shot- conpsttcooccsttioonooo film/
klm77 dtn/:ld0000tt-100
Camera Movement
ittps::/vwx.
blog/
httpa:
different- byperrettttpps--oeee ofapee----ccnees--eeoo www.-aaaneeeeeeeeeeeeeeeeee in- film/
IiyBo-qLDeN
E Societal Impact Statement
This work presents ShotBench, a benchmark for evaluating vision-language models (VLMs) on einematic language understanding, and ShotQA, large-scale dataset designed for training such capabilities. Additionally, we propose LhotVL, reasoning-enhaneee VLM series trained via SFT and GRPO.
Positive Societal Impact. By improving VLMs' understanding of professional cinematic con- ventions, our work can contribute to the development of AI systems that assist in film production. Specifically, cinematography aware models may support Al-assisted filmmaking tasks such as shot planning, automated style matching, and film-level image/video generation. These capabilities could help democratize access to professional filmmaking workflows, reduce production costs, and em- power creators with limited resources. In addition, our benchmark and dataset may foster research into multimodal reasoning, benefiting broader applications in video understanding and generation.
Negative Societal Impact. As with other generative or vision- technologies, there are potential negative applications. For example:
Disinformation and deepfakes: Enhanced understanding of cinematic language could be exploited to make Al-generated fake content more visually convincing or emotionally manipulative.
Creative job displacement: The use of cinematography-aware models in automated filmmaking pipelines may marginalize certain creative roles (c.g., assistant editors, junior cinematographers).
25

---

Bias propagation: If the training data or annotations reflect specific cultural aesthetics or norms (e.g.. Western cinematic styles). the resulting models may encode biased visual preferences or overlook underrepresented filmmaking traditions.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contributions are summarized in both abstract and introduction [
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Appendix A.I
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
we propose assumptions based on the findings in Section 3.3 and prove our assumptions through experiments in Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Details of our experiments reported in Appendix
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification:
We will opensource dataset, codes and models, as mentioned in Section] |
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
www.SSubbisiooooiiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Hyper-parameter parameters for evaluation and training are detailed in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experimental results of VLMs are basically consistent when the hyper- parameters are fixed, and repeating the experiments requires significant amount of comput- ing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Information of computer resources needed is reported in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The paper is conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: discuss positive impacts and negative impacts in Appendix Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We release ShotBench and ShotQA under non-commercial research license. All movie data are obtained from publicly available sources, and we filter out inappropriate content using automated tools (e.g., NSFW detectors). Additionally, for source traceability and clearly document intended use cases to discourage misuse. The trained model series ShotVL will be released under gated access and subject terms of responsible use.
include metadata
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available assets with proper attribution and license compliance. All third-party tools or models are cited in the main text or Appendix, with license names and official links included when available.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
patasets.papersstt
For existing datasets that are re-pace se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details of our dataset and benchmark are provided in Appendix C
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/coodemmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
This work does not involve crowdsourcing or experiments with human subjects in the traditional academic sense. The data annotation was carried out by third-pant professional data service provider. Annotators were paid according to the service agree- ment, and we ensured they had access to comprehensive publicly available cinematography reference materials, which we include Appendix [D] However, since the annotation was conducted externally and not through controlled experimental setup, we do not include internal instructions or screenshots.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
The annotations used this study were conducted by professional third-party annotation company. Our team did not interact directly with annotators collect any personal or sensitive information. Therefore, this study does not involve human subjects research under the definition requiring IRB equivalent approval.
Justification:
32

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We used Gemini-2.0-fie generate reasoning chains grounded in structured cinematography knowledge base as part of our ablation study (Section 5.3). Our model was initialized from Dwenn2.5vVL-3B-Insttuue
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standard components.
Please refer to our LLM policy (https://neura) httpr//ppppcccocofeeeeeee/////// for what should or should not be described.
33


Reviews:
[{"review_id": "9ru7dGBo6Y", "reviewer": "Reviewer_S5g8", "summary": "This paper introduces a new benchmark (ShotBench) specifically designed to evaluate how well VLMs understand the visual grammar and cinematic language of movie shots. The authors evaluate 24 sota VLMs on this curated benchmark and find that existing models share common limitations in cinematic understanding. To address these gaps, the authors construct a 70k QA dataset (ShotQA), and use it to train a new model (ShotVL). Experimental results demonstrate that ShotQA significantly enhances ShotVL's performance, enabling it to surpass all previously evaluated open-source and proprietary models on ShotBench and establish a new sota.", "questions": "Please refer to the weaknesses above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n* While prior work has addressed specific sub-tasks in film understanding (e.g., shot classification), this work has its unique novelty in proposing a comprehensive and structured evaluation of cinematography understanding since the proposed benchmark covers eight core aspects of cinematography (shot size, shot framing, camera angle, lens size, lighting type, lighting condition, composition, and camera movement).\n\n* This work presents extensive experiments on the proposed benchmark, evaluating 24 widely used VLMs from both open-source and close-source, and provides detailed quantitative results along with insightful qualitative analyses. The inclusion of annotated image examples and failure cases is effective in conveying the subtlety of the task and the types of errors current models make.\n\n* The proposed model (ShotVL) is trained with a two-stage pipeline combining supervised fine-tuning and GRPO on the proposed QA dataset (ShotQA). The resulting model achieves substantial improvements over strong baselines, including GPT-4o, which is a clear evidence to verify the effectiveness of the proposed dataset.\n\n\n### Weaknesses\n\n* One concern regarding the proposed benchmark/dataset is that, the QA pairs are provided in the form on multiple-choice. It is unclear how well the benchmark generalizes to more open-ended or generative tasks in cinematic language understanding. In addition, such task design may not fully capture the reasoning capabilities of tested VLMs that perform better on open-ended or generation tasks.\n\n* This is not a major concern, but it would strengthen the claims to show that ShotQA and GRPO training strategies is model-agnostic and may generalize across different VLM architectures of various sizes. Currently, the proposed ShotVL only use one relatively small VLM as backbone (Qwen2.5-VL-3B-Instruct).", "was_revised": true, "final_justification": "The authors' rebuttal has addressed my concerns. I am maintaining my rating of borderline accept.", "rebuttal": "Thank you for your positive assessment and for highlighting the novelty and extensive experiments of our work. We sincerely appreciate your valuable feedback and constructive suggestions. We have carefully reviewed your concerns and provide point-by-point responses below:\n\n### **Q1: The multiple-choice (MCQ) format limits the evaluation of generative capabilities.**\n\nWe agree that open-ended evaluation is an important direction for cinematic understanding. We adopt a multiple-choice format in ShotBench to ensure **objectivity**, **scalability**, and **reproducibility**, which are key qualities for building a reliable benchmark in a new domain. This design follows the common practice in recent representive VLM benchmarks (e.g., Video-MME [1], PhysBench [2], MMBench [3]) and enables fair comparison across models. While it does not capture generative reasoning, we view it as a necessary first step and will clarify this limitation in the revised version.\n\n### **Q2: Strengthen the claims to show that ShotQA and GRPO training strategies is model-agnostic and may generalize across different VLM architectures of various sizes.**\n\nWe appreciate the reviewer’s suggestion. In response, we have conducted additional experiments using a stronger backbone, Qwen2.5-VL-7B-Instruct, following the same two-stage training setting (SFT + GRPO). This larger model, **ShotVL-7B**, achieves better overall performance (53.1 $\\rightarrow$ **68.0**) than ShotVL-3B (**65.1**) on ShotBench.\n\nWe will include these results, along with more extended comparisons and analysis, in the next revision of the paper.\n\n### References\n\n> [1] Fu C, Dai Y, Luo Y, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 24108-24118.\n>\n> [2] Chow W, Mao J, Li B, et al. Physbench: Benchmarking and enhancing vision-language models for physical world understanding[J]. arXiv preprint arXiv:2501.16411, 2025.\n>\n> [3] Liu Y, Duan H, Zhang Y, et al. Mmbench: Is your multi-modal model an all-around player?[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2024: 216-233."}, {"review_id": "wxDGYUh6Uq", "reviewer": "Reviewer_K6tX", "summary": "This paper introduces ShotBench, a benchmark designed to evaluate the ability of Vision-Language Models (VLMs) to comprehend the visual grammar of movie shots. ShotBench comprises 3,049 still images and 500 video clips from over 200 films, with each sample annotated by trained annotators or sourced from professional cinematography resources, resulting in 3,608 high-quality question-answer pairs. The authors conducted a comprehensive evaluation of over 20 state-of-the-art VLMs across eight core cinematography dimensions and found significant limitations in their fine-grained perception and cinematic reasoning. To enhance VLMs' capabilities in this area, the authors constructed the large-scale multimodal dataset ShotQA and proposed the ShotVL model, which outperformed existing models and set a new state-of-the-art performance.", "questions": "The detailed questions are outlined in the weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n•\tInnovative Benchmark and Dataset: The paper introduces ShotBench, the first benchmark specifically designed to evaluate Vision-Language Models (VLMs) on cinematography understanding, covering eight core dimensions (e.g., shot size, framing, lighting). It includes 3,608 high-quality question-answer pairs, addressing a critical gap in VLM evaluation for cinematic language.\n\n•\tLarge-Scale Dataset: The ShotQA dataset, with approximately 70,000 QA pairs derived from movie shots, provides a valuable resource for training cinematography-aware VLMs.\n\n•\tNovel Model Proposal: The proposed ShotVL model, built on Qwen2.5-VL with supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO), achieves a significant performance improvement (average 19.8% gain) on ShotBench, surpassing both open-source and proprietary models.\n\n•\tComprehensive Evaluation: The evaluation of 24 leading VLMs provides a thorough analysis of current model limitations, offering clear directions for future research.\n\nWeaknesses：\n\n•\tThe determination of the eight core dimensions of cinematography primarily relies on Wikipedia, which may not be the most authoritative source. More authoritative classification methods could be considered, such as consulting industry standards, academic literature, or expert opinions in cinematography. This would enhance the credibility and robustness of the benchmark.\n\n•\tThe dataset contains 3,049 still images and 500 video clips, which is not considered large-scale in the context of modern large models.\n\n•\tThe paper does not provide detailed information on the data construction process, such as the number of annotators involved, the specific revision process, and the criteria for selecting and filtering data. A more detailed description of the data construction process, including the number of annotators, the training and oversight provided to them, and the specific steps taken to ensure data quality, would enhance transparency and reproducibility.\n\n•\tThe baseline models used for comparison have not been specifically trained for cinematography understanding. This may affect the fairness of the comparison. Including an additional comparison experiment where the baseline models are fine-tuned on ShotBench would provide a more comprehensive evaluation. This would help to better understand the incremental improvements brought by the proposed methods and ensure a fair comparison. In addition, consider including DeepSeek's related VLM model in the comparison to provide a more comprehensive evaluation.\n\n•\tThe failure cases shown in Figure 5 are not clearly attributed to specific models. It is unclear whether these failure cases are representative of all VLM models, including the proposed ShotVL model, or if they are specific to certain models. Specify which models the failure cases in Figure 5 represent. If these are common issues across all VLM models, including ShotVL, this should be explicitly stated. If the failure cases are specific to certain models, this should also be clearly indicated.\n\n•\tThere are some instances of awkward phrasing and grammatical errors in the paper, which can detract from the overall readability and professionalism of the work.", "was_revised": true, "final_justification": "The authors' rebuttal resolve most my conern.", "rebuttal": "Thank you for the detailed and constructive review, as well as for the positive evaluation of our contributions. We have carefully reviewed the concerns and provide point-by-point responses below:\n\n### **Q1: The eight core dimensions are primarily based on Wikipedia, which may not be the most authoritative source.**\n\nWe appreciate the reviewer’s comment. To clarify, the Wikipedia page we referenced also cites a number of authoritative cinematography textbooks (e.g., *The Focal Dictionary of Photographic Technologies* [6], *The Art Direction Handbook for Film & Television* [7]), and our intention is to provide readers with a broad and accessible entry point into the topic. Our definitions are likewise grounded in established academic and professional sources. As detailed in Table 9, annotators were trained using materials from industry platforms like StudioBinder, and our taxonomy aligns with widely adopted cinematography textbooks, including Blain Brown’s *Cinematography: Theory and Practice* [1] and David Bordwell’s *Film Art: An Introduction* [2]. We will clarify this in the revised manuscript to enhance the credibility of our benchmark.\n\n### **Q2: The dataset of ShotBench is not considered large-scale.**\n\nWe apologize if the distinction between our datasets was unclear. To clarify:\n\n- **ShotQA** (~70k QA pairs) is our **large-scale dataset** used for training.\n- **ShotBench** (3.6k QA pairs) is our diagnostic evaluation benchmark.\n\nWhile ShotBench is compact by design, this reflects the typical nature of benchmarks, which prioritize **evaluation efficiency** and **diagnostic precision** over scale. ShotBench is fully annotated and verified by trained annotators and experts, making large-scale expansion costly but ensuring high reliability. Despite its size, it provides **diverse and comprehensive coverage** across the eight core dimensions of cinematography and associated terminology (as detailed in Table 7), enabling fine-grained model analysis. Moreover, its size is **comparable to or larger** than several widely used VLM benchmarks (e.g., MMVU [3], Video-MME [4]).\n\nWe will clarify these roles and the rationale behind our design choices more explicitly in the revised manuscript.\n\n### **Q3: Lack of detail on the data construction process**\n\nTo clarify, we have provided a detailed data construction process in Section 3.2, including:\n\n**Data curation&filtering:** We apply datacuration and filtering based on aesthetic scoring, content safety checks, and automated shot segmentation.\n\n**Annotator training:** Before labeling, annotators are trained using curated cinematography tutorials and domain-specific guidelines.\n\n**QA annotation:** Question–answer pairs are generated using structured templates and professional annotations.\n\n**Expert verification:** Annotations undergo multi-round expert review to ensure consistency and quality.\n\nand **representative reference materials in Table 9**. In the revised version, we will expand our Appendix to include more detailed information on the data construction process, including the **aesthetic score threshold used for sample filtering (3.0)**, **the number of annotators involved (20)**, **additional materials and documentation** used during annotator training, and **more details on the expert review and validation** procedures.\n\n### **Q4: Fairness of baseline comparison without fine-tuning and missing DeepSeek-VL models.**\n\nTo clarify, our goal is to evaluate general-purpose VLMs under zero-shot settings to reflect their out-of-the-box understanding of cinematography. While fine-tuning all 24 baselines is infeasible, we agree it is valuable to test whether ShotQA improves other models. We have fine-tuned Qwen2.5-VL-7B on ShotQA and observed consistent gains (**53.1 $\\rightarrow$​ 68.0**). Additionally, we evaluated the performance of DeepSeek-VL[5] on ShotBench, and we found that **DeepSeek-VL models perform poorly** on ShotBench (**35.0** for DeepSeek-VL2-tiny and **36.0** for DeepSeek-VL2-small). \n\nWe will include these results in the next revision of the paper to provide a more comprehensive and fair comparison.\n\n### **Q5: On the attribution of failure cases in Figure 5.**\n\nThank you for highlighting this ambiguity. All failure cases shown in Figure 5 were **produced by GPT-4o**, the strongest proprietary model in our evaluation. We will revise the caption and discussion to clearly indicate the source model. Additionally, we will include failure cases from other representative models to offer a more balanced view of common challenges across models.\n\n### **Q6: Language and grammar issues.**\n\nWe sincerely appreciate your careful reading. We will thoroughly proofread the manuscript and correct all instances of awkward phrasing, grammatical issues, and unclear expressions to improve overall clarity and professionalism.\n\n### References\n\n> [1] Brown B. Cinematography: theory and practice: image making for cinematographers and directors[M]. Routledge, 2016.\n>\n> [2] Bordwell D, Thompson K, Smith J. Film art: An introduction[M]. New York: McGraw-Hill, 2004.\n>\n> [3] Zhao Y, Zhang H, Xie L, et al. Mmvu: Measuring expert-level multi-discipline video understanding[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 8475-8489.\n>\n> [4] Fu C, Dai Y, Luo Y, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 24108-24118.\n>\n> [5] Wu Z, Chen X, Pan Z, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding[J]. arXiv preprint arXiv:2412.10302, 2024.\n>\n> [6] Spencer D A. The Focal dictionary of photographic technologies[J]. London: Focal Press, 1973.\n>\n> [7] Rizzo M. The art direction handbook for film & television[M]. Routledge, 2014."}, {"review_id": "RBMQXEVKGY", "reviewer": "Reviewer_SCDV", "summary": "The paper introduces ShotBench, a benchmark designed to evaluate vision-language models (VLMs) on their understanding of cinematic language across eight key dimensions (e.g., shot size, camera angle, composition). It presents ShotBench with 3,608 high-quality QA pairs built from images and clips of over 200 films. The authors also construct ShotQA, a 70k QA dataset, and propose ShotVL, a new VLM trained with supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO), achieving performance gains over state-of-the-art models like GPT-4o. Experiments show ShotVL sets a new state-of-the-art in cinematic reasoning, despite its small size (3B parameters).", "questions": "1. I have a few questions about the datasets and experimental setup:\n- Will all three datasets - ShotBench, the 70k QA pairs for SFT, and the 8k QA pairs for GRPO - be made publicly available?\n- While the construction of the 3,608 QA pairs in ShotBench is well explained, the process and attributes of the 70k and 8k QA datasets are not clearly described. Could you provide more details on how these were created, perhaps in the appendix? It would also be helpful to include detailed features and attribute values for all datasets planned for release.\n- Could you clarify whether the datasets used for SFT and GRPO training are completely separate from ShotBench, ensuring that evaluation was done on truly unseen QA pairs? Clear information on the training, validation, and test splits would increase confidence in the reported performance improvements of ShotVL.\n\n2. Could you clarify whether permission was obtained to use the movie clips for research purposes?\n\n3. While this study covers eight dimensions comprehensively, did you conduct any comparisons of performance on individual dimensions with existing cinematography understanding methods, such as HunyuanVideo or SGNet?\n\n4. In ShotBench, for questions requiring inference of multiple keywords as answers - such as the example of “Lighting Condition” shown at the bottom left of Figure 1 - are models expected to mention all keywords? It would be helpful for readers if you could clarify how performance was measured and evaluated in such cases.\n\n5. Have you tried applying your approach to larger models, such as Qwen2.5-VL-7B-Instruct or InternVL3-8B? I am curious about the potential effects and performance improvements on these more capable models.\n\nIf these questions and concerns are adequately addressed, I would be willing to consider raising my score.", "limitations": "Yes, but further clarification is needed. Clearly stating whether the use of movie clips for research purposes meets copyright and licensing requirements could help prevent potential legal issues.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Generative models are increasingly applied to image and video creation across various domains, including commercial applications, yet they often fail to produce outputs that accurately align with prompts, particularly in cinematic contexts. This paper makes a contribution by developing a benchmark to test how well vision-language models (VLMs) understand cinematic language, which is essential for overcoming these limitations. The authors also curated a cinematic knowledge QA dataset, which they used to train ShotVL. By focusing on understanding cinematic language - a critical capability for applications in film production, video editing, and content generation - the research demonstrates practical value.\n2. The authors first organized expert knowledge in this domain to define eight core dimensions of cinematography and their corresponding attributes, then extended this structured framework to construct ShotBench, ShotQA, and ShotVL. This systematic research and development approach, combined with the commitment to make the resources publicly available, is a major strength of the work, as it will stimulate further research in the field and enhance the applicability and usability of the proposed benchmark and datasets in real-world scenarios.\n3. The paper thoroughly analyzes 24 diverse VLMs using standardized metrics, revealing the current limitations of these models. \n\nWeaknesses\n1. The paper does not clearly state the sources of the movie clips used or whether copyright permissions were obtained. The authors mention in Section 3.2 (Data Construction Process) that “Data are sourced from public websites and include high-resolution images and video clips,” but since the dataset includes clips from recent and high-profile films, including those nominated for or winning Academy Awards, it would be helpful to clarify whether the authors secured the necessary rights or permissions to use these materials for research. There have been cases where research groups were unable to secure usage rights for copyrighted film content and had to abandon or significantly modify their projects. Given current concerns about intellectual property and ethical use of data, a clear statement on this issue would improve the transparency of the work.\n2. The paper introduces three datasets: ShotBench, which includes 3,608 high-quality QA pairs annotated by trained experts; ShotQA, consisting of 70k QA pairs used for SFT training; and an additional 8k high-quality QA pairs used for GRPO training. While the construction process of ShotBench is described in detail, there is very limited information provided on how the 70k QA pairs for SFT and the 8k QA pairs for GRPO were created and what their specific attributes are. Additionally, since these datasets are used for model training and evaluation, it would be important to clarify whether there is any overlap between them to ensure the clarity and reliability of the experimental results.", "was_revised": true, "final_justification": "Considering the authors’ comprehensive description of the data sources and the additional experiments they conducted, they have fully addressed my concerns. Consequently, I have raised my evaluation score to reflect the substantially strengthened manuscript.", "rebuttal": "Thank you for the careful, constructive, and detailed review. We sincerely appreciate your recognition of the systematic methodology and practical value of our work. We have carefully reviewed your concerns and provide point-by-point responses below:\n\n### **Q1: On dataset availability, construction process, and train-test split integrity.**\n\n**Open-sourcing Datasets**\n\nWe **confirm** that we will **publicly release all resources** used in this paper, including the datasets **ShotBench and ShotQA**, model checkpoints, training hyperparameters, and evaluation code. All image and video samples in our datasets are obtained from **publicly accessible sources**, such as YouTube (for videos) and professional cinematography platforms (for images), and **each sample is directly downloadable via its corresponding URL**. To ensure transparency and long-term reproducibility, we will also release **detailed metadata** for each sample, including IMDb IDs, timestamps, and QA annotations, allowing users to trace or reconstruct the data even if source URLs become unavailable.\n\n**ShotQA Construction**\n\nShotQA is constructed in a similar manner to ShotBench (Section 3.2), including:\n\n**Data filtering:** We apply quality filtering based on aesthetic scoring, content safety checks, and automated shot segmentation.\n\n**Annotator training:** Before labeling, annotators are trained using curated cinematography tutorials and domain-specific guidelines.\n\n**QA annotation:** Question–answer pairs are generated using structured templates and professional metadata.\n\n**Expert verification:** Annotations undergo multi-round expert review to ensure consistency and quality.\n\nExcept that only video samples are totally manually annotated and verified by trained annotators and experts. For large-scale image annotations sourced from expert cinematography websites, we conducted random sampling checks and found their quality adequate for training use.\n\n**Dataset Splitting**\n\nWe adopt a two-stage filtering strategy to ensure no overlap between training and evaluation sets:\n\n1. We first remove duplicate samples based on IMDb IDs and timestamp as a coarse-level filtering step.\n2. Then, we extract CLIP features for all samples and exclude samples from the training set whose feature has a cosine similarity greater than 0.95 with any sample in ShotBench.\n\nThis strategy ensures no duplicate or near-duplicate samples are shared across splits. The 0.95 threshold is supported by existing works[1,2,3] and our own visual inspection. Additionally, we find that this choice allows us to retain samples that are visually similar yet differ in key cinematic attributes (usually adjacent shots within one scene),  such as shot size or composition, which are central to our task. This makes ShotBench more challenging and realistic.\n\nThe GRPO sub-dataset consists of a combination of mid-difficulty samples and uniformly sampled QA pairs across all eight dimensions. We identify mid-difficulty samples by prompting Qwen2.5-VL-3B to answer a subset of the data multiple times and selecting those for which both correct and incorrect answers are observed across different runs. The remaining QA pairs were uniformly sampled again and used for SFT. \n\nWe will add these details to Appendix in the revised version.\n\n### **Q2: On copyright and data licensing concerns.**\n\n All movie clips used in our dataset are sourced from **publicly available YouTube videos**, and we will provide:\n\n- Downloadable clips (in processed form)\n- Corresponding YouTube URLs\n- Metadata (title, timestamp, etc.)\n- Scripts for downloading and processing\n\nThis ensures full transparency and reproducibility. We will also add a clarification in the paper to address potential copyright concerns directly and explicitly.\n\n### **Q3: No comparison with existing cinematography understanding methods like SGNet.**\n\nWe thank the reviewer for this suggestion. Unfortunately, **HunyuanVideo’s camera movement classifier** and **SGNet** are not open-sourced. Moreover, these methods use **limited and inconsistent taxonomies** that do not align with standard cinematography principles.\n\nTo ensure fair comparison, we instead evaluated **CineScale** [4], an earlier open-source model for **shot size classification**. Note that CineScale only supports three coarse-grained categories (close-up, medium shot, wide shot). On the ShotBench shot size dimension, we tested CineScale in two settings:\n\n| Shot Size Samples                         | Accuracy (%) |\n| ----------------------------------------- | ------------ |\n| All samples                               | 21.2         |\n| Subset (close-up, medium shot, wide shot) | 51.2         |\n\nThe overall performance is much lower than most existing VLMs (regard that Shot-VL achieves 82.5 accuracy in this dimension), which highlights both the difficulty of ShotBench and the limited transferability of prior shot classification methods.\n\n### **Q4: How are multi-keyword answers evaluated?**\n\nAll ShotBench questions are designed as four-option single-choice questions. For questions involving multiple keywords (e.g., lighting condition), each option contains the same number of keywords to maintain balance. Only the correct option includes all the correct keywords, while the distractors may contain a mixture of correct and incorrect keywords to enhance the challenge and maintain fairness. A prediction is considered correct only when the model selects the fully correct option. This design follows existing works[5,6,7] and provides a clear and consistent evaluation protocol.\n\nWe will explicitly state this in the Evaluation section (Sec 3.3).\n\n### **Q5: Have you tried applying your approach to larger models?**\n\nYes, we have fine-tuned **Qwen2.5-VL-7B-Instruct** following the same two-stage training setting (SFT + GRPO), producing **ShotVL-7B**, it also achieves consistent improvements on ShotBench (**53.1 $\\rightarrow$ 68.0**) and outperforms our previous ShotVL-3B (**65.1**).\n\nWe are also conducting additional experiments on InternVL3-8B, and will report these results in the next version of the paper.\n\n### **Formatting and grammar issues:**\n\nThank you for the careful and helpful proofreading. We will comprehensively review the paper to correct all identified issues.\n\n### References\n\n> [1] Gadre S Y, Ilharco G, Fang A, et al. Datacomp: In search of the next generation of multimodal datasets[J]. Advances in Neural Information Processing Systems, 2023, 36: 27092-27112.\n>\n> [2] Caron M, Iscen A, Fathi A, et al. A generative approach for wikipedia-scale visual entity recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 17313-17322.\n>\n> [3] Caron M, Fathi A, Schmid C, et al. Web-scale visual entity recognition: An LLM-driven data approach[J]. Advances in Neural Information Processing Systems, 2024, 37: 34533-34560.\n>\n> [4] Savardi M, Kovács A B, Signoroni A, et al. CineScale: A dataset of cinematic shot scale in movies[J]. Data in Brief, 2021, 36: 107002.\n>\n> [5] Chow W, Mao J, Li B, et al. Physbench: Benchmarking and enhancing vision-language models for physical world understanding[J]. arXiv preprint arXiv:2501.16411, 2025.\n>\n> [6] Fu C, Dai Y, Luo Y, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 24108-24118.\n>\n> [7] Yue X, Ni Y, Zhang K, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 9556-9567."}, {"review_id": "e7EB3oYuEU", "reviewer": "Reviewer_xwhm", "summary": "The authors introduce ShotQA, a dataset for cinematographic understanding, along with ShotBench, the associated benchmark. They also propose the ShotVL model, trained on the ShotQA dataset and evaluated using the ShotBench benchmark. Their proposed model achieves substantial improvements over existing approaches.", "questions": "* Question 1: Was the same video clip potentially annotated by more than one annotator? If so, did the authors compute inter-annotator agreement (IAA) to assess consistency between annotators?\n\n* Question 2: Is it possible that clips from the same movie appear in both the training and test sets? If so, do the authors consider this a potential issue?\n\n* Question 3: Line 125-126: What does \"low-quality sample\" mean in this context? How can two clips from the same film differ in quality?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**  This is the first dataset and benchmark designed to evaluate vision-language models (VLMs) on their ability to understand cinematic language.The annotations have been validated by experts, ensuring their reliability. The authors establish comprehensive baselines using 24 VLMs, including both open-source and proprietary models. Additionally, an ablation study supports the effectiveness of their ShotVL model training strategy.\n\n**Weaknessess:**\n* Minor comment: reference 17 and 18 correspond to the same paper\n* There is a lack of detail regarding the annotation process (see Question 1 and 2) as well as the data separation process (Question 3)", "was_revised": true, "final_justification": "Having considered the authors’ replies to my questions, along with the other reviewers’ comments, I am choosing to maintain my initial evaluation.", "rebuttal": "Thank you for the support and for recognizing the novelty and impact of our work. We have carefully reviewed your questions and provide point-by-point responses below:\n\n### **Q1: Was Inter-Annotator Agreement (IAA) computed?**\n\nWhile we did not compute a formal inter-annotator agreement score, we employed a **rigorous multi-stage annotation and verification protocol to ensure quality and consistency** (as described in Section 3.2). All annotations were evenly divided into batches, each of which was audited by experts. A batch was only accepted if it achieved an accuracy above 90%; otherwise, it was sent back for re-annotation. Whenever a batch was rejected, annotators and experts engaged in discussion to reach a consensus. **We believe this expert-in-the-loop approach offers stronger guarantees of annotation quality than relying solely on IAA scores**.\n\n### **Q2: Is it possible that clips from the same movie appear in both training and test sets?**\n\nWe confirm that **there are no duplicate samples between the training and test sets.** To ensure this, we compute CLIP features for all samples and remove any training sample with cosine similarity greater than 0.95 (following [1,2,3]) to a test sample.\n\nTo clarify, our benchmark aims to evaluate fine-grained cinematic understanding, which often requires distinguishing between **visually similar but cinematographically distinct shots** (e.g., different shot sizes or compositions from the same scene). Therefore, we **allow different shots from the same movie to appear in both splits**, as this better reflects real-world challenges in cinematic understanding.\n\nWe will clarify these points in the revised paper.\n\n### **Q3: What does \"low-quality sample\" mean?** \n\nThe \"low-quality samples\" mentioned in Line 125–126 refer to those filtered out using the LAION aesthetic predictor. These are samples with low aesthetic scores, often characterized by **poor composition**, **motion blur**, or **chaotic visual content**. Such samples typically lack well-defined cinematic attributes and were excluded to maintain the overall quality and clarity of our dataset.\n\n### **Minor comment on references**\n\nThank you for your careful proofreading. We will merge the duplicate references (17 and 18) accordingly.\n\n### References\n\n> [1] Gadre S Y, Ilharco G, Fang A, et al. Datacomp: In search of the next generation of multimodal datasets[J]. Advances in Neural Information Processing Systems, 2023, 36: 27092-27112.\n>\n> [2] Caron M, Iscen A, Fathi A, et al. A generative approach for wikipedia-scale visual entity recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 17313-17322.\n>\n> [3] Caron M, Fathi A, Schmid C, et al. Web-scale visual entity recognition: An LLM-driven data approach[J]. Advances in Neural Information Processing Systems, 2024, 37: 34533-34560."}]

Author rebuttal:

Meta-review: Summary 

This paper introduces ShotBench, a new benchmark for evaluating Vision-Language Models (VLMs) on their understanding of cinematic language, covering eight core dimensions of cinematography. The authors curate a high-quality dataset (ShotBench: 3,608 QA pairs from 3,049 images and 500 video clips, annotated by experts) and conduct a comprehensive evaluation of 24 state-of-the-art VLMs, revealing significant limitations in fine-grained cinematic reasoning. To address these gaps, they construct ShotQA (∼70k QA pairs) and propose ShotVL, a VLM trained with supervised fine-tuning and Group Relative Policy Optimization (GRPO). ShotVL achieves substantial improvements over both open-source and proprietary baselines, establishing new state-of-the-art results on ShotBench.

Strengths
- Novelty and Scope: First benchmark and dataset specifically targeting cinematic language understanding in VLMs, covering eight well-defined dimensions (shot size, framing, camera angle, lens size, lighting type, lighting condition, composition, camera movement).
- Comprehensive Evaluation: Rigorous analysis of 24 VLMs, including both open-source and proprietary models, with detailed quantitative and qualitative results.
- High-Quality Annotation: Expert-validated annotations and a transparent, reproducible data construction process.
- Model and Training Innovations: Introduction of ShotVL, trained with a two-stage pipeline (SFT + GRPO), demonstrating clear performance gains.
- Resource Commitment: Authors commit to open-sourcing all datasets, code, and metadata, supporting reproducibility and future research.

Weaknesses
- Benchmark Format: The multiple-choice QA format, while objective and scalable, may not fully capture open-ended or generative reasoning capabilities of VLMs.
- Dataset Size and Construction Details: ShotBench is compact by design; some reviewers questioned whether its scale is sufficient for modern models. Initial manuscript lacked detail on annotation process, inter-annotator agreement, and data splits, though these were clarified in the rebuttal.
- Taxonomy Source: The eight dimensions were initially based on Wikipedia; reviewers suggested grounding in more authoritative academic or industry sources, which the authors addressed by referencing standard textbooks and professional materials.
- Baseline Comparisons: Some baselines were not fine-tuned for cinematography understanding, raising questions about fairness. Authors added new experiments with fine-tuned models and additional baselines (e.g., DeepSeek-VL).

This submission makes an important contribution to the evaluation and advancement of multimodal models in a domain (cinematography) that is both technically challenging and of growing practical importance, especially with the rise of text to video models. The benchmark, dataset, and model are novel, and the authors have demonstrated thoroughness in both evaluation and addressing reviewer concerns.
Decision: Accept (poster)