Video Scene Segmentation with Genre and Duration Signals
Authors: Jungu Cho
Seong Jong Ha
Hae-Gon Jeon
Venue: ICLR 2026 Poster (2026)
Keywords: Video Scene Segmentation
Movie Scene Boundary Detection
Video Temporal Segmentation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Video scene segmentation aims to detect semantically coherent boundaries in long-form videos, bridging the gap between low-level visual signals and high-level narrative understanding.
However, existing methods primarily rely on visual similarity between adjacent shots, which makes it difficult to accurately identify scene boundaries, especially when semantic transitions do not align with visual changes.
In this paper, we propose a novel approach that incorporates production-level metadata, specifically genre conventions and shot duration patterns, into video scene segmentation.
Our main contributions are three-fold:
(1) we leverage textual genre definitions as semantic priors to guide shot-level representation learning during self-supervised pretraining, enabling better capture of narrative coherence;
(2) we introduce a duration-aware anchor selection strategy that prioritizes shorter shots based on empirical duration statistics, improving pseudo-boundary generation quality;
(3) we propose a test-time shot splitting strategy that subdivides long shots into segments for improved temporal modeling.
Experimental results demonstrate state-of-the-art performance on MovieNet-SSeg and BBC datasets.
We introduce MovieChat-SSeg, extending MovieChat-1K with manually annotated scene boundaries across 1,000 videos spanning movies, TV series, and documentaries.

Published as a conference paper at ICLR 2026
VIDEO SCENE SEGMENTATION WITH GENRE AND DURATION SIGNALS
Jungu Cho Seong Jong Ha', Hae-Gon Jeon
1 AI R&D Division, CJ Corporation
2 Al Graduate School, GIST
Department of Artificial Intelligence, Yonsei University
ABSTRACT
Video scene segmentation aims to detect semantically coherent boundaries in long-form videos, bridging the gap between low-level visual signals and high- level narrative understanding. However, existing methods primarily rely on visual similarity between adjacent shots, which makes it difficult accurately identify scene boundaries, especially when semantic transitions do not align with visual changes. In this paper, we propose a novel approach that incorporates production- level metadata, specifically genre conventions and shot duration patterns, into video scene segmentation. Our main contributions are three-fold: (1) we lever- age textual genre definitions as semantic priors to guide shot-level representation learning during self-supervised pretraining, enabling better capture of narrative coherence; (2) introduce duration-aware anchor selection strategy that pri- oritizes shorter shots based on empirical duration statisties, improving pscudo- boundary generation quality; (3) we propose test-time shot splitting strategy that subdivides long shots into segments for improved temporal modeling. Experimen- tal results demonstrate state-of-the-an performance MovieNet-SSee and BBC datasets. We introduce MovieChat-SSeg, extending MovieChat- IK with manu- ally annotated scene boundaries across 1,000 videos spanning movies, TV series, and documentaries.
INTRODUCTION
Understanding long-form video content requires structuring it into semantically meaningful units that reflect the underlying narrative. Unlike short clips, narrative-driven videos such as movies, TV shows, and documentaries are naturally organized into distinct scenes that maintain narrative co- herence. Segmenting such content into scenes provides useful abstraction that supports wide range of downstream tasks, including content-based retrieval (Bain et al. (2020 202T)). ),, summariza- tion (Gan et al. (2023); Argaw et al. (2024)), and multi-modal question answering (QA) (Song et al. 2024 Chandrasegaran et al. (2024)). However, current approaches (Chen et al. (222): Wu et al. 2022 Mun et al. (2022); Islam et al. (2023); Yang al. (2023); Chen et al. (2024)) predominantly use visual features to identify scene boundaries, leading to errors when narrative transitions occur without accompanying visual discontinuities. These limitations highlight the need to move beyond purely visual approaches toward methods that can leverage richer contextual information.
Professional video production offers rich contextual cues through deliberately crafted elements such as editing patterns, camera movements, and temporal structures that signal narrative transitions. Among these production elements, genre conventions and shot duration patterns offer particularly promising signals for computational scene analysis. However, leveraging these signals presents sev- cral challenges: genre information is typically available only at the video level, while shot duration patterns vary significantly across genres and production styles. Addressing these challenges requires new methods that can integrate production metadata into scene segmentation while handling their inherent variability and granularity mismatches.
We propose to encode genre conventions through textual descriptions and use them as semantic priors for shot representation learning. Our framework integrates this genre information during
Cooresponding author

---

Published as a conference paper at ICLR 2026
Shot Sequence
Shot Encoder (in Section 3.3)
Representation Sequence
(hirzmes)
Sampled
Affinity Mapping
Projection Matrix
auchor
Attention
104
Boundary
413
Seurch Space
Genre Prompt
114
Geure embedding (Precomputed)
"[Sei-Fij, film [should be based on speculative scientific discoveries developments, environmental changes, spuce travel, life planets.]"
Sampled
Text Eacoder
Drama
Comedy
anchor
Documentary
102
Action
Crime
Romance
Western
139
Horor
Adventure
Pseudo Boundary
Generation (in Section 3.3)
Figure 1: Our framework integrates genre emheddings into the shot encoder via affinity- based resid- ual connections and uses duration-aware anchor sampling generate cecudobboundarie for self- supervised shot representation learning in scene segmentation.
pretraining enhance shot-level representations. We construct textual prompts from IMDb-based genre definitions and compute affinities with visual features to provide semantic guidance. This approach aims to capture semantic relationships that may not be apparent from visual features alone.
Building on prior self-supervised learning frameworks based on pseudo scene boundaries, we in- troduce duration-aware anchor sampling strategy that leverages shot duration patterns improve representation learning. Our approach prioritizes shorter shots as anchors to provide more diverse training samples, as these shots are more frequently distributed across different temporal contexts within sequences. Specifically, we model the empirical shot duration distribution within each video and assign higher sampling weights to shots the shorter duration percentiles during anchor pair selection for pseudo-houndary generation. This strategy enables the formation of diverse subse- quences within fixed-length sequences, providing varied training samples compared to fixed anchor selection methods (Mun al. (022); Islam et al. (2023)
We propose a test-time shot split strategy to handle long shots containing multiple semantic segments during inference. Our strategy subdivides shots exceeding duration threshold (10 seconds) into smaller segments, each represented by a keyframe. This preprocessing approach requires no model retraining and be applied to existing
segmentation frameworks.
We introduce MovieChat-SSeg a scene segmentation benchmark derived from MovieChat- 1 Song et al. (2024) with 1,000 annotated videos. The dataset contains manually annotated scene bound- aries, created through a multi-annotator process. The dataset includes diverse content types spanning movies, TV series, and documentaries. Each video represents a segment extracted from longer con- tent, enabling evaluation across different narrative contexts.
Our main contributions are summarized as follows:
We incorporate genre conventions as semantic priors by encoding textual definitions from IMDb guidelines, enabling conre-guided shot representation learning during pretraining.
We introduce duration- -warre anchor sampling strategy that prioritizes shorter shots based on empirical duration distributions, generating more diverse pseudo-boundaries compared to fixed-anchor approaches.
We propose a test-time shot splitting strategy that subdivides long shots into segments during inference, requiring no model retraining or architectural changes.
We introduce MovieChat- SSeg, a scene segmentation benchmark with 1,000 manually an- notated videos spanning films, TV series, and documentaries.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORKS
2.1 VIDEO SCENE SEGMENTATION
Video scene segmentation typically begins with shot boundary detection (Castellano) 2014 4); Soucek & Lokoe 2024)) to identify basic temporal units. Traditional methods group consecutive shots into scenes using low-level visual features such as color histograms (Rui et al. (1999); Chasanis et al. (008 and motion patterns (Rasheed & Shah (2005)). However, these feature-basee approaches often fail to capture higher-level semantic relationships between shots, limiting their effectiveness on complex narrative content.
The release of MovieNet (Huang et al. (2020)) provides a substantial benchmark for scene seg- mentation research with annotations across multiple movie genres. Supervised approaches (Huang) et al. (2020); Rao et al. (2020a) using this dataset achieve strong performance but require exten- sive manual annotations, limiting their applicability to large-scale unlabeled video collections. To overcome this limitation, recent works adopt self-supervised learning frameworks that leverage unla- beled videos. These approaches (Mun
[2022); Islam et al. (2023) Yang et al. (2023) typically follow a two-stage paradigm: (i) pretraining a shot encoder on an unlabeled large-scale dataset using pseudo-bounde boundaries generated by pretext tasks, and (ii) fine-tuning on smaller labeled dataset.
Various pretext tasks have been proposed for self- scene segmentation, including methods based on shot similarity (Chen based movie similarity (Chen et al. (202)). BaSSL Mun et al. (2022) introduces a method to generate pscudo scene boundaries by using the two fixed anchor shots located at the ends of a given shot sequence. The model then learns contextual representations by encoding the relationships across shots within the sequence. Subsequent approaches extend this framework by incorporating long-range inter-shot relationships (Islam al. (2033) and multi-scale temporal context (Yang al. (2023). While these methods show promising results, most rely primarily on visual and temporal cues, with limited exploration of comprehensive production-level metadata for semantic guidance.
2021 ) scene consistency (Wu al. (2022)), and metadata-
Movies2Scenes (Chen et al. 2023)) explores the use of metadata tags, such as co-watch patterns, genre labels, and synopsis text, for scene grouping. Specifically, it measures movie-level similarity using these metadata and constructs positive scene pairs across movies with similar tags, which are used train contrastive learning objective. This work demonstrates the potential of leveraging metadata for scene-level analysis, focusing on inter- metadata comparison.
VSS-MGP (Bouyahi & Ayed (2020) employs multimodal signals to train a classification model for the four most frequent genres, then performs scene segmentation by computing similarity between adjacent shots based on their shot-level genre prediction distributions. Movie-CLII (Zhang et al. (2024) proposes a multi-modal fusion architecture originally designed for genre prediction and shows this structure can be applied to scene segmentation. However, VSS-MGP (Bouyahi Ayed (2020) requires shot-level genre predictions, while Movie-CLIP (Zhang al. (2024)) does not explicitly utilize genre information for segmentation.
Beyond visual features, shot duration patterns have been recognized in film studies as indicators of narrative structure (Reisz & Millar (197I); Salt (2009)). However, the considerable variation in duration distributions across genres and production styles (Cutting & Candan (2015)) makes it chal- lenging to develop consistent supervision strategies, motivating the need for adaptive approaches.
2.2 LONG-FORM VIDEO UNDERSTANDING
Understanding long-form videos, including video question answering (Tapaswi et al. (2016); Song al. (204)) and narrative summarization (Song et al. Rohrbbch al. (2055). requires mod- cling semantic structures
extended temporal ranges. These tasks typically operate higher- level narrative units, such as scenes or events, rather than frame-level analysis. Scene segmentation provide useful structural units for such tasks by defining semantically coherent segments for temporal reasoning and content organization.
Recent datasets for long-form video understanding, such MovieChat- 1K (Song al. (2024) and TVQA (Lei et al. (018)), incorporate textual annotations including subtitles and dialogue to support multimodal reasoning tasks. However, many of these datasets lack explicit scene boundary annota-
3

---

Published as a conference paper at ICLR 2026
tions, limiting opportunities to study scene-level video organization. For instance, MovieChat-Ik focuses on multimodal QA tasks and does not include scene boundary annotations.
METHOD
3.1 TASK DEFINITION
Narrative-driven videos, such movies and TV shows, are typically structured into four-level hierarchy: frames, shots, scenes, and the whole video. A shor refers to sequence of consecutive frames captured without interruption by single camera, and can be reliably identified using low- level visual cues (Cotsaces et al. (2006); Castellano (2014)). In contrast, a scene is semantically cohesive unit composed of multiple shots that maintain continuity in action, location, and time (Katz (1979) ) serving as a key structural element of narrative storytelling.
While shot boundaries are often associated with visual discontinuities, detecting scene boundaries requires understanding higher-level semantic relationships, such as thematic coherence, location changes, and temporal continuity. Video scene segmentation aims to identify these boundaries by determining whether given shot marks the end of scene. This can be formulated as binary classification problem for each shot, where the model must integrate visual features with contextual information from surrounding shots to make boundary predictions.
Formally, video can be represented binary boundary labels {y1. .2..... UN }. Here, Vi E {0, 1} indicates whether s, marks the end of a scene. Each shot 8; transformed into a feature vector e; by a shot encoder Op. These shot representations serve as the input for boundary prediction, enhanced by auxiliary semantic signals described in the following sections.
sequence of N shots {si, $2...... with corresponding
3.2 GENRE-GUIDED SHOT REPRESENTATION
Genre conventions influence the visual and narrative elements of scenes, potentially providing mantic context for understanding scene structures. While visual appearances may vary across shots, genre conventions may provide consistent semantic signals that could aid in identifying narrative herence within scenes. However, applying genre directly is nontrivial, as most movies are associated with multiple genre tags, making difficult assign a unique genre each shot or scene.
To address this, we utilize genre conventions as a soft semantic prior during the self-supervised pre- training stage. We construct senre-specific textual prompts based on IMDb definitions (IMDb.com 2223)), incorporating characteristic visual and narrative descriptions for each genre. These prompts are encoded into embedding vectors using the text encoder from CLIP (Radford et al. (2021)), form- ing genre embedding set Ge N, where Ng is the number of genre categories and D is the embedding dimension.
Our shot encoder incorporates pre-computed genre embeddings as fixed parameters within the VIT (Dosovitskiy et al. (2020) architecture. Within the ViT layers, we compute affinities between visual token features V. N.D and the stored genre embeddings Ge using cosine similarity. The resulting affinity matrix A represents the relevance each genre to the visual tokens. The genre embeddings are then projected and integrated with the original visual features via residual connections:
V.G. VD
A softmax
V genre
V + AG.W,
(I)
where W E DxD projects the weighted genre information before integration. Following the VIT architecture, the enhanced token features V.geare are aggregated to form the final shot-level representation 64-
3.3 DURATION- AWARE SHEUDO-BOUNDAR GENERATION
Existing gecudobooundan generation methods, particularly BaSSL MMun al. (2022 )) fixed anchor positions at sequence endpoints, limiting their ability to capture diverse temporal patterns in video sequences. We observe that scenes often exhibit consistent shot duration patterns, with
4

---

Published as a conference paper at ICLR 2026
many scenes dominated by shorter shots that form coherent narrative units. To leverage this insight, we modify the anchor selection mechanism by introducing a duration- aware sampling strategy that prioritizes shorter shots over fixed endpoint selection.
Instead of selecting fixed anchors, we impose higher sampling probabilities based on inverse- normalized shot durations within sequence. We sample anchor shots {st, 8,) from the sub- sequences on both sides the target shot based on the sampling probability P(s;):
1/d;
P(s:)
2jm1 1/dj
where di represents the duration of the i-th shot, and N is the total number of shots in the subse- quence. We then search for a pseudo-boundary b" between these anchor shots using a similarity- based approach adapted from Dynamic Time Warping (DTW) (Berndt & Clifford (1994)):
(
(+12-1 Σ sim(er, e44))+ b b+L+1 1=-L
6" (i,2,))
argmax
Σ 2=b+1
sim esm( (3)
- b
where sim(2, y) is cosine similarity between shot representations, is the target shot index, and L, R denote distances left and right anchors respectively. This process divides the input sequence into two pseudo-scene sequences: S (Si-L...., 86- and 5 = {8b++1, 8,+
3.4 TEST-TIME SHOT SPLIT STRATEGY
While our training methods enhance shot representation learning, real-world inference faces chal- lenges from highly variable shot durations. Long shots may contain multiple semantic segments that are difficult to process as single units, potentially affecting boundary detection accuracy in requence-based approaches. To address this challenge, we propose a test-time shot split strategy that subdivides long shots during inference without requiring model retraining.
For any shot with duration d; > T (set to 10 seconds), we partition it into three equal-lengh segments, each treated as an independent shot. When pre-extracted keyframes are available (e.g., three keyframes shot in MovieNet (Huang al. (2020)) we directly assign them to the segments; otherwise, we extract keyframes from each segment using uniform temporal sampling:
{s}
(3) split(s;) if >
The strategy operates at the preprocessing level, making it compatible with existing architectures. After boundary prediction, detected boundaries are mapped back to original temporal coordinates.
4 MOVIECHAT-SSEE DATASET
We present MovieChat-SSeg, a manually annotated benchmark for the evaluation of video scene segmentation. The dataset builds upon the publicly available MovieChatIKK (Song al. (2024)) dataset, which contains 1,000 video clips originally curated for long-form video-language tasks. We extend these clips by manually annotating scene boundaries to create a testbed for scene segmenta- tion research.
Annotation Process. Scene boundaries are annotated by trained annotators following established guidelines that consider narrative coherence, location changes, and temporal transitions. The an- notation process involves multiple stages: initial independent annotation, discussion of boundary cases, and consensus resolution for disagreements. All annotations are reviewed to ensure consis- tency across the dataset.
Dataset Characteristics. MovieChat-SSS complements existing benchmarks by including diverse content types spanning movies, TV series, and documentaries. This content variety enables eval- uation across different narrative structures and editing styles. This diversity enables evaluation of scene segmentation methods across varied editing styles and storytelling approaches.
The dataset consists of focused segments (average 7.4 minutes) that enable efficient annotation while capturing multiple scene transitions per video. However, this focus on shorter segments may not capture long-term narrative patterns present in full-length content.
5

---

Published as a conference paper at ICLR 2026
We release MovieChat-SSeg publicly with annotation guidelines and evaluation protocols to support future research.
Dataset
#Title
#Video
Time(h)
#Shot
Category
BBC
11
4.9K
547
Documentary
(es))e))-SSeg(ees))
64
64
132
106K
8.IK
Movic
MovieChat (Sur)
143
IK
124
116K
7.8K
Movie, Documentary, TV series
Table Comparison of video scene segmentation dataset characteristics.
5 EVALUATIONS
5.1 DATASET AND EVALUATION METRICS
We use MovieNet (Huang et al. (2020) with 1,100 movies for pre-training. For supervised training and evaluation, utilize the MovieNet-SSSe subset, which includes 318 movies with scene bound- ary annotations, following the official training(190), validation(644, and test(64) splits. To assess the generalization ability of our model, we conduct zero-shot evaluation on both the BBC (Baraldi et al. 2015 dataset and our newly proposed MovieChat-SSeg dataset.
We evaluate model performances using Average Precision (AP), AUC-ROC, and F1 score for shot- level classification, and mean Intersection over Union (mloU) for sequence-level segment alignment.
5.2 TRAINING STRATEGY
Following prior works (Mun al. (2022): Islam al. (2023), ), adopt modular architecture consisting of a shot encoder, a context encoder Uc. and prediction head hy. The context encoder models temporal dependencies across shots, and the prediction head outputs binary scene boundary predictions. During pretraining, the shot encoder is optimized using pseudo- boundaries and then frozen during fine-tuning, where the context encoder and prediction head are updated only.
We employ two objective functions in the pretraining stage. First, we apply a contrastive loss based on InfoNCE (Oord al. (2018)), where the average representation of shots within a pseudo scene treated scene-level feature, and paired with an anchor shot from the same scene, as follows:
exp(sim(e,
(e, a) log
(S)
exp(sim(e, ē)) + +e,, exp(sim(en,e) + Ee,, exp(sim(e, Ex))
where denotes the anchor shot representation, and é represents the average feature of pscudo scene. en and en represent negative shot representations and negative pseudo-scene features, re- spectively. Since each shot sequence is divided into two pseudo scenes by pseudo boundary index 6", we compute the contrastive loss for both segments: Leon (nno(es, Et) + Lnne (Er, e, where el and e, are the anchor representations for the left and right pseudo scenes, respectively. El ovg(E,,.., Et.,..., ) and E, = avg(es- +1. +1. er) denote the average scene-level features of each segment. Second, we apply a binary cross-entropy loss to distinguish between the pseudo-boundary shot and a non-boundary shot randomly selected from the given sequence as below:
Lpo(co-, C,) log(h,((c-)) log(1 hp(c)))
where c-. and Cn denote context-aware shot representations of the boundary shot, respectively, obtained from the context encoder. The function h.,(.) denotes the prediction head, which maps the input representation objective in the pretraining stage is defined as the linear combination of the contrastive loss and the binary cross-entropy loss for pscudo- boundarry prediction.
and non-
the probability of a boundary. The overall
During fine-tuning, train the context encoder and prediction head using scene boundary labels with a standard binary cross-entropy loss as follows:
Lus (c, y:) yt log(h, + - ys) log(1 hp(())),
where c, is the context-aware representation of the i-th shot and ys {0, 1} is the ground truth scene boundary label. At inference time, we predict a shot as a scene boundary if the boundary probability hp( is greater than or equal to 0.5.

---

Published as a conference paper at ICLR 2026
Method
AP
AUC
mloU
Method
A1
A2
A3
A4
A5
GraphCut (Rasheed & Shab
2005
20.70
ShrrCol.
29.9
30.8
31.5
26.5
21.3
28.0
SCSA (Chisams el aL
2008
14.70
30.50
SCRL
32.5
32.5
33.3
28.4
24.3
30.2
BaSSI.
31.7
32.3
22,2
Scone Tillng
Wang et al
2025
19.95
72.15
29.36
36.96
TranS4mer'
35.6
33.6
29.8
27.6
25.0
30.3
Story Graph
Tapaswi ot
25.10
35.70
Ours
42.2
42.1
42.8
32.8
26,3
37.2
Grouping Rolman
2016
33.60
37.20
(b) Results on the BBC. The table presents
Siamese Baraldi
2015)
35.80
39.60
Average Precision (AP) scores computed
MS-I.TM [Hunng ot al (2:10
46.50
46.20
LGSS Rao Stall A0400
47.10
48.80
against ground truth annotations from five
Movie-
LTP
2024
54.45
independent expert annotators.
MIIRT Wei We al 20225
$4,80
90.30
46.30
$1.20
ShotCoL (Chen ct 2021
53.40
(Waret (2122))
2Neenss hen et
2023
55.03
CMS Park et al, 20241)
55.73
52.05
BaSSI (Mun
2V22
57.40
90.54
47.02
50.69
Method
Movic
TV
Docu.
Total
CAT
Yang st 2
59.55
91.81
51.93
53.67
TranS4mer (Isaan
2023
48.35 36
51.91
BaSSL"
37.6
TrunS4mer"
38.8
46.7
24.4
37.9
Ours
92.84
58.88
Ours
48.1
54.6
46.7
(a) Results on the MovieNet-SSS
(c) Results the MovieChat-SSeg
Table 2: Comparison results across three video scene segmentation benchmark datasets. In the
(b) BBC dataset and (c) MovieChat-SSeg dataset, evaluation results are presented using Average Precision (AP). The best numbers are in bold. The second best is in underline. implementation)
5.3 IMPLEMENTATION DETAILS
We use ViT-B/32 Dosovitskiy al. (020 the shot encoder backbone, initialized with Open- CLIP (Cherti et al. (023)) weights, processing 224x224 resolution images with 32 patch size.
We freeze the pre-tr ViT backbone weights and train only the genre integration module (pro- jection matrix W) and the final MLP classification head. Following (Huang et al. (2020); Zhhng ett al. 2024)), we use genre categories with textual definitions from IMDb (IMDb.com (2023), encoded using the OpenCLIP text encoder. For sequence-level boundary prediction, we employ BERTiDevlin
two-layer BERT Devlin et al. (2018) architecture with 768 hidden dimensions, trained from scratch.
We adopt two-stage training strategy. During pretraining, we train the shot and context encoders for 20 epochs with batch size of 256, using a base learning rate of 0.3 with cosine decay and linear warmup for 1 epoch. Fine-tuning is performed for 20 epochs using batch size of 1,024. We use the Adam (Kingma| (2014)) optimizer with a learning rate of 2.5 x 10 and a momentum of 0.9. All models trained on eight A100 GPUs. Results averaged over five runs with different fine-tuning seeds.
5.4 COMPARISON RESULTS
Results the Movie Dataset We compare model with the state-of-the-art methods MovieNet-SSeg. and report the results Tab. a). Pseudo-houndant boundary-based self-supervised meth- ods (Mun al. (2022); Yang et al. (2023): Islam al. (2033) achieve strong performance compared to traditional approaches. Our method shows consistent improvements over existing approaches us- ing various backbone architectures, including significant gains over methods employing the same VIT backbone (Islam et al. (2023)). This demonstrates that our contributions of genre guided rep- resentation learning and duration- aware sampling provide complementary benefits beyond architec- tural choices. Qualitative analysis in Fig. 2 illustrates improved boundary detection in challenging scenarios.
genre-guided
Cross-domain Evaluation on BBC Dataset For the cross-domain evaluation, we conduct an addi- tional experiment on the BBC dataset (Baraldi al. (2015)). The BBC dataset contains nature and wildlife documentaries with different narrative structures from MovieNet's theatrical content. The dataset provides multiple expert annotations for the same videos, enabling evaluation against diverse boundary interpretations. Our method achieves competitive performance compared to existing ap- proaches, as shown in Tab. bb) These results suggest that our approach generalizes reasonably well to documentary content beyond theatrical films.

---

Published as a conference paper at ICLR 2026
Index Method
Backbonc
Weight Init.
Train. Params
|nchhrr] Split Select.
MovieNet-SSog
BBC(AP)
MovieChat-SSeg(APP
Thre.
AP
AUCC
Fi
|mloU]
Movie]
TV
Docu.
Total
Ours
VIT-R/32
CL.IP
39M
Side
62.82
92.39
58.00
59.15
36.11
46.91
53.28
28.35
43.94
2 BaSSL"
VIT-B/32
CLIP
102M
Side
60:40
91.46
45.76
50.04
31.12
38.69
45.83
29.40
38.05
TranS4mer"
VII-B/32
CLIP
Side
60.32
91.44
47.04
46.57
38.34
BaSSL
ReaNet-50
scratch
56.82
190.00
442.51
34.73
5
BaSSL"
ResNet-50
scratch
39M
IDW
57.65
90.47
43.82
48.87
35.
44.69
49.72
24.31
40.27
TranS4mer"
VIT-S/32
Imagenet
43M
Side
58.60
90,92
47.07
50.29
29.68
33.76
41.35
19.47
31.97
TranS4mer
VIT-S/32
Imagenet
43M
IDW
61.17
91.78
49.09
52.42
34.11
46.56
53.74
25:42
42.69
8 BaSSL" ReaNet-50
scratch
39M
Side
10s
57.40
90.18
52.86
56.00
33.32
42.71
[45.82]
24.15
400.49
9
TranS4mer"
VIT-S/22
Imagenet
43M
Side
10s
59.42
91.21
46.91
50.52
32.14
39.53
41.72
23.40
(37.77
Table 3: Performance comparison video scene segmentation benchmarks. Methods are grouped by backbone architecture, with our contributions (duration-awart anchor sampling and test-time shot splitting) applied state-of-the-art baselines. ("our implementation)
Evaluation on MovieChat-SSeg Dataset MovieChat-SSee offers a diverse evaluation benchmark with video content from three distinct formats: movies, TV series, and documentaries, Each format differs in narrative structure, editing style, and scene organization, making valuable for assessing the generality of models across various storytelling conventions. As shown in Tab. Cc), our model achieves competitive performance across all formats, with an overall improvement of +8.8 points over TranS4mer (Islam et al. (2023)). The improvements are consistent across different content types, suggesting that our combined approach genre-guided learning and duration-aware sampling provides benefits across varied narrative contexts.
Comparison with Baseline Methods To ensure fair comparison, we re-implement BaSSL and TranS4mer with the same ViT-B/32 backbone and CLIP initialization. As shown in Tab, B both achieve competitive performance, confirming our gains stem from methodological rather than ar- chitectural advantages. Applying our duration-aware sampling these baselines yields consistent improvements (+0.8-2.1 AP on MovieNet), while test-time splitting provides additional gains (+0.6- 0.8 AP on MovieNet) without retraining. These results suggest our contributions show promise for broader applicability.
5.5 ABLATION STUDIES
We conduct ablation studies to evaluate our design choices and examine the impact of various con- figurations. Our analysis focuses on four key components: (1) integration methods for incorporating genre embeddings into shot representation learning (Tab. 4 rows 1-3, 8); (2) prompt design strate- gies for genre embedding generation (rows 4-5, 8); (3) parameter settings for duration-aware anchor sampling (rows 6-7, 8); (4) threshold configurations for test-time shot splitting across different du-
ration settings (rows 8-11).
Genre Embedding Integration To assess the impact of genre embedding integration, compare four strategies: no genre embedding, frame-level concatenation, token-level concatenation, and our
Cronching Tiper, Hidden Drngon(th) (11191332) (11332)-100 00146 0054
Jus81
-TarS4mss
-Ous
Top Gun/UD092009) 07788 - 0796
lussi
Oun
Figure 2: Qualitative examples of scene boundary detection. Top: visually dynamic single scene where method avoids over-prediction. Bottom: A cross-cuttigg sequence where approach accurately detects boundaries between scene translations. (GT: Ground Truth, BaSSL Mun et al.
2022) and TranS4mer Islam al. 2023 baseline methods)
8

---

Published as a conference paper at ICLR 2026
Index
Embedding
Prompe
Anchor
Split Thre.
MovieNet-SSog
BBC(AP)
MovisChat-Scgg(A))
AP
AUC
mloU
Movie
TV
Doeu.
Total
w/a Embed
Def
IDW
61.27
91.99
57.55
58.61
32.16
46.33
51,24
16.15
42.03
2
Frame Concat
Def
IDW
59. 13
91.36
55,29
57.08
32.25
44.25
48.27
23.22
39.47
Token Concal
Def
IDW
62.43
36.90
47.73
53.69
28.07
44.26
AG-Residual
Name
IDW
63.22
36.59
46.96
53.36
28.75
44.08
AG Residual
Style
IDW
63.42
92.75
58.76
59.56
36.39
47.09
53.10
26.75
43.23
6
AG Residual
Def
Side
62.82
92.39
58,00
59.15
36.11
46.91
53.28
28.35
43.94
AG-Residual
Def
DW
63.44
92.73
36.69
47.61
54.30
28.84
44.39
AG-Residual
Def
IDW
63.62
37.22
54.57
29.41
46.69
9
AG-Residual
Def
IDW
60s
63.63
92.83
58.97
59.86
37.28
48.15
54.58
29.45
46.75
10
AG-Residual
Def
IDW
30s
63.76
92.84
59.17
60.14
37.57
48.20
54.50
29.53
46.78
AG-Residual
Def
IDW
10s
63.80
92.81
59.60
60.38
38.87
48.56
54,86
29.62
46.83
Table 4: Ablation studies on MovieNet-SSeg. BBC, and MovieChat-SSeg datasets. We evaluate
genre embedding integration methods (rows 1-3, 8), prompt strategies (rows 4-5, 8), anchor sam- pling approaches (rows 6-7, 8), and test-time shot splitting thresholds (rows 8-11).
affinity-basse -baeed residual method. Frame-level concatenation selects the most similar genre embedding for each shot and concatenates it with the shot-level representation, while token-level concatenation adds genre embeddings to the image token sequence before passing through ViT attention blocks. As shown in Tab. Iowws 1-3, 8), frame-level concatenation actually degrades performance com- pared to using no genre information (59.13 vs 61.27 AP), while token-level concatenation provides modest improvements. Our affinity-based residual approach achieves the best performance (63.62 AP), demonstrating that dynamic integration of genre information through attention mechanisms more effective than simple concatenation strategies. These results indicate that the method of incorporating genre metadata significantly impacts its effectiveness for scene segmentation.
Genre Prompt Design We evaluate three genre prompt strategies generate genre embeddings: name-based (using only genre names), style-based (describing visual and narrative characteristics). and definition-based (incorporating comprehensive IMDb definitions). As shown in Tab. 4 rows 4-5, 8), definition based prompts achieve the highest performance (63.62 AP), outperforming name- based approaches by 0.39 points. This suggests that more detailed textual descriptions provide better semantic guidance for genre-aware representation learning. The complete list of prompts used for each genre is provided the appendix.
Shot Duration We evaluate three anchor sampling strategies for pseudo-boundary generation: sampling that prioritizes longer
side-based selection using fixed endpoints, turraton-weightediie shots, and our inverse-duration wwrre--duration-wwgghteeddDDDD approach that favors shorter shots. As shown in Tab. rows 6-7, 8), our inverse-duration wrreseddurationn-wiigeeddddd strategy achieves the best performance, improv- weighted sampling. This result demonstrates that prioritizing shorter shots as anchors provides more diverse training signals and enhances representation learning compared to conventional or length- biased approaches.
ing AP by 0.8 over the side-based baseline and 0.18 over duration-weiihted
Shot Split Threshold We evaluate the effectiveness test-time shot splitting with different duration thresholds to address challenges from long shots during inference. As shown in Tab. rows 8-11), progressively smaller thresholds lead to consistent performance improvements: no splitting (63.62 AP) + 60s threshold (63.63 AP) - 30s threshold (63.76 AP) - 10s threshold (63.80 AP). The 10-
second threshold achieves the best performance, suggesting that subdividing long shots into smaller segments improves boundary detection accuracy.
5.6 QUALITATIVE ANALYSTS OF FAILUREE CASES
We analyze the three lowest-performing videos in MovicNet-SSeg, test set to identify systematic failure patterns. Fig. illlssrattes representative cases across five common error types.
Insert shots and non-narrative elements. Brief inserts such as television screens, photographs, or archival footage are frequently misclassified scene boundaries. Our visual-only approach lacks narrative understanding to recognize these storytelling devices maintain continuity with surrounding sequences.

---

Published as a conference paper at ICLR 2026
The Green Mile(t10) 20689) 2239 2247
Durs
Green Mileruod 20689) 1402 1410
Ours
Cube(ro0 23755) 0402
Cerminaror(tt00822411
Oun
The Temminatortr00088244)) 0617-6012
Ouss
Figure 3: Common failure patterns in scene segmentation. Ground truth (GT) boundaries are shown with black arrows; our predictions with red X marks indicating errors.
Lighting variations. Sudden lighting shifts within continuous scenes trigger false boundaries. Lightning flashes, flickering lights, or rapid shadow changes create visual discontinuities misinter- preted scene changes, revealing over-reliance visual continuity cues.
Subtle spatial transitions. Gradual location shifts while maintaining narrative flow are often missed. When scenes expand from interior to exterior between adjacent locations, high visual similarity leads to mnder-segmentation, indicating limited semantic understanding of spatial context.
Extreme framing and unconventional compositions. Extreme close-ups or unusual camera gles create distinctive features that sometimes trigger spurious boundaries, revealing sensitivity to out-of-distribution visual patterns.
Special effects and visual artifacts. Digital effects such as compositing or CGI create scene- irrelevant visual discontinuities misinterpreted as boundaries, producing strong visual cues that mis- lead our appearance-basse approach.
CONCLUSION
In this paper, we present a novel approach for video scene segmentation that leverages production level metadata through genre-guided representation learning, duration-aware anchor sampling, and test-time shot splitting. We introduce MovieChat-SSeg, a scene segmentation benchmark compris- ing 1,000 aunan-annotated video clips across diverse content types, supporting evaluation across varied narrative structures. Extensive experiments demonstrate that our method achieves state- of-the-art performance on multiple benchmarks, with significant improvements over existing ap- proaches across diverse video content types.
Limitations. While our method successfully incorporates genre and temporal metadata, may face challenges in content where visual and production cues are insufficient for scene boundary de- tection, such as dialogue-heavy scenes with minimal visual variation. Additionally, our approach focuses on movie-style content and may require adaptation for other video formats with different production conventions. Our current pretraining relies solely on MovieNet data, and integrating di- verse large-scale video datasets could potentially improve generalization across varied content types. Furthermore, evaluation on broader video categories (e.g., news broadcasts, vlogs, educational con- tent) would provide insights into the method's 's applicability beyond narrative-driven content. Future work could explore additional production metadata and multi-n multi--odal approaches/ Argaw et al. (2023) incorporating audio and dialogue information for more comprehensive scene understanding.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Dr. ChiHoon Lec, Chief AI Officer of the AI R&D Division at CJ Corporation, and colleagues at CJ Corporation, for their insightful discussions and guidance that helped shape the di- rection this work. This work was supported by the National Research Foundation of KoRea(NRF) grant funded by the Korea governmen((MSSTTRR ( Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (R1-2020-1201661).
and Institute of Information &
REPRODUCIBILIIY STATEMENT
To ensure reproducibility, we provide comprehensive implementation details and resources. Our supplementary materials include source code and sample data from our newly created MovieChat- SSeg dataset. Detailed pseudocode for key algorithmic components-including the genre embed- ding integration process and duration-award anchor sampling for pscudo-boundary generation- presented in Appendix Implementation details, including hyperparameters, training procedures, []
and architectural specifications, are provided in Section [.3] Upon acceptance, we will publicly release complete codebase, dataset processing scripts, and the full MovieChat-SSeg dataset to facilitate future research and ensure full reproducibility of our results.
REFERENCES
Dawit Mureja Argaw, Joon- Joon-Young Lee, Markus Woodson, In So Kweon, and Fabian Caba Heilbron. Long-range multimodal pretraining for movie understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Dawit Mureja Argaw, Mattia Soldan, Alejandro Pardo, Chen Zhao, Fabian Caba Heilbron, Joon Son Chung, and Bernard Ghanem. Towards automated movie trailer generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. Condensed movies: Story based retrieval with contextual embeddings. In Proceedings of the Asian Conference on Computer Vision (ACCV). 2020.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. A deep siamese network for scene detection in broadcast videos. In Proceedings of the 23rd ACM international conference on Multimedia, 2015.
Donald Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In Proceedings of the 3rd International Conference Knowledge Discovery and Data mining, 1994.
Mohamed Bouyahi and Yassine Ben Ayed. Video scenes segmentation based on multimodal genre prediction. Procedia Computer Science, 17,11,, 2020.
Brandon Castellano. Pyscenedetect: Intelligent scene cut detection and video splitting tool, 2014.
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video- language understanding. Advances in Neural Information Processing Systems, 37:53168-5311 2024.
Vasilcios T Chasanis, Aristidis Likas, and Nikolaos P Galatsanos. Scene detection in videos using shot clustering and sequence alignment. IEEE transactions on multimedia, 11(1):9--100, 2008.
Lu Chen, Jiawei Tan, Pingan Yang, and Hongxing Wang. Semantic transition detection for self- supervised video scene segmentation. In International Conference on Multimedia Modeling, 2024.
11

---

Published as a conference paper at ICLR 2026
Shixing Chen, Xiaohan Nie, David Fan, Dongqing Zhang, Vimal Bhat, and Raffay Hamid. Shot con- trastive self-superviee learning for scene boundary detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Shixing Chen, Chun-Hao Liu, Xiang Hao. Xiaohan Nie, Maxim Arap, and Raffay Hamid. Movies2scenes: Using movie metadata to learn scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive iaguaage-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, 2023.
Costas Cotsaces, Nikos Nikolaidis, and Ioannis Pitas. Video shot detection and condensed represen- tation. a review. IEEE signal processing magazine, 23(2):28-3) -7,8, 2006.
James E Cutting and Ayse Candan. Shot durations, shot classes, and the increased pace of popular movies, 2015.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1exxiv:8/00.4440 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Bei Gan, Xiujun Shu, Ruizhi Qiao, Haoqian Wu, Keyu Chen, Hanjun Li, and Bo Ren. Collabo- rative noisy label cleaner: Learning scene-aware trailers for multi-modal highlight detection in movies. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2023.
Bo Han and Weiguo Wu. Video scene segmentation using a novel boundary evaluation criterion and dynamic programming. In 2011 I IEEE International conference multimedia and expo, 2011 1.
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In Proceedings of the European Conference on Computer Vision (ECCV),
2020.
IMDb.com. Genres.
gen.res/GZDRMS 6R JRGACH 2023.
Md Mohaiminul Islam, Mahmudul Hasan, Kishan Shamsundar Athrey, Tony Braskich, and Gedas Bertasius. Efficient movie scene detection using state-space transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Ephraim Katz. Ephraim katz's 's the film encyclopedia, 1979.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.600 2014.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. In EMNLP, 2018.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pp. 19730-19742. PMLR, 2023.
Jonghwan Mun, Minchul Shin, Gunsoo Han, Sangho Lee, Seongsu Ha, Joonseok Lee, and Fun- Sol Kim. Bassl: Boundary-aware self-supervised learning for video scene segmentation. In Proceedings of the Asian Conference on Computer Vision (ACCV), 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1 2018.
12

---

Published as a conference paper at ICLR 2026
Jinwoo Park, Jungeun Kim, Jaegwang Seok, Sukhyun Lee, and Junyeong Kim. Contrasting multi- modal similarity framework for video scene segmentation. IEEE Access, 12:32408 2132448-32419, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021.
Anyi Rao, Linning Xu, Yu Xiong, Guodong Xu, Qingqiu Huang, Bolei Zhou, and Dahua Lin. A local-to-global approach to multi-modal movie scene segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020a.
Anyi Rao, Linning Xu, Yu Xiong, Guodong Xu, Qingqiu Huang, Bolei Zhou, and Dahua Lin. A local-to-glohal approach to multi-modal movie scene segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020b.
Zeeshan Rasheed and Mubarak Shah. Detection and representation of scenes in videos. 1EEE
Iransactions on Multimedia, 167):009- 2005.
Karel Reisz and Gavin Millar. The technique of film editing. ERIC, 1971.
Xubin Ren, Lingrui Xu, Long Xia, Shuaiqiang Wang, Dawei Yin, and Chao Huang. Vide- orag: Retrieval-augmenned generation with extreme long-context videos. arXiv preprint 251.022002.01449, 2025.
Anna Rohrbach, Mareus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie descrip- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2015.
Daniel Rotman, Dror Porat, and Gal Ashour. Robust and efficient video detection using optimal sequential grouping. In 2016 IEEE international symposium on multimedia (ISM), 2016.
Yong Rui, Thomas Huang, and Sharad Mchrotra. Constructing tablc-of-content for videos. Mul- timedia systems, 7:359-368, 1999.
Barry Salt. Film style and technology: History and analysis. Starword, 2009.
Minho Shim, Young Hwi Kim, Kyungmin Kim, and Seon Joo Kim. Teaching machines to under- stand baseball games: large-scale baseball video database for multiple video understanding tasks. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token sparse memory for long video understanding, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tvsum: Summarizing web videos using titles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Tomás Soucek and Jakub Lokoc. Transnet v2: An effective deep network architecture for fast shot transition detection. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11218-112211 2024.
Jiawei Tan, Hongxing Wang, Jiaxin Li, Zhilong Ou, and Zhangbin Qian. Neighbor relations matter in video scene detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024a.
Jiawei Tan, Pingan Yang, Lu Chen, and Hongxing Wang. Temporal scene montage for self- supervised video scene boundary detection. ACM Transactions on Multimedia Computing, Com- munications and Applications, 20(7):1-19, 2024b.
Jiawei Tan, Hongxing Wang, Kang Dang, Jiaxin Li, and Zhilong Ou. Modality aware shot relating and comparing for video scene detection. In Proceedings of the AAAI conference on artificial intelligence, 2025.
13

---

Published as a conference paper at ICLR 2026
Makarand Tapaswi, Martin Bauml, and Rainer Stiefelhagen. Storygraphs: visualizing character interactions as a timeline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Yuxuan Wang, Yiqi Song, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long streaming video understanding with recurrent memory bridges. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), 2025.
Xi Wei, Zhangxiang Shi, Tianzhu Zhang, Xiaoyuan Yu, and Lei Xiao. Multimodal high-order relation transformer for scene boundary detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20001-2000000 1-22090, 2023.
Haoqian Wu, Keyu Chen, Yanan Luo, Ruizhi Qiao, Bo Ren, Haozhe Liu, Weicheng Xie, and Linlin Shen. Scene consistency representation learning for video scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Yang Yang, Yurui Huang, Weili Guo, Baohua Xu, and Dingyin Xia. Towards global video scene segmentation with context-aware transformer. Proceedings of the AAAI conference on artificial intelligence, 2023.
Zhongping Zhang. Yiwen Gu, Bryan A. Plummer, Xin Miao, Jiayi Liu, and Huayan Wang. Movie genre classification by language augmentation and shot sampling, In IEEE Winter Conference on Applications of Computer Vision (WACV), 2024.
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Wang HongFa, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Cai Wan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Language- bind: Extending video-language pretraining to n-modality by language- semantic alignment, 2023.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
Our supplementary material provides comprehensive details complementing the main paper. It cov- ers the MovieChat-SSeg dataset, including annotation procedures, quality assurance, and statisti- cal analyses in Section B We validate genre embeddings through zero-shot classification, explore prompt design strategies, analyze embedding similarities across different vision-language models, examine various types of production metadata and conduct clustering analysis to further validate Additionally, we assess robustness through genre vocabu- lary reduction experiments (21-6 genres) and analyze performance trends across videos with 1-5 genres per video in Sectior[] Section D discusses pseudo- bouud-b search techniques within shot sequences and validates our duration-aware anchor sampling approach. We analyze the computa- tional overhead of test-time shot splitting, evaluate performance-efficiinnn ent duration thresholds, and demonstrate the practical feasibility of our approach for deployment in Section We provide comparisons with recent supervised methods in Section [ Finally, addi- tional qualitative results are presented for the BBC dataset, generalization analysis on non-narrative formats (lectures, gaming) from the LongerVideos dataset and practical application the BBDB baseball dataset in Section H
our learned representations in Section C
trade-offss across differ-
B MOVIECHAT-SSEC DATASET
The MovicChat-SSeg dataset extends the MovieChat-IK (Song et al. (2024) dataset by adding scene boundary annotations to 1K videos. These videos trimmed about 8 minutes each and include a variety of content types, such as movies, TV shows, and documentaries. A scene consists of one or more shots, and the annotations are manually created following the shot-based approach used in MovieNet (Huang et al. (2020)).
Annotation Process The videos divided into shots using shot boundary detection (Castellano (014)). Within each shot, keyframes are extracted through uniform sampling, and we select 3 keyframes. As shown in Fig. 4 we develop a web-based annotation tool. Annotators view the shot sequence composed of keyframes, as shown in Fig. 4 a). and determine whether the target shot is scene boundary. To provide contextual information, surrounding shots are also displayed. Annotators are given the option to view 3, 5, or 7 shots at once, with most annotators using 5 consecutive shots. The timeline at the bottom of the tool allows annotators to use shot duration, which can be helpful for ambiguous cases, such long takes, where keyframe-basee judgment alone might be insufficient. Annotators can switch to video mode and review the actual video, as shown in Fig. bb).
Annotation Quality Annotators receive training before beginning their annotation tasks. This training includes hands-on practice with the MovieNet-SSeg dataset, where annotators perform annotations and compare their results with the ground truth (GT). This process helps them bet- ter understand the concept of scene boundaries. To ensure high-quality annotations, we employ multi-annotator process: each video is independently annotated by two, and any disagreements are resolved by a third annotator. On average, annotation takes approximately 2 minutes per video by each annotator. The average agreement rate between the first two annotators is 38%. The third annotator reviews the video and makes corrections based on the annotations provided by the first two.
Statisties As shown in Tab. 5 the dataset contains diverse video formats exhibiting distinct shot and scene characteristics. For example, documentaries tend to have longer shots and scenes, whereas TV shows have more shots and fewer scene transitions compared to movies. Unlike prior datasets Huang et al. 22000); Rotman et al. (2016). MovieChat-SSeg enables robust evaluation across varied narrative styles and editing conventions. By linking videos to IMDb-base genre tags, we confirm broad genre coverage. Combined with its diverse shot and scene structures, MovieChat- SSeg serves as a valuable benchmark for developing generalizable scene segmentation methods across domains.
15

---

Published as a conference paper at ICLR 2026
(a) Shot Mode
(a) Video Mode
Figure 4: Annotation interface. In Shot Mode, annotators assess scene boundaries based on keyframe sequences, while in Video Mode, they can review the actual video to refine their anno- tation decisions.
Type
#Videos
Avg Duration(s)
Arg Shots
Avg Seenes
Genre Tags
Movie
20XExcepi Documentary)
TV Show
252
421.41
132.83
7.15
StAct, Adv, Cri, Dra. Fan, Mys, Rom, Thr)
Documentary
217
448.48
90.00
5(Adv, Ani. Doc, Fam, His)
Total
1000
447.80 86
116.44
7.83
21
Table 5: Statistics of shot and scene counts for different video types in MovieChat-SSeg.
C GENRE EMBEDDING
Genre Embedding Validation To verify the relevance of genre information to scene-level visual representations, we conduct a zero-shot classification experiment on the MovieNet-SSeg dataset. First, we obtain shot-level visual features by passing keyframes through the OpenCLIP Cherti et al. (023) image encoder and averaging the resulting embeddings. Scene-level representations are then computed by averaging shot embeddings within each scene. Genre vectors are obtained by encod- ing -MDb-based textual genre prompts using the OpenCLIP text encoder. Using these scene-level visual features and genre embeddings, we perform zero-shot classification via cosine similarity. For evaluation, each movie has n genre tags, we measure the top-n accuracy by checking whether the true genres among the top-n predictions. Our method achieves top-n accuracy of 60.52%. The zero-shot classification procedure is summarized in Algorithm [ Additionally, t-SNE visual- ization of the scene-level features reveals distinct clusters aligned with genre categories, supporting the hypothesis that genre metadata-althougg annotated at the movie level-provides meaningful semantic guidance at the scene level (see Fig. 5). Based on these insights, incorporate genre information into shot representation learning to enhance video scene segmentation performance.
Algorithm Zero-shot Scene-Level Genre Classification
Input: Keyframes of shots {k, ). genre prompts {gm}
2: Compute shot representations e; by averaging CL.IP image embeddings of kj
3: Compute scene representations s, by averaging e, within each scene
4: Compute genre embeddings 9m from textual prompts via CLIP text encoder
5: for each scene 8, do
6: Compute cosine similarity between s, and each Im
7: Select top-n genres by similarity
8: Evaluate if true genre tags are in top-n
9: end for
Output: Top-n accuracy
Genre Prompt Design We categorize genre conventions into three types: name, style, and defi- nition. Inspired by CLIP Radford et al. (2021)ss prompts for zero-shot classification, we construct prompts according to types for each genre. First, we simply formulate the name type '[genre name], a film genre' (e.g.. 'Drama, a film genre'). In the style category, prompts describe com- visual characteristies of each genre by extending the name format with additional descriptions shown in Tab. [7 For the definition category, we incorporate IMDB's 's genre tagging guidelines IMDb.com (202)). extending the name format with additional prompts shown in Tab. T8
16

---

Published as a conference paper at ICLR 2026
1000626222 Sei-Fi
80087182 SeiFFi
Different movies, Same gente
u2488496
Same movic, Different genre
m2488496
Adventure
Comedd
Farrily
Scsi
82488496 Action
Homor
Mucicll
Drama
Thrller
Adventure
Muck
Crime
Figure 5: Each point represents scene's visual representation obtained by averaging shot-level features extracted from CLIP image encoder. The emergence of genre-specific clusters suggests that scenes within the same genre share common visual characteristics, despite the video-level genre annotations.
Genre Embedding Analysis We use the embedding vectors obtained by inputting these prompts into OpenCLIP/Cerrt (2023'ss 's text encoder representations that encapsulate each genre. We analyze relationships among film genres in Fig. G comparing embeddings from definition prompts (y-axis) against those from name, style, and definition prompts (X-axis). These maps show high similarities along their diagonals, indicating that each genre maintains its distinctive characteristics across name, style, and definition prompts. As shown in Fig. ((c), the figure demonstrates that definition embeddings effectively distinguish between genres, while capturing natural relationships between similar genre pairs.
Jorr
Drama
Dame
the kor
Theikr
Ramance
Somance
Samence
Cére
Orina
Crine
Adverture
Aherrre
Advnture
S0-11
Fatrasy
Netuay
fettaey
Blognshl
Bounghy
Haker
HOOY
Habyy
Har
Bor
Vuc
Maica
Pezcal
(a) Name
(b) Style
(c) Definition
Figure 6: Similarity analysis of genre embeddings across different prompt types (name, style, and definition), where brighter colors indicate higher similarities. With definition-based prompts fixed on the y-axis, the consistent patterns of genre embedding similarities across different prompt types suggest that genre-specific characteristics are effectively captured regardless of prompt variations.
Different Vision-Languare Models Our approach offers flexibility through component-wise sub- stitution, allowing different vision-language (V-L) text encoders to be integrated for genre embed- ding. As shown in Tab. [6, consistent performance improvements are observed across various V-L. text encoders, including BL.IP- al. (202)) and Language Zhu al. (2023). This demon- strates that the text encoders robustly capture genre characteristics expressed through our prompt- based framework. These findings validate the generality and adaptability of our method to different pretrained language models, highlighting its potential applicability new and more powerful V-L. models continue to emerge. Furthermore, consistent gains indicate that our genre embedding strat- egy is not dependent
specific encoder architecture, underscoring its robustness and flexibility.
Integration of Various Metadata
We further evaluate the effectiveness of incorporating different types of metadata into shot representation learning, presented in Tab.
17

---

Published as a conference paper at ICLR 2026
Method
AP
AUC
F1
mloU
Li et al. (2023)
63.44
92.73
59.41
59.85
LanguageBind Zhu et al. (203))
63.31
92.79
58.82
59.53
Ours
63.62
92.84
58.88
59.64
Table 6: Vision-Language Models for Genre Embedding
Specifically, we examine shot scale, shot angle, camera movement, and color grading metadata linked to visual storytelling techniques influenced by directorial intent during production. Similar to our approach for genre, we construct textual prompts based on category-specifie definitions within each metadata type and embed these prompts to generate semantic vectors. Our results confirm that integrating such metadata within our framework yields consistent performance improvements over approaches that exclude metadata information. These encouraging outcomes suggest that leveraging a wider range metadata types could further enhance the robustness and accuracy of video scene segmentation in future work.
Method
AP
AUC
mloU
Shot Scale 63.58
92.72
58.82
59.51
Shot Angle
63.40
92.75
58.72
59.57
Camera Movement
63.24
92.71
58.82
59.46
Color Grading
63.57
92.83
58.78
59.43
Ours
63.62
92.84
58.88
59.64
Table 7: Different types of Metadata
Clustering Analysis To further validate our shot representation learning, we conduct unsuper- vised clustering experiments on 64 movies from the MovieNet-SSeg test set. Using learned shot- level features, we perform hierarchical clustering with the number of clusters set equal the ground truth number of scenes per movie. The results show that genre integration improves clustering per- formance significantly (+0.088 ARI, +0.127 AMI), demonstrating that our genre-guidee features capture meaningful semantic relationships for scene organization, as shown in Tab. g These cluster- ing results provide complementary evidence to our boundary detection experiments, showing that the learned representations naturally group shots into semantically coherent clusters without explicit boundary supervision during clustering.
Method
ARI
AMI
BaSSL
0.209
0.486
TranS4mer
0.257
0.544
Ours (w/o genre)
0.146
0.390
Ours (w/ genre)
0.234
0.517
Table 8: Clustering performance using Adjusted Rand Index (ARI) and Adjusted Mutual Informa- tion (AMI). Higher values indicate better agreement with ground truth scene boundaries.
Genre Vocabulary Robustness We evaluate robustness to reduced genre coverage by progres- sively removing the least frequent genres from the embedding pool (21, 16, 11. genres). Tab. shows non-linear performance degradation, with AP highest at 21 genres, followed by 16, then 6, and lowest at genres. The 6-genre configuration unexpectedly outperforms 11 genres, suggesting that removing additional niche genres climinates conflicting priors, allowing the model to focus core genre patterns. Critically, even with only 6 genres (approximately 30% vocabulary), our method outperforms the metadata-free baseline by 2.14 AP, confirming that genre priors provide value regardless of coverage completeness. This demonstrates practical applicability to domains with limited genre taxonomies.
Impact of Genre Count on Performance explicit modifications, genre embeddings integrate through affinity mapping with visual to-
Our method naturally handles multi-genre videos with-
18

---

Published as a conference paper at ICLR 2026
#Genre
AP
AUC
F1
mloU
63.62
92.84
58,88
59.64
16
63.54
92.72
59.14
59.71
63.33
92.67
58.93
59.71
63.41
92.75
58.96
59.60
Table 9: Robustness reduced genre coverage. Least frequent genres are progressively removed from the embedding pool.
kens during shot representation learning rather than shot-level prediction. We analyze performance across videos with varying genre counts in MovieNet-SSeg test set as shown in Tab. TO Results show monotonic performance degradation as genre count increases, with AP declining from 67 .7.20 (1 genre) to 62.38 (4 genres). This trend reflects two factors. First, multi-genre videos exhibit complex narrative structures where multiple storytelling conventions interweave, creating ambiguous scene boundaries that do not align cleanly with single-genre patterns. Second, as genre count increases, individual genre priors provide less distinctive guidance ance competing conventions. Despite this challenge, our soft integration mechanism accommodates multi-genre complexity by dynamically weighting relevant priors.
affinity- integration must bal-
#Genre
#Title
AP
AUC
F1
mloU
1
3
67.20
93.29
59.11
58.13
2
18
65.69
93.76
60.50
59.33
3
23
63.42
92.23
58.20
59.72
4
62.38
93.30
59.13
60.10
5
58.66
87.48
51.95
55.33
Table 10: Impact of genre count performance. #Title indicates the number of movies in each category.
D PSEUDO-BOUNDAAS SEARCH
Scene Boundaries in Shot Sequence In video scene segmentation, sliding window approach
Rao (2020a): Chen al. (202)) Mun et (2022) Islam al., (2023); Yang al. (2033) Chen
et al. (2024) is commonly used to capture contextual information from surrounding shots. Existing
approaches configure their sequence length between 17 Mun et al. 2022) to 25 Islam et al. (2023)
shots as hyper-parameter to achieve optimal benchmark perlormance. Following BaSSL's protocol Mun et al. 2022), ) we use a sequence of 17 shots with neighboring shots on each side. During pre- training. we detect pseudo boundaries within shot sequences learn shot representations and inter- shot relations without scene boundary annotations. These pseudo boundaries must effectively reflect the characteristics of actual scene boundaries. As shown in Fig. aa),, since many shot sequences contain multiple scene boundaries, allowing multiple pseudo boundaries within a shot sequence proves beneficial for learning shot representations.
Pseudo Boundaries in a Shot Sequence We search for pseudo-boundaries by sampling anchors based on shot duration and applying Dynamic Time Warping (DTW) Berndt & Clifford (1994) with representation similarity between anchors and the shot sequence. The sampling weights are inversely proportional shot duration, favoring shorter shots. As shown in Fig. 7T)), we demonstrate the number of unique anchor combinations obtained through sampling performed 20 times, matching the number of pretraining epochs. Although diverse anchor combinations are formed, we observe that the number of pscudo-boundaries converges to meaningful subset of potential boundaries, demonstrating its effectiveness with respect to shot representation learning, as shown in Fig. c(.). Additionally, illustrated in the following Fig. g we show the examples how various combinations of anchors generate diverse pseudo boundaries.
Pseudo boundary validation against ground truth annotations We perform comparative exper- iments between fixed anchor and our proposed method to assess the similarity between generated pseudo boundaries and ground truth scene boundaries, obtaining the results presented in Tab. TI]
19

---

Published as a conference paper at ICLR 2026
Count
to
(a) Number of scene boundaries
(b) Number unique anchor combinations
(c) Number of pscudo boundaries
Figure/ 7: Distribution analysis of boundaries in shot sequences: (a) number of actual scene bound-
aries per 17-shot sequence MovieNet-SSeg dataset, showing significant cases multiple bound- aries, (b) diversity of anchor combinations obtained through shot duration-based sampling. and (c) resulting pseudo boundaries generated from these anchor combinations, demonstrating our model's capability to capture multiple scene transitions within a sequence.
While fixed anchors generate identical pseudo boundaries for each shot sequence, our probabilistic approach produces diverse results across iterations. To evaluate boundary discovery effectiveness, actual boundaries successfully identified. For sequences
we measured coverage- the proportion with multiple scene boundaries, fixed anchors can match most one boundary due determinis- tie anchor selection, whereas our probabilistic sampling generates varied pseudo boundaries across training iterations, increasing alignment probability with actual boundaries. This diversity enhances shot representation learning by exposing the model to multiple realistic transition patterns, enabling more robust and generalizable embeddings for scene segmentation.
Sequence Complexity
#Samples
#Boundaries
Fixed Anehor
Probabilistic Anchor
Improvement
0 boundary
45,488
0
boundary
25.240
25,240
0.285
0.463
+0.178
boundary
16,396
32,792
0.177
0.411
+0.234
boundary
8,560
25,680
0.137
0.391
+0.254
boundary
Table 11: Boundary coverage comparison between fixed anchor and probabilistic (duration-award) anchor sampling across different sequence complexities.
11 U68880
Scrne
Scene
Scene
Soene
Scene #10
Keylnames
66
67
68
70
71
72
73
74
75
76
77
78
79
80
81
82
67
78
89
75
76
72
71
76
GT Boundaries
Surpplid Roundary Figure 8: Visualization of pseudo boundary search process with diverse anchor combinations in shot sequence. Multiple ground truth scene boundaries (in green) exist within the sequence, and our approach generates pseudo boundaries (in blue) through various anchor combinations (in yellow). The example demonstrates how our anchor samplings lead to pseudo boundaries that effectively align with actual scene transitions, facilitating better shot representation learning during pretraining.
71
79
E TEST-TIME SHOT SPLITTING
Computational Cost of Test- Dest-Time Shot Splitting Test-time splitting introduces minimal computa- tional overhead, as additional processing is required only for shots exceeding the duration threshold. Tab. 12 shows the number of split shots and resulting total shot counts in MovieNet-SSeg test set
20

---

Published as a conference paper at ICLR 2026
across different thresholds. Since model processes shots independently, computational cost scales linearly with shot count. At our recommended 10-second threshold, 8,811 shots are split (8.3% of total), increasing the shot count from 105,976 123,598. This represents approximately 16.6% computational overhead-a a favorable trade-off given the 0.18 AP improvement over the no- split baseline. Higher thresholds (30s, 60s) further reduce overhead 2.0% and 0.4% respectively,, though with diminished performance gains.
Threshold
Split Shots
Total Shots
Overhead
None (baseline)
0
105,976
60s
210
106,396
+0.4%
30s
1,046
108,068
+2.0%
10s
8,811
123,598
+16.6%
Table 12:
Computational overhead test-time shot splitting at different thresholds in MovieNet-
SSeg test set.
F COMPARISON WITH RECENT METHODS
We compare our approach with recent supervised scene segmentation methods that focus contex- tual learning from shot-level features:
Method
Year
AP
AUC
mloU
LGSS
2020
47.10
48.80
VSMBD* Tan al. (2024b)
2024
49.17
87.16
35.87
43.47
NeighborNet* lan ct al. 2024a)
2024
50.07
87.54
40.89
47.76
MASRC* Tan
2025
52.62
89.39
45.84
49.63
Ours
63.62
92.84
58.88
59.64
Table 13: Comparison with recent supervised methods on MovieNet-SSeg. using available implementations with best-effort parameter tuning.
*Methods reproduced
Note that our reproduced results may not fully match the originally reported performance due to potential differences in implementation details, hyperparameters, evaluation protocols. These methods primarily focus on supervised contextual learning, while our approach emphasizes self- supervised representation learning with genre guidance.
G DIALOGUE AND GENRE-SPECIFIC ANALYSIS
Performance across dialogue levels We analyze our method's performance across different dia- logue densities to understand the impact of visual-only features in Tab. [4] We align dialogue to shots by matching timestamps in subtitles with shot boundaries in the MovieNet-SSeg test set. Di- alogue density is computed as total subtitle characters divided by average shot duration per scene, then categorized into tertiles (Low/Medium/High), Results show that AP decreases from 76.92 (Low) to 61 62 (High) as dialogue density increases, confirming that visual features alone struggle with dialogue-driven scenes.
Dialogue Density
#Scene
AP
AUC
F1
mloU
Low Medium
2676
76.92
87.80
60.00
47.23
2758
70.73
89.69
60.20
50.72
High
2676
61.62
95.20
57.25
58.48
Table 14: Performance by dialogue density in MovieNet-SSeg.
Genre-specific test-time split analysis We further analyze test-time split effectiveness across gen- in Tab. 15 The MovieNet-SSeg test contains 64 movies spanning 17 genres (avg. 2.95 genres
21

---

Published as a conference paper at ICLR 2026
per movie). Test-time shot split shows genre-dependent effectiveness. Mystery/Crimee genres benefit substantially, as their longer shots can subdivided into finer boundaries, improving precision in detecting narrative transitions. Conversely, Adventure/Wan genres show limited or negative impact because their rapid action sequences already produce short shots that offer minimal room for mean- ingful subdivision. This demonstrates method's adaptive behavior, where splitring enhances segmentation for extended compositions while avoiding over-segmentation in fast-paced content.
Genre
#Title
w/o Split
w/ Split
Improvment
Drama
37
66.25
66.44
0.19
Thriller
59.91
60.13
0.22
Action
20
59.94
60.09
0.15
Crime
60.81
61.37
0.56
Romance
16
71.33
71.53
0.20
Adventure
61.76
61.59
-0.17
Sci-Fi
14
58.38
58.66
0.28
Mystery
57.64
58.17
0.53
Comedy
67.30
67.48
0.18
Fantasy
61.86
62.07
0.21
War
60.87
60.41
-0.46
Biography
74.52
74.58
0.06
History
71.04
70.55
-0.49
Music
2
66.48
67.26
0.78
Horror
69.78
71.46
1.68
Musical
58.66
58.32
-0.34
Family
68.52
67.79
-0.73
Table 15: Genre-Specific Impact of Test-Time Shot Split. Performance is measured by AP (Average Precision). Test-time split threshold = 10 seconds
H ADDITIONAL QUALITATIVE RESULTS
BBC dataset The results on BBC Baraldi et al. (2015) dataset demonstrate our model's general- ization capability to TV series, where we present examples of accurately detected scene transitions various video formats. As shown in Fig. existing approaches like BaSSI. Mun al. (2022) and TranS4mer[Islam et (2023) exhibit limitations in handling significant visual variations within scenes. contrast, our method validate the robust performance by maintaining the consistency in scene boundary detection across diverse environmental conditions. This performance shows method's ability to capture high-level semantic changes while being resilient to low-level visual variations, making it particularly suitable for professional broadcast contents.
Generalization to non-narrative video formats We explore our method's applicability to non- narrative formats using the LongerVideos Ren al. (025) dataset, including lectures and entertain- ment content (gaming, award ceremonies). Qualitative analysis reveals fundamental limitations as shown in Fig. 10 Lecture videos show poor shot detection, resulting overly sparse segmentation compared to video length. Entertainment content similarly fails to identify boundaries adequately. These failures highlight that "scene" definitions in non-narrative content fundamentally differ from narrative structure. Lectures are organized by topic transitions rather than while gaming videos and award shows follow event-based structures that diverge from cinematic conventions. Additionally, these formats rely heavily on dialogue and audio for context, which our visual-only approach cannot capture.
changes,
Application to BBDB dataset To demonstrate the applicability of our method for general video segmentation, we apply to the Baseball Database (BBDB) /Shim et al. (2018) dataset a collection of baseball game broadcasts that differs substantially from our training data. First, we apply shot boundary detection Castellano 2014) to segment videos into shots and extract keyframes. These keyframes then fed into method to detect scene boundaries. After detecting scene boundaries, we merge shots within the same scenes. Finally, we filter these merged scenes based on scene duration to obtain actual baseball game segments. By identifying and removing brief scenes,
22

---

Published as a conference paper at ICLR 2026
01 From Pole to Pole Pole-247. - 255
Scene Boundary
BaSsL
TranS4met
O Ous
Not Predictees
Jo
Boundayy
Protictes
Not Prodicied
Figure 9: Qualitative comparison of scene boundary detection methods on BBC Planet Earth quences. Top: "From Pole to Pole" featuring polar landscapes. Bottom: "Deserts" showing varied terrain. Orange/purple boxes indicate false positives/negativs from baseline methods, while our approach (blue) correctly identifies scene boundaries aligned with ground truth (green)
Lecture climate- www.wwwweee..ooooo...... a--columbii--cnnnnerriggg
Lecture trading-fort for-beginners
(d) Entertainment Black Myth Wukong All S
(c) Entertainment 76th Primetime Emmy Awards Figure 10: Qualitative Results Non-Narrative Content. Our method shows fundamental limita- tions on lecture and entertainment videos, where scene definitions differ from narrative structure. Shots in the same scene share the same border color.
(e) Entertainment Black Myth Wukong Part
23

---

Published as a conference paper at ICLR 2026
Wide View After Hit
Pitcher's Set Position
Game Scene
M
Shangg
KR
For inning
Top 2nd
Botton 2nd
Top F
Botto Jnd
Top 4th
Bottam 4il-1
Bottom 4th
Tap Sth
Bottom Sth
Top eith
Batton eth
fop 2th-2
Bottom 7th
Top 8th
Rattan Ah
Top 9th
Buttom oth
Jop Iehh
Rottom Ioth
Top 1
Botoen 11th
Pop 12th
Botrom 12
00:00:00
01:00:00
02:00:00
03:00:00
04:00:00
05:00:00
Non-Game Scene
TWICE
Commercial Breaks
Season Highlights
Figure 11: Timeline visualization of scene segments in a baseball game video from BBDB
focus on significant game segments. These longer segments typically represent key transitions in the game, such changes between inmings pitcher replacements, which are marked by distinct scene changes, as shown in Fig. T1 We visualize scene segmentation results on BBDB dataset demonstrate practical utility, showing how model effectively organizes video content into coherent scenes, As shown in Fig. [2, our method successfully distinguishes between game-related scenes containing actual gameplay and non-game content such advertisements between gameplay segments.
We present a simple application leveraging scene segmentation on the BBDB baseball broadcast dataset. By filtering scenes based on their duration, we obtain segments of meaningful lengths (over 180 seconds) that better correspond to actual game events. As shown in Table T6 the full dataset consists of 1,172 videos with a total runtime of approximately 4,236 hours and over 237,000 scenes. After filtering. this reduced 23,528 scenes spanning about 3,588 hours. This corresponds to an average of around 20 scenes per game, reflecting semantic segmentation aligned with the baseball game structure. Importantly, the filtered scenes cover 94.75% of the game event annotations, de- spite representing only 84.69% of the total video duration. This suggests that scene- based filtering effectively narrows down the temporal search space while retaining most of the key events. Such segmentation can thus facilitate downstream tasks like temporal localization and highlight genera- tion by reducing search complexity and enabling more focused analysis on semantically coherent video segments.
Type
#Videos
#Shots
#Scenes
Duration(H)
#Plays
Total
1172
2,917,186
237,913
4236.87
404,964
Filtered
1172
1,708,972
23,528
3588.34
383,711
Table 16: Statistics of the BBDB baseball broadcast dataset before and after scene -raaad filtering. Filtering reduces the total duration and number of scenes while retaining the majority of annotated game events, demonstrating efficient temporal segmentation.
ALGORITHM FLOW
We provide our ViT-based shot encoder incorporating our approach in pseudocode form to explain how genre embedding operates and integrates with visual features.
24

---

Published as a conference paper at ICLR 2026
(b) Not-game content betw een gameplay segmetts Figure 12: Visualization of segmented scenes from BBDB dataset. (a) Baseball game-related scenes
[a] Game-related scenes
showing actual gameplay and related activities. (b) Non miscellaneous content between gameplay segments.
Non-game game scenes including advertisements and
Algorithm Genre-Aware Visual Feature Integration
1: Input: Visual tokens Vú
R N
2: Model parameters: Genre embeddings Ge E RRNxDD, Projection matrix W E RDD
3:
4: // Compute genre-visual affinity matrix
5: for 1 to N, do
6: for j to Ng do
7: affinity(i. +- V.l.
VD
8: end for
9: end for
10: A < softmax(affinit)) // normalize affinity scores
12: // Project genre embeddings
Gprojected [
14: for = to N do
Oproj - -Gll
W // apply learnable projection
16: Gprojected <- G projected Spoj
17: end for
18:
19: // Integrate genre information into visual features
20: for i = 1 to N, do
21: genre_conter context +-
22: for j = N₂ do
23:
genre + genre_contex. + A[i,jj] x Groojectdd [j] 24: end for
25: V V.gere
[i]] + [i + genre.context // additive integration
end for
27: Output: Genre-enhanced visual features //aere
E NiXD
Additionally, we provide the pseudo boundary generation flow incorporating our probabilistic an- chor sampling approach that utilizes shot duration in pseudocode form.
25

---

Published as a conference paper at ICLR 2026
Algorithm 3 Probabilistic Anchor- Pseudo Boundary Generation
1: Input: Shot encoder fenc. shot sequence S {s1, , SN sequence length L 17
2: Initialize: anchor Left t 9, anchor rrghtt +-
3:
4: // Compute duration-based sampling weights
5: for = =1 N do
6: Wi; < 1/d, // inverse duration weighting
7: end for
8: W + [wi/ Σ Wi, W2/ Σ W/ WN/ Σ wi] normalize to probabilities
9:
10: // Probabilistic anchor sampling
11: anchor left +- sample from S|: N/ N//2 using weights W[: N//2]
12: anchor_right +- sample from S/N//2 - 1 using weights W /N//2 1
13:
14: // Extract shot representations for subsequence
15: Esubsey +-1
16: for anchor left to anchor right do
17: G; < MLP((enn (s:)) project encoded features
18: Esuhsoq BBsbbsaqj ei append
19: end for
21: // Generate pseudo boundary using DTW alignment
22: e Exahsey [0] // left anchor representation
23: 6 <- Esubneag -1] // right anchor representation
b" DDT -ll__aign(ee, Cr Fssabeq) // find optimal boundary
25:
26: Output: Pseudo boundary position within subsequence [anchor left, anchor anchor.right]
Prompt
Drama
Emotional expressions and intimate character interactions captured in natural lighting.
Comedy
Exaggerated character expressions and reactions highlighted with clear visibility.
Thriller
High contrast lighting and strategic framing creating tense atmospheric composition.
Action
Dynamic movement and high energy composition with spectacular visual impact.
Intimate relationships emphasized through lighting and warm tones.
Horor
Unsettling visual elements extreme shadows and distorted perspectives.
Crime
Uihan environment details highlighted with stark comtrast lighting.
Documentary
Autbentic visual capture of real people in natural, unposed compositions
Adventure
Expansive euvironmental views emphasizing scale and exploration.
Sci-Fi
Distinctive tochnological elements presented with artificial lighting and futuristic composition.
Family
Bright, warm
composition creating welcoming atmosphere.
Fantaxy
Ethereal lighting and visually enhunced reality elements creating magical simosphere.
Mystery
Selective focus and carefully hidden details in shadowed composition.
Biugraphy
Documentary style framing of actual person with period specific details.
Animation
Stylized color palette and intentionally crafted visual elements in artistic composition.
History
Period-accurate visual eloments presented formal historical composition.
Music
Stage lighting and dynamic presentation of musical performance elements.
War
Military visual elements presented in desaturated colors with conflict intensity.
Sport
Physical action and competitive movement captured with clear focus.
Musical
Theatrical staging and choreographed elements with dramatic lighting.
Westem
Frontier period elements portrayed earthy tones western composition.
Table 17: Style prompts for each genre.
26

---

Published as a conference paper at ICLR 2026
Genre
Prompe
Trama
Should contain namerous conecurive scases of chameters portmyed to effect serious nirrative throughout the rife, usuily invliiig conflicts and casotions. This can exaggerated upon penduce meledmams Subjecthve
Comedyy
Vimmally all scenes should costain chancters partciputing in humocous comedic experiences, The comedd can be exclusively for the viewer, exgense of characters describe humor (ie, spuol, paruly, frony slpstick, satire, dark comedy, comedic scene, euc.) Subjective.
litl. be shared with them. Please submit qualifying keywards better
Thriller
Should contain sumenuss sensational scenes a mirraivve is sensational auspenseful. Note not confused with Mystory or Hoeror, and should only sometimes he accompanied by one (or both). Subjective.
Action
Should comain numerous scenes wh action is spectacnlar usually destructivee Otlen inchudes non- stup motion, high enerpy physical stunts, chases, battles, and destructive enies (hoods, explosions, natural disasters, fires, etu.). Note: if a movie contains jusi one action scene (even if prolonged. Le. urplane accident) it does not qualify. Subjective.
Romsanc
Should costain mamerous inter-eelate secnes of a chareter and their personal life with engghtsii on essctional attachmest or imvuleemnt with other characters, especially those chanaterized by a high level of purity and devotion. Note: Reminder, with getres this does descrihe the movic whally, only instead. Subjective.
subplor, then should he submited keyword
Horoor
Should contain numerous consecutive scenes of characters effecting a teriffiigg andior repugnant narrative throughou the titl. Note: not to be confused with Thriller which is not usually bused in fear or abhorreace. Subjective.
Crime
Whether the protagonists or antagonists criminals this should contain numerom consecurive and inter-relited scenes chame-- fers purticiputing aiding. abetting, andlor planning criminal bahavice or experiences usally for an illicit goal. Not to be confised with Film- and only sometimes should supllied with Subjective.
Ducumentary
Should contain sumerous consecutive scenes of real pensonuges and not characters postrayed by actors, This does not include fake or spoof decumensarics, which should instead have the fake documentury keyword. A documestary that includes creating evens should include the ksyword "roenactmen"" that those actors are not treased as "Himself." This geare should also be applied to all instances of stand up cometly and concent performances. Objective.
Adventure
Should contain numerous consecutive and inter-related scenes of characters purticppuing hazandous or exciting experiences specifle gool. Otten include scanches or expedition for lost continents and exotic lacales, characters embarking in traasue hont or hemic joaneys, navels, and quests for the unknown. Not to be confased with Action, and should only sometimes he supglied with it. Subjective.
Sci-Fi
Numerous scenes, andlor the entire background for the setting of the narrative, should be based on speculative scientiic discoveries or developments, envirummnttal changes, spuce travel, life on other planets. Subjective.
Family
Shauld be univensally secepted viewing for younger andience, eg.. aimed specificallly for the education andlor entertainment of children or entire fumily Oftenn festares children relates to them the context home and family. Note: Uuully, but not always, complementary to Animationn Objective.
Fantasy
Should contain numerous cossecutive scenes of characters portrayed to effect a magical and/or mystical narrative throughout the tile. Usually has elements of magic, supematural events, mythology. felklere, or exotic fantasy workds.Note Note: not to be confissd with Sci-Fi which usually based in magic mysikism Subjective.
Mystery
Should ovetain mumerons inter-rolated scenes of one or more chumeters endeavoring to widen their knowledge of mnything per- tining to thenselvvs others. Note: Usnally, but not alway nesociated with Crime. Subyective.
Biography
Primary focus the depiction of aclivities and personality of real person persons, for some of their lifetime, Evens in their may be reenacsed, described documentary style. re re-enacted, they should generally follow reasonably close to the factual record, within the limitations of dramatic necessity. A real person in a fictional setiing would not qualify a production for this genne. Objective.
Animation
Over 75% of the title's is running time should have acones that are wholly, or purt-animated. Any form of animution is acceptable, e.g., hand-dmww, contuutes-ggeer
geeeer,, stedd stop-morion, etc. Pupperry does noe count is animation, unless a fans of animation such stop-motion also applied. Incideatal animated sequenees should indicated with the keywards part-animated unimated- sopeene instead, Objective.
Hlistory
Primary focus on life events hissorical significance featuring real characters (allowing some artistic liconsel; current lerms, the sort of thing that might be expected dominate the front puge of national newspaper for least a week: for older times, the soet of thing likely to be included in any major history book. Objective.
Music
Contains significast masie-related elements while not acrually being Musical, this may mann a concert, a story about a hand (either Nicional or documentary). Subjective.
War
War, a time genre, Should contain numerous seenes and/or a narrative that pertann to a real war lie. current). Note: for tills that purtaay fectioal plase submit keyward only. Objective.
Sport
Spart, a film genee, Foess is on sports or sporting event, cither ficiioal or octuel. This includes fictional stocies focused on punicular sport event, documentarics about sports, and ielevision hmadccsts actual sporting ovents. fictional film, speet itself can also fictional, it should the printary focus the film. Objective.
Musical
Should contain several scines characters hurstnn into song aimed the viewer tihii excludes songs performed for the enjooment of other characters that may be viewing) while the rest of the time, usually but not exclusively, portayynng narrative that alludes to another Genre. Note: not so be added for tiles that are simply music related or have music performaness in them: e.g. pup concerts do not apply. Also, chesical opera, since it is entinely musical, does not apply and should instead be treated Music. Objective.
Wastern
Should contain mumerous andior narrative where the portrayal similar that of frontirr life in American West daring 1600s costemporay times. Objective.
Table 18: Definition prompts for each genre.
27


Reviews:
[{"review_id": "f1F71m58js", "reviewer": "Reviewer_DYd7", "summary": "This paper introduces a novel method for video scene segmentation, which extends beyond the traditional reliance on visual similarity between consecutive shots by leveraging production-level metadata in the form of genre conventions and shot duration patterns. Scene segmentation is designed to identify semantically consistent boundaries within long-form videos, thus linking low-level visual cues with broader narrative comprehension. The principal contribution of this work is the inclusion of narrative-oriented signals into a self-supervised learning framework, which  improves shot-level representation learning.", "questions": "* What additional forms of production metadata—apart from genre, duration, shot scale, and shot angle—could be utilised as semantic priors, and how might their integration differ in terms of complexity and potential impact compared to primary signals such as genre and duration? Possible examples include camera movement classifications (e.g., panning vs. static), or color palettes associated with mood shifts or emotional cues associated with character.\n\n* Do shorter video segments (avg. 7.4 min in MovieChat-SSeg) limit the model's ability to capture long-term narrative patterns in full-length content?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*  Integration of contextual movie level metadata: Inclusion of textual genre level definitions as semantic priors for guiding shot level representations.\n* Duration based pseudo boundary generation: Inverse duration-based anchor sampling strategy for providing more weights to shorter shots as compared to fixed anchor approaches like BaSSL\n* Introduction of a benchmark for visual scene segmentation called MovieChat-SSeg associated with domains like movies, TV series and documentaries.\n* State of the art performance on MovieNet-SSeg dataset when compared to other state-of-the-art self-supervised methods like TranS4mer and BaSSL.", "weaknesses": "* Non-visual cues like background music and audio events help identify scene divisions but are not included.\n* Genre based contextual information might be insufficient in dialogue heavy scenes with minimal visual variations.\n* The final ablation studies do not indicate whether performance varies by genre class. For instance, including the definition of an action genre may affect results more than drama.\n* Although the proposed method has been evaluated on movies, TV series, and documentaries, it can also be applied to other formats such as news broadcasts, vlogs, and educational videos.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zdB8v84dGn", "reviewer": "Reviewer_KJQR", "summary": "The authors target the problem of scene segmentation for long form videos. They propose exploiting the genre information to enhance the semantic information to learn better shot representations during the pretraining stage. They do this by constructing genre-level textual prompts and encode them into embedding vectors and then use the relevance of each genre to the given visual token an affinity matrix and add it to the visual features. Furthermore the authors also propose anchor sampling scheme that gives more importance to shorter duration shots as against just using the sequence endpoints. At the test time, the authors further counter the high variability of the shot durations by just splitting the sho greater than certain threshold duration into three shots and use uniform sampling to extract the keyframes. Also, the authors propose a new manually annotated dataset which spans movies, TV series and documentaries and thus consists of sufficiently wide variety of inputs. For the pretraining stage the authors just use a standard contrastive loss annd also another cross entrop loss to distinguish a boundary shot from the non-boudary counterpart and use also the CE loss for finetuning. The authors finally reporrt the resuls on MovieNet SSeg for supervised training and evaluate the generalization on their datasets showing significant gains in Average precision for both and F1 metric for the first.", "questions": "When comparing with other baselines do the authors incorporate the textual information since they mentioned that they reimplent the BaSSL method. Or they just do it to avoid architectural differences? If they don't use the textual information while comparing then isn't it unfair to them?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The overall proposed scheme for training including Genre guided shot representation and the inverse duration shot sampling seems to be sound semantically since in long videos the genre based prompts can help in the boundary detection. The test-time shot splitting strategy is standard and a sound way to handle the long videos considering variability in the shot duration. The results also sucessfully second that. The proposed dataset can also be used to drive the research further in this direction.", "weaknesses": "The proposed scheme doesn't seem to be very novel and can be more interpreted as standard way-out for solving this long-form video issue in the considered setup. The shot sampling although is a good finding but I am unsure of any novelty including the test time splitting. So method wise not a major contribution although results seem interesting. \n\nInitially I have given a rating 6 due to the impact solving the problem can make, but I might change it based on other reviews and the responses authors provide.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "inKknrFHBW", "reviewer": "Reviewer_NJjj", "summary": "This paper proposes a metadata-guided framework for video scene segmentation. Instead of relying only on visual similarity between adjacent shots, the authors incorporate genre conventions derived from IMDb textual definitions as semantic priors into a ViT-based shot encoder through affinity-based residual fusion, which functions similarly to cross-attention. They further introduce inverse-duration-weighted anchor sampling to improve pseudo-boundary generation during self-supervised pretraining and a test-time shot splitting strategy to handle long shots without retraining. In addition, they present a new benchmark, MovieChat-SSeg, with 1,000 manually annotated clips covering movies, TV series, and documentaries. Experiments on MovieNet-SSeg, BBC, and MovieChat-SSeg datasets show consistent improvements over previous methods such as BaSSL and TranS4mer. The authors also provide comprehensive ablations covering integration strategies, prompt designs, anchor sampling, and split thresholds.", "questions": "1.The novelties and advantages of the proposed genre and duration signals could be highlighted.\n\n2.How does the model behave if the genre prior is noisy or incorrect? Could you provide degradation curves or sensitivity analysis?\n\n3.Is there statistical evidence that shorter shots are more likely to be boundaries? Could you provide quantitative analysis?\n\n4.How do you handle multi-genre videos during training and inference?\n\n5. What is the computational overhead of test-time shot splitting?\n\n6.Could you show more failure case analysis, especially on non-narrative content such as news or user-generated videos?\n\n7. Are there any related state of the arts that are published in recent two years for the comparison in Table 2 ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.The idea of incorporating genre metadata as soft priors is well motivated and practical.\n\n2.The method is lightweight and can be easily applied to existing models.\n\n3.Experimental results show consistent improvement across multiple benchmarks and good zero-shot generalization.\n\n4.Ablation studies are detailed and analyze four aspects: genre integration, prompt strategy, anchor sampling, and shot split thresholds.\n\n5.The proposed MovieChat-SSeg benchmark adds valuable evaluation data with diverse narrative structures.", "weaknesses": "1. The novelties of the proposed method could be highlighted by comparing with the most related methods, e.g. whether these genre and duration signals are newly proposed ? What are exactly advantages of these characteristics over the existing ones, e.g. video context.\n[1] Video Scenes Segmentation Based on Multimodal Genre Prediction, Procedia Computer Science, 2020.\n\n2. Robustness to noisy or conflicting genre priors is not explored. There is no experiment showing how the model behaves when the genre is wrong or uncertain.\n\n3.Multi-genre scenarios are not addressed in detail, which may lead to bias or performance drop in real applications.\n\n4.The efficiency impact of test-time shot splitting is not reported. This makes it difficult to assess its practicality for deployment.\n\n5.Failure case analysis is weak and the qualitative examples focus mostly on positive results.\n\n6. Although the state of the arts are compared in Table 9 of appendix, the comparison in Table 2 should include the methods published in recent two years to validate the superiority of the proposed algorithm.   \n\n7. Typo errors, e.g. `a a scene` in line 162.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)