ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding
Authors: David Ma
Huaqing Yuan
Xingjian Wang
Qianbo Zang
Tianci Liu
Xinyang He
Yanbin Wei
Jiawei Guo
nijiahui
Zhenzhu Yang
Meng Cao
Shanghaoran Quan
Yizhi LI
Wangchunshu Zhou
Jiaheng Liu
Wenhao Huang
Ge Zhang
Shiwen Ni
Xiaojie Jin
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Timescale Benchmark
Long Video Understanding
Primary area: datasets and benchmarks
TL;DR: ScaleLong embeds four timescale questions in the same videos to benchmark MLLMs’ understanding, revealing a U-shaped performance curve.

Abstract: Although long-video understanding demands that models capture hierarchical temporal information—from clip and shot to event and story—existing benchmarks either neglect this multi-scale design or scatter scale-specific questions across different videos, preventing direct comparison of model performance across timescales on the same content. To address this, we introduce ScaleLong, the first benchmark to disentangle these factors by embedding questions targeting four hierarchical timescales\textemdash clip, shot, event, and story\textemdash all within the same video content. This within-content multi-timescale questioning design enables direct comparison of model performance across timescales on identical videos. ScaleLong features 269 long videos (avg. 86 min) from 5 main categories and 36 sub-categories, with 4–8 carefully designed questions, with at least one question targeting each timescale. Evaluating 23 MLLMs reveals a distinct U-shaped performance trend: higher accuracy at the shortest (clip) and longest (story) timescales, with a dip at intermediate levels. Furthermore, ablation studies demonstrate that increased visual token capacity consistently enhances reasoning across all timescales. ScaleLong offers a crucial fine-grained, multi-timescale benchmark for advancing MLLM capabilities in long-video understanding. The code and dataset are available at \url{https://github.com/multimodal-art-projection/ScaleLong}

Published as a conference paper at ICLR 2026
SCALELONG: A MULTI- BENCHMARK FOR LONG VIDEO UNDERSTANDING
David Ma'; Huaqing Yuan's Xingjian Wang'; Qianbo Zang'; Tianci Liu', Xinyang He'
Yanbin Wei Jiawei Guo', Jiahui Ni', Zhenzhu Yang', Meng Cao², Shanghaoran Quan'
Yizhi LI', Wangchunshu Zhou 1.3 Jiaheng Liu Wenhao Huang', Ge Zhang', Shiwen Ni5.6t
Xiaojic Jin
MM-A-P MBBUAI OPPO
*NJU
SAIRISUAT
&SIAT,CAS
ABSTRACT
Although long-video understanding demands that models capture hierarchical temporal iifommation-from clip and shot to event and story-existing bench- marks either neglect this multi-scale design scatter scale-specific questions across different videos, preventing direct comparison of model performance across timescales on the same content. To address this, we introduce Scale- Long, the first benchmark to disentangle these factors by embedding questions targeting four hierarchical timescales-clip. shot, event, and story-all within the same video content. This within-content multi-timescale questioning design enables direct comparison of model performance across timescales on identical videos. ScaleLong features 269 long videos (avg. 86 min) from 5 main cat- egories and 36 sub-categories, with 4-8 carefully designed questions, with at least question targeting each timescale. Evaluating 23 MILMs reveals distinct U-shaped performance trend: higher accuracy at the shortest (clip) and longest (story) timescales, with dip at intermediate levels. Furthermore, ab- lation studies demonstrate that increased visual token capacity consistently en- hances reasoning across all timescales. ScaleLong offers a crucial fine mnn-graieed, multi-timescale benchmark for advancing MI.LM capabilities in long-video un- derstanding. The code and dataset are available at / ttt.ttttt.uug..o com/ mult. wal..mmddal-artttaom
/ Sca leoon
INTRODUCTION
Recent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced their ability to integrate and interpret complex inputs such text, images, and vidcos (Liu et al.
2023a Chen et al. 2024b Guo et al. 2024 Zhang et al. 2024 Zhu et al. 2025 Team et al. 2025).
Consequently, variety of benchmarks have been leveloped
gauge their video understanding
capabilities across different scopes and tasks (Wu et al. 2024 Wang et al. 2023 Fu et al. 2024a Li
al. 2024e] Ma al. (025). True comprehension of long video, much like human understanding.
requires seamlessly integrating observations across a hierarchy of temporal scales: from recognizing momentary gesture grasping the overarching plot.
However, current long-video benchmarks are ill-equipped to assess the multi-timescale capabilities of MLLLMs-specfficallyy their distinct abilities across varying temporal granularities. By typically using isolated short segments (Li et al. 2024d) or evaluating different temporal scales across entirely different videos (Zhou al., 2024), these benchmarks inherently conflate temporal granularity with content variability. This makes it exceedingly difficult to disentangle MLLM's true performance at each specific timescale from content-driven adaptations. Thus, rigorous, fine- fine-grained methodology is critically needed to evaluate how MLLMs apply these distinct temporal capabilities to understand the hicrarchical temporal structures within long videos.
Equaal contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2026
Clip
Shot
Which part painting the modify first?
How does artist effect grass painting?
using a palette spenad yellow
paint
B. The grass
B. By applying beoud strokas with large brush and light blue paint
C. The distani mountries
By lightly louching with small brush and dark green point
dabbing canvas sponge orange point
Multi Timescale
Event
Story
10mn
How does the artist bold patches the grass?
What did
during
Adkling point
gress
A. First distant mountains, then shrubs, fially pasches, C. Fins shrubs, than grass, finally bald pasches
D. Applynng a thick layer white point
C. Mixing the white paint with different color before applying
Using white paint directly the
D. First then shrabs, then distant mountains, finally bald patches
Figure Representative samples from ScaleLong. Each sample ScaleLong comprises video paired with carefully designed questions, structured across four hierarchical temporal scales. The correct answers are indicated in green.
To bridge this gap, we introduce ScaleLong, a benchmark tailored for the fine-grained evaluation of MLLMs' multi-timescale capabilities long videos. Its core feature is embedding questions target- ing four hierarchical temporal scales (Clip, Shot, Event, Story) all within the same video content. This "within-con continnconntnt' design enables direct comparison of an MLLM's performance across these distinct temporal granularities identical narratives, thereby isolating its abilities each specific scale. ScaleLong includes 269 videos (averaging 86 minutes), each annotated with 4-8 questions, ensuring least question per time scale. As illustrated in Fig. 2 the benchmark spans 5 major categories and 36 suhcategories, enabling a comprehensive evaluation of MILMs* understanding of long videos across diverse timescales.
Leveraging ScaleLong, extensive evaluations of 23 MLLMs encompsnnomasssnng 19 open-source and proprietary models- reveal U-shaped performance curve across the defined temporal scales. These models generally perform better on questions at the shortest (Clip) and longest (Story) temporal scales, while performance noticeably drops at intermediate levels (Shot and Event). This pattern suggests that current MILMs often excel at processing either highly localized visual de- tails or overarching narrative structures, yet face challenges with intermediate temporal contexts. Furthermore, targeted experiments conducted ScaleLong indicate that an increased allocation of visual tokens systematically enhances MLLMs' performance across all evaluated timescales, pro- viding valuable insights for future advancements in model development.
In summary, our work makes three primary contributions:
We introduce ScaleLong, specifically designed assess the multi-timescale capabili- ties of MLLMs in long videos. By embedding questions at four hierarchical temporal scales (Clip, Shot, Event, and Story) within the same video content, it enables rigorous evaluation of MLLMs at each distinct scale. ScaleLong includes 269 diverse long videos (averaging 86 minutes), with 4-8 questions per video (at last one per scale), across major categories and 36 subcategories.
ScaleLong:
Comprehensive MILM Evaluation and Insights: Our extensive evaluation of 23 MILMs on ScaleLong reveals a consistent U-shaped performance trend. MLLMs generally exhibit stronger comprehension at the shortest (Clip) and longest (Story) temporal scales, while their performance discernibly dips at intermediate scales (Shot and Event). This finding offers critical insights into how MLLMs process information at distinct temporal granularities in long videos.
Insights for MLLM Development: Evaluations on Scalel.ong provide key insights for MILMM enhancement. For instance, increasing visual token allocation consistently enhances performance across all evaluated timescales. Furthermore, analysis of model error patterns reveals persistent weaknesses, offering guidance for future model improvements long-video understanding.
2

---

Published as a conference paper at ICLR 2026
Causal Ressoning 249 (10.27%)
Rar/Weett Shos
Gure
Kaler Showe
Live Megii Show
Greming Stretnne
Ruality Comp
Music Cansrnt
Crme Dria
infommn 743 (30.65%)
Serom
Newn
Art
Action understanding
badrrinton
AucioonSSSkic
393 (76.21%)
2
Uhe
tennis
Sunviar
motorcycte
ASMR
Sport
volleyhal hall
Seifmadia
Renctions
basketball
Lactures/ Toils
Counting Problem 408 (16.83%)
Social Iose Dos
fottall
Raval Dos Benncaa)
Doc
Cooking Gemes
Traver Adventurs
631 (26.03%)
Recodn
Resonniti cednition
Object
Scrfech Dic:
Contral Doc
Birgraphi comseeiial sl bos
inve: jo. Dgc
Meriical Di
Day Projer:
Beauty/aalee Fathien
Dec wutrr
(a)Task Distribution
(b) categories
Figure 2: (a) Task distribution in ScaleLong. ScaleLong consists of total 5 tasks, ensuring comprehensive evaluation of the model's capabilities. (b) Video Categories. ScaleLong includes videos spanning 5 major categories and 36 subcategories, ensuring diverse topical coverage.
2 SCALELONG
2.1 OVERVIEW
ScaleLong is specifically engincered for the fine-grained assessment of the multi-timescale capa- bilities of MLLMs in long videos. The benchmark comprises 269 diverse videos, each averaging 86 minutes and annotated with 4-8 questions. As illustrated in Fig. these questions address four hierarchical temporal scales: Clip, Shot, Event, and Story. Key features of ScaleLong include:
Multi Timescale Queries: Unlike existing benchmarks, ScaleLong structures queries at four metic- ulously defined temporal scales-Clip. Clip, Shot, Event, and Story-all all within each individual video. This "within-conten" embedding of questions targeting multiple, distinct temporal scales is crucial: effectively decouples the assessment of temporal understanding of specific video content. Such a design enables precise evaluation of how MLLMs handle different temporal granularities while keeping the narrative context consistent.
Diverse Video Content and Task Design: For comprehensive MLLM evaluation, ScaleLong offers extensive content diversity, featuring 5 main video categories (e.g., Sports, Documentaries) spanning 36 subcategories. also incorporates 5 distinct task types (c.g., Causal Reasoning, Action Under- standing) designed to probe deeper comprehension. This structured variety representative assessment across diverse, real-world long-video scenarios.
2.2 -ULES-TIMESCALE HIERARCHIES
The
Hierarchies within ScaleLong are established by categorizing questions into four distinct temporal levels. This classification is based on the video duration essential for answer- ing each question and how relevant information is distributed across the frames. These levels are detailed as follows:
Clip:
Questions solvable by analyzing a few consecutive frames, spanning only a few seconds (c.g., up to 3seconds), typically involving recognizing instantaneous actions, immediate visual details, or straightforward objects.
Questions requiring the analysis of information from multiple frames within single contin- wous shot, typically ranging from to seconds. These questions require interpreting short-term dynamics, simple actions, character interactions, semantic coherence within this timeframe.
Shot:

---

Published as a conference paper at ICLR 2026
Event: Questions concerning significant events that span multiple consecutive shots, with dura- tions from 16 seconds up to 10 minutes. These questions require integrating information across scenes, interpreting event sequences, identifying contextually relevant frames, and understanding more complex narrative developments or causal links.
Story: Questions addressing content from the entire video substantial portions thereof, typically exceeding minutes. These require holistic comprehension of the overall narrative, including overall narrative logic, causal relationships, character development, thematic analysis, long-term dependencies across the video.
2.3 TASK TYPES
ScaleLong incorporates five distinct task types, each designed to rigorously evaluate different facets of MLLM's comprehension abilities:
Questions requiring inference about causal relationships within the video content. These tasks assess the model's ability to deduce cause-and -nffffect dynamics and logical connections.
Causal Reasoning (CR):
Object Recognition (OR): Questions involving the identification and distinction of specific objects, scenes, or their attributes within the video. These tasks evaluate visual perception and fine- grained recognition capabilities.
Action Understanding (AU): Questions focused on interpreting the actions, movements, or behav- iors of entities (e.g., characters, objects) in the video. These tasks assess the capacity comprehend dynamic interactions and temporal movements.
Information Summary (IS): Questions that require summarizing or generalizing main content, key points, or details from the video. These tasks evaluate the ability to synthesize information and extract essential concepts.
Counting Problems (CP): Questions that are designed to rigorously assess an MILL's capacity for precise numerical grounding- that is, its ability to accurately identify, track, and count instances based purely on video content.
2.4
ANNOTATION METHODOLOGY AND QUALITY CONTROL
2.4.1
ANNOTATION METHODOLOGY
ScaleLong ensures high-quality annotations through multi-phase process involving curated video selection, structured question design, and multi-round quality control. This process emphasizes content-based understanding, requiring questions target video-specific information, answers be thoroughly video-grounded, and dependencies absolute time cues or external knowledge to be eliminated.
Video Curation and Collection: Our video acquisition process begins by defining 5 principal categories (further detailed into 36 subcategories) to cover diverse real-world scenarios. YouTube videos, typically around one hour in length, manually sourced for these categories. Each selected video undergoes inspection for high visual clarity, substantial information density, and appropriate duration, resulting in a final corpus of 269 videos.
Question, Answer, and Distractor Generation: For each video. annotators first conduct a full viewing. They then design questions (two for each of the four defined temporal hierarchy levels), ensuring a balance in task types. Correct answers are derived from careful analysis of multimodal information within the video. Each question is accompanied by one correct answer and three plau- sible distractors, which are constructed based on ten predefined types to offer varied challenges and facilitate error analysis.
2.4.2 RIGOROUS QUALITY CONTROL
Our quality control protocol involves principal rounds with distinct objectives:
4

---

trustworthy and statistically sound foundation Published as a conference paper at ICLR 2026
Table Comparison with other benchmarks, where the abbreviations are defined follows: Anno. (Annotation Method), A (Automatic Annotation), M (Manual Annotation), #Genres (Number of Video Genres). MTS is the abbreviation for Multi-Timescale and IV-MTS the abbreviation for Intra-Video
Benchmark
|Vides
Duration. (s)
ITasks
#QA Pairs
Anno.
#Genres
MIS
IV-MTS
MSVD-QA
1,970
10
13,157
A
X
MSRVTT-QA
2,900
15
72,821
X
ActivityNet-QA
5,800
111
800
M
NEXTQA
1,000
44
8,564
M
MVBench
3,641
16
20
4,000
A
CinePile
9,396
160
303,828
M & A
EgoSchema
5,063
180
5,063
M & A
X
LVBench
4,101
1,549
M
X
LONGVIDEORENCH
3,763
473
6,678
M
10
HourVideo
500
2,742
12,976
M & A
ALLVB
1,376
7,200
252,000
A
16
Vidco-MME
900
1,024
12
2,700
M
30
MoVQA
100
992
6
21,953
M
X
MLVU
1,730
930
3,102
M
31
X
ScaleLong
269
5,160
5
1747
M
36
This round focuses on the foundational correctness, clarity, and con- sistency of all annotations. Question stems are verified for precision. Critically, absolute time local- izations are replaced with descriptive cues to compel content-basee reasoning rather than timestamp reliance. Answer options thoroughly checked: correct answers must be unambiguously video- grounded, and distractors plausible yet definitively incorrect. Annotations also undergo checks to prevent undue concentration of questions within limited segments of the video's timeline, and to validate all categorizations (e.g., temporal levels, task types, distractor types).
First-Round Quality Control:
Recond-Round Quality Control: This round focuses eliminating confounding factors and en- suring questions exclusively assess understanding derived from the video content. Questions solv- able through common world knowledge reliant external prior information, rather than video- specific details, are systematically revised or removed to nullify such external dependencies. Finally, uphold dataset integrity, any questions exhibiting persistent ambiguities (e.g.. unclear grounding, indistinct features, or problematic categorization) are rigorously discarded.
COMPARISON WITH OTHER VIDEO BENCHMARKS
Existing video understanding benchmarks, as detailed Table are broadly categorized into short- video and long-video formats. Short-video benchmarks like NExTQA (Xiao et al.] (021) and MVBench (Li et al.] 2024b) utilize sub-minute clips, which restrict their capacity for evaluating long-range temporal understanding. Recent long-video benchmarks-inchheee
CinePile (Rawal) EgoSchema (Mangalam et al. 2023). MoVQA (Zhang et al.) 2023b), MLVU (Zhou Video-MME (Fu al. (0224). Long VideoBench (Wu et al. 2024), and ALLVB (Tan
et al.
2024),
al.
2024).
et al.
2025 feature extended durations. However, they generally do not decouple the targeted
temporal scales of questions from specific video content. This inherent coupling hinders precise assessment of how MLLMs handle varying temporal granularities and, consequently, their distinct multi-timescale capabilities.
ScaleL.ong adopts 'quality over quantity' philosophy, designed a high-fidelity evaluation benchmark, not training corpus. While large-scale, monel--ynthesized benchmarks like ALLVB risk introducing biases and quality fluctuations, focus meticulous manual annotation ensures high data integrity. We believe this compact, high-quality approach, proven effective by bench- marks like GPQA (Rein et al. 2024) offers for MILM assessment.
Unlike most existing video benchmarks which, noted, typically conflate temporal scale sessment with disparate video content, ScaleLong is distinguished by its primary design attribute: the Intra- Video
nature. This principle dictates that within every long video, ques-

---

Published as conference paper at ICLR 2026
Table The performance proprictary and open-source MLLMs on ScaleLong across granularities and task types. For each timescale and task, the best performance is indicated in bold, and the second- best performance indicated with underlining.
Granularities
Task Types
Models
Date
Input
Overall
Clip
Shot
Event
Story
18
Human
N/A
Whole Video
92.8
88.9
92.7
89.8
91.0
Proprietary Models
Gemini 2.0 Flash
2025-02
256 frm
65.7
52.4
48.4
53.4
53.5
64.8
54.6
55.9
41.9
55.0
GPT-4o
2024-05
64 frm
61.8
50.7
51.0
58.0
58.3
62.6
57.4
60.1
36.0
35.4
Douban 1.5-VI. Pro
2025-01
55.2
64.5
58.7
Gemini 2.5 Pro
2025-03
256 frm
71.5
62.8
68.0
69.0
66.0
72.5
65.8
74.6
51.2
67.9
Open source Models
LLaVA-Mini
2025-01
256 frm
29.7
25.3
28.8
25.2
27.6
29.8
29.4
27.2
22.1
27.3
Long VILA
2024-08
frm
29.1
23.8
28.6
29.2
30.0
26.8
23.9
27.5
LU
2024-10
fmm
40.9
37.2
33.5
35.6
43.9
44.1
37.5
38.1
21.7
36.8
Phi-3.5
2024-04
64 frm
44.8
35.8
34.3
43.0
43.3
47.9
33,9
40.1
30.2
39.5
Long
2024-06
frm
50.3
38.8
43.8
49.3
53.1
38.2
45.4
28.8
43.3
Flash-VStreum
2024-06
256
46,9
39.8
47.0
39.0
34.1
48.6
44.1
Phi-4
2025-03
128 frm
50.0
42.9
42.1
45.5
50.3
53.9
44.0
46.2
30.8
45.2
LLaVA-OV-7B(SI)
2024-08
frm
50.8
39.5
44.4
47.3
41.4
44.1
49.3
34.3
45.5
MiniCPM-V
2024-08
frm
5100
43.1
47.8
55.2
42.9
48.5
32.5
46.2
Owen2-VL-71
2024-09
frm
51.6
45.9
46.5
48.3
51.6
$1.9
53.1
50.6
33.9
48.1
LLaVA Video-7B
2024-10
frm
57.9
46.8
50.2
48.0
47.1
55.4
52.3
54.4
39.9
50.8
2024-12
128 frm
60.2
42.5
48.5
52.3
52.0
61.5
47.4
50.8
39.3
50.9
Qwen2.5-VL-7B
2025-01
256 frm
52.8
50.0
49.5
$2.7
50.2
$4.5
53.3
55.3
39.9
51.2
Aria
2024-10
57.2
48.7
53.4
48.5
52.4
41.6
51.5
LLaVA-OV-72B
2024-08
frm
56.1
51.5
55.2
53.6
59.3
53.5
52.3
45.5
53.1
IntemV1.2-5-200
2024-12
128 frm
60.2
50.1
48.5
56.8
57.9
61.7
53.6
$2.9
43.6
53.9
LLaVA-Video-72B
2024-10
60.0
50.7
53.2
51.8
54.6
62.7
55.5
55.2
39.0
53.9
IntemVL2-5-388
2024-12
256 frm
61.8
53.5
54.1
55.5
65.4
58.8
58.1
40.7
56.3
IntemV1.2-5-7888
2014-12
128 frm
65.2
54.3
53.4
61.5
57.2
65.4
54.4
63.9
46.2
58.6
tions are specifically designed to target multiple, distinct temporal scales. This inherent character- istic- materializz through questions probing four hierarchical levels (Clip, Shor, Event, and Story) all within the same video narrative-is fundamental to the engineering of ScaleLong for the precise assessment of multi-timescale capabilities of MILMS in Long Videos. Such design feature di- rectly facilitates disentangled evaluation; the distinct capabilities of an MLLM at various temporal granularities are thereby measured against the same video content, allowing for clear separation of temporal understanding performance from content-specific reactions.
4 EXPERIMENTS
In this section, we evaluate representative MLLMs on ScaleLong first outlining the setting of experiments, then analyzing the results of the 23 MLLMs ScaleLong. We assess how the visual tokens shape long video understanding, and finally examine error rates by distractor type to identify key failure modes.
4.1 SETTINGS
We evaluate total of 23 MLLMs, comprising 4 leading commercial models-Geminni22 pro [DeepMind] 2025), 1.5 1.5-vision pro (Doubao Team 2025 and 19 open-source models spanning from 7 billion to 78
2.0-flash (Team et al. 2024),, GPT-4o (OpenAil (024) and Doubao-
billion parameters, including representative models such as Qwen2.5-VI. (Bai et al. 2025). In-
tern terv VL.2.5 (Chen al. 2024a and LLaVA-OneViison (Liu et al. 2023b).
4.2
MAIN RESULTS
Table 2 presennt the performance of all evaluated models across four timescale questions as well as five task types defined in ScaleLong. In this subsection, for all experiments, we fix the resolution at 240p and use the highest frame count have tested. We draw the following key observations:

---

Published as a conference paper at ICLR 2026
Model performance varies significantly across timescales: In long-video understanding, models exhibit a pronounced U-shaped trend across four timescales- Clip, Shot, Event, and Story. Accu- racy peaks at the extremes (Clip and Story) but dips markedly intermediate timescales (Shot and Event). This suggests MLLMS excel at capturing brief cues and overarching narratives but struggle with temporal coherence
moderate-lengle segments. For instance, Gemini 2.5 Pro scores 71.5% on Clip and 69.0% on Story, yet drops to 62.8% on Shot and 68.0% on Event. This U-shaped pat- tern is consistent across all models. Crucially, this MLLM performance fluctuation contrasts sharply with human consistency. Human evaluators demonstrate near-uniform accuracy across timescales (e.g., 92.8% on Clip, 91.3% on Shot, 88.9% on Event, 91.0% on Story), confirming consistent ques- tion difficulty. Thus, the significant model performance variations reflect their inherent limitations handling different temporal spans, not uneven question design.
Closed-source models consistently outper- form open-source ones across all timescales. For example, Gemini 2.5 Pro, the leading closed-source model, surpasses the best open-source counterpart (IntemVL2..-78BB) by at least 6.3 percentage points on Clip and 14.6 points on Event. Within the Vn2ernVL2.5 series, scaling from 8B to 78B yields steady accuracy gains across timescales. Despite these improvements, even the top models remain substantially below human performance. Gemini 2.5 Pro (67.9% overall) trails the human baseline (91.0% overall) by 23.1 percentage points, with the widest disparity (40.6 points) Shot-scale tasks (human 91.3% vs. GPT-4o 50.7%). This underscores the considerable room for improvement MLLMs.
Performance differences across models are notable:
MLLMs exhibit substantial performance disparities across task types: For most models, Ob- jective Recognition tasks achieve the highest accuracy, while Counting Problems tasks incur the lowest. For example, Doubao 1.5-VL Pro shows a 23.7 percentage-point gap between OR and CP, and GPT-4o exhibits 26.6-point difference. This consistent gap highlights that MILMs' ability to perform counting remains a critical area for improvement in long-vided video understanding. In stark contrast, human performance remains consistently high across all types, indicating that the observed MILM disparities reflect genuine weaknesses in specific reasoning skills.
In summary, our findings pinpoint two critical directions for future MILM development. The U-shaped performance across timescales reveals core deficit in modeling intermediate temporal granularities, demanding architectures that better handle mid-range coherence. Furthermore, the pronounced weakness in specific tasks like counting underscores the need for targeted training to improve these reasoning skills. Addressing these dual challenges is essential for closing the sub- stantial gap with human performance.
CCip
Shot
Dact
Sty
- Cb
-Bh Shst - Saort &e Story
54
CPp
Tuert
Shot
Stoty
52
52
50
Accuracy (%6
50
48
48
46
44
8
16
32
64
120
72p
144p
240p
360p
480p
4x/2tp 3x450s www.oossiddansssieeppee..
(a)
(b)
(c)
Figure 3: Comparison of model performance under: (a) varying frame counts, (b) varying video resolutions, and (c) different frametresolltion combinations.
4.3 ABLATION STUDY
To investigate how total visual-token count and its allocation between frame number and resolution affect MLLM performance tiltii-ttmesaale long-video understanding, we conduct two ablation studies:
7

---

Published as a conference paper at ICLR 2026
Scaling Effect. How does performance change as increase the total number of visual to- kens either by sampling more frames or by raising resolution?
Token Allocation. When the total visual-token budget is held constant, does distributing tokens across more frames or into higher resolution yield greater gains?
4.3.1 ISOLATED SCALING OF FRAME NUMBER AND RESOLUTION
In this subsection, we evaluate how allocating extra visual lisens-ttmporallyy by sampling more frames spatially by increasing inoluutionaafecle model performance, illustrated in Fig. 3
Under a fixed resolution, increasing input frames consistently improves performance, with the greatest gains Clip-level tasks. As shown in Fig. g a), all models benefit from more frames, but the impact varies by timescale. Clip-level tasks, which depend on fine-gre teee-gr temporal changes, see the most substantial improvement. In contrast, gains are more moderate for Shot and Event levels. Notably, Event-level accuracy peaks at 64 frames, suggesting that too many frames can introduce redundancy. For Story-level tasks, performance gains are limited, indicating that smaller number of frames is often sufficient for long-range reasoning.
Under a fixed frame count, raising resolution offers moderate gains but is less effective than adding frames and can have negative returns. As seen in Fig. 3b), higher resolution generally helps across all timescales, but accuracy uplift considerably smaller than that achieved by increasing the frame count. Furthermore, the trend is not strictly monotonic; performance on Clip- level tasks slightly dips at the highest resolution (480p), suggesting that excessive spatial detail can introduce noise that hinders short-span reasoning.
4.3.2 TOKEN ALLOCATION: FRAMES vs. RESOLUTION
To analyze the trade-off between temporal and spatial information under fixed visual-token budget, we evaluated several frame-resolution combinations, averaging results across three Qwen2.5-V2 models (e.g., Qwen2.5-VL-7B. Qwen2.5-VL-333 Qwen2.5-VI.72BB)
The optimal allocation of visual tokens is highly dependent on the target timescale. As shown in Fig. Bc), short-span Clip-level tasks benefit most from high temporal density (many low-res frames). Conversely, long-range Story-level understanding peaks with a more balanced setting, showing diminishing returns from excessive frame counts. Intermediate timescales like Shot and Event also thrive on a moderate balance, as performance degrades when either temporal detail or spatial clarity overly sacrificed. This highlights that no single configuration is universally optimal for long-video understanding.
missing inforatiion
visusl replacement detail replscement
choracter reglacement action replacement
temporal replocemant sequential reglacement
rrequency replacement quantitative replacement
spatial replacement
$1.5
22.55
14.5%
10,0%
TEAL
19.35
20.5
113%
11.5%
MaR
43.5%
$8.2%
Gam ini 2.5 Pro
Doubao 1.5-VL Pro
GPT-4o
InternVL2-788
LLaVB-Video-728
Figure 4: Distractor-specific error rate distribution across five MLLMss
4.4 ERROR RATES ACROSS DIFFERENT DISTRACTOR TYPES
To analyze error patterns in long-video understanding, we evaluate several MLLMs the ten dis- tractor types in ScaleLong, as shown in Fig. Although overall error rates are comparable across models, two categories-missee missing information and spatial replacement-staan stand out with the high- est failure rates. For example, Gemini 2.5 Pro, our best-rernonning model, erroncously accepts missing-information distractors 53% of the time and spatial-replacement distractors 46.6% of the
8

---

Published as a conference paper at ICLR 2026
time. These findings indicate pervasive insensitivity to the completeness of evidential support, as well as a notable deficiency in reasoning about spatial relationships within complex vidco sequences.
In contrast, models exhibit markedly stronger performance frequency misdirection and quantita- tive misdirection. GPT-4o misclassifies these distractors only 19.8% and 28.7% of the time, respec- tively, while Gemini 2.5 Pro's error rates despite their struggles with semantic completeness and spatial inference, current MLLMs are adept leveraging statistical and numerical cues. Together, these results highlight the need for future work to incorporate eechanisms-eithhr through architectural enhancements or targeted training curric- ula that explicitly verify evidential completeness and model multi-view spatial configurations in long-video contexts.
lower (13.3% and 16.7%). This suggests that,
5 RELATED WORK
Multimodal Large Language Models Multimodal LLMs (MLLMs), which pair visual encoders with large language models, excel at image and short-video tasks (c.g., LLa VLoVvvionv (Li et al.]
2024a), Otter (Li al., 2023), mPLUG-OWl (Ye et al. 2023 ) To address the challenges of
long-video understanding, recent MILMs introduce specialized designs-such as using a ViT
Q-Former in Video-LLaMA (Zhang et al. (023a), efficient visual compression in LLaMA-Vid (Li ct al. 2024c), scalable multi-event modeling in mPLUG-ow13 (Ye et al. 2024), and audio integration OL-oooot (Zhao et al. 2025). In parallel, other approaches leverage massively expanded multimodal pretraining to improve temporal and contextual reasoning, as seen in state-of-the-at models like VL2. (Chen et al. 2024a) and Qwen2.5-VL (Bai et 2025).
Video Understanding Benchmarks Video understanding benchmarks have evolved significantly, progressing from short-clip formats (e.g.. MVBench (Li al.) (024b). NEXT-QA (Xiao et al.)
2021), through mid-length tasks (e.g., CinePile (Rawal et al.) 2024). EgoSchema (Mangalam
2024). Video-MME (?)), to (Wu et al. 2024). HourVideo (Chandrasegaran et al. 2004)). ALLVB (Tan et al. 2025), HLV-IK (Zou et al, 2025 However, despite this progress video duration, current video benchmarks are ill- equipped to properly assess the miltiitimmesaeee understanding of Multimodal Large Language Mod- els (MLLMs). By probing different temporal scales using entirely different videos (Zhou et al.) 2024) or relying on isolated video segments (Li et al.) 2024d), these benchmarks inherently con- flate model's temporal reasoning with its response to varying content. This makes it exceedingly difficult to disentangle an MLLM's true performance at each specific timescale from content-driven artifacts, thereby obscuring clear view it temporal processing capabilities. ScaleLong addresses this critical limitation with its novel within-content, multi-scale design. By embedding questions at four hierarchical levels from clip to story- within each individual video, enables precise, dis- entangled evaluation of an MLLM's capabilities across the temporal hierarchy. This approach not only reveals performance trends with longer contexts but also pinpoints specific strengths and fail- ure modes at distinct granularities, offering a more rigorous methodology for assessing long-video understanding.
al. 2023). MoVQA (Zhang al. 2023b). MLVU (Zhou et al.
hour-scale evaluations (c.g.. LVBench (Zhang et al.) 2025). LONGVIDEOBENTE
CONCLUSION
We introduce ScaleLong, the first benchmark to enable a fine-grained, within-content evaluation of MLLMs across hierarchical temporal scales. This design disentangles temporal reasoning from content variability, offering a more accurate assessment of celti--tmmesale understanding. Our eval- uation of 23 MILMS reveals a striking and consistent U-shaped performance curve: models excel at the shortest (Clip) and longest (Story) scales but falter at intermediate (Shot, Event) levels. This trend points to systemic weakness in current modeling paradigms, which are adept at local feature extraction and global summarization but lack robust mechanisms for mid-range temporal coherence. Furthermore, our analysis indicates that strategically increasing visual token allocation offers a tan- gible path toward mitigating these deficits, though is not complete solution. ScaleLong's findings thus challenge the community to develop novel architectures and training objectives specifically de- signed to bridge this critical temporal gap. Ultimately, ScaleLong provides crucial diagnostic

---

Published as a conference paper at ICLR 2026
tool to benchmark progress and steer the development of MLLMs capable of genuine multi-scale long-video comprehension.
7 ETHICS STATEMENT
We have undertaken comprehensive review of the ethical dimensions of this research, with par- ticular focus on data sourcing, potential biases, and labor practices. The ScaleLong dataset is com- posed entirely of videos sourced from YouTube, for which we employed a rigorous filtering process to select only content marked with a Creative Commons license, followed by a manual verification of each video's license status to ensure a legitimate copyright basis for academic research. Coneur- rently, we acknowledge the inherent limitations and potential biases of this approach; as our data is primarily sourced from platform dominated by Western and English-speaking creators, Scalel.ong may carry cultural, linguistic, and demographic skews. We transparently disclose this limitation to alert downstream users to the associated fairness risks and the possibility that models trained on this dataset may generalize equitably across all populations. Furthermore, in all stages of this project involving human labor, including data annotation and evaluation, we adhered to fair labor practices. All contributors were compensated equitably in accordance with fair living wage standards for the region, to respectfully acknowledge their contributions.
REPRODUCIBILITY STATEMENT
We are fully committed to the reproducibility of our work. To facilitate verification of our findings, all code required to reproduce our experiments is public. Furthermore, the ScaleLong dataset publicly and readily accessible via the Hugging Face Hub at: https:/hugginngfacec http://uugginggaae..ooodaattssss SatpesseeeeSaaaeeeonnnscaall
In the Experiments section of our paper, provide a detailed description of all experimental settings to ensure faithful replication such as the number of frames sampled per video. imput resolution, and our evaluation protocols. We are fident that these comprehensive resources will allow for the straightforward reproduction our experimental results.
ACKNOWLEDGEMEENTS
Shiwen Ni was supported by GuangDong Basic and Applied Basic Research Foundation 1203111515110718 and 2024A1515012003), and Shenzhen Science and Technology Program (CCY120250004112211002333
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 2rXiv:2502.13923, 2025.
Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video- language understanding. Advances in Neural Information Processing Systems, 37:531 17-5317, 2024.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 14205.241..05271, 2024a.
Zhe Chen. Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic lisual-linguiste tasks. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24198-24198. 2024b.
10

---

Published as a conference paper at ICLR 2026
Google DeepMind. Gemini 2.5 pro. gemini 2025. Accessed May 2025.
httess//deepiii..ooooo//////heoo
Doubao Team. Doubao 1.5 pro, 2025.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis, 2024a.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075, 2024b.
Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.05237, 2024.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-moda model with in-context instruction tuning, 2023. URL pottps://arxiv..r abs/2303 03726
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195-22206, 2024b.
Yanwei Li, Chengyao Wang, and JiayaJaa Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340, Springer, 2024c.
Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile benchmark for video understanding and reasoning. arXiv preprint arXiv:2406. 1303, 2024d.
Yunxin Li, Xinyu Chen, Baotian Hu, Longyue Wang, Haoyuan Shi, and Min Zhang. Videovista: A versatile benchmark for video understanding and reasoning, 2024e.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning, Advances in neural information processing systems, 36:34892-34916 2023a.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b. URL https://arxit: wwtp://orggab/////oooooooo
08485
David Ma, Yuanxing Zhang, Jincheng Ren, Jarvis Guo, Yifan Yao, Zhenlin Wei, Zhenzhu Yang, Zhongyuan Peng. Boyu Feng, Jun Ma, et al. Iy-hench: A benchmark for amage-grounder video perception and reasoning in multimodal 1lms. arXiv preprint arXiv:2504. 2025.
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic bench- mark for very long-form video language understanding. Advances in Neural Information Process- ing Systems, 36:46212 02-46222-6624,, 2023.
OpenAi, Gpt-4o, 2024.
Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv preprint 4rXiv:2405.08873, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-prood q&a bench- mark. In Fürst Conference Language Modeling, 2024.

---

Published as a conference paper at ICLR 2026
Xichen Tan, Yuanjing Luo, Yunfan Ye, Fang Liu, and Zhiping Cai. Allvb: All-in-one long video understanding benchmark. In Proceedings of the AAAI Conference on Artificial Intelligncce, volume 39, pp. 7211-7219, 2025.
Gemini Team, Petko Georgiev, Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint 2xXx:2403.005582 2024.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025.
Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. Paxion: Patching Action Knowledge in Video-Language Foundation Models. arXiv e-prints, art, arXiv:2305. 10683, May 2023, doi: 10.48550arXxv.2355..666
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua Tenenbaum, and Chuang Gan. STAR: A Benchmark for Situated Reasoning in Keal-World Videos. arXiv e-prints, art. arXiv:2405.09711 May 2024. doi: 10.48550/arxi..2405.00 09.11111....
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark wtw..//r/abbb////00///
benchaak for long-context
interleaved Iadoo-lagguage understanding, 2024. URL 15754
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 9777-9786, 2021.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi- large language models. arXiv preprint arXiv:2408.00 048-00 2024.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306. 028508 2023a.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08088 2024.
Hongjie Zhang. Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile question-answering for long-form movie understand- 2023b.
ing. arXiv preprint arXiv:2312.0443
Hongjie Zhang. Lu Dong, Yi Liu, Yifei Huang. Yali Wang. Limin Wang, and Yu Qiao. Lvbench: A benchmark for long-form video understanding with versatile multi-modal question answering. International Journal of Computer Vision, pp. 1-22, 2025.
Jiaxing Zhao, Boyuan Sun, Xiang Chen, Xihan Wei, and Qibin Hou. Llava-octopus: Un- locking instrvction-driven adaptive projector fusion for video understanding. arXiv preprint arXiv:2501.05067, 2025.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang. Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint 24Xi::446.0426660 2024.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10000.00449 10479, 2025.
Heqing Zou, Tianze Luo, Guiyang Xie, Fengmao Lv, Guangcong Wang, Junyang Chen, Zhuochen Wang, Hansheng Zhang, Huaijian Zhang, et al. Hlv-lk: A large-scale hour-long video benchmark for time-specific long video understanding. arXiv preprint arXiv:2501.01645. 2025.
12

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1
ANNOTATION TUTORIAL.
A.1.1
QUESTION TYPE
Question Type Details
Representative examples of all 5 task categories in ScaleL.ngg are shown in Figure
!
Causal Reasoning
These questions aim test the model's ability to infer causal relationships between events, actions, or phenomena the vidco. The model needs to understand the content frames over certain time segment, and identify the internal logic of the "cause-effect" chain.
Object Recognition
These questions aim assess model's ability to identify specific ohjects, scenes, and their features (such as color, shape, and state) in the video. The model needs to locate them within the video scenes, achieve cross-frame tracking and consistent recognition.
Action Understanding
These questions aim to test the model's ability to identify character actions or object move- ments in the vidco. The model needs to understand the temporal combination of actions and their semantic goals.
Information Summary
These questions aim to test the model's ability to summarize or generalize the main content or details of the video. The model needs to use clues or context within the video to go beyond the understanding of individual frames segments, grasp the content of the video, and extract the plot summary.
Counting Problems
These questions aim to assess the model's ability conduct quantitative analysis of the num- her of objects, frequeney of events, and temporal relationships in the video. The model needs to accurately identify and distinguish various elements, involving counting across multiple dimensions such as objects, plot elements, or actions.
A.1.2
DATA ANNOTATION STEPS
Operations:
Attributes that need to annotated:
Video Key: The vidco id in Youtube.
Video Type: Based on content of the video, select one category from TV, sport, live, self-media, and documentary.
Question Stems and Options: Use clear and concise language deserihe the ques- tion, ensuring that question stem explicit and specific. Each QA should include 4 options that are logically coherent and relevant to the question.
Answer: Provide unique and correct
Question Type: Select one of the 5 question types that align with the question stem, each question can have only question type.
Time Reference: Label the time segment in the video corresponding to the answer (the time segment format should be in string format "XX:XX-XO" XX:XXXXXXXX")
Hierarchy: Label the level to which question belongs (clip, shot, event, or story).
2. Watch Video and Determine Hierarchy Type:
After fully viewing the video content to be annotated, select the two most appropriate and valuable question types for each hierarchy. Then, pre-conceive the corresponiing question content in preparation for designing the question stems and distractors.
13

---

Published as a conference paper at ICLR 2026
3. Design Question Stem and Answer:
Question Design Requirements:
questions are concise straightforward, avoiding complex or lengthy expression.
Clear Expression:
Ensure
Explicit Description: Descrihe the elements in the video clearly and specifi- cally (such as scenes, characters, objects. actions, or weather), ensuring questions are unambiguous and refer to unique segment in the video.
Quantity Requirement: For each video, design and annotate 2 questions for each hierarchy (clip. shot, event, story), need total of questions per video.
Balanced Question Types: Maintain similar number of questions for each type (e.g.. Causal Reasoning, Object Recognition, etc.).
Targett Visual Information: Ensure that pure text-hased L.I.Ms cannot answer the questions correctly.
Answer Design Requirements:
-Unqquenss:: For each question, there must be a unique and clearly correct answer. Concise Language: The wording of answer should be concise and clear, avoiding complex sentence structures and uncommon vocabulary.
4. Design Distractor Options: Design three incorrect distractors based the question and correct answer. The distractors should also be described clearly, be of consistent length, and meaningful.
5. Option Format Requirements:
Consistent Length: Ensure that four options' length are similar, avoid making the correct option easily identifiable due to length differences.
Diverse Design: Design distractors a varied manner to avoid patterns, avoid consis- tently employing singular approach when design.
Concise Language: The words of distractor options should be concise and clear, avoiding complex sentence structures and uncommon vocabulary.
Option significance: The distractors should be of the same category as the correct answer and should be meaningful in relation to the question stem.
A.1.3 METHODS FOR DESIGNING INCORRECT ANSWER OPTIONS
Incorrect Option Design Methods
Visual Replacement: Replace a piece visual information in the video with information that is similar and incorrect. For example, altering the color or shape of an object.
Quantitative Replacement:
Change quantity detail in video.
Action Replacement: Describe action that is similar but different from the actually curred action.
Character Replacement: Associate the actual event with the wrong character.
Spatial Replacement: Incorrectly describe the location where event occurs.
Temporal Replacement: Change the time point of an event in the description.
Missing Information: Create
by omitting key details in the option. (e.g. leaving
out an important action or cause when describing an event)
Detail Replacement: Manufacture an error by exaggerating or minimizing a detail. (e.g. describing "rumning slowly" as "sprinting quickly")
Sequential Replacement: Arrange series of events that occurred in the video in the wrong order.
Frequency Replacement: Repeat frequency actions incorrectly.
A.2 MANUAL REVIEW PROCESS DETAILS
The quality inspection of ScaleL.og comprises rounds.
14

---

Published as a conference paper at ICLR 2026
Round 1: focuses standardizing problem structures and correcting elements that do not align with the question.
Round 2: addresses advanced quality requirements ensure task rigor.
A.2.1 ROUND QUALITY CONTROL
Purpose
Ensure comprehensiveness of basic structure the question the alignment among each elements, including a clear question stem, correct answer, and meaningful distractors.
Quality Assessment Dimensions
Question Stem:
Expression:Check whether the question stem is coherent and meaningful.
Duplicate question stem: Check whether the question content for the same video repetitive, if only change the words of question
consider the same question.
Question Type: Check whether the question type correct and whether the question stem corresponds the question type.
Temporal distribution: Check whether the question stem only focuses the specific time segments of the video.
2. Options:
Answer: Check the correctness the answer based on the original video.
Distractors Type: Check whether method of designing incorrect options same to the content of the incorrect options.
Distractors content: Check whether distractors are meaningful. Distractors should be as same category as the answer and should be meaningful in relation to the question stem.
Absolute Time: Replace question stems that use absolute references with vague time references.
4. Hierarchy: Check whether the hicrarchy is correctly labeled.
A.2.2 ROUND QUALITY CONTROL
Purpose
Enhance task validity by verifying multimodal necessity, question stem accurity, and distrac- tor plausibility, thereby preventing evaluation biases caused by design flaws.
Methods
Information Leakage Detection: Ensure that the question stem or options do not directly disclose the answer. (e.g. avoid using clothes color locate person when asking about the color of clothes)
2. Commonsense Dependency Screening: Check whether the content of the question is com- knowledge that be answer dircetly and whether the question requires prior knowl- edge answer. (c.g. "where does the sun rises", "What action did [Trumpp take?").
3. Duplicate question types: Check whether the question types in a video are overly concen- trated, and reannotate videos that are overly concentrated.
Distractor Optimization: Redesign meaningless distractors based on video content. Ide- ally, distractors should correspond to the question categories and create confusion.
5. Video Quality Filtering: Remove low-quality videos (difficult to describe with precise lan- guage) that cannot support effective question design.
15

---

Published as a conference paper at ICLR 2026
A.2.3 RETENTION STATISTICS AND INTER-ANNOTATOR AGREEMENT
To quantify the rigor of our manual review process, we track data retention rates and measure inter- annotator consistency throughout the quality control pipeline.
Verification Protocol: We implement strict "1-Generation, 2- 2-Verification" protocol. Each question is initially generated by one annotator and subsequently verified by two independent an- notators. We enforce unanimous consensus rule: sample retained only if both independent verifiers agree on its validity, including the accuracy of timescale categorization and answer correct- ness.
Inter-Annotator Agreement: We utilize Fleiss' Kappa (k) to evaluate the consistency between annotators throughout the quality control pipeline. The process demonstrates robust annotator consensus across both stages: Round 1 achieves temporal scale alignment, while Round 2 reaches an almost perfect 0.90 for advanced quality criteria such as information leakage detection and distractor plausibility. These consistently high metries confirm that our rigorous filtering criteria are applied uniformly, ensuring the reliability of the final benchmark.
= 0.85 for basic structural correctness and
Data Refention Analysis: The rigorous two-round filtering process results in highly curated dataset. As shown in Table 3 the overall question retention rate is 54.6%, reflecting our high stan- dard for quality assurance.
Table 3:
Statistics of data retention and inter-annotator agreement. Fleiss' Kappa is calculated independently for each verification round based on the agreement between the two verifiers.
Retained Volume
Step Retention
QC Stage
Focus
Kappa (s)
Videos
Questions
Videos
Questions
Round
Clarity & Timescale
331
2,430
82.8%
75.9%
0.85
Round 2
Leakage & Distractors
269
1,747
81.3%
71.9%
0.90
Final
Total Cumulative
269
1,747
67.3%
54.6 %
A.3 TIMESCALE ALIGNMENT VERIFICATION
To ensure the robust alignment of questions with their intended timescales, implement rig- orous annotation protocol, empirically validated through comprehensive experiments. These exper- iments unequivocally demonstrate both the sufliciency of our designated timescales- no additional performance benefit from expanded temporal contexts- and their necessity-vvidenee significant performance degradation when temporal contexts are restricted.
A.3.1 ANNOTATION PROTOCOL FOR TIMESCALE DEFINITION
Our annotation protocol is founded upon clearly defined timescales, meticulously applied by highly qualified annotators (Ph.D.s and Ph.D. candidates). To maintain the highest quality and consistency, each question undergoes thorough review process and is only accepted after successfully passing a stringent peer-review check, requiring independent approval from two additional annotators. This multi-stage validation ensures the precision and reliability of timescale annotations.
A.3.2 SUFFICIENCY OF TIMESCALE (UP-SCALING EXPERIMENT)
To empirically validate the sufficiency of our annotated timescales, conduct an up-scaling exper- iment using the Intern VL-2.5-8B model. This experiment involves evaluating questions by expand- ing their associated video segments to the next higher timescale (e.g., questions originally annotated for a 'Clip" scale are tested on 'Shot' segments). We quantify the change in performance using the accuracy difference metric, AcCupi
16

---

Published as a conference paper at ICLR 2026
AAccup ACCerpanded
ACtoiiginal
(I)
Where ACCoriginal denotes the accuracy obtained when the model is tested using the video segment corresponding to the originally annotated time span, and Accezpanded represents the accuracy when the model is provided with the expanded temporal context.
The results, presented Table indicate that expanding the temporal context yields only marginal performance improvements. The small positive AAccup values confirm that our original timescale annotations already encapsulate the essential visual information required to answer the questions effectively, thereby establishing the sufficiency of our defined timescales.
Table 4: Up-scaling Experiment Results
Up-Scaling Accuup
Clip + Shot
+1.6%
Shot + Event
+1.2%
Event - Story
+2.0%
A.3.3 NECESSITY OF TIMESCALE (DOWN-SCALING EXPERIMENT)
To establish the necessity of our annotated preescales-demonssie cannot be adequately answered with only local information perform down-scaling experiment with the Intern VL-2.5-8B model. In this setup, questions are evaluated on significantly shorter video segments extracted using sliding window approach. To ensure fair comparison, experiments are conducted across multiple shorter segments, and the best achievable performance is selected.
that long-timescale questions
We employ two key metrics quantify the observed performance degradation:
Absolute Performance Difference (AAccdown): ): This metric quantifies the absolute per- formance loss when the original timescale video segment is downscaled.
Accdoum AcCoriginal AcChsest
Where Accoriginal is the model's accuracy the original annotated timescale segment, and Aodnestddownscaeed is the best accuracy achieved using the shorter, downscaled video segments.
2. Visual Gain Retention Ratio (VGRR): This metric measures the proportion of perfor- mance gain, initially derived from the original full-timescale video segment, that is retained after down-scaling.
VGRR OoChees.downscuul -ACrezz X 100%
(3)
AcCoriginal ACCteat_onty
Where Accieut.only is the baseline accuracy when the model answers with video input, relying solely on textual priors.
The results, summarized in Table clearly demonstrate substantial performance degradation. The large positive values for Accaown indicate a significant loss in absolute accuracy, while the low VGRR values confirm that the majority of critical visual information and context provided by the original, broader timescale lost during down-scaling. This provides strong evidence that a broader temporal context is indeed necessary for answering long-timmn timescale questions effecttivly.
Table 5: Down-scaling Experiment Results
Down-scaling
A ACcatown
VGRR
Shot + Clip
15.1%
31.1%
Event - Shot
15.0%
25.0%
Story - Event
11.0%
33.1%
17

---

Published as a conference paper at ICLR 2026
A.3.4 CONCLUSION ON TIMESCALE ALIGNMENT
In conclusion, our stringent annotation protocol, rigorously validated by quantitative up- up-scaling and down-scaling experiments, unequivocally confirms that our questions are correctly aligned with their intended timescales. The up-scaling experiments establish the sufficiency of our annotations (demonstrating no significant benefit from adding more context), while the down-scaling experi- ments confirm their necessity (highlighting that removing context significantly harms performance). This robust methodology guarantees that questions designed to necessitate a broad understanding of video content cannot be trivially solved with only local temporal information, thus validating the integrity of our dataset's inherent timescale hierarchy.
A.4 SENSITIVITY ANALYSIS OF DISTRACTOR LENGTH BIAS
We acknowledge the importance of mitigating annotation artifacts, specifically the potential for models to exploit answer length as a heuristic. To rigorously validate the integrity of our benchmark, we provide comprehensive statistical and experimental evidence demonstrating that our distractors are well-balanced and that model performance is not driven by length cues.
A.4.1 STATISTICAL ANALYSIS OF LENGTII DISTRIBUTION
We conducted systematic analysis of answer and distractor lengths across all 1,747 questions using character count as the metric. The results indicate a negligible length bias: the Answer-to- Distractor Length Ratio is 1.083. This implies that correct answers are, on average, only 8.3% longer than incorrect options, suggesting that length is not salient statistical signal for distinguish- ing the correct answer.
A.4.2 EXPERIMENTAL VALIDATION VIA LENGTH- CONTROLLED SUBSETS
To further examine whether models systematically exploit length cues, we constructed two diagnos- tic subsets with contrasting distributions:
Length- BalgneeB Subset (n 148): Constructed via stratified sampling ensure equal representation across all length ranks (25% per quartile), thereby eliminating systematic length bias.
Length-Imbblane Subset (n 599): Adversarially constructed such that 94.3% of the correct answers are the longest option, maximizing the exploitability of length-based heuristics.
We evaluated six state-of-the-art MILMs on both subsets. The comparative results are reported in Table
Table 6: Performance comparison on Length-Baie Balanced and Emgth--mmalanced subsets. The small performance gap (A) indicates minimal reliance length heuristics.
Model
Balanced Subset
Imbalanced Subset
A Performance
Gemini-2.5-Pro
68.1 %
67.6%
+0.5%
Doubao-1.5
56.0%
61.8%
-5.8%
Intern V1.2.5-78B
54.8%
61.1% %
-6.3%
GPT-4o
53.2%
58.0%
-4.8%
LLa VA- Vidco
49.6%
54.4%
-4.8%
Aria
47.8%
53.6%
-5.9%
Mean
54.9% %
59.4%
-4.5%
A.4.3 PRINCIPAL FINDINGS
Our analysis yields three key observations that confirm the validity of the benchmark:
18

---

Published as a conference paper at ICLR 2026
1. Negligible Performance Differential: The mean performance gap between the balanced and maximally exploitable (imbalanced) subsets is merely 4.5%. This indicates that length cues confer minimal advantage, and models primarily relying visual-semante un- derstanding.
2. Robustness of Leading Models: Notably, the top-performing model, Geminii-2.5-Pre Balanced subset (68.1% 1%) compared to the
achieves superior performance the Length-Balanced Imbalanced subset (67.6%). This inverse behavior definitively excludes systematic length- based exploitation primary strategy for advanced models.
In conclusion, both statistical metries and experimental stress-testing confirm that our benchmark design effectively suppresses distractor length bias, ensuring that the evaluation reflects genuine multimodal reasoning capabilities.
A.5
MORE EXAMPLES
To provide clearer understanding of the key timescales proposed in this paper, present detailed specification of our annotation standards. We categorize temporal granularity into four distinet lev- els Clip, Shot, Event, and Story-based on their duration range and the visual scope required for reasoning. Table I outlines these specific criteria and decision rules. Furthermore, we include two illustrative examples to further elucidate the distinctions between these four different timescales.
Table 7: ScaleLong Annotation Criteria & Decision Rules
Timescale Clip
Duration Range
Decision Rule / Visual Scope
< Seconds
Single Moment: Solvable by analyzing a few consecu- tive frames.
Shot
4 15 Seconds
Single Continuous Shot: Requires analyzing informa-
tion within one uninterrupted camera take.
Event
16 Sec 10 Min
Multi-Shot Sequence: Spans multiple consecutive shots but remains within a localized context.
Story
10 Minutes
Global Narrative: Covers the entire video or substantial portions; cannot be solved by a single sequence.
19

---

Published as a conference paper at ICLR 2026
Clip
Where rider number 93 starting line?
Why did rider number crash?
the finst now At the back the
A. Ilis fromt tire exploded during turn
B. He was foroed off track by smother rider
row, near D. On the lirihast side of the grid
bike
causing balance
D. He look a risky move but losi control in high-speed cormer
Event
Timescale
Story
Bmin
process
overtaking rider 997
Was race winner in first plave throughout the entire race?
A. Overtook during a pit stop
geadually moved
B. Waited for to make mistake and then took advantage of
B. Yes, be stayed first place from start finish
C. stmight, using slipetremm
No,
the
D. Oventook on the comer, accelerated on the stright build
D. No, he started lastest place and made comehack win
Clip
Shot
156
What color of kanife sharpened by the penson to steak package?
What is the strainer doing while the butter being melted?
Gently stirring
utensil
Red
B. Removing the foam from the surface
Greem
C. Gradually beating the butler loww heat.
Straiinng hutter
Multi
Event
Timescale
Story
Bmin
How the lobsier cooked aller being taken out the tank?
What the feature the finst three steaks?
water,
slowly
They used rosemary
R. First marinsted in smec, then grilled over low hear with spices
They were grilled
First steamed with than brushed lightly with melted butter
C. They all included garlic
D. First pan fried, then stin fried with other ingredients
They
Figure 5: More representative samples from ScaleLong. The top and bottom panels illustrate dif- ferent examples, where each sample comprises video paired with carefully designed questions, structured across four hierarchical temporal scales. The correct answers are indicated in green.
20


Reviews:
[{"review_id": "ZU4nHYaX59", "reviewer": "Reviewer_dL49", "summary": "Authors introduce ScaleLong, a long-video benchmark that embeds questions targeting four hierarchical timescales, allowing direct comparison of model ability across temporal granularities. It contains 269 ~86-minute videos spanning 5 categories/36 subcategories, each with 4–8 carefully curated questions, rigorous quality control, and tasks covering causal reasoning, object/action understanding, summarization, and counting. Evaluating 23 MLLMs reveals a consistent U-shaped trend - stronger on the shortest and longest scales, weaker at intermediate shot/event levels - with ablations showing that allocating more visual tokens improves performance across timescales.", "questions": "My questions are based mainly on weaknesses - more examples, more statistics.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Novel Temporal Framework\nThe paper’s hierarchical division of video understanding into clip-, shot-, event-, and story-level timescales is a novel and insightful framework that enables fine-grained analysis of temporal reasoning abilities within a single benchmark.\n\nInteresting ablation studies\nThe ablation experiments are thoughtfully designed to show interesting insights about the effects of visual token allocation and temporal coverage\n\nRigorous Dataset Construction\nThe dataset creation process is meticulous, featuring multi-stage quality control, diverse content selection, and balanced question design to ensure fairness and reliability across temporal levels.", "weaknesses": "Limited Illustrative Examples\nThe paper provides only a small number of examples from the benchmark, which makes it difficult to fully appreciate the nuances of question design across different time scales.\n\nInconsistent Story-Level Definition\nAlthough the paper claims that story-level questions require holistic narrative understanding, the provided example of a story level question and answers focuses on a sequential event listing rather than true integration of information from the entire video. \n\nInsufficient Distractor Analysis – The distractor creation process lacks detailed evaluation of more superficial distractor properties (e.g., length of the answers). Given that large language models can exploit superficial answer features - such as selecting the longest or most specific option - further investigation into distractor balance would strengthen the benchmark’s validity. The annotator instructions state  to design distractors with length in mind but there is no statistics on it to verify.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2nUr8tlgMR", "reviewer": "Reviewer_6e8r", "summary": "This paper presents ScaleLong, a benchmark for evaluating multimodal large language models on long-video understanding across multiple temporal scales. Each of its 269 Creative Commons YouTube videos (average length ≈ 86 minutes) includes questions at four hierarchically defined levels: Clip, Shot, Event, and Story, allowing direct within-video comparisons of reasoning ability over time.\n\nThrough evaluation of state-of-the-art MLLMs, the study finds a consistent U-shaped performance trend: models handle short (Clip) and long (Story) reasoning relatively well but degrade sharply at intermediate timescales (Shot, Event). This reveals a systematic weakness in sustaining medium-range temporal coherence.\n\nThe ablation experiments show that temporal coverage (more frames) contributes more to performance than higher resolution, emphasising the primacy of temporal context over visual detail. The analysis of distractor errors further exposes persistent difficulties with evidential completeness and spatial grounding, even in top models.\n\nCompared with existing benchmarks (e.g., MVBench, NExT-QA, LVBench, CinePile), ScaleLong is different,  in providing multi-timescale evaluation within the same video, enabling controlled analysis of temporal reasoning rather than content variation.", "questions": "1) Do the Appendix A.3 timescale validation results replicate across architecturally diverse models beyond InternVL-2.5-8B?\n\n2) Can you provide human performance when restricted to the same sparse frame samples models receive, to decompose the 23.1-point gap into access vs. capability components?\n\n3) What are the explicit duration ranges or decision rules you provided to annotators for each of the four levels (especially Event vs Story)? A short table with time bounds/examples would help others replicate the labeling policy.\n\n4) The paper ensures ≥1 question per scale per video and cover five task families. But, it is not clear how does the authors balance task types across scales (e.g., Causal vs. Counting at Event vs. Story) to avoid systematic pairing effects that could bias the U-shape? A per-scale task histogram would be valuable.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Comprehensive benchmark to embed questions at four hierarchical temporal scales within identical video content, enabling within-content comparison that isolates timescale effects from content variability\n2) Section 4.3.2 and Figure 3(c) analyze different frame-resolution combinations described as under \"fixed visual-token budget,\" revealing timescale-dependent optimal allocations (Clip benefits from many low-res frames; Story from balanced configurations)\n3) Ten predefined distractor types enable systematic failure mode identification, revealing specific weaknesses (missing-information 53% error, spatial-replacement 46.6% error vs. frequency-replacement 13.3% error)\n4) The paper presents empirical validation of timescale boundaries. Up-scaling experiments demonstrate sufficiency (minimal performance gain from expanded context); down-scaling experiments demonstrate necessity (VGRR 25-33% shows substantial information loss from restricted context)\n5) The benchmark consists of 269 videos averaging 86 minutes across 36 subcategories address genuine gap in existing benchmarks which mainly focus on short clips or lack multi-timescale evaluation.", "weaknesses": "1) Humans evaluated on \"Whole Video\" (continuous playback, ~150,000 frames) while models receive sparse samples (32-256 frames). The 23.1-point human-model gap may conflate access differences with capability differences , thus making it hard to interpret.\n\n2) Given 5 task families × 4 scales × many categories, some slices will be small. The authors make strong qualitative claims (e.g., U-shape) but the paper has no per-slice CIs/bootstraps and no inter-annotator agreement figures for annotation reliability. This practically limits how confidently we can generalize beyond aggregate means.\n\n3) Each video contains 4-8 questions spanning timescales, but the paper never states whether the questions are evaluated independently (fresh context) or sequentially (shared context).\n\n4) The paper reports no quantitative reliability metrics (percent agreement) despite requiring \"independent approval from two additional annotators.\" Timescale categorization requires subjective judgment about temporal boundaries, without reliability validation, cannot assess measurement quality of fundamental category labels.\n\n5) The paper does not clearly state how frames are selected from 86-minute videos. In my understanding, ifferent strategies systematically advantage different timescales: 32 frames from 5,160 seconds requires explicit sampling algorithm specification for validity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ryXaUykxHI", "reviewer": "Reviewer_41fk", "summary": "This paper proposes ScaleLong, a benchmark for measuring the model's ability to understand content at different temporal scales (Clip, Shot, Event, and Story) in long videos. It uses five real-world video categories collected from YouTube, with an average video length of over one hour. This paper comprehensively tests existing MLLMs (both open-source and closed-source) on ScaleLong, revealing some of their shortcomings.", "questions": "1. How is the error defined and measured in Section 4.4, such as missing information and spatial replacement in L425?\n2. New models, such as InternVL3 and InternVL3.5, could also be tested before the paper is submitted.\n\nIn summary, I believe the benchmark's design is sound, but some details concern me. I would be happy to discuss the points raised in the weaknesses and questions with the authors.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation for this study is sound, and the comprehensive measurement of the model's ability to perceive and understand temporal information at different scales in long videos is meaningful.\n2. The paper is clearly organized and easy to understand.", "weaknesses": "1. Long videos collected from YouTube, particularly TV and sports videos, often feature famous events. The training corpus for MLLMs may contain information about these events. How can the authors ensure that the questions they ask are not answered by the inherent knowledge contained in the model? \n2. Where do the four time scales of Clip (less than 3 seconds), Shot (4-15 seconds), Event (16-10 minutes), and Story (more than 10 minutes) and their respective time range division standards come from?\n3. (1) If the upper limit of the model input is 256 frames, for an 89-minute video, the average time range contained in each frame will reach 89*60 / 256 = 20.85 seconds, which means that many CLIPs and Shots will be ignored when sampling video frames, while MLLMs performs relatively well at the Clip-level time scale. Does this mean that many Clip-level questions are related to a large number of clips in the video? Therefore, can they be truly called Clip-level questions? (2) On the contrary, some Shot and Event-level questions may be related to dense spatio-temporal motions, so sparse frame sampling cannot cover the complete spatiotemporal clues, resulting in low model performance.\n4. In addition, can part of the reason why counting questions cannot be answered well be attributed to insufficient input frames? After all, sparse video content sampling may cause the time when objects appear to be ignored? The authors should further explore the impact of the number of input frames on different question types and timescales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6WVuKMmGgg", "reviewer": "Reviewer_dYA8", "summary": "This paper proposes ScaleLong, a new benchmark designed to evaluate multi-modal large language models on long video understanding across multiple temporal scales. Unlike prior benchmarks that either use short clips or do not explicitly disentangle temporal granularity, ScaleLong embeds four levels of questions within the same video, including clip, shot, event, and story. It enables direct comparison across timescales. The benchmark includes 269 long videos (avg. length 86 minutes) from 5 main- and 36 sub-categories, each with 4-8 carefully designed questions. Extensive experiments on 23 commercial and open-source MLLMs reveal a consistent U-shaped performance trend, where models perform well on the shortest and longest timescales but struggle at intermediate ones. The authors also provide ablation studies indicating that increasing visual token capacity improves performance.", "questions": "Please refer to the weakness section for my questions. Overall, I leaning to accepted this paper given the considerable quality.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the problem addressed is well-motivated: existing benchmarks cannot reliably disentangle model performance across different temporal scales in long videos, and the within-content question design is a meaningful contribution.\n2. The dataset construction methodology is described clearly, including multi-stage annotation and quality control to ensure question grounding and discourage guessable answers.\n3. The experimental study is thorough, covering a diverse set of 23 models spanning multiple sizes and both closed & open-source settings, which makes the analysis convincing.\n4. The discovery of a U-shaped performance curve across timescales is interesting and insightful, and is further supported by human baselines for difficulty validation.", "weaknesses": "1. My major concern is the size of the benchmark. It contains 269 videos, which is slightly smaller than other recent long-video benchmarks, and the paper mostly argues for \"quality over scale\". It might be useful to include more justifications, e.g., a statistical analysis of annotation diversity, to demonstrate the effectiveness and significance of such a small benchmark.\n2. The U-shaped trend is an important and interesting observation, but the explanation provided is somewhat speculative (e.g., \"models excel at short and global context but struggle mid-term\"). More controlled experiments to identify why mid-level reasoning fails would strengthen the conclusion.\n3. Given the observed U-shaped trend, I believe it might be also interesting to investigate more types of video understanding models, e.g., grounded video understanding methods [1-3] and agentic solutions [4-5], rather than running experiments solely with end2end methods. More discussions on the performance trend of different types of models shall provide more insights to the readers.\n\n[1] SeViLA: Self-Chained Image-Language Model for Video Localization and Question Answering\n[2] VideoChat-TPO: Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment\n[3] VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning\n[4] VideoAgent: Long-form Video Understanding with Large Language Model as Agent\n[5] VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)