Video-LevelGauge: Investigating Contextual Positional Bias in Video Language Models.
Authors: Hou Xia
Zheren Fu
Fangcan Ling
Jiajun Li
Yi Tu
Zhendong Mao
Yongdong Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Contextual Positional Bais
Video Benchmark
Large Video Language Model
Primary area: datasets and benchmarks

Abstract: Large video language models (LVLMs) have made notable progress in video understanding, spurring the development of corresponding evaluation benchmarks. However, existing benchmarks generally assess overall performance across entire video sequences, overlooking nuanced behaviors such as contextual positional bias, a critical yet under-explored aspect of LVLM performance. We present **Video-LevelGauge**, a dedicated benchmark designed to systematically assess positional bias in LVLMs. We employ standardized probes and customized contextual setups, allowing flexible control over context length, probe position, and contextual types to simulate diverse real-world scenarios. In addition, we introduce a comprehensive analysis method that combines statistical measures with bias pattern recognition to characterize bias. Our benchmark comprises 438 manually curated videos spanning multiple types, yielding 1,177 high-quality multiple-choice questions and 120 open-ended questions, validated for their effectiveness in exposing positional bias. Based on these, we evaluate 27 state-of-the-art LVLMs, including both commercial and open-source models. Our findings reveal significant positional biases in many leading open-source models, typically exhibiting head or neighbor-content preferences. In contrast, commercial models such as Gemini 2.5 Pro show impressive, consistent performance across entire video sequences. Further analyses on context variation, context length, model scale, and multi-modal reasoning provide insights for mitigating bias and guiding model enhancement.

Published as a conference paper at ICLR 2026
VIDEO-LEVELGAUGE: INVESTIGATING CONTEXTUAL POSITIONAL BIAS IN VIDEO LANGUAGE MODELS
Hou Xia', Zheren Fu', Fangean , Li², Yi Tu², Zhendong Mao'; Yongdong Zhang' University of Science and Technology of China, "HUAWEI Technologies Ltd foverwhe Imed, Lfc20025000maal uste. edu, en, {izr, zdmao, {jiajun wjrkk work, xssg. tuyi @huaweii com
ABSTRACT
Large video language models (LVLMs) have made notable progress in video derstanding, spurring the development of corresponding evaluation benchmarks. However, existing benchmarks generally assess overall performance across entire video sequences, overlooking nuanced behaviors such as contextual positional bias, a critical yet under-explored aspect LVLM performance. We present Video-LevelGauge, a dedicated benchmark designed to systematically assess po- sitional bias LVLMs. We employ standardized probes and customized contex- tual setups, allowing flexible control over context length, probe position, and con- textual types to simulate diverse real-world scenarios. addition, we introduce a comprehensive analysis method that combines statistical measures with bias pat- tern recognition to characterize bias. Our benchmark comprises 438 manually cu- rated videos spanning multiple types. yielding 1,177 high-quality multiple-choice questions and 120 open-ended questions, validated for their effectiveness in ex- posing positional bias. Based on these, we evaluate 27 state-of-the-art LVLMs, including both commercial and open-source models. Our findings reveal signif- icant positional biases in many leading open-source models, typically exhibiting head or neighbor-conter
preferences. In contrast, commercial models such as Gemini 2.5 Pro show impressive, consistent performance across entire video se- quences. Further analyses context variation, context length, model scale, and multimmdal reasoning provide insights for mitigating bias and guiding model en- hancement. Project page:
INTRODUCTION
Large Video Language Models (LVLMs) have advanced rapidly in recent years, revolutionizing video understanding by integrating large language models (LLMs) with visual perception capabil- ities. These models (Zhang et al.) 2024bb 2025a, OpenAl) show impressive performance across diverse video tasks, with strong generalization abilities. Alongside these advancements, continu- ous efforts have been devoted to developing video-centric benchmarks to assess their effectiveness, providing critical insights for future improvement.
Existing video benchmarks (Maaz et al. [2024] Liu et al., 2024d) primarily evaluate models based on overall performance across entire video sequences and summarization. Notable benchmarks include MVBench (Li et al.] 2024a). TempCompass (Liu let al., 2024d), and MMVU (Zhao et al. 2025 for short videos, well as MLVU (Zhou et al., 2025) and Long VideoBench (Wu et al.) 2024a) for long video understanding. While these bench- marks offer broad task assessments, they provide limited insight into how models interpret content at different contextual positions. As illustrated in Figure [ (a), LVLMs can suffer from positional bias, i.e., inconsistent comprehension of identical content presented varying contextual locations. However, existing benchmarks can offer minimal diagnostic value for mitigating this issue.
variety of tasks, such temporal reasoning
The serial position effect (Murdock Jr 1962) in psychology suggests that humans tend better recall content presented at the beginning and end of sequence. Similar position sensitivity have
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
Query Content the Start
Query Content the Midele
Content the
transstions
transtione
through tasl yelloui
through yellour-oo
Eore elsphtente there?
three?
Upwr
there?
Four elephants can be seev walling along the shalow waterway, suroundedd vost worlord grosllndss
Soory theree don't seam to be any elophants the video the number of dlegranns canect be dienniied.
A can be seer that there snnteriay. and about five elephancs are moverg along the road.
V 100
X 10
50
(a) Illustration Contextual Positional Bias of LVLMs in Video Understanding.
Standardized Probes:
Which country recended by the video? Thalland
As the comera tranutioos namow wateewary fost pelins-rren How many elephonts thoce?
Position Preference:
Evaliaation
Overall Performance:
Acxiproben
A- Four
Aco/OaA, GP2, GAS)
Insert
Acofprobes)
Pcctppatees)
(b) Comparison Between Existing Benchmark (left) and Ours (right).
Figure 1: (a) Large Video Language Models (LVLMs) suffer from positional bias, characterized by uneven comprehension (marked by tick/cross and the score) of identical content presented different contextual positions, (b) Existing benchmarks typically assess models based on overall performance across the entire video sequence, which is limited in revealing this nuanced behav- ior. We explicitly investigates this by inserting standardized probes (curated segments tagged with meticulous questions) into varying positions of the context.
91.4
94.6
More Pronounced Positional Blas
67.7
60.0
31.7
34.9
35.3
35.5
39.4
40.0
16.4
21.4
22.9
Gemini Pro
GLM-4.5V
lat-sst
DoubaooSeed-11
QvQ-Max
Claude Sonnet-4
InternVL3-788
ULa Video-728
MiMo-VLLRRLN
LLaVA-OV-728
Video-XL2-78
VideoLLaMA3-78
Qwen2 5 VLV--28
VideoRefer
Qwen-ViL VL-Max
Figure 2: Performance of state-of-the-ate LVLMs on "ideo-LevelGauge, evaluated by our com-
posite metric, where higher values indicate more pronounced positional bias. Gemini 2.5 Pro (Co-
manici et al. 2025) exhibits the least positional bias, followed by GLM-4.5V (Team et al. 2025b), GPT-4o-lates (OpenAI) Doubao-Seed- -eeda----ee-.666 (Bytedance). and others.
been observed in language models (Liu al. 2024b) and multi-modal models (Tian et al. 2025 Tan et al.] 2024). To date, in video understanding, how various types of LVLMs, such as those incorporating memory components (He al. (2044) or trained with long-context (Qin al. 2025), perform on positional biases remains under-explored. Moreover, how positional bias manifests in video-text interleaved contexts is also an open question. In particular, LVLMs claiming excel at video understanding should be validated for their ability to maintain consistent and effective perception across the entire video sequence, with minimal positional bias.
To this end, we introduce Video-LevelGauge, dedicated benchmark for evaluating contextual po- sitional bias video understanding. Inspired by needle-in-an -aalle--nn--haytackk approaches (Nelson et al,, 2024 Lu et al. 2025), we propose standardized probe and customized context strategy for con- structing evaluation data. As shown Figure (b), our benchmark inserts standardized probes at varying positions within the context assess models' bias contextual positions. It supports flexi- ble control over context length, probe position, and context composition to evaluate positional biases in various real-world scenarios, such as long video comprehension and multi multi-modal interleaved in- puts. In addition, we propose a comprehensive analysis method for positional bias that combines statistical metrics and bias pattern recognition. Video-LLeeelGauge includes 438 manually curated
2

---

Published as a conference paper at ICLR 2026
multi-type videos and 1,177 high-quality multiple-choice question answering items spanning six tasks, along with 120 open-ended descriptive questions, constructed through a labor-efficient work- empirically validated to be highly sensitive to visual perception, making
flow. These questions them well-suited for detecting positional bias.
We evaluate 27 of-teoo-tthe-ar LVLMs, including 6 commercial and 21 open-source models based on diverse techniques. We reveal that most leading open-source models suffer from positional bias. They manifest various bias patterns, such as neighbor preference, U-shaped curves, and head pref- erence. In comparison, commercial models, e.g., Gemini 2.5 Pro, achieve consistent and superior performance across the entire sequence, as presented in Figure 2 There are substantial room for im- provement in mitigating positional bias among diverse LVLMs, which could significantly enhance their understanding capacity. Our further analysis of context variation, context length, and model scale provides key findings for positional bias in LVLMss. Our contributions are summarized as:
We emphasize the necessity of positional bias evaluation as a complement to existing benchmarks. To the best of our knowledge, this paper is the first systematic study of con- textual positional bias in video understanding.
We propose Video-LevelGauge, tailored benchmark for evaluating positional bias in video understanding with a comprehensive analysis method, which supports flexible con- figurations over context lengths and probe positions.
We experimentally investigate positional bias in 27 offte-of-thhaaae LVLMs and conduct an in- in-eppth analysis of the effects of context variation, context length, model size and multi- modal reasoning, offering insights for future refinement.
2 RELATED WORK
2.1 LARGE VIDEO LANGUAGE MODEL (LVLM)
LVLMs have made significant strides with the integration of large language models and visual en- coders. Early works (Lin al. 2024 Maaz et al. (2024) focus on short video understanding through video post-training. Later efforts (Liu et al. [202a Li et al. (024b) extend to long video under-
standing. Two-stage methods (Zhang et al. |0.4||aaa||Pa Park et al. (025) perform video understanding by inputting frame descriptions into LLMs. Others (Song et al. 2024 He et al. 2024) incorporate ex-
plicit memory modules to capture long-term dependencies. Besides, models such as Long (Zhang et al. 2025b). Long VILA (Chen et al., 2025) and Video-XL22 (Qin et al. (025) extend contextual lengths through long video training. More recently, models like Qwen2.5- VL (Bai et al., 2025). In- VL3 (Zhu al. 2025). and LLaVA-OV (Li et al. (225) built upon long- longecontext LLMs, exhibit strong performance in long video tasks. MiMo- VL. (Team et al. 2025a) and GLM-4.5V (Team et al., 2025b) deliver impressive performance equipped with multi-modal reasoning. Task-specifie models have also been developed. VideoRefer (Yuan et al.) (2025) enhances fine-grained spatial-temporal understanding and T-Star (Ye et al., 2025) improves contextual search. In this paper, we investigate whether these models can effectively comprehend entire videos without contextual positional bias.
2.2 VIDEO BENCHIARK
Alongside the advancement of LVLMs, many efforts have been made to develop video-centric benchmarks. Some benchmarks (Li et al. understanding. To extend evaluation to ong-videos, VideoMME (Fu et al. (025). MLVU (Zhou 2025). Long Beng (Wu et al. 2024a), LVBench (Wang et al. 2024a), and HLV- (Zou et al. comprehension. Another line of work adopts synthetic construction strategies, named Needle-in-a- Haystack (NIAH). For instance, some works (Yuan et al. 2024 Nelson et al. (2024) target long-text comprehension for LLMs, while others (Wang et al. 2025 2024b Wu et al. (2024b) focus on multi- modal contexts. More recently, VNBench (Lu et al.) 2025). LV-Haystack (Ye et al., 2025). and V-NIAH (Zhang et al., 2025b) are proposed to assess temporal search capabilities in LVLMs. Un- like these benchmarks, which primarily assess models' overall performance across the entire video sequence, our benchmark is specifically designed to evaluate the contextual positional bias in video understanding, providing dedicated analysis method to reveal nuanced model behaviors.
2024a Liu et al. 2024d) focus on evaluating short-video
al.
2025) focus on constructing videos of long duration and designing tasks involving long-range
3

---

Published as a conference paper at ICLR 2026
(a)
Optical Character Recognition
Attribute Recognition
Object Counting
slong
right,
Interacting graee pence
by equppeeet
riht?
ttte--piar
many
ICCamena
[6] Surver
running
Oui Whitee
(b) Red. Green [4]
(a) Fourteent.
211 Fight [r] Fleven
(d) Tweeken
Object Recognition
Relationship Recognition shirt down??
Action Recognition
Instructed Description
fliss dlsee
Q: Please discriss deta ly the segment
and
shisen serial slear of urben
street with podlsstrla evcrpais.
plastic
surple
the walks quidly.
(a) Dishes and bowls.
[c] Beer mags and pup
The persen in green.
The peroo pedto.
(b) ite ralses camera to take picture.
points drane bal.
busy
pois
toa cuos.
rags.
He sams
glses
Video Time axts (nonmalizes)
Start
Insert,
End
G
022
04
0.
08
1.0
(b)
Multi-video:
Customized Context
Long Video:
Multi-modal
Interleaved Input:
Template Background:
Figure 3: Overview of Video-LevelGauge, benchmark for contextual positional bias in video understanding. It adopts standardized probe and customized context paradigm, where crafted probes are inserted at varying positions within context. (a) Examples of standardized probes on six multi-choice question answering (MCQA) formatted evaluation tasks and one open-ended instructed description task. (b) Four customized context types for investigating positional bias under various real-world scenarios. More examples of probes and customized contexts are presented in Sec. A.7
VIDEO-LEVELGAUGE
We begin by presenting our philosophy and statistics (Sec. 3.1) followed by descriptions of probe
QA construction (Sec. 3.2) context construction (Sec. 3.3 and our analysis method (Sec. 3.4)
3.1 OVERVIEW
As shown in Figure 3 Video-LevelGauge introduces a standardized probe and customized con- text design paradigm, where carefully designed probe segments are inserted at varying posi- tions within customized contextual contents. By comparing model responses to identical probes
at different insertion points, we assess positional bias in video comprehension. Compared to approaches (Zhou et al.]
Recisnntion on
Action
OCK 10C
Betigaatinn 215
Reaatonnio Feongeiioe
180
116
2025 Zou et al., (2055) that densely for- mulate QA pairs in natural videos, this offers three advantages. (1) Controlled variables, eliminating confounding ef- fects of varying QA difficulty at differ- ent positions and contextual information leakage. (2) Flexible control over both the context length and evaluated posi- tions. (3) Simulation of diverse real- world scenarios, such as multi-video un-
Attribute Recegeitiin
Ohject Courting
rannucted Deser piion
Made
MCQA Formatted
Open-ended Answering
Question Type Distribution
barka
Count:
150
In sapord
100
90
QAs per
Distribution of Video Catagory
Distribution of Video Duration and QAs
Figure 4: Benchmark statisties.
4

---

Published as a conference paper at ICLR 2026
Step QA Generation
Step 2: QA Refinement
Step 3: Distractor Construction
Protocal
Blind Filtering
Human Refinement
LLM
QA
Refinement
QA
LLM
Distractor
MCQA
Hallucination Filtering
Video
LVLM
Caption
Figure 5: Probe QA construction workflow. We propose a labor-efficient workflow that combines automated generation with human refinement. Candidate QAs are first generated using LLM guided by manually defined protocols. These QAs are then enhanced using both LLM and LVLM to elim- inate hallucinations and information leakage, followed by human refinement. Finally, for challenge MCQAs, distractor options are generated by LLM and double checked by annotators.
derstanding, long videos, and interleaved video-text inputs. Beyond synthetic settings, we further construct extended set by annotating probes natural long videos, detailed in Sec. A.3
Vide encompasses six categories of structured video understanding tasks (e.g.. action recognition), along with open-ended descriptive task. As shown in Figure it includes 438 man- ually collected multi-type videos, 1,177 multiple-choice question answering (MCQA) items, and 120 open- instructed descriptive problems paired with annotations. Each question is described with scene mention and task instruction to ensure clarity, requiring genuine visual comprehension. In this way, Video- LvelGGuge provides comprehensive evaluation of positional bias LVLMs. We clarify how firmly our compact benchmark supports the evaluation of positional bias in Sec. A.9
3.2
STANDARDIZED PROBE CONSTRUCTION
The construction of standardized probes involves three stages: video collection, construction of multi-task QA, and probe requirement validation, which are elaborated below.
3.2.1 VIDEO COLLECTION.
As shown in Figure 4 six types of videos are collected from public test sets to avoid data leak- age and ethical concerns. Specifically, it includes 42 aerial videos from VisDrone (Zhu et al.) 202I], 49 surveillance videos from UCF-Crime (Sultani et al., 2018). 50 egucentric videos from Ego-4D (Grauman et al. 2022 152 media vidcos, 58 life-record videos, and 87 synthetic videos from MLVU and VideOMME. PySceneDetect is used to segment the original videos, followed by manual selection. Blurry, static, and duplicate videos filtered out.
3.2.2 CONSTRUCTION OF MULTI-TASK QA.
To comprehensively evaluate positional bias across various video understanding tasks, we construct six types of structured MCQA tasks: Optical Character Recognition (OCR), Attribute Recognition (AR), Object Recognition (OR), Object Counting (OC), Relationship Recognition (RR), and Action Recognition (ActR), along with one open-ended descriptive task. As shown in Figure 5 we construct probe QAs via a three-step workflow, involving automated generation and human refnement.
(1) QA Generation. Collected videos are frame-wise captioned using GPT-4o (OpenAl), and human annotators annotate the task definition along with three positive and negative QA examples for each video task. These video captions, task definitions, and QA examples are then fed into LLMs to generate task-specific question-answer pairs via prompt engineering.
(2) QA Refinement. The generated QA pairs may be problematic. We perform blind filtering with LLMs to eliminate questions that leak answers or can be answered using commonsense, and use GPT-4o to filter out items with hallucinations incorrect answers based on the original videos. Finally, all QA pairs undergo manual refinement.
(3) Distractor Construction. For MCQAs, LLMs are prompted to generate multiple deceptive dis- tractor choices, which are then manually vetted and refined by annotators ensure clarity and no
5

---

Published as a conference paper at ICLR 2026
obvious difference among the options. The choices randomly shuffled, and the average length of questions 30.4 words. Details of human refinement process are provided A.8
3.2.3 PROBE REQUIREMENT VALIDATION.
Although blind filtering is applied during QA construction, there may exist multi-modal Icak- age (Chen al. 2024). Therefore, we validate the quality of our QAs using (wen2.5-VL-771 and InternVL3-8B As shown Figure 6 left, compared the needle samples from MLVU, models achieve near-random accuracy (25%) under text-only input and perform far from sat- uration with single-frame inputs our MC- QAs. This indicates that QAs highly sensitive to the degree of visual perception, rendering them well-suited for evaluating posi- tional bias. Furthermore, as shown in Figure 6 right, the performance of each task scales with both number of frames and model size, highlight- ing the challenging nature of our QAs to discriminate models of varying capacities. Validation of
Text-only
Frame
100
50
16 Frames
Avg
OR
50
70
50
AR
OCR
10
10
Owen InternyL (lwnn InterniL
Qwennmax
Owen-78 lest-rrl
Qwen=78 1 Trome
Qoen-7 16 Trame
MLVU
Figure 6: Probe requirement validation.
potential leakuge between the probe clip and buckground videos is presented Tab
3.3 CUSTOMIZED CONTEXT CONSTRUCTION
Our design enables customized contextual contents simulate positional bias under various real- world scenarios: (1) Multi-video understanding. The probe is inserted into a context with multiple videos. (2) Long video understanding. The probe is inserted specific temporal positions within natural long-form video. (3) Multi-modal interleaved input. The probe is inserted into an alternating sequence of text and video, a type of context commonly used in retrieval-augmented generation and multi-turn dialogues (Agrawal et al., (024). (4) Template video background. Inspired by (Xing al. 2024), a template video initialized with ImageNet mean pixel values is studied as the context.
3.4 POSITIONAL. BIAS METRIC
Considering the inherent capability variance in models, commonly used absolute accuracy is inad- equate for fair cross-model positional bias comparison. this end, we introduce the relative score (RS) metric by normalizing positional score using model competence, defined as:
S
RS;
Smeta
where S; is the model accuracy when the probe is inserted at the i-th position, and Smeta is the accuracy when the probe provided standalone without any context. Based on this, positional bias is measured by combining both statistical metrics and bias pattern recognition.
For statistical metrics, we propose: Position mean score, Pmeen = mean( [ where the number of evaluated positions, indicates the model's average performance across positions. Position range, Pran mar R.S, ) min( ( (RS,) (...) measures the extent of the worst-case variation. Position variance, Pear ((.. M 1). evaluates the positional stability of the model.
To intuitively delineate different bias patterns, we propose Bias Pattern Recognition (BPR) to clas- sify models into five types. Stable (-) indicates that model's accuracy maintains consistent across different positions with milder bias. Head preference ('2) indicates that the model's 's accuracy is higher when probes are inserted at the head of the video, reflecting stronger comprehension of the video's 's start segment. Neighbor preference (/7) reflects stronger comprehension of the video's 's tail region, which is closest to the question and model's response the context sequence. Lost in the middle (U), adopted from prior LLM research Liu et al. (2024b), denotes stronger comprehen- sion of the video's ' head and tail visual contents and weaker understanding of the middle section. Volatile score (W) represents that the model's performance fluctuates repeatedly with probe inser- tion positions, showing no regional preference in the video but high sensitivity to positional changes. Classification is based polynomial fitting, detailed in Sec. A.5

---

Published as a conference paper at ICLR 2026
Table Contextual positional bias analysis of various LVLMs on Video-LevelGauge OCR. AR. OR, OC, RR, and ActR represent six video understanding tasks respectively, as shown in Sec. 3.2 Average denotes the average performance all six tasks. mmeta the accuracy with only probe input. Pient: the text-only score normalized by Smota for convenient comparison with P Pmean' Peeani the mean score across all evaluated positions; Pran: range of score between positions: Peari position variance; BPR: bias pattern recognition. Metrics are detailed in Sec. 3.4 multi-modal reasoning. Best results are highlighted in color
represents
Models
Size
Pran
OCR
Prin
AR
Pros
OR
Pren
OC
Pran
RR
ActR
Prest
Pmnaa
Average
Poor
BPR
Smete
Two mro-et
Cuption Qwen!
11.7
12.9
28.3
16.9
11.3
46.5
93.1
11.2
18.1
U
65.7
Caption GPT-4
4.3
11.3
7.7
12.9
15.6
7.9
45.9
96.1
7.4
5.8
U
67.7
Open-source Models
16.7
LongVA
7H
9.6
21.6
15.8
7.7
17.1
9.1
59.1
82.4
9.2
7.8
1
48.1
MiniGPT4-Videss
70
26.9
21.9
14.9
10.4
5.7
3.4
52.5
84.9
9.6
15,4
49.3
LLaMA-VII
22.0
24.2
25.4
12.3
31.2
Kangaroo
8B
14.3
13.2
10.4
24.6
38.9
15.6
$6.0
89.2
8.2
8.4
W
53.0
LLaVA-OV
7.0
4.8
15.5
6.7
43.7
72.0
LLaVA- Video
72B
6.9
7.2
10.7
4.6
47.5
93.4
3.7
1.2
74.6
Qwen2.5.VL
7B
8.1
27.4
9.8
16.9
3L.
15.0
41.9
89.6
12.4
11.7
U
68.2
Qwen2.5-VI.
20.2
22.2
17.7
92.2
73.7
InterVLL
8B
11.5
9.9
22.6
13.9
10.7
48.1
90.3
8.0
70.5
1ntemVL3
788
4.7
10.6
5.0
21.4
14.7
4.5
48.5
97.1
3.9
2.8
74.2
VedeoLLaMA2
LaM33
7.0
9.9
13.4
9.9
48.6
89.3
5.9
U
76.2
NVILA
12.5
34.2
17.2
15.1
17.7
14.0
48.9
79.1
13.9
64.5
IM
U
Video-XL.
7H
13.6
8.0
10.5
17.0
11.4
8.6
55.6
83.4
9.0
7.9
47.7
Videa-XL.2
13.2
13.2
48.2
6.3
73.5
VideoRefer
9.6
8.4
11.4
7.6
40.2
80.2
5A4
77.4
T-Star
7B
10.9
7.1
12.6
11.9
20.5
10.6
41.2
72.6
6.1
5.0
W
69.4
MiMo-VI.R RL
10.3
17.6
12.5
7.6
43.1
93.0
8.0
68.9
VI-R-V
7H
64
21.5
6.9
9.0
36.9
96.8
5.8
1.8
69.6
GLM-1.5V
108B
3.2
8.0
8.0
33.7
97.2
3.4
79.5
GLM-4.5VI
108B
5.2
6.6
6.7
30.9
97.8
2.7
79.9
Commeeriial Modells
Qwen-V Qwen-VL-Man
2.9
8.0
15.8
5.6
15.0
10.3
36.6
89.8
8.0
8.0
70.3
QVQ-Max!
2.9
6.0
7.8
38.2
93.4
2.4
75.8
Doubao Seed-LA
4.9
7.0
3.5
8.9
6.5
6.2
33.2
96.9
3.2
2,4
U
80.3
GPT-4p-latest
5.1
7.7
34.6
98.1
2.9
79.9
Claude-Somma 4
9.5
5.3
7.8
37.0
96.1
3.3
78.3
Gemini 2.5 Pro!
0.0
8.1
4.9
35.1
98.4
2.0
81.7
EXPERIMENTS
We first detail the evaluation protocol, and then present evaluations of wide range of LVLMs. Next, we analyse the effects of context type, context length, and model size.
4.1 EVALUATION PROTOCOL
We conduct a comprehensive investigation of 27 LVLMS using Video-LevelGauge, including commercial models, i... Gemini 2.5 Pro (Comanici et al. open-source LVLMs covering unified models like Intern VL3 (Zhu et al. 2025) long video models like Video-XI.2 QQn et al. 2025), specific optimized models like Videokefer Yuuan et al. 2025). multi-modal reasoning models like GLM-4.5V (Team et al., 20255), and two-stage methods like LLoVi (Zhang et al. (2244).. Specifically, evaluate
2025) and QVQ-Max (Alibaba); 21
uniformly distributed positions across the video context. For each probe, the context is built using nine same type collected videos, resulting in synthetic contexts with average duration of 7.2 minutes, which sampled following official implementations of each model. Unless specified, we sample frames probe and test the model's response by inserting them into different positions, ensuring consistent input of probe frames the model to isolate sampling impact. More details of evaluation protocol are provided in Sec. A.6
4.2 EVALUATION OF SOTA LVLMS
As shown in Table commercial models generally exhibit milder positional bias compared to open- source models. In particular, Gemini 2.5 Pro (Comanici al. 2025) demonstrates minimal posi-

---

Published as a conference paper at ICLR 2026
1002
IntemVL3-88 88
In0amVL VL3-9
106.
LLaVi-OV-77
100.0
Owen2 V-VL-28
Sweer 5-M-78
Roative Score
Reative Score
Score
55.0
20.0
85.0
Pudi
15.0
15.0
10.0
70.0
10
10.0
Posttonal Incc Frobe
Pesttional
Index
Postioonl
ProDe
Figure 7: Effect of context type on positional bias. For each model, we visualize the variation in relative scores across probe positions under six types of context (left), and quantify the severity of positional bias using the composite value of three proposed metrics (right), where longer bars indicate more pronounced bias. LVLMs exhibit milder positional bias in simple contexts, such as template videos, and degrade significantly in complex contexts like interleaved video-text inputs.
tional bias (with a small Pran and Pear, and stable BPR), followed by GPT-4o (OpenAI), Doubao- Seed-1.6 (Bytedance). and QVQ-Max (Alibaba), commercial models known for their long-context capabilities. Smeta measures a model's s accuracy on the probes and reflects its conventional visual understanding ability, which traditional benchmarks focus on. Although strong model may be strong in various aspects, Smeta not correlated with positional bias. For example, Qwen-VI. Max (Bai et al.] 2023), as an image-oriented model, attains a high Smeta. yet still exhibits severe positional bias. We speculate that this difference may be attributed to the longer training contexts and larger parameters of commercial models, which can mitigate bias as analysed in Sec. 43
Among the open-source models, two-stage methods that utilize LLMs to interpret sequential frame descriptions exhibit a U-shaped performance, found by Liu et al. (2024b). The state-of-the- art reasoning models (Team et al. 2025b]a) exhibit minimal positional bias, particularly GLM- 4.5V (Team et al., 2025b) with parameter scale of 10.8 billion. We observe that reasoning patterns can alleviate the positional bias issue to a certain extent (visualized in Figure [4). Besides, Video- LLaMA3 (Zhang al. 2025a) and LLaVA- (Zhang al. 2024b) are also impressive in terms of reduced positional bias. Video-XL2 (Qin et al. 2025). a model optimized for long videos, exhibits relatively small positional bias. However, Long VA (Zhang et al.) 2025b) and LongVILA (Chen et al., 2025) fail to achieve consistently robust performance across the entire sequence. This suggests that the effectiveness of long context training needs to be verified, as analysed in Sec.
VideoRefer (Yuan et al. 2025 designed to enhance fine-grained spatial-temporal understanding,
shows substantially lower positional bias than models of similar size. Training data with annota- tion bias is an undeniable contributor. For instance, MiniGPT4- Videoo (Ataallah et al. (0244), which exhibits head preference bias, is trained on WebVid (Bain et al.) 2021), which suffers from static appearance bias (Lei et al., (023). where visual cues the beginning of the video are often suffi- cient for understanding. We also observe that models tend to exhibit lower positional bias on tasks in which they excel. For example, Qwen2.5-VL VL shows reduced positional bias the OCR task compared to its bias on other tasks, while MiniGPT4-Vide (Ataallah et al., (024) displays more noticeable bias OCR and AR tasks. There remains considerable room for improvement in ad- dressing positional bias, and we discuss potential directions in Sec. 4.4 For most models, a clear margin between Pmoen and Ptezt is observed, indicating that although positional bias exists, the models do not exhibit complete blindness to the context. Finally, although the positional bias ob- served in LVLMs exhibits similar patterns in LLMs, mitigating positional bias in LLMs does not appear to directly transfer to LVLMs. A detailed comparative analysis is presented in the Sec. A.
4.3 FURTHER ANALYSIS
We further investigate the effect of context length, context variation, and model scale, providing actionable insights for future model enhancement.
Various Customized Contexts. Our benchmark supports flexible customized contexts. To explore positional bias under different real-world scenarios, we experiment with six distinct context types while keeping the context length fixed. These include: (1) template context, (2) short video context, (3) long video context, (4) multi-video context, (5) long-text context, and (6) interleaved video- text context. These contexts sourced from public datasets, detailed in Sec. A.4 Finding 1:
8

---

Published as a conference paper at ICLR 2026
Dwen2255VV.22
InternVL. 3-88
LL-VA-OV-78
Mideo-Xl2-78
1.5.0
1.5.0
106
100.7
Robetvv Store
ted
Rolative St00
$5.0
corined lan
Sccontedien
car lor
Relgeive
coried Im
63.0
$3.0
82.0
40
72.0
25
100
88.0
Normalleed Pax cun the Probu
Mormalized can the Probu
Norma Pokizon the Probu
Norma ized Posizon the Probu
LongVILA-11--8
LangvA-78
Viavo-Vide--8
LLaVA-Nide-7726
183.0
10..0
105.0
106.0
Reotive Scor
93.01
41.0
contett Ion
Relative Score
$2.0
accoraacinn
10k contecter
$6.0
$5.0
Recortaciin
13 le
206 conted
Secerteciion
car cartcction
$1.0
86.0
cortectioo
23.0
13.0
100
Normaleed Pos con the ProDo
Normalleed Pos can of the Probe
Norma 12.d Postion the Probe
Nora leod Posizon of the Probe
Figure 8: Effect of context length on positional bias. Each plot illustrates how positional bias manifests across different context lengths on specific LVLM. The x-axis denotes position of the probe normalized by the max context length tested, and the y-axis shows the model's 's relative score. Colored lines represent different context lengths. Positional bias is prevalent across various context lengths. It tends to intensify and the patterns of bias may shift as the context length increases.
LVLMs exhibit more pronounced positional bias in complex multi-modal context scenarios. As shown in Figure Z] models generally maintain high performance and exhibit milder positional bias in simpler contexts, such as template and short video context settings. However, as the contextual complexity increases, such in long video and multi-video scenarios, positional bias becomes more pronounced. Notably, the most severe bias emerges when textual contents are introduced into the context, such as in interleaved video-text and long-text contexts. We attribute this to the lack of training on long mixed-modal context data. This underscores the vulnerability of LVLMs to positional bias when exposed mixed-modal inputs, which are prevalent in multi-turn dialogue or strieval-augmenter generation. Refer to Sec. A.2 for more study on hour-long video context.
Effeet of Context Lenghh. Models tend to struggle with long-context scenarios, possibly due to inference lengths exceeding the training horizon (Liu et al., 2024c). As shown in Figure 8 we in- vestigate the effect of context length on positional bias across eight representative models, including models built upon long-con context LLMs, such as Intern VL3 and Long VA, and models specifically op- timized via long video training, such as Long VILA- (Chen et al.] 2025). Finding 2: Positional bias is prevalent across various context lengths and tends to intensify as the context length in- creases, accompanied by shifts in bias patterns. We observe that models such as Video-X1.2 (Qin) al. 2025), Long VILA (Chen al. 2025), and VLLVV (Li et al. 2025) undergo transitions among three patterns of positional bias: head preference (N) in shorter contexts, followed by a phase of lost in the middle (U), and eventually a shift toward neighbor bias (/)) in longer contexts. Moreover, models like Qwen2.5-VL LL. (Bai et al. 2025) and VA (Zhang et al. 2025b) exhibit U-shaped performance across different lengths. This suggests that LVLMs may inherit the biases of the LLM component. Optimization of positional encoding (Bai et al. (025) and the effectiveness of long video post-training (Qin et al. 2025) should verified through positional bias evaluation. Their overall performance gains videos may stem less from improved comprehension of the entire sequence, and more from improved visual understanding due to exposure to more video data.
Effect of Model Size. According to the scaling law (Kaplan et al. 2020), increasing the num- ber of model parameters generally improves performance across a wide range of capabilities. To investigate the effect of model size on posi- tional bias, we conduct experiments on four repre- sentative models, as presented in Figure Find- ing 3: Positional bias is significantly alleviated as model size increases, consistent with scaling law observed in other capabilities. Within the same model series, larger variants exhibit more stable performance across the entire sequence, as visualized by flatter bias curves and quantified by
InternVL3
LLavA-ov
Ielaaive
Positional Index of the Probe
Posttoral Index of Probe
Qwen2.5 5-VLL
LLaVA-Video
160.0
313.0
Ielekive Scuee
Inlative-Srcce
35.0
$3.0
120
lower Pour in Table They also show greater re- silience to context interference, reflected by con-
Positional Index of the Probc
Postdooal Index of the ProDe
sistently higher relative scores. Our experiments
Figure 9: Effect of model size.

---

Published as a conference paper at ICLR 2026
show that the superior overall performance of larger models long video tasks stems only from enhanced comprehension capabilities, but also from their improved handling of blind positions where small ones
prone to positional bias.
Open-ended Questions. Video-LevelGauge also contains 120 open open-ended descriptive questions, where LVLMs are instructed to describe mentioned scene. We use GPT-4 to evaluate model respo- nses by comparing them against annotations. As shown in Table 2 observe slightly more pronounced positional bias when using descrip- tive tasks compared MCQA formatted eval- uation. We attribute this to descriptive tasks requiring more fine-grained and comprehensive perceptions, whereas multiple-choice questions typically target specific object or action, mak- ing them relatively easy. As a result, portions of potential positional bias may go undetected in MCQA settings.
Table 2: Evaluation open-ended questions.
Models
Size
Pmenn
P von
Pons
Smein
InterVL33
8B
90.0
8.2
7.1
70.7
LLaVA- Video
7B
90.3
7.9
4.9
70.0
LLaVA-OV
7B
91.7
5.1
66.8
Qwen2.5-VL
7B
87.5
13.8
68.3
Qwen2.5-VI
32B
92.5
7.2
72.8
Vidco-XL2
7B
90.6
5.4
71.4
4.4 DISCUSSION.
Contextual positional bias, an inconspicuous but critical issue of LVLMs, may stem from annotation biases the training data, as prior works (Jung et al. 2025 Yuan et al. 202T) found that models may rely on the linguistic priors of the query for grounding rather than genuine visual understanding. Our analysis suggests that beyond scaling up model parameters, several strategies hold promise for mitigating positional bias, including long video training and enhanced context search algorithms. In addition, we observe amplified positional bias in multi-modal input scenarios, highlighting potential of training with interleaved video-text data and developing cross-modal context search algorithms. It tends to worsen with longer context lengths; thus, excellent video token compression may not only improve efficiency but also mitigate bias issues. Finally, evaluating positional bias is anticipated to assist in areas of hallucination mitigation and position encoding optimization. The key intuition is that the model should be able comprehend any part of the context answer the questions, given that the relevant content may appear anywhere in the sequence.
CONCLUSION
In this paper, we emphasize the importance of evaluating positional bias in LVLMs. To this end, introduce Lideo-LevelGauge, extensible benchmark tailored for assessing positional bias in video understanding. We systematically evaluate 27 state-of-the-art LVLMs and reveal disparity in positional bias between commercial and open-source models. Further analysis shows that complex contextual scenarios, especially multi-modal interlaced input, can exacerbate the bias. Moreover, positional bias is pervasive across various context lengths, intensifying and shifting in pattern the context length increases. Larger model variants tend to be robust. These findings suggest that future model advancement should not only focus on enhancing overall comprehension but also explicitly optimize positional bias, particularly in long video and video-text interleaved understanding tasks.
ACKNOWLEDGEMENT
This work was supported in part by the Artificial IntelligeneeNNationn Science and Technology Major Project under Grant 2023ZD012120 and in part by the National Natural Science Foundation of China under Grant 62121002.
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. this study, no human subjects or animal ex- perimentation involved. All datasets used sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. We have taken care to avoid any biases or discrimi- natory outcomes in research process. No personally identifiable information used, and no
10

---

Published as a conference paper at ICLR 2026
experiments conducted that could raise privacy security concerns. We are committed to maintaining transparency and integrity throughout the research process.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are reproducible. All code and datasets have been made publicly available facilitate replication and verification. The experimental setup, including model configurations, and data sources, is described in detail in the paper assist others in reproducing our experiments.
REFERENCES
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jes- sica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, al. Pixtral 12b. arXiv preprint arXiv:2410.00077. 2024.
Alibaba. Qvq-max. https::/gveelln.giihubbb-- 2025.
max max preve preview/
Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal Ilms for video understanding with interleaved visual-textual tokens, 2024. URL Btl.ttts:: aaaaiiiiiv. /ttp//iiiv...raaa////aa//aa/aaaa 2404. 03413
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond, 2023. Uttstttss::/arxiv.ooo abs///b///// 2308 12966
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392 2025.
Max Bain, Arsha Nagrani, Gäl Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In IEEE International Conference Computer Vision, 2021.
Bytedance. Doubao-seed-11 1.6. https://seede httpe::/saeedbbyteerrrccccmmm/m
2025.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we the right way for evaluating large vision- language models? In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), pp. 94237, 2024. WRLp:ttss://eet http:::///euurs.... ps. cc/vi twt.ttua///0 1/2024 poster/94237.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Yihui He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. LongVILA: Scaling long-context visual language models for long videos. The Thirteenth International Conference on Learning Representations, 2025.
URL http:htts///ppeeeeeee///e net. wtt.er/wwwwweeeeeeeoeeeeee.mcmce /tte//ouuuuu11ccCAAACCYC www.:ddwwwwwwccccc......
net. forum?
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Asya Fadeeva, Zach Gleicher, Thang Luong, and Niket Kumar Bhumihar. Gemini 2.5; Pushing the frontier with advanced reasoning, multimodality, long context, and next generation org/ abs
agentic capabilities, 2025. https://: wttp://wwwwrr////55.../.// 06261
Chaoyou Fu, Yuhan Dai. Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li. Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 24108- 24118 18, June 2025.
11

---

Published as a conference paper at ICLR 2026
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition (CVPR), pp. 18995-19012. IEEE, 2022.
Bo Hc, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao. Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: Memory-augmentt large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13504-13555 2024.
Minjoon Jung, Junbin Xiao, Byoung- Tak Zhang, and Angela Yao. On the consistency of video large language models in temporal comprehension. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13713-13722, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Hugo Laurengon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023. URL /Rttt:ts:
www.:orrggaa//////e 16527
Jie Lei, Tamara Berg, and Mohit Bansal, Revealing single frame bias for video- and lan-lagg learn- ing. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp. 487-507, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/ v1/2023. a22023.acl-long229 URL TtLLhtts::/ https/////aaaaatttaagoee /
orteerrg////// acl- 1on1--1oon.200 29/
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-onevision. Easy visual task trans- fer. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https:
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-moda. video understanding benchmark. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22195-22200 2024a.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340. Springer, 2024b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5971-5984, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10. emn p-ma n. 342/
main. 342. URLhttps: http:////caaatthhoooooo wrtp:/24..9/0020..
Jajjun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input, arXiv preprint arXiv:2408. 15542, 2024a.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024b. doi: 10.1162/acLLa.00633 URL
Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation, 2024c. URI.httpss: htt.htttss//aaiiiv..oo
05209
12

---

Published as a conference paper at ICLR 2026
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang. Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCompass: Do video LLMs really understand videos? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACI 2024, pp. 8731-8772, Bangkok, Thailand, August 2024d. Association for Computational URL
Linguistics. doi: 10. (0.18653/v1/20244
fin.inng--al..11. 0.55:////204..nnnngg-......111
org/
2024. findiiindngss ac1.5177
Haoyu Lu, Yuqi Huo, Yifan Du Zijia Zhao, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in a video haystack: tor for video milms. In Y. Yue, A. Garg, N. Peng. F. Sha, and R. Yu (eds.), International Conference on Representation Learning, volume 2025, pp. 99750-99782. 2025. URL URL http:/ttttppcceeeeeeeeee hetp.//eep.............../ppeeee fttp:/reeeeppppeellsseeeeeeeeeee wtt.s///aaaaerr/22//1L12
scalable synthetic evalua-
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volumee 1: Long Pa- pers), pp. 12585-12602 Bangkok, Thailand, 2024. Association for Computational Linguss- tics. doi: 10. 10.18655/112004.aclllog acl- 679/
URL https: http:s:s//acaatthhhhto walllanthooogy ogg.://00...g/22...
Bennet B Murdock Jr. The serial position effect of free recall. Journal of experimental psychology, 64(5):482, 1962.
Elliot Nelson, Georgios Kollias, Payel Das, Subhajit Chaudhury, and Soham Dan. Needle in the haystack for memory based large language models. arXiv preprint arXIvv2407.01437, 2024.
OpenAl. Gpt-4o.
cww/ httpa//aawaa.aaaaaaaaa//ao/aaaaa
2025.
Jongwoo Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryu, Donghyun Kim, and Michael S Ryoo. Too many frames, not all useful: Efficient strategies for long-form video QA. In Workshop on Video-Language Models @ NeurlPS 2024, 2025. URL TRRLhttps::/// httttp::://peeeeeeeeeeee
net
Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, Juenjie Zhou, Shitao Xiao, Bo Zhao, and Zheng Liu. Video-xl-2: Towards very long- video understanding through task-aware kv sparsification. arXiv preprint arXiv:2506. 19225, 2025.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18221-18232, 2024.
Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 6479-6488, IEEE, 2018
Zhijie Tan, Xu Chu, Weiping Li, and Tong Mo. Order matters: Exploring order sensitivity in multimodal large language models, 2024. /t///tttps////rvv.orr////// /2100 16983
Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, et al. Mimo-vl technical report, 2025a. URL https://aax.. org/ orq/abs/220 abs//506. 03569
V Team, Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng. Ji Qi, Junhui Ji, et al. Glm-4.5v and reasoning with scalable reinforcement learning. 2025b. URL htttps://arxiii
Towards versatile multimodal
abs/
2507.01006
Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Identifying and mitigating position bias of multi-image vision- Languaee models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10599-10609, 2025.
13

---

Published as a conference paper at ICLR 2026
Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long- context capability of multimodal large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Pupers), pp. 3221-3241, Albuquerque, New Mexico, April 2025, Association for Compu- tational Linguistics. ISBN 979-8-89176-189-6 doi: 10.18653/w1/0223 hatp222...naallooooo naacl-long. 166. URL attps://aclanthonn
wwt.oogg..66666 166/
Weihan Wang, Zehai He, Wenyi Hong. Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong. Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2024a.
08035
Weiyun Wang, Shuibo Zhang. Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang. Lewei Lu, et al. Needle in multimodal haystack. Advances in Neural Information Processing Systems, 37:20540-20565, 2024b.
Xilin Wei, Xiaoran Liu, Yuhang Zang. Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang. Xipeng Qiu, and Dahua Lin. VideoroPE: What makes for good video rotary position embedding? In Forty-second International Conference on Machine Learn- ing, 2025. URL httpt:ttttppeetteeee
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. Advances in Neural Information Processing Systems, 17:28828-288577 2024a.
Tsung-Han Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E Gonzalez, Trevor Dar- rell, and David M Chan. Visual haystacks: A vision-centric needle-in-a-baystack benchmark. arXiv preprint ar axXiv:207.13766, 2024b.
Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Mitigating object hallucination via concentric causal attention. Advances in Neural Information Processing Systems (NeurIPS) 37, pp. 96152, 2024. http://s///epeeeeeeeeeeeeee/e/ee
1b5mV
Poster Presentation.
Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyza- guirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking temporal search for long-form video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 8579-8599 June 2025.
Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao. Dahua Lin, Boxun Li, al. Lv-eval: A balanced long- long-context benchmark with 5 length levels up to 256k. arXiv preprint arXiv:2402.05136 2024.
Yitian Yuan, Xiaohan Lan, Xin Wang, Long Chen, Zhi Wang, and Wenwu Zhu. A closer look at tem- poral sentence grounding in videos: Dataset and metric. In Proceedings of the 2nd international workshop human-centric multimedia analysis, pp. 13-21, 2021.
Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing spatial-temporal object understanding with video 1lm. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18970-189988 2025.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501. 13106, 2025a.
Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple LLM framework for long-range video question-answering, In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 21715-21731 Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/v1/022. emnlp-main. 1209. 1209. URL. htt.pss//acllanthooooyy..o
2024. emn pp-main, in 12097
14

---

Published as a conference paper at ICLR 2026
Peiyuan Zhang. Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vi- sion. Transactions Machine Learning Research, 2025b. ISSN 2835-8856. URL https:
//openreview. net./ f forum? worrummmdd=000 1=00mmmd=300
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024b.
Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, al. Mmvu: Measuring expert-level multi-discipline video un- derstanding, In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 8475-8488 2025.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, Xi Yang, Yongping Xiong. Bo Zhang. Tiejun Huang, and Zheng Liu. Mlvu: Benchmarking multi-task long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 13691-13701, June 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 2025.
Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, and Haibin Ling. De- tection and tracking meet drones challenge. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4A(11):7800-739, 2021.
Heqing Zou, Tianze Luo, Guiyang Xie, Victor Xiao Jie Zhang, Fengmao Lv, Guangcong Wang, Junyang Chen, Zhuochen Wang, Hansheng Zhang, and Huaijian Zhang. Hlv-1k: A large-scale hour-long video benchmark for time-specific long video understanding, 2025. URL https:
7/a /aax.vv.rrg/bb//2501000
A APPENDIX
Video-LevelGauge developed for measuring the specific position video under- standing capability (i.e., positional bias) of Large Video Language Models. To isolate contextual interference and offer both flexibility and scalability, we adopt a synthetic approach to construct the dataset, facilitating more comprehensive evaluations. This paper actively explores and analyzes the influencing factors of positional bias, as well as the magnitude of bias exhibited by various models under different inputs in real-world scenarios. However, overall, the formation mechanisms and mitigation methods of positional bias remain open questions. On the other hand, traditional video benchmarks undoubtedly hold irreplaceable value. They reflect the model's overall performance on authentic videos. We believe that Video-LevelGand can serve as a useful complement to existing benchmarks, providing specific and controllable evaluation.
Limitations:
The Appendix is organized as follows:
Analysis LVLMs positional bias vs. LLMs positional bias (Sec. A.
Validation on hour-long video context (Sec. A.2).
Extended set constructed on natural long vidcos (Sec. A.3
Details of various customized contexts (Sec. A.4)
Bias Pattern Recognition (BPR) (Sec. A.5).
More details of evaluation protocol (Sec. A.6).
More examples of multi-task QAs (Sec. A.7p.
Detail of human refinement and annotation prompts (Sec. A.8).
Design criteria of Video-LevvllGagee (Sec. A.9
Effect of probe length (Sec. A.10
15

---

Published as a conference paper at ICLR 2026
Vides-XL2-78
LongWA-7B veanie
InterVL33-88 488
MBMG-V1-7B
MIMO-V 7B think
lounciy bek
Figure 10: Validation of positional bias on hour-long videos using six LVLMs.
A.1 LVLMS POSITIONAL BIAS vs. LLMS POSITIONAL BIAS
As observed, the positional bias in LVLMs exhibits similar patterns in LLMs, such as U-shaped performance. It seems that mitigating positional bias at the LLMs could directly transfer to the LVLMs. However, our experiments indicate that the positional bias in LVLMs differs from that in LLMs. We conduct in-depth analysis their distinction in light of the experimental results.
The process of multimodal training of LVLM may substantially alter the positional biases in the LLM component. For instance, as observed in Table Qwen2.5-VL-77, --2e 7B, VideoRefer, and the Qwen2.5-7B language model, yet they exhibit markedly different patterns of positional bias in video understanding. This suggests that mitigating positional bias in LLMs may not be directly transferable to LVLMs.
are all LVLMs derived from multi-modal training on
The severity of positional bias in LVLMs is correlated with their visual understanding abil- ity. As analyzed in Sec. 4.2 and illustrated in Figure [3 LVLMs exhibit reduced positional bias on visual tasks in which they excel. This suggests that visual positional bias in LVLMs is, some extent, attributable to insufficient visual capabilities, which must be acquired through visual learning, similar to the mitigation of the multi-modal hallucination issue.
The issue of positional bias is more complex in multi-modal scenarios, as interleaved multi- modal inputs can exacerbate positional bias (see Figure (7) and introduce more diverse position/order-rrlatee issues (Tan et al.] 2024 Tian et al., (025). In such scenarios, the composition and variability of contextual elements differ substantially: for instance, the embedding discrepancy between two adjacent video tokens is different from that of text tokens. This inherent distinction between textual sequences and multi-modal sequences highlights the need for community attention to multi-modal positional bias.
In summary, similar to other multi-modal capabilities, enhancing the LLM component is expected to improve video positional bias. However, discussed above, positional bias in multi-modal sce- narios deserves special attention, akin to multi-modal positional encoding (Wei et al. 2025).
A.2 VALIDATION ON HOUR-ILONG VIDEO CONTEXT
We further validate positional bias on hour-long videos. To isolate the influence of frame sampling, typically fix the frames sampled from the probe, insert them into the context, and then feed the sequence into the model. Here, as a form of real-world validation, we adopt a straightforward setting. We insert the same probe video clip into different positions within natural long video and construct the entire long video (.mp4). The synthetic long videos are then input to the LVLMs, which are sampled by the model's own sampling method. This straightforward setting suffers from two limitations. (1) Although the insertion positions are precise, the content and even the number of frames sampled from the probe by the model may vary for different positions, which can impact study of positional bias; (2) is inefficient in practice, since new video file needs to be synthesized and stored temporarily for every tested position. Considering this, we take as a supplementary validation method rather than the default setting of our benchmark.
50 long videos are collected from LVBench (Wang et al., 2024a) and HLV-1K (Zou et al. 2025 with an average length of 61.1 minutes. Six representative models are evaluated. Although some models adopt a default fps-basee sampling strategy, the video lengths exceed the maximum input limits of the models, leading them to perform uniform frame sampling. Specifically. Qwen2.5- VL. MiMo-VL, Mideo-XL2, and Long VA sample 768 frames for each hour- long video, InternVL3 samples 128 frames. The results presented in Figure 10
As described in Sec. A.4
16

---

Published as a conference paper at ICLR 2026
It can be observed that, in hour-long video scenarios, LVLMs exhibit more severe positional bias, i.., lower relative score and greater differences between positions, compared to synthetic video (Figure (). Among them, Video-XL2 (Qin et al. 2025), which is specifically optimized for long videos, shows relatively higher score on hour-long videos. Comparing the "think" and "no-think" modes of MiMo-VI (Team al.] (2255)), the "think" mode demonstrates slight improvement in mitigating positional bias. These results are consistent with the observations in the main paper.
A.3 EXTENDED SET CONSTRUCTED ON NATURAL LONG VIDEOS
Beyond synthetic settings, we further construct an extended set by annotating probes on natural long videos. We first collect 30 long videos from LVBench (Wang et al. (024a) and HLV-1K (Zou et al.) 2025j, with an average length of 58.8 minutes. Then, a sliding window of minutes applied to the long video extract 10 uniformly distributed, non-overlapping segments probes. For each segment, annotation is performed following the method described in Sec. 3.2 In total, we obtain 369 MCQAs, with the number of questions being approximately balanced across positions. During evaluation, each video is input to the model ten times, with each run evaluating the model's response to multiple-choice questions regarding one of the ten probes.
Notably, in addition to the lack of flexibility in adjusting probe positions and scaling context length, there are two potential issues in using such data. (1) There are uncontrollable relevant contents of probe segment in natural video context, i.e., information leakage, leading to inaccurate evaluation of positional bias. (2) The quality and number of frames sampled by the model for probe segments may be different, fundamentally impacting the model's understanding of the segment. Therefore, we treat this as an extended set for real-world validation, while our Video-LevelGauge primarily adopts the standardized probe and customized context design paradigm.
We adjust the computation of relative scores in Eq. !
defined as:
Swideo S Smeta
RS.
where St Video is the model accuracy answering MCQAs an- notated on the i-th position with the entire video as input, and Smeta the accuracy when the probe segment pro-
135.0
35.0
InterVL-8B
VL think
vided standalone without any context. We present S Smeta
Accuracy:s
25.0
Video XL2.7e 78
i = 2, .... 10 in Figure TT] It can be observed that the dif- ficulty of MCQAs constructed on different positions varies, indicating the procedure in Eq. 2 is necessary. Using the ex- tended set, six representative models are evaluated. Specif- ically, Qwen2.5-VL, MiMo-VL VL, Video-XL2, and Long VA sample 768 frames for each long video, Intern VL.3 samples 128 frames. Results are provided in Figure. [2
75.0-
26.0
$5.0
20.0
$5.0
Index of the Segments (Probes)
Figure 11: Illustration of St Smeta-
Compared with Figure 10 the positional bias observed natural long videos is obscure. In partic- ular, the differences across positions are smaller, and the bias pattern is not as evident, such as the U-shaped trend. We attribute this the fact that natural long videos contain many uncontrollable contextual correlations that lead to information leakage, thereby interfering with the measurement of the target position (see LVBench (Wang et al. 2024a) for video cases). To further validate this, we conduct experiments by comparing model response accuracy under three input settings. Text-only: only the MCQAs of segments on 3rd position are in- putted. Context: the visual content of segments on the first position together with the MCQAs annotated on 3rd segments. Target: the visual content of 3rd seg- ments together with the corresponding MCQAs. The results are shown in Table 3 Context is significantly higher than felt,only, indicating the existence of con- textual information leakage.
Table 3: Comparison of different input.
Models
Text-only
Context
Target
VL.-88
34.7
$6.5
73.9
HL-o-VL-7B
36.9
58.6
78.2
Vidco-XIL2-BB
32.6
50.1
67.3
17

---

Published as a conference paper at ICLR 2026
Owen2.5544 5-
Vides-XL2-78
LongWA-7B
IntervV-66 68
MBMG-V7-7B
NIMo-V 78 think
Figure 12: Evaluation of six LVLMs using the extended set constructed on natural long videos.
A.4 DETAILS OF VARIOUS CUSTOMIZED CONTEXTS
As shown in Figure we investigate positional bias across six distinct context types while keeping the context length fixed. These include: (1) template context, (2) short video context, (3) long video context, (4) multi-vide context, (5) long-text context, and (6) interleaved video-texs context. Visual We detail the configurations as below.
examples of six contexts are illustrated in Figure T5
Template context. We construct the template video by filling all frames with the ImageNet mean RGB pixel values: [123.675, 116.28. 103.53). The template video is uniformly sam- pled 54 frames as the context. We sample 6 frames per probe and insert them into different positions of context, ensuring the probe frames are consistent to isolate sampling impact.
Short video context. We randomly sample 100 videos from the short video subset of VideoMME (Fu al.) (025). During evaluation, each probe is randomly paired with a background video. which remains fixed throughout the positional evaluation. The back- ground video is uniformly sampled 54 frames, into which 6 probe frames are inserted..
Long video content. We randomly select 50 long videos from long video benchmarks of LVBench (Wang et al. (2024)) and HLV-1K (Zou et al. 2025). using the same processing and evaluation settings in the short video context. Length statisties of long videos: the shortest is 44.9 minutes, with an average of 61.1 minutes.
Multi-video context. This setting, described in Sec. [.1 and Sec. A.6 serves as our default configuration, as it provides flexibility in both context length and complexity. It is eco- nomical and effective approach that enables efficient utilization of video content (probe), while ensuring flexibility and diversity of the context and compactness of the dataset.
Long-text context. We extract 3,500 paragraphs from OBELICS (Laurençon et al. 2023
To ensure context length comparable to that of video-based contexts, each paragraph is truncated according to the tested model. For example, for Intern each paragraph is limited to approximately 1,500 tokens to match the token budget of a video clip (6 x 256 tokens). During evaluation, nine paragraphs are randomly selected to form the context, which remains fixed throughout the positional bias assessment.
Interleaved video-text context. This setting is variant of the long-text context. During evaluation, each background text is replaced with a video from our benchmark with a prob- ability of 0.5, while all other settings remain consistent.
A.5 BIAS PATTERN RECOGNITION
To recogmize the bias pattern (BPR) of each model, we apply both linear and quadratic fits to the rel- ative score (RS) at all positions. Specifically, we first perform a linear fit to estimate the global trend. Based on the slope and the mean squared error (MSE) of the fit, we clarify BPR into two coarse cat- egories: (I) monotonic type, which includes Stable with milder bias (-), Neighbor preference (/)) and Head preference (N) with small residuals and relatively consistent trend; (2) non-monotonic type, which includes Lost in the middle (U) and Volatile score (W), typically exhibiting larger resid- uals that indicate deviation from linearity. We then apply quadratic fitting to the non-monotonic type to further distinguish structured non-linear patterns.
Given the relative score {RS,, RS,.... RSw), }. where N is the number of evaluated positions, fit the data with both linear and quadratic polynomials:
(x) = kx + h,
(2)
(0) ax² + bar + C,
(3)
18

---

Published as a conference paper at ICLR 2026
pebeisrntie Recogrk.e
OCW
Oljec Recoyrtinn
Acvitude Recogrdor
Acten Recopn ton
Obect Countey
Averoon
ROAGHC Scate
Figure 13:
Illustration of positional bias of Qwen2.5-VL- VL-7BB across six video tasks. The x-axis repre- sents the positional index of probes, while the y-axis indicates the model's relative score. Qwen2.5- VL exhibits reduced positional bias on the OCR task compared to its bias on other tasks.
where (ar) and ((x( denote the fitted functions from the linear and quadratic fits, respectively; represents the slope coefficient of the linear fit.
Then we calculate the mean squared residuals for both fits separately:
MSE]
RS: - g) (())
(4)
1=1
MSE2 = Σ
RS: - g)2 (xx))
(5)
where (x;) and g(2) (x;) denote the predicted value at the i-th position from the linear and quadratic fits, respectively. Based on this, categorize BPR into five types:
MSEE < 3 and | S 0.5,
if MSE] I 3 and k > 0.5,
BPR
if MSE; < 3 and k < -0.5.
U if MSE; > 3 and MSE₃ < 2, W
otherwise.
A.6 EVALUATION PROTOCOL
To isolate the impact of frame sampling and to control the context length (see analysis in Sec. A.2 we sample frames from the probe and the background video separately. Specifically, 6 frames are uniformly sampled from the probe, which remain consistent across different testing positions. The background video is processed using each model's official implementations with 54 frames. The probe frames are then inserted into different positions within the context before being fed into the model. For commercial models, since their sampling strategies are not accessible, we input probe frames and 54 contextual frames jointly in the multi-image format.
We propose three statistical metrics, each capturing distinct aspect of the model's positional bias. To enable comprehensive evaluation and facilitate comparison across models, we introduce com- posite metric (CM), defined as:
CM = - Pren + a2 (100.0 - Pmeaa) + ary Pvar
(6)
where 022 a' are balancing parameters. We empirically set them to 4.0, 3.0, and 4.0, respec- tively, ensure that each component contributes comparably to the final score. Higher CM values indicate more pronounced positional bias, depicted in Figure 2
A.7 EXAMPLES OF MULTI-TASK QAs
As shown in Figure 13 model can exhibit varying degrees of positional bias across different tasks. Due to space constraints, only a limited number of multi-task probe examples are included in the main paper. Additional examples of multi-task QAs provided in Figure 16 and Figure Visual examples of six contexts are illustrated in Figure [5] Following VideoMME, we evaluate multiple- choice QA performance by prompting the L.LLM with:
19

---

Published as a conference paper at ICLR 2026
Select the best answer the following multiple-choice question based on the video. Respond with only the letter (A, B. C, or D) of the correct option. QUES- TION, OPTIONS. Answer with the option's 's letter from the given choices directly.
For open-ended items, we prompt LVLM with:
This video consists of multiple segmeni.. You are tasked to provide detailed description of the segment described as: QUESTION. Focus only on this part and do not describe other segments. Please describe the visual content of this segment in as detail as possible, including objects, actions, background, and any temporal progression.
The accuracy of responses for multiple-choice QA is computed locally using exact match. For open- ended questions, following (Maaz 2024), we use GPT-4 an evaluator. The prompt used for scoring is provided in Figure 24
A.8 DETAIL OF HUMAN REFINEMENT AND ANNOTATION PROMPTS
We construct probe QAs through a three-step workflow that combines automated generation with human refinement. Considering video collection process, the construction of Video-LevelGauge involves four major stages requiring human participation.
Video Collection. Diverse videos are manually collected from existing datasets, and blurry, static, and duplicate videos filtered out. Two annotators participate in this stage, tributing approximately 40 hours of human effort.
QA Generation. Two human annotators cross-validate the annotations, generating the task definition along with three positive and three negative QA examples for each video task.
QA Refinement. Twelve human annotators further review the QAs that pass the model- based blind filtering and hallucination filtering. Annotators are asked to remove or refine QAs that involve answer leakage, commonsense questions, answers that are inconsistent with the video content. The twelve annotators are divided into two groups. Each group reviews portion of the data first, and they then cross-check each other's assessments. Over 90 hours of human effort are expended in this stage.
Distractor Construction. Eight annotators examine the LLM-generated multiple-choice questions, filtering out distractors that are trivially distinguishable video content. MCQAs that do not meet quality requirements are LLM-rrgenerated until they rejected after three attempts. The eight annotators also divided into groups for cross-validation, contributing approximately 60 hours of human effort.
valid given the
Human annotators are encouraged to focus on validation and refinement rather than rewriting. thereby reducing their workload.
Besides, we elaborate the prompt of automated generation. (1) QA generation. The prompts used for frame captioning and QA generation are presented in Figure T8 and Figure [9] (2) QA refinement.
The prompts used for for blind filtering and hallucination filtering are provided in Figure 20 (3) Distractor construction. The prompt employed instruct LLMs in generating distractors is shown
in Figure 21
Task definitions are provided in Figure 22 and examples of discarded QAs shown in Figure
A.9 DESIGN CRITERIA OF VIDEO-LEVELGAUGH
VideoeLevelGauge consists of 438 videos and 1,177 MCQAs built upon them. Compared with Video-MME (Fu et al. 2025), which contains 900 videos and 2,700 QAs, its scale is relatively smaller. We clarify that we intentionally keep the number of probes moderate, although constructing more probes does not require substantial annotation labor under the adopted standardized probe and customized context paradigm. To evaluate positional bias, each probe needs to be inserted into multiple context positions. thereby expanding the scale of evaluation. For example, with 10 positions by default, evaluating 1,177 QA pairs results in over 10,000 model inferences.
20

---

Published as a conference paper at ICLR 2026
MiMo-VL-78
5V-1088
28.-7
Relative Score
50.1
Relstive Socore
$2.
6
Dink
DHIK
Positional
Posibional Index Probe
Figure 14:
Effect of thinking mode on positional bias. Thinking mode can alleviate the positional bias issue to certain extent.
Table 4: Effect of probe length. Probe length (PL.) is controlled by the number of sampled frames. Although increasing the number of sampled frames significantly improves the absolute performance (meeaa), the positional bias remains largely unchanged.
Models
PL
P
meaa
Pran
P
var
Smeta
87.7
7.7
5.8
59.2
Intern Intern VL3-8B
90.3
8.0
5.3
70.5
8
90.6
8.7
5.4
74.9
88.1
8.0
5.6
54.3
LLaVA-OV-7B
87.3
4.2
2.1
65.6
88.2
6.5
3.2
67.7
90.5
11.0
11.2
60.1
Qwen2.55VV..77B
89.6
12.4
11.7
68.2
8 90.9
10.7
10.0
72.7
This also why we primarily adopt the MCQA format, as its evaluation is significantly faster than open-end questions. To enhance the usability of the benchmark, number of probes moderate, and allocate more effort to improving their quality, For instance, we focus preventing single-frame blind-bias predictions, and to suppress potential context leakage (Sec. (.2.3). These quality- quality-validation efforts are essential for ensuring our reliable assessment of positional bias
intentionally keep the
Finally, the probes are highly extensible. They serve as "seeds" that can be flexibly inserted at arbitrary positions, with adjustable context lengths and categories depending the testing scenario. The key intuition is that the model should be able to comprehend any part of the context to answer the questions, given that the relevant content may appear anywhere in the sequence.
A.10 EFFECT OF PROBE LENGTH
Furthermore, examine how probe length affects positional bias by varying the number of sampled frames of each probe (2, 6, and 8). As shown in Table increasing probe length improves the Smetas richer information be sampled. Notably, positional bias basically maintains across different probe length. This suggests that a model's robustness to positional bias largely independent of probe length and is instead an inherent characteristic of the model.
B LLM USAGE
Large Language Models (LLMss) were employed to assist the writing and polishing of this manuscript. Specifically, LLM was utilized to refine the language, enhance readability, and ensure clarity across various sections of the paper. Its support focused on tasks such as sentence rephrasing, grammar correction, and optimization of the overall textual flow.
Notably, the LLM was not involved in ideation, research methodology design, or experimental plan- ideas, and analytical work were independently developed and
ning. All research concepts, conducted by the authors. The LLM's contributions were limited solely to improving the linguistic quality of the paper, without any involvement in the scientific content data analysis.
21

---

Published as a conference paper at ICLR 2026
Table 5: Validation of potential leakage between the probe clip and background videos. Smeto represents the accuracy when the probe clip is presented to LVLMs independently, without any surrounding context. B1-9 represents the accuracy obtained when all nine background videos are presented together with the QA pair, while B; (i=l...9) denotes the accuracy when each background video provided individually along with the QA pair. It is observed that when background videos and the QA pair are provided as input, the model's response accuracy approximates that observed with text-only imput. This indicates that the model cannot correctly answer the query question by referring to the background video content, which can be attributed to the manual deduplication performed during video collection (Sec 3.2.1
Models
Text-only
Smeta B1-9
B1
B2
Bg
B4
B3
Bg
13
Bg
By
Qwen2.5-VL-7B 28.6
32.5
28.7
31.0
30.7
30.9
30.2
IntemVL3-8B
33.9
34,5
35.3
35.6
35.6
34.4
33.5
35.5
35.6
37.5
33.4
Table 6: As supplement to Tab 1 P Pmean and Pran of each sub-task are reported, providing clearer view of the overall positional bias for each task.
Models
Size
OCR
Pran
AR
Pran
OR
From
OC
Frux
"moon
RR
Pres.
P Fooan
ActR
Pron
Two-stager Models
Caprio Owent
89.3
97.1
11.3
12.9
82.3
79.7
11.3
Caption GPT44
Models
MA-LMM
78
87.3
90.7
16.7
76.6
20.7
90.4
83.9
6.6
LongVA
MiniGPT4-VD Video
7B
84.6
26.9
88.6
21.9
$6.3
14.9
10.4
76.6
89.4
3.4
LLaMA-VID
138
70.9
23.2
87.5
22.0
85.2
24,2
65
35.5
82.7
34.5
78.8
25.4
Kangaroo
4B
87.4
14.3
78.6
13.2
$3.5
10.4
86.8
24.6
82.3
38.9
92.3
15.6
LLaVA OV
72B
93.9
93.4
92.1
5.4
87.1
15.5
91.9
6.7
Quen2.5
94.8
83.9
27.4
90.0
9.8
83.9
89.1
15.0
Owen2.5-VL
72B
95.3
89.3
20.2
94.9
15.8
90.1
22.2
S7.9
17.7
95.8
7.3
L3sernVL3
8B
91.7
6.7
86.8
11.5
95.0
9.9
91.4
22,6
$1.9
13.9
92.9
10.7
InserVL33
7SB
98.2
92.3
10.6
96.6
5.0
90.6
21
90.9
14.7
96.3
4.5
Lade
VideoLLaMA3
NVIAA
90.2
71.9
34.2
85.2
17.2
79.3
15.1
70.7
17.7
78.9
14.0
LongVILA-IN
7B
86.8
7.3
77.9
22.4
79.0
14,4
90.4
10.7
70.6
12.7
85.7
12.4
Video-X1.
7B
91.9
13.6
70.D
8.0
83.4
10.5
91
17.0
73.5
11.4
87.3
8.6
Vid
7B
97.9
$4.1
13.2
6.8
90.6
85.6
7.2
95.9
6.9
VidgoRofer
68.8
71.5
76.1
10.6
MiMe-VI-RI.
78
94.5
89.6
10.3
94.0
6.7
S3.0
17.6
90.5
12.5
94.5
7.6
VL-RL
7B
97.2
96.0
6.4
98.7
6.5
89.8
21.5
97.9
94.7
9.0
GL.--.5V
108B
GLM 1L5V.
L0SB
98.2
91.4
5.2
99.4
1.7
98.4
6.6
97.9
94.2
6.7
Qwen
QVQ-Max!
94.2
2.3
95.3
2.9
$3.9
6.1
90.1
6.0
90.0
5.6
93.7
7.8
Douban- Domban-Seed-1.6
95.1
4.9
94.3
7.0
98.2
3.5
95.3
8.9
94.0
6.5
98.4
6.2
GPT-4p-late
96.6
97.2
7.1
94.4
4.3
Souule-onnnt-44
Gemini 2.5 Pro"
99.4
99.4
0.0
98.3
7.5
97.4
8.1
98.0
97.9
4.9
The authors take full responsibility for the content of manuscript, including all text generated or polished with the LLM'' assistance. We confirm that all text produced with the LLM's support complies with ethical guidelines and does not lead to plagiarism any form of scientific misconduct.
22

---

Published as a conference paper at ICLR 2026
Standardized probe:
Customized Context:
Multi-video:
Q: the rinses begina wiping nanning waner wrings the counter around the sink, what can be seen in orange plassic box on the right?
(b) Kottle and tea
(6) Beer mugs and tea cups.
Short Video:
1010
Sampling
Long Video:
Insert
Video-text:
Select
multiple-chhioa question video. Respond with only the lettar (A, B, the correct option. As the red cloth option's
Whilepote growth alowed from 17t 5-16-177 tround It's posithve and 77% companies in the recent repor ring seasen...
The nuclese segmentation olgorithm is highlceffllont iififiiuuut shuarions...
oups, letter from the given choices directly.
While growth alowed from 1hth 2010 17 around it's poxtt wwadd 73%
The nuciiar segmentation aigerithm...
Text:
LVLMs
poinns Austral wcanomy yours now, manyyiove told that Austrolia..
FFR Stodowld
akimmshh...
more then actadium
Template Background:
Figure 15: Visual examples of customized contexts. We illustrate six types of customized contexts and how the probe is inserted into each of them. Refer to Sec A.4 for more details.
23

---

Published as a conference paper at ICLR 2026
T OCR
2
Attribute Recognition
@ Object Recognition
4 Object Counting
Qu The
Q: There tabll,
black high heels hat with a white lobel. What la the material meriiaalee of the hat??
athlute sprinting archsing Zealand to jump over a high jump bar. Which company' s sement did not appear in the background?
QU:
Q:: the of Men's
trident diagram chemical structure shown. What is labeled on the yellow wectnngllrr block neat?
on
Jump Final
athllee Tentoglou
runs, jumps into the sand pit, lands, and wvs wallss ewayy. How many athletess long jumps are recorded ded in the segment?
lat SWEET BUTTER.
[e] Wool.
[c] Leather.
lal SONY.
[c] TDK.
14) BUTTER SWEEE. BUTTER
[b] Knitted.
[d] Canvas.
[6] TCL.
[d] SFIKD
(a)
(1)
Icl
[dl
Relationship Recognition
@ Action Recagnition
Instructed Description
Q: The blue robotic czt holds up the bag with mushrooms
Q:: What does the man in the black
Q: Please describe in detail the sopment
painted on What is the correct layout of the model tree,
sliir do after handing the flowers the blonde
that shows person in headle measurin
robotic
(a) from to right robotic cat, tree, boy.
(a) Light the candle for the lady.
mezsure susuy day.
Annotation: The video shows person engayed
Ibl From left to right are model tree, robockc cat, and bay.
(b) Perform magic trick.
outdoor
from ieft to right are boy, model tree, and robotic cat.
(cl Invite the lody to dance.
woodworking on sunny day, standing on green grass throughout. Initialyy the person, weating a gray
jdr From left right are boy rabotic cat, and model tree.
[d] Pour wine lody.
BODSERR
Figure 16: Examples of probes for six MCQA formatted evaluation tasks and one open-ended in- structed description task. Each question is described with scene mention and task instruction to ensure clarity, requiring genuine visual comprehension.
① OCR
@ Attribute Recognition
T Object Recognition
X Object Counting
swnches view of the athleckes stadium with the blue track. What is the national abbreviation for lane
speaks camera, there as man wearing white top and red shorts walks up to the ping table, what color Blue.
girl was playing stalin, person in white T T-shirt
camera transitions narrow waterway through tal yellow green grass. How many elephants there?
walks by. What this person holding?
person carrys
bog
(a) TTO.
(c) JAM.
Black.
(b) This person pushis a baby carriage.
Four.
Five.
(b) GBR.
(4) UKR.
(b) White.
(4) Piak.
This pushes bicycle.
(b) Three.
[d] Stx.
person violin.
Relationship Recognition
Action Recognition
Instructed Description
Who closest to the person in the
Q: the recording drane fles closer the man in black and the woman in red
Q: Please descriee in detail the segment that shows an acrial view urban
dark blue shirt when he enters from
door
the
[a] The person the purple striped shirt.
(a) He walks out the shot quickly.
pedestrian overpass.
Annotation: This video presents
[b] the person yellow.
[b] He maises camera to take picture.
(c) person green.
[c] points drone the lady
scone pedestrian overpass spans the road. festuring a gloss railing and prominent red
[d] the person gryy shirt.
Figure 17: Examples of probes for six MCQA formatted evaluation tasks and one open-ended in- structed description task. Each question is described with scene mention and task instruction to ensure clarity, requiring genuine visual comprehension.
24

---

Published as a conference paper at ICLR 2026
Frame Caption Prompt
You are meticulous and detailloriented video frame anallzzer. Your core task to provide an exhaustive, objective, factually accurate description of the given frame, strictly based observable content without any subjective inferences or assumptions.
Please adhere to following guidelines when describing the frame:
1. Detailed Object Description
Identify all distinct objects present (e-g,, people, furniture, natural elements, text, etc.).
For each object, specify:
Quantification:
Exact or approximate number (e.g., "three chairs," "a single tree").
Physical properties: Color (e.g., "maroon with white stripes"), shape (e.g., "circular," "rectangular"), "), size relative to other objects (e.g.. "smaller than the table"), texture (e.g.,
"smooth metal," "fuzzy fabric"), and any notable features (e.g., "a cracked vase, "a sign with faded letters").
State: If applicable, describe the object's condition (e.g., "upright," "broken," " "moving").
2. Relationships Between Objects
Spatial relationships: Clearly define positions relative to one another (e.g.. "the lamp is on top of the desk," "the person is standing to the left of the door," "the cup is beside the book"). Include distance cues where observable (e.g., "closely adjacent," "several feet apart").
Interaction or connection: Note if objects are in contact, attached, interacting (e.g.. "the hand holding the pen," " "the wire is plugged into the socket," "the rain is falling the roof").
3. Language and Objectivity
Use clear, fluent, and precise natural language. Avoid ambiguous terms (e.g., "nearby" can be replaced with "immediately next to" if applicable).
Strictly base descriptions what visually verifiable. not include guesses about intent, context, unobserved details (e.g., instead "the person looks " describe "the person's mouth is downturned and eyebrows furrowed").
goal seeing
description that so thorough reader could visualize the frame accurately without
Figure 18: The collected videos are captioned Trame-wise at FPS using GPT-4o with this prompt. We instruct the model to generate detailed descriptions of each frame, encompassing elements such as the number of objects, visible text, and object relationships. This detailed captioning serves as foundation for subsequently generating multi-task question-answer pairs.
25

---

Published as a conference paper at ICLR 2026
QA Generation Prompt
You are tasked with creating 2-3 highly challenging questions that test memory recall specific details from a video. The descriptions of the video provided by frame-wise. You need to create questions defined as follow. Question Type definition: <TASK_NAME NAME> <TASK_ DEFINITION
The video is described in frames follows:
Frame 1: <FRAME1>
Frame 2: <FRAME1>
The generated questions need to STRICTLY meet the following requirements:
The type of question meets the Question Type definition: <TASK NAME> <TASK DEFINITION
The question is about the video content.
Questions should BRIEFLY refer to unique scenes, events, characters, avoiding vague descriptions.
The question should be as challenging as possible and can only be answered by relying on the video content, rather than commonsense.
Here are two POSITIVE example questions:
Positive Example1: <POSITIVE __A1>>; Positive Example2: <POSITIVE QA2>>
Positive Example3: <POSITIVE_OAS _QA>>
Here are two NEGATIVE example questions:
Negative Example1: <NEGATIVE QA1>;
Negative Example2: <NEGATIVE, QA2>>
Negative Example3: <NEGATIVE_QA3:
Please output questions and reference answers in following JSON format:
l'question': 'oox', 'answer': 'xoo'), )
l'question': 'xxx', 'answer': 'xox'),
Figure 19: Based detailed video captions and human- protocols, GPT-4 is instructed, using this prompt, to generate 2-3 task-specific question- answerr pairs for each video. Different types of questions are generated separately. In total, we obtain 7,319 candidate QAs at this stage. Definitions of each task are presented in Figure 22
26

---

Published as a conference paper at ICLR 2026
Blind Answer Prompt
Your task is to determine whether the given Question meets the provided Question Category Definition and then briefly answer the Question.
First, please carefully read the following definition of the question category:
Question Category definition: <TASK NAME> <TASK DEFINITION>.
Next, please read the following Question: <QUESTION>>
Now, follow these steps:
Analyze whether the example question meets the category definition and state your conclusion (Yes/No).
Use your knowledge briefly answer the example question, guess answer if your knowledge cannot answer it.
Please output your answers in the following JSON format: ("MeetsCategory's 'yes/no', 'answer': "xooox"""
Filtering Prompt
Your task evaluate whether the given be evaluated consistent with the reference answer and correctly answers the question.
First, please carefully read the following question: <QUESTION>.
Next, read the reference answer: SELERENCC_ANSWEER
Now, read the answer to evaluated: <LLM_ AANSWRRRRR.
When evaluating the answer be evaluated, please follow these steps: Carefully understand the question.
Compare the reference answer with the answer to be evaluated.
Check if the
be evaluated consistent with the reference answer.
You should answer with Yes/No first, and Provide brief explanation here.
LVLM Answer Prompt
Your task to evaluate whether the provided correctly responds to the question based exclusively the given video content.
Video content: <VIDEO <VIDEO_CONTEEEEEE CONTENT>.
Here the question: <QUESTION>.
And here the answer to be evaluated: <REFERENCE ANSWER>.
should carefully examine the video content. Fully understand the question and and make judgment.
Judgment Criteria:
Correct: Answer fully consistent with video content and completely addresses the question
Incorrect: Answer contradicts video content, adds unsupported information, or fails address the question.
You should answer with Correct/Incorrect first, and Provide a brief explanation here.
Figure 20: During QA refinement, the generated questions are submitted to GPT-4 for blind answer- ing using the Blind Answer Prompt. Subsequently, the QA pairs and blind answers are evaluated with the Filtering Prompt, which filters out questions answerable using text-only input. Addition- ally, GPT-4o tasked with verifying the correctness the QAs based on the original video content. Following manual selection, we obtain 1,177 final QAs and 120 descriptive items. While the struction of open- open-ended items has been completed, distractor generation for multiple-choice QAs requires additional step.
27

---

Published as a conference paper at ICLR 2026
Distractor Prompt
Your task is to generate three different seemingly reasonable interference answers based on the form of the given question and the correct
First, please carefully read the following question: <QUESTION>.
And here is the correct answer: <REFERENCE ANSWER>.
When generating the interference answers, please the following guidelines:
The interference answers should be in similar form or context as the correct answer.
They should seem reasonable at first glance, but not be the correct solution to the question.
Avoid using completely off topic obviously wrong
Please output your response in the following JSON format with English: "interference answer1', 'interference answer2', 'interference answer3'
For MCQAs, we input the verified question-answer pairs into GPT-4 and use a spe- cific prompt to generate three distractors. This step critical for creating high-quality, challenging multiple-choice questions, as low low-quality distractors may lead to information leakage, for instance, enabling model to guess the correct answer easily by elimination.
Figure 21:
Task Definition
"Object Recognition ": "Briefly describe the scene and specify the location of an object, and then ask question about the object(s) is(are).",
"Attribute Recognition "A question that briefly describe the scene and specify an object, and then ask to identify property of the object such color, shape, material, etc.",
"Object Counting": "A question that briefly describe the scene and specify a certain type of object/action, and then ask to identify the number of objects/actions",
"Relationship Recognition": : question that briefly describe the scene and specify several objects, and then ask to identify the relationship between them.",
"OCR": "A question that briefly describe the scene and specify an area, and then ask to identify the text content that area. can be subtitles any other text that appears in the video",
"Action Recognition "A question that briefly describe the scene and specify an object(human) and then ask about the action of the object(human).",
Task definitions for six video understanding tasks. Additionally, descriptive questions are formulated as instructions that prompt the model to generate detailed descriptions of the video segments specified by the scene mentions.
Figure 22:
28

---

Published as a conference paper at ICLR 2026
Q: In the scene featuring a red, curved blood vessel with a smooth inner surface, what the numerous red, disc-shaped objects flowing inside the vessel? Red blood cells.
Common Sense
the living room, the first-person places their white mug on coffee table that has yellow tray holding items such as two oranges and o pack labeled Viennetta'. Whot color is the tray? Yellow.
Information Leakage
Q: In the scene where a black SUV is parked on a residential street with trees and houses the background, what does the young girl do? A: Walk towards the car, open the front door, and reach for the door handle.
Hallucination
Figure 23: Three representative examples of filtered question-answer pairs are presented. The first example illustrates question answerable through commonsense knowledge. The second exempli- fies answer leakage originating from the scene description. The third demonstrates a hallucinated response, potentially caused by inaccuracies frame descriptions generated in Step
Evaluation Prompt for Open-ended Response
You are required to evaluate the performance of the respondent the video description task based on the standard answer and the respondent's answer. Provide single integer score from to 100 that reflects both:
COMPLETENESS how well the summary covers all key points and main content the video. CORRECTIESS the factual accuracy, clarity, and absence of contradictions in the summary.
Here the standard answer: <ANNOTATION>.
Here the respondent's answer: <MODEL RESPONSE>.
Reasonable additional details that do not conflict with the reference answer should not reduce the score. Return the integer score (0 to 100) with brief explanation. Please output the following JSON format: ["Score": xx, "explanation": "xxx").
Figure 24: We use GPT-4 to evaluate model generated open-ended outputs by comparing them against annotations.
29


Reviews:
[{"review_id": "gHBuTrxqd8", "reviewer": "Reviewer_3Lzq", "summary": "This paper introduces Video-LevelGauge, a benchmark for evaluating contextual positional bias in LVLMs by inserting standardized probes into different positions of video contexts. The benchmark systematically assesses 27 LVLMs, revealing that commercial models exhibit less positional bias than open-source ones. The study also explores the effects of context type, length, and model size on positional bias, providing insights for future improvements.", "questions": "Please see Weaknesses for details.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Novel Contribution: Highlights contextual positional bias, an underexplored issue in LVLMs.\n- Well-Designed Benchmark: Includes standardized probes, flexible context configurations, and comprehensive metrics.\n- Comprehensive Evaluation: Thorough analysis of 27 LVLMs, revealing actionable insights on bias patterns.", "weaknesses": "- No Mitigation Strategies: While the paper introduces contextual positional bias, it does not propose or evaluate methods to mitigate it.\n- Lack of Task-Specific Examples: The paper does not provide clear illustrations of how positional bias affects specific tasks, making the findings less interpretable.\n- Narrow Scope: Focuses solely on positional bias without addressing other LVLM limitations, such as hallucination or temporal reasoning errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EHapERVH31", "reviewer": "Reviewer_ZbYf", "summary": "This paper introduces Video-LevelGauge, a benchmark exploring positional bias in large video language models (LVLMs). The benchmark covers a range of models for comprehensive evaluation and reveals that existing open-sourced ones suffer significant positional bias, exhibiting unstabe video understanding affected by the position of target content. The paper also provides a deep analysis on models’ behavior across divers perspectives, such as context length and model scales.", "questions": "See the Weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles an important and underexplored problem in video understanding — positional bias in LVLMs. Overall, the paper is well written and easy to follow.\n- The benchmark is well structured, featuring diverse videos and mechanisms that prevent single-frame or blind-bias predictions. The positional bias metric is clearly defined and suitable for evaluation.\n- The paper provides extensive experimental results across 27 leading LVLMs with in-depth analyses, offering valuable insights into model behavior.", "weaknesses": "- While the benchmark design and evaluation criteria are solid, the results and analyses feel somewhat lukewarm and unsurprising. In Section 4.2, the main conclusion that positional bias tendencies vary across models and depend largely on training methods and model scale is expected. It is natural that larger models exposed to diverse video data perform better, and this conclusion could likely apply to other evaluation criteria beyond positional bias. I would appreciate a deeper, more tailored analysis of *why* such biases arise and *what* model characteristics contribute to distinct bias patterns (e.g., MR types). For instance, why do MiniGPT4-Video and InternVL3 (8B) show head preference? Are their training datasets skewed toward early-frame cues? \n- Consequently, it seems somewhat trivial that longer videos are more challenging and lead to lower performance. However, it remains unclear whether this truly amplifies positional bias or if the observed drop is simply due to increased video complexity.\n- The paper primarily identifies the problem of positional bias but does not propose or discuss concrete directions for mitigation. Including such discussion would make the contribution more complete and valuable for future work.\n- (Minor) Discussing inconsistent and biased video understanding found in prior works [1, 2] could also strengthen this analysis.\n\n**References** \n\n[1] A Closer Look at Temporal Sentence Grounding in Videos: Dataset and Metric, arxiv 2021\n\n[2] On the Consistency of Video Large Language Models in Temporal Comprehension, CVPR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ahh9IJRD2a", "reviewer": "Reviewer_TpSR", "summary": "This paper introduces a benchmark suite for evaluating contextual positional bias of large video language models (LVLMs) considering long contexts. The data collection involves three stages,\n1. QA Generation: They collect videos from existing video-text resources including ego-centric videos, and generate frame-wise captions using GPT-4o. These captions are combined with crowd-sourced task definitions, and then an LLM generates question-answer pairs.\n2. QA Refinement: The generated question-answer pairs are filtered by GPT-4o, considering hallucinations and cases where the question does not require visual information in order to be answered. Human validators filter out the invalid question-answer pairs before proceeding to next step.\n3. Distractors: LLMs generate distractors (i.e., incorrect answer choices) for the collected question-answer pairs, later, again refined by human validators.\n\nAdditionally, the benchmark is divided into 7 sub-categories, which are OCR, Attribute Perception, Object Reasoning, Count Problem, Relationship Recognition, Action Reasoning, and Instructed Description. The benchmarked models are tested under different conditions, which are denoted as customized contexts. The set of customized contexts within this work include multiple videos inputs, long videos, multimodal interleaved input and lastly template video with ImageNet mean pixel values. This work introduces 3 different metrics to measure the positional bias, where all 3 metrics are centered around the relative score (RS) measure. These 3 metrics are, average relative score ( $P_{mean}$ ), difference between maximum and minimum relative score ( $P_{ran}$ ), and the variance in relative scores ( $P_{var}$ ). The evaluations on the proposed benchmark include 27 LVLMs including both open-weight and proprietary models where the model scale is ranging up to 108 billion parameters. Further studies investigates the effect of context type, context length and model size on positional bias.", "questions": "- I think it would be good to show visual examples of customized contexts in the appendix part.\n- There is some typo on Fig2: GTP-4o-latest should be GPT-4o-latest.\n- There is also white text on the last figure (Fig. 22) which can be barely seen on the background: `\"Please output the questions and reference answers in the following JSON format: [ {'question': 'xxx', 'answer': 'xxx’}, {'question': 'xxx', 'answer': 'xxx’},\"`", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- A novel benchmark for assessing the positional bias in video-language models.\n- Focuses an important aspect which could help to assess robustness of LVLMs under different positional settings, which could help further developing more trusthworthy LVLMs.\n- Detailed experimentation: 27 models, further beneficial studies on the effect of different design choices.", "weaknesses": "- The presentation needs to be improved,\n\t- In Figure 3, the entire set of sub-tasks should be illustrated. This is not feasible.\n\t- In Figure 3, the tasks should be renamed by following the terminology already exist in the literature. I don't understand why someone should pose these tasks as *reasoning* tasks, where they are *recognition* tasks in fact. For instance, please see [Fig 1](https://arxiv.org/pdf/2306.13394) in MMU work to see the difference between reasoning and recognition type of tasks. So, the tasks should be renamed as,\n\t\t- Object Reasoning -> Object Recognition\n\t\t- Count Problem -> Object Counting (because actions can be counted also as well)\n\t\t- Action Reasoning -> Action Recognition\n\t\t- Attribute Perception -> Attribute Recognition (for consistency)\n\t- In Fig1(a), what do the numbers next to the tick and x marks?\n\t- Fig 2 is cluttered, I think it would be better if the only metric is positional variance.\n- Human refinement process remains entirely opaque. There are no details on this process, even how many human validators participated in the refinement process.\n- The proposed metric is not a standard metric used to evaluate the models, so I think this part should be expanded in the main text. This is such an important element of this paper but it only takes up 20 lines in the main text currently.\n\t- For instance, why should one use the relative score metric, and why should the standalone accuracy should be in the denominator?\n\t- What is the position $i$ ? What is the unit, seconds, or frame? Is this position absolute or relative? Are these positions randomly sampled for each individual example? Do these positions guarantee that the question-answer pairs remain valid for the video with custom context?\n\t- The morphological recognition (MR) term is confusing because morphology is a term which exists in NLP literature. Additionally, MR term currently seems unclear in the main text, and it is not motivated well enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vbFvcCV63C", "reviewer": "Reviewer_sdzE", "summary": "The authors address the problem of **contextual positional biases** in large video-language models (LVLMs), where the content of a clip is interpreted inconsistently depending on its position within a video. While this issue has been explored in language models, it has not been studied in multimodal video-language settings, making this work both unique and novel. The paper introduces **Video-LevelGauge**, a benchmark designed to evaluate contextual positional biases in LVLMs across diverse tasks and video types. Beyond standard accuracy, it proposes novel **statistical metrics** to quantify such biases. Using these metrics, the authors further characterize **morphological patterns** in LVLMs, enabling the identification of specific phenomena underlying positional bias. Extensive experiments and analysis conducted with Video-LevelGauge provide valuable insights to guide future LVLM development and contextual positional bias mitigation. Overall, the paper is well-written and presents notable contributions to a previously underexplored area of LVLM research.", "questions": "- “$\\nearrow \\text{if MSE}_1 \\leq 3 \\text{ and }k > 0.5$” [Appendix A.5] - Could the authors elaborate what the neighbor bias entails, and how the trend reflected by this condition correlates with this bias?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Video-LevelGauge incorporates a comprehensive coverage of six video-language tasks spanning diverse video types, including egocentric, media, and synthetic videos, enabling extensive evaluation of existing LVLMs for contextual positional biases across a multitude of settings.\n- Video-LevelGauge introduces three statistical metrics: $P_{mean}$, which captures the magnitude of positional bias, and $P_{ran}$ and $P_{var}$, which measure the volatility of model behavior. Together, these metrics provide a holistic assessment of positional biases in LVLMs.\n- The proposed morphological recognition provides a grounded approach to identifying potential root causes of contextual positional biases, providing a framework to diagnose and characterize positional biases in LVLMs.\n- Thorough analysis is performed on Video-LevelGauge using the proposed metrics across multiple factors, including context length and video type, revealing several insightful observations that could inform future efforts toward mitigating these biases.", "weaknesses": "- For the positional bias metric, the relative score is intuitive and easy to understand. However, greater details could be provided on what the score entails: How is $RS_i$ computed if $S_{meta} = 0$, and what does $RS_i$ reflect in this case? Additionally, for the five types of morphological patterns, several categories are not straightforward to understand. In particular, the morphological phenomena of “lost in the middle”, “neighbor bias”, and “volatile” could be somewhat confusing. I would suggest for the authors to perhaps provide additional elaboration for each of these types in Section 3.4, and possibly how they correlate with model performance.\n- Although the authors validated against multimodal information leakage within each probe instance, the paper does not seem to address potential leakage between the probe clip and background video(s) during evaluation. Background videos with similar contexts to the probe clip could lead to cross-video leakage, allowing the LVLM to answer the query correctly by referring to background video content, which may be unintended.\n- Evaluation results for individual tasks (e.g., OCR, AP, etc.) only report $P_{ran}$, which measures the worst-case variation, but omit $P_{mean}$, which captures the average performance across instances. Including $P_{mean}$ would provide a clearer view of the overall positional bias for each task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)