VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Authors: Xilin Wei
Xiaoran Liu
Yuhang Zang
Xiaoyi Dong
Pan Zhang
Yuhang Cao
Jian Tong
Haodong Duan
Qipeng Guo
Jiaqi Wang
Xipeng Qiu
Dahua Lin
Venue: ICML 2025 oral (2025)
Keywords: Rotary Position Embedding (RoPE)
Spatio-temporal Encoding
VideoRoPE
V-NIAH-D Task
Temporal Dimension Allocation
3D Position Embedding
Low-frequency Temporal Allocation
Diagonal Layout
Adjustable Temporal Spacing
Video Retrieval
Video Understanding
Video Hallucination
Position Encoding for Video
Distractor Handling in RoPE
Long-context Modeling
Primary area: applications->computer_vision
TL;DR: This paper identifies four key criteria for positional encoding: structure, frequency allocation, spatial symmetry, and temporal scaling. We propose VideoRoPE, which outperforms prior methods in video retrieval and understanding.

Abstract: While Rotary Position Embedding (RoPE) and its variants are widely adopted for their long-context capabilities, the extension of the 1D RoPE to video, with its complex spatio-temporal structure, remains an open challenge.
This work first introduces a comprehensive analysis that identifies four key characteristics essential for the effective adaptation of RoPE to video, which have not been fully considered in prior work.
As part of our analysis, we introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors) task, which adds periodic distractors into V-NIAH.
The V-NIAH-D task demonstrates that previous RoPE variants, lacking appropriate temporal dimension allocation, are easily misled by distractors.
Based on our analysis, we introduce VideoRoPE, with a 3D structure designed to preserve spatio-temporal relationships.
VideoRoPE features low-frequency temporal allocation to mitigate periodic oscillations, a diagonal layout to maintain spatial symmetry, and adjustable temporal spacing to decouple temporal and spatial indexing.
VideoRoPE consistently surpasses previous RoPE variants, across diverse downstream tasks such as long video retrieval, video understanding, and video hallucination.
Our code and model weights will be publicly released.

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Xilin Wei 12 Xiaoran liu 122
Yuhang Zang²
Xiaoyi Dong? 24 Pan Zhang² Yuhang Cao
Jian Tong"
Haodong Duan
Qipeng Guo
Jiaqi Wang? Xipeng Qiu 123
Dahua Lin 245
Abstract
Table I. Comparison between different RoPE variants for Video
While Rotary Position Embedding (RoPE) and its variants widely adopted for their long-context capabilities, the extension of the ID RoPE to video, with its complex spatio-temporal structure, remains an open challenge. This work first in- troduces a comprehensive analysis that identifies four key characteristics essential for the effective adaptation of RoPE video, which have not been fully considered in prior work. As part of our anal- ysis, introduce a challenging V-NIAH-D (Vi- sual Needle-In-A-Haaeeee In-A--aystack with Distractors) task, which adds periodic distractors into V-NIAH. The V-NIAH-D task demonstrates that previous RoPE variants, lacking appropriate temporal dimension allocation, are easily misled by distractors. Based our analysis, we introduce VideoRoPE, with a 3D structure designed preserve spatio-temporal relationships. VideoRoPE features low-frequency temporal allocation to mitigate periodic oscil- lations, a diagonal layout to maintain spatial symmetry, and adjustable temporal spacing to decouple temporal and spatial indexing. Vide- oRoPE consistently surpasses previous RoPE vari- ants, across diverse downstream tasks such long video retrieval, video understanding, and video hallucination. Our code is available at http:///ttttb..oommmsslllttttttt
Large Language Models (Video LLMs).
20/3 Structure
Frequency Allocation
Spatial Symmetty
Temporal Index Scaling
Vanilla RoPE (Sa 2024)
2024)
RoPa-Tle (Su, 20144)
x
x
M RoPE (Wang al., 2024a)
VideoRoPE (Ours)
V
VideoMME: 61.33
MLVU
LongVideoBench
S1.26
V-NIAHED 87.11
57.00 VideoHHaluueer (Spatial)
91.11
V-NIII
VideoHallucer (Temporal)
20
VideoHallucer
Vanilla RoPE
TAD-RoPE
M-RoPE
VideoRoPE(Ours)
Figure 1. VideoRoPE outperforms RoPE variants on benchmarks.
context understanding (Ding al., 2024b), and continues to be default choice in leading Large Language Models (LLMs) like the LLaMA (Touvron al., 2023a;b; Dubey et al., 2024) and QWen (Yang et al., 2024a;b) series.
The original RoPE implementation (Vanilla RoPH) (Su et al., 2024) is designed for sequential ID data like text. However, recent Video Large Language Models (Video LLMs) (Li et al., 2023; Lin et al., 2023a; Chen al., 2024a; Maaz et al., 2024b; Zhang al., 2024d; Wang et al., 2024c; Chen et al., 2024b; Zhang et al., 2024b) process video, which has more complex spatio and temporal structure. As shown in Tab. 1, although several RoPF-baaed approaches (Gao et al., 2024; Wang et al., 2024a) have been proposed to support video inputs, these variants exhibit limitations and do not fully satisfy the following key characteristies:
1. Introduction
Rotary Position Embedding (RoPE) (Su et al., 2024) helps Transformer models understand word order by assigning each token a unique positional 'marker" calculated using a mathematical rotation matrix. RoPE has advantages in long-
"Equal contribution 'Fudan University, Shanghai, China "Shanghai AI Laboratory, Shanghai, China "Shanghai Inno- vation Institute, Shanghai, China "The Chinese University of Hong Kong *CPII under InnoHK. Correspondence to: Yuhang Zang <zangyuhang pjlabborg.cn. en>> Qipeng Guo <guo- qipeng @ pjlab.org.cn. arg.cn>>,.........o....... Jiaqi Wang < <wangjiaqie pjlab.org.cn>.
(1) 2D/3D Structure. Some existing Video LLMs direct flatten the video frame into ID embeddings and apply the ID structure RoPE (Su et al., 2024; Gao et al., 2024). These solutions fail to capture video data's inherent 2D or 3D (temporal (t), horizontal (x), and vertical (1)) structure, thus hindering explicit spatial and temporal representation.
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

VidcoRoPE: What Makes for Good Video Rotary Position Embedding?
(a) V-NIAH A
0.0
1.0
(Visual-Needil Seall--eeeee-----e In-A- --Hayttack
Find the frame of couple wedding, what is the color of balloon on the bridegroom's 's head?
Question:
M-RoPE
Video RoPE (Ours)
Answer: Yellow
Needle
V-NIAH-D OOurs)
M-RoPE
Video RoPE (Ours)
Distractor
Needle
Distractor
Figure 2. Left:
To demonstrate the importance of frequential allocation, based on VIAH (a) we present a more challenging V-NIAH-D task (b) that similar images are inserted distractors. Right: Compared to M-RoPE, our VidcoRoPE is more robust in retricval and less affected by distractors. See Fig. 7 in the Experiments section for details on the horizontal and vertical axcs.
(2) Frequency Allocation. Previous approaches such as M- RoPE used in QWen2-VI. (Wang et al., 2024a) employ 3D structure, dividing feature dimensions into distinct subsets for (t, x, y) encoding, respectively. How to determine the optimal allocation of these dimension subsets, and their associated frequencies! are not well studied. Some previous work allocates the lower dimensions corresponding to the high frequency represent the L. However, the temporal dimension t is significantly tortured by periodic oscillation, and distant positions may have the same embeddings.
sions often exhibit different granularities (e.g., unit change in s/y differs from unit change in () (Gao et al., 2024), Employing varying index intervals in positional encoding allows for dimension-specific encoding, capturing diverse scales and enhancing efficiency.
Driven by our analysis, we present a new video position em- bedding strategy, VideoRoPE, which can simultaneously satisfy the four properties in Tab. 1. Specifically, 3D structure to model spatiotemporal information, allo- cating higher dimensions (lower frequencies), to the tem- poral axis (Low-frequency Temporal Allocation, LTA) to prioritize temporal modeling. The right panel of Fig. demonstrates that our LTA allocation mitigates oscillations and exhibits robustness distractors in the V-NIAH-D task. We further employ Diagonal Layout (DL) design en- sure spatial symmetry and preserve the relative position- ing between visual and text tokens. Regarding temporal index scaling, we propose Adjustable Temporal Spacing (ATS), where a hyper-parameter controls the relative tem- poral spacing of adjacent visual tokens. In summary, our proposed position encoding scheme demonstrates favorable characteristics for modeling video data, yielding robust and effective representation of positional information.
We present a simple setting to verify this point. Based on the previous long-video retrieval task V-NIAH (Visual Needle Inclee-I--A-Hlee In-A-Haystack) (Zhang et al., 2024d), we insert sev- eral similar images that do not affect the question's answer before and after the needle image distractor (Hsieh al., 2024; Yuan et al., 2024), forming new task, V-NIAH-D (Visual Needle-In-A A-aasstakkc with Distractors). As shown in Fig. 2, we find that previous M-RoPE is misled by dis- tractors, showing a significant performance decline from V-NIAH to V-NIAH-D. Our observation demonstrates that the periodic oscillation reduces Video LLMs' robustness.
(3) Spatial Symmetry. The distance between the end of the precedent textual input and the start of visual input equals the distance between the end of visual input and the start of subsequent textual input (Su, 2024b). Such a symmetry ensures that the visual input receives equal contextual influence from both the preceding and subsequent textual information.
Overall, the contributions of this work are summarized as:
(1) We present an analysis of four key properties essential for RoPE when applied to video. Motivated by this analysis, we propose VideoRoPE including Low-frequency Temporal Allocation (LTA), Diagonal Layout (DL), and Adjustable Temporal Spacing (ATS) to satisfy all four properties.
(4) Temporal Index Scaling, Spatial and temporal dimen-
'In RoPE, frequencies are determined by 9 g-2n/d where 3 a constant, n the dimension index, d is the total number of dimensions. Thus, choosing which dimensions represent t, x, and y directly determines the frequencies used for each.
(2) We introduce the challenging V-NIAH-D task to expose the drawbacks of current position embedding designs re- garding frequency allocation. We reveal that existing Video
2

---

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
LLMs are easily misled to frequency-based distractors.
where Al (1 - l2, the symbols It, and Kig are the query and key vectors at positions i and 1z. The relative rota- tion matrix R& is defined as Rs = exp(Atit,,, while i is the imaginary unit, 8, = B 3-22/d is the frequency of rotation applied to a specific n-th pair of dimensions (n = O..... d/2 - 1), and | is the frequency base param- eter. The vanilla RoPE uses Consequently, the An, 12 in Eq. (1) can be extended as:
(3) Extensive experiments demonstrate that VideoRoPE consistently achieves superior performance compared to other RoPE variants. For example, VideoRoPE outper- forms previous M-RoPE on long video retrieval (+12.4 on V-NIAH, + +12.4 on V-NIAH-D), video understanding (+2.9 on Long Vong +4.5 on MLVU,+14 +1.7 on Video-MME) and hallucination (+11.9 on VideoHallucer) benchmarks.
128, thus n = 0, O,..., 63.
2. Related Work
case Ar
win 8,44 cus 1
sin fucct
RoPE (Rotary Position Embedding). RoPE (Su et al., 2024) is pivotal mechanism for encoding positional infor- mation in LLM long-context modeling. Using a rotation matrix, RoPE unifies the advantages of both absolute and relative positional embedding schemes. In RoPE design, different feature dimensions are embedded with position information based on Trigonometrie functions sin and cos with different frequencies (Peng al., 2023; Liu et al., 2023b). Lower dimensions correspond to higher frequency given larger values of base frequency. The simplicity and effectiveness of RoPE have led to its widespread adoption in leading LLMS (Touvron et al., 2023a; Yang et al., 2024a; Team et al., 2024; Cai et al., 2024; Sun al., 2024).
4
((3RT)
(2)
While the vanilla RoPE operates on ID sequences, it can also be applied to higher-dimensional input by flattening the input into a I-D sequence. However, the flattening process discards crucial neighborhood information, increases the sequence length, and hinders the capture of long-range de- pendencies. Therefore, preserving the inherent 3D structure is essential when adapting RoPE for video data. Some recent RoPE-variants (c.g., M-RoPE in Qwen2-VL. (Wang et al., 2024a)) incorporate the 3D structure. The corresponding relative matrix A ((1..1.81) is computed as:
Extending RoPE to Multi-Modal Data. Extending RoPE to multi-modal or Video LLMs typically follows two ap- proaches. One approach directly applies standard RoPE, flattening visual tokens and treating text and visual tokens as a single ID sequence. Although variants (e.g.. TAD-RoPE (Gao et al., 2024)) introduce enhancements in indexing and attention mechanisms, these ID RoPE variants overlook the spatiotemporal structure of video and inherent inter-modal differences (Su, 2024a;b; Wang et al., 2024a). In contrast, several studies have explored incorporating structural infor- mation to formulate the 2D/3D RoPE For example, some previous works (Agrawal et al., 2024; Wang et al., 2024a) integrate RoPE-2D into visual encoders to improve spatial representation, particularly for resolution scaling. Based on the RoPE-Tie (Su, 2024a), M-RoPE (Wang et al., 2024a) used in QWen2-VL further generalizes RoPE to three dimen- sions to model both temporal and spatial dynamics. While effective, M-RoPE exhibits limitations, such as struggles with distractors in our V-NIAH-D task. This work presents a comprehensive analysis of the important characteristies essential for extending RoPE to video and proposes Vide- oRoPE according to our analysis.
A A(t.21::))) ((11.11m)) RACAAE (12.22233) (3)
where At = t1 - t2, Ax 11 I2, and Ay = - y22 M-RoPE divides the d = 128 feature dimensions into 3 groups: the first 32 for temporal positions (f), the middle 48 for horizontal positions (a), and the last 48 for vertical positions (y). As shown in Eq (4), M-RoPE is extended as:
in
Jous Hodi siu 9141 Gudt
<in Aiy
1000
sin Pr cas Pas
madeling temporal dependency with higher frequency
yo an
Fros Pua <in /ua styiycc moutee/hh
oos Pi
cce dap
modeling borizontal dependency with intermediate frequency
(a3)
foom Mandy
siis Panday sin duy con Surd
cos Raz des Pag Ay
3. Analysis
Der Bacca Byy cus Resibg
modeling vertical dependency with lower frequency
3D Structure. The vanilla RoPE defines a matrix A that represents the relative positional encoding between two positions f and to in a 1D sequence:
(4)
Frequency Allocation. Incorporating 3D structure raises the question of how to allocate the temporal (t), horizontal (æ), and vertical (y) components within the dimensions. Note that different allocation strategies are not equivalent
A = (q:, Re,) (kr, Rox) = qr, RAcK
(1)
3

---

VidcoRoPE:
What Makes for Good Video Rotary Position Embedding?
haystack
needle
haystack
located primarily through vertical positional information, rather than temporal features. Thus, the temporal dimension fails to capture long-range semantic dependencies, focusing on local relationships. Conversely, the spatial dimensions capture long-range rather than local semantic information. Lastly, the horizontal and vertical dimensions display dis- tinet characteristics, with the vertical dimension exhibiting phenomena reminiscent of attention sinks (Xiao et al., 2023). These suggest the performance decline primarily results from sub-optimal frequency allocation designs of M-RoPE.
Question: what is being transferred to the heaker in the laboratory? A. Solid substance Gas C. Nothing
D. Liquid tester
M- RrRoPE: Solid substance
Video RoPE; D. Liquid tester
M-RoPE-t
M-RoPE-X
M-RoPE- -y
10
20
30
Spatial Symmetry. Given the text tokens I and the visual tokens T,, spatial symmetry (Su, 2024b) claims that the distance between the end of the preceding textual input (Tuee and the beginning of the visual input (Tgart equal to the distance between the end of the visual input (Igml) and the beginning of the subsequent textual input (Tsub):
40
50
VideoRoPE- -t
VideoRoPE -
VideoRoPE -y
10
20
30
40
50
T
Tpr = Tsub - Tend
(5)
30
10
40
20
30
50
The spatial symmetrical structure can potentially simplify the learning process and reduce bias toward input order. However, existing 3D RoPE variants such as M-RoPE do not meet the spatial symmetry, we will elaborate related discussion in Fig. 6.
Figure 3. intention-based frequential allocation analysis. Middle: M-RoPE's temporal dimension (t) is limited to local information, resulting in diagonal layout. Bottom: VideoRoPE effectively retrieves the needle using the temporal dimension. The x and y coordinates represent the video frame number, c.g., 50 for 50 frames. For more details see Appendix E.
Temporal Index Scaling. The frame index in video and the token index in text are inherently different (Su, 2024b; Li et al., 2024a). Recognizing this difference, methods like TAD-RoPE, a 1D RoPE adaptation for Video LLMs, intro- duce distinct step offsets for image and text token indices: for image tokens and y + for text tokens. Consequently, an ideal RoPE design for video data should permit scaling of the temporal index to meet the inherent difference between the frame index and the text index.
in the rotation frequency 9n = 3 33/2//d As shown in Eq. (4), M-RoPE assigns higher frequencies (corresponding to lower dimensions) to the temporal dimension (!).
To highlight the importance of frequency allocation, we introduce a challenging retrieval task Visual Needle-In-A- Hastack-Distractor (V-NIAH-D). V-NIAH-D builds upon V-NIAH (Zhang al., 2024d), benchmark designed to evaluate visual long-context understanding. However, the straightforward retrieval- based task has been shown to pro- vide only a superficial form of long-context understand- ing (Hsich et al., 2024; Yuan et al., 2024). Therefore, We enhance V-NIAH by incorporating semantically similar distractors, obtained using Google Image Search (Google, 2025) Flux (Labs, 2023), to mitigate the possibility of correct answers through random chance. These distractors are designed to be unambiguous to the question in Fig. 2.
4. VideoRoPE
Based on some previous research and the above analysis, we claim that a good RoPE design for Video LLMs, especially for long videos, should satisfy four requirements. The first requirement has been solved by RoPE-Tie (Su, 2024a) and the subsequent M-RoPE (Wang al., 2024a). To solve the last three requirements and mitigate the performance decline observed in V-NIAH-D, we propose our VideoRoPE, comprising the following three key components.
As shown in Fig. 2, M-RoPE exhibits a clear performance drop from V-NIAH to V-NIAH-D. To investigate this de- cline, we follow previous works (Xiao et al., 2023; Liu et al., 2023b; Barbero et al.. 2024) to visualize the attention scores in Fig. 3. We decompose the attention scores into their corresponding temporal (4), horizontal (r), and vertical (v) components for visualization.
now-frequency Temporal Allocation (LTA). As shown in Eq. (2), the vanilla RoPE (Su al.. 2024) uses all di- mensions model the ID position information. And as indicated in Eq. (4), M-RoPE (Wang et al., 2024a) uses different dimensions to model temporal, horizontal, and ver- tical dimensions sequentially. However, previous frequency allocation strategies are suboptimal because different RoPE dimensions capture dependencies at varying ranges. As shown in Fig. 3, an interesting observation is that the local
Fig. 3 reveals unusual M-RoPE's attention patterns, despite locating the needle image, it fails to answer the multi-choice question. According to M-RoPE's attention, the needle is
4

---

VidcoRoPE: What Makes for Good Video Rotary Position Embedding?
00
0.
65
0.5
1.0
-00
200
Dim Index
16
2030
Dim Indox
52
Token Indez
32e
44
Token Index
10200
15020 20003
500
(a) Temporal Frequency Allocation in M-RoPE
(b) Temporal Frequency Allocation VideoRoPE (ours)
Figure (a) M-RoPE (Wang et al., 2024a) models temporal dependencies using the first 16 rotary angles, which exhibit higher frequencies and more pronounced oscillations. (b) In contrast, VideoRoPE models temporal dependencies using the last 16 rotary angles, characterized by significantly wider, monotonic intervals. Our frequency allocation effectively mitigates the mislcading influence of distractors in V-NIAH-D. For more detailed analysis, please refer to Appendix F.
attention branch (as reported in (Han et al., 2024)) corre- sponds to lower dimensions, while the global branch (or attention sink, in (Xiao et al., 2023)) corresponds to higher dimensions. To sum up, lower dimensions (higher frequency, shorter monotonic intervals, larger 6,,) tend to capture relative distances and local semantics (Men et al., 2024; Barbero et al., 2024), while higher dimensions (lower frequency, wider monotonic intervals, smaller 0,)) capture longer-range dependencies (Barbero et al., 2024).
OlL this text.
Coeriie mndes the RUPE text
Tdimmnstoon
Poetion index M-Roott video
dimeniioo
dimmnnsion
Based on analysis, VideoRoPE uses higher dimensions for temporal features in longer contexts and lower dimen- sions for spatial features, which are limited by resolution and have a fixed range. To avoid the gap between horizontal and vertical positions, we interleave the dimensions respon- sible for these spatial features. The dimension distribution for VideoRoPE is shown in Eq. (6):
Figure S. The position embeddings of adjacent text tokens for Vanilla RoPE (top row), the corresponding visual tokens in ad- jacent frames for M-RoPE (middle row) and our VideoRoPE (bottom row) with interleaved spatial and temporal last design.
leaved to occupy the lower dimensions, followed by tempo- ral I, which occupies the higher dimensions. We keep the same allocation number for x, y. and t as M-RoPE for fair comparison, with values of 48, 48, and 32, respectively.
(000 grory
Foos Pudd sain &i sin At
The advantages of this distribution are evident in Fig. 4. For a RoPF-bbased LLM with a 128-dimensional head (64 rotary angles 0,),, we visualize the function of cos 0,, for dimensions using parallel blue planes.
mochgchh 0
cos Rio sin Ap
sin Po
cos Pi
coefee eln Ray sin 22
coe 9as
modeling temporal dependency with lower frequency
As shown in Fig. 4 (a), M-RoPE's temporal position em- beddings are significantly distorted by periodic oscillations (Men et al., 2024), leading to identical embeddings for dis- tant positions. For instance, considering the last three rotary angles, the temporal embeddings are severely affected by these oscillations due to their short monotonic intervals (and even shorter intervals in lower dimensions). This pe- riodicity creates "hash collisions" (red planes), where dis- tant positions share near-identical embeddings, making the model susceptible to distractor influence. Fortunately, our VideoRoPE (Fig. 4 (b)) is free from oscillation and Hash collision in temporal modeling. The relationship between periodicity, monotonicity, and temporal modeling is visual- ized in Fig
fenm Au sin BuAr ene d.Ax 0
sin Arr
oos 9g Ax
sio du Byckem
(10) 1900
sis Pug
modeling borizontal dependency with interleaved high frequency
/oowlfy By Ay - win B,y sin 8i 2gg oos by 2a
6
Mao
cow B,Ayy
0y Ag
Am
(90 100
Pay Aey
sin Pur Rayidyy
modeling vertical dependency with interleaved high frequency
(6)
The horizontal position x and vertical position y are inter-
5

---

VidcoRoPE: What Makes for Good Video Rotary Position Embedding?
Taxis
axxs
axis
40
15.0
30
12.5
10.0
20
10
1.5
10
0
0.0
40
20
15.0
20
y yais
30
15
0.0
2.5 5.0 12.5 12.
7.5
12.5
axis
10
10
5.0
X axis
30
40
15
20
0.0
(a) 3D visualization for Vanilla RoPE.
(b) 3D visualization for M-RoPE.
(c) 3D visualization for VideoRoPE.
Figure 6. The 3D visualization for different position embedding. (a) The vanilla ID RoPE (Su et al., 2024) does not incorporate spatial modeling. (b) M-RoPE (Wang et al., 2024a). while have the 3D structure, introduces discrepancy in index growth for visual tokens across frames, with some indices remaining constant. (c) In contrast, our VideoRoPE. achieves the desired halance, maintaining the consistent index growth pattern of vanilla RoPE while simultaneously incorporating spatial modeling.
Diagonal Layout. Fig. provides a visual comparison of spatial symmetry in positional encodings. For vanilla RoPE (Fig. 6a), no spatial relation is considered and the index for every dimension increases directly. While M-RoPE (Fig. 6b), incorporates spatial information within each frame, it introduces two significant discontinuities between textual and visual tokens. This arises from M-RoPE's placement strategy, if the first visual token is at (0, 0), the last token in each frame will always be placed at (W - 1, - 1), creating a stack in the bottom-left corner. Furthermore, like vanilla ROPE, M-RoPE's indices increase with input length across all dimensions.
According to our adjustable temporal spacing design, for multi-modal input that consists of a text with T, tokens, a following video with T. frame with W x H patches in each frame, and an ending text with T. tokens, the position indices (t, x. y) of VidcoRoPE for 7-th textual token or (T, w. h)-th visual token defined as Eq. (7):
(T,
To 6(T T.). T. + TT) +
(t,,,)
T. + 6(r- T.)
+ (8 -IYT.
To address these limitations, VideoRoPE arranges the entire input along the diagonal, see Fig. 6c. The central patch's 's 3D position for each video frame is (t, t), with other patches offset in all directions. Our Diagonal Layout has two ad- vantages: (I) our design preserves the relative positions of visual tokens and ensures approximate equidistance from the image corners to the center, preventing text tokens from being overly close any corner. (2) It maintains the in- dexing pattern of vanilla RoPE (Fig. 5), as the position index increment between corresponding spatial locations in adjacent frames mirrors that of adjacent textual tokens.
+ (8 TYT..
+ T.
+ (8 TyT,
where and f represent the horizontal and vertical indices of the visual patch within the frame, respectively.
In summary, the parameter 8 in our adjustable temporal spacing allows for a flexible and consistent way to encode the relative positions of text and video tokens.
5. Experiment
5.1. Experimental Setup
Adjustable Temporal Spacing. To scale the temporal in- dex, we introduce scaling factor ó to better align temporal information between visual and textual tokens.
Training Data. We use subset of LLaVA-Video-178 dataset (Zhang et al., 2024c) to train VideoRoPE. The V.VVA-Viee-1788 dataset covers 178k videos and around million censsiion-answers. (QA) pairs from diverse sources such as HD-VILA (Xue et al., 2022), Kinetics (Kay et al., 2017), and ActivityNet (Fabian Caba Heilbron & Niebles, 2015). To balance training efficiency and long-video com- prehension, we randomly select 136k videos with durations under 2 minutes and 18k videos with durations between 2 and 3 minutes. This process yielded our training set of approximately 1.3 million pairs.
Suppose the symbol denotes the token index, for the start- ing text (0 S < T.). the temporal, horizontal, and vertical indices are simply set to the raw token index т. For the video input (T, S 7 < T2 + T.). The difference 7 - T, represents the index of the current frame relative to the start of the video, which is then scaled by 8 to control the space in the temporal dimension. For the ending text (T.+ TT, < < T. TT, +Te), the temporal, horizontal, and vertical index are the same, creating linear progression.
Implementation Details. Using the aforementioned video
6

---

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Table Comparison of different RoPE methods on Long Vedion MLVU, and Video-MME. The benchmarks evaluate perfor- mance across three context lengths: 8k, 16k, 32k, and 64k, where 8k represents context within the training range, and others represent context outside the training range. Our VideoRoPE outperforms other RoPE variants across all three benchmarks. The best results are marked in bold, and the second-best results are underlined. For more information on the evaluation, see Appendix B.
Long Bene
MLVU
Video-MME
Method
8k
16k
32k
64k
16k
32k
64k
16k
32k
64k
Vanilla RoPE (Su et al., 2024)
54.97
54.87
54.56
54.04
63.31
65.79
65.93
62.02
60.67
60.00
61.33
58.33
TAD-RoPE (Gao et al., 2024)
54.14
55.08
53.94
53.42
63.67
65.28
65.28
60.73
60.33
61.33
62.00
58.67
M-RoPE (Wang et al., 2024a)
53,42
52.80
53.11
54.35
60.41
60.68
61.56
61.10
60.67
59.67
61.00
59.67
VideoRoPE (Ours)
54.46
55.29
57.15
57.26
65.19
66.29
66.02
65.56
61.33
61.00
61.67
61.33
Vanilla RoPE
TAD
RePE
(2) CDYAAD-RO JAPE
Frume Depth
20.0
Frans Dephi
20.0
40.0
40.0
50.0
10.0
80.0
100.0
144
72.00
129
3414t
u0 Al
Jod
41 ai
14
129 cM
244
3e0 m
417 Fl
14.9k
72
120 ml
187.2
244
382
stank
a17 sk
14.4
72.4
129
187.2
244 A1
302
560 ok
a1 sk
Contert Longh
Contert Lmng
Contear Leggs
Comeer Lenga
RePE
VideoRaPE
(3) M-RaPE
(4) VileoRoPF
Flunte Depih
100
0.0
Freme Dopth
20.0
0.0
40.0
40.0
60.0
60.0
s0.0
100.0
300.0
144
72.0k
12948
187.36
344
102.4
4/7d
14.4
72
128
107 tt
244
4
dk
72.8
12
117.
346.
dz
4/7 dz
144
22.0k
127.4
187.1
.04
4k
417 cb
Contou Leught
Conear me
Cemiar Leag
Contes Lengut
(a/V-NIAH
(b/V-NIAH-D
Figure 7. Visualization of the retrieval results for V-NIAH and V-NIAH-D. The color gradient from green to red represents the progression of needle retrieval performance, from perfect zero.
training data, we fine-tune different models that use differ- ent positional encoding strategies, such as the Vanilla RoPE (Su et al., 2024), Time-Aware Dual RoPE (TAD-RoPE) (Gao et al., 2024), M-RoPE (Wang et al., 2024a), and our VodE All models are initialized with the Vision Trans- former from Qwen2-VI... -V. and L.MM (Vanilla RoPE) from Qwen2-7B (Yang et al., 2024a). Our fine-tuning incorpo- rates our oP to process the spatiotemporal nature of the video data effectively. We adopt Qwen2-VL: fine- tuning settings, processing each video at 2 fps with max- imum of 128 frames and dynamically adjusting the image resolution to maintain a consistent token count. However, to prevent memory overflow, we use a context window of 8192 tokens.
(Wu et al., 2024a) (8 seconds to hour), MLVU (Zhou et al.,
2024) (3 minutes hours), and Video-MME (Fu et al.,
2024) (11 seconds 60 minutes). For long video retrieval, we use Vision Needle-n----Haystael (V-NIAH) (Zhang et al., 2024d) and our proposed extension. Vision Needle- in-a-Haystack with Distractors (V-NIAH-D), which in- troduces distractor frames to increase the task difficulty. For video hallucination, we use VideoHallucer (Wang et al., 2024d), which evaluates the model's ability correctly answer both basic and hallucinated questions about video content. Details of these benchmarks can be found in Ap- pendix B.
5.2. Results on Long Video Understanding
Our fine-tuning process employs a batch size of 128, a cosine scheduler with a learning rate of le-5, warm-up ratio of le-2, and 704 Nvidia-A100 GPU hours in total. The evaluation involves sampling videos at 2 fps with minimum of 144 image tokens per frame. We use the vLLM framework (Kwon et al., 2023) to support inference on sequences longer than 32k tokens.
As shown in Tab. 2, we compare our VideoRoPE with ex- isting RoPE variants (vanilla RoPE (Su et al., 2024). TAD- RoPE (Gao et al., 2024), and M-RoPE (Wang et al., 2024a)) across three prominent video understanding benchmarks. Our VideoRoPE consistently outperforms all baseline meth- ods across these benchmarks, demonstrating its robustness and adaptability. Specifically, VideoRoPE achieves improve- ments of up 2.91, 4.46, and 1.66 points (64k context length) over the M-RoPE baseline on Long VideoBeB.,,,,,, MLVU, and Video-MME, respectively. These results em- phasize the superior ability of VideoRoPE to effectively
Evaluation Benchmarks. We evaluate our approach using six video benchmarks, including tasks related to long video understanding, long video retrieval, and video hallucination. For long video understanding, we use Long VedcoBench
7

---

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Table Performance comparison of different RoPEs on V-
Table S. Ablation study about different modules of Vide- oRoPE.
NIAH and V-NIAH-D. "Ace." refers to the average accuracy across haystack length and frame depth.
Long Vengoecceee
MLVV
Method
16k
32k
64k
16k
32k
64k
Method
V-NIAH Acc. V-NIAH-D Acc.
Baseline
53.42
52.80
53.11
54.35
60.41
60.68
61.56
61.10
Vanilla RoPF (Su et al., 2024)
31.78
30.22
+ DL
52.17
52.07
53.31
53.63
62.06
63.03
62.52
62.75
TAD-RoPE (Gao al., 2024)
29.33
29.56
&
54.46
55.49
54.66
55.60
63.35
64.09
64.00
63.26
DL & LIA & ATS
$4.46
55.29
57.15
57.26
65.19 66.29 66.02
65.56
M-RoPE (Wang et al., 2024a)
78.67
74.67
VideoRoPE
91.11
87.11
Hallucination task, specifically the Object-Relation Halluci- nation subtask, VideoRoPE achieves an impressive 18.0% improvement over existing methods, highlighting its abil- ity to better discern complex spatial interactions. These results underscore VideoRoPE's robustness in solving video hallucination and potential for real-world video analysis.
Table 4. Performance comparison of different RoPEs VideoHallucer, evaluated at context lengths of 8k, 16k, 32k, and 64k. The maximum result for each RoPE variant across these con- text lengths is displayed, with bold for the top result and underlined for the second-highest. 'OR'= Rblact-Relation, 'T' Temporal, "SD' = Semantic Detail, "F" = Factual, 'NF" = Non-factual.
5.5. Ablation Studies
Method
OR T SD
F
NF
Avg.
Vanilla RoPE (Su et al., 2024)
51.5 30.0 48.0
8.0
43.0
36.T
Ablation Studies on Module Design.
TAD-RoPE (Gao al., 2024)
51.0
37.0
48.0
11.5
47.5
39.0
M-RoPE (Wang et al., 2024a) 39.0
29.0 43.5
12.5
47.5
34.3
We conduct ablation experiments on the modules intro- duced in Section 4, quantitatively evaluating their impact on Long Becc and MLVU benchmarks. The experi- mental results presented in Tab. 5. The baseline setting, M-RoPE (Wang et al., 2024a), achieves scores of 54.35 on Vide and 61.10 on MLVU (both using a 64k context length). By progressively integrating the DL (Diagonal Layout), LTA (Low-frequency Temporal Alloca- tion), and ATS (Adjustable Temporal Spacing) modules, our method shows a continuous improvement in performance, achieving enhanced scores of 57.26 on Long BidcoBench and 65.56 on MLVU (both using a 64k context length). These results demonstrate the effectiveness of our approach in leveraging spatial-temporal positional information. To refine the analysis of x and y allocation in LTA, we quan- titatively evaluate interleaved vs. sequential layouts. We also compare strategies for allocating t, xr, and y. including M-RoPE, uniform interleaved layout, and our VideoRoPE design. Additionally, we explore the optimal ATS scaling factor by varying its value, and further ablate the diagonal layout module to validate the uymmetry-based design. See Appendix A.1 for details.
VideoRoPE
57.0 58,5 50.5 15.0 50.0 46.2
capture long-range dependencies and maintain performance across various challenging video data tasks.
5.3. Results on Long Video Retrieval
Fig. 7 illustrates the performance of V-NTAH and V-NIAH- D with VideoRoPE and other RoPE variants. Specifically, Fig. 7 (a) and (b) demonstrate that the proposed V-NIAH-D is more challenging than V-NIAH. Fig. 7 (1) and (2) show that both Vanilla RoPE and TAD-RoPE exhibit some extrap- olation ability beyond the visual training context. However, both methods fail once they exceed a certain extrapolation limit. In contrast, Fig. (3) and (4) highlight the supe- rior performance of VideoRoPE and M-RoPE in extrapolat- ing within the test context range. While both VodeoRoPE and M-RoPE successfully handle extrapolation, VideoRoPE consistently outperforms M-RoPE, showcasing the robust- ness of the task. Tab. 3 provides a quantitative analysis of the retrieval results, demonstrating a 12.44 % performance improvement of our method over M-RoPE on the Video Retrieval task in both settings, confirming the advantages of our proposed method in video retrieval scenarios.
6. Conclusion
This paper identifies four key criteria for effective positional encoding: 2D/3D structure, frequency allocation, spatial symmetry, and temporal index scaling. As part of our analy- sis, through the V-NIAH-D task, we demonstrate that pre- vious RoPE variants are vulnerable to distractors because of lacking proper temporal allocation. As a result, We pro- pose VideoRoPE that uses a 3D structure for spatiotemporal coherence, low-frequency temporal allocation to reduce oscillations, diagonal layout for spatial symmetry, and adjustable temporal spacing. VideoRoPE outperforms pre- vious RoPE variants in tasks like long video retrieval, video understanding, and video hallucination.
5.4. Results on Video Hallucination
As highlighted in Tab. 4, VideoRoPE significantly surpasses current RoPE methods on the VideoHallucer benchmark. In particular, for the Temporal Hallucination task, VideoRoPE demonstrates a substantial performance improvement of 29.5%, indicating its enhanced capability to accurately cap- ture and process temporal dependencies. This improvement suggests that VideoRoPE is better equipped to handle dy- namie video sequences, where the understanding of time- based relationships is critical. Similarly, for the Spatial
8

---

VidcoRoPE:
What Makes for Good Video Rotary Position Embedding?
Acknowledgments
Chai, W., Song, E., Du, Y., Meng, C., Madhavan, V., Bar-Tal, O., Hwang. J.-N., Xie, S., and Manning. C. D. Auroracap: Efficient, performant video detailed captioning and a new benchmark, 2024. URL htthttss: http://xxvvvv./.o. orgbbs// 03051..
This work was supported by National Key R&D Program of China 20222D0161600, Shanghai Artificial Intelligence Laboratory, the Centre for Perceptual and Interactive Intel- ligence (CPII) Ltd under the Innovation and Technology Commission (ITCY's s InnoHK. Dahua Lin is PI of CPII under the InnoHK.
Chen, L. and Xing, L. Open-llava-next: An open-source im- plementation of llava-next series for facilitating the large multi-modal model community. https: /g1 thub. com/ xi aalachhcee..... 98/ /Open-e----- /at///peennaaaaa LLaVA- -eLaV 2024.
Impact Statement
Chen, I.., Li, J., Dong, X., Zhang, P., He, C., Wang. J., Zhao, F,, and Lin, D. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:211..2277 2023.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, and none of which we feel must be specifically highlighted here.
Chen, L., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Lin, B., Tang, Z., Yuan, L.,, Qiao, Y., Lin, D., Zhao, F., and Wang. Sharegpt4video: Im- proving video understanding and generation with better captions. arXiv preprint arXiv:2406 04320, 2024a.
References
Agrawal, P., Antoniak, S., Hanna, E. B., Bout, B., Chaplot,
D., Chudnovsky, J., Costa, D., Monicault, B. D., Garg, S.,
Gervet, T., Ghosh, S., Héliou, A., Jacob, P., Jiang, A. Q Khandelwal, K., Lacroix, T., Lample, G., Casas, D. L., Lavril, T. Scao, T. L., Lo, A., Marshall, W., Martin, L., Mensch, A., Muddireddy, P., Nemychnikova, V., Pellat, M., Platen, P. V., Raghuraman, N., Rozière, B., Sablay- rolles, A., Saulnier, L., Sauvestre, R., Shang, W. Solet- skyi. R., Stewart, L., Stock, P., Studnia, J., Subramanian, S., Vaze, S., Wang, T., and Yang, S. Pixtral 12b, 2024. URL https::////. / / www.wwwwwwaw...............ww abaaabss22110 07073.
Chen, Y., Xue, F., Li, D., Hu, Q., Zhu, L.., Li, X., Fang. Y., Tang, H. Yang, S., Liu, Z., He, Y., Yin, H., Molchanov, P., Kautz, J., Fan, L.,, Zhu, Y., Lu, Y., and Han, S. Longvila: Scaling long-context visual language models for long videos. 2024b.
Ding, S., Qian, R., Dong, X., Zhang, P., Zang, Y., Cao, Y., Guo, Y., Lin, D., and Wang, J. Sam2long: Enhancing sam 2 for long video segmentation with training-free memory tree. arXiv preprint arXiv:2410. 16268, 2024a.
Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pascanu, R., and Veličković, P. Round and round we go! what makes rotary positional encodings useful? arXiv preprint arXiv:2410.06205, 2024.
Ding, S., Wu, S., Zhao, X., Zang, Y., Duan, H., Dong, X., Zhang. P., Cao, Y., Lin, D., and Wang, J. Mm-ifengine: Towards multimodal instruction following. arXiv preprint arXiv:2504007557, 2025.
Bertasius, G., Wang, H., and Torresani, L. Is space-time attention all you need for video understanding?. 2021.
Ding, Y., Zhang, L. L.,, Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., and Yang, M. LongRoPE: Extending Ilm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753, 2024b.
URL https://///// /aaaiiiiivvv//..///aaaaaaaaaavva www.v.wrg//....tttt.......twwott abs 2102. 05095.
Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H. Fan, Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo, Q., He, C.,, Hu, Y., Huang, T., Jiang, T. Jiao, P., Jin. Z., Lei, Z., Li, J., Li, J., Li, L., Li, S., Li, W., Li, Y., Liu, H., Liu, J., Hong, J., Liu, K., Liu, K., Liu, X., Lv, C., Lv, H., Lv, K., Ma, L., Ma. R., Ma, Z., Ning. W., Ouyang, L., Qiu, J., Qu, Y., Shang, F,, Shao, Y. Song, D., Song, Z., Sui, Z. Sun, P. Sun, Y., Tang, H., Wang, B., Wang, G., Wang, J.. Wang, J., Wang. R., Wang, Y. Wang, Z., Wei, X., Weng, Q. Wu, F., Xiong, Y., Xu, C., Xu, R., Yan, H., Yan, Y., Yang, X., Ye, H., Ying, H., Yu, J., Yu, Zang, Y., Zhang, C., Zhang, I.., Zhang, P., Zhang, P.. Zhang, R., Zhang, S., Zhang, S., Zhang, W., Zhang, W., Zhang, X., Zhang, X., Zhao, H., Zhao, Q., Zhao, X., Zhou, F., Zhou, Z., Zhuo, J., Zou, Y., Qiu, X., Qiao, Y., and Lin, D. Internim2 technical report, 2024.
Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L.., Wei, X., Zhang, S., Duan, H., Cao, M., Zhang, W., Li, Y., Yan, H., Gao, Y., Zhang, X., Li, W., Li, J., Chen, K., He, C., Zhang. X., Qiao, Y., Lin, D., and Wang. J. Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint Mxxi:22401.164222 2024.
internlmxcooppon
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788. 2024.
Fabian Caba Heilbron, Victor Escorcia, B. G. and Niebles, J. C. ActivityNet: A large-scale video benchmark for human activity understanding. CVPR, 2015.

---

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Fu, C.,, Dai, Y., Luo, Y., Li, L.., Ren, S., Zhang, R., Wang. Z.,, Zhou, C., Shen, Y., Zhang, M., et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.210755 2024.
LangChain.
Multi needle in a haystack. pottps::/blln http://bbbool..gcaiiiai.a www.ccainnhheeeer in haystacks, 2024.
Lei, J., Li, L.., Zhou, L.., Gan, Z., Berg. T. L., Bansal, M., and Liu, J. Less is more: Clipbert for aele--and-languaae learning via sparse sampling, 2021. URL https:// arg/abs////// axxiv.org/ abs 22022202222 06183.
Gao, M., Liu, J., Li, M., Xie, J., Liu, Q., Zhao, B., Chen, X., and Xiong, H. TC-LLaVA: Rethinking the transfer from image to video understanding with temporal considera- tions. arXiv preprint Xiix22409003266, 2024.
Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, I.., and Qiao, Y. VideoChat: Chat-centric video understanding, arXiv preprint arXiv:2305.0635: 2023.
Google. Google image search, 2025, URL https:// images. . google. com. Accessed: 2025-01-12.
Li, L., Liu, Y., Yao, L., Zhang, P., An, C., Wang, L., Sun, X., Kong, L., and Liu, Q. Temporal reasoning transfer from text to video. arXiv preprint ar arXiv:2410.061 66, 2024a.
Han, C., Wang, Q., Peng. H., Xiong, W., Chen, Y., Ji, H., and Wang. S. Lm-infinite: Zero-shot extreme length gen- eralization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3991-4008, 2024.
Li, Y., Wang, C., and Jia, J. Llama-vid: An image is worth 2 tokens in large language models. 2024b.
Li, Y., Niu, J., Miao, Z., Ge, C.,, Zhou, Y., He, Q., Dong, X., Duan, H., Ding. S., Qian, R., Zhang, P., Zang, Y., Cao, Y., He, C., and Wang, J. Ovo-bench: How far is your video- lims from real-world online video understanding?. 2025. URL https: arxiv. org/ /aag//////////////////////aa//// 2501. 05510.
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., and Ginsburg, B. Ruler: What's the real context size of your long-context language models? arXiv preprint ar arXiv:2404.06653 2024.
Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., and Yuan, L. Video-Ilava: Learning united visual representa- tion by alignment before projection. arXiv preprint arXiv:2311.10122 2023a.
Huang, B., Wang, X., Chen, H., Song, Z., and Zhu, W. Vtimellm: Empower Ilm to grasp video moments, 2023. URL https:ttp///m // //wxxvavvvvv////v////v/////////a v. orgg abs/2311. 48.
Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., and Yuan, L. Video-llava: Learning united visual representa- tion by alignment before projection. arXiv preprint 2Xiv:2311.10122 2023b.
Huang, Q., Dong, X., Zhang, P., Wang. He, C., Wang, J., Lin, D., Zhang, W., and Yu, N. Opera: Alleviat- ing hallucination in multi- via over-trust penalty and retroppcttion-allocation 2024. URL https::///// / arxiv. http/abb/222111111
large language models
Lin, K., Ahmed, F., Li, L., Lin, C.-C., Azarnasab, E., Yang, Z. Wang. J., Liang, L., Liu, Z., Lu, Y., Liu, C., and Wang, L. Mm-vid: Advancing video understanding with gpt- 4v(ision), 2023c. URL htth::::: URtpht:://axxxv..o //aa/axxvvvorgg//// /bs//
Jin, P., Takanobu, R., Zhang, C., Cao, X., and Yuan, L. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.0800 2023.
2310 19773.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurlPS, 2023a.
Kamradt, G. Needle in a haystack pressure test- ing lmms. https://githit: com/ www/////aaae gaaarrtt///// CaMTestt_eeeleeInAaaaaaaaa
2023.
Liu, X., Yan, H., Zhang, S., An, C., Qiu, X., and Lin, D. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.0520, 2023b.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv preprint arXiv: 1et.111050.9555 2017.
Liu, Y., Ma, Z., Qi, Z., Wu, Y., Shan, Y., and Chen, C. W. E.t. bench: Towards open-ended event-level video-language understanding, 2024a. URL a abs /bs/2409.1811
Kwon, W., Li, 7.,, Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, E.,, Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In ACM SIGOPS, 2023.
Liu, Z.,, Chu, T., Zang, Y., Wei, X., Dong, X., Zhang P., Liang, Z., Xiong, Y., Qiao, Y., Lin, D., et al. Mmdu: A multi-turn multi-image dialog understanding benchmark and weweeetiooottnnnnng dataset for Ivims. arXiv preprint arXiv:2406.I 11833, 2024b.
Labs, B. Flux. https: /tta::///itiii higghu.bb... com/ Black-forest: labs flux, 2023.
10

---

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Liu, Z., Sun, Z., Zang, Y., Li, W., Zhang, P., Dong, X., Xiong, Y., Lin, D., and Wang. J. Rar: Retrieving and ranking augmented milms for visual recognition, 2024c.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 2024.
Liu, Z., Zang, Y., Dong, X., Zhang. P., Cao, Y., Duan, H., He, C., Xiong, Y., Lin, D., and Wang, J. Mia- dpo: Multi-image augmented direct preference optimiza- tion for large vision-language models. arXiv preprint arXiv:2410. 17637, 2024d.
Sun, T., Zhang, X., He, Z., Li, P., Cheng, Q. Liu, X., Yan, H., Shao, Y., Tang, Q. Zhang, S., Zhao, X., Chen, K., Zheng, Y., Zhou, Z., Li, R., Zhan, J., Zhou, Y., Li, L., Yang, X., Wu, L., Yin, Z., Huang, X., Jiang, Y.GG,, and Qiu, X. Moss: An open conversational large language
Luo, R., Zhao, Z., Yang, M., Dong, J., Li, D., Lu, P., Wang, T., Hu, L.. Qiu, M., and Wei, Z. Valley: Video assistant with large language model enhanced ability, 2023. URL https: //t.iva..vvv//////////eae....av org/ wtt//bs/////oooooooro...ot....bt httpp///......... 07207.
model. Machine Intelligence Research, 2024. ISSN 2731-5398. doi: 10.11007/111333-024441330 https: htte::/ggitttbbt wtwp:///e///////////////m/m
URL
Sun, Z., Fang, Y., Wu, T., Zhang, P., Zang, Y., Kong, S., Xiong, Y., Lin, D., and Wang. J. Alpha-clip: A clip model focusing on wherever you want, 2023.
Maaz, M., Rasheed, H., Khan, S., and Khan, S. Video- chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024a.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L.., Rivière, M., Kale, M. S., Love, J., Tafti, P., Hussenot, L., Sessa, P. G., Chowdhery, A., Roberts, A., Barua, A., Botev, A., Castro-Ros, A., Slone, A., Héliou, A., Tacchetti, A., Bulanova, A., Paterson, A., Tsai, B., Shahriari, B., Lan, C. L., Choquette-Choo, C. A., Crepy, C., Cer, D., Ippolito, D., Reid, D., Buchatskaya, E., Ni, E., Noland, E., Yan, G., Tucker, G., Muraru, G.- C., Rozhdestvenskiy, G., Michalewski, H., Tenney, I., Grishchenko, I., Austin, J., Keeling, J., Labanowski, J., Lespiau, J., Stanway, J., Brennan, J., Chen, J., Ferret, J., Chiu, J., Mao-Jones, J., Lee, K., Yu, K., Millican, K., Sjoesund, L. L., Lee, L., Dixon, L., Reid, M., Mikula, M., Wirth, M., Sharman, M., Chinaev, N., Thain, N., Bachem, O., Chang, O., Wahltinez, O., Bailey, P., Michel, P., Yotov, P., Chaabouni, R., Comanescu, R., Jana, R., Anil, R., Mcllroy, R. Liu, R., Mullins, R., Smith, S. L., Borgeaud, S., Girgin, S., Douglas, S., Pandya, S., Shakeri, S., De, S., Klimenko, T., Hennigan, T., Feinberg, V., Stokowiec, W., hui Chen, Y., Ahmed, Z., Gong. Z., Warkentin, T., Peran, L., Giang, M., Farabet, C., Vinyals, O., Dean, J.. Kavukcuoglu, K., Hassabis, D., Ghahramani, Z., Eck, D., Barral, J., Pereira, F., Collins, F.,, Joulin, A., Fiedel, N., Senter, E., Andreev, A., and Kenealy, K. Gemma: Open models based on gemini research and technology, 2024. URL https: arwiv. org/ abs / 220//24433.0 08295.
Maaz, M., Rasheed, H., Khan, S., and Khan, F. S. Video- chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024b.
Men, X., Xu, M., Wang. B., Zhang, Q., Lin, H., Han, X., and Chen, W. Base of rope bounds context length. arXiv preprint arXiv:2405. 14591, 2024.
Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0000
2023.
Qian, R., Ding, S., Dong, X., Zhang, P., Zang, Y., Cao, Y., Lin, D., and Wang, J. Dispider: Enabling video lims with active real-time interaction via disentangled perception, decision, and reaction. arXiv preprint arXiv:2501.0322 2025a.
Qian, R., Dong, X., Zhang, P., Zang, Y., Ding, S., Lin, D., and Wang, Streaming long video understanding with large language models. Advances in Neural Information Processing Systems, 37:1 19336-119360 2025b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.. Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021. URL hwtph:tt//// /tt//axxvv.o// org/ abt////s///1/// 200000
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, F., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023a. URL https: /ttpi/awwwr.gaaa.aeaa////..oorot abs/ 2302. 13971.
00020.
Su, J. Transformer upgrade path: 17. insights into multi- modal positional encoding, March 2024a. URL https: // spaces ac. cn/ ar archives veeh0ie00e000...... 10040.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint MxXiv:2307002288 2023b.
Su, J. A brief discussion on multimodal thinking: 3. posi- tional encoding, Sep 2024b. URL https: htap:::/ss// /spaccs. ac. cn/ arrhivi ves /tee//1332. 1035..

---

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Wang, H., Shi, H., Tan, S., Qin, W., Wang, W., Zhang, T., Nambi, A., Ganu, T., and Wang, H. Multimodal needle in haystack: Benchmarking long-context capability of multimodal large language models. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics, 2025.
Xing, L., Huang, Q., Dong, X., Lu, J., Zhang, P., Zang. Y., Cao, Y., He, C., Wang, J., Wu, F., and Lin, D. Pyramid- drop: Accelerating your large vision-language models via pyramid visual redundancy reduction, 2024. URL https: http///xxvvvrrroo abs 2.10.17244
Xu, H., Ghosh, G., Huang. P.Y.,, Okhonko, D., Agha- janyan, A., Metze, F., Zettlemoyer, L., and Feichten- hofer, C. Videoclip: Contrastive pre-training for zero- shot video-text understanding, 2021. URL https: / //arxiv... www./org////a//////o.oeee /200////100 14084.
Wang, J., Chen, D., Luo, C., Dai, X., Yuan, L., Wu, Z., and Jiang, Y.G.. Chatvideo: A tracklet-centric multimodal and versatile video understanding system, 2023. URL ht https:://aae //axiv.. org/ httpabbs/23300 14407.
Xu, L., Zhao, Y., Zhou, D., Lin, Z., Ng. S. K., and Feng, J. Pllava Parameter- free llava extension from images to videos for video dense captioning, 2024. URL https: / araaivv..o org/ abs / 2404. 16994
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z.,, Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-VL: Enhanc- ing vision- languaee model's perception of the world at any resolution. arXiv preprint arXiv:2409 12191, 2024a.
Xue, H., Hang, T., Zeng, Y., Sun, Y., Liu, B., Yang, H., Fu, J., and Guo, B. Advancing high-resolution video- language representation with large-scale video transcrip- tions. CVPR, 2022.
Wang, W., Zhang, S., Ren, Y., Duan, Y., Li, T., Liu, S., Hu, M., Chen, Z., Zhang, K., Lu, L., et al. Needle in multimodal haystack. arXiv preprint S.Xv:22406.07230, 2024b.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F, et al. Qwen2 technical report. arXiv preprint a MxXv:2407.10671 2024a.
Wang, X., Song. D., Chen, S., Zhang, C., and Wang, B. Longllava: Scaling multi-moda ilms to 1000 images efficiently via hybrid architecture, 2024c. URL https: //arxiv. org/ abs/ / 2409. 02889.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. technical report. arXiv preprint arXiv:2412.15115 2024b.
Wang, Y., Li, K., Li, Y., He, Y., Huang. B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang, Z.,, Xing, S., Chen, G., Pan, J., Yu, J., Wang, Y.. Wang, L., and Qiao, Y. Internvideo: General video foundation models via gen- erative and discriminative learning, 2022. URL https: //arxiv. org/ abs/ 2212. 91.
Yuan, T. Ning, X., Zhou, D., Yang, Z., Li, S., Zhuang, M., Tan, Z., Yao, Z., Lin, D., Li, B., et al. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. arviv preprint ar Xriv:2442.05133. 2024.
Zang, Y., Dong, X., Zhang. P., Cao, Y., Liu, Z., Ding, S., Wu, S., Ma, Y., Duan, H., Zhang, W., al. Internim- xcomposer2. 5-reward: A simple yet effective multi- modal reward model. arXiv preprint arXiv:2501.12366. 2025.
Wang, Y., Wang. Y., Zhao, D., Xie, C., and Zheng, Z. Video- hallucer: Evaluating intrinsic and extrinsic hallucinations in large lideo-language models. arxiv, 2024d.
Wang, Z., Yu, S., Stengel-Esking E., Yoon, J., Cheng, F., Bertasius, G., and Bansal, M. Videotree: Adaptive tree- based video representation for IIm reasoning on long videos. arXiv preprint arXiv:2405. 19209, 2024e.
Zhang, B., Zhang, P., Dong, X., Zang, Y., and Wang, J. Long-clip: Unlocking the long-text capability of clip. arXiv preprint N.Xx:22403.15378 2024a.
Wu, H., Li, D., Chen, B., and Li, J. Long Beng:eehenn::h: A benchmark for long-context interleaved vangu-language understanding. arXiv preprint arXiv:2407.1575. 2024a.
Zhang, H., Li, X., and Bing, L. Video-llama: An instruction- tuned audio-visual language model for video under- standing, 2023a. URL https: pettts:://rxiiie watp/aavivvv..r...a..a.o..../.ra abs/ 2306. 2306.02858. 02858.
Wu, T.-H., Biamby, G., Quenum, J., Gupta, R., Gonzalez, E., Darrell, T., and Chan, D. M. Visual haystacks: A vision-centric needle-in-a-hee haystack benchmark. arXiv preprint 4xXiv:2407.13766 2024b.
Zhang, P., Dong, X., Wang, B., Cao, Y., Xu, C., Ouyang, L., Zhao, Z., Ding, S., Zhang, S., Duan, H., Zhang, W., Yan, H., Zhang, X., Li, W., Li, J., Chen, K., He, C., Zhang, X., Qiao, Y., Lin, D., and Wang, J. InternIm- -ncoppn-xcooppser: vision-language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 2023b.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. arXiv preprint arXiv:2309.1743 2023.
12

---

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Zhang, P., Dong, X., Cao, Y., Zang, Y., Qian, R., Wei, X., Chen, L.., Li, Y., Niu, J., Ding, S., Guo, Q., Duan, H., Chen, X., Lv, H., Nie, Z., Zhang, M., Wang, B., Zhang, W.. Zhang, X., Ge, J., Li, W., Li, J., Tu, Z., He, C., Zhang, X., Chen, K., Qiao, Y., Lin, D., and Wang, J. Internim- xcomposer2.5-ommiiiee A comprehensive multimodal system for long-term streaming video and audio interac- tions. arXiv preprint arXiv:2412.09596, 2024b.
Zhang, P., Dong, X., Zang, Y., Cao, Y., Qian, R., Chen, L.., Guo, Q. Duan, H., Wang, B., Ouyang, L., Zhang. S., Zhang, W., Li, Y., Gao, Y., Sun, P., Zhang, X., Li, W., Li, J.. Wang, W. Yan, H., He, C., Zhang, X., Chen, K., Dai, J., Qiao, Y., Lin, D., and Wang, Internim-xcomposerr 2.5: A versatile large vision language model support- ing long-contextual input and output. arXiv preprint arXiv:2407.03320, 2024c.
Zhang. P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang. Z., Tan, H., Li, C., and Liu, Z. Long text transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024d. prg/abs/2406..15
URL https://arme
Zhang, S., Fang, Q., Yang, Z., and Feng, Y. Llava-mini: Efficient image and video large multimodal models with one vision token, 2025. URL hattts://axxiv.oott abs/2501, 03895.
Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., and Li, C. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 2024e.
Zhao, X., Ding, S., Zhang, Z.. Huang, H., Cao, M., Wang, W., Wang, J., Fang, X., Wang, W., Zhai, G., Duan, H., Yang. H., and Chen, K. Omnialign-v: Towards en- hanced alignment of milms with human preference. arXiv preprint ArXiv:2502. 18411, 2024a.
Zhao, Z., Lu, H., Huo, Y., Du, Y., Yue, T., Guo, L., Wang. B., Chen, W., and Liu, J. Needle in a video haystack: A scalable synthetic framework for benchmarking video milms. arXiv preprint, 2024b.
Zhou, J., Shu, Y., Zhao, B., Wu, B., Xiao, S., Yang. X., Xiong, Y., Zhang, B., Huang, T., and Liu, Z. MLVU: A comprehensive benchmark for multi-task long video understanding. arXiv preprint 4xXiv2440.002644 2024.
13

---

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Appendix
This appendix provides additional resources to further enhance the understanding of our work. Section A, we present ablation studies and extrapolation experiments extending up to 128k, which are included here due to space constraints in the main text. Section B offers a more detailed discussion of the benchmarks used for evaluation. Section reviews related work on video LLMs and video haystack retrieval. Section D showcases examples from our proposed V-NIAH-D benchmark. In Section E, we provide additional attention visualizations to further support the observations discussed in Figure 3. Finally, Section F provides more detailed analysis of the frequency allocation and further elaborates on Figure 4.
A. MORE EXPERIMENTS
A.I. Supplementary Ablation Experiments
A Ablation Studies on the Scaling Factor ò for ATS. We conduct series of experiments further investigate the impact of the temporal scaling factor ó on the alignment between video and text representations. Accurate temporal alignment plays vital role in enhancing the model's understanding of both semantic and sequential aspects of video-langues data. To this end, we evaluate the model's performance across three representative video-language henchmarks-LLogg Loo--Lonoooooloo VideoBench, MLVU, and VideoMME- varying the temporal scaling factor from 0.5 to 3.0. As shown in Table 6, we observe a consistent trend across all benchmarks: performance improves as increases, peaking at 5 = 2 with an average score of 60.92. These results suggest that setting ò = strikes the best balance between temporal resolution and semantic alignment, resulting in optimal overall performance.
Table 6. Performance under different scaling factors across multiple benchmarks.
Scaling Factor ò Long Bide
MLVU
VideoMME
Avg
0.5
50.83
59.87
58.33
56.34
1.0
54.11
63.54
59.67
59.11
2.0
55.50
65.59
61.67
60.92
3.0
53.83
63.38
60.33
59.18
Ablation Studies on x, Allocation. To further investigate the impact of different allocation strategies, we conduct quantitative experiments on our proposed VideoRoPE, comparing sequential and interleaved allocations of x and y. The results, summarized in Table 7, indicate that interleaving æ and y leads to superior performance.
We hypothesize that this improvement arises because interleaving maintains the similarity between the x and y dimensions, whereas sequential allocation increases their disparity, thereby hindering model performance.
Table 7. Ablation Study on x, Allocation. VidcoRoPE (Sequential) represents the sequential allocation of x and following the pattem x, x, .,... Vn y, .. (similar M-RoPE (Wang al., 2024a)). VideoRoPE (Interleaved) represents the interlcaved allocation, following the pattern ..... (similar to Agrawal et al. (2024)).
Long LongVideoBench
MLVU
Method
8k
16k
32k
64k
8k
16k
32k
64k
(ideunoPE(Sequenttaa)
53.73
53.52
54.97
54.77
62.75
63.31
62.75
63.08
VideoRoPE (Interleaved)
54.46
55.29
57.15
57.26
65.19
66.29
66.02
65.56
Ablation Studies on Diagonal Layout Validated on More Benchmarks To further substantiate our claim that the Diagonal Layout (DL) enhances model's capability in video understanding tasks, we conduct additional ablation studies on four diverse and challenging benchmarks: MLVU, VideoHallucer, V-NIAH, and V-NIAH-D. These benchmarks cover wide range of evaluation perspectives, from multi-level video question answering to hallucination detection and fine-grained temporal alignment. As shown in Table 8, incorporating the DL module consistently improves performance over the bascline model across all benchmarks. Specifically, we observe notable gains on MLVU and V-NIAH and V-NIAH-D, suggesting that DL effectively facilitates better temporal reasoning and semantic alignment. These results reinforce the generalizability and robustness of the proposed Diagonal Layout design in understanding across various tasks.
14

---

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Table 8. Effect of Diagonal Layout (DL) across multiple henchmarks.
Method
MLVU
VideoHallucer
V-NIAH
V-NIAH-D
baseline
61.56
34.3
78.67
74.67
+ DL
63.03
34.8
80.44
76.44
Ablation Studies on Different Frequency Allocation Strategies We compare three different frequency allocation strategies: the M-RoPE approach, which emphasizes high-free Sigh-freuuency modeling of temporal information and follows a [t...x.... .........Y ..... format; an interleaved and evenly distributed pattern such as [t t x y x Y x y]; and our proposed VideoRoPE method, which prioritizes positional encoding followed by low-frequency temporal modeling, arranged in [x y...t.... format.
We evaluate these approaches the Long VideoBench benchmark under varying context lengths. This benchmark includes diverse set of video scenarios, ranging from rapidly changing dynamic scenes slowly evolving static content.
As shown in the results below, our low-frequency temporal allocation consistently outperforms the interleaved [t t x X y pattern on average. This suggests that our frequency design more effectively balances global temporal context modeling with local spatial dynamics, making better suited to handle a wide variety of video conditions.
Table 9. Comparison of different frequency allocation strategies under various context lengths.
Context
[t...
yttxyxyl
Ixy....... (Ours)
16k
60.05
59.95
62.03
32k
59.33
58.40
59.54
64k
58.71
57.73
59.12
Avg
59.36
59.06
60.14
A.2. Extrapolation to 128k Experiments
To explore the extrapolation limits of our approach, extend the visual context during inference to 128k. Specifically, utilize the vLLM framework (Kwon et al., 2023) in Server-API processing mode to enable efficient 128k inference.
Due to the prolonged evaluation time required for 128k processing, we focus on the Long Bencc benchmark. As shown in Table 10, although all four methods exhibit performance degradation at 128k, our proposed VideoRoPE experiences the least drop, demonstrating its robustness under extreme extrapolation settings.
Table 10. Comparison of model performance at 64k and 128k context lengths for different methods.
Long LongBideo
Method
64k
128k
Vanilla RoPE (Su et al., 2024)
54.04
48.01
TAD-RoPE (Gao et al., 2024)
53.42
45.77
M-RoPE (Wang et al., 2024a)
54.35
51.45
VideoRoPE
57.26
55.64
B. Additional Details on Evaluation Benchmarks
For long video understanding, we employ three benchmarks: (1) Long VideoBench highlights reasoning questions that depend on long frame sequences, which cannot be effectively addressed by a single frame or few sparse frames, with durations ranging from seconds to 1 hour. We retain only the questions that are free from subtitles. (2) MLVU provides a comprehensive benchmark tailored for assessing the performance of Multimodal Large Language Models in understanding long videos. The dataset features videos lasting between minutes and 2 hours, with nine diverse evaluation tasks. For analysis, we concentrate seven multiple-choice tasks, including Topic Reasoning, Anomaly Recognition, Needle QA, Ego Reasoning, Plot QA, Action Order, and Action Count. (3) Video-MME stands out as high-quality benchmark curated for broad scenario coverage, with videos drawn from six key visual domains and 30 subfields. Its dataset spans wide temporal range, including short clips of seconds and extended videos lasting up to hour.
15

---

VideoRoPE: What Makes for Good Vidco Rotary Position Embedding?
For long video retrieval, we adopt the following two benchmarks: (1) V-NIAH is specifically designed identify highly specific moments within long videos, simulating real-world scenarios where only a small segment of video is relevant within a vast corpus. The setup follows the same configuration Long where "needle" image is inserted at random position within a "haystack" of 3,000 frames. Each needle image corresponds to a particular question, which is unrelated to the content of the haystack. Each frame is encoded with 144 tokens, and the needle frame is inserted at 0.2 depth intervals. Validation begins at 100 frames, with checks every 200 frames up to 3,000. (2) Vision Needle-in-a- -at---aa with Distractors (V-NIAH-D), our proposed method, builds upon V-NIAH by periodically inserting a distractor 200 frames away from the needle. This distractor is semantically similar to the needle, but remains irrelevant to the specific question being asked. The insertion period for the distractor is calculated using 2 100000032 we directly use period of 200 for distractor insertion. For additional examples, refer to Figure 8.
198.7. In our experiments,
For the video hallucination, we use VideoHallucer for evaluation. VideoHallucer classifies hallucinations into two primary types: intrinsic and extrinsic. It further breaks these down into subcategories for detailed analysis, including object-relation, temporal, semantic detail, extrinsic factual, and extrinsic non- non-factual hallucinations. This framework assesses the model's s ability to accurately answer both basic and hallucinated questions about the video content.
C. More Related Works
Related Work on Video LLMs (Video Large Language Models) Video Large Language Models (Video LLMs) build upon the success of image-based vision-language models (VLMs) (Liu et al., 2023a; Zhang et al., 2023b; Dong et al., 2024; Zhang et al., 2024c; Zang et al., 2025; Chen et al., 2023; Chen & Xing, 2024; Liu et al., 2024c;b; Huang et al., 2024; Liu et al., 2024d; Xing et al., 2024; Zhao et al., 2024a; Ding et al., 2025), which align vision and language representations (Radford et al., 2021; Zhang et al., 2024a; Sun et al., 2023) but primarily focus on static images. Extending these models to video requires handling temporal dependencies (Xu et al., 2021; Lei et al., 2021; Bertasius et al., 2021; Huang al., 2023) and long-form video understanding (Wang et al., 2024c; Chen et al., 2024b; Zhang al., 2024d). Early video LLMs, such as Wang et al. (2023) and Li et al. (2023), leverage various Video Foundation Models (ViFMs), such as Intern Video (Wang et al., 2022), to extract video attributes, enabling LLM-based question answering. However, their ability to process video content is constrained by the limitations of ViFMs, restricting their effectiveness to short videos. To address this, Luo et al. (2023) introduces a Temporal Modeling Module, allowing end-to-end training of LLMs on video data. Building on this approach, Maaz et al. (2024a) further enhances spatiotemporal modeling to improve video comprehension. Meanwhile, Zhang et al. (2023a) and Lin al. (2023b) integrate multiple modalities, such as audio and images, to enrich video understanding. These advancements lay the groundwork for processing long videos with greater accuracy. To extend Video LLMs' capabilities to longer content, Lin et al. (2023c) first generates clip-level captions and then employs an LLM to integrate them into a comprehensive vidco caption, effectively representing the entire video. Various studies, such as those by Li et al. (2024b), Jin et al. (2023), Xu et al. (2024), and Zhang et al. (2025), explore different pooling strategies to reduce the number of video tokens, enabling LLMs to process longer videos more effectively. As the field progresses, there is a growing emphasis on long-form video understanding, exploring techniques such as Etreaming-basee processing (Qian et al., 2025a; Li et al., 2025), megory-augmented models (Qian et al., 2025b; Ding et al., 2024a), and hierarchical representations (Wang et al., 2024e) to efficiently model extended temporal structures for tasks like event-level comprehension (Liu et al., 2024a) and video summarization (Chai et al., 2024).
Related Work on Video Haystack Retrieval Originating from the Needle- -nalle-----HHystactt task in Natural Language Processing (Kamradt, 2023; LangChain, 2024), Video haystack tasks aim to locate specific needle, the target information, within vast haystack, collections of video or multi-modal content (Zhang et al., 2024d; Wang et al., 2024b). In the video domain, VNBench (Zhao et al., 2024b) first introduced a video haystack framework with diverse types of needles, such as subtitles, images, and video clips, specifically designed for retrieval tasks within three-minute timeframe. V-NIAH (Zhang et al., 2024d) further advanced the field by extending retrieval tasks to long durations of up to one hour, providing tools for comprehensive evaluation. In a broader modal domain, MMNeedle (Wang et al., 2025) modeled the retrieval task as locating the exact coordinates of a sub-image within larger multi-image haystack, while MM-NIAH (Wang et al., 2024b) introduced a setting where both haystack and needle could include images or text, emphasizing interleaved retrieval capabilitics.
However, video haystack retrieval still lags in difficulty compared with the QA or retrieval task in NLP (Hsieh et al., 2024; Yuan et al., 2024). Few attempts exist to enhance the discriminability of evaluations, such as multi-NIAH (LangChain, 2024; Hsich et al., 2024) or NIAH with distractors (Hsich et al., 2024). Although VHs (Wu et al., 2024b) introduces distractors into
16

---

VideoRoPE: What Makes for Good Video Rotary Position Embedding?
the multi-image haystack setting, where needles and distractors randomly inserted, VHs still did not consider temporal dependencies between video frames more structured approaches task evaluation. Our work builds upon V-NIAH by introducing distractors in systematic, periodic manner based rotary bases.
D. V-NIAH-D Examples
Figure 8 illustrates the five VQA needles used in V-NIAH-D, along with their corresponding distractors. The visual questions and their respective answers are the only components in V-NIAH-D that require human annotation, making it ideal benchmark for evaluating the long-context reasoning capabilities of LMMs.
E. Supplementary Attention Analysis
To further explain the attention pattern in Figure 3, we present additional visual analysis in Figure 9. An attention analysis comparing M-RoPE and VideoRoPE conducted using 8k-context imput, with video tokens from the same frame aggregated through average pooling. As result, one tick on the axis represents a single frame during inference. The evaluation setup for Figure 3 is the same as for Figure 9. M-RoPE relies on high-frequency temporal modeling, limiting it to local information and hindering effective needle identification for question answering. On the other hand, VideoRoPE employs low- fow-frequency temporal modeling, allowing it to capture long-range dependencies and successfully identify the needle for accurate responses.
F. Supplementary Explanation on Frequency Allocation
This section provides detailed explanation of the supplementary information related to Figure 4, highlighting the advantages of our frequency allocation. Consider RoPE-based L.M with head dimension size of 128, corresponding to 64 rotary angles 6n across different dimensions. In each illustration, we visually represent the function cos(0,,t) for 3 dimensions using parallel blue planes.
(a) For M-RoPE (Wang et al., 2024a), temporal dependency modeled using the first 16 rotary angles, which exhibit higher frequency and greater oscillation. Taking the last 3 rotary angles as an example, the position embedding for temporal modeling undergoes significant distortion due to periodic oscillations (Men et al., 2024), as these dimensions have shorter monotonous intervals. Lower dimensions have even shorter intervals. Notably, because the oscillation is periodic, two distant positions can have nearly identical position embeddings, resembling hash collision, shown by the red planes. This phenomenon is why distractors can easily mislead the model.
(b) In contrast, for VidcoRoPE, temporal dependency is modeled using the last 16 rotary angles, which have much wider monotonous intervals. Taking the first 3 rotary angles as an example, the position embedding for temporal modeling is free ferom oscillation (Men et al., 2024). As a result, the misleading effects of distractors are significantly suppressed.
17

---

VidcoRoPE: What Makes for Good Video Rotary Position Embedding?
V-NIAH-D Needles and Distractors
Question: Find the frame with the word 'zoo'. What is the animal outside the 2oo shop?
ZOO
A. lion
TST
B. tiger
C. horse
D. dog
Answer with the option's letter from the given choices directly.
Answer:
Question: frame a couple a wedling... What
color of that ballon?
A. Yellow
B. Red
C. Blue
D. White
Answer with the option's letter from the given choices directly.
Answer: A
Question: Find the frame with the image of Selenium tablets. How many mg does tablet contain? Answer question using single word or phrase,
Pouny SELENIUM meg
Answer:
Question: the frame a scientist. The scientist a...
A. Bird
Elephant
C. Panda
Dog
Answer with the option's letter from the given choices directly.
Answer: C
Question: Find the frame of a teddy bear. Where is this teddy bear?
A. Times Square
B. Eiffel Tower
C. Taj Mahal
D. Sydney Opera House
Answer with the option's letter from the given choices directly.
Answer: A
Figure 8. Five visual guestion-answering problems along with their corresponding needle and distractor.
18

---

VidcoRoPE: What Makes for Good Video Rotary Position Embedding?
haystack frames
needle frumes
baystack frames
M-RePPi dimeniion)
M. RoPE dimersion)
M-RaPPiiy dimension)
Ranos
Questton:: What type of dog, is the owner in sports stroking?
VidroRoFi11 dimenson)
ValeoR.PEIs dinernion)
VidcoRaPEiy dimension))
Jack Kussell
German Shepherd Dulldog
M-RoPE: Shepherd
Video RoPE: A Jack Russelll
haystnck frames
needle frames
haystack frames
RoPEit dimension)
M-RoPEix dimension)
M-RoPEIv dimension)
What popular activity the beach during the monsoon scason?
Question:
VidecitaPiitt dimension)
Videokorli dimension)
VideoitoPlly dimension.
Pienicking
Surting
Plainng volleyball Fishing
XI-RoPE: Surting
Video RoPB: D. Fishing
haystack frames
needle
haystack frames
Question: What painted in the colors Poland flag in the video?
Videoito/Uii dicsension)
VideoitaP
A 'The doctor's cost
B. doctor's medical mark
c. Thee doctor's is stethoscope D. The doctar's glasses
M-RoPE: B. The doctor's medical mask
glasses
huvstack frames
baystack frames
RoPPiyy
Question: What does the hand coming out the computer do?
VidcoRaPPi dimensson)
ValcoRoPRjp dimerssoo))
VidunReePPly dimension)
"tikee womont cand
Shabus venman's
M-RoPE: A. Takes the woman's is credit card
RoPE: D. Delivers product
haystack Tromes
frames
baystack Trames
M RoPEit dimension).
M RoPEix dimension)
RoPUly dimension)
Question: What is the gender the patient receiving the microblsttnng routine in the video?
VidcoRoPEi dimension)
VideoRoPEly
A. Not specified
B. Both male and fomale
C. Pemale
D. Male
M--oPE: A. Not spocified
Video RoPE: C, Pemale
Figure Additional visual analysis attention.
19


Reviews:
[{"review_id": "LHpMGse63W", "reviewer": "Reviewer_t9dP", "summary": "This paper proposes VideoRoPE, a new positional embedding for videos. VideoRoPE extends the 1D RoPE to 3D cases to encode spatiotemporal information. M-RoPE used in QWen2-VL (Wang et al., 2024b) employ 3D structure, dividing the feature dimensions into distinct subsets for spatiotemporal encoding. However, the authors argue that the allocation of feature dimensions is suboptimal. To address this issue, they propose to assign higher dimensions (lower frequencies) to the temporal axis, enhancing long-range temporal dependence modeling without interference from oscillations. Extensive experiments have been conducted to compare VideoRoPE against RoPE variants such as Vanilla RoPE, TAD-RoPE, and M-RoPE. The results on LongVideoBench and MLVU demonstrate VideoRoPE’s superior performance in long video understanding, retrieval, and hallucination.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer t9dP,\n\nThanks for your valuable feedback. We sincerely thank all reviewers for acknowledging that this paper is novel (t9dP, juMQ),  well-motivated (t9dP, 26sL, 3qrY), significant improvements (t9dP, 26sL, 3qrY), well-written (t9dP, juMQ, 26sL, 3qrY), and easy to follow (t9dP, 3qrY). Below we address your questions:\n\n> Q1: Only one LLM i.e., QWen2-VL, is evaluated. It would be more convincing to show improvement over more base LLMs.\n\nThanks for your suggestion. We have updated the table below with results on QWen2.5-VL. Performance is reported across three categories: long video understanding (LongVideoBench, MLVU, VideoMME) with a 64k context length, short video hallucination at an 8k context length, and average retrieval accuracy. Across all six benchmarks, VideoRoPE's performance remains consistently superior to other RoPE variants, including Vanilla RoPE, TAD-RoPE, and M-RoPE. We will add these results to our revised version.\n\n|Qwen2.5VL-7B|LongVideoBench|MLVU|VideoMME|VideoHallucer|V-NIAH|V-NIAH-D|\n|-|-|-|-|-|-|-|\n|Vanilla RoPE|53.37|63.13|60.0|45.1|31.77|29.99|\n|TAD-RoPE|54.62|66.78|59.0|45.1|28.44|24.88|\n|M-RoPE|58.71|68.09|60.6|45.3|77.33|75.11|\n|VideoRoPE|**59.20**|**68.84**|**61.6**|**45.8**|**86.44**|**86.22**|\n\n> Q2: More recent (SOTA) baselines should be compared. For example, LLaVA-Video, LLaVA-OneVision with different sizes.\n\nWe appreciate your feedback. We have added the experimental results of Video-RoPE with the SOTA models including LLaVA OneVision, LongVU, Apollo and LLaVA-Video with different sizes (3B and 7B). \n\nResults with 3B size models:\n|Model|LongVideoBench|MLVU|VideoMME|\n|-|-|-|-|\n|VILA1.5-3B|42.9|44.4|42.2|\n|Phi-3.5-Vision-4.2B|-|-|50.8|\n|LongVU-3.2B|-|55.9|51.5|\n|Qwen2.5-VL-3B-VideoRoPE|**54.7**|**62.6**|**58.3**|\n\nResults with 7B size models:\n|Model|LongVideoBench|MLVU|VideoMME|\n|-|-|-|-|\n|LLaVA OneVision-7B|56.3|64.7|58.2|\n|LongVU-7B|-|65.4|60.6|\n|Apollo-7B|58.5|**70.9**|61.3|\n|LLaVA-Video-7B|58.2|70.8|63.3|\n|Qwen2.5-VL-7B-VideoRoPE|**62.0**|70.7|**64.4**|\n\nNotably, our method achieves better results with significantly less fine-tuning data (0.33 million) than prior SOTA approaches (8.8 million for LLaVA OneVision, 3.75 million for LongVU, 3.2 million for Apollo, and 2.7 million for LLaVA-Video). Our results highlight the benefits of using our RoPE design in Video Large Language Models.\n\n> Q3: The performance degradation at 64k context length (Tab. 2) is not well explored in the main paper.\n\nIn the Tab. 2 of main paper, the performance of 64k (65.56 for MLVU, 61.33 for VideoMME) is slightly worse that 32k (66.02 for MLVU, 61.67 for Video-MME) on some benchmarks. The reason is that simply increasing the context window doesn't guarantee improved performance. It is challenging for the model's ability to effectively process and utilize extremely long contexts. For example, in Figure 5 of the technical report of QWen2-VL based on M-RoPE, the performance of 64k (70.4) is also slightly worse than 48k (71.3). We further tested the performance on our VideoRoPE on 48k and 80k context length. The results at 48k and 80k showed performance fluctuations, which are consistent with Figure 5 of the QWen2-VL technical report.\n\n|Method (VideoRoPE)|8k|16k|32k|48k|64k|80k|\n|-|-|-|-|-|-|-|\n|LongVideoBench|54.46|55.29|57.15|56.22|57.26|56.53|\n|MLVU|65.19|66.29|66.02|66.43|65.56|64.88|", "claims_and_evidence": "The paper makes a strong claim that their proposed VideoRoPE method is superior for video understanding tasks compared to existing RoPE variants. The claims are generally well-supported by the evidence presented. For example, VideoRoPE consistently outperforms Vanilla RoPE, TAD-RoPE, and M-RoPE across multiple popular video understanding benchmarks and the proposed Visual Needle-In-AHastack-Distractor benchmark.", "methods_and_evaluation_criteria": "Method: \n\nThe key idea of VideoRoPE is to improve the frequency allocation of the existing M-RoPE. Allocating higher dimensions (lower frequencies) to the temporal axis is a well-motivated strategy. It aims to prioritize temporal modeling and capture long-range dependencies, which are essential for video understanding. The justification based on the \"global branch\" corresponding to higher dimensions from other works also lends credibility to this. The diagonal layout is intended to maintain spatial symmetry and prevent bias toward input order, which is  reasonable too as it ensures that visual input receives equal contextual influence from preceding and subsequent textual information.\n\nEvaluation Criteria:\n\nThe paper evaluates VideoRoPE on a diverse set of video understanding benchmarks including Video-MME, MLVU, etc. The paper also proposes the Visual Needle-in-a-Haystack with Distractors benchmark to evaluate the retrieval capability. This paper follows the commonly used metrics like average accuracy to compare again baselines.", "theoretical_claims": "There is no formal proof and no theoretical claim in the paper.", "experimental_designs_or_analyses": "The experimental design looks sound. The experiments are quite extensive in terms of the number of benchmark and tasks evaluated. The main experiments include results on Long Video Understanding (covering 3 standard video QA benchmarks), results on Long Video Retrieval (the needle test), results on Video Hallucination. Moreover, ablation studies are provided to further justify the design choices.", "relation_to_broader_scientific_literature": "This paper proposes VideoRoPE, an new video positional embedding that is aware of spatiotemporal information. Compared to existing RoPE variants (Vanilla RoPE, TAD-RoPE, and M-RoPE), it has nice properties such as 2D/3D structure awareness, frequency allocation, spatial symmetry, and temporal index scaling. Overall, I believe this paper provides a valuable contribution to the field of video understanding with LLMs.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n-The proposed VideoRoPE is novel to the best my my knowledge.   \n\n-The improvement on many video benchmarks is significant.\n\n-The paper is well-structured and easy to follow. The problem is clearly defined, the proposed method is clearly explained, and the experimental results are clearly presented.\n\nWeaknesses:\n\n-Only one LLM i.e., QWen2-VL, is evaluated. It would be more convincing to show improvement over more base LLMs. \n\n-More recent (SOTA) baselines should be compared. For example, LLaVA-Video, LLaVA-OneVision with different sizes. \n\n-The performance degradation at 64k context length (Tab. 2) is not well explored in the main paper.", "other_comments_or_suggestions": "The authors are encouraged to address the concerns in the weakness section.", "questions_for_authors": "See the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DSQCtOXHOt", "reviewer": "Reviewer_juMQ", "summary": "VideoRoPE is a position embedding method designed for video large language models. It addresses four key issues: 3D structure, frequency allocation, spatial symmetry, and temporal index scaling. The authors demonstrate through a new benchmark V-NIAH-D that existing methods perform poorly when distractors are present. \nVideoRoPE solves these problems through low-frequency temporal allocation, diagonal layout, and adjustable temporal spacing, outperforming other RoPE variants in video retrieval, understanding, and hallucination benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer juMQ,\n\nThanks for your valuable feedback. We sincerely thank all reviewers for acknowledging that this paper is novel (t9dP, juMQ),  well-motivated (t9dP, 26sL, 3qrY), significant improvements (t9dP, 26sL, 3qrY), well-written (t9dP, juMQ, 26sL, 3qrY), and easy to follow (t9dP, 3qrY). Below we address your questions:\n\n> Q1: The paper covers most of the essential references related to position embeddings for transformer models and video understanding. However, there are a few relevant works that are not cited or discussed that could provide additional context.\n\nThank you for the suggestion. We will incorporate citations and a discussion of Auroracap, MVBench, and VidComposition into the revised manuscript. These benchmarks, which evaluate video large language models, offer valuable insights into detailed video captioning (Auroracap), general video understanding (MVBench), and compositional structure analysis (VidComposition). We also report the performance of our VideoRoPE on these benchmarks, see Q2 below.\n\n> Q2:  The paper could potentially be enhanced by discussing additional benchmarks that, while not included in the current comparison, might provide valuable insights, such as those works referenced in the \"Essential References Not Discussed\" section.\n\nThanks for your suggestion. We evaluate our method on the Video-Detailed-Caption (VDC) benchmark proposed by Auroracap and MVBench. Since VidComposition has not yet been open-sourced, we will evaluate it once the benchmark is released.\n\n|Model|VDC|MVBench|\n|-|-|-|\n|Auroracap-7B|38.2|-|\n|Vanilla RoPE|43.0|67.1|\n|TAD-RoPE|43.8|66.9|\n|M-RoPE|**44.0**|67.8|\n|VideoRoPE|**44.0**|**68.4**|\n\n> Q3: How would VideoRoPE need to be adapted for streaming video understanding tasks where the full temporal context isn't available upfront? Since there are many new datasets and benchmarks [1, 2] for streaming video understanding, which is more and more important.\n\nThank you for this insightful question regarding the adaptation of VideoRoPE to streaming video understanding tasks. We acknowledge the growing importance of streaming video understanding, as it closely aligns with how humans naturally perceive the world. Accordingly, we will reference the works you mentioned in our manuscript and discuss the significance of streaming modeling for video understanding tasks.\n\nBy design, VideoRoPE effectively supports streaming input scenarios, in contrast to RoPE-TIE-V2, which requires prior knowledge of the video's total length and thus cannot accommodate streaming data. Although VideoRoPE was not explicitly trained on streaming video data, we evaluated its effectiveness for streaming modeling on the StreamingBench Benchmark[2]. Specifically, we focused on the Real-Time Visual Understanding (RTVU) setting relevant to streaming modeling, as presented in the table below. The results demonstrate that VideoRoPE still exhibits superior performance in streaming video scenarios. \n\n|Model|StreamingBench (RTVU)|\n|-|-|\n| Vanilla RoPE|75.0|\n| TAD-RoPE|75.8|\n| M-RoPE|76.2|\n| VideoRoPE|**77.1**|", "claims_and_evidence": "Yes. VideoRoPE's superior performance is demonstrated through comprehensive evaluations across multiple benchmarks with significant improvements (+12.44% on V-NIAH-D, +11.9% on VideoHallucer).", "methods_and_evaluation_criteria": "Yes. The proposed method directly addresses video position embedding challenges, the new V-NIAH-D benchmark effectively tests model robustness to distractors, and the evaluation uses appropriate benchmarks covering understanding, retrieval, and hallucination tasks across multiple context lengths, which make sense to me.", "theoretical_claims": "Yes, I did. This paper doesn't contain formal mathematical proofs or theoretical claims requiring verification. These equations correctly describe implementation details of vanilla RoPE, M-RoPE, and VideoRoPE, but they serve as explanations rather than theoretical proofs.", "experimental_designs_or_analyses": "Yes, I did. The V-NIAH-D benchmark with periodic distractors effectively tests robustness to frequency-based issues. And the evaluations are comprehensive, testing across multiple datasets and context lengths (8k-64k) to assess both in-distribution and extrapolation capabilities.", "relation_to_broader_scientific_literature": "VideoRoPE extends position embedding research from text-only models to video, addressing the unique challenge of handling both spatial and temporal dimensions in one attention space. It builds on long-context understanding work (like LongRoPE) and attention mechanism research (particularly attention sinks), while making novel contributions to frequency allocation for effectively modeling video's complex spatio-temporal structure.", "essential_references_not_discussed": "The paper covers most of the essential references related to position embeddings for transformer models and video understanding. However, there are a few relevant works that are not cited or discussed that could provide additional context:\n\n[1] Chai, W., Song, E., Du, Y., Meng, C., Madhavan, V., Bar-Tal, O., ... & Manning, C. D. (2024). Auroracap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint arXiv:2410.03051.\n\n[2] Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., ... & Qiao, Y. (2024). Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22195-22206).\n\n[3] Tang, Y., Guo, J., Hua, H., Liang, S., Feng, M., Li, X., ... & Xu, C. (2024). VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?. arXiv preprint arXiv:2411.10979.", "other_strengths_and_weaknesses": "Other Strengths:\nS1: VideoRoPE addresses key issues of current position embedding for video-LLM including 3D structure, frequency allocation, spatial symmetry, and temporal index scaling.\nS2: The paper has very strong visualizations to explain frequency allocation and position embedding.\nS3: The V-NIAH-D benchmark is a valuable contribution that provides a way to test model robustness to distractors.\n\nOther weakness: no obvious drawbacks.", "other_comments_or_suggestions": "The paper could potentially be enhanced by discussing additional benchmarks that, while not included in the current comparison, might provide valuable insights, such as those works referenced in the \"Essential References Not Discussed\" section.", "questions_for_authors": "Q1. How would VideoRoPE need to be adapted for streaming video understanding tasks where the full temporal context isn't available upfront?\nSince there are many new datasets and benchmarks [1, 2] for streaming video understanding, which is more and more important.\n\n[1] Yang, Z., Hu, Y., Du, Z., Xue, D., Qian, S., Wu, J., ... & Xu, C. (2025). SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding. arXiv preprint arXiv:2502.10810.\n\n[2] Lin, J., Fang, Z., Chen, C., Wan, Z., Luo, F., Li, P., ... & Sun, M. (2024). Streamingbench: Assessing the gap for mllms to achieve streaming video understanding. arXiv preprint arXiv:2411.03628.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "66g7Sr8H9F", "reviewer": "Reviewer_26sL", "summary": "This paper identifies four key factors in extending position encodings from images to videos: spatio-temporal structures, frequency allocation, spatial symmetry, and temporal indexing. Drawing from these observations, the authors propose VideoRoPE, which (1) uses low-frequency temporal allocation to mitigate periodic oscillations, (2) applies a diagonal layout to maintain spatial symmetry, and (3) adopts adjustable temporal spacing to decouple temporal from spatial indexing. The method shows consistent performance improvements in long video retrieval, video understanding, and video hallucination tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 26sL,\n\nThanks for your valuable feedback. We sincerely thank all reviewers for acknowledging that this paper is novel (t9dP, juMQ),  well-motivated (t9dP, 26sL, 3qrY), significant improvements (t9dP, 26sL, 3qrY), well-written (t9dP, juMQ, 26sL, 3qrY), and easy to follow (t9dP, 3qrY). Below we address your questions:\n\n> Q1: Temporal allocation...showing some specific failure cases of high-frequency allocation...\n\nFailure cases of high-frequency allocation in M-RoPE have shown in Fig. 3 and Appendix E. Specifically, the M-RoPE responses illustrate its limitations, with attention visualizations revealing that high-frequency encoding hinders long-range temporal dependencies, favoring local patterns instead.\n\n> Q2: Spatial symmetry...more empirical qualitative study of how spatial inconsistency becomes problematic\n\nWe have quantitatively shown the impact of spatial symmetry in Tab. 5 (second row), with a +1.65 gain on MLVU under 64k context over M-RoPE. To further validate its importance, we evaluated it on 4 benchmarks with consistent improvements supporting our claim.\n\n|Method|MLVU|VideoHallucer|V-NIAH|V-NIAH-D|\n|-|-|-|-|-|\n|baseline|61.56|34.3|78.67|74.67|\n|+ DL|63.03|34.8|80.44|76.44|\n\n> Q3: While using only low frequencies...potentially risks missing short-term or local dynamics...\n\nSee Q4.\n\n> Q4: Why the temporal encoding frequencies are not interleaved with the spatial frequencies ... For example, ... [t t x y x y x y … t t x y x y x y] or [t x y t x y x y … t x y t x y x y] to preserve the ability to capture both long- and short-term variations.\n\nWe trained a model using the \"[t t x y x y x y]\" format and conducted additional comparative experiments with varying context lengths on the LongVideoBench benchmark, which features a wide range of video scenarios—including both rapidly changing dynamic content and slowly evolving scenes. Our results show below that, on average, low-frequency temporal allocation consistently outperforms the \"[t t x y x y x y]\" arrangement. This suggests that our frequency design effectively balances the modeling of both global context and local dynamics across diverse video conditions. As for the \"[t x y t x y x y … t x y t x y x y]\" pattern, we plan to report on it in future updates.\n\n|Context Length|[t...x...y...]|[t t x y x y x y]|[xy...t...] (Ours)|\n|-|-|-|-|\n|16k|60.05|59.95|62.03|\n|32k|59.33|58.40|59.54|\n|64k|58.71|57.73|59.12|\n|Avg|59.36|59.06|60.14|\n\n> Q5: ...The performance seems to be sensitive to the δ on LongVideoBench. Does the optimal δ vary significantly across benchmarks, or is it stable? Did you tune it separately for each dataset?\n\nTable 6 presents the ablation study of the scaling factor on LongVideoBench. Additionally, we report the average results on other benchmarks, MLVU and VideoMME, in the table below. Our analysis indicates that the optimal δ remains consistent across various tasks. We would also like to clarify explicitly in the revised manuscript that we did not tune this δ separately for each individual dataset.\n\n|δ|LongVideoBench|MLVU|VideoMME|Avg|\n|-|-|-|-|-|\n|0.5|50.83|59.87|58.33|56.34|\n|1|54.11|63.54|59.67|59.11|\n|2|55.50|65.59|61.67|**60.92**|\n|3|53.83|63.38|60.33|59.18|\n\n> Q6-1: The core findings and solutions (i.e., diagonal layout) for spatial symmetry was already proposed in (Su et al, 2024b). Personally, I think the proposed method essentially extends RoPE-Tie-V2(Su et al, 2024b) to the temporal domain.\n\nThis is not true. RoPE-Tie-V2 (Su et al, 2024b) introduced the concept of spatial symmetry but did not provide experimental validation, nor did it discuss the diagonal layout. Additionally, RoPE-Tie-V2 cannot handle streaming video inputs (as discussed by reviewer juMQ regarding streaming scenarios)\n\n> Q6-2: The newly featured low-frequency temporal allocation and temporal spacing are technically less significant or already proposed in (Li et al., 2024), respectively.\n\nThis is not true. Li et al. (2024) primarily focuses on enhancing video models from a data perspective, using only temporal reasoning data in pure text. Their work does not address low-frequency temporal allocation or temporal spacing—both of which are novel contributions introduced by our method. We will clearly highlight these distinctions in our revised manuscript.\n\n> Q7: [a] Heo et al., should be cite.\n\nWe will cite this work in our related work section. This paper is designed for 2D image tasks. On the contrary, our VideoRoPE is designed for 3D video tasks.\n\n> Q8: Captions in Fig.2 \n\nThank you for pointing that out. Fig. 2 and the last row of Fig. 7 are indeed the same, both showing M-RoPE and VideoRoPE performance on V-NIAH and V-NIAH-D benchmarks. The x-axis indicates context length, the y-axis shows frame depth, and the green-to-red color bar reflects needle retrieval accuracy—from perfect to zero.\n\n> Q9: Typo in Eq.7\n\nThank you for identifying this typo. We will correct and clarify this equation in our revised submission.", "claims_and_evidence": "The four key aspects conceptually sound but stronger empirical validation is required, especially on frequency allocation and spatial symmetry.\n\nTemporal allocation: The authors’ figures and retrieval results (e.g., Fig. 6) support that low-frequency allocation reduces periodic oscillations, but explicitly showing some specific failure cases of high-frequency allocation would strengthen the claim.\n\nSpatial symmetry: The only reference for spatial symmetry (Su et al., 2024b) is very weak. It’s just a blog not a peer-reviewed source. I understand the concept of the need of spatial symmetry but I think the authors should have provided more empirical qualitative study of how spatial inconsistency becomes problematic.", "methods_and_evaluation_criteria": "The authors’ approach—incorporating low-frequency temporal allocation, a diagonal layout, and adjustable temporal spacing—is broadly reasonable. However, one concern remains regarding low-frequency temporal allocation.\n\nWhile using only low frequencies may help capture global or long-range context, it potentially risks missing short-term or local dynamics, which are often critical in videos displaying rapid changes (e.g., fast-moving objects or abrupt scene transitions).\n\nA related question is why the temporal encoding frequencies are not interleaved with the spatial frequencies (like the 2D RoPE approach), balancing both low and high frequencies for the time dimension. For example, one could allocate 32, 48, 48 frequencies for (t,x,y) and interleave them as [t t x y x y x y … t t x y x y x y] or [t x y t x y x y … t x y t x y x y] to preserve the ability to capture both long- and short-term variations.", "theoretical_claims": "Theoretical claims appear to sound although further empirical and qualitative analysis should provide compelling support.", "experimental_designs_or_analyses": "Most of the experiments on different video tasks appear valid and sufficiently controlled, except scaling factor ablation in Tab.6.\nThe performace seems to be sensitive to the scaling factor $\\delta$ on LongVideoBench. Does the optimal scaling factor \\delta vary significantly across benchmarks, or is it stable? Did you tune it separately for each dataset?", "relation_to_broader_scientific_literature": "The core findings and solutions (i.e., diagonal layout) for spatial symmetry was already proposed in (Su et al, 2024b). Personally, I think the proposed method essentially extends RoPE-Tie-V2(Su et al, 2024b) to the temporal domain. The newly featured low-frequency temporal allocation and temporal spacing are technically less significant or already proposed in (Li et al., 2024), respectively.", "essential_references_not_discussed": "[a] Heo et al., “Rotary Position Embedding for Vision Transformer,” ECCV, 2024.\n\nThis reference would provide additional context for 2D extensions of RoPE in vision tasks.", "other_strengths_and_weaknesses": "Strengths:\n- The paper is well-written, motivating the problem clearly.\n- Empirical results consistently demonstrate improvements across diverse tasks.\n\nWeaknesses:\n- Please see above.", "other_comments_or_suggestions": "- Captions in Fig.2 should be more informative. Especially, on right figure, the caption should explain what the x, y axes, and color indicates\n\n- In Eq.7, I think the equation needs to be correct as, $(t,x,y)=(\\tau + (\\delta - 1)T_v, \\tau + (\\delta - 1)T_v, \\tau + (\\delta - 1)T_v)$ if $T_s+T_v \\leq \\tau \\leq T_s+T_v+T_e$", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dQUVlfjWS2", "reviewer": "Reviewer_3qrY", "summary": "In this work, the authors study criteria for video position embedding and propose a rotary position embedding method. They claim that a good video position embedding should handle 3D structure, and appropriate frequency allocation to prevent embedding collision, spatial symmetry and temporal index scaling between text and visual tokens. According to these criteria, they build their method upon M-RoPE (Wang et al., 2024b). They propose low-frequency temporal allocation to avoid embedding collision, and diagonal position ID layout to make the token distances more balanced, and adjustable temporal scaling that controls temporal interval between visual tokens with a hyperparameter. They validate the proposed idea on public benchmarks: long video understanding, long video retrieval, and video hallucination.\n\n\n\n## update after rebuttal\nMy major concerns are all resolved by the rebuttal. I have increased my rating to strong accept.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3qrY,\n\nThanks for your valuable feedback. We sincerely thank all reviewers for acknowledging that this paper is novel (t9dP, juMQ),  well-motivated (t9dP, 26sL, 3qrY), significant improvements (t9dP, 26sL, 3qrY), well-written (t9dP, juMQ, 26sL, 3qrY), and easy to follow (t9dP, 3qrY). Below we address your questions:\n\n> Q1: L266-270, second column. Why such collisions make the model susceptible to distractor influence?...Can the authors provide either logical explanations or more direct empirical evidence on this?\n\nWe appreciate your request for clarification. As shown in Fig. 4 and Appendix F, we have linked position embedding collisions in M-RoPE to distractor influence. Due to high-frequency oscillations, distant positions can share nearly identical embeddings (red planes in Fig. 4), making the model vulnerable to distractors. In contrast, VideoRoPE avoids such oscillations, enabling more robust temporal modeling. We have also conducted an ablation study (Table 5), confirming that LTA(Low-frequency Temporal Allocation)—by preventing temporal embedding collisions—improves video understanding performance.\n\n> Q2: L413-414, first column:  It is quite arguable that V-NIAH-D is more challenging than V-NIAH by merely looking into Fig. 7. Can the authors provide quantitative measure of challengingness of these two datasets?\n\nIn the table below, we provide the quantitative performance gaps observed in various models, including our Video-RoPE, tested on both V-NIAH and V-NIAH-D datasets:\n\n|Model|V-NIAH|V-NIAH-D|\n|-|-|-|\n|LLaVA-NeXT-Video-7B|14.66|3.99|\n|LongVA|90.88|80.44|\n|Qwen2.5-VL-7B-VideoRoPE|86.44|86.22|\n\nThese results indicate a significant performance drop on V-NIAH-D for most methods (-10.67 for LLaVA-NeXT-Video-7B and -10.44 for LongVA), supporting our claim that V-NIAH-D is more challenging than V-NIAH. Also, our VideoRoPE effectively mitigates the misleading influence of distractors in V-NIAH-D (-0.22 merely). We will include this quantitative comparison in our revised manuscript.\n\n> Q3:  I list a few errors, typos, and awkward sentences below.\n\nThank you for highlighting language issues and awkward expressions. We will carefully revise the manuscript to correct these:\n- \"uses dimensions\" (L214-216) will be corrected to \"uses different dimensions\".\n- \"The visualized relationship between the periodicity, monotonicity, and temporal modeling\" (L271-273) will be corrected to \"The relationship between periodicity, monotonicity, and temporal modeling is visualized in Fig 4.\"\n- \"modes\" will be corrected to \"models\" (L357).\n\n> Q4: In Fig. 2 right side, what do the axes mean and each cell color mean?\n\nThank you for pointing that out. Fig. 2 and the last row of Fig. 7 are indeed the same, both showing M-RoPE and VideoRoPE performance on V-NIAH and V-NIAH-D benchmarks. The x-axis indicates context length, the y-axis shows frame depth, and the green-to-red color bar reflects needle retrieval accuracy—from perfect to zero.\n\n> Q5: In Fig. 3 of the main paper, and Fig. 9 of the supplementary material, How do we interpret the results? For instance, in L165-176, second column, what does it mean by “the needle is located primarily through vertical positional information”? I do not understand this paragraph. Can the authors elaborate on this paragraph?\n\nIn Fig 3, we can derive the fact from different component of the attention score, that M-RoPE locates and retrieves the needle using the component with respect to the feature dimension capturing vertical positional information, namely the product of the last 48 dimension of Q and K in self-attention. This goes against the design intention of M-RoPE. \n\n> Q6: In L298-303, first column, what does it mean by “creating a stack in the bottom-left corner”? I do not understand this paragraph as well. Can the authors elaborate on this paragraph? Can the authors give more intuitive explanation on Fig. 6?\n\nThe three subfigures in Figure 6 represent the schema of Vanilla RoPE, M-RoPE, and VideoRoPE when encoding text-video-text input. The three dimensions of each subfigure represent the sequential indices (both textual and temporal indices), and the horizontal and vertical positions in each frame. We find that each dimension of Vanilla RoPE increases with the input token indices, regardless of the existence and the spatiotemporal characteristics of video, which is a straight diagonal line in 3D space. After that, M-RoPE uses part of dimensions to represent the spatial information of the video, but the horizontal and vertical position indices do not change with the video frames. Therefore, it appears as a vertical stack in the bottom-left corner in 3D space, deviating from the diagonal direction, different from the pre-trained features in the text. Comparatively, VideoRoPE allows the horizontal and vertical position indices of each frame to change with the frame indices, while maintaining the diagonal features, which makes it easier to transfer LLM to the video mode.", "claims_and_evidence": "Some of the claims are not fully supported by clear and convincing evidence.\n- L266-270, second column: “This periodicity creates “hash collisions” (red planes), where distant positions share near-identical embeddings, making the model susceptible to distractor influence.” $\\rightarrow$ Why such collisions make the model susceptible to distractor influence? I understand the proposed method helps avoiding such collisions and improves V-NIAH-D performance significantly upon M-RoPE baseline. However, I do not see any logical connection between the collision and distractor influence. Can the authors provide either logical explanations or more direct empirical evidence on this?\n- L413-414, first column: “Fig. 7 (a) and (b) demonstrate that the proposed V-NIAH-D is more challenging than V-NIAH.” $\\rightarrow$ It is quite arguable that V-NIAH-D is more challenging that V-NIAH by merely looking into Fig. 7.  Can the authors provide quantitative measure of challengingness of these two datasets?", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense for the problem of video positional embedding.", "theoretical_claims": "There is no theoretical claim made.", "experimental_designs_or_analyses": "I have checked all experimental designs and analysis. They all seem to be valid.", "relation_to_broader_scientific_literature": "The key contribution of this paper is related to the general multi-modal LLM field. The findings and proposed method could be helpful for the general audience of the field.", "essential_references_not_discussed": "I do not see any missing essential references not discussed.", "other_strengths_and_weaknesses": "- Strengths: This paper is well-motivated and the proposed method is sensible. The experimental results demonstrate the clear merit of the proposed video position embedding method. The paper is well-organized and easy to follow in general.\n- Weaknesses: There is missing evidence for some of the claims made. Please address the “Claims and Evidence” section. Although the paper is easy to follow in general, there are some errors, typos, and awkward sentences. Furthermore, there are some unclear points in the paper. The presentation quality could be improved to a more professional level.", "other_comments_or_suggestions": "I list a few errors, typos, and awkward sentences below.\n- L214-216, second column: “M-RoPE (Wang et al., 2024b) uses dimensions to model temporal, horizontal, and vertical dimensions sequentially” $\\rightarrow$ “uses dimensions” seems a bit awkward.\n- L271-273, second column: “The visualized relationship between the periodicity, monotonicity, and temporal modeling.” $\\rightarrow$ Not an English sentence.\n- L357, first column: “…different modes that ….” $\\rightarrow$ models", "questions_for_authors": "There are some unclear points in the paper. I list a few below.\n- In Fig. 2 right side, what do the axes mean and each cell color mean?\n- In Fig. 3 of the main paper, and Fig. 9 of the supplementary material, How do we interpret the results? For instance, in L165-176, second column, what does it mean by “the needle is located primarily through vertical positional information”? I do not understand this paragraph. Can the authors elaborate on this paragraph?\n- In L298-303, first column, what does it mean by “creating a stack in the bottom-left corner”? I do not understand this paragraph as well. Can the authors elaborate on this paragraph?\n- Can the authors give more intuitive explanation on Fig. 6?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received three accept recommendations and one strong accept, reflecting a positive consensus among reviewers. The work is well-written, with a clearly motivated problem and a methodologically sound approach to video position embedding. Empirical results show consistent improvements across multiple tasks, supporting the effectiveness of the proposed method. While the paper is generally clear and well-organized, reviewers noted minor issues related to typos, awkward phrasing, and a few claims lacking sufficient supporting evidence. The authors are advised to address these presentation and clarity concerns in the final version to strengthen the overall impact of the paper.
Decision: Accept (oral)