HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models
Authors: Haoran Li
Yingjie Qin
Baoyuan Ou
Lai Xu
Ruiwen Xu
Venue: NeurIPS 2025 poster ()
Keywords: Long Context
Position Embedding
Vision-Language Models
Transformers
Primary area: deep_learning
TL;DR: We propose a novel hybrid of position embedding to improve the length generalization ability of Vision-Language Models.

Abstract: Vision-Language Models (VLMs) have made significant progress in multimodal tasks. However, their performance often deteriorates in long-context scenarios, particularly long videos. While Rotary Position Embedding (RoPE) has been widely adopted for length generalization in Large Language Models (LLMs), extending vanilla RoPE to capture the intricate spatial-temporal dependencies in videos remains an unsolved challenge. Existing methods typically allocate different frequencies within RoPE to encode 3D positional information. However, these allocation strategies mainly rely on heuristics, lacking in-depth theoretical analysis. In this paper, we first study how different allocation strategies impact the long-context capabilities of VLMs. Our analysis reveals that current multimodal RoPEs fail to reliably capture semantic similarities over extended contexts. To address this issue, we propose HoPE, a Hybrid of Position Embedding designed to improve the long-context capabilities of VLMs. HoPE introduces a hybrid frequency allocation strategy for reliable semantic modeling over arbitrarily long contexts, and a dynamic temporal scaling mechanism to facilitate robust learning and flexible inference across diverse context lengths. Extensive experiments across four video benchmarks on long video understanding and retrieval tasks demonstrate that HoPE consistently outperforms existing methods, confirming its effectiveness.

HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models
Haoran Li Yingjie Qin
Baoyuan Ou Lai Xu² Ruiwen Xu²
Carnneie Mellon University "Xiaohongshu Inc.
haoran14@cs.
www.4400ssccue.e.m cmu. edu
Abstract
Vision-Language Models (VLMs) have made significant progress in multimodal tasks. However, their performance often deteriorates in long-context scenarios, particularly long videos. While Rotary Position Embedding (RoPE) has been widely adopted for length generalization in Large Language Models (LLMs), extending vanilla RoPE to capture the intricate spatial-temporal dependencies in videos remains an unsolved challenge. Existing methods typically allocate different frequencies within RoPE to encode 3D positional information. However, these allocation strategies mainly rely on heuristics, lacking in-depth theoretical analysis. In this paper, we first study how different allocation strategies impact the long-context capabilities of VLMs. Our analysis reveals that current multimodal RoPEs fail to reliably capture semantic similarities over extended contexts. To address this issue, we propose HoPE, Hybrid of Position Embedding designed to improve the long-context capabilities of VLMs. HoPE introduces hybrid frequency allocation strategy for reliable semantic modeling over arbitrarily long contexts, and dynamic temporal scaling mechanism to facilitate robust learning and flexible inference across diverse context lengths. Extensive experiments across four video benchmarks on long video understanding and retrieval tasks demonstrate that HoPE consistently outperforms cxisting methods, confirming its effectiveness. Our code is available at https: /tt///gtttttttt thub. ctm.b..com/iii hom//rrics//Ho//
1 Introduction
Vision- Language Models (VLMs) [1-5] have achieved remarkable success in multimodal tasks, including visual question answering [6-9], image captioning [10, 11], cross-modal retrieval [12, 13], and more [14-16]. However, VLMs suffer from significant performance degradation in long-context scenarios, particularly long videos [17-20]. In such settings, VLMS even struggle with simple tasks like object counting and temporal localization [21, 22], revealing critical limitation in their ability to model extended spatial-temports dependencies. This limitation substantially hinders their real- world deployment, where input length often exceeds the context window they have been pretrained
Rotary Position Embedding (RoPE) [23] has been widely adopted for length generalization in text- based LLMs [24-26]. Specifically, RoPE incorporates positional information by partitioning the query and key vectors into 2-dimensional pairs and rotating each pair at a unique frequency that decreases as the dimensional index increases. Despite its advantages, directly applying ID RoPE fails to capture the intricate spatial-temporal dependencies in videos. Several methods have been proposed to extend ID RoPE for multimodal inputs [2. 27, 28]. Among these, the most common approach is to allocate different frequencics to encode different positional components, as shown in the upper plots of Figure 1. For example, M-RoPE [2] allocates the highest frequencies for temporal modeling (t), and the remaining low frequencies for spatial modeling (x, y). In contrast, VideoRoPE [28] proposes allocate the lowest frequencies for temporal dimensions (t), and further
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Vanilla ROPE
(b) MoPE
(c) VideoRoPE
(d) HoPE (ours)
Temporal (t)
Horizontal (x)
Vertical (y)
Identity
T
T
2x
T
V
(a) & (b) No Scaling
(c) Fixed Scaling
(d) : Dynamic Scaling (ours)
Figure 1: Comparison of our HoPE and existing methods. Upper plots illustrate the frequency allocation strategies in different RoPE variants. Here, frequency decreases along the diagonal. (d) HoPE sets the lowest frequencies demonstrate different temporal scaling mechanisms. (d) HoPE proposes dynamic and bidirectional scaling learn temporal dynamics at multiple scales, facilitating robustness to various video speeds.
for reliable long-range semantic modeling. Lower plots
applics a fixed scaling factor to scale the temporal indices of visual tokens, as shown in the lower plots of Figure 1. Despite their improved performance, two significant challenges remain unsolved. Firstly, current methods mainly rely on heuristics rather than theoretical analysis to determine the frequency allocation strategy. Second, applying fixed and unidirectional scaling factor for all videos suboptimal in real-world scenarios, where videos proceed at different speeds and demonstrate significant variance information densities.
To address these challenges, we begin with an in-depth theoretical analysis in Section 3, outlining ideal properties that multimodal RoPE should possess. Our analysis reveals that: (1) the flattening operation vanilla RoPE inherently violates spatial- temporal locality prior, which is crucial in video modeling; (2) despite diverse frequency allocation strategies, existing multimodal RoPE variants fail to reliably capture semantic similarities over extended contexts; (3) temporal scaling of visual tokens should include both compression and expansion accommodate varying video speeds in real- world scenarios. Guided by these insights, we propose HoPE, a Hybrid of Position Embedding designed to improve the bong-context capabilities of VLMs. As shown in Figure 1, HoPE first introduces hybrid frequency allocation strategy to facilitate long-range semantic modeling. In this strategy, higher frequencies, which more sensitive positional differences and better at capturing local features, are allocated to spatial components (x, y) in an interleaved manner. Meanwhile, the lower frequencies are directly set to zero and allocated temporal component (t) enable reliable semantic modeling. Moreover, HoPE develops dynamic temporal scaling mechanism for lengthy visual tokens. This mechanism only enhances VLMs' robustness to various video speeds, which are common in real-world scenarios, but also offers flexible scaling during inference across diverse context lengths.
We summarize contributions follows:
To our best knowledge, we provide the first theoretical analysis how different frequency allocation strategies in multimodal RoPEs impact the long- long-context capabilities of VLMs, offering insights for the design and analysis of future multimodal RoPEs.
Guided by analysis, propose HoPE, which consists of hybrid frequency allocation strategy for reliable semantic modeling in long-context scenarios, and a dynamic temporal scaling mechanism for robust and flexible temporal comprehension.
Extensive experiments on four video benchmarks demonstrate that HoPE consistently outperforms existing RoPE variants, achieving improvements of 22.23% and 8.35% on long video retrieval and long video understanding tasks, confirming its effectiveness.

---

2 Preliminaries
Rotary Position Embedding (RoPE). Current Transformer- based LLMs rely on Positional En- codings (PEs) to incorporate sequential information into the attention mechanism. Among various PEs, Rotary Position Embedding (RoPE) [23] has emerged as a dominant approach for long-context modeling in text-based LLMs [24-26]. The key to RoPE's success lies in its ability to encode relative position information through an absolute positional encoding approach, ensuring both effectiveness and efficiency. Consider query and key vectors with d dimensions (where d is
number), RoPE partitions the dimensions into d/2 pairs, e.g., 9n [q 9n :............................. 9 (d/2-1)) Each pair of dimensions assigned unique rotation angle 0; = b-2i/d E {0, 1,.., .d/2
1}, where is pre-defined frequency base and set to 10, 000 by default [23]. This rotation can be achieved through a rotation matrix as follows:
( /cos0, sin 0 sin 8 cos 0;
The overall rotation matrix R, is constructed by concatenating each pair's rotation matrix along the diagonal to form block-diagonal matrix, i.e., R, diag(r(0),
r(1/2-1)) E Rdxd Therefore, during attention computation, the attention score! An,m between the n-th query q, and m-th key km is:
An,m (q, (km.mm) = qn Rn-kk
(2)
where the rotation matrix Ra can be formulated as:
cos (n m) sin 0o(n m)
sin Oo (n - m) cos (n m))
0
Ra-m
cos 0(4/2-1) (nc - m)) sin 1in0(a/2-11) 1) m))
sin (2/22 (n m. cos (2/22-1) m)
can be observed that through pairwise attention computation, the final rotation matrix naturally incorporates the relative position information (n m) between the query-key pair.
No Positional Encoding (NoPE). Despite the popularity of RoPE, several works have pointed out that the causal attention mechanism in current decoder-only LLMs implicitly learns absolute positional information [30-32]. This motivates the development of No Positional Encoding (NoPE). Specifically, the causal attention mask enforces An,m 0 for all m >> ensuring that each token only attends to itself and previous tokens. Under this constraint, the attention score with NoPE is simple dot product between the query vector 9n and the key vector km. i.e., An,m k providing no explicit positional information to Transformers.
Analysis
In this section, conduct comprehensive theoretical analysis of multimodal RoPE variants, aiming to answer the following questions: (1) Is vanilla RoPE enough for long-context VLMs? (2) How do different frequency allocation strategies impact semantic modeling in long-range multimodal contexts? (3) How should we assign the temporal indices for text and visual tokens?
3.1 Vanilla RoPE Fails in Spatial-Temma Temporal Structure
Several recent VLMs [1, 5, 20, 33-35] still use vanilla RoPE for multimodal inputs. In their approach. each video frame is first encoded by vision encoder (e.g., VIT [36]) and then flattened into sequence of patch-level tokens. These visual tokens will be treated equally as text tokens for
'erre, omit the softmax function and 1/ vd scaling in standard Transformer [29] for simplicity.

---

positional encoding, with each token incorporating only ID temporal information. We show in Proposition 3.1 that this approach, while easy implement, distorts spatial temporal localities and fundamentally limits VLMSs ability model extended spatial-temporal dependencies.
Proposition 3.1 (1D RoPE violates spatial-temporal locality priors). Given any query q at position (t, x. y) and relative distance of spatial or temporal dimensions, the flattening operation ID RoPE distorts the relative distance with a magnitude dependent on the frame resolution.
We provide the proof in Appendix A.1. This mismatch between positional encoding and the 3D struc- ture of videos creates distorted attention patterns, making it difficult for models learn meaningful spatial-temporal relationships essential for video-related tasks.
Conclusion Directly applying vanilla RoPE multimodal long-conter, inputs inherently fails to capture their complex spatial-tenne temporal dependencies.
3.2 Current Multimodal RoPEs Are Unreliable in Long- Range Semantic Modeling
To capture the spatial-temporal structure multimodal inputs, a recent VLM, Qwen2-VL [2], has introduced Multimodal Rotary Position Embedding (M-RoPE). Concretely, M-RoPE partitions the 128-dimensional rotary embedding into three distinct groups: the first 32 dimensions for temporal
information (f), the subsequent 48 dimensions for horizontal spatial information (x). and the final
48 dimensions for vertical spatial information (y), i.e., Ri,x,y = diag( (R, Rg, Ry). While this approach realizes naive extension for RoPE, fundamental question remains be answered:
How do different frequency allocation strategies impact the performance of multimodal RoPE?
This question arises from the fact that in RoPE, different dimensions carry unique frequencies (0, = 24/d lo....... d/2 1))), as shown in Equation 3. Therefore, different strategies exist for frrquency allocation in multimodal RoPE. As shown in Figure 1, M-RoPE allocates the highest frequencies for t. intermediate frequencies for n, and the lowest frequencies for y. In contrast, VideoRoPE [28] proposes assign the lowest frequencies temporal modeling (f) and high frequencies to spatial dimensions (x. y). Their empirical justification stems from attention pattern analysis, which reveals that dimensions encoded with the lowest frequencies exhibit more pronounced attention sink phenomenon [37]. which has proven to be effective in long-context modeling. However, argue that using the lowest frequencies for temporal modeling is still unreliable in capturing semantic similarities in extended multimodal contexts. Specifically, we first introduce semantic preference, a property where attention mechanisms should prioritize semantically similar tokens regardless of their relative distance, and formally define this concept in Definition 3.1.
Definition 3.1 (Semantic Preference). For any query vector q and a semantically similar key vector k' that can be expressed k' q + where d is zero-mean perturbation, the attention score with RoPE should satisfy:
Eg (RRAsssakk qRaiarsyk 0.
where k is the key vector of a semantically unrelated token. This preference should hold regardless of the relative distance (At, Ax. Ay) between the query and key.
Then, we show in Theorem 3.1 that all frequency allocation strategies of current multimodal RoPEs, including selecting the highest/lowest frequencies for temporal modeling, are unreliable maintain- ing the semantic preference property over extended contexts. This limitation arises because, with sufficiently long contexts, even the lowest frequencies can produce arbitrary rotations, ultimately undermining semantic preference. We provide the proof in Appendix A.2.
Theorem 3.1. Let X [2,, x1 be input sequence, and let RoPE use any fixed set of temporal frequencies (e.g.. highest or lowest). Then there exists a critical length La such that for all L' Lo the semantic preference property (Definition 3.1) is violated.
Conclusion 2. There exist different frequency allocation strategies to extend vanilla RoPE to multimodal RoPE. However, we prove that none of these strategies can reliably maintain the semantic preference property over a sufficiently long context.

---

3.3 How to Assign Positional Index for Multimodal Inputs?
Currently, most VLMs [1-4, 19, 20] adopt the same temporal stride for video frames and text tokens, as shown in Figure However, this approach overlooks the inherent difference in information densities between text and visual tokens. To address this issue, VideoRoPH [28] applies a fixed scaling factor (implemented as 2) adjust the temporal indices of visual tokens, achieving better empirical performance. However, this rigid scaling approach lacks the flexibility needed for diverse real-world videos, which naturally vary in pace and information density. A more ideal approach would incorporate both temporal compression and expansion capabilities, allowing the model to learn multi-scale temporal relationships, thereby enabling more robust temporal modeling.
Conclusion 3. Temporal index scaling of visual tokens is crucial for balancing multimodal information, yet current methods lack flexibility and bidirectionality.
4 HoPE: Hybrid of Position Embedding for Long Context VLMs
To address the above challenges, we propose HoPE, a Hybrid of Position Embedding designed to improve the long-context capability of VLMs. As illustrated in Figure and Figure 2, HoPE first introduces a hybrid frequency allocation (HFA) strategy to better preserve the semantic preference property (Definition 3.1) in long-context modeling. Under this strategy, spatial information will be encoded with higher frequencies to capture local semantics, while the lowest frequencies will be set to zero (as in NoPE [30]) facilitate long-range semantic modeling. Second, HoPE develops dynamic temporal scaling (DTS) mechanism to enhance VLMs' robustness to various video speeds and enable flexible inference under diverse context lengths. We will detail these strategies follows:
4.1 Hybrid Frequency Allocation Strategy
To extend vanilla RoPE to multimodal scenarios, a common approach is allocate different fre- quencies to encode different positional components (t, (t.,,,). 1). For example, M-RoPE [2] assigns the highest frequencies for temporal modeling and lower frequencies for spatial encoding. In contrast, VideoRoPE [28] allocates the lowest frequencies for temporal modeling, achieving better empirical results. However, in Theorem 3.1, we theoretically prove that, despite using lower frequencies being more ideal for semantic modeling, none of these frequency allocation strategies can maintain the ideal semantic preference property (Definition 3.1) over extended contexts.
To provide stronger theoretical guarantee for the semantic preference property, we propose hybrid frequency allocation strategy. As shown in Figure I, encode spatial information (x, y) with high frequencies, as high frequencies are more sensitive to positional differences and thereby better capturing local semanties [28, 38]. Following existing work [28], x and y are encoded in an interleaved manner to prevent biased spatial encoding. More importantly, unlike existing methods [23, 28, 2]. we directly the lowest frequencies to zero (as in NoPE [30]) to provide a stronger guarantee for the semantic preference property (Definition 3.1), as shown in Figure 2. Specifically, for d = 128, we interleave x and y positions in the first 96 dimensions of the rotation matrix and set the frequencies in the remaining 32 dimensions to zero, which corresponds to an identity matrix:
cos 0. sin Ar sin 2r cos drr
0
0
0
Ag
0
sin 0 Ay cos O, A,
Raz,Ay = diag(
.I22)
cos Pao Az 4a
sin 64n Aar coe Aag
0
cos 6 By sin Par
sin Ay con O47
We now provide theoretical analysis of how this hybrid strategy helps the attention mechanism to capture long-range semantic similarities. Building on Definition 3.1 and Theorem 3.1, we first formalize the condition under which semantic preference is preserved in multimodal RoPE.
5

---

sn
(a) High frequencies for temporal (b) Low frequencies for temporal modeling in M-RoPE.
(c) Zero frequencies for temporal modeling in HoPE (ours).
modeling VideoRoPE.
Figure 2: Multimodal RoPEs use different frequencies for temporal modeling. M-RoPE uses the highest frequencies, which are suboptimal for long-context modeling. VideoRoPE utilizes the lowest frequencies for more stable semantic modeling. Our HoPE, employing zero frequencies for temporal modeling, establishes the upper bound of semantic modeling capabilities across all strategies.
In particular, Lemma 4.1 establishes clear theoretical criterion for maintaining semantic preference with multimodal RoPE. It directly follows from our analysis in Theorem 3.1 and Appendix A.2, providing the theoretical foundation for our proposed method.
Lemma 4.1 (Necessary Condition for Semantic Preference). For a multimodal RoPE with rotation matrix R,, diag(R, Rz; R,,), the semantic preference property (Definition 3.1) holds if. for all possible relative distances,
200 (+)+ + Σ 202
Σ 20 20 Dy 0.) 2 0. ici,
IEN
iEL.
where o² the variance of each component in the querykey vector, i, bz by are dimensions H). Ay E 10.1..... W}.
allocated to t, x, and At E (0.1...., 1}. Ar E
Based on this Lemma, we now prove how our hybrid frequency allocation strategy provides stronger guarantees for the semantic preference property over extended contexts. Specifically, HFA set 8. 0 for all dg. Hence, the temporal terms in Lemma 4.1 reduce to Σiei. 2a 2 1, noting that Eier 202 12 Eies, 20 cos(Al 02) holds for any choice of temporal frequencies 0,, Adding the identical spatial terms both sides, we obtain:
2200 . 200 cos Ar 0,) + Σ 20 cos (Ay 0,)
20²
iE4
igka
X
(4)
Σ
cos At 0.) + Σ 20 20
20 cos
Az 0.) + Σ 20 cos(Ay 0,).
iEi+
iEiz
iE1y
This shows that our HFA strategy, by setting the lowest frequencies to zero, dominates any other choice of temporal frequencies and provides a stronger guarantee for preserving the semantic preference property under congecontext scenarios, as in Theorem 4.1.
Theorem 4.1. For multimodal position embeddings with dimensions allocated across temporal (i,), and spatial components (iz. setting 0 for all temporal dimensions maximizes the semantic preference guarantee in Definition 3.1, compared to any alternative frequency allocation strategy, particularly under extended context lengths.
Another interesting finding is that, if we set | d/4, i li, d/8 and 0; 0, i, for any context length and spatial size x, y, semantic preference property invariably holds, as Lemma 4.1 reduces to Ed/8-1 d/8-1 22 + cos( Az O2:) + cos(Ay 02++1)) > O. However, the empirical results of this approach are inferior to our proposed HoPE, probably due the decreased number of frequencies for spatial modeling. More discussions provided in Appendix B.3.

---

4.2 Dynamic Temporal Scaling Mechanism
Considering the distinct information densities of text and visual tokens, HoPE introduces a dynamic temporal scaling mechanism that adjusts the temporal strides of visual inputs. Specifically, we first define a set of scaling factors, e.g., I {0.5, 0.75, 1, 1.25, 1.5), which includes both stretching (7 > 1) and compressing (7 < 1) operations. During training, the scaling factor is randomly selected from T and applied to each video. This allows the model to learn temporal relationships at multiple scales, making it more robust to variations in video speed, which are common in real-world scenarios. Consider a multimodal input (tezt, video, text) of length Lt, Lv, and Le respectively. The position indices (t, y) for each token with our dynamic scaling factor y are:
(11,1).
0 0 l It
L + 7(1 Lt),
L +1 L1) 2
Le s S + Lu
(1,z,)) v)
La + 711 L1) h
(5)
(1 - b +
(7 1) + L
4 Ly l Ta + T + The
(7 1) +
Note that for visual tokens (L, S < L + L,), 1- L, indicates the distance of the current frame from the start frame. During inference, scaling factors can be flexibly selected from the set to accommodate videos of different lengths. is worth noting that unlike existing methods, which do not consider temporal scaling for visual tokens [1, 4, 5] or just apply fixed and unidirectional scaling factor for both training and testing [28], our methods not only help the model learn temporal relationships at multiple scales, but also offer flexibility during inference to accommodate various context lengths.
Experiment
In this section, evaluate the performance of HoPE four video benchmarks across long video understanding and long video retrieval tasks, aiming to validate its effectiveness in multimodal long- context modeling. Additionally, we conduct ablation studies investigate the individual contribution of each strategy to overall performance and the interplay between task type, context length, and scaling factor selection.
5.1 Experimental Setups
Implementation Details. We utilize Qwen2-1.5D and Qwen2-7B [39] as the backbone models.
By integrating these models with vision encoders from Qwen2-VL-2BB7B- Instructt [2], we obtain Dwen2-2BB7B--iddee respectively. During training, we adopt batch size of 128, a learning rate of 1e-5(BB)/e--(7BB) with cosine scheduler. Following the instruction tuning settings in Qwen2- VL [2]. we set the maximum video frames to 128 and the video sampling rate to 2. The training context length is set 8k, with the entire training process taking approximately 304 GPU hours on machines equipped with H800-80GB GPUs. During evaluation, the minimum tokens per frame are set to 144.
Training Data. We train the models on subset of LLaVA- ILLVV-VVdeo-117 178k [40], which consists of 178k videos ranging from 0 to 3 minutes and 5M instruction samples, including captions, free-form, and multiple-choice question answering. Our selected subset includes 30k videos with durations under minutes and 3k videos with durations between 2 and 3 minutes, resulting in roughly 300k pairs.
Baselines. We compare HoPE with the following RoPE variants: 1) vanilla RoPE [23], the standard approach in long-context LLMs, 2) M-RoPE [2], a famous RoPE extension in D0we2--VL for multimodal inputs, 3) VideoRoPE [28], specialized RoPE variant designed for video-related tasks.
Evaluation Benchmarks. We evaluate HoPE across four video benchmarks for long video under- standing and long video retrieval tasks. For long video understanding. we utilize Long VideoBench [41], Video-MME [42], and MIVU [43], covering videos ranging from a few seconds to 2 hours. For long video retrieval, we employ V-NIAH (Visual Needle- A-Haystack) [17]. In this task,
7

---

Table I: Performance comparison on long video understanding benchmarks. The training context length of all methods is set to 8k, and we report the performance on 8k, 16k, 32k, and 64k to evaluate length generalization. The best results are bold, while the second best results are underlined.
MLVU
Long VideoBench
Video-MME
Method
16k
32k
64k
8k
16k
32k
64k
8k
16k
32k
64k
Qwen2-2B- Video
Vanilla RoPE
55.10
55.21
54.36
39.06
51.57
50.29
51.00
34.21
50.70
51.48
51.44
20.31
M-RoPE
53.26
53.69
54.73
40.63
50.81
52.26
51.30
44.74
51.44
51.22
51.52
23.44
VidcoRoPE
54.75
55.19
54.00
42.19
52.17
52.02
51.31
36.84
50.89
50.52
50.56
15.63
HoPE (Ours)
54.89
56,36
55.70
45.12
52.31
52.97
51.66
46.27
51.79
51.87
51.69
26.03
Qwen2-7B- Video
Vanilla RoPE
59.75
61.13
61.03
34.38
51.17
50.31
51.29
39.47
56.70
57.96
57.99
26.13
M-RoPE
59.70
61.68
62.46
46.88
52.27
53.29
53.49
50.00
56.81
57.77
58.37
23.43
VidcoRoPE
60.40
61.82
62.51
45.31
52.89
53.13
53.82
47.37
57.51
59.00
59.13
26.52
HoPE (Ours)
61.09
63.48
63.85
50.01
54.11
55.09
55.34
51.22
57.74
59.33
59.44
27.34
"needle" image is randomly inserted into "haystack" video, and the VLM is required to answer question specifically about the embedded "needle" image. Following the protocol V-NIAH [17]. utilize haystack video with 1-hour duration (3,000 frames) and insert the needle image at 20% depth intervals (e.g., a frame depth of 0% would place the needle image at the very beginning of the video). For more detailed benchmark descriptions, please refer to Appendix B.1.
5.2 Results on Long Video Understanding
In this section, we provide comprehensive comparison HoPE and different RoPE variants in long video understanding. From Table 1, we observe that: (1) HoPE consistently outperforms all baselines across nearly benchmarks, context lengths, and backbone sizes. Specifically, under the 7B model scale and 32k context lengths, HoPE surpasses vanilla RoPE by 2.82, 4.05, and 1,45 on MLVU, Veng,, and Video-MME, respectively. This confirms its effectiveness and generalizability in multimodal long-context modeling. (2) The effectiveness of HoPE scales with backbone size. For instance, when the size of the backbone LLM increases from 2B to 7B, HoPE's performance gain on Long VideoBench (32k) significantly increases from 0.66 to 4.05 compared vanilla RoPE. Notably, the performance gap between different methods on the 2B scale is less significant, probably due to the limited capabilities of the backbone LLM. (3) For context lengths under 64k, performance Vidco-MME drops substantially, while the impact on MLVU and Long VideoBench is less pronounced. This suggests that extrapolating extreme context lengths (e.g.. up to 8x) remains highly challenging.
5.3 Results on Long Video Retrieval
evaluate HoPE against other RoPE variants on V-NIAH [17] to demonstrate the superiority of our method in long video retrieval, where VLMs are required to identify specific frames in a video to answer the question. Figure demonstrates that multimodal RoPEs significantly outperform vanilla RoPE, supporting claim in Proposition 3.1 that the flattening operation in vanilla RoPE hinders spatial-tempot modeling. Furthermore, HoPE achieves better extrapolation than M-RoPE and VideoRoPE, confirming its effectiveness in multimodal long-context modeling. Quantitative results in Table show that HoPE surpasses the hest baseline by significant margin of 22.23%.
5.4 Analysis
In this section, we first conduct ablation studies to analyze the effectiveness of each component in HoPE. We then present comprehensive analysis exploring how different factors, including task type, context length, and the scaling factor of visual tokens, interact and impact model performance.

---

(a) Vanilla RoPE
(b) M-RoPE
0.0
Depth Percent
20.0
40.0
0.5
60.0
80.0
100.0
100
500
1300
1700
2100
2550
2800
100
608
1350
1700
2100
2500
2800
Num. Frames
Num. Frames
(c) VidcoRoPE
(d) HoPE(ours)
Depth Porcent
0.2
100
550
600
1300
1700
2100
2500
2008
100
1900
1700
2100
2500
2000
Num. Frames
Num. Frames
Figure 3: Performance comparison on long video retrieval task (V-NIAH). Here, each frame
corresponds to 144 tokens. Cell colors indicate model accuracy (red: low, green: high), and the black dotted line marks the training context length (8k).
Ablation Studies. We conduct series of ablation experiments to evaluate the impact of each component in HoPE and summarize the results in Table 2. According to the results, we observe that
(1) The 3D structure effectively improves the per- formance of vanilla RoPE in multimodal contexts, supporting our Proposition 3.1. (2) Based on the 3D structure, the hybrid frequency allocation (HFA) strategy further enhances long-range semantic mod- eling, achieving an average improvement of 1.69 across all context lengths. (3) The dynamic tem- poral scaling (DTS) mechanism facilitates VLMs' robustness to varying video speeds in real-world scenarios, yielding further performance gain. By combining the above strategies, our HoPE achieves
Table 2: Ablation results on Video-MME
from 8k to 64k. Here, HFA: hybrid frequency allocation, DTS: dynamic temporal scaling.
Method
8k
16k
32k
64k
Vanilla RoPE
56.70
57.96
57.99
26.13
+ 3D structure
56.81
57.77
58.37
23.43
3D IIFA
57.66
59.19
59.31
26.98
+ +DD HIA + DIS
57.74
59.33
59.44
27.34
the best overall performance across different text lengths in multimodal long-context modeling.
Impact of Test-time Scaling Factor Se- lection. We conduct further experiments to investigate how different scaling factors in our dynamic temporal scaling mech- anism impact the performance of video- related tasks. We summarize the results on V-NIAH and Long Beng in Ta- ble 3. Our main observations are as fol- lows: (1) Long video retrieval generally prefers smaller scaling factors. As shown in Table 3, when utilize smaller scaling factors during inference, the performance on V-NIAH improves. We attribute this to the substantial length of I-hour videos
Table 3: Ablation studies on test-time scaling factor selection. We find that long video understanding gen- erally benefits from larger scaling factors, while long video retrieval yields better results with smaller ones.
Scaling Factor ry
Long VideoBench
V-NIAH
16k
32k
64k
0.50
54.48
54.29
54.36
52.63
60.89
0.75
1.00
54.11
54.48
54.97
52.63
62.67
1.25
54.11
54.84
55.70
52.63
62.67
1.50
54.11
55.09
55.34
51.22
61.78
(3,000 frames), which far exceeds the training length (128 frames). In such cases, smaller scaling factors indirectly prevent the spatial position indices from becoming excessively large (see Equa- tion 5), thereby providing a better guarantee for the semantic preference property. Therefore, we set 0.75 for long video retrieval. (2) Long video understanding generally benefits from larger scaling factors. contrast to retrieval, we find that long video understanding is relatively insensitive choice of scaling factor when the input context length is close to the training length. However, as the input length increases, employing larger scaling factors (7 > 1) results in better performance. We hypothesize that while smaller scaling factors help preserve the semantic preference property, larger

---

scaling factors are beneficial for maintaining spatial details (also see Equation 5), which are crucial for complex understanding tasks. This introduces natural tradeoff between semantic preference and spatial detail preservation. Compared to long video retrieval (3,000 frames, roughly 432k tokens), where extended temporal distances can significantly degrade semantic preference, in long video understanding tasks with context lengths of 16k-32k, the negative impact on semantic preference is relatively small. At the same time, the positive effect of larger scaling factors on capturing spatial details outweighs the semantic preference loss, making larger scaling factors overall more effective for complex video understanding. In our experiments, we set y = 1.5 for long video understanding.
6 Related Work
Position Embedding in LLMs. Rotary Position Embedding (RoPE) [23] has become a common choice for position embedding in modern LLMs [24-26, 44]. As discussed in Section 2, RoPE achieves this success through rotating query and key vectors, encoding relative position information through an absolute positional encoding approach. Despite its success, several works have pointed out that No Position Embedding (NoPE) still works for decoder-only LLMs, arguing that the causal attention mechanism implicitly learns absolute position information [30, 31, 38). These works even suggest that NoPE outperforms RoPE in out-of-distribution (OOD) scenarios. However, this observation remains unexplored in multimodal settings, where positional encoding strategies may have different implications for pross-modal interactions. Based on Lemma 4.1, we find that incorporating NoPE's zero frequency strategy indeed improves the length generalization of multimodal RoPE.
Multimodal Position Embedding in VLMs. In VLMs [1-5]. images are first processed by vision encoders and then flattened into ID tokens. Several early models [1. 4, 5] rely on vanilla RoPE for positional encoding, which distorts spatial-tempore locality (see Section 3) and limits VLMs' long-context capability. Recently, Qwen2-VL [2] introduced M-RoPE, which extends ID RoPE to multimodal settings by assigning distinct frequency ranges to different positional components. Specifically, M-RoPE allocates the highest frequencies to the temporal component i, while distributing the lower frequencies sequentially to the spatial components x and y. Conversely, VideoRoPE [28] allocates the lowest frequencies to to capture long-range dependencies, achieving stronger length generalization. However, these allocation strategies mainly rely on heuristics, lacking in-depth theoretical analysis. contrast, our work theoretically analyzes how different frequency allocation strategies impact the performance of multimodal RoPE. By zeroing out low frequencies for temporal modeling, our proposed HoPE provides the strongest theoretical guarantee for long-range semantic modeling. HoPE's strength is further enhanced by its dynamic temporal scaling of visual tokens, which enables robust temporal learning during training and flexible scaling during inference. By integrating these advantages, HoPE achieves sfateoof-the-ar performance in long video understanding and retrieval tasks, making it well-suited for long context VLMs.
7 Conclusion
This paper theoretically analyzes the limitations of current multimodal RoPE variants. Our analysis reveals that: (1) vanilla RoPE inherently fails in spatial-temporal modeling; (2) keeping all frequencies in multimodal RoPE is unreliable in capturing long-range semantic similarities; (3) temporal scaling of lengthy visual tokens should include both compression and expansion to accommodate various video speeds. Consequently, we introduce HoPF, a hybrid of position embedding designed to enhance the long-context capabilities of VLMs. HoPE proposes a hybrid frequency allocation strategy to facilitate long-range semantic modeling, and dynamic temporal scaling mechanism to enhance VLMs' robustness to varying video speeds in real-world scenarios. Experimental results on long video understanding and long video retrieval tasks demonstrate that HoPE consistently outperforms existing methods across diverse context lengths and backbone sizes, confirming its effectiveness.
Limitations. While HoPE's performance gains scale from 2B to 7B backbones, our work does not use larger models or training data. We observe that the performance all methods degrades significantly evaluations are
at 64k, though HoPE remains the most robust. While these resource-constraine essential for uncovering genuine algorithmic benefits of multimodal RoPE, we note that training with more data, particularly long-context data, could further improve length generalization.
10

---

References
[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning, Advances in neural information processing systems, 36:34892-34916, 2023.
[2] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-lane Languge model's perception of the world at any resolution. arXiv preprint MxXiv:240..12199 2024.
[3] Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, al. Internvl: Scaling up vision foundation models and aligning for generic disual-linguise tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2-185-24198 2024.
[4] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-mode large language model with modality collaboration. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 13040-13051, 2024.
[5] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language under- standing. arXiv preprint arXiv:2403.0552. 2024.
[6] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the JEEE/CVF Conference Computer Vision and Patlern Recognition, pages 14455-14465, 2024.
[7] Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang. Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. MMed-RAG: Versatile multimodal RAG system for medical vision language models. In The Thirteenth International Conference on Learming Representations. 2025.
[8] Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1081-1093, 2024.
[9] Peng Xia, Jinglu Wang, Yibo Peng, Kaide Zeng, Xian Wu, Xiangru Tang. Hongtu Zhu, Yun Li, Shujie Liu, Yan Lu, et al. Mmedagent-rl: Optimizing multi-agent collaboration for multimodal medical reasoning. arXiv preprint arXiv:2506. 000.6.005555000 2025.
[10] Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. Advances in Neural Information Processing Systems, 36:40924 2023.
[11] Junjie Fei, Teng Wang, Jinrui Zhang, Zhenyu He, Chengjie Wang, and Feng Zheng. Transferable decoding with visual entities for zero-shot image captioning. In Proceedings the international conference on computer vision, pages 3136-3146, 2023.
the IEEE/CVF
[12] Yongqi Li, Wenjie Wang, Leigang Qu, Liqiang Nie, Wenjie Li, and Tat-Seng Chua. Generative cross-modal retrieval: Memorizing images in multimodal language models for retrieval and beyond. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 11851-11861. 2024.
[13] Yabing Wang, Le Wang, Qiang Zhou, Zhibin Wang, Hao Li, Gang Hua, and Wei Tang. Mul- timodal IIm enhanced cross-lingual cross-modal retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 8296-8305, 2024.
[14] Feng Wang, Manling Li, Xudong Lin, Hairong Lv, Alexander G Schwing, and Heng Learning to decompose visual features with latent textual prompts. ICLR, 2022.
11

---

[15] Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision- language inference. In European Conference on Computer Vision, pages 315-332. Springer, 2024.
[16] Feng Wang, Yaodong Yu, Guoyizhe Wei, Wei Shao, Yuyin Zhou, Alan Yuille, and Cihang Xie. Scaling laws in patchification: An image is worth 50,176 tokens and more. ICML, 2025.
[17] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision. arXiv preprini arXiv:2406. 16852, 2024.
[18] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472- 19495, 2024.
[19] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.0744 2024.
[20] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
pages 12652, 2024.
[21] Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, et al. Needle in a multimodal haystack. Advances in Neural Information Processing Systems,
2024.
[22] Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tuny Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking long- context capability of multimodal large language models. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics, 2025.
[23] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputting 568:127063, 2024.
[24] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tune chat models. arXiv preprint arXiv:2307.09228: 2023.
[25] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2tt.::239..6609, 2023.
[26] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401 240.008000.04088, 2024.
[27] Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In European Conference on Computer Vision, pages 289-305. Springer, 2024.
(28) Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, et al. Videorope: What makes for good video rotary position embedding? In International Conference on Machine Learning, 2025.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[30] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer I Levy. Transformer language models without positional encodings still learn positional information. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1382-1390, 2022.
12

---

[31] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization transformers. Advances in Neural Information Processing Systems, 00124892-244229 2023.
[32] Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuan-Jing Huang, and Xiaoling Wang. Length generalization of causal transformers without position encoding. In Findings of the Association for Computational Linguistics ACL 2024, pages 14024-14040, 2024.
[33] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction- audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 543-553, 2023.
[34] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971-5984, 2024.
[35] Haoran Li, Junqi Liu, Zexian Wang, Shiyuan Luo, Xiaowei Jia, and Huaxiu Yao. LITE: Model- ing environmental ecosystems with multimodal large language models. In First Conference on Language Modeling, 2024.
[36] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
[37] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024.
[38] Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025.
[39] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15/11 15115, 2024.
[40] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, arXiv preprint arXiv:2410.02713, 2024.
[41] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding. Advances in Neural Information Processing Systems, 37:28828-28885 2024.
[42] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075. 2024.
[43] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.00426 2024.
[44] Mingyu Xu, Xin Men, Bingning Wang, Qingyu Zhang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Base of rope bounds context length. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 87386-87410. Curran Associates, Inc.. 2024.
[45] Sucheng Ren, Daquan Zhou, Shengfeng He, Jiashi Feng, and Xinchao Wang. Shunted self- attention via multi-scale token aggregation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10853-10862 2022.
13

---

[46] Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chiel Chen. Flowar: Scale-wise autoregressive image generation meets flow matching. ICML, 2025.
[47] Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang- Chen. Beyond next-token: Next-x prediction for autoregressive visual generation. In ICCV, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims are summarized in Figure 1. Section 3 provides theoretical analysis and Section 4 offers detailed methodology.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, please see Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide the proofs in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental setups provided in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: We use fully public datasets in our experiments and the details are provided in Section 5.1 and Appendix B.1. Code will be released in camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: // /nipp..ccocc cttps/cccppbbc//ggccceecec/ / ssi ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: In our experiments, the variance between different runs is negligible. Addition- ally, our training and evaluation pipelines are deterministic.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: btttp:///eeurii http:://eerrisppeeee http:/pp....ppb.ccecccceeeeeeeee
Answer: [Yes]
Justification:
We followed the NeurlPS Code of Fthics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses on improving the long-context capabilities of Vision- Language Models, which uses public datasets and is purely academic. We believe has no direct societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not pose safety risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We credited them in appropriate ways and followed their licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are introduced.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not include crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : //tt://is...../e //t/////eeeneee/2225//////////// /LLM) for what should or should be described.
21

---

A Proofs
In this section, we provide detailed proofs for the theoretical statements presented in this paper.
A.1 Vanilla RoPE Fails in Spatial-Temporal Structure
Given any query q position 2., y) and relative distance of I in spatial or temporal dimensions, the flattening operation in 1D RoPE distorts the relative distance with magniude dependent on the frame resolution.
Proposition 3.1.
Proof. Consider a video of shape T И X W, where each token at position (t, y) flattened by
f(t 2) = tHW + xW + y.
Now consider two types of local neighbors:
1. Spatial neighbors within the same frame:
Let (t, v) and (t, x 1, be adjacent in the spatial dimension. Then,
[f + 1,9) f(t, v) 1(( + 1)W - (xW v) W.
(6)
Note that relative distance of in æ becomes W after flattening, which grows linearly with the frame width.
2. Temporal neighbors at the same spatial position: Let (t, and (E + 1, be adjacent in time. Then,
11 - f(1, v) = [(1 1)H xW + (LHW + xW v) HW. (7)
For I-frame shift time, the index difference becomes HW, which grows with spatial resolution.
In both cases, spatially or temporally adjacent tokens are mapped to indices with significant differ- ences. Since vanilla RoPE incorporates positional information based on these ID index differences. such flattening leads to distorted spatial-tempordl relationships.
A.2 Semantic Preference Property
We now prove that the frequency allocation strategies in current multimodal RoPEs are unreliable in capturing semantic similarities over extended contexts, defined in Definition 3.1.
Definition 3.1. (Semantic Preference). For any query vector q and a semantically similar key vector k' that can be expressed as k' q + 8 where is zero-mean perturbation, the attention score with RoPE should satisfy:
(q.k,al qRasssapk] 0,
where k is the key vector of a semantically unrelated token. This preference should hold regardless of the relative distance (Ai, Ax, Ay) between query-key pairs
Firstly, we use Lemma A.1 show why using lower frequencies for temporal modeling more ideal in multimodal RoPE. Intuitively, larger rotation angles (frequencies) are more likely to produce negative cosine similarity values between semantically related tokens under long-context scenarios.
Lemma A.1. Let At be drawn uniformly from (0, 1, ..... L 1}, and define
1 - ( cos(@ 8 At) < 03|.
P-)0)
L
Then for any
1. 1f0 0
I]- then P-(0) =0.
2. For 0 2 2L-1j
P. (()) is non-decreasing in 0.
3. lim P0(
8-000
22

---

then for every Proof of Lemma A.I. 1. No negative region for small 6. If < 0 < 2(-1))
At E {O,..., 13 have
O S AA1 < (I. 8/2.
so cos(@ At) > O. Hence P-(0)= O.
2. Monotonicity once the first zero enters. As soon as 0 2 2[E-1)- the point satisfying 0 At = x/2 lies in L -- 1]. Each further increase in extends the interval of length 0(L 1), adding more half-periods of cosine. Each added half-period contains exactly one "negative" region of length и. Therefore the count {A cos(fA) < 0}| (and hence P-())) can only stay the same or increase,
up O rounding emoors the discrete grid.
3. Limit to one half for large 0. For large 0, the values 10 A become equidistributed mod 2T. Since the negative region { x mod 2x cos x < 0} has total length и over each 2v-cycle, one finds
lim P-(0) = π
01000
2x
2
We prove the frequency allocation strategies in current multimodal RoPE cannot reliably maintain the semantic preference property, i.c., semantically similar tokens should receive higher attention than semantically unrelated pairs.
Theorem 3.1. Let X T2,,. .22....,,11, xT. ) be an input sequence, and let RoPE use any fixed set of temporal frequencies (e.g., highest or lowest). Then there exists critical length Le such that for all LZ Lo the semantic preference property (Definition 3.1) is violated.
Proof. We first recall the definition of multimodal RoPE, where the rotation matrix is partitioned to encode different dimensions:
R = diagggRa (R,, R Ry),
where R., R2 and R. rotation matrices applied temporal, horizontal spatial, and vertical spatial dimensions, respectively, with each dimension carrying a frequency 6, b-2i/d i {O.... d/2 - 1}. Note that the (t, v, y) ordering is purely notational and does not constrain the actual dimension allocation strategy.
Assume that each component of the query vector q is independently and identically distributed with mean μ and variance o². We denote key vector that is semantically similar is zero-mean perturbation. The semantically unrelated key vector is independently drawn with the same distribution as q. Let At, Ar, Ay denote relative temporal and spatial distances between the query and each key. According to Definition 3.1, the semantic preference property requires that:
k' q 5, where
E (..ks|/RRr,Az.aa KT AR
= R...
=EllgRAAAAA 1 Eg.k
EEgl [ -2² D'RAiAAZAA
- + + 000) + Σ 2(2² + o² o²))oos(Ax00; +
2( 2 + +o))oos(Ay)0-- (9)
SEi:
iEiz
24-coost At)0; Σ (opc +
20²coo((Ay)l
vet.
iE
>> 20 cos(Att 0.) +
20 cos(AR Az 0:) + Σ
20 02) 20,
XE
iEky
where ie, iz, by denote dimensions allocated to encode temporal (t), horizontal spatial (x), and
vertical spatial (y) information. To satisfy the semantic preference property (Definition 3.1), the expected attention between query and its semantically similar key should remain higher than that
23

---

for an unrelated key, regardless of their relative distance. This implies the following condition must hold universally:
Σ 20 cos( At. 0.) Σ 20 202cos( (Ar +
Σ 2 Ay 0c) 0,
iEby
(10)
Al E {0, 10,1,.... 1), Ar E 10,1,..... H). Ay 10, W).
Now consider a long-context scenario, where L >> H, W, we can now theoretically prove that why VideoRoPE [28] (using lowest frequencies for is better than M-RoPE (using highest frequencies for !) in maintaining semantic preference property in long contexts. Simply, by Lemma A. we show that when the context length L is sufficiently large, the probability that cos( cos(At At 0.) leads to negative values becomes higher when 0g becomes larger. Therefore, lower frequencies, which rotate less, are less likely violate the semantic preference property.
However, despite using the lowest frequencies, VideoRoPE still fails to guarantee that the semantic preference property holds for all context lengths (Equation 10). Let VideoRoPE allocate only the smallest frequency to the temporal dimensions, instead of [&, smallest frequencies:
aiin -1//d
so that in Equation (10) the temporal sum reduces to:
20
del cos ( 0min).
Here, under the reasonable assumption that semantically related tokens co-occur in nearby spatial positions across frames, the spatial sums in Equation 10 remains non-negative. Thus we only consider the temporal sum in Equation 10:
20 cos(At mmnn).
Now pick any context length L large that there exists
At (0, 1, ..... L with
At 0 0min
E
(
Such A/ indeed exists as soon as min (I. 1)
3, i.e. for any
π
L > L
+1.
2 0 2mnn
For that choice of At, we have
cos( At -minn) < O,
and hence the left-hand side of Equation (10) becomes
20 2
Lel cos(Ar At 6min)
This single counterexample (At, Az, Ay) violates the semantic preference condition, since no further temporal frequencies are available to "rescue" the sum. Therefore, despite using the lowest frequencies for temporal modeling, VideoRoPE still fails to guarantee the semantic preference property. In conclusion, all frequency allocation strategies in current multimodal RoPEs fail to maintain the semantic preference property in Definition 3.1, completing the proof.
B Further Experimental Details
In this section, we provide further details of our experiments, including benchmark descriptions, experimental settings, and further results.
B.1 Detailed Benchmark Description
There a growing interest video generation and understanding [45-47, 34, 19], given their broad applications in content creation and analysis. In this subsection, we provide detailed descriptions of the video benchmarks we used the experiments, i.e., Bide [41], Video-MME [42], MLVU [43], and V-NIAH [17].
24

---

Needle Image
1,000.001
Question: Find the frame of scientist. The scientist is
Bird Elephant Panda
Please provide your answer stating the letter followed the option.
Answer: C
Figure 4: Illustration of V-NIAH, which consists of randomly inserted needle image, haystack video, and a specific question related to the needle.
LongVin LongVideoBench is comprehensive benchmark for evaluating Vision-Language Models on long video understanding tasks. Unlike traditional video benchmarks that focus on short clips under one minute, this dataset features videos ranging from 8 seconds to 1 hour across diverse sources, including everyday life, movies, knowledge, and news. The benchmark encompasses 17 fine-grained question categories organized into two levels: perception and relation. In our experiment, questions that are free from subtitles are retained.
Video-MME a full-spectrum evaluation benchmark of Vision-Language Models in video analysis, spanning 6 primary visual domains with 30 subfields ensure generalizability. features temporal diversity by incorporating both short- (<2 minutes), medium- (4-15 minutes), and long-term videos (30-60 minutes), ranging from seconds to hour.
MLVU a high-quality benchmark designed to evaluate the video understanding capabilities of Vision-Language Models. The temporal duration of videos within MLVU spans from minutes to 2 hours, covering genres such as movies, life records, and egocentric videos. In our experiment, we evaluate all methods on the following multiple-choice tasks: Action Count, Action Order, Topic Reasoning, Ego Reasoning, Needle QA, Plot QA, and Anomaly Recognition.
V-NIAH is challenging benchmark designed evaluate VI.Ms' ability to identify specific frames within long vidcos. this task, a "needle" image is inserted into a "haystack" video, and the VLMs are required to answer specific questions about this "needle" image, shown in Figure 4. Following the settings in V-NIAH [17], we utilize a haystack video with 1-hour duration (3,000 frames). The needle image is inserted at 20% depth intervals (e.g., frame depth of 0% would place the needle image at the very beginning of the video.)
Table 4: Quantitative performance of different RoPE variants on V-NIAH. Here, we report the average accuracy across different context lengths and frame depths.
Vanilla RoPE
M-RoPE
VideoRoPE
HoPE (ours)
V-NIAH
21.00
47.11
52.00
63.56
B.2
Quantitative Results V-NIAH
Here, we provide the quantitative results of different RoPE variants on long video retrieval task in Table 4. can be observed that our HoPE demonstrates 22.23% improvement compared the best baseline, justifying its effectiveness in multimodal long- long-context modeling.
B.3 Ideal Condition for Semantic Preference
As discussed after Theorem 4.1, the semantic preference property (Definition 3.1) invariably holds for any context length and spatial size æ. y when we set [ie| = d/ 44 [izl = | = d/8 and 0; 0, E it since Lemma 4.1 reduces to:
25

---

d/8-1 Σ (2 + cos(Ar. 02.) + cos(Ay 02.+1) 0. 1=0
In our original HoPE implementation, the frequencies allocated to 2; y are 16, 24, 24, respectively, with the lowest 16 frequencies for set to zero. For this proposed variant (HoPE-X), we redistribute these allocations 32, 16, 16 for 1, y, respectively, while setting the lowest 32 frequencies for t to zero. To evaluate the comparative effectiveness of these configurations, we conduct further experiments on LongVideoBench.
Table 5: Performance comparison between HoPE-X and HoPE.
LongVideo
Method
8k
16k
32k
64k
HoPE-X
52.68
52.73
53.01
46.32
HoPE
54.11
55.09
55.34
51.22
Table 5 demonstrates that HoPE consistently outperforms HoPE-X across diverse context lengths. We deduce that the inferior performance of HoPE-X due its decreased dimensions allocated for spatial modeling. While this configuration helps to maintain the semantic preference property, negatively impacts HoPE-X's ability to model local features. Therefore, it is necessary to keep adequate dimensions for spatial modeling in multimodal RoPE.
26


Reviews:
[{"review_id": "KhwOMW3TfU", "reviewer": "Reviewer_4YQx", "summary": "This paper addresses the critical challenge of long-context understanding in Vision-Language Models (VLMs), particularly for video tasks. It argues that existing methods for extending Rotary Position Embedding (RoPE) to the multimodal video domain are based on heuristics and fail to reliably maintain semantic understanding over long contexts.\nThe paper makes two main contributions. First, it provides a theoretical analysis of multimodal RoPE, introducing the concept of \"semantic preference\" and proving that existing frequency allocation strategies are inherently flawed and will fail beyond a certain context length. Second, guided by this analysis, it proposes HoPE, a new RoPE variant with HFA (Hybrid Frequency Allocation) and DTS (Dynamic Temporal Scaling).", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The primary strength of this paper is its rigorous theoretical analysis. Instead of relying on empirical trial-and-error, the authors formalize the desired property of \"semantic preference\" (Definition 3.1) and mathematically prove the limitations of existing methods (Theorem 3.1). This provides a solid, principled foundation for their proposed solution and is a significant contribution to understanding positional embeddings in multimodal contexts.\n2.  The proposed HoPE is a direct and elegant consequence of the theoretical analysis. The idea of setting temporal frequencies to zero is a novel insight, moving beyond the simple \"high vs. low\" frequency debate. The paper provides a clear proof (Theorem 4.1) of why this strategy is optimal for preserving long-range semantic similarity. The addition of Dynamic Temporal Scaling is also a practical and well-motivated enhancement for handling real-world video variance.\n3. The empirical evaluation is thorough and strongly supports the paper's claims.\n\nWeaknesses:\n1. Setting all temporal frequencies to zero is a radical and elegant solution derived from the theory. However, it means the model receives no relative temporal position signal from the RoPE mechanism itself. The paper argues that models can rely on other cues, but this is a significant ablation. This could have unintended side effects. For example, the model might struggle with tasks requiring precise temporal ordering or duration estimation, as it has lost a direct signal for \"how far apart\" two frames are in time.\n2. The DTS mechanism is introduced as a key component for robustness, but its analysis is relatively shallow. The main paper presents the mechanism, but the ablation of scaling factors is deferred to the appendix (Table 4). The results in the appendix show different optimal scaling factors for different tasks (retrieval prefers compression, understanding prefers expansion). This is a very interesting finding that deserves more attention in the main paper. Why does this happen? The paper offers a brief hypothesis (lines 562-571 in the appendix) but doesn't delve deeper. A more thorough analysis of the interplay between task type, context length, and the optimal scaling factor would significantly enhance the paper's contribution. Is there a way to learn the optimal $\\gamma$ per-instance instead of fixing it at inference?\n3. The paper is overall well-written, but some parts are dense. The jump from the formal proofs to the experimental results could be bridged more smoothly. Furthermore, Figure 3 is labeled as a performance comparison on the V-NAIH task, but it appears to be a visualization of attention scores or similarity, not a performance metric. This is confusing and should be clarified.", "was_revised": true, "final_justification": "My main concerns are about the temporal frequencies and hyper-parameter. As these concerns have been addressed, so I have decided to maintain my positive rating.", "rebuttal": "Thank you for your insightful review and suggestions to help us improve our paper! We are glad for your positive assessment of our proposed HoPE, both theoretically and empirically. We will detail our response below and please kindly let us know if our response addresses your concerns.\n\n> **Q1:** Setting all temporal frequencies to zero is a radical and elegant solution derived from the theory. However, it means the model receives no relative temporal position signal from the RoPE mechanism itself. The paper argues that models can rely on other cues, but this is a significant ablation. This could have unintended side effects. For example, the model might struggle with tasks requiring precise temporal ordering or duration estimation, as it has lost a direct signal for \"how far apart\" two frames are in time.\n\n**A1:** Thank you for pointing this out. Although setting all temporal frequencies to zero provides no explicit temporal signal from Position Embedding, the causal attention mechanism implicitly learns absolute positional information, as mentioned in NoPE [1, 2, 3]. Based on your suggestion, we evaluate different methods on tasks requiring precise temporal ordering (E3E, SSS) and long-term semantic preference (SOS, SAA) in LongVideoBench:\n- E3E: Determine the sequential order between two events.\n- SSS: Determine the sequential order among multiple frames.\n- SOS: Identify another scene where object *O* appears again. \n- SAA: Detect attribute changes of a queried object across frames.\n\n**Table R1:** Performance comparison between different methods on E3E, SSS, SOS, SAA tasks in LongVideoBench. We report the averaged results here.\n| Method   | E3E |  SSS    | SOS |  SAA    |\n|--------|------|----------|------|----------|\n| RoPE   | 51.43   |  41.05  | 56.15   |  51.12  |\n| M-RoPE   | 54.46   | 43.68 |  58.91 | 54.87 |\n| VideoRoPE   | **54.97**   | **44.01** | 59.77 |55.21|\n| HoPE (ours)   | 54.23   | 43.11 | **63.27** | **57.35**|\n\nAs shown in Table R1, our approach performs on par with VideoRoPE and M-RoPE in E3E and SSS tasks—even without temporal frequencies—suggesting that temporal information can still be recovered via the causal attention mechanism. More importantly, HoPE yields substantial gains in SOS and SAA, which require VLMs to capture long-term semantic similarities. Given the trade-off and the overall improvement across tasks, we chose this design. We will include this discussion in the appendix. \n\n***\n\n> **Q2:** The DTS mechanism is introduced as a key component for robustness, but its analysis is relatively shallow. The main paper presents the mechanism, but the ablation of scaling factors is deferred to the appendix (Table 4). The results in the appendix show different optimal scaling factors for different tasks (retrieval prefers compression, understanding prefers expansion). This is a very interesting finding that deserves more attention in the main paper. Why does this happen? The paper offers a brief hypothesis (lines 562-571 in the appendix) but doesn't delve deeper. A more thorough analysis of the interplay between task type, context length, and the optimal scaling factor would significantly enhance the paper's contribution. Is there a way to learn the optimal per-instance instead of fixing it at inference?\n\n**A2:** Thank you for your valuable suggestion! We will move the ablation results and findings of our DTS mechanism from appendix to the main paper for more attention. In addition, we would like to point out that, to the best of our knowledge, we are the first to study the interplay between task type, context length, and the optimal scaling factor. Besides our overall finding \"retrieval prefers compression, understanding prefers expansion\", we also point out the insensitivity of VLMs to scaling factors under training context length (8k) and much longer context (64k) in long video understanding, which is uncovered in previous research. We believe our analysis could provide insights for future designs and analysis of multimodal RoPE. Based on your suggestion, we use a trainable three-layer MLP to predict the optimal per-instance scaling factor to explore this possibility. We use the average‑pooled video embedding as input, and the MLP’s output logit is passed through a sigmoid function and then rescaled to the interval [0.5, 1.5]. As shown in Table R2, this strategy can improve HoPE's performance by a small margin, and it is expected since the predicted per-instance scaling factor is more adaptive than a fixed scaling factor. Considering the computation overhead and the limited performance gain, we find the original design in HoPE to be more balanced. We will include this further discussion in the appendix.\n\n**Table R2:** Performance comparison between HoPE and HoPE with predicted scaling factor. We report the results on MLVU and the 7B model scale.\n| Method   | 8k |  16k    | 32k |  64k    |\n|--------|------|----------|------|----------|\n| HoPE   | 61.09 | 63.48  | 63.85 | 50.01|\n| + predicted scaling factor  | 61.37 | 63.59 | 63.94 | 50.23|\n\n***\n  \n\n> **Q3:** The paper is overall well-written, but some parts are dense. The jump from the formal proofs to the experimental results could be bridged more smoothly. Furthermore, Figure 3 is labeled as a performance comparison on the V-NAIH task, but it appears to be a visualization of attention scores or similarity, not a performance metric. This is confusing and should be clarified.\n\n**A3:** Thank you for your advice! We will make the transition between formal proofs and results more smoothly. Apologies for the confusion in Figure 3. Figure 3 is an accuracy heatmap depicting long‑video retrieval performance under the V‑NIAH protocol, not attention scores or similarity. In V-NIAH, we insert a single “needle” image into a “haystack” video at a specified depth, then ask the VLM a question about that image. The horizontal axis represents the total number of frames of the “haystack” video, the vertical axis shows the insertion depth as a percentage of the video length (e.g., (1700, 60.0) means the needle was placed at frame 0.60 × 1700 = 1020), and each cell’s color indicates the model’s accuracy at that coordinate. Overall, more red colors represent lower accuracy, and more green colors represent higher accuracy in the long video retrieval task. We will provide a more detailed description of this task in the main paper.\n\n\n[1] Haviv, A., et al. Transformer Language Models without Positional Encodings Still Learn Positional Information. EMNLP 2022.\n\n[2] Kazemnejad, A., et al. The Impact of Positional Encoding on Length Generalization in Transformers. NeurIPS 2023.\n\n[3] Wang, J., et al. Length Generalization of Causal Transformers without Position Encoding. ACL 2024."}, {"review_id": "jFj8BmFE4B", "reviewer": "Reviewer_o59i", "summary": "This paper analyzes the limitations of current multimodal RoPE variants for spatio-temporal modeling. The authors observe that, in long videos, relative distances in the temporal domain are often much greater than in the spatial domain, which can lead to the loss of semantic properties after rotating features based on frequencies. To address this, the paper proposes setting temporal frequencies to zero, thereby preserving semantic similarity over long temporal spans. Experimental results show improvements in long video understanding benchmarks.", "questions": "Overall, I find more positive aspects than negative ones in this paper. Please refer to the weaknesses section for further details.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. **Addresses a Critical Problem**: The paper tackles the limitation of existing spatio-temporal RoPE modeling to deal with long contexts.\n2. **Rigorous Theoretical Analysis**: A notable strength is the paper's theoretical investigation into why existing spatio-temporal RoPE strategies fail in long contexts, especially regarding maintaining the semantic preference property.\n3. **Shown Effectiveness in Longer Contexts**: Experimental results on long video benchmarks demonstrate that the proposed method effectively handles longer contexts. For example, Table 1 shows that the performance on 32k → 64k is relatively maintained compared to other methods. Furthermore, It is interesting that HoPE (2B) achieves better or comparable performance compared to Vanilla RoPE (7B) on 64k length.\n\n### Weaknesses\n\n1. **Novelty:** While the combination of techniques and their application to multimodal long-context seems novel, some of the individual ideas (like setting frequencies to zero, similar to NoPE) might be seen as extensions of existing concepts in text-based LLM research or prior multimodal work.\n2. **Specific Allocation Might Still Be Empirical:** The idea of setting temporal frequencies to zero and allocating higher frequencies to spatial components makes sense theoretically. However, the exact number of dimensions assigned to each (e.g., 32 for temporal vs. 96 for spatial in the implementation) seems to rely on empirical tuning, rather than being fully derived from theory.", "was_revised": true, "final_justification": "My previous concerns regarding the novelty, empirical selection of hyper-parameters, and experimental setups have been resolved during the rebuttal and discussion phase. I recommend borderline accept.", "rebuttal": "Thank you for your valuable review and feedback to help us improve our paper! We truly appreciate your recognition of our work’s significance, the rigor of our theoretical analysis, and the strength of our empirical results. We detail our response below and please kindly let us know if our response addresses your concerns.\n\n\n> **Q1:** Novelty: While the combination of techniques and their application to multimodal long-context seems novel, some of the individual ideas (like setting frequencies to zero, similar to NoPE) might be seen as extensions of existing concepts in text-based LLM research or prior multimodal work.\n\n**A1:** Our primary contribution is a rigorous theoretical analysis of multimodal RoPEs. Existing works rely on heuristics to determine the frequency allocation strategy in multimodal RoPE. In contrast, we formally define the desired semantic preference property and prove the limitations of prior methods in Theorem 3.1. From this theoretical foundation, setting temporal frequencies to zero emerges as a natural and direct solution, and it has not been considered in previous multimodal RoPEs. Both our mathematical analysis and extensive empirical results confirm that this principled strategy effectively improves VLMs' performance on long‑context tasks, surpassing heuristic approaches. In addition, our Dynamic Temporal Scaling mechanism enhances VLMs' robustness to varying video speeds in real-world scenarios and enables flexible inference selection. As shown in Figure 1, prior works do not distinguish the temporal strides (information densities) of text and visual tokens, or just apply a fixed scaling factor during training and inference. We not only show the effectiveness of DTS in our ablation studies, but also provide the first analysis on the interplay between task type, context length, and the optimal scaling factor in the appendix (lines 562-571 and Table 4). We believe our theoretical analysis, empirical results, and findings could offer insights for future studies.\n\n***\n\n> **Q2:** Specific Allocation Might Still Be Empirical: The idea of setting temporal frequencies to zero and allocating higher frequencies to spatial components makes sense theoretically. However, the exact number of dimensions assigned to each (e.g., 32 for temporal vs. 96 for spatial in the implementation) seems to rely on empirical tuning, rather than being fully derived from theory.\n\n**A2:** Thank you for pointing this out. We would like to clarify that we have explored HoPE-X, a fully theory‑derived allocation strategy (`t:x:y = 64:32:32`; see Lines 206–210, 576–589). Specifically, HoPE-X provably preserves the semantic preference property (Definition 3.1) because Equation 5 remains ≥ 0. Based on your suggestion, we report the performance of HoPE-X and HoPE on subtasks of LongVideoBench that require precise spatial perception (**S2E**, **S2O**) and long-term semantic modeling (**SOS**, **SAA**) in Table R3. \n\n**Table R3:** Performance comparison between HoPE and HoPE-X on tasks requiring precise spatial perception (S2E, S2O) and long-term semantic modeling (SOS, SAA) in LongVideoBench. We report the averaged results here.\n| Method   | S2E |  S2O    | SOS |  SAA    |\n|--------|------|----------|------|----------|\n| HoPE-X |  65.12  | 51.82 | **63.98** | **57.61**|\n| HoPE  |  **67.39**  | **53.02**| 63.27 | 57.35|\n\n\nAs shown in Table R3, HoPE‑X further improves long‑term semantic tasks (**SOS**, **SAA**) but degrades spatial perception tasks (**S2E**, **S2O**) on LongVideoBench. To achieve a better balance between spatial and temporal capabilities, we adopt the `t:(x+y) = 32:96` split in M‑RoPE (Qwen2‑VL‑7B) and VideoRoPE. Besides the current overall performance comparison between HoPE and HoPE-X in Table 5, we will include this more detailed subtask performance in our appendix."}, {"review_id": "fu9eV1voGI", "reviewer": "Reviewer_3U4c", "summary": "This paper analyzes the characteristics of different frequency allocation strategies for multi-modal models in long-context understanding. The key insight is the disadvantages of how temporally high-frequency RoPE can harm the semantic relationship between the queries and keys. Based on this insight, the authors propose assigning the lowest frequency, which eventually turns out to be NoPE, to the temporal dimensions. Additionally, the authors scale the positional embedding for the multi-modal inputs, rather than treating them as the same text tokens. The performance improvement on several long video benchmarks indicates the effectiveness.", "questions": "See weaknesses above", "limitations": "Yes, the authors have discussed the limitations", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n* The paper is overall well written, and the analysis is insightful.\n* The proposition, lemmas, and theorems could provide insights to future researchers on thinking about the architectural modifications to existing models.\n* The improvement is notable on the long video benchmarks with only the modifications to PE.\n\n# Weaknesses\n\nDespite the insights from the paper, I have doubts from the following parts.\n\n1. When looking at the Figure 1, I have a question regarding the original RoPE in Qwen2. The RoPE assignment drawn in Figure 1 seems different from my understanding of Qwen2-VL's multi-modal rotary embedding. Specifically, I was thinking that each dimension of the temporal, height, and width would cover both high and low frequency, instead of each covering a band of the frequencies. I am looking at [the code of Qwen2-VL](https://github.com/huggingface/transformers/blob/2f50230c59ec9f17431236ed6625082cc385c76c/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L153) for this concern. Is my understanding incorrect? \n\n2. The major contribution of this paper is to set the temporal dimension totally identical, into absolutely low frequency. However, the authors might /might not be familiar with NTK-RoPE (Fu et al.), but an observation is that high-frequency RoPE is critical to the quality of text generation. In the case of Qwen, that is the temporal dimension. Therefore, a very necessary experiment is to evaluate the new model with HoPE on more text-dense tasks, such as captioning, to see how well it is. \n\nI would expect the performance to be worse but: (1) if the performance is worse with HoPE, I would suggest the authors clarify this in the paper. But no worry, I am still convinced with the analysis and insights from the authors, I am confident that HoPE would be an important analysis for building better multi-modal RoPE; (2) if the performance is better with HoPE, that's something very surprising, and of course you have an additional advantage to write in the paper.\n\nFu et al. Data engineering for scaling language models to 128k context\n\n3. An analysis of the bad effects of temporal PE (around Line 517) is under the assumption that t will get close to L. Therefore, a meaningful ablation would be not directly reaching the identity mapping, but using some lower-frequency interpolation so that t can at most be, e.g., L / 2. If the authors have limited resources, feel free to skip this point.", "was_revised": true, "final_justification": "After the discussion with the authors, I am happy to maintain my positive score.", "rebuttal": "Thank you for your constructive review and suggestions to help us improve our paper! We are grateful for your recognition of the strength of our writing, theoretical analysis, and empirical results. We detail our response below and would appreciate it if you could let us know if our response addresses your concerns.\n\n\n> **Q1:** When looking at the Figure 1, I have a question regarding the original RoPE in Qwen2. The RoPE assignment drawn in Figure 1 seems different from my understanding of Qwen2-VL's multi-modal rotary embedding. Specifically, I was thinking that each dimension of the temporal, height, and width would cover both high and low frequency, instead of each covering a band of the frequencies. I am looking at the code of Qwen2-VL for this concern. Is my understanding incorrect?\n\n**A1:** Thank you for your thoughtful review. In Qwen2-VL's M-RoPE, each positional component (*t, x, y*) only covers a specific band of frequencies rather than the full spectrum. For example, with `head_dim=128`, *t* uses dims `0–31`, *x* uses `32–79`, and *y* spans `80–127`, and each dimension carries only one frequency. In lines 186-191 in Qwen2-VL's code, `cos/sin` is first sliced according to `mrope_section`, and then (`m[i%3]`) selects the specific frequencies for the corresponding component in (*t, x, y*). This design ensures each positional component uses its designated frequency band.  \n\n***\n\n> **Q2:** The major contribution of this paper is to set the temporal dimension totally identical, into absolutely low frequency. However, the authors might /might not be familiar with NTK-RoPE (Fu et al.), but an observation is that high-frequency RoPE is critical to the quality of text generation. In the case of Qwen, that is the temporal dimension. Therefore, a very necessary experiment is to evaluate the new model with HoPE on more text-dense tasks, such as captioning, to see how well it is.\n\n**A2:** Thank you for your valuable suggestion! As derived from our theoretical analysis, setting temporal frequencies to zero is a natural design to better preserve the long-context semantic preference property (Definition 3.1). Based on your suggestion, we have conducted experiments to evaluate the performance of different multimodal RoPEs on the text-dense captioning task. Specifically, we select two video captioning benchmarks MSR-VTT [1], ActivityNet [2], and one image captioning benchmark COCO [3]. We utilize the widely used CIDEr score [4] as the evaluation metric.\n\n\n**Table R4:** Performance comparison between different methods on the video and image captioning task.\n\n| Method   |  MSR-VTT |  ActivityNet    | COCO |\n|--------|------|----------|------|\n| RoPE   |  24.3 | 11.4 |   121.2 |\n| M-RoPE   |  33.1  | 13.2 | 124.7  |\n| VideoRoPE   | 36.8  | 14.5 | 127.3 |\n| HoPE (ours)  | **38.9**  | **15.8** | **127.5** |\n\nFrom Table R4, we have the following observations: (1) on the image captioning task (COCO), M-RoPE is inferior to VideoRoPE and HoPE, as it uses the highest frequencies for temporal modeling (although $t=1$ here) and lower frequencies for spatial modeling, which are suboptimal in capturing local features. VideoRoPE and HoPE perform on par with each other, since under $t=1$, their spatial allocation strategy is identical. This confirms the insight from [5], and we will include this paper in our reference. (2) On the video captioning benchmarks, HoPE consistently outperforms prior methods. We think these results also support the insight in [5], as high frequencies are more helpful to capture local dynamics during auto-regressive generation. HoPE explicitly leverages this by reserving high-frequency components for better spatial modeling while setting temporal frequencies to zero to ensure reliable long-range semantic modeling, leading to smoother and higher-quality captions.\n\n***\n\n> **Q3:** An analysis of the bad effects of temporal PE (around Line 517) is under the assumption that t will get close to L. Therefore, a meaningful ablation would be not directly reaching the identity mapping, but using some lower-frequency interpolation so that t can at most be, e.g., L / 2. If the authors have limited resources, feel free to skip this point.\n\n**A3:** Thank you for your advice! Based on your suggestion, we have conducted experiments to explore an interpolated variant of HoPE where positions $ t \\leq L/3 $ remain unchanged, while positions $ t > L/3 $ are linearly compressed into the range $[L/3, L/2]$. We think this strategy behaves similarly to windowed attention by prioritizing nearby positions while attenuating the (bad) effects of long-range dependencies in existing multimodal RoPEs. In light of this, we report the results of this variant and HoPE on different duration groups in LongVideoBench.\n\n\n**Table R5:** Performance comparison between HoPE and interpolated HoPE on different duration groups in LongVideoBench.\n\n| Method   | (8s, 15s] |  [15s, 60s)    | [180s, 600s) |  (900s, 3600s]    |\n|--------|------|----------|------|----------|\n| HoPE-interpt | **59.7**  | **64.5** | 50.9 | 47.8 | \n| HoPE  | 58.2  | 63.4 | **53.1** | **51.8** |\n\nAs shown in Table R5, HoPE-interpt demonstrates better performance under short context, while its performance degradation is much more significant than HoPE under longer context. This is expected as HoPE-interpt behaves similarly to windowed attention, discarding long-range dependencies at a specific threshold.\n\n[1] Xu et al. MSR-VTT: A Large Video Description Dataset for Bridging Video and Language. CVPR 2016.\n\n[2] Krishna et al. Dense-Captioning Events in Videos. ICCV 2017.\n\n[3] Lin et al. Microsoft COCO: Common Objects in Context. ECCV 2014.\n\n[4] Vedantam et al. Cider: Consensus-based Image Description Evaluation. CVPR 2015.\n\n[5] Fu et al. Data engineering for scaling language models to 128k context. ICML 2024."}, {"review_id": "1PzubXIm6J", "reviewer": "Reviewer_iTBi", "summary": "This paper focuses on improving the RoPE for long context generalization in VLMs. The authors first analyse existing RoPEs that fail to generalize to long contexts. Then, based on the analysis, the authors propose a novel hybrid RoPE, trying to solve the generalization challenge. Extensive experiments are conducted to evaluate the proposed method.", "questions": "NA.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Theoretical analysis is provided for both the existing RoPE and the proposed RoPE.\n2. The proposed HoPE is interesting and novel.\n\nWeaknesses:\n1. Experiments cannot support the claim. We can see from Table 1, when scaling up from 8k to 64k sequence length, performance drops significantly and consistently across various benchmarks. The generalization problem is not solved at all to me. \n2. I checked the baseline method, VideoRoPE, which was mentioned and compared in this paper. In their paper Table 2., VideoRoPE does not have performance drops (even slightly increasing, which totally makes sense) when scaling the context sequence length.\n3. Potential wrong/unfair evaluation.  I do not know why in this paper, the reported VideoRoPE results drop significantly from 8k to 64k. \n\nVideoRoPE, https://arxiv.org/pdf/2502.05173\n\nGiven the above major concerns, I think this paper is far below neurips standard, even if the theoretical analysis seems interesting.", "was_revised": true, "final_justification": "Thanks the authors for the reply. I am not fully convinced.\n\nFrom the results on 8k - 32k, I find the evidence that HoPE performs better than previous baseline works, even though the experimental setting and results are quite different from the baseline work Video-RoPE.  I will raise my score.\n\nHowever, results on 64k are still a major concern. The long-context generalization ability of HoPE is not fully verified on the 64k setting. Video MME 64k acc is only 27.34%, only slightly better than a random guess. This cannot fully support the claim of solving the long-context generalization issue and outperforming Video-RoPE.\n\nThe theoretical analysis is really interesting and I encourage the authors to align the experiment settings and the results with baseline works like Video-RoPE, which could be more convincing.", "rebuttal": "Thanks for your review. We respectfully disagree with your current comments and assessment of our work. \n\nOverall, the three weaknesses mentioned here point to only one opinion held by the reviewer: because our 64k context length results seem to be different from those in VideoRoPE [1], the reviewer directly concludes that our evaluation is wrong, and even our paper is far below the NeurIPS standard. We would like to point out that: \n\n- **Difference in training data:** The training data used by VideoRoPE [1] and our paper is different. Please refer to Section 5.1 of VideoRoPE and Section 5.1 of our paper. Given this fact, it is unfair to assume that our experiments must yield the same results/trends as VideoRoPE [1].\n- **Existing evidence supporting our observations:** The reviewer holds the view that when increasing the input context length, the performance should increase as reported in VideoRoPE [1]; otherwise, it doesn't make sense on their side. However, the reviewer might not be familiar with current works on long-context VLMs. In Table 5 of LongVideoBench's paper [2], we can observe that VLMs (Idefics2 [3] and Mantis-Idefics2 [4]) with limited capabilities would face performance degradation when the input context is longer. Given the fact that all methods are trained from scratch in our paper with less compute and data compared to powerful open/closed-source models, it is expected that performance degrades under context length (64k) that are much longer than the training length (8k). Moreover, it can be observed that it is **not only** VideoRoPE [1], but all methods in our paper face performance degradation under 64k, with our HoPE being the most robust.\n\nGiven the above facts, it would be a rigid, unsupported expectation that all results must mirror those in prior work (particularly VideoRoPE [1] mentioned by the reviewer here), despite clear experimental differences.\n\nWe would appreciate it if the reviewer would reevaluate our work.\n\n*** \n[1] Wei et al. VideoRoPE: What Makes for Good Video Rotary Position Embedding? ICML 2025.\n\n[2] Wu et al. Longvideobench: A Benchmark for Long-Context Interleaved Video-Language Understanding. NeurIPS 2024.\n\n[3] Laurençon et al. What Matters When Building Vision-Language Models? NeurIPS 2024.\n\n[4] Jiang et al. Mantis: Interleaved Multi-Image Instruction Tuning. TMLR 2024."}]

Author rebuttal:

Meta-review: This work investigates the issue of extending RoPE for multimodal LLMs. The theoretical analysis in this work shows that the existing strategies can be sub-optimal and a better position embedding is proposed according to the analysis. Experiments demonstrate the effectiveness of HoPE on long video tasks. While reviewers had concerns about temporal frequencies, novelty, and experiment settings, most of them were addressed during rebuttal. All reviewers have positive scores after discussion. Please incorporate the comments and suggestions from reviewers into the final version.
Decision: Accept (poster)