HoPE: Hybrid of Position Embedding for Length Generalization in
  Vision-Language Models
Authors: name: Haoran Li; name: Yingjie Qin; name: Baoyuan Ou; name: Lai Xu; name: Ruiwen Xu
Published: 2025-05-26 18:37:40

Abstract: Vision-Language Models (VLMs) have made significant progress in multimodal
tasks. However, their performance often deteriorates in long-context scenarios,
particularly long videos. While Rotary Position Embedding (RoPE) has been
widely adopted for length generalization in Large Language Models (LLMs),
extending vanilla RoPE to capture the intricate spatial-temporal dependencies
in videos remains an unsolved challenge. Existing methods typically allocate
different frequencies within RoPE to encode 3D positional information. However,
these allocation strategies mainly rely on heuristics, lacking in-depth
theoretical analysis. In this paper, we first study how different allocation
strategies impact the long-context capabilities of VLMs. Our analysis reveals
that current multimodal RoPEs fail to reliably capture semantic similarities
over extended contexts. To address this issue, we propose HoPE, a Hybrid of
Position Embedding designed to improve the long-context capabilities of VLMs.
HoPE introduces a hybrid frequency allocation strategy for reliable semantic
modeling over arbitrarily long context, and a dynamic temporal scaling
mechanism to facilitate robust learning and flexible inference across diverse
context lengths. Extensive experiments across four video benchmarks on long
video understanding and retrieval tasks demonstrate that HoPE consistently
outperforms existing methods, confirming its effectiveness. Code is available
at https://github.com/hrlics/HoPE.