VideoRoPE: What Makes for Good Video Rotary Position Embedding?
Authors: name: Xilin Wei; name: Xiaoran Liu; name: Yuhang Zang; name: Xiaoyi Dong; name: Pan Zhang; name: Yuhang Cao; name: Jian Tong; name: Haodong Duan; name: Qipeng Guo; name: Jiaqi Wang; name: Xipeng Qiu; name: Dahua Lin
Published: 2025-02-07 18:56:04

Abstract: While Rotary Position Embedding (RoPE) and its variants are widely adopted
for their long-context capabilities, the extension of the 1D RoPE to video,
with its complex spatio-temporal structure, remains an open challenge. This
work first introduces a comprehensive analysis that identifies four key
characteristics essential for the effective adaptation of RoPE to video, which
have not been fully considered in prior work. As part of our analysis, we
introduce a challenging V-NIAH-D (Visual Needle-In-A-Haystack with Distractors)
task, which adds periodic distractors into V-NIAH. The V-NIAH-D task
demonstrates that previous RoPE variants, lacking appropriate temporal
dimension allocation, are easily misled by distractors. Based on our analysis,
we introduce VideoRoPE, with a 3D structure designed to
preserve spatio-temporal relationships. VideoRoPE features
low-frequency temporal allocation to mitigate periodic oscillations, a
diagonal layout to maintain spatial symmetry, and adjustable
temporal spacing to decouple temporal and spatial indexing. VideoRoPE
consistently surpasses previous RoPE variants, across diverse downstream tasks
such as long video retrieval, video understanding, and video hallucination. Our
code will be available at
https://github.com/Wiselnn570/VideoRoPE{https://github.com/Wiselnn570/VideoRoPE}.