Revisiting Multimodal Positional Encoding in Vision-Language Models
Authors: name: Jie Huang; name: Xuejing Liu; name: Sibo Song; name: Ruibing Hou; name: Hong Chang; name: Junyang Lin; name: Shuai Bai
Published: 2025-10-27 08:00:46

Abstract: Multimodal position encoding is essential for vision-language models, yet
there has been little systematic investigation into multimodal position
encoding. We conduct a comprehensive analysis of multimodal Rotary Positional
Embedding (RoPE) by examining its two core components: position design and
frequency allocation. Through extensive experiments, we identify three key
guidelines: positional coherence, full frequency utilization, and preservation
of textual priors-ensuring unambiguous layout, rich representation, and
faithful transfer from the pre-trained LLM. Based on these insights, we propose
Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and
plug-and-play variants that require no architectural changes. Our methods
consistently outperform existing approaches across diverse benchmarks, with
significant improvements in both general and fine-grained multimodal
understanding. Code will be avaliable at
https://github.com/JJJYmmm/Multimodal-RoPEs.