Revisiting Multimodal Positional Encoding in Vision–Language Models
Authors: Jie Huang
Xuejing Liu
Sibo Song
RuiBing Hou
Hong Chang
Junyang Lin
Shuai Bai
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Models
Multimodal Position Encoding
Primary area: foundation or frontier models, including LLMs
TL;DR: We analyze multimodal RoPE, distill three guidelines, and introduce MHRoPE and MRoPE‑I—plug‑and‑play variants that consistently outperform prior methods.

Abstract: Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) by examining its two core components: position design and frequency allocation. Through extensive experiments, we identify three key guidelines: positional coherence, full frequency utilization, and preservation of textual priors—ensuring unambiguous layout, rich representation, and faithful transfer from the pre-trained LLM. Based on these insights, we propose Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), two simple and plug-and-play variants that require no architectural changes. Our methods consistently outperform existing approaches across diverse benchmarks, with significant improvements in both general and fine-grained multimodal understanding. Code is avaliable at https://github.com/JJJYmmm/Multimodal-RoPEs.

Published as a conference paper at ICLR 2026
REVISITING MULTIMODAL POSITIONAL ENCODING IN VISION-LANGUAGE MODELS
Jie Huang!.28$ Hong Chang²
Xuejing Liu
Sibo Song Shuai Bai
Ruibing Hou
Junyang Lin!
'Qwen Team Alibaba Group.
Institute of Computing Technology, Chinese Academy of Sciences
"Equal contribution *Corresponding author
fyuzheng. ixj, sibo. sab, junyang. Ljy, baishuai, bs balbbbbaa -inc. com [huangjie24s houruibing, changhong @ @ict cn
ABSTRACT
Multimodal position encoding is essential for vision-language models, yet there has been little systematic investigation into multimodal position encoding. We conduct a comprehensive analysis of multimodal Rotary Positional Embedding (RoPE) by examining its two core components: position design and frequency allocation. Through extensive experiments, we identify three key guidelines: positional coherence, full frequency utilization, and preservation of textual pri- ors ors-ensuring unambiguous layout, rich representation, and faithful transfer from the pre-trained LLM. Based these insights, we propose Multi-Head RoPE (MHRoPE) and variants that require architectural changes. Our methods consistently outper- form existing approaches across diverse benchmarks, with significant improve- ments in both general and fine-grained multimodal understanding. Code is avali- able at /tttts:://gttthhe com/ JJJTmmmJ htt//MMuttmmomao RoPEs
(MRoPE-I), two simple and plug-and and--lay
INTRODUCTION
The cormutation-invarian nature the self-attention mechanism requires the use positional en- codings to inform Large Language Models (LLMs) of sequence order, relative distance, and struc- tural dependencies. While early methods relied on absolute position embeddings (Vaswani et al.] 2017). relative encodings-whing standard. Among these, Rotary Position Embedding (RoPE) (Su et al.) 2024) has emerged as the de facto choice in modern L.MSs such Llama (Grattafiori et al. 2024) and Qwen (Yang al. 2025
better generalize to varying sequence lengths have become the
Vision-Languaae Models (VLMs) also require positional encodings that can handle heterogeneous modalities, including ID and 2D/3D visual inputs. Current methods fall into two main cate- gories: ID sequential and multi-dimeesiionl designs. The former, exemplified by vanilla RoPE (Su) al. 2024) and V2PE (Ge al. 2024), flattens and concatenates all inputs into a single sequence. While simple, this approach discards the native visual geometry, leading to significant degradation in performance on tasks requiring visual grounding and spatial reasoning.
Multi-dimeensoonl designs, the second approach, extend RoPE to multiple axes (time, height, width) by partitioning embedding channels. Qwen2-VI. WWang et al. (2024a) adopts Multimodal RoPE (MRoPE) to unify positional encoding for text and visual tokens. However, MRoPE allocates the position embedding into t-h-w chunk, placing the temporal infomation entirely in the high figh-freeqency channels. This bias in temporal encoding harms long-range video modeling. Subsequent work has attempted to improve it, but this has led to a fragmented landscape of highly specialized solutions. Some methods focus exclusively
image understanding (Wang al. 2025), others video com- prehension (Wei et al. 2025 Li et al. 2025 Liu et al. 2025), and third group on image genera- tion (Liao al. 2025
Wu al. 2025 While these models achieve notable performance in their
This work done during research internship Qwen Team, Alihaba Group.

---

Published as a conference paper at ICLR 2026
Table I: Comparison of different RoPE methods.
Position Design
Freq Allocation
Compatible with Text-only RoPE
Method
3D Struct.
Modal Int.
Range
Gran.
Vanilla RoPE (Su et
2024
V2PE (Ge et a.. 2024)
RoPE-ti Su 2024)
MRoPE BBai et 2025
X
CircleRoPE [aag et al.
2025
X
X
X
X
VideoRoPE (Wei al. 2025
X
X
X
IL-RoPE (Liao et al. 2025)
X
Omni-RoPh (Wu 2025
МHRоPE
MRoPE-T
V
X
respective domains, the development of a truly robust and versatile VLM requires a more holistic positional encoding strategy. In this work, aim to develop ing strategy capable of supporting the core, unified capabilities of image and video understanding, complemented by fine-grained visual grounding.
holistic positional encod-
To build more robust multimodal positional encoding, we build on MRoPE and systematically ex- plore three underexplored design: (i) position design-how assign unambiguous, well-separated coordinates to text and visual tokens; and (ii) frequeney allocation- how to distribute rotary fre- quencies across embedding dimensions for each positional axis; (iii) compatibility with text-only ensuring the design defaults to vanilla RoPE for pure text inputs to enable effective trans- fer learning. Table D shows, we systematically compare recent methods across the three design axes and conduct extensive experiments. From this analysis, we identify common pitfalls: modal- ities confusion arising from positional ambiguity; degraded cross-modal fusion due to suboptimal modality intervals; impaired multi-scale modeling from restricted frequency allocations; and com- promised transfer learning caused by incompatibility with text-only RoPE.
Based on experiment, distill three core guidelines for designing robust VLM positional en- codings: (i) positional coherence, requiring unambiguous coordinates with well-defined modality interval; (ii) full frequency allocation, ensuring all positional axes have access the full frequency spectrum; and (iii) preservation of textual priors, keeping the text RoPE identical to the base LLM. To satisfy the guidelines of full frequency allocation, propose simple yet effective methods. Multi-Head RoPE dedicates distinct attention heads to different positional axes to preserve full fre- quency resolution. MRoPE-IIterleeve employs fine-grained, round-robin distribution of channels to ensure each axis is encoded with the full frequency spectrum. Besides, we introduce spatial-ress reset, a novel mechanism that resets the spatial position for visual content. This simple modification was found to significantly facilitate the model's focus to visual information.
Our methods consistently outperform strong baselines across key tasks, including image and video understanding and visual grounding. Our contributions are three-fold: (1) a systematic decom- position multimodal RoPE design; (2) two lightweight instantiations-MMllie MRoPE-Interleave that satisfy the guidelines; and (3) spatial-reset, general-purpose optimization for improved visual information flow.
RoPE and
ANALYSIS OF MULTIMODAL ROTARY POSITION EMBEDDING
This section provides systematic analysis of multimodal RoPE. We begin by revisiting the ba- sics of vanilla RoPE. We then evaluate existing multimodal extensions along three core design axes-position design, frequency allocation and compatibility with text-only RoPE. Through this analytical lens, we identify critical limitations in current approaches, directly motivating our pro- posal of two simple yet effective methods: Multi-Head RoPE and MRoPE--Iteereave
2

---

Published as a conference paper at ICLR 2026
2.1 PRELIMINARIES: VANILLA ROPE
Vanilla RoPE (Su et al. 2024) is pivotal method for encoding positional information in modern LLMss Unlike additive position embeddings, RoPE applies rotational transformation to the query and key vectors, thereby incorporating relative position dependencies directly into the self-attention mechanism. Given a query vector q at position m and a key vector at position n, the attention scores calculated as:
S = = (RRmn) (R,,k) R, R, = R Rn
The transformation R an orthogonal rotation, which causes the score S to depend solely on the relative position n m. This property is achieved by constructing R.m block-diagonal matrix parameterized by the absolute position m and a set of fixed frequencies 0, The rotation frequencies, 0 = base 2i/d for E [0, d/2 1], are set according to geometric sequence. This design creates a spectrum of frequencies ranging from high (for small i) to low (for large i), corresponding each pair of dimensions.
past-growing
Nosition IDs
Modalities
Mase
(a) Vanilla RoPE/ / V2PE
(b) MRoPE
VideoRoPE. HoPE
Attention Sink Mear (t,0,0)
Squeazed Video
Not only
with
Modalities
Focus on ?
(d) CircleRoPE
(c) IL.RRoP// Omni-RoPE
(f) MHRoPE MRoPE-I
Figure 1: Position design of different multimodal RoPE variants. The illustrated example follows an interleaved multimodal sequence: <system prompt>, <video 1>, <text>, <image 1>, <text>, <image 2>, <text>, <image 3>, <text>, <generated text>.
2.2
POSITION DESIGN
This section governs how the positional identifier m is assigned to text/visual tokens.
2.2.1
ID SEQUENTIAL DESIGN.
The most straightforward approach, employed by vanilla RoPE (Su et 2024) and V2PE (Ge et 2024], is to treat the multimodal input as a flattened, one-dimensional sequence. Position indices assigned incrementally, with the position m, of the i-th token defined as m, m4-1 Smod- where mmdd is a step size specific to the token's 's modality. For vanilla RoPE, all modalities are treated uniformly, with
1.
As shown in Figure Ia this design presents two significant drawbacks. First, it discards the inherent 3D structure of visual content, which can alter the apatio-temporal reasoning capabilities of a VLM. Second, position indices can grow exceedingly large in long sequences, negatively affecting the model's extrapolation performance (Wei al. 2025

---

Published as a conference paper at ICLR 2026
To address the issue of large position indices, V2PE (Ge et al. 2024) introduces dynamic posi- tion scaling for visual tokens, setting their step size svisual to a value in 11.1/2..... 1/256). This modification mitigates the rapid growth of position indices and has shown benefits in long video understanding. However, the 3D structure of visual content is still ignored in ID sequential design.
2.2.2 MULTI-DIMENSIONAL DESIGN.
To preserve the native 3D structure of visual content, methods like MRoPE (Wang et al., 2024a) (See Figure lbb extend the scalar position identifier to multi-dimensional tuple. For instance, token's position can be represented as m; = (m,, m. mm"), corresponding to its temporal, vertical, and horizontal axes. MRoPE conceptually treats each visual content (e.g., an image or a set of video frames) as a single, large "cube". The temporal position of the subsequent token is then set by "jumping" past the maximum coordinate value of current block. This is achieved with the update rule:
m = max(m m prev: prev)
This strategy guarantees that no positional overlaps occur between modalities. However, proponents of VideoRoPE: (Li et al. 2025) and HoPE (Li et al. 2025) argue that MRoPE's position design lacks inter-modal symmetry, they introduce a "diagonal layout" by centering the spatial coordinates (see Figure Io.. In this scheme, visual frames are not only stacked along the temporal axis but are also shifted along the vertical and horizontal axes. Despite its theoretical elegance, this diagonal layout introduces a critical flaw: the potential for position id overlap between visual content and generated text tokens. For high-resone nigh-resolution image content like documents, the spatial coordinates of visual tokens can extend into the index range subsequently assigned to the generated text tokens. We identify this positional ambiguity as a source of "modalities confusion in generation", failure mode that manifested as endless text repetition in our later experiments.
CircleRoPE (Wang et al., 2025 arranges image tokens in a circular layout, orthogonal to the linear axis of text positions (see Figure ld). A key property of this design is that it renders all visual tokens equidistant from any given text token, which theoretically promotes uniform attention across the image. However, CircleRoPE's 's design has two limitations. First, the large interval between modali- ties may impede effective modal interaction. Second, lacking temporal axis, it collapses all video frames onto a single ring, which introduces severe temporal ambiguity.
2.2.3 TOWARDS AN OPTIMAL POSITION DESIGN
Our preceding analysis, summarized in the first column of Table I. indicates that robust position design must satisfy several criteria, which we collectively term Positional Coherence: (i) preserve the 3D structure visual content; (ii) maintain a slow growth rate; (iii) avoid modalities confusion in generation; (iv) establish an appropriate modality interval.
While MRoPE fulfills most of these requirements, our analysis uncovers a crucial phenomenon: MRoPE exhibits visual "attention sink", where attention concentrates the top-left corner of each image or video frame, a behavior visualized in Figure 2 Specifically, for the image input from ChartQA, it is duplicated and paired with the prompt "Describe the images in detail" to reveal the attention sink in an interleaved pattern. This phenomenon analogous to the attention sink observed at the initial tokens in large language models. This insight directly motivates our proposal of spatial-reset; a mechanism that resets the spatial dimensions for each visual content. By applying spatial-reset, we aim to align this visual sink with the LLM's bias for small position IDs, accelerating visual adaptation.
Furthermore, spatial-reset reset provides another benefit for video understanding by disentan- gling the representation of motion. Consider an object token at spatial coordinates (hı, w1) at time & and a second token for the same object at (h2, W2) at time f2. Let their absolute posi- tion indices be mi and m2, respectively. Un- der the standard MRoPE formulation, the tem- poral and spatial dimensions are coupled. The absolute positions m. (41, l h1, +
Figure 2: Visual attention sink in MRoPE. Av- erage attention scores to input sequence from the ChartQA (Left) and VideoMME-short (Right).
4

---

Published as a conference paper at ICLR 2026
Wi and m2 (t2,tz h2,, + W2). The resulting relative position indices, mael m2 m1, becomes entangled:
Myel (12 11, (12 11) (h2 h1); (12 (1) (102 wx))
In contrast, our method with spatial-reset decouples these dimensions. The positions are defined as mi (t1, hı, W1) and my (f2, h2, W2). This yields purely spatio-temporal relative vector.
m/a (12 1,, h22
h1.
w2
w1)
This disentangled representation of motion more intuitive and provides a cleaner inductive bias for the model to learn from. Therefore, the position design we adopt for our proposed MHRoPE and MRoPE-I methods builds upon MRoPE by incorporating spatial-reset (as illustrated in Figure Tf
Temporal (T)
Vertical
Horizontal
Chunked Frequeacy
High Freq
Low Freq
Vanilla RoPE VRoPE
MRoPE. / CircleRoPE, OmniRoPP
VideoRoPE HoPE
IL-RoPE
MRoPE-1
MHRoPE
Figure 3: Frequence allocation of different multimodal RoPEs.
2.3 FREQUENCY ALLOCATION
Frequency Allocation governs the assignment of feature channels and their corresponding frequen- cies to the various axes of the position identifier m (temporal vertical h or horizontal w).
2.3.1 FREQUENCY ALLOCATION IN ID ROPE.
In ID methods like vanilla RoPE and V2PE, all feature channels allocated to encode the temporal axis. The frequencies 0g decay as the channel index increases, creating a spectrum from high-frequency (for short-range dependencies) cies). This design also imparts long-range decay property on attention scores, as their upper bound is function of the relative distance, which can be approximated by E |Si+1 where =
low-frequency (for long-range dependen-
S, D el(m=-)0=xsse Appendix D.2 for detailed derivation). V2PE's position scaling for visual
tokens effectively slows this decay, enhancing the model's ability to focus on long visual content.
2.3.2
FREQUENCY ALLOCATION.
The standard MRoPE partitions the d feature dimensions into three contiguous blocks, dedicating one each of the i, h, and axes. As rotational frequencies decrease with channel index, this design forces the temporal axis to be encoded entirely by the highest-frequency channels. This creates a strong inductive bias that is detrimental to long-sequence understanding, rapid decay of attention over time. Furthermore, because the h and w axes are assigned distinct, non-overlapping frequency ranges, they exhibit different long-range decay rates, as visualized in Figure 4a This asymmetry impair the model's ability to learn consistent spatial relationships
it leads
Subsequent methods have attempted to rectify this temporal bias through various frequency re- allocation strategies. VideoRoPE and HoPE, for example, move the temporal axis to occupy the low-frequency channels. IL-RoPE employs a form of interleaving but similarly reserves the lowest- frequency channels for the temporal dimension. While these approaches mitigate the long- context issue for the temporal axis, they introduce a critical, unaddressed trade-off: they force the spatial dimensions into restricted, and often exclusively high-frequency, band. This severely limits
5

---

Published as a conference paper at ICLR 2026
(a) MRoPE
(b) MIIOPPE/ MRoPE-I
Figure 4: The long-range decay property MRoPE, MHRoPE and MRoPE-I.
the model's ability to capture multi-scale spatial relationships, which can impair performance on tasks reliant fine-grained spatial reasoning, such as visual grounding. Furthermore, the very of partitioning feature dimensions inherently coarsens the frequency resolution for each positional axis. The performance implications of this reduced granularity is ender-exploration.
2.3.3 TOWARDS AN OPTIMAL FREQUENCY ALLOCATION
To address the limitations of frequency allocation, we propose two effective strategies, summa- rized in the second column of Table [ Both methods resolve the rapid temporal decay and asym- metric spatial decay of MRoPE, yielding unified decay profile for all axes shown in Figure 4b
Multi-Head Allocation. The first strategy, termed Multi-Head RoPE (MHRoPE), inspired by recent work demonstrating channel-level redundancy in RoPE (e.g.. partial RoPE (Barbero al. 2025 Based on the premise that similar redundancy exists at the attention head level, MHRoPE partitions the positional encoding task among different attention heads as shown in Figure 3 A primary advantage of this strategy is that each axis is encoded using the full frequency spectrum available within its assigned heads. This approach avoids the loss of frequency resolution inherent to channel-splitting methods. Moreover, it may be more scalable. As the number of positional axes grows (Liu al. 2025), partitioning a fixed channel channels (e.g., 128) becomes untenable, whereas dedicating distinct heads to new dimensions offers a far more robust and flexible approach.
Interleaved Allocation. Our second strategy, employed our MRoPE-Interleaved (MRoPE-I) method, distributes feature channels to the temporal (t), vertical (h), and horizontal (w) axes in fine-grained, round-robin manner, as shown in Figure This design ensures that each positional axis is encoded using the full frequency spectrum, from high to low, thereby enabling robust multi-scale modeling for each positional axis. Moreover, the uniform frequency distribution of our interleaved design compatible with extrapolation algorithms like NTK-aware (bloc97 2023) and YaRN (Peng
et al. 2024), which function by rescaling the frequency spectrum (see Appendix D33
2.4
COMPATIBILITY WITH TEXT-ONLY ROPE.
Most VLMs are adapted from LLMs, which typically use vanilla RoPE for positional encoding. This raises a natural question: should the encoding strategy for text tokens in VLMs remain identical to that of the base LLM? While most works implicitly agree, some methods have explored deviations.
From the Position Design aspect, methods like IL-RoPE Liao et al. 2025 and Omni- MoniiRoPEE (Wul al. 2025) modify the text encoding. As shown in Figure Ic| they reset spatial coordinates for each image to aid editing but concurrently set the spatial dimensions for text tokens to zero. This design choice breaks compatibility with the standard RoPE used in the pre-trained L.LM.
From the Frequency Allocation aspect, we also explored a potential modification. Since the coor- dinate range of spatial dimensions is much smaller than that of the temporal axis, smaller rotary 'base" could be used to better encode the spatial dimension. However, our experiments showed
For Group Query Attention (GQA), partition on KV heads and repeat corresponding query heads.

---

Published as a conference paper at ICLR 2026
that this strategy led to poor performance across most benchmarks. This outcome strongly indi- cates the critical importance of maintaining full compatibility with the text-only RoPE for effective knowledge transfer from pre-trained LLMs.
2.5 PROPOSED MULTIMODAL ROPES
Based on our analysis, we propose two novel multimodal RoPE variants: Multi-Head RoPE (MHRoPE) and MRoPP--nterleave (MRoPE-I). Both methods are built upon a shared set of design guidelines for robustness and performance.
For position design, we enhance MRoPE position design with spatial-ress to improve the model's focus on visual information. We also maintain strict compatibility with text-only RoPE, ensuring the effective transfer of pre-trained knowledge.
The key distinction between our variants lies in their frequency allocation strategy. MHRoPE em- ploys Multi-Head Allocation, dedicating distinct attention heads to different axes to preserve fre- quency resolution and offer scalability. In contrast, MRoPE-I uses Interleaved Allocation, fine- grained approach ensuring full-spectrum encoding and compatibility with extrapolation techniques. For detailed discussion the trade-off's between MHRoPE and MRoPE-I, see Appendix D.I
EXPERIMENT
3.1 EXPERIMENTAL SETUP
Training Details. All models use the Qwen ViT and connector from Qwen2.5V1 while initializing the VLM backbone with the Qwen2.5 7B LLM. During training, we freeze the VIT to fix the visual representations and unfreeze the connector and LLM backbone. This strategy is designed to isolate the effects of our proposed RoPE modifications while adhering to the standard VLM adaptation paradigm of building upon pre-trained LLM. Notably, this initialization setting aligns with that of VideoRoPE (Wei et al., 2025) and HoPE (Li et al. 2025), and differs from alternative architectural explorations in VLMs such Apollo (Orr Zohar& & Xial 2024) and CamBrain-1 (Tong al.) 2024).
The training process adopts a batch size 128 and uses the AdamW optimizer with a=0.9, (3=0.98, and weight decay of 0.05. The learning rate follows cosine decay schedule from initial value of 1 x 10 down to a minimum of 3 x 10 Each experiment consumes approximately 512 NVIDIA A100 GPU hours. The training context length is set to 32K, and the rotary base is set to 1000000.
All experiments share identical training data, model architecture, and hyperparameters, with the sole difference being the choice of multimodal RoPE
Training Data and Evaluation Benchmarks. We conduct experiments using approximately 2M high-quality supervised fine-tuning (SFT) samples, covering wide range of visual scenar- ios including image captioning, OCR, visual reasoning, visual grounding, document comprehen- sion, long video understanding and multi-turn dialogue. For evaluation, we adopt more than 20 benchmarks spanning images, videos, and grounding tasks. Specifically, the image bench- marks include MMMU ( Yuue et al.] (202)). MMBench (Liu et al., (2024)), MMStar (Chen et al.] 2024), OCRBench (Liu et al.] 2024b). A12D (Kembhavi et al. 2016). RealWorldQA (X.AI. 2024). DOA (Mathew et al. (2221) TextVQA (Singh et al. (2019), InfoVQA (Mathew et al. 2022
and ChartQA (Masry et al.) 2022). To evaluate multi-image reasoning capabilities, we use the
BLINK benchmark (Fu et al. 2024). The video benchmarks consist of MVBench (Li et al.) 2024), STAR (Wu al., (021). VideoMME (Fu et al., (025). LVBench (Wang et al. 2024b), MLVU (Zhou et al. (2024b), and Charades- (Zhou et al. 2024a). For grounding, we evalu- ate RefCOCO (Kazemzadeh et al. 2014) series.
Evaluation Setup. All results are reported following the official evaluation protocol of Qwen2VI. (Bai et al. 2025). Since all models are trained with 32K context length, we cap the maximum number of visual tokens at 24K and apply "smart resize" to preserve dynamic visual res- olution. For video inputs, we sample frames at 2 FPS by default; when this yields more than 768 frames, we fall back to uniform sampling to retain at most 768 frames.
For fair comparison with prior work which using Qwen2VI., we disable the absolute time encoding used in Owen2.5VI.

---

Published as a conference paper at ICLR 2026
Table 2: Overall performance of multimodal RoPEs variants on various benchmarks, The highest score is shown in bold, while the second-hip highest score is underlined.
Types
Benchmarks
Vanilla RoPE
MRoPE
VideoRoPE
HoPE
CircleRoPE
MH
MRoPE-I
MMMU
50,56
50.22
49.89
49.89
47.22
53.00
$3.22
MMBenchaarg
74.75
74.06
75,95
75.35
74.91
75.04
75.56
MMstar
49.13
49.93
49.60
50.33
47 27.00
49.60
51.13
OCRBench
73.40
72.70
66.20
66.60
70.60
73.40
74.00
A12D
76.20
74,94
74.45
75.45
75.36
Image
ReulworldQA
58,30
57.25
56.21
57.12
56.60
60.52
$7.39
DocVQA
82,94
81.49
60.13
60.12
77.70
81.32
83.72
TextVQA
66.80
65.85
66.58
66.77
65.54
66.49
66.91
InfoVQA
58.85
52,96
37.42
34.80
53.11
$2.01
58,24
CharQQA
56.84
63.56
54.88
55.44
53.72
62.44
62.12
BLINK
36.12
37.93
36.20
35.98
40.88
42.80
44.08
MVBench
$7.05
57.85
56.78
$8.00
$7.10
58.93
$7.05
STAR
$8.07
58.28
57.20
$8.30
57.94
59.48
$7.79
MLVU
64.69
63,26
66.05
64,81
62.69
65.69
65.46
Vidco
VideoMME
58.63
58.22
58.70
59.52
57.70
57.48
58.96
LVBench
38.93
39.22
40.15
40,99
38.80
40,32
40.54
Charades-STI
32.49
32,23
34.21
36.07
32.27
33.56
34.36
RefCOCOual
77.67
78.35
77.95
77.72
79.87
80.94
Re(COCOrest Orsttt
81.37
82.52
80.43
81,60
83.98
83.66
84.55
RefCOCO 1
72.66
72.31
72.62
71.44
74.35
73.20
75.05
Grounding
RefCOCO +yoi
69.16
68.80
68.
69.61
70.19
70.55
71.80
RefCOCO
74.48
75.95
73.14
74.55
76.77
76.88
77.44
RcfCOCO Ct
61.67
59.97
62.59
61.96
61.96
RefCOCOg go!!
75.45
75.86
74.06
74.69
76.10
76.55
77.70
Re(COCOg...
75.40
75.73
73.90
75.45
76.12
76.68
77.34
Image
62.17
61.90
57.03
57.14
60.16
62.92
63.79
Overall
Vidco
51.64
$1.51
52.18
52.95
51.09
$2.58
$2.36
Grounding
73.48
73.69
72.59
73.34
74.96
74.92
75.85
3.2
OVERALL PERFORMANCE
The overall performance of different multimodal RoPEs is presented in Table Both MHRoPE and MRoPE-I achieve consistently better performance across the majority of benchmarks. For instance, MRoPE-I outperforms the vanilla RoPE baseline by significant margin of +2.67% on MMMU, +5.28% on ChartQA, and +3.27% on RefCOCO
The results also reveal that while vanilla RoPE serves competitive baseline, its performance is noticeably impaired on benchmarks that demand fine-grained spatial reasoning, such as ChartQA and the RefCOCO series. This performance gap highlights the fundamental limitations of its flat- tened, ID position design. Vanilla RoPE also suffers from extrapolation, see Appendix D.4
While VideoRoPE and HoPE demonstrate stronger performance video benchmarks, they exhibit anomalous degradation on DocVQA, Info VQA, and ChartQA. We attribute this discrepancy to critical flaw in their position design: the overlap of position indices, which induces confusion be- tween visual and generated text tokens. The ablation study in Table 4 confirms the root cause of the degradation.
Tabferns that this confusion is
The suboptimal designs of MRoPE and CircleRoPE manifest in their performance. MRoPE breaks the full frequency spectrum for each positon axis. Consequently, struggles on tasks demanding specific frequency ranges, such as long-video understanding (MLVU, LV L.Bench), which requires robust low-frequency temporal encoding, and visual grounding (RefCOCO), which benefits from frequerry spatial encoding. Similarly, CircleRoPE introduces a large modality interval and collapses the video positions, results poor video understanding.
In contrast, MHRoPE and MRoPE-I leverage spatial-resel position design, which prevents modali- ties confusion and not introducing an improper modality interval. By providing each positional axis with full frequency spectrum (interleave multi-head allocation). they enable the models to bet- ter capture both fine-grained spatial details thigh-freqeent and long-range temporal dependencies (low- Now-frequencyy, leading their superior overall performance.
8

---

Published as a conference paper at ICLR 2026
3.3 GENERALIZATION ACROSS ARCHITECTURES
Current Vision-Languaal Language Models (VLMs) are converging towards a unified architectural paradigm comprising vision encoder, connector, and a Large Language Model (LLM), Since multimodal positional encodings primarily operate within the LLM backbone, which exhibits minimal structural variation across different VLM families. We hypothesize that our proposed MHRoPE, and MRoPE-] possess strong generalizability.
To empirically validate this hypothesis, we extended our evaluation to Qwen3-VL--H VL-4B- Instruct (QwenVL Team 2025) and Qwen3--L--S--Istrruut Although they belong to the same lineage as Qwen2.5-VL, they feature significant architectural distinctions: (1) the removal of win- dow attention in the vision encoder, (2) the introduction of DeepStack architecture between the connector and the backbone; (3) the application of QK-Norm within the LLM backbone; and (4) specific the 4B variant, the tying of weights between the embedding layer and the I.M head.
The results are presented in Table 3 (with full details in Appendix D.6). Despite these architectural shifts, methods consistently achieve the best performance. Furthermore, previously observed phenomena, such as the performance degradation caused by diagonal layouts, are corroborated in these new experiments, further solidifying the validity and robustness of our approach.
Table 3: Average performance across modalities for RoPE variants Owen3-VL-4B and Qwen3- VL-8B Instruct models. Highest scores are in bold, second-highest are underlined. Grnd: average grounding score RefCOCO benchmarks.
(a) Qwent-WenLccc VL-44B-Instruc
(b) Qwen3- VL-SB-Instrtutc
Method
Image
Video
Grnd
Method
Image
Video
Grnd
RoPE
47.00
49.90
20.77
RoPE
63.35
56.78
70.09
MRоPE
46.67
49.60
23.67
MRoPE
62.95
57.03
70.97
VideoRoPE
42.77
49.86
18.91
VidcoRoPE
60.26
57.75
67.99
HoPE
43.20
50.25
18.74
HoPE
59.96
57.81
70.19
CirclcRoPE
47.36
49.23
18.84
CircleRoPE
63.54
54.88
71.20
МHRоPE
48.22
50.80
25.23
МHRоPE
64.63
57.46
73.03
MRoPE-I
48.82
50.49
27.52
MRoPE-I
64.82
57.64
75.46
3.4
ABLATION STUDY
This section presents ablation studies on key design choices for our robust multimodal RoPEs. Ad- ditional results are provided Appendix D.5
3.4.1 ABLATION STUDY ON POSITION DESIGN
We previously argued that an optimal position design should: (1) incorporate 3D structure to capture native spatio-temporal information, (2) maintain proper modality interval, and (3) preserve com- patibility with text-only RoPE. To systematically dissect the impact of these factors, we conduct an ablation study, fixing the frequency allocation strategy to our interleaved allocation while varying the position design. The results are presented in Table 4 Simply introducing a 3D structure over the vanilla RoPE provides a notable boost to grounding performance. The addition of spatial-reset mechanism yields substantial gains across all benchmark categories, confirming its effectiveness. We also ablated other position designs proposed prior work, as shown Table 4
Diagonal Layout: Implementing the diagonal layout from VideoRoPE leads to a severe degradation in performance on document-centric benchmarks (Qo, InfoVQA and ChartQA). A qualitative analysis reveals specific failure mode: repetitive, nonsensical text generation (c.g.. "1111...)), which occurs even when the layout applied only at inference time. We attribute this behavior to modalities confusion induced by positional overlap, causing the model to misinterpret its own generated text tokens as visual tokens, resulting in this unpredictable repetitive output.
Enlarged Modality Interval:
We also tested artificially enlarging the modality interval to match that of vanilla RoPE, a strategy similar to RoPE-TI Tie (Su) (2024). This also resulted in poor document- related performance. However, the failure mode was distinct: the model generated fluent but contex-

---

Published as a conference paper at ICLR 2026
tually irrelevant text, effectively ignoring the visual input. This suggests that while clear modality interval is necessary, simply maximizing its size to align with vanilla RoPE can be detrimental by
Text spatial-reset. We also tested the strategy from IL-RoPE and Omni-RoPE, which resets spatial dimensions for visual tokens as well as text ones (Fig. Icl. This approach resulted notable per- formance degradation compared to the vanilla RoPE, emphasizing that preserving RoPE alignment for text is critical for successfully adapting LLMs into VLMs.
Scaling rotary base. Motivated by the smaller coordinate range of the spatial axes, experimented with scaling their corresponding rotary base (e.g., from 1,000,000 to 10,000). This consistently resulted in clear performance drop image benchmarks. This finding demonstrates that even well-intentioned deviations from the base LLM's RoPE formulation can break compatibility and
severely impair knowledge transfer.
Position Design
Image
Grounding
Video
Doc
InfoVQA
ChartQA
vanilla RoPE
65.69
73.48
51.64
82.94
58.85
56.84
+ 3D structure
65.87
74.40
51.29
82.33
57.24
61.44
3D + spatial-resed
66.65
75.85
52.36
83.72
58.24
62.12
+ diagonal layout
61.20
72.33
52.51
60.13
37.42
54.88
modality interval
62.80
73.19
50.88
70.43
42.18
51.28
text spatial-reset
58.27
68.2
50.71
77.30
52.15
44.33
scaling rotary base
60.15
74.13
52.11
80.44
52.16
58.80
Table Ablation study of different position design strategies.
3.4.2 ABLATION STUDY ON FREQUENCY ALLOCATION
To determine the optimal frequency allocation strategy, we fix the position design the same as MRoPE with spatial-resel enhancement, and vary only the frequency allocation scheme. shown in Table [S] more uniform allocation strategy consistently outperforms alternatives that split the spectrum into partial chunks. This highlights the importance of ensuring that each positional axes (time, height, width) retains access to the full frequency spectrum.
Allocation Type
Image
Video
Grounding
Overall
VideoRoPE-like
65.33
52.11
72.50
63.31
IL-RoPE-like
65.26
51.15
72.80
63.07
Multi-Head
66.40
52.58
74.92
64.63
Interleave
66.65
52.36
75.85
64.95
Table Ablation results of different frequency allocation strategies.
4 CONCLUSION
In this work, we conducted the first systematic investigation into multimodal Rotary Positional Em- bedding (RoPE) for Vision-Language Models (VLMs). From our systematic comparison and ex- tensive experiments, we identified three key design considerations for robust multimodal RoPE: positional coherence, full frequency utilization, and preservation textual priors from pre-trained LLMs. Guided by these insights, we proposed two plug-and-play RoPE variants: Multi-Head RoPE (MHRoPE) and MRoPE-IInerleav (MRoPE-I). Both methods adhere to our identified guidelines, effectively addressing common failure modes and achieving significant performance in both general and fine-graiie -reeeggrnndd multimodal understanding. This work offers a comprehensive guide for designing effective multimodal positional encodings, paving the way for future advancements in VLMs.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Ze- Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical re- port. CoRR, abs/2502. 13923, 2025. URL fttpt:tp/////......t...tt /oi. org 48550//88855/000 2502.
13923
10

---

Published as a conference paper at ICLR 2026
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Velick- ovic. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28. 2025. OpenReview.net, 2025. URL https: /ptps/////eeeeeeeeeeeeeee net/ fortm// du o
bloc97. atk-aware scaled rope allows llama models to have extended (8k+) con-
text size without any fine-tuning and minimal perplexity degradation., 2023. URL
attps://wwe
r/ocall LLaMA/ /tta:////n/nn//a//a//////aaa/aa/
scaled rope wwwwswwwww..........oooo.o.llw.w alamaamodelaaaaaaaaeeeaaaaaaaaae to_havee__
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision- language models? In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, pp. 24108-4240 24108-241183 18. Computer Vision Foundation / IEEE, 2025.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma. and Ranjay Krishna. BLINK: multimodal large language models can see but not perceive. 2024. URL https: /tw.///w.i./// org 10. 1007/978 3- 031 73337-6
Junqi Ge, Ziyi Chen, Jintao Lin, Jinguo Zhu, Xihui Liu, Jifeng Dai, and Xizhou Zhu. V2PE: improving multimodal long-context capability of vision-language models with variable visual position encoding. CoRR, abs/2412.09616, 2024. URL https::/doi.oot org/0.....g1.... 48550/
arXiv. 2412.422.10 09616
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, al. The llama herd of models, 2024.
URL htttps:///axii..tttt// wtt.or/gggb///....//./// 21783
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Referring to objects in photographs natural scenes. In Alessandro Moschitti. Bo Pang, and Walter Dacle- mans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), October 2014.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 235-251, Springer. 2016.
Haoran Li, Yingjie Qin, Baoyuan Ou, Lai Xu, and Ruiwen Xu. Hope: Hybrid of position embedding for length generalization in vision-language models. abs/R, 2025. URL httpp:
7/doi orooi.org/11 10. 48550/arXi11 2505. 20444
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Lou, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video understand- ing benchmark. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 22195-22206. IEEE, 2024.
Chao Liao, Liyang Liu, Xun Wang. Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved molti-modal generation. CoRR, abs/2505.0544 2025. URL http:///:...oo....////////////// 28550//0...555//..x...... 2505. 05472]

---

Published as a conference paper at ICLR 2026
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bi, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi- model an all-around player? In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gäl Varol (eds.), Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part VI, volume 15064 of Lecture Notes in Computer Science, pp. 216-233. Springer, 2024a.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang. Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: the hidden mystery of OCR in large multimodal models. Sci. China Inf. Sci., 67(12), 2024b.
Zikang Liu, Longteng Guo, Yepeng Tang. Junxian Cai, Kai Ma, Xi Chen, and Jing Liu. Vrope: Rotary position embedding for video large language models. CoRR, abs/2502. 1 1664,, 2025. URL //rg/./.... 48550 arxiv. 2502 11664
https://doi.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Compu- tational Linguistics: ACL 2022, Dublin, Ireland, May 22-27 2022, pp. 2263-2277 Association for Computational Linguisties. 2022.
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: dataset for VQA on doc- ument images. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, Jamuary 3-8, 2021, pp. 2199-2208. IEEE, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Emest Valveny, and C. Jawahar. Infographievqa. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022, pp. 2582-2591. IEEE, 2022.
Yann Dubois Nikhil Mehta Tong Xiao Philippe Gansen-EEtruch Licheng Yu Xiaofang Wang Felix Juefei-Xu Ning Zhang Serena Yeung-Levy Orr Zohar, Xiaohan Wang and Xide Xia. Apollo: An exploration of video understanding in large multimodal models. arXiv preprint 241:::2412. 10360, 2024.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context win- dow extension of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
QwenVL. Team. Qwen3-VL: The Most Powerful Vision- Language Model in the Qwen Series, 2025.
URL https::// thu.. com/ haaam//wwenn///mmm Qwen VI. Accessed: 2025-11-22.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 8317- 8326. Computer Vision Foundation IEEE, 2019.
Jianlin Su. Transformer upgrade path: 17. insights into multimodal positional encoding, March 2024. URLpattps://saceee ac en..e///aaa
Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024. URL /ttps::/doi..or www.aoo.orrgg// wwwl//..eeuoon............ neucom. 2023. 127063.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-l: A fully open, vision-centric exploration of multimodal 11ms, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 9998-6008 2017.
12

---

Published as a conference paper at ICLR 2026
Chengeheng Wang. Jianyuan Guo, Hongguang Li, Yuchuan Tian, Ying Nie, Chang Xu, and Kai Han. Circle-rope: Cone-like decoupled rotary positional embedding for large vision-languee mod- cls. CoRR, abs/2505.16416, 2025. URL htttp:://doo// org. 48550/ http:///1000000/1...vlooo
16416
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. CoRR, abs/2409. 12191, 2024a. URLhttps::/doo: org/10.48555 www.aaaivvvvv.... 12191
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark. CoRR, abs/2406. 08035, 2024b.
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao. Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, al. Videorope: What makes for good video rotary position embedding? In International Conference on Machine Learning, 2025.
Bo Wu, Shoubin Yu, Zhenfang Chen, Josh Tenenbaum, and Chuang Gan. STAR: A benchmark for situated reasoning in real-world videos. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks I, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang. Yexin Liu, Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng. Jiahao Wang. Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, and Zheng Liu. Omnigen2: Exploration to advanced multimodal generation. CoRR, hibs/2506.188711 2025. URL
https:://oot org/10. 48550/ wwr.vv.25000 18871
X.AI. Grok-1.5 vision
2024. URL URL.ttts://xxa//llloo////
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 1xXiv:2505.093880 2025.
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In IEEE/CVF Conference Computer Vision and Pattern Recog- nition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024 pp. 9556-9567. IEEE, 2024.
Junjie Zhou, Yan Shu, Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: A comprehensive benchmark for multi-task long video understanding. CoRR, abs/2406.04264, 2024a.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: A comprehensive benchmark for multi-task long video understanding. CoRR, abs/2406.04264 2024b.
A ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. this study, no human subjects or animal ex- perimentation was involved. All datasets used were sourced in compliance with relevant usage guidelines, ensuring no violation of privacy. We have taken care to avoid any biases or discrimi- natory outcomes in research process. No personally identifiable information experiments were conducted that could raise privacy or security concern. We are committed to maintaining transparency and integrity throughout the research process.
used, and no
13

---

Published as a conference paper at ICLR 2026
B REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper reproducible. Code will be made publicly available to facilitate replication and verification after inspection. The exper- imental setup, including training steps, model configurations, and hardware details, is described in detail in the paper. We believe these measures will enable other researchers to reproduce our work and further advance the field.
C LLM USAGE
Large Language Models (LLMs) were used to aid in the writing and polishing of the manuscript. Specifically, we used an LLM to assist in refining the language, improving readability, and ensuring clarity in various sections of the paper. The model helped with tasks such sentence rephrasing, grammar checking.
It is important to note that the LLM not involved in the ideation, research methodology, or experimental design. All research concepts, ideas, and analyses were developed and conducted by the authors. The contributions of the LLM were solely focused on improving the linguistic quality of the paper, with no involvement in the scientific content or data analysis.
The authors take full responsibility for the content of the manuscript, including any text generated or polished by the LLM. We have ensured that the LLM-generated text adheres to ethical guidelines and does not contribute plagiarism scientific misconduct.
D APPENDIX
D. PRACTICAL CONSIDERATIONS MHROPE VS. MROPE-I
While both of our proposed methods are effective, we currently recommend MRoPE-I over MHRoPE for two primary reasons: its consistent (albeit slight) performance advantage and its greater implementation simplicity. We attribute MHRoPE's minor performance deficit to its head- level information partitioning, which prevents the integration of different positional axes within the self-attention mechanism. From an engineering perspective, MRoPE-I is also simpler, avoiding the complexities that MHRoPE introduces with distributed training paradigms like tensor parallelism. Nevertheless, MHRoPE's design offers potentially more scalable architecture for future models that may need accommodate a larger number of positional axes.
D.2 DERIVATION OF THE ATTENTION SCORE UPPER BOUND IN MRoPE
Here, we provide a formal derivation for the upper bound of the RoPE attention score. The RoPE dot product between query and key is at relative position form as:
be expressed in complex
d/2-1
(R, = Re
Σ
(qp12411)
kipuz444 )e
(5)
1=
where x² denotes the complex conjugate of a 2D vector treated as a complex number, and the complex product.
To derive the upper bound, we analyze the magnitude of the summation term. To apply summation by parts, let us define content-dependent sequence h; [p2:2+1] k and a position- dependent sequence of partial sums S, Σ (mm-n) We also set the boundary conditions So and ha/2 = O. The standard summation by parts formula is Σ u, = [u,,,]b+t 1b+1 D Ue+1 Aut- Applying this, the magnitude of the summation be rewritten and bounded as
14

---

Published as a conference paper at ICLR 2026
follows:
d/2-1 Σ
d/2-1
lm-
[N:5:]0 1/2
Σ
(h+t hi)
1=0
==0
d/2-1
(ha/2Saf2 hu8o)
Σ
S Shot11o11 - hi)
==0
d/2-1
Σ 1h+t1 hi)
(6)
1=
d/2
S
Σ
|S 15+111/u411 - 14l
(=0
4/2-1 Σ
max 0Sicd/2
|
hil
S
This final expression reveals that the upper bound is product of two distinct components. max |/4+1 ha] is a content-dependent term that acts as a scaling factor based on the specific query purely position-dependent term whose value is deter-
and key vectors. The second. Σ 1 mined only by the relative position m n and the fixed frequencies 0.. Since the content-dependen term independent of position, the long-range decay property of the attention score governed primarily by this position-dependent term. Therefore, its average value, 2/2 Σ |St], serves as practical indicator to characterize how the upper bound attenuates with relative distance.
D.3 COMPATIBILITY WITH YARN EXTRAPOLATION
As shown in Figure 4b the interleaved frequency allocation of MRoPE-I makes it compatible with extrapolation algorithms like NTK-aware (bloc97 2023 and YaRN (Peng et al. 2024). Whereas standard MRoPE's partitioned spectrum complicates the application of consistent frequency scal- ing boundary, our interleaved design provides full spectrum across positional axes, enabling straightforward and symmetric application of these methods.
To validate this effect, we apply YaRN to both MRoPE and MRoPE-I under 256K context window and evaluate their performance on LVBench and MLVU. As shown in Table MRoPE-I with YaRN demonstrates substantially larger gains in long-video understanding compared to MRoPE.
Table 6: Performance comparison of MRoPE and MRoPE-I with and without YaRN under 256K context.
Method
LV LVBench
MLVU
MRoPE
41.5
62.9
MRoPE YaRN
41.2
63.3
MRoPE-I
42.0
63.2
MRoPE-I+ YaRN
43.6
64.1
D.4 LONG-CONTEXT VIDEO UNDERSTANDING
We further compare the performance different methods on long video understanding, with context lengths ranging from 32K to 256K. As shown in Figure apart from LVBench, do not observe clear performance improvements or degradation when extrapolating to longer sequences. The only exception is Vanilla RoPE, which suffers from sharp performance drop 128K/256K. We attribute this to excessively fast- aist-growing position IDs, which lead to degraded extrapolation capability, which also discussed in other works (Wei et al. 2025 et al. 2025).
Overall, methods such as VideoRoPE and HoPE, which allocate most low-frequency channels to the temporal axis, exhibit slightly better extrapolation ability in long video senario. However, when
15

---

Published as a conference paper at ICLR 2026
considering performance across images and grounding tasks, MHRoPE and MRoPE-I remain the most comprehensive and balanced designs.
59
03
50
37-
55
Context Length
Contect Lergth
Context
Wedi APC
Vileottoe
M-sort
Figure 5: Video extrapolation performance. Models are trained with context length of 32k (256 frames) and extrapolated 64k (512 frames), 128k (1024 frames), and 256k (2048 frames).
D.5 MORE ABLATION RESULTS.
D.5.1 ENHANCED VISUAL ATTENTION spatial-reset
To understand the mechanism driving the effectiveness of spatial-reset, we analyzed its impact on the model's attention patterns. As detailed in Table 7 we calculated the total attention scores on vi- sual tokens using the VoA test set. Specifically, we extracted attention scores from layers 12, 20, and 28, and averaged the scores across all attention heads and samples. The result demonstrates that MRoPE equipped with spatial-reset allocate more attention visual content, particularly in deeper layers, confirming it's effectiveness in enhancing the model's visual focus.
Method MHRoPE 40.31
Layer 4
Layer 12
Layer 20
Layer 28
21.76
32.05
19.00
w/o spatial-reset
35.99
19.68
22.02
9.93
MRoPE-I
37.48
15.68
28.08
23.23
w/o spatial-reset
31.22
17.66
16.02
11.69
Table 7: Average attention scores (%) on visual contents. The inputs are from DocVQA test set. And the scores are averaged between attention heads and samples.
D.5.2 ALLOCATION RATIO OF FREQUENCY
We further investigate different allocation ratios under the interleave frequency strategy. The results are summarized in Table The balanced allocation (t:h:w 24:20:20) achieves the best overall per- formance. Increasing the proportion of channels assigned the temporal axis reduces the available capacity for spatial dimensions. This leads to degradation in grounding ability and negatively impacts benchmarks involving spatial understanding in both images and videos.
Allocation Ratio
Image
Video
Grounding
Overall
0:32:32
66.42
51.01
76.02
64.48
12:26:26
66.30
51.93
75.77
64.67
24:20:20
66.65
52.36
75.85
64,95
32:16:16
64.07
51.15
74.65
63.29
48: 8: 8
65.06
51.17
72.87
63.03
Table 8: Ablation results of different frequency allocation ratios under interleave design.
D.5.3 TEMPORAL STRIDE VIDEO MODELING
This section investigates the impact of different temporal strides between video frames. Specifically, we experiment with = 0.5, 1, 2, as well as dynamic strides as used in V2PE and HoPE (with applied during inference). The results shown in Table
16

---

Published as a conference paper at ICLR 2026
Stride 0.5
MVBench
STAR
VideoMME
LVBench
MLVU
Charades
Overall
56.55
57.90
58.96
38.99
62.37
31.88
51.11
57.05
57.79
58.96
40.34
65.46
34.36
52.36
55.70
58.13
58.15
38.02
63.11
33.51
51.10
Dynamic
56.28
57.93
58.74
41.12
63.75
32.99
51.80
Table 9: Comparison temporal stride settings for video benchmarks on MRoPE-L.
From the results,
achieves the best overall performance, while smaller 0.5) larger
(Ō 2) strides lead to performance drops. Incorporating the dynamic stride from V2PE does not shows a significant benefit.
D.6 FULL EXPERIMENT RESULTS ON QWEN3-VL
We present the complete results of main experiments Qwen3-VL. Table T0 and Table I1 report the performance Qwen3-VL--B V.---------nstrrue and Owen3-VI-BB-Instruuc, respectively. Across both model scales, MHRoPE and MRoPE-I consistently outperform other multimodal posi- tional encoding variants. The experimental settings are identical to those in the main experiments.
Table 10: Overall performance of multimodal RoPEs variants on various benchmarks, evaluated Dwee3--VL-4B-Instrue The highest score shown in bold, while the second-highest score is underlined.
Types
Benchmarks
Vanilla RoPE
MRoPE
VideoRoPE
HoPE
CircleRoPE
MHRoPE
MRoPE-I
MMMU
45.78
42.89
44.44
45.44
45.33
46.11 11
46.33
MMBenchary
68.77
71.26
69.20
70.79
71.35
71.56
71.20
MMstar
43.67
42.87
41.87
43.53
43.73
43.73
45.00
OCRBench
35.20
37.30
32.10
28.70
37.10
38.60)
38.30
AI2D
66.71
67.29
63.21
63.46
66.06
68.32
69.29
Image
RealworldQA
$7.12
$6.60
57.25
57.52
57.25
58.86
59.78
DucVQA
46.59
43.13
25.27
27.25
44.60
46.77
46.89
TexVVQ
46.65
48.25
47.10
47.16
47.27
48.60
48.27
InfoVQA
30.55
28.14
16.22
17.32
30.02
30.93
32.46
ChanQA
39.64
38.80
39.52
40.68
39.76
41.64
BLINK
36.33
36.80
34.44
34.46
37.52
37.22
37.88
MVBench
$4.30
$3.95
53.83
54.53
53.13
54,98
$4.30
STAR
58.07
$7.90
58.77
56.73
$5.89
58.66 $8.66
$8.23
Video
MLVU
60.03
60.76
60.21
61.72
59.40
61.00
61.29
VidcoMME
50.93
50.41
50.19
51.41
50.44
$1,30
$0,70
LVBench
36,93
36.35
36.09
35,54
37,20
36,99
Charades-STA
39.16
38.20
40.06
40.08
40.98
41.68
41.44
Re(COCO... ml
22.51
25.53
19.58
19,74
20.38
26.71
28.82
ReiCOCO testt
22.91
27.40
20.03
21.35
21.21
26.60
28.91
ReFCOCO, rest
23.20
25.14
21.90
20.27
20.88
28.13
30.68
Grounding
RefCOCO+val
17.12
20.37
15.45
15,98
15.48
21.33
22.61
SefCOCO+taaa
18.49
21.76
16.66
16.59
21.87
23.85
RefCOCO+juat +
18.20
20.37
17.71
16.32
16.22
23.54
25.81
RefCOCOg. gma!
22.32
24.90
20.34
20.45
20.10
26.80
30,(K)
ReiCOCOg...
21.38
23.89
20.93
19.19
19.82
26,90
29.52
Image
47.00
46.67
42.77
43.20
47.36
48.22
48.82
Overall
Video
49.90
49.60
49.86
50.25
49.23
50.80
$0.49
Grounding
20.77
23.67
18.74
25.23
27.52
17

---

Published as a conference paper at ICLR 2026
Table 11: Overall performance of multimodal RoPEs variants on various benchmarks, evaluated on Ve-B--V--B--Insee The highest score shown in bold, while the second-highest score is underlined.
Types
Benchmarks
Vanilla RoPE
MRoPE
VideoRoPE
HoPE
CircleRoPE
MHRoPE
MRoPE-I
MMMU
51.18
50.11
51.78
51.89
50.67
$3.33
$3.89
MMBencharg
79.29
78.27
78.74
79.59
79.00
80.78
79.50
MMstar
51.88
52.53
53.33
$2.86
54.40
53.20
53.47
OCRBench
73.20
72,90
67.70
61.70
73.60
74.40
73,90
A12D
78.90
77.56
78.34
77.85
77.59
78.22
79.50
Image
RealworiaQA
63.27
64.05
62.48
62.48
61.66
67.33
65.22
DocVQA
82.41
82.85
71.71
72.95
82.28
82.41
83.67
TexVQQ
63.15
66.05
61.54
62.33
64.94
67.20
67.32
InfoVQA
53.84
49.60
39.15
41.69
51.06
$2,22
$2.62
ChariQA
63.44
BLINK
37.21
37.44
37.08
37.22
40.52
40,00
40.45
MVBench
60.35
59,90
59.70
60.53
59.53
60,20
60.70
STAR
61.65
62.29
61.21
62.19
61.51
62.33
62.45
MLVU
66.01
67.34
68.31
68.81
66.61
67.20
67.52
Vidco
VideoMME
60.07
59.78
61.48
60.56
$1.04
60.89
60.89
LVBench
41,06
42.93
42.48
42,00
41.58
Charades-STA
50.39
51.79
52.90
52.30
51.77
$2.13
$2.70
Re|COCO.a:
73.22
74.16
73.22
74.70
77.72
RefCOCOn.a
76.45
77.11
73.77
74.50
76.77
79.41
83.21
RefCOCO,..
71.40
72.09
69.89
70.93
73.22
72.89
76.45
Grounding
RefCOCO+vll
65.77
65.94
62.05
66.33
66.37
67.99
70.88
RefCOCO Ottest
71.17
72.25
68.08
72,89
72.41
75.08
77.79
ReiCOCO+zaa Ottts
$9.97
60.46
$9.26
63.18
64.33
Re(COCOg... g.n!
71.14
73.10
69.77
71.28
72.33
74,14
76.47
ReiCOCOg... Frest
71.57
72.68
69.45
71.82
72.67
74.65
76.79
Image Vidco
63.35
62.95
60.26
59,96
63.54
64.63
64.82
Overall
56.78
$7.03
57.75
57.81
54.88
$7.46
$7.64
Grounding
70.09
70.97
67.99
70.19
71.20
73.03
75.46
18


Reviews:
[{"review_id": "MAHAnirxbk", "reviewer": "Reviewer_KSo7", "summary": "The paper re-examines how multimodal positional encodings (especially RoPE) should be designed for vision–language models, arguing that current extensions are ad hoc and often break either visual geometry or LLM compatibility. It identifies three principles—positional coherence, full frequency utilization, and preservation of textual priors—as necessary to avoid modality confusion, attention sinks, and degraded long-context/video performance. Based on this, the authors propose two plug-and-play variants, Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), which give each axis (time, height, width) access to the full frequency spectrum while keeping text RoPE unchanged. Extensive experiments on >20 image, video, and grounding benchmarks show these designs consistently outperform vanilla RoPE, MRoPE, and recent video-focused RoPEs, especially on fine-grained and grounding tasks.", "questions": "Is MHRoPE and MRoPE-I more scalable than MRoPE and RoPE in the multimodal domain, such as better loss curve with proportionally increasing model size and data size?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. propose two plug-and-play variants, Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), which give each axis (time, height, width) access to the full frequency spectrum while keeping text RoPE unchanged.\n2. Extensive experiments on >20 image, video, and grounding benchmarks prove effectiveness of these designs\n3. Easy to understand and comprehend. Figures are clear.", "weaknesses": "1. Limited experimented VLMs. The experiments only cover Qwenvl2.5. Broader range of models such as llama, phi and internvl series would better demonstrates the generalizability of the benefits from the two novel designs.\n2. Increamental technical novelty compared with MRoPE. This paper propose spatial-reset and two frequency allocation modifications upon MRoPE. Despite effectiveness, the overall technical novelty compared to MRoPE and the original RoPE remains limited.\n3. experiment with rapid temporal decay and spatial asymmetry problem, such as thousands of frames or extreme interleaving patterns where attention sinks typically reappear and see the performance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mG4MT1k5mk", "reviewer": "Reviewer_PFUK", "summary": "The paper proposes MRoPE-I and MHRoPE, two improved methods for using rotary embeddings to encode position information in VLMs. MRoPE-I and MHRoPE improve on existing techniques by:introducing a \"spatial-reset\" to fully decouple position and time encoding and ensuring that positional infomration is encoded across the entire frequency spectrum. The work proposes with two methods for encoding frequency across the full spectrum: head specialization (MHRoPE) and interleaved (MRoPe-I).", "questions": "Several question can be found in the section above, in addition:  \n\n- What is the performance if the number of temporal channels is less than 24 in the interleaved strategy?\n\n- How are video frames sampled? Are a constant number of frames sampled uniformly or is fps sampling used? If so, what fps is used? Similarly, what image processing/scaling is used? Do the authors use QwenVL2.5’s dynamics scaling of visual tokens? Are the total number of frames sampled capped in long video? Do any of these settings differ between training and evaluation?\n\n- How was the ChartQA visualization created in figure 2? It is the reviewer’s understanding that ChartQA has a single image per input, but figure 2 has two images.\n\n---\n\n[1] S. Bai *et al.*, “Qwen2.5-VL Technical Report,” 2025, doi: 10.48550/arxiv.2502.13923.  \n\n[2] O. Zohar *et al.*, “Apollo: An Exploration of Video Understanding in Large Multimodal Models,” in *Proceedings (IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Online)*, IEEE, 2025, pp. 18891–18901. doi: 10.1109/CVPR52734.2025.01760.  \n\n[3] S. Tong *et al.*, “Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs,” 2024, doi: 10.48550/arxiv.2406.16860.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- In the reivewers opinion, the ideal way to encode position in VLMs is unclear and warrants further exploration. In particular, well put together ablations are interesting and useful.\n\n- The severe degradation of vanilla RoPE on long video inputs (and relatively strong performance on shorter visual tasks) and observation of a “visual attention sink“ across modalities, is interesting.\n    \n- The paper is clear in its presentation of current challenges in multimodal position embeddings, and its proposed solution.", "weaknesses": "- The vision encoder of Qwen2VL / connector MLP of Qwen2.5VL has been trained to output visual token that are used with MRoPE-like position encodings. This may disadvantage approach such as VideoRoPE, HoPE and CircleRoPE. Initializing from  vision encoder that has not been trained for use in a VLM would be better. This problem also seems to result in disabling the absolute time encoding scheme proposed for MRoPe [1]. Therefore, the MRoPe baseline is not one-to-one with how it is used in QwenVL-2.5. Overall, heavy use of QwenVL components makes interpreting experiments difficult and introduces confounding factors.\n    \n\n- The experimental setup is quite limited. Only 1 LLM is considered, and only at 7B parameter size. Showing generalization across architectures would be valuable.\n    \n\n- The stages followed for training the VLMs are not standard, especially the choices to initialize the vision encoder and MLP layers with QwenVL components. It would be better to recreate an established process for constructing a VLM from existing work [2,3].\n    \n\n- A discussion about the similarities of MRoPE-I and MHRoPE to MRoPE **given the training and testing distribution** is essential. In the case of a single visual element and no system prompt the proposed approach is very similar MRoPE. What/if system prompts are used for each evaluation is not clear. However, the single image case is certainly the dominant case among evaluations. Are system prompts are used during training? Is the system prompt of consistent length during training / evaluation? This is important for understanding exactly how position is encoded (i.e. what do equations 3 and 4 realistically evaluate to).\n    \n\n- The paper is very vague about what training data is used. The reviewer can only find that “2M high-quality supervised fine-tuning (SFT) samples” were used. Nothing about the breakdown between modalities, conversations vs. captioning, particular data sources, etc. Figure 1 demonstrates the position encoding for conversations containing multiple interleaved visual elements. Did the training contain conversations with multiple visual elements, if so, what proportion of the training data was it? Such details are crucial for understanding how position is realistically encoded and reproducibility.\n    \n\n- Some details are missing from training settings, such as the optimizer used and it’s respective settings. Justification for the choices of hyperparameters would also be valuable.\n    \n\n- It is claimed that MHRoPE / MRoPE-I result in “more focus on visual information.“ However, that is never justified experimentally or formalized. Do the proposed methods have larger attention scores (on visual tokens) than baselines?\n    \n\n- The benchmarks used to evaluate MRoPE-I and MHRoPE overwhelmingly follow the structure of a single visual document followed by some text (usually a short question). This does not test the proposed method across a diverse range of positional encoding setups. Comparisons across conversations with multiple visual documents (akin to Figure 1) would be valuable.\n\nMinor:  \n\n- Table 2 is a bit messy. Sometimes the best score is bold and in other rows the second best score is bold (for example, the Video row).\n    \n- Quotes open in the wrong direction in the manuscript, and are inconsistent throughout. Some are correct such as “1111...” whereas elsewhere it is flipped, such as ”attention sink,\" .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "89m7EuUVMi", "reviewer": "Reviewer_T4pq", "summary": "This paper presents a systematic study of multimodal rotary positional embeddings (RoPE) in Vision-Language Models (VLMs). The authors decompose the multimodal RoPE design space into three core aspects—position design, frequency allocation, and compatibility with text-only RoPE—and identify common pitfalls in existing methods. Building upon these insights, they propose two new variants: Multi-Head RoPE (MHRoPE) and MRoPE-Interleave (MRoPE-I), along with a spatial-reset mechanism. These designs aim to ensure positional coherence, full frequency utilization, and faithful transfer of textual priors from pre-trained LLMs. Through extensive experiments on over twenty benchmarks covering image, video, and grounding tasks, the proposed methods achieve consistent and notable improvements over prior work.", "questions": "- Carefully review Table 2. It seems that the calculations for some data are incorrect, especially in the \"overall\" column and the parts in red font.\n- How sensitive are the proposed methods to the choice of rotary base or context length during training? Would a smaller base or shorter context alter the observed trends?\n- Can the authors elaborate on whether spatial-reset interacts with patch embedding granularity (e.g., ViT patch size)?\n- Is there a possibility that interleaved frequency allocation could interfere with frequency-aware extrapolation methods when used in ultra-long video sequences?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides one of the most systematic and insightful examinations of positional encoding in multimodal contexts to date. The decomposition into design axes (position, frequency, and compatibility) clarifies the often opaque design space of VLM encodings.\n\n- MHRoPE and MRoPE-I are conceptually straightforward and “plug-and-play,” requiring no architectural modifications, which increases their practical utility.\n\n- The authors evaluate across a wide range of benchmarks, demonstrating consistent gains and analyzing failure cases of prior methods.\n\n- Extensive ablations provide convincing evidence for the effectiveness of spatial-reset and interleaved frequency allocation.", "weaknesses": "- While the analysis is systematic, the theoretical understanding of why certain frequency allocations or resets perform better remains mostly empirical. A deeper mathematical treatment could strengthen the claims.\n- Although improvements are consistent, in some cases (e.g., video benchmarks), the gains are modest, which might question the generality of the benefits.\n- Dependence on Qwen2.5-VL backbone. All experiments rely on one model family. It is unclear if the conclusions would hold equally well for other architectures (e.g., LLaVA or Gemini-style models).\n- The authors note MHRoPE’s potential scalability but provide little empirical evidence of its performance on extremely large-scale or high-resolution data.\n- The improvement in the experimental results is not significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f5gWQw46Hl", "reviewer": "Reviewer_kJVi", "summary": "This paper focuses on the design of multimodal position embeddings for VLMs. Specifically, the authors analyze existing RoPE variants and their designs used in VLMs, and summarize three key guidelines: the position design for different modalities, the frequency allocation for different positional axes, and the compatibility with vanilla text-only RoPE. Guided by these principles, the paper proposes two new designs: MHRoPE and MRoPE-I. Both methods enhance the MRoPE by incorporating a ‘spatial-reset' mechanism, and they introduce two novel frequency allocation strategies. Experiments demonstrate the effectiveness of the two proposed methods.", "questions": "For MHRoPE, is there any experiment or discussion on how the heads are allocated?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper provides a comprehensive and systematic discussion of the multimodal RoPE methods.\n* The proposed two methods MHRoPE and MRoPE-I are simple while well-motivated, and the results show their effectiveness.\n* There are extensive experiments, with various benchmarks across image understanding, video understanding and visual grounding tasks.", "weaknesses": "* Validation on only a single VLM instance (i.e. the experiments are based on Qwen2.5VL) may not be sufficient to demonstrate the robustness of the methods.\n* It is surprising that MRoPE underperforms Vanilla RoPE on many benchmarks (Table 2). Perhaps the authors could provide a deeper discussion about the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)