Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding
Authors: Zhanpeng Chen, Mingxiao Li, Ziyang Chen, nan du, Xiaolong Li, Yuexian Zou
Accepted at: acl25

Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding

Zhanpeng Chen1,2, Mingxiao Li2, Ziyang Chen2, Nan Du2, Xiaolong Li2, Yuexian Zou1,*
1Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University
2Tencent Hunyuan
troychen927@stu.pku.edu.cn, zouyx@pku.edu.cn

Abstract

Vision-language Models (VLMs) have shown remarkable capabilities in advancing general artificial intelligence, yet the irrational encoding of visual positions persists in inhibiting the models’ comprehensive perception performance across different levels of granularity. In this work, we propose Pyramid-descent Visual Position Encoding (PyPE), a novel approach designed to enhance the perception of visual tokens within VLMs. By assigning visual position indexes from the periphery to the center and expanding the central receptive field incrementally, PyPE addresses the limitations of traditional raster-scan methods and mitigates the long-term decay effects induced by Rotary Position Embedding (RoPE). Our method reduces the relative distance between interrelated visual elements and instruction tokens, promoting a more rational allocation of attention weights and allowing for a multi-granularity perception of visual elements and countering the over-reliance on anchor tokens. Extensive experimental evaluations demonstrate that PyPE consistently improves the general capabilities of VLMs across various sizes. Code is available at https://github.com/SakuraTroyChen/PyPE.

1 Introduction

Large Language Models (LLMs) (Touvron et al., 2023; Brown et al., 2020) demonstrate significant universal capabilities that contribute to the pursuit of general artificial intelligence. However, language constitutes only one aspect of communication. Visual information plays a crucial role in augmenting and enhancing our understanding of the world. Consequently, there is a growing interest in the development of Vision-language Models (VLMs) (Chen et al., 2024c; Peng et al., 2023; Wang et al., 2024; Bai et al., 2023) that can process and integrate visual modality. To effectively leverage the powerful contextual understanding capabilities of LLMs, VLMs project visual information to the same dimensionality as textual embeddings through specific projection layers (Chen et al., 2023; Li et al., 2023b; Zhou et al., 2024), which are then directly embedded into the text sequence to form the input for the foundation LLMs, enabling cross-modal alignment and instruction-following learning using next-token prediction.

Despite their commendable progress, the typical processing of visual information does not align with the distribution patterns of visual elements. Since visual information is composed of fixed-sized patches obtained through raster scanning, patches located closer to the bottom right corner of the image are positioned nearer to the instruction tokens within the sequence. Due to the long-term decay from Rotary Position Embedding (RoPE) (Su et al., 2024), visual tokens closer to the instruction tokens will be more likely to receive higher attention weights, and vice versa. This is counter-intuitive, as the importance of visual information is not defined by the order of raster-scanning. Xing et al. (2024) observe a similar phenomenon by visualizing the attention information flow from instruction tokens to visual tokens in the first layer of the decoder. Consequently, they propose Concentric Causal Attention (CCA), which starts assigning the position indexes of images from the peripheral and ends in the center, to alleviate the long-term decay in RoPE and improve causal attention following 2D spatial locality of images. Although CCA is both intuitive and effective, its applicability is constrained by the assumption that all significant elements related to the instructions are situated at the center of the image. This assumption inherently results in a loss of detail, limiting its effectiveness in capturing comprehensive information.

To further investigate the impact of raster-scan
Figure 1: Layer-wise attention visualization of visual-to-instruction information flow. Displayed from top to bottom are the attention heatmaps from LLaVA-1.5-7B trained with raster-scan and concentric PE, respectively. The example is derived from LLaVA-Bench (Liu et al., 2024b) and the query is "Describe this photo in detail".

and concentric PE on the fine-grained modeling of visual information, we extend the visualization to all layers of the decoder. As illustrated in Figure 1, CCA demonstrates exceptional performance in the first layer, alleviating the long-term decay caused by RoPE in the raster-scan approach, thereby directing the model’s attention to more significant areas. However, in the subsequent layers, both methods largely maintain the same attention patterns as observed in their respective third layers, with changes only occurring in the final layer. A similar phenomenon, namely "aggregation pattern", is observed in OPERA (Huang et al., 2024), where both LLMs and VLMs tend to generate new tokens by concentrating on a limited number of summary tokens (also referred to as anchor tokens (Wang et al., 2023)) rather than considering all preceding tokens. This tendency towards partial overtrust leads to the neglect of fine-grained image tokens, resulting in the generation that may be hallucinatory and do not accurately reflect the image content. Moreover, it has been demonstrated in OPERA that more hallucinations are generated when more summary tokens appear in the context.

To this end, we present Pyramid-descent Visual Position Encoding (PyPE), a novel position assignment approach for visual tokens, to alleviate the long-term decay induced by RoPE, avoid the "aggregation pattern" in the LLM, and ensure a comprehensive understanding of visual contents. PyPE reorganizes the flattened visual tokens into the 2D shape and assigns visual position indexes from the periphery to the center. This reduces the relative distance between interrelated visual elements, as well as the distance between significant visual elements and instruction tokens, thereby ensuring a more rational allocation of attention weights. Furthermore, to mitigate the impact of anchor tokens on the model’s fine-grained perception of visual elements, we draw inspiration from Pyramid Vision Transformer (PVT) (Wang et al., 2021): consistently combining global and local receptive fields. PyPE gradually expands the central receptive field, i.e., the central region of the position index matrix, at predetermined intervals of layers. Specifically, we expand the central region of the position index matrix by a circle every certain number of layers. Such expansion weakens the anchor tokens and enhances the model’s ability to perceive visual elements at varying levels of granularity (more cases can be found in Section 5.4).

With extensive experiments on visual question answering and general multimodal benchmarks, PyPE consistently improves general perception capabilities across VLMs of different sizes. In a nutshell, the main contributions of this work are as follows: (I) We make an in-depth analysis of how position encoding affects visual perception in VLMs. (II) Our proposed PyPE effectively mitigates long-term decay and the "aggregation pattern", which helps better perceive visual elements at different granularities. (III) Extensive evaluations demonstrate the superior performance of PyPE, a simple yet effective method that applies to any VLMs.

2 Related Work

2.1 Vision-language Model

Recent advancements in VLMs have demonstrated impressive performance in processing multi-format information (Huang et al., 2023; Achiam et al., 2023). VLMs are typically built upon existing LLMs and incorporate visual information as input tokens by utilizing an additional vision encoder (e.g., CLIP) and a bridging connector (e.g., MLP). For instance, LLaVA (Liu et al., 2024a) employs an MLP to project visual tokens and aligns the feature dimensions with word embeddings, while
BLIP-2 (Li et al., 2023b) utilizes a set of learnable query tokens to extract information in a query-based manner. Building upon these foundational works, MM1 (McKinzie et al., 2025) has further investigated the significance of the number of visual tokens and image resolution, identifying them as the most critical factors, while finding that the type of connector has minimal impact. By effectively connecting visual and textual modalities, VLMs significantly enhance human-AI interaction and exhibit remarkable capabilities in understanding and generating multimodal content (Chen et al., 2024b; Peng et al., 2023; Chen et al., 2023; Wang et al., 2024; Hu et al., 2024; Xie et al., 2024).

2.2 Position Encoding for Transformers
Since transformer-based models contain no recurrence (Hochreiter, 1997) and convolution (Islam et al., 2020) structure, additional information about the relative or absolute position of the tokens in the input sequence is required. Therefore, the community has witnessed the development of various position encoding methods, e.g. sinusoidal (Vaswani, 2017), learnable (Dosovitskiy, 2020), relative (He et al., 2020; Shaw et al., 2018), and conditional (Chu et al., 2021) position encoding. Among these studies, RoPE (Su et al., 2024) is introduced to encode absolute and relative positional information, showing superiority in LLMs (Touvron et al., 2023; Achiam et al., 2023). The success of LLMs has led to the continued adoption of the effective RoPE scheme in VLMs for the unified encoding of positional information across sequences that incorporate multimodal features. However, it is important to note that visual information does not conform to the same sampling paradigm as language. The raster scanning is insufficient for modeling the spatial correlations among different patches. Consequently, numerous recent studies (Chu et al., 2024; Xing et al., 2024; Lu et al., 2024) have sought to explore improved solutions that extend RoPE to visual tasks. In this paper, we investigate a novel multi-granularity position assignment strategy to enhance the VLM’s comprehension of visual information and improve the alignment between modalities.

3 Approach

3.1 Preliminaries
RoPE (Rotary Position Embedding) RoPE (Su et al., 2024) unifies both absolute and relative positional encodings, demonstrating a certain degree of extrapolation capability in LLMs and VLMs. Given the m-th query and n-th key vectors with a dimension D, denoted as \( \mathbf{q}_m, \mathbf{k}_n \in \mathbb{R}^{[D]} \), RoPE multiplies a bias to the key or query vector in the complex vector space as follows:

\[
f_q(\mathbf{q}_m, m) = e^{im\Theta} \mathbf{q}_m, \quad f_k(\mathbf{k}_n, n) = e^{in\Theta} \mathbf{k}_n
\]

where \( \Theta = \mathrm{Diag}(\theta_1, \cdots, \theta_{|D|/2}) \) is the rotary frequency matrix, where \( \theta_d = b^{-2d/|D|} \) and the rotary base \( b = 10000 \). In real space, for \( l = |D|/2 \), the rotary matrix \( e^{im\Theta} \) can be expressed as:

\[
\begin{bmatrix}
\cos m\theta_1 & -\sin m\theta_1 & \cdots & 0 & 0 \\
\sin m\theta_1 & \cos m\theta_1 & \cdots & 0 & 0 \\
\vdots & \vdots & \ddots & \vdots & \vdots \\
0 & 0 & \cdots & \cos m\theta_l & -\sin m\theta_l \\
0 & 0 & \cdots & \sin m\theta_l & \cos m\theta_l
\end{bmatrix}
\]

The attention score using RoPE is calculated as follows:

\[
A_n = \mathrm{Re}(f_q(\mathbf{q}_m, m), f_k(\mathbf{k}_n, n)) \\
= \mathrm{Re}(\mathbf{q}_m^T e^{i(m-n)\Theta} \mathbf{k}_n)
\]

where \( \mathrm{Re}(\cdot) \) is the real part of a complex number and \( e^{i(m-n)\Theta} = (e^{im\Theta})^\top e^{in\Theta} \). As the relative distance \( m - n \) increases, the attention score \( A_n \) correspondingly decreases due to long-term decay. This behavior aligns with the intuitive understanding that a pair of tokens separated by a significant
Figure 3: An overview of the proposed PyPE. We first reorganize the visual tokens from their vanilla flattened 1D sequence form into the 2D format. Subsequently, we assign visual position indexes from the periphery to the center and expand the central receptive field incrementally across the layers with an interval of \( t \).

relative distance should exhibit a weaker connection, and vice versa. However, a similar situation is observed in VLMs (Xing et al., 2024), which can lead to the model lacking attention to patches that are relatively far from the instruction token obtained through raster scanning.

All-One Position Encoding To further explore the impact of visual position encoding on the model’s perception of visual elements, we propose All-One Position Encoding: directly setting the relative distance between all image tokens and instruction tokens to 1. By doing so, the relative distances from all image tokens to the instruction token become equal, thereby excluding the influence of relative position decay introduced by RoPE. As a result, all patches are treated equally.

As indicated in Table 1, All-One PE performs weaker than the baselines in perception but keeps competitive in coarse-grained perception tasks on different sizes of models. This suggests that even when assigning the same positional weight to all image tokens, the VLM still possesses certain perception capabilities and performs better than raster-scan and concentric in coarse-grained situations. This is more pronounced on LLaVA-1.5-13B because larger models have stronger sequence modeling and feature capturing capabilities, which correspondingly bridge the gap in fine-grained abilities between All-One PE and other methods.

3.2 Pyramid-descent Visual Position Encoding Though presenting competitive coarse-grained perception capabilities, All-One PE still falls short in fine-grained perception. Using identical position weights hampers the model’s ability to differentiate the significance of image tokens, while the positional priors introduced by raster scanning conflict with general cognitive principles.

Similar challenges were also present in the early development of Vision Transformer (ViT) (Dosovitskiy, 2020). Due to the columnar structure of ViT, which uses coarse image patches as input, it is difficult to apply it directly to pixel-level dense predictions such as object detection and segmentation. This difficulty arises because its output feature map is single-scale and low-resolution. To address these issues, Wang et al. (2021) proposed the Pyramid Vision Transformer (PVT). They utilize fine-grained image patches as input to learn high-resolution representations and introduce a progressive shrinking pyramid to reduce the sequence length of the Transformer as the network deepens, significantly lowering the computational cost. Moreover, compared to CNNs, PVT consistently produces a global receptive field, ensuring a holistic perception of visual elements and benefiting its performance in detection and segmentation tasks.

In light of this, we propose the Pyramid-descent Visual Position Encoding (PyPE), a simple yet effective position assignment strategy for visual tokens in VLMs. As shown in Figure 3, we first reorganize the visual tokens from their vanilla flattened 1D sequence form into the 2D format. Subsequently, we adopt a decay pattern for the corresponding position indexes of the image tokens that spread outward from the center following concen-
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Perception</th>
    <th colspan="1">Commonsense QA (Reasoning)</th>
    <th colspan="4">Coarse-grained Perception Tasks</th>
    <th rowspan="2">Total Scores</th>
  </tr>
  <tr>
    <th>Existence</th>
    <th>Count</th>
    <th>Position</th>
    <th>Color</th>
  </tr>
  <tr>
    <td><i>TinyLLaVA-SigLIP-Phi-2</i></td>
    <td>w/ Raster-scan</td>
    <td>1488.30</td>
    <td>120.71</td>
    <td>185.00</td>
    <td>143.33</td>
    <td>133.33</td>
    <td><b>180.00</b></td>
    <td>762.37</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ Concentric</td>
    <td>1465.25</td>
    <td>114.29</td>
    <td>185.00</td>
    <td><b>160.00</b></td>
    <td>131.67</td>
    <td>170.00</td>
    <td>760.96</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ All-One</td>
    <td>1489.46</td>
    <td>117.14</td>
    <td><b>190.00</b></td>
    <td>158.33</td>
    <td>133.33</td>
    <td>175.00</td>
    <td><b>773.80</b></td>
  </tr>
  <tr>
    <td></td>
    <td>w/ PyPE (Ours)</td>
    <td><b>1500.66</b></td>
    <td>115.00</td>
    <td><b>190.00</b></td>
    <td>150.00</td>
    <td><b>138.33</b></td>
    <td><b>180.00</b></td>
    <td>773.33</td>
  </tr>
  <tr>
    <td><i>LLaVA-1.5-7B</i></td>
    <td>w/ Raster-scan</td>
    <td>1510.72</td>
    <td><b>135.71</b></td>
    <td>190.00</td>
    <td><b>158.33</b></td>
    <td>128.33</td>
    <td>175.00</td>
    <td>787.37</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ Concentric</td>
    <td>1485.67</td>
    <td>120.71</td>
    <td>190.00</td>
    <td>153.33</td>
    <td>133.33</td>
    <td>170.00</td>
    <td>767.37</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ All-One</td>
    <td>1474.13</td>
    <td>131.43</td>
    <td><b>195.00</b></td>
    <td>148.33</td>
    <td>141.67</td>
    <td>175.00</td>
    <td>791.43</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ PyPE (Ours)</td>
    <td><b>1542.19</b></td>
    <td>130.00</td>
    <td><b>195.00</b></td>
    <td>155.00</td>
    <td><b>146.67</b></td>
    <td><b>180.00</b></td>
    <td><b>806.67</b></td>
  </tr>
  <tr>
    <td><i>LLaVA-1.5-13B</i></td>
    <td>w/ Raster-scan</td>
    <td>1581.45</td>
    <td><b>132.14</b></td>
    <td>190.00</td>
    <td>155.00</td>
    <td>135.00</td>
    <td><b>195.00</b></td>
    <td>807.14</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ Concentric</td>
    <td>1607.40</td>
    <td>128.57</td>
    <td><b>195.00</b></td>
    <td><b>180.00</b></td>
    <td>141.67</td>
    <td>185.00</td>
    <td>830.24</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ All-One</td>
    <td>1608.12</td>
    <td>130.00</td>
    <td><b>195.00</b></td>
    <td>170.00</td>
    <td>146.67</td>
    <td>190.00</td>
    <td>831.67</td>
  </tr>
  <tr>
    <td></td>
    <td>w/ PyPE (Ours)</td>
    <td><b>1629.41</b></td>
    <td>130.71</td>
    <td>190.00</td>
    <td><b>180.00</b></td>
    <td><b>153.33</b></td>
    <td>180.00</td>
    <td><b>834.04</b></td>
  </tr>
</table>

Table 1: Performance evaluation on MME. <i>Existence, Count, Position, and Color</i> are coarse-grained subtasks of MME-Perception, while <i>Commonsense QA</i> is a subtask of MME-Cognition. <i>Total Scores</i> denotes the sum of the results from <i>Commonsense QA</i> and <i>Coarse-grained tasks</i>. The best results in each setting are in bold.

Algorithm 1 Pyramid-descent Visual Position Encoding

INPUT: Height \( H \), width \( W \), descent interval \( t \), current layer index \( i \), current \( \mathcal{P}_{max} \).
OUTPUT: Pyramid-descent position assignment matrix \( \mathcal{P} \), causal mask \( \mathcal{M} \) and \( \mathcal{P}_{max} \) for the next layer.
1: if \( i \mod t == 0 \) and \( \mathcal{P}_{max} > 1 \) then
2:   \( \mathcal{P}_{max} \leftarrow \mathcal{P}_{max} - 1 \)
3: end if
4: Initialize \( \mathcal{P} \).
5: for \( p \) in \( [1, \mathcal{P}_{max}] \) do
6:   \( \mathcal{P}[p : H - p, p : W - p] \leftarrow p \)
7: end for
8: Generate \( \mathcal{M} \) according to \( \mathcal{P} \).

tric PE (Xing et al., 2024). Given the maximum assignable position index \( \mathcal{P}_{max} \), the position assignment matrix \( \mathcal{P} \) is calculated as follows,

\[
\mathcal{P}(i, j) = p, \quad \forall p \in [1, \mathcal{P}_{max}], \\
s.t.\ \{(i,j)\mid i \in [p, H-p),\ j \in [p, W-p)\},
\]

where \( H \) and \( W \) represent the height and width of the input image, respectively. \( \mathcal{P}_{max} \) is initialized to \( \lfloor H/2 \rfloor \). This design maintains spatial continuity in the row and column dimensions. It reduces the average distance between significant image tokens and instruction tokens, facilitating cross-attention among the image tokens and cross-attention between the image tokens and instruction tokens.

Subsequently, we propose a gradual expansion of the central receptive field to diminish the influence of anchor tokens and enhance the model’s ability to perceive visual elements at varying levels of granularity. Specifically, we reduce \( \mathcal{P}_{max} \) every \( t \) layers, thereby controlling the granularity of perception through position encoding. When \( \mathcal{P}_{max} \) is reduced to 1, the corresponding position encoding transforms into an All-One PE, which perceives more coarse-grained elements. To maintain causal attention, we adjust the attention mask \( \mathcal{M} \) based on each assigned position matrix \( \mathcal{P} \).

By introducing hierarchical position indices, PyPE facilitates multi-granularity perception of visual elements, allowing the model to dynamically adjust its focus to capture both broad contextual information and fine-grained details within visual data. This innovative approach not only aligns more closely with human cognitive processes but also enhances the model’s overall performance in tasks that require both holistic and detailed perception of visual content.

4 Experiment Setup

4.1 Benchmarks
We evaluate PyPE on visual question answering and general multimodal benchmarks, including VQAv2 (Goyal et al., 2017), OK-VQA (Marino
<table>
  <tr>
    <th>Method</th>
    <th>VQAv2</th>
    <th>OK-VQA<sub>val</sub></th>
    <th>GQA</th>
    <th>VizWizQA</th>
    <th>TextVQA<sub>val</sub></th>
    <th>RealWorldQA</th>
    <th>ScienceQA<sup>1</sup></th>
  </tr>
  <tr>
    <th colspan="8"><i>TinyLLaVA-SigLIP-Phi-2</i></th>
  </tr>
  <tr>
    <td>w/ Raster-scan</td>
    <td>78.93</td>
    <td>56.71</td>
    <td>61.07</td>
    <td>34.30</td>
    <td>50.88</td>
    <td>53.99</td>
    <td>71.24</td>
  </tr>
  <tr>
    <td>w/ Concentric</td>
    <td>79.08</td>
    <td>57.35</td>
    <td>61.15</td>
    <td>41.08</td>
    <td>50.77</td>
    <td>53.59</td>
    <td>70.45</td>
  </tr>
  <tr>
    <td>w/ All-One</td>
    <td>78.89</td>
    <td>57.34</td>
    <td>61.33</td>
    <td>42.50</td>
    <td>50.94</td>
    <td>53.59</td>
    <td>70.55</td>
  </tr>
  <tr>
    <td>w/ PyPE (Ours)</td>
    <td><b>79.22</b></td>
    <td><b>57.48</b></td>
    <td><b>61.65</b></td>
    <td><b>44.45</b></td>
    <td><b>51.31</b></td>
    <td><b>54.12</b></td>
    <td><b>71.59</b></td>
  </tr>
  <tr>
    <th colspan="8"><i>LLaVA-1.5-7B</i></th>
  </tr>
  <tr>
    <td>w/ Raster-scan</td>
    <td>78.56</td>
    <td>54.32</td>
    <td>62.12</td>
    <td>50.34</td>
    <td>46.16</td>
    <td>54.80</td>
    <td>66.80</td>
  </tr>
  <tr>
    <td>w/ Concentric</td>
    <td>79.02</td>
    <td>52.70</td>
    <td>62.28</td>
    <td>52.52</td>
    <td>45.84</td>
    <td>54.77</td>
    <td>68.72</td>
  </tr>
  <tr>
    <td>w/ All-One</td>
    <td>79.02</td>
    <td>52.50</td>
    <td>62.00</td>
    <td><b>55.32</b></td>
    <td>45.98</td>
    <td>54.77</td>
    <td>68.32</td>
  </tr>
  <tr>
    <td>w/ PyPE (Ours)</td>
    <td><b>79.15</b></td>
    <td><b>54.96</b></td>
    <td><b>62.71</b></td>
    <td>53.11</td>
    <td><b>46.73</b></td>
    <td><b>55.42</b></td>
    <td><b>69.51</b></td>
  </tr>
  <tr>
    <th colspan="8"><i>LLaVA-1.5-13B</i></th>
  </tr>
  <tr>
    <td>w/ Raster-scan</td>
    <td>79.14</td>
    <td><b>57.38</b></td>
    <td>63.34</td>
    <td>53.75</td>
    <td>48.56</td>
    <td>55.95</td>
    <td>71.15</td>
  </tr>
  <tr>
    <td>w/ Concentric</td>
    <td>79.90</td>
    <td>53.81</td>
    <td>63.26</td>
    <td>56.38</td>
    <td>48.07</td>
    <td>55.42</td>
    <td>70.00</td>
  </tr>
  <tr>
    <td>w/ All-One</td>
    <td><b>79.95</b></td>
    <td>51.40</td>
    <td>63.34</td>
    <td>56.37</td>
    <td>48.15</td>
    <td>54.64</td>
    <td>71.39</td>
  </tr>
  <tr>
    <td>w/ PyPE (Ours)</td>
    <td><b>79.95</b></td>
    <td>55.66</td>
    <td><b>63.52</b></td>
    <td><b>58.06</b></td>
    <td><b>48.90</b></td>
    <td><b>56.86</b></td>
    <td><b>71.54</b></td>
  </tr>
</table>

Table 2: Performance evaluation on visual question answering. We utilize <i>accuracy</i> as the evaluation metric. OK-VQA<sub>val</sub> and TextVQA<sub>val</sub> denote the validation set of OK-VQA and TextVQA, respectively. ScienceQA<sup>1</sup> denote the image subset of ScienceQA. The best results in each setting are in bold.

et al., 2019), GQA (Hudson and Manning, 2019), VizWizQA (Bigham et al., 2010), TextVQA (Singh et al., 2019), RealWorldQA (X.AI, 2024), ScienceQA (Lu et al., 2022), MME (Yin et al., 2024), MMbench (Liu et al., 2025), SEED-Bench (Li et al., 2023a), POPE (Li et al., 2023c), AI2D (Kembhavi et al., 2016), MM-Vet (Yu et al., 2023), MMMU (Yue et al., 2024), MMT-Bench (Ying et al., 2024), and MMStar (Chen et al., 2024a). Refer to Appendix A for more details.

4.2 Implementation Details

To demonstrate the generalizability of our proposed method across models with different parameter sizes, we conduct experiments using three model architectures with 3B, 7B, and 13B parameters. For 3B models, we follow TinyLLaVA (Zhou et al., 2024) to use SigLIP (Zhai et al., 2023) as the visual encoder and Phi-2 (Li et al., 2023d) as the base LLM. For 7B and 13B models, we adopt pre-trained CLIP ViT-L/14 (336<sup>2</sup>) (Radford et al., 2021) as visual encoder and Vicuna v1.5 (Zheng et al., 2023) as the base LLM. Following Liu et al. (2024a), we pretrain the models on CC-558K dataset and finetune them on the mix-665K dataset. All experiments are conducted on 8 NVIDIA A100 and 8 NVIDIA H20 GPUs. See Appendix B for more training and implementation details.

5 Empirical Results and Analysis

We evaluate the visual capabilities of the models trained with the PyPE through various visual question answering and general multimodal benchmarks. This novel position encoding demonstrates highly competitive performance at different scales. Our proposed method consistently delivers top-tier performance across most evaluation metrics, frequently surpassing other baselines.

5.1 Results of Visual Question Answering Benchmarks

To rigorously evaluate the capabilities of our models in general visual question answering tasks, we conduct comprehensive assessments across a diverse array of state-of-the-art benchmarks. The results presented in Tables 1 and 2 indicate that the PyPE series demonstrates exceptional performance across all benchmarks, with the three variants consistently achieving or surpassing baseline performance. In the MME benchmark, PyPE exhibits a superior understanding of visual content at various levels of granularity. It retains a coarse-grained perception capability comparable to that of All-One PE while outperforming both Raster-scan and Concentric PE in terms of fine-grained perception. On the RealWorldQA benchmark, which assesses real-world spatial comprehension, PyPE achieves scores of 54.12, 55.42, and 56.86 for the 3B, 7B, and 13B variants, respectively. These results exceed all baseline performances and reflect an enhanced understanding of physical environments. VizWizQA is a dataset comprising images captured by visually impaired individuals using mobile phones, accompanied by recorded spoken questions. The images
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">POPE</th>
    <th colspan="2">MMBench</th>
    <th colspan="2">SEED<sup>1</sup></th>
    <th colspan="2">AI2D</th>
    <th colspan="2">MM-Vet</th>
    <th colspan="2">MMMU</th>
    <th colspan="2">MMT-Bench</th>
    <th colspan="2">MMSstar</th>
  </tr>
  <tr>
    <th>rand</th>
    <th>pop</th>
    <th>adv</th>
    <th>en</th>
    <th>cn</th>
    <th>68.54</th>
    <th>59.75</th>
    <th>33.00</th>
    <th>33.80</th>
    <th>48.93</th>
    <th>37.37</th>
    <th>68.51</th>
    <th>60.98</th>
    <th>33.40</th>
    <th>33.60</th>
    <th>48.86</th>
    <th>38.44</th>
    <th>68.25</th>
    <th>61.20</th>
    <th>32.70</th>
    <th>33.80</th>
    <th>48.00</th>
    <th>38.06</th>
  </tr>
  <tr>
    <td><i>TinyLLaVA-SigLIP-Phi-2</i></td>
    <td>88.50</td>
    <td>86.93</td>
    <td>85.60</td>
    <td>67.88</td>
    <td><b>45.07</b></td>
    <td>88.63</td>
    <td>87.27</td>
    <td>85.67</td>
    <td>67.83</td>
    <td>43.22</td>
    <td>68.51</td>
    <td>60.98</td>
    <td>33.40</td>
    <td>33.60</td>
    <td>48.86</td>
    <td>38.44</td>
    <td>88.53</td>
    <td>87.40</td>
    <td>86.00</td>
    <td>66.48</td>
    <td>43.11</td>
    <td>68.25</td>
    <td>61.20</td>
    <td>32.70</td>
    <td>33.80</td>
    <td>48.00</td>
    <td>38.06</td>
  </tr>
  <tr>
    <td>w/ PyPE (Ours)</td>
    <td><b>89.07</b></td>
    <td><b>87.70</b></td>
    <td><b>85.73</b></td>
    <td><b>68.33</b></td>
    <td>43.95</td>
    <td>68.55</td>
    <td>61.53</td>
    <td><b>35.00</b></td>
    <td>33.70</td>
    <td><b>48.93</b></td>
    <td><b>38.89</b></td>
  </tr>
  <tr>
    <td><i>LLaVA-1.5-7B</i></td>
    <td>88.33</td>
    <td>87.13</td>
    <td>85.63</td>
    <td>64.97</td>
    <td>57.90</td>
    <td>66.10</td>
    <td>55.25</td>
    <td>30.80</td>
    <td>31.00</td>
    <td>47.94</td>
    <td>35.25</td>
    <td>87.83</td>
    <td>86.40</td>
    <td>85.43</td>
    <td>65.41</td>
    <td>57.79</td>
    <td>66.31</td>
    <td>54.83</td>
    <td>29.70</td>
    <td>31.00</td>
    <td>49.02</td>
    <td>35.41</td>
    <td>87.30</td>
    <td>86.57</td>
    <td>85.53</td>
    <td>65.47</td>
    <td>55.89</td>
    <td>66.41</td>
    <td>54.73</td>
    <td>29.90</td>
    <td>30.70</td>
    <td>48.99</td>
    <td>36.24</td>
  </tr>
  <tr>
    <td>w/ PyPE (Ours)</td>
    <td>88.27</td>
    <td><b>87.43</b></td>
    <td><b>85.67</b></td>
    <td><b>66.65</b></td>
    <td><b>58.63</b></td>
    <td><b>67.01</b></td>
    <td><b>55.63</b></td>
    <td><b>31.10</b></td>
    <td><b>31.10</b></td>
    <td><b>49.70</b></td>
    <td><b>36.51</b></td>
  </tr>
  <tr>
    <td><i>LLaVA-1.5-13B</i></td>
    <td>88.77</td>
    <td><b>87.70</b></td>
    <td><b>85.90</b></td>
    <td>67.74</td>
    <td>63.17</td>
    <td>67.65</td>
    <td>59.49</td>
    <td><b>37.30</b></td>
    <td><b>33.20</b></td>
    <td>49.82</td>
    <td>36.81</td>
    <td>87.90</td>
    <td>87.13</td>
    <td>85.80</td>
    <td>68.89</td>
    <td>62.67</td>
    <td>67.59</td>
    <td>58.55</td>
    <td>35.90</td>
    <td>32.70</td>
    <td>48.54</td>
    <td>37.33</td>
    <td>87.93</td>
    <td>87.13</td>
    <td>85.77</td>
    <td>67.99</td>
    <td>63.06</td>
    <td>67.47</td>
    <td>58.84</td>
    <td>36.00</td>
    <td>32.90</td>
    <td>49.38</td>
    <td>37.32</td>
  </tr>
  <tr>
    <td>w/ PyPE (Ours)</td>
    <td>88.03</td>
    <td>86.97</td>
    <td>85.47</td>
    <td><b>69.23</b></td>
    <td><b>63.45</b></td>
    <td><b>68.50</b></td>
    <td><b>59.59</b></td>
    <td>36.60</td>
    <td><b>33.20</b></td>
    <td><b>50.40</b></td>
    <td><b>38.71</b></td>
  </tr>
</table>

Table 3: Evaluation on general multimodal benchmarks. We utilize accuracy as the evaluation metric. SEED<sup>1</sup> denotes the image subset of SEED-Bench. The best results in each setting are in bold.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">MME<sup>P</sup></th>
    <th colspan="2">OK-VQA<sub>val</sub></th>
    <th colspan="2">TextVQA<sub>val</sub></th>
    <th colspan="2">MMSstar</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td colspan="9"><i>TinyLLaVA-SigLIP-Phi-2</i></td>
  </tr>
  <tr>
    <td>PyPE 1x</td>
    <td>1479.53</td>
    <td>56.99</td>
    <td>50.13</td>
    <td>37.31</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PyPE 2x</td>
    <td><b>1500.66</b></td>
    <td><b>57.48</b></td>
    <td><b>51.31</b></td>
    <td><b>38.89</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PyPE 3x</td>
    <td>1470.45</td>
    <td>57.29</td>
    <td>50.28</td>
    <td>38.32</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PyPE 4x</td>
    <td>1466.70</td>
    <td>55.89</td>
    <td>50.59</td>
    <td>37.23</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="9"><i>LLaVA-1.5-7B</i></td>
  </tr>
  <tr>
    <td>PyPE 1x</td>
    <td>1507.19</td>
    <td>52.73</td>
    <td><b>46.77</b></td>
    <td>34.82</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PyPE 2x</td>
    <td><b>1542.19</b></td>
    <td><b>54.96</b></td>
    <td>46.73</td>
    <td><b>36.51</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PyPE 3x</td>
    <td>1503.95</td>
    <td>52.87</td>
    <td>45.99</td>
    <td>36.18</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PyPE 4x</td>
    <td>1497.18</td>
    <td>51.76</td>
    <td>46.20</td>
    <td>35.79</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td colspan="9"><i>LLaVA-1.5-13B</i></td>
  </tr>
  <tr>
    <td>PyPE 1x</td>
    <td>1608.01</td>
    <td>50.53</td>
    <td>48.60</td>
    <td>35.89</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PyPE 2x</td>
    <td><b>1629.41</b></td>
    <td>55.66</td>
    <td><b>48.90</b></td>
    <td><b>38.71</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PyPE 3x</td>
    <td>1583.84</td>
    <td>54.90</td>
    <td>48.52</td>
    <td>36.55</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>PyPE 4x</td>
    <td>1607.63</td>
    <td><b>57.42</b></td>
    <td>48.09</td>
    <td>37.07</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 4: Analysis of the descent interval t. PyPE t× denotes using PyPE with interval t. MME<sup>P</sup> denotes MME-Perception.

in this dataset tend to exhibit relatively low clarity, with subjects occupying a significant portion of the frame. Consequently, as shown in Table 2, All-One PE demonstrates competitive performance on this dataset, while our proposed PyPE exhibits superior zero-shot performance on both VizWizQA and ScienceQA. This improvement can be attributed to the flexible receptive field enabled by PyPE.

5.2 Results of General Multimodal Benchmarks

As illustrated in Table 3, the PyPE series demonstrates exceptional performance on mainstream general multimodal benchmarks. In the MMStar benchmark, which is designed to assess genuine multimodal capabilities using visually indispensable samples, PyPE outperforms all baseline models. On MM-Vet, which evaluates the integration of core vision-language capabilities across 16 complex multimodal tasks, the 3B model of PyPE achieves an impressive score of 35.00, significantly surpassing the scores of 33.00 and 33.40 obtained by Raster-scan and Concentric PE, respectively. In the MMT-Bench evaluation, which assesses advanced reasoning and instruction-following across 32 core meta-tasks and 162 subtasks in multimodal understanding, PyPE markedly exceeds baseline performance, demonstrating its ability to apply expert knowledge and execute deliberate visual recognition, localization, reasoning, and planning. On MMBench, which evaluates fine-grained abilities across 20 dimensions, PyPE exhibits strong performance, matching or leading the state-of-the-art. Additionally, we test the methods on AI2D, a benchmark focusing on multiple-choice questions related to scientific diagrams containing text. The results indicate that PyPE achieves state-of-the-art performance and demonstrates a strong comprehension of textual content within images.

5.3 Analysis of the Descent Interval

As shown in Table 4, we evaluate the performance of different models using PyPE with varying descent intervals on VQA and general multimodal benchmarks. Across all models, a moderate de-
Raster-scan: The image features a painting of a dog, dressed in a Renaissance-style outfit, sitting in a chair. The dog is wearing a hat and a scarf, giving it a unique and artistic appearance. The painting captures the dog's attention, as it looks directly at the viewer. The background of the painting includes a dining table and a chair, adding to the overall composition of the artwork. The scene is reminiscent of a famous painting of a woman, but with the dog taking the central role in this version.

All-one: The image features a painting of a woman with a brown dog sitting next to her. The woman is wearing a hat, and the dog is positioned in front of her, looking up at the viewer. The painting is quite detailed, capturing the woman's facial expression and the dog's attentive gaze. In the background, there is a boat visible, adding an interesting element to the scene. The overall composition of the painting is quite engaging, with the woman and the dog as the main subjects.

Concentric: The image features a painting of a dog, possibly a dachshund, dressed in a Renaissance-style outfit, resembling a woman. The dog is wearing a hat and a scarf, giving it a unique and artistic appearance. The painting is set against a backdrop of a landscape, which includes a river and a mountain. The dog's outfit and the surrounding scenery create a captivating and whimsical scene.

PyPE (ours): The image features a painting of a woman, possibly a nun, with a dog's face. The dog's face is painted in the style of a famous painting, giving it a unique and artistic appearance. The woman is wearing a long robe, and the dog's face is positioned in the center of the painting. The background of the painting includes a landscape with a river, adding depth and context to the scene. The combination of the woman's attire and the dog's face creates an interesting and intriguing visual experience.

![Illustration of the multi-granularity perception capability of PyPE with a sample from LLaVA-Bench. The case study is based on LLaVA-1.5-7B and the query is "Describe this photo in detail". The misunderstandings and hallucinations of visual contents are highlighted in red. We also provide a corresponding layer-wise attention visualization of PyPE, with the heatmap arranged from the upper left to the lower right, indicating layers 1 to 32.](page_184_693_1080_180.png)

Figure 4: Illustration of the multi-granularity perception capability of PyPE with a sample from LLaVA-Bench. The case study is based on LLaVA-1.5-7B and the query is *Describe this photo in detail*. The misunderstandings and hallucinations of visual contents are highlighted in red. We also provide a corresponding layer-wise attention visualization of PyPE, with the heatmap arranged from the upper left to the lower right, indicating layers 1 to 32.

scent interval PyPE 2x generally provides the best or near-best performance, which strikes a balance between the model’s ability to handle perception (MME), external knowledge integration (OK-VQA), text comprehension (TextVQA), and vision-critical tasks (MMStar). While the 2x interval is generally optimal, there are exceptions, such as the LLaVA-1.5-13B model performing best on OK-VQA with a 4x interval. This indicates that larger models might benefit from longer intervals for specific tasks.

5.4 Qualitative Results on LLaVA-Bench
Figure 4 demonstrates a case study on how, given identical prompts and images, other baselines mis-perceive or inadequately process visual information, resulting in the generation of hallucinatory content. For instance, in the displayed example, the baseline methods exhibit object hallucinations, identifying nonexistent items such as *dining table*, *hat*, *scarf*, and *boat*. In contrast, the implementation of PyPE notably mitigates these hallucination issues while simultaneously maintaining the coherence and informativeness of the output text. This can be attributed to the multi-scale visual modeling capability afforded by the dynamic local receptive fields of PyPE, in conjunction with the stable global receptive fields. Furthermore, the visualization results of layer-wise attention indicate that our proposed method effectively alleviates the phenomenon of "aggregation pattern", thereby creating a synergistic effect with the former. Refer to Appendix C for a more in-depth analysis of anchor tokens and Appendix E for more case studies.

6 Conclusion
In this work, we conduct an in-depth analysis of how visual position encoding affects visual perception in VLMs (Vision-Inlanguage Models), particularly from the aspect of long-term decay and the "aggregation pattern" (also known as "anchor tokens"). Our findings indicate that conventional visual position encoding methods are constrained by the "aggregation pattern" derived from LLMs (Large Language Models) and lack multi-scale perceptual capabilities. To address these limitations, we introduce Pyramid-descent Visual Position Encoding (PyPE), a novel approach designed to enhance the perception of visual tokens within VLMs. Extensive experiments across multiple benchmarks and VLM families demonstrate the efficacy of
PyPE in addressing these challenges and ensuring a thorough understanding of visual content.

Limitations

Although PyPE demonstrates exceptional performance in enhancing the overall capabilities of Vision-language Models (VLMs), it is currently limited to single-frame images and has not yet been extended to video and other modalities. Future research will focus on effectively integrating the temporal dimension for unified position encoding and extending PyPE to a broader range of VLMs.

Acknowledgement

This work is supported by Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology (Grant No. 2024B1212010006).

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966.

Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandy White, Samual White, et al. 2010. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333–342.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. 2024a. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330.

Wei-Ge Chen, Irina Spiridonova, Jianwei Yang, Jianfeng Gao, and Chunyuan Li. 2023. Llava-interactive: An all-in-one demo for image chat, segmentation, generation and editing. arXiv preprint arXiv:2311.00571.

Zhanpeng Chen, Chengjin Xu, Yiyuan Qi, and Jian Guo. 2024b. Mlm is a strong reranker: Advancing multimodal retrieval-augmented generation via knowledge-enhanced reranking and noise-injected training. arXiv preprint arXiv:2407.21439.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024c. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198.

Xiangxiang Chu, Jianlin Su, Bo Zhang, and Chunhua Shen. 2024. Visionllama: A unified llama interface for vision tasks. arXiv preprint arXiv:2403.00522.

Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, and Chunhua Shen. 2021. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882.

Alexey Dosovitskiy. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.

S Hochreiter. 1997. Long short-term memory. Neural Computation MIT-Press.

Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395.

Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427.

Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. 2023. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems, 36:72096–72109.
Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709.

Md Amirul Islam, Sen Jia, and Neil DB Bruce. 2020. How much position information do convolutional neural networks encode? arXiv preprint arXiv:2001.08248.

Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798.

Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2016. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–251. Springer.

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023a. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023c. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355.

Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023d. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024b. Visual instruction tuning. Advances in neural information processing systems, 36.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2025. Mm-bench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pages 216–233. Springer.

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521.

Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. 2024. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402.12376.

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11–20.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204.

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruvi Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. 2025. Mml: methods, analysis and insights from multimodal llm pre-training. In European Conference on Computer Vision, pages 304–323. Springer.

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326.
Jianlin Su, Muradtha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.

Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Label words are anchors: An information flow perspective for understanding in-context learning. arXiv preprint arXiv:2305.14160.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Wenhui Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. 2021. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 568–578.

X.AI. 2024. Grok-1.5 vision preview. https://x.ai/blog/grok-1.5v.

Yuxin Xie, Zhihong Zhu, Xianwei Zhuang, Liming Liang, Zhichang Wang, and Yuexian Zou. 2024. Gpa: Global and prototype alignment for audio-text retrieval. In Interspeech 2024, pages 5078–5082.

Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. 2024. Mitigating object hallucination via concentric causal attention. arXiv preprint arXiv:2410.15926.

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review, page nwae403.

Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. 2024. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruqii Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556–9567.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986.

Kaichen Zhang, Bo Li, Petyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. 2024. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. 2024. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289.

A Benchmarks

Visual Question Answering The VQAv2 dataset is currently the largest available dataset for visual question answering. OK-VQA includes questions that necessitate external knowledge beyond the multimodal inputs provided. GQA is specifically designed to assess the reasoning capabilities of the model. VizWizQA is composed of question-answer pairs derived from visually impaired users. TextVQA places a greater emphasis on evaluating the model’s ability to comprehend text within natural scenes. RealWorldQA is a benchmark specifically designed to evaluate the spatial understanding capabilities of multimodal AI models in real-world contexts. ScienceQA comprises multimodal multiple-choice questions across a diverse range of science topics. These datasets are strategically selected to comprehensively evaluate our method’s capacity to understand and reason across diverse visual contexts and knowledge domains.
(a) Raster-scan PE.

(b) All-One PE.

(c) Concentric PE.

(d) PyPE.

Figure 5: Visualization of anchor tokens in baselines and PyPE.

General Multimodal Benchmarks  MME measures both perception and cognition abilities on a total of 14 subtasks. MMBench comprehensively evaluates a model’s multimodal capabilities in both Chinese and English contexts. SEED-Bench focuses on assessing generative comprehension in Vision-language Models. POPE evaluates the extent of multimodal hallucinations present in a model. AI2D assesses a model’s ability to interpret scientific diagram inputs. MM-Vet evaluates the multimodal conversational abilities of a model using GPT-4 as a benchmark. MMMU is designed to assess multimodal models on extensive multi-disciplinary tasks that require college-level subject knowledge and deliberate reasoning. MMT-Bench is a comprehensive benchmark developed to evaluate VLMs across a wide range of multimodal tasks that necessitate expert knowledge and deliberate visual recognition, localization, reasoning, and planning. MMstar is a premier, vision-critical multimodal benchmark comprising 1,500 challenge samples meticulously curated by human experts.

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>Pretrain</th>
    <th>Finetune</th>
  </tr>
  <tr>
    <td>batch size</td>
    <td>256</td>
    <td>128</td>
  </tr>
  <tr>
    <td>lr</td>
    <td>1e-3</td>
    <td>2e-5</td>
  </tr>
  <tr>
    <td>lr schedule</td>
    <td colspan="2">cosine decay</td>
  </tr>
  <tr>
    <td>lr warmup ratio</td>
    <td colspan="2">0.03</td>
  </tr>
  <tr>
    <td>weight decay</td>
    <td colspan="2">0</td>
  </tr>
  <tr>
    <td>epoch</td>
    <td colspan="2">1</td>
  </tr>
  <tr>
    <td>optimizer</td>
    <td colspan="2">AdamW</td>
  </tr>
  <tr>
    <td>DeepSpeed stage</td>
    <td>2</td>
    <td>3</td>
  </tr>
</table>

Table 5: Hyperparameters of TinyLLaVA-SigLIP-Phi-2 and LLaVA-1.5-7B/13B.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">RefCOCO</th>
    <th colspan="3">RefCOCO+</th>
    <th colspan="2">RefCOCOg</th>
  </tr>
  <tr>
    <th>val</th>
    <th>test-A</th>
    <th>test-B</th>
    <th>val</th>
    <th>test-A</th>
    <th>test-B</th>
    <th>val</th>
    <th>test</th>
  </tr>
  <tr>
    <td><i>TinyLLaVA-SigLIP-Phi-2</i></td>
    <td colspan="8"></td>
  </tr>
  <tr>
    <td>w/ Raster-scan</td>
    <td><b>31.85</b></td>
    <td>15.77</td>
    <td>44.29</td>
    <td><b>31.28</b></td>
    <td>18.65</td>
    <td><b>40.83</b></td>
    <td>56.86</td>
    <td>56.98</td>
  </tr>
  <tr>
    <td>w/ Concentric</td>
    <td>30.89</td>
    <td>15.97</td>
    <td>42.22</td>
    <td>31.16</td>
    <td><b>19.31</b></td>
    <td>38.77</td>
    <td><b>59.45</b></td>
    <td><b>58.74</b></td>
  </tr>
  <tr>
    <td>w/ All-One</td>
    <td>29.13</td>
    <td>15.24</td>
    <td>39.11</td>
    <td>30.64</td>
    <td>18.38</td>
    <td>38.34</td>
    <td>54.81</td>
    <td>55.28</td>
  </tr>
  <tr>
    <td>w/ PyPE (Ours)</td>
    <td><b>31.33</b></td>
    <td><b>16.02</b></td>
    <td><b>45.13</b></td>
    <td><b>31.86</b></td>
    <td><b>19.40</b></td>
    <td><b>42.25</b></td>
    <td><b>59.72</b></td>
    <td><b>59.79</b></td>
  </tr>
  <tr>
    <td><i>LLaVA-1.5-7B</i></td>
    <td colspan="8"></td>
  </tr>
  <tr>
    <td>w/ Raster-scan</td>
    <td>34.19</td>
    <td><b>18.07</b></td>
    <td>46.89</td>
    <td><b>34.30</b></td>
    <td><b>21.63</b></td>
    <td><b>43.53</b></td>
    <td>61.21</td>
    <td>59.40</td>
  </tr>
  <tr>
    <td>w/ Concentric</td>
    <td>32.23</td>
    <td><b>16.51</b></td>
    <td>42.49</td>
    <td>32.66</td>
    <td>20.00</td>
    <td>40.41</td>
    <td>59.72</td>
    <td>58.47</td>
  </tr>
  <tr>
    <td>w/ All-One</td>
    <td>32.99</td>
    <td>16.46</td>
    <td>41.26</td>
    <td>33.28</td>
    <td>20.73</td>
    <td>39.83</td>
    <td><b>63.07</b></td>
    <td><b>61.90</b></td>
  </tr>
  <tr>
    <td>w/ PyPE (Ours)</td>
    <td><b>35.16</b></td>
    <td><b>16.51</b></td>
    <td><b>48.04</b></td>
    <td><b>34.17</b></td>
    <td><b>21.22</b></td>
    <td><b>41.46</b></td>
    <td><b>64.62</b></td>
    <td><b>64.13</b></td>
  </tr>
  <tr>
    <td><i>LLaVA-1.5-13B</i></td>
    <td colspan="8"></td>
  </tr>
  <tr>
    <td>w/ Raster-scan</td>
    <td>36.86</td>
    <td>19.29</td>
    <td>50.01</td>
    <td>36.12</td>
    <td>22.37</td>
    <td><b>43.59</b></td>
    <td><b>63.66</b></td>
    <td>60.96</td>
  </tr>
  <tr>
    <td>w/ Concentric</td>
    <td>35.87</td>
    <td>18.54</td>
    <td>48.17</td>
    <td>36.07</td>
    <td>21.94</td>
    <td>42.65</td>
    <td>61.66</td>
    <td>61.07</td>
  </tr>
  <tr>
    <td>w/ All-One</td>
    <td>36.84</td>
    <td>19.06</td>
    <td>49.16</td>
    <td><b>37.10</b></td>
    <td>22.71</td>
    <td>41.72</td>
    <td>61.58</td>
    <td>59.75</td>
  </tr>
  <tr>
    <td>w/ PyPE (Ours)</td>
    <td><b>37.81</b></td>
    <td><b>21.82</b></td>
    <td><b>51.88</b></td>
    <td><b>37.14</b></td>
    <td><b>25.74</b></td>
    <td><b>44.73</b></td>
    <td><b>63.16</b></td>
    <td><b>62.59</b></td>
  </tr>
</table>

Table 6: Performance comparison on referring expression comprehension tasks. We use CIDEr (Vedantam et al., 2015) to evaluate the quality of the descriptions. The highest results in each setting are indicated in bold, while the second-best results are underlined.

B Hyperparameters and More Implementation Details

We show the training hyperparameters for both first-stage vision-language alignment pretraining and the second-stage visual instruction tuning in Table 5. We use LMMs-Eval (Zhang et al., 2024) to conduct experiments on VQA and general multi-modal benchmarks.

C Visualization of Anchor Tokens

To further analyze the aggregating attention pattern, we visualize the attention score of each patch in the first 16 layers. As illustrated in Figure 5, both the All-One PE and the Concentric PE exhibit a relatively uniform distribution of attention in the initial two layers. However, a significant phenomenon of attention aggregation emerges in the subsequent layers, where non-anchor patches demonstrate a suppression of attention, particularly pronounced in Concentric PE. Though Raster-scan PE shows slight improvement, the attention in each layer tends to be preferentially allocated to patches that are closer to the instruction token, resulting in a discontinuous and fragmented attention pattern. This indicates a limitation of the Raster-scan PE in effectively modeling patches with similar semantics. In contrast, PyPE not only reduces the number of anchor tokens but also yields significantly lower attention scores for these tokens compared to the baselines, thereby facilitating the model’s exploration of image details more effectively. Furthermore, in each layer, the attention distribution of the PyPE is more continuous, highlighting the superiority of our proposed method in modeling semantically similar information.

D Performance on Referring Expression Comprehension

In the context of the visual localization task, we evaluate PyPE using the RefCOCO, RefCOCO+, and RefCOCOg datasets (Kazemzadeh et al., 2014; Mao et al., 2016). The results, presented in Table 6, indicate that PyPE achieves top-tier performance among baselines. Its superior structural design enables PyPE to effectively perceive intricate details within images, resulting in significant improvements over baseline models. The performance of PyPE underscores its potential to advance the field of visual localization and its applicability in real-world scenarios that require precise visual understanding.

E More Case Studies

We provide more examples of visual description in Table 7. As illustrated in the table, our proposed PyPE exhibits a reduced incidence of generating visual hallucinations or misunderstandings. More importantly, compared to other baseline methods,
PyPE demonstrates a finer granularity in perceiving visual elements, thereby uncovering additional information, such as "blueberries" in the first example and "My joke website (funny joke push to reveal punchline)" in the second example. To further analyze the model’s attention distribution across each decoder layer, we visualize the corresponding attention values for these examples. The results in Figure 6, 7, and 8 indicate that while other baselines remain hindered by anchor tokens, PyPE consistently mitigates this issue, facilitating a more rational allocation of attention.
Visual input example, Visual Description 1

<table>
  <tr>
    <th>User</th>
    <th>Describe this photo in detail.</th>
  </tr>
  <tr>
    <td>Raster-scan</td>
    <td>The image displays a refrigerator filled with various food items, including milk, juice, and fruits. There are multiple bottles of milk placed throughout the refrigerator, with some located on the top shelves and others on the bottom shelves. In addition to the milk, there are several bottles of juice, some of which are placed on the top shelves and others on the bottom shelves. A few carrots can be seen in the refrigerator, with some placed on the top shelves and others on the bottom shelves. The refrigerator is well-stocked with a variety of food items, making it a convenient place to store and access groceries.</td>
  </tr>
  <tr>
    <td>All-One</td>
    <td>The image displays a refrigerator filled with various food items. Inside the refrigerator, there are several bottles, including one on the left side, two in the middle, and one on the right side. There are also multiple carrots, with some placed in the middle and others on the left side of the refrigerator. In addition to the carrots, there are several strawberries, with some located on the left side and others on the right side of the refrigerator. A bowl can be seen on the left side of the refrigerator, possibly containing more food items. The refrigerator is well-stocked with a variety of fruits and vegetables, making it a healthy and diverse selection.</td>
  </tr>
  <tr>
    <td>Concentric</td>
    <td>The image displays a refrigerator filled with various food items, including a variety of fruits and vegetables. There are several carrots placed throughout the refrigerator, with some in the top left, middle, and bottom sections. A few strawberries can be seen in the bottom left corner of the refrigerator. In addition to the fresh produce, there are multiple bottles of milk and juice stored in the refrigerator. Some of these bottles are located in the top left, middle, and bottom sections of the refrigerator. The refrigerator is well-stocked with a diverse selection of food items, making it a convenient place to store and access groceries.</td>
  </tr>
  <tr>
    <td><b>PyPE (ours)</b></td>
    <td>The image displays a refrigerator filled with various food items, including several containers of berries, milk, and other foods. There are multiple bottles of milk placed throughout the refrigerator, with some located on the top shelves and others on the bottom shelves. In addition to the milk, there are several containers of berries, such as strawberries and blueberries, spread across the refrigerator. The refrigerator is well-stocked with a diverse range of food items, making it a convenient place to store and access groceries.</td>
  </tr>
</table>

Visual Description 2

<table>
  <tr>
    <th>User</th>
    <th>Describe this photo in detail.</th>
  </tr>
  <tr>
    <td>Raster-scan</td>
    <td>The image features a handwritten sign with a joke written on it. The sign is placed on a table, and it appears to be a website or a piece of paper with a humorous message. The joke is written in a foreign language, which adds a unique touch to the sign. The sign is placed in the center of the image, drawing attention to its content.</td>
  </tr>
  <tr>
    <td>All-One</td>
    <td>The image features a handwritten sign with a joke website written on it. The sign is placed on a white surface, possibly a piece of paper or a wall. The joke website is written in blue ink, and the text is accompanied by a question mark, indicating that the joke is a riddle or a puzzle. The sign is placed in the center of the image, drawing attention to the joke website.</td>
  </tr>
  <tr>
    <td>Concentric</td>
    <td>The image features a handwritten sign with a joke on it, likely meant to be displayed on a website. The joke is written in a foreign language, and the sign is accompanied by a question mark, indicating that it is meant to be a humorous or amusing statement. The sign is placed on a white background, which contrasts with the dark blue writing.</td>
  </tr>
  <tr>
    <td><b>PyPE (ours)</b></td>
    <td>The image features a handwritten sign with a joke on it, likely meant to be displayed on a website. The joke is written in blue ink, and it reads, "My joke website (funny joke push to reveal punchline)." The sign is placed on a white background, which contrasts with the blue ink used for the joke. The handwriting style suggests a casual and humorous tone, likely meant to entertain or amuse the viewers.</td>
  </tr>
</table>

Table 7: More examples from LLaVA-Bench. The misunderstandings and hallucinations of visual contents are highlighted in red. The descriptions that are not mentioned in baselines but are accurately represented by PyPE are highlighted in green.
(a) Raster-scan

(b) Concentric

(c) All-One

(d) PyPE

Figure 6: Layer-wise attention visualization (left to right, up to down) of the example from Figure 4.
(a) Raster-scan

(b) Concentric

(c) All-One

(d) PyPE

Figure 7: Layer-wise attention visualization (left to right, up to down) of the first example from Table 7.
(a) Raster-scan

(b) Concentric

(c) All-One

(d) PyPE

Figure 8: Layer-wise attention visualization (left to right, up to down) of the second example from Table 7.

Reviewer paper summaries: This paper focuses on the problem of visual encoding in vision-language models. Based on the analysis on different methods for encoding visual position, including the traditional raster-scan order and concentric position encoding, this paper introduces a novel position encoding method named of PyPE which boost the general multi-modal performance of existing VLMs.

This paper proposes a novel method called Pyramid-descent Visual Position Encoding (PyPE), which aims to enhance the visual perception capabilities of Vision-Language Models (VLMs). The work provides a detailed analysis of the role of visual position encoding in VLMs, particularly addressing issues such as long-term decay and "aggregation patterns." PyPE is introduced as a solution to these challenges, and its consistent effectiveness across VLMs of varying scales is thoroughly validated through extensive experiments.
Summary of strengths: 1. The motivation of this paper is strong and clearly described.
2. This paper provides a comprehensive analysis on existing visual encoding methods.
3. The proposed PyPE is well designed and can be easily applied to existing vision-language models.

The key contribution of this paper lies in proposing a novel visual position encoding method, Pyramid-descent Visual Position Encoding (PyPE), which effectively mitigates long-term decay and "aggregation pattern" issues in VLMs while enhancing the model's multi-granularity perception of visual information. 

This approach is both simple and effective, applicable to any VLM architecture, and its superior performance is rigorously demonstrated through extensive experiments.
Summary of weaknesses: 1. The main claims made in this paper are not well clarified.

1.1 It is not clear how could PyPE alleviate the "aggregation pattern".

1.2 In Table 2~3, the improvement brought by PyPE is not significant and not consistent across different datasets.

1.3 The idea of PyPE still follows the assumption that important information are mainly located at the center of the image, which may not holds for all situations.

2. The conducted experiments are not comprehensive:

2.1. lack of comparison between the proposed PyPE and 2D RoPE that is utilized in Qwen2-VL.

2.2 lack of hallucination diagnosis, only results on POPE is not persuasive enough.

2.3 The experiments are merely based on Vicuna, results on advanced backbones should be considered.

1. While PyPE avoids the assumption that "all important elements are located at the image center" (unlike Concentric PE), it may still impose certain implicit priors on the distribution of visual elements. For instance, by assigning position indices from the periphery to the center, PyPE might work better for centrosymmetric or near-centrosymmetric images (e.g., natural scenes with a clear focal point). 
However, its effectiveness could diminish for asymmetrically composed or complex layouts (e.g., abstract art, densely cluttered scenes), where the predefined pyramid descent may not align well with the actual semantic distribution.

2. PyPE achieves multi-granularity perception by progressively expanding the receptive field toward the center. While this design is intuitive, it might assume a specific hierarchy of visual granularity. For images with extremely fine-grained details (e.g., high-resolution textures, microscopic structures), the fixed pyramid descent could fail to capture nuanced local patterns, as the coarse-to-fine progression may inadvertently suppress high-frequency features.
Comments, suggestions and typos: 1. Some details are not well presented in this paper:

1.1 how to set the position indexes for the subsequent textual tokens? In Figure 3, when $P_{max}$ is reduced, should we reduce the position indexes of the subsequent tokens simultaneously?

2. experiments with unlocked the vision transformer are suggested. 

I think the raster-scan order information is already contained in the pre-trained visual encoder, could PyPE also applied to the visual encoder (also unlocking the visual encoder), making it consistent with the VLM?

No.
Overall assessment scores: 3

3.5