## 2 Materials and Methods
### Methods
#### 2.2.1 Overview
Assuming the n-th query vector and the m-th key vector are \(q_{m},k_{m}\in R^{1\times d_{head}}\), the application form of RoPE is

Figure 5:9

Figure 6:0

For images, a common method of extending one-dimensional position embeddings to two-dimensional embeddings is to apply embedding operations separately on each axis. This approach captures the positional relationships in the two-dimensional space by applying position encoding separately to the x-axis and y-axis. At the same time, representing the token's positional information as a two-dimensional vector helps the model more accurately capture the spatial correlations of the input data. Figure 6:1

Figure 6:2

Figure 6:3

The range of position indices \(p_{n}^{x},p_{n}^{y}\) in the spatial dimension needs to be scaled to ensure the frequency's validity. Therefore, the frequency \(\theta_{i}\) in RoPE is accordingly scaled down by the square root. This scaling operation effectively adjusts the frequency scale to accommodate positional information in the two-dimensional space, thereby enhancing the model's expressive power and generalization performance when processing two-dimensional inputs. Figure 6:4

The axial frequency evenly divides the original embedding dimension into two parts, with one part used to represent the position on the x-axis and the other part used to represent the position on the y-axis. Figure 6:5

Figure 6:6Here, \(Re[\cdot]\) denotes taking the real part of the attention score to avoid the influence of the imaginary part, and \(e^{i(\ldots)}\) represents the relative position rotation embedding introduced through the complex exponential form. Furthermore, by introducing mixed frequencies, the limitations of axial frequencies are overcome. Figure 6:7

In the process of introducing relative positional information, the relative positional information is represented by calculating the position difference \(p_{n}-p_{m}\). Figure 6:8

Figure 6:9

\(p_{n}^{x}-p_{m}^{x}\) and \(p_{n}^{y}-p_{m}^{y}\) represents the differences between positions n and m along the x-axis and y-axis, respectively. The differences are substituted into the rotation matrix formula to model the relative positional information between the query vector and the key vector. \[X_{gaf}=H\textit{swish}\Big{(}BN\Big{(}Conv_{3\times 3}\Big{(}X_{gaf}\Big{)} \Big{)}\Big{)} \tag{1}\]0

By applying the rotation encoding directly to the dot product between the query vector and the key vector, relative positional information is effectively introduced. \[X_{gaf}=H\textit{swish}\Big{(}BN\Big{(}Conv_{3\times 3}\Big{(}X_{gaf}\Big{)} \Big{)}\Big{)} \tag{1}\]1

In this formula, \(q_{n}\cdot k_{m}^{*}\) represents the dot product in the complex domain, where \(k_{m}^{*}\) is the complex conjugate of the key vector. C represents the complex number field, used for handling complex number operations. According to the relative position encoding principle of RoPE, the rotated positional information is integrated into the calculation of attention scores in a specific manner. \[X_{gaf}=H\textit{swish}\Big{(}BN\Big{(}Conv_{3\times 3}\Big{(}X_{gaf}\Big{)} \Big{)}\Big{)} \tag{1}\]2

\(\theta_{t}^{x}\) and \(\theta_{t}^{y}\) control the frequency of positional differences along the x-axis and y-axis, respectively. \(t\in\{0,1,\ldots,d_{head/2}\}\), where \(d_{head}\) represents the dimension of each head. By learning these parameters, the model is able to adapt to positional information along different axes, thereby improving its ability to model relative positional information in the input sequence. By applying the SoftMax operation to each row of \(A_{(n,m)}^{\prime}\), normalization is performed to obtain the final attention distribution \(a_{n,m}\). \[X_{gaf}=H\textit{swish}\Big{(}BN\Big{(}Conv_{3\times 3}\Big{(}X_{gaf}\Big{)} \Big{)}\Big{)} \tag{1}\]3

The normalized weights are used to weight the corresponding value vectors \(v_{m}\). The 2D RoPE pseudocode is shown in Algorithm 3.