## 2 Backgrounds and Methods
### Our Proposal: SBA-RoPE  
By observing the out-of-bounds values in certain dimensions of the NTK Interpolation, it becomes evident that for low-frequency dimensions, the maximum rotation angles learned during the pre-training process do not exceed \(2\pi\). These dimensions carry absolute position information, as the rotation angle corresponding to each position is unique. Extrapolating these angles would lead to significantly increased model perplexity, since the model has not been trained on these extrapolated angles. On the other hand, high-frequency dimensions have been thoroughly trained within the \([0,2\pi]\) range, and the sine and cosine values of these angles reoccur in RoPE due to the periodicity of trigonometric functions. Thus, we can simply extrapolate these angles, as they carry relative position information.  
To delineate the dimensions requiring interpolation from those needing extrapolation, we define the following notation as Equation (7) for the rotation angle at position _pos_ and dimension _dim_:  
\[\theta_{pos,dim,base}=pos\cdot base^{\frac{-2\cdot dim}{d}} \tag{7}\]  
where \(pos\in[0,L-1]\) represents the position index corresponding to the current rotation angle, \(dim=0,1,\dots,d/2-1\) represents the dimension corresponding to the current rotation angle, and \(d\) represents the embedding dimension of the attention head. Based on our discussion, we need to identify the smallest dimension \(dim^{\prime}\) such that it is just less than \(2\pi\); this dimension marks the boundary as Equation (8) between extrapolation and interpolation:  
\[dim^{\prime}=\min\{dim\in[0,1,\dots,d/2-1],\theta_{L-1,dim,base}<2\pi\} \tag{8}\]  
where _base_ and \(L\) are, respectively, the _base_ and the maximum context window used in RoPE during LLM pre-training. For \(dim>dim^{\prime}\), we employ an interpolation method similar to NTK, ensuring the scaling at dimension \(dim^{\prime}\) matches that of the Positional Interpolation (PI) method. The _base_\({}^{\prime}\) for these low-frequency dimensions \(dim\) can be calculated under the new fine-tuning length \(L^{\prime}\) as Equations (9)-(11):  
\[\theta_{L^{\prime}-1,dim^{\prime},base}= \theta_{L-1,dim^{\prime},base} \tag{9}\] \[(L^{\prime}-1)\cdot base^{\prime-\frac{2-dim^{\prime}}{d}}= (L-1)\cdot base^{\frac{-2\cdot dim^{\prime}}{d}}\] (10) \[base^{\prime}= base\cdot(\frac{L^{\prime}-1}{L-1})^{\frac{d}{2\cdot dim^{ \prime}}} \tag{11}\]  
This calculation determines the _base_ used for interpolating low-frequency dimensions. Consequently, we can express the rotation angles in Segmental Base Adjustment for RoPE (SBA-RoPE) as Equation (12):  
\[\theta=\begin{cases}\theta_{pos,dim,base},&\text{if }dim<dim^{\prime}\\ \theta_{pos,dim,base},&\text{if }dim\geq dim^{\prime}.\end{cases} \tag{12}\]  
Through our derivation, we have demonstrated that this approach yields rotation angles that, under the new fine-tuning length \(L^{\prime}\), do not produce out-of-bounds valuesfor high-frequency dimensions \(dim<dim^{\prime}\) nor low-frequency dimensions \(dim>dim^{\prime}\), maintaining consistency with the PI method. For high-frequency dimensions, we directly perform extrapolation to avoid the loss of high-frequency information, as seen with PI. For low-frequency dimensions, we adopt a method similar to NTK, distributing the interpolation load across all low-frequency dimensions.