## 3 Experiments and Results
### Passkey Retrieval  
To investigate the effective context window size of models after extension--that is, the maximum distance of tokens that can be effectively attended to during the inference process--we adhered to the passkey retrieval task as defined by [25]. In this task, the model is required to recover a hidden five-digit passkey embedded within a context of largely nonsensical text. The specific format of the prompt used for this task is detailed in Figure 2.  
To evaluate the models, we conducted the passkey retrieval task 20 times for each model variant, positioning the key as close to the beginning of the context as possible to more accurately reflect the model's capability to attend to the longest distance. The accuracy of models fine-tuned with a scaling factor \(s=2\) on the passkey retrieval task is shown in Table 3. It is observed that all fine-tuned model variants exhibit high accuracy in the extended window. Notably, PI outperforms NTK and YaRN across all context window sizes on this task, suggesting that evenly stretching all embedding dimensions, despite causing \"local distortion\" do not significantly impact the dependency relationships between local tokens in the context of passkey retrieval. Our SBA-RoPE achieves the best results in all context windows except for the 3 k window, by combining the advantages of PI's non-exceeding bounds and NTK's distribution of interpolation stress across all dimensions.  
To assess the extended capabilities of the models, we further tested them under the passkey retrieval task with the scaling factor increased to \(s=4\), representing an expanded context window up to 8 k tokens. The outcomes, as documented in Table 4, reveal that the models fine-tuned with this larger scaling factor continue to perform with notable accuracy across extended context lengths. Particularly, the PI method and our SBA-RoPE exhibit remarkably consistent performance, even at the higher context ranges of 5 k to 8 k tokens, underscoring the effectiveness of our approach in managing extended contexts. SBA-RoPE, especially, demonstrates superior adaptability and accuracy, effectively leveraging its  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Method** & **1024** & **2048** & **3072** & **4096** & **5120** \\ \hline Original & **1.0** & **1.0** & 0.00 & 0.00 & 0.00 \\ PI & 0.95 & 0.85 & **0.65** & **0.85** & 0.00 \\ NTK & 0.90 & 0.75 & 0.40 & 0.60 & 0.00 \\ YaRN & 0.70 & 0.80 & 0.50 & 0.55 & 0.00 \\ SBA-RoPE & 0.95 & 0.90 & 0.60 & **0.85** & 0.00 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy of passkey retrieval across different context window sizes for the original Pythia-2.8b model and its variants fine-tuned with a scaling factor of \(s=2\). Accuracy is measured by the model’s ability to correctly retrieve the hidden passkey within various context lengths. The highest and second-highest accuracy values at each length are highlighted in **bold** for the highest and underlined for the second-highest.  
Figure 2: We adopt the identical prompt structure for passkey recovery as suggested by [25]. In this setup, the specific passkey 12345 is substituted with randomly generated five-digit numerals for the evaluation phase.  
hybrid strategy to maintain high retrieval accuracy, a testament to its robust extrapolation capabilities in previously unseen context lengths.