## 3 Experiments and Results
### Long Sequence Language Modeling  
To evaluate performance in long sequence language modeling, we utilized the Proof-pile [24] dataset, which comprises numerous long sequence texts, specifically employing its test split for our analysis. We adopted the sliding window technique [6] to assess perplexity across various context window sizes, setting the stride \(S\) to 256. Initially, we assessed how model variants, fine-tuned using different methods, performed as the context window size increased. Following the approach used by YaRN, we selected 10 random samples from the Proof-pile dataset, each containing at least 10 k tokens. For the scaling factor \(s=2\), we evaluated the perplexity for sequence lengths from 1 k to 5 k tokens, in 1 k token steps. Table 1 presents a comparison of perplexity for the Pythia model, expanded from an original context window of 2048 to 4096, utilizing the original model, PI, NTK, YaRN, and SBA-RoPE. From the experimental results, it is evident that, initially, within the original context lengths (1 k and 2 k), all models fine-tuned with various context window extension methods experienced an increase in perplexity to varying degrees. The PI method saw the most significant increase within this range, likely due to the equal stretching of all dimensions, resulting in the loss of high-frequency information. In contrast, the model variants fine-tuned with our method exhibited the smallest increase in perplexity, indicating minimal performance degradation. This minimal increase is attributed to our method's direct extrapolation of high-frequency dimensions, preventing the loss of this part of the pre-trained information. In the extended context window (3 k to 4 k), the original model's perplexity significantly increased, indicating that the original model became impractical for extended windows. Our method achieved lower perplexity in the extended window compared to all other methods. Furthermore, at the 5 k window, which exceeds the extended context window (4 k), our method still maintained the lowest value, signifying superior extrapolation performance. We further increased the scaling factor \(s\) to 4, applying fine-tuning to the original model through various methods. The outcomes are presented in Table 2. Despite the maximum length of the fine-tuning data being 4 k, adjusting the scaling factor \(s\) to 4 allows us to infer that the model's maximum context length has now been extended to 8 k. This extension is noteworthy because, although the model has never been exposed to context lengths between 4 k and 8 k, it still demonstrates a degree of transfer learning capability. Data presented in Table 2 show performance within the 1 k to 4 k context window that is similar to the behavior observed with a scaling factor \(s=2\). Notably, even within the 7 k to 8 k length, which fal ls within the extended context window of NTK, this method exhibits significantly higher perplexity compared to others. This higher perplexity can be attributed to the NTK method's adjusted rotation angles producing out-of-bounds values. \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Method** & **1024** & **2048** & **3072** & **4096** & **5120** \\ \hline Original & **5.617** & **4.785** & 25.844 & 79.499 & 146.776 \\ PI & 6.185 & 5.227 & 4.756 & 4.622 & 10.543 \\ NTK & 6.064 & 5.134 & 4.678 & 4.595 & 11.173 \\ YaRN & 6.061 & 5.134 & 4.678 & 4.552 & 11.016 \\ SBA-RoPE & 6.036 & 5.016 & **4.596** & **4.363** & **10.233** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Perplexity comparison across expanded context window sizes, contrasting the original Pythia-2.8b model with variants fine-tuned via different methods. The scaling factor for all variants is \(s=2\), doubling the original context window size from 2 k to 4 k tokens. Evaluations of perplexity range from 1 k to 5 k tokens, with increments of 1 k. The lowest and second-lowest perplexity (PPL) values at each length are highlighted in **bold** for the lowest and underlined for the second-lowest.