PIC: Unlocking Long-Form Text Generation Capabilities of Large Language Models via Position ID Compression
Authors: Haoran Que, Wenge Rong
Accepted at: acl25

PIC: Unlocking Long-Form Text Generation Capabilities of Large Language Models via Position ID Compression

Haoran Que1, Wenge Rong2
1Sino-French Engineer School, Beihang University, China
2School of Computer Science and Engineering, Beihang University, China
2224124@buaa.edu.cn, w.rong@buaa.edu.cn

Abstract

Long-context understanding is crucial for large language models (LLMs) and has become a fundamental capability for most LLMs. However, beyond the focus on “input-long”, the ability to “output-long” is equally significant, yet it remains underexplored. To address this limitation, we propose a simple, efficient, and plug-in approach, Position ID Compression (PIC), to unlock the long-form text generation potential of LLMs. The idea is straightforward: by compressing the position ids of the context, we provoke and guide LLMs to generate coherent and longer output. Specifically, we find that directly reducing the position ids by a fixed ratio significantly impacts the generation quality. To mitigate this, we propose two variants of PIC: NTK-aware PIC and Dynamic PIC. Without additional training, both methods enable LLMs to extend their generation length by approximately 1.5 times without compromising generation quality. Furthermore, by integrating supervised fine-tuning (SFT) with PIC, we propose PIC-SFT, which further improves LLMs’ long-form text generation capabilities, achieving top performance on HelloBench and LongBench-Write. Extensive experiments demonstrate the effectiveness of our approach.

1 Introduction

Modeling long context is essential for Large Language Models (LLMs) as it meets the user’s need for long-range interactions (Ding et al., 2024; Lin et al., 2024) while enhancing the capabilities of LLM-based systems (e.g., Retrieval-Augmented Generation(Gao et al., 2023), Multi-Agent System(Li et al., 2024), etc.). Recently, a variety of methods for context window extension(Chen et al., 2023; Peng et al., 2023) and efficient inference (e.g., sparse attention(Beltagy et al., 2020; Xiao et al., 2023), KV cache compression(Dao, 2023; Kwon et al., 2023)) have been proposed. As a result, long-context modeling has become a fundamental capability of current LLMs(Dubey et al., 2024; Yang et al., 2024), with some models even capable of handling inputs with millions of tokens(Team et al., 2023; Zeng et al., 2022). Beyond long-context understanding, long-context modeling is also related to long-form text generation. With an extended context window, a model should theoretically be capable of both “input-long” and “output-long”. However, this is often not the case in practice: while models can process millions of tokens as input, they struggle to generate even 4000 tokens(Que et al., 2024). This motivates us: Can we efficiently unlock the long-form text generation capabilities of LLMs based on existing long-context modeling paradigms?

In this work, we propose Position ID Compression (PIC), a simple, efficient, and plug-in method to unlock the long-form text generation potential of LLMs. As illustrated in Figure 1, the core idea of PIC is to directly or indirectly compress the position ids, so that the relative positions perceived by the model are smaller than the actual relative positions, thereby extending the model’s generation length. To verify this idea, we first attempt to reduce the position ids by a fixed ratio, but we find that this method performs poorly. To address this, we propose two variants of PIC: NTK-aware PIC and Dynamic PIC. NTK-aware PIC indirectly reduces position ids by scaling down the rotation angle of Rotary Position Embedding (RoPE), while Dynamic PIC dynamically compresses position ids in the middle part of the context. Experimental results show that NTK-aware PIC and Dynamic PIC are effective and efficient. They can extend the model’s generation length by approximately 1.5 times without additional training and demonstrate generalization capabilities.

Furthermore, we combine supervised fine-tuning (SFT) with PIC and propose PIC-SFT, significantly extending the generation length to 5 times the original without losing the generation quality.
Figure 1: Illustration of Position ID Compression (PIC). (Top): The diagrams sequentially show the mechanisms of Naive PIC, NTK-aware PIC, and Dynamic PIC. The red areas represent the compressed areas, while the blue areas represent the uncompressed areas. The number on the block indicates the position id. (Bottom): The corresponding experimental results for the three PIC methods, where the blue lines represent the generated length and the red lines represent the generated quality.

Experimental results show that compared to traditional SFT, PIC-SFT achieves better performance on HelloBench (Que et al., 2024) and LongBench-Write (Bai et al., 2024). Moreover, through Naive Inference, PIC-SFT better preserves the short-form text generation capabilities of LLMs.

Our main contributions are as follows:

1. PIC: We introduce PIC, a method to extend the long-form text generation capabilities of LLMs by compressing position ids, allowing for longer generation length without additional training.

2. NTK-aware PIC and Dynamic PIC: We propose two variants, NTK-aware PIC and Dynamic PIC, to improve the efficiency and effectiveness of PIC.

3. PIC-SFT: We combine PIC with SFT, resulting in PIC-SFT, which extends the generation length up to 5 times the original while preserving short generation capabilities.

2 Background and Related Work

2.1 RoPE

RoPE was first introduced in RoFormer (Su et al., 2024) and has been widely used in current LLMs (Touvron et al., 2023; Bai et al., 2023a). For a sequence of vectors \((x_1, x_2, \ldots, x_i, \ldots, x_n)\) where \(x_i \in \mathbb{R}^d\) represents the word embedding of the \(i\)-th token in the sequence. RoPE aims to incorporate relative positional information into the computation of attention scores. Specifically, it ensures that the inner product of the query \(q_m\) and key \(k_n\) encodes position information in the relative form:

\[
\langle f_q(x_m, m), f_k(x_n, n) \rangle = g(x_m, x_n, m-n).
\]

\(m\) and \(n\) represent the position of the query and the key. To solve the functions \(f_q\), \(f_k\), and \(g\), RoPE applies a rotary matrix to \(q_m\) and \(k_n\):

\[
\begin{pmatrix}
\cos m \theta_1 & -\sin m \theta_1 & 0 & 0 & \cdots & 0 \\
\sin m \theta_1 & \cos m \theta_1 & 0 & 0 & \cdots & 0 \\
0 & 0 & \cos m \theta_2 & -\sin m \theta_2 & \cdots & 0 \\
0 & 0 & \sin m \theta_2 & \cos m \theta_2 & \cdots & 0 \\
\vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & 0 & 0 & 0 & \cdots & \cos m \theta_{d/2} & -\sin m \theta_{d/2} \\
0 & 0 & 0 & 0 & \cdots & \sin m \theta_{d/2} & \cos m \theta_{d/2}
\end{pmatrix},
\]

where \(\theta_j = b^{-2(j-1)/d}, j \in \{1, 2, ..., d/2\}\) and \(b\) is a fixed value called RoPE Base.

2.2 Position Interpolation

RoPE encodes a word embedding \(x_m\) by:
\[
\{q_m, k_m\} = f_{\{q,k\}}(x_m, g(m), h(\theta_j)) \tag{2}
\]
\[
g(m) = m, \quad h(\theta_j) = b^{-2(j-1)/d}. \tag{3}
\]

Based on RoPE, to further extend the context window of LLMs, Position Interpolation (PI) (Chen et al., 2023) alters \(g(m)\):

\[
g(m) = \frac{m}{s}, \quad \text{where} \quad s = \frac{L_t}{L_c}. \tag{4}
\]

where \(s\) is the scaling factor, \(L_t\) is the size of the extended context window and \(L_c\) is the current context window size. Compared to Position Extrapolation, PI is more stable and requires only a small amount of training data to extend the context window.

2.3 NTK-aware Interpolation

Besides scaling all dimensions equally, there is a class of interpolation methods based on Neural Tangent Kernel (NTK) theory (Jacot et al., 2018). We focus on two widely used methods: NTK-aware (bloc97, 2023b) and NTK-by-parts (bloc97, 2023a). NTK-aware modifies \(h(\theta)\) of the RoPE as follows:

\[
h(\theta_j) = \left(b \cdot s^{\frac{d}{d-2}}\right)^{-2(j-1)/d}. \tag{5}
\]

The core idea of NTK-by-parts is to interpolate for lower frequency dimensions while keeping higher frequency dimensions unchanged:

\[
h(\theta_j) = (1 - \gamma(r)) \frac{\theta_j}{s} + \gamma(r) \theta_j, \tag{6}
\]

where \(\gamma\) is a step function that depends on \(r\) and \(r\) represents the frequency of the dimension.

2.4 Related Work

Long-Context Understanding Long-context understanding focuses on enabling LLMs to handle, process, and retrieve information from very long inputs (Hsieh et al., 2024; Bai et al., 2023b). To achieve input lengths of even 100M tokens (Team et al., 2024), various methodologies have been proposed (Shen et al., 2021; Ainslie et al., 2023). Among them, speed and length are the key factors in long-context modeling. Context window extension (Cobbe et al., 2021; Zhu et al., 2023; Jin et al., 2024) focuses on expanding the context window of LLMs to support a larger number of input tokens. KV cache compression (Liu et al., 2024b; Ge et al., 2023) aims to compress the KV cache, enabling faster inference and eliminating redundant information.

Long-Form Text Generation Long-form text generation is a crucial capability for LLMs, closely tied to their practical applications in real-world scenarios (Guan et al., 2022; Hosseini et al., 2024; Wei et al., 2024). To explore the long-form text generation capabilities of LLMs, many benchmarks and methods have been proposed (Ye et al., 2025; Tan et al., 2024; Liu et al., 2024a). Bai et al. propose LongBench-Write (Bai et al., 2024), a comprehensive benchmark for evaluating ultra-long generation capabilities. Que et al. propose HelloBench (Que et al., 2024), a comprehensive, in-the-wild, and open-ended benchmark to evaluate LLMs’ performance in generating long text. Self-Lengthen (Quan et al., 2024) leverages the intrinsic knowledge and skills of LLMs to enable the model to generate longer content. Suri (Pham et al., 2024) and LongWriter construct high-quality datasets to train the model. These methods require extensive training. PIC is the first approach to enhance long-form text generation capabilities from the perspective of position id and can be plugged into many LLMs without training.

3 Method

3.1 PIC

The core idea of PIC is straightforward: RoPE encodes relative positional information for tokens at different positions, if we compress the position ids of tokens so that the relative positions perceived by the models are smaller than the actual relative positions, the models may be able to generate longer content. In Section 4.2, experimental results demonstrate that this simple starting point achieves highly effective results. Based on this idea, we propose three variants of PIC: Naive PIC, NTK-aware PIC, and Dynamic PIC.

Naive PIC As shown in Figure 1(a), Naive PIC is defined as compressing the position ids of all positions equally by a fixed ratio. This is equivalent to modifying \(g(m)\) in Equation (2) to:

\[
g(m) = \frac{m}{s_{cr}}, \tag{7}
\]

where \(m\) represents the actual position id of the current token and \(s_{cr}\) represents the **compression ratio**. The form of Naive PIC is identical to that of PI (Chen et al., 2023). However, the focus of PI is more on extending the context window, whereas the focus of Naive PIC is on enabling the model,
In the movies, time travelers typically step inside a machine and—poof—disappear. They then reappear (output 1) instantaneously among cowboys, knights or dinosaurs. (output 2) What these films show is basically time teleportation. (output 3)

![Illustration of Dynamic PIC-SFT. A complete text is split into segments, where the input for each subsequent segment is formed by concatenating the input and output of the previous segment.](page_184_120_1079_312.png)

Complete Training Data

Splitted Training Data

Figure 2: Illustration of Dynamic PIC-SFT. A complete text is split into segments, where the input for each subsequent segment is formed by concatenating the input and output of the previous segment.

which already has a sufficient context window, to generate more content.

NTK-aware PIC NTK-aware PIC indirectly compresses the position ids by modifying the RoPE Base \( b \), thus altering \( h(\theta_j) \) in Equation (2):

\[
h(\theta_j) = (b \cdot s_{rr})^{-2(j-1)/d},
\]

where \( b \) is the original RoPE base, \( s_{rr} \) is the RoPE base scaling ratio, \( j \) denotes the positional index of the token vector, and \( d \) represents the total dimension size of the token vector. As shown in Figure 1(b), the position ids are not directly compressed. Instead, the RoPE base is doubled, which affects the values in different dimensions of the token vector.

Dynamic PIC As shown in Figure 1(c), an input context \( C \) consists of three parts: \( C = \{ \text{Initial Context}, \text{Middle Context}, \text{Recent Context} \} \). The core idea of Dynamic PIC is to compress the position ids of the middle context only while keeping the position ids of the recent context and the initial context unchanged. Assuming the current total context length is \( l_c \), we have:

\[
g(m) = \begin{cases}
m & \text{if } m \leq l_{ic} \text{ or } m \geq l_c - l_{rc}, \\
\frac{m}{s_{cr}} & \text{otherwise.}
\end{cases}
\]

Here, \( s_{cr} \) represents the compression ratio, \( l_{ic} \) represents the length of the initial context, and \( l_{rc} \) represents the length of the recent context.

In the practical implementation of Dynamic PIC, we need to modify the forward function of the model. Originally, the KV cache stores the K and V after applying RoPE. In Dynamic PIC, we only compress the position ids of the middle context, and for each next token prediction, we need to update the position ids of all preceding tokens. Therefore, in the Dynamic PIC implementation, we need to store the K and V before applying RoPE. As a result, the steps for applying RoPE and updating the KV cache need to be swapped, as shown in Algorithm 1.

Algorithm 1 Dynamic PIC

Require: Query \( Q \), Key \( K \), Value \( V \), KV Cache \( C \)

Original Implementation
\( Q, K \leftarrow \text{apply\_rope}(Q, K) \)
if \( C \) is not null then
    \( K, V \leftarrow C.\text{update}(K, V) \)
end if

Dynamic PIC Implementation
if \( C \) is not null then
    \( K, V \leftarrow C.\text{update}(K, V) \)
end if
\( Q, K \leftarrow \text{apply\_rope}(Q, K) \)

3.2 PIC-SFT

To further enhance the long-form text generation capabilities of LLMs, we integrate PIC with SFT and propose PIC-SFT. Naive PIC-SFT is defined as compressing position ids to \( 1/s_{cr} \) of their original value during SFT, while NTK-aware PIC-SFT
is defined as expanding the RoPE base to \( s_{tr} \) times its original value during SFT. In the cases of Naive PIC-SFT and NTK-aware PIC-SFT, all tokens are processed in the same way, allowing direct combination with SFT. However, for Dynamic PIC, the position ids of the context need to be dynamically updated for each new token prediction. If directly combined with SFT, this would significantly increase storage and computation overhead during training, making parallel computation infeasible. To address these challenges, we propose **Dynamic PIC-SFT**, an SFT approach tailored to Dynamic PIC.

As shown in Figure 2, we split a complete piece of text into segments of equal length, starting from the end of the text and moving backward. These segments serve as the output, while all preceding content is treated as the input. Since we only compute the loss for the output, this approach effectively computes the prediction loss for the entire text. This approach ensures that the position ids for each segment remain fixed, allowing each segment to be treated as an independent sample for parallel computation, significantly reducing computational overhead during training. Furthermore, since the splits are fine-grained, the fixed position ids for each segment closely align with the behavior of Dynamic PIC during inference, ensuring that the training stage closely approximates the actual inference stage.

4 Experiments

4.1 Experimental Setup

**Models & Benchmarks** In this work, we mainly evaluate 3 LLMs: LLaMA-3.1-8B-Instruct (Dube et al., 2024), Qwen-2.5-7B-Instruct (Yang et al., 2024), and GLM-4-9B-Chat (Zeng et al., 2022). To evaluate the performance of PIC in long-form text generation, we select HelloBench (Que et al., 2024) and LongBench-Write (Bai et al., 2024) as the evaluation benchmarks. As the training dataset, we have chosen the high-quality bilingual SFT dataset LongWriter-6k\(^1\) which contains 6000 long-form instructional samples. We only selected samples from LongBench-Write with generation length requirements greater than 2000 as the test set. To ensure consistency with LongBench-Write, we focus on the “Heuristic Text Generation” subset of HelloBench for evaluation. Therefore, for PIC-SFT, HelloBench can be considered as out-of-distribution data, while LongBench-Write can be seen as in-distribution data, enabling a more comprehensive evaluation. To evaluate the model’s performance on short-form text generation, we select MMLU (Hendrycks et al., 2020) and GSM8K (Cobbe et al., 2021). The number of test samples for HelloBench, LongBench-Write, MMLU, and GSM8K are 123, 47, 14042, and 1319, respectively.

**Inference & Training** To ensure reproducibility, all inference is conducted using greedy decoding. For training, we use LLaMA-Factory (Zheng et al., 2024) with a learning rate of 5e-5, warmup of 0.1, and 3 epochs. All experiments are conducted on 64 NVIDIA H100 80GB GPUs.

**PIC** In Section 4.3, for Dynamic PIC-SFT, the length of the initial context is 10, and the length of the recent context is 600. For NTK-aware PIC-SFT, \( s_{tr} \) is set to 16.

**Metrics** We focus on two aspects: generation length and generation quality. We adopt the metrics for length and quality as defined in HelloBench and LongBench-Write. Specifically, \( S_l \) represents the length score, \( S_q \) represents the quality score, and \( \overline{S} \) is their average. Additionally, we have observed that generating overly long texts often leads to severe repetition. We consider such repetitive generations as failed outputs, and thus, we calculate the **Corrupted Ratio** to evaluate the effectiveness of the generation. The corrupted ratio is defined as the number of corrupted samples divided by the total number of samples:

\[
\text{CR} = \frac{n_{corrupted}}{n_{total}},
\]

where \( n_{corrupted} \) and \( n_{total} \) represent the number of corrupted samples and total samples, respectively. For MMLU and GSM8K, the evaluation metric is accuracy. To save resources and ensure reproducibility of results, we use LLaMA-3.3-70B-Instruct\(^2\) as the judge model, and the inference engine is vLLM (Kwon et al., 2023). More details are shown in Appendix A.

4.2 PIC

To verify the effectiveness and generalization of Naive PIC, NTK-aware PIC, and Dynamic PIC, we

\footnotetext{1https://huggingface.co/datasets/THUDM/LongWriter-6k}
\footnotetext{2https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct}
Table 1: The results of the three PIC methods, Naive PIC, NTK-aware PIC, and Dynamic PIC, with varying parameters are presented. Evaluation Benchmark is HelloBench. “CR” denotes Corrupted Ratio. The results for Quality and CR have been multiplied by 100. Only a subset of the results is shown here, more detailed experimental results can be found in Appendix B. The blue row represents the baseline.

<table>
  <tr>
    <th rowspan="2">Parameter</th>
    <th colspan="4">LLaMA-3.1-8B-Instruct</th>
    <th colspan="4">Owen-2.5-7B-Instruct</th>
    <th colspan="4">GLM-4-9B-Chat</th>
  </tr>
  <tr>
    <th>Length ↑</th>
    <th>Tokens ↑</th>
    <th>Quality ↑</th>
    <th>CR ↓</th>
    <th>Length ↑</th>
    <th>Tokens ↑</th>
    <th>Quality ↑</th>
    <th>CR ↓</th>
    <th>Length ↑</th>
    <th>Tokens ↑</th>
    <th>Quality ↑</th>
    <th>CR ↓</th>
  </tr>
  <tr>
    <td colspan="14"><b>Naive PIC, Parameter = Compression Ratio</b></td>
  </tr>
  <tr>
    <td>0.5</td><td>779.26</td><td>806.91</td><td>73.91</td><td>43.09</td><td>824.68</td><td>850.16</td><td>69.00</td><td>59.35</td><td>802.42</td><td>850.29</td><td>77.82</td><td>6.50</td>
  </tr>
  <tr>
    <td>0.75</td><td>959.35</td><td>998.88</td><td>78.52</td><td>4.07</td><td>832.16</td><td>860.22</td><td>77.65</td><td>11.38</td><td>946.47</td><td>994.15</td><td>79.64</td><td>0.81</td>
  </tr>
  <tr>
    <td>1</td><td><span style="color:blue;">1001.86</span></td><td><span style="color:blue;">1055.42</span></td><td><span style="color:blue;">79.85</span></td><td><span style="color:blue;">1.63</span></td><td><span style="color:blue;">958.33</span></td><td><span style="color:blue;">990.54</span></td><td><span style="color:blue;">80.17</span></td><td><span style="color:blue;">4.07</span></td><td><span style="color:blue;">990.09</span></td><td><span style="color:blue;">1039.23</span></td><td><span style="color:blue;">81.07</span></td><td><span style="color:blue;">2.44</span></td>
  </tr>
  <tr>
    <td>2</td><td>691.76</td><td>725.72</td><td>71.96</td><td>13.01</td><td>858.95</td><td>905.18</td><td>71.51</td><td>54.47</td><td><span style="color:blue;">1120.43</span></td><td><span style="color:blue;">1109.97</span></td><td>71.17</td><td>46.34</td>
  </tr>
  <tr>
    <td colspan="14"><b>NTK-aware PIC, Parameter = RoPE Base Scaling Ratio</b></td>
  </tr>
  <tr>
    <td>0.33</td><td>770.85</td><td>806.58</td><td>77.54</td><td>0</td><td>797.55</td><td>823.27</td><td>78.95</td><td>4.88</td><td>869.17</td><td>914.07</td><td>79.18</td><td>0</td>
  </tr>
  <tr>
    <td>0.67</td><td>882.79</td><td>925.79</td><td>79.49</td><td>0</td><td>962.98</td><td>997.83</td><td>78.67</td><td>5.69</td><td>944.34</td><td>989.40</td><td>80.02</td><td>0.81</td>
  </tr>
  <tr>
    <td>1</td><td><span style="color:blue;">1020.80</span></td><td><span style="color:blue;">1071.84</span></td><td><span style="color:blue;">80.97</span></td><td><span style="color:blue;">1.63</span></td><td><span style="color:blue;">954.75</span></td><td><span style="color:blue;">988.72</span></td><td><span style="color:blue;">80.98</span></td><td><span style="color:blue;">4.07</span></td><td><span style="color:blue;">986.97</span></td><td><span style="color:blue;">1031.88</span></td><td><span style="color:blue;">81.03</span></td><td><span style="color:blue;">2.44</span></td>
  </tr>
  <tr>
    <td>4</td><td>1522.97</td><td>1604.71</td><td><span style="color:blue;">82.38</span></td><td>7.32</td><td>1140.97</td><td>1178.38</td><td>80.98</td><td>6.50</td><td>1154.25</td><td>1203.4</td><td>82.16</td><td>1.63</td>
  </tr>
  <tr>
    <td>16</td><td><span style="color:blue;">1802.11</span></td><td><span style="color:blue;">1902.17</span></td><td>81.05</td><td>34.15</td><td><span style="color:blue;">1391.56</span></td><td><span style="color:blue;">1436.65</span></td><td><span style="color:blue;">81.77</span></td><td>13.01</td><td><span style="color:blue;">1352.04</span></td><td><span style="color:blue;">1404.38</span></td><td><span style="color:blue;">82.52</span></td><td><span style="color:blue;">1.63</span></td>
  </tr>
  <tr>
    <td colspan="14"><b>Dynamic PIC, Parameter = Compression Ratio, Length of Initial Context = 4, Length of Recent Context = 200</b></td>
  </tr>
  <tr>
    <td>0.25</td><td>816.65</td><td>857.19</td><td>79.30</td><td>0</td><td>823.67</td><td>854.88</td><td>78.88</td><td>4.88</td><td>930.33</td><td>973.37</td><td>79.92</td><td>3.25</td>
  </tr>
  <tr>
    <td>0.5</td><td>893.25</td><td>940.46</td><td>79.15</td><td>0</td><td>864.46</td><td>895.39</td><td>79.25</td><td>5.69</td><td>937.21</td><td>980.22</td><td>79.40</td><td>1.63</td>
  </tr>
  <tr>
    <td>1</td><td><span style="color:blue;">1024.05</span></td><td><span style="color:blue;">1076.16</span></td><td><span style="color:blue;">81.07</span></td><td><span style="color:blue;">1.63</span></td><td><span style="color:blue;">966.61</span></td><td><span style="color:blue;">1001.67</span></td><td><span style="color:blue;">80.19</span></td><td><span style="color:blue;">4.07</span></td><td><span style="color:blue;">984.50</span></td><td><span style="color:blue;">1029.95</span></td><td><span style="color:blue;">80.86</span></td><td><span style="color:blue;">2.44</span></td>
  </tr>
  <tr>
    <td>4</td><td>1380.63</td><td>1444.02</td><td><span style="color:blue;">81.31</span></td><td>0.81</td><td>1125.13</td><td>1164.26</td><td>81.58</td><td>17.07</td><td>1154.62</td><td>1206.48</td><td>81.75</td><td><span style="color:blue;">0.81</span></td>
  </tr>
  <tr>
    <td>16</td><td>1938.24</td><td>2028.61</td><td>81.12</td><td>24.39</td><td><span style="color:blue;">1520.81</span></td><td><span style="color:blue;">1569.01</span></td><td>80.93</td><td>22.76</td><td>1347.92</td><td>1406.73</td><td>82.72</td><td>4.88</td>
  </tr>
  <tr>
    <td>128</td><td><span style="color:blue;">1961.89</span></td><td><span style="color:blue;">2101.06</span></td><td>75.31</td><td>61.79</td><td>1490.53</td><td>1543.54</td><td><span style="color:blue;">81.96</span></td><td>35.77</td><td><span style="color:blue;">1476.12</span></td><td><span style="color:blue;">1538.12</span></td><td><span style="color:blue;">83.43</span></td><td><span style="color:blue;">4.07</span></td>
  </tr>
</table>

have tested the long-form text generation capabilities of the three models under different parameters. The experimental results are shown in Figure 1 and Table 1. Based on the experimental results, we draw the following conclusions:

(1). Naive PIC performs poorly: As observed in Table 5, when the compression ratio is not equal to 1, the generation quality tends to degrade significantly, with the quality score dropping to 0 and the corrupted ratio close to 100%. This result aligns with the conclusions from PI (Chen et al., 2023), where directly applying linear interpolation to position ids leads to undesirable outcomes.

(2). NTK-aware PIC and Dynamic PIC perform well: From the trend in Figure 1, it can be observed that as the Compression Ratio or RoPE Base Scaling Ratio increases, the generation length increases smoothly. At the same time, the generation quality does not deteriorate and even shows a slight improvement, which is consistent with our core idea: compressing position ids can stimulate the model’s inherent ability for long-form text generation. This validates the effectiveness and generalization of PIC across three models.

(3). Corrupted Ratio: Based on Table 6 and Table 7, in most cases, the corrupted ratio remains below 20% (with \( s_{cr} \) ranging from 0.25 to 64 and \( s_{cr} \) ranging from 0.25 to 64). This indicates that within a certain range, NTK-aware PIC and Dynamic PIC do not negatively impact generation quality. This result confirms the superiority of NTK-aware PIC and Dynamic PIC compared to Naive PIC. However, there is a threshold for these ranges. When the threshold is exceeded (e.g., when \( s_{cr} \) exceeds 8 on LLaMA-3.1-8B-Instruct), the generation quality deteriorates significantly.

(4). Generation Length: The trend of generation length is entirely consistent with the generation token. Additionally, it can be observed that NTK-aware PIC and Dynamic PIC are able to extend the model’s generated length to approximately 1.5 times the original length without compromising generation quality, and this is achieved without any additional training.

4.3 PIC-SFT

To further improve the long-form text generation capabilities of LLMs, we propose PIC-SFT and conduct experiments. The experimental results are shown in Table 2 and Figure 3. Based on the results, we summarize the following conclusions:

(1). SFT improves performance significantly: Fine-tuning with high-quality instruction datasets can significantly enhance the long-form text generation capabilities of the model. Regardless of the specific SFT method, fine-tuning improves both the generation length and generation quality. On HelloBench, the generation length increases by 4–5 times, and the generation quality improves from 82.25 to 89.26 (on LLaMA-3.1-8B-Instruct).
Table 2: The experimental results of different SFT methods. “CR” denotes Corrupted Ratio. All experimental results maintain consistency between training and inference, for example, the inference method for NTK-aware PIC-SFT is NTK-aware PIC.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">HelloBench-HTG</th>
    <th colspan="5">LongBench-Write</th>
  </tr>
  <tr>
    <th>Length ↑</th>
    <th>Quality ↑</th>
    <th>CR ↓</th>
    <th>S<sub>t</sub> ↑</th>
    <th>S<sub>q</sub> ↑</th>
    <th>S ↑</th>
    <th>CR ↓</th>
  </tr>
  <tr>
    <th colspan="8" style="background-color:#f2f2f2;">LLaMA-3.1-8B-Instruct</th>
  </tr>
  <tr>
    <td>w/o SFT</td>
    <td>1040.89</td>
    <td>82.25</td>
    <td><b>1.63</b></td>
    <td>18.87</td>
    <td>90.33</td>
    <td>54.60</td>
    <td>3.33</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>4342.24</td>
    <td>86.87</td>
    <td>3.25</td>
    <td>78.06</td>
    <td><b>98.73</b></td>
    <td>88.40</td>
    <td><b>1.67</b></td>
  </tr>
  <tr>
    <td>NTK-aware PIC-SFT</td>
    <td>4737.44</td>
    <td>89.17</td>
    <td>4.07</td>
    <td>77.71</td>
    <td>97.80</td>
    <td>87.75</td>
    <td><b>1.67</b></td>
  </tr>
  <tr>
    <td>Dynamic PIC-SFT</td>
    <td><b>5032.98</b></td>
    <td><b>89.26</b></td>
    <td>4.07</td>
    <td><b>82.08</b></td>
    <td>97.00</td>
    <td><b>89.54</b></td>
    <td><b>1.67</b></td>
  </tr>
  <tr>
    <th colspan="8" style="background-color:#f2f2f2;">Qwen-2.5-7B-Instruct</th>
  </tr>
  <tr>
    <td>w/o SFT</td>
    <td>965.51</td>
    <td>80.58</td>
    <td>4.07</td>
    <td>41.59</td>
    <td>98.12</td>
    <td>69.85</td>
    <td>8.33</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>4070.64</td>
    <td>87.85</td>
    <td>3.25</td>
    <td>81.92</td>
    <td><b>99.64</b></td>
    <td>90.78</td>
    <td>3.33</td>
  </tr>
  <tr>
    <td>NTK-aware PIC-SFT</td>
    <td>3964.59</td>
    <td>87.46</td>
    <td>8.13</td>
    <td>81.13</td>
    <td>98.48</td>
    <td>89.81</td>
    <td>1.67</td>
  </tr>
  <tr>
    <td>Dynamic PIC-SFT</td>
    <td><b>4282.14</b></td>
    <td><b>89.39</b></td>
    <td><b>0.81</b></td>
    <td><b>84.15</b></td>
    <td>99.57</td>
    <td><b>91.86</b></td>
    <td><b>0.83</b></td>
  </tr>
  <tr>
    <th colspan="8" style="background-color:#f2f2f2;">GLM-4-9B-Chat</th>
  </tr>
  <tr>
    <td>w/o SFT</td>
    <td>967.32</td>
    <td>81.09</td>
    <td><b>2.44</b></td>
    <td>27.03</td>
    <td>95.65</td>
    <td>61.34</td>
    <td>3.33</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>4057.83</td>
    <td><b>84.05</b></td>
    <td>14.63</td>
    <td><b>77.86</b></td>
    <td>96.94</td>
    <td>87.40</td>
    <td>8.33</td>
  </tr>
  <tr>
    <td>NTK-aware PIC-SFT</td>
    <td>3910.37</td>
    <td>81.82</td>
    <td>30.08</td>
    <td>77.52</td>
    <td>94.17</td>
    <td>85.84</td>
    <td>20.0</td>
  </tr>
  <tr>
    <td>Dynamic PIC-SFT</td>
    <td><b>4357.59</b></td>
    <td>82.57</td>
    <td>3.25</td>
    <td>76.48</td>
    <td><b>98.7</b></td>
    <td><b>87.59</b></td>
    <td><b>0</b></td>
  </tr>
</table>

![Bar charts comparing scores for different SFT methods on MMLU and GSM8K](page_1012_1042_410_312.png)

Figure 3: Results of different SFT methods evaluated on MMLU and GSM8K. The base model is LLaMA-3.1-8B-Instruct. * represents standard inference, † represents NTK-aware PIC inference, and ‡ represents Dynamic PIC inference.

(2). **Dynamic PIC-SFT performs best**: As shown in Table 2, Dynamic PIC-SFT achieves the best results in both generation length and generation quality, whether for in-distribution benchmark LongBench-Write or out-of-distribution benchmark HelloBench. Furthermore, Dynamic PIC-SFT does not compromise the validity of the generated content. For example, for GLM-4-9B-Chat, both SFT and NTK-aware PIC-SFT result in reduced validity (the corrupted ratio on HelloBench is 14.63 and 30.08, and on LongBench-Write is 8.33 and 20.0, respectively). In contrast, the corrupted ratio for Dynamic PIC-SFT is only 3.25 on HelloBench and 0 on LongBench-Write.

(3). **The decoupling of PIC-SFT**: Figure 3 shows the performance of different methods on MMLU and GSM8K. Since the fine-tuning focuses solely on long-form text generation tasks, the overall output distribution tends to favor longer content. As a result, the model’s ability for shorter text generation may decline, leading to lower scores on tasks such as MMLU and GSM8K. However, an interesting observation arises when the PIC-SFT model is used for standard (i.e., non-PIC) infer-
Table 3: Ablation Study on Dynamic PIC. The base model is LLaMA-3.1-8B-Instruct. “IC” denotes initial context, “RC” denotes recent context. The blue row represents the baseline.

<table>
  <tr>
    <th rowspan="2">Parameter</th>
    <th colspan="4">HelloBench-HTG</th>
    <th colspan="4">LongBench-Write</th>
  </tr>
  <tr>
    <th>Length ↑</th>
    <th>Quality ↑</th>
    <th>CR ↓</th>
    <th>S<sub>I</sub> ↑</th>
    <th>S<sub>q</sub> ↑</th>
    <th>Ś ↑</th>
    <th>CR ↓</th>
  </tr>
  <tr>
    <th colspan="8">Ablation study on the modules of Dynamic PIC</th>
  </tr>
  <tr>
    <td>Baseline</td>
    <td>1401.97</td>
    <td>81.13</td>
    <td>0.81</td>
    <td>27.73</td>
    <td>4.36</td>
    <td>16.05</td>
    <td>5.83</td>
  </tr>
  <tr>
    <td>w/o IC</td>
    <td>1400.53</td>
    <td>81.45</td>
    <td>1.63</td>
    <td>29.32</td>
    <td>4.44</td>
    <td>16.88</td>
    <td>6.67</td>
  </tr>
  <tr>
    <td>w/o RC</td>
    <td>306.11</td>
    <td>15.3</td>
    <td>78.05</td>
    <td>2.14</td>
    <td>1.39</td>
    <td>1.76</td>
    <td>48.33</td>
  </tr>
  <tr>
    <td>w/o IC&RC</td>
    <td>266.61</td>
    <td>10.72</td>
    <td>77.24</td>
    <td>0</td>
    <td>1.43</td>
    <td>0.72</td>
    <td>45.0</td>
  </tr>
</table>

![Ablation Study on length of IC and length of RC.](page_101_670_1297_312.png)

Figure 4: Ablation Study on length of IC and length of RC.

ence. In this case, it achieves higher scores on MMLU and GSM8K. In addition, we have conducted additional experiments to demonstrate this, as detailed in Appendix C. These Phenomena indicate that PIC-SFT exhibits decoupling. Specifically, in NTK-aware PIC, the RoPE Base value is modified, while in Dynamic PIC, the position ids of the middle context are compressed. When training with NTK-aware PIC-SFT, standard inference retains more of the model’s original capabilities, while NTK-aware PIC Inference focuses more on long-form text generation tasks. This observation suggests the potential for decoupling a model’s abilities in short-form text generation and long-form text generation through PIC-SFT.

4.4 Ablation Study on Dynamic PIC

To investigate the effectiveness of each module in Dynamic PIC, we have conducted an ablation study, with the results shown in Table 3 and Figure 4. It can be observed that the Recent Context (RC) is crucial for Dynamic PIC. Removing RC leads to a significant drop in both generation length and generation quality, as well as an increase in the corrupted ratio, resulting in an overall decline in output performance. As for the Initial Context (IC), its impact on the long-form text generation tasks is not significant. However, based on the conclusions from StreamingLLM (Xiao et al., 2023), the Initial Context is critical for long-context understanding. For consistency, we choose to include the Initial Context by default. Lastly, the length of RC has a noticeable effect on the generation length and generation quality. As shown in Figure 4, when the length of RC is smaller, the model tends to produce longer outputs, but the generation quality decreases.

5 Conclusion

In this work, we introduce Position ID Compression (PIC), a simple, efficient, and plug-in approach to enhance the long-form text generation capabilities of LLMs. By compressing position ids, PIC enables models to generate longer content without additional training. We propose two improved variants, NTK-aware PIC and Dynamic PIC, which achieve better efficiency and flexibility. To further improve long-form text generation, we combine PIC with SFT and propose PIC-SFT, which achieves up to a 5x increase in generation length while maintaining the generation quality. Through extensive experiments, we show that PIC-SFT achieves state-of-the-art performance on both in-distribution and out-of-distribution datasets, with significantly reduced corrupted ratios compared to other methods. Additionally, the observed ability of PIC-SFT to decouple short-text and long-text capabilities highlights its potential for more flexi-
ble deployment in future LLM applications. These findings suggest that proper design and configuration of position id compression strategies are key to unlocking the full potential of long-form text generation in LLMs.

Limitations

The choice of the judge model The quality evaluation for HelloBench and LongBench-Write involves the use of the judge model. In this work, to ensure the reproducibility of results while saving resources, we selected the latest open-source model, LLaMA-3.3-70B-Instruct, as our judge model. However, in the original works of HelloBench and LongBench-Write, GPT-4o was used for evaluation. In the future, we plan to conduct evaluations using GPT-4o and provide the corresponding results.

Further exploration on the decoupling of PIC-SFT In Section 4.3, we have discussed the decoupling phenomenon of PIC-SFT. This interesting finding suggests that the model’s capabilities might be decoupled and expressed through different inference methods. However, in this work, we focus on identifying and explaining this phenomenon, without conducting further exploration. We consider further investigation of this phenomenon as future work.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62477001.

References

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023a. Qwen technical report. arXiv preprint arXiv:2309.16609.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023b. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.

Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055.

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.

Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72.

bloc97. 2023a. Add ntk-aware interpolation "by parts" correction.

bloc97. 2023b. Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukas Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.

Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.

Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. 2024. Security and privacy challenges of large language models: A survey. arXiv preprint arXiv:2402.00888.

Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanuyan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Md Meftahul Ferdaus, Mahdi Abdelguerfi, Elias Ioup, Kendall N Niles, Ken Pathak, and Steven Sloan. 2024. Towards trustworthy ai: A review of ethical and robust large language models. arXiv preprint arXiv:2407.13934.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. 2023. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801.
Jian Guan, Zhuoer Feng, Yamei Chen, Ruilin He, Xiaoxi Mao, Changjie Fan, and Minlie Huang. 2022. Lot: A story-centric benchmark for evaluating chinese long text understanding and generation. Transactions of the Association for Computational Linguistics, 10:434–451.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Pedram Hosseini, Jessica M Sin, Bing Ren, Bryce-tom G Thomas, Elnaz Nouri, Ali Farahanchi, and Saeed Hassanpour. 2024. A benchmark for long-form medical question answering. arXiv preprint arXiv:2411.09834.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232.

Arthur Jacot, Franck Gabriel, and Clément Hongler. 2018. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31.

Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.

Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. 2024. A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Vici-nagearth, 1(1):9.

Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wen-cong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, et al. 2024. Infinite-llm: Efficient llm service for long context with distat-tention and distributed kvcache. arXiv preprint arXiv:2401.02669.

Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. 2024a. Longgenbench: Long-context generation benchmark. arXiv preprint arXiv:2410.04199.

Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kunlai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, et al. 2024b. Cachegeen: Kv cache compression and streaming for fast large language model serving. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 38–56.

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071.

Chau Minh Pham, Simeng Sun, and Mohit Iyyer. 2024. Suri: Multi-constraint instruction following for long-form text generation. arXiv preprint arXiv:2406.19371.

Shanghaoran Quan, Tianyi Tang, Bowen Yu, An Yang, Dayiheng Liu, Bofei Gao, Jianhong Tu, Yichang Zhang, Jingren Zhou, and Junyang Lin. 2024. Language models can self-lengthen to generate long texts. arXiv preprint arXiv:2410.23933.

Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, et al. 2024. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409.16191.

Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927.

Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. 2021. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3531–3539.

Jianlin Su, Muradtha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.

Haochen Tan, Zhijiang Guo, Zhan Shi, Lu Xu, Zhili Liu, Yunlong Feng, Xiaoguang Li, Yasheng Wang, Lifeng Shang, Qun Liu, et al. 2024. Proxyqa: An alternative framework for evaluating long-form text generation with large language models. arXiv preprint arXiv:2401.15042.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al.
2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro-Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, et al. 2024. Long-form factuality in large language models. arXiv preprint arXiv:2403.18802.

Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Owen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, and Danqi Chen. 2025. Longproc: Benchmarking long-context language models on long procedural generation. arXiv preprint arXiv:2501.05414.

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyuan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372.

Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wen-hao Wu, Furu Wei, and Sujian Li. 2023. Pose: Efficient context window extension of llms via positional skip-wise training. arXiv preprint arXiv:2309.10400.

A Details of Metrics

For HelloBench and LongBench-Write, we use different metrics to represent the generation length and generation quality. For HelloBench, we measure generation quality using the HelloEval scores from HelloBench, which are referred to as “Quality” in the experimental table for simplicity. It is worth noting that we do not adopt Score Rescaling as described in the original paper. For LongBench-Write, we use the \( \bar{S} \) metric as defined by Bai et al. (Bai et al., 2024) to represent the overall generation quality. For MMLU evaluation, we use 5-shot, and for GSM8K evaluation, we use 4-shot combined with Chain-of-Thought (CoT).

In addition, we have used the Corrupted Ratio to measure the model’s effective generation rate. Since different models may have varying numbers of corrupted samples, we ensure comparability by following a specific approach. For models with a Corrupted Ratio of less than 20%, we have calculated the average of non-corrupted examples shared among compared models as the final experimental result, ensuring direct comparability. For models with a Corrupted Ratio greater than 20%, including their corrupted examples in the shared non-corrupted set would result in too few examples for averaging, diminishing comparability. Therefore, for models with a Corrupted Ratio above 20%, we have computed their averaged results independently. We use the NLTK (Bird, 2006) library to calculate the generated length. Additionally, we have observed that most repetitive samples do not end with the typical end-of-token of LLMs. Therefore, we determine whether a sample is corrupted based on regular expressions and whether it ends with the end-of-token.

Through comparisons, we have observed that the error introduced by considering or not considering the shared non-corrupted examples is less than 5%. However, for the sake of rigor, we provide this clarification.

B Additional Experimental Results

Table 4: Results of different SFT methods evaluated on MMLU and GSM8K. The base model is LLaMA-3.1-8B-Instruct. * represents standard inference, † represents NTK-aware PIC inference, and ‡ represents Dynamic PIC inference.

<table>
  <tr>
    <th>Method</th>
    <th>MMLU</th>
    <th>GSM-8K</th>
  </tr>
  <tr>
    <td>w/o SFT</td>
    <td>66.17</td>
    <td>80.64</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>46.48</td>
    <td>64.82</td>
  </tr>
  <tr>
    <td>NTK-aware PIC-SFT*</td>
    <td>50.33</td>
    <td>65.73</td>
  </tr>
  <tr>
    <td>NTK-aware PIC-SFT†</td>
    <td>47.82</td>
    <td>61.71</td>
  </tr>
  <tr>
    <td>Dynamic PIC-SFT*</td>
    <td>54.77</td>
    <td>66.57</td>
  </tr>
  <tr>
    <td>Dynamic PIC-SFT‡</td>
    <td>50.10</td>
    <td>59.21</td>
  </tr>
</table>
Table 5: The experimental results of Naive PIC with varying Compression Ratio are presented. “CR” denotes Corrupted Ratio. The blue row represents the baseline.

<table>
  <tr>
    <th rowspan="2">Compression Ratio</th>
    <th colspan="4">LLaMA-3.1-8B-Instruct</th>
    <th colspan="4">Qwen-2.5-7B-Instruct</th>
    <th colspan="4">GLM-4-9B-Chat</th>
  </tr>
  <tr>
    <th>Length</th><th>Tokens</th><th>Quality</th><th>CR</th>
    <th>Length</th><th>Tokens</th><th>Quality</th><th>CR</th>
    <th>Length</th><th>Tokens</th><th>Quality</th><th>CR</th>
  </tr>
  <tr><td>0.25</td><td>0</td><td>0</td><td>0</td><td>100</td><td>0</td><td>0</td><td>0</td><td>100</td><td>0</td><td>0</td><td>0</td><td>100</td></tr>
  <tr><td>0.33</td><td>0</td><td>0</td><td>0</td><td>100</td><td>0</td><td>0</td><td>0</td><td>100</td><td>0</td><td>0</td><td>0</td><td>100</td></tr>
  <tr><td>0.5</td><td>779.26</td><td>806.91</td><td>73.91</td><td>43.09</td><td>824.68</td><td>850.16</td><td>69</td><td>59.35</td><td>802.42</td><td>850.29</td><td>77.82</td><td>6.5</td></tr>
  <tr><td>0.67</td><td>904.13</td><td>947</td><td>77.91</td><td>6.5</td><td>841.76</td><td>869.94</td><td>77.8</td><td>13.01</td><td>896.15</td><td>945.18</td><td>78.87</td><td>1.63</td></tr>
  <tr><td>0.75</td><td>959.35</td><td>998.88</td><td>78.52</td><td>4.07</td><td>832.16</td><td>860.22</td><td>77.65</td><td>11.38</td><td>946.47</td><td>994.15</td><td>79.64</td><td>0.81</td></tr>
  <tr><td>1</td><td>1001.86</td><td>1055.42</td><td>79.85</td><td>1.63</td><td>958.33</td><td>990.54</td><td>80.17</td><td>4.07</td><td>990.09</td><td>1039.23</td><td>81.07</td><td>2.44</td></tr>
  <tr><td>2</td><td>691.76</td><td>725.72</td><td>71.96</td><td>13.01</td><td>858.95</td><td>905.18</td><td>71.51</td><td>54.47</td><td>1120.43</td><td>1109.97</td><td>71.17</td><td>46.34</td></tr>
  <tr><td>3</td><td>593.78</td><td>618.3</td><td>43.9</td><td>43.9</td><td>876.75</td><td>917.61</td><td>66.44</td><td>77.24</td><td>1233.74</td><td>1288.02</td><td>60.05</td><td>65.85</td></tr>
  <tr><td>4</td><td>266.61</td><td>279.64</td><td>10.72</td><td>77.24</td><td>1660</td><td>1721.57</td><td>54.1</td><td>94.31</td><td>2304.38</td><td>2453.56</td><td>24.3</td><td>84.55</td></tr>
  <tr><td>8</td><td>27.67</td><td>28.76</td><td>0</td><td>22.76</td><td>0</td><td>0</td><td>0</td><td>100</td><td>5937.6</td><td>6642.4</td><td>0</td><td>95.12</td></tr>
  <tr><td>16</td><td>6.63</td><td>7.88</td><td>0</td><td>2.44</td><td>72.27</td><td>85.45</td><td>0</td><td>91.06</td><td>1923.5</td><td>2076</td><td>0</td><td>98.37</td></tr>
</table>

Table 6: The experimental results of NTK-aware PIC with varying RoPE Base Scaling Ratio are presented. “RBSR” denotes RoPE Base Scaling Ratio. “CR” denotes Corrupted Ratio. The blue row represents the baseline.

<table>
  <tr>
    <th rowspan="2">RBSR</th>
    <th colspan="4">LLaMA-3.1-8B-Instruct</th>
    <th colspan="4">Qwen-2.5-7B-Instruct</th>
    <th colspan="4">GLM-4-9B-Chat</th>
  </tr>
  <tr>
    <th>Length</th><th>Tokens</th><th>Quality</th><th>CR</th>
    <th>Length</th><th>Tokens</th><th>Quality</th><th>CR</th>
    <th>Length</th><th>Tokens</th><th>Quality</th><th>CR</th>
  </tr>
  <tr><td>0.25</td><td>693.05</td><td>721.01</td><td>76.45</td><td>0.81</td><td>812.38</td><td>837.8</td><td>77.14</td><td>4.07</td><td>866.12</td><td>909.25</td><td>79.39</td><td>0.81</td></tr>
  <tr><td>0.33</td><td>770.85</td><td>806.58</td><td>77.54</td><td>0</td><td>797.55</td><td>823.27</td><td>78.95</td><td>4.88</td><td>869.17</td><td>914.07</td><td>79.18</td><td>0</td></tr>
  <tr><td>0.5</td><td>849.8</td><td>888.56</td><td>78.97</td><td>0.81</td><td>890.93</td><td>922.84</td><td>78.91</td><td>6.5</td><td>909.26</td><td>954.31</td><td>79.62</td><td>0.81</td></tr>
  <tr><td>0.67</td><td>882.79</td><td>925.79</td><td>79.49</td><td>0</td><td>962.98</td><td>997.83</td><td>78.67</td><td>5.69</td><td>944.34</td><td>989.4</td><td>80.02</td><td>0.81</td></tr>
  <tr><td>0.75</td><td>924.98</td><td>972.79</td><td>80.79</td><td>0.81</td><td>866.75</td><td>897.66</td><td>78.89</td><td>5.69</td><td>980.88</td><td>1025.65</td><td>80.86</td><td>0.81</td></tr>
  <tr><td>1</td><td>1020.8</td><td>1071.84</td><td>80.97</td><td>1.63</td><td>954.75</td><td>988.72</td><td>80.98</td><td>4.07</td><td>986.97</td><td>1031.88</td><td>81.03</td><td>2.44</td></tr>
  <tr><td>2</td><td>1259.69</td><td>1326.76</td><td>81.59</td><td>0</td><td>1042.36</td><td>1080.24</td><td>80.22</td><td>3.25</td><td>1051.68</td><td>1096.56</td><td>81.62</td><td>2.44</td></tr>
  <tr><td>3</td><td>1381.08</td><td>1454.8</td><td>82.34</td><td>6.5</td><td>1092.58</td><td>1128.23</td><td>80.29</td><td>8.13</td><td>1106.7</td><td>1156.73</td><td>81.52</td><td>0</td></tr>
  <tr><td>4</td><td>1522.97</td><td>1604.71</td><td>82.38</td><td>7.32</td><td>1140.97</td><td>1178.38</td><td>80.98</td><td>6.5</td><td>1154.25</td><td>1203.84</td><td>82.16</td><td>1.63</td></tr>
  <tr><td>8</td><td>1675.86</td><td>1765.35</td><td>81.67</td><td>22.76</td><td>1256.05</td><td>1303.24</td><td>81.13</td><td>9.76</td><td>1245.97</td><td>1294.94</td><td>82.88</td><td>0.81</td></tr>
  <tr><td>16</td><td>1802.11</td><td>1902.17</td><td>81.05</td><td>34.15</td><td>1391.56</td><td>1436.65</td><td>81.77</td><td>13.01</td><td>1352.04</td><td>1404.38</td><td>82.52</td><td>1.63</td></tr>
  <tr><td>32</td><td>1811.1</td><td>1917.76</td><td>80.77</td><td>41.46</td><td>1519.52</td><td>1573.89</td><td>81.27</td><td>22.76</td><td>1453.24</td><td>1505.29</td><td>83.98</td><td>2.44</td></tr>
  <tr><td>64</td><td>1676.64</td><td>1812.77</td><td>77.61</td><td>61.79</td><td>1585.94</td><td>1651.06</td><td>80.71</td><td>47.15</td><td>1605.64</td><td>1666.28</td><td>84.03</td><td>4.07</td></tr>
  <tr><td>128</td><td>1493.38</td><td>1583.02</td><td>77.86</td><td>61.79</td><td>1761.41</td><td>1832.64</td><td>80.11</td><td>46.34</td><td>1701.72</td><td>1765.42</td><td>83.98</td><td>9.76</td></tr>
  <tr><td>256</td><td>1312.87</td><td>1382.3</td><td>73.13</td><td>81.3</td><td>1776.43</td><td>1867.52</td><td>77.41</td><td>62.6</td><td>1798.69</td><td>1865.31</td><td>83.05</td><td>23.58</td></tr>
  <tr><td>512</td><td>1066.64</td><td>1118.73</td><td>73.45</td><td>91.06</td><td>1900.31</td><td>2006.42</td><td>74.84</td><td>70.73</td><td>1880.8</td><td>1953.73</td><td>81.86</td><td>31.71</td></tr>
</table>

Table 7: The experimental results of Dynamic PIC with varying Compression Ratio are presented. “CR” denotes Corrupted Ratio. The blue row represents the baseline.

<table>
  <tr>
    <th rowspan="2">Compression Ratio</th>
    <th colspan="4">LLaMA-3.1-8B-Instruct</th>
    <th colspan="4">Qwen-2.5-7B-Instruct</th>
    <th colspan="4">GLM-4-9B-Chat</th>
  </tr>
  <tr>
    <th>Length</th><th>Tokens</th><th>Quality</th><th>CR</th>
    <th>Length</th><th>Tokens</th><th>Quality</th><th>CR</th>
    <th>Length</th><th>Tokens</th><th>Quality</th><th>CR</th>
  </tr>
  <tr><td>0.25</td><td>816.65</td><td>857.19</td><td>79.3</td><td>0</td><td>823.67</td><td>854.88</td><td>78.88</td><td>4.88</td><td>930.33</td><td>973.37</td><td>79.92</td><td>3.25</td></tr>
  <tr><td>0.33</td><td>858.72</td><td>902.11</td><td>79.84</td><td>1.63</td><td>858.68</td><td>889.78</td><td>78.36</td><td>4.07</td><td>899.69</td><td>941.86</td><td>80.5</td><td>0.81</td></tr>
  <tr><td>0.5</td><td>893.25</td><td>940.46</td><td>79.15</td><td>0.81</td><td>864.46</td><td>895.39</td><td>79.25</td><td>5.69</td><td>937.21</td><td>980.22</td><td>79.4</td><td>1.63</td></tr>
  <tr><td>0.67</td><td>945.2</td><td>994.1</td><td>80.16</td><td>1.63</td><td>883.42</td><td>916.3</td><td>78.91</td><td>5.69</td><td>931.02</td><td>974.52</td><td>81.11</td><td>1.63</td></tr>
  <tr><td>0.75</td><td>982.13</td><td>1033.77</td><td>80.23</td><td>0.81</td><td>902</td><td>933.7</td><td>79.13</td><td>5.69</td><td>951.4</td><td>993.65</td><td>80.63</td><td>0.81</td></tr>
  <tr><td>1</td><td>1024.05</td><td>1076.16</td><td>81.07</td><td>1.63</td><td>966.61</td><td>1001.67</td><td>80.19</td><td>4.07</td><td>984.5</td><td>1029.95</td><td>80.86</td><td>2.44</td></tr>
  <tr><td>2</td><td>1155.42</td><td>1211.42</td><td>81.89</td><td>1.63</td><td>1075.7</td><td>1112.8</td><td>80.12</td><td>8.13</td><td>1051.12</td><td>1101.73</td><td>81.11</td><td>1.63</td></tr>
  <tr><td>3</td><td>1283.81</td><td>1346.92</td><td>81.54</td><td>0.81</td><td>1104.83</td><td>1144.6</td><td>79.55</td><td>13.01</td><td>1130.18</td><td>1179.9</td><td>82</td><td>0.81</td></tr>
  <tr><td>4</td><td>1380.63</td><td>1444.02</td><td>81.31</td><td>0.81</td><td>1125.13</td><td>1164.26</td><td>81.58</td><td>17.07</td><td>1154.62</td><td>1206.48</td><td>81.75</td><td>0.81</td></tr>
  <tr><td>8</td><td>1656.45</td><td>1734.45</td><td>81.88</td><td>6.5</td><td>1282.11</td><td>1324.32</td><td>82.35</td><td>13.82</td><td>1234.58</td><td>1288.32</td><td>82.27</td><td>5.69</td></tr>
  <tr><td>16</td><td>1938.24</td><td>2028.61</td><td>81.12</td><td>24.39</td><td>1520.81</td><td>1569.01</td><td>80.93</td><td>22.76</td><td>1347.92</td><td>1406.73</td><td>82.72</td><td>4.88</td></tr>
  <tr><td>32</td><td>1919.77</td><td>2022.68</td><td>76.68</td><td>47.15</td><td>1479.34</td><td>1530.09</td><td>81.58</td><td>30.08</td><td>1428.65</td><td>1487.92</td><td>82.88</td><td>5.69</td></tr>
  <tr><td>64</td><td>1817.23</td><td>1916.1</td><td>76.08</td><td>60.98</td><td>1454.23</td><td>1501.58</td><td>79.96</td><td>31.71</td><td>1441.52</td><td>1502.96</td><td>83.45</td><td>4.88</td></tr>
  <tr><td>128</td><td>1961.89</td><td>2101.06</td><td>75.31</td><td>61.79</td><td>1490.53</td><td>1543.54</td><td>81.96</td><td>35.77</td><td>1478.12</td><td>1538.12</td><td>83.43</td><td>4.07</td></tr>
  <tr><td>256</td><td>1737.69</td><td>1858.37</td><td>74.45</td><td>71.54</td><td>1471.03</td><td>1526.04</td><td>79.8</td><td>38.21</td><td>1462.22</td><td>1523.08</td><td>82.91</td><td>4.07</td></tr>
  <tr><td>512</td><td>1576.7</td><td>1691.36</td><td>75.23</td><td>73.17</td><td>1460.7</td><td>1510.07</td><td>80.89</td><td>38.21</td><td>1507.17</td><td>1572.32</td><td>83.4</td><td>4.07</td></tr>
</table>
C  The impact of different inference methods

We further conduct experiments on the different inference methods on PIC-SFT, the experimental results are shown in Table 8. We observe that, compared to NTK-aware PIC Inference, standard inference tends to generate shorter content with lower generation quality. Similarly, compared to Dynamic PIC Inference, standard inference also produces shorter content and lower generation quality. This indirectly indicates that after NTK-aware PIC-SFT, the long-form text generation capability of standard inference weakens. This shows that PIC Inference remains effective after SFT, consistent with the trend identified in Section 4.2.

D  Potential Risks

LLMs have been observed to exhibit inherent biases, generating content that may contain discrimination in various aspects such as politics, gender, and race (Das et al., 2024; Ferdaus et al., 2024) due to biased training data. The harmful stereotypes manifested in the generated content can contribute to the oppression of those at social margins (Wiedlinger et al., 2021). Therefore, in various long-form text generation fields such as creative writing and story continuation, it is crucial to ensure that the relevant long texts generated by LLMs do not contain harmful stereotypes. Additionally, LLMs are prone to hallucinations, often generating information that is factually incorrect or non-existent (Huang et al., 2023; Sahoo et al., 2024). This issue is particularly prominent in applications requiring high accuracy, such as academic paper editing and news writing, where the dissemination of incorrect information can have serious consequences. Ensuring that LLMs generate reliable and accurate long texts is essential to maintain the credibility of the generated content.

E  Computational Budget for PIC-SFT

The computation budget for NTK-aware PIC-SFT is 20 GPU hours and the computation budget for Dynamic PIC is 60 GPU hours.
Table 8: The experimental results of different SFT methods with different inference methods. The base model is LLaMA-3.1-8B-Instruct. * represents standard inference, † represents NTK-aware PIC inference, and ‡ represents Dynamic PIC inference.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">HelloBench-HTG</th>
    <th colspan="4">LongBench-Write</th>
  </tr>
  <tr>
    <th>Length ↑</th>
    <th>Quality ↑</th>
    <th>CR ↓</th>
    <th>S_l ↑</th>
    <th>S_q↑</th>
    <th>Ś ↑</th>
    <th>CR ↓</th>
  </tr>
  <tr>
    <td>SFT*</td>
    <td>4207.51</td>
    <td>87.37</td>
    <td>3.25</td>
    <td>76.8</td>
    <td>98.37</td>
    <td>87.59</td>
    <td>1.67</td>
  </tr>
  <tr>
    <td>SFT†</td>
    <td>5166.1</td>
    <td>88.58</td>
    <td>21.14</td>
    <td>74.03</td>
    <td>98.13</td>
    <td>86.08</td>
    <td>13.33</td>
  </tr>
  <tr>
    <td>SFT‡</td>
    <td>4632.37</td>
    <td>87.95</td>
    <td>3.25</td>
    <td>80.02</td>
    <td>97.97</td>
    <td>89.0</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>NTK-aware PIC-SFT*</td>
    <td>3717.72</td>
    <td>87.34</td>
    <td>12.2</td>
    <td>80.29</td>
    <td>97.32</td>
    <td>88.80</td>
    <td>5.83</td>
  </tr>
  <tr>
    <td>NTK-aware PIC-SFT†</td>
    <td>4582.63</td>
    <td>89.03</td>
    <td>4.07</td>
    <td>80.08</td>
    <td>97.56</td>
    <td>88.82</td>
    <td>1.67</td>
  </tr>
  <tr>
    <td>Dynamic PIC-SFT*</td>
    <td>4413.57</td>
    <td>87.01</td>
    <td>3.25</td>
    <td>75.34</td>
    <td>97.56</td>
    <td>86.45</td>
    <td>3.33</td>
  </tr>
  <tr>
    <td>Dynamic PIC-SFT‡</td>
    <td>5098.45</td>
    <td>89.48</td>
    <td>4.07</td>
    <td>82.49</td>
    <td>96.99</td>
    <td>89.74</td>
    <td>1.67</td>
  </tr>
</table>

Reviewer paper summaries: The paper introduces Position ID Compression (PIC), a simple and efficient technique to enhance the long-form text generation capabilities of large language models (LLMs) by compressing their position IDs. The authors propose two refined methods, NTK-aware PIC and Dynamic PIC, enabling approximately 1.5 times longer generation without additional training or quality loss. Furthermore, combining PIC with supervised fine-tuning (PIC-SFT) boosts performance significantly, allowing models to generate outputs up to five times longer while maintaining high quality, as demonstrated by strong results on benchmark datasets HelloBench and LongBench-Write.

This paper focuses on enhancing the long - form text generation capabilities of large language models (LLMs). It proposes Position ID Compression (PIC) in three variants: Naive PIC, NTK-aware PIC (which modifies the RoPE Base to indirectly compress position ids), and Dynamic PIC (compressing middle-context position ids with adjusted KV cache handling). These are integrated with Supervised Fine-Tuning (SFT) to create PIC-SFT methods. The study evaluates generation length, quality, and corrupted ratio using metrics from HelloBench and LongBench-Write, with accuracy for MMLU and GSM8K. NTK-aware PIC and Dynamic PIC increase generation length without degrading quality, and Dynamic PIC-SFT maintains content validity. The paper also reveals PIC-SFT decoupling and through ablation shows the importance of the Recent Context in Dynamic PIC, overall demonstrating the effectiveness of these methods in optimizing long-form text generation in LLMs.
Summary of strengths: 1. This paper considers enhancing the long-text generation ability of the model from the perspective of position encoding, which is a very good perspective. The writing of the paper is also clear, and it demonstrates the relevant implementation details. However, the results are not satisfactory, at least for me.
2. By combining Supervised Fine-Tuning (SFT), the proposed method can effectively enhance the long-form generation abilities. However, this will bring about an additional alignment tax, which affects the generalizability.

1.The paper introduces PIC, a method to compress position ids in LLMs for extended long-form text generation without extra training. It also presents two variants, NTK-aware PIC and Dynamic PIC, which enhance PIC's efficiency and effectiveness.
2.By combining PIC with SFT to form PIC-SFT, the paper enables LLMs to extend generation length up to 5 times the original while still maintaining short-generation capabilities, beneficial for various applications.
3.Using metrics from multiple benchmarks, experiments show that NTK-aware PIC and Dynamic PIC increase generation length without degrading quality. The ablation study in Dynamic PIC reveals the significance of the Recent Context, offering practical guidance for LLM optimization.
Summary of weaknesses: 1. The specific calculation process of the Corrupted Ratio is not clear, which makes the results unconvincing.
2. When Supervised Fine-Tuning (SFT) is not used, although NTK-aware Position Interpolation (PIC) and Dynamic PIC can increase the generation length and even improve the generation quality, they significantly increase the Corrupted Ratio. Thus, it is difficult to directly use them as a plug-and-play method. Although the Corrupted Ratio decreases significantly after adding SFT, further SFT brings losses in general tasks, making it difficult for the method to generalize.
3. I admit that it is a good practice to use LLaMA-3.3-70B-Instruct to ensure reproducibility. However, considering the inherent capability gap between LLaMA-3.3-70B-Instruct and GPT-4o, the experimental results seem to be unconvincing.
4. I admit that using greedy decoding to ensure reproducibility is of great significance in complex reasoning tasks such as math and code. However, in text-writing, to ensure diversity and quality, hyperparameters such as temperature, top_p, and top_k are usually set. You can try strategies like setting a random seed to ensure reproducibility.

1.In Section 4.3, the lengths of the initial context and recent context in Dynamic PIC-SFT do not match the parameters of Dynamic PIC in Table 1. Is there a lack of hyperparameter tuning steps?
2.The authors' description of the text-splitting operation in the Dynamic PIC-SFT method is not clear enough, and there are no corresponding hyperparameter experiments. This makes the conclusion that "the training stage closely approximates the actual inference stage" less reliable.
3.Dynamic PIC requires real-time updating of position IDs and adjustment of the storage method of the KV cache, which may increase the computational complexity and memory usage during inference. However, the paper fails to provide specific data on time or resource consumption, such as latency and GPU memory usage.
Comments, suggestions and typos: See Weakness

1.It could be considered to add an experiment to verify the performance of applying NTK-aware PIC and Dynamic PIC to models without SFT in short-form text generation tasks.
Overall assessment scores: 3.5

3.5