What is Wrong with Perplexity for Long-context Language Modeling?

## Abstract

Abstract Handling long-context inputs is crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning. While recent approaches have extended the context windows of LLMs and employed perplexity (PPL) as a standard evaluation metric, PPL has proven unreliable for assessing long-context capabilities. The underlying cause of this limitation has remained unclear.
In this work, we provide a comprehensive explanation for this issue. We find that PPL overlooks key tokens, which are essential for long-context understanding, by averaging across all tokens and thereby obscuring the true performance of models in long-context scenarios. To address this, we propose LongPPL , a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them. Our experiments demonstrate that LongPPL strongly correlates with performance on various long-context benchmarks (e.g., Pearson correlation of -0.96), significantly outperforming traditional PPL in predictive accuracy. Additionally, we introduce LongCE (Long-context Cross-Entropy) loss, a re-weighting strategy for fine-tuning that prioritizes key tokens, leading to consistent improvements across diverse benchmarks. In summary, these contributions offer deeper insights into the limitations of PPL and present effective solutions for accurately evaluating and enhancing the long-context capabilities of LLMs. Code is available at https://github.com/PKU-ML/LongPPL .

## 1 Introduction

The ability to process long-context inputs is critical for large language models (LLMs) in many real-world tasks, such as long conversations (Maharana et al., ), document summarization (Chang et al., ), and many-shot in-context learning (Agarwal et al., ; Li et al., ). Despite many techniques for extending the context length (Han et al., ; Chen et al., ; Zhu et al., ; Xiong et al., ; Chen et al., ), the evaluation of long-context capabilities still heavily relies on perplexity (PPL) as the *de facto* metric. Many have claimed to extend context windows to 32k, 128k, or even millions of tokens, based on attaining a low perplexity score under long context. However, recent studies have challenged this common practice by revealing a huge discrepancy between perplexity and actual performance on long-context tasks (Hu et al., ; Hsieh et al., ). As shown in Figure 1(b) (top), the perplexity of LLMs shows almost no correlation to their long-context performance measured by Longbench scores (Bai et al., ). This raises the question:

*
Why does perplexity fail to reflect the long-context abilities of LLMs?*

Figure: (a) Illustration of how LongPPL is calculated.
Refer to caption: x1.png

To understand this phenomenon, we conduct a fine-grained analysis of the roles of different tokens at long-context tasks. Notably, we find perplexity computed only on the answer tokens to the long-context tasks strongly correlates with LongEval accuracy, whereas perplexity on non-answer tokens shows little to no correlation. Since most tokens are non-answer tokens, standard perplexity averaging over all token *equally* fails to represent the long-context abilities. This motivates us to average over the *key tokens* that reflect a model’s long-context abilities. A key obstacle is that natural texts have no ground-truth reference of key tokens, making it hardly applicable to general cases.

To tackle this challenge, we propose a principled method to measure the influence of long context on each token by performing a causal intervention on its context length. We find that tokens with significantly better predictions under long context are strongly tied to long-context information, even though they make up only a small portion of general text. Empirically, our proposed method can accurately identify the answer tokens in LongEval with up to 98.2% accuracy.

Built upon the accurate selection of key tokens, we propose LongPPL (Long-context Perplexity), where we compute perplexity by only averaging solely on the selected key tokens (Figure 1(a)). Extensive experiments across a diverse suite of LLMs and long-context benchmarks show that LongPPL computed on natural language corpus exhibits a consistently strong correlation with their benchmark scores computed over various long-context tasks, e.g., -0.96 correlation in Figure 1(b) (bottom). Thus, LongPPL offers a natural way to evaluate LLMs’ long-context capabilities in an *unsupervised* fashion.

Following the design of LongPPL, we further develop an efficient long-context training strategy by *emphasizing key tokens*. Specifically, we propose the LongCE (Long-context Cross-Entropy) loss that upweights the key tokens, which can be estimated by the model itself. In this way, LongCE can bootstrap its long-context abilities by alternating between estimating key tokens and optimizing key tokens. Experimental results across multiple LLMs show that LongCE consistently improves over the conventional CE loss, with a maximum accuracy gain of 22% on LongEval.

Our contributions are summarized as follows:

- •
We conduct a fine-grained analysis on the failure of perplexity at measuring long-context abilities. Specifically, we reveal the critical roles of key tokens in long-context tasks and propose principled metrics to identify key tokens with high accuracy.
- •
We propose LongPPL (Long-context Perplexity) that is solely based on the selected key tokens. Extensive evaluation shows that in contrast to standard PPL, LongPPL exhibits a strong correlation with long-context abilities across multiple LLMs and benchmarks.
- •
We introduce LongCE (Long-context Cross Entropy) loss that assigns larger weights to key tokens that gain more from the long context. LongCE attains consistent improvements in a plug-and-play solution, demonstrating its generality for learning long-context models.

## 2 A Fine-grained Analysis of Perplexity

Recent studies have shown that perplexity does not adequately reflect long-context performance of language models (Agarwal et al., ; Li et al., ), as we have also observed in Figure 1(b). In this section, we demystify this phenomenon with a fine-grained analysis of the roles of different tokens at long-context performance.

#### Preliminaries

Perplexity is a commonly used metric for evaluating a LM’s ability to predict the next word in a sequence (Bengio et al., ; Press et al., ). For a sequence of tokens $\bm{x}=(x_{1},x_{2},...,x_{n})$, a language model parameterized by $\theta$ is learned to predict the conditional probability of each token given the previous context $P_{\theta}(x_{i}|\bm{x}_{<i}),i\in[n]$. The perplexity (PPL) on this sequence is defined as the inverse of the geometric mean of all token probabilities:

$$ $\text{PPL}_{\theta}(\bm{x})=\exp\left(-\frac{1}{n}\sum_{i=1}^{n}\log P_{\theta }(x_{i}|\bm{x}_{<i})\right)=P_{\theta}(\bm{x})^{-\frac{1}{n}}.$ (1) $$

It quantifies the model’s uncertainty when encountering new tokens. A larger likelihood of $\bm{x}$ indicates better prediction and lower perplexity.

Figure: (a) Example of answer tokens.
Refer to caption: x3.png

### 2.1 Not All Tokens Matter for Long-context Performance

Despite the close connection between perplexity and token prediction accuracy, there is growing evidence that LMs’ perplexity does not indicate their performance on long-context benchmarks (Hu et al., ; Hsieh et al., ). There are two possible sources of this mismatch: either the log-likelihood-based metric is flawed, or the averaged tokens are not representative enough. In this work, we *champion the latter explanation* by showing that when selecting the proper “key tokens” for long-context understanding, perplexity can correlate very well with long-context performance.

To have an intuitive understanding, let us consider a real example from LongEval benchmark shown in Figure 2(a). Most tokens in the answer, “the <REGISTER_CONTENT> in
line tender-clause is”, are straightforward answer formats stemmed immediately from the question, without relying on any long-context information. Even short-context LMs can predict well on these tokens. Since most tokens are *long-context-agnostic* tokens, perplexity computed *equally* over all tokens do not represent long-context performance.

To quantitatively examine this hypothesis, we conduct experiments on LongEval (Li et al., ), a benchmark for long-context retrieval abilities, where we can separate the *answer tokens* that match the desired answers (e.g., <45129> in Figure 2(a)) from *non-answer tokens*.

We compare the perplexity computed with these two groups of tokens using two long-context LLMs. As shown in Figures 2(b) & 2(c), the perplexity on answer tokens correlates strongly with the LongEval accuracy that represents the long-context performance; instead, the perplexity on the non-answer tokens shows almost no correlation with LongEval accuracy, justifying our intuition that these tokens do not matter for evaluating long-context performance. In other words, we should evaluate the perplexity of the key tokens that really matter for long-context performance, which is the focus of our next section.

### 2.2 Extracting Key Tokens from Natural Texts

In natural texts used for training LLMs, we do not have knowledge of the answer tokens as in LongEval experiments (Figure 2). This motivates us to find a surrogate metric that can accurately identify the key tokens that matter for long-context performance.

Context Intervention.
To measure the influence of long context for each token $x_{i}$, we perform an *intervention* of context length. Specifically, given a sequence ${\bm{x}}$ and a language model $P_{\theta}$ (with strong long-context abilities), for each token $x_{i}$ that has a long context, we compute the difference between its log probability under the full *long context* ${\bm{l}}_{i}=(x_{1},\dots,x_{i-1})$ and the log probability under the truncated *short context* ${\bm{s}}_{i}=(x_{i-K},\dots,x_{i-1})$ (where $K$ is a short length, e.g., 64):

$$ ${\rm LPG}_{\theta}(x_{i})=\log P_{\theta}(x_{i}|{\color[rgb]{1,.5,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}{\bm{l}}_{i}})-\log P_{\theta} (x_{i}|{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{\bm{ s}}_{i}}).$ (2) $$

We call it Log Probability Gain (LPG), which measures the improvement in prediction accuracy *endowed solely by the long context*. From a causal perspective, ${\bm{s}}_{i}$ serves as the counterfactual context created by the intervention (dropping long context), and the LPG estimates the individual treatment effect (ITE) (Hernán & Robins, ) of long context using the language model $P_{\theta}$. Thus, a high LPG value indicates that long context plays an important part in the prediction of $x_{i}$, making them the key tokens to be considered for evaluating long-context performance. In other words, LLMs good at long-context understanding should be able to predict high-LPG tokens accurately.

Figure: (a) LPG of tokens on LongEval.
Refer to caption: x6.png

We evaluate the LPG score on LongEval, where we have knowledge of the key answer tokens. As shown in Figure 3(a), we compute the LPG score with a powerful long-context LLM, Mixtral-8x7B (Jiang et al., ), and find that answer tokens are clearly separated from the non-answer tokens: most answer tokens have LPG values higher than $2$, while most of the non-answer tokens concentrate around low LPG values (lower than $0.5$). When using LPG values alone to classify answer and non-answer tokens, we attain 85.6% accuracy (Figure 4(b)), indicating that LPG values are strongly indicative of the key tokens in long-context understandings.

From Figure 3(a), we find that a small proportion of non-answer tokens also have large LPGs (larger than $0.5$) and are thus confused together with key tokens.
After analyzing, we find that these tokens can be further separated out by inspecting their Log Probability Values (LPV) under long context:

$$ ${\rm LPV}_{\theta}(x_{i})=\log P_{\theta}(x_{i}|{\color[rgb]{1,.5,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}{\bm{l}}_{i}})=\log P_{\theta} (x_{i}|{\bm{x}}_{<i}).$ (3) $$

A lower LPV indicates that the language model hardly predicts accurately at $x_{i}$ even with the long context information. Figure 3(b) shows that these high-LPG non-answer tokens actually have lower LPVs than the corresponding answer tokens, indicating that these tokens are (strongly) mispredicted tokens even under a long context. In other words, these tokens are fundamentally hard to predict regardless of the context. Therefore, we can exclude them from the selection of key tokens.

#### Summary: Why does Perplexity Fail?

To summarize, we revisit our initial question why perplexity fails to represent long-context performance.
As shown in Figure 4(a), most tokens in a natural corpus, GovReport (Huang et al., ), are long-context-irrelevant tokens with low LPG (lower than $0.5$), while only less than 10% tokens are highly influenced by long context (with LPG$>2$) and represent long-context abilities. Therefore, perplexity that averages over all tokens (Equation 1) does not represent the real long-context performance. Instead, combining the LPG (Equation 2) and the LPV (Equation 3) scores, we are able to accurately identify the answer tokens in LongEval with an accuracy of 98.2% (Figure 4(b)). Based on this result, in the next section, we design a new perplexity measure, LongPPL, that is tailored to reflect the long-context performance of LMs, by focusing on the key tokens.

Figure: (a) LPG value distribution on GovReport.
Refer to caption: x8.png

## 3 Measuring and Enhancing Long-Context Capabilities with Key Tokens

In Section 2, we find that only key tokens correlate well with long-context performance (Section 2.1), and we identify two effective measures to select the key tokens from a natural corpus (Section 2.2). Based on these observations, we design a new perplexity measure, LongPPL, to measure the long-context abilities, and, following in the same vein, we propose a new training objective, LongCE, for finetuning LLMs with an emphasis on key tokens.

### 3.1 Long-context Perplexity (LongPPL)

Given a sequence ${\bm{x}}=(x_{1},\dots,x_{n})$ and a language model $P_{\theta}$ to be evaluated, we consider a generalized notion of perplexity for long context understanding, Long-context Perplexity (LongPPL), where we can assign an influence function $I(\cdot):{\mathbb{X}}\to{\mathbb{R}}_{+}$ to each token $x_{i}$:

$$ $\displaystyle{\rm LongPPL}({\bm{x}};\theta,\theta_{0})=\exp\left(\sum_{i=1}^{n }-\hat{I}(x_{i};\theta_{0})\log P_{\theta}(x_{i}|\bm{x}_{<i})\right),$ (4) where $\displaystyle I(x_{i};\theta_{0})=\begin{cases}1,&\text{if }{\rm LPG}_{\theta_ {0}}(x_{i})>\alpha\text{ and }{\rm LPV}_{\theta_{0}}(x)>\beta;\\ 0,&\text{otherwise}.\end{cases}$ $$

Here, the *long-context influence* of $x_{i}$, $I(x_{i};\theta_{0})\geq 0$, selects key tokens to have a large log probability gain under long context (${\rm LPG}$ , Equation 2) and a large log probability value (${\rm LPV}$, Equation 3) based on an evaluator model with parameters $\theta_{0}$, with two threshold parameters $\alpha,\beta$. $\hat{I}(x_{i})=I(x_{i})/\sum_{j}I(x_{j})$ is the relative influence after normalization. The first criterion ensures that the generation of the token is enhanced by the additional information in the long-context. The second criterion excludes the fundamentally hard (misclassified) tokens that long context information does not help. Based on these criteria, all tokens are divided into two categories. Tokens that meet the criteria are selected as key tokens and are included in the perplexity calculation with equal weight, while those that do not meet the criteria are excluded from the calculation. Later in Section 4.1, we show that in contrast to standard PPL, LongPPL computed on a natural language corpus for multiple LLMs correlates well with their performance on long-context benchmarks, including LongEval (Li et al., ), LongBench (Bai et al., ), and RULER (Hsieh et al., ). We also consider other similar variants of the influence function (e.g., with soft reweighting) and find them to be generally effective (though often less accurate).

Remark on the Evaluator Model $\theta_{0}$. Notably, the evaluator $P_{\theta_{0}}$ used for computing the long-context influence can be different from the evaluated model $P_{\theta}$. In fact, for the evaluator, we need a powerful model to ensure that they give a relatively accurate estimate of the token’s long-context influence. This requires the evaluator itself to have a strong long-context understanding ability. In practice, we adopt the medium-sized Qwen2-72B-Instruct (Yang et al., ) to select key tokens for all evaluated models for a fair comparison.

### 3.2 Improving Long-context Capabilities with LongCE

Due to the massive computational cost of pre-training an LLM from scratch on long texts, current long-text LLMs are pretrained on short contexts and then fine-tuned on longer contexts. By default, the long-context fine-tuning process adopts the Cross Entropy (CE) loss as in pre-training, which adopts a uniform average of all tokens, akin to standard perplexity (Equation 1):

$$ ${\rm CE}(x;\theta)=-\frac{1}{n}\sum_{i=1}^{n}\log P_{\theta}(x_{i}|{\bm{x}}_{< i}).$ (5) $$

Nevertheless, this *de facto* paradigm has the same issues that we discussed for perplexity in Section 2. We show that most tokens in a sequence are not influenced by the long context, while only a few key tokens require long-context information; and in turn, the model’s long-context performance depends crucially on its prediction on these key tokens (as measured in LongPPL, Section 3.1).

Following the methodology of LongPPL (Equation 4), we propose the LongCE (Long-context Cross Entropy) loss that reweights every token $x_{i}$ w.r.t. its gain $I_{\rm soft}(x_{i};\theta)$ from long context:

$$ ${\rm LongCE}(x;\theta)=-\sum_{i=1}^{n}I_{\rm soft}(x_{i};\theta)\log P_{\theta }(x_{i}|{\bm{x}}_{<i}).$ (6) $$

For the ease of differentiable optimization using all tokens, we adopt a *soft* long-context influence function $I_{\rm soft}:{\mathbb{X}}\to[0,\gamma]$ based on the likelihood ratio between the long-context probability $P_{\theta}(x_{i}|{\bm{l}}_{i})$ and short-context probability $P_{\theta}(x_{i}|{\bm{s}}_{i})$ (defined in Section 2.2):

$$ $I_{\rm soft}(x_{i};\theta)=\min\left(\exp\left({\rm LPG}_{\theta}(x_{i})\right ),\ \gamma\right)=\min\left(\frac{P_{\theta}(x_{i}|{\color[rgb]{1,.5,0} \definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}{\bm{l}}_{i}})}{P_{\theta}(x_{ i}|{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}{\bm{s}}_ {i}})},\ \gamma\right).$ (7) $$

Here, $\gamma>0$ is a hyper-parameter that sets a threshold on the maximal influence to avoid numerical instability. The weights are detached during the training process. As a consequence of this reweighting term, too easy tokens (both short and long context give accurate prediction) and too hard tokens (neither short or long context predicts correctly) will have a weight around $1$, while those long-context-dependent tokens (high $P_{\theta}(x_{i}|{\bm{l}}_{i})$ and low $P_{\theta}(x_{i}|{\bm{s}}_{i})$) will be upweighted above $1$, proportionally to the context informativeness.

Remark. Note that different from the influence function of LongPPL (Equation 4) that adopts a strong LM as the influence evaluator to better select tokens, in LongCE, we use the same model to evaluate the influence for training efficiency. Therefore, LongCE training does not require a separate evaluator model, but uses the model itself for long-context evaluation. By this, *LongCE bootstraps the model’s long-context capabilities in an EM (expectation-maximization) way*: the language model $P_{\theta}$ first uses itself to estimate long-context influence of each token $I_{\rm soft}$ (Equation 7); and we then use this estimate to update the model parameters by optimizing the LongCE loss $\rm LongCE$ (Equation 6). This way, the model can better focus on the key tokens that matter the most for long-context capabilities to improve training efficiency. Additionally, we observe that computing key tokens introduces extra computational overhead. However, later experiments demonstrate that this overhead is acceptable given the clear performance improvements.

## 4 Experiments

In this section, we conduct real-world experiments to analyze the applicability of the proposed LongPPL and LongCE. For all the experiments, we use LongBench (Bai et al., ), LongEval (Li et al., ) and RULER (Hsieh et al., ) as the long-context benchmarks. LongBench mainly contains real-world tasks, including multi-document QA and summarization, while LongEval and RULER focus on synthetic tasks such as key-value retrieval and needle-in-a-haystack (NIAH). We report the average score on LongBench, the accuracy on the subtask “lines” of LongEval, and the score on RULER. For LongBench and RULER, we restrict the prompt length to 32k tokens. For LongEval, we use 1350 lines as the prompt, which is approximately 32k tokens.

Practical Implementation. In the implementation of LongPPL and LongCE, we need to compute the log probabilities for each token under both the long and the truncated short context. For the truncated short context of length $K$, one can use the sliding window technique in Transformers for computing token predictions in parallel to improve computational efficiency. For computing LongPPL when the evaluator model and the evaluated model have different tokenizers, we only keep key tokens that form the longest common substrings of the evaluated tokens. More details can be found in Appendix A.1.

### 4.1 LongPPL Metric

Experimental Setup. We calculate LongPPL on the GovReport dataset (Huang et al., ), which consists of long sequences from government reports. We sample 50 documents with the context length up to 32k tokens. We set the hyperparameters as $\alpha=2,\beta=-2,K=4096$. We use Qwen2-72B-Instruct (Yang et al., ), an open-source LLM with the context length of 128k tokens, as the discriminator model $\theta_{0}$ to select the key tokens.

Figure: (a) LongEval
Refer to caption: x10.png

Figure: (a) Mistral Large 2
Refer to caption: x12.png

LongPPL Correlates Well with Long-context Performance. In Figure 1(b) and 5, we demonstrate the correlation between LongPPL and long-context benchmarks on various long-context LLMs. We observe that LongPPL exhibits a very strong negative correlation with performance on long-context tasks across different models, with pearson correlation coefficients exceeding -0.8 for all three tasks. In contrast, perplexity hardly shows a correlation with the long-context tasks. This indicates that LongPPL is sufficiently capable of measuring a model’s long-context capabilities.

**Table 1: The Pearson correlation between different perplexity measures and benchmark scores, which is lower the better (because we expect a lower perplexity indicates higher benchmark scores).**
| Metrics | Influence $I$ | LongBench | LongEval | RULER |
| --- | --- | --- | --- | --- |
| PPL | $I(x)\equiv 1$ | -0.11 | 0.31 | 0.33 |
| LongPPL-soft | $I_{\rm soft}$ (Equation 7) | -0.43 | -0.21 | -0.17 |
| LongPPL-hard (default) | $I$ (Equation 4) | -0.97 | -0.85 | -0.83 |

LongPPL is compatible with various evaluator models. To demonstrate that LongPPL is not restricted to a specific evaluator model, we additionally tested it with two other powerful long-context LLMs, Mistral Large 2, Llama-3.1-8B. As shown in Table 6, the LongPPL calculated using different models as the evaluator model also exhibits high correlation. We observe that even though the Llama-3.1 model we use is only 8B in size, it still obtains LongPPL with a strong correlation. This suggests that LongPPL is highly compatible with powerful long-context models, even with smaller parameter sizes.

Hard standard for key tokens is better than soft re-weighting standard.
In Equation 4, we use an indicator function $I$ as the influence function. Instead, we have also tried to use the soft reweighting function $I_{\rm soft}$ used in LongCE (Equation 7) to calculate LongPPL. Its token matching strategy is detailed in Appendix A.1. In Table 1, we show that LongPPL with soft criteria has a weaker correlation with the long-context benchmarks compared to LongPPL, indicating that the soft reweighting influence function is suboptimal for LongPPL.

### 4.2 Fine-tune with LongCE loss

**Table 2: Long-context performance of the fine-tuned models using the standard CE loss and our proposed LongCE loss. We fine-tune Llama-2-7b on long texts using various fine-tuning strategies (EABF and PI) and different training data (PG-19 and Pile-arxiv). The models are then assessed on benchmarks with prompts of up to 32k tokens.**
|  | LongBench | LongEval | RULER |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Training steps | 50 | 100 | 200 | 50 | 100 | 200 | 50 | 100 | 200 |
| Setting A (PG-19 dataset with EABF) |  |  |  |  |  |  |  |  |  |
| CE | 24.5 | 26.6 | 26.9 | 16.0 | 24.0 | 24.0 | 34.5 | 38.6 | 42.7 |
| LongCE (Ours) | 26.0 | 27.2 | 28.2 | 24.0 | 46.0 | 46.0 | 43.1 | 48.3 | 49.7 |
| Gain | (+1.5) | (+0.6) | (+1.3) | (+8.0) | (+22.0) | (+22.0) | (+8.6) | (+9.7) | (+7.0) |
| Setting B (PG-19 dataset with PI) |  |  |  |  |  |  |  |  |  |
| CE | 24.3 | 25.3 | 25.4 | 20.0 | 28.0 | 26.0 | 22.1 | 31.8 | 35.7 |
| LongCE (Ours) | 24.4 | 25.0 | 25.8 | 38.0 | 44.0 | 42.0 | 27.3 | 34.4 | 36.4 |
| Gain | (+0.1) | (-0.3) | (+0.4) | (+18.0) | (+16.0) | (+16.0) | (+5.2) | (+2.6) | (+0.7) |
| Setting C (Pile-arxiv dataset with EABF) |  |  |  |  |  |  |  |  |  |
| CE | 15.0 | 23.1 | 23.8 | 8.0 | 18.0 | 14.0 | 40.9 | 53.3 | 51.9 |
| LongCE (Ours) | 17.6 | 24.0 | 25.0 | 10.0 | 18.0 | 16.0 | 49.7 | 54.8 | 58.6 |
| Gain | (+2.6) | (+0.9) | (+1.2) | (+2.0) | (+0.0) | (+2.0) | (+8.8) | (+1.5) | (+6.7) |

Experimental Setup.
We primarily use Llama-2-7B (Touvron et al., ) as the base model to perform long-context finetuning. We also conduct experiments on Mistral-7B-v0.1 (Jiang et al., ) and Llama-2-13B. We use PG-19 (Rae et al., ), a book dataset sourced from a library, and Pile-arxiv (Gao et al., ), a dataset consisting of Arxiv papers, as the training dataset. The training sequences are organized to be the context length with 32k tokens. For the calculation of LongCE, we set $\gamma=5$ in Equation 7 and use the same sliding window approach as described in Section 4.1 to improve training efficiency. The context length of $\bm{s}_{i}$ is set to be $K=4096$.
We fine-tune the base models with Entropy-aware Adjusted Base Frequency (EABF) (Zhang et al., ) and Position Interpolation (PI) (Chen et al., ). Specifically, EABF applies a scaling mechanism to the attention and uses a higher base frequency for RoPE, while PI linearly downscales the position indices of the input tokens. These methods can significantly accelerate the convergence speed of long-context fine-tuning and have been widely adopted in many LLMs (Yang et al., ; Dubey et al., ; Chen et al., ).
Detailed training setups are available in Appendix A.2.

LongCE Outperforms CE in Various Settings. As shown in Table 2, we present the long-context capabilities of models fine-tuned with LongCE loss and CE loss under different fine-tuning strategies and training datasets. We also test the effectiveness of LongCE using different base models in Table 3. We find that models fine-tuned with LongCE loss consistently outperform those fine-tuned with CE loss across nearly all settings. This suggests that the LongCE loss, with its re-weighting strategy based on long-context token importance, can be applied as a plug-and-play module which can effectively improve the model’s long-context performance. We also provide fine-grained results in Appendix B.3 to illustrate the specific task domains in which LongCE outperforms the baseline.

Training Efficiency. In addition to the performance improvement brought by the LongCE loss, we also pay attention to the changes in training efficiency. In LongCE, we need an extra forward pass to calculate the probability under short context $P_{\theta}(x_{i}|\bm{s}_{i})$, which introduces additional computation costs. By using a sliding window technique (detailed in Appendix A.1), the computational overhead of training the model with LongCE is controlled to about 80% that of training with CE loss. In most settings shown in Table 2, the model trained with LongCE outperforms the model trained with CE, even when the latter is trained for twice as many steps. Additionally, we visualize in Figure 7 how the long-context performance of models fine-tuned with LongCE and CE changes over the course of training time. Most of the time, fine-tuning with LongCE loss is also a more efficient method.

**Table 3: Long-context performance of different fine-tuned models. We fine-tune Mistral-7B-v0.1 and Llama-2-13B with EABF adjustment strategy on Pile-arxiv dataset.**
|  | LongBench | LongEval | RULER |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Training steps | 50 | 100 | 200 | 50 | 100 | 200 | 50 | 100 | 200 |
| Mistral-7B-v0.1 |  |  |  |  |  |  |  |  |  |
| CE | 29.6 | 28.9 | 28.4 | 26.0 | 14.0 | 12.0 | 45.0 | 44.5 | 42.9 |
| LongCE (Ours) | 30.8 | 30.9 | 31.1 | 36.0 | 30.0 | 26.0 | 45.1 | 44.0 | 43.5 |
| Gain | (+0.8) | (+2.0) | (+2.7) | (+10.0) | (+16.0) | (+14.0) | (+0.1) | (-0.5) | (+0.6) |
| Llama-2-13B |  |  |  |  |  |  |  |  |  |
| CE | 26.3 | 26.9 | 28.2 | 14.0 | 14.0 | 14.0 | 45.4 | 50.4 | 52.3 |
| LongCE (Ours) | 26.4 | 28.5 | 28.9 | 20.0 | 18.0 | 18.0 | 55.1 | 61.9 | 62.5 |
| Gain | (+0.1) | (+1.6) | (+0.7) | (+6.0) | (+4.0) | (+4.0) | (+9.7) | (+11.5) | (+10.2) |

Figure: (a) LongBench
Refer to caption: x14.png

## 5 Related Work

Long-context Modeling.
Due to practical demands, numerous recent works have emerged that aim to enable large models to handle long contexts through improvements in architecture or algorithms. One mainstream direction is the study of positional encodings with length extrapolation capabilities, including Alibi (Press et al., ), xPOS (Sun et al., ), Kerple (Chi et al., ), and various RoPE (Su et al., ) variants (Chen et al., ; Zhang et al., ; Chen et al., ; Xiong et al., ; Peng et al., ). Others pay more attention to architecture improvements, using sparse attention mechanisms to prevent models from attending to overly long sequences (Han et al., ; Xiao et al., ; Chen et al., ; Ding et al., ), or exploring the use of recurrent mechanisms to compress and store key information from long texts, thereby effectively increasing the context window (Zhang et al., ; Bulatov et al., ; Martins et al., ).

Long-context Evaluation.
Recent studies have introduced several benchmarks to evaluate the long-context performance in downstream tasks. A widely used type of benchmark is retrieval-based synthetic task, including needle-in-a-haystack (Kamradt, ), passkey-retrieval (Mohtashami & Jaggi, ) and LongEval (Li et al., ). Some evaluation suites have also been gradually introduced, such as LongBench (Bai et al., ), RULER (Hsieh et al., ), ZeroSCROLLS (Shaham et al., ), including document question answering, summarization, few-shot learning, code completion, and other synthetic tasks, thereby offering a more thorough evaluation of a model’s long-context abilities. To further enhance the context length of the evaluation data, InfiniteBench (Zhang et al., ) has introduced evaluation data exceeding 100K tokens. In this work, we focus on understanding and addressing the limitations of perplexity at long-context training and evaluation.

Re-weighting methods in language model training. Re-weighting methods for language model training have been extensively studied, with a focus on enhancing model performance (Lin et al., ), improving training efficiency (Clark et al., ), and addressing token imbalance (Luo et al., ; Hu et al., ; Gu et al., ; Wang et al., ). Many works have also explored re-weighting through data selection techniques, addressing a wide range of challenges such as data quality (Lin et al., ; Li et al., ), data diversity (Liu et al., ), and distribution matching (Li et al., ; Ni et al., ). However, in long-context understanding, little attention was paid to the roles of individual tokens, which we find is pivotal for understanding the failure of perplexity measures. From a different perspective, a concurrent work LongRecipe (Hu et al., ) also proposes reweighting tokens during training by comparing token probabilities between a base model and a trained model. Instead, we identify key tokens by computing the differences under short and long contexts, inspired by causal intervention. As a result, we only require one model during training, which saves memory footprint especially for large models.

## 6 Conclusion

In this paper, we offer a comprehensive explanation for why perplexity fails to reflect the long-context capabilities of LLMs. We find that as perplexity treats all tokens equally, it lacks sufficient attention on the key tokens that are crucial for long-context understanding. To address this, we propose a novel metric, LongPPL, which focuses on the key tokens in natural texts through a long-short context constrastive method. We empirically demonstrate the strong correlation with the long-context capabilities of LLMs as indicated by LongPPL and the performance on long-context benchmarks. In addition, we utilize the concept of LongPPL to propose the LongCE loss, which reweights the CE loss used in the long-context fine-tuning. By up-weighting the key tokens, LongCE leads to consistent improvements across multiple long-context benchmarks with up to 22% gains in LongEval accuracy. We hope our analysis and approaches can provide insights for a better understanding into the essence of long-context generation.

## Appendix A Detailed settings in experiments

### A.1 Implementation Details of LongPPL

Sliding window algorithm to improve efficiency. Since the calculation of LongPPL requires computing the LPG for each token $x_{i},i\in[n]$, it necessitates calculating the probability under short context $P_{\theta}(x_{i}|\bm{s}_{i})$ for $n$ times. Since it cannot be computed in single forward pass, this leads to massive computational overhead. To overcome this problem, we use a sliding window algorithm to improve efficiency. Specifically, we introduce a step size $d$, which is smaller than the truncation length $l$ (we set it to $d=1024$). When calculating the short-context probabilities of $x_{i}$ to $x_{i+d-1}$, we set the starting token of the context uniformly as $x_{i-l}$. Formally speaking, we have

$$ $s_{kd+i^{\prime}}=(x_{(k-1)d},...x_{i^{\prime}-1}),$ (8) $$

where $k\in\mathbb{N},0\leq i^{\prime}<d$. This approach allows for the calculation of the short-context probabilities of $d$ tokens in a single forward pass.

Token matching method.
Since the used tokenizers between evaluator model $P_{\theta_{0}}$ and evaluated models $P_{\theta}$ could be different, we attempt to align the key tokens between different models. Formally, we define the encoding and decoding functions of tokenizers used in language models as $encode_{P}$ and $decode_{P}$. Let $\bm{t}=(t_{1},...,t_{N})$ be the original text contains of $N$ characters, and $\bm{x}=(x_{1},...,x_{n})=encode_{P_{\theta_{0}}}(\bm{t})$, $\bm{x}^{\prime}=(x^{\prime}_{1},...,x^{\prime}_{n^{\prime}})=encode_{P_{\theta
}}(\bm{t})$ be the token sequence encoded by $P_{\theta_{0}}$ and $P_{\theta}$, respectively. Let $\mathcal{X}=\{x_{k_{i}}\}_{i=1}^{n_{k}}$ be the set of key tokens calculated by the evaluator model $P_{\theta_{0}}$. We map these tokens to the text space as $\mathcal{T}=decode_{P_{\theta_{0}}}(\mathcal{X})$. Then, the key token set $\mathcal{X}^{\prime}$ of the evaluated model is the maximal subset of $\bm{x}^{\prime}$ which satisfies

$$ $decode_{P_{\theta}}(\mathcal{X}^{\prime})\subseteq\mathcal{T}.$ (9) $$

Besides, in Table 1, we also implement the LongPPL with the soft influence function $I_{\rm soft}$ (Eq. (7)). In this approach, we implement an reweighting algorithm to transfer the weight between different tokenizers. Specifically, denote $\bm{w}=(w_{1},...,w_{n})$ as the LPG weight on $\bm{x}$ calculated by $P_{\theta_{0}}$. The weight of $x^{\prime}_{i}$ is defined as

$$ $w^{\prime}_{i}=\sum_{t_{j}\in decode_{P_{\theta}}(x^{\prime}_{i})}w(t_{j})/| decode_{P_{\theta}}(x^{\prime}_{i})|,$ (10) $$

where $w(t_{j})$ is the weight of the token that $t_{j}$ belongs to. This assigns the weight of $\bm{x^{\prime}}$ with the string-level average of the weight in $\bm{x}$.

### A.2 Implementation Details of LongPE

Fine-tuning strategies. For EABF (Zhang et al., ), we adopt the identical settings in the original paper, with a RoPE base of 500k. For PI (Chen et al., ), we set the scaling factor to 8 since we want to extend the context window from 4k to 32k.

Training details. We use a learning rate of $2\times 10^{-5}$ for Llama and $1\times 10^{-6}$ for Mistral, with no weight decay and a linear warmup of 20 steps along with AdamW (Loshchilov, ) with $\beta_{1}=0.9$ and $\beta_{2}=0.95$. We apply a global batch of 64 on PG-19 and 8 on Pile-arxiv. We disable the sliding window mechanism when fine-tuning Mistral-7B-v0.1. We perform the experiments with 8 Nvidia A100 80GB GPUs using Pytorch (Paszke et al., ).

## Appendix B Supplementary experiment results

### B.1 Detailed results of LongPPL

We present the LongPPL calculated by different models in Table 4, and provide further visualization results for Mistral Large 2 and Llama-3.1-8B in Figure 8 and 9.

**Table 4: The perplexity-based metrics of various LLMs.**
| Metric | LongPPL | PPL |  |  |
| --- | --- | --- | --- | --- |
| Evaluator model | Qwen2-72B-Instruct | Mistral Large 2 | Llama-3.1-8B | - |
| Mixtral-8x7B-32k | 2.07 | 2.47 | 1.76 | 3.72 |
| FILM-7B-32k | 2.51 | 2.95 | 2.04 | 4.54 |
| Mistral-7B-32k | 2.68 | 3.35 | 2.19 | 4.33 |
| Qwen1.5-14B-128k | 2.90 | 2.80 | 2.39 | 5.30 |
| Qwen2-7B-128k | 2.96 | 2.73 | 2.36 | 5.03 |
| Phi-3-small-128k | 3.01 | 2.87 | 2.46 | 5.48 |
| CLEX-7B-64k | 3.54 | 4.27 | 2.89 | 3.93 |
| Yi-6B-200k | 3.66 | 3.76 | 2.93 | 5.17 |
| Yarn-7B-128k | 3.67 | 4.82 | 3.09 | 4.20 |

Figure: (a) LongEval.
Refer to caption: x17.png

Figure: (a) LongEval
Refer to caption: x19.png

### B.2 Ablation study

LPV. In the calculation of LongPPL, we employ LPV as an assistant for our core criterion, LPG, in selecting key tokens. In Figure 10, we demonstrate the LongPPL calculated without the LPV criterion. This version of LongPPL hardly has correlation with the long-context benchmark, showing that LPV is an indispensable part for LongPPL.

Figure: Figure 10: LongPPL without LPV.
Refer to caption: x21.png

Evaluator model. In the main text, we use a evaluator model $\theta_{0}$ to identify the key tokens. To validate the necessity of this approach, we calculate LongPPL using the model itself as the evaluator, as shown in Table 5. The results indicate that most models achieve similar LongPPL scores, suggesting that this self-evaluated version of LongPPL does not reflect the models’ long-context capabilities.

**Table 5: LongPPL using the evaluated model itself to calculate the key tokens.**
|  | Mixtral | FILM | Mistral | Qwen1.5 | Qwen2 | Phi-3 | CLEX | Yi | Yarn |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LongPPL | 1.67 | 1.64 | 1.68 | 1.67 | 1.65 | 1.65 | 1.68 | 1.75 | 1.92 |

### B.3 Fine-grained results of LongCE

In this section, we provide more detailed LongBench scores of the models from the experiments in section 4.2, as shown in Table 6. We observe that the models finetuned by LongCE outperforms the model finetuned with CE primarily in single/multi-document QA, summarization and synthetic tasks (including retrieval and counting tasks). This also explains why LongCE can significantly outperform CE on LongEval and RULER, as their synthetic tasks primarily assess models’ retrieval, summarization, and QA capabilities in long-context scenarios.

**Table 6: Detailed scores of LongBench in Table 2.**
| Task Domains | Single-Document<br>QA | Multi-Document<br>QA | Summarization | Few-shot<br>Learning | Code<br>Completion | Synthetic<br>Tasks | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Setting A (PG-19 dataset with EABF) |  |  |  |  |  |  |  |
| CE (50 steps) | 4.4 | 1.1 | 15.5 | 66.7 | 59.7 | 0.0 | 24.5 |
| CE (100 steps) | 5.9 | 2.0 | 21.9 | 67.5 | 61.8 | 0.4 | 26.6 |
| CE (200 steps) | 6.9 | 2.3 | 22.8 | 66.8 | 61.9 | 0.4 | 26.9 |
| LongCE (50 steps) | 7.6 | 2.1 | 22.0 | 66.1 | 57.9 | 0.5 | 26.0 |
| LongCE (100 steps) | 7.7 | 3.3 | 22.5 | 65.7 | 61.6 | 2.3 | 27.2 |
| LongCE (200 steps) | 9.3 | 4.8 | 23.9 | 66.0 | 61.9 | 3.2 | 28.2 |
| Setting B (PG-19 dataset with PI) |  |  |  |  |  |  |  |
| CE (50 steps) | 3.1 | 3.2 | 12.9 | 65.3 | 59.8 | 1.6 | 24.3 |
| CE (100 steps) | 4.1 | 3.5 | 17.5 | 65.2 | 59.9 | 1.8 | 25.3 |
| CE (200 steps) | 5.6 | 4.0 | 15.4 | 66.0 | 60.3 | 1.0 | 25.4 |
| LongCE (50 steps) | 4.5 | 2.2 | 15.6 | 63.1 | 58.4 | 2.7 | 24.4 |
| LongCE (100 steps) | 4.6 | 1.7 | 17.7 | 64.1 | 59.0 | 2.8 | 25.0 |
| LongCE (200 steps) | 6.0 | 4.3 | 19.0 | 63.6 | 59.2 | 2.7 | 25.8 |
| Setting C (Pile-arxiv dataset with EABF) |  |  |  |  |  |  |  |
| CE (50 steps) | 1.7 | 0.0 | 0.0 | 50.2 | 38.2 | 0.0 | 15.0 |
| CE (100 steps) | 4.2 | 5.4 | 4.9 | 65.0 | 58.9 | 0.0 | 23.1 |
| CE (200 steps) | 5.1 | 7.1 | 7.6 | 64.3 | 58.7 | 0.0 | 23.8 |
| LongCE (50 steps) | 3.5 | 0.0 | 2.6 | 52.9 | 46.7 | 0.0 | 17.6 |
| LongCE (100 steps) | 4.2 | 5.3 | 10.0 | 64.3 | 59.1 | 1.0 | 24.0 |
| LongCE (200 steps) | 3.7 | 6.1 | 14.3 | 64.7 | 59.8 | 1.3 | 25.0 |

## Appendix C Models

The models used in this paper are shown in Table 7.

**Table 7: Information of the models used in this paper.**
| Model | Size | Context Length | Huggingface |
| --- | --- | --- | --- |
| Llama-2-7B (Touvron et al., ) | 7B | 4K | meta-llama/Llama-2-7b-hf |
| Llama-2-13B (Touvron et al., ) | 13B | 4K | meta-llama/Llama-2-13b-hf |
| Llama-3.1-8B (Dubey et al., ) | 8B | 128K | meta-llama/Llama-3.1-8B |
| Mixtral (Jiang et al., ) | 8x7B | 32K | mistralai/Mixtral-8x7B-Instruct-v0.1 |
| Mistral-v0.1 (Jiang et al., ) | 7B | 8K | mistralai/Mistral-7B-v0.1 |
| Mistral (Jiang et al., ) | 7B | 32K | mistralai/Mistral-7B-Instruct-v0.2 |
| Mistral Large 2 (Jiang et al., ) | 123B | 128K | mistralai/Mistral-Large-Instruct-2407 |
| Qwen1.5 (Bai et al., ) | 14B | 128K | Qwen/Qwen1.5-14B |
| Qwen2-7B (Yang et al., ) | 7B | 128K | Qwen/Qwen2-7B |
| Qwen2-72B (Yang et al., ) | 72B | 128K | Qwen/Qwen2-72B-Instruct |
| FILM (An et al., ) | 7B | 32K | In2Training/FILM-7B |
| Phi-3 (Abdin et al., ) | 7B | 128K | microsoft/Phi-3-small-128k-instruct |
| CLEX (Chen et al., ) | 7B | 64k | DAMO-NLP-SG/CLEX-LLaMA-2-7B-64K |
| Yi (Young et al., ) | 6B | 200K | 01-ai/Yi-6B-200K |
| Yarn (Peng et al., ) | 7B | 128K | NousResearch/Yarn-Mistral-7b-128k |

## Appendix D Demonstration for the selected key tokens

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. I wonder how would fine tuned model perform on normal context length benchmarks, will it remain roughly at the same level or regress a bit?
2. How long is considered as "long-context". In experiments, 32k is used. I wonder how much this method would affect on other context lengths.
3. can we use equation (5) directly for LongCE instead of (8). so only differentiate on key tokens, would that be doable?
soundness: 3
strengths: The idea is simple and seems working well.
summary: This paper modified standard PPL and CE (cross entropy) loss, by weighting more to key tokens in long text. The key tokens are identified through a long-short context constrastive method. The experiments demonstrate the LongPPL correlates long-context capabilities of LLM, thus a good metric. Similarly, applying LongCE loss in long-context fine tuning shows improvement on long-context benchmarks.
weaknesses: n/a

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is the main reason perplexity fails due to its inability to correctly identify answer/key tokens?
2. Could you clarify what "task-aware" means in this context? Does it refer to distinctions between short-context and long-context tasks, or could it apply to other task definitions as well?
3. You mention that "even though the Llama-3.1 model we use is only 8B in size, it still obtains LongPPL with a strong correlation, suggesting high compatibility with powerful long-context models, even with smaller parameter sizes." Are you implying that LongPPL would show higher correlation with larger models? If so, could you explain why?
soundness: 3
strengths: This paper conducts a fine-grained analysis to investigate why perplexity (PPL) fails to capture long-context capabilities in LLMs. Few previous studies focus on re-weighting tokens to improve long-context performance; this work addresses this directly by re-weighting tokens based on their dependence on long-context information, providing a more fundamental and targeted solution.
summary: The paper critiques the use of perplexity (PPL) as a standard metric for evaluating large language models (LLMs) on long-context tasks, arguing that PPL’s averaging across all tokens fails to capture the importance of "key tokens" critical for understanding long-context dependencies. To address this, the authors introduce Log Probability Gain (LPG) and Log Probability Values (LPV) to identify these key tokens, leading to LongPPL—a refined metric that emphasizes these crucial tokens, thus providing a better measure of long-context performance. Additionally, they propose LongCE, a re-weighting strategy applied during fine-tuning that bolsters the model's ability to handle long-context tasks through an expectation-maximization (EM) approach. Although LongCE adds computational overhead, the authors show that it consistently outperforms traditional cross-entropy (CE) across various settings.
weaknesses: Some settings in the figures lack clear explanations, which affects the reliability of the figures and the conclusions drawn from them. For example, in Figure 2, the specific prompt lengths for each data point are not provided, making it difficult to interpret the statement, "Each point represents the results obtained from testing at a specific prompt length."
Although the correlation arguments seem promising and qualitatively sound, they would benefit from added statistical rigor, such as p-values from hypothesis testing to confirm the significance.
The naming of Log Probability Gain (LPG) and Log Probability Values (LPV) is not intuitive, which may hinder understanding and adoption of these concepts.
Additionally, the experiments are conducted with a sequence length of 32K, a length considered relatively manageable by recent research. Testing on longer sequences could introduce challenges that may not align with current trends, potentially impacting the generalizability of the results
Lastly, the criteria for selecting experimental models are not clearly defined, which introduces some ambiguity regarding the rationale behind the choice of models used in testing.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: - How does the performance of LongPPL and LongCE change when using a weaker model for key token identification? Have you tried using the same model for both evaluation and key token identification?
- The paper discusses the overhead of using LongCE in LLM fine-tuning compared to the original Cross Entropy. How does the evaluation time of using LongPPL compare to traditional perplexity (PPL) metrics?
soundness: 3
strengths: - The introduction of LongPPL and LongCE represents an innovation in the evaluation and fine-tuning of LLMs for long-context tasks. These metrics address the limitations of traditional perplexity and Cross Entropy loss, providing more accurate and task-relevant assessments of model performance.
- The concepts presented by the authors are easy to understand, including the motivation and method. The figures greatly help in understanding the concepts and the experiments.
summary: The authors state that perplexity metrics (PPL) are insufficient for evaluating the long-context capabilities of large language models (LLMs) because PPL considers both answer and non-answer tokens, resulting in perplexity scores that do not correlate with LongBench scores. To address this, the authors propose two new metrics, LongPPL and LongCE, which extract key tokens from the model output with the help of a relatively strong model as the evaluator. LongPPL is used as an evaluation metric for LLMs in long-context text, while LongCE serves as a loss function when fine-tuning LLMs on long-context tasks.
weaknesses: - LongPPL metric relies on a relatively strong model (medium-sized Qwen2-72B-Instruct) as an evaluator to identify key tokens. This dependence may limit the applicability of the metric in scenarios where such a strong model is not available or practical to use.
- The additional computational cost associated with identifying key tokens may still be significant. This could be a barrier to adoption in resource-constrained environments.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Why does finetuning with LongCE demonstrate such different results for mistral-7b and llama2 on RULER, with one hardly improving (<1) and one significantly improving (~10)?
- Why does figure 6 only contain correlation between longPPL and longbench score? What is the correlation between standard PPL vs longbench score?
soundness: 2
strengths: - This paper proposes a new metric LongPPL which better correlates with long-context downstream performance.
- This paper proposes a new training objective LongCE tailored for improving long-context performance.
summary: This paper proposes a new metric LongPPL and a new training objective LongCE. The LongPPL is calculated over selected tokens that satisfy the following conditions when evaluated with an evaluator model: (a) the tokens have higher probability conditioned on long context than short context (b) the tokens are predicted with high probability. Results show that the proposed metric better correlates with long-context downstream tasks than standard perplexity. The proposed LongCE loss adapts the standard focal loss by reweighting the tokens based on their performance given long and short context. Results demonstrate that models finetuned with LongCE achieve better long-context downstream task performance.
weaknesses: - The proposed LongPPL is limited by the evaluator model. Concretely, the validity of the evaluation is strictly bounded by the maximum effective length that an evaluator model can reliably handle. Additionally, the two thresholds, which depend on both the evaluation data and the evaluator model, can further complicate the evaluation process. These limitations weaken the universality of the proposed metric.
- The key token selection of LongPPL is motivated by observing large gains in log probability of the ‘answer’ tokens in LongEval, a retrieval-based synthetic test. It raises the question of whether the gains primarily come from the improved prediction of less frequent n-grams that re-appear later in the context, as studied by [1][2]. If so, the motivation of leveraging an evaluator model is obscure since one can simply evaluate those re-occurred n-gram tokens, which are much more convenient to find and overall a more universal setup than LongPPL. 
- The LongCE loss should be compared with the standard focal loss [3] baseline to understand if it is really necessary to add the contrasting term of long and short context perplexity. The standard focal loss does not require extra forward pass for computing short context log probability, thus can potentially improve training efficiency. 
- The LongCE loss seems to improve mostly the retrieval-based test (LongEval), however, demonstrate little gains on RULER, which has a few non-retrieval tasks, and almost zero improvement on LongBench, which contains quite a few non-retrieval tasks (Figure 7). It is unclear to what extent the proposed method benefits non-retrieval tasks.


[1] Zoology: Measuring and Improving Recall in Efficient Language Models

[2] Do Long-Range Language Models Actually Use Long-Range Context?

[3] Focal Loss for Dense Object Detection

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: -computed longPPL and longCE (as opposed to PPL) must be more costly; how much more costly (especially a you need to compute the LPG score for each token at training time) ?

-do you fine tune with longCE only or with a combination of CE and longCE ?

-what about using the needle-in-the-haystack test for your evaluation of long context models ?
soundness: 4
strengths: -Paper very clearly presented and very progressive, starting from identifying a problem, explain why it occurs and propose an efficient solution for it

- convincing explanation on why PPL is not a good indicator of quality for long context LLMs (with illustrations on a long context benchmark)

-Fine tuning with longCE loss leads to improvements on several long context benchmarks and for several LLMs ; very convincing results
summary: This paper first tries to understand why perplexity metric fail to reflect long-context ability of LLMs. For instance it has been shown that perplexity of LLMs shows almost no correlation with their long-context performance measured on a dataset such as Longbench.

Authors show the reason is because perplexity is computed over all tokens of the answer to a query, wether this tokens are ‘key’ or not to answer to the query. Consequently, perplexity computed equally over all tokens do not represent long-context performance. Therefore,  computing ppl by averaging only on key tokens (instead of all tokens)  avoid this dilution and correlates better with long context benchmarks scores such as the ones measured on Longbench.

A longPPL measure that takes into account this observation is proposed by the authors, where they compute perplexity by averaging solely on the selected key tokens.
But how do the authors extract the ‘key’ tokens from the natural text in the answer to a query ? 
 This is done by introducing a simple and elegant metric which consists in computing the difference - for each token x - of the probability p(x/long_context) and p(x/shortened_context). This metric called LPG score, measures the improvement of prediction accuracy induced only by the long-context => a high value indicates that long context plays an important role in the prediction of x, so x is a key token and should be taken into account to compute LongPPL (and it should not be taken into account - for compute the PPL - otherwise). 

Building on this finding, they develop an efficient long-context training strategy by prioritizing key tokens. Specifically, they introduce the LongCE (Long-context Cross-Entropy) loss, which assigns higher weights to key tokens, identified dynamically by the model itself.
Experiments across various LLMs demonstrate that fine tuning a model using the LongCE  loss consistently outperforms traditional CE loss, across multiple LLMs and multiple long context benchmarks.
weaknesses: -I don’t fully agree this the claim « the evaluation of long-context capabilities still heavily relies on perplexity (PPL) as the de facto metric » as we see now most papers relying on long-context benchmarks or use the needle-in-the-haystack test to evaluate long-context LLMs

-The reference to ppl is not the good one (Bengio did not invent perplexity!), you should credit people who actually were the first to introduce this metric. Perplexity was originally introduced in 1977 in the context of speech recognition by Frederick Jelinek, Robert Leroy Mercer, Lalit R. Bahl, and James K. Baker. See https://en.wikipedia.org/wiki/Perplexity

-I’d like to learn more about the computational implications of calculating LongPPL and LongCE, especially since the LPG score needs to be computed for each token (see my question below). I'm particularly curious about the costs associated with LongCE, as described in Section 3.2. While the Implementation details of LongPPL in the Appendix were helpful, I still want to understand more about how computationally expensive this process is.

Meta-review: This paper addresses the limitations of perplexity (PPL) as a metric for evaluating the long-context capabilities of large language models (LLMs). Standard PPL averages over all tokens, failing to highlight the role of "key tokens" essential for understanding long-context dependencies. The authors further propose two innovations to address the limitations of PPL. 1) LongPPL: A refined evaluation metric that focuses on key tokens, which are identified using a long-short context contrastive approach. The approach computes the difference in token probabilities between long and short contexts (Log Probability Gain, LPG). Tokens with high LPG scores are considered key. LongPPL demonstrates stronger correlation with long-context benchmarks (e.g., LongBench) compared to standard PPL. 2) LongCE: A re-weighted cross-entropy loss function for fine-tuning LLMs, to give key tokens higher importance during training, dynamically identified based on their significance in long-context scenarios. Inspired by focal loss, LongCE adapts weights to improve long-context understanding. Experiments show that models fine-tuned with LongCE outperform those trained with traditional cross-entropy loss on long-context benchmarks. The paper proposes a well-motivated and innovative approach to tackle the shortcomings of standard metrics and training strategies for long-context tasks in LLMs. Its clear methodology, strong empirical results, and practical relevance make it a significant contribution to the field. However, further exploration of computational efficiency and broader validation would enhance its impact.

Strength of this paper

- Novel Contributions: Introduces LongPPL, a new evaluation metric that correlates more effectively with long-context downstream performance than standard perplexity (PPL). Proposes LongCE, a novel training objective tailored to improve long-context performance by dynamically re-weighting tokens based on their dependence on long-context information. The identification of key tokens and their use in both metrics and training is a simple yet effective approach.
- Addressing Fundamental Limitations: Conducts a detailed analysis of why standard PPL fails to capture long-context capabilities in LLMs, providing a well-motivated and targeted solution. Offers a convincing explanation, supported by illustrations, that highlights the inadequacy of traditional metrics for long-context tasks.
- The paper is well-structured and progressive, starting from problem identification, explaining its causes, and presenting efficient solutions.
- Extensive experiments validate the efficacy of proposed metrics and training objectives, showing consistent improvements across diverse benchmarks.
- Impactful Findings: the work tackles an under-explored area of long-context performance in LLMs, offering a fundamental rethinking of token weighting and evaluation metrics, and provides tools and strategies that can be broadly applied to improve long-context tasks.


Weakness of this paper:

Several reviewers raised few concerns/limitations of this paper. By addressing these limitations, the paper could strengthen its experiment and expand impact.

- Computational Overhead: LongCE introduces additional computation by dynamically identifying and prioritizing key tokens, and requires additional forward passes for short-context log probability. Both  could be barriers in resource-constrained settings. Some analyses on computational costs would be helpful. Besides, simpler methods to avoid using evaluator model for key token selection , such as evaluating re-occurring n-grams, might offer similar benefits without the complexity.
- Dependency on Evaluator Models: LongPPL relies on an external evaluator to identify key tokens, which may add complexity to implementation. The validity of LongPPL is also bounded by the evaluator model's maximum effective length and thresholds, complicating its universal application.
- Experimental Limitations: The experiments use a sequence length of 32K, which is relatively manageable with current technologies. Testing on longer sequences could reveal additional challenges and generalizability issues. LongCE demonstrates limited improvements on non-retrieval benchmarks like RULER and LongBench, raising questions about its broader applicability beyond retrieval-based tasks. Besides, The LongCE loss is not compared against standard focal loss, which could provide insights into whether the proposed contrasting term of long- and short-context perplexity is necessary.
- Lastly, the claim that perplexity (PPL) remains the de facto metric for evaluating long-context LLMs is debatable, as many recent works rely on benchmarks or specific tests like the needle-in-the-haystack test. Impact claimed by the authors might be over-estimated.

Decision: Accept (Poster)