What is Wrong with Perplexity for Long-context Language Modeling?
Authors: Lizhe Fang
Yifei Wang
Zhaoyang Liu
Chenheng Zhang
Stefanie Jegelka
Jinyang Gao
Bolin Ding
Yisen Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Large language models
Long context
Primary area: foundation or frontier models, including LLMs

Abstract: Handling long-context inputs is crucial for large language models (LLMs) in tasks such as extended conversations, document summarization, and many-shot in-context learning. While recent approaches have extended the context windows of LLMs and employed perplexity (PPL) as a standard evaluation metric, PPL has proven unreliable for assessing long-context capabilities. The underlying cause of this limitation has remained unclear. In this work, we provide a comprehensive explanation for this issue. We find that PPL overlooks key tokens, which are essential for long-context understanding, by averaging across all tokens and thereby obscuring the true performance of models in long-context scenarios. To address this, we propose \textbf{LongPPL}, a novel metric that focuses on key tokens by employing a long-short context contrastive method to identify them. Our experiments demonstrate that LongPPL strongly correlates with performance on various long-context benchmarks (e.g., Pearson correlation of -0.96), significantly outperforming traditional PPL in predictive accuracy. Additionally, we introduce \textbf{LongCE} (Long-context Cross-Entropy) loss, a re-weighting strategy for fine-tuning that prioritizes key tokens, leading to consistent improvements across diverse benchmarks. In summary, these contributions offer deeper insights into the limitations of PPL and present effective solutions for accurately evaluating and enhancing the long-context capabilities of LLMs. Code is available at https://github.com/PKU-ML/LongPPL.

Published as a conference paper at ICLR 2025
WHAT IS WRONG WITH PERPLEXITY FOR LONG- CONTEXT LANGUAGE MODELING?
Lizhe Fang Yifei Wang2* Zhaoyang Liu
Chenheng Zhang'
Stefanie Jegelka 4,5
Jinyang Gao Bolin Ding
Yisen Wang" ,61
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
MIT CSAIL
3
Alibaba Group
TUM CIT, MCML, MDSI
MIT EECS, CSAIL
6 Institute for Artificial Intelligence, Peking University
ABSTRACT
Handling long-context inputs is crucial for large language models (LLMs) tasks such as extended conversations, document summarization, and many-shot in-context learning. While recent approaches have extended the context windows of L.LMs and employed perplexity (PPL.) as a standard evaluation metric, PPL has proven unreliable for assessing long-context capabilities. The underlying cause of this limitation has remained unclear. In this work, we provide a comprchensive explanation for this issue. We find that PPL. overlooks key tokens, which are essen- tial for long-context understanding. by averaging across all tokens and thereby ob- scuring the true performance of models in long-conten, scenarios. To address this, we propose LongPPL, novel metric that focuses key tokens by employing a long-short context contrastive method to identify them. Our experiments demon- strate that LongPPL strongly correlates with performance various long-contex, benchmarks (e.g., Pearson correlation of -0.96), significantly outperforming tra- ditional PPL in predictive accuracy. Additionally, we introduce LongCE (Long- context Cross Cross-Entropy loss, a weighting strategy for fine-tuning that prioritizes key tokens, leading to consistent improvements across diverse benchmarks. These contributions offer deeper insights into the limitations of PPL and present effective solutions for accurately evaluating and enhancing the long-context capabilities LLMs. Code is available at https://gtthu..om// www.//o////////o/a....
INTRODUCTION
The ability process long-context inputs is critical for large language models (LLMs) in many real- world tasks, such as long conversations (Maharana et al., 2024), document summarization (Chang et al.. 2024), and many-shot in-context learning (Agarwal et al., 2024; Li et al., 2024; Wei et al., 2023). Despite many techniques for extending the context length (Han et al., 2023; Chen et al., 2023; Zhu et al., 2024; Xiong et al., 2024; Chen et al., 2024a), the evaluation of long-context ca- pabilities still widely uses perplexity (PPL.) as the de facto metric. Many have claimed extend context windows to 32k, 128k, or even millions of tokens, based on attaining a low perplexity score under long context. However, recent studies have challenged this common practice by revealing a huge discrepancy between perplexity and actual performance on long-contex. tasks (Hu et al., 2024a; Hsieh et al., 2024). As shown in Figure (bb) (top). the perplexity of LLMs shows almost no correlation to their long-context performance measured by Longbench scores (Bai et al., 2023b). This raises the question:
Why does perplexity fail to reflect the long-context abilities of LLMs?
"Equal Contribution.
1 Corresponding Author: Yisen Wang (yisen, www.:aaw.g@@@aaeee

---

Published as a conference paper at ICLR 2025
Long contest
5.5
Sarah has dog named Buddy.
5.0-
[...]
H MMM
Sarah feels happy to play with Buddy.
Pyla, llng contictt
CLDV-70
Truncato
Long-Shart Defierence (Eq
0.89
0.00
2.00
corr=-0.6
Non-key token token
LongPPL
3.5
Longrat
0.1
0.6
0.3
Poort contica))
3.0
Sarah feels happy to with Buddy.
Owe331
Short context
2.0
30
35
40
45
50
LongBench score
(a) Illustration of how LongPPL. calculated.
(b) LongBench vs. PPL/ LongPPL. (Ours)
Figure I: (a) A constructed example to illustrate how LongPPL calculated. We truncate the long context and calculate the generation probability difference (long-short difference, LSD, Eq. (2)) for each token based on the long and short contexts. A high LSD score indicates that the token's genera- tion is significantly enhanced by the long context, making it a key token in the long text. LongPPL is then obtained by calculating perplexity Bench (Bai et al., 2023b)) vs. perplexity measures (PPL and our LongPPL) computed on GovReport (Huang et al., 2021), natural corpus. While PPL shows no correlation w.e.. Longbench score, LongPPL achieves -0.96 Pearson correlation coefficient.
these key tokens. (b) Long-context performance (Long-
To understand this phenomenon, we conduct a fine-grained analysis of the roles of different tokens at long-contex tasks. Notably, we find perplexity computed only on the answer tokens to the long- context tasks strongly correlates with LongEval accuracy, whereas perplexity on non-answer tokens shows little to no correlation. Since most tokens are non-answer tokens, standard perplexity averag- average
ing over all token equally fails to represent the long-context abilities. This motivates over the key tokens that reflect model's long-context abilities. key obstacle is that natural texts have no ground-truth -tuth reference of key tokens, making it hardly applicable to general cases.
To tackle this challenge, we propose principled method to measure the influence of long context on each token by performing a causal intervention on its context length. We find that tokens with significantly better predictions under long context strongly tied to long-context information, though they make up only a small portion of general text. Empirically, our proposed method can accurately identify the answer tokens in LongEval with up to 98.2% accuracy.
Built upon the accurate selection of key tokens, we propose LongPPL (Long-context Perplexity), where we compute perplexity by only averaging solely on the selected key tokens (Figure 1(a)). Extensive experiments across diverse suite of L.MS and long-context benchmarks show that LongPPI. computed on natural language corpus exhibits a consistently strong correlation with their benchmark scores computed over various long-context tasks, e.g., -0.96 correlation in Figure 1(b) (bottom). Thus, LongPPL offers a natural way to evaluate LLMs' long-context capabilities in an unsupervised fashion.
Following the design of LongPPL, we further develop an efficient long-context training strategy by emphasizing key tokens. Specifically, we propose the LongCE (Long-context Cross-Entropy) loss that upweights the key tokens, which can be estimated by the model itself. In this way, LongCE can bootstrap its long-contex abilities by alternating between estimating key tokens and optimizing key tokens. Experimental results across multiple LLMs show that LongCE consistently improves over the conventional CE loss, with a maximum accuracy gain 22% on LvngEval.
Our contributions are summarized as follows:
We conduct a fine- fine-grained analysis on the failure of perplexity measuring long-context abilities. Specifically, we reveal the critical roles of key tokens in long-context tasks and propose principled metrics identify key tokens with high accuracy.
2

---

Published as a conference paper at ICLR 2025
We propose LongPPL (Long-contex, Perplexity) that is solely based on the selected key tokens. Extensive evaluation shows that contrast to standard PPL, LongPPL exhibits strong correlation with long-context abilities across multiple LLMs and benchmarks.
We introduce LongCE (Long-context Cross Entropy) loss that assigns larger weights to key tokens that gain more from the long context. LongCE attains consistent improvements in a plug-and-play solution, demonstrating its generality for learning long-context models.
2 A FINE-GRAINEL ANALYSIS OF PERPLEXITY
Recent studies have shown that perplexity does not adequately reflect the long-context performance of language models (Agarwal et al., 2024; Li et al., 2024), we have also observed in Figure 1(b). In this section, we demystify this phenomenon with a fine-graine mne-grinedd analysis of the roles of different tokens at long-context performance.
Perplexity is commonly used metric for evaluating LM's ability predict the next word in sequence (Jelinek et al., 1977). For a sequence of tokens æ = (I) I2, C.), a language model parameterized by 8 is learned to predict the conditional probability of each token given the previous context Pe 22 (x) i E [n]. The perplexity (PPL) on this sequence is defined as the inverse of the geometric mean of all token probabilities:
1 Σ log Pa(x; = Pa
(2) = exp
-
(1)
quantifies the model's uncertainty when encountering new tokens. larger likelihood of indi- cates better prediction and lower perplexity.
2.1 NOT ALL TOKENS MATTER FOR LONG-CONTEXT PERFORMANCE
Despite the close connection between perplexity and token prediction accuracy, there is growing idence that LLMs' perplexity does not indicate their performance on long-context benchmarks (Hu et al., 2024a; Hsieh et al., 2024). There are two possible sources of this mismatch: either the log- likelihood- based metric flawed, or the averaged tokens are not representative enough. In this work, we champion the latter explanation by showing that when selecting the proper "key tokens" for long-context understanding, perplexity can correlate very well with long-context performance.
lina mindless- patrot: CONTENT is <28352>
2.5
answar tokens
2.5
Non-answer tokens
Prompt
clusse REGISTER Q: Tell mee what is the <REGISTER
Parplexity
Perplexity
xg
Xx
2.0
CONTENT> line tender-daws?
Standard Resporse
CONTENT
line tencer-cluuse <45129
Nur-amoowe tokerss Amsoett
40
60
80
100
20
40
60
80
Longeval accuracy
LongEval accuracy
(a) Example of answer tokens.
(b) PPL vs LongEval (Yi-6B)
(c) PPL. vs LongEval (CLEX-7B)
Figure 2: (a) An example of the answer tokens in the LongEval task. (b&c) The correlation between accuracy and perplexity on answer tokens non-answer tokens on LongEval. Each point represents the results obtained from testing at specific prompt length ranging from 2k to 28k. The experiments conducted using Yi-6B-200K (Young al., 2024) and CLEX-7B-64K (Chen al., 2024a).
To have an intuitive understanding, let consider a real example from LongEval benchmark shown CONTENT>> in line tender-clause is", straightforward answer formats stemmed immediately from the question, without relying any long-context information. Even short-context LLMs can predict well on these tokens. Since most tokens are cong-context-agnosiic tokens, perplexity computed equally over all tokens do not represent long-context performance.
in Figure 2(a). Most tokens in the answer, "the <RECIS SEECISTER <RECISTER_CONTENT
To quantitatively examine this hypothesis, we conduct experiments on LongEval (Li et al., 2023a), a benchmark for long-context retrieval abilities, where separate the answer tokens that match
3

---

Published as a conference paper at ICLR 2025
the desired answers (e.g., <45129> in Figure 2(a)) from non-answer tokens. We compare the per- plexity computed with these two groups of tokens using two long-context LLMs. As shown in Fig- ures 2(b) & 2(c) (result details in Appendix B.4), the perplexity answer tokens correlates strongly with the LongEval accuracy that represents the long-context performance; instead, the perplexity on the non-answer tokens shows almost no correlation with LongEval accuracy, justifying our intuition that these tokens do not matter for evaluating long-context performance. In other words, should evaluate the perplexity of the key tokens that really matter for long-context performance.
2.2 EXTRACTING KEY TOKENS FROM NATURAL TEXTS
In natural texts used for training LLMs, do not have knowledge of the answer tokens in LongEval experiments (Figure 2). This motivates us to find a surrogate metric that can accurately identify the key tokens that matter for long-context performance.
To measure the influence of long context for each token I₁, we perform an intervention of con- text length. Specifically, given a sequence æ and a language model Po (with strong long-context abilities). for each token 2; that has long context, ability under the full long context 1 = (Z1...... Z;-1) and the log probability under the truncated short context 8; (Ii-K, ........ ( (where K is short length, e.g., 64):
compute the difference between its log prob-
LSDp(a;) log Po (2) log (o.
We call it Long-Short Difference (LSSD), which measures the improvement in prediction accuracy endowed solely by the long context. From causal perspective, 8, serves as the counterfactual context created by the intervention (dropping long context), and the LSD estimates the individual treatment effect (ITE) (Hernán & Robins, 2010) of long context using the language model Pa. Thus, high LSD value indicates that long context plays important part in the prediction of I;, making them the key tokens to be considered for evaluating long-context performance. other words, LLMs good at long-context understanding should be able to predict high-LSD tokens accurately.
0.8
Answer tokens
Answer tokens
Non-answer tokens
0.8
Non-answer tokens
frequency
0.6
Taken frequency
0.6
0.4
0.4
0.2
0.2
0.0
0.55).0..
1) (1.555 (1.5.2) Long-short difference
[-2, -1.)1-15, -11-11 -0.5) Long-context likelihood
(a) LSD of tokens on LongEval.
(b) LCL of tokens on LongEval with large LSD.
Figure 3: (a) Token distribution categorized by long-short difference (LSD). (b) Distribution of tokens with LSD greater than 0.5 categorized by long-context likelihood (L.CL). The tokens are from the standard response of LongEval illustrated in Figure 2(a).
We evaluate the LSD score on LongEval, where we have knowledge of the key answer tokens. As shown in Figure 3(a), we compute the LSD score with a powerful long-context LLM, Mixtral- 8x7B (Jiang al., 2024), and find that answer tokens are clearly separated from the non-answer tokens: most answer tokens have LSD values higher than 2, while most of the non-answer tokens concentrate around low LSD values (lower than 0.5). When using LSD values alone classify answer and non-answer tokens, we attain 85.6% accuracy (Figure 4(b)), indicating that LSD values strongly indicative of the key tokens in long- congccotextt understandings.
From Figure 3(a), we find that small proportion non-answer tokens also have large LSDs (larger than 0.5) and are thus confused together with key tokens. After analyzing, we find that these to- kens can be further separated out by inspecting their Long-Context Likelihood (LCL) under long context:
L.C. (x:) log (x(x))l:)) log Pa(
(3)
4

---

Published as a conference paper at ICLR 2025
98.2%
0.35
Claxsificatio accuraty of arGmetten--srraa www.-oo------rr-ao www.araartttati colants
85.0%
TMlon frovonc
0.30
0.20
0.15
27.0%
0
0.00
[1.
0.0
Random
LGL
LSD
LCL
(a) LSD value distribution GovReport.
Criteria to identify answer tokens.
Figure 4: (a) Distribution of tokens in GovReport categorized by long-short difference. (b) The classification accuracy discriminating answer to non-answer tokens on LongEval with classifier using different metrics (Random refers 50-50 random guess two classes).
A lower LCL. indicates that the language model hardly predicts accurately at 2; even with the long context information. Figure 3(b) shows that these high-LSD non-answer tokens actually have lower LCLs than the corresponding answer tokens, indicating that these tokens are (strongly) mispredicted tokens even under long context. In other words, these tokens fundamentally hard to predict regardless the context. Therefore, we can exclude them from the selection of key tokens.
To summarize, revisit our initial question why perplexity fails represent long-context perfor- mance. As shown in Figure 4(a), most tokens in natural corpus, GovReport (Huang et al., 2021), are cong-context-irreleval tokens with low LSD (lower than 0.5), while only less than 10% tokens are highly influenced by long context (with LSD> 2) and represent long-context abilities. There- fore, perplexity that averages over all tokens (Fquation 1) does not represent the real long-context performance. Instead, combining the LSD (Equation 2) and the LCL (Equation 3) scores, able to accurately identify the answer tokens in LongEval with an accuracy of 98.2% (Figure 4(b)). Based on this result, in the next section, we design new perplexity measure, LongPPL, that is tailored reflect the long-context performance of LMs, by focusing on the key tokens.
MEASURING AND ENHANCING LONG-CONTEXT CAPABILITIES WITH KEY TOKENS
In Section 2, we find that only key tokens correlate well with long-context performance (Section 2. 2.1), and we identify two effective measures to select the key tokens from a natural corpus (Section 2.2). Based these observations, context abilities, and, following in the same vein, we propose new training objective, LongCE, for finetuning LLMs with emphasis on key tokens.
design new perplexity measure, LongPPL, measure the long-
3.1 LONG-CONTENT PERPLEXITY (LONGPPL)
Given a sequence x (I1, (Z1...., ,i....,,,y, Fr)) and a language model Pa be evaluated, we consider a gen- eralized notion of perplexity for long context understanding. Long- context Perplexity (LongPPL), where we can assign influence function I(.) R, each token v.:
LongPPL(x;0,0G) = exp
Σ ISDo, > and LCLg, (x) B; 1, otherwise.
i(x;; Ou) log Po(0s(IZ21)
[1,
where I(x;; 00)
(4)
and f(x,) = I(xx)/ Σ [(zz).
Here, the long-context influence of Ts, Txx 0g) 2 0, selects key tokens to have a large long- short difference (L.DD, Equation 2) and large long-context likelihood (I.CL,, Equation 3) based on

---

Published as a conference paper at ICLR 2025
evaluator model with parameters fb with two threshold parameters a, B. [xx) is the relative influence after normalization. The first criterion ensures that the generation of the token enhanced by the additional information in the long-context. The second criterion excludes the fundamentally hard (misclassified)) tokens that long context information does not help. Based on these criteria, all tokens are divided into two categories. Tokens that meet the criteria are selected as key tokens and are included in the perplexity calculation with equal weight, while those that do not meet the criteria are excluded from the calculation. Later in Section 4.1, we show that in contrast to standard PPL, LongPPL computed on natural language corpus for multiple LLMs correlates well with their performance on long-contex: benchmarks, including LongEval (Li et al., 2023a), LongBench (Bai et al., 2023b), and RULER (Hsieh et al., 2024). We also consider other similar variants of the influence function (e.g., with soft reweighting) and find them to be generally effective (though often less accurate).
Remark on the Evaluator Model Oo Notably, the evaluator Peo used for computing the long- context influence can be different from the evaluated model Po. In fact, for the evaluator, we need powerful model to ensure that they give a relatively accurate estimate of the token's long-context influence. This requires the evaluator itself to have a strong long-context understanding ability. Our empirical findings show that using the model Po itself as the evaluator Peo leads to LongPPL. being unable to distinguish the model's long-contex capabilities (Appendix B.2). In practice, we find that a small-sized model like Llama-3. 1amma-3.1-81 1-8B (Dubey et al., 2024) is enough to serve as good evaluator.
3.2 IMPROVING LONG-CONTEXT CAPABILITIES WITH LONGCE
Due to the massive computational cost of pre-training an LLM from scratch on long texts, current long-context LLMs are pretrained on short contexts and then fine-tuned on longer contexts. By default, the long-context fine-tuning process adopts the Cross Entropy (CE) loss as in pre-training which adopts a uniform average of all tokens, akin to standard perplexity (Equation l):
a
CE(x; = - Σ log Po (o(
(5)
Nevertheless, this de facto paradigm has the same issues that we discussed for perplexity in Sec- tion 2. We show that most tokens in sequence are not influenced by the long context, while only few key tokens require depends crucially on its prediction on these key tokens (as measured in LongPPL, Section 3.1).
information; and in turn, the model's long-contexs performance
Following the methodology of LongPPL (Equation 4), we propose the LongCE (Long-context Cross Entropy) loss that reweights every token 2; w.r.f. its gain (x,; 8) from long context:
LongCE(<;;)) 0) n
Tmott (x;;; 0) log Po(a;(anes).
For the ease of differentiable optimization using all tokens, we adopt a soft long-context influence function Isoft X + 10. 7 based on the likelihood ratio between the long- long-context probability (L,())) and short-context probability (s(x) (defined in Section 2.2):
Loote (x;; 0) = min (exp (L:D) 7) = min (o, Pa (x))
Here, 7 > is a hyper-parameter that sets threshold on the maximal influence to avoid numerical instability. As a consequence of this reweighting term, too easy tokens (both short and long con- text give accurate prediction) and too hard tokens (neither short or long context predicts correctly) will have a weight around 1, while those long-context-deeeeneer tokens (high (o(x::4:) and low Po(1))s)) will be upweighted above 1, proportionally the context informativeness.
Remark. Unlike the influence function of LongPPL (Equation 4), which uses a powerful LLM as an external evaluator to select tokens more effectively, LongCE leverages the same model to evaluate the influence for training efficiency. Therefore, LongCE training does not require a separate evalua- tor model, but uses the model itself for long context evaluation. In this way, LongCE bootstraps the model's long-context capabilities in an EM sntrecttttiommaxiiiaaeee
way: the language model Pg first uses itself to estimate long-context influence of each token Teott (Equation 7); and then this estimate is used to update the model parameters by optimizing the LongCE loss function LongCE

---

Published as a conference paper at ICLR 2025
corr-0.24, 24, p=0.54
cormo 27, pm0.09
5.5
Y1-22
5.0-
Owan 5-34
Owent 5-348
Swar1-7
Owe12:78
4.5
Mistre775 RIM-T
Masral-7
100m70
C.E8-7
Micral
3.
7am-70
corr=-0.86, p=0.002
3.5
1am-77
CLEX-78
60
corr= p-O 005
LorgPeL
5-15-7
Long
3.00
Ownnl
DoanLa
2715-30
2.5
M
Micoo 78
F1 M03
2.0
2.0-
20
30
70
80
50
80
LongEval accuracy
RULER
(a) LongEval
(b) RULER
Figure 5: Correlation between the PPL-based metrics (LongPPL. and PPL) on GovReport (Huang et al., 2021) and long-context benchmarks. LongPPL. is calculated using Qwen2-72B-Instee Re- sults of LongBench in Figure 1(b).
(Equation 6). This process enables the model to focus more effectively on the key tokens critical to long-context performance, thereby improving training efficiency. We also note that computing key tokens introduces some additional computational overhead. However, subsequent experiments show that this overhead is acceptable, given the clear performance improvements.
EXPERIMENTS
In this section, we conduct real-world experiments to analyze the applicability of the proposed LongPPL and LongCE. For all the experiments, we use LongBench (Bai et al., 2023b), LongEval (Li et al., 2023a), and RULER (Hsieh et al., 2024) as the long-context benchmarks. We report the average score on LongBench, the accuracy the subtask "lines" of LongEval, and the score on RULER. For LongBench and RULER, we restrict the prompt length to 32k tokens. For LongEval, use 1350 lines the prompt. which is approximately 32k tokens.
Practical Implementation. In the implementation of LongPPL. and LongCE, we need to compute the log probabilities for each token under both the long and the truncated short context. For the truncated short context of length K, one can use the sliding window technique in Transformers for computing token predictions in parallel to improve computational efficiency. For computing LongPPL when the evaluator model and the evaluated model have different tokenizers, only keep key tokens that form the longest common substrings of the evaluated tokens. More details can be found in Appendix A.I.
4.1 LONGPPL METRIC
Experimental Setup. We calculate LongPPL on the GovReport dataset (Huang et al., 2021), which consists of long sequences from government reports. We sample 50 documents with the context length up 32k tokens. We set the hyperparameters 72B-Instruce (Yang et al., 2024), an open-source LLM with the context length of 128k tokens, as the discriminator model du select the key tokens. We also consider using Llama-3.1 (Dubey et al., 2024) later and Mistral Large 2 (Jiang et al., 2023) Appendix B.
2, 3 -2, K 4096. We use Qwen2-
LongPPL Correlates Well with Long-context Performance. In Figure 1(b) and Figure 5, we demonstrate the correlation between LongPPL and long-context benchmarks on various long- context L.Mss. We observe that LongPPI. exhibits a very strong negative correlation with perfor- mance on long-context tasks across different models, with pearson correlation coefficients exceeding
7

---

Published as a conference paper at ICLR 2025
Cor--0.96 4x 40
corr=-0.8
corr=-0.5
p=0.001 0010
001
2.8
2.8
2.8
2.6
2,6
2.6
2.4
2.4
2.4
2.2
2.2-
2.2-
2.0
2.0
2,0
1.8
35
LongBench score
LongEval accurecy
RULER score
(a) LongBench
(b) LongEval
(c) RULER
Figure 6: Correlation between LongPPL on GovReport and long-context benchmarks. LongPPL is calculated using J.88--3.1-8B.
The Pearson correlation between different perplexity measures and benchmark scores, where a lower correlation is the better (since we expect lower perplexity indicates higher bench- mark scores).
Table 1:
Metrics
Influence I
LongBench
LongEval
RULER
PPL
(x) =1
-0.11
0.31
0.33
LongPPL-soft
soft (Equation 7)
-0.43
-0.21
-0.17
LongPPL-hard (default)
(Equation 4)
-0.96
-0.86
-0.84
-0.8 for all three tasks. In contrast, perplexity hardly shows a correlation with the long-context tasks. This indicates that LongPPL sufficiently capable of measuring a model's long-context capabilities.
LongPPL is Compatible with Small-sized Evaluator Models. To demonstrate that the effec- tiveness of LongPPL is not restricted by the size of the evaluator model, we additionally conduct experiments on a smaller model, 3lamma-311888 (Dubey et al., 2024). As shown in Figure 6, the LongPPL computed using 8B-sized model also achieves high correlation coefficients of -0.96, -0.89, and -0.90 with the three long-context benchmarks, respectively. In Appendix B.8, we have made discussion about the efficiency of LongPPI..
Hard Standard for Key Tokens Better than Soft Re-weighting Standard. In Equation 4, we use an indicator function as the influence function. Instead, we have also tried to use the soft reweighting function T used in LongCE (Equation 7) calculate LongPPL. Its token matching strategy is detailed in Appendix A. In Table 1, we show that LongPPL with soft criteria has a weaker correlation with the long-context benchmarks compared to LongPPL, indicating that the soft reweighting influence function is suboptimal for LongPPL. Besides, in Appendix B.2 and B.7, we have also explored some other alternative approaches, including using the model itself as the evalua- tor, removing the LCL discriminative condition, and using N N-grams as the key token discriminative condition. We find that all of these approaches led to worse performance.
LongPPL is not sensitive to the choice of hyperparameters of and B. To investigate the impact of the two threshold hyperparameters, i.e., a and 3 (in Equation 4), we conducted further ablation experiments. The results are presented in Table 2. Our findings reveal that when 33=1, a=1 or 2, the correlation between LongPPI. and the long-context benchmarks even improves. Notably, these hyperparameters were directly reused from the motivation experiments without any further tuning. The results indicate that LongPPL's performance largely insensitive to the choice of hyperparameters, with the correlation coefficient remaining below -0.8 in most cases.
4.2 FINE-TUNE WITH LONGCE LOSS
Experimental Setup. We primarily use Llama-2-77 (Touvron al.. 2023) the base model to perform long-context finetuning. We also conduct experiments on Mistral-7B-vO.1 (Jiang et al., 2023) and 2-3B.B We use PG-19 (Rae et al., 2020), book dataset sourced from library,
8

---

Published as a conference paper at ICLR 2025
Table 2: The Pearson correlation between LongPPI., calculated with different hyperparameters (o, 9), and the long-context benchmarks. In most cases, the correlation coefficients remain below -0.8.
LongPPL
LongBench
LongEval
RULER
2, 3 -2 (default)
-0.96
-0.86
-0.84
a 2, 3 -1
-0.96
-0.92
-0.92
a = 1,8 3 = -2
-0.91
-0.73
-0.69
1 -1
-0.97
-0.88
-0.87
Table 3: Long-context performance of the fine-tuned models using the standard CE loss and our proposed LongCE loss. We fine-tune Llama-2-7b on long texts using various fine-tuning strategies (EABF and PI) and different training data (PG-19 and Pile-arxiv). The models are then assessed on benchmarks with prompts of up to 32k tokens.
LongBench 100
LongEval
RULER
Training steps
50
200
50
100
200
50
100
200
Setting A (PG-19 dataset with EABF)
CE
24.5
26.6
26.9
16.0
24.0
24.0
34.5
38.6
42.7
LongCE (Ours)
26.0
27.2
28.2
24.0
46.0
46.0
43.1
48.3
49.7
Gain
(+1.5)
(+0.6)
(+1.3)
(+8.0)
(+22.0)
(+22.0)
(48.6)
(+9.7)
(+7.0)
Setting B (PG- 19 dataset with PI)
CE
24.3
25.3
25.4
20.0
28.0
26.0
22.1
31.8
35.7
LongCE (Ours)
24.4
25.0
25.8
38.0
42.0
27.3
34.4
36.4
Gain
(+0.1)
(-0.3)
(40.4)
(+18.0)
(+16.0)
(+16.0)
(45.2)
(42.6)
(+0.7)
Setting C (Pile-arxiv dataset with EABF)
CE
15.0
23.1
23.8
8.0
18.0
14.0
40.9
53.3
51.9
LongCE: (Ours)
17.6
24.0
25.0
10.0
18.0
16.0
49.7
54.8
58.6
Gain
(+2.6)
(40.9)
(+1.2)
(+2.0)
(+0.0)
(+2.0)
(48.8)
(+1.5)
(+6.7)
and Pile-arxiv (Gao et al., 2020), a dataset consisting of Arxiv papers, as the training dataset. The training sequences are organized to be the context length with 32k tokens. For the calculation of LongCE, we set y = Equation and use the same sliding window approach as described in Section 4.1 to improve training efficiency. The context length of 8; is set to be K = 4096. We fine- tune the base models with Entropy-awaid Adjusted Base Frequency (EABF) (Zhang al., 2024c) and Position Interpolation (PI) (Chen et al., 2023). Specifically, EABF applies a scaling mechanism to the attention and uses higher base frequency for RoPE, while PI linearly downscales the position indices of the input tokens. These methods can significantly accelerate the convergence speed of long-context fine-tuning and have been widely adopted in many LLMs (Yang et al., 2024; Dubey al., 2024; Chen al., 2024a). Detailed training setups are available Appendix A.2.
LongCE Outperforms CE in Various Settings. As shown Table 3, we present the long-context capabilities of models fine-tuned with LongCE loss and CE loss under different fine-tuning strategies and training datasets (see fine- reee-grained results LongBench effectiveness of LongCE using different base models in Table 4. We find that models fine-tuned with LongCE, loss consistently outperform those fine-tuned with CE loss across nearly all settings. This suggests that the LongCE loss, with its re-weighting strategy based on long-conten token importance, can be applied as plug-and-pla module which can effectively improve the model's long-context performance. To demonstrate the model's performance when the context length is over 32K, we provide the sosdll-in-a-Haystaan (Kamradt, 2023) evaluation results in Appendix B.5, which leads to similar conclusions. Besides, empirical results in Appendix B.6 demonstrate that LongCE does not cause any additional loss in the model's performance normal-length tasks.
Appendix B.3). We also test the
Training Efficiency. addition to the performance improvement brought by the LongCE loss, we also pay attention to the changes in training efficiency. In LongCE, we need an extra forward pass calculate the probability under short context Po(xy (s)). which introduces additional computation costs. By using a sliding window technique (as detailed Appendix A. I), the computational over- head of training the model with LongCE is controlled to about 80% that of training with CE loss.

---

Published as a conference paper at ICLR 2025
Table 4: Long-context performance of different fine-tuned models. We fine-tune Mistral-7B-vO.1
and Llama-2-133 2-13B with EABF adjustment strategy on Pile-arxiv dataset.
LongBench 100
LongEval
RULER
Training steps
50
200
50
100
200
50
100
200
Mistral-7B-ve
CE
29.6
28.9
28.4
26.0
14.0
12.0
45.0
44.5
42.9
LongCE (Ours)
30.8
30,9
31.1
36.0
30.0
26.0
45.1
44.0
43,5
Gain
(+0.8)
(+2.0)
(+2.7)
(+10.0)
(+16.0)
(+14.0)
(+0.1)
(-0.5)
(+0.6)
Llama-2-138
CE
26.3
26.9
28.2
14.0
14.0
14.0
45.4
50.4
LongCE (Ours)
26,4
28.5
28,9
20.0
18.0
18.0
55.1
61.9
62.5
Gain
(+0.1)
(+1.6)
(+0.7)
(+6.0)
(44.0)
(44.0)
(49.7)
(+11.5)
(+10.2)
LangCC
25-
Lingeerd
LongLE
10-
LonyCE
6
10
12
0
2
10
12
LongBench
Longeval
(c) RULER
Figure 7: Long-context fine-tur tne-tuning performance (PG-19 19 dataset with EABF) wall clock training time. LongCE demonstrates a stronger potential for enhancing long- long-context capabilities.
We visualize in Figure 7 how the long-context performance of models fine-tuned with LongCE and CE changes over the course of training time. Most of the time, fine-tuning with LongCE loss is more efficient method. Additionally, in Appendix B.2, we find that by changing the hyperparameters of LongCE, i.e., the short context-length and the sliding window length this overhead can be further reduced to 36%, with almost no loss in model performance.
CONCLUSION
In this paper, we offer a comprehensive explanation for why perplexity fails to reflect the long- context capabilities of LLMs. We find that as perplexity treats all tokens equally, it lacks suffi- cient attention on the key tokens that are crucial for long-context understanding. To address this, we propose a novel metric. LongPPL, which focuses on the key tokens in natural texts through long-short context constrastive method. We empirically demonstrate the strong correlation with the long-context capabilities of LLMs as indicated by LongPPL. and the performance on long-context utilize the concept of LongPPL to propose the LongCE loss, which
benchmarks. In addition, reweights the CE loss used in the long-context fine-tuning. By up-weighting the key tokens, LongCE leads to consistent improvements across multiple cong-context benchmarks with up to 22% gains in LongEval accuracy. We hope our analysis and approaches can provide insights for a better under- standing into the essence of long-context generation.
ACKNOWLEDGEMENT
Yisen Wang was supported by National Key R&D Program of China (2022ZD0160300), Na- tional Natural Science Foundation of China (92370129, 62376010), and Beijing Nova Program (20230484344, 20240484642). Yifei Wang and Stefanie Jegelka were supported in part by the NSF AI Institute TILOS, and an Alexander von Humboldt Professorship.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Luis Rosias, Stephanie CY Chan, Biao Zhang, Aleksandra Faust, and Hugo Larochelle. Many-shot in-context learning, In ICML 2024 Workshop on In- In-Context Learning, 2024.
Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. Make your Ilm fully utilize the context, arXiv preprint 248::2404..1811,,, 2024.
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Re. Zoology: Measuring and improving recall in efficient language mod- cls. In ICLR, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, LuoJi, Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. 14508, 2023b.
Aydar Bulatov, Yuri Kuratov, Yermek Kapushev, and Mikhail S Burtsev. Scaling transformer to Im tokens and beyond with rmt, arXiv preprint arXiv:2304. 10622 2023.
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. Booookscore: A systematic exploration of book-length summarization in the era of 1lms. In ICLR, 2024.
Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. Clex: Continuous length extrapolation for large language models. In ICLR, 2024a.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15559 2023.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. In ICLR, 2024b.
Ta-Chung Chi, Ting-Han Fan, Peter Ramadge, and Alexander Rudnicky. Kerple: Kernelized relative positional emhedding for length extrapolation. In NeurlPS, 2022.
Kevin Clark, Kelvin Guu, Ming- Wei Chang, Panupong Pasupat, Geoffrey Hinton, and Mohammad Norouzi. Meta-learning fast weight language models. arXiv preprint arXiv:2212.02475. 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 004557 2018
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02488, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llmaa 3 herd of models. arXiv preprint axX.v:2407.21788 2024.
11

---

Published as a conference paper at ICLR 2025
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint
200::20110000.,, 2020.
Shuhao Gu, Jinchao Zhang, Fandong Meng, Yang Feng Wanying Xie, Jie Zhou, and Dong Yu. Token-level adaptive training for neural machine translation. arXiv preprint arXiv:2010.04380, 2020.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint urivv2338..6137, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Miguel A Hernán and James Robins. Causal inference, 2010.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg, Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.0666 2024.
Nathan Hu, Eric Mitchell, Christopher D Manning, and Chelsea Finn. Meta-learning online adapta- tion of language models. arXiv preprint arXiv:2305. 15076, 2023.
Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng. Can perplexity reflect large language model's ability in long text understanding? In The Second Tiny Papers Track at ICLR 2024, 2024a.
Zhiyuan Hu, Yuliang Liu, Jinman Zhao, Suyuchen Wang. Yan Wang, Wei Shen, Qing Gu, Anh Tuan Luu, See-Kiong Ng. Zhiwei Jiang, et al. Longrecipe: Recipe for efficient long context general- ization in large language models. arXiv preprint arXiv:2409.0055 00500, 2024b.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In NAACL, 2021.
Fred Jelinek, Robert Mercer, Lalit R Bahl, and James K Baker. Perplexity- a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63, 1977.
Albert Q Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04008 2024.
Gregory Kamradt. Needle in haystack pressure testing Ilms., 2023. URL https:///itht: com/ homm/gkamradt...mm LLW./ttMMMtttt//
in.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations. In EMNLP, 2017.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Gonzalez Joseph E, Stoica Ion, Xuezhe Ma, and Hao Zhang. How long can open-source lims truly promise on context length?, 00---0662200 2916--291100 2ttp:/2222200-2---1-ghohhahaa longchat.
June 2023a. URL https: / /lmassmsssss////m/mssa.........a
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting I1m performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308. /2032, 2023b.
Tianle Li, Ge Zhang, Quy Duc Do. Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.000 02060, 2024.
12

---

Published as a conference paper at ICLR 2025
Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang. Min Yang. Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, et One shot learning as instruction data prospector for large language models. arXiv preprini arXiv:2312. 10302, 2023c.
Stephanie Lin, Jacob Hilton, and Owain Evans. falsehoods. In ACL, 2022.
Truthfulqa: Measuring how models mimic human
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, et al. Rho-l: Not all tokens are what you need. arXiv preprint N0V:24004.07655. 2024.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for align- ment? a comprehensive study automatic data selection in instruction tuning. arXiv preprint arXiv:2312. 15685, 2023.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05001. 2017.
Haocheng Luo, Wei Tan, Ngoc Dang Nguyen, and Lan Du Re-weighting tokens: A simple and effective active learning strategy for named entity recognition. arXiv preprint arXiv:2311.00 S.0:::331.000006, 2023.
Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of Ilm agents. arXiv preprint arXiv:2402.17753 2024.
Pedro Henrique Martins, Zita Marinho, and Andre Martins. oo-former: Infinite memory transformet-foreer Infinite memory transformer. In ACL, 2022.
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. arXiv preprint arXiv:2305. 16300, 2023.
Xinzhe Ni, Yeyun Gong, Zhibin Gou, Yelong Shen. Yujiu Yang, Nan Duan, and Weizhu Chen. Exploring the mystery of influential data for mathematical reasoning. arXiv preprint MxXi::2404.01067 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In ICLR, 2024.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In ICLR, 2021.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. ICLR, 2020.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding. In EMNLP, 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputiing 568:1168:00 127063, 2024.
Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. Do long-ran ranggrange language models actually use long-range context? In EMNLP, 2021.
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaud- hary, Xia Song, and Furu Wei. A length-extrappllaable transformer. In ACL, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In ACL, 2023.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In NAACI. 2019.
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09228, 2023.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural ma- chine translation. arXiv preprint MXXv:2004406644, 2020.
Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations, arXiv preprint arXiv:2310.00 0.06687, 2023.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In ICLR, 2024.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. In NAACL, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint 0rXiv:2403.04652. 2024.
Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Soaring from 4k to 400k: Extending Ilm's context with activation beacon. arXiv preprint arXiv:2401.03462, 2024a.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. ocbench: Extending long context evaluation beyond 100k tokens. In ACL, 2024b.
Yikai Zhang, Junlong Li, and Pengfei Liu. Extending llms' context window with 100 samples. arXiv preprint arXiv:2401.07000 2024c.
Dawei Zhu, Nan Yang. Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension of llms via positional skip-wise training. ICLR, 2024.
14

---

Published as a conference paper at ICLR 2025
A DETAILED SETTINGS IN EXPERIMENTS
A.1 IMPLEMENTATION DETAILS OF LONGPPL
Sliding window algorithm to improve efficiency. Since the calculation of LongPPL requires com- puting the LSD for each token T E [n]. necessitates calculating the probability under short context Pe (z, 8,) for n. K times, where K is the length of st. Theoretically, the computational complexity of this process is ((n K)) Since K is typically larger than (e.g., when K = 4096, K² = 16M, which is much greater than n = 32k), this complexity far exceeds the normal (n²²) complexity of a standard long-context forward pass. As result, the time cost of this process is quite significant.
To make this process more efficient, we use a sliding window algorithm to improve efficiency. Specifically,
introduce step size d, which is smaller than the truncation length / (we to d = 1024). When calculating the short-context probabilities of z; to Fi4d-1, we set the starting token of the context uniformly as -i Formally speaking, we have
Skd+'' = (# -I.,
(8)
where & N. < d. This approach allows for the calculation of the short-context probabilities of d tokens in a single forward pass, resulting in a complexity of O((N - K))3/d) To access better understanding on the selection of K and please refer Appendix B.2.
Token matching method. Since the used tokenizers between evaluator model Po, and evaluated models Pe could be different, attempt to align the key tokens between different models. For- mally, we define the encoding and decoding functions of tokenizers used in language models as encodep and decodep. Let t (4,...,(s (N) be the original text contains of N characters, and (I,, Fn)) encode Pao (t), x' (Fi...., x.....,,. xxx)) = encode P. (t) be the token sequence en- { n4 the set of key tokens calculated by the evaluator model Pa. We map these tokens to the text space as T = decode Pwg (X).. Then, the key
coded by Po, and Po. respectively. Let
token set X' of the evaluated model is the maximal subset of x' which satisfies
decodep, (X') CT.
Besides, in Table 1, we also implement the LongPPL with the soft influence function Looft (Eq, (7)). In this approach, we implement reweighting algorithm to transfer the weight between different tokenizers. Specifically, denote w = (Wi,.., wa) as the LSD weight on æ calculated by Pag. The weight of na is defined as
w
Σ
w(ts)//decodep.(ft (x:)),
(10)
ty Edecndery (m()
where w(tt) ) is the weight of the token that tj belongs This assigns the weight of x with the string-level average of the weight x.
A.2 IMPLEMENTATION DETAILS OF LONGCE
Fine-tuning strategies. For EABF (Zhang et al., 2024c), we adopt the identical settings in the original paper, with RoPE base of 500k. For PI (Chen et al., 2023), we set the scaling factor to since we want to extend the context window from 4k to 32k.
learning rate of 2 10-5 for Llama and 10-6 for Mistral, with no weight decay and a linear warmup of 20 steps along with AdamW (Loshchilov, 2017) with 31 = 0.9 and Bg 0.95. We apply global batch of 64 on PG-19 and 8 on Pile-arxiv. We disable the sliding window mechanism when fine-tuning Mistral-7B-vl A100 80GB GPUs using Pytorch (Paszke et al., 2019).
Training details. We
We perform the experiments with 8 Nvidia
B SUPPLEMENTARY EXPERIMENT RESULTS
B.1 DETAILED RESULTS LONGPPI.
We present the LongPPL. calculated by different models in Table 5, and provide further visualization results for Mistral I Large in Figure 8.
15

---

Published as a conference paper at ICLR 2025
Table 5: The perplexity-based metries of various L.Mss.
Metric
LongPPL Mistral Large 2
PPL
Evaluator model
Beenn--72B-Inttuue
Llama Llama-3.1-80 3.1--BB
Mixtral-8x7B-32k
2.08
2.50
1.74
3.67
FILM-7B-32K
2.49
3.17
2.03
4.47
Mistral-7B-324
2.68
3.49
2.19
4.25
Qwen 15--11B-12888
2.97
2.93
2.33
5.23
Qwen2-7B- -22k
2,99
2.73
2.29
4.97
Phi-3-small- 128k
2.98
2.86
2.41
5.42
CLEX-7B-64k
3.70
4.60
2.92
4.13
Yi-6B-200k
3.62
3.92
2.86
5.11
Yam--B-128k
3.67
4.88
3.10
4.17
5.0
5.0
5.0
corr=-0.79
corm=-0.38 91
corr=-0.99 pmix 10
pm0.01
0-0
4.5
4,0-
LongWW
LongP91
3.5
3.5
3.5
3.0
3.0
3.0
2.5
2.5
Longitench
Longlival accuracy
(a) LongBench
(b) LongEval
(c) RULER
Figure 8: Correlation between LongPPL on GovReport and long-context benchmarks. LongPPL is
calculated using Mistral Large 2.
B.2 ABLATION STUDY
LCL. In the calculation of LongPPL, we employ LCL as an assistant for our core criterion, LSD, in selecting key tokens. In Figure 9, we demonstrate the LongPPL calculated without the LCL criterion. This version of LongPPL hardly has correlation with the long-context benchmark, showing that LCL is an indispensable part for LongPPL.
cor--.0. 0.34
corr--0.03
corr-O.11
Longfrt Lo L7V
LongTench
Longfvel accurney
Figure 9: LongPPL without LCL.
Evaluator model. the main text, we use a evaluator model fg to identify the key tokens. To vali- date the necessity of this approach, calculate LongPPL using the model itself the evaluator, as shown Table The results indicate that most models achieve similar LongPPL scores, suggesting that this self-evaluated version of LongPPL does not reflect the models' long-context capabilities.
Hyperparameters LongCE. In the computation of LongCE, several hyperparameters are uti- lized, including the short context window length K and sliding window length d used in calculating LSD. Here, design ablation experiments analyze the selection of these hyperparameters, as
16

---

Published as a conference paper at ICLR 2025
Table 6: LongPPL. using the evaluated model itself to calculate the key tokens.
Mixtral
FILM
Mistral
Qwen1.5
Qwen2
Phi-3
CLEX
Yi
Yarn
LongPPL
1.67
1.64
1.68
1.67
1.65
1.65
1.68
1.75
1.92
Table 7: The performance and time cost of LongCE on long-context benchmarks under different hyperparameter settings of K and d. For the time cost, report the wall-clock time for training 200 steps.
Total training time h
LongBench
Longtival
RULER
Training steps
200
100
200
50
50
100
200
Serring (PG-19 dataser EABF)
CE
24.5
26.6
26.9
16.0
34.5
38.6
42.7
LongCEE (A 1k. d 1k. default)
12.5 (+79%)
26.0
27.2
28.2
24.0
46.0
46.0
43.1
48.3
49.7
LongCE (A lk, 1k)
10.0 (+43%)
25.8
26.9
20.0
48.0
45.6
51.1
55.9
LongCE (K 4k, d
9.5 (+36%)
25.4
25.8
25.8
28.0
56.0
56.0
42.5
48.0
51.2
LongCE (K = 4k, d 512)
17.5 (+150%)
25.4
25.8
27.3
26.0
48.0
60.0
42.4
50.1
$4.4
shown in Table 7. The results reveal that, on one hand, increasing K or decreasing d significantly improves the efficiency of LongCE (from +79% +36%/443%). On the other hand, under these settings, although the model's performance on real-world tasks (LongBench) slightly decreases, it achieves substantial improvements synthetic tasks (LongEval, RULER). This suggests that LongCE still holds potential for further efficiency enhancements.
B.3 FINE-GRAINED RESULTS OF LONGCE
In this section, provide more detailed LongBench scores of the models from the experiments in section 4.2, as shown in Table We observe the models finetuned by LongCE outperforms the model finetuned with CE primarily in single/multi-docunen QA, summarization and synthetic tasks (including retrieval and counting tasks). This also explains why LongCE can significantly outperform CE on LongEval and RULER, as their synthetic tasks primarily assess models' retrieval, summarization, and QA capabilities in long-context scenarios.
Table 8: Detailed scores of LongBench in Table 3.
Single-Document
Multi- Multi-Document
Few-shot Learning
Code Completion
Synthetic Tasks
Tisk Domains
QA
QA
Summarization
Avg.
Setting A (PG-19 dataset with EABF)
CE (50 steps)
4.4
1.1
15.5
66.7
59.7
0.0
24.5
CE steps)
0.4
26.6
CE (200 steps)
6.9
2.3
22.8
66.8
61.9
0.4
26.9
LongCEE (50 steps)
7.6
22.0
66.1
$7.9
0.5
26.0
LongCE (100 steps)
7.7
3.3
22.5
65.7
61.6
2.3
27.2
LongCE (200 steps)
9.3
4,8
23.9
66.0
61.9
3.2
28.2
Setting B (PG-19 dataset with Pf)
CE (50 steps)
3.1
3.2
12.9
59.8
1.6
24.3
CE (100 steps)
4.1
3.5
17.5
65.2
59.9
1.8
25.3
CE (200 steps)
5.6
4.0
15.4
66.0
60.33
1.0
25.4
LongCE (50 steps) LongCE (100 steps)
4.5
2.2
63.1
58.4
2.7
24.4
4.6
1.7
17.7
64.1
59,0
2.8
25,0
LongCE (200 steps)
6.0
4.3
19.0
63.6
59.2
2.7
25.8
Setting (Pile-arxiv dataset with EABF)
CE (50 steps)
1.7
38.2
0.0
15.0
CE (100 steps)
4.2
5.4
65.0
58.9
0.0
23.1
CE (200 steps)
5.1
7.1
7.6
64.3
58.7
0.0
23.8
LongCE: (50 steps)
17.6
LongCE (100 steps)
4.2
5.3
10.0
64.3
59.1
1.0
24,0
LongCEF (200 steps)
3.7
14.3
64.7
59.8
1.3
25.0
17

---

Published as a conference paper at ICLR 2025
Table 9: Detailed results of experiments in Figure including the accuracy LongEval, and
perplexity tested on answer and non-answers tokens, respectively.
Prompt Length
2k
5k
4k
5k
7k
9%
11k
13k
15k
17k
19k
21k
23k
25k
28k
Y--6B-2000
LongEval sccuracy
100.0
94.0
84.0
76.0
76.0
64.0
68.0
54.0
60.0
58.0
46.0
44.0
50.0
52.0
48.0
(answer tokens)
1.47
PPL (non answer tokens)
2.15
2.17
2.12
2.18
2.18
2.20
2.27
2.25
2.25
2.23
2.23
2.21
2.22
2.25
2.24
CLEX-7B-64K
I LongEval accuracy
82,0
34.0
84.0
82.0
58.0
62.0
58.0
56.0
50.0
44.0
46.0
24.0
22.0
28.0
24.0
PPL (answer tokens)
1.31
2.33
1.23
1,33
1447
1.43
1.51
1.54
1.63
1.78
1.89
2.23
2.50
2.61
2.59
PPL (non-answer tokens)
2.22
2.31
2.17
2,18
2.10
2.16
2.17
2.14
2.14
2.15
2.15
2.18
2.20
2.24
2.24
B.4 DETAILED RESULTS OF THE EXPERIMENTS IN SECTION 2.1
In Table 9, we present the detailed results from the experiments in Figure 2(b) and 2(c).
B.5 NEEDLE-IN-A A-HAYSTACK RESULTS
In this section, conduct the standard Needle-in-a-Haystaee (NIAH) evaluation to evaluate mod- els' long-context capability when context lengths greater than 32K.
We first test the models obtained in the main text, which are fine-tuned on 32K-length texts. As shown in figure 10, LongCE achieves score of 10 on 5 out of questions at the 40K length and out of 6 questions at the 48K length, outperforming CE, which achieves a score of 10 on out of and 0 out of 6 questions, respectively. Therefore, LongCE demonstrates longer effective context length.
Additionally, to demonstrate the generalization ability of LongCE longer context lengths, extend the context window of both models by increasing their RoPE base from 500K to 2M. The corresponding NIAH results are shown in Figure 11. The results show that model finetuned with LongCE answers all questions correctly at the 64K length and achieves score of 10 on 32 sequences with lengths of 232K, while CE only achieves this on 26 sequences. This indicates that LongCE generalize well longer lengths.
B.6 LONGCE'S PERFORMANCE ON NON-LONG-CONTEXT LANGUAGE TASKS
In this section, experimentally investigate whether LongCE will adversely impact non-long- context capabilities. In Table 10, we present the model performance on 6 common language tasks, i.e., MMLU (Hendrycks et al., 2021), ARC-Challenge (Clark et al., 2018). RACE (Lai al., 2017), BigBench Hard (Suzgun et al., 2023), TruthfulQA (Lin et al., 2022), and CommonsenseQA (Talmor et al., 2019). The results show that for non-long-conter with LongCE nearly identical to that of the model trained with CE, indicating that the long- context-specific characteristics of LongCE do not negatively affect the model's performance on tasks involving normal-length context compared the baseline.
tasks, the performance of the model trained
Table 10: The performance of models fine-tuned with CE and LongCE on non-long- context tasks. The models are finetuned with 200 steps under the setting A in Table 3.
Models
MMLU
ARC-C
RACE
BBH
TruthfulQA
CommonsenseQA
Avg.
Llama-2-7B
41.8
43.3
39.5
39.4
34.5
32.9
38.6
+CE (baseline)
40.8
42.8
40.3
36.4
29.3
31.5
36.9
+LongCE (ours)
39.9
43.9
39.3
37.5
30.0
30.8
36.9
18

---

Published as a conference paper at ICLR 2025
Retrieval Lengths HayStack")
10
0.0
20.0
Dopth Parcent
40.0
100.0
16k
24
40k
ank
sek
Token Lime
Model finetuned with CE.
Across Context Lengths ('Needle HayStack")
10
0.0
20.0
Depin Purcont
40.0
60.0
200.0
18k
24
A8k
sek
64k
Taken Limit
(b) Model finetuned with LongCE
Figure 10: Needle-in-a-haystack results of models trained with PG-19 datasets & EABF for 200steps.
B.7 SUBSTITUTING KEY TOKENS WITH RE-OCCURRED N-GRAM
In this section, we examine whether LongPPL works by retrieving the frequent N-gram in the con- text, concerned recent works (Sun et al., 2021; Arora al., 2024). We calculate perplexity solely on the re-occurred N-gram (word-level, N > 2) in the inputs, and present the correlation coefficients with the benchmarks in Table 11.
Table 11: The correlation coefficients between PPL calculated re-oceurred N-gram, and the benchmarks.
LongBench
LongEval
RULER
PPL
-0.11
0.24
0.27
PPL (N-gram)
-0.41
-0.10
-0.05
LongPPI.
-0.96
-0.86
-0.84
The results show that PPL on re-occurred N- N-grams has much weaker correlation with model's long- context capabilities. This indicates that LongPPL's powerful ability to capture long-oontett-related information cannot be simply explained by N-grams.
19

---

Published as a conference paper at ICLR 2025
Retrieval Lengths HayStack")
10
0.0
20.0
Dopth Parcent
40.0
100.0
16
18k
24k
32k
40K
Stak
64k
72k
B0k
ark
Token Lime
Model finetuned with CE.
Across Context Lengths ('Needle HayStack")
10
0.0
20.0
Depin Purcont
40.0
60.0
200.0
4
10k
24
12k
GAK
72k
gok
aek
Taken Limit
(b) Model finetuned with LongCE
Figure 11: seedle-in-a-haystar results of models trained with PG-19 datasets & EABF for 200steps. We increase the RoPE base from 500k to 2M after finetuning.
B.8 TIME CONSUMPTION OF LONGPPL
In Table 12, we test the time cost of LongPPL. It can be observed that the time cost of calculating LongPPL. using the 8B model the evaluator is approximately 3-4 times that of calculating PPL., while the overhead for using the 72B model is much higher.
Although the computational overhead of LongPPL is non-negligible, believe that such compu- tational cost will not have substantial impact on the practicality of LongPPL. On the one hand, users employ LongPPL. as benchmark, key tokens can be calculated offline, resulting in no online computation overhead. On the other hand, if LongPPL is used as an evaluation metric during train- ing, its computational overhead is negligible compared to the overall training cost (as evaluation steps are typically sparse during training).
Table The time consumption LongPPL.. The values in the table represent the average seconds required per sequence.
PPL
LongPPL (Llama-3.1-8B)
LongPPL
Mistral-7B
2.8
11.3 (+8.5, +304%)
56.4 (+53.6, +2014%)
Mixtral-8x7B (47B)
4.2
13.5 (+9.3, +221%)
58.4 (+54.2, +1390%)
20

---

Published as a conference paper at ICLR 2025
C RELATED WORK
Long-context Modeling. Due to practical demands, numerous recent works have emerged that aim to enable large models handle long contexts through improvements in architecture algorithms. One mainstream direction the study of positional encodings with length extrapolation capabilities, including Alibi (Press et al., 2021), xPOS (Sun et al., 2023), Kerple (Chi et al., 2022), and various RoPE (Su al., 2024) variants (Chen et al., 2023; Zhang et al., 2024c; Chen al., 2024a; Xiong et al., 2024; Peng et al., 2024). Others pay more attention to architecture improvements, using sparse attention mechanisms prevent models from atlending to overly long sequences (Han al., 2023; Xiao et al., 2024; Chen et al., 2024b; Ding et al., 2023), or exploring the use of recurrent mechanisms compress and store key information from long texts, thereby effectively increasing the context window (Zhang et al., 2024a; Bulatov et al., 2023; Martins et al., 2022).
Long-context Evaluation. Recent studies have introduced several benchmarks to evaluate the long- context performance in downstream tasks. A widely used type of benchmark is retrieval-based (Kamradt, 2023), passkey-retrieval (Mohtashami &
synthetic task, including needle-in-a-haystee Jaggi, 2023) and LongEval (Li et al., 2023a). Some evaluation suites have also been gradually introduced, such as LongBench (Bai et al., 2023b), RULER (Hsieh et al., 2024), ZeroSCROLLS (Shaham et al., 2023), including document question answering, summarization, few-shot learning, code completion, and other synthetic tasks, thereby offering a more thorough evaluation of model's long-context abilities. To further enhance the context length of the evaluation data, InfiniteBench (Zhang et al., 2024b) has introduced evaluation data exceeding 100K tokens. In this paper, we analyze the correlation between the Perplexity metric and specific evaluation tasks and propose an alternative LongPPL metric, which can better align the model's long-context performance on downstream tasks.
Re-weighting methods in language model training. Re-weighting methods for language model training have been extensively studied, with a focus on enhancing model performance (Lin et al., 2024), improving training efficiency (Clark et al., 2022). and addressing token imbalance (Luo et al., 2023: Hu al., 2023; Gu al., 2020: Wang et al. 2020). Many works have also explored re- weighting through data selection techniques, addressing a wide range of challenges such as data quality (Li et al., 2023b), data diversity (Liu et al., 2023), and distribution matching (Li et al., 2023c; Ni et al., 2024). However, few of these works focus on re-weighting tokens to enhance a model's long-context performance. The most recent and closely related work ours is LengRecipe (Hu al., 2024b), which re-weights tokens based on distribution shifts in model predictions during training. This approach does not capture the essential characteristics of key tokens. In contrast, our method directly re-weights tokens according to their dependence on long-context information, providing more fundamental and targeted solution.
21

---

Published as a conference paper at ICLR 2025
D MODELS
The models used in this paper are shown in Table 13.
Table 13: Information of the models used in this paper.
Model
Size
Context Length
Huggingface
Llama-2-78 (Touvron al., 2023)
7B
4K
meta-Ilama/l 1tap://ma/llaaa-22111111 Llama-2-7h-hf
Llama-2-13B (Touvron et al., 2023)
Seta-Ilama/Llamme 1aattua--aaaa----111 2-13bhhf
Alama-3.1-8B (Dubey et al., 2024)
8B
128K
meta- meta-Ilama/L
Mixtral (Jiang al., 2024)
8x7B
32K
1istral-v0..1 (Jiang al., 2023)
8K
mistralai/Miitral-78--001
Mistral (Jiang et al., 2023)
32K
Mistral Large 2 (Jiang et al., 2023)
123B
128K
Qwenl 5 (Bai al., 2023a)
14B
128K
Qwen/Qwenl.5-14BB
Qwen2-7B (Yang et al., 2024)
128K
Qwen/Qwen2-7B
Qwen2-72B (Yang et al., 2024)
72B
128K
Oween/Qwenn-722-IInsttuu
HILM (An al., 2024)
7B
32K
In2Training/HII.M-71
Phi-3 (Abdin al., 2024)
128K
-errooffffhi--mmmllll------nnm 128k-intruet
CLEX (Chen al., 2024a)
64k
Yi (Young et al., 2024)
6B
200K
01-ai/Yi-6B-200 6OOK
Yarn (Peng al., 2024)
128K
22

---

Published as a conference paper at ICLR 2025
E DEMONSTRATION FOR THE SELECTED KEY TOKENS
Demonstration for the selected key tokens in GovReport
Even though it has reimposed all U.S. sanctions on Iran, the Trump Administration has issued some exceptions that are provided for under the various U.S. sanctions laws, including the following: As noted above, on November 5, 2018, eight countries were given the SRE enable them to continue transactions with Iran's Central Bank and to purchase Iranian oil. At an April 10 hearing of the Senate Foreign Relations Committee, Secretary Pompeo appeared to indicate that the SREs would be renewed. However, on April 22 the Administration announced termination of the SREs as of their expiration on May 2019. On May 3, the Administration ended some waivers under IFCA and various antiproliferation laws (discussed above) that allow international technical assistance to Iran's 's three nuclear sites permitted to operate under the JCPOA-the Fordow facility, the Bushehr nuclear power reactor, and the Arak heavy water plant. The Administration ended the waiver that enabled Rosatom (Russia) to remove Iran's 's LEU that exceeds the 300kg allowed stockpile, and that allowed Iran to export heavy water that exceeded the limits on that product to Oman. The waiver limitations also will prohibit the expansion of the Bushehr reactor by any supplier. In response, President Rouhani announced that Iran would no longer abide by the JCPOA stockpile limits. The Administration waived Section 1247(e) of IFCA to enable Iraq to continue paying for purchases of natural gas from Iran. The waiver term that section up to 180 days, but the Administration has been providing the waiver for 90-day increments. The Administration has issued the permitted IFCA exception for Afghan reconstruction enable India continue work at Iran's 's Chahbahar Port. A U.S. State Department official told Afghan leaders in mid-May 2019 that the exception would continue. The Administration has renewed the licenses of certain firms enable them continue developing the Rhum gas field in the North Sea that Iran partly owns.
The JCPOA did commit the United States to suspend U.S. sanctions on Iran for terrorism or human rights abuses, on foreign arms sales to Iran or sales of prolifcration-sensiive technology such as ballistic missile technology, or on -.S.-Iran direct trade (with the selected exceptions of the latter discussed above). The sanctions below remained in place during JCPOA implementation and remain in effect trade with and investment in Iran; E.O. 13224 sanctioning terrorism entities, any sanctions related to Iran's 's designation sanctions. The JCPOA does not commit the United States to revoke Iran's 's placement on the terrorism list; E.O. 13382 sanctioning entities for proliferation; the Iran-Iray Arms Non-Proliferation Act; the Iran-North Korea-Syria Non-Proliferation Act (INKSNA); the section of ISA that sanctions WMD- and arms-related transactions with Iran; E.O. 13438 on Iran's interference in Iraq and E.O. 13572 on repression in Syria; Executive Orders (E.O. 13606 and E.O. 13628) and the provisions CISADA, ITRSHRA, and IFCA that pertain to human rights democratic change in Iran; all sanctions the IRGC, military, proliferation-related, and human rights- and terrorism-related entities, which were "delisted" from sanctions; Treasury Department regulations barring Iran from access to the U.S. financial system. Foreign banks can pay Iran in dollars out of their existing dollar supply, and the Treasury Department revised its guidance in October 2016 to stress that such transactions permitted.
E.O. 12959, the ban U.S.
state sponsor terrorism, and any other terrorism-related
23


Reviews:
[{"review_id": "8RNQNi7YfX", "reviewer": "Reviewer_aS6q", "summary": "This paper modified standard PPL and CE (cross entropy) loss, by weighting more to key tokens in long text. The key tokens are identified through a long-short context constrastive method. The experiments demonstrate the LongPPL correlates long-context capabilities of LLM, thus a good metric. Similarly, applying LongCE loss in long-context fine tuning shows improvement on long-context benchmarks.", "questions": "1. I wonder how would fine tuned model perform on normal context length benchmarks, will it remain roughly at the same level or regress a bit?\n2. How long is considered as \"long-context\". In experiments, 32k is used. I wonder how much this method would affect on other context lengths.\n3. can we use equation (5) directly for LongCE instead of (8). so only differentiate on key tokens, would that be doable?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is simple and seems working well.", "weaknesses": "n/a", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bLqUdYtrO2", "reviewer": "Reviewer_kFZS", "summary": "The paper critiques the use of perplexity (PPL) as a standard metric for evaluating large language models (LLMs) on long-context tasks, arguing that PPL’s averaging across all tokens fails to capture the importance of \"key tokens\" critical for understanding long-context dependencies. To address this, the authors introduce Log Probability Gain (LPG) and Log Probability Values (LPV) to identify these key tokens, leading to LongPPL—a refined metric that emphasizes these crucial tokens, thus providing a better measure of long-context performance. Additionally, they propose LongCE, a re-weighting strategy applied during fine-tuning that bolsters the model's ability to handle long-context tasks through an expectation-maximization (EM) approach. Although LongCE adds computational overhead, the authors show that it consistently outperforms traditional cross-entropy (CE) across various settings.", "questions": "1. Is the main reason perplexity fails due to its inability to correctly identify answer/key tokens?\n2. Could you clarify what \"task-aware\" means in this context? Does it refer to distinctions between short-context and long-context tasks, or could it apply to other task definitions as well?\n3. You mention that \"even though the Llama-3.1 model we use is only 8B in size, it still obtains LongPPL with a strong correlation, suggesting high compatibility with powerful long-context models, even with smaller parameter sizes.\" Are you implying that LongPPL would show higher correlation with larger models? If so, could you explain why?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper conducts a fine-grained analysis to investigate why perplexity (PPL) fails to capture long-context capabilities in LLMs. Few previous studies focus on re-weighting tokens to improve long-context performance; this work addresses this directly by re-weighting tokens based on their dependence on long-context information, providing a more fundamental and targeted solution.", "weaknesses": "Some settings in the figures lack clear explanations, which affects the reliability of the figures and the conclusions drawn from them. For example, in Figure 2, the specific prompt lengths for each data point are not provided, making it difficult to interpret the statement, \"Each point represents the results obtained from testing at a specific prompt length.\"\nAlthough the correlation arguments seem promising and qualitatively sound, they would benefit from added statistical rigor, such as p-values from hypothesis testing to confirm the significance.\nThe naming of Log Probability Gain (LPG) and Log Probability Values (LPV) is not intuitive, which may hinder understanding and adoption of these concepts.\nAdditionally, the experiments are conducted with a sequence length of 32K, a length considered relatively manageable by recent research. Testing on longer sequences could introduce challenges that may not align with current trends, potentially impacting the generalizability of the results\nLastly, the criteria for selecting experimental models are not clearly defined, which introduces some ambiguity regarding the rationale behind the choice of models used in testing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oImn5J9uZz", "reviewer": "Reviewer_KLF8", "summary": "The authors state that perplexity metrics (PPL) are insufficient for evaluating the long-context capabilities of large language models (LLMs) because PPL considers both answer and non-answer tokens, resulting in perplexity scores that do not correlate with LongBench scores. To address this, the authors propose two new metrics, LongPPL and LongCE, which extract key tokens from the model output with the help of a relatively strong model as the evaluator. LongPPL is used as an evaluation metric for LLMs in long-context text, while LongCE serves as a loss function when fine-tuning LLMs on long-context tasks.", "questions": "- How does the performance of LongPPL and LongCE change when using a weaker model for key token identification? Have you tried using the same model for both evaluation and key token identification?\n- The paper discusses the overhead of using LongCE in LLM fine-tuning compared to the original Cross Entropy. How does the evaluation time of using LongPPL compare to traditional perplexity (PPL) metrics?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The introduction of LongPPL and LongCE represents an innovation in the evaluation and fine-tuning of LLMs for long-context tasks. These metrics address the limitations of traditional perplexity and Cross Entropy loss, providing more accurate and task-relevant assessments of model performance.\n- The concepts presented by the authors are easy to understand, including the motivation and method. The figures greatly help in understanding the concepts and the experiments.", "weaknesses": "- LongPPL metric relies on a relatively strong model (medium-sized Qwen2-72B-Instruct) as an evaluator to identify key tokens. This dependence may limit the applicability of the metric in scenarios where such a strong model is not available or practical to use.\n- The additional computational cost associated with identifying key tokens may still be significant. This could be a barrier to adoption in resource-constrained environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a1bt8j6lPX", "reviewer": "Reviewer_At6Q", "summary": "This paper proposes a new metric LongPPL and a new training objective LongCE. The LongPPL is calculated over selected tokens that satisfy the following conditions when evaluated with an evaluator model: (a) the tokens have higher probability conditioned on long context than short context (b) the tokens are predicted with high probability. Results show that the proposed metric better correlates with long-context downstream tasks than standard perplexity. The proposed LongCE loss adapts the standard focal loss by reweighting the tokens based on their performance given long and short context. Results demonstrate that models finetuned with LongCE achieve better long-context downstream task performance.", "questions": "- Why does finetuning with LongCE demonstrate such different results for mistral-7b and llama2 on RULER, with one hardly improving (<1) and one significantly improving (~10)?\n- Why does figure 6 only contain correlation between longPPL and longbench score? What is the correlation between standard PPL vs longbench score?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a new metric LongPPL which better correlates with long-context downstream performance.\n- This paper proposes a new training objective LongCE tailored for improving long-context performance.", "weaknesses": "- The proposed LongPPL is limited by the evaluator model. Concretely, the validity of the evaluation is strictly bounded by the maximum effective length that an evaluator model can reliably handle. Additionally, the two thresholds, which depend on both the evaluation data and the evaluator model, can further complicate the evaluation process. These limitations weaken the universality of the proposed metric.\n- The key token selection of LongPPL is motivated by observing large gains in log probability of the ‘answer’ tokens in LongEval, a retrieval-based synthetic test. It raises the question of whether the gains primarily come from the improved prediction of less frequent n-grams that re-appear later in the context, as studied by [1][2]. If so, the motivation of leveraging an evaluator model is obscure since one can simply evaluate those re-occurred n-gram tokens, which are much more convenient to find and overall a more universal setup than LongPPL. \n- The LongCE loss should be compared with the standard focal loss [3] baseline to understand if it is really necessary to add the contrasting term of long and short context perplexity. The standard focal loss does not require extra forward pass for computing short context log probability, thus can potentially improve training efficiency. \n- The LongCE loss seems to improve mostly the retrieval-based test (LongEval), however, demonstrate little gains on RULER, which has a few non-retrieval tasks, and almost zero improvement on LongBench, which contains quite a few non-retrieval tasks (Figure 7). It is unclear to what extent the proposed method benefits non-retrieval tasks.\n\n\n[1] Zoology: Measuring and Improving Recall in Efficient Language Models\n\n[2] Do Long-Range Language Models Actually Use Long-Range Context?\n\n[3] Focal Loss for Dense Object Detection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "guF8y7Yhe3", "reviewer": "Reviewer_KQKC", "summary": "This paper first tries to understand why perplexity metric fail to reflect long-context ability of LLMs. For instance it has been shown that perplexity of LLMs shows almost no correlation with their long-context performance measured on a dataset such as Longbench.\n\nAuthors show the reason is because perplexity is computed over all tokens of the answer to a query, wether this tokens are ‘key’ or not to answer to the query. Consequently, perplexity computed equally over all tokens do not represent long-context performance. Therefore,  computing ppl by averaging only on key tokens (instead of all tokens)  avoid this dilution and correlates better with long context benchmarks scores such as the ones measured on Longbench.\n\nA longPPL measure that takes into account this observation is proposed by the authors, where they compute perplexity by averaging solely on the selected key tokens.\nBut how do the authors extract the ‘key’ tokens from the natural text in the answer to a query ? \n This is done by introducing a simple and elegant metric which consists in computing the difference - for each token x - of the probability p(x/long_context) and p(x/shortened_context). This metric called LPG score, measures the improvement of prediction accuracy induced only by the long-context => a high value indicates that long context plays an important role in the prediction of x, so x is a key token and should be taken into account to compute LongPPL (and it should not be taken into account - for compute the PPL - otherwise). \n\nBuilding on this finding, they develop an efficient long-context training strategy by prioritizing key tokens. Specifically, they introduce the LongCE (Long-context Cross-Entropy) loss, which assigns higher weights to key tokens, identified dynamically by the model itself.\nExperiments across various LLMs demonstrate that fine tuning a model using the LongCE  loss consistently outperforms traditional CE loss, across multiple LLMs and multiple long context benchmarks.", "questions": "-computed longPPL and longCE (as opposed to PPL) must be more costly; how much more costly (especially a you need to compute the LPG score for each token at training time) ?\n\n-do you fine tune with longCE only or with a combination of CE and longCE ?\n\n-what about using the needle-in-the-haystack test for your evaluation of long context models ?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "-Paper very clearly presented and very progressive, starting from identifying a problem, explain why it occurs and propose an efficient solution for it\n\n- convincing explanation on why PPL is not a good indicator of quality for long context LLMs (with illustrations on a long context benchmark)\n\n-Fine tuning with longCE loss leads to improvements on several long context benchmarks and for several LLMs ; very convincing results", "weaknesses": "-I don’t fully agree this the claim « the evaluation of long-context capabilities still heavily relies on perplexity (PPL) as the de facto metric » as we see now most papers relying on long-context benchmarks or use the needle-in-the-haystack test to evaluate long-context LLMs\n\n-The reference to ppl is not the good one (Bengio did not invent perplexity!), you should credit people who actually were the first to introduce this metric. Perplexity was originally introduced in 1977 in the context of speech recognition by Frederick Jelinek, Robert Leroy Mercer, Lalit R. Bahl, and James K. Baker. See https://en.wikipedia.org/wiki/Perplexity\n\n-I’d like to learn more about the computational implications of calculating LongPPL and LongCE, especially since the LPG score needs to be computed for each token (see my question below). I'm particularly curious about the costs associated with LongCE, as described in Section 3.2. While the Implementation details of LongPPL in the Appendix were helpful, I still want to understand more about how computationally expensive this process is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)