Long-context LLMs Struggle with Long In-context Learning
Authors: Tianle Li
Ge Zhang
Quy Duc Do
Xiang Yue
Wenhu Chen
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) have made significant strides in handling long sequences. Some models like Gemini could even be capable of dealing with millions of tokens. However, their performance evaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their true abilities in more challenging, real-world scenarios. We introduce a benchmark (LongICLBench) for long in-context learning in extreme-label classification using six datasets with 28 to 174 classes and input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprehend the entire input to recognize the massive label spaces to make correct predictions. We evaluate on 15 long-context LLMs and find that they perform well on less challenging classification tasks with smaller label space and shorter demonstrations. However, they struggle with more challenging task like Discovery with 174 labels, suggesting a gap in their ability to process long, context-rich sequences. Further analysis reveals a bias towards labels presented later in the sequence and a need for improved reasoning over multiple pieces of information. Our study reveals that long context understanding and reasoning is still a challenging task for the existing LLMs. We believe LongICLBench could serve as a more realistic evaluation for the future long-context LLMs.

Published in Transactions on Machine Learning Research (03/2025)
Long-context LLMs Struggle with Long In-context Learning
+ Tianle Li, Ge Zhang, Quu Duc Do, 'Xiang Yue, 4 Wenhu Chen
University of Waterloo
Carnegie Mellon University
Veetoor Institute, Toronto
Reviewed on OpenReview:
Abstract
Large Language Models (LLMs) have made significant strides in handling long sequences. Some models like Gemini could even be capable of dealing with millions of tokens. However, their performance ovaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their true abilities more challenging, real-world scenarios. We introduce benchmark (LongICLBenchh for long in-context learning in extreme cateeme-label classification using six datasets with 28 174 classes and input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprchend the entire input to recognize the massive label spaces to make correct predictions. We evaluate on 15 long-context LLMs and find that they perform well on less challenging classification tasks with smaller label space and shorter demonstrations. However, they struggle with more challenging task like Discovery with 174 labels, suggesting a gap in their ability to process long, context-rich soquences. Further analysis reveals a bias towards labels presented later the sequence and need for improved reasoning over multiple pieces of information. Our study reveals that long context understanding and reasoning is still challenging task for the existing LLMs. We believe LongICLBench could serve more realistic evaluation for the future long-context LLMs.
BANKING77
DialogueRE
Discovery
80
30
LLaMA 2.78 LongLara
ChalGLMS 6B 32K
Accuracy (%))
60
60
200K
InterLLM2 Ba-base
1.5-76-base
-0
40
10
Mesrai-7B-v0.22 78--0.2-bbe-- 2-base
Gomma 78 base
20
20
20
RWKV-S-World
Mamba-2 2 8B
GPT4-luroo
CPT4u
2K
4KC
7K
3K
14K
13K
19K
25K
32K
10K
20K
3CK
40K
50K
Claude3--puss
Context Token Length
Context Token Length
Context Token Length
Gemini-1 1.rinn--1.-Proo
Easy
Hard
Figure 1: LLM performance on long in-context benchmark across different lengths. We curate datasets with different difficulty levels. As we incrcase the difficulty of the datasct, LLMs struggle understand the task definition and suffer from significant performance degradation.

---

Published in Transactions on Machine Learning Research (03/2025)
1 Introduction
Large language models have already entered the long context era. A myriad of LLMs has been released to
support long context windows from 32K to 2M tokens. These methods (Hao et al. 2022 Chen et al. 2023a
Peng et al. 2023b Ratner et al. 2023 Xiao et al. 2024 Jin et al. 2024] can unlock lots of complex real-world
applications, such as long-document question-answering multi-document summarization, long-horizon agent
lasks, and repo-level code understanding.
One line research based on AliBi (Press et al. 2022 and RoPE (Su et al.
2024 embeddings, which
allows us to train Transformers with short sequences and subsoquently apply them to longer sequences
during inference. Recently, different approaches (Xiong al., 2023: Fu et. al.
2024 Liu et. al. 2024 help
the model to extrapolate to 128K window size with continued pre-training. Later on, LongRoPE (Ding
et al. 2024 was proposed to further extend the context window to 2M tokens. Another line of research also utilizes methodologies like context window sliding and segmentation to overcome the issue of the limited context window in original Transformers (Hao et al. 2022; Ratner et al., 2023). Furthermore, architectural innovations, transitioning from traditional Transformer- based designs to recurrent models or state space models, have shown promise in facilitating long-range computations naturally (Orvieto al.) 2023 Gu &
Dao 2023: Peng al.) 2023a). These techniques have been incorporated into several current open-source
LLMs to enhance long sequence undorstanding capability (Chen et al. 2023b Tworkowski ct al. 2023
These long-context models are primarily evaluated on three types of evaluations:
1. language model perplexity over long documents, which is used by most papers.
2. passkoy retricval (Mohtashami & Jaggi 2023 Chen et al.) 2023a Li et al. 2023a or ncedle-in-a-
haystack (Team el al. 2023 Fu et al. (202)). which requires reciting a randomly inserted information in a long sequence. Several LLMs achieve 99%+ on this synthetic task.
3. long-document question-anse answer or summarization over Qasper (Dasigi et al. 2021
Evaluations (1) and (2) only provide minimum bar for LLMs to pass, but their results cannot reflect LLMs' true ability to deal with realistic long-sequence tasks. Evaluation (3) provides a more realistic metric,
There is an important info hidden inside lot irrelevant text. Find it and memorize them. will quiz you about the important information there. <prefix filler by continuously repeating: The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again.> The pass key is <PASS KEY>, Remember it.
In-context Prompt:
Dialogue: Speaker 1: Hey Rach, talk you outside for second?
Speaker I: Ah well, can't blame guy for trying!".
Predict the relationship between the following entity pairs:
Speaker and Rach, Speaker and Phoebe, Speaker and Speaker
Answer: peraltemate names, perrroomatt,, per:girl/boyyrrcee
<PASS KEY> the pass key.
Dialogue:
Speaker 1:
You've got to get back out there, it's your party......
<suffix filler>
just 'nSpeaker 3: Didn't you like, just get your eyes phecked I: Well yeah, but, you know, uh, 27
What is the pass key? The pass key is
dangerous eye age.",
Predict the relationship between the following entity pairs:
(a) Passkey Retrieval
Speakerl and 27, Speakerl and Speaker 2, Speaker 2 and opthamologists
Answer: per:age, per:positive impression, per:tille
Passage: Mark Hunter (Slater), high school student sleepy suburb of Phoenix, Arizona, starts an FM pirate radio station that broadcasts from the basement of his parents' house. Mark loner, outsider.....
Dialogue: Speaker 1: Hello?inSpeaker 2: Joey just called. He's got
courtside Knicks tickets for him and me tomorrow night ..... 'nSpeaker 2: Yeah, ah, ah.. I'll think of something."
Predict the relationship between the following entity pairs: Speaker and Speaker 2, Joe and Knicks, Speaker and restaurant
Question: Who is Mark Hunter? Answer: He is high school student in Phoenix
Answer: per:spouse, per:positive impression, perpplae of work
(b) Long-document Question-answer
(c) Extreme-label In-context Learning
Comparison extreme-label ICL with existing evaluation tasks. Passkey Retrieval is a synthetic task. Long-document Question-answering does not require reading the entire document to find the answer. In extreme-label TCT, the model needs to scan through the entire demonstration to understand the whole label space to make the correct prediction.
Figure 2:
2

---

Published in Transactions on Machine Learning Research (03/2025)
Open-source Models
API-based Models
BANKING77: 9K
LLaMA-2-7B-32K
BANKIN077:99
GP T4-tuubb
80
80
Long-LLaMA-code-77 7B
GPT4o
60
Mistral-7B-40.2-ben
Accuracy (%)
DielogRE:
Claude3-Opus
Accuracy
TRCRED: 18K
Owen--15-7B-bbae
Cemini-1.5-Pro
40
RWKV-5-World
40
DielorRE: 32K
Mamba-2.8B
TacRED: 18K
20
Discovery 41K
Discovery:411
0
10000
20000
30000
40000
10000
20000
30000
40000
Context Token Length
Context Token Length
Figure Results for representative models across different evaluation datasets. The performance greatly decreasos as the task bocomes more challenging.
however, these tasks more focused retrieving correct information from the long input. In question answering, LLMs can take shorteut to read short snippet to predict the answer without reading the entire document as demonstrated in Figure case case (b). Similarly, summarization also suffers from the strong position bias, where LLMs can utilize the few leading sentences (Nallapati et al.] (017)) to achieve high performance. Therefore, these metrics are insufficient to measure LLMs' ability to comprehend and reason over the entire input sequence.
In this paper, we propose adopt in-context learning (ICL) on extreme-label classification tasks (Anil el al. 2022 Milios et al. 2023) to evaluate long-context LLMs. Unlike the prior tasks, in-context learning requires LLMs to recognize the task by scanning over the entire input to understand the label space. This task necessitates LLMs' ability to comprehend the entire input to make predictions. Due to the massive label space, the task demonstration could easily become a long sequence. For example, Discovery (Sileo et al., 2019) encompasses 174 classes with each example taking an average 61 tokens. Therefore, the minimum total demonstration length (1 shot per class) alrcady exceeds 10K tokens. Normally, LLMs demand more than 1 shot per class to understand the nuances of different fine- iie-grained labels. Having multiple shots can significantly extend the total demonstration length to above 32K. Therefore, this task becomes natural testbed for long-context understanding.
To systematically assess how these extended input capabilities affect model performance in the realm of fine-grained text classification with in-context learning, we have compiled a benchmark, i.c. LongICI.Bende consisting six carefully-selected tasks with different difficulty levels in terms of context length and label space. We evaluate the performance of a wide range of long-context LLMs and find that the performance the open-source models uniformly dips as the task becomes more complex (e-g. requiring longer demonstration) shown in Figure 3. Among the open-source models, the non-Transformer- based models, like RWKV and Mamba (Peng et al., 2023a Gu & Dao, 2023), perform far behind the Transformer-basee models. Simultaneously, within a task, most of the models can benefit from the extensive demonstration if the length is within a certain range. As the input grows longer, it either hurts or makes the performance fluctuate as shown in Figure [
On the most difficult extreme-label classification task Discovery (Silco et al. 2019), all LLMs achieve close-to- zero performance except Gemini-1.5-Pro with 14% accuracy. In contrast, fine-tuned BERT model (Kenton) & Toutanova 2019) can achieve 87%. This highlights the challenges that the long in-context learning pose for the existing L.Mss. Moreover, we make further analysis on the distribution of label position to investigate the factors that affect the long in-context learning capability of these models. It is shown that the position distribution of instances in the prompt can dramatically influence the performance of some of the evaluated models.
In a nutshell, our contributions to this work can be summarizod as follows:
1. We have identified in-context learning on extreme-label classification tasks as an ideal testbed for the evaluation of the long-context capability of the current LLMs. We developed LongICI.Bench, which serves as a complement carlier benchmarks that concentrated on tasks like long document summarization, question answering (QA), retrieval, focusing instead on long in-context learning.
3

---

Published in Transactions on Machine Learning Research (03/2025)
2. We evaluate line of recent long-context LLMs on LongICLBench and reveal their performances with gradually changed difficulty levels. Simultaneously, we find the sensitivity of some of the long-context. LLMs regarding instance position in the prompt. We hope the evaluation results can provide more insights for the improvement of the design of long-context large language models.
2
Related Work
Long In-context Learning on LLMs As pre-trained language models continue grow in size, in-context learning (TCL) has emerged as favored approach for addressing a wide array of tasks without the need for extensive fine-tuning (Dong et al. 2023). A body of research has established that increasing the number of examples demonstrations can enhance ICL performance (Liu et al. 2022 Wu et al.) 2023). Nonetheless, there are studies indicating that longer input prompts can actually diminish performance (Liu et al. 2023): with the effectiveness of prior large language models (LLMs) being constrained by the maximum sequence length encountered during their training. It is also claimed in previous works that LLM+ICL falls short on apecification-heny tasks due to inadequate long-text understanding ability (Peng et al. 2023c). To counter this issue, various works have introduced memory augmentation and extrapolation techniques to support ICL with an extensive set of demoosttationns [Li et al. 2023c Wang et al. 2023
Long Context Techniques over LLMs The effectiveness of Transformer baseed models is hindered by the quadratic increase in computational cost relative sequence length, particularly in handling long context inputs. Recent efforts have explored various strategies to address this challenge. Some studies have pursued continued fine-tuning of the LLM with longer context inputs (Rozière et al. 2024 Tworkowski et al. 2023). Others have leveraged position extrapolation interpolation, building upon relative rotary positional embedding (Su et al. 2021), to extend input length beyond the training phase (Press et al. 2022
Chen et al. 2023ap. Additionally, ly, more approaches have been proposed to mitigate computational issues, including sliding memory window and chunk segmentation (Hao et al.) 2022 Ratner et al., 2023 Zhu et al.
2024). Furthermore, alternative architectures beyond Transformer have been explored to handle long inputs more naturally, such as selective-state stace-ppcce models (Peng et al.) 2023a Gu Dao 2023). These diverse approaches claim that they can enhance the capabilities of LLMs in processing long context inputs more efficiently.
Long Context Evaluation Due to the imperious demands for the support of long-range LLMs, there is a series of benchmarks focusing on long context evaluation. Long- Bang-Range Arena (Tay et al. 2021 includes tasks consisting of sequences ranging from 1K to 16K tokens to evaluate variations of fast Transformers. LongBench (Bai et al., 2023b) comprises 21 bilingual datasets with an average length of around 6k words, which have been processed in a unified format to enable effortless evaluation. L-Eval Benchmark (An et al.) 2023) supports 20 sub-tasks with input lengths of 3K to 200K tokens. LooGLE (Li et al., 2023b) focuses summarization and long dependency QA tasks with test instances exceeding 100k words. Most recently, oo ocBench (Zhang et al. 2024) encompasses 12 tasks with an average length of 200K tokens. Another recent. work explores the impact of extending input lengths, especially on reasoning tasks @Levy et al. 2024
Extreme-label Classification Extreme-label Classification involves categorizing data into one of an extremely large number of labels, and finds application across a variety of real-world domains such as emotion classification, named entity recognition, and biological function prediction, each requiring precise differentiation among vast label spaces (Zhang al. 2017 Sileo et al.) 2019 Demszky al. 2020: Ding et al. b021b. Previous methods to tackle Extreme- label Classification tasks range from embedding based approaches to fino-tuned retrievals (Bhatia et al. 2015; Vulić et al., 2021). However, integrating this task with long-context large language models presents unique challenges. The large scale of the label space complicates the in-context learning process, where LLMs are expected to discern fine-grained differences among labels based on extensive context (Milios et al.) 2023). These challenges make the proposod LongICI.Bench with a range of difficulty levels a good testing scenario evaluate the capability of long-contex large language models.
4

---

Published in Transactions on Machine Learning Research (03/2025)
Table Statistics of the collected sub-dataset in LongICI.Bench. We evaluate from 1-shot /label to 5- shot /label, which results in the shown #total token range.
Dataset
Task Type
# Classes
# Tokens /Shot
Total Tokens
GoBmotion
Emotion Classification
28
[IK, 4K]
BANKING77
Intent Classification
77
28
[2K, 11K
TacRED
Relation Extraction
41
[4,, 18K]
Few-NERD
Entity Recognition
66
61
[5K, 23K
DialogRE
Relation Extraction
36
226
[8K, 32K]
Discovery
Discourse Marker Classification
174
61
[1KK, 50K]
3
Long In-context Evaluation
3.1 Long In-context Benchmark
To support the evaluation of long in-conlext learning on extreme-label classification lasks in different domains and various difficulty levels, collect six datasets containing context length from short to long. In order to balance the sequence token length within each dataset and the goal of evaluation for long in-context learning, we keep a subset of the classes among all the classes to format evaluation sets around round, 2 rounds, rounds, 4 rounds, and rounds correspondingly, where each round represent a complete set of examples containing all unique chosen labels. We sample the number of instances from each of the classes evenly to reduce the bias resulting from the label distribution. The statistics of the datasets are described in detail in Table
GoEmotions (Demszky et al. 2020 is the largest manually annotated dataset of 58k English comments from Reddit, which is labeled into 27 emotion categories or Neutral. Each selected example contains 28 tokens on average.
Casanueva et al. 2020 a banking-domain intent detection dataset comprising 13,083 annotated examples over 77 intents. We keep all of the types of intents, and each of the instances contains around 28 tokens.
BANKING77
TacRED (Zhang et al. 2017 is a large-scale relation extraction dataset with 106,264 examples built over news and web text. Only one relation is labeled for each of the sentences in the dataset. It covers 41 relation types in total, with an average length of 80 tokens for each example.
Few-NERD Ding et al. 2021 is a human-annotated name entity recognition dataset with a hierarchy of coarse-grained and 66 fine-grained entity types. Each of the instances is a paragraph with about 61 tokens on average and contains one multiple entity names as the ground truth answer.
DialogRE Yu et al. 2020 is human-annotated dialogue- based relation extraction dataset from American television comedy, Friends, with 36 possible relation types existing between an argument, pair in a dialogue. Each example contains an average of 226 tokens.
Discovery (Sileo et al. 2019) automatically discovers sentence pairs with relevant discourse markers and forms dataset containing 174 discourse markers with least 10K examples each. Each example contains around 61 tokens. This dataset is the most difficult task with fine-grained labels.
3.2 Model and Experimental Setup
In the exploration of in-context learning for extreme-label classification, we conduct comprehensive evaluation of popular open-source long-context language models of size around 7B parameters. We also include SoTA models like Semmini-1.--Pre investigated, highlighting the innovations in their architecture specifically for doaling with long context. We can observe that there are multiple strategies adopted extend the context window. Some of the models support the training context window size while some models support length extrapolation. RWKV (Peng
Claude3-Opus and GPT-4-turbo. [Table 2) provides an overview of the models

---

Published in Transactions on Machine Learning Research (03/2025)
et al. 2023a) and Mamba (Gu & Dao 2023) are the two new RNN-like architectures to decrease attention complexity which would allow the model to easily extrapolate to much longer inputs with linear time/memon complexity.
Table 2: The overview of the evaluated models. We utilize base models before Instruction-tuiing except
API-based models. LF means fine-tuning the model on longer-context corpus after pre-training.
Model
Size
Initialization
Strategy
Train
Support
Gemma- -B-base
7B
Gemma
RoPE + LF
8K
8K
2-M-A-2-BB-22
7B
LLaMA-2
Position Interpolation
32K
32K
CLa3GLM336B--3
6B
ChalGLM
Position Encoding Scheme
32K
32K
Qwen- 1.5-1.55..-1 1---7B-bse
7B
Qwen
NTK-Aware Interpolation
32K
32K
Histral-7B-vvle basee
7B
Mistral
L,F
32K
32K
L.aMAA-2-B--Loo L-.----ooooo..rr
7B
TLaMA-2
Shifted Short Attention
100K
100K
Yi-6B-2001
6B
Yi
Position Interpolation +LF
200K
200K
InternL 71-22----10 base
7B
InternLM
Dynamic NTK
32K
200K
Long MaaaaaMM---dde---e 7B
7B
LLaMA-2
Focused Transformer
8K
256K
RWKV-5-World
3B
RWKV
Attention-fres Model
4K
Mamba 2.8B
2.8B
Mamba
State Space Model
2K
00
GPT4-turbo
GPT-4
128K
GP
GPT-4
128K
Clande3- O-pus
Claude3
200K
Gemini- L.5-ni-L.5--Pro
Gemini
10M
We construct a prompt following the template as shown in A.2 for cach of the datasets. To fairly evaluate the open-source and API-based models with a series of input. lengths, we sample the same example set for all the models with labels distributed evenly to ensure an unbiased distribution for the in-context demonstration, For instance, an input of one round will include one set of examples traversing all the types, and 5 rounds will contain instances from each of the labels 5 times. For testing, sample 500 examples from the test set of each dataset, simultaneously ensuring models are loaded from the weights in Hugging Face] and inferred on eight NVIDIA RTX A6000 GPUs, while the API-based models are based on the official docunentations
even distribution in terms of the type of labels. All the open-source
3.3 Experiment Result
The main evaluation results are demonstrated in Table Table 4 Table Table Hand subsection A.1 For the entity recognition and relationship extraction dataset, Accuracy is utilized for the other datascts. From the presented results, generally, we can find that models of Transformer- architecture perform consistently better than the RNN-based ones in all the evaluated datasets. However, both of them are still falling behind the powerful API-based models. For a relatively simple task like BANKING77, whose context length from round to rounds is 2K to 14 K, most of the models can benefit from the extensive context with more demonstrations. As shown in Figure [ and Table B from 2K to 4K, there is either huge increase nearly doubling the accuracy, or a complete failure for most of the open-source models. After 3 rounds, limited performance gain can be achieved by adding more examples. When it comes to more complicated tasks like TacRED and DialogueRE in Table ! and Table 5 which are more urgently requiring the capability of long-context comprehension, the overall performance of all the few-shot models drops compared to BANKING77. As shown in the middle plot of Figure [ only GPT4-turbo and GPT4o can consistently benefit from more demonstrations, all of the other models reach their peak at the middle with context length around 13K to 25K.
the F1 score the evaluation metric, and
For the most challenging Discovery datasct, which has extremely large label space including 174 classes, one round of traversing for all the label possibilities has already made up context length of 10K. In this
https://huggingfacecco
openai
https://cloud.googe www.:ouud.goolg..com
6

---

Published in Transactions on Machine Learning Research (03/2025)
extreme case, all of the models except Gemini-1.5-Pre fail to tell the difference among the fine-grained types including GPT4-turbo, leading to score of 0. The results across different datasets reveal the models' capability to understand different types of tasks. Our initial hypothesis suggests that the strongest LLMs like GPT-4-turbo are capped at a certain complexity level between DialogRE and Discovery.
Another interesting observation we have that some L.LSs' performance on the extreme-label TCT. seems highly predictable. According to the left sub-graphi of Figure 3 the performance of Qwen and Mistral is almost linear w.r.t the demonstration length. This reveals that there might be an underlying mathematical relation between performance and the task complexity for ICL.
Table 3: BANKING77 result with respect to increasing context length. 1R represents one round of traversing the instances with a unique label.
Model
Param
Support
1R
2R
3R
4R
5R
Context Tokens
2K
4K
7K
9K
14K
Gemma- Baaama-7B-base
7B
8K
0
0
0
0
LLaMA-2-7B-32K
7B
32K
30.2
70.4
72.0
75.6
77.2
ChaiLLMB66D-32K
6B
32K
16.6
23.2
22.4
22.8
8.8
Qwen-1.5-7B-bas.
7B
32K
21.6
52.8
61.4
66.0
67.8
Mistral-7B-v0.2-baar
7B
32K
29.8
43.6
66.4
67.8
64.0
LLaMA-2--BB-LougLLorr
7B
100K
0
0
Yi-6B-200K
6B
200K
25.8
0
0
0
1.2
InternI.MM2-7B-basee
7B
200K
5.6
0
0
0
Long-JJaaAAcood-77
7B
256K
3.0
19.4
28.0
31.6
32.6
RWKV-5-World
7B
4K
8.6
21.2
0.4
0
0
Mamba-2.8B
2.8B
2K
0
0
0
0
0
GPT4-turbo
N/A
128K
73.5
80.5
82.0
83.5
84.4
GPT40
N/A
128K
80.8
79.8
81.2
71.2
71.4
Claude3- Opuss
N/A
200K
60.0
62.6
62.2
43.8
26.0
Gemini-LL5-Pro
N/A
10M
28.8
79.4
82.2
81.8
70.4
S6TA (RoBERTA + ICDA)
N/A
94.4
Table 4: TacRED result with respect to increasing context length.
Model
Param
Support
1R
2R
3R
4R
5R
Context Tokens
4K
7K
10K
14K
18K
Gemma- 7B-base
7B
8K
0.4
0.4
0
0
0
LLAMA-2-7B-322
7B
32K
0
0.4
0.4
0.8
0.4
ChaGGIMM3-6B-32K
6B
32K
29.7
36.1
38.9
40.1
25.2
Qwen-1.5-7B-bbae
7B
32K
38.7
17.3
45.2
43.6
40.6
Mistral-7B-v0.2-baer
7B
32K
53.3
53.1
51.6
48.0
42.3
1.AMA--2-7--Loglaae
7B
100K
0
0
Yi-6B-200K
6B
200K
5.6
1.9
8.0
9.5
2.0
InteraLM2-7B-luen
200K
29.6
27.2
15.5
10.7
8.0
7B
256K
3.8
7.1
4.1
6.6
4.9
RWKV-5-World
7B
1K
2.3
2.6
1.0
0
1.2
Mamba-2.8B
2.8B
2K
0
0
0
0
0
GPT4-turbo
N/A
128K
74.4
76.5
79.5
80.4
84.2
GPT40
N/A
128K
71.1
75.5
73.6
73.2
72.3
Claude3-Open Opuss
N/A
200K
68.7
74.1
35.4
43.4
44.3
1.5Pno-1.5-Prr
N/A
10M
72.6
81.4
79.6
81.4
82.3
SoTA (DeepStruct)
N/A
76.8

---

Published in Transactions on Machine Learning Research (03/2025)
Table 5: DialogRE result with respect increasing context length.
Model
Param
Support
1R
2R
3R
4R
5R
Context Tokens
8K
13K
19K
25K
32K
Gemma-7B-base
7B
8K
14.7
0
0
0
0
LLMMA-2-7B-3KK
7B
32K
6.6
13.5
6.0
5.4
5.5
ChatGLMB-66B33K
6B
32K
0.5
1.1
2.5
1.8
7.6
Qwen-1.5-7B- base
7B
32K
14.0
17.8
15.3
16.2
13.1
Mistral-7B-v0.22
7B
32K
24.0
23.0
23.2
22.0
21.1
LLaMA-2-7B-Longhh
7B
100K
0
0
0
0
0
Yi-6B-200K
200K
0.4
0.4
0
InteraLM2--B-base
7B
200K
12.0
13.2
5.8
1.8
0.7
Long-LLaaMA-cod-71
7B
256K
2.7
3.0
2.6
5.2
1.7
RWKV-5-World
7B
4K
0
0
0
0
0
Mamha-2.8B
2.8B
2K
0
0
0
0
0
GPT4-turo
N/A
128K
42.9
47.8
52.0
55.9
57.7
GPT4о
N/A
128K
40.6
11.5
41.0
17.3
45.3
Claude3-Opus
N/A
200K
16.8
30.3
15.3
0.8
0
Gemini-1.5-PP
N/A
10M
29.6
37.8
31.2
32.4
34.3
SoTA (HiDialog)
N/A
77.1
Table 6: Discovery result with respect to increasing context. length.
Model
Param
Support
1R
2R
3R
4R
5R
Context Tokens
10K
20K
30K
40K
50K
Gemma-7B-buse
7B
8K
LLaMA-2-7B-922
7B
32K
CharGL.M3-6BB32K
6B
32k
1.0
Qwen-1.5-7B-bass
7B
32K
O
Mistral-7B-v0.2--aas
7B
32K
LLaMA-2-7B-LLoggoor
7B
100K
0
0
0
0
Yi-6B-200K
200k
InternLMMP-TB-baaee
7B
200K
Long- -oodA-
7B
256K
RWKV-5-World
7B
4K
0.2
Mamba-2.8B
2.8B
2K
GPT4-turbo
N/A
128K
GPT4o
N/A
128K
2.8
0.8
0.8
0.6
0.4
Claude3-Opus
N/A
200K
1.2
0.6
Gemini-1.5-Pro
N/A
10M
14.0
6.0
3.2
1.8
2.8
SoTA (MTL)
N/A
87.4
4
Exploratory Experiment
Inspired by the Lost in the Middle phenomenon (Liu et al. 2023), we take analysis experiments to explore whether the position distribution of the instances will make difference in the performance for long in-context learning with extreme-label classification tasks.
4.1 Scattered Distribution
In our investigation, we conduct pilot experiments on TacRED, medium-complexitty dataset, with each label type demonstrated three times, resulting in total of 123 distinct instances (calculated 41 3).

---

Published in Transactions on Machine Learning Research (03/2025)
InternLM2-78 ba--ae-e-ee-eeeee---- Scatter
ChatGLM3-6B-3B Scatter
GPT4-turbo Scatter
100
100
u.a
100
Class Accuracy (%)
per:religion peragee persdate of birth
80
80
60
60
60
40
40
40
20
20
20
0
25
50
75
100
125
0
25
50
75
100
125
25
50
75
100
125
InternLM2-7B-basee Grouped
ChatGLM3-6B-32K Grouped
GPT4-turbo Grouped
100
100
100
perstitle
Class Accuracy (%)
80
80
80
60
60
60
40
40
40
20
20
20
0
50
75
100
125
25
50
100
125
25
50
75
100
125
Instance Position
Instance Position
Instance Position
Figure 4: Visualization of accuracy for every class when instances from the same class are scattered V.S. grouped in the demonstration prompt.
Within these experiments, instances bearing the same labels are distributed randomly to form a scattered configuration. For each instance, we track its relative position within the prompt alongside its corresponding label, thereafter computing the accuracy for each label class. As illustrated in the first row of Figure 4 the visualization delincates the accuracy of each label, aligned with its position within the prompt, where diverse colors symbolize various label lypes. In scenarios where class instances are scaltered, certain models, such as InternLM2-77 -B-bee, demonstrate acceptable performances approximately 60% accuracy merely on specific labels, as highlighted by red circle in Figure 4 regardless of the instance placements. Conversely, other models, like ChaiGLM3-6B-32K, exhibit robust. performance across broad spectrum of labels. Remarkably, the GPT4-turoo model consistently surpasses 80% accuracy threshold for the majority of label types, with only minimal count of exceptions.
4.2 Grouped Distribution
To facilitate clear comparison between scattered and grouped distributions, we organize instances of the same class to be adjacent within the demonstration prompts. The impact of this reorganization model performance, both pre and post-grouping, is presented in (suuseecciion A.3 It is easy to observe that there is a general decline in performance across most models after grouping instances by class. Notably, models such as Mistral and InternLM2 exhibit significant performance drops, underscoring a pronounced sensitivity to an effort to delve deeper into this phenomenon, we visualize the accuracy of grouped labels in relation to their positions within the prompt, as illustrated in Figure 4 This visualization reveals that instances of the same class, denoted by dots of the same color, positioned nearby. It became evident that some models, like InternLM2 or Mistral shown in subsectiin A.3] demonstrate high sensitivity to the distribution of instances, only handling instances with labels positioned at the end of the prompt. Conversoly, other open-source models such as CLa3GGMM366B-3K, with a modest 3.3% drop in accuracy, proved to be more resilient to changes in instance positioning. Surprisingly, even the GPT4-turbo and Geminil.5 P-o are not immunc to the challenges posed by grouped distributions, experiencing a notable decline performance by 20.3% and 22.3%. This observed decrease in performance is consistent across models, unaffected by the specific positions of the labels within the prompt.
instance grouping.
9

---

Published in Transactions on Machine Learning Research (03/2025)
The potential reasons why there is substantial drop in performance after grouping the labels in demonstrations are worthy of further research to explore and offer valuable insights for future research and development in this area. One of the potential explanations is that Large Language Models can develop biases based on the position of examples within the prompt. Grouping similar examples together may accidentally reinforce such biases, leading the model to overfit to specific palterns associated with certain positions (like the end or the beginning of the demonstration context). Scattering demonstrations helps distribute examples more evenly, reducing the likelihood of position-inducee biases as discussed in teddin- (Liu et al. 2023)
5 Conclusion
In summary, our research explores the capability of LLMs on long in-context learning tasks, particularly in extreme-label classification scenarios. We curate dataset LongICI.Bench consisting of long in-context. learning tasks with different difficulty levels in terms of context length. Through our study, we have discovered that LLMs demonstrate dramatic performance degradation when it comes to more difficult tasks. Our exploratory experiments further highlight the impact of the distribution of examples within prompts on model performance. We hope LongICLBench and our findings contribute to the ongoing efforts to enhance LLMs' understanding of long contexts.
Broader Impact Statement
The development of long LLMs evaluation benchmarks and the corresponding insights can boost the development of long-context techniques, which can revolutionize fields requiring deep contextual understanding, such as legal analysis, long-form form journalistic content generation, and comprehensive academic summarization. However, there are potential risks associated with the deployment of such powerful models. Enhanced long-context capabilities could be misused for generating misinformation, especially in political or social contexts, where nuanced long-form content can have significant inftuence. There is also the risk of dependency automated systems in icalical decision-making processes, which could lead to over-reliance on technology at the expense of human judgment.
References
The claude 3 model family: Opus, sonnet, haiku. URL 268232499
org. org/CCrpusslii ID:
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 23X::2303.08774, 2023.
01. AT, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models, 2023.
Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Venkatesh Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /tttps://operrevie net.:/oruum/aaasskeeeeetooo
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mci Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian
10

---

Published in Transactions on Machine Learning Research (03/2025)
Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023a.
Yushi Bai, Xin Lv, Jiajic Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2023b.
Kush Bhatia, Himanshu Jain, Purushottam Kar, Manik Varma, and Prateek Jain. Sparse local embeddings for extreme multi-label classification. In Neural Information Processing Systems, 2015. URL https //api. www.ntiicshoholrrrr...eooocccaa. wwwmmaatttcccolaaa.ro//oooorrooo Cttp/CrrpsID::::: 11419932
Zheng Cai, Maosong Cao, Haojiong Chen, .... Yu Qiao, and Dahua Lin. Internim2 technical report. arXiv preprint ar Xiv:2400. 17297, 2024.
Inigo Casanueva, Tadas Temčinas, Danicla Gerz, Matthow Henderson, and Ivan Vulié. Efficient intent detection with dual sentence encoders. In Tsung-Hsien Wen, Asli Celikyilmaz, Zhou Yu, Alexandros Papangelis, Mihail Eric, Anuj Kunar, Inigo Casanueva, and Rushin Shah (eds.), Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AT, pp. 38-45, Online, July 2020. Association for Computational Linguisties. doi:
URL https://acllanthologg org/
2020. 2020.nlp4convai-1.5
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. ArXiv, abs/ 2222230115000 2023a. URL https::/appl
semanticscholar.c :it.aaarrrrrrrrroossss:::::: 2552222222 259262376
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient
fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, 2023b.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. A dataset of information-seekingg questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599 4610, 2021.
Dorottya Demszky, Dana Movshovitz-Ation
Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoEmotions: A dataset of fine-ge ane-grained emotions. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Anreual Meeting of the Association for Computational Linguistios, pp. 4040 4054, Online, July 2020. Association for Computational Linguisties. doi: 10.15653/w1/2022 acl-main.377 URL
acl-main.:3 wwin.ain..... 372
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. Few-NERD: A few-shot named entity recognition dataset. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 3198 3198-3213, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/
URL http://g://ccaaaaaooooooooo/////
248
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. orXiv preprint Areiv:2402.1177 2024.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannanch Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arXiv preprint NrXiv:2402. 10171, 2024.
11

---

Published in Transactions on Machine Learning Research (03/2025)
Albert Gu and Tri Dao. Mamba: Lincar-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.0075: 2023.
Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. Structured prompting: Scaling in- context learning to 1, 000 examples. ArXiv, abs/2212.067133, 2022. URL https://api. httpps//appl.emmmntiiccccooooo semanticscholar
Ntg/CorpusID:25445 254599668
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. Llm maybe longlm: Self-extend Ilm context window without tuning, 2024.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pro-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HLTT, pp. 4171-4186, 2019.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models, 2024.
Dacheng Li. Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source LLMs truly promise? In NewrIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a. URL https:/ net/ forun'id=LyyiiIPIWV
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts?, 2023b.
Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jun Zhang, Zhiyong Wu, and Lingpeng Kong. In-context learning with many demonstration examples, 2023c.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Eneko Agirre, Marianna Apidianaki, and Tvan Vulié (eds.). Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Archilectures, pp. 100 114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics. doi: 10.18653/v1 10/55/33/1/2222eeel// deelio- 1.10. URL httpp:://ccaatthoo wtwphhoooogooo.gg/222/22// or2/2022 deelio-1.100
Jiaheng Liu, Zhiqi Bai, Yuanxing Zhang, Chenchen Zhang, Yu Zhang, Ge Zhang, Jiakai Wang, Haoran Que, Yukang Chen, Wenbo Su, et al. E 2-1lm: Efficient and extreme length extension of large language models. arXiv preprint ar (xxx:2401.06955 2024.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang Lost in the middle: How language models use long contexts. Trunsactions of the Association for Computational Linguistics, 12:157-173, 2023. URL http://tttpppss:/p/ssseeaaaaaa hosps/c/app..nnnnniccccoooo.rrrr hagg/CorpuIDD
259360665
Aristides Milios, Siva Reddy, and Dzmitry Bahdanau. In-context learning for text classification with many labels, 2023.
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. In Workshop on Efficient Systems for Foundation Models ICML22023, 2023.
Ramesh Nallapati, Feifci Zhai, and Bowen Zhou. Sunmarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
12

---

Published in Transactions on Machine Learning Research (03/2025)
Antonio Orvicto, Samuel L. Smith, Albert Gu, Amushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. ArXic, abs/2330.00639, 2023. URL
2577496664
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derezynski, et al. Rwkv: Reinventing rmns for the transformer cra. In Findings of the Associalion for Compulalional Lánguistics: EMNLP 2023, pp. 10488-14077 2023a.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023b.
Hao Peng, Xiaozhi Wang, Jianhui Chen, Weikai Li, Yunjia Qi, Zimu Wang, Zhili Wu, Kaisheng Zeng, Bin Xu, Lei Hou, and Juanzi Li. When does in-context learning fall short and why? a study on specilccation-eavy tasks, 2023c.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022. URL https: //openreview wwwp://ee/foomiicccmmm wtt.sprPprn
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown. and Yoav Shoham. Parallel context windows for large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6383 6402, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: /1053///1112223cclll https://aclanthonn www.hhooooooooooooo...ooooooo org. 20wgg//223.... ww2...a.l....m.......e acl-long........... 352
URL
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean- Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, loannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katic Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mclroo,, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem W. Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lene, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, Luke Vilnis, Oscar Chang, Nobuyuki Morioka, George Tucker, Ce Zheng, Oliver Woodman, Nithya Attaluri, Tomas Kocisky, Evgenii Eltyshev, Xi Chen, Timothy Chung, Vittorio Selo, Siddhartha. Brahma, Petko Georgiev, Ambrose Slone, Zhenkai Zhu, James Lottes, Siyuan Qiao, Ben Caine, Sebastian Riedel, Alex Tomala, Martin Chadwick, J Christopher Love, Peter Choy, Sid Mittal, Neil Houlsby, Yunhao Tang, Matthew Lamm, Libin Bai, Qiao Zhang, Luheng He, Yong Cheng, Peter Humphreys, Yujia Li, Sergey Brin, Albin Cassirer, Ying-Qi Miao, Lukas Zilka, Taylor Tobin, Kelvin Xu, Lev Proleev, Daniel Sohn, Alberto Magni, Lisa Anne Hendricks, Isabel Gao, Santiago Ontan 'on Oskar Bunyan, Nathan Byrd, Abhanshu Sharma, Biao Zhang, Mario Pinto, Rishika Sinha, Harsh Mchta, Dawei Jia, Sergi Caelles, Albert Webson, Alex Morris, Becca Roelofs, Yifan Ding, Robin Strudel, Xuehan Xiong, Marvin Ritter, Mostafa Dehghani, Rahma Chaabouni, Abhijit Karmarkar, Guangda Lai, Fabian Mentzer, Bibo Xu, YaGuang LI, Yujing Zhang, Tom Le Paine, Alex Goldin, Behnam Neyshabur, Kate Baumli, Anselm Levskaya, Michael Laskin, Wenhao Jia, Jack W. Rae, Kefan Xiao, Autoine He, Skye Giordano, Lakshman Yagati, Jean- Baptiste Lespiau, Paul Natsev, Sanjay Ganapathy, Fangyu Liu, Danilo Martins, Nanxin Chen, Yunhan Xu, Megan Barnes, Rhys May, Arpi Vezer, Junhyuk Oh, Ken Franko, Sophie Bridgers, Ruizhe Zhao, Boxi Wu, Basil Mustafa, Sean Sechrist, Emilio Parisotto, Thanumalayan Sankaranarayana Pillai, Chris Larkin, Chenjie Gu, Christina Sorokin, Maxim Krikun, Alexey Guseynov, Jessica Landon, Romina Datta, Alexander Pritzel, Phocbe Thacker, Fan Yang, Kevin Hui, A.E. Hauth, Chih-Kuan Ych, David Barker, Justin Mao-Jones, Sophia Austin, Hannah Sheahan, Parker Schuh, James Svensson, Rohan Jain, Vinay Venkatesh Ramasesh, Anton Briukhov, Da- Woon Chung, Tamara von Glehn, Christina Butterfield, Priya Jhakra, Matt Wicthoff, Justin Frye, Jordan Grimstad, Beer Changpinyo, Charline Le Lan, Anna Bortsova, Yonghui Wu, Paul Voigtlaender, Tara N. Sainath, Charlotte Smith, Will Hawkins, Kris Cao, James Besley, Srivatsan Srinivasan, Mark Omernick, Colin Gaffney, Gabriela de Castro Surita, Ryan
13

---

Published in Transactions on Machine Learning Research (03/2025)
Burnell, Bogdan Damoc, Junwhan Ahn, Andrew Brock, Mantas Pajarskas, Anastasia Petrushkina, Seb Noury, Lorenzo Blanco, Kevin Swersky, Arun Ahuja, Thi Avrahami, Vedant Misra, Raoul de Liedekerke, Mariko linuma, Alex Polozov, Sarah York, George van den Driessche, Paul Michel, Justin Chiu, Rory Blevins, Zach Gleicher, Adria Recasens, Alban Rrustemi, Elena Gribovskaya, Aurko Roy, Wiktor Gworek, S'ebastien M. R. Arnold, Lisa Lee, James Lee-Thorp, Marcello Maggioni, Enrique Piqueras, Kartikeya Badola, Sharad Vikram, Lucas Gonzalez, Anirudh Baddepudi, Evan Senter, Jacob Devlin, James Qin, Michael Azzam, Maja Trebacz, Martin Polacek, Kashyap Krishnakumar, Shuo yinn Chang, Matthew Tung, Ivo Penchev, Rishabh Joshi, Kate Olszewska, Carrie Muir, Mateo Wirth, Ale Jakse Hartman, Joshua Newlan, Sheleem Kashem, Vijay Bolina, Elahe Dabir, Joost R. van Amersfoort, Zafarali Ahmed, James Cobon-Kerr, Aishwarya B Kamath, Arnar Mar Hrafnkelsson, Le Hou, Ian Mackinnon, Alexandre Frechette, Eric Noland, Xiance Si, Emanuel Taropa, Dong Li, Phil Crone, Anmol Gulati, S'ebastien Cevey, Jonas Adler, Ada Ma, David Silver, Simon Tokumine; Richard Powell, Stephan Lee, Michael B. Chang, Samer Hassan, Diana Mincu, Antoine Yang, Nir Levine, Jenny Brennan, Mingqiu Wang, Sarah Hodkinson, Jeffrey Zhao, Josh Lipschultz, Aedan Pope, Michael B. Chang, Cheng Li, Laurent El Shafey, Michela Paganini, Sholto Douglas, Bernd Bohnet, Fabio Pardo, Seth Odoom, Mihaela Rosca, Cicero Nogueira dos Santos, Kedar Soparkar, Arthur Guez, Tom Hudson, Steven Hansen, Chulayuth Asawaroengchai, Ravichandra Addanki, Tianhe Yu, Wojeiech Stokowiec, Mina Khan, Justin Gilmer, Jachoon Lee, Carrie Grimes Bostock, Keran Rong. Jonathan Caton, Pedram Pejman, Filip Pavetic, Geoff Brown, Vivek Sharma, Mario Luvei'e, Rajkumar Samuel, Josip Djolonga, Amol Mandhane, Lars Lowe Sjosund, Elena Buchatskaya, Elspeth White, Natalie Clay, Jiepu Jiang, Hyeontaek Lim, Ross Hemsley, Jane Labanowski, Nicola De Cao, David Steiner, Sayed Hadi Hashemi, Jacob Austin, Anita Gergely, Tim Blyth, Joe Stanton, Kaushik Shivakumar, Aditya Siddhant, Anders Andreassen, Carlos L. Araya, Nikhil Sethi, Rakesh Shivanna, Steven Hand, Ankur Bapna, Ali Khodaei, Antoine Miech, Garrett Tanzer, Andy Swing, Shantanu Thakoor, Zhufeng Pan, Zachary Nado, Stephanie Winkler, Dian Yu, Mohammad Saleh, Lorenzo Maggiore, Iain Barr, Minh Giang, Thais Kagohara, Ivo Danihelka, Amit Marathe, Vladimir Feinberg, Mohamed Elhawaty, Nimesh Ghelani, Dan Horgan, Helen Miller, Lexi Walker, Richard Tanburn, Mukarram Tariq, Disha Shrivastava, Fei Xia, Bhung-Cheng, Chiu, Zoe C. Ashwood, Khuslen Baat Beatarsukh, Sina Samangooei, Fred Alcober, Axel Sijennrren, Paul Komarek, Katerina Tsihlas, Anudhyan Boral, Ramona Comanescu, Jeremy Chen, Ruibo Liu, Dawn Bloxwich, Charlic Chen, Yanhua. Sun, Fangxiaoyu Feng, Matthow Mauger, Xerxes Dotíwalla, Vincent Hellendoorn, Michael Sharman, Ivy Zheng, Krishna Haridasan, Gabriel Barth-Maron, Craig Swanson, Dominika. Rogozi nska,, Alek Andreev, Paul Kishan Rubenstein, Ruoxin Sang, Dan Hurt, Gamaleldin Elsayed, Renshen Wang, Dave Lacey, Anastasija ll Yao Zhao, Lora Aroyo, Chimezie Iwuanyanwu, Vitaly Nikolacv, Balaji Lakshminarayanan, Sadegh Jazayeri, Raphael Lopez Kaufman, Mani Varadarajan, Chetan Tekur, Doug Fritz, Misha Khalman, David Reitter, Kingshuk Dasgupta, Shourya Sarcar, T. Orndulf, Javier Snaider, Fantine Huot, Johnson Jia, Rupert Kemp, Neje Trdin, Anitha Vijayakumar, Lucy Kim, Christof Angermueller, Li Lao, Tianqi Liu, Haibin Zhang, David Engel, Somer Greene, Anais White, Jessica Austin, Lilly Taylor, Shereen Ashraf, Dangyi Liu, Maria Georgaki, Irene Cai, Yana Kulizhskaya, Sonam Goenka, Bremnan Saeta, Kiran Vodrahalli, Christian Frank, Dario de Cesare, Brona Robenek, Harry Richardson, Mahmoud Alnahlawi, Christopher Yew, Priya Ponnapalli, Marco Tagliasacchi, Alex Korchemniy, Yelin Kim, Dinghua Li, Bill Rosgen, Kyle Levin, Jeremy Wiesner, Praseemn Banzal, Praveen Srinivasan, Hongkun Yu, cCauglar Unlu, David Reid, Zora Tung, Daniel F. Finchelstein, Ravin Kumar, Andre Elisscoff, Jin Huang, Ming Zhang, Rui Zhu, Ricardo Aguilar, Mai Gim'enez, Jiawei Xia, Olivier Dousse, Willi Gierke, Soheil Hassas Yeganeh, Damion Yales, Komal Jalan, Lu Li, Eri Latorre- Latorre-Chinolo, Duc Dung Nguyen, Ken Durden, Praveen Kallakuri, Yaxin Liu, Matthew Johnson, Tomy Tsai, Alice Talbert, Jasmine Liu, Alexander Neitz, Chen Elkind, Marco Selvi, Mimi Jasarevic, Livio Baldini Soares, Albert Cui, Pidong Wang, Alek Wenjiao Wang, Xinyu Ye, Krystal Kallarackal, Lucia Loher, Hoi Lam, Josef Broder, Daniel Niels Holtmann-Rice, Nina Martin, Bramandia Ramadhana, Daniel Toyama, Mrinal Shukla, Sujoy Basu, Abhi Mohan, Nicholas Fernando, Noah Fiedel, Kim Paterson, Hui Li, Ankush Garg, Jane Park, Donghyun Choi, Diane Wu, Sankalp Singh, Zhishuai Zhang, Amir Globerson, Lily Yu, John Carpenter, Félix de Chaumont Quitry, Carey Radebaugh, Chu- Chu-Cheng Lin, Alex Tudor, Prakash Shroff, Drew Garmon, Dayou Du, Neera Vats, Han Lu, Shariq Tqbal, Alexey Yakubovich, Nilesh Tripuraneni, James Manyika, Haroon Qureshi, Nan Hua, Christel Ngani, Maria Abi Raad, Hannah Forbes, Anna Bulanova, Jeff Stanway, Mukund Sundararajan, Victor Ungureanu, Colton Bishop, Yunjie Li, Balaji Venkatraman, Bo Li, Chloe Thornton, Salvatore Scellato, Nishesh Gupta, Yicheng Wang, lan Tenney, Xihui
14

---

Published in Transactions on Machine Learning Research (03/2025)
Wu, Ashish Shenoy, Gabriel Carvajal, Diana Gage Wright, Ben Bariach, Zhuyun Xiao, Peter Hawkins, Sid Dalmia, Cl'ement Farabet, Pedro Valenzuela, Quan Yuan, Christoper A. Welty, Ananth Agarwal, Mianna Chen, Wooyeol Kim, Brice Hulse, Nandita Dukkipati, Adam Paszke, Andrew Bolt, Elnaz Davoodi, Kiam Choo, Jennifer Beattic, Jennifer Prendki, Harsha Vashisht, Reboca Jarek Wilkiewiez, David Madras, Ali Elqursh, Grant Uy, Kevin Ramirez, Matt Harvey, Tyler Liechly, Heiga Zen, Jeff Seibert, Clara Huiyi Hu, A. Ya. Khorlin, Maigo Le, Asaf Aharoni, Megan Li, Lily Wang, Sandeep Kumar, Alejandro Lince, Norman Casagrande, Jay Hoover, Dalia El Badawy, David Soergel, Denis Vnnkov, Matt Miecnikowski, Jifi Simsa, Anna Koop, Praveen Kumar, Thibault Sellam, Daniel Vlasic, Samira Daruki, Nir Shabat,, John Zhang, Guolong Su, Jiageng Zhang, Jeremiah Liu, Yi Sun, Evan Palmer, Alireza Ghaffarkhah, Xi Xiong, Victor Cotruta, Michael Fink, Lucas Dixon, Ashwin Sreevatsa, Adrian Goedeckemeyer, Alek Dimilriev, Mohsen Jafari, Remi Crocker, Nicholas A Fitzgerald, Aviral Kumar, Sanjay Ghemawat, Ivan Philips, Frederick Liu, Yannie Liang, Rachel Sterneck, Alena Repina, Marcus Wu, Laura Knight, Marin Georgiev, Hyo Lee, Harry Askham, Abhishck Chakladar, Annic Louis, Carl Crous, Hardie Cate, Dessie Petrova, Michael Quinn, Denese Owusu-Afriyiie Achintya Singhal, Nan Wei, Solomon Kim, Damien Vincent, Milad Nasr, Christopher A. Choquette-Chov Reiko Tojo, Shawn Lu, Diego de Las Casas, Yuchung Cheng, Tolga Bolukbasi, Katherine Lee, Saaber Fatehi, Rajagopal Ananthanarayanan, Miteyan Patel, Charbel El Kaed, Jing Li, Jakub Sygnowski, Shreyas Rammohan Belle, Zhe Chen, Jaclyn Konzelmann, Siim Poder, Roopal Garg, Vinod Koverkathu, Adam Brown, Chris Dyer, Rosanne Liu, Azade Nova, Jun Xu, Slav Petrov, Demis Hassabis, Koray Kavukeuoglu, Jeffrey Dean, and Oriol Vinyals. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv, 0es/2203.055533 2024. URL https://app. www.eniisscoollaa...... org/CorpusID 268297180
Luis C. Cobo,
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code laaaa Open foundation models for code, 2024.
Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller. Mining discourse markers for unsupervised sentence representation learning. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1 11t/1//11111111//11 URL
Minneapolis,
hrw.laathology..og///11S www.:/99g///11911 1351
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. ArXiv, abs/2104.09864, 2021. URL https: lttt::://api.smmaattat sww.nti .rgg CCrpppsppss
233307138
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Nearocomputing, 568:127063, 2024.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL qVyeW-grC2k
http///rrevvii.eeeefoooortaaaa net/forum?id
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptisle Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXio preprint orXiv:2312.0 222122118000 11805, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya. Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0880 08295, 2024.
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miloć. Focused transformer: Contrastive training for context scaling, 2023.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Ivan Vulié, Pei-Hao Su, Samuel Coope, Daniela Gerz, Pawel Budzianowski, Iñigo Casanueva, Nikola Mrkšić, and Tsung-Hsien Wen. ConvFiT: Conversational fine-tuning of pretrained language models. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1151-1168, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 2021. emnlp-main. 88
emnlp-main.86. URL httpp://aclanthoooogy
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. In seieth- Conference on Neural Information Processing Systems, 2023. URL
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering, 2023.
Guangxuan Xiao, Yuandong Tian, Boidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL httpps://openrevier net/f http//formmmmddd/mem 2d
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajiwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309. 16039, 2023.
Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. Dialogue-based relation extraction. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4927-4940, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/vl 100/3//12220.aa// /2000..cl-main.4444
URL htttss:/aclanthhollgg
wttp////....///oooo/ acl- -main.ain... 444.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm- An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, 2022.
Xinrong Zhang, Yingla Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. oobench: Extending long context evaluation beyond 100k lokens, 2024.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Tanguage. Processing (EMNLP 2017), pp. 35-45, 2017. URL URLhhttps: httt::://nnl..saaaee pdf
Dawii Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient context window extension of LLMs via positional skip-wise training. In The Twelfth International Conference Learning Representations, 2024. URL https::/ wwtp://www..eeeoooor////////2//e
A
Appendix
A.1 Additional Datasets
We list few additional datasets as follows:
GoEmotions (Demszky et al. 2020) is the largest manually annotated dataset of 58k English comments from Reddit, which is labeled into 27 emotion categorics or Neutral. There are 27 types of emotion types and drop the rare ones with few examples. Each selected example contains 28 tokens on average.
Few-NERD (Ding et al.) 2021 is a large-scale camman-annotated name entity recognition dataset with a hicrarchy of 8 coarso-grained and 66 fino-grained entity types. Each of the instances is a paragraph with approximately 61 tokens on average and contains one or multiple entity names as the ground truth answer. There are 66 types of entities in the collection.
16

---

Published in Transactions on Machine Learning Research (03/2025)
The performance for the two tasks is demonstrated in Table and Table g
Model
Param
Support
1R
2R
3R
4R
5R
Context Tokens
0.8K
1.6K
2.4K
3.2K
4K
Gemma-7B-base
7B
8K
0
0
0
0
7B
32K
0
0
0.2
0.2
ChatGLM3-6B-33 BBKKKK
6B
32K
22.0
17.0
15.0
12.6
10.6
Qwen-1.5-7BBbaase
7B
32K
14.8
18.2
18.6
19.0
14.2
Mistral-7B-v0.2--
7B
32K
2.6
11.4
7.4
11.6
12.4
I.L.MA-2-7B-T.oogl.om
7B
100K
0
0
0
0
Yi-6B-200K
6B
200K
0.8
4.0
IuternLM2--B--baee
7B
200K
0
0
0
0
0
Long-LLaMAA-odde
7B
256K
0.2
RWKV-5-World
7B
1K
8.8
7.4
4.6
5.2
Mamba-2.8B
2.8B
2K
0
0
0
0
GPT4-turbo
N/A
128K
36.5
34.4
35.0
33.3
32.0
GPT4o
N/A
128K
23.0
23.8
21.2
21.2
22.2
Claude3-Opus
N/A
200K
25.8
7.4
17.0
12.6
19.6
Gemini-1.5-Pro
N N/A
10M
19.0
10.4
9.2
10.6
9.4
SoTA (BERT)
N/A
58.9
Table GoEmotion Result.
Model
Param
Support
1R
2R
3R
4R
5R
Context Tokens
5K
9K
14K
19K
24K
Gemma-7B-base
7B
8k
44.0
44.2
LLaMA-2-7B-32K
7B
32k
36.9
40.8
41.1
41.6
41.3
ChatGLM3-6BB322
32k
10.4
Qwen-1.5-7B-bass
7B
32k
40.0
46.4
47.6
47.3
47.8
Mistral-7B-vv..2-hal
7B
32K
42.2
47.4
48.9
50.0
50.0
LLaMAA--77B-Longloorr
7B
100K
Yi-6B-200K
34.3
40.2
12.3
IntemLMM-7BB-ase
7B
200k
43.6
46.2
46.5
47.8
48.3
Long-
Long-LLLMA-code-7BB
7B
256K
22.3
25.5
26.5
29.4
27.0
RWKV-5-World
7B
1k
13.9
0.7
9.9
Mamba-2.8B
2.8B
2k
GPT4-turbo
N/A
128k
53.4
55.3
55.6
56.8
GPT4o
N/A
128k
46.7
41.4
42.8
39.0
44.4
Claude3-Opus
N/A
53.5
51.3
51.2
52.5
Gemini-L.5-P2
N/A
10M
55.4
47.8
49.5
41.4
42.4
SoTA (PL-Marker)
N/A
70.9
Table 8: Few-NERD Result.
A.2
Prompting Template
The prompting template for each of the datasets presented at Table
A.3 Additional Distribution Analysis
To facilitate a clear comparison between random and grouped distributions, we organize instances of the same class be adjacent within the demonstration prompts. The impact of this reorganization on model performance, both pre and post-grouping, is presented in Table T0
17

---

Published in Transactions on Machine Learning Research (03/2025)
Dataset
Prompt
GoEmotion
Given a comment, please predict the emotion category this comment. The prediction from the demonstration examples with the exact format. The examples are as follows:
fcomment: "..comment..."
emotion category: "..eemtioon.."
repeal n lines
BANKING77
Given a customer service query, please predict the intent of the query. The predicted answer must come from the demonstration examples with the exact format. The examples are as follows:
(service query: "...ervvcee.." intent category: "...intent...
repeat times
TacRED
Given sentence and pair subject and object entities within the sentence, please predict the relation between the given entities. The examples are follows: (sentence: "..sentence..
the subject is "..subjecct.." the object is "...object..."
the relation between the two entities is: "...relation..'
x repeat n times
Few-NERD
Given the sentence, please find the name entities in the sentence and their corre- sponding entity types the strict format the given examples following (Eutity: Entity'Typej: ("...eutity...' "...entity type...' repeal times
DialogRE
Given the dialogue, please find the name pair entities in the dialogue and their correspouding relation types in the strict format of given examples as following (note that the number of entities has strictly have the same value the number of respective relation): (Dialogne:
"..dialogue..."
The list entity pairs are "s.bjeujeet,,, objectl), (subject2, object2), etc... The ...mumber of pairs...' respective relations between each entity pair are: '...relation, relation2, elc...
repeat times
Discovery
Given two sentencel and sentence2, please predict the conjunction word between the two sentences. The predicted answer must come from the demonstration examples with the exact format. The examples follows: {" ("...sentencel.." () ...sentence2...
the conjunction word in () ...conjuucttoo...
repeat n times
Table
The data prompt format of each dataset. Fach dataset has a unique prompt format to effectively utilize the context and format of its respective data to get the best output response.
The distribution plots for other models are shown Figure and Figure 6
A.4 Data Accessibility
Our CLLnnhL is set under MIT license, thus permission is granted, free of charge, to any person obtaining copy of this dataset and associated documentation files. The datasests are curated under the rules guaranteed by the original dataset. There is no personally identifiable or offensive content in the dataset.
18

---

Published in Transactions on Machine Learning Research (03/2025)
Model
Param
Support
Scatter
Grouped
A
Context Tokens
10K
Gemma-7B-base
0
LLaMA-2-7B-32K
7B
32K
0.4
3.0
+2.6
CharGLM3-66B3322
6B
32K
38.9
35.6
-3.3
Qwen-1.5-7B-bar
7B
32K
45.2
33.0
-12.2
Mistral-7B--v0.241
7B
32K
51.6
5.1
-46.5
LLa.MA-2-77B-LonggL.ora
7B
100K
0
0
Yi-6B-200K
6B
200K
8.0
0
-8
InternLMM--7B-base
7B
200K
15.5
1.8
-9.7
Long-
Long-LLaMA-coder
7B
256K
4.1
0
-4.1
RWKV-5-World
7B
4K
1.0
3.6
+2.6
Mamba-2.8B
2.8B
0
GPT4-turbo
N/A
128K
79.5
59.2
-20.3
Gemini-LL.-Pro
N/A
10M
79.6
57.3
-22.3
Table 10: Exploratory Result on TacRED 3 Round. Grouped means forcing the same-typed demonstration examples near by each other instead randomly distributing in the prompt.
Mistral-7B-v0.2- Scatter
Swen--.5-78-bbae Scatter
LLaMA-2-7B-32K Scatter
100
100
100
Class Accuracy (%)
80
80
80
60
60
40
40
20
20
20
0
25
125
100
125
75
100
125
Mistral-7B-v0. 2 2-base Grouped
Qwen-1.5- 1..--115-BBBB. 5-7B-base Grouped
LLaMA-2-7B-924 Grouped
100
Class Accuracy (%)
80
80
60
60
60
40
40
20
20
0
0
25
50
75
100
125
0
25
50
75
100
125
0
25
50
75
100
125
Instance Position
Instance Position
Instance Position
Figure Visualization of accuracy for every class when instances from the same class are scattered V.S. grouped in the demonstration prompt.
19

---

Published in Transactions on Machine Learning Research (03/2025)
Long-LLaMA-code-7B Scatter
Yi-6B-200K Scatter
Gemini-1.5 Scatter
100
100
100
Class Accuracy (%)
80
80
80
20
20
20
50
125
100
125
100
Long-LLaMA-codee Grouped
Yi-6B-200K Grouped
Gemini-1.5 Grouped
100
Class Accuracy (%)
80
80
80
60
60
60
25
50
75
100
125
25
50
75
100
125
25
75
100
125
Instance Position
Instance Position
Instance Position
Figure Visualization of accuracy for every class when instances from the same class are scattered V.S. grouped in the demonstration prompt.
20


Reviews:
[{"review_id": "HXDorlsgTd", "reviewer": "Reviewer_4FTQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The introduction of the new benchmark in this paper could contribute to evaluating long-context capabilities in LLM for more practical scenarios focusing on tasks that require understanding large contexts and extreme-label classification.\n* The authors provide some insights which are the degradation of performance with increased context length and potential bias affecting prediction outcomes.\n\nWeaknesses\n1. This paper lacks an in-depth explanation of why LLMs fail to generalize with longer contexts or extreme labels. It just reported that the length of contexts affects the performance degradation.\n2. This paper lacks specific explanations for their datasets. The author should contain more concrete information about the datasets such imbalance of the dataset or what the classification labels are. Furthermore, since this paper makes ICL scenarios with existing benchmark datasets, it also should note how differently those existing datasets were used. \n3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors offer a new benchmark dataset, LongICLBench, for long in-context learning in extreme-label classification. This dataset comprises six tasks with different difficult levels regarding context length and label space. Using this dataset, this paper exhibits the performance of numerous long-context LLMs and demonstrates that the models'  performance consistently degrades as given tasks get more complex.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Regarding the two weaknesses above, could the author add more information to the paper?\n2. It would be helpful to report the performance achieved by employing other learning techniques with the entire dataset, not using ICL, or human evaluation. This would provide an oracle performance level, offering clearer context for how underwhelming the ICL performance truly is.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ItmSpcZC6o", "reviewer": "Reviewer_7dKU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper proposes a novel and convincing approach for evaluating the long in-context learning capabilities of LLMs.\n2. The authors conducted extensive evaluations across diverse models.\n3. The paper is clearly written and easy to follow.\n\nWeaknesses\n1. There is no explanation for how the specific datasets were selected. Can the same approach be directly applied to other extreme-label classification datasets to extend the benchmark?\n2. The authors demonstrate that scattering the demonstrations consistently performs better than grouping them, but they do not provide an explanation for the suspected reasons.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes LongICLBench, a new benchmark for evaluating long-context LLMs. The core idea is to use extreme-label classification datasets to construct the benchmark. The authors compare 15 long-context LLMs across 6 datasets and suggest that modern long-context LLMs still struggle with these tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There is no critical adjustment required. Along with the weaknesses mentioned above, here is an additional change that could strengthen the work.\n\n1. Can the dataset be easily extended to include more demonstrations (e.g., 10-shot)? Having the flexibility to provide longer inputs would be beneficial for future use of the benchmark, as most LLMs are already claiming context windows of 128k tokens or even millions.", "broader_impact_concerns": "I have no concerns about the ethical implications of the work beyond those stated in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rcb4VDLTuM", "reviewer": "Reviewer_iSUr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper introduces a novel benchmark, LongICLBench, which will undoubtedly aid in the future development of LLMs by assessing their performance on long sequences. \nThe experimental studies effectively highlight the shortcomings of current LLMs.\n\n\n\nWeaknesses:\n\n- Although the paper points out the issues with current LLMs, it does not provide concrete solutions to address them. \nAs such, this paper serves primarily as a diagnostic tool. \nThat being said, it may have limited impact on researchers looking to propose solutions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides an experimental evaluation on the performance of LLMs processing long sequences. \nThe current evaluation for LLMs replies on basic metrics like perplexity and synthetic tasks.\nHowever, these metrics may not be able capture the real world scenarios.\nIndeed, previous results showed that LLMs handle shorter and simpler tasks well but they struggle on more difficult tasks.\n\nIn this paper, the authors introduce a new benchmark called LongICLBench which focuses on long in-context learning for extreme-label classification.\nThe result suggests that there are some areas needed to be improved.\nThe first one is Bias towards later labels in the sequence.\nThe second one is performance degradation as input length and label space increase.\nThe third one is the need for better reasoning over long contexts and multiple pieces of information.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Note:\n\n- Abstract: \"Some models like Gemini could even to be capable ...\" $\\to$ \"Some models like Gemini could even be capable ...\"", "broader_impact_concerns": ".", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces LongICLBench, a new benchmark that leverages extreme-label classification tasks to evaluate the long-context capabilities of Large Language Models (LLMs). The benchmark provides a more realistic assessment compared to traditional metrics like perplexity, demonstrating through extensive experiments across multiple models and datasets that current models struggle with longer contexts and larger label spaces.

The reviewers unanimously agree that the benchmark's design and comprehensive evaluation make a valuable contribution to the field. The authors have adequately addressed the reviewers' concerns in their rebuttal, including clarifications on dataset selection criteria, the impact of demonstration ordering, and detailed dataset descriptions. While one reviewer noted the lack of proposed solutions, the Action Editor agrees that establishing a robust diagnostic benchmark is an important contribution that will enable future research on improving long-context capabilities.
Decision: Accept (poster)