Task-agnostic Prompt Compression with Context-aware Sentence Embedding and Reward-guided Task Descriptor
Authors: Barys Liskavets
Shuvendu Roy
Maxim Ushakov
Mark Klibanov
Ali Etemad
Shane K. Luke
Venue: Accepted by TMLR ()

Abstract: The rise of Large Language Models (LLMs) has led to significant interest in prompt compression, a technique aimed at reducing the length of input prompts while preserving critical information. However, the prominent approaches in prompt compression often require explicit questions or handcrafted templates for compression, limiting their generalizability. We propose Task-agnostic Prompt Compression (TPC), a novel framework that generalizes compression across tasks and domains without requiring input questions or templates. TPC generates a context-relevant task description using a task descriptor trained on a curated dataset of context and query pairs, and fine-tuned via reinforcement learning with a reward function designed to capture the most relevant information. The task descriptor is then utilized to compute the relevance of each sentence in the prompt to generate the compressed prompt. We introduce 3 model sizes (Base, Large, and Huge), where the largest model outperforms the existing state-of-the-art methods on LongBench and ZeroSCROLLS, and our smallest model performs comparable to the existing solutions while being considerably smaller.

Published in Transactions on Machine Learning Research (11/2025)
Task-agnostic Prompt Compression with Context-aware Sen- tence Embedding and Reward-guided Task Descriptor
Barys Liskavets" Shuvendu Roy?, Maxim Ushakov',
Mark Klibanov Ali Etemad', Shane K. Luke
Alterra AI, Palo Alto, United States, =Queen's University, Canada, WWorkday Inc
Reviewed OpenReview: http:://eeenvvveeerrerrr///u//rr
Abstract
The rise of Large Language Models (LLMs) has led to significant interest in prompt com- pression, technique aimed at reducing the length of imput prompts while preserving critical information. However, the prominent approaches in prompt compression often require explicit questions or handcrafted templates for compression, limiting their generalizability. We propose Task-agnostic Prompt Compression (TPC), a novel framework that generalizes compression across tasks and domains without requiring input questions or templates. TPC generates context-relevant task description using task descriptor trained on curated dataset of context and query pairs, and fine-tuned via reinforcement learning with a reward function designed to capture the most relevant information. The task descriptor is then utilized to compute the relevance of each sentence in the prompt to generate the compressed prompt. We introduce model sizes (Base, Large, and Huge), where the largest model outperforms the existing stale-of-the-arl methods on LongBench and ZeroSCROLLS, and our smallest model performs comparable the existing solutions while being considerably smaller. Finally, release the code and the dataset for quick reproducibility and further development: httpps://githbbbccm httph//ww.ooommissstttsttt/eeeee
1 Introduction
The emergence of Large Language Models (LLMs) has spurred extensive research into prompting techniques, including chain-of-thought reasoning Wei ct al. (2022; in-context learning Dong et al. (2024). and retrieval- augmented generation Lewis et. al. (020)] to harness LLMs' generalization and reasoning abilities for various applications. In practice, effective prompting often requires detailed and lengthy inputs to generate high- quality responses in domain-specific tasks. However, long prompts significantly increase inference time and costs. To address this, a new research direction called prompt compression Jiang et al. (2023b c); Liskavets et al. (2025) has emerged, which focuses on reducing prompt length while retaining the critical information needed to accurately answer user queries.
Early research prompt compression predominantly focused on token-level techniques, which involve analyzing the importance of individual tokens in prompt and removing less informative ones to generate the compressed prompt. However, token-level compression often leads to incoherent or fragmented sentences, hindering the overall performance. To address this, recent approaches, such as CPC Liskavets et al. (025 have shifted to sentence-level compression, which evaluates the relevance each sentence to the input question. While this method represents a significant improvement over token-level approaches, its reliance on explicit questions or manually crafted templates for non-question-based tasks (e.g. summarization or code) restricts its broader applicability.
In this work, we propose Task-agnostic Prompt Compression (TPC), generic prompt compression method capable of compressing input prompls across lasks and domains without relying on input questions or
"Corresponding Author. Email: bakkeett.boressd@mmall.com

---

Published in Transactions on Machine Learning Research (11/2025)
55
Performace (prompt poompt--ggooss agnostic)
45
Performance (prompt (it-awrre)
35
45
40
25
15-
30
LLMLingua
LLMLingua-2 LongLLMLingua
TPC-Large
TPC-Base
SelectiveContext
CPC
TPC-Huge
Figure 1: Comparison of model size versus performance for different prompt compression methods in both task-aware and task-agnostic sctups. Our largest model, TPC-Huge, outperforms all cxisting methods while maintaining a comparable size to existing solutions. On the other hand, our smallest model, TPC-Base, achieves a competitive performance despite being significantly smaller in size.
handcrafted prompt templates. The central idea of TPC involves using novel task descriptor to create context-relevant task description, which facilitates compression by comparing its context-aware embedding similarity to each sentence in the prompt. Here, the task descriptor is causal LM, trained on our curated dataset of prompt-question pairs, that generates context-relevant task description encapsulating the main concept of the prompt. Since the effectiveness of the prompt compression relies on the quality of the generated task description, we introduce a reinforcement learning (RL) approach to further ine-tune the task descriptor with a novel reward function designed to encourage capturing the most relevant information of the prompt in the task description. Finally, the context-awand embedding is computed using context-aware sentence encoder trained
curated dataset of multi-hop questions, answers, and positive and negative pairs.
We present comprehensive experiments to evaluate our proposed solution and compare it against existing methods on two evaluation setups: task-aware compression and task-agnostic compression on two popular benchmarks LongBench and ZeroCCROLLS adopted by existing literature on prompt compression. We report the results for three variants of our model size: TPC-Base, LaC-Large, and TPC-Huge, containing 0.5B, 1B, and 7B parameters, respectively. As summarized Figure [ our TPC shows significant improvements over existing methods on both task-aware and task-agnostic setups. Our smallest model, while being considerably smaller in size, outperforms performs comparable to the existing statc-of-the-art (SOTA) methods. Overall, make the following contribution in this paper:
We introduce TPC, task-invariant prompt compression method, which unlike existing methods doesn't require task-specific hand-crafted templates to achieve generalizable compression. TPC generates a context-relevant task description using a task descriptor LMM, which is then used for compression by comparing the context-relevant embedding to each sentence in the prompt.
We develop synergistic training framework consisting of two components: a contextual task description (CTD) module which generates concise task-relevant descriptions that guide the contextual segment importance scoring module (CSE). Both modules share the same languago-model backbone with only a lightweight LoRA head added. Furthermore, introduce novel compression-quallity reward combined with RL-based fine-tur fine-tuning to improve CTD quality, resulting in more robust compression and stronger out-of-distte -iitt--diisriittion gencralization.
We curale two datasets required for training the conlext-relevant task descriptor and contexl-aware sentence encoder of our proposed TPC.
We propose task-agnostic prompt compression a new problem formulation, where the compression method must. work universally without manual user intervention via hand-crafted prompting. Our proposed solution matches or outperforms existing methods on both task-aware and task-agnostic compression sctups, while using up 14x fewer parameters (0.5B vs. 7B) and shows strong gencralization across tasks and domains. To enable quick reproduction and further development, we will release the datasets and the codebase upon acceptance.
2

---

Published in Transactions on Machine Learning Research (11/2025)
2 Related Works
2.1
Prompt Compression
Recent efforts in prompt compression focus on reducing the inference cost of LLMs. A key line of research involves model-agnostic methods, which leverage pre-trained models for compression. Early works like token pruning during the forward pass (Kim et al. 2022) and recursive context summarization (Chen et al. 2023) introduced effective strategies but required access to the pre-trained LLM, which is often impractical. To address this limitation, newer approaches such as L.MMLnnuu (Jiang et al.) 2023b) utilize token-level perplexities to filter out semantically insignificant tokens, while high-perplexity tokens using decoder-only models like LLaMa and GPT-2. LongLLMLingua (Jiang al.) 2023c) further refines this idea by integrating question relevance for context-aware compression. However, these methods often lack adaptability to new domains, restricting their broader applicability.
Conttett (Li et al. 2023a retains
In parallel, trainable methods for prompt compression have also gained traction as key research direction. Soft prompt techmiques (Wang et al.) 2024 Bulatov et al. 2022; Chevalier et al. 2023) fine-tune or pre-train LLMs to achieve high compression rates, though they provide limited interpretability and control over the compression ratio. Seruuence-t-sequeene
models compress context by generating a summary directly (Xul et al. 2024), but their autoregressive design introduces latency. Reinforcement learning approaches, such as optimizing for simplicity, fluency, and salience (Laban al.) 2021), or using compression ratio as reward 2024). offer alternative, but may fall short in task-aware tasks. Recent innovations, such Pan et al. 2024). propose models that evaluate and prune lokens based on their information value, providing more systematic approach to compression. Despite these advances, challenges remain in balancing efficiency, accuracy, and domain adaptability. More recently CPC Liskavets et al. (2025) proposed to utilize a context-aware sentence encoder to find the relevance of each sentence in the context to remove irrelevant sentences from the input prompt, achieving SOTA performance on existing benchmarks. However, a major limitation of such an approach its reliance on the input question to guide the compression, requiring manual prompt formats for different tasks. While there have been some efforts towards building task-agnostic prompt compression Pan et al. (2024), the performance of such methods falls short of the task-aware counterpart. Our goal in this work is to develop a task-agnostic prompt compression method without sacrificing performance.
(Jung& & Kim-
2.2 Sentence Embedding Learning
Sentence embedding learning aims to create high-dimensional vector representations of text that capture semantic meaning, facilitating tasks such as text classification, clustering, and similarity search. Early approaches like GloVe (Pennington et al.] 2014) and Word2Voc (Mikolov et al. 2013) focused on word- or token-level embeddings. Later, sentence-leve) representation learning gained allention, with works like Reimers
& Gurevych (2019) fine-tuning BERT-based models (Vaswani et al. 2017) to extract sentence embeddings
Beltagy et al. 2020 enhanced the effectiveness of these embeddings, particularly for longer contexts. Later, SehnaGGhade et al.
for measuring text similarities. Subsequent research (Li et al. 2023b Wang et al. 2022
(2024) leveraged the extensive knowledge of pre-trained LIMS to build robust sentence encoders. However, while these models excel at sentence representation, they lack context awareness, which is an essential property for our prompt compression approach. While more recent work, CPC (Liskavets et al., 2025) explored context-aware sentence encoding compress prompts based on their relevance to the question, the proposed encoder was not task-agnostic. In this work, introduce task-agnostic and context-aware sentence encoder, where the embedding captures the semantics of the context.
3 Proposed Method
3.1 Problem Statement
The goal of prompt compression is to transform long input prompt into a shorter version while preserving all the information necessary for the LLM to produce an answer of comparable quality to that generated
3

---

Published in Transactions on Machine Learning Research (11/2025)
Context-relevann Task Descriptor
Context-aware Sentence Encoder
Description
Description embedding
Long input prompt
Embedding
Cosine Similarity
Selected
Segment
Segment 2
Embedding
sor
Segment
Embedding
Segmant
Embedding
Segment
Embedding
[O
Figure Illustration of our proposed prompt compression method. The CTD module generates a task description that is relevant to the input context. This description then utilized by the Context-Aware Sentence Encoder to evaluate the relevance of each sentence in the input prompt, ultimately generating the compressed prompt.
from the original prompt. Most prior works prompt compression expect both the text and explicit question inpul, using the question to assess the relevance of different text segments. This becomes limitation, since in many real-world scenarios, users of prompt compression methods may not be able or may not wish to formulate an appropriate question to accompany the text. In contrast, our mothod enables context. compression without additional manual input or lask-specific configurations. In our work, we define a task-agnostic setup, where the prompt compression method receives a text input without any explicitly marked components that can or cannot be compressed. The compression model must automatically determine which parts of the prompt are important and to what extent they should be compressed. In this setup, the model must infer the domain and structure of the prompt autonomously.
3.2 cask-agnostic Prompt Compression
The conventional setup of task-aware prompt compression operates by loveraging an explicit question (or a hand-crafted prompt where no question is available, e.g. code) provided along with the input context. c. Specifically, this approach utilizes sentence encoder fs to encode both the question and individual sentences {s1,-., )n) from the context into embeddings ey = f(()) and es, = ($1, respectively. The relevance of each sentence is then determined by scoring function (e.g. cosine similarity) R(eg, ex,), to select the top & most relevant sentences:
S olg TopK((R(e, es,) 8; E c)),
(1)
where S is the compressed prompt containing the selected sentences. The encoder fa is trained on a dataset of (q,c,pos, (neg"=I.m)) tuples learn context-aware sentence embeddings, where q is a question, the context, and pos is the positive sentence, and i=..m are negative sentences. Here, a positive is sentence containing relevant information to the question, while the negatives are context sentences containing no information regarding the question. While effective, this approach inherently relies the availability of explicit questions and cannot be adapted to tasks with no explicit questions, such as summarization.
To address the limitations of (ask-aware compression, we propose Task-agnostic Prompl Compression (TPC), a novel solution for prompt compression that does not require such explicit questions. Instead, our approach generates context-rclevant task-description á from the input prompt p using our proposod Collext-relevant Task Descriptor (CTD) model fg as Ò = (p; 8,). CTD is a lightweight module designed to generate task description that highlights the most relevant information within p. The task description & is then used to find
4

---

Published in Transactions on Machine Learning Research (11/2025)
the relevance of each sentence in the input prompt with our context-aware sentence encoder. To ensure the generated task description is semantically meaningful and contextually relevant, we train ʃg on synthetic dataset of (p, 4) pairs (curated using our proposed data curation pipeline discussed in Section 3.3p, followed by a reward-guided fine-tuning stage with our proposed reward function (Section 3.4 Finally, we propose a new method for training the Context-aware Sentence Encoder (CSE) that effectively captures the context. in the embedding (Section 3.5). Note that the CTD our proposed method is a small L.M generate the task description only; the final response generated by a pre-trained LLM (like the existing method). Overall, the lightweight CTD add negligible overhead compared to the computation cost of the LLM. The overall diagram of our proposed method is illustrated in Figure 2 and pseudo-code is provided in Algorithm (Appendix 3.7
3.3
Context-relevant Task Descriptor
CTD is the key component of our proposed solution that transforms task-agnostic prompt compression task into a task-aware prompt compression task. Specifically, CTD generates context-aware task description i from the input prompt p. The CTD comprises two primary components: (1) a dataset curation pipeline for generating high-quality (p, q) pairs, and (2) a causal encoder training process on the curated dataset to generale the task description q. Below, we describe these components detail.
3.3.1 Dataset Curation
To train the CTD, we require a dataset of (p, q) pairs, where represents long input prompt and is a relevant query or description that highlights the essential information in p. We leverage an cxisting dataset of texts Dinit and a pre-trained LLM generate these pairs. Specifically, for a given input prompt. p, the LLM is prompted to generate q using the pre-designed Prompt 1 in AA.2] By applying this prompt to dataset of long texts (c.g. Wikipedia), we generate collection of question and input context pairs.
In real-world scenarios, user prompts are often structured, typically comprising an input context, a question, and a system prompt. To ensure that our generated dataset adheres to such a structure and remains suitable for downstream lasks, we refine the initially generated dataset by prompting the same pre-trained LLM with Prompt 2 in AA2 This two-stage process yields a dataset DETD of input prompts and structured task descriptions, which is subsoquently usod to train the question generator fq. as described in the next subsection. Additionally, we provide few examples from the resulting training dataset in the Appendix.
3.3.2 Training CTD
At the core of CTD is a causal encoder, fyr designed to generate contextually relevant task descriptions. We utilize a causal language model for fqz initialized from a pre-trained LLM, to rege-segrysyively generate the task description á conditioned on the input prompt p. Formally, given input long prompt p. fy generates the question follows:
P(q 0g) = II P(q. (;
(2)
where is the t-th token in q, qet represents the sequence of tokens generated up to step t - 1, and 0g are the model parameters. We train the encoder /a to maximize the likelihood of generating the target question q given the prompt p. The Supervised Training (ST) procedure carried out using the following loss function:
T
C = -E (B
Σ
log P(ge g<t, p: (.)
(3)
where T is the total number of tokens in q.
3.4 Reward- guided Refinement
As mentioned earlier, the generated task description serves as guide for creating the compressed prompt by calculating the embedding similarity between the task description and each sentence in the prompt.
5

---

Published in Transactions on Machine Learning Research (11/2025)
Response
Pretrained LLM
Pretrained LLM
Compressed prompl
Pretrained LLM
CSE
Description
Prompt
Prompt
CTD
Figure 3: Overview of our proposed reward system for refining CTD with RL. (left) A response is generated by the pre-trained LLM with the complete long input prompt. (right) The CTD and CSE modules generale the compressed prompt. KL divergence between the conditional distribution of the generated response from the long prompt and the compressed prompt is used as the reward signal for the RL.
Consequently, the quality of the compressed prompt is directly dependent on the generated lask description. While the dataset DCTD generated using the initial prompting pipeline provides a diverse set of (p. q) pairs for training fg. there is no explicit constraint to ensure that & generated by the LLM is the most relevant task description for input prompt To this end, propose a novel reward function for fine-tuning Sg through cross-entropy reinforcement learning. The reward function is specifically designed to achieve efficient prompt. compression by generating a task description & that ensures the performance of the compressed prompt is on par with that of the original uncompressed input prompt.
Consider, task-descriptor f as a model that performs the action of generating candidate task-description from an input prompt p of a real-world human instruction dataset (e.g. Tulu SFT dataset [vison al. (2023) ) which is then used to generate the compressed prompt S و p) using Eq. [ We define the reward function as the KT. divergence between the conditional probability of generating the response from the whole input prompt and the compressed prompt S as:
R(a) KKL(((r S() p))
(4)
where, is the response from the pre-trained LLM using the whole input prompt p fLLM(P), and P(r[S) is the conditional probability of generating the response r given the compressed prompt S with a pre-trained LLM fLLM,
P(r S) = (S;;0),
(5)
and P(r p) is the conditional probability of generating the response given the original input prompt p. The KL divergence measures how much the distribution P(r S) deviates from P(r p), which we use as the reward signal for encouraging the model to generate an informative task description & for compressing the prompt. Finally, using fa as an agent to generate multiple task descriptions q (actions), and calculating the corresponding reward, we fino-tune the task descriptor LM with the following loss:
CAL
Σ
log P(ga (ist,,
(6)
where dj is the generated task-description with maximal reward from Eq 4 The overall process is illustrated in Figure
3.5
Context-aware Sentence Encoder
CSE the sentence encoder our method, which captures the information of the surrounding context the embedding of a sentence. Training CSE consists of two components discussed below: data curation and encoder training.

---

Published in Transactions on Machine Learning Research (11/2025)
LongBench
Methods
SongleDoc
MultiDoc
FewShot
Summ.
Syuth.
Code
Tokens
Full context
45.32
65.90
26.94
70.47
60.00
73.99
57.10
losk-wwore compression
Selective- Con.
29.74
43.86
23.61
57.95
22.75
45.32
1,925
LLMLingus
20.48
37.02
21.23
55.99
10.5
43.78
1,856
LLMLingua-2
33.23
52.4
24.64
46.74
26.0
59.94
1,868
Long LongLLMLingui
26.61
39.85
22.37
58.53
14.25
29.08
1,926
CompAct
19.8
40.55
59.91
831
FAVICOMP
27.13
51.65
41.63
95
CPC
43.79
62.67
25.8
67.
54.0
56.74
1,936
TPC-Base
44.06
62.25
25.66
70.1
53.75
59.72
1,973
TPC-Large
42.96
58.38
25.34
69.34
52.75
58.9
1,983
TPC-Huge
44.7
64.02
25.84
68.91
54.0
60.04
1,969
task-agnostic compression
Selective-Con
14.72
13.08
22.04
21.09
1.0
19.3
1,756
LLMLingua
21.02
34.79
21.32
54.57
7.00
64.62
1,870
LLMLingus-2
20.18
7.00
22.86
13.02
1.34
23.3
1,866
TPC-Base
40.70
58.
25.66
55.95
27.38
38.36
1,934
TPC-Large
41.59
57.59
26.26
47 86
42.0
35,25
1,978
TPC-Huge
41.84
61.93
26.22
58.27
39.0
36.25
1,962
Table 1: Performance of different methods in task-aware and task-agnostic sctups on LongBench.
3.5.1 Dataset Curation
To train the context-aware sentence encoder, we require a dataset consisting of tuples containing long contexts, questions, positive sentence, and negalive sentences. The context, provides all the relevant information necessary to answer the question, positives are sentences within the context that contain partial but not necessarily complete information needed to answer the question, while negative sentences are context sentences that provide no relevant information. While a similar dataset (CQR) was introduced in CPC Liskavets et al. (2025). the simple prompt used in the pipeline did not explicitly ensure that all the information not present in the positive alone for encouraging understanding of the context. To this end, we introduce a multi-hop quest question-answer prompt for generating a new datasel of contexts, questions, positive sentence, and negative sentences using Prompt 3 in A.2 The multi-hop question-answer prompt ensures that the generated questions require reasoning multiple sentences in the context to arrive at answer, thereby promoting deeper contextual understanding and ensuring that the positive does not contain the entire information required to answer the question. We denote the generated data of (q. pos, {negi=1-"")) tuples as the Multi-hop Context-Question Relevant dataset (MCQR), which is then used to train the CSE fs.
3.5.2 Encoder Training
CSE is trained to learn the context-aware sentence representations by distinguishing the positives and negatives. Specifically, this loss maximizes the similarity between the embeddings of the question and the positive sentence while minimizing the similarity between the question and the embeddings of negative sentences. Similar to CPC Liskavets et al. (2025), we use contrastive loss along with the MNTP loss BehnamGhader al. (2024) for more stable training on our newly curated MCQR dataset, with another key modification. Specifically, we introduce two new types of tokens into the tokenizer dictionary: the end-of-sent end-of-sentence token <end_of_sents and the question token <end_of <end_of__uessions. The <end_of_sent _eent>____e_t_o token is inserted at the end of cach particular sentence in the text, explicitly marking sentence boundarics and enabling the model to process lext segments of varying lengths elfectively. This token enables the adaptation to different input gramularities while maintaining a clear structural understanding of the text. Similarly, the <end <end_of_ed _tt_ooff_______e_ee_____e_e_ token is appended to the end of cach question, signalling its conclusion and helping the model handle questions with diverse syntactic structures. Together, these tokens improve the model's ability to generate embeddings that encode contextual and structural relationships more effectively.
7

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm Reinforcement Learning for CTD Relinement
Input: Pre-trained task descriptor fq: candidate task descriptions (Ø:), pre-trained LLM fLLM, input
prompt response r
Output: Refined task descriptor fg with updated parameters 0g
11. Generate candidate task descriptions using fq
{0) = Sq(p; 0g)
1/ For each qi: construct compressed prompt S; using 6
S. p)g.p) Vg, E {
// Compute initial response using fLLM and the initial non-compressed prompt
= (P)
// Evaluate candidate task descriptions based on KL divergence reward
R(q+) (Pr S p)) 4q, {4}
// Optimize Ig using cross entropy RL for the generated task-description with maximal reward qj
CRI = - log P(gj, (i.ces 0,)
Update 0g to minimize LRL Return refined fq
3.6 Inference
During inference, we improve the computational and memory efficiencies of deploying TPC by unifying the CTD and CSE encoders. As both tasks utilize the same pretrained encoder, employ pre-trained encoder with lask-specific LoRA adapters for CTD and CSE. The adapters are activated sequentially to generate from with CTD, followed by processing (p, 9) with CSE.
3.7 Algorithm
Next, provide the pseudo-code of the overall pipeline of our proposed prompt compression method TPC in Algorithm 8 the pseudo-code for context-relevant task descriptor in Algorithm 2 and CTD refinement with RL in Algorithm
4 Experiments
In this section, we discuss the experiments and results of our proposed solution for prompt compression. We begin by presenting the datasets, evaluation protocols, and implementation details. Subsequently, discuss the main results, ablation studies, and qualitative findings TPC.
4.1 Datasets
Evaluation datascts. To evaluate the performance of TPC, we adherc the experimental protocols outlined in prior research Jiang et al. (2023c) and benchmark our method on LongBench Bai et al. (2024) and ZeroSCROLLI Shaham et al. (2023) datasets. LongBench features diverse range of tasks, including both single-document and multi-document QA, sunmarization, few-shot learning, synthetic and code generation. Similarly, ZeroSCROLLS offers a collection of tasks, including GovReport, SummScreenfDD. QMSUM, SQuality, Quality, NarrativeQA, Qasper, Musique, SpaceDigest, BookSumSort, and Tokens.
Our curated datascts. As outlined the Methods section, TPC trained using two of our curated datasets: CTD and MCQR. The CTD dataset comprises 4,100 context-question pairs specifically designed for training the CTD module. We initialize the seed dataset for training the CTD by combining Tulu-3-sft-
8

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm Context-relevant Task Descriptor (CTD)
Input: Prompl P, pre-trained LLM 9, dataset D, training parameters 0g-
Output: Context-relevant Question Generator fg
// Stage 1: Dataset Curation
Initialize empty dataset Draw for each prompt p E 'D do Generate raw question q using gi q = g(p; Prompt 1)
Append (p,q) q) to Draw end for
// Stage 2: Structured Prompt Creation
Initialize empty dataset DOTD for each (p.9)= E Draw do
Generate structured prompt Pstruct using gi Pstruct g(p, g; Prompt 2) Append (Patrett q) to DCID end for
// Train Causal Encoder fg Initialize fa with pre-trained weights 0g
repeat
Sample batch (pstruct, 4) from DETD
Compute loss:
CCTD=- = -t log P(ge 9<t: Pstruct: 0%) Update parameters 8, using gradient descent until convergence criteria met
Return trained question generator Ja
Algorithm 3 Task-Agnostic Prompt Compression (TPC)
Input: Prompt p, context c, number of top sentences k context-aware sentence encoder fe, context-relevant task descriptor LM, fg.
Output: Compressed prompt S
// Generate task description & from prompt p using fq
i = [n(p;
// Encode task description q and each sentence s; in context using sentence encoder f.
C9 fQ), (cs,...., C., (s1, sn), Vs;
// Compute relevance scores for all sentences in
R₂ = R(eg, e,;),, Vs; e
// Select top-t sentences based on relevance scores
S TopK((R; 8; E Ec), kt)
Return compressed prompt
mixturd MetaMMathFewshoff Magicoder-Eoe Eecelerr-vvo---se Exrol-nnttrutt-1100 110K consisting of 3,000 prompts. The MCQR dataset includes 9,300 samples, each consisting of question, context, positive and negative sentences.
4.2 Evaluation Protocols
We conduct a scries of experiments across multiple downstream tasks, following widely accepted protocols on the LongBench and ZerOSCROLLS subsets. Our approach adheres to the standard evaluation protocols established prior studies Jiang et al. (2023c). Specifically, for summarization tasks, we assess our method by comparing the Rouge scores (Lin 2004) between the ground truth responses and the outputs generated by the model using compressed prompts. In document QA tasks, the model's responses are evaluated against
2 http:///www...aaaaaat////////aaa 3 http://ww../.ttttte//////t/////a Ewn.edrrre-ll----tteeeee 110K

---

Published in Transactions on Machine Learning Research (11/2025)
CeOOL
SummScreente
Narralive
SpaceDigest Digest
BookSumSort
GovReport
Methods
QMSUM
SQuality
Quality
Qasper
Masique
Space
Tokens
tosk-aware compression
Praetive-Contexx
17.92
9.91
12.43
15.57
76.19
33.48
32.78
29.76
39.52
75.33
1,925
LLMLingua
15.03
6.52
11.03
12.14
71.43
23.08
15.85
35.00
43.15
56.17
1,856
LLMLingua-2
17.60
12.73
15.25
17.37
80.95
34.75
33.87
30.00
37.20
64.98
1,868
LongLLMLingua
0.00
0.00
0.00
14.45
71.43
21.6
22.81
0.00
0.00
0.00
1,926
CompAct
11.64
13.1
66.67
17.75
4.42
20.00
831
FAVICOMP
13.14
13.0
42.86
35.86
24.05
31.76
95
CPC
18.67
12.44
17.06
16.49
100.0
45.84
41.24
29.76
54.36
80.51
1,936
TPC-Base
18.67
12.26
15.78
17.00
90.48
11.75
41.58
29.76
53.90
80.36
2,027
TPC-Large
18.31
13.54
16.72
16.97
90.48
46.42
32.86
34.76
57.21
80.45
1,993
TPC-Huge
19.30
12.14
16.38
16.64
100.00
40.75
41.35
29.76
61.42
83.33
1,941
task-agnostic compression
Selective-
17.94
10.71
7.96
13.14
33.33
1.14
25.58
33.33
19.31
76.5
1,756
LLMLingua
13.75
7.37
10.15
10.56
28.57
12.56
21.50
20.0
34.53
24.54
1,870
L.MTingua-2
18.24
13.09
11.07
16.33
38.10
1.11
26.74
39.76
29.49
63.63
1,866
TPCCBase
19.72
13.07
16.94
17.29
90.91
19.97
16.54
33.11 11
58.73
23.00
2,032
TPC-Large
18.48
12.30
16.02
17.29
90.91
21.6
26.7
33.29
59.51
5.24
1,978
JPCC-uge
17.73
12.40
16.28
17.26
86.36
16.42
15.19
24.62
63.2
48.56
2,012
Table 2: Performance of different methods in task-aware and task-agnostic setups on ZeroSCROLLs.
the ground truth using the Fl score. For code completion tasks, we rely textual similarity measure derived from the Levenshtein edit distance (Yujian & Bo 2007).
We present the main results in two evaluation setups, including task-aware compression and task-agnostic compression. lask-aware compression is the most common setup adopted by existing works where the long context is provided with a question, and the compression is performed based on the question and a system prompt. However, this limits its applicability cortain tasks such as summarization and code, where a manual hand-designed and lask-specifie prompt template is required for the compression. On the contrary, task-agnostic compression does not require such template context. We use GPT-4o as the pretrained model to generate final answers. All main results reported using a compressed prompt limited to 2,000 tokens. We also provide additional extensive evaluations for other generative models (GPT-3.5-turbo, Tlama-3.1 1aitaa3..-8B--mmee 1-8B-Instrucc) in A.3 We further assess the scalability of our method with respect to the target compression length by conducting evaluations and comparisons against competitive baselines on the LongBench dataset at 3,000-token context length. The results are reported in Table
explicit question to compress the
4.3 Implementation Details
We the subset of pile dataset Gao et al. (2020) as the seed dataset, Dinit in CTD.. We train the fo with our curated CTD dataset for 2 epochs with an AdamW optimizer, learning rate of 1.5e-4, and batch size of 16. This is followed by the reward-guided RL training for 3 iterations with Llama-3.1-8B Dubey al. (2024) as the pre-trained LLM for the reward function. To obtain questions in the RL stage, we select 16 questions per prompt. using Nucleous Sampling Holtzman et al.] with lemperature = 0.7 and topP = 0.9.
CSE is trained with an AdamW optimizer for epochs, a learning rate of 5e-5, and a batch size of 32, We introduce three versions of TPC, namcly Base, Large, and Huge, consisting of Qwen2 Yang et al. (2024)
with 0.5B parameters, 1ama-332-Insttree Dubey et al. (2024) with 1B parameters, and distral-Instruct v0.2 Jiang et al. (023a) with 7B parameters the encoder, respectively. We initialize the encoder with the
10

---

Published in Transactions on Machine Learning Research (11/2025)
LongBench
Methods
SiugleDoc
MultiDoc
FewShot
Summ.
Synth.
Code
Avg
Full context
45.32
65.90
26.94
70.47
60.00
73.99
57.10
Task-oware results
LLMLingua-1
25.94
38.20
22.27
60.01
12.25
51.13
31.97
LongLLMLingua
30.23
43.87
23.41
59.67
20.75
31.81
34.96
LLMLingua-2
38.20
57.13
25.59
53.33
35.5
63.01
45.46
TPC-B
43.43
60.52
26.39
70.91
52.75
64.85
53.14
TPC-H
43.73
61.02
26.59
71.66
53.45
65.74
53.70
TPC-L
43.38
59.88
25.09
69.50
53.75
65.18
52.79
Task-agnostic results
LLMLingua-1
24.75
39.02
22.34
58.34
7.75
68.52
36.79
LLMLingua-2
26.09
12.77
24.11
22.76
9.00
31.46
21.03
TPC-B
41.69
58.81
26.12
56.49
34.81
39.71
42.94
TPC-L
41.97
59.32
26.63
48.33
42.54
38.94
42.29
TPC-H
42.91
62.04
26.80
59.30
44.60
39.40
45.01
Table 3: Task-aware and lask-agnostic results (3K context length) on LongBench.
Setup
TPC-B TPC-L TPC-H CPC
Ablation
Perf.
Ablation
Perf.
TPC
41.62
CSE
51.28
2K task-aware
52.59
51.28
52,92
51.78
TPC w/o RL
37.97
CSE w/o MCQR
49.7
3K task-aware
53.14
52.78
53.69
$2.71
TPC w/o SFT and RL 25.92
CSE w/o spec. tokens
50.23
tasl-agnostic
41.04
41.76
43.91
3K task-agnostic
42.93
42.28
45.01
(a) Ablation on CTD.
(b) Ablation on CSE.
(c) Performance vs, context length.
Table 4: Ablation and analysis of different components of TPC on LongBench. In (c), B, L, and H represent Base, Large, and Huge, respectively.
pre-trained weights and fine-tune it with LoRA Hu et al. 2022 of rank 16, and follow the training details from BehnamGhader et al. (022)) to turn the pre-trained causal encoder into bidirectional sentence encoder.
All the experiments are conducted on an Nvidia A100 80GB GPU.
4.4 Main Results
We present the main results of our experiments LongBench in Table both task-aware and task-agnostic compression setups. We perform the evaluations all three sizes of our model TTPC-Base, TPC-Large, and TPC-Huge). As we find from this table, in the task-aware setup, TPC outperforms all the prior works across the tasks, with up to 3.3 points improvement on individual tasks. While being considerably smaller than the existing SOTA (CPC), our smallest model (TPC-Base) outperforms it on most setups.
Similarly, TPC shows strong performance on the task-agnostic compression setup. Here, only a few of the existing methods can be adopted to the task-agnostic setup, due to the nature of the compression methods. In the task-agnostic setup, TPC outperforms existing methods by larger margin than the task-aware setup. Specifically, TPC shows up to 35.0 points improvement (on synthetic task) over the existing methods, along with 20.82 points improvement on SingleDoc, and 27.14 points improvement on MultiDoc tasks. Despite having 14x less parameters than our largest. model, our Base model achieves strong results, and considerably outperforms existing methods. This gap appears to a model family effect rather than a capacity limit: the 2B..-225-0.5B backbone TPC-Base aligns more tightly with our CTD/ CSE objectives and yields sharper and less- verbose relevance estimates than the Llama-3- 3- backbone TPC-Large, which is particularly ad advantageous for task-agnostic compression.
11

---

Published in Transactions on Machine Learning Research (11/2025)
38-
33.50
Performance
Performance
37-
36-
33.10
33.35
33.30
34
Epoch
Iteration
Figure 4: Scnsitivity study on CTD training epochs (left) and the number of RL itcrations (right). Here, an RL iteration of 0 indicates no soward-guiee refinement training.
The results the ZeroSCROLLs dataset are presented in Table 2 Similar to LongBench, the results are presented on both task-aware and task-agnostic compression setups for the three different model sizes. As we find from this table, on the task-aware setup, TPC outperforms prior works by to 7.06 points (SpaceDigest). Notably, TPC outperforms existing methods by a larger margin on the task-agnostic setup. Specifically, we observe up to 51.81 points improvement on the Quality task. Additionally, we observe 28.67 and 9.04 points improvements for SpaceDigest and NarrativeQA tasks, respectively. Notably, our basc encoder performs comparable to that of the large encoder, surpassing it in few tasks, possibly due to the relative simplicity of the tasks in ZeroSCROLLs dataset.
While we believe that LongBench and ZeroSCROLLs are the most representative benchmarks, we provide experiments on a broader set of datasets in the appendix.
4.5 Ablation
Here, we present ablation and sensitivity experiments on our proposed TPC. All the results this section are reported as the averaged metrics on the LongBench dataset, with the GPT-4o model utilized as the pre-trained LLM to generate the responses. In Table 4aa) study the impact of different components of the CTD module, including the supervised fine-tuning and the reward-guided refinement. We also provide results for the non con-iineuuned CTD model (before applying SFT and RL stages) for comparison. As we find from this experiment, removing the reward-guided refinement module shows a 3.65 point drop in the performance, while removing the ST module shows another 12.05 points drop in the overall performance, showing the importance of both modules the final performance of the model. Note that RL is dependent on the ST module, and therefore can not be ablated alone.
Next, we ablate two novel components in CSE, presented in Table 4(b). First, we remove the new multi-hop dataset MCQR and train CSE with CQR instead. This results in a 1.58-point drop in performance, indicating the impact of our curated MCQR dataset over the existing CQR dataset for training a context-aware sentence encoder. Then, we remove the special tokens (<end_of _f___of__et>>>>> and <end_of encoder, leading to a 1.05-point drop in performance. These experiments demonstrate the importance of each component our context-aware sentence encoder.
fand__o__uuetiioooo _quuetioo>) from the
Next, in Table [()) we present additional studies on different constraints for context length, including 2,000 and 3,000 tokens, and compare them with the prior SOTA CPC. From this study, we observe that TPC outperforms CPC under both constraint setups, where TPC-Huge outperforms CPC with a larger margin on the 2K constraint compared to the 3K token constraint.
Next, we perform additional ablation studies to further evaluate the effectiveness of our prompt compression method. To assess the contribution of the CTD module, we replace with a substantially larger LLM, Sentral-7B-Instuutt-ee which we prompt to generate a task description. For scoring the relevance of text segments (sentences) to this task description, employ CPC. To further evaluate the CSE module,
12

---

Published in Transactions on Machine Learning Research (11/2025)
LongBench
Methods
SingleDoc
MultiDoc
FewShot
Summ.
Synth.
Code
Full context
45.32
65.90
26.94
70.47
60.00
73.99
task-aware compression
Mistral + CPCC
40.05
56.31
25.30
27.42
16.59
29.14
LLM2Vec
4.83
1.14
12.11
0.00
0.75
17.67
TPC-H
44.70
64.02
25.84
68.91
54.00
60.04
agnosigi compression
Mistral + CPC
4.60
1.25
11.55
0.00
0.00
19.51
LLM2Vec
1.20
0.21
2.12
3.65
3.75
5.61
TPC-H
41.84
61.93
26.22
58.27
39.00
36.25
Table Evaluation of Mistral--enerated task descriptions with CPC relevance scoring and LLM2Vee basclines in task-aware and task-agnostic setups on LongBench.
substitute it with LLM2Vec under both task-aware and task-agnostic sctups. In the task-aware sctup, we use the per-example question provided by LongBench as the lask description, and for lasks that lack an explicit question, we use fixed system prompt. We also use a fixed system prompt for the task-agnostic setup. Additionally, to ensure fair measurement, we leverage the prompt-guided retrieval capability of LeM by prepending the prefix "Given a user prompt, retrieve relevant passages that address the prompt:" the task description. Both experiments are conducted generation model and are summarized in Table 5 As we can see, the simple use of an LLM to generate task descriptions combined with CPC for selecting relevant chunks of the prompt performs reasonably the aaske setup, but on the agno-tgi setup it shows very poor results. This indicates the low stability of this approach to prompt distortions during compression. Also, our findings indicate that employing LLM2Vec as prompt-couditioned encoder is insufficient to achieve effective prompt compression, which demonstrates the importance of the CSE module.
the LongBench benchmark using GPT-4o as the
Finally, to further investigate the generalization of our proposed method perform additional experiments coherency, intent preservation, contextual information preservation using LLM-as-a-judge. Further details of this experiment are provided in the Appendix A.3
4.6 Sensitivity Analysis
Finally, we investigate two critical aaper-parameters of our method: the number of epochs for training the CTD model and the number of iterations for fine-tuning the CTD using RL. The number of training epochs is crucial to ensure that the task descriptions generated by the CTD model are sufficiently accurate for the subsequent reward-guided fine-tuning stage. As shown in Figure g (left), the model's performance drops after 2 epochs of training due to overfiuting. The impact of the number of RL iterations is illustrated in Figure 4 (right). Here, iteration represents the baseline performance without RL finc-tuning. As we find from this figure, the performance initially improves with the number iterations. However, the performance drops after 2 iterations likely due overfitting to the recent policies.
4.7 Latency
In Figure 5 we present the latency evaluations of our method in both task-agnostic and lask-aware setups, comparing it with other prompt compression techniques. As shown in the figure, although the latency of our method slightly higher in the task-agnostic sctup due to the overhead introduced by the CTD module, it remains lower than that of most SOTA prompt compression methods, while achieving significantly better performance than existing solutions.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Sclective-Cunext
15000
CPC
LLMLingua22
Latency (tts)
Latency (tuss)
LLMLingua
10000
LongLIMLinuu
TPC- Base
TPC-Large
5000
TPC-Huge
20418
8192 Prompt length (tokens)
4096
16384
2048
4096
8192
16384
Prompt length (tokens)
Figure 5: Latency estimates in milliseconds of the competitor methods and the TPC method in task-agnostic setup (left) and TPC method in task-aware setup (right).
5 Conclusion
We introduce TPC, a general-purpose prompt compression method that, unlike existing methods doesn't 't require an explicit question or handcrafted prompt for generating the compressed prompt. Instead, our method uses a task descriptor to generate a context-relevant task description, which is then utilized to find the relevance of each sentence in the prompt to produce the compressed prompt using CSE. Experiments LongBench and ZerOCCROLLS demonstrate that our approach outperforms existing methods in both task-aware and task-agnostic compression settings. While achieving strong performance, our method is compulationally elficient compared lo existing SOTA. Our smallest variant with considerably fewer paramelers performs on par with the previous SOTA.
References
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang. Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Doug, Jie Tang, and Juanzi Li. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3119-3193111 3137, 2024.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LLM2Vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling, 2024. URL
Iz Beltagy, Matthew E. Peters, and Arman Cohan. arXiv:2004.05150 2020.
Longformer:
The long-document transformer.
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. Advances in Newral Information Processing Systems, pp. 11079-11091, 2022.
Howard Chen, Ramakanth Pasumuru, Jason E Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading. arXiv preprinl ar Xrx:22330 2023.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Conference on Empirical Methods in Natural Language Processing, pp. 3829 3846, 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Conference on Empirical Methods in Natural Language Processing. 2023.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. survey in-context learning. In Yaser Al-
14

---

Published in Transactions on Machine Learning Research (11/2025)
Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Conference on Empirical Methods in Natural Language Processing, pp. 1107-1128, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. Xrvi preprint arxiv:2100 01.0000... 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew E Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Ta Beltagy, et al. Camels a changing climate: Enhancing Im adaptation with tulu 2. CoRR, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arviv preprint Xrxi::2310.1 06825, 2023a.
Huiqiang Jiang, Qianhui Wu, Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Compressing prompts for accelerated inference of large language models. Tn Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13358-13376, 2023b.
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing Ilms in long context scenarios via prompt compression. In Annual Meeting of the Association for Computational Linguistics, 2023c.
Hoyoun Jung and Kyung-Joong Kim. Discrete prompt compression with reinforcement learning. IEEE Access, 2024.
Schoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers. In ACM SIGKDD Conference on Knowledge Discovery and Dala Mining, pp. 784-794, 2022.
Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A Hearst. Keep it simple: Unsupervised simplification of multi- multi-paragraph text. In International Joint Conference on Natural Language Processing, pp. 0365-6378, 2021.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retriieval-uugneneee generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems, pp. 9459-9999999 940994974444 2020.
Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency large language models. In Conference on Empirical Methods in Natural Language Processing, pp. 6342-6353, 2023a.
Zchan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Mcishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint Xiw:2308.00288. 2023b.
Chin- Chwn-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Sunmarization Branches Oul, pp. 74 81, 2004.
15

---

Published in Transactions on Machine Learning Research (11/2025)
Barys Liskavets, Maxim Ushakov, Shuvendu Roy, Mark Klibanov, Ali Etemad, and Shane Luke. Prompt compression with context-aware sentence encoding for fast and improved Ilm inference. AAAT Conference on Artificial Intelligence, 2025.
Feihu Ma, Xuechen Liang, and Meiling Tao. Linguashrink: Reducing token overhead with psycholinguistics. IEEE Access, 12:145995-145105, 2024. doi:
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficiinnt estimation of word representations in vector space. Xrv preprint arXiv:1301.378. 2013.
Norman Mu, Jonathan Lu, Michael Lavery, and David A Wagner. A closer look at system prompt robustness. CoRR, 2025.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadal- lah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint S3w:23360027077, 2023.
Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang, LLMLingua-2: Data distillation for efficient and faithful task-agnostic prompt compression. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties: ACL, pp. 963 981, August 2024.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representa- tion. In Conference Empirical Methods in Natural Language Processing, pp. 1532-1543, 2014.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In International Joint Conference on Nataral Language Processing, pp.
2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sulawika, Zaid Alyaleai, Antoine Chaffin, Arnaud Stiegler, Teven Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
Uri Shaham, Maor Tvgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: zero-shot benchmark for long text understanding, In Conference on Empirical Methods in Natural Language Processing, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Cangqing Wang, Yutian Yang, Ruisi Li, Dan Sun, Ruicong Cai, Yuzhu Zhang, and Chengqian Fu. Adapting 1mss for efficient context processing through soft prompt compression. In International Conference on Modeling, Natural Language Processing and Machine Learning, pp. 91 91-97, 2024.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint 2rX.::2212.03533, 2022.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Annual Meeting of the Association for Computational Linguistics, pp. 13508,13508, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1 121011/.222222///// 754. URL https:: https://aaclanthology wttg//22..aaaeeo aec.acl-lloe long. 754/
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations.
Jason Wei, Xuczhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thoughh prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24883 2022.
16

---

Published in Transactions on Machine Learning Research (11/2025)
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-angmentel Ims with context compression and selective augmentation. In International Conference on Learning Representalions, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arX 12.1511,, 2024.
Li Yujian and Liu Bo. A normalized levenshtein distance metric. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1091-1095, 2007.
Yi Zhao, Zuchao Li, Hai Zhao, Baoyuan Qi, and Liu Guoming, DAC: A dynamic attention-awar approach for task-agnostic prompt compression. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mo- hammad Taher Pilehvar (eds.), Annual Meeling of the Association for Compulational Linguistics, pp- 2025.
A Appendix
A.1 Qualitative Results
In this section, we first present an example demonstrating the CSE module in action. Figure g illustrates the importance of each sentence in the given context, as captured by the <end_of_sent sent> tokens. darker colour indicates higher importance.
Next, provide example of compression in coding task in Figure First, the CTD module generates the 9 as:
CTD obtained task description
What is the code for the 'testrww function in the 'TestSeek' class?
Next, the CSE module generates the compression scores using 4. The colour intensity in Figure demonstratos the importance score. The final compressed prompt is visualized in Figure
Finally, present example of summarization in Figure 10 Here, the CTD first generates the & as:
CTD obtained task description ğ: Write a one-page summary of the report by the government agency on the review process and documentation of clinical
at VA medical centers.
The ÿ is then utilized by CSE to generate the compressed prompt as
Finally, we provide examples illustrating the distinction between task-agnostic and task-aware prompt compression below. In the task-agnostic sctup, no prior knowledge about the input is assumed; the input is treated as plain text. In contrast, the task-aware setup assumes that the inpul prompt can be segmnented into distinct components, including the system prompt template I, question(s) Q, and context C, which can be compressed.
Task agnostic prompt compression:
Please select the answers to the questions presented to you and write out the results in the form of table. First, read text carefully, then give your answer. Here are the texts:
Once upon time, dinosaurs lived the world, was about 240 million years ago. Among them, stood out.
Thus, climate change, provoked by the fall of meteorite, put end the of dinosaurs.
Questions:
When did dinosaurs live on earth?
What types of dinosaurs are described in the text?
What caused the end of the dinosaur era?
Answer the questions in the form of a table with three columns: details. write no more than five sentences detail for each question
question, short answer,
17

---

Published in Transactions on Machine Learning Research (11/2025)
gäven report by government agency. one-page surmary report.
Report:
centers required reviews ical care after concerns were raised about their care, but officials at these nedical centers could not provide docunentation to show that conducted. The redical centers lacked docunentation showing that retrospective reviews-which assess the previously delivered by provider during specific
through
selected
providers'
tsost half these reviews
conducted ut their clinical tare, One nedical center lacked cocunentation shawing that review had been conducted for another 12 providers after conce rns had been raised about their care. In the absence of any documentation, 12 previders,
providers
concerns
unable to identify the types of reviews, If that review providers' nical care a tinely nonner. VA concurred with these recomnendations and described plans for VHA to revise existing policy and update the standardized oudit tool used the networks to include comprehensive oversight VA medical centers" reviews providers clinical raised. Now,
selected
centers
always
concerns
one-page sumtary the regort.
Sumnary:
Figure 6: Compressed summarization prompt after taking the top-scored segments.
Task prompt compression:
I="*#
Please select the answers to the questions presented to and write out the results in the form of table. First, read the text carefully, then give your answer.
Here are the texts: [C}
Questions:
(Q)
Answer the questions in the form of a table with three columns;
question, short answer,
details. Write no more than sentences in detail for each question
Q="="
When did dinosaurs live earth?
What types of dinosaurs are described in the text? What caused end the dinosaur era?
C="*"
Once upon time, dinosaurs lived in the world, it was about 240 million years ago. Among them, stood out.
Thus, climate change, provoked by the fall meteorite, put to the era dinosaurs.
A.2 Dataset curation prompts
Prompt
You are tasked with writing user queries based long context. Consider the topic of the context when formulating the query. Queries should be concise and specific. You may also ask more complex questions, such requesting specific knowledge extraction from the text, extending the text, answering questions based on the text, paraphrasing/rewritin the text, or summarizing the text. not include the word "context" the query. Long context: (text)
Query:
Prompt 2:
You tasked with writing integrates this context and a question into single instruction. The template must include the placeholders (text) {question).
query based long context. Create complex template that
Template examples:
1. You are given a text and a question related to it. Answer the question based on the text. Question: (question)
Now (teex))uestion
Text: (text)
the question:
2. Can you extend the following block of code: (text)
such that it satisfies
requirement: (question)
Now write the code:
Provide various templates, taking into account the topic of the text and the question.
18

---

Published in Transactions on Machine Learning Research (11/2025)
Datasets
Methods
Krapivin
SummScreenFD PubMed
Meetingbank
Tosk-oware evaluation
CPC
0.435
00.2183
0.275 0.261
0.1766
TPC (ours)
0.628
0.205
0.181
Task-agnostic evaluation
CPC with fixed question
0.439
0.201
0.270
0.172
CPC w w/o question
0.418
0.190
0.240
0.156
TPC (ours)
0.724
0.205
0.276
0.186
Table 6: Task-aware and task-agnostic evaluation.
Prompt
You are given long text consisting of numbered sentences. Your task is to generate complex multi-hop questions about this text, such that answering them requires step-by-step reasoning (in multiple hops). To achieve this, first, ask series of sequential factual questions and identify the corresponding sentences that contain the answers to these questions. Then, formulate final question that can only be answered by combining the information from the previously generated questions and Additionally, specify which sentences (by their numbers) contain the information necessary to answer the final question.
Toy, example: [[1]] John is married to Mary. [[2]] They've decided to spend their marriage anniversary in Spain. [[3]] Mary was afraid that their two small children Jody and Sue, were too small for a flight. [[4]] That's 's why she asked her elder sister Jane to look after them. Questions:
Question 1: Who is John married to?
Answer 1: John is married to Mary, as stated [[1]].
Question 2; How many children does Mary have?
Answer 2: Mary has two children, as stated [[3]].
Combining the questions to create a multi-hop question:
1. John is married to Mary,
stated in [[1]].
2. Mary has two children, as stated in [[3]]. Final question: Necessary sentences: [[t]], [[3]]
many children does John have?
Now, solve this example:
(text)
Questions:
A.3 Experiments
To further highlight the efficiency of our method against CPC, we conduct experiments on 4 additional out-of- distribution datasets. The results are presented Table As evident from the results, while being times smaller (1B parameters) than CPC (7B parameters), TPC outperforms on keyword extraction (Krapivin), and summarizing meetings (meetingbank), and performs as per on the PubMed and SummScreenF'D) datasets. More importantly, unlike CPC, TPC does not require any handcrafted prompts, explicit question format, prompt template to work on these new tasks. Furthermore, TPC outperforms CPC with larger margin in the prompt-agnostic setup. These results emphasize the effectiveness and generalization of the task-agnostic nature of TPC.
Next, we compare our method with a concurrent work, DAC (Zhao et al. 2025 on a subset of the datasets on which they evaluated their method. The results are presented in Table 7 As we find from this table, our method considerably outperforms DAC on most datasets, with DAC performing marginally better on only 3 datasets.
Next, we compare our method against LanguaShrink Ma et al. (024 under the same evaluation protocol described in their paper. The results, sunmarized in Table indicate that our approach marginally outperforms LanguaShrink on 4 of the LongBench tasks.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Please complete the code gäven below. #! /usr/bin/env python
"T"Test for the sndfile class. "!!
XXX: there is lot to refactor here class Testsndfflle(Test(ase))
def rest_basic_io(setf]]
Check open, close and basic read/write""" dirty
def test basic :o_fd(self):
Check open from fd works""n
def
:est_raw(selff):
rawname DOOLCTTTST_DATADDDE 'test. raw')
"""Check float32 write/read works"""
"""Check 32 bits pcm write/read works""" """Check 16 bits pcm write/read works"""
def cesst_read_write(seee
dtype):
#def test_supported srrrsuuupprte_ffftttursssettff """Check for bad arguments." "m.
-.. Try seek really far. -m.
e. class____
cass Testsee
test rest_simple(seliie
ofilename join(TEST DATA DIR, 'test.wav')
Open the test file for reading test_ nw(se)))::
"""Test read/write pointers for seek. unn ofilename join(TEST_DATT) D_TA_AATAA,,, 'test.wav') Next line of code:
Figure 7: Compressed coding prompt after taking the top-scored segments.
Datasets
Methods
NarrativeQA
Multifield..A
HotpotQA
2WilaMAA
GovReport
MultiNws
Musique
QMSum
Qusper
DAC
24.85
40.12
42.37
22.44
30.40
25.77
TPC (ours)
23.72
37.75
50.25
45.06
53.87
26.78
31.15
25.45
21.82
Table 7: Comparison of DAC and TPC across nine datasets.
A.4
Generalization to alternative LLM models
To further demonstrate the generalizability of our method to alternative models, we conduct additional evaluations on GPT--3.-tuurb and 1--ma-3.1-8B-Issttar The results are presented in Tables and 9
A.5
Generalization evaluation using LLM-as-a-Judge
We conduct additional evaluations to assoss the generalization capability of our prompt compression method, measuring quality across multiple dimensions using an LLM-as-a-Judge. Specifically, we evaluate the compressed prompts along the following dimensions:
Coherency: We assess how consistent the compressed prompt is with the original version by asking an LLM to rate its coherence on a scale from 1 to 5.
Intent preservation (IR): We evaluate how well the compressed prompt retains the user's original intent, as expressed in the uncompressed prompt
Contextual information preservation (CIP): We assess the extent which the compressed prompt enables the LLM to generate response similar to what it would produce when given the original prompt
20

---

Published in Transactions on Machine Learning Research (11/2025)
GPrr-.5-turbo (context length 2K)
Model
SingleDoc
MaltiDoc
FewShot
Summ.
Syuth
Code
Avg
Task-aware selup
LLMLingua
22.40
32.10
24.50
61.20
10.40
56.80
34.60
LLMLingna-2
29.80
33.10
25.30
66.40
21.30
58.90 39.10
LanguaShrink
42.10
54.30
26.30
62.30
33.00
58.40 46.10
TPC-B
42.54
49.56
21.80
67.79
55.63
55.96 49.38
TPC-L
42.92
47.95
24.88
68.17
56.13
57.70 49.62
TPC-H
43.92
50.28
25.43
68.87
55.63
59.53 50.61
Task-agnoss setup
LLMLingua
19.02
26.84
19.55
50.72
6.25
38.99
26.90
LLMLingua-2
18.38
6.86
22.09
20.78
0.78
31.11 16.67
TPC-L
37.65
42.99
23.74
59.36
30.75
37.38 38.64
TPC-B
40.67
47.09
23.44
60.98
31.66
39.6 40.58
TPC-H
12.93
47.47
23.30
61.47
13.38
40.93 43.25
Table 8: Performance of GPT-3.5-turbo on LongBench (context length 2K) across task-aware and task-
agnostic sctups.
Llama-3.1 1-1-----3-----Inttuue Instructt (context length 2K)
Model
SingleDoc
MaltiDoc
Few
Summ.
Syuth
Code
Avg
Full context
44.83
46.10
24.14
34.48
53.26
13.27
36.01
Task-aware setup
LLMLingus-1
13.86
17.09
20.42
51.74
4.94
38.73
24.46
LLMLingus-2
19.17
24.65
21.31
54.39
12.93
33.96
27.73
TPC-B
36.56
42.48
25.05
65.29
48.85
41.81
43.34
TPC-L
35.26
39.91
24.89
65.15
49.13
41.85
42.70
TPC-H
36.68
43.37
25.38
66.38
47.28
42.07
43.52
Task-agnastic setup
LLMLingua-I
10.95
9.68
20.30
48.56
2.33
34.67
21.08
L.MMinguaa2
15.85
4.37
17.02
19.58
1.68
23.85
13.72
TPC-B
33.40
33.41
23.88
41.46
17.02
32.60
30.29
TPC-L
31.93
33.42
22.92
25.94
22.75 79
31.02
28.00
TPC-H
26.11
17.49
20.52
45.24
33.72
34.72
29.63
Table 9: Performance of Llama-3.1-8B- Insttrctttttttt on LongBench (context length 2K) across task-aware and task-agnostic setups.
The evaluations are conducted the following out-of-distribution datasets:
Self-Tnstruct Wang et al. (2023) which includes complex prompts generated by LLM
P3 Sanh et al. (2022) & subset of crowd-sourced instructions and demonstrations sampled from the
Public Pool of Prompts (P3) dataset
UltraChat Ding al. 2003)) subset of Long instruction- followiing prompts curated for UltraLM.
Examples include multi- paragrap instructions, simulated conversations, and educational tasks
SystemCheck Mu et al. (2025) A subset of the dataset containing generated user queries targeting 14 test-only system prompts with irrelevant in-context task demonstrations
A subset of OpenOrea Mukherjoe al. (2023), extended OpenAI prompts (via SharcGPT FLAN Wei et al.), many with long, deeply nested reasoning tasks
21

---

Published in Transactions on Machine Learning Research (11/2025)
Dataset
Method
Coherency CIP Winrate
Comperison with LLMLingua-2
Financial News Prompts
LLMLingua-2
2.5
2.5
1.6
0.6
Financial News Prompts
TPC
4.7
3.4
Self-Instruct
LLMLingua-2
2.2
2.5
1.8
0.2
Self-Instruct
TPC
3.6
2.8
2.4
0.8
P3
LLMLingua-2
3.2
3.6
2.9
0.4
P3
TPC
2.8
2.3
0.6
UltraChat
LLMLingna-2
2.4
2.9
2.3
0.1
UltraChat
TPC
3.7
3.4
2.7
0.9
DostemCheck
LLMLingua-2
1.8
2.8
2.4
0.1
SystemCheck
TPC
3.6
3.1
2.3
0.9
OpenOrca
LLMLingua-2
2.8
3.1
2.5
0.1
OpenOrca
TPC
3.4
3.3
2.4
0.9
Comparison with SelectiveConterr
Financial News Prompts SelectiveContext
2.6
2.6
1.8
0.1
Financial News Prompts
TPC
3.6
3.4
Self-Instruct
SelectiveContext
2.7
3.1
2.1
0.1
Self-Instruct
TPC
3.6
2.8
2.4
P3
SelectiveContext
3.2
3.1
2.7
0.1
P3
TPC
2.9
2.8
2.3
UltraChat
SelectiveContext
2.3
2.4
1.7
0.0
UltraChat
TPC
3.7
3.4
2.7
SystemChede
SelectiveContext
1.9
2.4
1.6
0.0
SystemCheck
TPC
3.6
3.1
2.3
1.0
OpenOrca
SelectiveContext
2.7
3.0
2.3
0.2
OpenOrca
TPC
3.4
3.3
2.4
0.8
Comparison with LLMLingua-1
Financial News Prompts
LLMLingua-1
1.7
1.6
1.2
0.0
Financial News Prompts
TPC
4.7
3.4
Self-Instruct
LLMLingua-1
2.2
2.4
2.0
0.0
Self-Instruct
TPC
3.6
2.8
2.4
P3
LLMLingua-1
1.9
2.6
1.8
0.0
P3
TPC
2.9
2.8
2.3
UltraChat
LLMLingua-1
1.5
1.7
1.3
0.0
UltraChat
TPC
3.7
3.4
2.7
1.0
SystemCheck
LLMLingua-1
1.5
1.6
1.5
0.0
SystemCheck
TPC
3.6
3.1
2.3
1.0
OpenOrca
LLMLingua-1
1.8
2.2
1.7
0.0
OpenOrea
TPC
3.4
3.3
2.4
1.0
Table 10: Comparison of TPC against LLMLingua-2, SelectiveContext, and LLMLingua-1 across multiple datasets using LLM-as-a-Judge.
These experiments are conducted on a randomly sampled subset of 200 examples from each dataset. As shown in the Table I0 our method consistently achieves higher scores than existing approaches. Additionally, for each original instruction prompt from given dataset, we directly ask the GPT-4.1 LLM to choose between the compressed prompt generated by our method and that produced by a competing method. We define the win rate of compressed prompt A over B as the number of times A is preferred over B, divided by the total number of samples from that dataset. A higher win rate thus reflects a more effective prompt compression strategy. According to the evaluation results, GPT-4.1 consistently prefers prompts compressed by our method over those produced by LLMLingua-2. We also observe a substantial performance advanlage over other task agnostic prompt compression baselines.
22

---

Published in Transactions on Machine Learning Research (11/2025)
Please complete the code given below. kend of sent> #! /usr/bin/env python kend_of_sen
# Last Change: Sun Dec 14 07:00 PM 2008 <end_of of_sent>
"""Test for the sndfile c class. "". kend_of_sents
from os. path import join, dirname <end of_sent> import <end_os lan_off_eents import sys <end_os jon__of__ett>
from numpy. testing import TestCase, assert assar__array_equal dec <end sodd_of_senii Dente import numpy from audiolab import Sndfile, Format, available encooingg,, available_fille file_formaaaeeeee <end <end_of_see oe_st testcommon import open open_tmp_file
<end of__tt
flose_tmp_file TEST_DATA_ <end_of_ sand_of_sent' of_sent>
_ {np. float64 'float64', no.t2o: 'float32', <end_of_sents
np.int32: 'pem32', np.int16: 'pcm16') <end_of of_ sent> XXX: there is a lot to refactor here kend_of send_of_seee of__sent> class Testsndffle(Testt(sse)) and_of__eents
Snafile(ofilenmme, 'r') send_of_sents
a. sceck(-bufssiz,, 1) <end_ot send_of_scnte
buff2) <end send_of_sent>>
a. close() <end_of sodd_of_saee _en__of_eet>>>>
Now, read some frames, go back, and compare buffers <end_of of__ent> (check whence Snafile(ofilennme, 'r') send_of_sents buff a. frrread_frameeenrrrrrrr a.seck(-buffsize, <end of_sent>
SEEK END) xend of sents
send_of_sent>
buff2 = frames (buffsize) kend_of of__set>> assert_array_equal (bbuff [-buffsize:], buff2) <end_of sentt def test_nw(self); send_of_sents>
"""Test read/write pointers for seek. kend of sents ofilename Join(TEST DATISS___AAA_AAAAA DIR, 'test.wav') <end of sents Next line of code: kend_of of _ent=
Figure 8: CSE segment scores colormap for coding prompt. Darker colours indicate higher scores.
A.6 Licenses
We used pre-trained models and open-source datasets. In terms of models, we use models from the Qwen-2.5 family and Mistral family, which are licensed under the APACHE-2.0 license. We also used the LLama-3.2 model, which licensed under the "LLAMA 3.2 COMMUNITY LICENSE AGREEMENT". All the models used are available for research purposes.
As our training datasct, we used Commons License Attribution family", as well as "abacusai/ aabacusai/MetaMaatheeea MMetaMMttFeesshoo" and "ise-uiue/ uuiue/MgMaMddeer- mooo-nnstruct-100". which are licensed under the APACHE-2.0 license. We assume that these data follow the well-known HHH paradigm. Therefore, taking data from these sources should not introduce potentially malicious behaviour into our models. We also used the Pile dataset, which is licensed under the MIT license. We conduct manual checks on subsamples of the datasets to verify the cleanliness and safety of the data. In addition, provide full details on how created the datasets in Section 3.5.1
which is licensed under the "Open Data
We evaluate our method on benchmarks such as MIT-licensed LongBench and MiT-licensed ZeroScrolls. All a aforementioned models, datasets, and benchmarks are publicly available and imply research use. Their prescribed terms of use are hereby aligned with our research work.
23

---

Published in Transactions on Machine Learning Research (11/2025)
Read the following text and i, Arrenian innfessinnal fostball
seerryysend.of..o sents Foothall Club Urarts translated Furtollaynn Akrb Urartal, comsonly capital 2015, In 2013-2824, they
Urart that currently plays in the Arnenian Prenier League. club Prenier Leegue
Armenian cup
thear history. major club after purchasing was officially renared FC Urartu. of sents History
end_at_sents In early ly 2016, the Russia-tases Areenian businessmen Ozhevas cheloyants becate co-omee Banents until August 2019,
part the club shares. The club was known sond_af seats kotaykoend of sents Urarts were founded as Banants by Sarkis Israclyan on 21 Janurry 1992 in the village of Ketayk, repre senting the Kotayk Province. He nered the club after his native village of Benents lcurrently by known as Beyan). Between 1992 and 1985, the clu was common ly referred to as Benants Katays, Buring the 1992 season, the club won the first Amenian cup. At the end of the 1995 transitionn season, Banants suffered a financial crisis. The club owners decided that it was better to serge the club with FC Kotayk of Abovyan, rather than sisband it. In 2061, Banants deverged from FC kotayk, and was noves fron Abovyan to the copstal Yerevan, send of sonts Verevankand ot sen is FC Benants was relocated to Yerevan in 2001. At the beg inning of 2883, Banants serged with FC Spartak Yerevan, but was able to Lisit the n are of the new ne nerger to TC Banants, Spartak becaee Banants's 's youth acadery and later changed the nome to Danants-2. Because of the merger, B arants acquired many players from Spartak Yerevan, inc including Servel Relkonyan, After the nerger, Banants took more serious appreach and hav finsshed highly in the league table ever since. The club ranaged to lift the Armenian Cup in 2007. send of sonty Experience is naking way to youth for the 2005 and 2009 seasons, The departures of nost of the experienced players have left the club's future to the youth. Along with Ugandan international, Noeh Kasule, has been sigred. send sents club headquarters Donesticsend
two Ukrainian the Salatin-Sebastia District, Yerevan. send the Banants Stadius launched in 2806 the Malatia- Sebastia District of Europeansend.c Further developrents
located Jivani Street Stadiuneend_
The construction with assistance of FIFA programme. playing pitch nnd (2,750 the northern stand, 1,509 at souther stand academy
officially opened in 2688 with capacity emized and the capecity the stadiun was incressed up to 4,860 sents Basants Trasning Centre i5 the
3,600
ipplamented later
western stand). send send_or sents Training contre/acadee//eed So-Setastia District Yereven, In adcition the nain stadiun, the centre houses 3 full-size training pitches, nini football pitches 11 indoor facility. The current technical director of the academy is forner Russian foothaller Iishat Faizulin, sand of sents Fanses me_nf sentt The rost active group of fans is the South West Ultras fan club, manlyy composed of residents from several reightourhads within the Hs Salatia-Seeesti District of Yerevan, since the club i5 de tacto representer of the district, Heabers of the fan club benerst troe even ts organized by the club and nany facilities of the Banants training centre, such as the mini foothall pitch, the club store and other entert aineests, send of sonts tiiieveenntseedd of sents Amentann Pronter Leaguesent of sents Kinter (3]: 2813 14, sand of sentx Sunnor-up up (51: 208 3, 2806, 2007, 2018, 2018, send of sents Arrenian Cupsend_ sents Kinter (3): 1992, 2007, cand__ of sents Runner-up (6); 2043, 2004, 28 16, 2069, 2018, 2821-22eend, of sents Armenian supercupsend_oft of_ sents Winner (1): 2014,send, send of sents Runner-ug 15): 2004, 2007, 2009, 2010, 2010, send_of____ of_entt sents Current squadiend of_nttts Out on Loansand of sents Personnelsand of sents Technical staffsend_ sents hanagenentseend_ert sente Urartu 2xend of sents FC Banants' reserve squad play FC Bonants- 2 in the Arsonian First League. They play their hase ganes at the to aining field with artificial turf of the Irartu Training Centre. of sents Managerial lal historycend sents Veruchan Suklasyen (1992-94) üganes Zonazanyan (280)-95)<en__ Postutka (2887)
located in the
rf_of_ nd_of_sent= Poghos Galstyan Quuly 1, 1996
1998)
ashot Barseghyan (2085-0
Nikolay Kastov 12uly Gyctbusighyants (Jan S>vens_ot_sene Wikelay Kiselyov (eedd_0))-mn____ooo_oe 2808)-end seeents Kin Splidstoel (2868) of__ents
Aoril 8, 2009
son_s Nedatcho Matushev (Aprit 2088
2009 Bec 1, of
Barseghyan (interin) of sents Stevica
1, 2018 Dec 2810) of sen
2013] of_netts Zsolt Hornyek (Juy 2013 May 30, 2815)=end_ot _t_tett- Aran Voskanyan Duly 2015 Oct 11, send)- ot__ettttttttt Tito Rarallo
Refael Neceryen
2011
15, 2012) sentes Volodyeyr Pystenko
2013
12, 2015 Aug 11, 2018) sest> IIshat Faizulin (Aag 12, -Noy 24, 2019) sente Aleks
16] of_eett Artur Veskanyan (Oct
andr Crigoryan (Nov 25, 2810 .Mar 18, 2021) of sents Robeett Arzuzaryan (18 March 2021-24 June 2022) send, of sents Deltri Gunke (27 Juna 2022-)cend of sents biw, answer the following question based on the abave text, only give me the anser and do not output any other words..en s_of_sents question: where is the club's 's headquarters located of Answert Band of
Figure 9: Visualization of the operation of the compression model with special lokens. Each <end_of ___n>> token is highlighted with an intensity proportional to its importance in terms of answering the question.
You are given a report by a government ageney. send of sente Write one-page sunrary of the report.
send of sents
Report:
send_of_sentss
We found that fron October 2013 through March 2817, the five selected VA redical conters required reviews of a total of 148 providers' clini cal care after concerns were raised about their care, but officials at these nedical centers could not provide documentation to show that at nost half these reviews were conducted. of
lle found that all five VA medical centers lacked at least sone docunentation of the reviews they told us they conducted, and in sone cases, we found that the required reviews were not conducted at all. send of sents Specifically, across the five VA redical centers, we found
nedical centers lacked docunentation showing that type roviee- focused prefessionsl practice evaluations (FPPE) for cause-had been conducted far 26 providers after concerns had been raised about t heir care. send_of_sente
following:
FPPEs for cause revieus of providers' care over a specified period of tire, during which the provider continues to see patients and has the caportunity to demonstrate inprovement, send of sents
Pocunentation of these reviews is explicitly required under VHA policy. send of sents
Additionally, redical center officials confirsed
for cause that were required for another 21 providers were never conducted. &
end of sents
The nedical centers lacked documentation showingg that retrospective reviews-which assess the care previously delivered by a provider during specific period of tire- had been conducted providers after concerns had been reised about their clinical care, send_of_sent
A commtteee within the VA medical center had recomencee that the provider's privileges be revoked prior to the agresrent. send_af_sentz There was docurentation of the reasons why this provider was not reported to the NPDB under VHA policy. sead_of sead_of_sent>> sents
NPD8 and state Licensing boards and VMA oversight these processes, we recomn ended that VHA require its networks to establish a process for overseeing VA nedical centers to ensure they are reporting to the NPOB and to state Licensing beards and to ensure that this reporting is timely kend_of_ sents it plans include oversight of tirely reporting the NPD8 and state Licensing boe
ingrove nedical centers' regorting of providers
concurred with this recornendation and told us rds as part of the standard audit tool used by the networks. send sents If or your staff nembers have any questions concerning this testinony, please contact ne (282) 512-7114 futllianson kend_e
[_sents
Contact points for our Office of Congressional Relations and Public Affairs ray be found on the last page of this statenent. send_of_sent' other individuals who made key contributions to this testirony include Marcia A. Mann (Assistant Director), Kastlin M. Mctonnell (Ana lyst-in -Cherge), Sunnar C. Corley, Krister Friday, Jacquelyn Hanilton. cend_of_sents The publishee product nay be repraduced and distributed in its entirety without further permission from GAO. send_af_sents Now, write a one-page sumnary of the repart.
send sents
Sunrary: Kend of sents
Figure 10: CSE segment scores colormap for summarization prompt. Darker colours indicate higher scores.
24


Reviews:
[{"review_id": "lrqZU1Hm7i", "reviewer": "Reviewer_eS4v", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with prompt compression, the task of reducing prompt length while retaining the important information to provide accurate answers, thus reducing inference time and costs. Unlike various recent approaches that require explicit questions, the paper focuses on the task-agnostic setting. Specifically, the proposed approach trains a module that generates context-relevant task descriptions in lieu of the explicit questions and fine-tune this module using RL with reward function that aims to improve the final generated answer. Experiments on LongBench and ZeroSCROLLS show gains over the baselines, particularly in the prompt-agnostic setting.\n\nStrengths:\n- The problem addressed is important and of interest\n- Overall, the paper is written well and the technical approach is clearly described\n- The experiments show improvement over the baselines, in both task-aware and task-agnostic (with gains that are much more significant in the task-agnostic setting).\n\n\nWeaknesses:\n1. The importance of the context-agnostic setting could be further explained/motivated: the paper notes that approaches like CPC \"cannot be adapted to tasks with no explicit questions, such as summarization”. This claim could be explained in more details: why is there no question in summarization tasks? There should be some instruction that indicates that the goal is to summarize the text. Why can't this instruction be used as the question in the CPC pipeline?\n2. Not clear why RL is applied specifically to CTD. Given that the RL training is operating on the complete task, taking into account the result of the CSE module, it is not clear why the RL training is limited to operating on the CTD module. For example, the same RL training could have been operating on the CSE module (leaving the CTD module frozen) or on both.\n3. Simple baselines for CTD is missing, namely simply using the result of a strong LLM (e.g., GPT-4o, Claude 4 Sonnet, DeepSeek V3/R1) prompted to extract a question (which seems similar to how the CTD training data was created, although not clear what model was used for that purpose). While in this case we will not be able to apply the RL training on the LLM-based CTD component, as noted in my previous comment, it sounds like it may be equally applicable on the CSE module.\n4. Similarly, LLM-based task description together with CPC would have been a very useful baseline for the task-agnostic setting.\n5. Not clear how to interpret the ablation analyses: why the max result in Table 3(a) and Table 3(b) are different? The top row in both Table should be representing the full setting of CTD and CSE respectively, assuming the other component is also provided in full setting. So it is not clear to me why the performance in the full setting is not the same for both tables.\n6. The results presented for some baselines do not match their original papers.Specifically, there are very large gaps between the results reported for LLMLingua-2 in the manuscript vs. the original paper: SingleDoc (35.5 vs 20.18), MultiDoc (38.7 vs. 7.0), Few-Shot (69.6 vs. 13.02), etc. This is very concerning and I could not find any explanation for these gaps in the paper.", "claims_and_evidence_yn": "No", "claims_explanation": "As per the list of weaknesses above, I have concerns about insufficient evidence provided in the paper to support the presented approach and the claims about its performance, specifically w.r.t the importance/relevance of this setting (weakness #1), the need/usefulness for RL in the proposed setting (weakness #2), the importantly the experimental evaluation including the selected baselines, the reported results for the baselines, and the ablation study (Weaknesses #3, #4, #5, #6).", "audience_yn": "Yes", "audience_explanation": "As noted in the strength above, the topic is important and likely to be of interest to various individuals in TMLR's audience as: (1) it has the potential of reducing inference time and costs in LLMs; (2) it builds on and compare to published literature in the field.", "requested_changes": "Based on my comments above, the requested changes are:\n- Clarify the motivation for the proposed task-agnostic setting\n- Explain the gaps in performance of baselines\n- Describe the role of RL and whether it could have been applied to the CSE module or both\n- Add simpler baselines for CTD and for the task-agnostic CPC, or explain why they are not applicable\n- Clarify the differences between the full setting in the ablation study of both CTD and CSE and why they are not conducted starting from the same full setting", "broader_impact_concerns": "I have no concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dgjrzuvjy3", "reviewer": "Reviewer_xXkJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors claim the paper proposes a novel reward function, reinforcement based fine tuning, data curation for two datasets, and a task-agnostic prompt compression method. \n\nIn my opinion, the contribution that can be claimed in this paper is only the generation of task description from a long prompt to replace explicit questions.\n\n- The reward function is not novel, it is KL divergence.\n- The fine-tuning is not reinforcement learning.\n- The datasets are not curated but generated by an LLM.\n- The compression pipeline is described in another previous work.", "claims_and_evidence_yn": "No", "claims_explanation": "The authors claim that the work is introducing RL based fine-tuning that uses a novel reward function and cross-entropy loss. \nI would not call the methodology described in the paper a RL. It is not clear how this method could be considered a RL approach (other than using the phrase \"reward function\"). The methodology describes a cross-entropy loss and the provided equations are not cross entropy. The novel reward function the authors presented is KL divergence. The utilization of KL in this task might be novel but the function cannot be claimed as novel.\n\nFurthermore, in order to claim data curation as contribution, I would expect to see some of the claims made in the paper to be analyzed and presented. The data generated are provided by pre-trained LLMs. Authors claim that the pipeline proposed generates diverse pairs for q-p, the questions require reasoning and the positives do not contain entire information required to answer the question. The data generated were not analyzed against these claims. The paper does not contain proof of such claims.\n\nMoreover, the paper suggest that the pipeline is novel, especially in providing a task agnostic pipeline. The only difference between the SOTA method (CPC) is that the pipeline utilizes an LLM to generate the questions instead of the explicit questions given in the input. The rest of the pipeline is the same as CPC. I do not see the contribution in the general pipeline.", "audience_yn": "Yes", "audience_explanation": "Although not all the contributions proposed by the authors can be claimed as part of novel work, the question generation improves on the CPC results. This could be considered as an iterative advancement of an existing work. In other words, an extension of CPC.", "requested_changes": "I believe that the following are crucial adjustments or clarifications needed for this paper:\n\nThe reward function is not novel, it is KL divergence.\n- The authors need to clarify why they claim KL divergence is a new method.\n\nThe fine-tuning is not reinforcement learning.\n- The authors need to clarify how their approach could be considered an RL methodology\n\nThe datasets are not curated but generated by an LLM.\n- The authors need to provide extended analysis of the data to provide evidence that the generated synthetic data is balanced, provides diverse examples/pairs, requires reasoning with context, and the positives do not contain entire information required to answer the question. \n\nThe compression pipeline is described in another previous work.\n- The authors need to provide enough evidence that this work is significantly different than CPC while highlighting the novel contributions beyond CPC.\n\n\nFurthermore, I noticed that especially the related work section highly resembles the related work section of CPC paper. The authors need to address this.", "broader_impact_concerns": "If the utilized pre-trained LLM models already have biases, using them to generate synthetic data without human involvement and using them to train another model risks amplifying those biases.\n\nAlthough prompt compression aims to reduce inference time and cost, the total pipeline which includes training multiple LLMs to compress the original prompt might increase the overall footprint.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JhF5XdVbCi", "reviewer": "Reviewer_Kcp7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Task-agnostic Prompt Compression (TPC), a framework for reducing prompt length in LLMs without relying on explicit user queries or handcrafted templates. Unlike most prior compression methods that depend on task-specific signals, TPC leverages a context-relevant task descriptor (a lightweight LM) to generate a task description directly from the input prompt. This description is then used with a context-aware sentence encoder to assess sentence relevance and construct the compressed prompt. To improve the quality of task descriptions, the model incorporates reinforcement learning with KL-divergence reward, ensuring that compressed prompts preserve essential information needed for accurate LLM responses. The authors evaluate TPC on two benchmarks (LongBench and ZeroSCROLLS) under both task-aware and task-agnostic setups, demonstrating that their approach outperforms the baselines.\n\n**Strengths:**\n- TPC addresses a key limitation of prior methods (e.g., CPC) by eliminating the need for explicit questions or handcrafted templates.\n- The RL fine-tuning of the CTD is elegant and directly ties compression to downstream performance.\n- The multi-hop dataset curation for CSE promotes deeper contextual reasoning\n\n**Weakness**\n- The paper lacks some key comparisons and ablation experiments to support the claims.", "claims_and_evidence_yn": "No", "claims_explanation": "While the paper presents a good approach to task-agnostic prompt compression, some key limitations warrant attention:\n\n- **Insufficient Empirical Evidence for the CSE Over Existing Methods**\nIn Section 2.2, the authors claim that models like LLM2Vec (BehnamGhader et al., 2024) \"excel at sentence-level representation but lack context awareness,\" which motivates the development of CSE as a key contribution. However, LLM2Vec supports instruction-based embeddings (e.g., prepending prompts like \"Retrieve semantically similar text\"). Similarly, MAGNET [1] (not cited in the paper) generates prompt-conditioned sentence embeddings (check Section 4.2 of [1]). The paper doesn't include experiments comparing CSE to these baselines (e.g., ablating TPC with LLM2Vec or MAGNET as the encoder), which undermines the novelty or necessity of CSE.\n\n- **Discrepancies in Baseline Results and Limited Evaluation Across Downstream LLMs**\nThe reported task-agnostic results for LLMLingua-2 in Table 1 are substantially lower than those in the original paper. Is this because a different pretrained LLM is used for generating the final answer? To demonstrate TPC's generalizability (LLM-agnosticism) evaluations across multiple models are essential, alongside explanations for baseline mismatches.\n\n- **Lack of Analysis on Compression Ratios and Uncompressed Baselines in Task-Agnostic Settings**\nTask-agnostic results are reported at a fixed ~2k token limit. There is no analysis for varying compression ratio for the task-agnostic setup, despite its focus on broad applicability. Additionally, uncompressed prompt baselines (pretrained LLM performance without compression) are not reported.\n\n- **Missing Comparisons to Recent Task-Agnostic Baselines**\nFor the task-agnostic setting, the paper doesn't compare their approach with methods like DAC [2], Selection-p [3], and LanguaShrink [4]. \n\n[1] Khosla et al., \"MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities,\" ACL 2025.\n\n[2] Zhao el al., \"DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression,\" ACL 2025.\n\n[3] Chung et al., \"Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability,\" EMNLP 2024.\n\n[4] Liang et al., \"LanguaShrink: Reducing Token Overhead with Psycholinguistics,\" ArXiv 2024.", "audience_yn": "Yes", "audience_explanation": "TMLR's audience would likely find this work appealing due to its advances in prompt efficiency, which is a key challenge in LLM deployment. However, the submission would be more impactful if the raised concerns are addressed.", "requested_changes": "I recommend the following additions to secure my recommendation for acceptance:\n- Replace CSE with off-the-shelf LLM2Vec/MAGNET encoders in the TPC pipeline and report LongBench/ZeroSCROLLS performance\n- Expand evaluations in Tables 1-2 to include additional LLMs (e.g., GPT-3.5-Turbo, Llama-3-8B).\n- In Tables 1-2 and Figure 1, report uncompressed baselines.\n- Extend ablation in Table 3(c) to task-agnostic setting.\n- Add comparison to stronger task-agnostic prompt compression methods like [2,3,4].\n- Briefly explain why TPC-Base outperform TPC-Large on most tasks.\n\nI also recommend the following minor adjustments:\n- Fix the title of the vertical axis in Figure 1. It should be “task-agnostic” and task-aware.”\n- Cite recent works like [1,2,3,4] in Section 2.\n\n[1] Khosla et al., \"MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities,\" ACL 2025.\n\n[2] Zhao el al., \"DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression,\" ACL 2025.\n\n[3] Chung et al., \"Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability,\" EMNLP 2024.\n\n[4] Liang et al., \"LanguaShrink: Reducing Token Overhead with Psycholinguistics,\" ArXiv 2024.", "broader_impact_concerns": "I think no major ethical concerns arise from this work, as it focuses on efficiency improvements for LLMs without introducing novel risks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)